hubertlepicki.com : Amazon SimpleDB and Rails. BDD style.

Amazon SimpleDB and Rails. BDD style.

Amazon SimpleDB is schema-free, non-relational database engine developed by Amazon and available as paid service as one of Amazon Web Services. It’s simple to use, fast and, combined with EC2 gives you great scalability out of the box.

One of biggest problems I had to face, when developing applications that use SimpleDB backend, is that I couldn’t run SimpleDB locally. No one can. It’s not available to purchase or download, you can’t do “apt-get install amazon-simpledb”. Using remote database for development is not a thing you want to do. It’s slow when used from outside of EC2 network. You probably need separate account for each of your developers. This sucks even more if you realize that you pay for running your tests… Hey, we want to run them as much as we can in BDD style – don’t want to be bothered by CPU usage it generates.

Solution to my troubles appeared to be M/Gateway’s M/DB. This is free and open source database, compatible with most recent (2009-04) API of SimpleDB, and you can run it on your own hardware. While it is possible to compile it from source, you probably need to pay developers for support on info how to do it ;). You won’t find also any packages for Ubuntu or Debian on the Internet (but if you do, please let me know!). The easiest way to get it up and running locally is to grab VMWare image and run it locally. Images and documentation how to set it up can be found on M/DB’s site. I run it on VirtualBox, which is my favourite among desktop virtualisation technologies, and have customized image a bit, but it’s equally easy to use it it VMWare. I’ll try to write detailed instructions on how to set up M/DB in VirtualBox some time next week on this blog. You also need to create 2 databases on your M/DB instance, and save access key id an secret access keys of those.

When you have M/DB installed locally, you need M/DB adapter for Ruby. Hey, didn’t I say it’s compatible with Amazon SimpleDB? So, existing adapters should work. Sort of.

The one I am using is Appoxy’s AWS combined with Appoxy’s SimpleRecord. First library is fork of abandoned RightScale’s AWS. I have made simple modifications to it, and now, you get M/DB support out of the box in master Git branch. You’ll need to manually build and install gem until nev version is released (which I show below). Second library is ActiveRecord pattern implemented on top of SimpleDB. Not as big, powerful and famous as Rails’ ActiveRecord, but good enough to provide object-oriented abstraction for your data for daily use.

To get started with these, install required gems:

# Until we have appoxy-aws version 1.11.36, we need to build gem manually:

$ git clone git://github.com/appoxy/aws.git
Initialized empty Git repository in /home/hubert/aws/.git/
remote: Counting objects: 377, done.
remote: Compressing objects: 100% (296/296), done.
remote: Total 377 (delta 198), reused 116 (delta 53)
Receiving objects: 100% (377/377), 167.74 KiB | 108 KiB/s, done.
Resolving deltas: 100% (198/198), done.
$ cd aws
$ gem build aws.gemspec 
WARNING:  no rubyforge_project specified
WARNING:  description and summary are identical
  Successfully built RubyGem
  Name: aws
  Version: 1.11.36
  File: aws-1.11.36.gem
$ sudo gem install aws
Successfully installed aws-1.11.36
1 gem installed
Installing ri documentation for aws-1.11.36...
Installing RDoc documentation for aws-1.11.36...

# in in future we'll just do:
# gem install appoxy-aws --source http://gems.github.com

# install simple-record
$ gem install appoxy-simple_record --source http://gems.github.com

Create your new M/DB Rails project:

$ rails mdb && cd mdb

Put these two lines in your config/environment.rb:

  config.gem "aws", :lib => "right_aws"
  # uncomment line below when version 1.11.36 is ready on github, and remove line above
  # config.gem "appoxy-aws", :lib => "right_aws", :source => "http://gems.github.com"
  config.gem "appoxy-simple_record", :lib => "simple_record", :source => "http://gems.github.com"

Now, you need to connect to your local M/DB instance when starting Rails. Edit your config/environment.rb to contain:

SimpleRecord.establish_connection(
  YOUR_KEY_ID_HERE,
  YOUR_SECRET_ACCESS_KEY_HERE,
  :mode => :per_thread,
  :server => IP_OF_YOUR_VIRTUAL_MACHINE,
  :protocol => "http",
  :port => 80,
  :signature_version => 2,
  :service => "/mdb/request.mgswi"
)

I advise you use app_config or AmberBit Config to use different keys for development and test environments, which will result in using different databases.

In order to check if your database is set up properly, let’s create a model “Brick”:

class Brick < SimpleRecord::Base
  has_attributes :weight, :color
end

There’s no migrations. To create table, for bricks, start up console and type:

$ ./script/console

Loading development environment (Rails 2.3.4)
>> Brick.create_domain
=> {:box_usage=>"0.0010000000", :request_id=>"eb4e3424-6371-89c4-330c-0c808d09ec52"}
>>

Hooray! Now, let’s see some more operations:

>> Brick.create(:weight => "1kg", :color => "red")
=> #, @attributes={"weight"=>["1kg"], "id"=>"1a9d944c-afaa-11de-9808-00a0d1a202b3", "color"=>["red"], "updated"=>[Sat, 03 Oct 2009 01:19:59 +0200], "created"=>[Sat, 03 Oct 2009 01:19:59 +0200]}, @new_record=false>
>> Brick.create(:weight => "0.5kg", :color => "brown")
=> #, @attributes={"weight"=>["0.5kg"], "id"=>"23111360-afaa-11de-9808-00a0d1a202b3", "color"=>["brown"], "updated"=>[Sat, 03 Oct 2009 01:20:13 +0200], "created"=>[Sat, 03 Oct 2009 01:20:13 +0200]}, @new_record=false>
>> Brick.find(:all).first.color
=> "red"
>> Brick.find(:all).last.weight
=> "0.5kg"

Cool, huh?