Top Secret Project

My favourite new plugin for Ruby on Rails is backup_fu.

backup_fu makes managing Amazon S3 backups really simple.

I have been running my own Rake Task for MySQL Backup to Amazon S3, but I’ve switched my projects over to backup_fu instead because it has much better control over your backups and will even backup stored files.

I’ve also created a patch for backup_fu that allows you to specify the mysqldump options. It should get rolled into an upcoming release, but if you would like it sooner, just contact me at toby…@info-architects.net.

Share This

With the announcement of Amazon SimpleDB Scoble has asked if Amazon Web Services are going to kill MySQL and Oracle.

I think the short answer is no, but the game is changing rapidly, and Amazon is at the vanguard.

SimpleDB basically acts like a big structured bucket.
The Model is represented by Domains. A Domain can have Attributes. Attributes are key/value pairs.

There is no schema attached to Domains, you can PUT any combination of attribute-values in the domain:
PUT (item, 123), (description, sweater), (color, blue), (color, red)
PUT (item, 456), (description, dress shirt), (color, white), (color, blue)
PUT (item, 789), (description, shoes), (color, black), (material, leather)


The real issue here is that developers are strongly tied to the RDBMS. Frameworks assume you are running on top of a relational database, so there is an instant barrier to entry. However, I suspect it won’t be long before the first set of libraries and drivers for SimpleDB are developed.

The much bigger issues are in the features that SimpleDB doesn’t have:

• No ACID
• No transactions (see ACID, above)
• No relationships
• No data integrity
• No SQL
• Latency

SQL is actually pretty incredible - it makes accessing structured data very simple. If you’ve ever played with other types of data store you miss SQL almost instantly (Prevayler, anyone?).

All of that said, Amazon continues to lead the way with Web Services. Consider SimpleDB a warning shot to the database incumbents.

Share This

I’ve just completed work on another Facebook application: Songbook.

This was a relatively straight forward conversion of the existing application to use S3 for data storage. Songbook allows users to upload and display a song in their Facebook Profile. The original development stored the files on disk and hit limits really fast … hosting large numbers of files becomes very expensive very quickly, not to mention performance issues.

I converted the whole thing to S3 very quickly - it’s actually easier in some respects to handle storing files in S3 than to disk. The application now has essentially limitless and very cost-effective storage capacity.

S3 really is incredible.

Share This

Last week Amazon announced “Start-Up Challenge“:

Amazon Web Services (AWS) is searching for the next hot start-up that is leveraging AWS to build its infrastructure and business.

Do you have a great idea or an existing application that uses services from AWS? Enter the AWS Start-Up Challenge to win $100,000 in cash and AWS credits, and receive an investment offer from Amazon.

This got me all excited because I LOVE AWS! I use S3 for backup  (both personal and for live database backups) and the message-queue service is forming up to be an integral part of my “Secret Project” architecture. The money would keep me rolling for 12 months at my current burn rate too.
So I just went to register and realised that I am locked out of the challenge because I am not a US resident.

 Curse my antipodean geography!

Share This

There are a couple of these around, but I rolled my own because I wanted something quite simple.

# S3 Backup Task for MySQL
# Assumes InnoDB tables
# Database User needs the "reload" permission on the database (for --flush-logs in mysqldump)
#
# Stores files in Amazon S3 using the excellent AWS Gem: http://amazon.rubyforge.org/
# For information about Amazon S3: http://aws.amazon.com/s3
#
# Installation
# 1) Install AWS Gem
# 2) Enter your S3 Bucket, access_key_id and secret_access_key in this file
# 3) Run (rake db:backup)
#
# Inspired by code from:
#   http://blog.craigambrose.com/articles/2007/03/01/a-rake-task-for-database-backups
#   http://www.rubyinside.com/advent2006/15-s3rake.html

namespace :db do
  require "aws/s3"

  desc "Backup database to Amazon S3"

  task :backup => :environment do
    BUCKET = "bucket"

    begin
      AWS::S3::Base.establish_connection!(
        :access_key_id     => "access_key_id ",
        :secret_access_key => "secret_access_key "
      )

      db_config = ActiveRecord::Base.configurations[RAILS_ENV]

      backup_path = "db/backup"
      File.makedirs(backup_path)

      datestamp = Time.now.strftime("%Y%m%d%H%M%S")
      file_name = "#{RAILS_ENV}_#{db_config['database']}_#{datestamp}.sql.gz"

      backup_file = File.join(backup_path, file_name)

      sh "mysqldump -u #{db_config['username']} -p#{db_config['password']} --single-transaction --flush-logs --add-drop-table --add-locks --create-options --disable-keys --extended-insert --quick #{db_config['database']} | gzip -c > #{backup_file}"
      puts "Created backup: #{file_name}"

      puts "Storing file in S3: #{BUCKET}"
      AWS::S3::S3Object.store(file_name, open(backup_file), BUCKET)

      puts "Backup Complete"
    rescue AWS::S3::ResponseError => error
      puts error.response.code.to_s + ": " + error.message
    end

  end

end

Share This

While we’re talking Amazon S3, I found the Mac OS X S3 Browser, it works really well for managing your S3 account.

Share This

I’m speaking at the Melbourne Ruby User Group next week. Either on using Amazon Web Services. or hacking with Prototype (probably focussed on the new 1.6.0 features).

Lineup & Location

ThoughtWorks
Level 11, 155 Queen St

• Pete Yandell - SpiffyAttributes
• Clifford Heath - HAML
• Mike Bailey - Jester
• Ben Schwarz  - JS Hacking
• Toby Hede - Amazon AWS and/or Prototype Hacking
• Marty Andrews - Logging and/or the Rails Plugin Architecture

Share This

1. attachment_fu
   From install to an Image Upload with Amazon S3 storage in about 10 minutes. See the attachment_fu tutorial.
   
   
2. RESTful rails
   An XML API for nothing? Why thanks, Rails.
   
   

The REST stuff is really fun to play with … I am finding it breaks a little in the real-world with non-trivial use-cases. Start adding auto-completes and other AJAX elements to your page and you start bending the RESTful model a little. I am figuring it’s best to have the core REST architecture and build on it, rather than bend my code to adhere mindlessly to some REST ideal.

I {heart} Rails.

Share This

Just lost an hour of my life trying to work out why the test server received 403 errors when attempting to store an image using Amazon S3.

The server date was a day out and S3 uses the date as part of its authentication scheme.

Note to self: check server date when seeing inexplicable errors in Amazon S3.

Share This