Archive for the ‘Amazon Web Services’ Category

Persistent Storage on Amazon EC2 Announced

Tuesday, April 15th, 2008

The final piece in the EC2 web-hosting puzzle has been announced:

This new feature provides reliable, persistent storage volumes, for use with Amazon EC2 instances. These volumes exist independently from any Amazon EC2 instances, and will behave like raw, unformatted hard drives or block devices, which may then be formatted and configured based on the needs of your application. The volumes will be significantly more durable than the local disks within an Amazon EC2 instance. Additionally, our persistent storage feature will enable you to automatically create snapshots of your volumes and back them up to Amazon S3 for even greater reliability.

Until now the only way to get real MySQL storage has been to jump through some hoops with multiple EC2 instances or back the database onto S3 (with subsequent latency overhead). With 100GB of outbound data transfer the smallest ECV2 image will cost you about $90/month for 1.7GB of memory and a dedicated CPU core. The price just cannot be beat.

Amazon SimpleDB: death of the database?

Saturday, December 15th, 2007

With the announcement of Amazon SimpleDB Scoble has asked if Amazon Web Services are going to kill MySQL and Oracle.

I think the short answer is no, but the game is changing rapidly, and Amazon is at the vanguard.

SimpleDB basically acts like a big structured bucket.
The Model is represented by Domains. A Domain can have Attributes. Attributes are key/value pairs.

There is no schema attached to Domains, you can PUT any combination of attribute-values in the domain:
PUT (item, 123), (description, sweater), (color, blue), (color, red)
PUT (item, 456), (description, dress shirt), (color, white), (color, blue)
PUT (item, 789), (description, shoes), (color, black), (material, leather)

The real issue here is that developers are strongly tied to the RDBMS. Frameworks assume you are running on top of a relational database, so there is an instant barrier to entry. However, I suspect it won’t be long before the first set of libraries and drivers for SimpleDB are developed.

The much bigger issues are in the features that SimpleDB doesn’t have:

  • No ACID
  • No transactions (see ACID, above)
  • No relationships
  • No data integrity
  • No SQL
  • Latency

SQL is actually pretty incredible - it makes accessing structured data very simple. If you’ve ever played with other types of data store you miss SQL almost instantly (Prevayler, anyone?).

All of that said, Amazon continues to lead the way with Web Services. Consider SimpleDB a warning shot to the database incumbents.

Converting a FaceBook App to use Amazon S3

Friday, November 16th, 2007

I’ve just completed work on another Facebook application: Songbook.

This was a relatively straight forward conversion of the existing application to use S3 for data storage. Songbook allows users to upload and display a song in their Facebook Profile. The original development stored the files on disk and hit limits really fast … hosting large numbers of files becomes very expensive very quickly, not to mention performance issues.

I converted the whole thing to S3 very quickly - it’s actually easier in some respects to handle storing files in S3 than to disk. The application now has essentially limitless and very cost-effective storage capacity.

S3 really is incredible.

Locked out of the Amazon Start-Up Challenge

Monday, September 17th, 2007

Last week Amazon announced “Start-Up Challenge“:

Amazon Web Services (AWS) is searching for the next hot start-up that is leveraging AWS to build its infrastructure and business.

Do you have a great idea or an existing application that uses services from AWS? Enter the AWS Start-Up Challenge to win $100,000 in cash and AWS credits, and receive an investment offer from Amazon.

This got me all excited because I LOVE AWS! I use S3 for backup  (both personal and for live database backups) and the message-queue service is forming up to be an integral part of my “Secret Project” architecture. The money would keep me rolling for 12 months at my current burn rate too.
So I just went to register and realised that I am locked out of the challenge because I am not a US resident.

 Curse my antipodean geography!

Rake Task for MySQL Backup to Amazon S3

Saturday, August 25th, 2007

There are a couple of these around, but I rolled my own because I wanted something quite simple.

# S3 Backup Task for MySQL
# Assumes InnoDB tables
# Database User needs the "reload" permission on the database (for --flush-logs in mysqldump)
#
# Stores files in Amazon S3 using the excellent AWS Gem: http://amazon.rubyforge.org/
# For information about Amazon S3: http://aws.amazon.com/s3
#
# Installation
# 1) Install AWS Gem
# 2) Enter your S3 Bucket, access_key_id and secret_access_key in this file
# 3) Run (rake db:backup)
#
# Inspired by code from:
#   http://blog.craigambrose.com/articles/2007/03/01/a-rake-task-for-database-backups
#   http://www.rubyinside.com/advent2006/15-s3rake.html

namespace :db do
  require "aws/s3"

  desc "Backup database to Amazon S3"

  task :backup => :environment do
    BUCKET = "bucket"

    begin
      AWS::S3::Base.establish_connection!(
        :access_key_id     => "access_key_id ",
        :secret_access_key => "secret_access_key "
      )

      db_config = ActiveRecord::Base.configurations[RAILS_ENV]

      backup_path = "db/backup"
      File.makedirs(backup_path)

      datestamp = Time.now.strftime("%Y%m%d%H%M%S")
      file_name = "#{RAILS_ENV}_#{db_config['database']}_#{datestamp}.sql.gz"

      backup_file = File.join(backup_path, file_name)

      sh "mysqldump -u #{db_config['username']} -p#{db_config['password']} --single-transaction --flush-logs --add-drop-table --add-locks --create-options --disable-keys --extended-insert --quick #{db_config['database']} | gzip -c > #{backup_file}"
      puts "Created backup: #{file_name}"

      puts "Storing file in S3: #{BUCKET}"
      AWS::S3::S3Object.store(file_name, open(backup_file), BUCKET)

      puts "Backup Complete"
    rescue AWS::S3::ResponseError => error
      puts error.response.code.to_s + ": " + error.message
    end

  end

end

Managing Amazon S3 on Mac OS X

Friday, August 24th, 2007

While we’re talking Amazon S3, I found the Mac OS X S3 Browser, it works really well for managing your S3 account.

Speaking at Melbourne Ruby User Group

Friday, August 24th, 2007

I’m speaking at the Melbourne Ruby User Group next week. Either on using Amazon Web Services. or hacking with Prototype (probably focussed on the new 1.6.0 features).

Lineup & Location

ThoughtWorks
Level 11, 155 Queen St

  • Pete Yandell - SpiffyAttributes
  • Clifford Heath - HAML
  • Mike Bailey - Jester
  • Ben Schwarz  - JS Hacking
  • Toby Hede - Amazon AWS and/or Prototype Hacking
  • Marty Andrews - Logging and/or the Rails Plugin Architecture

Amazon Flexible Payment Services

Tuesday, August 7th, 2007

Amazon continues to lead the way into a service-driven future with the release of the Amazon Flexible Payments Service:

Amazon Flexible Payments Service (Amazon FPS) is the first payments service designed from the ground up specifically for developers. The set of web services APIs allows the movement of money between any two entities, humans or computers. It is built on top of Amazon’s reliable and scalable payment infrastructure.

FPS supports micro-payments and the fees look very competitive.

Payments are the last piece of the service puzzle and with the release Amazon now has a complete Operating System for services.