I was watching the Pivotal Labs video of their luncheon with New Relic and the speaker, CEO Lewis Cirne said something that struck me as really odd yet kind of clever.
As he was describing the architecture of their system, Cirne revealed they they run a constant "synthetic" load on their production system. He referred to the faux load agents as their "canaries in the coal mine." The purpose of these canaries, as he explained it, is so that when they reach a dangerous level of traffic - when performance begins to suffer - they can kill off the fake load, freeing up resources and consequently giving them a gauge as to how much breathing room they have before they are really in trouble.
Right off the bat this ingenious ruse throws up two warning flags to me. First off, is all this fake load unnecessarily reducing the lifetime of the hardware, such as thrashing the disk drives, generating excess heat, sucking electricity, etc.? And secondly, how sure are they that this fake load is representative of real load, and hence a true measurement of spare capacity once it's removed?
What do you think?
Tuesday, October 14
Monday, October 6
Bootstrapping GitHub Access on a Fresh EC2 Instance
If you'll recall from my last post, I've spent an extended weekend getting a mature Rails application running on Amazon's Elastic Computing Cloud (EC2). Yes, I took two vacation days off work (where I suffer with Java and ATG) to spend some quality time with Rails and EC2, and more accurately Capistrano.
My short-term goal was to get the application running (which I did), then to automate deployment. Since EC2 affords you the luxury of conjuring up fresh severs on a whim, I wanted to make sure I had a simple single Capistrano task that could take a virgin machine and impregnate it with my application. (Wow, that metaphor kinda strayed into the inappropriate zone, eh?)
Out of the box, the EC2 on Rails gem gives you some handy recipes for setting up a fresh instance and deploying to it, but it misses a couple little chicken-and-egg bootstrapping niggles. One of those is GitHub access.
In order to pull your repository off GitHub, you have to have your SSH keys set up. This is generally a manual and slightly tedious process. The GitHub guides do an excellent job of walking you through it. However, I don't want to run through that rigamarole every time I christen a new server; I want that chore to be automated.
This certainly isn't the most elegant way to solve it, and I don't recommend it for a long-term large-scale operation, but here's how I handled it...
I generated the SSH keys on a single machine then copied the "id_rsa" and "id_rsa.pub" files into my "config/deploy" folder. So I now have a copy of the super secret files necessary for accessing my remote repository in the repository itself. I'm aware of how dumb this is from a security perspective, and you should be too, so just put down the flamethrower and back away from the keyboard. I'm keeping things simple and practical here; this isn't the source code for an electronic voting machine after all.
Next I created a Capistrano recipe to copy the SSH key files from my "config/deploy" directory up to the target server (being deployed to) into its "~/.ssh" directory (where the remote ssh command expects to find them). Note that this puts the same keys on every server.
I then used the "after" hook to ensure this recipe runs after the EC2 on Rails gem completes its native "setup" recipe.
Now every time I create a brand-spanking-new instance on EC2 and I run the "cap ec2onrails:setup" command it does everything it needs to do to prepare that server for receiving the only other command that needs to be run, "cap deploy".
My short-term goal was to get the application running (which I did), then to automate deployment. Since EC2 affords you the luxury of conjuring up fresh severs on a whim, I wanted to make sure I had a simple single Capistrano task that could take a virgin machine and impregnate it with my application. (Wow, that metaphor kinda strayed into the inappropriate zone, eh?)
Out of the box, the EC2 on Rails gem gives you some handy recipes for setting up a fresh instance and deploying to it, but it misses a couple little chicken-and-egg bootstrapping niggles. One of those is GitHub access.
In order to pull your repository off GitHub, you have to have your SSH keys set up. This is generally a manual and slightly tedious process. The GitHub guides do an excellent job of walking you through it. However, I don't want to run through that rigamarole every time I christen a new server; I want that chore to be automated.
This certainly isn't the most elegant way to solve it, and I don't recommend it for a long-term large-scale operation, but here's how I handled it...
I generated the SSH keys on a single machine then copied the "id_rsa" and "id_rsa.pub" files into my "config/deploy" folder. So I now have a copy of the super secret files necessary for accessing my remote repository in the repository itself. I'm aware of how dumb this is from a security perspective, and you should be too, so just put down the flamethrower and back away from the keyboard. I'm keeping things simple and practical here; this isn't the source code for an electronic voting machine after all.
Next I created a Capistrano recipe to copy the SSH key files from my "config/deploy" directory up to the target server (being deployed to) into its "~/.ssh" directory (where the remote ssh command expects to find them). Note that this puts the same keys on every server.
namespace :spumco do
namespace :ec2 do
desc "copy the GitHub keys"
task :copy_github_keys do
upload "config/deploy/id_rsa", "~/.ssh", :via => :scp
upload "config/deploy/id_rsa.pub", "~/.ssh", :via => :scp
end
end
end
I then used the "after" hook to ensure this recipe runs after the EC2 on Rails gem completes its native "setup" recipe.
after "ec2onrails:setup", "spumco:ec2:copy_github_keys"
Now every time I create a brand-spanking-new instance on EC2 and I run the "cap ec2onrails:setup" command it does everything it needs to do to prepare that server for receiving the only other command that needs to be run, "cap deploy".
Sunday, October 5
Rails on EC2: A Tale of Perseverance (Part One?)
One of my pet projects has been getting a lot of "yeah but can it really scale?" naysaying of late so I decided to find out if my on-paper theory translated to a real-world infrastructure. Essentially I need to take my all-on-one-server Rails application and split it up into several segregated modules across several servers: the main web site, main database, slave databases, web services layer, caches, queues, workers, etc. I'm not mister moneybags, so I opted to use Amazon's Elastic Computing Cloud (EC2) service to virtualize all the servers.
Up until Friday morning the only thing I knew about EC2 was what I'd read in blog posts. The first thing I had to do was RTFM. I can't stress enough how phenomenal is Amazon's documentation. Everything I needed to know to get started was spelled out step by step in the Getting Started Guide. It walks you through setting up your account, configuring your local environment for communicating with the cloud, then setting up your first virtual server. It all worked as advertised and I didn't hit any of those "oh crap I'm going to have to ask on IRC or the Google Groups WTF this means" speed bumps.
Once I was proficient with the basics of EC2, I wanted to get Rails on it, and I was pretty sure somebody out there had already done all the hard work for me. I was right, of course, and that somebody was the EC2 on Rails project. It was a bit confusing at first as there seems to be some stale information over on RubyForge but apparently the project moved to GitHub; at least that's where the most recent activity and updated documentation can be found.
EC2 on Rails is simply a Ruby gem - accompanied with an EC2 image pre-configured for hosting Rails applications - and a toolbox of Capistrano recipes for deploying your application to the cloud. Like Amazon's documentation, EC2 on Rails walks you through the process of installation and configuration pretty well. However, I did run into a plethora of thorns. The first and most painful of which was the Capistrano incompatibility - it insists on version 2.4.3. I tried at first to hack it to work with the latest, but after falling down far too many rabbit holes I gave up and conceded to downgrade my copy of Capistrano.
Once the Capistrano conflict was resolved I ran into problems with the eyecap gem and some of my app's recipes (too many definitions of the "symlink" task), then configuring the virtual server to talk to GitHub was an outside-of-the-box somewhat-convoluted experience (which I might detail in a follow-up post), and I wasn't able to initialize the database due to an issue with migrations and a bad model name conflicting with some native libraries, so I ended up scp'ing a dump from another host and importing it. But most, if not all, of those issues were related to the quirks of my Rails application and not necessarily issues with the EC2 on Rails package. After everything was ironed out, I could deploy to the cloud with a single command.
Although it spanned three calendar days I only clocked in six hours on the project; from EC2 noob to a live Rails application. Not too shabby for a weekend.
But, I've thus far only replicated what I already have: an all-on-one-server environment. My next step is to start up a half-dozen other virtual servers and start breaking off pieces of the application to see just how well it can scale up and out.
Up until Friday morning the only thing I knew about EC2 was what I'd read in blog posts. The first thing I had to do was RTFM. I can't stress enough how phenomenal is Amazon's documentation. Everything I needed to know to get started was spelled out step by step in the Getting Started Guide. It walks you through setting up your account, configuring your local environment for communicating with the cloud, then setting up your first virtual server. It all worked as advertised and I didn't hit any of those "oh crap I'm going to have to ask on IRC or the Google Groups WTF this means" speed bumps.
Once I was proficient with the basics of EC2, I wanted to get Rails on it, and I was pretty sure somebody out there had already done all the hard work for me. I was right, of course, and that somebody was the EC2 on Rails project. It was a bit confusing at first as there seems to be some stale information over on RubyForge but apparently the project moved to GitHub; at least that's where the most recent activity and updated documentation can be found.
EC2 on Rails is simply a Ruby gem - accompanied with an EC2 image pre-configured for hosting Rails applications - and a toolbox of Capistrano recipes for deploying your application to the cloud. Like Amazon's documentation, EC2 on Rails walks you through the process of installation and configuration pretty well. However, I did run into a plethora of thorns. The first and most painful of which was the Capistrano incompatibility - it insists on version 2.4.3. I tried at first to hack it to work with the latest, but after falling down far too many rabbit holes I gave up and conceded to downgrade my copy of Capistrano.
Once the Capistrano conflict was resolved I ran into problems with the eyecap gem and some of my app's recipes (too many definitions of the "symlink" task), then configuring the virtual server to talk to GitHub was an outside-of-the-box somewhat-convoluted experience (which I might detail in a follow-up post), and I wasn't able to initialize the database due to an issue with migrations and a bad model name conflicting with some native libraries, so I ended up scp'ing a dump from another host and importing it. But most, if not all, of those issues were related to the quirks of my Rails application and not necessarily issues with the EC2 on Rails package. After everything was ironed out, I could deploy to the cloud with a single command.
Although it spanned three calendar days I only clocked in six hours on the project; from EC2 noob to a live Rails application. Not too shabby for a weekend.
But, I've thus far only replicated what I already have: an all-on-one-server environment. My next step is to start up a half-dozen other virtual servers and start breaking off pieces of the application to see just how well it can scale up and out.
Subscribe to:
Posts (Atom)
