A better progress meter for your (Rails) scripts

As a follow-up to my post a couple weeks back on putting a progress meter in your long-running migrations, I've whipped up a more helpful and re-usable tool.

I've called it simply "Progress" and here it is in its entirety:

class Progress
  require 'action_view/helpers/date_helper'
  include ActionView::Helpers::DateHelper
  
  def initialize(total, interval = 10)
    @total = total
    @interval = interval
    @count = 0
    @start = Time.now
  end
  
  def tick
    @count += 1
    if 0 == @count % @interval
      sofar = Time.now
      elapsed = sofar - @start
      puts "been running for #{distance_of_time_in_words(@start, sofar)}"
      rate = elapsed / @count
      puts "at a rate of #{distance_of_time_in_words(sofar, (sofar - rate), true)} per item"
      finish = sofar + ((@total * rate) - elapsed)
      puts "should finish around #{distance_of_time_in_words(sofar, finish)}"
    end
  end
end

Usage is pretty simple. Here's an (slightly-abridged) example from a Rake task I wrote to clean out bad references to removed YouTube videos:

namespace :videos do
  desc "Remove videos from SpumCo if YouTube also removed them"
  task :purge => :environment do
    videos = Video.find(:all, :order => 'created_at asc')
    progress = Progress.new(videos.size)
    videos.each do |video|
      progress.tick
      video.destroy unless video.still_on_youtube?        
    end
  end
end

And what you see in the console as the script runs is periodic updates like so:

been running for less than a minute
at a rate of less than 5 seconds per item
should finish around 14 minutes

Put a progress meter in your long-running migrations

Uploaded with plasq's Skitch!

I'm working on a Rails project now that requires lots of database massaging and repair. This repair work needs to be tried and tested on a development workstation, reviewed on a staging server, then applied to the production system. Since the work needs to be repeatedly applied to several environments, I'm logically using migrations.

One nuisance I got sick of real quick is staring at a terminal running a long migration and wondering "is it doing anything?" and "how much longer is it going to take?" So I decided to add some progress indicators.

The simple yet effective method I've settled on is an on-screen count-down. Most of the migrations consist of the same pattern: 1) get a list of the records that need repaired, then 2) iterate over each record and repair it. I set the counter to the size of the record set I'll be iterating over, decrement it on each iteration, and print it to the screen. Seeing the numbers scroll across the screen lets me know the migration is working, and since the migrations count down to zero, I can gauge how long it's going to take to complete based on how fast the numbers are shrinking.

Here's an example:

class FixTheThingsWithTheStuff < ActiveRecord::Migration
  def self.up    
    query = <<-SQL
      select name
      from things
      where stuff = 1972
      and deleted_at is null
      group by name
      having count(id) > 1
      order by name
    SQL
    broken_rows = select_all(query)
    count = broken_rows.size
    broken_rows.each do |row|
      printf "[#{count-=1}]"
      # ... fix it! ...
    end
  end

  def self.down
    # ... re-break it! ...
  end
end

Disabling third-party services when they stop performing (in Rails)

Uploaded with plasq's Skitch!

One of my clients uses the hosted version of CompanyX (not their real name) to serve ads on their site. A couple weeks back, CompanyX applied some "upgrades" and things didn't go as planned, so for nearly a week their service was up and down like a yo-yo. That resulted in me getting calls along the lines of, "Hey our site is loading slow because of the CompanyX ads, please take them all off," followed a few hours later with another call, "Hey CompanyX seems to be OK now please turn their ads back on," and a little while later the cycle repeats itself. That got real old, real quick.

So, I decided to whip up a little automated solution. I needed two core components:

1. A way to programatically turn the ads on and off.

2. A way to periodically test the third-party service, and enable or disable the ads based on its response time.

For disabling and enabling the serving of ads, I created a new model called CompanyXStatus which is essentially a toggle switch, it's either on or off, and for auditing purposes I have it store the date and time whenever it's flipped. The database table looks like this:

create_table "companyx_statuses" do |t|
  t.column "enabled",    :boolean
  t.column "created_at", :datetime
end

And the app-facing API of the model looks like this:

class CompanyxStatus < ActiveRecord::Base
  
  class << self
    
    def enabled?
      latest.enabled
    end
    
    def disabled?
      !enabled?
    end
    
    def disable!
      CompanyxStatus.create!(:enabled => false) if enabled?
    end
    
    def enable!
      CompanyxStatus.create!(:enabled => true) if disabled?
    end

    private

    # I'm not using a named scope because this client is on an OLD version of Rails...
    def latest
      CompanyxStatus.find(:first, :order => 'created_at desc') || CompanyxStatus.new(:enabled => true)
    end

  end

end

So in the views when I'm building a page I just have to check if CompanyxStatus.enabled? before rendering an ad tags.

Now, for the actual toggling logic, I'm using the Benchmark module to call out to the service and measure the response time. If if exceeds the threshold (2.5 seconds in this case) the service is disabled, otherwise enabled. Here's the rest of the model:

  def test
    if 2.5 > latency
      CompanyxStatus.enable
    else
      CompanyxStatus.disable
    end
  end

  private

  def latency
    Benchmark::measure{ connect }.real
  end

  def connect
    socket = Socket.new( AF_INET, SOCK_STREAM, 0 )
    sockaddr = Socket.pack_sockaddr_in( 80, 'blah.companyx.org' )
    socket.connect( sockaddr )
    socket.write( "GET /blah.php HTTP/1.0\r\n\r\n" )
    results = socket.read
  end

Finally I need a way for the system to periodically make these checks and toggles so my client and I don't have to worry about babysitting the site. For this I wrote a simple rake task:

namespace :companyx do
  desc "Ping CompanyX and disable it if too slow"
  task :ping => :environment do
    CompanyxStatus.test
  end
end

And scheduled it as a cron job to run every minute:

* * * * * cd /home/client/apps/production/site/current && RAILS_ENV=production rake companyx:ping

That's it! Now I can get a good night's sleep knowing that the next time CompanyX has a burp in their service, my client's site is going to automatically shut them off until they get their act back together again.

Laptop recovery with Twitter? Scratching another itch...

One morning last week a curious idea popped into my head:



... and that afternoon I had accomplished this:



The premise is simple:

1. I set up a private Twitter for just my laptop status, and subscribe to it from my personal Twitter account.

2. I set up a cron job on my laptop to run ever hour, tweeting the geographical location of my laptop to the private account.

The theory is that if my laptop were ever stolen and the crook for some reason didn't format it before connecting it to the 'net again, I might have a fart's chance in a whirlwind of recovering it. Yeah, probably not, but it was fun to build, and on the coolness factor I'll have an automated journal of my travels.

As an added bonus, Twitter automatically rejects duplicate tweets, so the account doesn't get spammed every hour if the laptop hasn't moved (since the updates would be identical).

So how did I do it? Well the first thing I did was realize that I couldn't get my public IP address from the laptop when it was behind a NAT router, so I had to reach out to find a service that could provide me that information. My initial Google attempts came up dry and I was about to resort to scraping the IP Chicken page (blech) when a helpful Twitter follower came to my rescue:



That service is exactly what the doctor ordered (thanks @nu2rails). I used HTTParty to parse the response and Jeweler to turn my little library into a gem and tossed it up on my GitHub account for the world to enjoy.

So please check it out, try it, fork it, improve it, and let me know how you like it. Oh, and extra brownie points if you get the name of the gem!

Twitter2RSS: Scratching my own itch at Acts As Conference 2009

The beauty of RSS is that I can aggregate all of my information sources, read them at my leisure without worry of expiration, and organize them however I desire. It's always bugged me that I couldn't have that luxury with Twitter. Sure Twitter has RSS feeds, but they don't include the avatars, and they don't include direct messages, and they commonly truncate the tweets (blech). There are some pretty slick clients out there, sure, but why would I want to run yet another application for just another information feed? Why can't I have my cake and eat it too?

Last week, while I was up in Orlando for Acts As Conference, I did the same thing I do at most conferences, I started another damn pet project. Except this time I also completed it during the conference. Thanks to my bud Bryce Kerley from the Miami Ruby Meet-up for giving John Nunemaker's Twitter gem a little massaging, I had all the tools I needed to alleviate myself of the aforementioned headaches and get my Twitter goodness piped right into my RSS reader.

It's a proxy server, running on Rails (perhaps a bit overkill of a framework for such a simple application, but it's what I know best) that sucks up your Twitter business using their API and spits it back out in a consumable RSS feed, with avatars, and without truncation.

If you want to run it yourself, I've opened up the source over on GitHub at github.com/trak3r/twitter2rss

Or, if you're too lazy or don't have a cheap hosting provider, you can use mine over at twitter2rss.anachromystic.com

Enjoy!

Tracking AJAX calls with Google Analytics (and Rails)

With my recent pet project I used AJAX to provide a majority of the site functionality on the main page with it never having to reload itself or load another page. This resulted in a drastic drop of "page" views in my Google Analytics reports, because the only "page" being loaded on each visit was the sole main page; every click after that was an AJAX call to update only a portion of the already loaded page, hence not triggering any calls back to the Google Analytics tracking server. So I set out to see if I could remedy that, and I did.

I started out with a Google search for assistance. Surprisingly, the results were grim. The highest ranked result required a "donation" to read the answer, and I opted not to. Other results were vague or dated or focused on the same issue with Flash. I had to crack open the code and get my hands dirty.

When you set up a profile to track with Google Analytics, they give you a little snipped of JavaScript code to paste into your website pages. It looks something like this:

<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-1377203-6");
pageTracker._trackPageview();
} catch(err) {}</script>

Seems cryptic enough, right? If you look closely you'll see it's actually two scripts, not one. The first script imports the Google Analytics tracking library:

<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>

And the second script calls the library to track the page.

<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-1377203-6");
pageTracker._trackPageview();
} catch(err) {}</script>

In order to track a page load, including an AJAX call, you merely need to call the tracking library, but you need to import the library only once. That's where things get a little tricky.

For starters, I separated the two scripts into their own respective partials. I render both partials on the main page, so the library is loaded and then called to track the page load. However, from the AJAX calls, I only render the second script, the one that calls the library, since it will have already been loaded by the main page.

Here's the first curve ball. The first time I render and serve the main page, the AJAX-y portions of the page are rendered in-line. Since each of them in turn renders the call to the tracking library, the loading of the main page could erroneously track several hits. In order to prevent this, I wrapped the script with a global flag check and set, so no matter how many times it's rendered for a single page, it only injects the script once:

<% unless @already_tracked %>
<% @already_tracked = true %>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-1377203-6");
pageTracker._trackPageview();
} catch(err) {}</script>
<% end %>

The second curve ball is that you need to render the first partial - the one that loads the Google Analytics library - at the top of your page, rather than the bottom as Google recommends in its documentation. I put mine immediately inside the body tag. Why? Because the first partial, with an embedded tracker call, to render on the page is going to try to call the tracking library when the browser processes it, and if the tracking library isn't yet loaded, the call will silently fail (thanks to the try/catch block).

Now that the main page loads the tracking library, and the HTML snippets returned from the AJAX calls in turn call the already-loaded tracking library, each AJAX call is tracked as a page view on your Google Analytics report. Note that this solution is specific to AJAX calls that inject pre-rendered HTML into the page. If you need to track AJAX calls that deal with behind-the-scenes processing you should be able to simply make the same JavaScript calls you see in the second script; wrap them up in a helper function for convenience.

How to Launch 2 Sites in 20 Hours

Since I "became independent" exactly two months ago to the day, I've launched two pet project web sites: Pocket Rails and Rate Marina's Outfits. I tracked my time on each, just like I would for a billable client, and oddly enough they each took about twenty hours from inception to launch. Turns out it's pretty damn easy, and cheap too. Here's how I did it.

Get organized

I'm a "to do" list guy. Everything I do on a daily basis centers around "to do" lists. Whenever I think to myself, "I need to..." it goes straight on to the list. For these two projects I used Ta-da Lists. I created a new list for each project, and started adding "to do" items as I thought of them, and checked them off as I completed them. It's a great way to ensure you don't forget anything, nothing falls through the cracks, and it gives you a decent visual representation of your progress and how much you have left to do.

Get a domain

The first thing you need is a domain name for your site so people have a way to surf to it. You don't necessarily need a separate domain for every site; you can can host several sites on a single domain via sub-domains. For example, I registered anachromystic.com for my company then hosted one of my projects at marina.anachromystic.com.

I get all my domain names through GoDaddy. It's usually the cheapest, and it's convenient to manage them all through a single central service. If you plan on sending/receiving e-mail through the domain, I strongly recommend Google Apps for Business. It's dirt simple to set up, their tutorials cover every major registrar, and requires no maintenance.

Get a host

As the name "pet" project implies, these sites are hobbies. They are not generating any money, and it's not critical that they be up all the time and fast to respond. So I went with the cheapest host I could find, DreamHost. Pull up Google and search for DreamHost promo codes and try all the ones you find. I ended up getting an entire year of hosting for about $20 (that's for the entire year, not a monthly rate).

Choose a platform (Hint: Use Rails, dummy)

Not only is Ruby on Rails the best platform for getting a site up and running quickly, you can get a head start with a "base" application like Bort which comes with a plethora of pre-shaved yaks including registration, e-mail activation, log-in, password reset, pre-configured routes, deployment scripts, etc. It's not perfect - I had to tweak it a bit - but it saved me hours of laying the groundwork and let me get to the meat of the project quicker.

Use hosted source control

Why hosted? First of all, it's essentially a cheap back-up of your work. Secondly, it makes it a lot easier to collaborate if you're working with other developers. There's a billion to choose from, and if you're willing to let other people see your code, they're free. I decided to make the source for Pocket Rails open but keep the source for Marina private (for now). I'm a huge git fanboy so GitHub was the natural choice for me. My open-sourced projects are hosted for free and I pay a measly $7 per month for the privilege of keeping some of them private.

Test all the fscking time

If you aren't test infected yet, it's time to wake up. Testing demonstrates that what you've written works, and testing ensures that when you modify or enhance it you don't break any of the old stuff. Don't let yourself fall into the quagmire of, "I'll add testing later after I get everything working." You'll waste endless hours of debugging issues that could have been prevented with preemptive testing. But don't take my word for it, take the word of Bryan Liles.

Also, on a somewhat related note to testing, use one of the plug-ins and accompanying services like Hoptoad or Exceptional to track and alert you when something breaks on your site.

If you want to get hard core, and why not, install the New Relic plug-in which will track and report on the performance of your application, so you can find out where the bottlenecks are.

Automate deployment

Get automated deployment working from the get go. Don't save it for the end. You should be able to deploy your site from your hosted source control to your hosting provider with a single command. Capistrano is the tool for the job if you're using Rails. Open up a terminal window and type "cap deploy" and watch it all unfold. If that one command doesn't do everything you need it to, make it! For example, I deploy a lot, and Capistrano doesn't clean up after itself automatically, so I hacked it to run the "clean" command after every "deploy". It also doesn't run database migrations by default, so I added the "migrate" task to "deploy" as well. When you can update your live site with one command, you'll sleep better at night.

If you build it, they will come

Well, they wont actually come until you promote it, but that's a later section. Once I had all the aforementioned steps in place, I buckled down and coded. My Ruby and Rails skills were a little rusty, so I'd occasionally have to visit a documentation site like APIdock or search for an issue on Google or as a last resort post my problem on gist then tweet the link on Twitter (thankfully I have quite a few smart and helpful Rails guys following me).

Package your dependencies

Don't expect your hosting provider, or the next developer to work on your project, to have all the necessary third-party dependencies for your project. Package them up with your application if possible. In Rails this is pretty simple, just declare the gem dependencies in your environment.rb file then rake gems:unpack to extract them locally. Do it for Rails itself too, and don't forget to add them all to source control.

Track it

If you want to know how many people are visiting your site, how many pages your site is serving, which page is the most popular, etc., you need to track it all. The quickest, easiest, and cheapest way I know of accomplishing that is Google Analytics. Create a profile for your site and they will generate a little snippet of JavaScript code to paste into your pages. Assuming you have a template that's common to ever page on the site (like a header or footer), that's the logical place to put it. Google will track everything for you and give you some super slick reports.

Tell people about it

Once you've got your new site coded, tested, and deployed, it's time to draw people to it. People aren't likely to find it on their own, so you need to announce it. With Pocket Rails I first started tweeting about it on Twitter. This attracted a few visitors and provided some initial feedback. Once I'd ironed out a few kinks I shot an e-mail over to the guys as the Rails Envy Podcast explaining what I'd built. They seemed to like it and mentioned it on their show. That drove a sizable burst of traffic which quickly died off. A week later I posted the link to reddit for ruby hackers and holy macaroni the site was deluged with visitors, and I began to see other people talking about it and linking to it thanks to tools like Google Alerts and Twitter Search. I added their RSS feeds to my news reader so I can keep on top of the chatter.

Finally, blog about it

Hey, my blog post about launching your site is recommending you blog about launching your site. How meta is that? It makes my brain hurt a little. But seriously, share your story so others may learn from it, as I hope you've learned something from my story. I'd love to hear your comments and criticisms, perhaps you'd have done something differently.

Rate Marina's Outfits

Yesterday I soft-launched my latest pet project, which I wrote on New Year's Eve of all nights, with graphic design graciously contributed by Allan Branch of Less Everything.

What is it? Well, for those of you too lazy to click the link, it's essentially a mash-up of YouTube and the Ajaxful Rating plug-in for Ruby on Rails that lets you rate the outfits worn by Marina Orlova on her popular webisode sensation HotForWords.

Why did I write it? To get rich and famous, of course, right? Ha, no. I wrote it because I've been interviewing for new gigs the last couple months and I keep getting asked for sample code that I've written. All the projects I've worked on for the last decade have been private and proprietary code bases, so I can't share them. Now I have something to share. For now I'm only sharing the code with potential employers, but I do plan to release it to the public in the very near future.

If you like what you see and you're looking for a solid developer (remote only, sorry, no relocation) please give my resume a peek. Thanks.

iPhone RDoc Template

I just made public my GitHub repository for a pet-peeve project: an RDoc template that looks good on the iPhone browser.

I used this template to generate the Rails documentation and you can check it out at http://pocketrails.com (visit it from your iPhone).

Ruby Revelations: An Introspection

I was code-reviewing a cohort's contribution to adPickles the other day and I came across a line of Ruby that had me completely perplexed:

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

That's quite a mouthful. If you know what that does, go ahead and stop reading now, 'cause the rest of this post is just self flagellation.

I'm no Ruby expert, but this guy is (in case you couldn't discern that from this one line masterpiece). I had to ask him to break it down for me, and after a few back-and-forths I think I've got it. Here's the skinny, for those of you that are still reading and, like me, want to see the secret unravelled.

My gut instinct is to start from the deepest nested block and work my way out, but you first need a little nugget of context.

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

An Advertisement has several aspects, such as "new", "approved", "rejected", etc. Knowing that, now we can work from the bottom up.

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

Here he's referencing a variable named advertisements which we can safely assume is an Array of objects, each being an instance of Advertisement.

He wants to call a method on the array (which in turn calls the method on each contained element) but at runtime he doesn't know the name of the method, so he constructs it on the fly.

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

In Ruby, if you want the value of a variable to be evaluated in a string, you wrap the reference in #{}, so in this case if the value of aspect is "rejected", the method name being constructed from "count_#{aspect}" is going to be count_rejected.

So each member of the Array named advertisments is going to have the method count_rejected called.

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

And for each aspect, a two-element array is created, where the first element is the value of aspect and the second element is the results returned by the on-the-fly-generated-method-name.

For example, one of these arrays might look like ['rejected',23].

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

Each of these arrays are collected into an outer containing array by the map method, which will leave us with something like [['new',14],['live',51],['rejected',23]].

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

The the whole thing gets flattened into a one-level (flat) array, like ['new',14,'live',51,'rejected',23].

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

As my cohort explains the next part, "Hash doesn't take an array as an argument, so the asterisk breaks the array into multiple arguments. Hash[*[1,2,3,4]] is the same as Hash[1,2,3,4]."

Hash[*Advertisement.aspects.map {|aspect| [aspect,advertisements.send("count_#{aspect}")]}.flatten]

Which bring us to the crusty outer shell, leaving us with a simple hash which looks like {'new' => 14,'live' => 51,'rejected' => 23}, where as you can see, each aspect is now mapped to its respective count.

Clear as mud? Excellent!

That Didn't Take Long

One day into development and I've already got a rant... but it's a tame one.

I needed to do some Globally Unique Identifier (GUID) generation for my new project so I Googled up "ruby guid" and arrived at this convenient little library.

Somebody has already done the work for me, and shared it! I love that. My thanks go out to the author.

I downloaded it, installed it, incorporated it, tested it, and everything was golden... on my Windows machine.

After I committed to source control and ran an update on my Mac to continue development there, I started getting this error:

/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require': No such file to load -- Win32API (MissingSourceFile)

Oops, that's rather queer. Why would the code running on OSX be attempting to load a Windows API?

Thanks to the wonders of Open Source Software (and the author) I was able to look at the offending line and this is what I found:

if RUBY_PLATFORM =~ /win/i

The author reports on his download site that he "only tested this library under Win2k and Redhat," so he didn't realize or anticipate another environment other than Windows that might have "win" in its name... like, maybe, Darwin :-)

A little knowledge can be dangerous

<arturaz> any ideas how i can get @foo from :foo ?

<inono> magic

<TeflonTed> eval "@#{:foo.to_s}" ? :-)

<arturaz> eval's evil :)

<TeflonTed> magic is evil