Friday, April 3

Disabling third-party services when they stop performing (in Rails)

Chain
Uploaded with plasq's Skitch!
One of my clients uses the hosted version of CompanyX (not their real name) to serve ads on their site. A couple weeks back, CompanyX applied some "upgrades" and things didn't go as planned, so for nearly a week their service was up and down like a yo-yo. That resulted in me getting calls along the lines of, "Hey our site is loading slow because of the CompanyX ads, please take them all off," followed a few hours later with another call, "Hey CompanyX seems to be OK now please turn their ads back on," and a little while later the cycle repeats itself. That got real old, real quick.

So, I decided to whip up a little automated solution. I needed two core components:

1. A way to programatically turn the ads on and off.

2. A way to periodically test the third-party service, and enable or disable the ads based on its response time.

For disabling and enabling the serving of ads, I created a new model called CompanyXStatus which is essentially a toggle switch, it's either on or off, and for auditing purposes I have it store the date and time whenever it's flipped. The database table looks like this:
create_table "companyx_statuses" do |t|
t.column "enabled", :boolean
t.column "created_at", :datetime
end

And the app-facing API of the model looks like this:
class CompanyxStatus < ActiveRecord::Base

class << self

def enabled?
latest.enabled
end

def disabled?
!enabled?
end

def disable!
CompanyxStatus.create!(:enabled => false) if enabled?
end

def enable!
CompanyxStatus.create!(:enabled => true) if disabled?
end

private

# I'm not using a named scope because this client is on an OLD version of Rails...
def latest
CompanyxStatus.find(:first, :order => 'created_at desc') || CompanyxStatus.new(:enabled => true)
end

end

end

So in the views when I'm building a page I just have to check if CompanyxStatus.enabled? before rendering an ad tags.

Now, for the actual toggling logic, I'm using the Benchmark module to call out to the service and measure the response time. If if exceeds the threshold (2.5 seconds in this case) the service is disabled, otherwise enabled. Here's the rest of the model:
  def test
if 2.5 > latency
CompanyxStatus.enable
else
CompanyxStatus.disable
end
end

private

def latency
Benchmark::measure{ connect }.real
end

def connect
socket = Socket.new( AF_INET, SOCK_STREAM, 0 )
sockaddr = Socket.pack_sockaddr_in( 80, 'blah.companyx.org' )
socket.connect( sockaddr )
socket.write( "GET /blah.php HTTP/1.0\r\n\r\n" )
results = socket.read
end

Finally I need a way for the system to periodically make these checks and toggles so my client and I don't have to worry about babysitting the site. For this I wrote a simple rake task:
namespace :companyx do
desc "Ping CompanyX and disable it if too slow"
task :ping => :environment do
CompanyxStatus.test
end
end

And scheduled it as a cron job to run every minute:
* * * * * cd /home/client/apps/production/site/current && RAILS_ENV=production rake companyx:ping

That's it! Now I can get a good night's sleep knowing that the next time CompanyX has a burp in their service, my client's site is going to automatically shut them off until they get their act back together again.

2 comments:

Dr Nic said...

This is a neat idea and could perhaps be DSL-ified with config/third_parties/companyx.rb files that contain the guts of the #connect method in a little DSL:

Rails::ThirdParty.dependency "companyx" do
# is companyx.com up?
end

Thoughts?

pete said...

I had to deal with this once when embedding dynamically found youtube videos on a page; sometimes youtube's api would be slow to respond and our page loading time suffered. Our solution was a lot less elegant than this: we just made the code that searched for and embedded the videos a javascript callback that was executed on page load.