Disabling third-party services when they stop performing (in Rails)

Uploaded with plasq's Skitch!

One of my clients uses the hosted version of CompanyX (not their real name) to serve ads on their site. A couple weeks back, CompanyX applied some "upgrades" and things didn't go as planned, so for nearly a week their service was up and down like a yo-yo. That resulted in me getting calls along the lines of, "Hey our site is loading slow because of the CompanyX ads, please take them all off," followed a few hours later with another call, "Hey CompanyX seems to be OK now please turn their ads back on," and a little while later the cycle repeats itself. That got real old, real quick.

So, I decided to whip up a little automated solution. I needed two core components:

1. A way to programatically turn the ads on and off.

2. A way to periodically test the third-party service, and enable or disable the ads based on its response time.

For disabling and enabling the serving of ads, I created a new model called CompanyXStatus which is essentially a toggle switch, it's either on or off, and for auditing purposes I have it store the date and time whenever it's flipped. The database table looks like this:

create_table "companyx_statuses" do |t|
  t.column "enabled",    :boolean
  t.column "created_at", :datetime
end

And the app-facing API of the model looks like this:

class CompanyxStatus < ActiveRecord::Base
  
  class << self
    
    def enabled?
      latest.enabled
    end
    
    def disabled?
      !enabled?
    end
    
    def disable!
      CompanyxStatus.create!(:enabled => false) if enabled?
    end
    
    def enable!
      CompanyxStatus.create!(:enabled => true) if disabled?
    end

    private

    # I'm not using a named scope because this client is on an OLD version of Rails...
    def latest
      CompanyxStatus.find(:first, :order => 'created_at desc') || CompanyxStatus.new(:enabled => true)
    end

  end

end

So in the views when I'm building a page I just have to check if CompanyxStatus.enabled? before rendering an ad tags.

Now, for the actual toggling logic, I'm using the Benchmark module to call out to the service and measure the response time. If if exceeds the threshold (2.5 seconds in this case) the service is disabled, otherwise enabled. Here's the rest of the model:

  def test
    if 2.5 > latency
      CompanyxStatus.enable
    else
      CompanyxStatus.disable
    end
  end

  private

  def latency
    Benchmark::measure{ connect }.real
  end

  def connect
    socket = Socket.new( AF_INET, SOCK_STREAM, 0 )
    sockaddr = Socket.pack_sockaddr_in( 80, 'blah.companyx.org' )
    socket.connect( sockaddr )
    socket.write( "GET /blah.php HTTP/1.0\r\n\r\n" )
    results = socket.read
  end

Finally I need a way for the system to periodically make these checks and toggles so my client and I don't have to worry about babysitting the site. For this I wrote a simple rake task:

namespace :companyx do
  desc "Ping CompanyX and disable it if too slow"
  task :ping => :environment do
    CompanyxStatus.test
  end
end

And scheduled it as a cron job to run every minute:

* * * * * cd /home/client/apps/production/site/current && RAILS_ENV=production rake companyx:ping

That's it! Now I can get a good night's sleep knowing that the next time CompanyX has a burp in their service, my client's site is going to automatically shut them off until they get their act back together again.