I was watching the Pivotal Labs video of their luncheon with New Relic and the speaker, CEO Lewis Cirne said something that struck me as really odd yet kind of clever.
As he was describing the architecture of their system, Cirne revealed they they run a constant "synthetic" load on their production system. He referred to the faux load agents as their "canaries in the coal mine." The purpose of these canaries, as he explained it, is so that when they reach a dangerous level of traffic - when performance begins to suffer - they can kill off the fake load, freeing up resources and consequently giving them a gauge as to how much breathing room they have before they are really in trouble.
Right off the bat this ingenious ruse throws up two warning flags to me. First off, is all this fake load unnecessarily reducing the lifetime of the hardware, such as thrashing the disk drives, generating excess heat, sucking electricity, etc.? And secondly, how sure are they that this fake load is representative of real load, and hence a true measurement of spare capacity once it's removed?
What do you think?