One of the questions we've been asked more frequently is: how can we filter Mogotest traffic in our analytics tool? While our spider uses a customer user agent (mogobot) that's easy to filter on, the test browsers have been considerably trickier. We use actual browsers to run our tests and we can't change the user agent string, since that would break tests for the large number of sites still doing user agent sniffing. Additionally, we use EC2 to scale up our service, so our block of IPs can change at any given time, making traditional IP filtering all but useless.

Well, I'm happy to say that after some heavy internal testing we now finally have a good answer to the problem. All outgoing traffic from the Mogotest browsers is filtered through a pool of proxy servers with static IP addresses so we can employ IP or hostname-based filtering. We wrote up a howto for filtering Mogotest traffic in Google Analytics specifically, but the approach is general enough to apply to all analytics packages. This was a rather large architecture change for us that we wanted to ensure wouldn't break test runs, so we very much appreciate your patience as we rolled it out.

A tangential benefit is we can now internally cache resources your site marks as cacheable. This should yield faster tests in general, but for those of you that were testing on weaker hardware that couldn't handle all the traffic Mogotest would send your way at once, this will avoid the "thundering herd" problem you were experiencing that was potentially knocking your site offline. The gains overall should be quite substantial. Try running a test and let us know what you think.