Something like:

var RT=new Image();
RT.src=”/__rt.png?x=”+(12345 + 67890);

Previously I left out the “.png” from the URL and a higher percentage of robots did request the URL. Adding the extension provided a 10% improvement to my detection rates.

For A/B testing purposes relying on JavaScript like Patrick described — is a good solution.
For blocking excessive request — relying on JavaScript validation is not the best approach.

Liked the idea so much, that my subconscious expanded on what hypothetical / theoretical alternatives us programmers could put the users browser and therefore CPU to good use.

What if we were able, or sites like facebook were able to use the users impressions, using a small part of the users cpu to do some sort of tasks.

My first thought was hashing md5’s or more complicated things like searching subsets of dna sequences for patterns.

Slightly longer version.
http://thoughtsofaprogrammer.tumblr.com/post/677654730

For information, the websites are http://www.chemeo.com (physical properties of chemical components) and http://www.indefero.net (hosted git/subversion repositories and project management). These are not websites where one make money selling attorney services or things like that. This is maybe why the bots coming are not that bad.

Maybe my procedure to manage my tests is also helping, here is what I am doing:

– always have at least one A/A test open to ensure that the results are sound. That is, I test 2 times the same alternative and they come out with the same score.
– rerun on a regular basis the tests which were not that significant.
– keep long term stats to be sure that if my test gave an alternative which went from a 15% to a 20% conversion rate, it stays like that on the long run (or improve).

So I “fire and forget” a lot of tests, but I keep track of the effects on the long run and adjust. I accept that the numbers in the tests can be a bit off and always check against my real conversion rates at the end of the day. Note that the 10% deviation is that my framework (at the application level) picks up 10% more visitors, they can be bots (but they will not convert) or people without javascript enabled.

I hope it helps understanding my particular situation. The code of the A/B testing framework (PHP+MongoDB) is available (LGPL) here:
http://projects.ceondo.com/p/pluf/source/tree/master/src/Pluf/AB.php

Loic d’Anterroches – either you are lucky not to have these misreporting bots or because they misreport themselves you just haven’t noticed.

public static function isBot($user_agent)
{
static $bots = array(‘robot’, ‘checker’, ‘crawl’, ‘discovery’,
‘hunter’, ‘scanner’, ‘spider’, ‘sucker’, ‘larbin’,
‘slurp’, ‘libwww’, ‘lwp’, ‘yandex’, ‘netcraft’,
‘wget’, ‘twiceler’);
static $pbots = array(‘/bot[\s_+:,\.\;\/\\\-]/i’,
‘/[\s_+:,\.\;\/\\\-]bot/i’);
foreach ($bots as $r) {
if (false !== stristr($user_agent, $r)) {
return true;
}
}
foreach ($pbots as $p) {
if (preg_match($p, $user_agent)) {
return true;
}
}
if (false === strpos($user_agent, ‘(‘)) {
return true;
}
return false;
}

It is not perfect, but it has been working very nicely for more than a year on my websites. Anyway, there are no perfect methods to detect if an agent is a bot… good enough is just the way to go.