I might be way of base with this, but does your ABingo zscore function contain a small bug?

numerator = cr1 – cr2
frac1 = cr1 * (1 – cr1) / n1
frac2 = cr2 * (1 – cr1) / n2

Shouldn’t the frac2 line deal with cr2 only? That’s how I read the http://20bits.com/articles/statistical-analysis-and-ab-testing/ posting at least.

FYI, I noticed this because I am trying to do a ABingo port to Java, just like you were hoping. Things are going reasonably, but without the uniformity of something like Rails my biggest struggle it to determine which (of the thousands) of Java web frameworks and the various back-ends to integrate with first.

Thanks for the inspiration,

Ivan

Your article was *awesome* and I hope to employ the ideas in practice.

I am starting a blog called “scrumpilot” which is basically taking your concepts but for large engineering firms around coming up with smart, metrics for agile development and at some point would like maybe to have a podcast interview with you.

There’s nothing on the blog, I’ve just started it but wanted to start reaching out….

Thanks…Tim

All I see is up side to this. If the test you conducted wasn’t sufficient in the UI pretty department and you get higher conversions from a form from one of the nay sayers they hey you get a better form and they can say I told you so. If on the other hand the simpler current form keeps the title as best conversion method for bingo then you can thumb your nose at the critiques that are a bit more abundent on this piece then other pieces you’ve articulated.

Either way the whole A/B testing wins and lessons are still highlighted to your captive readership. :) Love to see a throw down. Hey and of course it has to win with statistical significance. 95% I presume is the statistical setting.

On another note have you come across type I errors in the A/B testing. I was reading about health research and the number of times they community retracts and refutes a previously studied health bennefit or risk and that they found that the rate of these retractions is following about the rate of expected type I errors.

Yes, you “conversationalized” the form, but in a very awkward and geeky way. “Save them under” and “use this word ______ as my password” are no better than the form because it’s still techno-speak.

“My email address is ________ and, by the way, use ________ as my password.” Is a step closer to what you should have done. Also, the rest of his for was left unchanged. So it’s still basically a form.

A point of opinion: when you’re discussing the gap between two percentages, referring to said gap as a relative percentage is initially confusing.