Introducing A/Bingo: Rails Split Testing

Regular readers of this blog know I’m a bit obsessive about testing and measuring small, iterative improvements to my website.  Previously I used Google Website Optimizer but I found it had some annoying limitations.  In particular, it was too much work to start tests, too much work to code tests, and too much work to maintain expired URLs after tests were over. 

So I went away to the code salt mines for the last week, and coded my own A/B testing framework in Ruby on Rails.  It is called A/Bingo .  (The name sort of popped into my head and wouldn’t leave.  It is a pun, it relates to my business and personal brand, and it gives the impression of Hitting the Target that successful testing should give you.  My designer took that and ran with it for the logo.  I’m very impressed with the results.

A/Bingo is, without question, the most technically impressive code I’ve ever written.

  • Setting up a test takes one line of code.  It is so easy you’ll find yourself doing it automatically when writing copy or making new features.
  • It tracks absolutely any event as a conversion, in one line of code.
  • It does statistical significance testing for you, producing human-readable output which tells you exactly how confident you can be that the test results are accurate.
  • It is fast.  No, it is fast.  Like, “A/Bingo could handle being linked from the front page of a social news site.  Heck, it could be used on the front page of many social news sites.”

If you want to read any more about it, I encourage you to take a look at the website.  A/Bingo is already powering all the Bingo Card Creator A/B tests.  I expect that the ease of use and copious features will mean I run a lot more tests.  I’ll continue sharing what I learn through them. 

P.S. Incidentally, I scored a 16.9% improvement to funnel completion thanks to Mixpanel.

No Responses to “Introducing A/Bingo: Rails Split Testing”

  1. Jeff Lewis August 15, 2009 at 10:32 am #

    Patrick, very easy implementation of A/B testing for rails – where’s the .NET version? ;-)

    You mention that you are A/B testing the “Get the code” link using bold or italics. It changed on me from italics to bold when I changed pages. Shouldn’t it always stay the same for me?

  2. Jeff Lewis August 17, 2009 at 11:06 am #

    Never mind. You’ve fixed it…

  3. Pius Uzamere September 7, 2009 at 9:21 am #

    Thanks for sharing your hard work! Unfortunately your git server seems to be down. Do you have an alternate download link for A/Bingo?

  4. Patrick September 7, 2009 at 9:27 am #

    Hiya Pius. I got the git server back up — it went down when this server got crushed by an avalanche of traffic yesterday, and it seems that it didn’t restart automatically when the server did. Try again now.

  5. Remi October 4, 2009 at 1:05 pm #

    Patrick, A/Bingo looks amazing. I have a JS-heavy app and I’ve been struggling with Google Analytics trying to do funnels and A/B testing.
    I figured I would do a quick google search to see if there was anything better, A/Bingo + mixpanel look awesome together. I have to try them out!
    Thanks for developing the plugin and all the great documentation!

  6. Patrick October 5, 2009 at 7:09 pm #

    Thanks Remi, I’m glad you liked it. I’m going to be integrating Mixpanel into the default A/Bingo distribution later.


  1. Kai’s daily tech #28 … | Kai Richard König - September 4, 2009

    […] Introducing A/Bingo: Rails Split Testing: MicroISV on a Shoestring […]

  2. A/Bingo 1.0.0 Official Release: MicroISV on a Shoestring - February 14, 2010

    […] Back in August I released A/Bingo, an MIT-licensed OSS Rails A/B testing framework.  I have been using it continuously on Bingo Card Creator, and judging from the support requests I’ve been getting it has gotten some traction in the Rails world.  The 5,000 or so people seeing A/B tests on my site on Valentine’s Day are almost certainly less than 1% of the beneficiaries of the software now. Yay. […]