The Most Radical A/B Test I've Ever Done

About four years ago, I started offering Bingo Card Creator for purchase.  Today, I stopped offering it.

That isn’t true, strictly speaking.  The original version of Bingo Card Creator was a downloadable Java application.  It has gone through a series of revisions over the years, but is still there in all its Swing-y glory.  Last year, I released an online version of Bingo Card Creator, which is made through Rails and AJAX.

My personal feeling (backed by years of answering support emails) is that my customers do not understand the difference between downloadable applications and web applications, so I sold Bingo Card Creator without regard to the distinction.  Everyone, regardless of which they are using, goes to the same purchasing page, pays the same price, and is entitled to use either (or both) at their discretion.  It is also sold as a one-time purchase, which is highly unusual for web applications.  This is largely because I was afraid of rocking the boat last summer.

The last year has taught me quite a bit about the difference between web applications and downloadable applications.  To whit: don’t write desktop apps.  The support burden is worse, the conversion rates are lower, the time through the experimental loop is higher, and they retard experimentation in a million and one ways.

Roughly 78% of my sales come from customers who have an account on the online version of the software.  I have tried slicing the numbers a dozen ways (because tracking downloads to purchases is an inexact science in the extreme), and I can’t come up with any explanation other than “The downloadable version of the software is responsible for a bare fraction of your sales.”  I’d totally believe that, too: while the original version of the web application was rough and unpolished, after a year of work it now clocks the downloadable version in almost every respect.

I get literally ten support emails about the downloadable application for every one I get about the web application, and one of the first things I suggest to customers is “Try using the web version, it will magically fix that.”

  • I’m getting some funky Java runtime error.  Try using the web application.
  • I can’t install things on this computer because of the school’s policies.  Try using the web application.
  • How do I copy the files to my niece’s computer?  By the way it is a Mac and I use a Yahoo.  Try using the web application.

However, I still get thousands of downloads a month… and they’re almost all getting a second-best experience and probably costing me money.

Thus The Experiment

I just pushed live an A/B test which was complex, but not difficult.  Testers in group A get the same experience they got yesterday, testers in group B get a parallel version of my website in which the downloadable version never existed.  Essentially, I’m A/B testing dropping a profitable product which has a modest bit of traction and thousands of paying customers.

This is rather substantially more work than typical “Tweak the button” A/B tests: it means that I had to make significant sitewide changes in copy, buttons, calls to action, ordering flow, page architecture, support/FAQ pages, etc etc.  I gradually moved towards this for several months on the day job, refactoring things so that I could eventually make this change in a less painful fashion (i.e. without touching virtually the entire site).  Even with that groundwork laid, when I “flipped the switch”  just now it required changing twenty files.

Doing This Without Annoying Customers

I’m not too concerned about the economic impact of this change: the A/B test is mostly to show me whether it is modestly positive or extraordinarily positive.  What has kept me from doing it for the last six months is the worry that it would inconvenience customers who already use the downloadable version.  As a result, I took some precautions:

The downloadable version isn’t strictly speaking EOLed.  I’ll still happily support existing customers, and will keep it around in case folks want to download it again.  (I don’t plan on releasing any more versions of it, though.  In addition to being written in Java, a language I have no desire to use in a professional capacity anymore, the program is a huge mass of technical debt.  The features I’d most keenly like to add would require close to a whole rewrite of the most complex part of the program… and wouldn’t generate anywhere near an uptick in conversion large enough to make that a worthwhile use of my time, compared to improving the website, web version, or working on other products like Appointment Reminder.

I extended A/Bingo (my A/B testing framework) to give a way to override the A/B test choices for individual users.  I then used this capability to intentionally exclude from the A/B test (i.e. show the original site and not count) folks who hit a variety of heuristics suggesting that they probably already used the downloadable version.  One obvious one is that they’re accessing the site from the downloadable version.  There is also a prominent link in the FAQ explaining where it went, and clicking a button there will show it.  I also have a URL I can send folks to via email to accomplish the same thing, which was built with customer support in mind.

I also scheduled this test to start during the dog days of summer.  Seasonally, my sales always massively crater during the summer, which makes it a great time to spring big changes (like, e.g., new web applications).  Most of my customers won’t be using the software again until August, and that gives me a couple of months to get any hinks out of the system prior to them being seen by the majority of my user base.

My Big, Audacious Goal For This Test

I get about three (web) signups for every two downloads currently, and signups convert about twice as well as downloads do.  (Checking my math, that would imply a 3:1 ratio of sales, which is roughly what I see.)  If I was able to convert substantially all downloads to signups, I would expect to see sales increase by about 25%.

There are a couple of follow-on effects that would have:

  • I think offering two choices probably confuses customers and decreases the total conversion rate.  Eliminating one might help.
  • Consolidating offerings means that work to improve conversion rates automatically helps all prospects, rather than just 60%.

Magic Synergy Of Conversion Optimization And AdWords

Large systemic increases in conversion rates let me walk up AdWords bids.  For example, I use Conversion Optimizer.  Essentially, rather than bidding on a cost per click basis I tell Google how much I’m willing to pay for a signup or trial download.  I tell them 40 cents, with the intention of them actually getting the average at around 30 cents, which implies (given my conversion from trials/signups to purchase) that I pay somewhere around $12 to $15 for each $30 sale.  Working back from 30 cents through my landing page conversion rate, it turns out I pay about 6 cents per click.

Now, assuming my landing page conversion is relatively constant but my trial to sale conversion goes up by 25%, instead of paying $12 to $15 a sale I’d be paying $9.60 to $12 a sale.  I could just pocket the extra money, but rather than doing that, I’m probably going to tell Google “Alright, new deal: I’ll pay you up to 60 cents a trial”, actually end up paying about 40 cents, and end up paying about 8 cents per click.  The difference between 6 and 8 will convince Google to show my ads more often than those of some competitors, increasing the number of trials I get per month out of them.  (And, not coincidentally, my AdWords bill.  Darn, that is a bloody brilliant business model, where they extract rent every time I do hard work.  Oh well, I still get money, too.)

We’ll see if this works or not.  As always, I’ll be posting about it on my blog.  I’m highly interested in both the numerical results of the A/B test as well as whether this turns out being a win-win for my customers and myself or whether it will cause confusion at the margin.  I’m hoping not, but can’t allow myself to stay married to all old decisions just out of a desire to be consistent.

About Patrick

Patrick is co-founded Starfighter, founded Appointment Reminder and Bingo Card Creator, and presently works at Stripe on Atlas. (Opinions on this blog are his own.) Want to read more stuff by him? You should probably try this blog's Greatest Hits, which has a few dozen of his best articles categorized and ready to read. Or you could mosey on over to Hacker News and look for patio11 -- he spends an unhealthy amount of time there.

15 Responses to “The Most Radical A/B Test I've Ever Done”

  1. Eric Burdo June 4, 2010 at 5:47 am #

    I’m interested in your results. I think the idea of the web based apps are going to push out the desktop apps in many cases.

    When I was working for “the man”, we were pushing everything out in web form, just because it was much easier to maintain and service (all was in-house apps).

    My new project (2 of them) are both going to be web-based.

  2. Stephan Wehner June 4, 2010 at 9:06 am #

    You wrote that you made “… significant sitewide changes in copy, buttons, calls to action, ordering flow, page architecture, support/FAQ pages”.

    I thought the trick with A/B Tests is to only change one feature or aspect (text,colour,box-left vs. box-right etc). See section “How To Do Your A/B Split Test?”

    On the other hand, I got the impression you already have enough reason to drop the desktop application, without testing.



  3. Zack June 4, 2010 at 9:38 am #


    You’re on the right track with your testing, and it’s fun to watch your adventures in optimization on this blog. I predict an ultimate win for the online version.

    One note – have you split-tested the pop-up signup form vs. a simple HTML page?

    We’ve found that pop-ups can confuse users and are one of the toughest things to get working cross browser (in IE8 for instance, the “close” button doesn’t seem to work – it appears that the image ref is slightly skewed thanks to CSS – maybe adjust close to HTML rather than image?).

    Anyways – keep up the great posts, looking forward to seeing how this experiment goes!


  4. lynn June 4, 2010 at 12:25 pm #

    Well, I’ve been following your blog for a while, and was about to launch my own windows desktop app, but now you have given me pause… Hm. Okay, so if I get someone on scriptlance or rentacoder to take my app specifications and make me a web-based program instead, that won’t take too long I guess… I don’t really want to deal with all the support tickets a buggy desktop app will probably generate. One question for you: why e-junkie and not Clickbank? Out of curiosity, is one micro-ISV friendlier than the other?

  5. lynn June 4, 2010 at 12:26 pm #

    that should be “is one site micro-ISV-friendlier than the other” :-)

  6. Map Guy June 5, 2010 at 2:37 am #

    Hi Patrick,

    I have read your post for sometime and this is the first time I wrote you. I have developed Windows version of my interactive map software, as well as a web version.

    However my results are very different from yours. Most of my customers still prefer to buy the Windows version, even I get more ‘signups’ everyday than ‘downloads’. But the conversion ratio to sale is much higher for users who download the windows version.

    My understanding is that many of my customers are actually business buyers, so they would prefer to pay for a ‘license’ which is very easy for them to explain to their management or boss.

    The web-based concept is still new to many business users. I would think it would be good for your business (at least no harm), if you keep both versions on your website. As you were giving out both to your users anyway. You can simply state that you will no longer support the Windows version?

  7. Map Guy June 5, 2010 at 2:41 am #

    Also Patrick are you still using Adwords? I noticed your expenses on advertising dropped significantly in recent months. Are you cutting adwords budget as you are getting sufficient organic search traffic, or the CPA model really saves that much money?

  8. Patrick June 6, 2010 at 5:00 am #

    I still use AdWords — quite a bit, actually. I just haven’t done bookkeeping in a few months (since the last big batch in January), due to a problem with my bookkeeping system that I just resolved recently.

  9. Steve Moyer June 6, 2010 at 9:21 am #

    Interesting post and I reread “Why I’m Done Making Desktop Applications” to make sure I remembered your salient points. I whole-heartedly agree and disagree.

    My plan for my next project is that it will be both a web application and a desktop application. And all from the same build system. This is a bit bleeding edge (just a little bit), but my looking at the features included in HTML5 make this possible.

    First a little teaser … What’s the one language (framework, run-time) that’s guaranteed to be installed on almost every desktop regardless of the underlying OS? Yes it’s JavaScript. But programming in JavaScript, even with libraries like JQuery and Prototype still leaves a lot to be desired.

    So my plan is as follows:

    – Use GWT to compile Java to JavaScript and to allow an automated build/test cycle.
    – Allow all users to run the application within their web browsers.
    – Allow users with an HTML5 compliant browser (specifically browsers that support the Offline Web Application and Web Storage portions of the specification) to run the web application locally on their machine (but still within the browser).
    – Create an installer that runs on Windows, Mac and Linux to install Prism and the web application, so that people who prefer a desktop like application can still run one that way.
    – Allow the application to synchronize its data with the server when on-line.
    – Force the application to verify the registration when on-line.
    – Force the application to upgrade to the latest version when on-line.

    So … I think that with the possible exception of a slightly weaker defence against piracy, I’ve managed to put together a platform with a single build that satisfies both those that prefer desktop application (and those that prefer web applications but are off-line much of the time so are forced into using desktop software) and those that prefer web applications, but the system still maintains the benefits you describe in both the articles you’ve written on the subject. It potentially has the benefits of AIR and Silverlight, but doesn’t require that additional run-time environments be installed. It will even work on the IPhone/IPad because no flash is required!

    What do you think? Is JavaScript our new “run anywhere” environment as others have proposed?

  10. william June 8, 2010 at 2:47 am #

    to lynn,

    e-junkie is better than clickbank in all aspects. i would personally suggest u e-junkie.

  11. Steve O'Brien June 8, 2010 at 8:16 am #

    Steve Moyer, your idea seems like it would be a real big pain to operate. You’re betting a lot on the assumption that desktop versions are still valuable when a web version is offered – betting your whole infrastructure and a lot of time and frustration. Hopefully, you can test your assumption somehow, or all that time and frustration could be wasted.


  1. links for 2010-06-04 « Daniel Harrison's Personal Blog - June 4, 2010

    […] The Most Radical A/B Test I’ve Ever Done : MicroISV on a Shoestring (tags: startups entrepreneurship) […]

  2. The Most Radical A/B Test I’ve Ever Done » Ever, Done, Test, Radical, Link, Most » Adjoozey - June 5, 2010

    […] Link: The Most Radical A/B Test I’ve Ever Done […]

  3. Link to Fit: The Top 5 Links of the Week - June 11, 2010

    […] Krzastek: A Radical A/B Test. Not just shifting a button or text but removing an entire well-used product from the site, […]

  4. Listen to 'The Most Radical A/B Test I’ve Ever Done' narrated by professionals, from 'MicroISV on a Shoestring' - Hear a Blog - June 13, 2010

    […] About four years ago, I started offering Bingo Card Creator for purchase.  Today, I stopped offering it.That isn’t true, strictly speaking.  The original version of Bingo Card Creator was a downloadable Java application.  It has gone through a series of revisions over the years, but is still there in all its Swing-y glory.  Last year, I […] Original post […]