Comments on: Results of Native Compilation vs. Java Split Test

By: Roedy Green

Roedy Green — Wed, 01 Jul 2009 20:12:29 +0000

“Of the 75 purchases, 44 were for the JET version and 31 were for the native Java compiled version.”

What do you meant by “native Java compiled version”?
Using standard Java.exe hotspot?
using GCJ native compiler?
something else?

Java.exe uses hotspotting, not native compilation.

By: Nick

Nick — Fri, 17 Apr 2009 12:08:47 +0000

What was the donwload size difference between the JET version and the standard Java version? Just curious as to whether the JET version is smaller than the normal version and if so, how much. If it is much smaller, perhaps the download finishing sooner means more people remember they downloaded it, try it out and ultimately buy it.

By: Patrick

Patrick — Tue, 24 Mar 2009 02:51:27 +0000

>>
Do I understand correctly that if Java is not found, you download JRE distribution, start it and let user to complete the installation?
>>

When they attempt to run BingoCardCreator.exe it will tell them they need Java to run it, then direct them to the appropriate page on Sun’s website. After that it is up to them.

By: Vladimir Dyuzhev

Vladimir Dyuzhev — Mon, 23 Mar 2009 00:29:10 +0000

We had an argument over your results with a few Java/.NET/native C++ folks.

Do I understand correctly that if Java is not found, you download JRE distribution, start it and let user to complete the installation? I had an idea that the difference may be because of ugly look of JRE installer. Unprepared user would have an urge to cancel it — after all, he was doing BingoCardsCreator download, not some coffee maker software or something.

I’m interested in one more version: when Java is not found, download JRE and install it yourself (simply unpack a prepared ZIP into your application folder privately), with no extra dialogs or anything. I wonder how that would be different from JET results.

By: Jason Cohen

Jason Cohen — Fri, 27 Feb 2009 15:51:29 +0000

Congrats on a great blog, and I love that you’re testing this stuff, but I’m sorry to report that your statistics are wrong.

The question you’re trying to answer is:

My hypothesis is that both downloads are equally likely to result in sales. My sample is that in 75 trials, the distribution was 44/31. Given this sample, have I disproved my hypothesis?

To answer this, you cannot just look at one tail of a binomial chart. Rather, you need the Pearson chi-squared test. Ref: http://en.wikipedia.org/wiki/Pearson’s_chi-square_test

(Read the first two paragraphs of that link if you’re still unconvinced that this is the correct statistic.)

Since your situation is binomial (two outcomes, one degree of freedom), you can use the calculation listed under “Two cells” in that article.

For you, O1=44, n=75, p=0.5, so the x^2 statistic is 2.253. Looking at the chi-square integration table for 1 df (Ref: http://www.medcalc.be/manual/chi-square-table.php) you’re around the 0.15 confidence interval.

Typically you want to see at least the 0.05, probably 0.01 confidence interval before you declare the hypothesis false.

Still, 0.15 is interesting…

My conclusion: This is *possibly* significant, but needs more testing. I would continue the A/B test and see if the CI moves.

Fun stuff! I’m excited to see what happens in the next few months. And congrats also on your 100-buster milestone!