Some months ago when the 99designs fixed-price logo store launched, my buddy Thomas at Matasano remarked that it would have been the perfect choice for my business. I said that, while I’m quite impressed with 99designs’ business model and many of the logos on offer, I thought the less-generic-looking logo which I had custom-made fairly cheaply (by the folks at Logo Samurai — I bought three logos and chucked the two I liked least) was better than a generic logo. Thomas disagreed, and said that the more “professional” look of the off-the-rack logos would offset the genericness.
If Thomas and I were bigwigs at a multinational firm with entire departments devoted to preventing anything from getting accomplished, that disagreement might have been settled by a great deal of harumphing. However, we’re engineers, so we made a wager: I would purchase a logo handpicked by Thomas and A/B test it against my present logo sitewide, and whomever’s logo lost would donate $100 to a charity named by the victor.
The Original Logo:
The 99 Designs Logo:
Technical Details
This A/B test was done sitewide — on first access of the site, via landing page, organic search, or any other source, users were randomly assigned one of the two logos to be the “official” Bingo Card Creator logo. They saw the same logo for all subsequent accesses. This was done with A/Bingo, the Rails A/B testing framework I wrote. This particular test required A/Bingo to be slightly extended to ignore bots, because — since the test was sitewide and visible prior to a bot-blocking login screen — the deluge of bots this site deals with was drowning out the signal with their non-converting noise. Happily, since A/Bingo is open source, all other users of the software benefit from Thomas and I fighting over geek cred.
We agreed to measure conversions to the free trial rather than conversions to purchase. Measuring conversions to purchase would have been technically feasible, but since much of my market is out of school during the summer, getting enough data to be statistically significant would probably have taken months. Conversions to the trial are much more frequent so it was reasonable to expect sufficient data in a week or two. The test was allowed to run for two weeks.
The Results
My original logo ended up winning by a virtual mile: 11.05% of all users seeing the classic logo converted, versus 9.92% of those seeing the 99designs logo. Just by eyeballing that you might think “Hmm, 1% difference, that’s probably just noise”, but when you factor in the sample sizes (8,500 real people saw each variation) it turns out that the difference is statistically significant at the 99% confidence level.
(If those conversion numbers strike you as fairly low, remember that the test is running sitewide rather than on a single landing page. Many folks who visit the site are drawn to free activities created for SEO purposes, and downloading those activities doesn’t necessarily require signing up. Conversion rates for my AdWords landing pages are closer to 25 ~ 30%. Relatedly, consolidated conversion rate is a poor metric to make decisions on, but that is a discussion for another day.)
Even though in this particular case the A/B test did not succeed in identifying a profitable change to make in my business, I consider it a wonderful use of my time — it only required $99, a few minutes of time from my local designer (to put the logo on the background I use for my header without it getting aliased), and a single line of code. I can’t possibly stress enough how valuable it is to have your A/B tests require only a single line of code. You’ll test more frequently, you’ll test more diligently, and you’ll discover more valuable things about your business.
Implications For Other Businesses
Graphic designers: Some of you appear to be highly opposed to spec work, at least partially because you believe it to produce inferior work. You might consider measuring the effect of well-designed logos (for any value of “well-designed”, for example, “uses a business model I approve of”) versus other logos. I have the graphical skill of a molerat — I should be linking to this article on your blog, professional graphic designer, not writing it.
99designs: Don’t get me wrong, I love you guys. My Appointment Reminder site uses an off-the-rack logo from your site that I’m very, very fond of. I’m very, very open to the notion that there exist circumstances under which this test would have come down the other way.
Software companies: Your logo could potentially add or subtract 10% from enterprise value. Bolded for emphasis because holy cow. (10% higher conversion rate to a trial flows virtually directly to my bottom line.) The majority of companies in our industry get a logo done early in their corporate life, generally exactly once. For some companies, brand equity might be of paramount concern, and the risk of creating confusion in a two-week A/B test swamps even a 10% potential increase in conversion rates. For those of us who are more transaction-oriented than brand-oriented, though, this is an easy, obvious, fairly inexpensive area to test.
(Obligatory “I passed Stats 101″ disclaimer: the significance test I performed says that logo #1 is 99% likely to truly be better than logo #2, rather than saying that logo #1 is 99% likely to be 10% better than logo #2.)
One question: did the designer of the new logo know how it would sit within your site header, and did Thomas take that into account when he chose it? It looks to me like the biggest problem with the new logo is that the product name disappears into the green swoop. That logo in that context is poor, but out of that context it’s probably somewhat better.
I thought your original logo was way better, too: it’s much more informative and says a lot about the product. Logos like the second one give off an aura of vagueness that makes me feel wishy-washy.
I think this proves that the logo actually does matter. The second logo is, to be blunt, very poorly designed. It doesn’t speak to the product or the site, and as mentioned above gets lost in the header. It looks exactly like it is… a generic logo, creating an artistic element out of the first letter of the sitename (and not much more). No surprise that the first logo resonated better with the audience.
Just some thoughts for improvements from a non-designer… Your blog’s text logo is butted up against my toolbar and your business logos, while nice, could be more readable.
The “a” in “Creator” is hardly distinguishable from an “o” at a distance. The shininess effect or faint white gradient (whatever it’s called) on the text of your Appointment Reminder causes the text to look faded because it’s on a white background. Additionally, the text needs a bit of breathing room.
Two reasons the logo doesn’t matter here:
– The new logo doesn’t fit in with the design of the header, or the design of the layout in general. Whereas the old logo appears to have been designed to be more consistent and fit better in the layout.
– The layout as a whole is so clunky and aesthetically lacking, that anyone put off by aesthetics is already put off by the layout before they even worry about how the logo, specifically, looks.
Because of these points, I wouldn’t say the lesson learned applies more generally.
I think the more interesting experiment would be A/B testing a whole design. You could even keep every element exactly the same, just have a designer “re-skin” the site.
As much as I’d like to think it would make a positive difference, I fear that the more aesthetically pleasing option doesn’t always convert better…
Holy cow indeed. I am surprised to see the logo makes such a difference.
It would be interesting to test and see what the difference is between your current logo and no logo at all. (A much shorter test run, if there is a difference it should be obvious pretty fast).
The second logo reminds me of ads I saw for
http://www.birthright.org/
Sincerely,
Gene Wirchenko
Your website has always been — and still is one thing: ugly.as.shit.
(so were both logos)
Sorry buddy.
Oh, maybe.
Another explanation is that some people arriving were already familiar with you. They had already seen the previous logo. Now when confronted with the “B” thing, the first thought they had was “I am on the wrong site” and they back off.
There was a difference of 96 people. If the number of repeat visitors withing say last 6 months is significantly more than 96, you can’t really separate out the effect of previous branding.
That said, the stylized B is void of personality and poorly integrated into your site so I would expect it to lose. The good thing about your old logo is that it has a bingo card in it, immediately reassuring people that they are on a site having something to do with bingo cards.
This is A/B testing gone totally wrong. As Dennis said, the lost sales are most likely due to burning the logo recognition you had with your previous logo.
I’m a designer, and I’m the first one to denounce the bullshitting my colleages do. Change your logo to a giraffe and you would still have the same numbers. Logos (or branding in general) allows you to build meaning into a recognizable symbol. If you think the logo of Apple, Nike, or Coca Cola logo is great you probably also think the company is great.
If you provide great service, a solid product, friendly support and good value, your logo becomes a visual representation of it to your customer’s eyes. Provide shitty service and the same thing happes (AT&T). There are some things a designer can do to make it stick (being simple, or having a visual pun such as FedEx) but after that you’re on your own.
I love A/B testing, but this is impossible to track.
And BTW, I don’t think your site looks terrible, it looks like a thrift store: personal, honest and cheap.
Hey Patrick, did you ever do a logo vs no logo A/B test. I have a feeling that logos are remnants of print publisihing where they served as a hook to grab buyer’s attention when looking at the magazine rack. The closest visual element that accomplishes this is the favicon appearing at the tab bar or address drop down.
Unless you have a widget based marketing strategy, where you reinforce a brand visually, logos are a waste of time and money.
The 99designs logo looks pretty bad compared to some designs I’ve seen on there. And it really doesn’t fit with the “bubble” design of the site or the color scheme (especially as compared to the original). Of course, it’s easy to call the losing logo out after the fact, but this one seems abnormally bad.
Yea, the original logo is way better than the new one, which doesn’t seem to fit at all. Not only that, but the name of your product is obscured in the new logo. So, is this a fair A/B test? I would have to agree that it is a fair test and it does tell us something. I think it tells us that poorly designed logos impede the buyer’s positive experience.
Chris
Hi Patrick, I discovered your blog today (linked from an article on currybet).
I found this article incredibly informative and helpful! I guess it really highlights the effect a good quality logo can have. I think where people run into trouble is mistaking a logo for a brand. It’s really only one element (in fact, I recently wrote an article on logo’s and branding concepts.
In any case, thanks for the interesting article, I’d be interested in seeing the statistics over a longer time period, too.
Thanks,
Daniel
This is an argument that is hard to concretely solve because it’s so subjective. For instance, I think the chosen ‘new logo’ is absolutely awful. It says nothing about your business and clashes with the top header style in general. The original logo is almost certainly ‘bad’ in most people’s opinion, but not as bad as some generic no-substance flowy B that doesn’t fit with the site.
It’s as if I wanted to do an A/B test to determine if FedEx or Amazon had a better logo, and I tested it by putting Amazon’s logo on FedEx’s website. Well, no shit, Amazon’s logo looks terrible at the top of fedex.com!
Try this again with a _real_ new logo and I bet you’ll see the opposite happen.
I think the logo chosen from 99Designs is a nice logo, but not implemented well in this design at all. It’s not fair to just drop it into a design that was created for another logo. For one, if you
Oops, got cut off. Silly curly quote. I think the logo chosen from 99Designs is a nice logo, but not implemented well in this design at all. It’s not fair to just drop it into a design that was created for another logo. For one, if you’re going to go with that logo, then you need to have a picture or illustration of a bingo card elsewhere on the page. For two, the green swoop in the header was obviously designed to complement the first logo. And throwing the second logo on top of it doesn’t work, b/c the text of the logo gets lost. Not fair to throw 99Designs under the bus, in my opinion.
Question, how were you able to quantify the results?
You said that people would log on to the site and randomly get a different image. How did you measure that yours actually won?
A/B testing. I use http://www.bingocardcreator.com/abingo