Lesson from Madlibs Signup Fad: Do Your Own Tests

Periodically, news of an innovative, goofy, compelling, or compellingly goofy design decision will sweep across the Internets like wildfire.  Most recently, this happened with a madlibs-looking lead generation form.

I think it has much to recommend it in the context of lead generation forms (long, arduous monstrosity that you sign up for in the hopes you are contacted but not spammed to death), but I didn’t see much possible upside for using it on a new user registration form (short form which you sign up to use something).

However, I’m wary of trusting my instincts on such things when I could trust data instead.  There is a key point about A/B testing: trust your data, not somebody else’s data.  After all, you only make money when it improves your conversion rate, not their conversion rate.  You can feel free to use other folk’s successful experiments for inspiration but for heaven’s sake use them to inspire you to run tests, rather than inspire you to fire blindly.

I was particularly wary about trusting this result because, as pointed out by numerous people in the Hacker News discussion, roughly seven things changed between the two forms in the A/B test performed on the standard form versus the madlib form, and there is no particular reason to assume that the salient difference was caused by the part which strikes us as creative as opposed by more boring things like e.g. the call to action in the header.

When In Doubt, Test.  (When Not In Doubt, Test Twice.)

No less than six people said “Hey Patrick have you seen this madlibs thing yet?  You’ve got to try it.”, and because knocking something together would take less than 10 minutes because I have an A/B testing framework that makes this a one-line proposition, I decided I’d humor them.  I isolated just the madlibs versus standard style for the test, knocked up an alternative in about ten minutes with my (decidedly limited) CSS and Javascript skills, and set them against each other.  My conversion goal for this test is successfully inducing someone to sign up for the free trial of Bingo Card Creator.

My Usual Registration Form

The Madlibs Registration Form

P.S. If you have good eyes you’ll spot the other A/B test ongoing on this page.  I’m using the traditional way of mitigating cross-test interaction… ignoring the possibility of it.  Don’t tell your college stats professor, but this actually works pretty well in practice.

Results

I ran this test until A/Bingo, my A/B testing framework for Rails, told me that further testing was just a waste of my time.  It didn’t take long at all — 34 hours after the test alternative went live for the site, the first time I checked the results, they were already overwhelming.  Let me copy/paste right off my public results page:

Signup Madlibs Versus Standard Standard (27.55%) winner
Madlibs (21.73%)
95%

By my count that is a 22% decrease in conversion rates for using the madlibs signup style over the standard signups style, and the fact of the decrease (but not the magnitude) is significant at the 95% confidence level.

For the curious: there were 736 participants in this test, split roughly 50/50, as you would expect.  I love the Internet because where else can you get 736 people to help you improve your website while you sleep, work at the day job on Saturday, have an evening out with friends, and then sleep some more?

Anyhow: test ended, not touching the madlibs idea again.  Before adopting this or any other fad (or good suggestion, for that matter): do your own A/B tests.

No Responses to “Lesson from Madlibs Signup Fad: Do Your Own Tests”

  1. Mathew February 27, 2010 at 8:22 pm #

    Your point is well taken, testing with your own users is essential.

    Your test was not quite madlibs style – it doesn’t make sense as a sentence when you fill in the blanks. It actually sounds really awkward, and visually since it *almost* lines the fields up vertically it looks like it could be intentional but innacurate design.

    Of course that’s hardly the point, but it would be interesting to see the style tested on a number of websites with forms of different sizes and complexity.

  2. Pi February 27, 2010 at 8:33 pm #

    Have you considered that it’s because your madlibs style form is kinda ugly looking? (Please, please don’t be offended, am being sincere here) The copy isn’t very good, and the stacked input fields don’t look very good. These are factors that matter. Your original form is much nicer and cleaner. If you go back to the original article, the madlibs style forms are rather nice to read/fill. Importantly, while you can’t really do much to improve your original form, there is a lot of scope for improving your madlibs style form. Only then wouldn’t it be a fair comparison?

    I’m also guessing that the madlibs style form would works great for stuff that requires a lot of information to be filled. But that’s just me speculating.

    Finally, with just one experiment as a data point, please don’t disregard something as a ‘fad’. True, it might not work for you, but your wording influences a lot of people.

    Thanks.

  3. JB February 27, 2010 at 8:35 pm #

    I’m sorry, but I agree with Matthew in that your execution was lacking in the mad-lib form. Your A/B test only proved that the particular mad-lib form you tested isn’t any better than your standard-style form.

    This is just my opinion and interpretation of your post, but at the risk of starting an argument it kind of seems to me like you were skeptical that the mad-lib style has merit and so you set out to create an A/B test to prove yourself correct using a sub-par implementation of it.

    If you are truly feeling that not enough people are signing up for your service, and your stats show that the abandon rate is high on your signup form, then maybe you should try various styles of mad-lib form to see if you can maximize the success rate of that particular form and then compare those results to the success rate of your standard form.

  4. Tom February 27, 2010 at 8:41 pm #

    It’s a very reasonable point, you should always test assumptions with your own users.

    I think you have a much simpler signup form that most madlibs examples that have been shown. As such the madlibs style doesn’t add any clarity, and it certainly makes it less familiar. I’m not surprised that the traditional form performed better in your case.

    I think this post does prove the use of have a good testing framework in place so it’s easy to compare new features without apparently breaking too much of a sweat.

  5. zeldman February 27, 2010 at 8:53 pm #

    As others have said, it’s about your audience, their expectations, and your execution.

    On a site visited by digital hipsters, a well-executed, clever, off-handed form variant appealed to users’ in-the-knowness and resulted in higher conversions.

    I’d venture to guess that your site appeals to a more mainstream crowd. If that’s so, then the professional-looking “usual registration form” would be more appealing (and less confusing) than the alternative which you’re calling a “madlibs” form (but which essentially looks like a normal form that is not quite professionally put together).

  6. zeldman February 27, 2010 at 9:41 pm #

    That being said, I also don’t think Luke’s A/B test proves anything except that on the test site in question, conversion increased after a change in form structure and design.

    What part of that change *caused* that uptake is hard to say, and you certainly can’t abstract those results and expect them to work on every site.

  7. rick February 27, 2010 at 10:29 pm #

    I saw your attempt at Mad Libs and I knew it wouldn’t do any better. That other guy’s mad lib form was sweet… no offense, but your’s wasn’t.

    I bet if you do it up in a way that looks nothing like your usual form AND do a better job of the copy, it’ll work.

    But I think that’s the key to the success of that other guy’s form. It looks better, it’s simpler, etc.

  8. Loup Vaillant February 28, 2010 at 3:45 am #

    They all say your madlib is ugly, but didn’t say why. Personally, I see 2 whys: first, your madlib form uses many words, while your standard form is far more concise. Second, the rest of your box stayed in second person form. You didn’t went madlib all the way. This is inconsistent, and maybe even confusing.

  9. Melvin Ram at Web Design Company February 28, 2010 at 4:39 am #

    Hey Patrick,

    I think your first form is much more approachable than the new madlibs one… versus in the LukeW experiment, the madlibs version was much more approachable than the normal version.

    Also, in LukeW example, the madlib version of the form does work for you. It tells the sales person which vehicle you are interested in and helps you create the email by filling a few fields. Your madlibs version makes you (the user) do more work by making you read more without accomplishing anything more.

    I’m surprised your madlibs version performed as good as it. I would expect a much larger decrease in conversions so maybe thats an indicator that you could tweak this to make it really perform well.

    Here is what I thought might make more sense in the madlibs version of your form:

    I want to try out the Bingo Card Creator. Sign me up for a free trial account so I can create some cards, save them and print them out. My email is _________. Set my password to _________.

    I [would/would not] like to get 2 emails with hints for using the Bingo Card Creator and [please do/please don't] send me newsletters (about once a month) about new bingo activities.

    [Yes/No] this part would be drop down menu.

  10. Melvin Ram at Web Design Company February 28, 2010 at 4:48 am #

    Also, I would increase the line height on the form so the text fields don’t touch each other. It just feels weird visually.

  11. Dev February 28, 2010 at 6:50 am #

    I agree with the other commenters. How you can equate your madlibs form with the original post is beyond me. They are not visually similar at all. Seriously, how do you claim to do justice to the original idea with your version? It’s baffling.

  12. Dev February 28, 2010 at 6:57 am #

    I just wanted to point out that it’s how you present the madlibs form. Your english is totally wrong. This is what it should look like in my opinion:

    I’m [so-and-so] and I want to make bingo cards! You’ll know it’s me because I use [some-secret] as my password.

    Second, you absolutely have to lay out your paragraph correctly. The cramped “pickup truck in the front yard” looking mash of letters and fields in your example don’t qualify.

    Finally, I really don’t like the sign in as a guest idea. I think it’s absolutely not helping you. KISS. Demonstrate your value without begging me to come inside twice on the same page. It just seems cheap to me, which I suppose could be a good thing, but I’ll leave that one up for debate.

  13. a February 28, 2010 at 7:18 am #

    People don’t want to think. If you see a standard signup form that we are all familiar with, then you instantly switch off, recognise the pattern and fill the form in.

    However, if you see one of these mad libs forms, then it is completely unfamiliar. You pause and figure out how to fill the form in, and read the whole text, and thought is required on how to fill in the form. This break in thought, gives the option of signing up and not signing up. Hence the results, which I am not sure are statistically significantly different.

    I would say, that wording is particularly important in the mad libs version. It can be tuned to see if it has an effect based on different wording.

  14. rjp February 28, 2010 at 7:42 am #

    Why does the madlib form have a “sign in as guest” option that’s missing from the first one?

  15. MIke February 28, 2010 at 12:54 pm #

    I would also have to agree with your commenter(s) and feel you missed the whole idea of the mad-libs style form.

    First – in the original example – the madlibs form actually IMPROVED upon the original. It made things more readable, guided a user to a specific goal, and compacted a few of the form fields. Instead of them typing out a all the information, it saved the user time (or at the very least, gave the perception of it).

    In addition – your text heavy A/B example just seems off and unfriendly. The simple fact you changed the text to sound like a person “asking” for a password is just strange. We are trained to never tell anyone our password yet you make it seem like we are verbally telling someone our password. “We’d like to use THIS word as our password ol’ buddy!”

    Just strange.

    However, your LESSON is spot on. Always do your own testing, and just because Site A has feature B, does not mean your site needs it.

  16. Charles Boyung February 28, 2010 at 8:47 pm #

    You people have all missed the point (at least those of you questioning his “mad-libs” format). The original example changes SEVERAL items, not just the layout. He essentially removed the big, ugly comment box and also changed the phone number field from a really annoying group of 3 text boxes to a nice, simple single text box. These are not the only changes, either, but I think that those two changes alone would increase conversion rates. A true test of the success of this form would be to go back to the old style, but keep the same set of fields and functionality as the new form. Then compare the results of that with the mad-libs form. That is the only way that you could (semi-)accurately determine if the mad-libs form increases conversions or if it is just the fact that the fields on the form are now more user-friendly as-is.

    That said, I like the mad-libs style form if done right, and I think Luke’s example is done right, but his data is just as wrong as (or more wrong than) the data listed here.

  17. John March 1, 2010 at 8:42 am #

    I think the stats test is completely valid and I really love how fast he can verify that something improves something versus not improve something. Have you tried utilizing regression to get an idea of size of the impact.

    I use to do some shortage analysis (retail loss) at a major chain and could use that to get an idea of dollar amount stolen versus loss through operations (margin neutral).

    I believe you could set some of the factors in a regression as 1 or 0 and have multiple tests of the form. Position (2 positions of an element) , color (could test 2), font style (test 2) so have mutliple mutual exclusive questions at once to get a comparison of conversion significance for each binomial question. I think you could then get an idea of the level of lift for a multiquestion test. Perhaps you would then learn what each independent variable coeficient is and its signficiance. Though its better with whole number data questions like $ of ad exp versus sales.

  18. Ariel March 1, 2010 at 8:50 am #

    I see that quite a few of your site visitors are saying the same as I feel.

    This is a poor test, imo. Not because of the concept to proof, but the implementation of it.

    Nearly everything implementation of a design concept needs… design. Your original form has a lot of white space which easily and visually differentiates areas for the users. It allows them to know where the fields are vs. the information.

    Your “test” takes this away while leaving the same margins but filling it in with words. This made your design entirely chaotic, confusing, and would likely make your audience feel almost stupid because they had to go through a frustrating and tiresome form. Had things been redesigned and properly spaced, I wouldn’t be surprised if you at least hit your current target rate if not exceeded it as it makes a mundane task like signing up for something, into something fun and playful.

    Your implementation of this in your test was like a political party surveying a group, where their questions are so pointed towards one direction, leaving almost no room for objectible thought. Those types of surveys aren’t real deductions of a person’s true impression. Nor is your test a true test of this design concept.

  19. grimen March 1, 2010 at 2:26 pm #

    I guess you got a point, but the example was quite obious: A person with slightly design interaction (or cognitive psychology) knowledge would be able to state the winner with 2 seconds peeking. A/B-testing can be over-used; employ a interaction designer too and you will save some development time. ;)

  20. Joshua Clanton March 1, 2010 at 3:44 pm #

    Well, I didn’t expect to find that quite this many people had already made my point for me. I definitely agree that you should test everything yourself, because context makes a huge difference. But I think that you should also give Melvin Ram’s version a try in your next round of testing.

  21. gwenhwyfaer March 3, 2010 at 9:43 pm #

    > A/B-testing can be over-used

    No. A/B testing is how we know that what we think is obvious is actually true. Anyone who thinks that doesn’t need to be checked more often is… too confident.

  22. John Haugeland March 4, 2010 at 10:34 am #

    Good article, and thanks for saving me the hassle.

    A point of opinion: when you’re discussing the gap between two percentages, referring to said gap as a relative percentage is initially confusing.

  23. Ric March 5, 2010 at 5:38 am #

    Sometimes your use of the English language can skew results on an English language-based interface.

    Yes, you “conversationalized” the form, but in a very awkward and geeky way. “Save them under” and “use this word ______ as my password” are no better than the form because it’s still techno-speak.

    “My email address is ________ and, by the way, use ________ as my password.” Is a step closer to what you should have done. Also, the rest of his for was left unchanged. So it’s still basically a form.

  24. John March 9, 2010 at 7:57 am #

    I think the lessons learned from this are really powerful but hearing so many critiques on the form to form comparison I think a throw down should be done. Put a challange out for a better Madlibs form that meets all the sign up needs so it can go through the rigors of the A/B testing.

    All I see is up side to this. If the test you conducted wasn’t sufficient in the UI pretty department and you get higher conversions from a form from one of the nay sayers they hey you get a better form and they can say I told you so. If on the other hand the simpler current form keeps the title as best conversion method for bingo then you can thumb your nose at the critiques that are a bit more abundent on this piece then other pieces you’ve articulated.

    Either way the whole A/B testing wins and lessons are still highlighted to your captive readership. :) Love to see a throw down. Hey and of course it has to win with statistical significance. 95% I presume is the statistical setting.

    On another note have you come across type I errors in the A/B testing. I was reading about health research and the number of times they community retracts and refutes a previously studied health bennefit or risk and that they found that the rate of these retractions is following about the rate of expected type I errors.

  25. Sylvain March 31, 2010 at 6:49 pm #

    I would tend to agree that the form design of your test was not good enough for the reasons mentioned by others.

    However, I think there is another factor: The type & purpose of the form.

    The form in the example was designed to contact a dealer regarding a car for sale. The contact form on the left is blank and requires the user to write the message, even though what the message should be is obvious.

    The mad lib form looks like a regular letter and is already written for the user. The user just fills in the blank with his contact info and send. He doesn’t have to think of a message to write.

    Even though the comment section could be left blank in the original form, the user doesn’t know if the recipient will know what this is about, and so feels he need to write a message, which takes time, while the mad lib form feels much faster to fill out.

    So my conclusion is that the “Mad libs” version on the other site works mostly because it makes things easier for the user.

    On the other end, on a sign up form with 2 fields, the Mad Libs form makes things harder for the user: he has to read a message that he wouldn’t have written otherwise.

    About your “Mad Libs” form, there is also a problem with the copy: “save cards under email address” sounds like the cards will be sent to the email address or even stored directly the user’s email.

  26. Steve February 21, 2011 at 4:16 pm #

    This was not a well thought out test. You want to do it when there a number of fields to fill out and a more naturally flowing sentence. Also, you shouldn’t use text fields and the spacing is really close, making for difficult reading. The main issue here though, is this is a 2 field form. The madlibs is decidedly for a much longer form.

  27. John Laura April 27, 2011 at 5:32 am #

    I think that, like everything, it needs testing. I did found the article cited above really interesting. The link provided before is in French, and i couldn’t understand any of it !

    Here’s the english version of the article, it has a great comparison , with loads of images and Pros&Cons

    http://www.yucentrik.ca/en/2011/03/17/should-we-use-mad-libs-style-form/

Trackbacks/Pingbacks

  1. jardenberg kommenterar – 2010-02-28 | jardenberg unedited - February 28, 2010

    [...] Lesson from Madlibs Signup Fad: Do Your Own Tests: MicroISV on a Shoestring [...]

  2. -bridging data (collection) with story and context. « define:happiness - February 28, 2010

    [...] – :  http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-own-tests/ [...]

  3. Theo Mandel’s Usability Blog » Counterpoint to the “Madlibs Form Style - February 28, 2010

    [...] Lesson from Madlibs Signup Fad: Do Your Own Tests [...]

  4. Weekly Links and Comments | FrontEndBook - February 28, 2010

    [...] guy named Patrick McKenzie tried  the Mad Libs technique on a form and came to the conclusion that in his case with his users, the result was the opposite. I think [...]

  5. Listen to Lesson from Madlibs Signup Fad: Do Your Own Tests - MicroISV on a Shoestring - Hear a Blog - February 28, 2010

    [...] http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-own-tests [...]

  6. UXstudy–生活总是赏赐不浮躁的人们 — 换个风格: 像Mad Libs(疯狂填词)一样填写表单 - March 1, 2010

    [...] 欢迎分享你的主意。最新看到Madlibs风格表单试验, 不一定能增加转化率, 如果你使用的不够好,可能还不如原来那个。 [...]

  7. Labnotes » Rounded Corners 249 — Life before Google - March 1, 2010

    [...] only reliable data is yours. Rather than echo the latest discovery — the madlibs signup form — Patrick tested how well it performs on his site: By my count that is a 22% decrease in conversion rates for using the madlibs signup style over the [...]

  8. Daily Links #156 | CloudKnow - March 1, 2010

    [...] Patrick McKenzie: Lesson from Madlibs Signup Fad: Do Your Own Tests [...]

  9. Revolutie in formdesign? Mad-libs style verhoogt conversie met +20% | Lammert Postma - March 2, 2010

    [...] McKenzie maakt terecht een kritische opmerking dat het hier om een specifieke situatie gaat. Of zoals hij zelf zegt, vertrouw alleen je eigen data [...]

  10. Increase conversion rates with narrative forms | Ben Lloyd on digital marketing, media and culture. Evolution of The Digital Ape - May 12, 2010

    [...] technique hasn’t worked for everyone, Patrick from Kalzemeus found a deterioration in conversion rate with his take on a  madlib form, but his example could [...]

  11. Using Natural Language To Improve Conversion Rates - May 12, 2010

    [...] of fairness I should also link to this post from software developer Patrick McKenzie arguing these forms don’t work at all! Personally I don’t really like the implementation but that post emphasizes the most important [...]

  12. The Ultimate Guide To A/B Testing - Smashing Magazine - June 24, 2010

    [...] [...]

  13. TG Designer » The Ultimate Guide To A/B Testing - June 24, 2010

    [...] "Mad Libs" style form increases conversion 25-40%Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test. [...]

  14. The Ultimate Guide To A/B Testing « FED视野 - June 24, 2010

    [...] “Mad Libs” style form increases conversion 25-40% Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test. [...]

  15. The Ultimate Guide To A/B Testing | i know idea - June 26, 2010

    [...] "Mad Libs" style form increases conversion 25-40%Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test. [...]

  16. 王赟的分享博客 » Blog Archive » The Ultimate Guide To A/B Testing - June 28, 2010

    [...] "Mad Libs" style form increases conversion 25-40% Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test. [...]

  17. The Ultimate Guide To A/B Testing » abdie.web.id - June 28, 2010

    [...] "Mad Libs" style form increases conversion 25-40%Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test. [...]

  18. Conversion war: l’illusione delle best practice - Alberto Mucignat - June 30, 2010

    [...] leggevo di a/b testing e lo sguardo mi è caduto (sarà l’eyetracking che mi contagia) su un articolo che spiega perché le cosiddette “mad libs” style form di cui ha parlato recentemente [...]

  19. A/B测试终极指南 « Max's UED - July 4, 2010

    [...] “Mad Libs” style form increases conversion 25-40% Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test. [...]

  20. Just. Act. Natural? | Kerpak Blog - July 30, 2010

    [...] developers have claimed that it provided no help to users and in fact became a barrier to success.  So, putting on my Market research hat, I went looking [...]

  21. Quora - December 7, 2010

    Where are some well executed mad-lib style websites?…

    The huffduffer sign-up form is a good and fairly well-known example of this. Luke Wroblewski redesigned the Contact Dealer form at vast.com as mad-libs-style forms, and reported that it increased conversion rates by 25-40%. Rustin Jessin has a mad-libs…

  22. Yu Blog, le blogue de Yu Centrik, Expertise en Utilisabilité et ergonomie des interfaces - March 28, 2011

    [...] contre exemple vient de l’article Lessons from Mad Libs Signup Fad : Do your own tests. Le ton de l’article montre clairement l’opinion de l’auteur sur les Mad Libs. Il y fait [...]

  23. The Ultimate Guide To A/B Testing — Mafuge - May 16, 2011

    [...] “Mad Libs” style form increases conversion 25-40% Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test. [...]

  24. Mad Lib style forms | 削面一宛 - November 22, 2011

    [...] [7].http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-own-tests [...]

  25. UX Designer - January 5, 2012

    [...] “Mad Libs” style form increases conversion 25-40% Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test. [...]

  26. Стоит ли нам использовать повествовательные веб-формы? — UX Fox — дизайнеру интерфейсов - January 5, 2012

    [...] http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-own-tests/ [...]

Loading...
Grow your software business:
(1~2 emails a week.)