What My User Survey Taught Me

Some weeks ago I mentioned that I was implementing a user survey in Bingo Card Creator, using Wufoo.  About forty of my customers have now taken the time to give me detailed advice.  I thought I’d share some things I learned.  A few takeaways may be applicable to your business, and at the very least “detailed, actionable advice can be yours if you just ask for it” should convince you to start a survey this week.

Incentivize Any Surveys You Do

I ran two A/B tests in taking the survey.  The first selected whether a user was asked to take it at all.  The second tested what prompt was more effective at inducing people to take it.  Half of participants were invited to to take it for altruistic reasons (“Would you like to take a survey about Bingo Card Creator?  We’d like your feedback so that we can improve the website and software for all of our customers.”)  The other half were incentivized (“Would you like to take a survey about Bingo Card Creator?  If you do, we’ll let you print 5 extra free bingo cards.”)

I gave all users who completed the survey the free cards, regardless of whether they had been promised it or not.

I have never had a more lopsided A/B test.  The response rate of incentivized users so resoundingly crushed the response rate of non-incentivized users right out of the gate that I was scared of a bug.  It ended up being more than a 2X difference in conversion: 2.51% vs 1.17%, which is significant at 95% confidence.

Thus the conclusion: if you want responses, give away free stuff.  (Incidentally, you might think that you’d get lower quality feedback from incentivized users.  I did not get that impression from reading the results, but I can’t reduce that to a simple statistical measure.)

Surveys Don’t Cost You Sales

A major worry I had was that putting the survey prominently on the trial version of my application would cost conversions.  Nope — there was no significant difference in sales given to folks asked to take the survey versus folks not asked to take the survey.  (I actually got very, very marginally more sales from the folks asked — but not statistically significant.)

Opinions Confirmed About My Customers

Prior to instituting the survey I had guessed that my customers were mostly female, older (I was guessing normally distributed around 40), and that they were probably teachers.

  1. I was dead on the money on gender.  85% of participants were ladies — this roughly comports with my experience looking at customer names and doing customer support.
  2. My customers are, indeed, mostly older: fully half of them are over 40.  Another 30% are in their 30s.  About 10% are in their 20s and a little less than 10% are below that.
  3. I now have data to substantiate that most of my users are teachers: 30% of respondents use BCC in elementary schools and another 20% use it in high schools (that number is much higher than I would have expected).  I was mildly surprised with the number of people playing with adult family members (~15%).  All other uses are fairly marginal.  Confirming that most of my users are teachers is going to be very helpful in crafting my marketing messages.  (Though I suppose I have to worry about stumbling into the local optimum: I might be seeing this because I market to teachers well and market to, e.g., parents poorly.)
  4. Two survey takers, as predicted, used the survey free response box to ask customer support questions.  One of them I was able to de-anonimize and fix the issue she had (she had created a new trial account rather than using the old registered account and wondered why it was asking her to purchase).  The other one, sadly, was using an anonymous guest account from an IP I’ve never seen before, so I couldn’t track them to an email address to resolve their issue with the purchase.

Things I Didn’t Know Prior to Asking

  1. A lot of my users are very, very appreciative of Bingo Card Creator.  I mean, I knew it saved them some time and that lots liked it, but I got stories of it saving a lesson plan, brightening the day of a room full of seniors, and teaching a son to read.  A few of the results warrant follow up emails to ask if I can quote them publicly.  (Like, right next to the Buy Now button.)
  2. I am terrible at Quality Assurance.  Well, I knew that.  But, specifically, a particular combination of A/B tests would result in a user getting textual instructions on one screen which conflicted with what was written on the buttons on the next screen.  Thank you for the report, ma’am — bug squashed.
  3. A few customers reported generalized anxiety about using “new programs” and said they wanted more handholding in the instructions.  I beefed up instructions throughout the application.
  4. Surprisingly few of my customers reported problems with ease of use — over 90% rated BCC either “Very easy to use” or “Mostly easy to use”.  I also got a lot of free-form comments praising the daylights out of that — many of my customers compared it favorably to other unpleasantness they had had doing routine computer tasks.
  5. Surprisingly many of my customers self-evaluate as comfortable with computers.  50% were “very comfortable”, and 30% were “mostly comfortable”.  These numbers are, candidly speaking, not what I would have assigned on the basis of reading support requests for three years.  It is possible both the survey and I are right, just looking at different segments of reality: 80% of the customers are good with computers and fairly rarely email me, and 80% of my inbox is caused by the remaining 20%.
  6. Customers respond very strongly to features I consider so core to the program I scarcely mention them — in particular, I  learned twelve different ways to express the thought “Every card is different!” and another eight for “I love that I can customize the word list for the lessons I am teaching that week”  I will be incorporating additional variations of those into my copy.  Repetition never hurt any teacher.

The Number One Complaint My Users Have

Bingo Card Creator isn’t free.  Some variations on the theme:

Make it totally free by having advertisers. Many, many sites that elementary teachers use are kept free by the magic of Google AdWords.  I should know — I’m the guy paying for the ads.  While you can keep the free bubble going by passing around VC dollars to Internet firms to Google to Internet firms to Google to Internet firms to Google to … for a while, eventually, if you’re not getting money from customers, everyone dies.  This was, essentially, the last Internet bubble.  I will continue charging because charging keeps the rest of the ecosystem alive.

Relatedly, the fact that Bingo Card Creator is so easy to use is directly related to the fact that I charge money for it.  I obsess over getting customers (or trial users) through the pages to their beautiful bingo cards, because people who see beautiful bingo cards are very inclined to pay me money.  The sites that I advertise on do not optimize for their user experience because if their website is better than my textual ad, they don’t get paid.  That is why the experience of using them sucks.  (I won’t out anybody who I essentially have a business relationship with, but take a look at the free options in my market some time, or look at pages which compete with mine in the search results.)

Charging also subsidizes the experience of the 97.3% of trial users who don’t actually pay me money.

15 card limit does not allow for a class set in which everyone can have a different card 25 would be more appropriate for teachers.”  I regularly get asked what my business model is — i.e. how I convince people to pay money for my software when I give so much away for free.  This lady nailed it in one sentence.

Incidentally, I don’t feel any ranchor at folks who believe that everything should be free on the Internet.  I just will not accomodate your preferences.  You’re welcome to use my free competitors if they better fit your needs.  (I actually provide folks  with lists, on request.)

Takeaway Lesson

What are you waiting for, go sign up for Wufoo (or whatever — they have a few competitors) and do a survey.  You’ll learn stuff that you can use to make helpful decisions for your business.

No Responses to “What My User Survey Taught Me”

  1. Patrick Sullivan February 7, 2010 at 9:07 am #

    The fact that you did not see anything to indicate they blew through the survey just to get the free stuff is interesting to me, as I had a similar concern to using such methods.

    I wonder if it’s widely applicable, or if the teacher subsection you so heavily target/get sales from are more prone to “do the right thing,” since that makes a difference in how useful this info is to the rest of us ;-). Obviously some market subsets WILL be more prone to just blow through it, but it is a question of which is the norm for any given random group.

    Thanks for continuing to share so much great info.

  2. Pascal February 7, 2010 at 3:53 pm #

    So instructive, as always. Thank you for sharing all the details, Patrick.

  3. Matt Mazur February 7, 2010 at 6:48 pm #

    Patrick, thanks for all the excellent articles you’ve written and posts you’ve made on HackerNews. I make it a point to read your posts when scanning through the comments. The quality is always top notch. Appreciate it.

    • Patrick February 8, 2010 at 5:07 am #

      Thanks guys. I’m glad you liked the article.

  4. Steve Hanov February 8, 2010 at 5:18 am #

    I would like to add that surveys should always have the opportunity to enter free-form text. Often, I have a specific complaint or experience to share and it is not possible to communicate it through multiple choice questions.

  5. Hrishi Mittal February 8, 2010 at 5:32 am #

    Patrick, thanks for another excellent article. I really enjoy reading your articles and comments on HN.

    I am not really into bingo, but I think I’m going to buy a copy of BCC because I think I could learn as much from your creation as from your words (and maybe I’ll become a bingo grandmaster). I have one question though – does the online version give unlimited usage? Sorry if I overlooked the details, but I couldn’t find any indication of it being a monthly or annual subscription. I imagine that the customer demographic you are dealing with would not like the idea of a recurring cost for an app like this. But then, how do you plan to provide this service indefinitely into the future?

    I’m trying to come up with a pricing model for Pretty Graph (a webapp for making err.. pretty graphs. See http://prettygraph.com). I’ve been considering a single one-time fee but can’t think of how to support it outside a limited timeframe. I’d appreciate your comments.

    Thanks once again and keep writing!

  6. Marcin February 8, 2010 at 8:33 pm #

    With regards to what Steve mentioned, as anyone experimented putting such free-form text fields in the beginning of the survey rather than the end? Hypothesis being that the survey taker is more likely to provide accurate feedback on the multiple choice after having vented their frustrations (if any) out up front.

  7. Rob Walling February 8, 2010 at 10:38 pm #

    It’s amazing how much you can learn about your customers just by handling support emails. It sounds like you had almost everything correct in your head but without the stats to back it up.

    But of course, now you have them.

  8. Andy Brice February 10, 2010 at 4:13 pm #

    >Surprisingly many of my customers self-evaluate as comfortable with computers. 50% were “very comfortable”, and 30% were “mostly comfortable”. These numbers are, candidly speaking, not what I would have assigned on the basis of reading support requests for three years.

    I had similarly surprising survey results a while back when I asked people to self-assess their computer competence. One explanation is that people who are unskilled at something lack the skill to judge just how unskilled they are. Look up ‘unconsciously incompetent’ on wikipedia. This maybe why you never hear anyone admit to being a bad driver – the bad drivers don’t enough knowledge of what good driving is to appreciate how bad they are!

  9. Avery Otto February 11, 2010 at 4:59 pm #

    User feedback is mission critical and a really good way to do that is through the use of a User Survey. We do software development and we conduct user surveys through the team interview feature of Code-Roller. See http://www.dynamicalsoftware.com/cr/prod/home.php?entry=18 for a short introduction to that. Code-Roller is how all members of the team collaborate.


  1. Steven’s weblog » Another reason that ad-supported == teh suck - February 7, 2010

    […] this article from ‘MicroISV on a shoestring’ (which is a blog worth keeping up with, if you ask me) has an interesting insight. Patrick says […]

  2. What Happens When All Our Content Comes from Content Factories? | Byrne's Blog - February 8, 2010

    […] in making that interaction frustrating, but not infuriating (it bas to be frustrating because, as Patrick MacKenzie points out, the content must be less useful than the ads. If it’s infuriating, visitors will just leave. […]

  3. Listen to What My User Survey Taught Me - MicroISV on a Shoestring - Hear a Blog - February 9, 2010

    […] http://www.kalzumeus.com/2010/02/07/what-my-user-survey-taught-me/ […]

  4. 8 ideas for understanding product / market fit using the web – Grayson Technology - February 9, 2010

    […] Here’s an excellent post on “What my Survey Taught Me” . […]

  5. Unskilled and unaware of it « Successful Software - February 16, 2010

    […] Note how skewed the results are and bear in mind that relatively few of my customers are IT professionals. Similarly skewed results were reported recently by Patrick McKenzie in his blog: […]

  6. Late Stage Co-Founder Wanted, Screencasting Handbook Released, Triumph of the Nerds on DVD, and more… | Software by Rob - May 24, 2010

    […] What My User Survey Taught Me – Insights from Patrick McKenzie. My favorite quote: “Incidentally, I don’t feel any rancor at folks who believe that everything should be free on the Internet.  I just will not accommodate your preferences.  You’re welcome to use my free competitors if they better fit your needs.” […]