You Should Probably Send More Email Than You Do

For the last six years or so, I’ve blogged, participated in online watering holes like Hacker News and the Business of Software boards, tweeted, spoken at conferences, and been very public about what I’ve learned about making and marketing software. Folks tell me that my advice has made their lives and businesses better, which is enormously motivational to me. Also, all this online participation has a variety of helpful side-effects for my business: more consulting leads, link juice to help my SEO out for the product businesses, reputation/credibility enhancement, yadda yadda.

One medium I’ve never made much use of personally is email. This is objectively suboptimal, because email is the best broadcast channel ever invented for business purposes. There’s no time to fix that like the present, so a) I created a mailing list, you should probably sign up and b) I would like to explain why your business should probably send more email than it is right now.

The Thirty Second Sales Pitch

Give me your email address and I’ll send you things that you’ll enjoy. For example, immediately after you confirm your email address, I’ll send you a link to watch a free 45 minute training video on improving the first run experience of your software. Why would you care about that? Because you’ll learn stuff like:

  • how two hours of work lets me sell $10,000 extra of Bingo Card Creator a year
  • how I more-than-doubled customer lifetime value on Appointment Reminder by implementing a product tour
  • why some software companies that you’ve heard of pay a fair bit of money for similar advice
  • tactics that two folks have already reported made their businesses more successful after implementation

In addition to that video, I’ll periodically send you other stuff you might be interested in. For example, last Friday I sent the mailing list a very well-received, in-depth look at how to price SaaS plans. Twenty people wrote me back and said they found it useful and would implement at least one recommendation in their own business. (Didn’t see it? If you’re on the list, send me an email.  If you’re not, sign up, you’ll get it automatically tomorrow.)  That’s a specific example, in generalities you’ll get:

  • exclusive looks at software/marketing topics which for whatever reason aren’t right for this blog
  • announcements when I produce something of particular interest to you (like e.g. my Business of Software presentation, which is the smartest/funniest I’ve ever been in 7.5 minutes)
  • announcements of new product offerings, should I ever decide to make anything that I think you’ll find interesting

Email: Geeks Ignore The Best Marketing Channel Ever

A long time ago, I used to be an anti-spam researcher. I was very, passionately committed to getting 95% less email in people’s inboxes. This lead me to have an enormous psychological block against collecting and sending emails for my businesses. My perceptions were that:

  • users bucket email into “tolerably annoying” and “hated with passion unmatched by a thousand burning suns”
  • email marketing is universally kind of seedy
  • asking for email addresses would damage customer confidence
  • other methods of communication made email obsolete
  • ethical use of email is economically marginal for the business

These perceptions were catastrophically wrong. If you are currently where I was six years ago, let’s have an intervention.  You should start collecting emails from people interested in your topic of expertise and periodically dusting off their inboxes, too.  Here’s why:

Presentation Of Content Influences Perception of Value

I try to avoid the word “content” because I think that “content” auto-commoditizes the value of whatever it is you are offering. Shakespearean plays would sound like terrible wastes of time if they were described as “content.” Regardless, I’m going to use “content” illustratively for the next few paragraphs.

Have you ever heard the phrase “You can’t judge a book by its cover”? It’s the worst kind of horsepuckey. In addition to the fact that it is trivially possible to judge a book by its cover, it is an empirically observable fact that most people, when presented with a book, will judge it by its cover.

Content gets judged by it’s cover.

  • Visual presentation is treated as a proxy for content quality and is provably dominated by first impressions formed in 3 seconds or less (see “Blink”).
  • Content which bears social proof (including e.g. visual elements suggesting endorsement by news media, tastemakers, or one’s peer group) will, in A/B tests, often ROFLstomp absolutely equivalent content lacking social proof.
  • Content appearing next to things which suggest high value will tend to inherit that high value. (This is the entire justification for the NYT’s rate card.) Content appearing next to things which suggest low value will tend to inherit that value.
  • Content has perceived value accutely dependent on what you describe it as. Novels are valuable, text files are not, even text files which are bitwise equivalent to novels. Reports are valuable, white papers are less valuable, PDFs are virtually valueless. (Seriously — A/B test this on a lead capture form.)

Here’s a concrete example for you: 37signals recently offered to give away free copies of their best-selling book Getting Real in return for mailing list signups. The positioning for it was, unfortunately, as a “free PDF.” Some members of HN proceeded to question the value of the give-away, including by interpreting the positioning for it as “a PDF file full of content that was published years ago to the Internet.” That’s painful. For what it’s worth, Getting Real is a great book with good advice in it, some of which is very actionable.

Given that reading Getting Real is a win for the reader and a win for 37signals, as a marketer, you want to convince people that reading it is in their interest. This begins with positioning it in such a way that people perceive value out of it. For example, rather than calling it a “PDF file”, which is begging to get valued based on the price of bits by people who think all bits deserve to be free, you can call it a best-selling business manifesto written by beloved industry experts. (That’s certainly marketing but, crucially, not a word is false. If you believe that cynical Internet-types would be less likely to download it if it were phrased that way, you could hypothetically A/B test that, and you would hypothetically lose.)

What Does That Have To Do With Email?

Consider this blog post (we’ll walk it back to email in a second). How are you reading it right now? Statistically speaking, you’re probably reading it either through a feedreader or on an aggregator like Hacker News. It is sandwiched somewhere in the middle of 30 other posts, all presented with the same visual weight. You will spend, statistically, an average of under three minutes on it. You will, statistically, not read all of it. This is a shame, because this post is orders-of-magnitude more important than a lot of the dross it is located next to.  However, because humans are humans, you (statistically) are going to associate it with dross because that is where you found it.

I’m not immune to this: my feedreader has over 1,000 items which I’ll never get to. Here’s what I see on those rare occasions when I open it:

Don’t look at this from my perspective.  Look at it from Aaron Wall’s perspective.  He’s one of the savviest SEOs alive and has a SEO training business which he would like to promote to people.  (Disclosure: I used to pay him $300 a month for it.  I spent waaaay too much time on his forums and answered so many questions that he made me a moderator and comped my subscription, demonstrating further that given the opportunity and a text input field I pretty much can’t help myself.)

Anyhow, Aaron is competing with thousands of unread items which are all presented as being approximately equivalent in value to his offering.  They’re not!  Egads, they’re not!  As much as I love with keeping up with the news coming from commentators about World of Warcraft, SEOBook is a bazillion times more valuable to me.  But I’m still probably not going to read that article!

Now let’s compare this to my inbox:

See those twelve unread messages?  How many of them do you think I’m going to read today?

That’s right, I’m going to read all messages in my inbox, because like most white collar professionals reading and responding to email is my effing job.  If Aaron Wall had hypothetically sent me his actionable tactics on how to market my business online, then that would get a halo effect from being in my inbox.  It wouldn’t be competing with the news about the Diablo 3 launch, it be in my workflow right between answering a question about the software I sell and re-sending a five figure invoice to a client.  I read and pay attention to my email, whereas blog posts get skimmed or dismissed outright.

Emails Get Opened And Change Behavior

Having done this for six years, I have good horse-sense with regards to the anatomy of blog posts and where links go on a page.  For example, if I put a link like “If you do nothing else to improve your copywriting go take a look at CopyHackers” I can tell you that, at this point in the post, maaaaaybe 1% of readers are going to actually click on that.  (The rest of y’all are missing out, seriously.  I could practically justify my consulting rates just doing dramatic readings of that for the web teams at my clients.  Application of the advice therein has made tens of thousands of dollars.  This personal recommendation may get that click through rate up to 3%.)

Now, if you’re an email-skeptic, you’re going to say “Well sure, but that’s got to be better than email, right?”   Wrong!

I soft-launched my training email list a few weeks ago, and have some seven hundred odd subscribers on it.  When I sent them an email last Friday about SaaS pricing, fully sixty percent of them read it.  (Sidenote: Folks thought I was going to get an email list full of mailinator.com or throwaway email addresses, since I was giving an incentive for signing up to the mail list.  cough  Not so much, no.  Folks scamming you for the incentive are a — cheap — cost of doing business.)

Roughly 25% of folks who read the first email (or a full 15% of the mail list) clicked on a link with approximately similar prominence to that one.  (It was to Ruben Gamez’s writeup on how and why he changed pricing for BidSketch.  Seriously worth a read.)

Changing behavior can be substantially more lucrative than just getting a link clicked on.  For example, if you’re not just publishing anonymous “content” but rather trying to reactivate users of an application (lifecycle emails) or educate-them-to-the-point-where-they-want-to-buy-something (drip marketing), a little nudge in behavior could result in adding thousands of dollars of customer LTV.  One of my consulting clients could trivially afford to include a ten dollar bill with every email.  (You’re already thinking of mails hitting spam filters, mails not getting read, and mails not getting acted on.  Yep.  You’re overestimating all of those numbers, but yep.  A few percent survive all those hurdles and result in thousands of dollars of sales at the margin.  Email is a numbers game and the numbers are very motivational indeed.)

Blog posts are very poor at changing actual behavior.  Here’s a “conversion” I’d like to get out of everyone in our space: I want you to start A/B testing, because it will make you a substantial amount of money.  I’ve beaten that drum so many times that folks identify A/B testing with me and ask me for my opinions about it.  I have probably told a hundred anecdotes like “I just did an A/B test and increased software sales by 70% with 99% statistical confidence.  The change was a two-character configuration tweak that I dismissed on a hunch six years ago.”  (That totally happened this May.  Ask me for details later.)  My consulting clients frequently tell me they’ve read my things on A/B testing for years, been very intrigued, and yet no amount of saying “This is worth millions of dollars” has ever convinced them to have one of their twenty employees spend 15 minutes implementing an A/B test.  It was downright surreal for me the first half-dozen (!) times this happened: folks who have followed me for years and could, in many cases, literally quote from my writing on this subject had never acted on it.

(Speaking of presentation changing perceived value: If you pay nothing for expert advise you will value it at epsilon more than nothing, if you pay five figures for it you will clear your schedule and implement recommendations within the day.  In addition to this being one of consulting’s worst-kept secrets, it suggests persuasive reasons why you should probably extract a commitment out of software customers prior to giving them access for the software.  Doing this will automatically make people value your software more.)

People — Including Geeks — Actually Like Getting Email!

I always thought I really hated getting email.  It turns out that I was not a good reporter of my own actual behavior, which is something you’ll hear quite a bit if you follow psychological research.  (For example, something like 75% of Americans will report they voted for President Obama, which disagrees quite a bit with the ballot box.  They do this partially because they misremember their own behavior and partially because they like to been seen as the type of person who voted for the winner.  99% of geeks will report never having bought anything as a result of an email.  They do this because they misremember their own behavior and partially because they believe that buying stuff from “spam” is something that people with AOL email addresses do, and hence admitting that they, too, can be marketed to will cause them to lose status.  The AppSumo sumo would be a good deal skinnier if that were actually the case, but geeks were all people before they were geeks, and people are statistically speaking terrible at introspection.)

For example, when I started going back through my inbox and memory to find emails that had really struck a chord with me, I remembered that two onboarding emails from Twilio had been particularly good.  I mean, sure, “I hate email”, but when Jeff Lawson sent me a note asking how Twilio was doing I read it.  (Subject line: “RE: Welcome to Twilio”  This is the point where geeks point out that they hate when evil marketers hack their brains and make them click on the message to open it and discover that Jeff Lawson didn’t write it to me personally.  Trust me on this one: I not only was not pissed off at that — I didn’t remember it until I opened it again just now — I went on to build an entire business on top of Twilio, with the first step largely being prodded by a well-executed series of email which kept bringing them back to mind over a few weeks.)

I assume you can probably see a similar mail sequence if you sign up for Twilio right now, so go do that.

My first business, Bingo Card Creator, makes extraordinarily light use of email.  I send out a newsletter on every month with a major holiday, telling people to make $HOLIDAY bingo cards with Bingo Card Creator.  The newsletters practically write themselves off the usual template, with slightly different graphics and a find/replace for the name of the holiday and tracking codes.  If I forget to send one of these newsletters real customers actually find my email and complain.  That blew my mind the first three times it happened — probably because part of me thought the newsletters were not really delivering value.  In hindsight, that was stupid.  People pay money for Thanksgiving bingo cards.  Of course they want to hear about them when go out of their way to check “I’d like to get a newsletter about new bingo activities, about once a month.”

Permission And Trust Are Worth Money

I had the Rails request log opened with tail -f e it when I launched my email list.  (Geek-to-English translation: I was watching who signed up in real time.)  Sixty seconds later, a particular email address caught my eye when it was confirmed.  Let’s call it Totally Not Mark Zuckerberg (TNMZ) since it wasn’t Mark Zuckerberg but was a person who I wanted to make the acquaintance of.

TNMZ has apparently been reading my blog for years.  I never knew that (d’oh!) and would have never presumed to contact TNMZ because I assumed that TNMZ would have better things to do than read email from me.  Explicitly asking for email from me makes that worry seem sort of silly.  The ability to reach out to TNMZ and know that contact would be welcomed is worth meaningful amounts of money to my business, even if nothing ever came from the mails sent by the actual mailing list.

Similarly, if you sell SaaS, and get say a hundred prospects to give you their email addresses, you can do motivational things for your business just by sending out 10 “Hey I saw you signed up.  I’m the CEO.  What questions can I answer for you?” mails to the people who look like the most promising prospects.

This is an ability that you don’t get with blog posts and largely won’t get with other broadcast methods like e.g. Twitter.

There are, of course, other companies which make scads of money from carefully curated email lists.  DailyCandy sold for nine figures.  Groupon is an email marketing company, growing to billions of dollars of sales on exploiting an arbitrage between “It’s comparatively cheap to get young, professional women to sign up for an email list” and “After they’re on the email list, they spend motivational amounts of money at its direction, and we get half of gross.”  (P.S. That arbitrage opportunity is now fished to heck and gone.)

Almost every first-class e-commerce company treats their house email campaigns like they are the goose that lays the golden eggs, chiefly because they are.  For companies which have repeat-purchase models, direct response to emails can represent half or more of customer lifetime value.

One of the largely-unsung secrets to Facebook having such an insanely high user retention rate is that they use activity from your friends to give you highly personalized emails designed to bring you back to the site and post stuff.  (A detail I really like: you can reply to email and then post things on Facebook without even visiting it.  Facebook will then, predictably, hit your friend with an email and use social pressure from you to bring them back in.)  Many social startups are extraordinarily aggressive with this early in their lifecycle (I’m looking at you, Quora) and gradually back it off over time.

Some folks might think I’m saying Build Trust With X, Monetize Trust With Email Sales Pitches, but it can work in exactly the opposite fashion, too.  For example, I wrote a drip marketing campaign (a series of emails scripted to go out at particular intervals) for WPEngine.  They sell high-end WordPress hosting, and every sale requires a strong commitment to change on the part of the customer — migrating your blog is not easy or fun for most people.  They also charge multiples of what the typical WordPress host charges, because they’re not the typical WordPress host, so it is imperative to educate the customer on the difference.  Many customers will not sit down and read 10,000 words of your marketing copy just because your face is pretty, but if you pitch them something like e.g. a course on how to improve their own WordPress installation, they’ll happily sign up for email from you.  (Really.)  You can then spend, e.g., eight emails over the course of a month educating them on what happens to WordPress under load and how to improve that, what WordPress’ security record is like and how to lock it down, and how browsers fetch and display content with reference to how to optimize one’s site to take advantage of it.  As you gradually build up trust as a respected provider of optimization advice that your customers (if they are diligent) can see working for themselves week in and week out, you can get more aggressive with “So we talked about X and Y and Z, and you’ve implemented some of it and not implemented some of the rest yet.  We could talk your ears off about this, but tell you what, if you switch to our service you’ll get it all because we breathe this stuff.”)

This type of approach demonstrably works well at selling software, even software which you might expect would require high-touch consultative sales processes.  Indeed, this is a way to scale the initial parts of the high-touch sales funnel while simultaneously passively collecting information about customers to help you do lead qualification for the really intensive portions, like say webinars or live sales presentations.  It also scales down beautifully to low-touch self-service sales approaches, too.

Email Keeps Your Audience Warmed Up

You might have heard the old advertising saw that someone needs to hear about your brand seven times before they buy.  I personally think that one is poorly sourced industry lore.  In place of it, let me cite something which is verifiably true: the probable value of a prospect goes down over time, the provable value of a customer goes up over time.  (The predicted future value of a customer is an odd duck for many SaaS companies.  I’ll sketch out the shape of the curve some time.  It’s a weird snake that requires a bit of explanation.)

You can break out the numbers in your own CRM to see this: your likelihood of getting an order from a prospect goes down sharply the longer they’ve been in the CRM, largely because the “lead gets cold.”  (Indeed, in a lot of fields you can lose up to 90% of the predicted value of the lead within the hour.  Crikey.)  Software businesses will see a similar effect in their own sales: for example, Bingo Card Creator trials (which are not time-limited) will lose 90% of their expected value after the first day (i.e. 90% of purchases happen in the first day), another 90% in the first week, and another 90% in the first month.  (i.e. Only one out of about a thousand sales happens more than one month after the trial starts.  That’s actually an old stat — it went waaaay up when I started emailing folks who opted-in on approximately a monthly basis.  Every time I blast out an email I get a few hundred bucks.)

Regularly getting in contact with customers or otherwise getting on their radar keeps leads warm.  Some of my more successful clients and confidants successfully get orders from people who’ve been in their company “ecosystem” (fan of the blog, trialed a product four years ago, signed up for email list, etc etc) for literally years.  And not “An order here and and order there” either, no, this is a repeatable process that they put in place because it works.  They plan out email campaigns with more sophistication than many of us would plan out actual product launches.

Email Helps Get Marginal Revenue Out Of Existing Customers For Almost Nothing

This worked on me, recently: I signed up for the Blue Nile mailing list back in October when I ordered my engagement ring, and they have sent me approximately 45 newsletters (I counted) since then.  I think I opened maybe two of them…  but sure enough, when I remembered I needed wedding rings too, they were sitting at the top of my inbox.  Guess who got the order?

You can do so, so much more with emails than the typical SaaS company does.  Here’s two stupidly simple things you can do to celebrate a customer’s first year anniversary with your service:

  • offer them a month free if they pay for a year of their current plan in advance
  • offer them a substantial discount (say, 25%) if they upgrade to the next higher plan level  (optionally, if they do it for a year in advance)
For the right companies and the right offers, that can mean $100 per email sent in extra customer LTV.  Plus, if you ask them for money in advance, you get the opportunity to spend it immediately.  (This is enormously motivational for SaaS businesses because they typically have godawful terrible cash-flow issues with customer acquisition costs front-loaded and revenue dripped out at a mostly steady pace for years.  That is typically the reason that SaaS companies attempt to seek investment: despite the cost of producing SaaS cratering over the last years, scalable customer acquisition channels are, if anything, getting more expensive over time.)
Paul Stamatiou wrote up how Picplum used this to their advantage.  I plan on covering the topic in substantially more depth, later.

Does This Email List Mean You Going To Stop Blogging?

One might reasonably ask where I’m going with this private email list.  Good question!  I’m not entirely sure — I have a feeling that I’ll write a mix of free material for it and, eventually, produce something which will add a lot of value for software companies.  That offering will be priced commensurate with value.

The amount of effort I put into my blog waxes and wanes with where I am in life at any given time. Some folks have recently noticed that I haven’t been posting as much as I used to. I plead “upcoming wedding” with the extenuating circumstance of “business is on fire (in a good way).”  Going forward, if I have something which makes a good fit for the blog, I intend to write it up as life permits.

Have a topic which you’d like to hear about?  Drop me a comment or email.

P.S. To save you a scroll-back-to-the-top, if you’re looking for the signup link for emails from me, it is here.

33 Comments

Kalzumeus Podcast Ep. 2 with Amy Hoy: Pricing, Products, And Passion

Keith and I recorded the second podcast, this time with special guest Amy Hoy. (If you missed the first podcast, see here.) We’re still searching for a format which really works for us, so this is a work in progress. Please share your thoughts with us on what you like/don’t like about it.

This podcast was recorded two months ago, largely because Keith and I got too busy to do the editing and post it. We’ll outsource more of that in the future. Of particular note: the 30×500 class that Amy talks about is already started, so if you’re interested in it, sign up for her email announcement (link waaay down on that page) for when it opens the next time.

Major topics for this podcast included:

  • why businesses are not price sensitive and how to price SaaS directed at them
  • how bootstrapping product businesses with a side of consulting worked out
  • the psychology of happiness

Download Links

Podcast link (MP3, 23 MB, approximately 80 minutes.)

Subscribe in iTunes &tc: The feed http://www.kalzumeus.com/feed/atom/ technically includes all posts on this blog, but if you put it into iTunes or your iDevice, it will slurp in only the audio posts. (Have a more finnicky client than iTunes? Try http://www.kalzumeus.com/category/podcasts/feed/ instead.)

Transcript

Patrick: Okey-dokey. You guys want to get started on the formal talking to people aside from the three of us thing?

Amy: Yeah.

Keith: Alright. Sounds good. OK, so we do the intro music [mimics intro music]. We don’t have intro music.

Patrick: We don’t have intro music.

[laughter]

Patrick: This is a third-rate podcast.

Keith: Welcome back to the Kalzumeus podcast and…

Patrick: This is episode two with…

Keith: Episode two, well, 2.5, because we actually recorded an episode two and then trashed it because it sucked.

Patrick: Yeah. This is uh…

Keith: Two alpha? Two Beta.

Patrick: Two Beta. It was an MVP of a podcast and then we shot it in the head because it was not accomplishing customer goals or anything.

Keith: Exactly.

Patrick: And we are joined today by special guest Amy Hoy.

Keith: Hello Amy.

Amy: Hello.

Keith: For our people, you want to do the introduction, Patrick?

Patrick: Oh, introduction, yeah. I’m Patrick McKenzie, better known as Pattio11 on the Internet and…

Keith: I’m Keith Perhac, not at all known on the Internet. Joining us today is Amy Hoy, who is the founder of Freckle and the new product 30×500, which seems awesome. We’re going to have Amy talk about that a little more coming up. Amy, do you have anything you want to add?

Amy: There are a couple more products other than those. [laughs]

Keith: Those are the big one’s that I know of, that I’m most familiar with. And, of course, your blog Unicorn Free, which is freaking awesome, by the way.

Amy: Thanks.

Keith: I was actually… I listened to the 5×5 podcast where you talked about where you came up with the name for Unicorn Free, and were explaining all that. Ever since then I’ve really enjoyed that.

Amy: Oh, well thank you. Yeah. I figure like, I was drunk and I wrote a note that I forgot and there was a Narwhal horn involved. It was a pretty good story.

[laughter]

Patrick: Ba dum bum.

Keith: Ba dum bum. Alright.

Patrick: So, let’s see. So, the 30×500 classes opening up in the near future [Patrick notes: We didn't get this episode out on time, so if you want into 30x500, you'll have to do it the next time Amy opens up the course.] and I feel it’s probably of interest to some people that are listening to this so why don’t you give us the rundown on that for the folks listening at home who don’t know what it is?

Amy: I’d love to, if I can pronounce my own product’s name. Sidebar, don’t name your product with numbers in it. I never can say 30×500 properly unless I do it like that in slow mo. In 2008 I got absolutely sick and tired of consulting, even though our consulting business was doing really well, mine and my husband’s. I knew that I didn’t want to be a consultant forever, I just fell into it because I had these skills.

I knew the way to make money was product. I had been watching 37 Signals from their rise back when they had zero products and were just designers and then they had products and all this stuff and all these people I knew who were starting businesses and were making money not by the hour but by the product.

I pushed my husband to help me make a software service, which is Freckle, which is nearly three years old now. We also wrote, after that, an e-book on JavaScript performance and from there on we built our product empire and weaned away from consulting entirely. During this process I felt entirely alone. There was almost no discussion about this stuff.

I knew what I knew by reading business books, which were not tailored to me, a two person company who was starting on the side. They were tailored to larger business but I was able to extrapolate the advice and apply it to myself. I read startup stuff, most of which was totally useless for what I was doing.

I wasn’t trying to get millions of users, I couldn’t spend money to acquire users, I couldn’t use venture capital, I couldn’t hire the best people. My husband and I were pretty great, but it was pretty much us and we were free.

Keith: You were pretty much stuck between the two worlds, weren’t you? What you wanted was a standard business model, but there weren’t any real books or information on standard business models in the Internet age and what information there was, was for startups and startups generally assume venture capital, which of course you weren’t going for, which is a great role to go down.

Just because you’re a startup does not mean you need venture capital and does not mean you need to do the seven billion users in two months for a free medium app, whatever.

Amy: But the hockey stick. The hockey stick. How do you survive without the hockey stick? I don’t know. In fact, I was anti-venture capital, for me. I feel like a lot of venture capital is quite deceptive, but I don’t begrudge anyone for taking it if that’s what they want to do. I’m like go after the dealers, not the users. I really didn’t want anyone else in-between me and my customers and my success.

I didn’t want anyone fiddling with my stuff. Until 2008, I had not been able to ship a project I worked on for years because management kept screwing with it, whether it was Behr Sterns, well they went out of business, or Limewire, where my CEO was a 12-year-old multimillionaire boy scout on crack. I never did anything for years and I was dying. I was like no more intermediation. No more between me and them. I was like no venture capital.

You’re right, there was nothing really to rely on other than old school business books and what I had detected by working around and following 37 Signals and Mail Chimp and these other businesses. I observed them over the years, lurked. They weren’t writing about it at that time. Even 37 Signals was not writing about their Bootstraps and Proud series. That was not in 2008, that was a lot later. So it was really lonely.

Patrick: Yeah. I start Bingo Card Creator in 2006 and, man, the state of the art back then. I was literally unsure it was legal to actually just setup a shingle on the Internet and start selling software. I tried to get advice from people in our community, which has some issues about things like that. They’re like no, God man. What if someone does a refund? You can get sued. You should never, ever, ever try to get away from the day job and, besides, they own your soul.

Keith: Going back to that a little, as much as we are three people on the Internet giving advice, you should never listen to people giving advice on the Internet.

[laughter]

Amy: Yeah, just us.

Keith: Yeah, just the three of us. I think that’s a very safe assumption. I would preface the “don’t take advice from people” with “random people on the Internet.” Find people who are successful and who are practicing what they preach, essentially.

Patrick: I think that’s life advice as to hanging around with the kind of people you want to be with rather than anybody else.

Keith: Rather than people who scream the loudest, yeah.

Patrick: This goes to picking your community, both in real life and in the Internet. If you routinely hang around people who run businesses, your perception of the world is going to get influenced by people who run businesses, and if you’re routinely hanging around people who will perpetually have a business some day you have a risk of that warping your perspective on things.

Amy: Absolutely. My rule of thumb is also that even if someone has what you want and people ask, “What’s the secret of your success?” and they say, “Hard work,” stop listening. Let’s face it, it’s a lot more than that and if they don’t have the insight or the willingness to share beyond the phrase “hard work” then there’s nothing you can learn from them from what they explicitly try to teach you. You can observe and be the learner and analyze.

Yeah, it really sucked and there were a lot of naysayers. Not maybe quite as serious as Patrick: did because we’re a couple of years later. Honestly, I really don’t listen to people.

Keith: Very good advice. Probably the best advice we’ll get out of this podcast is don’t listen to people.

Amy: Don’t listen to people. [laughs] People are like no, no, no, no, no. I’m like whatever. We started making a product business and it sort of was harder than I thought. A lot of it was easier than I thought, but it was all hard because we were alone. We were doing this on the side with consulting and we had these short bursts with super intense consulting contracts, mostly for Fortune 500 companies or Fortune 50 companies.

Thomas did a lot of JavaScript security and JavaScript performance. We did a lot of these media art installations based around Twitter and other APIs, Foursquare and stuff, for like Pepsi, who was awesome, and all the other people we worked for who weren’t awesome. So these intense projects that paid a lot but they demanded a lot of us.

So we were doing that and trying to build this product (Freckle) and then ship this product and then support the product after we shipped it and add new features and stuff. And it was really chaotic for about a year and a half.

And at the end of 2009, I knew that I could not do that anymore. That’d been nearly a year and a half, 18 months, 24 months almost, of doing this. And I was like, “I cannot consult and do this other stuff anymore. The product’s suffering. I’m suffering. I have to quit. I’m going to quit now.”

We just got this big check from this two-week project that turned into two months of hell. Then I quit, and I thought, “What can I do to help shore up this income?” And I thought, “I know. I can help people who are like me two years ago.” I wasn’t sure if people were into it or not, so I did a three-hour teleconference and invited some people. (Patrick notes: This idea is genius. It’s the MVP of a more involved product.) They paid me, not a lot.

And the number-one feedback was “more,” which I wrote down in my notebook as “M-O-A-R-R-R…” I counted the R’s the other day. I was like, wow. [laughs]

Keith: Just love writing those R’s, huh? [laughs]

Amy: I did. Everyone was like, “I want more.” That was the number-one feedback. So I was really excited, I guess, so I put extra R’s.

And that slowly turned into this three-and-a-half-month-long class that it is today, 30×500. I built the first one with my friend Alex Hillman, who’s awesome, who’s bootstrapped a community and a physical office space, for co-working and a whole bunch of other stuff, in Philadelphia. He’s amazing. He’s been a business consultant as well as stuff, and he helped me with the first version, and then he couldn’t do it anymore, time-wise.

And I flipped all the stuff around and created a community to go along with the class. I say, “created a community.” I basically meant open the mailing list and invited people to talk on it. The students have really created the community.

And so now I have this three-and-a-half-long mentorship program, which includes a lot of kind of mind-bending lessons an exercises that will help somebody get away from what I call “idea quicksand,” where you have a fantastic idea and then you either see someone else already did it or you get depressed.

You’re like, “There’s competitors. Is the market saturated? Can I validate this idea? What if I can’t validate this idea? What if I can’t build the whole thing and it won’t be like I thought it was?” Basically, everyone goes… [makes bomb-dropping sound] It ends in death, and never shipping anything, or shipping things no one wants. And so the very first thing we do in 30×500 is turn that around. I say, “No ideas,” [laughs] for the first month almost.

It’s all about learning about business, about how to come up with an infinite number of ideas, valid ideas which are pre-validated. You don’t have to have the idea then validate it. It’s all about turning the process on its head, coming up with as many potential profitable products as possible, learning to do market research, learning to really sell those ideas to people before you build it, and all sorts of good stuff: marketing, productivity, how to trim it down to a tiny thing you can ship, all that stuff.

The class is all about that, and it’s pretty intense. But we’ve had an amazing wave of launches lately, and I’m so happy.

Patrick: Yeah. People will not be surprised, based on my karma score, but I get most of my news through Hacker News and saw multiple of them on the front page in the last couple of days. Let’s see, there was, oh, shoot, projector, the one that does designer stuff and client stuff. (Patrick notes: I was thinking of PlanScope). I’m sorry, I can never…

Amy: It’s like project management and budget and scope management for freelancers and agencies to use with their clients. The client sees where their money is going, which is a huge problem, as you may know.

Patrick: Right. Yeah. I think you’ve mentioned this before, but there’s… so, a piece of received wisdom in the community, which, I will be totally honest, I have said this many times, is, “Don’t make things for people who are like you, because developers/designers, et cetera, don’t pay for software.”

I think you have an opinion on this as somebody who successfully sells time-tracking management and who just had multiple customers, or multiple, I guess, students from the program launch straight into the designer community and totally kill it. But what’s your opinion on that?

Amy: It’s not true.

[popping sound]

Keith: [laughs]

Amy: My water bottle. That was dramatic! That was my water bottle going, “Pop!”

[laughter]

Amy: It’s simply not true, pop, which is a polite way to put it.

[laughter]

Amy: I mean, if you look around at all these companies that are making so much money off the developer and designer audience like, GitHub, linda.com, Basecamp started exclusively with developers and designers. That was their market. They marketed to people like them, the small agencies and the individuals, and it grew out from there. Massively, it did. But they started with that core audience. That’s how I knew about it. I found out about it on an invite-only design community way back in 2005.

I usually have a whole list. PeepCode, look up PeepCode. Jeff’s does very well. Let’s see, Rueben from Bidsketch. It’s a total one-man operation. He just dramatically increased his prices, and he’s doing fantastically. Bidsketch is all about preparing beautiful, templatized proposals for your clients. There’s so much, FreshBooks, Harvest, all these other time trackers which make way more money than I do. There’s a lot. A lot.

And not all of those are exclusively designer/developer anymore, but a lot of them started that way and they branched out as they grew. But there are a lot of developer-only ones as well. Look at all these apps for server monitoring (Patrick notes: use it, love it) or Rails Screencasts from many different people, not just Jeff. There’s just tons, and tons, and tons of stuff.

So, I don’t know where the idea that they don’t buy comes from because there are products everywhere that are successful.

Patrick: I think it’s partly a projection thing, like, “I don’t buy anything and therefore people like me must not buy stuff.” Which, there are many issues with projecting your behavior onto other people.

Keith: And really, I think there’s also a… so, this is not just the Hacker News crowd, this is not just the Slashdot crowd, this is not just the techie crowd, there are a lot of people. I think the naysayers are the people who have more time than money, is honestly what it comes down to.

Because, honestly, if I had a ton of time, if I was working a nine-to-five job, had a set number of hours a day I worked at a fixed income, at that, and I needed time-tracking software, I would probably write my own on the weekend because I have more time than I have money at that point.

For someone who’s trying to run or start their own business, they suddenly have more money than they have time. Not that they’re making tons of money but because their time is much more valuable because there are so many other things they could be doing.

Patrick: and I have actually talked about this because I need to write my own invoicing software and stuff like that. I finally did not because I thought I could set up an MVP in about a week and take another week to fix any bugs so that’s two weeks of my time that it would take to build just my invoicing software. Or I could pay $20 to $50 a month for someone else’s invoicing software. That’s a no-brainer. Two weeks worth of billable work versus $50 a month. No-brainer at all.

Amy: Absolutely, but how long did you think about buying the podcasting equipment?

Keith: Actually, we just kind of fell into that.

Amy: A lot of people say they don’t buy stuff. They actually buy stuff left and right, they just weren’t paying attention. Not that you bought it mindlessly while you were sleep walking or anything. When you think back in your memory you think when did I buy things? It just doesn’t pop up.

Keith: I’m gradually getting better about it. The podcasting equipment, I was like we need podcasting equipment. OK, done. A couple years ago I started Bingo Card Creator on a $60 budget and when you only have 60 in the budget you get very creative about not spending money, but these days the budget is much more than $60 and I have to sometimes slap myself and say no, implementing this myself is absolutely not the right call.

I was talking to a buddy of mine and asked, “Is there any way I can optimize Redis such that it will use 15 megabytes less of RAM on the server? Then I won’t have to upgrade to the next higher tier of VPS.” He said, “What’s the next higher tier of VPS?” “$20.” “Do we need to have the rest of this conversation?” “OK, OK. I get it, I get it.” Gradually, very gradually I’m starting to get it.

Amy: It takes time. Most people aren’t like you, even developers. That’s fairly unusual. Most of us, especially Americans, we just tend to throw money at stuff.

Keith: I think we’re very much conditioned by living in Japan. It’s weird. Once you get to certain price points, like low price points for some reason people hee and haw over much more than they would decide over something of a large price point. If you’re spending $1,000 it’s easy to spend another $100. If you’re spending $10 or $15 people seem to think about it a lot more.

Amy: It’s true.

Keith: The old president of my company, multi, multi-millionaire (Patrick notes: he is credited with bringing the Internet to Japan, you do the math), he does a lot of donations and stuff to colleges and stuff and he had done this multi-million dollar donation to a college and he had just finished signing the contract and he’s leaving and he goes to a convenience store and picks up a bottle of water and he just goes, “I can’t believe water at a convenience store is 135 yen. That’s just so expensive.”

I’m like you just contributed millions of dollars and you’re complaining about $1.35.

Amy: I think we all fall prey to that one. The other day I was in a convenience store and I realized I needed a toothbrush and I bought a toothbrush and I was like why is this toothbrush $1.75? I’ve been paying like 3.50 each. I caught myself thinking I’m going to buy my toothbrushes there from now on. I was like wait a minute.

Keith: You actually consider should I go to a different store to get the cheaper toothbrush?

Amy: It only passed a moment, I have to say, to my credit, but it occurred to me. I was like come on, Amy. I’ve occasionally thought I should have kept the exact amount, 20 percent or whatever, and I’m like come on, really? Am I even wasting a cycle of my brain on 50 cents?

I’m not like that with buying software tools for business at all, so I think what you said earlier about selling to businesses is totally true, but I think there’s a lot more people who are just not commenting on things who just quietly buy the things they want or need whether or not they have a business.

Keith: This might not be a new thing for the 30×500, but you’re starting to focus more on building apps and things for businesses than for B2C stuff which, as somebody who knows and loves many B2C customers for his Bingo product, is totally the right way to go.

Amy: I like how you used love in a negative way. [laughs]

Patrick: I do love my customers.

Amy: I understand.

Patrick: I love them even when they write I can’t access your product from the blue Googles, only from the green Googles, can you please help me out? That’s still from a place of love.

It’s just from a place of I can do my math on what my hourly is on that versus an appointment reminder where I get to charge a car dealership every month. I’ve had car dealerships say is it 200 a day or 200 a month? I’m like it’s 200 a month and they’re like “Whatever. Either would have worked.”

Amy: That’s nice.

Patrick: In other news, I’m re-pricing.

Keith: The correct response to that is not $200 a month. If they ask is it 200 a day or 200 a month you say a day. If they say that’s too expensive you say then a week. [laughs] Start with the expensive one first.

Amy: You’ve got to capture that customer surplus. You want that.

Patrick: We were thinking about talking about pricing grids. One of the things that you can actually learn if you spend too much time hanging out on the Internet and talking to people about this is that a lot of SaaS companies use the four column pricing grid strategy typically.

I’ve talked to a lot of folks about this one. The one that really prints the cash, usually about 50% of sales, is the one to the extreme right that’s priced at businesses at prices that people think no one can will pay. Say, $250 a month for Wufoo.

It’s just that people who are spending essentially other people’s money, it just comes out of a budget so it doesn’t matter to them whether it’s 200 or 250 or 500. As long as it still only requires one signature or zero signatures it’s whatever it is.

Amy: Yeah. I hear the amount per signatures for an employee generating expenses in a large company is usually around $500. I think under $500 it does not require a signature.

Patrick: That’s consistent with my experience. Anybody who’s doing a SaaS pricing grid where the top price tops out at $20 or $30 should really…

Keith: Rethink what they’re doing.

Patrick: …put anything you need to just get an enterprise level, even if you don’t necessarily call it the enterprise level, and price it at $250 or $500. This is free money. And, oh, goodness…

Keith: So, not even looking at this as a huge business. When people think of business, they think of huge corporations. But even for small companies that are making a good amount of money, let’s say that a company has maybe six people working there that are contractors.

And a good contractor will run you about 100 to 200 an hour, depending on what they’re doing. If a product on the Internet costs you the same as one hour of that person’s time and saves them over an hour a month, then it’s a no-brainer to get that, right?

Amy: And that is exactly why, actually… Patrick, you asked this earlier. I’ve always told people, in 30×500 and “Year of Hustle”: do not sell to consumers. And some people will say, “But I have this idea.” I’m like, “No. You can do it, but I’m not going to support you because you’re going exactly against what I told you to do.” For that reason. For that reason.

You can sell on value to businesspeople. You can say, “You spend $80 an hour on this freelancer. I can save you 45 minutes of their time and charge you $60, and that is a win.” It’s certainly not a loss. I think I did the math wrong; I think that’s exactly $60. But let’s say half an hour of their time for 50 bucks or whatever. No, that’s still wrong. [laughs]

Patrick: We get the general idea.

Amy: You know what I’m saying. [laughs]

Keith: So there is really two places that you can really provide enumerable benefits to your customers. One is saving time. Because a consultant takes time. The amount of money that you save is directly related to how much you can charge, right? The other one is anything that increases sales.

Amy: Totally.

Keith: For example, Visual Website Optimizer prints money for customers. Anyone who is using Visual Website Optimizer is literally printing money with every single test that they do.

Patrick: This is the A-B testing software we often recommend to clients.

Keith: It is so amazing. So their largest for enterprise is “call us.” That’s fine. Their second-largest is $250 a month. OK? $250 a month for increases of sales starting at two percent, five percent, 10 percent, 50 percent. As good as your test can be, that’s how much money you are making with their $250-a-month service. It’s amazing.

Amy: It is amazing. They could probably charge more for that.

Keith: They could.

Amy: Ruben Gomez, who does Bidsketch, I mentioned earlier, he tweeted repeatedly and told me personally how much more money he was making when he drastically increased his prices. And I’ve been nagging him to write a blog post. So I’m going to keep nagging him until it happens, but his story is pretty incredible.

I’m not going to release the numbers because he hasn’t done it yet. You would think that he’s looking at private bids instead of people, the freelancers and consultants. And it worked out so well for him, so well. It’s such a big deal.

Patrick: I think he’s coming to MicroConf. We’re going to lock him in a hotel room in Las Vegas and not let him out until the blog post is written. (Patrick notes: Did not actually happen at MicroConf.)

Amy: Yes. Yes!

Keith: [laughs]

Amy: I vote yes. Let’s do that. [laughs] He’s a cool guy. I like him a lot. So yeah, you were saying more sales, or saving time. And I also think of, people usually go for cost reduction, I think, when they talk about monetary value. But I don’t see nearly as many products being successful for reducing costs, unless it’s extreme, because penny-pinchers aren’t people who spend money.

Patrick: One thing that’s great for software is if you can tell a story. Ultimately, all sales is about telling stories and painting the right picture in peoples’ minds, but tell a story where it reduces the amount of employee labor required to do something, particularly if that either allows them to switch them to tasks that actually generate money or, I hate sounding like a business, but “decreasing headcount.”

If you can successfully pitch to a business that you are going to “decrease headcount,” that is a total win in 99.95 percent of cases. So, Appointment Reminder, my software that does phone calls to the clients of professional-services businesses. I often say that if you have an office manager who costs you $4,000 month, who half of her time is literally talking to people’s voicemail to attempt to get them into the office at the proper time, then spend $200 a month and save $2,000 on salary costs.

One of my more successful clients is saying that, basically, I saved him enough on that to put his daughter through Harvard.

Amy: So you need to raise your prices. There’s that customer surplus.

Patrick: I need to raise my prices. Yeah, I so do. I did something very stupid when I launched Appointment Reminder, and I’ll just tell it to everybody to have you avoid doing it. I launched with the four-tier pricing structure, like usual, and the bottom plan was $9 for a, quote-unquote, “personal plan.”

So my idea was, “I don’t really care about the $9. I just want people using this.” I should’ve wanted people using it at 30 bucks a month for the cheap plan, because the people who pay $9 are, my word is “pathological customer,” for people who are penny-pinching, and they have every kind of support issue that you could possibly imagine, like, “How do I record telephone calls if I don’t have a telephone? Can I log into the website from my Kindle, which doesn’t really have a web browser in it?” Yadda-yadda-dee-da-da.

And they expect turnaround times of two minutes or less to customer-support issues arising at 3:00 AM in the morning.

Keith: I would really like to see someone. I don’t know if anyone has done a blog post about this, about a breakdown of the number of support calls and support messages you have, broken down by which plan they’re in.

Patrick: I will totally bet that it is the cheap-o Charlies who contribute a vastly, vastly disproportionate…

Keith: Like 80 percent…

Amy: Me too. Heard that over and over again from everybody. Also, this is something that you could absolutely do with our upcoming software-as-a-service product, Charm, which is a customer-support and true customer-relationship management tool. Everyone says CRM, they mean lead tracker, which I find to be terribly dishonest. [laughs] It’s like, one, they’re not customers yet, they’re leads.

And then after they are customers, doesn’t help you at all. The only exception I’ve seen is Intercom, which is pretty neat. But they don’t call themselves a CRM, I don’t think. But Charm will let you filter requests by plans or price points.

And so you’ll be able to profile feature requests and issues, specific ones, like, “Please add invoicing feature,” that kind of thing, by which plan or how much your customers are paying you. But also, you will be able to see how many incidents you get from which type of customer. But everyone I’ve…

Patrick: That would be great blog-post fodder if you can get anonymized data for that. Well, it’s not a great idea, but yeah.

Keith: No. I mean, that’s easy to get anonymized data from. You say, “My support numbers are X, and they belong to the lowest tier.” You can use even general numbers for that, I would think.

Amy: Do you mean if you blog about it, Patrick, anonymized?

Patrick: Well. So, I’ve already retracted this idea, but the idea I have now retracted was, oh, you could aggregate across all Charm customers, whether it was the cheap-o plan or whatever that generated the most customer support inquiries. I’m like, “No, that’s a wee bit aggressive.” [laughs]

Amy: Do you think that’s like a cats.jpg moment? Because I don’t really think so. I don’t think it’s a cats.jpg moment.

Patrick: The rational part of me thinks that it’s not a cats.jpg moment, but I think that loud people will perceive it as a cats.jpg moment. Really, this is inside baseball here, so let me tell everyone what a cats.jpg is.

Keith: First of all, yeah, I have no idea… [laughs]

Amy: Sorry. [laughs]

Patrick: So, 37signals said, “Oh, the 100 millionth file has just been uploaded to Basecamp, and it was called cats.jpg.” And they tweeted that out or put it in a blog post or something. And the folks who were worried about the Facebookization of all services were like, “Oh my God! 37signals can view all this data that we’re uploading to their servers, and they’re not treating it in a privacy-conscious manner! Brar!”

Amy: [laughs]

Patrick: So, OK, yes, A, it is totally technically possible for people to view data that you upload to your servers. That’s kind of how it works. And if you don’t trust them on that, you definitely should not be using Basecamp. But it was kind of a tempest-in-a-teapot kind of thing about whether it’s OK to publish that even if it’s a trivial amount of customer data. No one’s business is going to collapse over the words “cats.jpg” getting out, where if it was “letter of intent to dismiss Mary Smith for sexual misconduct.doc…”

Amy: [laughs]

Keith: They might’ve anonymized that. [laughs]

Patrick: Right.

Amy: I don’t think they would’ve put that up there. I think people were more upset over the idea that they were looking at individuals’ accounts. But there’s a lot of apps out there which say how many bookmarks or how many dollars have been invoiced or how many hours have been tracked. I’ve never seen anyone ever complain.

Keith: Complain about that, right.

Amy: I think FreshBooks and Hunch, they all do these infographic-style breakdowns of the data. But it’s totally anonymized, like you said, so it’s totally in aggregate. I can’t imagine. Well, you know what? I’m going to do it with Freckle anyway, so we’ll find out. [laughs]

Keith: The noisy people, the people who are complaining, I think, about the cats.jpg, I mean, aggre-data… [sputters] Aggregate data.

Amy: Aggre-data.

Keith: Aggre-data. There we go, aggre-data.

Amy: That’s great.

Keith: Aggre-data is brought from individual data, right? So if you have source to create the aggregate data, you have the original source data. So there’s really no difference in the privacy, right? It’s not like they purposely were looking at anyone’s single Basecamp to find cats.jpg. They just did, “Query, item number one million. What is name of that item?” Right? I don’t know, like Patrick said, tempest in a teapot.

Amy: How different would it feel if I wrote a blog post on Freckle, which is a time-tracking productivity tool, that said that 30 percent of all hours logged yesterday were overhead hours that are non-billable, versus I said the 100 millionth hour was logged to a project called “Cat.” You know what I’m saying? I don’t think people would care.

Keith: They didn’t even mention the project name, right?

Amy: No…

Keith: See, I think it would be different if you said that. But if you said, “The millionth task that was logged was overhead,” I don’t know how interesting that is. [laughs] See, me personally, that’s perfectly fine.

Patrick: See, this is the reason why it’s a tempest in a teapot. The only reason that anecdote was put into the post anyhow is because it’s harmless and silly and trivial.

Amy: Hilarious. [laughs]

Patrick: And if the 100 millionth item had been a business document, it just wouldn’t have been mentioned, because, A, privacy issues, but B, it isn’t funny. But because it’s stupid cat photos, it’s funny. And, brar, tempest in teapot. I avoided commenting on those threads because I thought commenting would make me dumber.

Amy: [laughs]

Keith: [laughing] We’re doing a whole podcast about it.

Amy: It’s true. I’m sorry I brought it up. [laughs]

Patrick: I feel myself getting more stupid with every sentence that comes out of my mouth.

Amy: Oh, no! I killed Patrick: McKenzie’s brain cells.

Patrick: What were our value-creating topics we were going to talk about…?

Keith: OK, so value-creating topics. Number one was the cat picture that Basecamp…

Patrick: No, that was not on the list.

Keith: Oh, that was not on the list. OK. [laughs]

Patrick: We were going to talk about…so, Amy, your business trajectory has been from one where you were consulting and not really loving it, to put it mildly. Now you’re 100 percent on the products. I started with the product/day job and got quit of the day job as of two years ago this week, which was the second-best decision ever.

Amy: Happy anniversary!

Patrick: Thank you.

Keith: Oh yeah, that’s right.

Patrick: But I kind of got sucked into consulting, starting about the same time I quit the day job, because people threw motivation on my company at me. And it was just hard to say no to the checks. And they generally come from people who are not Fortune 500 companies and have a little less BS associated with it, like the minimum BS that you can possibly have while still taking money from other people, I think.

So I’m pretty happy with that. But in the future, I would love to transition back into 100 percent product. And Keith is kind of at the end of the totem pole. Keith also quit. Like we talked about on the podcast last time, he quit his crazy Japanese day job and now works for consulting clients who are much better. But he also nurses dreams of having a…

Keith: A product and actually creating something of my own, right?

Amy: Yeah. That’s an awesome feeling.

Keith: It is.

Amy: I can tell you, from over here, it’s great.

[laughter]

Amy: Why didn’t I do this three years earlier? Keep at it. It’s worth it.

Patrick: So people have told me that they’re actually interested in how the lifestyle works out. Everyone grows up knowing lots of examples of people who work day jobs, and everyone kind of knows, “OK, you work for about 40 hours a week. You go commute to an office.” You know what the packaged lifestyle deal of working for a day job is, whereas they don’t know what it is to run a product. So, can we just spill the beans and say it’s F’ing awesome all the time?

Keith: [laughs]

Amy: Yeah. [laughs]

Keith: Except for customer support.

Patrick: Even for customer support…

Amy: Whenever I have to touch any other institution, like government or healthcare or banking, I kind of want to kill myself.

Patrick: Yeah, that’s true.

Amy: But it’s no worse than working with marketing people, [laughs] which is what I was doing as a consultant.

Patrick: And we never have to talk to an HR department, which is worth its weight in gold.

Amy: I haven’t worked at company big enough to do that, so I’ve avoided a special kind of hell. I feel very lucky for that.

Keith: [laughs]

Patrick: One of the things that I’m really appreciating this year is I’m getting married in June — yay — and pretty much taking off. I just told my consulting clients that, basically, either get themselves in by the end of May or it’s not happening till September, and just took off the entire stretch in there and will just not be working.

Amy: That’s awesome.

Patrick: I get a lot of people asking me, “How can you do that? The servers are going to burn down in a fiery badness.”

Amy: [laughs]

Patrick: Just verify for me that I’m not insane here. That’s not really how things work, right? These businesses…

Amy: Of course it’s how it works. The moment you turn your head, everything explodes in a fireball, then Godzilla comes out of the ocean. [laughs] Come on.

Keith: This is half-true, especially with Patrick’s track record.

Patrick: That’s not actually true.

Keith: OK, let me put it this way. [laughs]

Amy: Uh-oh. Is there dirt here? Is there dirt to dish? Do we get to dish dirt? [laughs]

Keith: No, no, there’s no dirt. He’s actually blogged about this. Whenever he is fully available, he generally has no support costs on his products, right? I think like, what, an hour of support a day or something like that?

Patrick: Way less than that, dude.

Keith: Way less than that. OK. Maybe 10 minutes…

Patrick: 20 minutes a week.

Keith: 20 minutes a week. OK. Anytime he gets on an airplane, or anything where he has no Internet connectivity, the server goes down. [laughs]

Patrick: This is not actually true. It just happened…

Keith: One out of 10 times. [laughs]

Amy: Just seems like it.

Patrick: It happened when I was doing an intercontinental flight back at Halloween, which is unfortunately the busy season for bingo cards.

Keith: You had another one when you were moving. You were moving and didn’t have phone access for one day, and the server crashed.

Patrick: Oh, God. [laughs] So this is two events in six years.

Keith: [laughs]

Patrick: The key that we were trying to emphasize to impressionable youngsters who are listening to this podcast is that you can step away from the business and it will not consume your life.

Keith: You can.

Amy: Absolutely.

Patrick: People will happily pay you money, even if you’re not working on the product every day, because people don’t care if you’re working on the product every day, they only care what they’re getting out of it.

Keith: The point I’m trying to make is, and what you say is true. 20 hours a week, you can go off and do what you want. People don’t care if you are working eight hours a day on your product, and you really shouldn’t be after you’ve launched to any certain degree. But keep your phone on. [laughs]

Amy: I’m sorry, I couldn’t parse that sentence. You can only not work 20 hours a week and people don’t care? I’m confused. [laughs]

Keith: Sorry, sorry. Did I really say that?

Amy: Yes.

Patrick: The English…

Keith: English? OK. So, as I’m sure everyone knows, we’ve been in Japan way, way, way too long.

Amy: OK.

Keith: So your customers do not care that you’re only working 20 minutes a week, or they don’t expect you to be working eight hours a day, because as long as the service works, they don’t care.

Amy: That’s right.

Keith: And you should not be working that much once your product is launched. However, you should always have your cell phone or some sort of Internet connection on in case things do explode, or someone to watch it for you.

Amy: So, we just took a month off in New Zealand. And then we came back for a week and a half. We’re doing city hopping in the US, San Francisco and Atlanta. And I actually did do email every two to four days, because I wanted to keep up with my class. We had somebody handling front-line support for the two apps.

We did have a server problem with Charm, but we haven’t launched that product publicly yet because we’re still ironing out those infrastructure kinks, right? And so I think my husband actually worked like two hours the entire trip, because he doesn’t have his class that he’s running. And nothing bad happened.

So here’s the thing, right? When you have a lot of customers, something bad can happen, and you can lose a few and you can be like, “You know what? I lost $200 a month of business and I took a month off. Who cares? Who cares?” And you can just gain those back. You come back, you’re like, “I’ll get new customers.” It’s not a big deal.

Someone will always cancel for some reason. It doesn’t really matter. In Freckle, we’ve gone down quite a few times. But it’s a product where you’re not in it all day, and something goes down once in a while. People don’t even get mad as long as you try to get back on and apologize. If it happens in the middle of the night, so be it. I’m not getting up in the middle of the night. No way.

Keith: And this is one thing. I think a lot of people on the Internet think that there is a limit to the number of customers you can have. They always talk about market shares and stuff like that. And talking about market shares when you’re going after big companies or products that need millions and millions of users is one thing.

So Bingo Card Creator is a very good one, because people always say, “How much of a market is there for teachers that need bingo cards?” Right? And there’s, compared to the number of programmers in the world, probably not many. But there are a f-ton, right?

Patrick: “More than I could ever hope to get to my website” is the short version.

Keith: If you were to even get one percent, you would never have to work again.

Patrick: I hate the one-percent math…

Amy: Oh, me too.

Patrick: Just as a comparable for folks, Bingo Card Creator, which is almost like the canonical example of, “Oh, God, that was a poor choice in niche selection, Patrick. Why did you do that?” has over 200,000 users and 6,000 paying customers. So if you think your thing for programmers is going to be more niche than that, you probably need to recalibrate expectations.

Amy: You’re probably wrong, also.

Patrick: And if I only had recurring revenue. That’s another thing.

Keith: [laughs]

Patrick: Recurring revenue, man, that’s the best kind of revenue, isn’t it?

Amy: It is crack, in a good way. It’s crack that doesn’t make you sick. [laughs]

Keith: [laughs]

Amy: And it’s legal and stuff. And you don’t have to inject it. I don’t know. How do you take crack? Stop me now… [laughs]

Keith: [laughs] Our street cred is going down the toilet right now. [laughs]

Patrick: So, definitely, if you have the opportunity to make a SaaS business, do the monthly charge thing that all the cool kids are doing, because it does wonderful, wonderful things for your cash flow. It helps you absorb advertising costs better. It will allows you to have high customer lifetime values without your customers perceiving the service as being expensive at all.

Amy: It’s true, that is a very good point! Recurring revenue is the Holy Grail and I love it. And back to your market share comment, recently, I mean it seems recently but it was like six or eight weeks ago, people were like…someone on Hacker News like, ‘Do people still pay for porn or these other things?’

And you were trying to be like, ‘I don’t know about that but I know people pay for a lot of these other tools, among which what you deem time tracking.’ And then the same or different person, very skeptical is like, ‘People don’t pay for time tracking!’

Keith: All people pay for time tracking! [laughs]

Amy: That’s how I read Hacker News, by the way. And then mentioned me and that’s just one example. In any industry there can be one example which makes money and I had to try them and then go, ‘You know what? There’s at least six to eight companies which make geometrically more money than I do!’ [laughs]

Keith: [laughs]

Amy: And then it went silent, surprisingly. [laughs]

Keith: [laughs] Always does!

Amy: I think a lot of people don’t, they don’t have any clue but they think that they do, about market share. What I hear a lot is, ‘Oh, but that market is saturated.’ You don’t even know what that means. That’s not what you think it means. Saturated means people don’t buy stuff anymore but they do.

If you have a pool that is very popular, has a lot of customers, there’s got to be a significant portion of those customers who are being ill served by that product.

Keith: Right, right.

Amy: It cannot be all things to all people. So someone like us who just needs a few thousand customers to live like a king, can swoop in and serve a segment of those customers, which were created for you by…

Keith: By someone else.

Amy: …this competitor which is allegedly saturating the market.

Patrick: That’s something I’ve been telling people for a while, it’s that competitors are a wonderful thing because it’s an engraved invitation from God that tells you that there’s money to be made in a particular place.

Amy: Yeah.

Keith: And there are always going to be people using your competitor’s products that are not happy with them that might want to go somewhere else. If you have a feature that other places don’t have, and even if you have a combination of features, so everyone else in this space might have the exact same features but they don’t have them in the same combination, you then have a niche of an already proven market share that want the features that you’re offering.

Patrick: We shouldn’t be the engineers here, either . We start talking about feature, feature, feature but we can honestly take something which is feature equivalent or even at less than the feature parity and just market it in such a way that, you know, it actually worked for people who it isn’t working for right now.

And that would itself justify a different business. Like, you know, there must be 500,000 big freaking enterprise project management/time tracking/Sa* , yada-yada things. Freckle doesn’t have to compete with them because you’re addressing just a different market than the kind of folks who want to buy consulting ware from IBM. So even with just a fraction of the “feature set,” you can just say, “Look, it will do what you need to do and get you back to charging your customers money.” Then that makes it a viable option for them, whereas the IBM consultingware wouldn’t be. Who would you consider to be Freckle’s big competitors?

Amy: Harvest.

Keith: Harvest.

Amy: “No Tool At All” I think is our biggest competitor.

Patrick: That is a big one.

Keith: That is a big one, that is a big one.

Amy: It’s huge! [laughs]

Patrick: Folks ask me how I convince people to stop using whatever their business’ scheduling software is and start using Appointment Reminder because you have to have the appointment schedule to send out the appointments reminders at the right time. And the easiest answer to that is, all you have to do is out-compete paper. It’s not very hard.

Keith: Moves people, especially techies, think that there is a solution out there already that people are using in the space that they don’t understand. And one of the things that I’ve seen with my clients especially is, they don’t have a solution other than Excel and a piece of paper.

Amy: Oh, it’s so true.

Keith: If you can beat down Excel you’re winning.

Patrick: Yeah.

Keith: The sad thing is how many don’t beat out Excel, right? [laughs]

Amy: Be careful about that because a lot of… so, I teach my students a lot of different things, one of which is a list of failure archetypes. Type one failures, failures that cannot be resuscitated by more work and marketing and repositioning and all that stuff.

And one of them is a “Cure for Religion:” trying to solve something that people don’t see as a problem.

Keith: Don’t want to see, right.

Amy: Lots of people love Excel and you will never pry it from their cold, dead fingers. Because they friggin love it. So you can be better than Excel and they’ll be like, ‘I don’t care, I’m not interested. I love Excel.’ And a lot of people cannot be reformed by software! [laughs]

Keith: It’s actually funny. My old company, they were having, not cash flow issues but reporting issues on their invoices and monies received and everything. It was taking so long because they were doing it over seven or eight Excel files and nothing was tied together and the sales guys were not reporting right.

So they commissioned me as an employee to spend a month or two creating an invoicing system that would tie back to all their sales and everything and just make it really easy to use.

I got all the requirements, made it all. I thought it was probably the easiest thing to use ever. Everyone said, ‘Oh, this is so easy to use.’ No one used it. [laughs] Like what they would do …

Amy: I think that was worth where that was going! [laughs]

Keith: Yeah, actually the sales guys really like it. The sales guys would put it the data, copy it into Excel and send it to the accounting firm. [laughs]

Amy: Yeah!

Keith: I mean, the saddest thing ever, to have your software simply be a copy paste solution for Excel. [laughs]

Amy: Yeah, that sounds really terrible.

Patrick: I don’t know if that’s sad or opportunity because I have definitely created things where for, largely not in publicly accessible parts of the product but if people say “the workflow requires X at the end of it”. If that is the issue that’s preventing you from paying me a motivational amount of money every month then wham! There’s a button on your dashboard now that exports CSV files. Go to town!

Although that’s an issue I think we’ve all talked about before. Customers, the things they tell you are they reasons they’re not buying the software are generally not the reasons they’re actually not buying the software!

Amy: They’re usually, yes. I find it is a mistake to listen to people. Not just in like, I don’t take their advice, this is different. I watch they do, so the whole ‘programmers don’t buy things’, I see people saying that, meanwhile they pay for like Apple products and GitHub and PeepCode.

And they say it with a straight face when they say it, “I would sign up for your service if XYZ.” And I’m like, “What would that look like? Why do you need that?” And they come up with something that’s so bizarre. I’m like, “Why don’t you do it this way?” And they’re like, “Oh…”

Because when people ask for features, like a client, most of us who are experienced consultants know that you can’t take anything they say at face value. You’ll be like, “What is your purpose?” They’re like, “I need this animated Flash widget, blah blah blah.”

And then you find out they need something really simple, and they just came up with that because it looked likely and they like to sound like they know what they’re doing. But they don’t. [laughs] It’s our job to figure that out and look at what they actually do.

Keith: Customers, businesses, clients, all of them together, most of them have really no idea how their business runs, I think. Patrick: always says that there’s a key number to any business that directly influences the bottom line of sales. And the number of companies that actually know that key number are few and far between, I think.

Amy: What kind of number are we talking about?

Keith: I did a recent re-jiggering of an online registration service (Patrick notes: more natural English might be “a hotel booking website”), and I did some consulting for them, and they were under the impression that 90 percent of their reservations came from the website instead of phone or walk-ins. And they were under the impression that they were having about a 60-percent, or a really high, conversion rate from people who came into the system.

And once we brought out the actual numbers, they saw that there was only 20 percent actually using the website. And of those 20 percent, only, I think, like nine percent actually completed a reservation on the website. And so it’s not that those numbers were necessarily bad, but they had a completely opposite view of the reality of their business, right?

And they had been doing that for five, six years. If they had noticed that five, six years earlier, they could’ve completely changed their strategy, but instead they were poking along because they were under a misconception.

Amy: Right. That’s a pretty big misconception.

Keith: That’s a pretty big [laughs] misconception, I know.

Patrick: That happens over and over again in my consulting career. I’m lucky I get to work with savvy, intelligent people. I mean, hey, they pay me.

Keith: [laughs]

Patrick: They’re all good companies run by smart people, and yet many of them don’t have the infrastructure in place to tell them material facts about the business that you can’t get just by looking at a screen in Google Analytics. That directly influences decision-making about those material facts.

Amy: Right. We tend not to notice what isn’t there. We just work on whatever’s in front of us. We don’t look for the thing that’s missing.

Patrick: That’s an interesting topic. As one business operator to another, what kind of things do you track for your business?

Amy: So, since we last talked, it hasn’t really changed that much. [laughs] We have a lot more… Actually, that’s not true. We set up KISSmetrics, since we track a lot of things now. But we do not have a very good sales funnel tracking, and that’s because we plan to redo the sales page completely. This is my white whale, perhaps, or some other thing that will never get finished…

[laughter]

Amy: …and I should give up on before I become a horrible novel, or something. Because that’s going to happen. But we track a lot of revenue, we track churn rate, we track feature adoption now. But, I’ll be honest, I haven’t looked at it lately. And by lately, I mean the last three months.

We’ve been totally occupied with other stuff. In fact, we haven’t developed, or even deployed finished features, for Freckle for months because of the international move, all the other drama we had in our personal lives, travel, and Thomas getting his immigration stuff sorted out. It’s kind of like your three month vacation, only we weren’t really having fun but for one month of it.

[laughter]

Patrick: Back to a previous topic, because you charge customers monthly, the revenue went up every month anyhow.

Amy: Yes. It did. It did. It went up no matter what. What’s really awesome is that I, a few years ago, got sick with mono for the second time.

Keith: Oh my God.

Amy: I developed chronic fatigue syndrome, which kind of blew. For a while I was so sick I couldn’t do anything. The best thing I could do in the day was to get up out of bed and go to the sofa and watch stupid TV. I couldn’t even watch smart TV because it felt like I was having an agoraphobic attack in a crowd with all the facts.

Literally, I was averse to facts. I couldn’t cope. It turns out that was low cortisol, believe it or not. I couldn’t make any decisions or do anything at all for three months, work wise. Zip. Thomas manned the support, he talked to the one developer who was doing work for us, and it was fine.

Our business grew even when I was on practically bed rest, and that was a really transformative moment for me. I knew we could take these vacations. I knew we could do this stuff. But that was like a, “Holy shit!” moment. Am I allowed to say that? [laughs]

Keith: Yeah.

Amy: Awesome. It was. I was just like, “Oh my…”

Keith: [laughs] We’ve been cursing like sailors the whole time, so…

Amy: Oh, OK. [laughs]

Patrick: You have, I haven’t.

Keith: Patrick: doesn’t.

Patrick: My half of the podcast is PG, his is PG-13.

Amy: Once I felt better and actually had the cortisol to think about it…

[laughter]

Amy: …it was like a sky has opened up, ray of light, choir of angels singing and throwing cash.

[laughter]

Patrick: I love that image.

Amy: It was the best. I was like, “You know what? I can’t be fired. I cannot be laid off. I do not have to worry about unpaid sick leave. I have it made.” I think that’s one of the big reasons that I’m such a tireless promoter of what I call Bacon Business. Products that bring home the bacon, that make money that you sell directly to people who buy them. Not advertising, not marketplaces, not venture backed, because they can change lives.

To get all philosophical for a moment, it’s epic to be able to live this kind of lifestyle. Isn’t it? It’s amazing. I think not enough people promote it in a way that isn’t like, “Oh, well, they’re just super successful. That’s not standard and I could never live like that.” The examples out there are just too lofty. And then there are people like us.

Patrick: Yeah. I know, so I’ve been hanging around with the small software developer crowd for a while and there’s a lot of businesses that might be like Bingo Card Creator in terms of scope, but maybe up to an order of magnitude and more in terms of revenue, just by doing things that you wouldn’t expect that people could do as a full time thing that they’re doing as a full time thing. They get all the benefits of the lifestyle.

It’s like being a rock star minus the groupies. You never have to show up anywhere at any time. Money just appears magically in the bank account. Seriously, guys. For any of you who are on the fence try it. It’s awesome.

Amy: It is.

Patrick: I’ve got a lot of respect for the Silicon Valley startup types and I’ve kicked around doing that myself a couple of times and have been offered motivational amounts of money to do that. “This is awesome!” is something that you will not hear from lots of the folks over there.

Amy: I wonder why.

Patrick: It’s like being a lawyer or consulting like management consulting. There are people the lifestyle works for and there’s people that the lifestyle just does not work for. I don’t know if I could think of anyone off the top of my head who has started their own software business and went full time at it and was like “no.”

Keith: “I want to go back to my nine to five.”

Patrick: “I really want more challenges in life.” That’s something I hear from a lot of people. “Don’t you feel bored like you don’t have enough challenges?” No, I can spin up challenges any time I want.

Amy: No one has ever said that to me. I think most people assume it’s way harder and more stressful than it is, and I understand why. I actually had a short Twitter conversation with Jason Cohen who I absolutely adore. He writes a fantastic blog, A Smart Bear. Before I say this, I want to say that I just think he’s great. I wanted him to speak at Schnitzelconf, but it was just too far for him to go.

He tweeted why do startup founders beat themselves up? It’s like why do hamsters eat their young? They just have to. I was like no. For starters, I didn’t say this in the Twitter conversation, but I used to breed gerbils and none of them ever ate their young because I took good care of them so I feel like I’m sort of an expert on both parts of this equation. I was like that’s not true.

My gist was that people do it to themselves. He said it was easier to be lenient after you’ve had objective success. I said I wouldn’t call it lenient, I call it self-respect. The truth is it doesn’t get easier after you’ve had objective success. I think a lot of people, they actually are worse to themselves after they’ve had objective success because they feel like they have something important to lose.

I know a few people who run businesses like ours, they may be more involved, some less involved, and they feel like they can’t go on vacation, they feel like they have to answer email in the middle of the night, and they do it to themselves. It’s not external. It’s all internal and I don’t think Jason believes me. [laughs]

Patrick: I’m constrained at how much I can say because he’s one of my wonderful, lovely clients, but I’ve heard that feedback from other people who, again, much like Jason I respect.

It is a psychological thing that this is a meme we really need to kill, but I think there’s a deep seeded… zeitgeist. Is that the right word? People are afraid to allow themselves to be happy and believe that success must require a certain quantum of suffering and if you’re not suffering you’re clearly not on the successful route.

Even people who are clearly by any objective measurement successful. It’s kind of a personality thing, too. Jason is a very hard-charging, type A kind of guy. He’s got a ridiculously successful company right now, he’s sold one successful company previously. If you want to look at somebody who’s got it made, Jason has it made.

There’s no external feature that would necessarily need to make Jason feel the need to beat himself up. Actually, something you said to me was very profound, that if there’s ever an issue between you and another person it’s not about you, it’s about them.

Amy: Absolutely. So true.

Patrick: I think that recompiled part of my source code when I heard it because it was just so f’ing true. I find myself quoting that to people a lot.

People have asked me, “You didn’t answer my email. Was it something I said?” I’m like, “Nope. Just an FYI, any time someone does something it’s probably because something that was just going on in their life because they’re in their life 24 hours a day and they’re in their relationship with you for like 36 seconds a day. Just don’t worry about it.”

Similarly, don’t worry about what other people are thinking of you because they’re probably thinking of you a lot less than you think they’re thinking of you. They’ve got better things to do by their perspective. Same with software, by the way. We see our own software eight+ hours a day. We know where all the skeletons are buried. We see every little imperfection. Customers, by and large, don’t care about the little things.

Amy: They don’t.

Patrick: If it makes their life better, great. You’ll have complainers who largely won’t buy it anyhow.

Keith: Especially on the backend.

Patrick: 90 percent of the customers if it accomplishes the big 48 point font promise that’s on the front page of the website they’re good. If it gets better over time, that’s great, but fundamentally they’re good. If it has a bug, no problem. Computers eat things all the time. Whatever. They will say “I’ve got better things to do than worry about it.”

Amy: All those things come from the same route, if you ask me. What Terry Pratchett called being trapped in the dark behind the eyes. It’s just that we go through our lives 100 percent privy to everything that goes on inside us, even if we don’t understand it, which most of us don’t. This has been proven by research.

When we make a mistake or we choose something we have an elaborate reason why. When someone else does the same thing we get really glib and superficial like well, I did this because I made a mistake but he did that because he’s a jerk. It all comes from being self-involved, which is the default nature of humanity.

I think you were saying it’s like a zeitgeist. That was the right word. I think it’s just human. I think a lot of us, especially in Western cultures, we tend to self-flagellate for no good reason. People call it the Puritan work ethic or whatever.

Keith: Well, it’s not just Western. I was going to say…

Patrick: Japan could teach everybody about self-flagellation. (Patrick notes: If I were not talking in real time I’d say “I could have a very long discussion on the degree to which Japan counts as ‘non-Western’ here if you wanted me to.” Side effect of getting a degree regarding that subject.)

Amy: [laughs] Fair enough.

Keith: Looking at it from the Japanese perspective, I wonder how much of it is almost like an arms race. So one of the things that happens in Japanese companies…

Patrick: Oh, God, yes. Oh.

Keith: So, going back to the startup, where people have to suffer. So they have to work the 20 hours a day kind of thing, only four hours of sleep, constantly working, not taking care of their health and stuff like that. There are people out there who only need eight hours of sleep, who enjoy working 15, 18, 20 hours a day.

I’m actually close to that. I love working. And I work much more than I probably should, because I enjoy it. It’s my hobby to be creating things. And I think people see, especially people like that who have become successful and think, “Oh, this person is successful because he only sleeps four hours a day. In order for myself to be successful, I have to only sleep four hours a day as well.” And I think it becomes an arms race for trying to be successful.

And in Japan, there’s a very similar thing with the amount of hours people work. So people think that people in Japan work long hours and they are productive for all those hours. That is the furthest thing from the truth on the planet. They sleep. They clean their ears. I had my coworker assemble a bicycle in his cubicle [laughs] during work hours, for no apparent reason whatsoever.

It’s assumed that, just like in the startup business, there are people who work long hours because they are really good and they are successful. There are people who work long hours because they are idiots and not successful, and it takes them time to do everything. But the longer the people are there, if everyone is there, it’s so much harder for you to go home, right?

Amy: Right.

Keith: If successful guy number one is working 12, 14 hours a day, you think, “Oh, I have to be there as long as he’s there. Otherwise I’m not going to be seen as being as productive as him.” So what it comes down to is a bunch of people sitting in an office for 14, 16 hours a day, only doing about four to five hours of actual work.

Amy: So it’s cargo-culting mixed with social contagion. (Patrick notes:Great line!)

Keith: Exactly.

Amy: Right.

Patrick: And like a massive game of chicken…

Amy: Yeah. [laughs]

Patrick: Chicken or prisoner’s dilemma, I guess, one of them. The first person to decide to go home gets the evil eye. I think that’s part of the startup culture, too, in that, “Oh, you quit after only 10 hours today. You must not want success enough.” We construct our own cultural pathologies, because people don’t have enough exemplars of folks in companies that said, “We worked four, six, eight hours today, and we go home to the kids, and things are fine.” The cultural pathology of overwork ends up getting celebrated.

Keith: We need more Fog Creeks of the world.

Patrick: Fog Creek, the office is a ghost town after five o’clock.

Amy: As well it should be.

Keith: Except on game night.

Patrick: Except on game night. (Patrick notes:Every Thursday. Third-best reason to work there. They’re hiring, go work for them.)

Keith: [laughs]

Amy: So you were saying, Keith, the examples you had were the guy who works 12 to 14 hours and is successful and then the guy who works, I think you said 12 to 14 hours and was not successful.

Keith: Yeah.

Amy: I thought you said lower numbers for the second guy. But anyway, when you said that I was thinking that what you don’t have room for in Japan, apparently, but also not in Silicon Valley, is the person who works five hours a day and actually outperforms the person who works 12 hours a day. And that’s not uncommon.

Keith: Right.

Amy: I have a lot of people who use Freckle who’ve written in to me and said, “You know what I discovered, which is really freeing, is that I actually only get two to four hours of work done at my computer done every day. The rest is dicking around. And so I’m going to spend all the rest of the time that I would waste on the computer going out and playing music or walking around or reading, and then I’ll get more work done in the two to four hours I actually work.” And I think that’s true. It’s backed up by a lot of research.

Keith: Oh, definitely, definitely. And going back to the Japanese side of it, the problem is that when everyone is forced to work 12 to 14 hours a day, you then have the problem of, why would I work smarter? Why would I try to automate my process so that I can work more during those 12 to 14 hours instead of dicking around?

So it’s actually, because you’re in a trapped system here, then there’s no reason to better yourself. But using a product like Freckle, and especially for consultants and people who define their own time, it’s the biggest win you can possibly have. If you find out that you are dicking around on the Internet for two, three hours a day while you’re working, and a time-tracking software like Freckle actually makes you realize that, and then you gain two to three hours a day…

Amy: That’s true.

Keith: Because, as soon as you realize that you’re dicking around, you go, OK, I’m just going to leave the computer. I’m going to de-screen. I’m going to go off, play with my child, play with my friends, go out drinking, get slammed, or whatever you want to do, right?

Amy: Absolutely.

Patrick: This is one of the benefits of doing your own thing. You have social pressure coming from yourself, which always happens, but you don’t have social pressure from other people who can tell when you leave the office. The vast majority of days, I have a two to four-hour peak of productivity, and after that I’m pretty much shot. And since I know this about myself now, I just don’t work the rest of it.

Keith: You probably shouldn’t say that. Your financier probably… [laughs]

Patrick: I will tell this to any client. (Patrick notes: Any clients in the audience? You presumably know I have a sense of humor and can judge my pace of working, having sat next to me for a while. Any prospective clients? Productivity for me tends to be bursty, interspersed with periods of introspection, much like your engineers.)

Keith: That you only work two hours a day? [laughs]

Patrick: This is why I have you pay the week rate, guys, because work gets done, but assuming 480 minutes of equally productive time is not a good assumption for working with me, which you will probably notice as I check Hacker News in the middle of the day.

Amy: [laughs]

Patrick: But, no. It’s funny, though. There’s people who I respect enormously who have found out the same thing about themselves. Four hours a day is kind of the productivity limit, and after that it suffers. I know one friend in particular, and I won’t mention his name because he asked me not to mention it publicly, but the point is that he asked me not to mention it publicly.

He thought people would think less of him if they thought that his business, which is wildly successful, was just a part-time gig. Which, that breaks my brain. Half the reason I do the blog and the podcast and whatnot is to give people examples of there being multiple paths to the cheese of success in life.

Amy: Hear, hear.

Patrick: I wish everybody happiness. That’s kind of like a foundational philosophical thing for me, but once you introduce people to other ways to getting to happiness, which can include not working all the time.

Amy: Or much at all. [laughs]

Patrick: Or much at all.

Amy: That was not a slam on you or anything. I was actually thinking about myself when I said that.

Patrick: It’s no problem.

Amy: Not that I thought you would think it was an insult.

Patrick: I’m lazy like a fox.

Keith: He’s very proud of that. He’s very proud of that. He gets on my case all the time for working too much.

Amy: I thought foxes were pretty brown.

Patrick: I’m going to convert Keith.

Keith: You converted me to quitting my day job, so you might be successful yet.

Patrick: I’m only saying I’m going to convert Keith because Keith is my best friend so the less he works the more time I have to play League of Legends with him.

Keith: And the more time he has to exploit me for his own product development and stuff.

Patrick: Keith, in addition to being my best friend, is also the designer, but I can’t get any of his time because it’s been filled up with client work. Anyhow, what was I saying? If other people are sincerely happy working 16 hours a day in the coding salt mines then bully for them.

But I know because I’ve talked to and met a lot of people who are doing the 16 hour days in the coding salt mines because they think either that’s required to be successful or because they are strongly socially pressured by people that that is the behavior you should emulate.

If you are folks out there like that, if it makes you happy, thumbs up, go for it, but if you’re not truly happy by that then start doing something that will make you happy because there are so many ways to succeed in this. In business, in life, in general.

Amy: Absolutely. You’re not talking about runners up, either. We did not quite hit my revenue estimates for 2011, but we did have $550,000 of revenue and in 2008 we had zero.

Keith: Not too shabby, right?

Patrick: High five.

Amy: I’m sorry, what?

Keith: Not too shabby, right?

Amy: Not too shabby, right. Exactly. I was hoping for 600 grand and we didn’t quite make it, but that was with a lot of drama where we didn’t work for a lot of that year. I had surgery, I was really sick. That was the three months. That was last year. I had surgery. I was out of commission for six weeks then. We had this hiring and firing drama. That year was screwed and we made $550,000. I’m basically retired and I don’t want to be this way forever.

I really enjoy working and I enjoy having impact and I enjoy touching people’s lives with my software and my course, and I do spend a lot of time on my course. 30×500, that is. We could sit on our asses and rake in $550,000 a year and really work just a couple hours a day on average and we could really cut our overhead.

Most of our overhead we spend on developing new features and our new app, Charm. We spent a lot of money on that the last year. That’s because I had bigger ambitions, but I’m never going to work a 40 hour work week. I had this near death experience, basically, with chronic fatigue. I had this priority change. I was a workaholic. No, no more. Now I’m a hippy, but you can be a hippy and earn $550,000 a year if you pick the right product and if you keep at it.

The first year and a half kind of really sucked, but we got over that and now we’re making really good money and it’s not that hard. Patrick, I found out about you because you blog about this stuff because you’re trying to be a positive example and I also don’t understand why the person you know refuses to be named because he’s afraid of being shamed and that’s just really sad, I think, that by telling people you make the world a better place.

I understand why he’s afraid, but I would rather put it all out there and be a positive example because there are so few of them.

Patrick: I think that is one reason, too. Going back to a topic we were just talking about, like you said, we’re not runners up here. How do I want to phrase this? I like celebrating other people’s successes. You guys make more money for me, bully for you. 37signals can buy everybody in Chicago a sports car these days, bully for them. It makes me happy to hear that other people are doing well.

Where is this topic going? Just a life tip for everybody listening, if you compare yourself to other people and think you’re not successful unless you’re beating them by some metric you will generally be less happy than you are if you’re comparing yourself against either where you were previously or what your goals are.

Success, for me, is either beating where I was last year or beating where I thought I was going to be this year. That generates happy points for me, whereas I never really cared about comparing with other folks. I think folks like the friend of mine who success is partly defined as being seen as successful, that kind of screws up your priorities a little bit, although I’m not totally immune to that myself.

That was rambling. I’m sorry.

Amy: Not as much as me.

Keith: I think we’re going to have to close this down because we’ve gotten complaints in the past about us talking too much.

Patrick: This one is only an hour and a half or so.

Keith: We also shot the shit for about 15 minutes so after editing it’ll be about an hour. That’s pretty good.

Patrick: OK. Only an hour. Thanks very much, Amy. Let’s give folks the actionable information with the call to action at the end. If they want to sign up for 30×500 how would they do that? (Patrick notes: Again, you can’t, because it took a while to get this posted. Sorry about that.)

Amy: First you actually have to apply. We’ve been creating more and more successes each time I run the class and I want to keep that turned up, so I want to basically help decide with you if 30×500 is right for you at this time. That application is launching on April 13. That’s Friday the 13th. All the information is on my blog at unicornfree.com.

Patrick: Sounds great. Thanks very much for doing the podcast with us, Amy. It was insightful as always.

Amy: Thank you for having me.

Keith: It was great to meet you and looking forward to seeing more about 30×500 and more Freckle stuff, too.

Patrick: For all you folks in the audience, we’ll probably be doing this again in a month or two. See you next time.

Keith: Depending on when we can all get together with the microphone. All right. Thanks for joining us, Amy. You take care.

Amy: Thank you for having me.

Special Bonus Prize For People Who Read The Whole Transcript

I did a 45 minute video on improving the first-run experience of your software, to increase customer happiness and conversion rates. Some of the folks who have implemented advice from it have already told me that it made them appreciable amounts of money. You should probably watch it. Click here to give me your email address on the right side of the page, and I’ll give it to you free. You’ll also get an occasional email from me about things you’ll find interesting, like e.g. a new podcast getting posted, not-for-the-blog thoughts about software/marketing/etc, and (possibly) a product announcement someday.

11 Comments

I Redesigned My Software. Users: Thrilled. Conversion Rates: Up. Sales: Unchanged.

My oldest software product, Bingo Card Creator, is currently in maintenance mode.  For the last year and change, I’ve done very little to actively improve or market it — I just send emails, cut checks, and collect profits.  That was pretty much the plan for this year, too.

Then, I got an email out of the blue from Ashraful Sheik, a designer who had seen a years-old HN post by me about my design needs and wanted to see if I needed any work done.  I don’t usually rush to employ people who send me unsolicited emails, but I’m always happy to read emails, so I took a glance at his portfolio in case I could recommend him the next time one of my clients needed a designer.

I noticed that he had previously done a design for VLC Player.  It’s software that does… actually, I’m not really sure what it does, but I remember it from an HN thread waaaay back about them having SEO problems, and the reason I remember it was because I really liked their website design.  Simple, elegant, modern…  and very much not what Bingo Card Creator looked like.  So I mulled it over for a few minutes, figured I had a few days free in April, and asked Ashraful for a quote for a full-blown redesign of BCC.  I thought I’d try A/B testing it against the existing site.

Technical Sidenote: Why I Never Do Big-Bang A/B Tests

People have often asked me why I’ve never tested full redesigns before, and the answer is always “They’re a metric tonne of work to do correctly.”  You might naively assume that you just create two versions of your application’s template and, bingo, the money starts rolling in, but it is never that simple for non-trivial applications.

If you only have one site-wide template and you are totally religious about not including presentational code in any view/template/partial and the before and after redesigns are very compatible at the DOM level, then doing the A/B test isn’t that bad.  This was very much not the case for BCC and will likely not be the case for most live applications.

I actually considered making a complete copy of the BCC application with a shared database, then doing the split testing with some sort of software load balancer redirecting people to two entirely separate Rails stacks, but that promised to be a whole heck of a lot of maintenance pain going forward in return for avoiding coding pain for the integration.  So having nixed that idea, I did some plumbing in the Rails 2.3 internals to override how Rails magically picks layouts. This let me make duplicates of my existing layout structure for the redesign, do the appropriate HTML changes to them, and then start worrying about the main content areas of the layouts (where the real work began).

#goes in application_controller.rb
#Hack hack HACKEY HACK to re-direct all layouts to /layouts/redesign if this user is in that A/B test.
alias_method :old_active_layout, :active_layout

def active_layout(passed_layout = nil, options = {})

  redesign_choice = session[:big_bang_redesign] || ab_test("big_bang_redesign", ["default", "redesign"], :conversion => "purchase")
  # Exclude abingo controller and one blog article from scope of redesign -- customers don't see them and they'd take real work to get right.
  @use_redesign = (redesign_choice == "redesign" && (params[:controller] != "abingo") && (params[:action] != "developing_shopping_cart"))
  unless (@use_redesign)
    old_active_layout(passed_layout, options)
  else
  layout_name = old_active_layout(passed_layout, options).to_s rescue nil
  return nil if layout_name.nil?
  chosen_layout = "layouts/redesign/#{layout_name.gsub("layouts/", "").gsub("redesign/", "")}"
  find_layout(chosen_layout, default_template_format, options[:html_fallback]) if chosen_layout
end

Ahh, DOM conflicts.  One of my requirements for the new redesign, to save my sanity, was that it be built on a grid system.  I picked 960.gs because I happen to like it, but Bootstrap would have worked just as well.  BCC is presently written without the benefit of a grid system, so the internals of many pages require a bit of tweaking to fit onto one.  Also, the redesign omits elements of the previous design in some places in such a way that “display:none” doesn’t really cut it, so I wanted to be able to quickly turn off and on bits of HTML based on whether someone was using the redesign or not. I made a quick helper to do so: redesign(true) { # renders only if someone is seeing the redesign}, redesign(false) { #renders only if someone is seeing the old version}.

  def redesign(for_redesign = true, &block)
    test_choice = session[:big_bang_redesign] || ab_test("big_bang_redesign", ["default", "redesign"], :conversion => "purchase")
    # Exclude abingo controller and one blog article from scope of redesign -- customers don't see them and they'd take real work to get right.
    @use_redesign = (test_choice == "redesign" && (params[:controller] != "abingo") && (params[:action] != "developing_shopping_cart"))
    #If for_redesign and use_redesign are both truth or both false, yield.
    if (!(for_redesign ^ @use_redesign))
      yield
    end
  end

This let me start attacking the marketing site and application for Bingo Card Creator, adding code-spackle where required to get things actually working correctly. It turned out to be necessary in 60 places, consuming most of the three days it took to get the redesign working after receiving the HTML and CSS mockups for it.

The final technical measure was for user experience: anyone who has ever done a redesign knows that a vocal contingent of users hates them.  As long as I was doing an A/B test anyhow, I gave users the ability to flip between which version they were seeing, buried waaaaaaaaay at the bottom of the page in the footer.  This way when people complained (and two inevitably did) I could tell them how to opt-out of the new version.  (It is also indispensable when testing to see that the site was functional in both versions.)  Feel free to use this feature if you want to see both versions of the site live.

Enough Technical Mumbo-Jumbo, Let’s See Some Screenshots

 

The old version of the home page:

The new version:

As you can see, the new version is cleaner, more modern, and (partially as a consequence of finally adapting to a world with wider displays than 800×600) has quite a bit more room to breathe between elements.  It probably still won’t get featured in any design galleries, but that isn’t the point: this site exists to sell software.  (I rush to add that this isn’t a reflection on my designer’s skill: my brief constrained him into favoring the commercial imperative over design imperatives in a few ways.  As always I’m ultimately responsible for anything which looks bad and the designer gets the credit for anything that doesn’t, since if I were left alone to design things they’d look like big balls of blue-green mud with large orange BUY NOW buttons stuck in them.)

Redesigning The UX of The App

As long as we were giving the site a facelift, I decided to see if without majorly tweaking the underlying application we could make it more usable.  I thought of adding a prominent graphical element suggesting what steps it requires to make bingo cards and tracking user progress, something which has been reported to work frequently among UX folk.  (I also have a motivational result or three from clients about this.)

This is what a new trial user previously saw:

Here’s what they see now:

I added a few affordances to that design.  For example, clicking on the elements of the progress bar makes my best context-based guess of how to move you backward or forward along the path of making bingo cards.  It also highlights showing you how far you’ve gone, as seen here:

You’ll note that I haven’t fixed the Next Step button yet.  Ahh well, always one more thing to do…

So How Did Things Go?

I generally do far less extensive A/B tests than this, and track them only to a single conversion.  (That is actually a limitation of my A/B testing software, A/Bingo, because I never really saw the need before to track a change’s effect on multiple conversions in my own business.)

However, since this redesign affects every part of my funnel from the AdWords landing pages to the internals of the application to the purchasing page, I thought it might reasonably be the case that the redesign was a win somewhere and a loss elsewhere, so just tracking to the final conversion (purchase) might cause me to have an incomplete view of the implications of  the redesign.

Enter KissMetrics, my current favorite funnel tracking software.  They’re wonderful, you should use them.  I’ve happily paid $150 a month for the last year or so and barely log in — that is totally justified now.

KissMetrics lets you include custom properties as people cause events in your site/app/etc (which you can then retrospectively organize into funnels on their website).  I simply included which version of the site someone was seeing as a custom property, then fiddled with their UI a bit until I had the filters set properly, and voila, I can now see the A/B test affect every funnel I have.

In some cases, the redesign was a win.

AdWords Landing Pages: Did Registrations Go Up?

Consider the AdWords landing pages, where I measure conversion to the free trial (all stats taken from last 7 days just for convenience, but they mostly match results since start of test):

old version Visits: 1,403 Conversions: 293 (20.9%)
redesign Visits: 1,349 Conversions: 311 (23.1%)

I’ll spare you the z-test: That modest increase is statistically significant at the 10% confidence level, but not at the 5%.  So middling evidence of a change in the right direction.  So far, so good.

It was actually a much more dramatic increase on my non-AdWords landing pages (SEO-ified content, you’ve heard me talk about it before), but that pales next to the following improvement.

The Application: Did User Success Increase?

I’ve written and presented previously about using funnel tracking to improve your application such that users are more likely to succeed with it.  Success with Bingo Card Creator involves, predictably, actually creating bingo cards.  Somewhat surprisingly, back when I started tracking it, only 48% of free trials actually got as far as actually successfully downloading a bingo card.  I’ve worked on that and gotten the number to 60% over the years — roughly a 25% lift in user success with a huge, honking increase in my bottom line results as a consequence.  My consistent experience has been that the more users succeed with BCC, the more money I make.

So if we look at the workflow for BCC:

 

I had to stitch that together from a pair of screenshots because it won’t fit on one interface in KissMetrics, but the numbers are accurate.  Let me direct your attention to the salient bits:

  • The redesign has a lot more people start at the Dashboard than the “default” (old) version does.  This is because the redesign is, as we previously discussed, crushing the old version at getting user signups to the free trial.
  • Each step of the funnel includes a percentage, which is the percent of the folks who started at Dashboard that made it all the way through that step.  (They’re not inter-step percentages, they’re cumulative.)  If you want to back out the math, you’ll see that the redesign outperforms the drop-off rate of the old site at every step in the funnel — 1% here, 2% here, 5% here.
  • This compounds multiplicatively.  By the end of the funnel, the redesign has a 9.2% increase (a 15% lift) in user success compared with the old version of the software.  To give you some appreciation of that: if you’re working with mature software and have already grabbed your low hanging fruit, the magnitude of that improvement is staggering.  That is literally better than a year’s worth of active tweaking at this point.
  • When you compound the pre-workflow increase in trial signups and the increase in user success, the actual number of teachers who successfully create bingo cards out of a given number sent to my site in the first place goes up by 45%.  This makes this the most successful A/B test I’ve ever run, at least by that metric.

Egads, So This Printed Money For You, Right?

Cue the bad news!  Teachers are so successful with the newly redesigned BCC that, out of any 100 using the software, less of them decide to purchase it.  This almost exactly cancels out gains in trial registrations and user success.  It is downright painful: last week, I got 26 sales, and would you believe they were split exactly 13/13?  That’s practically a textbook null result — it was so improbable to me that I spent a few hours checking stats to see if I hadn’t made a systematic error somewhere, but no, crosschecking in a few places makes it look legitimate.  (The split since I started the test is 50/46 in favor of the old version, which is comfortably in the null result territory as well.)

Does that result sound counterintuitive to you?  It is the sort of thing that, when I have to report it to clients, always sets me walking on eggshells.  The first rule of A/B testing is that everything you know is wrong.

Since I have the luxury of well-instrumented funnels, I can tell you where the problem isn’t:

We did a complete redesign of the purchasing page and shopping cart.  I omitted showing it above to save space, but I’m pretty proud of how it turned out .  The new purchasing page/shopping cart is not the problem: precisely as many people will, once getting to the purchasing page, complete a purchase as they would previously.  (On the subject of plumbing-that-takes-money: Stripe.  I’ve promised them a case study post someday, but the capsule version is that if you can use Stripe you shouldn’t be using anything else.)

The problem appears to be, simply, that less users hit our trial limitations now.  (Hitting the 15 card trial limit is the overwhelming cause of going to the purchasing page.)  This suggests that either a) the new site is converting more parents than the old one used to, and since parents rarely have 15 children they’re simply having a happy bingo experience and not paying (net win for the world, not really a win for me personally though) or b) for indefatigable reasons, users simply get what they need out of the free trial and don’t convert.  It is entirely possible that any of the sixty small tweaks I had to make to the site nudged people away from hitting those limitations.

This is one of the reasons I hate big-bang A/B tests — when you have a huge batch like that, isolating the exact cause of any observed effect is difficult (other than “Well, clearly, something changed in the redesign”), whereas A/B testing an individual element structurally gives you configurable levels of confidence that a particular element was the one and only cause of an observed effect if the stats shake out that way.

So Where Does That Leave You?

If I had a great desire to do more work in BCC, this would provide a good place to target.  I could figure out why the new version of the site is having less folks hit trial limitations, and either tighten those limitations or tweak the UX such that the site nudges more people into hitting them.  That said, the free time on my schedule is rapidly drying up as we get closer to my wedding, and even if I had captured all of that 45% increase to the bottom line that isn’t really the path forward for my business, so BCC is going back into maintenance mode.  This one is getting written off as an amusing and partially successful experiment which helped out my users but didn’t succeed at making me money.  I will likely finalize the redesign and kill the old version in the coming weeks.

How did existing users respond to the change?

Well, half of them don’t know about it yet, obviously. With two exceptions, the feedback has been overwhelmingly positive. Many of them were appreciative that they got the Totally New Software Absolutely Free. It is actually functionally identical to the old version — not one line of model code or business logic changed — but I have received many compliments about the wonderful new features, performance increase, and improved compatibility with Epson printers. Before you laugh, consider that probably 95% of software businesses don’t A/B test yet, so my users are in good company making inferences from observed behavior changes over time even when their explanation for the changes has no relationship to reality whatsoever.

Sidenote: If new software is assumed to be worth money and reskins make software “new” in the minds of the only people that matter (users), that probably suggests a viable marketing strategy. Your engineers don’t have to like it.

How much did it cost you?

BCC is growing like a weed for reasons totally unrelated to my work on it (long story but you can see the recent stats).  This means I have quite a bit of flexibility to cut checks to make things happen.  To give you a rough idea, we settled on a price of $1,X00 for PSD, HTML, and CSS mockups of my front page, my pricing page (complete with minor JS for the shopping cart), and the main page of the application.  I then munged that into site- and application-wide templates to get things to their current state.

Ashraful was wonderful to work with: very responsive to email, timely, and receptive to feedback in a way that improved the quality of the designs vis-a-vis my target market while also decreasing the amount of integration work I had to do.  I’d work with him again in a heartbeat.  You should hire him.

25 Comments

Why I Don’t Host My Own Blog Anymore

I moved my blog over to WPEngine recently. Why? Read on.

I started blogging about 375,000 words ago (about three full-length novels… crikey). At first, I was on a subdomain of WordPress.com, mostly because a) it was free and b) I had no intention of ever writing anything more significant than a few observations I made while developing a summer project.  My posts about making and marketing software turned out to be rather more popular than I anticipated, and I eventually made them my main professional presence and moved them to their own domain name.

Eventually, I decided that I had to be in control of my site rather than being locked into only the themes and functionality that WordPress.com supported, so I started hosting the blog myself.  I installed WordPress on a modest 512 MB VPS from Slicehost, with the standard Apache 2 / PHP / MySQL stack, and thought that would be the end of my hosting decisions.  Sadly, it was only the beginning.

WordPress, PHP, and Apache: The Trifecta Of Finnicky Software

I’m a web developer by trade, so I’m very skeptical of unevidenced comments like “lol, PHP can’t scale.”  It’s transparently obvious that PHP can scale — one look at who uses it (Facebook is really all you need to know) proves that.  Sadly, my system crashed frequently under load, even after upgrading the 512 MB slice to a 1,024 MB slice, and after tweaking Apache and PHP’s memory-usage settings for hours.  I read practically everything written about caching plugins for WordPress and it availed me not.  The loads at issue weren’t generally that impressive, either: 10,000 pageviews here,  100,000 pageviews there, either way a modern server in 2006 ~ 2012 should eat those numbers for breakfast.

In addition to tweaking the heck out of my settings, I was hacking WordPress to lessen the load on the server.  I went so far as to putting all the static resources for this blog (CSS, images, etc) on a server used for my software business, since no level of traffic has ever managed to give Nginx a problem in my experience.  This made the blog much more stable under load, but it still crashed occasionally.

I eventually found the culprit: a setting in Apache named, ironically, KeepAlive.  Since then I have been seized by missionary zeal in trying to convince people that leaving KeepAlive on will probably not be in your best interests.  I turned off KeepAlive and have had a much more stable time since then.

Even with all this work, my blog crashed four times in 2011.  Each time, my monitoring software woke me up, and I learned things like “Oh, Jimmy Wales tweeted about an article of mine” at 4 AM in the morning while trying to restart the server.  I’ve probably lost in excess of 100,000 readers over the years for the blog being down.

I eventually got so fed up with Apache that I spent a day to migrate this (and 15 other sites) from using Apache as a front-end webserver to using Nginx to serve all requests and proxy dynamic requests to a backend Apache instance.  (Why not use Nginx to execute the PHP directly?  Long story — I do that elsewhere, but it isn’t painless.)

Routine Maintenance Sucked, Too

You’ve heard about WordPress’ somewhat spotty security record, right?  I’m aware of it as a practicing engineer and have even reported a few doozies in commonly deployed themes/plugins myself.  So I took some fairly extraordinary measures to secure the blog versus a stock WordPress installation:

  • required all access of the admin to happen through a proxy that I control
  • locking down all files on the WordPress installation such that the webserver could not write to them — this makes plugin installation/maintenance a manual chore and breaks some plugins, but makes it less likely that a vulnerability in a plugin or theme will ruin your day
  • performed some sporadic code audits on things I am inexpert on in a language I detest.  I eventually stopped doing this because reporting vulnerabilities to the WordPress ecosystem could easily be my full time job.

As a result of this, to my knowledge my blog never got compromised.  Whee, great.  It only took me a few billable weeks.  Plus I had all the usual fun of applying patches, making backups, restoring from backups when MySQL decided to eat the wp_posts table (still no clue why that happened), tweaking settings for rotating logs, migrating my hosting provider (Rackspace bought Slicehost so I had to move servers), yadda yadda yadda.

“Do You Enjoy Hosting WordPress?”

Two years ago at a conference I ran into Jason Cohen, a very smart guy who had sold his previous software business.  He told me that his new venture was managed WordPress hosting.  I was outwardly interested and inwardly cringing, because I thought “There are already WordPress hosts available for $4 a month, they all suck, and the software is pretty much irredeemable.  If it weren’t the best blogging software available I’d take a hammer to my backups then burn the shards and bury them on sanctified ground so that they never troubled the world again.”

At some point Jason asked me if I liked hosting WordPress.  I told him “I love hosting WordPress!  I love all the power and tweakability!”  And while I do appreciate control, I still can’t imagine what possessed me to say that.  Hosting WordPress has been a black hole of my time.

Last year I ran into Jason Cohen at another conference. He told me that WPEngine, the WordPress hosting company he’d told me about, was live and doing well.  In my haste to demonstrate that I had learned something from cutting myself on WordPress for the last five years, I mentioned “I guess you guys figured out to turn KeepAlive off, huh?”

Jason said “Actually, no.  I mean, sure, in the general case for a VPS, you want it off because otherwise your site gets non-responsive under load.  However, if you’ve got Apache talking to the outside world, something is wrong.  Apache only handles the request after it’s been through a load balancer, a Varnish caching proxy, and then Nginx, because Nginx does static content so well.  If the request gets that far you want KeepAlive on because your Apache will only be talking within your datacenter, only have a handful of connections, and you want those connections to be alive almost indefinitely because setup/teardown is always waste.”

You know how often I talk to software company CEOs and get not just corrected but destroyed and then re-educated about a point of technical fact?  Suffice it to say it made an impression.

So when Jason invited me over to WPEngine to do some marketing work, I leaped at the chance.  I went down to Texas for a week, met the team, and did my thing.  (Sidenote: Want to see a fairly typical week’s work for me?  Take a gander at their speed test tool, which you can point at an arbitrary WordPress site and learn why your page load speed isn’t optimized enough yet.  The punchline is, of course, that if you were with WPEngine they would have already fixed that for you.  I assisted with a redesign of this, wrote the month-long WordPress optimization course that the tool will let you sign up for, and generally improved copy and the like on their marketing site.)

So I Switched

In the course of hearing the sales pitch from them several times so I could write it accurately, I became convinced: WPEngine is absolutely superior in every way to me continuing to host the blog myself.  So I took out my credit card, signed up for their $200 a month plan (prices got reduced recently, see here), and migrated my blog over.  It has been quietly hosting my blog for the last several weeks, including through two of my highest traffic days ever, without a hitch.  For the first time, I can watch a post go to the top of HN and not have to have “top” open to keep an eye on swap consumption.

A few things that particularly impressed me about WPEngine:

  • A few hours after migrating my blog I got an automated email saying that they had found an outdated copy of TimThumb in my WordPress install and had upgraded it for me.  It wasn’t a vulnerability (permissions locked down saved the day for that one), but I’m very, very glad they keep an eye on things so that I don’t have to.
  • Migration was almost painless.  I just dumped the WP database, grabbed my existing files, and copied them over as instructed.  I needed to speak to support to get a setting tweak done for me (the plan I bought has WordPress multi-site not single site like my old blog, and this resulted in a minor issue), but all told I was up and running in about two hours of elapsed time.
  • Just like their speed tool promised, my site did get modestly faster.

I’m a bit of a YSlow fanboy and ever once in a while I go through my sites and make an optimization pass, so I usually have all the low-hanging fruit like gzipping, static content loaded from multiple domains, and the like taken care of.  I wasn’t expecting WPEngine to shave much time from my page loads, given the amount of optimization work I had already done, but I kept the old server around to do a fair test on AOL’s speed tool.  Take a look what happens: here’s a video showing (left) my old 1,024 MB VPS versus (right) WPEngine showing the same page from my blog.

If you didn’t watch the movie, I’ll spoil it: content pops in about half a second earlier (and finishes loading .8 seconds earlier) on WPEngine with no manual tweaking versus my tweaked-to-limit-of-my-ability VPS.

That test uses IE8 on a reasonable residential Internet connection coming from the US. For accessing from Japan or on a mobile device, it is viscerally faster for me, probably due to the CDN which WPEngine uses.

Do You Have A Blog For Business? Use WPEngine

The VPS that I used to run my blog on cost a bit over $100 a month (1 GB Rackspace slice + 160 GB of bandwidth + backups = I am not actually sure). WPEngine runs me $200 a month because I have high anticipated traffic. They have a $29 option for folks who don’t.

$2,500 a year for blog hosting sounds a bit on the high side, but it is honestly nothing against the amount of time that I will no longer have to invest supporting this sucker. I love being out of the hosting business. I never intended on being in it in the first place — it was always just something I needed to do to write for people. Less time poured down that black hole means more time to work on my businesses and share what I learn doing so.

WPEngine has been a total, epic win for me. I suggest that you use them if you are using WordPress for a business: go ahead, make with the clicky clicky. Want to just geek out on how they have their infrastructure setup? See here.

P.S. Long-time readers are aware of this, but just to reiterate: I don’t take money for blog posts, was not asked to write this by WPEngine (who are, again, clients of mine), and would not have written it except that their service really rocks.

66 Comments

Salary Negotiation: Make More Money, Be More Valued

[Editor's note: At nearly 7,000 words, you probably don't want to try reading this on an iDevice.  Bookmark it and come back later.]

Imagine something a wee bit outside your comfort zone.  Nothing scandalous: just something you don’t do often, don’t particularly enjoy, and slightly more challenging than “totally trivial.”  Maybe reciting poetry while simultaneously standing on one foot.

If I told you I would pay you a hundred thousand dollars if you did five minutes of poetry recital while standing on one foot, would you do it?  It’s an absurd image, but play it straight.  There is no hidden gotcha here.  You won’t be videotaped.  Your friends will never see you make a fool of yourself.  The revolution will not be YouTubed.  The offer is exactly as simple as you think it is: poetry, foot, $100,000.

Would you read poetry for me?

Of course you would.  You’d be screamingly stupid not to.  In fact, not only would you read poetry, you’d probably take a poetry class to make sure you did it right, or go to the gym to verify “Yep, sure enough, I can stand on one foot.  Phew.  Pass me the Shakespeare.”  If you couldn’t stand on one foot, you’d fix that, because you know that is much easier than other things you routinely accomplish and you suddenly have a hundred thousand wonderful reasons to learn it, too.

What if you were talking about this at dinner with your friends, and one of them said “Oh, no, I’d never do that.  I just don’t do poetry.  I’m an engineer.  And besides, my father told me that people who stand on one foot look silly.  And what do I need $100,000 for anyhow?”  You would not clap them on the back and say “Damn straight!  Man, poets, always trying to tempt virtuous engineers into their weird poetry-spouting flamingo-standing ways.”  You’d say “Dude, it’s five minutes.  Heck, I’ll help you practice.”

This is pretty much how I feel every time I talk to my engineering friends about salary negotiation.  We overwhelmingly suck at it.  We have turned sucking at it into a perverse badge of virtue.  We make no affirmative efforts to un-suck ourselves and, to the extent we read about it at all, we read bad advice and repeat it, pretending that this makes us wise.

Dude, it’s five minutes.  Let’s un-suck your negotiation.

(New to the blog?  Hiya.  I generally write as an engineer for engineers.  Non-engineers can benefit from many of the same techniques, though the hiring market isn’t nearly as in your favor at the moment as it is for engineers in most major US metro areas.)

Why Negotiation Matters

Your salary negotiation — which routinely takes less than 5 minutes to conclude — has an outsized influence on what your compensation is.  Compensation can include money or things which are more-or-less fungible replacements for money, but it can also include interesting things which you value from “more time with your family” to “opportunities to do tasks which you find fulfilling” to “perks which make a meaningful difference in your day-to-day quality of life.”  That makes your negotiation five very important minutes.  You generally can’t do a totally bang up job on any five minutes of work this year and have your boss give you an extra $5,000.  You can trivially pick up $5,000 in salary negotiations just by sucking less.

Since salaries are shockingly durable over time, particularly if one is not good at negotiating, you can expect a $5,000 increase in salary to compound with your standard annual read-the-HR-chart-percent raise, cause a similar increase in your 401k contribution (which also compounds), and establish a higher peg for any further jobs you take (if you’re unsavvy and allow these other jobs access to your prior salary history, at any rate).  Accordingly, over the next ten years, the value of $5,000 a year extra salary is close to $100k gross, and the value of $15,000 a year extra (very achievable if you’re e.g. a young engineer who doesn’t realize that the hiring market is on fire right now, even outside of tech epicenters like Silicon Valley) is over $100k even net of taxes.

Shifting Your Mindset To Embrace Negotiation

We’ll discuss tactical advice in a moment, but let’s talk about the psychology of negotiation first.  I think that middle class Americans are socialized from a very young age to view negotiation as something that is vaguely disreputable and engaged in only by poor people.  Think of the associations you have with the word “haggling”: do you think of a successful young professional talking about thousands of dollars in a brightly lit office?  No, you probably think of an old woman arguing over a trivial sum of money in a dirty flea market.

If I were a little more paranoid and a little more Marxist, I’d honestly think that you’re so misinformed about reality that that is almost prima facie evidence of a conspiracy to keep you in the dark about this, to the advantage of people who a) you won’t negotiate with and b) who will feel absolutely no compunctions about negotiating with you.  Principally, this will be your employers.  People say that your house is the biggest purchase you’ll ever make, but it won’t be the most consequential negotiation.  If you’re sane only about 25% or so of your gross income is subject to the results of real estate negotiations.  Close to 100% is subject to the results of salary negotiations.  Thus, your salary negotiations are probably going to be the most important financial decisions you will ever make.  We socialize middle class Americans to go into them unprepared, demotivated, and fearful of success.

The reality is that rich, successful people negotiate.  (This is one important way in which they get — and stay — rich.)  It is an all-day-every-day thing in much of the business world, which is where most rich people get their money.

Your Counterparty Does Not Share Your Mental Model of Negotiation

Salary negotiations are very asymmetrical.   Companies know this and routinely exploit it.  Job seekers don’t, perhaps because they think doing so would be unfair and the word “exploit” makes them acutely uncomfortable.  So we often default by pretending that the employer is evaluating the negotiation like we would.  This is not true, and acting like it is true will harm both your interests and the interests of your future employer.

For example, many people’s mental model of employment is that an employee with a $60,000 a year salary costs about $60,000 a year to hire.  If they negotiate $65,000 instead, that’s $5,000 extra which has to come from… somewhere.  If the negotiation breaks down, then that is $60,000 saved.  This mental model is broken.

First, get into the habit of seeing employees like employers see them: in terms of fully-loaded costs.  To hire someone you need to pay for their salary, true, but you also have taxes, a benefits package, employer contributions to retirement, healthcare, that free soda your HR department loves mentioning in the job ads, and what have you.  (Trivia: for a US employer of professionals, the largest component after salary is usually healthcare, followed by payroll taxes.)  The fully-loaded costs of employees are much higher than their salary: exactly how much higher depends on your locality’s laws, your benefits package, and a bunch of other HR administrivia, but a reasonable guesstimate is between 150% and 200% of their salary.

The fully loaded cost of an engineer receiving market salaries these days in California or New York is close to $20,000 a month.  It is “only” $10,000 a month if they’re receiving a heavily below-market salary, such as if they’re working for a startup.  If you have a kid brother who majored in Flemish Dance and got a modest full-time job at a non-profit, his fully-loaded cost is still probably $4,000 a month or more.

This is a roundabout way of telling you that companies are not sensitive to small differences in employee wages because employees are so darned expensive anyhow.  You see $5,000 and think “Holy cow, even after taxes that’s a whole new vacation.  Five thousand dollars.  Five thousand dollars.  It would be so very, very greedy of me to ask for five thousand whole dollars.”  The HR department sees $5,000 and thinks “Meh, even after we kick in the extra taxes, that is only about 3% of their fully-loaded cost for this year anyhow, or seven hundredths of one percent of that team’s hiring budget.  I wonder if the cafeteria has carrot cake today?”

Virtually any amount of money available to you personally is mouse droppings to your prospective employer.  They will not feel offended if you ask for it.  (I received a comment that this is untrue for startups by someone today.  For a funded startup which has enough engineers to warrant a foosball table, the company payroll is well north of $100,000 a month.  Making a new hire is a big commitment, but they still have a lot of flexibility on the  details because the details do not shave months off of their runway.)

We’ve been talking about your employer as an abstraction, but in the instant case you’re talking to an actual person.  Let’s call him Bob.  It is Bob’s job to get you signed with the company as cheaply as possible, but Bob is not super motivated to do so, because Bob is not spending Bob’s money to hire you.  Bob is spending Bob’s budget.  Bob generally does not get large performance incentives for shaving money off of his hiring budget: you get a new Macbook if you convince Bob to give you $5k extra, but Bob gets (if he is anomalously lucky) a dinner at TGIFridays if he convinces you to take $5k less.  In fact, there are many organizations (and Bobs) for whom power, status, and money come from asking for more budget every year.  If you turn out to be on the expensive side, that is great for Bob, because a) he manages a high-powered peon so he must be a high-powered manager and b) this will help Bob get more budget next quarter.  So if you’re worried about what Bob will think of your moral character, or you want to compensate Bob because you feel you owe him for this job opportunity, do Bob a solid and negotiate in a spirited fashion with him.

You don’t owe Bob for giving you this job opportunity, by the way.  Internalize this: everyone in this discussion is a businessman.  (Some might call themselves “regular employees,” which just means they’re businessmen with self-confidence issues and poor business skills.)  If the deal makes economic sense, it will happen.  If it doesn’t, firm handshakes will be exchanged, non-specific promises will be uttered, and everyone will forget about this discussion in a matter of hours.  You will not be blackballed for negotiating.  Bob couldn’t care less and, even if he did care, he has better things to do with his time than worry about a candidate he didn’t hire.  Bob is working through a list of a dozen people right now, and his manager Dave is being such a hard case about that project’s schedule, and he’s not sure he can make his daughter’s piano recital, and the cafeteria’s carrot cake was a little dry.  Bob is far, far less invested in this negotiation than you are.

Your Negotiation Started Before You Applied To This Job

Your negotiation doesn’t happen in a vacuum.  Generic career advice is a little outside the scope of this post (though I’ve previously written a bit with engineers in mind that folks from many walks of life tell me was useful), but to make a long story short, many people think job searches go something like this:

  1. See ad for job on Monster.com
  2. Send in a resume.
  3. Get an interview.
  4. Get asked for salary requirements.
  5. Get offered your salary requirement plus 5%.
  6. Try to negotiate that offer, if you can bring yourself to.

This is an effective strategy for job searching if you enjoy alternating bouts of being unemployed, being poorly compensated, and then treated like a disposable peon.  (I served three years as a disposable peon in a Japanese megacorp and might be projecting a tad bit here.  Regardless, my loss is your gain.)

You will have much, much better results if your job search looks something more like:

  1. (Optional but recommended) Establish a reputation in your field as someone who delivers measurable results vis-a-vis improving revenue or reducing costs.
  2. Have a hiring manager talk with you, specifically, about an opening that they want you, specifically, to fill.
  3. Talk informally (and then possibly formally) and come to the conclusion that this would be a great thing if both sides could come to a mutually fulfilling offer.
  4. Let them take a stab at what that mutually fulfilling offer would look like.
  5. Suggest ways that they could improve it such that the path is cleared for you doing that voodoo that you do so well to improve their revenue and/or reduce their costs.
  6. (Optional) Give the guy hiring you a resume to send to HR, for their records.  Nobody will read it, because resumes are an institution created to mean that no one has to read resumes.  Since no one will read it, we put it in the process where it literally doesn’t matter whether it happens or not, because if you had your job offer contingent on a document that everyone knows no one reads, that would be pretty effing stupid now wouldn’t it.

You might think that desirable jobs at well-managed companies (Google, Microsoft, hot startup FooWithTheWhat.ly, etc) have layers and layers of bureaucratic scar tissue (a great image from 37Signals) to ensure that their hiring will conform to established processes and that offers will not be given to candidates sourced by using informal networks and interpersonal connections.  If you believe this, you have a dangerously incomplete mental model of how the world operates.  I have a specific recommendation for you to make that model more complete: start talking to people who actually work for those companies and who have hiring authority.  Virtually no company has a hiring process which is accurately explained by blog posts about the company.  No company anywhere has a hiring process which is accurately explained by their own documents about how the hiring process works.

I won’t give names, but all of the following are companies you’ve heard of:

  • Ironclad non-compete with an IP assignment provision of major multinational… struck from the contract with four sentences of discussion.
  • Major popular tech employer offered desirable employee $X as a salary because “it was the max the HR department allows for that position.”  He got hired that week at $2X.  All parties — hiring organization, HR, and employee — think they pulled one over on the other participants.
  • Funny story goes here. I now can’t tell you the funny story, because literally two hours before publication someone emailed me for advice about a situation that he believes is incredibly unjust at his company, and it is exactly the funny story to the letter.  Now if I tell you the funny story he might think “Dang, I write Patrick in confidence and it ends up on the blog.”  So, no funny story today.  Suffice it to say that in my old age I treat Dilbert less as farce and more as documentary.
  • “We can’t hire engineers fast enough through our standard processes so, meh, I guess we’ll circumvent them by just tossing $1 million per employee at whomever they currently work for.  Who cares, it isn’t my million.”

When Does A Salary Negotiation Happen?

Only negotiate salary after you have agreement in principle from someone with hiring authority that, if a mutually acceptable compensation can be agreed upon, you will be hired.

This is really, really important because it has direct implications for your negotiating strategy.  First, the company is going to spend a lot of time and effort on getting you to the point of agreement-in-principle.  Pretend you’ve gone through six rounds of interviews.  (You probably won’t if you get hired on informal networks, because all barriers vanish when important people want a deal to get done, but let me give some advice to someone a little less well-situated.)  Do some quick mental math on what that actually cost the company, with reference to “one man-month of an engineer’s time costs $20k” like we discussed earlier.  You’ll quickly reach the conclusion that the company has spent thousands of dollars just talking to you, and that doesn’t even count the thousands they spent deciding to talk to you instead of whoever isn’t in the room right now.  Walking away from the negotiation means that they lose all that investment.  (Yeah, sunk cost fallacy and all, but since people predictably act in this fashion you should, well, predict that they will act in this fashion.)  They really want to reach an agreement with you.

The second implication is that the inner serf worrying “If I even attempt to negotiate this, the deal will fall through” is worrying for nothing.  They’ve got thousands invested in this discussion by this point.  They want you.  The absolute worst outcome of negotiating an offer in good faith is that you will get exactly the contents of that offer.  Let me say that again for emphasis: negotiating never makes (worthwhile) offers worse.  This means you need what political scientists call a commitment strategy: you always, as a matter of policy, negotiate all offers.  (In this wide world I’m sure you can find a company who still makes exploding offers, where you get one yay-or-nay and then the offer is gone.  You have a simple recourse to them: refuse them and deal with people who are willing to be professionals.  You’re not a peasant.  Don’t act like one.)

This also means you do not start negotiating until you already have a Yes-If.  (Yes-If we agree on terms.)  Do not start negotiating from No-But.  (No-But we might hire you anyway if you’re really, really effing cheap.)  You don’t want to work for a No-But for the same reasons that smart employers hate hiring candidates who are a No-But (No-But maybe if not on my team, etc).  If they’re leaning to not hiring you, you will compromise excessively on negotiation to get them to hire you.  Compromising excessively is not the point of the exercise.  It is a seller’s market for talent right now: sell to someone who is happy to buy.

This means that any discussion of compensation prior to hearing Yes-If is premature.  If you’re still at the job interview and you’re talking price you are doing something wrong.  (Read the room: it is entirely possible that you came for a job interview, finished it, and proceeded directly to a salary negotiation.  That’s probably suboptimal, but it is OK.  Just don’t give the employer the option of having the schedule be job interview, salary negotiation, and back to job interview if they discover that you have a spine.)  The ideal resolution to the job interview is for both sides to be optimistic about the arrangement, and then you close with a warm handshake and “I look forward to receiving your offer by, oh, would tomorrow be enough time for you to run the numbers?”

You then have a high likelihood of doing your salary negotiation over email, which is likely to your advantage versus doing it in real time.  Email gives you arbitrary time to prepare your responses.  Especially for engineers, you are likely less disadvantaged by email than you are by having an experienced negotiator talking to you.

The First Rule Is What Everyone Tells You It Is: Never Give A Number First

Every handbook on negotiation and every blog post will tell you not to give a number first.  This advice is almost always right.  It is so right, you have to construct crazy hypotheticals to find edge cases where it would not be right.

For example, if your previous salary was set during the dot-com bubble and you are negotiating after the bubble popped, you might mention it to anchor your price higher such that the step down will be less severe than it would be if you engaged in free negotiations unencumbered by the bubbilicious history.  Does this sound vaguely disreputable to you?  Good.  This vaguely disreputable abuse of history is what every employer asking for salary history, salary range, or desired salary is doing.  They are all using your previous anomalously low salary — a salary which did not reflect your true market worth, because you were young or inexperienced or unskilled at negotiation or working at a different firm or in another line of work entirely — to justify paying you an anomalously low salary in the future.

Never give a number.

“But Patrick,” you cry.  “I don’t want to be difficult.”  You’re not being difficult.  You’re not doing anything immoral.  You’re not being unprofessional.  They’re businessmen, sometimes they don’t have all the information they would love to have prior to making a decision.  They’ll deal.

They already deal with every employee that they’ve ever had who was not a doormat at negotiations, which includes essentially all of the employees they really value.  Ramit Sethi (more on him later) introduced me to a concept that he calls Competence Triggers: basically, if you have to judge someone’s skill based on a series of brief interactions, you’re going to pattern match their behavior against other people who you like.  When people with hiring authority think of winners, they think of people like them who live and breathe this business thing.  They negotiate things as a matter of course: that is a major portion of the value they bring to the company.  Volunteering a number when asked says the same thing to people with hiring authority that flunking FizzBuzz says to an engineer: this person may be a wonderful snowflake in other regards, but on the thing I care about, they’re catastrophically incompetent.  It will also cause them to retroactively question competencies they’d previously credited you with.

I have literally heard that feedback, in so many words, from folks with whom I’ve had successful business dealings.  (A funny in hindsight story: I cost myself five figures with a single email.  The particulars are boring, but suffice it to say I fairly recently made a wet-behind-the-ears-engineer error in quoting a client.  He noticed.  So did my bank statement.  My bank statement kept quiet, but the client opined that it made him think less of me until we actually got to work together.)

So anyhow, you may well hear reasons why you should give a number.

Objection: “I really need a number to move the process forward.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “I’m more concerned at the moment with talking to you about discovering whether we’re a mutual fit.  If we’re a great fit, then I can be flexible on the numbers with you and you can be flexible on the numbers with me.  If we’re not a great fit, then the numbers are ultimately irrelevant, because your company only hires A players and I only work at roles I would be an A player at.”

(Don’t talk like that normally?  Fine then, talk like yourself, but say substantially the same things.  Engineers overestimate how different we really are from business people: we say “10x engineer,” they say “A player,” but at the end of the day we believe that there are vast differences in productivity between workers.  OK, gut check: is this something we actually believe to be true or just something we wish for?  If it is actually your best guess about the state of reality, that has immediate news-you-can-use implications about how you should conduct your life.)

 

Objection: “This form needs a number.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “Give me git access and I’ll fix it in a jiffy!  both people laugh No, seriously, speaking, I’m more concerned at the moment with discovering whether we’re a mutual fit…  Oh, it’s physically impossible?  Put in $1 then to get the ball rolling, and we’ll circle back to this later.”

 

Objection: “We want to figure out whether you’re an appropriate candidate for the position.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “It’s so important to me that this is a good mutual fit for us.  Let’s talk about why I’m a great fit for this position: I know you’re concerned about $FILL_IN_THE_BLANK.  In addition to my previous successes doing it, I have some great ideas for what I’d do about that if I was working at your company.  Would you like to drill into those or is there another job area you’re more concerned about to start with?”

 

Objection: “I’m sorry, great try at a dodge there, but I just can’t go forward without a number.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say (if you’re an engineer): “Well, you know, I would hate to have to walk away from the negotiation over this.  Working with your company looked it would have been such a wonderful opportunity.  I hear the hiring market is super-tight right now, would you like me to introduce you to other candidates?  Maybe we can shave a couple of months off of you filling this position.”

What you should say (if you’re not an engineer): “Damn, I guess I should have studied engineering.”

What you should say (if you’re a little put out by that comment): “Well, you know, salary is only one component of the total compensation package.  In terms of total compensation, we’re probably looking at something like $FILL_IN_NUMBER_HERE.”  (Suggested calculation: take the package value from your last company and add 5~10%.  If you don’t know how to calculate the value of your compensation package, learn that, but as a rough guesstimate salary + 30 ~ 50% for full-time employees in professional roles and the multiplier tends to scale up as your base salary scales up.)

P.S. I double majored in making things and making things up.  The joking comes from a place of love.  OK, love and schadenfreude, in solution with each other.

Listen To What People Tell You.  Repeat It Back To Them.

Properly run negotiations are not jockeying contests, they’re persuasive exercises.  (We’ll give the company a pass on the “what’s your number?” question because it is an established social ritual that they get one free pass at screwing you.  You still don’t have to cooperate with it, though.)  You know what people find persuasive?  Their own words.  People love their own words.  When you talk to them, you should use their own words.  Seriously, watch the eyes light up.

Did the solicitation for the job say “We are seeking someone with strong skills at scaling traffic in a fast-moving environment”?  Pick out the key words.  Scaling traffic.  Fast-moving environment.  “Scaling traffic” doesn’t sound like how I’d phrase it if I were writing or speaking for myself, but if you’ve just described your need to me as scaling traffic, by golly I will tell you how great I am at scaling traffic.  Reinterpret or rephrase the (true!) bits of your own story such that it fits the narrative framework which they have conveniently told you that they are going to respond to.  Did you previously work at a small business which was unencumbered by lots of process?  Sounds like a fast-moving environment, right?  Call it exactly that, then.

Micro-tip: Take notes during job interviews and salary negotiations.  It’s easy: go to the convenience store before the job interview, buy a writing instrument and a $1 notebook, jot down occasional notes when appropriate.

Can I do that?!  Of course you can.  Do you know anyone who you’ve ever thought “Man, I thought they were competent, but then it turned out they had a notebook so I had to write them off?”  No.  Taking notes says “I’m attentive and detail-oriented and I care about what you say.”  (Make sure you can take notes without playing with your pen or otherwise appearing to fidget.)  In terms of specific things that should get your pen moving, among others, I would focus on specific words they use and concerns they have so that you can come back to them later in the conversation.  Numbers are another good thing to hit the notebook, because numbers should only ever trend in a direction of “Better to you,” so you don’t want to do something stupid like saying “So how many days of vacation was that again?” and let a 24 suddenly become a 20.  (You might think “I’m going to write down the offer so I have proof of it for later.”  Get offers written, that goes hopefully without saying, but get it written by them and/or follow-up the discussion with an email recapping the important points and asking if you understood them correctly.  Your notes will not convince their HR apparatus to honor the agreement in event of a retroactive miscommunication, but an email from their decisionmaker likely will.)

People say the damnedest things.  For example, someone might spontaneously volunteer during a job interview that they’ve been interviewing for the position for six months.  (None of my clients would ever say that, of course, but then again one would hope none of their consultants would chop five figures off their own invoice with an email.)  If they say the position has been open for six months, take a note of that.  During the salary negotiation, if they have a pricing objection, one of your first responses should be “I appreciate that this is a little more money than you might have been thinking about, but this is an opportunity to get this position filled without delaying your business by another six months.  What is the value of that six months of execution to you?”  (Conversely, don’t say stupid things during job interviews such as “I need this job because…”  You never need a job.  Being needy means that the party who is not needy has automatic leverage over you: your BATNA to the negotiation is very poor.  Instead of being needy, aim for “I’m enthusiastic about the opportunity with working with you, assuming we can come to mutually satisfactory terms.”)

Micro-tip: Notice how often I say “We” and variations on “mutual win.”  Those work pretty well.  The only thing better than “We” is “You” (and variants), because people care a heck of a lot more about their problems than about your problems.  (This advice is stolen shamelessly from Dale Carnegie.)  This means that a) you should talk about their problems, concerns, and wishes and b) you should guard against your own natural tendency to bring up irrelevant things like your own problems, which typically will not help you sell the decisionmaker on adopting the mutual win you’re proposing.  Similarly, I generally try to phrase things positively rather than score debating points.  (“You just said X, but that was contradicted by your earlier statement Y, which means…” wins debating points but does not win friends and influence people.  You might try something like “Good good, but taking into account your earlier concerns about Y…”)

Research, Research, Research

Many people will tell you that you should familiarize yourself with the approximate salary range for the position in your region.  This advice is easy to act on (go to a salary aggregation site, guess what “the position” is, pray that this gives you a better number than rand(40000,120000)), but it leaves a lot to be desired.  It is 2012.  Facebook and LinkedIn exist.  You should, before any job interview, have intimate knowledge of the target company.  Prospective peers within the company are one obvious way to get it.  So are ex-employees, folks who’ve had dealings with them professionally, etc.  Key things you want to learn:

  • What do they value?
  • Who do they value within the company?  (Roles?  Titles?  Groups?)
  • What does the career path look like for successful people within the company?
  • Roughly speaking, how generous are they with regard to axes that you care about?
  • Do they have any compensation levers which are anomalously easy to operate?  (For example, if you asked around, you might hear a few people say that a particular firm pushes back modestly on out-of-band increases in salary but they’ll give in-the-money option grants like candy.)
  • All the fuzzy stuff: what’s the corporate culture like?  Yadda yadda.

You can even bring a lot of these questions to the job interview, which is (again) prior to the negotiation.  (Maybe not “So are you guys tightwads?” but culture-esque questions like “What are the projects this company thinks are really key to its future and how would a motivated person go about getting on them?” are both a) totally fair game and b) will win you brownie points just for asking.  Similarly, a lot of employees will, out of company loyalty, attempt to sell you on taking the job with the company by trading you very useful information.)

The more you know, the more options you have when doing negotiation, because you’ll have more things and more motivational things which you can offer in exchange for things you want.  It will also help you avoid making mistakes like e.g. getting into a rigid classification system where the classification you’re aiming at will make forward advancement towards your goals very difficult.  (Example: there are some companies where Product and QA are run like separate fiefdoms which haven’t forgotten the most recent war, and in those companies getting hired as an engineer may not be a career enhancing move if you like making things for a living.  There are other companies where people cross-function in those responsibilities all the time and applying for a job advertising as “Support Engineer” makes lateral moves onto customer-facing projects trivial.  You can find which one you’re applying to by taking any engineer out for coffee.)

New Information Is Valuable And Can Be Traded For Things You Want

There was a post recently on Hacker News about someone’s experience with a job offer from Google.  They wanted more money.  The recruiters offered to think it over, and came back with the reply that Google’s food benefit was worth a significant amount of money, with a calculation to back it up.  That is a pretty brilliant reply.  Google’s food benefit is about as old as the company.  Approximately all people wanting to work at Google are aware of its existence.  However, the explicit calculation of what it is worth is new, so if you bring up that calculation, by implication you’re offering newly found value to the negotiation.  This successfully convinces people that they didn’t really need that extra money.  It is so successful at this that Google recruiters apparently have this entire interaction scripted, since multiple people report having the exact same experience.

You should steal this tactic.  You are an expert in your own skill set, life story, and (ideally) value you can create for the company.  However, the person you are talking to is not.  If they ever resist about something which you want, consider reaching into the treasure chest that they are buying mostly blind and revealing one of the many glittering jewels inside.  They are going to get them all anyhow if they buy the chest, but each one you bring out decreases the perceived risk of buying it and therefor increases its perceived value.

Company: We can’t see our way to $88,000.

Applicant: Well, I know you do a significant amount of business with your online store.  At my last company, I increased sales by 3% by $YADDA_YADDA.  What would a 1% increase in sales be worth to you?

Company: Well, I don’t have that figure in front of me, but…

Applicant: Would it be safe to say “millions of dollars”?

Company: That sounds about right, yeah.

Applicant: Great, I can’t wait to get started.  Getting me that extra $4,000 would make this a much easier decision.  Considering that this is conceivably worth millions to you, we’d be silly not to do business with each other.

Company: I’ll see what I can do.

Applicant: Let me help give you some options!  [See below.]

(This hypothetical applicant is doing well on the negotiation but apparently needs to do more research on what conversion optimization specialists can get away with charging these days.  Here, let me help: six figure salary with all the usual perks as an employee, “senior engineer project rates” through “you might not believe me if I told you” as a consultant.)

Anyhow, simply by bringing attention to something which was hopefully already in bold print on their resume, they just increased their perceived value to the company, thus justifying the company moving a lever which (again) the company isn’t really sensitive to at the end of the day.

You Have A Multi-Dimensional Preference Set.  Use It.

Don’t overly focus on your salary number.  It is important (of course), but there are many parts of your compensation package, and many more things that you value.  Should you and the other party reach an impasse on any part of it, offer to table that part of the discussion (to be returned to later) and bring up a different topic.  You can then trade improvements for concessions (or apparent concessions) on the earlier topic.

Employer: “We were thinking $80,000.”

Applicant: “$80,000 is interesting (*) but not quite where we need to be to get this done.  Do you have any flexibility on that number?”

Employer: “I think I can convince HR to approve $84,000 but that is the best I can do.”

Applicant: “I appreciate that.  $84,000, huh.  Well, it isn’t quite what I had in mind, but the right package offer could make that attractive.  How much vacation comes with the package?”

Employer: “20 days a year.”

Applicant: “If you could do 24 days a year, I could compromise on $84,000.”

Employer: “I think I can do that.”

For those keeping score at home: the applicant never gives up anything but the employer will walk away feeling he got a good deal.

* Micro-tip: “Interesting” is a wonderful word: it is positive and non-commital at the same time.  If they tell you a number, tell them it is an “interesting” number, not a “wonderful” number.

Hoping around the offer also helps you defuse common negotiating tactics like “I have to go to $EXTERNAL_AUTHORITY to get approval of that.”  (This is in the negotiation playbook, because it works well: it injects an automatic delay in the process, and gives you a scapegoat for refusing a request while not being guilty of the refusal yourself.  You should strongly consider having an $EXTERNAL_AUTHORITY of your own.  Significant others work well.  Note that in the US your would-be employer is legally prohibited from breathing about the subject of your marital status, so something like “We’ll have to talk that over” or “That sounds reasonable, but I’ll have to run it by the family” has the dual virtues of being a socially acceptable reason to delay any major decision while also being equally available to unattached young’uns.  I talk shop with my family all the time.  I’ll certainly continue discussing employment with my family after it includes my fiancee, too.)

Anyhow, say your decisionmaker says that approving deviations from the company’s salary structure is outside of his discretion and those evil ogres in HR will likely deny his request.  That’s fine.  Express sympathy with him, because he just said he wants to give you more but can’t, then refocus the discussion on things which are within his personal authority.  (Vacation days, work hours, project assignments, travel opportunities, professional development opportunities, and the like are good areas to probe at.)  You can then use the unspent “You wanted to do something nice for me” obligation which he just acknowledged on one of the things which he has authority to grant you.

For Your Further Perusal

I’m deeply indebted to a few buddies of mine, principally Thomas at Matasano and Ramit Sethi, for teaching me to be less of a doormat in terms of negotiation.  Thomas has forgotten more than I’ll ever know about doing negotiations with clients.  Check out HNSearch.com with [tptacek negotiation] for some good advice, or (if you’re in Chicago) take him out to coffee.  By the way, if you’re an engineer and want to practice salary negotiation, Matasano is hiring and a great place to work for.

Ramit is extraordinarily persuasive about how psychology influences how people act in negotiations, and how a deeper understanding of this gives you better preparation than trying to focus on particular tactics.  When I told him that I was finally going to blog about one of our dinner conversations (you’re reading it), he said he’d give my readers a free video on negotiation tactics and the psychology underlying them in return for your email address.  The video is good and elaborates on some subjects that I haven’t covered, plus I know some people find them easier to learn from than lots of text, so that strikes me as a very fair offer. You’ll also get Ramit’s emails, which have never wasted my time.

47 Comments

Bingo Card Creator (and etcetera) Year In Review 2011

I’m Patrick McKenzie (patio11 on the Internets) and for the last several years I’ve run a small software company.  My first product was Bingo Card Creator, my current product focus is Appointment Reminder, and I do occasional consulting for a variety of clients, mostly on helping them sell more of their software over the Internet.

Traditionally, right before Christmas every year I release an annual report.  See, for example, 2006, 2007, 2008, 2009, and 2010.  (Crikey, have I really been doing this for that long?)  I’ve also traditionally published live stats for Bingo Card Creator, but not my other lines of business.

Writing the annual report is partially to keep me grounded, partially to talk through my thoughts on the year and goals for next year, and partially to (hopefully) give other folks ideas that they can use in their own businesses.  I hope you find it interesting or, at the very least, mildly amusing.

Obligatory disclaimers: Assume any statistics that I give are “roughly accurate, to the best of my knowledge, at the time this report was written.”  There are still a few weeks left in the year.  Sales are typically low in the last two weeks, but the exact timing of credit card charges can cause a bit of jitter in the December stats.  From past experience, I have a high degree of certainty that there are about $1,000 or $2,000 of expenses (across all lines of business) which aren’t in the bookkeeping  system yet and won’t be until I sit down in March and check things for taxes.

Capsule summary: Best year ever, by a lot.  Broke $100,000 in sales for the first time and increased total profits to ~$70k.  2012 has inflection points coming for life and the business.

The Year In Brief

I put Bingo Card Creator into maintenance mode for approximately 48 weeks out of 2011: I only answered emails and kept systems running, but took no action to improve the product or marketing.  (The other four weeks I tried a few minor things out.)  This was, theoretically, supposed to free me to spend most of my efforts on Appointment Reminder…

… but that didn’t end up happening.  For a variety of reasons, most of my focus business-wise went into consulting.  Although I technically only did about 10 weeks of consulting during the year, I spent quite a bit of overhead time on e.g. arranging deals which ended up falling through, arranging the deals which did actually go through, and doing general promotion activities like speaking at conferences.  (I had the opportunity to speak at about a half dozen conferences this year, and assorted other events.  It is great fun, but since I generally have to fly to America for them, they tend to munch a full week out of my schedule each.  I spent almost three months of the year in the US, doing a combination of family events, consulting, prospecting, speaking, and meeting some Internet buddies to discuss plans for later.)

I also lost two solid months due to dealing with legal issues, mostly centering around Immigration.  I’d love to fill you in on the nitty-gritty, but have been asked not to by people close to the situation.  Suffice it to say that I was a shoe-in for a Japanese visa back when I worked at a large megacorp, was not a shoe-in for a visa when doing my own thing, and had a very hairy experience with getting them to approve me as a “self-employed engineering consultant.”  Tips of the hat to my Japanese clients, particularly Makeleaps / Webnet IT and myGengo, whose support was instrumental in getting Immigration to approve my renewal.

Despite not having nearly as much time to work on Appointment Reminder as I would have liked, I did manage to firm up its technical underpinnings, add new features requested by clients, and do a small amount of work marketing it.  I hope to make that more of my focus in 2012.

Bingo Card Creator

Despite being in maintenance mode, BCC continued performing like a trooper.  People always ask “Could you afford to live on it only?” and the answer is “Yes, but barely, and it would require a lifestyle adjustment, mostly in the don’t-fly-across-the-Pacific-so-often department.”  BCC did not meet the numeric goals that I had for the year.

Stats:

Sales: 1,539 (up 6% from last year’s 1,451)

Refunds: 14 (down from 22 last year, to .9% of sales from 1.5%)

Sales Net Of Refunds: $45,479.93 (up 5% from $43,398.55)

Expenses: $22,560.00 (up from $18,287.93, but largely just due to an accounting issue — I can’t split costs in my homegrown bookkeeping software, so the ~$3,000 I paid for servers for AR is hiding in that number)

Profits: $22,919.93 (see above accounting issue, essentially flat from last year’s $25,904.66)

Wage per hour: Let’s see, ~15 hours of programming, 20 minutes a week on customer support…  about $700 an hour.  Not too bad.

 

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 821k (up from 777k)

Unique visitors: 670k (up from 655k)

Page views: 2.9 million (up from 2.7 million)

Traffic sources of note: Google (46%), AdWords (18%), Binghoo (13%)

Trial signups for online version: 82,000 (up from 72,000)

Approximate online trial to purchase conversion rate: 1.8%

 

Narrative Version:

Aside from kicking up AdWords spend modestly (to no good effect) and running a few A/B tests, nothing really substantial happened with Bingo Card Creator this year.  I lost probably $1,000 to $2,000 of sales when the site crashed right during the middle of the Halloween rush for ~9 hours while I was on an airplane.  That was a little disappointing, but while it broke my candy budget it won’t exactly put me in the poorhouse.

Projections that BCC would continue to grow despite not being actively worked on turned out to be totally wrong.  I forecast 50% growth, reasoning “Hey, most of the systems work pretty much without my intervention, so I think the overall growth of the Internet plus a few A/B test means, oh, 50% or so.”  It mostly tread water.  I’m not hugely disappointed.

 

What Went Right:

  • Not having to work hardly at all for it.
  • Aside from the Halloween crash, the system was largely stable for the year.  I think I got woken up by the automated alarm maybe once.
  • SEO, AdWords, email marketing, and the usual scalable marketing stuff continued to be my bread and butter even when I was too lazy to actually cut and butter bread.

What Didn’t Work So Well:

  • Crashing on the third busiest day of the year, in such a way that it depresses my AdWords campaigns for the first and second busiest days of the year.
  • I integrated Stripe and expected a huge lift in conversions for going from Paypal to a simple CC-based payment system.  I tested this extensively in A/B tests.  I love everything about the Stripe system, but I have no evidence for “Stripe is better than Paypal/Google Checkout”, “Stripe/Paypal/Google Checkout is better than Paypal / Google Checkout”, etc etc.  That said, it might be something as simple as my buttons being ugly.  I’ll probably take a whack at it in the future, or better yet, have my designer take a whack at it.

Consulting

I did a few weeks of consulting this year, for several different clients.  Mostly, I do my engineering / marketing shtick for software companies, although some of my clients have been a wee bit farther afield.  I wrote up a fairly typical engagement with Fog Creek.  That one was a mutual success and we’ll continue to work together in the future.  (To the best of my knowledge, all of my consulting clients are happy with my work.)

One thing I’m going to do differently in the future is to work for less clients.  Don’t get me wrong: I love all my clients.  I was privileged to work with them.  However, it takes approximately X units of work to set up an engagement with a previous satisfied customer, 5X units of work to get a new prospect to the go/don’t-go decision on a new engagement, and I generally have to get three to four prospects to that point to actually wind up with a signed contract.  As my buddy Thomas at Matasano says, “That is life in the big leagues.”  However, since I’m not in a position where 100% utilization is a huge overriding goal of mine, I don’t need to keep the new prospect pipeline totally full… so I’m probably going to cut back on it quite a bit in 2012.  I’ll continue doing follow-up engagements for established clients where it makes mutual sense to do so, and I’m still of course available for interesting projects, but I’m not going to be doing six-week fly-across-America-four-times tours to drum up new business.

The following numbers are approximations only.  NDAs and having the sense God gave a tadpole constrain me from revealing my “going rate.”

Consulting sales: $55,000

Consulting expenses: $13,000  (mostly hotels and airfare for prospecting, which I pay for out of pocket.)

What Went Right:

  • Client selection.  I was, again, privileged to work for people who have interesting businesses, problems that I could make substantial contributions on, and the willingness and ability to pay all invoices in a timely fashion.
  • Raising rates.  My first guesstimate at my rate, back in 2010, was $X.  It turns out that I could do just about as much work as I wanted regardless of whether I charged $X, $2X, or $5X.  As a result, I typically quote fairly high rates and mostly stick with them, unless there is another reason I really, really want an engagement to happen.

What Didn’t Work So Well:

  • Disorganization.  At one point I was juggling something like five simultaneous proposals out while preparing for three conferences, two engagements, and six weeks of travel.  It got so bad that I showed up at a city once and checked at the airport for where I was staying, quickly seeing that I mistimed a conference by three days and thus had no hotel booked, booking a hotel from the taxi, and then arriving at the hotel to recheck my schedule and discover that I had used the previous year’s schedule and was actually simultaneously at a different hotel across Brooklyn.  (Shoutout to the Brooklyn Beta guys for saving me from my own stupidity that week.)  There were multiple points in the year where I found myself wishing for either a boss or a secretary or somebody to just say “Show up to X on Monday and Do Stuff and all the stuff that is not Stuff will be taken care of.”  My occasional slipups in dealing with the demands of a growing business caused me to drop balls in ways that were sometimes client-visible, too.  This is a major part of the motivation for cutting back next year.  (There is Plan B, of course: hire folks to do either the execution or the admin and take whichever part they’re not doing, but I don’t think I’m moving in that direction.)
  • Too much work!  Largely due to overhead and travel, plus the outsize distraction generated by the same, consulting munched a heck of a lot more time than I thought it was going to.  I wanted to have a solid eight months of the year to work on AR.  I think I probably got maybe two.

 

Appointment Reminder

I launched Appointment Reminder last December, with the goal of having approximately 200 customers and $10k in monthly recurring revenue by now.  I had planned on focusing for most of 2011 on marketing and selling it to more businesses.  That largely didn’t happen, but since I got the fundamentals of my SEO strategy in place (while largely ignoring the modestly more advanced content creation / etc that runs BCC and that I usually help clients with), the business grew despite my best efforts at totally neglecting it to focus on consulting and not getting deported.

AR has been hanging around at a crossroads for a while now.  There are two very different trajectories it could go down.  In one, I grow it organically, and it grows into a modestly profitable software business which will provide handsomely for my family and (in the fairly near future) employees.  In two, I take outside investment, and attempt to grow as quickly as possible to $N million a year in revenue, at which point options would include either a) selling to one of the larger players in the small business software space or b) continued operations at scale with a focus on growth.  Luckily, I have  the luxury of waiting on making that decision: my runway is infinite, the market opportunity is only getting bigger, and the perceived value of my involvement with a startup among investors does not appear to be depreciating.

This is one of the reasons I can’t be as open as I would like to be about the current status of the business.  BCC has essentially no secrets, and would not really benefit from having them, as — aside from elementary school English teachers — there is nobody out there who has something I want for BCC.  However, if I hypothetically wanted to take investment, then accredited investors suddenly have something I want very much and having secrets about AR gives me something with which to trade to get it.  (It is similar to not putting prices on an Enterprise Software website.  You can trivially get them, but the price of getting them is giving a salesman permission to give you the spiel.  Similarly, folks who ask about AR’s numbers these days are generally asking in the hopes that they eventually receive a phone call asking them for a check.)

The other reason I can’t talk about AR numbers so much is that I radically underestimated how important the enterprise market would be to the business, and you can’t spell enterprise without NDA.

So: I wanted to have two hundred customers by now.  For the publicly available plans, I currently have a few dozen paying customers.  There are ways to get things from me that don’t involve paying the numbers on the Pricing page.

AR is modestly profitable — it covers all of its own costs.  I plow most of the money it generates back into the business, though, rather than taking distributions.  For example, I’m now about 95% certain that I will have significant contractor or employee involvement on it in 2012.

Revenue: Undisclosed

Expenses: Undisclosed (very modest ongoing expenses, reinvested most profits)

Profits: I took about $5k just to have a number that would minimize disbelief at the tax office.

What Worked Right:

  • Twilio.  The Twilio API and service have been unalloyed epic wins for Appointment Reminder.  I had zero disruptions in service attributable to them, their customer support has been fast, responsive, and technically savvy (even helping me debug my own code at points), and they’ve been very supportive of me.  Plus they have these awesome red track jackets that they keep sending me, which you’ve probably seen if you’ve seen a picture of me doing any talk this year.  (I actually wear them mostly because I love the color red, but apparently I wear them so often that folks at the Fog Creek office thought the Twilio logo was my logo.)
  • Sendgrid: It’s like Twilio, except for email.  Great service.  No red jackets.
  • Unit testing & staging servers.  I am gradually getting more sophisticated in my engineering practices, and have been ramping up my testing activities since starting to code AR.  It has transformed the way that I do development, for the better, and made it easier to respond to customer requests to change things while decreasing the number of problems I have caused.  Total win.  See my presentation at TwilioConf for examples of the specific ways I use it for AR.
  • Exact match domain names.  “Hey Patrick, how is it that with no marketing budget and nearly no marketing work you rank #1 for [appointment reminder]?”  I told everybody that I was buying the .org specifically because that would happen but apparently folks didn’t believe me.
  • Using the self-service site as lead generation for enterprise sales.  Fairly self explanatory.
  • The service itself: AR solves a clear customer need, and my customers are raving fans of it.  There exist many services businesses which incur hundreds in direct costs and thousands in forgone revenue for a single missed appointment.  (Think, say, an HVAC company which sends a three-man team of tradesmen out to your house to replace your heater, which is a $2,000+ job, only to discover that you aren’t home to let them in.)  One of my customers reports that just the delta in no-shows since starting to use AR would pay for his mortgage and his daughter’s college education.  Many of my other customers report that their office managers, who previously did telephone reminder calls manually, are ecstatic to not have to do them any more.  Customer retention among folks who actually use the system (as opposed to signing up, doing a test call, and forgetting about it) is virtually 100%.
  • Talking to smart people for advice: Since I’ve been going back and forth on the investment question, I talked to a lot of entrepreneurs and investors whose opinions I respect.  I really appreciate their feedback, which ranged from “Are you kidding?  You’d hate it.” to “I want to invest in you, but realistically, you would lose nothing by waiting until you are sure.” to “Best decision I ever made.” and helpfully included a lot of actionable advice on how to do things in the meanwhile such that options remain open.

What Didn’t Work So Well:

  • Catastrophic engineering failures.  I had one combination outage/catastrophic failure in February (the details are recounted in that TwilioConf presentation) and a ~3 day period of sporadically degraded operations after my move to Rackspace, which I finalized over the Thanksgiving holiday.  Both of those were my fault, for architecting the system in a way which did not gracefully handle its multiple moving parts getting out-of-sync with each other.  I’ve since done significant work on making it more stable.  (Overall reliability for the year has been excellent, but those periods were easily the most stressed I’ve ever been about any business issue.)
  • Lack of focus: I’ve been commenting above on this, so I won’t belabor the issue, but I really didn’t get to work on AR as much as I wanted.
  • Enterprise sales: I’m actually fairly decent at Enterprise Sales, and am working with someone in the industry who has a deep Rolodex among folks who would be great candidates for AR, but (partly due to the focus issue and partly due to my own comfort level) I didn’t put nearly enough effort towards it this year.  What I should honestly do is go to a conference some time, prospect like a madman, and then make following up on those leads my only job until I’ve got contracts signed.  (The prices for enterprise SaaS make this very economically viable.)

Goals For 2012

Bingo Card Creator

  • I’d be happy with continued flatness ($~30k profits on $50k sales), maybe.  It isn’t the source of growth for my business anymore.
  • Continue using it as a laboratory for weird ideas I have on conversion optimization.
  • Don’t break it during Halloween.

Consulting

  • Do less work prospecting for new clients.
  • Do more work for existing clients.
  • Modestly increase billings, if that makes sense for where my overall business is.  (If I take external investment in AR, that will likely require shuttering the consulting business.)

Appointment Reminder

  • Figure out whether I want to take investment or not.  If so, do so.
  • Convince Keith (who I do my podcast with) to work with me, if possible.  (Don’t worry, he knows this is on the agenda.  We’re best friends.)
  • See about transferring responsibility for the engineering (particularly front-end) side of things so I can focus on marketing/sales.
  • 10x current sales numbers.  That seems to be a fairly safe bet regardless of whether I shoot for a small business or for a high-growth business.  (1,000x-ing would be another story.)

A personal note: The last 3,300 words ultimately matter much, much less than the next 3: she said yes.  We’re announcing to our family on Christmas, as per our family tradition.

27 Comments

Productizing Twilio Applications

This post includes video, slides, and a full-text writeup. I recommend bookmarking it if you’re on an iPhone right now.

I make extensive use of Twilio (a platform company that lets you do telephony with an API) in running Appointment Reminder, my core product focus at the moment.  (Wait around a day or two and I’ll tell you a bit about how it is doing in my annual end-of-the-year wrapup.)

Twilio has a very passionate developer community and fairly good documentation on their website, but I’ve sometimes been frustrated at it, because it teaches you the bare minimum to get phones ringing.  That is truly a wonderful thing and a necessary step to building a telephony application.  However, if you continue developing your application in the way that the Quick Start guides suggest, you will routinely run into problems regarding testing your code, maintaining it, securing it, and generally providing the best possible experience to your customers and the people they are calling.

I have a wee bit more than a year of practical experience with a Twilio application in production, so I went to TwilioConf and did a presentation about how to “productize” Twilio applications: to take them past the “cool weekend hack” stage and make them production-ready.  Twilio has graciously released videos of many of the presentations at TwilioConf, so I thought I’d write up my presentation for the benefit of folks who were not at the conference.

The Video (~30 Minutes)

Twilio Conference 2011: Patrick McKenzie – Productizing Twilio Apps from Twilio on Vimeo.

The Presentation (40 Slides)

The Writeup

 

Why I Think Twilio Will Take Over The World

(This was not actually in the presentation, because I didn’t have enough time for it, but I sincerely believe it and want to publish it somewhere.)

I think Twilio is, far and away, the most exciting technology I’ve ever worked with.  The world needs cat photos, local coupons, and mobifotosocial games, too, but it needs good telephony systems a lot more, as evidenced by companies paying billions of dollars for them.

Additionally, Twilio is the nascent, embryonic form of the first Internet that a billion people are going to have access to, because Twilio turns every phone into a smartphone.  The end-game for Zynga’s take-over-the-world vision is the human race slaved to artificial dopamine treadmills.  The endgame for Twilio’s vision is that every $2 handset in Africa is the moral equivalent of an iPhone.  I know which future I want to support.

Smartphones aren’t smart because of anything on the phones themselves, they’re smart because they speak HTTP and thus get always-on access to a universe of applications which are improving constantly.  Twilio radically reduces the amount of hardware support a phone needs to speak HTTP — it retroactively upgrades every phone in the world to do so.  After that, all you need is the application logic.  And what application logic there is — because the applications live on web servers, they have access to all the wonderful infrastructure built to run the Internet, from APIs that let you get Highly Consequential Data like e.g. weather reports or stock prices or whatever, to easy integration with systems which were never built to have a phone operating as part of them.

You can’t swing a stick in a business without hitting a problem which a phone application makes great sense for.  I filled up three pages of a notebook with them in just a week after being exposed to Twilio for the first time.  Order status checking for phone/fax/mail orders.  Integrated CRMs for phone customer service representatives.  Flight information.  Bank balances.  Server monitoring.  Appointment reminders.  Restaurant reservations.  Local search.  Loyalty programs.  Time card systems.  Retail/service employee support systems.  Shift management.  The list goes on and on and on.

Seriously, start writing Twilio apps.

What This Presentation Will Actually Cover

I’m tremendously optimistic about the futures of Twilio and the eventual futures of companies which make Twilio applications, but I’m pessimistic about your immediate future as an engineer writing a Twilio app, because it is going to be filled with pain.  You’re probably going to make some choices which will cause you and your customers intense amounts of suffering.  I’ve already done several of them, so use me as the inoculatory cowpox and avoid dying.

Crying In A Cold, Dark Room

Back in February 2011, I moved from my previous apartment to my current house.  I unwisely decided to push a trivial code change prior to boxing things up.  This trivial code change did not immediately take down the server, but did cause one component (queue worker processes) to fail some hours later.  The most immediate consequence of this was that outgoing appointment reminder phone calls / SMSes / emails failed to go out.  Since I was busy moving, I did not notice the automated messages about this failure.

When I discovered the failure (8 hours into customer-visible downtime), I panicked.  Rather than reasoning through what had happened, I reverted the code change and pushed reset on the queue worker processes.  This worked, and the queue quickly went from 2,000 pending jobs to 0 pending jobs.  I then went to bed.

At roughly 3 AM, I woke up with a vague feeling of unease about how I had handled the situation, and checked my email.  My customers (small businesses using AR to talk to their clients) had left several incensed messages about how their client had reported receiving dozens of unceasing calls on the behalf of their business, in a row, at 7:30 PM the night before (right after I had restarted the queue workers).

Here was the error: my application assumed that the queue would always be almost clear, because queue workers operate continuously.  A cron job was checking the DB every 5 minutes to see whether a particular client had been contacted about her appointment yet.  If she hadn’t, the cron job pushed another job to the queue to make the phone call / SMS / email.  When the queue came back up, each client received approximately ~100 queued events simultaneously.  These did not themselves check, at the start of the job, whether the job was still valid, because the application assumed that the cron job would only schedule valid reminder requests and not execute 100 times in between queue clearings.

This resulted in approximately 15 people receiving a total of 600 phone calls, 400 SMSes, and 200 emails, in approximately a 5 minute period of time.

There are a variety of ways I could have avoided causing this problem for my customers:

  1. Don’t make code changes prior to planned unavailability, even if they look trivial.
  2. Don’t ever leave your phone that gets emergency messages out of your pocket.
  3. Switch to idempotent queues, so that adding the same job multiple times does not result in multiple copies of the job.
  4. Add per-client rate limits, so that application logic errors don’t cause runaway contact attempts.
  5. Add failsafes for historically unprecedented levels of activity, shutting down the system until I could manually check it for correctness.

Testing Twilio Applications

Unit testing and integration testing are virtually required to produce production-quality Twilio applications, and will make it much less likely for you to create catastrophically bad bugs in production.  Unfortunately, testing Twilio applications is much harder than testing traditional CRUD web applications, because of how TWIML is different than HTML (in terms of how minor syntax errors actually cause business problems), how it is not easy to replicate telephone operation in integration testing, and because Twilio sometimes has poor separation of concerns between the MVC of a web application, the Twilio helper library, and the Twilio service itself.

Twilio testing is inherently dangerous, because non-production environments (testing, staging, development, etc) could conceivably generate actual, real-world phone calls to phone numbers which were in your database but not actually under your control.  The first and most important tip I have for Twilio testing is to make it explicitly impossible to contact anyone not on a whitelist from code when you’re not in production.  I have a quick snippet that I put in a Rails initializer which monkeypatches my Twilio library to force it to only make phone calls or SMSes to whitelisted numbers.  (I don’t suggest actually re-using this code, particularly as you may not be using Rails or the same Twilio library that I am using, but you can reuse the idea of enforcing safety in non-production environments.)

 

 

A lot of Twilio testing will, unfortunately, require manual button-pressing (or scripts which simulate button-pressing on a telephone).  This is easier to accomplish if you can expose your local development machine to the actual Internet.  There are strong security reasons why you don’t want to do this but, if you’re comfortable with doing it, LocalTunnel is a great way to actually accomplish it.

Also see the section below on Modeling Phone Calls, because it will make Twilio phone trees and call logic much more tractable to unit testing.

You Should Have A Staging Server

A staging server is just a copy of the entire production system minus the actual customers.  (You probably shouldn’t put production data on it, because staging systems are designed to break and as a result they may leak data through e.g. SQL injections.  This is an easy way to lose your DB.)  You should use firewalls and/or server rules to make the staging server inaccessible to the world (aside from Twilio and any other APIs which need to access your site for it to work), but assume you will botch this.

Staging servers are virtually mandatory for Twilio applications, because Twilio apps can fail in ways which will not be detected until they are actually accessed over the Internet.  For example, even with unit and integration testing, failing to properly deploy all audio assets (MP3 files, etc) will cause Twilio to throw hard, customer-visible errors in production.  I have automated systems which check for this now, but since that isn’t an exhaustive list of things that can go wrong in production, part of my workflow for deploying all changes on Twilio is to push them to the staging server first, and then having automated scripts exercise the core functionality of the application and ensure that it continues to work.

How To Model Phone Calls

Twilio Quick Start guides generally don’t suggest modeling phone calls explicitly, instead relying on just taking user input and doing if/then or switch statements on it.  This is ineffective for non-trivial use cases, because as the application logic gets more complicated, it will tend to accumulate lots of technical debt, be hard to visually verify for correctness, and be extremely difficult to automatically test.  Instead, you should model Twilio calls as state machines.  I am a big fan of state_machine in the Ruby world.

I’ll skip the CS201 description of what a state machine actually is.  If you didn’t take that course, Google is your friend.

You should model calls such that they start in a predictable state and user input moves the call between states, causing side effects of a) running any business logic required and b) outputting Twiml to Twilio, to continue driving the call.  This lets you replace case statements with a lot of parallel structure with well-defined transition tables within the call models.  Those models are then trivial to unit test.  Additionally, adopting coding conventions such as “the Twiml to be executed at a given state is always state_name.xml and any audio assets go in /foo/bar/state_name/*.mp3 “allows you to write trivial code which will test for their presence, which will save you from having to manually go through the entire phone tree every time on the staging server to verify that refactoring didn’t break anything.

Additionally, state machines are much easier to reason over than masses of spaghetti code which case statements tend to produce.  For example, consider the following code, which attempts to implement the phone prompt “Press 1 to confirm your appointment, press 2 to cancel your appointment, press 3 to ask for us to contact you about your appointment.”  Spot the bug.

There are actually over six bugs in that code, above the trivial ones you probably saw with numbers not lining up to action names:

  • The Twilio API will pass this code params[:Digits] not params[:digits], which will cause an error that won’t be caught until you physically pick up the phone.
  • The comparisons of params[:digits] with integers will fail, because it includes string representations of numbers.
  • There are several mistakes in mapping numbers to actions.
  • One of the action names is spelled improperly.

These are very easy to miss because our brains get lulled into a false sense of security by parallel structure.  Instead, the model should be taking care  of that mapping between user input and state transitions.  This would radically simplify the code and make the controller virtually failure-free, while letting the model exhaustively unit-test possible user input, expected transitions, and business logic side effects.

State machines might seem like an unnecessary complication when you only have three branches in your code, but production Twilio applications can get very, very complicated.  Here is a state diagram from Appointment Reminder.  You do not want to have to test these transitions manually!

Dealing With Answering Machines

Dealing with the case where the phone calls is answered by an answering machine or voicemail system has been the hardest application design problem for me in doing outgoing phone calls in Twilio.  The documentation suggests using an IfMachine feature, which will cause Twilio to listen to a few seconds of the phone call prior to executing your code.  They do some opaque AI magic to determine whether the entity speaking (or not speaking) in that interval is a machine or not, and tell your application whether it is talking to a machine or a human.  In my experience, this has error rates in the 20% region, and many customers intensely dislike the gap of dead air at the start of their phone calls.  Also, if the heuristic improperly detects the beep, your message will start playing early, causing the recording to be cut off in the middle.

There are several ways you could attempt to deal with this:

  • Ignore the issue and treat both machines and humans the same.  This will produce the optimal result for humans, but your system will be virtually unusable when it gets a machine.  (This happens very frequently in my use case.)
  • Force a keypress (“1″) prior to playing your message, then give all users the same message.  This will force most machines to start recording immediately, stopping the cut-off-in-the-middle problem but annoying some clients.
  • Play instructions such as “This is an automated message from $COMPANY.  To hear it, press 1.”  Assume that anyone who doesn’t press 1 in 5 seconds is a machine and play the machine message.  If they interact with the call, play the human message.  This is my preferred solution (although not actually implemented in AR publicly yet, because customers don’t really grok this issue until it bites them personally).

There is one particular problem with recording messages on answering machines: if you give a user instructions such as “Press 1 to confirm your message” and they follow that instruction when listening to their voicemail, that keypress will not be caught by your application, it will be caught by their voicemail system, with unpredictable results (such as deleting the message) and an absolute certainty of not doing what your keypress would normally do.  Users do not understand why this happens.  They expect your instructions to them to work.

Securing Twilio Applications

Twilio applications have a superset of the security issues of web applications.  In addition to the usual SQL injections / XSS issues / etc, use of the telephone has unique security issues associated with it.

One issue is that confidential information is only confidential until you repeat it into a telephone.  Even assuming that the phone call isn’t intercepted (which is, ahem, problematic), there are very common user errors and use cases which will cause that information to be disclosed to third parties.  For example:

  • User error in inputting telephone numbers causes the message to go to the wrong party.
  • The message goes to corporate voicemail, where it will routinely be accessible to third parties.
  • The message is played over a speakerphone / cell phone / etc within earshot of third parties.
  • The message is saved on a physical device which can predictably leave the physical control of authorized parties.
  • etc, etc

Don’t ever put confidential information into an outgoing message, unless you have an automated way to authenticate who you are speaking with.

For incoming phone calls, Caller ID is not sufficient authentication.  It can be trivially spoofed, indeed, your phone company will probably sell you a product whose sole aim is to spoof Caller ID.  Instead, you should use a circumstance where the user is already authenticated and authorized, such as a face-to-face meeting or using a username / password pair in a web application, and then give them one-time PINs to do whatever they need to do on your system.  Alternatively, you can implement an entire password system for your incoming phone calls, but users tend to hate them, so I try to keep to the one-time PIN metaphor.  (When a user does something on the AR site which requires calling the system, such as setting up a recording for a reminder, I tell them “Call 555-555-5555 and put in your Task Code 1234″, which (since it is time-sensitive) both helps me look up what they were doing on a multi-user system and also conclusively demonstrates that they were able to read a web page which already verified their identity.

Not in the presentation because the slide got deleted for some reason: the 4chan rule.  Even if your free trial is discovered by 4chan, the world should not become a darker, more evil place.  There exists tremendous possibilities for abuse of free-form input/output to people’s telephones.  I gate access to my trial by requiring a valid credit card, and demonstration calls and the like have strict rate limits which prevent them from being used to spam someone’s phone to death.  (I should also make it impossible to send demo calls outside of standard work hours.  This is easy to say but a little tricky to implement across multiple time zones while still encouraging legitimate use of demo calls, which is why I haven’t done it yet.)

Twilio Scales Impressively

Twilio and modern web technologies scale impressively well by the standards of traditional businesses.  However, you should probably continue to rate-limit your systems, even though you could theoretically do substantially more volume.  For example, many customers who ask about scaling issues do not sufficiently understand that your application scales several orders of magnitude better than their business processes.  For example, a prospective client asked if my system could handle 10,000 phone calls a month.  I told them that I could handle that in under an hour.  They were quite excited about that, but as we continued to speak about their needs, it developed that actually doing that would have crushed their business.  They would have made 10,000 phone calls in an hour, received over 1,000 callbacks, and their two full-time telephone operators would have been overwhelmed by incoming demand for their time.

Grab Bag of Random Advice

  • Never contact Twilio, or any external API, inside the HTTP request/response cycle.  Doing so imposes an unacceptable delay in performance and slaves your reliability to that of the worst performing API you use.  (Twilio has never had user-visible downtime, but some APIs I rely on have.) Queue the request and tell the browser that you’ve done so.  You can drizzle AJAX magic on your website to make this feel responsive for your users.
  • The Twilio Say verb will have a robot read your message.  This is adequate for development, but for production, people prefer listening to people.  Fiverr.com is great for finding voice actresses for $5.
  • You can’t record too much information about Twilio requests, responses, and errors.  I stuff everything in Redis these days.  I strongly wish I had started doing this earlier, rather than writing “An error happened” to a log file and being unable to determine exactly what the error was or easily figured out whose account it actually affected.
  • When in doubt, don’t make that phone call.  Design your system to fail closed.  This is a continuous discipline, but it will drastically cut down on catastrophic problems.

Wrapup

That’s it for the presentation contents.  I remain very interested in Twilio apps, and am happy to talk to you about them whenever. My contact details are trivially discoverable.

I’m going to attempt to write a more comprehensive guide to developing Twilio applications, eventually. We’ll see what form that takes — I would really like to provide people an (even) easier way to get started, but at the same time I can’t justify dropping two months of my schedule to write a traditional book on it.

14 Comments

Inaugural Kalzumeus Podcast: Japan, Startups, A/B Testing, And More

Hiya guys.  My good friend Keith and I decided to do something a little different and tried recording a podcast.  We’re still rather new at this, so it took for form of a freewheeling conversation.  Major topics included:

  • the experience of working at large and small tech companies in Japan
  • the Japanese web application market
  • career advice for programmers don’t call yourself a programmer
  • us trying to sell you on starting A/B testing
  • conversion optimization stories, including actionable tips which have actually worked for people
  • a wee bit of generic geekery.
The podcast weighs in at about 79 minutes long.

Podcasts take a metric truckload of work to put together.  If you like it, please, say so.  If folks have interest in it, we’ll do it again.  If not, well, one more data point as to what the market wants.

Podcast Link (M4A.  Click to play, right click to download.  The play feature may not work quite right in Chrome.  Feel free to put this on your iDevice.  (You may find this URL helpful: http://www.kalzumeus.com/feed/atom/ That technically includes all the posts on the blog, but iTunes and similar software will automatically pluck out the audio ones.)

Podcast Link (MP3, for Chrome. Click to play, right click to download.)

 

Transcript (Because We Love You)

Patrick McKenzie:   Hi, everybody. I’m Patrick McKenzie, better known as patio11 on the Internets.  This is my buddy Keith.

Keith:  Hi, I’m Keith Perhac. I live next to Patrick’s and I’m pretty much unknown on the Internets.

Patrick:  So when we tell people we’re right next to each other, we’re right next to each other in Ogaki, Japan. How the heck did we end up here?

Keith:  Long, long story. So I’ve been here for nine years, you’ve been here for eight, pretty much. And we came here on the JET program. I was working as an English teacher, you were working at Softopia, which is apparently our prefecture’s gift to web development and iPhone development right now.

Patrick:  Yeah, the prefectural technology incubator.

Keith:  That was an interesting little incubator because it’s been losing money for the last seven years. And then when the iPhone came out, all of the iPhone developers and everything moved in there because rent is cheap. And suddenly the incubator is making tons and tons of money thanks to iPhone apps.

Patrick:  Yeah. That was crazy. The number one, number two, and number three most popular Japanese iPhone apps of all time were literally right next door to each other, all in Softopia. They’ve been saying that they were going to make Sweet Valley, the name of this region, into the Silicon Valley of Japan. And I always thought it was a pipe dream. And now we have three successful software companies in literally like a 10‑square‑meter space in one building here. [Editor's note: Perhaps not quiiiiite Silicon Valley yet...]

Keith:  Yeah.

Patrick:  It blows my mind.

Keith:  Yeah, essentially we’re in the Kansas of Japan. There’s nothing here, like literally nothing here.

Patrick:  Yeah, the engineering culture of this region is much less startup‑friendly than say a Silicon Valley or a New York would be. It’s dominated by one monopsony employer of engineering talent, which we’ve both had business dealings with.

Keith:  We’ll just say a very large automaker which everyone knows of and who pretty much pioneered the idea of kaizen, which I’m sure you all know of right now.

Patrick:  Yeah, the big buzzword for the lean startup movement was lean manufacturing before it was lean startup and before it was lean manufacturing it was just the [REDACTED] way. Whoo, shoot.

Keith:  [laughs]

Patrick:  Sorry, we’re editing that out.

Keith:  The T way.

Patrick:  Yeah, T. A big T-Corp here. Somewhat surprisingly, T-Corp and its various affiliated industries don’t really apply the T-Corp way to software development, do they?

Keith:  No, and I think the real reason that they don’t do that is because software development is not seen as a manufacturing process. It’s really seen as this kind of artsy‑fartsy kind of thing here. And it’s interesting because Web is definitely seen as artsy‑fartsy, and programmers themselves are pretty much seen as code monkeys.

Patrick:  Yeah.

Keith:  They’re seen as people on a factory floor that can… you just give them the spec and they punch out the pieces and that’s what programmers are for. And so what you have is a lot of managers who are used to managing assembly lines then trying to manage software projects in the same way. And it doesn’t really work quite as well.

And unfortunately, when big T feels that way about programmers, you find out that all their subsidiaries and everyone who works for them and everyone who deals with those subsidiaries and everyone who deals with the people who deal with those subsidiaries, the whole sort of culture kind of filters down to that.

Patrick:  This is true. Even at my old day job, which was a multinational corporation in Nagoya, which there is no company in Nagoya that does not have business with the big T in one way or another. It’s like Detroit times 1,000.

Keith:  Very true, very true.

Patrick:  Everyone is connected with the automobile manufacturing industry here. Even if you’re not directly connected into your day‑to‑day work. And it trickles down into things like the way engineers are treated, the way the discipline is treated, and the prevailing salary and wages for engineers here.

Even at a large multinational corporation as a salaryman, which is a full‑time employee who is expected to be at the company until death do you part, the wages for programmers are not that great here. Would you say the algorithm is $100 per year of age, per month, give or take?

Keith:  About that, yeah.

Patrick:  So Keith and I are right around 30 right now. That means we’d expect at a regular company here to be making $3,000 a month more or less, which as you might have seen is a wee bit less than you guys get in Silicon Valley.

Keith:  Yeah. Now I do have to say, the pricing structure for payments for programmers and for anyone really, is a holdover from the bubble days. And when you retire, you are guaranteed a very nice retirement fund. And every year because the company will do so well, you’ll get a big, fat bonus check, sometimes three times a year. And the problem is that Japan has been in a recession since about what, I think it’s ’90, ’92, something like that?

Patrick:  Yeah, somewhere in there.

Keith:  Long, long recession. And we don’t get bonuses. There’s no more job security for this until you retire at 60, now 65, soon to be 70. So we’ve lost all the benefits of working for peanuts at these companies and yet we’re still working for peanuts. That’s pretty much what it comes down to.

The salaries have not evolved with the change in Japan’s social contract, pretty much.

Patrick:  Mm‑hmm.

Keith:  We had a social contract where you go onto a company and they will take care of you until the day you retire and now it’s you go into a company and it’s pretty much hit or miss.

Patrick:  Now, for the peanut gallery that doesn’t know, the peak of the salaryman/lifetime employment system was in about the 1970s or so. And at that time it was about 30 percent of the Japanese labor force who was both working in a large multinational corporation and had this particular seishain [editor’s note: “full company employee”] status that we’re talking about. The rest of the 70 percent were not subject to the lifetime employment guarantee including, most prominently, women.

Keith:  Women, yeah.

Patrick:  Who we have many, many stories about how Japanese companies do not treat ladies right.

Keith:  That’s for a different time.

Patrick:  Yeah, another time.

Speaking of the social contract, a lot of Japanese society is built around this presumption that if you go to a good school you will get a good job at a nice megacorp like T-Corp and then have this lifetime employment guarantee. That’s one of the reasons it’s so hard to hire for startups out here, because basically you get one shot at that brass ring of lifetime employment and if you don’t take it right after you get out of the university, you are damaged goods for the rest of your life.

Keith:  Well, I do want to cut in there. That’s not really how it works, but that’s how everyone thinks it works.

Patrick:  I agree with both parts of that statement.

Keith:  And the problem is when everyone thinks that’s the way it works, especially for people who have just gotten out of college and don’t know what it’s like in the world, they’re going to go where it’s a safe bet. And we’re talking about lifetime employment and stuff and how that’s no longer a guarantee. However, if you get into a place like NTT, which is the phone company, even the big T, in most cases if you are a proper contracted employee, then you will probably be there until you retire.

Patrick:  Right.

Keith:  However, there are only a handful of those companies left. I would say that there’s less than 100, maybe less than 50 in all of Japan that are large enough to really be able to take care of you for your entire life like that.

Patrick:  And there’s definitely a difference in statuses at those big companies. For instance, one of the ways they manage to have a lifetime employment guarantee is that T-Corp has this… it’s like an industry within an industry of supporting companies, built around themselves, like suppliers, vendors, that sort of thing.

And while they won’t cut their employees come hell or high water, they’ll cut their vendors like nobody’s business and/or tell the vendors, “Listen, we can only afford to pay you a quarter of what we did last year, so make the appropriate adjustments.” And then the vendors will go down and cut their regular employees or, most commonly, cut their contractors.

Keith:  And T-Corp themselves have a lot of contractors. And when you think of contractors, you usually think of consultants or stuff like that. I’m talking about contract employees working on the assembly lines and whatnot. And those people do get cut.

Patrick:  Right, for example, right before the recent economic crisis, which is known in Japan as the Lehman shock for some reason, right before the Lehman shock there were about 6,000 foreigners, mostly Brazilians, living in our town of Ogaki. And so I go to a church out here and after the Lehman shock for about a year, every week at church we had a family X or a family Y or family Z having lost all the jobs in the family is going back to Brazil. So that population of foreigners, who were almost all contract laborers at the local manufacturing industries ,went from about 6,000 to about 1,000 in a little less than a year.

Keith:  That was a heavy, heavy downturn. But that gives you a pretty good idea of what the labor force is like in Japan. And why it’s difficult to get a good startup community here. I go to a lot of programming conferences and stuff and we really, as programmers, picked a really crappy place to be, to be perfectly honest, because the entire area is ruled by T-Corp.  Also, there’s just a very different way of thinking about business in Nagoya in the Nagoya area than in, let’s say, Tokyo and Osaka. And even for Japanese people, and other Japanese programmers, looking at Nagoya they say it’s very difficult for people who are not in Nagoya to start a business here. And it’s very difficult for new businesses to start here.

Patrick:  So we’ve been doing this for the last couple of years and we know how it actually works, but people in America always tell me I’m lying when I talk about working conditions for professionals in Japanese societies.

Keith:  [laughs]

Patrick:  Why don’t you tell me about those lies I’ve been telling for the last couple of years.  Just sketch some out.

Keith:  So I’ve heard some of the lies Patrick has been telling all of you and I have to tell you that I don’t think any of them were actually even slight exaggerations.

Patrick:  Well, let’s recount some stories, shall we?

Keith:  Let’s recount some stories. So coming home every night at 12:30 AM, if you could get home.

Patrick:  Right.

Keith:  OK, that’s definitely true.

Patrick:  Last train of the day is 30 minutes after midnight, I missed that train quite a bit of the time.

Keith:  Yeah, the business hotel near the station actually knew Patrick by name and face. And Patrick had this thing Tuesdays where there were longhalf‑day meetings.

And so you get pretty much nothing done on that day. So you’d have to work until about 12:30, 1:00, end up missing the train and he would arrive at the business hotel and they would have a room ready for him. And if he wasn’t there one week, then in the following week when he did show up, they would ask, “Why weren’t you here last week?”

Patrick:  It got so bad at one point I left my Kindle there overnight on a Monday and it was waiting at the front desk for me the following day with a note, this is Mr. McKenzie’s Kindle, he’ll be in, it’s a Tuesday.

Keith:  So really, no exaggerations about the workload. And I, fortunately, was not as bad as him. I could usually get home every night. I usually got home around 11:00 or so. So I was very lucky to get home at 11:00.

Patrick:  So what are we doing for this 12‑to‑19‑hour day? Is it actually programming for the entire time? No, not so much.

Keith:  OK, so Patrick and I had very different employers. He worked at a very large, how many people were in your company?

Patrick:  Well, let’s see, at my office there were about 70, but companywide maybe 1,000‑1,200 employees or so.

Keith:  So he’s with a very large company that does only programming. I, on the other hand, was in what is pretty much the Japanese startup.  A startup is called a “venture” here. It’s generally a single person putting his money on the line, getting a loan from the bank, and trying to make a company out of it. I was the only programmer at this IT company. So we were a company of around seven or eight people, and I was the only programmer left after my boss, the other programmer, quit.

So pretty much I had a lot of free rein in what I was doing, but, on the other hand, for any product that my company had to ship, I had to make it from scratch. And that’s not just programming, that’s the design, development, graphic design, figuring out what the customer wants, how we’re going to accomplish it, pretty much everything. The other people in the company were sales and management.

Patrick:  My company was a multinational consultancy.  Think of like an Accenture or IBM, which goes to a client and say, “Hey, we’ll fix your problem, whatever the problem is, and we’ll sell you smart people to make it happen.” We were in the wetware business. I was the least wet bit of the wetware, we’ll put it that way.

My official title was “System Engineer.” That’s one of these things in Japan, programmers are considered code‑monkeys, but system engineers, who tell programmers what to do, are assumed to have a bit of discretion and professional ability. Maybe half the company was people in sales or support, who would sell a university on, “You should use our systems for doing your backend course management or payroll or whatever.”

Then the other half of the company was the engineers like myself who would talk to the customer and figure out what they actually needed for integration with their systems and then build out the code to do it. Or assist outsourcing operations for outsourcing the business process management to low‑wage countries like India.

The feeling in the company was that even though we were a software company, writing software did not add value. That’s almost a direct quote from the CEO. So, we would figure out the software that needed to be written and then actually get the software written by cheap people because that would provide more value for our customer.

This resulted in me doing outsourcing management to India for about three years because I was the only one in the company who could speak good enough English to manage it. That was an experience.

Keith:  You had actually told me about some of the technical docs that they had sent before you were on the project where they would actually put the Japanese technical docs into Google Translate and then just ship them over.

Patrick:  Yeah. I tried to stop that and was less successful than I wanted to be. At one point I told my boss, “Listen, boss, we’re shipping out several thousand pages of technical documentations for this particular subsystem which just absolutely puts the ‘critical’ in ‘business critical.’ You need to clear the next month of my schedule so I can translate them.”

He said, “Your fully loaded cost as an engineer is about $6,000 a month [ salary of $3,000 times a factor of two for contribution, taxes and other overhead involved with hiring me]. $6,000 a month is too much to pay for translation. We want to get it done for about $4,000, so we’re going to have an Indian company to do the translation for us.” I said, “I don’t really love that plan, but I’ll do it. But I want to spot‑check the quality of their translation just to make sure we’re getting our money’s worth.”

The first day I get back an Excel file. I can’t remember exactly what the error was, but on the first page that was describing black as white, a clear error. I mail the Indian subsidiary and say, “Hey, I was spot‑checking the quality of your translation and there’s just this glaring error on the first page that makes me worry about the rest of the documents. Can you please tell your translators to be careful.”

They mail back to me a few hours later, “We don’t agree this is an error.” It’s literally a translation of black is white or something. So I mail back, “This is absolutely an error. I haven’t had the opportunity to discuss it with a native speaker of Japanese, but there’s no possible way that your translator is translating this correctly.”

I get a mail back, “We don’t agree.” I said, “You need to put your translator on the phone with me now because we need to discuss this. The project will not proceed if you have this level of understanding of what the requirements are.” They said, “We discussed with three authorities and all of them agree with our translations. So you’re outvoted.”

I said, “Who are your three authorities?” They said, “Google Translate, BabelFish and AltaVista Translate.” So I was outvoted by three computers who all use the same dictionary apparently. That sort of thing happened all the time. They had billed us literally $4,000 for having people do the manual copy‑pasting work into Translate and billed it as translation services. Yeah. That project didn’t work out very well.

Keith:  I actually want to jump back real quick. You had mentioned the difference between the System Engineers and programmers. You had a blog post, I think it was two weeks ago and it got put on Hacker News. Everyone was like, “Oh, I’m a programmer, so that’s what I’m going to call myself. I’m not going to call myself System Engineer.”

This is really one of the places where it’s a very clear‑cut difference. It doesn’t matter what it is that you do all day. A simple name change raises your pay grade pretty much double. The amount of things that you are trusted with, the amount of things that is decided that you do. This is not to say that programmers are not given system engineer responsibilities, they’re just not believed to able to accomplish them.

Patrick:  Right.

Keith:  So, it doesn’t matter what you’re doing in Japan, as long as you’re a programmer, call yourself a system engineer and you’d double your pay.

Patrick:  Right. Actually every time I had a bug for about the first two years of working on a company, often because I didn’t quite understand the requirements in the document because all the documents are in highly technical Japanese. And my co‑workers would be berating me for the bug or the absence of testing, which failed to reveal the bug. Or just being careless, because I’m often careless. They would say, “Listen, Patrick. This isn’t acceptable, you are not a programmer. This should have been taken care of.”

Definitely with regards to Japan, call yourself a system engineer if you have the choice, even with regards to America.

I do programming, I like programming, I love programming. The craft and actual activity of programming really, really appeal to me. But as a consultant when I’m talking to businesses, I never describe the value that I’m adding to the business as, “OK. I’m going to program some stuff for you in Rails.” Because businesses don’t really see it on that same level.

Even businesses that are run by really engineers’ engineers, like the Fog Creeks of the world, which really love programming and the craft of programming: at the end of the day they are a business.  So you need to be connecting to them on that business level of, “Here are the concrete benefits you’re going to get from doing this engagement with me.”

Keith:  It’s just like when you sell a product, you don’t sell the features, you sell the benefits.

Patrick:  Right, right.

Keith:  You are programming, and that’s the feature that you’re selling. But that’s not the benefit. The benefit is not, “I’m going to write code for you.” The benefit is, “I’m going to make a system that does this awesome thing. And this awesome thing is going to make you a million‑up in dollars.” And that’s what you sell. That’s why you call yourself a system engineer instead of a programmer.

Patrick:  Or providing solutions instead of providing software.

Keith:  Solution Architect. It was so funny. My job title in my company has changed so many times. I think it changed seven or eight times in the last four years. And my favorite one was “Solution Architect,” because it just says nothing about what I do whatsoever.

Patrick:  But civilians, or whatever you call people who aren’t engineers, sometimes respond to things like what is printed on your business card. That being the reality, adjust yourself to the reality rather than what you think your ideal world would look like.

Keith:  Yeah. Really, if you want to put “programmer” on your business card, write it in binary on the back or put it in as part of the design or something.

Patrick:  Right. You can always call yourself what you want when you’re around just us geeks, but when off Hacker News and you’re talking rate negotiations with customers, adjust to their view of the world.  It will make you happier in life.

Keith:  Exactly, exactly. Don’t believe us, A/B test it. I actually did this. I went to a prospective client with a programming friend of mine. We do the exact same thing. We have the roughly the same levels of technical and business acumen. We decided, “Hey, we’re going to try this out.” He called himself a system engineer, I called myself a programmer. For the rest of our meeting they deferred to him over me for every single thing. [Editor’s note: You’re right, this is not an A/B test.]

So just changing that one little word on your business card or when you introduce yourself makes a huge difference. It’s really just you have to change yourself to fit your customers’ expectations.

Patrick:  Obviously the culture of businesses in Japan is very different than what many of our listeners are used to.  Well, I say the culture of Japan: Japan is a big place. It’s like saying, “the culture in America.” Kansas is a different place than Texas, is a different place than a tech startup specifically in Mountain View, California. But the business culture of, say, metropolitan Nagoya, Japan is very different than the business culture of New York City, or at a tech startup in New York City, or at a financial services firm in New York City, or specifically Goldman Sachs in New York City.

You have to adjust yourself to whatever that particular situation you find yourself in. That being said, there’s very, very few places I’m aware of calling yourself a programmer is a career enhancing move.

Keith:  Yeah. Let’s stop berating everyone for calling themselves programmers.

Patrick:  Right.

Keith:  What do you say? You want to move on to kaizen?

Patrick:  Sure. Let’s move on to kaizen.

Keith:  OK. Everyone, I’m sure, knows kaizen.

Patrick:  Actually, I don’t think that’s true.  Why don’t you explain what kaizen is.

Keith:  Really? Because I got a photo from one of my friends in Nashville of all places. She went to a burger joint and had the Kaizen Burger. This huge burger with Kobe beef and cheese and all the stuff, and they just called it “Kaizen Burger.”

Patrick:  That sounds delicious, but I think the rest of America is not caught up to Nashville yet. So let’s go.

Keith:  Everyone should catch up with Nashville, I swear. [laughs]

OK. Kaizen is a term in Japanese which means “improvement.” That’s the general translation. What it has come to mean in American English is the reiteration and iterative improvement of systems and processes that were founded pretty much by Toyota. Essentially it’s reiterations over a design in order to get rid of flaws and to make a better product.

This is pretty much the the cornerstone of any physical manufacturing process. Any car that you see is built with kaizen. Apple does it with their Macs. Everything is iterated and iterated. They make a design, they put it out there, they find out what’s wrong and they slightly change it, they slightly fix it and they keep reiterating and reiterating. Then they finally get a great product. That’s how Toyota has worked, that’s how most manufacturing works.

Patrick:  I think this idea is so powerful. There have been hundreds of volumes written about the Japanese economic miracle. Honestly this and the related Toyota process improvement were so powerful that they were able to cover up every mistake made at the same time. If you have that going for you and the rest of the world hasn’t caught on too it yet, you can do all manner of stuff totally stupidly, like work your people to near death by exhaustion. You’ll still ROFLstomp over the competition.

Keith:  Exactly. Where kaizen comes in to our conversation now is that kaizen has left the physical manufacturing world and is starting to go into the software development.

Patrick:  If you read Eric Ries’s book The Lean Startup, which by the way is a fantastic read and you should read it if you haven’t already, it’s one of the ideas he touches on a lot.

Keith:  Yeah, yeah. I actually want to pick up a copy of that. I saw that in your house when I was over there.

Patrick:  One of the ways that kaizen is relevant, not just to people who are manufacturing physical products but also to software developers who are making software or an experience delivered over the Internet, is that we can do things like A/B testing. So we can tell in almost real‑time which of multiple versions of a website, once exposed to users, causes those users to have a happy experience. You could define “happy experience” in a few ways: judged by business goals like whether they buy more software or sign up for your email list more often, or judged by whether they achieve success with your software.

For example, I sell software that makes bingo cards for elementary school teachers. If the schoolteachers can’t successfully make bingo cards, that’s a problem for me. So, I can redesign the interface of my software such that they actually achieve task success.

Keith:  The great thing about this iteration, when we’re dealing with software and especially web apps, is that the feedback is so immediate and the cost for doing these iteration tests, these split tests is so low. How long does it take you to make a split test?

Patrick:  One line of code, it’s the only way to do it.

Keith:  One line of code. All right. In one line of code you can test which creates the better user experience.

This kaizen originated in manufacturing. In manufacturing, it’s a much more expensive process. You have to build it, which means you have to use the raw materials, you have to create the building process, then you have to ramp up production and ship it to the customer and the customer has to give you feedback. We’re talking, maybe, two or three years to get feedback on a single design before you can iterate.

Now, we can iterate… If you have enough users on your site, you can get an answer with a split test in about a day.

Patrick:  At Facebook or Google scales you can get it literally in seconds. Even at fairly minor scales, like the Bingo Card Creators of the world, I can throw out a test every Monday and have a statistically confident results generally by the following week. It’s a wonderful thing when you don’t have much time to work on a site.

If you’re getting about a hundred visitors a day then you can throw some up, come back a week later, and have results.  Doing the A/B test doesn’t block any of your other activities for the week. You can continue talking to customers or continue writing documentation, continue producing the new feature. Then, when you’re ready for it, take a look at the results of the A/B tests and generally just make a one click to pick result A over result B or whatever the stats tell you to do.

Keith:  What we’re actually seeing… Patrick’s A/B testing software, he made it himself. It’s called A/Bingo. It’s amazing and you should pick it up. I use Visual Website Optimizer, which is a great software, all JavaScript based, works on any page, I recommend that as well.

But what we’re seeing is a change from manufacturing where in manufacturing you can have a thousand ideas, but your iteration time is around maybe six months at the fastest, a year, maybe two or three if you’re talking something huge like a car.

With the software iterations, one line of code takes me five minutes to implement. VWO does the same thing, it takes me 5‑10 minutes to implement. What takes more time now is to actually think of what we’re testing. The time it takes to test something is so infinitesimal that thinking up good tests takes more time. Which is amazing that we’re in this technological area that we can test things as fast as we literally think them up.

Patrick:  Right. Also it prevents a lot of waste in the company. Have you ever been a really pathological meeting? I once had a six‑hour meeting involving four people to discuss the text that was going to be placed on a button for signing up to a mailing list. The mailing list was only getting 50 sign ups a week anyhow. There’s an actual cost to that. The fully loaded cost for that meeting was on the order of $300 an hour times six hours, so almost $2,000 spent just to determine the call‑to‑action on a button that didn’t really matter anyhow.

If you have a culture in your organization of testing things, that would literally have been a 45 second discussion. Like, “I think it should be, ‘Sign up for mailing list.’” “I think it should be ‘Sign up now.’” “OK. We’ll throw it in the test.” Done. And you save that $2,000 and spend it on stuff that actually matters to your business and your customers and the world at large.

Keith:  Yeah. Going back to the Japanese side of things, this is a big thing where they don’t consider software to be a manufacturing process. Testing in Japan… Now, we have unit testing and testing like that, but the idea of user‑based testing is pretty much non‑existent, almost completely non‑existent. Unless you’re going into the very large, very in‑the‑know tech companies. I’ve dealt with one or two, and they’re not common at all.

Patrick:  For example, one of the several big startups based in Tokyo are in the mobile gaming space. There’s an American company you’ve probably heard of, they allow you to make virtual cabbages and water the virtual cabbages. Sometimes the virtual cabbages rot.

This company is very, very sophisticated and they iterative improvement processes including A/B testing and collecting user metrics and what have you. They recently opened a Tokyo office and somebody there said, “Yeah. We’re pretty much planning on effing killing all the Japanese companies in the this space.”

I do not love that company, but I think they’re likely to achieve that goal.  Just like Toyota versus the American manufacturers, if there are 10 people in a fight and only one of them understands how to use science, I already know who’s going to win.

Keith:  I like that science is the thing that’s going to win the fight. [laughs]

Patrick:  It seriously is the revenge of the geeks out there right now. Really.

Keith:  Really it is. For the first time in history we have so much data that we don’t have enough people to process it. We have so much information that really what we need to do now is just process all this great information and put it to good use. And we just don’t have the resources yet.

Patrick:  Right.

Keith:  What we need is mathematicians and statisticians to find, look over, and analyze this data.

Patrick:  Yeah. Also, this is a good thing for you to pick up for your career. A/B testing is a skill set that is not really all that hard to pick up. Grab a book, study it for a week. The mechanics are dead simple. After that, it’s just a matter of coming up with what actually to test.

But that skill is fantastically rare, much more rare than, say, skill with creating web apps in PHP or Ruby on Rails. Because it has direct bottom‑line implications to the business, to the tune of, “What’s five percent of your sales for the next quarter?” When you’re talking to a company the size of say a T-Corp or just the line of business from Bank of America’s website, five percent is a very significant number.

If you can routinely quote numbers like that, you are no longer in the $60‑to‑$100‑an‑hour bucket with other programmers, you are in the bucket with strategic initiatives that add five percent to Bank of America’s bottom line. Scary stuff.

Keith:  I really hate to keep coming back to this. System engineer versus programmer, implementing an A/B test takes one line of code, takes five minutes to do. If you are the person that used that one line of code, changed the color of a button from blue to red, and added five percent to a multi‑million dollar company’s bottom line, you’re no longer the programmer who implemented that, you’re the guy who added five percent to a multi‑million dollar bottom line. That’s what matters.

Patrick:  Right. Billing also scales right to match, particularly when you get clueful clients. I won’t be comfortable talking about mine. Suffice it to say if you have a portfolio website… Not that I have a portfolio website.

Keith:  You need to make one.

Patrick:  I don’t know. Yes and no. When someone comes to me and says, “Hey, we’d like to hire you for this thing, do you have an example of people that it’s worked for before?” I can informally say, “Well, I’ve worked for XYZ, and XYZ have reported this kind of result.”

That’s, by the way, a trick when you’re dealing with clients… It’s not a trick because both of you will benefit from this. But say, “Look, if we have a successful result with this project, I would like to write that successful result up as a blog post or something. That will be to our mutual benefit, you’ll get additional attention because of me writing the blog post, and I’ll get something to keep for my portfolio.”

Phrased like that, many clients will say yes to it. It is a wonderfully, wonderfully wonderful thing to arrange for yourself as a consultant. For example, when I was working with Fog Creek, when it became obvious that the engagement was going to work out for both of us, I said, “Hey, you guys are always looking for new topics for your blog post, why don’t let me ghostwrite one about what we did for the marketing?”

They said, “Oh, yeah. Let’s do that.” So I wrote something called “Our Marketing Is Up Fog Creek and How We Fixed It” or something to that effect. Where we talked about things like we made changes to the website and as a result Fog Creek’s conversion rate went up by, say,  10 percent. Just make a guess of how much money Fog Creek makes in a year, add 10 percent to that, it’s meaningful, right?

So, Fog Creek is obviously very happy with this. But in terms of dealing with my other clients, when they say, “Oh man, Patrick. That number you just quoted to me is a lot of money. How do I justify that to the investors or how do I justify that to my team?” I say, “Well, if you look at this post over here by Fog Creek, they said I added about 10 percent.” And done, that’s one way to deal with pricing objections.

[Editor's note: Patrick wasn't clear on this, but 10% was a handwavy approximation number here.  The actual number is undisclosed.]

Keith:  And that’s something I’ve been working on with. We were talking earlier about how this split testing your stuff is not known in Japan. I’m trying to bring split testing and  modern iterative development practices into Japan.

One thing I found when dealing with my customers is that they feel the same way, it’s like, “How do I add this to my bottom line?” I say, “Well, I improved such and such company, who makes a million dollars a year, by 10 percent.” Sometimes they understand it and sometimes they don’t.

When they don’t understand it I say, “Well, why don’t we do it like this. You pay me half of what I’m charging. And for every percent improvement we see over your current baseline, you give me another X dollars.” People seem very happy about that. Until, of course, the results come in and they end up paying me three times to what they were originally going to pay me.

Patrick:  Yeah. Pay‑per‑performance is an interesting model.

Keith:  It’s not the best.

Patrick:  Yeah, I am not a big fan of pay‑per‑performance arrangements. Not to say I would never do them, but I typically try against them.

Here’s one reason. In any sort of project you have execution risk. Generally as a consultant you want to be responsible for your own conduct, but not risks external to your own execution. If, for example, you want to do a pay‑per‑performance arrangement for search engine optimization or for A/B testing or conversion optimization for a startup, that startup’s priorities might change on a dime.  This can happen one week after you get out of the building. That actually happens frequently. The project you were working on can get shelved.  This even happens at multinational corporations.

If this happens, you’re not going to make your pay‑per‑performance numbers no matter how good the quality of your underlying work. Typically, with a pay‑per‑performance arrangement you’re going to be getting a smaller number upfront, plus a payment if you hit particular milestones. You won’t get that milestone payment and you’re going to end up working for a fraction of your rate. And working for a fraction of your rate is not the purpose of the exercise. Unless it was client who had a clear reason not to pivot on the…

Keith:  Right. I should have prefaced that with saying that these are clients that I have had worked with before in my previous companies and that I do trust. And who I know are not going to shelve the entire project and I’m going to get put in the works.

These are things that part of the contract of doing the performance‑based payment is that I’m going to be setting it up and they are going to be putting it out there and that it is going to run. Like you said, it’s not a 100 percent, nothing’s ever 100 percent. But I would never do it especially for a large company that I’ve never worked with before, I would never do a performance‑based review. I don’t think.

Patrick:  Right. Engineers have a funny notion of what the price of things is. If I say “$3,000″, that sounds like a lot of money for us because, hey, we worked for months for that. But $3,000 for a real company that has real assets and real employees, that’s literally below their capability of measuring stuff. It isn’t a line item on the annual report, it isn’t even in the appendix to an appendix on the line item of the annual report. You really have to get into the five‑figures to even hit on the radar screen at all.

Given that, if you were thinking, “OK. We’ll could charge a variable price between $3,000 and $15,000, or just charge $20,000 fixed,” there’s a lot of businesses that will say, “Give me the higher $20,000 price, because that will make my life easier. Getting a variable charge approved through the accounting division or my higher‑ups or whatever is going to be more difficult, because that’s not our established process.”

If it is just a fixed  $20,000 coming out of budget X, well, budget X has hundreds of thousands of dollars in it because there’s employees in there, too.  [Editor's note: Employees are very expensive.  There is a cynical but true saying: "Overhead walks on two legs."] So $20,000 out of that budget is not difficult to approve.

Keith:  Right. Working, not with T-Corp in particular, but with other companies of that size, we would have our sales reps on their side come to us and say, “Our budget is X amount of dollars. Make it under that.”  That’s not their maximum possible budget, that’s the budget they have before they need additional signoffs.  You could spend within $1 of that, and it would be no problem. However, if they go over that line, they have to get the sign off of their manager or, god forbid, the vice‑president or someone. As long as it’s under that number even by a dollar, they can sign off themselves and that makes their lives so much easier.

Patrick:  You were saying something interesting earlier, which was that you hope to bring in A/B testing and these modern techniques into Japan. That’s something I’ve heard a lot in working with my other companies, because the Japanese engineering, with specific regards to, say, Web engineering, is kind of lagging a couple of years behind the U.S. right now, right?

Keith:  Oh, so far behind.

Patrick:  Let’s talk about the status things. So the Joel Test is getting almost 10 years old now.

Keith:  Is it 10 years already?

Patrick:  Almost 10 years I think. [Editor's Note: Even older!]The Joel Test is about basic things like source control, having a quality department, yada, yada. Japan, not quite there yet. So I worked with a company that worked with other companies. We had source control, thank God. Subversion, though, not git. Sorry, I didn’t mean to be a snobby geek there. Subversion is an excellent source control system. It’s heads and tails better than no source control system, which is the competition here in Japan.

Similarly, we write big freaking enterprise applications in Java with homegrown web frameworks, which were absolute pain in the butts to work with. One of the reasons I have like 60,000 Hacker News karma was there was a compile and deploy step that literally took like five minutes to run. So if I wanted to change, say, the number of columns on one particular table on a page, testing that would take upwards of an hour.  Oh, whoops, I had a one‑character bug in my HTML, rerun the compile. It’s going to take five minutes. Yada, yada.

The amount of friction involved in that process destroyed my productivity.  [Editor's Note: It didn't help that I'm fundamentally a lazy git.] However, we measured productivity by number of engineering hours expended, so by that metric I made my department look very good. But modern web frameworks like Ruby on Rails, Django, all the fun stuff, make life so much better.

Keith:  It’s really ironic that Ruby was created by a Japanese man, and yet Ruby on Rails is pretty much unknown here.

Patrick:  Maybe this will change a little bit. I think EngineYard has hired Matz, right?

Keith:  Right. Was it EngineYard ? I thought it was Heroku?

Patrick:  Yeah, it was Heroku. Sorry. Heroku by the way, is the most Japanese name on an American company I’ve ever heard. [Editor's note: I heard at Startup School that they were explicitly inspired by the Japanese aesthetic... in case the samurai and Godzilla on the pricing page had not clued you in yet.]

Keith:  Yeah. I know. But actually there are Rails conferences here and it’s getting more and more popular. The problem is that web apps themselves are still not that popular here. I think part of that is done in by the fact that, there are maybe four or five countries in the world that have IE6 shares above 25 percent.  [Editor's note: I don't know about this one, but wasn't sure enough to correct him.]

I believe they include Thailand, China, and Japan. China actually has I think 23 or 22. It’s really quickly dropping. But Japan still relies on IE6 so much. And the reason for this is that back when IE6, when XP came out, we were still, not really in the recession, but we still have a lot of money as a country. And so everyone had their IT departments, and everyone had these huge reduxes and they made all their systems compatible with XP. And it was great and wonderful. And then the money starts drying up, and the first thing to go is the tech departments.

I’ve worked with couple large companies that literally had outsourced their entire IT departments. And so there was no one at the company anymore who could even do anything. Let’s say they needed a new computer setup or new proxy or one of their NATs died or something, there was no one at the company who could fix that.

Patrick:  That brings up a great topic. You’ve probably heard that engineering productivity is worth pretty much any amount of money relative to engineering salaries.  Everyone tells you to get two big wide screens, as it’s only 400 bucks extra.

Despite my previous employer being a software company, they had outsourced the internal IT. So then rather than buying each programmer a computer, they would lease a computer through blah‑blah‑blah, whatever, for 80 bucks a month. And by default, it was a years‑old computer with, let’s say, one gig of RAM, a processor from 2003, and a 14‑or‑15‑inch monitor for a desktop system. And there was literally no button you could push at the company to get a better computer than that.

Keith:  Once the rules are set, it’s not worth the bureaucratic hassle and even the time to think about to change it in a company. What it does, in the end, is it just hurts everyone.

Patrick:  Another factor about web apps in Japan is that a huge portion of Japanese use of the Internet is not actually on real browsers, it’s on the lovely fake browsers that you find in old pre‑iPhone Japanese feature phones. That’s an entire topic in itself. Japan, hardware‑wise, were probably the most advanced cell phones in the world until…

Keith:  Also software‑wise. When we were still dealing with WAP and wedge TML and WEKO, WACKO, WACO, WACO, whatever the hell we had back in early ’90s and stuff, Japanese cell phones had iMode, which is a full-featured HTML subset, and they were browsing websites in ’91. On their cell phone.

Patrick:  Websites with pictures and JavaScript. It was amazing. And then it didn’t really change for like the next 15, 16 years.

Keith:  Yeah, they’re still browsing the same Web pages 20 years later.

Patrick:  Yeah. They call this the Galápagos effect. Like the Galápagos Islands, which have a variety of different finches that are found nowhere else in the world, Japan had a variety of very advanced hardware platforms found nowhere else. The software for each of the hardware platforms would be literally handwritten assembly each time a new model came out. That was the dominant way to do cell phone development, with improvements like, “OK, we’ll put a JRE on it and allow you to run Java code.” But that was the dominate cell phone paradigm up until the iPhone came out. And the iPhone proceeded to ROFLstomp over the Japanese carriers. Right?

Keith:  Pretty much. Pretty much. And now we have Android phones as well — a lot of Android phones. The problem is, and I think it’s pretty much the same in America right now, is that you don’t get the updates. The Galápagos model is still the common way of thinking for any cell phone besides the iPhone. So you have this Android‑based amazing phone that is outdated in one to two months. You will never receive an update for it through the life of the contract.

Patrick:  Just a feature of the Japanese consumer electronics market. People tend to upgrade devices very quickly here.

Keith:  Very quick. Very quick.

Patrick:  Japanese cell phones are marketed under the assumption that the core of the market updates their cell phones at least yearly.

Keith:  Yeah. In America, because everyone is on the two year contract, the companies assume that you’ll update your phone when you’re two year contract is up. And in Japan, even though there’s a two‑year contract, most people update their phone, I would say, once a year. And depending on how trendy you are, even more.

Patrick:  One of the main early adopters for the cell phones in Japan, and I swear I’m not making this up, high‑school girls.

Keith:  Oh, yes.

Patrick:  Their use of the phone as a mobile/Internet platform drives quite a bit of the development here. I can never remember which of the companies it was, but a couple of years ago when smartphones were starting to get popular, one of the companies was literally pitching them as a fashion accessory item.

So in addition to pitching, “It has a camera. You can take photos with your friends. And here’s what the software does.” They would pitch like, “And here are the varieties of things you can stick on the phone to make it look different but similar to your girlfriends’ phones.  This way, you can be trendy in your own individual-but-not-too-individual way.” [Editor's note: If you think I'm cynical about the social motivations of teenage girls, you should hear me talk about geeks.]

Keith:  Right. And they’re starting to do that with the Androids now. So each of the Androids has pre‑installed software for each kind of clique for the Japanese high‑school girls. It’s really interesting to look at. It is a bit depressing as a programmer. It’s like, “You could just customize the wallpaper yourself and not have to pay the $600 for the new phone.” But, oh well.

Patrick:  Yeah. Meet the customers where they’re at. Right?

Keith:  Right. Right. Do we want to go on to websites as… I still not have been able to figure out how to really put this other, not in Japanese.

Patrick:  Well, say it in Japanese and I’ll translate for you.

Keith:  Now I’m going to forget it in Japanese. What was it? Ore ore homupeiji kind of thing.  [Editor's note: The rough sense of this: "Me-me-me home page!" ] So what it really is, so you have the ore ore framework.  It is the framework that you built, and you love to death, and is not used for anything at all.

So the web page is pretty much in a lot of cases… OK, here’s how I’m going to say it. The business card web page, I think really is the best way to put it. [Editor's note: Brochure site, maybe?] A business card web page that doesn’t accomplish anything first of all. And in addition to not accomplishing anything, is made in a way so that the president of the company feels vindicated that he has spent enough money on his web page.

So even if they have like a goal like get customers or tell customers about our company, it’s not something that they’re working towards. They are not investing any time or effort into the home page other than saying, “Oh, we need a home page. I spent $6,000 on a designer to design me a home page in Dreamweaver.”

I think we should talk about those.

Patrick:  Let’s talk about this.

Keith:  Yes.

Patrick:  So we’ve got this idea of having a brochure website for maybe an offline corporation or even for companies that are SaaS Internet startups what not. The home page is not designed for the interests of the business but rather for interests of stakeholders in the business, which are two very different things.

Man, I’ve seen home pages that were literally battlegrounds. There was a war between the engineering department and the marketing department and both sides lost. Engineering said “We want to build features. We don’t want to work on any of that crap on the home page that doesn’t require any technical expertise whatsoever.” Marketing said  “We have this message and this brand that we want to get across.” The result of that war was a page that didn’t convert anyone and wasn’t clearly owned by any group in the company.

There are excellent companies which have little dysfunctional issues like this.  Engineering’s job is to ship more features to the customer. Marketing’s job is to brand or whatever their magic stuff is. But nobody’s job was to measure the conversions on the page, or to increase those over time.

I’ve gone into very smart, successful, well‑managed companies and asked questions like “So, how many trials did you guys do last week?” And the answer would be, “We don’t know. There’s probably a SQL query we can run somewhere that will tell us that.”

Keith:  Yeah. I had a similar thing. This was actually very eye‑opening. I did a reservation system at a really cool place.  They have online and phone reservation systems.  The phone reservations have a backend to their online system, which the front desk uses to place reservations.

So I could see, when I was doing the analytics for their site, how many reservations they had but not where they came from. I couldn’t see if they were self-service reservations online or assisted reservations over the telephone. So I asked the person in charge. I said, “What percentage is online? And what percentage is phone?” He said, “I think at least 80 percent of our reservations are from online. I only think 20 percent are from the phone.” I said, “Wow, that’s much more than I thought. I thought that the phone would be much, much more popular. And he said, “Yeah, but those are the numbers I think.”

So I installed Google Analytics, and I dug into the numbers. It turned out that less than 20% of their volume was online, and more than 80% was on the telephone. So they had no idea of even the number of reservations that were coming in and where they were coming from. And doing a little more analytics, I found out that those 20 percent were the customers who had survived through the reservation process.  The system was difficult to use and the conversion was horrible: only about 15% of people who started a reservation actually got done with making one. That was very eye‑opening for them.

Patrick:  It’s easy to laugh at them in hindsight, but I guarantee if you run a business right now, there is some number that is key to your business that you are not tracking and that would be very eye‑opening to you. The one that I routinely bring up for clients is, what percentage of people who use the software/free trial/etc stop using it after the first time, versus what percent use it at least twice?

The number that use it at least twice is almost always depressingly low. When I started tracking it for Bing Card Creator it was only 40 percent of people came back a second time. These days, it’s up to 60 percent, which a 50 percent lift, did wonderful things for my business, but still 40 percent of people would not even give it a second look.

40% is a decent ballpark number for people who don’t track this.  That has been fairly consistent across a lot of clients of mine. No one ever gives you a book that says you should be tracking this because it’s abysmal right now and you should improve it, but if you’re not tracking that, I guarantee you it is probably abysmal right now and you should probably try to improve it.

Keith:  You should write a book about that. You should just write a book about these are the things you should be tracking, why the hell aren’t you?

Patrick:  Yeah, everybody tells me I should write a book, but writing a book is a terrible idea.

Keith:  Maybe a pamphlet. [laughs]

Patrick:  It’s funny. If I write a blog post people will read it and maybe comment on it for Hacker News. I know it would be forgotten the day after. Ideally, if I wrote a book it would be referenceable by people. Despite the underlying information being essentially the same product, if you put 2,000 words in a book it is perceived as having value, but if you put 2,000 words in a blog post your readers are going to be “TL;DR dude.”

That is the theory at any rate.  The reality is that putting 2,000 words in a book is a terrible, terrible use of my time business‑wise.  Even if I sell the book (or e-book) at $20 or $50 or $100 a piece, and sell hundreds of copies, that is not a meaningful amount of money relative to getting one of my happy consulting clients on the phone and saying, “Hey, I have an idea that I want to try for you guys. Hopefully it will do as well as the last time. Let’s get this going.”

Keith:  However, one thing you might want to try is even just a blog collection and get a physical, printed book. One of my clients, he’s actually considering giving his book away just because of the amount of press it gets him. We did some welcome page tests.

The first one was a picture of him, where he graduated from, and the things that he’s accomplished.

The second one was a big old picture of his book on the “New York Times Bestseller.”

The third one was “I’ve been on NBC” with a video of him in NBC.

The one that says “I’ve done all this” did fairly OK. The one that says “I published a book that was a bestseller” I think was 100 percent better. 100 percent better than that was “I was on TV.” Pretty much once you get on TV, people will buy your shit, no matter what you’re hawking. If you have a clip of you being on TV, they will buy anything you want.

Patrick:  This, by the way, is hilariously underexploited by engineers and people in the startup community, but it’s well known among direct marketers.  After you have press about you, if the New York Times blurbs about you, even one sentence worth: in all your copy from that point out you put the logo of the New York Times and you say “as appeared in theNew York Times” and that will increase your conversion rate up the wazoo.

You’ve suddenly jumped from “A website that I’ve never heard of and don’t trust” to “Endorsed by the paper of record.” You will routinely see this on some of the skeazier direct marketing pages. Even if they haven’t actually been on the New York Times, they’ll say the product category’s been endorsed by the New York Times.

For example, if they’re hawking a social network, they’ll say “As seen in the New York Times.”  They haven’t actually been reported about, but a social network — probably Facebook — has been reported about. I wouldn’t be happy about doing that. That said, if you’re a Fog Creek or a company with  is legitimately high enough profile to getink from the New York Times, that should probably be on the website somewhere.  Fog Creek has even had their office written about before. [Editor's Note: Like Lonely Island says, "Doesn't matter, still counts."]

I don’t know if that is on their website, actually.

Keith:  You should just talk to Joel about that. One thing I do want to say you don’t want to put on your site is “as seen on TV.” Never put “As seen on TV”, especially the red circular logo with all the explosions that you see on $50‑dollar abs stretchers or whatever. That’s not the best thing.

Patrick:  Yeah. People may have cottoned on to the fact that that means they paid for an infomercial. However, retrofitting an explanation like I just did is witchcraft. Have your intuitions and have your thoughts about what would probably work for the site. Like we mentioned earlier, it’s so cheap to be test these things, you should be continuously testing them.

Keith:  Even if you have a banner space on your home page, just switch out the banners. Do an A/B test with that, switch out the banners, see which one performs better. It’s so simple.

Patrick:  It’s so simple and it’s absolutely crazy the amount of lift you can get for an established business that produces value and has been doing so for a decade just by switching out banners.

Keith:  Speaking of which, are you going to tell about the recent A/B test result you got from Bingo Card Creator?

Patrick:  I should tell about the Bingo Card Creator story, right? OK. I went on a six‑week America tour recently, partly to drum up consulting business, partly to go see Silicon Valley and meet the people I’ve only been reading about in Hacker News, and partly just a change of place. So I went to New York City, went over to Fog Creek, chatted with them a little bit, and while I was in New York City I went to something called Hacker School.

It’s a bunch of people who are learning to become better programmers. They gave me an opportunity to talk to everybody and to teach a lesson on anything I thought they would find valuable. I think a lot of hackers don’t know about A/B testing, so I did a live demo of A/B testing using A/Bingo and I said, “This is literally how easy it is to make an A/B test. We’re going to live code one for my front page right now.”

“I’ve told you in like two sentences what Bingo Card Creator does and for whom.  Now, I’ve been doing this for five years and none of you all heard about it for more than 10 seconds, but just throw out a new headline for Bingo Card Creator. Try to beat my champion headline,  ‘Make Bingo cards on your computer.’”

Someone said, “Why don’t you try ‘Create Your Own Bingo Cards Now?” I said, “That sounds like a great answer. It won’t possibly beat my best thing, but sure, we’ll just try coding it up.” I come back a week later and there was literally a 40 percent lift. 40 effing percent.

Keith:  And that’s on sales.

Patrick:  Yeah, that was tracked direct to sales.

Keith:  That’s not trials, that’s not, “Oh, I want to know more,” that’s direct sales. 40 percent increase to sales.

Patrick:  Yeah. Somebody who knew virtually nothing about my business was able to beat something that I’d worked on for a couple of years. Granted, with A/B testing there’s often a bit of reversion to the mean. You get a confidence interval for whether there’s a difference between A and B, but with the typical way that you do stats with A/B testing, you don’t get a confidence interval on what the percent lift is.

It could be anywhere from say two percent to 80 percent and you wouldn’t know where unless you do more sophisticated statistical techniques that I don’t usually bother doing.  Then, that results in a reversion to the mean effect, where a couple weeks later you’ll see it’s only a 20 percent lift now.  You think “what gives?” It was probably only 20 percent that whole time, you’ve just got a bit of a ghost in the machine there. [Editor's note: This is not the only way reversion to the mean can happen.  For example, it may be the case for some sites that change itself tends perform well, irrespective of whether the changed version is distinguishable from the old version from the perspective of users who haven't seen the old version.  I generally just ignore the possibility of that, but it is certainly possible]

Be that as it may, you would be amazed how often, big, successful, established companies, like 37 Signals, Fog Creek, Google, Microsoft, et cetera, see huge lifts from A/B testing.  They run tests years or decades after having released products and that has a meaningful impact on their bottom line without a huge amount of work.

Something I often tell my clients who have 10‑year‑old products, “How much work would it be to increase the value of this product to your customers by one percent? If you have a team of 20 engineers working on a project for 10 years there’s 200 man‑years invested in it already.  You’ve probably already snipped all the low-hanging fruit in terms of features. Making it one percent better by adding features is going to require five man‑years, 10 man years, 20 man‑years of work.  Making it one percent better in terms of perceived value  can be as simple as changing button copy or changing a headline that you use on your product’s landing page.” [Editor's note: Remember, the last feature added to the product gets seen by a fraction of a percent of the user base, but the headline gets read by most folks who open the site.]

One percent is definitely not the ceiling. You can get 5, 10, 20, 100 percent in some cases. Basecamp has done many A/B tests, but just tweaking the header on the pricing page was worth 40‑60 percent. That should blow your mind. [Editor's note: Not sure whether I was remembering my facts correctly here: just trying to Google for a citation, the only article I'm quickly bringing up is a story about Highrise, another 37signals product, getting +30%.]

Think back. A week ago you were doing something for your business, right? What were you doing last week? Did it get you 40 percent to 60 percent improvement in your business’s bottom line? If not, why weren’t you doing A/B testing instead? When you phrase it like that, it always scares the heck out of me but it’s kind of true. [Editor's note: I often get very handwavy with that phrase "bottom line": SaaS math does not work out such that 50% growth in rate of new account signups results in 50% to the bottom line. Work with me here -- I'm from the land of one-time purchases, where modeling the effects of an A/B test is simple without having to literally start using calculus.]

Keith:  You do need the other stuff, you can’t just A/B test your way to a successful business, but at the same time if you have down time, if you have any amount of time, like we said one line of code, five minutes, it’s not hard to do. If you have that time, you should be doing it.

Patrick:  This should be your routine practice.  If you make a new feature and you’re wondering if users are actually going to respond to it or not, put the new feature in an A/B test. It’s as simple as putting an if block around it. [Editor's note: I'm trying to sell you something, not describe engineering reality here.  Yeah, it can get substantially more complicated than "Wrap that in an if block."  Buy what I'm selling -- you'll be happy you did.]

Keith:  I’m sure a lot of people are thinking, “But then you have to hire a designer and they have to make a mock‑up.” Even though it takes one line of code, it’s a pain to think about all these things. I do graphic design as well, but I show all my stuff to my wife and my friends to get a different perspective.  Even if I don’t agree with their comments, even if their comments are batshit crazy, I test it because the cost of testing is so low.  [Editor's note: I do not endorse testing batshit crazy ideas.]

If it fails, it failed for three days and then it’s off the site and we’re never going to see it again. Sometimes you get nice results. I had a very nicely designed welcome page. My wife looked at it and said, “It’s too busy. Why don’t you try having no background?” Tried no background. 40 percent lift.

Patrick:  You often see in design advice, for example at Smashing Magazine. I love designers — Keith is a designer and Keith is my best friend — but on the medical science scale, the discipline of design is closer to leeches than it is to medicine. I don’t like this design, so let’s add more leeches. We should make design more rigorous, like medicine: we’re going to test a treatment against a placebo and then see if it doesn’t kill people. Less leeches, more penicillin.

There was an article recently on Smashing Magazine that said if your body copy is less than 16 points you’re losing money. That’s a great headline, isn’t it, because that is a prediction about the measurable nature of reality.  So you should be able to test that. I actually did test that. For Bingo Card Creator, I changed the body copy sitewide to 14 points for half of the population  and 16 points in the other half of the population. [Editor's note: BCC has used 14px for several years.]

Sure enough, I got a null result. “Null result” means that after a week and 20,000 people or whatever saw both variations, there did not exist data to conclude that 16 percent was different than 14 percent along the axis I was measuring. Good to know, right? [Editor's note: 49,000 participants, actually.  It was a good week.]

Keith:  I do want to follow that up a little, doing a lot of graphic design myself. A lot of it is gut instinct. There are a lot of rules to design, but when you say something like “anything less than 16 point font is unreadable,” that really has to do with each page. You can’t just take any page, make all the font 16, and you’re suddenly going to have increased conversions. [Editor's note: Bah, designers.]

Smashing Magazine spinning the story in that fashion was essentially linkbait. That’s the whole reason they did that. For some sites it might get you 20‑30 percent. It’s worth testing and in Patrick’s case it didn’t work. It all has to do with the design, it has to do with how the site feels, and for the particular users. Some users may want a green button, some users might want a red button.

If your users are the ones who want the green button and you give them a red button, no matter how well it converts on another person’s page, it’s not going to convert well for yours. For any design for anything you should test it on your own users instead of following just what other people say.

Patrick:  That brings up a funny anecdote about users and culture. One of the versions of my Bingo Card Creator website was designed by a lady in India.  (Editor’s note: Gursimran Kaur, whose name I avoided saying because I am unsure of the pronunciation, having never verbally spoken to her.)  I asked her to design a pair of buttons for me, to represent downloading and creating cards.  She thought that the action associated with creation was getting something from the website.  Think of your hand going out, grabbing something, and taking it back to you.

So her button looked like a palm with splayed fingers oriented upwards on a red background, which in America means “Stop!”:

I had to say, “Hey, in America, this gesture does not suggest ‘Come get this!’ It means ‘No, no, no! Before you click this button think carefully. Do you really want to download this software? It will give your Googles a virus.’” [Editor's note: I am poking light fun of my composite customer, who is not very technical, and as a result might use "the Googles" to mean, variously, the Internet, the web browser, or the entire experience of using a computer.]

We settled on a design of the download button that did not give their Googles a virus.  It probably worked better.

Keith:  You’ve made many buttons since then, of course. You’re very big into the color contrast. You want big, flashy buttons that say download this now.

Patrick:  People make fun of me. I’ve been saying this stuff about A/B testing and whatnot for years. I make buttons bigger and it almost always works better. One of the English as a second language folks in one of my forums was talking one day and he’s like, “I know Patrick loves his big, orange, pancake buttons.” Big, orange, pancake button is my keyword for this now because big, orange, pancake buttons really effing work. Go ahead.

Keith:  Except for these ones you tried last time. You had put these ones in and we were working together and he just turns to me and says, “I have these huge, new buttons and I really like them, but they’re just not converting well.” I look at them, they were really the ugliest things I’ve ever seen. They completely did not fit with the site at all. They were just offenses against God, honestly. [laughs]

Patrick:  Right. I’ve been trying to get Keith to be my cofounder for the last couple of years. We’ll see if that happens.

Keith:  We’ll see if that happens.

Patrick:  One of the reasons I have been trying to recruit Keith for this kind of thing is because there’s this, I don’t know.  Maybe it’s a skill, maybe it is an aptitude, maybe it is something your born with, but it is called taste. Whatever day God handed out the taste I was studying stats for D&D characters or something and totally missed it.

Keith:  For action buttons for Patrick it’s generally bigger, oranger, more pancake‑y. It’s pretty much the order it goes in.

Patrick:  With a little bit of rounding on the side.

Keith:  You love the rounding.

Patrick:  Good pancakes should be rounded and huge. Huge, rounded pancakes.

Keith:  I think your Bingo Card Creator is just going to have a giant orange circle on it that says “Buy now.” You should split test that for April Fool’s Day.

Patrick:  People wanted me to test that for a while. I was about to put an 800 by 600 button on my landing pages, but that would violate the Google AdWords content policies, which is problematic because most of the traffic to the landing pages comes from AdWords.

Speaking of which, interesting result from Fog Creek. Fog Creek decided to replace their FogBugz home page. It’s text‑heavy and tries to sell software to enterprise users. They replaced it with their kiwi logo and text similar to “FogBugz: The World’s Best Bug Tracking Software.  Start Now.” No text.  No screenshots.  No philosophy.  Just “Kiwi.  Start now!”

Conversions went way up. At the same time, they fell off the Internet in terms of SEO.  SEO and A/B testing exist to help the business. The business does not exist to help them. Despite the fact that that A/B test was successful, we couldn’t justify burning their Internet brand to continue it, the result was amusing.

Keith:  Right. Actually, one thing you got from a similar test from Fog Creek is that people love to watch Joel talk.

Patrick:  People love to watch Joel talk.

Keith:  I love to listen to Joel talk. I actually listen to the StackExchange podcast just to hear Joel talk.

Patrick:  This is why we’re doing a podcast right now, by the way, because Keith was so inspired by listening to the StackExchange podcast.

I don’t want to give the impression that all Fog Creek’s optimizations are my doing, by the way. I helped Fog Creek a few times with regards to their marketing, A/B testing and whatnot, but there are people in the company that are doing it pretty much every day. [Editor's note: Fog Creek folks are ridiculously smart and they're a wonderful company to work for.  Does being their data-driven marketing guy interest you?  Get in touch with them or get in touch with me -- they're hiring.]

One page frequently tested is the front page for FogBugz. For example, a page with a flat photo versus a design with a short three‑minute feature video versus a page with Joel Spolsky talking for an hour.  The video is indirectly a sales pitch for FogBugz, but it’s mostly a pitch of how we should do software development better.

People watch that video for the entire hour. This just blows my mind. They’ve shown me a graph of what the level of attention is at various points in the interview. If you can imagine the graph for a Justin Bieber song on YouTube where everybody is listening at the first second and then X percent listen through the entire song, it’s that graph… but stretched out to an hour.

Granted, there’s probably not too many 14‑year‑old girls in the market for bug tracking software. A very different audience wants to watch Joel Spolsky talk for an hour about FogBugz. After they watch him for an hour, they are effing sold. Where is that button? They want it now.

Keith:  Actually, that is an interesting trend. Joel talking for an hour is definitely above and beyond what I think most other companies can do. Video has always been the anathema of landing pages. You never put video on your landing page and stuff. Video now on landing pages is converting so much better than just text and I think it’s really going back to people don’t like to read.

Patrick:  Right. People do not read on the Internet. If you take one thing from this podcast, people do not read on this Internet. [Editor's note: It's a pity that of the tens of thousands of people who see this post, only about five will actually read this comment.  You'll know who they are because they'll race to mention it.]

Keith:  They will click on a play button and listen to a 10 second clip over reading a paragraph.

Patrick:  Yeah. It’s insane. This is one of those things where engineers have a totally different grasp of how the world works because most of us were precocious readers. We devour 600‑page Lord of the Ring books with no problem. That is severely anomalous behavior among the population at large.

If you install Crazy Egg, Crazy Egg will show you what percentage of people actually made it to a particular portion of a blog post or whatnot. The world, in general, is TL;DR after four words. It’s crazy. That’s why headers matter so much, by the way, because if they’re only going to read one sentence, make that sentence count.

Keith:  Crazy Egg, if you use their scroll map, you will have so much wonderful information on your pages, especially if you have any page that requires scrolling for more than maybe three clicks.

Patrick:  We just got done saying people don’t read on the Internet, but man, long copy. Long copy is those lovely pages that have hundreds and hundreds of paragraphs of talking you to death about the product. They just beat you into submission and then give you the buy button and then you actually buy. Do people actually buy from these, Keith?

Keith:  One of my clients has very, very large long, long, long sales pages. I think it’s 68 pages printed. I looked at them and I was like, “There is no way people read these.” While 100 percent do not read the entire thing, 40 percent read the entire thing. 40 percent.

Patrick:  That just blows my mind.

Keith:  It absolutely blows my mind. The interesting thing now is that the sales button is not at the end of the page, it’s somewhere in the middle, and yet people read, go to the sales button, actually continue reading, and come back to the sales button in order to click it. That’s just amazing.

That’s 40 percent. 60 percent either don’t read it at all or just skim around. It’s really cool because you can see exactly where on the page people will stop, even for a slight amount of time. There’s this picture of a kind of cute girl with a blue Mohawk on. I hope I’m not giving out too much but it’s not a public page so I think it’s OK. 60 percent of people stop at that picture. It’s 40 percent, 40 percent, 40 percent and then blue Mohawk 60 percent. The area is just bright yellow on the heatmap. Everyone loves looking at this person.

Patrick:  People love faces, by the way. Dave McClure has a great quote about this.  [Editor's note: profane, amusing, and not paid attention to nearly enough.] If you put faces on anything, you’ll draw attention to both the face and to everything surrounding the face.

One of my buddies, Ian, runs HelpSpot. It’s customer help desk software. He just put a photo of himself on the front page. It’s the size of the author photo at the end of this post — really tiny. He just put a photo of himself under the main content area of both default and the front page with the preexisting copy about I’m the founder, this is wonderful software, here’s why, yada, yada. Just adding the photo increased the number of trials by 20 percent.

I think it’s hardwired in people — biologically —  that we track gazes very well.  If you find photos of people looking at something and you track where their eyes are on the page, all the visitors are going to be looking on that line of sight too. Have that line of sight terminate in the form or the sign up button or whatever it is that you want people to take action on. Really, really works well.

Keith:  There’s a lot of good articles on that on Google if you search for, I can’t remember, what is it eye tracking and product placement?

Patrick:  I think it’s gaze. The magic word is gaze.

Keith:  They’ve done actual eye tracking experiments and it amazing that people will look first at the face and then where the face is looking. If you have a face with someone looking to the left and to the left is your purchase now button, then you have a much higher chance of people clicking on that button than anywhere else on the page.

Patrick:  Speaking about the faces, who here has seen a comely young lady in a headset? Girls in headset photos are the visual equivalent of Cosmo’s top 10 things that your employer doesn’t want you to know headlines. Those persist over hundreds of websites done by hundreds of different highly sophisticated companies because they effing work. Go to HeadsetHotties.com. [Editor's note: Some people claim they thought of Facebook before Facebook.  I thought of HeadsetHotties.  I have really really tiny sour grapes about getting beaten to market.] It’s a real website. Take a look at the comely photos of young ladies with headsets. Put it on your page. A/B test it. You’ll see it go up.

I’ve seen also great variations on that. A lot of engineers told that their software will do 20 percent better stuff for customers if it just had a photo of a young lady with a headset get very skeptical of that.  They’re just not willing to try it on their website for whatever reason. Hey, it’s your business. Ultimately that is your decision.

I once worked for a particular company.  They had an employee who they wanted to get contacted from a particular page.  Let’s call him Bob.  Bob is very much not a female, blonde, 20‑something with a headset, but we put Bob in a headset and took a photo of Bob and just put a photo of Bob on the pages with a line that said, “Bob isn’t pretty, but he actually works here. Send Bob an email.” Bob got a lot of emails.  This was a happy result for the company, since people were practically begging Bob to tell them about a five-figure enterprise solution.

So authenticity works too, but if you’re going to be authentic, be authentic by using a picture with a person’s face. It helps.

Keith:  All right. We’ve gone almost an hour and a half now. Some of this is going to go on the cutting room floor. I think that’s about time to wrap up.

Patrick:  All right. Thanks very much, Keith. We’re going to probably be doing this later. Feel free to drop us a comment either in the comment thingy below or on Hacker News or through the emails. You want to give your contact information? We’ll put it in the post.

[Editor's note: Patrick is patrick@ this domain.  Keith Perhac is k dot HIS_LAST_NAME@delfi-net.com We'll also read your comments on the Internets.]

Keith:  We’ll put it in the post. As always, Patrick is patio11 and I’m harisenbon on Hacker News and Twitter.

Patrick:  Thanks so much for listening everybody. We’ll see you next time.

Keith:  All right. Good talking to you.

Patrick:  Take care.

Keith:  Bye bye.

 

Transcript by CastingWords, with light editing by me.  It cost about $100, and was worth every penny, since it freed up basically an entire day of my time.  Assume any remaining errors are my fault.

42 Comments

Image Ad Blending Works Really, Really Well

So I’ve hung around with SEOs for the last couple of years, including ones who pay the rent based on their ability to convince people to click on AdSense ads, and I’ve learned a trick or three.  One that Google will actually tell you in as many words is to make the ads seem less garishly out of place on your site.  In the words of an Internet buddy, “You want to not look like an ad at the first glance, but to look like an ad on the second glance.”  This way you avoid banner blindness, the phenomenon in which Internet users mentally tune out portions of a website which look like advertisements.

Google says:

We’ve outlined a few strategies below that are designed to decrease ad blindness, the tendency for users to ignore anything that’s separate from the main content of your site. By making these changes, you’ll be making your ads more visible to users. The goal isn’t to confuse users into thinking ads are content, but to get users to see and read the ads so they can click on those that interest them.

That’s why if you buy ads on Google itself, they’ll look something like this:

As you can see, the top ad is styled to resemble the main content on the page, with a bit of a subtle yellow background and the notation that it is an ad there, if you take a gander for it.  (Note that a lot of users think that yellow is used for marking the best results.  Non-technical customers, yay.  If you don’t believe me, watch someone who can’t program Google sometime.)

I buy a lot of ads on the Google Content Network, but I don’t get a say in how they are presented usually.  Since I pay per click, it is in the interests of webmasters to make the ads look content-esque, so that they catch lots of clicks.  They occasionally get clicks with less-than-true-user-intent volition behind them, but that is a cost of doing business for me.

Anyhow, I’m always experimenting with different ways to advertise profitably for my businesses other than Google AdWords, which a) are expensive (I spend something like 50~60% of my gross on a sale at the margin) and b) have limited volume.  I recently was inspired to try something new when listening to a (paid) Mixergy video with Ilya of Mixrank, whose blog you should really be reading if you are interested in online advertising.  The gist of the video was to try negotiating direct deals with advertisers with access to the right targeted demographics, and I’m going down that route as well, but for the moment I wanted a get-my-feet-wet option that was more self-service, so I went with BuySellAds.

BuySellAds basically lets you pick a website (in their network), pick a particular type of image ad inventory on it, and pay the displayed rate for advertising there for a month.  Sadly, their options for inventory appropriate to teachers are rather lacking, but I found one wonderful website accepting ad placements through them: BusyTeacher.  BusyTeacher takes 728×90 image ads on their category pages, like this one.  So I went into Paint.NET to exercise my meager pixel-pushing skills, slapped something together, and submitted it for their approval.  After they approved the ad, my credit card got charged for a month of placement ($135 for an estimated 500k impressions or thereabouts), and it went live.

Let’s Play “Spot That Ad”

You can click on the photo to see the full size version.

 

You probably saw-but-didn’t-see the “Get A College Degree For Easy Loan Payments of Only…” spammy ad in the middle.  If you routinely surf sites visited by middle-aged women, you’ve seen-but-not-seen so many thousands of them that you tune them out.  But you probably didn’t automatically tune out the Bingo Card Creator ad.

Spot it yet?  Hint: it’s the row without the Facebook button.

Is This Evil Or Just Evil Genius?

Once upon a time I was an engineer totally scornful of effective marketing, but I have gradually gotten over it.  After thinking it over, this is aggressive but within my comfort envelope.  The ad is honest about being an ad, makes a straightforward commercial proposition (“Sign up for a free trial”) to an audience that I think will respond well to that, and is pretty true by the standards of marketing copy.  It is designed to catch clicks only from people interested in signing up for a free trial of Bingo Card Creator, and sends them straight to a landing page where they can do just that.

I wish there was a way to dynamically generate the image such that I could provide a more exact star valuation, but in the context of a sponsored placement, “Rated 5 stars by lots” is both non-specific and true.  Lots of people have used BCC, and when I ask for star ratings in internal surveys I get something like 4.8 on a volume of hundreds or thousands.  I think this compares favorably with “9 out of 10 dentists agree” and other pretty banal marketing copy.

So Does It Work?

Oh heeeeeeeck yes.  That ad has a 2.2% click through rate (astronomical by the standards of banner advertising), generates about 500 views of my landing page a day, and of that about 11% or so convert to the free trial.  (This is lower than my landing page average by quite a bit, but that makes sense.  Most people who come to one of my landing pages just got done searching for e.g. “make bingo cards”, so they’re clearly in a bingo mood.  The user here was just looking at a page about generic teaching activities then shown the bingo option, and might not be totally sold on bingo for her classroom yet.)

Anyhow, at about $5 to run on this site per day, the CPA (cost per action = how much money do I spend per free trial signup driven) has been running somewhere in the 8 cent region.  Yowza.  I pay Google closer to 30 ~ 40 cents.  At typical conversion rates to a trial of about 1.8~2%, that means that this costs me $4 or so to generate a $30 sale rather than $20.  Sold!

I just wish there was more inventory available.  Many of the sites in the teaching niche either only do Google AdWords, so I’m already saturating them (and paying a cut to Google), or they only accept advertising through large networks, which tend to favor e.g. brands with $X00k marketing spends and not guys who want to experiment with a few hundred bucks at a time.  I’m following up on the advice to get in touch with smaller sites directly, but I need to hit the sweet spot of “Small enough for my $X00 to matter, large enough that they send me enough traffic such that my time in negotiating an ad buy and preparing a creative is worthwhile.”  For an experiment that looks like it will net in the neighborhood of $1,000 a month in sales for $135 and ~10 minutes of pixel pushing, this one is going in the win column.  I hope to get bigger and better results later.

Anyhow, check out BuySellAds, direct ad buys, and ad blending.  They can be made to work.

34 Comments

I Saw An Extremely Subtle Bug Today And I Just Have To Tell Someone

This post will not help you sell more software. If you’re not fascinated by the inner workings of complex systems, go do something more important. If you are, grab some popcorn, because this is the best bug I’ve seen in years.

Have you ever been logged into a site and get suddenly asked to log in again?  This is one of those minor nuisances of using the Internet, right?  If you’re not technically inclined, you think “Hmm, ghosts in the Googles.  Oh well, here is my username and password again.”  If you are technically inclined, you might think “Hmm, funny, my cookie picked a bad time to expire, or maybe my session was getting stored in Memcached on a machine which just went down.  Oh well, here is my username and password again.”

It turns out that Bingo Card Creator has been doing this pervasively to a fraction of my users for the last few months.  I never noticed it, and no one ever complained.

Here’s the scenario: Bingo Card Creator is a Rails application, originally coded against Rails 2.1.X and then gradually updated with Rails security releases.  Like many Rails applications, it stores sessions in a cookie (using CookieStore), and uses the session to hold only very limited data.  Specifically, it holds the (critical) user ID for logged in users and the (nice to have) pre-login session ID.  I use the pre-login session ID to tie some analytics stuff together on the back end — basically, it lets me associate newly created accounts with search terms and whatnot that bring them to my site.  The exact mechanism for doing that isn’t important to this bug — you just need to understand that the session resetting is a minor nuisance if it only happens once in a blue moon, and a huge nuisance if it happens pervasively.

Subtle Indications My Site Was Borked

BCC maintains a whole lot of internal analytics, because I’m something of stats junkie.  Because BCC is in maintenance mode this year, I don’t actually view the stats on a regular basis — as long as the server stays up and users don’t have any complaints, I let the sleeping dog lie.  (I’ve been busy with other projects.)  Anyhow, one example of such a stat is “Of recently created trial accounts, how many were referred from the Halloween bingo cards mini-site?”  For most of the year, that should be a negligible number.

Except right about on Halloween, when the mini-site sees on the order of 30,000 visits or more.  This usually sells several thousand dollars worth of software.  That is fairly hard to miss, because if several thousand dollars don’t show up in my bank account, I’d know right away.  (Sidenote: I did lose about $1,000 due to an ill-timed server crash while I was on a cross-continental plane ride right during the middle of the pre-Halloween rush. Oof.)  So naturally, several thousand dollars implies a hundred or more sales (at $30 each) which implies thousands of trials, right?

Well, my internal analytics code was telling me that the Halloween site had referred ~100 trials of which 6 converted.   Which means that I should have expected a $200 bump in my bank balance.  Which was not what happened.

I mentally filed this away under “Hmm, that’s odd” but didn’t investigate immediately because I had not lost any money (or so I thought) and was busy that week.  Then recently, after doing an unrelated code push (I integrated Stripe, it is awesome, full details later), I did my usual post-deploy smoke test and, after creating a new account, I got suddenly logged out of the application.

“Hmm, that’s odd.”  And I tried it again, twice, couldn’t produce the error, and mentally wrote it off to gremlins.

In Which I Become Doubtful Of The Existence Of Gremlins

Four hours ago, my brain suddenly decided to put these facts together. The discrepancy for the sales statistics strongly suggests that, prior to accounts getting created, the session was getting cleared.  This meant that, when the account actually got created, the referrer was not associated with the account in the DB, which threw off subsequent stats gathered by my internal analytics.  Sessions getting randomly cleared would also cause the user to become instantly signed out.

I tried to reproduce the problem in development mode and was pervasively unable to do so.  Then I started trying to reproduce it on the live site and was able to, sporadically, but only in Incognito Mode in Chrome, and only if I clicked fairly fast.  (Don’t ask how many dozens of tries it took to figure out that fast clicking was the culprit.)

Having verified that it actually existed, I added instrumentation to tell me what my session ID was, and noticed — like expected — that it changed when I was suddenly logged out.  Sure enough, the session was getting wiped.  But why?

Racking my brains to figure out “What could reset a session in Rails other than explicitly trying to do it?”, I started listing up and discarding some candidates:

  • The cookie expired in the browser — nope, expiry was set correctly
  • The cookie got eaten by the hop from Nginx to Mongrel — nope, after investigation, cookies always matched on both sides (like expected)
  • The cookie got too big and failed to serialize properly — nope, after looking through the Rails codebase, that looked like it would throw an exception
  • The cookie got reset when Rails detected malicious behavior coming from the browser — bingo!

CSRF Protection: When It Breaks, It Breaks Very Quietly

Cross-site request forgery (CSRF) is tricking the browser with a malicious (or compromised) site B to access something on site A.  Since requests for site A will carry A’s cookie whether requested by A or not, an image tag or embedded Javascript on B can do anything on A that a logged-in user can do, like accessing /accounts/wire_all_your_money_to_switzerland with the appropriate POST parameters to make it happen.  This is, to put it mildly, a bad thing.  Rails has lovely magic which defends against CSRF for you: all you have to do is include two lines of code
#In application_controller.rb
protect_from_forgery

#In your templates' HEAD somewhere

Rails will then basically generate cryptographically secure random number, totally transparently to you. This is called the CSRF token.

One copy goes in your Rails session, where only your server and the client can see it.  (n.b. Rails sessions are a bit opaque since they are Base64 encoded, but they can be trivially decoded by anyone who can read the cookie, including the end-user.  They can’t be forged because of another security feature, but don’t put anything you don’t want the user to know in the session.)

Another copy goes in the document’s HEAD (for access via Javascript) and in Rails-generated forms as a hidden value.  When Rails makes a PUT or POST request to the server (via helper-generated form or helper-generated Javascript), Rails will submit the copy included in the HTML code with the request, compare it to the one in the session, and bounce requests where they don’t match. Bad actions on other sites shouldn’t be able to read either a) a page on your site (the same origin policy prevents this) b) the contents of your cookie from your site, so this is secure.

The specifics of how it “bounces requests” are very important.

Point Releases Sometimes Contain Doozies

My personal understanding of Rails up until an hour ago was that a CSRF violation would raise an exception.  This would practically never get seen by a legitimate user operation, so few people are aware of that, but I had seen it a time or two when security auditing BCC.  (Some of my geeky friends had, back in the day, exploited BCC with a CSRF and helpfully told me how to fix it.  Naturally, after fixing it I verified that the site worked as expected with the fix.)

So if the CSRF protection was somehow eating sessions, I would expect to see that exception getting logged and emailed to me by Airbrake (formerly Hoptoad — it emails you when an exception happens in production, highly recommended).   That wasn’t happening.

Then I decided to dig into the Rails source.  Whereupon I learned that Rails 2.3.11 changed the behavior of CSRF protection: instead of throwing exceptions, it would silently just clear the session and re-run the request.  For most sensitive operations (e.g. those which require a signed in user), this would force a signout and then any potentially damaging operation would be averted.

Here’s the relevant code in Rails 2.3.11:

def verify_authenticity_token
  verified_request? || handle_unverified_request
end

def handle_unverified_request
  reset_session
end

Versus the relevant code in Rails 2.3.10 (sidenote: you can see all of this easily in Github because Rails is diligent about tagging releases, a practice you should certainly follow in your own development):

def verify_authenticity_token
  verified_request? || raise(ActionController::InvalidAuthenticityToken)
end

And, sure enough, checking Subversion showed that I upgraded the version of Rails I was using in January of this year in response to this security advisory. I read that, made the required modifications to my application, tested, and had no problems.

So What Went Wrong Then?

After I was sure that sessions were being reset (but only in production), I added a bit of instrumentation to the live site to record the session ID for people coming from my IP address and to log when it changed. This let me find the culprit: a bit of Javascript that A/Bingo, my A/B testing library, uses to verify that people are human. It assumes that robots generally won’t run Javascript engines capable of doing POST requests, so it does an ajax-y POST to my server to assert humanity of the end-user, thus keeping almost all bots out of my stats.

That code has been live over a year. Why did it suddenly start causing session resets? Oh, another change in the 2.3.11 upgrade:

The old code:

  # Returns true or false if a request is verified.
  # Comment truncated by Patrick
  def verified_request?
      !protect_against_forgery?     ||
        request.method == :get      ||
        request.xhr?                ||
        !verifiable_request_format? ||
        form_authenticity_token == form_authenticity_param
  end

Notice that request.xhr? will cause this request to be verified if it evaluates to true, regardless of the other things in the OR statements. request.xhr? tests whether a request is ajax-y in nature. A/Bingo’s humanity-verifying POST is, so it didn’t trigger the CSRF check.

The new code, however:

  # Returns true or false if a request is verified.
  # Comment truncated by Patrick
  def verified_request?
    !protect_against_forgery?                            ||
      request.get?                                       ||
      form_authenticity_token == form_authenticity_param ||
      form_authenticity_token == request.headers['X-CSRF-Token']
  end

Yep, as announced in the patch notes, we lost the exemption for XHR requests. So the A/Bingo mark_human request will, because it makes no particular effort to include a CSRF token (which I will be changing very quickly, as A/Bingo is my project), with certainty cause the CSRF check to fail in 2.3.11. This will result in not a noisy exception (the previous behavior) but instead a silent reset followed by re-running the action. A/Bingo, which doesn’t care a whit whether you’re logged in, will then mark your freshly new session as human. If the previous contents of your session mattered, for example to keep you signed in, they are now gone. A/Bingo will not reaudit your humanity, though, because your session now marks you as human, so this will only ever happen to your browser once.

Race Conditions: Not Just In Java Thread Programming

So why did this never show up in development and why did it show up only sporadically in production? Well, consider how a browser interprets a page presented to it: it first downloads the HTML, then downloads the assets, blocking when it discovers e.g. CSS or Javascript which alters the document. This means that Javascript very low on a page may never execute if someone above it blocks them until the user navigates away. (This is a pretty gross simplification of how multiple pieces of very complicated and often incompatible software do something very difficult. If you want details, read stuff by the people behind YSlow. They’re geniuses and taught me all that I successfully learned about this process.) Someone like, say, external analytics utilities loaded over the public Internet. My page includes a few such scripts, like Google Analytics and CrazyEgg. They are off in development to avoid polluting my stats.

This plus the lack of network latency means that, on development, a browser which sees a page that includes the humanity testing Javascript will almost certainly execute it. That will cause the session to be burned, once, on the first page load. Since my invariable workflow for manual testing is “Start with a new browser at the homepage or landing page, do stuff, make sure it works”, the order of execution is:

  1. I load the front page or a landing page. The session is initialized to some value S1.
  2. (A few milliseconds later.) The A/Bingo Javascript checks for my humanity, resetting the session to some new value S2.
  3. I hit the registration or login button, and the site works as I expect it to.
  4. Since the site knows I am human now, that never gets checked again, and the session never gets reset again.

In production though, the workflow could very well be:

  1. The user arrives at the front page or landing page. The session is initialized to some value S1, including (say) their referrer information.
  2. A bunch of Javascript starts loading ahead of the A/Bingo check.
  3. The user, within one or two seconds (depending on network latency to those external scripts), either logs in or creates an account.
  4. The browser never successfully executes the A/Bingo check.
  5. The user arrives at their dashboard. When it is rendered, the server (robotically) decides it isn’t quite sure if they are human yet, and includes that Javascript again. (This behavior is designed because I was aware of the timing issue, I just didn’t realize how it would shake out with the 2.3.11 upgrade.
  6. This time, the user ponders their dashboard enough for the A/Bingo Javascript to post successfully. This resets their session to some new value S2.
  7. The user clicks anything on the page, and (because S2 doesn’t include their logged in user ID) gets taken to a login screen.
  8. The user is now durably marked as human, so the A/Bingo check never fires again, preventing a second unceremonious logout.

This neatly explains the logged out users. How to explain the missing referrer information? Well, if the user is NOT very fast on the click on the first page, they’ll have their referrer cleared out of the session before they successfully signup. They’ll get marked as a human prior to creating their account, though, so they’ll never even notice the unceremonious logout. This is the behavior of the overwhelming bulk of new users, which is why the stats were getting comprehensively borked but almost no users thought to complain.

This difference in behavior based on the hidden interaction of two concurrent processes is called a race condition. Race conditions are why sane programmers don’t program with threads or, if they do, they use shared-nothing architecture and pass all communication between the threads through a message queue written by someone who knows what they are doing (if you have to ask, it isn’t you — seriously, multithreaded programming is hard). I haven’t seen a race condition in years, because the genre of web applications I write and their architectures makes me mostly immune to them. Well, I just got busted back to CS102. Sadly, the core lesson of CS102 hasn’t changed: reasoning through why race conditions happen is very hard.

Saved By Unsophisticated Users, Sort Of

Users returning after the session naturally expired (2 weeks) would go through the dance again, potentially getting asked to log in twice. However, it took most of them enough time to have the human check prior to finding where the Sign In button was, so the percentage of users who actually visibly saw the bug was fairly small. (I’m guessing, from a quick heuristic run on my log files, that it was below 1% of accounts. That’s the optimistic way to say it. The pessimistic way is to say that this bug negatively affected the better part of a thousand people, and probably cost me sales from some of them.)

Whose Fault Is This?

If my users are inconvenienced, it is my fault, always. I should have read the patch notes for 2.3.11 more diligently, to discover the very consequential line “In addition to altering the templates, an application’s javascript must be changed to send the token with Ajax requests.”, and I should have been more aware that there was a one-line Javascript method pulled in by a library (which I wrote, so that is no excuse) which was not automatically upgraded with the Rails helper methods.

I’m not sure if more diligent testing would have caught this. Race conditions are hard to diagnose, and while I might have caught it by including production levels of external Javascript in my development environment, the symptoms would only have been visible a fraction of the time anyhow, and in ways which didn’t visibly break the application most of the time. (Who checks their stats for the development version to make sure they’re sensible after implementing that function correctly the first time?)

What I really should have done about this is addressing it earlier, when I first got the inkling that there was some weird edge case which would cause a logged in user to become logged out. I futzed around with my configuration once or twice and saw the problem go away (because it was non-deterministic), but rather than futzing I should have figured out a complicated but reducible series of steps that would always cause the issue. That would have sent me down the right road for fixing it.

So How Do You Address This

Immediate term, a one-line patch turns off CSRF protection for the A/Bingo mark_human action, preventing it from accidentally resetting the session.

skip_filter :verify_authenticity_token, :only => :mark_human

I also added a note about this to the A/Bingo documentation. I’ll patch A/Bingo after I have enough brain cells left to do that in a way which won’t break anyone’s applications. After I patch A/Bingo, that work-around won’t be necessary.

Why’d You Write This Post?

Because, after hours spelunking in Firebug, my codebase, and the innards of obsolete version of Rails to understand what was happening, I had to tell somebody. Some people have water coolers. I have the Internet. Hopefully, someone in this wide world will find this discussion useful.

If you’re wondering what the day-to-day life of an engineer is like or why it’s so dang hard some of the time, this might be a good example (of the pathological case — the typical case is writing boring code which solves boring problems, like laying out a 5×5 grid on a bingo card and randomizing the word order). Bingo Card Creator is not terribly complicated software when compared to most applications, but it sits on top of other pieces of code (Rails, the web server the browser, the TCP/IP stack, the underlying OS, the hardware on both ends, etc) which collectively are orders of magnitude more complicated than any physical artifact ever created by the human race.

Most of the time that complexity is abstracted away from both the user and the developer, both as blissfully ignorant of the layers below as an ant walking on an aircraft carrier is ignorant of the depth of the ocean. But when a problem bubbles up and writing it off to gremlins isn’t getting the job done, you have to start looking at the lower levels of abstraction. That is rather harder than dealing with just the higher levels of abstraction. (Joel Spolsky has an article about this subject.)

43 Comments
Loading...
Free video + email advice on making & selling software:
(1~2 emails a week.)