Archive by Author

Hacking Customers' Technology Adoption Cycles

YCombinator just released their semi-annual application for companies to incubate.  One of the new questions this time around is “How will you get users?”  I think that is a great question to think about for everybody in business — perhaps the great question to think about.  Customer acquisition is one of the easiest places to screw up as a startup, particularly for technical founders (who, in their previous lives, have probably never had to do it for anything).

I’m not applying to YC this time around, but I always fill out the application to force myself to talk through my business strategy.  I had one thought which sounded worthwhile enough to share: customer acquisition can be hacked.  You can take the current conventional wisdom in the market of how to get customers to use solutions, identify it’s weak points, and aggressively target them.  That can, potentially, be as important (or more important) than the same applied to the actual product.

Enemy #1: The Technology Adoption Cycle

Let’s assume that you’re capable of successfully identifying a problem customers have and solving it.  Those are both highly non-trivial, but put them out of scope for the moment: people’s hair is on fire, you’re selling fire extinguishers, life should be grand.  Life is often not quite so grand, because you can produce a wonderful product which creates value and fail to sell it to folks.

Most startups are not creating an entirely new solution out of whole cloth.  Somewhere out there people are currently experiencing the problem you are solving, and they’re dealing with it somehow.  They might be ignoring it or gritting their teeth.  They might be using some inferior solution which they got from your competitors, you have competitors (you should have competitors — if you don’t, you probably aren’t doing something people care about).

Your competitors had to see people through a product adoption cycle:

  1. Identify people with the problem
  2. Teach them that the solution exists
  3. Successfully sell them on the solution
  4. Prevent them from leaving the solution for a competing solution

In actual practice, this adoption cycle is frequently long and arduous.  (If it were short and easy, there wouldn’t be any money in it.)

Your competitors, if they are established businesses, are probably very good at maneuvering customers through the technology adoption cycle as it exists in the market today.  For example, if grading students is a problem, your competitor might very well be successfully selling school districts on their gigantic consultingware grading solution which costs six figures an installation.  Since they can still make the rent and keep the lights on, you can infer that their business probably works.   Their marketing team is generating sufficient leads, their sales team converts some of them.

But you probably don’t want to do what they’re doing, because they’re better at being them than you are.

Hacking The Product Adoption Cycle

Startups are not the world’s most obvious choice of employment for people who enjoy coloring between the lines.  If you execute the competition’s playbook for acquiring customers, you are probably going to get crushed by them, because

  • they know more about the market than you do
  • they have a commanding head start
  • they have large amounts of resources to throw at the problem

On the other hand, it is entirely possible that:

  • they have stopped learning about the market
  • they have a commanding head start running in a suboptimal direction
  • they have large amounts of resources which, for reasons of switching costs and politics, can’t be reallocated to more efficient approaches

These statements aren’t just true about the product — sure, they might have a crufty old VB6 app and you have the new Node.js hotness.  They are equally true about the customer acquisition process.  You’re competing with their business, not with their product, so you could possibly either focus your innovation on customer acquisition or, more likely, use innovation on both customer acquisition and product in a mutually supportive manner.

Examples Of Hacking This

Freemium isn’t a business model so much as it is a customer acquisition tactic.  In markets dominated by expensive solutions with huge switching costs and uncertainty about success with technology changes, freemium can be very compelling: the self-serve model allows you to do less consultative sales (with the multi-month purchase cycles, large sales teams, and politicking that entails) and instead focus your efforts on getting leads and converting them.  This plays to a very different skill set versus traditional enterprise B2B sales, and it is a much more forgiving of small teams, since you’re deputizing your free users as internal sales champions and praying that they can do the consultative sales that your non-existent sales force isn’t doing.  This also lets you crack into markets where any model which requires consultative sales automatically is priced out of the market — essentially, anything where customer lifetime value is less than $75,000, give or take.

Monthly billing is another hack.  Customers are irrational and their processes are broken.  One artifact of those practices is that there is a stepwise increase in difficulty if prices increase by $1, as long as the price was already at whatever the company’s magic number is for maximum to be put on a corporate credit card or signed for on a non-manager’s authority.    Monthly billing defeats this step function because even if the total lifetime cost of the solution goes up the largest amount ever billed at once might well cross under that critical threshhold again.  This means that there is no longer a total no-man’s land between $1,000 and $75,000 in lifetime value. (Is this a hack?  Yes.  If you bill a Fortune 500 company product manager $80 a month, you are essentially conspiring with him against his accounting controls.  Not that there is anything wrong with that.  You can even explicitly sell that as a benefit to him, just like you sell SaaS as allowing him to avoid having to talk to IT to get the stuff he needs to do his job.)

Online marketing expertise hacks through the ridiculous inefficiencies of offline marketing.  Many startups can run rings around their traditional competitors in online marketing, for example due to savvier SEO that leverages their strengths in execution speed, technological savvy, and community ties.  For example, my wee little business competes directly with Scholastic Publishing, who has 10,000 employees and access to public capital markets.  They also couldn’t spell SEO if you handed them a set of alphabet flash cards, which is good news for me.  You would think that “Well, a business which doesn’t have online marketing expertise could just hire for it”, but after you get past the level of “let’s make a website — it should probably have title tags and some of those keywords, too”, everyone who tries this finds that it is murderously difficult to hire competent SEOs right now.  (If you disagree, I have some clients who would love to meet you.)  At the same time, I couldn’t possibly compete with the relationships which get their competing products on shelves at tens of thousands of retail locations… but then again, I don’t have to pay 50 cents of every dollar of sales to the retailer, either.

Taking A Hack From Tactic To Strategy

I think this isn’t exactly a new insight.  There are lots of folks who, when asked for their marketing plan, will say “Oh, we’re going to get lots of search traffic” (indeed, that is probably second only to “it will grow virally” in terms of signaling “has probably not thought this through.”)  What separates hopes and dreams of future success from very valuable businesses is a strategy which, with execution and refinement over time, will actually achieve the goals.

We often hear products described using something like “It’s like Facebook, except for dogs.”  How about, instead, describing businesses like “It’s like Quicken, except Quicken sells primarily through boxed software channels and we’re going to sell primarily through banks which will deal with us for a cut of the sale price and the ability to deepen relations with small business customers, who consume lots of high margin services and stay locked in for decades at at time.”  (That may or may not actually be true.)

We often accept previous experience or minimal proof-of-concept prototypes/MVPs in lieu of a functioning product when evaluating whether someone is capable of executing on building something.  Why not do essentially the same for proving that one is likely to get customers?  A previous background in revenue maximization through negotiating cross selling deals for banks, or evidence that you have enthusiasm from a few bankers who like the concept and want to hear more when you have something to show, demonstrates a certain likelihood that marketing challenges will be overcome like technical challenges will be overcome.

Similarly, for a startup hoping to make inroads for SEO, I’d be thinking less along the lines of “we’ll sprinkle some SEO on our website” and more along the lines of specific plans for scalable content generation, securing backlinks at scale, and winning the support of influencers either in the niche or in other addressable niches which your competition may not be aware are relevant to that facet of their business.

Product Supports The Marketing And Vice Versa

I have a wee little heresy as an engineer: I think that you can make a perfectly viable business out of a product which is not better than competitors, solely by improving on the method of selling it.  Farmville (and whatever Zynga has reskinned it as this week), for example, is not superior to all other options for entertainment… it just beats the pants off of most of their viral spread patterns, because promoting your use of the game is the core gameplay mechanic.  (You can also do this in more socially beneficial ways than Farmville, don’t worry.  I have a competitor in the bingo business whose product is very close in quality to my product.  They sell to schools via a catalog.  I sell to teachers via a website.  Despite solving the same problem for the same end-users our businesses are like ships passing in the night.  Hilariously, at least a few of my customers actually own both pieces of software, because the people who buy from the catalogs never bothered telling the people who use the websites.)

However, this doesn’t mean you can’t innovate on both the marketing and the product.  Indeed, since they feed off each other, that is probably substantially more effective than innovating on one or the other.  Imagine what a juggernaut World of Warcraft would be if they nailed their game’s quality as much as they did and also had Zynga’s viral loop and monetization model.  That hypothetical WoW could probably deal with Chinese net regulators by buying China.

(It’s easy to say this in retrospect: empirically, millions of employed adults with lots of disposable income spend much of their free time playing WoW.  They spend huge amounts of money on buying status for themselves — cars, diamonds, big houses.  They clearly value their experience in the game.  Therefore, they should be willing to buy status in the game, too… and since buying status is more being seen as having paid lots of money than it is about any particular artifact received, this should go over very well.  I mean, crikey, in a world where encrusted mollusk discharges say “I love you” anything is possible… anyhow, it is easy to say that in hindsight.  The challenge for startups is identifying that sort of synergy between customer adoption and the product in advance, and communicating that it is likely enough to happen to risk betting on.)

Hacking A Non-Computer System Whose Source Is Closed And Updated Continuously

We all know the first iteration of the product is going to suck (hopefully in the sense of “not meet customers needs” more than “a broken, unreliable mess”).  The first iteration of the marketing strategy is also going to suck (hopefully in the sense of “fail to generate the expected level of success” rather than “like shouting to an audience of deaf ants during a hurricane”).  Just like you can use the Lean Startup principles to modify your product and marketing message such that it comes closer to achieving a match with what some people actually need, you can also use spiritually similar disciplines to iterate on customer acquisition strategies.  There is as large a solution space in them as there is in the product space.  Maybe you need to try SEO and see that it doesn’t do a great job in your market, for your customers, while an affiliate channel performs better.  If you’re experimenting, measuring, and moving with a purpose as opposed to the traditional method of “throw stuff at the wall and see what sticks”, you will hopefully have a bit of success.

I’d love to hear if you have comments.

[Memo to self: Prior to ever actually applying for YC, I should practice thinking big thoughts and then writing small thoughts.  Those form fields are tiny!]

Quantifying The Value Of A College Degree (By Major)

“Your single best investment is your own education.” “The average new graduate is drowning in $22,000 in debt.” “A degree in English is just as valuable as a degree in Biology — it teaches you critical thinking!” “Follow your dreams and you’ll find financial success whatever your major is!”

You’ve probably heard a thousand pieces of educational/career advice like these, and if your family/friends/advisors are anything like mine they’ve virtually never been substantiated by data.  That is a shame, because choosing a course of study is one of the largest transactions you’ll ever make — the sticker price at my alma mater was close to $140,000 and prices nationwide have risen faster than inflation for virtually a generation now.  We have the Blue Book to tell you what a ten year old beater is worth down to the dollar, there are entire industries devoted to assessing every type of security to determine their valuation, and the closest thing most students have to insight on the degree selection process is getting advice from Aunt Sue.  This is insane.

Information Asymmetry In Employment Outcomes

Any college could rectify this situation virtually overnight: take that lovely list of alumni that they obsessively curate for squeezing donations, send out a two question survey (“What did you major in?” and “What was your salary this year?”), and give a sociology grad student a bowl of ramen to do some very simple number crunching.  No college will actually do this because transparency goes directly against their interests: if all degrees from a particular institution are valued at “An uncertain, but certainly large and roughly constant number”, then the standard practice of pricing them all identically makes sense.  If not, it is the academic equivalent of pricing stocks by length of ticker symbol.

(I understand many folks enjoy the non-economic component of their education.  I did and do, too, but since I’ve never spent $120,000 and signed myself up for a decade of debt for the sheer enrichment offered by attending a ballet or reading about the Japanese economy I can only conclude that I don’t value it anywhere near what I paid for my degrees.  Your objection to the narrowness of this study is duly noted, though.)

You might assume that the government would track this, somewhere, but you’d be wrong.  The Census Bureau produces a report every ten years tying level of education (associate’s degree, bachelor’s degree, master’s degree, etc) to salary, which invariably produces the result “More education is better”, but they don’t answer very interesting questions like “Is a bachelor’s degree in computer science better than a master’s in elementary education?” or “Are there fields with a sharply diminishing return to additional education?”

However, the Bureau of Labor Statistics does very comprehensive work in administering a National Compensation Survey, which gathers huge amounts of raw data about employment hours and wages broken down by region, job (over 800 classifications available, from CEOs to ship loaders), and a few other axes.  They use this and other studies to produce a variety of government reports, such as the Occupational Outlook Handbook, which does a good job of providing per-career advice but probably intentionally omits comparisons between careers.

The Bureau of Labor Statistics also makes the data from the NCS available for download on their website.  It is hefty — over a gigabyte of plain text — but it contains a virtual treasure trove of value… if you just know how to read the map.

Liberating Conclusions From Open Data

Recently, a big buzzword in the tech community has been Open Data: the notion that the huge, monstrous streams of raw facts collected by the government can be exploited for our benefit if they are merely shared.  I think this is mostly true: the best single example I’ve heard of is that since your local health authorities inspect every restaurant’s premises as a matter of course they must know where they all are located, and therefore one should be able to get those locations from them and jumpstart the creation of a guide to local restaurants without having to find every one by yourself (a monumental undertaking).

However, raw facts are uninteresting.  Here’s a line from the BLS data:

NWU009910010200000000000016260,2008,M07,69.71

Scintillating stuff, right?  What we are really interested in is what those facts can teach us — in particular, what can they teach us that allows us to make decisions such as what to major in.  This is where your friendly neighborhood computer programmer comes in: with a bit of elbow grease and a laptop, you can reduce the 856,000-odd facts in the government’s salary data to some useful conclusions about college majors.

A Bit Of Science And A Dash of Art

Sadly, there are limitations to what we can accomplish with the BLS data: it groups salary data by occupation rather than by major and degree level.  The BLS separately identifies for each occupation what the probable degree level is, but going from occupation to degree requires a bit of guesswork.  Rather than associating all occupations with sets of related degrees by hand, and injecting my own biases into the analysis, I decided to crowd source the problem: I paid Mechanical Turk workers for their two cents (quite literally) on what degree an e.g. elementary school teacher likely had.  This produced answers like Education, Early Childhood Education, Teaching, English, etc.

I then used an arbitrary level of consensus as a cutoff, and was able to pair ~60 very popular degrees (Computer Science, English, etc) and ~250 less common ones (Vocational Education, Media Technology, etc) with associated occupations.  Some additional number crunching let me construct rough estimate salary-versus-age curves for the occupations, which could then be reduced into a simple net present value calculation.  Long story short: a lot of data, a bit of science, and a dollop of absolute voodoo — it’s sort of like most social science, except I’m going to be honest about the voodoo upfront.

After doing the calculations I used Ruby on Rails and some open source graphing libraries to present the results in a comprehensible, searchable fashion — similar to the data visualizations done by the New York Times, which are some of the best work they produce.  (Check this one on the geography of the recession for the general feel.)

Why Go To All This Trouble?

Short story: Intellectual interest plus a nice paycheck.

Longer story: I do very occasional consulting work for a variety of clients.  In case you haven’t noticed from the six-figure sticker prices, offering degrees is a very big business.  Any flowing river of cash that large attracts, as if by magic, a variety of service providers around it.  In education, one major problem colleges have is finding prospective customers to sell degrees to.  This is hardly a unique problem for businesses.  (Colleges may prefer to phrase this as “students” to “award” degrees to, because they are intellectually committed to a view of themselves as custodians of the lamp of human knowledge rather than rapacious money-grubbing institutions.  I don’t know of any reason they can’t be both.)

One thing colleges — from the Ivies to state schools to online for-profit institutions — spend absolutely gobsmacking amounts of money on is “lead generation”.  Basically, since a percentage of applicants will eventually matriculate (and pay five or six figures for the privilege), when a qualified prospect fills out an application that is an economically beneficial event.  You can compare this to a conversion to the free trial of a web service.  Universities are willing to pay quite a bit of money if you can induce someone to apply: the payout varies by university and agreement, but payments in the $10+ region just for requesting an application packet are not uncommon.  (And if you had some magic way of generating sought after candidates — say, highly qualified African American students — you could almost certainly negotiate much, much higher payouts.  There might still be some Marxists on the faculty but it is all capitalists in the administrations.)

Anyhow, with universities offering to pay for lead generation, there is an entire value chain created from the ether: sites to capture the leads, affiliate programs to direct folks to the lead capture sites, advertising to attract visitors to the affiliates, publishers to create content which displays advertising, etc.  One of the publishers in the industry, Online Degrees, hired me with an open brief: make something compelling for our website.  I thought since universities, academics, and the government have failed to produce any actionable data on which degrees make sense to go after, I could do some independent research on the subject.  Online Degrees.org will host it on their website, and in the course of providing value to potential students researching the subject, they’ll have an opportunity to display ads for degree programs.

You might be concerned about the impartiality of this.  I don’t blame you.  I’ve got no particular dog in this fight: I get paid by the hour no matter what degree wins.  (Cards on the table: I have degrees in Computer Science — which is in the data set — and East Asian Studies, which is not.)

Online Degrees.org obviously has a vested interest in convincing you that a having a degree is better than not having one, but they’re agnostic about which one you apply for.  Indeed, they’d love to tell you which fields are better than others because somebody in the industry needs to have the credibility to say that e.g. culinary school is tantamount to grand theft (and most of the victims take out loans for the privilege of going through it).

Besides, do you really have a better alternative?  If I had a PhD in Sociology, would that make me a less biased source of information on the desirability of becoming a cheap source of exploitable labor a master’s candidate in Sociology?

Anyhow, I have been intellectually interested in this subject for several years now.

Quick Overview Of Results

For the results of most particular interest to you, take a gander at the degree value calculator.

Regular readers of this blog are mostly technologists of one flavor or another, and degrees in technical majors do very, very well.  Computer Science and Computer Engineering are near the top among all options for bachelor’s degrees.  It is narrowly bested by a handful of degrees tailored around resource extraction: for example, if you study Geology, Big Oil will apparently pay you Big Bucks.

Hard sciences such as Physics and Biology pay rather less well than I would have expected.  Degrees in the humanities perform about as poorly as people often joke.  The largest surprise to me was that degrees, even advanced degrees, in some caring professions (like Social Work) are apparently terrible options.  Looking at the underlying data suggests that this because many social workers do it as a part time job.  (That is a recurring theme among many jobs that I expect people would classify as more likely to be female than the typical occupation.  Food for thought the next time someone brings up the wage gap.)

You can see the results of this research on their website.  [Edit as of 2/19/2013: You’ll have to search for this directly, due to link rot.]

Questions?  Comments?  Criticisms?  I’d love to hear them.

Bingo Card Creator (& etc) Year In Review 2010

I’m Patrick McKenzie.  For the last few years, I’ve run a small software company which, most prominently, makes Bingo Card Creator.  It creates… well, you probably get the idea.  I recently launched Appointment Reminder, which… yeah.  I also do occasional consulting, shockingly not as Pay Me Lots Of Money And I Solve Your Problems LLC.

In addition to publishing pretty much live stats about the business, I always do a year-end wrapup (see: 2006, 2007, 2008, 2009) covering my thoughts on the year.  I hope folks find it interesting or informative.

Disclaimers: Stats are accurate as of publication, but the year isn’t quite over yet.  Ordinarily the last two weeks of December are fairly slow, but I would expect there to be a few hundred dollars more sales and possibly a few hundred more in expenses, depending on the timing of people charging my credit card.

My business is a good deal more complicated now than it was previously, which changes how open I can be about some bits of it.  See below.

The Big Change

After several years working as a Japanese salaryman, I quit my day job and went full time on my business as of April 1st of this year.  This was the best decision I have ever made.  Words cannot describe how happier I am with everything about my life: I see my family more often, I see friends more often, I travel more, I work less, I make more money, I’m healthier, my apartment is cleaner (well, OK, modestly cleaner), and I enjoy work a whole lot more than I ever did when I was working for somebody else.  Self-employment is awesome.  There are occasionally challenges to it, but since I spent a few years dipping my toes into the water prior to doing it full-time, I have had very little of the “Uh oh, I might not be able to make rent” uncertainty that a lot of folks report.

Bingo Card Creator

Bingo Card Creator remains my bread and butter for the time being, but I think this is likely to be the last year for that.

Stats:

Sales: 1,422 (up 38% from last year’s 1,049)

Refunds: 20 (down from 24 last year, to 1.4% of sales from 2.3%)

Sales Net Of Refunds: $42,589.90 (up 33% from $31,855.08)

Expenses: $16,685.24 (up from $15,485.28)

Profits: $25,904.66 (up 58% from $16,369.80)

Wage per hour: $200~250ish, based on my best guess of time spent on BCC (yeah, I went “full time” and work less than ever.  This is mostly because BCC is mature software.)

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 777,000 (up from 546k)

Unique visitors: 655,000 (up from 470k)

Page views: 2.7 million (up from 1.6 million)

Traffic sources of note: Google (44%), AdWords (20%), Binghoo (11%)

Trial downloads: 21,000 (down from 56,000)

Trial signups for online version: 72,000 (up from 17,000)

Approximate online trial to purchase conversion rate: 1.7%

Narrative Version:

The major change in BCC this year was unofficially deprecating the downloadable version of the software, for a variety of reasons.  This led to a huge cut in my support investment — the downloadable version generates 10 times the work per customer that the web app version does — which helped enormously when I dropped BCC into maintenance mode, which it spent over half the year in.  (Maintenance mode means I answer questions and collect payments and that is about it.)  I did a bit of experimentation over the summer in terms of conversion funnels, and also did some major work in October with regards to seasonal promotions and using my mailing list.

My conversion rate for BCC is slipping steadily.  This is sort of ironic, because it is the result of an unalloyed good thing for the business: as I get better at getting organic search traffic, the population of people using my web site moves from the teacher-heavy target-rich-environment it has historically been and towards a broader audience who don’t have quite such a pressing need for bingo cards.  Sales go up, since converting 1% of a big number is still a good thing, but it dilutes my aggregate conversion rate.  Oh well.

I’m mildly disappointed that I missed my sales and profitability targets this year (by about $2k and $4k respectively).  Oh well.

What Went Right:

  • Deprecating the downloadable version reduces my work and stress level attributable to BCC enormously.
  • Getting serious about using MailChimp and email marketing in general. My Halloween, Thanksgiving, and Christmas mail blasts made well over a thousand dollars in sales for me, for about $30 worth of virtual postage stamps and perhaps twenty minutes of writing.  One thing to note for next year: there does not seem to be a substantial difference in conversion rates for when I put a $5 off coupon in the mail versus when I don’t, so I shouldn’t next time.  Also, given that a huge percentage of folks bounce on the password screen coming from the email, I need to think about either putting a token in the URL to let them in automatically, or making dedicated landing pages for these campaigns.  (I don’t have good numbers for how effective the autoresponder sequence is — i.e. automated emails I send to people on the first and sixth days after they sign up.)
  • Meat and potatoes SEO continues to be my bread and butter (how is that for a mixed metaphor).  My conversion rate has been gradually sliding as I get more parents in for holiday bingo activities through my Sprawling Bingo Empire, but 1% of a very large number is still worthwhile.  I should probably get serious about optimizing those sites individually, but my desire is waning.

What Didn’t Work So Well:

  • In the wake of the FireSheep release, I decided to implement SSL for Bingo Card Creator, right in the middle of Halloween busy season.  This broke my pages in several different ways, from causing security popups on the login screen in IE (whoops) to not showing key images on landing pages in some browsers (whoops).  I really should have put off that implementation another few weeks, or thought through my testing strategy for it better.
  • I don’t have a staging server for Bingo Card Creator yet.  Having seen the enormous advantages from having a staging server through my Appointment Reminder, I am retroactively dinging myself for not making one in the last four years.
  • My Halloween promotion could have been handled better: $6,000 in sales is nothing to sneeze at, but I’m still of the opinion I could have broken $10k with a little better execution.  Maybe next year.
  • AdWords has been on auto-pilot for virtually the entire year, and when it goes onto auto-pilot it seems to optimize for Google’s bottom line over mine.  I should block off some time to get it under control, and aggressively cut weakly performing aspects of my campaigns.

Consulting

Last Christmas Thomas Ptacek at Matasano (whose office I am in as we speak) suggested to me that people would pay for my expertise in software marketing.  I was skeptical, but put the word out quietly that I was available for hire, and did three large projects this year for a few clients.  The only one I can publicly comment on at the moment is that I worked for Matasano, on stuff.  My general field of expertise is in engineering marketing outcomes: A/B testing, SEO, and the like.  Basically, I bring the fanatical iterative improvement mindset and apply it to things other than bingo cards.

I love talking about what I do.  Unfortunately, consulting clients pay a lot of money to get me to shut up.  This means, for example, not blabbing on the specifics of new projects which are as-yet unreleased, and not blabbing on the particulars of engagements.  Since I had only a handful of clients, giving exact numbers tends to give a wee bit too much detail, since if you happen to know one data point then you can guess the other two fairly easily.  So treat these as very ballpark numbers:

Consulting sales: $45,000

Consulting expenses: $10,000 (Plane tickets and hotel rooms get expensive quick, what can I say.)

Consulting Profits: $35,000

I know somebody is going to ask my rate, and I’m torn between a desire to quote it and the knowledge that there is absolutely nothing good that can come of quoting it.  The reality for consultants is that clients pay you exactly how much you can negotiate with them.  Everybody knows this and yet everybody would be shocked, shocked to know that someone else got a better deal.  In addition to causing problems with existing clients, it complicates raising rates for new clients in the future.  (Given that I could fill 100% of the hours I wish to, have been saying ‘Nope, sorry, not available’ to interesting work frequently, and now have a few CEOs singing my praises, my rates are only going up next year.)

What Went Right:

  • Client selection.  All three clients knew me well from the Internets, all had confidence in my ability to do what they wanted me to do, and all were model clients: they knew what they want, they communicated perfectly, and they paid all invoices in a timely fashion.
  • Pricing: Aside from frightening my bank a few times when I got large wire transfers from America, charging a lot of money is a great idea in every possible way.  It makes your clients respect your time more, keeps you motivated, and helps pay the rent during the lean summer months of the bingo calendar.

What Didn’t Work So Well:

  • Newbie consultant blues: I did my first consulting project for a friend.  Unfortunately, due to a combination of it being my first gig ever and my first experience with using Heroku, I greatly underestimated the amount of time the project would actually end up taking.  What I thought would be 20 hours over a two week period stretched into many more hours over months and months.  Luckily, my client was sympathetic, but I ended up doing a lot of writeoffs for good will and diverting my attention for longer than I wanted to.
  • Juggling consulting with project work: I worked 90 hours a week and still had cycles to spare for BCC.  How hard could it possibly be to do 10 hours of consulting in a week and still get productive work done?  Very freaking hard, because the 10 hours tended to be splayed across several days, require creativity and focus to execute, and not include the whole email/sales/support dance that comes with consulting.  This is in no way a criticism of the client, it is just to illustrate how consulting works: I got a request from a C-level exec at a company you’ve heard of in April, and despite us being mutually enthusiastic about the project it took forty emails and five months until billable work actually started on it.  I had to gracefully extricate myself from my clients and block off November and December to get Appointment Reminder launched.

Appointment Reminder

I finally released Appointment Reminder in early December.  (Don’t worry, I haven’t forgotten about writing the rest of the series about developing it.)  I have one customer already and a handful of prospects currently trying the software out, so revenue is negligible as of yet.  Now I just have to do the other 90% of marketing the software, which I am going to do a bit of in December and then start in earnest in January.  The plan is, unsurprisingly if you know me, heavily reliant on organic SEO.  I have also had a lot of interest about whitelabeling the software, and will do that as well.  That gives me a built-in on-the-ground sales force of people with intimate knowledge of potential clients and the desire to sell them my service — and judging from the numbers thrown around by one of the guys interested in white labeling, that could be “quite lucrative indeed.”

In terms of technical direction, I’ve engaged a freelancer to get it working on iDevices (I could have done it myself, but I’ve got plenty on my plate as it is).  I am about 60% of the way to getting a very impressively hacky solution working on every major US smartphone, because many prospects have asked to be able to access schedules when on the move.  The implementation is so devious that if I had a mustache, I would be stroking it while cackling maniacally.

Semi-exciting news: I received an acquisition offer from a foreign telecommunications firm.  The CEO believed AR fit a hole in their product line, and offered me their estimated development budget for it if I would sell them the whole business.  That was a generous number relative to the amount of work I have put in, but it would not have been lifechanging for me or my family.  I declined and told him I’d like to try to run the business myself and see where it takes me.

The stats:

Sales: Nothing yet.  (Well, one test transaction to make sure Paypal/Spreedly works.  Spreedly is impressively painless, by the way.)

Expenses: ~$1,600 (a few hundred bucks in design work, $800 of Twilio credit, and one or two other things.  Servers and online services which I also use for BCC got filed under BCC because I’m lazy and my bookkeeping software doesn’t support splitting bills: a more accurate accounting would be closer to $3,000.)

What Went Right:

  • It exists and mostly functions! These are both handy properties in software one wants to sell.
  • MVP available for several months.  I didn’t have the cycles to create AR back in April/May, but I did get the MVP of it — basically, an interactive demo of the core of the service — out early.  This helped me in both getting a bit of age and trust on Google prior to the official launch, and also in getting a base of interested prospects to try selling to (I’m currently talking to a few of them).

What Didn’t Work So Well:

  • Delaying release: April, May, June, July, August, September: that is six months, and I got very, very little accomplished (not one line more than the MVP, as a matter of fact).  The distraction from consulting work, working on BCC, and reacclimating myself to a human existence after years of salarymanhood just totally destroyed any desire to do heavy lifting on a new project.  I’m very obliged to the HN-based folks who started a Launch Something in November mini-sprint, which helped get me the energy to actually sit down at the computer for six hours a day five days a week and bang it out.
  • Insufficient pre-launch marketing: I really should have been constantly adding new content to the website for the last six months.

Goals for 2011

Bingo Card Creator

  • Sales of $60k, profits of $40k
  • Getting AdWords back under control
  • Getting the holiday promotions ready for all those domains I own but have never successfully exploited yet

Appointment Reminder

  • 200 paying customers by December of 2011.  This implies the revenue run rate will be somewhere north of $10k then.  Cross my fingers: it might be well north of that, if SEO or white labeling works well.
  • I’m about 70% certain that I’m going to hire a front end developer for AR.

Consulting

  • Perhaps a wee bit more of it.
  • At higher rates.
  • Knock things out of the park for clients #1 ~ #3, and tell stories when possible.

See you all in 2011!

Staging Servers, Source Control & Deploy Workflows, And Other Stuff Nobody Teaches You

I worked for almost three years as a cog in a Japanese megacorporation, and one of the best parts about that experience (perhaps even worth the 70 hour weeks) was that they taught me how to be a professional engineer.  Prior to doing so, my workflow generally involved a whole lot of bubble gum, duct tape, and praying.  I spent a lot of time firefighting broken software as a result, to the detriment of both my customers and myself.  Talking to other software developers has made me realize that I’m not the only person who was never taught that there are options superior to bubblegum.  If you aren’t lucky enough to work at a company that has good engineering in its very DNA, you’re likely to not know much about them.

This strikes me as the industry’s attitude to source control a few years ago, prior to a concerted evangelization movement by people like Joel Spolsky.  It is virtually impossible to overstate how much using source control improves software development.  Our industry has changed in major ways since 2000, but our best practices (and knowledge of those best practices) are lagging a few years behind.  We could really use a Joel Test 2010 edition, for a world where “you should have build scripts for desktop software which can complete the build in one step” is largely an anachronism and where the front page to the website is no longer hand-coded in Notepad but, rather, is a shipping piece of software which can break in two hundred ways.

You’re not going to get the Joel Test 2010 here, mostly because I’m not Joel and there is no particular reason any company should judge its development practices relative to mine.  What I would like to give is some practical pointers for implementing three practices which, if you’re not already doing, will greatly improve the experience of writing software for the web:

  1. Staging servers
  2. Version control workflows
  3. Tested, repeatable deployments

Staging Servers

What is a staging server?  The basic idea is that it is staging = production – users. (If you’re Facebook, Google, or IMVU, you are lightyears ahead of this article and have some system where there are multiple levels of staging/production and where you can dynamically change them.  You already have geniuses working on your infrastructure.  Listen to them.  This article is for people who don’t have any option between “code runs on developer’s laptop” and “code runs in production.”)

Why do we have staging servers?  So that anything that is going to break on production breaks on the staging server first.  For this reason, you want your staging server to be as similar to the production environment as you can possibly make it.  If the production environment processes credit cards, the staging environment processes credit cards.  This means that if, e.g., your configuration for the payment gateway is borked, you’ll find out about that on the staging server prior to pushing it live to production and, whoopsie, not actually being able to get money from people.  If your production server uses Ruby 1.9, your staging server uses 1.9.  If the production server uses memcached on port 12345, the staging server uses memcached on port 12345.

(Many folks have systems which exist on more than one physical machine.  I don’t — I’m a small business where 2 GB of RAM is enough for anything I want to do.  If you have multiple machines, strike “staging server” and read as “staging system” below: all the benefits for having a separate staging server are still beneficial when your staging environment actually has fifteen physical servers running 47 VMs.)

Setting up a staging server should be easy.  If it is not easy, you already have a problem in your infrastructure, you just don’t know it yet: you’ve cobbled together your production server over time, usually by manually SSHing into it and tweaking things without keeping a log of what you have done.  (Been there, done that, got the “I Created A Monster” T-shirt.)  There isn’t a written procedure or automated script for creating it from the bare metal.  If you had that procedure written, you should be able to execute it and create a staging server that works inside of an hour.

Most people won’t be able to do this if they haven’t given thought to the matter before.  That is fixable, and should be fixed.  It has substantial benefits: if you have a repeatable procedure for provisioning a production system, then when disaster strikes you will be able to confidently execute that procedure and end up with a production system.  (Confidence is important since you’ll probably be terrified and rushed when you need to do this, and rushed terrified people make unnecessary mistakes.)

If you’re working on Rails, I highly recommend using Deprec/Capistrano with all new projects.  In addition to making it very easy to get a full Rails stack working on your deployment environment of choice , it helps automate routine deployment and server maintenance, and has mostly sensible defaults.  (I have only one quibble with deprec : it installs software from source rather than using your system’s package manager.  That means that upgrading e.g. Nginx two years down the road is needlessly hard and error prone, when instead you could just have used apt-get in the first place and then updating is a piece of cake.)

You can also use Fabric, Chef, Fog, or a similar system to script up building new environments.  Pick whichever strikes your fancy.  Try to recreate your production environment, down to a T, on another host at your VPS/cloud/etc provider, or on another physical machine if you actually still own machines.  Keep tweaking the script until it produces something which actually matches your production environment.  You now have a procedure for creating a staging server, and as an added bonus it also works for documenting your production environment in a reproducible fashion.

One nice thing about keeping your server configuration in scripts rather than just splayed across fifteen different places on the server (/etc/environment, /etc/crontab, /usr/local/nginx/conf/apps/AppName.conf, etc) is that it lives in source control.  Your cron jobs?  If they’re in source control, you’ll have a written record of what they are, what they’re supposed to do, and why they just blew up when you bork the underlying assumptions eight months down the line.  Your Nginx config?  If it is in source control, you’ll understand why you added that new location setting for static images only.  The voodoo in your postfix config?  A suitably descriptive commit note means you’ll never have to think about reproducing the voodoo again.

After you have the script which will produce your staging environment, you probably want to make a minimum number of alterations from production.  Many companies will want their staging environment to be non-public — that way, customers don’t see code before it is ready, and critical issues never affect the outside world.  There are many ways to do this: ideally, you’d just tweak a setting on your firewall and bam, nobody from the public Internet can get to your staging environment.  However, this is a wee bit difficult to pull off for some of us.  For one, I don’t actually have a hardware firewall (I use iptables on each VPS individually).

My staging environment simply includes a snippet in Nginx which denies access to everyone except a particular host (which I can proxy through).  This breaks integration with a few outside services (e.g. Twilio and Spreedly, which needs callbacks), so I make exceptions for the URLs those two need to access.  The more complicated your staging server configuration gets relative to production, the more likely you are to compromise its utility.  Try to avoid exceptions.

That said, there are a couple that are too valuable to not make.  For example, my staging server has a whitelist of email addresses and phone numbers owned by me.  Through the magic of monkeypatching, attempting to contact anyone else raises an exception.  That sounded a little paranoid until that day when I accidentally created an infinite loop and rang every number in the database a hundred times.  (My cell phone company loves me, but folks who accidentally collided with test data sure would not have.)

How do you get data to populate the staging server?  I use seed scripts and add more data by hand.  (I also have DB dumps available, but they tend to go stale against the current schema distressingly quickly: I recommend seed scripts.)  You can also dump the production DB and load it into the staging DB.  Think long and hard before you do this. For one, it is likely to be way, way the heck out of bounds for regulated industries.  For another, your staging server is probably going to periodically be insecure — insecurity is failure and failure is what the staging server is for.  Slurping all of the data out of a staging environment has caused many companies smarter than you to have to go into disaster management mode.  Please be careful.

So you’ve got a staging server?  Now what?

At the simplest, you access your staging server with a browser and try to break things.  When you break things, you fix things, then you redeploy the staging server and try to break them again.  This is what you are probably doing right now with production, except that your customers don’t have to see broken things when you break things.

Eventually, you can script up attempts to break things, using e.g. Selenium.  Then when you break things, you add them to the list of things that Selenium tries to break.  If you run that against the staging server after every code check in (a process known as continuous integration), you’ll quickly catch regressions before they disrupt paying customers.  This is a wee bit harder than just having a staging server — OK, a lot harder — but you’ll get clear, obvious advantage out of every increment of work you do on this path, so don’t let present inability to be Google prevent you from getting started.

Version Control & Deployment Workflows

Everyone should use version control, but people tend to use version control differently.  Git is very popular in the Rails community, but there are probably no two companies using git the same way.  The key thing is that you agree with your team on how you use version control — document your assumptions, document your processes, then apply them religiously.  This will reduce conflicts on the team, reduce mistakes, and help you get more out of your tools.

There are a million ways to use version control and most of them are perfectly OK.  I’m going to mention mine, but it isn’t the canonical Right Way, it is just one way which works for a (very) small company.  Yours will likely be different, but you can see some of the things which go into design of a version control workflow.

Assumptions I Make About Life, The Universe, And Everything

  1. I use git.  Git has notion of branches, tags, and remotes (physically distinct repositories) — if you don’t know what these are, Google for [getting started with git].
  2. I generally work alone or with a very small team. (This assumption underpins very important parts of my workflow.  It won’t expand very well to a 200 man distributed team, but it might well work for 2 ~ 5 people.)
  3. There is exactly one canonical repository, origin.  Developers maintain other repositories on their workstation.  Automated processes like deployment happen only with reference to the origin.  Code existing outside of the origin does not officially exist yet, for any purpose.  The history preserved in the origin is, in principle, sacred.
  4. There is a branch called deploy.  The HEAD of deploy (the most recent code on it) is presumptively ready to be put into production.
  5. Tags are used to take snapshots of the code base and preserve them in amber with a human readable name.  Right before we deploy to either production or staging, the HEAD gets tagged, so that we can easily find it later, with a simple naming convention (I use production_release_X and staging_release_X, where X just increments upwards — some people might prefer timestamps).  Production release tags are never deleted.  Staging tags get periodically culled when convenient to do so.
  6. Development of any feature expected to take longer than a few hours happens on a feature branch.  (I do occasional work right on deploy locally, for issues of the “Minor copy edit on dashboard.” variety.  This would be one of the first things to go if I were working on a larger team.)

So how does this actually work in practice?  Let’s say I’m implementing a new feature.  I create a new branch to work on.  I code a bit, creating local commits with wild abandon any time I have accomplished something which I don’t want to lose.  When I believe code to be functional, I fire a capistrano task which tags the current head of my branch, pushes that tag to origin, and deploys it to the staging server.  I then continue testing on the staging server, for example verifying that Twilio integration actually works with Twilio (which cannot conveniently access localhost:3000 on my laptop).  I continue writing code, committing, tagging, and pushing to the staging server until the feature is ready.

Then, I switch back to the deploy branch and merge in my feature branch (with –no-ff, which creates a commit message just for the merge — this handily groups the twenty or thirty commits I just made into one easily readable story for myself later).  I then tag a production release (manually — this is entirely to force me to think through whether I’m ready for a production release), verify that there is no diff between it and the most recent staging release, and then push the new tag to origin.  I then fire the Capistrano task which checks out the new deployment tag and restarts the server.

What does this get me versus my previous SVN workflow for Bingo Card Creator, which was “Work only on one branch, commit stuff when I think it is ready, and deploy the trunk manually on occasion”?

  1. I cause much less downtime for the production server due to reasons like svn commit -m ‘Whoops, forgot a setting in production.rb’ and svn commit -m ‘r1234 introduced dependency on foobar gem without putting it in environment file, causing rake gems:install to not load it.  Mongrels then failed to restart.’
  2. My deploy branch has a relatively clean history, so when things start to break next year in production, finding the change sets which eventually caused the breakage will be less of a needle in the haystack search than finding them in SVN is.  SVN’s history is 1800 unedited commits, recording my stream of consciousness as they happened.  My stream of consciousness is frequently stupid, particularly when I’m panicking because the server is down.
  3. This decouples the staging server from production in a clean fashion (so that I can advance the staging server a feature or three at a time if I want to), but guarantees that when I’m actually ready to deploy, I’m deploying exactly what did not break on the staging server.
  4. Tagging releases gives you an Oh Crikey button, as in Oh Crikey, that last release broke stuff.  You can quickly rollback the deploy to a known good tag, isolate the changes which broke production, and fix them.
  5. Deploy scripts manage releases with multiple moving parts a lot better than I do, even when I’m working from a checklist.

By the way, git gives you many options for recovering from problems — even severe problems — without requiring either gymnastics or a full-blown CSI investigation to discover what happened later.  For example, let’s pretend I just deployed tag production_deploy_82, and have discovered some issue serious enough to require an immediate rollback to production_deploy_81, which is known to be good:


#Assuming we are on our local workstation on the deploy branch.

git branch something-in-here-is-broken

git reset --hard production_deploy_81  #All changes made between deploy 81 and 82 just vanished from the deploy branch locally.

#Clean up the deploy, using any option discussed below.

git checkout something-in-here-is-broken  #Those changes which you just disappeared are now living on this branch, ready for you to fix.  After you've fixed and verified it works on the staging server (and, ahem, that you have addressed the issue that allowed this to get OKed for release last time), you merge this branch back into deploy, and do a tag-and-release cycle.

How you clean up the mess on the server is up to you: good options include “deploy 81 again”, “tag a release 83 equivalent to 81, then deploy it”, and “rollback to the copy of 81 which still exists on the server.”  (Capistrano includes deploy:rollback, which will do exactly this.)  Any of these will work, just always do it the same way to avoid stepping on each others’ toes.  I prefer tagging a new release so that I can add a descriptive message explaining why 82 just created an emergency.

This is important because it leaves a paper trail — if you’re pulling a release from production, something just went seriously wrong with your processes.  Emergencies are not supposed to happen – anything that lets an issue get that far isn’t just a one-off failure of whatever broke, it is a series of failures of the systems/processes designed to prevent failures from getting that far.  After you’ve put out the fire, investigate what went wrong and tweak your processes such that a similar failure in the future gets caught prior to bringing down production.  The sleep you save may be your own.

Scaling this to more programmers: Do whatever works for you!  I would probably create a staging branch and have folks integrate stuff into the staging branch when it was ready to go to the official staging environment.  I also might make per-developer staging environments: since creating one from the bare metal is supposed to be essentially free, let them all have their own where they can be reckless without spoiling the experience of other developers.  We can worry about code interaction on the “real” staging server.  Then, have folks communicate when they consider everything they have on staging ready for release, and release when everybody says it is ready.

The important thing is that, whatever process you use, you document it, teach it, and enforce it.

Stuff Your Deployment Script Might Not Do Today But Probably Should

  1. Depending on your scale and how you use e.g. memcached, it might be safe to purge the cache on every re-deploy, which will prevent some hard to diagnose bugs.  At a certain scale, this is virtually a recipe for taking your site down in a cache stampede, but I’m not Facebook and having capacity problems means that I am probably already vacationing at my hollowed-out volcano lair.
  2. Tell everybody on the team that you just deployed.  I know some teams who have an IRC channel with a bot who announces redeploys.  A quick email CCed to five developers also probably suffices.
  3. Restart worker processes.  This is easy to forget but, if you do it by hand, you’ll eventually forget and then have two versions of the application in production at once.  If you’re not prepared for that, it will bite you on the hindquarters when, e.g., the application servers ask the workers to execute methods that the workers do not know now exist in the code base.
  4. Do sanity checks.  You can go arbitrarily deep with complexity here.  For a first cut, mine for Appointment Reminder restarts the application server, counts ten seconds, then tries to access an internal URL.  If the application server isn’t up, or if the action at that URL blows up for any reason, the deployment script fails the deploy, rolls back to a known-good version, and sends me a very crossly worded email.  (You can do this for the staging server, too.)
  5. Integrate with other systems which manage the state of your code/business.  For example, I use Hoptoad.  Hoptoad keeps track of exceptions and mails you when they happen, in such a fashion that your inbox doesn’t get buried by e.g. Googlebot deciding to do an impromptu fuzz test on your website.  I mark all exceptions as resolved every time I deploy to the environment they happened in.  You could also e.g. update an internal wiki by adding a new page specific to the deployment, automatically update your bug tracker to change the status of the bugs that you (presumably) just squashed, or start a new cohort for your stats tracking.

Is There Anything You Would Like To Add?

The trend towards openness with regards to technical practices on the Internet is a major, major win for everybody in our industry.  Best practices do not have to be passed from master to apprentice as oral lore.  Like OSS, since they’re often glue rather than competitive advantages (for many companies — not all), sharing them mostly doesn’t hurt you and can improve the situation of everybody.  So, in that spirit, if you’ve got anything you’d like to add, particularly for how you do things or how you would adapt this advice outside the scope of a very small business, I’d love to hear about it either in the comments or on your own blog (feel free to leave a link if it is relevant.)

Appointment Reminder Has Launched

Appointment Reminder has launched.  I will continue blogging about the development process (and post-launch stuff) later, but at the moment I am up to my eyeballs.

Getting A New Product Off The Ground: Part Two

Along with some other users from Hacker News, I’m doing my darndest to get Appointment Reminder into the hands of customers by the end of November.  It looks like it is going to happen, too.  (There was an early blog post about this here.)

Rails Deployment Options

Somebody asked me to talk about this subject, so I will cover it briefly.  Rails deployment has historically been very painful.  These days it is a little better.  The big options are:

  • Run a Rails stack on a server or VPS under your control
  • Use Heroku
  • Run a Rails stack on Amazon EC2

I have used Heroku for a client project before.  It is a wonderful service, don’t get me wrong, but I have almost five years of managing production Rails applications on a VPS and less than five weeks of managing production Rails applications on Heroku.  I know from experience that I will try to do things during development where Heroku’s limitations (like the read only filesystem) will trip me up, and tripping myself up doesn’t get things in the hands of my customers any quicker.

Additionally, my anticipated traffic for Appointment Reminder is way, waaaaay within the capacity of single largish Slicehost slice to handle.  If I manage to bring my Slice to its knees, it will be because my revenue has hit several million dollars a month.  That will give me many attractive scalability options, such as hiring someone to care about scalability while I sip chilled juices on the beach of a tropical island.

I built a staging server, from scratch, and obsessively documented the process.  (I strongly recommend that you do this for building servers, since you will have a burning need for that documentation if something ever goes wrong.)  It relied heavily on the Deprec gem, which is far and away the easiest way to get a Rails stack running on a bare Ubuntu install.   I still spent almost 4 hours filing down sharp edges, though.  (Sample: I wanted to run Rails 2.3.10 rather than upgrade to Rails 3, there were issues with compiling Mongrel on Ubuntu that had to be resolved via manually grabbing packages, etc etc.) I only really recommend this if you know what you’re doing with Rails system administration.

In terms of software choices for my deployment stack:

  • Ruby 1.8.7 via the Matz Ruby interpreter.  It isn’t the fastest, but speed is hardly of the essence to this application and it is the least likely to die horribly when using a gem or library, since everybody tests against it.
  • Rails 2.3.10.  I don’t want to learn Rails 3 at the moment.  Maybe next year
  • Mongrel cluster: I have experience managing it.
  • Nginx: The best web server I’ve ever had the pleasure of working with.  Also plays well with PHP, which I need on the box to handle the marketing portion of the website outside of the application.  (The marketing site is in WordPress.)
  • God process monitoring: I have experience using it.
  • DelayedJob job queuing: I have experience using it.

I will probably eventually put WordPress on a physically separate server (hey, it’s WordPress), but that doesn’t strike me as the biggest priority at launch.  I did go to a wee bit of trouble to secure it:

  • Access to the admin directory is denied at the server level for requests not from my private network.
  • WordPress has its own database and database user, and can’t touch anything elsewhere.
  • The WordPress directories are only writable by root, not by the web server user.  If I want to install plugins, I get to SCP them to my account on the server then SSH in and start sudoing to actually copy them in.  Ditto for uploading files.  Facilities to upload and immediately execute PHP code are, ahem, a persistent source of vulnerabilities.

Code Is About 70% Complete

I have sustained pretty close to my lifetime peak productivity at programming (say that five times fast) for the last two weeks, averaging 4 to 6 hours per weekday.  Appointment Reminder has most core functionality implemented on the staging server, for a single single-user account.  Twilio has been an absolute joy to work with — I really can’t express how much of the pain they take out of running a telephony company.

What’s done:

  • Client creation/management: 100%
  • Appointment creation/management: 90% (still need to associate scripts with appointments)
  • SMS reminder loop: 100% implemented.  (web app -> Twilio -> user’s phone -> web app)
  • Phone reminder loop: 100% implemented (web app -> Twilio -> user’s phone -> web app)
  • Dashboard: 90% implemented.  (Needs to be a wee bit more configurable, particularly for multi-user environments.)

Things are looking fairly slick thanks to jQuery UI and a lot of AJAX, but it has been a bit of a handful to test.  If you have suggestions for doing so, I’d love to hear about them in the comments.  Mostly I have been doing unit testing for models which are likely to have failures, and then doing manual testing to verify that the UI works how I expect it to.

I am currently using an absolute riot of colors on the dashboard to convey status information.  That probably should get reduced.  SASS has been very, very helpful in keeping me from having to handwrite CSS to manage all of these.

What still needs work:

  • Email reminder loop: 0%, should take ~4 hours for simplest thing that will work
  • Script creation/management: 0%, should take 1~2 days (this requires hooking up another phone system)
  • User/account management: 0%, should take ~2 days
  • Reports: 0%, should take ~1 day for simplest thing that will work
  • Feedback to user on appointment status changing: 0%, 2~4 hours depending on how full-featured I want to launch with
  • Charging people money (using Spreedly — shouldn’t take more than a day)

Outsourcing Up A Storm

I have had a lot of luck using Fiverr for getting recordings done cheaply.  Try the Appointment Reminder demo to hear one of the voiceover artists in action.  My plan is to launch with about four different sets of voices to give customers some variety: I expect that most will actually record their own reminder scripts, but anything which decreases the amount of work it takes before they get a phone call from the system is a win for me.

I also plan on eventually using reminder scripts as a cheap way of marketing.  There are virtually infinite marketing angles once you have the ability to make a phone ring, since it is so ridiculously compelling.  (I mean, people paid how many billions of dollars for ringtones?)  For example, I have an absolutely unhealthy interest in the Old Spice Man.  Getting a script inspired by the character is easy, remarkable, and is both a win for my customers (“That phone call you gave me earlier was hilarious!  Thanks!”) and lets my customers market AR inside their organizations (“Cindy, you have to hear this.  Quick, what is your phone number?”)

There are virtually infinite variations on that: you could imagine getting a phone call from a Humphrey Bogart impersonator, etc etc.

Speaking of outsourcing, I recently started using a Virtual Assistant, something I have been meaning to try since reading about them in Rob Walling’s book.  I found mine through an agency called Pepper (that name is a bit of an easter egg for comic book geeks in the audience).  For about $300 a month, I get twenty hours of time from one particular lady who works at their office.  She does, within reason, anything I tell her to do.  This has been a huge win for me in cleaning boring, time-sucking tasks off my plate so that I can concentrate on development and marketing.

For example, up until quite recently I was ten months behind on bookkeeping.  That’s a long story: I do expenses bookkeeping via a home grown application so that I can display nice graphs on my website, and that application got broken by an unrelated update back in February.  (Bookkeeping of revenue is totally automated, and didn’t break.)  Since customers don’t pay me to do bookkeeping, fixing it went on the back burner.  I eventually got it working again sometime in summer… and then had six months of credit card statements to process.  Well, that sure sounds like work, so I procrastinated… until there were ten months of statements to process.  Crikey.  It kept getting worse, it kept weighing on me that it wasn’t done, and I didn’t want to spend a day or two just going through 20 PDF files and typing some 300 transactions into the computer.

Enter my Virtual Assistant.  I spent ten minutes typing up my decision tree for classifying transactions (the credit card statements have mixed business and personal charges — whoops, got to fix that one next year), instructions for operating the bookkeeping interface, and the like.  Then I created an account for her in the system, zipped up the 20 statements, sent them over, and answered an email or two from her over the course of the next week.  Managing her and reviewing her output took perhaps 30 minutes in total, instead of somewhere on the order of three to six hours to do the data entry myself — plus, it didn’t totally wreck me for other productive work that day.

I’m now almost caught up with bookkeeping (got another statement delivered by my bank in the interim), and now that there is a process in place for it keeping caught up is simplicity itself: download statement, shoot it to my VA, go on to do things that actually matter.

This was astoundingly beneficial for me.  I have a nagging worry taken off my plate and a process to make sure it never comes up again.  I got to invest several extra hours in things which matter to the business, instead of doing data entry.  That covers the first week of the month: if I never got another stitch of work done, I’d consider my $300 well spent, but I’m guessing that I can almost certainly find other stuff for her to do.

When I get some time to breathe this month, I’m going to figure out other tasks I can assign to her: perhaps drawing up lists of keywords or blog post topics.  I mean, I could spend a day coming up with every possible permutation of a professional service plus every possible synonym for “scheduling software”, but that doesn’t require my personal attention.  (Why I would want a list of hundreds of keywords about my software is fairly straightforward if you’ve followed my bingo card creation endeavors: implement system to create content at scale, farm out content creation to freelancers, cover the organic search longtail for the topic, sell thousands of accounts to organic searchers).

Speaking To Customers

Dharmesh Shah, I think, said that there are phone people and email people, and that most engineers are email people.  I am definitely an email person.  I hated speaking on the phone even before I started using the Internet.  But I’ve been chitchatting with prospects (at Ungodly O’Clock AM) for the last couple of weeks to hear about their needs, and have both learned stuff I didn’t know about my market, confirmed some stuff I did know, and — importantly — gotten some promises to pay me money immediately when the software is ready.

Here is my mental model of the typical customer for Appointment Reminder: Martha owns a massage therapy practice.  If Cindy, her client, doesn’t make her 4:00 PM massage, then Martha loses $60 of revenue immediately and will likely be unable to rebook the slot, since she’ll find out about the gap at about 4:05 PM.  Appointment Reminder either gets Cindy in on time or gets Martha 24 hours of notice, so she can book the slot.  Martha happily buys the service.

Note the embedded assumption there: Appointments are something you go to.  I never even realized I was assuming that.  It turns out that I was, and that there is a broader market than I expected, for appointments where the service provider comes to you.

Why does this distinction matter?  Because Martha is greatly annoyed when her customers don’t come in, but she has not literally lost money out of her pocket.

Consider Earnest the Exterminator.  He has a team of three guys and a van filled with lots of dangerous chemicals.  When Earnest makes an appointment, he finds out that you’ve forgotten after he drives 25 miles to your house.  Not only did Earnest just lose out on the revenue from killing your bugs, he just burned up three hours of salary at skilled labor rates because you forgot that today was, indeed, Tuesday.  Martha is greatly inconvenienced by her no-show problem.  Earnest is sputtering with rage about his now-show problem.  I now know this because I’ve gotten on the phone with three different Earnests now and just shut up while they talked about their problems.

I’m going to make changes to my marketing posture to reflect this (the Marthas of the world are overwhelmingly female, but there are an awful lot of guys in the Earnest category along with some ladies), and I’ve also started to create some features for supporting Earnest and his team.  Some of my Earnests have been surprisingly tech savvy (“Oh, my programming team wants to know about your API.”  “You have a programming team?!?”  “Bleep yes I do, you think I’m going to run the systems myself?”), and the knowledge that the user of the software could very possibly be mobile rather than at her place of business really rejiggers my development priorities for e.g. checking your schedule via a phone call.

Plus, as you can probably guess, somebody who has a programming team and a very good idea for how many tens of thousands of dollars they lost last year to no-shows is, shall we say, quite willing and able to pay any price I want to charge.

Next Steps

More of the same, really.  I just got confirmation from one of my freelancers that she is working on the MP3 files for phone calls.  Tomorrow I build out the script interface to allow people to pick or record the telephone/SMS/email scripts that they use.  After that is done, I think I’ll do the email integration, which is the last bit of actual functionality for the site.  After that, user management (if worse comes to worse, all I really need is to have users have separate settings, and I can do all creation of secondary users by waiting for emails from customers and then doing it via the Rails console rather than with a slick UI on top of it).

If you have anything in particular you’d like me to cover in the next installment, say the word.

Getting A New Product Off The Ground: Part One

There was overwhelming enthusiasm from people when I offered to blog about the development of Appointment Reminder, so I will be doing it during the month of November, prior to my planned release.  (Tentatively planned for the end of the month, if it is ready in time.)

Achieving Activation Energy

Appointment Reminder has, theoretically speaking, been on my plate since April 1st, which was my first day of self-improvement.  I released the MVP — basically, a functioning demo of the core interaction between service provider and customers — halfway through May.  And then I sat on my hands for about six months.

Oh, stuff happened, don’t get me wrong.  I went travelling internationally, twice.  I kicked butt and took names for some consulting clients, got (and turned down) about a dozen job offers, won an award for best presentation at the Business of Software conference, started writing an article for the ACM, met a young lady, and broke my old bingo card sales record.  But while my life is firing on all cylinders, happiness does not write jQuery or Rails code.

My last client project wrapped up in mid-October, I told clients I would be mostly unavailable for the next few months, and I picked the end of November as an arbitrary deadline to finally get Appointment Reminder out the door.  Deadlines tend to coerce me into doing my best work.  Specifically, deadlines with subgoals that have small units of measurable accomplishment work best for me.  It must be the WoW player in me, I swear.  So I broke the month or so of work up into a series of mini-quests which take about half a day to accomplish, and then logged the first dozen or so in EpicWin (productivity management for recovering WoW players — my other quests include going to the gym, doing the dishes, and calling each of my younger brothers).  I have been polishing them off more or less according to plan.

Technology Choices

When I start a greenfield project (and AR is still new enough to be mostly greenfield), one of the first things I write down in the project notebook is what technology stacks I’m good with and which make a fit for the product.  Given that my options for web development are Big Freaking Enterprise Java and Rails, Rails was the clear winner.  I am literally an order of magnitude more productive in Rails than I am with Java, and I lack the experience with Java architecture astronomy to build an application from the ground up (which I have done successfully in Rails before).

I sometimes also use the opportunity of new projects as an excuse to broaden my horizons.  For example, I’ve wanted to get into jQuery for a while since the world seems to be moving that way (away from Prototype, my old Javascript framework of choice), so I decided to take the minor productivity hit with starting a new framework and moved to jQuery.

My other professional growth goal with AR was to get a little more serious about interface design.  Thomas Ptacek has been raving to me about how much better SASS/Compass made the experience of getting CSS to work right, so I decided to move to SASS/HAML from my previous CSS/ERB standbys.  This delivered huge, huge wins within two days of starting exploratory coding with the project.  I can’t recommend SASS enough.  This also led to a bonus: the lack of markup going into my views means that I can develop against a cheapo CSS template and then swap to a professional design later, without having to have the design be on the critical path for November.

Design on Paper

I generally keep all notes for a project in a $1 notebook.  However, my first Appointment Reminder notebook is nowhere to be found.  Drats.  I probably left it in a hotel room in America.  So I bought a new notebook and started sketching out database tables, screens, features I would like to include, etc etc.  Then I started cutting, ruthlessly.

For example: Users need to be able to manage clients.  OK.  Hypothetically, they may have hundreds or thousands of them, so they’ll need a search interface… but will they have hundreds or thousands on launch day?  No?  Search is out of scope.  Next!

Users need to be able to give custom reminders to clients.  This requires recording their voice.  Uploading files is a pain in the keister — out of scope!  I’ll make do by having them just narrate the reminder over the phone to Twilio, which would obviate the need for me to do any file encoding or management myself.

And so it continued.  By the end of my first working lunch, the feature list that was in scope ended up looking something like this (broken down roughly along controller/model lines):

Accounts

Account creation

User management

User permissions (Enterprise-y feature, enterprises aren’t going to adopt in first two weeks: out of scope.)

User preferences

Login / Logout / Forgot Password

Clients

CRUD app for users to add clients

Track phone numbers and email

Communication preferences

Opt out of reminders in case of abuse (Barely avoided being out of scope: it is important but doesn’t sell software.)

Schedules (one schedule manages one resource — a room, service provider, etc)

Default schedule (most users will have only one)

CRUD app for schedules

Change hours of business day, days of business week.  Eventually that might change on week-to-week basis, for now, simplest thing that works.

Appointments

Appointment calendar — starred several times because this is going to be the hardest part of the app

CRUD for appointments, based on calendar

Status tracking for appointments

Reports for number of appointments missed, etc Can’t be used on day one, out of scope.

cron job to send reminders about appointments

Reminders

Spawned automatically from appointments

State machine (see below)

Twilio integration

Twilio

Inbound calls

“Who the heck are you and why did you call me?” auto-response

Reset trial (already implemented)

Record custom reminder

Get schedule for day dictated over phone great feature, out of scope for now

Voice-assisted tour only in scope if I have two days left over

Outbound calls

Reminders

If answered live, capture user input and update Appointment state accordingly (coming, canceled, needs call, etc)

If answered by machine, leave message, update Appointment state as notified.

Reminders with custom recordings

Text messages

Email

Thank you for signing up, blah blah blah

Password reset

Appointment reminders

User notifications on appointment confirmation, cancellation, etc out of scope

Billing

Get paid (oh heck yes, in scope)

Assorted Stuff

Account deletion out of scope

Admin interface God gave us console for a reason

Custom analytics Nobody to track on first day

A/B tests (would be out of scope, but hey, free since I have A/Bingo)

Whitelabel version

HIPPA compliant version

Change tracking (e.g. audit trail for deletion, etc)

Exploratory Coding Begins

After playing around with the signup screen for a little while, mostly to get a feel for SASS/Haml, I started working on the first feature of the app that would provide user value and be mostly self-contained.  Ideally, I would start with the first thing I want them to do (schedule an appointment), but that requires adding a first client anyhow, so I started with the client administration screens first.

I don’t use Rails scaffolding, although since I got publicly told off invited to try improvements by DHH for having insufficiently RESTful routes I thought I’d give that a try.  (Turns out it actually does save enough pain to matter, although I still think publicly visible RESTful routes are a mistake.  For internal features of a CRUD app, though, they make a lot of sense.)

What I generally do is start with the index action, verify that I can display records added to the database directly (via the console or a seed script), then start on the show action.  This results in me getting frequent incremental visual progress that the program is, in fact, getting better.  Anything which can’t be used yet gets stubbed out with lorem ipsum.  When I feel myself reaching the stopping point for a day, I go to the next action on my list and get the view working, even if it is as simple as displaying a screen which says “This action does not really exist yet but if it did it would show #{@client.name}”.  That gives me a place to pick up again in the morning (or, ahem, afternoon, given my frequent work/sleep habits).

Anyhow, after getting show done, I work on the edit/save loop, which requires building out the model a bit, adding validations, and writing some less trivial controller logic.  Here I frequently discover something like “Hmm, it would be handy to have a way to display messages out of the flash” and start working up a helper to do it in a minimalistic way.  You can see it behind the example of me operating the new (error-enabled) edit window.

Working with jQuery slowed me down a bit when creating some of these screens.  I used jRails to ease the transition (it lets you use all the old Rails-esque Javascript helpers like link_to_remote), but it does not play well with jQuery 1.4 out of the box.  It turns out that the issue I was having was mostly that jQuery executes any Javascript it grabs from the server prior to updating the DOM with the rest of the data it got back, which borks the Prototype-esque habit of sending back both HTML and then Javascript which relies on that HTML being in the DOM to function.  I got around this by simply returning only Javascript — e.g. for the “pop a window to put in a new client” action I return:

:javascript
  $('#client_dialog').html("#{escape_javascript(render :partial => 'edit_form')}");
  $('#client_dialog').dialog({
    //blah blah
  });
  $('#client_dialog').dialog('open');
  $('#client_dialog').find('input').focus();

This is hacky as heck, and obviates some of the reason of using jRails in the first place, but it works. I can always DRY up the helpers a little more later.

After getting the validations and whatnot working for Clients, and doing some light testing to make sure things worked to my satisfaction, I did similar work for Schedules. I got the basic CRUD functions working in record time, mostly by copy/pasting everything I had done for clients and then just changing the parts of the model and view that were different. The controllers are mostly identical, at least until I actually create the ability to render a schedule.

I’ve finished the CRUD logic for my first two parts of the problem domain, and am impressively ahead of schedule. To celebrate, I spent a day upgrading jQuery to the latest stable version (not quite so fun), upgraded the jQuery week calendar widget I was using to a fork that is being actively maintained, and wrote code to render the minimum calendar possible without actually having any appointment data available.

Schedule Calendar

And that is about it for today. Tomorrow: hooking up the ability to add appointments to the schedule, then CRUD logic for them, then the ability to create an appointment and client at the same time. That will complete the first take on the core interface logic for the application, leaving me next week to tackle integration with Twilio.  (I had originally planned on that taking at least two weeks, but I remember it being much easier than I expected back when doing the MVP, and I seem to be progressing faster on this project than I typically do.)

Plans after that are roughly:

  • Turn on user account creation, log in/log out, etc
  • Create staging server, taking care to document my setup process on a computer this time (darn missing notebook)
  • Add integration with Spreedly/Paypal for billing
  • Polish as much as possible.
  • Test as much as possible.
  • Ship at or near end of November.

After Shipping

After shipping, of course, comes the 90% of the remaining work needed to turn an application into a business.  (Of course, I have been doing some of it all along: speaking with customers, gathering requirements, thinking about marketing angles, etc.)  In particular, I’m starting to devote my free cycles at lunch into thinking about scalable content generation strategies for organic SEO, which is the form of marketing that I’ve had the best results with in the past.

But it is highly likely that, immediately after launch, I’ll have a bare handful of customers and a fairly relaxing December with answering customer support queries (and pre-sales inquiries), mapping out future development, starting the marketing engine, and enjoying Christmas with my family.  Then, on to January.

How A Half-Broken Halloween Promotion Smashed Revenue Records

Happy Halloween!  Everyone’s favorite secular American holiday where kids are in school is invariably the best month of the year for Bingo Card Creator.  This year, it ended up being a very happy Halloween indeed.  I ended up selling $6,024.45 worth of software after returns, beating my previous record (set last Halloween) by about 30%.  Bingo Card Creator has now sold over $100,000 worth of software (now that is spooky) and is on target to hit $45,000 in sales in 2010.

I did a bit of extra work getting the Halloween promotion to work this year, and also automating/systemizing so that I’ll be able to exploit similar opportunities without needing to take time off to do custom coding.  Some parts of the strategy worked very, very well.  Other parts lagged expectations.  And, naturally, I managed to make a critical CSS error and cost myself a few thousand dollars in sales… again.

Timing A Seasonal Promotion

The best time to start preparing for Halloween is in September… preferably, a September many years ago.  This is because there is a huge, huge surge of Halloween-related queries on Google starting at about October 15th or so, and if you wait you won’t have your site ranking in time to ride the tidal wave.

[Halloween costumes], [Halloween parties], [Halloween ideas], etc etc, all follow almost the same distribution.  Most importantly for my business, so does [Halloween bingo cards].

A brief digression to understand why this is important: teachers want to play a game on Halloween because t is a children-centered holiday.  Halloween is a much more festive occasion in American schools than many other holidays, because kids are not given any time off for it and it is secular.  To make a long story short, there is a wide cultural rift in US education about the proper place of religion in public schools, and that makes celebrating e.g. Easter with a rousing game of bingo not exactly a career-enhancing move.  Despite Halloween being theoretically based on All Hallow’s Eve, in standard American practice it is totally secular in character.

Anyhow, to make a long story short, upwards of a hundred thousand people will go to the Googles and search for [Halloween bingo cards] and a passel of related terms in October.  For the last few years I’ve really wanted to be #1 for that.  This year, I was — my mini-site, established in September 2008 or so, finally hit the big time.

Halloween Bingo Cards is what SEOs call a mini-site.  It has one mission in life: for two weeks out of the year, it is supposed to be the best, most focused site on the Internet for, well, Halloween bingo cards.  Between on-page optimization, a handful of inlinks, the exact match domain bonus (it is a .net, which along with .com and .org receives a huge bonus to rankings if the domain name exact matches the query), and a bit of age, it finally edged out the competition.  The extra traffic I got was, well, a little underwhelming.

Why was I underwhelmed by 27,000 visits?  Well, the site was designed to generate trial signups of Bingo Card Creator, and most of the obvious candidates for action on the site linked people directly to a trial signup form.  Conversion rates were unspectacular, in part due to a technical failure describe later and and in part because a combination of design issues and just overall low traffic quality meant fairly few people clicked off Halloween Bingo Cards in the first place.

So:

  • 27,000 people saw the Halloween mini-site
  • 3,500 (only about 13%) clicked to the main site (and the signup form)
  • 1,300 (37%) signed up for the free trial (that is actually extraordinarily good — normally I top out at about 28 ~ 30% for scalable traffic sources)

So of those 1,300 people, how many do you think actually signed up for BCC?  Well, since I finally have end-to-end analytics running (through a combination of KissMetrics/Mixpanel and a bit of glue code I wrote myself), I can get away from half-accurate Google Analytics reports and give concrete numbers on this.  So I know, with certainty, that that answer is 15 sales directly attributable to this page.  At about 1.2% conversion that isn’t that much to write home about — for the right audiences, BCC gets about 2%+.  (It should be noted that Halloween traffic is almost always going to be the wrong audience, because most searchers are parents but most of my customers are teachers: parents have their needs 100% met by the free trial.)

So What Worked Better Then?

Email.  Holy cow, email.  This has been a blindspot of mine for years since I used to be an anti-spam researcher, do not really send or read mail that regularly outside of doing customer support, and hate newsletters with a passion.  Big mistake: my customers empirically do not feel the same way.

Halfway through last year I signed up for MailChimp with the intention of doing great things with it, but I struggled to get it to work out right.  I would lose interest for several months at a time and not send email to the list, which causes people to forget who you are and then get very peevish when they receive mail from you.  About the only thing that worked very well for me was my auto-responder, which automatically mails people 1 and 6 days after they sign up for it with hints and tricks, to basically remind them that I am still here.  (A huge percentage of BCC users never sign in after their first day, and anything I can do to remind them of my existence is worth serious money.)

Sadly, I got an automated notice from MailChimp several months ago about excessive unsubscribes from my autoresponder — about 1% (they’re serious about list quality at MailChimp), and paused it while editing to make it obvious who I was and why they were getting email from me.  (The phrase “because you went out of your way, twice, to ask for it” may have been in an initial draft.)  And then I left the autoresponder paused for several months.  sigh So I wrote a new alert for my dashboard about that, since I’m serious about making mistakes only once.

Anyhow: Rob Walling had an amazing presentation at the Business of Software conference about using email to sell more software, and I was so inspired I decided to get serious about it this year.

  • I used MailChimp filters to go down from the 8,000 or so people on my list to only the 800 signing up since June, figuring they would still remember me.  (In hindsight, I should have cut further, and I will make a point about absolutely, positively keeping the list “fresh” from here on out.)
  • I tweaked my from address from “Bingo Card Creator” to be “Patrick McKenzie (Bingo Card Creator)”, at Rob’s suggestion.
  • I mulled over reasons why teachers might open the mail, and settled on “A Halloween Activity (And Discount) From Your Friends at Bingo Card Creator”

The time-limited discount was repeatedly stressed in Rob’s presentation and in his book about selling software.  (He gave me a free copy.  You should buy it just for the email chapter.)  So I spent a few hours tweaking my shopping cart so it could use discounts, and then polished until the experience of receiving them was totally transparent to the user — they never have to fumble for a code, all they have to do is click a link in their email and they’re in.

This discount was extraordinarily effective at motivating sales, relative to many things I have tried.

  • 775 mails sent.
  • 170 (23%) opened
  • Of those, 6 people bought.

At a little less than 1% conversion from an email to a sale, I’m absolutely giddy about the future potential for email marketing.  I got perhaps excessively giddy and sent another email two weeks later to people who didn’t open the first one.  At least, that was my intention: I actually hit the button for sending it to people who didn’t open last year’s Halloween newsletter (i.e. essentially everyone who had just been mailed two weeks prior).  Doh.

One person did not appreciate getting two newsletters in one month after not getting them for several months.  I got a handful of spam complaints for that (four from the first send, one from the second).  Since MailChimp is hypersensitive to spam complaints and I likewise care about my reputation, I’m going to be more careful in the future.  I anticipate the best thing to do to keep them down is to just email people early and often to keep myself in their mind, which is almost the opposite of my natural inclination.

Anyhow, the two campaigns together:

  • 1,600 emails (total cost: $32 of MailChimp credits)
  • 60 reactivated accounts
  • 15 sales (~ $370 in revenue)

Very little I have done has scalable 1,000% ROIs on marketing spends.  You can bet I will be doing more of this in the future.  (Granted, this doesn’t count approximately 3 hours of effort upgrading my shopping cart and site to do discounts, but I can amortize that over every time I use them from here on out.  The newsletter was the 2009 Halloween newsletter with about 5 minutes of editing.)

Speaking Of Discounts

So after having huge results early in the month with the emailed discount offer, I thought “Hey, why not extend that to everybody?  I’m getting crazy 10% conversion rates on reactivated accounts — if I got 10% conversion rates on new accounts I would be making hats out of money.”  Out of an abundance of caution, I A/B tested that for the last two weeks.  Surprisingly, discounts failed to make a dent in sales numbers in the general (non-email) population.  Roughly 2.08% of people seeing the discount converted, and 1.87% converted without seeing the discount.  That is despite prominent notices on the first screen after you log in, plus the pricing page, and the exploding nature of the discount (act now or you won’t get it).  Not only is that difference below the floor of statistical significance, when you factor in the fact that I make $5 less for every discounted copy, the full-price version did better for me.

Things to try next time:

  • Only give the discount to people who I’m reactivating — their accounts are writeoffs if they don’t sign back in, after all.
  • Pick the cutoff date for the discount better.  I gave people to November 1st, because Halloween is the 31st and I wanted to be generous.  However, that diminishes the perceived urgency of the sale — on Thursday night, they still have “3 days left!” That was stupid of me because I know, from doing this for several years, that my window of opportunity for Halloween sales closes as soon as she goes to sleep on Thursday.  I should have had the Halloween sale go until Thursday night, which makes it a pretty cruddy “Halloween” sale but would almost certainly have goosed sales.

That said, that was a pretty good day for me — Wednesday before Halloween was my best day ever, right until Thursday smashed that record.  Last Halloween I was beginning to pull weeks upon weeks of crunch, and I think I probably made less in a 70 hour workweek than I did in those two days… while sleeping.  I love self-employment.

Google Enjoyed Halloween, Too

While I was #1 on Google for [Halloween bingo cards], [Halloween bingo] and a thousand words on the long tail also receive significant traffic, and many of them are sewn up by about.com, eHow, etc etc.  Luckily, most of the content mills run AdSense ads, and the guy at the front of the auction screaming “Pick me pick me!” was yours truly. I spent $1,764.84 on AdWords ads in October.  Sadly, much of it got wasted.

Why?  Well, I made one extraordinarily poor decision and followed it up with a mistake: when FireSheep (the sniff-your-credentials Firefox plugin) was announced, I immediately bumped SSL support from “something I’d eventually like Bingo Card Creator to have” to “must be done today”.  If I had had any sense, I would have let that wait until November.  Enabling SSL took about a day, but I missed some edge cases which came back to cut me: in particular, I had the registration form throwing SSL errors for about 6 hours (painful, particularly since it was during an email delivery window) and, even more painfully, I had the AdWords landing page throwing errors for 48 hours (not fun when you’re spending $150 a day driving people to it).

But the worst thing was keeping the freaking AdWords conversion code on HTTP, which caused some browsers to not load it from an SSL page, and cut my conversion rate in half.  Since I use Conversion Optimizer, this caused Google’s algorithms to think “Hmm, Patrick must really not want this flood of traffic we’re sending him since it apparently converts like crap.  Oh well.  I guess we’ll just send it somewhere else instead.”  Between money that I essentially set fire to, an AdWords campaign thrown into disarray (I still haven’t recovered my previous conversion rates, since the bots are in a bit of confusion and haven’t replaced me on my best performing sites yet), and missed opportunities, that mistake probably cost me a few thousand dollars.  Ouch.

Still, I generally try to see things as half-full rather than half-empty, and my jack o’lantern is overflowing at the moment: previous sales records smashed, a new scalable marketing tactic which actually works, infrastructure ready to go for Thanksgiving, Christmas, and Valentine’s Day (and a few hundred more email addresses to send to), and the SSL issues now mostly under control.  (I still am not transitioned to 100% SSL, which ironically means that the effective increase in security was minimal.  sigh)

Mini-update on Appointment Reminder

I was very busy with ongoing improvements to Bingo Card Creator, consulting, travel, and non-business life for the first six months of being self-employed.  Since midway through October I’ve refocused to get Appointment Reminder out the door (really — most of the Halloween campaign was taking advantage of things which already existed), and since previous experience has taught me that artificial deadlines really work for me, I have joined up with a bunch of Hacker Newsies to make November the Get Your Startup Accomplished month.  After some fumbling around with jQuery and other infrastructural issues, I feel like I really hit my stride today, and if I keep it up I should launch right around the end of the month.

If you’d be interested in hearing about this on a regular basis, leave me a comment — if folks care, I will post regular updates to the blog.

How To Use SSL To Secure Your Rails App Against FireSheep And Other Evils

The post on Hacker News today about FireSheep, a Firefox addon which lets you trivially compromise the web application cookies/sessions of anyone on the same wireless network, gave me the much-needed impetus to upgrade my business to SSL security.  For the last several years, I haven’t encrypted traffic between the server and the user.  My theory was that my customer’s didn’t store anything security critical in their elementary school bingo card games, but the increasing amount of information available to the admin (me) plus a customer support story from this morning convinced me that compromise would be a Very Bad Thing.

Implementing SSL in Rails was very painful and resulted in my site being down or unusable for a large portion of my customers for much of the day.  (If I were doing it again, I would have paid the extra few bucks to set up a staging environment with its own certificate and verified everything worked on that prior to trying to fight my way through the process.) Luckily, since I am time-shifted from them by over 12 hours, few noticed.  In the interests of saving you and your customers some difficulty, I thought I would write up what I learned.

What You Need Before You Get Started

  1. A SSL certificate from a certificate authority which is trusted by the major browsers.  GoDaddy sells them for $12.99 and they are perfectly adequate.
  2. SslRequirement and AssetHostingWithMinimalSsl, both plugins by DHH.
  3. Rails to be fronted by Nginx.  The explanation for Apache is similar but the magic server configuration is different. (If you use Nginx+Passenger, can skip the Mongrel-specific bit below.)

Nginx configuration

Nginx manages configurations on a per-server basis, and cannot have a single server declaration listen to both HTTP and HTTPS requests. We’re going to get around having to copy/paste our entire configuration (and maintain it in two places) by DRYing it into a single external snippet, then including it twice.

Right now, your Nginx config looks something like this:

server {
    listen       80;

    //You have a lot of stuff here.
}

Cut everything out of the server declaration (yes, everything) and externalize it into a separate file. It should now look like:

server {
    listen       80;
    // This path is relative to the conf directory
    include apps/EverythingJustCut.conf;
}

You can now create a separate declaration for your SSL server without causing much overlap:

  server {
  listen 443;
  ssl on;

  #GoDaddy's instructions will walk you through setting these up
  ssl_certificate /usr/local/nginx/conf/ssl/your_certificate_combined.crt;
  ssl_certificate_key /usr/local/nginx/conf/ssl/your_key_pair.key;

    include apps/EverythingJustCut.conf;
}

I remember setting up the SSL certificate to be more of a nuisance than a difficulty, but if not, you can find very detailed instructions online.

Now, if you are using Mongrel, you need to do one more thing: withing EverythingJustCut.conf, you’ve got to find the place where Nginx is proxying to Mongrel and have Nginx set the X_FORWARDED_PROTO to https for HTTPS requests. This is the one and only signal that Rails, running on Mongrel, is going to get that a particular request is for SSL or not. This is trivial to do if you know you have to do it: just find all your existing proxy_set_header statements and add this after them:

  proxy_set_header X_FORWARDED_PROTO $scheme;  // http or https, evaluated per request by Nginx

Very Carefully Scrub Your Website For Absolute URLs

Absolute URLs containing the scheme (i.e. anything starting with http:// ) are dangerous to your application, because if your page transmitted over HTTPS references a “certain type of resource” on HTTP, your browser may display a Scary Error Message to your users. Unhappily, the browsers get to pick what constitutes a security-critical resource.

The general rules for maintaining SSL security on HTTPS pages are:

  • Javascript files: must always be loaded from HTTPS
  • Image files: must always be loaded from HTTPS, except for Firefox and Safari
  • CSS files: must always be loaded from HTTPS, except for Safari
  • other files: may or may not be loaded from HTTPS

Do you think you’re going to remember that? Yeah, me neither. Hence, you don’t want any hardcoded http:// anywhere. Let Rails take care of it for you with AssetHostingWithMinimalSsl: all you have to do is be consistent about always loading e.g. images through the image_tag or image_path helper, CSS and Javascript through their associated helpers, etc, and you will always get the proper behavior. The tough part is that you’re going to have to take IE and drive through every page on your site verifying that it does not accidentally include a resource transmitted in the clear.

The configuration for AssetHostingWithMinimalSsl goes in your config/environments/production.rb file and is trivial:

config.action_controller.asset_host = AssetHostingWithMinimumSsl.new(
  "http://images%d.example.com",  #In this example, I have images1..images4 .example.com configured in DNS
  "https://www.example.com"  #Your SSL-secured domain
)

Note that it is very easy to miss a http:// URL hidden in a CSS file, Javascript file, or analytics-package JS include somewhere. If you do that, even in an unused CSS selector, you will cause the browser to throw Big Scary Errors. Test that you have gotten everything very thoroughly prior to proceeding.

Require SSL for Any Security Sensitive Actions

Why are we requiring SSL? To prevent against an attack where the bad guy can sniff the cookie out of there air, thereby appropriating it for himself and logging in as the logged-in user (either by compromising a session ID or, in Rails, compromising the user_id you probably stored in the tamper-proof CookieStore). So what do we have to SSL? Everything Rails sends or accepts a session cookie with. What is that? Everything that an actual browser can access. (If you are, like me, in the unenviable situation where some URLs are going to be hit by user agents which cannot support either HTTPS or cookies, don’t forget that requiring SSL for all actions won’t help you.)

The SslRequirement plugin makes it easier to do this sitewide than it otherwise would be:

#in application_controller.rb
 unless RAILS_ENV == "production"
    def self.ssl_required(*splat)
      false
    end

    def self.ssl_allowed(*splat)
      false
    end
  else
    include SslRequirement
  end

This sets it so that SSL is required when we say it is in production only, and in other environments the statements which set up SSL requirements are merely silently ignored. Those functions are:

  • ssl_required: takes a list of :symbols representing action names to require SSL for. Should be nearly all of your actions. If someone tries to access one of these actions over HTTP, they will be redirected to HTTPS (+).
  • ssl_allowed: takes a list of :symbols representing action names to allow SSL for. If they aren’t required and aren’t allowed, then they’ll be redirected to HTTP if they try getting to this action over HTTPS.

You set them in each controller, on a per controller basis. There does not appear to be a handy mass assignment option like :all for this method, unlike most of the before_filter and similar things you find in Rails.

Note there is a subtlety here: if you let Rails share a session cookie over both HTTP and HTTPS, then if it is ever used over HTTP, byebye cookie security. This means you can either be very, very careful that you never let anyone access Rails actions over HTTP (and you pray a malicious attacker never tricks your users into clicking to a valid link to your website), or you ban your session cookie from being sent over HTTP at all:

#production.rb
config.action_controller.session = {
    :key     => 'name_of_session_goes_here',
    :secret          => 'you need to fill in a fairly long secret here and obviously do not copy paste this one',
    :expire_after    => 14 * 24 * 3600, #I keep folks logged in for two weeks
    :secure => true #The session will now not be sent or received on HTTP requests.
  }

This option will probably break your site, possibly subtly, the first time you switch it on. Test thoroughly. I haven’t got it 100% working for myself yet.

Host downloadable files on SSL? You just broke IE.

After several hours of frustration, failing my way forward, and finally getting things working on Chrome/Firefox, I received a bug report from an IE using customer who couldn’t download her bingo cards. Some deep Googling revealed that IE, for architectural reasons known only to the IE team, does not play well with downloadable files over SSL unless you set some very specific headers:

  response.headers["Pragma"] = " "
  response.headers["Cache-Control"] = " "
  response.headers["Content-Type"] = "application/pdf"  #Put your favorite MIME type here
  response.headers["Content-Disposition"] = "attachment; filename=#{filename}"  #
  response.headers["X-Accel-Redirect"] = "/path/to/file.pdf"
  render :nothing => true

Note I am using X-Accel-Redirect to have the file served directly through Nginx, rather than through Mongrel, as described here.

Conclusion

I hope that saved you and your customers some pain and insecurity. If you haven’t done this yet, and I think there are many Rails apps as open to exploitation as I was until this afternoon, I suggest you download FireSheep and see how quickly any idiot with wireless can compromise your admin session. Then, fix this as soon as possible.

Lessons Learned At Business of Software 2010

Author’s note: This post is gigantic — over 10,000 words.  I don’t recommend reading this in one sitting — bookmark it and come back later.
//
Early this October, I was privileged to attend the Business of Software conference as both attendee and a speaker (see disclaimer waaaaay at the bottom of this post).  It was far and away the best conference I’ve ever been to.  You should go to it next year — I know I will, if I have to swim from Japan to Boston to do so.  (If you would like to but don’t think it is possible, see waaaaay at the bottom of this post.)

The featured speakers were, virtually without exception, remarkable.  The 7.5 minute Lightning Talks (one of which I presented — more about that after they post the video, you will be amused or your money back) were often entertaining and at least a few of them were packed with better detail than 15 Powerpoint slides have any right to be.

The phenomenal content of the conference, however, is just an excuse to get a few hundred ferociously smart people in the same hotel for three days.  I got to finally meet in person a few of the folks who either personally assisted me in my early days as a businessman or have been tremendously influential in my thinking — a short list of shoutouts includes Ian Landsman, Dave Collins, Joel Spolsky, Eric Sink, Eric Ries, Peldi from Balsamiq, and I could put another dozen names here without breaking a sweat.  One speaker at one point asked all attendees who had founded a software business to stand at one point, and by my eye probably 60% of the room stood up.  During breakout sessions, I heard from everyone from folks doing small software businesses like myself to a company which sells monitoring software for nuclear power plants.

The spirit of generosity at the conference was off-the-charts amazing.  However, many conversations were either by explicit agreement or common courtesy off the record — not everybody publishes sales stats, and that is their right — so I’m going to err on the side of caution and only repeat things I learned from the public sessions.  That said: beg, borrow, or steal tickets to it next year.  I can’t repeat that enough.  Although the conference swung to business topics far more often than technical topics, almost everyone I talked to both inspired and holding a notebook full of actionable advice to try.

My notes are incomplete and my memory is fallible, but here were some of the takeaways I got from the public presentations.  These are more a combination of my notes and thoughts than transcriptions of what was said.  If you want those, wait for the videos to be released.

Seth Godin

Seth posted a riff on his remarks on his blog.  The general tone and content was not surprising if you’ve followed him for a while.  (If you don’t, do so — aside from being a great communicator, he’s one of the better big-picture thinkers on marketing that I know of.)

Specifically, he says that the software business is undergoing radical changes because technical competence is no longer a scarce commodity: with open source tools, an increasing supply of trained programmers, and the explosion of cost-lowering innovations for creating and marketing software on the Internet, things which were previously the purview of a technical elite are now within what talented teenagers can cook up from their kitchen table.  The marketing side of the equation is also changing, since increased crowding means that the traditional model of paying salesmen to push software has devolved into using humans as inefficient spam bots.  (See his whole spiel about Permission Marketing here.)

A brief guide to evaluating opportunities:

  • Who can I reach with my marketing message?
  • Will they talk to people on my behalf?
  • Can I earn and maintain permission to continue the conversation with them?
  • Will they pay for my solution to their problems?

Perceptive readers will note that technical difficulty appears nowhere above.

Best four words of talk: “Ideas that spread, win.”  Software is now about creating a tribe or otherwise achieving leadership of one.  Take AutoCAD, for example: it isn’t a product, it is a movement: we architects who use AutoCAD are here to change the industry away from using pencil&paper drafting.  You are either in the movement or opposed to it, because it is an existential threat to your business, but you certainly aren’t neutral on the question.  Boring software creates no movements and has to be sold the old fashioned (expensive, ineffective) way.

Another example of movements is the trend towards measuring things, which was a recurring theme during the conference.  Toyota succeeded in changing the face of manufacturing because they got religion about statistics, and a similar sea change is finally happening in software.  This meta-trend is creating a lot of extraordinarily valuable companies directly and also making the rest of us leaner and meaner.  (Bingo Card Creator has a more sophisticated approach to metrics than many Fortune 500s.  That’s a shame, but also a tremendous opportunity for smart, savvy software firms.)

I was quite amused by a throw-away example of the difference between capitalists and socialists inspired by pin-making machines: prior to the industrial revolution, it took a skilled artisan a day to hone 5 pins (small bits of metal which one uses to, e.g., attach fabric to each other) by hand.  A pin-making machine lets any 5 people who can be trusted to learn a simple, repetitive set of instructions make 10,000 pins a day.  Socialists hearing this thought “Oh no, that is going to totally destroy the livelihood of pinmakers.”  Capitalists hearing this thought “Memo to self: be the guy who owns the pin-making machine.”  As the cost of software approaches zero, you want to be the guy who owns the machine.

Relatedly, with regards to your future employment prospects: draw a Venn diagram of job skills you have and skills which can be automated (or, by extension, anything which can be adequately explained by any amount of writing on paper, because anything that can be written down can get shipped to India).  Anything in the intersection represents career suicide.

Seth claims that no “competent” people work at his current software company, Squidoo, because “if we only needed competency, we can get that from [outsourcing]”.  You hire exceptional people and let them execute on things which require exceptional talent, and write down things which only require competency.  Low-wage grunts can do the competent stuff.  “Your job is a platform, not a list of tasks.”

Some personal thoughts: Seth was bearing on charging for software because of competition from OSS competitors, preferring instead to lead movements where joining was free (to maximize the size of the movement).  Happily for those of us who sell software, almost all OSS projects are incompetent at marketing, which matters more than technical execution.  Seth gave an example of trying to sell water: what are you going to do, make water wetter?  That is a wonderful example, since bottled water didn’t exist as a category 50 years ago and sales are going up every year, over a century after America solved the problem of giving everybody what amounts to free, infinite, drinkable water.  It amused me greatly to hear that bottled water was going away while attending a conference at a luxury hotel which sold $4 bottles of bottled water placed in the restroom literally next to the tap of ever-flowing unmetered municipal goodness.

David Russo

David gave a presentation about hiring at software companies, in particular about hiring to create and sustain a corporate culture as the company approaches scalability.  I regret I was insufficiently attentive during it, as this topic is several years ahead of need for me.

Briefly, there are five models for corporate culture in the literature:

  • Bureaucracy  (the traditional BigCorp model)
  • Autocracy   (Apple-style, where the company is an extension of the Will of Jobs)
  • Engineering (Google, where the engineers run the asylum)
  • Star culture (a large consulting firm or, perhaps, 37Signals, where individual contributors are expected to be stellar and you can put up with rough edges because, hey, rockstars)
  • Commitment — a consensus culture with loyalty receiving strong priority (and, as you can probably guess by position on the list, the clear favored choice of the presentation)

There was an example of Westinghouse having a non-engineering CEO who proved to be disastrous for the company.  Afterwards, the board resolved never to hire a non-engineer again, because the culture of Westinghouse was ferociously engineering-dominated and when engineers talk to each other they break out the secret engineer decoder ring.  The CEO, who couldn’t speak the language or achieve respect among the engineers, was sidelined and failed at achieving his objectives.

Culture generally grows organically out of a business rather than being imposed from the top-down.  Changing a firm’s culture midstream is very painful, and Harvard studies suggest it has quantifiable negative impacts: 50% reduction in odds of an IPO, tripled likelihood of business failure, and 3% less growth monthly.  (The engineer in me winces at the compounding there.)

David’s advice was to invest in employees: hire brilliant people, and let them be brilliant.  The outcomes they achieve are more important than the work itself.

Dharmesh Shah

Dharmesh is CEO (most recently) of Hubspot, a marketing tool for small businesses.  They have raised quite a bit of money and, more importantly, have a lot of paying customers, consistent growth, and a revenue curve which virtually defines “up and to the right”.  You should be reading his blog already.

He presented a few techniques from Hubspot which were “interesting and not obvious”.  For example, successful SaaS companies which IPOed had a median capital raise of $42 million dollars.  It doesn’t match with our expectation that you can make a SaaS startup in your garage, partially because SaaS companies essentially offer their customers upfront financing (by spending their cost of customer acquisition in a front-loaded fashion and then recouping it over the 4-5 year lifespan of that relationship).

A key metric to keep track of: customer churn rate, because a 50% decrease in it doubles customer average lifetime value (LTV).  One of Dharmesh’s periodic hobbyhorses is that if you know LTV and know the cost to aquire a customer (CAC), then if CAC is less than LTV by a fair bit, you take as much investment as humanly possible, spin straw into gold, and walk away with money hats.  (This is far and away the best pitch for a software company taking VC money that I’ve ever heard.  See also the video of him speaking last year, if I remember right he covers it there.)

There are a few ways to measure churn: as percent of customers, as percent of revenue (better accounts churning faster than entry-level accounts: uh oh!), and “discretionary churn”, which is churn among accounts who are actually able to make the decision to cancel and are not, e.g., locked into a one-year contract with you at the moment.  However, overoptimizing on churn rates is myopic, because absence of churn does not indicate the presence of delight, and customer delight is a key to long-term success at marketing the product.

One particular way Hubspot maximizes for customer delight is by creation and use of a Customer Happiness Index (CHI), which is a backtested predictive measure of someone’s likeliness of churning next month.  CHI is dominated by three factors and adjusted as they get more data:

  1. Frequency of use (more use = less churn)
  2. Breadth of use (customers who use more features churn less)
  3. Use of “sticky features” (there exist some features which are so screamingly better than alternatives that a customer who uses and gets value from them is very reluctant to stop using the service)

If I can call out #3 for a moment: many companies I have worked with have “sticky features”, although not all of them know it.  (Some day when I don’t have another few thousand words to write I’ll tell you about what they are for Bingo Card Creator and how I know.)

After you have CHI available in a transparent fashion throughout your organization, you can exploit it in a variety of ways:

  • Tell your sales representatives to call customers with low CHI proactively and ask them what you can do to make them happy.  This simple technique saves 1/3 of canceled accounts, and is virtually free.
  • Judge sales reps not just on sales but on CHI of new accounts.  (This counters the incentive to armtwist to sell to marginal customers.)
  • Judge new features / policies / etc based on their impact to CHI.  Concerned about whether charging for consulting for setup is a good play for the business or not?  A/B test doing it versus not doing it, and optimize for CHI.  (I get the feeling that they might not explicitly be doing A/B testing… but they should be.  Sorry, hobbyhorse of mine.)

All meetings at Hubspot include a teddy bear named Molly, who is a stand-in for the customer (they have two customer archetypes: Margaret the marketing director and Olly the owner, collectively, Molly).  When they have contentious debates internally, they are frequently decided by someone saying “Hey, what would Molly want us to do here?”  Brilliant hack, practically free.

Hubspot sells services, primarily consulting for setting up new clients.  This represents a fraction of their revenue — only 7% or so — and decreases their gross margins, because consulting services are intrinsically lower margin than selling software products.  But “services are low margin… except when they’re not”: since getting people set up for success increases their CHI and wildly decreases churn rate, selling someone a few hundred bucks worth of engineer time at low margins can lead to thousands of dollars of marginal increase in LTV delivered by high margin renewals of the core product (which has 90+% margins).  Star this advice, gentlemen, because it has obvious implications for B2B SAAS startups.

So if you know customers and the company benefit hugely by increasing consumption of consulting services, why charge for them?  Answer: perceived value increases consumption, and perceived value is driven by cost.  When you give people 5 hours of free telephone support with an engineer, they — being successful businesspeople — find difficulty clearing 5 hours free in their schedule, fail to show up for the call, get distracted, etc.  When you charge them $500 for the call, they damn well show up.  (Patrick notes: why hello, recent discussion about Chargify free customers and support costs.)

In terms of explaining the value proposition of your software to customers, cribbing from Kathy Sierra: don’t make _____ software, make _____ superstars.  Make your customers awesome at their jobs / tasks, and they will pay any price and/or follow you to the gates of Hell.

Hubspot is big about being transparent to a fault within the company, particularly by using a company-wide Wiki which includes financial details, plans, and pretty much everything except salary information.  (Apparently, that being public is viewed as possibly toxic to morale without having obvious upsides.  Having financial details open to employee audit has obvious upsides: employees can tell you when you’ve let the entrepreneur reality distortion field go wild in the board meeting.)

Best sentence about branding I’ve heard recently: “Brand is what people say about you after you have left the room.”

Eric Ries

I will refrain from giving an in-depth summary of this talk because if you have read his blog you pretty much know the gist of it before.  If you haven’t read his blog, start doing so.  With the obligatory disclaimer about letting any methodology take over your higher brain processes, the Lean Startup movement (and some of the specific, implementable tactics it suggests) are so effective I sometimes worry about whether established competitors will try to illegalize them.  The best effect of this presentation was taking the idea away from the Silicon Valley echo chamber (where “pivot” has become something of a buzzword) and bringing it to an audience of people who largely sell things to people for money, but who still could benefit immensely from some of the Lean Startup lessons.

If you want an hour-long video introduction by Eric to the Lean Startup, you can see parts 1 and 2 of a previous explanation on the Internet.

Some specific lessons I haven’t heard before in a couple years of lurking on his blog or I thought were worth highlighting:

  • The key takeaway: Lean Startup matters because the traditional model of development wastes people’s lives.  They pour out years of hard-charging startup workweeks making software which doesn’t actually improve the lives of any customer because it solves a problem no one has.  This waste of potential is criminal and can be fixed.
  • Machine shops and software firms are self-regulating ecosystems if you do them right.  For example, if you use continuous deployment and five whys, when you start doing development too fast, you’ll get the brakes applied by having to make investments in quality control repeatedly because you’re making too many mistakes.  This lets you find the optimal pace of learning without having to fiat an inaccurate number (“You will write 10,000 lines of code a week!” — which as we all know means nothing, and definitely nothing good) from on high.
  • The Agile methodology is a wonderful solution to the peculiar problems of internal IT departments at big companies, where the problem is extraordinarily well-known (“We need our existing payroll system, except on a new architecture where extending it doesn’t require sacrificing goats”) and solutions are unknown (“What is the Rails code to do this?”).  It has less to offer startups, where both the problem and solution are unknown.  (“Do people even need the thing we’re fumbling forward on building right now?”)
  • Since the only people you can plausibly get to use a new product are early adopters, any additional quantum of work done to get the product ready for mainstream users is a form of waste.  (This is a bold statement with interesting implications.  I’m not sure I agree with it, but it sounds potentially very powerful.)

Scott Farquhar

The president of Atlassian had some thoughts gained from bootstrapping the company from 2 people to a $60 million capital raise and beyond.  Regrettably for my notes, this talk was structured as a List of N Things, which ensured that I wrote down N things and grabbed few supporting details.

  1. Start with 2 founders.  For example, when their password database was compromised while Scott was off on his honeymoon in the wilds of Africa, knowing that his cofounder had the situation under control alleviated a portion of the anxiety, and after an emergency flight back to Australia Scott was able to let the cofounder get some sleep.
  2. You need a business model.
  3. Dogfood your products.
  4. Measure everything.
  5. Always be marketing.  Atlassian excels at this — for example, they had a charity give-away where they gave licenses of JIRA (a bug tracker) away for a song to small companies ($5 for 5 licenses or something to that effect), donated the profits to charity, and got massive good press from the deal while also increasing sales in the market segments which drive most of their actual revenue.  They also had very appealing marketing of another few initiatives: calling something the Stimulus Package, for example, was an excellent PR hook in the financial crisis environment.  I wish I had better notes here.  Memo to speakers: please do not autocommoditize your best points by numbering them.
  6. Your first idea will fail.  Atlassian was originally going to market plugins for an obscure product from a company in Europe.  It turns out the bug tracker they wrote to make that product was more valuable than the product itself — shades of Fog Creek and CityDesk here.  (We heard this theme several times during the conference: Peldi later said that Mockups was originally intended strictly as a plugin for another software, and the standalone version which it grew into now dominates Balsamiq’s revenues.)
  7. Long term thinking is good.
  8. Know when to switch gears.
  9. Build somewhere you want to work.  Many precious anecdotes here about, e.g., organizing a city-wide scavenger hunt for employees.
  10. Experiences are more effective at motivating employees than and customers than money.  This comports with both social science research I’ve read and my own experience.  (I have recent experience of this at a company I consulted for, but since it is identifying, I’ll wait until they announce what we did together to go into it.  Still: it is amazing how effective turning money into an experience is at motivating people than awarding them the money directly, even though this is crazily irrational.  Japanese companies understand this better than anyone: “why pay people so that they can buy objects suggesting social status when you can just award social status directly”, which is the Rosetta Stone for so much of corporate life here it is scary.)

Jason Cohen

I was flagging by this point in the conference and a little terrified about my upcoming speech, so my notes aren’t that great.  Jason had a lot of amusing anecdotes from taking Smart Bear code review software from a one-man operation in a small room in Texas through to a sale to another company.  (Despite being told, repeatedly, that “Smart Bear” was unprofessional and the name would scare off big Fortune 500s, the company which bought him out later reorganized their portfolio of products under that brand name.)

One amusing story was about Sales Guru Frank.  He was a silver-haired slick talker who, for a mere 50% of the equity of the company, promised to sell Smart Bear software to the sorts of huge corporate clients which would bring it from a six figure business to a smashing success.  Jason had an absolutely priceless anecdote about how us engineers perceive the sales process: “You know, it’s when you’re in the boardroom drinking some expensive brand of alcohol and you start the conversation with ‘Hey, did you see the game?’ and everybody understands what game you meant and then, suddenly, sales happen.”  It turns out that sales is a little more complicated than this (for details, see the Paul Kenny presentation) and, thankfully, Jason avoided getting suckered by Frank.

Similar fun stories abounded but I was too terrified of my upcoming presentation to write them down (d’oh).

The single best takeaway was how to evaluate advice you receive from noted software bloggers.

There are two competing motivations for people who start software companies: wanting to maximize one’s financial outcome as quickly as possible (you want to be Rich) and wanting to sculpt the perfect personal niche in which you will be respected and enjoy your day-to-day work (you want to be King).  There are two fairly well-known markets for software: B2B (business to business) and B2C (business to customer).

This gives us the traditional four quadrant graph, which Jason decorated with some examples:

The take-away: only listen to advice given from people in the same quadrant as yourself, and you’ll save yourself a lot of pain and cognitive dissonance from getting conflicting advice.  (This is overstating the conclusion a wee bit I think, and people change on both axes over time.  That said, I’m definitely onboard with “all advice is a product of the circumstances faced by the person giving it”, but I think you can pull lessons from 37Signals in a B2C business or from Joel Spolsky regardless of whether you want to craft the ideal place to work or go big with a VC company or both.)

Sidenote: my business got mentioned from the stage three times, and I’m seriously honored and humbled about being put in that company.

Paul Kenny

Paul talked about the sales function — not just the sales job title — at software companies.  See his talk from last year for details about that.

Why do so many salespeople suck?  Well, the typical career path requires:

  • One year from hiring to get up and running with the particular company/market segment/product.
  • One year to achieve momentum in building up one’s list of contacts and starting the sales cycle.
  • One year to demonstrably fail to achieve results.
  • One year for HR to convert the salesman from non-performing to “no longer with the firm.”

Given that it takes four full years to go through this cycle, you could see a salesman with an impressive resume of 3 ~ 5 year stints at name brand companies with impressive sales to their credit… and not learn, from their resume, that their sales skills are atrocious and their successes were actually the result of products which could be sold solely on the strength of their quality/marketing/brand even with the salesman impeding customer perception of all three.  For example, Jason’s Frank probably fit into this category.

Paul nonetheless believes that strictly depending on inbound marketing restricts your business to plucking the low-hanging fruit, and that going solely after low-hanging fruit is not a great business strategy.  (This would be an excellent opportunity to whip out the above quadrant map, by the way: if you’re in either of the B2C quadrants, congratulations, low-hanging fruit is your business strategy, and you can be quite successful with it.  Ditto 37Signals and many similar cheaper B2B SAAS apps, by the way.)

The point of speaking to customers, which is all sales is, is to reach a mature understanding of what your client values the most.  Then, and only then, you offer that to them.

Engineers routinely suck at doing demos for customers because they demo every bit of the interface, from start to finish, going through all the little knobby settings bits.  Nobody cares about these things.  Demo the bits of the software your customers find interesting.  When you email customers, talk about the bits of the software your customers find interesting.  When you blog about the software, blog about the bits of the software your customers find interesting.  But above all things recognize that your customers find themselves interesting and the software boring unless the software makes a demonstrable improvement in their lives.

Quick tip when speaking to customers: “The most interesting people you will ever meet are the people who are most interested in you.”  The average sales rep spends about 47 seconds in talking about customers prior to talking about the solution.  That number should be increased, greatly.  On the plus side, because the industry is so comprehensively screwed up here, even minor improvements provide an opportunity to surprise and delight customers.  “Goodness, the CEO of that company spent five whole minutes talking to me about how we go about our business!  They really care!

(Sidenote: This also goes if you’re selling yourself as an employee to a company.  If you’re in an interview and talking about the employer, you are probably going to get hired.  For reasons beyond my ken they don’t teach this lesson in college.)

Key takeaway: The quality of your dialogue with customers is directly proportional to the quality of the customers you will acquire.  If you understand their needs better, you will close bigger deals with happier customers who consume less resources.

Customer faith in your product as a solution to their problem is directly proportional to how well they believe that you understand their problem.  This again counsels spending more time talking to them and asking perceptive questions, then repeating their own language right back at them.  If they call it a foo, you call it a foo, even if internally everyone knows it is “really” a bar and, after all, it implements IBarable in the source code.  Relatedly, you cannot tell your way into a dialogue with the customer, you can only ask your way in.

Customers have DNA: Drivers, Needs, and Aspirations.  You should be capturing your understanding of these as you talk to customers, or you aren’t learning what you need to learn to bring the customer and the firm to a mutually satisfactory relationship.

Some factors to consider

  • Customer needs
  • Timescale for implementation
  • Scalability
  • Integration with existing systems/processes
  • Affordability
  • Results

Customers have many priorities:

  • Ego (underrated by engineers in my opinion… even those who own iPhones because they’re worth owning iPhones)
  • Perceived gain
  • Sense of belonging (“Nobody ever got fired for…”)
  • Security
  • Ease of use

If you think of a funnel of concerns prior to sale, starting at the top:

  • The experiences customers have right now.
  • Their business case or project which may benefit from your solution.
  • The utility your solution can offer.  (Note to engineers: many of you stop inquiring here.  That is a mistake.)
  • Options they have competing with your solution.
  • What the company values in terms of outcomes, drivers, etc.
  • Resources they have to solve the problem (i.e. talk budget last, not first)

Founders have an advantage in doing sales, even if they’re engineers who think they can’t do sales.  Nobody understands the problem domain like a founder, nobody understands the solution’s capabilities better than a founder, and nobody else can say “You can trust me on this because I build the bloody company” or inspire the same degree of emotional response in a customer that talking to a founder provides.  (This is so true in my personal experience.  I hear, over and over again, from my customers that they trust me because they know they get an answer “straight from the top” when they send me an email.  I hear that most often from people who never emailed me in the first place, or who emailed me with issues that could be easily dealt with by Level 1 CS.  This is one compelling reason why I don’t outsource customer service.)

Rob Walling

Rob writes Software By Rob, runs a stable of small software businesses as a one-man show (including, most amusingly to me, selling beach towels — he focuses just on software these days), and wrote a book I really enjoyed.  (Flip to the chapter about Virtual Assistants, it is worth the price of the book and then some.)  The slides and outline for his talk are online.

After review of statistics generously donated by several software companies, Rob is of the mind that the most important goal of your website is to bring people back for additional visits, because the conversion rate of your website is much, much higher among engaged visitors than it is among first-time visitors.  I didn’t write down the exact numbers, but I recall him citing DotNet Invoice (“written in Ruby on Rails… no, just kidding”), CrazyEgg, and the beach towel site among a few others.  Interestingly, a few of the examples had higher average order values for returning visitors, too.

I only investigated this for about 30 seconds, but I have lower conversion rates to the online free trial of my software from returning visitors than first time visitors.  That actually isn’t contrary to his thesis, though — it is an artifact of having a very scalable SEO strategy and customer acquisition highly dependent on high-converting organic and AdWords landing pages.  This is one of the threats of making decisions based on shallow understanding of what Google Analytics tells you is the “truth” of your business.  (One minor surprise: about 60% of my customers who convert from the free trial convert within 3 hours of starting it.  Nail that first run experience and nail it hard, guys.)

“The ineffective marketer asks you to buy too soon.”  A customer on their first visit could have multiple reasons why they are not currently prepared to buy your software:

  1. They don’t have enough info about it.
  2. They don’t trust you.  (Oh goodness, this topic is worth a book.)
  3. They don’t have money to buy it right now.  (A particular issue for a $300 invoicing software.)
  4. The don’t need the software right now.
  5. They’re never going to buy this.

You can recover from all the issues but #5 if you establish a relationship with them.  Permission-based email marketing is the best way to do this, bringing them back to your website via a communication channel everyone has and uses.  Repeated, trust-building contact with them will gradually wear down their resistance to parting with their cold-hard cash.

Email is far superior to competing methods of repeatable contact with customers — such as Twitter, blogs, RSS feeds, or your favorite social network — because it allows “personalized broadcasts”, where you can scalably communicate with an arbitrarily high number of people while also making it feel like everyone is getting individualized attention.  (Not nearly enough people use this to its maximum potential, particularly with SaaS.  Free tip from me: note what search term sent them to your website, put a variation of it in the subject line of the email, watch your open rate go through the freaking roof.)

To get the maximum value out of emailing your customers:

  1. Have a killer landing page laser targeted on the “give us your email in return for this immediate benefit to you” call to action.  Give them a whitepaper or similar resource in return for the email.  It greatly, greatly will increase your conversion rate.  (Rob is not a fan of free-to-use trials.  That is an interesting idea to me, as I come from a “of course the software is free to try out” background.  He has achieved truly eyepopping conversion rates to an email submission — 40%+ on some pages.  I can’t get that to my free trial, which also requires an email address, and I have pages which are very, very compelling at “selling” that to customers who were looking for exactly what the free trial offers.)
  2. Give something away in return for the email address.  See the slides for specifics, but this has a monumental ROI.  (I have some confidential numbers from consulting clients.  Suffice it to say this is right on the money.)  Interestingly, if you just call a white paper a “report” it has higher perceived value.  (News you can use!)
  3. Set follow-up for emails.  MailChimp, my mail provider of choice, calls this an autoresponder sequence.  If you take one thing from this presentation, learn and use autoresponders.  They will change your email marketing life.

Some excellent tactical suggestions:

  • Spam: Don’t spam!  Don’t mistakenly get classified as spam by hitting filters.  You can pre-test your emails against common filters, like SpamAssassin, to reword against accidental false positives.  (A long time ago I worked on anti-spam systems for a living.  Oh, the hilarious stories I could tell you.  Suffice it to say that a home builder might want to avoid promising that they could “increase the size [of your patio]”.)  Incidentally, MailChimp will do this for you for a nominal per-email fee.
  • Some words kill open rates if you use them in subject lines.  Reminder, Special, % off, and “Help [us]…” are all culprits.
  • From Name: your own personal name is best, and role names are a bit worse, company name is a bit worse, and webmaster@example.com is worst of all.
  • All advice on emails can be A/B tested!

Peldi (Giacomo Guilizzoni)

Peldi runs Balsamiq, a small software company which has become practically synonymous with low-fidelity mockups for software.  He wrote the best blog post on the Internet about seeking coverage for your startup.  You should execute on it.

The theme of the talk was worries: worries you shouldn’t have, worries you will have, and what keeps Peldi up at night these days.

Things you should not be worried about:

  • Asking customers to pay you money.  (Both Peldi and I were shy about this prior to starting, apparently, because we had the genetic flaw all engineers have that says that there might be a bug and, therefore, it is too early to accept appreciable amounts of money for the product.  I promise to do my part and say “Charge more” to any person that will listen to me until we have cleansed this from our collective gene pool.)
  • Pirates.  (Not an issue in practice, you can pretty much fire-and-forget a low-pain DRM system, and since if you’re smart you’re doing web apps anyway this is disappearing from the radar screen entirely.)
  • Picking a niche which is too small.  (Peldi put up a screenshot of Bingo Card Creator here, to the widespread amusement of the crowd.  I later heard more than one attendee say that they were under the impression it was “a funny Italian joke” until I gave my lightning talk.  That actually worked out great for my talk’s reception.)

Early worries in the lifecycle of his company:

  • When to make the jump from full-time employment to starting your own business.  If I recall the talk correctly, Peldi scrimped and saved for a while, sold his shares of Adobe, and emptied his IRA so that he would have a year of runway in hand prior to going full-time on Balsamiq.  He also had written code on nights and weekends for several months, so he was not doing a standing start.  (And his rise was truly meteoric after that — a combination of product quality and being a marketing genius.  If you don’t understand why he is a marketing genius, go back and read that post I cited earlier.)
  • Work/life balance.  Regrettably, you’re going to have to get your family on board with losing you for a few months while you disappear into the Bat Cave and hammer out your software.  (This is my one bone to pick: no, you do not.  You can virtually always stretch out your schedule.)  Peldi then delivered the best line of the conference: “If you work while they sleep, they won’t know that you’re ignoring them.”

Many people are under the impression that Balsamiq was an instant, overnight success, largely based on Peldi’s embrace of radical transparency and publishing revenue numbers.  Peldi had a series of very interesting graphs: the first shows the hockey stick revenue curve that competitors salivate over.  Then it zooms out to include the time while the software was getting written (i.e. no sales).  Then it zooms out to include his previous career at Macromedia/Adobe where he learned to polish software to perfection in the way that separates Balsamiq from its many, many competitors.  Then it zooms out to include his programming career starting at age 12.  Overnight success only took thirty years.  In a conference with many profound insights, I think this was one of the best ones.

Let me give you an example from my own experience: I recently received a wire transfer from a consulting client, for an amount of money which would be unexceptional to some people and which is staggering for me — more than I’ve ever had accessible in my entire life, by a factor of lots.  It was for two weeks of work and was roughly equivalent to what I used to make for six months at my previous day job.  The story for how I got to do those two weeks of work starts out with participation on the Joel on Software discussion boards some four and a half years ago.  I started dabbling in the topic the work covered almost six years ago.  The focus on communication which made it possible to sell the client (and eventually, their team) on the importance of the work getting done can be traced back probably to middle school (when I took up public speaking to get over a speech impediment).  You literally needed an entire lifetime to prepare for what you’re doing today.

Back to Peldi: don’t be afraid of an inability to deliver.  You’re a talented engineer, you’ll figure it out.  If you fall in love with the problem and can discuss it with boundless energy, then every interaction with customers will make your product better.  At early stages in the product lifecycle, feedback from customers is a much better thing to capture than mere money.  (Peldi gave away licenses like they were candy during the first few years.  Again, read the above blog post, in addition to getting great feedback this was a great PR and linkbuilding hook.)

Peldi had an extended discussion of how you can use Obama’s debate script as a model for addressing your fear of confrontation with customers.  While I’m not a fan of his politics, I agree, he is an excellent communicator.  There was a flow-chart involved, but the core is:

  • Thank the customer for taking time out of their busy life to communicate with you.
  • Empathize with them.
  • Apologize if they’re ticked off, even if they have no right to be.  (Oh, crikey, how much do I agree with this.)

Lightning Talks

All of the Lightning Talks were delivered by non-professional speakers.  They followed a murderously difficult format: you get 15 slides which get auto-advanced every thirty seconds, and that’s it.  You might think that means you can get away with slapping together something in an hour: oh no.  In discussions with my fellow lightning talkers, we agreed that there has been something of a “lightning talk arms race”: the two talks I was most impressed by took over 24 hours to prepare, and mine took at least twelve solid hours over two months, with probably half of that being rehearsal until I could literally count out thirty seconds with a prepared spiel delivered in a voice other than my own.

I’m going to refrain from talking about my own lightning talk.  Why?  First, it relies on a “reveal” for a great deal of the impact.  Second, it was, far and away, the best speech I’ve ever given (out of thousands — some people play sports, I did public speaking).   The combination of heartstopping terror and the crowd reaction produced a strange alchemy which made the talk as delivered a lot better than the videos I made of myself performing it in a room for my webcam.

I’m told it was recorded and the video will be made available to the public.  I’ll tell you when it is ready.  I promise you, you’ll enjoy that a lot more than you’ll enjoy me posting the slides and a video of me saying virtually the same words to a wall.

As to the lightning talks other than mine, a combination of heartstopping terror about my talk and the exigencies of the format mean that my notes on them are woefully inadequate.  I expect that a few of the lightning talks will eventually be put online at some point.  Those videos will do them better justice than my hazy recollections, so I recommend waiting and seeing them.

Eric Sink

Eric presented the nuts and bolts of what happens if your company (or, in his case, a product thereof) gets acquired by BigCo (in this case, Microsoft).  Short version: a lot of pain and suffering, followed by a check so large that the VP of his community bank said putting it in his account would throw their company into absolute havoc because of the impossibility of putting so much money to work.  Let me emphasize the pain and suffering part, though.  (The low point: when the deal had consumed his waking life so much that Eric was reduced to taking a phone call from his lawyer while in church.)

One of the weirder points for Eric was that, with a cast of literally dozens working the Microsoft side of the deal, he never met or learned the identity of the person who actually made the decision to buy the product.  In a gang of people with confusingly similar job titles and descriptions, it seemed that “nobody was in charge.”  The deal was lead by a particular point of contact — we’ll call him Fred — who, as a trained negotiator whose full-time job is assimilating biological and technological distinctiveness, was a thousand times more prepared than any entrepreneur will ever be.

The process involves a long rigmarole of procedural steps.  You’ll want to see the full talk for the breakdown, because I’m hazy on it, but it involved multiple levels of due diligence, getting signed agreements covering IP rights from every person who had ever put a finger to a keyboard within a meter of the source code (cough start work on this today if you want to sell cough), etc etc.

While many entrepreneurs obsess about The Number, what you actually get is The Deal, which ended up being 150 pages of bullet points all individually negotiated covering every aspect of the transfer.  This included guarantees of treatment for employees who were relocating with their product from central Illinois to Microsoft, all manner of indemnifications, tax treatment of the deal, escrow (code and otherwise), the exact structure of compensation for the deal, the terms of the NDA covering material terms of the deal, etc etc.

The deal nearly fell through at several points, over contentious negotiating issues.  Nearly might not be strong enough: it was dead.  The deal was off, and the other side went to total radio silence for a period of weeks.  This was necessary to get the best achievable outcome for Eric and the team.  Eric is of the opinion that you need to walk away about twice: less and you’re getting screwed, more and you’re buying more heart attacks than the marginal improvement in terms is worth.  (One friend of his — in the elite fraternity of “people who have sold a company”, whose stories “you only become privy to after you have joined the club” — had to walk away five times.)

One interesting point raised in discussion was the importance of having multiple interested buyers so that you can play them against each other.  (Michael Pryor from Fog Creek is especially articulate about this one.  The short version: if your BATNA is “get bought by the other guy”, you have more leverage than if your BATNA is “you don’t get acquired, six months of work flushes down the drain, and the company potentially withers.”  The long version: ask Michael.)

Dan Bricklin

So here is a generation gap question for you: I didn’t know who Dan Bricklin was, and someone said “Well, duh, he’s the guy who wrote VisiCalc.”  I then had to Wikipedia VisiCalc because I had never heard of it.  For those of us who were born in the eighties or later: he invented spreadsheet software.  Yowza.

In addition to a hundred and one precious anecdotes about inventing spreadsheet software while a poor grad student (my favorite: the development machine and sole source repository, which cost the team’s life savings, being protected from a plumbing problem by constant vigilance with mops), there were a couple of generally useful takeaways:

  • Customers have jobs that they need to get done.  Identify those jobs.  Build general purpose software which allows them to get the jobs done.
  • The perceived value of general purpose software, which solves a variety of current and anticipated needs of a customer, is higher than that of software which solves one specific pain point.  (I respectfully disagree, but then again, I didn’t invent the spreadsheet.)
  • Software which the customer can customize to their particular needs allows “tailoring at the ends.”
  • The two-week payback rule: software sells itself if you can reliably demonstrate that your product will recoup the purchase price within two weeks.  VisiCalc (spreadsheet on a mini-computer) was so disruptively better than using a time-sharing system to do calculations that you could buy the software and the system to run it for just half of one month’s metered use of the big mainframe you leased time on to run your TPS reports.  That contributed to it selling 700,000 copies, back when distribution channels for software were stone age compared to the alternatives we have today.  (My unconscious brain was virtually screaming: “You can sell software without a website?!” when I saw the magazine ad slides.)
  • Games and relative cheapness were very important in driving new technology adoption.  Note that this, and the “general purpose applications win” rule, are demonstrated in abundance by the runaway success of the iPhone.

Derek Sivers

Derek had originally planned on delivering a talk about various profit models for software companies (spiritually similar to this talk).  I met him for the first time at a speaker’s dinner.  During the course of dinner, one topic frequently returned to was why we do what we do.  Derek told a very moving story about how he sold CDBaby, his labor of love for many years, and someone (I think it was Rob Walling) suggested that he do that as a talk instead.  He ended up doing so.

Where to start:

Derek originally got into CDBaby to sell CDs of his own band’s songs.  Back at the dawn of the Internet, getting a merchant account for selling CDs was a multi-month process, with plenty of administrative hassle and pain involved.  Derek persevered and, with no programming background, managed to hack together enough CGI scripts to get a buy now button on his website in a mere three months.  (And again I bless my lucky stars that I had Paypal and e-junkie.)

After he started selling CDs, some of his friends with bands said “Hey man, that’s really cool.  Can you sell my CDs, too?”  So he’d hack the scripts again and then he was selling another CD.  Eventually, word of mouth had gotten to the point where his band’s website was colonized by other bands who lacked a good way to sell their CDs online, and CDBaby was born.  His blog covers this story in detail.

Derek was once interviewed by NPR and, when asked about his exit plan, said that long after the sun had set on CDs and when he was old and grey, he’d still be shipping CDs to the last aging person who swore that the CDs they played when he was young were far superior to that newfangled hologram 7D garbage that passed for music in 2050.

This turned out not to be the case: after a major rewrite of the CDBaby source code, where Derek achieved pretty much every technical goal he had for the site (and had paid back years of code debt), and after changing his management style such that he became totally superfluous in day-to-day operation of the company, Derek became an absentee owner of the company he founded.  Some people might view that as a wonderful outcome.  It turns out that Derek’s employees viscerally hated him for it: convinced that they were the true bearers of the CDBaby vision, rather than the unseen owner who many of them had never even met, they began organizing what could be described as a corporate coup.

When Derek found out about this (oh, the joys of reviewing MP3 recordings of meetings at midnight), he realized that he was well past the point of no return with regards to employee loyalty.  So he turned Apache off and started composing an emailed pink slip for the entire staff.  No, really.  Luckily for all concerned, he eventually decided to sleep on it, and came back to advice he had received from Seth Godin: if you are no longer passionate about the problem, you owe it to your customers to sell to someone who is.  Apache went back on and the email was not sent.

Derek pursued a few options for getting rid of the company.  One was to simply give it away: he actually pursued pulling off a Willy Wonka and putting five golden tickets in CDs delivered by his company, with the five holders invited to present their visions for the future growth of the company, and the winner (as decided by a vote of customer musicians) receiving the entire company.  Due to fortunate advice from friends regarding the desirability of being sixty and having no money, Derek threw away the golden tickets he had gotten printed and decided to do the more usual thing and just sell.

The sale came with a wrinkle: Derek first transferred the company into a charitable trust, then sold it, meaning that he is forced to draw 5% of the sale price per year, and when he passes away the principal and investment gains (if any) of the trust will be distributed to the charitable causes of his choice.  Derek claims this is mostly not for altruistic purposes: he asked his lawyer how to achieve “I want to have a stable income until I die, and that is all I need”, and the charitable trust was an attractive option.

Derek mentioned that having competing bidders was wonderful, since they could be played against each other.  This helped him achieve some goals he had in selling the company:

  • I’m out.  (Founder participation as a consultant or employee for a defined term is a frequent requirement of sales.  Derek took that option off the table entirely as a precondition of sale.)
  • I keep my database.  (A decade of running essentially a marketplace between music sellers and music buyers gives Derek a powerful foundation for future endeavors aimed at helping musicians, his perennial concern.)
  • I keep helping musicians.  (The non-compete was narrowly tailored so that he would not sell CDs again, but could continue participating in the music industry.  Sometimes non-competes are are very broadly drafted — I’ve heard possibly apocryphal reports from one that, boiled to its essence, barred a technical founder from programming for the duration that a large company with a small-sounding name continued selling operating systems.)

Joel Spolsky

I’m pretty sure most people know Joel co-founded Fog Creek and Stack Overflow.

Continuing the theme of the later part of the conference about big picture questions like “Why?”, Joel’s presentation was probably the bravest speech I’ve ever heard.  Theoretically, it was about the story of transitioning from a low-growth (ish) software business like Fog Creek to  a high-growth VC-funded shoot-the-moon attempt like StackOverflow.  However, it dwelled on some intensely personal questions about leadership, philosophy, and intra-team conflict, and I don’t know if I could have discussed them at that level in public.

Let’s see: the brief takeaway on seeking investment is that if you’ve got a successful track record as a well-known CEO and a good idea which also has demonstrable traction, raising capital is fairly easy (Joel and company were practically booked solid when they went to Silicon Valley to meet with VCs).  However, it still has some of the industry seediness.  One VC who they met with, who ended up not investing in the company despite early promise, later tried his darndest to cause a rift between Joel and his co-founder (Jeff Atwood) so that he could pick up the pieces.  Another VC was quite interested in StackOverflow, provided it was CEOed by anyone other than Joel.

Interference from VCs aside, they eventually got signed by Union Square Ventures, who have treated Joel and company fairly.  The cultural gap between a self-funded software company and the VC world has caused a few disconnects (for example, at one point they found that StackOverflow had $200,000 lying around that they had forgotten to disclose based on revenue, and they asked what corporate structure should receive it or whether they could licitly award it to employees: the VCs were totally uninterested in $200,000, and rather amused at the notion that a company they had funded could actually have had a profit from operations at the stage they were at).

StackOverflow has a very different corporate culture than Fog Creek, which Joel describes as a software company founded with the goal of becoming an “intentional community” whose mission in life was providing programmers a great place to work.  It was explicitly analogized to a kibbutz, with collective ownership, the profit being redistributed among employees, and decisions specifically made to promote community (one which stuck with me was the huge importance placed on breaking bread together on a daily basis).

The culture involves decision-making by consensus.  For a variety of reasons, StackOverflow does not operate in this fashion.  This lead to some contentious battles between Joel and Jeff, exacerbated by the aforementioned evil VC, who tried to split the founders to pick up the company on the cheap.  Among other things, After frequent screaming and crying, Joel figured that his traditional style of hiring smart people and letting them get things done wasn’t working, and he called in a CEO coach for assistance.  He thought he’d get management advice on how to better run meetings and write org charts and do the things that CEOs do.

Instead, he got corporate psychoanalysis, and was introduced to the notion that the CEO’s job is not to manage the company, but to lead it.  This was particularly important as he was CEOing two companies simultaneously.  Once he started accepting that “I am the CEO, I have heard your viewpoints, we do it my way and that is final” was also a valid managerial style, conflict in the team ironically decreased greatly.

On the question of why we do what we do: Fog Creek is the company Joel designed to have a great place to work.  StackOverflow, on the other hand, is targeted at contributing one significant thing to the world: fixing the broken style of asking expert questions on the Internet.  They want to bring the improvement StackOverflow brought to programmers to the wider community (and communities) of interest who Google questions which have canonical good answers and find a SERP stuffed with 10 year old VBulletin threads where the answer is on page 7 in between a cat photo, a flame war, and the announcement of the forum’s guru’s goddaughter’s wedding.

Joel would prefer to work at Fog Creek, but thinks he has an ethical responsibility to make StackOverflow succeed first, because — irrespective of Fog Creek maximizing his personal enjoyment — StackOverflow has the potential to contribute a large social good to the world at large, and he views this as ethically mandatory given the opportunity.  He contrasts this to the 37Signals-esque notion of being content to run a small business because it allows you financial success and the lifestyle you enjoy (although he was quick to note that, while saying this, 37Signals gave the world Rails and teaching, so their own ethical bases are covered).

For an example, see CraigsList, which has moved a huge amount of wealth from newspapers to the posters of classified advertisements by making classifieds free.  CraigsList is run with a goal other than profit, and would instantly be a gigantic company if they charged for ads in categories other than the ones they do now (which they do as a spam-control device).  Joel argues that Craig should do this, because the massive wealth transfer his business enabled has redistributed the producer surplus from newspapers (which once used it for a socially beneficial purpose: underwriting unprofitable but socially beneficial local investigative journalism) to a consumer surplus for “people selling couches, landlords, and pimps.”  His point was that, when we know the trade-off our businesses are making, we have an obligation to choose the results wisely.

I am perhaps not doing this point justice: the two minute thumbnail sketch of the argument I heard Joel deliver profoundly altered my perception of where my company fits into the larger tapestry of my life.  I would previously have characterized, e.g., the decision to keep the company small as being essentially a personal aesthetic choice with no moral consequences, akin to picking colors for one’s bedroom.  I will be digesting the implications of this for a while, but I think I am now convinced that that is probably untrue.

A sidenote: since, pace Seth Godin, the cost of producing software is cratering and the competition is exploding, Joel is gradually coming to the belief that the value in a software company isn’t the code, it is the community.  The thing StackOverflow had to offer VCs wasn’t the code (which, to quote what has become an in-joke among HN readers, “could be written in a weekend”), it was the “large number of engaged users.”  Microsoft estimates there are 9 million programmers in the world.  StackOverflow has upwards of 8 million monthly uniques.  They’ve become the canonical go-to place for answers in the programming niche, and have dreams of becoming the canonical go-to place for answers for all expert queries which have canonical right answers, from tax advice to math theorems to gardening and many points beyond.

Commitment To Community & Disclaimer

(Back to top.)

Like I mentioned earlier, I strongly recommend you attend Business of Software.  Even at close to 10,000 words, this post captures a bare fraction of the value I got out of going to the conference.  I understand that both the ticket and the ancillary costs of attending are quite high for bootstrapped startups.  Trust me on this one: a plane ticket from Japan to Boston and four days in a hotel wipe out nearly a month of revenue for me — and it was worth every penny and then some.

Neil Davidson, the conference organizer, was very generous with giving away tickets to the event to deserving bootstrappers.  In the spirit of the overwhelming generosity of the community and the thoughts sparked by talk of our ethical responsibilities, I will likewise make an opportunity for at least one marginal person to go next year.  Watch this space — I’ll hammer out specifics closer to the actual event.

Now, the obligatory disclaimer: have never accepted advertising on my blog and don’t feel like starting anytime soon.  I was given a free ticket to the conference in return for presenting my Lightning Talk at it, and after winning the competition for best Lightning Talk (which I literally did not know existed until five minutes before it was announced that I won it), I received a Kindle and an invitation to a very nice dinner.  I’m easy to impress, but not that easy: my effusive enthusiasm for this conference comes from it being one of the highlights of my professional career, not because of compensation.