Archive | Uncategorized RSS feed for this section

Quantifying The Value Of A College Degree (By Major)

“Your single best investment is your own education.” ”The average new graduate is drowning in $22,000 in debt.” ”A degree in English is just as valuable as a degree in Biology — it teaches you critical thinking!” “Follow your dreams and you’ll find financial success whatever your major is!”

You’ve probably heard a thousand pieces of educational/career advice like these, and if your family/friends/advisors are anything like mine they’ve virtually never been substantiated by data.  That is a shame, because choosing a course of study is one of the largest transactions you’ll ever make — the sticker price at my alma mater was close to $140,000 and prices nationwide have risen faster than inflation for virtually a generation now.  We have the Blue Book to tell you what a ten year old beater is worth down to the dollar, there are entire industries devoted to assessing every type of security to determine their valuation, and the closest thing most students have to insight on the degree selection process is getting advice from Aunt Sue.  This is insane.

Information Asymmetry In Employment Outcomes

Any college could rectify this situation virtually overnight: take that lovely list of alumni that they obsessively curate for squeezing donations, send out a two question survey (“What did you major in?” and “What was your salary this year?”), and give a sociology grad student a bowl of ramen to do some very simple number crunching.  No college will actually do this because transparency goes directly against their interests: if all degrees from a particular institution are valued at “An uncertain, but certainly large and roughly constant number”, then the standard practice of pricing them all identically makes sense.  If not, it is the academic equivalent of pricing stocks by length of ticker symbol.

(I understand many folks enjoy the non-economic component of their education.  I did and do, too, but since I’ve never spent $120,000 and signed myself up for a decade of debt for the sheer enrichment offered by attending a ballet or reading about the Japanese economy I can only conclude that I don’t value it anywhere near what I paid for my degrees.  Your objection to the narrowness of this study is duly noted, though.)

You might assume that the government would track this, somewhere, but you’d be wrong.  The Census Bureau produces a report every ten years tying level of education (associate’s degree, bachelor’s degree, master’s degree, etc) to salary, which invariably produces the result “More education is better”, but they don’t answer very interesting questions like “Is a bachelor’s degree in computer science better than a master’s in elementary education?” or “Are there fields with a sharply diminishing return to additional education?”

However, the Bureau of Labor Statistics does very comprehensive work in administering a National Compensation Survey, which gathers huge amounts of raw data about employment hours and wages broken down by region, job (over 800 classifications available, from CEOs to ship loaders), and a few other axes.  They use this and other studies to produce a variety of government reports, such as the Occupational Outlook Handbook, which does a good job of providing per-career advice but probably intentionally omits comparisons between careers.

The Bureau of Labor Statistics also makes the data from the NCS available for download on their website.  It is hefty — over a gigabyte of plain text — but it contains a virtual treasure trove of value… if you just know how to read the map.

Liberating Conclusions From Open Data

Recently, a big buzzword in the tech community has been Open Data: the notion that the huge, monstrous streams of raw facts collected by the government can be exploited for our benefit if they are merely shared.  I think this is mostly true: the best single example I’ve heard of is that since your local health authorities inspect every restaurant’s premises as a matter of course they must know where they all are located, and therefore one should be able to get those locations from them and jumpstart the creation of a guide to local restaurants without having to find every one by yourself (a monumental undertaking).

However, raw facts are uninteresting.  Here’s a line from the BLS data:

NWU009910010200000000000016260,2008,M07,69.71

Scintillating stuff, right?  What we are really interested in is what those facts can teach us — in particular, what can they teach us that allows us to make decisions such as what to major in.  This is where your friendly neighborhood computer programmer comes in: with a bit of elbow grease and a laptop, you can reduce the 856,000-odd facts in the government’s salary data to some useful conclusions about college majors.

A Bit Of Science And A Dash of Art

Sadly, there are limitations to what we can accomplish with the BLS data: it groups salary data by occupation rather than by major and degree level.  The BLS separately identifies for each occupation what the probable degree level is, but going from occupation to degree requires a bit of guesswork.  Rather than associating all occupations with sets of related degrees by hand, and injecting my own biases into the analysis, I decided to crowd source the problem: I paid Mechanical Turk workers for their two cents (quite literally) on what degree an e.g. elementary school teacher likely had.  This produced answers like Education, Early Childhood Education, Teaching, English, etc.

I then used an arbitrary level of consensus as a cutoff, and was able to pair ~60 very popular degrees (Computer Science, English, etc) and ~250 less common ones (Vocational Education, Media Technology, etc) with associated occupations.  Some additional number crunching let me construct rough estimate salary-versus-age curves for the occupations, which could then be reduced into a simple net present value calculation.  Long story short: a lot of data, a bit of science, and a dollop of absolute voodoo — it’s sort of like most social science, except I’m going to be honest about the voodoo upfront.

After doing the calculations I used Ruby on Rails and some open source graphing libraries to present the results in a comprehensible, searchable fashion — similar to the data visualizations done by the New York Times, which are some of the best work they produce.  (Check this one on the geography of the recession for the general feel.)

Why Go To All This Trouble?

Short story: Intellectual interest plus a nice paycheck.

Longer story: I do very occasional consulting work for a variety of clients.  In case you haven’t noticed from the six-figure sticker prices, offering degrees is a very big business.  Any flowing river of cash that large attracts, as if by magic, a variety of service providers around it.  In education, one major problem colleges have is finding prospective customers to sell degrees to.  This is hardly a unique problem for businesses.  (Colleges may prefer to phrase this as “students” to “award” degrees to, because they are intellectually committed to a view of themselves as custodians of the lamp of human knowledge rather than rapacious money-grubbing institutions.  I don’t know of any reason they can’t be both.)

One thing colleges — from the Ivies to state schools to online for-profit institutions — spend absolutely gobsmacking amounts of money on is “lead generation”.  Basically, since a percentage of applicants will eventually matriculate (and pay five or six figures for the privilege), when a qualified prospect fills out an application that is an economically beneficial event.  You can compare this to a conversion to the free trial of a web service.  Universities are willing to pay quite a bit of money if you can induce someone to apply: the payout varies by university and agreement, but payments in the $10+ region just for requesting an application packet are not uncommon.  (And if you had some magic way of generating sought after candidates — say, highly qualified African American students — you could almost certainly negotiate much, much higher payouts.  There might still be some Marxists on the faculty but it is all capitalists in the administrations.)

Anyhow, with universities offering to pay for lead generation, there is an entire value chain created from the ether: sites to capture the leads, affiliate programs to direct folks to the lead capture sites, advertising to attract visitors to the affiliates, publishers to create content which displays advertising, etc.  One of the publishers in the industry, Online Degrees, hired me with an open brief: make something compelling for our website.  I thought since universities, academics, and the government have failed to produce any actionable data on which degrees make sense to go after, I could do some independent research on the subject.  Online Degrees.org will host it on their website, and in the course of providing value to potential students researching the subject, they’ll have an opportunity to display ads for degree programs.

You might be concerned about the impartiality of this.  I don’t blame you.  I’ve got no particular dog in this fight: I get paid by the hour no matter what degree wins.  (Cards on the table: I have degrees in Computer Science — which is in the data set — and East Asian Studies, which is not.)

Online Degrees.org obviously has a vested interest in convincing you that a having a degree is better than not having one, but they’re agnostic about which one you apply for.  Indeed, they’d love to tell you which fields are better than others because somebody in the industry needs to have the credibility to say that e.g. culinary school is tantamount to grand theft (and most of the victims take out loans for the privilege of going through it).

Besides, do you really have a better alternative?  If I had a PhD in Sociology, would that make me a less biased source of information on the desirability of becoming a cheap source of exploitable labor a master’s candidate in Sociology?

Anyhow, I have been intellectually interested in this subject for several years now.

Quick Overview Of Results

For the results of most particular interest to you, take a gander at the degree value calculator.

Regular readers of this blog are mostly technologists of one flavor or another, and degrees in technical majors do very, very well.  Computer Science and Computer Engineering are near the top among all options for bachelor’s degrees.  It is narrowly bested by a handful of degrees tailored around resource extraction: for example, if you study Geology, Big Oil will apparently pay you Big Bucks.

Hard sciences such as Physics and Biology pay rather less well than I would have expected.  Degrees in the humanities perform about as poorly as people often joke.  The largest surprise to me was that degrees, even advanced degrees, in some caring professions (like Social Work) are apparently terrible options.  Looking at the underlying data suggests that this because many social workers do it as a part time job.  (That is a recurring theme among many jobs that I expect people would classify as more likely to be female than the typical occupation.  Food for thought the next time someone brings up the wage gap.)

You can see the results of this research on their website.  [Edit as of 2/19/2013: You'll have to search for this directly, due to link rot.]

Questions?  Comments?  Criticisms?  I’d love to hear them.

Bingo Card Creator (& etc) Year In Review 2010

I’m Patrick McKenzie.  For the last few years, I’ve run a small software company which, most prominently, makes Bingo Card Creator.  It creates… well, you probably get the idea.  I recently launched Appointment Reminder, which… yeah.  I also do occasional consulting, shockingly not as Pay Me Lots Of Money And I Solve Your Problems LLC.

In addition to publishing pretty much live stats about the business, I always do a year-end wrapup (see: 2006, 2007, 2008, 2009) covering my thoughts on the year.  I hope folks find it interesting or informative.

Disclaimers: Stats are accurate as of publication, but the year isn’t quite over yet.  Ordinarily the last two weeks of December are fairly slow, but I would expect there to be a few hundred dollars more sales and possibly a few hundred more in expenses, depending on the timing of people charging my credit card.

My business is a good deal more complicated now than it was previously, which changes how open I can be about some bits of it.  See below.

The Big Change

After several years working as a Japanese salaryman, I quit my day job and went full time on my business as of April 1st of this year.  This was the best decision I have ever made.  Words cannot describe how happier I am with everything about my life: I see my family more often, I see friends more often, I travel more, I work less, I make more money, I’m healthier, my apartment is cleaner (well, OK, modestly cleaner), and I enjoy work a whole lot more than I ever did when I was working for somebody else.  Self-employment is awesome.  There are occasionally challenges to it, but since I spent a few years dipping my toes into the water prior to doing it full-time, I have had very little of the “Uh oh, I might not be able to make rent” uncertainty that a lot of folks report.

Bingo Card Creator

Bingo Card Creator remains my bread and butter for the time being, but I think this is likely to be the last year for that.

Stats:

Sales: 1,422 (up 38% from last year’s 1,049)

Refunds: 20 (down from 24 last year, to 1.4% of sales from 2.3%)

Sales Net Of Refunds: $42,589.90 (up 33% from $31,855.08)

Expenses: $16,685.24 (up from $15,485.28)

Profits: $25,904.66 (up 58% from $16,369.80)

Wage per hour: $200~250ish, based on my best guess of time spent on BCC (yeah, I went “full time” and work less than ever.  This is mostly because BCC is mature software.)

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 777,000 (up from 546k)

Unique visitors: 655,000 (up from 470k)

Page views: 2.7 million (up from 1.6 million)

Traffic sources of note: Google (44%), AdWords (20%), Binghoo (11%)

Trial downloads: 21,000 (down from 56,000)

Trial signups for online version: 72,000 (up from 17,000)

Approximate online trial to purchase conversion rate: 1.7%

Narrative Version:

The major change in BCC this year was unofficially deprecating the downloadable version of the software, for a variety of reasons.  This led to a huge cut in my support investment — the downloadable version generates 10 times the work per customer that the web app version does — which helped enormously when I dropped BCC into maintenance mode, which it spent over half the year in.  (Maintenance mode means I answer questions and collect payments and that is about it.)  I did a bit of experimentation over the summer in terms of conversion funnels, and also did some major work in October with regards to seasonal promotions and using my mailing list.

My conversion rate for BCC is slipping steadily.  This is sort of ironic, because it is the result of an unalloyed good thing for the business: as I get better at getting organic search traffic, the population of people using my web site moves from the teacher-heavy target-rich-environment it has historically been and towards a broader audience who don’t have quite such a pressing need for bingo cards.  Sales go up, since converting 1% of a big number is still a good thing, but it dilutes my aggregate conversion rate.  Oh well.

I’m mildly disappointed that I missed my sales and profitability targets this year (by about $2k and $4k respectively).  Oh well.

What Went Right:

  • Deprecating the downloadable version reduces my work and stress level attributable to BCC enormously.
  • Getting serious about using MailChimp and email marketing in general. My Halloween, Thanksgiving, and Christmas mail blasts made well over a thousand dollars in sales for me, for about $30 worth of virtual postage stamps and perhaps twenty minutes of writing.  One thing to note for next year: there does not seem to be a substantial difference in conversion rates for when I put a $5 off coupon in the mail versus when I don’t, so I shouldn’t next time.  Also, given that a huge percentage of folks bounce on the password screen coming from the email, I need to think about either putting a token in the URL to let them in automatically, or making dedicated landing pages for these campaigns.  (I don’t have good numbers for how effective the autoresponder sequence is — i.e. automated emails I send to people on the first and sixth days after they sign up.)
  • Meat and potatoes SEO continues to be my bread and butter (how is that for a mixed metaphor).  My conversion rate has been gradually sliding as I get more parents in for holiday bingo activities through my Sprawling Bingo Empire, but 1% of a very large number is still worthwhile.  I should probably get serious about optimizing those sites individually, but my desire is waning.

What Didn’t Work So Well:

  • In the wake of the FireSheep release, I decided to implement SSL for Bingo Card Creator, right in the middle of Halloween busy season.  This broke my pages in several different ways, from causing security popups on the login screen in IE (whoops) to not showing key images on landing pages in some browsers (whoops).  I really should have put off that implementation another few weeks, or thought through my testing strategy for it better.
  • I don’t have a staging server for Bingo Card Creator yet.  Having seen the enormous advantages from having a staging server through my Appointment Reminder, I am retroactively dinging myself for not making one in the last four years.
  • My Halloween promotion could have been handled better: $6,000 in sales is nothing to sneeze at, but I’m still of the opinion I could have broken $10k with a little better execution.  Maybe next year.
  • AdWords has been on auto-pilot for virtually the entire year, and when it goes onto auto-pilot it seems to optimize for Google’s bottom line over mine.  I should block off some time to get it under control, and aggressively cut weakly performing aspects of my campaigns.

Consulting

Last Christmas Thomas Ptacek at Matasano (whose office I am in as we speak) suggested to me that people would pay for my expertise in software marketing.  I was skeptical, but put the word out quietly that I was available for hire, and did three large projects this year for a few clients.  The only one I can publicly comment on at the moment is that I worked for Matasano, on stuff.  My general field of expertise is in engineering marketing outcomes: A/B testing, SEO, and the like.  Basically, I bring the fanatical iterative improvement mindset and apply it to things other than bingo cards.

I love talking about what I do.  Unfortunately, consulting clients pay a lot of money to get me to shut up.  This means, for example, not blabbing on the specifics of new projects which are as-yet unreleased, and not blabbing on the particulars of engagements.  Since I had only a handful of clients, giving exact numbers tends to give a wee bit too much detail, since if you happen to know one data point then you can guess the other two fairly easily.  So treat these as very ballpark numbers:

Consulting sales: $45,000

Consulting expenses: $10,000 (Plane tickets and hotel rooms get expensive quick, what can I say.)

Consulting Profits: $35,000

I know somebody is going to ask my rate, and I’m torn between a desire to quote it and the knowledge that there is absolutely nothing good that can come of quoting it.  The reality for consultants is that clients pay you exactly how much you can negotiate with them.  Everybody knows this and yet everybody would be shocked, shocked to know that someone else got a better deal.  In addition to causing problems with existing clients, it complicates raising rates for new clients in the future.  (Given that I could fill 100% of the hours I wish to, have been saying ‘Nope, sorry, not available’ to interesting work frequently, and now have a few CEOs singing my praises, my rates are only going up next year.)

What Went Right:

  • Client selection.  All three clients knew me well from the Internets, all had confidence in my ability to do what they wanted me to do, and all were model clients: they knew what they want, they communicated perfectly, and they paid all invoices in a timely fashion.
  • Pricing: Aside from frightening my bank a few times when I got large wire transfers from America, charging a lot of money is a great idea in every possible way.  It makes your clients respect your time more, keeps you motivated, and helps pay the rent during the lean summer months of the bingo calendar.

What Didn’t Work So Well:

  • Newbie consultant blues: I did my first consulting project for a friend.  Unfortunately, due to a combination of it being my first gig ever and my first experience with using Heroku, I greatly underestimated the amount of time the project would actually end up taking.  What I thought would be 20 hours over a two week period stretched into many more hours over months and months.  Luckily, my client was sympathetic, but I ended up doing a lot of writeoffs for good will and diverting my attention for longer than I wanted to.
  • Juggling consulting with project work: I worked 90 hours a week and still had cycles to spare for BCC.  How hard could it possibly be to do 10 hours of consulting in a week and still get productive work done?  Very freaking hard, because the 10 hours tended to be splayed across several days, require creativity and focus to execute, and not include the whole email/sales/support dance that comes with consulting.  This is in no way a criticism of the client, it is just to illustrate how consulting works: I got a request from a C-level exec at a company you’ve heard of in April, and despite us being mutually enthusiastic about the project it took forty emails and five months until billable work actually started on it.  I had to gracefully extricate myself from my clients and block off November and December to get Appointment Reminder launched.

Appointment Reminder

I finally released Appointment Reminder in early December.  (Don’t worry, I haven’t forgotten about writing the rest of the series about developing it.)  I have one customer already and a handful of prospects currently trying the software out, so revenue is negligible as of yet.  Now I just have to do the other 90% of marketing the software, which I am going to do a bit of in December and then start in earnest in January.  The plan is, unsurprisingly if you know me, heavily reliant on organic SEO.  I have also had a lot of interest about whitelabeling the software, and will do that as well.  That gives me a built-in on-the-ground sales force of people with intimate knowledge of potential clients and the desire to sell them my service — and judging from the numbers thrown around by one of the guys interested in white labeling, that could be “quite lucrative indeed.”

In terms of technical direction, I’ve engaged a freelancer to get it working on iDevices (I could have done it myself, but I’ve got plenty on my plate as it is).  I am about 60% of the way to getting a very impressively hacky solution working on every major US smartphone, because many prospects have asked to be able to access schedules when on the move.  The implementation is so devious that if I had a mustache, I would be stroking it while cackling maniacally.

Semi-exciting news: I received an acquisition offer from a foreign telecommunications firm.  The CEO believed AR fit a hole in their product line, and offered me their estimated development budget for it if I would sell them the whole business.  That was a generous number relative to the amount of work I have put in, but it would not have been lifechanging for me or my family.  I declined and told him I’d like to try to run the business myself and see where it takes me.

The stats:

Sales: Nothing yet.  (Well, one test transaction to make sure Paypal/Spreedly works.  Spreedly is impressively painless, by the way.)

Expenses: ~$1,600 (a few hundred bucks in design work, $800 of Twilio credit, and one or two other things.  Servers and online services which I also use for BCC got filed under BCC because I’m lazy and my bookkeeping software doesn’t support splitting bills: a more accurate accounting would be closer to $3,000.)

What Went Right:

  • It exists and mostly functions! These are both handy properties in software one wants to sell.
  • MVP available for several months.  I didn’t have the cycles to create AR back in April/May, but I did get the MVP of it — basically, an interactive demo of the core of the service — out early.  This helped me in both getting a bit of age and trust on Google prior to the official launch, and also in getting a base of interested prospects to try selling to (I’m currently talking to a few of them).

What Didn’t Work So Well:

  • Delaying release: April, May, June, July, August, September: that is six months, and I got very, very little accomplished (not one line more than the MVP, as a matter of fact).  The distraction from consulting work, working on BCC, and reacclimating myself to a human existence after years of salarymanhood just totally destroyed any desire to do heavy lifting on a new project.  I’m very obliged to the HN-based folks who started a Launch Something in November mini-sprint, which helped get me the energy to actually sit down at the computer for six hours a day five days a week and bang it out.
  • Insufficient pre-launch marketing: I really should have been constantly adding new content to the website for the last six months.

Goals for 2011

Bingo Card Creator

  • Sales of $60k, profits of $40k
  • Getting AdWords back under control
  • Getting the holiday promotions ready for all those domains I own but have never successfully exploited yet

Appointment Reminder

  • 200 paying customers by December of 2011.  This implies the revenue run rate will be somewhere north of $10k then.  Cross my fingers: it might be well north of that, if SEO or white labeling works well.
  • I’m about 70% certain that I’m going to hire a front end developer for AR.

Consulting

  • Perhaps a wee bit more of it.
  • At higher rates.
  • Knock things out of the park for clients #1 ~ #3, and tell stories when possible.

See you all in 2011!

Staging Servers, Source Control & Deploy Workflows, And Other Stuff Nobody Teaches You

I worked for almost three years as a cog in a Japanese megacorporation, and one of the best parts about that experience (perhaps even worth the 70 hour weeks) was that they taught me how to be a professional engineer.  Prior to doing so, my workflow generally involved a whole lot of bubble gum, duct tape, and praying.  I spent a lot of time firefighting broken software as a result, to the detriment of both my customers and myself.  Talking to other software developers has made me realize that I’m not the only person who was never taught that there are options superior to bubblegum.  If you aren’t lucky enough to work at a company that has good engineering in its very DNA, you’re likely to not know much about them.

This strikes me as the industry’s attitude to source control a few years ago, prior to a concerted evangelization movement by people like Joel Spolsky.  It is virtually impossible to overstate how much using source control improves software development.  Our industry has changed in major ways since 2000, but our best practices (and knowledge of those best practices) are lagging a few years behind.  We could really use a Joel Test 2010 edition, for a world where “you should have build scripts for desktop software which can complete the build in one step” is largely an anachronism and where the front page to the website is no longer hand-coded in Notepad but, rather, is a shipping piece of software which can break in two hundred ways.

You’re not going to get the Joel Test 2010 here, mostly because I’m not Joel and there is no particular reason any company should judge its development practices relative to mine.  What I would like to give is some practical pointers for implementing three practices which, if you’re not already doing, will greatly improve the experience of writing software for the web:

  1. Staging servers
  2. Version control workflows
  3. Tested, repeatable deployments

Staging Servers

What is a staging server?  The basic idea is that it is staging = production – users. (If you’re Facebook, Google, or IMVU, you are lightyears ahead of this article and have some system where there are multiple levels of staging/production and where you can dynamically change them.  You already have geniuses working on your infrastructure.  Listen to them.  This article is for people who don’t have any option between “code runs on developer’s laptop” and “code runs in production.”)

Why do we have staging servers?  So that anything that is going to break on production breaks on the staging server first.  For this reason, you want your staging server to be as similar to the production environment as you can possibly make it.  If the production environment processes credit cards, the staging environment processes credit cards.  This means that if, e.g., your configuration for the payment gateway is borked, you’ll find out about that on the staging server prior to pushing it live to production and, whoopsie, not actually being able to get money from people.  If your production server uses Ruby 1.9, your staging server uses 1.9.  If the production server uses memcached on port 12345, the staging server uses memcached on port 12345.

(Many folks have systems which exist on more than one physical machine.  I don’t — I’m a small business where 2 GB of RAM is enough for anything I want to do.  If you have multiple machines, strike “staging server” and read as “staging system” below: all the benefits for having a separate staging server are still beneficial when your staging environment actually has fifteen physical servers running 47 VMs.)

Setting up a staging server should be easy.  If it is not easy, you already have a problem in your infrastructure, you just don’t know it yet: you’ve cobbled together your production server over time, usually by manually SSHing into it and tweaking things without keeping a log of what you have done.  (Been there, done that, got the “I Created A Monster” T-shirt.)  There isn’t a written procedure or automated script for creating it from the bare metal.  If you had that procedure written, you should be able to execute it and create a staging server that works inside of an hour.

Most people won’t be able to do this if they haven’t given thought to the matter before.  That is fixable, and should be fixed.  It has substantial benefits: if you have a repeatable procedure for provisioning a production system, then when disaster strikes you will be able to confidently execute that procedure and end up with a production system.  (Confidence is important since you’ll probably be terrified and rushed when you need to do this, and rushed terrified people make unnecessary mistakes.)

If you’re working on Rails, I highly recommend using Deprec/Capistrano with all new projects.  In addition to making it very easy to get a full Rails stack working on your deployment environment of choice , it helps automate routine deployment and server maintenance, and has mostly sensible defaults.  (I have only one quibble with deprec : it installs software from source rather than using your system’s package manager.  That means that upgrading e.g. Nginx two years down the road is needlessly hard and error prone, when instead you could just have used apt-get in the first place and then updating is a piece of cake.)

You can also use Fabric, Chef, Fog, or a similar system to script up building new environments.  Pick whichever strikes your fancy.  Try to recreate your production environment, down to a T, on another host at your VPS/cloud/etc provider, or on another physical machine if you actually still own machines.  Keep tweaking the script until it produces something which actually matches your production environment.  You now have a procedure for creating a staging server, and as an added bonus it also works for documenting your production environment in a reproducible fashion.

One nice thing about keeping your server configuration in scripts rather than just splayed across fifteen different places on the server (/etc/environment, /etc/crontab, /usr/local/nginx/conf/apps/AppName.conf, etc) is that it lives in source control.  Your cron jobs?  If they’re in source control, you’ll have a written record of what they are, what they’re supposed to do, and why they just blew up when you bork the underlying assumptions eight months down the line.  Your Nginx config?  If it is in source control, you’ll understand why you added that new location setting for static images only.  The voodoo in your postfix config?  A suitably descriptive commit note means you’ll never have to think about reproducing the voodoo again.

After you have the script which will produce your staging environment, you probably want to make a minimum number of alterations from production.  Many companies will want their staging environment to be non-public — that way, customers don’t see code before it is ready, and critical issues never affect the outside world.  There are many ways to do this: ideally, you’d just tweak a setting on your firewall and bam, nobody from the public Internet can get to your staging environment.  However, this is a wee bit difficult to pull off for some of us.  For one, I don’t actually have a hardware firewall (I use iptables on each VPS individually).

My staging environment simply includes a snippet in Nginx which denies access to everyone except a particular host (which I can proxy through).  This breaks integration with a few outside services (e.g. Twilio and Spreedly, which needs callbacks), so I make exceptions for the URLs those two need to access.  The more complicated your staging server configuration gets relative to production, the more likely you are to compromise its utility.  Try to avoid exceptions.

That said, there are a couple that are too valuable to not make.  For example, my staging server has a whitelist of email addresses and phone numbers owned by me.  Through the magic of monkeypatching, attempting to contact anyone else raises an exception.  That sounded a little paranoid until that day when I accidentally created an infinite loop and rang every number in the database a hundred times.  (My cell phone company loves me, but folks who accidentally collided with test data sure would not have.)

How do you get data to populate the staging server?  I use seed scripts and add more data by hand.  (I also have DB dumps available, but they tend to go stale against the current schema distressingly quickly: I recommend seed scripts.)  You can also dump the production DB and load it into the staging DB.  Think long and hard before you do this. For one, it is likely to be way, way the heck out of bounds for regulated industries.  For another, your staging server is probably going to periodically be insecure — insecurity is failure and failure is what the staging server is for.  Slurping all of the data out of a staging environment has caused many companies smarter than you to have to go into disaster management mode.  Please be careful.

So you’ve got a staging server?  Now what?

At the simplest, you access your staging server with a browser and try to break things.  When you break things, you fix things, then you redeploy the staging server and try to break them again.  This is what you are probably doing right now with production, except that your customers don’t have to see broken things when you break things.

Eventually, you can script up attempts to break things, using e.g. Selenium.  Then when you break things, you add them to the list of things that Selenium tries to break.  If you run that against the staging server after every code check in (a process known as continuous integration), you’ll quickly catch regressions before they disrupt paying customers.  This is a wee bit harder than just having a staging server — OK, a lot harder — but you’ll get clear, obvious advantage out of every increment of work you do on this path, so don’t let present inability to be Google prevent you from getting started.

Version Control & Deployment Workflows

Everyone should use version control, but people tend to use version control differently.  Git is very popular in the Rails community, but there are probably no two companies using git the same way.  The key thing is that you agree with your team on how you use version control — document your assumptions, document your processes, then apply them religiously.  This will reduce conflicts on the team, reduce mistakes, and help you get more out of your tools.

There are a million ways to use version control and most of them are perfectly OK.  I’m going to mention mine, but it isn’t the canonical Right Way, it is just one way which works for a (very) small company.  Yours will likely be different, but you can see some of the things which go into design of a version control workflow.

Assumptions I Make About Life, The Universe, And Everything

  1. I use git.  Git has notion of branches, tags, and remotes (physically distinct repositories) — if you don’t know what these are, Google for [getting started with git].
  2. I generally work alone or with a very small team. (This assumption underpins very important parts of my workflow.  It won’t expand very well to a 200 man distributed team, but it might well work for 2 ~ 5 people.)
  3. There is exactly one canonical repository, origin.  Developers maintain other repositories on their workstation.  Automated processes like deployment happen only with reference to the origin.  Code existing outside of the origin does not officially exist yet, for any purpose.  The history preserved in the origin is, in principle, sacred.
  4. There is a branch called deploy.  The HEAD of deploy (the most recent code on it) is presumptively ready to be put into production.
  5. Tags are used to take snapshots of the code base and preserve them in amber with a human readable name.  Right before we deploy to either production or staging, the HEAD gets tagged, so that we can easily find it later, with a simple naming convention (I use production_release_X and staging_release_X, where X just increments upwards — some people might prefer timestamps).  Production release tags are never deleted.  Staging tags get periodically culled when convenient to do so.
  6. Development of any feature expected to take longer than a few hours happens on a feature branch.  (I do occasional work right on deploy locally, for issues of the “Minor copy edit on dashboard.” variety.  This would be one of the first things to go if I were working on a larger team.)

So how does this actually work in practice?  Let’s say I’m implementing a new feature.  I create a new branch to work on.  I code a bit, creating local commits with wild abandon any time I have accomplished something which I don’t want to lose.  When I believe code to be functional, I fire a capistrano task which tags the current head of my branch, pushes that tag to origin, and deploys it to the staging server.  I then continue testing on the staging server, for example verifying that Twilio integration actually works with Twilio (which cannot conveniently access localhost:3000 on my laptop).  I continue writing code, committing, tagging, and pushing to the staging server until the feature is ready.

Then, I switch back to the deploy branch and merge in my feature branch (with –no-ff, which creates a commit message just for the merge — this handily groups the twenty or thirty commits I just made into one easily readable story for myself later).  I then tag a production release (manually — this is entirely to force me to think through whether I’m ready for a production release), verify that there is no diff between it and the most recent staging release, and then push the new tag to origin.  I then fire the Capistrano task which checks out the new deployment tag and restarts the server.

What does this get me versus my previous SVN workflow for Bingo Card Creator, which was “Work only on one branch, commit stuff when I think it is ready, and deploy the trunk manually on occasion”?

  1. I cause much less downtime for the production server due to reasons like svn commit -m ‘Whoops, forgot a setting in production.rb’ and svn commit -m ‘r1234 introduced dependency on foobar gem without putting it in environment file, causing rake gems:install to not load it.  Mongrels then failed to restart.’
  2. My deploy branch has a relatively clean history, so when things start to break next year in production, finding the change sets which eventually caused the breakage will be less of a needle in the haystack search than finding them in SVN is.  SVN’s history is 1800 unedited commits, recording my stream of consciousness as they happened.  My stream of consciousness is frequently stupid, particularly when I’m panicking because the server is down.
  3. This decouples the staging server from production in a clean fashion (so that I can advance the staging server a feature or three at a time if I want to), but guarantees that when I’m actually ready to deploy, I’m deploying exactly what did not break on the staging server.
  4. Tagging releases gives you an Oh Crikey button, as in Oh Crikey, that last release broke stuff.  You can quickly rollback the deploy to a known good tag, isolate the changes which broke production, and fix them.
  5. Deploy scripts manage releases with multiple moving parts a lot better than I do, even when I’m working from a checklist.

By the way, git gives you many options for recovering from problems — even severe problems — without requiring either gymnastics or a full-blown CSI investigation to discover what happened later.  For example, let’s pretend I just deployed tag production_deploy_82, and have discovered some issue serious enough to require an immediate rollback to production_deploy_81, which is known to be good:


#Assuming we are on our local workstation on the deploy branch.

git branch something-in-here-is-broken

git reset --hard production_deploy_81  #All changes made between deploy 81 and 82 just vanished from the deploy branch locally.

#Clean up the deploy, using any option discussed below.

git checkout something-in-here-is-broken  #Those changes which you just disappeared are now living on this branch, ready for you to fix.  After you've fixed and verified it works on the staging server (and, ahem, that you have addressed the issue that allowed this to get OKed for release last time), you merge this branch back into deploy, and do a tag-and-release cycle.

How you clean up the mess on the server is up to you: good options include “deploy 81 again”, “tag a release 83 equivalent to 81, then deploy it”, and “rollback to the copy of 81 which still exists on the server.”  (Capistrano includes deploy:rollback, which will do exactly this.)  Any of these will work, just always do it the same way to avoid stepping on each others’ toes.  I prefer tagging a new release so that I can add a descriptive message explaining why 82 just created an emergency.

This is important because it leaves a paper trail — if you’re pulling a release from production, something just went seriously wrong with your processes.  Emergencies are not supposed to happen – anything that lets an issue get that far isn’t just a one-off failure of whatever broke, it is a series of failures of the systems/processes designed to prevent failures from getting that far.  After you’ve put out the fire, investigate what went wrong and tweak your processes such that a similar failure in the future gets caught prior to bringing down production.  The sleep you save may be your own.

Scaling this to more programmers: Do whatever works for you!  I would probably create a staging branch and have folks integrate stuff into the staging branch when it was ready to go to the official staging environment.  I also might make per-developer staging environments: since creating one from the bare metal is supposed to be essentially free, let them all have their own where they can be reckless without spoiling the experience of other developers.  We can worry about code interaction on the “real” staging server.  Then, have folks communicate when they consider everything they have on staging ready for release, and release when everybody says it is ready.

The important thing is that, whatever process you use, you document it, teach it, and enforce it.

Stuff Your Deployment Script Might Not Do Today But Probably Should

  1. Depending on your scale and how you use e.g. memcached, it might be safe to purge the cache on every re-deploy, which will prevent some hard to diagnose bugs.  At a certain scale, this is virtually a recipe for taking your site down in a cache stampede, but I’m not Facebook and having capacity problems means that I am probably already vacationing at my hollowed-out volcano lair.
  2. Tell everybody on the team that you just deployed.  I know some teams who have an IRC channel with a bot who announces redeploys.  A quick email CCed to five developers also probably suffices.
  3. Restart worker processes.  This is easy to forget but, if you do it by hand, you’ll eventually forget and then have two versions of the application in production at once.  If you’re not prepared for that, it will bite you on the hindquarters when, e.g., the application servers ask the workers to execute methods that the workers do not know now exist in the code base.
  4. Do sanity checks.  You can go arbitrarily deep with complexity here.  For a first cut, mine for Appointment Reminder restarts the application server, counts ten seconds, then tries to access an internal URL.  If the application server isn’t up, or if the action at that URL blows up for any reason, the deployment script fails the deploy, rolls back to a known-good version, and sends me a very crossly worded email.  (You can do this for the staging server, too.)
  5. Integrate with other systems which manage the state of your code/business.  For example, I use Hoptoad.  Hoptoad keeps track of exceptions and mails you when they happen, in such a fashion that your inbox doesn’t get buried by e.g. Googlebot deciding to do an impromptu fuzz test on your website.  I mark all exceptions as resolved every time I deploy to the environment they happened in.  You could also e.g. update an internal wiki by adding a new page specific to the deployment, automatically update your bug tracker to change the status of the bugs that you (presumably) just squashed, or start a new cohort for your stats tracking.

Is There Anything You Would Like To Add?

The trend towards openness with regards to technical practices on the Internet is a major, major win for everybody in our industry.  Best practices do not have to be passed from master to apprentice as oral lore.  Like OSS, since they’re often glue rather than competitive advantages (for many companies — not all), sharing them mostly doesn’t hurt you and can improve the situation of everybody.  So, in that spirit, if you’ve got anything you’d like to add, particularly for how you do things or how you would adapt this advice outside the scope of a very small business, I’d love to hear about it either in the comments or on your own blog (feel free to leave a link if it is relevant.)

Lessons Learned At Business of Software 2010

Author’s note: This post is gigantic — over 10,000 words.  I don’t recommend reading this in one sitting — bookmark it and come back later.
//
Early this October, I was privileged to attend the Business of Software conference as both attendee and a speaker (see disclaimer waaaaay at the bottom of this post).  It was far and away the best conference I’ve ever been to.  You should go to it next year — I know I will, if I have to swim from Japan to Boston to do so.  (If you would like to but don’t think it is possible, see waaaaay at the bottom of this post.)

The featured speakers were, virtually without exception, remarkable.  The 7.5 minute Lightning Talks (one of which I presented — more about that after they post the video, you will be amused or your money back) were often entertaining and at least a few of them were packed with better detail than 15 Powerpoint slides have any right to be.

The phenomenal content of the conference, however, is just an excuse to get a few hundred ferociously smart people in the same hotel for three days.  I got to finally meet in person a few of the folks who either personally assisted me in my early days as a businessman or have been tremendously influential in my thinking — a short list of shoutouts includes Ian Landsman, Dave Collins, Joel Spolsky, Eric Sink, Eric Ries, Peldi from Balsamiq, and I could put another dozen names here without breaking a sweat.  One speaker at one point asked all attendees who had founded a software business to stand at one point, and by my eye probably 60% of the room stood up.  During breakout sessions, I heard from everyone from folks doing small software businesses like myself to a company which sells monitoring software for nuclear power plants.

The spirit of generosity at the conference was off-the-charts amazing.  However, many conversations were either by explicit agreement or common courtesy off the record — not everybody publishes sales stats, and that is their right — so I’m going to err on the side of caution and only repeat things I learned from the public sessions.  That said: beg, borrow, or steal tickets to it next year.  I can’t repeat that enough.  Although the conference swung to business topics far more often than technical topics, almost everyone I talked to both inspired and holding a notebook full of actionable advice to try.

My notes are incomplete and my memory is fallible, but here were some of the takeaways I got from the public presentations.  These are more a combination of my notes and thoughts than transcriptions of what was said.  If you want those, wait for the videos to be released.

Seth Godin

Seth posted a riff on his remarks on his blog.  The general tone and content was not surprising if you’ve followed him for a while.  (If you don’t, do so — aside from being a great communicator, he’s one of the better big-picture thinkers on marketing that I know of.)

Specifically, he says that the software business is undergoing radical changes because technical competence is no longer a scarce commodity: with open source tools, an increasing supply of trained programmers, and the explosion of cost-lowering innovations for creating and marketing software on the Internet, things which were previously the purview of a technical elite are now within what talented teenagers can cook up from their kitchen table.  The marketing side of the equation is also changing, since increased crowding means that the traditional model of paying salesmen to push software has devolved into using humans as inefficient spam bots.  (See his whole spiel about Permission Marketing here.)

A brief guide to evaluating opportunities:

  • Who can I reach with my marketing message?
  • Will they talk to people on my behalf?
  • Can I earn and maintain permission to continue the conversation with them?
  • Will they pay for my solution to their problems?

Perceptive readers will note that technical difficulty appears nowhere above.

Best four words of talk: “Ideas that spread, win.”  Software is now about creating a tribe or otherwise achieving leadership of one.  Take AutoCAD, for example: it isn’t a product, it is a movement: we architects who use AutoCAD are here to change the industry away from using pencil&paper drafting.  You are either in the movement or opposed to it, because it is an existential threat to your business, but you certainly aren’t neutral on the question.  Boring software creates no movements and has to be sold the old fashioned (expensive, ineffective) way.

Another example of movements is the trend towards measuring things, which was a recurring theme during the conference.  Toyota succeeded in changing the face of manufacturing because they got religion about statistics, and a similar sea change is finally happening in software.  This meta-trend is creating a lot of extraordinarily valuable companies directly and also making the rest of us leaner and meaner.  (Bingo Card Creator has a more sophisticated approach to metrics than many Fortune 500s.  That’s a shame, but also a tremendous opportunity for smart, savvy software firms.)

I was quite amused by a throw-away example of the difference between capitalists and socialists inspired by pin-making machines: prior to the industrial revolution, it took a skilled artisan a day to hone 5 pins (small bits of metal which one uses to, e.g., attach fabric to each other) by hand.  A pin-making machine lets any 5 people who can be trusted to learn a simple, repetitive set of instructions make 10,000 pins a day.  Socialists hearing this thought “Oh no, that is going to totally destroy the livelihood of pinmakers.”  Capitalists hearing this thought “Memo to self: be the guy who owns the pin-making machine.”  As the cost of software approaches zero, you want to be the guy who owns the machine.

Relatedly, with regards to your future employment prospects: draw a Venn diagram of job skills you have and skills which can be automated (or, by extension, anything which can be adequately explained by any amount of writing on paper, because anything that can be written down can get shipped to India).  Anything in the intersection represents career suicide.

Seth claims that no “competent” people work at his current software company, Squidoo, because “if we only needed competency, we can get that from [outsourcing]“.  You hire exceptional people and let them execute on things which require exceptional talent, and write down things which only require competency.  Low-wage grunts can do the competent stuff.  ”Your job is a platform, not a list of tasks.”

Some personal thoughts: Seth was bearing on charging for software because of competition from OSS competitors, preferring instead to lead movements where joining was free (to maximize the size of the movement).  Happily for those of us who sell software, almost all OSS projects are incompetent at marketing, which matters more than technical execution.  Seth gave an example of trying to sell water: what are you going to do, make water wetter?  That is a wonderful example, since bottled water didn’t exist as a category 50 years ago and sales are going up every year, over a century after America solved the problem of giving everybody what amounts to free, infinite, drinkable water.  It amused me greatly to hear that bottled water was going away while attending a conference at a luxury hotel which sold $4 bottles of bottled water placed in the restroom literally next to the tap of ever-flowing unmetered municipal goodness.

David Russo

David gave a presentation about hiring at software companies, in particular about hiring to create and sustain a corporate culture as the company approaches scalability.  I regret I was insufficiently attentive during it, as this topic is several years ahead of need for me.

Briefly, there are five models for corporate culture in the literature:

  • Bureaucracy  (the traditional BigCorp model)
  • Autocracy   (Apple-style, where the company is an extension of the Will of Jobs)
  • Engineering (Google, where the engineers run the asylum)
  • Star culture (a large consulting firm or, perhaps, 37Signals, where individual contributors are expected to be stellar and you can put up with rough edges because, hey, rockstars)
  • Commitment — a consensus culture with loyalty receiving strong priority (and, as you can probably guess by position on the list, the clear favored choice of the presentation)

There was an example of Westinghouse having a non-engineering CEO who proved to be disastrous for the company.  Afterwards, the board resolved never to hire a non-engineer again, because the culture of Westinghouse was ferociously engineering-dominated and when engineers talk to each other they break out the secret engineer decoder ring.  The CEO, who couldn’t speak the language or achieve respect among the engineers, was sidelined and failed at achieving his objectives.

Culture generally grows organically out of a business rather than being imposed from the top-down.  Changing a firm’s culture midstream is very painful, and Harvard studies suggest it has quantifiable negative impacts: 50% reduction in odds of an IPO, tripled likelihood of business failure, and 3% less growth monthly.  (The engineer in me winces at the compounding there.)

David’s advice was to invest in employees: hire brilliant people, and let them be brilliant.  The outcomes they achieve are more important than the work itself.

Dharmesh Shah

Dharmesh is CEO (most recently) of Hubspot, a marketing tool for small businesses.  They have raised quite a bit of money and, more importantly, have a lot of paying customers, consistent growth, and a revenue curve which virtually defines “up and to the right”.  You should be reading his blog already.

He presented a few techniques from Hubspot which were “interesting and not obvious”.  For example, successful SaaS companies which IPOed had a median capital raise of $42 million dollars.  It doesn’t match with our expectation that you can make a SaaS startup in your garage, partially because SaaS companies essentially offer their customers upfront financing (by spending their cost of customer acquisition in a front-loaded fashion and then recouping it over the 4-5 year lifespan of that relationship).

A key metric to keep track of: customer churn rate, because a 50% decrease in it doubles customer average lifetime value (LTV).  One of Dharmesh’s periodic hobbyhorses is that if you know LTV and know the cost to aquire a customer (CAC), then if CAC is less than LTV by a fair bit, you take as much investment as humanly possible, spin straw into gold, and walk away with money hats.  (This is far and away the best pitch for a software company taking VC money that I’ve ever heard.  See also the video of him speaking last year, if I remember right he covers it there.)

There are a few ways to measure churn: as percent of customers, as percent of revenue (better accounts churning faster than entry-level accounts: uh oh!), and “discretionary churn”, which is churn among accounts who are actually able to make the decision to cancel and are not, e.g., locked into a one-year contract with you at the moment.  However, overoptimizing on churn rates is myopic, because absence of churn does not indicate the presence of delight, and customer delight is a key to long-term success at marketing the product.

One particular way Hubspot maximizes for customer delight is by creation and use of a Customer Happiness Index (CHI), which is a backtested predictive measure of someone’s likeliness of churning next month.  CHI is dominated by three factors and adjusted as they get more data:

  1. Frequency of use (more use = less churn)
  2. Breadth of use (customers who use more features churn less)
  3. Use of “sticky features” (there exist some features which are so screamingly better than alternatives that a customer who uses and gets value from them is very reluctant to stop using the service)

If I can call out #3 for a moment: many companies I have worked with have “sticky features”, although not all of them know it.  (Some day when I don’t have another few thousand words to write I’ll tell you about what they are for Bingo Card Creator and how I know.)

After you have CHI available in a transparent fashion throughout your organization, you can exploit it in a variety of ways:

  • Tell your sales representatives to call customers with low CHI proactively and ask them what you can do to make them happy.  This simple technique saves 1/3 of canceled accounts, and is virtually free.
  • Judge sales reps not just on sales but on CHI of new accounts.  (This counters the incentive to armtwist to sell to marginal customers.)
  • Judge new features / policies / etc based on their impact to CHI.  Concerned about whether charging for consulting for setup is a good play for the business or not?  A/B test doing it versus not doing it, and optimize for CHI.  (I get the feeling that they might not explicitly be doing A/B testing… but they should be.  Sorry, hobbyhorse of mine.)

All meetings at Hubspot include a teddy bear named Molly, who is a stand-in for the customer (they have two customer archetypes: Margaret the marketing director and Olly the owner, collectively, Molly).  When they have contentious debates internally, they are frequently decided by someone saying “Hey, what would Molly want us to do here?”  Brilliant hack, practically free.

Hubspot sells services, primarily consulting for setting up new clients.  This represents a fraction of their revenue — only 7% or so — and decreases their gross margins, because consulting services are intrinsically lower margin than selling software products.  But “services are low margin… except when they’re not”: since getting people set up for success increases their CHI and wildly decreases churn rate, selling someone a few hundred bucks worth of engineer time at low margins can lead to thousands of dollars of marginal increase in LTV delivered by high margin renewals of the core product (which has 90+% margins).  Star this advice, gentlemen, because it has obvious implications for B2B SAAS startups.

So if you know customers and the company benefit hugely by increasing consumption of consulting services, why charge for them?  Answer: perceived value increases consumption, and perceived value is driven by cost.  When you give people 5 hours of free telephone support with an engineer, they — being successful businesspeople — find difficulty clearing 5 hours free in their schedule, fail to show up for the call, get distracted, etc.  When you charge them $500 for the call, they damn well show up.  (Patrick notes: why hello, recent discussion about Chargify free customers and support costs.)

In terms of explaining the value proposition of your software to customers, cribbing from Kathy Sierra: don’t make _____ software, make _____ superstars.  Make your customers awesome at their jobs / tasks, and they will pay any price and/or follow you to the gates of Hell.

Hubspot is big about being transparent to a fault within the company, particularly by using a company-wide Wiki which includes financial details, plans, and pretty much everything except salary information.  (Apparently, that being public is viewed as possibly toxic to morale without having obvious upsides.  Having financial details open to employee audit has obvious upsides: employees can tell you when you’ve let the entrepreneur reality distortion field go wild in the board meeting.)

Best sentence about branding I’ve heard recently: “Brand is what people say about you after you have left the room.”

Eric Ries

I will refrain from giving an in-depth summary of this talk because if you have read his blog you pretty much know the gist of it before.  If you haven’t read his blog, start doing so.  With the obligatory disclaimer about letting any methodology take over your higher brain processes, the Lean Startup movement (and some of the specific, implementable tactics it suggests) are so effective I sometimes worry about whether established competitors will try to illegalize them.  The best effect of this presentation was taking the idea away from the Silicon Valley echo chamber (where “pivot” has become something of a buzzword) and bringing it to an audience of people who largely sell things to people for money, but who still could benefit immensely from some of the Lean Startup lessons.

If you want an hour-long video introduction by Eric to the Lean Startup, you can see parts 1 and 2 of a previous explanation on the Internet.

Some specific lessons I haven’t heard before in a couple years of lurking on his blog or I thought were worth highlighting:

  • The key takeaway: Lean Startup matters because the traditional model of development wastes people’s lives.  They pour out years of hard-charging startup workweeks making software which doesn’t actually improve the lives of any customer because it solves a problem no one has.  This waste of potential is criminal and can be fixed.
  • Machine shops and software firms are self-regulating ecosystems if you do them right.  For example, if you use continuous deployment and five whys, when you start doing development too fast, you’ll get the brakes applied by having to make investments in quality control repeatedly because you’re making too many mistakes.  This lets you find the optimal pace of learning without having to fiat an inaccurate number (“You will write 10,000 lines of code a week!” — which as we all know means nothing, and definitely nothing good) from on high.
  • The Agile methodology is a wonderful solution to the peculiar problems of internal IT departments at big companies, where the problem is extraordinarily well-known (“We need our existing payroll system, except on a new architecture where extending it doesn’t require sacrificing goats”) and solutions are unknown (“What is the Rails code to do this?”).  It has less to offer startups, where both the problem and solution are unknown.  (“Do people even need the thing we’re fumbling forward on building right now?”)
  • Since the only people you can plausibly get to use a new product are early adopters, any additional quantum of work done to get the product ready for mainstream users is a form of waste.  (This is a bold statement with interesting implications.  I’m not sure I agree with it, but it sounds potentially very powerful.)

Scott Farquhar

The president of Atlassian had some thoughts gained from bootstrapping the company from 2 people to a $60 million capital raise and beyond.  Regrettably for my notes, this talk was structured as a List of N Things, which ensured that I wrote down N things and grabbed few supporting details.

  1. Start with 2 founders.  For example, when their password database was compromised while Scott was off on his honeymoon in the wilds of Africa, knowing that his cofounder had the situation under control alleviated a portion of the anxiety, and after an emergency flight back to Australia Scott was able to let the cofounder get some sleep.
  2. You need a business model.
  3. Dogfood your products.
  4. Measure everything.
  5. Always be marketing.  Atlassian excels at this — for example, they had a charity give-away where they gave licenses of JIRA (a bug tracker) away for a song to small companies ($5 for 5 licenses or something to that effect), donated the profits to charity, and got massive good press from the deal while also increasing sales in the market segments which drive most of their actual revenue.  They also had very appealing marketing of another few initiatives: calling something the Stimulus Package, for example, was an excellent PR hook in the financial crisis environment.  I wish I had better notes here.  Memo to speakers: please do not autocommoditize your best points by numbering them.
  6. Your first idea will fail.  Atlassian was originally going to market plugins for an obscure product from a company in Europe.  It turns out the bug tracker they wrote to make that product was more valuable than the product itself — shades of Fog Creek and CityDesk here.  (We heard this theme several times during the conference: Peldi later said that Mockups was originally intended strictly as a plugin for another software, and the standalone version which it grew into now dominates Balsamiq’s revenues.)
  7. Long term thinking is good.
  8. Know when to switch gears.
  9. Build somewhere you want to work.  Many precious anecdotes here about, e.g., organizing a city-wide scavenger hunt for employees.
  10. Experiences are more effective at motivating employees than and customers than money.  This comports with both social science research I’ve read and my own experience.  (I have recent experience of this at a company I consulted for, but since it is identifying, I’ll wait until they announce what we did together to go into it.  Still: it is amazing how effective turning money into an experience is at motivating people than awarding them the money directly, even though this is crazily irrational.  Japanese companies understand this better than anyone: “why pay people so that they can buy objects suggesting social status when you can just award social status directly”, which is the Rosetta Stone for so much of corporate life here it is scary.)

Jason Cohen

I was flagging by this point in the conference and a little terrified about my upcoming speech, so my notes aren’t that great.  Jason had a lot of amusing anecdotes from taking Smart Bear code review software from a one-man operation in a small room in Texas through to a sale to another company.  (Despite being told, repeatedly, that “Smart Bear” was unprofessional and the name would scare off big Fortune 500s, the company which bought him out later reorganized their portfolio of products under that brand name.)

One amusing story was about Sales Guru Frank.  He was a silver-haired slick talker who, for a mere 50% of the equity of the company, promised to sell Smart Bear software to the sorts of huge corporate clients which would bring it from a six figure business to a smashing success.  Jason had an absolutely priceless anecdote about how us engineers perceive the sales process: “You know, it’s when you’re in the boardroom drinking some expensive brand of alcohol and you start the conversation with ‘Hey, did you see the game?’ and everybody understands what game you meant and then, suddenly, sales happen.”  It turns out that sales is a little more complicated than this (for details, see the Paul Kenny presentation) and, thankfully, Jason avoided getting suckered by Frank.

Similar fun stories abounded but I was too terrified of my upcoming presentation to write them down (d’oh).

The single best takeaway was how to evaluate advice you receive from noted software bloggers.

There are two competing motivations for people who start software companies: wanting to maximize one’s financial outcome as quickly as possible (you want to be Rich) and wanting to sculpt the perfect personal niche in which you will be respected and enjoy your day-to-day work (you want to be King).  There are two fairly well-known markets for software: B2B (business to business) and B2C (business to customer).

This gives us the traditional four quadrant graph, which Jason decorated with some examples:

The take-away: only listen to advice given from people in the same quadrant as yourself, and you’ll save yourself a lot of pain and cognitive dissonance from getting conflicting advice.  (This is overstating the conclusion a wee bit I think, and people change on both axes over time.  That said, I’m definitely onboard with “all advice is a product of the circumstances faced by the person giving it”, but I think you can pull lessons from 37Signals in a B2C business or from Joel Spolsky regardless of whether you want to craft the ideal place to work or go big with a VC company or both.)

Sidenote: my business got mentioned from the stage three times, and I’m seriously honored and humbled about being put in that company.

Paul Kenny

Paul talked about the sales function — not just the sales job title — at software companies.  See his talk from last year for details about that.

Why do so many salespeople suck?  Well, the typical career path requires:

  • One year from hiring to get up and running with the particular company/market segment/product.
  • One year to achieve momentum in building up one’s list of contacts and starting the sales cycle.
  • One year to demonstrably fail to achieve results.
  • One year for HR to convert the salesman from non-performing to “no longer with the firm.”

Given that it takes four full years to go through this cycle, you could see a salesman with an impressive resume of 3 ~ 5 year stints at name brand companies with impressive sales to their credit… and not learn, from their resume, that their sales skills are atrocious and their successes were actually the result of products which could be sold solely on the strength of their quality/marketing/brand even with the salesman impeding customer perception of all three.  For example, Jason’s Frank probably fit into this category.

Paul nonetheless believes that strictly depending on inbound marketing restricts your business to plucking the low-hanging fruit, and that going solely after low-hanging fruit is not a great business strategy.  (This would be an excellent opportunity to whip out the above quadrant map, by the way: if you’re in either of the B2C quadrants, congratulations, low-hanging fruit is your business strategy, and you can be quite successful with it.  Ditto 37Signals and many similar cheaper B2B SAAS apps, by the way.)

The point of speaking to customers, which is all sales is, is to reach a mature understanding of what your client values the most.  Then, and only then, you offer that to them.

Engineers routinely suck at doing demos for customers because they demo every bit of the interface, from start to finish, going through all the little knobby settings bits.  Nobody cares about these things.  Demo the bits of the software your customers find interesting.  When you email customers, talk about the bits of the software your customers find interesting.  When you blog about the software, blog about the bits of the software your customers find interesting.  But above all things recognize that your customers find themselves interesting and the software boring unless the software makes a demonstrable improvement in their lives.

Quick tip when speaking to customers: “The most interesting people you will ever meet are the people who are most interested in you.”  The average sales rep spends about 47 seconds in talking about customers prior to talking about the solution.  That number should be increased, greatly.  On the plus side, because the industry is so comprehensively screwed up here, even minor improvements provide an opportunity to surprise and delight customers.  ”Goodness, the CEO of that company spent five whole minutes talking to me about how we go about our business!  They really care!

(Sidenote: This also goes if you’re selling yourself as an employee to a company.  If you’re in an interview and talking about the employer, you are probably going to get hired.  For reasons beyond my ken they don’t teach this lesson in college.)

Key takeaway: The quality of your dialogue with customers is directly proportional to the quality of the customers you will acquire.  If you understand their needs better, you will close bigger deals with happier customers who consume less resources.

Customer faith in your product as a solution to their problem is directly proportional to how well they believe that you understand their problem.  This again counsels spending more time talking to them and asking perceptive questions, then repeating their own language right back at them.  If they call it a foo, you call it a foo, even if internally everyone knows it is “really” a bar and, after all, it implements IBarable in the source code.  Relatedly, you cannot tell your way into a dialogue with the customer, you can only ask your way in.

Customers have DNA: Drivers, Needs, and Aspirations.  You should be capturing your understanding of these as you talk to customers, or you aren’t learning what you need to learn to bring the customer and the firm to a mutually satisfactory relationship.

Some factors to consider

  • Customer needs
  • Timescale for implementation
  • Scalability
  • Integration with existing systems/processes
  • Affordability
  • Results

Customers have many priorities:

  • Ego (underrated by engineers in my opinion… even those who own iPhones because they’re worth owning iPhones)
  • Perceived gain
  • Sense of belonging (“Nobody ever got fired for…”)
  • Security
  • Ease of use

If you think of a funnel of concerns prior to sale, starting at the top:

  • The experiences customers have right now.
  • Their business case or project which may benefit from your solution.
  • The utility your solution can offer.  (Note to engineers: many of you stop inquiring here.  That is a mistake.)
  • Options they have competing with your solution.
  • What the company values in terms of outcomes, drivers, etc.
  • Resources they have to solve the problem (i.e. talk budget last, not first)

Founders have an advantage in doing sales, even if they’re engineers who think they can’t do sales.  Nobody understands the problem domain like a founder, nobody understands the solution’s capabilities better than a founder, and nobody else can say “You can trust me on this because I build the bloody company” or inspire the same degree of emotional response in a customer that talking to a founder provides.  (This is so true in my personal experience.  I hear, over and over again, from my customers that they trust me because they know they get an answer “straight from the top” when they send me an email.  I hear that most often from people who never emailed me in the first place, or who emailed me with issues that could be easily dealt with by Level 1 CS.  This is one compelling reason why I don’t outsource customer service.)

Rob Walling

Rob writes Software By Rob, runs a stable of small software businesses as a one-man show (including, most amusingly to me, selling beach towels — he focuses just on software these days), and wrote a book I really enjoyed.  (Flip to the chapter about Virtual Assistants, it is worth the price of the book and then some.)  The slides and outline for his talk are online.

After review of statistics generously donated by several software companies, Rob is of the mind that the most important goal of your website is to bring people back for additional visits, because the conversion rate of your website is much, much higher among engaged visitors than it is among first-time visitors.  I didn’t write down the exact numbers, but I recall him citing DotNet Invoice (“written in Ruby on Rails… no, just kidding”), CrazyEgg, and the beach towel site among a few others.  Interestingly, a few of the examples had higher average order values for returning visitors, too.

I only investigated this for about 30 seconds, but I have lower conversion rates to the online free trial of my software from returning visitors than first time visitors.  That actually isn’t contrary to his thesis, though — it is an artifact of having a very scalable SEO strategy and customer acquisition highly dependent on high-converting organic and AdWords landing pages.  This is one of the threats of making decisions based on shallow understanding of what Google Analytics tells you is the “truth” of your business.  (One minor surprise: about 60% of my customers who convert from the free trial convert within 3 hours of starting it.  Nail that first run experience and nail it hard, guys.)

“The ineffective marketer asks you to buy too soon.”  A customer on their first visit could have multiple reasons why they are not currently prepared to buy your software:

  1. They don’t have enough info about it.
  2. They don’t trust you.  (Oh goodness, this topic is worth a book.)
  3. They don’t have money to buy it right now.  (A particular issue for a $300 invoicing software.)
  4. The don’t need the software right now.
  5. They’re never going to buy this.

You can recover from all the issues but #5 if you establish a relationship with them.  Permission-based email marketing is the best way to do this, bringing them back to your website via a communication channel everyone has and uses.  Repeated, trust-building contact with them will gradually wear down their resistance to parting with their cold-hard cash.

Email is far superior to competing methods of repeatable contact with customers — such as Twitter, blogs, RSS feeds, or your favorite social network — because it allows “personalized broadcasts”, where you can scalably communicate with an arbitrarily high number of people while also making it feel like everyone is getting individualized attention.  (Not nearly enough people use this to its maximum potential, particularly with SaaS.  Free tip from me: note what search term sent them to your website, put a variation of it in the subject line of the email, watch your open rate go through the freaking roof.)

To get the maximum value out of emailing your customers:

  1. Have a killer landing page laser targeted on the “give us your email in return for this immediate benefit to you” call to action.  Give them a whitepaper or similar resource in return for the email.  It greatly, greatly will increase your conversion rate.  (Rob is not a fan of free-to-use trials.  That is an interesting idea to me, as I come from a “of course the software is free to try out” background.  He has achieved truly eyepopping conversion rates to an email submission — 40%+ on some pages.  I can’t get that to my free trial, which also requires an email address, and I have pages which are very, very compelling at “selling” that to customers who were looking for exactly what the free trial offers.)
  2. Give something away in return for the email address.  See the slides for specifics, but this has a monumental ROI.  (I have some confidential numbers from consulting clients.  Suffice it to say this is right on the money.)  Interestingly, if you just call a white paper a “report” it has higher perceived value.  (News you can use!)
  3. Set follow-up for emails.  MailChimp, my mail provider of choice, calls this an autoresponder sequence.  If you take one thing from this presentation, learn and use autoresponders.  They will change your email marketing life.

Some excellent tactical suggestions:

  • Spam: Don’t spam!  Don’t mistakenly get classified as spam by hitting filters.  You can pre-test your emails against common filters, like SpamAssassin, to reword against accidental false positives.  (A long time ago I worked on anti-spam systems for a living.  Oh, the hilarious stories I could tell you.  Suffice it to say that a home builder might want to avoid promising that they could “increase the size [of your patio]“.)  Incidentally, MailChimp will do this for you for a nominal per-email fee.
  • Some words kill open rates if you use them in subject lines.  Reminder, Special, % off, and “Help [us]…” are all culprits.
  • From Name: your own personal name is best, and role names are a bit worse, company name is a bit worse, and webmaster@example.com is worst of all.
  • All advice on emails can be A/B tested!

Peldi (Giacomo Guilizzoni)

Peldi runs Balsamiq, a small software company which has become practically synonymous with low-fidelity mockups for software.  He wrote the best blog post on the Internet about seeking coverage for your startup.  You should execute on it.

The theme of the talk was worries: worries you shouldn’t have, worries you will have, and what keeps Peldi up at night these days.

Things you should not be worried about:

  • Asking customers to pay you money.  (Both Peldi and I were shy about this prior to starting, apparently, because we had the genetic flaw all engineers have that says that there might be a bug and, therefore, it is too early to accept appreciable amounts of money for the product.  I promise to do my part and say “Charge more” to any person that will listen to me until we have cleansed this from our collective gene pool.)
  • Pirates.  (Not an issue in practice, you can pretty much fire-and-forget a low-pain DRM system, and since if you’re smart you’re doing web apps anyway this is disappearing from the radar screen entirely.)
  • Picking a niche which is too small.  (Peldi put up a screenshot of Bingo Card Creator here, to the widespread amusement of the crowd.  I later heard more than one attendee say that they were under the impression it was “a funny Italian joke” until I gave my lightning talk.  That actually worked out great for my talk’s reception.)

Early worries in the lifecycle of his company:

  • When to make the jump from full-time employment to starting your own business.  If I recall the talk correctly, Peldi scrimped and saved for a while, sold his shares of Adobe, and emptied his IRA so that he would have a year of runway in hand prior to going full-time on Balsamiq.  He also had written code on nights and weekends for several months, so he was not doing a standing start.  (And his rise was truly meteoric after that — a combination of product quality and being a marketing genius.  If you don’t understand why he is a marketing genius, go back and read that post I cited earlier.)
  • Work/life balance.  Regrettably, you’re going to have to get your family on board with losing you for a few months while you disappear into the Bat Cave and hammer out your software.  (This is my one bone to pick: no, you do not.  You can virtually always stretch out your schedule.)  Peldi then delivered the best line of the conference: “If you work while they sleep, they won’t know that you’re ignoring them.”

Many people are under the impression that Balsamiq was an instant, overnight success, largely based on Peldi’s embrace of radical transparency and publishing revenue numbers.  Peldi had a series of very interesting graphs: the first shows the hockey stick revenue curve that competitors salivate over.  Then it zooms out to include the time while the software was getting written (i.e. no sales).  Then it zooms out to include his previous career at Macromedia/Adobe where he learned to polish software to perfection in the way that separates Balsamiq from its many, many competitors.  Then it zooms out to include his programming career starting at age 12.  Overnight success only took thirty years.  In a conference with many profound insights, I think this was one of the best ones.

Let me give you an example from my own experience: I recently received a wire transfer from a consulting client, for an amount of money which would be unexceptional to some people and which is staggering for me — more than I’ve ever had accessible in my entire life, by a factor of lots.  It was for two weeks of work and was roughly equivalent to what I used to make for six months at my previous day job.  The story for how I got to do those two weeks of work starts out with participation on the Joel on Software discussion boards some four and a half years ago.  I started dabbling in the topic the work covered almost six years ago.  The focus on communication which made it possible to sell the client (and eventually, their team) on the importance of the work getting done can be traced back probably to middle school (when I took up public speaking to get over a speech impediment).  You literally needed an entire lifetime to prepare for what you’re doing today.

Back to Peldi: don’t be afraid of an inability to deliver.  You’re a talented engineer, you’ll figure it out.  If you fall in love with the problem and can discuss it with boundless energy, then every interaction with customers will make your product better.  At early stages in the product lifecycle, feedback from customers is a much better thing to capture than mere money.  (Peldi gave away licenses like they were candy during the first few years.  Again, read the above blog post, in addition to getting great feedback this was a great PR and linkbuilding hook.)

Peldi had an extended discussion of how you can use Obama’s debate script as a model for addressing your fear of confrontation with customers.  While I’m not a fan of his politics, I agree, he is an excellent communicator.  There was a flow-chart involved, but the core is:

  • Thank the customer for taking time out of their busy life to communicate with you.
  • Empathize with them.
  • Apologize if they’re ticked off, even if they have no right to be.  (Oh, crikey, how much do I agree with this.)

Lightning Talks

All of the Lightning Talks were delivered by non-professional speakers.  They followed a murderously difficult format: you get 15 slides which get auto-advanced every thirty seconds, and that’s it.  You might think that means you can get away with slapping together something in an hour: oh no.  In discussions with my fellow lightning talkers, we agreed that there has been something of a “lightning talk arms race”: the two talks I was most impressed by took over 24 hours to prepare, and mine took at least twelve solid hours over two months, with probably half of that being rehearsal until I could literally count out thirty seconds with a prepared spiel delivered in a voice other than my own.

I’m going to refrain from talking about my own lightning talk.  Why?  First, it relies on a “reveal” for a great deal of the impact.  Second, it was, far and away, the best speech I’ve ever given (out of thousands — some people play sports, I did public speaking).   The combination of heartstopping terror and the crowd reaction produced a strange alchemy which made the talk as delivered a lot better than the videos I made of myself performing it in a room for my webcam.

I’m told it was recorded and the video will be made available to the public.  I’ll tell you when it is ready.  I promise you, you’ll enjoy that a lot more than you’ll enjoy me posting the slides and a video of me saying virtually the same words to a wall.

As to the lightning talks other than mine, a combination of heartstopping terror about my talk and the exigencies of the format mean that my notes on them are woefully inadequate.  I expect that a few of the lightning talks will eventually be put online at some point.  Those videos will do them better justice than my hazy recollections, so I recommend waiting and seeing them.

Eric Sink

Eric presented the nuts and bolts of what happens if your company (or, in his case, a product thereof) gets acquired by BigCo (in this case, Microsoft).  Short version: a lot of pain and suffering, followed by a check so large that the VP of his community bank said putting it in his account would throw their company into absolute havoc because of the impossibility of putting so much money to work.  Let me emphasize the pain and suffering part, though.  (The low point: when the deal had consumed his waking life so much that Eric was reduced to taking a phone call from his lawyer while in church.)

One of the weirder points for Eric was that, with a cast of literally dozens working the Microsoft side of the deal, he never met or learned the identity of the person who actually made the decision to buy the product.  In a gang of people with confusingly similar job titles and descriptions, it seemed that “nobody was in charge.”  The deal was lead by a particular point of contact — we’ll call him Fred — who, as a trained negotiator whose full-time job is assimilating biological and technological distinctiveness, was a thousand times more prepared than any entrepreneur will ever be.

The process involves a long rigmarole of procedural steps.  You’ll want to see the full talk for the breakdown, because I’m hazy on it, but it involved multiple levels of due diligence, getting signed agreements covering IP rights from every person who had ever put a finger to a keyboard within a meter of the source code (cough start work on this today if you want to sell cough), etc etc.

While many entrepreneurs obsess about The Number, what you actually get is The Deal, which ended up being 150 pages of bullet points all individually negotiated covering every aspect of the transfer.  This included guarantees of treatment for employees who were relocating with their product from central Illinois to Microsoft, all manner of indemnifications, tax treatment of the deal, escrow (code and otherwise), the exact structure of compensation for the deal, the terms of the NDA covering material terms of the deal, etc etc.

The deal nearly fell through at several points, over contentious negotiating issues.  Nearly might not be strong enough: it was dead.  The deal was off, and the other side went to total radio silence for a period of weeks.  This was necessary to get the best achievable outcome for Eric and the team.  Eric is of the opinion that you need to walk away about twice: less and you’re getting screwed, more and you’re buying more heart attacks than the marginal improvement in terms is worth.  (One friend of his — in the elite fraternity of “people who have sold a company”, whose stories “you only become privy to after you have joined the club” — had to walk away five times.)

One interesting point raised in discussion was the importance of having multiple interested buyers so that you can play them against each other.  (Michael Pryor from Fog Creek is especially articulate about this one.  The short version: if your BATNA is “get bought by the other guy”, you have more leverage than if your BATNA is “you don’t get acquired, six months of work flushes down the drain, and the company potentially withers.”  The long version: ask Michael.)

Dan Bricklin

So here is a generation gap question for you: I didn’t know who Dan Bricklin was, and someone said “Well, duh, he’s the guy who wrote VisiCalc.”  I then had to Wikipedia VisiCalc because I had never heard of it.  For those of us who were born in the eighties or later: he invented spreadsheet software.  Yowza.

In addition to a hundred and one precious anecdotes about inventing spreadsheet software while a poor grad student (my favorite: the development machine and sole source repository, which cost the team’s life savings, being protected from a plumbing problem by constant vigilance with mops), there were a couple of generally useful takeaways:

  • Customers have jobs that they need to get done.  Identify those jobs.  Build general purpose software which allows them to get the jobs done.
  • The perceived value of general purpose software, which solves a variety of current and anticipated needs of a customer, is higher than that of software which solves one specific pain point.  (I respectfully disagree, but then again, I didn’t invent the spreadsheet.)
  • Software which the customer can customize to their particular needs allows “tailoring at the ends.”
  • The two-week payback rule: software sells itself if you can reliably demonstrate that your product will recoup the purchase price within two weeks.  VisiCalc (spreadsheet on a mini-computer) was so disruptively better than using a time-sharing system to do calculations that you could buy the software and the system to run it for just half of one month’s metered use of the big mainframe you leased time on to run your TPS reports.  That contributed to it selling 700,000 copies, back when distribution channels for software were stone age compared to the alternatives we have today.  (My unconscious brain was virtually screaming: “You can sell software without a website?!” when I saw the magazine ad slides.)
  • Games and relative cheapness were very important in driving new technology adoption.  Note that this, and the “general purpose applications win” rule, are demonstrated in abundance by the runaway success of the iPhone.

Derek Sivers

Derek had originally planned on delivering a talk about various profit models for software companies (spiritually similar to this talk).  I met him for the first time at a speaker’s dinner.  During the course of dinner, one topic frequently returned to was why we do what we do.  Derek told a very moving story about how he sold CDBaby, his labor of love for many years, and someone (I think it was Rob Walling) suggested that he do that as a talk instead.  He ended up doing so.

Where to start:

Derek originally got into CDBaby to sell CDs of his own band’s songs.  Back at the dawn of the Internet, getting a merchant account for selling CDs was a multi-month process, with plenty of administrative hassle and pain involved.  Derek persevered and, with no programming background, managed to hack together enough CGI scripts to get a buy now button on his website in a mere three months.  (And again I bless my lucky stars that I had Paypal and e-junkie.)

After he started selling CDs, some of his friends with bands said “Hey man, that’s really cool.  Can you sell my CDs, too?”  So he’d hack the scripts again and then he was selling another CD.  Eventually, word of mouth had gotten to the point where his band’s website was colonized by other bands who lacked a good way to sell their CDs online, and CDBaby was born.  His blog covers this story in detail.

Derek was once interviewed by NPR and, when asked about his exit plan, said that long after the sun had set on CDs and when he was old and grey, he’d still be shipping CDs to the last aging person who swore that the CDs they played when he was young were far superior to that newfangled hologram 7D garbage that passed for music in 2050.

This turned out not to be the case: after a major rewrite of the CDBaby source code, where Derek achieved pretty much every technical goal he had for the site (and had paid back years of code debt), and after changing his management style such that he became totally superfluous in day-to-day operation of the company, Derek became an absentee owner of the company he founded.  Some people might view that as a wonderful outcome.  It turns out that Derek’s employees viscerally hated him for it: convinced that they were the true bearers of the CDBaby vision, rather than the unseen owner who many of them had never even met, they began organizing what could be described as a corporate coup.

When Derek found out about this (oh, the joys of reviewing MP3 recordings of meetings at midnight), he realized that he was well past the point of no return with regards to employee loyalty.  So he turned Apache off and started composing an emailed pink slip for the entire staff.  No, really.  Luckily for all concerned, he eventually decided to sleep on it, and came back to advice he had received from Seth Godin: if you are no longer passionate about the problem, you owe it to your customers to sell to someone who is.  Apache went back on and the email was not sent.

Derek pursued a few options for getting rid of the company.  One was to simply give it away: he actually pursued pulling off a Willy Wonka and putting five golden tickets in CDs delivered by his company, with the five holders invited to present their visions for the future growth of the company, and the winner (as decided by a vote of customer musicians) receiving the entire company.  Due to fortunate advice from friends regarding the desirability of being sixty and having no money, Derek threw away the golden tickets he had gotten printed and decided to do the more usual thing and just sell.

The sale came with a wrinkle: Derek first transferred the company into a charitable trust, then sold it, meaning that he is forced to draw 5% of the sale price per year, and when he passes away the principal and investment gains (if any) of the trust will be distributed to the charitable causes of his choice.  Derek claims this is mostly not for altruistic purposes: he asked his lawyer how to achieve “I want to have a stable income until I die, and that is all I need”, and the charitable trust was an attractive option.

Derek mentioned that having competing bidders was wonderful, since they could be played against each other.  This helped him achieve some goals he had in selling the company:

  • I’m out.  (Founder participation as a consultant or employee for a defined term is a frequent requirement of sales.  Derek took that option off the table entirely as a precondition of sale.)
  • I keep my database.  (A decade of running essentially a marketplace between music sellers and music buyers gives Derek a powerful foundation for future endeavors aimed at helping musicians, his perennial concern.)
  • I keep helping musicians.  (The non-compete was narrowly tailored so that he would not sell CDs again, but could continue participating in the music industry.  Sometimes non-competes are are very broadly drafted — I’ve heard possibly apocryphal reports from one that, boiled to its essence, barred a technical founder from programming for the duration that a large company with a small-sounding name continued selling operating systems.)

Joel Spolsky

I’m pretty sure most people know Joel co-founded Fog Creek and Stack Overflow.

Continuing the theme of the later part of the conference about big picture questions like “Why?”, Joel’s presentation was probably the bravest speech I’ve ever heard.  Theoretically, it was about the story of transitioning from a low-growth (ish) software business like Fog Creek to  a high-growth VC-funded shoot-the-moon attempt like StackOverflow.  However, it dwelled on some intensely personal questions about leadership, philosophy, and intra-team conflict, and I don’t know if I could have discussed them at that level in public.

Let’s see: the brief takeaway on seeking investment is that if you’ve got a successful track record as a well-known CEO and a good idea which also has demonstrable traction, raising capital is fairly easy (Joel and company were practically booked solid when they went to Silicon Valley to meet with VCs).  However, it still has some of the industry seediness.  One VC who they met with, who ended up not investing in the company despite early promise, later tried his darndest to cause a rift between Joel and his co-founder (Jeff Atwood) so that he could pick up the pieces.  Another VC was quite interested in StackOverflow, provided it was CEOed by anyone other than Joel.

Interference from VCs aside, they eventually got signed by Union Square Ventures, who have treated Joel and company fairly.  The cultural gap between a self-funded software company and the VC world has caused a few disconnects (for example, at one point they found that StackOverflow had $200,000 lying around that they had forgotten to disclose based on revenue, and they asked what corporate structure should receive it or whether they could licitly award it to employees: the VCs were totally uninterested in $200,000, and rather amused at the notion that a company they had funded could actually have had a profit from operations at the stage they were at).

StackOverflow has a very different corporate culture than Fog Creek, which Joel describes as a software company founded with the goal of becoming an “intentional community” whose mission in life was providing programmers a great place to work.  It was explicitly analogized to a kibbutz, with collective ownership, the profit being redistributed among employees, and decisions specifically made to promote community (one which stuck with me was the huge importance placed on breaking bread together on a daily basis).

The culture involves decision-making by consensus.  For a variety of reasons, StackOverflow does not operate in this fashion.  This lead to some contentious battles between Joel and Jeff, exacerbated by the aforementioned evil VC, who tried to split the founders to pick up the company on the cheap.  Among other things, After frequent screaming and crying, Joel figured that his traditional style of hiring smart people and letting them get things done wasn’t working, and he called in a CEO coach for assistance.  He thought he’d get management advice on how to better run meetings and write org charts and do the things that CEOs do.

Instead, he got corporate psychoanalysis, and was introduced to the notion that the CEO’s job is not to manage the company, but to lead it.  This was particularly important as he was CEOing two companies simultaneously.  Once he started accepting that “I am the CEO, I have heard your viewpoints, we do it my way and that is final” was also a valid managerial style, conflict in the team ironically decreased greatly.

On the question of why we do what we do: Fog Creek is the company Joel designed to have a great place to work.  StackOverflow, on the other hand, is targeted at contributing one significant thing to the world: fixing the broken style of asking expert questions on the Internet.  They want to bring the improvement StackOverflow brought to programmers to the wider community (and communities) of interest who Google questions which have canonical good answers and find a SERP stuffed with 10 year old VBulletin threads where the answer is on page 7 in between a cat photo, a flame war, and the announcement of the forum’s guru’s goddaughter’s wedding.

Joel would prefer to work at Fog Creek, but thinks he has an ethical responsibility to make StackOverflow succeed first, because — irrespective of Fog Creek maximizing his personal enjoyment — StackOverflow has the potential to contribute a large social good to the world at large, and he views this as ethically mandatory given the opportunity.  He contrasts this to the 37Signals-esque notion of being content to run a small business because it allows you financial success and the lifestyle you enjoy (although he was quick to note that, while saying this, 37Signals gave the world Rails and teaching, so their own ethical bases are covered).

For an example, see CraigsList, which has moved a huge amount of wealth from newspapers to the posters of classified advertisements by making classifieds free.  CraigsList is run with a goal other than profit, and would instantly be a gigantic company if they charged for ads in categories other than the ones they do now (which they do as a spam-control device).  Joel argues that Craig should do this, because the massive wealth transfer his business enabled has redistributed the producer surplus from newspapers (which once used it for a socially beneficial purpose: underwriting unprofitable but socially beneficial local investigative journalism) to a consumer surplus for “people selling couches, landlords, and pimps.”  His point was that, when we know the trade-off our businesses are making, we have an obligation to choose the results wisely.

I am perhaps not doing this point justice: the two minute thumbnail sketch of the argument I heard Joel deliver profoundly altered my perception of where my company fits into the larger tapestry of my life.  I would previously have characterized, e.g., the decision to keep the company small as being essentially a personal aesthetic choice with no moral consequences, akin to picking colors for one’s bedroom.  I will be digesting the implications of this for a while, but I think I am now convinced that that is probably untrue.

A sidenote: since, pace Seth Godin, the cost of producing software is cratering and the competition is exploding, Joel is gradually coming to the belief that the value in a software company isn’t the code, it is the community.  The thing StackOverflow had to offer VCs wasn’t the code (which, to quote what has become an in-joke among HN readers, “could be written in a weekend”), it was the “large number of engaged users.”  Microsoft estimates there are 9 million programmers in the world.  StackOverflow has upwards of 8 million monthly uniques.  They’ve become the canonical go-to place for answers in the programming niche, and have dreams of becoming the canonical go-to place for answers for all expert queries which have canonical right answers, from tax advice to math theorems to gardening and many points beyond.

Commitment To Community & Disclaimer

(Back to top.)

Like I mentioned earlier, I strongly recommend you attend Business of Software.  Even at close to 10,000 words, this post captures a bare fraction of the value I got out of going to the conference.  I understand that both the ticket and the ancillary costs of attending are quite high for bootstrapped startups.  Trust me on this one: a plane ticket from Japan to Boston and four days in a hotel wipe out nearly a month of revenue for me — and it was worth every penny and then some.

Neil Davidson, the conference organizer, was very generous with giving away tickets to the event to deserving bootstrappers.  In the spirit of the overwhelming generosity of the community and the thoughts sparked by talk of our ethical responsibilities, I will likewise make an opportunity for at least one marginal person to go next year.  Watch this space — I’ll hammer out specifics closer to the actual event.

Now, the obligatory disclaimer: have never accepted advertising on my blog and don’t feel like starting anytime soon.  I was given a free ticket to the conference in return for presenting my Lightning Talk at it, and after winning the competition for best Lightning Talk (which I literally did not know existed until five minutes before it was announced that I won it), I received a Kindle and an invitation to a very nice dinner.  I’m easy to impress, but not that easy: my effusive enthusiasm for this conference comes from it being one of the highlights of my professional career, not because of compensation.

Automatically Minimizing Javascript and CSS with Rails

If you obsess over your YSlow and/or Page Speed performance scores, you might be wondering how to get your Rails app to automatically crunch Javascripts and CSS files without actually requiring cumbersome rake tasks to be repeated during every deploy.  Fear not: you can quickly monkeypatch Rails to do this.

  1. Copy this gist into your lib/ directory.
  2. Create a config/initializers/javascript_minimization.rb as described here.
  3. Use your javascript_include_tag and stylesheet_include_tag helpers as normal, with the caching option turned on.
  4. Watch your users save 100kb.

Organized, Curated Lists of Best Posts From This Blog

When I started this blog four years ago, I never thought it would be seen by more than a few dozen people, so I never planned any sort of information architecture to it.  500 posts (300,000 words!) and several hundred thousand readers later, it is unwieldy to get to the good stuff unless you have been following along for the last couple of years, or have a week free.  To make this a little less annoying, with the help of an assistant from Hacker News I found the best 72 articles I’ve written, grouped them by category, and for ones which are logically related explained what the connections are (in particular, for the “experiment” / “results” pairs).

I hope you find it useful.  If I let out anything or if there is a category you know I publish on but did not include in the list, please let me know in the comments.

Dealing With Market Seasonality

One of the attractions of having a website is that it operates twenty four hours a day, 365 days a year.  However, depending on what your market is, it might not operate evenly on all of those days.  The phenomenon of market seasonality is well-understood in offline businesses — ice cream shops do most business in the summer, retailers have a 4th quarterly sales spike — but understanding of it seems to be fairly limited among software companies.  Since market seasonality has non-trivial impact on many software businesses, I thought I would write about what I’ve learned.

Brief Background

I sell software which makes bingo cards to elementary schoolteachers, who overwhelmingly purchase the software with their own money directly in advance of the day they plan to use it.  This gives me a very quirky calendar relative to many software businesses.  Collapsing my sales chart for the last several years over the calendar year produces…

Average Sales Per Month

As you can see, I have a trough of sales every summer and a spike in October, among other seasonal anomalies.  More about why that is later.  This is just the sales graph, but almost any other statistic for my business — page views, signups, ad impressions, search traffic, etc etc — more or less tracks this curve, although that isn’t going to be the case for all businesses.

Know Your Market

I came into this situation more or less blind: prior to starting my business, if I had thought about it, I would have expected “Yeah, sales are probably going to pick up when the school year starts”, but the huge spike in October and the mini-trough in November/December were all surprises to me.  If possible, don’t be surprised.  Presumably you are expert or near-expert at your particular problem domain, and should be accustomed to what your business calendar probably looks like in your market.  If you aren’t, time to make a courtesy call to either customers or competitors and ask them.  (Sure, you can ask your competitors this.  Why not?  If you’re worried about being laughed off the phone, then find someone who is publicly traded and look at their quarterly earning reports.  The numbers and explanatory notes will often give you all the information you need.)

For highly seasonal businesses, get used to the words year-over-year.  As in, sales in July for me were up 42% year-over-year, meaning that they were up 42% as compared to last July, as opposed to comparing them with this June.  This gives you a much more accurate picture of where the business is going, because for businesses which are not growing at truly meteoric rates, seasonal effects can very easily swamp the underlying trend.  (If you’re a viral Silicon Valley darling, you may be able to discount this.  However, if you’re a retailer, it is virtually impossible to have a January better than your December.  In fact, if it happens, it means you probably had a catastrophe.)

Cash Flow Management For The Off-Season

When people stop buying what you sell, you’ll tend to have less money.  That sounds vacuous, but it has important implications for your cash management.  It means that you’ll need to model your behavior on the ant, not the grasshopper: when times are easy, save up for when times are going to be difficult.

There are other options besides just banking money made during the on-season.  For many businesses, support loads, marketing activities, and the like all roughly coincide with the level of sales.  The number of emails I send regarding Bingo Card Creator falls by roughly 80% during the summer months.  This frees up a lot of time and focus in your organization.  One option to keep the lights on for software businesses is to do consulting or similar billable work during the off-season and go back to the product during the on-season — I took advantage of that this summer, to avoid breaking the piggybank too badly.

Another option, very popular among offline businesses but not used much online, is to do seasonal sales.  For example, if you sell software to teachers, you could do a back to school sale in August and, essentially, cut three weeks off of summer.  I did a 20% discount one year from late July through mid-August, which helped cushion the blow a bit.

Many SaaS startups are currently scratching their heads wondering “Why does this matter?”, because they get consistent revenue on a monthly basis regardless of when the customer’s signup was, and hence can’t conceive of having a growing user population and falling revenues.  This is yet another enviable reason to sell software on a subscription basis.

Other Ways To Exploit The Off Season

If you find that your marketing, customer support, and what have you decline in the off season, and you don’t need to find someone to bill to keep the lights on, you have a variety of fun options:

  • Deploying new features, particularly ones which involve significant engineering changes (like wholescale site redesigns or business model changes), gets much easier when 80% of the user population is not actually on the site.  In the event you have an issue, the number of people affected by it is likely to be much smaller, and you’ll have a few months to get the kinks out prior to sales hitting their peaks again.  When I released the online version of my software last year, conversion rates plunged by about 30% while I was getting debugging the software and (more importantly, as it turns out) the marketing message.  Since it was the dog days of summer, though, that didn’t end up costing me all that much money (although it was a sock in the gut to have my first year-over-year decline ever).
  • For similar reasons, there is never a time like the present to start on a new project.  It might be worth doing one which either has a different seasonal cycle, or none at all, as this eliminates the cash flow challenge.  (On the other hand, having a built-in 3 month vacation isn’t the worst problem in the world.)

Communicating Seasonal Effects

To the extent that outside parties do not understand the seasonality in your market, you need to educate them.  Investors, for example, might assume if they come from a non-seasonal background that any dip in the “up and to the right” story means you are losing traction.  It is a virtual certainty, for example, that GroupOn is going to sell less coupons in January than they will in December.  Having an article on TechCrunch saying “Groupon Growth Finally Puttering Out?” would not be positive for them (well, to the extent that TechCrunch coverage matters to a profitable business…).

Here are two very different graphs:

And here’s the same data, presented in another light:

The first business looks like it has had mostly flat sales for the last year, and a recent slump.  The second business looks like it is killing it.  They are the same business.  In the rather unlikely scenario that I was seeking investors, I know which of these two graphs I’d put in my slide deck.

Other presentational techniques to use:

  • Second derivative of (pick a metric: sales, signups, etc).  The story is “We’re growing faster all the time.”  (Note as you can tell from eyeballing the above graph this is not actually true for me.)
  • Pick a metric where the seasonal effects are less apparent.  Conversion rates are wonderful for this, since often you’ll have an increased conversion during the off-season.  (If someone comes into an ice cream shop in December, that is because they really, seriously are in the mood to buy ice cream.  They’ll also find the place absolutely deserted.)

Plan In Advance For Peak Season, Too

The top of the year for educational bingo is in October, due to a massive influx of teachers wanting to play Halloween bingo at their class party.  (A brief aside: Halloween is considered a holiday of special importance to young children in America, but they aren’t given the day off, which means teachers want to do a fun activity in class to commemorate it.  It has also been mostly secularized, which means public schoolteachers are at no risk to their career if they throw a Halloween party, whereas throwing an Easter party is a wee bit tricky.  You have to do it without mentioning Jesus and, simultaneously, without mentioning that you’re not mentioning Jesus.  Most teachers just have a chocobunny and hope for break to arrive.)

When I was young and stupid, I started planning for Halloween in October.  That works very poorly for Internet marketing: it will take months for your holiday-specific content (such as my page about Halloween bingo cards) to start ranking in the search engines where you want it to be.  If you do AdWords advertising, starting a campaign right before a holiday is likely going to be disastrous, because a) if it gets plunged into approval purgatory you just missed your holiday (this has happened to me quite frequently) and b) your campaign won’t have time to build up the positive history which will get it the widest exposure and best costs.

For SEO projects, start several months in advance of holiday or season you’re targeting.  For AdWords, I’d recommend at least a few weeks.  These schedules can be attenuated if you have a national-level brand to fall back on or a built-in tribe of evangelists like 37Signals has assiduously cultivated, but in general, more time is better.  (If this sounds like a whole lot of work, that is only because it is a whole lot of work, but you can have stupendously disproportionate returns to seasonal promotions.  I set records every year at Halloween even without ranking #1 for the term I was most interested in, and now that I am ranking #1 I am cautiously optimistic that I’ll absolutely smash my records this year.)

If you have another facet of this you’d like covered, or if you have some tactics which have worked for you, I’d love to hear about them in the comments.

Does Your Product Logo Actually Matter?

Some months ago when the 99designs fixed-price logo store launched, my buddy Thomas at Matasano remarked that it would have been the perfect choice for my business.  I said that, while I’m quite impressed with 99designs’ business model and many of the logos on offer, I thought the less-generic-looking logo which I had custom-made fairly cheaply (by the folks at Logo Samurai — I bought three logos and chucked the two I liked least) was better than a generic logo.  Thomas disagreed, and said that the more “professional” look of the off-the-rack logos would offset the genericness.

If Thomas and I were bigwigs at a multinational firm with entire departments devoted to preventing anything from getting accomplished, that disagreement might have been settled by a great deal of harumphing.  However, we’re engineers, so we made a wager: I would purchase a logo handpicked by Thomas and A/B test it against my present logo sitewide, and whomever’s logo lost would donate $100 to a charity named by the victor.

The Original Logo:

The 99 Designs Logo:

Technical Details

This A/B test was done sitewide — on first access of the site, via landing page, organic search, or any other source, users were randomly assigned one of the two logos to be the “official” Bingo Card Creator logo.  They saw the same logo for all subsequent accesses.  This was done with A/Bingo, the Rails A/B testing framework I wrote.  This particular test required A/Bingo to be slightly extended to ignore bots, because — since the test was sitewide and visible prior to a bot-blocking login screen — the deluge of bots this site deals with was drowning out the signal with their non-converting noise.  Happily, since A/Bingo is open source, all other users of the software benefit from Thomas and I fighting over geek cred.

We agreed to measure conversions to the free trial rather than conversions to purchase.  Measuring conversions to purchase would have been technically feasible, but since much of my market is out of school during the summer, getting enough data to be statistically significant would probably have taken months.  Conversions to the trial are much more frequent so it was reasonable to expect sufficient data in a week or two.  The test was allowed to run for two weeks.

The Results

My original logo ended up winning by a virtual mile: 11.05% of all users seeing the classic logo converted, versus 9.92% of those seeing the 99designs logo.  Just by eyeballing that you might think “Hmm, 1% difference, that’s probably just noise”, but when you factor in the sample sizes (8,500 real people saw each variation) it turns out that the difference is statistically significant at the 99% confidence level.

(If those conversion numbers strike you as fairly low, remember that the test is running sitewide rather than on a single landing page.  Many folks who visit the site are drawn to free activities created for SEO purposes, and downloading those activities doesn’t necessarily require signing up.  Conversion rates for my AdWords landing pages are closer to 25 ~ 30%.  Relatedly, consolidated conversion rate is a poor metric to make decisions on, but that is a discussion for another day.)

Even though in this particular case the A/B test did not succeed in identifying a profitable change to make in my business, I consider it a wonderful use of my time — it only required $99, a few minutes of time from my local designer (to put the logo on the background I use for my header without it getting aliased), and a single line of code.  I can’t possibly stress enough how valuable it is to have your A/B tests require only a single line of code.  You’ll test more frequently, you’ll test more diligently, and you’ll discover more valuable things about your business.

Implications For Other Businesses

Graphic designers: Some of you appear to be highly opposed to spec work, at least partially because you believe it to produce inferior work.  You might consider measuring the effect of well-designed logos (for any value of “well-designed”, for example, “uses a business model I approve of”) versus other logos.  I have the graphical skill of a molerat — I should be linking to this article on your blog, professional graphic designer, not writing it.

99designs: Don’t get me wrong, I love you guys.  My Appointment Reminder site uses an off-the-rack logo from your site that I’m very, very fond of.  I’m very, very open to the notion that there exist circumstances under which this test would have come down the other way.

Software companies: Your logo could potentially add or subtract 10% from enterprise value.  Bolded for emphasis because holy cow.  (10% higher conversion rate to a trial flows virtually directly to my bottom line.)  The majority of companies in our industry get a logo done early in their corporate life, generally exactly once.  For some companies, brand equity might be of paramount concern, and the risk of creating confusion in a two-week A/B test swamps even a 10% potential increase in conversion rates.  For those of us who are more transaction-oriented than brand-oriented, though, this is an easy, obvious, fairly inexpensive area to test.

(Obligatory “I passed Stats 101″ disclaimer: the significance test I performed says that logo #1 is 99% likely to truly be better than logo #2, rather than saying that logo #1 is 99% likely to be 10% better than logo #2.)

SEO for Software Companies

This is a rough outline of my verbal remarks while giving a presentation at the Software Industry Conference.  Regular readers of my blog will notice it has a lot of overlap with previous posts on the topic, but I thought posting it would save presentation watchers from having to take copious notes on URLs and expand the reach of the presentation to people who couldn’t attend this year.

Brief Biography

My name is Patrick McKenzie.  For the last six years I was working in Japan, primarily as a software engineer.  In the interim, I started a small software company in my spare time, at about five hours a week, which recently allowed me to quit my day job.  Roughly half my sales and three quarters of my profits come as a result of organic SEO, and the majority of the remainder come from AdWords.  If you need to know about AdWords, talk to Dave Collins, who is also attending. This presentation is about fairly advanced tactics — if you need beginner-friendly SEO advice, I recommend reading SEOMoz’s blog or taking a look at SEOBook.

Bingo Card Creator makes bingo cards for elementary schoolteachers.  This lets them, for example, teach a unit on chemistry and then, as a fun review game, call out the names of compounds like “ozone” and have students search out the chemical symbols on their bingo cards.  Students enjoy the game because it is more fun than drilling.  Teachers enjoy the game because it scales to any number of students and slides easily into the schedule.  However, making the cards by hand is a bit of a pain, so they go searching on the Internet for cards which already exist and find my website.  I try to sell them a program which automates card creation.

SEO can be a powerful tool for finding more prospects for your business and increasing sales.

SEO In A Nutshell

People treat SEO like it is black magic, but at the core it is very simple: Content + Links = You Win.

Content: Fundamentally, users searching are looking for keywords, and Google wants to send searchers to content which is responsive to the intent of the searcher.  Overwhelmingly, this means to content directly responsive to the keywords.  This is particularly true on the long tail, meaning queries which are not near the top of the query frequency distribution.  Many more people search for “credit cards” than for “How do I make a blueberry pie?”

For the most popular queries, the page that ranks will likely not be laser-targeted on “credit cards”.  However, for the long tail, a page that is laser-targeted will tend to win if it exists.  The reason is that Google thinks that your wording carries subtle clues of your intent, so it should generally be respected.  Someone looking for “How to make a blueberry pie?” isn’t necessarily as sophisticated a cook as someone who searches for “blueberry pie recipe” — they might not even be looking with the intention of making a blueberry pie, but rather out of curiosity as to how it is made, and so a recipe does not directly answer their intent.

Links: With billions of pages on the Internet, there needs to be a way to sift the wheat from the chaff and determine who wins out of multiple close pages.  The strongest signal for this is how trusted a site and how trusted a page is, and this is overwhelmingly measured by links.  A link from a trusted page to another page says “I trust this other page”, and the aggregate graph shows you which pages are most trusted on the Internet.  Note that trust is used as a proxy for quality because it is almost impossible to measure quality directly.

It is important to mention that links to one bit of content on your site help all other content — perhaps not as much as the linked content, but still substantially.  Wikipedia’s article on dolphins doesn’t necessarily have thousands of links pointing to it, but over their millions of articles like the History of the Ottoman empire, they have accumulated trust sufficient that a new page on wiki is assumed to be much better than a new page on a hobbyist’s blog.  Note that because Wiki ranks for nearly everything they tend to accumulate new citations when people are looking for someone to cite.  This causes a virtuous cycle (for Wiki, anyway): winners win.  You’ll see this over and over in SEO.

Despite this equation looking additive, SEO very rarely shows linear benefits.  Benefits compound multiplicatively or exponentially.  Sadly, many companies try to develop their SEO in a linear fashion: writing all content by hand, searching out links one at a time, etc.  We’ll present a few techniques to do it more efficiently.

The Biggest Single Problem

The biggest single problem with software company’s SEO is that they treat their website like a second class citizen.  The product gets total focus from a team of talented engineers for a year, and then the website is whipped up at 2 AM in the morning on release day and never touched again.  You have to treat your website like it was a shipping software product of your company.

It needs:

  • testing
  • design
  • strategic thought into feature set
  • continuous improvement
  • for loops

“For loops?”  Yes, for loops.  You’d never do calculations by counting on your fingers when you have a computer available to do them for you.  Hand-writing all content in Notepad is essentially the same.  Content should be separated from presentation — via templates & etc — so that you can reuse both the content and presentational elements.  Code reuse: not just for software.

Scalable Content Generation

Does anyone have a thought about how large a website’s optimal size should be?  10 pages?  A hundred pages?  No, in the current environment, the best size for a website is “as large as it possibly can be”, because of how this helps you exploit the long tail.  As long as you have a well-designed site architecture and sufficient trust, every marginal topic you cover on your website generates marginal traffic.  And if you can outsource or automate this such that the marginal cost of creating a piece of content is less than the marginal revenue received from it, it makes sense to blow your website up.

This is especially powerful if you can make creation of content purely a “Pay money and it happens” endeavor, which lets you treat SEO like a channel like PPC: pour in money, watch sales, laugh all the way to the bank.  The difference is that you get to keep your SEO gains forever rather than having to rebuy them on every click like PPC.  This is extraordinarily powerful if you do it right.  Here’s how:

Use a CMS

The first thing you need to enable scalable content generation is a CMS.  People need to be able to create additional content for the website without hand-editing it.  WordPress is an excellent first choice, but you can get very, very creative with custom CMSes for content types which are specific to your business if you have web development expertise.

Note that “content” isn’t necessarily just blog posts.  It is anything your customers perceive value out of, anything which solves problems for them.  That could be digitizing your documentation, or answering common questions in your niche (“How do I…” is a very common query pattern), or taking large complex data sets and explaining their elements individually in a comprehensible fashion.  Also note that it isn’t strictly text: you can do images and even video in a scalable fashion these days.

For example, using Flickr Creative Commons search, you can tap millions of talented photographers for free to get photos, so illustrating thousands of pages is as simple as searching, copying, and crediting the photographer.  You can use GraphicsMagick or ImageMagick to create or annotate images algorithmically.  You can use graphing libraries to create beautiful graphs from boring CSV files — more on that later.

The reasons why you’d use a CMS are they make content easy to create and edit, so you’ll do more of it.  Additionally, by eliminating the dependency on the webmaster, you can have non-technical employees or freelancers create content for you.  This is key to achieving scale.  You can also automate on-page SEO optimization — proper title tags, interlinking, etc — so that content creators don’t have to worry about this themselves.

Outsource Writing

You are expensive.  English majors are cheap.  Especially in the current down economy, stay at home moms, young graduates, the recently unemployed, and many other very talented folks are willing to write for cheap, particularly from home.  This lets you push the marginal cost of creating a new page to $10 ~ $15 or lower.  As long as you can monetize that page at a multiple of that, you’ll do very well for yourself.  Demand Media is absolutely killing it with this model.

Finding and managing writers is difficult.  If you use freelancers and find good ones, hold onto them for dear life, since training and management are your main costs.  Standardize instructions to freelancers and find folks who you can rely on to exercise independent thinking.

You can also get content created as a service, using TextBroker.  Think of the content on your website as a pyramid: you have a few pages handwritten by domain experts with quality off the chart, and then a base of the pyramid which is acceptable but perhaps not awe-inspiring.  At the 4 star quality level, you can get content in virtually infinite quantity at 2.2 cents per word.  You can either have someone copy/paste this into your website or do a bit of work with their API and automate the whole process.

You can use software to increase the quality of outsourced content.  For example, putting a picture on it automatically makes it better.  You can automate that process so your editors can quickly do it for all pages.  You can remix common page elements — calls to action, etc — which are polished to a mirror shine with the outsourced content.  You can also mix content from multiple sources to multiply its effectiveness: if you have 3 user segments and 3 features they really value, that might be 9 pages.  (If you use 2 features per page, that is 18.  As you can see, the math is gets very compelling very quickly.)

Milk It

Now that you’re set up to do content at scale, you can focus on doing it well.  The best content is:

Modular: You can use it in multiple places on the website.  You paid good money for it.  If you can use it in two places, the cost just declined by half.

Evergreen: The best possible value for an expiration date is “never”.  Chasing the news means your content gets stale and stops providing value for the business.  Answering the common, recurring, eternal problems and aspirations of your market segment means content written this year will never go out of style.  That lets you treat content as durable capital.  Also, because it tends to pick up links over time, it will get increasing traffic over time.

The first piece of content I made for my website took me two hours to write.  It made $100 the first month.  Not bad, but why only get paid once?  It has gone on to make me thousands over the years, and it will never go out of style.

Competitively Defensible: One of the tough things about blog posts is that any idiot can get a blog up as easily as you can.  Ideally, you want to focus on content which other people can’t conveniently duplicate.  OKCupid’s blog posts about dating data are a superb example of this: they use data that only they have, and they’ve made themselves synonymous with the category.  No wonder they’re in the top 3 for “online dating”.  Proprietary data, technical processes which are hard to duplicate, and other similar barriers establish a moat around your SEO advantage.

Process-oriented: If something works, you want to be able to exploit it in a repeatable fashion.  Novelty is an excellent motivational factor and you can’t lose it, but novelty that can be repeated is a wonderful thing to have.  You also want to have a defined step where you see what worked and what didn’t, so that you can improve your efforts as you go on.

Tracking:

Track what works!  Do more of that!  Install Google Analytics or similar to see what keywords people are reaching for on your site.  Keywords (or AdWords data) are great sources of future improvements.  Track conversions based on landing page, and create more content based on the content which is really winning.  If content should be winning but isn’t, figure out why for later iterations — maybe it needs more external or internal promotion, a different slant, a different target market, etc.

Case Study

Getting into the heads of my teachers for a moment — a key step — most teachers have a lesson planned out and need an activity to slot into it.  For example, they know they have a lesson about the American Revolution coming up.  Some of them, who already like bingo, are going to look for American Revolution bingo cards.  If my site ranked for that, that would be an opportunity to tell them that they could use software to create not just American Revolution activities but bingo for any lesson if they just bought my software.

So I made a CMS which, given a list of words and some explanatory text, would create a downloadable set of 8 bingo cards (great for parents, less great for teachers) on that topic, make a page to pitch that download in, and put an ad for Bingo Card Creator on the page.  Note how I’m using this content to upsell the user into more of a relationship with me: signing up for a trial, giving me their email address, maybe eventually buying the software.

I have a teacher in New Mexico who produces the words and descriptions for me.  The pages end up looking like this for the American Revolution.  She produces 30 activities a month for $100, and I approve them and they go live instantly.  This has been going on for a few years.  In the last year, I’ve started doing end-to-end conversion tracking, so I can attribute sales directly to the initial activity people started with.

This really works.  Some of the activities, like Summer bingo cards or Baby Shower Bingo cards, have resulted in thousands of dollars in sales in the last year.  $3.50 in investment, thousands in returns.  And there is a long tail of results:

This graph shows the 132 of the 900 activities which generated a sale in the last year.  You can see that there is a long tail which each generated one sale — in fact, a hundred of them.  Sure, you might not think that Famous Volcano bingo cards would be that popular, but I’ll pay $3.50 to get a $30 sale as often as the opportunity is offered.  These will also continue producing value in the next years, as they already have over the last several years: note that roughly half of these which produced a sale in the last 12 months were written in 2007 or 2008.

This took only a week or two to code up, and now 5 minutes a month sending my check and a thank-you note to the freelancer.  I’ve paid her about $3,000 over the last few years to write content.  In the last year alone, it has generated well over $20,000 in sales.  If you do things this efficiently, SEO becomes a channel like PPC — put in a quarter, get out a dollar, redeploy the profits to increase growth.

Any software company can create content like this, with a bit of strategic thinking, some engineering deployed, and outsourced content creation.  Try it — you can do an experiment for just a few hundred dollars.  If it works, invest more.  (Aaron Wall says that one of the big problems is that people do not exploit things that work.  If you’ve got it, flaunt it — until it stops working.)

Linkbait

Linkbait is creating content intended to solicit links to your website.  This can be by exploiting the psychology of users — they show things to friends because they agree with them strongly, or they hate them.  They create links because it creates value for them, not value for you — it increases their social status, it flatters their view of the world, it solves their problems.

All people are not equal on the Internet: twenty-something technologists in San Fransisco create hundreds of times more links per year than retired teachers in Florida.  All else being equal, it makes sense to create more of your linkbait targeted at heavy linking groups.  They’ve been labeled the Linkerati by SEOMoz, and I recommend the entire series of posts on them highly.

Software developers have some unique, effective ways to create linkbait.  For example:

Open Source Software

OSS developers and users are generally in very link-rich demographics.  OSS which solves problems for businesses tends to pick up links from, e.g., consultants deploying it — they will cite your website to justify their billing rate.  That is a huge win for you.  There are also numerous blogs which cover practically everything which happens in OSS.

OSS is fairly difficult to duplicate as linkbait, because software development is hard.  (Don’t worry about people copying it — you’ll be the canonical source, and the canonical source for OSS tends to win the citation link.  Make sure that is on your site rather than on Github, etc.)

OSS in new fields in software — for example, Rails development the last few years — has landgrab economics.  The first semi-decent OSS in a particular category tends to win a lot of the links and mindshare.  So get cracking!  And keep your eyes open for new opportunities, particularly for bits of infrastructural code which you were going to write for your business needs anyhow.

Case Study: A/Bingo

I’m extraordinarily interested in A/B testing, and wanted to do more of it on my site.  At the time, there was no good A/B testing option for Rails developers.  So I wrote one.  It went on to become one of the major options for A/B testing in Rails, and was covered on the official Rails blog, Ajaxian, and many other fairly authoritative places on the Internet.  It is probably the most effective source of links per unit effort I’ve ever had.

Some tactical notes:

  • Put it on your website.  You did the work, get the credit for it.
  • Invest in a logo — you can get them done very cheaply at 99designs.  Pretty things are trusted more.
  • Spend time “selling” the OSS software.  Documentation, presentation of benefits, etc.
  • OSS doesn’t have to be a huge project like Apache.  You can do projects in 1 day or 1 week which people will happily use.  (Remember, pick things which solve problems.)

Conclusion

I’m always willing to speak to people about this.  Feel free to email me (patrick@ this domain).

Speaking at Software Industry Conference

I’m currently in Dallas at the Software Industry Conference, where I’ll be giving a presentation about SEO strategies on Saturday.  In the meanwhile, if you’re at the conference or feel like coming out to the Hyatt Regency, feel free to get in touch with me.

As you have probably guessed, I’ll be posting the presentation and some textual elaboration on it right after I finish delivering it.  (I don’t know if video will be available this time.)

Loading...
Grow your software business:
(1~2 emails a week.)