Why I Don’t Host My Own Blog Anymore

I moved my blog over to WPEngine recently. Why? Read on.

I started blogging about 375,000 words ago (about three full-length novels… crikey). At first, I was on a subdomain of WordPress.com, mostly because a) it was free and b) I had no intention of ever writing anything more significant than a few observations I made while developing a summer project.  My posts about making and marketing software turned out to be rather more popular than I anticipated, and I eventually made them my main professional presence and moved them to their own domain name.

Eventually, I decided that I had to be in control of my site rather than being locked into only the themes and functionality that WordPress.com supported, so I started hosting the blog myself.  I installed WordPress on a modest 512 MB VPS from Slicehost, with the standard Apache 2 / PHP / MySQL stack, and thought that would be the end of my hosting decisions.  Sadly, it was only the beginning.

WordPress, PHP, and Apache: The Trifecta Of Finnicky Software

I’m a web developer by trade, so I’m very skeptical of unevidenced comments like “lol, PHP can’t scale.”  It’s transparently obvious that PHP can scale — one look at who uses it (Facebook is really all you need to know) proves that.  Sadly, my system crashed frequently under load, even after upgrading the 512 MB slice to a 1,024 MB slice, and after tweaking Apache and PHP’s memory-usage settings for hours.  I read practically everything written about caching plugins for WordPress and it availed me not.  The loads at issue weren’t generally that impressive, either: 10,000 pageviews here,  100,000 pageviews there, either way a modern server in 2006 ~ 2012 should eat those numbers for breakfast.

In addition to tweaking the heck out of my settings, I was hacking WordPress to lessen the load on the server.  I went so far as to putting all the static resources for this blog (CSS, images, etc) on a server used for my software business, since no level of traffic has ever managed to give Nginx a problem in my experience.  This made the blog much more stable under load, but it still crashed occasionally.

I eventually found the culprit: a setting in Apache named, ironically, KeepAlive.  Since then I have been seized by missionary zeal in trying to convince people that leaving KeepAlive on will probably not be in your best interests.  I turned off KeepAlive and have had a much more stable time since then.

Even with all this work, my blog crashed four times in 2011.  Each time, my monitoring software woke me up, and I learned things like “Oh, Jimmy Wales tweeted about an article of mine” at 4 AM in the morning while trying to restart the server.  I’ve probably lost in excess of 100,000 readers over the years for the blog being down.

I eventually got so fed up with Apache that I spent a day to migrate this (and 15 other sites) from using Apache as a front-end webserver to using Nginx to serve all requests and proxy dynamic requests to a backend Apache instance.  (Why not use Nginx to execute the PHP directly?  Long story — I do that elsewhere, but it isn’t painless.)

Routine Maintenance Sucked, Too

You’ve heard about WordPress’ somewhat spotty security record, right?  I’m aware of it as a practicing engineer and have even reported a few doozies in commonly deployed themes/plugins myself.  So I took some fairly extraordinary measures to secure the blog versus a stock WordPress installation:

  • required all access of the admin to happen through a proxy that I control
  • locking down all files on the WordPress installation such that the webserver could not write to them — this makes plugin installation/maintenance a manual chore and breaks some plugins, but makes it less likely that a vulnerability in a plugin or theme will ruin your day
  • performed some sporadic code audits on things I am inexpert on in a language I detest.  I eventually stopped doing this because reporting vulnerabilities to the WordPress ecosystem could easily be my full time job.

As a result of this, to my knowledge my blog never got compromised.  Whee, great.  It only took me a few billable weeks.  Plus I had all the usual fun of applying patches, making backups, restoring from backups when MySQL decided to eat the wp_posts table (still no clue why that happened), tweaking settings for rotating logs, migrating my hosting provider (Rackspace bought Slicehost so I had to move servers), yadda yadda yadda.

“Do You Enjoy Hosting WordPress?”

Two years ago at a conference I ran into Jason Cohen, a very smart guy who had sold his previous software business.  He told me that his new venture was managed WordPress hosting.  I was outwardly interested and inwardly cringing, because I thought “There are already WordPress hosts available for $4 a month, they all suck, and the software is pretty much irredeemable.  If it weren’t the best blogging software available I’d take a hammer to my backups then burn the shards and bury them on sanctified ground so that they never troubled the world again.”

At some point Jason asked me if I liked hosting WordPress.  I told him “I love hosting WordPress!  I love all the power and tweakability!”  And while I do appreciate control, I still can’t imagine what possessed me to say that.  Hosting WordPress has been a black hole of my time.

Last year I ran into Jason Cohen at another conference. He told me that WPEngine, the WordPress hosting company he’d told me about, was live and doing well.  In my haste to demonstrate that I had learned something from cutting myself on WordPress for the last five years, I mentioned “I guess you guys figured out to turn KeepAlive off, huh?”

Jason said “Actually, no.  I mean, sure, in the general case for a VPS, you want it off because otherwise your site gets non-responsive under load.  However, if you’ve got Apache talking to the outside world, something is wrong.  Apache only handles the request after it’s been through a load balancer, a Varnish caching proxy, and then Nginx, because Nginx does static content so well.  If the request gets that far you want KeepAlive on because your Apache will only be talking within your datacenter, only have a handful of connections, and you want those connections to be alive almost indefinitely because setup/teardown is always waste.”

You know how often I talk to software company CEOs and get not just corrected but destroyed and then re-educated about a point of technical fact?  Suffice it to say it made an impression.

So when Jason invited me over to WPEngine to do some marketing work, I leaped at the chance.  I went down to Texas for a week, met the team, and did my thing.  (Sidenote: Want to see a fairly typical week’s work for me?  Take a gander at their speed test tool, which you can point at an arbitrary WordPress site and learn why your page load speed isn’t optimized enough yet.  The punchline is, of course, that if you were with WPEngine they would have already fixed that for you.  I assisted with a redesign of this, wrote the month-long WordPress optimization course that the tool will let you sign up for, and generally improved copy and the like on their marketing site.)

So I Switched

In the course of hearing the sales pitch from them several times so I could write it accurately, I became convinced: WPEngine is absolutely superior in every way to me continuing to host the blog myself.  So I took out my credit card, signed up for their $200 a month plan (prices got reduced recently, see here), and migrated my blog over.  It has been quietly hosting my blog for the last several weeks, including through two of my highest traffic days ever, without a hitch.  For the first time, I can watch a post go to the top of HN and not have to have “top” open to keep an eye on swap consumption.

A few things that particularly impressed me about WPEngine:

  • A few hours after migrating my blog I got an automated email saying that they had found an outdated copy of TimThumb in my WordPress install and had upgraded it for me.  It wasn’t a vulnerability (permissions locked down saved the day for that one), but I’m very, very glad they keep an eye on things so that I don’t have to.
  • Migration was almost painless.  I just dumped the WP database, grabbed my existing files, and copied them over as instructed.  I needed to speak to support to get a setting tweak done for me (the plan I bought has WordPress multi-site not single site like my old blog, and this resulted in a minor issue), but all told I was up and running in about two hours of elapsed time.
  • Just like their speed tool promised, my site did get modestly faster.

I’m a bit of a YSlow fanboy and ever once in a while I go through my sites and make an optimization pass, so I usually have all the low-hanging fruit like gzipping, static content loaded from multiple domains, and the like taken care of.  I wasn’t expecting WPEngine to shave much time from my page loads, given the amount of optimization work I had already done, but I kept the old server around to do a fair test on AOL’s speed tool.  Take a look what happens: here’s a video showing (left) my old 1,024 MB VPS versus (right) WPEngine showing the same page from my blog.

If you didn’t watch the movie, I’ll spoil it: content pops in about half a second earlier (and finishes loading .8 seconds earlier) on WPEngine with no manual tweaking versus my tweaked-to-limit-of-my-ability VPS.

That test uses IE8 on a reasonable residential Internet connection coming from the US. For accessing from Japan or on a mobile device, it is viscerally faster for me, probably due to the CDN which WPEngine uses.

Do You Have A Blog For Business? Use WPEngine

The VPS that I used to run my blog on cost a bit over $100 a month (1 GB Rackspace slice + 160 GB of bandwidth + backups = I am not actually sure). WPEngine runs me $200 a month because I have high anticipated traffic. They have a $29 option for folks who don’t.

$2,500 a year for blog hosting sounds a bit on the high side, but it is honestly nothing against the amount of time that I will no longer have to invest supporting this sucker. I love being out of the hosting business. I never intended on being in it in the first place — it was always just something I needed to do to write for people. Less time poured down that black hole means more time to work on my businesses and share what I learn doing so.

WPEngine has been a total, epic win for me. I suggest that you use them if you are using WordPress for a business: go ahead, make with the clicky clicky. Want to just geek out on how they have their infrastructure setup? See here.

P.S. Long-time readers are aware of this, but just to reiterate: I don’t take money for blog posts, was not asked to write this by WPEngine (who are, again, clients of mine), and would not have written it except that their service really rocks.

66 Comments

Salary Negotiation: Make More Money, Be More Valued

[Editor's note: At nearly 7,000 words, you probably don't want to try reading this on an iDevice.  Bookmark it and come back later.]

Imagine something a wee bit outside your comfort zone.  Nothing scandalous: just something you don’t do often, don’t particularly enjoy, and slightly more challenging than “totally trivial.”  Maybe reciting poetry while simultaneously standing on one foot.

If I told you I would pay you a hundred thousand dollars if you did five minutes of poetry recital while standing on one foot, would you do it?  It’s an absurd image, but play it straight.  There is no hidden gotcha here.  You won’t be videotaped.  Your friends will never see you make a fool of yourself.  The revolution will not be YouTubed.  The offer is exactly as simple as you think it is: poetry, foot, $100,000.

Would you read poetry for me?

Of course you would.  You’d be screamingly stupid not to.  In fact, not only would you read poetry, you’d probably take a poetry class to make sure you did it right, or go to the gym to verify “Yep, sure enough, I can stand on one foot.  Phew.  Pass me the Shakespeare.”  If you couldn’t stand on one foot, you’d fix that, because you know that is much easier than other things you routinely accomplish and you suddenly have a hundred thousand wonderful reasons to learn it, too.

What if you were talking about this at dinner with your friends, and one of them said “Oh, no, I’d never do that.  I just don’t do poetry.  I’m an engineer.  And besides, my father told me that people who stand on one foot look silly.  And what do I need $100,000 for anyhow?”  You would not clap them on the back and say “Damn straight!  Man, poets, always trying to tempt virtuous engineers into their weird poetry-spouting flamingo-standing ways.”  You’d say “Dude, it’s five minutes.  Heck, I’ll help you practice.”

This is pretty much how I feel every time I talk to my engineering friends about salary negotiation.  We overwhelmingly suck at it.  We have turned sucking at it into a perverse badge of virtue.  We make no affirmative efforts to un-suck ourselves and, to the extent we read about it at all, we read bad advice and repeat it, pretending that this makes us wise.

Dude, it’s five minutes.  Let’s un-suck your negotiation.

(New to the blog?  Hiya.  I generally write as an engineer for engineers.  Non-engineers can benefit from many of the same techniques, though the hiring market isn’t nearly as in your favor at the moment as it is for engineers in most major US metro areas.)

Why Negotiation Matters

Your salary negotiation — which routinely takes less than 5 minutes to conclude — has an outsized influence on what your compensation is.  Compensation can include money or things which are more-or-less fungible replacements for money, but it can also include interesting things which you value from “more time with your family” to “opportunities to do tasks which you find fulfilling” to “perks which make a meaningful difference in your day-to-day quality of life.”  That makes your negotiation five very important minutes.  You generally can’t do a totally bang up job on any five minutes of work this year and have your boss give you an extra $5,000.  You can trivially pick up $5,000 in salary negotiations just by sucking less.

Since salaries are shockingly durable over time, particularly if one is not good at negotiating, you can expect a $5,000 increase in salary to compound with your standard annual read-the-HR-chart-percent raise, cause a similar increase in your 401k contribution (which also compounds), and establish a higher peg for any further jobs you take (if you’re unsavvy and allow these other jobs access to your prior salary history, at any rate).  Accordingly, over the next ten years, the value of $5,000 a year extra salary is close to $100k gross, and the value of $15,000 a year extra (very achievable if you’re e.g. a young engineer who doesn’t realize that the hiring market is on fire right now, even outside of tech epicenters like Silicon Valley) is over $100k even net of taxes.

Shifting Your Mindset To Embrace Negotiation

We’ll discuss tactical advice in a moment, but let’s talk about the psychology of negotiation first.  I think that middle class Americans are socialized from a very young age to view negotiation as something that is vaguely disreputable and engaged in only by poor people.  Think of the associations you have with the word “haggling”: do you think of a successful young professional talking about thousands of dollars in a brightly lit office?  No, you probably think of an old woman arguing over a trivial sum of money in a dirty flea market.

If I were a little more paranoid and a little more Marxist, I’d honestly think that you’re so misinformed about reality that that is almost prima facie evidence of a conspiracy to keep you in the dark about this, to the advantage of people who a) you won’t negotiate with and b) who will feel absolutely no compunctions about negotiating with you.  Principally, this will be your employers.  People say that your house is the biggest purchase you’ll ever make, but it won’t be the most consequential negotiation.  If you’re sane only about 25% or so of your gross income is subject to the results of real estate negotiations.  Close to 100% is subject to the results of salary negotiations.  Thus, your salary negotiations are probably going to be the most important financial decisions you will ever make.  We socialize middle class Americans to go into them unprepared, demotivated, and fearful of success.

The reality is that rich, successful people negotiate.  (This is one important way in which they get — and stay — rich.)  It is an all-day-every-day thing in much of the business world, which is where most rich people get their money.

Your Counterparty Does Not Share Your Mental Model of Negotiation

Salary negotiations are very asymmetrical.   Companies know this and routinely exploit it.  Job seekers don’t, perhaps because they think doing so would be unfair and the word “exploit” makes them acutely uncomfortable.  So we often default by pretending that the employer is evaluating the negotiation like we would.  This is not true, and acting like it is true will harm both your interests and the interests of your future employer.

For example, many people’s mental model of employment is that an employee with a $60,000 a year salary costs about $60,000 a year to hire.  If they negotiate $65,000 instead, that’s $5,000 extra which has to come from… somewhere.  If the negotiation breaks down, then that is $60,000 saved.  This mental model is broken.

First, get into the habit of seeing employees like employers see them: in terms of fully-loaded costs.  To hire someone you need to pay for their salary, true, but you also have taxes, a benefits package, employer contributions to retirement, healthcare, that free soda your HR department loves mentioning in the job ads, and what have you.  (Trivia: for a US employer of professionals, the largest component after salary is usually healthcare, followed by payroll taxes.)  The fully-loaded costs of employees are much higher than their salary: exactly how much higher depends on your locality’s laws, your benefits package, and a bunch of other HR administrivia, but a reasonable guesstimate is between 150% and 200% of their salary.

The fully loaded cost of an engineer receiving market salaries these days in California or New York is close to $20,000 a month.  It is “only” $10,000 a month if they’re receiving a heavily below-market salary, such as if they’re working for a startup.  If you have a kid brother who majored in Flemish Dance and got a modest full-time job at a non-profit, his fully-loaded cost is still probably $4,000 a month or more.

This is a roundabout way of telling you that companies are not sensitive to small differences in employee wages because employees are so darned expensive anyhow.  You see $5,000 and think “Holy cow, even after taxes that’s a whole new vacation.  Five thousand dollars.  Five thousand dollars.  It would be so very, very greedy of me to ask for five thousand whole dollars.”  The HR department sees $5,000 and thinks “Meh, even after we kick in the extra taxes, that is only about 3% of their fully-loaded cost for this year anyhow, or seven hundredths of one percent of that team’s hiring budget.  I wonder if the cafeteria has carrot cake today?”

Virtually any amount of money available to you personally is mouse droppings to your prospective employer.  They will not feel offended if you ask for it.  (I received a comment that this is untrue for startups by someone today.  For a funded startup which has enough engineers to warrant a foosball table, the company payroll is well north of $100,000 a month.  Making a new hire is a big commitment, but they still have a lot of flexibility on the  details because the details do not shave months off of their runway.)

We’ve been talking about your employer as an abstraction, but in the instant case you’re talking to an actual person.  Let’s call him Bob.  It is Bob’s job to get you signed with the company as cheaply as possible, but Bob is not super motivated to do so, because Bob is not spending Bob’s money to hire you.  Bob is spending Bob’s budget.  Bob generally does not get large performance incentives for shaving money off of his hiring budget: you get a new Macbook if you convince Bob to give you $5k extra, but Bob gets (if he is anomalously lucky) a dinner at TGIFridays if he convinces you to take $5k less.  In fact, there are many organizations (and Bobs) for whom power, status, and money come from asking for more budget every year.  If you turn out to be on the expensive side, that is great for Bob, because a) he manages a high-powered peon so he must be a high-powered manager and b) this will help Bob get more budget next quarter.  So if you’re worried about what Bob will think of your moral character, or you want to compensate Bob because you feel you owe him for this job opportunity, do Bob a solid and negotiate in a spirited fashion with him.

You don’t owe Bob for giving you this job opportunity, by the way.  Internalize this: everyone in this discussion is a businessman.  (Some might call themselves “regular employees,” which just means they’re businessmen with self-confidence issues and poor business skills.)  If the deal makes economic sense, it will happen.  If it doesn’t, firm handshakes will be exchanged, non-specific promises will be uttered, and everyone will forget about this discussion in a matter of hours.  You will not be blackballed for negotiating.  Bob couldn’t care less and, even if he did care, he has better things to do with his time than worry about a candidate he didn’t hire.  Bob is working through a list of a dozen people right now, and his manager Dave is being such a hard case about that project’s schedule, and he’s not sure he can make his daughter’s piano recital, and the cafeteria’s carrot cake was a little dry.  Bob is far, far less invested in this negotiation than you are.

Your Negotiation Started Before You Applied To This Job

Your negotiation doesn’t happen in a vacuum.  Generic career advice is a little outside the scope of this post (though I’ve previously written a bit with engineers in mind that folks from many walks of life tell me was useful), but to make a long story short, many people think job searches go something like this:

  1. See ad for job on Monster.com
  2. Send in a resume.
  3. Get an interview.
  4. Get asked for salary requirements.
  5. Get offered your salary requirement plus 5%.
  6. Try to negotiate that offer, if you can bring yourself to.

This is an effective strategy for job searching if you enjoy alternating bouts of being unemployed, being poorly compensated, and then treated like a disposable peon.  (I served three years as a disposable peon in a Japanese megacorp and might be projecting a tad bit here.  Regardless, my loss is your gain.)

You will have much, much better results if your job search looks something more like:

  1. (Optional but recommended) Establish a reputation in your field as someone who delivers measurable results vis-a-vis improving revenue or reducing costs.
  2. Have a hiring manager talk with you, specifically, about an opening that they want you, specifically, to fill.
  3. Talk informally (and then possibly formally) and come to the conclusion that this would be a great thing if both sides could come to a mutually fulfilling offer.
  4. Let them take a stab at what that mutually fulfilling offer would look like.
  5. Suggest ways that they could improve it such that the path is cleared for you doing that voodoo that you do so well to improve their revenue and/or reduce their costs.
  6. (Optional) Give the guy hiring you a resume to send to HR, for their records.  Nobody will read it, because resumes are an institution created to mean that no one has to read resumes.  Since no one will read it, we put it in the process where it literally doesn’t matter whether it happens or not, because if you had your job offer contingent on a document that everyone knows no one reads, that would be pretty effing stupid now wouldn’t it.

You might think that desirable jobs at well-managed companies (Google, Microsoft, hot startup FooWithTheWhat.ly, etc) have layers and layers of bureaucratic scar tissue (a great image from 37Signals) to ensure that their hiring will conform to established processes and that offers will not be given to candidates sourced by using informal networks and interpersonal connections.  If you believe this, you have a dangerously incomplete mental model of how the world operates.  I have a specific recommendation for you to make that model more complete: start talking to people who actually work for those companies and who have hiring authority.  Virtually no company has a hiring process which is accurately explained by blog posts about the company.  No company anywhere has a hiring process which is accurately explained by their own documents about how the hiring process works.

I won’t give names, but all of the following are companies you’ve heard of:

  • Ironclad non-compete with an IP assignment provision of major multinational… struck from the contract with four sentences of discussion.
  • Major popular tech employer offered desirable employee $X as a salary because “it was the max the HR department allows for that position.”  He got hired that week at $2X.  All parties — hiring organization, HR, and employee — think they pulled one over on the other participants.
  • Funny story goes here. I now can’t tell you the funny story, because literally two hours before publication someone emailed me for advice about a situation that he believes is incredibly unjust at his company, and it is exactly the funny story to the letter.  Now if I tell you the funny story he might think “Dang, I write Patrick in confidence and it ends up on the blog.”  So, no funny story today.  Suffice it to say that in my old age I treat Dilbert less as farce and more as documentary.
  • “We can’t hire engineers fast enough through our standard processes so, meh, I guess we’ll circumvent them by just tossing $1 million per employee at whomever they currently work for.  Who cares, it isn’t my million.”

When Does A Salary Negotiation Happen?

Only negotiate salary after you have agreement in principle from someone with hiring authority that, if a mutually acceptable compensation can be agreed upon, you will be hired.

This is really, really important because it has direct implications for your negotiating strategy.  First, the company is going to spend a lot of time and effort on getting you to the point of agreement-in-principle.  Pretend you’ve gone through six rounds of interviews.  (You probably won’t if you get hired on informal networks, because all barriers vanish when important people want a deal to get done, but let me give some advice to someone a little less well-situated.)  Do some quick mental math on what that actually cost the company, with reference to “one man-month of an engineer’s time costs $20k” like we discussed earlier.  You’ll quickly reach the conclusion that the company has spent thousands of dollars just talking to you, and that doesn’t even count the thousands they spent deciding to talk to you instead of whoever isn’t in the room right now.  Walking away from the negotiation means that they lose all that investment.  (Yeah, sunk cost fallacy and all, but since people predictably act in this fashion you should, well, predict that they will act in this fashion.)  They really want to reach an agreement with you.

The second implication is that the inner serf worrying “If I even attempt to negotiate this, the deal will fall through” is worrying for nothing.  They’ve got thousands invested in this discussion by this point.  They want you.  The absolute worst outcome of negotiating an offer in good faith is that you will get exactly the contents of that offer.  Let me say that again for emphasis: negotiating never makes (worthwhile) offers worse.  This means you need what political scientists call a commitment strategy: you always, as a matter of policy, negotiate all offers.  (In this wide world I’m sure you can find a company who still makes exploding offers, where you get one yay-or-nay and then the offer is gone.  You have a simple recourse to them: refuse them and deal with people who are willing to be professionals.  You’re not a peasant.  Don’t act like one.)

This also means you do not start negotiating until you already have a Yes-If.  (Yes-If we agree on terms.)  Do not start negotiating from No-But.  (No-But we might hire you anyway if you’re really, really effing cheap.)  You don’t want to work for a No-But for the same reasons that smart employers hate hiring candidates who are a No-But (No-But maybe if not on my team, etc).  If they’re leaning to not hiring you, you will compromise excessively on negotiation to get them to hire you.  Compromising excessively is not the point of the exercise.  It is a seller’s market for talent right now: sell to someone who is happy to buy.

This means that any discussion of compensation prior to hearing Yes-If is premature.  If you’re still at the job interview and you’re talking price you are doing something wrong.  (Read the room: it is entirely possible that you came for a job interview, finished it, and proceeded directly to a salary negotiation.  That’s probably suboptimal, but it is OK.  Just don’t give the employer the option of having the schedule be job interview, salary negotiation, and back to job interview if they discover that you have a spine.)  The ideal resolution to the job interview is for both sides to be optimistic about the arrangement, and then you close with a warm handshake and “I look forward to receiving your offer by, oh, would tomorrow be enough time for you to run the numbers?”

You then have a high likelihood of doing your salary negotiation over email, which is likely to your advantage versus doing it in real time.  Email gives you arbitrary time to prepare your responses.  Especially for engineers, you are likely less disadvantaged by email than you are by having an experienced negotiator talking to you.

The First Rule Is What Everyone Tells You It Is: Never Give A Number First

Every handbook on negotiation and every blog post will tell you not to give a number first.  This advice is almost always right.  It is so right, you have to construct crazy hypotheticals to find edge cases where it would not be right.

For example, if your previous salary was set during the dot-com bubble and you are negotiating after the bubble popped, you might mention it to anchor your price higher such that the step down will be less severe than it would be if you engaged in free negotiations unencumbered by the bubbilicious history.  Does this sound vaguely disreputable to you?  Good.  This vaguely disreputable abuse of history is what every employer asking for salary history, salary range, or desired salary is doing.  They are all using your previous anomalously low salary — a salary which did not reflect your true market worth, because you were young or inexperienced or unskilled at negotiation or working at a different firm or in another line of work entirely — to justify paying you an anomalously low salary in the future.

Never give a number.

“But Patrick,” you cry.  ”I don’t want to be difficult.”  You’re not being difficult.  You’re not doing anything immoral.  You’re not being unprofessional.  They’re businessmen, sometimes they don’t have all the information they would love to have prior to making a decision.  They’ll deal.

They already deal with every employee that they’ve ever had who was not a doormat at negotiations, which includes essentially all of the employees they really value.  Ramit Sethi (more on him later) introduced me to a concept that he calls Competence Triggers: basically, if you have to judge someone’s skill based on a series of brief interactions, you’re going to pattern match their behavior against other people who you like.  When people with hiring authority think of winners, they think of people like them who live and breathe this business thing.  They negotiate things as a matter of course: that is a major portion of the value they bring to the company.  Volunteering a number when asked says the same thing to people with hiring authority that flunking FizzBuzz says to an engineer: this person may be a wonderful snowflake in other regards, but on the thing I care about, they’re catastrophically incompetent.  It will also cause them to retroactively question competencies they’d previously credited you with.

I have literally heard that feedback, in so many words, from folks with whom I’ve had successful business dealings.  (A funny in hindsight story: I cost myself five figures with a single email.  The particulars are boring, but suffice it to say I fairly recently made a wet-behind-the-ears-engineer error in quoting a client.  He noticed.  So did my bank statement.  My bank statement kept quiet, but the client opined that it made him think less of me until we actually got to work together.)

So anyhow, you may well hear reasons why you should give a number.

Objection: “I really need a number to move the process forward.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “I’m more concerned at the moment with talking to you about discovering whether we’re a mutual fit.  If we’re a great fit, then I can be flexible on the numbers with you and you can be flexible on the numbers with me.  If we’re not a great fit, then the numbers are ultimately irrelevant, because your company only hires A players and I only work at roles I would be an A player at.”

(Don’t talk like that normally?  Fine then, talk like yourself, but say substantially the same things.  Engineers overestimate how different we really are from business people: we say “10x engineer,” they say “A player,” but at the end of the day we believe that there are vast differences in productivity between workers.  OK, gut check: is this something we actually believe to be true or just something we wish for?  If it is actually your best guess about the state of reality, that has immediate news-you-can-use implications about how you should conduct your life.)

 

Objection: “This form needs a number.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “Give me git access and I’ll fix it in a jiffy!  both people laugh No, seriously, speaking, I’m more concerned at the moment with discovering whether we’re a mutual fit…  Oh, it’s physically impossible?  Put in $1 then to get the ball rolling, and we’ll circle back to this later.”

 

Objection: “We want to figure out whether you’re an appropriate candidate for the position.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “It’s so important to me that this is a good mutual fit for us.  Let’s talk about why I’m a great fit for this position: I know you’re concerned about $FILL_IN_THE_BLANK.  In addition to my previous successes doing it, I have some great ideas for what I’d do about that if I was working at your company.  Would you like to drill into those or is there another job area you’re more concerned about to start with?”

 

Objection: “I’m sorry, great try at a dodge there, but I just can’t go forward without a number.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say (if you’re an engineer): “Well, you know, I would hate to have to walk away from the negotiation over this.  Working with your company looked it would have been such a wonderful opportunity.  I hear the hiring market is super-tight right now, would you like me to introduce you to other candidates?  Maybe we can shave a couple of months off of you filling this position.”

What you should say (if you’re not an engineer): “Damn, I guess I should have studied engineering.”

What you should say (if you’re a little put out by that comment): “Well, you know, salary is only one component of the total compensation package.  In terms of total compensation, we’re probably looking at something like $FILL_IN_NUMBER_HERE.”  (Suggested calculation: take the package value from your last company and add 5~10%.  If you don’t know how to calculate the value of your compensation package, learn that, but as a rough guesstimate salary + 30 ~ 50% for full-time employees in professional roles and the multiplier tends to scale up as your base salary scales up.)

P.S. I double majored in making things and making things up.  The joking comes from a place of love.  OK, love and schadenfreude, in solution with each other.

Listen To What People Tell You.  Repeat It Back To Them.

Properly run negotiations are not jockeying contests, they’re persuasive exercises.  (We’ll give the company a pass on the “what’s your number?” question because it is an established social ritual that they get one free pass at screwing you.  You still don’t have to cooperate with it, though.)  You know what people find persuasive?  Their own words.  People love their own words.  When you talk to them, you should use their own words.  Seriously, watch the eyes light up.

Did the solicitation for the job say “We are seeking someone with strong skills at scaling traffic in a fast-moving environment”?  Pick out the key words.  Scaling traffic.  Fast-moving environment.  ”Scaling traffic” doesn’t sound like how I’d phrase it if I were writing or speaking for myself, but if you’ve just described your need to me as scaling traffic, by golly I will tell you how great I am at scaling traffic.  Reinterpret or rephrase the (true!) bits of your own story such that it fits the narrative framework which they have conveniently told you that they are going to respond to.  Did you previously work at a small business which was unencumbered by lots of process?  Sounds like a fast-moving environment, right?  Call it exactly that, then.

Micro-tip: Take notes during job interviews and salary negotiations.  It’s easy: go to the convenience store before the job interview, buy a writing instrument and a $1 notebook, jot down occasional notes when appropriate.

Can I do that?!  Of course you can.  Do you know anyone who you’ve ever thought “Man, I thought they were competent, but then it turned out they had a notebook so I had to write them off?”  No.  Taking notes says “I’m attentive and detail-oriented and I care about what you say.”  (Make sure you can take notes without playing with your pen or otherwise appearing to fidget.)  In terms of specific things that should get your pen moving, among others, I would focus on specific words they use and concerns they have so that you can come back to them later in the conversation.  Numbers are another good thing to hit the notebook, because numbers should only ever trend in a direction of “Better to you,” so you don’t want to do something stupid like saying “So how many days of vacation was that again?” and let a 24 suddenly become a 20.  (You might think “I’m going to write down the offer so I have proof of it for later.”  Get offers written, that goes hopefully without saying, but get it written by them and/or follow-up the discussion with an email recapping the important points and asking if you understood them correctly.  Your notes will not convince their HR apparatus to honor the agreement in event of a retroactive miscommunication, but an email from their decisionmaker likely will.)

People say the damnedest things.  For example, someone might spontaneously volunteer during a job interview that they’ve been interviewing for the position for six months.  (None of my clients would ever say that, of course, but then again one would hope none of their consultants would chop five figures off their own invoice with an email.)  If they say the position has been open for six months, take a note of that.  During the salary negotiation, if they have a pricing objection, one of your first responses should be “I appreciate that this is a little more money than you might have been thinking about, but this is an opportunity to get this position filled without delaying your business by another six months.  What is the value of that six months of execution to you?”  (Conversely, don’t say stupid things during job interviews such as “I need this job because…”  You never need a job.  Being needy means that the party who is not needy has automatic leverage over you: your BATNA to the negotiation is very poor.  Instead of being needy, aim for “I’m enthusiastic about the opportunity with working with you, assuming we can come to mutually satisfactory terms.”)

Micro-tip: Notice how often I say “We” and variations on “mutual win.”  Those work pretty well.  The only thing better than “We” is “You” (and variants), because people care a heck of a lot more about their problems than about your problems.  (This advice is stolen shamelessly from Dale Carnegie.)  This means that a) you should talk about their problems, concerns, and wishes and b) you should guard against your own natural tendency to bring up irrelevant things like your own problems, which typically will not help you sell the decisionmaker on adopting the mutual win you’re proposing.  Similarly, I generally try to phrase things positively rather than score debating points.  (“You just said X, but that was contradicted by your earlier statement Y, which means…” wins debating points but does not win friends and influence people.  You might try something like “Good good, but taking into account your earlier concerns about Y…”)

Research, Research, Research

Many people will tell you that you should familiarize yourself with the approximate salary range for the position in your region.  This advice is easy to act on (go to a salary aggregation site, guess what “the position” is, pray that this gives you a better number than rand(40000,120000)), but it leaves a lot to be desired.  It is 2012.  Facebook and LinkedIn exist.  You should, before any job interview, have intimate knowledge of the target company.  Prospective peers within the company are one obvious way to get it.  So are ex-employees, folks who’ve had dealings with them professionally, etc.  Key things you want to learn:

  • What do they value?
  • Who do they value within the company?  (Roles?  Titles?  Groups?)
  • What does the career path look like for successful people within the company?
  • Roughly speaking, how generous are they with regard to axes that you care about?
  • Do they have any compensation levers which are anomalously easy to operate?  (For example, if you asked around, you might hear a few people say that a particular firm pushes back modestly on out-of-band increases in salary but they’ll give in-the-money option grants like candy.)
  • All the fuzzy stuff: what’s the corporate culture like?  Yadda yadda.

You can even bring a lot of these questions to the job interview, which is (again) prior to the negotiation.  (Maybe not “So are you guys tightwads?” but culture-esque questions like “What are the projects this company thinks are really key to its future and how would a motivated person go about getting on them?” are both a) totally fair game and b) will win you brownie points just for asking.  Similarly, a lot of employees will, out of company loyalty, attempt to sell you on taking the job with the company by trading you very useful information.)

The more you know, the more options you have when doing negotiation, because you’ll have more things and more motivational things which you can offer in exchange for things you want.  It will also help you avoid making mistakes like e.g. getting into a rigid classification system where the classification you’re aiming at will make forward advancement towards your goals very difficult.  (Example: there are some companies where Product and QA are run like separate fiefdoms which haven’t forgotten the most recent war, and in those companies getting hired as an engineer may not be a career enhancing move if you like making things for a living.  There are other companies where people cross-function in those responsibilities all the time and applying for a job advertising as “Support Engineer” makes lateral moves onto customer-facing projects trivial.  You can find which one you’re applying to by taking any engineer out for coffee.)

New Information Is Valuable And Can Be Traded For Things You Want

There was a post recently on Hacker News about someone’s experience with a job offer from Google.  They wanted more money.  The recruiters offered to think it over, and came back with the reply that Google’s food benefit was worth a significant amount of money, with a calculation to back it up.  That is a pretty brilliant reply.  Google’s food benefit is about as old as the company.  Approximately all people wanting to work at Google are aware of its existence.  However, the explicit calculation of what it is worth is new, so if you bring up that calculation, by implication you’re offering newly found value to the negotiation.  This successfully convinces people that they didn’t really need that extra money.  It is so successful at this that Google recruiters apparently have this entire interaction scripted, since multiple people report having the exact same experience.

You should steal this tactic.  You are an expert in your own skill set, life story, and (ideally) value you can create for the company.  However, the person you are talking to is not.  If they ever resist about something which you want, consider reaching into the treasure chest that they are buying mostly blind and revealing one of the many glittering jewels inside.  They are going to get them all anyhow if they buy the chest, but each one you bring out decreases the perceived risk of buying it and therefor increases its perceived value.

Company: We can’t see our way to $88,000.

Applicant: Well, I know you do a significant amount of business with your online store.  At my last company, I increased sales by 3% by $YADDA_YADDA.  What would a 1% increase in sales be worth to you?

Company: Well, I don’t have that figure in front of me, but…

Applicant: Would it be safe to say “millions of dollars”?

Company: That sounds about right, yeah.

Applicant: Great, I can’t wait to get started.  Getting me that extra $4,000 would make this a much easier decision.  Considering that this is conceivably worth millions to you, we’d be silly not to do business with each other.

Company: I’ll see what I can do.

Applicant: Let me help give you some options!  [See below.]

(This hypothetical applicant is doing well on the negotiation but apparently needs to do more research on what conversion optimization specialists can get away with charging these days.  Here, let me help: six figure salary with all the usual perks as an employee, “senior engineer project rates” through “you might not believe me if I told you” as a consultant.)

Anyhow, simply by bringing attention to something which was hopefully already in bold print on their resume, they just increased their perceived value to the company, thus justifying the company moving a lever which (again) the company isn’t really sensitive to at the end of the day.

You Have A Multi-Dimensional Preference Set.  Use It.

Don’t overly focus on your salary number.  It is important (of course), but there are many parts of your compensation package, and many more things that you value.  Should you and the other party reach an impasse on any part of it, offer to table that part of the discussion (to be returned to later) and bring up a different topic.  You can then trade improvements for concessions (or apparent concessions) on the earlier topic.

Employer: “We were thinking $80,000.”

Applicant: “$80,000 is interesting (*) but not quite where we need to be to get this done.  Do you have any flexibility on that number?”

Employer: “I think I can convince HR to approve $84,000 but that is the best I can do.”

Applicant: “I appreciate that.  $84,000, huh.  Well, it isn’t quite what I had in mind, but the right package offer could make that attractive.  How much vacation comes with the package?”

Employer: “20 days a year.”

Applicant: “If you could do 24 days a year, I could compromise on $84,000.”

Employer: “I think I can do that.”

For those keeping score at home: the applicant never gives up anything but the employer will walk away feeling he got a good deal.

* Micro-tip: “Interesting” is a wonderful word: it is positive and non-commital at the same time.  If they tell you a number, tell them it is an “interesting” number, not a “wonderful” number.

Hoping around the offer also helps you defuse common negotiating tactics like “I have to go to $EXTERNAL_AUTHORITY to get approval of that.”  (This is in the negotiation playbook, because it works well: it injects an automatic delay in the process, and gives you a scapegoat for refusing a request while not being guilty of the refusal yourself.  You should strongly consider having an $EXTERNAL_AUTHORITY of your own.  Significant others work well.  Note that in the US your would-be employer is legally prohibited from breathing about the subject of your marital status, so something like “We’ll have to talk that over” or “That sounds reasonable, but I’ll have to run it by the family” has the dual virtues of being a socially acceptable reason to delay any major decision while also being equally available to unattached young’uns.  I talk shop with my family all the time.  I’ll certainly continue discussing employment with my family after it includes my fiancee, too.)

Anyhow, say your decisionmaker says that approving deviations from the company’s salary structure is outside of his discretion and those evil ogres in HR will likely deny his request.  That’s fine.  Express sympathy with him, because he just said he wants to give you more but can’t, then refocus the discussion on things which are within his personal authority.  (Vacation days, work hours, project assignments, travel opportunities, professional development opportunities, and the like are good areas to probe at.)  You can then use the unspent “You wanted to do something nice for me” obligation which he just acknowledged on one of the things which he has authority to grant you.

For Your Further Perusal

I’m deeply indebted to a few buddies of mine, principally Thomas at Matasano and Ramit Sethi, for teaching me to be less of a doormat in terms of negotiation.  Thomas has forgotten more than I’ll ever know about doing negotiations with clients.  Check out HNSearch.com with [tptacek negotiation] for some good advice, or (if you’re in Chicago) take him out to coffee.  By the way, if you’re an engineer and want to practice salary negotiation, Matasano is hiring and a great place to work for.

Ramit is extraordinarily persuasive about how psychology influences how people act in negotiations, and how a deeper understanding of this gives you better preparation than trying to focus on particular tactics.  When I told him that I was finally going to blog about one of our dinner conversations (you’re reading it), he said he’d give my readers a free video on negotiation tactics and the psychology underlying them in return for your email address.  The video is good and elaborates on some subjects that I haven’t covered, plus I know some people find them easier to learn from than lots of text, so that strikes me as a very fair offer. You’ll also get Ramit’s emails, which have never wasted my time.

47 Comments

Bingo Card Creator (and etcetera) Year In Review 2011

I’m Patrick McKenzie (patio11 on the Internets) and for the last several years I’ve run a small software company.  My first product was Bingo Card Creator, my current product focus is Appointment Reminder, and I do occasional consulting for a variety of clients, mostly on helping them sell more of their software over the Internet.

Traditionally, right before Christmas every year I release an annual report.  See, for example, 2006, 2007, 2008, 2009, and 2010.  (Crikey, have I really been doing this for that long?)  I’ve also traditionally published live stats for Bingo Card Creator, but not my other lines of business.

Writing the annual report is partially to keep me grounded, partially to talk through my thoughts on the year and goals for next year, and partially to (hopefully) give other folks ideas that they can use in their own businesses.  I hope you find it interesting or, at the very least, mildly amusing.

Obligatory disclaimers: Assume any statistics that I give are “roughly accurate, to the best of my knowledge, at the time this report was written.”  There are still a few weeks left in the year.  Sales are typically low in the last two weeks, but the exact timing of credit card charges can cause a bit of jitter in the December stats.  From past experience, I have a high degree of certainty that there are about $1,000 or $2,000 of expenses (across all lines of business) which aren’t in the bookkeeping  system yet and won’t be until I sit down in March and check things for taxes.

Capsule summary: Best year ever, by a lot.  Broke $100,000 in sales for the first time and increased total profits to ~$70k.  2012 has inflection points coming for life and the business.

The Year In Brief

I put Bingo Card Creator into maintenance mode for approximately 48 weeks out of 2011: I only answered emails and kept systems running, but took no action to improve the product or marketing.  (The other four weeks I tried a few minor things out.)  This was, theoretically, supposed to free me to spend most of my efforts on Appointment Reminder…

… but that didn’t end up happening.  For a variety of reasons, most of my focus business-wise went into consulting.  Although I technically only did about 10 weeks of consulting during the year, I spent quite a bit of overhead time on e.g. arranging deals which ended up falling through, arranging the deals which did actually go through, and doing general promotion activities like speaking at conferences.  (I had the opportunity to speak at about a half dozen conferences this year, and assorted other events.  It is great fun, but since I generally have to fly to America for them, they tend to munch a full week out of my schedule each.  I spent almost three months of the year in the US, doing a combination of family events, consulting, prospecting, speaking, and meeting some Internet buddies to discuss plans for later.)

I also lost two solid months due to dealing with legal issues, mostly centering around Immigration.  I’d love to fill you in on the nitty-gritty, but have been asked not to by people close to the situation.  Suffice it to say that I was a shoe-in for a Japanese visa back when I worked at a large megacorp, was not a shoe-in for a visa when doing my own thing, and had a very hairy experience with getting them to approve me as a “self-employed engineering consultant.”  Tips of the hat to my Japanese clients, particularly Makeleaps / Webnet IT and myGengo, whose support was instrumental in getting Immigration to approve my renewal.

Despite not having nearly as much time to work on Appointment Reminder as I would have liked, I did manage to firm up its technical underpinnings, add new features requested by clients, and do a small amount of work marketing it.  I hope to make that more of my focus in 2012.

Bingo Card Creator

Despite being in maintenance mode, BCC continued performing like a trooper.  People always ask “Could you afford to live on it only?” and the answer is “Yes, but barely, and it would require a lifestyle adjustment, mostly in the don’t-fly-across-the-Pacific-so-often department.”  BCC did not meet the numeric goals that I had for the year.

Stats:

Sales: 1,539 (up 6% from last year’s 1,451)

Refunds: 14 (down from 22 last year, to .9% of sales from 1.5%)

Sales Net Of Refunds: $45,479.93 (up 5% from $43,398.55)

Expenses: $22,560.00 (up from $18,287.93, but largely just due to an accounting issue — I can’t split costs in my homegrown bookkeeping software, so the ~$3,000 I paid for servers for AR is hiding in that number)

Profits: $22,919.93 (see above accounting issue, essentially flat from last year’s $25,904.66)

Wage per hour: Let’s see, ~15 hours of programming, 20 minutes a week on customer support…  about $700 an hour.  Not too bad.

 

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 821k (up from 777k)

Unique visitors: 670k (up from 655k)

Page views: 2.9 million (up from 2.7 million)

Traffic sources of note: Google (46%), AdWords (18%), Binghoo (13%)

Trial signups for online version: 82,000 (up from 72,000)

Approximate online trial to purchase conversion rate: 1.8%

 

Narrative Version:

Aside from kicking up AdWords spend modestly (to no good effect) and running a few A/B tests, nothing really substantial happened with Bingo Card Creator this year.  I lost probably $1,000 to $2,000 of sales when the site crashed right during the middle of the Halloween rush for ~9 hours while I was on an airplane.  That was a little disappointing, but while it broke my candy budget it won’t exactly put me in the poorhouse.

Projections that BCC would continue to grow despite not being actively worked on turned out to be totally wrong.  I forecast 50% growth, reasoning “Hey, most of the systems work pretty much without my intervention, so I think the overall growth of the Internet plus a few A/B test means, oh, 50% or so.”  It mostly tread water.  I’m not hugely disappointed.

 

What Went Right:

  • Not having to work hardly at all for it.
  • Aside from the Halloween crash, the system was largely stable for the year.  I think I got woken up by the automated alarm maybe once.
  • SEO, AdWords, email marketing, and the usual scalable marketing stuff continued to be my bread and butter even when I was too lazy to actually cut and butter bread.

What Didn’t Work So Well:

  • Crashing on the third busiest day of the year, in such a way that it depresses my AdWords campaigns for the first and second busiest days of the year.
  • I integrated Stripe and expected a huge lift in conversions for going from Paypal to a simple CC-based payment system.  I tested this extensively in A/B tests.  I love everything about the Stripe system, but I have no evidence for “Stripe is better than Paypal/Google Checkout”, “Stripe/Paypal/Google Checkout is better than Paypal / Google Checkout”, etc etc.  That said, it might be something as simple as my buttons being ugly.  I’ll probably take a whack at it in the future, or better yet, have my designer take a whack at it.

Consulting

I did a few weeks of consulting this year, for several different clients.  Mostly, I do my engineering / marketing shtick for software companies, although some of my clients have been a wee bit farther afield.  I wrote up a fairly typical engagement with Fog Creek.  That one was a mutual success and we’ll continue to work together in the future.  (To the best of my knowledge, all of my consulting clients are happy with my work.)

One thing I’m going to do differently in the future is to work for less clients.  Don’t get me wrong: I love all my clients.  I was privileged to work with them.  However, it takes approximately X units of work to set up an engagement with a previous satisfied customer, 5X units of work to get a new prospect to the go/don’t-go decision on a new engagement, and I generally have to get three to four prospects to that point to actually wind up with a signed contract.  As my buddy Thomas at Matasano says, “That is life in the big leagues.”  However, since I’m not in a position where 100% utilization is a huge overriding goal of mine, I don’t need to keep the new prospect pipeline totally full… so I’m probably going to cut back on it quite a bit in 2012.  I’ll continue doing follow-up engagements for established clients where it makes mutual sense to do so, and I’m still of course available for interesting projects, but I’m not going to be doing six-week fly-across-America-four-times tours to drum up new business.

The following numbers are approximations only.  NDAs and having the sense God gave a tadpole constrain me from revealing my “going rate.”

Consulting sales: $55,000

Consulting expenses: $13,000  (mostly hotels and airfare for prospecting, which I pay for out of pocket.)

What Went Right:

  • Client selection.  I was, again, privileged to work for people who have interesting businesses, problems that I could make substantial contributions on, and the willingness and ability to pay all invoices in a timely fashion.
  • Raising rates.  My first guesstimate at my rate, back in 2010, was $X.  It turns out that I could do just about as much work as I wanted regardless of whether I charged $X, $2X, or $5X.  As a result, I typically quote fairly high rates and mostly stick with them, unless there is another reason I really, really want an engagement to happen.

What Didn’t Work So Well:

  • Disorganization.  At one point I was juggling something like five simultaneous proposals out while preparing for three conferences, two engagements, and six weeks of travel.  It got so bad that I showed up at a city once and checked at the airport for where I was staying, quickly seeing that I mistimed a conference by three days and thus had no hotel booked, booking a hotel from the taxi, and then arriving at the hotel to recheck my schedule and discover that I had used the previous year’s schedule and was actually simultaneously at a different hotel across Brooklyn.  (Shoutout to the Brooklyn Beta guys for saving me from my own stupidity that week.)  There were multiple points in the year where I found myself wishing for either a boss or a secretary or somebody to just say “Show up to X on Monday and Do Stuff and all the stuff that is not Stuff will be taken care of.”  My occasional slipups in dealing with the demands of a growing business caused me to drop balls in ways that were sometimes client-visible, too.  This is a major part of the motivation for cutting back next year.  (There is Plan B, of course: hire folks to do either the execution or the admin and take whichever part they’re not doing, but I don’t think I’m moving in that direction.)
  • Too much work!  Largely due to overhead and travel, plus the outsize distraction generated by the same, consulting munched a heck of a lot more time than I thought it was going to.  I wanted to have a solid eight months of the year to work on AR.  I think I probably got maybe two.

 

Appointment Reminder

I launched Appointment Reminder last December, with the goal of having approximately 200 customers and $10k in monthly recurring revenue by now.  I had planned on focusing for most of 2011 on marketing and selling it to more businesses.  That largely didn’t happen, but since I got the fundamentals of my SEO strategy in place (while largely ignoring the modestly more advanced content creation / etc that runs BCC and that I usually help clients with), the business grew despite my best efforts at totally neglecting it to focus on consulting and not getting deported.

AR has been hanging around at a crossroads for a while now.  There are two very different trajectories it could go down.  In one, I grow it organically, and it grows into a modestly profitable software business which will provide handsomely for my family and (in the fairly near future) employees.  In two, I take outside investment, and attempt to grow as quickly as possible to $N million a year in revenue, at which point options would include either a) selling to one of the larger players in the small business software space or b) continued operations at scale with a focus on growth.  Luckily, I have  the luxury of waiting on making that decision: my runway is infinite, the market opportunity is only getting bigger, and the perceived value of my involvement with a startup among investors does not appear to be depreciating.

This is one of the reasons I can’t be as open as I would like to be about the current status of the business.  BCC has essentially no secrets, and would not really benefit from having them, as — aside from elementary school English teachers — there is nobody out there who has something I want for BCC.  However, if I hypothetically wanted to take investment, then accredited investors suddenly have something I want very much and having secrets about AR gives me something with which to trade to get it.  (It is similar to not putting prices on an Enterprise Software website.  You can trivially get them, but the price of getting them is giving a salesman permission to give you the spiel.  Similarly, folks who ask about AR’s numbers these days are generally asking in the hopes that they eventually receive a phone call asking them for a check.)

The other reason I can’t talk about AR numbers so much is that I radically underestimated how important the enterprise market would be to the business, and you can’t spell enterprise without NDA.

So: I wanted to have two hundred customers by now.  For the publicly available plans, I currently have a few dozen paying customers.  There are ways to get things from me that don’t involve paying the numbers on the Pricing page.

AR is modestly profitable — it covers all of its own costs.  I plow most of the money it generates back into the business, though, rather than taking distributions.  For example, I’m now about 95% certain that I will have significant contractor or employee involvement on it in 2012.

Revenue: Undisclosed

Expenses: Undisclosed (very modest ongoing expenses, reinvested most profits)

Profits: I took about $5k just to have a number that would minimize disbelief at the tax office.

What Worked Right:

  • Twilio.  The Twilio API and service have been unalloyed epic wins for Appointment Reminder.  I had zero disruptions in service attributable to them, their customer support has been fast, responsive, and technically savvy (even helping me debug my own code at points), and they’ve been very supportive of me.  Plus they have these awesome red track jackets that they keep sending me, which you’ve probably seen if you’ve seen a picture of me doing any talk this year.  (I actually wear them mostly because I love the color red, but apparently I wear them so often that folks at the Fog Creek office thought the Twilio logo was my logo.)
  • Sendgrid: It’s like Twilio, except for email.  Great service.  No red jackets.
  • Unit testing & staging servers.  I am gradually getting more sophisticated in my engineering practices, and have been ramping up my testing activities since starting to code AR.  It has transformed the way that I do development, for the better, and made it easier to respond to customer requests to change things while decreasing the number of problems I have caused.  Total win.  See my presentation at TwilioConf for examples of the specific ways I use it for AR.
  • Exact match domain names.  ”Hey Patrick, how is it that with no marketing budget and nearly no marketing work you rank #1 for [appointment reminder]?”  I told everybody that I was buying the .org specifically because that would happen but apparently folks didn’t believe me.
  • Using the self-service site as lead generation for enterprise sales.  Fairly self explanatory.
  • The service itself: AR solves a clear customer need, and my customers are raving fans of it.  There exist many services businesses which incur hundreds in direct costs and thousands in forgone revenue for a single missed appointment.  (Think, say, an HVAC company which sends a three-man team of tradesmen out to your house to replace your heater, which is a $2,000+ job, only to discover that you aren’t home to let them in.)  One of my customers reports that just the delta in no-shows since starting to use AR would pay for his mortgage and his daughter’s college education.  Many of my other customers report that their office managers, who previously did telephone reminder calls manually, are ecstatic to not have to do them any more.  Customer retention among folks who actually use the system (as opposed to signing up, doing a test call, and forgetting about it) is virtually 100%.
  • Talking to smart people for advice: Since I’ve been going back and forth on the investment question, I talked to a lot of entrepreneurs and investors whose opinions I respect.  I really appreciate their feedback, which ranged from “Are you kidding?  You’d hate it.” to “I want to invest in you, but realistically, you would lose nothing by waiting until you are sure.” to “Best decision I ever made.” and helpfully included a lot of actionable advice on how to do things in the meanwhile such that options remain open.

What Didn’t Work So Well:

  • Catastrophic engineering failures.  I had one combination outage/catastrophic failure in February (the details are recounted in that TwilioConf presentation) and a ~3 day period of sporadically degraded operations after my move to Rackspace, which I finalized over the Thanksgiving holiday.  Both of those were my fault, for architecting the system in a way which did not gracefully handle its multiple moving parts getting out-of-sync with each other.  I’ve since done significant work on making it more stable.  (Overall reliability for the year has been excellent, but those periods were easily the most stressed I’ve ever been about any business issue.)
  • Lack of focus: I’ve been commenting above on this, so I won’t belabor the issue, but I really didn’t get to work on AR as much as I wanted.
  • Enterprise sales: I’m actually fairly decent at Enterprise Sales, and am working with someone in the industry who has a deep Rolodex among folks who would be great candidates for AR, but (partly due to the focus issue and partly due to my own comfort level) I didn’t put nearly enough effort towards it this year.  What I should honestly do is go to a conference some time, prospect like a madman, and then make following up on those leads my only job until I’ve got contracts signed.  (The prices for enterprise SaaS make this very economically viable.)

Goals For 2012

Bingo Card Creator

  • I’d be happy with continued flatness ($~30k profits on $50k sales), maybe.  It isn’t the source of growth for my business anymore.
  • Continue using it as a laboratory for weird ideas I have on conversion optimization.
  • Don’t break it during Halloween.

Consulting

  • Do less work prospecting for new clients.
  • Do more work for existing clients.
  • Modestly increase billings, if that makes sense for where my overall business is.  (If I take external investment in AR, that will likely require shuttering the consulting business.)

Appointment Reminder

  • Figure out whether I want to take investment or not.  If so, do so.
  • Convince Keith (who I do my podcast with) to work with me, if possible.  (Don’t worry, he knows this is on the agenda.  We’re best friends.)
  • See about transferring responsibility for the engineering (particularly front-end) side of things so I can focus on marketing/sales.
  • 10x current sales numbers.  That seems to be a fairly safe bet regardless of whether I shoot for a small business or for a high-growth business.  (1,000x-ing would be another story.)

A personal note: The last 3,300 words ultimately matter much, much less than the next 3: she said yes.  We’re announcing to our family on Christmas, as per our family tradition.

27 Comments

Productizing Twilio Applications

This post includes video, slides, and a full-text writeup. I recommend bookmarking it if you’re on an iPhone right now.

I make extensive use of Twilio (a platform company that lets you do telephony with an API) in running Appointment Reminder, my core product focus at the moment.  (Wait around a day or two and I’ll tell you a bit about how it is doing in my annual end-of-the-year wrapup.)

Twilio has a very passionate developer community and fairly good documentation on their website, but I’ve sometimes been frustrated at it, because it teaches you the bare minimum to get phones ringing.  That is truly a wonderful thing and a necessary step to building a telephony application.  However, if you continue developing your application in the way that the Quick Start guides suggest, you will routinely run into problems regarding testing your code, maintaining it, securing it, and generally providing the best possible experience to your customers and the people they are calling.

I have a wee bit more than a year of practical experience with a Twilio application in production, so I went to TwilioConf and did a presentation about how to “productize” Twilio applications: to take them past the “cool weekend hack” stage and make them production-ready.  Twilio has graciously released videos of many of the presentations at TwilioConf, so I thought I’d write up my presentation for the benefit of folks who were not at the conference.

The Video (~30 Minutes)

Twilio Conference 2011: Patrick McKenzie – Productizing Twilio Apps from Twilio on Vimeo.

The Presentation (40 Slides)

The Writeup

 

Why I Think Twilio Will Take Over The World

(This was not actually in the presentation, because I didn’t have enough time for it, but I sincerely believe it and want to publish it somewhere.)

I think Twilio is, far and away, the most exciting technology I’ve ever worked with.  The world needs cat photos, local coupons, and mobifotosocial games, too, but it needs good telephony systems a lot more, as evidenced by companies paying billions of dollars for them.

Additionally, Twilio is the nascent, embryonic form of the first Internet that a billion people are going to have access to, because Twilio turns every phone into a smartphone.  The end-game for Zynga’s take-over-the-world vision is the human race slaved to artificial dopamine treadmills.  The endgame for Twilio’s vision is that every $2 handset in Africa is the moral equivalent of an iPhone.  I know which future I want to support.

Smartphones aren’t smart because of anything on the phones themselves, they’re smart because they speak HTTP and thus get always-on access to a universe of applications which are improving constantly.  Twilio radically reduces the amount of hardware support a phone needs to speak HTTP — it retroactively upgrades every phone in the world to do so.  After that, all you need is the application logic.  And what application logic there is — because the applications live on web servers, they have access to all the wonderful infrastructure built to run the Internet, from APIs that let you get Highly Consequential Data like e.g. weather reports or stock prices or whatever, to easy integration with systems which were never built to have a phone operating as part of them.

You can’t swing a stick in a business without hitting a problem which a phone application makes great sense for.  I filled up three pages of a notebook with them in just a week after being exposed to Twilio for the first time.  Order status checking for phone/fax/mail orders.  Integrated CRMs for phone customer service representatives.  Flight information.  Bank balances.  Server monitoring.  Appointment reminders.  Restaurant reservations.  Local search.  Loyalty programs.  Time card systems.  Retail/service employee support systems.  Shift management.  The list goes on and on and on.

Seriously, start writing Twilio apps.

What This Presentation Will Actually Cover

I’m tremendously optimistic about the futures of Twilio and the eventual futures of companies which make Twilio applications, but I’m pessimistic about your immediate future as an engineer writing a Twilio app, because it is going to be filled with pain.  You’re probably going to make some choices which will cause you and your customers intense amounts of suffering.  I’ve already done several of them, so use me as the inoculatory cowpox and avoid dying.

Crying In A Cold, Dark Room

Back in February 2011, I moved from my previous apartment to my current house.  I unwisely decided to push a trivial code change prior to boxing things up.  This trivial code change did not immediately take down the server, but did cause one component (queue worker processes) to fail some hours later.  The most immediate consequence of this was that outgoing appointment reminder phone calls / SMSes / emails failed to go out.  Since I was busy moving, I did not notice the automated messages about this failure.

When I discovered the failure (8 hours into customer-visible downtime), I panicked.  Rather than reasoning through what had happened, I reverted the code change and pushed reset on the queue worker processes.  This worked, and the queue quickly went from 2,000 pending jobs to 0 pending jobs.  I then went to bed.

At roughly 3 AM, I woke up with a vague feeling of unease about how I had handled the situation, and checked my email.  My customers (small businesses using AR to talk to their clients) had left several incensed messages about how their client had reported receiving dozens of unceasing calls on the behalf of their business, in a row, at 7:30 PM the night before (right after I had restarted the queue workers).

Here was the error: my application assumed that the queue would always be almost clear, because queue workers operate continuously.  A cron job was checking the DB every 5 minutes to see whether a particular client had been contacted about her appointment yet.  If she hadn’t, the cron job pushed another job to the queue to make the phone call / SMS / email.  When the queue came back up, each client received approximately ~100 queued events simultaneously.  These did not themselves check, at the start of the job, whether the job was still valid, because the application assumed that the cron job would only schedule valid reminder requests and not execute 100 times in between queue clearings.

This resulted in approximately 15 people receiving a total of 600 phone calls, 400 SMSes, and 200 emails, in approximately a 5 minute period of time.

There are a variety of ways I could have avoided causing this problem for my customers:

  1. Don’t make code changes prior to planned unavailability, even if they look trivial.
  2. Don’t ever leave your phone that gets emergency messages out of your pocket.
  3. Switch to idempotent queues, so that adding the same job multiple times does not result in multiple copies of the job.
  4. Add per-client rate limits, so that application logic errors don’t cause runaway contact attempts.
  5. Add failsafes for historically unprecedented levels of activity, shutting down the system until I could manually check it for correctness.

Testing Twilio Applications

Unit testing and integration testing are virtually required to produce production-quality Twilio applications, and will make it much less likely for you to create catastrophically bad bugs in production.  Unfortunately, testing Twilio applications is much harder than testing traditional CRUD web applications, because of how TWIML is different than HTML (in terms of how minor syntax errors actually cause business problems), how it is not easy to replicate telephone operation in integration testing, and because Twilio sometimes has poor separation of concerns between the MVC of a web application, the Twilio helper library, and the Twilio service itself.

Twilio testing is inherently dangerous, because non-production environments (testing, staging, development, etc) could conceivably generate actual, real-world phone calls to phone numbers which were in your database but not actually under your control.  The first and most important tip I have for Twilio testing is to make it explicitly impossible to contact anyone not on a whitelist from code when you’re not in production.  I have a quick snippet that I put in a Rails initializer which monkeypatches my Twilio library to force it to only make phone calls or SMSes to whitelisted numbers.  (I don’t suggest actually re-using this code, particularly as you may not be using Rails or the same Twilio library that I am using, but you can reuse the idea of enforcing safety in non-production environments.)

 

 

A lot of Twilio testing will, unfortunately, require manual button-pressing (or scripts which simulate button-pressing on a telephone).  This is easier to accomplish if you can expose your local development machine to the actual Internet.  There are strong security reasons why you don’t want to do this but, if you’re comfortable with doing it, LocalTunnel is a great way to actually accomplish it.

Also see the section below on Modeling Phone Calls, because it will make Twilio phone trees and call logic much more tractable to unit testing.

You Should Have A Staging Server

A staging server is just a copy of the entire production system minus the actual customers.  (You probably shouldn’t put production data on it, because staging systems are designed to break and as a result they may leak data through e.g. SQL injections.  This is an easy way to lose your DB.)  You should use firewalls and/or server rules to make the staging server inaccessible to the world (aside from Twilio and any other APIs which need to access your site for it to work), but assume you will botch this.

Staging servers are virtually mandatory for Twilio applications, because Twilio apps can fail in ways which will not be detected until they are actually accessed over the Internet.  For example, even with unit and integration testing, failing to properly deploy all audio assets (MP3 files, etc) will cause Twilio to throw hard, customer-visible errors in production.  I have automated systems which check for this now, but since that isn’t an exhaustive list of things that can go wrong in production, part of my workflow for deploying all changes on Twilio is to push them to the staging server first, and then having automated scripts exercise the core functionality of the application and ensure that it continues to work.

How To Model Phone Calls

Twilio Quick Start guides generally don’t suggest modeling phone calls explicitly, instead relying on just taking user input and doing if/then or switch statements on it.  This is ineffective for non-trivial use cases, because as the application logic gets more complicated, it will tend to accumulate lots of technical debt, be hard to visually verify for correctness, and be extremely difficult to automatically test.  Instead, you should model Twilio calls as state machines.  I am a big fan of state_machine in the Ruby world.

I’ll skip the CS201 description of what a state machine actually is.  If you didn’t take that course, Google is your friend.

You should model calls such that they start in a predictable state and user input moves the call between states, causing side effects of a) running any business logic required and b) outputting Twiml to Twilio, to continue driving the call.  This lets you replace case statements with a lot of parallel structure with well-defined transition tables within the call models.  Those models are then trivial to unit test.  Additionally, adopting coding conventions such as “the Twiml to be executed at a given state is always state_name.xml and any audio assets go in /foo/bar/state_name/*.mp3 “allows you to write trivial code which will test for their presence, which will save you from having to manually go through the entire phone tree every time on the staging server to verify that refactoring didn’t break anything.

Additionally, state machines are much easier to reason over than masses of spaghetti code which case statements tend to produce.  For example, consider the following code, which attempts to implement the phone prompt “Press 1 to confirm your appointment, press 2 to cancel your appointment, press 3 to ask for us to contact you about your appointment.”  Spot the bug.

There are actually over six bugs in that code, above the trivial ones you probably saw with numbers not lining up to action names:

  • The Twilio API will pass this code params[:Digits] not params[:digits], which will cause an error that won’t be caught until you physically pick up the phone.
  • The comparisons of params[:digits] with integers will fail, because it includes string representations of numbers.
  • There are several mistakes in mapping numbers to actions.
  • One of the action names is spelled improperly.

These are very easy to miss because our brains get lulled into a false sense of security by parallel structure.  Instead, the model should be taking care  of that mapping between user input and state transitions.  This would radically simplify the code and make the controller virtually failure-free, while letting the model exhaustively unit-test possible user input, expected transitions, and business logic side effects.

State machines might seem like an unnecessary complication when you only have three branches in your code, but production Twilio applications can get very, very complicated.  Here is a state diagram from Appointment Reminder.  You do not want to have to test these transitions manually!

Dealing With Answering Machines

Dealing with the case where the phone calls is answered by an answering machine or voicemail system has been the hardest application design problem for me in doing outgoing phone calls in Twilio.  The documentation suggests using an IfMachine feature, which will cause Twilio to listen to a few seconds of the phone call prior to executing your code.  They do some opaque AI magic to determine whether the entity speaking (or not speaking) in that interval is a machine or not, and tell your application whether it is talking to a machine or a human.  In my experience, this has error rates in the 20% region, and many customers intensely dislike the gap of dead air at the start of their phone calls.  Also, if the heuristic improperly detects the beep, your message will start playing early, causing the recording to be cut off in the middle.

There are several ways you could attempt to deal with this:

  • Ignore the issue and treat both machines and humans the same.  This will produce the optimal result for humans, but your system will be virtually unusable when it gets a machine.  (This happens very frequently in my use case.)
  • Force a keypress (“1″) prior to playing your message, then give all users the same message.  This will force most machines to start recording immediately, stopping the cut-off-in-the-middle problem but annoying some clients.
  • Play instructions such as “This is an automated message from $COMPANY.  To hear it, press 1.”  Assume that anyone who doesn’t press 1 in 5 seconds is a machine and play the machine message.  If they interact with the call, play the human message.  This is my preferred solution (although not actually implemented in AR publicly yet, because customers don’t really grok this issue until it bites them personally).

There is one particular problem with recording messages on answering machines: if you give a user instructions such as “Press 1 to confirm your message” and they follow that instruction when listening to their voicemail, that keypress will not be caught by your application, it will be caught by their voicemail system, with unpredictable results (such as deleting the message) and an absolute certainty of not doing what your keypress would normally do.  Users do not understand why this happens.  They expect your instructions to them to work.

Securing Twilio Applications

Twilio applications have a superset of the security issues of web applications.  In addition to the usual SQL injections / XSS issues / etc, use of the telephone has unique security issues associated with it.

One issue is that confidential information is only confidential until you repeat it into a telephone.  Even assuming that the phone call isn’t intercepted (which is, ahem, problematic), there are very common user errors and use cases which will cause that information to be disclosed to third parties.  For example:

  • User error in inputting telephone numbers causes the message to go to the wrong party.
  • The message goes to corporate voicemail, where it will routinely be accessible to third parties.
  • The message is played over a speakerphone / cell phone / etc within earshot of third parties.
  • The message is saved on a physical device which can predictably leave the physical control of authorized parties.
  • etc, etc

Don’t ever put confidential information into an outgoing message, unless you have an automated way to authenticate who you are speaking with.

For incoming phone calls, Caller ID is not sufficient authentication.  It can be trivially spoofed, indeed, your phone company will probably sell you a product whose sole aim is to spoof Caller ID.  Instead, you should use a circumstance where the user is already authenticated and authorized, such as a face-to-face meeting or using a username / password pair in a web application, and then give them one-time PINs to do whatever they need to do on your system.  Alternatively, you can implement an entire password system for your incoming phone calls, but users tend to hate them, so I try to keep to the one-time PIN metaphor.  (When a user does something on the AR site which requires calling the system, such as setting up a recording for a reminder, I tell them “Call 555-555-5555 and put in your Task Code 1234″, which (since it is time-sensitive) both helps me look up what they were doing on a multi-user system and also conclusively demonstrates that they were able to read a web page which already verified their identity.

Not in the presentation because the slide got deleted for some reason: the 4chan rule.  Even if your free trial is discovered by 4chan, the world should not become a darker, more evil place.  There exists tremendous possibilities for abuse of free-form input/output to people’s telephones.  I gate access to my trial by requiring a valid credit card, and demonstration calls and the like have strict rate limits which prevent them from being used to spam someone’s phone to death.  (I should also make it impossible to send demo calls outside of standard work hours.  This is easy to say but a little tricky to implement across multiple time zones while still encouraging legitimate use of demo calls, which is why I haven’t done it yet.)

Twilio Scales Impressively

Twilio and modern web technologies scale impressively well by the standards of traditional businesses.  However, you should probably continue to rate-limit your systems, even though you could theoretically do substantially more volume.  For example, many customers who ask about scaling issues do not sufficiently understand that your application scales several orders of magnitude better than their business processes.  For example, a prospective client asked if my system could handle 10,000 phone calls a month.  I told them that I could handle that in under an hour.  They were quite excited about that, but as we continued to speak about their needs, it developed that actually doing that would have crushed their business.  They would have made 10,000 phone calls in an hour, received over 1,000 callbacks, and their two full-time telephone operators would have been overwhelmed by incoming demand for their time.

Grab Bag of Random Advice

  • Never contact Twilio, or any external API, inside the HTTP request/response cycle.  Doing so imposes an unacceptable delay in performance and slaves your reliability to that of the worst performing API you use.  (Twilio has never had user-visible downtime, but some APIs I rely on have.) Queue the request and tell the browser that you’ve done so.  You can drizzle AJAX magic on your website to make this feel responsive for your users.
  • The Twilio Say verb will have a robot read your message.  This is adequate for development, but for production, people prefer listening to people.  Fiverr.com is great for finding voice actresses for $5.
  • You can’t record too much information about Twilio requests, responses, and errors.  I stuff everything in Redis these days.  I strongly wish I had started doing this earlier, rather than writing “An error happened” to a log file and being unable to determine exactly what the error was or easily figured out whose account it actually affected.
  • When in doubt, don’t make that phone call.  Design your system to fail closed.  This is a continuous discipline, but it will drastically cut down on catastrophic problems.

Wrapup

That’s it for the presentation contents.  I remain very interested in Twilio apps, and am happy to talk to you about them whenever. My contact details are trivially discoverable.

I’m going to attempt to write a more comprehensive guide to developing Twilio applications, eventually. We’ll see what form that takes — I would really like to provide people an (even) easier way to get started, but at the same time I can’t justify dropping two months of my schedule to write a traditional book on it.

14 Comments

Inaugural Kalzumeus Podcast: Japan, Startups, A/B Testing, And More

Hiya guys.  My good friend Keith and I decided to do something a little different and tried recording a podcast.  We’re still rather new at this, so it took for form of a freewheeling conversation.  Major topics included:

  • the experience of working at large and small tech companies in Japan
  • the Japanese web application market
  • career advice for programmers don’t call yourself a programmer
  • us trying to sell you on starting A/B testing
  • conversion optimization stories, including actionable tips which have actually worked for people
  • a wee bit of generic geekery.
The podcast weighs in at about 79 minutes long.

Podcasts take a metric truckload of work to put together.  If you like it, please, say so.  If folks have interest in it, we’ll do it again.  If not, well, one more data point as to what the market wants.

Podcast Link (M4A.  Click to play, right click to download.  The play feature may not work quite right in Chrome.  Feel free to put this on your iDevice.  (You may find this URL helpful: http://www.kalzumeus.com/feed/atom/ That technically includes all the posts on the blog, but iTunes and similar software will automatically pluck out the audio ones.)

Podcast Link (MP3, for Chrome. Click to play, right click to download.)

 

Transcript (Because We Love You)

Patrick McKenzie:   Hi, everybody. I’m Patrick McKenzie, better known as patio11 on the Internets.  This is my buddy Keith.

Keith:  Hi, I’m Keith Perhac. I live next to Patrick’s and I’m pretty much unknown on the Internets.

Patrick:  So when we tell people we’re right next to each other, we’re right next to each other in Ogaki, Japan. How the heck did we end up here?

Keith:  Long, long story. So I’ve been here for nine years, you’ve been here for eight, pretty much. And we came here on the JET program. I was working as an English teacher, you were working at Softopia, which is apparently our prefecture’s gift to web development and iPhone development right now.

Patrick:  Yeah, the prefectural technology incubator.

Keith:  That was an interesting little incubator because it’s been losing money for the last seven years. And then when the iPhone came out, all of the iPhone developers and everything moved in there because rent is cheap. And suddenly the incubator is making tons and tons of money thanks to iPhone apps.

Patrick:  Yeah. That was crazy. The number one, number two, and number three most popular Japanese iPhone apps of all time were literally right next door to each other, all in Softopia. They’ve been saying that they were going to make Sweet Valley, the name of this region, into the Silicon Valley of Japan. And I always thought it was a pipe dream. And now we have three successful software companies in literally like a 10‑square‑meter space in one building here. [Editor's note: Perhaps not quiiiiite Silicon Valley yet...]

Keith:  Yeah.

Patrick:  It blows my mind.

Keith:  Yeah, essentially we’re in the Kansas of Japan. There’s nothing here, like literally nothing here.

Patrick:  Yeah, the engineering culture of this region is much less startup‑friendly than say a Silicon Valley or a New York would be. It’s dominated by one monopsony employer of engineering talent, which we’ve both had business dealings with.

Keith:  We’ll just say a very large automaker which everyone knows of and who pretty much pioneered the idea of kaizen, which I’m sure you all know of right now.

Patrick:  Yeah, the big buzzword for the lean startup movement was lean manufacturing before it was lean startup and before it was lean manufacturing it was just the [REDACTED] way. Whoo, shoot.

Keith:  [laughs]

Patrick:  Sorry, we’re editing that out.

Keith:  The T way.

Patrick:  Yeah, T. A big T-Corp here. Somewhat surprisingly, T-Corp and its various affiliated industries don’t really apply the T-Corp way to software development, do they?

Keith:  No, and I think the real reason that they don’t do that is because software development is not seen as a manufacturing process. It’s really seen as this kind of artsy‑fartsy kind of thing here. And it’s interesting because Web is definitely seen as artsy‑fartsy, and programmers themselves are pretty much seen as code monkeys.

Patrick:  Yeah.

Keith:  They’re seen as people on a factory floor that can… you just give them the spec and they punch out the pieces and that’s what programmers are for. And so what you have is a lot of managers who are used to managing assembly lines then trying to manage software projects in the same way. And it doesn’t really work quite as well.

And unfortunately, when big T feels that way about programmers, you find out that all their subsidiaries and everyone who works for them and everyone who deals with those subsidiaries and everyone who deals with the people who deal with those subsidiaries, the whole sort of culture kind of filters down to that.

Patrick:  This is true. Even at my old day job, which was a multinational corporation in Nagoya, which there is no company in Nagoya that does not have business with the big T in one way or another. It’s like Detroit times 1,000.

Keith:  Very true, very true.

Patrick:  Everyone is connected with the automobile manufacturing industry here. Even if you’re not directly connected into your day‑to‑day work. And it trickles down into things like the way engineers are treated, the way the discipline is treated, and the prevailing salary and wages for engineers here.

Even at a large multinational corporation as a salaryman, which is a full‑time employee who is expected to be at the company until death do you part, the wages for programmers are not that great here. Would you say the algorithm is $100 per year of age, per month, give or take?

Keith:  About that, yeah.

Patrick:  So Keith and I are right around 30 right now. That means we’d expect at a regular company here to be making $3,000 a month more or less, which as you might have seen is a wee bit less than you guys get in Silicon Valley.

Keith:  Yeah. Now I do have to say, the pricing structure for payments for programmers and for anyone really, is a holdover from the bubble days. And when you retire, you are guaranteed a very nice retirement fund. And every year because the company will do so well, you’ll get a big, fat bonus check, sometimes three times a year. And the problem is that Japan has been in a recession since about what, I think it’s ’90, ’92, something like that?

Patrick:  Yeah, somewhere in there.

Keith:  Long, long recession. And we don’t get bonuses. There’s no more job security for this until you retire at 60, now 65, soon to be 70. So we’ve lost all the benefits of working for peanuts at these companies and yet we’re still working for peanuts. That’s pretty much what it comes down to.

The salaries have not evolved with the change in Japan’s social contract, pretty much.

Patrick:  Mm‑hmm.

Keith:  We had a social contract where you go onto a company and they will take care of you until the day you retire and now it’s you go into a company and it’s pretty much hit or miss.

Patrick:  Now, for the peanut gallery that doesn’t know, the peak of the salaryman/lifetime employment system was in about the 1970s or so. And at that time it was about 30 percent of the Japanese labor force who was both working in a large multinational corporation and had this particular seishain [editor’s note: “full company employee”] status that we’re talking about. The rest of the 70 percent were not subject to the lifetime employment guarantee including, most prominently, women.

Keith:  Women, yeah.

Patrick:  Who we have many, many stories about how Japanese companies do not treat ladies right.

Keith:  That’s for a different time.

Patrick:  Yeah, another time.

Speaking of the social contract, a lot of Japanese society is built around this presumption that if you go to a good school you will get a good job at a nice megacorp like T-Corp and then have this lifetime employment guarantee. That’s one of the reasons it’s so hard to hire for startups out here, because basically you get one shot at that brass ring of lifetime employment and if you don’t take it right after you get out of the university, you are damaged goods for the rest of your life.

Keith:  Well, I do want to cut in there. That’s not really how it works, but that’s how everyone thinks it works.

Patrick:  I agree with both parts of that statement.

Keith:  And the problem is when everyone thinks that’s the way it works, especially for people who have just gotten out of college and don’t know what it’s like in the world, they’re going to go where it’s a safe bet. And we’re talking about lifetime employment and stuff and how that’s no longer a guarantee. However, if you get into a place like NTT, which is the phone company, even the big T, in most cases if you are a proper contracted employee, then you will probably be there until you retire.

Patrick:  Right.

Keith:  However, there are only a handful of those companies left. I would say that there’s less than 100, maybe less than 50 in all of Japan that are large enough to really be able to take care of you for your entire life like that.

Patrick:  And there’s definitely a difference in statuses at those big companies. For instance, one of the ways they manage to have a lifetime employment guarantee is that T-Corp has this… it’s like an industry within an industry of supporting companies, built around themselves, like suppliers, vendors, that sort of thing.

And while they won’t cut their employees come hell or high water, they’ll cut their vendors like nobody’s business and/or tell the vendors, “Listen, we can only afford to pay you a quarter of what we did last year, so make the appropriate adjustments.” And then the vendors will go down and cut their regular employees or, most commonly, cut their contractors.

Keith:  And T-Corp themselves have a lot of contractors. And when you think of contractors, you usually think of consultants or stuff like that. I’m talking about contract employees working on the assembly lines and whatnot. And those people do get cut.

Patrick:  Right, for example, right before the recent economic crisis, which is known in Japan as the Lehman shock for some reason, right before the Lehman shock there were about 6,000 foreigners, mostly Brazilians, living in our town of Ogaki. And so I go to a church out here and after the Lehman shock for about a year, every week at church we had a family X or a family Y or family Z having lost all the jobs in the family is going back to Brazil. So that population of foreigners, who were almost all contract laborers at the local manufacturing industries ,went from about 6,000 to about 1,000 in a little less than a year.

Keith:  That was a heavy, heavy downturn. But that gives you a pretty good idea of what the labor force is like in Japan. And why it’s difficult to get a good startup community here. I go to a lot of programming conferences and stuff and we really, as programmers, picked a really crappy place to be, to be perfectly honest, because the entire area is ruled by T-Corp.  Also, there’s just a very different way of thinking about business in Nagoya in the Nagoya area than in, let’s say, Tokyo and Osaka. And even for Japanese people, and other Japanese programmers, looking at Nagoya they say it’s very difficult for people who are not in Nagoya to start a business here. And it’s very difficult for new businesses to start here.

Patrick:  So we’ve been doing this for the last couple of years and we know how it actually works, but people in America always tell me I’m lying when I talk about working conditions for professionals in Japanese societies.

Keith:  [laughs]

Patrick:  Why don’t you tell me about those lies I’ve been telling for the last couple of years.  Just sketch some out.

Keith:  So I’ve heard some of the lies Patrick has been telling all of you and I have to tell you that I don’t think any of them were actually even slight exaggerations.

Patrick:  Well, let’s recount some stories, shall we?

Keith:  Let’s recount some stories. So coming home every night at 12:30 AM, if you could get home.

Patrick:  Right.

Keith:  OK, that’s definitely true.

Patrick:  Last train of the day is 30 minutes after midnight, I missed that train quite a bit of the time.

Keith:  Yeah, the business hotel near the station actually knew Patrick by name and face. And Patrick had this thing Tuesdays where there were longhalf‑day meetings.

And so you get pretty much nothing done on that day. So you’d have to work until about 12:30, 1:00, end up missing the train and he would arrive at the business hotel and they would have a room ready for him. And if he wasn’t there one week, then in the following week when he did show up, they would ask, “Why weren’t you here last week?”

Patrick:  It got so bad at one point I left my Kindle there overnight on a Monday and it was waiting at the front desk for me the following day with a note, this is Mr. McKenzie’s Kindle, he’ll be in, it’s a Tuesday.

Keith:  So really, no exaggerations about the workload. And I, fortunately, was not as bad as him. I could usually get home every night. I usually got home around 11:00 or so. So I was very lucky to get home at 11:00.

Patrick:  So what are we doing for this 12‑to‑19‑hour day? Is it actually programming for the entire time? No, not so much.

Keith:  OK, so Patrick and I had very different employers. He worked at a very large, how many people were in your company?

Patrick:  Well, let’s see, at my office there were about 70, but companywide maybe 1,000‑1,200 employees or so.

Keith:  So he’s with a very large company that does only programming. I, on the other hand, was in what is pretty much the Japanese startup.  A startup is called a “venture” here. It’s generally a single person putting his money on the line, getting a loan from the bank, and trying to make a company out of it. I was the only programmer at this IT company. So we were a company of around seven or eight people, and I was the only programmer left after my boss, the other programmer, quit.

So pretty much I had a lot of free rein in what I was doing, but, on the other hand, for any product that my company had to ship, I had to make it from scratch. And that’s not just programming, that’s the design, development, graphic design, figuring out what the customer wants, how we’re going to accomplish it, pretty much everything. The other people in the company were sales and management.

Patrick:  My company was a multinational consultancy.  Think of like an Accenture or IBM, which goes to a client and say, “Hey, we’ll fix your problem, whatever the problem is, and we’ll sell you smart people to make it happen.” We were in the wetware business. I was the least wet bit of the wetware, we’ll put it that way.

My official title was “System Engineer.” That’s one of these things in Japan, programmers are considered code‑monkeys, but system engineers, who tell programmers what to do, are assumed to have a bit of discretion and professional ability. Maybe half the company was people in sales or support, who would sell a university on, “You should use our systems for doing your backend course management or payroll or whatever.”

Then the other half of the company was the engineers like myself who would talk to the customer and figure out what they actually needed for integration with their systems and then build out the code to do it. Or assist outsourcing operations for outsourcing the business process management to low‑wage countries like India.

The feeling in the company was that even though we were a software company, writing software did not add value. That’s almost a direct quote from the CEO. So, we would figure out the software that needed to be written and then actually get the software written by cheap people because that would provide more value for our customer.

This resulted in me doing outsourcing management to India for about three years because I was the only one in the company who could speak good enough English to manage it. That was an experience.

Keith:  You had actually told me about some of the technical docs that they had sent before you were on the project where they would actually put the Japanese technical docs into Google Translate and then just ship them over.

Patrick:  Yeah. I tried to stop that and was less successful than I wanted to be. At one point I told my boss, “Listen, boss, we’re shipping out several thousand pages of technical documentations for this particular subsystem which just absolutely puts the ‘critical’ in ‘business critical.’ You need to clear the next month of my schedule so I can translate them.”

He said, “Your fully loaded cost as an engineer is about $6,000 a month [ salary of $3,000 times a factor of two for contribution, taxes and other overhead involved with hiring me]. $6,000 a month is too much to pay for translation. We want to get it done for about $4,000, so we’re going to have an Indian company to do the translation for us.” I said, “I don’t really love that plan, but I’ll do it. But I want to spot‑check the quality of their translation just to make sure we’re getting our money’s worth.”

The first day I get back an Excel file. I can’t remember exactly what the error was, but on the first page that was describing black as white, a clear error. I mail the Indian subsidiary and say, “Hey, I was spot‑checking the quality of your translation and there’s just this glaring error on the first page that makes me worry about the rest of the documents. Can you please tell your translators to be careful.”

They mail back to me a few hours later, “We don’t agree this is an error.” It’s literally a translation of black is white or something. So I mail back, “This is absolutely an error. I haven’t had the opportunity to discuss it with a native speaker of Japanese, but there’s no possible way that your translator is translating this correctly.”

I get a mail back, “We don’t agree.” I said, “You need to put your translator on the phone with me now because we need to discuss this. The project will not proceed if you have this level of understanding of what the requirements are.” They said, “We discussed with three authorities and all of them agree with our translations. So you’re outvoted.”

I said, “Who are your three authorities?” They said, “Google Translate, BabelFish and AltaVista Translate.” So I was outvoted by three computers who all use the same dictionary apparently. That sort of thing happened all the time. They had billed us literally $4,000 for having people do the manual copy‑pasting work into Translate and billed it as translation services. Yeah. That project didn’t work out very well.

Keith:  I actually want to jump back real quick. You had mentioned the difference between the System Engineers and programmers. You had a blog post, I think it was two weeks ago and it got put on Hacker News. Everyone was like, “Oh, I’m a programmer, so that’s what I’m going to call myself. I’m not going to call myself System Engineer.”

This is really one of the places where it’s a very clear‑cut difference. It doesn’t matter what it is that you do all day. A simple name change raises your pay grade pretty much double. The amount of things that you are trusted with, the amount of things that is decided that you do. This is not to say that programmers are not given system engineer responsibilities, they’re just not believed to able to accomplish them.

Patrick:  Right.

Keith:  So, it doesn’t matter what you’re doing in Japan, as long as you’re a programmer, call yourself a system engineer and you’d double your pay.

Patrick:  Right. Actually every time I had a bug for about the first two years of working on a company, often because I didn’t quite understand the requirements in the document because all the documents are in highly technical Japanese. And my co‑workers would be berating me for the bug or the absence of testing, which failed to reveal the bug. Or just being careless, because I’m often careless. They would say, “Listen, Patrick. This isn’t acceptable, you are not a programmer. This should have been taken care of.”

Definitely with regards to Japan, call yourself a system engineer if you have the choice, even with regards to America.

I do programming, I like programming, I love programming. The craft and actual activity of programming really, really appeal to me. But as a consultant when I’m talking to businesses, I never describe the value that I’m adding to the business as, “OK. I’m going to program some stuff for you in Rails.” Because businesses don’t really see it on that same level.

Even businesses that are run by really engineers’ engineers, like the Fog Creeks of the world, which really love programming and the craft of programming: at the end of the day they are a business.  So you need to be connecting to them on that business level of, “Here are the concrete benefits you’re going to get from doing this engagement with me.”

Keith:  It’s just like when you sell a product, you don’t sell the features, you sell the benefits.

Patrick:  Right, right.

Keith:  You are programming, and that’s the feature that you’re selling. But that’s not the benefit. The benefit is not, “I’m going to write code for you.” The benefit is, “I’m going to make a system that does this awesome thing. And this awesome thing is going to make you a million‑up in dollars.” And that’s what you sell. That’s why you call yourself a system engineer instead of a programmer.

Patrick:  Or providing solutions instead of providing software.

Keith:  Solution Architect. It was so funny. My job title in my company has changed so many times. I think it changed seven or eight times in the last four years. And my favorite one was “Solution Architect,” because it just says nothing about what I do whatsoever.

Patrick:  But civilians, or whatever you call people who aren’t engineers, sometimes respond to things like what is printed on your business card. That being the reality, adjust yourself to the reality rather than what you think your ideal world would look like.

Keith:  Yeah. Really, if you want to put “programmer” on your business card, write it in binary on the back or put it in as part of the design or something.

Patrick:  Right. You can always call yourself what you want when you’re around just us geeks, but when off Hacker News and you’re talking rate negotiations with customers, adjust to their view of the world.  It will make you happier in life.

Keith:  Exactly, exactly. Don’t believe us, A/B test it. I actually did this. I went to a prospective client with a programming friend of mine. We do the exact same thing. We have the roughly the same levels of technical and business acumen. We decided, “Hey, we’re going to try this out.” He called himself a system engineer, I called myself a programmer. For the rest of our meeting they deferred to him over me for every single thing. [Editor’s note: You’re right, this is not an A/B test.]

So just changing that one little word on your business card or when you introduce yourself makes a huge difference. It’s really just you have to change yourself to fit your customers’ expectations.

Patrick:  Obviously the culture of businesses in Japan is very different than what many of our listeners are used to.  Well, I say the culture of Japan: Japan is a big place. It’s like saying, “the culture in America.” Kansas is a different place than Texas, is a different place than a tech startup specifically in Mountain View, California. But the business culture of, say, metropolitan Nagoya, Japan is very different than the business culture of New York City, or at a tech startup in New York City, or at a financial services firm in New York City, or specifically Goldman Sachs in New York City.

You have to adjust yourself to whatever that particular situation you find yourself in. That being said, there’s very, very few places I’m aware of calling yourself a programmer is a career enhancing move.

Keith:  Yeah. Let’s stop berating everyone for calling themselves programmers.

Patrick:  Right.

Keith:  What do you say? You want to move on to kaizen?

Patrick:  Sure. Let’s move on to kaizen.

Keith:  OK. Everyone, I’m sure, knows kaizen.

Patrick:  Actually, I don’t think that’s true.  Why don’t you explain what kaizen is.

Keith:  Really? Because I got a photo from one of my friends in Nashville of all places. She went to a burger joint and had the Kaizen Burger. This huge burger with Kobe beef and cheese and all the stuff, and they just called it “Kaizen Burger.”

Patrick:  That sounds delicious, but I think the rest of America is not caught up to Nashville yet. So let’s go.

Keith:  Everyone should catch up with Nashville, I swear. [laughs]

OK. Kaizen is a term in Japanese which means “improvement.” That’s the general translation. What it has come to mean in American English is the reiteration and iterative improvement of systems and processes that were founded pretty much by Toyota. Essentially it’s reiterations over a design in order to get rid of flaws and to make a better product.

This is pretty much the the cornerstone of any physical manufacturing process. Any car that you see is built with kaizen. Apple does it with their Macs. Everything is iterated and iterated. They make a design, they put it out there, they find out what’s wrong and they slightly change it, they slightly fix it and they keep reiterating and reiterating. Then they finally get a great product. That’s how Toyota has worked, that’s how most manufacturing works.

Patrick:  I think this idea is so powerful. There have been hundreds of volumes written about the Japanese economic miracle. Honestly this and the related Toyota process improvement were so powerful that they were able to cover up every mistake made at the same time. If you have that going for you and the rest of the world hasn’t caught on too it yet, you can do all manner of stuff totally stupidly, like work your people to near death by exhaustion. You’ll still ROFLstomp over the competition.

Keith:  Exactly. Where kaizen comes in to our conversation now is that kaizen has left the physical manufacturing world and is starting to go into the software development.

Patrick:  If you read Eric Ries’s book The Lean Startup, which by the way is a fantastic read and you should read it if you haven’t already, it’s one of the ideas he touches on a lot.

Keith:  Yeah, yeah. I actually want to pick up a copy of that. I saw that in your house when I was over there.

Patrick:  One of the ways that kaizen is relevant, not just to people who are manufacturing physical products but also to software developers who are making software or an experience delivered over the Internet, is that we can do things like A/B testing. So we can tell in almost real‑time which of multiple versions of a website, once exposed to users, causes those users to have a happy experience. You could define “happy experience” in a few ways: judged by business goals like whether they buy more software or sign up for your email list more often, or judged by whether they achieve success with your software.

For example, I sell software that makes bingo cards for elementary school teachers. If the schoolteachers can’t successfully make bingo cards, that’s a problem for me. So, I can redesign the interface of my software such that they actually achieve task success.

Keith:  The great thing about this iteration, when we’re dealing with software and especially web apps, is that the feedback is so immediate and the cost for doing these iteration tests, these split tests is so low. How long does it take you to make a split test?

Patrick:  One line of code, it’s the only way to do it.

Keith:  One line of code. All right. In one line of code you can test which creates the better user experience.

This kaizen originated in manufacturing. In manufacturing, it’s a much more expensive process. You have to build it, which means you have to use the raw materials, you have to create the building process, then you have to ramp up production and ship it to the customer and the customer has to give you feedback. We’re talking, maybe, two or three years to get feedback on a single design before you can iterate.

Now, we can iterate… If you have enough users on your site, you can get an answer with a split test in about a day.

Patrick:  At Facebook or Google scales you can get it literally in seconds. Even at fairly minor scales, like the Bingo Card Creators of the world, I can throw out a test every Monday and have a statistically confident results generally by the following week. It’s a wonderful thing when you don’t have much time to work on a site.

If you’re getting about a hundred visitors a day then you can throw some up, come back a week later, and have results.  Doing the A/B test doesn’t block any of your other activities for the week. You can continue talking to customers or continue writing documentation, continue producing the new feature. Then, when you’re ready for it, take a look at the results of the A/B tests and generally just make a one click to pick result A over result B or whatever the stats tell you to do.

Keith:  What we’re actually seeing… Patrick’s A/B testing software, he made it himself. It’s called A/Bingo. It’s amazing and you should pick it up. I use Visual Website Optimizer, which is a great software, all JavaScript based, works on any page, I recommend that as well.

But what we’re seeing is a change from manufacturing where in manufacturing you can have a thousand ideas, but your iteration time is around maybe six months at the fastest, a year, maybe two or three if you’re talking something huge like a car.

With the software iterations, one line of code takes me five minutes to implement. VWO does the same thing, it takes me 5‑10 minutes to implement. What takes more time now is to actually think of what we’re testing. The time it takes to test something is so infinitesimal that thinking up good tests takes more time. Which is amazing that we’re in this technological area that we can test things as fast as we literally think them up.

Patrick:  Right. Also it prevents a lot of waste in the company. Have you ever been a really pathological meeting? I once had a six‑hour meeting involving four people to discuss the text that was going to be placed on a button for signing up to a mailing list. The mailing list was only getting 50 sign ups a week anyhow. There’s an actual cost to that. The fully loaded cost for that meeting was on the order of $300 an hour times six hours, so almost $2,000 spent just to determine the call‑to‑action on a button that didn’t really matter anyhow.

If you have a culture in your organization of testing things, that would literally have been a 45 second discussion. Like, “I think it should be, ‘Sign up for mailing list.’” “I think it should be ‘Sign up now.’” “OK. We’ll throw it in the test.” Done. And you save that $2,000 and spend it on stuff that actually matters to your business and your customers and the world at large.

Keith:  Yeah. Going back to the Japanese side of things, this is a big thing where they don’t consider software to be a manufacturing process. Testing in Japan… Now, we have unit testing and testing like that, but the idea of user‑based testing is pretty much non‑existent, almost completely non‑existent. Unless you’re going into the very large, very in‑the‑know tech companies. I’ve dealt with one or two, and they’re not common at all.

Patrick:  For example, one of the several big startups based in Tokyo are in the mobile gaming space. There’s an American company you’ve probably heard of, they allow you to make virtual cabbages and water the virtual cabbages. Sometimes the virtual cabbages rot.

This company is very, very sophisticated and they iterative improvement processes including A/B testing and collecting user metrics and what have you. They recently opened a Tokyo office and somebody there said, “Yeah. We’re pretty much planning on effing killing all the Japanese companies in the this space.”

I do not love that company, but I think they’re likely to achieve that goal.  Just like Toyota versus the American manufacturers, if there are 10 people in a fight and only one of them understands how to use science, I already know who’s going to win.

Keith:  I like that science is the thing that’s going to win the fight. [laughs]

Patrick:  It seriously is the revenge of the geeks out there right now. Really.

Keith:  Really it is. For the first time in history we have so much data that we don’t have enough people to process it. We have so much information that really what we need to do now is just process all this great information and put it to good use. And we just don’t have the resources yet.

Patrick:  Right.

Keith:  What we need is mathematicians and statisticians to find, look over, and analyze this data.

Patrick:  Yeah. Also, this is a good thing for you to pick up for your career. A/B testing is a skill set that is not really all that hard to pick up. Grab a book, study it for a week. The mechanics are dead simple. After that, it’s just a matter of coming up with what actually to test.

But that skill is fantastically rare, much more rare than, say, skill with creating web apps in PHP or Ruby on Rails. Because it has direct bottom‑line implications to the business, to the tune of, “What’s five percent of your sales for the next quarter?” When you’re talking to a company the size of say a T-Corp or just the line of business from Bank of America’s website, five percent is a very significant number.

If you can routinely quote numbers like that, you are no longer in the $60‑to‑$100‑an‑hour bucket with other programmers, you are in the bucket with strategic initiatives that add five percent to Bank of America’s bottom line. Scary stuff.

Keith:  I really hate to keep coming back to this. System engineer versus programmer, implementing an A/B test takes one line of code, takes five minutes to do. If you are the person that used that one line of code, changed the color of a button from blue to red, and added five percent to a multi‑million dollar company’s bottom line, you’re no longer the programmer who implemented that, you’re the guy who added five percent to a multi‑million dollar bottom line. That’s what matters.

Patrick:  Right. Billing also scales right to match, particularly when you get clueful clients. I won’t be comfortable talking about mine. Suffice it to say if you have a portfolio website… Not that I have a portfolio website.

Keith:  You need to make one.

Patrick:  I don’t know. Yes and no. When someone comes to me and says, “Hey, we’d like to hire you for this thing, do you have an example of people that it’s worked for before?” I can informally say, “Well, I’ve worked for XYZ, and XYZ have reported this kind of result.”

That’s, by the way, a trick when you’re dealing with clients… It’s not a trick because both of you will benefit from this. But say, “Look, if we have a successful result with this project, I would like to write that successful result up as a blog post or something. That will be to our mutual benefit, you’ll get additional attention because of me writing the blog post, and I’ll get something to keep for my portfolio.”

Phrased like that, many clients will say yes to it. It is a wonderfully, wonderfully wonderful thing to arrange for yourself as a consultant. For example, when I was working with Fog Creek, when it became obvious that the engagement was going to work out for both of us, I said, “Hey, you guys are always looking for new topics for your blog post, why don’t let me ghostwrite one about what we did for the marketing?”

They said, “Oh, yeah. Let’s do that.” So I wrote something called “Our Marketing Is Up Fog Creek and How We Fixed It” or something to that effect. Where we talked about things like we made changes to the website and as a result Fog Creek’s conversion rate went up by, say,  10 percent. Just make a guess of how much money Fog Creek makes in a year, add 10 percent to that, it’s meaningful, right?

So, Fog Creek is obviously very happy with this. But in terms of dealing with my other clients, when they say, “Oh man, Patrick. That number you just quoted to me is a lot of money. How do I justify that to the investors or how do I justify that to my team?” I say, “Well, if you look at this post over here by Fog Creek, they said I added about 10 percent.” And done, that’s one way to deal with pricing objections.

[Editor's note: Patrick wasn't clear on this, but 10% was a handwavy approximation number here.  The actual number is undisclosed.]

Keith:  And that’s something I’ve been working on with. We were talking earlier about how this split testing your stuff is not known in Japan. I’m trying to bring split testing and  modern iterative development practices into Japan.

One thing I found when dealing with my customers is that they feel the same way, it’s like, “How do I add this to my bottom line?” I say, “Well, I improved such and such company, who makes a million dollars a year, by 10 percent.” Sometimes they understand it and sometimes they don’t.

When they don’t understand it I say, “Well, why don’t we do it like this. You pay me half of what I’m charging. And for every percent improvement we see over your current baseline, you give me another X dollars.” People seem very happy about that. Until, of course, the results come in and they end up paying me three times to what they were originally going to pay me.

Patrick:  Yeah. Pay‑per‑performance is an interesting model.

Keith:  It’s not the best.

Patrick:  Yeah, I am not a big fan of pay‑per‑performance arrangements. Not to say I would never do them, but I typically try against them.

Here’s one reason. In any sort of project you have execution risk. Generally as a consultant you want to be responsible for your own conduct, but not risks external to your own execution. If, for example, you want to do a pay‑per‑performance arrangement for search engine optimization or for A/B testing or conversion optimization for a startup, that startup’s priorities might change on a dime.  This can happen one week after you get out of the building. That actually happens frequently. The project you were working on can get shelved.  This even happens at multinational corporations.

If this happens, you’re not going to make your pay‑per‑performance numbers no matter how good the quality of your underlying work. Typically, with a pay‑per‑performance arrangement you’re going to be getting a smaller number upfront, plus a payment if you hit particular milestones. You won’t get that milestone payment and you’re going to end up working for a fraction of your rate. And working for a fraction of your rate is not the purpose of the exercise. Unless it was client who had a clear reason not to pivot on the…

Keith:  Right. I should have prefaced that with saying that these are clients that I have had worked with before in my previous companies and that I do trust. And who I know are not going to shelve the entire project and I’m going to get put in the works.

These are things that part of the contract of doing the performance‑based payment is that I’m going to be setting it up and they are going to be putting it out there and that it is going to run. Like you said, it’s not a 100 percent, nothing’s ever 100 percent. But I would never do it especially for a large company that I’ve never worked with before, I would never do a performance‑based review. I don’t think.

Patrick:  Right. Engineers have a funny notion of what the price of things is. If I say “$3,000″, that sounds like a lot of money for us because, hey, we worked for months for that. But $3,000 for a real company that has real assets and real employees, that’s literally below their capability of measuring stuff. It isn’t a line item on the annual report, it isn’t even in the appendix to an appendix on the line item of the annual report. You really have to get into the five‑figures to even hit on the radar screen at all.

Given that, if you were thinking, “OK. We’ll could charge a variable price between $3,000 and $15,000, or just charge $20,000 fixed,” there’s a lot of businesses that will say, “Give me the higher $20,000 price, because that will make my life easier. Getting a variable charge approved through the accounting division or my higher‑ups or whatever is going to be more difficult, because that’s not our established process.”

If it is just a fixed  $20,000 coming out of budget X, well, budget X has hundreds of thousands of dollars in it because there’s employees in there, too.  [Editor's note: Employees are very expensive.  There is a cynical but true saying: "Overhead walks on two legs."] So $20,000 out of that budget is not difficult to approve.

Keith:  Right. Working, not with T-Corp in particular, but with other companies of that size, we would have our sales reps on their side come to us and say, “Our budget is X amount of dollars. Make it under that.”  That’s not their maximum possible budget, that’s the budget they have before they need additional signoffs.  You could spend within $1 of that, and it would be no problem. However, if they go over that line, they have to get the sign off of their manager or, god forbid, the vice‑president or someone. As long as it’s under that number even by a dollar, they can sign off themselves and that makes their lives so much easier.

Patrick:  You were saying something interesting earlier, which was that you hope to bring in A/B testing and these modern techniques into Japan. That’s something I’ve heard a lot in working with my other companies, because the Japanese engineering, with specific regards to, say, Web engineering, is kind of lagging a couple of years behind the U.S. right now, right?

Keith:  Oh, so far behind.

Patrick:  Let’s talk about the status things. So the Joel Test is getting almost 10 years old now.

Keith:  Is it 10 years already?

Patrick:  Almost 10 years I think. [Editor's Note: Even older!]The Joel Test is about basic things like source control, having a quality department, yada, yada. Japan, not quite there yet. So I worked with a company that worked with other companies. We had source control, thank God. Subversion, though, not git. Sorry, I didn’t mean to be a snobby geek there. Subversion is an excellent source control system. It’s heads and tails better than no source control system, which is the competition here in Japan.

Similarly, we write big freaking enterprise applications in Java with homegrown web frameworks, which were absolute pain in the butts to work with. One of the reasons I have like 60,000 Hacker News karma was there was a compile and deploy step that literally took like five minutes to run. So if I wanted to change, say, the number of columns on one particular table on a page, testing that would take upwards of an hour.  Oh, whoops, I had a one‑character bug in my HTML, rerun the compile. It’s going to take five minutes. Yada, yada.

The amount of friction involved in that process destroyed my productivity.  [Editor's Note: It didn't help that I'm fundamentally a lazy git.] However, we measured productivity by number of engineering hours expended, so by that metric I made my department look very good. But modern web frameworks like Ruby on Rails, Django, all the fun stuff, make life so much better.

Keith:  It’s really ironic that Ruby was created by a Japanese man, and yet Ruby on Rails is pretty much unknown here.

Patrick:  Maybe this will change a little bit. I think EngineYard has hired Matz, right?

Keith:  Right. Was it EngineYard ? I thought it was Heroku?

Patrick:  Yeah, it was Heroku. Sorry. Heroku by the way, is the most Japanese name on an American company I’ve ever heard. [Editor's note: I heard at Startup School that they were explicitly inspired by the Japanese aesthetic... in case the samurai and Godzilla on the pricing page had not clued you in yet.]

Keith:  Yeah. I know. But actually there are Rails conferences here and it’s getting more and more popular. The problem is that web apps themselves are still not that popular here. I think part of that is done in by the fact that, there are maybe four or five countries in the world that have IE6 shares above 25 percent.  [Editor's note: I don't know about this one, but wasn't sure enough to correct him.]

I believe they include Thailand, China, and Japan. China actually has I think 23 or 22. It’s really quickly dropping. But Japan still relies on IE6 so much. And the reason for this is that back when IE6, when XP came out, we were still, not really in the recession, but we still have a lot of money as a country. And so everyone had their IT departments, and everyone had these huge reduxes and they made all their systems compatible with XP. And it was great and wonderful. And then the money starts drying up, and the first thing to go is the tech departments.

I’ve worked with couple large companies that literally had outsourced their entire IT departments. And so there was no one at the company anymore who could even do anything. Let’s say they needed a new computer setup or new proxy or one of their NATs died or something, there was no one at the company who could fix that.

Patrick:  That brings up a great topic. You’ve probably heard that engineering productivity is worth pretty much any amount of money relative to engineering salaries.  Everyone tells you to get two big wide screens, as it’s only 400 bucks extra.

Despite my previous employer being a software company, they had outsourced the internal IT. So then rather than buying each programmer a computer, they would lease a computer through blah‑blah‑blah, whatever, for 80 bucks a month. And by default, it was a years‑old computer with, let’s say, one gig of RAM, a processor from 2003, and a 14‑or‑15‑inch monitor for a desktop system. And there was literally no button you could push at the company to get a better computer than that.

Keith:  Once the rules are set, it’s not worth the bureaucratic hassle and even the time to think about to change it in a company. What it does, in the end, is it just hurts everyone.

Patrick:  Another factor about web apps in Japan is that a huge portion of Japanese use of the Internet is not actually on real browsers, it’s on the lovely fake browsers that you find in old pre‑iPhone Japanese feature phones. That’s an entire topic in itself. Japan, hardware‑wise, were probably the most advanced cell phones in the world until…

Keith:  Also software‑wise. When we were still dealing with WAP and wedge TML and WEKO, WACKO, WACO, WACO, whatever the hell we had back in early ’90s and stuff, Japanese cell phones had iMode, which is a full-featured HTML subset, and they were browsing websites in ’91. On their cell phone.

Patrick:  Websites with pictures and JavaScript. It was amazing. And then it didn’t really change for like the next 15, 16 years.

Keith:  Yeah, they’re still browsing the same Web pages 20 years later.

Patrick:  Yeah. They call this the Galápagos effect. Like the Galápagos Islands, which have a variety of different finches that are found nowhere else in the world, Japan had a variety of very advanced hardware platforms found nowhere else. The software for each of the hardware platforms would be literally handwritten assembly each time a new model came out. That was the dominant way to do cell phone development, with improvements like, “OK, we’ll put a JRE on it and allow you to run Java code.” But that was the dominate cell phone paradigm up until the iPhone came out. And the iPhone proceeded to ROFLstomp over the Japanese carriers. Right?

Keith:  Pretty much. Pretty much. And now we have Android phones as well — a lot of Android phones. The problem is, and I think it’s pretty much the same in America right now, is that you don’t get the updates. The Galápagos model is still the common way of thinking for any cell phone besides the iPhone. So you have this Android‑based amazing phone that is outdated in one to two months. You will never receive an update for it through the life of the contract.

Patrick:  Just a feature of the Japanese consumer electronics market. People tend to upgrade devices very quickly here.

Keith:  Very quick. Very quick.

Patrick:  Japanese cell phones are marketed under the assumption that the core of the market updates their cell phones at least yearly.

Keith:  Yeah. In America, because everyone is on the two year contract, the companies assume that you’ll update your phone when you’re two year contract is up. And in Japan, even though there’s a two‑year contract, most people update their phone, I would say, once a year. And depending on how trendy you are, even more.

Patrick:  One of the main early adopters for the cell phones in Japan, and I swear I’m not making this up, high‑school girls.

Keith:  Oh, yes.

Patrick:  Their use of the phone as a mobile/Internet platform drives quite a bit of the development here. I can never remember which of the companies it was, but a couple of years ago when smartphones were starting to get popular, one of the companies was literally pitching them as a fashion accessory item.

So in addition to pitching, “It has a camera. You can take photos with your friends. And here’s what the software does.” They would pitch like, “And here are the varieties of things you can stick on the phone to make it look different but similar to your girlfriends’ phones.  This way, you can be trendy in your own individual-but-not-too-individual way.” [Editor's note: If you think I'm cynical about the social motivations of teenage girls, you should hear me talk about geeks.]

Keith:  Right. And they’re starting to do that with the Androids now. So each of the Androids has pre‑installed software for each kind of clique for the Japanese high‑school girls. It’s really interesting to look at. It is a bit depressing as a programmer. It’s like, “You could just customize the wallpaper yourself and not have to pay the $600 for the new phone.” But, oh well.

Patrick:  Yeah. Meet the customers where they’re at. Right?

Keith:  Right. Right. Do we want to go on to websites as… I still not have been able to figure out how to really put this other, not in Japanese.

Patrick:  Well, say it in Japanese and I’ll translate for you.

Keith:  Now I’m going to forget it in Japanese. What was it? Ore ore homupeiji kind of thing.  [Editor's note: The rough sense of this: "Me-me-me home page!" ] So what it really is, so you have the ore ore framework.  It is the framework that you built, and you love to death, and is not used for anything at all.

So the web page is pretty much in a lot of cases… OK, here’s how I’m going to say it. The business card web page, I think really is the best way to put it. [Editor's note: Brochure site, maybe?] A business card web page that doesn’t accomplish anything first of all. And in addition to not accomplishing anything, is made in a way so that the president of the company feels vindicated that he has spent enough money on his web page.

So even if they have like a goal like get customers or tell customers about our company, it’s not something that they’re working towards. They are not investing any time or effort into the home page other than saying, “Oh, we need a home page. I spent $6,000 on a designer to design me a home page in Dreamweaver.”

I think we should talk about those.

Patrick:  Let’s talk about this.

Keith:  Yes.

Patrick:  So we’ve got this idea of having a brochure website for maybe an offline corporation or even for companies that are SaaS Internet startups what not. The home page is not designed for the interests of the business but rather for interests of stakeholders in the business, which are two very different things.

Man, I’ve seen home pages that were literally battlegrounds. There was a war between the engineering department and the marketing department and both sides lost. Engineering said “We want to build features. We don’t want to work on any of that crap on the home page that doesn’t require any technical expertise whatsoever.” Marketing said  ”We have this message and this brand that we want to get across.” The result of that war was a page that didn’t convert anyone and wasn’t clearly owned by any group in the company.

There are excellent companies which have little dysfunctional issues like this.  Engineering’s job is to ship more features to the customer. Marketing’s job is to brand or whatever their magic stuff is. But nobody’s job was to measure the conversions on the page, or to increase those over time.

I’ve gone into very smart, successful, well‑managed companies and asked questions like “So, how many trials did you guys do last week?” And the answer would be, “We don’t know. There’s probably a SQL query we can run somewhere that will tell us that.”

Keith:  Yeah. I had a similar thing. This was actually very eye‑opening. I did a reservation system at a really cool place.  They have online and phone reservation systems.  The phone reservations have a backend to their online system, which the front desk uses to place reservations.

So I could see, when I was doing the analytics for their site, how many reservations they had but not where they came from. I couldn’t see if they were self-service reservations online or assisted reservations over the telephone. So I asked the person in charge. I said, “What percentage is online? And what percentage is phone?” He said, “I think at least 80 percent of our reservations are from online. I only think 20 percent are from the phone.” I said, “Wow, that’s much more than I thought. I thought that the phone would be much, much more popular. And he said, “Yeah, but those are the numbers I think.”

So I installed Google Analytics, and I dug into the numbers. It turned out that less than 20% of their volume was online, and more than 80% was on the telephone. So they had no idea of even the number of reservations that were coming in and where they were coming from. And doing a little more analytics, I found out that those 20 percent were the customers who had survived through the reservation process.  The system was difficult to use and the conversion was horrible: only about 15% of people who started a reservation actually got done with making one. That was very eye‑opening for them.

Patrick:  It’s easy to laugh at them in hindsight, but I guarantee if you run a business right now, there is some number that is key to your business that you are not tracking and that would be very eye‑opening to you. The one that I routinely bring up for clients is, what percentage of people who use the software/free trial/etc stop using it after the first time, versus what percent use it at least twice?

The number that use it at least twice is almost always depressingly low. When I started tracking it for Bing Card Creator it was only 40 percent of people came back a second time. These days, it’s up to 60 percent, which a 50 percent lift, did wonderful things for my business, but still 40 percent of people would not even give it a second look.

40% is a decent ballpark number for people who don’t track this.  That has been fairly consistent across a lot of clients of mine. No one ever gives you a book that says you should be tracking this because it’s abysmal right now and you should improve it, but if you’re not tracking that, I guarantee you it is probably abysmal right now and you should probably try to improve it.

Keith:  You should write a book about that. You should just write a book about these are the things you should be tracking, why the hell aren’t you?

Patrick:  Yeah, everybody tells me I should write a book, but writing a book is a terrible idea.

Keith:  Maybe a pamphlet. [laughs]

Patrick:  It’s funny. If I write a blog post people will read it and maybe comment on it for Hacker News. I know it would be forgotten the day after. Ideally, if I wrote a book it would be referenceable by people. Despite the underlying information being essentially the same product, if you put 2,000 words in a book it is perceived as having value, but if you put 2,000 words in a blog post your readers are going to be “TL;DR dude.”

That is the theory at any rate.  The reality is that putting 2,000 words in a book is a terrible, terrible use of my time business‑wise.  Even if I sell the book (or e-book) at $20 or $50 or $100 a piece, and sell hundreds of copies, that is not a meaningful amount of money relative to getting one of my happy consulting clients on the phone and saying, “Hey, I have an idea that I want to try for you guys. Hopefully it will do as well as the last time. Let’s get this going.”

Keith:  However, one thing you might want to try is even just a blog collection and get a physical, printed book. One of my clients, he’s actually considering giving his book away just because of the amount of press it gets him. We did some welcome page tests.

The first one was a picture of him, where he graduated from, and the things that he’s accomplished.

The second one was a big old picture of his book on the “New York Times Bestseller.”

The third one was “I’ve been on NBC” with a video of him in NBC.

The one that says “I’ve done all this” did fairly OK. The one that says “I published a book that was a bestseller” I think was 100 percent better. 100 percent better than that was “I was on TV.” Pretty much once you get on TV, people will buy your shit, no matter what you’re hawking. If you have a clip of you being on TV, they will buy anything you want.

Patrick:  This, by the way, is hilariously underexploited by engineers and people in the startup community, but it’s well known among direct marketers.  After you have press about you, if the New York Times blurbs about you, even one sentence worth: in all your copy from that point out you put the logo of the New York Times and you say “as appeared in theNew York Times” and that will increase your conversion rate up the wazoo.

You’ve suddenly jumped from “A website that I’ve never heard of and don’t trust” to “Endorsed by the paper of record.” You will routinely see this on some of the skeazier direct marketing pages. Even if they haven’t actually been on the New York Times, they’ll say the product category’s been endorsed by the New York Times.

For example, if they’re hawking a social network, they’ll say “As seen in the New York Times.”  They haven’t actually been reported about, but a social network — probably Facebook — has been reported about. I wouldn’t be happy about doing that. That said, if you’re a Fog Creek or a company with  is legitimately high enough profile to getink from the New York Times, that should probably be on the website somewhere.  Fog Creek has even had their office written about before. [Editor's Note: Like Lonely Island says, "Doesn't matter, still counts."]

I don’t know if that is on their website, actually.

Keith:  You should just talk to Joel about that. One thing I do want to say you don’t want to put on your site is “as seen on TV.” Never put “As seen on TV”, especially the red circular logo with all the explosions that you see on $50‑dollar abs stretchers or whatever. That’s not the best thing.

Patrick:  Yeah. People may have cottoned on to the fact that that means they paid for an infomercial. However, retrofitting an explanation like I just did is witchcraft. Have your intuitions and have your thoughts about what would probably work for the site. Like we mentioned earlier, it’s so cheap to be test these things, you should be continuously testing them.

Keith:  Even if you have a banner space on your home page, just switch out the banners. Do an A/B test with that, switch out the banners, see which one performs better. It’s so simple.

Patrick:  It’s so simple and it’s absolutely crazy the amount of lift you can get for an established business that produces value and has been doing so for a decade just by switching out banners.

Keith:  Speaking of which, are you going to tell about the recent A/B test result you got from Bingo Card Creator?

Patrick:  I should tell about the Bingo Card Creator story, right? OK. I went on a six‑week America tour recently, partly to drum up consulting business, partly to go see Silicon Valley and meet the people I’ve only been reading about in Hacker News, and partly just a change of place. So I went to New York City, went over to Fog Creek, chatted with them a little bit, and while I was in New York City I went to something called Hacker School.

It’s a bunch of people who are learning to become better programmers. They gave me an opportunity to talk to everybody and to teach a lesson on anything I thought they would find valuable. I think a lot of hackers don’t know about A/B testing, so I did a live demo of A/B testing using A/Bingo and I said, “This is literally how easy it is to make an A/B test. We’re going to live code one for my front page right now.”

“I’ve told you in like two sentences what Bingo Card Creator does and for whom.  Now, I’ve been doing this for five years and none of you all heard about it for more than 10 seconds, but just throw out a new headline for Bingo Card Creator. Try to beat my champion headline,  ’Make Bingo cards on your computer.’”

Someone said, “Why don’t you try ‘Create Your Own Bingo Cards Now?” I said, “That sounds like a great answer. It won’t possibly beat my best thing, but sure, we’ll just try coding it up.” I come back a week later and there was literally a 40 percent lift. 40 effing percent.

Keith:  And that’s on sales.

Patrick:  Yeah, that was tracked direct to sales.

Keith:  That’s not trials, that’s not, “Oh, I want to know more,” that’s direct sales. 40 percent increase to sales.

Patrick:  Yeah. Somebody who knew virtually nothing about my business was able to beat something that I’d worked on for a couple of years. Granted, with A/B testing there’s often a bit of reversion to the mean. You get a confidence interval for whether there’s a difference between A and B, but with the typical way that you do stats with A/B testing, you don’t get a confidence interval on what the percent lift is.

It could be anywhere from say two percent to 80 percent and you wouldn’t know where unless you do more sophisticated statistical techniques that I don’t usually bother doing.  Then, that results in a reversion to the mean effect, where a couple weeks later you’ll see it’s only a 20 percent lift now.  You think “what gives?” It was probably only 20 percent that whole time, you’ve just got a bit of a ghost in the machine there. [Editor's note: This is not the only way reversion to the mean can happen.  For example, it may be the case for some sites that change itself tends perform well, irrespective of whether the changed version is distinguishable from the old version from the perspective of users who haven't seen the old version.  I generally just ignore the possibility of that, but it is certainly possible]

Be that as it may, you would be amazed how often, big, successful, established companies, like 37 Signals, Fog Creek, Google, Microsoft, et cetera, see huge lifts from A/B testing.  They run tests years or decades after having released products and that has a meaningful impact on their bottom line without a huge amount of work.

Something I often tell my clients who have 10‑year‑old products, “How much work would it be to increase the value of this product to your customers by one percent? If you have a team of 20 engineers working on a project for 10 years there’s 200 man‑years invested in it already.  You’ve probably already snipped all the low-hanging fruit in terms of features. Making it one percent better by adding features is going to require five man‑years, 10 man years, 20 man‑years of work.  Making it one percent better in terms of perceived value  can be as simple as changing button copy or changing a headline that you use on your product’s landing page.” [Editor's note: Remember, the last feature added to the product gets seen by a fraction of a percent of the user base, but the headline gets read by most folks who open the site.]

One percent is definitely not the ceiling. You can get 5, 10, 20, 100 percent in some cases. Basecamp has done many A/B tests, but just tweaking the header on the pricing page was worth 40‑60 percent. That should blow your mind. [Editor's note: Not sure whether I was remembering my facts correctly here: just trying to Google for a citation, the only article I'm quickly bringing up is a story about Highrise, another 37signals product, getting +30%.]

Think back. A week ago you were doing something for your business, right? What were you doing last week? Did it get you 40 percent to 60 percent improvement in your business’s bottom line? If not, why weren’t you doing A/B testing instead? When you phrase it like that, it always scares the heck out of me but it’s kind of true. [Editor's note: I often get very handwavy with that phrase "bottom line": SaaS math does not work out such that 50% growth in rate of new account signups results in 50% to the bottom line. Work with me here -- I'm from the land of one-time purchases, where modeling the effects of an A/B test is simple without having to literally start using calculus.]

Keith:  You do need the other stuff, you can’t just A/B test your way to a successful business, but at the same time if you have down time, if you have any amount of time, like we said one line of code, five minutes, it’s not hard to do. If you have that time, you should be doing it.

Patrick:  This should be your routine practice.  If you make a new feature and you’re wondering if users are actually going to respond to it or not, put the new feature in an A/B test. It’s as simple as putting an if block around it. [Editor's note: I'm trying to sell you something, not describe engineering reality here.  Yeah, it can get substantially more complicated than "Wrap that in an if block."  Buy what I'm selling -- you'll be happy you did.]

Keith:  I’m sure a lot of people are thinking, “But then you have to hire a designer and they have to make a mock‑up.” Even though it takes one line of code, it’s a pain to think about all these things. I do graphic design as well, but I show all my stuff to my wife and my friends to get a different perspective.  Even if I don’t agree with their comments, even if their comments are batshit crazy, I test it because the cost of testing is so low.  [Editor's note: I do not endorse testing batshit crazy ideas.]

If it fails, it failed for three days and then it’s off the site and we’re never going to see it again. Sometimes you get nice results. I had a very nicely designed welcome page. My wife looked at it and said, “It’s too busy. Why don’t you try having no background?” Tried no background. 40 percent lift.

Patrick:  You often see in design advice, for example at Smashing Magazine. I love designers — Keith is a designer and Keith is my best friend — but on the medical science scale, the discipline of design is closer to leeches than it is to medicine. I don’t like this design, so let’s add more leeches. We should make design more rigorous, like medicine: we’re going to test a treatment against a placebo and then see if it doesn’t kill people. Less leeches, more penicillin.

There was an article recently on Smashing Magazine that said if your body copy is less than 16 points you’re losing money. That’s a great headline, isn’t it, because that is a prediction about the measurable nature of reality.  So you should be able to test that. I actually did test that. For Bingo Card Creator, I changed the body copy sitewide to 14 points for half of the population  and 16 points in the other half of the population. [Editor's note: BCC has used 14px for several years.]

Sure enough, I got a null result. “Null result” means that after a week and 20,000 people or whatever saw both variations, there did not exist data to conclude that 16 percent was different than 14 percent along the axis I was measuring. Good to know, right? [Editor's note: 49,000 participants, actually.  It was a good week.]

Keith:  I do want to follow that up a little, doing a lot of graphic design myself. A lot of it is gut instinct. There are a lot of rules to design, but when you say something like “anything less than 16 point font is unreadable,” that really has to do with each page. You can’t just take any page, make all the font 16, and you’re suddenly going to have increased conversions. [Editor's note: Bah, designers.]

Smashing Magazine spinning the story in that fashion was essentially linkbait. That’s the whole reason they did that. For some sites it might get you 20‑30 percent. It’s worth testing and in Patrick’s case it didn’t work. It all has to do with the design, it has to do with how the site feels, and for the particular users. Some users may want a green button, some users might want a red button.

If your users are the ones who want the green button and you give them a red button, no matter how well it converts on another person’s page, it’s not going to convert well for yours. For any design for anything you should test it on your own users instead of following just what other people say.

Patrick:  That brings up a funny anecdote about users and culture. One of the versions of my Bingo Card Creator website was designed by a lady in India.  (Editor’s note: Gursimran Kaur, whose name I avoided saying because I am unsure of the pronunciation, having never verbally spoken to her.)  I asked her to design a pair of buttons for me, to represent downloading and creating cards.  She thought that the action associated with creation was getting something from the website.  Think of your hand going out, grabbing something, and taking it back to you.

So her button looked like a palm with splayed fingers oriented upwards on a red background, which in America means “Stop!”:

I had to say, “Hey, in America, this gesture does not suggest ‘Come get this!’ It means ‘No, no, no! Before you click this button think carefully. Do you really want to download this software? It will give your Googles a virus.’” [Editor's note: I am poking light fun of my composite customer, who is not very technical, and as a result might use "the Googles" to mean, variously, the Internet, the web browser, or the entire experience of using a computer.]

We settled on a design of the download button that did not give their Googles a virus.  It probably worked better.

Keith:  You’ve made many buttons since then, of course. You’re very big into the color contrast. You want big, flashy buttons that say download this now.

Patrick:  People make fun of me. I’ve been saying this stuff about A/B testing and whatnot for years. I make buttons bigger and it almost always works better. One of the English as a second language folks in one of my forums was talking one day and he’s like, “I know Patrick loves his big, orange, pancake buttons.” Big, orange, pancake button is my keyword for this now because big, orange, pancake buttons really effing work. Go ahead.

Keith:  Except for these ones you tried last time. You had put these ones in and we were working together and he just turns to me and says, “I have these huge, new buttons and I really like them, but they’re just not converting well.” I look at them, they were really the ugliest things I’ve ever seen. They completely did not fit with the site at all. They were just offenses against God, honestly. [laughs]

Patrick:  Right. I’ve been trying to get Keith to be my cofounder for the last couple of years. We’ll see if that happens.

Keith:  We’ll see if that happens.

Patrick:  One of the reasons I have been trying to recruit Keith for this kind of thing is because there’s this, I don’t know.  Maybe it’s a skill, maybe it is an aptitude, maybe it is something your born with, but it is called taste. Whatever day God handed out the taste I was studying stats for D&D characters or something and totally missed it.

Keith:  For action buttons for Patrick it’s generally bigger, oranger, more pancake‑y. It’s pretty much the order it goes in.

Patrick:  With a little bit of rounding on the side.

Keith:  You love the rounding.

Patrick:  Good pancakes should be rounded and huge. Huge, rounded pancakes.

Keith:  I think your Bingo Card Creator is just going to have a giant orange circle on it that says “Buy now.” You should split test that for April Fool’s Day.

Patrick:  People wanted me to test that for a while. I was about to put an 800 by 600 button on my landing pages, but that would violate the Google AdWords content policies, which is problematic because most of the traffic to the landing pages comes from AdWords.

Speaking of which, interesting result from Fog Creek. Fog Creek decided to replace their FogBugz home page. It’s text‑heavy and tries to sell software to enterprise users. They replaced it with their kiwi logo and text similar to “FogBugz: The World’s Best Bug Tracking Software.  Start Now.” No text.  No screenshots.  No philosophy.  Just “Kiwi.  Start now!”

Conversions went way up. At the same time, they fell off the Internet in terms of SEO.  SEO and A/B testing exist to help the business. The business does not exist to help them. Despite the fact that that A/B test was successful, we couldn’t justify burning their Internet brand to continue it, the result was amusing.

Keith:  Right. Actually, one thing you got from a similar test from Fog Creek is that people love to watch Joel talk.

Patrick:  People love to watch Joel talk.

Keith:  I love to listen to Joel talk. I actually listen to the StackExchange podcast just to hear Joel talk.

Patrick:  This is why we’re doing a podcast right now, by the way, because Keith was so inspired by listening to the StackExchange podcast.

I don’t want to give the impression that all Fog Creek’s optimizations are my doing, by the way. I helped Fog Creek a few times with regards to their marketing, A/B testing and whatnot, but there are people in the company that are doing it pretty much every day. [Editor's note: Fog Creek folks are ridiculously smart and they're a wonderful company to work for.  Does being their data-driven marketing guy interest you?  Get in touch with them or get in touch with me -- they're hiring.]

One page frequently tested is the front page for FogBugz. For example, a page with a flat photo versus a design with a short three‑minute feature video versus a page with Joel Spolsky talking for an hour.  The video is indirectly a sales pitch for FogBugz, but it’s mostly a pitch of how we should do software development better.

People watch that video for the entire hour. This just blows my mind. They’ve shown me a graph of what the level of attention is at various points in the interview. If you can imagine the graph for a Justin Bieber song on YouTube where everybody is listening at the first second and then X percent listen through the entire song, it’s that graph… but stretched out to an hour.

Granted, there’s probably not too many 14‑year‑old girls in the market for bug tracking software. A very different audience wants to watch Joel Spolsky talk for an hour about FogBugz. After they watch him for an hour, they are effing sold. Where is that button? They want it now.

Keith:  Actually, that is an interesting trend. Joel talking for an hour is definitely above and beyond what I think most other companies can do. Video has always been the anathema of landing pages. You never put video on your landing page and stuff. Video now on landing pages is converting so much better than just text and I think it’s really going back to people don’t like to read.

Patrick:  Right. People do not read on the Internet. If you take one thing from this podcast, people do not read on this Internet. [Editor's note: It's a pity that of the tens of thousands of people who see this post, only about five will actually read this comment.  You'll know who they are because they'll race to mention it.]

Keith:  They will click on a play button and listen to a 10 second clip over reading a paragraph.

Patrick:  Yeah. It’s insane. This is one of those things where engineers have a totally different grasp of how the world works because most of us were precocious readers. We devour 600‑page Lord of the Ring books with no problem. That is severely anomalous behavior among the population at large.

If you install Crazy Egg, Crazy Egg will show you what percentage of people actually made it to a particular portion of a blog post or whatnot. The world, in general, is TL;DR after four words. It’s crazy. That’s why headers matter so much, by the way, because if they’re only going to read one sentence, make that sentence count.

Keith:  Crazy Egg, if you use their scroll map, you will have so much wonderful information on your pages, especially if you have any page that requires scrolling for more than maybe three clicks.

Patrick:  We just got done saying people don’t read on the Internet, but man, long copy. Long copy is those lovely pages that have hundreds and hundreds of paragraphs of talking you to death about the product. They just beat you into submission and then give you the buy button and then you actually buy. Do people actually buy from these, Keith?

Keith:  One of my clients has very, very large long, long, long sales pages. I think it’s 68 pages printed. I looked at them and I was like, “There is no way people read these.” While 100 percent do not read the entire thing, 40 percent read the entire thing. 40 percent.

Patrick:  That just blows my mind.

Keith:  It absolutely blows my mind. The interesting thing now is that the sales button is not at the end of the page, it’s somewhere in the middle, and yet people read, go to the sales button, actually continue reading, and come back to the sales button in order to click it. That’s just amazing.

That’s 40 percent. 60 percent either don’t read it at all or just skim around. It’s really cool because you can see exactly where on the page people will stop, even for a slight amount of time. There’s this picture of a kind of cute girl with a blue Mohawk on. I hope I’m not giving out too much but it’s not a public page so I think it’s OK. 60 percent of people stop at that picture. It’s 40 percent, 40 percent, 40 percent and then blue Mohawk 60 percent. The area is just bright yellow on the heatmap. Everyone loves looking at this person.

Patrick:  People love faces, by the way. Dave McClure has a great quote about this.  [Editor's note: profane, amusing, and not paid attention to nearly enough.] If you put faces on anything, you’ll draw attention to both the face and to everything surrounding the face.

One of my buddies, Ian, runs HelpSpot. It’s customer help desk software. He just put a photo of himself on the front page. It’s the size of the author photo at the end of this post — really tiny. He just put a photo of himself under the main content area of both default and the front page with the preexisting copy about I’m the founder, this is wonderful software, here’s why, yada, yada. Just adding the photo increased the number of trials by 20 percent.

I think it’s hardwired in people — biologically —  that we track gazes very well.  If you find photos of people looking at something and you track where their eyes are on the page, all the visitors are going to be looking on that line of sight too. Have that line of sight terminate in the form or the sign up button or whatever it is that you want people to take action on. Really, really works well.

Keith:  There’s a lot of good articles on that on Google if you search for, I can’t remember, what is it eye tracking and product placement?

Patrick:  I think it’s gaze. The magic word is gaze.

Keith:  They’ve done actual eye tracking experiments and it amazing that people will look first at the face and then where the face is looking. If you have a face with someone looking to the left and to the left is your purchase now button, then you have a much higher chance of people clicking on that button than anywhere else on the page.

Patrick:  Speaking about the faces, who here has seen a comely young lady in a headset? Girls in headset photos are the visual equivalent of Cosmo’s top 10 things that your employer doesn’t want you to know headlines. Those persist over hundreds of websites done by hundreds of different highly sophisticated companies because they effing work. Go to HeadsetHotties.com. [Editor's note: Some people claim they thought of Facebook before Facebook.  I thought of HeadsetHotties.  I have really really tiny sour grapes about getting beaten to market.] It’s a real website. Take a look at the comely photos of young ladies with headsets. Put it on your page. A/B test it. You’ll see it go up.

I’ve seen also great variations on that. A lot of engineers told that their software will do 20 percent better stuff for customers if it just had a photo of a young lady with a headset get very skeptical of that.  They’re just not willing to try it on their website for whatever reason. Hey, it’s your business. Ultimately that is your decision.

I once worked for a particular company.  They had an employee who they wanted to get contacted from a particular page.  Let’s call him Bob.  Bob is very much not a female, blonde, 20‑something with a headset, but we put Bob in a headset and took a photo of Bob and just put a photo of Bob on the pages with a line that said, “Bob isn’t pretty, but he actually works here. Send Bob an email.” Bob got a lot of emails.  This was a happy result for the company, since people were practically begging Bob to tell them about a five-figure enterprise solution.

So authenticity works too, but if you’re going to be authentic, be authentic by using a picture with a person’s face. It helps.

Keith:  All right. We’ve gone almost an hour and a half now. Some of this is going to go on the cutting room floor. I think that’s about time to wrap up.

Patrick:  All right. Thanks very much, Keith. We’re going to probably be doing this later. Feel free to drop us a comment either in the comment thingy below or on Hacker News or through the emails. You want to give your contact information? We’ll put it in the post.

[Editor's note: Patrick is patrick@ this domain.  Keith Perhac is k dot HIS_LAST_NAME@delfi-net.com We'll also read your comments on the Internets.]

Keith:  We’ll put it in the post. As always, Patrick is patio11 and I’m harisenbon on Hacker News and Twitter.

Patrick:  Thanks so much for listening everybody. We’ll see you next time.

Keith:  All right. Good talking to you.

Patrick:  Take care.

Keith:  Bye bye.

 

Transcript by CastingWords, with light editing by me.  It cost about $100, and was worth every penny, since it freed up basically an entire day of my time.  Assume any remaining errors are my fault.

42 Comments

Image Ad Blending Works Really, Really Well

So I’ve hung around with SEOs for the last couple of years, including ones who pay the rent based on their ability to convince people to click on AdSense ads, and I’ve learned a trick or three.  One that Google will actually tell you in as many words is to make the ads seem less garishly out of place on your site.  In the words of an Internet buddy, “You want to not look like an ad at the first glance, but to look like an ad on the second glance.”  This way you avoid banner blindness, the phenomenon in which Internet users mentally tune out portions of a website which look like advertisements.

Google says:

We’ve outlined a few strategies below that are designed to decrease ad blindness, the tendency for users to ignore anything that’s separate from the main content of your site. By making these changes, you’ll be making your ads more visible to users. The goal isn’t to confuse users into thinking ads are content, but to get users to see and read the ads so they can click on those that interest them.

That’s why if you buy ads on Google itself, they’ll look something like this:

As you can see, the top ad is styled to resemble the main content on the page, with a bit of a subtle yellow background and the notation that it is an ad there, if you take a gander for it.  (Note that a lot of users think that yellow is used for marking the best results.  Non-technical customers, yay.  If you don’t believe me, watch someone who can’t program Google sometime.)

I buy a lot of ads on the Google Content Network, but I don’t get a say in how they are presented usually.  Since I pay per click, it is in the interests of webmasters to make the ads look content-esque, so that they catch lots of clicks.  They occasionally get clicks with less-than-true-user-intent volition behind them, but that is a cost of doing business for me.

Anyhow, I’m always experimenting with different ways to advertise profitably for my businesses other than Google AdWords, which a) are expensive (I spend something like 50~60% of my gross on a sale at the margin) and b) have limited volume.  I recently was inspired to try something new when listening to a (paid) Mixergy video with Ilya of Mixrank, whose blog you should really be reading if you are interested in online advertising.  The gist of the video was to try negotiating direct deals with advertisers with access to the right targeted demographics, and I’m going down that route as well, but for the moment I wanted a get-my-feet-wet option that was more self-service, so I went with BuySellAds.

BuySellAds basically lets you pick a website (in their network), pick a particular type of image ad inventory on it, and pay the displayed rate for advertising there for a month.  Sadly, their options for inventory appropriate to teachers are rather lacking, but I found one wonderful website accepting ad placements through them: BusyTeacher.  BusyTeacher takes 728×90 image ads on their category pages, like this one.  So I went into Paint.NET to exercise my meager pixel-pushing skills, slapped something together, and submitted it for their approval.  After they approved the ad, my credit card got charged for a month of placement ($135 for an estimated 500k impressions or thereabouts), and it went live.

Let’s Play “Spot That Ad”

You can click on the photo to see the full size version.

 

You probably saw-but-didn’t-see the “Get A College Degree For Easy Loan Payments of Only…” spammy ad in the middle.  If you routinely surf sites visited by middle-aged women, you’ve seen-but-not-seen so many thousands of them that you tune them out.  But you probably didn’t automatically tune out the Bingo Card Creator ad.

Spot it yet?  Hint: it’s the row without the Facebook button.

Is This Evil Or Just Evil Genius?

Once upon a time I was an engineer totally scornful of effective marketing, but I have gradually gotten over it.  After thinking it over, this is aggressive but within my comfort envelope.  The ad is honest about being an ad, makes a straightforward commercial proposition (“Sign up for a free trial”) to an audience that I think will respond well to that, and is pretty true by the standards of marketing copy.  It is designed to catch clicks only from people interested in signing up for a free trial of Bingo Card Creator, and sends them straight to a landing page where they can do just that.

I wish there was a way to dynamically generate the image such that I could provide a more exact star valuation, but in the context of a sponsored placement, “Rated 5 stars by lots” is both non-specific and true.  Lots of people have used BCC, and when I ask for star ratings in internal surveys I get something like 4.8 on a volume of hundreds or thousands.  I think this compares favorably with “9 out of 10 dentists agree” and other pretty banal marketing copy.

So Does It Work?

Oh heeeeeeeck yes.  That ad has a 2.2% click through rate (astronomical by the standards of banner advertising), generates about 500 views of my landing page a day, and of that about 11% or so convert to the free trial.  (This is lower than my landing page average by quite a bit, but that makes sense.  Most people who come to one of my landing pages just got done searching for e.g. “make bingo cards”, so they’re clearly in a bingo mood.  The user here was just looking at a page about generic teaching activities then shown the bingo option, and might not be totally sold on bingo for her classroom yet.)

Anyhow, at about $5 to run on this site per day, the CPA (cost per action = how much money do I spend per free trial signup driven) has been running somewhere in the 8 cent region.  Yowza.  I pay Google closer to 30 ~ 40 cents.  At typical conversion rates to a trial of about 1.8~2%, that means that this costs me $4 or so to generate a $30 sale rather than $20.  Sold!

I just wish there was more inventory available.  Many of the sites in the teaching niche either only do Google AdWords, so I’m already saturating them (and paying a cut to Google), or they only accept advertising through large networks, which tend to favor e.g. brands with $X00k marketing spends and not guys who want to experiment with a few hundred bucks at a time.  I’m following up on the advice to get in touch with smaller sites directly, but I need to hit the sweet spot of “Small enough for my $X00 to matter, large enough that they send me enough traffic such that my time in negotiating an ad buy and preparing a creative is worthwhile.”  For an experiment that looks like it will net in the neighborhood of $1,000 a month in sales for $135 and ~10 minutes of pixel pushing, this one is going in the win column.  I hope to get bigger and better results later.

Anyhow, check out BuySellAds, direct ad buys, and ad blending.  They can be made to work.

34 Comments

I Saw An Extremely Subtle Bug Today And I Just Have To Tell Someone

This post will not help you sell more software. If you’re not fascinated by the inner workings of complex systems, go do something more important. If you are, grab some popcorn, because this is the best bug I’ve seen in years.

Have you ever been logged into a site and get suddenly asked to log in again?  This is one of those minor nuisances of using the Internet, right?  If you’re not technically inclined, you think “Hmm, ghosts in the Googles.  Oh well, here is my username and password again.”  If you are technically inclined, you might think “Hmm, funny, my cookie picked a bad time to expire, or maybe my session was getting stored in Memcached on a machine which just went down.  Oh well, here is my username and password again.”

It turns out that Bingo Card Creator has been doing this pervasively to a fraction of my users for the last few months.  I never noticed it, and no one ever complained.

Here’s the scenario: Bingo Card Creator is a Rails application, originally coded against Rails 2.1.X and then gradually updated with Rails security releases.  Like many Rails applications, it stores sessions in a cookie (using CookieStore), and uses the session to hold only very limited data.  Specifically, it holds the (critical) user ID for logged in users and the (nice to have) pre-login session ID.  I use the pre-login session ID to tie some analytics stuff together on the back end — basically, it lets me associate newly created accounts with search terms and whatnot that bring them to my site.  The exact mechanism for doing that isn’t important to this bug — you just need to understand that the session resetting is a minor nuisance if it only happens once in a blue moon, and a huge nuisance if it happens pervasively.

Subtle Indications My Site Was Borked

BCC maintains a whole lot of internal analytics, because I’m something of stats junkie.  Because BCC is in maintenance mode this year, I don’t actually view the stats on a regular basis — as long as the server stays up and users don’t have any complaints, I let the sleeping dog lie.  (I’ve been busy with other projects.)  Anyhow, one example of such a stat is “Of recently created trial accounts, how many were referred from the Halloween bingo cards mini-site?”  For most of the year, that should be a negligible number.

Except right about on Halloween, when the mini-site sees on the order of 30,000 visits or more.  This usually sells several thousand dollars worth of software.  That is fairly hard to miss, because if several thousand dollars don’t show up in my bank account, I’d know right away.  (Sidenote: I did lose about $1,000 due to an ill-timed server crash while I was on a cross-continental plane ride right during the middle of the pre-Halloween rush. Oof.)  So naturally, several thousand dollars implies a hundred or more sales (at $30 each) which implies thousands of trials, right?

Well, my internal analytics code was telling me that the Halloween site had referred ~100 trials of which 6 converted.   Which means that I should have expected a $200 bump in my bank balance.  Which was not what happened.

I mentally filed this away under “Hmm, that’s odd” but didn’t investigate immediately because I had not lost any money (or so I thought) and was busy that week.  Then recently, after doing an unrelated code push (I integrated Stripe, it is awesome, full details later), I did my usual post-deploy smoke test and, after creating a new account, I got suddenly logged out of the application.

“Hmm, that’s odd.”  And I tried it again, twice, couldn’t produce the error, and mentally wrote it off to gremlins.

In Which I Become Doubtful Of The Existence Of Gremlins

Four hours ago, my brain suddenly decided to put these facts together. The discrepancy for the sales statistics strongly suggests that, prior to accounts getting created, the session was getting cleared.  This meant that, when the account actually got created, the referrer was not associated with the account in the DB, which threw off subsequent stats gathered by my internal analytics.  Sessions getting randomly cleared would also cause the user to become instantly signed out.

I tried to reproduce the problem in development mode and was pervasively unable to do so.  Then I started trying to reproduce it on the live site and was able to, sporadically, but only in Incognito Mode in Chrome, and only if I clicked fairly fast.  (Don’t ask how many dozens of tries it took to figure out that fast clicking was the culprit.)

Having verified that it actually existed, I added instrumentation to tell me what my session ID was, and noticed — like expected — that it changed when I was suddenly logged out.  Sure enough, the session was getting wiped.  But why?

Racking my brains to figure out “What could reset a session in Rails other than explicitly trying to do it?”, I started listing up and discarding some candidates:

  • The cookie expired in the browser — nope, expiry was set correctly
  • The cookie got eaten by the hop from Nginx to Mongrel — nope, after investigation, cookies always matched on both sides (like expected)
  • The cookie got too big and failed to serialize properly — nope, after looking through the Rails codebase, that looked like it would throw an exception
  • The cookie got reset when Rails detected malicious behavior coming from the browser — bingo!

CSRF Protection: When It Breaks, It Breaks Very Quietly

Cross-site request forgery (CSRF) is tricking the browser with a malicious (or compromised) site B to access something on site A.  Since requests for site A will carry A’s cookie whether requested by A or not, an image tag or embedded Javascript on B can do anything on A that a logged-in user can do, like accessing /accounts/wire_all_your_money_to_switzerland with the appropriate POST parameters to make it happen.  This is, to put it mildly, a bad thing.  Rails has lovely magic which defends against CSRF for you: all you have to do is include two lines of code
#In application_controller.rb
protect_from_forgery

#In your templates' HEAD somewhere

Rails will then basically generate cryptographically secure random number, totally transparently to you. This is called the CSRF token.

One copy goes in your Rails session, where only your server and the client can see it.  (n.b. Rails sessions are a bit opaque since they are Base64 encoded, but they can be trivially decoded by anyone who can read the cookie, including the end-user.  They can’t be forged because of another security feature, but don’t put anything you don’t want the user to know in the session.)

Another copy goes in the document’s HEAD (for access via Javascript) and in Rails-generated forms as a hidden value.  When Rails makes a PUT or POST request to the server (via helper-generated form or helper-generated Javascript), Rails will submit the copy included in the HTML code with the request, compare it to the one in the session, and bounce requests where they don’t match. Bad actions on other sites shouldn’t be able to read either a) a page on your site (the same origin policy prevents this) b) the contents of your cookie from your site, so this is secure.

The specifics of how it “bounces requests” are very important.

Point Releases Sometimes Contain Doozies

My personal understanding of Rails up until an hour ago was that a CSRF violation would raise an exception.  This would practically never get seen by a legitimate user operation, so few people are aware of that, but I had seen it a time or two when security auditing BCC.  (Some of my geeky friends had, back in the day, exploited BCC with a CSRF and helpfully told me how to fix it.  Naturally, after fixing it I verified that the site worked as expected with the fix.)

So if the CSRF protection was somehow eating sessions, I would expect to see that exception getting logged and emailed to me by Airbrake (formerly Hoptoad — it emails you when an exception happens in production, highly recommended).   That wasn’t happening.

Then I decided to dig into the Rails source.  Whereupon I learned that Rails 2.3.11 changed the behavior of CSRF protection: instead of throwing exceptions, it would silently just clear the session and re-run the request.  For most sensitive operations (e.g. those which require a signed in user), this would force a signout and then any potentially damaging operation would be averted.

Here’s the relevant code in Rails 2.3.11:

def verify_authenticity_token
  verified_request? || handle_unverified_request
end

def handle_unverified_request
  reset_session
end

Versus the relevant code in Rails 2.3.10 (sidenote: you can see all of this easily in Github because Rails is diligent about tagging releases, a practice you should certainly follow in your own development):

def verify_authenticity_token
  verified_request? || raise(ActionController::InvalidAuthenticityToken)
end

And, sure enough, checking Subversion showed that I upgraded the version of Rails I was using in January of this year in response to this security advisory. I read that, made the required modifications to my application, tested, and had no problems.

So What Went Wrong Then?

After I was sure that sessions were being reset (but only in production), I added a bit of instrumentation to the live site to record the session ID for people coming from my IP address and to log when it changed. This let me find the culprit: a bit of Javascript that A/Bingo, my A/B testing library, uses to verify that people are human. It assumes that robots generally won’t run Javascript engines capable of doing POST requests, so it does an ajax-y POST to my server to assert humanity of the end-user, thus keeping almost all bots out of my stats.

That code has been live over a year. Why did it suddenly start causing session resets? Oh, another change in the 2.3.11 upgrade:

The old code:

  # Returns true or false if a request is verified.
  # Comment truncated by Patrick
  def verified_request?
      !protect_against_forgery?     ||
        request.method == :get      ||
        request.xhr?                ||
        !verifiable_request_format? ||
        form_authenticity_token == form_authenticity_param
  end

Notice that request.xhr? will cause this request to be verified if it evaluates to true, regardless of the other things in the OR statements. request.xhr? tests whether a request is ajax-y in nature. A/Bingo’s humanity-verifying POST is, so it didn’t trigger the CSRF check.

The new code, however:

  # Returns true or false if a request is verified.
  # Comment truncated by Patrick
  def verified_request?
    !protect_against_forgery?                            ||
      request.get?                                       ||
      form_authenticity_token == form_authenticity_param ||
      form_authenticity_token == request.headers['X-CSRF-Token']
  end

Yep, as announced in the patch notes, we lost the exemption for XHR requests. So the A/Bingo mark_human request will, because it makes no particular effort to include a CSRF token (which I will be changing very quickly, as A/Bingo is my project), with certainty cause the CSRF check to fail in 2.3.11. This will result in not a noisy exception (the previous behavior) but instead a silent reset followed by re-running the action. A/Bingo, which doesn’t care a whit whether you’re logged in, will then mark your freshly new session as human. If the previous contents of your session mattered, for example to keep you signed in, they are now gone. A/Bingo will not reaudit your humanity, though, because your session now marks you as human, so this will only ever happen to your browser once.

Race Conditions: Not Just In Java Thread Programming

So why did this never show up in development and why did it show up only sporadically in production? Well, consider how a browser interprets a page presented to it: it first downloads the HTML, then downloads the assets, blocking when it discovers e.g. CSS or Javascript which alters the document. This means that Javascript very low on a page may never execute if someone above it blocks them until the user navigates away. (This is a pretty gross simplification of how multiple pieces of very complicated and often incompatible software do something very difficult. If you want details, read stuff by the people behind YSlow. They’re geniuses and taught me all that I successfully learned about this process.) Someone like, say, external analytics utilities loaded over the public Internet. My page includes a few such scripts, like Google Analytics and CrazyEgg. They are off in development to avoid polluting my stats.

This plus the lack of network latency means that, on development, a browser which sees a page that includes the humanity testing Javascript will almost certainly execute it. That will cause the session to be burned, once, on the first page load. Since my invariable workflow for manual testing is “Start with a new browser at the homepage or landing page, do stuff, make sure it works”, the order of execution is:

  1. I load the front page or a landing page. The session is initialized to some value S1.
  2. (A few milliseconds later.) The A/Bingo Javascript checks for my humanity, resetting the session to some new value S2.
  3. I hit the registration or login button, and the site works as I expect it to.
  4. Since the site knows I am human now, that never gets checked again, and the session never gets reset again.

In production though, the workflow could very well be:

  1. The user arrives at the front page or landing page. The session is initialized to some value S1, including (say) their referrer information.
  2. A bunch of Javascript starts loading ahead of the A/Bingo check.
  3. The user, within one or two seconds (depending on network latency to those external scripts), either logs in or creates an account.
  4. The browser never successfully executes the A/Bingo check.
  5. The user arrives at their dashboard. When it is rendered, the server (robotically) decides it isn’t quite sure if they are human yet, and includes that Javascript again. (This behavior is designed because I was aware of the timing issue, I just didn’t realize how it would shake out with the 2.3.11 upgrade.
  6. This time, the user ponders their dashboard enough for the A/Bingo Javascript to post successfully. This resets their session to some new value S2.
  7. The user clicks anything on the page, and (because S2 doesn’t include their logged in user ID) gets taken to a login screen.
  8. The user is now durably marked as human, so the A/Bingo check never fires again, preventing a second unceremonious logout.

This neatly explains the logged out users. How to explain the missing referrer information? Well, if the user is NOT very fast on the click on the first page, they’ll have their referrer cleared out of the session before they successfully signup. They’ll get marked as a human prior to creating their account, though, so they’ll never even notice the unceremonious logout. This is the behavior of the overwhelming bulk of new users, which is why the stats were getting comprehensively borked but almost no users thought to complain.

This difference in behavior based on the hidden interaction of two concurrent processes is called a race condition. Race conditions are why sane programmers don’t program with threads or, if they do, they use shared-nothing architecture and pass all communication between the threads through a message queue written by someone who knows what they are doing (if you have to ask, it isn’t you — seriously, multithreaded programming is hard). I haven’t seen a race condition in years, because the genre of web applications I write and their architectures makes me mostly immune to them. Well, I just got busted back to CS102. Sadly, the core lesson of CS102 hasn’t changed: reasoning through why race conditions happen is very hard.

Saved By Unsophisticated Users, Sort Of

Users returning after the session naturally expired (2 weeks) would go through the dance again, potentially getting asked to log in twice. However, it took most of them enough time to have the human check prior to finding where the Sign In button was, so the percentage of users who actually visibly saw the bug was fairly small. (I’m guessing, from a quick heuristic run on my log files, that it was below 1% of accounts. That’s the optimistic way to say it. The pessimistic way is to say that this bug negatively affected the better part of a thousand people, and probably cost me sales from some of them.)

Whose Fault Is This?

If my users are inconvenienced, it is my fault, always. I should have read the patch notes for 2.3.11 more diligently, to discover the very consequential line “In addition to altering the templates, an application’s javascript must be changed to send the token with Ajax requests.”, and I should have been more aware that there was a one-line Javascript method pulled in by a library (which I wrote, so that is no excuse) which was not automatically upgraded with the Rails helper methods.

I’m not sure if more diligent testing would have caught this. Race conditions are hard to diagnose, and while I might have caught it by including production levels of external Javascript in my development environment, the symptoms would only have been visible a fraction of the time anyhow, and in ways which didn’t visibly break the application most of the time. (Who checks their stats for the development version to make sure they’re sensible after implementing that function correctly the first time?)

What I really should have done about this is addressing it earlier, when I first got the inkling that there was some weird edge case which would cause a logged in user to become logged out. I futzed around with my configuration once or twice and saw the problem go away (because it was non-deterministic), but rather than futzing I should have figured out a complicated but reducible series of steps that would always cause the issue. That would have sent me down the right road for fixing it.

So How Do You Address This

Immediate term, a one-line patch turns off CSRF protection for the A/Bingo mark_human action, preventing it from accidentally resetting the session.

skip_filter :verify_authenticity_token, :only => :mark_human

I also added a note about this to the A/Bingo documentation. I’ll patch A/Bingo after I have enough brain cells left to do that in a way which won’t break anyone’s applications. After I patch A/Bingo, that work-around won’t be necessary.

Why’d You Write This Post?

Because, after hours spelunking in Firebug, my codebase, and the innards of obsolete version of Rails to understand what was happening, I had to tell somebody. Some people have water coolers. I have the Internet. Hopefully, someone in this wide world will find this discussion useful.

If you’re wondering what the day-to-day life of an engineer is like or why it’s so dang hard some of the time, this might be a good example (of the pathological case — the typical case is writing boring code which solves boring problems, like laying out a 5×5 grid on a bingo card and randomizing the word order). Bingo Card Creator is not terribly complicated software when compared to most applications, but it sits on top of other pieces of code (Rails, the web server the browser, the TCP/IP stack, the underlying OS, the hardware on both ends, etc) which collectively are orders of magnitude more complicated than any physical artifact ever created by the human race.

Most of the time that complexity is abstracted away from both the user and the developer, both as blissfully ignorant of the layers below as an ant walking on an aircraft carrier is ignorant of the depth of the ocean. But when a problem bubbles up and writing it off to gremlins isn’t getting the job done, you have to start looking at the lower levels of abstraction. That is rather harder than dealing with just the higher levels of abstraction. (Joel Spolsky has an article about this subject.)

43 Comments

Don't Call Yourself A Programmer, And Other Career Advice

If there was one course I could add to every engineering education, it wouldn’t involve compilers or gates or time complexity.  It would be Realities Of Your Industry 101, because we don’t teach them and this results in lots of unnecessary pain and suffering.  This post aspires to be README.txt for your career as a young engineer.  The goal is to make you happy, by filling in the gaps in your education regarding how the “real world” actually works.  It took me about ten years and a lot of suffering to figure out some of this, starting from “fairly bright engineer with low self-confidence and zero practical knowledge of business.”  I wouldn’t trust this as the definitive guide, but hopefully it will provide value over what your college Career Center isn’t telling you.

90% of programming jobs are in creating Line of Business software: Economics 101: the price for anything (including you) is a function of the supply of it and demand for it.  Let’s talk about the demand side first.  Most software is not sold in boxes, available on the Internet, or downloaded from the App Store.  Most software is boring one-off applications in corporations, under-girding every imaginable facet of the global economy.  It tracks expenses, it optimizes shipping costs, it assists the accounting department in preparing projections, it helps design new widgets, it prices insurance policies, it flags orders for manual review by the fraud department, etc etc.  Software solves business problems.  Software often solves business problems despite being soul-crushingly boring and of minimal technical complexity.  For example, consider an internal travel expense reporting form.  Across a company with 2,000 employees, that might save 5,000 man-hours a year (at an average fully-loaded cost of $50 an hour) versus handling expenses on paper, for a savings of $250,000 a year.  It does not matter to the company that the reporting form is the world’s simplest CRUD app, it only matters that it either saves the company costs or generates additional revenue.

There are companies which create software which actually gets used by customers, which describes almost everything that you probably think of when you think of software.  It is unlikely that you will work at one unless you work towards making this happen.  Even if you actually work at one, many of the programmers there do not work on customer-facing software, either.

Engineers are hired to create business value, not to program things:  Businesses do things for irrational and political reasons all the time (see below), but in the main they converge on doing things which increase revenue or reduce costs.  Status in well-run businesses generally is awarded to people who successfully take credit for doing one of these things.  (That can, but does not necessarily, entail actually doing them.)  The person who has decided to bring on one more engineer is not doing it because they love having a geek around the room, they are doing it because adding the geek allows them to complete a project (or projects) which will add revenue or decrease costs.  Producing beautiful software is not a goal.  Solving complex technical problems is not a goal.  Writing bug-free code is not a goal.  Using sexy programming languages is not a goal.  Add revenue.  Reduce costs.  Those are your only goals.

Peter Drucker — you haven’t heard of him, but he is a prophet among people who sign checks — came up with the terms Profit Center and Cost Center.  Profit Centers are the part of an organization that bring in the bacon: partners at law firms, sales at enterprise software companies, “masters of the universe” on Wall Street, etc etc.  Cost Centers are, well, everybody else.  You really want to be attached to Profit Centers because it will bring you higher wages, more respect, and greater opportunities for everything of value to you.  It isn’t hard: a bright high schooler, given a paragraph-long description of a business, can usually identify where the Profit Center is.  If you want to work there, work for that.  If you can’t, either a) work elsewhere or b) engineer your transfer after joining the company.

Engineers in particular are usually very highly paid Cost Centers, which sets MBA’s optimization antennae to twitching.  This is what brings us wonderful ideas like outsourcing, which is “Let’s replace really expensive Cost Centers who do some magic which we kinda need but don’t really care about with less expensive Cost Centers in a lower wage country”.  (Quick sidenote: You can absolutely ignore outsourcing as a career threat if you read the rest of this guide.)  Nobody ever outsources Profit Centers.  Attempting to do so would be the setup for MBA humor.  It’s like suggesting replacing your source control system with a bunch of copies maintained on floppy disks.

Don’t call yourself a programmer: “Programmer” sounds like “anomalously high-cost peon who types some mumbo-jumbo into some other mumbo-jumbo.”  If you call yourself a programmer, someone is already working on a way to get you fired.  You know Salesforce, widely perceived among engineers to be a Software as a Services company?  Their motto and sales point is “No Software”, which conveys to their actual customers “You know those programmers you have working on your internal systems?  If you used Salesforce, you could fire half of them and pocket part of the difference in your bonus.”  (There’s nothing wrong with this, by the way.  You’re in the business of unemploying people.  If you think that is unfair, go back to school and study something that doesn’t matter.)

Instead, describe yourself by what you have accomplished for previously employers vis-a-vis increasing revenues or reducing costs.  If you have not had the opportunity to do this yet, describe things which suggest you have the ability to increase revenue or reduce costs, or ideas to do so.

There are many varieties of well-paid professionals who sling code but do not describe themselves as slinging code for a living.  Quants on Wall Street are the first and best-known example: they use computers and math as a lever to make high-consequence decisions better and faster than an unaided human could, and the punchline to those decisions is “our firm make billions of dollars.”  Successful quants make more in bonuses in a good year than many equivalently talented engineers will earn in a decade or lifetime.

Similarly, even though you might think Google sounds like a programmer-friendly company, there are programmers and then there’s the people who are closely tied to 1% improvements in AdWords click-through rates.  (Hint: provably worth billions of dollars.)  I recently stumbled across a web-page from the guy whose professional bio is “wrote the backend billing code that 97% of Google’s revenue passes through.”  He’s now an angel investor (a polite synonym for “rich”).

You are not defined by your chosen software stack: I recently asked via Twitter what young engineers wanted to know about careers.  Many asked how to know what programming language or stack to study.  It doesn’t matter.  There you go.

Do Java programmers make more money than .NET programmers?  Anyone describing themselves as either a Java programmer or .NET programmer has already lost, because a) they’re a programmer (you’re not, see above) and b) they’re making themselves non-hireable for most programming jobs.  In the real world, picking up a new language takes a few weeks of effort and after 6 to 12 months nobody will ever notice you haven’t been doing that one for your entire career.  I did back-end Big Freaking Java Web Application development as recently as March 2010.  Trust me, nobody cares about that.  If a Python shop was looking for somebody technical to make them a pile of money, the fact that I’ve never written a line of Python would not get held against me.

Talented engineers are rare — vastly rarer than opportunities to use them — and it is a seller’s market for talent right now in almost every facet of the field.  Everybody at Matasano uses Ruby.  If you don’t, but are a good engineer, they’ll hire you anyway.  (A good engineer has a track record of — repeat after me — increasing revenue or decreasing costs.)  Much of Fog Creek uses the Microsoft Stack.  I can’t even spell ASP.NET and they’d still hire me.

There are companies with broken HR policies where lack of a buzzword means you won’t be selected.  You don’t want to work for them, but if you really do, you can add the relevant buzzword to your resume for the costs of a few nights and weekends, or by controlling technology choices at your current job in such a manner that in advances your career interests.  Want to get trained on Ruby at a .NET shop?  Implement a one-off project in Ruby.  Bam, you are now a professional Ruby programmer — you coded Ruby and you took money for it.  (You laugh?  I did this at a Java shop.  The one-off Ruby project made the company $30,000.  My boss was, predictably, quite happy and never even asked what produced the deliverable.)

Co-workers and bosses are not usually your friends: You will spend a lot of time with co-workers.  You may eventually become close friends with some of them, but in general, you will move on in three years and aside from maintaining cordial relations you will not go out of your way to invite them over to dinner.  They will treat you in exactly the same way.  You should be a good person to everyone you meet — it is the moral thing to do, and as a sidenote will really help your networking — but do not be under the delusion that everyone is your friend.

For example, at a job interview, even if you are talking to an affable 28 year old who feels like a slightly older version of you he is in a transaction.  You are not his friend, you are an input for an industrial process which he is trying to buy for the company at the lowest price.  That banter about World of Warcraft is just establishing a professional rapport, but he will (perfectly ethically) attempt to do things that none of your actual friends would ever do, like try to talk you down several thousand dollars in salary or guilt-trip you into spending more time with the company when you could be spending time with your actual friends.  You will have other coworkers who — affably and ethically — will suggest things which go against your interests, from “I should get credit for that project you just did” (probably not phrased in so many words) to “We should do this thing which advances my professional growth goals rather than yours.”  Don’t be surprised when this happens.

You radically overestimate the average skill of the competition because of the crowd you hang around with:  Many people already successfully employed as senior engineers cannot actually implement FizzBuzz.  Just read it and weep.  Key takeaway: you probably are good enough to work at that company you think you’re not good enough for.  They hire better mortals, but they still hire mortals.

“Read ad.  Send in resume.  Go to job interview.  Receive offer.” is the exception, not the typical case, for getting employment: Most jobs are never available publicly, just like most worthwhile candidates are not available publicly (see here).  Information about the position travels at approximately the speed of beer, sometimes lubricated by email.  The decisionmaker at a company knows he needs someone.  He tells his friends and business contacts.  One of them knows someone — family, a roommate from college, someone they met at a conference, an ex-colleague, whatever.  Introductions are made, a meeting happens, and they achieve agreement in principle on the job offer.  Then the resume/HR department/formal offer dance comes about.

This is disproportionately true of jobs you actually want to get.  ”First employee at a successful startup” has a certain cachet for a lot of geeks, and virtually none of those got placed by sending in a cover letter to an HR department, in part because two-man startups don’t have enough scar tissue to form HR departments yet.  (P.S. You probably don’t want to be first employee for a startup.  Be the last co-founder instead.)  Want to get a job at Googler?  They have a formal process for giving you a leg up because a Googler likes you.  (They also have multiple informal ways for a Googler who likes you an awful lot to short-circuit that process.  One example: buy the company you work for.  When you have a couple of billion lying around you have many interesting options for solving problems.)

There are many reasons why most hiring happens privately.  One is that publicly visible job offers get spammed by hundreds of resumes (particularly in this economy) from people who are stunningly inappropriate for the position.  The other is that other companies are so bad at hiring that, if you don’t have close personal knowledge about the candidate, you might accidentally hire a non-FizzBuzzer.

Networking: it isn’t just for TCP packets: Networking just means a) meeting people who at some point can do things for you (or vice versa) and b) making a favorable impression on them.

There are many places to meet people.  Events in your industry, such as conferences or academic symposia which get seen by non-academics, are one.  User groups are another.  Keep in mind that user groups draw a very different crowd than industry conferences and optimize accordingly.

Strive to help people.  It is the right thing to do, and people are keenly aware of who have in the past given them or theirs favors.  If you ever can’t help someone but know someone who can, pass them to the appropriate person with a recommendation.  If you do this right, two people will be happy with you and favorably disposed to helping you out in the future.

You can meet people over the Internet (oh God, can you), but something in our monkey brains makes in-the-flesh meeting a bigger thing.  I’ve Internet-met a great many people who I’ve then gone on to meet in real life.  The physical handshake is a major step up in the relationship, even when Internet-meeting lead to very consequential things like “Made them a lot of money through good advice.”  Definitely blog and participate on your industry-appropriate watering holes like HN, but make it out to the meetups for it.

Academia is not like the real world: Your GPA largely doesn’t matter (modulo one high profile exception: a multinational advertising firm).  To the extent that it does matter, it only determines whether your resume gets selected for job interviews.  If you’re reading the rest of this, you know that your resume isn’t the primary way to get job interviews, so don’t spend huge amount of efforts optimizing something that you either have sufficiently optimized already (since you’ll get the same amount of interviews at 3.96 as you will at 3.8) or that you don’t need at all (since you’ll get job interviews because you’re competent at asking the right people to have coffee with you).

Your major and minor don’t matter.  Most decisionmakers in industry couldn’t tell the difference between a major in Computer Science and a major in Mathematics if they tried.  I was once reduced to tears because a minor academic snafu threatened my ability to get a Bachelor of Science with a major in Computer Science, which my advisor told me was more prestigious than a Bachelor of Science in Computer Science.  Academia cares about distinctions like that.  The real world does not.

Your professors might understand how the academic job market works (short story: it is ridiculously inefficient in engineering and fubared beyond mortal comprehension in English) but they often have quixotic understandings of how the real world works.  For example, they may push you to get extra degrees because a) it sounds like a good idea to them and b) they enjoy having research-producing peons who work for ramen.  Remember, market wages for people capable of producing research are $80~100k+++ in your field.  That buys an awful lot of ramen.

The prof in charge of my research project offered me a spot in his lab, a tuition waiver, and a whole $12,000 dollars as a stipend if I would commit 4~6 years to him.  That’s a great deal if, and only if, you have recently immigrated from a low-wage country and need someone to intervene with the government to get you a visa.

If you really like the atmosphere at universities, that is cool.  Put a backpack on and you can walk into any building at any university in the United States any time you want.  Backpacks are a lot cheaper than working in academia.   You can lead the life of the mind in industry, too — and enjoy less politics and better pay.  You can even get published in journals, if that floats your boat.  (After you’ve escaped the mind-warping miasma of academia, you might rightfully question whether Published In A Journal is really personally or societally significant as opposed to close approximations like Wrote A Blog Post And Showed It To Smart People.)

How much money do engineers make?

Wrong question.  The right question is “What kind of offers do engineers routinely work for?”, because salary is one of many levers that people can use to motivate you.  The answer to this is, less than helpfully, “Offers are all over the map.”

In general, big companies pay more (money, benefits, etc) than startups.  Engineers with high perceived value make more than those with low perceived value.  Senior engineers make more than junior engineers.  People working in high-cost areas make more than people in low-cost areas.  People who are skilled in negotiation make more than those who are not.

We have strong cultural training to not ask about salary, ever.  This is not universal.  In many cultures, professional contexts are a perfectly appropriate time to discuss money.  (If you were a middle class Japanese man, you could reasonably be expected to reveal your exact salary to a 2nd date, anyone from your soccer club, or the guy who makes your sushi.  If you owned a company, you’d probably be cagey about your net worth but you’d talk about employee salaries the way programmers talk about compilers — quite frequently, without being embarrassed.)   If I were a Marxist academic or a conspiracy theorist, I might think that this bit of middle class American culture was specifically engineered to be in the interests of employers and against the interests of employees.  Prior to a discussion of salary at any particular target employer, you should speak to someone who works there in a similar situation and ask about the salary range for the position.  It is <%= Date.today.year %>; you can find these people online.  (LinkedIn, Facebook, Twitter, and your (non-graph-database) social networks are all good to lean on.)

Anyhow.  Engineers are routinely offered a suite of benefits.  It is worth worrying, in the United States, about health insurance (traditionally, you get it and your employer foots most or all of the costs) and your retirement program, which is some variant of “we will match contributions to your 401k up to X% of salary.”  The value of that is easy to calculate: X% of salary.  (It is free money, so always max out your IRA up to the employer match.  Put it in index funds and forget about it for 40 years.)

There are other benefits like “free soda”, “catered lunches”, “free programming books”, etc.  These are social signals more than anything else.  When I say that I’m going to buy you soda, that says a specific thing about how I run my workplace, who I expect to work for me, and how I expect to treat them.  (It says “I like to move the behavior of unsophisticated young engineers by making this job seem fun by buying 20 cent cans of soda, saving myself tens of thousands in compensation while simultaneously encouraging them to ruin their health.”  And I like soda.)  Read social signals and react appropriately — someone who signals that, e.g., employee education is worth paying money for might very well be a great company to work for — but don’t give up huge amounts of compensation in return for perks that you could trivially buy.

How do I become better at negotiation?  This could be a post in itself.  Short version:

a)  Remember you’re selling the solution to a business need (raise revenue or decrease costs) rather than programming skill or your beautiful face.

b)  Negotiate aggressively with appropriate confidence, like the ethical professional you are.  It is what your counterparty is probably doing.  You’re aiming for a mutual beneficial offer, not for saying Yes every time they say something.

c)  ”What is your previous salary?” is employer-speak for “Please give me reasons to pay you less money.”  Answer appropriately.

d)  Always have a counteroffer.  Be comfortable counteroffering around axes you care about other than money.  If they can’t go higher on salary then talk about vacation instead.

e)  The only time to ever discuss salary is after you have reached agreement in principle that they will hire you if you can strike a mutually beneficial deal.  This is late in the process after they have invested a lot of time and money in you, specifically, not at the interview.  Remember that there are large costs associated with them saying “No, we can’t make that work” and, appropriately, they will probably not scuttle the deal over comparatively small issues which matter quite a bit to you, like e.g. taking their offer and countering for that plus a few thousand bucks then sticking to it.

f)  Read a book.  Many have been written about negotiation.  I like Getting To Yes.  It is a little disconcerting that negotiation skills are worth thousands of dollars per year for your entire career but engineers think that directed effort to study them is crazy when that could be applied to trivialities about a technology that briefly caught their fancy.

How to value an equity grant:

Roll d100.  (Not the right kind of geek?  Sorry.  rand(100) then.)

0~70: Your equity grant is worth nothing.

71~94: Your equity grant is worth a lump sum of money which makes you about as much money as you gave up working for the startup, instead of working for a megacorp at a higher salary with better benefits.

95~99: Your equity grant is a lifechanging amount of money.  You won’t feel rich — you’re not the richest person you know, because many of the people you spent the last several years with are now richer than you by definition — but your family will never again give you grief for not having gone into $FAVORED_FIELD like a proper $YOUR_INGROUP.

100: You worked at the next Google, and are rich beyond the dreams of avarice.  Congratulations.

Perceptive readers will note that 100 does not actually show up on a d100 or rand(100).

Why are you so negative about equity grants?

Because you radically overestimate the likelihood that your startup will succeed and radically overestimate the portion of the pie that will be allocated to you if the startup succeeds.  Read about dilution and liquidation preferences on Hacker News or Venture Hacks, then remember that there are people who know more about negotiating deals than you know about programming and imagine what you could do to a program if there were several hundred million on the line.

Are startups great for your career as a fresh graduate?

The high-percentage outcome is you work really hard for the next couple of years, fail ingloriously, and then be jobless and looking to get into another startup.  If you really wanted to get into a startup two years out of school, you could also just go work at a megacorp for the next two years, earn a bit of money, then take your warchest, domain knowledge, and contacts and found one.

Working at a startup, you tend to meet people doing startups.  Most of them will not be able to hire you in two years.  Working at a large corporation, you tend to meet other people in large corporations in your area.  Many of them either will be able to hire you or will have the ear of someone able to hire you in two years.

So would you recommend working at a startup?  Working in a startup is a career path but, more than that, it is a lifestyle choice.  This is similar to working in investment banking or academia.  Those are three very different lifestyles.  Many people will attempt to sell you those lifestyles as being in your interests, for their own reasons.  If you genuinely would enjoy that lifestyle, go nuts.  If you only enjoy certain bits of it, remember that many things are available a la carte if you really want them.  For example, if you want to work on cutting-edge technology but also want to see your kids at 5:30 PM, you can work on cutting-edge technology at many, many, many megacorps.

(Yeah, really.  If it creates value for them, heck yes, they’ll invest in it.  They’ll also invest in a lot of CRUD apps, but then again, so do startups — they just market making CRUD apps better than most megacorps do.  The first hour of the Social Network is about making a CRUD app seem like sexy, the second is a Lifetime drama about a divorce improbably involving two heterosexual men.)

Your most important professional skill is communication: Remember engineers are not hired to create programs and how they are hired to create business value?  The dominant quality which gets you jobs is the ability to give people the perception that you will create value.  This is not necessarily coextensive with ability to create value.

Some of the best programmers I know are pathologically incapable of carrying on a conversation.  People disproportionately a) wouldn’t want to work with them or b) will underestimate their value-creation ability because they gain insight into that ability through conversation and the person just doesn’t implement that protocol.  Conversely, people routinely assume that I am among the best programmers they know entirely because a) there exists observable evidence that I can program and b) I write and speak really, really well.

(Once upon a time I would have described myself as “Slightly below average” in programming skill.  I have since learned that I had a radically skewed impression of the skill distribution, that programming skill is not what people actually optimize for, and that modesty is against my interests.  These days if you ask me how good of a programmer I am I will start telling you stories about how I have programmed systems which helped millions of kids learn to read or which provably made companies millions.  The question of where I am on the bell curve matters to no one, so why bother worrying about it?)

Communication is a skill.  Practice it: you will get better.  One key sub-skill is being able to quickly, concisely, and confidently explain how you create value to someone who is not an expert in your field and who does not have a priori reasons to love you.  If when you attempt to do this technical buzzwords keep coming up (“Reduced 99th percentile query times by 200 ms by optimizing indexes on…”), take them out and try again.  You should be able to explain what you do to a bright 8 year old, the CFO of your company, or a programmer in a different specialty, at whatever the appropriate level of abstraction is.

You will often be called to do Enterprise Sales and other stuff you got into engineering to avoid: Enterprise Sales is going into a corporation and trying to convince them to spend six or seven figures on buying a system which will either improve their revenue or reduce costs.  Every job interview you will ever have is Enterprise Sales.  Politics, relationships, and communication skills matter a heck of a lot, technical reality not quite so much.

When you have meetings with coworkers and are attempting to convince  them to implement your suggestions, you will also be doing Enterprise Sales.  If getting stuff done is your job description, then convincing people to get stuff done is a core job skill for you.  Spend appropriate effort on getting good at it.  This means being able to communicate effectively in memos, emails, conversations, meetings, and PowerPoint (when appropriate).  It means understanding how to make a business case for a technological initiative.  It means knowing that sometimes you will make technological sacrifices in pursuit of business objectives and that this is the right call.

Modesty is not a career-enhancing character trait: Many engineers have self-confidence issues (hello, self).  Many also come from upbringings where modesty with regards to one’s accomplishments is culturally celebrated.  American businesses largely do not value modesty about one’s accomplishments.  The right tone to aim for in interviews, interactions with other people, and life is closer to “restrained, confident professionalism.”

If you are part of a team effort and the team effort succeeds, the right note to hit is not “I owe it all to my team” unless your position is such that everyone will understand you are lying to be modest.  Try for “It was a privilege to assist my team by leading their efforts with regards to $YOUR_SPECIALTY.”  Say it in a mirror a thousand times until you can say it with a straight face.  You might feel like you’re overstating your accomplishments.  Screw that.  Someone who claims to Lead Efforts To Optimize Production while having the title Sandwich Artist is overstating their accomplishments.  You are an engineer.  You work magic which makes people’s lives better.  If you were in charge of the database specifically on an important project involving people then heck yes you lead the database effort which was crucial for the success of the project.  This is how the game is played.  If you feel poorly about it, you’re like a batter who feels poorly about stealing bases in baseball: you’re not morally superior, you’re just playing poorly

All business decisions are ultimately made by one or a handful of multi-cellular organisms closely related to chimpanzees, not by rules or by algorithms: People are people.  Social grooming is a really important skill.  People will often back suggestions by friends because they are friends, even when other suggestions might actually be better.  People will often be favoritably disposed to people they have broken bread with.  (There is a business book called Never Eat Alone.  It might be worth reading, but that title is whatever the antonym of deceptive advertising is.)  People routinely favor people who they think are like them over people they think are not like them.  (This can be good, neutral, or invidious.  Accepting that it happens is the first step to profitably exploiting it.)

Actual grooming is at least moderately important, too, because people are hilariously easy to hack by expedients such as dressing appropriately for the situation, maintaining a professional appearance, speaking in a confident tone of voice, etc.  Your business suit will probably cost about as much as a computer monitor.  You only need it once in a blue moon, but when you need it you’ll be really, really, really glad that you have it.  Take my word for it, if I wear everyday casual when I visit e.g. City Hall I get treated like a hapless awkward twenty-something, if I wear the suit I get treated like the CEO of a multinational company.  I’m actually the awkward twenty-something CEO of a multinational company, but I get to pick which side to emphasize when I want favorable treatment from a bureaucrat.

(People familiar with my business might object to me describing it as a multinational company because it is not what most people think of when “multinational company” gets used in conversation.  Sorry — it is a simple conversational hack.  If you think people are pissed off at being manipulated when they find that out, well, some people passionately hate business suits, too.  That doesn’t mean business suits are valueless.  Be appropriate to the circumstances.  Technically true answers are the best kind of answers when the alternative is Immigration deporting you, by the way.)

At the end of the day, your life happiness will not be dominated by your career.  Either talk to older people or trust the social scientists who have: family, faith, hobbies, etc etc generally swamp career achievements and money in terms of things which actually produce happiness.  Optimize appropriately.  Your career is important, and right now it might seem like the most important thing in your life, but odds are that is not what you’ll believe forever.  Work to live, don’t live to work.

281 Comments

How Running A Business Changes The Way You Think

A few months ago I had the opportunity to have dinner with Ramit Sethi.  We shot the breeze about business topics for a little while — optimizing email opt-in rates, A/B testing to victory, pricing strategies, and the like.  Writing and selling software is solidly in my comfort zone and is what I usually talk about on this blog.  If you want that, skip this post.  We are going deep, deep into the fluffy bits!

The conversation wandered into the softer side of things: psychology, and how running businesses has changed us as people.  I told Ramit a story, and he encouraged me to share it a little more widely, so here we go.

Knowing What Motivates You

Have you ever heard the expression “That really pushes my buttons”?  The notion that there is a particular set of things that uniquely motivates a person has always been a very powerful one with me.  I don’t always have the best handle on what I want in life, but I have enough introspection to know where my buttons are.  A big one, for as long as I can remember, is labeled “Praise To Release Dopamine.”  When I was growing up, I was a praise-seeking missile.  I got very good grades.  I got very good at getting very good grades — not because I wanted good grades for their own sake, and not out of some notion that good grades would get me something valuable in the abstract future, but mostly because I learned very early that parents, teachers, and other people I respected would say “Attaboy, Patrick” if I brought home the A.

That particular story might evoke shades of Tiger Mom parenting.  That would be overstating it: I love my parents deeply, they never pushed me into anything I didn’t want to do, and we’re rather paler than most of the folks in the Tiger Mom crowd.  (Though friends in high school often joked that we’re the most Asian white people they know — and seeing as how my siblings and I can together handle Japanese, Chinese, and Korean, that might not be totally off the mark.)

I realized something fairly early on, though: being a good, hardworking student and getting good grades are two very distinct skill sets.  I was not a particularly “good student”, I just got particularly good at being a student.  I got hooked on a lifetime love of optimizing systems when I realized that 89.5% rounds up to 90% rounds up to an A. That meant I could blow off one more homework assignment than if I didn’t understand decimal arithmetic. Math gives you superpowers.

That’s also why I got started writing.  You see, after you set a multi-year expectation of always getting good grades, the praise mostly dries up.  Drats!  But if you particularly impress a teacher with your flair for writing, and she reads it out to the entire class, then you get to beam a little bit and you collect a story redeemable for an Attaboy at the dinner table.  So I got hooked on writing.  Roughly contemporaneously, I got introduced to the Internet, which if you write well and enjoy having an audience praise your ideas is like your own personal Skinner box.  (If anyone ever wonders why I spend a wee bit too much time on message forums, well, there you go.)

Is there a point to that anecdote?  Sort of.  If you have a good handle on what really motivates you, you probably have a good handle on things which do not really motivate you.  At the top of the list for me: money.  Don’t get me wrong, I am fascinated by money, in the same fashion that the engineer in me is fascinated by bits and electricity and experience points and red blood cells and everything else that whirs around animating interestingly complex systems.  The notion of seeking money strikes me a lot like the notion of seeking red blood cells: ewww.  As long as I don’t drop dead from lack of them, honestly, I could take it or leave it.  (At least qua currency.  Money is also the convenient method of keeping score for optimizing businesses, which feels like a game to me.  I really enjoy winning games with complicated rules sets, especially by optimizing the heck out of play, because optimization is often as much fun as actually playing the game.)

Careers as a Multi-Dimensional Preference Space

My preference set is more complicated than the above micro-sketch, but suffice it to say that I have a good handle on what I value.  One would think I used that introspection to actually achieve happiness.  Was I happy a few years ago?  No, I was very unhappy.

There was a point in my life where I thought what I really wanted was a safe, comfortable, professional career at a big freaking megacorp.  I even had the position picked out: I wanted to be the Project Manager in charge of the Japanese version of Microsoft Office.  If that sounds like a curiously specific goal in life, just call me quirky: I had set about optimizing for a safe middle-class career and that looked to be the high-percentage route through the maze to the cheese.  Why was I so concerned with security at the time?  That’s a long story, but suffice it to say it was a combination of encouragement from my family, a culture that I was in which emphasized good grades to good college to secure employment as the thing to aspire to, and some other factors.  So, true to form, I did what I thought I needed to get to that pot of goal at the end of the rainbow: got into good university, check.  Studied Japanese, check.  Went to Japan to perfect business Japanese, check.  Got a job at a Japanese megacorp, check.  Became totally miserable, check check check.

Along the way, almost totally by accident, I discovered that you could open a small software business on the Internet pretty much without asking permission from anyone.  I discovered that I really, really liked almost every aspect of doing business, in the same way that I really, really did not like most aspects of working at a megacorp.

I worked for several years as a Japanese salaryman.  For those of you not acquainted with the term, it basically means that a company owns you body and soul in return for guaranteeing you gainful employment and social status until you die.  I was working 70 ~ 90 hour weeks for very little money doing “fun” things like translating Excel design docs into Java code and valiantly trying to fix programs delivered by an Indian outsourcing team working from design docs translated via Babelfish.  Why does anyone put up with this?  A lot of people, asked that question, would say “Well, that’s just the culture in Japan.”

Let’s talk about a different quirky culture for a moment: you go from a place you like being, surrounded by people you love, to a place where you do not really enjoy being, surrounded by people who you pretend to like but honestly wouldn’t choose to be friends with.  You stay there for half of your waking hours, five days a week.  Why does anyone put up with this?  I don’t know, it’s the culture or something.

I may have said this before becoming a salaryman, but I never truly understood it: the way we work is, essentially, arbitrary.  40 hour work weeks for middling-good salaries are no more a law of nature than 90 hour workweeks for middling-poor salaries.  The story we hear growing up about working hard to get into a good school so you can work hard at a good job so you can be well rewarded isn’t a lie, per se, but it is a story.  A narrative.  There’s a core of truth buried in it somewhere, but we don’t tell it because it is a true story, we tell it because it is a crackling good story.  If you’re committed to the world being a meritocracy, that story isn’t just good, it is practically mandatory.

The world isn’t a meritocracy.

Here’s a story which we’ve all heard before:

This isn’t a particularly true story.  I mean, let’s start fitting some points to the line:

The  graph isn’t to scale and the points don’t matter: pick your own examples if you disagree with these.  The point is that “work harder and you’ll make more money” is bunk, and we know it to be bunk, and yet we tell that story anyway.

There are many, many more points that we could add to that graph: you could theoretically pick from uncounted thousands of tradeoffs on making money versus working hard.  And that is with only two things under consideration!  What if you considered just one more, such as “Where I want to live?”  Maybe you so want to live in a particular small town in Kansas that you’re willing to accept a smaller salary.  Now we have a three-dimensional preference space.  The true preference space for careers has more dimensions than you can imagine, and sliced along most of them, the straight lines are a just a story.

So if we’ve got a mindbogglingly large solution space of possible jobs, some better in some areas than others, and some areas mattering more to us than others, why don’t we just we just make our mental graph just one wee bit more complex, and add in the 3,207th dimension, how happy we are with our lives if we work at that job?  And then optimize for that?  Heck, since the straight lines are basically not real, would that point even be a job, per-se?  Running your own business is an option — and a laundromat is very, very different from selling software to elementary schoolteachers is different from building the next Google.  Would it even be a point?  Maybe it bobs and weaves a little bit (or quite a lot), as we find out more about ourselves, as we explore new options, and as our preference set changes over time.  I cared about job security once, or at least thought I did.  I honestly could not care less about what “normal” people would consider a secure job now.  When we re-rate that, the points which look like good options for me change dramatically.

I’ve been an engineer, and a technical translator, and a teacher, and a paperboy.  These days, I guess I’d answer to “I own a software company.”  But my job, on any given day, is doing something which makes me happy.  (For, again, a complex definition of “happy”, which includes things not strictly related to me, such as “Is the world better for this having been done?” and “Are the people who matter to me well taken care of if I do this?”)

Because if you’re not happy, and you’re not moving to happy, you should do something else.  We should be happy.

It’s All Negotiable

I grew up in a world of simple stories with clear morals.  As far as careers were concerned, straight lines abounded.  Does anyone remember the day we learned that, say, a middle class man goes to work every single day, Monday through Friday, and that days not like that must be celebrations because they’re clearly the exceptions to the rule?  Were we ever told that, or did it just seep in, somehow?

Here’s a radical notion, let’s try it on: never work Thursdays.  Why?  It has a T and an H in it — T +H = no work, that’s just the way it is, end of discussion.

Would any company ever go for that?  Five years ago, I would have said that is ludicrous.  Everyone works Monday through Friday.  Well, OK, technically speaking university professors could schedule classes and office hours such that they never did anything on Thursday.  And most of the cafes in my town are closed on Thursday.  And teachers get three whole continuous months of Thursdays off.  But that doesn’t matter — the rule is, you work on Thursday.

There may well be excellent reasons why people work on Thursday.  Other people also work on Thursday, so that is very convenient to their company if their workers are around to answer inquiries on Thursday.  But what if the company just said “Sorry customers, no service on Thursdays.”  That would work much of the time — it does work, if you’re a sushi shop in Ogaki.  And it wouldn’t necessarily even be the entire company.  It could just be Bob.  T + H = Bob isn’t there on Thursdays, deal with it.

But what company would deal with it?  Isn’t it written in some book of laws or manual somewhere that Bob has to work on Thursday?  Probably not.  Actually, it might well be written down somewhere, but we ignore things that are written down all the time, so Thursday seems as good a thing to ignore as any.  This is one of the things being a business owner has taught me: there are two parties in any negotiation, and if they agree on one particular point, then that point goes the way they agree.  If Bob and the company agree that Bob doesn’t work on Thursdays, then Bob doesn’t work on Thursdays and damn what the “standard” contract says.  All Bob needs is negotiating leverage to convince his company that agreeing with him on the Thursday issue is in their interests.  Maybe he trades them a salary cut.  Maybe he’s just the best Bob there is and putting up with his idiosyncrasies is worth having him on board.  Maybe just, when push comes to shove, nobody really cares about Thursdays at all.  Whatever.  If Bob wants it, and asks for it, and gets it agreed to, Bob gets what he wants.

You can often get what you want.

Let’s take an example a wee bit less far-fetched than Thursdays.  Let’s say you’re an engineer and you want to be able to pick your technology stack, always.  You have many options here.  You can found your own company, and then you use whatever technology stack appeals to you.  You could consult only on the technology stacks you like.  You could work for a job and just tell them “I quit if you ever put me on a project using a stack I don’t like.”

That might be considered a little disloyal, for a certain value of disloyal which occurs only in the context of a particular type of commercial relationship.  Nobody expects the company to be loyal to their suppliers.  The company expects you to be loyal to them, largely because it is in their interest, and they will often do a lot to convince you that their values are your values.  Is this synthetic, external value really one of your values?   After much consideration, I’ve come to a conclusion about company loyalty: stuff company loyalty.  Companies are legal fictions which we find convenient to use to move capital around and balance accounting ledgers.  I’ll save my loyalty for people.  You’re welcome to your own preference set on this: stick with the company if that is more important to you than working with your favorite technology stack, but make that your choice, not somebody else’s.

Working five days a week?  Your choice.

A job which doesn’t excite you?  Your choice.

Money insufficient to buy whatever it is you want?  Your choice.

Thinking About Money

I come from a fairly modest background, and remember distinctly that one of the perks of my first job was that I didn’t have to worry any more about buying something below $5.  Then I got my first real job, and my care floor went up to $20.  Then my business started doing well, and video games slid under at $60.  Sometime recently I realized that trans-Pacific plane tickets were no longer an expense I needed to save for months for.  So it goes.

You can imagine that if $3.50 is a meaningful amount of money to you, you lead a fairly frugal existence.  I used to live like a monk, and I even idealized that a little bit, but I’ve come to realize that not-spending-money is not something I particularly value one way or another.  Back when my business had no budget, I would gladly do web designs by myself, even though I don’t particularly enjoy it and I am not particularly good at it.  These days, the business is doing rather well, so if I need a web design done I’ll just pay someone who is good at it and likes it, and I’ll get back to doing more important things like strategizing or doing marketing or maybe reading a supernatural romance.  (internal monologue: “But Patrick, you’re not allowed to read in the middle of the day! … Oh wait, I guess I am.”)

For a business, money is a tool to get things that you want.  That started creeping into my personal life as well.  Consider my apartment move in February.  I could have spent a week boxing things up and cleaning my apartment.  Or, I could have paid $2,000 and gotten people to do it for me.  Earlier in my life I would have thought $2,000 is an incredible amount of money (almost a month’s salary!) and that there was something vaguely lazy about not doing the work myself.  The businessman in me made the fairly simple calculation that I could pay the movers $2,000 and get back to charging customers something rather more, and I would not be exhausted and bored at the end of the day.  Here is the money, gentlemen, I’ll be programming at the cafe if you need me.

Confidence Issues

I have longstanding confidence issues.  I often feel like maybe I’m just getting praised for playing the game really well as opposed to demonstrating actual merit.  (I have since found that many, many people I respect likewise worry they’re faking it.  Anybody in my audience got the same issue?  If so, and if you respect me, there you go — you’re not alone on this one.)

I have found that actually showing confidence issues, on the other hand, does not do great things for one’s business.  For example, let’s say you’re doing pricing for a product or service your offering — maybe your software, maybe your labor, whatever.  If you have confidence issues, you will undercut your own negotiating position by underpricing.  Been there, done that, got the T-shirt.

I do consulting, on occasion, and it makes up most of my income and rather little of my day these days.  Consulting is a topic for it’s own post some other week, since I learned by a combination of mentorship and making expensive mistakes and would love to save other folks with less access to good mentors some pain.

At some point in discussing a consulting engagement, you have to quote a price to the client and get them to say “Yes.”  You could write entire books about the psychology of that fifteen seconds, but in a nutshell, you don’t want it to cause you to think “Oh my goodness I’m not really good at this and charging any money is basically stealing so maybe I’ll give them a discount to my lowball offer and then work extra hours to make up for how terrible I’m being right now.”

I got clued into consulting by Thomas Ptacek at Matasano, who has gone from Internet buddy to client to personal friend.  It’s a funny story: I spend too much time on Hacker News, he spends too much time on Hacker News, we both had reason to be in Chicago, so I suggested we get coffee and talk about… whatever message board geeks talk about when they get coffee, I don’t know.  He was very receptive to the idea, so back in December 2009 we grabbed a coffee and went back to his office to drink it.  He then locked me in his conference room for three hours and proceeded to interrogate me with one of his co-workers about everything I knew about SEO and marketing.  It was, seriously, the most fun I had all year.  (Remember, praise-seeking missile.  That entire situation was basically pure brain crack for me and I was getting a free drink out of the deal.)

At the end of the discussion, Thomas mentioned that, if I had phrased my invitation as “Why don’t I consult for you” rather than “Why don’t we grab coffee”, he would have gladly written me a check.  I had a vague idea that a semi-decent engineer (my mental positioning for myself at the time) was worth about $100 an hour, and mentioned that $300 didn’t seem to be enough money for either of us to worry about.  He told me that he would have paid $15,000, and it blew my mind.  (There was a bit of discussion on that point — his coworker thought that to justify $15,000 they’d really need a written report, but the petty cash drawer could have maybe swung $5,000 for it — and my mind got blown again.)

I would like to say that after starting consulting I went immediately to charging $5,000 an hour, but sadly for my bank balance, not so much.  I distinctly remember the first time I quoted someone, though, over email.  I typed a number, agonized over it for ten minutes, then typed a number 50% higher, then curled up in the fetal position for a few minutes, then impulsively hit Send and started hyperventilating.  Ten minutes later I got an email back and, to my enduring surprise, they did not hate my guts for picking that number that was 50% bigger than a number I was already barely comfortable with.

Since then, I’m gradually getting better at the confidence thing.  Partially, it is as a result of trying new things for me and realizing that, no matter how intimidating they were when I started, I largely don’t end up dead.  Consulting engagements go well.  When a new client asks what my rates are, I pick a bigger number.  They’re generally OK with it.  If not, oh well, it turns out there are often many ways to negotiate such that we’re mutually happy.  (Including me just letting them go.  I mean, I don’t need to work everyday.)

Speaking of confidence, sending email to people used to be a bit of a hurdle for me.  Even asking Thomas out was on the edge of my comfort zone.  After trying it many times since then, and observing the behavior of other people in business who I admire, I have come to realize that it is not a big deal.  (Relatedly, almost nothing is as big a deal as we think it is.)  This comes up over and over for young engineering-types on Hacker News so, if you get one actionable piece of advice out of this, get comfortable with sending email to people and asking them to give you what you want.  This will virtually never cause a mortal catastrophe.  I still have a ways to go to do it routinely, but it gets easier every time I do it.  Definitely explain to them why giving you what you want is in their interest, but ask.  They might say “Not interested”, or they might not reply, but they’ll very rarely put a contract out on your head.

Can Your Job Change Who You Are?

After you have asked a CEO or two to coffee and not gotten your hand bitten off, you would be amazed how less scary “Say, why don’t we have dinner together two weeks from now?” is.  That could, possibly, have been the most important sentence of my life to date.  She didn’t put a contract on me, either.

(I generally keep my work life and personal life fairly compartmentalized, so I won’t talk too much about that, but a funny anecdote: we were talking about money one day and future career plans.  She told me she’d continue working at an office job which she doesn’t particularly enjoy so that we’d have enough money to support a family.  I was confused for a few seconds and then started laughing: I had shared less with my girlfriend about my finances than I routinely publish on the Internet.)

And if you’re getting comfortable with addressing folks as your equal and reaching mutually acceptable solutions, why ever stop?

I have to deal with city hall from time to time.  I used to be very timid about it, afraid that I was wasting their time or about to step on a bureaucratic landmine.  These days, they’re a service provider and I’m a customer — a customer with limited options for changing providers, granted, but a customer.  CEOs get what they want.  When I go to city hall, I put on my suit, slide into professional mode, and get things accomplished.  They can’t do something?  Of course they can, or they can find someone else to do it, or there is a particular fact which they need from me to justify doing it, or maybe I’m just asking the wrong way and I should try again.  And I don’t have to be intimidated or embarrassed about doing this — it’s just a negotiation, and not even a really important one, at that.   Plus, if worse comes to worse, I could always just send a lawyer to do it.

I missed a flight a month ago, and got one sentence of a very stern dressing-down from the clerk at the Delta counter.  Years ago, that would have paralyzed me.  I was totally unphased: my brain immediately supplied “They likely sold the seat to someone on standby and, if not, Delta’s revenue maximization is not your moral imperative.  You paid them for a service.  The price of your ticket and standard social graces is the maximum extent of your obligation to them.”  I immediately cut in with “Sorry for the inconvenience ma’am, these things happen.  What are our options?”  (It turns out that if you fly ~100,000 miles a year, your options include them putting you on the next flight, for free, and getting a discount on your hotel room that evening.)

My friends also tell me that I’m almost a different man these days than even two years ago.  The most striking quote to me, from my best friend: “You look… healthy.”  Apparently the old day job was beating me down so thoroughly that I looked about as bad as I felt, and even on those days when I wasn’t dog tired I walked with a bit of a stoop.  These days, I even stand straighter.  I think that is almost too convenient to be true, but hey, it’s a story.

This Actually Can Matter

If you don’t have a good handle on what you want, or even worse, you don’t actually consult it, you could make decisions which are not really in your interests.

I have been somewhat seriously kicking around the idea of taking investment recently.  Thankfully, at Microconf, I got the chance to bounce some ideas off some smart people, including Hiten Shah.  He asked a very perceptive question: why did I want to take investment?  And honestly, I didn’t have a really good answer for that.

“I have the ‘ran a small business’ merit badge already, and am looking for new challenges.”

“Is 10x-ing your business a challenge?”

“Yeah.”

“Do you need investment to do it?”

“Nope, I have done it before.  Besides, even 10x-ing it wouldn’t produce a favorable outcome for most investors.”

“Then why not skip investing and continue doing what you love?”

“…”

The business is growing quickly.  I’m bringing someone on in August.  My life is better than it has ever been.  Taking investment would commit myself to a few years of a very different trajectory, including some things that I have no reason to assume I would enjoy, like managing employees, getting involved with intrapersonal conflicts, and having to give up ownership (not the equity, the responsibility) of parts of the business that I genuinely enjoy to focus on CEO-ing.  In return, yeah, I guess you get a quirky kind of social status and a shot at getting a lot of money, but if I had so much money that a dragon would be embarassed sleeping on it, very little about my life would change.

I’m pretty happy with where I am.  Today rocked.  Bring on tomorrow.

 
50 Comments

Software Businesses In 5 Hours A Week: Microconf 2011 Presentation (1 hour)

Rob Walling (who wrote a book on starting software businesses that I enjoyed) and Mike Taber produced Microconf, a conference for small solo software entrepreneurs.  That sounded right up my alley, and I was extraordinarily happy when asked to speak at it.  The organizers have generously given me permission to post the slides and video of my talk.  (Sidenote: editing videos is going on my Never Doing This Again list.  I should have just thrown a few hundred dollars at someone and saved six hours.  It probably would have ended up better, too.)

If you have an hour free, I recommend the video.  I am told that it was funny, though the genre of humor was very different than my Business of Software presentation.  If you can’t take an hour to watch it, though, you can certainly just read the slides and my commentary below.

How I Ended Up In Central Japan

I have long wrestled with fairly severe self-confidence issues.  (The psychology of entrepreneurship is the major theme of an upcoming post, inspired by a talk I had with Ramit Sethi, who also spoke at Microconf.)  When I was in college, I knew I wanted to be a software engineer, but I was worried about my job prospects competing with 100,000 engineers graduating every year in China and India.  My family was very big on me getting a nice stable job at a megacorp, and I didn’t think I had the chops for it.  So I played the Venn Diagram game: if I could do one very hard thing plus engineering, the intersection of those two would be only a half dozen people, and I’d be set for life.

Learning languages is very hard, so I went down the list that my university offered, and Japanese jumped out at me.  The US and Japan trade billions upon billions of dollars of high-tech stuff every year.  Virtually no Americans speak Japanese.  Practically no Japanese people are fluent in business-level English.  Bingo, if I spoke Japanese and can do engineering, I thought Microsoft (my favored employer at the time) would have to put me in a nice safe job in the Office group for the rest of my life.  So I doubled majored in the two.

Quick tangent: A major multinational advertising firm with an anomalously high number of PhDs on the payroll recently approached me about being a Product Manager in Japan, so this strategy really does work.  It’s funny, three years ago that would have been my dream job and now it was totally untempting.

Anyhow, after graduating college, I was still not confident in my business Japanese, so I decided to go to Japan, work in a Japanese office, and firm it up.  Did I think I could actually call the VP of the multinational who gave me his card and said “Call for a job when you graduate?”  No, confidence issues.  So instead I applied to an international exchange program, which places mostly English teachers and, crucially, some translators at Japanese governmental and quasi-governmental institutions.  One of them was the prefectural technology incubator in Gifu.  (I’m not comfortable telling you which one, but suffice it to say that really narrows it down.)  They took me on as a technical translator, in the expectation that their quickly-growing incubated companies would need technical translation to close big deals with foreign companies and governments.

Gifu is the Kansas of Japan, with more rice and less white people.  I’ve lived here for the last seven years and love Ogaki, my adopted home town, to pieces.  Being a technical translator, however, was not very professionally fulfilling.  Professional ethics require you to translate everything exactly, without elaboration.  I like to think I have something to add to the conversation, so when my contract elapsed in 2007 I switched to being an engineer at a Japanese megacorp.  Prior to doing that, though, I launched Bingo Card Creator.

From Humblest Beginnings

BCC was originally the hobbiest of hobby projects.  One of my assorted job duties as a heavily-underused technical translator was to help out the prefecture’s 200-strong mailing list of (foreign) English teachers, who didn’t speak Japanese and as a result often had issues with coworkers, landlords, government, and the like to muddle through.  Someone asked the mailing list how to make bingo cards for an activity she had planned for later in the week.  I told her to Google it.  She told me that she had and that Google was showing her solutions which were grossly inadequate to her needs.  So I got permission from my boss and spent the rest of the day putting together what might today be called the Minimum Viable Bingo Card Creator.

It was terrible: a Java swing app, distributed as a .jar file, which would accept words in one text-box and, when you hit Print, dump a directory of card0.html … card29.html and ask you to print them from IE because I didn’t know how to actually do that in Java at the time.  But it did actually create bingo cards, and they were of sufficient quality to give to a 7 year old Japanese kid without feeling embarrassed, so I sent it to the mailing list, and went home for the day.  I thought that was the end of it.

The next day, I had 60 emails in my inbox when I got into work.  They were split 50/50 between “THANK YOU!  BEST SOFTWARE EVER!” and “THIS SUCKS!  IT DOESN’T WORK ON MY MACHINE! FIX IT BECAUSE I NEED IT NOW!”

So later in June 2006, when I decided to create a business on the side to try my hand at the SEO/AdWords/etc stuff I had been reading about, bingo jumped out at me as a good topic for software.  I mean, if I could find 60 people who wanted it in Gifu, surely there must have been a market back in the US.  So I budgeted $60 (one video game) and one week to rewriting and productizing it (outside the day job this time, naturally), and set myself a goal: some day, after months of work, I wanted to make $200 in sales a month.

Since I had been inspired by other tales of success on the Internet, I started blogging (you’re reading the result, 5 years later) and publishing my statistics, including sales.  You can see annotated graphs in the slides, so I won’t put them in this post.

Early Days: Filling A Hole In The Internet

BCC exceeded $200 in sales in its second month, largely on the strength of two pages I wrote about Dolch sight words bingo.  (Not an English teacher?  No problem.  Dolch was an English pedagogist who compiled lists of the 220 or so words early English learners need to know on sight.  Teachers know they should teach these but often don’t know which words are on the lists for what year.  I put lists of them online and monetized them with self-ads for the strongly-related bingo activity, on the assumption that almost any teacher wanting to teach them would want a review activity, too.)

This was a good thing, since I had no budget at the time for AdWords.  The success of the content marketing also clued me into one of the core features of the software: writing pre-made word lists that shipped with it, so that teachers didn’t have to type up their own.  So I spent the next year or so in very part-time fashion improving the software, launching new versions, polishing the site, and generally learning more about running a business.  (“Schedule C?  What is that?  Ooooh.”)

Got Google AdWords To Work

In 2007, I started trying my hand at AdWords.  It was a fistful of fail — I could not seem to get either positive ROI or meaningful volumes for the life of me. A buddy of mine from the Business of Software forums advised me to try the Content Network (i.e. ads on sites other than Google.com).  I had turned this off, as prevailing sentiment on the Internet was that the Content Network was a hive of scum and villainy, filled with spammers and MFA (Made For AdSense) sites which sent traffic that did not convert.  But my buddy was sufficiently credible that I trusted him…

… and that ruined my summer.  (His advice ended up turbocharging my business, so I’m retroactively happy for it, but try telling that to me at the time.)  See, every day after coming home from work I would check into AdWords, and every day I would have a new list of spam sites to have to manually ban.  They sent non-converting traffic and I didn’t want to subsidize them.

Towards the end of summer, Google came out with Conversion Optimizer.  In brief, it automatically increased your bid on sites/keywords which sent traffic that converted and decreased it on sites/keywords which didn’t.  This meant that non-converting traffic from spam sites essentially got optimized away without me having to manually ban it.  I loved that, and became an early adopter, writing a pair of blog posts on it.

Concurrently with adopting Conversion Optimizer, October rolled around.  Halloween happens at the end of October, and hundreds of thousands of teachers look for a Halloween activity to play with class.  (Why Halloween, over every other holiday?  Because it is kid-focused, because kids are in school for it, and because as a largely secular holiday it can’t get public school teachers in trouble.)  This meant that sites with content responsive to Halloween bingo, like about.com (which was a content farm before content farming was cool), suddenly had hundreds of thousands of page views to sell AdSense against.  And who was in the front row of the auction for halloween bingo ad impressions?  Me, because Conversion Optimizer figured out that I was making out like a bandit and aggressively moved to spend my money.

Sentiment on the Internet towards Conversion Optimizer had been primarily negative, but I was killing it with it.  My blog post also ranked #3 for Conversion Optimizer right below two posts on google.com… above much of the official documentation.  I think that was probably what clued the Product Manager into talking to me.  Anyhow, to my total surprise, Google asked to do a white paper about my experience with the product.  That was my proudest professional accomplishment for a while, actually.

Content Marketing Seems To Be Working Out… Let’s Scale It

So I was doing well for Halloween bingo in spite of not having any page about it (remember — AdWords ads only), and had done even better for Dolch sight words.  If only I could make a page about Halloween bingo, and Thanksgiving bingo, and addition bingo, and any kind of bingo a teacher could possibly want to play.  Then instead of paying Google to lease the traffic, I would get it for free myself, forever.

This struck me as an unachievably huge amount of work while full-time employed, so I decided to partially automate and partially outsource it.  I taught myself Rails and rewrote the website as a Rails application (rather than 100% HTML-written-in-Notepad), then wrote a script that would populate the Cards table by reading in text files.  Each card got its own page on the website, complete with image of the card, downloadable PDF of 8 randomly created cards, and copious oppotunities to download the free trial of my software.

Creating the GIF and PDF was originally very difficult: you had to use BCC, print to a virtual PDF-ing print driver, open the PDF, screencapture it, crop the capture manually, and then send me the words you used, the resulting GIF, and the PDF.  Repeat thirty times over.  My freelancers understandably got bored, so I had someone write a script which would use a particular Windows macroing utility to drive my laptop’s mouse and do the work.  This took about an hour to get through 30 cards, and required my presence if the script broke in the middle (which was “often”), but it still cut production time down by 90%.

This ended up working out scandalously well for me.  See Scalable Content Creation under Greatest Hits if you want the story in detail.  (I also did a video with Andrew Warner and AppSumo on the topic, if you want it described in a more organized fashion than my blog’s usual stream-of-consciousness approach.)  Eventually, after optimizing the process, I had nearly a thousand pages like this created.

I also have a variety of micro-sites written on exact match domain names, like my favorite, Halloween bingo cards.  Honestly, they’re not that material to my strategy anymore, but if you want to hear more about them see the blog from a few years back.

On Being A Salaryman

Around this time my contract elapsed at the cushy translation job, where I left at 4:00 most days, and I got a job as an engineer at a megacorporation in Nagoya.  No, not that one… not technically, at any rate.  Somewhat to my surprise, the job they offered was as seishain,  which means full-time company employee.  The more commonly known coinage for this status is salaryman, because the job is designed to take over your life.  And take over my life it did.

I rush to point out that I have no ill-will against my old employers: they treated me fairly, by the standards of Japanese corporations, and I learned a lot at that job.  I had my eyes wide open going into it, too — I just didn’t realize how bad 70 ~ 90 hour work weeks would actually be.

This was intended more as a tangent for the speech, but I did more than a bit of venting in the video, much of it humorous.  See that for the full version.

Web Applications Are The Bomb

By 2009, I had advanced sufficiently in my Rails and web programming skills that I could re-release BCC as a web application.  That decision roughly doubled sales, largely due to increased conversion rates both to the trial and from trial to purchase.  I strongly, strongly, strongly suggest developers build web applications in preference to desktop apps, for reasons I have gone into before.  Or, alternately, see this bingo card:

Web apps: do ‘em.

Quitting The Day Job

The combination of these and a hundred other smaller improvements (A/B tests, etc) eventually got my sales to the point in late 2009 where I could seriously consider quitting the day job.  I went home for Christmas, talked it over with my family, then came back and told my bosses that I was through.  They let me go with a mere four months notice.  (Theoretically, the law only requires two weeks in Japan.  In practice, well, see the video.)

Also at Christmas I had a conversation with Thomas Ptacek at Matasano (conveniently in Chicago, close to my family), who opened my eyes regarding consulting.  I owe Thomas a lot for that, because consulting turned quitting from a dicey proposition (a dip in sales could have imperiled my ability to fly home or expand the business, to say nothing of making rent) into a total no-brainer.  Last year I made a bit more from BCC than consulting.  This year I’ll make quite a bit more from consulting than BCC.  The goal is still building a software product business (my current focus on that score is Appointment Reminder), but as of late the caliber of clients, work, and paychecks for consulting has been so attractive that I have been unable to say no.

Tactical Advice

The last half of the presentation was tactical advice on running a small business in one’s spare time — over the first 4 years of doing BCC, I averaged about 5 hours a week on it.  (This last year, even less: it is in maintenance mode.  I send out customer support emails and that is about it.)

The five quick hits:

Charge more money.

Most engineers severely undercharge for their products.  This is particularly true for products which are aimed at businesses — almost all SaaS firms find that they make huge portions of their revenue from the topmost plan which is bought by people spending other people’s money, but instead of optimizing for this we optimize for charging “fair” prices as determined by other software developers who won’t pay for the service anyway.  This is borked.  Charge more.

Make it a web app.

Covered above.

Put more of your iceberg above the water line.

Businesses create value with almost everything they do.  The lion’s share of the value is, like an iceberg, below the waterline: it cannot be seen from anywhere outside the business.  Create more value than you capture, certainly, but get the credit for doing so, both from Google and from everybody else.  This means biasing your business to producing value in a public manner.  Did you write software not core to  your line of business?  Great, OSS it.  Get the credit.  Have you learned something novel about your customers or industry?  Write about it.  Get the credit.  Are your business’ processes worthy of emulation?  Spread what you know.  Get the credit.

37Signals is amazing at this.  You can do it, too.

Get good at SEO.

I talk about this extensively on my blog.  In a nutshell:

  1. You need more links.  Create ways to justify people who aren’t in a commercial relationship with you linking to you anyway.  My favorite way for doing this is getting the credit for things you do, as described above.
  2. Create quality content at scale which solves problems for people in your niche.  See earlier discussion on Scalable Content Creation.

Optimize Everything

I’ve blogged extensively on A/B testing and funnel optimization (see Greatest Hits).  The big take away is, as Steve Pavlina said, all factors in the success of a software business are multiplicative.  If you increase conversions to the trial by 10% and conversions to sale by 10%, your sales go up by 21%, because 1.1 * 1.1 = 1.21.  This is awesomely powerful, particularly for businesses which don’t require hockey-stick trajectories.  You can hill-climb your way to very, very nice places in life for a one-man shop or small company.  (I mean, what real company offers 70% raises per year just for doing an A/B test every week and collecting a +5% improvement on one out of every four?)

Outsource / Automate / Eliminate To Actually Do It In 5 Hours A Week

I have previously written about Outsource / Automate / Eliminate extensively on my blog, so see here and here.

Comments?

I’d love hearing what you thought of the presentation.  I sincerely enjoy talking to people about this and other topics, so if there is a topic you’d like to hear more (or less!) on in the future, tell me and I’ll try to work it in to future presentations.  I never deliver the same one twice.

21 Comments
Loading...
Free video + email advice on making & selling software:
(1~2 emails a week.)