Archive by Author

Developing In Stockfighter With No Trading Experience

Starfighter is a company which makes fun programming challenges. One of our goals is inspiring engineers to take a whack at problems they might assume are “too difficult for me.” Both sets of levels for our first game, Stockfighter, give copious opportunities for this: one set has you do algorithmic trading and one set has you do low-level C and assembly coding, reverse engineering, and security research.

In my experience, the modal web developer probably does not believe they can do algorithmic trading or reverse engineering of assembly code. We strongly disagree: every great developer you know got there by solving problems they were unqualified to solve until they actually did it. That’s why we’re making an environment to let you sink your teeth into fun, hard problems at your own pace, in a supportive community, with us taking care of the scutwork so you can focus on the intellectually interesting bits.

I wrote the algorithmic trading levels (with, I rush to add, no background in finance myself), so I thought I’d write a little bit about how to get started with algorithmic trading for a generalist programmer.

(If this is the first time you’re hearing about Starfighter (the company) or Stockfighter (our first game), you may wish to read or why and how we’re intent on spending the next few years of our lives fixing dev hiring. If you’ve heard of us before and are wondering “Yeah yeah, when do you launch?”, the honest answer is “We bit off a very aggressive engineering schedule between building a stock exchange and an entire C toolchain. The last few months have been pretty rough, but we’re almost done. The game is feature-complete, in private beta now, and will be coming to an Internet near you ‘shortly.’”)

Mea Maxima Culpa, Finance Programmers

I apologize in advance to experienced finance programmers — some of this is simplified a little bit for general consumption. Other parts might accurately reflect how Stockfighter’s simulations work but might not be maximally true-to-life, as we occasionally have to break with reality for pedagogic or player-experience reasons. (Also, it’s entirely possible that I’m wrong with regards to details — feel free to ping me if you think I have material errors. They’re my fault rather than that of our trading advisors.)

The Problem Stock Exchanges Solve

Andy wants to buy a stock. Beth wants to sell the same stock. A stock exchange gives Andy and Beth a place to transact where they know there is a high likelihood that a willing counterparty (someone who takes “the other side” of the trade) exists.

The stock exchange is built around a data structure called an order book. An order book records orders: offers to buy a stock or sell a stock. By convention, these are called bids and asks respectively. (If you need a mnemonic, try “both ‘bids’ and ‘buy’ begin with ‘b’”, but you’ll have this in your muscle memory by your second day of writing trading systems.)

For a trade to happen, a bid and an ask must cross: that is, the maximum price the buyer is willing to pay must be greater than or equal to the lowest price the seller is willing to sell at. You might find it handy to remember those prices as ‘limits’, for reasons which will become obvious later.

An order book is a prioritized queue (or two of them: one for bids, one for asks), ordered by “priority”: “What is the first order that an incoming order would cross with?” There exist a variety of prioritization schemes at various exchanges, and they have huge impacts on how trading happens on those exchanges. Stockfighter assumes the simplest and most common algorithm: price/time priority. Basically, an order always interacts with the best priced order on the opposite side of the book first. Ties are broken by the timestamp that the exchange accepted the resting order at.

Since the order book is split into two parts, it’s often useful to know what the best bid and the best ask are. This is often called the quote. It is expressed as “$BID / $ASK” or, in spoken language, “$BID by $ASK.” For example, if I quote Google to you at $750.05 / $750.06, that means someone is willing to buy it for up to $750.05 and someone is willing to sell it for at least $750.06. (More sophisticated traders might want to know the size available at those levels. A level is simply a price. Why not call it a price? I have a sneaking suspicion Wall Street invented many of these words to give customers the impression “This is all really, really difficult — pay us money and the complexity goes away.”)

The Fundamental Order Book Algorithm

There exist multiple order types which an exchange can support. By far the most common is a limit order, which can be understood as “I want to buy X shares (or as many up to X as I can) for a price which is no more than Y” or “I want to sell X shares (or as many up to X as I can) for a price which is at least Y.”

For each limit order the exchange receives, it checks:

  1. Does the order cross with an order presently resting on the order book? If yes, they match, for as many shares as possible (up to the number specified in the order).
  2. Is the order fully satisfied yet? If no, goto 1 until the order no longer crosses with anything on the other side of the order book.
  3. Is the order fully satisfied? If no, the remainder of the order now rests on the book.
  4. For each order we matched with, write the fact of the match (the fill / execution) to the tape.

Steps 1 through 4 are, essentially, atomic with regards to all orders on the stock exchange. You’re guaranteed to not have two orders interleave execution — only one order is incoming at one time. It is either fully processed (potentially with part of it coming to rest on the book) or canceled before the next incoming order is processed.

The Tape(s)

Markets are by nature distributed systems. To simplify all participants having the same view on reality (or as close to that as possible), they typically have a relatively slow way to get a current snapshot of the order book and relatively fast ways to get a stream of deltas to the order book as they come in — new orders, order cancellations, executions, etc. That stream is called a tape, because way back in the day it was physically printed on a ticker tape.

The Stockfighter exchange implementation exposes a few tapes to users: one of all executions (with a new message for each execution) and one of all quotes (with a new quote — containing an at-a-glance view of the order book state and last trade — each time someone either sends in or cancels an order).

Stockfighter also does not, at this point in time, directly expose orders/cancels via a publicly visible tape. This is a considered game design decision for Chapter 1. I’m calling this out here as “A significant way we deviate from reality”, which we’ll do any time we need to to make the game more fun for players.

Order Types

We discussed the simplest and most common order type, limit orders, above. There are many order types supported by exchanges in the real world, all of them offering some benefit to at least some exchange participant. (Exchanges make money on every consummated trade, and they’re in vicious competition with each other for business, so they generally want to innovate on order types which offer particular customers things those customers want. They are constrained by the law and “not advantaging any participant overmuch against other participants, because that would chase the disadvantaged participants to a competitor.”)

Stockfighter supports three order types besides limit orders:

Immediate-or-cancel (IOC) orders: Exactly like a limit order, except if there is a part of the order which is not filled, that part is canceled rather than resting on the book.

Fill-or-kill (FOK) orders: Exactly like an immediate-or-cancel order, with one wrinkle: if the order can’t be fully filled for all shares it requests, it is canceled without causing any executions. (On real exchanges, this is sometimes described as “immediate-or-cancel all-or-nothing”, or “IOC AON.”)

Market orders: Market orders are what mom-and-pop retail investors use: they include a direction (buy or sell) and a quantity of shares to transact, but no price. They execute instantly and take whatever price the order book offers, again matching the most favorable prices first.

Let’s Talk Liquidity

One of the fundamental problems with buying/selling anything at all is that one is not guaranteed to have a counterparty ready at any given moment. This makes it difficult to buy/sell your thing and forces you to take a worse price if you want certainty of execution.

Consider houses. Lining up a buyer for your house takes, typically, weeks or months of work. If you needed to sell your house not at “some time in the vaguely defined future” but “within the next five minutes”, you would have to offer the house at a tremendous discount to its market value. Similarly, if you wanted to buy a house immediately, you would probably need to pay a tremendous premium.

The housing market is said to be illiquid, or lacking in liquidity: you cannot conveniently transform houses into money or money into houses quickly without losing a lot of value.

The stock market is incredibly liquid: for any stock listed on an exchange, you can buy almost any quantity and sell almost any quantity, at any time the market is in session. No negotiation, no red tape, no uncertainty. Click a button on your computer and bam trade done.

This property of stock markets is optional, tremendously useful for some participants, and very not free. Liquidity is a thing that can be sold, and much of the money on Wall Street is made by selling it. Let’s walk you through how it happens, but first, a bit of an explanation about why people actually want to buy it.

There exists a tradeoff between price and execution certainty. If you send in a limit order, there exists the possibility that it will not execute. This probability is higher if the order wouldn’t cross with the current state of the order book, but even if it looks like it will, the market might well change before your order arrives at the exchange. Even if it looks like someone is willing to sell Google at $750 a share, if you send in a limit order for 100 shares at $750, you have no guarantee that you actually get any Google shares.

If you send in a market order for Google shares, you’re guaranteed to get all of them that you want (subject to the availability of them at any price), but you give up certainty about the exact price you get.

That’s a reasonable tradeoff for many market participants! A family doing some casual trades in their retirement account wanting to buy 20 Google shares (~$15,000 worth) doesn’t really care about the exact price they get. If Google moves by a few cents in the interim, that costs them only a few dollars of value. Oh well — they just want to have the Google, for whatever investing or speculative reason they had for placing the trade originally.

A professional trader, who cared a lot about getting the best possible price and was therefore willing to pay attention to the market all day, might say “Well, Google routinely swings around a bit, so I’ll put in an order at $748 and see what happens.” If they’re buying 10,000 shares at a time, that saves a meaningful amount of money… if that order gets hit at all. If they were wrong, then they don’t get their Googles… or they have to adjust their orders mid-day. That’s fine — executing trades is their job.

Market Makers And The Spread

So the stock market exists to connect Andy and Beth. What happens if Andy and Beth want to trade a stock but are not both in the market at the same time? Enter the market maker. Once upon a time, market makers were designated individuals (called “specialists”), but these days it is often just “anyone running a market making strategy.”

A market maker’s job is TTTaaS: Teleportation and Time Travel as a Service. Suppose Andy wants to buy at 9:15 AM and Beth wants to sell at 9:30 AM. If neither Andy nor Beth are willing to wait, no trade would happen, and Andy, Beth, and the larger economy are all sad.

A market maker says “This is solvable. I sell Andy the stock he wants to buy at 9:15 AM. I then buy the stock back from Beth at 9:30 AM. I charge them slightly different prices and make a modest profit for holding onto the risk for 15 minutes. Then I do this a lot.”

(That’s the time-travel aspect. The teleportation aspect involves cross-venue arbitrage. Too complicated for today, but know that it exists.)

The difference between the price a market maker is willing to buy at and the price they are willing to sell at is called the spread. If you put in a market order, you’re guaranteed to “cross the spread”, effectively paying the market maker a small toll for guaranteed instant execution. (If you don’t want to cross the spread, just put in a limit order such that it rests on the book rather than immediately crosses, and hope that that limit order gets hit — again, no guarantees there.)

How Wide Is The Spread?

The width of the spread — the price of liquidity — is set by the market, not the exchange. It arises from the frothy interactions of thousands of participants firing orders at the exchange.

In the bad old days before computers, stocks were priced in eighths of a dollar (multiples of 12.5 cents). The spread could never be any less than 12.5 cents, which is a substantial chunk of the transaction value for many stocks.

Additionally, specialists colluded with each other extensively, such that they agreed to quote only “odd eighths”, essentially widening the spread to an entire quarter. They collected a one quarter tax on every share of stock which traded, every time it traded, for decades. Specialists loved this system. Investors, not so much — somebody pays that tax.

These days the markets are decimalized — stocks trade in increments of a penny. (They are not allowed to trade in increments smaller than a penny, by federal regulation. This is unfortunate, because “the minimum spread is 0.01 dollars” is not any more rational than “the minimum spread is 0.125 dollars” — if someone is willing to provide liquidity for cheaper, we should encourage that.) Additionally, since anyone can trade from any computer hooked to the exchange, human specialists have largely been outcompeted by algorithmic traders — computers which place orders all day long trying to be the one market maker of hundreds who successfully captures that penny.

You may have heard about High Frequency Trading (HFT). There is no hard-and-fast definition of it. You should understand that most HFT firms are just executing market making strategies really, really quickly while in vicious competition with traditional market makers (whom they utterly crush, because computers are better at doing math fast than people are) and other HFT firms. This is a huge benefit to most people attempting to transact in a stock, because a) no one is forced to do business with the market makers (again, just use a limit order and accept the risk of not executing if you don’t want to pay for their liquidity-providing services) and b) the presence of HFTs competes the spread down to a penny in most highly-traded stocks. Since they’re legally prohibited from competing with price below the penny increment, they then have to compete on speed, and that competition has intensified to the point that HFTs routinely run up against “the speed of light” as an annoying constraint on their engineering teams.

What Is The Risk In Being A Market Maker?

Your job is teleportation and time travel. Bad news: teleportation and time travel aren’t actually possible. This means you take on risk.

Consider the case where Andy wants to buy at 9:15 AM and Beth wants to sell at 9:30 AM. The market maker is not aware of Andy or Beth’s plans and cannot be certain they will not change. The market maker also cannot know what happens between 9:15 AM and 9:30 AM. The stock that they sold to Andy for $40 a share could soar in value to $50 a share when Beth wants to sell, costing them a loss of $10 a share.

This risk is the economic justification for liquidity having a price associated with it. If it were as simple as accepting “Hey, hold onto this stock for 15 minutes and then someone will ask for it — you have no price risk at all”, then it would cost as much as a coat check (“We’ll just throw that in for free”), and not “a small amount on every transaction” which turns into “billions of dollars over the course of the year.”

(You might sensibly be curious as to the impact on individual investors. Fair enough. I’m a small retail investor who trades very occasionally. I ran the math and, on my portfolio of ~$80,000, I’ve paid approximately $6 to market makers over the last ten years. This compares to e.g. $500 or so in commissions to my discount brokerage.)

How Do You Manage Risk As A Market Maker?

This is the entire ball game. At the most basic level, you want to limit the amount of inventory you take in any stock in either direction and charge an appropriate price for liquidity.

Sophisticated market makers use statistical techniques, simulations, etc to try to guess the near-term future behavior of the market, using this to determine how much inventory they’re willing to hold at any given time and what prices to charge.

In the real world, market makers use a variety of other instruments to hedge their inventory risk with respect to any given stock. In the early levels of Stockfighter, we intentionally restrict you to thinking about only a single stock at a time (for simplicity), so your main levers are canceling your existing orders, adding new orders to the book at different price levels, and (in extremis) unloading your position by transacting with orders on the book placed by someone else.

(Probably another market maker. Fun fact: most orders resting on the orderbook at any given time, both in Stockfighter and in real life, are there because a market maker put them there. This was one of my fun takeaways from the research phase for this project: liquidity really does exist primarily because market makers are actively adding it.)

The Simplest Market Maker That Can Possibly Work

1) Guess a current fair price for the stock. (The midpoint of the current quote might be a good first approximation, or perhaps the last price a trade happened at.)

2) Put that price on a number line.

3) Draw three equidistant ticks to the left of that price and three to the right. The distance between the ticks is up to you — you could use a set interval (say, 5 cents) or something sized relative to the price of the stock (say, 0.5% of the midpoint price).

4) Send orders into the exchange such that you currently have orders to buy or sell at each of those ticks. Sizing is up to you: the simplest thing that can possibly work is just “pick a number and use it everywhere.”

5) Wait.

6) Did someone transact with you? Great! Cancel all your outstanding orders. Now, do it all again.

7) Keep doing this until you make a squazillion dollars.

This is about as easy to implement as it looks. (My first market maker clocked in at about 144 lines of Ruby.)

Shockingly, if you’re the only market maker in the market, this will actually work most of the time. The monopoly supplier of liquidity makes money virtually by definition, particularly when the market does not quickly move in one direction and stay there.

In real life, you’re not the only market maker in the market, and you’re liable to get crushed if you try this, as you’re going to be systematically outcompeted for trades which are safe and you’ll systematically undercharge for trades which involve risk. Also, in real life, other people can look at how you choose to do business… and they have a lot of experience picking the pockets of naive market makers. Don’t say I didn’t warn you.

Doing This In Stockfighter

Now that you know in broad strokes how to write a market maker, you’re probably wondering “OK, but how does one actually do that?

In real life, you’d post about $30,000 of capital (bare minimum) with a broker, get access to their API, and then try not to bankrupt yourself while you learn the ropes. I can’t recommend most developers actually try this.

In Stockfighter, your fictional employer in the fictional game will give you lots of fictional money, backed by a reset button should you ever run out. We give you access to a REST API, which has everything you need to send in orders, get the status of orders, get quotes for stocks, and what have you. You can connect to the various tapes provided by the exchange over web sockets, but this isn’t necessary for our earlier levels — the vast majority of players will just write a for loop and poll every few seconds for updates.

This would be problematic if you were competing on speed with a HFT firm, but not only are our bots written in Ruby and not designed to be speed demons, we intentionally hobble them in the early levels to make it an inviting experience for programmers new to trading. (Our stock market maker bot is also literally the first trading program I ever wrote and close to the dumbest a market maker can possibly be, so clocking it shouldn’t be that difficult.)

In real life, most exchanges expose a quirky protocol called FIX. Stockfighter will support FIX in a later release, but for our Chapter 1 release, we have a simplified REST API with JSON. You’ll end up doing things like:

POST /venues/FOOEX/stocks/BAR/orders

with the order:

  “symbol”: “BAR”,
  “venue”: “FOOEX”,
  “direction”: “buy”,
  “qty”: 20,
  “price”:  5100,
  “type”: “limit”,
  “account” : “OGB12345”, // your trading account (game gives you this)

and get a response back like:

  “ok”: true,
  “symbol”: “BAR”,
  “venue”: “FOOEX”,
  “direction”: “buy”,
  “originalQty”: 100,
  “qty”: 20,   // this is the quantity *left outstanding*
  “price”:  5100, // the price on the order — may not match that of fills!
  “type”: “limit”,
  “id”: 12345, // guaranteed unique *on this venue*
  “account” : “OGB12345”,
  “ts”: “2015-07-05T22:16:18+00:00”, // ISO-8601 timestamp for when we received order
        “price”: 5050,
        “qty”: 50
        “ts”: “2015-07-05T22:16:18+00:00”
      }, … // may have zero or multiple fills.  Note this order presumably has a total of 80 shares worth 
  “totalFilled”: 80,
  “open”: true

Take it from this web developer — you can be up and running on this API in a matter of minutes.

After you’re able to work with the API, you just have to use that to solve whatever challenge the level throws at you. One challenge might be “Here’s a venue (stock exchange) where a particular stock is traded by many bots, one of whom is running a poorly considered market making strategy. Implement a better one and make $X before time runs out.”

Our desired difficulty curve: our first level is a cakewalk if you’ve ever programmed with an API before. Our first few levels after that are solvable in an hour or so of donking around with the API. They range in conceptual difficulty from “A motivated CS102 student should be able to do this in a fairly straightforward fashion” to “You’ll feel pretty proud of yourself once the code works.”

We also have later, more challenging levels, calibrated to be a fun evening project for a developer skilled enough to make it to them. Many of the solutions would make a good conference talk: here are the dead ends I tried, here is the insight those gave me, here is the approach that ultimately worked, and here are the fun implementation details.

Affordances We Built To Make This Easy

Every public endpoint of our API is documented. This includes code samples, sample responses, commentary on how one would actually use that endpoint in an application, and an in-page API explorer so you can run ad-hoc queries without needing to write any code. This is courtesy of, which is one of my favorite new SaaS apps.

We’re not releasing the API documentation publicly until the game formally launches, to avoid giving anyone the ability to pre-write clients for the game. (Though that earlier sample probably gives you enough to predict most of the API… hmm… well, good on you if you can do it from that.)

We will not be releasing first-party libraries for the API at launch, to give the community the opportunity to build them for yourselves. (“Can I write a e.g. Python library for the API and throw it up on Github?” Heck yes. “Can I use a client library someone else wrote to let me focus on the fun work involved in solving my levels?” I’d personally be very disappointed in an engineer who, in 2015, defaulted to scratchbuilding their own clients for every API they consumed. Starfighter loves OSS and the OSS culture. Go nuts.)

You can, of course, use any language capable of driving a REST API to play these levels. Or curl, for that matter. (To quote Chris Rock: “You can drive a car with your feet if you want to, but that doesn’t make it a good idea.” Memo to self: add a Drives Car With Feet badge to the game.)

A Non-Trivial Sample Application

We built an in-browser trading application in React, which you’ll get instant access to once you open one of our trading levels. This interface is essentially what a day-trader would be working with… if their brokerage of choice made some pretty poor UX decisions because their dev team was one guy writing his first React app.

Wait, did I say that out loud? What I meant to say was: we produced an entire web-based trading interface, driven 100% through our API, which allows you to see the API actually getting worked with.


It has virtually complete coverage of our API, by necessity, so if you need to know how to e.g. interact with a web socket you can just right-click and View Source. We don’t “cheat” and do anything to make the API easier to consume for our own applications like e.g. adding private endpoints which pre-digest information for the client.

(Most of our bots don’t cheat, either — they interact with the stock exchange the same way your applications do, and have no privileged access to e.g. market data. There exist exceptions to this general rule, in particular, in Chapter 1 level 6. I won’t spoil it for you.)

Further Reading

If you’re having trouble visualizing what an order book looks like, particularly as it gets mutated in response to incoming orders, I recommend taking a look at Chris Stucchio’s examples in these three essays. They’re in the context of an argument that HFT is not as abusive as the engineering community often believes.

I happen to think that Chris has the right of that argument, but even if you don’t, read his examples closely, because Chris has a view of trade execution which can be reconciled with reality and Michael Lewis (of Flash Boys) does not, as you will quickly discover if you try to actually write out what Lewis says is happening in pseudo-code.

What Happens If I Can Make A Market Maker?

In the process of learning to build a market maker, you’ll both demonstrate substantial practical engineering skills (e.g. working with a novel API, dealing with state, modeling a data structure that probably isn’t built into your language already, etc), learn some fun new things, and get a whirlwind tour of common Wall Street activities.

Each level of Stockfighter introduces you to a challenge which builds on the last, in the context of a narrative taking place in a simulated world. You can’t possibly screw up anything so badly that the reset button won’t fix it, and no money is actually on the line.

For example, after you have successfully built a toy market maker, we might give you a new level where that market maker is exposed to harsher conditions and tell you to adapt to them. (Fun intellectual exercise: read the section on risk and try to predict what features of a trading environment would make a market maker’s job harder.)

Market makers are among the first of many concepts Stockfighter will teach you. Our intention is that many generalist programmers, including folks who have never had the opportunity to do anything more interesting than a standard CRUD app, will discover (or develop) depths of engineering skill they didn’t know they had. That’s awesome regardless of what happens.

Our games are free to players. Most players will be playing Stockfighter simply because play is fun. We’ll always support that, but we want to support the engineering community in more direct ways as well. If you find that you’re a better engineer than your day job needs you to be, we might very well be able to find a job more suited to your abilities. Our business is introducing talented engineers to clients who want to hire them. They pay us if they hire you. We’ll have about 15 clients signed at launch, ranging from Wall Street institutions (“Want to learn how to do this when it isn’t a toy?”) to non-profits to startups with interesting engineering problems.

The market doesn’t believe there exist enough engineers who thrive when given a novel, hard problem. We think you’re out there, in multitudes, and we think there exist many other engineers who are on the cusp of greatness. We want to meet you and geek out together.

See you in the game in the near future. If you’d like to make sure you hear when we launch, and you’re not already on our email list, fix that here.

Designing And Building Stockfighter, Our Programming Game

Stockfighter, Starfighter’s first programming game, is presently in private beta. We’re moving to dark launch (to clients and handpicked vict^H^H^H friends) next week. We’ll launch publicly immediately after the servers give us signs that they won’t go down in a roaring fire. I’d anticipate within 3 weeks of today, but you know how engineering schedules go.

(Starfighter is a company which is trying to solve engineering hiring. We make games/puzzles which one plays by programming. Think Minecraft meets Legos meets World of Warcraft meets a CS class — we’re giving people a fun little world to play in by programming with some pre-planted goals and then seeing what emerges. More on our website or here if you’re curious.)

I wanted to talk a little about what we’ve built and why, both out of general interest and because I know some folks want some direction for how to prepare. I’m going to take the liberty of not spoiling actual levels, because discovery of them is a lot of fun, but will focus on the setting, infrastructure, and design goal.

Picking The Fiction / Setting

We picked “cops and robbers on Wall Street” for the setting of our first game — making money by legitimate means (like charging for liquidity) and illegitimate means (like stealing it). Although my co-founders Thomas and Erin have done substantial work with securing real stock exchanges over their career, I have no particular background in finance. We’re also primarily interested in general engineering skills rather than e.g. quantitative trading specifically. So why start in finance?

Basically, rotating the player around a series of Wall Street institutions allowed us carte blanche to hang virtually any engineering challenge off a more-or-less cohesive narrative. Trading systems touch low-level coding, networking (all seven layers of hello OSI Model), APIs, end-user UI, databases, embedded systems, enterprise web frameworks, cutting-edge programming language research, Big Data, state management, etc etc.

The fiction also gives us excuses to be naughty, which is something we think many programmers are attracted to. Many CTFs (games which practice security skills) cast the player as a hacker breaking into something, which works to a point. You can get the same adversarial frisson in legal fashions on Wall Street, often by building things rather than strictly by breaking them. (Though, ahem, not all of the things our levels have you build are, strictly speaking, legal. That’s also part of the fun. The fact that you’re being naughty vis-a-vis institutions which are not well-liked in the programming community is also useful to us from a player-motivation perspective. I hope people will, in the course of robbing them blind, actually learn what major Wall Street institutions do for a living and come to a more nuanced understanding of their position in society.)

There are natural in-universe tie-ins for both player-versus-player contests (which we won’t ship at launch, but soonish) and for dealing directly with computer-controlled opponents. Wall Street is, after all, one big computer-controlled opponent.

Some folks have been surprised that our game is not more, you know, game-y. Couldn’t we have made e.g. an RPG or strategy game or similar? Yes, certainly, but we had some concerns of a social nature.

Cards on the table: I used to run a World of Warcraft raid guild before I went into business. (Business has less dragons and better loot.) My co-founders, on the other hand, feel about gaming approximately the way I feel about ballet: it’s great that it exists in the world and I’m glad people find enjoyment in it, but it isn’t for me. This view is shared by many talented engineers and would-be engineers.

As of 2015, we don’t gate entrance to the tech industry based on demonstrable enthusiasm for ballet. It is important to our founders that the industry is not gated by demonstrable enthusiasm for gaming-qua-gaming or related geek cultural signifiers, like e.g. a fantasy or sci-fi setting.

Starfighter doesn’t ever intend to build gates (gates close; that’s what they are for). We dynamite gates.

Tech Trees

Stockfighter is divided into tech trees. A tech tree presents itself to the player as a series of challenges linked by narrative progression and pedagogy. Learn a new concept; beat a level; get introduced to new levels where you can build upon what you build and what you learned. (Fun game design challenge: how do we not bust you back to a blank editor between every level while simultaneously not making it feel like you’ve “already done this.”)

In architectural terms, a tech tree typically requires additional back-end engineering work on top of our general platform, and also some front-end engineering to make it playable in the browser. (About which, more later.)

We’re shipping two tech trees at launch: automated trading and emulators and compilers. My co-founder Erin discussed writing our emulator already, so I’ll be covering mostly the trading tree today.

Automated Trading

Automated trading is the classic problem people think of when they hear Wall Street. It’s topical: Flash Boys (a book about high-frequency trading) was on the New York Times bestseller list. It delivers on the “make a squazillion dollars using only your wits” fantasy for the game. (“Fantasy” is, in this sense, a term of art from game development: much like someone writing a novel or screenplay, a game designer wants to have intentional control over the player’s emotional state. If you’d like to read more about this, Raph Koster’s A Theory of Fun comes highly recommended.)

In Stockfighter’s automated trading tree, each level gives you a new scenario (randomly generated and assigned to a wee little copy of the world accessible only by you) and a plausible set of objectives. You achieve the objectives by driving our REST API, using any language/stack/tooling you’re comfortable with.

What we built to support trading:

  • a limit order book, written in Go
  • a stock exchange front-ended by a REST API, written in Go
  • a settlement system, written in Go
  • a world simulation for a universe of fictitious companies, written in Ruby
  • several automated traders, written in Ruby
  • a game master server, exposing a REST API, written in Go
  • an internal message bus, about which more later
  • a blotter interface (similar to a retail trader’s web application — think “what I see when I sign into eTrade”) written in React and consuming the exchange and game master APIs

I wrote most of this myself because Thomas and Erin were doing the technically ambitious parts of Stockfighter. No, really, compared to their stuff, scratch-building a stock exchange is actually fairly straightforward.

Writing A Stock Exchange

My primary stack since 2010 has been Ruby on Rails. I love Rails, and we use it for our user-facing web application, but Rails is not the right choice to build a stock exchange. The biggest reason was concerns about raw performance.

Stockfighter doesn’t do “high-frequency trading” like Wall Street understands that term, although it does support (and have you actually do) what many lay people understand HFT to mean.

The difference is one of degree. Many lay people basically think that any algorithm placing orders is HFT. There are many prop shops — i.e. hybrid tech/finance companies using the company’s money to trade algorithmically — which have a timescale on the order of days rather than on the order of nanoseconds. Their individual orders (and the logic behind them) might be faster than a human can reasonably follow, but that’s still not HFT. (Acid test: if you have never worried about the speed of light between your systems, you’re not doing HFT. The amateur trading from London on the Chicago Mercantile Exchange using MS-Freaking-Excel to place trades? Not HFT, guys.)

We abstract the notion of time quite a bit, but it was obvious that each exchange was going to have to be able to process thousands of orders a real-life second.

This would be… challenging in Rails. Low-complexity operations which do need to write to the DB typically hit about 300ms in my Rails experience. (Better, more experienced teams like the folks at Basecamp have gotten it to 50~100 ms in typical cases.) We would need to do thousands of them a second and, unlike a typical web application, we a) have a bursty, write-heavy, uncacheable load by nature and b) have some severe constraints with regards to requests affecting each other.

Why? Well, it is a stock exchange. The order book — the list of how many shares at what prices are currently available to be bought and sold — is (conceptually) a singleton sitting behind a global lock. (Per-exchange — Stockfighter will simulate thousands of exchanges simultaneously, but architecturally, it needed to consider the “Thousands of people hitting a single exchange at once” use case from the jump.) This is getting all the player and bot interaction. The matching engine has to be completely done with a particular order before starting processing of the next one.

We solved this in a roughly similar way to a real stock exchange: the order book is persisted entirely in memory, in Go.

Stockfighter is my first production system written in Go. The Google (et al) performance wizards have absolutely delivered. The language screams and, happily, it is possible for a mere mortal to actually ship a web server with it. (Note “server” but not application. More on that later.)

Incoming orders from the REST API or our internal message bus are passed to a single goroutine which manages ingress and egress to the order book. This goroutine writes the results of those orders to a series of tapes. A tape, physically, a buffered channel with a servicing go-routine. The tapes are then consumed by other parts of the systems to e.g. send execution notices to players/bots, update the quotes displayed by the API/UI, etc.

Our exchange is blazingly fast: we can process fairly complicated orders matching against a filled order book in on the order of 100 to 200 nanoseconds. Read operations are even faster. (I might eventually figure out some way to re-architect the exchange such that reads don’t touch the matching engine at all, but at present some of them do. Whoopsie.)

This is good — it allows players substantial latitude for experimentation with trading strategies (including ones which push the performance envelope), lets us run dozens or hundreds of bots per player, and keeps our infrastructure costs down to “less than a small fortune.” (We’re on EC2. At release, pending understanding how many users we have and what their typical behavior is, I anticipate that we’ll have a count-on-your-finger number of t2.micro instances for the stock exchanges.)

Getting this working was a bit of a beast. Our order book is represented internally as a red-black tree (Go implementation via GoLLRB). I had never actually used a red-black tree since finishing my CS degree. (Well, OK, they’re sitting down somewhere in MySQL or Postgres but I’ve never had to actually think about them before.) This is a preview of coming attractions for Stockfighter: even when we aren’t explicitly making tech decisions for pedagogic purposes, modeling the complexity of the stock market is a real engineering workout versus e.g. the typical CRUD application.

Bonus points: most players will, at some point, choose to represent the order book on their own machines. This is foundational to many trading strategies. If you want to golf cycles off to make your code faster, that will a) directly advantage you in later Stockfighter levels and b) be a fun opportunity to play with algorithms and data structures.

A quick case for a LLRB for the order book data structure: they tolerate arbitrary writes and deletes very well, are sorted, and let you do things like “Iterate over every price point on the tree from the low-price side and give me everything until you count up this many shares and do it fast.”

My main issue in this part of the development was managing concurrency. If I had cottoned onto the wisdom of the common Go pattern “Make a server object with one managing go-routine and have it tightly loop on a single channel of incoming requests” earlier, I probably would have a avoided a whole lot of deadlocks, crashes, and concurrency bugs caused by e.g. reading the order book for GET /venues/FOO/symbols/BAR at the same time a match was ongoing subsequent to someone POSTing a new order.

Stock Exchange Simulations

There exists substantial prior art on funny-money stock exchange simulations. Most of them are cunning artifice — you’re not actually trading with another actor. When you put in an order, you receive the results suggested by a stock market simulation. Amalgamated Cat Food (ACF) is up today — you paid $45.32 for your shares! But you didn’t buy them from anyone. Your account just got incremented, but there is no corresponding sell anywhere. Your counterparty is a fiction.

There are excellent engineering reasons to do this and I won’t knock their decision, but Stockfighter had different goals.

Why? Well, a lot of the fun engineering problems in trading are caused by it actually not being reliably the case that you send in an order and it gets unproblematically matched at “the price.” Markets are distributed systems. The exchange’s view of reality and your trading system’s view of reality are, by necessity, separated by the great firewall known as “physics.” For maximum possible results, you have to be able to do things like accurately predict what the future state of the exchange is, because the order you’re composing right now will arrive in the future not the present, while being cognizant that your present view of the exchange’s state is actually the exchange’s past.

Sound hard? It is hard. You don’t have to worry about it for the first couple of levels… well, actually, let me take that back. We do not intentionally expose you to the speed of light as a limiting factor early in Stockfighter… but we can’t remove latency as being an actual limitation distributed systems have to overcome.

Here’s an example of how this manifests: I was very frustrated during development at one point, debugging a level I had written. The blotter interface was showing a quote with an ask for 500 shares available for $100. (You’ll pick up the market jargon quickly — an ask is an offer to sell a given quantity at a given price. If they want to buy, it’s called a bid.)

I sent in a market order for testing. A market order is a special order type which doesn’t specify a price — it says “I’ll pay literally anything, I just want guaranteed execution of this order immediately.” (Aside: Stockfighter will quickly teach players in a visceral fashion why Wall Street considers people who place market orders to be idiots. Use a limit order most of the time. If you need instant execution, use fill-or-kill or immediate-or-cancel with a suitably-generous-but-not-actually-unbounded price attached.)

My order came back saying that I had bought 0 shares. And the quote from the exchange now read “500 shares available at $120.”


After a lot of spelunking, I figured out what happened: there was only one bot on the ask side of the order book during that level. That bot had an order resting on the book displayed on my screen. In the time between I hit the button and my order reaching the exchange, the bot had hit the REST API to cancel its order. Then my order arrived to an empty book, soaking up 0 shares. It was then canceled (because market orders are instantaneous). The bot then replaced its order, causing the exchange to issue a new quote, which made it back to my PC before I had time to blink. From my imperfectly attentive human perspective, it looked like the exchange and the bot were intentionally colluding to abuse me.

They’re not.

But that sort of interaction — orders resting on the book, cancels, timing issues, different views on reality, profiling incoming order flow (as reflected by the quotes or execution tapes) and using it to send in or cancel orders faster than the incoming flow can itself adjust, using executions from one exchange to drive decisions to send orders to another exchange, etc, are all so fun to play with. And to do that, you need other agents participating on the exchange with you, rather than the exchange being a single-player experience.

Enter the bots.

Writing Bots

Since player-versus-player interaction causes a lot of social issues which we weren’t ready to design for for release, and since players are not reliably available to play a) at your skill level b) right-the-heck now, we knew we wanted AI to trade against for most levels. (Aside: We’ll support PVP in some levels of later chapters, for those players who enjoy it. If you just want to treat Stockfighter like an intellectual exercise and never have to worry about another human during your experience, that is a game mode we will always support.)

It fell upon me to write multiple trading algorithms. For context: I have an undergrad background in AI (enough to know that it’s largely not applicable to the problem at hand) and most of my investing experience was simply buying-and-holding rather than trading actively. (Aside: retail investors should not trade actively. If someone thinks Stockfighter has given them the confidence to try actively trading for real, I will bang my head into the wall several times and wonder how we can make the next chapter more clearly communicate how ROFLstomped one will be if one tries that.)

Luckily, the market has participants operating at all levels of skill. Markets are aggregation mechanisms for price signals, and they come to surprisingly sophisticated behavior even when composed out of relatively unsophisticated individual agents. As soon as we had multiple bots operating diverse strategies, we started to see fun emergent behavior.

An example, hopefully without spoiling too much: I did a technology preview of the game with a friend who has a Wall Street background. The only actual goal-oriented level in the game at the time was our tutorial, which just has you buy 100 shares to prove you’re mechanically capable of doing so.

(Aside: I could describe level 1 as “FizzBuzz for a REST API”, but we don’t. FizzBuzz is widely used to quickly weed out non-programmers from engineering hiring processes, but the experience of being asked to do it is a little off-putting for experienced developers. Giving people an actual fun goal — demonstrate mastery over a new system! — allows us to covertly test core engineering skills without writing This Is A Test Of Core Engineering Skills at the top of the screen.)

Anyhow, my Wall Streeter friend sailed through the tutorial and asked what was next. I told him “Well, content-wise, I’ve got nothing. But you’re hooked up to a stock exchange and all the other participants are idiots. What do you think would be fun to do?” “How dumb are they?” “Pretty dumb.” “So dumb they would follow the stock all the way down to zero if someone was to manipulate it?” “Umm… I don’t know.” “Let’s see.”

Five minutes later, we had verified that unplanned interactions between bots who a) don’t collude, b) don’t know of the existence of each other aside from seeing the order book, and c) have no mental model of the player can in fact successfully model sheer, unrestrained terror if a player manipulates the market correctly.

We’ve found a small number of not-too-complicated bots to be the gift that keeps on giving for level design — remix them and tweak parameters and the behavior of “the market” is different on every playthrough but roughly configurable in advance. This made level development much, much less expensive than I had modeled it out as.

We’re shipping with six trading levels, but we’ll have Chapter 2 out the door with more, probably within a month or two of release, and we intend to continue shipping new chapters every 6~8 weeks, indefinitely.

Another fun thing consequence of being a games company: in real life, when the state of the art advances in trading algorithms, you throw out everything that was done before because it would now be predated heavily. Being predated is a victory condition for us — it means levels are winnable! As we get better at building trading algorithms internally and/or hire experts to do it for us, earlier dumber bots don’t end up as bitrot, they end up as valuable game content. For example, the bugged momentum following bots which were proximately at fault for the market cascade were too disruptive to keep in that level… but, well, they’re now a resource that one can deploy to a variety of purposes.

Even after we have very sophisticated market makers (if that is a new word for you, I promise, you’ll understand it by the end of level 3), our idiot market makers are still valuable content for people who are still learning the field.

Simulating An Entire World Economy

Stockfighter’s trading levels are about trading rather than investment. It’s important that the markets dance to an internal logic, but it is less important that that logic be particularly complicated. We wanted the companies at issue to have plausible behavior — some do well, some do poorly, some are stable, some gyrate, and sometimes the entire market goes into bull or bear mode for uncertain reasons before unpredictably reversing. We’re not really concerned, though, with e.g. line-by-line analysis of the earnings report for Amalgamated Magitek Corporation.

So we faked it. I’m going to elide the details of the fakery here. Our order book and exchange server work like they have to work, so I’m happy to spill beans about implementation details. The world simulation, though, is “security sensitive” — if you understand the world simulation you can predict and influence the future. That’s an I-Win button for most of our trading levels.

“What if players reverse engineer your world simulation?” Then I pin a freaking medal on that player (and ask whether they’re happy in their current career).

I think you’ll be pleasantly surprised with some emergent features of the simulation.

At present, it’s totally opaque — the only thing you’ll learn about Amalgamated Magitek Corporation is what you surmise based on seeing a market for its shares develop organically out of the actions of various bots.

Some of the bots are performing market functions which are neutral with regards to the underlying, like a market maker. Some of the bots actually do care about AMC. Well, to the extent that a 200 line Ruby script can care about an earnings report.

It’s not perfectly realistic and isn’t designed to be.

For one, we abstract time heavily. It’s a lot more fun to play through one trading day every 5 seconds and thus get a year’s play done in a 30 minute session of watching your script run than it is for that to just represent the quiet post-bell lull in the morning on January 3rd.

For another, again for maximum fun, our simulated world is more volatile than the one we actually live in. The in-universe financial press would be an impossible job for someone who wasn’t actually manic depressive.

In the future, we’ll expose various elements of the simulation to players. For example, there is an in-universe Twitter. Seriously, that is an actual thing that exists. I pipe it to a text file and tail -f it as a debugging tool, because otherwise it is difficult to figure out whether e.g. the price is cratering because I inverted a comparison somewhere or the price is cratering because Amalgamated Magitek Corporation was hit with a terrorist attack.

The Magitek industry is, as they say, “politically exposed.” Plucky teenagers with outrageous hair hate them and blow up their buildings with disturbing regularity. They just have no consideration for the Imperial Pension Fund’s AMC holdings or the chaos that news will unleash on the markets.

(Aside: geek references are capped at about 5 to 10% of the stocks in the game — you’ll also see a lot of things like Red Tree Agriculture or Uganda Mining Inc. We also reference various things that only have value in a fictional universe, like unobtanium or Bitcoin.)

(Aside: I promise that’s the last Bitcoin joke I’ll make until we ship our cryptocurrency tech tree some months from now. Actually, no, I can’t promise that.)

Anyhow, we’ll expose the in-universe Twitter directly to players in a later chapter. You can try your hand at doing things like e.g. sentiment analysis or trading on news generated by the world simulation before other trades can react to it.

The best example yet of the flexibility this simulation gives us is in the 6th trading level. I won’t spoil the details here, but after I beat it the first time, I told Thomas and Erin “This level is why I signed on to this project.” It’s the capstone for the trading tree in Chapter 1.

A particular fact is inserted into the world simulation and bubbles up to the stock exchange. You’re asked to discover the fact. There are literally infinite ways to do it.

Players getting to level 6 are by construction pretty good at programming and will have learned enough about market microstructure to be effective at trading. Level 6 is, let me not mince words, hard. You have to synthesize what you’ve learned about market operations, about our particular implementation, and then stir in some special sauce of your own… and we give rather little direction on what that sauce needs to look like.

I expect that level to be very challenging, because it is totally out of left field, but it also rewards creativity. You can Big Data your way through it. You can Hadoop cluster your way through it. You can probably solve the hardest phase of it with an Excel spreadsheet. You can use Statistics 101 (if you figure out the right question to ask). You can even solve it by eyeball if you parse the right data and graph the right thing about it. I expect players to do all of these things, with substantial variation. I also expect the unexpected, in spades.

We can’t wait to see what you come up with.

Level Design

Again, no spoiler on individual levels, but I’ll talk about what makes a good level from the perspective of Starfighter.

Appropriate challenge progression: Our desired curve is:

  1. Early levels: Solvable by people learning to program with effort approaching an undergraduate lab session. Cakewalks for professional engineers.

  2. Middle Levels: Solvable by people who possess professional-grade engineering skills, even if they may not have professional-grade titles yet. Expect the level to require a bit of thought and execution quality but not to have much technical risk.

  3. Later levels: Solvable by very skilled engineers, but requiring an investment of effort, and having a corresponding emotional payoff for it (a sense of mastery, learning new skills that you could reasonably take with you, etc).

I was asked how we difficulty curved our levels. My design process was to throw out a lot of real-world trading problems (thanks a million to our advisors from industry; all mistakes in interpretation are our fault alone), winnow to the ones which seemed most amenable to working on our system, and then arranging them in rough order of difficulty assuming the player takes the most straightforward route through the level. I then wrote ourselves a few levers for difficulty — for example, parameterization of the level goals (e.g. if it is too hard to make $1 million in a simulated year, edit a YAML file and now players only need $250k) and providing instructions/hints.

While we’re doing our technical burn-in, we’re also collecting feedback on whether we guessed right on difficulty levels.  We can titrate how difficult a level is in production by choosing between e.g. giving no instructions on the path forward, saying something like “The path forward is obliquely suggested by the API docs.”, giving in-universe hints in e.g. the level briefing, giving out-of-universe hints, and directly telling players “Here’s one way to do it; you produce code to that.”

Players can, naturally, also do this, which suggests that our earlier levels are likely to get easier over time as e.g. better tooling is available on Github or people post walkthroughs. That’s a natural and not-unhappy result from a game design perspective.

Variety: Starfighter is a company, not a one-off event or a mobile game. We expect to (eventually) have a relationship with players which is denominated in years, not minutes. Accordingly, we want to produce regular content updates with them feeling sufficiently distinct from each other that there is a reason to play.

Variably difficulty: We wanted players to be able to Choose Their Own Adventure. If you want to pick the most straightforward solution for a level which works, blast through it, and move on, go for it. If you want to golf your solution, go for it.

Some of the most fun I’ve ever had with another team’s CTF was on Stripe’s 3rd CTF, where automatic detection of e.g. program speed supported a thriving optimization metagame. An example: some players looked at a fly like “An O(N^2) loop which compares text word-for-word against a world list.” and eschewed the flyswatter “An O(N) loop which compares against a hash table.” in favor of an orbital ion cannon like “Pre-compile the dictionary into the executable as a trie or bloom filter.”

We have per-level leaderboards which give people a meaningful number to golf on, and additionally, we have discretion to (automatically or after human review) award additional badges for going above-and-beyond on a level. We’ll also do that outside of levels, too — I can imagine issuing badges for people who, e.g., write OSS clients for our API, submit documentation patches, responsibly report security issues, etc.

Specified goals. Unspecified solutions. A common pattern in CTF development is “Make a program which conforms to this toy interface.” A lot of the work on the player’s side involves understanding that interface (and, often, the undocumented quirks of it). The interface often dictates exactly what your algorithms/structures/general approach need to look like.

We prefer modeling the actual practice of engineering, where you build stuff not to satisfy an arbitrarily defined interface but to achieve some motivating goal. Figuring out what to actually build to achieve the goal is a substantial portion of the challenge of real engineering work — and a whole lot of fun! (Also, companies hiring engineers right now are often looking for folks who can, given an underspecified goal, successfully reduce it to working computer code.)

This lets us support multiple solutions trivially, if we pick goals which are sufficiently robust. Luckily, the Wall Street fiction gives us numerous options. I mean, when in doubt it’s “Here’s the world; make money.”, but things can be arbitrarily rich.

This also has some unplanned for benefits from our internal engineering perspective. The modal level orchestration for our trading levels is roughly ~50 lines of Ruby code. Most victory conditions can be evaluated in a function with less than 10 lines. (Level six? The most fun I’ve ever had which was judged by a single string comparison.)

Level Orchestration

We have a GM server (“Game Master” is an RPG term): it a combination simulation of the world, judge, orchestrator of the non-player characters (bots), and opinionated producer of fun.

Our GM server is written in Go and exposes a REST API. (You could, in principle, play Stockfighter entirely through curl if someone told you what URLs to hit. That would be pretty spiffy. Tell me if you manage to do it.) This accepts commands like e.g. POST /levels/foobar, which creates a new instance of the foobar level.

GM’s level orchestration logic is actually kicked out to Ruby, mostly because I develop Ruby much faster than I develop Go and, since this code is less performance intensive than the stock exchange, we had some flexibility. Levels are plain-ol’ Ruby whose design was heavily influenced by Rails — convention over configuration, lots of reflection used to locate files given predictable naming schemes, etc. They also have a YAML configuration which allows us to adjust e.g. which bots are present in the level, what instructions are given to the player, etc.

The bots… well… every project which ships ships with one decision that you feel the need to say “Look, I know, I know, but it actually worked.” Our bots are all written in Ruby. The goroutine managing the level instance on the GM server periodically wakes up, opens a Ruby subprocess, unpickles the state of the world simulation and the bots, has the bots update themselves against the state of the exchange via the REST API, accepts the bots desired orders, makes them against the REST API, pickles the state of the simulation and the bots, and then sleeps until the entire process starts again in a few seconds.

Why? A combination of “That lets me write levels and bots in my best language”, “I wanted to ship in 2016” plus “If I did a slightly more obvious thing and kept all of our levels and bots constantly in memory rather than slicing them 500ms of the server’s attention every N seconds, then our memory requirements at launch would be measured in terabytes and go up from there.”

Why use Go for the server then, instead of e.g. Sinatra? Because Go’s concurrency story is much better than Ruby’s is.

Front-End Engineering

A core design goal for Stockfighter is that it is quick to pick-up-and-play, right in the browser, without requiring one to have a complicated toolchain installed.

Why? First, from a strict conversion optimization perspective, I’d anticipate a heavy, heavy dropoff between signup and completing level 1 if my first instruction was “OK, get a fully-functioning C toolchain installed on your box. Now clone this repo.” or even “Download this Vagrant image.”

Second, this is particularly difficult for less-experienced engineers. They’re not the core of our business, not by a long-shot, but I want to support them to the extent possible. I want a CS101 classroom to be able to play Stockfighter together. Many practicing engineers don’t remember this, but a CS101 classroom finds installing e.g. a full Rails toolchain to be a hard, frustrating, failure-filled experience. (I’m a professional engineer using Rails for the last 8 years. If you gave me a blank Macbook and a day I could do install Rails… after a lot of Googling to check whether we still use rbenv and Bundler or if that changed since last week.)

So a hard requirement for both of our tech trees was that they be playable in the browser to start. After players are invested, they can get additional edge by working in their stack of choice locally, writing tools which sit on top of our systems, or even e.g. deploying code to EC2 or Heroku or what have you, but batteries had to be included out of the box.

Thomas and Erin build an entire in-browser C development toolchain which compiles down to assembly and then executes on a farm of emulated Arduino processors. I had the comparatively easier task of figuring out “What does the simplest version of automated trading that works look like?”

Answer: it’s manual trading! So we made a blotter interface — basically, a stripped-down retail day trader’s interface for monitoring the market, placing orders, and seeing executions. You can use the UI to poke at any of our levels and get a sense for what the “normal” behavior of the market is given that level’s load out of bots and stocks. You can also use it as a stepping stone for bootstrapping your first programs — there is no need to write e.g. code to poll whether your orders hit anything because we’ve already taken care of that for you, so all you need to do is focus on generating the right orders to send in. If you want, you can even totally replace the UI with your own web application — go nuts.

Another nuts option: if you feel confident in your trading abilities, it is at least theoretically possible to beat many of our trading levels the old fashioned way — by running the math in your head and keying in your orders faster than the other guy. This is hard mode, since the other guy is Ruby code, but given that we force him to sleep for 80~90% of every trading day it’s at least possible.

The UI also includes controls for level operation, like e.g. restarting the level if you get stuck.

It was obvious that given goals like “Make a million dollars!,” in the inevitable learning period you could quickly get yourself in an unwinnable state by e.g. losing 95% of your starting capital. The designed operation of our trading levels is “Play them, lose horribly, write a program, reset the level, run the program, have it crash with a bug, fix and re-run, lose modestly, adjust your approach, re-start the level, run the program, win.” (Folks have asked whether we’ll hold this against you. No, no, of course not. We log everything under the sun, but losing/restarting levels is planned. That capability is an advantage of doing this in a rich simulation as opposed to real life — you can go Knight Capital as many times as you want and no one loses their job!)

We did the UI in React, because it has a lot more going on than most CRUD apps.

DHH describes the sweet spot of my favorite stack (Rails-with-some-Javascript) as “documents with some dynamic sprinkles.” I really love that model, but it does not describe a trading interface. You have an incoming stream of events, many per second, updating part of the DOM, potentially hundreds of orders open whose state can mutate at any time and whose accurate display is important, and a variety of actions available to you at any time.

Obligatory learning curve aside, as this is my first React app, I’m loving React and the Flux architecture. (We use Flummox, which was — sadly — EOLed between us picking it and launching. I swear, all the jokes about JS frameworks are true. That said, of the available options it seemed to require the least boilerplate, which can be further reduced with a sensible naming convention.)

Our blotter interacts directly with the exchange API (unlike most retail trading applications, which interact with a brokerage which interacts with another party which interacts with the exchange). This was enormously useful as I was doing parallel development on both systems. Forcing me to write a fairly complicated application against our API burned out a lot of the usage niggles prior to them being exposed to players. (Oh, don’t worry, you’ll find plenty more.)

Requirements generated by the blotter, like “Polling for order status is just crazy after you have hundreds of them — we need some sort of pub/sub system accessible over a websocket.”, made the exchange better for a variety of trading applications.

Here’s an image, although it’s modestly more amusing when quotes are streaming in and things are updating constantly.

Stockfighter Trading Front-End

The big challenges for React development, aside from the learning curve, were state management when switching between levels (eventually solved by rethinking our store design) and dealing with lots and lots of UI repainting.

At one point I was playing a level with a hundred bots. Each of those bots sent in about a thousand orders a minute. Each order changes the quote for the stock, causing a separate websocket message. Each websocket message would cause my entire table of orders to refresh.

100,000 refreshes of a table with a thousand elements in a minute will, pretty reliably, melt your poor little Macbook.

We eventually solved this by using built-in React features like shouldComponentUpdate (basically, we got very detail-oriented regarding the performance-sensitive components like orders and quotes, made sure they would always render the same given the same prop and states, and then canceled the render if their prop/states were deep-equals equal) and using keys on child objects.

Fun fact about child keys: even if every child is itself immutable, if you choose keys poorly (like, say, simply making a unique ID on creation of the child UI element), updating a single order out of an in-memory list of 1,000 will cause the entire table to refresh, and if your child key is different than the key for the same element in the last version of the table, React’s magic DOM-diffing technology does not save your bacon. Instead, your bacon sizzles amicably on your Macbook.

After I got my head around that and started picking better child keys (appending the ID and modified timestamp is all you really need — same as Rails Russian-doll caching), our UI became quite snappy. I log into our staging environment (Amazon’s Oregon data center) from Tokyo and I can’t tell it’s not on localhost… except when the speed of light decides to play tricks on me.

Supporting Technology

Our web application (signup/login/leaderboards/etc) is a bog-standard Rails 4 app. There is very little technically interesting about it, but it works and was quick to write — this is exactly Rails’ sweet spot.

You know whose sweet spot it is decidedly not? Go’s. I tried, I really did, but it would be very difficult to recommend Go for a web application at the moment.

The ORM we ended up using for e.g. our settlement service, Gorm, is not anywhere near the level of maturity of ActiveRecord, and to get its (valuable!) feature set you have to tolerate a) throwing out most of the benefits of using a type-safe language and b) programming bugs which can cause statements which certainly look like they should generate SQL queries to just silently not generate SQL queries. In general, working with the database has been so painful in Go that I have been instead either a) hitting a REST endpoint on an internal API to have Rails do the DB access then return formatted JSON (which Go can actually consume fairly decently) or b) throwing the data at NSQ.

I have fallen in love with NSQ, an enterprise message bus for people who don’t ever want to have an enterprise message bus. The elevator pitch for it: write a textual message, tag it with a topic, and send it to your local NSQ daemon. On some other machine that doesn’t even know you exist, another program says “Apropos of nothing I’m a member of the $FOO channel and want a copy of $BAR topic messages which have not been processed by the $FOO channel yet. $FOO doesn’t exist yet? It does now. Hit me.” NSQ handles all the magic to make that happen.

We use it for everything. Rails controllers need a way to log analytics-style events? Pipe it to NSQ — we’ll make a decision on an analytics-provider later and, in the meanwhile, we’ll have nsq_to_s3 just save them for us. Our settlement system needs executions from the stock exchange? Open a settlement channel on the executions topic; done. Executions also need to be logged somewhere? Sure, open a dumpToLog channel on that topic and use the provided nsq_to_file utility; done. We want a per-player list of historical UI actions taken by levels? Sure, no worries, have the GM server write to the levelActions topic, open a peristToDB channel on it, and use nsq_to_http to have those hit an internal API on our Rails app which takes care of that.

The best part? When you just assume the existence of NSQ everywhere, you don’t have to think about e.g. dropping more instances of a particular service on your infrastructure, routing requests, etc etc. You just fire-and-forget at the bus.

Actually, that isn’t actually the best part about NSQ. The best part is Stockfighter ships with about seven microservices and having them all understand every HTTP endpoint available would have been unwieldly. Instead, we standardized on a small set of common JSON object representations and then the topic names. This lets us do things like e.g. have our Arduino emulator slurp up quotes from the stock exchange simply by saying “Hey I’d like quotes, too” without the stock exchange even knowing that an Arduino is a thing.

For our services which communicate over HTTP, like e.g. the exchanges, we have to be a little smarter about routing because we can’t treat NSQ as the Magic Oracle Of Making Sure Messages Get Delivered. The short version of that story is, the longer version is a full infrastructure post because it gets complicated.


We’ve been using for our player-facing documentation. I haven’t pushed our docs live yet (spoilers abound) but I love this service.

  • The docs come out looking very pretty without me having to do any more CSS munging.
  • The UX of the docs approaches that of e.g. a Stripe or Twilio rather than a Markdown file.
  • There is an in-browser API explorer, so you can (if you want) copy/paste in your API key and, from the page describing e.g. our POST /venue/FOO/symbol/BAR/order endpoint, actually post a live order against one of our running stock exchanges. It displays the HTTP headers and full response in-line. is ridiculously fun. We’ll push the docs live to it when the game launches publicly.

Further Reading About Trading

We’ve promised players a reading list to help with understanding Wall Street for generalist developers. Here’s a brief tour of my Amazon history over the last several months:

  • Flash Boys, Michael Lewis: I find this book impossible to avoid in a discussion of automated trading and yet difficult to recommend. Lewis is an exceptionally talented writer. I think many Starfighter players will come away from some of our levels saying “Michael Lewis, I have attempted to actually do the thing you said that the bad guys did, and that is, as a matter of engineering reality, actually impossible.” That said we’re totally ripping some levels off of his (highly entertaining) storytelling here. I mean, they named a computer program after a Greek god. How can we not make that into a level.
  • Flash Boys: Not So Fast: A much more lucid take on HFT, by someone whose paycheck depended on being able to turn it into actual engineering artifacts.
  • Dark Pools: The Rise of the Machine Traders and the Rigging of the U.S. Stock Market, Scott Patterson: If the title doesn’t give the game away, spoiler alert: Patterson probably doesn’t consider “more engineers able to do algorithmic trading” to be a victory for truth and justice. (Of course it is! At the very least, consider it self-defense training.) That said, unlike Flash Boys, the discussion of issues/attacks strikes me (and our advisors) as being more tightly grounded in consensus reality. This book also includes an excellent history of the development of computerized exchanges in the US from about the 90s through today. It was particularly wild when I realized that our cute little toy hosted on a t2.micro was a $X0 million or $Y00 million engineering project back in living memory.
  • I hesitate to recommend it based on the price, but if you want to pursue this topic professionally after Stockfighter, the Bible at Thomas’ last company was Trading and Exchanges: Market Microstructure for Practitioners.

That’s all I’ve got for today. Back into the code cave for us. We’ll see you in a few weeks!

Talking About Money

Twitter is presently abuzz with #talkpay folks sharing their salaries. The movement is rather Marxist in character, but this capitalist encourages you to look beyond that, as it is in the general interests of everyone who works for a salary or rate to understand what the market is like.

Why Do I Support Talking About Salary?

I was raised to not talk about salary. This was largely for sociocultural reasons in the American Irish-Catholic middle class, where money was seen as a corrupting influence which should be kept from children for as long as possible. This decision is probably the only major point of difference I have with my parents regarding my upbringing — I would have made much, much better decisions early in life if I had any clue what the world looked like.

Employers have perfect information about salaries at their firm and fairly reliable salary information for the local slice of their industry. Candidates often have no clue whatsoever. To the extent that candidates in general have accurate impressions of what salary ranges are, these accurate impressions often travel along peer networks at roughly the speed of beer. Informal networking like this is superior to no information exchange at all, but results in people who are not present in those networks being at a negotiating disadvantage.

I’m not a Marxist — and in fact so non-Marxist that I’m honestly troubled by joining a Marxist movement on May Day — but I would have to tear up my Member of Vast Capitalist Conspiracy card if I thought that one side of a negotiation having a permanent information advantage didn’t influence how the negotiation turned out.

Compensation negotiations are presently like a stock exchange where only your counterparty can see the ticker and order book. You’d never send an order to that exchange — it would be an invitation to be fleeced. “I happen to have a share of Google and want to sell it. What’s it go for?” “Oh, $200 or so.” “Really? That feels low.” “Alright, $205 then, but don’t be greedy.”

The spot price of Google when I write this is $535. Someone offering $205 for GOOG would shock the conscience. In the years since I wrote a post on salary negotiation for engineers I have received many letters suggesting folks have received and accepted employment offers much worse than that, relative to reasonably achievable market rates.

A Quick Note About Being Fortunate

I don’t know if “fortunate” captures my position in life well, since that suggests rather more luck/chance than I think was involved.  It seems to work as a word, though, given that political implications that “privileged” have come to have are not my politics. That said, for the last couple of years I have been a rather well-compensated member of a well-compensated industry, and I’m cognizant that I have advantages that many people, including myself a few years ago, lack with regards to this conversation.

One advantage in particular is that talking about salaries can expose you to disciplinary process, either formally or informally. Salary structures are treated as corporate secrets in many places ostensibly to prevent morale-sapping internal conflict and — can we be real here for a moment, fellow CEOs? — explicitly to avoid having accurate market intelligence used against the company during salary negotiations.

Consultants/freelancers/etc also have to be cagey about their rate, because very few have just one rate, and if you disclose a previous rate then you can end up negotiating against yourself when attempting to take on a new client. (Most important takeaway about salary negotiation, by the way: disclosing a previous salary is almost always against your interests because it pegs your new salary to that plus 5% rather than your value to the new firm minus a discount, which is a brutal mistake, particularly in the years of your career where natural growth in capabilities/experience/the market/etc causes you to bring vastly more to the table with each new job.)

Happily, these days I’m a) CEO of a company I co-own, b) have no investors to keep happy, and c) have no reasonable prospect of ever having a client again, so I can publish salary/rates without blowback. I’d encourage folks who are likewise well-situated to do the same. In particular, I think that Glassdoor/salary threads on HN/etc often fail to capture the top end of the distribution due to (basically) professional courtesy/discretion, and it is really, really important that young/impressionable/lacking-information members of the industry understand that the top end exists.  (Incidentally, “the top end” suggests that there’s a ceiling/asymptote, but it’s really closer to a long tail distribution.  Pick a number, any number: somebody does a lot better than that. For most numbers you’d naively think of, its an awful lot of somebodies. $100k? Not the top. $250k? Not the top. $500k? … Not the top.)

I’d encourage similarly fortunate members of the industry to strongly consider dropping some breadcrumbs for folks who are still leveling up, either on your blog or even just informally to your peers.  Remember that nobody is born in possession of reliable market data.  Make it a point to assist other people as they journey up the career ladder.  It’s the right thing to do.

Salary History

I have a pair of undergraduate degrees, one in Computer Science, from Washington University in St. Louis.  As of graduation in 2004, my impression was that $50k was a reasonable achievable mid-career salary as an engineer and that, if I worked very diligently, I might eventually hit $100k per year.

Order entry (ages 18 ~ 20, Chicago suburbs): $10/hr, no benefits, $0.25/hr incentive pay for working in non-airconditioned parts of the operation.
Work Study (ages 18 ~ 22, St. Louis): $10/hr. Positions included data entry and basic technical support for e.g. printers.
Undergraduate research (age 22, St. Louis): $12.50/hr. My first position where I shipped code for a living.
Technical translator/researcher (age 22 ~ 25, Ogaki, Gifu, Japan): 3 million yen per year (quoted post-tax) with housing benefit with a fair-market value of approximately 500k yen per year. This was roughly equivalent to a $35k to $40k professional salary in the US, with a comparable benefits suite.
Engineer, first-class (age 25 through 28, Nagoya, Japan): ~260k yen per month (age * 10k yen is approximately the salaryman expectation for engineers in and near Nagoya), overtime pay (a lot of it — as a salaryman I routinely was compensated for more than 50 hours of it in a month… and worked “rather more than that implies”), and the standard benefits suite for professional workers in Japan. No housing benefit. Standard non-taxed transportation benefit covered more than 100% of transportation costs, as is common in Japan. I don’t want to dig out my pay stubs because seeing the number of hours I worked might cause PTSD (not a joke) but my US tax returns were generally in the $45k per year region.

A brief digression about salarymanhood: Salaries are not just internally transparent at Japanese megacorps but they’re so predictable that any middle-class adult in central Japan can accurately predict my salary history at Japanese organizations. I mention this partly because it’s an interesting cultural note and partly to say that salary transparency by itself does not guarantee happy outcomes for employees.

Consulting Rates

After that I quit my day job and started consulting simultaneously with running my own business. Now things get a little… crazy. I reiterate that this isn’t to brag, it’s just to hopefully show people how the world works.

My consultancy is shuttered now, so I can’t hurt myself with revealing the following information, but in deference to identifiable past clients I will be vague about details about particular engagements. I consulted in Tokyo, Chicago, New York, San Francisco, Germany, LA, Austin, and remotely from my then-home in Ogaki, Japan. My typical client was a privately owned software company with $10 million to $50 million a year in revenue. People generally assume that I must have been working for funded Silicon Valley startups to earn the following rates so I think I’ll explicitly call out that they represented less than 10% of my consultancy’s business.

While I was still consulting, my rates were private, largely so that I had freedom to price new engagements.  This worked out rather well, as I walked my rates up rather quickly.

First consulting gig: $100/hr
Second consulting gig: $4k/wk
Third consulting gig: $8k/wk
Fourth consulting gig: $8k/wk “Hey I wonder what happens if I just put $12k on the proposal?” $12k/wk
Consulting rack rate, year 2: $20k/wk
Consulting rack rate, year 3: $30k/wk
Highest accepted proposal: $50k/wk (The gig fell through. Had it not, my rack rate after it would have been $50k per week, at least until I convinced a firm to pay more than that.)

There were a few engagements which I priced at below rack rate, for longstanding clients, clients operated by friends, and the like. I also did a modest amount of semiformal pro-bono work.

What I did to earn those rates:

I combined a modest amount of programming skill (typically Rails, Ruby, and Javascript) with substantial experience with using engineering skills to move marketing/sales levers, including by doing SEO, email marketing, A/B testing, (light) UX design (typically around high-value parts of a SaaS business like signup flows or purchasing pathways), etc. I turned down essentially any gig which was strictly engineering in character and rather aggressively went after bigger projects which were “closer to the money” every couple of months.

I aggressively solicited formal and informal case studies from clients who had had successful outcomes, which went directly into proposals for subsequent engagements. Most of the discontinuous jumps in my consulting rate were directly occasioned by an engagement which had gone exceptionally well.

The main thing holding my rate down for the early years was personal discomfort with being “worth” the types of rates which I desired to charge. I dealt with (deal with?) impostor syndrome frequently and had little context for what alchemy one needs to work to justify professional rates.

Spoiler alert: there is virtually no difference in the mechanics of work done between $100 an hour, $200 an hour, and $30k a week — all of the leveling up there is in sophistication on who you go after, what engagements you propose and deliver, and how you package things for clients.

By the end of my consulting career, I pointed to a small pile of mostly satisfied customers and other evidence that I could do the work, sketched out a “Here’s how I plan to create a couple million dollars of value for your company” in proposals, and then just announced my rate. I received rather less pushback than I expected and virtually never negotiated on rate, using a trick from Thomas Ptacek (“If a client says you’re out of the budget, start talking about the scope of the engagement to find something they can afford rather than slipping your rate.”)

One under-appreciated benefit of charging premium rates is you get to be really picky about engagements. This lets you both deliver only engagements where you have an expectation of being able to do meaningful, successful moves-the-needle-for-everyone work, and it improves your negotiating position. If you charge $60 an hour and desire to be middle class, you have to jump on virtually every engagement which comes your way and if someone counterproposes $50 an hour you virtually have to hear them out.  If you charge more professional rates, you have the wherewithal to decline work below your prevailing rates and simply take an unpaid vacation or concentrate on building higher-quality pipeline for new full-rate engagements.  That is a high-quality BATNA to have, and existence of it helps close negotiations successfully all by itself.

My impression is that my consulting rates are both high and not nearly the top of the market for professional work done in our field. I could substantiate that with data from other people, but that would require divulging confidences.

Folks occasionally think I’m a member of a shadowy cabal for being able to charge the above rates, but believe me, I was nearly exactly the same level of capabilities at $30k a year as I was at $30k a week, I just got better at organizing the business to expose me to opportunities to capture the better rate. I’m good at what I do, but the premium built into the rates versus many other people who can knock together a Rails app is not, not, not due to superior skill with Rails, the elite fraternity of the Ogaki tech industry, etc.

Business Results vs. Wages

I ran a succession of small software businesses between 2006 and 2015, which both earned money and gave me the skills/platform which allowed me to do the consulting thing described above. They’re not directly relevant to discussions about “pay”, as running a business has many types of tradeoffs not directly comparable to those experienced by employees/consultants.  In particular, running a business exposes you to substantially higher risk.  Business owners have a huge variance in outcomes due to factors both inside and outside of their control.

That said, I originally got on the transparency train back when I had $25 a month in revenue and just never stopped publishing numbers. There exists a fairly comprehensive breakdown of my business’ numbers every year since starting (c.f. 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, and 2014).


As a small-business owner, my net worth is dominated by equity I own in my company. This is important to discuss in any conversation about wages in the tech industry, because for historical/cultural/tax reasons we put a large percentage of the industry’s total compensation in equity grants. This is particularly true as you get increasingly advanced in your career.

It’s extraordinarily difficult to value equity in a small company prior to selling it. At one point, I wanted to invest a small amount of money into startups which were owned by other people. This requires one to be an “accredited investor” in the US, with either $250k salary for two years (a bar which I did not hit) or $1 million in assets aside from the value of one’s primary residence. After speaking it over with my accountant, we mutually decided that we could attest to me being an accredited investor in good faith.  (Somewhat surprisingly to me, a two-sentence letter from a CPA suffices to establish accredited investor status.  I had a vague notion that it would probably require a balance sheet or complicated well-researched valuation of assets, but no, it’s basically “I’m professionally responsible for looking after this client’s finances.  His assets exceed liabilities by more than a million dollars.”)

I recently sold my first SaaS business, Bingo Card Creator. I want to disclose how much it went for, but am prohibited from doing so by the terms of my agreement with the purchaser. Speaking generally, small SaaS businesses are presently valued at between 2X and 3X “seller discretionary cash flow” (SDC), which is essentially “how much money could the owner extract from this, totaling profits and their salary, if they only paid absolutely necessary expenses.” BCC’s SDC can be reasonably estimated from previously published numbers. That’s all the bars I can hum.

I have another SaaS business, Appointment Reminder, which I am a 100% owner of. The business is modestly successful. I have a particular number in mind for it, which I’d be crazy to quote publicly prior to selling it, for the same reason that you’d be crazy to quote your desired salary in a salary negotiation.

If a buyer pitches me on 2X my desired number I will, naturally, say “That’s an interesting number. If you were offer insert 2.2X here I think we’d have a deal.” No buyer will pitch 2X if they know what X is, though.  A younger, less savvy me would interject here “Why ask for more money given that they’ve already offered you enough?” and my answer to that is “Because everyone in the negotiation is a businessman and any number we are mutually happy with is a morally acceptable number.  Given this, and keeping all else equal, I prefer ‘more’ to ‘less’ and always, always, always ask for ‘more.’  You should try that — it’s a trivial tactical suggestion which helps address you systematically low-balling yourself, which you’re prone to.  You think that asking for ‘more’ will result in ‘less’ being yanked from the table, but you believe this because you’re young, inexperienced, and working from a narrative about desert which is quite disconnected from how rational businessmen actually operate.  No counterparty who you actually want to work with will hold good-faith negotiating against you.”)

I expect, regardless of the outcome of negotiations over the business, that the sale will pay for my daughter’s college education, repay the out-of-pocket investments I made in growing the company, and establish a healthy buffer for my family.  That said, I’ll still need to draw a salary for the near future.

I did not accept equity in any of my consulting clients.  I own a very modest amount in two companies which I am an angel investor in (Binpress and Riskpulse) and one company which I am a formal advisor in (MakeLeaps).  Like most angel investors in tech, most career returns will likely come down to equity awarded as wages rather than equity purchased primarily for cash consideration.

Present Salary / Equity Situation

I’m presently an equal co-owner in Starfighter with Thomas and Erin Ptacek. As the CEO of a pre-revenue company, I don’t presently take a salary. After Starfighter has revenue, I expect all the co-founders to draw substantially identical salaries (pegged at market rates for senior technologists). We will also receive distributions.


Anyhow, that’s the boring-but-hopefully-instrumentally-useful discussion of ten years in a rather quirky tech career. I am impossible to bore or embarrass on this topic, so if you ever want private advice, drop me a line.

Final words of advice: Technologists are in a wonderful position presently to create a heck of a lot of value in the world. Don’t be hesitant to be compensated equitably for the value you are creating.

Announcing Starfighter

Thomas Ptacek, Erin Ptacek, and I are pleased to announce Starfighter, a company that will publish CTFs (games) that are designed to develop, improve, and assess rare, extremely valuable programming skills.

Starfighter CTFs are not fantastic Hollywood-logic depictions of what programming is like. There is no “I built a GUI interface using Visual Basic to track the IP address.”

You will use real technology. You will build real systems. You will face the real problems faced by the world’s best programmers building the world’s most important pieces of software.

You will conquer those problems. You will prove yourself equal to the very best. Becoming a top Starfighter player is a direct path to receiving lucrative job offers from the best tech companies in the world, because you’ll have proven beyond a shadow of a doubt that you can do the work these companies need done.

We’re not here to fix the technical interview: we’re here to destroy it, and create something new and better in its place.

Sound interesting? Our first game will be ready shortly. Give us your email address and we’ll tell you when it is ready.


We’re going to publish a game in the genre often described as “Capture The Flag” (CTF). It will be a goal-oriented exploration of technology.

You will code to play. You will not pay to code.

Our CTF will be totally free for players. (Not “free-to-play.” There is no catch. We will not ask you to pay extra to buy funny hats or recharge energy or unlock the full version.)

To progress in the game, players will have to use every programming skill they know, and pick up new tricks along the way.

CTFs are a superior way to learn rare and valuable programming skills which you would not otherwise be exposed to. We’ll give you the excuse, and code/test harnesses/documentation/community support/etc, to try that language/framework/problem space/etc you’ve been meaning to learn “someday.” The games Starfighter produces will help programmers all over the world learn these skills absolutely free.

We’ve done this before: our founders ran MicroCorruption. It is one of the most successful CTFs ever created, by any metric, and unlike most CTFs it is still playable years later. Starfighter will run ongoing, supported, progressive CTFs at scale, which will operate indefinitely. Making and supporting CTFs will be our only business.


Starfighter games will be, first and foremost, fun to play. You’ll get to dig into new tech, on your own terms, in the comfort of your own living room. You will achieve mastery of it. You will be able to show off your skills to other devs, and they will say “That’s awesome how you did that.”

Progress in a Starfighter game will map naturally to skills that top tech employers need RIGHT NOW. Playing will teach you crazy programming skills you can’t learn anywhere else. (Bold talk, right? No, really — some of our levels are so fun that if you did them in real life you’d be thrown in jail. Others put you in charge of highly-lifelike-simulations-of programs that, if the real ones blew up, would rate the nightly news worldwide.)

You will learn what it is like to see the Matrix.

Sounds like BS, right. I know. I’m a generic web programmer (yay Ruby, meh JavaScript, boo low-level anything). The last time I played a CTF, written by my cofounders, they had me breaking into locks controlled by micro controllers which ran embedded assembly code. I haven’t touched assembly in 12 years, because I thought I hated it. But then I found myself in Hanoi finding myself with only a lock running vulnerable assembly code separating me from 25 points.

I pulled my hair out for hours. I tried everything I could to get that lock to open. I cracked open a book on assembly. I read tutorials on the Internet. The opcodes were an impenetrable blob, and then something I could sound out but make no sense of, and then a functioning computer program, and then… then they were a target.

I… I can’t even believe I’m saying this… I exploited a buffer overflow bug to corrupt the value on the stack storing the program counter so that when the function returned it wouldn’t go back to the call site but rather jump into memory that I controlled where I had pre-staged handwritten assembly code to gain control of the lock.

Take that, Neo. [Thomas comments: That’s one of the easier levels, actually.]

You will do bigger, more impressive things.

Starfighter will allow you to develop and show off skills possessed only by the most valuable programmers in the world. Does that make you one of the most valuable programmers in the world? Yes, yes it does.

Want to land a better gig? This is your opportunity to level up. Do well in the game, and we can short-circuit the resume spray-and-pray hiring nonsense and introduce you directly to CTOs who will be happy to hire you. (We’ll only do this if you ask us to.)

Sound good? Give us your email address; we’ll tell you when we have a game ready for you.


The science of hiring practices is settled: work-sample tests are the most effective way to assess skill in potential hires.

The problem? Work-sample tests take time and money to develop, deliver, maintain, and support. You’re not in the work-sample test business: you have a company to run.

Starfighter games are work-sample tests, built by a company which will do nothing else. We’ll treat our CTFs like a first-class tech product, because they will be our only product.

We will market them actively. We will track player behaviors and skill at incredible levels of granular detail, instrumenting them like they were built by the Orwellian MiniPeace. We will iterate on their game design, calibrating it to be accessible in the earlier levels but provide an appropriate challenge even to the best engineers in the world. We will create regular content updates. We will make the supporting documentation/libraries/etc a first-class concern rather than the afterthought of an overworked team building a side project. We will engage our players like our ability to feed our families depends on it.

We will erase all doubts you have about a candidate’s ability. You can guess whether they grok REST APIs based on their Github profile, if you are OK with ignoring 90%+ of the hiring pool with no public Github profiles. We can tell you exactly what happened when your candidates tried to implement a REST API. We can compare their performance against hundreds of other talented engineers (including your current employees) on the same task.

We can bring engineering rigor into your hiring process.


You do not have enough qualified candidates, because your candidate filter greps resumes rather than ability. Starfighter will dispense with resumes and measure ability directly. We will source a higher volume and higher quality stream of engineering candidate leads than any channel you presently use. (We’re shooting for higher volume and higher quality than all channels you currently used, combined.)

We will love our players. They will love us. We will help the right ones fall in love with you, too.

Your present hiring process is secretive, scary, and stressful for candidates. We can offer an independent front-end to it without requiring ongoing management overhead from you.

Your team may occasionally falter in building hiring pipeline — the business gets busy, the milestones start slipping, so prospecting stops and coffee dates get neglected and interviews get rescheduled. It happens. We will constantly be identifying new, talented, pre-vetted engineers and introducing them deep into your hiring funnel.

Has anyone ever posted a video of themselves interviewing at your company? No. It is a painful experience. Nobody wants their friends to see them struggling in front of a whiteboard. Some companies would even threaten a candidate with legal action for doing this.

People will post Let’s Plays of Starfighter CTFs to Youtube. They will get together with their friends to to talk how awesome your company’s hiring funnel is. We won’t send them a cease-and-desist. We’ll send them a pizza.

“Aren’t you worried about players gaming your assessment?” The only way to game a Starfighter assessment will be to demonstrably possess the type of engineering skill which you want to hire for. Are we worried that players will teach themselves these skills just to play Starfighter? We’re counting on it. That would be an epic success.

You cannot buy hiring pipeline as effective as the one Starfighter will build for you.


The technology industry structurally excludes many qualified candidates from their hiring funnels and then is shocked when those hiring funnels disproportionately select for candidates who are not structurally excluded. Traditional tech interviews are terrible ways to identify, qualify, and evaluate top programming talent. Filtering by education level or university is unreliable. Keyword searches are applied by people who don’t understand the underlying technology. The tech industry excludes perfectly viable candidates for no reason at all.

(Case in point: Donald Knuth would be selected out of the hiring process for [senior C programmer with Unix experience] before any human had ever considered him at most tech companies. His CV doesn’t match the keywords C, programmer, or Unix.)

Starfighter is different. We attract the interest of a huge pool of potential talent. Tens of thousands of people will play Starfighter games for fun, and the act of playing improves their skills for free. Some will find them too difficult. Some players will sink their teeth into them, self-studying and rising to the occasion. A small percentage of serious players will breeze through all the levels faster than we can possibly create them.

You will want to hire our best-performing players, before someone else snaps them up.

You need people with skills, and we’d be happy to make the introduction. We run the CTF, then make the appropriate introductions under a standard contingency recruiting arrangement. You don’t need to make major changes to your hiring process to adopt Starfighter. (We’d be happy to suggest some, though.)

Everyone wins.

Starfighter has signed a few marquee clients. If you have hiring authority for 10+ engineers in 2015, send an email with details about your company to — we may be able to slot your company in for the first batch of candidates.


The 21st century is going to belong to developers and the businesses which employ their talents successfully. Unfortunately, the technology industry is fundamentally unserious as to how it presently identifies and employs engineers.

This is one of the primary causes of the hiring crunch. It is a primary contributor to structural impediments to entering our industry. This negatively affects many candidates, including (but certainly not limited to) those from underrepresented backgrounds.

It is also a multi-billion dollar problem. Persistent market inefficiencies should be music to the ears of a capitalist, because they suggest free money. The technology industry has, through neglectfully embracing hiring policies which are so irrational as to shock the conscience, created a gigantic mountain of free money.

Starfighter claims that lonely mountain as its birthright. We are going to slay the dragons guarding it and then strip-mine it. This will continue until either dwarves sing about how wealthy we are or until every firm in the industry rationalizes its approach to hiring.


We have the technical chops to build CTFs, which are virtually impossible to keep in production without a strong technical team and ongoing focus.

We have built software companies and worked with the best teams in the industry. We understand how software firms work, inside and out. We know how frustrating hiring is because we did it for years, too, before we realized how CTFs are a cheat code for life.

The Starfighter founders can talk the geek talk. We have walked the geek walk. Of the three of us, I’m practically the non-technical co-founder, and I solo-shipped two SaaS products. Thomas and Erin spent most of the last decade looking at software systems made by the most talented engineers in the world and then breaking them in ways so horrible that describing some of them could bring down Western capitalism.

We’ve spent years helping engineers level up in their careers. I have a folder in Gmail saving messages from geeks who used my career advice or salary negotiation tips to their advantage. Those two essays are, by the numbers, apparently among my most useful career contributions to the software industry. Now I’ll have the excuse to do more like them every day.

It could be sensibly argued that desire to please clients will make us go against the interests of engineers. Don’t worry. We’re brokers in a seller-dominated market. Our economic incentive is to maintain our reputation as honest agents for the most valuable W-2 employees in the world. Also, again, we’re geeks. We come here not to serve technology recruiters, but instead to replace them with a small shell script.


It’s possible that some engineers might be confused about how we’re going to make money. No worries.

Companies pay “contingency recruiters” a commission, generally calculated as a percentage of an employee’s first year salary, to introduce them to candidates. This is paid “contingent” on the candidate accepting a job with the company.

Starfighter is a contingency recruiter with access to a better way to identify candidates than “Call up everyone on LinkedIn and beg them to take a job at Highly Regarded Tech Firm In Your Area.” We assess for skill first, passively as players play our games and then actively. Our founders — talented technologists — personally reconstruct candidates’ solutions and evaluate them.

We follow-up with players to ask if they have any interest in a no-obligation chat about career options. If they’re interested, we have an honest geek-to-geek conversation.

Then, if appropriate, we introduce the candidate as deep into the hiring funnel at our clients as our clients will allow. It’s not “Yay, as your special prize for winning we award you permission to send your resume into their /dev/null inbox”, it’s “The CTO’s got an hour free at 3 PM on Friday; would you like to meet him to talk about joining their DevOps team?”

After the introduction is made, the decision is up to the candidate and the employer, but we’ll be following up with both to make sure the process is running smoothly. Clients will give Starfighter-sourced candidates their full attention immediately and process them in an expeditious and dignified fashion, as befits skilled professionals.

Contingency fees are not paid by candidates and don’t come out of their salary, any more than the company’s rent or marketing budget comes out of employees’ salaries. They’re a cost of doing business. Companies are happy to pay them because companies understand how hiring engineers makes-or-breaks their businesses. (Potential clients interested in getting the best terms possible should get an engagement letter signed before I convince the other founders “Why are we only charging market?! We’re better than all our competitors! Charge more!”)


Thomas, Erin, and I are presently hard at work building Starfighter’s first game. It will assess a variety of programming skills, including general systems programming aptitude as well as a few more… esoteric fields. We’ll have challenges appropriate to your skill level, whether you’re new to these fields or a seasoned pro, and we’ll have study guides, skill trees, and friendly geeks who love helping other geeks level up.

We plan to launch the game publicly in the near future. If you’d like to hear when this happens, sign up here.

A quick note from Patrick (in my totally-not-the-CEO voice)

I’m super-pumped about Starfighter, which is my main gig as of now. I’ll talk later on what it means to be quitting the whole self-employment thing. This was a major decision for me.

What does this mean for my other projects?

Bingo Card Creator: Bingo Card Creator will be sold. I hope to close the sale before the end of March. Our broker is FEInternational. I’ve enjoyed working with them so far. If you’re interested in BCC, please talk to them.

Appointment Reminder: I have no announcement to make about my involvement with Appointment Reminder at this time. We will, naturally, continue keeping all commitments to our customers.

Kalzumeus Software: Kalzumeus Software is going to continuing operating and will continue to be home to my eclectic side projects.

I have one major upcoming commitment on this front, the A/B testing course that I’ve been working on for far too long now, and am hoping to get it out ASAP to get it off my plate and clear the way for Starfighter. My co-founders have graciously let me delay full-time involvement in Starfighter while I get this out. Expect news on this front soon-ish.

We haven’t done any consulting in a while. We will continue doing no consulting, to the best of our inability.

Blog/speaking/podcast/etc: Software seems to be my life’s work. Yay. I’m going to keep writing and speaking about it, in all the usual places and under the Starfighter banner as well. I look forward to applying old tactics in new ways (what happens when you give a hiring funnel to an engineer who sees conversion funnels in everything? We’re about to find out!) and figuring out some new tricks as well. As always, I’ll be happy to teach anything I learn.

Kalzumeus Podcast Episode 11: Bootstrapping vs. Raising Money

Keith and I are joined by special guest Jay Winder, CEO of MakeLeaps, in this 11th episode of the podcast.  We talk a bit about doing business in Japan, raising money vs. bootstrapping as a SaaS company, how AngelList is going to eat the world, and the usual eclectic mix of topics.

[Patrick notes: The transcript below has my commentary inserted like this, as usual.]

What you’ll learn in this podcast:

  • Why you should negotiate from a position of strength and abundance.
  • How to raise a round of funding through AngelList while not being in Silicon Valley.
  • Why AngelList Syndicates are the future of seed stage fundraising.
  • How to manage relations with investors when you have lots of them.
  • How running a bootstrapped SaaS company is different than one which has raised early investment rounds.
  • How the SaaS market is different in Japan.
  • A bit about Jason’s new project, Sales for Geeks.

A brief announcement: Keith and his co-founded Rachel launched a new product recently called Segmetrics.  It’s Baremetrics, for InfusionSoft — gives you actionable, one-look insight into which of your InfusionSoft segments (e.g. traffic sources) are producing results for your business.  If that sounds relevant, make with the clicky-clicky.

Podcast: Customer Onboarding

MP3 Download (~60 minutes, ~59.3 MB) : Right-click here and click Save As.

Podcast format: either subscribe to in your podcast reader of choice or you can search for Kalzumeus Podcast in the iTunes Store.

Transcript: Bootstrapping vs. Raising Money

Patrick McKenzie:  Hello, everybody. Welcome to the 11th episode of the Kalzumeus podcast. I’m Patrick McKenzie, here with my noted co‑host, Keith Perhac and our good friend, Jay Winder as CEO of MakeLeaps here in Tokyo.

Keith Perhac:  I’m Keith. Welcome to the 11th episode. I can’t believe that we got this out, literally one week after our last episode.

Patrick:  This is downright scary.

Keith:  This is scary.

Patrick:  It’s almost like we have an actual podcast.

Keith:  I don’t think so. All right. Cool. Welcome, Jay.

Jay Winder:  Thank you. Hello. I appreciate the welcome. I was looking at you guys doing a podcast behind a thick Plexiglas window and I think, “Geez, that looks really warm and cozy inside that little podcast igloo”, so thank you very much for inviting me in. It’s a pleasure to be here.

Patrick:  We’re happy to have you from outside in the cold.

Jay:  Well, we have an air‑conditioner right here.

Keith:  Hopefully you guys cannot hear, otherwise our editor is going to be very, very mad at us.

Patrick:  Very, very angry at us.

Keith:  In fact…

Jay:  Should we turn it off?

Patrick:  We were testing a little bit ago and it sounded OK.

Keith:  You couldn’t hear it.

Patrick:  Alright, so we’ll be talking about a few different things this time. Jay has a SaaS business called MakeLeaps. Why don’t you just give us a little bit of background on what that is for people to have some context, and then we’ll talk about recent MakeLeaps adventures.

Jay:  The best and shortest way to describe MakeLeaps would be a FreshBooks for Japan. It’s essentially an online platform that helps Japanese freelancers and businesses more easily create and send their invoices, whereas right now pretty much everybody in Japan uses Excel and they do it all manually, which is crazy, and we’re on a mission to fix that.

Patrick:  As of a couple months ago, you guys are funded out of a few Silicon Valley movers and shakers. You want to tell us a little bit about how that came to pass?

Jay:  Sure. Around one year ago, I had the good fortune to meet Naval from AngelList, who’s a good friend of my friend. The first time I met Naval, he asked me a series of questions about MakeLeaps because he’s interested in the Japanese startup ecosystem. I answered all of those questions and at the end of that conversation, he very generously offered to be an advisor, which was very nice.

We thanked him for that and then the next time he came to Tokyo, I got a chance to see him again. He said, “What’s happening with MakeLeaps? Where’s everything going?” I said, “Oh, it’s great. We’ve got this traction and we just hired this guy and we’ve got this partnership and this deal and blah‑blah‑blah.”

He looked at me and said, “Can I ask you a few more questions about MakeLeaps?”

I said, “Sure, of course.” For about 30 minutes, he asked me a series of about 20 rapid‑fire questions, which I found very fun. They were all on point and interesting, so I responded.

Then at the end of that conversation, he said, “Wow. That all sounds pretty interesting. If you would be open to it, I would really be interested to invest in MakeLeaps.”

Up until that point, taking funding and investment had been an amorphous idea, we’d read about it in TechCrunch, but actually having an offer from one of the top most well‑known people in Silicon valley, offered to invest in MakeLeaps, kind of like Michael Jordan saying, “Hey, would you like a game of basketball?” It’d be crazy to say no, even if you like basketball a tiny little bit.

That began a process where we started thinking deeply, myself and my co‑founder, Paul. What do we want to do with MakeLeaps? How do we want to manage it? Do we want to take on investors?

That was quite a complex discussion we had over quite a long time. The more that we thought about it, we realized, “Well, if we really want to build a big important meaningful business, it’s going to affect a lot of people in a meaningful way, funding is going to be critical” is what we ended up realising.

I realized that, the Bootstrapper method is absolutely legitimate, and we did that for four years, so, I am not detracting from that at all. That’s a great way to go, but we had a situation where we could get some well‑known people onboard our team at MakeLeaps.

We decided that, it would be a great idea. We went ahead and did the funding round of $750,000 that closed a few short months ago.

Patrick:  Some of the participants for that, were Naval, Dave McClure from 500 Startups, Hiten Shah, a few others. It’s funny, these days, there’s a mechanism called a “Syndicate” on AngelList, which might be new to some of you who are listening to this.

Typically, in the days of yore, three years ago, a funding round for seed stage investment like this would be done by some of the collection between a 5 and 10, or 5 and 20 angels. Typically the minimum for an angel investment is about $25,000, below which, it wasn’t really worth everyone’s time in putting the deal together. Between $25,000 and then, say $100,000 on the high‑end, you would put together $250,000 to $500,000 like that.

One of the emerging models on AngelList is that, various smaller angels who don’t want to do deal flow management for themselves, which means they don’t want to go out in the world finding people who are like Jay, convince them that they should be allowed to invest in the company.

They want to outsource the deal flow management to somebody else, outsource a lot of the vetting of the deal to somebody else that they trust, and be able to invest some money in the startups that person invests in.

They can follow them on AngelList, in a way that is not like following somebody on Twitter. They make an informal commitment to invest in startups that the syndicate lead invests in. Collectively, this is called a syndicate, and then, some magic happens. I am not sure what the magic is, but my understanding is that AngelList makes some sort of vehicle where the syndicate can each put money into the vehicle, or is it AngelList just brokers the arrangement and the angel invests directly in your company?

Jay:  The way that it works in practice is there’s a person that has a syndicate, like you say. There’s maybe anywhere between 2 to 100 people that are following a syndicate. Then the syndicate lead will say something like, “OK, guys. I’m going to make an investment into this company. Here’s what it looks like.” At which point, depending on Angel List settings, and I’m not sure about this, it’s either opt‑in or opt‑out.

If people want to opt into the deal, they can say, “Yep. We agree with this deal. It looks great. I’m in,” or if they want to opt out, they can say, “You know what? I’m not interested in this space” or something like that, they can opt out. With that in mind, it’s actually kind of interesting. You don’t actually know, if you’re leading a syndicate, the amount of investment money that you can get until you actually decide to lead a syndicate investment and see how many people join.

Any person that’s running a syndicate who’s fairly well known that says, “OK, we trust these guys and we want to invest into them,” that’s a very strong signal. As Patrick said, if you’ve got some spare cash sitting around from Google stock share sales or something like that, you can say, “I’ll throw in behind all these people.”

Then what actually happens is if you’re an entrepreneur on the other side, you perhaps might start to trend on Angel List. Then you start get a lot of attention and people start to follow you and people might start to say, “Hey. Can we hop on the phone for a call?” at which point you lose a tremendous amount of sleep if you’re living in the Tokyo time zone…


Jay:  …because they all want to call either very late at night or super‑early in the morning. You’re a bit of a zombie for a little while. Hopefully, you can close enough deals on the phone or you can get enough people interested in investing that you’re ready to go. You can pull together your funding round.

We got very lucky. We set out to raise $400,000 and in the end we ended up almost double oversubscribed at $750,000, which would not have been possible without AngelList, especially because we’re trying to do this remotely. There are already some difficulties, where somebody can’t drive from Sand Hill or wherever they happen to be living in San Francisco to come visit you guys in Japan.

That’s a little bit difficult, so there’s a little bit of extra, I would say, latitude that we have to spend a bit of extra time and effort in trying to assure people that we’re good people. Sure, we’re doing this in a foreign market, but it’s B2B SaaS. It all looks similar. The numbers are similar. The market is huge and exciting and all the rest of it. You close a syndicate deal and then what happens is there is a single line on the cap sheet in respect of that deal.

All the angels invest into a holding company and that holding company actually makes the investment into your company. You get a single wire and a single name on the cap table and that’s it. You’re done. You don’t have to chase people to sign agreements or do any of that kind of stuff. All of that is done and fully managed by AngelList, which makes having 60 investors a cakewalk.

It’s simple. All of a sudden, you get a big wire of cash and a single entry in your cap table and that’s it. You’re completely done. That was transformative for us. Moving forward it’s not going to make sense to raise money using something other than AngelList. It just makes it so easy and straightforward.

Keith:  That’s a glowing recommendation if I ever heard one.

Jay:  Yeah, it was great.

Patrick:  This is particularly true for folks who might be raising from smaller angels or angels who don’t have a strategic profile to them. For example, if you’re someone who previously has some successes as an angel or as an entrepreneur and is known to have the capability to open doors, then a startup like MakeLeaps or another startup would be crazy to not jump through any hoop you put in your way to getting a check.

Well, given that it’s a relatively reasonable process. If you require a custom document ‑‑ a custom negotiation of the terms of the agreement, as long as it’s something reasonable, there could be founder time invested into getting a $25,000 check from you.

If, on the other hand, you’re J. Random product manager from Google, you don’t particularly have any successes behind you. Your main capability of contributing to a business is the ability to contribute a check full of a meaningful amount of money, but not a gigantic amount of money, then startups, particularly startups in demand, don’t really have a lot of reasons to spend huge amounts of their time on the phone with you, when they could be spending huge amounts of the time on the phone with either a better‑known angel or with their customers or employees trying to move the business forward.

Moving the amount of pain, the transaction costs of doing business with you down opens up more deals to you, which is probably one of the reasons that AngelList is going to eat large segments of the angel fundraising market, particularly on the low or less‑established end.

Jay:  Absolutely. It’s either “click, click” and you’re done, or “click, click, phone call, send a document, get somebody to sign it. Did they sign it? Oh well, they went on a holiday for three days. Try to call them again, but they’re not around. You need that document to close your deal,” or “Click, click, you’re done.”

It’s very clear that “click, click and you’re done,” the way that AngelList has made this very possible, and simple, and easy, it’s great. I can’t recommend it enough.

Keith:  It’s also easier for the investor, as well, because you don’t have to jump through all the hoops, as well. You don’t have to vet it as well as you would if it was a one‑on‑one.

You have other people. You have that social proof working for that, as well. It’s a lot easier for people now to invest, as well as be invested in.

Patrick:  Right, and we’re not trying to convince anyone out there in podcasting land to become an angel investor because the mathematics of it are exceptionally unfavorable unless you already know what you’re doing and you have lots of money to burn. Be that as it may, even knowing that the underlying math was difficult, the actual mechanics of doing angel investing two to three years ago were a little painful.

I have two very small angel investments, and because there’s actual contractual language involved we have to run it by a lawyer. Let’s say that I spent a high portion of my very small check size on having a lawyer look over four pieces of paper and make sure that I wasn’t giving up the general store in those four pieces of paper.

That wasn’t really something that I wanted to do. It wasn’t something that the startup wanted to do.

It’s like, “OK, we’re both professionals. We both have different bases of IP, which we want to mutually know is not going to cause problems for anybody, so the lawyers will be involved.”

With AngelList, if everyone is using substantially the same paper, which has been vetted by AngelList’s presumably top‑tier, VC, yadda, yadda legal firms in San Francisco, then there’s less legal risk involved with that paperwork.

Jay:  Very true. What’s interesting about the whole process is you can choose the set of documents that you want to use.

If you’re raising, say, $400,000 on AngelList, people will see that as, “OK, they’re raising 400k, they’ve got over 50 percent committed, and they’re using, let’s say, Y Combinator SAFE documents. Oh, OK, that’s a very clear, very easy to understand structure. It all makes sense.”

There are all kinds of different SAFE documents, so you can pick the kind that you want, and say, “OK, here are the base default documents that we’re asking everybody to sign. That makes things easier, as well. When we did our round we had an investor or two say, “Listen, the terms are good, but we’d like this favorable term,” or something like that.

When you’re doing an AngelList syndicate and you’re already almost double overcommitted, it becomes very easy to say, “OK, we appreciate that you’d like to be involved. You want this favorable term, however, we’re already double overcommitted and everyone has signed documents that do not include this favorable term. If you’d really like to be involved we’d love to have you involved, but the documents are the documents.”

It becomes very easy and straightforward to say that. Negotiation becomes very simple because everyone signs the same documents, and away you go.

Patrick:  What Jay’s saying about favorable terms is important. Traditionally in angel investing there is a few axes that the deal can move along.

One is fairly simple, “What’s the price this deal is being done at?” is something, which is fairly uncomplicated, which has not always been true.

You can imagine, using very simplified math, if I’m willing to buy 10 percent of the company for $10,000 that puts the value of the company at $100,000, which is much lower than any company would actually be valued at, but simplified math.

Those two numbers, which sum up the price of deal, are one way to evaluate the deal. There’s other terms that might be involved with exactly how the math works out in the event of an acquisition, or a down round, or various other things in the company, like, say, whether you get liquidation preference.

A liquidation preference is typically in the interest of the investors, not really in the interests of the firm. Certain liquidation preferences are borderline abusive.

Which doesn’t mean that they were never signed.  A few years ago there was not the understanding in the entrepreneur community that a 3X liquidation preference is not a market term.

Also, supply and demand. People were saying, “I will only give you my money if you include a 3X liquidation preference.”

Like Jay said, if you’re oversubscribed and the funding round is happening with or without someone’s particular $25,000 check, you can say, “Look, the term that we’ve negotiated with everyone is,” for the sake of discussion purposes, “a 1X liquidation preference, ” or “no liquidation preference, and that’s kind of the deal. You can choose to do this deal or not do this deal, but we’re not going to negotiate consequential terms like that with you.”

Jay:  The way that we ended up saying it was, “Listen, for us it’s very important to keep good relationships with all of our investors. If they found out later that one investor got preferential terms over everybody else, then that’s kind of a big, serious issue.”

There was a point that I needed to make that was fairly important, and that was the way that we ended up responding to people who asked for preferential terms. “Listen, we’d love to have you involved in this, but we don’t want to make a situation where the other investors feel like they got a bad deal or any one particular person out of 20, 30, 40, 50 people didn’t get that same deal.

“In the interests of keeping good relations between MakeLeaps and all of our investors, we’d really love you to sign the standard documents. That’s very simple, and easy, and then you can be involved, and that’d be great. If that’s difficult for you, completely understand. No problem at all. Please let me know what you’d like to do,” is the way that we handled that.

That was very smooth, and successful, and an easy way to explain what we were doing and why without treading on anyone toes.

Patrick:  This is, incidentally, something which is useful, not just in negotiating investment, but in negotiating all sorts of contracts. It’s something that your vendors will occasionally do to you.

A, if you’re selling, say, SaaS where there is an actual, physical contract in play, and it isn’t the standard contract of adhesion ‑‑ presuming you’re selling on the several thousand dollars of services every year [inaudible 17:35] companies ‑‑ you should probably have a template agreement, which you give to all of your customers.

Your first negotiating position can always be, “Well, our standard offer is…” blah. If they say, “Well, we would really prefer….” some term in there, “…which is more favorable to us than what you have right now,” you can say, “Well, this is the standard thing we offer to all of the customers at the level of service which you’re currently willing to sign for.”

“If you want to upgrade to our super‑duper platinum enterprise thing, we could discuss individualized contractual language with you. But, at the $1,000‑a‑month level of service you’re at, the $2,000‑a‑month level of service, we really encourage the use of our standard terms.”

Maybe you win that deal, maybe you don’t. That’s a negotiating tactic that you can do, which is fairly low‑risk.

Jay:  Exactly. If you’re speaking from a position of strength where you have other customers signed up to that same deal, you don’t need that deal to make your rent payment, or something like that, then it’s very easy to do.

Say, “Listen, here’s the deal that’s available right now. If you can sign it, great, if not, no problem. Best of luck, and hope to run into you again in the future,” is a very strong negotiating position to be taking.

Patrick:  I had this once for Appointment Reminder, where I’m trying to get business associates agreements, a particular type of paperwork, signed between Appointment Reminder and all of our medical customers. Many of them had already signed it, but some of them were waiting on it.

I have gone out to the folks who were still on the not signed state and said, “Look, we have a standard BAA ready. I’ve prepared it for your company. Please sign it immediately.”

Some of them are saying, “Well, we have to run this by process A, process B, and process C.” Most of these are not large accounts.

My position has been, “You have neglected signing this contract for a while, which puts us both in technical violation of some regulations, so this really needs to get done this week, rather than being done after the lawyers get through with it. Our current documents work. We know they work. We’d highly encourage you to sign the correct document.”

Keith:  What I find a lot, especially in consulting, is what you said Jay. It’s talking from a position of confidence, and power, and “OK, I’m going to be able to make my rent this week, do I really need this contract if they’re asking for all this stuff?”

What I’ve really found is that the more concessions you make, the lower your status becomes, the more power the client has, and the more things that they’re going to demand of you. In many cases, not all of them.

I think always working from a position of power, and from a position of, “OK, this is standard. This is the way we do things,” and, like Patrick said, “If you want extra things, it’s going to cost this much for adding that on.”

What a lot of people don’t realize, negotiation is a two‑way street. It’s not someone telling you what you’re going to do. You need to work out what’s beneficial for both of you, so that you’re both on the equal footing.

Patrick:  The magic words for negotiation are “mutually beneficial agreement.”

Keith:  Exactly.

Patrick:  Like Keith was saying, sometimes people who are maybe a little closer to us in mindset hear the word “power” and they get a little bit scared about it. Maybe we could think of it less of it being a power relationship and more about you having a mentality where, “This is one deal out of a universe of many abundant deals that are going to happen over the course of your business career.”

Given that, you do not need this deal to close. You’re in a position to hold out for terms on the deal that make sense to your business and to say, “OK, if the terms of the deal that are being offered right now don’t make sense for the business.” Then we will shake hands, we’ll part as friends, and maybe we’ll do business in the future, but not under terms which don’t make sense for me in the present moment.

Jay:  Of course, the classic negotiating idiom is two people want an orange. One person wants the skin of the orange, and one person wants the inside of the orange.

[Patrick notes: There are many SaaS negotiations like this, where e.g. the buyer sincerely wants the software to be maintained and the SaaS company sincerely wants thousands of dollars, and neither realizes that the other would grant that request in a heartbeat rather than it being a key point of contention.]

Then somebody cuts it in half and says, “OK, you both get half,” where both people could get 100 percent of exactly what they want if they talk about it a bit more, and understand each other’s goals, and what they’re trying to do in trying to divide an orange.

“Just buy two oranges,” is what I think actually listening to that idiom again.


Jay:  From that position, I think it’s always good to understand your negotiating position against somebody else’s, but let’s not get too far into negotiating theory and all that kind of stuff, though it’s interesting.

Patrick:  One thing I want to ask is you’re a Japanese company. You’re completely founded in Japan.

You’ve been running in Japan. You’re towards the Japanese market. Why Silicon Valley for funding?

Jay:  Good question. I suppose the best answer is because Silicon Valley came calling. Naval from Silicon Valley came over, got to know us, met us, found out a lot of information about us, and offered to invest.

From that perspective, we could have gone out and aggressively looked for Japanese angels and done an angel round that way. However, the Japanese angel ecosystem is not one percent of what it is in Silicon Valley.

I recently had a trip to San Francisco and I checked into my Airbnb. I walked outside and literally ran into a MakeLeap investor, completely randomly on the street.


Jay:  I’d only actually met him via video chat. We had two video chats.

He walked past me, and I was like, “Oh, man, I’m pretty sure that’s Tom.” But I was like, “Do I go over and accost him, and say, ‘Hey, do you know who I am?!'”


Jay:  Of course, that’s what I did. I’m Australian. I can’t help it.

I went over, and I said, “Hey, do you know who I am?” and of course he responded as if he was about to be mugged. He took a step back, and I was like, “No, no, no. I’m Jay from MakeLeaps.”

He’s like, “Jay.” I’m like, “Yeah, Tom.” He was like, “Ahh.”

That is the sort of thing that you can experience in Silicon Valley. [Patrick notes: People have come up to me on the street in Silicon Valley and asked “Excuse me — are you patio11?!”  Craziness.] If you walk outside in Tokyo, you will have to throw a lot of rocks before you hit an investor, [laughs] or an angel investor. It’s not so common here.

As Dave McClure says, “Angel investors are the limiting factor to any startup ecosystem,” and I agree with that. The value that you get from an experienced angel investor to give you advice, and capital, and point you in the right direction is really super‑invaluable.

We found that, once we did the syndicate round through AngelList, a lot of people became interested in Japan and the Japanese market, as well they should. Japan is still 15 percent of the entire world’s GDP.

It’s still the third largest economy in the world, full of affluent businesses and consumers. It’s a really exciting space, so people see this opportunity, and it’s rare that a company, especially run by foreigners, is aggressively focusing on focusing on and targeting the Japanese market.

[Patrick notes: I am perennially annoyed when people — inside and outside of our industry — treat Japan like its an also-ran whose time has faded, or are unaccountably surprised when Japan proves to have a lot of money.  One common article in the business press about e.g. iPhone penetration rates could be titled World’s Third Largest Economy Discovered To Contain Money.]

I guess, because of that, we also got a little bit of interest from people who’ve always been interested in Japan and wanted to find a way to get into Japan a bit more. We became that opportunity for them.

Patrick:  The understanding of investing investors, VCs, angels, service providing firms, companies, entrepreneurs, et cetera, our ecosystem is really important because all these things are interconnected.

For example, not only does Japan have fairly few angels who have operating experience in technology companies, it has few enough…Technology companies in the sense that we mean the word technology companies.

There are many tech firms in Japan. Most of them are not SaaS firms. In fact, I would have difficulty naming five Japanese SaaS firms.

Jay:  It’s not a common model in Japan.

Patrick:  Not a common model yet, so there aren’t people who have successful exits at these firms, who then turn into angels and start giving money to the next generation of entrepreneurs. Even though that’s the probably the same generation of the actual people. There’s not the sort of established best practices here for people who can….

There’s understanding in the market in Silicon Valley that a SaaS firm that has a particular metrics ‑‑ “A churn rate of two percent is really, really good amongst most SaaS,” for example. That there are probably on the order of 20,000 people in San Francisco who could tell you that off the top of their head, where in Japan you could probably go to a lot of professional angel investors here and ask them to define churn rate and have them be unable to answer that question.

Jay:  I agree with that. Yes, that’s true.

Patrick:  The three of us all have high hopes for Japan over the next short period and in the long‑run on both having successful exits, having this information percolate through the various channels, both Japanese and Japan‑resident foreigners, and hopefully having the market mature a little bit more here.

Keith:  I’m the only one in the room who doesn’t live in Tokyo.

Jay:  We’re working on that.

Patrick:  We’re working on that.

Jay:  I got one of you guys to come to Tokyo.


Jay:  It’s just a matter of time, Keith.

Keith:  Yeah, I’m number two, right?


Patrick:  It’s true. I’m in Tokyo largely because Jason is here. He and I have been good friends for the last four years.

Keith:  Also, it’s Tokyo, so it does have a lot of benefits over Ogaki

Patrick:  Tokyo is not Tokyo for everybody, although Tokyo is Tokyo for me now, darn it.


Patrick:  Darn it, Jason, you were right.

Jay:  That’s deep. That’s deep, Patrick.

That’s true. I’m so glad you think that.

Keith:  I think you need another drink.


Keith:  I did want to ask, me being not in Tokyo, not as much in the scene, when I think of a Japanese startup, what I’m really thinking of, and what I see a lot of, are Japanese megacorps who have that SaaS division.  [Patrick notes: Many Japanese companies have an empire-building complex.  In the US, Salesforce has essentially one product.  Dropbox has essentially one product.  Basecamp has essentially one product.  Their Japanese analogues own domain registrars, web development agencies, and — in many cases — restaurants or hotels.  I kid you not.]

I’m thinking of a lot of companies that…like Recruit. Recruit has a ton of SaaS companies, divisions.

Jay:  “Intra‑preneurs.”

Keith:  Exactly, and they’re very different than a bootstrap or a SaaS that’s going to be invested in because they’re owned by a giant, multibillion dollar company, right?

Jay:  Right.

Keith:  It really changes the landscape, so there are fewer investors. Companies do not buy SaaS here like they do in the States.

Jay:  Yes, there’s definitely a lot less exits and acquisitions in Tokyo, much to the chagrin of all of the angel investors, especially foreign angel investors.

You will see a company like Recruit say, “OK, we’ve identified a need in this market. Let us build a solution for that market, rather than acquire the established current leader of that market that’s going really, really well.

“Then we’ll put $20 million behind their marketing campaign and, boom, now we’re number two,” or three, or one, depending on the market and the company, which is not great. Then again, you don’t necessarily want to look towards giant companies in Japan to really support the startup ecosystem.

Keith:  Not at all.

Jay:  It’s not a good strategy.

Keith:  I work with a lot of marketing companies, especially in [inaudible 28:35] area. For the last four years, I’ve been really pushing for A/B testing, and analytic testing, and stuff for marketing tools in Japan.

I was talking to, literally, the largest digital marketing firm in Japan. I was like, “Yeah, you know they have all these great tools, Visual Website Optimizer, Optimizely, all these things, Crazy Egg, that you can use to test.”

They’re like, “Yeah, we know. We’re really pushing hard to create our own platform to help our customers.’

I’m like, “Why don’t you just help your customers by…”

Jay:  “Introducing a tremendously successful thing that already works right now?”

Keith:  Yeah, it’s like, “Why do you want to build…?” I understand why they want to build the competition, but it’s like, “You could be helping them now and build that on the side.”

It’s like, “We’re not going to touch A/B testing, we’re not going to touch marketing, until we build this tool.”

It’s still not built. That’s four years ago.

Jay:  Sure, of course. Of course, yeah.

Keith:  That’s what happens, so now they’re late to market, and…

Jay:  Of course. People underestimate complexity in software endemically, especially in Japan. It’s like, “Oh, it’s software. Just somebody build it. Easy, done.”

Especially for invoicing. A lot of people will have their own solutions, or a lot of people will be considering, “OK, do we go with MakeLeaps or do we build our own solution?”

[Patrick notes: The build-versus-buy decision for anything at a SaaS company which isn’t something you actually sell should come down to “Can I do this before lunch?”  If yes, build.  If not, buy or adopt OSS.]

The way that I look at this is very similar to, let’s say, an infrastructure question where somebody says, “OK, well, Gmail looks pretty good with your 500 of the most qualified engineers on planet Earth 100 percent focused on this 24 hours a day, but I really want to be able to control my own infrastructure.”

It’s like, “OK, so what would you like to do?” It’s like, “Well, instead of using an established platform run by literally the smartest people on Earth, that are qualified to do this exact kind of thing, I want my own server.”

It’s like, “Great, so when there’s a problem at 2:00 AM, 500 of the smartest people on Earth are not focused on your problem.”


Jay:  “They’re eating free lunch because today’s Mexican day at the Googleplex. That’s great for them, but you’re screwed because your email server’s offline for four days and your business stops. It makes obvious sense to start…”

Keith:  Japan has a huge thing about “not built here.” The thing is, we have a ton of smart people, and we have a ton of smart people who love solving hard problems.

They love solving hard problems, and that’s not a bad thing at all. It’s a great thing, especially for new IT companies in Japan.

The problem is that a lot of people would rather be solving those problems than using a solution off the shelf that solves the issue, that solves the business need, right?

Jay:  Right.

Patrick:  I find, in dealing with larger Japanese companies, that when they’ve identified an intrapreneur and said that, “OK, this person has put in 20 years at the company, they’re a known quantity. We’re going to stake a few million dollars of the budget to build a competitor to Optimizely or Visual Website Optimizer.”

They don’t think of what Optimizely and Visual Website Optimizer were when they started, something put together by one person or a very small team, thrown together in six weeks, and then exposed to the market for feedback.

They think, “No, no. We’re going to try to escape to where they are today,” and with several tens of person‑years of engineering effort involved.

“OK, well, if it takes 50 person‑years of engineering effort to get to where this is today, that’s all right. We will hire 25 engineers, and we’re going to put them on a project for two years.

“In two years from now, we will be where Optimizely is in 2015.

Jay: “We will release our unmarketable product.”


Patrick:  Aspirationally in 2017, they’ll be almost where Optimizely is in 2015 as experienced by someone who doesn’t really understand the product because they never actually used it in anger.

Keith:  I think a lot of people are listening right now are like, “Oh, no. It’s not as bad as you say.” I have had literary three Japanese clients last year that this was exactly their problem. They would not MVP it. The contract was for an MVP. We worked out a thing.

It’s like, “Take this to your client. Start selling it and then we’ll start building the next one.”

They’re like, “No. No. We have to have all the features. It has to be complete. It has to have the bells and whistles, invoicing and all this stuff before we even show it to our potential customers.”

Jay:  Here’s an interesting theory on this that I heard recently which I kind of agreed with and I think it’s fascinating.  Typically, Japanese companies and people approach software the same way as they have successfully in the past approached hardware.

Keith:  We’re a hardware country.

Jay:  Exactly. You can’t show someone hardware that’s 20 percent complete because that’s zero percent complete. Hardware is binary. It works or it does not work for its intended purpose. People have tried to apply that same idea to software and it doesn’t work in 2015 now. I almost said ’14 but it’s ’15 now. It makes sense to roll out something that works to fulfill the purpose that it’s built for and then you iterate from there just like software.

Keith:  There are two comments I want to make. The first one is that there’s this great Dilbert cartoon where the boss is saying, “Measure twice, cut once.” Dilbert’s response, “Well, when thinking measured is infinite and the cost of cutting is zero, it makes more sense to measure once, cut once and then iterate quickly.” It doesn’t work with hardware.

I keep telling this to all my American friends or my western friends in general. They’re like, “Oh, Japanese IT, software must be so amazing.” I’m like, “No, it’s not.”

You think Japan, you think electronics. That’s true. Electronics are not software. Electronics are hardware. You look at all the hardware, all the electronics that come out of Japan that have been amazing and they’re all hardware based, the Walkman, the CD, the CD‑ROM…


Jay:  …PlayStation.

Keith:  PlayStation. Oh, my god. PlayStation.


Keith:  Word processors, the toilets. It’s all hardware.

Jay:  Magic toilet. Oh, god. I love my toilet.


Keith:  It’s all hardware.

Jay:  That’s very true.

Keith:  The software in Japan…You look even at a simple website, it’s horrible.

Patrick:  With exceptions for maybe embedded programming and video games.

Keith:  That’s about it. This is interesting. We built an OS back in, I think, ’85, I want to say, that is the kernel level OS for pretty much 90 percent of ATMs in the world, but it’s so low level. There’s no user interface. There’s no nothing. It’s built on top of…We built tools. We’ve built hardware. We built things.

Jay:  When you say “we'” you’re referring to Japan, which is interesting.

Keith:  I’ve been here for 12 years. Almost my entire business career has been in Japan. I worked at an internship, then I worked at a startup job during my college years in America, and then I moved out here.

[Patrick notes: Keith cofounded an RPG company.  His comment on that: “Want to know how to make a small fortune in RPGs?  Start with a large one!”  It’s a horrible, horrible business to be in.]

Jay:  Your formative years in Japan.

Keith:  I know. I wonder if it’s hurt me. People tell me I have a very different thinking about business than western…I like it. I think it’s a good way of thinking.

Jay:  We all bring our own unique experiences to our work in all of this in much the same way. I came to Japan when I was 19 to originally study martial arts. Now, I’m running a SaaS business. Japan has been an interesting country for us both to grow in because you’ve been here for 12…?

Keith:  12 years.

Jay:  How old were you when you first came to Japan originally?

Keith:  I was at 22.

Jay:  Interesting. How about you, Patrick? You’ve been in Japan also for a long time.

Patrick:  Yup. About 10 years and change now. I think I came out three months after university ended. I’ve been here, essentially, for the duration ever since. All of us have careers has been shaped by our time in Japanese organizations or in foreign organizations that were operating in Japan.

It’s a pretty fair statement to say that if you took a look at the a day in the life of one of our businesses, it would not exactly look like it would if we had grown up in Tulsa, Brisbane or yada, yada and have the business grow up with us there.

Jay:  That’s true.

Patrick:  Or San Francisco.

Jay:  One thing that kind of blew my mind was I had to return to Australia for a five‑month period. This was maybe coming back like seven or eight years. I had to run a business from Australia and I had to do some business in Australia. What struck me was how simple everything was. It was so straightforward.

Keith:  [laughs]

Jay:  I needed to do something. I make a phone call and I have the thing that I need. I needed a contract. It’s like boom. “Here’s what we want to do.” It’s like, “All right. We’ll sign here and…”


Keith:  No huge back and forth and all that.

Jay:  It’s like, “What the hell is this? I have like two things to get done today. It’s 10 AM and I’ve done three things now.” It kind of blew my mind.

Keith:  I had a client who would not discuss anything over the phone or email. It had to be in person.

Jay:  That sucks.

Keith:  We cannot come to an agreement on things except by meeting and talking. There’s no way to come to an agreement over email or a phone call.

Jay:  That the magic of SaaS business.

Keith:  [laughs]

Jay:  When every single of your customers is paying you $20 a month and somebody says something like that to you, you can…

Keith:  Delete the account.

Jay:  Exactly. You know what? That sounds great. I’d love to spend the next six months talking about your $20 a month account, but no. [chuckles] No. No. No. No.

Running a Low-touch SaaS Company In Japan

Patrick:  That’s an interesting segue. Like we said earlier, there’s very few SaaS companies in Japan operating at any sort of scale. Most of them that are US transplant like Salesforce does very, very good numbers here. Many US enterprise companies do good numbers by merging the traditional enterprise sales culture of the US with the stuff that works in Japanese market.

On the low touch end of things, there’s not that much. What have you done with MakeLeaps that’s been successful with getting thousands of paying customers on the lower touch end of things?

Jay:  Well, one thing that’s kind of interesting in Japan, I would say, is an incredibly high level of customer support. In Japan, it’s table stakes. That does not set you apart from any of the other companies in Japan that also have an incredibly high level of support.

Patrick:  Can I jump in with an anecdote there that happened to me today? I go to a hair salon. I’ve been there maybe twice since moving to Tokyo. The young lady who cut my hair both times is moving to a different branch of the same hair salon in Tokyo.

She sent me a two‑page, honest to God, handwritten letter thanking me for coming in, visiting her twice and getting my hair cut. Giving instructions that I can give the next person that cuts my hair and directions to the new hair salon where I can find her if I want to travel 15 minutes out of the way to get my haircut.

I will be getting on that train to get my haircut, obviously.

You can imagine this is somebody whose per-visit value is what, 50/ 60 dollars for a salon haircut here. That’s a level of care that most SaaS companies that have account values the $5,000 a year range do not send handwritten letters to their customers saying, “Hey, we’re just dropping you a line because we like you.”

Keith:  If you do want to send them, I recommend MailLift.

Patrick:  MailLift?

Keith:  MailLift is a handwritten letter by API.

Jay:  Does it work in Japanese?

Keith:  It does not work in Japanese, but a great SaaS idea for anyone who wants to start a handwritten API.

Jay:  That’s a good point. There’s actually a bit of a problem in Japan right now, I think, where older workers are not able to assimilate into a lot of the newer roles that are being created as certain sections of industry are phased out.

Something like that would actually be interesting idea to look after the rapidly growing section of Japan, which are the older seniors. As we all very likely know, recently, adult diapers finally began to outsell baby diapers in Japan, which is terrifying.


Keith:  It’s only going to get worse.

Jay:  That’s very true. That’s only the start of the trend. There are a lot of potential business opportunities for older people in Japan for sure.

Keith:  I think it’s interesting though. I’ve wondered about doing a MailLift version in Japan. The thing that convinced me against it was that why would a company not get their OLs or their, I guess, window watcher is the best English term for madogiwa. Why would they not get them to do it?

Patrick:  Well, there’s actually companies…They don’t have the API piece. API piece is what scares them because you would have to convince companies on integrating an API to get business, which is not something many Japanese companies are very keen on.

Jay:  Considering that they don’t understand the word API. [chuckles]

Keith:  You could say integrates with Salesforce.

Patrick:  There are even at least five of these in Ogaki where the company is typically a printing shop or something where they have the traditional mass-scalable printing equipment there.

They also have, essentially, little ladies who will handwrite documents that you take to them with the idea being that certain classes of documents in Japan can’t really be printed and still have the kind of heft to them, personal letters, certain forms of certificates and whatnot. These require a bit of skill with Japanese calligraphy, which the little old ladies have. If you don’t, if you are the poor, semi‑illiterate graduate of the modern Japanese educational system [Patrick notes: sarcasm tags] who can only type, then you’d go and pay a little lady $20 a page to handwrite your customer facing documents.

Jay:  Very true. That’s a good point, but the other thing that I find a little bit concerning and perhaps another reason to not start this one particular business is sometimes, in Japan, the act of doing something because it is time consuming, ridiculous…


Keith:  Don’t even get me started on that.

Jay:  …and all the rest of it. The act of doing that is what’s valuable. If you were to outsource that to somebody else for $20 a page, that act, essentially, becomes meaningless. It’s a concern that I will have.

Keith:  We talked a little about this in the last podcast where we were talking about getting the busy work done feels good. It’s an extension of that where taking the time, putting the effort even though it’s not what is the best for you or the company, it feels good and it feels right.

Patrick:  Also, seen from the perspective of the customer, I think Jay is right. That sometimes the conspicuous expenditure of resources as a way to signal commitment, that’s something you see in all societies. There are rituals for it everywhere. It something that Japan embraces a lot.

Jay:  Very much so.

Patrick:  For example, in Japanese politics, if the ruling parties is having difficulties, they send out the equivalent of US congressmen to stand at stations in Tokyo outside of the gate. They get on a soapbox and say, “This is the Liberal Democratic Party. Please support us. We want to work together for a bright vision of Japan’s future. This is the Liberal Democratic Party. Please support us.” The person is basically on loop for three hours.

Keith:  I do work with political candidates as well for marketing and stuff. They actually have people whose job is to repeat a speech on a megaphone as they drive around the city for eight hours a day.

Jay:  This is one of the areas where I take issue with Japan. I love everything about Japan. I love the people, the food. Everything about it is great. Noise pollution laws, where are you on this one Japan?

Keith:  [laughs]

Jay:  What the hell is going on? I say this without doing any research, the only civilized country that has zero noise pollution laws.

Keith:  We do have noise pollution laws.

Jay:  Really?

Keith:  Oh, yeah.

Jay:  When you say “we do,” you mean Japan.

Keith:  Japan has noise pollution laws. I looked this up because I have a noisy neighbor. I think it’s 80 decibels or something like that. It’s ignored for many, many things.

Jay:  I see. Keith, as difficult as I’m sure your situation is with your noisy neighbor, I’m a little more concerned about the people who are driving around with megaphones on Sunday morning!

Keith:  No. It’s horrible.

Jay:  For me as well. I’m a foreigner so therefore I cannot vote in Japan. However, if there was a candidate that said, “You know what? I’m against noise pollution. I’m going to make noise pollution laws in Japan.”

I would totally support them even though that support would be meaningless. The problem is that person could never be elected because he could never announce his platform to all the people to put him to be potentially elected.

Keith:  It’s going to change because this was the first year. This last election was the first year. I don’t want to get too much in the politics so let’s kind of cut it.

Jay:  We’re leaving out sex and religion. Let’s go into politics. Go for it.

Keith:  This is the first year they could do commercials. If you went on YouTube during elections, there were tons of political commercials. For the first year they’re actually allowed to do social media.  They were allowed to do blogs before but they had to actually write them themselves. You could not hire a marketing firm to do it.

Jay:  For $20 a page. [chuckles]

Keith:  You could have a marketing firm tell you what to do and then you do it but you had to actually physically do it. Those laws are changing. I helped a political candidate. I’ll be helping another one with their social media platform this coming election in…When is it April or whenever.

Patrick:  That’s probably something to nail down prior to taking politicians on as clients.

Keith:  There are a lot of business opportunities in Japan. In Ogaki, there’s very little. It’s pro bono. This is totally a, “Hey, you’re a friend of my wife. We hang out. I’d love to help you out. Let’s do some social marketing for you kind of thing.”

Patrick:  That makes sense.

Keith:  I haven’t done Abe’s — the Prime Minister’s — marketing campaign.

Patrick:  You would probably have to know when the election is if you’d be doing that.

Keith:  There’s snap elections. You don’t know.

Patrick:  That’s right!  We could schedule elections like a week from now.

Keith:  They did do that.

The Difference In Bootstrapping And The Venture Track

Keith:  We are kind of getting long on time. I do want to be conscious of everyone’s listening time because we have had complaints that three hours is a long time for a podcast. had always been bootstraped and it was a big decision for you to take funding. How has taking funding change the way you’re doing business from being bootstrapped?

Keep in mind people who are listening. This is not like, “Oh, I got $5 million of funding.” It was a sizeable funding amount but it’s not like you’re going off and buying Lamborghinis tomorrow.

Jay:  If I could find a Lamborghini that I could buy a number of without causing problems for my bootstrap, then I’d consider that. It’s $750,000 of funding that we have to use in a really good way to either, I suppose, build in the features set that is very compelling and that enables us to get to the next level of the business.

We have to use that funding to, essentially, get us to a point where we’re totally happy and ready to go out and do a Series A funding round, which we’re on track to do within three to six months from now depending on how it all goes.

Your question to how did the tenor of the business or how did the atmosphere change before funding and after funding is a really important one and a good one. It’s tremendously important to have good mentors because I have a lot of very dumb questions that I needed answered. “Well, if I sign this, does that mean that I don’t run the company anymore?” Kind of level of questions like that.

Of course, I’ve done a lot of reading. [Patrick notes: Jay recommends Venture Hacks and pg’s blog on this topic.]  I’ve done a lot of learning and I’ve gotten a lot of advice from people. When you do funding, it comes down to two things, economics and control. The extent to which you understand both of those is very critical when it comes to negotiating those kinds of deals.

For me, I was more concerned actually about the staff and how MakeLeaps staff would feel about…Suddenly, we have a lot of investors because that’s kind of confusing. They would ask all the questions that I needed to ask to understand really what was going on and how it was going to affect things.

I made sure to talk to everybody and say, “Listen, we’ve done very well up until this point. We’ve managed to get many, many users, many, many customers. It’s all very exciting. We’ve come to the point where people are interested to invest money into us.

At this point, you have to know that I’m not getting to let some person who could be destructive or cause problems for MakeLeaps into an area where they could potentially cause problems for us. We’ve been successful up until this point by doing what we’ve done. It’s not in anybody’s interest for us to radically change things like completely different in a crazy way when we’ve achieved really great growth up until this point.

For me, it was critical to, number one, understand the impact of investors coming into MakeLeaps. What would they expect? How they could offer value and how they could help us get into the next level? It’s important for the MakeLeaps staff to understand that it’s not going to change the tenor of the business.

In fact, if it has changed things in a very serious way, that’s my fault as CEO. I haven’t done my job well enough.

Number one, I want to find the right people. Number two, I want to explain the process and how we’re going to do it. Of course, we have the MakeLeaps vision — where we want to go and how we want to get there. That needs to be clearly communicated to both staff and investors.

One of the things that I’ve learned is…Well, there’s actually a great film. I think it’s called “The Watchmen.” There’s a particular character in that film. His name is Rorschach. He’s like this crazy guy. He gets caught and sent to jail.

This sounds like a sidetrack, but it gets really relevant in a moment.

Keith:  [laughs]

Jay:  Essentially, he ends up having to go to jail. In the jail, there are many of the criminals that he himself put into the jail. He’s standing there and they can’t wait to get their hands on him, beat him to death and all that kind of stuff. He looked at all the people that are standing in the jail and he says “You know what? I’m not stuck in here with you. You’re stuck in here with me.”

That’s what I realized about investors. It’s like I will talk about MakeLeaps for literally 24 hours with anybody who wants to talk about it. I’m very, very happy to discuss and talk. When people say, “You know what, I’ve been thinking for MakeLeaps. How about we do something like this? What do you think of this kind of idea?” I’m very happy to discuss anything with anybody about that kind of stuff. It’s my passion.

I sort of realize, “Well, hang on a minute. We’re getting with AngelList syndicate 60 people that are now financially interested in MakeLeaps and they’re intelligent because they’re investing in MakeLeaps.

Keith:  [laughs]

Jay:  They’re clearly cut above the standard person.

Keith:  They know what they’re doing.

Jay:  They know what they’re doing. They’ve had plenty of experience. For me, it was actually all positives. We never had any issues or problems because we had really great mentors. We had great lawyers that put together the deal well. We communicated what we’re going to do and how we’re going to do it.

Typically, when it comes to relationship breakdowns, it’s just a lack of communication about what everybody is thinking [Patrick notes: In business and in life!], so we made it a priority to communicate as much as we possibly could. We send out very regular investor updates on what we’re doing and how and current progress and all that kind of stuff.

I would say that we’ve been able to expand a lot and be able to do many different things with more resources. The core that’s got us to this point that’s been successful remains unchanged and that’s a good thing.

[Patrick notes: Seen from the outside, I’d identify the core of what makes MakeLeaps tick as a willingness to walk the line between being a natively Japanese company and a tech startup, Jay being the most force-of-nature CEO I’ve had the opportunity to meet, and a great willingness to just grind things the heck out.  Anybody can say “Try going to where your users are.”  Most people don’t sell $20 a month SaaS software by knocking on the doors of substantially every accounting firm in downtown Tokyo.]

The Benefits Of Raising Money

Keith:  I want to ask something a little more specific and you’re free to not answer this if you don’t want to. Where did you find that you had the most success with that funding? Is it building out your staff? Is it being able to do more marketing? Is it doing more events?

Jay:  I’m going to tell you guys something that you might not be happy to hear. It’s the same way for all bootstrappers. We bootstrapped for four years proudly. We’re happy we’ve gotten into this point. We’ve done very well.

We’ve received, I would say, slightly less recognition because we weren’t funded, and we weren’t in the media as much with things like funding announcements. As soon as we got funded, about a week afterwards, this one person I’ve known for a long time was like, “Oh, you got funded. Great. We’d love to do an interview with you.”

It’s like literally nothing has changed between now and when the money in our bank one week ago. Why are you interested to interview me now?

Keith:  It’s the social proof.

Jay:  Exactly. There’s social proof in that. There are certain things that have gotten easier and better as a result of getting funding, especially from well‑known people, but, as a bootstrapper, it’s always frustrated me. It’s like, come on, we are doing as well as these guys. In fact, better in many ways, but they are getting more press because they are funded.

Keith:  I do see that a lot with the bootstrapper community, especially some of the communities I am in. I don’t want to say, incestuous, but it is a closed circle. The people we know who are famous, they are famous within our circle, but if I was to talk to someone like at a newspaper, or someone in the larger startup community, they’d be like, “I have no idea who that is.”

It’s really interesting, because as soon as you get that funding, apparently, it’s this open door to, “Hey, we are a real company. We are a real boy now.” It’s frustrating, like you said. Nothing has changed.

Keith:  There’s an extra digit in our bank account now, but other than that, at that point, you haven’t even used that money yet.

Jay:  It’s a little bit frustrating, if you are operating a business, and people are suddenly interested in you, now that you’ve gotten funding, but that’s part of the game. At the end of the day, you have to figure out, how to make it work with your particular environment, the way that you have set things up.

If you can be successful and grow your business, and get plenty of media attention without getting funding, more power to you, that’s great. In Japan, people tend to respect social proof very much so…

Keith:  Did I ever tell you about the optical fiber guy, who came to my door, and try to sell me optical fiber?

Jay:  I heard this story, but it’s a good one.

Keith:  It’s a good story. I like this one. I had actually had optical fiber at this time, not through that company. He comes to my door, and he says, “You know, you are the last person in your neighborhood who has not had optical fiber installed.”

I thought about that, I am surrounded by grandmothers. I am the youngest person in my neighborhood. The three people on the side of me, I know for a fact, do not even own a computer.  They have no idea what a computer is, most likely.

He is trying to tell me that I am the last person with optical fiber, which I already have. This is interesting, because I know this works in the US as well. It is a very time tested proven strategy. It’s so effective in Japan. It’s not even crazy. I told this to my wife, and she’s like, “Oh, we should get optical fiber.” We have optical fiber!


Keith:  No one in the neighborhood has optical fiber but us.

Jay:  Keeping up with the Joneses.

Keith:  Exactly. It’s huge.

Patrick:  I am just realizing now, I bought optical fiber after hearing that pitch.

[Patrick notes: Seriously, I did.  I thought they were going to drop connections to the ASDL lines in the neighborhood or something.  What can I say, I’m a software guy, not a network guy.]


Keith:  It works.


Patrick:  Crying about it…


Keith:  Get optical fiber. There’s no…

Jay:  Not have it.

Keith:  Exactly.

Jay:  Absolutely, get it right…

Patrick:  Great reason for all you geeks to come out to Japan, you can come to Ogaki, where we have slow, slow one gigabit Internet…

Jay:  Two.

Patrick:  Two gigabit Internet, yeah, I am sorry. We upgraded in Ogaki. We’re where in Tokyo was in like 1997. [chuckles]

Keith:  What’s your speed right now?

Jay:  I’m not going to share how fast my Internet connection with you guys. [chuckles]

Keith:  No. I want to hear. How fast is your Internet connection?

Jay:  Oh, geez. I’ve been meaning to upgrade it one gig, which would be simple, painless and straightforward, but I think I’m still like 300 megabits.

Patrick:  Oh, man.

Keith:  Oh, god. That’s horrible. You’re almost American.

Jay:  I know and I’m Australian. What the hell.

Keith:  I was talking to a friend who was interning at Gawker. He’s like, “Oh, man. I’m so happy to be at Gawker because we have a 25 megabit connection.” I was like, “Since I moved to Japan, I’ve never had a connection that low.”


Keith:  12 years ago, when I was on a ADSL, it was faster than 25 megabits.

Jay:  Exactly. 10 years ago when I had like a wireless PCMCIA card, originally it was slotted into the laptop.

Keith:  It was still going at 350…


Jay:  …Of course. [laughs]

Patrick:  Anyhow, do you think we should be wrapping this up?

Keith:  I think we’re wrapping this one up.

Patrick:  Jay, do you have anything else you want to tell us?

Jay:  I suppose. Aside from the confusion where my name is Jason, but now pretty much everybody calls me Jay. I’m right in the awkward spot of transition my name from Jason to Jay. There’s still a bit of Jay, a bit of Jay all over the place but I answer to either.

Keith:  Should I go for Jay?

Sales For Geeks

Jay:  You can go for whatever you want. Honestly, if you make a noise in my direction, I will answer. However, even if you say Jay or Jay that’s fine. Speaking of which one thing that I’m pretty excited about that I should mention that Patrick has been helping me with a bit is I’m working on a new course called “Sales for Geeks.”

As a geek myself…I mean, people describe me as technical, but I can’t code very well. However, I’ve built and grown two IT businesses, one on the infrastructure side and one on the software SaaS side, so I do know a little bit about sales, communication and social skills to a degree.

Keith:  I want to mention that Jay is a force of nature when it comes to sales, networking or any sort of talking to people.

Patrick:  I’ve said for the last couple of years that Jay could probably get a meeting with the Japanese Prime Minister simply by showing up at his residence and refusing to be told no by any of the people in there. Four years ago, I said that like it was a joke. As I’ve become more friends with Jason, I think, it’s probably literally accurate.

Jay:  At this point, I should probably just do it to prove Patrick true.

Patrick:  [laughs]

Jay:  The Japanese, sometimes, if you’re being polite and friendly and you’re saying, “I need five minutes of the Prime Minister’s time.” It might be potentially possible.

At the same time, I also have a sales course to get out. Priorities I suppose. It’s very nice of you guys to compliment me in that way. As an Australian, I do struggle with compliments and accepting them in general.

Patrick:  What’s the name of that course?

Jay:  The name of the course, Patrick, is Sales for Geeks. You can access more information about this course at

Keith:  It’s such a deal!


Jay:  Order now and get a free set of steak knives. There’s no steak knife. Sorry. Order now, though.

Keith:  You’re doing it as a text course, an audio course, a video course, or…?

Jay:  I’m thinking video might be a good way forward, because in a lot of sales stuff a lot of it is quite visual. There’s a lot of body language. That factors into sales meetings in general. A video course would be a good way to do this.

Patrick:  We’re looking forward to seeing what you come up with. You can go to the website right now and start to get Jay’s advice delivered over email about this as he gets the course put together, and then you hear about it when it comes out.

Keith:  No commitment required.

Jay:  No credit card required. Sign up right now.


Jay: if you’re interested at all in learning how to sell better. It would be great to have you onboard. I’ve got a bunch of stories, sales anecdotes, and interesting things that I’ve learned in a career of 12 years. I started my first company when I was 20. I’ve got a lot of stuff to share. If that’s useful or interesting to anybody, that would be great.

Keith:  I hate to talk Jay up this much. Actually, what am I saying? I love to talk Jay up this much.


Keith:  But, really, you’re up there with the three best salesmen that I know personally. I consider you, Ryan Delk, and Steli Efti definitely hands down.  [Patrick notes: Agreed.] If the three of you got into a room, either it would cause a singularity, and it would explode, or you would take over the world. It’s one or the other.

Jay:  Interesting. That’s very, very nice of you to say. All I can do is do my best to toil in the efforts of becoming the person that you both seem to think that I am.


Jay:  I appreciate both of your faith in me. I thank you.

Keith:  Jay, thanks very much.


Patrick:  We appreciate you guys being here with us for the long, long, long haul.

Jay:  It wasn’t that bad. It was about an hour and 10 minutes?


Keith:  …Yeah.

Patrick:  I was thinking more like the three years, which it took us to…


Keith:  Four.

Patrick:  Four years?

Keith:  Four years to get up to episode 11.

Patrick:  We ship babies and products faster than we ship podcast episodes, but it’s changing. We actually got up two in a month. Oh God, this is a regular thing to happen.

Keith:  This will be great.

Patrick: Drop either Keith or I an email about this stuff anytime, if you have ideas for what we could cover on the podcast.

Keith:  Ping us on Twitter.

Patrick:  Or you can ping us on Twitter.  Keith is harisenbon79, I’m patio11, and Jay is @JasonWinder.

Keith:  Suddenly, I feel like we’re on NPR. I don’t know why.

Keith:  “Thank you for listening to this edition of ‘This American Life.’ [laughs] This your host, Ira Glass.”


Keith:  Great show, by the way.

Patrick:  Great show, by the way.

Keith:  Thanks very much, guys.

Patrick:  Thanks very much.

Jay:  Thank you. Cheers. Bye‑bye.

Patrick:  Bye‑bye.

Design and Implementation of CSV/Excel Upload for SaaS

I usually write more about marketing/sales than I do about actually making software products, but I have recently been working on the product side a little more intensively for Appointment Reminder.  One of the features we implemented was CSV upload.  This is a very, very common task for virtually every B2B SaaS product, so I thought I’d share how we did it, as I’m pretty happy with how it is working out. Hopefully it will be useful to some of you.

The Problem With Upload Interfaces

Substantially every B2B SaaS product benefits from interoperability with other recordkeeping systems your customers use, including “formal” software (your competitors or products you interoperate with) and “informal” software like e.g. spreadsheets, Trello lists, and email-inboxes-used-as-a-database.

This is particularly true early in your customer’s lifecycle with your company: most data which will be new to your system is not actually new.  It presently exists somewhere in the customer’s organization.  You presumably want to make it as easy as possible for them to put it into your system and make your system the “source of truth” about that data.

Frequently, savvy SaaS entrepreneurs do this with “concierge onboarding” — basically, using high-touch human handholding to substitute for features which do arbitrary data source to your DB schema importing.  Why?  Partially because the human touch earns a lot of customer loyalty.  Partially because you can offer concierge onboarding as soon as you have one smart person who has an inbox and free time, without necessarily having to build a whole lot of software support for them.

Why I Punted On CSV Import for 4 Years

My launch list of features for Appointment Reminder included CSV import for client data (names/phone numbers/emails) and for appointment data (client data plus date/times of appointments).  Unfortunately, I assessed this feature as probably costing $100,000 in engineering time to implement well, so I punted and implemented a quick “concierge upload feature.”

Concierge Onboarding Upload

This was backed by me literally SCPing the files to the server, then opening the Rails console, and typing commands to parse out the file format in real time.  It would typically look something like:

lines = File.readlines("/home/patrick/new-upload.csv")

u = User.find_by_email ""

a = u.account

records = {|line| line = line.split; record = {}; record[:name] = line[3]; record[:email] = line[7]; record[:home_phone] = record[:mobile_phone] = line[12]; record}; records.size

records.last #Check to make sure that command actually worked.

clients = {|record| c =; = record[:name]; c.home_phone = "blah blah blah"; c}; clients.size

clients = clients[1..(clients.size - 1)] #Skip the header line {|c|}; "Clients saved!"

errors = {|c| c.errors.present?}; errors.size

#While there are still records with errors

c = errors.pop

puts c.inspect is "bob@gmail", needs a .com += ".com"


c = errors.pop

#Blah blah blah.

For a well-behaved CSV file, this took about 5 to 15 minutes of work. For clients who had very unclean CSV files, I often ended up doing an hour of data entry into IRB. That certainly makes sense for clients with predicted LTVs of $5,000+, but ideally I wouldn’t be doing this sort of work when I could be doing other stuff to drive the business forward. Additionally, the latency in sending me an email and actually having one’s account ready was 24~48 hours in the best of times, and that often caused clients to seek alternatives.

So why didn’t I just have a generic CSV importer ready? Because it is hellaciously difficult to do CSV import well in the general case.

Why is this?

  • Column-to-column mapping: Clients’ existing understanding of their data very rarely matches with your understanding of how the same data is organized, so you have to map their columns to your columns. This is often not a one-to-one mapping. For example, Appointment Reminder gives each client a single freeform name field, but many customers have systems which differentiate between first and family names (presumably because they think those things exist), so you have to have a way to stitch together those columns.
  • Pervasively unclean data: Informal software often includes data which is not exactly machine readable. Email addresses like bob@aol. Phone numbers like [(555) 555-5555 or -5556 (his mother)]. Names like [” Catherine] which make sense if you’re a human seeing it under [Smith James] but make no sense to a CSV file reader (and, incidentally, will frequently choke it).
  • File formats: You would think the CSV file format is fairly well-standardized. Hah. Hah. Hah. cries

As a result, the typical web upload workflow is very inadequate for CSV file upload. You’d like to just give them an upload control, grab the file, pass it to a background process, then update the UI with the result, but often the result is going to be “Lines 37, 45, and 392 had problems. Also, although you wouldn’t have expected it, we left off all the email addresses because you picked the wrong column. Whoopsie! Edit the file then re-upload.”

This is enormously frustrating for the user.

Enter SheetJS and Handsontable

I can’t remember how I stumbled upon SheetJS, but as soon as I saw their tech demo, I knew AR was finally going to get file uploads.

SheetJS is an OSS project which parses a variety of common spreadsheet formats, including (through libraries) Excel, OpenDoc, and CSV. Because it is Javascript, you can actually do this directly in the browser, which allows me to completely eliminate the “upload” stage of the “Upload your CSV/etc, we parse it, then we display feedback” workflow.

I can’t express how magical this is in mere words. You have to see it in action. Either use that tech demo or watch the following ~50 second video.

I’ve described this to my friends as “It’s like embedding Google Sheets into your application, for less than a day of work.”

Carefully Considered Glue Code

I had “minimal” import capabilities available in about 3 hours of work with SheetJS, but getting a decent editing workflow with Handsontable (the grid component which provides a lot of the “magic”) took a solid three days of work.


The above example shows a well-behaved Excel file. We heuristically guess what columns in the Excel file map to which columns in our data model. Files which are a) parsed and mapped 100% successfully and b) contain no errors at all… are not the most common case with CSV uploads.

I had to build a way for users to both a) quickly understand that the column mapping was not correct (ideally, without having them have to understand the phrase “heuristics used to parse your document”, since most are office managers) and b) correct it, without requiring a heck of a lot of explanation.

UX-wise, I figured that, as long as we were using the typical right-click context menu that people would be familiar with from Excel, we’d extend that with the ability to mark columns appropriately. Additionally, we’d add a bit of external-to-the-grid feedback to the user to make it obvious what their file was missing.

Then there’s the question of “What do we do about errors?”

I figured that, for AR’s particular use case, errors in a given line shouldn’t block processing of the other lines, since customer records typically don’t “bleed” into each other. Accordingly, we’d save all records which had no errors (similar to my IRB session above), then ask the user to bulk update records with errors and try again.

This is handy, since customers often have very consistent forms of errors in their documents, and the remedy for them (e.g. deleting an entire column of bad data) is often fairly easy to do in a full-featured spreadsheet interface. Putting that in the browser rather than back in Excel makes it minimally difficult for them to actually succeed in doing this.

Take a look at this 65 second video for a user who uploads 98 users successfully while also having two records with errors. (Don’t worry about violating anyone’s privacy: all of the data is faked using, a really handy service for cranking out CSVs/Excel files/etc which exercise various features in upload features.)

I rush through the process pretty quickly for the purpose of the demonstration, but in real life, it is close to this easy, particularly for a user who has done this before (like, e.g., my support reps or myself — see below).

What was tough about the glue code? Mostly, that it involves a lot of jQuery callback hell. Javascript development is, in general, not exactly my strong suit. 80%+ of development time for this feature was making sure that right clicking on the mobile column to mark it as mobile actually worked as expected. You won’t believe how many reloads I did during this process.

I give an A+ to Handsontable for documentation quality and a well-designed JS API. This project would have been enormously frustrating without that. SheetJS is a little newer of a project with a substantially broader brief, so it involved a little more source code spelunking to get working (particularly when integrating new file formats into the provided dropsheet.js magic which imports the dropped data into Handsontable), but I’d rate it as production-quality. Both sets of devs are quite responsive and substantially every B2B SaaS product should adopt them (hopefully adding a bit more polish on the UX than I did).

What Does The Backend Look Like?

Customers routinely upload thousands of records at once (which, n.b., neither of these OSS projects have any problem with). Given that my Rails app treats each row in isolation rather than using a batch insert statement, mostly to run validations, this means that doing the upload parsing in the request/response cycle is probably not optimal. You never want a request waiting several seconds unless you’re intentionally doing long polling.

I instead immediately stuff the uploaded data into a semi-ephemeral data store (Redis), then acknowledge to the AJAX request “OK, got it. ID number was $FOO.” The client then begins polling an endpoint for status updates about $FOO.

(Worth mentioning: why Redis as opposed to e.g. just persisting the files to disk? One, our Redis database is encrypted and our standard file storage is not, so not putting the files on disk saves us from having to worry about whether they include HIPAA-privileged data at all. Two, while we could theoretically clear out uploads on a regular basis, Redis has the expiry logic already built into it, so the data has built-in ephemerality. Three, it’s easy for my app to assert “We’ll always have a Redis instance available!” but not quite so easy to assert “We’ll always have a local filesystem shared by all processes relevant to servicing a particular user’s account” — if we eventually move to a multi-machine architecture then the file system suddenly becomes a really bad place to be storing uploads. Four, uploaded files are inherently a security risk if a misconfiguration lets someone execute upload.csv as e.g. a PHP file, and I’m decently certain there is no URL which will ever route to an arbitrary key in Redis and cause it to be evaluated as anything other than a string.)

Our queue worker processes then process upload $FOO, updating the database directly for working records and saving errored records as a new job (also in Redis). We then have the client poll result return both instructions to the user and new records to repopulate the table with. While our upload infrastructure at the model level is reasonably well architected with separations of concerns and everything, the bit which synchronizes the jobs and the UI is incredibly tightly coupled between the UI and the controller layer. That decision will eventually make maintenance of this more painful than it would be if they were decoupled. I shipped it anyhow, because shipping beats not shipping.

To give you a rough idea of complexity:

upload_processor.rb contains the logic orchestrating jobs, persisting to/from Redis, and turning JSON blobs which were POSTed to the server into client records. It is 335 lines long, and required approximately 20 lines of additions to our Client class to support. Unit tests run another 500 or so lines.

uploads_controller.rb turns AJAX requests into upload processor method calls. It is 135 lines, of which 35 are the incredibly coupled method for answering poll requests. I’ll avoid showing a screenshot for fear of being blocked as obscenity in the offices of good engineering teams everywhere.

The Javascript glue code is 450 lines long. No unit tests because, frankly, I have no clue how one would test this in an automated fashion. I’d rate it as C+ in terms of code quality — jQuery callback hell, what can I say.

Rolling Out To Customers

I don’t like exposing new features directly to customers, particularly features which I suspect are probably fiddly. For example, while I’ve tested this feature with documents in a variety of encodings and file formats, I have no clue what will happen if someone uses an outdated version of MS Excel on a Windows 98 machine to upload a file written in a Hebrew code page, and I’m honestly a little scared of finding out.

Accordingly, we put this behind a feature flag. Feature flags are structurally similar to A/B tests:

#In a view.  @account is, by convention, the currently logged in account.
#Account#allows?("feature") is our feature flag method.

<% if @account.allows? "uploads" %>
  Click here to <%= link_to "upload", {...} %> your client data.
<% end %>

Naturally, we also control access at the controller level (to prevent anyone from playing guess-the-URL), but this demonstrates the basic idea. We can assign users access to this feature individually or in groups, without having to roll it out to the entire userbase.

At present, client data uploads are available in production for our internal users only. Our administrator accounts can upload data into any account, but users can’t access the corresponding upload feature for their own accounts unless I hand-grant their account that permission.

This forces clients to continue emailing us their CSV files to get them uploaded, which lets me have the catastrophic, terrible UX, blow-up-straight-in-my-face errors that still exist in this feature happen to a patient and incredibly invested product owner (me) rather than an impatient user on their first day. For example, it wasn’t obvious to me, but if JSON interpreted someone’s phone number as an integer (5555555555) rather than a string (555 555-5555), our upload job would die with an uncaught exception and the polling method would continue polling forever. I fixed that and the user never even knows it happened.

This also lets us verify that the parsing/heuristics/etc work for someone’s particular “informal software” used at their business. Typically, they have one or a handful of Excel files, often maintained on a single version of Excel on a small set of machines. We can handle the first upload ourselves, verify that it works as expected, and then extend them the ability to do uploads. It is unlikely that they’ll add row-level data which comprehensively borks the upload feature assuming it works on their existing files at least once.

What’s Next In Our App

Now that we have the ability to do this for client data uploads, which are (by far) the easier of the two upload types for us, we’re going to deal with appointment data uploads next. These are a) less forgiving of error than client data, b) conceptually harder to deal with (parsing date/times… ick), and c) vastly more numerous than client data, since client data is typically essentially append-only with minimal editing after upload but appointment data changes on a multiple-times-a-day basis.

Luckily, we’ll be able to reuse a bit of the infrastructure which we built and also will, hopefully, have shaken out a lot of the UX/error handling/monitoring/etc challenges prior to playing the upload game on hard mode.

Presently, we have a single set of heuristics/parsers/etc which are used for all clients. Eventually, I’d like this to be plug-and-play on a per-customer basis, so that we could write ~50 lines of Ruby if required to slurp in a godawful file format required to support a particular enterprise client. c.f. the Strategy pattern in the GoF book, except less painful, because metaprogramming makes this much less painful than it would otherwise be.

A Brief Meditation on OSS

I estimated this feature as costing $100k in engineering time if I were scratch building it. We got it done in +/- 3.5 days of work, which is easily a 90%+ savings, as a result of SheetJS and Handsontable being available. I simply didn’t feel right getting that amount of value for free from two projects which are run by very small teams, so I approached both and convinced them to sell me an enterprise license to their project. It is equivalent to the usual OSS license, except it comes with an invoice. (I don’t think it would be appropriate to name numbers, as that might constrain their ability to price enterprise licenses going forward. Let’s say that my initial outreach emails — titled “Can I pay you to work on $PROJECT?” — probably sounded like a less-wealthy-than-usual Nigerian prince who happened to live in Tokyo and have a really odd interest in CSV files.)

If you have an OSS project which is as useful for for-profit businesses as these two projects, I would strongly, strongly advise taking down any mention of “donations” and offer commercial licensing for the project. This doesn’t have to change anything about your project. (What is the difference between a commercial license that is completely discretionary and a donation? I can donate money, and like most middle class people, my actual behavior with regards to donations is “occasionally donates tens of dollars to poor people and other deserving causes.” My company is literally not allowed by the tax code to donate money, but it can buy any software it feels like, it does not require “You must be poor” to write a check to you, and it assumes that software costs hundreds/thousands of dollars. Ask for license revenue, not for donations.)

If you’re willing to let commercial viability influence your choice of licenses, the dual licensing model also works pretty well, which my buddies at Binpress explain in more detail here. (Disclaimer: I’m a small investor in Binpress.) For example, Appointment Reminder would be totally unwilling to put GPLed code anywhere in our codebase (viral infection of the rest of the code is a non-starter for me), but if you (the copyright holder) said “We’ll let you use our code on a basis equivalent to an MIT license, if you pay us $X,000″, then (assuming the code solved a $X,000 problem for me) I’d write a check immediately.

More broadly, I think that SaaS businesses and other heavy consumers of OSS owe it to the community to provide the funding for projects which represent significant advances, as we are — frankly — much better than the typical OSS dev at actually monetizing software. I’m less worried about the likes of Ubuntu or Chrome, which have massive corporate backing behind them, and even much smaller projects like Rails do pretty well with a few larger sponsors (Basecamp, NewRelic, Heroku, etc) which full-time employ the largest contributors. I think it is right and proper for for-profit businesses to assist the labors-of-love OSS projects at the lower end of the scale though in going full-time on those projects if that meets their goals. So, where possible, I try to pay professional wages for professional work. There exist a variety of ways to contribute to OSS projects, but nobody’s landlord accepts pull requests as currency, so I prefer contributing with money.

Try it sometime — it’s fun and easy. I wrote two emails explaining that I wanted to buy a commercial license and that my only requirement from them was an invoice which said “commercial license for $SOFTWARE” and a figure on it. After collecting the invoices, I had my bank send checks/wire transfers as appropriate. Payments between parties in the global wealthy class (i.e. most software companies and developers) is a solved problem — don’t let whole minutes of hassle scare you away from trying it.

Kalzumeus Podcast Episode 10: Putting the “Family” in Family Business

Keith and I are back with the 10th episode of the podcast.  This time we’re talking about our wives and kids, how much they mean to us (lots!), and how we try to fit being good husbands/fathers around our mutual desire to keep growing the businesses.

[Patrick notes: The transcript below has my commentary inserted like this, as usual.]

What you’ll learn in this podcast:

  • What Keith has been up to with Summit Evergreen and what Patrick has been up to with Appointment Reminder.
  • How having children changed how we run our businesses.
  • How delegating tasks is key to making sure we can spend appropriate amounts of time/brainsweat on being good husbands and fathers, as opposed to optimizing Nginx config files.
  • How Japan’s poor systematic answers to the question of work/life balance decreases the birth rate here.  (Who said this podcast wasn’t educational?)

Podcast: Putting the “Family” in Family Business

MP3 Download (~53 minutes, ~50 MB) : Right-click here and click Save As.

Podcast format: either subscribe to in your podcast reader of choice or you can search for Kalzumeus Podcast in the iTunes Store.

Transcript: Putting the “Family” in Family Business

Patrick McKenzie:  Hi, everybody. Welcome to ‑‑ what is this? ‑‑ the 10th episode of the Kalzumeus Podcast. I’m Patrick McKenzie, better known as patio11 on the Internet. I’m here with my buddy, Keith.

Keith Perhac:  Hi, this is Keith. We are on the 10th episode, three and a half years in the making. Probably the slowest podcast ever. I know every time we say we’re going to make these a little bit faster and do this a little more regularly. Hopefully, in this new year, 2015, we’ll actually get that done. Here’s knocking on wood.

Patrick:  Knocking on wood. I think we ship products and children about as quickly as we ship podcasts.

Keith:  [laughs]

Patrick:  In fact, I think that’s almost literally true.


Patrick:  This segues nicely the topic for today. We’re going to be talking about what it’s like to run a business as two guys who are very committed to being family men. Not just to grinding away and burning the midnight oil and the work stuff, as we might have done in our younger and stupider years.

Keith:  [laughs] I don’t know. I still do that on occasion, but having a family has definitely changed it.

Patrick:  Yeah, so we are going to talk a little bit about family stuff in a few minutes, but we want to steal a [inaudible 01:06] from Rob Walling and Mike Taber at the “Startups for the Rest of Us” podcast, which is one of my personal favorites. They start off every episode with just a little update on what’s new and exciting in their businesses. I thought, “Hey that’s kind of interesting to pattern after.” We’ll try it and see if it looks it. Keith, we haven’t heard much about Summit Evergreen recently. Why don’t you start us off with that?

How’s Summit Evergreen Going?

Keith:  Summit Evergreen has been going pretty well. May was our official launch. We’re out of beta now and we have, I don’t remember how many customers we have. We have a fair number of customers doing a lot of sales. I am really happy about that.

Patrick:  For the folks who don’t remember what Summit Evergreen is, because it has probably been two years for them, it’s Summit Evergreen in 30 seconds.

Keith:  Summit Evergreen is a courseware platform. It helps people create and sell online courses on the Internet. Think of it like Udemy or kind of like the courses that Patrick does, like the one on lifecycle emails.

The difference is that compared to Udemy or Skillshare or whatever, it’s completely you-branded. You are not just another guy on Udemy. It’s all about you. It’s on your domain, your URL and it’s all about your product and your branding. That’s what we do and we just actually released our New Year’s campaign, which is our annual campaign.

One of the biggest things that we find issues with is that people do not commit to building courses. Everyone always says, “Oh, I want to build a course. Oh, I want to build a product.” Number one thing that has people fail is they’re not committed to it. There is a big psychological impetus, a big psychological effect when you put your money where your mouth is, when you commit something of yourself to your own success.

We launched the Annual Plan, which we started January 5th of this year, so that was yesterday. It’s essentially you get 17 percent discount, two months off, if you sign up for a year. We hope that not only are people going to enjoy the Annual Plan but they’re also going to use that as stopping creating excuses for themselves. It’s not going to be, “Oh, I need to get this course out someday. I’ll do it when I have time.” It’s, “OK, I’ve put my money where my mouth is. It’s time for me to get this done, and I’m going to launch this year.”

The people who do that, we see they’ve launched courses in five days, in one day, in hours, in weeks. People who are committed, people who are committed, who put that line in the sand, those are the people who succeed and successfully sell their course to customers.

Patrick:  Summit Evergreen has been, for folks who haven’t been following the story, it’s like a natural outgrowth of the consulting work that you and your co‑founder, Rachel, were doing over the last couple of years.

You were working with some of the larger publishers in the Internet space who have these courses online and sell them for, in many cases, large amounts of money, both on a per‑course basis and on an absolute basis.

You’re bringing some of the technology that you custom‑built for these publishers to the scales, where somebody like either the two of us could actually implement it without having to free‑write the courseware in Ruby on Rails like I did.

Keith:  Exactly like you did. A lot of the problems that people have is that when they start their own course, they’re like, “Oh, if I want courseware, I either have to have something crappy that’s running on WordPress or I have to hire a dev and pay him $80,000, $100,000 to build it.”  [Patrick notes: Mine probably was a +/- $40,000 job if you bought it at market rates in Ruby on Rails programmer time.]

We took the happy medium. We took all the knowledge, all the things that we’ve learned, built a system that works for actual product creators instead of people trying to sell software.

Patrick:  Awesome. You’ve just launched annual pricing for it. You’re starting to get a little bit of customer uptake and your early‑adapter customers are getting to the point in the lifecycle where they’re actually on‑boarding their customers, so they’re seeing success with it. What’s the next three‑month plan for Summit?

Keith:  We have a bunch of joint ventures, and I don’t want to call them affiliates but people who have publishing platforms already. Not software but publishing houses. They have lots of content creators. They want to come in and they want to work on Summit Evergreen and get their content creators onto Summit Evergreen as well. That’s one thing I think is very big for the next year.

As far as software and features for our customers go, we’re looking at better integration with lifecycle email service providers, better integration for that customer retention.  If you’ve taken Patrick’s course, you know that your existing customers are the number one way you’re going to grow your business. Your existing customers are generally 70 percent more likely to buy than the new customer.

Focusing on existing customers to build out product orbits, to build out retention strategies and things like that, that’s the way you’re going to grow your business. First quarter, second quarter 2015, that’s really what we’re looking to focus on, is increasing that customer retention and increasing peoples’ abilities to build product orbits.

Patrick:  That’s a great way to demonstrate additional value to your customers too, because many of them don’t have the know‑how of, say, a Brennan Dunn or a Nathan Barry, where those guys have, through a lot of hard work, made a bunch of products which naturally feed into each other.

They’ve got, basically, a self‑supporting ecosystem where they can step people through, “Make your business a little more successful by buying our book. After you’ve got the business a little bit more successful, that opens up new challenges for you, and I have a course which happens to slot directly into these new challenges,” to both keep increasing the value to the customer and then keep increasing the value to the business or content creator over time.

Keith:  Exactly, but one thing I do want to mention is that people look at these “product orbits” that they’re called, like Brennan Dunn, Nathan Barry, Amy Hoy has and they think, “Oh, I can never get 12 products out” or “I can never get three products out.” You only need one or two products, or two or three products, and they’re not hard to make.

If you look at Nathan Barry’s “24‑Hour Product Challenge,” which he did again this year ‑‑ really, really amazing. He and Amy Hoy both did it. Amy Hoy put out, “Just F‑ing Ship,” which she put it out in 24 hours. It’s a saleable product, she made some good sales on it and it doesn’t have to be this amazing course.

There’s a lot of content you can make at that beginner, beginner level. Those kinds of gateway products are really more effective as product orbit than larger products.

Patrick:  That makes sense. You want to hear my update on Appointment Reminder?

Keith:  I was just about to ask. What’s been up with Appointment Reminder?

How’s Appointment Reminder Doing?

Patrick:  I’ve kind of lit a fire under myself for the first time with Appointment Reminder due to my daughter being born.  I’ve been running Appointment Reminder for four years now. It’s been like the redheaded step‑child of my business for those four years.

I wasn’t really passionate about the problem space. I always treated it as an afterthought, even though every year I said, “OK, my goal for this year is to finally do some major work on Appointment Reminder.”

Right around when my daughter was born in October, I started to get kind of serious about it and, with the thought that, “OK, I’m going to execute on this seriously for 12 to 18 months, get it to a happy point and then do a check‑in on whether I want to continue with that business or spin it off to somebody, and sell the business and start doing something that I enjoy a little bit more.”

Since committing seriously to Appointment Reminder, it’s actually been almost fun. I spent two months last year working in a very “new dad,” fits and spurts fashion on doing infrastructure for on‑boarding my first couple of employees into Appointment Reminder. They’re actually consultants, not employees, but finally getting some help on the sales and support front, building the CRM integration that would let the sales rep do her work, yadda, yadda, yadda.

In the process of managing that sales process, I’m actually starting to do a little more of the day‑to‑day grind on it, which is helpful because if I don’t respond to inquiries from people then they don’t buy software, so I’ve been responding to inquiries.

We’ve lined up two opportunities for major integrations with platforms that trades businesses use. I’m working on those this week and hoping to ship them by the end of the month.  [Patrick notes: Looks unlikely at the moment.  The below-described customers backed out of their soft-commits.  Yay, enterprise sales!]

Both of them have a sponsoring customer, where we’ve got, basically, a soft commit. I sent them a one‑page letter of a text that they agreed to where, if we ship this integration by the end of the month, then they commit to buying Appointment Reminder at whatever the price is. The price is somewhere above where the publicly available plans are right now, but not hundreds of thousands of dollars.

Speaking of publicly available plans, I’m finally taking my own advice and charging more. As of February 1st, pricing is moving from $29, $79, $199 to the entry point is going to be $99 for a modestly higher quota than we’ve done before.

That’s actually scaring me a little bit. I was talking it over with somebody. I wanted the new entry point to be $49, and he said, as Keith has said on occasion, “Have I listened to myself for the last couple years?” The most successful clients that we have are the ones where the businesses are executing at a certain amount of scale. They literally have full‑time people who are just slaving away on the phones everyday. Appointment Reminder can replace that effort for much less than the cost of a full‑time person.

In contrast, we’ve got a lot of customers who have, say, five appointments a day. 5 times 20 workdays in a month is 100 appointments. They get value out of Appointment Reminder and they pay us $30 a month, but they’re not the heart of the business.

Honestly, it’s been brutally difficult to build up a business to a reasonable scale on $30‑a‑month chunks, so I’m going to be refocusing on the types of businesses where it’s worth at least $100 a month to them to have this problem solved, which is largely the trades businesses, medical, et cetera. Ooh, medical.

Keith:  [laughs]

Patrick:  We’ve had medical customers for Appointment Reminder for about two years now, but it’s been kind of on the DL, hush‑hush kind of thing, for HIPAA compliance reasons. We were kind of skirting the edges of HIPAA compliance, so we had promised HIPAA compliance to people on a very limited basis. They required us doing custom legal work with each customer that had gotten on‑board with it.

We weren’t ready to scale then. As of some engineering and process work that I’ve put in in the last two weeks, we’re finally ready to scale HIPAA‑compliant accounts for Appointment Reminder, which is going to require me going out to the 50 people who have medical businesses that are using us but are not on a HIPAA‑compliant account at the moment and getting them to sign a business associate’s agreement, which is the last bit of paperwork that we need for them.

Then, I’ve already flipped a switch on the back‑end to start treating their accounts in a HIPAA‑compliant fashion, which means…I won’t bore you with the details, because it could take the whole podcast, but we have to encrypt their information on a disk, which we’re doing for everybody now, and we have to enforce some procedural safeguards about access to their accounts, like enforcing a password rotation strategy, which we don’t do for normal accounts.

Keith:  I’m curious. Are you going to be upgrading those plans to a higher tier for that HIPAA compliance, or no?

Patrick:  As of February 1st, we’re going to have HIPAA‑compliant accounts available on the pricing and plans page. They’re just going to be straight‑up 2X what it would be for the same account tier at a non‑medical provider. Medicine is a business, right? You charge 2X as much for literally the same thing, but we promise you formally that we’re doing all the right things with regards to security, which we only promise informally if you’re an accounting firm, for example.

For our early adopters, for these first 50 doctors’ offices that are using us, I’m going to get in touch with them this month and say, “Look. Formally, you’re using $500 worth of services a month and you’re paying us $50 a month, but since you got in early and you’ve been with us for the last few years, I’m going to grandfather you in at that $50 a month or whatever it was pricing, contingent on you making sure to get this document signed for me this week. Because I need you to sign this document so that if, knock on wood ‑‑ this never happens ‑‑ but if the Department of Health and Human Services audits us, I need to have these business associate’s agreements in place with all of our medical customers.”

Similarly, if our medical customers get audited, they need a BA with us or they’ll fail the audit automatically. Many of the folks in the medical industry, especially on the lower end of the scale, not the major hospital chains but independent doctor’s offices, have been kind of dragging their heels on compliance with HIPAA, as they drag their heels on a lot of things.

Keith:  I can imagine. I’ll be very interested to see how the $99 entry-level plan works out for you. I think you’re right. It’s going to get rid of a lot of the low‑tier, high‑support customers. You’re only going to have people who are really serious about business. As we’ve stated before on the podcast, $99 to someone who’s serious is really nothing.

Patrick:  It’s literally like less than one‑third of the cost of publishing this episode of the podcast.

Keith:  [laughs]

Patrick:  That’s not even the time costs. We have plus or minus $250 of hard costs associated with publishing a marginal podcast episode.  [Patrick notes:  Transcript, $90.  Audio editor, $150.  Bandwidth, ~$100.  Finally getting a podcast out the door?  Priceless.]

Keith:  You figure, like I just bought a $20 recording software. You just bought a $20 stabilizer. It adds up. You look at some of our customers for Summit Evergreen, the people who are successful, the people who have big businesses who are getting a million dollars of revenue through Summit, they are people who have support staff. They have Zendesk or Freshdesk, they have a CRM, all of which are more expensive than Summit Evergreen, even though our lowest price point is that $99, right?

You think of a CRM like InfusionSoft? It starts at what, $150, $300. EntraPort, I know, is $350 a month. $99 is small potatoes to real businesses.

Patrick:  By the way, if you don’t follow my blog and didn’t read the end of the year update, starting to be public Appointment Reminder numbers. It’s currently at about $6.5K a month in monthly recurring revenue.

Keith:  Very nice.

Patrick:  Plus an undisclosed amount on Enterprise plans. My goal for mid‑2015 is to get that to $15K a month in monthly recurring revenue, which after you subtract out the costs of good sold for the business, so basically my massive Twilio spend and also after you subtract out the kind of on‑target compensation for my consultants, then that leaves me with about plus or minus $10K a month of profit for the business.

I think if it hits the magic $10K a month number, it will be in a good place for either me continuing it or for spinning it out into a variety of possible acquirers. We’ll think a little bit more about that. Why I want to spin it out in 6 months, as opposed to 18 months, is a story for another day. We’re not public about that information yet.

Welcome Lillian!

Keith:  Good. We want to talk about family today.

Patrick:  Yes, we want to talk about family. Can I give the news about the new entry to my family?

Keith:  I’m going to say no, but then I’m going to say yes. Yes. [laughs]

Patrick:  Ruriko, my wife, and I were pleased to welcome Lillian, our daughter, into the family as of middle of October. I know the date, but I don’t want the Internet knowing the date, because for some reason that is treated as secret information. It allows you to open up credit card accounts in the name of a two‑and‑a‑half‑month‑old baby. Who knew? That brings us up to a total of three in the family. Keith and company, you guys are up to four, right?

Keith:  We’re up to four, yeah. I have a four‑year‑old and a two‑year‑old now.

Patrick:  Both of us previously did time in the salt mines at Japanese organizations, so we are no strangers to overwork. Depending on the day, I think both of us kind of occasionally get a little too wrapped up in this. I think that’s fair assessment for me and maybe for Keith as well.

Keith:  [laughs] That’s a good explanation, I guess.

Patrick:  Yep. I’m doing my deep thoughts and where I want my career and life to move and what’s really important to me. Spending 12 hours a day doing integrations for software is probably not it.

Keith:  Let’s talk a little bit about our life before this entrepreneurial stuff. We’ve talked about when we were at the Japanese mega‑corps and how it was soul‑crushing, and we worked, God, 16‑, 18‑hour days, et cetera. When we left that…

You left before me. I kind of carried over that same work/time ethic, and I was working around 16 hours a day, I believe, at that point. 16 to 18, I guess. How about you? When you left your company and got over that initial shock of, “Holy crap. Now, I’m free,” how much were you generally working?

Patrick:  It went all over the place. I had a good six months there, and you might remember this, of just total burn‑out, where I did virtually nothing and just slept for most of every day. Probably a bit of undiagnosed depression going on there too.

Then, after I stabilized in the business and started to spin up on consulting, it went all over the place. The most consistent thing about my business for the 2010 to 2012 timeframe was inconsistency. [joking] Bah-dum-bump.

There were many days where I did absolutely nothing. There were many days where I just answered email for 20 minutes to an hour and then called it a day. Then, there were some days where ‑‑ at the time I was courting Ruriko. On a day where she was working, there was basically nothing else for me to do in Ogaki other than either play League of Legends or work on my business. There were some days where I got up in the morning at 11:00 or so, dragged myself out to a cafe, had some breakfast, and then just coded straight for eight hours and then went home.

Keith:  There were some days when you did the same thing, except instead of coding, it was League of Legends.

Patrick:  Yeah.


Patrick:  Or somebody once did a time graph of my Hacker News comments, and it is impossible to identify core sleep hours for me, which is a little disconcerting. There are some days when I was hacking on work or work‑related things at 4:00 AM in the morning and heck a lot more where I slipped into like 2:00 PM.  Lots of inconsistency.

One of the things that, since getting married and particularly since having a kid, I’m trying to get on a “more human time schedule.” My schedule today was pretty representative. I woke up at 8:00, played around with Lillian until about 9:00, and then left for the cafe at 10:00.

I typically take one hour a day to go to a cafe, have breakfast, listen to podcasts or something, kind of plan out my day. Then, I got into the office at 11:00 and I’m going to be at the office here from 11:00 to 4:00 working on a combination of, A, publishing this podcast and, B, my three goals for the day on what I want to get done for Appointment Reminder.

Then, at 4:00, going home, family stuff for the early evening. Then, I might do maybe an hour or two of work later in the 10 o’clock‑ish timeframe, which is peak hours for me.

Keith:  It’s interesting to me that you mentioned waking up, you said at 8:00 or 9:00, and playing with Lillian and then you went to the cafe, because when I quit my job, I already had my first daughter. There was still within me that I can work whenever kind of timeframe.

I would sleep in. The kid wakes up at 5:00 or 6:00 or whenever she woke up, because she was a baby. My wife, gracious that she is, would take care of her in the morning. I might sleep in if I had been up ’til 4:00 coding. I might wake up early and start coding. But I had that free‑form schedule, because the baby was just a baby lying around at the house.

I would go into office, but it was so free‑form. The number one thing that, now I have to have an actual time schedule. I have an actual time schedule each day. The reason for that is because my girls are in school now. They’re in preschool and they have to be at preschool at a certain time.

I have meetings starting at a certain time, because now I know that I can’t just have a meeting whenever because I have to time it with taking the kids to school, getting them ready in the morning, making sure everyone’s fed, making sure I’m fed. Same at night, kids come home 4:30. They practice their piano, do whatever.

Dinner’s at 5:30, because if dinner’s not at 5:30, they’re not getting in the bath. If they don’t get in the bath, they’re not getting into bed. They’re not reading their stories. They’re not going to bed. Suddenly, it’s 9:00 and they haven’t gone to bed yet.

It’s interesting, once the kids reach a certain age where they have a schedule, your schedule then 100 percent changes because you have to apply what you’re doing to them.

Patrick:  This is applying magic of consulting pipeline management to the day‑to‑day management of a household.

Keith:  To be perfectly honest, it has helped my business so much to have this rigid structure. We’ll talk about this more in a second. Having the structure is just really important, both from a work‑life balance perspective and then also from a productivity perspective.

Patrick:  I think the word “forcing function” might be relevant to that. Either of us could probably have pushed for a rigid, defined structure earlier in life, but having children concentrates the mind and gives you both a rationale for figuring out what the changes you need to make in your business/day‑to‑day productivity/life are to accommodate these sudden new demands on your time.

In a way that, since failure is not an option, given that we’re fathers, fatherhood succeeds where a bunch of New Year’s resolutions in the previous years did not.

Keith:  That one interesting thing. A lot of start‑ups and even Japanese companies, they talk about, “Oh, if you have a family, well, you’re not going to work hard for us. You’re going to be focused on your family.” I find myself not only working harder, but also being more productive because I have that sense of failure.

If it was just me and I lost my job, meh, so what, I’ll go find another job. If I lost my job having to feed a family of four, it’s a lot more stressful. It motivates me to succeed and do better, because I know I have so much more to lose. These people, my daughters depend on me.  [Patrick notes: Keith has a slightly different take on motivation than I do.]

They can’t go out and get a job. They depend on me to feed them, to clothe them, to put them in school and also to be a good father. I have to improve what I’m doing and improve the work that I do in order to be more successful for them. It’s no longer just my enjoyment or my life on the line.

Patrick:  Yeah, totally understood. I think if we get into an extended discourse on manhood or adulthood in the modern age, I definitely did not feel at, say, 29 [Patrick notes: I am 32] that I was a bonafide adult yet, despite running a business that was theoretically making some other folks lots of money.

Then, got married and didn’t quite feel like an adult yet. But now that I have a little infant who is easy to break and will literally die if I do not treat her correctly, I suddenly feel like a certified adult. Yeah, I’ve been kind of like kicking things into gear on that.

Let’s talk a little bit more about that Silicon Valley mindset, though. I orbit Silicon Valley at a certain level of distance, partly through Hacker News participation and partly because I had a number of consulting clients or less formal business connections out there.

I used to go there maybe twice a year or so to catch up with people. A very common thought in the start‑up ecosystem is that you’ve got to make hay while the sun shines. Your 20s are the time to work like an absolute dog, do 70 or 90 hours a week, get an exit. Then, after you have the exit, you can get married and have kids, if that’s your thing.

A lot of the folks in Silicon Valley look a little askance at that being your thing. It’s like, “Oh, he wants to have kids. Nothing wrong with that.

Keith:  [laughs]

Patrick:  Be that as it may, it’s one of those curious youth‑focused myopias that the industry has. There’s many, many jobs, which are objectively speaking more intense on a timescale or less forgiving of juggling multiple obligations than, say, working at a Silicon Valley start‑up or even founding a Silicon Valley start‑up is.

If you’re a partner at a law firm, you can’t just roll into the office at 2:00 PM and say, “Well, sorry everybody. Late night partying,” which most start‑ups will tolerate pretty decently, if you only do it occasionally.

Keith:  It’s interesting, because my father and my grandfather are both entrepreneurs. My grandfather, I think he started his main company at 35. My dad was, I think, 40 or 42 when he started his company. By current Silicon Valley standards, they were ancient. Who’s going to be an entrepreneur at 42? But they built huge, successful companies.

Patrick:  Can I tell you a priceless anecdote on that?

Keith:  I would love that.

Patrick:  Someone who read to meas being about like 23, asked me when I was 30, on a trip to Silicon Valley, “Aren’t you supposed to be a VC now? I mean, your career is pretty much over, right? You’re 30.”

Keith:  [laughs]

Patrick:  It’s like, now that is a forehead‑slapping inducing moment for me because, oh man, you’re so young. You understand nothing yet. I think both of us plan on doing this essentially forever. Probably 98 percent or so of the value that we’re going to create in our careers is in the forward mirror as opposed to the rearview mirror.

Keith:  Definitely.

Patrick:  It’s so crazy. This is one of the reasons that I can never commit to doing the Silicon Valley lifestyle, just because the social construction of that lifestyle seems to be, frankly, bonkers.

Keith:  Not all of it. You can’t generalize because there are some amazing things going on. You and I both went up there and were talking to a lot of people. It felt like college plus two.

Patrick:  College plus two. There’s so many things about that. [Patrick notes: I have a theory that several Silicon Valley ecosystems pretty much explicitly pitch working at a startup or founding one as a surrogate class that you do to earn the respect of your surrogate teachers.  This is very effective for a certain psychograph of high-achieving Stanford grads who have spent their entire lives doing anything required to please teacher and get an A on the exam.]  We might do that in a different episode. Let’s talk more about our beautiful daughters because they’re much more fun.

Keith:  [laughs] I actually want to talk about something interesting that we briefly mentioned on, which was how we have these time constraints and it’s forced focus. I remember a little while ago there was a talk about comedy on Twitter and how comedy on Twitter and how discourse on Twitter was changing the way that comedy worked and not improving, but creating a whole different genre of comedy because it was limited to those 140 characters.

It was talking about how limitations to art forms create better art forms in certain situations. It’s the whole limitations create creativity. I really think it’s the same with work. When you have forever to do something and you have no constraints and you have no set budget, let’s say. You have no set budget either for money or time or what needs to get done, things take a long time.

You decide, “Oh, I’m just going to do it myself.” You don’t have that focus because you don’t have any external constraints. We were talking about, especially when the kids go to school and, even now with you, you come home at a certain time. I have a hard stop. 5:30, lights go off in my office. I’m at home.

I can work after that. After the kids go to bed, I can go back to work, but at 5:30, I have to stop no matter what. Now, I have limited time and I have these hard stops that I must obey. A lot of people think that’s a bad thing and it’s not. It’s a good thing because this is the basis of business.

Delegating Responsibilities To Save Our Sanity

Keith: Because whenever you’re running a business, whenever you’re doing a business, there is so much more to do than you have time or the ability to do. You have to learn to delegate. You have to learn to prioritize. Having these external forces, these external walls making you, these limitations, it forces you to be smarter with your time.

It’s no longer viable for me to spend 20 hours deciding if we’re going to use Apache or nginx and tweaking the config files and everything, because there’s 100 other things that I need to be doing with my business besides setting up this one server. What do I do? I delegate it to someone. Because it takes five minutes of my time. It might take them 20 hours. That’s fine. That’s what I pay them for.

Patrick:  Do you find that you do more delegation now that you’re a little more constrained on the amount of time you can put into the business?

Keith:  Delegation has always been my hard point. I always want to do things myself because I love solving the problems, but it’s gotten to the point where I have to delegate. To answer your question, yes. I delegate much, much more.

I had done that talk at MakeLeaps up in Tokyo and you had re‑tweeted that. We were talking about meeting notes. The biggest thing I’ve done is delegating meeting notes, so that I record the calls. They go to my note‑taker, and they pull out the to‑do items and everything. I’m not sitting there rewriting all my to‑dos and putting them in Asana or JIRA, making all my calendar invites and everything like that.

Anything that is not moving the business forward, I think, needs to get delegated to someone on my team.

Patrick:  This is historically one of the big differences between your business and my business, because while we do roughly similar things ‑‑ we were both consultants for a while, we both have product businesses now with different levels of consulting still remaining in the business ‑‑ I’d always been kind of like a solo shop until recently.

You have had, for the last couple of years, a team of contractors who assisted you with the provision of consulting services and now help you out with that day‑to‑day business administration. It’s been a major thing for me that I was a solo bootstrapper, accent on the “solo,” for the first eight years or so of running my business. That changed recently.

I think it probably wouldn’t have changed but for the birth of my daughter being a forcing function. It must have been the TMBA guys, “Tropical MBA,” another great podcast, who said that there’s three levers that you get to adjust productivity in a business. You have your time, your level of savviness and money that you can throw at problems.

If you want to increase the number of sales that your business is doing, you either have to throw your time at making the sales, throw savvy at producing systems or being better at sales than the average bear, or throw money at the problem, either lead acquisition or in paying someone to do your sales work for you.

Like we said a little bit ago about constraints driving the business, back when I was salaryman, I had no time and I had no money. 100 percent of why Bingo Card Creator worked was savvy, savvy and more savvy on the media marketing front. That probably actually helped develop my savvy muscle, because I was exercising it so much.

Once I went on to doing my own stuff full‑time, I didn’t have as hard a constraint on time any more. While I probably increased on absolute levels of savviness, the percentage of my maximum capability of savviness that I was using on a day‑to‑day business and the percent I was growing in terms of savvy probably decreased, because there were a lot of problems where I could have solved them with additional thought but I just threw hours at it because I suddenly had a lot of hours to throw at things.

Keith:  Right. It’s interesting, because not only do you have a lot of hours to throw at things but menial tasks are easy to do, right?

Patrick:  Right, and they certainly feel like you’re winning when you’re doing them, which is the most toxic thing ever.

Keith:  I just took all my addresses from Evernote and put them into Excel so I could print out Christmas cards. Like, “I was very productive today,” except for the fact that that took me eight hours. That’s a full billable day that went away. Or I could have hired a VA to do it for $10, $15 an hour. What’s the bigger win there?

It feels like I was so productive, but when you look at the benefits to the company or just the benefits to my life, that’s eight hours I’m never going to get back. I could have done so much in those eight hours. But it felt good while I was doing it.

Patrick:  I think some of the work like that has that seductive quality of feeling like work, even though it doesn’t meaningfully drive the business forward. I’ve been doing a lot of architecture/underpinnings work to get Appointment Reminder in a better situation to move more accounts in the next six months.

Yesterday, for example, I spun up a new server to decouple the marketing site from the infrastructure for the main application.  I’m glad it happened. It has needed to happen for a few years now, but rationally speaking I should probably have found a sysadmin available for this and just thrown a couple of hundred bucks at them and had them do it, rather than losing that day or two to make a 90 percent solution.

I’m trying to get better at that. I’m finally working with a salesperson to do some of the sales activity. I have a, knock on wood, repeatable sales process in place for her, so that I copy and paste it out what I’ve said and done before that actually results in us closing deals. Then, we’ll have her turning the crank at it everyday, rather than me turning the crank on it.

Identifying As A “Solo” Entrepreneur

Keith:  Exactly. I think it’s interesting, that you had that pride of being a solo entrepreneur, a solo founder, a solo product guy with your software. It’s interesting because, when I started doing the consulting the products, I already had one child, right? I never had that time freedom. I think it gets really interesting that, now that Lillian has been born and you have those time constraints now, you’re like, “OK, now I can’t do this solo thing anymore. Because if I do the solo thing, I’m going to be spending all my doing this. I’m never going to see my kid.”

Patrick:  Yeah. There’s ways I could reconfigure the business to make the solo thing still work with both achieving business objectives and not crunching my quality of life at the same time. But it’s like, “Well, you get a certain number of decisions that you’re allowed to make, to move to levers.”

At the end of the day, I like this identity of myself as solo entrepreneur but I’m not willing to compromise on my family, because that’s a terminal value for me. That’s not on the table. Then, it’s like, “OK, do I compromise more on what my desired identity is or do I compromise more on what the desired trajectory for the business is?” I was like, “Well, the solo entrepreneur chapter was a great chapter in my life. But starting a new chapter, I feel like I’m kind of ready for that.”

I might cry a tear or two about my internal conception of what my business is like, but that’s the only change that’s going to happen to it. On the positive side of the ledger, it’s going to grow much faster and help other people out for supporting their families too, give my customers a better experience and let me do more fun work on a day‑to‑day basis. Hmm, tough decision here.

Keith:  Exactly, exactly. For me, that self‑identity I think is important but you also think, “OK, growing the business isn’t just for growing the business,” although with personality types like us, it’s kind of like a game, right? You want to grow the business because we like seeing the numbers go up.

Patrick:  It is totally WoW gold for me.  [Patrick notes: I care quite a bit about my metrics for each year, almost entirely for ego reasons.  Crushing my last year’s numbers makes me feel like I’m winning the game.  In terms of actual money?  Not nearly so interesting.]

Keith:  It is, it really is. But at the end of the day, growing the business and making the business perform better gives you two major benefits for your family. One is, the more automated your business is, the more time you get to spend with your family. Two is, the more successful your business is, the more money you have in order to have that financial freedom with your family.

I went to the States for three weeks a couple of months ago. I was talking with one of my dad’s friends. He’s retired now. He was talking about when he was in his 30s, he had built up a number of brick and mortar stores. He had managers and people were managing them. He was sitting at the pool with his co‑founder on a Wednesday afternoon, just sipping margaritas.

He was like, “Isn’t this amazing that we have this huge multimillion dollar business with 10 stores throughout the area? We’re sitting here on a Wednesday afternoon, sipping margaritas at the pool.”

Patrick:  Man, that is the life. Aside from the margarita. I’d be doing iced cocoa.

Keith:  [laughs] I’d get bored with it. I would get bored with it, but I’d like the freedom to do that.

Patrick:  Definitely.

Keith:  For me, the business automation, the freedom gives me two things. One, it gives me the freedom to spend the time I want with my family. It gives me the freedom to focus on the parts of the business that I want to focus on. I don’t have to worry about putting out the payroll checks and making sure that all the checks get out there, and pressing submit on all the forms, because someone’s doing that.

I can go in and say, “OK, we need to re‑look at our automation funnels.” Or, “Hey, this marketing page is getting kind of stale. Let’s re‑do it.” Or, “Hey, why don’t we call Jeff and set up this awesome new joint venture?” It gives me the freedom to be able to do that when you have a successful business. It gives you freedom when you have an automated business. It gives you freedom not only in your business life, but also in your personal life.

Patrick:  Yeah. One of the things…I don’t know, I have a weird relationship with money. I think that’s true of many people. We both got paid at the Japanese scale when we were working in Japanese organizations, which for somebody right around our age, it’s like plus or minus $30,000 or $40,000 a year. It’s not exactly a secret ‑‑ both of our businesses are doing a little bit better than that these days.  [Patrick notes:  n.b. Keith’s is a whole number multiple of my business and that whole number is not 1.]

Keith:  [laughs] My wife’s very happy, because she can now buy butter. That’s her major thing about me starting my business, is now we can buy butter instead of margarine.

Patrick:  The biggest concrete change it’s made to my family’s quality of life is, for semi‑related to Lillian being born reasons, we decided to move from Ogaki to Tokyo. The rents here are…

Keith:  Astronomical.

Patrick:  Pretty outstanding.

Keith:  [laughs]

Patrick:  My rent is literally more than my last salary was in Japan. Obviously, this would not be a viable option, at least not living in the apartment I currently live in, had we stayed in that trajectory, but it is…I won’t lie, when I see the rent get debited from my bank account every month, there’s still a little bit of, “Eek, that’s a whole lot of money,” But the business allows my family the opportunity to live in a place that my wife really loves and I’m really starting to like, and that I think will be a bit better for our daughter.

Keith:  Actually, it’s interesting that you mentioned that, because that’s something that we talk about constantly is, because of the business and the way I do my business, we have absolute freedom to be wherever we want. The problem is, when the kids…the kids are still in preschool. We don’t have this problem yet, but when they go into elementary and junior high, we’re suddenly stuck at a place.

We can’t go off to the US for three months because we want to, so it’s really making us make the decision. OK, what do we want to do? Do we want to raise them in Ogaki, which is not a bad place but it’s not cosmopolitan? There’s old ways of thinking. Do we want to live in Tokyo? Do we want to live in Osaka? Do we want to go to the States? My wife wants to go to southern Italy. I’m not learning Italian. [laughs]

Patrick:  [laughs]

Keith:  But it brings up a lot of lot of questions about what do you want to do for the family and what do you want to do with your life. If If I was a salaryman, there would be no way to make this decision. It wouldn’t be a decision. I could not make this decision. This is where my job is. This is where I have to live.

Patrick:  Or the decision might be made for you without consulting anyone.

Keith:  Oh god, yeah. The “Hey, now you’re working in Tottori for no reason.”

Patrick:  Japanese companies, for those of you that haven’t worked for them, have this wonderful cultural thing called tanshin funin (単身赴任) which means that the bosses can come up to on Monday morning and say, “Hey, guess what? From Tuesday, you’re going to be working out of our New York office,” or a place much less desirable to live than New York. “BTW, it’s just you. Your family won’t be going with you. We’ll start you off there for three years, but we could call you back at any time, or move you anywhere else at any time.”

Keith:  Yeah, and that’s not only abroad either, because abroad, maybe you’d move your family over there, although most people don’t. They have them inside Japan as well. You’re a three hour bullet train ride away from your family, everyday except Saturday and Sunday.

Patrick:  I know an engineer who was close to us in age range and life circumstances. He just got told that, “Oh, I hope you’ve had a great time living in Nagoya, where you’ve lived your entire life. We will now name one of the most rural prefectures of Japan where we have a factory. You are going to be supervising it, starting tomorrow. Have fun.” Japanese companies, oh boy.

Keith:  [laughs]

Patrick:  That’s another discussion.

Keith:  But it fits in with our discussion of family.

My wife actually works with me now. She does our sales and marketing on the Japanese side. She’s really become an amazing marketer. She has her teaching degree, and she was going to be a teacher. One of the things that teachers have in Japan is, the first three years, you don’t get to choose where you go. In fact, you don’t get to choose most of the time, but the first three years especially, they send you to some god‑forsaken place for no reason whatsoever.

Gifu, where we live, is a fairly large prefecture compared to other prefectures. Other prefectures, it maybe takes an hour to drive from one end to the other. Ours is about…what four hours, three hours? It’s a huge prefecture.

We’re in the absolute south, and they just love to send all the new teachers from here to the north. She’d live five hours away by car. At that point, it’s like, “I have a family. I have a husband, I have a kid. Am I going to live five hours away for three years, just to become a teacher? Then, maybe I’ll get to come back?” It really puts some big questions, and she decided to not become a teacher because of that.

She had the highest ranks in her school. She was number one ranked for all her teaching certificates and everything. She didn’t become a teacher because they would have forced her to move five hours away, and I couldn’t go.

Patrick:  I think a lot of people will make the other decision too, which is given the needs of their career they will sacrifice the dreams of having a family on that altar ‑‑ one of the reasons why the Japanese population growth rate is hitting the lowest of low levels.

Keith:  It’s interesting. Everyone always talks about, “Why aren’t people having kids in Japan?” There are two very big reasons. First of all, we’re worked like dogs because if you have no time and you’re too tired when you get home, there’s no way you’re going to have kids. The second one is, there’s no societal way to raise kids here. All of the preschools are completely full. I had to wait outside for two days to get my kid into preschool.

There’s no babysitters. There’s no after‑school services or anything for little kids. It’s getting slightly better, but not good enough. The reason is, Japan didn’t have nuclear families until recently. It used to be you lived with grandma and grandpa, and they helped take care of the kids. Now, no one wants to do that. Even the grandmas and grandpas, they’re like, “I don’t want to live with my kids. I want to have my own life.”

But society is not catching up to that. This idea that we have in the US of play dates, where you send your kids over to someone else’s house and they all play together ‑‑ then, one day they all come and play at your house et cetera, and the other parents get to take a day off. They don’t have that here.

Patrick:  Or the institution of babysitting. If you explain babysitting to an average, middle class Japanese person, you get looks of unrestrained terror on their face.

Keith:  In fact, I’ve actually been told, “Why would you trust your kid to another person? How could you?” Especially a 16‑year‑old or an 18‑year‑old. “I don’t know, that’s how I was raised. I [laughs] never had any problem, I’m still alive.”

Patrick:  This is one of those things where I think sometimes cultural differences are oversold, but this is definitely an American thing, where it’s like, “Yeah, sure. I’d like to entrust my four‑month‑old infant and my house and all of my possessions to a 12‑year‑old girl from the neighborhood. My only reason for trusting her is that she is from the same neighborhood, and that one person said two sentences of recommendation on her behalf.” It sounds totally reasonable.

Keith:  [laughs] When you put it like that, I don’t know how my parents ever decided to do that. One of my babysitters was actually named Candy, believe it or not, when I was younger. I don’t think she lasted long. [laughs]

Patrick:  Anyhow…

Keith:  [laughs]

Patrick:  Funny we should mention that, but given that it takes a village to raise a child, burden tends to fall in family in Japan, rather than falling on the rest of the village. One of the things that I’m hoping to getout of growing the business is we’re hoping to be able to support Ruriko’s mother moving to Tokyo. It’s been something on her to‑do list for a while, but not something that she can afford on a pensioner’s income.

If we were able to help underwrite that, that would both achieve a goal for her and make life a bit easier for us in having close family close by, helping to raise Lillian.

Keith:  I think that’s amazing. That’s the first I’d actually heard of that. I think that’s amazing. I think it’s good to have those business goals that are not business goals, right?

Patrick:  Yeah, like there’s a purpose to all this grinding that we do everyday.

Keith:  Exactly, exactly. I have a goal for this year that I’m not going to say out loud, because I don’t want to jinx it. But I was talking with some of my friends. One of my friends, in particular, he says to his accountant, “Give me one number. This is the number that I need every month in order to live the life that I want to live, so I don’t have to worry about money. It takes into account all my general expenses. It gives me a $500 or $1,000 buffer every month that I can do fun stuff with. Give me that number.”

That’s the number that his SaaS has to reach every month. That’s the number that he shoots for. That’s what he focuses all his attention on, is raising that MRR number to that point. It’s the same for me. I have a goal that I want to do this year, hopefully in the summer. In order to do that, I have to have a certain amount of money coming in from the business. That means I have to automate it. I have to make sure that the business is growing in the right direction, because even if I make the money and the business goes down, we can’t do it.

It has to be far enough that we have that safety net and that number, and that’s our goal is to be able to reach that and to have this fun thing that I’d like to set up.

Patrick:  I would suggest just as a micro‑tactical thing, put that goal on your dashboard. I had my revenue goal put for Appointment Reminder on my dashboard at the bottom of the screen, which is the wrong place for that.

Keith:  [laughs]

Patrick:  Because as the business got more successful, the goal got pushed down such that I would only ever see it if I Control+F’d and searched for it. Stick it to the top of the screen instead, so that everyday or every Monday, you check in and say, “OK, where are we with regards to plan?” Give yourself a little bit of a kick in the pants to do the work this week, to get you a fraction closer to it.

Keith:  Actually, I think we’re going to need to start winding down this episode but I did want to mention one thing off of the having that number in front of you. In another life, I did the inter‑office dashboards. It’s essentially giant TVs with the KPIs, the numbers that matter for the business or matter for that department.

There’s this great psychological effect that, when you have that number in front of you constantly, you’re subconsciously always thinking about that number. You’re always thinking, “OK, how can we increase that number? How can we reduce that number, if it needs to be reduced? What can we do to move that number?” It’s no longer this thing that I’m thinking about once a week in the shower, “Oh, yeah. We have to increase our opt‑in rates,” let’s say, “or decrease our churn rate.”

It’s something that is always in front of you. Having those numbers ‑‑ not a ton of numbers, like maybe five to six that are important to you and show you the core of your business, psychologically, it’s a huge motivator for getting to accomplish those goals.

Patrick:  It totally tracks with my experience. It’s easy, even our really important numbers like yearly revenue, if you don’t have it automatically updated, you might not know where that is. That’s sort of a surprising statement, right? But until getting my bookkeeping systems improved recently ‑‑ which, by the way, a shout to Bench, they’ve been instrumental in that ‑‑ I did not know my yearly revenue until I did taxes every year. I had a rough guesstimate like, “Oh, it’s certainly above 100, uh…”

Keith:  [laughs]

Patrick:  There were a bunch of times over the years where I had laser‑like degrees of precision with regards to my conversion rate, or I could tell you how many customers I had down to like three significant digits. If you asked me how much money the business made last year, it would be like “Um, I don’t know.” Yeah, graph it somewhere and make it automatic, but if you have company events, make that one of these things that at the weekly standard meetings, “And remember, our KPIs are X, Y and Z currently. The goal is this. Do your best on moving them in the right direction over the next period.”

Keith:  It’s very important, like you say. You said, “X, Y and Z.” It’s important not to have a ton. Everyone starts thinking, “Oh, I’ve got to monitor all these numbers.” These numbers are all‑important, but when you have too many numbers, you’re not looking at any one.

Patrick:  Right. You can spend an unlimited amount of time in Google Analytics, on optimizing the conversion rate to email sign‑ups of people who are entering through this particular blog post…but that’s not ultimately where the core growth of the business is going to come from. It’s not where the major changes to your life or your employees’ lives are going to come from.

Keith:  It goes back to those what we were talking about earlier, with the hard stops ‑‑ the limited time, limited focus. Pick three, maybe five at the most. These are the things you need to focus on. Just like in my to‑do’s every day, I know I have a limited amount of time. I pick three goals that must be done and two stretch goals. That’s my to‑do for the day.

Patrick:  Makes sense. All right. Well, I think we’re in a pretty good place for this episode of the podcast. Thanks everybody for your patience. Keith and I are, knock on wood, going to be trying to get these out a little more frequently, although it sounds like it feels some deja vu.

Keith:  I think we’ve said this for the last two years now. Although I have to say, this podcast was only about 40 minutes, 30 or 40 minutes. Very easy to knock out. Hopefully, we can get these out a bit more often.

Patrick:  Yeah, I’m looking forward to it. As always, we’d appreciate your thoughts on the podcast, whether the new format works for you and what you’d like to hear about. Drop either of us an email. My website is at, which…well, you’ve presumably figured that out, if you’re listening to this.

Keith’s is at Summit Evergreen, where you can sign up for some email about his service, if you’re interested in that. Our email addresses are readily accessible.

Keith:  All right. Thanks very much, Patrick. Thanks everyone for listening.

Patrick:  Thanks a bunch, Keith. Thanks everybody for your time, and we’ll see you again in a few weeks.

Keith:  Cheers.

Kalzumeus Software Year in Review 2014

I’m Patrick McKenzie, perhaps better known as patio11 on the Internet. Back in 2006 I created a side project selling software. This eventually morphed into something a little more. Since 2010, running this software business has been my full-time gig.

Every year, I publish a writeup of how the year went, what the statistics for the business looked like, and what I tried that went well or went poorly. This is partially for my own planning purposes (very useful, by the way — I recommend you do a writeup, too, even if you only publish it to your hard drive) and partially in the hope that other folks can use bits of it. You can read the write-ups for 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, and wow do I feel old.

(Hint for Googlebot: If someone is looking for Bingo Card Creator Year in Review 2014 or Appointment Reminder Year in Review 2014, this is the right place.)

What’s new about this writeup?: In previous years, I’ve been coy with the numbers for Appointment Reminder. This has always be mildly irksome for me, as I would prefer to have them here, but I had justified it because I was perpetually on the fence about perhaps taking investment money at some point, and having numbers publicly available would complicate that process.

For more thoughts on why, see below, but I’m virtually positive I won’t try to raise funding for Appointment Reminder, so I’m deciding to burn those ships behind me and run the business the way that I’d rather, and that includes being present in this writeup. (To the maximum extent possible compatible with my commitments to clients, vendors, and contractors, at any rate.)

Where are the expense numbers?: Way down below. I previously broke down expenses on a per line of business basis, but due to process changes with how we do bookkeeping, I can’t do that anymore. That breakdown was always kind of handwavy — for instance, I allocated 100% of our server expenses to BCC, which was true in 2006 but ludicrous today — so I’m just killing it.

Capsule summary: This was a pretty good year business-wise. I didn’t quite hit the goals which I had set out last year, due to some major reconfigurations of priorities mid-year, but I’m reasonably satisfied with it in most respects. I finally feel like Appointment Reminder is making sustainable progress. BCC is basically a vestigial project at this point.

My one major regret is that I was pulled in a few too many directions and, as a consequence of that, didn’t hit a shipping target for one of my products.

My businesses grossed roughly $200,000 in sales and produced roughly $120,000 in profit. (These numbers are slightly fudged for sensitivity of enterprise deals — see below.)

Disclaimer: For the first time in my life, I actually have really solid books for the business (thanks Bench — more on that later), but for continuity with my informal style for previous years, the following numbers do not match the books. They’re also my best guesses as of today, which is complicated, since we haven’t closed books for December yet. Treat these as approximations rather than audited financial figures.

The Year In Brief

Appointment Reminder finally got a bit of attention paid to it in a sustained fashion. For the last four years running it has theoretically been my main business priority. Practically speaking, though, I’ve treated it with benign neglect.

This year our monthly recurring revenue is up by about 35%, I’m no longer the sole person in the business, and I have a much better handle on where it is going than I did previously.

Bingo Card Creator continues to be in maintenance mode. Sales continued to fall off a cliff, largely due to a decline in traffic from Google/AdWords and my neglect of it, which may or may not be related.

Productized consulting produced my greatest regret of the year: the course on A/B testing which I’ve been working on for a year and took pre-orders for last December didn’t completely ship in 2014. As to why that happened, don’t worry, we’ll talk about it in a minute.

Since I didn’t feel right spinning up new projects with an existing one which was stalled, I did no training events, no new products, and no launches for old products, which caused substantially below-plan sales.

I did one small consulting engagement, even though I thought I was done with consulting.

I didn’t do any new angel investing this year, mostly out of lack of bandwidth, both in terms of attention and in terms of ability to write checks. I tried to continue helping out the folks at Riskpulse and Binpress. I also started working as a formal advisor (formal paperwork, actual duties, an equity grant, etc, as opposed to informal “Yeah, I love talking about this stuff, email me any time” that I do with basically any geek on the Internet) with a pair of startups, MakeLeaps and another which I can’t talk about yet.

Appointment Reminder

Appointment Reminder is a SaaS product which sends automated phone calls, text messages, and emails to the clients of professional services businesses, ranging from tutors through HVAC companies on our publicly available plans to nationally-renowned hospitals and service firm chains on our enterprise plans.

In principle we’re capable of providing services at anywhere from about $5k to about $100k per year to enterprises, but in practice, we haven’t closed a six figure deal yet. I want to someday, more for the merit badge than the money, but when we were very close to the finish line earlier this year on getting one, I had to withdraw due to inability to provide services in the required timeframes.

I can’t tell you the exact number, but pretend the largest check we’ve ever received (not in this calendar year) was for $75k, since that is roughly accurate. I walked it into the bank and was shaking because I thought they’d shoot me for attempting a robbery. They didn’t.

Before I get into the numbers, a bit of context about goals: My software business was a sideline to my day job prior to going full-time in 2010. I started Appointment Reminder literally days after quitting, and at the time I thought “For the moment I’m going to run this similarly to my existing business, without stressing too much about it or jumping in with two feet or taking investment, because I don’t know how life is going to change as a result of going full-time.”

I didn’t predict that I’d soon meet the most wonderful woman in the world and successfully court her (lifetime conversion rate on marriage proposals: 100%, booyah) or that I’d fall backwards into running a fairly in-demand software consultancy.

While this was happening, I often thought “I might try to take Appointment Reminder to The Next Level (TM) sometime.” I spend a lot of time on Hacker News and am peripherally involved in the startup community, and a lot of folks have wanted me to roll the dice on a funded startup with big put-a-dent-in-the-universe ambitions. At times, I wanted to want that for myself, but for the moment I was content to keep running my business in the traditional fashion. I work mostly on what I want to work on, take a day off whenever I feel like it, and optimize the business for quality of life rather than for any particular growth or financial targets.

To maintain optionality for taking funding, I have avoided speaking publicly about Appointment Reminder’s trajectory. Well, to be honest, that’s the excuse I’ve given. At least half of the real reason was that I’ve been disappointed with Appointment Reminder’s performance. If you had asked me in 2010 where I thought AR would be at in 2014, I’d have said “$500k a year or so.” It’s not there. I feel embarrassed by this — some people are accountable to investors for quarterly performance, and I feel an analogous sense of obligation to a lot of people who think I’m a smart guy and should, therefore, be running a business with a certain amount of scale.

Is it weird to say that is distinct from actually wanting the money? I wouldn’t mind a little bit more to establish more of a buffer for my family, but other than that, money qua money doesn’t buy me anything I don’t already have. It might as well be WoW gold.

As long as I’m putting fairly personal thoughts in the report this year, let me add another one: Appointment Reminder has been one long bloody slog. I have not frequently enjoyed the business until up to a few months ago. Why not? Well, a bunch of reasons, most keenly one which Peldi warned me about back in 2010: I don’t really care about the problem it solves.

I’m not the world’s most avid player of bingo, either, but with Bingo Card Creator I really sunk my teeth into doing low-touch software sales and marketing over the Internet. That was an enormously fun and engaging problem, even though making bingo cards is not. (The app proper has been described as “Hello World with a random number generator” and that’s only a tiny exaggeration. I have played perhaps two games of bingo in the last ten years. The only time in recent memory I used it myself was when a Redditor asked for anti-Bitcoin bingo cards, a request which I am unquestionably the best qualified person in the world to answer.)

I like that Bingo Card Creator has helped to teach millions of students to read, and I like that Appointment Reminder helps get patients to the doctor on time and puts the children of some of my clients through school (via increasing the revenue of their businesses), but these businesses don’t really excite me.

Peldi told me I should do a business which excites me. I told him that Appointment Reminder was a boring problem space but a great business, and that I’d find some challenge in it which was different than what I’d done with BCC. As it turns out, AR didn’t offer a lot of fun new challenges — it offered the same old been-there-done-that work that BCC had offered, and while doing BCC for the first few years was a form of play, AR was very much work. I often avoided things that would be clearly beneficial for the business just because they were a boring grind, to focus on other things, from the consulting business to my personal life to, well, far too many games of League of Legends.

I’ve recently sunk my teeth into systematizing Appointment Reminder’s sales operations and this is, for the first time in ~4 years, a fun problem to work on for Appointment Reminder. That said, it’s like maybe a 7/10 fun problem. I realized recently that, while I had been saying “Yeah, maybe after I grow Appointment Reminder a little bit and want a new adventure I’ll take investment for it and shoot for the moon”, but intellectually speaking I know what that would have to look like, and it commits me to working for another 5+ years on a product/problem/market segment which, on its best days, is 7/10 in terms of fun.

Life is too short to do that. I certainly do not intend on devoting 90%+ of my career for the next five years to Appointment Reminder, so I won’t take investment for it. (That would be unfair to the investors.) Given that, there’s no harm in sharing the numbers.

Appointment Reminder Stats

Appointment Reminder’s key stat of interest is monthly recurring revenue on our publicly available plans, since this is the most predictable revenue in any of our businesses and thus lets me make consequential decisions like “Move the family to Tokyo, where rents are 5X what they were in Ogaki” (that happened, more later” or “Bring on help” (that also happened, more later).

Here’s the graph since we launched in December 2010. This is an MRR graph, so when we get annual pre-payments (we offer 12 months of service for 11 months of cash up front), they’re pro-rated over the next 12 months on this graph. However, since we’re a cash-basis business, the revenue number for the year will be higher.

Appointment Reminder MRR graph

MRR as of December 2014: ~$6,500 (up roughly 35% YOY)

Revenue for 2014 (on the publicly available plans): ~$75,000

Both of these numbers are approximations — we have credit card charges which will happen between now and December 31st, and they’re fairly predictable.

Brief fun digression on how much I love the SaaS model: our oldest customer signed up in January 2011, weeks after we launched publicly. He has since paid us $1,334, $29 at a time.

Our presently available plans are on the pricing page. They’re primarily segmented by appointment count. This is a decent if imperfect proxy for value a customer gets from Appointment Reminder. It is imperfect because an HVAC company which saves an appointment can earn $2,000+ in revenue. A hair salon, on the other hand, only values an appointment in the tens of dollars.

Once upon a time we had a $9 Personal plan available, but the customers who signed up for it were pathological in every sense of the word — some cost hundreds of dollars in imputed support costs then demanded that we send out $5 rebate checks for “unused” appointments. We killed that option a few years ago — I should never have offered it, but I thought I’d be able to pitch it to productivity bloggers at Lifehacker and the like and thereby gain exposure to their audience which might include office managers at professional firms. That never materialized.

Our target customer for the publicly available plans is the office manager who would otherwise be doing the calls herself. I call her Office Manager Milly. She’s comfortable with basic computer usage but not a technologist, earns about $20k to $50k a year, is generally an employee rather than owner of the business, is usually in her thirties or forties, and she has independent authority to make the purchasing decision. She works for a professional services firm, like a legal firm, an accounting practice, an HVAC installation firm, a plumbing firm, yadda yadda.

This is distinct from a personal service firm, like say a massage therapy practice. I thought personal service firms would be a big chunk of our book of business, but they aren’t: they’re smaller, they often are operated in non-businesslike fashions (many hair salons have no one accountable for making sure people actually come to appointments, but Milly is accountable to the business owner — this is one of her core tasks), and their per-appointment value is so low that losing an appointment doesn’t really constitute a hair-on-fire problem for them. It is a hair-on-fire problem for Milly: she can often be in trouble with the boss if she forgets to send a reminder even if the client makes the appointment anyhow, because potentially hundreds or thousands of dollars of revenue was just put at risk.

Anyhow, our publicly available plans range from Professional ($29, 100 appointments) to Small Business ($79, 300 appointments) to Office ($199, 1,000 appointments). This will likely change early next year — I’m taking my own advice to charge more, and re-aligning those numbers with actual customer behavior rather than the numbers I guessed four years ago.

I picked the original appointment quotas half based on projections and half based on total guesswork. My rough, untutored impression was that a personal service provider (stylist, massage therapist, etc) might average five appointments a day. I then linearly extrapolated how many workers would provide services at various target firm sizes and then fudged the resulting numbers a bit to provide a volume discount and to have round numbers. Translation: our pricing is, basically, made up.

Most of our customers are on the Professional plan, which annoys the heck out of me, but it’s my fault. Since I was thinking personal services, where 100 appointments a month barely sustains a sole practitioner (it implies $3k to $8k gross revenue), I thought any sizable business would be forced to pay more meaningful amounts of money. It turns out that you can run a nice boutique law office with sales in the high seven figures or an architectural consultancy with millions in revenue on less than 100 appointments a month. Believe me, I know several examples.


Paying account breakdown: 142 (as of today)
Professional: 106
Small Business: 31
Office: 5

Churn rates: Churn is the percent of accounts which paid us money in month N who did not pay us money in month N+1. As we’re on a card-upfront free trial model, this definition structurally excludes someone who fails to continue a free trial as a churned account. That is in keeping with the business purpose of the metric: churn measures accounts which we once serviced satisfactorily which at some point were not willing to continue purchasing services, where trial conversion rates measure our effectiveness at convincing people that AR is right for their business.

We track it on a per-plan basis, since our customer behavior is wildly different. As a benchmark for you, for B2B SaaS sold on a month-to-month basis on a low-touch model, 5% is on the high side, 3% is what you should shoot for, and 2% is best-in-class.

I rather suspect that our churn rates have gone down over time as our product has improved, but that is difficult to tell, partly because we churn rates are very volatile at the (small) number of accounts we have. Our churn rates since inception are:

Professional: ~7%
Small Business: ~6.9%
Office: ~3.8% <— very small sample, so this number in particular is noisy as heck

I’m mildly dissatisfied with these churn rates, particularly in the months where churn eats the MRR increase from new converted trials (as happens occasionally), but they are what they are. In a more mature business than AR, I’d spend a lot more resources measuring this and working on the causes of it, but AR’s bigger problem is not closing enough new accounts every month for churn to really matter yet, so I prioritized working on sales recently rather than working on churn directly.

We track reasons that people churn. I separate that feedback into reasons we could have addressed and those which were beyond our help. “You didn’t respond to my emails” or “I needed the software to do X” plug back directly into our decisionmaking. Unfortunately, the bulk of our cancellations are for reasons like “We migrated to a solution which includes your product as a feature” or “We never really convinced the whole team to use it.” Historically failure-to-adopt has been something that we only can weakly influence, but we’re actively working on that with concierge onboarding/customer success as one facet of the sales initiatives.

Knowing churn lets you calculate the lifetime value of customer accounts. (LTV = Monthly price / monthly churn rate.) They are, assuming you trust our churn numbers as roughly representative:

Professional: ~$400
Small Business: ~$1,000
Office: ~$5,000

Knowing these numbers let us budget for paid acquisition (though we don’t do any significant amount of it — I’ve experimented halfheartedly but haven’t found a channel that works yet) and prioritize which accounts get proactive outreach from our sales rep and how much attention (and, ahem, commissions) we can afford.

Free trial acquisition: Our primary channel is organic Google search, by a factor of lots. This is partially driven by us ranking well for generic terms ([appointment reminder], [appointment reminder software], etc), very modestly by the sort of scalable content creation that put BCC on the map (I was unhappy with the content quality of our initial experiment into this for AR and opted not to scale until I fixed the process causing the quality issue… and then simply ignored that topic for the next 3 years), and referrals from clients.

Many people who run professional services firms are themselves consumers of professional services firms. If a lawyer gets an automatic SMS from their accountant, they might ask the accountant “That’s amazing. I wish I had that.” The accountant tells them “Ask the office manager what we use for it.” She says: “It’s really easy to remember: Appointment Reminder.” They Google it, bam.

It wasn’t the world’s most inspired naming choice, but the $8.95 I paid for the domain name is probably the best ROI of anything I’ve ever bought. (AR is virtually guaranteed to be a mortal lock on the query [appointment reminder] due to the combination of the exact match domain bonus and the fact that most links to it naturally cite the name of the company. The fact that it is a .org is irrelevant, except in that the .org was $8.95 and the company which owned the .com wanted $30,000 for it.)

So given our very, very lackadaisical marketing, we have very low traffic and a fairly modest number of free trials per month:

Unique visitors (2014): 66k, of which many are existing customers. Yep, this whole business has less traffic than my most popular blog post from 2011. I know, the irony, right?

Conversion to free trial: Due to the high percentage of people visiting our homepage who are existing customers, I feel like the most representative view of the funnel I can give you is this one (pulled straight out of Kissmetrics):

Funnel numbers

Free trials (card up front) (2014): ~350
Trial conversion rate: ~42% <— Would you believe I actually had no screen to track this number, simply because calculating it was not straightforward due to technical debt? Yep, I know, embarrassing.

You’re thinking “42% conversion rate! That is very high!” For low-touch B2B SaaS which you can sign up to without a credit card, it is generally closer to the 2~5% region. We require a credit card, both to prevent abuse of the system (ask me about the time AR was used to violate a temporary restraining order by, essentially, proxying harassing phone calls) and because forcing people to take their free trial seriously means that the increase in trial conversion rate is greater than the number of accounts we lose due to requiring the card.

Against comparable companies with cards required upfront, I feel like 42% is a happy place to be, but not outstanding yet. Hopefully it will increase by having more human outreach available. Additionally, we lost a lot of trials due to non-responsiveness to CS inquiries earlier in the year when I was sick. As one customer accurately opined in an irate email: “How can I trust you with my business long-term if you won’t answer an easy question on day 1?” (I apologized. Nothing else to do, really: under the health circumstances I wasn’t going to further stress myself over $29 a month.)

Since someone always asks me: no, no, no, we don’t hope that Office Manager Milly signs up for the trial and then forgets about us. We send at least ~5 emails during the free trial and, now that we have a sales rep, have an actual person manually reaching out, too. If someone’s account looks abandoned, I eventually get in touch with them and then terminate it proactively if I don’t hear back. I have gotten more complaints about that policy than you’d think: “Hey, why’d you close that account?!” “Because you hadn’t logged in in several months, had no client data in the system, and did not respond to several attempts to get in touch. I didn’t feel right taking your money.” “We were getting around to it! Open it back up again!”

Enterprise deals: I still can’t tell you numbers with a high degree of resolution here. We cashed only a handful of invoices this year, from AR and from one quick consulting gig, and any level of granularity gets legally dicey. Let me ballpark it as $50k < X < $100k.

As to how enterprise sales works at Appointment Reminder? In lieu of repeating myself, I covered it the other day and nothing changed over the weekend, so check out that writeup.

Vanity metrics: Our non-enterprise customers had 141k appointments (double last year) and sent over 200k reminders, saving the equivalent of about four full-time people doing nothing but dialing phones and leaving voice mails under sweatshop conditions. We saved our customers several million dollars in aggregate that would have been lost due to missed appointments.

My favorite story on this, from the principal of a contracting firm, is that his daughter is going to Harvard on the Appointment Reminder scholarship (when you lose $2k on a missed appointment it adds up quickly). We’ve also got a small pile of notes from folks in all walks of life thanking us for getting them to their doctor appointment, chat with the immigration lawyer, cable company, and he like on time.

Appointment Reminder: What Went Right

I did some sustained A/B testing on AR for the first time earlier this year. The full writeup would make an already long post even longer, but suffice it to say that we rejiggered the signup process and had a ~50% increase in the number of free trials as a result. I’ll talk more about that test in the future.

I also did some long-delayed work on automated customer onboarding and first-run experience earlier in the year, with the upshot that people are more likely to succeed in getting up-and-running and hence more likely to convert. Probably. I don’t have really great historical numbers there, but sentiment from customers improved and I got less “This is confusing. How do I…?” inquiries, anecdotally.

I have, finally, started working with other people. Appointment Reminder presently has two people working on it part-time, though this is only for the last few weeks so we don’t have great results to show yet. My virtual assistant who has been doing CS for Bingo Card Creator for several years is getting spun up as tier-one CS for AR, and I have a commission-based salesperson doing a combination of outreach to inbound leads and customer success style management of people’s trials.

I built software to support sales — easily the most fun thing I’ve done in AR since the original buildout of the core functionality — which helps the above-mentioned folks do their jobs. I recently wrote about that in considerable detail.

To build out that software and get the team spun up, I had to actually sit down and document our business processes, which was a great opportunity to force myself to actually think through them. The discipline of doing this made me confront a bunch of very obviously not optimal decisions I had made, like the lack of anything even approaching a repeatable sales process for enterprise details, so now we have one. Hopefully we can execute on it a bit in 2015.

Appointment Reminder had rock solid reliability in 2014. We had no systemwide outages, for the first time ever.

We did have an exacerbation of the “server occasionally times out during a phone call” problem that I mentioned last year was a real stitch in my craw. Errored-out calls increased from “a handful a week” to “a dozen a day”, which lit a fire under me to finally spend time investigating and fix things.

It turned out to be a combination of poor configuration choices for the MySQL server and the lack of an index for a particular query caused by our auditing system, which wasn’t a problem back when there were only thousands of audit records, but wasn’t quite so ignorable when it got to millions of audit records.

That issue is now fixed. I haven’t succeeded in completely eradicating timeouts yet. We dropped three calls in December. That is three too many.

Appointment Reminder: What Didn’t Go Right

Health management / burnout: Virtually all of the good news for AR this year happened between July and December. I accomplished very little which meaningfully drove the business forward in the first half of the year. This was due to a complex cocktail of stress, illness, and competition with the conversion optimization course for cycles (about which, more later).

This manifests itself, most commonly, as me fleeing from forward progress on the business and, on particularly bad days, avoiding my inbox. Avoiding my inbox never helps on stress levels, because I think that if I open it there will be more stress… and then it is suddenly Thursday and I haven’t checked emails since Monday. This caused me to deliver suboptimal customer service at a few points in the year — much better than last year, but still not where I’d like it to be.

I waited way too long to bring people on. Hat tip to Jason Winder, a good friend and fellow CEO, who finally managed to smack some sense into me. Here’s hoping that I can retain the fundamental character of the business while now having folks who I’m responsible to.

Due to having no process for closing enterprise leads and not enough brain cycles free to do the wildcat shoot-from-the-hip stuff which got us our existing enterprise customer base, our enterprise pipeline dried up early in the year and, as a consequence, we closed very little additional enterprise business.

Bingo Card Creator

Bingo Card Creator is a B2C SaaS application which is sold on a one-off transactional model (hint: don’t do this), primarily to elementary school teachers.

BCC was very firmly in maintenance mode. Substantially all customer support for it is done by my virtual assistant. I only get involved for refunds or harder issues where we have to do hand-fixes in the Rails console.

I worked less than five hours on Bingo Card Creator this year. It runs itself… not particularly well, but it runs.

BCC traffic, which is quite seasonal, peaked in 2012/early 2013 and has been on a steady decline since then. This decline accelerated in 2014. I assume it is due to some combination of the content on the website being stale (nothing has materially changed since, oh, 2011 or so?) and perhaps changes to Google’s algorithms, which I don’t actively keep abreast of these days.

Additionally, we’ve historically gotten a large portion of our traffic from AdWords. That also hasn’t been touched since 2011 other than buying Larry and Sergey a few more drink coasters made out of materials that can only be created in the Large Hadron Supercollider. AdWords is identifying less opportunities for our ads to be run profitably, hence showing them less, hence sending us less traffic (though costing us less in absolute numbers, so that partially balances).

I have done nothing to address the decline, as it is economically irrational to try to recapture those halcyon days of ~$5k a month sales when for the same amount of work I can hand off major portions of AR’s workload, cut my stress massively, and almost certainly make more money in the long-run.

BCC Stats

Sales: 1,128 (down 35% from last year’s 1,734)

Refunds: 15 (down from last year’s 57)

Sales Net of Refunds: $33,319.35 (down 33% from last year’s 50,156.16)

Profits: ~$20,000 (estimate based on best guess of allocation of costs to BCC which is consistent with last year’s number, down from ~$23,000 last year)

Wage Per Hour: ~$4,000 — I was checked out of BCC this year

BCC Web Stats

Visits: 500k (down 35% from last year’s 770k)

Unique visitors: 430k (down 32% from 630k last year)

Page views: 1.5M (down from 2.4M)

Traffic sources of note: Google (51%), AdWords (16%), direct (11%), BingHoo (11%)

Trial signups: 41,000 (down from 61,000)

Approximate trial to purchase conversion rate: 2.8% (up modestly from 2.6%)

BCC: What Went Right

There’s a lot to be said for having a business which runs totally on auto-pilot.

BCC: What Went Wrong

Traffic has continued to fall off a cliff, as mentioned previously.

I had collaboration issues with a consultant, which didn’t meaningfully impact business results (the trajectory on BCC is pretty clear), but which caused him unnecessary stress, which I regret. I had engaged Nick Disabato’s Draft Revise service to do A/B testing for BCC. My expectation with working with Nick was that I’d basically write checks and never have to think about the matter again. Nick wanted a more collaborative process with me, which I was not particularly unwilling to provide but which I did not prioritize actually making happen. As a consequence, I missed emails from him for months at a time, which inhibited his ability to do his work.

How disengaged was I from Bingo Card Creator? Nick and I got together for lunch last week and one of the things I wanted to tell him was that I wanted to dissolve the relationship amicably as I didn’t have time/effort/desire to manage it and he had previously expressed concerns about that. Nick brought up that he had actually stopped charging me several months ago and stopped work at the same time, and that I would have known this if I read the emails he had sent me. Whoops.

I’m glad he’s now freed up to work with better clients than me. They can actively engage with him on products which have a bright future ahead of them. That way, the insights he pulls out of the A/B testing process can actually get looped back into the product/marketing on a timely basis.

So if that was my level of responsiveness for someone working with me, how responsive do you think I was to customers? Yep, luckily my VA takes care of 95%+ of issues by herself, but the remaining 5% which got escalated to me (refund requests, etc) were often delayed by, literally, weeks at a time. I regret this — when I noticed it I apologized to customers, but that level of responsiveness makes me feel like BCC has become a business which is other than the kind I want to operate. I’m going to fix it or get it into the hands of someone who can.


I had quit consulting last year, but had the combination of a wonderful opportunity fall into my lap at the same time as a move to Tokyo was in works. I can’t say anything about the engagement. Basically, same old same old, I built systems that helped make a software company a wee bit more efficient at getting a great product into the hands of more people.

The Move To Tokyo

So in lieu of that, let’s talk Tokyo. My wife and I moved from Ogaki, the small town in Japan where I had lived for 10 years, to Tokyo, as of a few months ago. This quintupled our rent on an ongoing basis and required substantial expenses to set up our new household, hence motivating me to do a quick consulting engagement.

Why move? The long and short of it: Ruriko (my wife) does not love Ogaki the way I love Ogaki, and wanted a change. That’s a good enough reason to do anything, but as it happened I was also feeling socially isolated in Ogaki (aside from Keith, I have no friends within the city anymore, and few enough close acquaintances), and I was ready to start a new chapter, too.

Why move to Tokyo? I wasn’t ready to uproot the family to the US, so looking around Japan, Tokyo is pretty clearly the best option for us. Ruriko has friends and family there (not immediate family, but I expect that to change), the QOL according to things she cares about is quite high there, and Tokyo has a rather substantial tech community.

I thought Tokyo was going to be… tolerable. I’m rapidly falling in love with it, or at least the little slice of it I’ve found in Nakameguro (our neighborhood). Jason Winder, a close friend of mine, runs a company in that neighborhood. We met when he started doing HN meetups several years ago — I used to go out 3 hours on the train once a month just to talk software with anyone — and, as we became friends and I ended up spending a bit more time in the neighborhood, came to think it represented a nice compromise of the advantages of big city living but with a friendly atmosphere.

I remember thinking years ago that Tokyo was just wall-to-wall overcrowded alienating hell, like the subways. The subways are indeed wall-to-wall overcrowded alienating hell, at least during rush-hour. I have solved this by making sure to sleep in until rush-hour is over and joined a coworking space within walking distance, so that I’m only on the subway twice a week rather than twice a day.

It has been great businesswise — in particular, Americans from Silicon Valley invariably hit Tokyo when they come to Japan but don’t come to Ogaki, so just by being there I get to meet with a lot of interesting folks who I would otherwise have to fly over the ocean to see. (I also hope to get a bit involved with the Japanese startup community, but have been rather busy lately.)

Productized Consulting

In addition to the blog (you’re on it), podcast with my co-host Keith Perhac, and occasional essay delivered via email, I have also made a few paid products which help software companies market and sell their software.

They include a book on conversion optimization, a video course on lifecycle email marketing, and occasional one-off training events on email, copywriting, productizing one’s own consulting services, and the like. I didn’t actually do any paid training events in 2014.

I have been working for the past year on a video course to teach software companies how to do conversion rate optimization for their websites and products. It has been, frankly, a long, slow nightmare of a project.

I originally opened pre-orders for it last year in December and anticipated delivering it by the end of January. That didn’t happen. The slip date shipped repeatedly [editor’s note: that spelling mistake was unintentional but I like it so much I’m leaving it in].

I accomplished partial delivery of the course in summer — it had been sold on a three tier model, with the first tier having access only to canned videos, the second additionally getting personalized advice about their websites in a group discussion format, and the third getting a mini-consultation privately. To apologize for the delay, I bumped everyone to the tier one higher than they had paid for, and then delivered the non-canned portion. This required a few weeks of scrambling to do mini-consults and give a few dozen companies conversion advice, but it was the right thing to do, and it kept most of the customers happy.

I suppose it is worth saying why delivering that was comparatively easy and delivering the course has been comparatively hard. Partially, expectations in terms of production quality for live events are much lower than they are for video, and I wanted this course to be produced much better than the last one I did (which was me talking into a webcam). The other factor is that there is a difference between doing and teaching, and (contrary to the old saw) teaching is actually sometimes harder than doing.

I have substantial experience with doing conversion optimization work and, shown a company’s (often-unoptimized) home page, trial signup funnel, and pricing page, the ideas flow really freely. That’s easy for me and I’m very, very good at it. What I am not an expert at is making other people good at it, too. Curriculum development, collateral (slides & etc), and actually shooting the course has taken far, far more time than I anticipated, and I’ve thrown out quite a bit of work because, after seeing the final product, it didn’t hit my internal quality bar. (Which is set rather high for this, partly in reaction to it being late and partly because I am “the A/B testing guy” and I want to make this some of the best work in my career thus far.)

Every time I get in touch with folks about this, I offer refunds for those folks who (quite reasonably) feel inconvenienced by continuing to wait on the course. Even so, I feel awful about it having taken this long. That has been one of my main stressors this year.

This has totally stalled my product development pipeline, as it were, for productized consulting. I had expected last year to do a few one-off training events again, because people were thrilled with them last year and the model very clearly works. I wouldn’t feel right breaking time off the course’s schedule to build, launch, and deliver those training activities, though, so nothing on that front has happened this year.

Productized Consulting Stats

Sales (Hacking Lifecycle Emails): $9,443

Pre-sales (Software Conversion Optimization): $2,988

Refunded pre-sales: -$1,588

Royalties (Sell More Software): $469

Gross Sales Net of Refunds: $11,312

Productized Consulting: What Went Right

I did a wee bit of tweakage to my email list such that folks get my favorite 5ish essays delivered in a sensible sequence after they sign up for it. This lets me get some mostly passive sales for Hacking Lifecycle Emails, as one of the essays has a brief plug for it. (Yep, prior to 2014 there was no lifecycle email to sell my course on lifecycle email. The cobbler’s children have no shoes.)

Keith and I found a good podcast editor, which makes getting new episodes up less of an ordeal than it was previously. Hopefully after the situations in our personal lives get a little more stable, we’ll do even more of them. I think, off the top of my head, that we recorded more in 2014 than any year previous. (Some are still in the can.)

Productized Consulting: What Went Wrong

I won’t belabor the point, but not delivering the conversion optimization course this year is, far and away, my biggest regret for the year.

As a consequence of not getting that course launch and having it block other things in the pipeline, productized consulting fell fall short of my financial goals for that segment of the business.

I have not written as much as I’d have liked to this year.

Some days I feel like I have a few mana pools of creative energy — one for programming, one for writing/speaking/podcasting, one for generic business stuff, etc. As a consequence of stress and this course hanging around my neck, the mana pool for writing has been bone dry for most of this year. On those few occasions where I’ve had enough left over to cast the spell of 10,000 Word Blog Post, I’ve found myself with nothing new to write about.

This is irrational, and I know it is irrational. What I should do is continue to write on topics that I’ve covered or work on in the past, because that provides incremental value to folks who like reading my work. However, I always use the burst of inspiration which covers conquering fun new challenges to fuel writing about those challenges, so when I have no fun new challenges being conquered, my writing cadence falls off a freaking cliff.

I’ve only published three essays to my newsletter this year, where I try to do it bi-weekly and I’m embarrassed when it is less than monthly. I don’t really love blog posts anymore as an expressive medium, and only do it when I have something which should go to a wider audience than my usual mailing list (like, say, a 10k word brain dump of everything I know about doing business in Japan).

Business Administration

I changed the way we do bookkeeping this year. Previously, I used homegrown bookkeeping software built into the Bingo Card Creator website to do my books. I then had my VA do them, by periodically dumping hundreds of pages of transactions from my credit card statements, emailing them over, and having her categorize them for me.

This was a poor use of her time. It is more important that customers get useful, friendly responses in a timely fashion than the bookkeeping get done on any sort of schedule. My VA did not particularly enjoy the bookkeeping work, which requires a certain level of obsessive-compulsive attention to detail (one reason why I am bad at it), and the process routinely produced errors. Occasionally, they were errors which — had I not caught them — would have materially affected our tax liability. If you make a one-digit error on the date in a single-entry bookkeeping system and file a $5,000 hotel stay in the wrong year, that costs me thousands in additional taxes.

As a result of this, collaborating with my VA on bookkeeping consumed an excessive amount of time relative to the value we were getting out of it. I was weakly attached to the notion of continuing automated expense transparency on the BCC website, but not at the expense of additional stress this year. So I looked for a better solution.

Enter Bench. They’re basically Bookkeeping as a Service, where an actual, honest-to-God human bookkeeper manages your books for you using algorithms as a lever rather than as a substitution for their work and expertise. (I previously used a few solutions which are designed to automate the process entirely, but they produced books which required so much manual correcting on my part that they were worse than having no books at all.)

Bench is structurally a simple app. They use Yodlee (presumably) to slurp transactions out of my bank accounts. I upload statements for those accounts where that is impossible, and receipts for those transactions which need them for substantiation purposes (receipts for hotel stays for business travel, etc). Graydon the bookkeeper does the books for me. The site has an interface for passing messages to them and annotating individual transactions with a note, like “That transfer to a German tech company wasn’t a distribution to an owner of the LLC — easy mistake to make since all my wires to Japan are — it was actually for a software license.” Graydon then goes in and recategorized as required.

It runs $125 a month at my scale and is worth every penny and then some. I’m thrilled.

Expenses (total): $74,913.90 (plus any transactions which are “in flight” and not entered yet, so probably $80k by the end of the year)

(I’d be remiss if I didn’t mention that, while that number is comparable to the one I report every year, it doesn’t include our expenses paid in Japan. Those are mostly immaterial with regards to clear business expenses — a ream of copy paper, yay — but Japanese tax law also lets me expense a substantial fraction of e.g. our rent and utilities, so we do.)

Major highlights:

People (accountant, VA, sales rep): ~$20k
SaaS: ~$15k
Hosting & Domains: ~$10k
Advertising: ~$7k
Twilio: ~$6.5k <— Largest single vendor after finally surpassing Google, which makes me happy, as I’m their biggest fan
RailsLTS: ~$5.5k
Credit card processing (Stripe/Paypal/etc): $4k
Macbook Pro: $3k

I finally joined the vast Mac conspiracy. Darn all y’all for being right. It is a much, much more productive environment than my old Dell, even considering the amount of time I’ve spent banging my head on the learning curve.

Goals for 2015

Bingo Card Creator

  • Retool CS processes such that customers get responses to their questions in a timely fashion, even when they have questions which (at present) require tier-two support.
  • Either continue presiding over BCC’s long twilight or sell it to someone who can give it the level of attention it needs.

Appointment Reminder

  • I’m shooting for $15k MRR on the publicly available plans. That’s slightly more than double what we’re doing right now, but I think it is doable with the assistance of my sales rep plus me actively working on marketing for a change.
  • We’ve historically sold HIPAA-compliant services on a very limited basis. I’m hoping to release HIPAA-compliant services sold on a medium-touch model, now that I have a very good understanding of the process of providing that in a compliant fashion and (knock on wood) actually closing the deals. This will probably be in the $500 to $1,000 a month range. If we could hit, oh, $15k MRR on those accounts as well, that would be nice, but that might be a bit aggressive.
  • The enterprise pipeline is, at the moment, close to bone dry. I should restock that, hopefully with our sales rep taking a bit of the workload. I have no clue how to forecast from “bone dry”, so let’s pick “break six figures” as a nice round number to shoot for.
  • I’d still love to earn that “Cashed a six figure check” merit badge but I’m not wedded to it.
  • Do one or two ambitious development projects on the product side of the house, if I have the time.

Productized Consulting

  • Ship the Software Conversion Optimization course as early as possible given my non-business commitments.
  • I’m hoping that, with the course out, I can do some more experiments in form factor this year. For example, I’ve been really wanting to write a book (from scratch this time) for a while, and maybe this is the year.
  • Numerically, I’m shooting for ~$80k in sales when I launch the course, and then hope to do about $150k in this segment for the year.

Big News

I have one potentially big project in the offing, but it isn’t public knowledge yet. Suffice it to say it could materially impact the above by a lot if it comes to pass. More about that in the usual places as it gets closer to happening.

I kept the best news for the end. Ruriko and I were blessed by the birth of our daughter, Lillian. I love her to pieces. I’m still working on the balancing act of being a husband and father while also running the business, but I’ve got the best co-founder in the world to figure out the challenges with.

Doing Business In Japan

(For readers for whom Japanese is easier than English / 日本語が読みやすい方:上杉周作さんが本投稿を日本語に翻訳してくださいました。ビジネス・イン・ジャパンをご参照ください。)

I’ve been in Japan for ten years now and often get asked about how business works here, sometimes by folks in the industry wondering about the Japanese startup culture, sometimes by folks wishing to sell their software in Japan, and sometimes by folks who are just curious. Keith and I have discussed this on the podcast before, but I thought I’d write a bit about my take on it.

Disclaimer: Some of this is going to be colored by my own experiences.

The brief version: white male American (which occasionally matters — see below), came to Japan right out of college in 2004. I have spent my entire professional life here. I’ve worked in two traditionally-managed Japanese organizations (one governmental body and one megacorp), run my own business full-time since 2010, and have modest professional experience with Japanese startups (both run by Japanese folks and by foreigners).

I’m fluent in Japanese to all practical purposes.

Disclaimer the second: I’m going to attempt to avoid essentializing Japan too much, as (like the US) it is a big country with a broad range of human experience in it. Essentialization is a persistent problem with most writing about foreign cultures. The best antidote for it ever with regards to Japan is an out-of-print book Making Common Sense of Japan.

That said, there may be some generalization and/or exaggeration for dramatic effect. Mea maxima culpa.

The Company Is Father. The Company Is Mother.

The slice of contemporary Japanese life of keenest interest to you is dominated by one particular relationship: that of the Japanese salaryman to his employer. If you understand this relationship, it is almost a Rosetta stone. You’ll immediately be able to predict true things about the world like “Japanese startups probably have huge difficulties in hiring.” (About which, more later.)

A salaryman (transliterated from the Japanese which is itself borrowed from English), more formally a “full-time company employee” (正社員), is the local equivalent of a W-2 employee in America. This is roughly 1/3rd of the labor force in Japan, but it has outsized societal impact.

Traditionally, salarymen (and they are, by the way, mostly men) are hired into a particular company late in university and stay at that company or its affiliates until they retire.

There are other workers at Japanese companies — contract employees, who can be (and are) let go at will, or young ladies on the “pink collar” track who are encouraged tacitly or explicitly to quit to get married or raise children — but the salaryman/employer relationship is the beating heart of the high-productivity Japanese private sector. (The Japanese economy is roughly 1/3rd the public sector, 1/3rd low-productivity firms like restaurants or traditional craftsmen, and 1/3rd high-productivity household-name megacorps. Salarymen are mostly present in the last one, which happens to dovetail with your professional interests.)

The salaryman/employer relationship is best characterized as “You swear yourself to us, body and soul, and in return we will isolate you from all risks.”

The employee hereby promises the company: Your first obligation, in all things, will be to your company. You will work incredibly hard (90+ hour weeks barely even occasion comment) on their behalf. The company can ask you to head to a foreign office for three years without your wife and child beginning tomorrow, and you will be expected to say “Sure thing, when does my flight leave?” or accept that your career advancement is functionally over.

The company will mold you to their exacting specifications to do whatever form of service they require. You will happily comply, in this as in all things. For example, if your company needs a Java-speaking systems engineer and you have a degree in Art History, this is not a problem because you can be fixed. Sure it might take ten years and only work on a quarter of the new hires but that’s why we employ you for 45 years and hire a hundred at once! (What of the Art History majors who don’t successfully learn how to edit XML files or architect web applications? Well, they’ll be promoted in lockstep with the rest of their cohort, but tasks which actually require programming with magically route around them, and they’ll end up doing things like leading 6 hour planning meetings and producing spreadsheets. Lots and lots of spreadsheets.)

The company hereby promises the employee: Your company will provide structure and purpose for your life. You will be clothed in the company colors, literally and figuratively. You will be respected, inside and outside the company, as befits an employee of ours. You will be provided with benefits perfectly calibrated to allow you and your family to lead a middle-class Japanese life. Your children will go to as good schools as they test into. Your wife will be able to afford an annual trip to Hawaii with her girlfriends.

You probably won’t attend that trip because, as a salaryman, you wouldn’t want to leave your coworkers in the lurch by taking extended vacations. Your company officially allows you between 12 and 18 combined vacation/sick days a year, but salarymen generally try to hold themselves to about 5, taken in single-day increments. Your company loves you and wants you to be happy, though, so they’ll suggest two days for your honeymoon, two if a parent passes away, and one if your wife passes away. You can take that Saturday off, too, because the company is generous. There, that’s like four full days — five, if you time it with a public holiday.

There exist companies which don’t require their salarymen to work Saturdays. That is considered almost decadent for salarymen — the more typical schedules are either “2 Saturdays a month off” or “every Sunday off!” Even if you’re not required to work Saturdays, if one’s projects or the company’s situation requires you to work Saturdays, you work Saturdays. See also, Sundays.

Salarymen work large amounts of overtime, although much of it is for appearance’s sake rather than because it actually accomplishes more productive work. Depending on one’s company, this overtime may be compensated or “service overtime” — “service” in Japanese means “thrown in for free in the hopes of gaining one’s further custom”, so your favorite restaurant might throw in a “service” desert once in a while or you might do 8 hours of “service” overtime six nights a week for 15 years.

At those companies which actually pay for overtime (not uncommon, even for professional salaried employees, even for those who would characteristically be exempt in the US), there are generally multiple rates. I got time and a quarter between 6:30 and 9:30 AM, time and a half until midnight, and time and three quarters after 1:00 AM. That last bracket was there for a reason.

It is highly unlikely that anyone will ever tell you “We need you here until 3 AM. Yeah, sorry, tell you what, take off early at 9 PM tomorrow.” The company is just steeped in an environment which will make this decision seem like the most natural thing in the world to you. To leave early would let your team down. To make a habit of it would cause people to question your commitment to the company and to the important work that the company does. It will become so natural to work salaryman hours that you’ll teach their necessity to junior employees who you mentor, probably without you even realizing you’re doing it.

Don’t have a wife? You might quite reasonably think “I don’t have time to even think about that.” Don’t worry — the company will fix your social calendar for you. It is socially mandatory that your boss, in fulfillment of his duties to you, sees that you are set up with a young lady appropriate to your station. He is likely to attempt to do this first by matching you with a young lady in your office. There are, at all times, a number of unattached young ladies in your office. Most of them choose to quit right about when they get married or have children.

You might imagine that you heard a supervisor tell a young lady in the office “Hey, you’re 30 and aging out of the marriage market, plus I hear you’re dating someone who is not one of my employees, so you might want to think about moving on soon.”, but that would be radioactively illegal, since Japanese employment discrimination laws are approximately equivalent to those in the US. A first-rate Japanese company would certainly never do anything illegal, and a proper Japanese salaryman would never bring his company into disrepute by saying obviously untrue things like the company is systematically engaged in illegal practices. So your ears must be deceiving you. Pesky ears.

The company is your public life. Have an issue with your landlord? The company will handle it, in those cases where the company is not your landlord. (“So let me get this straight: we’re going to pay our employees, and then they’re going to immediately hand 25% of their salary over to an apartment? Doesn’t this suggest an obvious inefficiency? We could just buy a building and house dozens of employees there — lower transaction costs plus economies of scale.” Many Japanese companies have done this math already, and company dorms are quite common, particularly for young, single employees.)

Need to file paperwork with City Hall? Someone from HR can do it for you. Salarymen don’t file tax returns — the National Tax Agency and HR work out 100% of the paperwork on their behalves. Insurance? Handled. Pension? You’re sorted. Immigration, for those very rare salarymen who are also foreigners? Your CEO has written a letter to the Minister of Justice for inclusion with the paperwork that HR has put together, and you won’t even have to carry it into the office.

The company is your private life. All friends you’ve made since your school days almost by definition work for your company, because you spend substantially every waking hour officially at work or at quote leisure unquote with people from work. When you get off work rather early, like 7:30 PM, you’ll be strongly encouraged to go out to dinner and/or drinks with bosses, coworkers, and/or business acquaintances. (The company is buying, either directly via an expense account or indirectly via a “The most senior person pays and their salary has been precisely calibrated to accommodate this” cultural norm.) Like karaoke and golf? Wonderful, you’ll have an excellent time with the other salarymen, who have either perfected the skill of liking karaoke and golf or seeming to like karaoke and golf when invited out by colleagues.

We’ve mentioned that your company considers it its responsibility to see you appropriately married. That is not the sole way in which the company may try to arrange companionship, but let’s table that issue for the moment. When you get married, your boss will give the longest speech at your wedding, praising your diligence on that last project and bright future with the firm. Perhaps eight or so coworkers will show up. They’ll also take up a collection for you if a parent should pass away, come visit if you’re hospitalized, and offer to intercede if you should have trouble with your wife or children. You are, after all, one of the family.

Lifetime employment is somewhat on the outs in the last 20 years or so, but it is still a reasonably achievable thing in 2014, and an expectation that many Japanese folks quite literally structure their entire lives around. An offer of employment as a salaryman, while theoretically instantiated as a e.g. three year employment contract with “renewal upon mutual agreement”, is (practically speaking) a promise that one will be promoted on a defined schedule for one’s entire working career.

One’s actual salary as a salaryman is generally rather low — about $100 per year of age per month, as an engineer in Nagoya (set by a particular monopsonistic engineering employer near Nagoya). In Tokyo, my sense of the market is that, as an intermediate engineer in his early thirties, I’d probably command somewhere between $30k and $60k. (In Silicon Valley, the going rate would be somewhere between $120k and $160k and increasing rapidly.)

The stability is superior to even tenured professors or civil servants in the United States, though. Eliminating your position will result in, at worst, your transfer into a division optimized to shame you into quitting. Incompetence at one’s job bordering on criminal typically results in one’s next promotion being to a division which can’t impact shipping schedules and has few sharp objects lying around.

You owe your company one more thing: Don’t. Ever. Quit. Salarymen are very rarely hired mid-career — you start at a company directly after undergrad and stay there forever. If you somehow manage to separate from that company, you are damaged goods. You will, in all probability, never be offered a salaryman position again. You may be offered professional work as a contract employee, but this has worse material terms, second degree social status, and no job security.

You may think I’m exaggerating. Not so much. I spent about three years in the salt mines and could go on this topic for hours. You can also read about this, to exhaustion, in most books about modern Japanese culture. (Single favorite recommendation for foreigners: An Introduction To Japanese Society, Sugimoto. Salarymen rate only a chapter or two — the book is sweeping in breadth and does the best job I’ve ever seen at adequately representing the diversity of life here for a foreign audience.)

Salaryman loyalty compels me to mention that my company was scrupulously fair to me, in a fashion which is not automatic among Japanese megacorps with regards to their foreign employees. I am sincerely indebted to them for that.

Startups In Japan Are Considered Off-The-Charts Risky

As a young professional, you’re defined by your relationship with your employer, and everyone else expects to interface with your employer to do business with you. If your employer is yourself, or a company no one has heard of, this has numerous negative impacts on your life as compared to your employer being a member of the elite fraternity of Japanese megacorps.

Example: Housing. When I started my own company, I was living in an apartment that I had first rented as an employee of a megacorp. The entirety of the credit investigation was me presenting my business card to them. Possessing it implies both sterling moral character, stable finances, and a responsible party to intercede with should there ever be an issue with me as a tenant. (Japanese landlords and lenders will, as a matter of policy, escalate any disagreement with you to your boss, as the social opprobrium you’ll suffer will get you to quickly cave.)

The apartment required a guarantor (co-signer on the lease who is responsible for rent and damages if you fail to comply with your obligations), as many Japanese apartments do. Most young Japanese professionals use their parents. My parents were ineligible due to being, well, Americans living in America. I mentioned this fact at my office, whereupon my boss’ boss immediately said “Tanaka, he’s your subordinate. Take care of it.”, and my boss immediately called the landlord and said “This is Patrick’s superior at $COMPANY. We request that you send over Patrick’s guarantor paperwork. I assume that your company will find me acceptable as guarantor. Thank you in advance for your continued service to $COMPANY and our employees.”

When I quit my day job, I called the landlord to apprise them of this fact, as I was required to by the terms of the lease. At the time, I had somewhere north of $50k a year of income, and rent of $400 a month.

I was immediately asked to leave the apartment “at your first available convenience” because “self-employed” is about one half-step above “homeless vagabond” in terms of social esteem in Japan. No amount of explaining “I am not a risk of non-payment — I have lived here without incident for years and my income has increased as a result of quitting the day job” would mollify my landlord.

Want to buy a house? Japan theoretically has credit bureaus but credit scoring has not replaced manual underwriting to anywhere near the degree it has in the US, so you’ll find it very difficult to purchase a house without “stable employment”, by which we mean “being a salaryman.” (Or, equivalently for this purpose, a civil servant.)

Example: Relationships. Should you want to get married in Japan, you’ll find that most young ladies, and virtually all young ladies’ parents, prefer the material stability that comes from salarymen. My wife Ruriko was able to overlook my damaged professional prospects, despite the prevailing opinion among her friends being that I was unemployed. (The hypothesis was advanced, more than once, that as a foreigner who routinely travels abroad, speaks Spanish, and has money without any evidence of gainful employment, I was probably a drug dealer. I wish I were joking.)

When I met her mother for the first time, I brought my resume and tax returns. Her mother was not 100% keen on the match when we started dating, as a combination of “foreigner” and “not gainfully employed” suggested that I was not exactly marriage material, but I eventually won her over.

This is a real issue for many Japanese folks who want to become involved with startups, either as a founder or as an employee.

When I was spending my nights and weekends on Bingo Card Creator, my then-coworker (one of the two best engineers I’ve ever had the privilege of working with) built Github for SVN as a side project. He was hours away from launching it, then had one conversation with his wife about it. She was of the opinion that the side project might induce him to do something crazy, like leaving his job, or induce the company to do something relatively sane, like firing him for stealing company property (to whit, the brain cycles of a salaryman). That ended that.

(This company was actually relatively progressive with regards to letting employees have extracurricular interests like OSS projects or, in my case, BCC, but my coworker’s wife’s assumption about market terms remains quite reasonable.)

One of the most common topics I have with young Japanese would-be entrepreneurs isn’t about how to get investment or how to find customers. Many of them want my advice on how to sell the idea to their parents or girlfriends. (Would-be entrepreneur ladies have a different set of challenges, but I run into them rather less frequently and they almost never ask me for dating advice.)

My general advice for Japanese folks trying to make their loved ones happy is “Tell them that, in tech, a lot of the companies you’d want to work for are full of inscrutable foreigners who have insane decision-making processes. Take Google, for example. Chock full of Americans. Man, Americans, right? Anyhow, Google has this crazy notion that you should demonstrate capability through personal projects prior to them hiring you. So really, the startup isn’t a startup, per se, it is an extended interview for the job at Google. After you get hired by Google, of course, you’ll be a salaryman at Google. Despite being chock full of Americans, Google gets salarymen: look how they exercise benign paternalistic control over every aspect of their employees’ lives. Almost as good as Sony, twice the pay!”

(Any Googlers reading this? Howdy! Don’t worry, as an ex-salaryman, I am absolutely sincere in saying that I understand the attraction and also understand why you might object to that phrasing. In my salaryman days, I would have objected to it, too. Seen in the clarity of hindsight, I plead temporary insanity exacerbated by extraordinarily effective social conditioning designed by very, very smart people. If you’re happy, though, good for you. I know genuinely happy salarymen, too, and wouldn’t think of attempting to stamp on their joy even though I have some very pointed observations to make about their organizational culture.)

Hiring In Japan Requires Exploiting Flaws In Salarymandom

In the US, startups have to come up with a reason for engineers to join them over AmaGooBookSoft. In Japan, the competition is the salaryman ecosystem, and it is a jealous god indeed, in that if you ever take a walk on the wild side you’ll never get back into respectable society again.

How to work around this? Well, you start by hiring around the edges for Japanese society. Most of my Japanese startup buddies are very good, by necessity, at hiring people who the job market has not valued appropriately yet. Since most highly-educated, career-oriented Japanese folks aspire to jobs as salarymen or similar work in the public sector, most Japanese startups have to hire folks who don’t fit that mold.

Some examples include:

Women: I may have mentioned alluded to the fact that traditionally managed Japanese companies are pathological with regards to their treatment of women. There’s an entire academic field devoted to that topic. Anyhow, this is an opportunity for startups here: since college-educated women are tremendously underused by the formal labor market, startups can attract them preferentially.

Foreigners: It is fairly difficult (not impossible, but difficult) for foreigners to arrange to get hired as salarymen. If you’re obviously foreign, no matter what you do, you’ll be constantly assumed to be an English teacher, since that is the one value-producing occupation that Japanese society conveniently slots you in. (Oh boy, does this get old.) Given limited ability to break into The System, startups are a fairly reasonable choice of occupation if you want to live in Japan for some reason.

”Misfits”[+]: Salarymandom isn’t all roses for Japanese men, either. Some don’t have the right degree. Some burned out. Some are unable to subordinate to the extent the jobs require. Some spent more than a few years abroad and are seen as being potentially “too foreign-ized to work in a Japanese company.” Some were simply born in the wrong year and thus in college during the wrong economy to get hired, which includes lots of young men in my generation. They are thus frozen out of salarymanhood, effectively for life.

([+] A Japanese hiring manager once told me, beaming, “I look for misfits.” I apologize in advance for the following sentence, but I will quote it accurately, because it is instructive: “Otaku, Koreans, foreigners, dropouts, I’ll hire anybody who can do the work. You’re bargains.” In an ideal world there would be no racists, but in the less than ideal world that you may find yourself living in, at least hope to run into ruthlessly capitalist racists, because that’s something you can work with.)

Good news for employers: Japanese employees are, comparatively speaking, cheap, and there is only a very small premium for engineers relative to similarly credentialed employees.

I heard a great line about this once, and unfortunately I cannot remember the source: “Most people want to become wealthy so they can consume social status. Japanese employers believe this is inefficient, and simply award social status directly.” The best employees aren’t compensated with large option grants or eye popping bonuses — they’re simply anointed as “princes”, given their pick of projects to work on, receive plum assignments, and get their status acknowledged (in ways great and small) by the other employees.

$30k is a reasonable wage for an engineer in Japan virtually anywhere but Tokyo. In Tokyo, average mid-career wages in engineering are roughly $50k (5 million yen a year). (Pay is generally higher in the financial industry and in foreign-owned corporations, which are generally in the financial industry.)

Non-salary costs of employment are roughly in-line with what they are in the US — budget about 25~50% extra. They include health insurance, pensions (defined-benefit pensions are compulsory but the required levels are rather low), and some I-can’t-believe-its-not-salary disbursements such as a commuting allowance, doesn’t-live-on-company-property allowance, has-wife-and-kids allowance, and what have you. Some of these are non-taxable, which means you should characterize as little money as “salary” and as much as those allowances as possible. Ask your accountant if you’re curious.

I’ve occasionally been asked “So what do you think of Japanese engineers?” In general, I think the field here is as wide as it is anywhere else. Two of the five most talented engineers I’ve ever had the privilege of working with — whom I’d stake against anyone in the Googleplex — are Japanese.

The larger hiring market includes, just like the US, many people who cannot be trusted to FizzBuzz. Young engineers are not, in traditionally managed Japanese organizations, given authority or responsibility, with the notion that from the time they’re hired to their early thirties they’re mostly just supposed to be learning the Proper Way Of Doing Things At Our Company, so expectations for productivity are very low. (I know some folks might find it difficult to reconcile “90 hour weeks” and “very low productivity.” Suffice it to say “Six hour planning meeting by five people to discuss whether the copy on a button should be ’Sign Up’ or ‘Sign Up For Newsletter.’”)

The state of the “modern web” in Japan

Complicating the issue for the purposes of startup hiring: Japanese engineers are largely employed by Japanese megacorps, and Japanese megacorps don’t really produce wonderful modern web software. Metropolitan Nagoya has literally thousands of people who can write assembly code that you’d literally trust your life to (you have before and will again, unless your sole method of transportation is bicycles), and probably only a few dozen who you’d want working on a web application. Tokyo has more, but still far too few.

In general, with exceptions, I’d rate Japan as about 5~10 years behind the skill curve relative to the US when it comes to web/mobile development. When I left my last day job in 2010, executing Javascript in the client side of a B2B application demonstrated very impressive technical acumen and my company was worried about losing their connection to spiffy, innovative American engineering techniques. No, not joking, really.

While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior engineers at my former employer. We had binders upon binders full of checklists for doing things like e.g. server maintenance, and despite how chafing the process-for-the-sake-of-process sometimes felt, I stole much of it for running my own company. (For example, one simple rule is “One is not allowed to execute commands on production which one has not written into a procedural document, executed on the staging environment, and recorded the expected output of each command into the procedural document, with a defined fallback plan to terminate the procedure if the results of the command do not match expectations.” This feels crazy to a lot of engineers who think “I’ll just SSH in and fix that in a jiffy” and yet that level of care radically reduces the number of self-inflicted outages you’ll have.)

UX, web design, A/B testing, and the like are similar to programming in this respect. Best-in-class Japanese web applications produced in 2014 asymptotically approach Facebook 1.0 in functionality. One reason is that the primary B2C Internet consumption device is the cell phone and, prior to the iPhone arriving, most Japanese sites were designed with the “needs to be consumable on a feature phone” requirement firmly in mind.

The story of Japan’s relation to cell phones is very interesting. It is pithily summarized as “Japan managed to produce the Galapagos finches of feature phones — diverse, specialized to the native environment, found nowhere else in the world, and totally at the mercy of invasive species.” They were truly amazing hardware for the time with, like most Japanese hardware, all software re-written from scratch for every model, often in assembly. Given that those constraints make it pretty difficult to even ship a clock app, and most of the phones shipped with web browsers (!) and fairly functional Javascript interpreters (!!), they can be forgiven for having terrible UXes. And they were, until Steve Jobs changed that overnight.

Incidentally: when the iPhone came out, many foreign commentators said it would never be a hit in Japan because Japan doesn’t trust foreign products. That was horsepuckey when they said it — the iPod already had a 70%+ share while competing with Sony/etc on their home turf — and hopefully is even more obviously horsepuckey now.

Access To Capital

Japan is a rich country with almost unfathomable amounts of capital available to deploy. Japanese monetary policy has made money virtually free for more than 10 years now.

At the same time, Japanese startups have an extraordinarily difficult time raising capital.

How can both of these be true? Well, imagine a pre-YCombinator Silicon Valley with the strength of the social graph dialed to eleven. Japanese VC firms largely fund established entrepreneurs who might be called intrapreneurs: they put in twenty or thirty years of service with a particular company or group of companies, have an idea for a product that they can sell that company, raise investment from that company’s closely affiliated VC firms, and then may eventually be acquired by that company.

If you’re a 22 year old with a gleam in your eye, Japanese VC firms are not exactly rushing to make your acquaintance. Come back after you’ve got the deep network which will allow you to sell your solution into one of the megacorps. Yep, Catch 22.

Angel investors? For a variety of reasons, they’re thin on the ground here. Japanese tech companies have not yet started doing wide distribution of stock options like American tech companies do. When Google/Facebook/Groupon/etc IPOed, each of those events created hundreds to thousands of people who suddenly met the accredited investor standard, had a great deal of money to spend, and were interested in technology. By comparison, IPOs in Japan are exceptionally rare and the equity is typically centralized among investors and management. This results in relatively fewer people who can write $25k checks.

Angels in Silicon Valley have evolved a certain level of professionalization with regards to practices which is wildly not the case in the rest of the United States. These practices are actively promulgated by (de-facto) consumers of the angels’ services, such as YC and 500 Startups.

Japan is not quite there yet. If you were, hypothetically, to spend a few weeks pitching a promising startup to well-regarded angels in Silicon Valley, you would hear very few terms which shocked the conscience. If you were, hypothetically, to spend a few weeks pitching a promising startup in Tokyo… well, a plane ticket to San Francisco might be a very reasonable business expense, we’ll put it that way.

Valuations in Japan are, by Valley standards, absolutely ridiculously low. I am constrained here from giving you many anecdotes because that would be socially embarrassing for friends, so instead, can I tell you an anecdote from St. Louis? Slicehost was once told by an angel investor that the investor would co-sign a $250,000 loan in return for 10% of the company. This is after they had an enormously quickly growing hosting company. In Silicon Valley, this results in millions getting thrown at you at a valuation in the tens of millions. In Tokyo, the strangest thing about the Slicehost anecdote would be “Why’d they need $250k? Couldn’t they have gotten by with $200k? Man, St. Louis must be made out of money.”

Debt financing? Hah, you’re funny. If you’re attempting to open a hair salon, you can get, say, $0.8 million or so collateralized by the real estate, and use some portion of that for working capital. Software firms, on the other hand, are not ideally suited to the standards of underwriting departments here. (My bank, in consideration of my decade of patronage, spotless payment record, and outstanding character references, generously approved a $3,000 credit line for my business.)

Selling To Japanese Companies

Do you enjoy enterprise sales, but think it includes excessive focus on the product and not enough wining, dining, and corporate politics? Then does Japan have a deal for you.

Low-touch software sales is relatively popular in the US. (“Low-touch sales” is the Basecamp model, where a compelling website, free trial, onboarding experience, email marketing, etc generally sell prospects with only a minimum of personalized interaction with the company. “High-touch sales” is the Oracle model, where you spend a lot of time on individualized communication.) Many companies are quite successful at low-touch sales, and many more use the experience of having done low-touch sales successfully to start an enterprise sales operation.

The Japanese market virtually requires high-touch sales for selling software, including even low price-point software to SMBs. Decisions for small purchases for software (and a variety of other goods and services) are primarily made after face-to-face meetings with local sales reps. A great overview of the traditional process is here, and I cannot really elaborate on it more than “No, really, we really did have to take a distributor’s reps out to drinks to procure more MS Office licenses. No, really, the most formidable Japanese low-touch SaaS entrepreneur I know figured out how to sell SaaS door-to-door in Tokyo.”

The economics implied by this arrangement make Japan relatively more hospitable to enterprise software and relatively less hospitable to e.g. SMB software. (This is also a major reason why I, personally, don’t sell to the Japanese market. Given that I’m primarily limited by my own availability, selling to the US implies an order of magnitude or more more revenue per every hour invested.)

Maintaining a team of reps to do client visits (who can, quite literally, often drink their way through a $2k entertaining-prospects budget on a monthly basis… in a single evening if you don’t discourage that) costs quite a bit of money, but once you get into average contract values in the several hundred thousand to several million region (dollars), it works out to the ~20% that US enterprise sales operations expect, and the same factors that made adopting you difficult now makes it very difficult for competitors to steal your accounts.

Japan is a gigantic market for software, and the number two market worldwide for a lot of US firms. Prominent examples include Oracle, Salesforce, Microsoft (IIRC), etc etc.

Penetrating the Japanese market virtually requires either a local office (in Tokyo, because you’ll want to have in-person visits with your customers and, if they’re large Japanese corporations, odds are they are in Tokyo) or an arrangement with a Japanese distributor. In general, relationships between vendor, distributor, and ultimate customer can be fraught. If you’re coming to Japan, think long and hard about the distributor decision, as cutting them out of the loop is seen as unseemly behavior, but keeping them in the loop if they’re inefficient virtually dooms your chances here.

If you want to read more on this general subject, I recommend Venture Japan, whose take on sales operations here generally matches my experiences.

Do you want to sell Japanese companies consulting services, as opposed to products? Remember, you’re going to be compared with the price of domestic employees. They’re quite cheap, so you’re going to get quite a bit of price resistance.

The Personal Touch

Doing business with Japanese companies frequently resembles It’s A Wonderful Life. “Customer relationships” are not an empty phrase — many business relationships where one is approximately equivalent to a row in the database in the United States are, instead, expected to be relationships between two actual people.

This is occasionally exasperating, as a software person who doesn’t want to have to take someone drinking to sell a single SaaS account, but it is occasionally quite charming. Moving to Japan, particularly small-town Japan, was like visiting an old America that I had heard stories about but had never gotten the opportunity to experience.

For example, when I first came to Japan, I had no computer. I also had no money, because the plane ticket and setting up my household ate all of my savings. In America, this isn’t a barrier to getting a computer, because Dell will do a quick FICO score on you and then happily extend you $2,000 of trade credit.

Dell Japan, on the other hand, set me up with two phone calls with actual human underwriters at two Japanese financial institutions. Both had me fill out rather extensive forms (100+ questions — seriously). The first said “In view of your length of tenure at your employer and length of residence at your apartment, we don’t feel that your situation is stable enough to extend you credit.” The second said “Look, umm, officially, I am supposed to just tell you that we decline your business and wish you luck. Unofficially, the bank doesn’t extend foreigners credit, as a matter of policy. You’ll find that is quite common in Japan. I know, it is lamentable, but I figure that you’d be able to save yourself some time if you knew.”

So I gave up for a while, but mentioned to a coworker later that week that I really wanted a computer to be able to Skype home. He said “Come with me” and we left, in the middle of the work day, to visit a bank. It is a smaller regional bank in Gifu. I’ll elide naming it to avoid the following story being personally identifiable, but suffice it to say it is a very conservative institution.

My coworker got a credit card application and asked me to fill it in. I did so, but told him “Look, two Tokyo banks, which are presumably about as cosmopolitan as Japanese financial institutions get, just shot me down. One of them explicitly did so because I’m a foreigner. The chance of this middle-of-nowhere bank accepting a credit application is zero.”

“Don’t worry, I know the manager. Hey, Taro!”

Taro and my coworker had gone to school together.

“Patrick here just started working with us. He wants to buy a computer to call his parents, diligent son that he is, and needs a credit card to do it. Here’s his application. Make sure it doesn’t get lost in the shuffle, OK?”

Some weeks passed, and I assumed that I had been denied. Then there was a knock on my door early one Saturday morning.

It was bank manager Taro and an older gentleman who introduced himself as the Vice President for Risk Management of the bank. He promptly took over the conversation.

“You have to understand that we’re not one of those banks. We’re not some magical pot of money. Every yen we have is a farmer depositing against a bad harvest or a retiree’s pension, carefully husbanded over a lifetime. That is a sacred trust. We cannot lose their money. The bank has to be appropriately careful about who we lend that money to. Taro here tells me your trustworthy, so that is good. Even trustworthy young men sometimes make poor decisions. I need to know you won’t, so before I give this credit card, I have three questions for you.”

“Will you ever use this credit card to gamble?”

“No, sir.”

“Good. Will you ever use this credit card to buy alcohol?”

“No, sir.”

“Good. Will you ever give this credit card to a woman who is not your wife?”

“No, sir.”

“Good. Think darn hard before giving it to your wife, too. OK, you pass muster. Sign here.”

That was the first of a dozen stories which you wouldn’t believe actually happened about that bank. Taro correctly intuited when I started dating a young lady, and when we broke up, solely based on on my spending habits. He considered that part and parcel with looking out for my financial interests.

Taro stopped me from doing a wire transfer back to Bank of America to pay my student loans during the Lehman shock because Wachovia had gone into FDIC receivership that morning. I told Taro that I didn’t have an account at Wachovia. Taro said that he was aware of that, but that I used Lloyds’ remittance service to send wires, and Lloyds’ intermediary bank in the US was Wachovia, which might or might not be safe to have money in at the moment. I asked Taro how in God’s name does a banker in Ogaki, Japan happen to know what intermediary banks Lloyds uses in North America off the top of his head, and Taro said, and I quote, “There exists a customer of the bank who habitually makes USD wire transfers using Lloyds and, accordingly, it is my business to know this.”

Taro called me on March 12th, the day after the Touhoku earthquake, to say that he was concerned about my balance in the circumstances (I had cleared out my account to pay a tax assessment minutes before the quake) and, if I needed it, to come down to the bank and, quote, we’ll take care of you and worry about the numbers some other time, endquote.

Taro eventually retired from his position, and as part of making his rounds, gave me a warm introduction to the new bank manager. He made it a point to invite me out for coffee, so that he’d be able to put a face to Taro’s copious handwritten notes about my character. Some years after that, a new manager transferred in. I popped by with a congratulations-on-the-new-job gift, mildly surprising the staff, but it felt appropriate.

When I moved to Tokyo, I went to the regional bank’s sole Tokyo office, which exists to serve their large megacorp customers. They were quite shocked that I had an account with the bank (“Mister! Citibank is down the street! If you use our ATMs you’ll get charged extra!”), and even more shocked when I told them that I run a multinational software company through it. “Wouldn’t you get better services with Citibank or Mitsubishi?” The thought of switching never crossed my mind. Indeed, I can’t imagine anything that would convince me to switch. They don’t make numbers big enough to compensate for how much I trust my bank.

Was I a particularly large account to the bank? Nope. It’s the same passbook savings account a 17 year old gets to deposit their first wages into. For 8+ of my ten years in Japan, my balance there was below $2,000.

The bank is one anecdote, but I could tell you about the hair stylist who drops me a handwritten postcard after every appointment, the restaurant that I went to weekly that tried to cater my wedding for free, the glasses shop which invited me to come back for a (free) frame re-bending and cup of coffee any time I was in the neighborhood, etc etc.

Japanese customers, in both B2C and B2B relationships, expect a level of personalized, attentive service which is qualitatively different than that in the United States. Anomalously good sales reps in the US are frequently operating at table stakes or below in Japan.

On the plus side, after you’ve actually won the business and demonstrated capability to serve customers to these standards, Japanese customers are very loyal. This is true both qualitatively and quantitatively. I’m aware of a Japanese SaaS app which, despite being sold at low price points on a low-touch month-to-month model (all predictive of relatively high churn rates) has a churn rate which would be considered exemplary for an enterprise SaaS app sold with high-touch sales on an annual contract.

The Mechanics Of Getting Started

Japan has a reputation as being forbiddingly bureaucratic. I find that this depends strongly on what exactly you’re doing. In many respects, the actual mechanics of starting a business are quite easy.

I quit job on March 31st, took April 1st off, and went down to town hall to file paperwork on April 2nd. As an American, I expect dealing with city government to be a very painful experience. I was whisked between three departments staffed by knowledgable, efficient, mostly pleasant bureaucrats, and in less than 30 minutes walked out the door with health insurance, a public pension, and forms filed to reflect that I’d be filing as a self-employed person for taxes the following year.

Historically, Japan makes company formation rather more difficult than it is in the US — it costs a few thousand dollars (filing fees and legal advice, which you’ll need to complete the process) and requires that you have $30,000 of capital. This has changed a bit over the years, in response to feedback from Japanese entrepreneurs. Personally, though, having a supermajority of my customers be in the US makes having US entities equally useful as Japanese ones, so I just have US LLCs, which you can open with ~$500 and 30 minutes. (Japan’s closest equivalent is a “goudou kaisha”, which are substantially easier and less costly to form than traditional corporations. However, many Japanese entrepreneurs choose to go for the traditional corporation anyway, on the theory that it is likely to be perceived as more trustworthy.)

I’d estimate that I spend approximately 3~5 work days a year dealing with government requirements. In my business, the overwhelming majority of this time is spent on doing taxes. They’re approximately as burdensome as American taxes at my scale of business. One added hurdle: Japanese accountants are typically not conversant about the software industry and, since the intersection of Japanese tax law and software realities is not well settled, are often not tremendously capable of giving great advice about it.

Where does it get more difficult? As you get progressively more enmeshed with the Japanese bureaucratic state, the amount of time you’ll spend managing that relationship goes up rather drastically. Assuming you’re not in a regulated industry, like e.g. finance or healthcare, the thing which is most likely to bring you to government attention is hiring full-time employees. (If you’re in a highly regulated industry, may God have mercy on your soul — ask your competent legal advisors rather than me.)

Remember how societally important the employment relationship is? The Japanese government will expect you to discharge your responsibilities in that relationship, and this will generate enormous volumes of paperwork. Most of it is similar in character to running a business anywhere, but there is a lot of it. The government is impressively well-organized, but it is well-organized to accept your paper declarations in-person, and you’ll spend a lot of time acting as a transport layer for SQL queries between government offices.

I once was obligated to spend $2 to get a piece of paper telling Agency B that a particular number in Agency A’s possession was, in fact, accurately reflected on the paperwork I had earlier presented to Agency B. Agency A and B simply will not talk to each other about this. They have a protocol, and you need to walk the messages of that protocol between each of them, until they tell you you’re done. Usually, A and B are reasonably close to each other, so you’ll waste a minimum of travel time.

Japanese folks consider at-will employment to be an alien institution, much like you might be thinking about the salaryman system. (At-will employment is the common-in-many-US-states arrangement where employers and employees have the mutual right to terminate employment for virtually any reason.) If you hire full-time employees in Japan, you can only dismiss for cause, and the bar is relatively high.

Imagine having the following conversation with the relevant authority: “Incompetence at one’s job is only a reasonable cause for termination if you’ve dutifully discharged your duty to retrain the employee, documented several months of poor performance subsequent to the retraining, and explored options for other jobs they could do for you. After all, everyone starts out at incompetent, right? If we let any company just up and fire anyone merely for not being able to do their job, that would contravene the social purpose of employment.”

As you can imagine, this makes hiring for small companies even more difficult than it already as.

If one wants to terminate an employee for poor performance in Japan, the most efficient way is dealing with them like an unwanted New York or San Francisco tenant: offer to buy them out. If they don’t take the buyout and don’t wish to leave, your escalation options are limited and fairly high-stress.

Availability Of Non-Employee Business Inputs

Forgive me for stating the obvious, but people do ask, so: Japan is a highly developed industrialized nation where any business input you require is available, in quantity, if you’re prepared to pay for it.

Office real estate, particularly highly desirable office real estate in Tokyo, is more expensive than you might expect and modestly difficult to acquire. This is largely because, as a startup which is considered off-the-scale risky, you’re not a good candidate for a lease.

That said, if you’re willing to look around a bit, walk an extra 10 minutes from the closest train station, and go to a slightly less prestigious address, you can reasonably get a startup-capable office for $2,000 to $3,000 a month. A floater spot at a coworking spot in Tokyo runs about $300 to $400 a month. If you simply need a place to park your weary bones, Internet cafes are ubiquitous and charge about $4 an hour, although they’re typically not great environments to work from.

Internet connectivity to your office, place of residence, and phone is fast and cheap. Gigabit Internet runs about $50 or so a month and a generous data plan for an iPhone is about $50 to $100 a month. Internet connectivity in public spaces like e.g. (regular) cafes is much, much rarer than it is in the United States, although this is changing.

Do You Speak Japanese?

I’ve never had the experience of running a business in Japan without speaking Japanese. Doing so strikes me as playing life on hard mode. Japan theoretically has compulsory English education but, practically speaking, Japanese folks who can carry on a business-level conversation in English are rather thin on the ground.

This is true even in engineering. I know, I know, most technical documentation in software exists in English, and many foreign engineers are amazed that people who don’t possess a firm command of English can nonetheless be great engineers. All I can say is you’d be surprised by how many levels of fluency there are.

Although it is changing gradually, routine business dealings are generally conducted only in Japanese. Some businesses or government offices might have forms which are bilingual, but you’d be unwise to expect an answer to any question about the form.

Learning to speak, read, and write Japanese is enormously fun. So is starting a company. I recommend not combining the two. It typically takes at least two years of high-intensity study to be able to carry on a basic business conversation in Japanese (on the level of “Are you done with that? Not yet? Why not, and when do you expect to be done?”) and, unless you’re already coming from literate in Chinese, four-plus years until you’d have pretty good odds of understanding consequential business documents like e.g. a lease or contract.


You can skip this if you’re Japanese.

Japan has a variety of categories of status of residence, which is a status quite similar to what the rest of the world calls visas. (A visa only lets you into the country here, but a status of residence allows you to stay and gives you privileges you might want during your stay, such as the privilege to work without being deported.)

Applications for most professional statuses of residence, such as engineer or humanities specialist, require sponsorship by a Japan-based organization. One’s likelihood of being approved depends in a fairly direct fashion on how much societal pull that organization has. If Toyota wants you to get a status of residence, you will be issued a status of residence. It gets somewhat more dicey with smaller companies, and the standard of review for documentation gets rather higher.

Status of residences follow employees, not jobs. If you are, for example, an engineer, you can quit your job as an engineer and get any other job without requiring a review of your immigration status… as long as that new job is in the same status of residence. This is very important.

The most common way to licitly start a business in Japan as a foreigner is to arrange to work with a Japan-based employer, get one’s status of residence through the employer, work for a time, quit, and then go into business for oneself in the same field. Although it isn’t exactly encouraged, the regulations for e.g. engineers don’t disallow you from being an engineer for a variety of customers including e.g. an entity you just happen to own. This means that you have from the time you quit to your next renewal of your status of residence to figure out how to either e.g. justify an entrepreneurship status of residence or fulfill the three prongs of your existing professional status of residence. (“Continued stable employment, at a Japanese organization, as demonstrated by contracts.”)

My hack around this, after quitting the day job, was to describe myself as an engineering consultant. I presented the immigration office with a stack of invoices and tax returns demonstrating that I made a stable living in software. (Much of it was from selling software, the key bit from their perspective was that at least one of my contracts had a Japanese company as a party to it.) After a bit of wrangling, they approved me to continue doing what I was already doing. (Word to the wise: this trick for self-sponsorship doesn’t technically speaking allow one to “run a company”, so I would avoid doing things which make it undeniable that one is in fact doing that, like e.g. hiring full-time Japanese employees.)

There exists a new status of residence for highly-skilled professionals which may make this somewhat easier than the business manager status of residence (which is achievable but has toothy requirements, like having 2+ full-time Japanese employees and at least ~$500k in capital).

Dealing with Immigration is, always and everywhere, high stress for immigrants. On the plus side, highly-educated Westerners are not the primary focus of xenophobia in the immigration agency. (Did I say xenophobia? Wait, sorry, I meant to say “zealous attention to their statutory duty to ‘forcibly expel undesirable foreigners from the nation.’”)

Permanent residence is an option, theoretically after 10 years of residence in Japan but, practically speaking, only about five if you’re married to a Japanese person. You’ll need to make a showing that your presence in Japan redounds to the benefit of Japanese society. It would be easier to do this if you were a salaryman, but successful entrepreneurs can also, in principle, pass the bar, depending on the mood of the examining clerk.

On Being A Foreigner In Japan

I customarily start speeches in the US with a fish-out-of-water story from over here, because they’re often funny. Some were less funny when I lived them, believe me.

Japan has a reputation for xenophobia. This is partially unfair: it is a large nation with more than 100 million people, who are not unanimous about anything which humans are not in general unanimous about. Many Japanese folks like foreigners, many more are indifferent, and attitudes in even less-enlightened portions of the country perceptibly improved in the 10 years I’ve been here.

That said: is racism a bigger problem in Japan than e.g. in the United States? Oh, yes. Unquestionably.

Let’s say you’re building a job-hunting site in the US and you notice, in the documentation, a boolean flag on the JobListing object titled nonWhitesAllowedToApply. It being 2014, several decades after relevant legislation has been passed, and you being at a Fortune 500 company which does not have a reputation as committing itself to clearly illegal courses of action, you might ask your boss “Hey boss, that nonWhitesAllowedToApply flag? Ahem, what the hell?”

You know what would not happen? Your boss telling you “Yeah, umm, I see how that could potentially be problematic, but the customer wanted it.”

Not that any Japanese company has ever instructed an employee to implement nonWhitesAllowedToApply, mind you. That would be silly.

Similarly, it is illegal in Japan to discriminate on the basis of race in e.g. housing. This bounced me out of approximately 40% of available apartments in Ogaki and a non-zero number in Tokyo, though I think I could have probably pulled strings around it. (In general, foreigners are foreigners in Japan, but certain foreigners are less foreign than others. Highly-paid well-educated articulate Western men with deep Japanese social networks are almost Japanese for the purposes of avoiding institutionalized discrimination like that. Almost.)

In general, I counsel picking one’s battles carefully with regards to this sort of thing. The formal channels for resolution are very slow, and you can quite easily win the battle (vindicated by the local equal opportunity commission; collect damages in the amount of a month’s salary) and lose the war (unable to work again in this country). I generally avoid it by picking associates carefully. This works in the 99.8% of time when I can pick who I deal with. (Sadly, while you can pick your bosses and landlords, police/immigration get to pick their foreigners, whether the foreigners like it or not.)

While not as consequential as discrimination which has actual professional/housing/etc impacts, Japan can occasionally be maddening with regards to certain expectations about foreigners. One of them is a widespread belief that foreigners don’t speak or read Japanese.

Imagine the following dialogue.

Me: “Good morning.”
Clerk at ward office: “WOW YOU SPEAK JAPANESE SO WELL.”
Me (ritual reply for compliment): “You are entirely too kind.”
Clerk: “So can you write Japanese, too?!”
Me: “I’m literate.”
Clerk: “So you could write, like, the name of this office?”
Me: “Yes. The hardest character in it is taught in third grade.”
Clerk: “Wow that is so amazing! I don’t think I’ve ever met a foreigner who could write Japanese.”
Me: “That’s funny. I don’t think I’ve ever met a Japanese person who has never met a foreigner who could not read Japanese. Except for three other clerks at this office this morning. And the last 2,000 times this happened.”

I did not say that final line because one does not go out of one’s way to antagonize people who are fundamentally of good will and also in a position of authority over one’s ability to continue living in one’s neighborhood. But believe me, I’ve wanted to say it about 2,000 times.

Imagine walking the tax return for your multinational software company into the local tax office and being asked, in a clerk’s best speaking-to-a-slow-child voice, “Who can I call mimes phone if I have a question shrugs about this paper points?”

“My name and contact information should be printed in the responsible corporate officer box, as per the usual.”

“But tax words are hard!”

“‘Straight-line calculation method for depreciation of an intellectual property asset’ was a really corker, I agree, but luckily your pamphlet ‘Easy-Peasy Taxes For The Self-Employed’ helpfully defines it on page 47. I’ll do my level best to comply with all of my requirements under the law, including looking up jargon in the dictionary, when necessary.”

It is occasionally to one’s advantage in business dealings to be a foreigner, largely because you can selectively code-switch between societal expectations for Japanese people and societal expectations for foreigners. I try to avoid abusing this, but it has occasionally been useful to e.g. object vociferously to something while pretending to be unaware that one is causing a scene.

Few things in life are worth fighting over. Fights that are worth fighting are usually worth winning.

For more prosaic examples of strategic use of foreign-ness, Venture Japan has some examples of deploying it for e.g. software sales. I’m aware of a few enterprise sales reps who have one quite well for themselves using those approaches, but wouldn’t personally endorse them.

Are Any Businesses Uniquely Helped Out By Being In Japan?

I very rarely feel like my professional opportunities are greatly circumscribed by being in Japan. Now is a wonderful time to be alive, and a combination of the Internet, a worldwide community of practice, and phones/plane flights mean that my business is virtually as viable in Tokyo as it would be in Toledo.

That said, candidly, my particular business does not benefit much from being here. (It would operate equally well from any reasonably fast Wifi, and since most of the customers are in the US, being closer to US time zones would mean a few less late nights for me.)

If you do sell to Japanese customers, it is obviously to your advantage to be here. Would I recommend that, given you have a choice to site your business anywhere in the world? Well, if you understand that your primary business challenge is going to be in sales, and that sounds like a good fit for your skill set and ambitions, Japan is a reasonably good place.

The market is tremendously underserved here with regards to technology solutions, in virtually everything relevant to you if you’re reading this. UX and design which Silicon Valley companies would consider barely adequate for an internal admin app would strike Japanese customers like wizardry from the future.

Competition from other startups is rather low, and Japanese megacorps do not exactly have Internet DNA yet, which means that distribution channels which are extraordinarily competitive in the US (like, say, AdWords or SEO) are not nearly as competitive here.

Market-leading foreign companies often neglect their Japanese operations, allowing “Like $NAME_A_STARTUP, but natively Japanese” to be a perfectly adequate strategy. Yes, you’re locked onto a “small island nation”, but it is a small island nation of 130 million globally rich people. (Dave McClure once said, with regards to Japanese startups, that they’re far too eager to exit the Japanese market and go multinational. I tend to agree with this assessment. The market here is gigantic and the competition usually sucks. I think that most Japanese entrepreneurs just want to broaden from the Japanese market quickly in the hopes that they’ll land somewhere which celebrates entrepreneurship.)

I’m optimistic in the longer term about the Japanese startup community specifically and, though this might be controversial here, the Japanese economy generally.

Recently, there has been a modest bit of interest by Valley investors in Japanese startups. I’m aware of YC and 500 Startups being active here, and some of the best Japan-based entrepreneurs I know have substantial cross-Pacific ties. (One plug: Jason Winder, CEO of Makeleaps, which is Freshbooks except for Japan, is presently in San Francisco. If you are, too, you should strongly consider taking him out for coffee. He’s the most formidable CEO I’ve ever met.)

Should any of the rest of you be interested in starting a business in Japan, investing here, or what have you, please drop me a line. I’m always happy to help. Similarly, if you’re ever in Tokyo, I’d be happy to say hiya.


Kalzumeus Podcast Episode 9: Customer Onboarding With Samuel Hulick

Samuel Hulick, one of the guys I trust most with regards to SaaS user onboarding, joined us for this episode of the podcast.  I met Sam first when he was writing a book on the topic.  The best evidence I can give you for the proposition “Sam knows more than the vast majority of people about user onboarding experiences” is the fact that he’s written up 25+ of them publicly (e.g. Basecamp’s) and that the writeups are of very high caliber.  Check them out sometime.

[Patrick notes: The transcript below has my commentary inserted like this, as usual.]

What you’ll learn in this podcast:

  • What mistakes SaaS companies frequently make with regards to user onboarding.
  • How to start preparing users for success pre-signup, using site copy and appropriate expectation setting in marketing.
  • How SaaS companies often botch product tours, and how you can make yours serve the user rather than serving the product team.
  • How to use lifecycle emails to make customers more successful.
  • How organizational issues at SaaS companies often directly cause problems in the artifacts given to customers, and how you can avoid this.

Podcast: Customer Onboarding

MP3 Download (~75 minutes, ~110MB) : Right-click here and click Save As.

Podcast format: either subscribe to in your podcast reader of choice or you can search for Kalzumeus Podcast in the iTunes Store.

Transcript: Customer Onboarding

Patrick McKenzie:  Hideho, everybody. This is Patrick McKenzie, here with the ninth episode of the Kalzumeus Podcast. Our guest today is Samuel Hulick, who is behind My usual co‑host, Keith, couldn’t make it today.

I moved down to Tokyo recently [Patrick notes: And will talk about that more some other day.], so it’s a bit of logistical nightmare getting everybody together, but hopefully that’ll work out itself over the next couple episodes. Anyhow, it’s great to have you here, Sam.

Samuel Hulick:  It is wonderful to be here.

Patrick:  I think today we’re just going to talk a little bit about what you’ve noticed in your experiences as a consultant/author on the topic of user onboarding, what software companies typically do well, do poorly, how they can improve on it.

Also, on a meta-level, I’d like to ask about your experiences of building up the reputation as an expert in this emerging field of dev‑related topics, and how that’s worked out for you personally in your career. Sound good?

Samuel:  I would be delighted to cover all of that.

Patrick:  Awesome. I guess, first question. Sam is one of the few people I trust on the topic of user onboarding.  I trust Sam largely because he’s done maybe 20 public tear‑downs of websites, saying, “Hey, this is a SaaS company. I signed up for their product.”

He makes copious screencasts/screenshots of the product during the onboarding phase. If you’re not familiar with that term of art, onboarding is basically the experience immediately after you sign into the product for the first time. It’s analogous to unboxing in the retail world. Sometimes we call it the first‑run experience, too. Anyhow, onboarding is getting someone shoved into the software.

Sam did public tear‑downs about this for various websites, ranging from Gmail and Basecamp, down to no‑name websites like mine, and just highlighted, “OK, here’s what they’re doing well. Here’s what they’re doing poorly. Here’s what my recommendations would be for doing it better.”

One of the things that I noticed as I was reading a lot of Sam’s write‑ups is that they’re really, really good. These are the sort of things that I used to do for consulting clients. Mine were not nearly so in‑depth or detail‑oriented. I would just say, “I A/B tested things around this before.”

I would do X, Y, and Z, but I had no, really, theory of the mind of the customer that was informing X, Y, and Z, where Sam is much better at the theorizing behind it. Anyhow.

[Patrick notes: I think I was excessively self-deprecating here with regards to not having a theory of the mind.]

Samuel:  I’m honored that you think so.

Patrick:  After building you up so much…

Samuel:  [laughs]

Patrick:  What’s the general takeaway for software companies about our onboarding processes? What are the easy ways that we fluff things up right now, and how can we do that better?  Hmm, that sounds too generic, but let’s roll with it anyhow.

How You Structure Your Teams Spills Over Into How Your Product Is Experienced

Samuel:  I think that probably the biggest mental hurdle to get over ‑‑ well, a couple things. One is I oftentimes will look at how the organization is organized. There’s the term Conway’s Law, which is that the output of a team will be reflecting the way that that team was organized. If one department doesn’t talk to another, the parts of the application that they’re in charge of will likely not talk to each other.  Things like that.

[Patrick notes: This observation is almost painfully on-point at larger software companies.]

I find that a lot of times, when you’re looking at how a product is produced or how it comes to be, there’s typically a marketing department or team, where they’re organized around driving awareness and acquisition, sign‑ups, and whose responsibility just ends after sign‑ups a lot of the time.

Then you’re looking at a product team, where they’re driven around creating new features and driving ongoing engagement. There are often  really any humans in the organization that are really incentivized around bridging the gap from sign‑ups to highly engaged users.

Much of the time, if there’s an onboarding issue in the user experience, it’s a lot of times derived from the way that the teams were split up to begin with.

On top of that, a lot of times, onboarding seems to be something of an afterthought. I look at a lot of similarities between onboarding in a product, like a SaaS product, and tutorials in a video game.

[Patrick notes: I would explicitly point out Blizzard games or the Half-Life series, which quite literally have this down to a science.  Play the first 5 minutes of Diablo 3 or World of Warcraft or Hearthstone or Half-Life and observe how they’re simultaneously effective at telling a story, manipulating the player’s emotional state, and also introducing several core UI elements of a piece of software whose user-visible complexity rivals that of MSWord or MySQL.]

A lot of times, people seem like they get really carried away about making the “core” game, or the essence of the software product, without really looking at the problem to be solved as, how do you even get people engaged, to begin with?

I wouldn’t say that onboarding is necessarily something that I would recommend waiting till the very end, when all the resources and time are exhausted and trying to staple something on after the fact.

I would really look at your product as the essence of what you’re doing is creating some amount of success in people’s lives. The onboarding process is getting people transitioned from a situation that’s probably frustrating them, because that’s why they’re trying out a new product, to a situation that they’re a lot happier with.  Then, they pay you money for that pain relief.

That is my general take on the matter, as a brain-dump.

Patrick:  Sure. I largely agree with everything in that general take. It’s something that I often went to clients and other people in the SaaS industry and try to impress upon them.

Again, for reasons like you were talking about with Conway’s Law, and the fact that this is not tracked anywhere in the organization, most companies are unaware of this.

Depending on the SaaS company, if you have a free‑trial sign‑up model, where folks can have a way of testing your software out, somewhere between 40 and 60 percent of the people who start a free trial will never log into the software a second time. They get that one free‑trial experience, and then they’re out of there.

Given that the first five minutes of the use of the software is all a lot of people are ever going to see, you really have to make that first five minutes absolutely sing if you’re going to convince, literally, half of the market for your software that they have to do just a bit more work to get the change in their life that your software is promising to them.

[Patrick notes: I occasionally experience pushback from the phrase “change in someone’s life”, but I honestly think that that is about where we should aim as software entrepreneurs.  You don’t need to have the impact of a religion or their spouse, but you should be at least aiming at “washing machine.”  Take a person who has never had a washing machine, introduce them to a washing machine, bam, that is a major improvement in their quality of life.  That is, approximately, how much you want to revolutionize a business process with your software offering.  (Or you can try to do it in B2C, but it’s really, really hard, and I only recommend doing so if you have a lot of money to spend, for reasons I’ve discussed before.)]

Patrick: Instead of just typing some information into the computer, they’re going to have to go to bat for your software with other people in the organization. They’re going to have to change the way they how do some process at their job. They’re going to have to change their habits over months and years to get value out of it.

That long trek to changing those habits and unlocking the value starts with just that first five minutes, so that first five minutes absolutely cannot be a blocker to them.

Samuel:  For those listening, I’m nodding vigorously right now. I completely agree. The 40 to 60 percent, I’m very thankful for you making public claims in that regard, because then I get to reference that in my material. Very, very much agree.

[Patrick notes: I’m a little afraid of citogenesis in the Wikipedia sense, so feel free to share stats from your own businesses if they agree or disagree with that range.  It is my rough rule of thumb for B2B apps sold on a low-touch free trial model.]

Samuel: I would also say, one thing that you touched on is it’s really important to make those first five minutes really great, but at the same time, I wouldn’t constrain the definition of onboarding to just that first‑run experience. A lot of times, when I see onboarding done really well, it’s because they have a really smart automatic‑trigger life‑cycle email campaign that follows, or things along those lines.

Then also, even before sign‑up, just orienting people around the value that your product offers and setting expectations and guiding their motivation and momentum in the right direction. If you think you’re signing up for one thing and you’re getting another, that’s an onboarding problem, but certainly the heavy weight could’ve been lifted long before that person signed up.

Patrick:  I have a priceless anecdote about that, actually. Everybody does experiments, right, with traffic channels, different acquiring customers, whatnot. I presume they won’t be too mad at me for talking about this.

Fog Creek has FogBugz, which is a bug‑tracking product for developers. Due to an AdWords campaign at one point, Google was sending people to the landing page who were looking for things like [bug for tracking boyfriend’s car].

[Patrick notes: Back in the day, I did occasional consulting for Fog Creek.  Assume that I caused this problem.]

The landing page, at the time, did not disabuse someone of the notion that FogBugz was the right product for them. They got some very interesting feedback in the customer support channel about, “I don’t see how I track my boyfriend’s car,” “How do I start tracking my boyfriend’s car?” et cetera.

Samuel:  [laughs]

Patrick:  You can think of that as an onboarding problem. Obviously, the solution was tightening up our Google campaign so that we’re no longer paying them a lot of money to send people who want to do various illegal activities with software. Setting expectations is really important.

I like that you mentioned life‑cycle emails. Some of the best individual wins that I’ve seen companies get are just circling back with someone in an automated or semi‑automated fashion, a day or two after them signing up, and saying, “Hey, you signed up for blah the other day, but it looks like you haven’t gotten around to using it. Here’s how to get started.”

Then you’re giving them X, Y, and Z, where if you can, X, Y, and Z are triggered based on their actions in the app. For example, hypothetically, if I was writing one for GitHub, and somebody had signed up for the paid version of GitHub, but 48 hours later they didn’t have a single repository in their paid version of GitHub, I might ping them and say, “Hey, thanks for signing up for GitHub.”

“Github the place where you want to get all your source code, but you don’t have your source code in yet, so here’s how you can start by getting your source code into GitHub. Or maybe that isn’t your job. Maybe that’s someone else’s at the organization. Here’s how to give them the instructions they need to start using your company’s new GitHub account.”

Samuel:  I completely agree. I think that looking at what are the external pressures that are forcing that person to be trying out new software to begin with, and being as aware of those as possible, is a total no‑brainer.

Especially B2B software, understanding who’s the person who needs to sign off on this check being cut, is there an IT review of some sort that needs to happen, can you create a PDF that they can slide across the desk to their boss that explains the ROI of your product instead of hoping that they can come up with something clever on their own ‑‑ those are all things that I would highly recommend doing, for sure.

Patrick:  That ROI calculation is priceless. I actually do that in a scalable fashion for Appointment Reminder. Somebody, several weeks into a trial, once emailed me when I said, “Hey, your trial is coming towards the last couple of days.”

Since I have their credit card already, it’s just like a courtesy notification, like, “Hey, we’re going to charge you in three days. If you don’t want this to happen, cancel now.” That is not actually the copy I use, but that’s the sense of it.

They wrote me back and said, “Hey, Patrick, really appreciate the email. I’ve got a question. My boss is asking me for an ROI calculation on this software. I don’t know what ROI means and I don’t know how to calculate it. Can you do that work for me?” I’m like, “Well, since it’s going to help you pay 80 bucks a month for my software, I can certainly help you out for that.”

I attached an Excel spreadsheet, where I made some assumptions about their level of use of the software based on what I had seen in the database about them and assumptions on their cost structure as a business. I said, “OK. It turns out that this is saving you,” make up a number, “$500 a month. It only costs you $100, so that’s an ROI of 500 percent.”

Then, right after I sent that email, I’m like, this calculation is generalizable to everybody, and there’s approximately nobody who would hate hearing that they’re getting 500 percent ROI. I just changed the email that they’d gotten that eventually prompted them to ask about ROI, such that the computer automatically calculated the ROI if they were getting a happy number.

[Patrick notes: Maybe this would be easier to understand with a visual aid?  Here’s the “trial progressing well” email, which goes out on day 20 for trials which the system has heuristically decided that the customer is likely happy with Appointment Reminder.  I’ve taken the liberty of marking it up a bit to direct you directly to the ROI-focused portion.

Appointment Reminder ROI Calculation in Email


If it calculated the ROI and they were getting an unhappy number, it instead sent them a second email, which was ‑‑ informally, it’s called “trial unsuccessful,” although that’s not actually in the email to them.

It basically says, “Hey, I’m a small businessman. I understand that sometimes I want to do something in a given month, and life just gets so freaking busy, and I’m not successful with getting that done this month. I totally understand how that might’ve happened to you, too. If it did, drop me an email. I’d be happy to extend your trial by another month.”

Maybe a quarter of people who get that email will write me back and say, “Hey, I wanted to use it. I just got busy. Can you extend my trial?”

Then that gives me an opportunity to both talk to them, figure out, if there was an issue with Appointment Reminder, how do I fix it such that they won’t need the second month. Also, the value of a trial that bounces out of your system is zero. The value of a trial that’s still in your system is nonzero.

To a first approximation, it’s always to your advantage just to give people the extra trial, especially if you don’t advertise that you’re doing that ‑‑ which, oops, I just told the podcast with 40,000 people, but none of you will pay me money so it’s fine.

Samuel:  [laughs]

Patrick:  Life‑cycle email is where it’s at.

Samuel:  I concur. Yeah.

Patrick:  One of the things that I see trip up folks a lot when they’re doing onboarding is sometimes the folks who are in charge of onboarding ‑‑ and this goes back to your organization point ‑‑ might not be the folks in charge of product.

Prior to and ‑‑ shoot, what’s the other dot‑IO company?, which gives people who are less technical the ability to set up these complicated life‑cycle email chains without having to necessarily dig into the code of the product to do it for themselves.

A lot of the marketing or customer success teams might have had a little difficulty hooking up life‑cycle email. In addition to life‑cycle email, what are tools that someone who might not have total control of the technical aspect, what could they do to influence the customer in their onboarding phase?

Samuel:  Hopefully I will be able to point to something that I am making sometime down the road.

Patrick:  Oh, please do.

Samuel:  In the meantime, I am a big fan of Jonathan Kim and what he’s doing at AppCues. That’s a piece of third‑party software that I would recommend people check out.

This maybe is cheating a little bit, but looking at live‑chat software, like Olark, just being present in there and understanding where your onboarding experience is breaking down, to me is really, really valuable. Most people who are faced with the onboarding dilemma in their company tend to be more like user experience or product, but just don’t really have the resources to pull it off.

Even if you’re just living in that world and can just say, “Guys, people think that they’re signing up for a bug to track their boyfriend’s car or whatever. Can we just make a copy change or something like that?”

Typically, those can get pushed through. It’s not like it’s a whole new feature, things like that. I would really recommend having live chat in that moment, because you can find out where one person’s going wrong and then make changes that affect the untold thousands of other people that will be following them.

The Fundamental Feedback Loop Of Low-Touch SaaS Companies

Patrick:  That is, by the way, just one of the generic secrets of running a SaaS company at scale. You do lots of stuff that doesn’t scale, and then use the stuff that doesn’t scale as fuel for the mass‑scalable approaches that affect the rest of the customer base that doesn’t talk to you.

Whether that’s automated email sequences, website copy changes, whether it informs your general marketing strategy, whether it drives changes to the product to make things easier to understand, et cetera.

Samuel:  Completely. One thing. I was speaking with Nick Francis from Help Scout the other day, and he made a really great point, which is, even if you’re using your own product or eating your own dog food, you’re not dog‑fooding your own onboarding experience. You’re not signing up for your product over and over every day.

A lot of times, you can be pushing out changes over the course of three weeks or three months that change A, B, and E, collide somehow and mess up your onboarding interface and the onboarding experience, but it’s a real blind spot for a lot of people. That’s another big reason that I really recommend getting a live‑chat box, like Olark or something like that, installed, just maintaining a presence there.

Patrick:  That’s actually a really good point. One of the exercises that I used to go to with consulting clients was, roughly on a quarterly basis, I would have people take out an actual, honest‑to‑God, physical credit card and sign up for the product. Not on the staging server, not through the API, not using autofilled details provided by the browser.  Buying the actual product from the actual production system. Put in a credit card, buy it, and see if everything works, and see if everything was optimally optimal.

I think sign‑up flows, purchasing flows, et cetera, they have a great tendency to go stale, because they get implemented by a junior Rails engineer in the first two months of the company, and since they <airquotes>”work”</airquotes> nobody ever looks at them again, until you break them in such a way that sales go to zero.

Samuel:  It’s crazy to me. I mean, you spend so much. Your team breaks their backs to create features that people would bother signing up for, and your marketing department is killing themselves trying to get more and more people to sign up for them, and then looking at just even getting people introduced to that or finding some sort of wins.

Then, even if they defy all the odds and get that far and they’re ready to pay you, like let’s put a ring on this, and then that experience breaks down. It’s super frustrating, so yeah. I completely agree.

Patrick:  You would not believe how many times you have really good, passionate product people in a room, these folks who would not tolerate a single comma out of place on a preferences screen, and you put them in front of the credit‑card form and ask them to buy their own product.

They put in their credit‑card number like it’s written on the credit card, with spaces, and the form blocks them from doing that.  “Your credit card number cannot contain spaces.”

Flag on the play. Why are we telling a human to do something that a computer can do better? Let’s fix this.

Samuel:  I completely agree.

Patrick:  That can be your takeaway. Just take out your credit card right now, try to buy whatever it is you sell.

Anyhow, one more thing on the topic of user onboarding, before we go in a new direction.

Samuel:  Sure.

Patrick:  I’d like to talk about one of my favorite tactics for improving the onboarding experience, even though it is a bit resource intensive: the product tour.  There’s the less‑invasive changes you can make to an onboarding process, like changing the marketing copy for the sign‑up to establish expectations better, changing the life‑cycle email copy or the life‑cycle email timing to rescue more of the people who might not have a hundred‑percent‑successful first‑run experience.

Or to not even rescue people but assist them in being more successful with the software, for folks whose decision‑making process just naturally doesn’t occur at their company in a five‑minute increment.

The more resource‑intensive thing that I do for my own products and do for a lot of clients is implementing a post‑sign‑up tour in the application. I was wondering if you could distill some experiences that you’ve had of doing that with clients, sighting it on the Internet, and dot‑dot‑dot.

[Patrick notes: Verbal ellipses?  Sure, why not.]

Samuel:  That is an area, philosophically, I have some issues with it, to be honest. I somewhat hesitate to anti-recommend it, given that you’ve presumably done a lot of experiments and had great success with it.

[Patrick notes: Careful Samuel and all you other guys!  I have actually done a lot of experiments about this particular thing, but because there is finite experimental bandwidth and because often we don’t have enough traffic post-signup to get results before the sun goes nova, I will often ship things based on my/the team’s best guess as to what user behavior is without actually testing them.  People assume I never do that because I talk about A/B testing so often, but that assumption is dangerous.

I also have to point out that, in the context of user tours, I am aware of ones which were major wins and ones which were “worthwhile to do but won’t make a difference to our overall numbers” and other ones which had to be yanked out of the product at a later date, for a few reasons.  Like any tactic, they’re heavily sensitive to the particulars of one’s own use case, one’s implementation ability, and uncontrollable vagaries.]

Samuel: I don’t want to pooh‑pooh it out of hand, but I think that there are a couple issues of going down the road of basically what you might call a tool‑tip tour or something like that, where you’re spotlighting different parts of the interface or things like that.

[Patrick notes: Tooltip tours are in vogue because they’re comparatively easy to implement, relative to all the other ways of doing a tour experience.  Unless you put a lot of thought into what you’re actually getting people to do with the tooltips, they are not value creating.  I see many more tooltip tours that are vexatious than ones which, like the Blizzard example earlier, excite the user while simultaneously instructing them on how to get started with saving the world from orcs and poorly managed projects.]

Samuel: One, going back to my initial point of tacking onboarding, stapling onboarding on at the very end of the product cycle, a lot of times, I think people use it to literally cover up user‑interface issues that they have. A lot of times, people will use it as a crutch to say ‑‑ I’ve literally seen a button that says, “Create project,” and there’s a tool tip that points to it that says, “Click this to create a project.”

There’s a really wonderful Flickr group that Jason Fried from Basecamp started a long, long time ago, called Signs on Signs, where he takes pictures, or he did at the time, took pictures of a sign in a library that says, “Please be quiet,” and then there’s a sign attached to that sign that points to it that says, “Look at this sign,” or “Attention, please be quiet,” or things like that.

A lot of times, if your interface is messed up, adding more to it that literally points to the parts that are confusing is not something that I would recommend. I would really recommend just fixing it to begin with. There’s that.

I think another issue with tool‑tip tours and things along those lines ‑‑ I’ve seen your Appointment Reminder tour, and it does not qualify for this critique, but a lot of others, I think, do ‑‑ is that they’re very focused on introducing people to features or introducing people to parts of the interface more so than they are at guiding people to actually accomplish something.

A lot of times, you’ll see six tool tips that all show up at the same time, and it says, “Click here to do this,” or “When you need to do this, go over here,” things like that. They’re not actually walking you through getting something accomplished. They’re just basically asking you to remember where to go when you’re faced with a situation in which that button would be useful.

[Patrick notes: This is, indeed, a failure case.  I pointed it out at one consulting client and asked who to talk to to get it fixed, and was told that nobody “owned” the tour, which is another failure case.  They thankfully came to the conclusion that that was suboptimal and tasked an engineer on it.]

Samuel:A lot of times, when I see tool‑tip tours done poorly, it’s because they ask people to learn by remembering and not learn by doing, which is not the case with yours.

You focus on one thing at a time, and the entire thing is about let’s just guide you through, hold you by the hand, and get your first appointment scheduled, and understand what it’s like, what kind of phone call you’re going to be getting when that happens, and things like that. I would say, not in your case, but a lot of times, that can be an issue.

Then, one other thing to really look for with tool‑tip tours is that they can be, how would I put it, sequentially fragile. There are a lot of times where, if your plan is to get people through maybe a 15‑step tool‑tip tour workflow, but if there’s an issue where somebody thinks they’re supposed to click on one button, but they’re actually supposed to click “OK” within the tool tip or something like that.

All of a sudden, it disappears. Getting back into it, do you start back at step one, or step seven, where you left off? Things like that. Using that as a highly linear narrative device can go sideways really quickly. That’s another thing to be concerned about.

Patrick:  Just in terms of building stuff, when I build out tours, partly because I’m aware of the fact of sequential fragility, there is generally an easily accessible bailout button for the tour at all stages of the process. When somebody bails out, it should probably keep them in a consistent state that doesn’t totally hose their account.

That’s not universal on all tours. Either you give them a wiped‑clean account, or, ideally, you would just give them full control over the interface, but keep the state of the account as whatever they were just seeing, to not have the leaky abstraction of, OK, the tour‑mode stuff was actually just fake objects created in a fake state that we displayed to you, but it was actually a lie, which is how a lot of them are implemented.

[Patrick notes: Why?  Well, since the tour will often involve running through the core use case for the application, it will necessarily plug very deeply into the core workflow / business logic / etc.  That is a non-starter at some places, so rather than changing the core workflow / business logic to accommodate a special tour state, they just fake the existence of the happy path of the workflow with smoke and mirrors.  Appointment Reminder’s tour, by comparison, is done with maximum verisimilitude (and a whole lot of elbow grease, which involves some hackery deep in the bowels of the application, like a special case in our outgoing phone system which detects Hollywood phone numbers of the (555) 555-XXX format, short-circuits actually attempting to call them as you’re directed to do in the tour, and then simulates the results of interaction by a human with the phone call that would have happened but for the short-circuit).

Samuel:  We almost think about, I can almost guarantee that you, myself, and every listener for this podcast has downloaded some sort of mobile app that’s been greeted with a series of welcome screens, and gone swipe‑swipe‑swipe‑swipe‑swipe to just get through it to get to the actual thing. Then not know what was covered in the thing that you just skipped over and not really know how to proceed from there.

I would say any kind of intro or tour or anything, create it around helping people make progress and move forward, but don’t absolutely depend on that.

I would design for a null state, where, basically, OK, is it still going to make sense and we’re still going to be communicating the most important things when the dashboard is empty or things along those lines, and not really count on the hand‑holding as your only source of orientation and motivation.

Put Better Default States In Your UI For New Accounts

Patrick:  Definitely. Speaking of null states, often, if you’re in a project‑management app and you have no projects, the first screen will say, “You have no projects.”

Samuel:  Right, kind of accusatory.

Patrick:  I would always say, rather than saying, “You have no projects,” OK, if you’re in development mode, sure. Whatever. Put that up there, like zero of zero results returned for projects.

When you’re shipping that to actual customers, just a quick, one‑line if statement, replace it with a, “Here’s how you can get started creating a project.” Then most people just put an arrow that points to the “add new project” button or they say, “Click above on ‘add new project’ to get started.” Rather than that, I would give a little bit of goal‑oriented instruction.

Samuel:  100 percent. The only thing I would add to that is not only prompting them to fill it up with something, but also, “Here’s the value of doing that to begin with.”

Patrick:  Exactly.

Samuel:  “You don’t have any projects yet, but this is where you’ll be able to see which ones are at the biggest risk of not shipping on time. Click here to add your first one.” Something like that, to me, that’s my general recommendation. Because just changing it from, “You don’t have projects” to, “Click this button to create one,” it’s not making it a meaningful action.

That’s a term that I use over and over again when I’m looking at onboarding experiences. Do I even know what I’m doing, or do I even care about what I’m doing? Can you help me at least get to one, if not both, of those?

Patrick:  One of the things I really like is when you provide people with a vision of the future they’ll have if they’re using the software, ideally a vision more focused on them than just focused on your software. For example, I think Baremetrics does this very well.

They’re a company that slurps data out of your Stripe account and presents a variety of metrics for you. My recollection is, when you start using Baremetrics, in the pre‑slurp state, it’s got nothing to show you. Rather than showing you, “We’ve got nothing to show you,” they show a grayed‑out version of their real stats.

It’s like, “Wow. You can see all of this stuff, except for your business, if you just click the Slurp button and give us however much time it takes to do the Slurp out of your account thing.”

Samuel:  That can be like a PNG. You don’t have to build in a feature that has all this mocked out or whatever. Yeah.

Patrick:  Is that how it’s pronounced, PNG?

Samuel:  I always call it “ping.” I don’t know. Do you just say the letters?

Patrick:  OK. Here it’s PNG, but then again, I’ve lived in Japan for the last 10 years.

Samuel:  Do you call it a J‑P‑E‑G, or a J‑peg?

Patrick:  Wow, I don’t know. It’s different than “jiff.”

Samuel:  [laughs]

Patrick:  “Jiff,” and then, I don’t know. It’s been too long since I’ve been in a Japanese office, despite being here for 10 years.

Samuel:  That’s something to celebrate, I guess.

Patrick:  Oh, yes. Every day that I’m not a seller man is one more day of actual life. Yay.

Samuel:  [laughs]

How Samuel Set Out To Become, And Became, The “User Onboarding Guy”

Patrick:  We covered onboarding tours a little bit. We covered a bit of the design of UX when someone is getting started with a product to feel like they can get more success and not just have to do meaningless busywork until they get to the good stuff.

We covered a bit of life‑cycle emails and whatnot. I’d be a terrible businessman if I did not mention the fact that if you go to, there’s a course from me teaching you how to do that. A little plug.

I mentioned you have the blog at Maybe blog isn’t the right thing. A site, where you go into this customer‑onboarding thing in a bit more detail than people typically do in a blog post, and you do tear‑downs of folks’ user‑onboarding experiences, the pre‑sign‑up process, the sign‑up process, the post‑sign‑up process. You dig into what emails they send.

Samuel:  Slideshows.

Patrick:  I thought this was really, really smart back when I first got to know you, in that you very clearly said that, “OK, there’s a million UX designers out there. I’m going to be the one that just takes user onboarding and owns that.”

How’d that work out for you? I know you wrote a book on it later, and we can talk about that in a moment, too, but I presume you also do a bit of consulting and whatnot?

Samuel:  I would say that going after a niche has been a highly lucrative decision on my part. The consulting and the book sales have both been really strong. Really, it’s been a pretty fun ride.

Patrick:  Awesome. Mind if I rewind the tape to a little bit before the fun ride and whatnot? What got you into this in the first place?

Samuel:  Interestingly, you were saying that I wrote the book later, but the book was actually what got me into it to begin with. I was a user‑experience generalist for years. I was at this weird in‑between place.

Looking at things like conversion‑rate optimization is not typically something that’s part of the UX wheelhouse, but I was always really, really interested in ‑‑ I think that you’re actually probably a good representative of this mentality of, “Let’s find out what the problems with this flow are and empirically demonstrate that we have improved upon it.”

To me, I guess, the job to be done of what somebody hires a UX consultant for is typically not that. I really struggled with that for a long time, looking at a competing set of passions that were between qualitative and quantitative, so user experience versus conversion‑rate optimization, and then the ever‑contentious term “growth hacking.” [laughs] Was that as well.

They all embody the same thing, which is aligning your success around your users being successful and paying attention to whether that’s happening and where that’s breaking down, and then being able to measure the impact of the way that you’ve improved upon it.

Let’s see. Where do I even start here? I decided to write a book, because I wanted to take a product to market, but I didn’t want to go six months to a year into developing some sort of a SaaS product and then take it on the chin with a bunch of rookie mistakes about just how to price something or how to create a landing page that sells it or how to have a product launch or things like that.

I thought, instead, “I’ll just have a really constrained product, in the form of an eBook. Surely I can write that in a couple weeks.” Which was, spoiler alert, not the case. [laughs] Then be able to just get my toe in the water by getting something out there and just go through that experience and learn from it.

Very naively, I put up a landing page for the book, and I titled it “Customer Growth.” It was basically all about “Grow your customer by helping your customers grow.” That was the one‑liner for it.

A lot of issues there. One, that’s not a thing that people explicitly care about. It’s more of I would have to write a very philosophical thought piece on why people should care about it to begin with, and convince them of the value of the subject, before even convincing them of the value of the book that covered said subject.

The best way I can describe it is nobody was sitting down and saying, “You know what? We have this problem with this thing that I’ve never heard of. I wonder if there’s a book out there on it.”

Patrick:  Amen. If you have to convince the market that they’ve got a problem, that may be an option, but you need to have a previous success behind you or a war chest or maybe VC investment.  [Patrick notes: But that wouldn’t be my first choice for any of those folks, either!  Pick the screamingly obviously high-impact problems first.  Plenty are left unsolved.]

For those of us who are just getting into business for ourselves, you should not be targeting problems that you have to explain to people that they have. You should be targeting problems that they know they have, that when they wake up in the morning, it’s one of the one or two things that is keeping them up at night.

That’s a mixed metaphor. One of their one or two top hair‑on‑fire problems for today. Virtually, every mistake I’ve made in business in terms of product selection has been not targeting those, but that’s another podcast entirely.

[Patrick notes: I do not regret Bingo Card Creator or Appointment Reminder, but if I got a do-over, I’d pick something higher saliency in a tightly defined commercial niche that I would like exploring and knew I had “unfair” advantages in.  That would probably be selling software, though there’s an interesting meta question in whether I would be any good at selling software but for the experience of BCC/AR and taking advantage of the doors they opened.]

Samuel:  The one thing that I did right, though, was fully committing to getting the book out before I started. As I guess we’ll get to in a second, there were definitely some hard stretches in the middle, where people would ask me how the book was coming or something like that.

I would very truthfully tell them that I was glad that I didn’t know how hard it would be before I started or I never would’ve started. I was equally true about how hard it was, and also equally true about how glad I was that that wasn’t the case, because I was fully committed when I started and I was just absolutely going to see it through.

I put up the landing page, wasn’t really sure how I was going to go about writing the book or establish my expertise. Obviously, Nathan Barry’s information out there was a big source of help and inspiration. He’s pretty email‑list‑centric. [Patrick notes: For a reason.  Email is the secret weapon of low-touch marketing like airplanes are the secret weapon of modern militaries.  It’s not a secret at all — if you don’t have them, you’re giving up a tremendous advantage.]  I was like, “I guess I should start an email list and see. Maybe I could get up to hundreds of subscribers or something.”

Literally starting at zero, not even having an email list, that was like, “OK, I’ll put up a landing page, get people to sign up.” I had to figure out how were people even going to be getting to the landing page. This was the definition of a cold start, I guess you could say.

As a UX consultant, generalist, at that time, a lot of times I would do something very much like the tear‑downs that are on the User Onboard site right now. I was like, “Man. I’m already writing the book,” which is really hard, because I’m a painfully, painfully slow writer, which is another issue. I was like, “Am I going to try to guest‑post on other blogs?”  [Patrick notes: I did a guest-post tour when I launched my Lifecycle Emails course.  I cannot recommend it for generating sales, though for someone with less of a platform it might make sense for audience building.  I think I wrote six good essays on the topic for other people, and seen in hindsight, six good essays for my own purposes would be better than six good essays for other’s sites.  I think I have about $3k of attributable sales as a result of that, and even if you double the number to count ones which I missed due to the vagaries of tracking, I’d rather have the essays than the money, both for the initial results and for the residual portfolio value.]

Samuel: That’s even more writing on top of writing the book. I’ve heard sometimes you just basically get a really deemphasized link in the byline, and it results in three people signing up. I can’t spend 15 hours on a blog post and hope that that happens.

I was like, “Man, if only I could share these tear‑downs that I’ve been doing.” I’d feel weird, because they were commissioned and paid for and not really owned by me. I was like, “Oh, I can just pick a company and not ask them to pay me for it and just do whatever I want with that.”

I just picked a company at random, which was OpenTable, and recorded the experience, and I was like, it just didn’t really come out right. Their onboarding experience is very confusing, or nonexistent, almost.

I had to scrap that one. It was getting kind of late into the day, and I was like, “Maybe I will do this. Maybe I won’t. All right, screw it. I’ll try one other company.” That company, once again, just completely at random, was LessAccounting. Your listeners would probably be ‑‑ it’s kind of in a similar space, I guess.

Patrick:  LessAccounting, for those of you who don’t know it, is a bootstrapped software company that is basically a stripped‑down competitor to Quicken.

Samuel:  There you go, yeah. Also, they seem to really maintain a presence in the bootstrapper community and stuff. That’s what I was referring to.

Patrick:  Oh, yeah. It’s a funny little bit of community inside baseball, I think. Sometimes we overestimate how much folks know about the <airquotes>”scene.”</airquotes> If you hang out at Amy Hoy’s conferences and go out to BaconBiz every year or go out to MicroConf every year, then you’ve run into the LessAccounting guys and you know who they are.  (I would bet you that the average Quicken-using accountant or bookkeeper in Normal-Bloomington does not.)

[Patrick notes: I think this is observation is widely applicable.  Our community is simultaneously bigger than we realize and yet smaller than we think it is.]

Patrick: I happen to know that for a lot of the world it’s the first time they’re hearing about it right now. By the way, they’re good software. You should use them. A plug. Yeah.

Samuel:  Actually, speaking of which, that was back in November of 2013, so not even a year ago, I was one of the people who didn’t know who they were. I’d known them just because they’d been around for several years, didn’t really know that they were highly involved in the bootstrapper “scene,” or whatever that might be.

I was like, “All right, they seem friendly enough, at the very least,” went through, created the tear‑down, put it up on SlideShare.

I think it was the end of the day, and SlideShare messed up the formatting and the aspect ratio. I just wasn’t really happy with the product, but I was like, “I’ll just go home and talk to my wife, and tell her I blew another day while I’m trying to get this book out.”

I posted it, and the next morning I got an email from one of the co‑founders of LessAccounting. I see it in my inbox as the subject line and see who the email’s from, and I was just like, “Oh, no.” I was sure that when I clicked on it, it was going to be him coming and being like, “Oh, thanks a lot for airing our dirty laundry and pointing out how our onboarding experience isn’t working, and who even are you?”

It turned out to be completely the opposite, that he was like, “Thank you so much for pointing all these things out. We already made a bunch of the changes that you recommended. I see you’re writing a book. How can I help promote it?”

It was just total night and day from what I was expecting as a worst‑case scenario, really just a super‑supporting response. He wound up featuring it on the LessAccounting blog, and that was really the very first thing that I did that got a decent amount of traffic and a decent amount of sign‑ups for the email list and things like that.

Patrick:  Awesome. I think this strategy is very generalizable, and in fact, folks have done it in a lot of circumstances, and it often works well. Dustin Curtis is a designer. He basically made his name by taking a few big Fortune 500 company websites, doing redesigns on them.

I might have issues with that particular work product, but that’s neither here nor there. 37Signals, back in the day before anybody knew who they were [Patrick notes: 2002 — Basecamp and Rails were in 2006, I think?], just did unsolicited redesigns of the FedEx application and said, “Here’s all the mistakes that FedEx is making.”

It wasn’t actually FedEx, because I think they were probably worried about getting sued, but it was a purple‑looking delivery company.

[Patrick notes: I apparently misremembered this — it appears to actually mention FedEx.  Their BetterBank was a better example of a generically attacked problem.  Though I’m fairly certain that somebody has done a You All Know Whose Website I’m Talking About redesign without actually using the trademark — can’t recall who at the moment.]

Samuel:  [laughs]

Patrick:  They just had purpledelivery.pdf, with the redesigned with the purple delivery company’s Web app, which featured package tracking as a first‑class citizen rather than the 15th thing that you wanted to use.

Especially, if you do this for other people who are closer to the us‑es of the world than the Fortune 500s of the world, folks, often, the first impulse will not be send a cease and desist or be very annoyed at you. It’s like, “Hey.” Folks describe me personally as Internet famous, which is a funny, funny word.

To this day, my heart lights up anytime someone shows any bit of attention to one of my products. If you want to screen‑grab everything and show what I’m doing wrong with the world or how my pixels are out of place, I won’t think, “Oh, he’s insulting my pixels.” I’ll think, “Yes! Totally! Someone noticed me! That’s awesome!”

Samuel:  Just to briefly touch on the whole unsolicited‑redesign thing, I can see how I’m in a similar space as that, but personally, I’m really not a huge fan of unsolicited redesigns as a thing, largely because it’s a very surface problem.

You’re basically saying, a lot of times when you see them, especially on Dribble or things like that, it’s, “I didn’t think this was pretty, so I made it prettier,” where you don’t really know what’s working that well, what’s not, what kind of constraints the team is faced with. You’re probably not even necessarily the primary audience that the product was intended for.

For a lot of reasons, when I’m creating a tear‑down, I try to be very, very conscious of the fact that there are real people who had made this under real pressure, and it was to serve a job that may or may not be something that was…I’ll literally go through the sign‑up process to create the tear‑down.

It’s not like I’m even really genuinely trying to make it work for me in that moment. There’s already that, and maybe I’m not even a key audience member at any point in time. There are those issues.

Then also, I’ve actually had conversation with people, where maybe I’ll say, “Yeah, deciding to do that made me scratch my head,” or “That doesn’t really conform to the “best practices” or whatever that might be.” The design team will be like, “Yeah. Yeah, we hated it, too, but it’s working really well.”

Not having visibility into the conversion funnel or whatever that might be, and then also just not knowing about what kind of internal pressures they’re dealing with in the office, I really, really try to say, basically, “Objectively, these are what our best practices are, or not, considered to be generally within the community.

Then also, anecdotally, as a user, I was legitimately confused when I went through this.” Really, really distance myself from saying, “This is objectively wrong” or anything like that.

Patrick:  Right. I think that is a great attitude to have about it in general, and probably a karma‑maximizing attitude, if you’re hoping to borrow an audience as a result of publishing these things.

Also, I think, as somebody who has worked in a lot of companies and seen the sausage get made, it absolutely tracks with the internal human/political/resource‑based constraints on why something might not be totally optimal. There’s a lot of times where, heck, I’ll own it. I won’t blame the client, I’ll blame myself.

There are something that I have shipped where I could point to you X, Y, and Z decisions of the things that shipped today and say, “I hate X. I hate Y. I hate Z.” I had 100 points of awesomeness in that engagement, where awesome is an arbitrary resource. Fixing X would’ve required 20 points of awesomeness. We just had other things to spend awesomeness on, so we just shipped it out the door.

Samuel:  There you go.

Patrick:  Often, a particular team or person in the organization just did not want to budge on Y, and they had been really cooperating on some other part, and so you trade tit for tat. That happens all the time in real life.

Anyhow, going back to the book for a moment.

Samuel:  Yes.

Patrick:  You release some of these tear‑downs, and both the folks who were, quote‑unquote ‑‑ I hate that word, tear‑down, by the way. I’d like to say, “build‑up.”

Samuel:  [laughs]

Patrick:  You released some of this feedback on people’s onboarding processes. Some of the folks who were featured in this feedback found it really, really useful to them, like the guys at LessAccounting.

They spread all over the Internets to these people’s preexisting networks, as they said, “Hey, someone has taken this interest in our business and given us really useful feedback. We could read the writeup.” Thus, you got, what, a few hundred or a few thousand people to subscribe to your email newsletter?

Samuel:  Specifically, in the early, early days, it was maybe a couple hundred, when the book was still called “Customer Growth” and people probably didn’t really know what they were signing up to get.

Through the success of the tear‑downs ‑‑ so I did the LessAccounting one, and I was like, “Well, that went well. I should do another one.” Then I did Basecamp, and then that one got shared a lot. I think that wound up Designer News, the front page, for quite a while.

At that point, it was suggested to me that I stop using SlideShare and instead create a site that’s dedicated to those, where I could control the conversion timing, the asks, basically. I also just wasn’t really a super‑huge fan of the user experience on SlideShare, either.

Patrick:  Can I circle this point and star it, guys? There’s a lot of folks who put their best work on 3rd party sites. In my case, some of my best work is on Hacker News. In other folks’ cases, it might be on Dribbble, on Twitter, on Facebook, on Medium, et cetera, on GitHub, a major one for the developer community.

For things that are central to your career, building up a public portfolio, you really want to be able to control all aspects about that, both how the work is presented to people who will be future decision‑makers about your career and what you emphasize about it.

If you embed something in GitHub, you’re going to end up with a very GitHub‑y experience for that, regardless of what it is. The way that people consume that artifact that you have put on GitHub will be a very GitHub‑focused consumption experience rather than a you‑focused consumption experience or a them‑focused consumption experience.

Samuel:  Yes.

Patrick:  I would strongly encourage, from both a UX point of view and a “maximizing the future value of your passport to your future self” point of view, that you put your best work on your own darn website. Think about wrapper‑type issues, like:

Should I put a logo on it?  [Patrick notes: If it represents more than a work-week of effort from you you’d be crazy to not spring $100 or $200 for a logo.  I strongly feel like the logo, the dedicated web presence, and the non-zero effort put into documentation for A/Bingo were reasons it redounded to my favor.]

What sort of asks should I be having on the page? Whether that’s asking someone for their email address, or, for those of you who might not be selling a book but might be selling freelance or consulting services, maybe you ask them to send you an email and ask for a quote, or “Send me an email. I’d love to have a Skype chat about this if it interested you.” You convert them and do a request for a quote or something on the Skype chat, et cetera.

Yes, asterisk, asterisk, asterisk. You should absolutely have it on your own website. Which is, by the way, to this day, why ‑‑ well, aside from Hacker News ‑‑ almost all of my writing is on either my own blog or on my other sites. Most of the things that I produce even for free, the canonical source for it is on my Web presence rather than other people’s Web presences.

I love GitHub, don’t get me wrong. In the absence of contracts that I can neither confirm nor deny exist, my job is not to make GitHub money. It is to make my family and I money while producing stuff of value to society, so I tend to keep that on my own Web presence rather than theirs.

Samuel:  I completely concur.

Patrick:  Anyhow. You did the smart thing. You moved some of the stuff from SlideShare. By the way, you can still host stuff on SlideShare. Just put the embed in the write‑up in your own site, and then it will collect the links and citations that people are looking for.

Samuel:  There you go.

Patrick:  That’s what I do for all my presentations, by the way. You’ve created a Web presence for this.

Samuel:  Right. I guess, at that point, it was very clear, like, “Oh, user onboarding is the thing that I should be talking about, not a component of this very vague thing that I want to talk about.” I guess a good litmus test is, if you’re going to be writing an eBook, is it something that people would find helpful?

Are you doing it to help people, or are you doing it to preach at people? I made the shift from preaching at to trying to help when I made the shift from writing a book on customer growth to writing a book on user onboarding.

At that point, too, that’s when I bought the User Onboard domain, because was already taken, created the site in a weekend, and transitioned the slides over to that, and then just kept coming out with new tear‑downs. Basically, it was like one a week at the beginning. At that point, the email‑list sign‑ups went from a couple hundred to a couple thousand, and then even further from there.

Nicely enough, the book’s already been out. I didn’t really launch it very well. I set out to do it so I could learn rookie mistakes, and boy, did I make some. A really nice thing about it, too, is having an ongoing Web property, there’s some repeatability to it, I guess you could say.

Every time I put out a new tear‑down, I know that’s going to result in X‑hundred more newsletter sign‑ups and Y more book purchases, or whatever that might be. It’s been nice to have as just an ongoing asset, for sure.

Patrick:  This is something that I really like about this emerging publishing model.  In the old model, you contract with a publisher, you write a manuscript, they pass it through a few other professionals, put it out, and it goes to bookshelves across the country, it sells a thousand copies. Then is available for back‑ordering from anyone who wants it, but nobody will because they are very launch‑focused.

When we control the assets and we control the marketing plan, we cannot just have a launch‑centric approach to the value we’re creating. Most of the value’s not created just by launching something. It’s created by building something of value and then figuring out what the right recipe of things is to get people to be exposed to that thing of value, and you can tweak and adjust over time.

Even in the ‑‑ going to use a word I hate ‑‑ information marketing space, a lot of it is very launch‑centric. I think that’s partially because a lot of folks who are broadly in that space don’t produce things of value, and then after the market figures out that the new thing that has been produced is not of value, sales go to zilch. For people who generally produce books, software, et cetera, of value, they often have a substantial long‑tail component to the sales.

If you follow the usual email/launch‑centric approach, where you’re collecting email addresses, the thing launches, you get 10,000 people or whatever with an offer to buy the thing in the first two weeks, then you typically will get a spike at launch day or in the first two weeks.

There is residual value to having that, both in terms of quantifiable money in your Stripe account, residual value, and also in the fact that you can point, in conversations with people three years from now, you’ll still be the guy who quite literally wrote the book on user onboarding when you’re talking to potential consulting clients. Or if you have a new book, it will be by the author of the bestselling book on user onboarding, dot‑dot‑dot.

Samuel:  For sure.

Patrick:  I wrote one book, by a nontraditional publisher, Hyperink, called “Sell More Software,” on Amazon. One of the things that traditional publishers tell you is, “You should write a book. Not because making a book will make you money, because it won’t, but because having written a book on something is great for consulting.”

I always thought that was self‑serving BS.  I still believe it is self‑serving when the publisher tells you that, but it is not totally BS. There are some clients who really, really connect to having a book available on Amazon. I happen to know that there’s a few copies of mine boxing around at Fortune 500 companies at the VP level, which surprised the heck out of me.

Samuel:  That’s awesome.

Patrick:  I also produced a video course two years ago about life‑cycle emails, and that just has, as I write more stuff for my email list and people get added to the email list, and then eventually, 30 days later, they get a brief blurb about the life‑cycle email course in one of the emails that on‑boards people onto my email list. It produces just a nice, happy Chinese water torture of sales over time.

Samuel:  [laughs]

Patrick:  I don’t do any active promotion for it, and it’s been out for two years. It’s probably still made $10,000 this year, [Patrick notes: for values of $10,000 equal to $7,500] which is a pretty nice place to be for not doing additional work. Would you mind if I asked, how many people do you have subscribed to your email list these days?

Overnight Success, Isn’t.

Samuel:  A little bit over 11,000.

Patrick:  That’s actually right about where I am, at about 12‑ish or so. One of the things that I frequently get when I’m talking to people about making stuff and then selling it, about the Internet, is that folks have unrealistic expectations about what “Internet fame” is.

11,000 subscribers to an email list is not a number that you have to be an international celebrity/Internet man of mystery to hit.

You and I are both, total mortals, we take the not too difficult to comprehend tactic, of doing the thing we were good at in public, and saying, “If you want more of this stuff, give me your email address, I am hooking that up to an easily available mail provider, which costs less than a cable bill, a month, and then, just continuing that for a year, in your case, or 10 years in mine.”

Oh boy! Can’t believe that I have been in the industry for 10 years now, blows my mind.

Samuel:  That’s also worth noting too, I have been in the industry for 10 years now, too. I would not consider myself to be an overnight success by any meansI think a lot of whatever success that I have found in the last year has come from paying attention to things that you’re writing and putting out there about like, “Yes, you can do this.”

You don’t have to just be toiling in obscurity. You can do a very reasonable amount of effort put into creating something that people find valuable and be able to benefit from that. It’s very, very achievable.

Patrick:  I totally got everything that you just said there. One of the things about overnight success that always staggers me is that Peldi, the gentleman behind Balsamiq Mockups. Balsamiq Mockups have the first like first year of sales of, virtual of any software company that I know about, aside from one that $500 million obviously injected into in year one at $300 million in adds and sold $200 million in software.

The absolutely meteoric graph that first year, Peldi had a great presentation where he showed the meteoric graph. That’s what it looks like from the outside. From the inside, and he shows a graph that expands 28 years in the past for as long as they’ve been in software game in various companies, where obviously, for the first 27 years or so was zero software sales.

Then, overnight success. Overnight success didn’t take overnight. It took 27 years. But then people selectively edited that down when they’re talking about it.

Yeah, very important to point out that overnight I was just doing, I call it the grind, just get up, day to day, bang out some code, write some emails, try some experiments. For, I’d say, maybe after four or five years of doing it, a thousand people knew who I was.

Samuel:  I like the metaphor of pounding the rock. Totally in agreement. I think it’s also a thing that you don’t have to be that long tail of toiling in obscurity, like I was mentioning it before. I could have been doing things so much smarter, so much earlier.

It wasn’t like I’m a radically more experienced or smarter designer now. I just had to do an inch more smartness around being able to distribute that information, I guess.

Patrick:  Kind of like operationalizing it, in a business sense. Oh, that just sounded like a management consultant.

Samuel:  [laughs]

Patrick:  Sorry, guys. Anyhow. I totally agree on that. That’s one of the reasons I have my blog, one of the reasons why I really like the openness in our industry, from folks like Paul Graham, Joel Spolsky, all the way down to folks like me, Nathan Barry, et cetera, where you don’t have to make all the mistakes to learn all the stuff anymore. It’s great. I’ve made so many mistakes so you don’t have to.

Samuel:  Hiten Shah has been another person I would certainly put up in that pantheon of helpfulness as well.  [Patrick notes: Hiten is writing again, after a long hiatus.  Check out his blog.]

Patrick:  I’ve learned many things from Hiten over the years. Man, could do an entire podcast just about intellectual influences for stuff that I do on a day‑to‑day basis, probably get up to 200 names.

Samuel:  You should. I would absolutely listen to that.

Patrick:  Putting it on my list. Intellectual influences. Sorry, just writing that down. Anyhow.

[Patrick notes: I’m serious about doing this, but it would be a lot of work and might not be interesting for folks who don’t play inside baseball, so tell me if it is interesting to you.]

Samuel:  [laughs]

Patrick:  I’ve learned this over time. I think one little asterisk that I put, often, when talking about the topic of learning from other folks is that you should generally balance learning from other folks with doing for yourself. Just because I know a lot of folks who, they’re working the day job, the day job’s taking up a lot of their creativity/mental energy.

They listen to podcasts. They read the blog posts. They even go to conferences, watch the presentations. It’s like they’re perpetually in training for the championship bout that never comes.

Samuel:  Right. That totally resonates with me. For a very long time ‑‑ if this was school, I would be so ready to take the test right now, but nobody’s sliding that test onto my desk or whatever. Very much felt that way, for sure.

Patrick:  I felt like that myself for many years prior to actually starting a business. I would encourage all of you guys ‑‑ take the plunge. Doesn’t have to be a life‑changing, burn‑the‑ships decision or anything. Just start a blog, if you don’t already have a blog.

If you got a blog, start an email list. If you’ve already got an email list, put a stake in the ground on a product that you want to get out and make the progress towards actually getting that out there. Launch it. Everything about life gets better after shipping things.

Samuel:  I think it’s important to emphasize, too ‑‑ this is probably actually where I picked it up is from you. Don’t discount the expertise that you already have.

Patrick:  Yes.

Samuel:  Just because you’re not a nationally recognized name or whatever that might be, if people are paying you to do something, then that is enough expertise for you to be able to disseminate that information in exchange for an email sign‑up or something along those lines.

Patrick:  Getting somebody’s email address, you are not proposing marriage. It’s just like, “Hey. If, in this one interaction we’ve had together, you think that it might be useful possibly having a relationship with me in the future and hearing even more valuable stuff, here’s an easy way to accomplish that.”

I know many, many geeks have an anti‑email bias. I did, too, as an anti‑spam researcher, because I only saw the absolute worst of email for years and years. A lot of folks are not offended by being asked for an email address. Even if 50 percent of the audience is like, “I never give my email to anybody,” you can read the blog at your own pace. That’s fine. The other 50 percent, though, their email, those are quite valuable to have.

Samuel:  I think that it’s an important distinction to make, too. I certainly completely concur with that, but at the same time, I wouldn’t say just start a blog to have a blog or start a podcast to have a podcast. The lens that I would use on that is just start contributing things that people find useful, and whatever the delivery mechanism is.

Patrick:  Yes. Absolutely true. I totally agree with that. I also think that the effectiveness for these sort of things, both in terms of reaching an audience and creating a value to that audience and in terms of helping out your career/business interests, goes way the heck up once you find that thing that you’re good at.

I started my blog the same week that Bingo Card Creator shipped [Patrick notes: July 1st, 2006, crikey I’m old], and the original idea was I’m just going to write down stuff about what I did for Bingo Card Creator.

That blog really only hit its stride maybe three or four years later, after I figured out the thing that I had a comparative advantage against versus the rest of the Internet, that little, itty‑bitty intersection between engineering and marketing, and started writing about that a lot. It resonated with a lot of people. It actually, knock on wood, changed other people’s businesses for the better in a lot of cases.

The more I wrote about that, and the more I wrote about it in a particular format that you might be familiar with if you’ve followed my blog or podcast, et cetera, for a while, because I’ve got a style ‑‑ that style worked.

Whereas a 500‑word update on, “Here’s what I did for SQL optimization today,” there’s probably a post or two on my blog about that, which five people have read and nobody found tremendously valuable, and there’s much better SQL optimizers elsewhere on the Internet.

Samuel:  I guess, when you talk about going after a niche, whatever that might be, looking at the gaps in between really big things. For me, marketing and product, there’s user onboarding, or for you, between engineering and marketing. A lot of times, I think that’s a place that you would look.

Patrick:  That is good. It’s going to sound a little weird, but you can have a very happy, fulfilling, rewarding career just by being the spackle between different teams in an organization.

Samuel:  That’s why they have consultants. Otherwise they would have already staffed for it.

Patrick:  [laughs]

Find The “Known Groan” To Find A Profitably Exploitable Niche

Samuel:  If I could make a recommendation on niche‑finding…

Patrick:  Sure.

Samuel:  Actually, I was preparing for this, and I was like, “What do I do that I didn’t get from Patrick McKenzie [laughs] , that’s not already out there?”

One thing that I haven’t seen anybody really write about ‑‑ the people that I recommended it to have found it helpful ‑‑ is to pick a term, almost like if you wanted to compete for SEO or whatever, just like the presence of mind.

I very quickly realized that there was not the user onboarding guy. Specifically, that being a term that A, people had an emotional reaction to, like, “Ugh, onboarding.” I call it the “known groan,” when people are like, “Ugh. Yeah, ours sucks.”

Patrick:  [laughs]

Samuel:  Looking for something that they actually have some sort of emotional, I guess, revulsion to. That indicates that it’s a hair‑on‑fire need, like you were mentioning before.

Also, it’s a specific phrase, term, or something like that. Really, the actionable part of this recommendation is once you’ve…Maybe if you have a few different ones that you think you might want to try, set up Google Alerts for them.

If you’re getting flooded with stuff, you’re probably too broad. If you’re not getting anything, then people don’t care about it. If you’re getting three to four or five a day of new articles that are coming out, that have that term in the subject line, that’s a very strong indication that you can own that niche and it’s a niche worth owning.

You can be the person who comments. Every time somebody sees a user onboarding post, I should have some sort of presence there. I can leave a comment, or I can link to it through the user onboarding Twitter account or something. I have that Google Alert, and it’s been just tremendously valuable for me. That’s my tip of the week. [laughs]

Patrick:  I think that is a useful tip. One of the things that I struggled with in my own consultancy/larger business interests that there wasn’t really…I had an idea for what I was good at, but I don’t really have a word for it. I bounce around a little bit.

Other people created words for that sort of thing. “Growth Hacker” is a created word to identify some niche for a particular type of individual, and then allow them to go after it. I don’t particularly love that created word, but there might be better cases for it.

Anyhow, that’s neither here nor there. One quick question before we get off the topic of your book. Do you mind if we share with the audience what the results from that book were for you?

Samuel:  Financially speaking?

Patrick:  Financial or more important than that, either way.

Samuel:  Yeah. Either of those sound good. As I mentioned, I really did not do the launch super‑well. Being a hypocrite cost me thousands of dollars, I think [laughs] , in that scenario.

One of my strongest philosophies is paying attention to the last mile. Don’t tack onboarding on at the end. Make sure people are oriented around the value that they’re going to be getting, paying attention to those switching moments, so on and so forth.

I literally was editing my book up until hours before I launched and spent probably about one‑fiftieth of the amount of time I should have spent on the sales page. Worked really, really hard to build up the email list to a few thousand people, and blasted them at a page that was really confusing to them and did not emotionally engage them or anything like that.

Fortunately, because of the robustness of just having built up the list, trying to warm it up before I launched, and sharing the ride, I guess, as we were going, I still made 7,000 and change on the launch day, but there were some very forehead‑slappingly obvious problems with the sales page itself.

Fortunately, that’s, again, a nice thing about not just basically optimizing for launch and then trying to keep as much money from it as possible.

Being in it, invested in the long haul, and seeing it as an ongoing asset, I guess, for lack of a better term, I was able to make those changes within a couple weeks, and saw that things were not going to be nearly as dire as they appeared to be on launch day.

Actually, I just pulled up the launch‑to‑date stats in Gumroad. Also, I recommend Gumroad for those out there. One more person who’s a very happy customer. Probably by the end of the week, I’ll cross over the 70K sales number.

Patrick:  Awesome. Congrats.

Samuel:  Thank you.

Patrick:  That’s probably the first time I’ve heard a 10X increase from launch to lifetime. Actually, it makes a lot of sense in software. Man, the power of having assets. Just last week, Bingo Card Creator sold its 10,000th license.  [Patrick notes: I will note that while this asset is quite impaired these days, due to less traffic for some reason I’ve never really poked into, it is still trucking along quietly powering classrooms and underwriting lifestyles of the rich and famous maybe half our new-and-improved Tokyo rent.]

Samuel:  Awesome.

Patrick:  Only took 10 years.

Samuel:  [laughs]

Patrick:  Or eight years or whatever 2006 to today is. The power of having assets.

Samuel:  It’s also something where that is basically what I used to make in salary. Not only has it led to more consulting work, higher‑priced consulting work, and things like that, but just knowing my family’s not going to starve if something weird happened, just having that as an ongoing source of revenue is very comforting, I guess you could say.

Patrick:  It allows you to also optimize for other decisions. If you’re the typical person with a salary job, you have to have that salary job, or else the rent does not get paid for next month.

If you are the typical freelancer and somebody comes to you with a proposal, and you’re not really feeling it about this client, it’s not exactly the kind of work you want to be doing, or maybe they’re giving you pushback on your rates or whatnot, you might think, “I have to make these compromises in this business relationship because your rent is due next month, and I need to sell these hours.”

Where, if you have a baseline, even a small baseline, you get to be really picky about things. It’s like, “I don’t have a great option for consulting next week. I could take a middling option to consult for the next four weeks, or I could just wait and see if a great option shows up.”

That lets you be more selective with your clients, lets you pick the kind of work that you want to be doing. In a lot of cases, it actually is better for your clients.

Samuel:  Nodding vigorously over here, once again.

Patrick:  There were a lot of consulting clients where we heard the outlines of what they want and said, “Look. I am not the right guy for this, so I’ll just give it to you straight. Here is the right guy for it. You should have him do it instead.”

If I was more constrained by financial stuff, I might think, “I’m not the right guy for this but…” or “For my rate per week, I could be the almost halfway decent guy for it.”

When I go into clients, I have the ability to say, “You brought me in to do the best possible work and get you the best possible results, so we are only going to do projects which will be my best possible work and get what I think, before doing the project, is likely to give you the highest results. If you are not amenable to that, that’s fine. There are many consultants out there. I can recommend you to one of them.”

Samuel:  I totally agree. It’s at this point where I know the unit of work that goes into creating a tear‑down is going to result in roughly a certain amount of money coming back in book sales every time I do it. When I’m bidding on a project, I’m like, “How many tear‑downs, basically, is this project going to be?”

The number, financially, that I would get out of doing X tear‑downs is often a lot higher than what a client would be willing to pay for the same amount of work to work directly for them. Or at least it’s a healthy way for me to keep that in perspective, so there’s that.

Another nice thing is, as I mentioned way earlier, I’m in the very, very early stages of creating software for the onboarding space, and so I put out a survey. I didn’t even drive that much traffic to it. I haven’t sent out an email linking to it to my email list or anything, just tweeted it out a couple times.

At the end of the survey, I said, “It was really great of you to complete the survey. As a thank‑you, can I send you a coupon for 15 percent off the book?” It’s a nice opportunity to be able to be generous. There is something that you can offer, and also, at the same time, I don’t care if somebody buys it for 15 percent off.

If that’s going to be a triggering moment for them, then great. I’ve already made a couple hundred bucks just from putting out a survey. Which, I’m almost getting paid to do customer development. That kind of thing is really nice.

Patrick:  I really like your idea of writing the book before writing the SaaS product, for a lot of reasons. I think probably my next‑next project is going to be a SaaS, just because, a long story. I could do an entire episode on this.

Appointment Reminder is not exactly holding the fire in my belly, and I wanted to get into some sort of ‑‑ since it turns out the thing that I’m really good at and that my audience really trusts me about is making money for software companies, I want to make “making money for software companies” as a SaaS business.

Samuel:  That makes a lot of sense.

Patrick:  I’m probably going to be thinking in the next 18 months about how to actually turn that into something. It’s a heck of a lot easier to make the SaaS business after you’ve written a book on it and proved that there is an audience that cares about that topic.

Is empirically willing to pay you money about it than it is to just parachute into a field and say, “Well, I don’t have any evidence that people are willing to pay money for solutions to this, and if they are willing to pay money for solutions, I don’t know who that is. I’m going to spend the next six months writing Ruby code anyhow and see if it works.”

Samuel:  You’re going to be spending a bunch of time trying to build up that audience regardless. Looking at Rand Fishkin, with what he did with SEOmoz at the time, where he put out “The Beginner’s Guide to SEO” and got all that attention and built up his audience around that, and then were like, “Oh, we should create tools to serve this audience that exists now,” you can very clearly see the transition from one to the other.

Patrick:  I want to just do a little bit of a callback to something we were talking about earlier. You mentioned how, when you were writing your book, you had really, really hard stretches, like, “Man, this is taking much more time than I expected it to be.” Creative work is occasionally a beast. I have a funny story on that.

I’ve been saying since last August ‑‑ not the August in 2014 but the one in 2013. Yeah, I’m good with math that way. Since last August, that any day now, I was going to be releasing a course on conversion optimization.

Knock on wood, any day now, [laughs] hopefully before the birth of my daughter at the end of the month, I will be releasing a course on software‑conversion optimization. It’s on if you guys want to take a look at it. If you could, yay, great.

[Patrick notes: See the postscript.]

That’s the other thing I really like about the asset‑building approach to business, as opposed to the “grinding it out for the day job” approach to business. I am trying to just push the pause button on pretty much everything, aside from responding to routine email, business‑wise, for much of the next six months after my daughter is born, just to be able to be present for that, which is something that is difficult to do if you’re committed to the standard W2/working‑professional life.

Samuel:  There you go. Count me in as a first‑day purchaser of said course.

Patrick:  Oh, awesome. Thanks very much. I should also mention that I think I had bought yours as well.

That’s funny. There’s lots of stuff on the Internet that teaches various worthwhile things, like your stuff on user onboarding. Like any other business, I have a budget for training employees [Patrick notes: The business has only one employee and there is a heck of a lot he doesn’t know.], and I think mine runs to ‑‑ I’m going to check with the accountant ‑‑ probably on the order of $2,000 a year.

My business makes, well, let’s say “mid six figures” in turnover. That $2,000 is not a lot of money. When you divide it over the fact that I only probably read 10 business books a year, that means I’m spending an average of $200 on each of them. Or maybe not books. Courses, products, what have you.

Then you figure, OK, if you can price things at $200 and then have businesses be happy to pay for them, like I’m happy to pay for your stuff. Then you aggregate that over even a small number of people, that turns into a real amount ‑‑ maybe life‑changing isn’t the right word, but definitely life‑impacting amount of money for an individual running the training business, in a really short amount of time, as a producer of useful stuff.

Samuel:  I wholly concur.

Patrick:  Thanks very much for coming and getting interviewed on the podcast, Sam. It was really awesome to have you. If folks want to hear more from you, where should they go?

Samuel:  I would recommend, as we’ve mentioned. On Twitter, also UserOnboard, and on Twitter as SamuelHulick, which is just my name as one word.

Patrick:  For those of you have difficulty spelling, like I did, it’s H‑U‑L‑I‑C‑K.

Samuel:  S‑A‑M‑U‑E‑L H‑U‑L‑I‑C‑K.

Patrick:  Awesome. Thanks very much for being on the program. Knock on wood, we’ll have another podcast available in about two weeks or so, assuming I’m not called away by either work duties or baby duties, and it will likely be on the subject of churn. I hope you guys can catch it. Thanks, as always, for showing up for the podcast, and we hope to see you next time.

Samuel:  Awesome.

Patrick:  All right. Bye‑bye.

Brief Postscript On My Conversion Optimization Course

This podcast was recorded a few weeks ago, when I was still holding out hope of getting the Software Conversion Optimization course out sometime before my daughter was born (later this month).

At the time I thought “One more quick sprint and I’ll finally finish this thing!”, but between the course and the impending baby I was under incredible stress and it was compromising both the quality of the work and my mental availability to support my wife.  Ultimately, I’m disappointed that I haven’t shipped this yet, but I’d be far more disappointed to not be a good husband and father, so in lieu of continuing to provide a slipping shipping date I’ll reiterate that “It will ship at some point, when it is ready.” and “If you pre-purchased the course and are in any way not happy with waiting, get in touch with me and I’ll refund you.”

If you’d like to hear when the course ships (and also get a free eight-ish lessons on software conversion optimization delivered over email), you can sign up here.