Archive by Author

Running A Software Business On 5 Hours A Week

Some four years ago, I started Bingo Card Creator, a business which sells software to teachers.  At the time, my big goal for the future was eventually making perhaps $200 a month, so that I could buy more video games without feeling guilty about it.  The business has been successful beyond my wildest expectations and has made it possible to quit my day job at the end of this month.  The amount of time I’ve spent on it has fluctuated: the peak was the week I launched (50 hours in 8 days), a very busy week in the last few years spiked up to as many as 20 hours, and the average over the period is (to my best estimate) about 5 hours.

During the majority of the time I’ve had the business, I’ve also been a Japanese salaryman at a company in Nagoya.  For those of you who are not acquainted with the salaryman lifestyle, I leave the office at 7:30 PM on a very good day, and have an hour and a half of commute both ways.  In our periodic bouts of crunch time, such as the last three months, I end up sleeping at a hotel next to the office (about 25 times this calendar year).

I’m not saying this to brag about my intestinal fortitude — this schedule is heck on your body and life, and absolutely no one should aspire to it.  That said, I snort in the general direction of anyone saying a nine-to-five job is impossible to juggle with a business because “businesses require 100% concentration”.

Here are practical, battle-tested ways for you to improve the efficiency of your business and deal with some of the niggles of partial self-employment.  They’ll hopefully be of use whether you intend to try running it in your spare time or just want to squeeze more results out of the time you’re already spending.  Many of these suggestions are specific to the contours of running a software business on the Internet, which has a lot to recommend it as far as part-time businesses go — take care before trying these willy-nilly with an unrelated industry.  (Part-time pacemaker research is probably not the best idea in the world.)

Time as Asset; Time as Debt

The key resource if you’re running a business by yourself is your time.  Other businesses might worry about money — however, you’ve probably got all your needs and then some covered by your day job salary, and capital expenditures in our business are so low as to be insulting.  (I started my business with $60.  Literally.)  And the key insight about time is that software lets us take the old saying about how “Everyone gets the same 24 hours per day” and break it open like a pinata.

Time can be stored.  One of the great features about currency is that it functions as a store of value: you create some sort of value for someone via your labor, trade that value for currency, and then the currency will retain value even after the physical effect of the labor has faded.  For example, a pumpkin farmer might not be able to conveniently store pumpkins, but if he sells them the currency will (under normal circumstances) not rot.

Most people think, intuitively, time always rots.  You get 24 hours today.  Use them or lose them.  The foundation of most time management advice is about squeezing more and more out of your allotted 24 hours, which has sharply diminishing returns.  Other self-help books exhort you to spend more and more of your 24 hours on the business, which has severely negative effects on the rest of your life (trust the Japanese salaryman!)

Instead of doing either of these, build time assets: things which will save you time in the future.  Code that actually does something useful is a very simple time asset for programmers to understand: you write it once today, then you can execute it tomorrow and every other day, saving you the effort of doing manually whatever it was the code does.  Code is far from the only time asset, though: systems and processes for doing your work more efficiently, marketing which scales disproportionate to your time, documentation which answers customers’ questions before they ask you, all of these things are assets.

The inverse of time assets is time debt.  Most programmers are familiar with technical debt, where poor technical decisions made earlier cause an eventual reckoning where forward progress on the program becomes impossible until the code is rearchitectured.  Technical debt is one programmer-specific form of time debt.  Basically, time debt is anything that you do which will commit you to doing unavoidable work in the future.

Releasing shoddy software, for example, commits you to having to deal with customer complaints about it later.  So don’t do that.  Better yet, rather than a useless bromide like “don’t release bad software”, spend time creating systems and processes which raise the quality of your software — for example, write unit tests so that regressions don’t cause bugs for customers.

However, not all time debt comes from intrinsically negative activities: there are many things that successful businesses do which cause time debt and you probably do not have the luxury of engaging in them.  For example, high touch sales processes incur time debt almost as soon as you put out your shingle: you’re committed to spending many, many hours wining and dining clients, often on a schedule that you cannot conveniently control.  That is generally a poor state of affairs to be in for a part-time entrepreneur, even though there are many wonderful businesses, small and large, created in high-touch industries.

Code Is About 10% Of Your Business.  Maybe Less.

Are you considering starting up a business because you wish to work on wonderfully interesting technical problems all of the time?  Stop now — Google is hiring, go get a job with them.  90% of the results of your business, and somewhere around 90% of the effort, are caused by non-coding activities: dealing with pre-sales inquiries, marketing, SEO, marketing, customer support, marketing, website copywriting, marketing, etc.

Bingo Card Creator has been memorably described as “Hello World attached to a random number generator.”  If anything, that probably overstates its complexity.  Customers do not care, though — they have problems and seek solutions, regardless of whether the solution required thousands of man years of talented engineers (Excel) or one guy working part-time for a week.  (You’ll note that you can make bingo cards in Excel, too.  Well, you could.  Many people can’t.  If I sell to them, I don’t necessarily have to sell to you.)

Relentlessly Cut Scope

37Signals had many good ideas in their book Getting Real, but probably the best one is to “Build Less”.  Every line of code you write is time debt: it is another line that has to be debugged, another line that has to be supported, another line that may require a rewrite later, another line that might cause an interaction with a later feature, another line to write documentation for.

Cutting your feature set to the bone is the single best advice I can give you which will get you to actually launching.  Many developers, including myself, nurse visions of eventually releasing an application… but always shelve projects before they reach completion.  First, understand that software is a work in progress at almost every stage of maturity.  There is no magic “completion” day on an engineer’s schedule: “complete” is 100% a marketing decision that the software as it exists is Good Enough.  If you have to cut scope by 50% to get the software out the door, you’re not launching with a 50% product: you’re launching with 100% of the feature set that is implemented, with 100% of (hopefully decent) ideas for expansion in the future.

Pick Your Problem Well

Long before you sit down to write code, you should know what your strengths are and what your constraints are.  If you can only afford to spend 10 hours a week and your schedule is inflexible, then anything which requires calling customers in the middle of the day is out.  Scratch B2B sales for you.  If you have the graphical skills of a molerat, like myself, you probably should not develop for iPhones.  (Minor heresy: while Mac developers are very graphically intensive people who will buy software just to lick it if the UI is good enough, many Mac users are just regular people.  My Mac version has a conversion rate fully twice that of the Window version, and it is not noticeably pretty.)

Some people profess difficulty at finding applications to write.  I have never understood this: talk to people.  People have problems — lots of problems, more than you could enumerate in a hundred lifetimes.  Talk to a carpenter, ask him what about carpentry sucks.  Talk to the receptionist at your dentist’s office — ask her what about her job sucks.  Talk to a teacher — ask her what she spends time that she thinks adds the least value to her day.  (I’ll bet you the answer is “Prep!” or “Paperwork!”)

After you’ve heard problems, find one which is amenable to resolution by software and that people will pay money for solving.  One quick test is to see whether they pay money for solving the problem currently: if people are spending hundreds of thousands of dollars on inefficient, semi-manual ways to do something that you could do with Hello World and a random number generator, you may be on to something.  (For example, if you knew nothing about the educational market, you can infer that there are at least several hundred thousand dollars sold of reading vocabulary bingo cards every year, just by seeing those cards stocked in educational stores across the country and doing some quick retail math.  So clearly people are spending money on reading vocabulary bingo.  It isn’t that much a reach to assume they might pay money for software.)

Other things you would look for in your idea are anything you see yourself using in your Benefits section of the website to entice people to buy it.  (Benefits, not Features.  People don’t buy software because of what it does, they buy it for the positive change it will make in the life.)  If you think “People should buy this because it will make them money, save them time, and get them back to their kids faster”, then you probably have a viable idea.

Another thing I’d look for prior to committing to building anything is a marketing hook — something you can take advantage of to market your product in a time-effective way.  For bingo cards, I knew there were more activities possible than any one company could ever publish, and that gave me hope that I could eventually out-niche the rest of the market.  (This is core idea still drives most of my marketing, four years later.)  Maybe your idea has built-in virality (nice if you can get it — I really envy the Facebook crowd sometimes, although I suppose they probably envy having a customer base which pays money for software), a built-in hook for getting links, or something similar.  If you can’t come up with anything, fix that before you build it.

This should go without saying, but talk to your customers prior to building anything.  People love talking about their problems to anyone who will listen to them.  Often they won’t have the first clue about what a solution looks like, but at the very least repeated similar emotional reactions from many people in a market should tell you that the problem is there and real.  After that, it is “just” a matter of marketing.

One note about business longevity: you will likely be involved in this business until you decide to quit.  That means planning for the long term.  Markets which change quickly or where products rot, such as applications for the iPhone (which have a sales window measured in weeks for all but the most popular apps) or games (which have constantly increasing asset quality expectations and strong fad-seeking in mechanics/themes/etc) interact very poorly with the constraints you are under.  I would advise going into those markets only with the utmost caution.

Get Your Day Job Onboard

Don’t do work on your business at your day job.  DO NOT do work on your business at your day job.  Do NOT do work on your business at your day job.  It is morally and professionally inappropriate, it exposes you to legal liability (particularly if your business ends up successful), and it just causes headaches for all concerned.

As long as you follow that one iron law of doing a part time business, all other obstacles are tractable.  Many engineers these days code outside the clock — for example, contributing to OSS projects.  Tell your boss that you have a hobby which involves programming, that it will not affect your performance at work, and that you want to avoid any misunderstandings about who owns the IP.  You can do something culturally appropriate to actually effect that: it might involve a contract, a memorandum of understanding, or even just a promise that there is no problem.

(Aside: I know many Americans consider the last option shockingly irresponsible.  My ability to prevail over my employer — a major multinational — in a lawsuit is effectively nil.  A contract is just a formalization of a promise.  In Japan, the ongoing relationship with my bosses is the part of the agreement that provides security, not the piece of paper.)

One sweetener you can offer any employer: providing you with discretion to continue with your hobby costs the employer nothing, but it will result in you getting practical experience in technologies and techniques you wouldn’t normally get at the day job, and they can then make use of that expertise without having to send you to expensive training or seminars.  I generated conservatively six figures in business for my day job as a result of things I learned from my “wee little hobby.”  Feel free to promise them the moon on that score — all they have to do in return is not object to your hobby.

Speaking of day jobs: if you know entrepreneurship is in your future, you might pick a job which dovetails nicely with it.  Prior to becoming a salaryman I was employed by a local government agency which had stable salaries and a work-day which ended at 4:30 PM.  Hindsight is 20/20, but that would have been perfect for nuturing a small business on the side.  (What did I do with my free time back when I had so much of it?  I played World of Warcraft.  sigh Youth, wasted on the young…)

Avoid Setting Publicly Visible Deadlines

One thing I did not know four years ago was how dangerous it is to promise things to customers.  For example, suppose a customer asks for a feature which is on the release roadmap.  I might, stupidly, commit to the customer that “Yes, this will be available in the next release, which I hope to have ready on next Monday.”  If the day job then has me spend the rest of the week at the hotel, or I have a family emergency, I will miss that deadline and have one ticked-off customer to deal with.  That is 100% avoidable if you simply don’t commit to schedules.  (Also note that committing to a schedule is time debt, by definition.  If you ever say “Yes, I will implement that”, you’ve lost the ability to decide not to implement it if your priorities change.)

One of the most useful things I learned in college was a line from my software engineering professor.  “The only acceptable response to a feature request is: ‘Thank you for your feedback.  I will take it under advisement and consider it for inclusion in a later version of the software.’”  That line actually works.  (There are industries and relationships in which it won’t work — for example, if you’re in a regulated industry and the regulations change, you can’t fob the regulatory authority off with that.  Don’t be in a regulated industry.)

Release schedules are not the only type of deadline out there.  Ongoing relationships with freelancers will occasionally have deadline-like characteristics, too.  For example, if you have a pipeline where you generate requests for work and then the freelancer fills it, if you unexpectedly are unable to do your part, the freelancer will be idle.  Thus, you want a bit of scheduling flexibility with them, a store of To Be Done On A Rainy Day requests queued up, or a rethink of your relationship such that your brain is not required for them to be able to do their job.

Cultivate Relationships With Effective Freelancers

Dealing with outside talent is one of the most important skills of being a part-time entrepreneur.  It lets you work more hours than you have personally available, it lets you use skills that you don’t possess, and especially when combined with software you’ve written you can do truly tremendous things with with a little bit of elbow grease.  Many folks get started with freelancing from posting to sites like Rentacoder (awesome article about which here) or Craigslist.  That is fine — everyone has to start somewhere.  However, you’ll quickly find that there is literally a world of people out there who are willing to work for $1.50 an hour… and would be terrifically overpaid at that price.

My suggestion is that, when you find a freelancer who you click with, hold onto them for dear life.  Pay them whatever it takes to keep them happy.  Additionally, since most clients are just as incompetent as most freelancers, don’t be one of the flakes.

  • Pay freelancers as agreed, promptly.  I jokingly refer to my payment terms as Net 30 (Minutes), and that ends up being true 90% of the time.
  • Provide sufficient direction to complete the task without being overbearing.  (Freelancers with a bit of personal initiative are worth their weight in gold.)
  • Don’t schedule things such that freelancers are ever blocking on you or that you are ever blocking on freelancers.  You have all the time in the world if you get things done well in advance of need.  For example, I just got my St. Patrick’s Day wordpress theme done — for next year.  If I was getting the Easter bingo site cranked out now, any hiccup would mean it missed my window.  (Technically speaking it would already be too late for SEO purposes, but that is a long discussion.)
  • Recurring tasks are a great thing to systemize and outsource.  You can write software to do the painful or boring bits, greatly increasing productivity, and as your freelancers get more experienced at the task you take on less time debt for explanation and review of their work.

Speaking of which, the most successful freelancing relationships are ones where you correct the labor market’s estimation of someone’s value.  (That is the positive way to say “You spend much less on them than you’d pay someone else for the same work and they’re happy to get it because you’re the highest paying offer.”)  Much ink has been spilled about how the globalization of labor makes it possible to get work done by folks in low-wage countries.  To the extent that you identify skilled, reliable workers, this is certainly one way to do things, but it is not the only way.  The current economic malaise has left many folks in high-wage countries either unemployed or underemployed.  In addition, the labor markets have huge structural impediments to correctly valuing the expertise of stay-at-home mothers, retirees, and college students.  All of those are potentially good resources for you.

Understand the Two Types of Time

There are two types of time involved in business: wall clock time and calendar time.

Wall clock time: minutes/hours which you spend actually working.

Calendar time: days/weeks/months/years where time passes so that something can happen.

We expect the world to be very, very fast, because the Internet is very, very fast, but when dealing with non-Internet processes we are frequently reminded of how slow things are.

Paul Graham mentions this as one of the hard things to learn about startups.  I really like his metaphor for how to deal with it: fork a process to deal with it, then get back to whatever you were doing.  For example, while Google rebuilds its index in a matter of minutes these days (this blog post will be indexed within fifteen minutes of me hitting the post button, guaranteed), getting a new site to decent rankings still takes months of calendar time.  That doesn’t mean you stand around waiting for months — you get your site out and aging as fast as humanly possible, and then start working on other things.  Get good at task switching — you’ll be doing it a lot.  (I literally just alt-tabbed to Gmail and squashed a support inquiry.)

You can incorporate calendar time into your planning, too, and since it is essentially free to you (you’re planning on being here in a week, right?) it is often advantageous to do it.  For example, A/B testing requires lots of calendar time but very little wall-clock time: you spend 15 minutes coding up the test and then have to wait a week or two for results.  That works very, very well in a part-time business.  Often, you can get into a rhythm for feedback loops like that.  Do whatever works for you: for me, Saturday typically sees me end my old tests and start new ones.

Avoid Events, Plan For Processes

There is a temptation to see business as series of disconnected events, but that should probably be avoided.  For example, you might see a dozen emails as a dozen emails, but it is probably just as true that it is six of Email A, 3 of Email B, and three emails with fairly unique issues.  You should probably turn your response to Email A and Email B into some sort of process — address the underlying issue, write your web page copy better, add it to your FAQ, create an auto-text to answer the problem, etc etc.

Similarly, spending your time on things which help your business once is almost always less effective than making improvements which you can keep.  For example, running a sale may boost sales in the short term, but eventually the sale will end and then you cease getting additional advantage from it.  There is a time overhead assorted with running the sale: you have to promote it, create the graphics, code the logic, support customers who missed the sale by 30 minutes but want the price (give it to them, of course), etc etc.  Spend your time on building processes and assets which you get to keep.

Another example: attempting to woo a large blog to post about you may require quite a bit of time in return for one fleeting exposure to a fickle audience.  Instead, spend the time creating a repeatable process for contacting smaller blogs, for example something along the lines of Balsamiq’s very impressive approach.  (Other examples: repeatable piece of linkbait such as the OKCupid’s series on dating also works, or a repeatable method of building linkable content, or a repeatable way of convincing customers to tell their friends about you.)

You can also avoid spending hours on incident response if you spend minutes planning your testing and QA procedures to avoid it.  When they fail — and they will fail — fix the process which permitted the failure to happen, in addition to just responding to the failure.

Document.  Everything.

I’m indebted to my day job for teaching me the importance of proper internal documentation.  As weeks stretch into months stretch into years, no matter how good of a memory you have, you will eventually have things fall through the cracks.  Your business is going to produce:

  • Commit notes.  Thousands of them.
  • Bug reports.
  • Feature requests.
  • Pre-sales inquiries
  • Strategic decisions
  • Statistical analyses

… etc, etc.  The exact method for recording these doesn’t matter — what matters is that you will be able to quickly recall necessary information when you need it.

I tend to have short-term storage and long-term storage.  Short term things, like “What do I need to do this week?”, get written down in a notebook that I carry with me at all times.  (I lock it in the drawer when I get to work, but feel no compunction about sketching things on my train ride.)  Things that actually need to get preserved for later reference go into something with a search box.  This blog actually serves as a major portion of my memory, particularly for strategic direction, but I also have SVN logs (with obsessive-compulsive commit notes… often referencing bugs or A/B tests by number), email archives, and the like.  (One habit I picked up at the day job is sending an email when I make a major decision outlining it and asking for feedback.  Note this works just as well even if you’re the only person you send it to — at least you’ll force yourself to verbalize your rationalizations and you can compare your expectations with the results later.)

There are a million-and-one pieces of software that will assist in doing this.  My day job uses Trac, which has nice SVN integration.  I have heard good things about 37Signals’ stuff for project planning/management and also about Fogbugz for bug tracking.  Use whatever works for you.

Note that quality documentation of processes both prevents operator error and makes it possible for you to delegate the process to someone else.  Also, if you have eventual designs on selling this business, comprehensible and comprehensive documentation is going to be a pre-requisite.

Dealing With The Government

I’ve been pleasantly surprised how little pain I’ve suffered in dealing with the government.  Part of this is because software is such a new industry that we often slide by on regulation — if I ran an actual Italian restaurant instead of the software analogue, I would have to keep health inspectors happy on a regular basis, but there is (thankfully) no one auditing my code quality.  Speak with competent legal advice if you’re not sure, but for the most part the only thing Japan and America want from me is that I pay my taxes on time.

Paying taxes is weeks of hard work really freaking easy.  The typical Italian restaurant has to do lots of bookkeeping involving thousands of sales, most of them involving cash, juggle record-keeping demands for half-dozen employees, and has expenditures ranging from rent to wages to capital improvements to food with a thousand rules about depreciation, etc, to worry about.  By comparison, the typical software business gets half of bookkeeping for free (if you can’t tell me to a penny how much your software business has sold this year with a single SQL query… well, I don’t know whether to deride your intelligence or congratulate you on your evident success), we have absurdly high margins so if you forget to expense a few things it won’t kill you, the number of suppliers we deal with is typically much lower, and the vast majority of what we do is amenable to simple cash accounting.

Additionally, your local government almost certainly has a bureau devoted to promoting small businesses.  They are happy to give you pamphlets explaining your legal responsibilities — in fact, sometimes it seems the only thing they do is create ten thousand varieties of pamphlets.  Your local tax office will also fall over backwards telling you how quick and easy it is to pay them more money.

Incorporation?  Incorporate when you have a good reason to.  (I still don’t, but I might do it after I go full-time, largely for purposes of dealing with Immigration.)  If you’re selling B2C software, your number one defense against getting sued is promptly refunding any customer who complains, and that pretty decisively moots the LLC’s (oft-exaggerated) ability to limit your personal liability.  You’ll be personally liable for debts from the business, but since the business is fundable out of your personal petty cash that isn’t the worst thing in the world.  If sales collapsed tomorrow I’d be on the hook for my credit card bill, which runs about $1,200 a month — not a financial catastrophe for an employed professional, particularly when the business generates far more than that in profits well in advance of the bill being due.  Sole proprietorship — i.e. merely declaring “I have a business” — is the most common form of business organization, by far.

Ask Someone Else About Health Insurance

I’m only putting this here to mention I have no useful information, because I live in a country with national insurance.   That isn’t a veiled political statement — I am not really emotionally attached to either model, I just don’t have useful experience here.  (My impression is that young single businessmen around my age are probably well-served with getting cheap catastrophic coverage.)

Keep A Routine, When Appropriate

Through sickness, health, and mind-numbing tedium, I’ve woken up every day for the last four years, checked email, gone through the day, checked email, and gone to sleep.  This is the single best guarantee that I would deliver on the promised level of service to customers — almost all questions answered within 24 hours.  There have been many, many weeks where this is literally all I’ve done for the business.

I try to keep creative work — such as writing, coding, or thinking up new tacts for marketing — to a bit of a routine, too, with flexibility to account for days where I’m not mentally capable of pushing forward.  For example, generally I do planning for the week at dinner on Monday and have four hour block to the business on Saturday.  If on Saturday it turns out that I can’t make forward progress on the business, I clock out and go enjoy life.

Routines aren’t limited to the business, either.  They help me incorporate my other priorities — family, friends, church, gym, hobbies — into a schedule that would otherwise descend into total anarchy.  (If you want to see what happens to the things that I don’t prioritize when the day job starts knocking, well, suffice it to say that I was cleaning today and removed 13 pizza boxes from my kitchen table.  I hope to put both cleaning and cooking back in the rotation after separating from the day job.)

Seek The Advice Of Folks You Trust.  Disregard Some Of It.

One of the major things which pushed me to (a small measure) of success these last four years has been advice from the communities at the Business of Software boards and Hacker News and the writings of folks like Joel Spolsky, Paul Graham, and the 37Signals team.  Much of the advice I received has been invaluable.  I disagree quite strongly with some of it.  When reading advice from me or anyone else, keep in mind that it is a product of particular circumstances and may not be appropriate for your business.  And always, always, always trust the data over me if the data says I’m wrong.  (That’s the easy part.  The hard part is trusting the data when it is overruling you.)

I’m thinking of making this first in a series.  If you have topics you’d like me to cover in more detail, please, let me know in the comments.

Getting Ready For Going Full Time

I’m quitting my day job as of March 31st.  Today I was running out to lunch prior to a day packed with various uISV-centered activities and it struck me: crikey, this is really happening.  I’m exhilarated and that weird not-scared-not-settled feeling one gets when one has run out the door in a hurry and is absolutely positive they have left something behind despite having verified the presence of cellphone, wallet, and day job ID card.  (Oh, to no longer need the day job ID card.)

Let’s see, what is new and exciting:

Mini-sites: I met a new designer over the Internet who is helping me out with my stable of mini-sites, such as the (previously created) Easter bingo cards that I’m sure are going to be very popular over the next two weeks.

A/B tests: I have six A/B tests currently ongoing, which I think is my personal record for simultaneous A/B tests.

1) Landing page: Text heavy versus an image, a few sentences, and a signup button.

2) Shopping cart: A tweak of some microcopy (“Thank you for your interest in Bingo Card Creator!” -> “Get instant access to Bingo Card Creator!”)

3)  Shopping cart: Addition of microcopy (“You don’t need an account to pay with your credit card though Paypal.”)

4)  Online version dashboard: Addition of one-click access to top 10 word lists.  Seems to be a crashing failure at increasing task completion, incidentally.  I’ll be reverting this one in a moment.

5)  Guest signup: Presence/absence of guest signup seems to have no effect whatsoever on sales.  Good — I’m going to probably disable it, as they cost me support issues through the wazoo.

6)  Button redesign.  I found another talented designer recently, and decided to get a few dozen buttons drawn up to start testing versus my existing ones.

The old buttons:

The new buttons:

After seeing whether those new buttons are roughly comparable to the old ones or not, I had the designer make many, many variations on color and button texts.  As always, I’m testing my way to victory, one 3~5% lift at a time.

Next Application: I have more or less mentally committed myself to my next application, although I should probably give it some more thought.  Hint: it uses Twilio.  Stay tuned to the same blog time, same blog channel for when I have something to announce.

Substantive Blog Posts: I’m working on a few long-form posts about what it is/was like running a business in my side time and how to do that without losing your sanity.  If there are particular aspects of that you’d like to hear about, I’d love to hear them.  Folks keep telling me to focus on time management techniques, so I guess I’ll be covering that, but I think they’re a little dry.

Stats Bug In A/Bingo v1.0.0 and earlier

Many thanks to Ivan for reporting this one: there is a significant bug in A/Bingo calculation of z-scores for versions 1.0.0 and earlier, which borks substantially all z-score calculations and in some cases can change whether A/B test results are reported as statistically significant or not. 

The bug is all of one character long:

def zscore()
#omitted for clarity
 cr1 = alternatives[0].conversion_rate
 cr2 = alternatives[1].conversion_rate
 n1 = alternatives[0].participants
 n2 = alternatives[1].participants

 numerator = cr1 - cr2
 frac1 = cr1 * (1 - cr1) / n1
 frac2 = cr2 * (1 - cr1) / n2   #this line is bugged
 numerator / ((frac1 + frac2) ** 0.5)
end

I have fixed the bug (via the Slicehost console, on a Japanese cafe Internet PC, because I am stuck in Nagoya today again) and pushed the fix to the git repository.

Does this make my results invalid?

You can probably still have confidence in results you got from A/Bingo previously. While the numerical calculation of the z-score was borked, it was borked in a subtle enough fashion that most statistically significant tests will retain their statistical significance under the borked calculation and most statistically insignificant tests will not gain statistical significance magically as a result of the borked calculation. (My quick eyeball suggests that it causes BCC to overstate the significance of tests which are very significant and understate the significance of tests which are insignificant, which is a very fortuitous set of properties for a random bug to have in an A/B testing framework.)

I have re-run statistical confidence tests for everything I’ve ever done for BCC that I still have data for, and no experimental results changed as a result of the error. Nonetheless, I deeply regret the bug, and will write unit tests for the statistics code as soon as I am physically capable of doing so to rule out the possibility of this sort of thing in the future.

Business Stats On A Photo Frame

I got inspired by a blog post from Panic, a Mac software company, and created myself a dashboard for the business, currently residing on a photo frame right on my desk.  The full writeup is on my main site, including code if you want to use it.

Lesson from Madlibs Signup Fad: Do Your Own Tests

Periodically, news of an innovative, goofy, compelling, or compellingly goofy design decision will sweep across the Internets like wildfire.  Most recently, this happened with a madlibs-looking lead generation form.

I think it has much to recommend it in the context of lead generation forms (long, arduous monstrosity that you sign up for in the hopes you are contacted but not spammed to death), but I didn’t see much possible upside for using it on a new user registration form (short form which you sign up to use something).

However, I’m wary of trusting my instincts on such things when I could trust data instead.  There is a key point about A/B testing: trust your data, not somebody else’s data.  After all, you only make money when it improves your conversion rate, not their conversion rate.  You can feel free to use other folk’s successful experiments for inspiration but for heaven’s sake use them to inspire you to run tests, rather than inspire you to fire blindly.

I was particularly wary about trusting this result because, as pointed out by numerous people in the Hacker News discussion, roughly seven things changed between the two forms in the A/B test performed on the standard form versus the madlib form, and there is no particular reason to assume that the salient difference was caused by the part which strikes us as creative as opposed by more boring things like e.g. the call to action in the header.

When In Doubt, Test.  (When Not In Doubt, Test Twice.)

No less than six people said “Hey Patrick have you seen this madlibs thing yet?  You’ve got to try it.”, and because knocking something together would take less than 10 minutes because I have an A/B testing framework that makes this a one-line proposition, I decided I’d humor them.  I isolated just the madlibs versus standard style for the test, knocked up an alternative in about ten minutes with my (decidedly limited) CSS and Javascript skills, and set them against each other.  My conversion goal for this test is successfully inducing someone to sign up for the free trial of Bingo Card Creator.

My Usual Registration Form

The Madlibs Registration Form

P.S. If you have good eyes you’ll spot the other A/B test ongoing on this page.  I’m using the traditional way of mitigating cross-test interaction… ignoring the possibility of it.  Don’t tell your college stats professor, but this actually works pretty well in practice.

Results

I ran this test until A/Bingo, my A/B testing framework for Rails, told me that further testing was just a waste of my time.  It didn’t take long at all — 34 hours after the test alternative went live for the site, the first time I checked the results, they were already overwhelming.  Let me copy/paste right off my public results page:

Signup Madlibs Versus Standard Standard (27.55%) winner
Madlibs (21.73%)
95%

By my count that is a 22% decrease in conversion rates for using the madlibs signup style over the standard signups style, and the fact of the decrease (but not the magnitude) is significant at the 95% confidence level.

For the curious: there were 736 participants in this test, split roughly 50/50, as you would expect.  I love the Internet because where else can you get 736 people to help you improve your website while you sleep, work at the day job on Saturday, have an evening out with friends, and then sleep some more?

Anyhow: test ended, not touching the madlibs idea again.  Before adopting this or any other fad (or good suggestion, for that matter): do your own A/B tests.

Women, Men, And Other Things Done Wrong By Silicon Valley

This post is waaaaaaay outside the usual ambit of my blog, as it is at least arguably political and about cultural norms in Silicon Valley.  (I’m a sometimes visitor and spiritual resident, but I’ve never lived there.)  I’ll be back to software blogging on the weekend if all goes well. 

There was a bit of a dustup recently about there not being enough women in Startup Land.  By this, they really mean “the startups we can see in the Valley”, because the Valley thinks that it is the beginning and end of all things tech and startup.  This bit of hubris has a lot more going for it than some other bombastic nonsense I’ve heard over the years, since quite a lot of tech innovation has indeed happened in the Valley.   I live in the heart of Japan’s automobile industry.  We’re justifiably proud of our cars.  That doesn’t mean I think your cars suck.

Strictly speaking the complaint was phrased in terms of “diversity”.  This is the peculiar diversity of the American academy, where a gay Jewish man in New York, an Englishman in London, a 4th generation zainichi kankokujin (ethnically Korean who was born in Japan), and an Irish Catholic dogmatist living in a rice field in Central Japan are so close they are practically brothers.  True diversity, of course, is the 5-member iStockPhoto of attractive twenty-somethings sitting on the college quad who check different boxes on the demographic inventory and think alike in every way that matters.

A serious question for metrics-focused individuals: If demographic diversity is a proxy for diversity of thought, is there some reason we’re not measuring diversity of thought?  Is it hard to measure somehow?  We sound like we’re counting hits because grepping the Apache log is easy and implementing conversion funnel tracking is hard, despite the fact that we know hits are meaningless and we’re really interested in conversions.  (Cards on the table: I think we’re the pathological PHBs who learned only half the lesson and now seek hits as a goal unto themselves.)

I mean, I would be sympathetic to “We can’t build products for women if we don’t have more women in the room” if it weren’t so laughably false.  (Context if you need it: 90% of my customers are ladies.  They’re also older, better educated, less coastal, and more religious than would be anticipated of the customer base of most B2C startups.  I’m pretty much your typical 27 year old male engineer… well, for certain quirky values of “typical”.)  If you wanted to recruit a team with experience building products for women, rather than quickly polling their X chromosome count, you could just ask “What have you made for women?”

That is probably worth doing as the Valley creates a persistent undersupply of products targetting the needs of women.  “Persistent undersupply” is one of those words that should be music to your ears if you are a capitalist, because market failures are opportunities to make lots and lots of money.  (For that matter, if you think that there is a vast pool of untapped female talent working for 80% of the price of equivalent male talent… what are you doing hiring men, again?  That would suggest that you could field whole teams of ladies and clean up.  I tend to be skeptical of the “persistent underpricing of female labor” hypothesis in the large, but there’s at least one example which has produced the outcome Econ 101 suggests: you can hire a stay-at-home mom with a graduate degree in Middle America for less than $10 an hour.  If you figure out a way to exploit that, you’ll end up very, very rich.  This is the unsung secret to Demand Media’s success.  If you think I’m wrong on the probable lack of this opportunity for female computer programmers, please go prove me wrong and make billions.)

My beef with the discourse of “diversity” in a nutshell: it screams “give us more women” and whispers “give us more women like us”.  We want more women to be early stage startup employees working for equity and battling code until 2 AM in the morning.  We want more women making products to pump VC cash into so that they can be flipped to Google in two years despite having less paying customers than your local Girl Scout troop’s worst cookie salesman.  We want more women mentors and women VCs and women industry group organizers so that we can pat ourselves on the back for embracing change while making sure that the Valley stays the way it is.

After quite a bit of time studying diversity in college (yay, liberal arts degree — there was a buy-one-get-one deal the day I majored in CS) I can rattle off all the hypotheses for you: there’s a biological basis, no it’s a cultural issue, no it’s a pipeline issue, no it’s a lack of role models, no it’s a …  and it is probably a witches brew of all of the above and more.  But my gut instinct has always been that people avoid joining startups because joining startups sucks.  The question isn’t what are we doing that’s keeping ladies out of the Valley, gentlemen.  The question should be why in God’s name are we still here.

Let’s review:

  • Most startups require you to be at or near the top of the game in a very difficult, competive field, requiring a college degree (or equivalent education gained in the School of Hard Knocks) in a subject which is widely agreed to be difficult.
  • The no-risk option for anybody capable of doing a startup is to go to their local insurance company and get a job cranking out CRUD apps.  They will immediately be in the upper middle class, have ample opportunities for professional advancement, and leave work each day at about 5 PM.
  • Of course, just because you might possibly be good at programming doesn’t mean you’re limited to doing it.  You could go into a host of fields in engineering or outside of it.  You could have the societal respect of being a doctor, or the material rewards of going into finance, or the work/life balance of teaching, or the rock-solid stability of being a technocrat ensconced in a minor government office somewhere.
  • Your sales pitch as a startup is “Turn your back on all that!  We’ll work you 100 hours a week, pay you nothing while requring you to live in a freakishly expensive area, give you social status one rung above the homeless, take two to three years of your life, ruin your relationships, and with better than 90% probability subject you to the most crushing defeat of your professional career with no lateral move except into doing the same thing over again.”
  • Your upside, should you make it to the pinnacle of your profession and do everything right, is theoretically unbounded but, practically speaking, what’s left after the VCs get their share will probably work out to a few million for most founders and barely cover the opportunity cost for early non-founder employees.  I don’t mean to say that is totally insane, but it requires that you have the risk-tolerance slider bumped to the maximum.

Unsurprisingly, the conversion rate for the above sales pitch has lagged expectations.

I don’t know which factor makes the Valley most gender-skewed but I’m pretty certain that casting a jaundiced eye at the reality of the situation turns off many intelligent women.  It certainly has to turn off many intelligent men, too.

So here’s a quick action plan to fix some Valley pathologies and make the whole thing a little more palatable:

  1. Make stuff for people who pay money for stuff.  This is a shortcut for getting paid money for stuff.  It also puts fun little infusions of funds between startup, series A round, and flipping to Google… which creates a whole spectrum of successful options for the business other than “achieve flipping to Google”.  (If you want more evangelizing on this subject, I suggest checking out DHH’s speech at Startup School.)
  2. Ditch the Valley.  Recognize that the same factors which make that tiny tip of the Silicon Valley distribution go infinite can make just about anybody, anywhere scale freakishly well — with respect to capital, with respect to time, with respect to team size, with respect to any metric you want to name.  OSS doesn’t stop working because you’re not in the Valley.  You can write an A/B test while sitting in a rice field.  (Trust me.)  SEO can be accomplished from anywhere on the intertubes and still scales worldwide.  There is a  rich ecosystem of businesses and APIs which so lower costs and barriers to entry for us — many of them built by the blood, sweat, and tears of twenty-something guys in the Valley (thanks for doing the work so I don’t have to, guys).  It is the best time in the history of the world to create a business.
  3. Don’t worry about the gatekeepers.  A lot of the angst about the old boy’s network is that the old boys are perceived as controlling opportunities to funding, which gives them power in the same way that the cartels gain power by rationing access to cocaine.  OK, quick solution: don’t seek funding.  Wham.  With a single stroke you’ve just managed to make the opinions of everyone but your customers utterly irrelevant.  (After you’re profitable, if you really want to, you can go to the Valley.  My guess is they’ll fall over themselves trying to give you money because they have no freaking clue how to create success in a reproducible fashion — c.f. 90+% failure rate.)
  4. Send people home at six.  I’m a poor example of this because I’m a Japanese salaryman (which means I get to say “100 hours a week!?  Slackers!” and then cry into my sake softly) , but you really can get an awful lot done in something vaguely resembling a human existence.  (I had originally described this as “a traditional workweek”, but I’ve got no particular love for 40 hours.  It is one arbitrary point you could find success at.  I know some people with businesses at 60 and at least one at five.)  One of the curious cultural pathologies of the Valley — and I suppose if I were a gender feminist I might describe it as “macho”, except I’m a Republican so I’ll just go for “stupid” — is that we treat overwork as a badge of pride and model it as the correct behavior.  This is insanity on a societal scale.  You mean we can scale to millions of visitors, our PCs got a bazillion times faster, and you can download twenty thousand man-years of software created by engineers smarter than you or I will ever be legally, for free, with explicit encouragement to build a business on it… and the only thing that hasn’t gotten better is the work week?  What.  The.  Heck.  I checked my Ruby standard library: there is no TriedToHaveALifeOutsideOfWorkException. 
  5. Take all advice with a grain of salt.  OK, so in deference to my liberal arts degree I’ll hit a few notes from it: institutions tend to try to perpetuate themselves.  Valley culture is a lot like an institution: it has its peculiar jokes and rhythms and closely-held shadow beliefs which owe a bit more to repetition than they owe to empirical reality.  A quick survey of the rest of this post should show you what I think of a few of these memes.  Remember that people, and I’m no exception, have a tendency to privilege the things they know and can think of easily over the things which are foreign to their experience.

If we fix this, it will result in more ladies at the margin seeing startups as an attractive career choice.  It might not change the percentages in the Valley.  Heck, it might even make it more skewed towards the guys.  I don’t profess to know and, honestly, I don’t really care that much either — it is worth doing regardless for the benefits to human welfare.

I Had Downtime Today. Here's What I'm Doing About It.

I screwed up in a major way yesterday evening. This post is part of my attempt to fix it.

This morning I woke up to an email from a paying customer saying that they tried to print cards but couldn’t. Specifically, they said that they were able to use the Print Preview feature, but that using the actual print button, quote, “caused the server to hang.” That can’t actually happen but it was sufficiently detailed as a bug report to immediately clue me in one what probably happened: the Delayed::Job workers must be down. A quick check of the server (ps -A | grep ruby) showed that this was indeed the case.

I quickly restarted the Delayed::Job workers then logged into the Rails console to check how many jobs had piled up. Six thousand.  Oof.  Most of them were low priority tasks (e.g. pinging the Mixpanel server with stats updates, which I do asynchronously to avoid having a failure there affect my users), but sixty users were affected — their print jobs were delayed.  Print jobs normally take under five seconds to execute and are checked with a bit of AJAX magic which polls the server until the job is ready, which means that most of these users probably got an animated GIF spinner to look at until they got tired and closed the web page.  The worst affected jobs took over twelve hours.

Happily, the downtime hit on a Saturday, which is the lightest day of the week for me.  If this had happened a week ago right before Valentine’s Day over 5,000 users would have been affected.

Apologizing To Affected Users

I used the Rails console to create a list of users affected by this, and have sent individual apology emails to the 2 paying customers affected (including attachments for the cards they had tried to print).  I will be contacting the trial users in a more scalable fashion.  Since I don’t have permission to email free trial users (the anti-spam guarantee I give is fairly strict), I dropped the development I had planned for this morning and built a simple messaging system into the site (~20 lines of code — I love you, Rails).  It gives me one-way “drop a message directly to your dashboard” functionality.

For example:

I prefer using this feature to the standard industry responses to outages:

  • “Outage?  What outage?”
  • “Please see our status page, which we’ve conveniently located in electronic Siberia.”
  • “ATTENTION ALL USERS!  0.7% of you were affected by very serious sounding things yesterday!  Please be worried unnecessarily even if you weren’t affected, and swamp our support line, who we will provide no effective tools to to tell you whether you’ve been affected or not!”

It allows me to apologize directly to affected users, makes minimal demands on their attention while still almost certainly reaching them, and does not cause any issue for the other 25,000 users.  Plus I can re-use this feature later in the event of needing to contact specific users without needing to email them (one obvious candidate would be plopping something straight on the screens of anonymous guests if I found something they individually needed to know, for example, if one of my automated processes caught that a recent print job of theirs did not come out right).

Preventing It From Happening Again

I’m something of a fan of Toyota’s Five Whys methodology for investigating issues like this.  (It has recently been popular with the lean startup crew.  My coworkers at the day job enjoyed some mostly justifiable smirks when I told them that.)

  1. Why couldn’t my users print?   Because the Delayed::Job workers were terminated when I upgraded the production server to Ubuntu Karmic Koala last night.
  2. Why didn’t the post-deploy checklist catch that users couldn’t print?  The post deploy checklist has “manually verify you can print cards” on it. I didn’t follow the post-deploy checklist with sufficient attention to detail because it was late (midnight) and I was tired (because I worked a six day crunch week at the day job… 30 days to go).  Here, I used the Print Preview feature to verify that I could print cards (“Hey, it tests the same code path, right?”), not realizing that while it tests the same code path they have different failure scenarios if e.g. Delayed::Job workers are down.  Fix: Quit day job and, regardless of how tired you are, follow the freaking checklist.
  3. Why weren’t you woken up by the Ride of the Valkyries playing on your cell phone when the site failed?  Don’t we have a system in place to do that? It turns out that the automated diagnostic (an external service pings a URL, the URL runs various tests and throws an HTTP error if any fail, the service mails my cell phone if there is an HTTP error twice in a row) tests nginx, mongrel, the D/B, and core program logic but doesn’t test the Delayed::Job processes or sanity check the job counts.  Fixed.
  4. Why didn’t the ‘god’ process monitor detect the workers were down? God sees every sparrow, but god only knows about the processes you tell it to manage, and my god_config.rb file has the Delayed::Job bits commented out with the notation “#This is buggy.”  I don’t remember why it was buggy and my notes in SVN are similarly unhelpful.  New task: unbuggy it.
  5. Why don’t you have commit notes, comments, or a development journal telling you what you were thinking when you found it was “buggy”? Failure to keep adequate records for “minor” changes and failure to follow up on a bug that was prioritized “Eh, get to that whenever” and then never gotten to.  Fix:  Look into beefing up developer documentation practices.

In the course of investigating this I discovered the update to Koala also killed Memcached on the server.  (Thankfully, Memcachedb — where I persist long-term user data that for whatever reason isn’t in the database, such as A/B testing participation data — is on another server.)  Unbeknownst to me, my use of memcached fails totally silently: if Rails can’t find the data in the cache it just regenerates it.  That would have had very unpleasant consequences for users if it had continued until Monday, and none of my automated tests would have picked up on it, because they all ignore timing.  I’ve added an explicit check to see if memcached is up and running.  I’ll also look into doing something about monitoring response times.

What I Learned From Japanese Engineering

I’m indebted to my day job for teaching me both a) how to do this and b) the absolute necessity of doing it, in spite of my longtime cavalierness with software testing. It was quite a culture shock for me the first time I logged into the test server at work to deploy something and got a rap on the knuckles for not:

  • Having a written explanation of exactly what commands I was going to enter.
  • Having a written checklist describing what tests to perform to ensure the deploy worked, and what the expected results would be.
  • Writing in the wiki that I was doing the deploy for a particular version done to close out a particular bug, so that there would be a trail to follow if the version I was about to deploy failed years from now.

That’s what we do for the test server.

All of the writing, test suites, automated test processes, and monitoring takes some time to set up and much of it generates additional overhead on all your tasks.  However, in the last three years, I’ve come to recognize that it is a net time-savings over writing apology letters and doing emergency incident response, neither of which are ever fun or quick.

Alright, development journal entry over.  Back to new development.

A/Bingo 1.0.0 Official Release

Back in August I released A/Bingo, an MIT-licensed OSS Rails A/B testing framework.  I have been using it continuously on Bingo Card Creator, and judging from the support requests I’ve been getting it has gotten some traction in the Rails world.  The 5,000 or so people seeing A/B tests on my site on Valentine’s Day are almost certainly less than 1% of the beneficiaries of the software now. Yay.

As A/Bingo has grown in popularity, I have begun to get requests for features that I did not need urgently for my own development, as well as the usual support requests, patches, and the like.  I want to make your use of the software as pleasant as possible to further evangelize the cause of A/B testing, so here you go:

New features:

A/Bingo now ships with a default dashboard.  Previously, I assumed that everyone would be writing their own dashboard code, so I just included the absolute minimum to show you what you’d need to do to get data out of A/Bingo.  Many people have remarked that they would really appreciate a “works out of the box” solution.  Your wish is my command — you can now enable a default dashboard in about ~30 seconds. It would work totally out of the box, but there are security implications, so I wanted you to have to think for a moment prior to enabling it.

#Create a new controller.  The name is up to you -- this example uses abingo_dashboard_controller.rb
class AbingoDashboardController  :abingo_dashboard

You can customize the dashboard code yourself. Nota bene: it uses your application layout, and has CSS classes applied to most of the elements, so you can style it quickly with CSS if you desire to. By default, it probably looks terrible. If you want to send me a patch to make it pretty, be my guest.

Experiments can now be stopped: Using either the built-in links on the above controller or, if you prefer programmatically scripting things, experiment.end_experiment!(alternative_content), you can now stop an experiment without touching the code.  Stopping an experiment causes all users to get the specified alternative rather than what they would have gotten randomly.  It also ceases stats collection.  Stopping an experiment is irreversible (currently — that might change later).  I tried to make this feature not affect the performance of A/Bingo for larger sites — it makes each test require one extra cache access.  (*cough* Rounding error, hopefully.)

A/Bingo internals are now fairly thoroughly tested: Unit tests are not exactly my cup of tea (“Argh, it works in production, what else do you want from me?!”), but Rails developers look askance at software that does not include them.  So I knuckled down and wrote a test suite.  (Hat tip to Nathaniel Talbott for mentioning A/Bingo in a conference presentation.  The constructive criticism regarding testing drove this change.)

I have not written thorough integration tests for the syntax sugar that you get via the included helper methods, but I’ll fix that eventually.

Named conversions: Previously, all A/Bingo tests required one line to add the test and one line somewhere else to track conversions.  Typically, since businesses have very many tests and fairly few conversion events, this resulted in code like:

#A controller method
def purchase
#Business logic goes here.
  bingo!("new_button_test")
  bingo!("email_copy_test_january")
  bingo!("microcopy_test")
  bingo!("button_colors")
  bingo!("login_button_alignment")
end

That isn’t very DRY at all.

Now, A/Bingo will take an optional parameter :conversion (or :conversion_name) when you’re defining a test, telling it to listen to a particular named conversion. This way, you can reuse the same conversion for as many tests as you want, decreasing the lines of code needed to create most new tests from two to one.

def some_method_with_a_test
  @alternative = ab_test("some_test_name", %w{altA altB}, :conversion => "purchase")
end

def some_other_method_with_a_test
  @foo = ab_test("bar_test", %w{coke water}, :conversion => "purchase")
end

def purchase
  #Business logic goes here!
  bingo!("purchase")  #Calls conversions for both of the above tests.
end

A/Bingo handles tests with spaces in them more gracefully: Although I still don’t recommend doing it, A/Bingo has been improving its handling of test names which have a space in them.  (The reason I don’t recommend it is because some cache stores — particularly memcached — do not support this well.)

Official support for Redis: Assaf Arkin picked Redis for his awesome Vanity project (which also does A/B testing for Rails, among other things), which inspired me to take a look into it.  It appears to be a much, much better alternative for a key/value store than Memcachedb, which is what I use for persistence.  A/Bingo has always accepted any cache store that Rails does, but I want to make it explicitly clear that I run tests against Redis, Memcached, and MemcacheDB. Just add the following to your environment:

#Goes in environment.rb
config.gem  'ezmobius-redis-rb',
  :source => 'http://gems.github.com',
  :lib => false

config.gem  'jodosha-redis-store',
  :source => 'http://gems.github.com',
  :lib => 'redis-store'

#Goes in whatever environment you're using:
require 'redis-store'
Abingo.cache = ActiveSupport::Cache::RedisStore.new

I intend to migration my own deployment to Redis when it becomes reasonably convenient for doing so.

Versioning: Previously I’ve just released patches to the A/Bingo git repository when I got done coding them, but I feel that is suboptimal now that there are substantial deployments which I could potentially break with changes.  So, here’s the skinny: A/Bingo is now, as of this blog post, 1.0.0.  I’ll communicate breaking changes by bumping that number up.  If it goes up by a tenth or more, expect that you need to re-run the migrations and that you will probably lose data on any tests in-progress, so plan ahead for that.  Version increases in that last number should be safe to apply directly.

I do not anticipate breaking the published A/Bingo API (i.e. methods mentioned in the docs) until at least v2.0.0, if ever, so upgrading A/Bingo should almost never cause you to need to update your own code.

How To Contribute

I would like to thank everyone who has submitted bug reports and patches. As usual, I’m always happy to get bug reports or feature requests. If you’d like to contribute code, make it available via git anywhere you please, and then send me an email telling me about it.

How Do I…

If the question isn’t answered in the (copious) documentation, feel free to ask me over email. If your business has particular needs for A/Bingo or you just want to talk A/B testing strategy with somebody who breathes it, I’m available for consulting engagements starting April 1st.

You Should Be Doing A/B Testing

I really can’t stress this enough: A/B testing is an easy, reproducible process that you can use to improve your marketing, website copy, product, user experience, etc. If you haven’t started yet, take A/Bingo, Vanity, or your other framework of choice for a spin. It won’t take you five minutes until you’re getting actionable data which you can use to make money.

Using CrazyEgg on Pages Requiring A Login

Long-time readers of this blog know I’m absolutely goo-goo for CrazyEgg, principally because they keep making me money.  They’re seriously my favorite $19 to pay every month, even when I don’t actually use them, because some day I know I’ll get the itch again and then bam actionable insights into what my customers are doing. Today is an itchy day.

One thing I have never gotten around to is tracking how users click on pages in my web application, behind the login screen.  CrazyEgg can do this but you need a bit of magic to let their screenshot bot grab samples of the page being interacted with — otherwise, they won’t be able to match up the events their Javascript recorded with the form fields on, um, your login screen.

CrazyEgg’s current user-agent string is:

Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.4) Gecko/20091016 (CrazyEgg 2 screenshot agent) Firefox/3.5.4

Then just shortcircuit your login procedure for people with that user agent.  (This may be undesirable if you are very, very security conscious.  I sell bingo cards to 60 year old women.  If you are security conscious, email them and they’ll provide a listing of IPs to whitelist.)

In Rails, doing something on the basis of the user agent is easy but, and I know this might come as a surprise, not covered in the documentation.

def authenticate_user
  if (request.env["HTTP_USER_AGENT"] =~ /CrazyEgg/)
    #Whatever you need to do to let them in as CrazyEgg
  else
    #Actual logic goes here
  end
end

For a quick 30 second solution, I signed up as a trial user for my own service with the email address crazyegg@bingocardcreator.com, and have the analogue to the above method on my site just pretend that anybody with the appropriate User Agent is authenticated as that user.

Dashboard Design For Metrics-savvy Software Companies

I have a confession to make: I’m something of a metrics junkie.  I have lost entire days of my life just staring at Google Analytics reports.  Metrics have always activated that same part of my brain that WoW did: ooh, a page view, ooh, a sale, ooh, if this had purple bars on it I’d pay $15 a month.  So I would flip from email to Analytics to e-junkie (the extremely appropriately named payment processor I use) to Analytics to… and end up accomplishing nothing of real importance.

Because although I’m a metrics junkie, I’m a smart metrics junkie.  And any smart metrics junkie can tell you that if your metrics aren’t giving you actionable insights to make decisions that matter to your business, well, you might as well go play WoW for all the good you’re doing.

That’s why, in one of my periodic bits of investing in the business, I built myself a dashboard.

Goals of the Dashboard

A dashboard is, simply, an easy to digest one-glance view on how your business is doing.  Mine is implemented in Rails and it is the page that greets me if I visit my site (password protected, naturally).  The purpose is threefold:

  1. Minimize the time it takes me to do common repetitive tasks.
  2. Show me information about my business at a glance, so that I don’t feel the need to log into my various other sources of info.
  3. Arrange for easy access to drilling down into important things for me.

Like all of my other software, it is a work in progress.  (Incidentally, some folks have offered to buy it when I have mentioned it previously.  I can’t sell it, since it is tied very tightly to my business needs and data available.  At all of ~200 lines of code, though, you can knock one for yourself out in an afternoon.)

Here’s a screengrab from it:

Some comments on what this shows:

Search box (“The Omnibox”): The Omnibox is my Swiss Army Knife support tool.  Given absolutely anything I know about a customer — from her name to email address to transaction number to a (hopefully unique) phrase she has used in her bingo card — it goes off and fetches her customer record.  This is a real timesaver because many of my customers don’t remember what email address they purchased the software under, told Google to obscure their email address (a misfeature in Checkout, if you ask me), purchased with their husband’s credit card, etc etc.  The Omnibox saves me from having to actually do work to find customer records about 80% of the time.  Search results are the same as…

Customer Entries (Latest First): The vagaries of the bingo business mean that I’m overwhelmingly more likely to get a support incident from you in your first 24 hours of use than at any other time.  Accordingly, as soon as I open the dashboard, the last 10 customers pop straight up so I don’t even have to search for them.  (This also serves as a quick visual health check and lets me see if a customer’s transaction didn’t go through.)  This illustrates a core principle of dashboards: do less work, get more done.  Ten keystrokes saved doesn’t sound like a lot until you’ve done it 200 times.

Customer Support Options:  You’ll note some customers have their names in green.  This means they are using the online version of Bingo Card Creator, rather than (or in addition to) the desktop version.  Previously I just made that green to satisfy my curiosity (it gives instant, accurate visual feedback that 70% of my sales come from the online version), but as I got a feel for customer support needs I decorated their records with a pair of hyperlinks.

“Ghost me”: This is the same link you get if you try the Forgot Password function in the online app — one click logs you in.  I right click it, select Chrome’s Private Browsing option (to avoid overwriting my cookie), and suddenly I’m you.  This lets me see exactly what a customer is seeing, so that I can diagnose problems easier, or in a pinch just do what they need done.

“Email password”: The same as the password recovery functionality on my site — mails them a link to let them in so that they can change their password.

Sales Counter: This is only on the page because otherwise I’d a) log into e-junkie to check and then b) start trying to guesstimate how much money I was going to make this month.  Both are bad habits.  Putting it front and center decreased my logins to e-junkie from several times per day to once a blue moon (only when I need to speak with them, which since they run a very tight ship is “Almost never”).

Edit Bingo Cards: Takes me into the original core of the Bingo Card Creator site: a CMS which makes, you guessed it, bingo cards.  This is where I approve work submitted by freelancers, make minor content edits to it, or create new bingo activities.  Again, instant access on the dashboard saves time.  (There are also a few affordances there, like AJAX approval so that all I have to do is mouseover a new card and click OK to approve it.  Do less work, get more done.)

Downloads Per Month: This shows a graph of how many PDFs have been downloaded from my site, on a monthly basis.  It is a quick one glance indicator of how effective my SEO is, from back when I didn’t have the user stats to look at.  I could probably demote it from the dashboard these days.  This information is public, by the way.

All-time Sales: Graph of sales by month.  I mostly use this to check on market seasonality, year over year increase (70%, ho), and whether I’m on pace to make my revenue goals.  This is also public.

A/B Test Results: If you’ve been around this blog much you probably know that I obsessively test and measure.  When I have a particular A/B test on the front burner of my mind it gets promoted to the top of the dashboard.  At the moment nothing I have running is super-critical so they’ve been placed a click away.  You can see some of my A/B test results here — I’m a big believer in code reuse so a portion of my test results are also used for the documentation for my A/B testing library.

E-junkie: Convenience link to my payment processor, where I used to log into a lot.  This should be demoted from the dashboard.

uISV email: The Google Apps for Domains email account I use.  (uISV stands for “Micro-independent software vendor” — i.e. like “small software company” with a more pretentious acronym.)

Logins Today / This Week: This is my “one glance health check” for the business.  I have a rough idea of what these numbers should be.  If they go down far below that, I have broken the login button and need to fix it immediately.  If they go high above that, it must be Halloween.  If they stay flat I might have broken the login button close to Halloween.

Vanity Stats: I keep these around just to satisfy my primal WoW player urges.  They have no particular relevance to my business, but they’re fun to quote to people.

You might think “conversion rate” is a really important number.  See, conversion rate for a channel or a creative or a landing page is a very important number.  Conversion rate for all of your visitors, on the other hand, is very, very sensitive to traffic mix.  Since customers arriving by AdWords always outconvert customers arriving by organic search in aggregate (which is an artifact of my SEO strategy and not to be worried about), a gyration in conversion rate is generally caused more by a change in prospect mix than a strong reaction to a change in my product.

Things Not Pictured

This dashboard can potentially display a few other types of information.  Thankfully, I don’t have an example to show you today.

Exceptions in payment processing: An exception happening in most places in my code is probably a misbehaving spider or a bug.  An exception happening in the callback for successful payments is a bug.  A very critical bug, because it often means that a customer loses access to the software they paid for.  If that happens, the computer sends me an email, my cell phone gets rung, and this whole page gets hidden under a stacktrace written in blaze orange until I hit the “I dealt with it” button.  (This obvious probably won’t happen if the whole site goes down — mon.itor.us watches that for me.)

Regular exceptions get written to my log files.  That is, candidly speaking, where data goes to die.

Bug of The Day: I have a confession to make: I’m not much of a test driven developer.  It is really, sincerely difficult to anticipate every corner case which could happen with customers trying all possible combinations of words and bingo card options.  This causes a huge portion of the user-visible bugs in BCC, and because the symptom is typically “It looks… ugly?”  (for a value of “ugly” not known at runtime) it is sort of hard to capture ahead of time in the predominant Rails testing frameworks.

That’s why I have a wee little daemon who periodically sanity checks all the print jobs people have run recently, looking for anything I’ve identified as the symptom of the Bug of the Day.  Currently, the Bug of the Day is that under certain circumstances the combination of a very long title and a bingo card with four to six short words in a certain box can cause the box to render across multiple pages, which is not desirable.  This is algorithmically checkable, but only in retrospect.

If the daemon detects that a particular print job caused one of the Bugs of the Day to pop up, it displays that fact in red, along with a link which copies the word list to my personal account so that I can reproduce the behavior and start trying to squash it.

This system also lets me sanity check upgrades to the site.  If I cause a regression in behavior, typically (squashed) Bugs of the Day will typically resurface, and my dashboard lights up like a Christmas tree.

What’s On Your Dashboard?

I love hearing implementation details from other businesses, and this is about as wonky as it gets.  What do you put on your dashboard?  Do you have any suggestions for things I should consider adding to mine?

Loading...
Grow your software business:
(1~2 emails a week.)