I am not an accomplished Perl hacker and something has been frustrating me at work for the last several weeks.  If you know of an easy way to solve this, I will be forever in your debt:

We are using Perl to run a HTTP server (HTTP::Daemon, incidentally, works like a charm) to receive information from a rich client application.  The client has to upload text files via HTTP post.  Everything is A-OK on the client’s side of things, but I have no clue how to go about extracting the uploaded text file from the HTTP request on the server side of things.  My experiments have been a bunch of failures, I can’t seem to find any CPAN Module which will do this cleanly, and there are other deliverables up in the air which are getting held up by the server being only 98% complete (that last pesky 2% is critical to testing the client application, sadly).  Does anyone have any suggestions?  Perl’s documentation is, well…  up to its usual standard of lucidity.

Steph of LandLordMax fame recently posted about how to deal with email spam.  This just happens to be the focus of my day job (we’re busy exploiting the synergies of a new paradigm of… actually, no, I don’t work in Dilbert’s office).  I use spam filtering religiously on all my email accounts (4 at last count) and it saves me roughly 2,000 spam a day (my business email address is forwarded to by my college email address, which was widely circulated back when I was young and stupid).

Here is everything you need to know about spam prevention in two sentences sentence: If you are comfortable administering your own email server (and already do), use SpamAssassin.  If you are not, use Popfile.  There, you’re done and didn’t even have to pay me my company’s consulting rate.

SpamAssassin takes a multi-tiered approach to dealing with spam: it has rules written by the administrator (or, if you’re a good outsourcer like any uISV should be, you just download them from the Rules du Jour website) which are handwritten regular expressions which look for spam signs, it has RBL checking, and it has a Bayes statistical engine.  Popfile just does Bayes.  You can find descriptions of exactly how Bayes works elsewhere on the Internet (Paul Graham was one of the early pioneers of the idea, incidentally: see his seminal article A Plan for Spam, with the caveat that the field has improved significantly since then and that you should consider this more of a historical piece than a tech blueprint).

Both SpamAssassin and Popfile, when trained properly, will produce well over 99% accuracy (both in not improperly labeling good mail as spam, and in not missing spam).  Exactly how you do the training depends on your installation.  If you are a mail administrator, start reading about SpamAssassin and understand you will be spending much of your free time feeding the monster.  If you aren’t, can I say for the second time, Popfile will save your life.

Popfile works as an email proxy, which means instead of having your email client (Outlook, Thunderbird, whatever) dial up your server when it wants mail, you have it dial up a port on your local computer instead.  There POPfile says “Hold on chap, just one second” and runs to fetch your mail from the actual server.  Popfile then scans the mail, uses statistical witchcraft to determine whether it is spammy or not, and marks it appropriately (via one or both of adding another message header to it or changing the subject line to read “[spam] Do 1+ 4 yur l0v3d 0n3″).  You then either just scan those subjects yourself or, if you’re smart, configure your mail client to automatically move everything tagged as spam into a special folder, which you periodically run through (look for missclassified emails and then purge).

Classifying emails is a bit of a nuisance: you have to visit a special web page (on your local machine), which you can do with a click of an icon in your system tray.  Then you have to locate the proper email (easy with a search-by-subject or sender feature) and click the appropriate classification (Popfile calls it “bucket”) and click reclassify. 

POPfile isn’t just for spam versus ham — you can have as many buckets as you want.  For example, I use Thunderbird at home to download from both my ISP email account and my bingocardcreator.com account(s).  I have POPfile configured to automatically tag any emails which are autogenerated (Amazon sales receipts, bounce mails, subscription confirmations, and the like), from family, regarding Bingo Card Creator, spam, from 2 particular mailing lists, and the like.  After you classify a shockingly small number of emails (generally less than a day’s worth in my experience, your mileage WILL vary), POPfile gets the hang of things very quickly: it “knows” that a genuine message from Paypal is invariably about Bingo Card Creator (despite it not saying “bingo” anywhere) and it “knows” that a fake message from Paypal (which it is very good at sniffing out) is spam.  My accuracy hovers in the 99.5% range.

The main problem with ANY anti-spam system is dropping legitimate emails.  I lost 3 in the last week, although two were identical (my friend hit the send button twice, apparently).  On a scale of criticality from 1-10 the email that was duplicated was like a 9.2, which was unfortunate (luckily I noticed it in a periodic scan of my spam folder).  The other one was a subscription confirmation (no big deal, since I was expecting it a search found it instantly).  For folks you absolutely want to get mails from, you can set up “magnets”, which are basically hand-written rules which override the statistics.  For example, I’ve got a magnet for my mother — anything she sends goes into family.  I’ve had another magnet for any email which quotes one of my own receipts, because that is what most of my customer support emails look like.  I stopped using that magnet because after the first 3 times that sort of mail happened POPfile was on it like white on rice.

OK, enough plugging POPFile (same story as always: opinions are my true feelings and totally my own, and nobody is sliding me money under the table from them, although you can reasonably expect that since my day job is directly related to this I am not an unbiased observer).

 Steph mentioned another method in his post: challenge/response.  Basically, if someone who you don’t know sends you an email, you send them an automated message saying “Hey, prove you’re a human by responding to this CAPCHA and I will deliver your message”.  DO NOT USE THESE.  The potential problems are numerous and they get even more numerous the more people who use them — and, since if you’re reading this you are a business, you should know that end-users HATE THEM with a burning passion.

… I am extremely busy at the day job at the moment.  Today was a deliverable for the project I am working on.  There are at least two deliverables to go, and final acceptance testing was supposed to start on, oh, August 31st?  Yep, you could say we’re under a weeeee bit of pressure to get things finished.  As a result my time on my uISV consists of responding to an email or three a week, and the odd transfer from Paypal to my bank account.  I hope to get “caught up” with it after we’re out of the woods at work.  I have the data from Oct. 1st through 15th stored for folks who are interested in stats, and will try to get that posted tonight or tomorrow night.

I’ve mentioned a couple of times that I like Direct Access, and use it on a daily basis (and by “daily” I mean “I type in a Direct Access shortcut about every 30 seconds to five minutes while working, but I do this every day, so thats daily”).  You can see my full review of the program earlier on my blog.  Actually, perhaps review is the wrong word to use: that implies objectivity, and I’ve got none here: I love the program like I love chocolate.  Anyhow, the reason I’m mentioning it out of the blue is that another site I rather like, Bits Du Jour, is running a one day 50% off sale on it.  So, if for some reason you’re still fence-sitting, you can get it for $20.

As usual, when I plug something here I’m not paid to do it.

I spent quite a bit of time in the last two weeks getting my personal finances in order (I generally do this once a year around tax time, and since I have an extension for living abroad tax time is October for me).  My situation, which is already a wee bit of a mind-bender as a result of having accounts scattered in Japan and the US, is going to be made a wee bit more difficult next year as a result of the uISV I set up.  Here is what I am doing to make this as easy as possible:

I got myself a credit card.   I have always been a cash-only person in my dealing with American companies (interest!  ick!), but I am starting to see the value of having a paid-in-full-monthly credit card.  For example, I have dreams of eventually owning a home, and that is going to require some demonstration that I am a responsible credit risk.  From the perspective of the American consumer finance industry my financial health is like that of a corpse, since the only activity for the last few years has been student loan payments.  Anyhow, I got myself a nice little Visa to help that out, and since all of my business expenses are dollar-denominated I’ll have something to charge every month.  Additionally, and crucially, with all of my business expenses and nothing else on one card it becomes very easy to file my Schedule C-EZ next year.  Schedule C is the form American small business owners report their taxes on, and it essentially asks you for your sales and expenses (capped at $5,000).  You pay taxes on the difference, at your individual income tax rate.

Its crucial that you be able to account for every dime you spent on your business, so that you can properly claim every dime worth of business expenses.  Back when I started out I paid for my business expenses on my Japanese credit card, my American debit card, and through Paypal, and as a result my record-keeping was a mess.  (I will seriously be reading my own blog when filling out my Schedule C, thats how bad it is to reconcile my credit card statements.)  With everything on one card, all you have to do is pull your statements online and sum.  This ensures that you don’t miss a single expense, because any expense you miss costs you money in taxes you pay but don’t owe.  For example, if I forgot come next October that the previous October I had paid $29.95 for stock icons (clearly a deductible business expense for me), then I would end up paying my tax rate (call it 25%) on $30 of income that those stock icons should have offset.  Thats $7.50 worth of idiot tax for forgetting a single transaction.  Don’t pay the idiot tax, keep copious records in an easy-to-access place!

The credit card also allows you to smooth out cash flow issues.  I get paid at odd times throughout the month, whenever someone buys my program and I transfer the money out of Paypal.  My bills, however, come both at set times during the month (web hosting, e-junkie), variable times during the month (Goooogle), and periodically (software/services purchases).  Having a credit card around lets me essentially reschedule those bills to come after I have had the sales to pay for them.  (N.B. I have a day job, and the day job can easily cover all my business expenses without blinking an eye.  However, the day job and the day job’s paycheck are on the wrong side of the ocean, and if my American checking account is dry when the Google AdWords bill comes that would be less than fun.  I would eat an insufficient funds fee, a week of below-minimum charges on my checking account, and a bank transfer, wire transfer, and currency conversion fee for fixing that.  So my credit card gives me a lot of piece of mind.)

I’m keeping paper files, too:  I have always been a direct-to-digital person when it comes to dealing with my finances.  However, to keep myself organized, I got myself a folder marked “BINGO TAXES” and have started feeding it three pieces of paper a month on the 31st: my copy of e-junkie’s sale tracking page, my copy of my Paypal account statement, and my copy of my credit card statement.  That allows me to reconstruct income and expenses in about 3 minutes per month.  Even if I were selling 10 times what I am now (wouldn’t that be nice), it would still be trivial.  I also keep PDF printouts of these documents on my hard drive (side note: if you don’t have a virtual printer which exports to PDF, can I recommend PDFCreator, it saves me from having to have a printer at all) and will soon be backing them up somewhere else in event of a catastrophe at my residence.

Roughly 80% of my customers do not buy software.  I don’t mean this in the sense that they are pirates, who grab software which they want without paying for it.  I mean this in the sense that their relationship to their computer is not primarily mediated through applications.  Many of them do not particularly like using their computers, and use only the programs which do what they need to do, rather than seeking out new, exciting things that they can do with their machine.  And what they need to do is generally simple: they need to create documents, they need to write emails to their nephews and coworkers, and they need to check the status of that lovely knickknack they are secretly hoping to snipe on eBay.  (Side note: about half of my customers have used eBay.  Yep.  It is that mainstream.)

Now, if I do say so myself, I’m pretty decent at selling software to these people who don’t particularly buy software.  You might be, too, if you’re wandering off the well-trodden “I am going to sell to software developers like myself” uISV path.  (Not that there is anything wrong with that, by the way, but it has its own challenges: you’ll be forever worrying about OSS coming along and eating your lunch, and your customers are in general loathe to spend money to accomplish something they think they can just whip up themselves… if they had a couple months free.)  Here are some of the things I have learned over the last few months of this.

The look of your program is critical.  Since release, I have upgraded my program with precisely three new features worthy of bullet point: a font selector, word wrap, and stock icons.  Before the upgrade (which, off the top of my head, I think was v1.03), I was getting two emails a week asking whether there was any way to do font selection, and perhaps one every two weeks regarding word wrap.  Do you know how many people ever mailed me saying “Your program looks kind of drab”?  Not a single one.  Do you know how many compliments I get in the average week now that my program looks bright, inviting, and the perfect thing for Grandma the bingo enthusiast who runs Windows98 and types her emails in ALL CAPS because that shift key is too difficult for her?  More than I have gotten for every feature request, ever.  The amount of space download sites allocate to screenshots is tiny, and mine having a splash of color in them instead of looking like a spreadsheet probably gets me some clicks I would not otherwise have gotten.

They want to know someone is home.   A certain segment of my customers needs a little hand-holding before they are willing to part with their credit card information over the Internet.  Typically, what they do is send me a single email asking a question about features: “Can I print a call list?”  “Yes, ma’am, go to the File Menu and select Print Call List.”  And then I go to sleep for the evening and wake up to You’ve Got Money.  My theory on this is that people want to know that the program is being actively maintained by a real live human being, and they want to know that if there is a problem somebody will be on the case.  Which is why I have my email address plastered to nearly every page on my website and in the program itself.  Some people say that if you’re selling at my price point ($24.95) you cannot afford to offer personalized customer support.  I say you can’t afford not to: your skittish prospective customers seeking reassurance will run away.  (I also write reassurances into my web site copy: your credit card information is safe, full money-back guarantee, upgrades available forever, free support from someone fanatically dedicated to making you happy, etc).

The ordering pathway needs to be failure proof:  I have taken significant steps towards making my ordering pathway failure proof: it is the same for PC and Mac, there are extremely few options and I bury all the ones except the most common (single copy of the software available via download), I ask for no additional information besides what I need to process the order (telephone numbers and shipping addresses?  Why?), and all roads lead to Rome from my purchasing.htm page: practically anything you click within the main body of the page will take you directly to the ordering page.  (Side note: purchasing.htm is NOT the ordering page because people who hit it typically have come from a link in my application.  Many of them are just interested in seeing what the link does or checking out the price.  As a direct consequence, I use purchasing.htm to make my “Close the deal” sales pitch to wavering prospects while simultaneously allowing folks who have their credit card out to consumate the purchase in 15 seconds or less.  My average is actually 42 seconds, most of it spent waiting for Paypal’s site to load by my guess.)

Previously, every time someone has posed the question “Should I make my blog part of my business domain, on a subdomain, or as a different domain altogether?”, I have said “Put it on your main domain, this will allow you to share pagerank between your blog and your business”. What can I say, I really wish that circa June, 2007 I had known enough about my own advice to take it. :)

Steph of real estate software fame recently posted a link to a tool which will tell you your pagerank for different domains after the currently-in-progress PageRank update gets pushed to all the Google clusters. At present, it seems that Bingo Card Creator is a PR4, which is neither terrible nor anything to write home about after I’ve spent a good deal of time trying to build up some links for it. This blog, on the other hand, is PR5 — solely due to people deciding what is here has value and linking it from their blogs (and other content aggregators which are spiderable, such as Slashdot or the WordPress front page).

The difference between PR4 and PR5 is a whole lot of searches, particularly for low-competition niches like, well, Dolch sight words. I now get an absolutely insane number of people who search for my very favorite search terms and come *here* first, then follow the link to my actual product site. This is utterly insane not because of the magnitude, but because I know I must be losing a huge percentage of the potential traffic there. Ah well, lesson learned. Don’t let it happen to you!

After my first post today I apparently triggered some insidious bot lurking in a dark corner of the Interweb: within 45 seconds of bringing my blog back from the dead I got over 200 spam comment submissions, mostly to old posts.  More distressingly, 180 of them sailed right past the spam filter.  All of them were about gambling.  As a result, to prevent me getting repetitive stress disorder from hitting the “mark as spam” button and keep this blog easy to comment on without requiring registration or anything intrusive, I’ve banned the following three words from comments: poker, casino, and gambling.  I don’t anticipate anyone needing to use them, but if for some reason you do you may want to write c@sino or you’ll have your post discarded transparently.