“instead of throwing exceptions, it would silently just clear the session and re-run the request.”

I will say it is annoying when something that used to throw an exception, in this case swallows the exception, and marches on. I guess that’s considered “handling” the exception, but not even a warning in the log? Surely some of the blame lies in making a change like this.

An environment as close to production as possible would have helped in snaring this bug, but obviously there are limits and one could never truly replicate a prod environment, and in this case with a fairly subtle error due to timing issues. There’s a reason why Doug Lea’s excellent util.concurrency was brought into Java 1.5 – concurrency is hard to get right.

This does remind me of “The Cuckoo’s Egg”, about an astronomer that notices a 2 cent difference between the standard Unix (BSD in this case) accounting program, and a custom accounting program, written to provide a little extra functionality that the standard tool did not provide. Long story short, it put the author (Clifford Stoll) on the trail of same hackers who gained access into the Lawrence Livermore Lab (Hagbard?). Fascinating story, there are clips on YouTube, including Cliff’s story, and a movie about the Germans (one of the guys apparently committed suicide but the circumstances were strange. Movie is in German but might have subtitles?) Anyway – all because this crazy astronomer wanted to figure out why there was a 2 cent difference in his accounting programs!

I think Mike Ferrier has the right nothion: “it reminds you of what you did in the past”.

Once upon a time (30 or 40 years ago), I used to be working in COBOL and assembly language on a 128 KiB mainframe, and someday the machine started halting erratically with invalid operation codes in the resident monitor (the kernel, for you young’uns). After delving deep enough in the code (yes, we had the mainframe’s full OS source in those times, and searching around, even in assembly-language source — C didn’t exist yet — isn’t so hard when you have written down the IP) I found out that two instructions had been written in the wrong sequence, causing an invalid operation code halt in privileged code (so that a “privileged operation code” /exception/ wouldn’t be triggered) if an external interrupt (such as the end of a tape block read, or maybe an interval-timer interrupt) happened while handling an internal interrupt (such as authorizing a certain specific privileged opcode which had been encountered in a usermode program). So I swapped these two instructions in the relevant system macro, recompiled and reinstalled the OS over a weekend, and submitted a software report (by snail mail, of course: no Internet back then). I still remember the exhilaration when the software notice came back, telling me my “temporary solution” had been found equivalent to the official fix.

A bug might be considered “pervasive” if it showed up in every part of your application (or was visible in many different parts of your source code) – while “persistent” would be for something that keeps on happening (even if, as in this case, it keeps happening only 1% of the time).

A bug that’s pervasive would probably also be persistent, but not necessarily vice-versa. :)

That’s day I swore never to use Thread in Ruby again. Now whenever I need concurrency, I just spin out processes. Redis is my hammer.