If you’re a technologist and you’re not living under a rock, you’ve heard about Heartbleed, which is a Severity: Apocalyptic bug in the extraordinarily widely deployed OpenSSL software. Heartbleed lets anyone capable of finding a command line read encryption keys, passwords, and other private data out of affected systems. If you don’t remember addressing this in the last 48 hours close this window immediately and get to work.
Now that we’re past the immediate panic phase, though, I want to share some lessons learned. Security experts can tell you more than I can about what it means for good C coding practices in high-criticality security libraries. I want to take a moment to point at the marketing aspects of it: how the knowledge about Heartbleed managed to spread within a day and move, literally, hundreds of thousands of people to remediate the problem.
Heartbleed is much better marketed than typical for the OSS community, principally because it has a name, a logo, and a dedicated web presence.
What’s In A Name
Remember CVE-2013-0156? Man, those were dark days, right?
Of course you don’t remember CVE-2013-0156.
The security community refers to vulnerabilities by numbers, not names. This does have some advantages, like precision and the ability to Google them and get meaningful results all of the time, but it makes it very difficult for actual humans to communicate about the issues.
CVE-2013-0156 was the Rails YAML deserialization vulnerability. “Oh! I remember that one!”, said the technologists in the room. Your bosses don’t. Your bosses / stakeholders / customers / family / etc also cannot immediately understand, on hearing the words “Rails YAML deserialization vulnerability”, that large portions of the Internet nearly died in fire. After I wrote a post about that vulnerability I was told for weeks by frustrated technologists about e.g. VPs nixing remediation efforts due to not understanding how critical it was. That’s a failure of marketing.
Compare “Heartbleed” to CVE-2014-0160, which is apparently the official classification for the bug. (I say “apparently” because I cannot bring myself to care enough to spend a minute verifying that.) Crikey, what a great name that is.
- It references the factual underlying technical reality of the vulnerability, which is data leakage during a heartbeat protocol.
- It is very emotionally evocative. Think of your associations — “my heart bleeds for you”, the Sacred Heart and associated iconography, etc.
- It sounds serious and/or fatal.
Geeks sometimes do not like when technical facts are described in emotionally evocative fashion. I would agree if it were for the purpose of distortion, but “If you use OpenSSL 1.0.1a-f you could be leaking server memory” actually is serious and/or fatal, so describing it as such has the benefit of making people seek immediate resolution, which should be our goal as technologists.
Unique names (and “Heartbleed” is unique, given that you’d be hard pressed to find any mention of it which predates the vulnerability) are useful for communicating shared concepts between people. My Twitter stream for the last few days is people sensibly discussing e.g. “Don’t forget, you can be heartbled in a client context”, “How do you fix Heartbleed on Ubuntu?” “Depends — older versions aren’t vulnerable, newer versions can just apt-get update & upgrade” “Thanks!”
This is a substantial improvement on conversations I’ve had about previous vulnerabilities, where you often end up discussing, e.g., “the Rails bug.” Which one? You know, THE bug. Wait THE bug or the other bug? The YAML bug. Wait wait the YAML bug in the XML handling or the class of bugs caused by YAML deserialization? Man, would that have been an easier month if we had all been talking about DeserialKiller.
Names which don’t involve arcane trivia like “OpenSSL 1.0.1g” are also easy to communicate with non-technical stakeholders. If you had a launch yesterday, and you were forced to choose between making the launch date and fixing Heartbleed, you absolutely should have scrubbed the launch. We were all racing against for loops and the prize for 2nd place was “Our customers’ security gets horribly abused.” To actually scrub the launch, you might need to convince e.g. a manager that despite the company having dropped $100k on a splashy ad campaign, Heartbleed was priority #1. The image of your lifeblood dripping out was more likely to successfully accomplish that than a CVE number.
Clear Communication
The Heartbleed announcement should be taught in Technical Writing courses. It is masterful communication. Let me quickly excerpt the first three paragraphs:
The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by the SSL/TLS encryption used to secure the Internet. SSL/TLS provides communication security and privacy over the Internet for applications such as web, email, instant messaging (IM) and some virtual private networks (VPNs).
The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users.
What leaks in practice?
We have tested some of our own services from attacker’s perspective. We attacked ourselves from outside, without leaving a trace. Without using any privileged information or credentials we were able steal from ourselves the secret keys used for our X.509 certificates, user names and passwords, instant messages, emails and business critical documents and communication.
That is tight, precise, hard-hitting writing, of the sort which we normally associate with journalists rather than cryptographers or software engineers. It is both technically accurate and yet comprehensible if you are not a technologist. It doesn’t bury the lede about severity: “popular $MUMBOJUMBO software library” “allows stealing the information protected” on your “web, email, IM, and virtual private networks” “without leaving a trace” including “user names and passwords, instant messages, emails, and business critical documents and communication.”
The website goes on to provide technical details and remediation advice, but you can already tell your boss “I can’t do that today, boss. We have to respond to heartbleed.com.” If he spends even 30 seconds glancing at that executive summary he’ll say “Crikey. Yep, you do.” I particularly liked the recognition that most remediation of Heartbleed would be done by businesses, which is probably why the writer focused on “business critical documents” rather than the more anodyne “data.” Data gets weighed by the gigabyte but business critical documents spur immediate action when threatened.
The Benefits Of A Dedicated Web Presence
I often tell OSS practitioners to use dedicated web presences for projects they consider important, as opposed to dangling them off of (without loss of generality) Github. Why?
People will generally try to link to something to describe a project / vulnerability / etc, and having an easy and obviously linkable canonical description is both best for clarity and best for your own personal interests as the project/etc creator. Heartbleed.com is the canonical explanation of Heartbleed, both because people trust $8.95 domain names and because it was first published, came with a design/logo and comprehensive information, and is suitably authoritative in character.
Compare it to the best canonical reference you can find about CVE-2013-0156. That would be an archived copy of a plain-text email, hosted on Google Groups. It isn’t particularly attention grabbing, doesn’t really scream “citable” to either a technical or non-technical audience, and is optimized for a fairly narrow strand of practitioners rather than the much larger audience of people who should have cared about CVE-2013-0156.
Visual Identity Is Important
The Heartbleed logo is probably one of the highest ROI uses of ~$200 in the history of software security. (I don’t actually know whether they got it done for $200, but that is about what I paid the last time I had a logo done for an OSS project.)
Heartbleed Logo
I saw some kvetching on Twitter to the effect that the logo designer heard about Heartbleed before the distribution maintainers at e.g. Ubuntu and RedHat did. This kvetching is wrongheaded, because the logo designer only needed the instruction “We have a project named Heartbleed. Come up with a logo which says serious danger.” rather than “Apropos of nothing, mostly non-technical logo designer, the heartbeat protocol in OpenSSL 1.0.1a through 1.0.1f has been fubared for 2 years now. Don’t tell the Ubuntu guys though, we’re trying to keep it a secret!”
(I am, for what it is worth, absolutely agnostic on who should have preferential access to information of upcoming vulnerabilities with regards to a particular project. This strikes me as something which should be bought from maintainers/security researchers if you care about it, but I’m only weakly committed to that.)
Why spend the extra money for a logo? Because it suggests professionalism and dedicated effort, because it will be used exhaustively in media coverage of the vulnerability, because it further deepens the branding association of the vulnerability, the name, the logo, and the canonical web presence, and because it also suggests danger. Is it the best logo in the history? No. This one won’t win design awards. But it certainly does the job with aplomb.
OSS projects often don’t have logos or, ahem, do not devote to them the level of technical excellence that they devote to their products. I will refrain from pulling in examples here to make my point.
Marketing Helps Accomplish Legitimate Goals
There exists a huge cultural undercurrent in the OSS community which suggests that marketing is something that vaguely disreputable Other People do which is opposed to all that is Good And Right With The World, like say open source software. Marketing is just a tool, and it can be used in the cause of truth and justice, too.
As technologists, the Heartbleed vulnerability posed an instant coordination problem. We literally had to convince hundreds of thousands of people to take action immediately. The consequences for not taking action immediately were going to be disastrous. They were not limited to “mere” violations of computer security, but would have had dire economic and social consequences in the real world. Livelihoods (and, likely, lives) were at stake.
Given the importance of this, we owe the world as responsible professionals to not just produce the engineering artifacts which will correct the problem, but to advocate for their immediate adoption successfully. If we get an A for Good Effort but do not actually achieve adoption because we stick to our usual “Put up an obtuse notice on a server in the middle of nowhere” game plan, the adversaries win. The engineering reality of their compromises cannot be thwarted by effort or the feeling of self-righteousness we get by not getting our hands dirty with marketing, it can only be thwarted by successfully patched systems.
This makes marketing an engineering discipline. We have to get good at it, or we will fail ourselves, our stakeholders, our community, and the wider world.
More OSS marketing like Heartbleed, please.
Totally agree. I was surprised by the level of clarity and effort put into the marketing of Heartbleed. Whoever was behind that deserves a medal.
When I started working at Bell Labs back in the late 70s, they made us all take a short technical writing course. The main emphasis was “write like a journalist doing a newspaper article, and STOP writing like a grad student doing an academic paper.”
Think about what your readers needs to know, what they’ll be interested in, and hit them with the important points first so they know whether to take the time to read the rest. If you’re writing about complicated ideas, make your language simpler so they can spend the limited amount of attention they’re giving you on the ideas and not on how fancy your writing is. Your goal is to get the point across before tl;dr sets in.
I’ve read the comments below, and while I agree with most comments. I do find it strange that an SSL vulnerability that has existed for 2 yrs is brought to the world the same week that WindowsXP support stops, along with microsoft Essentials support… I think this elaborate marketing genius might be brought to you by our good friends at Microsoft… Stimulate OS and antivirus sales..
Viola..Now saying that, I’m not downplaying the seriousness of the “Bug”, just keeping it in perspective..
Right on. Who *was* behind the marketing of Heartbleed, anyway? I’d love to hear the full story on how this came about.
I think that the marketing was done by Codenomicon, a security tools vendor that independently discovered and reported the Heartbleed bug: http://www.codenomicon.com/ .
I wonder if this will become standard practice for major bugs found by security companies, since their products and services are likely to get some attention from the publicity attached to the vulnerability.
If it’s not it really, really should be.
I’d like you to think about the amount of pagerank they just got by being the big link at the bottom of heartbleed.com for a moment.
Better yet, I’d like all the *other* security companies to think about that, and then market their next serious vulnerability accordingly.
Is there any fear of the boy who cried wolf effect; of this type of marketing being used too often and the impact of each vulnerability being lessened by the confusion of this new marketing ecosystem? “Is this new vulnerability ‘Brainleak’ as bad as ‘Heartbleed’ or is it just another name?” I fear it becoming like Weather.com’s winter storm naming scheme: universally recognized as unreliable hype.
Is there anything we can do to keep the marketing efforts as consistently effective as this first round?
I think of virus names as a middle ground between CVE identifiers and “Heartbleed”. This is a pendulum swinging through history—here’s an article complaining that taking the time to name viruses delays the response: http://www.brighthub.com/computing/smb-security/articles/52909.aspx
(I tend to agree with the present article rather than that one, but the situation is somewhat different.)
Case in point, freeswitch logo: http://freeswitch.org/sites/all/themes/spreadfirefox/logo.gif
In a twisted way, the CVE number, obscure bug descriptions, and impenetrable nature of bug reports is a ‘feature’, not a failure. There is still a lot of wrongheaded obfuscation done by vendors in an attempt to hide vulnerabilities, and the associated bloody nose that a company gets when one is found in their product. While the CVE numbers aren’t built to provide that obfuscation by design, nobody at the enterprise layer is clamoring to change the process (except maybe to keep it as quiet as possible).
The CVE also provides a context free placeholder for a bug that has been found, but not yet disclosed, which is probably an opportunity to set up the marketing message asynchronously to the original report I suppose.
Nice article.
As a non-native speaker I found your thesis on the name evoking any sort of emotion especially interesting, for me was just a short “aha, nice pun” chuckle. That means I’m not totally sold on your interpretation, but an interesting point indeed :)
This blog is a noteworthy piece of writing as well. I am in the media and we had to choose whether to talk about this over the past few days. You might think, “Of course!”
But we are a top-40 radio station and this is outside our typical news reporting. The Heartbleed name made this so much easier to choose to report because it was so much easier TO report. Easier to explain and easier to write a memorable story about.
Heartbleed was spread on news outlets that never would have mentioned it without the well-handled campaign.
The only thing I didn’t agree with you about… The logo is perfectly fine! Lol.
Tom Steele
Yes, everything you state, Patrick, is true except for one detail. As a security person charged with assessing heartbleed, I cannot rely on statements such as on the heartbleed web site; I must have hard technical details to assess. there were none. I had to wait until a PoC came out, then read that code in order to affirm that, yes indeed, the guys at Codenomicon had found a baddie.
Good marketing, yes. Good technical source? No. Note even the CVE had any detail. somewhere, somehow, I would have appreciated a technical source, as well.
There’s some background of the site in
http://techcrunch.com/2014/04/09/heartbleed-the-first-consumer-grade-exploit/
On the other hand, naming can be done wrong. Look at what happens when the antivirus companies name viruses — each company gives different names to the same virus (e.g. Conficker = Downup = Downadup = Kido), resulting in mass confusion.
I completely agree that marketing serves an important service and, like it or not, if you want to spread a message and have people take it seriously, you need good “marketing”. The question i still haven’t heard much about is “What about the companies responsibility to inform their customers that their servers are now safe?” I couldn’t figure out if my bank or other sites had actually fixed the problem Maybe they need to read this article too.
I agree on the evocative marketing approach for critical security bugs – until the point of diminishing returns. One critical bug with an evocative name becomes a major flaw begets a minor bug released creates a FUD campaign for attention. I would be concerned that Heartbleed has created an example for would-be attention seekers rather than serious contenders.
I’ll be among the first to admit the OSS community (and to an extent the broader technical community) does a generally poor job in properly marketing itself, but attributing VPs nixing nixing remediation efforts to _a failure of marketing_ seems extreme. I’m more inclined to say this a failure of technical competence by the VP. If they can’t understand the significance of a security exploit without having it properly packaged in _evocative buzzword bingo_ they should not be part of the decision making process.
And yes – I can appreciate management are all too often promoted to their level of incompetence and this is how it is, and we have to deal with it. I just want to point out that thinking of this as a _failure of marketing_ is implicitly accepting a broken system.
I completely agree with your message, but think it might be more accurate to call this a success of “branding” not “marketing”.
Serious kudos to Codenomicon for contributing to the resolution, and a big “grow the fsck up” to everyone who jumped to the conclusion that the discovery team distracted themselves and further jumped to publicly complaining based on their jumped-to-conclusion instead of contributing to the public outreach.
The question that I have is: If they spent this much time in preparing a polished PR message, might more consideration have been made into whether release of the openssl bug fix and propagation to various distros was the more important action to take time doing?
It seems like the high visibility of the bug works against us as well as for us.
If you look into heartbleed.com, you’ll see that authors of the site weren’t the same ones that went public with the vulnerability.
Also I’m pretty sure that such company has different people doing PR and vulnerability research, so those functions are not affecting each other.
The heartbleed branding was so strong that it took me around an hour of googling to get to the actual information past all the nonsense press stories that said nothing of value and did not link to the original site. I think the tone and concern got out far stronger than the actual information, and I think this is not surprising when the tone and broadcast of concern were prioritized this high relative to the core problem.
It took you an hour to find the top ranked site for “Heartbleed Vulnerability”, aka heartbleed.com?
That’s probably on you.
Desirable component of content material. I simply stumbled upon your website plus accession cash to assert i get into point cherished profile the web site threads. By any means I shall be following with your augment and also I actually fulfillment you will get appropriate of entry to continually speedy.
I was receive lots of email regarding heartbleed, but my questions is why many software developers and also security expert did not aware this issues before? What happen with it? Are they did not have lots of experience?
SSL security also use in many bank for protect security but they are using 128 bits of SSL, oh come one… should i change my password?
Thanks for the great post. I’m glad it’ all mostly solved. Keep up the good work!
Thanks for adding the marketing note. Yes for all products, software and inventory, marketing is what gets it to the finish line.
http://www.siberia-nw.ya.ru
Outstanding overcome! I’d like to beginner while you actually amend your site, how do i join for a website web-site? This bank account made it easier for us a applicable option. I have been small bit common of the the sent out presented dazzling distinct plan