What Heartbleed Can Teach The OSS Community About Marketing

If you’re a technologist and you’re not living under a rock, you’ve heard about Heartbleed, which is a Severity: Apocalyptic bug in the extraordinarily widely deployed OpenSSL software.  Heartbleed lets anyone capable of finding a command line read encryption keys, passwords, and other private data out of affected systems.  If you don’t remember addressing this in the last 48 hours close this window immediately and get to work.

Now that we’re past the immediate panic phase, though, I want to share some lessons learned.  Security experts can tell you more than I can about what it means for good C coding practices in high-criticality security libraries.  I want to take a moment to point at the marketing aspects of it: how the knowledge about Heartbleed managed to spread within a day and move, literally, hundreds of thousands of people to remediate the problem.

Heartbleed is much better marketed than typical for the OSS community, principally because it has a name, a logo, and a dedicated web presence.

What’s In A Name

Remember CVE-2013-0156?  Man, those were dark days, right?

Of course you don’t remember CVE-2013-0156.

The security community refers to vulnerabilities by numbers, not names.  This does have some advantages, like precision and the ability to Google them and get meaningful results all of the time, but it makes it very difficult for actual humans to communicate about the issues.

CVE-2013-0156 was the Rails YAML deserialization vulnerability.  “Oh!  I remember that one!”, said the technologists in the room.  Your bosses don’t.  Your bosses / stakeholders / customers / family / etc also cannot immediately understand, on hearing the words “Rails YAML deserialization vulnerability”, that large portions of the Internet nearly died in fire.  After I wrote a post about that vulnerability I was told for weeks by frustrated technologists about e.g. VPs nixing remediation efforts due to not understanding how critical it was.  That’s a failure of marketing.

Compare “Heartbleed” to CVE-2014-0160, which is apparently the official classification for the bug.  (I say “apparently” because I cannot bring myself to care enough to spend a minute verifying that.)  Crikey, what a great name that is.

  • It references the factual underlying technical reality of the vulnerability, which is data leakage during a heartbeat protocol.
  • It is very emotionally evocative.  Think of your associations — “my heart bleeds for you”, the Sacred Heart and associated iconography, etc.
  • It sounds serious and/or fatal.

Geeks sometimes do not like when technical facts are described in emotionally evocative fashion.  I would agree if it were for the purpose of distortion, but “If you use OpenSSL 1.0.1a-f you could be leaking server memory” actually is serious and/or fatal, so describing it as such has the benefit of making people seek immediate resolution, which should be our goal as technologists.

Unique names (and “Heartbleed” is unique, given that you’d be hard pressed to find any mention of it which predates the vulnerability) are useful for communicating shared concepts between people.  My Twitter stream for the last few days is people sensibly discussing e.g. “Don’t forget, you can be heartbled in a client context”, “How do you fix Heartbleed on Ubuntu?”  “Depends — older versions aren’t vulnerable, newer versions can just apt-get update & upgrade”  “Thanks!”

This is a substantial improvement on conversations I’ve had about previous vulnerabilities, where you often end up discussing, e.g., “the Rails bug.”  Which one?  You know, THE bug.  Wait THE bug or the other bug?  The YAML bug.  Wait wait the YAML bug in the XML handling or the class of bugs caused by YAML deserialization?  Man, would that have been an easier month if we had all been talking about DeserialKiller.

Names which don’t involve arcane trivia like “OpenSSL 1.0.1g” are also easy to communicate with non-technical stakeholders.  If you had a launch yesterday, and you were forced to choose between making the launch date and fixing Heartbleed, you absolutely should have scrubbed the launch.  We were all racing against for loops and the prize for 2nd place was “Our customers’ security gets horribly abused.”  To actually scrub the launch, you might need to convince e.g. a manager that despite the company having dropped $100k on a splashy ad campaign, Heartbleed was priority #1.  The image of your lifeblood dripping out was more likely to successfully accomplish that than a CVE number.

Clear Communication

The Heartbleed announcement should be taught in Technical Writing courses.  It is masterful communication.  Let me quickly excerpt the first three paragraphs:

The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by the SSL/TLS encryption used to secure the Internet. SSL/TLS provides communication security and privacy over the Internet for applications such as web, email, instant messaging (IM) and some virtual private networks (VPNs).

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users.

What leaks in practice?

We have tested some of our own services from attacker’s perspective. We attacked ourselves from outside, without leaving a trace. Without using any privileged information or credentials we were able steal from ourselves the secret keys used for our X.509 certificates, user names and passwords, instant messages, emails and business critical documents and communication.

That is tight, precise, hard-hitting writing, of the sort which we normally associate with journalists rather than cryptographers or software engineers.  It is both technically accurate and yet comprehensible if you are not a technologist.  It doesn’t bury the lede about severity: “popular $MUMBOJUMBO software library” “allows stealing the information protected” on your “web, email, IM, and virtual private networks” “without leaving a trace” including “user names and passwords, instant messages, emails, and business critical documents and communication.”

The website goes on to provide technical details and remediation advice, but you can already tell your boss “I can’t do that today, boss.  We have to respond to heartbleed.com.”  If he spends even 30 seconds glancing at that executive summary he’ll say “Crikey.  Yep, you do.”  I particularly liked the recognition that most remediation of Heartbleed would be done by businesses, which is probably why the writer focused on “business critical documents” rather than the more anodyne “data.”  Data gets weighed by the gigabyte but business critical documents spur immediate action when threatened.

The Benefits Of A Dedicated Web Presence

I often tell OSS practitioners to use dedicated web presences for projects they consider important, as opposed to dangling them off of (without loss of generality) Github.  Why?

People will generally try to link to something to describe a project / vulnerability / etc, and having an easy and obviously linkable canonical description is both best for clarity and best for your own personal interests as the project/etc creator.  Heartbleed.com is the canonical explanation of Heartbleed, both because people trust $8.95 domain names and because it was first published, came with a design/logo and comprehensive information, and is suitably authoritative in character.

Compare it to the best canonical reference you can find about CVE-2013-0156.  That would be an archived copy of a plain-text email, hosted on Google Groups.  It isn’t particularly attention grabbing, doesn’t really scream “citable” to either a technical or non-technical audience, and is optimized for a fairly narrow strand of practitioners rather than the much larger audience of people who should have cared about CVE-2013-0156.

Visual Identity Is Important

The Heartbleed logo is probably one of the highest ROI uses of ~$200 in the history of software security.  (I don’t actually know whether they got it done for $200, but that is about what I paid the last time I had a logo done for an OSS project.)

Heartbleed Logo

I saw some kvetching on Twitter to the effect that the logo designer heard about Heartbleed before the distribution maintainers at e.g. Ubuntu and RedHat did.  This kvetching is wrongheaded, because the logo designer only needed the instruction “We have a project named Heartbleed.  Come up with a logo which says serious danger.”  rather than “Apropos of nothing, mostly non-technical logo designer, the heartbeat protocol in OpenSSL 1.0.1a through 1.0.1f has been fubared for 2 years now.  Don’t tell the Ubuntu guys though, we’re trying to keep it a secret!”

(I am, for what it is worth, absolutely agnostic on who should have preferential access to information of upcoming vulnerabilities with regards to a particular project.  This strikes me as something which should be bought from maintainers/security researchers if you care about it, but I’m only weakly committed to that.)

Why spend the extra money for a logo?  Because it suggests professionalism and dedicated effort, because it will be used exhaustively in media coverage of the vulnerability, because it further deepens the branding association of the vulnerability, the name, the logo, and the canonical web presence, and because it also suggests danger.  Is it the best logo in the history?  No.  This one won’t win design awards.  But it certainly does the job with aplomb.

OSS projects often don’t have logos or, ahem, do not devote to them the level of technical excellence that they devote to their products.  I will refrain from pulling in examples here to make my point.

Marketing Helps Accomplish Legitimate Goals

There exists a huge cultural undercurrent in the OSS community which suggests that marketing is something that vaguely disreputable Other People do which is opposed to all that is Good And Right With The World, like say open source software.  Marketing is just a tool, and it can be used in the cause of truth and justice, too.

As technologists, the Heartbleed vulnerability posed an instant coordination problem.  We literally had to convince hundreds of thousands of people to take action immediately.  The consequences for not taking action immediately were going to be disastrous.  They were not limited to “mere” violations of computer security, but would have had dire economic and social consequences in the real world.  Livelihoods (and, likely, lives) were at stake.

Given the importance of this, we owe the world as responsible professionals to not just produce the engineering artifacts which will correct the problem, but to advocate for their immediate adoption successfully.  If we get an A for Good Effort but do not actually achieve adoption because we stick to our usual “Put up an obtuse notice on a server in the middle of nowhere” game plan, the adversaries win.  The engineering reality of their compromises cannot be thwarted by effort or the feeling of self-righteousness we get by not getting our hands dirty with marketing, it can only be thwarted by successfully patched systems.

This makes marketing an engineering discipline.  We have to get good at it, or we will fail ourselves, our stakeholders, our community, and the wider world.

More OSS marketing like Heartbleed, please.

About Patrick

Patrick is the founder of Kalzumeus Software. Want to read more stuff by him? You should probably try this blog's Greatest Hits, which has a few dozen of his best articles categorized and ready to read. Or you could mosey on over to Hacker News and look for patio11 -- he spends an unhealthy amount of time there.

50 Responses to “What Heartbleed Can Teach The OSS Community About Marketing”

  1. Josh April 9, 2014 at 6:40 am #

    Totally agree. I was surprised by the level of clarity and effort put into the marketing of Heartbleed. Whoever was behind that deserves a medal.

    • Bill April 10, 2014 at 3:38 pm #

      When I started working at Bell Labs back in the late 70s, they made us all take a short technical writing course. The main emphasis was “write like a journalist doing a newspaper article, and STOP writing like a grad student doing an academic paper.”

      Think about what your readers needs to know, what they’ll be interested in, and hit them with the important points first so they know whether to take the time to read the rest. If you’re writing about complicated ideas, make your language simpler so they can spend the limited amount of attention they’re giving you on the ideas and not on how fancy your writing is. Your goal is to get the point across before tl;dr sets in.

    • Dereck Jansen April 11, 2014 at 8:01 am #

      I’ve read the comments below, and while I agree with most comments. I do find it strange that an SSL vulnerability that has existed for 2 yrs is brought to the world the same week that WindowsXP support stops, along with microsoft Essentials support… I think this elaborate marketing genius might be brought to you by our good friends at Microsoft… Stimulate OS and antivirus sales..
      Viola..Now saying that, I’m not downplaying the seriousness of the “Bug”, just keeping it in perspective..

  2. Tristan April 9, 2014 at 6:54 am #

    Right on. Who *was* behind the marketing of Heartbleed, anyway? I’d love to hear the full story on how this came about.

    • JP Lewicke April 9, 2014 at 7:25 am #

      I think that the marketing was done by Codenomicon, a security tools vendor that independently discovered and reported the Heartbleed bug: http://www.codenomicon.com/ .

      I wonder if this will become standard practice for major bugs found by security companies, since their products and services are likely to get some attention from the publicity attached to the vulnerability.

      • Noah Gibbs April 9, 2014 at 7:27 am #

        If it’s not it really, really should be.

        I’d like you to think about the amount of pagerank they just got by being the big link at the bottom of heartbleed.com for a moment.

        Better yet, I’d like all the *other* security companies to think about that, and then market their next serious vulnerability accordingly.

        • Tristan April 9, 2014 at 7:40 am #

          Is there any fear of the boy who cried wolf effect; of this type of marketing being used too often and the impact of each vulnerability being lessened by the confusion of this new marketing ecosystem? “Is this new vulnerability ‘Brainleak’ as bad as ‘Heartbleed’ or is it just another name?” I fear it becoming like Weather.com’s winter storm naming scheme: universally recognized as unreliable hype.

          Is there anything we can do to keep the marketing efforts as consistently effective as this first round?

  3. Ryan Platte April 9, 2014 at 8:02 am #

    I think of virus names as a middle ground between CVE identifiers and “Heartbleed”. This is a pendulum swinging through history—here’s an article complaining that taking the time to name viruses delays the response: http://www.brighthub.com/computing/smb-security/articles/52909.aspx

    (I tend to agree with the present article rather than that one, but the situation is somewhat different.)

  4. Josh Nankin April 9, 2014 at 8:20 am #

    Case in point, freeswitch logo: http://freeswitch.org/sites/all/themes/spreadfirefox/logo.gif

  5. Derek Anderson April 9, 2014 at 8:41 am #

    In a twisted way, the CVE number, obscure bug descriptions, and impenetrable nature of bug reports is a ‘feature’, not a failure. There is still a lot of wrongheaded obfuscation done by vendors in an attempt to hide vulnerabilities, and the associated bloody nose that a company gets when one is found in their product. While the CVE numbers aren’t built to provide that obfuscation by design, nobody at the enterprise layer is clamoring to change the process (except maybe to keep it as quiet as possible).

    The CVE also provides a context free placeholder for a bug that has been found, but not yet disclosed, which is probably an opportunity to set up the marketing message asynchronously to the original report I suppose.

  6. Florian April 9, 2014 at 10:21 am #

    Nice article.

    As a non-native speaker I found your thesis on the name evoking any sort of emotion especially interesting, for me was just a short “aha, nice pun” chuckle. That means I’m not totally sold on your interpretation, but an interesting point indeed :)

    • Tom Steele April 9, 2014 at 5:52 pm #

      This blog is a noteworthy piece of writing as well. I am in the media and we had to choose whether to talk about this over the past few days. You might think, “Of course!”

      But we are a top-40 radio station and this is outside our typical news reporting. The Heartbleed name made this so much easier to choose to report because it was so much easier TO report. Easier to explain and easier to write a memorable story about.

      Heartbleed was spread on news outlets that never would have mentioned it without the well-handled campaign.

      The only thing I didn’t agree with you about… The logo is perfectly fine! Lol.

      Tom Steele

  7. Brook Schoenfield April 9, 2014 at 11:49 am #

    Yes, everything you state, Patrick, is true except for one detail. As a security person charged with assessing heartbleed, I cannot rely on statements such as on the heartbleed web site; I must have hard technical details to assess. there were none. I had to wait until a PoC came out, then read that code in order to affirm that, yes indeed, the guys at Codenomicon had found a baddie.

    Good marketing, yes. Good technical source? No. Note even the CVE had any detail. somewhere, somehow, I would have appreciated a technical source, as well.

  8. -M- April 9, 2014 at 11:49 am #

    There’s some background of the site in
    http://techcrunch.com/2014/04/09/heartbleed-the-first-consumer-grade-exploit/

  9. Derek April 9, 2014 at 2:18 pm #

    On the other hand, naming can be done wrong. Look at what happens when the antivirus companies name viruses — each company gives different names to the same virus (e.g. Conficker = Downup = Downadup = Kido), resulting in mass confusion.

  10. debbie rosen April 9, 2014 at 3:50 pm #

    I completely agree that marketing serves an important service and, like it or not, if you want to spread a message and have people take it seriously, you need good “marketing”. The question i still haven’t heard much about is “What about the companies responsibility to inform their customers that their servers are now safe?” I couldn’t figure out if my bank or other sites had actually fixed the problem Maybe they need to read this article too.

  11. Mike April 9, 2014 at 6:00 pm #

    I agree on the evocative marketing approach for critical security bugs – until the point of diminishing returns. One critical bug with an evocative name becomes a major flaw begets a minor bug released creates a FUD campaign for attention. I would be concerned that Heartbleed has created an example for would-be attention seekers rather than serious contenders.

  12. adante April 9, 2014 at 6:52 pm #

    I’ll be among the first to admit the OSS community (and to an extent the broader technical community) does a generally poor job in properly marketing itself, but attributing VPs nixing nixing remediation efforts to _a failure of marketing_ seems extreme. I’m more inclined to say this a failure of technical competence by the VP. If they can’t understand the significance of a security exploit without having it properly packaged in _evocative buzzword bingo_ they should not be part of the decision making process.

    And yes – I can appreciate management are all too often promoted to their level of incompetence and this is how it is, and we have to deal with it. I just want to point out that thinking of this as a _failure of marketing_ is implicitly accepting a broken system.

  13. Hans Gerwitz April 10, 2014 at 2:33 am #

    I completely agree with your message, but think it might be more accurate to call this a success of “branding” not “marketing”.

    Serious kudos to Codenomicon for contributing to the resolution, and a big “grow the fsck up” to everyone who jumped to the conclusion that the discovery team distracted themselves and further jumped to publicly complaining based on their jumped-to-conclusion instead of contributing to the public outreach.

  14. Roy April 10, 2014 at 10:48 am #

    The question that I have is: If they spent this much time in preparing a polished PR message, might more consideration have been made into whether release of the openssl bug fix and propagation to various distros was the more important action to take time doing?

    It seems like the high visibility of the bug works against us as well as for us.

    • -M- April 10, 2014 at 12:41 pm #

      If you look into heartbleed.com, you’ll see that authors of the site weren’t the same ones that went public with the vulnerability.
      Also I’m pretty sure that such company has different people doing PR and vulnerability research, so those functions are not affecting each other.

  15. jrodman April 10, 2014 at 9:22 pm #

    The heartbleed branding was so strong that it took me around an hour of googling to get to the actual information past all the nonsense press stories that said nothing of value and did not link to the original site. I think the tone and concern got out far stronger than the actual information, and I think this is not surprising when the tone and broadcast of concern were prioritized this high relative to the core problem.

    • tclarke April 14, 2014 at 2:35 am #

      It took you an hour to find the top ranked site for “Heartbleed Vulnerability”, aka heartbleed.com?

      That’s probably on you.

  16. ročne ure April 11, 2014 at 1:54 am #

    Desirable component of content material. I simply stumbled upon your website plus accession cash to assert i get into point cherished profile the web site threads. By any means I shall be following with your augment and also I actually fulfillment you will get appropriate of entry to continually speedy.

  17. Jon Erlichman April 11, 2014 at 7:07 pm #

    I was receive lots of email regarding heartbleed, but my questions is why many software developers and also security expert did not aware this issues before? What happen with it? Are they did not have lots of experience?

    SSL security also use in many bank for protect security but they are using 128 bits of SSL, oh come one… should i change my password?

  18. Vivien Toledo   April 14, 2014 at 2:10 pm #

    Thanks for the great post. I’m glad it’ all mostly solved. Keep up the good work!

  19. Michelle April 15, 2014 at 2:05 am #

    Thanks for adding the marketing note. Yes for all products, software and inventory, marketing is what gets it to the finish line.

  20. Siberia Phorum April 15, 2014 at 4:47 pm #

    http://www.siberia-nw.ya.ru

  21. site nhu loz April 18, 2014 at 2:01 am #

    Outstanding overcome! I’d like to beginner while you actually amend your site, how do i join for a website web-site? This bank account made it easier for us a applicable option. I have been small bit common of the the sent out presented dazzling distinct plan

Trackbacks/Pingbacks

  1. Lessons in software bug marketing: Heartbleed’s evocative branding, dedicated web presence clearly communicate need for action (Patrick McKenzie/Kalzumeus Software) | NYC Startup News - April 9, 2014

    […] McKenzie / Kalzumeus Software:Lessons in software bug marketing: Heartbleed’s evocative branding, dedicated web presence cle… — What Heartbleed Can Teach The OSS Community About Marketing — If you’re a […]

  2. Lessons in software bug marketing: Heartbleed’s evocative branding, dedicated web presence clearly communicate need for action (Patrick… | Killer Apps TV - April 9, 2014

    […] Originally posted here: Lessons in software bug marketing: Heartbleed's evocative branding, dedicated web presence clearly c… […]

  3. Heartbleed: Yes, it's bad. Yes, the Russia mafia are probably involved. Now here's what to do – Telegraph Blogs - April 10, 2014

    […] "Heartbleed" by the researchers who have discovered it. It's been described as "catastrophic", "apocalyptic" and "the single biggest f––– up in the history of the […]

  4. The Heartbleed Bug | Continuous Updating - April 10, 2014

    […] If you don’t live behind the moon you probably heard already about the Heartbleed bug in openssl. This bug is so critical for the security of the internet that it even gets his own domain, logo and marketing campaign. […]

  5. Heartbleed and HealthIT | Health2.0 SF - April 10, 2014

    […] interesting is that this is the first bug disclosure that came pre-packaged its own marketing campaign, complete with dedicated web presence, graphic design and it’s own favicon. By doing so, the […]

  6. NEWS: HEARTBLEED BUG MAJOR WEB SECURITY FLAW | euzicasa - April 10, 2014

    […] What Heartbleed Can Teach The OSS Community About Marketing […]

  7. Friday always comes too late – TGIF! (28) - April 10, 2014

    […] What Heartbleed can teach the OSS community about Marketing […]

  8. ‘Brands face reputation risk over Heartbleed for not contacting users directly’ | The Best Spinner - April 11, 2014

    […] McKenzie, founder of Kalzumeus Software, wrote in a blog post entitled ‘What HeartBleed Can Teach The OSS Community About Marketing’: “Heartbleed is much better marketed than typical for the OSS community, principally because it […]

  9. What Can App Developers Learn From Heartbleed? | Deploy360 Programme - April 11, 2014

    […] McKenzie had a great post out titled “What Heartbleed Can Teach The OSS Community About Marketing” that nicely hits on key elements of why we’re seeing so much attention to this – […]

  10. Heartbleed Bug – Off topic but critical | Old Bones Genealogy of New England - April 11, 2014

    […] What Heartbleed Can Teach The OSS Community About Marketing […]

  11. The Heartbleed Bug | Internet Crime Fighters Org - April 11, 2014

    […] What Heartbleed Can Teach The OSS Community About Marketing […]

  12. Au Courant || WNBTv - will not be televised - April 12, 2014

    […] course there’s a Tupperware side of the issue as […]

  13. My Heart Bleeds (or, What’s Going On With Heartbleed) | The Licquia Blog - April 13, 2014

    […] from: the heartbeat signal bleeds data, so “Heartbleed”.  There’s been some fascinating commentary on how well this bug has been marketed, by the way; hopefully, we in the techie community will learn something about how to explain […]

  14. What to do about Heartbleed as a normal (non-geek, non-tech savvy) person - April 13, 2014

    […] What Heartbleed Can Teach The OSS Community About Marketing […]

  15. Les liens de la semaine – Édition #75 | French Coding - April 14, 2014

    […] La leçon de marketing que Heartbleed peut nous enseigner. […]

  16. Heartbleed, Sudden Change, and Managing Your Time - April 14, 2014

    […] What Heartbleed Can Teach The OSS Community About Marketing […]

  17. You reap what you sow | The Bankers' Bane - April 17, 2014

    […] What Heartbleed Can Teach The OSS Community About Marketing (kalzumeus.com) […]

  18. Heartbleed /3 | Ilcomizietto - April 17, 2014

    […] What Heartbleed Can Teach The OSS Community About Marketing di Patrick McKenzie […]

  19. Episode 18 – Heartbleed | tektalkv101 - April 17, 2014

    […] What Heartbleed Can Teach The OSS Community About Marketing […]

  20. alexhosein - April 17, 2014

    […]  McKenzie, Patrick . “What Heartbleed Can Teach The OSS Community About Marketing”.http://www.kalzumeus.com/2014/04/09/what-heartbleed-can-teach-the-oss-community-about-marketing/ […]

  21. Some Perspective on Heartbleed® | Techrights - April 18, 2014

    […] logo,” as a British FOSS professional put it. It’s like a branding exercise. Also see this post titled “What Heartbleed Can Teach The OSS Community About Marketing”. “Ties in a bit with what you’ve posted,” iophk told me after I had noted the […]

Loading...
Grow your software business:
(1~2 emails a week.)