What The Rails Security Issue Means For Your Startup

January has been a very bad month for Ruby on Rails developers, with two high-severity security bugs permitting remote code execution found in the framework and a separate-but-related compromise on rubygems.org, a community resource which virtually all Ruby on Rails developers sit downstream of.  Many startups use Ruby on Rails.  Other startups don’t but, like the Rails community, may one day find themselves asking What Do We Do When Apocalyptically Bad Things Happen On Our Framework of Choice?  I thought I’d explain that for the general community.

Nota bene: I’m not a professional security researcher.  Mostly, I sell software.  In the course of doing that, I (very occasionally) do original security research.  I did no significant amount for these bugs, aside from mitigating them for my own applications.  I am currently engaged in a Ruby on Rails security safari, and anticipate publishing the results of that in February, after responsible disclosure to the relevant security teams.  If you don’t know enough to know whether I’m trustworthy with regards to generic advice, pay someone you trust to get advice on this.

Don’t skip this post because you’re not a Rails developer.  If you’re reading this blog, this matters to you.

Background: What Has Been Happening in Rails-land?

Ruby on Rails recently released two sets of security patches (announcements here and here), in response to related vulnerabilities discovered in the frameworks.

How bad were those bugs? Severity: Apocalyptic.  They permitted attackers to execute arbitrary code on virtually ever Ruby on Rails application, without requiring that the application do anything to enable the attack other than “be hooked up to the Internet.”

What does “execute arbitrary code” mean?  Literally, it means that the attacker can choose to have your server execute any code they can dream up.  In practice, it means that you lose the server that the code is executing on.  Any further access to that server or applications on should be assumed to be compromised.

What went wrong?  This has been covered in more detail by security researchers, in posts such as here and here.  A brief description: Ruby on Rails makes extensive use of a serialization format called YAML, most commonly (you might think) for reading e.g. configuration files on the server.  The core insight behind the recent spat of Rails issues is that YAML deserialization is extraordinarily dangerous.  YAML has a documented and “obvious” feature to deserialize into arbitrary objects.   Security researchers became aware in late December that just initializing well-crafted objects from well-chosen classes can cause arbitrary code to be executed, without requiring any particular cooperation from the victim application.  Since then, the bug hunt has been on: security researchers have been actively finding lots of ways in the Ruby on Rails code base, and in related code bases, to cause the application to deserialize YAML which is in some way under the control of the attacker.

So far this has included:

  • Rails, for programmer convenience, used YAML to implement JSON deserialization.  JSON is designed to get into Rails quite easily indeed — just POST it at the server, wham, YAML.load(attacker_data) happened.  (The actual mechanics of achieving that were more complicated, but that’s the practical upshot.)
  • Rails allows XML documents to include YAML attributes.  That decision has caused a bit of head scratching, since it seems like a curious choice for most programmers in the community, but be that as it may this allowed posting XML at Rails apps to be trivially exploited.
  • Rubygems used YAML to hold metadata about each gem submitted to it.  An attacker was able to create a malicious gem, cause the Rubygems web application to evaluate the metadata contained in it, and thereby compromise the Rubygems server infrastructure.
  • February will see more compromises, with my certainty of this prediction approaching my certainty that the sun will rise tomorrow.  There exist many, many other code paths in Rails to get to YAML.load().  Some of them will be found to be amenable to attackers, either (worst case) for all or substantially all Rails applications or (still bad case) to Rails applications whose application logic involuntarily cooperates with the attack.  (i.e. In the worst case, attackers root every unpatched Rails app on the Internet.  In the best case, attackers only root some apps and they often have to have an expert do a modicum of marginal work to do so.)

Ruby on Rails security sucks lolz amirite? No.  Well, no to the nuance.  Software security does, in general, suck.  Virtually every production system has security bugs in it.  When you bring pen testers in to audit your app, to a first approximation, your app will lose.  While Ruby on Rails cherishes its Cool-Kid-Not-Lame-Enterprise-Consultingware image, software which is absolutely Big Freaking Enterprise consultingware, like say the J2EE framework or Spring, have seen similar vulnerabilities in the past.  The recent bugs were, contrary to some reporting, not particularly trivial to spot.  They’re being found at breakneck pace right now precisely because they required substantial new security technology to actually exploit, and that new technology has unlocked an exciting new frontier in vulnerability research.  It sucks for users of Rails that Rails is currently on the bleeding edge — believe me, after having lost 3 consecutive nights to patching my own applications, I know — but it would suck much, much worse if the Bad Guys had found these first and just proceeded to remote-own every Rails app on the Internet.  That is, by the way, an achievable scenario.

Was anyone actually compromised?  Yes.  The first reported compromise of a production system was in an industry which hit the trifecta of amateurs-at-the-helm, seedy-industry-by-nature, and under-constant-attack.  It is imperative that you understand that all Rails applications will eventually be targeted by this and similar attacks, and any vulnerable applications will be owned, regardless of absence of these risk factors.

Will anyone else be compromised?  Yes.  Thousands upon thousands of Ruby on Rails applications will be compromised using these vulnerabilities and their spiritual descendants, and this will happen for years.

  • Many Rails developers have not reacted to this news with the alacrity they should have.  (See next question.)  These applications may be compromised already.
  • There are many Rails applications which were created years ago, which are not under active development any more, for whom no-one is responsible for applying security patches.  Any of these applications which are publicly routable on the Internet will be compromised.
  • There are many Rails applications which are installed by end users, some of whom do not have security expertise.  For example, Redmine — an open source developer productivity tool — is commonly installed at individual companies.  Every publicly accessible Redmine instance which is not patched will be compromised.
  • Ruby on Rails lacks a CMS with the mindshare of, say, WordPress, which is good, because every unpatched Ruby on Rails CMS delivered to a non-technical company to serve as their website or backend to their mobile application will be compromised.
  • There are many developers who are not presently active on a Ruby on Rails project who nonetheless have a vulnerable Rails application running on localhost:3000.  If they do, eventually, their local machine will be compromised. (Any page on the Internet which serves Javascript can, currently, root your Macbook if it is running an out-of-date Rails on it. No, it does not matter that the Internet can’t connect to your localhost:3000, because your browser can, and your browser will follow the attacker’s instructions to do so. It will probably be possible to eventually do this with an IMG tag, which means any webpage that can contain a user-supplied cat photo could ALSO contain a user-supplied remote code execution.)
  • Many companies — including ones which do not even consider themselves Ruby on Rails shops — nonetheless have a skunkworks project running somewhere.  For example, they might have a developer who coded a quick one-off analytics app, which is accessible outside the firewall so that sysadmins could check server loads from home.  If the app is on the public Internet, it will be compromised.
  • Many Ruby on Rails shops have good development practices and no longer have the “monorail” anti-pattern, where everything their company does is in one gigantic Rails app.  They have already patched most of their main apps, but they missed one.  Maybe it is the customer support portal at admin.example.com.  Maybe it is a publicly accessible staging server at EC2 spun up by a developer who has since left the company and not shut down because, hey, $20 a month.  Maybe it is a 20% project by a junior engineer which he has on the back burner for the moment.  It doesn’t matter why this app was forgotten: if it is publicly accessible, it will be compromised.

What was the proper way to react to these patches?  Patch immediately, install a workaround immediately, or pull the plug on your application.  (“Pull the plug” means disconnect it from the Internet or shutdown the server while you get a mitigation plan into place.)  You should have distinct memories of you or someone under your employ having at least two separate incidents in the last four weeks in which they dropped everything they were doing and immediately took action to resolve these problems.  Immediately means exactly that: right now, not during the next schedule code spring, not tomorrow, not in an hour.

I was up at 3 AM Japan time applying these patches, twice.  If the next patch drops at 3 AM your local time, someone should be applying it immediately.  Computers can count to big numbers very quickly indeed.  A six hour window between a patch dropping and the start of business the next day is more than enough for an automated scanner running on a botnet to have tried compromising substantially every Rails app on the Internet.  (Do you disagree?  You are overestimating how hard it would be to find your application.)

Aren’t you exaggerating?  Our application isn’t particularly high risk!  We aren’t high-profile, it doesn’t have obvious monetary return for exploiting it, etc etc. Good thing you aren’t really saying that, but you might be at an Internet cafe next to an engineer who has poor reading comprehension, so help me explain this to him: nobody needs to care about your application to compromise it using these vulnerabilities. They can be exploited in a totally automated manner over the open Internet, requiring zero knowledge of e.g. what version of Ruby you are running, what version of Rails you are running, what your URL structure looks like, etc.  (Somebody suggested “How would you determine which servers were running Ruby on Rails?”  Answer: It’s absolutely trivial to detect Rails applications in a scalable fashion, but why bother?  Fire four HTTP requests at every server on the Internet: if the server is added to your botnet, it was running a vulnerable version of Ruby on Rails.)

Aren’t you exaggerating? Clearly this would take a lot of specialized expertise to exploit! Yep… the first time. Now that people know how the exploitation is done, however, you could do it by just copy/pasting one of the proof-of-concept scripts or b) running a browser bookmarklet. (I am not passing out that browser bookmarklet, because I think that would inevitably lead to mischief, but just know that you’re rootable in a click if you didn’t take action on this. And, by the way, have been for three weeks or so now.)

We’re A Startup.  What Happens If We Lose A Server?

If you lose one server, you will lose every server, with very high confidence.  If, for example, you are a Python-using shop which had a Redmine instance running around with no code on it, and you lose that Redmine server, you can expect a skilled attacker to then pivot from that privileged location within your network to start compromising other servers on your network.  At this point, you need to have done absolutely everything right to make it impossible for that skilled attacker to prevail, and you almost certainly have not.  (Compelling evidence that you’re not as good as you think you are: you already had one vulnerable application which could be compromised over the open Internet.  To a certain philosophy, that isn’t your fault, but the attacker gets root regardless of whose fault it is.)

The actual steps a pen-tester would take to root your other boxes are pretty academic after they have one.  (For example, you can start probing other machines on the network for vulnerable services, use credentials found on your compromised machine to suborn other machines, take over routing hardware using vulnerable administration panels and then start intercepting all network traffic, etc etc.)  Just take it as a given, you will lose.  Companies much larger and smarter than you lose everything when this happens, essentially every time it happens.

We’re A Startup.  What Happens If We Lose Every Server?

A short preview of coming attractions:

  • You will lose the next several weeks out of your schedule dealing with this issue.
  • You will have to take down all of your applications and rebuild all your servers from scratch.
  • You can assume the attacker now has a copy of your source code, all credentials you have, all your databases, and all information you had like e.g. log files.
  • Do you take credit cards?  Were you taking credit cards through an exploited application?  You now have a PCI-reportable data breach.
  • Your local jurisdiction may have legal requirements that you notify the people whose data just got exposed.
  • You now have a public relations nightmare on your hand.
  • In addition to compromising any customer data you possessed, you have made it possible for diligent attackers to compromise those customers elsewhere.  The most trivial example is, if you did not implement password storage correctly, you have just handed the attackers a list of email addresses and associated passwords which they can now re-use on higher value targets like e.g. bank accounts or Gmail, because many users re-use their passwords everywhere.  (You use bcrypt?  Wonderful.  Did  the attackers turn it off when they rooted all your applications?  Can you conveniently check that, knowing that you cannot trust the contents of any logs on those compromised servers?  No?  OK, so instead of losing all the passwords, we can upper bound exposure at only all users who logged in since the attack started.  That’s an improvement… sort of.)

Basically, it’s Very Bad, but not the end of the world.  You’ll probably need expert help to get through it, like you would need if e.g. you got sued.  Unfortunately, lawsuits generally give you weeks of notice and progress slowly, but security vulnerabilities often give you negative several hours notice and get worse for every minute left unchecked.

We’re A Startup.  We Don’t Use Ruby on Rails So We’re Totally Cool, Right?

Can you enumerate every account on the Internet where you have a password and also every service consumed by your business?  Go ahead, take as long as you need: it is very important that you don’t miss one.

OK, let’s start with the obvious: Look for analytics providers and other folks on that list who have instructed you to embed JS on your website.  If I do this exercise, I come up with at least three results here.  Do any of them use Ruby on Rails?  (Are you sure?  Remember, if they have at least one Rails app on their network…)  Great.  If they didn’t patch in a timely manner, you should assume that Javascript you’re embedding on your website is in the hands of the enemy.  It is now a cross-site scripting vulnerability against every page it is embedded on.  Do you embed it on e.g. log in pages or anywhere your admins expose their own all-powerful admin cookies?  Boo, now the enemy has your password / cookies / etc.

Alright, let’s move down the line: Look for anybody who implements OAuth/Facebook Connect/etc.  Do any of them use Ruby on Rails?  Are you sure?  If they haven’t patched, you’ve handed the union of all privileges over the linked accounts to the attackers.

Alright, let’s move down the line: Consider everybody who has a copy of a password which you re-use elsewhere.  (You shouldn’t be re-using passwords, or variants of passwords, but I ignored that advice for years so I’m betting a lot of you did, too.  Maybe not you, specifically, but you know that chap in marketing who is great with people but thinks MSWord is complicated?  Consider whether he has access to anything sensitive in your company.  He does?  Well, sucks to be you then, but good on your for password security.)  Do any of them use Ruby on Rails?  Are you sure?  Did they use bcrypt/scrypt/etc to properly secure passwords at rest, and did they patch these vulnerabilities fast enough to prevent attackers from pulling them off of the wire?  Are you sure?  If you’re not sure of all of these things, consider every password compromised and take action appropriately.

One of my friends who is an actual security researcher has deleted all of his accounts on Internet services which he knows to use Ruby on Rails.  That’s not an insane measure.  (It might even be inadequate, because all the folks who are compromised are probably going to lose their database backups as well.  Well, if they have database backups.)

These are just a sample of ways in which these vulnerabilities can ruin your day.  They are very much not an exhaustive list.  If you believe in karma or capricious supernatural agencies which have an active interest in balancing accounts, chortling about Ruby on Rails developers suffering at the moment would be about as well-advised as a classical Roman cursing the gods during a thunderstorm while tapdancing naked in the pool on top of a temple consecrated to Zeus while holding nothing but a bronze rod used for making obscene gestures towards the heavens.

Somebody Dropped A 0-Day On Rubygems. What If It Happens To Me?

Yes, that certainly sucks royally.  Rubygems wasn’t even exploited using the patched Rails vulnerabilities — an attacker just learned something which worked (again, we’re on the leading, bleeding edge of security research here), applied it in a novel fashion, and compromised the Rubygems application.  As of me writing this it looks like we avoided the Ruby-ecosystem-wide apocalypse that would have happened if they had started backdooring gems, but let’s just focus on the immediate fallout: their system got compromised.  What if one of yours did, like that?

The first step is a preventative inoculation: If you run an application on the Internet, you should today establish a security contact page.  It only needs to include two things: a working, monitored email address and a PGP key.  Bonus points for giving some sort of public recognition to people who report security vulnerabilities to you in a responsible matter.  This helps to co-opt some security researchers so that they e.g. get in touch with you about the problem prior to just going ahead an exploiting it.  Software security has a curious system of social norms, where scalp collecting both builds both karma and pseudo-currency.  It’s bizarre, but just take this on faith: having a security page with a working email gives you a certain amount of We Should Avoid #’#(ing Their #()#% Up Without Asking First street cred.  (Naturally, like any social norm, that is a preventative measure rather than a panacea.  However, given that it is a well-understood norm, it gives you a bit of an edge in the PR battle should someone decide to just drop a 0-day on you.)

Good security pages to pattern after: 37signals (I particularly like how this page works for responsible disclosure while, in a dual-audience fashion, also doubles as being great marketing copy), Twilio, Heroku (again, dual audience), etc.

Have a plan for responding to security incidents. I call mine the Big Red Button. Thomas, a security consultant friend of mine, accurately observed that these probably caused the first Big Red Button events that many folks in the Rails community have ever had to deal with. We should learn from our experiences here.

For example: I pushed the Big Red Button at 3 AM in the morning, twice this month, to apply critical security patches and work-arounds.

So did I do a great job of addressing this problem? No, I did a pretty effing atrocious job of addressing this problem. Specifically, I have two old-as-the-hills Rails apps running on 2.3.X at the moment. Waaaaay back in 2010, Mongrel and Rails had a bit of a compatibility issue, and I solved it via a monkeypatch. The monkeypatch relied on a hardcoded version number, which I have been hand-incrementing every time I update Rails. It’s literally on the redeploy checklist, next to a note “TODO: This is stupid and should be fixed when I get a moment.”

I did three Rails app upgrades locally, three test suite runs, and three sets of smoke tests when applying one of these patches. The one in the middle happened to be Appointment Reminder, which is an application that has to be up during the US workday. Unfortunately, because I was exhausted while following my deployment and smoke test checklists, I a) forgot to bump the version number in that monkeypatch and b) did not follow the part of the smoke test which would have clued me on to “This is going to cause log-ins to fail on some browsers.” That resulted in some breaking downtime for some customers during the US workday, and me having to send an apology to all customers. That sucked horribly.

I have now fixed my monkeypatch to not require hard-coding the Rails version, simplified some of my deploy procedures, and am working in the next several months on beefing up my testing suite. Also, lesson learned about resolving “TODO: This is stupid” when it would take 5 minutes to do rather than having it blow up in my face.

There, that’s an experience I went through. Now you know the punchline, so hopefully you don’t have to go through it as well.

Similarly, we can observe:

  • We need an updated list of all applications running on our servers, so that we know when a problem with a technology stack affects them, even though this sounds like a boring Big Freaking Enterprise IT Department requirement. (And gulp their dependencies.)
  • For each tech stack we support, we need at least one expert following the primary source for security news for that tech stack.
  • We need whomever is responsible for product development and/or ops to, effectively, carry a pager for drop-everything-and-do-it-now resolution of security issues, just like we’d do for e.g. the server has fallen over or “our building is, physically, on fire.”
  • These requirements suggest minimizing the number of tech stacks we work with, even if that means passing up the new hotness occasionally.
  • Just like we have e.g. insurance on the building physically burning down, we should have some upfront investment in security. Good forms might include security training, outside consulting, or (if we’ve got a lot of money) contributing work towards securing tech stacks we rely on.

You Should Be At Defcon 2 For Most Of February

Big security vulnerabilities tend to be discovered in bunches.

Why does this happen?

  • Blood in the water attracts sharks. Some of my security friends would hate this phrasing, because “researchers don’t cause vulnerabilities, they find vulnerabilities”, but as a businessman who depends on software for his livelihood, I had exactly zero days of the last six years spent sleepless because of the latent vulnerability in Rails, but two days this month due to the pressing need for immediate mitigation. There are many more eyes pouring over Rails — and related projects — more closely now than typically. Many of them are white hats (yay!). Some aren’t. In general, there is a virtually infinite need for software security expertise, just like there is an infinite need for software, and there is a crushing lack of expertise which can meet it. Some folks who are capable of finding vulnerabilities are, due to attention/topicality/renewed interest/commercial potential/etc, now looking at Rails as of today.
  • Technology marches on. After you have a new exploit vector to play with, you can start applying some of the technology used to discover / develop / exploit it against other code bases, code paths, etc etc. For example, the first Rails vulnerability was parlayed within a day into a similar vulnerability in the MultiXml gem. The same underlying “YAML is very dangerous” realization enabled the Rubygems compromise. If I were working on e.g. Django, I would strongly suspect that security researchers are going to see whether they can find similar patterns on Django — it wouldn’t be the first time, since e.g. HMAC tampering vulnerability disclosures in Rails were followed up by similar findings on Django the same week.

I previously had a version of this post queued up right after the first bug dropped, but didn’t hit Publish because I got busy that weekend and thought it wouldn’t be timely anymore. That post included the lines “I will bet $1,000 at 100-to-1 odds that Rails suffers another code execution vulnerability before the end of January.” If you had hypothetically taken that bet, you would have lost.

You should expect February to be a very trying month for the Rails community and startups in general. Your security team should be at Defcon 2: be ready to respond to patches with particular alacrity, and expect there to be failures in the ecosystem outside of your ability to control them. For example, I’d make sure that you can rebuild systems without requiring access to Github / Rubygems / etc, and that’s (unfortunately) the tip of the iceberg.

This Sounds Like A #$()#%ing Disaster

That is primarily because this is a #$()#%ing disaster.

For my part, in addition to taking steps to fortify my own businesses, I’m (as time permits) doing some pro-bono security work on Rails. I do not have results which can be published yet. I strongly suspect based on early research that I will, in February, and I strongly suspect that other researchers (both white hats and the Bad Guys) are much, much better at this than I am.

Get ready. It will get worse before it gets better.

About Patrick

Patrick is co-founded Starfighter, founded Appointment Reminder and Bingo Card Creator, and presently works at Stripe on Atlas. (Opinions on this blog are his own.) Want to read more stuff by him? You should probably try this blog's Greatest Hits, which has a few dozen of his best articles categorized and ready to read. Or you could mosey on over to Hacker News and look for patio11 -- he spends an unhealthy amount of time there.

70 Responses to “What The Rails Security Issue Means For Your Startup”

  1. Henley January 31, 2013 at 8:29 am #

    Is there a list anywhere of ALL the vulnerabilities and fixes that have been found so far? I know the YAML one is a big one but is there a consolidated list somewhere? I know I can upgrade to the latest Rails and that might fix all of them, but for some people that’s not a possibility.

  2. Patrick January 31, 2013 at 8:37 am #

    The Ruby on Rails security announcement mailing list will list most new security patches coming on Rails 3.2.X, 3.1.X, and 2.3.X. This is not necessarily all of the threats you will be dealing with in the next month. If you are not sure you are good, take your application offline and seek the advice of a security professional.

  3. KaeseEs January 31, 2013 at 8:43 am #

    But you see these issues aren’t such a big deal, rails is ~omakase~

  4. David January 31, 2013 at 10:45 am #

    Every big framework has had security exploits like these. I work at a start up myself, and we use Ruby on Rails. We have about 12 clients, and all 12 servers were patched immediately. Not harm was done, but that doesn’t mean much. I think it means more if a start-up is not patching their servers more so than it does for the Ruby on Rails framework.

  5. Bartosz January 31, 2013 at 10:53 am #

    You’re being over dramatic about this.

    The more bugs are exposed the more are fixed. Personally, I think that’s a good thing. Every framework has bugs like these.

    • Fuzz January 31, 2013 at 5:17 pm #

      No, the Rails community has downplayed security with attitudes like this for years; it is just a matter of time before an actual apocalyptic event finishes convincing people.

      Yes, exposing bugs is great. That does not mean the author is being overdramatic about what needs to happen. First startup I worked at was in 2000, I found a compromised system and recommended exactly the above. I was told I did not understand how much work it would be to wipe machines and rebuild from scratch but I most certainly did–I just understood the danger of not doing so. We cleaned up best we could without wiping but never got rid of the intruder, I kept finding evidence. That happened to be a worst case scenario, we were trying to compete with a company that was also a customer and someone from their company broke in and in accessing our email systems was able to learn that if they withheld a certain payment we would be forced to shut down most of our operations immediately. And that’s what they did. Overnight almost everyone lost their jobs because we could not make payroll otherwise. I will never forget that morning the CEO with his hands in the air saying, “How did they know!?” and then suddenly looking at me with horror and regret.

      • Mike February 1, 2013 at 5:59 am #

        Who are these people you know that have lead you to believe “Rails community has downplayed security” as if the rails community is somehow apart from web developers in general? Anyone that downplays security is simply a bad developer for real projects.

  6. Scott January 31, 2013 at 12:18 pm #

    Great post. Thx!

    Probably a little over alarmist, but its good to see the issue from this perspective.

    Great for convincing the bosses they need to spend on this.

    Thx again

  7. Dimetry January 31, 2013 at 12:42 pm #

    2 vulnerabilities in past year? Таке a look at the enterprise sector (java, for example) and take a breath

  8. Elsebeth January 31, 2013 at 12:42 pm #

    Thank you. I’d reacted to the first notices about the YAML vulns, but I hadn’t thought through the implications for e.g. analytics code and backburnered Rails 20% projects. I think I closed three major vulnerabilities since reading this and forwarding it to my colleagues. :)

  9. Scott Nixon January 31, 2013 at 1:24 pm #

    It’s no shortcut for patching your servers/apps, but I’d also recommend using something like Cloudflare to help protect against distributed attacks. If they see sources that look like attacks they will block all traffic. Security is a layered approach.

  10. Alan January 31, 2013 at 2:43 pm #

    This was clearly written for developers using Rails. It sounds alarmist, because it was meant to. For developers using Rails, it’s probably best to err on the side of caution, so that’s good. This article was not meant to provide a balanced view of the actual security situation with Rails today. It was not meant to say that Python is somehow better than Ruby, or any nonsense like that.

    I maintain a large Rails app, and when I saw the first advisory, I checked our code to see if we were vulnerable. We weren’t (but we upgraded anyway). It isn’t literally every Rails app that is vulnerable: to be open to a YAML attack, you have to have some HTTP endpoint which accepts and deserializes untrusted YAML. Not every app does. The patches for these issues are downright tiny, because they only need to cover one specific hole.

    Furthermore, just because you’ve upgraded to Rails 3.2.999 doesn’t mean you’re immune to security flaws. It just means you’re not vulnerable to the same security holes that everybody else was, and so your app can’t be trivially attacked in an automated fashion. I bet a good percentage of Rails apps still have gigantic homemade security holes.

    The lesson people should take from this is to be proactive with security. One specific case of that is keeping up-to-date with the latest security patches for all the libraries you’re using. That’s good practice, but on its own that is neither necessary nor sufficient.

    • Donald Ball January 31, 2013 at 6:28 pm #

      I’m sorry, you’ve been misinformed. All Rails applications were exploitable by the YAML attack by default. You would only have been protected against that exploit if you’d disabled the XML and JSON params parsers.

  11. Daniel January 31, 2013 at 2:53 pm #

    Is this issue really that bad? If the application is running with the least possible privileges (ie, not as root) only the app would be compromised, not the whole server.

    • Fuzz January 31, 2013 at 5:02 pm #

      In the world of Unix security it is considered a given that a compromised user account will eventually lead to root access, the only question being how long it will take.

      • dawmail333 February 1, 2013 at 1:39 am #

        I understand the value of worst-case assumptions, but wouldn’t that assume that the UNIX security model is broken at the core? :/

        • barrucadu February 1, 2013 at 3:53 am #

          It’s not broken at the core, but new privilege-escalation exploits can always be found – and are found. The only solution is to fix the problem before they gain root and, if you’re unsure at all, assume the worst case.

          • dawmail333 February 1, 2013 at 3:58 am #

            Fair enough, but I figured most of those would be ‘social-engineering’ attacks – e.g. modifying something the root user would call. On an automated system, that wouldn’t be a problem if root was never really used.

  12. Andrew January 31, 2013 at 6:40 pm #

    This is a terrible article. Really, it’s a fucking atrocity and represents everything wrong with the web development ecosystem. Here’s an idea. Instead of ranting about how every Rails application is compromised and everyone is screwed how about you actually explain how you’re supposed to protect against this vulnerability and not come off like such a pompous douche. I’m really tired of shit like this, where people tell you that all Rails applications are compromised and don’t offer any solutions to help you fix it.

    Maybe to you saying something like “patch your application immediately” makes perfect sense, but I have no idea how you’re supposed to upgrade a legacy Rails application to a new version. To a lot of people who have made Rails applications or are currently working on them, “patch your application immediately” is a completely meaningless phrase and they have no idea what you’re actually supposed to do to fix these problems.

    If you really wanted to help fix these vulnerabilities you would spend 2,500 words explaining how to patch against the exploits and secure your application. Junior developers need examples of what you’re supposed to do if you’re running a legacy Rails version to get it to work with the security fixes not rants telling you that you’re an idiot if you don’t upgrade.

    Seriously, try helping people out next time instead of being such a condescending douchebag.

    • Elsebeth January 31, 2013 at 7:14 pm #

      Really, hon? Shooting the messenger is oooolllld school, like Roman times. :P

      There was nothing wrong with the article’s tone or intent. I found it very useful and, with irate comments like these, entertaining. I smell psychology of previous investment.

    • Richard January 31, 2013 at 9:56 pm #

      This is a terrible comment. It’s an %$*! atrocity and represents everything wrong with public anonymous comments.

      Rather than bitch about his article, why not post your own article with specifics on how to patch the vulnerabilities?

      Not a Ruby dev, so I can’t help you.

    • Evan R. Murphy January 31, 2013 at 11:56 pm #

      You had a question – just ask it. No need to sling insults at the author.

      Regarding your question about patching, I found the original announcements of the security vulnerabilities to have pretty clear instructions about what to do:

      * https://groups.google.com/forum/?fromgroups=#!topic/rubyonrails-security/61bkgvnSGTQ

      * https://groups.google.com/forum/?fromgroups=#!topic/rubyonrails-security/1h2DR63ViGo

      Basically, in all the Rails apps that you maintain, you just need to upgrade the version of Rails you’re using to one in which they’ve fixed the vulnerabilities. You can do this by changing the version number of the “gem ‘rails'” line in your application’s Gemfile and then running “bundle” from the command line inside your application’s root directory. Finally, test to make sure everything works, and deploy.

      In case you can’t upgrade for some reason (like if you found in testing that upgrading Rails broke some part of your application that can’t be easily fixed), they suggest some ways that you can work around the vulnerabilities without having to upgrade Rails.

    • Matteo February 1, 2013 at 1:37 am #

      > To a lot of people who have made Rails applications or are currently working on them, “patch your application immediately” is a completely meaningless phrase

      Those people shouldn’t be in the ICT business in the first place, then.

      • Andres February 1, 2013 at 8:09 am #

        Yes they should. And that pretentious attitude is what is so off-putting to beginners in the first place.

        The entire article is saying “I patch my Rails applications so you should too” without explaining how. It’s really really useless and condescending and is a serious negative addition to the Rails ecosystem.

          • Jonas February 1, 2013 at 11:34 am #

            How the fsck does that help me, as an ops guy who actually has the responsibility (!) to keep stuff running smoothly, when I have a multitude of apps running some Rails 1.x or 2.x version?

            (Not more than three or four years old perhaps, and keep in mind that ZERO of my Perl, Python or even Java apps gives me these problems. Not to mention the black magic I’ve had to endure over the years to get this crap to scale.)

            Please do tell.

    • Neil February 1, 2013 at 5:40 am #

      You’re an asshole, Andrew. I agree with everything Elsebeth, Richard and Matteo said in their replies. This is a free blog — Patrick isn’t being paid to work for you! I’m glad to read Patrick’s blog; it puts a sense of urgency for us to review our code base, servers, etc. for vulnerabilities.

    • Jason February 1, 2013 at 5:07 pm #

      That’s a lot of typing and bad energy, that could’ve been channelled into, finding the patch you need, and reducing your blood pressure.

    • Jez February 3, 2013 at 8:22 am #

      The author offers a great deal of helpful advice and meta-advice in this post.

      If you can’t get to detailed instructions on how to patch Rails in less than a minute using Google, you shouldn’t be in IT.

      If you can’t learn by yourself how to patch the exploits and secure your application, and to take down any vulnerable services in the meantime, you shouldn’t be in IT.

      As can be seen on this comment thread – in spades – many people in our community have a highly developed ability to downplay these kinds of events and re-interpret what they “read” such that it can’t possibly apply to them.

      The author performs a great service trying to get it into developers thick heads that actually they need to pay attention to this, and offering a great deal of valuable information on the possible impact of this kind of vulnerability and how to think about security in general.

      Your assertion that the author is a condescending douchebag just makes you look defensive.

  13. Scott Bartell January 31, 2013 at 10:51 pm #

    FYI, “on virtually ever Ruby on Rails application”, it should be “every” not “ever”.

    • Noah Vawter February 1, 2013 at 10:23 am #

      Nice catch, dude!

  14. Ryan Hellyer February 1, 2013 at 1:29 am #

    Nice article. I’m a WordPress developer and am just waiting for the day we get our first zero day exploit like this. Fingers crossed it never happens, but if it does then I’m gonna be happy to have a solid backup system in place.

    • das February 1, 2013 at 8:30 am #

      There has been (and still are) zero day exploits for WordPress, but the difference here was that the problem was in the deeper level than just in a CMS application.

      Also, there have been, are, and will be many zero day exploits for PHP. Many of them haven’t ever been published. It’s a game of shadows and there are many individual researchers and even few security companies who make money selling exploits for the highest bidder.

      Point is, regardless of the application or backend software stack, things should be kept updated. Even that is not enough: problem is that security of service development should be proactive, actively trying to find problems in their own systems (while even that does just make things a little bit less potential). Another side of problems is that things have gotten so complicated that even most professionals can assume that it’s not whether security breakins happen, it’s matter of when. Risk can be reduced (and should be) but it is always there.

  15. Ben February 1, 2013 at 2:07 am #

    What a load of overblown shit.

  16. makomk February 1, 2013 at 3:37 am #

    “If I were working on e.g. Django, I would strongly suspect that security researchers are going to see whether they can find similar patterns on Django”

    I’d be disappointed if they do find any. This class of vulnerabilities has been well known in Python for literally years (though the Python equivalent is slightly more powerful, since Python doesn’t really distinguish between classes and other callables).

  17. Jason February 1, 2013 at 3:54 am #

    Embedding third-party JS from a compromised site will not provide them access to cookie data from the originating site because cookies are restricted per domain. The only scenario I can think of where this could be the case would be a cookie from the wild-card domain and where the embedded JS is on a subdomain of the original site.

    • dawmail333 February 1, 2013 at 4:00 am #

      I thought this was the case, but not being 100% sure, I didn’t want to risk spreading incorrect information.

      Malicious javascript IS still a problem though, which is why I only ever use Google Analytics and JQuery as my remote javascript calls, if I can help it.

    • Shannon February 1, 2013 at 7:46 am #

      You are missing the point. The YAML is executed by Ruby, not the browser, and you can call arbitrary code.

      The impact is at the server level. The browser is not relevant.

      • Wes February 1, 2013 at 1:00 pm #

        I believe the browser becomes relevant if that’s the mechanism for launching a request at something running on localhost://3000, or on some other internal server not accessible to the public internet. Those sites can’t be accessed directly from the outside, but they can from my browser. If javascript from an external site can hit them though, they suddenly become vulnerable.

    • Nick Grimshaw February 3, 2013 at 3:05 pm #

      The phrase you’re looking for is “Cross Site Request Forgery”.

      If you’re executing JS from a compromised site, that JS could cause a request to be sent to the Facebook endpoint that changes your password, and your browser will cheerfully send your Facebook cookie to Facebook. The compromised site doesn’t get the Facebook cookie, but it *can* get sent your Facebook username and new password.

    • dino February 4, 2013 at 3:11 pm #

      Cross domain restriction doesn’t apply. If you load a library from a compromised 3rd party site, the code is executed in your domain. That code can collect any info and send it back to the bad guy.

    • Danillo Nunes February 13, 2013 at 7:51 pm #

      It’s true that a cross-domain JS cannot read your cookies, but there’s many ways as it can compromise your visitor’s data. For example, it can alter the whole page to simulate a “expired session” scenario and asks your visitor’s username and password, which he’ll provide without even think because the form is in your site after all.

  18. Jonas February 1, 2013 at 11:27 am #

    What every commenter here seems to miss is that when every seasoned programmer, including myself, learned Rails they went “ooh, aah” and then “wat! this is batshit insane! why would you want to do _that_?!?”.

    (Whereas every OO-head who found refuge from Java-land whined about haters of course, and did not pay one single bit of attention. They went on bro’gramming ecstatically and making up new puns to name their next authentication package. All of which were found to be severly broken of course.)

    That’s on everything from automatic deserialization of code, decoding of yaml-in-xml, obvious complexity attacks, etc. And then Rails blew up hard but no one cared until Github was pwned. And then Rails blew up again. And again.

    Not to mention all those authentication packages, which every single one failed to keep the bad people out. And now the main repository of packages turns out how have been controlled by others for god knows how long.

    But somehow that’s all ok, since one or two Java or Python actually also had a security issue a few years ago. (That may have turned out to be an XSS in some admin interface of course, while Rails actually executes code from the Internet of course, but who’s counting?) Yeah, why learn anything? Who cares? Everything has security holes! It’s just the State of Software(tm)! It’s not my responsibility, that can some sysadmin in operations (like yours truly) take care of.

    • cha0s February 3, 2013 at 12:28 am #

      Shhhhhh, Rails is Omakase.

  19. Ryan Hellyer February 1, 2013 at 12:53 pm #

    Hi, I’m just wondering if anyone here knows if it is possible to setup Rails to auto-update? (I don’t actually use Rails, I’m just interested).

    I see a lot of suggestions in the comments stream for people to patch their systems, but I always assumed Rails would have some sort of auto-update system built into it to make that sort of thing happen by default. WordPress has a one click upgrade system, and if you want to you can just use SVN to update your install once every few hours. It seems that a system like that would at least avoid issues of n00bs not bothering to update their Rails installs.

    Or is there something obvious I’m missing here?

    • Evan R. Murphy February 1, 2013 at 1:29 pm #

      By default, Rails does auto-update (as do all gems), in the sense that the simplest thing to put in your Gemfile is “gem ‘rails'”, which will fetch the latest stable version every time you run “bundle”. However, I have found this sort of auto-updating to be problematic because it breaks dependencies, so I consider it a necessary evil to hard code all version numbers and only do updates manually, when there’s time to thoroughly test and make sure nothing breaks.

  20. Ben February 2, 2013 at 5:34 pm #

    In regards to Redmine being hosted on a server, would it be sufficient to just use http authentication on the domain to protect it?

    I’ve already upgraded rails to 3.2.11 as well.

    • Mark IJbema February 6, 2013 at 12:07 am #

      It would make an attack less likely. A remote computer cannot exploit it anymore. But it could still be exploited by you, your browser, and sites your browser goes to. So if you use a seperate browser for redmine and use http authentication it would be safe (but if you have no means of upgrading, using http authentication would be a good intermediate safety measure)

  21. Michael Selik February 3, 2013 at 1:38 pm #

    That’s just implausible. A classical Roman would be dancing at Jupiter’s temple, not Zeus’.

  22. Dan February 4, 2013 at 1:06 pm #

    For Ruby developers looking for a quick, stopgap fix against YAML deserialization exploits, it’s worth taking a look at SafeYAML (https://github.com/dtao/safe_yaml), a gem which prevents arbitrary object deserialization by default. It won’t be an appropriate solution for every application (it may break some functionality that relies on YAML parsing, for example), but for most websites that don’t leverage the full power of Ruby’s YAML library, it’d worth considering.

    Full disclosure: I’m the author of the gem, so obviously take this with a grain of salt.

  23. Jamon Holmgren February 6, 2013 at 10:27 am #

    I built a lightweight gem and web app for tracking your Rails app versions. It’s too difficult to remember every little Rails app we’ve deployed and what version they’re on.


    It’ll email you if something is out of date. Super easy to install too.

  24. Fuzz February 6, 2013 at 11:02 am #

    In response to this I have written up some bits about hardening your infrastructure to mitigate the need for a full rebuild following a compromise.


  25. Tim February 10, 2013 at 7:25 pm #

    Great read.

    As a non-developer but as a site owner I’d like to know

    – what major 3rd party apps are affected, e.g. Analytics, Crittercism, Chartbeat, etc etc
    – what would happen if you were executing JS calls to affected Rails apps, from within a php environment, on a shared server? Because in that case, with a shared server, all hell could break loose (and where does the responsibility lie? I don’t know if the server owner can quarantine the damage to your site, whether it’s virtualised or not) — as I said, I am not a developer/security/sysops person.

    I’m also super interested in your comment on putting a Security page up with simple contact form and pgp key. How and why do you implement the pgp key?



  1. If you’ve ever had ANYTHING to Do with Rails. ever. Please read this now. | kangaroo court - January 31, 2013

    […] What The Rails Security Issue Means For Your Startup! […]

  2. What The Rails Security Issue Means For Your Startup | Kalzumeus Software | Ethical Hackers for Hire. Hacker for Hire, Hire a Hacker, Hack Facebook - January 31, 2013

    […] URL: http://www.kalzumeus.com/2013/01/31/what-the-rails-security-issue-means-for-your-startup/ […]

  3. What The Rails Security Issue Means For Your Startup | Kalzumeus Software | Hacker for Hire, Hire a Hacker, Hack Facebook Ethical Hackers for Hire. - January 31, 2013

    […] URL: http://www.kalzumeus.com/2013/01/31/what-the-rails-security-issue-means-for-your-startup/ […]

  4. What The Rails Security Issue Means For Your Startup - February 1, 2013

    […] wise, I would still suggest having a secure CDN can reduce the zero-day attacks to a minimum. What The Rails Security Issue Means For Your Startup | Kalzumeus Software […]

  5. pinboard February 1, 2013 — arghh.net - February 1, 2013

    […] What The Rails Security Issue Means For Your Startup | Kalzumeus Software […]

  6. Fluffy Logic » Blog Archive » I knew rails couldn’t be trusted… - February 1, 2013

    […] Here’s why. […]

  7. Design for Upgradability and the Rails DigiD Outage | Arie van Deursen - February 4, 2013

    […] a more detailed account of the impact of the Rails vulnerabilites, have a look at What The Rails Security Issue Means For Your Startup by Patrick […]

  8. Reading Notes 2013-02-04 | Matricis - February 4, 2013

    […] What The Rails Security Issue Means For Your Startup (Patrick McKenzie) – A security breach in Ruby on Rails, comprimising a huge part of web data. […]

  9. What The Rails Security Issue Means For Your Startup | Kalzumeus Software | WebTekNeqs' - February 5, 2013

    […] What The Rails Security Issue Means For Your Startup | Kalzumeus Software. […]

  10. Did you work on a Rails app? It’s extremely vulnerable. Tell someone to update it! | Art & Logic Blog - February 5, 2013

    […] McKenzie’s excellent blog post inspired this commentary.  Read on for a reiteration of his […]

  11. En dårlig måned for Ruby on Rails | Hennings blog - February 6, 2013

    […] on Rails, et web framework til Ruby, har ikke haft en god januar og der går nok lidt tid før de afslørede sikkerhedshuller er lukkede. Sådan noget sker og på […]

  12. Ruby on Rails Security Vulnerability Throws Apps Off Track | Software Testing Blog - February 7, 2013

    […] Today, anyone who runs a Ruby on Rails server who hasn’t applied an update is probably already compromised. Think that’s overstating things a bit? Patrick McKenzie sounds the alarm loudly in his blog post titled What The Rails Security Issue Means For Your Startup: […]

  13. De-Railing Security Bugs - Pittsburgh Web Design & Hosting - February 8, 2013

    […] Patrick McKenzie noted, this may affect you even if your project doesn’t use Ruby or Rails—it could even impact you if […]

  14. Noticias 08-02-2013 - La Web de Programación - February 8, 2013

    […] El grave agujero de seguridad de Ruby on Rails y su gravedad: Veredicto: Muy grave. […]

  15. De-Railing Security Bugs | Underwaterpistol - February 9, 2013

    […] Patrick McKenzie noted, this may affect you even if your project doesn’t use Ruby or Rails—it could even impact […]

  16. Off the Rails | Coder Radio 36 | Jupiter Broadcasting - February 11, 2013

    […] What the Rails Security Issue Means… […]

  17. Critical Rails Security Issue | The Ruby Railroad - February 13, 2013

    […] http://www.kalzumeus.com/2013/01/31/what-the-rails-security-issue-means-for-your-startup/ […]