Fixing the “Heartbleed” OpenSSL Bug: A Tutorial for Sys Admins

So what exactly is the bug anyway?

Here’s a very quick rundown:

A potentially critical problem has surfaced in the widely used OpenSSL cryptographic library. It is nicknamed “Heartbleed” because the vulnerability exists in the “heartbeat extension” (RFC6520) to the Transport Layer Security (TLS)  and it is a memory leak (“bleed”) issue.  User passwords and other important data may have been compromised on any site affected by the vulnerability.

The vulnerability is particularly dangerous for two reasons:

1. Potentially critical data is leaked.
2. The attack leaves no trace.

The affected OpenSSL versions are 1.0.1 through 1.0.1f, 1.0.2-beta, and 1.0.2-beta1.

Who is affected by the problem?

Short answer:  Anyone and everyone who uses these versions of OpenSSL.

And that’s a LOT of companies and a LOT of people.

Before we get into our Heartbleed tutorial, here’s just a brief sampling of major companies and websites that are known to have been affected and that needed to patch their sites:  Gmail, Yahoo Mail, Intuit TurboTax, USAA, Dropbox, Flickr, Instagram, Pinterest, SoundCloud, Tumblr, GitHub, GoDaddy, Boingo Wireless, and many more.

Many, many corporate websites, of companies of all sizes, have been (or still need to be!) patched to fix the Heartbleed vulnerability.

The vulnerability has existed since December 31, 2011, with OpenSSL being used by about 66% of Internet hosts.

As a user, chances are that sites you frequent regularly are affected and that your data may have been compromised. As a developer or sys admin, sites or servers you’re responsible for are likely to have been affected as well.

So what do I need to do to protect myself if I use any of the affected sites?

The main thing you should do immediately is to change your passwords for any of the affected sites for which you have a login account.

And what do I need to do to fix and protect against Heartbleed if I’m the sys admin for a site that uses OpenSSL?

If you’re using OpenSSL 1.0.1, do one of the following immediately:

• Upgrade to OpenSSL 1.0.1g, or
• Recompile OpenSSL with -DOPENSSL_NO_HEARTBEATS.

If you’re using OpenSSL 1.0.2, the vulnerability will be fixed in 1.0.2-beta2 but you can’t wait for that.  In the interim, do one of the following immediately:

• Revert to OpenSSL 1.0.1g, or
• Recompile OpenSSL with -DOPENSSL_NO_HEARTBEATS.

Most distributions (e.g., Ubuntu, Fedora, Debian, Arch Linux) have upgraded their packages already.  In cases like Gentoo, you can upgrade to a patched ebuild.

Once you’ve upgraded (or recompiled) and have established a secure version on your server:

• Be sure to restart all potentially affected processes.  Major daemons affected by the bug include Apache, Nginx, OpenVPN, and sshd; basically anything and everything linked against libssl. (Note that a restart of these daemons should be sufficient.  There should be no need to rebuild these binaries since they are dynamically linked with the openssl libraries.)
• Verify that you are no longer vulnerable using tools like this online test or this tool on GitHub or this tool on Pastebin.

If your infrastructure was vulnerable, there are Heartbleed tutorial steps that you can and should take.  A useful list of such mitigations is available here.

More gory Heartbleed details, for those who are interested…

As explained in the GitHub commit for the fix, a missing bounds check in the handling of the TLS heartbeat extension could be exploited to reveal up to 64k of memory to a connected client or server.

While the exposed memory could potentially just be garbage, it could just as easily turn out to be extremely valuable to a malicious attacker.

Here’s how the Heartbleed vulnerability works:  An attacker provides the payload as well as the payload length.  However, no validation is done to confirm that the payload length was actually provided by the attacker.  If the payload length was not provided, an out-of-bounds read occurs, which in turn leaks process memory from the heap.

Leaking previous request headers can be a very serious security problem. Specifically, a prior user’s login post data might still be available with their username, password, and cookies, all of which can then be exposed and exploited. Moreover, although private key leakage through Heartbleed was initially deemed to be unlikely, it has been verified that private SSL keys can be stolen by exploiting this vulnerability.

The vulnerability is also made possible due to OpenSSL’s silly use of a malloc() cache.  By wrapping away libc functions and not actually freeing memory, the exploitation countermeasures in libc are never given the chance to kick in and render the bug useless.

Additional details on these ways to fix Heartbleed are available here and here.

And, for what it’s worth, here’s a more amusing perspective.

Kudos to the discoverer, Neel Mehta of Google Security, as well as Adam Langley and Bodo Moeller who promptly provided the patch and helped sys admins determine how to fix Heartbleed. I also encourage you to educate yourself on some of the other common web security vulnerabilities to avoid issues in the future.