Debian's OpenSSL maintainer (Kurt Roeckx) should be changed

Given enough eyeballs, all bugs are shallow.

Eric Steven Raymond: The Cathedral and the Bazaar

Warning: this article is deliberatly fingerpointing. On a [1,10] flame scale this is 567.89. I give the readers a lot of links at the end of the page, so they can read. I hoped that people will have a sense of humor...

The issue

All OpenSSL randomness from 2006-09-17 to 2008-05-13 was just some hash of the current process ID in Debian, Ubuntu and other distros based on one of these.

In practice, this means that all your keys (RSA, DSA, whatever) generated on these systems in this timeslice is compromised. And all of your DSA keys generated wherever and whenever is compromised if you did a single digital signature or authentication with it from a bad system. See Wikipedia DSA article for the description of DSA: if the k is known, then x (the secret key) can be calculated from s.

Why Kurt did this, is he evil?

No, I think he is not. He just doesn't have the qualifications to create, fix or patch crypto code, espically not in a so widely used project as OpenSSL. He doesn't understand C, in-out parameters, preprocessing, ifdefs and memory management issues.

Probably he would never have been allowed to contribute anything to OpenSSL. He was able to do it just because he is a Debian maintainer, so he can "fix" things directly in Debian. This is why this is a Debian only issue, he just uploaded his change to Debian and forgot to try to push it upstream. He tried once, but his change weren't committed to upstream and he never wondered why.

Bad people will tell you, that Kurt simply works for the NSA, this is not true, they don't employ silly ones.

Comic about a fix random number (4)

But what he tried to fix in OpenSSL, it was buggy?

No, it was not. Kurt doesn't know anything about memory management issues, so he thought that he found a bug in OpenSSL, but he didn't. He was hit by an uninitialized memory issue, pointed out by valgrind, a very handy tool for programmers btw. For example:

#include <stdio.h>
/tmp/a% cat valgrind-test.c
int main() {
	char buf[1];
	printf("%d\n", buf[0]);
	return 1;
}
/tmp/a% gcc -Wall -g -o valgrind-test valgrind-test.c
/tmp/a% ./valgrind-test
-73
/tmp/a% valgrind ./valgrind-test
...
==11175== Use of uninitialised value of size 4
==11175==    at 0x4080C0B: (within /lib/libc-2.7.so)
==11175==    by 0x40827D0: vfprintf (in /lib/libc-2.7.so)
==11175==    by 0x408A7E2: printf (in /lib/libc-2.7.so)
==11175==    by 0x804839B: main (valgrind-test.c:6)
...
/tmp/a%

As you can see, this program finds some garbage in the memory (-73) and prints it out, valgrind points out this bug, so we can fix it. OpenSSL does the same, when you ask some random from it with the function ssleay_rand_bytes(unsigned char *buf, int num) (used as RAND_bytes from applications):

... (do some precomputations, like hashing current process ID)
#ifndef PURIFY
#if 0 /* Don't add uninitialised data. */ (BASTARDIZATION BY Kurt)
                MD_Update(&m,buf,j); /* purify complains */
#endif (BASTARDIZATION BY Kurt)
#endif
... (create random from a PRNG and put it to buf, return)

OpenSSL does this to gather some extra entropy from the uninitialized state of the memory, if you don't like this, you can call RAND_bytes with an initilized buf, or you can compile OpenSSL with -DPURIFY=1.

Purify is similar to valgrind, just older. As you can see, Kurt never realised that he can simply build OpenSSL with "purify support" by defining the PURIFY constant, he inserted an extra #if 0 without any useful comments, like "Debian change by xy". But just let's call this step fine, nothing wrong with this, double #if is quite silly, yeah, but the method is effective, he could be sure that this code won't be compiled into the binary.

But he also changed ssleay_rand_add(const void *buf, int num, double add), usually used through RAND_add or RAND_seed, where buf is an input buffer to seed PRNG with:

... (some precomp)
/*
 * Don't add uninitialised data.
        MD_Update(&m,buf,j);
 */
... (some postcomp)

He commented out the only important line of a one hundred line long function, the one which adds the entropy to the PRNG or as experts call this, "seed the PRNG" (note the name of the functions I just describe). Again, the comment is meaningless, a more relevant comment would be "Don't add any data to the random pool, Debian change for easily invertible keys".

Note again, that in this case buf is clearly an input parameter, not an out as in the previous case. The api documentation (man 3ssl RAND_seed) clearly states this:

       RAND_add() mixes the num bytes at buf into the PRNG state. Thus, if the
       data at buf are unpredictable to an adversary, this increases the
       uncertainty about the state and makes the PRNG output less predictable.
       Suitable input comes from user interaction (random key presses, mouse
       movements) and certain hardware events. The entropy argument is (the
       lower bound of) an estimate of how much randomness is contained in buf,
       measured in bytes. Details about sources of randomness and how to
       estimate their entropy can be found in the literature, e.g. RFC 1750.
  
RAND_add() may be called with sensitive data such as user entered passwords. The seed values cannot be recovered from the PRNG output.
Comic about other distros' sec holes

Nobody noticed this?

I don't know, but no one announced it for two years, when Luciano Bello found it finally.

The changelogs created by Kurt wasn't very helpful anyway:

openssl (0.9.8b-1) unstable; urgency=low
...
  * Don't add uninitialised data to the random number generator.  This stop
    valgrind from giving error messages in unrelated code.
    (Closes: #363516)
...
 -- Kurt Roeckx <kurt@roeckx.be>  Thu,  4 May 2006 20:40:03 +0200

openssl (0.9.8c-1) unstable; urgency=low ... * Move the modified rand/md_rand.c file to the right place, really fixing #363516. ... -- Kurt Roeckx <kurt@roeckx.be> Sun, 17 Sep 2006 14:47:59 +0000
openssl (0.9.8g-9) unstable; urgency=high ... [ Kurt Roeckx ] * ssleay_rand_add() really needs to call MD_Update() for buf. -- Kurt Roeckx <kurt@roeckx.be> Wed, 07 May 2008 20:32:12 +0200

As the changelog shows, he tried to punch the hole on 4 May 2006, but he failed, he moved the modified md_rand.c to the wrong place, but he "fixed" his mistake in september.

He doesn't use any tool like dpatch, so it is quite hard to quickly look up what he exactly changes when he writes a cryptic changelog line. Debian changes more than 100 lines in the OpenSSL source, but don't separate this changes from the Debian packaging details in their .diff.gz's.

Correct and open disclosure by Kurt and the Debian project as a whole

If you say that anyone can make a mistake, then I will give you the final shock now. When he found it out, he silently updated the package on 2008-05-07 in unstable. All uploads to unstable with changelogs and source level changes are public for a wide audience.

All the machines were in direct danger from that time to the release of the Debian Security Advisory 1751 on 2008-05-13. Some crackers used this time to start attacks.

Debian gave (after some painfully long hours) some tools and advices for replacing bad keys. For example they have created a search tool for bad keys. But they have never released the info about the methods used during the creation of the tool. The tool has a large (~5Mb) sequence of binary hash values, but nobody knows how it was generated. They are the same guys, who cries out loud if a vendor create a driver with a 1kbyte binary-only firmware file in it.

The Debian Project will learn from this...

I hardly believe. They already did something very similar exactly seven years ago with cron. That time though they tried to fix a security hole in-house, changed the code and forgot to drop privileges at a very important place.

The security issue they tried to fix was quite nasty, but far wasn't as serious as the introduced issue. All you had to do that time was to edit your crontab with crontab -e, make some mistake while editing, save-and-quit (:wq), acknowledge that you made an error and ask for re-editing, then in the new editor ask for a shell (:!/bin/sh). Voila, there was the root shell!

Comic about a fix random number (4)

Please do not solve problems, without consulting upstream!

What can I do to secure my machines

Assume that they are compromised. Reinstall with fixed OpenSSL and generate new keys. After the reinstall, do not reuse any password, used previously! Yes, this is a lot of work, PITA! Thanks Kurt! Do not allow your users to use bad keys for authentication. Never again use a package maintained by Kurt Roeckx! Compile from source instead, vendors like this shouldn't be trusted!

And search some distro, which works together closely with upstream, has clever maintainers and doesn't fix non-problems in a problematic way. If you find one, contact me!

Links, further reading

SimpsonsDebian already did it
Bug report about the issue Kurt tried to fix
Kurt's "fix" for the valgrind "problem"
Kurt's fix for the security problem
Other packages maintained by Kurt
DSA-1571
Ubuntu's way more correct Security Advisory on the issue
USN's for related packages too
Ben Laurie's absolutely correct reaction I do sympathize with
Ben Laurie's answers to Debian loving morons
Exploit 1
Exploit 2
One point on the side of Kurt: OpenSSL Team's communication skills are bad, to say at least
Hungarian translation of this article
Financial Cryptography on the issue
LWN on the code failure
LWN on the communication failure
SSH attacks started to rise on May 10,
two days before the disclosure

Big hooray for superb, really clever and skilled Debian maintainers!