Just before midnight on New Year’s Eve 2011, one of four software developers at OpenSSL Software Foundation (OSF) instructed his computer to accept a suggested new feature which had been voluntarily submitted by a PhD student, and so to merge it into the core code base of his company’s software. Three months later, on March 14th 2012, a new version of the code base, including the PhD student’s new feature, was made freely available to the global software community.
The consequence subsequently turned out to be arguably the most disastrous mistake ever made in the history of the software industry.
“OpenSSL” is a moniker for “open secure sockets layer”. OpenSSL implements basic cryptographic functions for use by the software communication protocols used on the internet. It is “open” in the sense that the software source code is published freely for any software developer to peruse and to verify its professionalism, as well as being free to use (no royalty or licensing fees) for anyone who wishes to do so. There are several alternatives to OpenSSL which perform the same task, some of which are similarly “open” and others which are commercially licensed. But OpenSSL is by a very long way the most popular implementation.
OpenSSL is widely used in Cisco and Juniper networking products used to build much of the internet; in operating systems such as those from HP, Microsoft, VMWare and Google's Android for smart devices; and in very many (maybe as many as 70 per cent) of the world's web services including Amazon, Facebook, Wikipedia and Yahoo. A dangerous fault in OpenSSL would threaten pretty much the entire internet.
On Monday April 7th last, OSF notified the world of a catastrophic flaw in OpenSSL, latent since March 2012. The flaw, “Heartbleed”, can be anonymously exploited by any adversary to remotely examine the memory contents of any machine using OpenSSL anywhere on the internet. A series of such undetected, but fraudulent, requests over a period of just a few seconds, can result in the entire contents of the memory being downloaded. In turn, it is highly likely that these contents will include copies of user passwords, secret encryption keys and the like. As a consequence, the machine, and both its users and administrators, become compromised.
In fact, the flaw had been discovered within Google on or about March 21st last. Over the subsequent fortnight, it now appears that word of the problem was spread quietly to a select few. On April 2nd, Finnish security company Codenomicon independently identified the fault. There was a delicate balance of developing a repair for the flaw, concurrent with telling various people in a small number of companies, without at the same time leaking news of the flaw to the attention of any malicious hackers. Finally, on April 7th, both Codenomicon and OpenSSL went fully public on the disaster.
Every single computer which has the Heartbleed fault must be specifically updated to repair the security breach. Until this is done, each such computer is now particularly vulnerable since the fault has now been widely advertised to all, including potential attackers.
The incident has been a strident alarm for the global software industry. How could almost the entire internet be so vulnerable, and have remained so for a full two years before discovery?
One concern is that some may have discovered the issue, and quietly exploited it without telling anyone else. The US National Security Agency has denied such an allegation, but perhaps other intelligence agencies worldwide, as well as some criminals, have had a free rein since the flaw was first published in 2012.
Blatantly obvious
The shock to the industry has been even more severe because the software source code for OpenSSL is freely published for anyone to read, albeit in the now slightly archaic "C" programming language. A basic tenet of open source software is that since it is openly available, any blemishes and issues are therefore quickly spotted by vigilant developers. Thus, open source software should be of higher quality and "safer" than proprietary software, which is examined only by a relatively small number of software engineers.
Heartbleed has totally destroyed that myth. Despite OpenSSL being openly published, despite potentially hundreds of thousands of developers who could have read the code and identified a very major problem, and despite the fault being very blatantly obvious in the OpenSSL source code, nobody – at least, no “good guys” – spotted the problem for 25 months. It would appear that everyone thought somebody else had checked the source code. Since almost everyone else was using it, everyone assumed it thus must be safe.
Until Heartbleed, OSF had just one full-time developer, alongside three part-time, and received only about $2,000 in donations per year. Severely chastened by Heartbleed, a number of major IT companies recently each pledged $100,000 a year for at least three years, to critical open source internet infrastructure initiatives such as OpenSSL.
It is very challenging to engineer correct and safe software. Open source is clearly no guarantee of thoroughness. Software tools greatly assist the automatic detection of common faults, and testing verifies that a software system does what it is supposed to do, within the time and resource constraints available. However, it is, in general, difficult to build automated tools to uncover all vulnerabilities liable to malicious attack. There is still plenty of scope for sorely-needed innovation in the professional engineering of software.
