The majority of digital tools that you use at some point are built on a foundation of open source projects. If you use an iPhone, it most likely stores data in a SQLite database, an open source database that was designed to be easily embedded in devices. The websites you browse are often made with React.js, Vue.js, or Angular, open source JavaScript frameworks that web developers use to build websites with a lot of interactivity.
Open source and digital infrastructure are not quite the same thing, but they do go hand in hand. While AWS is certainly not open source but is digital infrastructure, it does make use of open source tools like various databases, Linux, and countless other tools that engineers have developed in their spare time. And while Facebook and Google’s various websites are not open source, they were built on the open source JavaScript frameworks listed earlier.
One way to visualize this system is it is like an iceberg. On top are various applications that users interact with such as Chrome, Gmail, Amazon, or if more technical, AWS or Google Cloud for example. However, just below the surface are a wide variety of packages usually produced for free that hundreds of millions of people depend on either for the public facing applications they use, or that developers use to build the services users depend on. It is impressive how developers can depend on these countless packages to help build the products that users see. Instead of having to code everything yourself, one can download bits of code that other people developed to do the hard work for you. For example, instead of implementing data manipulation code yourself in Python, you can download the Pandas package so you don’t have to reimplement the basics. However, the average user doesn’t see that the products they use are made up of combined bits of code created by other people.
While it’s admirable that people would do this work for free, it does lead to quite a few issues.
The Trouble with the Iceberg Model
Society depends on programmers who develop and maintain projects that allow the consumer internet to run properly. But how these major open source projects function is often far more informal than one might think.
Many projects are maintained only by a single person who not only codes all the features for the project, but also manages all the bugs and reports that users send in. While projects that grow in size may get additional maintainers, this can lead to conflict in what parts of a project should be prioritized. Oftentimes feature development is prioritized more than quashing potential security issues. Understandably this happens given that open source maintainers don’t have a security background and code audits by 3rd parties can be pricey.
To make matters worse, most open source projects, even those that are key for various parts of the internet to work, are maintained for free! While some projects eventually get funded, many times this is through private donations facilitated by a platform such as Patreon. For example, Vue.js, a major JavaScript library, is funded by the Patreon of Evan You and for many years Evan could only support the project part time because he wasn’t provided enough money to work full time on Vue.js, and as such needed a traditional job to live.
Consequences
The trouble of the lack of support and funding rears its head when a major disaster occurs and turns out a major project was only maintained by a small group of people who are suddenly overwhelmed.
Several years ago, the Heartbleed bug crippled the cryptographic tool OpenSSL which secured many Windows products across the world along with hundreds of thousands of websites. However, before this bug happened, the OpenSSL project, critical to securing the internet, typically secured $2000 dollars a year according to the person managing an increase in donations, and was only maintained by a single person full-time. In other words, a major global cybersecurity vulnerability was the result of overwhelmed open source maintainers not being able to address all possible issues
One might think that this is an anomaly. But no, the state of OpenSSL prior to the increase in donations spurred by Heartbleed is what a normal, even critical open source project looks like when it comes to general and monetary support.
Another example where the informal nature of open source work crippled large amounts of digital infrastructure was the left-pad incident. Left-pad was a simple library that padded out the left side of strings with zeros or spaces. While the library eventually was reuploaded to NPM, while the package was unpublished, projects such as Node and Babel, essential for frontend development, failed in development and deployment.
The founder problem also causes critical security issues. The library event-stream was maintained by Dominc Tarr, but he soon got overwhelmed, and passed on the development to a different maintainer. This new maintainer added a bit of malicious code that stole money from a type of Bitcoin wallet to siphon the funds to a server in Kuala Lumpur. While the backdoor was removed, this highlights the major issue of managing critical upstream projects with overwhelmed maintainers
What Can be Done
In both the private and public sector, work has been done to support key projects. GitHub recently rolled out a sponsorship program to help support key open source projects that allow our digital infrastructure to keep running. Projects such as Curl or Homebrew are now easily supported through this program.
The Open Technology Fund, a branch of the U.S. Agency for Global Media, also supports a variety of key projects. The OTF for example, improved PyPi’s security by helping introduce two-factor login authentication to make it harder for bad actors to install malware into major Python libraries. The OTF also funds Briar, a open source encrypted messaging system, which has recently seen use in countries that turn off their internet such as Iran or Belarus.
Private organizations are increasingly taking responsibility for critical upstream works. Consider the example of the heartbleed bug. The Linux Foundation along with some partners jumped into action founding the Core Infrastructure Initiative. While the initiative itself has not actually implemented any changes, it has greatly improved the visibility of critical upstream projects through the organization’s steering committee comprised of security experts. A recent evolution in the open source community highlighting security is the Open Source Security Foundation. This is another example of multiple critical projects not getting enough attention. However, this time the community recognized the problem before they turned into a public relations disaster. The failing efforts have been rolled into a new organization with a proper funding model.
Increasing the amount of funding projects get in general would alleviate multiple problems, from projects receiving a pittance compared to how critical they are, to allowing maintainers to potentially have more than one person maintain a critical project.
As policymakers consider ways to address cybersecurity vulnerabilities, including within private and public sector information systems, recognizing how the software development process involves open source tools is necessary to consider potential solutions or policy changes.
