When the Last Line of Defense is the Weakest

Around 2016, Armijn Hemel took apart a KVM-over-IP switch. The box a sysadmin reaches for when a server stops answering, it opens a path into the machine over the network. This one ran a version of Linux that, by Hemel’s estimate, was about 16 years old. The equipment designed to be a data center’s last line of defense was the least defended thing in the rack.

Hemel dismantles firmware for a living, so this pattern is familiar. The devices most people never think about (the home router, the cheap camera) tend to ship with already-old software that then runs for years. The owner rarely notices when one gets exploited, because the camera still films and the router still routes. “Would you be okay if a team of burglars set up shop in your home?” he asks. That’s roughly what happened with Mirai: malware that, without obvious disruption, used hundreds of thousands of Internet protocol (IP) cameras and similar devices to launch attacks.

Old flaws linger for the same reason: No one is forced to fix them. Philippe Ombredanne, who leads the open source project ScanCode, points to log4j to show “the inertia is huge.” A small, ubiquitous piece of Java software, log4j was found in December 2021 to be dangerously vulnerable — and present in a vast number of Internet-facing systems, including banks. More than four years later, a contact at one of the largest cloud providers told Ombredanne that more than 20 percent of its Java customers were still running a flaw that was already years old at discovery.

The scale of this gap is almost invisible to the people relying on the software. Ombredanne likes a thought experiment: Subtract all the open source software from the world and see what stops. The phones, the computer, the networks that connect them. Most cars built in the last few decades and the pumps that would refuel them. Cash registers and the banks behind them. “It would be total chaos,” he says.

For years, a small number of people have been building the defenses for this hidden layer of dependency. They did it quietly, on public money, before any law required it. Now a law has arrived.

A Supply Tree, Not a Supply Chain

The company listed on a typical consumer device box is often little more than a sales office: a support engineer, someone to answer the phone. A step back, the original design manufacturer, or ODM, builds the anonymous white-box hardware that later gets a brand attached; ODMs frequently pick a design from a chipset catalog, Hemel says, the way you might order “that one, in blue.” The chipset maker hands the ODM a software development kit (SDK) to drive its silicon, and that SDK is often already at the end of its supported life by the time the finished device reaches a shelf. Hemel once visited a chipset maker at an ODM’s request and learned that the chip going into a ready-to-launch product had been discontinued two years earlier.

The Common Vulnerabilities and Exposures (CVE) system was designed decades ago to pin a flaw to a vendor and a product: Microsoft, Adobe, Windows, Acrobat. It was never built for a world where tens of millions of devices share one WiFi chipset and the same aging code. A vulnerability, Ombredanne argues, should be treated as a software attribute and tracked across every device that carries it, rather than as a problem belonging to one vendor. “That's how it should be,” he says. “It's not how it is.”

This is why Hemel talks about a supply tree rather than a supply chain. One root feeds many near-identical devices wearing different logos. Trace enough Blu-ray recorders back, and they converge on the chipset maker MediaTek. The root of the tree — the chipset and the ODM — is also the root of the problem; the branding companies sit out at the branches, closest to the customer and furthest from the code.

Hemel points to a book, The Box by Marc Levinson, about how the standardized shipping container collapsed the cost of moving goods and reshaped global trade. Linux was the embedded software’s version of the container, a standard base that let manufacturers stop paying for proprietary real-time operating systems and drove the cost of a capable device toward the floor. Security was the thing that gave way to market pressure: “Fast, cheap, secure,” Hemel says. “Pick two.”

The result is a strange allocation of risk. The company whose sticker is on the box carries the product’s legal liability. But in nearly every case, that company does not know what software is inside — and cannot fix it.

A European law is about to make device safety someone’s legal duty. The Cyber Resilience Act, or CRA, requires anything sold with software in it to be built securely and kept patched for years. Ombredanne and Hemel would sharpen that summary: The law rolls out by sector and will prioritize genuine risks. Its first hard deadline, mandatory reporting of exploited vulnerabilities, lands on September 11, 2026; the full set of obligations follows in December 2027. The law reaches toward the root of the tree, even though the liability starts out at the branches.

The Toolkit’s Ready, the Data’s in Poor Shape

The CRA will require an inventory of every component in a product, with its licenses and dependencies: a software bill of materials, or SBOM. The tools to produce that list already exist. Reconstructing that list after the fact, when nobody kept the label, is the discipline of software composition analysis — and it requires reliable data.

Ask Ombredanne what his tools do, without a demo, and he reaches for LEGO. ScanCode identifies the origin and license of third-party components inside a piece of software. “You built a LEGO car,” he says. “It tells you which bricks you used and what color they are.” Its companion, VulnerableCode, adds the security layer. “It tells you this brick here has a crack. Don’t use it.” Between them, they answer the two questions any company facing the CRA must know: What is in my product, and which parts of it are known to be dangerous?

Tools are the easy half. The hard half is the data they run on. Ombredanne’s team keeps finding vulnerability reports for packages that do not exist: a flaw filed against an unreleased version or a package name that appears nowhere anyone actually publishes software. He attributes the flood to AI systems generating plausible-looking reports. Nobody can use software that isn’t real, but the fake report lends the fake name credibility; an attacker can then register that name, point to the vulnerability database as proof it exists, and wait for someone to pull it in. Even the real reports arrive faster than anyone can sort them. Ombredanne compares it to an emergency room where a thousand people with paper cuts and one person about to die all sit in the same waiting room, looking alike. And the incentives make it worse. Fame and money in security flow to discovering vulnerabilities, not to fixing them; the Linux kernel’s security team holds, by his count, a standing list of more than 500 serious open issues and keeps receiving more reports than fixes.

The work of getting good data is unglamorous and often lonely. To make sense of the security advisories from NGINX, the web server behind a large share of the Internet, Ombredanne’s team spent a month decoding a format whose conventions lived mostly in the authors’ heads. When he asked the Debian project which license covered its security data, so he could legally reuse it, he was told he was the first person ever to ask.

Data also degrades as it travels. Ombredanne traces one Apache log4j advisory through its journey: Correct at the source, it passed to the MITRE CVE project, then to the US National Vulnerability Database where a format conversion damaged the contents of the version list. GitHub drew its feed from there and damaged it a little more, and GitLab and Google drew theirs from GitHub. The upstream project had it right; a US federal agency, a major open source company and two of the largest technology firms all ended up with a version that was wrong. “It’s a game of telephone,” Hemel says, and his real objection is the opacity. When the data is open, you can find where it broke; when it is closed, you cannot.

Ombredanne is blunt about why the mess persists. The commercial database and scanning-tool vendors that are the paid competitors to his own tools have little incentive to clean it up, he argues, because the complexity is what justifies their pricing. He points to a US venture-backed startup selling “CVE-free” container images without contributing its fixes back upstream. To him, that is a kind of enclosure: “You don’t put a tax on oxygen,” so why require paid access to vulnerability data that’s just as vital? In his view, these records are a commons that has to stay open — not merely visible, but licensed for reuse and produced through a process anyone can inspect. That covers three kinds of data: which versions are vulnerable, what license a component carries, and where a component actually came from.

None of this looked like a product anyone would buy. The risk of shipping insecure software was low, and the people who understood the problem were rarely the people holding the budget. Asked whether the market would ever have paid for an open vulnerability database or firmware scanner, Ombredanne does not hesitate. “No,” he says.” “I don't think so.”

Public money provided the answer. Hemel started out of frustration and on his own time. He kept his tools alive with Dutch research and development tax credits and a series of public grants that include support through NGI Zero, the EU-funded programs that fund open source research and development. One product of that work is DeviceCode, an open, searchable catalog assembled by cleaning and unifying enthusiast-sourced facts about real hardware. In a world where vendors and manufacturers keep the contents of our devices obscured, it can list every device shipped with the password “password” and filter the results by ODM and chipset. Though Hemel is the first to admit it is still too raw for a non-technical user, it combines resources with tools like VulnerableCode to track vulnerabilities across clusters of similar devices. Ombredanne’s ScanCode has drawn on NGI Zero, as well, on the German Sovereign Tech Agency, and on grants from companies including Bloomberg, Zeiss, Porsche, Mercedes, and Google. He is now building the AboutCode Foundation, a member-funded nonprofit, to keep the work going.

Both men have landed in the same place: CodeSupply. A project funded by the European Commission, it catalogs the software and firmware inside real devices and records each binary’s origin, license, health, and security as open data. “More than the tools,” Ombredanne says, “the frontier is good data.”