Why it matters: By happenstance Microsoft researcher Andres Freund found malicious code that could break sshd authentication. If it hadn’t been discovered it could have posed a grave threat to Linux. The open source community has reacted to the incident, acknowledging the fortuitous nature of the discovery and how it was fortunately caught early before it could pose a significant risk to the broader Linux community.

Andres Freund, a PostgreSQL developer at Microsoft, was doing some routine micro-benchmarking when he noticed a small 600ms delay with ssh processes, noticing that these were using a surprising amount of CPU even though they should be failing immediately, according to his post on Mastodon.

One thing led to another and Freund eventually stumbled upon a supply-chain attack involving obfuscated malicious code in the XZ package. He posted his discovery on the Open Source Security Mailing List and the open source community took it from there.

i tried explaining my nontech friends today that an engineer debugging a 500ms delay has saved the entire web, potentially the entire civilisation

– Peer Richelsen – oss/acc (@peer_rich) March 30, 2024

The dev community has swiftly been uncovering how this attack was craftily injected into XZ utils, a small open-source project maintained by a single unpaid developer since at least 2009. The account associated with the offending commits seemingly played the long game, slowly gaining the trust of XZ’s developer, which has led to speculation that the author of the malicious code is a sophisticated attacker, possibly affiliated with a nation-state agency.

Officially called CVE-2024-3094, it has the highest possible CVSS score of 10. Red Hat reports that the malicious code modifies functions within liblzma, which is a data compression library that is part of the XZ utils package and is a foundational part of several major Linux distributions.

Open source maintainer burnout is a clear and present security danger. What are we doing about that? https://t.co/GZETWimy5i

– Ian Coldwater �”��’� (@IanColdwater) March 29, 2024

This modified code can then be used by any software linked to the XZ library and allow for the interception and modification of data used with the library. Under certain conditions, according to Freund, this backdoor could allow a malicious actor to break sshd authentication, allowing the attacker to gain access to an affected system. Freund also reported that XZ utils versions 5.6.0 and 5.6.1 are impacted.

The xz backdoor is, well, setting a fire under the entire Linux ecosystem… but I’m also so impressed with how it was set up: 2-yr maintainership, oss-fuzz, etc.

…and who knows how long it would’ve stayed undetected if the injected sshd code ran faster (<600ms)

Highlights:

– Danny Lin (@kdrag0n) March 30, 2024

Red Hat has identified vulnerable packages in Fedora 41 and Fedora Rawhide, advising users to cease usage until an update is available, though Red Hat Enterprise Linux (RHEL) remains unaffected. SUSE has released updates for openSUSE (Tumbleweed or MicroOS). Debian Linux stable versions are safe, but testing, unstable, and experimental versions require xz-utils updates due to compromised packages. Kali Linux users who updated between March 26 and March 29 need to update again for a fix, while those who updated before March 26 are not impacted by this vulnerability.

However, as many security researchers have noted, the situation is still developing and more vulnerabilities could be discovered. It is also unclear what the payload was going to be. The US Cybersecurity and Infrastructure Security Agency has advised people to downgrade to an uncompromised XZ utils version, which would be earlier than 5.6.0. Security firms are also advising developers and users to conduct incident response tests to see if they’ve been impacted and if they have, to report it to CISA.

This is explains how the xz backdoor was found pic.twitter.com/n9rNjvawHU

– myq (@mippl3) March 30, 2024

Fortunately it doesn’t appear as if those affected versions were incorporated into any production releases for major Linux distributions, but Will Dormann, a senior vulnerability analyst at security firm Analygence, told Ars Technica that this discovery was a close call. “Had it not been discovered, it would have been catastrophic to the world,” he said.