Near-miss Linux cyberattack puts US officials, tech industry on edge

German software developer Andres Freund was running detailed performance tests last month when he noticed odd behaviour in a little-known program. What he found has sent shudders across the software world and drawn attention from tech executives and government officials.

Freund, who works for Microsoft in San Francisco, discovered that the latest version of the open-source software program XZ Utils had been sabotaged, a move that could have carved out a secret door to millions of Linux-based servers across the internet.

Security experts say it’s only because Freund spotted the change before the latest version of XZ had been widely deployed that the world was spared a digital security crisis.

“We really dodged a bullet,” said Satnam Narang, a security researcher with Tenable, a cybersecurity company based in the US, who has been tracking the fallout from the find. “It is one of those moments where we have to wipe our brow and say, ‘We were really lucky with this one.’”

The near-miss has refocused attention on the safety of open-source software, which is free and serves as the foundation for the internet due to its transparency and flexibility. Many such projects depend on a tiny circle of unpaid volunteers who battle to cope with demands from a global community of users for fixes and upgrades.

XZ, a suite of file compression tools packaged into distributions of the Linux operating system, was long maintained by a single author, Lasse Collin. In recent years, he appeared to be under strain.

In a message posted to a public mailing list in June 2022, Collin said he was dealing with “long-term mental health issues”, and hinted that he is working with a new developer named Jia Tan and that “perhaps he will have a bigger role in the future”.

Update logs available through the open-source software site Github show that Tan’s role quickly expanded. By 2023, Tan was merging his code into XZ, a sign that he had won a trusted role in the project.

But cybersecurity experts who’ve scoured the logs say that Tan was masquerading as a helpful volunteer and that he introduced a nearly invisible backdoor into XZ over the next few months.

Collin didn’t return messages seeking comment and said on his website that he would not respond to reporters until he understood the situation well enough, while Tan did not return messages sent to his Gmail account.

Reuters has been unable to ascertain who Tan is, where he is or who he was working for but many of those who've examined his updates believe Tan is a pseudonym for an expert hacker or group of hackers — likely one working on behalf of a powerful intelligence service.

“This is not kindergarten stuff,” said Omkhar Arasaratnam, the general manager of the Open Source Security Foundation (OSSF), a cross-industry forum for collaborative improvement of open-source software which defends projects like XZ. “This is incredibly sophisticated.”

‘WE LUCKED OUT’

Tan could easily have got away with it had it not been for the Microsoft developer whose curiosity was piqued when he noticed the latest version of XZ intermittently using an unexpected amount of processing power on the system he was testing.

Microsoft declined to make Freund available for an interview, but in publicly-available emails and posts to social media, Freund said a series of easy-to-miss clues prompted him to discover the backdoor.

The find “really required a lot of coincidences,” Freund said on the social network Mastodon.

Microsoft CEO Satya Nadella congratulated Freund over the weekend, saying in a post to the social network X that he loved seeing how the developer “with his curiosity and craftsmanship, was able to help us all.”

In the open-source community, the discovery has been sobering. The volunteers who maintain the software aren't strangers to little pay or recognition, but the realisation that they were now being hunted by well-resourced spies pretending to be Good Samaritans was “incredibly intimidating,” said Arasaratnam of the OSSF.

Government officials are also weighing the implications of the near-miss, which has underlined concerns about how to protect open-source software. Assistant National Cyber Director Anajana Rajan told Politico that “there’s a lot of conversations that we need to have about what we do next” to protect open-source code.

The Cybersecurity and Infrastructure Security Agency (CISA) says it has been leaning on US companies that use open-source software to plough resources back into the communities that build and maintain it. CISA adviser Jack Cable said that the burden was on tech companies not just to vet open software but to “contribute back and help build the sustainable open-source ecosystem that we get so much value from”.

It’s not clear that software companies are properly incentivised to do so. Online forums are teeming with complaints about tech giants demanding that volunteers troubleshoot issues with open-source software from those companies who use it to earn billions of dollars.

Whatever the solution, almost everyone agrees the XZ episode shows something has to change. “We got unreasonably lucky here,” said Freund in another Mastodon post. “We can't just bank on that going forward.”

Reuters