Insight and commentary on the latest news stories

• Thoughts on the XZ Utils Backdoor2024/04/02

  The open-source community narrowly avoided a major security crisis with the discovery of a critical vulnerability (CVE-2024-3094) and backdoor that was discovered in XZ Utils, a data compression library used by many Linux distributions.

  XZ Utils provides a collection of command-line tools and libraries for lossless data compression using the LZMA algorithm. The most prominent program within this suite is xz, a powerful compressor frequently used for tasks like archiving files, creating software packages, and compressing disk images. Due to its efficiency and reliability, XZ Utils is installed on many mainstream Linux distributions.

  The discovery of the backdoor

  Andres Freund, a Microsoft PostgreSQL developer focused on performance, first noticed something strange on his machine when developing new functionality for an upcoming PostgreSQL release. Andres noticed that his CPU would randomly spike and after some intense investigation was able to pinpoint the issue to XZ Utils and upon further review found that malicious code had been introduced. Andres notified the wider community about this backdoor by sending a mail to OpenWall’s security mailing list:

  After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with sshd taking a lot of CPU, valgrind errors) I figured out the answer: The upstream xz repository and the xz tarballs have been backdoored. At first I thought this was a compromise of debian’s package, but it turns out to be upstream.

  This vulnerability was intentionally introduced into XZ as a backdoor to allow a remote attacker access to a targeted system via remote code execution. The malicious code that was introduced could tamper with SSH authentication.

  This backdoor targeted popular Debian and RPM-based distributions, which includes e Ubuntu, Red Hat, Fedora, and CentOS. These distributions rely on package managers that download and install software. If a backdoored version of XZ Utils had made it into these repositories, a significant number of users could have unknowingly installed the vulnerable software.

  The backdoor specifically affected xz versions 5.6.0 and 5.6.1, released in February 2024. Thankfully, due to its late discovery, the backdoored version hadn't achieved widespread deployment.

  How the backdoor was introduced

  The entire internet is underpinned by infrastructure and services that rely on open-source software. Open-source projects thrive on collaboration where code lives on a central platform (Frequently GitHub) and developers from around the world can propose improvements by creating changes (code commits) and then pushing these proposed changes to the open-source software project. These proposals are then reviewed by other developers, often the maintainers of the project, for functionality, performance issues, style, and security. After discussion and additional tweaks, these changes are approved and then are merged into the code branches that eventually will find themselves into a release branch. Over time, these merged changes accumulate, and project maintainers decide when a new release, containing bug fixes, new features, or both, is ready. This updated code is then made available to everyone.

  Open-source projects have a distributed trust model that can be subverted by determined threat actors as we have seen with the introduction of this backdoor. The backdoor is made possible by several Linux components including OpenSSH, systemd, liblzma and xz utils. OpenSSH and systemd do not have any vulnerabilities as this backdoor is made possible specifically from liblzma and the wider xz project.

  The xz project was created and maintained by a developer named Lasse Collin and has been stable since 2010 with a few updates since then. The introduction of this backdoor was the result of a planned scheme.

  In 2021, a GitHub user calling themselves “Jia Tan” (@JiaT75) was created. Tan consistently delivered valuable contributions as a developer, gaining the confidence of the project maintainers. In 2022, Tan submitted a patch for xz along with other commits that were strongly supported by several other users. After analysis it appears that these users who supported the patch were sock puppet accounts and did not exist elsewhere on the internet. This trust allowed Tan to introduce the backdoor code into xz versions 5.6.0 and 5.6.1. Efforts were then made to expedite the adoption of these backdoored versions into Linux distributions, essentially trying to push the compromised project onto millions of machines.

  The backdoor

  The backdoor enables a Remote Code Execution (RCE) exploit and grants attackers unauthorized access to vulnerable systems. To trigger the backdoor, a unique Ed448 private key could bypass the standard SSH authentication protocols. Once authentication was bypassed, attacks would be able to steal data, install malware or even use the affected system to move laterally to other networks or systems.

  The stealthy activation mechanism of this backdoor made it hard to detect, however since the backdoor requires a specific key, it limits its exploitability.

  Developer Sam Jones, has an excellent summary  of the backdoor:

  This backdoor has several components. At a high level:

  • The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
  • There are crafted test files in the tests/ folder within the git repository too. These files are in the following commits:
    • tests/files/bad-3-corrupt_lzma2.xz (cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0, 74b138d2a6529f2c07729d7c77b1725a8e8b16f1)
    • tests/files/good-large_compressed.lzma (cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0, 74b138d2a6529f2c07729d7c77b1725a8e8b16f1)
  • A script called by build-to-host.m4 unpacks this malicious test data and uses it to modify the build process.
  • IFUNC, a mechanism in glibc that allows for indirect function calls, is used to perform runtime hooking/redirection of OpenSSH's authentication routines. IFUNC is a tool that is normally used for legitimate things, but in this case it is exploited for this attack path.
  • Normally, upstream publishes release tarballs that are different than the automatically generated ones in GitHub. In these modified tarballs, a malicious version of build-to-host.m4 is included to execute a script during the build process.

  Akamai has a good explanation of the backdoor and cryptography expert Flippo Valsorda also did an analysis of the backdoor.

  Who was behind this supply chain attack?

  This attack was not the work of a lone person, but rather a targeted operation by a nation state actor. At the time of writing there is no definitive evidence of which nation was behind the attack. We will likely know the answer to this question once further investigations have been completed.

  The intelligence agencies of the Five Eyes (United States, Canada, United Kingdom, Australia, and New Zealand) are unlikely to behind the attacks given the mechanism that was used. The United States can be excluded as they would not get the legal authority to undermine a crypto subsystem.

  Russia and China are the likely culprits behind this attack. Russian actors have already performed supply chain attacks such as the NotPetya Attack in 2017 and the SolarWinds attack in 2020. In the SolarWinds attack, code was added to the build pipeline that could allow unauthorized access to systems. Linguistical analysis of the language used by the person pushing the commits suggest that the person behind it is Russian due to the language mistakes that were made that are commonly made by Russians trying to speak English. Time zone analysis shows that who ever was behind this, was likely working in the Asia Pacific time zone which would include Russia and China.

  China is could also behind the attacks as they have been targeting critical infrastructure around the world in preparation for their invasion of Taiwan. One of these operations was Volt Typhoon, which targeted critical infrastructure. The FBI and intelligence agencies shut down infrastructure associated with Volt Typhoon earlier this year. It is also likely that China hired Russians to carry out this attack.

  The calculated approach employed by Tan highlights the potential dangers of social engineering within open-source projects, especially with regards to advanced supply chain attacks. Working and building trust over years to introduce a backdoor is a significant operation. While we got lucky that a developer spotted this backdoor, what other backdoors could have been introduced without anyone knowing?

• Crypto King Sentenced: FTX scandal leads to accountability2024/03/30

  FTX, once a leading cryptocurrency exchange, crumbled in November 2022 after a massive fraud orchestrated by its founder, Sam Bankman-Fried (SBF). SBF diverted billions of dollars in customer funds for personal use, risky investments, and political donations.

  The house of cards began to topple when reports surfaced about FTX's close ties to Alameda Research, a crypto trading firm also owned by SBF. Concerns mounted over FTX's financial health, leading to a surge of customer withdrawals.  The exchange was unable to meet these demands, exposing a massive shortfall in its reserves.

  The fallout was devastating. Investors and everyday users lost billions as FTX filed for bankruptcy. The scandal sent shockwaves through the already volatile cryptocurrency market, causing prices to plummet.  Trust in the entire crypto industry was shaken, with stricter regulations being called for.

  Prosecutors painted a picture of SBF as a greedy mastermind who built a web of lies to deceive investors and regulators. They presented evidence of hidden financial records, inflated asset valuations, and misappropriated funds. SBF pleaded not guilty, claiming he was unaware of the full extent of the problems at FTX.

  The defense argued that SBF was a young, ambitious entrepreneur who made mistakes but never intended to defraud anyone. They portrayed him as a victim of circumstance, caught off guard by the sudden market downturn.  However, the jury wasn't swayed.  In November 2023, SBF was found guilty on all counts, including wire fraud and conspiracy to commit money laundering.  Yesterday, SBF was sentenced to 25 years in prison.

  The Crypto Industry starts to see accountability

  The world of cryptocurrencies holds potential, but trust and fraud remain major concerns for users. Unlike traditional financial institutions with regulations and consumer protections, crypto services are a relatively new frontier, leaving many vulnerabilities open for exploitation.

  One key issue is the lack of transparency surrounding some crypto exchanges.  Unlike banks, which are required to hold reserves and regularly report their financial health, some crypto exchanges operate with limited oversight. This opacity makes it difficult for users to assess the true value of their holdings and the solvency of the exchange itself.  The FTX scandal, is a stark example of the dangers of such a lack of transparency.

  Another major concern is the prevalence of scams and hacking attacks.  Cryptocurrencies operate on decentralized networks, making them attractive to cybercriminals. Phishing schemes, malware designed to steal private keys, and exit scams – where fake exchanges disappear with user deposits – are just a few of the threats users face.  The anonymity associated with some crypto transactions also makes it difficult to track down and prosecute criminals.

  Building trust in crypto services requires a multi-pronged approach. Stronger regulations that mandate transparency and robust security measures are essential.  This could involve requiring exchanges to hold reserves in traditional assets and to undergo regular audits. Additionally, user education is crucial.  Understanding the inherent risks of cryptocurrencies and how to protect oneself from scams is vital for anyone considering entering this space.

  The FTX scandal sent shockwaves through the cryptocurrency industry, prompting regulators around the world to take a closer look at this once-unfettered market. Calls for stricter regulations are growing louder, with governments aiming to bring more transparency and accountability to crypto exchanges.  Measures being discussed include requiring exchanges to hold reserves in traditional assets, mandating regular audits, and implementing stricter know-your-customer (KYC) procedures to combat money laundering.

  This newfound regulatory focus also means stiffer penalties for fraudsters.  The sentencing of Sam Bankman-Fried, the founder of FTX, to 25 years in prison serves as a stark warning to others considering manipulating the crypto space.  Law enforcement agencies are also increasing their efforts to track down and prosecute criminals involved in crypto scams and hacking attacks. With tighter regulations and a harsher legal environment, the Wild West days of cryptocurrency may be coming to an end.

• UN Investigating 58 crypto heists linked to North Korea’s Cyberattacks2024/03/25

  A United Nations panel is investigating a series of cyberattacks allegedly conducted by North Korea. The report, released in March 2024, details how these attacks, carried out between 2017 and 2023, netted an estimated $3 billion for the regime.

  From the UN report:

  The Panel is investigating 58 suspected cyberattacks by the Democratic People’s Republic of Korea on cryptocurrency-related companies between 2017 and 2023, valued at approximately $3 billion, which reportedly help to fund the country’s development of weapons of mass destruction. The high volume of cyberattacks by hacking groups subordinate to the Reconnaissance General Bureau reportedly continued. Trends include targeting defence companies and supply chains and, increasingly, sharing infrastructure and tools.

  The U.N. report identified several hacking groups linked to North Korea's Reconnaissance General Bureau (RGB), including Kimsuky, Lazarus Group, Andariel, and BlueNoroff. These groups use various methods like spearphishing, exploiting vulnerabilities, social engineering, and "watering hole" attacks to steal valuable data and generate revenue.

  The report highlights North Korea's interest in stealing intellectual property for technological advancements and resale. Cryptocurrency hacks are a growing concern, with the panel investigating 17 incidents in 2023 alone, totaling $750 million in stolen funds.

  The hackers target defense companies, software supply chains, and even nuclear engineers. They've compromised organizations in Spain, Netherlands, Poland, and Russia. Social engineering tactics include fake recruiter profiles on LinkedIn and manipulating job seekers on messaging apps. South Korea is a frequent target, with stolen data ranging from defense information to university research.

  The report exposes supply chain attacks on software companies like JumpCloud, compromised to launch cryptocurrency heists. It confirms collaboration among hacking groups with overlapping membership across agencies within North Korea.

  The U.N. panel documented North Korean involvement in ransomware attacks, with Andariel linked to a $360,000 bitcoin theft and Lazarus Group collaborating with a South Korean company to collect millions in ransom payments. Laundering stolen funds remains a challenge, with the report mentioning Lazarus Group's renewed use of Tornado Cash, a cryptocurrency mixing service.

  The U.N. urges member states to strengthen financial institution cyber defenses, consider sanctions on hacking groups, and disrupt North Korea's money laundering activities. Blockchain security firm Elliptic confirms ongoing large-scale laundering efforts by Lazarus Group through Tornado Cash.

• Tor releases WebTunnel to bypass censorship by mimicking HTTPS traffic2024/03/14

  Roger Dingledine presented a talk at the 37th Chaos Communication Congress about the challenges of web censorship and the censorship attempts that countries including Russia, Iran and Turkenistan have attempted. Tor effectively has an technological arms race with these censors as they race to build new ways of bypassing network censorship in adversarial countries. One of the anti-censorship mechanisms that Tor uses is pluggable transports which disguise internet traffic as something else to evade detection or uses alternative protocols which are problematic to block as they will lead to civil unrest if implemented.

  Tor currently supports the following pluggable transports:

  • obfs4: One of the more popular pluggable transports, it adds an extra layer of encryption that disguises Tor traffic as seemingly random data and is hard to distinguish from other traffic.
  • meek: This transport is designed to blend in with cloud traffic, making it useful in situations where Tor itself might be blocked. However, it can be slower than other options. There are also variations like meek-azure or meek-amazon that leverage specific cloud providers.
  • snowflake: This transport uses a different technique to disguise Tor traffic, making it appear like legitimate web browsing activity.
  • fte: This is a more experimental transport that offers good performance but might not be as widely supported as others.

  The Tor project has recently announced a new pluggable transport called WebTunnel which is designed to circumvent internet censorship by disguising online activity.

  From the Tor project blog:

  "WebTunnel is a censorship-resistant pluggable transport designed to mimic encrypted web traffic (HTTPS) inspired by HTTPT. It works by wrapping the payload connection into a WebSocket-like HTTPS connection, appearing to network observers as an ordinary HTTPS (WebSocket) connection. So, for an onlooker without the knowledge of the hidden path, it just looks like a regular HTTP connection to a webpage server giving the impression that the user is simply browsing the web. 

  In fact, WebTunnel is so similar to ordinary web traffic that it can coexist with a website on the same network endpoint, meaning the same domain, IP address, and port. This coexistence allows a standard traffic reverse proxy to forward both ordinary web traffic and WebTunnel to their respective application servers. As a result, when someone attempts to visit the website at the shared network address, they will simply perceive the content of that website address and won't notice the existence of a secret bridge (WebTunnel)."

• CISA takes down 2 systems after Ivanti compromise2024/03/10

  The Cybersecurity and Infrastructure Security Agency (CISA) confirmed a cyberattack in February that exploited vulnerabilities in Ivanti software used by the agency. While CISA declined to disclose details about the attackers or data accessed, the compromised systems reportedly included critical infrastructure data.

  CISA acknowledged the incident but offered limited information.  They confirmed taking two systems offline and emphasized the importance of incident response plans.  Unnamed sources claim the compromised systems may have housed critical infrastructure data and chemical security plans.

  Jonathan Greig and Suzanne Smalley for The Record:

  A CISA spokesperson confirmed to Recorded Future News that the agency “identified activity indicating the exploitation of vulnerabilities in Ivanti products the agency uses” about a month ago.

  “The impact was limited to two systems, which we immediately took offline. We continue to upgrade and modernize our systems, and there is no operational impact at this time,” the spokesperson said. “This is a reminder that any organization can be affected by a cyber vulnerability and having an incident response plan in place is a necessary component of resilience.”

  CISA previously warned about vulnerabilities in Ivanti products exploited by hackers. These vulnerabilities potentially allowed attackers to gain full access to compromised systems.  A separate advisory noted that an Ivanti tool designed to detect such breaches may be ineffective.

  While the number of affected federal agencies remains unclear, CISA previously stated that around 15 agencies used the vulnerable software. The potential exposure of sensitive infrastructure data raises security concerns, highlighting the need for robust cybersecurity measures and the importance of addressing software vulnerabilities promptly.