The xz-utils backdoor in security advisories by national CSIRTs

Published: 2024-04-01
Last Updated: 2024-04-02 06:24:29 UTC
by Jan Kopriva (Version: 1)
9 comment(s)

Errata: It turns out I have missed two advisories when going through the CSIRT sites the first time - I have amended the text and table to reflect this.

For the last few days, the backdoor in xz-utils[1] has been among the main topics of conversation in the global cyber security community.

While it was discovered before it made its way into most Linux distributions and its real-world impact should therefore be limited, it did present a very real and present danger. It is therefore no surprise that it was quickly covered by most major news sites devoted to information and cyber security[2,3,4,5].

However, since the first information about existence of the backdoor was published on Friday 29th[6], which was a public holiday in many countries around the world, and the same may be said of today, it is conceivable that some impacted organizations and individuals might not have learned about the danger from these news sites, as they might only monitor advisories from specific sources – such as national or governmental CSIRTs – during the holidays.

Fast response from national or governmental CSIRTs, or other, similar organizations, in situations like these can therefore be of paramount importance. Consequently, it occurred to me that the current situation might present a good opportunity for a quick analysis to see how many national or governmental CSIRTs/their host organizations/similar entities (e.g., national coordination centers, multi-national or regional CSIRTs, etc.) publish up-to-date warnings and advisories even during holidays.

I have therefore gone over the FIRST membership list[7], which includes (among many other teams) a large percentage of national and/or governmental CSIRTs from around the globe, and identified 105 teams which have a national or governmental constituency and which might therefore possibly function as an “early warning system” for a specific country, region or nation. I have then gone through the official websites of these teams to see which ones did warn about the xz-utils backdoor and when.

The results were interesting, and – at least to me – somewhat surprising. At the time of writing, only 11 (e.g., approximately 10.5%) of the 105 teams/organizations had published an advisory covering the existence of the backdoor. Four of them did so on March 29th, the same day when the existence of the backdoor was first made public, six of them did so the next day – on Saturday 30th – and one did so three days later, on Monday 1st.

At this point, it should be stressed that not all the identified national or governmental CSIRTs (or other relevant organizations) provide a public “advisory service” to their constituencies, so the numbers mentioned above don’t tell the whole story. Additionally, even for CSIRTs/organizations that do provide such a service, a lack of warning about this specific issue is not necessarily an indication that the service doesn’t function efficiently and effectively – every team has its own standards and processes which one can hardly judge from the outside perspective… In short, this article is not intended as a criticism of any of the CSIRTs which did not publish an advisory corresponding to the aforementioned backdoor.

That said, let’s take a look at the list of CSIRTs that were identified as potentially relevant for our purposes. In the following table, you will find the name of each such team/organization, the country it belongs to and information about any relevant advisory it published.

Country/region Team/organization Advisory published Link Note
Albania AL-CSIRT No - -
Australia AUSCERT No - -
Australia Australian Cyber Security Centre No - -
Austria CERT.at 29.3.2024 link -
Azerbaijan CERT.AZ No - -
Azerbaijan CERT.GOV.AZ No - -
Bahrain CERT BH No - -
Belarus CERT.BY No - -
Belgium CERT.be No - -
Benin bjCSIRT 30.3.2024 link -
Bhutan BtCIRT No - -
Botswana Botswana-CSIRT No - -
Brazil CERT.br No - -
Brazil CTIR Gov-BR No - -
Brunei BruCERT No - -
Canada CanCERT ? ? The official CSIRT site was inaccessible
Canada Canadian Centre for Cyber Security No - -
Catalonia CATALONIA-CERT No - -
Côte d'Ivoire CI-CERT No - -
Croatia CERT ZSIS No - -
Croatia CERT.hr No - -
Cyprus National CSIRT-CY No - -
Czech Republic CSIRT.CZ No - -
Czech Republic GovCERT.CZ No - -
Denmark Centre For Cyber Security No - -
Denmark DKCERT No - -
Dominican Republic CSIRT-RD No - -
Egypt EG-CERT No - -
Estonia CERT-EE No - -
Ethiopia ETHIO-CERT ? ? The official CSIRT site was inaccessible
European Union CERT-EU 30.3.2024 link -
Finland NCSC-FI No - -
France CERT-FR No - -
Georgia CERT.DGA.GOV.GE No - -
Germany CERT-Bund No - -
Ghana CERT-GH No - -
Hong Kong GovCERT.HK No - -
Hungary NCSC Hungary No - -
Chile CSIRT GOB CL No - -
China CNCERT/CC ? ? The official CSIRT site was inaccessible
Iceland CERT-IS No - -
India CERT-In No - -
Indonesia ID-SIRTII/CC No - -
Ireland NCSC Ireland 29.3.2024 link -
Italy CSIRT-IT 30.3.2024 link -
Japan CDI-CIRT No - -
Japan JPCERT/CC No - -
Japan NISC No - -
Jordan JoCERT ? ? The official CSIRT site was inaccessible
Kazakhstan KZ-CERT No - -
Kenya National KE-CIRT/CC No - -
Korea KN-CERT No - -
Latvia CERT.LV No - -
Liechtenstein CSIRT.LI No - -
Lithuania CERT-LT No - -
Luxembourg CIRCL 30.3.2024 link -
Luxembourg GOVCERT.LU No - -
Malawi mwCERT No - -
Malaysia MyCERT No - -
Malta govmtCSIRT No - -
Mauritius CERT-MU No - -
Mexico CERT-MX ? ? List of vulnerability advisories did not load
Moldova CERT-GOV-MD No - -
Monaco CERT-MC No - -
Mongolia MNCERT/CC No - -
Montenegro CIRT.ME ? ? The official CSIRT site was inaccessible
Morocco maCERT No - -
Netherlands NCSC-NL 30.3.2024 link -
New Zealand CERT NZ No - -
Nigeria ngCERT ? ? The official CSIRT site was inaccessible
North Macedonia MKD-CIRT No - -
Norway NCSC-NO 29.3.2024 link -
Oman OCERT No - -
Panama CSIRT Panama No - -
Poland CERT POLSKA No - -
Portugal CERT.PT No - -
Romania Romanian National Cyber Security Directorate No - -
Russia RU-CERT No - -
Rwanda Rw-CSIRT No - -
Saudi Arabia Saudi CERT ? ? The official CSIRT site was inaccessible
Serbia GOVCERT.RS No - -
Serbia SRB-CERT No - -
Singapore SingCERT 1.4.2024 link -
Singapore SG-GITSIR No - -
Slovakia SK-CERT 29.3.2024 link -
Slovakia CSIRT.SK No - -
Slovenia SI-CERT No - -
Spain CCN-CERT No - -
Spain Basque CyberSecurity Agency No - -
Spain INCIBE-CERT No - -
Sri Lanka Sri Lanka CERT/CC No - -
Sudan Sudan-CERT No - -
Sweden CERT-SE 30.3.2024 link -
Switzerland NCSC.ch No - -
Taiwan TWCSIRT ? ? The official CSIRT site was inaccessible
Taiwan TWNCERT ? ? The official CSIRT site was inaccessible
Taiwan TWCERT/CC No - -
Tanzania TZ-CERT No - -
Thailand ThaiCERT No - -
Tunisia tunCERT No - -
Turkey TR-CERT No - -
UAE aeCERT ? ? The official CSIRT site was inaccessible
Uganda UG-CERT No - -
Ukranie CERT-UA No - -
United Kingdom National Cyber Security Centre No - -

As we already mentioned, the fact that some of the teams listed in the table didn’t publish an advisory for the xz-utils backdoor doesn’t necessarily reflect badly on them – many of these teams don’t provide any security advisory services, vulnerability warnings, etc. at all, and still do a good job, while others might have simply decided not to publish an advisory for this specific issue given that it didn’t meet their internal criteria for advisories.

Still, there will certainly also be a number of teams/organizations which didn’t publish the advisory because they don’t have sufficient personnel or processes to do so during public holidays… So, for any organization which wishes to monitor only a limited number of information sources during such times, it might be advisable to chose those sources very carefuly.

[1] https://isc.sans.edu/podcastdetail/8918
[2] https://www.theregister.com/2024/03/29/malicious_backdoor_xz/
[3] https://www.helpnetsecurity.com/2024/03/29/cve-2024-3094-linux-backdoor/
[4] https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
[5] https://securityaffairs.com/161224/malware/backdoor-xz-tools-linux-distros.html
[6] https://www.openwall.com/lists/oss-security/2024/03/29/4
[7] https://www.first.org/members/teams/

-----------
Jan Kopriva
@jk0pr
Nettles Consulting

Keywords:
9 comment(s)

The amazingly scary xz sshd backdoor

Published: 2024-04-01
Last Updated: 2024-04-01 15:25:33 UTC
by Bojan Zdrnja (Version: 1)
1 comment(s)

Unless you took the whole weekend off, you must have seen by now that Andres Freund published an amazing discovery on Friday on the Openwall mailing list (https://www.openwall.com/lists/oss-security/2024/03/29/4).
The whole story around this is both fascinating and scary – and I’m sure will be told around numerous time, so in this diary I will put some technical things about the backdoor that I reversed for quite some time (and I have a feeling I could spend 2 more weeks on this).
There is also a nice gist by smx-smx here that gets updated regularly so keep an eye there as well.
The author(s) of the backdoor went a long way to make the backdoor look as innocent as possible. This is also why all the reversing effort is taking such a long(er) time. Let’s take a look at couple of fascinating things in this backdoor.

String comparison

One of the first things a reverse engineer will do is to search for strings in the code they are looking at. If strings are visible, they can usually tell a lot about the target binary. But if we take a look at the library (and for this diary I am using the one originally sent by Andres) we will see practically no visible strings.
The authors decided to obfuscate all strings – in order to do that, they stored strings as a radix tree (also known as prefix tree or trie, more info at https://en.wikipedia.org/wiki/Radix_tree). This allows them to store all strings as obfuscated, however now one of the challenges they had was to lookup strings – they implemented a function that checks whether a string exists in the radix tree table, and if it does, it returns back the offset:


The image above shows start of the function (originally called Lsimple_coder_update_0) where I also expanded one of the radix tree tables (_Llzip_decode_1). So as input we send a string, and if it exists in the radix tree, we get back its offset. The code needs to do this for every single string lookup, but that should not cause the delay as radix tree lookups are quite fast.

Anti-debugging

Anti-debugging efforts were implemented on several levels. The simplest one includes code where the backdoor is checking if a function starts with the ‘endbr64’ instruction. Let’s check how this works – first the snippet below shows how the .Llzma_block_buffer_encode.0 function gets called. Notice the value 0x0E230 being passed in rdx:



The following figure shows how the backdoor checks whether the ‘endbr64’ instruction is still there (if not – it has been probably changed by a debugger which signals the backdoor to abort):

Another important register here is rdi, which will points to the location of the ‘endbr64’ instruction. In other words, if everything is OK, value of [rdi] must be 0xFA1E0FF3, which is the opcode for the ‘endbr64’ instruction.
So now when we go through code, the line add eax, [rdi] should sum 0xFA1E0FF3 and 0x5E2E230 which results in 0xF223. If that is correct 1 is returned, otherwise 0 (indicating that the 'endbr64' instruction was not there). Quite cool.
There is more another function that dynamically modifies code, but that is horrible for reversing – perhaps later ?

Collecting usernames and IP addresses?

The final function we’ll take a look at is also interesting – it will parse every log created by the sshd service and will try to extract valid usernames and IP addresses. The function is .Llzma12_mode_map.part.1 and we can see the initial comparisons (with my comments) here:

So with this, the attacker is parsing a log line such as the following:

Accepted password for root from 192.168.14.1 port 62405 ssh2

Further below in the function we can see that they match on the ‘from’ and ‘ssh2’ keywords and extract what’s between them:

Finally, they pass this to another function, however that’s not visible in static analysis so this is where I lost it.

Guess that’s it for today – depending on developments we might update this diary but in any case check the resources posted at the beginning as they get updated quite often.
If you are looking at this too - hope you're having fun. I quite enjoyed this (too bad it's back to dayjob soon heh), but I'm also scared about how *close* this was to infect zillion servers.

--
Bojan
@bojanz | @linkedin
INFIGO IS

Keywords: backdoor ssh xz
1 comment(s)
ISC Stormcast For Monday, April 1st, 2024 https://isc.sans.edu/podcastdetail/8918

Comments


Diary Archives