The xz-utils backdoor in security advisories by national CSIRTs
Errata: It turns out I have missed two advisories when going through the CSIRT sites the first time - I have amended the text and table to reflect this.
For the last few days, the backdoor in xz-utils[1] has been among the main topics of conversation in the global cyber security community.
While it was discovered before it made its way into most Linux distributions and its real-world impact should therefore be limited, it did present a very real and present danger. It is therefore no surprise that it was quickly covered by most major news sites devoted to information and cyber security[2,3,4,5].
However, since the first information about existence of the backdoor was published on Friday 29th[6], which was a public holiday in many countries around the world, and the same may be said of today, it is conceivable that some impacted organizations and individuals might not have learned about the danger from these news sites, as they might only monitor advisories from specific sources – such as national or governmental CSIRTs – during the holidays.
Fast response from national or governmental CSIRTs, or other, similar organizations, in situations like these can therefore be of paramount importance. Consequently, it occurred to me that the current situation might present a good opportunity for a quick analysis to see how many national or governmental CSIRTs/their host organizations/similar entities (e.g., national coordination centers, multi-national or regional CSIRTs, etc.) publish up-to-date warnings and advisories even during holidays.
I have therefore gone over the FIRST membership list[7], which includes (among many other teams) a large percentage of national and/or governmental CSIRTs from around the globe, and identified 105 teams which have a national or governmental constituency and which might therefore possibly function as an “early warning system” for a specific country, region or nation. I have then gone through the official websites of these teams to see which ones did warn about the xz-utils backdoor and when.
The results were interesting, and – at least to me – somewhat surprising. At the time of writing, only 11 (e.g., approximately 10.5%) of the 105 teams/organizations had published an advisory covering the existence of the backdoor. Four of them did so on March 29th, the same day when the existence of the backdoor was first made public, six of them did so the next day – on Saturday 30th – and one did so three days later, on Monday 1st.
At this point, it should be stressed that not all the identified national or governmental CSIRTs (or other relevant organizations) provide a public “advisory service” to their constituencies, so the numbers mentioned above don’t tell the whole story. Additionally, even for CSIRTs/organizations that do provide such a service, a lack of warning about this specific issue is not necessarily an indication that the service doesn’t function efficiently and effectively – every team has its own standards and processes which one can hardly judge from the outside perspective… In short, this article is not intended as a criticism of any of the CSIRTs which did not publish an advisory corresponding to the aforementioned backdoor.
That said, let’s take a look at the list of CSIRTs that were identified as potentially relevant for our purposes. In the following table, you will find the name of each such team/organization, the country it belongs to and information about any relevant advisory it published.
Country/region | Team/organization | Advisory published | Link | Note |
Albania | AL-CSIRT | No | - | - |
Australia | AUSCERT | No | - | - |
Australia | Australian Cyber Security Centre | No | - | - |
Austria | CERT.at | 29.3.2024 | link | - |
Azerbaijan | CERT.AZ | No | - | - |
Azerbaijan | CERT.GOV.AZ | No | - | - |
Bahrain | CERT BH | No | - | - |
Belarus | CERT.BY | No | - | - |
Belgium | CERT.be | No | - | - |
Benin | bjCSIRT | 30.3.2024 | link | - |
Bhutan | BtCIRT | No | - | - |
Botswana | Botswana-CSIRT | No | - | - |
Brazil | CERT.br | No | - | - |
Brazil | CTIR Gov-BR | No | - | - |
Brunei | BruCERT | No | - | - |
Canada | CanCERT | ? | ? | The official CSIRT site was inaccessible |
Canada | Canadian Centre for Cyber Security | No | - | - |
Catalonia | CATALONIA-CERT | No | - | - |
Côte d'Ivoire | CI-CERT | No | - | - |
Croatia | CERT ZSIS | No | - | - |
Croatia | CERT.hr | No | - | - |
Cyprus | National CSIRT-CY | No | - | - |
Czech Republic | CSIRT.CZ | No | - | - |
Czech Republic | GovCERT.CZ | No | - | - |
Denmark | Centre For Cyber Security | No | - | - |
Denmark | DKCERT | No | - | - |
Dominican Republic | CSIRT-RD | No | - | - |
Egypt | EG-CERT | No | - | - |
Estonia | CERT-EE | No | - | - |
Ethiopia | ETHIO-CERT | ? | ? | The official CSIRT site was inaccessible |
European Union | CERT-EU | 30.3.2024 | link | - |
Finland | NCSC-FI | No | - | - |
France | CERT-FR | No | - | - |
Georgia | CERT.DGA.GOV.GE | No | - | - |
Germany | CERT-Bund | No | - | - |
Ghana | CERT-GH | No | - | - |
Hong Kong | GovCERT.HK | No | - | - |
Hungary | NCSC Hungary | No | - | - |
Chile | CSIRT GOB CL | No | - | - |
China | CNCERT/CC | ? | ? | The official CSIRT site was inaccessible |
Iceland | CERT-IS | No | - | - |
India | CERT-In | No | - | - |
Indonesia | ID-SIRTII/CC | No | - | - |
Ireland | NCSC Ireland | 29.3.2024 | link | - |
Italy | CSIRT-IT | 30.3.2024 | link | - |
Japan | CDI-CIRT | No | - | - |
Japan | JPCERT/CC | No | - | - |
Japan | NISC | No | - | - |
Jordan | JoCERT | ? | ? | The official CSIRT site was inaccessible |
Kazakhstan | KZ-CERT | No | - | - |
Kenya | National KE-CIRT/CC | No | - | - |
Korea | KN-CERT | No | - | - |
Latvia | CERT.LV | No | - | - |
Liechtenstein | CSIRT.LI | No | - | - |
Lithuania | CERT-LT | No | - | - |
Luxembourg | CIRCL | 30.3.2024 | link | - |
Luxembourg | GOVCERT.LU | No | - | - |
Malawi | mwCERT | No | - | - |
Malaysia | MyCERT | No | - | - |
Malta | govmtCSIRT | No | - | - |
Mauritius | CERT-MU | No | - | - |
Mexico | CERT-MX | ? | ? | List of vulnerability advisories did not load |
Moldova | CERT-GOV-MD | No | - | - |
Monaco | CERT-MC | No | - | - |
Mongolia | MNCERT/CC | No | - | - |
Montenegro | CIRT.ME | ? | ? | The official CSIRT site was inaccessible |
Morocco | maCERT | No | - | - |
Netherlands | NCSC-NL | 30.3.2024 | link | - |
New Zealand | CERT NZ | No | - | - |
Nigeria | ngCERT | ? | ? | The official CSIRT site was inaccessible |
North Macedonia | MKD-CIRT | No | - | - |
Norway | NCSC-NO | 29.3.2024 | link | - |
Oman | OCERT | No | - | - |
Panama | CSIRT Panama | No | - | - |
Poland | CERT POLSKA | No | - | - |
Portugal | CERT.PT | No | - | - |
Romania | Romanian National Cyber Security Directorate | No | - | - |
Russia | RU-CERT | No | - | - |
Rwanda | Rw-CSIRT | No | - | - |
Saudi Arabia | Saudi CERT | ? | ? | The official CSIRT site was inaccessible |
Serbia | GOVCERT.RS | No | - | - |
Serbia | SRB-CERT | No | - | - |
Singapore | SingCERT | 1.4.2024 | link | - |
Singapore | SG-GITSIR | No | - | - |
Slovakia | SK-CERT | 29.3.2024 | link | - |
Slovakia | CSIRT.SK | No | - | - |
Slovenia | SI-CERT | No | - | - |
Spain | CCN-CERT | No | - | - |
Spain | Basque CyberSecurity Agency | No | - | - |
Spain | INCIBE-CERT | No | - | - |
Sri Lanka | Sri Lanka CERT/CC | No | - | - |
Sudan | Sudan-CERT | No | - | - |
Sweden | CERT-SE | 30.3.2024 | link | - |
Switzerland | NCSC.ch | No | - | - |
Taiwan | TWCSIRT | ? | ? | The official CSIRT site was inaccessible |
Taiwan | TWNCERT | ? | ? | The official CSIRT site was inaccessible |
Taiwan | TWCERT/CC | No | - | - |
Tanzania | TZ-CERT | No | - | - |
Thailand | ThaiCERT | No | - | - |
Tunisia | tunCERT | No | - | - |
Turkey | TR-CERT | No | - | - |
UAE | aeCERT | ? | ? | The official CSIRT site was inaccessible |
Uganda | UG-CERT | No | - | - |
Ukranie | CERT-UA | No | - | - |
United Kingdom | National Cyber Security Centre | No | - | - |
As we already mentioned, the fact that some of the teams listed in the table didn’t publish an advisory for the xz-utils backdoor doesn’t necessarily reflect badly on them – many of these teams don’t provide any security advisory services, vulnerability warnings, etc. at all, and still do a good job, while others might have simply decided not to publish an advisory for this specific issue given that it didn’t meet their internal criteria for advisories.
Still, there will certainly also be a number of teams/organizations which didn’t publish the advisory because they don’t have sufficient personnel or processes to do so during public holidays… So, for any organization which wishes to monitor only a limited number of information sources during such times, it might be advisable to chose those sources very carefuly.
[1] https://isc.sans.edu/podcastdetail/8918
[2] https://www.theregister.com/2024/03/29/malicious_backdoor_xz/
[3] https://www.helpnetsecurity.com/2024/03/29/cve-2024-3094-linux-backdoor/
[4] https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
[5] https://securityaffairs.com/161224/malware/backdoor-xz-tools-linux-distros.html
[6] https://www.openwall.com/lists/oss-security/2024/03/29/4
[7] https://www.first.org/members/teams/
-----------
Jan Kopriva
@jk0pr
Nettles Consulting
The amazingly scary xz sshd backdoor
Unless you took the whole weekend off, you must have seen by now that Andres Freund published an amazing discovery on Friday on the Openwall mailing list (https://www.openwall.com/lists/oss-security/2024/03/29/4).
The whole story around this is both fascinating and scary – and I’m sure will be told around numerous time, so in this diary I will put some technical things about the backdoor that I reversed for quite some time (and I have a feeling I could spend 2 more weeks on this).
There is also a nice gist by smx-smx here that gets updated regularly so keep an eye there as well.
The author(s) of the backdoor went a long way to make the backdoor look as innocent as possible. This is also why all the reversing effort is taking such a long(er) time. Let’s take a look at couple of fascinating things in this backdoor.
String comparison
One of the first things a reverse engineer will do is to search for strings in the code they are looking at. If strings are visible, they can usually tell a lot about the target binary. But if we take a look at the library (and for this diary I am using the one originally sent by Andres) we will see practically no visible strings.
The authors decided to obfuscate all strings – in order to do that, they stored strings as a radix tree (also known as prefix tree or trie, more info at https://en.wikipedia.org/wiki/Radix_tree). This allows them to store all strings as obfuscated, however now one of the challenges they had was to lookup strings – they implemented a function that checks whether a string exists in the radix tree table, and if it does, it returns back the offset:
The image above shows start of the function (originally called Lsimple_coder_update_0) where I also expanded one of the radix tree tables (_Llzip_decode_1). So as input we send a string, and if it exists in the radix tree, we get back its offset. The code needs to do this for every single string lookup, but that should not cause the delay as radix tree lookups are quite fast.
Anti-debugging
Anti-debugging efforts were implemented on several levels. The simplest one includes code where the backdoor is checking if a function starts with the ‘endbr64’ instruction. Let’s check how this works – first the snippet below shows how the .Llzma_block_buffer_encode.0 function gets called. Notice the value 0x0E230 being passed in rdx:
The following figure shows how the backdoor checks whether the ‘endbr64’ instruction is still there (if not – it has been probably changed by a debugger which signals the backdoor to abort):
Another important register here is rdi, which will points to the location of the ‘endbr64’ instruction. In other words, if everything is OK, value of [rdi] must be 0xFA1E0FF3, which is the opcode for the ‘endbr64’ instruction.
So now when we go through code, the line add eax, [rdi] should sum 0xFA1E0FF3 and 0x5E2E230 which results in 0xF223. If that is correct 1 is returned, otherwise 0 (indicating that the 'endbr64' instruction was not there). Quite cool.
There is more another function that dynamically modifies code, but that is horrible for reversing – perhaps later ?
Collecting usernames and IP addresses?
The final function we’ll take a look at is also interesting – it will parse every log created by the sshd service and will try to extract valid usernames and IP addresses. The function is .Llzma12_mode_map.part.1 and we can see the initial comparisons (with my comments) here:
So with this, the attacker is parsing a log line such as the following:
Accepted password for root from 192.168.14.1 port 62405 ssh2
Further below in the function we can see that they match on the ‘from’ and ‘ssh2’ keywords and extract what’s between them:
Finally, they pass this to another function, however that’s not visible in static analysis so this is where I lost it.
Guess that’s it for today – depending on developments we might update this diary but in any case check the resources posted at the beginning as they get updated quite often.
If you are looking at this too - hope you're having fun. I quite enjoyed this (too bad it's back to dayjob soon heh), but I'm also scared about how *close* this was to infect zillion servers.
Comments