DigiNotar breach - the story so far
I've been following the DigiNotar story as it evolved for a few days now with growing concern and increasing alarm.
I'm by far not privy to the inside information to be able to really assess and audit the situation, so this is purely based on what is publicly known. Being a Dutch native speaker I have access to what the press in the Netherlands writes about it with the subtle nuances that an automated translation will not capture. I do lack the resources to independently double verify everything and as such some errors might still be in it, consider this a best effort at creating some overview and leading up to conclusions with the limited information that is available.
If we do attract the attention of DigiNotar and/or Vasco: please do contact us, we'd love to talk to you and get more information!
So who is DigiNotar and what do they do when all is normal?
DigiNotar is a CA. They sell SSL certificates, also the EV kind.
But there is more that's mostly of interest to those in the EU or the Netherlands only:
They are also (I'm simplifying a bit, I know) an accredited provider in the EU and provide qualified certificates and approved SSCDs to customers to create digital signatures that -by law- in the EU are automatically considered to be qualified digital signatures and as such they are automatically equivalent to manual signatures. This status forces regular 3rd party audits against the relevant Dutch law and standards such as ETSI TS 101 456.
They also provide certificates services under the PKIOverheid umbrella in the Netherlands. This has even more and stricter rules. e.g. Things that are suggested in the ETSI standards, but not mandatory, can become mandatory for PKIOverheid.
DigiNotar is a 100% daughter company of Vasco (since Jan 2011), so if you see Vasco sometimes doing things like press releases regarding the incident, that's why.
So what do we know in a chronological order ?
- Dating back as far as May 2009, the portal of DigiNotar has been defaced, these hacks remained in place till this week after f-secure exposed them in their blog.
Source: f-secure blog
- On July 10th 2011, 283 rogue certificates were signed
Source: spreadsheet released by torproject, and claimed to come from the Dutch government
This included one certificate that was issued with a CN of *.google.com by DigiNotar this is so far the only certificate we have seen.
Source: pasted certificate
- On July 18th 2011, another 124 rogue certificates were signed
Source: spreadsheet released by torproject, and claimed to come from the Dutch government - On July 19th 2011, 128 rogue certificates were revoked
Source: spreadsheet released by torproject, and claimed to come from the Dutch government - On July 20th 2011, another 124 rogue certificates were signed
Source: spreadsheet released by torproject, and claimed to come from the Dutch government - On July 20th 2011, 130 rogue certificates were revoked
Source: spreadsheet released by torproject, and claimed to come from the Dutch government - On July 27th 2011, 75 rogue certificates were revoked
Source: spreadsheet released by torproject, and claimed to come from the Dutch government - On an unknown date, an unknown external auditor did not catch the fraudulent certificate for *.google.com. as well as any others that might be missed as well. Not did they catch the defaced pages.
The specialized press in the Netherlands seems to conclude the auditor was PwC, but there is not much solid proof of that to be found so far.
PwC was DigiNotar's certifying auditor for a lot of their PKI activities as can be seen in the DigiNotar certification list. - On Aug 28th 2011, (some sources claim 27th) a user from Iran posted on a forum using Chrome was warned by his browser the certificate was not to be trusted.
Source: Forum post
Chrome does additional protections for gmail since chromium 13. - On Aug 29th 2011, the *.google.com certificate was revoked by DigiNotar
This can be seen in the CRL at http://service.diginotar.nl/crl/public2025/latestCRL.crl [do not click on this URL, most browsers "understand" CRLs], see further. - On Aug 29th 2011, the response from Google and the other browser makers came: Basically the "sh*t hit the fan" as the browser vendors are pulling the plug on DigiNotar and not trusting their processes anymore.
- On Aug 30th 2011, issue 7791032 in chromium was created. it blocklisted 247 Serial Numbers from certificates issued by DigiNotar and 2 more intermediate DigiNotar certificates. The Serial numbers are available in the patch.
- On Aug 30th 2011, Vasco issued a press release reporting the incident.
- On Aug 30th 2011, various claims of both Vasco, and the Dutch government try to stress that the activities of DigiNotar under the PKIOverheid root were not affected. Some arguments used in the press such as that the root certificate of PKIOverheid is not at DigiNotar (they have an intermediate) are obvious and irrelevant.
- On Aug 30th 2011, DigiNotar released information for users of Diginotar certificates [in Dutch]. This includes a very painful statement: (my translation): "Users of SSL certificates can depending on the browser vendor be confronted with a statement that the certificate is not trusted. This is in 99,9% of the cases incorrect, the certificate can be trusted". I've got nothing positive to say about that.
They also offer a free upgrade to the PKIOverheid realm for those holding a SSL or EVSSL certificate.
- On Aug 31st 2011, Jan Valcke, Operational director at Vasco in an interview with "webwereld" [in Dutch] claims that "dozens" of fake certificates were issued by intruders and that most were recoked on july 19th (minus the one of *.google.com and others that might have been missed).
- On Aug 31st 2011, it is confirmed security company Fox-IT is performing a forensic audit of the systems of DigiNotar. Results are expected next week at the earliest.
Source: webwereld article [in Dutch] - On Sept 3rd 2011, a press released by the Dutch government [in Dutch] shows that after a crisis meeting the Dutch government cancels the trust they had maintained in DigiNotar after the audits by Fox-IT cannot preclude there were no PKIOverheid rogue signatures issued. They take following measures:
- They will switch to other providers in the short term
- They chose for a controlled transition where they take over the operational management of all DigiNotar certificates
- By taking over the operational management they hope to monitor for abuse during the transition. They will invite security specialists to complete the transition as soon as possible.
Vasco actually issued a very short press release on the cooperation as well. It's dated Sept. 2nd (likely due to timezones). - On Sept 4th 2011, the torproject published a spreadsheet (excel and csv) claimed to come from the Dutch government that finally gives an overview of what known rogue certificates had been signed.
Analysis of the spreadsheet published by the torproject
A list of CAs was affected and has issued rogue certificates:
- DigiNotar Cyber CA
- DigiNotar Extended Validation CA
- DigiNotar Public CA - G2
- DigiNotar Public CA 2025
- Koninklijke Notariele Beroepsorganisatie CA
- Stichting TTP Infos CA
Revocations:
- 128 revocations on July 19th
- 130 revocations on July 20th
- 75 revocations on July 27th
- 1 revocation in August 29th
- 198 with a revocation status of "unknown"
The CN list (some had multiple certificates):
- *.*.com
- *.*.org
- *.10million.org
- *.android.com
- *.aol.com
- *.azadegi.com
- *.balatarin.com
- *.comodo.com
- *.digicert.com
- *.globalsign.com
- *.google.com
- *.JanamFadayeRahbar.com
- *.logmein.com
- *.microsoft.com
- *.mozilla.org
- *.RamzShekaneBozorg.com
- *.SahebeDonyayeDigital.com
- *.skype.com
- *.startssl.com
- *.thawthe.com
- *.torproject.org
- *.walla.co.il
- *.windowsupdate.com
- *.wordpress.com
- addons.mozilla.org
- azadegi.com
- Comodo Root CA
- CyberTrust Root CA
- DigiCert Root CA
- Equifax Root CA
- friends.walla.co.il
- GlobalSign Root CA
- login.live.com
- login.yahoo.com
- my.screenname.aol.com
- secure.logmein.com
- Thawte Root CA
- twitter.com
- Verisign Root CA
- wordpress.com
- www.10million.org
- www.balatarin.com
- www.cia.gov
- www.cybertrust.com
- www.Equifax.com
- www.facebook.com
- www.google.com
- www.hamdami.com
- www.mossad.gov.il
- www.sis.gov.uk
- www.update.microsoft.com
This spreadsheet shows revocation of 205 certificates between July 19th and the time of the interview with Jan Valcke, Operational director at Vasco where he claimed DigiNotar had revoked all but the *.google.com one that was found in the wild on the 19th.
Analysis of the Public 2025 CRL
DigiNotar claims all breaches were under the "Public 2025 Root" ref [in Dutch]. What "root" does in there is somewhat unclear to the technical inclined mind, and the "public 2025" just seems to be some sort of internal name. Let's assume they meant the fraudulent certificates all were signed by the same intermediate.
The CRL indicated in the fraudulent *.google.com certificate does indeed point in the same "public 2025" direction, so let's get that CRL:
$ wget http://service.diginotar.nl/crl/public2025/latestCRL.crl
Let's make this file human readable:
$ openssl crl -text -inform DER -in latestCRL.crl >/tmp/t
And let's verify there is indeed the Serial Number in there of the *.google.com fake certificate we found on pastebin:
$ grep -i "05e2e6a4cd09ea54d665b075fe22a256" /tmp/t
Serial Number: 05E2E6A4CD09EA54D665B075FE22A256
So yes, it's revoked. Getting the other relevant lines (it means first figuring out how many, but I skip the boring part).
$ grep -i -A4 "05e2e6a4cd09ea54d665b075fe22a256" /tmp/t
Serial Number: 05E2E6A4CD09EA54D665B075FE22A256
Revocation Date: Aug 29 16:59:03 2011 GMT
CRL entry extensions:
Invalidity Date:
Aug 29 16:58:47 2011 GMT
So that checks out nicely. [One should of course check that all signatures are valid everywhere]
Unfortunately one can only see the Serial Number of the certificates revoked, not the more juicy fields like the CN or so that would allow to see what and when other (fake) certificates were revoked.
But since we have the revocation date, maybe we can see the peak where they revoked the fraudulent certificates. I know the nature of revocation and any other work in a CA/RA can be highly cyclic with huge peaks in it, and I know not to worry about any revocation as such, users loosing control over a certificate happens all the time.
So let's see revocation activity in July 2011 split out per day:
$ grep "Revocation Date:" /tmp/t | sed 's/^.*Date: //' | sed 's/..:..:.. //' |sed 's/GMT//' | sort -n | uniq -c | grep 'Jul .* 2011' 1 Jul 1 2011 3 Jul 4 2011 3 Jul 5 2011 6 Jul 6 2011 6 Jul 7 2011 1 Jul 8 2011 2 Jul 11 2011 6 Jul 14 2011 1 Jul 15 2011 1 Jul 18 2011 2 Jul 19 2011 1 Jul 20 2011 1 Jul 21 2011 3 Jul 22 2011 3 Jul 26 2011 7 Jul 28 2011 5 Jul 29 2011
Uhmm, where is the "dozens" on July 19th ?
Since the *.google.com one was made on Jul 10th, there is no dozens neither before nor shortly after the 19th.
They might have been added to another CRL, hard to say as DigiNotar does not allow directory listing and doesn't have an easy to find list of CRLs they publish either.
Still, even if we look at the "normal" workload in 2011:
$ grep "Revocation Date:" /tmp/t | sed 's/^.*Date: //' | sed 's/..:..:.. //' |sed 's/GMT//' |grep 2011| sed 's/ .. 2011//'| sort -n | uniq -c 93 Apr 34 Aug 112 Feb 144 Jan 52 Jul 18 Jun 118 Mar 118 May
We see that the Jun/Jul and Aug months are very light on revocations. [Note that August was not yet complete in GMT time when I downloaded the CRL file].
I know my sed, grep commands could be optimized to save a few CPU cycles, but this isn't a unix lesson.
I'd love to see the "dozens" of revocations around July 19th in a DigiNotar CRL, but I simply cannot find them.
[for a time we showed here that a number of SNs we had found (246 to be exact) were not included in this CRL]
It's now clear that by far not all fraudulent certificates were revoked in this CRL as the initial information from DigiNotar that all was under the public 2025 CA is simply shown to be untrue. Still it should be carefully examined that all of the known bad certificates were indeed revoked and when that was done.
So what's the known impact right now:
- If you're a general Internet user: you're unlikely to see much impact, maybe you'll run into a website with a DigiNotar certificate somewhere that will now warn the certificate is not trusted anymore.
Keep your browser up to date!
The longer term impact will still have to manifest itself, and for sure breaches such as these will prompt thinking of other solutions.
If the add-ons of Mozilla were indeed attacked using a MitM approach, impact might be more widespread, but that becomes somewhat less likely.
If you really need to access a website that is using a DigiNotar SSL certificate that your browser is not trusting anymore, I'd encourage you not to ignore the warning of the browser, certainly not to add the yanked DigiNotar root certificate back in. Instead the safe procedure is to go examine the certificate and contact the website operator out of band (e.g. by phone). Make them tell you what the fingerprint is of their certificate, verify that with what you see and only then accept the certificate. If you want to be sure you're talking to the right website, you need to perform the work the 3rd party used to do for you, not blindly click OK. - If you're a user in Iran, and had something to hide from your government, odds are you're in trouble with your government.
- Tor users: the torproject confirms the tor network itself is not reliant on SSL certificates. Downloading their client should be done with great care, but the fraudulent certificates that DigiNotar informed them about have by now all expired on their own - revocation can't be confirmed yet.
- You're a customer of DigiNotar: DigiNotar lost the trust from the browser makers and the Dutch government, how permanent that is is too soon to say, but it's a huge unprecedented dent.
Your best option is to seek another provider, if you have not done so already. - If you're a CA or RA, this is yet another big wake-up call. If you're a 3rd party auditor of said, it's the exact same thing. CAs are now a target. Trying to keep breaches such as these secret is hopefully proven to be a disastrous recipe.
What is the biggest thing we all lack to better see what impact there is/was ?
- Full list of fraudulent certificates (CN, SN fields at the very least)
After the publication of the list on Sept 4th we're getting there slowly, but there are still a lot of certificates with an SN as "unknown". - Clarity on when each certificate was created and when it was revoked
After the publication of the list on Sept 4th, we're getting there slowly, but there are still a lot of certificates with a status of revocation as "unknown". - In order too keep trust in other CAs/RAs that were audited by the same auditor that missed the fraudulent *.google.com certificate as well as the defaced pages on the portal, it becomes critical for the auditor in question to speak up and reassure the world this will not repeat itself. This is not intended to publicly humiliate the auditor, but much more a matter of getting confidence back into the system. So a compromise that an unnamed auditor working for well known audit company X is now not an auditor anymore due to this incident is maybe a good start. Add in some measures and guarantees to prevent it from occurring again.
- Clarity over what was affected by the hackers, a full report would be really nice to read. Special attention should be given to explain how it is sure PKIOverheid, the qualified certificates etc. are for sure not affected and how privacy of other customers e.g. was affected. Similarly the defacements should be covered in detail as well as how they could be missed for so long.
Obviously it's unlikely we'll get all those details publicly, but the more we get the easier it will be to keep the trust in the SSL "system" in general.
Trust
A CA has the function of a trusted third party. Their business at the core is based on getting and keeping that trust.
Getting hacked, being reluctant to report the hack, being slow in detecting the hack, being slow in responding to the hack, not being willing or able to provide clarity about the impact of the hack, ... all are tremendously damaging to that trust.
Getting caught spreading incorrect or incomplete information about the hack is however of a whole other magnitude in loosing trust. Compare e.g. this:
- A press release on Aug 30th by DigiNotar containing a statement in Dutch "Gebleken is dat vanuit één subroot (de zogenaamde Public 2025 Root) voor een aantal domeinen ten onrechte certificaten in omloop zijn gekomen. Uit het ingestelde onderzoek is gebleken dat het hier louter om SSL certificaten en EVSSL certificaten gaat die onder deze specifieke subroot zijn uitgegeven. Andere roots zijn onaangetast gebleven, zo ook de root waaruit PKIOverheid certificaten worden uitgegeven en de subroot waaruit de DigiNotar gekwalificeerde certificaten worden uitgegeven. DigiNotar trok toen onmiddellijk alle gecompromitteerde certificaten die uit het onderzoek naar voren zijn gekomen, inwaardoor deze onbruikbaar zijn." my (somewhat free) translation to English: "It is apparent that from one subroot (the so-called Public 2025 Root) certificates for a number of domains have unrightfully been put in circulation. From the investigation it is apparent that it is limited to SSL certificates and EVSSL certificates issued under this specific subroot. Other roots have been unaffected, such as the root from wich the PKIOverheid certificated are issued and the subroot from where the qualified DigiNotar certificates are issued. DigiNotar immediately revoked all compromised certificates revealed by the investigation, making them unusable". It then goes on to cover the missed *.google.com one.
to the analysis of the Sept 4th excel reportedly from the Dutch government that lists:
- 6 CAs that issued rogue certificates, including the DigiNotar Public CA 2025 one.
- That the revocations happened on July 19th, 21st and 27th, and that almost 200 still have an unknown revocation status. [The rogue certificates were issued on July 10th, 18th and 20th].
With the browser vendors and the Dutch government cancelling their trust in DigiNotar and said government taking over the operational management till it can transition, the fate of DigiNotar seems sealed.
Glossary
- CA: Certificate Authority
- CN: Common Name, in case of a SSL certificate for a web server this contains the name of the website, can be a wildcard as well in that case.
- CRL: Certificate Revocation List a machine readable list of revoked certificates, typically published over http. Contains the Serial Numbers (SN) of the revoked certificates along with some minimal supporting data.
- "dozens" used in my text above is a somewhat freely translation of the Dutch "tientallen", literally, "multiple tens"
- ETSI TS 101 456: A technical specification on "policy requirements for certification authorities issuing qualified certificates"; used as a norm in audits of said providers.
Can be freely downloaded from ETSI: version 1.4.3. - EV: extended validation: essentially the same SSL certificate, but with a slightly stricter set of rules on issuing. Most browsers render something like the URL in the address bar in a green color when they see such a certificate
- PKIOverheid: a PKI system run under very strict requirements by and for the Dutch Government. There are 7 providers recognized to deliver certificates under a root certificate held by the Dutch government. This PKI not only issues certificates to (web) servers, but also to companies and individuals to do client authentication against government websites as well as provide means to create qualified digital signatures.
- RA: Registration Authority
- SN: Serial Number
- SSCD: Secure Signature Creation Device. Mostly a smartcard or smart USB token that holds key pairs used for signing and protects the secret keys
Update History
- version 1: initial release
- version 2: updated with more information from the torproject, thanks for the pointer Gary!
- version 3: update to include the DigiNotar press release of Aug 30th.
- version 4: update to add information from the chromium source code
- version 5: update after checking the SNs found in the chromium source code if they appear in the known CRL
- version 6: update with information of the Dutch government as published by the torproject
- version 7: update with information that the Dutch government is stopping their trust in Diginotar and moving away from them
- version 8: some clean up and additions
--
Swa Frantzen -- Section 66
Comments