Thoughts on the Equifax Data Breach

Categories: Security

In 2017, a large amount of data was stolen from Equifax, a US-based company specializing in online creditworthiness checks. In September 2018, a report on the issue from the US Goverment Accountability Office (GAO) was finally released.

One good news article on the subject is from The Register. It is typical in that the emphasis is placed on two issues:

  • The failure of a system-monitoring tool to detect unusual behaviour within the company network, due to an expired SSL cert.
  • The failure of internal processes intended to detect software packages with known security holes (Struts in this case)

However in my opinion, there are more significant issues.

If I have read the GAO report correctly (see pdf page 15 and 16):

  • The “disputes portal” was not registered in the central software registry as something with a dependency on struts - and thus the people responsible for updating struts code did not know about it
  • The sysadmins responsible for the portal were not on the appropriate email list - a communications screwup
  • The disputes portal was not in a DMZ; apparently once the attackers got a shell on the portal servers, they could access all sorts of servers. That’s a real sysops WTF.

  • Equifax had a database which “contained unencrypted credentials for accessing additional databases”. That’s a real design WTF - having a central DB of credentials makes any system breach into a major one.

  • A scanning tool that was intended to detect struts-based services did not detect this one - that’s probably just bad luck.
  • Some kind of IDS system wasn’t working correctly. It isn’t clear if this was looking for odd system logs on the portal server, or working at the network level.
  • There was no rate-limiting on database queries from the portal to the database. Interesting - I had not previously considered query-rate-limitings as a security measure. Might make some sense, but could also backfire.

Question: was the DB account used by the portal to (legitimately) access its backing DB properly limited to just the tables it needed to access? Once an attacker is on the system, there is no easy way to prevent the attacker from getting the configured creds for the app on that system - but those creds should be appropriately limited. Given that the attackers could access this “database of credentials”, the answer is perhaps not - which would be a WTF.

The GAO report states:

The company’s investigation was facilitated by the use of electronic logs that had not been damaged or erased by attackers on the affected systems.

It is not clear whether these logs were inaccessible to users (eg shipped in real-time to another system), whether the attackers were unable to affect the logs (eg they were not able to gain root privileges), or whether the attackers just didn’t bother to modify the logs.

The register article states that:

Equifax did get lucky on one score: had the attackers erased some of the logs, …

but it is not clear whether that was indeed “luck” or actually good design (storing logs remotely). In either case, ensuring robust logs is obviously an important step..

My guess is that the struts flaw allowed attackers to load up their own code onto the portal server, giving them shell access through which they uploaded and ran tools like nmap. But they probably never got root on the box (or any box); the data they extracted simply came from local-network-accessible databases, including critically the database containing credentials for other databases. And probably the struts-based app itself contained a config-file with credentials for that database.

So in my opinion, this shows:

  • 1 big WTF from the design team - whoever set up this “database of credentials” must have been smoking something
  • 1 big WTF from the ops team (no network segmentation aka DMZ) - a DMZ has been “best practice” since the last millenium.
  • 3 smaller WTFs:
    • software registry portal not correctly holding lists of dependencies (bad, but such registries can be tricky to get right and keep up-to-date)
    • sysadmins not on right email lists (oops, but can happen)
    • IDS system not working correctly (an important screwup, but one that is easy to make and hard to detect)
  • Two unknowns:
    • were the DB credentials for the portal correctly limited?
    • were logs being correctly shipped off-server, or were they just lucky?
  • One dose of bad luck
    • struts detection tool did not fire on this site
  • And something that would have been a good idea:
    • rate limiting of the portal DB connection

And overall, most of these issues would be detected by a competent threat modelling process.

There is of course yet a higher layer of security management that should be providing “meta-policies” that ensure that a company has proper security guidelines for projects, that these are reviewed regularly, and that projects are checked for compliance with policies. That has clearly also failed - a regular internal audit would have detected that either no sensible security policies were present, or that this project was not compliant with them.

Interesting that most of the press articles on this issue have focused on what I described as “one dose of bad luck” - the failure to detect the struts library automatically. I guess that counts as successful PR spin on the part of Equifax!

Update: House Report

In Dec 2018, the US House of Representatives Oversight and Government Reform Committee released a report on the Equifax breach with more details than the GAO report, including:

  • Attacks appear to have originated from Chinese IP addresses
  • Equifax had allowed over 300 security certificates to expire, including 79 certificates for monitoring business critical domains
  • Equifax did not have file integrity monitoring enabled on the (vulnerable) system
  • Equifax had a massive internal legacy-code problem
  • The application that had the Struts-related issue was also later found to have two additional unrelated but severe security issues (SQL injection, Insecure Direct Object Reference)
  • There is no indication that the attackers gained root access on the vulnerable servers
  • An August 2016 report by the financial index provider MSCI Inc. assigned Equifax ’s data security efforts a rating of zero out of ten.
  • There was “a lack of accountability and no clear lines of authority in Equifax’s IT management structure”
  • 3 senior managers “took early retirement” or left the company; one middle manager was fired.
  • Legal action by Government agencies against Equifax is being considered

Interestingly, the explanation of why the automated scan failed to detect the use of Struts is complete nonsense (the scan was executed “on the root directory, not the subdirectory where Apache Struts was listed”). This explanation apparently came from the Equifax Interim CSO; clearly, basic IT competence was not a requirement for this position - or for being on the House review board. See p29 of the report. And directly afterwards, comes the statement that “attackers ran the whoami command to discover other potentially vulnerable servers”; did nobody review this review for basic sense? Later statements claiming “a database of credentials was accessible via a mounted filesystem” is of dubious believability - was this a file or a database? Equally dubious is the claim that an expired SSL certificate “made encrypted data unreadable” - maybe they meant out-of-date in the sense that the primary certificate had been rolled over?

The report is worth reading in full - with either a notepad or a bowl of popcorn, depending on whether you are responsible for such systems or not.

Update 2: Senate Report

In Feb 2019, the United States Senate Permanent Subcommittee on Investigations/ Committee on Homeland Security and Governmental Affairs (phew, what a title!) has released a report:

The content is mostly waffle; there is a big focus on “patch management” which is important but IMO less important than good layered design. Patches are almost always running behind the bad guys, fixing things only after exploits have been detected in the wild and thus while applying them is better than not doing so, designs that protect systems even in the presence of vulnerabilities are better.

This report includes a lot of content also present in the House Report above. However there are a few gems.

Regarding credentials for other systems:

hackers could access credentials for certain other databases and applications because some Equifax employees saved those credentials on a file share.

Ah, so at least a “database of credentials” was not part of the design.

Saving such credentials in a file is dumb, but it is not easy to control what staff do sometimes. Two questions spring out, however:

  • presumably not all Equifax staff are idiots, and some knew this was a bad idea - so is the company lacking any way for its staff to raise security concerns internally?
  • and why would the vulnerable webserver have access to a “file share” at all, let alone one on which users also store their data?

Regarding network segmentation:

Second, once the hackers gained access, they could access certain databases because of a lack of network segmentation within the relevant environment.346 Such network segmentation restricts unnecessary access to other systems once a user is inside a particular environment, such as the dispute portal.347 The lack of segmentation was a conscious decision by Equifax to support efficient business operations and functionality over security protocols.

I’m speechless. A conscious decision not to use firewalls on a server exposed to the internet??!

The scathing title of the report appears to be well deserved..