Facebook Stored Hundreds of Millions of Passwords as Unmasked Text

"This is something that should have been caught years ago. Why wasn't it?"

The cause of the masking failure—passwords are usually encrypted using a process called hashing that scrambles the readable text into gibberish—was the result of software engineers apparently building applications on their platform that, through an apparent series of errors, ended up recording the unmasked, readable passwords and logged them internally without hashing them properly.

Originally flagged by Krebs on Security, Canahuati’s acknowledgement that “some” users were affected could be seen as a slight understatement. According to Krebs, anywhere from 200 million to 600 million Facebook users had the passwords to their Facebook accounts exposed, some as far back as 2012.

Facebook acknowledges that the affected passwords number in the hundreds of millions of Facebook Lite users—a version of Facebook designed to be accessible to those with poor connectivity or low end devices—, tens of millions of regular Facebook users, and tens of thousands of Instagram users.

Passwords Possible Viewable—And Searchable—by More Than 20,000 Facebook Employees

Canahuati says that “these passwords were never visible to anyone outside of Facebook and we have found no evidence to date that anyone internally abused or improperly accessed them.”

The investigation is still ongoing, however, and there's no way to know the veracity of these assurances given the repeated blows to the company's credibility over the past year and a half when it comes to privacy and data security concerns. What we do know is that these passwords may have been accessible and retrievable via search by the more than 20,000 Facebook employees with access to Facebook’s internal server where the passwords were stored.

That is far too much power over users' privacy and data security for a Facebook employee to have, no matter how well intentioned they may be.

An anonymous Facebook employee told Krebs that “access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.”

In an interview with Krebs, a second Facebook employee, software engineer Scott Renfro, says that so far there isn’t any evidence that anyone deliberately tried to collect this password data.

“We’ve not found any cases so far in our investigations where someone was looking intentionally for passwords, nor have we found signs of misuse of this data,” said Renfro. “In this situation what we’ve found is these passwords were inadvertently logged but that there was no actual risk that’s come from this. We want to make sure we’re reserving those steps and only force a password change in cases where there’s definitely been signs of abuse."

While these may have been unrelated queries that served some other legitimate purpose and no harm may have come from them, Facebook is essentially asking us to take their word for it.

It's Literally Unbelievable That This Simply Slipped Through the Cracks for Years

If I can editorialize a bit here, to say that this should not have happened is understating the case by a few orders of magnitude.

Software malfunctions occur all the time, this is to be expected and sometimes it can take a long time to uncover the cause of particularly subtle software malfunction; a misplaced pair of { } brackets in a piece of code can radically change the behavior of a program even though the program seems to run just fine.

A bot can make 9 million queries of a database very quickly if using a powerful enough processor, which Facebook employees unquestionably have. Facebook also stores an unfathomable amount of raw data on its servers. That being the case, these 9 million searches likely represent a very small portion of the queries made by Facebook employees over a several year period. Its highly understandable that such a small sample size makes detecting the password exposure isn't a gaurantee.

"We’ve not found any cases so far in our investigations where someone was looking intentionally for passwords, nor have we found signs of misuse of this data." — Facebook Software Engineer Scott Renfro, Interview with KerbsOnSecurity

It is also likely that the engineers and developers who made these queries retrieved a data node containing user information, including the unmasked passwords, and never even looked at the data they were looking to query. Programmers can simply used a script or a function to take the data from a specific, unrelated data field of an exposed user's data node and feed that data directly into whatever program they were working on.

In that case, they could make millions of queries an hour and never have to look at a single line of user data, much less the exposed passwords.

The nature of this kind of programming can make it difficult to track down a bug like this by looking at the code and tracing the logic of your program. The systems are simply too complex for this to be a possibility and issues with inputs—especially user-entered inputs like passwords—are among the most unpredictable challenges that programmers have to try to anticipate when designing software.

These sorts of unpredictable problems are precisely why entire libraries of sophisticated testing APIs were created. Using automation, you can test a software module through millions of repetitions using different inputs to stress-test your module and try to break it, thereby exposing hidden vulnerabilities before deploying the software.

Likewise, you can feed in millions of different inputs into a function and validate that the output is what it should be; like, I don't know, maybe whether a password passed into a hashing function actually returns an encrypted password.  Sure, no test is perfect, and nothing can be made 100% secure, but this isn't a exceptionally rare occurrance that exposed a few hundred passwords as unmasked, plain test as an offering to the Random Number God.

Facebook has about 2.5 billion active monthly users, so the 200 million to 600 million users whose passwords were exposed represent, in a rough approximation of the percentage of total Facebook users, around 8-24% of Facebook's active monthly userbase.

That is a massive percentage to have slipped through the cracks for years. It simply isn't possible for these unmasked, plain text passwords to have not shown up during the kinds of rigorous testing you need when dealing with something as sensitive as stored password data. The fact that these unmasked, plain text passwords were "missed" by some of the most "elite" quality assurance teams, security analysts, and developers accessing these data elements for ostensibly unrelated purposes is gobsmacking.

Even if every single one of the exposed passwords represented a user who quit their social media accounts years earlier, it wouldn't matter. The data was still sitting there, fully accessible by internal employees, waving a red flag for anyone to see who bothered to look. This is something that should have been caught years ago. Why wasn't it?

Hell, a bot running a regex algorithm on user passwords fields contained in a user's data file for less than a day would have picked up that recognizable words were showing up in user passwords and set off alarms about this security lapse; masked passwords don't contain the words bronco, patriot, or ILoveBetoORourkeABunch.

[see-also]

Checking billions of user accounts for recognizable patterns in stored passwords that would have exposed this vulnerability sounds like a lot of work, but this is literally what Facebook's algorithms do every moment of every day. This type of data analysis is exactly what Facebook exists on this Earth to do, but it looks like they'd much rather set their algorithms loose on our data to try and figure out what kind of clothes we like so they can sell our preferences to advertisers.

Facebook will undoubtedly disclose more information about this security lapse and what they're going to do to fix the problem, but given Facebook's recent run of scandal around issues of privacy and data security, this is not an encouraging development to say the least. The fact that it was only discovered in January after engineers performing "routine" security testing saw that passwords weren't being masked begs the question why didn't earlier "routine" security tests expose this problem sooner?

Needless to say, failing to secure the data contained in hundreds of millions of their users' accounts by leaving the keys to these accounts—the unmasked, plain text passwords—exposed on their internal company servers is the most spectacular fail yet in what has already been a pretty awful year and a half for Facebook.