Cloudflare, one of the larger content-delivery networks and DNS providers on the Internet, had a critical bug in one of its services that resulted in sensitive customer data such as cookies, authentication tokens, and encryption keys being leaked and cached by servers around the world.
The vulnerability was in an HTML parser that Cloudflare engineers had written several years ago but had recently replaced by a newer one. The company was migrating various services from the old parser, written using Ragel, to the new one, and a change made during that process is what caused the bug to activate and begin leaking memory with private information in it. The bug active for several days, and Cloudflare said the most critical period was Feb. 13 to Feb. 18.
“It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself,” John Graham-Cumming of Cloudflare said in a post-mortem on the response to the vulnerability.
Cloudflare has a massive and diverse customer base that includes companies such as Uber, Yelp, OkCupid, Medium, and 1Password. There is a running list being maintained of all of the known customers, including some that are known not to have been affected by the vulnerability. 1Password is among those who have said their data was unaffected.
The bug had a broad potential effect for Cloudflare’s customers, as well as for the company itself. Because of the way the company’s infrastructure is set up, a request to one Cloudflare site affected by the vulnerability could end up revealing private information from a separate site. Also, search engines routinely cache web content for faster serving, and some of the leaked private data from Cloudflare sites had been cached by Google and other engines.
“We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data.”
“The infosec team worked to identify URIs in search engine caches that had leaked memory and get them purged. With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines,” Graham-Cumming said.
“We also undertook other search expeditions looking for potentially leaked information on sites like Pastebin and did not find anything.”
Some of the sensitive data leaked by the vulnerability belonged to Cloudflare itself rather than its customers. Although no customer encryption keys were leaked, an SSL key Cloudflare used to encrypt connections between its own machines did, as did some other internal authentication secrets.
A researcher with Google’s Project Zero discovered the memory leak last week while doing unrelated research, and after confirming what he had found, reached out to CloudFlare’s security team immediately.
“It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output. My working theory was that this was related to their ‘ScrapeShield’ feature which parses and obfuscates html – but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers,” researcher Tavis Ormandy of Google said in his initial analysis of the flaw.
“We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users.”
Cloudflare implemented a partial fix for the memory leak within a few hours of Ormandy’s initial report and fully fixed it earlier this week.
Image: Maarten Van Damme, CC By license.
