Cloudflare Reverse Proxies are Dumping Uninitialized Memory - Sensationalist Headlines

[URL]https://bugs.chromium.org/p/project-zero/issues/detail?id=1139[/URL] [URL]https://news.ycombinator.com/item?id=13718752[/URL] [URL]https://arstechnica.com/security/2017/02/serious-cloudflare-bug-exposed-a-potpourri-of-secret-customer-data/[/URL] [B]Google Project Zero Bug Report:[/B] [Quote=Tavis Ormandy] On February 17th 2017, I was working on a corpus distillation project, when I encountered some data that didn't match what I had been expecting. It's not unusual to find garbage, corrupt data, mislabeled data or just crazy non-conforming data...but the format of the data this time was confusing enough that I spent some time trying to debug what had gone wrong, wondering if it was a bug in my code. In fact, the data was bizarre enough that some colleagues around the Project Zero office even got intrigued. It became clear after a while we were looking at chunks of uninitialized memory interspersed with valid data. The program that this uninitialized data was coming from just happened to have the data I wanted in memory at the time. That solved the mystery, but some of the nearby memory had strings and objects that really seemed like they could be from a reverse proxy operated by cloudflare - a major cdn service. A while later, we figured out how to reproduce the problem. It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output (kinda like heartbleed, but cloudflare specific and worse for reasons I'll explain later). My working theory was that this was related to their "ScrapeShield" feature which parses and obfuscates html - but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers. We fetched a few live samples, and [B]we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users[/B]. Once we understood what we were seeing and the implications, we immediately stopped and contacted cloudflare security. [/Quote] If your site uses Cloudflare, you need to consider every user password, every SSL private key, anything that is transferred over HTTPS and is considered secure compromised. [B]HackerNews User Comments:[/B] [QUOTE=tptacek] Oh, my god. Read the whole event log. If you were behind Cloudflare and it was proxying sensitive data (the contents of HTTP POSTs, &c), they've potentially been spraying it into caches all across the Internet; it was so bad that Tavis found it by accident just looking through Google search results. The crazy thing here is that the Project Zero people were joking last night about a disclosure that was going to keep everyone at work late today. And, this morning, Google announced the SHA-1 collision, which everyone (including the insiders who leaked that the SHA-1 collision was coming) thought was the big announcement. [B]Nope. A SHA-1 collision, it turns out, is the [I]minor[/I] security news of the day.[/B] This is approximately as bad as it ever gets. A significant number of companies probably need to compose customer notifications; it's, at this point, very difficult to rule out unauthorized disclosure of anything that traversed Cloudflare. [/QUOTE] [QUOTE=jkells] My first thought was relief, thank god I'm not using Cloudflare.Where would you even start to address this? Everything you've been serving is potentially compromised, API keys, sessions, personal information, user passwords, the works. You've got no idea what has been leaked. Should you reset all your user passwords, cycle all or your keys, notify all your customers that there data may have been stolen? My second thought after relief was the realization that even as a consumer I'm affected by this, my password manager has > 100 entries what percentage of them are using CloudFlare? Should I change all my passwords? What an epic mess. This is the problem with centralization, the system is broken. [/QUOTE] [QUOTE=fagnerbrack]TL;DR for the lazy ones: > The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything. This is huge. I mean, seriously, this is REALLY HUGE.[/QUOTE] FP is using Cloudflare, it would be wise to update your password

are you fucking serious ARE YOU FUCKING SERIOUS AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA what notable websites use cloudflare so I can protect myself

[QUOTE=Vitisus;51866577]are you fucking serious ARE YOU FUCKING SERIOUS AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA what notable websites use cloudflare so I can protect myself[/QUOTE] Its probably easier to list the ones that don't use it. Cloudflare is ubiquitous as fuck. Boy my workplaces platforms manager is going to have a fun weekend. I hope the rest of the dev team aren't dragged into this mess.

Well, we're gonna have a lot to talk about during my cybersecurity class on Tuesday

[QUOTE=RocketSnail;51866561] FP is using Cloudflare, doesn't look good [/QUOTE] Panic now or later?

Well thats embarrassing. Glad I've only ever used it for the dns functions.

[QUOTE=Vitisus;51866577]are you fucking serious ARE YOU FUCKING SERIOUS AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA what notable websites use cloudflare so I can protect myself[/QUOTE] Here is a list of sites for a start [URL]https://stackshare.io/cloudflare/in-stacks[/URL] Sorry, looks like you need to log in to see more. I'll post if I can find any more resources

as a relatively tech-illiterate person, do I need to do anything and if so what should I do?

[QUOTE=Vitisus;51866577]are you fucking serious ARE YOU FUCKING SERIOUS AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA what notable websites use cloudflare so I can protect myself[/QUOTE] [url]https://twitter.com/taviso/status/834900838837411840[/url] [QUOTE=Tavis Ormandy] Cloudflare have been leaking customer HTTPS sessions for months. Uber, 1Password, FitBit, OKCupid, etc. [/QUOTE]

[QUOTE=Headhumpy;51866619]as a relatively tech-illiterate person, do I need to do anything and if so what should I do?[/QUOTE] as a relatively tech-literate person, I'm wondering the same thing

[QUOTE=Headhumpy;51866619]as a relatively tech-illiterate person, do I need to do anything and if so what should I do?[/QUOTE] This broke 2 hours ago. it's advised to change passwords on any website that uses Cloudflare's services Sites that are suspected to have been vulnerable to this bug (literally this is huge, this news broke 2 hours ago and effected thousands of websites): OKCupid Uber 1Password Reddit Lyft Yelp Pingdom Digital Ocean Montecito Bank and Trust RapGenius Coinbase Bitpay Product Hunt Udemy Crunchyroll FitBit Hacker News Stack Overflow Zendesk Discord

Just a quick skim of the cloudflare blog leads me to believe they fixed it almost as soon as they were aware if it. But they didn't really tell anyone who doesn't follow the right mailing lists until yesterday. A week later. The only thing you can hope for is that nobody found this exploit and used it quietly. Cloudflare estimate a tiny number of requests were impacted in the end. But even a tiny amount of requests is a good few million at their scale.

[QUOTE=RocketSnail;51866635]This broke 2 hours ago. it's advised to change passwords on any website that uses Cloudflare's services Sites that are suspected to have been vulnerable to this bug (literally this is huge, this news broke 2 hours ago and effected a lot more websites): OKCupid Uber 1Password Reddit Lyft Yelp Pingdom Digital Ocean Montecito Bank and Trust RapGenius Coinbase Bitpay Product Hunt Udemy Crunchyroll FitBit Hacker News Stack Overflow Zendesk Discord[/QUOTE] Thankfully I'm only using two of those services, changed both passwords to be safe.

[QUOTE=hexpunK;51866641]Just a quick skim of the cloudflare blog leads me to believe they fixed it almost as soon as they were aware if it. But they didn't really tell anyone who doesn't follow the right mailing lists until yesterday. A week later. The only thing you can hope for is that nobody found this exploit and used it quietly. Cloudflare estimate a tiny number of requests were impacted in the end. But even a tiny amount of requests is a good few million at their scale.[/QUOTE] Yes, but keep in mind that Cloudflare serves over 1 billion people

That's quite a fuck-up.

Oh man, this is going to seriously hurt CloudFlare's reputability.

It's been patched. It's been patched for awhile now. [URL]https://bugs.chromium.org/p/project-zero/issues/detail?id=1139[/URL] Yes, it's a horrible bug and a huge information leak for Cloudflare services. You can't exploit it anymore, but it's been suspected that some crawlers might have picked up some info. They responded almost instantly to Google Project Zero's lead alerting them. It affected 0.06% of the HTTP traffic coming from Cloudflare from their statistics. [B]I repeat: This cannot be exploited anymore and there is nothing you can do but hope your data didn't get saved by a web crawler accidentally at some point in the last few days[/B]

[QUOTE=Gbps;51866676]It's been patched. It's been patched for awhile now. [URL]https://bugs.chromium.org/p/project-zero/issues/detail?id=1139[/URL] Yes, it's a horrible bug and a huge information leak for Cloudflare services. You can't exploit it anymore, but it's been suspected that some crawlers might have picked up some info. They responded almost instantly to Google Project Zero's lead alerting them. It affected 0.06% of the HTTP traffic coming from Cloudflare from their statistics. [B]I repeat: This cannot be exploited anymore and there is nothing you can do but hope your data didn't get saved by a web crawler accidentally at some point in the last few days[/B][/QUOTE] You cannot imagine how many secrets are left in cached pages on Google, DuckDuckGo, Baidu, and other search engines. We're not in the past tense, this [B]is [/B]one of the biggest fuckups in internet history Imagine your password *accidentally* being inserted to the HTML that Google's webpage caching robot sees. your password is now shown in that page when people go to Google search and click "Cached" on a result

[QUOTE=RocketSnail;51866698]You cannot imagine how many secrets are left in cached pages on Google, DuckDuckGo, and other search engines. Data can and is being scraped by web crawlers right now.[/QUOTE] That's what I'm saying. It's absolutely horrible, but the amount of leaked data is finite. We're talking one random person's information [B]maybe[/B] per cached page. The reason the Google engineers saw the data they saw was from repeated use of the exploit they discovered. That means many page loads on many vulnerable pages. More people's information get stolen from large websites on a daily basis just from malware and phishing. There's only so many caches and so many web crawlers.

[QUOTE=Gbps;51866676]It's been patched. It's been patched for awhile now. [URL]https://bugs.chromium.org/p/project-zero/issues/detail?id=1139[/URL] Yes, it's a horrible bug and a huge information leak for Cloudflare services. You can't exploit it anymore, but it's been suspected that some crawlers might have picked up some info. They responded almost instantly to Google Project Zero's lead alerting them. It affected 0.06% of the HTTP traffic coming from Cloudflare from their statistics. [B]I repeat: This cannot be exploited anymore and there is nothing you can do but hope your data didn't get saved by a web crawler accidentally at some point in the last few days[/B][/QUOTE] According to the blog post from Cloudflare as well [quote]The infosec team worked to identify URIs in search engine caches that had leaked memory and get them purged. With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines.[/quote] [URL]https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/[/URL] Also Includes how it came about [quote]The root cause of the bug was that reaching the end of a buffer was checked using the equality operator and a pointer was able to step past the end of the buffer. This is known as a buffer overrun. [b]Had the check been done using >= instead of == jumping over the buffer end would have been caught.[/b] The equality check is generated automatically by Ragel and was not part of the code that we wrote. This indicated that we were not using Ragel correctly. [B]The Ragel code we wrote contained a bug that caused the pointer to jump over the end of the buffer and past the ability of an equality check to spot the buffer overrun.[/B][/quote] And then Cloudflare had a fit when I tired to edit this :v:

[QUOTE=Gbps;51866676]It's been patched. It's been patched for awhile now. [URL]https://bugs.chromium.org/p/project-zero/issues/detail?id=1139[/URL] Yes, it's a horrible bug and a huge information leak for Cloudflare services. You can't exploit it anymore, but it's been suspected that some crawlers might have picked up some info. They responded almost instantly to Google Project Zero's lead alerting them. It affected 0.06% of the HTTP traffic coming from Cloudflare from their statistics. [B]I repeat: This cannot be exploited anymore and there is nothing you can do but hope your data didn't get saved by a web crawler accidentally at some point in the last few days[/B][/QUOTE] Further to this, they had email obfuscation (where the vast majority of the vulnerability was) disabled world wide 47 minutes after Google advised them, and had the other vulnerable module disabled 3 hours and 44 minutes after that. Then patched versions were reenabled a couple of days after. Their blog post is [URL="https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/"]here[/URL] if you want to read it in more detail.

[QUOTE=toyg] Lol, Google just purged that search. EDIT: but there's still plenty of fish: [URL]http://webcache.googleusercontent.com/search?q=cache:lw4K9G2F1WgJ:lightnetwork.ph/ofw-family-day-december-1/&num=1&hl=en&gl=uk&strip=0&vwsrc=1[/URL] This will take weeks to clean, and that's just for Google. EDIT2: found other oauth tokens, lots of fitbit calls... And this just by searching for typical CF internal headers on Google and Bing. There is no way to know what else is out there. What a mess. [/QUOTE] This was commented 1 hour ago

[QUOTE=helifreak;51866712]Further to this, they had email obfuscation (where the vast majority of the vulnerability was) disabled world wide 47 minutes after Google advised them, and had the other vulnerable module disabled 3 hours and 44 minutes after that. Then patched versions were reenabled a couple of days after. Their blog post is [URL="https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/"]here[/URL] if you want to read it in more detail.[/QUOTE] This. It's all up to crawlers to purge their vulnerable site's caches now. This is not up to any particular person now, just large companies with big cache stores.

[QUOTE=jgrahamc] We identified 3,438 unique domains. I'm not sure if those were all sent to Tavis because we were only sending him things that we wanted purged. [/QUOTE] Cloudflare rep in response to how many sites are effected by this bug

Personally, as a security engineer, I know it would be a nightmare to be at Cloudflare security right now. That being said, they did everything they could do. 1) They identified and stopped the service leaking the data 47 minutes after the vulnerability was discovered. This is a very fast turnaround time. Sometimes it takes companies months to patch. 2) They fixed the vulnerability and had the patched service up in a few days 3) They worked with search engine companies to purge caches of known vulnerable pages 4) They were 100% transparent on their blog about the problem, its cause, and what they did to solve it, and what they will do in the future to prevent it. It's like a spacecraft crash. It's horrible, but it is highly unlikely to affect you, and it also doesn't mean NASA isn't qualified to send people into space anymore. Feel me?

-

[QUOTE=Gbps;51866676]It's been patched. It's been patched for awhile now. [URL]https://bugs.chromium.org/p/project-zero/issues/detail?id=1139[/URL] Yes, it's a horrible bug and a huge information leak for Cloudflare services. You can't exploit it anymore, but it's been suspected that some crawlers might have picked up some info. They responded almost instantly to Google Project Zero's lead alerting them. It affected 0.06% of the HTTP traffic coming from Cloudflare from their statistics. [B]I repeat: This cannot be exploited anymore and there is nothing you can do but hope your data didn't get saved by a web crawler accidentally at some point in the last few days[/B][/QUOTE] What data, precisely? If I logged into my bank and instructed my browser to save my session as a cookie to bypass 2FA and I used a CloudFlare site, is my bank account at risk? What exactly happened here and what precisely is at risk? Everyone is talking about how fucked up this is but nobody is talking at all about what steps I need to take immediately to protect myself.

[QUOTE=mn_chaos;51866742]GUYS, CHANGE YOUR PASSWORD, PLEASE. There might not be verifiable pastes floating around (thank fucking goodness) but I know for a fact that this is a 0 day that has been extensively used, even on this site. I apologize for not creating a thread earlier, but I am blocked from this site and this is the soonest I can reach it.[/QUOTE] It's not a 0 day if it's patched... I'm searching caches right now. I can't find a hit on facepunch.com Please with the sensationalism here. [editline]23rd February 2017[/editline] [QUOTE=SGTNAPALM;51866751]What data, precisely? If I logged into my bank and instructed my browser to save my session as a cookie to bypass 2FA and I used a CloudFlare site, is my bank account at risk? What exactly happened here and what precisely is at risk? Everyone is talking about how fucked up this is but nobody is talking at all about what steps I need to take immediately to protect myself.[/QUOTE] That's the thing that makes this so crazy. No one knows what was leaked. Uninitialized memory is highly volatile. What YOU (an individual) need to know is that, statistically, your information being leaked, traced back to you, and then used against you, is [b]highly unlikely[/b] The big issues are with things having to do with the site itself. Private website certificates, etc.

[QUOTE=SGTNAPALM;51866751]What data, precisely? If I logged into my bank and instructed my browser to save my session as a cookie to bypass 2FA and I used a CloudFlare site, is my bank account at risk? What exactly happened here and what precisely is at risk? Everyone is talking about how fucked up this is but nobody is talking at all about what steps I need to take immediately to protect myself.[/QUOTE] Just change your passwords to accounts that you care about. should be very safe if your account has 2FA.

[QUOTE=Gbps;51866763]It's not a 0 day if it's patched... I'm searching caches right now. I can't find a hit on facepunch.com Please with the sensationalism here. [editline]23rd February 2017[/editline] That's the thing that makes this so crazy. No one knows what was leaked. Uninitialized memory is highly volatile. What YOU (an individual) need to know is that, statistically, your information being leaked, traced back to you, and then used against you, is [b]highly unlikely[/b] The big issues are with things having to do with the site itself. Private website certificates, etc.[/QUOTE] Whose uninitialized memory? My memory, or Cloudflare servers' memory? Did my browser transmit this data to the faulty Cloudflare website, or did the Cloudflare website not receive any "unkosher" memory from me but still leak its own memory and all that entails?