Critical CloudFlare Vulnerability Causes Sensitive Data Leak. Are Reverse Proxy Solutions Worth It?
User data of Uber, Fitbit, Ok Cupid, 1Password and leading companies was risked for weeks together due to a critical CloudFlare vulnerability. The ‘Cloudbleed Bug’ was caused because of servers running past the buffer and returning memory containing private information. Something similar was seen in the heartbleed bug reported in 2014 too. The vulnerability was reported by Google security researcher Tavis Ormandy. Graham-Cumming, CTO at CloudFlare said that it if difficult to point out which of the 6-million websites have been affected. A number of security experts believe that Cloudbleed has much more impact that CloudFlare is claiming.
Consequences of the Cloudflare Vulnerability
A number of CloudFlare’s services rely on parsing HTML and modifying it on the fly. Cloudflare was using a parser written in ragel and last year decided to write their own parser. Both the ragel and new parser where deployed as nginx modules. While the bug causing the vulnerability was present in ragel parser always, when the new parser was introduced the way parsers interacted with the server changed. This caused a memory overflow and hence the vulnerability surfaced. An in-depth analysis of the cause can be found here. The direct consequences of Cloudbleed are:
- Search engines cached a lot of leaked data and are displaying them on the search results
- Leakage of personal chats on social/dating websites
- User identity information leakage
- Cookies/IP information
The examples we’re finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I’ve informed cloudflare what I’m working on. I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything. -Travis Ormandy
Immediate Precautionary Steps
- Change you passwords. Right from FTP, cpanel, admin panel to all others.
- Force a password change to your users
- If possible, use two-factor authentication
- Clear your websites cache. Purge old cache.
The bigger question we should be asking here is not whether or not to trust Cloudflare, but whether or not we should trust other parties with our traffic at all. As it stands now with this bug (referring cloudbleed) squashed, there’s no reason to trust them any less than counterparts such as Imperva.
Since surfacing of this vulnerability, CloudFlare’s stance has been very firm and upfront. Both the CEO & CTO of CloudFlare have given public statements and taken charge of the situation. However, this entire incident raises questions about reverse proxy solutions expecting you to route your web traffic through them. The biggest consequences of such solutions is that if the server gets hacked, all the websites go down. As seen with CloudFlare, with one flaw millions of websites came to risk.
We interacted with Rafay Baloch, a well-known security researcher and author to take his inputs on the subject. This is what Rafay had to say:
Cloudflare, Akamai, incapsula, Sucuri and many other CDN’s are being used by majority of websites nowadays for handling their traffic. Imagine the amount of critical data that is at risk if a similar vulnerability is found in any other Cloud providers. Another thing is that vast majority of these cloud providers do not have a bug bounty programs. These companies should consider running a lucrative bug bounty program in order to harness the power of crowd to build a safer internet.
A very interesting point by Rafay indeed. Security companies should not ignore the power of security researchers and their ability to bypass best WAFs of the world. We would soon write more on the subject of how a complete security solution should contain a ‘Responsible Disclosure’ component.
While building Astra, one major reason why we did not use reverse proxy implementation was to avoid a situation like this where all of our users are put at risk because of one vulnerability in our infrastructure.