Comment by sikhnerd

8 years ago

jgrahamc: can you list which public caches you worked with to attempt to address this? It does not inspire confidence when even google is still showing obvious results

92 comments

sikhnerd

Reply
eastdakota  8 years ago

Google, Microsoft Bing, Yahoo, DDG, Baidu, Yandex, and more. The caches other than Google were quick to clear and we've not been able to find active data on them any longer. We have a team that is continuing to search these and other potential caches online and our support team has been briefed to forward any reports immediately to this team.

I agree it's troubling that Google is taking so long. We were working with them to coordinate disclosure after their caches were cleared. While I am thankful to the Project Zero team for their informing us of the issue quickly, I'm troubled that they went ahead with disclosure before Google crawl team could complete the refresh of their own cache. We have continued to escalate this within Google to get the crawl team to prioritize the clearing of their caches as that is the highest priority remaining remediation step.

  • taviso  8 years ago

    Matthew, with all due respect, you don't know what you're talking about.

    view-source:http://cc.bingj.com/cache.aspx?q=&d=4857656909960944&w=rj9cg...

    view-source:http://cc.bingj.com/cache.aspx?q=&d=4901023173710126&w=n3mEZ...

    view-source:http://cc.bingj.com/cache.aspx?q=&d=4558611265887320&w=urwoW...

    view-source:http://cc.bingj.com/cache.aspx?q=&d=4592983872701813&w=Ghwdd...

    view-source:http://cc.bingj.com/cache.aspx?q=&d=4997243316273666&w=wdpFH...

    Not as simple as you thought?

    • phaed  8 years ago

      Thousands of years from now, when biological life on this planet is all but extinct and superintelligent AI evolving at incomprehensible rates roam the planet, new pieces of the great PII pollution incident that CloudFlare vomited across the internet are still going to be discovered on a daily basis.

      5 replies →

    • dorianm  8 years ago

      Some IPv6 internal connections, some websocket connections to gateway.discord.gg, rewrite rules for fruityfifty.com's AMP pages, and some internal domain `prox96.39.187.9cf-connecting-ip.com`.

      And some sketchy internal variables: `log_only_china`, `http_not_in_china`, `baidu_dns_test`, and `better_tor`.

    • acqq  8 years ago

      Exactly, it looks that the cleaning people up to now only looked for the most obvious matches (just searching for the Cloudflare unique strings). There's surely more where "only" the user data are leaked and are still in the caches.

    • tempz  8 years ago

      The event where one line of buggy code ('==' instead of '<=') creates global consequences, affecting millions, is great illustration of the perils of monoculture.

      And monoculture is the elephant in the room most pretend not to see. The current engineering ideology (it is ideology, not technology) of sycophancy towards big and rich companies, and popular software stacks, is sickening.

    • loup-vaillant  8 years ago

      How about clearing all the cache? (Or at least everything created the last few months.)

      I've never seen anyone suggest it, I suppose It cannot or should not be done for some reason?

      7 replies →

    • amenod  8 years ago

      Incredible. Are they really trying to pin it on Google? Yes, clearing cache would probably remove some part of the information from public sources. But you can never clear all cache world-wide. Nor can you rely that the part that was removed was really removed before being copied elsewhere.

      The way I see it, time given by GZero was sufficient to close the loophole, it was not meant to give them chance to clear caches world-wide. They have a PR disaster on their hands, but blaming Google won't help with it.

    • sah2ed  8 years ago

      The scope of this is unreal on so many levels.

      20 hours since this post and these entries are still up ...

    • dgrealy  8 years ago

      > After I explained the situation, cloudflare quickly reproduced the problem, told me they had convened an incident and had an initial mitigation in place within an hour.

      for what it's worth I think they deserve a little credit

  • __jal  8 years ago

    I find it troubling that the CEO of Cloudflare would attempt to deflect their culpability for a bug this serious onto Google for not cleaning up Cloudflare's mess fast enough.

    Don't use CF, and after seeing behavior like this, don't think I will.

    • fishywang  8 years ago

      On a personal note, I agree with you.

      Before Let's Encrypt is available to public use (beta), CF provided "MITM" https for everyone: just use CF and they can issue you a certificate and server https for you. So I tried that with my personal website.

      But then I found out that they replace a lot of my HTML, resulting mixed content on the https version they served. This is the support ticket I filed with them:

        On wang.yuxuan.org, the css file is served as:

  <link rel="stylesheet" title="Default" href="inc/style.css" type="text/css" />

  Via cloudflare, it becomes:

  <link rel="stylesheet" title="Default" href="http://wang.yuxuan.org/inc/A.style.css.pagespeed.cf.5Dzr782jVo.css" type="text/css"/>

  This won't work with your free https, as it's mixed content.

  Please change it from http:// to //. Thanks.

  There should be more similar cases.

      But CF just refuse to fix that. Their official answer was I should hardcode https. That's bad because I only have https with them, it will break as soon as I leave them (I guess that makes sense to them).

      Luckily I have Let's Encrypt now and no longer need them.

  • Lazare  8 years ago

    This comment greatly lowers my respect for Cloudflare.

    Bugs happen to us all; how you deal with this is what counts, and wilful, blatant lying in a transparent attempt to deflect blame from where it belongs (Cloudflare) onto the team that saved your bacon?

    I've recommended Cloudflare in the past, and I was planning, with some reservations, to continue to do so even after disclosure of this issue. But seeing this comment? I don't see how I can continue.

    (For the sake of maximum clarity: I take issue: 1) with the attempt at suggesting the main issue is in clearing caches, not on the leak itself. It doesn't matter how fast you close the barn door after the horse is gone and the barn has burned down. 2) With the blatantly false claim that non-Google caches have been cleared, or were faster to clear than Google's. Cloudflare should know, better than anyone, the massive scope of this leak, and the fact that NO search engine's cache has or could be cleared of this leak. If you find yourself in a situation so bad you feel like you need to misdirect attention to someone else, and it turns out no one else is actually doing anything so you have to like about that...maybe you should just shut up and stop digging?)

  • glass_of_water  8 years ago

    > I agree it's troubling that Google is taking so long.

    Google has absolutely no obligation to clean up after your mess.

    You should be grateful for any help they and other search engines give you.

    • swsieber  8 years ago

      You're right, I guess. (Disclaimer: Not affiliated with any company affected / involved)

      But I still find it troubling. Is it their mess? No. Does it affect a lot of people negatively - yes. I expect Google to clean this up because they're decent human beings. It's troubling because it's not just CloudFare's mess at this point.

      It reminds me of the humorous response to "Am I my brother's keeper?", which is "You're your brother's brother"

      3 replies →

  • kentt  8 years ago

    I despise the way you've dealt with this issue with as much dishonesty as you thought you could get away with.

    I will be migrating away from your service first thing Monday. I will not use you services again and will ensure that my clients and colleagues are informed of you horrific business practices now and in the future.

  • kentt  8 years ago

    For this who haven't been following along, this is the CEO of CloudFlare lying in a way that misrepresents a major problem CloudFlare created. Additionally, they are trying to blame parts of this problem on those that told them about the problem they created.

  • obituary_latte  8 years ago

    >I'm troubled that they went ahead with disclosure before Google crawl team could complete the refresh of their own cache.

    It sounded like they (cf) were under a lot of pressure to disclose ASAP from project zero and their 7 day requirement...

    • cesarb  8 years ago

      eastdakota is one of the cloudflare guys, so "they" in that sentence can only refer to Google (see also the previous paragraph/sentences, where eastdakota used "we" for cloudflare).

      1 reply →

  • winteriscoming  8 years ago

    >> We have continued to escalate this within Google to get the crawl team to prioritize the clearing of their caches as that is the highest priority remaining remediation step.

    If you are using the same attitude as you use in this comment, with their team, i'm pretty sure they will be thrilled to keep aside all their regular work and help you out cleaning up a enormous mess created by a bug in your service.

  • easuter  8 years ago

    Oh wow, taking a shit on Google after they helped you by reporting a critical flaw in your infrastructure.

    I'm no longer using CF for my own projects, but you've just cemented my decision that none of my clients will either.