Comment by eastdakota
7 years ago
We don’t block archive.is or any other domain via 1.1.1.1. Doing so, we believe, would violate the integrity of DNS and the privacy and security promises we made to our users when we launched the service.
Archive.is’s authoritative DNS servers return bad results to 1.1.1.1 when we query them. I’ve proposed we just fix it on our end but our team, quite rightly, said that too would violate the integrity of DNS and the privacy and security promises we made to our users when we launched the service.
The archive.is owner has explained that he returns bad results to us because we don’t pass along the EDNS subnet information. This information leaks information about a requester’s IP and, in turn, sacrifices the privacy of users. This is especially problematic as we work to encrypt more DNS traffic since the request from Resolver to Authoritative DNS is typically unencrypted. We’re aware of real world examples where nationstate actors have monitored EDNS subnet information to track individuals, which was part of the motivation for the privacy and security policies of 1.1.1.1.
EDNS IP subsets can be used to better geolocate responses for services that use DNS-based load balancing. However, 1.1.1.1 is delivered across Cloudflare’s entire network that today spans 180 cities. We publish the geolocation information of the IPs that we query from. That allows any network with less density than we have to properly return DNS-targeted results. For a relatively small operator like archive.is, there would be no loss in geo load balancing fidelity relying on the location of the Cloudflare PoP in lieu of EDNS IP subnets.
We are working with the small number of networks with a higher network/ISP density than Cloudflare (e.g., Netflix, Facebook, Google/YouTube) to come up with an EDNS IP Subnet alternative that gets them the information they need for geolocation targeting without risking user privacy and security. Those conversations have been productive and are ongoing. If archive.is has suggestions along these lines, we’d be happy to consider them.
Honestly, Cloudflare choosing not to hastily slap a band-aid on a problem like this just makes me feel more compelled to continue using 1.1.1.1.
I hesitate to compare this to Apple calling themselves “courageous” when removing the headphone jack, but in this case, I think the word is appropriate. I’ll happily stand behind you guys if you take some PR hits while forcing the rest of the industry to make DNS safer – since it is understandable, admittedly, for users to conclude that “Cloudflare is blocking websites, sound the alarms!” at first glance.
For the moment, I also do trust CloudFlare's intentions, but it's wrong to classify this as some kind of stoic resolve in not "slapping a band-aid on a problem" since that's exactly what they did after their business decision about not responding to "any" queries.
What do you mean with this? Refuse ANY is now a proposed RFC https://datatracker.ietf.org/doc/rfc8482/ How is that a band aid?
2 replies →
Please note that incentive to "hastily slap a band-aid" did appear, but was overcome by the team. At least, they deserve praise for honesty.
Apple's move is solely based on greed. Sell people a product they eill easily lose as well as having a limited lifetime. Easy money.
Say you remove/don't proxy the ECS information, and I get some generic, non-geo-location aware response back. In the majority of cases, wouldn't my next step be to open a TCP connection to the IP in the response, and immediately leak my full IP address to the other end? While I get (and appreciate!) the concern for the user's privacy, I'm having a hard time seeing what practical effect not proxying the subnet the user is on has?
(This is not meant to suggest that archive.is's DNS response is appropriate, or that CF's setup is inappropriate.)
(Just to check my understanding of ECS: it's an extension to DNS that sends the user's subnet in the request, and gets relayed with the request, s.t. an authoritative server can respond with a geo-location appropriate response/IP.)
> Say you remove/don't proxy the ECS information, and I get some generic, non-geo-location aware response back. In the majority of cases, wouldn't my next step be to open a TCP connection to the IP in the response, and immediately leak my full IP address to the other end?
That assumes that the nameserver and the actual server are run by the same party which quite often is not the case.
> That assumes that the nameserver and the actual server are run by the same party which quite often is not the case.
Cloudflare can check if nameserver and the actual server are run by different parties, and if so omit subnet information from EDNS response. It is not hard to implement — Google and OpenDNS used to require manual whitelisting to receive EDNS subnet responses (not sure if they still do).
Cloudflare's CDN leaks user's full online identity to Google via reCaptcha, especially when you use Tor. Maybe they should ask Google to be satisfied with client's subnet too?
2 replies →
Alternatively:
Cloudflare simply is making a subversive play against their competitor CDNs. Client subnet of a DNS request is used for initial rough mapping by Cloudflare competitors such as Akamai (definitely) and I believe Fastly ( and probably others) . Stripping it easily adds at least a few milliseconds to the time to first byte and most likely results a request re-routing on the second or third request.
After all, no other CDN is operating a well used public resolver.
As this is related to CDN, I am gonna leave it here.
The irony is one.one.one.one is marketed as getaway to faster internet, while making CDNs that use GeoDNS slower.
All it takes is a bad route to a far away cloudflare POP to make your internet really slower. Case in point. [1]
I really don't find why no EDNS is considered private, as it only sends the IP subnet.[2] And on IPv6 the IP is far more protected.
If you care that much about privacy, you should be using a VPN.
[1] https://pastebin.com/raw/QnbWXU1a
[2] https://tools.ietf.org/html/rfc7871#section-11.1
> If you care that much about privacy, you should be using a VPN.
Another point; if you care about privacy, why use a 3rd party resolver that you have to "trust"?
Use the ISP resolver; they can see all your traffic anyway if they want to.
Alternatively, cut out all the middle men and run your own recursive resolver. It's not complicated to do so, there's other software than Bind for doing so.
Google has its own public DNS and CDN, I'm pretty sure that counts.
Isn't Google CDN a public beta or did it just exit a public beta into a GA? If so, it is a non-entity for at least a year long contracts that the other CDNs have with its customers. Probably a non-entity for years to come.
1 reply →
Thank you for your comment.
Since HTTPS traffic already reveals communicating IPs to nation-state actors, could you clarify what attack vector removing user IP info from authoritative DNS queries protects against?
In what way does Cloudflare publish its PoP geolocation? Is it a Cloudflare-specific API? Why not fake EDNS subnet info by providing the PoP’s?
I notice of course that Google, Facebook, and Netflix still work on 1.1.1.1. Does this mean they’re currently using Cloudflare PoP geolocation in lieu of EDNS subnet information?
Its preventing the DNS authority to know the IP of who is making the request.
CloudFlare decided its DNS should be the authority to the end user and Archive.is's DNS should be the authority only to CloudFlare. CloudFlare is breaking the bond between the end user and the Service provider.
What CloudFlare is doing is centralizing authority to itself rather allowing authority to be distributed to all owners of the domains as intended. An argument can be made that by using 1.1.1.1 you are granting CF permission to act in this role - some users may even prefer it.
This is no different than any 3rd party DNS service. If the resolving DNS server you hit doesn't have a cached response, it reaches out to the upstream resolver. It doesn't pass your IP along to the upstream resolver
2 replies →
An example of something that Cloudflare's approach provides some protection from: http://dnscookie.com/
> We publish the geolocation information of the IPs that we query from. That allows any network with less density than we have to properly return DNS-targeted results.
The operator of archive.is claims that they suffer from a "massive mismatch" between those query IPs and actual traffic. Any idea why? [Is that claim wrong? Is archive.is to blame? Is cloudflare to blame? Are ISPs badly routing the DNS queries?]
Do you have stats on how well the geolocation works in practice?
Well no, CloudFlare doesn't get to talk about not "violating the integrity of DNS" after you stopped responding to "any" queries in violation of the standard. You started by doing your own thing and then proposed a change to the standard to fit your business decision. [0]
[0] https://www.rfc-editor.org/info/rfc8482
There's a difference between changing results (or adding) and not supporting a feature that is dangerous and rarely used. Kind-of like banning handguns vs. providing unknownly modified guns.
They could have allowed "any" only via TCP. Instead Cloudflare told everyone "our software can't handle any, so yours shouldn't either".
Wait, but both of these are horrible ideas. Horrible analogy; theres no need to bring politics into this.
1 reply →
They also stopped responding to all DNS queries for some neonazi asshats because of public pressure and politics. They're definitely jerks but they still should be treated like any other customer unless they're actively breaking the law.
Thanks for the detailed response! I think your team is handling this the right way.
@eastdakota what about just failing without response on archive.is calls so the second resolver address configured in the client will be used? I understand this is also a DNS integrity violation, however the result for the end user would be either the same if they don’t have a second resolver configured or enhanced if they do.
The current effect is I stop using 1.1.1.1 when I need archive.is (often) and set it back the next time I’m messing with my network settings.
DNS either has integrity or it doesn’t. We get a response from an Authoritative server and, as a Resolver, we believe our responsibility is to return it. If we start making exceptions because of bad PR, how can you trust us to do the right thing when the stakes are even higher (e.g., nationstate pressure)?
As an aside, I used to think that when Emerson said that “a foolish consistency is the hobgoblin of little minds” he meant that we were foolish to try and be consistent. Increasingly I wonder if instead he meant that when you’re trying to reason with people who may not have the same detailed knowledge of a problem as you, there’s an enhanced importance to being consistent. Unfortunately, most policy makers globally don’t have a detailed understanding of how technical systems like DNS work, so we think it’s especially important we be consistent.
My take on the Emerson quote you mention is to be mindful instead of mindless when it comes to consistency. I respect the commitment to consistency you convey (and I do think it is mindful).
I would recommend you leave exegesis of Emerson to the experts. What he meant is much closer to "pave the cowpaths" than "break things that currently work by enforcing arbitrary standards".
If you're going to that much trouble I suggest you just hardcode an IP address for archive.is into /etc/hosts. I've only had to change it once in the whole time I've used Cloudflare DNS (i.e. since the first day it was public).
If you use dnsmasq, you can special case archive.is to not be resolved via 1.1.1.1.
Also: it'd be nice if CloudFlare made a secondary DNS resolver (1.1.2.2?) that didn't pass along EDNS information, as a backup for websites like archive.is (and for anyone who cares about privacy).
I think you may have typo’d, but just in case:
1.1.1.1 does not send EDNS ECS data, specifically because of the privacy concern. So the hypothetical secondary resolver would need to send that data, for people who aren’t concerned about the privacy implications / want to get to archive.is.
Given CloudFlare’s stated message of prioritizing privacy, it seems unlikely they’d stand up infrastructure that behaved like 1.1.1.1 except that it leaked more private information.
2 replies →
I just added an entry for archive.is in my etc/hosts.
How do I do that on my iPhone?
4 replies →
If you're for integrity of DNS, why did you suspend the free speech of the admittedly bigoted, hateful neonazis on dailystormer?
https://blog.cloudflare.com/why-we-terminated-daily-stormer/
"Earlier today, Cloudflare terminated the account of the Daily Stormer. We've stopped proxying their traffic and stopped answering DNS requests for their sites. We've taken measures to ensure that they cannot sign up for Cloudflare's services ever again."
I'll keep using non-logging, encrypted OpenNIC servers, since you seem to selectively censor instead of only blocking terrorists and cp.
How does doing this censor them?
The Daily Stormer is free to get their business elsewhere and it's still up on the internet. Cloudflare didn't want to be associated to this kind of content, and thus terminated their business relation.
We don't NEED Cloudflare to keep the internet integrity (if we did, it will go pretty badly...) but we do need DNS to keep the internet integrity.
> I'll keep using non-logging, encrypted OpenNIC servers, since you seem to selectively censor instead of only blocking terrorists and cp.
Why are you censoring Cloudflare? /s
I don't think this argument is well stated, so I'll give it a shot.
CloudFlare is very basic infrastructure and there are a handful of companies providing such infrastructure, thus a group can be effectively deleted from the Internet if these companies decide or are pressured to do so. (Example of pressuring: Patreon dropped some accounts at the behest of Mastercard.)
So maybe the real question is, "does this notion of the integrity of DNS extend to other basic infrastructure services?"
Why not just send the subnet of the machine at cloudflare doing the querying?
The full IP of the Cloudflare resolver doing the recursive resolution is already provided to the authoritative server, as the source IP for the DNS query traffic.
I think the parent is saying, why not spoof the EDNS client subnet information?
6 replies →
Could you use your own subnet in the EDNS that matches client's country or could you let user configure what data would be shared?
Ongoing work with big companies to replace existing technologies don't convince me. Though, neither does whining when the authoritative nameserver itself is returning bogus responses.
The experience for the user is that the page just keeps loading indefinitely, while showing a blank page. Is there nothing Cloudflare could do to inform the browser about the situation, so that the browser can show some kind of a message to the user? As it stands, to the user it just looks like their connection died.
I don't see what they can do, short of sending fake DNS replies with their own webserver IPs, which is worse for the integrity of DNS.
Shouldn’t there be some kind of standard DNS response for this? There are dozens of different HTTP status codes for all kinds of scenarios. Doesn’t DNS have something like that? I know almost nothing about DNS, I’m just curious.
1 reply →
Awesome response! Is there already a blog post on this?
Such a post might A) get better SEO than an HN thread for 'cannot access archive.is [or ...]' and B) help change its behaviour.
If the govt of your jurisdiction (American I assume?) commanded you to censor a certain domain or block of IPs with a court order, what exactly happens? I'm not sure if this has been done on the DNS level before but do you guys have a plan in case it ever does happen?
Jurisdiction would include every place where they have a business presence. Their page https://www.cloudflare.com/en-au/about-overview/ lists quite a few international phone numbers, which may or may not correspond with offices and subsidiary companies in those locations.
I assume they'd just have to go along with such legal demands, or withdraw from the relevant country, unless the penalty for not complying was very small.
It will probably become an issue some day. In Australia, for example, courts can issue DNS bans of particular sites to individual ISPs. You can avoid these bans entirely by using a service like Cloudfare DNS.
Or you know you Piss off CloudFare CEO and he directs them to censor a site... Which has happened in the past
Can you cite any source on this?
2 replies →
Exactly. If it's not "the right kind" of content behind a domain, it doesn't even take a court order for CloudFlare to censor it.
2 replies →
Are there any other known sites that don't work with 1.1.1.1 but work fine on other resolvers?
Typically, if you experience that, it’s because DNSSEC fails. 1.1.1.1 enforces DNSSEC. As does 8.8.8.8 in most, but not all, cases. Many other DNS resolvers do not enforce DNSSEC. Archive.is (and its directly affiliated sites) are the only exception like this I am aware of. And, to be clear, as a policy the 1.1.1.1 DNS does not block any sites from resolution.
I (random HN user) happen to know of lancaster.ac.uk (there was a comment thread a while back where this was mentioned).
In what way doesn't it work? This is with my ISP's DNS (using which I can visit https://www.lancaster.ac.uk/ in a browser):
and this is with Cloudflare's:
Looks the same to me.
This is a problem of the 1^4 resolver not implementing DNAME support (either not a priority, or just in the backlog): https://community.cloudflare.com/t/www-lancaster-ac-uk-not-r...
Thanks for the explanation. One more reason to keep using you instead of anything else.
tl;dr: Don't use 1.1.1.1 if you use any services that use DNS geolocation to bring you resources from the closest datacentre. These include: Office 365, Netflix, Facebook, Google services, ...
Huh, don’t see many CEOs writing and talking like this. And I don’t think a flunky wrote this.
Wow, I'm totallly not on the x company awesome today- x company terrible tomorrow with x = cloudflare but I'm impressed that you guys are doing due diligence!
archive.is is a very important tool in online extremism research and you've taken money from far-right extremists, your explanation for why it's inaccessible seems incomplete.
This is probably where I get banned from Hn but it has to be said - to posture as if you care about end users while in the same breath taking money from extremists and turning over personal identifiable information to far-right outlets like DailyStormer, is disingenuous at best and I can think of other ways to describe it which are less charitable.
You also host and protect 8chan.
https://twitter.com/ncweaver/status/1124091916520497153
https://twitter.com/klarajk/status/1122625367490146304
https://twitter.com/Riverseeker/status/1122612031234945024
https://twitter.com/slpng_giants/status/1123592717341200384
https://twitter.com/NathanBLawrence/status/10562868097418199...
https://twitter.com/NJDemocrat/status/897147112273608705
https://twitter.com/InvestMib/status/1123308004873515015
https://twitter.com/jwz/status/1124415034610860033
This is amusing, They Banned the DailyStormer which I why I will never support them. While I disagree 100% with the DailyStormer it is not up to cloudflare to decide who can and can not speak, who can and can not access the internet.
The concept of Free Speech is the most important right we have as humanity, while I may not agree with some peoples words I will fight for their right to say those words
And do not even come at me with "well they are private company" we impose all kinds of regulations on private companies when it comes to basic human rights like free speech and Free Association for example private companies can not refuse service based on race, sex, age, etc.
yet you WANT them to censor content, censor speech. You want them to apply your left authoritarian world view to legal speech, and yes everything you have cited is LEGAL SPEECH in the USA.
If there are actual threats, True Threats as defined in US law, then the police should be involved and the people arrested. If there is defamation or other illegal speech then the courts should be involved
It should NOT be the position of private companies to regulate speech online
Platform Access Is A Civil Right. https://humanevents.com/2019/05/03/platform-access-is-a-civi...
Yes, I want them to censor lies and misleading speech. People or services that feed the public dangerous misinformation should be silenced.
I realize that’s a slippery slope, but I just don’t trust the public to filter for themselves any more.
7 replies →
I think you're being downvoted because of the bit about regulation. At least, that is what I choose to believe, because imagine our state of affairs if you are being downvoted because of your comments about the idea of free speech.
2 replies →
Eastdakotas replies here and in previous threads indicate that his team have more oversight into his own wants and desires. He also writes more about being consistent in actions than they had when he did this action. I think that's a positive way forward. But the person at the top will be the weakest link. If I was a nation state that's where I would be applying the force not at the company level. Maybe he also realised this?
race, sex, age, etc.
Where does daily stormer fall in the “etc.” part?
1 reply →
I think the tone of this comment as "courageous" is especially humorous coming from a throwaway account.
Anonymity online. If you have nothing to hide you have nothing to fear.
I'm far from courageous, in fact I'm scared because these groups regularly DoX those who draw attention to this pressing issue:
https://thenextweb.com/opinion/2018/07/17/the-daily-callers-...
http://www.sfweekly.com/news/daily-caller-doxxes-the-s-f-guy...
Journalist like Robert Evans are courageous: https://www.bellingcat.com/news/americas/2019/04/28/ignore-t...
Researchers like Whitney Phillips are courageous https://www.wired.com/story/existential-crisis-plaguing-onli...
I'm just disgusted.
Encrypting dns is bad for end users. Please cut this shit out. You are acting like you are defending against the NSA, but in reality we will have a bunch of shitty IoT phoning data to indecipherable IP addresses without any meaningful defense of consumer privacy.
It is hostile to customers who want to troubleshoot wtf apps are doing.
Normal DNS queries aren't encrypted. It's normal queries on port 53.
Users/programs/IoT can choose to use DNS-over-TLS or DNS-over-HTTPS, but that's not Cloudflare's fault.
Nothing in his response is about encrypting DNS. Go grind your axe elsewhere.
In my country, government/ISP blocks websites and changes the DNS results of 8.8.8.8 since it is not encrypted. If ISP can create a valid certificate, that browsers trust [1], they may be able to access my Gmail or Github account.
[1] https://www.zdnet.com/article/mozilla-to-chinas-wosign-well-...