Some aspects of the described behavior are as we intended and some are not. The cause is not exactly as described in the blog post. As for mitigation, we are already testing a patch of the unintended behavior on a subset of our infrastructure. If any of you try to reproduce the blog post's findings you may get confusing results throughout the day.
We will also re-evaluate whether the intended behaviors are acceptable or not. Some of this is a trade-off between multiple aspects of privacy, and multiple aspects of user experience.
Please note that this is my current understanding, which may change. I was only made aware of this an hour ago, and most of that time was spent talking with Ops, considering what to do immediately, and writing this post.
Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away.
You really do provide a reassuring, good service -- thank you.
It's also worth stating that the client (including the cli client -- which, with a bit of work, you can get running in most situations where you'd use native wireguard) by default has a key rotation interval of I think 72 hours.
`mullvad tunnel get` will show it and `mullvad tunnel set rotation-interval <hours>` will change it. This is the preferred mitigation method of the post.
I personally don't mind having a pseudo-static IP (some other suppliers offer a static IPv4 as a feature!) as I wish to prevent network-level snooping from my ISP and governments. It's also worth stating that I think having a smaller IP space is an advantage for a privacy VPN: there are more potential users acting behind any given externally visible IP. Combined with technologies like DAITA (which effectively adds chaff to the tunnel) and multi-hop entrances and I personally think that this service really does plausibly make harder the life of those who snoop netflows all day.
I just want to say I absolutely love Mullvad! You guys did a fantastic job at designing a genuinely good and trustworthy (as much as possible) VPN vendor. You communicating here is just another data point towards this.
It already has official packaging for Tumbleweed, see https://github.com/mullvad/mullvadvpn-app/issues/2242 for the upstream issue. Leap can use the normal Linux application, you will just have to provide the dependencies yourself.
> Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings
How to report a bug or vulnerability
... we (currently) have no bug bounty program ... send an email to support@mullvadvpn.net
I'm a long-time Mullvad user. I will continue to buy and use Mullvad VPN services (with my credit card that has my name on it) so long as it is legal to do so in my country.
VPNs are not 100% anonymous. They are not meant to be. Instead, they are meant to provide some level of privacy to law-abiding adults.
Most people would be embarrassed if their co-workers and neighbors knew the intimate personal details of their lives. Things they like, things the buy, things they do, etc. So, most people should use a VPN to protect their privacy.
By definition, 'most people' don't want or expect 100% anonymity online. They just want a bit of privacy in their personal life and their relationships. That's it.
VPNs don't protect (and are not intended to protect) criminals who want 100% anonymity from governments while committing online crimes. This is an important distinction. 'Most people' are not criminals and do not have this unrealistic expectation from Mullvad and other VPN providers.
> As an example, imagine that you are a moderator on a forum and you suspect that a new face is actually a sockpuppet of a user you banned the day prior. You check the IP logs, and despite using different Mullvad servers, both accounts resolve to the overlapping float ranges 0.4334 - 0.4428 and 0.4358 - 0.4423. This gives you a >99% chance that they are the same person.
This sounds like how I'd design a VPN if I were an intelligence agency.
Why? If I was an intelligence agency and designing a VPN I would simply log all the IPs connecting to my VPN and not rely on statistics on exit nodes to identify the users, even more so because they rely on the users to pick different servers.
Yeah I'm sure one day it will transpire Cloudflare is affliated with intelligence agencies too. The solution to a "sudden DDoS" is to put their website behind Cloudflare. Wonder who can do those sudden attacks?
That’s been my pet theory from day 1, and not because of DDoS. Simply because they are the SSL terminator for most of the internet and can see anything going on in cleartext (and I’ve seen them protecting some shady stuff)
I recall a PRISM slide showing the diagram of Google and the public internet, with a big arrow on GFE saying, quote, “SSL added and removed here! :-)”
If NSA aren’t installed at Cloudflare, I wonder what they are even doing.
Yeah, their origin is a story of absolute incredible luck. Cloudflare came out of nowhere and suddenly massive sites with huge user bases around the world, including places like 4chan, were getting DDoSed. Then they immediately announce that they transitioned to Cloudflare. Hell of a lucky time to make a company that the entire internet suddenly became absolutely dependent on.
The funny thing about that era is you knew they started using Cloudflare because they went from stable with constant uptime to going down and showing a Cloudflare banner randomly all the time for a good year or so. They ran worse with Cloudflare than they did while they were allegedly getting DDoSed. The whole company glows, as the late great HN commenter Terry Davis would've said.
Well there is still the small detail of them not storing any logs.
This is a massive issue in my view, it allows correlation across multiple VPNs exit nodes, but that’s it. It doesn’t allow to identify you automatically. It does significantly lower the bars for identifying you though, but the requirements are still high.
Hopefully they fix this soon.
I can’t believe this type of “let’s make it a hash or something sensitive” still happen, and at mullvad, of all places. Why not randomise it simply?
> It does significantly lower the bars for identifying you though, but the requirements are still high
If you squint a bit, it looks a lot like a "Nobody But US" (NOBUS[1]) scheme. A few more identifying bits could tip the scale for party that has a whole host of other bits on a list of suspects, without being useful to most other people.
Mullvad predates the Snowden leaks by several years and was not mentioned anywhere in them.
Sure, there are other intelligence agencies, but that's the one I'd be the most worried about. Since either they run it, or they would know of it and want to emulate the idea, or know of it and have access to it from the partner agency running it. Or they are not a threat to me.
There's also the issue of no publicly known cases where someone that used Mullvad being deanonymized through the VPN but instead being discovered through some other opsec failure. If an intelligence agency has this capability they have been sitting on it for almost 2 decades without making use of the data. Hard to believe.
> Mullvad predates the Snowden leaks by several years and was not mentioned anywhere in them.
Wow, I didn't realize Mullvad was this old! Then again, maybe they weren't popular enough back then for intelligence agencies to target them? For instance, Mullvad kinda rode WireGuard's popularity wave by being the first(?) VPN provider to implement the protocol. Big ads on billboards came even later. So maybe they only became a target in recent years?
In this particular case I'm quite sure it's not the case. Good arguments in the other comments (why not just log more if that's the case), but I also happen to know a little bit about the workings of Mullvad (I live in Gothenburg where they're from...)
Every now and then there are articles like this one about something that Mullvad may or may not be able to do better, and there are always comments about whether they're an intelligence front.
I don't know the answer, but there are two ways to take it:
1. Submarining to destroy confidence in an actually trustworthy, decent VPN company
2. They're an intelligence front.
For me, Mullvad have the appearance of the greatest likelihood of being legit since they're not aggressively pushing their product with lies and fear mongering. That gels with my vibe. If they're an intelligence front, well, most VPNs probably are as well, so I'm no worse off.
Luckily I'm not doing anything that would get me in the kind of trouble for which multi-jurisdictional cooperation is worthwhile.
> As an example, imagine that you are a moderator on a forum and you suspect that a new face is actually a sockpuppet of a user you banned the day prior. You check the IP logs, and despite using different Mullvad servers, both accounts resolve to the overlapping float ranges 0.4334 - 0.4428 and 0.4358 - 0.4423. This gives you a >99% chance that they are the same person.
I don't see how the author is arriving at this ">99% chance" purely from the numbers provided in the article. Assuming the first (banned IP) seed and the second seed are both in the range 0.4423 - 0.4358 (a stronger assumption than is justified by the example), all this tells us is that the first and second IP addresses both have seeds in a range that would contain 0.4423 - 0.4358 = 0.65% of all Mullvad users, which 0.0065 * 100,000 = 650 users. We've eliminated >99% of users as "suspects", but we haven't actually gotten >99% accuracy in identifying an individual across multiple exit IPs.
In more Bayesian thinking, the overlap in potential seeds is great evidence to think these IP addresses represent one and the same person (or Mullvad VPN account at least), but as far as I can tell, that's not what the author is saying.
I think you are (informally and correctly) doing Bayes theorem here. The prior is combined with the conditional to give the posterior estimate; the conditional is not itself the estimate.
The purpose of a VPN does not include anonymizing users with respect to the sites they visit,so it shouldn't be too surprising that Mullvad doesn't enforce unique exit IPs. Users who want anonymity should use networks like Tor.
If I'm on a public VPN, I don't want anyone to know who is making the request, including the terminating IP.
Think about it. By your logic, VPNs shouldn't be used for torrents because VPNs shouldn't anonymize you to the terminating IP. Whereas they work gangbusters for that.
If you are talking about private VPNs.. Mullvad isn't one.
I think you are misreading his comment. He is saying that on a VPN it is standard behavior that if you visit site A and site B they will both see you connecting from the same IP and can infer you are potentially the same person.
Great find by the author and I have no trouble believing this is an oversight by Mullvad. Kind of shocking that something this simple slips by them but I could see myself missing it.
Putting aside the IP correlation across multiple servers, at first I wondered why even keep the user IP stable on one server. But I think it makes sense because as the author states other VPNs usually have only one IP per server so they are essentially simulating that. The advantages for the user are, if they find a server that works for accessing some service they can connect to that server again and it will work again because they get the same IP.
The IP correlation across multiple servers they should fix though with something like rand.seed(user_pub_key + server_id)
> The advantages for the user are, if they find a server that works for accessing some service they can connect to that server again and it will work again because they get the same IP.
On the flip side, if they’re getting banned by a service because of a noisy neighbor on the same IP, they’d have no way to work around that, no?
> Surprisingly, the exit IP you are given is not randomized each time you connect to the server, but deterministically picked based on your WireGuard key, which rotates every 1 to 30 days (unless you use a third-party client, in which case it never rotates).
I'm a little confused on this... what is stopping third parties from doing key rotations like the main app clients if it is detailed in the repo how to do it?
Third party clients include e.g. the WireGuard driver in the Linux kernel. It's definitely not the network driver's job to mitigate an attack against one specific commercial service.
It seems surprising that people would expect a VPN to be comparable to Tor.
It does seem ridiculous once you spell it out like that, and then you have to realize that it’s plausible to de-anonymize even Tor users by controlling exit nodes.
It is privacy with respect to your ISP. A lot of ISPs are pretty shitty. Some will rat out their own customers to copyright mongrels and threaten to disconnect you - which is important when there's a local monopoly.
Things you connect to or log in to are clearly going to be able to ID you at least with in the context of the login that you use regardless of what the VPN does.
I'm logged into HN through Mullvad as it happens. I usually leave it on regardless of what I'm doing because what I'm doing isn't my ISP's business even though I'm pretty happy with them.
But what privacy do you think majority of people who not doing something badly illegal expect from VPNs?
Most likely these people just look to hide their torrenting, saying political shit on Twitter from employer and not share their choice of porn with local ISP. Also just adding one more layer between them and occasional scammer who can sometimes infer more broad geodata from their IP leaked from yet another database. Oh and now to avoid "Show your ID" page on the same porn sites.
It works well enough for this goal. Not everyone needs NSA-proof solution.
PS: Obviously more tech savvy people understand importance of hiding traffic on public WiFi, but I doubt average Joe the VPN user will buy VPN for this.
"identifying" is the wrong word here--that's only possible if Mullvad stores a mapping between IP addresses and people, which according to them, a 3rd party audit, and a law enforcement raid they do not. It's also worth saying it's possible to use Mullvad entirely anonymously by mailing them cash, which I do.
Also if the threat model you're addressing w/ VPN usage is anything other than "I don't want my ISP to know what I'm doing" you need to use/do something else.
Given that Mullvad is basically a bulletproof VPN host[1], it would be great if site operators could rely on this property to enact bans. Given that the solution is simple (add a pseudorandom seed), Mullvad will likely push out a fix within a couple days.
I work for IPinfo. Even though we are in the VPN detection business, I will give Mullvad the benefit of the doubt, to be honest. They were one of the three VPN providers we found that did not attempt to submit inaccurate geolocation information to IP geolocation providers like us. I am sure they will fix the issue.
blocked IPs they contain all VPN providers. Often VPN providers seed Geofeeds with wrong data, this is why i use traceroute and ping network to locate their real location.
I have a script that logs IPs for any traffic coming in to my servers on ports that don't accept traffic. I then block those IPs from accessing ports behind which there are services.
If they're checking my locked doors, I don't want them coming in my unlocked doors.
This might be a good idea, but consider banning them for, say, a couple hours at a time. It’s easy to rotate IP, especially if you’re using a residential proxy service, and there’s a good chance you’ll end up blocking real users using the same ISP.
Closed ports are not "locked doors", and open ports are not "unlocked doors"
That is a binary thought process with a lot of assumptions. You might introduce even more attack surface in pursuit of this "security" measure by installing additional software like fail2ban, for example. Close your ports, maybe assign a non-standard port to the popular ones (like SSH) to reduce log spam, and patch your server often. Anything more complicated than that is not worth it, IMO.
You know that people use VPNs for perfectly legitimate reasons, right?
Like when I was travelling, sites would routinely use the language of my IP address location, not the language preference as I set it in my browser. So I would be served a site that I couldn't read. My only option was to use a VPN to spoof my location so that it would serve me a site in a language I understand.
I use aVPN when I’m traveling and want to order food delivery for my 93 year old mother in NY. UberEats and InstaCart will stop me from ordering when logged in my mom’s NY account if I’m in China, Saudi Arabia, India, Vietnam, etc.
It's a game of cat and mouse. The service keeps banning IP ranges, the user keeps reconnecting to different servers and regions. The server can't know exactly who's who, just that a bunch of users are using mullvad, while the user just need to find one server on one IP range that works.
Seems like a good deal to me. I don't care if they know I use mullvad, I care they don't know I'm me, and that's not something mullvad will easily disclose.
I'll go ahead and answer that it can't. It knows I'm mullvad user X, thus deanonimization, "it knows I use mullvad", but it doesn't know my original IP, so "it doesn't know I'm me".
surprising that the mapping may be stable enough to become a user-level signal. and rotating away from deterministic assignment seems like a cheap way to avoid creating an extra fingerprint
>Surprisingly, the exit IP you are given is not randomized each time you connect to the server, but deterministically picked based on your WireGuard key
What's the point of this? This seems more complicated to implement than mapping exit ips at the server level, so surely they must be doing this for a good reason?
It's simpler to implement because it's more stateless, and it's a better user experience.
If you get a new exit IP each time you connect, you need something like a NAT table to look up "key 0xabc exits ip 1.2.3.4", and that grows to be the size of the number of users you have active, and you need to save it forever so that when the NSA asks who used the IP for what duration you can tell them.
With a static mapping derived from the key, you don't need a table like that.
It's also better UX since it means reconnecting your VPN software (say you switch wifi hotspots) doesn't give you a different IP address, so things like SSH sessions can resume, which wouldn't be possible if it were a different public IP each time.
I'd guess that this is to ensure one abusive user doesn't get every other user blocked from a large service (say, Google) for botting over the VPN and constantly rotating IPs.
It's a practical measure, but definitely has a privacy cost though.
It's possible that contributes, but to be honest most VPN users are split "privacy seeking" and "abusive". Though I grant you paid users are probably slightly more circumspect than users of Tor, etc.
It seems more likely this is just about load-balancing use against their available nodes.
My guess is deterministic assignment makes load distribution and debugging easier. But for a privacy product, that convenience probably needs to be reconsidered
I imagine there are a bunch of things on the internet that break if you start trying to connect to them from varying IP addresses. Things like the various CAPTCHA schemes and rate limiting etc, IP reputation etc.
> I imagine there are a bunch of things on the internet that break if you start trying to connect to them from varying IP addresses. Things like the various CAPTCHA schemes and rate limiting etc, IP reputation etc.
Given how much of the world is stuck behind CGNAT now, I would expect any major sites to handle it.
This is why VPNs have always been crap. The pool of IPs are backlisted/tainted, so you will run into various roadblocks and cpatchas, in addition to slow speed. If you are serious about privacy and don't want blocks and blacklists, buy high speed private proxies. Don't use a pooled service.
Reusing the same VPN between multiple identities is a horrible idea regardless. And let's be real. As a forum moderator if you ban a Mullvad user and then a new Mullvad user signs up the next day it is probably the same person. You should be using residential or mobile proxies if you want privacy and to blend in to everyone else.
Let's see, short summary of the article, saying nothing new or important. It's not x it's y. Comment history is exactly this type of comment everywhere.
VPNs are not snake oil. They transfer the trust of your internet activity from a place of low-trust, your ISP, to a place of high-trust, ideally a trustworthy VPN like Mullvad, IVPN, or Proton. Among other benefits. If you don't like your ISP creating a profile of you and selling it to target ads to you, you should use a VPN.
>Should I use a VPN?
Yes, almost certainly. A VPN has many advantages, including:
1. Hiding your traffic from only your Internet Service Provider.
2. Hiding your downloads (such as torrents) from your ISP and anti-piracy organizations.
3. Hiding your IP from third-party websites and services, helping you blend in and preventing IP based tracking.
4. Allowing you to bypass geo-restrictions on certain content.
How is private company (VPN) is more trustworthy than an other private company (ISP) and how do you expect them to protect your identity in face of determined state actors that are afer you?
What power is in $2.99/month that it offers so much security?
Why is that at least 40% of sponsorship to YouTube Creators seem to be from VPN industry?
> 4. Allowing you to bypass geo-restrictions on certain content.
In theory, but as someone who uses Mullvad in the UK on a day-to-day basis on my personal laptops (not my phone) - I'm using it now, I'm afraid there's quite an additional downside I've found, in that because Mullvad's (at least UK, but also French and Dutch ones I've tried) exit IPs are known, many companies (Cloudflare, Akamai) at the very least know about them, and several sites block access when using Mullvad, returning 403s.
Santander bank for example, I can't always (sometimes I can) connect to when using Mullvad, and sometimes have to turn it off, as I get 403 responses from the bank otherwise (using Firefox).
Sometimes using IPv6 in the Mullvad settings gets around this, but more and more recently I've found it doesn't, so there sites where I'm having to stop using Mullvad to actually access sites.
(I'm still a happy customer, and 1 to 3 are still true and why I use it otherwise).
Making your traffic cross jurisdictional boundaries also adds a level of difficulty for tracking usage.
Local law enforcement can tap a local ISP for their records, but it would take a scale more effort to then tap a non-local service provider for their records. Each additional level of difficulty adds a cost, and at some point those costs aren't worth the potential results.
(assuming that the VPN provider doesn't just roll over due to an email inquiry, or isn't a front for very cooperative law enforcement).
Marcus Hitchens (security researcher who blackholed the WannaCry ransomware domain) made a post on LinkedIn today comparing VPNs to snake oil. With regard to the way they're advertised in internet ads, they are. VPNs will not protect ordinary users from ad tracking or commercial data mining. They're marketed as a privacy tool when their privacy value is very limited.
> place of low-trust, your ISP, to a place of high-trust, ideally a trustworthy VPN like Mullvad
This is highly subjective statement.
Almost all commercial VPN services farm and sell your data. Just by that, my ISP is definitely high trust point while any commercial VPN is a low trust.
Mullvad is a tiny world-famous ISP in Sweden that has zero KYC and explicit zero-log policy, specifically designed that way to enable mild abuses, that also accept PayPal, credit cards, and today I learned, cash in an anonymous envelope for payments. That doesn't scream US three-letter organization at all.
I do all my illegal shit over Mullvad and I've only been raided once.
(yes, I've been raided)
(I started using Mullvad after - because of - that)
(I don't do illegal shit, I just like some obfuscation of my trail because I enjoy fiddling with this stuff - which may have been why I ended up a raid target in the first place)
I was just talking to a friend who believes that the feds poison privacy communities by spewing nonsense like this. I don't think wg0 is a fed, and my friend didn't have any proof for his claim. My feeling is that it is probably people acting like regular humans. They hear things, they have opinions and they don't provide proof or adhere to community norms. Eternal september or something. Regardless of if it's federal agents disrupting the discussion or human nature, the response should be the same—push back with proof, and demand proof and avoiding logical fallacies.
>Also. This is how they ruined any meaningful talks about privacy
There is so much noise
"Use braive. Don't use braive. Use vpn. Don't use vpn"
Then the debate spreads to all other aspects
password managers, emails and etc
If people using some tool made my job harder id be vocally against it during off hours. But lets be real any powerful group interested in tracking people would just be working with or running vpn companies. Or perhaps providing free vpn. Either way I think its all moot as for tracking you have to question who you do and do not want to be tracked by and for other purposes vpn works just fine
VPNs as marketed to "normies" is absolutely snake oil. It won't improve anyone's "privacy" in any meaningful way to simply proxy all their regular traffic through a VPN.
VPNs are a technical tool for technical people. You need to know exactly why you need it in order for it to be useful.
The most generous way of reading that would be the fact that every YouTube pushing for a VPN as an essential tool just to use the internet outside of your house without getting hacked is a big exaggeration or fear mongering but there's good reasons for using a VPN for a lot of reasons and it's not snake oil.
I work at Mullvad. (co-CEO, co-founder)
Some aspects of the described behavior are as we intended and some are not. The cause is not exactly as described in the blog post. As for mitigation, we are already testing a patch of the unintended behavior on a subset of our infrastructure. If any of you try to reproduce the blog post's findings you may get confusing results throughout the day.
We will also re-evaluate whether the intended behaviors are acceptable or not. Some of this is a trade-off between multiple aspects of privacy, and multiple aspects of user experience.
Please note that this is my current understanding, which may change. I was only made aware of this an hour ago, and most of that time was spent talking with Ops, considering what to do immediately, and writing this post.
Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away.
You really do provide a reassuring, good service -- thank you.
It's also worth stating that the client (including the cli client -- which, with a bit of work, you can get running in most situations where you'd use native wireguard) by default has a key rotation interval of I think 72 hours.
`mullvad tunnel get` will show it and `mullvad tunnel set rotation-interval <hours>` will change it. This is the preferred mitigation method of the post.
I personally don't mind having a pseudo-static IP (some other suppliers offer a static IPv4 as a feature!) as I wish to prevent network-level snooping from my ISP and governments. It's also worth stating that I think having a smaller IP space is an advantage for a privacy VPN: there are more potential users acting behind any given externally visible IP. Combined with technologies like DAITA (which effectively adds chaff to the tunnel) and multi-hop entrances and I personally think that this service really does plausibly make harder the life of those who snoop netflows all day.
I just want to say I absolutely love Mullvad! You guys did a fantastic job at designing a genuinely good and trustworthy (as much as possible) VPN vendor. You communicating here is just another data point towards this.
Can we have an Open Suse client.
Sorta odd you don't support one of Europe's most popular distros.
It already has official packaging for Tumbleweed, see https://github.com/mullvad/mullvadvpn-app/issues/2242 for the upstream issue. Leap can use the normal Linux application, you will just have to provide the dependencies yourself.
1 reply →
> Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings
https://mullvad.net/en/help/how-report-bug-or-vulnerability / https://archive.vn/BeHhr
Not having a bug bounty or dedicated email address does not make it OK to go public immediately
9 replies →
To support? Oof.
4 replies →
I'm a long-time Mullvad user. I will continue to buy and use Mullvad VPN services (with my credit card that has my name on it) so long as it is legal to do so in my country.
VPNs are not 100% anonymous. They are not meant to be. Instead, they are meant to provide some level of privacy to law-abiding adults.
Most people would be embarrassed if their co-workers and neighbors knew the intimate personal details of their lives. Things they like, things the buy, things they do, etc. So, most people should use a VPN to protect their privacy.
By definition, 'most people' don't want or expect 100% anonymity online. They just want a bit of privacy in their personal life and their relationships. That's it.
VPNs don't protect (and are not intended to protect) criminals who want 100% anonymity from governments while committing online crimes. This is an important distinction. 'Most people' are not criminals and do not have this unrealistic expectation from Mullvad and other VPN providers.
> As an example, imagine that you are a moderator on a forum and you suspect that a new face is actually a sockpuppet of a user you banned the day prior. You check the IP logs, and despite using different Mullvad servers, both accounts resolve to the overlapping float ranges 0.4334 - 0.4428 and 0.4358 - 0.4423. This gives you a >99% chance that they are the same person.
This sounds like how I'd design a VPN if I were an intelligence agency.
Why? If I was an intelligence agency and designing a VPN I would simply log all the IPs connecting to my VPN and not rely on statistics on exit nodes to identify the users, even more so because they rely on the users to pick different servers.
How would you claim it's a no log VPN?
15 replies →
Yeah I'm sure one day it will transpire Cloudflare is affliated with intelligence agencies too. The solution to a "sudden DDoS" is to put their website behind Cloudflare. Wonder who can do those sudden attacks?
That’s been my pet theory from day 1, and not because of DDoS. Simply because they are the SSL terminator for most of the internet and can see anything going on in cleartext (and I’ve seen them protecting some shady stuff)
I recall a PRISM slide showing the diagram of Google and the public internet, with a big arrow on GFE saying, quote, “SSL added and removed here! :-)”
If NSA aren’t installed at Cloudflare, I wonder what they are even doing.
22 replies →
I don’t see how they couldn’t be. Either on purpose, secretly my coercion, or secretly without their own knowledge. It’s so valuable
Yeah, their origin is a story of absolute incredible luck. Cloudflare came out of nowhere and suddenly massive sites with huge user bases around the world, including places like 4chan, were getting DDoSed. Then they immediately announce that they transitioned to Cloudflare. Hell of a lucky time to make a company that the entire internet suddenly became absolutely dependent on.
The funny thing about that era is you knew they started using Cloudflare because they went from stable with constant uptime to going down and showing a Cloudflare banner randomly all the time for a good year or so. They ran worse with Cloudflare than they did while they were allegedly getting DDoSed. The whole company glows, as the late great HN commenter Terry Davis would've said.
3 replies →
> Wonder who can do those sudden attacks?
Anyone with a few crypto currencies in their wallet that can click a button on any of the booter services with botnets for hire.
3 replies →
Well there is still the small detail of them not storing any logs.
This is a massive issue in my view, it allows correlation across multiple VPNs exit nodes, but that’s it. It doesn’t allow to identify you automatically. It does significantly lower the bars for identifying you though, but the requirements are still high.
Hopefully they fix this soon.
I can’t believe this type of “let’s make it a hash or something sensitive” still happen, and at mullvad, of all places. Why not randomise it simply?
> It does significantly lower the bars for identifying you though, but the requirements are still high
If you squint a bit, it looks a lot like a "Nobody But US" (NOBUS[1]) scheme. A few more identifying bits could tip the scale for party that has a whole host of other bits on a list of suspects, without being useful to most other people.
1. https://en.wikipedia.org/wiki/NOBUS
9 replies →
Mullvad predates the Snowden leaks by several years and was not mentioned anywhere in them.
Sure, there are other intelligence agencies, but that's the one I'd be the most worried about. Since either they run it, or they would know of it and want to emulate the idea, or know of it and have access to it from the partner agency running it. Or they are not a threat to me.
There's also the issue of no publicly known cases where someone that used Mullvad being deanonymized through the VPN but instead being discovered through some other opsec failure. If an intelligence agency has this capability they have been sitting on it for almost 2 decades without making use of the data. Hard to believe.
> Mullvad predates the Snowden leaks by several years and was not mentioned anywhere in them.
Wow, I didn't realize Mullvad was this old! Then again, maybe they weren't popular enough back then for intelligence agencies to target them? For instance, Mullvad kinda rode WireGuard's popularity wave by being the first(?) VPN provider to implement the protocol. Big ads on billboards came even later. So maybe they only became a target in recent years?
In this particular case I'm quite sure it's not the case. Good arguments in the other comments (why not just log more if that's the case), but I also happen to know a little bit about the workings of Mullvad (I live in Gothenburg where they're from...)
> This sounds like how I'd design a VPN if I were an intelligence agency.
So does your comment...
> how I'd design a VPN if I were an intelligence agency
I think its safe to assume that intelligence agencies have other options available to them, such as country-wide timing attacks.
Makes you wonder...
Every now and then there are articles like this one about something that Mullvad may or may not be able to do better, and there are always comments about whether they're an intelligence front.
I don't know the answer, but there are two ways to take it:
1. Submarining to destroy confidence in an actually trustworthy, decent VPN company
2. They're an intelligence front.
For me, Mullvad have the appearance of the greatest likelihood of being legit since they're not aggressively pushing their product with lies and fear mongering. That gels with my vibe. If they're an intelligence front, well, most VPNs probably are as well, so I'm no worse off.
Luckily I'm not doing anything that would get me in the kind of trouble for which multi-jurisdictional cooperation is worthwhile.
3 replies →
> As an example, imagine that you are a moderator on a forum and you suspect that a new face is actually a sockpuppet of a user you banned the day prior. You check the IP logs, and despite using different Mullvad servers, both accounts resolve to the overlapping float ranges 0.4334 - 0.4428 and 0.4358 - 0.4423. This gives you a >99% chance that they are the same person.
I don't see how the author is arriving at this ">99% chance" purely from the numbers provided in the article. Assuming the first (banned IP) seed and the second seed are both in the range 0.4423 - 0.4358 (a stronger assumption than is justified by the example), all this tells us is that the first and second IP addresses both have seeds in a range that would contain 0.4423 - 0.4358 = 0.65% of all Mullvad users, which 0.0065 * 100,000 = 650 users. We've eliminated >99% of users as "suspects", but we haven't actually gotten >99% accuracy in identifying an individual across multiple exit IPs.
In more Bayesian thinking, the overlap in potential seeds is great evidence to think these IP addresses represent one and the same person (or Mullvad VPN account at least), but as far as I can tell, that's not what the author is saying.
Say your forum is a big one and has 1000 active users, with 1 joining every day. Most will be a lot smaller/less active.
What are the chances that someone uses this vpn, joins your forum the day after someone was banned, and has an ip in a similar range?
For most small websites this would be strong evidence.
I think you are (informally and correctly) doing Bayes theorem here. The prior is combined with the conditional to give the posterior estimate; the conditional is not itself the estimate.
The purpose of a VPN does not include anonymizing users with respect to the sites they visit,so it shouldn't be too surprising that Mullvad doesn't enforce unique exit IPs. Users who want anonymity should use networks like Tor.
Why not? Why can’t it be the purpose of a given VPN service?
If you use the VPN for the Web, browser fingerprinting is a major threat outside of specialized scenarios
1 reply →
That is exactly the point of public VPNs..
If I'm on a public VPN, I don't want anyone to know who is making the request, including the terminating IP.
Think about it. By your logic, VPNs shouldn't be used for torrents because VPNs shouldn't anonymize you to the terminating IP. Whereas they work gangbusters for that.
If you are talking about private VPNs.. Mullvad isn't one.
Public VPNs only protect you from your ISP
1 reply →
I think you are misreading his comment. He is saying that on a VPN it is standard behavior that if you visit site A and site B they will both see you connecting from the same IP and can infer you are potentially the same person.
3 replies →
Isn't Tor a us government project that has been shown to be deanonymizable?
Sort of. There are a bunch of timing attacks bug in general it still works fairly well.
1 reply →
It has been successfully deanonymized, and resistance to NSA-level capabilities is explicitly not a stated goal.
2 replies →
and so is ARPANET
Missing from the story: did they reach out to Mullvad? Would have been interesting to see how their security team responded.
As far as I can tell they did not, and I've asked both our operations and support teams. I will update this post if I am mistaken.
Edit: In hindsight I regret making this comment. It was unnecessary, but removing it now would look weird.
Seems fine. You didn’t exactly demand a 90 day embargo or something.
Great find by the author and I have no trouble believing this is an oversight by Mullvad. Kind of shocking that something this simple slips by them but I could see myself missing it.
Putting aside the IP correlation across multiple servers, at first I wondered why even keep the user IP stable on one server. But I think it makes sense because as the author states other VPNs usually have only one IP per server so they are essentially simulating that. The advantages for the user are, if they find a server that works for accessing some service they can connect to that server again and it will work again because they get the same IP.
The IP correlation across multiple servers they should fix though with something like rand.seed(user_pub_key + server_id)
> The advantages for the user are, if they find a server that works for accessing some service they can connect to that server again and it will work again because they get the same IP.
On the flip side, if they’re getting banned by a service because of a noisy neighbor on the same IP, they’d have no way to work around that, no?
You mean if the neighbor somehow burned every VPN location?
1 reply →
> Surprisingly, the exit IP you are given is not randomized each time you connect to the server, but deterministically picked based on your WireGuard key, which rotates every 1 to 30 days (unless you use a third-party client, in which case it never rotates).
I'm a little confused on this... what is stopping third parties from doing key rotations like the main app clients if it is detailed in the repo how to do it?
Third party clients include e.g. the WireGuard driver in the Linux kernel. It's definitely not the network driver's job to mitigate an attack against one specific commercial service.
> what is stopping third parties from doing key rotations
Knowing to do so, primarily.
It seems surprising that people would expect a VPN to be comparable to Tor.
It does seem ridiculous once you spell it out like that, and then you have to realize that it’s plausible to de-anonymize even Tor users by controlling exit nodes.
Most of the big consumer VPNs include "privacy" with an implication of anonymity in their marketing, so it shouldn't really be surprising
It is privacy with respect to your ISP. A lot of ISPs are pretty shitty. Some will rat out their own customers to copyright mongrels and threaten to disconnect you - which is important when there's a local monopoly.
Things you connect to or log in to are clearly going to be able to ID you at least with in the context of the login that you use regardless of what the VPN does.
I'm logged into HN through Mullvad as it happens. I usually leave it on regardless of what I'm doing because what I'm doing isn't my ISP's business even though I'm pretty happy with them.
But what privacy do you think majority of people who not doing something badly illegal expect from VPNs?
Most likely these people just look to hide their torrenting, saying political shit on Twitter from employer and not share their choice of porn with local ISP. Also just adding one more layer between them and occasional scammer who can sometimes infer more broad geodata from their IP leaked from yet another database. Oh and now to avoid "Show your ID" page on the same porn sites.
It works well enough for this goal. Not everyone needs NSA-proof solution.
PS: Obviously more tech savvy people understand importance of hiding traffic on public WiFi, but I doubt average Joe the VPN user will buy VPN for this.
11 replies →
"Not knowing who a user is" privacy may still be useful even if you don't have, "not knowing two users are the same user" privacy.
"identifying" is the wrong word here--that's only possible if Mullvad stores a mapping between IP addresses and people, which according to them, a 3rd party audit, and a law enforcement raid they do not. It's also worth saying it's possible to use Mullvad entirely anonymously by mailing them cash, which I do.
Also if the threat model you're addressing w/ VPN usage is anything other than "I don't want my ISP to know what I'm doing" you need to use/do something else.
We keep adding layers of encryption and the metadata keeps snitching on us anyway.
Given that Mullvad is basically a bulletproof VPN host[1], it would be great if site operators could rely on this property to enact bans. Given that the solution is simple (add a pseudorandom seed), Mullvad will likely push out a fix within a couple days.
1. It's the preferred VPN of TeamPCP.
Source? Been googling for this but I don’t see any relevant info
oopsie, has someone burned their proprietary intel for internet points?
I work for IPinfo. Even though we are in the VPN detection business, I will give Mullvad the benefit of the doubt, to be honest. They were one of the three VPN providers we found that did not attempt to submit inaccurate geolocation information to IP geolocation providers like us. I am sure they will fix the issue.
Who else ?
I maintain a list of
"23034 IPs to blocklist.txt"
blocked IPs they contain all VPN providers. Often VPN providers seed Geofeeds with wrong data, this is why i use traceroute and ping network to locate their real location.
I have a script that logs IPs for any traffic coming in to my servers on ports that don't accept traffic. I then block those IPs from accessing ports behind which there are services.
If they're checking my locked doors, I don't want them coming in my unlocked doors.
This might be a good idea, but consider banning them for, say, a couple hours at a time. It’s easy to rotate IP, especially if you’re using a residential proxy service, and there’s a good chance you’ll end up blocking real users using the same ISP.
18 replies →
That’s nice, I need to implement this.
Closed ports are not "locked doors", and open ports are not "unlocked doors"
That is a binary thought process with a lot of assumptions. You might introduce even more attack surface in pursuit of this "security" measure by installing additional software like fail2ban, for example. Close your ports, maybe assign a non-standard port to the popular ones (like SSH) to reduce log spam, and patch your server often. Anything more complicated than that is not worth it, IMO.
You know that people use VPNs for perfectly legitimate reasons, right?
Like when I was travelling, sites would routinely use the language of my IP address location, not the language preference as I set it in my browser. So I would be served a site that I couldn't read. My only option was to use a VPN to spoof my location so that it would serve me a site in a language I understand.
By the way, if you’re a webmaster doing this, look at the Accept-Language header instead: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
I use aVPN when I’m traveling and want to order food delivery for my 93 year old mother in NY. UberEats and InstaCart will stop me from ordering when logged in my mom’s NY account if I’m in China, Saudi Arabia, India, Vietnam, etc.
1 reply →
It's a game of cat and mouse. The service keeps banning IP ranges, the user keeps reconnecting to different servers and regions. The server can't know exactly who's who, just that a bunch of users are using mullvad, while the user just need to find one server on one IP range that works.
Seems like a good deal to me. I don't care if they know I use mullvad, I care they don't know I'm me, and that's not something mullvad will easily disclose.
> I don't care if they know I use mullvad, I care they don't know I'm me
That's exactly what the article is about, a side channel information leak that de-anonymises users, did you read it?
Can it get my IP?
I'll go ahead and answer that it can't. It knows I'm mullvad user X, thus deanonimization, "it knows I use mullvad", but it doesn't know my original IP, so "it doesn't know I'm me".
1 reply →
surprising that the mapping may be stable enough to become a user-level signal. and rotating away from deterministic assignment seems like a cheap way to avoid creating an extra fingerprint
>Surprisingly, the exit IP you are given is not randomized each time you connect to the server, but deterministically picked based on your WireGuard key
What's the point of this? This seems more complicated to implement than mapping exit ips at the server level, so surely they must be doing this for a good reason?
It's simpler to implement because it's more stateless, and it's a better user experience.
If you get a new exit IP each time you connect, you need something like a NAT table to look up "key 0xabc exits ip 1.2.3.4", and that grows to be the size of the number of users you have active, and you need to save it forever so that when the NSA asks who used the IP for what duration you can tell them.
With a static mapping derived from the key, you don't need a table like that.
It's also better UX since it means reconnecting your VPN software (say you switch wifi hotspots) doesn't give you a different IP address, so things like SSH sessions can resume, which wouldn't be possible if it were a different public IP each time.
I'd guess that this is to ensure one abusive user doesn't get every other user blocked from a large service (say, Google) for botting over the VPN and constantly rotating IPs.
It's a practical measure, but definitely has a privacy cost though.
It's possible that contributes, but to be honest most VPN users are split "privacy seeking" and "abusive". Though I grant you paid users are probably slightly more circumspect than users of Tor, etc.
It seems more likely this is just about load-balancing use against their available nodes.
My guess is deterministic assignment makes load distribution and debugging easier. But for a privacy product, that convenience probably needs to be reconsidered
I imagine there are a bunch of things on the internet that break if you start trying to connect to them from varying IP addresses. Things like the various CAPTCHA schemes and rate limiting etc, IP reputation etc.
> I imagine there are a bunch of things on the internet that break if you start trying to connect to them from varying IP addresses. Things like the various CAPTCHA schemes and rate limiting etc, IP reputation etc.
Given how much of the world is stuck behind CGNAT now, I would expect any major sites to handle it.
1 reply →
This is why VPNs have always been crap. The pool of IPs are backlisted/tainted, so you will run into various roadblocks and cpatchas, in addition to slow speed. If you are serious about privacy and don't want blocks and blacklists, buy high speed private proxies. Don't use a pooled service.
A VPN by any other name would smell as sweet.
Reusing the same VPN between multiple identities is a horrible idea regardless. And let's be real. As a forum moderator if you ban a Mullvad user and then a new Mullvad user signs up the next day it is probably the same person. You should be using residential or mobile proxies if you want privacy and to blend in to everyone else.
I just use it to watch iPlayer outside of the UK lol
[flagged]
[dead]
[dead]
[flagged]
Let's see, short summary of the article, saying nothing new or important. It's not x it's y. Comment history is exactly this type of comment everywhere.
This is an AI comment from an AI account.
Doesn't matter much as long as it is a pseudonymous identity
It’s also not that difficult to fix, so I expect a fix to roll out soon enough.
VPNs are snake oil. Exit IPs are a public information.
VPNs are not snake oil. They transfer the trust of your internet activity from a place of low-trust, your ISP, to a place of high-trust, ideally a trustworthy VPN like Mullvad, IVPN, or Proton. Among other benefits. If you don't like your ISP creating a profile of you and selling it to target ads to you, you should use a VPN.
>Should I use a VPN?
Yes, almost certainly. A VPN has many advantages, including:
1. Hiding your traffic from only your Internet Service Provider.
2. Hiding your downloads (such as torrents) from your ISP and anti-piracy organizations.
3. Hiding your IP from third-party websites and services, helping you blend in and preventing IP based tracking.
4. Allowing you to bypass geo-restrictions on certain content.
(https://www.privacyguides.org/en/basics/vpn-overview/)
How is private company (VPN) is more trustworthy than an other private company (ISP) and how do you expect them to protect your identity in face of determined state actors that are afer you?
What power is in $2.99/month that it offers so much security?
Why is that at least 40% of sponsorship to YouTube Creators seem to be from VPN industry?
What is that they know and we don't know?
6 replies →
> 4. Allowing you to bypass geo-restrictions on certain content.
In theory, but as someone who uses Mullvad in the UK on a day-to-day basis on my personal laptops (not my phone) - I'm using it now, I'm afraid there's quite an additional downside I've found, in that because Mullvad's (at least UK, but also French and Dutch ones I've tried) exit IPs are known, many companies (Cloudflare, Akamai) at the very least know about them, and several sites block access when using Mullvad, returning 403s.
Santander bank for example, I can't always (sometimes I can) connect to when using Mullvad, and sometimes have to turn it off, as I get 403 responses from the bank otherwise (using Firefox).
Sometimes using IPv6 in the Mullvad settings gets around this, but more and more recently I've found it doesn't, so there sites where I'm having to stop using Mullvad to actually access sites.
(I'm still a happy customer, and 1 to 3 are still true and why I use it otherwise).
1 reply →
Making your traffic cross jurisdictional boundaries also adds a level of difficulty for tracking usage.
Local law enforcement can tap a local ISP for their records, but it would take a scale more effort to then tap a non-local service provider for their records. Each additional level of difficulty adds a cost, and at some point those costs aren't worth the potential results.
(assuming that the VPN provider doesn't just roll over due to an email inquiry, or isn't a front for very cooperative law enforcement).
6 replies →
Unfortunately, the largest and most well-marketed VPNs are, in fact, less trustworthy than your average ISP.
3 replies →
Marcus Hitchens (security researcher who blackholed the WannaCry ransomware domain) made a post on LinkedIn today comparing VPNs to snake oil. With regard to the way they're advertised in internet ads, they are. VPNs will not protect ordinary users from ad tracking or commercial data mining. They're marketed as a privacy tool when their privacy value is very limited.
VPNs are useful for the reasons you mentioned.
> place of low-trust, your ISP, to a place of high-trust, ideally a trustworthy VPN like Mullvad
This is highly subjective statement.
Almost all commercial VPN services farm and sell your data. Just by that, my ISP is definitely high trust point while any commercial VPN is a low trust.
25 replies →
Mullvad is a tiny world-famous ISP in Sweden that has zero KYC and explicit zero-log policy, specifically designed that way to enable mild abuses, that also accept PayPal, credit cards, and today I learned, cash in an anonymous envelope for payments. That doesn't scream US three-letter organization at all.
I do all my illegal shit over Mullvad and I've only been raided once.
(yes, I've been raided)
(I started using Mullvad after - because of - that)
(I don't do illegal shit, I just like some obfuscation of my trail because I enjoy fiddling with this stuff - which may have been why I ended up a raid target in the first place)
> That doesn't scream US three-letter organization at all.
They have their own tools + tor, they do not need mullvad.
Interesting handle to make that comment. I'm assuming you mean commercial VPN providers, and not wireguard (or other such VPN implementations).
I was just talking to a friend who believes that the feds poison privacy communities by spewing nonsense like this. I don't think wg0 is a fed, and my friend didn't have any proof for his claim. My feeling is that it is probably people acting like regular humans. They hear things, they have opinions and they don't provide proof or adhere to community norms. Eternal september or something. Regardless of if it's federal agents disrupting the discussion or human nature, the response should be the same—push back with proof, and demand proof and avoiding logical fallacies.
>Also. This is how they ruined any meaningful talks about privacy
There is so much noise
"Use braive. Don't use braive. Use vpn. Don't use vpn"
Then the debate spreads to all other aspects password managers, emails and etc
If people using some tool made my job harder id be vocally against it during off hours. But lets be real any powerful group interested in tracking people would just be working with or running vpn companies. Or perhaps providing free vpn. Either way I think its all moot as for tracking you have to question who you do and do not want to be tracked by and for other purposes vpn works just fine
VPNs as marketed to "normies" is absolutely snake oil. It won't improve anyone's "privacy" in any meaningful way to simply proxy all their regular traffic through a VPN.
VPNs are a technical tool for technical people. You need to know exactly why you need it in order for it to be useful.
> VPNs are snake oil
The most generous way of reading that would be the fact that every YouTube pushing for a VPN as an essential tool just to use the internet outside of your house without getting hacked is a big exaggeration or fear mongering but there's good reasons for using a VPN for a lot of reasons and it's not snake oil.
> Exit IPs are a public information.
Yes, obviously.
> VPNs are snake oil
Huh?