Comment by TZubiri
21 hours ago
Hi, I’m helping a hospital in Buenos Aires that has lost their IPv4 due to their ISP moving to CGNAT, putting their on prem imaging servers offline for 7 days. (it’s ok for us though, we have a backup failover isp)
Might have to do something horrible like:
- setup an ipv4 tunnel, which will require hacking a VPN to allow TCP listening, and routing connections to some external proxy with a dedicated static ipv4 address
- or do some modem router level nat port forwarding, and use a ddns to avoid ip changes breaking stuff. but that won’t work probably since public ip is shared
will investigate the isp to see if they are truly short in ipv4 or if they are just moving clients off them in order to sell them to higher bidders and plans, which is likely since I’ve been upsold into optic fiber, static ip surcharge, sd-wan, dedicated connection so far
if you have any advice or RIR contacts (or an ipv4 block to donate, invest or sell), I’d be thankful, my email is in my profile
The first thing you need to ask is: Why do you need IPv4?
I don't know much about your imaging servers: Do they accept incoming https connections, or is it some other kind of TCP or UDP protocol? Assuming it's https, why do you need to stick with IPv4?
> Hi, I’m helping a hospital in Buenos Aires that has lost their IPv4 due to their ISP moving to CGNAT, putting their on prem imaging servers offline for 7 days. (it’s ok for us though, we have a backup failover isp)
Two more things:
1: If you require IPv4, that needs to be part of the contract with your ISP. If they just arbitrarily turned you off, it means that you didn't contract for the services you thought you had.
2: IPv6 is supposed to solve a lot of these problems... So again, why can't you move to IPv6?
> I don't know much about your imaging servers: Do they accept incoming https connections, or is it some other kind of TCP or UDP protocol?
Yes
>why do you need to stick with IPv4?
To support devices and routes that do not have ipv6 support?
>1: If you require IPv4, that needs to be part of the contract with your ISP. If they just arbitrarily turned you off, it means that you didn't contract for the services you thought you had.
That is correct in principle. Although, in the absence of a contract, if a service has been provided for a long time in one way, I'd say there's an argument for breach of contract when that way changes for the worst.
Another argument for breach of contract, which can work jointly with the previous, are industry standards, as far as they define the terms that are implied by common words like "internet connection".
Finally, there's the specifications themselves, since the service is billed as "internet", one can refer to the Internet Protocol Specification, although if they do offer IPv6, that might be reasonably considered a substitute, it is a material change that was not agreed to originally (this being a pre IPv6 contract). That said, an IPv4 with a local address might technically still be defined as internet, what is broken in this case is TCP, not the IP, although by lacking a public IP address, TCP is broken via an IP protocol failure, so there's an argument that something is being sold as Internet but it isn't really internet, there's no written contract necessary, the term used to describe the product creates the contract in the same way that selling a carton labeled Milk would create a contract based on milk coming from mammaries of a domestic mammal, and not pressed Soy Juice with artificial coloring.
>2: IPv6 is supposed to solve a lot of these problems... So again, why can't you move to IPv6?
I'm open to proposals to moving IPv6, I'm not content with having that be forced and removing it from people and institutions, if there's a shortage, then that means no new IP addresses can be issued, why are we normalizing the removal of IPv4 addresses? Are they being removed from lower priority users in order to serve high priority usecases? If so is a hospital a low priority user?
The main issue with IPv6 seems to be IPv6 proponents that try to kill IPv4, no one will migrate if you force them to, if you propose, we will consider.
> The main issue with IPv6 seems to be IPv6 proponents that try to kill IPv4, no one will migrate if you force them to, if you propose, we will consider.
My experience is more the opposite. I keep seeing people say things like "I don't do v6 at home because I don't need it". These people are outright telling us that they're not going to start using v6 unless we make them need it.
Note that I didn't say "migrate to" v6. They're not even willing to do the first step of that ("start using v6"), even with no pressure to do the last step ("stop using v4").
I read an anecdote from Facebook that they had huge trouble getting their developers to stop writing v4-only code. Eventually they ended up removing v4 from their developer machines, who all of a sudden were able to write v6-capable code -- and it didn't take long for them to start asking when they could drop support for v4.
People will moan and moan and moan about v6 until you make them use it, at which point they discover that it was never as bad as they imagined it being.
1 reply →
The professional services that I work with give plenty of notice (years) before turning off services. Were you using an internet connection that's intended for hosting servers? (As in you were paying for a dedicated static IP.) If so, then you have a case.
Otherwise, your arguments about what "internet" is are flawed. IPv6 has been a thing for about 30 years, with a well-publicized problem with IPv4 address space and migration plan. I suspect that it's (IPv6) older than your hospital's contract. Even in the IPv4 world, your IP can change at any time and you can be put behind a NAT. That's why ISPs will sell you a static IP if you need one.
>> why do you need to stick with IPv4?
> To support devices and routes that do not have ipv6 support?
> Are they being removed from lower priority users in order to serve high priority usecases? If so is a hospital a low priority user?
After reading your response, I suspect the real problem is attitude and experience. The hospital should have IT staff that know to purchase a static IPv4 IP. This should be part of the contract with the ISP. If it wasn't, ultimately it's up to the IT staff to approach the ISP and arrange to purchase one.
2 replies →
Have you noted that in the mainline Linux build config file, the option to enable IPv4 functionality is CONFIG_LEGACY_IP= ?
I can tell you that in cicd for production Linux systems today, we have
which in a Linux build config file means "do not include this in the build".
_
What might work well for your organization is ZeroTier which will make you immune to what any ISPs do. The downside is your organization would be dependent on ZeroTier, but ZeroTier is much less likely to screw you over than any ISP.
_
I appreciate your speculation on the principles of contract law. Like you, I believe those principles should be upheld and I believe in morality. If you lived under a monarchy and I were of the monarch, I would cross-examine your ISP in public and solve your problem in minutes. I would also on occasion take someone to the side of the road and chop off their head in front of the entire village, and everyone would know on a visceral level they are all much better off with someone taking out the trash now and then.
But, as many technical workers know, the contract stuff takes months or years to resolve, and technical issues involving IP addresses are best worked around with technical approaches.
_
What I would suggest your network team (you and whoever) do in your shoes is:
- set up NAT64 and DNS64 at the edges of subnets that contain devices that don't do IPv6 properly, because if these devices do not have updates to do IPv6 properly then they are broken in a fundamental way, and a workaround is necessary;
- dictate that starting on December 1st, all new devices and VMs have IPv6 addresses with all userspace services listening on IPv6 addresses or some nftables rule to kludge IPv6:$port -> 127.0.0.1:$port if necessary, with firewall configured to block that IPv6 traffic, then at a later date firewalls not blocking those IPv6 addresses, then at a later date prohibit any new IPv4 addresses being assigned anywhere with special exception to devices tagged "legacy" or "end of life" requiring documentation of such;
- use a combination of IPv6 GUA addresses assigned by whatever ISP and IPv6 ULA addresses chosen by your team -- in IPv6, multiple local gateways and multiple addresses on a network interface issued by those local gateways via RA is relatively a norm rather than an exception (requires userspace tooling in IPv4, DHCP doesn't have this functionality), so leverage this advantage;
- use IPv6 AH (authenticated headers) wherever possible;
- because in an ideal world, ISPs would behave consistently and IPv6 GUA addresses would always be stable, but we aren't living in an ideal world, so be ready to update your internal DNS to drop AAAA records containing IPv6 GUA that an upstream ISP fumbled;
- or, if you have the internal resources, register your own AS and command your own official block of IPv6 addresses;
- use a combination of OpenTOFU (Terraform) and Salt or Ansible, all versioned with Git, be very strict with your team about versioning with Git, to manage everything, and use
and Salt or Ansible equivalents for continuous monitoring;
- move userspace services to Podman where possible.
Personally I like Salt. I know Ansible is more popular out there with more inertia.
_
>> why are we normalizing the removal of IPv4 addresses?
This was normalized a long time ago actually. "The future is already here, it's just not evenly distributed".
You and I are total strangers on different corners of the globe. You may not like what you're about to read, but it's better that you get a hint where the wind is blowing sooner than not.
If my team were consulting with your organization's execs and we caught a hint that the existing internal resources were not already proactively staying current with internet protocol (which is IPv6), we would point out there is urgency to start right away with emphasis that the existing internal team is over two decades behind staying current.
Or, if a detected hint the internal team were at all resistant to IPv6, we would advise the executive team to start developing a plan to flush out the replacements for the internal team ASAP.
Many network engineers at many ISPs would gladly drop all IPv4 traffic if they thought they could get away with it.
IPv6 only seems more complicated if you still need to unlearn IPv4. IPv6 does have a steeper learning curve, but once that learning curve is overcome, IPv6 is actually much simpler than IPv4. More people than not don't like reading RFCs, but professionals do slowly read RFCs on their own time.
7 replies →
You're a hospital. You have an actual contract with the ISP, right? Does it say you get an IPv4 address? If not, ask them for one, and pay up.
You don't need an address block from an RIR. You just need your ISP to give you an address.
You could also update to IPv6.
>You're a hospital.
I work for the hospital, occasionally, they are my client.
> You have an actual contract with the ISP, right?
They do not, it's an implied consumer contract, like when someone buys a hot dog from a stand.
I'd like a written contract too, but when trying to negotiate one, they want a lot more money for them, it depends of the nuances of the terms, but I've talked to sales reps, and it comes down to having a 1Gbps line that works right now vs a 100Mbps line that's more expensive, with the only difference being that the contract is now written and there's a symbolic SLA with some 10% penalty, and a single professional beration receiver instead of multiple anonymous on call rotation.
It's not a clear choice, especially considering that the dedicated route with SLA means introducing change risk to the operational and SLA benefit of the ISP, fiber, new holes, new lines, etc..
>You could also update to IPv6.
I would, they don't offer IPv6
>You don't need an address block from an RIR. You just need your ISP to give you an address.
Yeah, maybe. If I get an IPv4 block though what I'd do is a regional entity whether for profit or NFP to offer emergency regional tunneling, potentially while finding alternative revenue streams like collecting lists of wronged ISP customers and notifying them of class action lawsuits they may participate in. A 25K check, or a 5K check with an ipv4 block and I'm golden.
Well if your hospital is running on the cheapest possible internet made for houses, then you have no rights. You should GET a contract that includes what you actually need.
Email sent!