Comment by gwbas1c
1 day ago
The first thing you need to ask is: Why do you need IPv4?
I don't know much about your imaging servers: Do they accept incoming https connections, or is it some other kind of TCP or UDP protocol? Assuming it's https, why do you need to stick with IPv4?
> Hi, I’m helping a hospital in Buenos Aires that has lost their IPv4 due to their ISP moving to CGNAT, putting their on prem imaging servers offline for 7 days. (it’s ok for us though, we have a backup failover isp)
Two more things:
1: If you require IPv4, that needs to be part of the contract with your ISP. If they just arbitrarily turned you off, it means that you didn't contract for the services you thought you had.
2: IPv6 is supposed to solve a lot of these problems... So again, why can't you move to IPv6?
> I don't know much about your imaging servers: Do they accept incoming https connections, or is it some other kind of TCP or UDP protocol?
Yes
>why do you need to stick with IPv4?
To support devices and routes that do not have ipv6 support?
>1: If you require IPv4, that needs to be part of the contract with your ISP. If they just arbitrarily turned you off, it means that you didn't contract for the services you thought you had.
That is correct in principle. Although, in the absence of a contract, if a service has been provided for a long time in one way, I'd say there's an argument for breach of contract when that way changes for the worst.
Another argument for breach of contract, which can work jointly with the previous, are industry standards, as far as they define the terms that are implied by common words like "internet connection".
Finally, there's the specifications themselves, since the service is billed as "internet", one can refer to the Internet Protocol Specification, although if they do offer IPv6, that might be reasonably considered a substitute, it is a material change that was not agreed to originally (this being a pre IPv6 contract). That said, an IPv4 with a local address might technically still be defined as internet, what is broken in this case is TCP, not the IP, although by lacking a public IP address, TCP is broken via an IP protocol failure, so there's an argument that something is being sold as Internet but it isn't really internet, there's no written contract necessary, the term used to describe the product creates the contract in the same way that selling a carton labeled Milk would create a contract based on milk coming from mammaries of a domestic mammal, and not pressed Soy Juice with artificial coloring.
>2: IPv6 is supposed to solve a lot of these problems... So again, why can't you move to IPv6?
I'm open to proposals to moving IPv6, I'm not content with having that be forced and removing it from people and institutions, if there's a shortage, then that means no new IP addresses can be issued, why are we normalizing the removal of IPv4 addresses? Are they being removed from lower priority users in order to serve high priority usecases? If so is a hospital a low priority user?
The main issue with IPv6 seems to be IPv6 proponents that try to kill IPv4, no one will migrate if you force them to, if you propose, we will consider.
> The main issue with IPv6 seems to be IPv6 proponents that try to kill IPv4, no one will migrate if you force them to, if you propose, we will consider.
My experience is more the opposite. I keep seeing people say things like "I don't do v6 at home because I don't need it". These people are outright telling us that they're not going to start using v6 unless we make them need it.
Note that I didn't say "migrate to" v6. They're not even willing to do the first step of that ("start using v6"), even with no pressure to do the last step ("stop using v4").
I read an anecdote from Facebook that they had huge trouble getting their developers to stop writing v4-only code. Eventually they ended up removing v4 from their developer machines, who all of a sudden were able to write v6-capable code -- and it didn't take long for them to start asking when they could drop support for v4.
People will moan and moan and moan about v6 until you make them use it, at which point they discover that it was never as bad as they imagined it being.
So you admit that IPv6 people routinely create the problem if it doesn't exist. Maybe there's some nuance where you need to go backwards in order to go forwards?
In any case it's not relevant to the case from the root comment, since the ISP doesn't offer IPv6. IPv6 folk are siding with the ISP here or getting distracted with IPv6.
Essentially IPv6 people are delusional about the state of support of the tech and the extent of issues it brings, there's regions where there's no IPv6 support, so when a network with IPv4 has an IP related problem, IPv6 folk will blame it on IPv4 even if it's entirely unrelated, even if the infrastructure isn't even at dual-stack phase.
One of the thing the ISP did was try to upgrade the connection from copper to fiber optics, like instead of understanding the issue, "let's just try upgrading from your OLD technology which is OUTDATED, no wonder you were having issues? Also have you tried turning it off an on again that helps? Talk to you tomorrow.". IPv6 guys so far have been a distraction, get walmart to run their website on IPv6 before you push it to a hospital in southamerica who is having downtime.
The professional services that I work with give plenty of notice (years) before turning off services. Were you using an internet connection that's intended for hosting servers? (As in you were paying for a dedicated static IP.) If so, then you have a case.
Otherwise, your arguments about what "internet" is are flawed. IPv6 has been a thing for about 30 years, with a well-publicized problem with IPv4 address space and migration plan. I suspect that it's (IPv6) older than your hospital's contract. Even in the IPv4 world, your IP can change at any time and you can be put behind a NAT. That's why ISPs will sell you a static IP if you need one.
>> why do you need to stick with IPv4?
> To support devices and routes that do not have ipv6 support?
> Are they being removed from lower priority users in order to serve high priority usecases? If so is a hospital a low priority user?
After reading your response, I suspect the real problem is attitude and experience. The hospital should have IT staff that know to purchase a static IPv4 IP. This should be part of the contract with the ISP. If it wasn't, ultimately it's up to the IT staff to approach the ISP and arrange to purchase one.
> The hospital should have IT staff that know to purchase a static IPv4 IP
I purchased a static IPv4 IP, and they made the CGNAT IP static.
I thought about making a blog post about it, but I realize there's so much nuance and details. I can't tell if it's just venting about work, or a genuinely interesting clusterfuck in some forlorn corner of the new world.
1 reply →
Have you noted that in the mainline Linux build config file, the option to enable IPv4 functionality is CONFIG_LEGACY_IP= ?
I can tell you that in cicd for production Linux systems today, we have
which in a Linux build config file means "do not include this in the build".
_
What might work well for your organization is ZeroTier which will make you immune to what any ISPs do. The downside is your organization would be dependent on ZeroTier, but ZeroTier is much less likely to screw you over than any ISP.
_
I appreciate your speculation on the principles of contract law. Like you, I believe those principles should be upheld and I believe in morality. If you lived under a monarchy and I were of the monarch, I would cross-examine your ISP in public and solve your problem in minutes. I would also on occasion take someone to the side of the road and chop off their head in front of the entire village, and everyone would know on a visceral level they are all much better off with someone taking out the trash now and then.
But, as many technical workers know, the contract stuff takes months or years to resolve, and technical issues involving IP addresses are best worked around with technical approaches.
_
What I would suggest your network team (you and whoever) do in your shoes is:
- set up NAT64 and DNS64 at the edges of subnets that contain devices that don't do IPv6 properly, because if these devices do not have updates to do IPv6 properly then they are broken in a fundamental way, and a workaround is necessary;
- dictate that starting on December 1st, all new devices and VMs have IPv6 addresses with all userspace services listening on IPv6 addresses or some nftables rule to kludge IPv6:$port -> 127.0.0.1:$port if necessary, with firewall configured to block that IPv6 traffic, then at a later date firewalls not blocking those IPv6 addresses, then at a later date prohibit any new IPv4 addresses being assigned anywhere with special exception to devices tagged "legacy" or "end of life" requiring documentation of such;
- use a combination of IPv6 GUA addresses assigned by whatever ISP and IPv6 ULA addresses chosen by your team -- in IPv6, multiple local gateways and multiple addresses on a network interface issued by those local gateways via RA is relatively a norm rather than an exception (requires userspace tooling in IPv4, DHCP doesn't have this functionality), so leverage this advantage;
- use IPv6 AH (authenticated headers) wherever possible;
- because in an ideal world, ISPs would behave consistently and IPv6 GUA addresses would always be stable, but we aren't living in an ideal world, so be ready to update your internal DNS to drop AAAA records containing IPv6 GUA that an upstream ISP fumbled;
- or, if you have the internal resources, register your own AS and command your own official block of IPv6 addresses;
- use a combination of OpenTOFU (Terraform) and Salt or Ansible, all versioned with Git, be very strict with your team about versioning with Git, to manage everything, and use
and Salt or Ansible equivalents for continuous monitoring;
- move userspace services to Podman where possible.
Personally I like Salt. I know Ansible is more popular out there with more inertia.
_
>> why are we normalizing the removal of IPv4 addresses?
This was normalized a long time ago actually. "The future is already here, it's just not evenly distributed".
You and I are total strangers on different corners of the globe. You may not like what you're about to read, but it's better that you get a hint where the wind is blowing sooner than not.
If my team were consulting with your organization's execs and we caught a hint that the existing internal resources were not already proactively staying current with internet protocol (which is IPv6), we would point out there is urgency to start right away with emphasis that the existing internal team is over two decades behind staying current.
Or, if a detected hint the internal team were at all resistant to IPv6, we would advise the executive team to start developing a plan to flush out the replacements for the internal team ASAP.
Many network engineers at many ISPs would gladly drop all IPv4 traffic if they thought they could get away with it.
IPv6 only seems more complicated if you still need to unlearn IPv4. IPv6 does have a steeper learning curve, but once that learning curve is overcome, IPv6 is actually much simpler than IPv4. More people than not don't like reading RFCs, but professionals do slowly read RFCs on their own time.
Thanks, I'll take a look into the recommended protocols.
One clarification I'd like to make is that IPv6 is not offered, I can only get IPv4 addresses through the ISP DHCP, there is no ipv6 being offered with command 'dhclient -6' for example on linux, neither does window pick any ipv6 up so it doesn't seem to be an us problem.
It's often that Argentinians miss nuances when copying ethical gray areas from the Americans, and without those slight nuances the debate often gets mixed up with a much more subtle debate.
To make a parallel, Trump issued a memecoin days before entering office, it wasn't technically while being president, it was a memecoin so there was no underlying asset, a gray area, see? America.
Now Milei saw and culted the cargo, he made a coin during his office mandate, and he mentioned it would be used to fund Argentina tech projects, so it's an unregulated security. The nuances are lost, and it's just plain bad.
EDIT1:
>Or, if a detected hint the internal team were at all resistant to IPv6, we would advise the executive team to start developing a plan to flush out the replacements for the internal team ASAP.
>we would point out there is urgency to start right away with emphasis that the existing internal team is over two decades behind staying current.
I appreciate the candor
>You and I are total strangers on different corners of the globe.
This is true and it might impact the nuances and extra-technical semantics of the protocol. For example, in here dropping IPv4 support might mean dropping support for older devices, which might be less common in America, ISPs might not support it, so it's not just about the device but the network, to give you an idea I just got off the phone with the ISP which is THE biggest ISP in Argentina, they don't offer IPv6, period.
>If my team were consulting with your organization's execs
Here might be a big disconnect, my organization execs would probably be way out of budget for whatever your org may charge, that would never happen.
If you want to point the salesmen of your org to try and stir some shit up like that with the ISP, then that may make a bit more sense, the budgets may align for importing American consulting, and berating a big org chart that includes engineers might make more sense than trying to replace a 'team' of 1 or 2 fractionalist generalist IT guys on <500$ month contracts.
The ISP brand is Personal, local corp is Telecom Argentina S.A, Asynchronous System Number AS7303, trades as TEO in NYSE I think, owned by Grupo Clarin which comes is a traditional Media and news company. IPv6 is not offered to consumers right now.
>we would advise the executive team to start developing a plan to flush out the replacements for the internal team ASAP.
go nuts
6 replies →