But plenty has changed since then. It seems all my complains about the API are now addressed. Maybe it's a good time to actually run with MPTCP again :)
In my private affairs, I realised I need MPTCP less, since I started using tailscale. My SSH sessions tend to last longer when going over it.
My biggest disappointment with MPTCP was that seemingly, you need both the old and new address to be available when doing handover. So you cannot suspend your laptop, go to a different place, and resume the TCP sessions on the new Wi-Fi.
Of course you can do that. There are different timeouts (MPTCP level, TCP (and SSH) keep alive, etc.) to prevent having dangling connections for a while, but they can be changed if needed.
MPTCP being enabled on a server is what makes clients running on Apple devices magically not time out when you walk out of Wi-Fi range or switch to LTE (and is behind the “Wi-Fi Assist” setting/feature). iOS and macOS have had it quietly on by default for years: at first just iCloud etc. used it under the hood but for instance WeChat started enabling it like a decade ago for the improved performance.
With MPTCP, the same TCP session hops networks mid-flight. Without it, these seamless handoffs are at best fast reconnects. It’s one of those Apple things that “just works”; on your Linux server you need to flip it on in `socket()` or look into `mptcpize` last I checked but it’s no big deal. I dont think it’s well supported as a client yet and who knows if Android will ever.
(the “Wi-Fi Assist” toggle in Settings doesn’t enable/disable MPTCP, it is on regardless, but it decides if iOS will spin up a cellular subflow when Wi-Fi starts flaking out. It will use some metered data, hence the user-facing toggle.)
More and more apps (mostly server apps) have a dedicated option to enable MPTCP. Some server apps have even decided to enable MPTCP support by default, which makes sense: if MPTCP is not requested, TCP is used like before. Note that server apps written in Go usually have MPTCP enabled by default (if supported by the OS/kernel). See: https://www.mptcp.dev/apps.html
mptcp.io monitors servers supporting MPTCP.
> I dont think it’s well supported as a client yet
It is: by default, NetworkManager will configure MPTCP endpoints, so app can use multiple interfaces (if any). See: https://www.mptcp.dev/pm.html
> who knows if Android will ever
Sadly, it is difficult to talk to people in charge there. A few years ago, they were interested in MPTCP, but it was not available in the official Linux kernel. Now it is, and easily accessible (especially for small actors)... but Google has enough resources to find and use alternatives they fully control.
Sort of related, I'm curious why SCTP did not take off more in this space? It might have had more telecom origins maybe but seemed to fill some of the same needs back in the day.
Windows doesn't have kernel mode SCTP so it was slow for most consumer devices for a long time. Even now, Linux SCTP is slow in comparison to other protocols. Plus, it's complicated enough already to get UDP and TCP traffic to make it's way through middleboxes. Also, not a lot of consumer routers support things like port forwards and combining SCTP with NAT doesn't seem to be widely tested. Things just didn't work out when SCTP stood to gain adoption.
It's an interesting protocol, but these days I think the internet has ossified so far that you're probably better off relying on hacks like QUIC and MPTCP to get the protocol features that SCTP stood to introduce.
Coming from the telecoms space, I was slightly amazed to see how little well known SCTP actually is in the networking world.
My old company/offices had site internet provided by one of the top 50 UK Managed Service Providers. They swapped out the on-site router not many years ago as the fibre to site was being upgraded from 100mbit to gigabit and so a new Juniper firewall with GBE ports was required.
Turns out the newer, faster, shinier, though albeit lower model numbered'd Juniper SRX fundamentally didn't support passing SCTP data and suddenly we lost access to all our remote stuff that used it. Ended up on a call with the MSPs Head of Networks (who was not a stupid person), but their opening gambit was "Are you sure you mean SCTP? Oh. What is that then?"
There was also numerous weird kernel bugs with implementations on CentOS 5, 6 and 7 which all would manage to get themselves into weird states where only a reboot would clear - not really what you want from a multi-endpoint, 'copes and recovers well from network weirdness' tunnelling protocol.
What are those flaws? And are they inherent, or something that could be fixed with enhancements/revisions?
TCP as originally specified had fundamental design flaws too, but the TCP of today has significant differences from the 1981 TCP standard (not the first version of TCP, but the first version to see significant production use).
I've tried to get MPTCP support in Chrome (default OFF, with a knob to turn on in GUI) but the reason most often people gave was that there is how to handle support.
Heck. I even tried to add it into git because i was having issues with reliable connectivity with WiFi and 5G (i was in a hotel at that time) while working on a project.
So unless, if there is some reason why people kept giving reasons of not include it. I just do not have a reason to add support for $(name your favorite software)
z/OS has this interesting feature AT-TLS (Application Transparent TLS)-an app requesting a plaintext socket may silently get a TLS socket instead (with the TLS implemented in the OS), based on a policy defined by the system administrator.
Maybe the solution might be something similar… an app asks for a TCP socket, but (if the request matches a policy) it gets an MPTCP socket instead-so you could make apps use MPTCP even if they weren’t compiled to support it.
Maybe you could implement this using LD_PRELOAD/ptrace/eBPF/etc
But the best is to let the app (and their users) controlling that, with a nice option. With Chrome/Firefox/..., we could enable MPTCP per domain for example.
Seems like a fool's errand. The cited RFC8684 starts from the wrong premise:
> TCP/IP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers.
In reality, IP modules of all the hosts and routers can load-balance over a set of all available interfaces, as long as global routing information is available.
It is very rare that a single TCP connection will be load-balanced over multiple interfaces, since that would frequently cause out-of-order delivery, which sucks quite a lot if the receiver isn't prepared for it.
Hey! MPTCP again!
Back in Jan I wrote a piece about how to actually use MPTCP
https://blog.cloudflare.com/multi-path-tcp-revolutionizing-c...
But plenty has changed since then. It seems all my complains about the API are now addressed. Maybe it's a good time to actually run with MPTCP again :)
In my private affairs, I realised I need MPTCP less, since I started using tailscale. My SSH sessions tend to last longer when going over it.
My biggest disappointment with MPTCP was that seemingly, you need both the old and new address to be available when doing handover. So you cannot suspend your laptop, go to a different place, and resume the TCP sessions on the new Wi-Fi.
Of course you can do that. There are different timeouts (MPTCP level, TCP (and SSH) keep alive, etc.) to prevent having dangling connections for a while, but they can be changed if needed.
If anyone is unaware:
MPTCP being enabled on a server is what makes clients running on Apple devices magically not time out when you walk out of Wi-Fi range or switch to LTE (and is behind the “Wi-Fi Assist” setting/feature). iOS and macOS have had it quietly on by default for years: at first just iCloud etc. used it under the hood but for instance WeChat started enabling it like a decade ago for the improved performance.
With MPTCP, the same TCP session hops networks mid-flight. Without it, these seamless handoffs are at best fast reconnects. It’s one of those Apple things that “just works”; on your Linux server you need to flip it on in `socket()` or look into `mptcpize` last I checked but it’s no big deal. I dont think it’s well supported as a client yet and who knows if Android will ever.
(the “Wi-Fi Assist” toggle in Settings doesn’t enable/disable MPTCP, it is on regardless, but it decides if iOS will spin up a cellular subflow when Wi-Fi starts flaking out. It will use some metered data, hence the user-facing toggle.)
More and more apps (mostly server apps) have a dedicated option to enable MPTCP. Some server apps have even decided to enable MPTCP support by default, which makes sense: if MPTCP is not requested, TCP is used like before. Note that server apps written in Go usually have MPTCP enabled by default (if supported by the OS/kernel). See: https://www.mptcp.dev/apps.html
mptcp.io monitors servers supporting MPTCP.
> I dont think it’s well supported as a client yet
It is: by default, NetworkManager will configure MPTCP endpoints, so app can use multiple interfaces (if any). See: https://www.mptcp.dev/pm.html
> who knows if Android will ever
Sadly, it is difficult to talk to people in charge there. A few years ago, they were interested in MPTCP, but it was not available in the official Linux kernel. Now it is, and easily accessible (especially for small actors)... but Google has enough resources to find and use alternatives they fully control.
Sort of related, I'm curious why SCTP did not take off more in this space? It might have had more telecom origins maybe but seemed to fill some of the same needs back in the day.
https://docs.kernel.org/networking/sctp.html
PS the kernel work goes back to 2003!
Windows doesn't have kernel mode SCTP so it was slow for most consumer devices for a long time. Even now, Linux SCTP is slow in comparison to other protocols. Plus, it's complicated enough already to get UDP and TCP traffic to make it's way through middleboxes. Also, not a lot of consumer routers support things like port forwards and combining SCTP with NAT doesn't seem to be widely tested. Things just didn't work out when SCTP stood to gain adoption.
It's an interesting protocol, but these days I think the internet has ossified so far that you're probably better off relying on hacks like QUIC and MPTCP to get the protocol features that SCTP stood to introduce.
I believe it's because of firewalls. You need to be either UDP or TCP to work in the Internet at large.
Though SCTP did find its place as a layer in WebRTC.
Coming from the telecoms space, I was slightly amazed to see how little well known SCTP actually is in the networking world.
My old company/offices had site internet provided by one of the top 50 UK Managed Service Providers. They swapped out the on-site router not many years ago as the fibre to site was being upgraded from 100mbit to gigabit and so a new Juniper firewall with GBE ports was required.
Turns out the newer, faster, shinier, though albeit lower model numbered'd Juniper SRX fundamentally didn't support passing SCTP data and suddenly we lost access to all our remote stuff that used it. Ended up on a call with the MSPs Head of Networks (who was not a stupid person), but their opening gambit was "Are you sure you mean SCTP? Oh. What is that then?"
There was also numerous weird kernel bugs with implementations on CentOS 5, 6 and 7 which all would manage to get themselves into weird states where only a reboot would clear - not really what you want from a multi-endpoint, 'copes and recovers well from network weirdness' tunnelling protocol.
6 replies →
SCTP was (and probably still is) supported in many NAT devices, firewalls, and other middleboxes.
SCTP has several fundamental design flaws, which are sufficient to discourage anyone from actually trying to make all the middleware support it.
What are those flaws? And are they inherent, or something that could be fixed with enhancements/revisions?
TCP as originally specified had fundamental design flaws too, but the TCP of today has significant differences from the 1981 TCP standard (not the first version of TCP, but the first version to see significant production use).
1 reply →
I've tried to get MPTCP support in Chrome (default OFF, with a knob to turn on in GUI) but the reason most often people gave was that there is how to handle support.
Heck. I even tried to add it into git because i was having issues with reliable connectivity with WiFi and 5G (i was in a hotel at that time) while working on a project.
So unless, if there is some reason why people kept giving reasons of not include it. I just do not have a reason to add support for $(name your favorite software)
https://chromium-review.googlesource.com/c/chromium/src/+/63...
z/OS has this interesting feature AT-TLS (Application Transparent TLS)-an app requesting a plaintext socket may silently get a TLS socket instead (with the TLS implemented in the OS), based on a policy defined by the system administrator.
Maybe the solution might be something similar… an app asks for a TCP socket, but (if the request matches a policy) it gets an MPTCP socket instead-so you could make apps use MPTCP even if they weren’t compiled to support it.
Maybe you could implement this using LD_PRELOAD/ptrace/eBPF/etc
There are different ways to force an app to use MPTCP, where the most convenient method is 'mptcpize run <cmd>', see: https://www.mptcp.dev/setup.html#force-applications-to-use-m...
But the best is to let the app (and their users) controlling that, with a nice option. With Chrome/Firefox/..., we could enable MPTCP per domain for example.
Check out http://www.openmptcprouter.com which works excellent!
Track MPTCP adoption:
https://mptcp.io/
MPTCP is supported by more and more servers these days!
Note: if you don't see the two large graphs at the top, disable ad-blockers and/or try with another browser.
[flagged]
Seems like a fool's errand. The cited RFC8684 starts from the wrong premise:
> TCP/IP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers.
In reality, IP modules of all the hosts and routers can load-balance over a set of all available interfaces, as long as global routing information is available.
It is very rare that a single TCP connection will be load-balanced over multiple interfaces, since that would frequently cause out-of-order delivery, which sucks quite a lot if the receiver isn't prepared for it.
Isn't every TCP implementation required to deal with receiving packages out-of-order? That is one of the abstraction TCP provides.
1 reply →