Comment by Trev123

This post hits close to home, I've run into all of these myself.

On checksums: Incremental updates are the path of least pain only if the packet’s checksum is valid and not CHECKSUM_PARTIAL. With modern offloads (TSO/GSO/GRO/checksum offload), the checksum visible to XDP is often zero/garbage because the NIC fills it later. In practice, either disable offloads for that traffic or recompute from scratch with bpf_csum_diff() plus bpf_l3_csum_replace() / bpf_l4_csum_replace().

The verifier: This is a fun one, when you make a small change and suddenly the verifier won't allow it.

And the moment you start modifying packets too much yourself, you're on the hook for everything the kernel used to do for you.

I once went down the rabbit hole of building a minimal TCP stack, and the experience was exactly as you'd expect. Getting to 95% done felt quick, but that last 5% was a nightmare (if 100% is even achievable)