Comment by ryao

7 months ago

I do not know exactly what is being done here, but I can say that I am aware of two techniques for sending bit streams over parallel wires while keeping the bits in order:

1. The lengths of all wires are meticulously matched so that signals arrive at the same time. Then the hardware simply assumes that the bits coming off each wire are in sequence by wire order. This is done in computers for high speed interfaces such as memory or graphics. If you have ever seen squiggly traces on a PCB going to a high speed device, they were done to get the lengths to be exactly the same so the signals arrive at the same time across each. This is how data transfers from dual channel DDR4 RAM where 64 bits are received simultaneously occur without reordering bits.

2. The lengths of wires are not matched and may be of different lengths up to some tolerance. Deskew buffers then are used to emulate matched lengths. In the case of twisted pair Ethernet, the wire pairs are not equal length because the twist rates vary to avoid interference from having the same twist rates. The result is the Ethernet PHY must implement a deskew buffer to compensate for the mismatched lengths and present the illusion of the wire lengths being matched. This is part of the Ethernet standard and likely applies to Ethernet over fiber too. The IEEE has a pdf talking about this on 800 Gb/sec Ethernet:

https://www.ieee802.org/3/df/public/23_01/0130/ran_3df_03_23...

LAG was never intended to have the sequence in which data is sent be preserved, so no effort was made to enable that in the standard.

That said, you would get a better answer from an electrical engineer, especially one that builds networking components.

I just noticed a typo in this. I should have written 128 bits, not 64 bits. Data transfers in dual channel DDR4 are 128-bits at a time.