Comment by gwright
13 years ago
Reminds me of a problem I had with a T1 circuit corrupting packets.
Shortly after bringing up a second T1 into a remote location we discovered that some web pages would show broken JPG images at the remote site.
Some troubleshooting revealed that this only happened when traffic was routed over the new T1. The old T1 worked just fine. Pings, and other IP traffic seemed to work over either line but we kept seeing the broken image icon for some reason when traffic came over the new T1.
We tried several times to confirm with the telco that the T1 was provisioned correctly and that our equipment matched those telco parameters. Still had some mangled bits going over that new T1.
Finally had the telco check the parameters over every span in the new (long-distance) T1 circuit and they eventually found one segment that was configured for AMI instead of B8ZS (if I can remember correctly, certainly it was a misconfigured segment though).
The net result is that certain user-data patterns that didn't include sufficient 0/1 transitions would lead to loss of clock synchronization over that segment and corrupted packets. Those patterns were most likely to occur in JPGs.
Once they corrected the parameters on that segment, everything worked as expected.
Quite a bit of head scratching with that one and lots of frustration as the layer-1 telco culture just couldn't comprehend that layer-2/3 Internet folks could accurately diagnose problems with their layer-1 network.
One of my fellow ISP admins (silicon.net, I was elite.net) had the same problem around 1995. GIFs would load on web pages but not JPEGs. They'd load for some amount of time and then hang.
The way he diagnosed it was to do a transfer of /dev/zero out one link. It worked. But it stopped almost immediately out the problem link. It turned out to be the same problem -- no zero bit stuffing configured on the line.
By the way, this same technique, known as "weak bits", was used as a floppy and CD/DVD copy protection scheme.
Yes. I remember being so excited that the option for pinging with particular bit patterns finally came in handy when I was debugging a new T1.
Nice story! I like it! Takes me back to my TDM days too.