About the same time the 500-mile email problem happened (mid 1990s), I had a difficult to understand issue with my office PC. Every morning, I'd come in, slide my hard drive sled in, and turn the computer on. We had 128 Kbps ISDN internet at the office and I had the same at home, but that was too slow to do much work. So I'd take the drive home so I could work at night, especially in the winter when the office was too cold at night.
Suddenly one winter morning, the PC wouldn't boot. I had to run to a meeting. When I got back, I turned the PC off and on again and everything was fine. The next morning, the same thing happened. The third day, I didn't have a meeting. I turned it off and back on, still no boot. I'd gotten in late, so I just turned it off and took an early lunch. When I got back, it still wouldn't boot. But I had a meeting, so I ran to that, leaving the computer on. When I got back, it booted fine.
The next morning, same thing. I decided to look inside, not having any idea what might cause such symptoms. As I took the shell off, a tiny mouse came out, jump off my desk, and ran across my lap before jumping on the floor and scurrying out of sight. From inside the computer came the smell of mouse urine. Apparently he'd been crawling in through the open drive bay to keep warm every night, and urinating while he was in there. Once the computer had been on for a while, the heat and airflow would dry it out enough to eliminate whatever electrical short was keeping it from booting. I went to the store and bought an empty drive sled to put in the drive bay whenever I took my drive out, and the problem never came back. I felt lucky that the liquid didn't cause permanent damage.
Someone posted a similar story on one of the other times the 500 mile email was posted - where a car would fail to start if the owner bought strawberry ice-cream from the store, but would work if they have vanilla. I love the processes that go into finding the actual issue (regardless of if the ice cream story is true!): https://www.snopes.com/fact-check/cone-of-silence/
Mice can fit through tiny holes. An old rule says that if a pencil can get through - a mouse will get through.
Some mice even fly.
I once had a bat clinging on my good old CAT cable. So even leaving windows open at night might affect bandwidth...
Another classic is the "Frog on Keyboard error". Software developers have to be prepared for everything...
There used to be an option called "Cat guard" built into several historical (BBS ) software. On (and cannot remember the name) one software that did synchronization with other networks (e.g., FIDO, uunet) it was considered a major feature.
Primary purpose was to lock the keyboard so when the cat walked all over it, it would not disconnect.
He doesn't give the chairman due credit, IMHO. The chairman collected information to help solve the problem AND it actually was the information needed. Without it, the author might look for "randomly unreachable servers" for a long time.
It's almost raw data -- exactly what you would wish for. By lecturing people that "email does not work that way", next time you either get no data at all because people don't even try, or no data because people hide it thinking email doesn't work that way, or a misguided conclusion when a layman tries to make a better guess at the cause of the problem.
Absolutely. It's one of my all time favourites stories and this is pretty much the reason why. I wish my users gave me such specific steps to reproduce!
What's my recent annoyance is that users will describe their problem in great detail if they are talking to LLM, yet same people make just as shit support tickets as before
I do wonder if they already had a feeling it was not supposed to work that way hence the info gathering. This is one of my favorite all time IT stories because the client was right, and the engineer was left almost going crazy.
When I was a Junior I asked an honest question to the senior I was working with at the time, great dude, I basically asked him because everyone joked about the "works on my machine" crowd, so I said, so what the heck do I do if it only works on my machine? He said you have to figure out what's different. It sounds obvious or simple, but if you go with that mindset, when someone's stuck in the "it works on my machine idk why" sure enough I ask "what is DIFFERENT from your machine and this one?" and it almost always leads to the right answers. It triggers something in our brains. I usually follow up whats different with "what was the last change?" in the case of a production issue.
These kind of posts are why I check HN pretty much every day for 15+ yrs now. Hard to believe I've missed this one. Glad I caught it this time! This posts reminds me to stay humble and avoid jumping to conclusions without analysis.
> Btw for those wondering about reposts: reposts on HN are just fine after a year or so (https://news.ycombinator.com/newsfaq.html), and reposts of classics every now and then are good because it's important for new users to learn the classics!
It would be nice if HN would simply "float" to the front page the classics, a year or two after their submission. That would avoid duplication (specially of comments!), and allow people in the lucky 10,000 group know about it.
This gets posted just infrequently enough that I remember that I've read it before but forget why those emails weren't delivered, so I read it every time :-)
Every time see this story I think "oh, this is the story about the packet TTLs being set stupidly low or something but you wouldn't be able to narrow that exactly to 500 miles" and have to click and learn again the the first time it's about the connection timeout being set stupidly low.
Last night I downloaded a TV episode and played it in VLC. 30 seconds in, the power failed. Fine, it's an old laptop I'm using as a media server, battery is long dead - this never happened before but maybe something is loose. I checked the power supply and restarted it. It failed again at the same point in the video, and again a third time. Something about that video causes my laptop to die.
I turned it off and went to bed. Maybe I'll troubleshoot it today. But I'd love to understand what could have happened. The closest thing I know of is the Janet Jackson video that could crash hard drives [0]. In this case the sound was playing on a different device (my TV) so I don't think it's the same explanation.
For extra weirdness, the episode was Black Mirror S7E01. Exactly the kind of thing the creators would like to build into a Black Mirror episode.
Dying on the exact same frame, or just generally in the same spot?
In the case of the latter my first thought would be thermals. Different video codecs have significantly different decoding costs, and may also stress different parts of your system. You could check for that by playing that same video but not starting at the beginning and see if it's the same duration. Or jump to just before it dies and see if it plays through.
If by "downloaded" you mean The High Seas, those who provision the high seas are often on the cutting edge of using codecs with every last feature turned on to make the videos smaller to squeeze every last bit out of the encodings that they can, which can make them unusually expensive to decode. Or so I've heard.
I didn't get to dig much further into it, but for those of you who suggested ideas:
- not always the same frame. The first three failures were within seconds of each other, possibly the same frame. I tried again the next night and it got through that part of the video, but crashed a minute later
- I was able to play the video using a different app (Ubuntu's built-in Videos app from an old Ubuntu release, maybe 20.04)
Some of these video codecs have pathological cases that might be maxing out your video while doing the decoding. If you're only using it as a media server, that might exceed the (possibly age-degraded) capacity of your power supply. Replacing the power supply might help in that case.
It's also possible that something in a particular frame is triggering a bug in your driver and crashing that way. In that case, your best bet might be to transcode the video to a different codec or something.
Maybe your particular video download is from an entirely trustworthy source, but it's not unheard of that untrustworthy folks will modify a file with the intent of causing this to happen.
This, Stalking the Wiley Hacker[1], and others were the stories that got me into computers. I wish so much the experience of working in this industry hadn't so thoroughly annihilated the joy they once brought.
I had a chance to meet Cliff Stoll a couple summers ago. He was giving a presentation to a quilt society and it was great. If you ever have a chance to see him in person, no matter the topic, you will be greatly rewarded! He is such an energetic and enthusiastic person and he finds the beauty in all sorts of everyday things. I was captivated by him the entire time on a topic I only had a passing interest in at the beginning.
What I don't get is how the author can't pin the year down to anything narrower than "between 1994 and 1997," especially considering he wrote the article in 2002: only a few years later.
I'm not at all implying the story was fake; just this particular thing feels weird.
You just reminded me of my time working at Sendmail, where I often had to telnet to port 25 of some machine, and pretend to be a mail server sending email.
I used to be able to send all the commands without having to look them up. Not sure I could still do that today.
I think can still do it, 30 years after I last had to. The trauma of debugging sendmail m4 config issues for
hours while the company e-mail remained dysfunctional has permanently etched it into my mind.
EHLO example.com
MAIL FROM:<foo@example.com>
RCPT TO:<bar@example.com>
DATA
Subject: Hello, World
I have crawled through the depths of hell to deliver unto you this message.
.
Units is a cool piece of software, but I have since switched to qalculate.
Mostly units has some silly defaults like needing to type tempC(30) instead of 30C; and it's nice to have a full calculator.
I know it's a way to specify that the conversion is absolute rather than relative, but qalculate just asks you about it the first time you convert, and since converting oven and outside temperatures is most of what I do, I don't havr to bother with remembering a different syntax.
Also qalculate is an awesome piece of software in general, so if you're excited by units you should check it out!
Favorite feature: you can type in any equation, writing 'x' for an unknown quantity, and it will solve for x. This comes in handy to avoid having to engage brain even for simple calculations. How many pixels per mm is 96 DPI? Just type 96/inch = x/mm. Sure you could rearrange it yourself but why bother?
Our email systems are mostly mediated by giant hyper-scale companies (Microsoft, Google etc). The location of mail servers being where the recipient is seems quaint (and wonderfully decentralised).
And even if we do manage our own servers they are automated, and apps often containerised. Nobody ends up with older MTA due to an OS upgrade.
Remember reading this like 20 years ago nice to see it again.
It was a long time before I understood that remark.
Since Mel knew the numerical value
of every operation code,
and assigned his own drum addresses,
every instruction he wrote could also be considered
a numerical constant.
He could pick up an earlier "add" instruction, say,
and multiply by it,
if it had the right numeric value.
His code was not easy for someone else to modify.
…I almost choked on my breakfast bacon reading this. This is some fabulous “greybeard wizard” lore from the early days of the WWW that I just love hearing about.
Bless OP for sharing this gem today. I needed the laughter.
Never get tired of seeing this resurface every once and a while. There needs to be a /greatest for posts like these (while still allowing people to repost them every so often)
Honestly burst out laughing as I saw the FAQ section covering the timeout.
Thanks for sharing the link.
The ultimate explanation that he just pinged known distances to calculate the time and distance relation is actually brilliant I'm not sure it would have occured to me particularly quickly to just experiment.
About the same time the 500-mile email problem happened (mid 1990s), I had a difficult to understand issue with my office PC. Every morning, I'd come in, slide my hard drive sled in, and turn the computer on. We had 128 Kbps ISDN internet at the office and I had the same at home, but that was too slow to do much work. So I'd take the drive home so I could work at night, especially in the winter when the office was too cold at night.
Suddenly one winter morning, the PC wouldn't boot. I had to run to a meeting. When I got back, I turned the PC off and on again and everything was fine. The next morning, the same thing happened. The third day, I didn't have a meeting. I turned it off and back on, still no boot. I'd gotten in late, so I just turned it off and took an early lunch. When I got back, it still wouldn't boot. But I had a meeting, so I ran to that, leaving the computer on. When I got back, it booted fine.
The next morning, same thing. I decided to look inside, not having any idea what might cause such symptoms. As I took the shell off, a tiny mouse came out, jump off my desk, and ran across my lap before jumping on the floor and scurrying out of sight. From inside the computer came the smell of mouse urine. Apparently he'd been crawling in through the open drive bay to keep warm every night, and urinating while he was in there. Once the computer had been on for a while, the heat and airflow would dry it out enough to eliminate whatever electrical short was keeping it from booting. I went to the store and bought an empty drive sled to put in the drive bay whenever I took my drive out, and the problem never came back. I felt lucky that the liquid didn't cause permanent damage.
Someone posted a similar story on one of the other times the 500 mile email was posted - where a car would fail to start if the owner bought strawberry ice-cream from the store, but would work if they have vanilla. I love the processes that go into finding the actual issue (regardless of if the ice cream story is true!): https://www.snopes.com/fact-check/cone-of-silence/
I can hear Click and Clack, the Tappett Brothers, hooting and guffawing on Car Talk as I’m reading this Snopes article!
2 replies →
> Vanilla, being the most popular flavor, was in a separate case at the front of the store for quick pickup.
wish modern stores optimized for customer convenience instead of seeing most shelves along the way to the usual
1 reply →
Finally a real computer mouse! What a funny story :)
Mice can fit through tiny holes. An old rule says that if a pencil can get through - a mouse will get through. Some mice even fly. I once had a bat clinging on my good old CAT cable. So even leaving windows open at night might affect bandwidth...
Another classic is the "Frog on Keyboard error". Software developers have to be prepared for everything...
https://thedailywtf.com/articles/Classic-WTF-Cursed-and-ReCu...
There used to be an option called "Cat guard" built into several historical (BBS ) software. On (and cannot remember the name) one software that did synchronization with other networks (e.g., FIDO, uunet) it was considered a major feature.
Primary purpose was to lock the keyboard so when the cat walked all over it, it would not disconnect.
My mouse doesn't do that
This is almost the origin story for the EDM producer deadmau5's name.
Kind of similar to the story about the origins of the word "bug" in software
If this would have caught on we might have called bugs mice
Too many people remember the “bug” story as “Grace Hopper invented the term ‘bug’” when the real takeaway is “Grace Hopper was very funny.”
1 reply →
Isn't that story more myth than reality?
The history section of the Wikipedia entry for "bug" [1] suggests it predates computers by decades.
1- https://en.wikipedia.org/wiki/Bug_(engineering)
8 replies →
Seems like 'he' came out without damage, too :)
In another world, I guess bugs would be called mice. But then what would we call mice?
He doesn't give the chairman due credit, IMHO. The chairman collected information to help solve the problem AND it actually was the information needed. Without it, the author might look for "randomly unreachable servers" for a long time.
It's almost raw data -- exactly what you would wish for. By lecturing people that "email does not work that way", next time you either get no data at all because people don't even try, or no data because people hide it thinking email doesn't work that way, or a misguided conclusion when a layman tries to make a better guess at the cause of the problem.
Absolutely. It's one of my all time favourites stories and this is pretty much the reason why. I wish my users gave me such specific steps to reproduce!
What's my recent annoyance is that users will describe their problem in great detail if they are talking to LLM, yet same people make just as shit support tickets as before
3 replies →
It helps to have a statistician and a geostatistician as your clients!
I do wonder if they already had a feeling it was not supposed to work that way hence the info gathering. This is one of my favorite all time IT stories because the client was right, and the engineer was left almost going crazy.
When I was a Junior I asked an honest question to the senior I was working with at the time, great dude, I basically asked him because everyone joked about the "works on my machine" crowd, so I said, so what the heck do I do if it only works on my machine? He said you have to figure out what's different. It sounds obvious or simple, but if you go with that mindset, when someone's stuck in the "it works on my machine idk why" sure enough I ask "what is DIFFERENT from your machine and this one?" and it almost always leads to the right answers. It triggers something in our brains. I usually follow up whats different with "what was the last change?" in the case of a production issue.
Fantastic comment. Indeed.
Popular in:
2023 (1164 points, 198 comments) https://news.ycombinator.com/item?id=9338708
These kind of posts are why I check HN pretty much every day for 15+ yrs now. Hard to believe I've missed this one. Glad I caught it this time! This posts reminds me to stay humble and avoid jumping to conclusions without analysis.
That story has been repeated in various places for decades now.
One of the lucky 10,000! https://xkcd.com/1053/
3 replies →
[dead]
You guys beat me to it - I was working on the list!
Btw for those wondering about reposts: reposts on HN are just fine after a year or so (https://news.ycombinator.com/item?id=123489 - Feb 2008 (7 comments)
I can say that a new user (me) did learn the classic (this).
2 replies →
Pallet from the shipping room is a different story, though this one is mentioned in the comments
> Btw for those wondering about reposts: reposts on HN are just fine after a year or so (https://news.ycombinator.com/newsfaq.html), and reposts of classics every now and then are good because it's important for new users to learn the classics!
It would be nice if HN would simply "float" to the front page the classics, a year or two after their submission. That would avoid duplication (specially of comments!), and allow people in the lucky 10,000 group know about it.
1 reply →
I wonder when we will see a timeless classic like this on HN, but for AI.
We can then poke some fun at what AI did, what went wrong, and our incredibly illogical "debug" of AI
AI bots on dead internet will bring them up and have an earnest metallic laughter together.
This gets posted just infrequently enough that I remember that I've read it before but forget why those emails weren't delivered, so I read it every time :-)
Every time see this story I think "oh, this is the story about the packet TTLs being set stupidly low or something but you wouldn't be able to narrow that exactly to 500 miles" and have to click and learn again the the first time it's about the connection timeout being set stupidly low.
I believe I'm up for submitting it in a couple of months.
And still a treat. I love that story.
I thought it sounded familiar. Still a good read.
Last night I downloaded a TV episode and played it in VLC. 30 seconds in, the power failed. Fine, it's an old laptop I'm using as a media server, battery is long dead - this never happened before but maybe something is loose. I checked the power supply and restarted it. It failed again at the same point in the video, and again a third time. Something about that video causes my laptop to die.
I turned it off and went to bed. Maybe I'll troubleshoot it today. But I'd love to understand what could have happened. The closest thing I know of is the Janet Jackson video that could crash hard drives [0]. In this case the sound was playing on a different device (my TV) so I don't think it's the same explanation.
For extra weirdness, the episode was Black Mirror S7E01. Exactly the kind of thing the creators would like to build into a Black Mirror episode.
[0] https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...
Dying on the exact same frame, or just generally in the same spot?
In the case of the latter my first thought would be thermals. Different video codecs have significantly different decoding costs, and may also stress different parts of your system. You could check for that by playing that same video but not starting at the beginning and see if it's the same duration. Or jump to just before it dies and see if it plays through.
If by "downloaded" you mean The High Seas, those who provision the high seas are often on the cutting edge of using codecs with every last feature turned on to make the videos smaller to squeeze every last bit out of the encodings that they can, which can make them unusually expensive to decode. Or so I've heard.
I didn't get to dig much further into it, but for those of you who suggested ideas:
- not always the same frame. The first three failures were within seconds of each other, possibly the same frame. I tried again the next night and it got through that part of the video, but crashed a minute later - I was able to play the video using a different app (Ubuntu's built-in Videos app from an old Ubuntu release, maybe 20.04)
3 replies →
Some of these video codecs have pathological cases that might be maxing out your video while doing the decoding. If you're only using it as a media server, that might exceed the (possibly age-degraded) capacity of your power supply. Replacing the power supply might help in that case.
It's also possible that something in a particular frame is triggering a bug in your driver and crashing that way. In that case, your best bet might be to transcode the video to a different codec or something.
Maybe your particular video download is from an entirely trustworthy source, but it's not unheard of that untrustworthy folks will modify a file with the intent of causing this to happen.
If you manage to find out what the cause is, I'd love to hear it.
Reminds me of this classic that resurfaces here every few years: if I buy vanilla ice cream my car won’t start https://www.netscrap.com/netscrap_detail.cfm?scrap_id=501
For me, it brings to mind the SR-71 speedcheck story just as a similar classic. https://www.thesr71blackbird.com/Aircraft/Stories/sr-71-blac...
A classic. It's like the hacker version of the SR-71 Blackbird speed check story [0]. Every time it comes up, I have to read it.
0: https://www.thesr71blackbird.com/Aircraft/Stories/sr-71-blac...
This, Stalking the Wiley Hacker[1], and others were the stories that got me into computers. I wish so much the experience of working in this industry hadn't so thoroughly annihilated the joy they once brought.
[1] https://archive.org/details/5626281-Clifford-Stoll-Communica...
I had a chance to meet Cliff Stoll a couple summers ago. He was giving a presentation to a quilt society and it was great. If you ever have a chance to see him in person, no matter the topic, you will be greatly rewarded! He is such an energetic and enthusiastic person and he finds the beauty in all sorts of everyday things. I was captivated by him the entire time on a topic I only had a passing interest in at the beginning.
FAQ about this, which answers such questions as "Did this actually happen, or were you just spinning a yarn?"
https://ibiblio.org/harris/500milemail-faq.html
As many times as I've read this story, I've never come across this.
Pity, as the constant handwaving in the answers makes the entire thing seem made up.
What I don't get is how the author can't pin the year down to anything narrower than "between 1994 and 1997," especially considering he wrote the article in 2002: only a few years later.
I'm not at all implying the story was fake; just this particular thing feels weird.
How about sending mail 500 miles more?
Just to be the man/woman/non-binary who sends mail 500 miles to your front door?
You had me at EHL0.
> You had me at EHL0.
You just reminded me of my time working at Sendmail, where I often had to telnet to port 25 of some machine, and pretend to be a mail server sending email.
I used to be able to send all the commands without having to look them up. Not sure I could still do that today.
I think can still do it, 30 years after I last had to. The trauma of debugging sendmail m4 config issues for hours while the company e-mail remained dysfunctional has permanently etched it into my mind.
Wietse Venema saved us all.
3 replies →
I immediately did a "apt install units". Very cool!
Units is a cool piece of software, but I have since switched to qalculate. Mostly units has some silly defaults like needing to type tempC(30) instead of 30C; and it's nice to have a full calculator.
I know it's a way to specify that the conversion is absolute rather than relative, but qalculate just asks you about it the first time you convert, and since converting oven and outside temperatures is most of what I do, I don't havr to bother with remembering a different syntax.
Also qalculate is an awesome piece of software in general, so if you're excited by units you should check it out!
+1 for qalculate
Favorite feature: you can type in any equation, writing 'x' for an unknown quantity, and it will solve for x. This comes in handy to avoid having to engage brain even for simple calculations. How many pixels per mm is 96 DPI? Just type 96/inch = x/mm. Sure you could rearrange it yourself but why bother?
I did as well. I found a bug right away. If I use "units" the resulting calculations are reversed.
You have: 1 mile You want: kilometers * 1.609344 / 0.62137119
You have: 1 unit You want: 1.609344 units * 0.62137119 / 1.609344
The output format is weird. I recommend `units --one-line` (or `units -1` for short).
I was disappointed to find Arch doesn't have it. :(
It has it in the AUR: https://aur.archlinux.org/packages/units
There's also the "magic" and "more magic" switch.
https://users.cs.utah.edu/~elb/folklore/magic.html
I once had a computer that would turn itself off when I left the room to get a drink.
Turned out be an old building with loose floorboard. The force of standing up was just enough to short out a failing power supply.
"Thankfully, it failed." So relatable, in general, when debugging systems.
So funny to think about this now.
Our email systems are mostly mediated by giant hyper-scale companies (Microsoft, Google etc). The location of mail servers being where the recipient is seems quaint (and wonderfully decentralised).
And even if we do manage our own servers they are automated, and apps often containerised. Nobody ends up with older MTA due to an OS upgrade.
Remember reading this like 20 years ago nice to see it again.
This is one of my favorite Old Internet tales. It's up there with "Mel, The Real Programmer."
Absolute classic if anybody else is interested in reading:
https://users.cs.utah.edu/~elb/folklore/mel.html
Mel was for sure on another level:
It was a long time before I understood that remark. Since Mel knew the numerical value of every operation code, and assigned his own drum addresses, every instruction he wrote could also be considered a numerical constant. He could pick up an earlier "add" instruction, say, and multiply by it, if it had the right numeric value. His code was not easy for someone else to modify.
I also enjoy the one about testing bird strike tolerances with store-bought chicken.
Here is another classic: wrong password when standing. https://www.reddit.com/r/talesfromtechsupport/comments/3v52p...
This is a good read! and something i have in the back of my head when debugging spooky bugs.
I never realized this was 2002 and when I first read it, how new it was.
And here we are almost 25 years later.
Everytime this pops up I immediately think of this classic: https://www.cartalk.com/radio/puzzler/flavors
…I almost choked on my breakfast bacon reading this. This is some fabulous “greybeard wizard” lore from the early days of the WWW that I just love hearing about.
Bless OP for sharing this gem today. I needed the laughter.
Never get tired of seeing this resurface every once and a while. There needs to be a /greatest for posts like these (while still allowing people to repost them every so often)
My favorite bug story. I created a curated list of similar bugs at https://500mile.email
> It hadn't been altered -- it was a sendmail.cf I had written. And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES" option.
This is gold.
I love that the statistics department decided not to contact IT until they had enough data to be statistically significant!
All time classic.
https://www.snopes.com/fact-check/shoot-me-kangaroo-down-spo...
I'm sure this part of the "boring details" omitted.
But what was the actual timeout and distance?
Presumably 60-70% VF of PVC coated copper?
So a 5ms timeout would be a 500mile run?
There's an FAQ here: https://www.ibiblio.org/harris/500milemail-faq.html
Honestly burst out laughing as I saw the FAQ section covering the timeout.
Thanks for sharing the link.
The ultimate explanation that he just pinged known distances to calculate the time and distance relation is actually brilliant I'm not sure it would have occured to me particularly quickly to just experiment.
It was a fake story he made up to help in his job search. Don't expect any of the details to add up.
This post always go back, like the doctors in Brazin using tilapia skin to heal burn wounds article.
This story travels at light speed and will never get old.
Reminds me of the time I went to Ceti Alpha 6
A classic.
Related:
Can an email go 500 miles in 2025?
https://news.ycombinator.com/item?id=44466030
TIL about 'units'
Heisenbug
Now someone post the "we can't print on Tuesdays story" too
This made me LOL so many times.
[dead]
[dead]
[dead]