Long ago I worked on the firmware for a game controller. We started getting reports back of ghost inputs like stuck buttons and false presses after we sent some early hardware to media reviewers. Given the power of game media at the time, this was an immediate code red. We took shifts playtesting various video games for nearly a week straight just to try and replicate the issues. No luck, only the reviewers could manifest it. We were about to put reviewers on a plane to demonstrate the issue in person when I decided to clean my desk. In doing so I tossed a bare PCB running debug to the other side of the desk and my console went wild.
Turns out the PCBs were shock/pressure sensitive, and the debouncing was just a bit off. Reviewers were getting really into their games and mechanically stressing the controllers. Stressed hard enough, the PCB would bend slightly, causing line level fluctuations and eventually ghost inputs. Back in the office we were just doing a job and not getting too emotionally involved in our playtesting.
Some new molds and review units later we shipped the working system. Percussive debugging has solved a number of otherwise intractable bugs over my career.
Had a bug about two years ago that I just could not reproduce. In fact, only the engineer who reported it could reproduce it. Finally, I did "go to gemba" and went into the test lab and watched him use the machine.
He was doing a complex operation and was young and fast enough to overcome the keypress reporting interval. Literally it was just someone doing something we didn't expect and doing it fast enough that the sequence got messed up.
was more widely read (and that it was updated with stories of turning OPENSTEP into Mac OS X), and if there is a similar site for Windows which collected stories such as:
Folklore.org starts off as an engaging and fascinating whistle-stop tour of a bunch of mad geniuses creating a personal computing revolution.
It ends up being a series of cautionary tales about Steve Jobs from a defeated and depressed Woz. Mostly about how he can't understand how someone can act so devoid of empathy.
Dealing with a "more magic switch" this week with Claude.
The AI prompting business feels a lot more like the analog circuit where something being "near" something else causes capacitance or inductance without actually being connected.
To me these tools are pointless unless they're 100% reproducible. If they're not reproducible, they create more problems than they solve.
Thankfully local models are a thing and the proper ones already have a "random" seed and given the same seed an the same prompt shall always give the same picture / answer / etc.
If they can't do that, they're the road to impossible to fix bugs, impossible to diagnose products defects.
Many who rely on non-deterministic models are in for a world of hurt that's going to make these post-modern of insane bugs look like cheap stakes.
I’m trying to track down an old story about the network not working in a firm on Wednesday morning near 7 am.
I remember the details being that only on Wednesday did they use the freight elevator to deliver donuts to an all-staff meeting and when they did, the network would fritz.
It turned out that if you move a very large electromagnet (on the elevator) along the network cable, it makes the network go crazy.
Combine that with cable-pullers that rather than putting holes in multiple ceilings and floors in the cable closet, saw that the elevator shaft was a pre-existing hole between floors anyway, simply took the path of least resistance and hilarity (and a real hard network debugging problem) ensues.
Does anyone know when or where this anecdote occurred ?
I have definitely found software bugs by wiring up solenoids to act as programmable button-pressers. Unlike junior testers, scripts and solenoids will work 24-hour shifts and never ask for pizza and coke.
Long ago I worked on the firmware for a game controller. We started getting reports back of ghost inputs like stuck buttons and false presses after we sent some early hardware to media reviewers. Given the power of game media at the time, this was an immediate code red. We took shifts playtesting various video games for nearly a week straight just to try and replicate the issues. No luck, only the reviewers could manifest it. We were about to put reviewers on a plane to demonstrate the issue in person when I decided to clean my desk. In doing so I tossed a bare PCB running debug to the other side of the desk and my console went wild.
Turns out the PCBs were shock/pressure sensitive, and the debouncing was just a bit off. Reviewers were getting really into their games and mechanically stressing the controllers. Stressed hard enough, the PCB would bend slightly, causing line level fluctuations and eventually ghost inputs. Back in the office we were just doing a job and not getting too emotionally involved in our playtesting.
Some new molds and review units later we shipped the working system. Percussive debugging has solved a number of otherwise intractable bugs over my career.
Had a bug about two years ago that I just could not reproduce. In fact, only the engineer who reported it could reproduce it. Finally, I did "go to gemba" and went into the test lab and watched him use the machine.
He was doing a complex operation and was young and fast enough to overcome the keypress reporting interval. Literally it was just someone doing something we didn't expect and doing it fast enough that the sequence got messed up.
> Percussive debugging
S-tier term. Will need to add that to my repertoire.
Nothing a bit of percussive maintenance cannot solve!
1 reply →
Arguably, _The Jargon File_:
http://www.catb.org/jargon/html/index.html
and _Zen and the Art of the Internet_
https://www.gutenberg.org/ebooks/34
should be a part of the school curriculum covering the internet.
While specific to the Mac, one wishes:
https://folklore.org/0-index.html
https://www.goodreads.com/en/book/show/40492.Revolution_in_T...
was more widely read (and that it was updated with stories of turning OPENSTEP into Mac OS X), and if there is a similar site for Windows which collected stories such as:
https://www.joelonsoftware.com/2006/06/16/my-first-billg-rev...
Folklore.org starts off as an engaging and fascinating whistle-stop tour of a bunch of mad geniuses creating a personal computing revolution.
It ends up being a series of cautionary tales about Steve Jobs from a defeated and depressed Woz. Mostly about how he can't understand how someone can act so devoid of empathy.
Yeah, it was kind of saddening when watching a video interview which covers the later timeframe:
https://news.ycombinator.com/item?id=42585332
1 reply →
I wish the jargon file was still updated, it's such an interesting window into the compsci culture back then
Dealing with a "more magic switch" this week with Claude.
The AI prompting business feels a lot more like the analog circuit where something being "near" something else causes capacitance or inductance without actually being connected.
What is old is again new!
Yeah there's going to be AI version of row hammer attacks where you pollute the training data to get a favourable bias somehow.
We should have skipped this phase of AI development and just created terminators.
I'm playing a lot with AI but...
To me these tools are pointless unless they're 100% reproducible. If they're not reproducible, they create more problems than they solve.
Thankfully local models are a thing and the proper ones already have a "random" seed and given the same seed an the same prompt shall always give the same picture / answer / etc.
If they can't do that, they're the road to impossible to fix bugs, impossible to diagnose products defects.
Many who rely on non-deterministic models are in for a world of hurt that's going to make these post-modern of insane bugs look like cheap stakes.
I’m trying to track down an old story about the network not working in a firm on Wednesday morning near 7 am.
I remember the details being that only on Wednesday did they use the freight elevator to deliver donuts to an all-staff meeting and when they did, the network would fritz.
It turned out that if you move a very large electromagnet (on the elevator) along the network cable, it makes the network go crazy.
Combine that with cable-pullers that rather than putting holes in multiple ceilings and floors in the cable closet, saw that the elevator shaft was a pre-existing hole between floors anyway, simply took the path of least resistance and hilarity (and a real hard network debugging problem) ensues.
Does anyone know when or where this anecdote occurred ?
You can find a it of Unix history here, docs, pics, and videos. All from an old geek. http://crn.hopto.org/unix/
previous https://news.ycombinator.com/item?id=23005140
I've got a project kind of similar but I try to give well referenced historical evidence to try to suss out the true narrative, if any
I’ll need to read up on those. Only recognized two from title (more magic and 500 mile)
The number of computer-related problems that I solved by putting a heavy object on my keyboard and taking a break is mind-boggling.
I have definitely found software bugs by wiring up solenoids to act as programmable button-pressers. Unlike junior testers, scripts and solenoids will work 24-hour shifts and never ask for pizza and coke.
"we have a tradition in our family of ice cream for dessert after dinner each night. "
I immediately geolocated this story.
Care to elaborate? IME this is fairly common. Do you just mean that you guessed that it was in America?
Correct. The abundancy of ice cream for dessert everynight just sets the mental imagery for a big american family.
They're probably guessing italy.
5 replies →
I didn't!
from the hackers test https://stuff.mit.edu/afs/net/user/tytso/archive/hackers.tes...
...my fave:
0015 Ever change the value of 4?
0016 ... Unintentionally?
0017 ... In a language other than Fortran?
cool stories to read during my lunch