Comment by pkolaczk
15 hours ago
> Which is why when folks nowadays say "you cannot use XYZ for embedded", given what most embedded systems look like, and what many of us used to code on 8 and 16 bit home computers, I can only assert they have no idea how powerful modern embedded systems have become.
Yet, I still need to wait about 1 second (!) after each key press when buying a parking ticket and the machine wants me to enter my license plate number. The latency is so huge I initially thought the machine was broken. I guess it’s not the chip problem but terrible programming due to developers thinking they don’t need to care about performance because their chip runs in megahertz.
There's no pressure to make a good product because nobody making this decision has to use the machine. Everywhere I've worked purchase decisions are made by somebody with no direct contact to the actual usage, maybe if you're lucky they at least asked the people who need the product what the requirements are, otherwise it's just whatever they (who don't use this product) thought would be good.
"Key presses are 15x slower than they should be" gets labelled P5 low priority bug report, whereas "New AI integration to predict lot income" is P0 must-fix because on Tuesday a sales guy told a potential customer that it'd be in the next version and apparently the lead looked interested so we're doing it.
Not just that, nobody chooses their parking spot based on the UI of the machine.
Banks and phone manufacturers now care about UI, because some of them started to do so, and people started switching to them en masse. US carriers were bleeding subscribers left and right when the iPhone was only available on AT&T, which was the first time people started switching plans to get a specific phone instead of the other way around.
People usually choose their parking based on where they want to go and how far it is from that place, and that trumps all other considerations. Paying more for programmers or parking machine processors would be a waste of money.
Interesting story; I went to park at a downtown lot in my local city (Vancouver BC) and the machine had an unusual UI. So I skipped the machine and scanned the QR code for the app. By the time I had taken the elevator up to the lobby of the building I had the app.
But then the usability on the app was so bad, that I actually could not figure out how to buy parking. The instructions were clear, but the latency on the app was unusable. The Internet connection was fine. It was the app. So I skipped the whole thing, went to dinner, and was happy when I found my car without a ticket.
"Unable to buy a ticket" would have been an interesting day in court.
> Paying more for programmers or parking machine processors would be a waste of money.
The rise of parking apps on mobile adds an interesting angle to this.
No doubt, many of us favour apps because the UX is so much better. Not quite sure if that affects the bottom line short-term, but long-term I’m sure it will.
[dead]
> There's no pressure to make a good product because nobody making this decision has to use the machine.
Most software sucks, even when people have to chose using it. Everything is buggy and slow, people are just used to software being bad.
While this is a decision-making problem, it is also an engineering incompetence problem. No matter what pointy haired boss is yelling about "priorities" ultimately software developers are the ones writing the code, and are responsible for how awful it is.
When it comes to priorities about what to write and what to focus on, the buck stops at management and leadership. When it comes to the actual quality of the software written, the buck stops at the developer. Blame can be shared.
Precisely this. We love to put our colleagues as competent victims of the system, but a competent engineer is unlikely to build an embeeded UI with high latency at their first try. It's a combination of cheap, underqualified labour and careless management.
To paraphrase Upton Sinclair: “It is difficult to get a man to prioritize something when his salary depends upon his not prioritizing it.”
1 reply →
This is only partially true.
If developers prioritize customer experience instead of velocity and cost in situations where that isn't warranted, the company they work for can no longer sell products as cheaply as their competitors do. This decreases their market share and their revenue, which means they'll employ fewer developers in the future.
This is almost an evolutionary process, many (but not all) markets choose for developers which don't care about such things.
You mean the developer hired by (and fired by) management?
> When it comes to the actual quality of the software written, the buck stops at the developer. Blame can be shared.
No. The quality is not prioritized by management. A dev that fails to ship a feature because they were trying to improve "quality" gets fired.
We have no labor power because morons spent the good times insisting that we don't need a professional organization to solve the obvious collective action problem.
2 replies →
My first guess was debouncing. They assume that the switches are worn out, deeply weathered, and cheaply made. Each press will cause the signal to oscillate and they're taking their sweet time to register it.
When the device is new this is an absurd amount of time to wait. As the device degrades over 10, 20 years, that programming will keep it working the same. Awful the entire time, yes, but the same as the day it was new.
I was late for a train at my local station and the parking machine was taking ages to respond to keypresses. I could see the training pulling up to the platform and I was still stuck entering the second digit of seven. In my shameful frustration I hit the machine fairly hard. While the button presses might take a while to register, the anti-tamper alarm has really low latency and is also quite loud.
You need to find the right person to complain to. Here we are sympathetic, but can't do anything.
The right person is the other riders on the train - but the hard part is to frame this such that they join you on a march to the the agency that owns that machines to complain. I wish you the best of luck figuring out how to do that (I don't know how to do it - and if I did there are might higher priority things that need to be fixed).
Debouncing would be smart, sure. But sometimes, these sorts of embedded machines are weirder than that.
At Kroger-brand gas stations near me, I get to interact with the buttons on gas pumps to select options and enter a loyalty ID.
Those buttons have visible feedback on a screen, and also audible feedback consisting of a loud beep. And there's always delays between button press and feedback.
Some combination of debounce and wear might explain that easily enough.
Except... the delay between pushing a button and getting feedback is variable by seemingly-random amounts. The delay also consistently increases if a person on the other side of the pump island is also pushing buttons to do their own thing.
It's maddening. Push button, wait indeterminate time for beep, and repeat for something like 12 or 13 button presses -- and wait longer if someone else is also using the machine.
I can't rationally explain any of that variability with debounce.
They are running it on a Java VM in a container - on a 386?
Over WiFi?
2 replies →
That's a good point. When I use them I assume they're making API calls to a central server to validate (or something) them.
Making API calls to a server to do button debouncing does sound like something so stupid a tech company would do it
1 reply →
One of the more inspired design choices of the parking ticket devices in my area is the inclusion of a key repeat feature.
If you keep your finger on the touchscreen for just long enough, it helpfully repeats the keystroke while you're entering a license plate.
Given the inevitable hardware issues, this means that what should be a single tap frequently becomes a burst of identical characters.
The programmers who worked on this probably would've liked to be game developers instead.
That's programmer incompetence. Unfortunately pervasive, especially with devices like parking meters, EV chargers, and similar, where the feedback loop (angry customer) is long (angry customers resulting in revenue decrease) or non-existent.
It could be a management problem instead also, while developers are just following instructions sent by management
I disagree — developers are not sheep.
And nobody with options would settle for the low pay and terrible working conditions, so the quality of the output also reflects that.
They also prefer you to use the mobile app so they can gather more data so they do not want the devices to work well in the first place.
It's a nice theory, but many of those terrible parking ticket machines predate smartphones, so it might be the case for machines built now, but it's really hard to imagine that that was the original intention
1 reply →
It's organizational incompetence driven by companies that see software development as a cost centre rather than a key asset.
It's usually clear when this has happened. Buggy bargain basement output.
Give it some slack, it's probably doing its best to inexplicably run windows.
Disagree. Windows for embedded runs extremely well, though can take a minute to boot.
My underpowered cash register that hadn't been updated in a decade could run POS on top of Windows 7 Embedded POSReady buttery smooth.
Occasionally they would start performing poorly, and it was always a network issue.
Or maybe they think they should be sending each keystroke to a server and waiting for the response.
A server on Mars?
Na each key press goes to a separate lambda invocation that gets submitted to a Kafka queue, and what happens after that is a mystery to all involved.
We can make crazy latency ourselves just fine, no space transmission necessary
1 reply →
Probably a Celeron-powered PC tower barely keeping up with Windows Server 2008 R2 in a closet of a public office ;)
Gotta have multiple AZs.
The server is probably running Python
1 reply →
Everyone was locked out in a building am staying at (40 something stories) for several hours. When I asked the concierge if I can have a look at the system, it turns out they had none. The whole thing communicated with AWS for some subscription SaaS that provided them with a front-end to register/block cards. And every tap anywhere (elevators/doors/locks) in the building communicated back with this system hosted on AWS. Absolute nightmare.
I wonder what happened to the building when us-east-1 went down.
As the parent said: “Everyone was locked out in a building am staying at (40 something stories) for several hours.”
Now I am waiting for time when they move us-east-1 physical security to run in us-east-1... Thus locking themselves out when needing some physical intervention on servers to get backup.
2 replies →
I wonder what happened to the building when the internet went down. How do you get into the room to reboot the router?
5 replies →
This is in SEA. They probably operate from ap-southeast-1 or 2. But yeah, if the internet goes down, the provider service goes down or AWS goes down they are cooked.
> Absolute nightmare.
Yes, but still probably a million times easier for both the building management and the software vendor to have a SaaS for that, than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.), and have someone deploy, install, manage, update, etc. all of that.
Easier maybe. But significantly worse. Parts of these systems have been build and engineered to be entirely reliable with automatic hand-overs when some component fails or with alternative routings when some connection is lost.
>than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.), and have someone deploy, install, manage, update, etc. all of that.
You don't need any of that. You need one more box in the electrical closet and one password protected wifi for all the crap in the building (the actual door locks and the like) to connect to.
6 replies →
Its absolutely possible to have both a SaaS based control plane and continue functioning if the internet connection/control plane becomes unavailable for a period. There's presumably hardware on site anyway to forward requests to the servers which are doing access control, it wouldn't be difficult to have that hardware keep a local cache of the current configuration. Done that way you might find you can't make changes to who's authorised while the connection is unavailable, but you can still let people who were already authorised into their rooms.
> with redundant power, cooling, etc
The doors the system controls don't have any of this. Hell, the whole building doesn't have any of this. And it definitely doesn't have redundant internet connections to the cloud-based control plane.
This is fear-mongering when a passive PC running a container image on boot will suffice plenty. For updates a script that runs on boot and at regular intervals that pulls down the latest image with a 30s timeout if it can't reach the server.
12 replies →
It's also easier to keep all the water for fighting fires in trucks that are remote, than to run high pressure water pipes to every room's ceilings, with special valves that only open when exposed to high heat. Imagine the overhead costs!
Cooling for a card access system?
A card access system requires zero cooling, it’s a DC power supply or AC transformer and a microcontroller that fits in a small unvented metal enclosure. It requires no management other than activating and deactivating badges.
There is no reason to have any of the lock and unlock functionality tied to the cloud, it’s just shitty engineering by a company who wants to extract rent from their customers.
9 replies →
The system was not built with resiliency in mind and had no care/considerations for what a shit-show will unfurl once the system or the link goes down. I wonder if exit is regulated (you can still fully exit the building from any point using the green buttons and I think these are supposed to activate/still work even if electricity is down).
> Yes, but still probably a million times easier for both the building management and the software vendor to have a SaaS for that, than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.)
A isolated building somewhere in the middle of the jungle dependent for its operation on some American data-center hundreds of miles away is simply negligence. I am usually against regulations but clearly for certain things we can trust that all humans will be reasonable.
1 reply →
Only in America. America is deigned to make you mad that public common life isn't keeping up with whats in everybody's pocket.
Gently forcing the individual to choose sapient or insentient.
Sorry to rant, but this kind of stuff is the only thing that triggers me. It's gotten so bad that my family makes me put a dollar in a 'complain jar' everytime I talk about how poor quality software has become.
Just one recent example: few months ago, I replaced a Bosch dishwasher with the latest version of the same model. Now, when I press the start button to initiate the cycle, it takes over 3 seconds for it to register! Like, what is going on in that 3 seconds?
How was it possible that even 'kind of good' developers like me were able todo much more with much less back in the 90s? My boss would be like, "Here's this new hardware thingy and the manual. Now figure out how to do the impossible by Monday." Was it because we had bigger teams, more focus, fewer dependencies?
I think we've been trained to accept bad software at this point, and a lot of people don't know anything different.
I suspect that a lot of it is caused by shoving Android onto underpowered devices because it is cheap and seems like an easy button. But I don't know for sure, that's just an impression. I have no numbers.
Could there be an opportunity here, for a specialized kiosk OS or something like that?
Whilst I can not see a motivation I refuse to accept that parking machines are not advisarial design. Why do they have haf a dozen things that look a bit like tap n pay if they are not trying to make it eaiser for card skimmers.
And the self service kiosks/checkouts at supermarkets. So infuriating! Like I'd have to try to make something that slow myself on purpose!
Besides the fact that scanning a barcode seems beyond much of the general population, they do it so sloooow.
Some of these are just dumb terminals with the entire state handled on a server. I've seen a bunch of them freeze at once where no UI would respond (but the interactions were buffered) and then when the network hiccup was over they all unfroze and reflected the input.
The self service kiosks are intentionally throttled when scanning barcodes, at a guess to prevent people accidentally scanning the previous/wrong item - I once had some problems with one and a staff member flipped it into supervisor mode at which point they were able to scan at the same rate you'd see at a manned checkout.
I think that's handled by the barcode scanner itself, at least on the ones I've used. The scanner will not recognize the same code immediately, but will immediately pick up a different code.
What's slow is that after each scan it needs to check the weight which means it lets the scales settle for one second before accepting another scan.
1 reply →
Idt that's it, at least in my experience.
I scan as fast as a manned checkout (I did my time in retail). And I can scan my groceries at the speed whilst the people next to me spend most of their time rotating an item to find the barcode.
It could also be intentional UX design. Or a result of the keyboard hardware.
What can you expect, when people assume as normal shipping the browser alongside the "native" application, and scripting languages using an interpreter are used in production code?
Maybe that ticket machine was coded in MicroPython. /s
Interpreters don't have to be slow.
Forth is usually interpreted and pretty fast. And, of course, we have very fast Javascript engines these days. Python speed is being worked on, but it's pretty slow, true.
Classic Forth Dimensions article: Why Forth Isn't Slow
* https://www.forth.org/fd/FD-V06N5.pdf
Basically it is because Forth programs are fairly flat and don't go deep into subfunctions. So the interpreter overhead is not that great and the processor spends most of the time running the machine code that underlays the primitives that live at the bottom of the program.
It's not really "interpreted", in the way that for example BASIC or Java is.
It's a list of jumps to functions.
Some Forths are dog slow such as PFE compared to GForth. Meanwhile others running in really slow platforms such as subleq (much faster in muxleq) run really fast for that the VM actually as (almost something slightly better than a 8086).
- TCL/Tk slowish under P3 times, decent enough under P4 with SSE2. AMSN wasn't that bad back in the day, and with 8.6 the occasional UI locks went away.
- Visual Basic. Yes, it was interpreter, and you used to like it. GUI ran fast, good for small games and management software. The rest... oh, they tried to create a C64 emulator under VB, it ran many times slower than one created in C. Nowadays, with a P4 with SSE2 and up you could emulate it at decent speeds with TCL/Tk 8.6 since they got some optimized interpreter. IDK about VB6, probably the same case. But at least we know TCL/Tk got improved on multiprocessing and the like. VB6 was stuck in time.
- TCL can call C code with ease, since the early 90's. Not the case with Electron. And JS really sucks with no standard library. With Electron, the UI can be very taxing, even if they bundle FFMPEG and the like. Tk UI can run on a toaster.
- Yeah, there is C#... but it isn't as snappy and portable TCL/Tk with IronTCL, where it even targets Windows XP. You have JimTCL where it can run on scraps. No Tk, but the language it's close in syntax to TCL, it has networking and TLS support and OFC has damn easy C interops. And if you are a competent programmer, you can see it has some alpha SDL2 bindings. Extend those and you can write a dumb UI with Nuklear or similar in days. Speed? It won't win against other languages on number crunching, but for sure it could be put to drive some machines.
I worked on a startup that was mostly powered by Tcl, the amount of rewriting in C that we had to do between 1999 and 2003, when I left the company among all those dotcom busts, made me no longer pick any language without at least a JIT, for production code.
The founders went on creating OutSystems, with the same concepts but built on top of .NET, they are one of the most successful Portuguese companies to this day, and one of the few VB like development environments for the Web.
Anything that makes the world worse for car drivers is a huge bonus for The planet.