Comment by shoddydoordesk
1 day ago
> it suddenly ballooned in size in April 2025 after its operators breached a TotoLink router firmware update server and infected approximately 100,000 devices
This is scary. Everyone lauds open source projects like OpenWRT but... who is watching their servers?
I imagine you can't run an army of security people on donations and a shoestring budget. Does OpenWRT use digital signing to mitigate this?
Why, OpenWRT firmware and packages are both signed, of course. You can manually and independently check the image signature before flashing an update.
The build infrastructure is, of course, a juicy target: infect the artifact after building but before signing, and pwn millions of boxes before this is detected.
This is why bit-perfect reproducible builds are so important. OpenWRT in particular have that: https://openwrt.org/docs/guide-developer/security#reproducib...
This exchange is somewhat hilarious. Oh how on earth do we keep things safe and secure if everyone can see the code and verify what it does! Who would keep us safe if we turn our backs to unverifiable, unvetted, unprofitable security fixes, by for-profit companies!
The biggest joke is most of the proprietary routers both consumer and enterprise grade often are running some old outdated version of custom tuned openwrt lol, this goes for tp-link, and everyone else almost.
> how on earth do we keep things safe and secure if everyone can see the code and verify what it does!
That's not always the silver bullet you seem to think it is. Have you ever tried to build something like Chromium, Firefox, or LLVM yourself? It's not realistic to do that on a mid tier let alone low end device.
Even when you go to the trouble of getting a local build set up, more often than not the build system immediately attempts to download opaque binary blobs of uncertain provenance. Try building some common pieces of software in a network isolated environment and you will likely be surprised at how poorly it goes.
If projects actually took this stuff seriously then you'd be able to bootstrap from a sectorlisp and pure human readable source code without any binary blobs or network access involved. Instead we have the abomination that is npm.
3 replies →
> You can manually and independently check the image signature before flashing an update.
Of course you can. You can also read the ToS before clicking accept, but who does that?
I'm sure there are dozens of us.
People who don't want to find themselves inadvertently participating in a botnet.
Bit-Reproducible infrastructure could also result in some of the wildest build distribution architectures if you think about it. You could publish sources and have people register like in APT mirrors to provide builds, and at the end of the day, the build from the largest bit-equal group is published.
I do see the Tor-Issue - a botnet or a well-supplied malicious actor could just flood it. And if you flip it - if you'd need agreement about the build output, it could also be poisoned with enough nodes to prevent releases for a critical security issue. I agree, I don't solve all supply chain issues in one comment :)
But that in turn could be helped with reputation. Maybe a node needs to supply 6 months of perfect builds - for testing as well - to become eligible. Which would be defeated by patience, but what isn't? It'd just have to be more annoying to breach the distributed build infrastructure than to plant a malicious developer.
This combination of reproducible, deterministic builds, tests across a number of probably-trustworthy sources is quite interesting, as it allows very heavy decentralization. I could just run an old laptop or two here to support. And then come compromise hundreds of these all across the world.
Sounds overly complex and completely unnecessary, like some kind of blockchain/defi scheme shoehorned onto distributed builds.
Reproducible isn't quite enough, you also need bootstrap from almost-zero binaries.
https://bootstrappable.org/
>It'd just have to be more annoying to breach the distributed build infrastructure than to plant a malicious developer.
It really wouldn't. You don't even need a powerful build server since you can mirror whatever someone else built. You can also buy / hack nodes of existing trusted people.
> Try building some common pieces of software in a network isolated environment and you will likely be surprised at how poorly it goes.
I have yet to experience a straight shot install or build of anything in an air gapped environment. Always need to hack things to make it work.
The distribution system you're describing exists and has been in use for decades. You just distribute the build using bittorrent.
4 replies →
I don't follow.
> run an army of security people
Do you think these private companies do this? They don't. They pay as little as humanly possible to cover their ass.
Botnets comprised of compromised routers is common and commercial/consumer routers are a far juicer target than openwrt.
> They pay as little as humanly possible to cover their ass.
They probably spend more on the team who ends up writing the "We take your security very seriously" breach notification message than they do on "security people". At least until then get forced into brand-name external Cyber Security Consultants to "investigate" their breach and work out who they can plausibly blame it on that's not part of the C suite.
> They pay as little as humanly possible to cover their ass.
It’s probably helpful that open source teams aren’t hampered by standards and 20 year outdated audit processes either.
This is exactly why OpenWRT has no unattended updates by default )
You are dismissing the seriousness of this. Their package manager is widely used. One would only need to compromise their build servers to wreak havoc.
Didn't they have a vulnerability in their firmware download tool like a minute ago?
The difference between OpenWRT and Linux distros is the amount of testing and visibility. OpenWRT is loaded on to residential devices and forgotten about, it doesn't have professional sysadmins babysitting it 24/7.
Remember the xz backdoor was only discovered because some autist at Microsoft noticed a microsecond difference in performance testing.
I'm confused why you're so honed in on OpenWRT as a third-party open-source project here when the vulnerability you quoted (TotoLink) was the official firmware update server of a brand of devices.
Is it "scary" to think about OpenWRT potentially getting hacked? If you get scared by theoretical possibilities in software, sure. Is it relevant? Not exactly. Are companies' official servers more secure than an open-source project's servers? In this case, apparently not.
2 replies →
The post is nothing more than "but what about security" meant to deflect away from the discussion at hand and towards OpenWRT
I recently had some issues getting one of our embeded devices connect through passive ftp. Because the exact same device worked at a different site I knew it wasn't the device or it's settings. Long story short, it turned out the problematic site hadn't been updating its routers which meant they couldn't VPN passive FTP traffic. Anyway, we have literal thousands of those routers maintained by hundreds of different companies, who are mainly there to maintain the actual mechanical equipment and not the network. Turned out the site where the technicians updated things weren't in the majority.
I'm in the process of getting the business to implement better security, and it's going better than you might expect. If it wasn't because having a plan for how to update your OT security is required to meet EU compliance, however, I doubt we would've done anything beyond making sure we could do passive FTP when it was needed.
As an example, there is still no plans to deal with the OT which we know has build in hardware backdoors from the manufactures. Wnich is around 70% of our dataloggers, but the EU has no compliance rules on that...
As always, hundreds watch the open repositories, maybe one watches a company's build servers, if they're lucky. :-)
Hundreds watch, but how closely?
Plenty of stories of fairly major projects having evil commits snuck in that remain for months.
Name a few.
3 replies →
Digital signing wouldn't defend you from a compromised build server.
Reproducible Builds and multiple distributed builders would though.
https://reproducible-builds.org/
What in that act says OpenWrt would be made illegal? If anything, OpenWrt would roll out automated security updates for a supported branched release to comply with these regulations.
Also, if you actually read it, there are exceptions for open source software!
OP claims almost daily that some benign thing is actually illegal but practically never provides any useful proof when asked.
(please prove me wrong, Alex)