Comment by dofm
4 hours ago
You may never hear me say this again because I am a gruff cynic who is not sure that hanging out here is good for his psyche, but thank goodness for HN.
While I have been chasing around today, trying to resolve this on different servers, it turns out a three line nginx mitigation (blocking multipart uploads) I wrote in the wee small hours GMT has been quietly protecting a vulnerable machine I could not shut down (from attacks powered by every single mainstream agentic AI tool — Anthropic, OpenAI, Qwen cloud, Baidu, seemingly also AWS, Meta)
Including an hour before the PHP patch actually shipped.
So without the HN discussion I wouldn't have found enough people pointing at possible mitigations and I would have lost a server to this, and I thank everyone who pointed out that second mitigation in particular.
YES!!!!
I absolutely loved being here today.
So many brilliant minds screaming and hating together, trying to survive the tide.
There aren’t many places like this left on the commercial internet, especially in this brave new AI-driven world.
I felt a great disturbance in the Force. But in a good way.
Together, we are strong. May the Force be with us all!
> from attacks powered by every single mainstream agentic AI
Is it so bad? Have you measured the number of attacks?
> So without the HN discussion I wouldn't have found enough people pointing at possible mitigations and I would have lost a server to this
I run a tiny server. I like this server a lot. I only patched an hour after the Debian package was ready and didn't think about it again. Do you mean to imply there's a real chance my little server has been compromised?
[Edit:] My server is quite public too. It hosts four SLDs and many subdomains and receives both human and bot traffic. Only one or two subdomains use PHP, but a reasonable attacker surely tries all of them.
> Is it so bad? Have you measured the number of attacks?
It is not a torrent, no. Dozens up to hundreds, rather than hundreds to thousands.
But they would very likely have succeeded, right? It's a fairly turnkey vulnerability.
> I run a tiny server. I like this server a lot. I only patched an hour after the Debian package was ready and didn't think about it again. Do you mean to imply there's a real chance my little server has been compromised?
Can't say one way or another. Have a look at your log file for POST requests in quick succession for things like
> [Edit:] My server is quite public too. It hosts four SLDs and many subdomains and receives both human and bot traffic. Only one or two subdomains use PHP, but a reasonable attacker surely tries all of them.
All my "obviously PHP app" domains got visited. None of my bespoke app domains (which are not obviously PHP). Though over time, the bespoke domains do get a whole bunch of boring probes for WP vulnerabilities etc., like all websites get.
Thanks (this is a django site) -
From a Google Cloud IP it seems.
> Have a look at your log file
Surely it doesn't take a very sophisticated attacker to clean up after themselves at least a little bit?
4 replies →
HN was my source of truth for mitigating this
Agreed. It made my ancient old man face tear pores by smiling when I saw people organizing in the comments to throw sandbags against the wall while waiting to be rescued.
It’s scary that people haven’t been able to move uphill though.
I mean, this isn't a things-made-in-PHP-are-bad problem.
It's a bit more a "certain C programming techniques are bad" problem.
This is more like log4j, heartbleed etc.
Won't be the last of these.
It was fun being here today. Validated a lot of the live frustrations and quirks and discovery.
It is bizarre to me that servers are considered something to be “lost” in this day and age. They should as borderline ephemeral as us.
By "lost" I mean, it might have been broken, commandeered, exfiltrated.
This is PHP world; it may be a share nothing architecture, but it's full of long-running VMs and very long-running apps.
I think calling private data "lost" if it is accessed is not a misnomer, even if you have backups.
You lost its privacy
Care to share it? Still trying to convince my provider to go this route...
I'd like to express my dissatisfaction that the "big guys" are basically getting free security over getting earlier disclosure.
Just saying "hey blocking multipart uploads can mitigate most of attacks early would save people so much trouble. Of course not every CVE will have workaround but many do and whole secrecy about releasing it is only to benefit of few big tech companies and nobody else
Is it just me, or is it the truly brave that decide to deploy and protect PHP instances? Especially if WordPress or Drupal are involve?
I finished moving the remainder of my stuff off of PHP and WordPress last year and haven't missed it at all. Backwards compatibility was typically pretty bad.
It seemed like maintaining existing working PHP code was always a cycle of: upgrade to newer PHP, that version has breaking changes, fix that code or update to a new WordPress (plus all of the plugins that would inevitably need updating too).
I don't really want to be in the WordPress world anymore (I still maintain a couple alas).
But this isn't about PHP code, really — this is about C code (with less-than-ideal exception handler patterns). This problem is more Heartbleed than WP vulnerability.
I quite like modern PHP, I don't really think it's necessarily any less secure (exception handler weirdness notwithstanding), and more to the point, I don't think not using it is any kind of automatic improvement in web app security. It's not like server-side JavaScript is intrinsically better.
I am certainly doing some thinking today; I switched off a couple of web servers so I didn't have to think about them while I was working on other stuff, and actually I think they might not get switched back on. But it's not so much that they are running PHP as that they were kind of neglected and it's always worth reconsidering what you need to keep around.
I think I have reached a point where low value stuff is not worth subsidising just for fun.
(I have also reached the point where in future I won't be using distro packages for PHP)
Other runtimes certainly have their vulnerabilities. It is just that it seems like PHP values backwards compatibility less than Node, Python, Java, or C#. It seems like there is better tooling for running mixed versions of Node or Python than what I've experiences with PHP.
And with the security fixes only landing on the relatively recent PHP versions, it means vulnerabilities force you into the PHP upgrade churn.
In nearly every org I've been in or contracted to, the public landing page of the company gets a special place outside of any ops of general technical team, and is instead the work of a marketing group or a "creative" outsourcer who will default to Wordpress on PHP.
This is how it began and should stay, when IT teams started to get involved in this they ran heavy overhead and aren't clued in on creative requirements. Stick to keeping emails and erp servers up.
Seems like most main company sites at places I've worked. If you are lucky, that WP instance is a fully managed service where they proactively deal with keeping it on the latest PHP, WordPress, and plug-in versions.
I’ve been to a few places like this and managed to get rid of the Wordpress infestation.
[flagged]
Please don't cross into personal attack.
https://news.ycombinator.com/newsguidelines.html
There's no personal attack here.
Unless your position that any statement whatsoever made in the second person that isn't a compliment is then an attack?
San Francisco is a magical place where "you are doing X wrong" without slathering on six paragraphs of fake kindness constitutes some kind of personal attack
4 replies →
Well, OK then:
1) you're being rude, just stop it and get a grip
2) actually, this particular mitigation is very clearly stopping all the unusual patterns of POST request attempts I can see (it's not usually a busy server)
3) I know it is not the only mitigation that is necessary
4) … so it isn't the only mitigation I have applied
5) I am not working under a false sense of security, and I know it is temporary
6) I am not an idiot, and I am working on patching this machine, but this particular server has a unique and complicated configuration and needs a bespoke build
7) I have been prioritising much more serious risks for other people, all day, for no pay
8) again, to reiterate, you're rude.
Please don't respond to a bad comment by breaking the site guidelines yourself. That only makes things worse.
https://news.ycombinator.com/newsguidelines.html
2 replies →
[dead]
You would have been fine without HN, just use AI
I have a Muse Glimmer session helping me keep track of what I am doing, but I don't think any cloud AI would have been as helpful to warn me about something I didn't know and orient me towards possible fixes as other actual people.
Nice try, Diddario...