It's 5:20 AM here in Europe and I just rotated a bunch of keys.
Only to read,
> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.
So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.
I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
Same situation. Frankly unbelievable - how do you not lead with this? You can't tell customers to rotate their keys and not inform them in the same breath them that actually, environment variable updates currently cannot deploy.
They're actively firefighting an incident, and are likely going to make some mistakes along the way. In this case, they've erred on the side of security over availability, which in some cases is the better outcome. I know not in all, but some.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
You may think that sounds like a very nice salary but $4MM annual gross is a very small business. I can’t imagine how Railway would be in business if that were considered a large customer.
300K ARR means you get 300K in annual recurring revenue.
Subtract costs for employees. Which, given current economy and inflation, that gets you a founder scraping on their savings, one or two other people (remember that if you're paying someone unless it's totally offshored through a house which has it's own inflation, you have other factors) and then your own taxes... yeah... much more than that and you are in the black.
As a general factor I keep in mind, when I worked at a 'normal' but toxic place before I was an actual software engineer, my manager's rules were something like '40 percent L2R' (might have been lower) for in-house employees and 70-
80% L2R for contractors.
IOW, for every dollar billed for an actual employee, we paid the employee 40 cents. For every dollar billed for a contractor, we paid them 70-80 cents.
I've heard similar figures from others (e.x. 'in house employee has a real cost of 1.5-2x their actual pay').
And yes it sucks and yes the structure at least here in the US is a factor.
Yup, got my OpenRouter key. They managed to run up a whole $2 on Opus 5.5 and Fable. Well, time for a new hosting provider I guess...
Incidentally, I'm testing Muse as an email triage agent today - it buried "oh also, railway and anthropic say yours keys have been leaked and you need to rotate asap" under 50 routine deploy / admin emails notifications. Oh to live through such an AI powered software renaissance!
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
Brother do not cancel anything for these people, they do not care about you, it's their fault for not considering bus factor in their org chart. Why can't the person above you handle that?
Thanks, yes I will. Left without the laptop(s). Let the chips fall where they lay. If they need me back so urgently, they will have to go through the full cancellation of booked flights and hotels first. Lets enjoy the time off.
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
> Use environment variables to point to where secrets are located, not to store their values.
Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.
It doesn't sound like they got them via a shell environment, they just hacked into the place that stored them. If the place you store secrets can be hacked into, it doesn't really matter if they are injected into environment variables or not.
Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself.
The security team at my company is working with a frontier lab to scan all our code/systems. They are in a borderline panic to close every possible issue, even innocuous ones that are buried under defense in depth. Whatever they are seeing from our partner has them spooked -- as in "acceptable risk" is no longer a thing anymore.
I fully believe the only sustainable future is that hot patching will need to itself become fully automated. Hot patch now with a minimal allowed amount of line changes to head off the AIs, review it in office hours for auditing and the team is actually awake.
My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.
>We are going back to static htmls for public pages...
God yes, please. I backpack and mountaineer in the PNW and have been trying to research trip ideas up in British Columbia, and as a result I discovered https://bivouac.com.
Here's a website that's still frequently used by a large number of people as a valuable resource, is regularly updated and maintained, and yet looks like it's straight out of 1995.
Every pages loads instantaneously, everything is simple yet packed with detail, and it's such a basic, straightforward, delightfully pleasant experience to just browse through the site. I was so thrilled after just 5 minutes on the site that I threw them $25 for a year's full access.
Sorry for the tangent, but oh man, it's such a refreshingly stark contrast to the rest of the bullshit that fills the web these days.
I think we'll see a reduction in complexity and an increase in transparency, rather than just on-prem.
One medium-sized server can now easily serve the entire banking system of a large country. So you don't need clusters of servers with careful tradeoffs between consistency and availability because of the CAP theorem.
Just get a big honking host, slap Postres on it, and run your application server in a VM on the same host. For backup, export the WAL logs from Postgres onto your favorite off-site blob storage. UI for it? There's Pigsty.
It still makes sense to host this in a cloud provider, just because you probably don't want to deal with actual hardware. And maybe get a second standby server for redundancy.
That's it.
You'll need a bunch of external dependencies for email, 2FA, observability, LLMs, and so on. But they should all be outgoing dependencies that should not allow access back into your system.
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
We have gotten a lot of comments about why didn't we just message the breach sooner when people have noticed compromise with their accounts. I think that crit is fair and justified, and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.
As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
We take the view that we would like to "aviate, navigate, then communicate" when we deal with incidents like these then we prepared communications to the public and all users. While it was ongoing, we were in contact with external partners and vendors to revoke keys (the source of some online chatter) with investigators that required time and coordination.
On the incident side of the house, it's a suite of bad options, we're still in the middle of it making sure customers are safe. But, hopefully this offers a view into our thought process.
Formal postmortem after we fully restore service and complete the investigation. Apologies to all, I personally I feel like I let everyone down.
> and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
From my timezone, this started last night and is ongoing into today. Has this been going for longer than the status page and email indicated? If so, why was notification delayed so long?
I love your product, have an incredible amount of sympathy for you as humans (having been in similar sleep-consuming nightmares) and was willing to forgive the GCP-triggered outage as an understandable teething pain, but sadly this response has really knocked my confidence.
I understand the hard reality of security issues -- "urgently rotate your keys, they're already compromised" is an important message to share -- but "sorry, that will also require your site going down because we've blocked deploys" (presumably as a response to the incident...) is an incredibly frustrating situation to find ourselves in.
Railway / OpenAI: I had a key compromised this morning. It hit our limit, so I revoked the key... and figured, well, we'll have to track down the leak.
Then this security notice came out, telling all of their customers to rotate all their keys... As the guy whose spent the entire evening rotating keys and etc. doing
this is... it's a huge message to send.
The cost and implications go really far.
We have many production services that we've taken offline or disabled because we can't deploy to fix the credentials. because of the railway deploy problem...
Railway was stuck between some pretty nasty tradeoffs. I get that the few hour delay in communication caused some of us more cost, issues, and potential breach... and maybe larger suff.
But I also realize that rolling the keys while things may still be compromised, with deployments blocked, could make thins even more sticky
leaves only very bad options. Also we don't know how much they we're spending time actively stopping the attack. the time and focus could be very much consumed. if it was 5% of exposure and they prevented the 95% exposure.. i'm sure glad they aviated instead of communicated.
We make the best decision we can in these moments.. and they all just suck.
"aviate, navigate, then communicate" only matters if you are the one in control of outcomes.
Customers could have simply revoked keys and not re-launched services with rolled credentials (that from your post, may have been re-stolen), but instead you let this fester.
What a disaster for Railway. I hope you get through this stressful moment, but this should be the end of Railway.
IMO "aviate, navigate, then communicate" does not work in this situation. There is more than one pilot. The moment you detected a breach, hell, even knowing that an attacker was inside your system, you should have pointed to anyone. Hell, even AI, to draft a disclosure with more info coming.
I changed he api key in the env var as openrouter revoke the previous one based on this incident. However, as the redeploy cannot be done in my region, the env cannot be updated and now the product cannot be used. Any command in railway to fix this?
What about buckets? I tried to regenerate my secrets but deploys are paused. I don't know if they actually rotated or not because the old keys seem to still be working! Dear attackers, please don't read or mutate my files while my app is offline :)
I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.
The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.
I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.
Then you have real security incidents that you need to tune into constantly.
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.
Like did someone vibe code too hard, how do env secrets leak for everyone...
This is so fucked up. I was reviewing my account and I just saw this in the logs... Claude says database credentials and bucket credentials go in via env vars so likely they had access to database and bucket contents.
Audit Logs
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
Defending is so much harder than attacking. You don't even have to focus on one target, as you can just mass-AI-script a lot of attacks wherever it's possible, and see what sticks. Even if you did everything right, there's always the social-engineering angle, or a single compromised device, or just accidental bad upgrade through supply chain. If you can mass-spread the seed, and plant something that "calls home" constantly, you can just have an agent that would start the attack the minute it gets infected, and go deeper.
It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.
Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.
> It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!”
This isn't in the article? They do not at all claim "the attacked used AI!".
The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.
> your own red teaming should have found them before the attacker did
Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.
It's a nice saying, but if you did enough things right, the attacker still needs multiple things to line up in order to get anything of value, if you get everything right. No one competent who's doing everything right is leaving open S3 buckets open to the world. Doing some things right means using multiple buckets, using multiple time limited-keys with limited read-only access, using special write-access, not having admin keys on developer laptops, a SIEM system that monitors buckets for traffic going in & out and sending emails (both amount and who it's going to) to the person on call. A firewall (inc IP-address-based blocks so eg China can't access the customer data from China at all). A WAF.
That's not an exaustive list but if the attacker gets one of those "right" once, it is not game over. (They aren't of equal strength, mind you.)
I didn’t actually see that in this post at all. They mention that the end goal of the attackers might be to get to AI inference providers. But I don’t think they are crying “AI was used” here.
We (Railway) make no claims about the tools the attacker used in the above post.
However, from discussions with partners, it resembled the pattern of attacks to secure AI API tokens and credentials. Preliminary forensics tell us that this was the primary goal of their attack. That said, I want to be careful here commenting until we have our formal public post-mortem.
So far I found they accessed my S3 buckets, used my OpenAI key (which OpenAI thankfully detected odd behaviour and revoked the key automatically), did readonly operations on my Valkey instance, etc.
Of course, it was a giant scam that tested for nothing and optimized for how to pass a test.
Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.
Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not
This is exactly why we should own our deployment stacks. Going to push up the launch here as pdt is a railway alternative thttps://github.com/autoidm/pdt for scheduled jobs
AI makes this so much easier to do now, we should all own our own code, and deployment infrastructure (use a cloud instead of someone wrapping aws for you)
No, and this isn't about a supply chain attack on Python libs. This is about running your service(s) on a service that runs in front of a cloud provider. My argument is simply that it's needless extra risk
You do have an argument to be made about vendoring our dependencies, or removing them in general. Not a bad idea
It's 5:20 AM here in Europe and I just rotated a bunch of keys.
Only to read,
> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.
So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.
I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
The customers of my company are angry. It is 9.45 AM here in SE Asia. Things are bad.
Same situation. Frankly unbelievable - how do you not lead with this? You can't tell customers to rotate their keys and not inform them in the same breath them that actually, environment variable updates currently cannot deploy.
They're actively firefighting an incident, and are likely going to make some mistakes along the way. In this case, they've erred on the side of security over availability, which in some cases is the better outcome. I know not in all, but some.
They did.
2 replies →
Yeah, I had to rotate my data store keys ASAP. Now I'm stuck with a broken application. Best option, horrible option.
Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway.
And it’s all just getting started imo.
https://blog.cloudflare.com/emudhra-certificate-compromise
https://www.wiz.io/blog/php2shell-php-rce
I think the pingcap hack might be the biggest of them all, somehow it has gone under the radar.
[0] https://news.ycombinator.com/item?id=50173716
Related ongoing thread:
PingCap TiDB Leaks - https://news.ycombinator.com/item?id=50173716 - Oct 2026 (32 comments)
(It had only spent 3 hours on the frontpage so I re-upped it.)
I guess I picked the wrong week to stop sniffing glue.
2 replies →
I find it wild how much they have downplayed this in their statement. https://www.pingcap.com/trust-hub/security/security-update-o...
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
hmm, I don't think that's true.
3 replies →
That was a single company, the cert problem hit something like 70% of all websites and devices.
Honestly…I’ve never heard of pingcap
Jesus that one is bad
2 replies →
Also GitLab
https://docs.gitlab.com/releases/patches/patch-release-gitla...
Also Shopify, they have been delisting a lot of apps without giving a reason, most lickely a supply chain attacked that spoofed/stol checkout data
Madness..
And about 50+ sev 9+ CVEs for the linux kernel
And litellm too
true =(
"It's just marketing" HN mostly told us
[dead]
Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up:
https://x.com/wickedguro/status/2113015432974405658
Why is a 300k mrr company (his tiwtter bio) running on raleway?
You may think that sounds like a very nice salary but $4MM annual gross is a very small business. I can’t imagine how Railway would be in business if that were considered a large customer.
5 replies →
300K ARR means you get 300K in annual recurring revenue.
Subtract costs for employees. Which, given current economy and inflation, that gets you a founder scraping on their savings, one or two other people (remember that if you're paying someone unless it's totally offshored through a house which has it's own inflation, you have other factors) and then your own taxes... yeah... much more than that and you are in the black.
As a general factor I keep in mind, when I worked at a 'normal' but toxic place before I was an actual software engineer, my manager's rules were something like '40 percent L2R' (might have been lower) for in-house employees and 70- 80% L2R for contractors.
IOW, for every dollar billed for an actual employee, we paid the employee 40 cents. For every dollar billed for a contractor, we paid them 70-80 cents.
I've heard similar figures from others (e.x. 'in house employee has a real cost of 1.5-2x their actual pay').
And yes it sucks and yes the structure at least here in the US is a factor.
2 replies →
Lots of companies run on railway
Why not?
Why are you even commenting? And it's Railway not raleway.
1 reply →
Yup, got my OpenRouter key. They managed to run up a whole $2 on Opus 5.5 and Fable. Well, time for a new hosting provider I guess...
Incidentally, I'm testing Muse as an email triage agent today - it buried "oh also, railway and anthropic say yours keys have been leaked and you need to rotate asap" under 50 routine deploy / admin emails notifications. Oh to live through such an AI powered software renaissance!
Hahaha, you're giving Muse access to a mailbox? Good luck, have fun.
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
Brother do not cancel anything for these people, they do not care about you, it's their fault for not considering bus factor in their org chart. Why can't the person above you handle that?
Prioritize your mental health, go enjoy your vacation.
Thanks, yes I will. Left without the laptop(s). Let the chips fall where they lay. If they need me back so urgently, they will have to go through the full cancellation of booked flights and hotels first. Lets enjoy the time off.
4 replies →
Strongly concur
Not your problem. Enjoy your vacation. Bus factor of 1 is poor planning of leadership.
Either a) your company will reimburse you for your costs incurred and give you a different week off, or b) they don’t need you there that badly.
Take more days off. Today was my first day back after 2.5 weeks.
I say this as a Railway customer, go take your vacation. The only time you should even reconsider is a massive bonus plus more time off for staying.
Correct me if I am wrong - if the API keys are compromised, then a large bill could be racked up?
3 replies →
None of this matters if you zoom out.
Services you were responsible for
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
Environment variables are a terrible place to store secrets.
I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.
The 12 Factor manifesto was 100% wrong on this point.
Use environment variables to point to where secrets are located, not to store their values.
Or we could just use our operating systems for the things we do well, and have a service to send our API calls to that inserts the appropriate token.
I don't have to enter my Mac's password every time I open an app.
> Use environment variables to point to where secrets are located, not to store their values.
Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.
1 reply →
It doesn't sound like they got them via a shell environment, they just hacked into the place that stored them. If the place you store secrets can be hacked into, it doesn't really matter if they are injected into environment variables or not.
How does this help? If your container gets compromised it's trivial to exfiltrate the secrets.
Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself.
3 replies →
systemd makes it easy: https://systemd.io/CREDENTIALS/
I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
The security team at my company is working with a frontier lab to scan all our code/systems. They are in a borderline panic to close every possible issue, even innocuous ones that are buried under defense in depth. Whatever they are seeing from our partner has them spooked -- as in "acceptable risk" is no longer a thing anymore.
I fully believe the only sustainable future is that hot patching will need to itself become fully automated. Hot patch now with a minimal allowed amount of line changes to head off the AIs, review it in office hours for auditing and the team is actually awake.
My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.
> hot patching will need to itself become fully automated.
Great, even less control over the things we use and more power ceded to corporations.
3 replies →
Hot patching is itself a liability
4 replies →
The internet is over. We are going back to static htmls for public pages, and Point to Point VPN bases connections for B2B service provision.
B2C? too risky to publish services online haha.
Static HTML, yeah.
Not sure about VPN. Seems likely to provide a false sense of security.
>We are going back to static htmls for public pages...
God yes, please. I backpack and mountaineer in the PNW and have been trying to research trip ideas up in British Columbia, and as a result I discovered https://bivouac.com.
Here's a website that's still frequently used by a large number of people as a valuable resource, is regularly updated and maintained, and yet looks like it's straight out of 1995.
Every pages loads instantaneously, everything is simple yet packed with detail, and it's such a basic, straightforward, delightfully pleasant experience to just browse through the site. I was so thrilled after just 5 minutes on the site that I threw them $25 for a year's full access.
Sorry for the tangent, but oh man, it's such a refreshingly stark contrast to the rest of the bullshit that fills the web these days.
Still some industries where you cable colocated servers in a shared DC together where neither server has internet access.
1 reply →
[flagged]
It's only just starting. It'll get worse before it gets better.
I think the only way out is to go back to on-prem for everything, like in the 00s.
LAN cables will go up in price soon.
Ha, Cyberpunk 2077 is becoming reality. We're proposing the Blackwall.
1 reply →
"So you want to build your own data center?"
https://blog.railway.com/p/data-center-build-part-one
I think we'll see a reduction in complexity and an increase in transparency, rather than just on-prem.
One medium-sized server can now easily serve the entire banking system of a large country. So you don't need clusters of servers with careful tradeoffs between consistency and availability because of the CAP theorem.
Just get a big honking host, slap Postres on it, and run your application server in a VM on the same host. For backup, export the WAL logs from Postgres onto your favorite off-site blob storage. UI for it? There's Pigsty.
It still makes sense to host this in a cloud provider, just because you probably don't want to deal with actual hardware. And maybe get a second standby server for redundancy.
That's it.
You'll need a bunch of external dependencies for email, 2FA, observability, LLMs, and so on. But they should all be outgoing dependencies that should not allow access back into your system.
2 replies →
How does on-prem help if your product is a SaaS.
6 replies →
surely once there's been enough patches the code will be perfect though?
Railway was not honest about the reason for their last big outage.
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
Hi Angelo from Railway here. This is very much me. https://x.com/ngeloxyz/status/2113077263386239309?s=20
We have gotten a lot of comments about why didn't we just message the breach sooner when people have noticed compromise with their accounts. I think that crit is fair and justified, and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.
As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
We take the view that we would like to "aviate, navigate, then communicate" when we deal with incidents like these then we prepared communications to the public and all users. While it was ongoing, we were in contact with external partners and vendors to revoke keys (the source of some online chatter) with investigators that required time and coordination.
On the incident side of the house, it's a suite of bad options, we're still in the middle of it making sure customers are safe. But, hopefully this offers a view into our thought process.
Formal postmortem after we fully restore service and complete the investigation. Apologies to all, I personally I feel like I let everyone down.
> and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
From my timezone, this started last night and is ongoing into today. Has this been going for longer than the status page and email indicated? If so, why was notification delayed so long?
I love your product, have an incredible amount of sympathy for you as humans (having been in similar sleep-consuming nightmares) and was willing to forgive the GCP-triggered outage as an understandable teething pain, but sadly this response has really knocked my confidence.
I understand the hard reality of security issues -- "urgently rotate your keys, they're already compromised" is an important message to share -- but "sorry, that will also require your site going down because we've blocked deploys" (presumably as a response to the incident...) is an incredibly frustrating situation to find ourselves in.
You know... I get it.
"aviate, navigate, then communicate" only matters if you are the one in control of outcomes.
Customers could have simply revoked keys and not re-launched services with rolled credentials (that from your post, may have been re-stolen), but instead you let this fester.
What a disaster for Railway. I hope you get through this stressful moment, but this should be the end of Railway.
IMO "aviate, navigate, then communicate" does not work in this situation. There is more than one pilot. The moment you detected a breach, hell, even knowing that an attacker was inside your system, you should have pointed to anyone. Hell, even AI, to draft a disclosure with more info coming.
Anyone can help?
I changed he api key in the env var as openrouter revoke the previous one based on this incident. However, as the redeploy cannot be done in my region, the env cannot be updated and now the product cannot be used. Any command in railway to fix this?
How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...
What about buckets? I tried to regenerate my secrets but deploys are paused. I don't know if they actually rotated or not because the old keys seem to still be working! Dear attackers, please don't read or mutate my files while my app is offline :)
yeah .. this is the suck
I've already taken my applications offline and will be migrating to a new host...
I'm curious where you decided to go
1 reply →
In general, the new host will have its own problems, otherwise you would've used them to begin with.
Yep. What an incredibly frustrating notice. I like Railway and I guess their hands are tied here, but wow.
I rotated mine. If you just change the keys and hit deploy it works
It appears to work in the UI, but go test your product. Mine didn't receive the new keys.
Saw someone on Twitter indicating that they did this, and new keys got compromised.
1 reply →
Jeez, what a rough past 24 hours for all security folk. Hang in there
It's going to be a rough decade.
I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.
The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.
18 replies →
It already _is_ a rough decade. I have hated the 20s so damn much.
Yeah, the future of our jobs in this industry... for the type of work we do. are concerning. Explosure has whole new meaning
It feels like the only solution is going to be proactively running your own AI agents looking for patches and applying them.
6 replies →
[dead]
I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.
Then you have real security incidents that you need to tune into constantly.
job security!
Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?
Not a bad idea
It is… for their business.
sighs
Thankfully I'm only using Railway for a hobby project (aptly, for a real-time transit dashboard) but still
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.
Like did someone vibe code too hard, how do env secrets leak for everyone...
Easy, all the bad guys need to do is access whatever system stores the decryption keys.
It's time for competent programmers to get rehired to help patch massive exploits that are about to unfold before us.
This is so fucked up. I was reviewing my account and I just saw this in the logs... Claude says database credentials and bucket credentials go in via env vars so likely they had access to database and bucket contents.
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
Defending is so much harder than attacking. You don't even have to focus on one target, as you can just mass-AI-script a lot of attacks wherever it's possible, and see what sticks. Even if you did everything right, there's always the social-engineering angle, or a single compromised device, or just accidental bad upgrade through supply chain. If you can mass-spread the seed, and plant something that "calls home" constantly, you can just have an agent that would start the attack the minute it gets infected, and go deeper.
It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.
Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.
> It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!”
This isn't in the article? They do not at all claim "the attacked used AI!".
The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.
> your own red teaming should have found them before the attacker did
Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.
It's a nice saying, but if you did enough things right, the attacker still needs multiple things to line up in order to get anything of value, if you get everything right. No one competent who's doing everything right is leaving open S3 buckets open to the world. Doing some things right means using multiple buckets, using multiple time limited-keys with limited read-only access, using special write-access, not having admin keys on developer laptops, a SIEM system that monitors buckets for traffic going in & out and sending emails (both amount and who it's going to) to the person on call. A firewall (inc IP-address-based blocks so eg China can't access the customer data from China at all). A WAF.
That's not an exaustive list but if the attacker gets one of those "right" once, it is not game over. (They aren't of equal strength, mind you.)
Unless they edited the post, they never said this. You should delete this; it's false information.
I didn’t actually see that in this post at all. They mention that the end goal of the attackers might be to get to AI inference providers. But I don’t think they are crying “AI was used” here.
Or did I miss something?
We (Railway) make no claims about the tools the attacker used in the above post.
However, from discussions with partners, it resembled the pattern of attacks to secure AI API tokens and credentials. Preliminary forensics tell us that this was the primary goal of their attack. That said, I want to be careful here commenting until we have our formal public post-mortem.
This one doesn’t?
meanwhile on railway.com:
Ship software peacefully
I'm rotating production secrets all night... great. I confirmed that they've actually used many of my secret keys already.
Which secrets were they using?
So far I found they accessed my S3 buckets, used my OpenAI key (which OpenAI thankfully detected odd behaviour and revoked the key automatically), did readonly operations on my Valkey instance, etc.
The secret ones
i revoked my api keys, changed with new ones but guess what? railway does not allow deployments at the moment, so my app is down.
The past two days have been awful as someone who uses Railway.
Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36
Related: https://news.ycombinator.com/item?id=50177413
If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.
someone managed to get the environment variables AND the encryption keys for those? i feel for the secops engineers over there right now. brutal
Is this our new AI daily reality?
script kiddies now have access to frontier models. bad
[dead]
companies probably have php and Python applications exposed to the web and nobody knows.
the speed of irresponsible deployments is infinite now, by design
I’m lucky I moved away from it to host my own db with coolify
That’s coming for all platforms unfortunately
Take coolify admin off the web, put it on a private address only accessible via tailscale
Aw shit. I just got done patching our PHP servers, now I gotta do the Railway too? I swear I'm gonna wake up to a Vercel SNAFU.
We are not manifesting this energy
Yeah I’m not sure what happened but there was a huge bot swarm the last few days
I had an open gitea host pulled into some botnet
Luckily digitaloceam sent an alert.
Actually I will move my projects away from Railway. With AI what it is today, no reason to shy away from hosting on actual VPS.
What is the best VPS to use today? Is it still hetzner or ovh?
Ugh. Love Railway.
Damn I guess all those l33t hiring practices were for nothing
Of course, it was a giant scam that tested for nothing and optimized for how to pass a test.
Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.
Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not
the worst service I've signed up for has to be railway. What a terrible product.
Fuck, I thought this was about trains
[flagged]
[dead]
>We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.
Is that an accusation? As in, major AI players taking out competitors ?
I don’t think that’s what’s implied here. Possibly they are trying to get API tokens of providers to distill their own models.
This is exactly why we should own our deployment stacks. Going to push up the launch here as pdt is a railway alternative thttps://github.com/autoidm/pdt for scheduled jobs
AI makes this so much easier to do now, we should all own our own code, and deployment infrastructure (use a cloud instead of someone wrapping aws for you)
Do you own argcomplete, backoff, certifi, cffi, charset-normalizer, cryptography, google-auth, google-auth-oauthlib, idna, markdown-it-py, mdurl, msal, oauthlib, pyasn1, pyasn1-modules, pycparser, pygments, pyjwt, python-dotenv, pyyaml, requests, requests-oauthlib, rich, rich-argparse, shellingham, and urllib3?
No, and this isn't about a supply chain attack on Python libs. This is about running your service(s) on a service that runs in front of a cloud provider. My argument is simply that it's needless extra risk
You do have an argument to be made about vendoring our dependencies, or removing them in general. Not a bad idea