Security Disclosure
blog.railway.com
blog.railway.com
Only to read,
> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.
So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.
I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
> What you should do: As such, we strongly recommend that you immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs for those services for unusual activity.
> All deployments are paused while we investigate. Until builds are restored, you will not be able to roll out new versions. We apologize for this inconvenience.
They lead with rotating keys.
Next, I did not realize that this meant not being able to restart an existing deployment to bring up new env vars. They also speak of builds. I don’t need a new build. I need my existing deployment to work.
It was not at all clear ‘paused’ mean this; maybe I should have realized, but I had interpreted it as doing new deployments or builds. New services. Code changes. New builds as they even say. It’s fully reasonable to pause that in this sort of situation and so I had interpreted their statement that way.
If there is a clear impact from following advice — ‘do what we recommend and your service will not function’ — those who provide that advice need to make that clear.
You’re better off waiting and monitoring until you’re sure you can get back online if your service supports any kind of revenue.
Thankfully, they clarified they're in Europe, so GDPR prevents them from ignoring known leaked credentials, lest they have that to deal with, on top of the revenue loss they'll make when all of their customers eventually find out: "they ran leaked credentials knowing our data was at risk, but decided to suck it and see".
It's a key to your creditcard. Your revenue will be wiped out if you get the AWS bill after hackers booted hundreds of GPU instances under your account, moved terabytes in your data-transfer budgetm, burned tens of thousands of CPU-minutes in github actions under your creditcard, or used thousands of $s worth of tokens with your AI provider.
Naturally "a stolen key" here means that the key has access to something worth stealing. And my point was that there's far more worth stealing than your users' private data which many companies don't value that much.
If you architected things correctly, your root DB key should be useless without an additional compromise of an independent system.
From Margin Call:
> It's just money; it's made up. Pieces of paper with pictures on it so we don't have to kill each other just to get something to eat.
Even if that were not absolutely wrong¹ these days², if your risk is mitigated by being one of millions the damage done if you are hit could still be so high that you can't afford to³ knowingly take that risk.
--------
[1] if the credentials can be easily linked to the relevant account, cycling through them to see what useful stuff can be accessed using which is not difficult to quickly automate
[2] if it were ever right
[3] or your customers can't afford for you to
With cloud credentials I understand that they are able to create machines, but with api keys... what they are able to do to take profit from?
We reached a state where whatever is on the web that cannot be trusted. Users worry, privacy and secrets are compromised on a huuuge scale, while the creators who did this way are saying 'this is normal business'. That's not right! Privacy is sacrificed for developer's convenience.
The only alternative I can think of is "own and operate all of your site's hardware on physical premises to which you have sole access", and even that's not perfect.
Are you arguing against hosting/cloud/datacenter services in general? Or is there a specific technique you would prefer people use to secure their keys in a non-owned environment?
Right now I use ECS and coolify in some cases.
My point was just use the cloud directly want their tools instead of something like this.
Making a security incident recovery a paid feature is a new low. I guess this is how they plan to recoup the money they lost from the incident.
I think if redeploying was not possible, I would have preferred they automatically hid or revoked keys, or the sites or services were automatically turned offline. I suppose this may not be everyone's preference. I know for my site, a simple 'This is hosted by Railway and we have an outage' (and for our service tier, all REST service requests 500-ing, or 404-ing, or...) would be far preferable to the site being online, appearing to work, and functionality just failing for people due to the keys being revoked. Something clear is preferable to something hidden. I worry that to our users, who don't know our host, when the service is down but we cannot change the deployment, it makes us look like we're unreliable or flaky.
I like Railway but have been burned a little by them. FWIW, while not using them for one service where I felt really let down, one other service remained and just a week ago I recommended them again... ouch!
I feel they've grown fast but may not have the infrastructure fully fleshed out. A mass 'disable all services' button probably would not exist until the day they need one.
Last job had a ton of .Net applications once when we did multiple Kubernetes redeploys, CPU spiked for 4 minutes on a bunch of nodes causing problems.
And it’s all just getting started imo.
> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out. Eighty-eight consecutive exports, not one detection event.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
hmm, I don't think that's true.
PingCap TiDB Leaks - https://news.ycombinator.com/item?id=50173716 - Oct 2026 (32 comments)
(It had only spent 3 hours on the frontpage so I re-upped it.)
I do not thing there is (any) security in cyberspace, and this is exposed big now with the use of AI.
I dont understand using any PaaS that essentially functions by arbitraging aws to people that are scared of an aws dashboard at a 3x markup.
Seems like a decent platform for toy projects not for enterprise.
tripadvisor and bilt, maybe?
It's pretty common to have "toy" infrastructure in a non-tech business
Of course "uses" doesn't mean they run critical services.
Median annual revenue for a small business (or just business) is a paltry $280k.
Subtract costs for employees. Which, given current economy and inflation, that gets you a founder scraping on their savings, one or two other people (remember that if you're paying someone unless it's totally offshored through a house which has it's own inflation, you have other factors) and then your own taxes... yeah... much more than that and you are in the black.
As a general factor I keep in mind, when I worked at a 'normal' but toxic place before I was an actual software engineer, my manager's rules were something like '40 percent L2R' (might have been lower) for in-house employees and 70- 80% L2R for contractors.
IOW, for every dollar billed for an actual employee, we paid the employee 40 cents. For every dollar billed for a contractor, we paid them 70-80 cents.
I've heard similar figures from others (e.x. 'in house employee has a real cost of 1.5-2x their actual pay').
And yes it sucks and yes the structure at least here in the US is a factor.
Technical people should understand they're going to get their ass beat by non-technical people right now
You don't need to know how to code to write Rust anymore
Stop pearl clutching and start problem solving
Or:
The implementation no longer matters
People really, really need to understand that
The cynical take for the state of the industry, which is maybe more aligned with Railway's statements, is that even people who do know how to code can get hacked, given how good the LLMs are at attacking targets now.
I imagine both camps would agree that without a red team that's leveraging the frontier, you're much more likely to fall victim to an attack.
> Bus factor of 1
Chef Finway: [laughing awkwardly] Okay...
Rob: Derek, why do you care about these people? They don't care about you. None of them. They don't even know you because you haven't shown them. Every day, you'll wake up, and there'll be less of you. You live your life for them, and they don't even see you. You don't even see yourself.
[long pause; Derek's tight smile has slowly faded into a look of sorrow and regret]
Rob: We don't get a lot of things to really care about.
I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.
The 12 Factor manifesto was 100% wrong on this point.
Use environment variables to point to where secrets are located, not to store their values.
I'm not sure I get the difference - the issue I would have thought was having the system code have access to the secrets themselves? Env file vs env vars injected at runtime seems the same there - either can be exfiltrated. The difference would be having it injected when used.
Is the problem with files about what happens outside that, being committed, etc?
The headline here is that there's no reason to believe "environment variables" were the reason this data was breached; presumably, the database those variables came from was what actually got hit.
These problems were all solved a long time ago.
We have a couple of VMs that are not mission critical, and they don't hold sensitive data. We just inject the secrets into the runtime. Access to those machines is restricted to a single IP, there's not much more protection in front of them.
A meaningful attack on one of those machines would require a LOT of effort, and almost nothing in return. Some systems don't NEED to have government level security, as a developer it's my job to provide the best solution for each scenario.
(IE, I wouldn't put 12 security cameras and an armed guard around a fruit stand, but I might around a jewelry store)
Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.
Oh no, guess they'll have to hire developers that can adapt!
Docker in 2026 is a perfect example of this (long lasting habits due to familiarity, not best practice).
In addition to the developers not interested in using AI to augment their skills and code.
I don't have to enter my Mac's password every time I open an app.
Incidentally, I'm testing Muse as an email triage agent today - it buried "oh also, railway and anthropic say yours keys have been leaked and you need to rotate asap" under 50 routine deploy / admin emails notifications. Oh to live through such an AI powered software renaissance!
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
LAN cables will go up in price soon.
Sometimes I do miss owning the hardware to run a thing.
What I don't miss was having all of those appliances run a different base distro of Linux, each with varying levels of update support, while the security team keeps reading security scans to me.
One medium-sized server can now easily serve the entire banking system of a large country. So you don't need clusters of servers with careful tradeoffs between consistency and availability because of the CAP theorem.
Just get a big honking host, slap Postres on it, and run your application server in a VM on the same host. For backup, export the WAL logs from Postgres onto your favorite off-site blob storage. UI for it? There's Pigsty.
It still makes sense to host this in a cloud provider, just because you probably don't want to deal with actual hardware. And maybe get a second standby server for redundancy.
That's it.
You'll need a bunch of external dependencies for email, 2FA, observability, LLMs, and so on. But they should all be outgoing dependencies that should not allow access back into your system.
Do you think you can hit these load levels? For compute, a 64-core CPU can saturate a 10G network with simple template rendering. If the code is written in something reasonably fast (Go, Rust, Java).
Of course, if you're on a scale of Amazon or Stripe, you'll need more than that.
Postgres fanboys are insufferable.
My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.
Great, even less control over the things we use and more power ceded to corporations.
The barbarians are at the gates and they're interested in laundering your compute and network for other attacks, scams, and things like transferring CSAM.
How much time do you need to audit? You might have an hour - for now.
The actual solution largely involves the same technologies but in reverse: leveraging AI for live security monitoring, better correctness validation and security testing before code goes live, etc. And of course more pressure to use battle-tested widely-adopted libraries and systems, even though of course that comes with its own tradeoffs.
Importantly, as an engineer, when someone tells you something has liabilities or tradeoffs, the response is not "Everything is a liability, existing is a liability"; that is unprofessional. The appropriate response is to engage with them on these tradeoffs and working through the best ones, because that's your job. Otherwise defer to security engineers who are ready to back up their arguments. It could be as simple as following up with, "But do you think those liabilities outweigh the benefits of hot patching without proper review? What are some of the main liabilities you see, and what methods/technologies do you think properly address them?"
AI is quite good at finding vulnerabilities, but it's been rough for patching them. The old adage "defenders have to be right every time, attackers only have to be right once" applies to AI security patches.
B2C? too risky to publish services online haha.
God yes, please. I backpack and mountaineer in the PNW and have been trying to research trip ideas up in British Columbia, and as a result I discovered https://bivouac.com.
Here's a website that's still frequently used by a large number of people as a valuable resource, is regularly updated and maintained, and yet looks like it's straight out of 1995.
Every pages loads instantaneously, everything is simple yet packed with detail, and it's such a basic, straightforward, delightfully pleasant experience to just browse through the site. I was so thrilled after just 5 minutes on the site that I threw them $25 for a year's full access.
Sorry for the tangent, but oh man, it's such a refreshingly stark contrast to the rest of the bullshit that fills the web these days.
Not sure about VPN. Seems likely to provide a false sense of security.
We have gotten a lot of comments about why didn't we just message the breach sooner when people have noticed compromise with their accounts. I think that crit is fair and justified, and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.
As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
We take the view that we would like to "aviate, navigate, then communicate" when we deal with incidents like these then we prepared communications to the public and all users. While it was ongoing, we were in contact with external partners and vendors to revoke keys (the source of some online chatter) with investigators that required time and coordination.
On the incident side of the house, it's a suite of bad options, we're still in the middle of it making sure customers are safe. But, hopefully this offers a view into our thought process.
Formal postmortem after we fully restore service and complete the investigation. Apologies to all, I personally I feel like I let everyone down.
Customers could have simply revoked keys and not re-launched services with rolled credentials (that from your post, may have been re-stolen), but instead you let this fester.
What a disaster for Railway. I hope you get through this stressful moment, but this should be the end of Railway.
From my timezone, this started last night and is ongoing into today. Has this been going for longer than the status page and email indicated? If so, why was notification delayed so long?
I love your product, have an incredible amount of sympathy for you as humans (having been in similar sleep-consuming nightmares) and was willing to forgive the GCP-triggered outage as an understandable teething pain, but sadly this response has really knocked my confidence.
I understand the hard reality of security issues -- "urgently rotate your keys, they're already compromised" is an important message to share -- but "sorry, that will also require your site going down because we've blocked deploys" (presumably as a response to the incident...) is an incredibly frustrating situation to find ourselves in.
I would rather have known earlier that I should put a notice on my site, or my own social media, or notify my own customers ASAP and disable some functionality (or make my own choice about porting somewhere else as a mitigation) than get caught out like this after the horse has bolted and it turns out you've been desperately trying to get it back into the paddock for days.
Railway / OpenAI: I had a key compromised this morning. It hit our limit, so I revoked the key... and figured, well, we'll have to track down the leak.
Then this security notice came out, telling all of their customers to rotate all their keys... As the guy whose spent the entire evening rotating keys and etc. doing
this is... it's a huge message to send.
The cost and implications go really far.
We have many production services that we've taken offline or disabled because we can't deploy to fix the credentials. because of the railway deploy problem...
Railway was stuck between some pretty nasty tradeoffs. I get that the few hour delay in communication caused some of us more cost, issues, and potential breach... and maybe larger suff.
But I also realize that rolling the keys while things may still be compromised, with deployments blocked, could make thins even more sticky
leaves only very bad options. Also we don't know how much they we're spending time actively stopping the attack. the time and focus could be very much consumed. if it was 5% of exposure and they prevented the 95% exposure.. i'm sure glad they aviated instead of communicated.
We make the best decision we can in these moments.. and they all just suck.The options railway has given us are "rotate your keys and bring down your service" or "keep your service up and let the exposed keys remain at risk." Kind of a crap position, no? Mistakes are going to happen, but when the suggested mitigations are to stab yourself in the leg or stab yourself in the ass, it's hard to be sympathetic.
I have to imagine quite a few users are asking their agents to port their stacks over to alternate providers like I am. The cost of switching has never been cheaper thanks to AI.
> As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
Once you realize there is compromise, you message accordingly, as soon as you are able. If you can't immediately determine exactly what subset of customers were affected, this means you have to tell everyone. Then you provide follow-up updates to fill in gaps as you develop your understanding of scope+impact and work on remediation.
What you should never do is hold off on reaching out to possibly affected stakeholders that they may be affected by a compromise because you still have uncertainties about current status.
A situation like this does not in any way a constitute a dilemma or tough balance regarding when or to whom to do initial disclosure - you do it as soon as you can, to everyone except those you know are not impacted.
IMO Railway's response was evasive, underplayed the incident, and did not reflect the severity of this failure. As a response we pulled all our big projects off Railway.
All I want is that my data is reliably secure on some iron somewhere, without a million vibe coding features or whatever they are trying to build, distracting from basic security.
Anything hosted on Railway is at a huge risk, IMO.
https://blog.railway.com/p/incident-report-march-30-2026-acc...
I too in my org are having to navigate the fallout of this and begin migrations to another platform. Once is okay, but this isn't their first big outage this year.
Infra providers have only their name and stats back themselves up with, let's be honest from a tooling perspective it's largely all same same. Real shame, has been easy to use Railway up till now.
Interestingly enough, from the logs they were only using Anthropic as the provider, with a Go HTTP client. Pretty sure the main target was LLM API keys...
The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.
I don't have experience with proof based tools, so I'm not sure what's feasible.
I can imagine prompting an LLM to build provably correct software. Aren't we already doing something similar already when we remind it to write unit tests and account for all edge cases?
Actually writing provably correct software from scratch sounds terribly tedious, which is why I've never explored it. But maybe an LLM could make that task more approachable.
It gave me something which looked pretty legit. Of course, the generated output very likely to be subtly wrong, but as starting input to a formal checked specification/proof process it seemed like a good start.
I wrote some UTF-8 based string handling software back in the day. It doesn't seem unreasonable to write up a set of requirements and edge cases and then have a formal mechanism prove correctness for common UTF-8 string manipulation operations.
[PEAK HN ON] Now if we could just write provably correct exception handling, we could solve one of the two PHP vulns from today. [PEAK HN OFF]
I should check it against my email address validator that would be a nice benchmark. It's interesting that there are people that are using email addresses that are not actually valid according to the standard.
We can have an LLM enumerate all those edge cases, and we can assemble them into a spec. Then we can integrate that spec into a proof.
And because the spec is functional rather than expressive, we can produce copyright-free, provably correct reference implementations whose correctness is as strong as our spec and our proof process.
This reference implementation would then be available for use by a language like PHP, or perhaps an LLM generating a vibe-coded dependency.
Over time, then, the accumulation of such proven reference works grows. We'll never prove all the application code out there correct, but incremental progress is possible.
I think this, the supply chain attacks and binary reverse engineering will cause us to go back to thin clients, html over the wire and vibe coded libs instead of external open source deps.
In the near to medium term, the current open source regime has issues because bugs have become extremely discoverable and easy to exploit. But I am skeptical that obscurity will provide vibe coded deps sufficient protection over the long term. As soon as the source code for those vibe-coded deps leaks, you are screwed.
The value of 'code written before 2024' is going to be substantially higher than of code written afterwards.
Sort of like a 'low background steel', a comparison someone made on HN before which I think was pretty good.
Pre-2024 code is full of human-written vulnerabilities.
Prompting the LLM to "look for vulnerabilities in PHP's multipart upload handling code" is going to find stuff that "look for vulnerabilities in PHP" won't.
Those can leak / be stolen of course, but a bunch of my clients php apis survived as without a cert they’re told to fuck right off.
It does make their client onboarding a bit harder but it’s a trade off which seems to have been the right one.
Then you have real security incidents that you need to tune into constantly.
Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.
That's not an exaustive list but if the attacker gets one of those "right" once, it is not game over. (They aren't of equal strength, mind you.)
It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.
Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.
Or did I miss something?
However, from discussions with partners, it resembled the pattern of attacks to secure AI API tokens and credentials. Preliminary forensics tell us that this was the primary goal of their attack. That said, I want to be careful here commenting until we have our formal public post-mortem.
This isn't in the article? They do not at all claim "the attacked used AI!".
The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.
Rotated all our secrets but since deploys are paused our app is down.
I'm unimpressed.
All my apps are on Railway, deployments has been down since yesterday morning (24 hours), feels nuts that they have waited until 2AM to say this.
Like did someone vibe code too hard, how do env secrets leak for everyone...
I'm really curious how hackers have monetised the compromised API keys? Do they submit them to a shady AI-multiplexer that provides cheaper inference using stolen keys? One would expect completely random/disparate usage patterns/prompts if so...
managers, accountants, c-suite, sales dept, shareholders, stock-must-go-up, etc...
its no secret the industry wide trash of "move fast and break things" has been the mantra far too often as opposed to "take the time to build it correctly"
for much of the industry its "ship features. now."
we regularly read here how often engineer's concerns are ignored by company salesmen and executives.
What's happening is that machines are now better at coding than humans are, and they are finding copious numbers of vulnerabilities in software written by "competent programmers" over the last 40 years.
This is only going to get worse.
> All deployments are paused
Can you explain us how can we rotate secrets without going down too?
I changed he api key in the env var as openrouter revoke the previous one based on this incident. However, as the redeploy cannot be done in my region, the env cannot be updated and now the product cannot be used. Any command in railway to fix this?
What is the best VPS to use today? Is it still hetzner or ovh?
>Actually I will move my projects away from Railway
If you want a railway like experience, i have been building an opensource alternative to railway: https://github.com/axernel-labs/stackdome . Its mit-licensed and can run on a single vps to full-scale k8s clusters.
Also it's impossible to get verified on the Discord server to see what's going on and the support link in the dashboard leads to a 404 page.
I am kinda losing patience here.
the speed of irresponsible deployments is infinite now, by design
Also they regenerated credentials for buckets but stopped deployments so the services could not connect to the buckets. Completely stupid. If you are going to regenerate the credentials, then let us redeploy the services.
Was it just to create damage? i.e. create unnecessary spend to cause trouble? Or were they trying to use the LLMs for a specific reason?
Audit Logs
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway StaffShip software peacefully
Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.
I had an open gitea host pulled into some botnet
Luckily digitaloceam sent an alert.
Thankfully I'm only using Railway for a hobby project (aptly, for a real-time transit dashboard) but still
What can i do? Move to render ?
So basically, the same shit as in the crypto heydays - rich techbros providing extremely lucrative incentives for unscrupulous actors to hack anyone and anything.
Is that an accusation? As in, major AI players taking out competitors ?
AI makes this so much easier to do now, we should all own our own code, and deployment infrastructure (use a cloud instead of someone wrapping aws for you)
You do have an argument to be made about vendoring our dependencies, or removing them in general. Not a bad idea
Our own open-source, cross-platform, multi-language, cross-framework SecureStore is an embedded secrets store (sort of like SQLite) that lets you version your secrets alongside your code in git (and deploy updates via CI, accordingly), only securely.
Available for rust, golang, php, python, js/ts (node, bun, or in-browser), C#/.NET, PowerShell, bash, and more. Open Source (MIT), not vibe coded, and lovingly maintained for a decade.
The somewhat widely known Mozilla SOPS is a tool in the same broad category.
I hadn't seen securestore-rs before. Thank you for sharing this.