Went down a bit of a rabbit hole reading the post-mortem and status logs on the Oct 21 Railway incident. A few thoughts on the technical vs. comms execution here.
First, the timeline. At 07:24 UTC, Railway opens a status entry titled "Railway Platform Outage." By 09:28, they pull the plug on the dashboard, API, and CLI, and pause builds across all regions. Then, for ~14.5 hours, from the outside, you basically think you're just sitting through a standard platform outage. It's not until 21:51 UTC that they post on X that an intruder may have read and decrypted customer environment variables.
Think about what actually lives in those env vars—database passwords, Stripe keys, OpenAI/Anthropic keys, sometimes even hot crypto wallet keys. And to be clear, this isn't users holding it wrong; this is the exact abstraction Railway built the product around. So for an entire working day, the state of the world was: an adversary potentially has your master keys in plaintext, and you don't know you need to be rotating them.
Reading through the status updates, the disclosure, the preliminary report, and the customer email floating around, you have to separate this into two distinct layers: the engineering response and the comms layer.
On the systems/engineering side, I'd actually give the team a pretty solid grade. Doing incident response live while an attacker might be in the perimeter is brutal, high-stress work. They rotated all storage bucket credentials, nuked the GitHub tokens they held, invalidated customer API tokens, and pushed out concrete runbooks for rotating keys across providers. Jake Cooper (the founder) stepped up and signed the disclosure himself. The low-level mechanics were handled well.
Where the leadership stack failed is the comms protocol. In systems engineering, humans tend to forget the exact root cause pretty quickly (some permissions bug, a leaked token, etc.), because distributed security is just fundamentally hard. What permanently updates their trust weights is the information latency: what you told them during the crisis, and the exact timestamp of when you told them.
Let's unpack where the state machine broke down.
1. Railway told the vendors before it told the customers
In its preliminary report, Railway's team said they learned at 09:25 UTC on October 21 that the attacker had accessed keys. They started working with GitHub, GitLab, Anthropic, OpenAI, OpenRouter, Stripe and others to revoke them. That was the right call. Then Railway's team waited about twelve more hours to tell the people who owned those keys.
So customers heard from their vendors first. Postiz, a social media scheduling tool, shut down for all its users at 21:11 UTC after Anthropic and Stripe flagged its keys as compromised. Railway posted its warning forty minutes later.
Security researcher Taylor Monahan has tied wallet drains starting at 19:36 UTC to keys stored on Railway, more than two hours before the public notice. Railway hasn't confirmed the link. Customers who lost money in those two hours won't wait for Railway's confirmation before they decide what they think of Railway.
2. Railway's team called a breach an outage
When a company announces an outage, customers wait. When it announces a breach, customers act. Railway's team kept the "outage" label through most of the day, so customers waited when they should have been rotating keys.
Nobody outside Railway knows what the team understood at 07:24. But by 09:25 they knew enough to ask Stripe to revoke keys. They kept calling it an outage on the status page until 21:59.
3. Railway published two timelines
In the first disclosure, Railway's team said they detected anomalous activity on the morning of October 21. In the preliminary report a day later, they said they became aware of a "sustained and targeted campaign" against their infrastructure at 19:53 UTC on October 20.
Railway's team may have told the truth both times. An engineer may have spotted an odd signal on Tuesday evening that the team only recognised as an attack on Wednesday morning. But Railway's team never explained the gap, so readers on Hacker News and X explained it for them; they picked the worst version.
If you publish two timelines, give people one sentence on why they differ.
4. "A subset of users" became 100,000 workspaces
In the first disclosure, Railway described the impact as limited to a subset of users. In the preliminary report, the team put the ceiling at 1.85% of all workspaces, or about 100,000.
Railway's team may have meant both statements honestly. But customers read "subset" as permission to relax. Customers anchor on the first number a company gives them; they hold every larger number against it.
5. Railway asked customers to act, then blocked them from acting
Railway told every customer to rotate every secret. Railway had also paused builds, so customers couldn't deploy the new secrets. Enterprise and Pro customers got builds back early on October 22. Hobby, Free and Trial customers waited more than 27 hours from the start of the incident.
Customers couldn't reach the support site, station.railway.com, for much of the crisis either. Railway's team lost the site again on October 23 and spent days restoring it.
Railway's leaders told customers to move fast, then left them with no way to ship the fix and no way to ask for help.
6. Railway sent an email that broke off mid-sentence
Railway's support team sent the customer email at 22:37 UTC, after midnight across most of Europe. Dennis Weidner, a German founder whose company runs on Railway, published his copy. Whoever wrote it left the second instruction, about external registry credentials, unfinished halfway through the sentence.
Nobody leaves a platform over a typo. But customers read a breach email twice, looking for proof that someone is in control. Customers who hit a half-finished instruction conclude that nobody proofread it.
7. Railway's team made the attacker sound unbeatable
In the preliminary report, Railway's team said the attacker probably had uncensored AI models and large amounts of compute, which let them run highly sophisticated exploits. They also said they had taken other steps they would not disclose.
Railway's team may be right on both counts. But when a company leads with how powerful its attacker was, customers hear "don't blame us." Customers can't check either claim. They can check when Railway told them.
8. Railway said nothing about the cost to customers
Railway's team announced compensation after an outage in February. This time, they mentioned no credits in the disclosure or on the status page. Customers spent days rotating keys, reconnecting integrations and reading logs. Railway's leaders could have acknowledged those lost days in one line from the founder. They didn't.
9. Railway's team didn't plan for the second wave
On October 24, Railway's team posted a clarification on the status page saying Railway was not under attack. AI providers had begun revoking keys as a precaution; customers read the revocation emails as a fresh breach. Railway's team had set that wave of emails in motion on October 21. They didn't warn customers it was coming.
How SELF handles crisis comms
Railway's engineers didn't cause any of these nine problems. Railway lacked a person whose only job, for those fourteen hours and the week after, was the words: what to call the incident, who hears first, how to state the scope, what to promise, what the email says, and what customers will think when the vendors start sending their own notices.
At SELF, we do that job for technology companies.
When an outage, a breach, an AI controversy, a social media backlash or a business crisis puts a company in the public eye, we work inside the response team with the engineers, the executives and legal counsel. We help leaders decide what to say, who says it and who needs to hear it first. We write the holding statements, status updates, customer notices and press responses, from the first statement to the final update; we check each one against the incident log. We set out what silence will cost next to every other option, so leaders choose with the price in front of them. We stay on after the fix ships, because customers keep talking long after engineers close the ticket.
We also do the work before anything happens. We plan for the risks a company can already see, set up response protocols and draft the statements in advance.
We're AI-native. Our agents watch the forums, X, Hacker News, the trade press and status page traffic and flag which claims are spreading and which need an answer within the hour. A senior person checks their work, makes the calls and approves every word that goes out under a client's name.
I've watched the first hour of enough incidents to know how it usually goes. A hundred people pile into the incident channel and nobody owns the words. Legal wants to say nothing; engineering wants to explain everything. The team drafts the holding statement nine times in a shared doc while a stranger's theory gets upvoted on Hacker News. Customers read the status page next to a screenshot from Discord; they trust the screenshot.
What to put in place before your bad week
If you run a platform that holds other people's secrets, here's what I'd set up now:
- Pick one person to own the words. Engineers fix the system; that person writes everything customers read.
- Write the breach notice before you need it. Draft the email and the status page language now, while nobody is panicking.
- Tell customers when you tell vendors. If you've asked Stripe to revoke keys, your customers deserve the same news within the hour.
- Give scope as a ceiling. "Up to 1.85% of workspaces" holds up better than "a subset."
- Keep one timeline. When new facts change it, say what changed and why.
- Don't ask customers to do what you've blocked. If builds are down, tell people to revoke at the provider first and tell them when they can deploy.
- Plan the second wave. Every vendor you ask for help will email your customers. Warn your customers first.
- Name the cost. Tell customers you know what the incident cost them and what you'll do about it.
If your company could have a week like Railway's, I'd rather meet you before it starts. Send SELF a brief at thisisstudioself.com or email hello@thisisstudioself.com.
