{"id":49594,"date":"2026-10-22T16:20:15","date_gmt":"2026-10-22T10:50:15","guid":{"rendered":"https:\/\/www.getastra.com\/blog\/?p=49594"},"modified":"2026-10-22T16:39:11","modified_gmt":"2026-10-22T11:09:11","slug":"railway-breach-what-to-rotate-and-what-rotation-wont-fix","status":"publish","type":"post","link":"https:\/\/www.getastra.com\/blog\/security-advisory\/railway-breach-what-to-rotate-and-what-rotation-wont-fix\/","title":{"rendered":"Railway Breach: What to Rotate, and What Rotation Won\u2019t Fix\u00a0"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><strong>This post updates as Railway publishes new details.<\/strong> Check the update log at the bottom before you act on anything older than a few hours.<\/p>\n\n\n<div class=\"gb-container gb-container-e43a8917\">\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"TLDR\"><\/span>TL;DR<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Railway says an unauthorised party <strong>may have<\/strong> accessed customer environment variables and, for some, <strong>may have<\/strong> obtained decrypted values: API keys, database credentials, registry credentials. Railway strongly recommends that every customer revoke all environment variables stored on Railway. (<a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-wednesday-october-21st\" target=\"_blank\" rel=\"noopener\">disclosure<\/a>)<\/li>\n\n\n\n<li><strong>Priority action:<\/strong> For any deployment with variables stored on Railway, the priority is to review those variables and rotate or revoke the affected secrets. Pausing or stopping the application doesn\u2019t remove exposure to credentials already read.<\/li>\n\n\n\n<li>After a day, three facts that should determine rotation and containment strategy remain publicly unknown: whether the intruder&#8217;s access has been closed, when it began, and whose variables were read.<\/li>\n\n\n\n<li><a href=\"https:\/\/thedefiant.io\/news\/security\/developer-links-370-000-theft-to-railway-secrets-exposure\" target=\"_blank\" rel=\"noopener\">The Defiant<\/a> reports that a developer linked a $370,000 theft to secrets stored on Railway. (Railway hasn&#8217;t confirmed the claim, and we haven&#8217;t verified the on-chain transfer or the investigator attribution.)&nbsp;<\/li>\n\n\n\n<li>Most guidance, Railway&#8217;s included, treats rotation as swapping a variable and redeploying. That closes the exposure for bearer API keys. It does not close it for signing secrets, encryption keys, webhook secrets, wallet keys, or the infrastructure map your variables reveal.<\/li>\n\n\n\n<li><strong>Order of work<\/strong>: contain at the provider, rotate money-moving secrets first, redeploy as your tier&#8217;s builds return, then hunt backward through logs.<\/li>\n<\/ul>\n\n<\/div>\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_happened_what_we_still_need_to_know\"><\/span>What happened &amp; what we still need to know<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Railway&#8217;s disclosure says it detected anomalous activity in the systems that store customer environment variables on the morning of October 21. Railway\u2019s investigation to date indicates the third party may have accessed encrypted variables and, for some, the decrypted contents.<\/p>\n\n\n\n<div id=\"tablepress-490-scroll-wrapper\" class=\"tablepress-scroll-wrapper\">\n<table id=\"tablepress-490\" class=\"tablepress tablepress-id-490 column1-color tablepress-responsive\">\n<thead>\n<tr class=\"row-1\">\n\t<th class=\"column-1\">Question<\/th><th class=\"column-2\">What Railway has said<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">Was there unauthorised activity in the variable-storage systems?<\/td><td class=\"column-2\">Yes, Railway detected anomalous activity there.<\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">Were secrets exposed?<\/td><td class=\"column-2\">Possibly. Railway says encrypted variables may have been accessed and, for some, decrypted. Treat any variable stored on Railway as potentially exposed.<\/td>\n<\/tr>\n<tr class=\"row-4\">\n\t<td class=\"column-1\">Is every customer affected?<\/td><td class=\"column-2\">Not established. Railway says impact may be a subset, but recommends everyone act.<\/td>\n<\/tr>\n<tr class=\"row-5\">\n\t<td class=\"column-1\">How did the attacker get in?<\/td><td class=\"column-2\">Not disclosed.<\/td>\n<\/tr>\n<tr class=\"row-6\">\n\t<td class=\"column-1\">Who is behind it?<\/td><td class=\"column-2\">Not disclosed. Railway links it to a broader campaign against platform companies.<\/td>\n<\/tr>\n<tr class=\"row-7\">\n\t<td class=\"column-1\">How many accounts?<\/td><td class=\"column-2\">Not disclosed. Railway says it will contact confirmed affected customers directly by email.<\/td>\n<\/tr>\n<tr class=\"row-8\">\n\t<td class=\"column-1\">Was customer application data, source code or databases accessed?<\/td><td class=\"column-2\">Not stated in the disclosure or status updates we reviewed.<\/td>\n<\/tr>\n<tr class=\"row-9\">\n\t<td class=\"column-1\">Has misuse been confirmed?<\/td><td class=\"column-2\">Not stated.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<!-- #tablepress-490 from cache -->\n\n\n\n<p class=\"wp-block-paragraph\">Most independent coverage repeats Railway&#8217;s wording. We treat <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-wednesday-october-21st\" target=\"_blank\" rel=\"noopener\">&nbsp;the disclosure<\/a> and the <a href=\"https:\/\/status.railway.com\/\" target=\"_blank\" rel=\"noopener\">status<\/a> page as the record and mark everything else as inference.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What we still don&#8217;t know<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Is the intruder still inside?<\/strong> Decides whether anything you write to Railway today is safe.<\/li>\n\n\n\n<li><strong>When did access begin?<\/strong> Sets your log look-back. Until Railway says, use the full retention of every log you have.<\/li>\n\n\n\n<li><strong>How were decrypted values obtained?<\/strong> Tells you whether sealed or reference variables were protected differently. We couldn&#8217;t confirm this.<\/li>\n\n\n\n<li><strong>Whose variables were read?<\/strong> Railway says it will email confirmed accounts. No email means unconfirmed, not clear.<\/li>\n\n\n\n<li><strong>Was customer data touched, or only credentials?<\/strong> Decides whether you also have a notification question for legal, not only a rotation task.<\/li>\n\n\n\n<li><strong>Did upstream providers see misuse?<\/strong> Railway asked AI providers to revoke keys. Ask each provider whether yours was revoked or flagged.<\/li>\n\n\n\n<li><strong>Have customers who need them replaced their Railway API tokens?<\/strong> Railway invalidated all of them at 04:17 UTC.<\/li>\n\n\n\n<li><strong>When is the postmortem?<\/strong> We haven&#8217;t seen one. Until it lands, the three questions at the top stay open.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Who_is_affected\"><\/span><strong>Who is affected<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Any deployment with variables stored on Railway should be treated as affected until its variables are reviewed. Railway has not published which accounts were touched.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">No email from Railway doesn&#8217;t mean you&#8217;re clear. Railway says it will email accounts as they&#8217;re confirmed, and confirmation may lag the investigation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Risk is highest for teams whose Railway variables include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>AI keys, because Railway ties the incident to attacks on AI inference providers.<\/li>\n\n\n\n<li>Cloud credentials that can create resources or change IAM.<\/li>\n\n\n\n<li>Production database URLs (including public proxy URLs).<\/li>\n\n\n\n<li>Payment or CI\/CD tokens, which carry supply-chain and financial risk.<\/li>\n\n\n\n<li>Private registry credentials, which Railway specifically calls out.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">If you use storage buckets, Railway has already reset those credentials. Expect to see the revocations in your audit log and plan a redeploy. If you use the GitHub integration, Railway&#8217;s guide covers reconnecting after the reset. (<a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-redeploy-buckets-and-reconnect-github-after-the-credential-reset\" target=\"_blank\" rel=\"noopener\">Railway guide<\/a>)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Open question:<\/strong> we could not confirm from public sources whether variables marked as sealed, or variables injected through reference chains, were treated differently. Don&#8217;t assume they were safe unless Railway says so.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Our_lens_the_fix_runs_through_the_system_thats_down\"><\/span><strong>Our lens: the fix runs through the system that&#8217;s down<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A common assumption is that pausing or stopping an affected service removes the risk. It does not. Any credential that was stored and possibly read stays usable by whoever holds it, so rotation has to happen whether or not the app is running.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On a platform like Railway rotating a secret takes three steps, and the last one was blocked for most of the first day:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Issue a new credential at the upstream provider.<\/li>\n\n\n\n<li>Put it in the Railway variable.<\/li>\n\n\n\n<li>Redeploy, because containers read variables at start-up.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Railway explicitly said guides \u2018may require redeployment\u2019 and that those steps had to wait until builds were available.\u201d&nbsp; So for every secret you had a trade-off:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Revoke now.<\/strong> The attacker loses the key immediately. Your service loses it too. Revoking a provider key can immediately break upstream calls until you deploy a replacement.<\/li>\n\n\n\n<li><strong>Leave it live.<\/strong> Your service keeps working, and so can anyone holding a copy.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">There&#8217;s a third option: contain in place. If you can shrink what the key is allowed to do at the provider, you don&#8217;t have to choose between a working key and a safe key, and you don&#8217;t have to touch Railway.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There&#8217;s also a fourth operational reality. Railway&#8217;s API-token invalidation can break CI\/CD, automation and third-party integrations, independent of your application secrets. Recreate those tokens before you assume your pipelines are healthy.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>A decision rule we use<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Rank every secret by two questions: what&#8217;s the worst thing this key can do, and can I afford the downtime of revoking it?<\/p>\n\n\n\n<div id=\"tablepress-491-scroll-wrapper\" class=\"tablepress-scroll-wrapper\">\n<table id=\"tablepress-491\" class=\"tablepress tablepress-id-491 column1-color tablepress-responsive\">\n<thead>\n<tr class=\"row-1\">\n\t<th class=\"column-1\">Secret type<\/th><th class=\"column-2\">Worst case<\/th><th class=\"column-3\">Recommended move while builds are paused<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">AI provider keys (OpenAI, Anthropic, OpenRouter, Grok)<\/td><td class=\"column-2\">Direct spend, quota theft, resale of access<\/td><td class=\"column-3\">Revoke, or at minimum set a hard spend or rate cap and watch usage by key. Railway told upstream providers to revoke proactively, so some keys may already be dead.<\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">Cloud keys (AWS, GCP)<\/td><td class=\"column-2\">Create infrastructure (GPU mining), change IAM, read buckets<\/td><td class=\"column-3\">Contain in place with a deny policy or IP condition, or deactivate if downtime is tolerable.<\/td>\n<\/tr>\n<tr class=\"row-4\">\n\t<td class=\"column-1\">Stripe secret or restricted keys<\/td><td class=\"column-2\">Refunds, payouts, data access, webhook tampering<\/td><td class=\"column-3\">Rotate with Stripe's overlap window if you can deploy, otherwise, restrict permissions and watch the logs.<\/td>\n<\/tr>\n<tr class=\"row-5\">\n\t<td class=\"column-1\">Database passwords<\/td><td class=\"column-2\">Data theft or tampering<\/td><td class=\"column-3\">Regenerate, then redeploy consumers. Remove public access if nothing outside the project needs it. Check for sessions you don't recognize.<\/td>\n<\/tr>\n<tr class=\"row-6\">\n\t<td class=\"column-1\">Registry and GitHub tokens<\/td><td class=\"column-2\">Supply-chain tampering<\/td><td class=\"column-3\">Revoke and re-issue. Railway reset its own storage-bucket credentials; its guide covers redeploying and reconnecting GitHub afterward.<\/td>\n<\/tr>\n<tr class=\"row-7\">\n\t<td class=\"column-1\">Internal service-to-service tokens<\/td><td class=\"column-2\">Lateral movement<\/td><td class=\"column-3\">Rotate after the above; lower priority unless they reach production data.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<!-- #tablepress-491 from cache -->\n\n\n\n<p class=\"wp-block-paragraph\">This rule is our judgment, not Railway&#8217;s. Railway&#8217;s guidance is to roll everything.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Rotating into a store that may still be open<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Rotation assumes the place you write the new secret is clean. In everything we reviewed, Railway has not said the intruder&#8217;s access is closed, or how decrypted values were reached. Until it does, a fresh key written to a Railway variable carries the same exposure as the old one. This is our assessment, not Railway&#8217;s. We handle it three ways:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Scope replacements tighter than the originals.<\/strong> One key per service, read-only where the workload allows, spend caps on metered providers, and provider-side allow-lists pinned to Railway&#8217;s <a href=\"https:\/\/docs.railway.com\/networking\/static-outbound-ips\" target=\"_blank\" rel=\"noopener\">static outbound IPs<\/a> if you use them.<\/li>\n\n\n\n<li><strong>Prefer a replacement that&#8217;s useless elsewhere over one that&#8217;s merely new.<\/strong> For crown-jewel secrets (cloud admin, payment secret keys, anything that signs transactions), IP-bound, short-lived or narrowly scoped credentials beat a fresh long-lived string.<\/li>\n\n\n\n<li><strong>Alert on first use.<\/strong> A new key that authenticates from an address outside your egress range tells you the store leaked again.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Example: contain an AWS key in place<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This attaches an inline policy to the IAM user that owns the exposed key. It denies privilege escalation (<code>iam:* <\/code>and <code>sts:*<\/code>, so the key can\u2019t assume a more powerful role), the common compute-launch routes (instances, fleets, spot requests, Auto Scaling, ECS and Batch), and Bedrock model invocation. An explicit deny wins over any allow.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Adapt the action list to your workload. <code>sts:* <\/code>breaks apps that assume roles, and <code>bedrock:* <\/code>breaks apps that call Bedrock, so narrow or remove those if you depend on them. A deny list only blocks what it names, so treat it as a stopgap. If you can tolerate the downtime, deactivating the key is safer.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": &#091;{\n    \"Effect\": \"Deny\",\n    \"Action\": &#091;\n      \"iam:*\",\n      \"organizations:*\",\n      \"sts:*\",\n      \"ec2:RunInstances\",\n      \"ec2:CreateFleet\",\n      \"ec2:RequestSpotInstances\",\n      \"ec2:CreateKeyPair\",\n      \"autoscaling:*\",\n      \"ecs:RunTask\",\n      \"batch:SubmitJob\",\n      \"bedrock:*\",\n      \"s3:PutBucketPolicy\"\n    ],\n    \"Resource\": \"*\"\n  }]\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Save that as <code>deny.json<\/code>, then run:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>aws iam put-user-policy \\\n  --user-name railway-app \\\n  --policy-name contain-railway-incident \\\n  --policy-document file:\/\/deny.json<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If you use Railway\u2019s static outbound IPs, you can go further and deny everything not coming from those addresses with an <code>aws:SourceIp <\/code>condition. Without static IPs, Railway egress comes from shared addresses, so an IP restriction would be unreliable.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For GCP, Railway&#8217;s guide suggests disabling the whole service account or applying an IAM deny policy if you need an immediate cut-off, and warns that tokens already issued from the key can live for up to an hour. Remove the containment only after you&#8217;ve deployed the new key, and wait at least 60 minutes before re-enabling a disabled account.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_secrets_that_dont_rotate_cleanly\"><\/span><strong>The secrets that don&#8217;t rotate cleanly<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A variable swap closes the exposure for bearer API keys. For the secrets below it doesn&#8217;t, and most Railway projects hold at least two of them.<\/p>\n\n\n\n<div id=\"tablepress-492-scroll-wrapper\" class=\"tablepress-scroll-wrapper\">\n<table id=\"tablepress-492\" class=\"tablepress tablepress-id-492 column1-color tablepress-responsive\">\n<thead>\n<tr class=\"row-1\">\n\t<th class=\"column-1\">Secret class<\/th><th class=\"column-2\">Typical variable<\/th><th class=\"column-3\">Why a swap isn't enough<\/th><th class=\"column-4\">What closes it<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">Wallet and signing keys<\/td><td class=\"column-2\">PRIVATE_KEY, MNEMONIC, relayer or deployer keys<\/td><td class=\"column-3\">The key is the identity. There is no new key for the same account.<\/td><td class=\"column-4\">Move assets to a new address from a clean machine now, then retire the old one. Treat any balance as already exposed.<\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">Session and token-signing secrets<\/td><td class=\"column-2\">JWT_SECRET, SESSION_SECRET, NEXTAUTH_SECRET<\/td><td class=\"column-3\">Whoever holds it can mint valid sessions for any user, admins included.<\/td><td class=\"column-4\">Rotate (everyone is logged out), then search auth logs across the exposure window for admin actions with no matching login.<\/td>\n<\/tr>\n<tr class=\"row-4\">\n\t<td class=\"column-1\">Application-layer encryption keys<\/td><td class=\"column-2\">ENCRYPTION_KEY, field-level or backup keys<\/td><td class=\"column-3\">Swapping the key breaks reads of existing data. Any ciphertext copied earlier is now decryptable.<\/td><td class=\"column-4\">Versioned keys plus re-encryption. Assume earlier dumps and backups are readable and check for database exports.<\/td>\n<\/tr>\n<tr class=\"row-5\">\n\t<td class=\"column-1\">Webhook signing secrets<\/td><td class=\"column-2\">Stripe whsec_, GitHub and Slack signing secrets<\/td><td class=\"column-3\">An attacker can forge inbound events your app trusts, such as a fake \"payment succeeded\".<\/td><td class=\"column-4\">Rotate at the sender, then reconcile events you processed against the sender's own event log.<\/td>\n<\/tr>\n<tr class=\"row-6\">\n\t<td class=\"column-1\">OAuth client secrets and SSO<\/td><td class=\"column-2\">Google, GitHub, Microsoft client secrets<\/td><td class=\"column-3\">Lets someone impersonate your app in consent flows.<\/td><td class=\"column-4\">Rotate, then review redirect URIs and the authorized-app lists.<\/td>\n<\/tr>\n<tr class=\"row-7\">\n\t<td class=\"column-1\">Email and messaging credentials<\/td><td class=\"column-2\">SMTP, SendGrid, SES, Twilio<\/td><td class=\"column-3\">Phishing and password-reset abuse from your own domain and reputation.<\/td><td class=\"column-4\">Rotate, review send logs, check DKIM and SPF alignment, look for new templates or forwarding rules.<\/td>\n<\/tr>\n<tr class=\"row-8\">\n\t<td class=\"column-1\">Infrastructure metadata and network exposure<\/td><td class=\"column-2\">Hostnames, internal service URLs, public database proxy URLs, bucket names<\/td><td class=\"column-3\">The exposure survives rotation. It is reconnaissance on your stack.<\/td><td class=\"column-4\">Remove public database proxies, add network allow-lists, shrink what is reachable from the internet.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<!-- #tablepress-492 from cache -->\n\n\n\n<p class=\"wp-block-paragraph\">Most teams rotate in the order secrets appear in the Variables tab. Rotate in the order of what each one can move: money first, then identity (signing secrets, OAuth), then data keys, then everything else.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The loss claim we&#8217;ve seen so far fits the first row. The Defiant reports that a developer linked a $370,000 theft to secrets stored on Railway. The report is associated with ZachXBT\u2019s exploitation coverage, but Railway has not confirmed the claim, and we have not independently verified the on-chain transfer, victim attribution, or causal link to this incident.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_to_do_a_recovery_runbook\"><\/span><strong>What to do: a recovery runbook<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 0: Freeze and get organized (15 minutes)<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Review every variable on each affected deployment and record which ones are credentials. Do this before changing anything, and do it even if the application is paused.<\/li>\n\n\n\n<li>Stop pasting secrets into chat, tickets and docs. Use a password or secrets manager.<\/li>\n\n\n\n<li>Start an incident log with timestamps for every action. You&#8217;ll need it for any later customer, auditor or insurer conversation.<\/li>\n\n\n\n<li>Check <a href=\"https:\/\/status.railway.com\/\" target=\"_blank\" rel=\"noopener\">status.railway.com<\/a> for build availability. This page is Railway&#8217;s stated fastest source of updates.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Inventory what is stored (names, not values)<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">List the variable names for every project, service and environment so you can map each to an owner and an upstream system. This prints names only, not values. Verify the flags against your CLI version first.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Print every name and let the keywords highlight the likely secrets. The highlighting is a guide, not a filter: during an incident, read the whole list. Names that match no keyword can still hold secrets.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One run is not a full inventory. This loop covers one project at a time, so link or select each project in turn and repeat it. Edit the environment and service lists to match what each project actually has.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"># KEYWORDS only highlights. Every name is printed.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>KEYWORDS='key|secret|token|pass|pwd|dsn|url|uri|credential|private|auth|cert|mnemonic|seed|salt|sign|jwt|whsec|webhook|cookie'\nset -o pipefail\n\nfor env in production staging; do        # every environment, including previews\n  for svc in api worker web; do          # every service in this project\n    echo \"=== $env \/ $svc\"\n   railway variables --service \"$svc\" --environment \"$env\" --json \\\n\n      | jq -r 'keys&#091;]' | sort \\\n      | grep -Ei --color=always \"$KEYWORDS|$\" \\\n      || echo \"!! failed for $env \/ $svc, check by hand\"\n  done\ndone<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The trailing <code>|$ <\/code>makes grep match every line, so nothing is dropped and the keywords only add colour. If you redirect the output to a file, drop <code>--color=always <\/code>to avoid raw escape codes. A failed service prints a warning, so it can\u2019t pass for an empty one.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This loop doesn\u2019t see preview environments you haven\u2019t listed, shared variables set in Project Settings, or variables you copied outside Railway (local <code>.env <\/code>files, CI). Check those separately.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Triage by blast radius<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Use the table above. Mark each secret <em>revoke now<\/em>, <em>contain in place<\/em> or <em>rotate after redeploy<\/em>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Contain, then rotate, per vendor<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Railway published vendor-specific guides on Oct 22.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Railway API tokens<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Recreate Railway API tokens if you use the Public API, CLI integrations, automation or third-party tools authenticated with a Railway token. Railway invalidated all of them at 04:17 UTC on Oct 22.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>AWS access keys<\/strong> (<a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-anomalous-aws-access\" target=\"_blank\" rel=\"noopener\">Railway guide<\/a>)<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>In IAM, find the user for the exposed key ID, create a replacement key, then deactivate (don&#8217;t delete) the old one. Deactivating first lets you reactivate briefly if you&#8217;ve forgotten a consumer.<\/li>\n\n\n\n<li>Edit both <code>AWS_ACCESS_KEY_ID <\/code>and <code>AWS_SECRET_ACCESS_KEY <\/code>in Railway (once in Project Settings if shared) and deploy the staged change.<\/li>\n\n\n\n<li>Confirm logs show successful AWS calls with no <code>InvalidClientTokenId <\/code>or <code>SignatureDoesNotMatch <\/code>errors, and update CI and local environments.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>GCP service-account keys<\/strong> (<a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-anomalous-gcp-access\" target=\"_blank\" rel=\"noopener\">Railway guide<\/a>)<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Create a replacement JSON key. Google doesn&#8217;t keep a copy, so it downloads once.<\/li>\n\n\n\n<li>Disable the exposed key by ID using the Google Cloud console or CLI.&nbsp;<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Note: Previously issued tokens can remain valid for up to an hour, use a service-account disable or IAM deny policy if you need an immediate cutoff.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gcloud iam service-accounts keys disable KEY_ID \\\n  --iam-account=SA_NAME@PROJECT.iam.gserviceaccount.com \\\n  --project=PROJECT<\/code><\/pre>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li>Paste the new JSON into the Railway variable as-is (multiline is supported), deploy, check for <code>invalid_grant <\/code>or <code>PERMISSION_DENIED<\/code>, then delete the old key once your review is done. Review roles on the account and remove any you don&#8217;t recognize, in particular Service Account Token Creator on other accounts.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Databases<\/strong> (guides for <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-rotate-postgresql-database-credentials\" target=\"_blank\" rel=\"noopener\">PostgreSQL<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-rotate-mysql-database-credentials\" target=\"_blank\" rel=\"noopener\">MySQL<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-rotate-mongodb-database-credentials\" target=\"_blank\" rel=\"noopener\">MongoDB<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-rotate-redis-database-credentials\" target=\"_blank\" rel=\"noopener\">Redis<\/a>)<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Before rotating, list consumers, check whether Public Access (TCP proxy) is on, and note anything using the public URL (local dev, CI, BI tools).<\/li>\n\n\n\n<li>In the Database tab, use Regenerate. It changes the password inside the database and updates the template variables together. Don&#8217;t hand-edit password variables or run <code>ALTER USER <\/code>yourself, because that leaves the database and the variables out of sync.<\/li>\n\n\n\n<li>Redeploy each service that consumes the variables. Replace pasted connection strings with references such as <code>${{ MySQL.MYSQL_URL }} <\/code>so the next rotation propagates automatically.<\/li>\n\n\n\n<li>Afterward, check for sessions you don&#8217;t recognize (queries in Step 5). Any session using the new password means the new password leaked too, so find out how.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>AI providers<\/strong> (<a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-openai-usage\" target=\"_blank\" rel=\"noopener\">OpenAI<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-anthropic-usage\" target=\"_blank\" rel=\"noopener\">Anthropic<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-openrouter-usage\" target=\"_blank\" rel=\"noopener\">OpenRouter<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-grok-usage\" target=\"_blank\" rel=\"noopener\">Grok<\/a>)<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Revoke or delete the key in the provider console (immediate), create a replacement, update the Railway variable and deploy.<\/li>\n\n\n\n<li>Open usage, group by API key, and compare token volume, request counts and model mix against normal traffic. Check team members and project membership for people you didn&#8217;t invite.<\/li>\n\n\n\n<li>If your key was already auto-revoked upstream, you still need to deploy a replacement and review usage.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stripe<\/strong> (<a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-usage-in-stripe\" target=\"_blank\" rel=\"noopener\">Railway guide<\/a>)<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Rotate the exposed secret or restricted key. For a confirmed leak, Railway&#8217;s guide describes setting the old key to expire immediately; otherwise use Stripe&#8217;s overlap window. Rotate every other secret and restricted key on the account too.<\/li>\n\n\n\n<li>Update the key in Railway and redeploy.<\/li>\n\n\n\n<li>Review per-key request logs, team members and security history for changes you didn&#8217;t make.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>GitHub and registries<\/strong> (<a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-anomalous-github-access\" target=\"_blank\" rel=\"noopener\">Railway guide<\/a>)<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Review your security log, authorized OAuth apps and GitHub Apps, and SSH and GPG keys. Revoke anything unfamiliar.<\/li>\n\n\n\n<li>For organizations, check the audit log and third-party access.<\/li>\n\n\n\n<li>Confirm Railway only deployed commits you recognize, then replace registry tokens.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 4: Redeploy when builds return<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Builds are being restored in stages: Enterprise first (back at 04:00 UTC on Oct 22), then Pro, Hobby, and finally Free and Trial. When your tier is available, deploy the staged variable changes, verify health, then delete old credentials. Keep a short list of anything you revoked but couldn&#8217;t yet replace, so you don&#8217;t leave an outage behind.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 5: Hunt for misuse (rotation doesn&#8217;t undo what already happened)<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The exposure window is unknown. Railway only says it detected activity on Oct 21, not when access began. Look back as far as your logs allow.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>AWS: every Region, one key.<\/strong> CloudTrail event history is per Region and keeps 90 days of management events (for longer retention, query an audit trail, security lake, or exported log destination). Run it for each Region and read the source IPs.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>KEY=AKIAXXXXXXXXXXXXXXXX\nSINCE=$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%SZ)\nfor r in $(aws ec2 describe-regions --all-regions --query 'Regions&#091;].RegionName' --output text); do\n  echo \"== $r\"\n  aws cloudtrail lookup-events --region \"$r\" \\\n    --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=\"$KEY\" \\\n    --start-time \"$SINCE\" --output json \\\n  | jq -r '.Events&#091;].CloudTrailEvent | fromjson | &#091;.eventTime,.eventName,.sourceIPAddress,.awsRegion] | @tsv'\ndone<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Red flags from Railway&#8217;s guide: <code>RunInstances<\/code>, <code>CreateUser<\/code>, <code>CreateAccessKey<\/code>, <code>AttachUserPolicy<\/code>, <code>CreateRole<\/code>, <code>PutBucketPolicy<\/code>, and anything from a Region you don&#8217;t use. Also pull the IAM credential report, and compare spend by service:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>aws ce get-cost-and-usage \\\n  --time-period Start=2026-10-01,End=2026-10-23 \\\n  --granularity DAILY --metrics UnblendedCost \\\n  --group-by Type=DIMENSION,Key=SERVICE<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>AWS: last use of a key.<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><code>aws iam get-access-key-last-used --access-key-id \"$KEY\"<\/code><\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>GCP: what did this service account do?<\/strong> In Logs Explorer, set the exposure window and run:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>logName:\"cloudaudit.googleapis.com\"\nprotoPayload.authenticationInfo.principalEmail=\"SA_NAME@PROJECT.iam.gserviceaccount.com\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then check for impersonation with<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>protoPayload.authenticationInfo.serviceAccountDelegationInfo.firstPartyPrincipal.principalEmail<\/code>, look at the key&#8217;s last authentication with Activity Analyzer, and compare Compute, GPU and Vertex AI billing against the prior month. Data Access logs are off by default except <code>BigQuery<\/code>, so absence of data-read entries doesn&#8217;t prove nothing was read.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>PostgreSQL: who&#8217;s connected after rotation?<\/strong> Railway notes PostgreSQL doesn&#8217;t log successful connections or client addresses by default, so live inspection matters.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>SELECT pid, usename, client_addr, application_name, backend_start\nFROM pg_stat_activity\nWHERE pid &lt;&gt; pg_backend_pid()\nORDER BY backend_start;\n-- terminate anything you don't recognize\nSELECT pg_terminate_backend(&lt;pid&gt;);<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>MySQL: accounts and sessions.<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>SELECT id, user, host, db, command, time FROM information_schema.processlist;\nSELECT user, host FROM mysql.user;\nSHOW GRANTS FOR 'root'@'%';<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Railway itself.<\/strong> Review audit logs (retention depends on plan) to see who viewed or changed variables, filter deploy logs for &#8220;Access denied&#8221; and unfamiliar hosts, and check that deployments reference commits you recognize.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Billing as a canary.<\/strong> The fastest signal that a stolen AI or cloud key is in use is an unexplained cost line. Set budget alerts and hard caps today, even if you rotate everything.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Vendor_guide_index\"><\/span><strong>Vendor guide index<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>Vendor<\/strong><\/td><td><strong>Railway&#8217;s guide<\/strong><\/td><td><strong>Upstream reference<\/strong><\/td><\/tr><tr><td>Railway disclosure<\/td><td><a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-wednesday-october-21st\" target=\"_blank\" rel=\"noopener\">Security Disclosure: Oct 21<\/a><\/td><td><a href=\"https:\/\/status.railway.com\/\" target=\"_blank\" rel=\"noopener\">Status page<\/a><\/td><\/tr><tr><td>AWS<\/td><td><a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-anomalous-aws-access\" target=\"_blank\" rel=\"noopener\">Check for Anomalous AWS Access<\/a><\/td><td><a href=\"https:\/\/repost.aws\/knowledge-center\/potential-account-compromise\" target=\"_blank\" rel=\"noopener\">AWS: unauthorized activity<\/a>, <a href=\"https:\/\/docs.aws.amazon.com\/IAM\/latest\/UserGuide\/id-credentials-access-keys-update.html\" target=\"_blank\" rel=\"noopener\">update access keys<\/a>, <a href=\"https:\/\/docs.aws.amazon.com\/guardduty\/latest\/ug\/guardduty_finding-types-iam.html\" target=\"_blank\" rel=\"noopener\">GuardDuty IAM findings<\/a><\/td><\/tr><tr><td>GCP<\/td><td><a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-anomalous-gcp-access\" target=\"_blank\" rel=\"noopener\">Check for Anomalous GCP Access<\/a><\/td><td><a href=\"https:\/\/docs.cloud.google.com\/docs\/security\/compromised-credentials\" target=\"_blank\" rel=\"noopener\">Handling compromised credentials<\/a>, <a href=\"https:\/\/docs.cloud.google.com\/iam\/docs\/keys-disable-enable\" target=\"_blank\" rel=\"noopener\">disable keys<\/a><\/td><\/tr><tr><td>PostgreSQL \/ MySQL \/ MongoDB \/ Redis<\/td><td><a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-rotate-postgresql-database-credentials\" target=\"_blank\" rel=\"noopener\">Postgres<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-rotate-mysql-database-credentials\" target=\"_blank\" rel=\"noopener\">MySQL<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-rotate-mongodb-database-credentials\" target=\"_blank\" rel=\"noopener\">MongoDB<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-rotate-redis-database-credentials\" target=\"_blank\" rel=\"noopener\">Redis<\/a><\/td><td><a href=\"https:\/\/docs.railway.com\/databases\/database-view#connection\" target=\"_blank\" rel=\"noopener\">Railway Database View<\/a><\/td><\/tr><tr><td>OpenAI \/ Anthropic \/ OpenRouter \/ Grok<\/td><td><a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-openai-usage\" target=\"_blank\" rel=\"noopener\">OpenAI<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-anthropic-usage\" target=\"_blank\" rel=\"noopener\">Anthropic<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-openrouter-usage\" target=\"_blank\" rel=\"noopener\">OpenRouter<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-grok-usage\" target=\"_blank\" rel=\"noopener\">Grok<\/a><\/td><td>Each provider&#8217;s console usage and key pages<\/td><\/tr><tr><td>Stripe<\/td><td><a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-unexpected-usage-in-stripe\" target=\"_blank\" rel=\"noopener\">Check for Unexpected Usage in Stripe<\/a><\/td><td>Stripe&#8217;s guidance on compromised API keys<\/td><\/tr><tr><td>GitHub \/ buckets<\/td><td><a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-check-for-anomalous-github-access\" target=\"_blank\" rel=\"noopener\">Anomalous GitHub Access<\/a>, <a href=\"https:\/\/blog.railway.com\/p\/security-disclosure-recommendation-redeploy-buckets-and-reconnect-github-after-the-credential-reset\" target=\"_blank\" rel=\"noopener\">Redeploy buckets and reconnect GitHub<\/a><\/td><td>GitHub security log and audit log docs<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Where_Astra_can_help\"><\/span><strong>Where Astra can help<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Post-rotation validation.<\/strong> After you rotate, a pentest or targeted assessment can confirm old credentials are dead, that no attacker-created users, keys or roles remain, and that your cloud IAM can&#8217;t be escalated from a single leaked key.<\/li>\n\n\n\n<li><strong><a href=\"https:\/\/www.getastra.com\/pentesting\/cloud\">Cloud <\/a>and credential exposure review.<\/strong> Review AWS and GCP configurations for over-broad permissions, long-lived keys and missing logging, so the next leaked variable has a smaller blast radius.<\/li>\n\n\n\n<li><strong><a href=\"https:\/\/www.getastra.com\/ptaas\">Continuous scanning<\/a>.<\/strong> Catch exposed services, public database proxies and misconfigurations that turn a credential leak into a breach.<\/li>\n\n\n\n<li><strong>Triage support.<\/strong> If you&#8217;ve found suspicious activity and want a second pair of eyes, <a href=\"https:\/\/www.getastra.com\/\">contact Astra<\/a>.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Timeline_UTC_from_Railways_status_page\"><\/span>Timeline (UTC, from Railway&#8217;s status page)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"630\" height=\"825\" src=\"\/cdn-cgi\/image\/quality=80,format=auto,onerror=redirect,metadata=none\/https:\/\/cdn-blog.getastra.com\/2026\/10\/eb8a1fc0-image.png\" alt=\"\" class=\"wp-image-49604\" style=\"aspect-ratio:0.7637698898408812;width:624px;height:auto\"\/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Oct 21, 07:24:<\/strong> Railway starts investigating an issue. Later entries say the dashboard, API and CLI were unavailable and builds were paused in all regions. The support site was also down. These early entries describe an outage and do not mention security.<\/li>\n\n\n\n<li><strong>Oct 21, 19:55\u201321:01:<\/strong> Railway works toward re-enabling builds in stages (Enterprise, then Pro, Hobby, Free and Trial). It invalidates OAuth sessions because of impact on adjacent cache systems, so some users are signed out.<\/li>\n\n\n\n<li><strong>Oct 21, 21:59:<\/strong> Railway publishes the security notice. Deployments stay paused, Enterprise included.<\/li>\n\n\n\n<li><strong>Oct 21, 22:10:<\/strong> Railway starts rotating all storage-bucket credentials itself. New ones appear in dashboards automatically.<\/li>\n\n\n\n<li><strong>Oct 22, 00:22:<\/strong> Railway says storage-bucket credential revocations will appear in audit logs under a Railway employee&#8217;s name. It calls them expected precautions.<\/li>\n\n\n\n<li><strong>Oct 22, 01:22:<\/strong> Railway says remediation guides will be published shortly.<\/li>\n\n\n\n<li><strong>Oct 22, 02:19:<\/strong> Railway publishes vendor guides for OpenRouter, PostgreSQL, Grok, buckets\/GitHub, Anthropic, OpenAI, Stripe, GitHub, MongoDB, Redis, MySQL, AWS and GCP.<\/li>\n\n\n\n<li><strong>Oct 22, 03:00:<\/strong> Builds remain paused; Railway promises an update by 03:45.<\/li>\n\n\n\n<li><strong>Oct 22, 03:32:<\/strong> Railway says builds remain paused and an update for Enterprise users is expected soon.<\/li>\n\n\n\n<li><strong>Oct 22, 04:00:<\/strong> Railway restores builds for Enterprise workspaces so those customers can complete remediation steps that need a redeploy. Pro, Hobby, then Free and Trial follow in stages.<\/li>\n\n\n\n<li><strong>Oct 22, 04:17:<\/strong> Railway invalidates all Railway API tokens. Customers using the Public API must create new ones.<\/li>\n\n\n\n<li><strong>Oct 22, 04:39 UTC:<\/strong> Railway said the Redeploy feature was unavailable and offered a workaround: use Deploy Latest Commit.<a href=\"https:\/\/status.railway.com\/\" target=\"_blank\" rel=\"noopener\">&nbsp;<\/a><\/li>\n\n\n\n<li><strong>Oct 22, 04:52 UTC:<\/strong> Railway said the Redeploy feature was fixed for Enterprise customers.<a href=\"https:\/\/status.railway.com\/\" target=\"_blank\" rel=\"noopener\">&nbsp;<\/a><\/li>\n\n\n\n<li><strong>Oct 22, 05:20 UTC:<\/strong> Deployments were progressing slowly, with some stuck at \u201cRunning pre-deploy command.\u201d<\/li>\n\n\n\n<li><strong>Oct 22, 06:01 UTC:<\/strong> Builds remained available only for Enterprise workspaces. Pro, Hobby, Free and Trial were still pending.<a href=\"https:\/\/status.railway.com\/\" target=\"_blank\" rel=\"noopener\">&nbsp;<\/a><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">We can&#8217;t say from public information how the morning outage relates to the intrusion beyond Railway&#8217;s own statements, so we won&#8217;t guess.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_the_first_day_of_public_reaction_gets_right_and_what_it_misses\"><\/span><strong>What the first day of public reaction gets right, and what it misses<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>The sequencing complaint is correct.<\/strong> Developers asked why customers were told to roll secrets while deployments were blocked, in replies under <a href=\"https:\/\/x.com\/Railway\/status\/2113025284148715845\">Railway&#8217;s announcement<\/a>. Railway later reset bucket credentials itself and published vendor guides, but nothing on its side removes the dependency on redeploying. Containing at the provider does.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>The &#8220;stolen credit cards for compute&#8221; framing for AI keys is mostly right.<\/strong> The correction is that it shows up as a billing and logging problem before it looks like a data problem. Reports of keys draining in real time are individual accounts, which Railway hasn&#8217;t confirmed. If you run models on AWS, look in CloudTrail for <code>InvokeModel <\/code>from unfamiliar IPs, regions or model IDs.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It&#8217;s often preceded by calls that probe what&#8217;s enabled (<code>ListFoundationModels<\/code>) and how invocation is logged (<code>GetModelInvocationLoggingConfiguration<\/code>). If model invocation logging was off, spend data and CloudTrail may be among the few available evidence sources.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For OpenAI, Anthropic, OpenRouter and Grok, group usage by key and look for a key that suddenly handles volumes or models your application never calls.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Moving off Railway won&#8217;t fix this.<\/strong> Any host that stores your secrets has the same exposure on its worst day. Short-lived credentials (workload identity or OIDC federation), keys bound to an IP or identity, and secrets fetched at runtime from a manager with its own audit log are what make the next incident smaller. Without them, a migration leaves the next breach the same size. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The <a href=\"https:\/\/www.reddit.com\/r\/devops\/comments\/1xc6bbp\/railways_security_incident_is_more_serious_than_a\/\" target=\"_blank\" rel=\"noopener\">community discussion<\/a> is already arriving at this point.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"After_the_fire_harden_so_the_next_one_is_smaller\"><\/span><strong>After the fire: harden so the next one is smaller<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Long-lived strings in a platform&#8217;s variable store are exposed whenever that platform is. These changes shrink the damage:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Least privilege per key.<\/strong> One key per service, scoped to the exact actions and resources it needs.<\/li>\n\n\n\n<li><strong>Short-lived credentials where possible.<\/strong> Workload identity or federation for cloud access, instead of static keys in variables.<\/li>\n\n\n\n<li><strong>Network-bound credentials.<\/strong> Static outbound IPs plus provider-side allow-lists turn a stolen key into a mostly useless string.<\/li>\n\n\n\n<li><strong>Spend caps and alerts<\/strong> on every metered provider.<\/li>\n\n\n\n<li><strong>Reference variables and a written rotation runbook<\/strong>, so rotating is a routine deploy and not a scramble.<\/li>\n\n\n\n<li><strong>Separate production from preview.<\/strong> Don&#8217;t reuse production secrets in preview or staging.<\/li>\n\n\n\n<li><strong>Know your secret inventory.<\/strong> You can&#8217;t rotate what you can&#8217;t list.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Live_updates\"><\/span><strong>Live updates<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We&#8217;ll add dated entries here as Railway publishes new information. Times are UTC.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Oct 22, 24-hour update:<\/strong> Added the section on secrets that don&#8217;t rotate cleanly, the first loss allegation (unverified), and a note on rotating into a store that may still be open. Railway restored Enterprise builds at 04:00 UTC and invalidated all Railway API tokens at 04:17 UTC. Pro, Hobby, Free and Trial follow in stages.<\/li>\n\n\n\n<li><strong>Oct 22:<\/strong> Initial post published.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"FAQs\"><\/span>FAQs<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n<div id=\"rank-math-faq\" class=\"rank-math-block\">\n<div class=\"rank-math-list \">\n<div id=\"faq-question-1792667251892\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">What is the Railway security incident?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>The Railway breach is an October 21, 2026 security incident in which an unauthorized party may have accessed customer environment variables stored on Railway, including decrypted API keys, database credentials and registry credentials.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>This post updates as Railway publishes new details. Check the update log at the bottom before you act on anything older than a few hours. TL;DR What happened &amp; what we still need to know Railway&#8217;s disclosure says it detected anomalous activity in the systems that store customer environment variables on the morning of October &#8230; <a title=\"Railway Breach: What to Rotate, and What Rotation Won\u2019t Fix\u00a0\" class=\"read-more\" href=\"https:\/\/www.getastra.com\/blog\/security-advisory\/railway-breach-what-to-rotate-and-what-rotation-wont-fix\/\" aria-label=\"Read more about Railway Breach: What to Rotate, and What Rotation Won\u2019t Fix\u00a0\">Read more<\/a><\/p>\n","protected":false},"author":24,"featured_media":49603,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[819],"tags":[],"class_list":["post-49594","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-security-advisory"],"_links":{"self":[{"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/posts\/49594","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/comments?post=49594"}],"version-history":[{"count":2,"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/posts\/49594\/revisions"}],"predecessor-version":[{"id":49622,"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/posts\/49594\/revisions\/49622"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/media\/49603"}],"wp:attachment":[{"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/media?parent=49594"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/categories?post=49594"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.getastra.com\/blog\/wp-json\/wp\/v2\/tags?post=49594"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}