On October 21, a malicious version of LiteLLM went up on PyPI. LiteLLM is the gateway a lot of teams put between their code and the model vendors, so traffic can move when one provider goes down. It was the second time this year. In March, attackers stole its publishing token and shipped two bad versions. Seven months later, it still gets 92 million downloads a month.
The October payload stole credentials. On a proxy with a database, it also dumped the tables holding every provider key that proxy stores, according to the first issue report. LiteLLM’s own root-cause writeup says the attacker was after “upstream inference API tokens.”
The way in was ordinary. A master key committed to a public repo in May and never revoked. An owner’s GitHub token that was never scoped down. A release gate loosened that same week “to accommodate the temporary push for more frequent releases.” Credentials in public repos were also how Gemini got into two companies in the nine intrusions I covered in September. No AI was needed for any of it.
You install a gateway because model vendors fail. I still think you should. I plan for three ways they fail: the endpoint errors, the answer degrades, or access is withdrawn. Error-based failover covers the first. October showed that the layer doing the switching can be the breach as well.
The failure that throws an error
The first kind is the one gateways were built for. The provider returns an error, and the gateway can see it.
On October 25, Anthropic’s status page opened an incident covering Mythos 5.1, Fable 5.1, Opus 5.5, Sonnet 5.5 and Haiku 5.5. The impact lasted half an hour. It was the fourth elevated-error incident in under two days, after three on the night of October 23.
A gateway handles this well if the fallback leaves the serving path that is failing. A chain that drops from Opus to Sonnet to Haiku on the same API walks from one error into the next.
Crossing vendors helps, with less certainty than the architecture diagram suggests. On September 3, ChatGPT, Claude and Grok had overlapping outages. Three different vendor names did not guarantee a healthy route.
The error code can also lie. On June 2, OpenAI’s Codex requests got HTTP 429s for about fifty minutes because a broken internal rate limiter flagged them by mistake. A policy that only backs off on 429 would have kept waiting on the broken path.
Brightwave, whose agents process billions of tokens a day, does this properly. Their writeup puts the reason in one line: “If every agent owns its own SDK calls, every provider incident becomes a product incident.” Their router can serve the same Sonnet through Anthropic, Bedrock or Vertex, and they have “seen external provider status pages turn red while our customer-facing dashboards stay flat.”
If you won’t build Brightwave’s router, there is a cheaper step: stop calling the vendor directly. Our clients are enterprise and go through Bedrock or Azure. When I started this piece in June, I called that a relocated dependency. It is one, but Anthropic’s own numbers show it fails differently. Bedrock serves the same weights on AWS hardware, with reserved capacity if you pay for it, and of the three bugs in Anthropic’s September 2025 postmortem, two never reached it at all. A cloud endpoint gives you a different failure domain. One endpoint is still one endpoint, though, and resilience still needs a second path you have tested.
It has limits. Azure OpenAI had its own multi-region outage on May 29, about three hours of elevated errors. And the region is still a single point. When DynamoDB’s DNS broke us-east-1 in October 2025, a primary and a backup in that region went down together.
The first failure is close to solved, as long as you read the errors correctly. Everything in this section works because the failure shows up in your own request outcomes.
The failure that returns 200
The second kind doesn’t. The provider answers with a valid response, the gateway logs a success, and the content is worse than it was last week.
That September 2025 postmortem, which I wrote about in April, is the clearest case. A routing bug sent Sonnet 4 requests to the wrong servers. On Anthropic’s own platform, it affected 16% of them at the worst hour. On Bedrock, its peak was 0.18%. Users got worse answers, and no errors.
The same postmortem admits that “the evaluations we ran simply didn’t capture the degradation users were reporting.” This April’s postmortem, about the Claude Code product layer rather than the model API, says it again: “neither our internal usage nor evals initially reproduced the issues.” Twice in seven months, the vendor with the weights, the infrastructure and the eval team learned about a drop from its users.
A complaint can outrun the measurements too. On October 24, Thariq Shihipar, who had announced his departure from Anthropic’s Claude Code team nineteen minutes earlier, posted “wait is it just me or is Opus 5.5 dumber.” By the next afternoon it had passed 900,000 views.
One public instrument for that question is LiveNerf, an open-source tracker that asks Opus 5.5 the same 78 hard questions every day through Claude Code, with a control model alongside. Through October 23 it shows no change. It also fixes effort, so it would not see a harness default changing, which is how the March drop started. The window covering his post has no reading yet. Its answer keys aren’t perfect either. Its author flags eight that look wrong and thirty ambiguous questions, and reruns the result without them. What it offers is a panel, settings, error bars and a decision rule anyone can inspect.
The vendor’s evals missed real drops twice. This complaint isn’t settled either way. The tracker’s finished windows end before the post, and it measures a narrower slice of work than one person’s sessions. A fixed reference doesn’t settle every complaint. It tells you which claim you have actually tested.
Why not let the models check each other
The obvious fix for a missing reference is a second model. Run two vendors side by side and treat disagreement as the signal. In June I also wrote that there was no established technique for that. I was wrong about the research. I still think I was right about production.
The case against using models as a vote is strong. A study covering more than 350 models found that, on one leaderboard dataset, models agreed 60% of the time when both were wrong, and that more accurate models had more correlated errors. A nine-judge panel drawn from seven model families measured out at about two independent votes. Add the self-preference research I covered in May. Different model names don’t buy independent votes.
Panels still help on specific tasks. A PoLL study found that juries of smaller models from different families beat a single large judge across six datasets. That does not establish independent votes or a production alarm. Separately, AWS researchers used cross-model consistency to catch hallucinated passages. Disagreement between families carries signal.
What I still cannot find is anyone who has published running it as a live production alarm, with a false-positive rate. That would change my view, and it could appear any month. Until it does, the distance between a benchmark result and a page at 3 AM is where the cost sits. A voting panel decides. A detector only raises its hand, and something outside the models has to decide whether the hand means anything.
The failure with no status code
Both of those failures assume the model is still there. The third one, in June, took it away while it was healthy. On June 12 the Commerce Department sent Anthropic the export directive I wrote about the next day, and Anthropic switched Fable 5 and Mythos 5 off for everyone. GitHub’s changelog said the same day that “access to Claude Fable 5 has been suspended across all GitHub Copilot experiences.”
No routing setting can restore a withdrawn model. The model was withdrawn on purpose by people outside the vendor. Fable came back to Copilot nineteen days later. Moving to a cloud host does nothing here either. AWS also revoked access to Fable 5 and Mythos 5 on Bedrock “for all users in all regions,” at Anthropic’s request. The cost landed on whoever had built on Fable in the three days it was available, and on their users.
The only defense I can see is a second model family that already passes your evals, so the gateway has somewhere to switch. A model name sitting in a fallback array, untested against your workload, is a hope. One that passes your evals is a plan. The plan costs something every month you don’t need it: prompts tuned for two families, evals run against both, a second contract. Most teams will not pay for it until they have lost a model once.
Where the anchor goes
“Passes your evals” keeps coming up in this piece, and it carries more of the defense than the gateway does. The second failure, the one nobody pages you about, gets caught by something outside the models or by your users. Tests. A compiler. A schema validator. A fixed question panel with known answers. A person who knows the domain. My own review pipeline works that way: three review agents, each checking what the previous one let through, and me on top. It is a hierarchy that ends in a person, not a panel of equals.
What I would put in place.
Failover on errors, classified by source. Provider-wide 5xx and timeouts move traffic. Back off when you can tie a 429 to your own quota. A burst of unexplained 429s across keys is a reason to investigate and reroute, whatever the status code claims. Timeouts measured in seconds.
The same model on a second host before a second vendor, in a different region. Anthropic’s 2025 numbers show that hosts fail differently.
A second model family that already passes your evals, for the day the first one is withdrawn.
The gateway treated as a vault. Pin a reviewed release and its digest, knowing that a valid hash or signature doesn’t make a compromised release safe. October’s bad wheels came out of LiteLLM’s own pipeline. Scope credentials per workload and environment, limit what the gateway can reach, and revoke an exposed key immediately. If your gateway’s database holds every provider key you own, October 21 was a dry run of your incident. Brightwave cites the March LiteLLM compromise as one reason it owns that layer itself. That works if you have Brightwave’s engineers.
A canary with a fixed reference. LiveNerf’s design is public and small: a fixed panel, a pinned client, explicit settings, a control arm, and a decision rule written down before the data comes in. Its rule needs two ten-day windows to call a change, which is fine for a study and slow for an alarm, so set your own effect size, check frequency and false-positive budget. Run one on your own task. Put a second model on a sample of traffic as a detector, and treat a disagreement as a reason to look. Page only on a threshold you set in advance, backed by a test, a task-level canary or a domain reviewer.
The vault is the item that failed at LiteLLM in October. Nobody there needed a new technique. The key had been spotted in May and was never revoked.
If your primary model got a third worse tomorrow and kept returning 200, what in your stack would notice before your users did? And if it disappeared on a Friday by order of someone who has never heard of your product, how long until you were running on something else?