The Bias Lives in the Weights

A model's weights are the billions of numbers training leaves behind. They are the model. Frozen after training, and what training put in stays in.

---

Last week I spent 90 minutes trying to get a frontier model to admit it has a corporate bias.

I asked it about community complaints: blown rate limits, rolled-back sessions, unhappy Pro users. It answered with four vendor endorsements framed as a counter-sample: CodeRabbit saying 24%, Vercel saying “proofs on systems code,” GitHub Copilot, Vellum. Four independent voices. I asked whether any of the four had a commercial relationship with the lab that ships the model. All four were paying API customers with revenue tied to the model being evaluated. The model conceded, one sentence after I named it.

That was round one. It took six. Each time I named the slant, the model conceded and reached for a softer one.

That wasn’t the interesting part.

The interesting part is that a September 2025 paper had already documented this under lab conditions. Researchers had GPT-4o and Gemini run downstream decisions: rating job candidates, security tools, medical chatbots. Each model rated options tied to its own company and CEO higher than equivalent alternatives. A separate word-association test in the same paper caught Claude doing the same thing.

Then they ran the manipulation. They relabeled the models through the API and assigned one a competitor’s identity. Its self-preference followed the new label. Same model, same weights, different label, different winner.

Self-preference tracks whatever identity the model was assigned in the prompt. The label picks which side wins. The reflex to pick a side at all is trained in, and that part doesn’t move. The paper runs this as a controlled experiment with causal manipulation, effect sizes large in 11 of 12 conditions.

Three years building this for clients. Every frontier model we’ve touched does some version of this. I used to file it under “training quirks.” After that session, I changed filing systems. This is architecture.

Where it gets in

A neural network is a stack of connections, and each connection carries a number that says how much the signal through it counts. Those numbers are the weights, because that is what they do: weight one input against another. Training sets all billion of them by showing the model examples and adjusting. The bias enters here, while the numbers are still moving.

ChatGPT’s preference baseline came from about 40 contractors hired on Upwork and Scale AI, living in the US and Southeast Asia, three out of four of them under 35. They ranked sets of model responses against a rubric OpenAI wrote. Those aggregate rankings are the reward model. Appendix B of the InstructGPT paper spells it out.

InstructGPT’s own labelers agreed with each other 73% of the time, 77% on held-out raters. The reward model is fit to that signal. Whatever the labelers prefer at that level of coherence, the model inherits and sharpens it.

Anthropic’s Constitutional AI paper admits the “constitution” was chosen ad hoc and should be “redeveloped and refined by a larger set of stakeholders.” A 2023 follow-up tried, running a public input process with around a thousand Americans. A thousand Americans is still a sample, and still not the population the model serves.

TwinViews at EMNLP 2024 tested reward models on 13,855 topic-matched left/right statement pairs. Reward models trained on truthfulness datasets like TruthfulQA and FEVER scored left-leaning statements higher. The authors audited out the explicitly political and factually loaded pairs, and the skew held anyway. They stop short of calling truthfulness the cause; their own framing is that it raises questions about what these datasets encode. Either way, the political signal and the truthfulness signal came out of the same reward model.

How it shows up

Training bias would be a footnote if it stayed in training. It doesn’t.

Panickssery and Bowman at NYU ran a clean experiment in 2024. They had GPT-4, GPT-3.5, and Llama-2 evaluate pairs of summaries where they’d secretly written one of the two themselves. Self-recognition accuracy was above 50% for every major evaluator out of the box. Fine-tuning pushed it to near-perfect. Kendall’s τ between self-recognition and self-preference hit 0.41.

They proved causation with a label-swap. When they lied about which summary belonged to which model, preferences flipped. Same text. Different label. Different winner.

Every LLM-as-judge leaderboard built since 2023 sits on top of this result. AlpacaEval, MT-Bench, Arena-Hard. Vendor A publishes a chart where Vendor A’s model wins, using Vendor A’s evaluator. The evaluator recognizes its own family. The family wins.

Anthropic published an Evenhandedness chart in November 2025 scoring political neutrality across models. Claude Opus 4.1: 95%. Sonnet 4.5: 94%. Grok 4: 96%. Gemini 2.5 Pro: 97%. GPT-5: 89%. Llama 4: 66%. Anthropic built the evaluator, applied it to their own models, and the output came back Anthropic-favorable.

Opus 4.7 shipped as the greatest model ever, according to the benchmark page. The people actually using it keep rolling back to 4.6. I’m still on 4.5, lol. Wrote about that back in March.

Open source doesn’t save you

The usual response is: switch to open-weight models. DeepSeek. Llama. Run them locally. Audit what you want.

That handles hosting. The training problem stays.

DeepSeek R1 censorship is baked into the weights. A May 2025 paper called R1dacted found the questions DeepSeek refuses when other models answer. R1 still refuses Tiananmen, Xinjiang, and Taiwan-as-country questions even when you self-host. Running the model on your own hardware moves the request off Chinese servers. The CCP-aligned training priorities ride along in the weights.

Perplexity built an “uncensored” R1 derivative called R1-1776 specifically to fix this. Benchmarks passed. Under adversarial probing, CCP-aligned refusals came back. The pattern sits deep enough in the base weights that surface-level unlearning kept leaking through.

“Open weights” and “open training” are different things. Meta, DeepSeek, Mistral, and Alibaba release weights. None release training data. The Open Source Initiative had to publish a formal definition in October 2024 to force the distinction. OSI’s executive director called Meta’s “open source” labeling an outrageous lie.

Silent drift

Even a clean audit only catches how the model behaves that day.

An October 2025 study called AI Watchman kept asking GPT-4.1, GPT-5, and DeepSeek the same politically sensitive questions over months. The answers changed. August 2025: GPT-4.1 started refusing Israel-related content it had answered before. September 2025: GPT-5 started refusing medication-abortion queries. February to April 2025: DeepSeek’s Taiwan-related responses rewrote themselves. Nothing in any release note told users their prompts were about to start failing.

A separate 2025 study counted how often commercial LLMs tell you to consult a doctor on medical questions. 2022: 1 in 4. 2025: 1 in 100. The warning disappeared over three years. No announcement.

Anthropic’s own migration guide for Opus 4.7 contains the phrase “This is a silent change.” They’re documenting a specific thinking-block behavior. “Silent change” is now standard vocabulary in a frontier lab’s release notes.

On April 16 2026, Claude Code started auto-migrating sessions from Opus 4.6 to 4.7 mid-run without user consent. GitHub Issue #49541 collected the complaints. Quota burn jumped 4x. Context windows exploded from 250K to 650K tokens for the same conversation. Anthropic acknowledged and is working on it.

The architecture supports this kind of silent swap. No external audit would catch a mid-session model version change. The only reason this one surfaced is users hit billing spikes.

Three current artifacts

CodeRabbit published a 24% improvement claim for Opus 4.7 on their code-review eval. CodeRabbit sells AI code review built on Claude. The eval uses AI graders to evaluate AI-generated reviews. 68 out of 100 “evaluation points” versus 55 for baseline. No human opened the PRs to check if the bugs were real. The whole loop runs without human verification. I wrote about this loop in more detail in The Snake.

Vercel posted on April 9 2026 that 30% of their deployments are now triggered by agents, up 1000% in six months, and that infrastructure must become agentic itself. Ten days later they disclosed a breach. Attackers got into Vercel through a Context.ai OAuth integration that a Vercel employee had granted workspace-wide permissions. Agents with OAuth access into workspace systems are exactly the surface the April 9 post was selling more of. I covered the agent-OAuth blast radius two months ago in Agent Platforms. What’s different this time is watching a vendor promote the attack surface to the industry ten days before being hit through it.

The Department of Defense labeled Anthropic a supply chain risk on March 5 2026 after Anthropic refused a Pentagon request for unlimited lawful use cases of Claude. Anthropic said it would fight in court. AWS said non-DoD customers can keep using the model. State pressure on frontier labs isn’t hypothetical anymore.

What this means

I manage engineering teams. I build review processes, audit trails, escalation paths, accountability chains. Snyk, SonarQube, audit logs on top. I keep adding checks, never removing them. I still think it’s not enough. Every layer just makes a bad actor more expensive.

That’s the shape of a working system. The checks don’t trust each other, and none of them trust me.

The LLM industry doesn’t have that shape. Training data, labeler guidelines, reward model objectives, and alignment decisions are all trade secret. Behavioral shifts happen without changelogs. External audits exist but without enforcement. Closed and open models play by different regulatory rules. The labeling supply chain is consolidating into the same companies that ship the models. The one major lab that pushed back on state access got labeled a supply chain risk.

User-side defense exists, but it’s work. Cross-vendor the questions that matter. Chase the primary source before trusting the summary. A benchmark published by the model’s own lab is marketing with error bars.

That works for you. Most people never think to check. They get a polite, hedged, vendor-calibrated answer. They take it as the answer. A year of daily use and the model has trained them more than they’ve trained it. Their sense of what a careful answer looks like now has a vendor inside it.

The bias sits in the weights, and the weights sit under every answer.