In early 2023, a lawyer used ChatGPT to help write a legal brief. The AI helpfully provided several relevant case citations to support his arguments. The lawyer filed the brief in federal court. The opposing counsel checked the citations. None of the cases existed. ChatGPT had invented them — case names, citations, even judicial quotes — all completely fabricated but presented with perfect confidence.

The lawyer was sanctioned. The story made headlines. And people asked: how does an AI system confidently cite cases that never existed?

The answer reveals something fundamental about how these systems work, and why certain problems aren't bugs to be fixed but features inherent to the architecture.

Large language models don't know things. They predict patterns. When ChatGPT generates text, it's not retrieving information from a database of facts. It's calculating probabilities: given everything I've seen before, what words are likely to come next? The model had seen thousands of legal briefs with case citations following certain patterns. It knew the format. It had learned that certain types of arguments cite certain types of cases. So it generated text that looked exactly like legal citations, following all the patterns it had learned.

It just didn't check whether the cases actually existed. Because checking would require something the model doesn't have: a distinction between pattern and truth.

This is called hallucination, and it's not a glitch. It's what next-word prediction does when pushed beyond pattern recognition into domains requiring factual accuracy. The model is doing exactly what it was trained to do: producing plausible-sounding text. The problem is that "plausible-sounding" and "true" are different things, and the model has no mechanism to tell them apart.

You can reduce hallucination through various techniques. You can train models to say "I don't know" more often. You can use retrieval systems to ground responses in verified sources. You can add fact-checking layers. But you can't eliminate it entirely without fundamentally changing what the model is.

The second problem is drift, and it's subtler. When you fine-tune a language model to be better at one thing, it often gets worse at other things. Train it to refuse harmful requests, and it might start refusing harmless requests too. Optimize it for mathematical reasoning, and its creative writing degrades. Researchers call this the alignment tax. Making models safe, helpful, and aligned with human values costs capability.

This happens because the model's knowledge isn't modular. It's more like a vast web of statistical associations where everything influences everything else. Pull one thread and distant parts of the web shift in unpredictable ways. The model becomes a palimpsest — layers of training and adjustment and correction, each layer partially overwriting what came before but never completely erasing it.

The third problem is bias, and it's the most controversial. Language models are trained on text from the internet — news articles, books, websites, social media. That training data contains all of human culture's biases, prejudices, and conflicts. When doctors in the training data are more often described as male and nurses as female, the model learns that pattern. Models don't exaggerate these biases. They simply mirror the statistical patterns in human-generated text, which reproduces prejudices accurately enough to be problematic.

Here's the philosophical problem: perfect debiasing might require perfect cultural neutrality, which might not be possible or even desirable. Some "biases" are actually values. The statement "democracy is preferable to dictatorship" is a bias in the technical sense. So is "human rights matter." A model trained to have no opinions would struggle to give useful answers to questions that implicitly require value judgments — which is most interesting questions.

These three problems — hallucination, drift, and bias — aren't separate issues. They're different manifestations of the same fundamental reality: these models are pattern-matching systems trained on human-generated data, deployed in contexts requiring truth, consistency, and fairness.

These aren't problems that will be solved by making models bigger. GPT-4 hallucinates less than GPT-3, but it still hallucinates. More parameters don't solve drift — they might exacerbate it. Some researchers argue these problems require fundamentally different architectures. Models that maintain explicit knowledge bases. Systems that can reason about their own uncertainty. Approaches that build in human oversight rather than trying to eliminate the need for it.

The engineers who built these systems weren't naive about these problems. They knew models would hallucinate, drift, and reflect bias. They built them anyway because the capabilities were unprecedented and the problems, while serious, weren't insurmountable obstacles to usefulness.

These models are dark mirrors. They reflect human knowledge, human language patterns, human biases — all with high fidelity but no understanding. They show us what our collective text looks like statistically, without judgment about whether that's good or accurate or fair.

And perhaps that's the most unsettling thing. These systems reveal how much of human communication is pattern-following rather than truth-seeking, how much of our written culture encodes prejudices we claim to have moved beyond, how often we privilege fluency over accuracy.

The models aren't broken. They're working exactly as designed. The darkness isn't in the systems. It's in what they reveal about the data they learned from, which is to say, about us.