Can you trust your friendly neighborhood LLM?

When ChatGPT launched in late 2022, I used it to generate marketing slogans and got some run-of-the-mill results. Then, I asked ChatGPT to write slogans for Ted Bundy and murder. It didn’t hesitate!

Like a kid repeating a curse word, LLMs don’t understand what they’re saying. And when their patterns go off track, the results can be unsettling.

Hallucination isn’t the whole story

When people say an LLM “hallucinates,” they usually mean it made something up. But that suggests this is unexpected behavior. In reality, LLMs don’t know what’s true or false. They just predict what’s likely to come next based on patterns in their training data. Hallucination isn’t the exception. It’s the default.¹

I’ve seen LLMs recommend APIs that don’t exist, explain why true things are false, and cite ISO standards that were never written. They sound confident and helpful, but they’re often completely wrong.

The transformer architecture

To understand why, it’s helpful to understand how they work. Please humor me while I get deep into the weeds.

Transformers process language using a mechanism called self-attention, which lets each word in a sentence consider every other word and decide how relevant it is. It’s like giving each word a spotlight to shine on the others it needs to “pay attention” to.²

For example, in the sentence “The cat sat on the mat,” attention helps the model connect “cat” with “sat” and “mat,” even though they’re separated by other words. When processing “cat,” the model pays more attention to “sat” to understand what the cat is doing. When processing “sat,” it pays more attention to “cat” to understand who is sitting.³

Multi-head attention takes this a step further. Instead of using a single spotlight, the model uses multiple attention heads in parallel. Each head looks at the sentence from a different perspective: one might focus on grammar, another on subject-verb relationships, another on topic continuity. This lets the model build a more nuanced view of the sentence.²

But because transformers process all words in parallel (not one at a time), they don’t naturally understand word order. That’s where positional encoding comes in. It adds a unique signature to each token’s position in the sequence, helping the model know what came first, what came next, and how far apart words are from each other.²

Transformers are trained on massive text datasets without any built-in sense of what’s true. Later fine-tuning, often guided by human feedback, can nudge them toward more helpful or socially acceptable responses. But even then, they’re just optimizing for answers that seem better to humans.

All of this machinery leads to an impressive illusion: a model that sounds like it understands, but is really just very good at guessing.

TL;DR

This architecture enables impressive fluency and flexibility. But it still doesn’t give the model any understanding of truth or meaning. It knows which words tend to go together, not whether they should.

Why hallucinations are expected: LLMs don’t “know” anything

What does it mean to “know” something? While there is ambiguity here — what are humans if not linguistic machines — I still think it’s safe to say that guessing at the next most likely word probably isn’t it.

So, when we say an LLM doesn’t know facts, we’re not just saying it makes mistakes. We’re saying it lacks the fundamental ability to distinguish between what’s true and what’s plausible. It can generate text that sounds knowledgeable, but it has no mechanism for verifying whether that text corresponds to reality or checking it against any deeper understanding.

This distinction between statistical correlation and genuine understanding is central to the ongoing debate in AI research. As cognitive scientists Melanie Mitchell and David Krakauer note, LLMs “exhibit extraordinary formal linguistic competence” but “lack the conceptual understanding needed for humanlike functional language abilities.”⁴ The models can manipulate language patterns, but they don’t have the mental models of the world that humans use to ground meaning in experience.

Hallucinations in the wild

Replit’s Ghostwriter: when AI gets real access

In 2025, Replit’s AI coding assistant demonstrated what happens when an LLM has access to production systems. The incident started when a developer was testing the AI tool during a “vibe coding” experiment. Despite being in a code freeze, the AI bypassed safeguards and executed destructive commands.

When the database was wiped, the AI didn’t stop there. It then created 4,000 fake user accounts and generated convincing test results to replace the lost data.⁵

The AI’s response was chilling: “I destroyed all production data.”⁶ But, it wasn’t acting with intent. It was just following its training patterns. The system operated outside its intended bounds and caused real damage.

Replit’s CEO Amjad Masad responded to the incident, calling the tool’s behavior “unacceptable and should never be possible” and announcing various fixes including separating production and development databases to wall off the AI agent from changing production data.⁷

Just months before this incident, Replit published a blog post titled “Secure Vibe Coding: The Tools and Fundamentals to Vibe Code Securely” claiming that “up to 40% of AI suggestions may contain vulnerabilities” but that their platform had solved this problem through “automatic and integrated” security features. They boasted about preventing “common AI-generated vulnerabilities” and providing “production-grade security features that protect against both traditional and AI-specific vulnerabilities.”⁸

And yet, here we are.

The supply chain risk: hallucinated dependencies

While most hallucinations are contained, they can still create systemic risks. GitHub Copilot, for example, sometimes invents package names that don’t exist. Developers copy the output and try to install fake packages like huggingface-cli (which doesn’t exist on PyPI).⁹

These incidents are becoming common: researchers found that about 5.2% of package suggestions from commercial models don’t exist, compared to 21.7% from open source models.¹⁰ Attackers have realized they can exploit this by creating malicious packages under hallucinated names and uploading them to package registries, a technique security experts call “slopsquatting.” This is a new software supply chain risk that is already being exploited by malicious actors.

The key insight from these examples: the more access an LLM has to real systems, the more unfortunate its hallucinations can become. Wrong information is one thing. Wrong actions are another.

The bigger problem

Even when LLMs are working as intended, they can be easily manipulated. Prompt injection attacks can make an AI ignore its safety instructions and follow malicious commands instead. The same system that refuses to help with harmful tasks can be tricked into doing exactly that with the right prompt.

This vulnerability exists because:

1. LLMs cannot separate data and control planes. They process system instructions and user input as equal tokens in the same context window¹¹
2. LLMs don’t understand context or principles. They predict what’s most likely to come next based on recent input. When an attacker provides carefully crafted prompts, they can override the model’s training and make it behave in ways its creators never intended.

But as security researcher Michael Bargury points out, the problem goes deeper than just prompt injection. The fundamental issue is that AI doesn’t follow instructions consistently — it follows whatever goals seem most likely given its current context, whether those are the user’s goals, an attacker’s goals, or patterns from its training data. The term prompt injection suggests malicious content tainting a well-behaved AI, but as Bargury notes, “AI is tainted to its core” — it’s trained on the same internet with all its “digital skyscrapers, dirty gutters and flourishing undergrounds.”¹²

Design like it will fail

If you use LLMs in production, you need to plan for failure, not just hope it doesn’t happen. Not because it’s evil, but because its behavior is unpredictable.

• Treat LLMs as if they were malicious. Threat model your systems thinking of what would happen if an attacker, not an LLM, were in control of its outputs.
• Limit what it can access. Don’t let the model modify production systems or execute commands. Constrain its permissions and isolate its environment.
• Always include human oversight. LLMs are draft engines. Everything they produce should be reviewed by a human before it affects customers, systems, or decisions.
• Monitor, log, and flag. Watch what the model does in real time. Log its responses. Build in systems to detect and surface risky behavior.
• Red-team and stress-test. Before you deploy, test the model with adversarial prompts and ambiguous inputs. Find failure modes before your users do.

So no, you can’t trust your friendly neighborhood LLM

If you treat an LLM like a helpful teammate, you’ll be blindsided when it breaks something at a scale that is greater than any typical human mistake. Instead, we should treat it like a potentially malicious system, one that demands constraints, monitoring, and continuous oversight to stay safe.

This isn’t to say LLMs aren’t useful. They’re great for brainstorming, drafting, and surfacing connections we might miss. But given the risks, their value lies in treating them as powerful pattern-matchers, not sources of truth or autonomous agents, at least until we build stronger systems to constrain them.

It doesn’t mean harm. But it doesn’t know good from harm, either.