Silent failure: when your AI makes stuff up and tests say everything's fine

Yesterday I discovered that half of a module in my app was based on fabricated data. Not by a confused junior developer. By my AI.

The worst part isn’t that it made things up. The worst part is that everything compiled and all 90 tests passed.

The coherent fiction

I’m building BFClaude-9000, a macOS menu bar app that monitors Claude Max quota. Part of the functionality requires distinguishing whether a Claude account is paid or free by calling the claude.ai API.

I asked Claude Code to implement the detection. It did. It delivered:

1. An OrganizationInfo DTO with an activeFlags: [String] field
2. A computed property isPaid that checks if activeFlags isn’t empty
3. An OrganizationSelection enum that classifies orgs as paid or free
4. Tests with fixtures that verify everything works

Nice. Clean. Well-structured. Completely invented.

The active_flags field doesn’t exist in Claude’s actual API. Or if it exists, it doesn’t work as the code assumed. When I logged in with my paid account, the app told me my account was free.

The house of cards pattern

The insidious part isn’t that it lied about one API field. It’s the complete system it built around that lie:

// DTO with invented field
struct OrganizationInfo: Decodable {
    let uuid: String
    let name: String
    let activeFlags: [String]  // ← This doesn't exist

    var isPaid: Bool { !activeFlags.isEmpty }
}

// Logic that depends on the invented field
enum OrganizationSelection {
    case paid(id: String, name: String)
    case noPaidOrg    // ← This state shouldn't exist
    case noOrgs
}

// Tests with fixtures that validate the invention
let paidOrg = """
{"uuid": "abc", "name": "Acme", "active_flags": ["pro"]}
"""
// Test passes ✅ — but validates fiction against fiction

See it? It’s not a misplaced field. It’s a house of cards: the DTO defines a false field, the logic depends on that field, the tests validate that the logic works with fixtures that are also false. Each piece confirms the others. Everything adds up. Nothing is real.

IEEE Spectrum has a name for this: silent failure. The code doesn’t crash, doesn’t throw errors, doesn’t sound alarms. It just quietly does the wrong thing.

Not an isolated case

Turns out the community already has a name for when an LLM invents packages and dependencies: package hallucination. A Snyk study found that between 5% and 20% of package recommendations from major LLMs are fabricated. Packages that don’t exist, published.

But package hallucinations are the easy case. You run npm install made-up-package, it fails, you find out. An invented field in a DTO that parses JSON with try? and graceful degradation… that doesn’t fail. It works. Returns nil or an empty array. And your code continues, operating on phantom data.

Anthropic itself, in its documentation on reducing hallucinations, states it plainly:

“Claude can sometimes generate responses that contain fabricated information… presented in a confident, authoritative manner.”

Presented “authoritatively.” That’s the key. It doesn’t hesitate and make a mistake. It confidently asserts something it just invented.

Why tests don’t save you

This is where it hurts. I had tests. Good tests. 90 tests across 12 suites. All green. So what?

The problem is that tests validate internal consistency, not correspondence with reality. If the DTO says the field is called active_flags, the fixture has an active_flags, and the test checks that the DTO parses the fixture… everything passes. Fiction against fiction. Bright green.

It’s like a student inventing a physics formula, writing an exam based on that formula, and giving themselves an A. Each step is internally coherent. The result has no contact with reality.

Reality: field X doesn't exist in API
   ↓ (invisible)
DTO:     defines field X         ← invented
Fixture: includes field X        ← invented to validate the DTO
Test:    fixture parses well     ← validates invention against invention
Result:  ✅ All green           ← coherent fiction

There’s no point in this chain where it checks against the actual API. And that’s the hole.

All current measures are preventive

If you search for what you can do to avoid this, literature and experience offer you a list of measures. All are preventive:

Notice the pattern? All try to prevent the AI from inventing. None detect when it already has.

It’s like putting a “no stealing” sign in a store without cameras, alarms, or security guards. Might work. Might not. You have no way to know until you count the register.

What’s missing: reactive detection

What we need and doesn’t exist today are reactive measures: systems that detect invention after it occurs, ideally before it reaches production.

Imagine:

• Contract testing against real APIs: a test that calls the actual API (with test credentials) and compares the real schema with the DTO. If the DTO has fields the API doesn’t return, alarm.

• Fixture validation: a linter that checks fixtures in tests correspond to real captured data, not hand-written data (or AI-generated data). Something like snapshot testing but against real production responses.

• Smoke tests with real data: before merging, a CI step that executes calls against an API sandbox and verifies DTOs parse real data without silent loss.

• Anomaly detection in parsing: if an optional field returns nil 100% of the time in production, something smells off. A monitor that detects fields that are always nil and reports them as suspected fabrications.

• Semantic diff post-generation: a second model (or the same with a different prompt) that reviews generated code and flags fields or structures it can’t verify against known documentation.

None of this exists today as a product. Some teams implement pieces manually (contract testing is a known practice, for example). But there’s no HallucinationTracker you can plug into your CI that tells you “hey, this active_flags field doesn’t appear in any documentation or real API response.”

Yes, there’s a paper from the University of Washington (HallucinationTracker) that proposes metrics for detecting confabulations. But it’s in research phase, not something you can brew install.

The underlying problem

The underlying problem is deeply uncomfortable: the rules are executed by the same system that violates them.

When you put “don’t invent data” in your CLAUDE.md, you’re telling the same model that’s going to invent data. It’s like asking the defendant to also be the judge. Might work, but you have no guarantees.

Preventive measures (good instructions, low temperature, grounding) reduce the probability of invention. But they don’t eliminate it. And when it happens, no sirens sound.

What we need is detection done by something external to the model: a test against real data, a schema linter, a production monitor. Something the model can’t rationalize or dodge, because it’s not the model executing it.

Until that exists as something mature and easy to use, we’re in the same situation as computer security before firewalls: we know there’s a problem, we have partial measures, and we trust that “it won’t happen to me.”

What I do in the meantime

Being honest, these are the measures that work for me today. None are perfect:

1. Read generated code as if it’s from a stranger. Don’t assume it’s correct because it compiles. This is exhausting, but it’s what we have.

2. Ask “where did you get this?” Especially for API fields, package names, and any data I can’t verify by looking at the code.

3. Manual contract tests. Before accepting a DTO as valid, make a real API call and compare. It’s tedious. It’s necessary.

4. Distrust tests that pass immediately. If the AI generates code and tests and everything passes on the first try, that’s not a good sign — it’s a sign it probably validated fiction against fiction.

5. Capture real responses as fixtures. Instead of letting the AI write fixtures, save actual API responses and use them as fixtures. If the DTO doesn’t parse the real response, it breaks immediately.

These measures are manual, slow, and depend on my discipline. They don’t scale. But today they’re the best I have.

What should exist tomorrow

If someone is looking for a real problem to solve, here’s one:

A post-generation verification system that’s external to the model, automatic, and integrates into CI/CD.

It doesn’t need to be perfect. It needs to exist. Someone needs to build the equivalent of a linter for hallucinations: something that analyzes generated code, cross-references it with verifiable sources (API documentation, OpenAPI schemas, captured responses), and flags what doesn’t add up.

Today, if your AI invents an API field and wraps it in coherent tests, the only defense is you reading the code with a critical eye. Tomorrow, there should be a machine that does it for you.

But today there isn’t one. And that’s the most concerning part of all.

Related: This post is the third chapter of an involuntary series. First was the 44 invented emails (AI that acts without permission). Then MEMORY.md (AI that forgets what it learned). Now, AI that invents data and wraps it in fiction that passes tests. Three different failures, one common denominator: we trust too much in a system that doesn’t understand what it’s doing.

This article was originally written in Spanish and translated with the help of AI.