Bugs that look like success: testing AI-generated content

The card that passed every check

During the final end-to-end test before LAFwords launched, I tried a deliberately broken input. I typed asdfgh as a word to learn, and when the app asked whether I meant something else, I chose to keep my original word.

The app built a card anyway. It had ten example sentences, an AI image, pronunciation audio and a success dialog, and it went into my study queue. Here are some of the sentences, with the word the card was teaching in bold:

"I'm sorry, but asdfgh isn't a recognized English word, so I can't create valid usage samples for it."
"Please provide a real English word so I can generate varied usage samples."
"Tell me a real target word and I will produce ten diverse sentences."

Every sentence was a piece of the model's apology. And every automated check had passed. All the words were real dictionary words. The entered word appeared in a sentence. There were enough sentences.

That bug is the best summary I have of what testing LLM output is like. The dangerous failures don't look like failures. They look like success.

This post is about how I check the content LAFwords generates: what broke, what each break taught me, and what my checks still don't catch. My background is in software QA, with hands-on work in prompt engineering and in evaluating non-deterministic LLM output. This has been the most interesting testing problem I've worked on.

What gets generated, and why errors cost more here

LAFwords is a vocabulary app for intermediate and advanced learners. You enter a word you met, in English or Spanish, and the app builds a card: ten sentences that show the word in different situations, each with a short meaning, a context label and an image, plus native pronunciation. There are no translations. Ten varied contexts give you a feel for the word's range, so you can use it in situations the ten sentences never showed.

That design allows no compromise on quality. A learner can't spot the errors because they don't know the word yet. If a sentence uses the word in the wrong sense, or invents a form that doesn't exist, the learner memorizes the mistake. The generated content is the curriculum.

1. The model can't check its own inventions

It gets worse: language models are prone to inventing word forms. They apply patterns like -ity and -ness that work for thousands of words, but real vocabulary doesn't follow them everywhere—borrowed words and one-off formations break the rules.

Early on, the model produced holisticity (for holistic) and rigorousnesses (for rigorous). Neither is an English word. Both look plausible, which is the problem.

The obvious idea is: let's ask the model whether a word is real! It doesn't work reliably. The reason is simple: we judge the invented word using the same model knowledge that generated it. A checker that shares the generator's blind spot passes exactly the errors you most need to catch.

So the first lesson I learned: don't let the model check itself. The fabrication check doesn't use an LLM at all. Every word in every sentence is checked against real spelling dictionaries, the same kind a word processor uses. The dictionary brings ground truth from outside the model. If a sentence contains a word the dictionary doesn't know, it's rejected and regenerated.

A side lesson came from the re-test. The model used rigorousness, and my first reaction was that it was another invention. It isn't. It's in the dictionary. My instinct as a reviewer was no better than the model's, and only the external source could decide.

2. The quality rule that caused the defect

Why was the model inventing words in the first place? Because I asked it to.

The early prompt required several different forms of the word across the ten sentences, so the learner would see decide, decided, deciding, decides. It sounded like good teaching. But holistic has only two real forms: holistic and holistically. When a requirement can't be met with real words, the model doesn't refuse. It fills the gap with plausible inventions.

Removing the requirement fixed it. A re-test of holistic produced the two real forms and nothing else. Different forms are still welcome when they appear naturally, but they are never forced.

So, the next lesson: keep your eye on edge cases where the input makes the constraint impossible to follow. Ask what the model will do. Unfortunately, it usually sticks to the letter of the rule and pays for it in correctness.

3. Reasonable gates, garbage output

Back to asdfgh. Each check in the pipeline was sensible on its own. Are all the words real? Yes, an apology is made of real words. Does the entered word appear? Yes, once, inside the apology. Are there enough sentences? Yes.

What I missed was checking the relationship between each sentence and the target. Every sentence highlights the word it teaches. In the broken card, the highlighted words were please, tell, choose: real words, just not the word the card was about.

The fixes were specific. Each sentence's highlighted word must now match the entered word or a form of it, and anything else is rejected. Known refusal phrasings are detected and rejected too. And the exact sentences from that card became a regression test, so this failure can't quietly come back.

The lesson I took here: a list of property checks is not a specification. Each gate checked that something looked normal. None checked that the output did its job.

4. When the check measures a proxy

In September, I noticed a card for triage in production with ten medical sentences: the ER, a field clinic, a disaster drill, a triage nurse. Every sentence was correct. But triage has a second sense that an office worker probably meets more often: prioritizing anything scarce, like a bug backlog, an inbox, or a budget. That sense was missing entirely.

Ten sentences in one sense defeat the point of the card. They teach one corner of the word ten times.

The worse news was in my review checklist. It checked variety by counting distinct context labels, and the triage card had ten different labels: "ER", "disaster response", "drill" and so on. By my own checklist, the card passed. The checklist measured a proxy, labels, instead of the thing itself: meanings.

The cause turned out to be a model setting. To save cost during bulk generation, the model ran with low reasoning effort, and at that level it follows its first association and stays there. The original card had zero non-medical sentences out of ten, and a regeneration had two. After raising the setting one step, the next generation gave five: IT support, a newsroom, email, a budget, project planning.

Two lessons. When a check uses a proxy, test it against a known bad case before trusting it. And a setting chosen for cost is a quality decision, whether or not anyone treats it as one.

5. Strict gates fail silently

The dictionary check is the strongest gate I have, and it has a cost I only found by looking for it.

During Spanish testing, the dictionary rejected four real words as inventions: afrutadas (fruity), carísimos (very expensive), desglasar (to deglaze) and risotto. Spanish builds forms like carísimo freely, and dictionaries don't list them all. Loanwords and specialist terms are often missing too.

Nobody noticed, because nothing visibly broke. The rejected sentences were regenerated, the card still had ten sentences, and it shipped. But the replacements were plainer. A strict gate quietly strips out exactly the rich, natural vocabulary an advanced learner wants.

English has the same problem. Our English dictionary doesn't know triaging, a perfectly regular form. And the check that catches refusal text once threw out "I cannot condone lying to my friends", a natural sentence for condone, because it sounded like the model saying no. In both cases the sentence was quietly replaced, and the card looked fine.

A gate's misses show up as bugs. Its false alarms show up as nothing at all.

My lesson here: watch what the validator removes, not only what it lets through.

6. Every new language reopens old failures

Spanish needed its own dictionaries and its own rules. Accents, for example, are part of the word: sí and si, tú and tu, papá and papa are different words. A human review of generated Spanish cards then found failure types that English testing never showed:

Another language leaking in. Italian congiuntivo instead of Spanish subjuntivo in a grammar field, and Chinese characters inside a Spanish context label.
The model's doubt written into the output. A part-of-speech field that read "verbo? (incorrecto)".
One generic meaning copied onto all ten sentences, instead of the meaning the word has in each sentence.
A neighboring word taught instead of the target. restar (to subtract) for resto (remainder).
Gender that changes the meaning. La radio is broadcasting; el radio is a radius. A wrong article silently changes the word.

Most of these get past the dictionary, because it checks the sentence, not the metadata fields around it. Inside the sentences, the words involved are real.

Italian and German come next. German uses gender to separate meanings even more often than Spanish does. I treat the list of past failures as the first draft of the test plan for each new language.

7. When the prompt and the validator disagree

In October, during a large import of new words for the app's word library, more than half of the cards came out short: six, seven, eight, or nine sentences instead of ten. There were no errors and no timeouts. I noticed it in the admin word list, where sixes and sevens kept showing up. The logs gave the scale: of 698 words generated in twelve hours, only 326 had all ten.

The cause was two rules that disagreed. The prompt allowed close relatives of the word: decide, decision, decisive. The validator, written after asdfgh, accepts only grammatical forms of the same word, so it rejected abhorrence on a card for abhorrent. The target-word check caused 907 of 1,045 rejections, and most of them were derived words the prompt had invited. Both rules were reasonable, and both sat in the same methodology document, a few lines apart. Nobody noticed that they contradicted each other.

Rejections alone would not have produced short cards, because a retry loop refills rejected slots. But in the batch import, one setting meant two things: the minimum number of sentences a card needs to ship, and the point where generation stopped trying. Once six sentences passed, the loop stopped. It took both bugs together to ship short cards, and every one of them passed as finished.

We chose one rule: a card shows the entered word and its grammatical forms, never derived words, which belong on their own cards. Generation now keeps going until there are ten sentences or the retry limit is reached. Replaying the old rejections through the new validator showed that 93 of them had been real forms of the word all along, lost to a gap in the tool that maps a word to its dictionary form. We paused the import, fixed the rules, and regenerated 833 cards before any images were made, so no image money was spent twice. Of the 2,599 imported words, 2,594 now have ten sentences.

The lesson: the generator's instructions and the checker's rules are one specification, so test them against each other, starting with every example in the prompt. Running decision for decide through the validator would have failed in a second. And watch the distribution of your output, not only pass and fail. A column full of sixes and sevens told me more than any single check.

What the checks still miss

Now the honest part. The lessons above apply to almost any LLM product. What follows doesn't. These are gaps specific to a vocabulary app, and that is exactly why they are still open: problems rooted in the domain don't give way to general rules. I list them less as lessons than as a sample of how varied quality problems get, even inside one narrow product:

Sense variety has no automated check. It depends on the prompt and the model setting. The triage fix made it better, but it's not guaranteed. This is the biggest open gap.
Sentences about the word instead of with it. For example, instead of using the word bass, the model defined it: "The term bass often refers to the lowest range in a musical ensemble." Every word is real, and the target is present, so the gates pass it. Only the prompt discourages it.
Words outside the word family. Mousetrap is highlighted as a form of mice. It's a real word, so the dictionary accepts it.
Grammar. A dictionary checks words, not how they combine. The same grammar mistake repeated across a card gets through.
Sensitive words. The classifier that flags vulgar or offensive words under-flags in practice.
Images that contradict the sentence, such as a person of a different gender than the sentence describes.

If you are new to evaluating non-deterministic LLM output, expect a list like this in your own domain. It will be different, and it won't be shorter.

Some of these problems can get a cheap mechanical check: a question mark in a grammar field is easy to catch. Others need a semantic judgment, and that is where an LLM judge may finally earn a place. Not to decide whether a word exists, but whether ten sentences really cover different meanings.

As long as these gaps remain, human review covers what the gates don't.

How I keep it honest

Three habits hold this together. None of them is clever. They just stop knowledge from leaking away between one fix and the next.

A failure gallery. Every real failure goes into one file: the actual output (verbatim, where I still have it), why it passed the checks, what caused it, and the rule it produced. Real cases only, no hypotheticals. Entries are never deleted when fixed, only marked fixed, because fixed failures come back.
One methodology document with a decision log. What makes a good card is written down in one place, with the date and the reason for every rule. Any change to the generation prompt or a validator must update that document in the same pull request. On its own, it didn't prevent the contradiction in section 7; that takes tests, not just a document.
Regression tests built from real output. The asdfgh sentences are a test fixture. A real failure makes a better test than one I would invent, because I would never have invented it.

Much of the day-to-day review is done with AI assistants, which makes the written rules more important, not less. An assistant reviewing cards follows the same checklist a human would, and the triage story shows what happens when the checklist is wrong.

If you're building something similar

Don't let the model grade its own facts. Bring ground truth from outside: dictionaries, schemas, a database, anything the model didn't write.
Read every prompt constraint as a test case. What does the model do when the input makes the constraint impossible?
Check that the output does its job, not only that it looks normal.
Test each check against a known bad case, especially when it measures a proxy.
Watch what your gates remove. False alarms cost quality that nobody sees.
Keep real failures, verbatim. They are the best test plan you will have for the next model, the next prompt or the next language.
Test the generator's instructions against the checker's rules. Start with every example in the prompt; one document is not enough.
Look at the distribution of your output, not only at pass and fail. A histogram of results can show a bug that no single check reports.

LAFwords is live at lafwords.com for learners of English and Spanish. If you work on LLM quality and see a gap in this approach, I'd like to hear about it.