Recorded run
Catching what a document extraction got wrong, before it reaches the database
When an AI model reads a document, most fields come out right and a few come out invented. This is the step that finds the invented ones. Below is one real run of that step from KOTA’s menu onboarding pipeline, with its actual output.
Real
The validation code and every message shown in the results. Run on November 12, 2026 against commit d15e5d7 of the KOTA repository.
Synthetic
The menu. We wrote a small fictional menu by hand, with mistakes of the kind an extraction model makes, to stand in for the model’s output. No restaurant’s data is shown.
Not shown
The extraction from photos, which calls a vision model, and the database load. Both exist in the pipeline but were not part of this run.
The input
A structured menu for a fictional noodle house: two sections, four items with English and Chinese names, prices, dietary tags, a required noodle choice, serving suggestions, and a personality description for the phone agent.
- Beef Noodle Soup 红烧牛肉面13.95
- Wonton Soup 馄饨汤9.50
- Scallion Pancake 葱油饼6.50
- Jasmine Tea 茉莉花茶2.50
What the validation step reported
Five issues. Four were fixed automatically because the menu itself proves them wrong. One was left alone and flagged, because fixing it would mean guessing.
Beef Noodle Soup → upsell note
Mango Pudding is not on this menu. A phone agent reading this note would offer a dish the kitchen can’t make.
Reported by the code (upsell_note_ungrounded)Upsell note references unknown item(s): "Mango Pudding".
- Before
- Pairs well with "Scallion Pancake" or "Mango Pudding".
- After
- Removed
Beef Noodle Soup → aliases
A caller asking for wonton soup would have been matched to the wrong dish.
Reported by the code (alias_collides_item_name)Alias 'wonton soup' collides with item 'Wonton Soup'.
- Before
- beef noodles, Beef Noodle Soup, wonton soup
- After
- beef noodles
Beef Noodle Soup → dietary tags
Only tags from a fixed list are allowed, so the agent can’t repeat a dietary claim nobody checked.
Reported by the code (dietary_tag_unknown)Unknown dietary tag 'keto_friendly' removed.
- Before
- contains_beef, contains_wheat, keto_friendly
- After
- contains_beef, contains_wheat
Restaurant → agent persona
Peking Duck is not on the menu. Dish recommendations must come from menu data, not from a personality description.
Reported by the code (persona_mentions_menu_item)Persona references menu items; persona must be style-only.
- Before
- A warm, efficient host. Recommend popular items like Beef Noodle Soup and Peking Duck.
- After
- Replaced with a style-only persona that tells the agent to use only verified menu items.
Beef Noodle Soup → Noodle type options
Not an error the code should guess at. It goes on the review list.
Reported by the code (option_descriptions_missing)All option descriptions are missing for slot 'Noodle type' (Thin, Wide).
- Before
- Thin, Wide (no descriptions)
- After
- Unchanged, flagged for a person to review
The run also made two changes without reporting an issue. It dropped the alias “Beef Noodle Soup” because it only repeated the item’s own name, and it normalized Wonton Soup’s “contains_shrimp” to the standard “contains_shellfish”.
What it did not touch
- All four items and both sections
- Every price (Beef Noodle Soup stayed 13.95)
- The required “Noodle type” choice and its two options
- English and Chinese item names
That matters as much as the fixes. A cleanup step that quietly changes a price is worse than no cleanup step.
Limits
This step checks the extracted data against itself. It can tell that Mango Pudding isn’t on the menu. It cannot tell that a price was misread from the photo, because nothing else in the document contradicts it. That is why a person reviews the flagged list and the prices before a menu goes live.
We have built this for menus. The same pattern applies to invoices, intake forms, and other documents (extract, check against known records, flag what doesn’t match), but we have not yet delivered those, and each needs its own rules.
Have documents that need this?
Tell us what the documents are, where the data needs to go, and what you check by hand today.