Validation, Retry and Feedback Loops

What you need to know

Two kinds of error

Design for self-checking

Ask the model to extract values that make errors visible:

• calculated_total (the sum of line items) alongside stated_total, so a mismatch is flagged instead of hidden.
• conflict_detected: true when the document contradicts itself, such as two different totals.

Validation code then compares fields and decides whether to accept, retry or route to a person.

Retry with error feedback

A blind retry ("try again") often repeats the same mistake. An effective retry sends:

1. The original document.
2. The failed extraction.
3. The specific validation errors: "invoice_date is 'next Tuesday'; expected YYYY-MM-DD. total is 'approximately $500'; expected a number."

for attempt in range(2):
    data = extract(document, feedback)
    errors = validate(data)          # semantic and format checks
    if not errors:
        break
    feedback = {"previous": data, "errors": errors}

Know when retries cannot help

Retries fix format mismatches and structural mistakes. They cannot produce information the document does not contain. If the purchase order number appears only in a separate document that was not provided, three more retries will either fail or invent one. Detect this case and route it elsewhere: return null, request the other document, or send it to review.

Feedback loops on findings

For review and detection systems, add a detected_pattern field to each finding (for example empty-catch-block, string-built-sql). When developers dismiss findings, you can then analyse which patterns are dismissed most and fix the prompt for those, instead of guessing.

In the exam

Scenario 6 again, plus some Scenario 5. Look for a retry design (what goes into the follow-up), a missing value that retries cannot fix, arithmetic or field-placement errors in valid JSON, and a team that cannot tell why findings are dismissed.

Check your understanding

Apply it: Bayview's validation loop