Validation, Retry & Feedback Loops

When a model's output fails validation, retrying with the specific error attached is one of the most reliable correction patterns. This lesson covers when retries help, when they don't, and how to structure feedback loops.

The Retry-With-Error-Feedback Pattern

Standard pattern for correctable failures:

1. Model produces output
2. Validate the output
3. If validation fails:
     - Append the specific error message to the conversation
     - Ask the model to retry, given the error
4. Repeat up to N times

The key word is "specific." Don't say "something was wrong" — say "Field total_amount is required but was missing." The model uses the error to correct.

What Retries Fix

• Format mismatches. The model got the data right but used the wrong date format. Tell it the format; it fixes it.
• Structural output errors. Missing fields, wrong types, malformed JSON (when not using tool_use). The model corrects when shown the issue.
• Failed validation rules. Calculations that don't sum, dates that don't make sense given context, IDs that don't match expected patterns.

What Retries DON'T Fix

Retries do not work when the required information is simply absent from the source. If the document doesn't contain a PO number, retrying won't make one appear. The model will either fabricate (bad) or repeatedly fail to find it (better, but still failure).

For absent information, the right pattern is different: design the schema to allow null, and design the prompt to instruct the model to set the field to null when not found. No retry needed — the "missing" case is a valid output, not an error.

The Self-Correction Flow

Input: invoice document
First call: extracts data
Validation: total_amount is missing
Retry call:
  Original document
  + "Your previous extraction was missing total_amount.
     The total appears at the bottom right of page 2.
     Please retry, including total_amount."
Output: complete extraction with total_amount

Tracking detected_pattern for Improvement

If your task is dismissing false positives (e.g., security review where reviewers reject some flagged issues), track what pattern triggered each false positive. Over time, you see clusters: "all false positives in category X are dismissed because Y." That's the signal to either improve the prompt for category X or temporarily disable it.

Retry Budget

Set a maximum retry count. After N retries (typically 2-3), give up and either escalate or accept the partial result. Infinite retry loops happen when the underlying issue is unfixable (absent information) and the validation logic doesn't recognize that.

Skills to Develop

1. Append specific validation errors to the prompt on retry — not generic "please try again."
2. Design schemas to make "not found" a valid output via nullable fields, so retries aren't needed for absent data.
3. Track what patterns trigger false positives to improve prompts over time.
4. Set retry budgets to prevent infinite loops on unfixable issues.