Explicit Criteria to Reduce False Positives

Vague instructions feel reasonable and don't work: "be conservative," "only report high-confidence findings," "use good judgement." None of these actually constrain behavior — they read as guidance while leaving the model to decide what "conservative" means, run to run. What works instead is explicit criteria: "flag comments only when the claimed behaviour contradicts the actual code behaviour" gives the model something concrete to check against.

Practical guidance beyond the basic contrast:

• Write criteria defining which categories of issue to report (bugs, security) versus which to skip (minor style, local team patterns) — rather than leaning on confidence-based filtering, which just relocates the vagueness into a threshold nobody can actually calibrate.
• Define severity levels with concrete code examples at each level. Labels alone ("high," "medium," "low") without anchoring examples classify inconsistently between runs, because the boundary between levels is exactly the thing that needs anchoring.

The discriminator: false positives are a trust problem, not just a precision problem. A noisy category doesn't just waste attention on its own bad flags — it makes reviewers start skimming past all findings in that category, including the accurate ones. This is why temporarily disabling a demonstrably noisy category, while its criteria get rewritten with concrete examples, can be the right immediate call rather than a concession. Leaving a known-noisy category enabled "while tuning it in place" keeps damaging trust in the interim.

A quick-reference version of the wrong fix vs. the right fix for each named problem: