What you need to know
Vague instructions do not move precision
"Be conservative", "only report high-confidence findings" and "be very, very careful" sound like they should cut false positives. In practice they change little, because they do not tell Claude what counts. Precision improves when the prompt defines the categories to report and the ones to skip.
| Vague | Explicit |
|---|---|
| "Check that comments are accurate" | "Flag a comment only when the behaviour it claims contradicts what the code does" |
| "Only report important issues" | "Report: bugs that change behaviour, security issues, data loss. Skip: style, naming, patterns used consistently elsewhere in this repository" |
| "Be conservative" | "If the issue depends on an assumption you cannot verify from the diff, do not report it" |
False positives destroy trust in everything
Developers do not judge categories separately. If the style category is wrong 70% of the time, they start dismissing all findings, including accurate security ones. A high false-positive category damages the categories that work.
A practical response: temporarily disable the noisy category so developers trust the rest again, and improve its prompt offline before re-enabling it.
Define severity with examples
"Critical" and "minor" drift unless each level is defined with criteria and a concrete code example:
| Severity | Criteria | Example |
|---|---|---|
| Critical | Exploitable security flaw or data loss in production | SQL built by string concatenation from request input |
| High | Wrong behaviour users will hit | Refund calculated before discount is applied |
| Medium | Wrong behaviour in rare paths | Missing null check on an optional field |
| Low | Maintainability | Duplicated helper that already exists in utils |
In the exam
Scenario 5 again: a reviewer with a high false-positive rate, developers ignoring all findings, or inconsistent severity labels. The answer defines report and skip categories, uses examples for severity, or disables a noisy category temporarily.
Check your understanding
A review prompt says "only report high-confidence findings" and false positives stay at 35%. What should change?
Show answer
Answer: C. Explicit categories tell Claude what counts. Confidence wording and intensity do not change what it considers an issue.
The style category has a 70% false-positive rate and developers now ignore security findings too. What is the best immediate step?
Show answer
Answer: B. Turning off the noisy category restores trust in the accurate ones. Improve its prompt offline and re-enable it once it is reliable.
Nearly identical issues are labelled critical one day and minor the next. What fixes this?
Show answer
Answer: A. Criteria plus concrete examples anchor each level, so similar issues get the same label.
Apply it: Rewrite Keystone's review criteria
Keystone's reviewer says "be thorough but conservative". Write explicit criteria.
- List what to report and what to skip, at least four of each.
- Define four severity levels with one Keystone-specific code example each.
- Decide what to do with the style category now, and what must be true before it returns.
Show model answer
1. Report and skip
| Report | Skip |
|---|---|
| Logic errors that change payment amounts or states | Formatting the linter already enforces |
| Security issues: injection, secrets, card data exposure | Naming preferences |
| Missing error handling on calls to the card network | Patterns used consistently elsewhere in the repo |
| Concurrency bugs on shared balances | Comments, unless they contradict the code |
| Data loss on retries | Speculative issues needing facts outside the diff |
2. Severity
| Level | Example |
|---|---|
| Critical | Full card number written to logs |
| High | Retry path charges the customer twice |
| Medium | Missing timeout on the fraud-check call |
| Low | Duplicate currency-format helper |
3. Style. Disable it now. Re-enable it only after the new criteria reach a false-positive rate the team agrees (for example under 10%) on a sample of 100 past pull requests.