The business
Keystone Payments, based in Philadelphia, Pennsylvania, processes card payments for online merchants. About 90 engineers open roughly 400 pull requests a week across a dozen services.
The pipeline
On every pull request, a CI job runs Claude Code to:
- Review the change and post findings as comments.
- Generate unit tests for changed code.
A separate nightly job writes a technical debt report across all repositories, read by team leads the next morning.
What goes wrong today
- The CI job sometimes hangs until the 30-minute timeout. The logs show Claude Code waiting for input.
- Findings arrive as prose paragraphs; a script tries to turn them into inline comments and often fails.
- After each new commit the reviewer re-posts the same findings, including ones already fixed.
- The style category has about a 70% false positive rate. Developers now ignore all findings, including real security ones.
- Severity labels are inconsistent: near-identical issues are "critical" one day and "minor" the next.
- About 80% of generated tests are rejected: they ignore the team's fixture library, test trivial getters, or repeat scenarios the suite already covers.
- The same Claude session that wrote a suggested fix also reviews it, and approves it.
Constraints
- Pre-merge checks block merging and must finish within 10 minutes.
- The nightly report has until 08:00 the next day.
- Review findings must be machine-readable so they can be posted as inline comments.
People
| Stakeholder | Cares most about |
|---|---|
| Head of Platform (sponsor) | A reviewer developers read again |
| Security engineer | Security findings never lost in noise |
| Team leads | Fewer, better comments; useful tests |