When a review fails
Never red, never silent. Every way a review can fail degrades to a neutral check with the next step written on it.
A design reviewer that flaps is worse than no reviewer at all: the first time a check goes red for a reason nobody can explain, the team learns to ignore it, and every true finding after that is ignored too.
Two consequences, everywhere. A check is only ever success or neutral, and a neutral check's title names its findings or the next step.
What happens on your pull request
| Situation | What you see |
|---|---|
| Duplicate webhook delivery | nothing twice: a commit is reviewed once |
| Push storm | collapsed into one review against the newest commit |
| Many pushes to one pull request | the deterministic checks run on every push; the AI review reads a pull request twice, the first time and once more after new UI lines, and never the same lines twice; past that a member's @crocotaste review reads it again for one review, an ask with nothing new is free, the summary says when the AI review did not run, and a finding already posted keeps counting while its line is unchanged; a push whose read was due and withheld passes on the checks alone, titled checks only, without an approving review |
| Two events for one pull request | never reviewed side by side; the second waits for the first |
| Draft pull request | waits silently until marked ready for review; no check, no comment |
| No UI files changed | Skipped · no UI files changed, neutral; no comment, no credit |
| Too large to review | Skipped · too large to review, neutral; the check states the measured size and the ceiling (5000 added UI lines, 100 UI files, or 300 changed files), no credit. Split the pull request; each smaller one is reviewed normally |
| No design system found | Skipped · no design system found, neutral; no DESIGN.md, no tokens, no components and no UI rules in an agent guide were parsed, so there was nothing to check against and no credit is used. Add a DESIGN.md or your tokens and the next push is reviewed |
| An outside contributor's pull request | Skipped · ask a member for @crocotaste review, neutral, no credit. Pull requests from people who cannot push to the repository (bots and agents are never counted) are reviewed only when a member asks with @crocotaste review; once that review has spent its credit, later pushes to that pull request are reviewed like any other. A clean pull request from an outside contributor gets the green check and the summary but no approving review |
| A file too large for GitHub to diff | the review runs on the rest, and the summary says how many files had no diff and were not reviewed |
| Model failure | findings that need no model still post as a normal review, which uses the pull request's one review. With nothing else to say the check goes neutral and costs nothing |
| Malformed AI output | a review without the AI part, and a note saying so; never a fake finding. After two unanswered calls on one pull request its pushes stop calling the model, and @crocotaste review tries again |
| AI finding without a valid citation | deleted before posting |
| Review balance exhausted | Review budget used · top up or upgrade, neutral, with the billing link; no model call is made (billing) |
| Our own error | Couldn't complete · re-run with @crocotaste review, neutral; the summary comment says the same |
| A comment GitHub rejects | the finding is still reported in the summary and the check, rather than failing the review |
| An approval GitHub refuses | the findings and the check still land; only the approving review is skipped |
| Our service down | queued reviews wait and run when it is back; a review interrupted part-way is run again |
| Webhook GitHub could not deliver | GitHub does not retry a failed delivery, so that pull request gets no check until you ask for one: comment @crocotaste review, or press Re-run in the dashboard |
| Our database lost | the next review reads its own markers back from GitHub and still posts nothing twice |
Why a lost webhook is yours to re-run
This is the one failure Crocotaste does not recover on its own, and it is a decision rather than an omission.
GitHub does not redeliver a webhook it failed to deliver. Polling every repository to find what we missed would spend a large share of an installation's GitHub budget to discover that nothing had happened, and it still would not recover a lost @crocotaste command. Both recovery paths already exist and both are one action: @crocotaste review, or Re-run in the dashboard.
Limits say so where they bind
Crocotaste reads a bounded amount per review: changed files, stylesheets, Tailwind configs, component files, parsed tokens and findings posted at once. When one of those limits applies to your pull request, the summary and the check say so on that pull request. Silent truncation is never acceptable: a review that saw half your system has to admit it, or the green check means nothing.
The checks that need no model are bounded the same way. An added line longer than 2000 characters is not scanned for raw values, at most 20 raw values are read per line, and at most 2000 per review. Those numbers sit far above anything a person writes; they exist for a minified bundle or a generated sheet, where one line can hold thousands. Each one is a line in the summary when it applies.
Quiet and correct
A false positive on a clean pull request is this product's worst failure. A missed nit is not. That asymmetry decides every close call:
- Checks stay silent when they cannot be certain.
- The AI review prefers no finding, and is not run at all when there is nothing left for it to look at.
- Every AI finding without a resolving citation is deleted in code, not asked about in a prompt.
- Nothing is posted twice, and a finding you dismissed does not come back on that pull request.
Consistency is the product. A check that flaps or spams is a bug, and worth reporting as one: hello@crocotaste.com.