feat(review): preserve findings across review reruns - #2722
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
75348a5 to
d9a235a
Compare
PR Summary by QodoPreserve review findings across reruns via persistent comment state
AI Description
Diagram
High-Level Assessment
Files changed (7)
|
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
|
Code review by qodo was updated up to the latest commit 2ca9a24 |
|
Code review by qodo was updated up to the latest commit 0b66cd9 |
|
Code review by qodo was updated up to the latest commit 53f75f8 |
|
Code review by qodo was updated up to the latest commit c04cc62 |
IsmaelMartinez
left a comment
IsmaelMartinez
left a comment
There was a problem hiding this comment.
Thanks for taking #2453 on, and for turning the Qodo findings around so fast. Four notes inline, two with suggestions; the suite stays green with both applied.
| """Serialize state deterministically so repeated updates are diffable.""" | ||
| if not _is_valid_state(state): | ||
| raise ValueError("Invalid review finding state") | ||
| payload = json.dumps(state, ensure_ascii=False, sort_keys=True, separators=(",", ":")) |
There was a problem hiding this comment.
A finding body containing --> closes the marker early and the rest of the payload renders in the review, and parse_review_state still calls that state valid, so it repeats every run. Escaping both brackets fixes it without changing the decoded state.
It does not close Qodo's item 12: append_review_state prints bodies verbatim in the human section too.
| payload = json.dumps(state, ensure_ascii=False, sort_keys=True, separators=(",", ":")) | |
| payload = json.dumps(state, ensure_ascii=False, sort_keys=True, separators=(",", ":")) | |
| # '<' and '>' occur only inside JSON strings, so escaping cannot change the decoded state | |
| payload = payload.replace("<", "\\u003c").replace(">", "\\u003e") |
| allow_resolution = ( | ||
| bool(self.prediction) | ||
| and not bool(getattr(self.incremental, "is_incremental", False)) | ||
| and not bool(self.remaining_files_list) | ||
| and parsed.valid | ||
| and current_findings is not None | ||
| and len(current_findings) < max_findings | ||
| ) |
There was a problem hiding this comment.
With num_max_findings at its default of 3, two findings enable resolution, and nothing checks that the commit moved. Re-running /review on an unchanged head marks a finding RESOLVED just because the model did not repeat it.
Gate on the previous SHA rather than the current one: _review_head_sha reads last_commit_id, which only GitHub and Gitea set, so testing the current SHA turns two of your own tests red.
| allow_resolution = ( | |
| bool(self.prediction) | |
| and not bool(getattr(self.incremental, "is_incremental", False)) | |
| and not bool(self.remaining_files_list) | |
| and parsed.valid | |
| and current_findings is not None | |
| and len(current_findings) < max_findings | |
| ) | |
| previous_head_sha = str(((parsed.state or {}).get("last_run") or {}).get("head_sha") or "") | |
| current_head_sha = self._review_head_sha() | |
| allow_resolution = ( | |
| bool(self.prediction) | |
| and not bool(getattr(self.incremental, "is_incremental", False)) | |
| and not bool(self.remaining_files_list) | |
| and parsed.valid | |
| and current_findings is not None | |
| and len(current_findings) < max_findings | |
| and (not previous_head_sha or current_head_sha != previous_head_sha) | |
| ) |
| artifact={"error": e}) | ||
| else: | ||
| get_logger().exception(f"Failed to edit github comment", artifact={"error": e}) | ||
| raise |
There was a problem hiding this comment.
Not behind persistent_finding_state, and the same line lands in azuredevops_provider.py:237 and bitbucket_provider.py:438, so edit_comment propagates on all three, including the 403 branch this code calls "usually due to polling".
Mostly a gain: a failed edit on the suggestions comment now republishes them where main drops them. It escapes the function only when the fallback write fails too, and then /improve publishes nothing. Both worth a line in the description.
| ) | ||
| except Exception as e: | ||
| get_logger().exception(f"Failed to edit comment, error: {e}") | ||
| raise |
There was a problem hiding this comment.
Hard conflict with #2724, which returns False here. Worth agreeing the resolution before either merges, since as noted on github_provider.py the two are not equivalent.
There was a problem hiding this comment.
Hard conflict with #2724, which returns False here. Worth agreeing the resolution before either merges, since as noted on
github_provider.pythe two are not equivalent.
Agreed. Since #2724 owns the broader Azure path and already validates the False return contract end-to-end, I’ll align #2722 with that behavior rather than keep the competing exception-propagation change.
There was a problem hiding this comment.
Thanks. I traced the edit_comment() call sites further and narrowed #2722 so the two PRs no longer need to own the same Azure behavior.
#2722 now leaves AzureDevopsProvider.edit_comment() and the /improve path untouched. It only updates the shared persistent-comment publisher to treat an explicit False return as an edit failure, while preserving fallback_on_error=False for lifecycle updates so a failed edit cannot create a duplicate persistent review.
That means #2724 can keep the Azure True/False contract and its /improve caller handling independently.
I also added regression coverage for both paths:
False+fallback_on_error=False→ no fallback commentFalse+fallback_on_error=True→ normal fallback
The full unit suite is green locally.
|
Code review by qodo was updated up to the latest commit e97e318 |
IsmaelMartinez
left a comment
IsmaelMartinez
left a comment
There was a problem hiding this comment.
Thanks for chasing this down, but the narrowing has not landed on this branch.
AzureDevopsProvider.edit_comment() is still modified here: head e97e3184 adds raise to its except block, which is the line the conflict was about. Merged onto today's main, this and #2724 still collide on azuredevops_provider.py and git_provider.py.
Worth knowing before you change anything: simply dropping the Azure raise would not be safe on its own. On main edit_comment returns None, not False, so the is False guard you added to publish_persistent_comment_full would never fire, and your fallback_on_error=False callers would read a failed edit as success. That only becomes safe once #2724's True/False contract is in.
So this is an ordering question. If #2724 lands first, this can drop the Azure change entirely. If this lands first, #2724 has to adopt the raise instead.
You're right. I found the residual change: the Azure raise and its regression test were introduced in an earlier commit and survived my later scope narrowing. My last update only narrowed the newest patch, not the full PR diff against main. I also understand why simply removing it now would be unsafe while main still returns None on that path. I'll coordinate the merge order: once the Azure True/False contract is established, I'll rebase #2722, remove the Azure-specific change and test, and verify the complete PR diff against main before updating the PR. |
3 participants
Summary
/reviewrerunsDesign notes
Resolution is conservative:
findings are only marked resolved after complete valid reviews.
Incremental reviews, partial analysis, malformed predictions, and provider failures do not trigger negative state transitions.
Validation
pytest tests/unittest -q— 1960 passed, 1 skipped, 1 xfailedgit diff --checkAI disclosure
This PR was developed with assistance from Codex and validated with targeted tests by the contributor.
Closes #2453