fix: show partial code suggestion coverage

[YaoSong808]

Summary

• track failed and total /improve analysis chunks
• add a configurable coverage notice when only successful chunks contribute suggestions
• cover partial, complete, fallback, disabled, and artifact-output paths with focused tests

Testing

• PYTHONPATH=. ./.venv/bin/pytest tests/unittest/test_pr_code_suggestions_core.py -q
• PYTHONPATH=. ./.venv/bin/pytest -q tests/unittest/test_pr_code_suggestions_core.py tests/unittest/test_pr_code_suggestions_rendering.py tests/unittest/test_pr_code_suggestions_filtering.py tests/unittest/test_code_suggestions_comment_identity.py tests/unittest/test_run_details_wiring.py tests/unittest/test_self_reflect_fallback.py
• ./.venv/bin/pre-commit run --files pr_agent/tools/pr_code_suggestions.py pr_agent/settings/configuration.toml tests/unittest/test_pr_code_suggestions_core.py

Closes #2855

[qodo-code-review]

PR Summary by Qodo

Show partial coverage for /improve suggestions

🐞 Bug fix 🧪 Tests ⚙️ Configuration changes 🕐 10-20 Minutes

AI Description

• Track failed and total /improve chunks while preserving successful suggestions.
• Warn when generated suggestions cover only successful analysis chunks.
• Allow operators to disable the coverage footer and test affected output paths.

Diagram

graph TD
  D["PR Diff"] --> C["Chunk Analysis"] --> A["Result Aggregation"] --> S["Suggestion Summary"] --> O["Comment or Artifact"]
  A -->|failed totals| F["Coverage Footer"] --> S
  CFG["Coverage Setting"] --> F

Loading

High-Level Assessment

The current approach is appropriately scoped: retaining coverage counts on the suggestion tool avoids changing prediction result contracts, while one footer helper keeps published and artifact summaries consistent. Returning a richer aggregation object was considered but would create unnecessary API churn for two internal counters.

Files changed (3) +80 / -2

Bug fix (1) +17 / -1

pr_code_suggestions.pyTrack chunk failures and disclose partial suggestion coverage +17/-1

Track chunk failures and disclose partial suggestion coverage

• Records total and failed analysis chunks for each prediction attempt. Appends a configurable warning to summarized published output and artifact output when suggestions come from only successful chunks, while completed fallback runs reset the coverage state.

pr_agent/tools/pr_code_suggestions.py

Tests (1) +62 / -1

test_pr_code_suggestions_core.pyCover chunk accounting and coverage footer behavior +62/-1

Cover chunk accounting and coverage footer behavior

• Extends parallel, sequential, and fallback tests with chunk-count assertions. Adds focused coverage for partial warnings, disabled output, uninitialized tool safety, and artifact rendering.

tests/unittest/test_pr_code_suggestions_core.py

Other (1) +1 / -0

configuration.tomlEnable partial suggestion coverage warnings by default +1/-0

Enable partial suggestion coverage warnings by default

• Adds a '/improve' setting that controls whether incomplete chunk coverage is disclosed. The footer is enabled by default.

pr_agent/settings/configuration.toml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

1. Partial runs claim no suggestions ✓ Resolved 🐞 Bug ≡ Correctness

Description

If some chunks fail while successful chunks yield no accepted suggestions, run returns through
publish_no_suggestions() before the new footer is appended. The output consequently claims no
suggestions were found for the PR—or emits an empty artifact—even though the failed chunks were
never able to contribute results.

Code

pr_agent/tools/pr_code_suggestions.py[211]

+                    pr_body += self._get_suggestions_coverage_footer()

Relevance

●●● Strong

This is a deterministic partial-run correctness gap; recent accepted history favors preserving
useful output over misleading early exits.

PR-#2649
PR-#2491

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The result can be partial because chunk errors only abort when every chunk invocation failed;
otherwise successful predictions continue through filtering. If the resulting suggestion list is
empty, run exits at lines 195-198, while publish_no_suggestions never invokes the coverage
helper and sets the disabled/artifact result to an empty string.

pr_agent/tools/pr_code_suggestions.py[195-211]
pr_agent/tools/pr_code_suggestions.py[311-335]
pr_agent/tools/pr_code_suggestions.py[1183-1218]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A partially failed analysis that produces no accepted suggestions bypasses the coverage notice and is reported as an unconditional no-suggestions result.

## Issue Context
The empty-result return occurs before both new footer call sites. The no-suggestions method must incorporate partial-coverage state for comment and artifact/provider-specific output paths while still respecting the feature flag.

## Fix Focus Areas
- pr_agent/tools/pr_code_suggestions.py[195-211]
- pr_agent/tools/pr_code_suggestions.py[302-335]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Inline results omit coverage ✓ Resolved 🐞 Bug ≡ Correctness

Description

When some analysis chunks fail but others produce suggestions, the inline/committable branch
publishes those partial suggestions without any coverage warning because the new footer is added
only inside the summarized-comment branch. Users therefore see an apparently complete inline review
even though failed_chunk_count proves part of the PR was not analyzed successfully.

Code

pr_agent/tools/pr_code_suggestions.py[211]

+                    pr_body += self._get_suggestions_coverage_footer()

Relevance

●●● Strong

Recent accepted correctness precedents favor fixing misleading partial-result paths in
pr_code_suggestions.

PR-#2649
PR-#2404

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

prepare_prediction_main retains successful predictions while recording per-chunk failures, but
run calls the footer helper only in the summarized branch. The sibling inline branch calls
push_inline_code_suggestions, whose implementation publishes provider suggestions and fallback
comments without any coverage message.

pr_agent/tools/pr_code_suggestions.py[1183-1200]
pr_agent/tools/pr_code_suggestions.py[201-271]
pr_agent/tools/pr_code_suggestions.py[755-817]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Partial suggestions published through the inline/committable path omit the new failed-chunk coverage notice.

## Issue Context
The footer is currently appended only while building a summarized comment; `push_inline_code_suggestions` publishes the same partially generated data through provider APIs without exposing the counters.

## Fix Focus Areas
- pr_agent/tools/pr_code_suggestions.py[201-271]
- pr_agent/tools/pr_code_suggestions.py[755-817]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Fallback chunks evade coverage 🐞 Bug ≡ Correctness ⭐ New

Description

After line-number conversion fails, the fallback regenerates numbered chunks but leaves the old
unnumbered chunk list in place; zip() can truncate or mispair them because adding line numbers
changes token-based chunk boundaries. Setting total_chunk_count from that truncated list makes the
new coverage notice omit chunks that were never analyzed and can present partial output as more
complete than it is.

Code

pr_agent/tools/pr_code_suggestions.py[1183]

+            self.total_chunk_count = len(chunk_pairs)

Relevance

●●● Strong

Clear fallback-state bug affecting coverage counts; recent same-file correctness fixes were
accepted, with no close rejection precedent.

PR-#2649

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The non-decoupled list is saved at lines 1161-1167, but conversion failure replaces only the
numbered list at lines 1168-1174. The changed code then derives the advertised total from zip() at
lines 1182-1183; get_pr_multi_diffs proves numbered and unnumbered forms have different
content/token sizes and uses those sizes to split chunks, so equal list lengths and matching
boundaries are not guaranteed.

pr_agent/tools/pr_code_suggestions.py[1161-1183]
pr_agent/algo/pr_processing.py[462-498]
pr_agent/tools/pr_code_suggestions.py[1245-1279]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
When conversion fails, the fallback numbered chunks are paired with the stale original unnumbered chunks using `zip()`. Their token-based chunk boundaries can differ, so chunks may be omitted or mismatched and the coverage denominator becomes inaccurate.

## Issue Context
Regenerate or derive an unnumbered counterpart from the fallback numbered list so both lists have identical chunk membership before constructing `chunk_pairs`. Avoid silently accepting unequal lengths, and add a test where fallback chunking produces different list lengths.

## Fix Focus Areas
- pr_agent/tools/pr_code_suggestions.py[1161-1183]
- tests/unittest/test_pr_code_suggestions_core.py[274-333]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Coverage footer lacks documentation ✓ Resolved 📘 Rule violation ⚙ Maintainability

Description

The PR adds a user-visible partial-coverage footer and a new configuration toggle, but the
/improve documentation does not explain either behavior. Users therefore cannot discover when the
notice appears or how to disable it.

Code

pr_agent/tools/pr_code_suggestions.py[R308-309]

+        return (f"\n\n⚠️ **Suggestion coverage:** {failed_chunk_count} of {total_chunk_count} analysis chunks failed; "
+                "the suggestions above are based on the successful chunks only.")

Relevance

●●● Strong

Recent accepted precedent documents new user-visible behavior and configuration, including /improve
behavior changes.

PR-#2649
PR-#2491
PR-#2659

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 2694680 requires documentation for new user-facing output and configuration
options. The changed code emits the new Suggestion coverage warning, while the existing /improve
configuration table documents neighboring output controls but has no entry for
enable_suggestions_coverage_footer.

Rule 2694680: Update docs when user-facing behavior changes
pr_agent/tools/pr_code_suggestions.py[302-309]
pr_agent/settings/configuration.toml[179-185]
docs/docs/tools/improve.md[308-340]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Document the new partial suggestion-coverage footer and the `enable_suggestions_coverage_footer` configuration option.

## Issue Context
The `/improve` tool now appends a warning when some analysis chunks fail, and the behavior is enabled by default through a new setting. The existing `/improve` configuration table currently ends without this option.

## Fix Focus Areas
- docs/docs/tools/improve.md[308-340]
- pr_agent/settings/configuration.toml[182-182]
- pr_agent/tools/pr_code_suggestions.py[302-309]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

• Author self-review: I have reviewed the code review findings, and addressed the relevant ones.

Context sources

✅ Compliance rules (platform): 34 rules

Review mode: ⚖️ Balanced: This behavioral feature spans chunk accounting, summary/inline/artifact publishing, provider APIs, configuration, and fallback paths; it has real integration risk, but remains one cohesive concern suitable for a careful single-pass review.

Tip of the day

💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous reviews

Review updated until commit 9e293c4

Results up to commit 0455f9d ⚖️ Balanced

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 📜 Skill insights (0)

1. Partial runs claim no suggestions ✓ Resolved 🐞 Bug ≡ Correctness

Description

If some chunks fail while successful chunks yield no accepted suggestions, run returns through
publish_no_suggestions() before the new footer is appended. The output consequently claims no
suggestions were found for the PR—or emits an empty artifact—even though the failed chunks were
never able to contribute results.

Code

pr_agent/tools/pr_code_suggestions.py[211]

+                    pr_body += self._get_suggestions_coverage_footer()

Relevance

●●● Strong

This is a deterministic partial-run correctness gap; recent accepted history favors preserving
useful output over misleading early exits.

PR-#2649
PR-#2491

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The result can be partial because chunk errors only abort when every chunk invocation failed;
otherwise successful predictions continue through filtering. If the resulting suggestion list is
empty, run exits at lines 195-198, while publish_no_suggestions never invokes the coverage
helper and sets the disabled/artifact result to an empty string.

pr_agent/tools/pr_code_suggestions.py[195-211]
pr_agent/tools/pr_code_suggestions.py[311-335]
pr_agent/tools/pr_code_suggestions.py[1183-1218]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A partially failed analysis that produces no accepted suggestions bypasses the coverage notice and is reported as an unconditional no-suggestions result.

## Issue Context
The empty-result return occurs before both new footer call sites. The no-suggestions method must incorporate partial-coverage state for comment and artifact/provider-specific output paths while still respecting the feature flag.

## Fix Focus Areas
- pr_agent/tools/pr_code_suggestions.py[195-211]
- pr_agent/tools/pr_code_suggestions.py[302-335]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Inline results omit coverage ✓ Resolved 🐞 Bug ≡ Correctness

Description

When some analysis chunks fail but others produce suggestions, the inline/committable branch
publishes those partial suggestions without any coverage warning because the new footer is added
only inside the summarized-comment branch. Users therefore see an apparently complete inline review
even though failed_chunk_count proves part of the PR was not analyzed successfully.

Code

pr_agent/tools/pr_code_suggestions.py[211]

+                    pr_body += self._get_suggestions_coverage_footer()

Relevance

●●● Strong

Recent accepted correctness precedents favor fixing misleading partial-result paths in
pr_code_suggestions.

PR-#2649
PR-#2404

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

prepare_prediction_main retains successful predictions while recording per-chunk failures, but
run calls the footer helper only in the summarized branch. The sibling inline branch calls
push_inline_code_suggestions, whose implementation publishes provider suggestions and fallback
comments without any coverage message.

pr_agent/tools/pr_code_suggestions.py[1183-1200]
pr_agent/tools/pr_code_suggestions.py[201-271]
pr_agent/tools/pr_code_suggestions.py[755-817]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Partial suggestions published through the inline/committable path omit the new failed-chunk coverage notice.

## Issue Context
The footer is currently appended only while building a summarized comment; `push_inline_code_suggestions` publishes the same partially generated data through provider APIs without exposing the counters.

## Fix Focus Areas
- pr_agent/tools/pr_code_suggestions.py[201-271]
- pr_agent/tools/pr_code_suggestions.py[755-817]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Coverage footer lacks documentation ✓ Resolved 📘 Rule violation ⚙ Maintainability

Description

The PR adds a user-visible partial-coverage footer and a new configuration toggle, but the
/improve documentation does not explain either behavior. Users therefore cannot discover when the
notice appears or how to disable it.

Code

pr_agent/tools/pr_code_suggestions.py[R308-309]

+        return (f"\n\n⚠️ **Suggestion coverage:** {failed_chunk_count} of {total_chunk_count} analysis chunks failed; "
+                "the suggestions above are based on the successful chunks only.")

Relevance

●●● Strong

Recent accepted precedent documents new user-visible behavior and configuration, including /improve
behavior changes.

PR-#2649
PR-#2491
PR-#2659

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 2694680 requires documentation for new user-facing output and configuration
options. The changed code emits the new Suggestion coverage warning, while the existing /improve
configuration table documents neighboring output controls but has no entry for
enable_suggestions_coverage_footer.

Rule 2694680: Update docs when user-facing behavior changes
pr_agent/tools/pr_code_suggestions.py[302-309]
pr_agent/settings/configuration.toml[179-185]
docs/docs/tools/improve.md[308-340]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Document the new partial suggestion-coverage footer and the `enable_suggestions_coverage_footer` configuration option.

## Issue Context
The `/improve` tool now appends a warning when some analysis chunks fail, and the behavior is enabled by default through a new setting. The existing `/improve` configuration table currently ends without this option.

## Fix Focus Areas
- docs/docs/tools/improve.md[308-340]
- pr_agent/settings/configuration.toml[182-182]
- pr_agent/tools/pr_code_suggestions.py[302-309]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[YaoSong808]

Addressed the automated review findings in 9e293c4e:

• partial coverage is now disclosed for inline/committable suggestions without duplicating the notice in dual-publishing mode
• partial runs with no accepted suggestions now qualify the result for comments and artifacts
• local improve.md artifacts preserve the same coverage context
• documented enable_suggestions_coverage_footer
• added focused regression coverage

Validation: 2585 passed, 1 skipped, 1 xfailed; all configured pre-commit hooks pass.

[qodo-code-review]

Code review by qodo was updated up to the latest commit def3814

[qodo-code-review]

Code review by qodo was updated up to the latest commit 9e293c4

[IsmaelMartinez]

Merging this, and it is a good first PR. It does what #2855 asked, and you avoided the trap the issue warned about: getattr on both counters plus a regression test for it. I broke it deliberately to check, and bare attribute access fails 27 tests across seven files.

The coverage line matches /review's footer on naming, default and placement, which is exactly what I was hoping for.

CI is green now that the first-run workflow approval has gone through, so this is ready.