feat: detect-only mode for docs drift checking by Benkapner · Pull Request #68 · redhat-community-ai-tools/code-to-docs

Benkapner · 2026-08-17T05:45:32Z

Summary

Adds a detect-only mode that identifies docs affected by a code change without generating anything. Runnable on pull_request events as a lightweight docs drift check.

New mode input: comment (default, current behavior) or detect-only
New docs-drift-severity input: warn (exit 0) or error (exit non-zero for use as a required status check)
New src/detect.py module with diff parsing and comparison logic
README section with example workflow

The failure message names specific files and tells the user what to do next.

Test plan

uv run pytest -v passes (423 tests)
Lint clean
detect-only mode reports untouched affected docs correctly

The detection half of the tool is far more reliable and cheaper than generation, but currently they are bundled. Add a detect-only mode that identifies docs affected by a diff, compares against docs actually changed in the PR, and reports which files may need updates. Generates nothing, opens no PR. Runnable on pull_request events, not just issue_comment.

Add a README section with an example workflow for detect-only mode on pull_request events. Frame it as the first thing a cautious team should adopt before using the full generation pipeline.

fullsend-ai-review · 2026-08-17T05:46:38Z

🤖 Finished Review · ✅ Success · Started 5:46 AM UTC · Completed 6:03 AM UTC

Commit: 777de71 · View workflow run →

fullsend-ai-review · 2026-08-17T06:03:47Z

Review

Findings

High

[logic-error] src/suggest_docs.py:270 — In detect-only mode, setup_docs_environment() is called before get_diff(). setup_docs_environment() changes the working directory (via os.chdir) to the docs subfolder or a cloned docs repo. get_diff() then runs git diff in that new CWD. In separate-repo mode, this produces the docs repo's diff instead of the source code diff. In the existing comment-based flow, get_diff() correctly runs before setup_docs_environment(). The detect-only block reverses this order.
Remediation: Move diff = get_diff() and the empty-diff guard before the setup_docs_environment() call, matching the ordering used by the comment-based flow.
[logic-error] src/detect.py:21 — run_detect_only compares relevant_files (paths relative to the docs root, e.g., guide.md) against changed_docs (paths extracted from the source repo diff via extract_changed_doc_paths, e.g., docs/guide.md). In same-repo mode with DOCS_SUBFOLDER, these path namespaces differ — relevant_files lacks the subfolder prefix while changed_docs retains it. The set subtraction affected_set - changed_docs will never find matches, so every affected file will be reported as untouched even when updated in the PR.
Remediation: Normalize paths before comparison — strip the DOCS_SUBFOLDER prefix from changed_docs paths or prepend it to relevant_files.

Medium

[missing-test] tests/test_detect.py — exit_with_severity is not tested. The severity='error' path that calls sys.exit(1) is untested, and the severity='warn' with non-empty untouched (silent return) is also untested. This function controls whether the GitHub Action fails the status check.
[import-style] src/suggest_docs.py:262 — The detect-only mode block uses a lazy inline import (from detect import ...). Every other module import in suggest_docs.py is at the top of the file (lines 26–57). No other conditional/deferred import exists in the file, and detect.py has no heavy dependencies that would justify deferring.
[documentation-coherence] CLAUDE.md — CLAUDE.md was not updated to reflect the new detect-only mode: (1) the Source modules table does not list detect.py, (2) the Environment variables table does not list MODE or DOCS_DRIFT_SEVERITY, and (3) the Command flows section does not document the detect-only flow.

Low

[edge-case] src/detect.py:37 — When relevant_files is empty and changed_docs is also empty, run_detect_only reports "All affected documentation files are already updated in this PR." This is misleading — no docs were identified as affected.
[module-docstring-style] src/detect.py:1 — The module docstring is a single line. Most other source modules in src/ use a multi-line docstring with a summary line, blank line, and elaboration.
[test-docstring-convention] tests/test_detect.py:1 — The test docstring """Tests for detect-only mode.""" references the feature rather than the module. Other test files follow """Tests for <module>.py — <description>.""" convention.
[missing-authorization] No linked issue for a non-trivial feature addition. The PR description adequately explains the motivation, but a tracking issue would establish authorized scope.
[scope-coherence] src/suggest_docs.py:261 — The detect-only block sits between the --build-index handler and COMMENT_BODY parsing. The existing comment partially explains this but could be more explicit about why the position matters (must run before comment-driven flow).

Labels: PR adds a new detect-only execution mode with Python source, tests, and GitHub Action config changes.

Next steps:

/fs-fix — agent addresses review findings automatically
/fs-fix <your instruction> — agent fixes with your specific guidance
Push commits directly — review re-runs automatically on push
/fs-fix-stop — disable automatic fix runs for this PR

fullsend-ai-review

See the review comment for full details.

fullsend-ai-review · 2026-08-17T06:03:49Z

+        if not setup_docs_environment():
+            print("Failed to set up docs environment")
+            return
+        diff = get_diff()


[high] logic-error

In detect-only mode, setup_docs_environment() is called before get_diff(). setup_docs_environment() changes the working directory to the docs subfolder or a cloned docs repo. get_diff() then runs git diff in that new CWD. In separate-repo mode, this produces the docs repo's diff instead of the source code diff. In the comment-based flow, get_diff() correctly runs before setup_docs_environment().

Suggested fix: Move diff = get_diff() and the empty-diff guard before the setup_docs_environment() call in the detect-only block, matching the ordering used by the comment-based flow.

fullsend-ai-review · 2026-08-17T06:03:49Z

+
+def run_detect_only(diff, relevant_files, changed_docs):
+    """Compare affected docs against docs actually changed in the PR.
+


[high] logic-error

run_detect_only compares relevant_files (paths relative to docs root) against changed_docs (repo-relative paths from extract_changed_doc_paths). In same-repo mode with DOCS_SUBFOLDER, these path namespaces differ. The set subtraction will never find matches, so every affected file will be reported as untouched even when updated in the PR.

Suggested fix: Normalize paths before comparison. Strip the DOCS_SUBFOLDER prefix from changed_docs paths or prepend it to relevant_files.

fullsend-ai-review · 2026-08-17T06:03:49Z

        return

+    # Handle detect-only mode (runs on pull_request events, not comments)
+    mode = os.environ.get("MODE", "comment")


[medium] import-style

The detect-only mode block uses a lazy inline import. Every other module import in suggest_docs.py is at the top of the file. No other conditional/deferred import exists in the file, and detect.py has no heavy dependencies justifying deferral.

Suggested fix: Move the import to the top-level imports section alongside the other from-module-import statements.

fullsend-ai-review · 2026-08-17T06:03:49Z

+            "Comment [review-docs] on the PR to review suggested changes, "
+            "or [update-docs] to generate updates directly."
+        )
+    else:


[low] edge-case

When relevant_files is empty and changed_docs is also empty, run_detect_only reports 'All affected documentation files are already updated in this PR.' This is misleading -- no docs were identified as affected.

Suggested fix: Add a check: if affected_set is empty, output a distinct message like 'No documentation files were identified as affected by this change.'

fullsend-ai-review · 2026-08-17T06:03:49Z

@@ -0,0 +1,51 @@
+"""Detect-only mode: identify docs affected by a diff without generating anything."""


[low] module-docstring-style

The module docstring is a single line. Most other source modules in src/ use a multi-line docstring with summary, blank line, and elaboration.

fullsend-ai-review · 2026-08-17T06:03:49Z

@@ -0,0 +1,52 @@
+"""Tests for detect-only mode."""


[low] test-docstring-convention

The test docstring references the feature rather than the module. Other test files follow the convention of referencing the source module filename.

fullsend-ai-review · 2026-08-17T06:03:49Z

        print(f"Index build complete: {result['status']}")
        return

+    # Handle detect-only mode (runs on pull_request events, not comments)


[low] scope-coherence

The detect-only mode block sits between the --build-index handler and COMMENT_BODY parsing. The existing comment partially explains this but could be more explicit about why the position matters.

Benkapner added 2 commits August 17, 2026 08:44

docs(readme): position detect-only as the low-cost entry point

777de71

Add a README section with an example workflow for detect-only mode on pull_request events. Frame it as the first thing a cautious team should adopt before using the full generation pipeline.

Benkapner requested a review from csoceanu August 17, 2026 05:47

Benkapner self-assigned this Aug 17, 2026

fullsend-ai-review Bot suggested changes Aug 17, 2026

View reviewed changes

fullsend-ai-review Bot added feature python Pull requests that update python code github_actions Pull requests that update GitHub Actions code labels Aug 17, 2026

Navigation Menu

Uh oh!

feat: detect-only mode for docs drift checking - #68

feat: detect-only mode for docs drift checking#68
Benkapner wants to merge 2 commits into
mainfrom
feat/detect-only

Benkapner commented Aug 17, 2026

Uh oh!

fullsend-ai-review Bot commented Aug 17, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Aug 17, 2026

Uh oh!

fullsend-ai-review Bot left a comment

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant


		def run_detect_only(diff, relevant_files, changed_docs):
		"""Compare affected docs against docs actually changed in the PR.

		@@ -0,0 +1,51 @@
		"""Detect-only mode: identify docs affected by a diff without generating anything."""

Uh oh!

Conversation

Benkapner commented Aug 17, 2026

Summary

Test plan

Uh oh!

fullsend-ai-review Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Aug 17, 2026

Review

Findings

High

Medium

Low

Uh oh!

fullsend-ai-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fullsend-ai-review Bot commented Aug 17, 2026 •

edited

Loading