Add agentic documentation framework with metrics - #2792
Open
ashwindasr wants to merge 1 commit into
Open
Conversation
Implements the Agentic Documentation Rulebook for art-tools: structured documentation for AI agents including navigation (AGENTS.md), architecture, 10 domain concept docs, 5 component docs, 3 ADRs, exec-plan templates, CI validation workflow, and Phase 7 metrics/quality scoring with scripts and an HTML dashboard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Contributor
WalkthroughThis PR introduces a comprehensive agentic documentation framework for the art-tools repository, including domain concepts, design documentation, architecture decision records (ADRs), development guides, security documentation, quality scoring, a GitHub Actions validation workflow, and automated metrics measurement scripts. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 14
🧹 Nitpick comments (14)
agentic/exec-plans/tech-debt-tracker.md (1)
5-20: Add language specifier to template code block.The item template code block should specify markdown as the language for proper syntax highlighting and linter compliance.
♻️ Add language specifier
When adding a new item, copy this template: -``` +```markdown ### [DEBT-NNNN] Short description🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@agentic/exec-plans/tech-debt-tracker.md` around lines 5 - 20, Update the Item Template code block in agentic/exec-plans/tech-debt-tracker.md to add a markdown language specifier on the opening fence so the template (starting with "### [DEBT-NNNN] Short description") is fenced as ```markdown; ensure the opening fence includes "markdown" and the closing fence remains present so syntax highlighting and linting pick it up.agentic/decisions/adr-0001-monorepo-structure.md (1)
1-9: Minor inconsistency: Frontmatter key naming differs from template.The frontmatter uses
superseded-by(hyphenated) while the template atagentic/decisions/adr-template.mdusessuperseded_by(underscore). Both are valid YAML, but consistency across ADRs would improve maintainability.♻️ Align with template naming
deciders: [art-team] supersedes: null -superseded-by: null +superseded_by: null🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@agentic/decisions/adr-0001-monorepo-structure.md` around lines 1 - 9, The ADR frontmatter for ADR-0001 uses the hyphenated key `superseded-by`; update it to the template's underscore form `superseded_by` so the frontmatter matches `adr-template.md` naming; locate the frontmatter block in ADR-0001 (id: ADR-0001) and replace the `superseded-by` key with `superseded_by`, ensuring no other keys or values change and run any YAML/CI lint that validates ADR frontmatter.agentic/QUALITY_SCORE.md (1)
99-122: Consider consolidating same-date progress entries.Both progress entries are dated 2026-04-16 with the same 81/100 score. Since they document different aspects of the same PR (metrics implementation + framework implementation), consider consolidating them into a single entry for clarity.
♻️ Consolidate into single entry
## Recent Changes and Progress -### 2026-04-16: Metrics and Quality Scoring Implementation +### 2026-04-16: Initial Agentic Documentation Framework **Score**: 81/100 (baseline with metrics) **What Changed**: +- Complete directory structure (8 directories) +- AGENTS.md (141 lines) and ARCHITECTURE.md +- 10 concept docs, 3 workflow docs, 5 component docs +- 3 ADRs, exec-plan template, tech-debt tracker - Added metrics measurement scripts (`agentic/scripts/`) - Created QUALITY_SCORE.md with actual measured scores - Added Makefile targets (`check-docs`, `docs-dashboard`) +- CI validation workflow with freshness checks +- DESIGN.md, DEVELOPMENT.md, TESTING.md, SECURITY.md - Added doc-update guidance to DEVELOPMENT.md - -### 2026-04-16: Initial Framework Implementation - -**Score**: 81/100 (baseline) - -**Created**: -- Complete directory structure (8 directories) -- AGENTS.md (141 lines) and ARCHITECTURE.md (150 lines) -- 10 concept docs, 3 workflow docs, 5 component docs -- 3 ADRs, exec-plan template, tech-debt tracker -- CI validation workflow with freshness checks -- DESIGN.md, DEVELOPMENT.md, TESTING.md, SECURITY.md🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@agentic/QUALITY_SCORE.md` around lines 99 - 122, Consolidate the two same-date entries under "## Recent Changes and Progress" (the headings "2026-04-16: Metrics and Quality Scoring Implementation" and "2026-04-16: Initial Framework Implementation") into a single 2026-04-16 entry that lists the shared Score (81/100) once and merges the "What Changed" and "Created" bullet items into one combined section so the metrics additions (agentic/scripts/, Makefile targets, QUALITY_SCORE.md changes) and the framework artifacts (directory structure, AGENTS.md, ARCHITECTURE.md, concept/workflow/component docs, ADRs, exec-plan template, tech-debt tracker, CI validation, and documentation files) are all included under a single header for clarity.agentic/domain/concepts/ocp-build-data.md (1)
37-53: Add language specifier to code block.The repository structure code block should specify a language for better rendering and linter compliance.
♻️ Add language specifier
### Repository Structure -``` +```text openshift-4.17/ # Group branch🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@agentic/domain/concepts/ocp-build-data.md` around lines 37 - 53, The code block under the "### Repository Structure" section that begins with "openshift-4.17/" should include a language specifier for proper rendering and linting; replace the opening triple backticks (```) with a language-tagged fence such as ```text (i.e., change the code block delimiter in ocp-build-data.md's Repository Structure code block to ```text) so the block is explicitly marked as plain text.agentic/design-docs/components/validator.md (1)
27-29: Add language specifier to code block.The code block showing the command usage should include a language specifier for proper syntax highlighting and markdown compliance.
💅 Proposed fix for code block formatting
-``` +```bash validate-ocp-build-data FILE [FILE ...]</details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against the current code and only fix it if needed.
In
@agentic/design-docs/components/validator.mdaround lines 27 - 29, The code
block showing the CLI usage for validate-ocp-build-data lacks a language
specifier; update the fenced code block that contains the line
"validate-ocp-build-data FILE [FILE ...]" to include "bash" (or "sh") after the
opening backticks so the block reads as a bash shell snippet for proper syntax
highlighting and markdown compliance.</details> </blockquote></details> <details> <summary>ARCHITECTURE.md (1)</summary><blockquote> `25-45`: **Add language specifier for ASCII diagrams.** The fenced code blocks for ASCII diagrams lack a language specifier, triggering markdownlint MD040. While `text` or `plaintext` work, many prefer an empty specifier comment or just leaving it. However, for consistency and to silence linters, consider adding `text`: <details> <summary>📝 Suggested fix</summary> ```diff -``` +```text +----------+Apply similarly to the data flow diagram at line 67.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ARCHITECTURE.md` around lines 25 - 45, The fenced ASCII diagram blocks in ARCHITECTURE.md are missing a language specifier causing markdownlint MD040; update the opening triple-backtick fences for the diagram(s) (the block that begins with the ASCII art showing pyartcd/doozer/elliott/artcommon and the other data flow diagram mentioned at line 67) to include a language specifier such as text (e.g., change ``` to ```text) so the linter stops complaining and the diagrams remain plain text.agentic/scripts/generate-metrics-dashboard.py (1)
23-28: Use tuple unpacking for cleaner list construction.Per ruff RUF005, prefer unpacking over concatenation for combining lists:
♻️ Suggested fix
result = subprocess.run( - ['python3', str(script_path)] + list(args), + ['python3', str(script_path), *args], capture_output=True, text=True, cwd=Path.cwd() )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@agentic/scripts/generate-metrics-dashboard.py` around lines 23 - 28, The subprocess.run call builds the argv list by concatenating ['python3', str(script_path)] and list(args); replace this concatenation with tuple/list unpacking so the argv is constructed with the base items followed by the unpacked args (i.e., use the unpacking form with *args) in the subprocess.run invocation where result is created, referencing subprocess.run, script_path, and args.agentic/domain/workflows/advisory-management.md (1)
117-119: Add language specifier for ASCII diagrams.The fenced code blocks for ASCII state diagrams (lines 117 and 140) lack language specifiers. Consider adding
textto satisfy markdownlint MD040.Also applies to: 140-149
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@agentic/domain/workflows/advisory-management.md` around lines 117 - 119, The ASCII state diagram code fences (e.g., the block containing "NEW_FILES --> QE --> REL_PREP --> PUSH_READY --> IN_PUSH --> SHIPPED_LIVE" and the subsequent diagram block later in the file) are missing a language specifier; update both fenced code blocks to use a language tag such as ```text so markdownlint MD040 is satisfied and the diagrams are treated as plain text.agentic/domain/concepts/metadata.md (1)
26-36: Consider adding a language hint to the class-hierarchy fence.Optional cleanup: tag this block as
textfor consistency with markdown linting conventions across the doc set.Suggested patch
-``` +```text MetadataBase (artcommon/artcommonlib/metadata.py) | +-- Metadata (doozer/doozerlib/metadata.py) @@ |-- ImageMetadata (elliott/elliottlib/imagecfg.py) |-- RPMMetadata (elliott/elliottlib/rpmcfg.py)</details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against the current code and only fix it if needed.
In
@agentic/domain/concepts/metadata.mdaround lines 26 - 36, The fenced block
showing the Metadata class hierarchy uses a bare code fence (); update that fence to include a language hint (usetext) so the block becomes a fenced
"text" block; locate the block containing "MetadataBase
(artcommon/artcommonlib/metadata.py)" and the indented tree with "Metadata
(doozer/doozerlib/metadata.py)" and "Metadata (elliott/elliottlib/metadata.py)"
and change the opening fence totext (and the closing fence remains);
this aligns the block with the project's markdown linting conventions.</details> </blockquote></details> <details> <summary>agentic/domain/workflows/release-preparation.md (1)</summary><blockquote> `76-111`: **Consider adding a language identifier to the fenced code block.** The sequence diagram uses a bare fenced code block. Adding a language identifier (e.g., `text` or `plaintext`) would satisfy the MD040 lint rule and may help some markdown renderers. <details> <summary>💡 Suggested fix</summary> ```diff -``` +```text Assembly Author pyartcd elliott doozer Brew/Konflux ``` </details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against the current code and only fix it if needed.
In
@agentic/domain/workflows/release-preparation.mdaround lines 76 - 111, The
fenced sequence-diagram code block that begins with "Assembly Author
pyartcd elliott..." is missing a language identifier; update the
opening fence fromtotext (or ```plaintext) so the block is recognized
as plain text and satisfies MD040; locate the diagram block in
release-preparation.md and change only the opening fence to include the language
identifier.</details> </blockquote></details> <details> <summary>agentic/scripts/measure-navigation-depth.py (1)</summary><blockquote> `57-58`: **Consider narrowing the exception type.** While catching a broad `Exception` is defensive for file parsing, it could mask unexpected errors. Consider narrowing to specific exceptions. <details> <summary>💡 More specific exception handling</summary> ```diff - except Exception as e: + except (IOError, OSError, UnicodeDecodeError) as e: print(f"Warning: Could not parse {file_path}: {e}", file=sys.stderr) ``` </details> <details> <summary>🤖 Prompt for AI Agents</summary> ``` Verify each finding against the current code and only fix it if needed. In `@agentic/scripts/measure-navigation-depth.py` around lines 57 - 58, The catch-all except in the parsing code currently printing "Warning: Could not parse {file_path}: {e}" should be narrowed: replace "except Exception as e" around the parsing logic that references file_path with targeted except clauses (e.g., except json.JSONDecodeError as e, except UnicodeDecodeError as e, except FileNotFoundError as e, except ValueError as e) each logging the warning with the same print call; optionally add a final bare "except Exception:" that re-raises unexpected errors so truly unexpected issues are not silently swallowed. Ensure you keep the same print message and only handle specific parse/IO-related exceptions for the code that reads/parses file_path. ``` </details> </blockquote></details> <details> <summary>agentic/scripts/measure-all-metrics.sh (1)</summary><blockquote> `89-91`: **Quote variable expansions to handle paths with spaces.** The script uses unquoted variable expansions which could break if paths contain spaces. <details> <summary>💡 Suggested fix</summary> ```diff if [ -f "$SCRIPT_DIR/measure-navigation-depth.py" ]; then - NAV_OUTPUT=$(python3 $SCRIPT_DIR/measure-navigation-depth.py --max-depth 3 2>&1) + NAV_OUTPUT=$(python3 "$SCRIPT_DIR/measure-navigation-depth.py" --max-depth 3 2>&1) echo "$NAV_OUTPUT" ``` Similarly for lines 115 and 372: ```diff - BUDGET_OUTPUT=$(python3 $SCRIPT_DIR/measure-context-budget.py --max-budget 700 2>&1) + BUDGET_OUTPUT=$(python3 "$SCRIPT_DIR/measure-context-budget.py" --max-budget 700 2>&1) ``` ```diff - python3 $SCRIPT_DIR/generate-metrics-dashboard.py + python3 "$SCRIPT_DIR/generate-metrics-dashboard.py" ``` </details> <details> <summary>🤖 Prompt for AI Agents</summary> ``` Verify each finding against the current code and only fix it if needed. In `@agentic/scripts/measure-all-metrics.sh` around lines 89 - 91, The unquoted variable expansions can break on paths with spaces; update the invocation that sets NAV_OUTPUT to quote the SCRIPT_DIR path and the command substitution (use python3 "$SCRIPT_DIR/measure-navigation-depth.py" --max-depth 3 2>&1 inside the $(...) so the script path is quoted) and likewise apply the same quoting fixes for the other two occurrences mentioned (around lines with similar python3 $SCRIPT_DIR/... calls), keeping echo "$NAV_OUTPUT" as-is. ``` </details> </blockquote></details> <details> <summary>agentic/scripts/test-metrics.sh (1)</summary><blockquote> `38-40`: **Fragile field extraction may break on output format changes.** The `awk` field positions assume a specific output format. Looking at the output from `print_report()` in `measure-navigation-depth.py` (context snippet lines 165-167), the format includes variable leading whitespace: ``` Total documents found: 42 Reachable documents: 40 Unreachable documents: 2 ``` With leading spaces and padding, `awk '{print $4}'` on "Total documents found:" would correctly get the number, but this is fragile if formatting changes. <details> <summary>💡 More robust extraction using colon delimiter</summary> ```diff -TOTAL=$(echo "$OUTPUT" | grep "Total documents found:" | awk '{print $4}') -REACHABLE=$(echo "$OUTPUT" | grep "Reachable documents:" | awk '{print $3}') -UNREACHABLE=$(echo "$OUTPUT" | grep "Unreachable documents:" | awk '{print $3}') +TOTAL=$(echo "$OUTPUT" | grep "Total documents found:" | awk -F: '{print $2}' | tr -d ' ') +REACHABLE=$(echo "$OUTPUT" | grep "Reachable documents:" | awk -F: '{print $2}' | tr -d ' ') +UNREACHABLE=$(echo "$OUTPUT" | grep "Unreachable documents:" | awk -F: '{print $2}' | tr -d ' ') ``` </details> <details> <summary>🤖 Prompt for AI Agents</summary> ``` Verify each finding against the current code and only fix it if needed. In `@agentic/scripts/test-metrics.sh` around lines 38 - 40, The current extraction of totals from OUTPUT using fixed awk field positions is brittle; instead parse by the colon-delimited value and trim whitespace so changes in leading spacing or padding won't break TOTAL, REACHABLE, and UNREACHABLE. Update the three assignment lines that reference OUTPUT (TOTAL, REACHABLE, UNREACHABLE) to split on ":" (e.g., using awk -F: or cut -d:) and trim surrounding whitespace from the right-hand field; this keeps the extraction in sync with the print_report() output formatting in measure-navigation-depth.py. ``` </details> </blockquote></details> <details> <summary>agentic/domain/concepts/distgit.md (1)</summary><blockquote> `22-22`: **Line count may become stale.** The specific line count "(~3062 lines)" will drift as the codebase evolves. Consider removing this detail or using a more general description like "one of the largest modules". <details> <summary>💡 Suggested fix</summary> ```diff -- **Core module:** `doozer/doozerlib/distgit.py` -- One of the largest modules (~3062 lines). Contains: +- **Core module:** `doozer/doozerlib/distgit.py` -- One of the largest modules. Contains: ``` </details> <details> <summary>🤖 Prompt for AI Agents</summary> ``` Verify each finding against the current code and only fix it if needed. In `@agentic/domain/concepts/distgit.md` at line 22, The markdown references a stale specific line count "(~3062 lines)" for the Core module; update the sentence in agentic/domain/concepts/distgit.md to remove the numeric line count and use a general phrase such as "one of the largest modules" or "a large module" when referring to doozer/doozerlib/distgit.py, ensuring the offending substring "(~3062 lines)" is deleted or replaced. ``` </details> </blockquote></details> </blockquote></details> <details> <summary>🤖 Prompt for all review comments with AI agents</summary>Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/validate-agentic-docs.yml:
- Around line 167-184: The problem is the inner "grep ... | while read -r path;
do" creates a subshell so updates to the broken variable are lost; replace the
piped while with a while loop that reads from process substitution or capture
grep output into a variable and iterate in the same shell (e.g. change "grep ...
| while read -r path; do" to "while read -r path; do ... done < <(grep -oP
'...pattern...' "$doc" 2>/dev/null)" so increments of broken (the broken
variable) occur in the main shell and the final if ("if [ "$broken" -gt 0 ]")
sees the correct value.- Around line 88-117: The broken-link counter never increments because the inner
"grep ... | while read -r link; do ... broken=$((broken + 1)); done" pipelines
run in subshells; change those piped while loops to use process substitution
(e.g., replace "grep ... | while read -r link; do" with "while read -r link; do
... done < <(grep ... )") so broken is updated in the main shell, apply this to
both the per-file inner loop and the root-file loop, and ensure after the scans
you check "if [ $broken -gt 0 ]; then exit 1; fi" to fail the job when broken >
In
@agentic/decisions/adr-0002-runtime-pattern.md:
- Around line 23-27: The fenced code block showing the runtime hierarchy
(containing artcommonlib.runtime.GroupRuntime, doozerlib.runtime.Runtime,
elliottlib.runtime.Runtime) is missing a language identifier and triggers
markdownlint MD040; update the triple-backtick fence to include the identifier
text(i.e., replace the openingwithtext) so the block is treated as
plain text.In
@agentic/design-docs/components/doozer.md:
- Line 110: The doc incorrectly claims cycles are detected/rejected; update the
sentence in doozer.md to either remove the cycle-detection assertion or point to
the real implementation: reference Runtime.generate_image_tree() and the actual
function that performs cycle checks (or its absence) in the runtime codebase.
Concretely, edit the line to say "Cyclic dependencies are not explicitly
detected here; generate_image_tree() produces image_order (parents before
children) and cycle handling is implemented elsewhere if present" or replace
with a correct reference to the real cycle-check routine if you locate it in the
runtime module.In
@agentic/design-docs/core-beliefs.md:
- Around line 29-33: The fenced code block containing "GroupRuntime (ABC) ->
doozerlib.runtime.Runtime -> elliottlib.runtime.Runtime" should include a
language identifier to satisfy MD040; change the opening fence from ``` toremains ```), updating the block that lists GroupRuntime and the runtime paths. In `@agentic/design-docs/index.md`: - Line 13: Update the broken component link in agentic/design-docs/index.md: replace the current link target "./components/ocp-build-data-validator.md" (the list entry referencing the validator component) with "./components/validator.md" so it matches the docs mapping/checks that expect agentic/design-docs/components/validator.md; ensure the link text remains descriptive and run the docs check to confirm the internal link is resolved. In `@agentic/domain/concepts/errata-advisories.md`: - Line 14: Replace the incorrect singular "erratum" with the plural "errata" in the descriptive sentence about advisories so it reads "an advisory groups one or more errata together with associated metadata…" — update the sentence text in the Errata Tool description accordingly. In `@agentic/domain/workflows/advisory-management.md`: - Around line 105-125: The documentation currently implies elliott change-state can advance advisories through the full state machine, but the CLI (elliott/elliottlib/cli/change_state_cli.py) only accepts --state values NEW_FILES, QE, and REL_PREP; transitions to PUSH_READY and IN_PUSH are handled by the Errata Tool. Update the advisory-management.md “6. Change State” section to explicitly state that elliott change-state only manages the first three states (NEW_FILES -> QE -> REL_PREP), clarify that PUSH_READY and later transitions are performed externally by Errata Tool, and adjust any examples or the state diagram to avoid implying elliott covers PUSH_READY/IN_PUSH. In `@agentic/exec-plans/active/implement-agentic-docs.md`: - Line 18: Update the success criteria text in implement-agentic-docs.md that currently reads "38 new files created under `agentic/`" to "42 new files created under `agentic/`" so the documented file count matches the actual number of new files; locate the exact phrase "38 new files created under `agentic/`" and replace it with "42 new files created under `agentic/`". In `@agentic/QUALITY_SCORE.md`: - Around line 75-88: The manual scoring table under the "Total Score" header (the 7-category table showing **109/120**) conflicts with the "Automated Score (from `measure-all-metrics.sh`): **81/100**"; either reconcile them or separate them clearly: update the manual table to compute and display the automated metrics (Navigation Depth, Context Budget, Structure Compliance, Documentation Coverage with values 50, 75, 100, 100) so both scores align, or add explicit section headers and a short explanatory sentence separating the "Manual Assessment" table (keep the 7-category breakdown) from the "Automated Score" block (retain the `measure-all-metrics.sh` result) so readers understand they are different measurements—make the chosen change in QUALITY_SCORE.md, preserving both the "Total Score" heading and the `measure-all-metrics.sh` reference. In `@agentic/scripts/generate-metrics-dashboard.py`: - Around line 126-133: In generate_html_dashboard, structure_score and coverage_score are hardcoded to 100 which inflates overall_score; change the logic so structure_score and coverage_score are treated as optional: read actual values if present (e.g., nav_metrics.get('structure'), budget_metrics.get('coverage') or dedicated keys), otherwise set them to None and render "N/A" in the dashboard, and compute overall_score by averaging only the available (non-None) scores instead of always dividing by 4; update the assignments for structure_score, coverage_score and the overall_score calculation in generate_html_dashboard accordingly. In `@agentic/scripts/measure-context-budget.py`: - Around line 227-231: Current code uses Path.cwd() for base_dir which makes measurements depend on invocation directory; replace base_dir = Path.cwd() with logic that locates the repository root (for example run git rev-parse --show-toplevel and fallback to Path(__file__).resolve().parent/ancestors) to produce an absolute Path for the repo root, assign that to base_dir, and continue passing it into workflow.measure for each item in WORKFLOWS; update references to base_dir, WORKFLOWS, and workflow.measure accordingly so file lookups are performed relative to the repo root instead of the current working directory. - Around line 35-37: The catch in count_lines() is too broad and should only handle expected I/O/decoding failures; replace the generic "except Exception as e" with a narrow handler like "except (OSError, UnicodeDecodeError) as e" so file open/read and decoding errors are caught and logged/return 0 while other exceptions continue to propagate; update the except block around the file open/read in count_lines() accordingly and leave all other logic unchanged. In `@agentic/TESTING.md`: - Around line 5-17: Add a language label to the unlabeled fenced code block in TESTING.md (the directory-layout snippet) to satisfy MD040; change the opening triple-backticks to include "text" so the block becomes a ```text fenced block that preserves the tree-style formatting. --- Nitpick comments: In `@agentic/decisions/adr-0001-monorepo-structure.md`: - Around line 1-9: The ADR frontmatter for ADR-0001 uses the hyphenated key `superseded-by`; update it to the template's underscore form `superseded_by` so the frontmatter matches `adr-template.md` naming; locate the frontmatter block in ADR-0001 (id: ADR-0001) and replace the `superseded-by` key with `superseded_by`, ensuring no other keys or values change and run any YAML/CI lint that validates ADR frontmatter. In `@agentic/design-docs/components/validator.md`: - Around line 27-29: The code block showing the CLI usage for validate-ocp-build-data lacks a language specifier; update the fenced code block that contains the line "validate-ocp-build-data FILE [FILE ...]" to include "bash" (or "sh") after the opening backticks so the block reads as a bash shell snippet for proper syntax highlighting and markdown compliance. In `@agentic/domain/concepts/distgit.md`: - Line 22: The markdown references a stale specific line count "(~3062 lines)" for the Core module; update the sentence in agentic/domain/concepts/distgit.md to remove the numeric line count and use a general phrase such as "one of the largest modules" or "a large module" when referring to doozer/doozerlib/distgit.py, ensuring the offending substring "(~3062 lines)" is deleted or replaced. In `@agentic/domain/concepts/metadata.md`: - Around line 26-36: The fenced block showing the Metadata class hierarchy uses a bare code fence (```); update that fence to include a language hint (use ```text) so the block becomes a fenced "text" block; locate the block containing "MetadataBase (artcommon/artcommonlib/metadata.py)" and the indented tree with "Metadata (doozer/doozerlib/metadata.py)" and "Metadata (elliott/elliottlib/metadata.py)" and change the opening fence to ```text (and the closing fence remains ```); this aligns the block with the project's markdown linting conventions. In `@agentic/domain/concepts/ocp-build-data.md`: - Around line 37-53: The code block under the "### Repository Structure" section that begins with "openshift-4.17/" should include a language specifier for proper rendering and linting; replace the opening triple backticks (```) with a language-tagged fence such as ```text (i.e., change the code block delimiter in ocp-build-data.md's Repository Structure code block to ```text) so the block is explicitly marked as plain text. In `@agentic/domain/workflows/advisory-management.md`: - Around line 117-119: The ASCII state diagram code fences (e.g., the block containing "NEW_FILES --> QE --> REL_PREP --> PUSH_READY --> IN_PUSH --> SHIPPED_LIVE" and the subsequent diagram block later in the file) are missing a language specifier; update both fenced code blocks to use a language tag such as ```text so markdownlint MD040 is satisfied and the diagrams are treated as plain text. In `@agentic/domain/workflows/release-preparation.md`: - Around line 76-111: The fenced sequence-diagram code block that begins with "Assembly Author pyartcd elliott..." is missing a language identifier; update the opening fence from ``` to ```text (or ```plaintext) so the block is recognized as plain text and satisfies MD040; locate the diagram block in release-preparation.md and change only the opening fence to include the language identifier. In `@agentic/exec-plans/tech-debt-tracker.md`: - Around line 5-20: Update the Item Template code block in agentic/exec-plans/tech-debt-tracker.md to add a markdown language specifier on the opening fence so the template (starting with "### [DEBT-NNNN] Short description") is fenced as ```markdown; ensure the opening fence includes "markdown" and the closing fence remains present so syntax highlighting and linting pick it up. In `@agentic/QUALITY_SCORE.md`: - Around line 99-122: Consolidate the two same-date entries under "## Recent Changes and Progress" (the headings "2026-04-16: Metrics and Quality Scoring Implementation" and "2026-04-16: Initial Framework Implementation") into a single 2026-04-16 entry that lists the shared Score (81/100) once and merges the "What Changed" and "Created" bullet items into one combined section so the metrics additions (agentic/scripts/, Makefile targets, QUALITY_SCORE.md changes) and the framework artifacts (directory structure, AGENTS.md, ARCHITECTURE.md, concept/workflow/component docs, ADRs, exec-plan template, tech-debt tracker, CI validation, and documentation files) are all included under a single header for clarity. In `@agentic/scripts/generate-metrics-dashboard.py`: - Around line 23-28: The subprocess.run call builds the argv list by concatenating ['python3', str(script_path)] and list(args); replace this concatenation with tuple/list unpacking so the argv is constructed with the base items followed by the unpacked args (i.e., use the unpacking form with *args) in the subprocess.run invocation where result is created, referencing subprocess.run, script_path, and args. In `@agentic/scripts/measure-all-metrics.sh`: - Around line 89-91: The unquoted variable expansions can break on paths with spaces; update the invocation that sets NAV_OUTPUT to quote the SCRIPT_DIR path and the command substitution (use python3 "$SCRIPT_DIR/measure-navigation-depth.py" --max-depth 3 2>&1 inside the $(...) so the script path is quoted) and likewise apply the same quoting fixes for the other two occurrences mentioned (around lines with similar python3 $SCRIPT_DIR/... calls), keeping echo "$NAV_OUTPUT" as-is. In `@agentic/scripts/measure-navigation-depth.py`: - Around line 57-58: The catch-all except in the parsing code currently printing "Warning: Could not parse {file_path}: {e}" should be narrowed: replace "except Exception as e" around the parsing logic that references file_path with targeted except clauses (e.g., except json.JSONDecodeError as e, except UnicodeDecodeError as e, except FileNotFoundError as e, except ValueError as e) each logging the warning with the same print call; optionally add a final bare "except Exception:" that re-raises unexpected errors so truly unexpected issues are not silently swallowed. Ensure you keep the same print message and only handle specific parse/IO-related exceptions for the code that reads/parses file_path. In `@agentic/scripts/test-metrics.sh`: - Around line 38-40: The current extraction of totals from OUTPUT using fixed awk field positions is brittle; instead parse by the colon-delimited value and trim whitespace so changes in leading spacing or padding won't break TOTAL, REACHABLE, and UNREACHABLE. Update the three assignment lines that reference OUTPUT (TOTAL, REACHABLE, UNREACHABLE) to split on ":" (e.g., using awk -F: or cut -d:) and trim surrounding whitespace from the right-hand field; this keeps the extraction in sync with the print_report() output formatting in measure-navigation-depth.py. In `@ARCHITECTURE.md`: - Around line 25-45: The fenced ASCII diagram blocks in ARCHITECTURE.md are missing a language specifier causing markdownlint MD040; update the opening triple-backtick fences for the diagram(s) (the block that begins with the ASCII art showing pyartcd/doozer/elliott/artcommon and the other data flow diagram mentioned at line 67) to include a language specifier such as text (e.g., change ``` to ```text) so the linter stops complaining and the diagrams remain plain text.🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID:
ecd1fd43-2108-4eca-8294-f9032770450a⛔ Files ignored due to path filters (1)
agentic/generated/README.mdis excluded by!**/generated/**📒 Files selected for processing (47)
.github/workflows/validate-agentic-docs.yml.gitignoreAGENTS.mdARCHITECTURE.mdCLAUDE.mdMakefileagentic/DESIGN.mdagentic/DEVELOPMENT.mdagentic/QUALITY_SCORE.mdagentic/SECURITY.mdagentic/TESTING.mdagentic/decisions/adr-0001-monorepo-structure.mdagentic/decisions/adr-0002-runtime-pattern.mdagentic/decisions/adr-0003-dual-build-system.mdagentic/decisions/adr-template.mdagentic/decisions/index.mdagentic/design-docs/components/artcommon.mdagentic/design-docs/components/doozer.mdagentic/design-docs/components/elliott.mdagentic/design-docs/components/pyartcd.mdagentic/design-docs/components/validator.mdagentic/design-docs/core-beliefs.mdagentic/design-docs/index.mdagentic/domain/concepts/assembly.mdagentic/domain/concepts/brew-koji.mdagentic/domain/concepts/distgit.mdagentic/domain/concepts/errata-advisories.mdagentic/domain/concepts/konflux.mdagentic/domain/concepts/metadata.mdagentic/domain/concepts/model-missing.mdagentic/domain/concepts/ocp-build-data.mdagentic/domain/concepts/plashet.mdagentic/domain/concepts/runtime.mdagentic/domain/glossary.mdagentic/domain/index.mdagentic/domain/workflows/advisory-management.mdagentic/domain/workflows/image-build-lifecycle.mdagentic/domain/workflows/release-preparation.mdagentic/exec-plans/active/implement-agentic-docs.mdagentic/exec-plans/tech-debt-tracker.mdagentic/exec-plans/template.mdagentic/references/index.mdagentic/scripts/generate-metrics-dashboard.pyagentic/scripts/measure-all-metrics.shagentic/scripts/measure-context-budget.pyagentic/scripts/measure-navigation-depth.pyagentic/scripts/test-metrics.sh
Comment on lines
+88
to
+117
| - name: Check for broken internal links | ||
| run: | | ||
| broken=0 | ||
| while IFS= read -r file; do | ||
| # Extract relative markdown links | ||
| grep -oP '\[.*?\]\(\./[^)]+\)' "$file" 2>/dev/null | grep -oP '\(\K[^)]+' | while read -r link; do | ||
| # Remove anchor | ||
| path="${link%%#*}" | ||
| # Resolve relative to the file's directory | ||
| dir=$(dirname "$file") | ||
| resolved="$dir/$path" | ||
| if [ ! -f "$resolved" ] && [ ! -d "$resolved" ]; then | ||
| echo "Broken link in $file: $link (resolved to $resolved)" | ||
| broken=$((broken + 1)) | ||
| fi | ||
| done | ||
| done < <(find agentic -name "*.md" -type f) | ||
|
|
||
| # Also check root md files | ||
| for file in AGENTS.md ARCHITECTURE.md; do | ||
| if [ -f "$file" ]; then | ||
| grep -oP '\[.*?\]\(\./[^)]+\)' "$file" 2>/dev/null | grep -oP '\(\K[^)]+' | while read -r link; do | ||
| path="${link%%#*}" | ||
| if [ ! -f "$path" ] && [ ! -d "$path" ]; then | ||
| echo "Broken link in $file: $link" | ||
| fi | ||
| done | ||
| fi | ||
| done | ||
| echo "Link check complete" |
Contributor
There was a problem hiding this comment.
Broken link counter never increments due to subshell.
The piped while read loops create subshells, so broken=$((broken + 1)) modifies a local copy. The outer $broken stays 0 and the check always passes. This is a critical issue that renders the link validation ineffective.
🐛 Suggested fix using process substitution
- name: Check for broken internal links
run: |
broken=0
- while IFS= read -r file; do
- # Extract relative markdown links
- grep -oP '\[.*?\]\(\./[^)]+\)' "$file" 2>/dev/null | grep -oP '\(\K[^)]+' | while read -r link; do
- # Remove anchor
- path="${link%%#*}"
- # Resolve relative to the file's directory
- dir=$(dirname "$file")
- resolved="$dir/$path"
- if [ ! -f "$resolved" ] && [ ! -d "$resolved" ]; then
- echo "Broken link in $file: $link (resolved to $resolved)"
- broken=$((broken + 1))
- fi
- done
- done < <(find agentic -name "*.md" -type f)
+ while IFS= read -r file; do
+ # Extract relative markdown links
+ while IFS= read -r link; do
+ # Remove anchor
+ path="${link%%#*}"
+ # Resolve relative to the file's directory
+ dir=$(dirname "$file")
+ resolved="$dir/$path"
+ if [ ! -f "$resolved" ] && [ ! -d "$resolved" ]; then
+ echo "Broken link in $file: $link (resolved to $resolved)"
+ broken=$((broken + 1))
+ fi
+ done < <(grep -oP '\[.*?\]\(\./[^)]+\)' "$file" 2>/dev/null | grep -oP '\(\K[^)]+')
+ done < <(find agentic -name "*.md" -type f)
+
+ echo "Found $broken broken links"
+ if [ "$broken" -gt 0 ]; then
+ exit 1
+ fi🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/validate-agentic-docs.yml around lines 88 - 117, The
broken-link counter never increments because the inner "grep ... | while read -r
link; do ... broken=$((broken + 1)); done" pipelines run in subshells; change
those piped while loops to use process substitution (e.g., replace "grep ... |
while read -r link; do" with "while read -r link; do ... done < <(grep ... )")
so broken is updated in the main shell, apply this to both the per-file inner
loop and the root-file loop, and ensure after the scans you check "if [ $broken
-gt 0 ]; then exit 1; fi" to fail the job when broken > 0.
Comment on lines
+167
to
+184
| - name: Validate code paths referenced in docs | ||
| run: | | ||
| broken=0 | ||
| while IFS= read -r doc; do | ||
| # Extract backtick-quoted file paths that look like Python source files | ||
| grep -oP '`(?:artcommon/artcommonlib|doozer/doozerlib|elliott/elliottlib|pyartcd/pyartcd|ocp-build-data-validator/validator)/[a-zA-Z0-9_/]+\.py`' "$doc" 2>/dev/null | tr -d '`' | sort -u | while read -r path; do | ||
| if [ ! -f "$path" ]; then | ||
| echo "BROKEN PATH in $doc: $path does not exist" | ||
| broken=$((broken + 1)) | ||
| fi | ||
| done | ||
| done < <(find agentic AGENTS.md ARCHITECTURE.md -name "*.md" -type f 2>/dev/null) | ||
|
|
||
| if [ "$broken" -gt 0 ]; then | ||
| echo "Found references to non-existent source files. Update the docs." | ||
| exit 1 | ||
| fi | ||
| echo "Code path validation OK" |
Contributor
There was a problem hiding this comment.
Same subshell issue affects code path validation.
The nested pipe at line 172 (grep ... | while read) creates a subshell where broken is incremented locally. The outer check at line 180 always sees broken=0.
🐛 Suggested fix
- name: Validate code paths referenced in docs
run: |
broken=0
while IFS= read -r doc; do
# Extract backtick-quoted file paths that look like Python source files
- grep -oP '`(?:artcommon/artcommonlib|doozer/doozerlib|elliott/elliottlib|pyartcd/pyartcd|ocp-build-data-validator/validator)/[a-zA-Z0-9_/]+\.py`' "$doc" 2>/dev/null | tr -d '`' | sort -u | while read -r path; do
+ while IFS= read -r path; do
if [ ! -f "$path" ]; then
echo "BROKEN PATH in $doc: $path does not exist"
broken=$((broken + 1))
fi
- done
+ done < <(grep -oP '`(?:artcommon/artcommonlib|doozer/doozerlib|elliott/elliottlib|pyartcd/pyartcd|ocp-build-data-validator/validator)/[a-zA-Z0-9_/]+\.py`' "$doc" 2>/dev/null | tr -d '`' | sort -u)
done < <(find agentic AGENTS.md ARCHITECTURE.md -name "*.md" -type f 2>/dev/null)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - name: Validate code paths referenced in docs | |
| run: | | |
| broken=0 | |
| while IFS= read -r doc; do | |
| # Extract backtick-quoted file paths that look like Python source files | |
| grep -oP '`(?:artcommon/artcommonlib|doozer/doozerlib|elliott/elliottlib|pyartcd/pyartcd|ocp-build-data-validator/validator)/[a-zA-Z0-9_/]+\.py`' "$doc" 2>/dev/null | tr -d '`' | sort -u | while read -r path; do | |
| if [ ! -f "$path" ]; then | |
| echo "BROKEN PATH in $doc: $path does not exist" | |
| broken=$((broken + 1)) | |
| fi | |
| done | |
| done < <(find agentic AGENTS.md ARCHITECTURE.md -name "*.md" -type f 2>/dev/null) | |
| if [ "$broken" -gt 0 ]; then | |
| echo "Found references to non-existent source files. Update the docs." | |
| exit 1 | |
| fi | |
| echo "Code path validation OK" | |
| - name: Validate code paths referenced in docs | |
| run: | | |
| broken=0 | |
| while IFS= read -r doc; do | |
| # Extract backtick-quoted file paths that look like Python source files | |
| while IFS= read -r path; do | |
| if [ ! -f "$path" ]; then | |
| echo "BROKEN PATH in $doc: $path does not exist" | |
| broken=$((broken + 1)) | |
| fi | |
| done < <(grep -oP '`(?:artcommon/artcommonlib|doozer/doozerlib|elliott/elliottlib|pyartcd/pyartcd|ocp-build-data-validator/validator)/[a-zA-Z0-9_/]+\.py`' "$doc" 2>/dev/null | tr -d '`' | sort -u) | |
| done < <(find agentic AGENTS.md ARCHITECTURE.md -name "*.md" -type f 2>/dev/null) | |
| if [ "$broken" -gt 0 ]; then | |
| echo "Found references to non-existent source files. Update the docs." | |
| exit 1 | |
| fi | |
| echo "Code path validation OK" |
🧰 Tools
🪛 actionlint (1.7.12)
[error] 168-168: shellcheck reported issue in this script: SC2016:info:4:12: Expressions don't expand in single quotes, use double quotes for that
(shellcheck)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/validate-agentic-docs.yml around lines 167 - 184, The
problem is the inner "grep ... | while read -r path; do" creates a subshell so
updates to the broken variable are lost; replace the piped while with a while
loop that reads from process substitution or capture grep output into a variable
and iterate in the same shell (e.g. change "grep ... | while read -r path; do"
to "while read -r path; do ... done < <(grep -oP '...pattern...' \"$doc\"
2>/dev/null)" so increments of broken (the broken variable) occur in the main
shell and the final if ("if [ \"$broken\" -gt 0 ]") sees the correct value.
Comment on lines
+23
to
+27
| ``` | ||
| artcommonlib.runtime.GroupRuntime (ABC) | ||
| |-- doozerlib.runtime.Runtime | ||
| |-- elliottlib.runtime.Runtime | ||
| ``` |
Contributor
There was a problem hiding this comment.
Add a language identifier to the fenced code block.
Line 23 uses a bare triple-backtick block, which triggers markdownlint MD040 in docs CI. Use text for this hierarchy block.
Suggested patch
-```
+```text
artcommonlib.runtime.GroupRuntime (ABC)
|-- doozerlib.runtime.Runtime
|-- elliottlib.runtime.Runtime</details>
<details>
<summary>🧰 Tools</summary>
<details>
<summary>🪛 markdownlint-cli2 (0.22.0)</summary>
[warning] 23-23: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
</details>
</details>
<details>
<summary>🤖 Prompt for AI Agents</summary>
Verify each finding against the current code and only fix it if needed.
In @agentic/decisions/adr-0002-runtime-pattern.md around lines 23 - 27, The
fenced code block showing the runtime hierarchy (containing
artcommonlib.runtime.GroupRuntime, doozerlib.runtime.Runtime,
elliottlib.runtime.Runtime) is missing a language identifier and triggers
markdownlint MD040; update the triple-backtick fence to include the identifier
text (i.e., replace the opening withtext) so the block is treated as
plain text.
</details>
<!-- fingerprinting:phantom:poseidon:hawk:2623d9d4-04a9-45b5-a6bf-f6d52c1694f5 -->
<!-- This is an auto-generated comment by CodeRabbit -->
|
|
||
| ### Build Ordering | ||
|
|
||
| Images are built in dependency order. `Runtime.generate_image_tree()` builds a tree of parent-child relationships and produces `image_order` -- a flat list where parents always precede children. Cyclic dependencies are detected and rejected during initialization. |
Contributor
There was a problem hiding this comment.
Remove the unsupported cycle-detection claim.
Line 110 states cyclic dependencies are detected/rejected, but doozer/doozerlib/runtime.py:1005-1031 shows dependency ordering construction without explicit cycle detection logic. Please either reference the actual cycle-check implementation or remove that claim.
Suggested patch
-Images are built in dependency order. `Runtime.generate_image_tree()` builds a tree of parent-child relationships and produces `image_order` -- a flat list where parents always precede children. Cyclic dependencies are detected and rejected during initialization.
+Images are built in dependency order. `Runtime.generate_image_tree()` builds a tree of parent-child relationships and produces `image_order` -- a flat list where parents always precede children.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| Images are built in dependency order. `Runtime.generate_image_tree()` builds a tree of parent-child relationships and produces `image_order` -- a flat list where parents always precede children. Cyclic dependencies are detected and rejected during initialization. | |
| Images are built in dependency order. `Runtime.generate_image_tree()` builds a tree of parent-child relationships and produces `image_order` -- a flat list where parents always precede children. |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@agentic/design-docs/components/doozer.md` at line 110, The doc incorrectly
claims cycles are detected/rejected; update the sentence in doozer.md to either
remove the cycle-detection assertion or point to the real implementation:
reference Runtime.generate_image_tree() and the actual function that performs
cycle checks (or its absence) in the runtime codebase. Concretely, edit the line
to say "Cyclic dependencies are not explicitly detected here;
generate_image_tree() produces image_order (parents before children) and cycle
handling is implemented elsewhere if present" or replace with a correct
reference to the real cycle-check routine if you locate it in the runtime
module.
Comment on lines
+29
to
+33
| ``` | ||
| GroupRuntime (ABC) # artcommon/artcommonlib/runtime.py | ||
| -> doozerlib.runtime.Runtime # doozer/doozerlib/runtime.py | ||
| -> elliottlib.runtime.Runtime # elliott/elliottlib/runtime.py | ||
| ``` |
Contributor
There was a problem hiding this comment.
Add a language identifier to the fenced code block.
Line 29 uses an untyped fenced block; this can fail MD040 markdownlint checks.
Proposed fix
-```
+```text
GroupRuntime (ABC) # artcommon/artcommonlib/runtime.py
-> doozerlib.runtime.Runtime # doozer/doozerlib/runtime.py
-> elliottlib.runtime.Runtime # elliott/elliottlib/runtime.py</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.22.0)
[warning] 29-29: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@agentic/design-docs/core-beliefs.md` around lines 29 - 33, The fenced code
block containing "GroupRuntime (ABC) -> doozerlib.runtime.Runtime ->
elliottlib.runtime.Runtime" should include a language identifier to satisfy
MD040; change the opening fence from ``` to ```text so the block becomes a
labeled text code fence (ensure the closing fence remains ```), updating the
block that lists GroupRuntime and the runtime paths.
Comment on lines
+75
to
+88
| ## Total Score: 81/100 (approximately) | ||
|
|
||
| | Category | Score | Max | | ||
| |----------|-------|-----| | ||
| | Navigation | 15 | 20 | | ||
| | Completeness | 20 | 20 | | ||
| | Freshness | 18 | 20 | | ||
| | Consistency | 20 | 20 | | ||
| | Correctness | 13 | 15 | | ||
| | Utility | 8 | 10 | | ||
| | Automation | 15 | 15 | | ||
| | **Total** | **109** | **120** | | ||
|
|
||
| **Automated Score** (from `measure-all-metrics.sh`): **81/100** |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check what scores the metrics script generates
if [ -f "agentic/scripts/measure-all-metrics.sh" ]; then
echo "=== Metrics script exists ==="
grep -A 20 "TOTAL_SCORE=" agentic/scripts/measure-all-metrics.sh | head -30
fiRepository: openshift-eng/art-tools
Length of output: 716
🏁 Script executed:
#!/bin/bash
# Run the metrics script to see actual output
if [ -f "agentic/scripts/measure-all-metrics.sh" ]; then
echo "=== Running metrics script ==="
bash agentic/scripts/measure-all-metrics.sh 2>&1 | tail -50
else
echo "Script not found"
fiRepository: openshift-eng/art-tools
Length of output: 2122
🏁 Script executed:
#!/bin/bash
# Check the actual QUALITY_SCORE.md file at the specified lines
if [ -f "agentic/QUALITY_SCORE.md" ]; then
echo "=== Full QUALITY_SCORE.md content ==="
cat -n agentic/QUALITY_SCORE.md
else
echo "File not found"
fiRepository: openshift-eng/art-tools
Length of output: 7477
Remove or reconcile the conflicting scoring systems in QUALITY_SCORE.md.
The manual scoring table at lines 77–86 (109/120 = 90.8%) uses seven categories, while the automated script produces 81/100 using four different metrics (Navigation Depth, Context Budget, Structure Compliance, Documentation Coverage). These two systems measure different aspects and yield incompatible totals.
The file should either:
- Align the table at lines 77–86 to reflect the actual automated metrics (50, 75, 100, 100), or
- Clearly separate the manual assessment table from the automated score with explicit headers explaining they serve different purposes.
Currently, placing them side-by-side without clarification risks confusion about the actual documentation quality.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@agentic/QUALITY_SCORE.md` around lines 75 - 88, The manual scoring table
under the "Total Score" header (the 7-category table showing **109/120**)
conflicts with the "Automated Score (from `measure-all-metrics.sh`):
**81/100**"; either reconcile them or separate them clearly: update the manual
table to compute and display the automated metrics (Navigation Depth, Context
Budget, Structure Compliance, Documentation Coverage with values 50, 75, 100,
100) so both scores align, or add explicit section headers and a short
explanatory sentence separating the "Manual Assessment" table (keep the
7-category breakdown) from the "Automated Score" block (retain the
`measure-all-metrics.sh` result) so readers understand they are different
measurements—make the chosen change in QUALITY_SCORE.md, preserving both the
"Total Score" heading and the `measure-all-metrics.sh` reference.
Comment on lines
+126
to
+133
| def generate_html_dashboard(nav_metrics: dict, budget_metrics: dict, output_path: Path): | ||
| """Generate HTML dashboard.""" | ||
| nav_score = 100 if nav_metrics['status'] == 'pass' else 50 | ||
| budget_score = 100 if budget_metrics['status'] == 'pass' else 75 | ||
| structure_score = 100 | ||
| coverage_score = 100 | ||
|
|
||
| overall_score = (nav_score + budget_score + structure_score + coverage_score) // 4 |
Contributor
There was a problem hiding this comment.
Hardcoded scores may produce misleading dashboard results.
structure_score and coverage_score are hardcoded to 100, which inflates the overall score regardless of actual structure/coverage status. If these metrics aren't implemented yet, consider:
- Documenting this as a TODO
- Excluding them from the overall calculation
- Displaying them as "N/A" in the dashboard
📝 Suggested fix (option: exclude from calculation)
nav_score = 100 if nav_metrics['status'] == 'pass' else 50
budget_score = 100 if budget_metrics['status'] == 'pass' else 75
- structure_score = 100
- coverage_score = 100
+ # TODO: Implement structure and coverage metric scripts
+ # For now, only use navigation and budget scores
- overall_score = (nav_score + budget_score + structure_score + coverage_score) // 4
+ overall_score = (nav_score + budget_score) // 2🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@agentic/scripts/generate-metrics-dashboard.py` around lines 126 - 133, In
generate_html_dashboard, structure_score and coverage_score are hardcoded to 100
which inflates overall_score; change the logic so structure_score and
coverage_score are treated as optional: read actual values if present (e.g.,
nav_metrics.get('structure'), budget_metrics.get('coverage') or dedicated keys),
otherwise set them to None and render "N/A" in the dashboard, and compute
overall_score by averaging only the available (non-None) scores instead of
always dividing by 4; update the assignments for structure_score, coverage_score
and the overall_score calculation in generate_html_dashboard accordingly.
Comment on lines
+35
to
+37
| except Exception as e: | ||
| print(f"Warning: Could not read {file_path}: {e}", file=sys.stderr) | ||
| return 0 |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify broad exception handlers in this script via AST (read-only).
python - <<'PY'
import ast
from pathlib import Path
p = Path("agentic/scripts/measure-context-budget.py")
tree = ast.parse(p.read_text(encoding="utf-8"))
for node in ast.walk(tree):
if isinstance(node, ast.ExceptHandler):
t = node.type
is_broad = t is None or (isinstance(t, ast.Name) and t.id == "Exception")
if is_broad:
print(f"Broad except detected at Line {node.lineno}")
PYRepository: openshift-eng/art-tools
Length of output: 99
🏁 Script executed:
cat -n agentic/scripts/measure-context-budget.pyRepository: openshift-eng/art-tools
Length of output: 8938
Catch only expected I/O and decoding errors in count_lines().
Line 35 catches Exception, which can hide real logic bugs and silently turn them into 0 lines. The function only performs file I/O and string operations, so it should catch only OSError (from file open/read) and UnicodeDecodeError (from encoding issues).
Suggested patch
- except Exception as e:
+ except (OSError, UnicodeDecodeError) as e:
print(f"Warning: Could not read {file_path}: {e}", file=sys.stderr)
return 0📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| except Exception as e: | |
| print(f"Warning: Could not read {file_path}: {e}", file=sys.stderr) | |
| return 0 | |
| except (OSError, UnicodeDecodeError) as e: | |
| print(f"Warning: Could not read {file_path}: {e}", file=sys.stderr) | |
| return 0 |
🧰 Tools
🪛 Ruff (0.15.10)
[warning] 35-35: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@agentic/scripts/measure-context-budget.py` around lines 35 - 37, The catch in
count_lines() is too broad and should only handle expected I/O/decoding
failures; replace the generic "except Exception as e" with a narrow handler like
"except (OSError, UnicodeDecodeError) as e" so file open/read and decoding
errors are caught and logged/return 0 while other exceptions continue to
propagate; update the except block around the file open/read in count_lines()
accordingly and leave all other logic unchanged.
Comment on lines
+227
to
+231
| base_dir = Path.cwd() | ||
|
|
||
| results = [] | ||
| for workflow in WORKFLOWS: | ||
| results.append(workflow.measure(base_dir)) |
Contributor
There was a problem hiding this comment.
Resolve file paths relative to repo root, not current working directory.
Using Path.cwd() on Line 227 makes results dependent on where the command is invoked; running from a subdirectory can mark most files missing and under-report context usage.
Suggested patch
- base_dir = Path.cwd()
+ base_dir = Path(__file__).resolve().parents[2]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@agentic/scripts/measure-context-budget.py` around lines 227 - 231, Current
code uses Path.cwd() for base_dir which makes measurements depend on invocation
directory; replace base_dir = Path.cwd() with logic that locates the repository
root (for example run git rev-parse --show-toplevel and fallback to
Path(__file__).resolve().parent/ancestors) to produce an absolute Path for the
repo root, assign that to base_dir, and continue passing it into
workflow.measure for each item in WORKFLOWS; update references to base_dir,
WORKFLOWS, and workflow.measure accordingly so file lookups are performed
relative to the repo root instead of the current working directory.
Comment on lines
+5
to
+17
| ``` | ||
| artcommon/tests/ -- artcommon unit tests | ||
| doozer/tests/ -- doozer unit tests | ||
| tests/cli/ -- mirrors CLI command structure | ||
| tests/backend/ -- Konflux backend tests | ||
| tests/test_distgit/ -- distgit-specific tests | ||
| tests/resources/ -- test fixtures | ||
| elliott/tests/ -- elliott unit tests | ||
| pyartcd/tests/ -- pyartcd tests | ||
| ocp-build-data-validator/tests/ -- validator tests | ||
| doozer/tests_functional/ -- doozer functional tests (6 test files) | ||
| elliott/functional_tests/ -- elliott functional tests | ||
| ``` |
Contributor
There was a problem hiding this comment.
Specify a language for the directory-layout fenced block.
Line 5 uses an unlabeled fenced code block and triggers MD040. Use text for this tree-style snippet.
Suggested patch
-```
+```text
artcommon/tests/ -- artcommon unit tests
doozer/tests/ -- doozer unit tests
tests/cli/ -- mirrors CLI command structure
@@
doozer/tests_functional/ -- doozer functional tests (6 test files)
elliott/functional_tests/ -- elliott functional tests</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.22.0)
[warning] 5-5: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@agentic/TESTING.md` around lines 5 - 17, Add a language label to the
unlabeled fenced code block in TESTING.md (the directory-layout snippet) to
satisfy MD040; change the opening triple-backticks to include "text" so the
block becomes a ```text fenced block that preserves the tree-style formatting.
Contributor
|
@ashwindasr: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
joepvd
reviewed
Apr 17, 2026
| superseded-by: null | ||
| --- | ||
|
|
||
| # ADR-0003: Dual Build System Support (Brew and Konflux) |
Contributor
Author
|
/hold |
openshift-ci
Bot
added
the
do-not-merge/hold
Contributor
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-ci
Bot
added
the
needs-rebase
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://docs.google.com/document/d/1EqijnuwpKr4FtBLNQRojq7TU2VInE7AgBAYAXcLlI7U/edit?tab=t.0
Summary
What is included
Test plan
Generated with Claude Code (https://claude.com/claude-code)