Canvas Accessibility Linter

Hybrid accessibility auditing for Canvas pages using:

• deterministic browser-side checks (contrast, structure, pedagogy, link patterns), and
• AI-assisted remediation + vision review through a local FastAPI backend.

This README is the current implementation reference. Treat it as the source of truth for the code in this repository.

Current Architecture

Extension (Extension/)

• content_script.js
  • Runs deterministic checks directly in the page.
  • Assigns stable element selectors (data-ai-audit-id) for cross-process mapping.
  • Returns per-element metadata (selectors, bounding rects, DPR) for server-side cropping.
  • Waits briefly for visible images to finish loading before building the analysis snapshot.
• sidepanel.js
  • Calls backend endpoints for analysis and AI suggestions.
  • Sends element-level context for each action (not generic page prompts).
  • Refresh & Analyze refreshes the active tab, waits for load completion, extracts DOM metadata, then captures the screenshot used for cropping.

Backend (Backend/)

• main.py (FastAPI)
  • Runs deterministic issue normalization.
  • Runs AI vision checks for images.
  • Generates alt text, link text, and paragraph rewrites.
  • Handles retry, response parsing, and CreateAI payload shaping.

AI Data Flow (Important)

1) Vision Findings (/api/analyze)

1. Extension refreshes the page before analysis.
2. Content script waits briefly for visible images to load, then returns image element rects + selectors + DPR.
3. Extension captures the visible tab screenshot after extraction so DOM rects and screenshot pixels stay aligned.
4. Backend crops each image from screenshot using rect + DPR.
5. Backend normalizes crops (RGB flatten, resize/compress, payload variants).
6. Backend calls CreateAI endpoint: "vision" per image and converts the plain-text response into issue cards such as Semantic Mismatch and Decorative Image.

2) Alt Text Generation (/api/generate_alt)

1. Extension requests the target element rect (get_element_rect) from content script.
2. Extension sends full screenshot + element rect + DPR to backend.
3. Backend crops only the target element image.
4. AI receives element-level crop only (not full-page screenshot).
5. Backend asks CreateAI for plain text only, then returns JSON to the extension as { "alt_text": "..." }.

3) Paragraph Rewrite (/api/rewrite_paragraph)

1. Extension fetches exact paragraph text by selector (get_element_text).
2. If element text cannot be retrieved, request fails explicitly (no generic fallback text).
3. Backend rewrites only provided text.
4. Backend asks CreateAI for plain text only, normalizes the response, and returns { "rewritten_text": "..." }.

4) Non-Descriptive Link Rewrite (/api/generate_link_text)

1. Extension sends link URL + link text + parent context.
2. Parent context is capped to 500 chars to avoid noisy prompt injection.
3. Backend prompts AI to use only provided fields.

CreateAI Behavior

Request Source

• Vision requests use request_source: "override_params" because they require endpoint: "vision" and image_file.
• Alt text generation and paragraph rewrite also use request_source: "override_params" in a prompt-only mode so the backend can disable search/history and avoid relying on CreateAI's project retrieval state.
• Link text generation uses the normal text query flow unless explicit overrides are supplied.

Model Selection

• Vision requests do not force a model override unless one is supplied through explicit AI overrides.
• Prompt-only alt/rewrite requests preserve any explicit model_provider / model_name override the user supplies.
• If no model override is provided, CreateAI platform defaults are used.

Response Scope Guardrails

• Vision, alt-text, and rewrite flows do not require CreateAI to return model-generated JSON.
• The backend normalizes plain-text CreateAI responses locally and only the backend API response to the extension is JSON.
• Rewrite prompts explicitly instruct the model to use only the provided text.
• Alt-text prompts explicitly instruct the model to return only the alt text, with no preamble or JSON wrapper.

Vision Constraints

• Only images currently visible in the viewport are sent for AI analysis.
• Images smaller than 50x50 pixels are skipped.
• At most the first 15 visible images are analyzed per pass.
• Semantic mismatch is conservative: it is intended to flag objectively wrong alt text, not minor wording differences.

Setup

1) Backend environment

python3 -m venv .venv
source .venv/bin/activate
pip install -r Backend/requirements.txt

2) Backend .env

Create Backend/.env with at least:

CREATEAI_BASE_URL=https://api-main.aiml.asu.edu
CREATEAI_API_KEY=...
AI_PROVIDER=createai

Optional:

CREATEAI_TIMEOUT_SECONDS=120
CREATEAI_MAX_RETRIES=3

3) Extension

1. Open chrome://extensions
2. Enable Developer Mode
3. Load unpacked Extension/
4. Use the gear icon in the side panel to:
   • enable or disable Brand Violation analysis
   • adjust Brand Strictness from 0.0 (lenient) to 1.0 (strict)
   • edit or clear AI override JSON

Running

Use the root launcher:

python run.py

Expected:

• backend on http://127.0.0.1:8000
• auto-reload enabled

Health check:

curl http://127.0.0.1:8000/health

Local Test Harnesses

The repo includes two isolated scripts for debugging CreateAI behavior without the extension UI:

python3 test_vision.py /absolute/path/to/image.png
python3 text_rewrite.py --file test_paragraph.txt

Use these when you need to determine whether a failure is coming from:

• the browser extension payload,
• backend normalization logic, or
• CreateAI itself.

Troubleshooting

"Cannot answer because not part of database" (rewrite/link)

• Rewrite now uses a prompt-only CreateAI path and should return the rewritten paragraph itself, not database commentary.
• If this still appears, verify the backend was restarted after changes and confirm the paragraph text is actually being fetched from the page.
• Link generation still depends more heavily on surrounding context quality and may still drift if the available DOM context is weak.

Alt text or rewrite returns wrappers / commentary instead of the requested text

• The backend now strips common wrappers such as Suggestion: and Alt Text: before returning the result.
• The backend API still returns JSON to the extension, but CreateAI itself is no longer required to emit JSON for alt text or rewrite.

Link suggestion returns design standards instead of link wording

• Confirm capped link context is being sent (500 chars max).
• Confirm prompt-only scope instructions are present in backend prompts.

CreateAI vision error: "Failed to load image, your image is invalid"

• Confirm file uploads are enabled for the model on CreateAI platform.
• Backend now sends normalized image variants (JPEG/PNG, raw/data URI fallback).
• Check backend logs for variant diagnostics (name, dimensions, bytes).

Alt generation seems unrelated to target image

• Verify extension can fetch target selector rect.
• Alt endpoint now requires element rect; it should fail explicitly if rect is missing.
• The screenshot is captured after DOM extraction so crop alignment should now be more stable on refreshed pages.

Vision findings cards are missing

• Make sure the image is visible in the viewport when running Refresh & Analyze.
• Images smaller than 50x50 or beyond the first 15 visible images are not sent to CreateAI.
• Semantic mismatch is only expected when the existing alt text is meaningfully wrong. Broadly correct alt text like dog for a dog image may not be flagged.
• If needed, use test_vision.py to confirm CreateAI can describe the image independently of the extension.

I want to disable brand checks

• Open the side panel settings using the gear icon.
• Turn off Analyze Brand Violations.
• Run Refresh & Analyze again. This skips Brand Violation issue creation in deterministic analysis.

I want brand checks to be stricter or more lenient

• Open the side panel settings using the gear icon.
• Use Brand Strictness:
  • 0.0 = lenient, only clearly off-brand colors are flagged
  • 0.5 = current balanced default
  • 1.0 = strict, even small palette deviations are more likely to be flagged
• Run Refresh & Analyze again after saving.

Security Notes

• Keep Backend/.env private.
• Do not commit API keys or tokens.
• Restrict CORS + extension host permissions before production deployment.