Plan 007: Remove dead dependencies and unreachable code; align tooling target with Python 3.12

[strickvl]

Plan 007: Remove dead dependencies and unreachable code; align tooling target with Python 3.12

Executor instructions: Follow this plan step by step. Run every
verification command and confirm the expected result before moving to the
next step. If anything in the "STOP conditions" section occurs, stop and
report — do not improvise. When done, update the status row for this plan
in plans/README.md.

Drift check (run first): git diff --stat 5b5c634..HEAD -- pyproject.toml src/utils/embeddings.py src/eval src/frontend/routes/events.py src/frontend/app_config.py
If any in-scope file changed since this plan was written, compare the
"Current state" excerpts against the live code before proceeding; on a
mismatch, treat it as a STOP condition.

Status

• Priority: P2
• Effort: S
• Risk: LOW
• Depends on: none (do BEFORE plan 009 — fewer imports for the type checker to chase)
• Category: tech-debt
• Planned at: commit 5b5c634, 2026-06-12

Why this matters

The dependency manifest forces ~150MB+ of scraping tooling (Playwright
browsers, etc.) onto every install even though it is used by exactly one
optional script; two declared dependencies (pyclean, aiohttp) are
imported nowhere; arrow exists for three lines that stdlib datetime
covers. Separately: src/utils/embeddings.py is unreachable — a package
directory of the same name shadows it, so Python can never import the file —
and src/eval/ contains only orphaned __pycache__ bytecode from a deleted
feature. Finally, ruff/black are configured for Python 3.9 while the project
requires ≥3.12, so the linter enforces compatibility with a Python this
project can't even run on.

Current state

All verified at the planning commit:

• pyproject.toml:
  • [project] dependencies includes pydantic, pyclean, requests, tqdm, beautifulsoup4>=4.13.3, playwright>=1.50.0, aiohttp>=3.11.12, ..., arrow>=1.3.0, ... and requires-python = ">=3.12".
  • [tool.black] target-version = ["py39"] — note black is not a
    dependency anywhere (not in dev extras or dependency-groups); the
    whole section is vestigial.
  • [tool.ruff] target-version = "py39".
• Import facts (re-verify with the grep gates in the steps):
  • pyclean: zero imports in src/, scripts/, tests/.
  • aiohttp: zero imports anywhere.
  • requests + bs4 + playwright: imported ONLY by
    scripts/get_miami_herald_articles.py (lines 10–12).
  • arrow: imported ONLY by src/frontend/routes/events.py (line 3; used
    at lines 42, 56, 61 — arrow.get(...) to parse/normalize dates for
    filtering).
• src/utils/embeddings.py — an 11-line re-export shim. The package
  src/utils/embeddings/ (with __init__.py) exists alongside it; Python's
  import system resolves packages before same-named modules, so the shim can
  never be imported. Proof command in Step 1.
• src/eval/ — contains ONLY __pycache__/ (no .py sources). Git history
  shows the eval harness was added and later removed; the bytecode is
  untracked leftovers.
• src/frontend/app_config.py:288 — main_layout(page_title, filter_panel, content, page_header_title=None, current_domain=...); the
  page_header_title parameter is never read in the function body.
• justfile — recipes fetch-miami and import-miami run the scraper
  scripts.
• README "Install dependencies" section (~line 97) documents uv sync and
  uv sync --extra local-embeddings.

Commands you will need

Purpose	Command	Expected on success
Relock	uv sync	exit 0
Lint+format	just format && just lint	exit 0
Tests	just test	all pass
Full CI parity	just ci	exit 0
Frontend smoke	just frontend + open /events	events page renders, date filter works

Scope

In scope:

• pyproject.toml, uv.lock
• src/utils/embeddings.py (delete)
• src/eval/ (delete directory)
• src/frontend/routes/events.py (arrow → datetime)
• src/frontend/app_config.py + callers of main_layout (drop dead param)
• README.md (scraping-extra note)
• justfile (comment on scraper recipes)

Out of scope (do NOT touch):

• scripts/get_miami_herald_articles.py itself — it keeps working once the
  extra is installed; no code changes.
• scripts/import_miami_herald_articles.py — possibly obsolete, but
  deciding its fate is not this plan.
• numpy, opentelemetry-*, instructor, and all other dependencies —
  only the four named ones move/go.
• src/utils/embeddings/ (the package) — only the shadowed FILE is deleted.

Git workflow

• Branch: advisor/007-deps-and-dead-code-cleanup
• Commit per step (these are independent, revertable units).
• Do NOT push or open a PR unless the operator instructed it.

Steps

Step 1: Delete unreachable and orphaned code

First prove the shim is shadowed:

uv run python -c "import src.utils.embeddings as m; print(m.__file__)"

Expected: a path ending in src/utils/embeddings/__init__.py (the package,
NOT embeddings.py). Then:

rm src/utils/embeddings.py
rm -rf src/eval

Verify: just test → all pass; uv run python -c "from src.utils.embeddings.manager import EmbeddingManager; print('ok')" → ok

Step 2: Replace arrow with stdlib datetime in events route

Read src/frontend/routes/events.py lines 30–70 for full context. The three
usages parse a value (string or datetime) into a comparable date object:
arrow.get(dt) (line 42), arrow.get(start_date) (line 56),
arrow.get(end_date) (line 61). Replace with a small local helper:

def _to_dt(value: Any) -> Optional[datetime]:
    if value is None:
        return None
    if isinstance(value, datetime):
        return value
    try:
        return datetime.fromisoformat(str(value))
    except ValueError:
        return None

and adjust the comparison sites accordingly (preserve the existing behavior
for missing/invalid dates — read what the current code does when
arrow.get fails and match it; if arrow.get failures are currently
uncaught, matching means letting ValueError propagate — in that case drop
the try/except from the helper).
Remove import arrow.

Verify: grep -rn "arrow" src/ --include="*.py" → no matches;
just test → all pass; frontend smoke on /events with a date filter

Step 3: Trim the manifest

In pyproject.toml:

1. Delete pyclean and aiohttp and arrow from [project] dependencies.

2. Move requests, beautifulsoup4>=4.13.3, playwright>=1.50.0 from
   dependencies into a new optional extra:

   [project.optional-dependencies]
   scraping = [
       "requests",
       "beautifulsoup4>=4.13.3",
       "playwright>=1.50.0",
   ]

   (Keep the existing anthropic / local-embeddings / dev extras as-is.)

3. Run the gate greps BEFORE relocking — all must return nothing:

   grep -rn "import pyclean\|import aiohttp\|from aiohttp\|import arrow\|from arrow" src scripts tests --include="*.py"
   grep -rln "import requests\|from requests\|from bs4\|import playwright\|from playwright" src tests --include="*.py"

4. uv sync to relock.

Verify: uv sync → exit 0; just test → all pass;
uv run python -c "import src.process_and_extract; print('pipeline imports ok')" → ok

Step 4: Align lint targets with Python 3.12

In pyproject.toml: delete the entire [tool.black] section (black is not
installed); change [tool.ruff] target-version = "py39" to "py312". Then
run just format — if the target change reformats files, include those
hunks; they are the point.

Verify: just lint → exit 0

Step 5: Drop the dead main_layout parameter

List callers: grep -rn "page_header_title" src/. Remove the
page_header_title: Optional[str] = None, parameter from main_layout
(src/frontend/app_config.py:288) and remove the argument from every caller
found. Because callers may pass current_domain positionally after it,
check each call site's argument style carefully — convert positional tails
to keywords where needed.

Verify: grep -rn "page_header_title" src/ → no matches; just test → all pass; frontend smoke (any page renders)

Step 6: Document the scraping extra

• README.md install section: add
  uv sync --extra scraping # only needed for the Miami Herald scraper scripts.
• justfile: comment above fetch-miami:
  # Requires: uv sync --extra scraping (playwright, beautifulsoup4, requests).

Verify: just ci → exit 0

Test plan

No new tests — this plan removes never-executed code and unused packages.
The full suite plus the two frontend smoke checks (events date filter, any
page rendering after the main_layout change) are the regression net.

Done criteria

• just ci exits 0
• src/utils/embeddings.py and src/eval/ do not exist
• grep -n "pyclean\|aiohttp\|arrow" pyproject.toml → no matches in dependencies
• grep -n "py39" pyproject.toml → no matches; no [tool.black] section
• grep -rn "page_header_title" src/ → no matches
• uv run python -c "import src.process_and_extract" exits 0
• No files outside the in-scope list modified (git status)
• plans/README.md status row updated

STOP conditions

Stop and report back (do not improvise) if:

• The Step 1 proof prints the embeddings.py path instead of the package —
  the import-shadowing premise is wrong; deleting the file would break imports.
• Any gate grep in Step 3 returns matches (a new import of these packages
  appeared since planning).
• arrow.get in events.py turns out to handle timezone-aware values or
  human-readable formats ("2 days ago") that datetime.fromisoformat
  cannot — report the actual formats observed in the data.
• Removing page_header_title requires touching more than ~10 call sites or
  any caller relies on it positionally in a way that can't be safely
  rewritten — stop and report the call-site list.

Maintenance notes

• New scripts that need HTTP/scraping should import from the scraping
  extra and say so in their docstring; core src/ must not grow
  dependencies on it.
• Reviewer should scrutinize: the events-date helper's behavior on malformed
  dates (must match the old arrow behavior), and the relocked uv.lock diff
  (expect only removals plus extras reshuffling).
• Deferred deliberately: deciding whether scripts/import_miami_herald_articles.py
  (old JSONL→SQLModel importer) should be deleted — likely yes, but confirm
  with the maintainer first.