Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
55e5d18
fix(reports): enforce readiness recovery budget
fitzee Jul 30, 2026
163057e
fix(reports): narrow recovery and capture compatibility
fitzee Jul 30, 2026
7236b4c
style(reports): apply pinned Ruff formatting
fitzee Jul 30, 2026
3f232a2
fix(reports): preserve alert timeout semantics
fitzee Jul 31, 2026
830e6cb
fix(reports): address reliability review
fitzee Jul 31, 2026
d81ef61
fix(reports): terminalize failed captures
Jul 31, 2026
f27092c
fix(reports): preserve active same-id replays
Jul 31, 2026
ac3f21c
fix(reports): scope terminal retry to row owner
Jul 31, 2026
4ae2324
fix(reports): terminalize refused executions
Jul 31, 2026
2b745d0
test(reports): allow metadata timestamp precision
Jul 31, 2026
01a373e
Merge remote-tracking branch 'oss/master' into fix-report-readiness-r…
eschutho Jul 31, 2026
4b1ec35
fix(reports): honor per-schedule working_timeout and preserve default…
eschutho Jul 31, 2026
b2e0937
fix(reports): anchor non-report tiled budget clock at overall screens…
eschutho Aug 1, 2026
9b84ce7
test(reports): align scheduler budget test with the 3600s default
eschutho Aug 1, 2026
d06a3b5
test(reports): cover the shared soft-timeout handler for alerts
eschutho Aug 1, 2026
8feae1a
fix(ci): auto-walrus compliance, test persistence, and review-suggest…
eschutho Aug 1, 2026
ba7777d
docs(reports): remove stale 15-minute soft-limit reference
eschutho Aug 1, 2026
7e2010f
test(reports): cover delivery-phase gate and retry-net failure path; …
eschutho Aug 1, 2026
c77359a
fix(reports): chart-capture readiness logs report container state, no…
eschutho Aug 4, 2026
af5bcc2
chore: retrigger CI
eschutho Aug 5, 2026
f674957
Merge remote-tracking branch 'oss/master' into fix-report-readiness-r…
eschutho Aug 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions UPDATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,34 @@ assists people when migrating to a new version.

## Next

### Scheduled report execution now enforces one application deadline

Scheduled report (not alert) executions are now governed by a single
end-to-end deadline shared by browser readiness, capture/PDF generation,
notification delivery, and terminal-state persistence, configured via
`ALERT_REPORTS_EXECUTION_BUDGET_SECONDS` (with per-phase reserve settings).
Behavior changes to be aware of:

- The effective budget for a schedule is
`min(ALERT_REPORTS_EXECUTION_BUDGET_SECONDS, working_timeout)`. The default
budget (one hour) matches the historical `working_timeout` model default,
so default installations see no change in how long a report may run —
but reports now fail cleanly (with an error notification) at the deadline
instead of being killed silently by Celery.
- For REPORT schedules, the Celery `soft_time_limit`/`time_limit` are now
derived from that same effective budget plus
`ALERT_REPORTS_EXECUTION_HARD_TIMEOUT_GRACE_SECONDS`, replacing the
previous `working_timeout + ALERT_REPORTS_WORKING_TIME_OUT_LAG` /
`+ ALERT_REPORTS_WORKING_SOFT_TIME_OUT_LAG` derivation. Alert schedules
keep the previous behavior.
- A `working_timeout` smaller than the summed phase reserves is floored at
the minimum viable budget (reserves + 30s) with a warning; such reports
fail fast at the first phase check rather than erroring at setup.
- Dashboard reports whose charts have not mounted are no longer captured
blank: readiness is polled until the deadline, and the report fails loudly
if charts never mount. Thumbnails and non-report screenshots keep their
previous behavior.

### Principal listing APIs now honour related-field filters

Two authorization-related listing behaviors changed for API clients. Neither
Expand Down
47 changes: 47 additions & 0 deletions docs/admin_docs/configuration/alerts-reports.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,53 @@ class CeleryConfig:
}
CELERY_CONFIG = CeleryConfig

# Scheduled reports share one deadline across browser readiness, capture/PDF
# generation, delivery, and terminal-state persistence. The effective budget
# for a schedule is min(this value, the schedule's working_timeout), so the
# per-schedule field keeps its meaning as a user-facing cap. The default (one
# hour) matches the historical working_timeout default, so upgrading changes
# no default behavior; lower it to enforce a tighter report SLA.
ALERT_REPORTS_EXECUTION_BUDGET_SECONDS = 3600

# These reserves are part of (not additions to) the total budget and their sum
# must be less than it. Readiness polling stops in time to leave capacity for
# the later phases.
ALERT_REPORTS_EXECUTION_CAPTURE_RESERVE_SECONDS = 60
ALERT_REPORTS_EXECUTION_DELIVERY_RESERVE_SECONDS = 120
ALERT_REPORTS_EXECUTION_CLEANUP_RESERVE_SECONDS = 30

# Celery's hard limit leaves this additional window for terminal cleanup after
# the soft limit, which equals the resolved execution budget (the configured
# budget capped by each schedule's working_timeout).
# ALERT_REPORTS_WORKING_TIME_OUT_KILL controls these Celery limits; disabling
# it does not disable the application deadline above.
ALERT_REPORTS_EXECUTION_HARD_TIMEOUT_GRACE_SECONDS = 30
Comment on lines +263 to +268

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: This comment says the hard-limit grace follows a 15-minute soft limit, but the documented default execution budget and resulting soft limit are 3600 seconds (one hour). This gives operators an incorrect timeout expectation; describe the grace as following the resolved execution budget instead of naming 15 minutes. [comment mismatch]

Severity Level: Minor 🧹
- ⚠️ Operators receive incorrect Celery timeout guidance.
- ⚠️ Deployment timeout sizing may use the wrong soft-limit assumption.

Fix in Cursor Fix in VSCode Claude

(Use Cmd/Ctrl + Click for best experience)

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** docs/admin_docs/configuration/alerts-reports.mdx
**Line:** 262:265
**Comment:**
	*Comment Mismatch: This comment says the hard-limit grace follows a 15-minute soft limit, but the documented default execution budget and resulting soft limit are 3600 seconds (one hour). This gives operators an incorrect timeout expectation; describe the grace as following the resolved execution budget instead of naming 15 minutes.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct — stale leftover from the 900s draft. Fixed in ba7777d94a: the comment now describes the grace as following the resolved execution budget (configured budget capped by working_timeout) instead of naming 15 minutes.


# Invalid budget/reserve combinations fail application startup instead of
# allowing every scheduled report to fail later. A report Celery soft timeout
# records ERROR and increments `reports.execute.celery_soft_timeout`; it does
# not attempt an in-band customer error notification during the hard-limit
# grace window. Alert schedules retain their existing timeout notifications.
#
# The application deadline is cooperative between synchronous phases. The
# Celery limits provide the final preemption boundary when the worker pool
# supports them; PDF construction is checked immediately before and after the
# synchronous builder but cannot be interrupted inside that call.
#
# Sizing the budget against infrastructure limits:
# - Kubernetes (or similar) pod termination grace must exceed
# budget + hard-timeout grace, or in-flight reports are killed mid-run on
# every deploy/node drain despite the application deadline.
# - The web server's per-request timeout (e.g. gunicorn ``timeout``) bounds
# each individual chart data request made by the headless browser -- not
# the report as a whole. Readiness allowance beyond that per-request
# ceiling buys nothing for a single slow chart (its request dies at the
# web layer and the chart reaches an error state), but multi-chart and
# tiled captures legitimately accumulate total time well past it.

# Screenshot-specific waits continue to apply to thumbnails and other
# standalone screenshot calls. Scheduled reports derive their waits from the
# shared execution deadline above.
SCREENSHOT_LOCATE_WAIT = 100
SCREENSHOT_LOAD_WAIT = 600

Expand Down
Loading
Loading