reporter-worker OOMKilled: unbounded memory usage in report generation pipeline

[gandalf-at-lerian]

Problem

The reporter-worker is getting OOMKilled in Firmino (namespace midaz-plugins-dev, container limit 4Gi). Multiple pods killed within minutes during report generation.

Root Cause

The entire report generation pipeline loads everything into memory with no pagination or streaming:

1. Queries without LIMIT

Query() and QueryWithAdvancedFilters() in pkg/postgres/datasource_query.go execute SELECT without LIMIT. If the template requests data from a large table, all rows are loaded into heap as []map[string]any.

2. Full data accumulation in memory

In generate-report.go, the result map accumulates data from all tables of all databases before passing to the renderer. Peak memory = sum of all queried data.

3. HTML → PDF pipeline without streaming

Flow: data in memory → Pongo2 renders full HTML string → writes HTML to temp file → Chrome headless loads and generates PDF as []byte → writes PDF to temp file → reads back to []byte → sends to SeaweedFS as string. At each stage the entire content lives in memory simultaneously.

4. Chrome + Go combined footprint

PDFChromeMaxOldSpaceSize is 512MB, container limit is 4Gi. The problem is the combination: raw query data + rendered HTML + Chrome DOM + PDF bytes all coexisting in the same container.

Proposed Optimizations (by impact)

1. Pagination/cursor on queries (highest impact) — Add LIMIT+OFFSET or cursor-based iteration in Query() and QueryWithAdvancedFilters(). A configurable MAX_QUERY_ROWS env var as safety net.
2. Template streaming — Pongo2 renders everything to a string. For large reports, consider a renderer that writes directly to an io.Writer (file-backed) instead of holding the full HTML in memory.
3. Report concurrency limiter — If two large reports run simultaneously, each loads its own data. A semaphore or bounded queue prevents parallel memory spikes.
4. PDF generation in sidecar — Move Chrome headless to a separate sidecar container with its own memory limit, isolating it from the Go process.

Files involved

• components/worker/internal/services/generate-report.go — main orchestration
• components/worker/internal/services/generate-report-data.go — data fetching (no pagination)
• components/worker/internal/services/generate-report-render.go — HTML rendering + PDF conversion
• pkg/postgres/datasource_query.go — unbounded SELECT queries
• pkg/pongo/renderer.go — in-memory template rendering
• pkg/pdf/pool.go — Chrome PDF worker pool
• pkg/constant/pdf.go — PDF constants (PDFChromeMaxOldSpaceSize = "512")