Fix Snowflake external table 0-row bug with flat JSON array format

[concaf]

Summary

• Snowflake's VARIANT type has a 16MB per-row limit. The current format writes each stage's output as a single JSON object per source document (e.g., 195MB for embeddings), which exceeds this limit and causes external tables to silently return 0 rows despite files being registered in EXTERNAL_TABLE_FILE_REGISTRATION_HISTORY.
• Switch all pipeline stages (converted, chunks, embeddings) to output JSON arrays of flat row objects. With STRIP_OUTER_ARRAY=TRUE already set on the external tables, each array element becomes its own row (~10KB each), well under the 16MB limit.
• Add new row types (ConvertedRow, ChunkRow, EmbeddingRow) and update all controllers to write the new format while reading both old and new formats for backward compatibility.

Companion DPO MR: Updates materialized views to read the flat row format (no more LATERAL FLATTEN needed).

Test plan

• Deploy to sandbox and trigger a pipeline for a data product (e.g., aiftestgdrivear)
• Verify select count(*) from <db>.staging.embeddings_ext_tbl returns > 0
• Verify select count(*) from <db>.staging.chunks_ext_tbl returns > 0
• Verify materialized views (EMBEDDINGS_MV, CHUNKS_MV, CONVERTED_MV) return correct data
• Verify MCP tools (get_chunks, search) still work end-to-end
• Test with an existing data product that has old-format files — verify re-processing generates new format

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d2b983f2-4d69-4348-99b4-7f0c9e286704

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}