fix(airbnb): materialize mom/wow agg_reviews as table to avoid incremental + LAG NULL bug - #145
Open
sahrizvi wants to merge 1 commit into
Open
Conversation
…ental + LAG NULL bug mom_agg_reviews and wow_agg_reviews are materialized=incremental with LAG(REVIEW_TOTALS, N) window functions (N=29 for MoM, N=6 for WoW). On incremental runs, dates_cte collapses to a single date, so the LAG window partition has only one row per sentiment. LAG(N) then returns NULL, and the unique_key='DATE_SENTIMENT_ID' upsert overwrites the correctly- computed first-run values. dbt-labs#106 added --full-refresh to test_setup to mask the failure at grading time, but the underlying SQL bug is unchanged — any consumer running dbt build twice still gets NULL last-date MoM/WoW values. Table materialization eliminates the cross-batch lookback problem entirely. The airbnb fixture is small; the incremental optimization is not load-bearing here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
mom_agg_reviewsandwow_agg_reviewsarematerialized="incremental"withLAG(REVIEW_TOTALS, N)window functions (N=29 for MoM, N=6 for WoW). On incremental runs,dates_ctecollapses to a single date:{% if is_incremental() %} AND DATE_ACTUAL = (SELECT MAX(REVIEW_DATE::DATE) FROM {{ref('fct_reviews')}}) {% endif %}final_ctethen produces 3 rows total (one per sentiment, all for the max date).LAG(N)overPARTITION BY REVIEW_SENTIMENT ORDER BY AGGREGATION_DATEoperates on a 1-row partition and returns NULL. Theunique_key='DATE_SENTIMENT_ID'upsert then overwrites the correctly-computed first-run values with NULL.Prior art
#106 (merged 2026-02-09) added
--full-refreshtotest_setupfor tasksairbnb001,airbnb002,airbnb008. That masks the failure at grading time, but the underlying SQL bug in the model itself is unchanged — any consumer (agent, user, downstream pipeline) runningdbt buildtwice still sees NULL last-date MoM/WoW values.Fix
Switch
materializedfrom"incremental"to"table"for both models. The airbnb fixture is small; the incremental optimization isn't load-bearing here. Table materialization sidesteps the cross-batch dependency cleanly.Reproduction
Run
dbt runtwice against the airbnb fixture without--full-refresh. After the second run, themom_agg_reviewsrows for the max review date (one per sentiment) haveMOM = NULL, overwriting the correct first-run values. Same forwow_agg_reviews.Alternative considered
Keeping
materialized="incremental"and looking up the prior comparison via a self-join to{{ this }}(joining onAGGREGATION_DATE = new_date - N) preserves the incremental optimization. The trade-off is that the full-refresh path needs to keep the original LAG (since{{ this }}is empty/being-rebuilt on full refresh), so the final SELECT bifurcates onis_incremental(). Happy to revise to that shape if you prefer it.Out of scope
{% if is_incremental() %}blocks become dead code undermaterialized="table"but are left in place to keep the diff minimal. Happy to clean up in a follow-up.unique_key='DATE_SENTIMENT_ID'config is ignored fortablematerialization but left in place for the same reason.task.yaml --full-refreshworkarounds from fix(airbnb): use --full-refresh in test_setup for incremental models #106 become redundant under this fix but are not reverted here.Discovery
Surfaced while running ADE-Bench DuckDB against an internal coding agent (altimate-code) on 2026-04-23 and tracing the failures in
airbnb001,airbnb002,airbnb008to a common root cause distinct from agent-side issues.