Skip to content

fix(airbnb): materialize mom/wow agg_reviews as table to avoid incremental + LAG NULL bug - #145

Open
sahrizvi wants to merge 1 commit into
dbt-labs:mainfrom
sahrizvi:fix/airbnb-mom-wow-incremental-lag
Open

fix(airbnb): materialize mom/wow agg_reviews as table to avoid incremental + LAG NULL bug#145
sahrizvi wants to merge 1 commit into
dbt-labs:mainfrom
sahrizvi:fix/airbnb-mom-wow-incremental-lag

Conversation

@sahrizvi

@sahrizvi sahrizvi commented May 19, 2026

Copy link
Copy Markdown

Problem

mom_agg_reviews and wow_agg_reviews are materialized="incremental" with LAG(REVIEW_TOTALS, N) window functions (N=29 for MoM, N=6 for WoW). On incremental runs, dates_cte collapses to a single date:

{% if is_incremental() %}
    AND DATE_ACTUAL = (SELECT MAX(REVIEW_DATE::DATE) FROM {{ref('fct_reviews')}})
{% endif %}

final_cte then produces 3 rows total (one per sentiment, all for the max date). LAG(N) over PARTITION BY REVIEW_SENTIMENT ORDER BY AGGREGATION_DATE operates on a 1-row partition and returns NULL. The unique_key='DATE_SENTIMENT_ID' upsert then overwrites the correctly-computed first-run values with NULL.

Prior art

#106 (merged 2026-02-09) added --full-refresh to test_setup for tasks airbnb001, airbnb002, airbnb008. That masks the failure at grading time, but the underlying SQL bug in the model itself is unchanged — any consumer (agent, user, downstream pipeline) running dbt build twice still sees NULL last-date MoM/WoW values.

Fix

Switch materialized from "incremental" to "table" for both models. The airbnb fixture is small; the incremental optimization isn't load-bearing here. Table materialization sidesteps the cross-batch dependency cleanly.

Reproduction

Run dbt run twice against the airbnb fixture without --full-refresh. After the second run, the mom_agg_reviews rows for the max review date (one per sentiment) have MOM = NULL, overwriting the correct first-run values. Same for wow_agg_reviews.

Alternative considered

Keeping materialized="incremental" and looking up the prior comparison via a self-join to {{ this }} (joining on AGGREGATION_DATE = new_date - N) preserves the incremental optimization. The trade-off is that the full-refresh path needs to keep the original LAG (since {{ this }} is empty/being-rebuilt on full refresh), so the final SELECT bifurcates on is_incremental(). Happy to revise to that shape if you prefer it.

Out of scope

  • The {% if is_incremental() %} blocks become dead code under materialized="table" but are left in place to keep the diff minimal. Happy to clean up in a follow-up.
  • The unique_key='DATE_SENTIMENT_ID' config is ignored for table materialization but left in place for the same reason.
  • The task.yaml --full-refresh workarounds from fix(airbnb): use --full-refresh in test_setup for incremental models #106 become redundant under this fix but are not reverted here.

Discovery

Surfaced while running ADE-Bench DuckDB against an internal coding agent (altimate-code) on 2026-04-23 and tracing the failures in airbnb001, airbnb002, airbnb008 to a common root cause distinct from agent-side issues.

…ental + LAG NULL bug

mom_agg_reviews and wow_agg_reviews are materialized=incremental with
LAG(REVIEW_TOTALS, N) window functions (N=29 for MoM, N=6 for WoW). On
incremental runs, dates_cte collapses to a single date, so the LAG window
partition has only one row per sentiment. LAG(N) then returns NULL, and
the unique_key='DATE_SENTIMENT_ID' upsert overwrites the correctly-
computed first-run values.

dbt-labs#106 added --full-refresh to test_setup to mask the failure at grading
time, but the underlying SQL bug is unchanged — any consumer running
dbt build twice still gets NULL last-date MoM/WoW values.

Table materialization eliminates the cross-batch lookback problem
entirely. The airbnb fixture is small; the incremental optimization is
not load-bearing here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant