[v2 Bug] [Databricks] Unit tests on incremental models with complex (STRUCT/MAP) columns fail: dbt1308 fetching schema / dbt1005 non-primitive type

[safdar-a]

Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x?

• I believe this is a new bug in dbt v2.x
• I have searched the existing issues and could not find a duplicate

Current Behavior

On Databricks, unit tests for materialized='incremental' models whose output schema contains complex columns (STRUCT / MAP / ARRAY) fail under dbt v2 (tested 2.0.5). Two error shapes, same root cause:

1. During unit-test rendering, v2 derives the expected-output schema by describing the model — this fails with:

[error] [DbDriverFailed (dbt1308)]: Remote database error while fetching schema for incremental unit test model-under-test relation from `expect` '`my_catalog`.`my_schema`.`my_model`'

With --debug, the schema-derivation step issues DESCRIBE TABLE EXTENDED … AS JSON on the model relation, a describe query over the model's rendered SQL, and information_schema.key_column_usage / table_constraints lookups — then errors.

2. When the schema is derived, complex columns are rejected outright:

[error] [InvalidConfig (dbt1005)]: The column 'attrs' has a non-primitive type 'Struct(...)'. Only primitive types like numeric, temporal and varchar are supported for unit_testing

Observed regardless of whether the model relation already exists (so not just a fresh-schema/ordering issue), and under both dbt test and dbt build. The same project and tests pass on dbt Core 1.11.11. Fixtures follow the documented Spark pattern (SQL-expression strings like named_struct(...) / map(...) in dict rows).

Expected Behavior

Unit tests on incremental models with complex-typed columns run under v2 as they do under v1. The given inputs are SQL expressions evaluated warehouse-side and the expect comparison also happens warehouse-side, so no client-side (Arrow) representation of complex values should be required.

Steps To Reproduce

models/my_incremental_model.sql:

{{ config(materialized='incremental', unique_key='id') }}

select
    id,
    attrs
from {{ source('raw', 'events') }}

models/_unit_tests.yml (documented Spark fixture pattern):

unit_tests:
  - name: my_incremental_model_struct_test
    model: my_incremental_model
    given:
      - input: source('raw', 'events')
        format: dict
        rows:
          - {id: 1, attrs: 'named_struct("a", 1, "b", "x")'}
    expect:
      format: dict
      rows:
        - {id: 1, attrs: 'named_struct("a", 1, "b", "x")'}

1. dbt run --select my_incremental_model (relation exists)
2. dbt test --select my_incremental_model_struct_test → fails with dbt1308/dbt1005 on v2.0.5; passes on dbt-core 1.11.11 (dbt-databricks 1.12.5)

Relevant log output

See error snippets above; full --debug log available on request.

Environment

• dbt v2.0.5 (self-contained binary via the dbt PyPI shim)
• Adapter: Databricks (built-in v2 adapter, ADBC) against a Databricks SQL warehouse
• Also reproduced on a Databricks serverless job environment (dbt_task)
• Baseline where it passes: dbt-core 1.11.11 / dbt-databricks 1.12.5
• Python 3.13, macOS (local); Linux (CI/jobs)

Which database adapter are you using?

databricks

Is this a discrepancy vs. dbt 1.x?

• Yes — this works in dbt 1.x but not in dbt v2.x

Additional Context

Related work in this area: #16038 (TIMESTAMP_NTZ — fixed), #15803 (expected-output column resolution — fixed), #15882 (ARRAY given input, BigQuery — fixed), #13867 (semistructured dict fixtures — fixed); still open: #15878 (BigQuery nested constraints), #16040 (incremental unit tests, dbt9002). The remaining uncovered case appears to be Databricks + incremental + STRUCT/MAP output columns. In our real project this affects unit tests so it blocks adopting v2 for CI.