Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

delete+insert materialization does not work with location property specified #467

Open
1 task done
yakovlevvs opened this issue Feb 20, 2025 · 1 comment · May be fixed by #468
Open
1 task done

delete+insert materialization does not work with location property specified #467

yakovlevvs opened this issue Feb 20, 2025 · 1 comment · May be fixed by #468
Labels
bug Something isn't working

Comments

@yakovlevvs
Copy link

Expected behavior

Dbt runs models with delete+insert materialization strategy and specified location consistently

Actual behavior

On the second and subsequent runs model refresh fails with error message:
message="Cannot create a table on a non-empty location: s3a://<location>, set 'iceberg.unique-table-location=true' in your Iceberg catalog properties to use unique table locations for every table.", query_id=<query id>)

Steps To Reproduce

Create a model with following config block and run it two times

    config(
        materialized='incremental',
        incremental_strategy='delete+insert',
        unique_key=['state', 'county', 'city'],
        properties= {
            "format": "'PARQUET'",
            "format_version": "2",
            "location": "'s3a://datalake/model'"
        }
    )

Log output/Screenshots

I think this happens because when we create a temporary table we pass properties block unchanged and Trino tries to create that table on the same location. Maybe if we add __dbt_tmp suffix to the name of temp table, we could patch the location the same way? This would make it's location unique and resolve this issue.

Operating System

Ubuntu 24.04.1 LTS

dbt version

1.9.2

Trino Server version

455

Python version

3.12.3

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@ArtemiyNaumov
Copy link

Heavy plus sign to this issue, it has pretty big negative impact on users, which can be confirmed by other issues (example 1, example 2)
Here we also put more detailed description of this problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants