Delta table not created as expected

[nicoladeseta]

What happened?

We're trying to use the databricks.Table to create an empty external Delta table in our Unity Catalog. The table is created but no transaction log is written in the external location (s3 in our case) making the table unusable (no insert can be performed).

Expected Behavior

A delta table is created along with transaction logs so that such table can be lately filled by spark jobs.

Steps to reproduce

Create a databricks.Table resource with the following configuration:

table_args = TableArgs(
            name="table_name",
            catalog_name="catalog_name",
            columns=[TableColumnArgs(...), ...],
            data_source_format="DELTA",
            schema_name="schema_name",
            table_type="EXTERNAL",
            owner="some_user",
            storage_location="s3://bucket/some/prefix/",
        )
Table(
            resource_name="resource_name",
            args=table_args,
            opts=ResourceOptions(
            replace_on_changes=["*"], delete_before_replace=True
)

Output of pulumi about

CLI          
Version      3.62.0
Go Version   go1.20.2
Go Compiler  gc

Plugins
NAME        VERSION
auth0       2.21.0
aws         5.41.0
cloudamqp   3.14.0
cloudflare  4.16.0
databricks  1.15.0
datadog     4.18.1
kubernetes  3.29.0
python      unknown
random      4.13.2

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in python: executable='/usr/bin/python3' version='3.10.6
'

Current Stack: organization/datasets/datasets.staging

TYPE                                URN
pulumi:providers:databricks         urn:pulumi:datasets.staging::datasets::pulumi:providers:databricks::default
pulumi:pulumi:Stack                 urn:pulumi:datasets.staging::datasets::pulumi:pulumi:Stack::datasets-datasets.staging
prima:index:DataProduct             urn:pulumi:datasets.staging::datasets::prima:index:DataProduct::data-product-test
pulumi:providers:databricks         urn:pulumi:datasets.staging::datasets::pulumi:providers:databricks::default_1_16_1
databricks:index/table:Table        urn:pulumi:datasets.staging::datasets::databricks:index/table:Table::ssdp_96_pulumi
databricks:index/table:Table        urn:pulumi:datasets.staging::datasets::databricks:index/table:Table::ssdp_96_pulumi_view
databricks:index/table:Table        urn:pulumi:datasets.staging::datasets::databricks:index/table:Table::cancellation_refund
databricks:index/grants:Grants      urn:pulumi:datasets.staging::datasets::databricks:index/grants:Grants::ssdp_96_pulumi_grants_284820640
databricks:index/grants:Grants      urn:pulumi:datasets.staging::datasets::databricks:index/grants:Grants::ssdp_96_pulumi_view_grants_284833021
databricks:index/grants:Grants      urn:pulumi:datasets.staging::datasets::databricks:index/grants:Grants::cancellation_refund_grants_284839307
databricks:index/grants:Grants      urn:pulumi:datasets.staging::datasets::databricks:index/grants:Grants::ssdp_96_pulumi_view_grants_284828176
databricks:index/grants:Grants      urn:pulumi:datasets.staging::datasets::databricks:index/grants:Grants::cancellation_refund_grants_284832154
databricks:index/grants:Grants      urn:pulumi:datasets.staging::datasets::databricks:index/grants:Grants::ssdp_96_pulumi_grants_284814011
databricks:index/sqlTable:SqlTable  urn:pulumi:datasets.staging::datasets::databricks:index/sqlTable:SqlTable::test_sql_table


Found no pending operations associated with datasets.staging

Backend        
Name           L-PA-PF2SL0B2
URL            s3://state-bucket
User           ****
Organizations  

Dependencies:
NAME               VERSION
black              22.12.0
data-prima-pulumi  0.3.0
flake8             3.9.2
isort              5.12.0
mypy               0.931.0
parameterized      0.9.0
pip                23.1.2
prima-pulumi       0.17.27
setuptools         67.7.2
wheel              0.40.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Thank you for reporting this issue!

Could this possibly be a configuration issue with the credentials not being picked up or not granting access to the bucket? Are you configuring them with

https://www.pulumi.com/registry/packages/databricks/api-docs/storagecredential/ and
https://github.com/databricks/terraform-provider-databricks/blob/master/docs/resources/storage_credential.md#example-usage

It does appear that even if it is the case Pulumi should be reporting an error when credentials are invalid or missing.

[gab-txt]

Hi, I am encountering the same issue. The table appears in Databricks web UI but no transaction log is written to S3, and if I navigate to the Sample Data tab I get an error:

Your request failed with status FAILED: [BAD_REQUEST] [DELTA_TABLE_NOT_FOUND] Delta table <table_name> doesn't exist.

Code:

table_resource = table.Table(
    table_name,
    catalog_name=catalog,
    schema_name=schema,
    name=table_name,
    columns=[
        table.TableColumnArgs(
            name="col1", position=0, type_name="STRING", type_text="varchar(64)", comment="first column",
            type_json="{\"name\":\"col1\",\"type\":\"varchar(64)\",\"nullable\":true,\"metadata\":{}}",
        ),
    ],
    data_source_format="DELTA",
    table_type="EXTERNAL",
    storage_location="<REDACTED>"
)

Pulumi about:

CLI          
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  archive     0.0.5
resource  auth0       2.24.3
resource  aws         5.43.0
resource  cloudamqp   3.17.5
resource  cloudflare  5.29.0
resource  command     0.9.2
resource  databricks  1.46.3
resource  datadog     4.28.0
resource  kubernetes  4.15.0
language  python      unknown
resource  random      4.16.1