Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Data Factory is not loading from Fabric Lakehouse with SchemaTypePropertiesSchema #48545

Open
mikeblakeuk opened this issue Mar 4, 2025 · 0 comments
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. Data Factory Mgmt This issue is related to a management-plane library. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@mikeblakeuk
Copy link

mikeblakeuk commented Mar 4, 2025

Library name and version

1.1 Azure.ResourceManager.DataFactory 1.6.0

Describe the bug

2.1 SchemaTypePropertiesSchema doesn't appear to work for Fabric Lakehouses

Expected behavior

3.1 Read the lakehouse table that has a table schema name

Actual behavior

4.1 Data Preview Fails with
Image

at Source 'source1': Path /1543dbb1-01eb-485d-8676-ec7f75752582/Tables/sample_datasets/ does not resolve to any file(s). Please make sure the file/folder exists and is not hidden. At the same time, please ensure special character is not included in file/folder name, for example, name starting with _

Reproduction Steps

4.1 Part A in Fabric
Create a Fabric Lakehouse with Schemas enabled
Create a schema
Load in a table
Image

Make sure the SP has access to the lakehouse/workspace

4.2 Part B In Data Factory
Add a Linked Service to the Fabric Lakehouse
Create a new Lakehouse Table dataset

Image

4.3 Try the Data Preview / Debug the pipeline

Image

Important
4.4 Data Preview on the source is not the same as the Preview data on the Dataset [works]

Environment

5.1 Azure Data Factory (before we even get to the SDK)
5.2 Example code

var linkedServiceReference = new DataFactoryLinkedServiceReference(DataFactoryLinkedServiceReferenceKind.LinkedServiceReference, dataFactoryLinkedServiceName);
var sourceDataset = new LakeHouseTableDataset(linkedServiceReference)
{
    Table = "sample_datasets",
    SchemaTypePropertiesSchema = "myschema"
};

sourceDataset.GetType().GetProperty("DatasetType", BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic)?.SetValue(sourceDataset, "LakehouseTable");

var datasetData = new DataFactoryDatasetData(sourceDataset);
var datasetOperation = dataFactoryResource.GetDataFactoryDatasets().CreateOrUpdate(WaitUntil.Completed, dataFactorySourceName, datasetData);
Console.WriteLine(datasetOperation.WaitForCompletionResponse().Content);

based on quickstart-create-data-factory-dot-net

@mikeblakeuk mikeblakeuk changed the title [BUG] [BUG] Data Factory is not loading from Fabric Lakehouse with schema Names Mar 4, 2025
@github-actions github-actions bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Mar 4, 2025
@mikeblakeuk mikeblakeuk changed the title [BUG] Data Factory is not loading from Fabric Lakehouse with schema Names [BUG] Data Factory is not loading from Fabric Lakehouse with SchemaTypePropertiesSchema Mar 4, 2025
@jsquire jsquire added Data Factory Service Attention Workflow: This issue is responsible by Azure service team. Mgmt This issue is related to a management-plane library. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team and removed needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. labels Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. Data Factory Mgmt This issue is related to a management-plane library. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

2 participants