[improve][test] Azure Data Explorer E2E test is always skipped in CI — add a runnable integration test

[david-streamlio]

Current state

azure-data-explorer has ADXSinkE2ETest, but it throws SkipException unless live Azure credentials (kustoCluster, kustoDatabase, kustoAadAppId, ...) are present in the environment — so CI never exercises the sink. Effective integration coverage is zero.

Proposal

Use the Microsoft Kusto emulator container image (mcr.microsoft.com/azuredataexplorer/kustainer-linux) via Testcontainers so the sink can be exercised hermetically in CI, keeping the credential-gated E2E test for live-cluster verification.

Part of an integration-test coverage sweep of all connector modules.

[david-streamlio]

Investigated the Kusto emulator approach end-to-end (ran mcr.microsoft.com/azuredataexplorer/kustainer-linux locally, wrote the test, ran the real sink against it). The emulator is not viable for this connector — documenting so no one repeats the effort.

What works against kustainer: plain HTTP unauthenticated (auth is a non-issue — /v1/rest/auth/metadata returns AzureAD: null, so the SDK skips AAD), control commands (.create table, .ingest inline), and queries (Table | count).

What doesn't — and it's exactly what the sink uses. ADXSink only ever calls ingestClient.ingestFromStream(...) (ADXSink.java:148). Verified. That routes through one of two SDK clients, and the emulator services neither:

• Queued ingest (default): the SDK's ResourceManager runs .get ingestion resources, which the emulator rejects — AdminCommandWrongEndpointException: Cannot get ingestion resources from this service endpoint. The emulator is a single engine node with no Data-Management endpoint, blob storage, or queues, which queued ingestion requires.
• Managed streaming ingest: returns BadRequest_StreamingIngestionPolicyNotEnabled; after enabling the streamingingestion policy at both DB and table level (confirmed IsEnabled: true), it then returns BadRequest_EntityNotFound. The emulator advertises the policy but doesn't implement the streaming data node.

End-to-end result: sink wrote 10 records, Table | count stayed at 0, background flush threw the SDK errors above. The only way data enters kustainer is via .ingest inline control commands, which IngestClient never issues — so a passing test would have to insert data itself and assert on it, exercising none of the sink's real code. That is vacuous, so nothing was shipped.

Recommendation. The emulator route (option in the issue title) is a dead end — a known Microsoft limitation. Realistic alternatives:

1. Keep ADXSinkE2ETest as the real coverage but gate it on CI repository secrets (run when creds are present, skip-with-notice otherwise) instead of the current silent SkipException, so the coverage is at least exercised on scheduled/secret-backed runs.
2. Add a mock-based unit test (mock IngestClient) asserting ADXSink batches, acks, and retries correctly — no emulator, and it covers the connector's own logic.
3. (Larger) refactor ADXSink to optionally ingest via .ingest inline control commands, which would be emulator-testable but changes production behavior — needs maintainer buy-in.

Suggest retitling this issue away from 'add a runnable emulator test' toward option 1+2.