Implement databricks creds manager

[m-abulazm]

Changes

What does this PR do?

• Support databricks credentials
• Standardize usages of credentials manager

Linked issues

Progresses #1008

Functionality

• added relevant user documentation
• modified existing command: databricks labs lakebridge ...

Tests

• manually tested
• added unit tests
• added integration tests

[codecov]

Codecov Report

❌ Patch coverage is 79.48718% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.30%. Comparing base (34d089d) to head (cf4fe77).

Files with missing lines	Patch %	Lines
.../labs/lakebridge/connections/credential_manager.py	79.48%	8 Missing ⚠️
src/databricks/labs/lakebridge/reconcile/utils.py	0.00%	6 Missing ⚠️
.../labs/lakebridge/reconcile/connectors/snowflake.py	84.61%	2 Missing and 2 partials ⚠️
...s/assessments/synapse/dedicated_sqlpool_extract.py	0.00%	3 Missing ⚠️
.../assessments/synapse/monitoring_metrics_extract.py	0.00%	3 Missing ⚠️
.../assessments/synapse/serverless_sqlpool_extract.py	0.00%	3 Missing ⚠️
...resources/assessments/synapse/workspace_extract.py	0.00%	3 Missing ⚠️
...abs/lakebridge/assessments/configure_assessment.py	33.33%	2 Missing ⚠️
...abs/lakebridge/reconcile/connectors/credentials.py	95.12%	1 Missing and 1 partial ⚠️
...abs/lakebridge/reconcile/connectors/data_source.py	75.00%	2 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2123      +/-   ##
==========================================
+ Coverage   64.11%   64.30%   +0.19%     
==========================================
  Files         101      100       -1     
  Lines        8675     8747      +72     
  Branches      902      918      +16     
==========================================
+ Hits         5562     5625      +63     
- Misses       2941     2947       +6     
- Partials      172      175       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 51/51 passed, 6 flaky, 3m43s total

Flaky tests:

• 🤪 test_transpiles_informatica_to_sparksql (21.685s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (19.137s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (20.3s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (4.311s)
• 🤪 test_transpile_teradata_sql (6.741s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (8.061s)

_{Running from acceptance #3273}

[m-abulazm]

@asnare @sundarshankar89

The goal of the dict is to contain the connection properties, for example:

{
  "user": "scope/key",
  "password": "scope/key2",
  "host": "scope3/key4",
  "port": "scope4/key5",
  "database": "scope/key3",
  "encrypt": "scope/key",
  "trustServerCertificate": "scope/key"
}

i.e. mapping logical connection options to (scope, key) references so that, at runtime, we only read those specific secrets via the vault, not entire scopes. The use of the __secret_scope key is just to support older configs that only specified a single scope and didn’t use this more dynamic mapping.

In other words, my intent in this PR was always to model the connection properties / OPTIONS(...) side (N) and how they reference secrets, not to introduce a general vault configuration model. The stacked PRs already use this dict in the connection resolution path; the vault backend shape (M) can still be tightened separately if we want.