Revise SQL handling in #2172

[asnare]

Changes

This PR revises #2172 so that SQL is handled more safely: in particular we no longer dynamically build SQL queries to run.

Relevant implementation details

DuckDB can bulk insert from Pandas data frames, and with CTAS will properly (and safely) set the table schema. This is preferable to building SQL statements dynamically.

Linked issues

Revises #2172.

[codecov]

Codecov Report

❌ Patch coverage is 60.00000% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.52%. Comparing base (dd357cb) to head (7b67aad).
⚠️ Report is 1 commits behind head on feature/handle_empty_sql_output.

Files with missing lines	Patch %	Lines
...databricks/labs/lakebridge/assessments/pipeline.py	58.82%	7 Missing ⚠️
...ks/labs/lakebridge/connections/database_manager.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@                         Coverage Diff                         @@
##           feature/handle_empty_sql_output    #2203      +/-   ##
===================================================================
- Coverage                            63.52%   63.52%   -0.01%     
===================================================================
  Files                                  100      100              
  Lines                                 8508     8512       +4     
  Branches                               886      886              
===================================================================
+ Hits                                  5405     5407       +2     
- Misses                                2936     2938       +2     
  Partials                               167      167              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 52/52 passed, 5 flaky, 5m32s total

Flaky tests:

• 🤪 test_transpiles_informatica_to_sparksql (23.765s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (26.613s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (21.408s)
• 🤪 test_transpile_teradata_sql (23.825s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (24.559s)

_{Running from acceptance #3217}

[sundarshankar89]

LGTM