Skip to content

Switch UD reporting to 5-group test matrix, fix concurrency + verification #4329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
sfc-gh-fpawlowski wants to merge 1 commit into
base: SNOW-2912540-write-pandas-raw-msg
Choose a base branch
from
Draft

Switch UD reporting to 5-group test matrix, fix concurrency + verification #4329

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
375 changes: 375 additions & 0 deletions .github/workflows/ud-inline-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,375 @@
name: UD compatibility tests (Snowpark on Universal Driver)

# Runs the Snowpark test suites with the Universal Driver (UD) swapped in for the
# legacy snowflake-connector-python, to surface which tests fail on UD.
#
# Scope is split into per-area matrix groups (unit-integ, scala, modin, datasource,
# doctest). At-risk groups run with continue-on-error so every gap is visible in a
# single run. UD is built once (build-ud job) from a pinned ref and consumed as a
# prebuilt wheel via the ud_connector_path swap in scripts/tox_install_cmd.sh.

on:
workflow_dispatch:
inputs:
ud-ref:
description: 'universal-driver ref to build (commit SHA preferred for reproducibility; branch allowed)'
required: false
default: 'snowpark-compatibility'
type: string
python-version:
description: 'Python version'
required: false
default: '3.13'
type: choice
options:
- '3.10'
- '3.11'
- '3.12'
- '3.13'
cloud-provider:
description: 'Cloud provider'
required: false
default: 'aws'
type: choice
options:
- 'aws'
- 'azure'
- 'gcp'
pytest-addopts:
description: 'Extra pytest args, e.g. "-k test_dataframe" or "--maxfail=5"'
required: false
default: ''
type: string
schedule:
# Nightly, off-peak, off the :00 mark.
- cron: '23 7 * * *'
# For now, run on every PR for visibility (jobs are continue-on-error, so they
# never block merges). `labeled` is included so adding a label also triggers a
# run. NOTE: PRs from forks get no secrets, so build-ud / decrypt will fail
# there — revisit with label-gating once UD is closer to green.
pull_request:
types: [opened, synchronize, reopened, labeled]

permissions:
contents: read

# No concurrency group here (unlike the source branch): odbc-reports dispatches this
# same ref repeatedly, in parallel, with different ud-ref values per historical
# snapshot date. A group keyed on github.ref alone would cancel one in-flight
# backfill run whenever another started. Confirmed none of odbc-reports' other UD-
# dispatch sibling jobs (snowflake-cli, snowflake-sqlalchemy, dbt-adapters, airflow)
# use a concurrency block either.

env:
PYTHON_VERSION: ${{ inputs.python-version || '3.13' }}
CLOUD_PROVIDER: ${{ inputs.cloud-provider || 'aws' }}
UD_REF: ${{ inputs.ud-ref || 'snowpark-compatibility' }}
EXTRA_PYTEST_ADDOPTS: ${{ inputs.pytest-addopts || '' }}

jobs:
# ---------------------------------------------------------------------------
# Build the UD Python wheel once, from a pinned ref. The wheel is cpXY/platform
# tagged (Rust core compiled via hatch_build.py). Cached by resolved UD commit
# so repeat runs at the same ref don't rebuild.
# ---------------------------------------------------------------------------
build-ud:
name: "Build UD wheel py${{ inputs.python-version || '3.13' }}"
runs-on: ubuntu-latest-64-cores
steps:
- name: Checkout universal-driver
uses: actions/checkout@v4
with:
# snowflakedb/universal-driver is a PUBLIC mirror and has snowpark-compatibility.
# Do NOT pass a custom token: an empty/invalid SNOWFLAKE_GITHUB_TOKEN sets a bad
# Authorization header and GitHub 401s instead of serving the public repo. Omitting
# `token:` lets checkout use the default GITHUB_TOKEN, which reads public repos fine.
repository: snowflakedb/universal-driver
ref: ${{ env.UD_REF }}
persist-credentials: false
fetch-depth: 1

- name: Resolve UD commit
id: udsha
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}

- uses: astral-sh/setup-uv@v6

- name: Ensure Rust toolchain
run: rustup toolchain install stable --profile minimal && rustup default stable

- name: Install protoc
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler

- name: Cache UD wheel
id: udcache
uses: actions/cache@v4
with:
path: python/dist
key: ud-wheel-${{ steps.udsha.outputs.sha }}-py${{ env.PYTHON_VERSION }}-${{ runner.os }}

- name: Build UD wheel
if: steps.udcache.outputs.cache-hit != 'true'
run: |
uv tool install hatch
cd python
hatch build -t wheel
ls -al dist

- name: Upload UD wheel
uses: actions/upload-artifact@v4
with:
name: ud-wheel-py${{ env.PYTHON_VERSION }}
path: python/dist/*.whl
if-no-files-found: error

# ---------------------------------------------------------------------------
# Run each Snowpark test area against the UD wheel. continue-on-error keeps the
# whole matrix running so every gap is visible; promote a group to blocking
# once it is green.
# ---------------------------------------------------------------------------
test:
needs: build-ud
strategy:
fail-fast: false
matrix:
group: [unit-integ, scala, modin, datasource, doctest]
name: "${{ matrix.group }} py${{ inputs.python-version || '3.13' }}-${{ inputs.cloud-provider || 'aws' }}"
runs-on: ubuntu-latest-64-cores
continue-on-error: true

steps:
- uses: actions/checkout@v4
with:
persist-credentials: false

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}

- run: python -c "import sys; print(sys.version)"

- uses: astral-sh/setup-uv@v6

- name: Download UD wheel
uses: actions/download-artifact@v4
with:
name: ud-wheel-py${{ env.PYTHON_VERSION }}
path: ud-wheel

- name: Locate UD wheel
id: udwheel
run: |
whl=$(ls "$PWD"/ud-wheel/*.whl | head -1)
test -n "$whl"
echo "path=$whl" >> "$GITHUB_OUTPUT"
echo "UD wheel: $whl"

- name: Decrypt parameters.py
run: .github/scripts/decrypt_parameters.sh
env:
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
CLOUD_PROVIDER: ${{ env.CLOUD_PROVIDER }}

- name: Install protoc
run: .github/scripts/install_protoc.sh

- name: Install tox
run: uv pip install tox --system

- name: Install MS ODBC Driver
if: matrix.group == 'datasource'
run: |
curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list \
| sudo tee /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 unixodbc-dev

- name: Run tests
id: tests
run: |
set +e
mkdir -p reports
V="${PYTHON_VERSION//.}"
case "${{ matrix.group }}" in
unit-integ)
TOX="py${V}-notdoctest-ci"
set -- --ignore=tests/integ/scala --ignore=tests/integ/datasource ;;
scala)
TOX="py${V}-notdoctest-ci"
set -- tests/integ/scala ;;
modin)
TOX="py${V}-snowparkpandasnotdoctest-modin-ci"
set -- ;;
datasource)
TOX="datasource"
set -- ;;
doctest)
TOX="py${V}-doctest-notudf-ci"
set -- ;;
esac
echo "tox env: $TOX posargs: $*"

python -m tox -e "$TOX" --notest

echo "=== ud-job-testing: force-reinstalling UD directly (bypasses tox_install_cmd.sh env-passing entirely) ==="
if [ -n "${ud_connector_path}" ]; then
.tox/"$TOX"/bin/pip install --force-reinstall --no-deps "${ud_connector_path}"
echo "reinstall exit code: $?"
elif [ -n "${snowflake_path}" ]; then
.tox/"$TOX"/bin/pip install --force-reinstall --no-deps ${snowflake_path}/snowflake_connector_python*.whl

@graphite-app graphite-app Bot Aug 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing quotes around ${snowflake_path} variable. This will cause word splitting and glob expansion if the path contains spaces or special characters.

# Current (broken):
.tox/"$TOX"/bin/pip install --force-reinstall --no-deps ${snowflake_path}/snowflake_connector_python*.whl

# Fixed:
.tox/"$TOX"/bin/pip install --force-reinstall --no-deps "${snowflake_path}"/snowflake_connector_python*.whl

This is inconsistent with line 221 which correctly quotes "${ud_connector_path}".

Suggested change
.tox/"$TOX"/bin/pip install --force-reinstall --no-deps ${snowflake_path}/snowflake_connector_python*.whl
.tox/"$TOX"/bin/pip install --force-reinstall --no-deps "${snowflake_path}"/snowflake_connector_python*.whl

Spotted by Graphite

Fix in Graphite

Is this helpful? React 👍 or 👎 to let us know.

echo "reinstall exit code: $?"
else
echo "neither ud_connector_path nor snowflake_path set -- nothing to reinstall"
fi

cat > /tmp/verify_ud_snippet.py <<'PYEOF'
#!/usr/bin/env python3
"""Standalone UD-connector verification, run inside a tox venv's python via
`<venv>/bin/python verify_ud_snippet.py`. Exit 0 = UD genuinely active, exit 1 = not
(legacy connector, whether by name collision with the pinned dependency range, or
never installed at all). Mirrors run-ud-tests.sh's verify_ud_loaded(): only UD ships
a `snowflake.connector._internal` subpackage and a compiled .so anywhere under
`_core/`, regardless of which name/version the wheel happens to use -- so this check
is independent of package-naming quirks (the "-ud" suffix vs plain name vs version
string), which is exactly what silently defeated a naive `pip freeze` check earlier.
"""
import glob
import importlib.util
import pathlib
import sys

internal_spec = importlib.util.find_spec("snowflake.connector._internal")

try:
import snowflake.connector as _c
pkg_dir = pathlib.Path(_c.__file__).resolve().parent
except Exception as e:
print(f"UD_VERIFY: FAIL -- could not import snowflake.connector at all: {e}")
sys.exit(1)

core_so = glob.glob(str(pkg_dir / "_core" / "*.so"))
ud_active = bool(internal_spec) and bool(core_so)

print(f"UD_VERIFY: loaded={pkg_dir} _internal={bool(internal_spec)} core_so={core_so}")
if ud_active:
print("UD_VERIFY: PASS -- Universal Driver active")
sys.exit(0)
else:
print("UD_VERIFY: FAIL -- legacy (pure-Python) connector active, UD not found")
sys.exit(1)
PYEOF
.tox/"$TOX"/bin/python /tmp/verify_ud_snippet.py
VERIFY_RC=$?
if [ "$VERIFY_RC" -ne 0 ]; then
echo "UD_VERIFICATION_FAILED -- legacy connector active, not testing UD. See job log for UD_VERIFY details." | tee reports/test-output.log
exit 1
fi

# --skip-env-install: reuse the just-hardened venv. A plain second
# `tox -e "$TOX" -- "$@"` would reconcile deps again and silently
# reinstall legacy, undoing the reinstall above.
python -m tox -e "$TOX" --skip-env-install -- "$@" 2>&1 | tee reports/test-output.log
env:
cloud_provider: ${{ env.CLOUD_PROVIDER }}
# Consume the prebuilt UD wheel via the safe swap path (deps installed
# first, then UD force-reinstalled last so it wins regardless of the
# connector version range).
ud_connector_path: ${{ steps.udwheel.outputs.path }}
# Honest counts for compatibility measurement: no reruns.
UD_RERUN_FLAGS: ''
# Write JUnit XML to a known dir for artifact upload / triage.
JUNIT_REPORT_DIR: ${{ github.workspace }}/reports
PYTEST_ADDOPTS: --color=yes --tb=short ${{ env.EXTRA_PYTEST_ADDOPTS }}
TOX_PARALLEL_NO_SPINNER: 1

- name: Extract results
if: always()
run: |
# Human-readable per-group counts. The pytest terminal summary line is
# left intact in the job log for odbc-reports log-scraping.
grep -oP '\d+ (failed|passed|skipped|errors?|warnings?)' reports/test-output.log \
> "reports/summary-${{ matrix.group }}.txt" || true
echo "== ${{ matrix.group }} ==" && cat "reports/summary-${{ matrix.group }}.txt" || true
# Failing/errored test IDs from JUnit XML (triage material).
python3 - "${{ matrix.group }}" <<'PY' || true
import glob, sys, xml.etree.ElementTree as ET
group = sys.argv[1]
out = []
for p in glob.glob("reports/junit*.xml"):
try:
root = ET.parse(p).getroot()
except Exception:
continue
for tc in root.iter("testcase"):
if any(tc.iter("failure")) or any(tc.iter("error")):
out.append(f"{tc.get('classname','')}::{tc.get('name','')}")
with open(f"reports/failing-{group}.txt", "w") as f:
f.write("\n".join(sorted(set(out))))
if out:
f.write("\n")
print(f"failing/errored testcases: {len(set(out))}")
PY

- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: results-${{ matrix.group }}
path: reports/
if-no-files-found: warn

# ---------------------------------------------------------------------------
# Aggregate per-group counts into the GitHub step summary (UI only; odbc-reports
# derives its own totals from the job logs). Fails the run if any group had
# failures/errors, so humans see red until the suite is green on UD.
# ---------------------------------------------------------------------------
summary:
if: always()
needs: test
runs-on: ubuntu-latest
steps:
- name: Download all results
uses: actions/download-artifact@v4
with:
pattern: results-*

- name: Generate summary
run: |
python3 - <<'SCRIPT'
import glob, re, collections, os

per_group = {}
totals = collections.Counter()
for path in sorted(glob.glob("results-*/summary-*.txt")):
group = re.search(r"summary-(.+)\.txt$", path).group(1)
g = collections.Counter()
with open(path) as f:
for line in f:
m = re.match(r"(\d+)\s+(\w+)", line.strip())
if m:
g[m.group(2)] += int(m.group(1))
per_group[group] = g
for k, v in g.items():
totals[k] += v

def fmt(c):
order = ["failed", "errors", "passed", "skipped", "warnings"]
return ", ".join(f"{c[k]} {k}" for k in order if c.get(k, 0)) or "no results"

lines = ["## UD compatibility test results", "", "| group | result |", "|---|---|"]
for group in sorted(per_group):
lines.append(f"| {group} | `{fmt(per_group[group])}` |")
lines += ["", f"**Total:** `{fmt(totals)}`"]
summary = "\n".join(lines)
print(summary)
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
f.write(summary + "\n")

if totals.get("failed", 0) or totals.get("errors", 0):
raise SystemExit(1)
SCRIPT
Loading
Loading