InternalCompat: Internal Python tests request a Maven coordinate that was never published (HHmm timestamp race) #2653

Summary

The SynapseML-Internal Compatibility Check job fails on essentially every PR with an opaque JAVA_GATEWAY_EXITED. The underlying cause is a timestamp race in the Internal Maven coordinate: the version string embeds a live HHmm value that is recomputed on each invocation, so the Run Internal Python tests (ExcludeAIFunc) step requests an artifact that was never published.

The job is already continueOnError: true and is not a required status check, so this does not block merges — but it produces a permanently red check run, which trains reviewers to ignore a signal that is supposed to catch real OSS→Internal breakage.

Evidence

Two builds from different PRs, different Scala/Python targets, same failure shape:

Same git SHA, same python tag, same base version — only the HHmm component differs.

Package Internal Python against OSS succeeds and logs the publish:

[info] published ...-internal_2.13 to file:/home/vsts/.m2/repository/com/microsoft/azure/...-internal_2.13/1.1.3.0-python3.13-5-5474bb77-20260817-1132-SNAPSHOT/...

Run Internal Python tests (ExcludeAIFunc) then dies with:

Exception in thread "main" java.lang.RuntimeException: [unresolved dependency:
  com.microsoft.azure#...-internal_2.13;1.1.3.0-python3.13-5-5474bb77-20260817-1136-SNAPSHOT: not found]
	at org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:539)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:340)

Because spark-submit exits before writing its port, PySpark surfaces only:

pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED]
  Java gateway process exited before sending its port number

which is why every test in the step reports as an ERROR at fixture setup rather than pointing at the real cause.

Why the existing mitigation is insufficient

pipeline.yaml already documents this failure mode and mitigates half of it:

# packagePython and publishM2 MUST share one sbt session. Internal's version embeds an HHmm timestamp, so separate invocations can differ by a minute: packagePython bakes the jar coordinate into the generated Python package, publishM2 then publishes a *different* version, and the Spark launch inside make_mlflow_models.py fails to resolve it - surfacing only as an opaque JAVA_GATEWAY_EXITED. Observed: packaged 1022, published 1023.

That fix works — in both builds above, packagePython and publishM2 agree (1132 / 1240). The remaining gap is the next step: make_mlflow_models.py runs as a separate process 3–4 minutes later and recomputes the timestamp. Since the publish→test gap is reliably longer than a minute, this now fails almost deterministically rather than intermittently.

Suggested fix

The durable fix belongs in SynapseML-Internal: the Python-side coordinate should read the version baked in at packagePython time rather than recomputing it from the wall clock. Any of these would work:

1. Have the generated Python package expose the exact coordinate written during packagePython, and have init_spark / session_utils read that constant.
2. Compute the timestamped version once per build and thread it through as an environment variable that both sbt and the Python fixtures honour.
3. Drop HHmm from the coordinate entirely for CI builds — the git SHA already makes it unique.

Option 2 could be driven from the OSS pipeline.yaml if Internal accepts a version override; options 1 and 3 are purely Internal-side.

Impact / non-impact

• Not caused by any particular PR — reproduced on two unrelated PRs, and pipeline.yaml cites build 230974877 hitting it on a PR that changed only pipeline.yaml.
• Internal Scala tests are unaffected: they resolve through sbt's classpath, never through Maven, and pass in the same builds where the Python step fails.
• The check is advisory (continueOnError: true) and is not in the required status checks, so merges are not blocked.

Repro

Any PR build of definition 17563. Compare the version in the published ...-internal_... line of Package Internal Python against OSS against the version in the unresolved dependency line of Run Internal Python tests (ExcludeAIFunc).