Repository navigation
CLOUDSEC-27 Add script for pre-downloading Spotless dependencies - #127
mstachniuk wants to merge 1 commit into
Conversation
hashicorp-vault-sonar-prod
Bot
changed the title
SummaryAdds a Python utility script to pre-download Spotless formatter dependencies and avoid OkHttp timeouts on CloudFlare WARP networks. Problem: The Spotless Groovy/Java Eclipse formatters download large JARs via OkHttp with a 10-second read timeout. When behind CloudFlare WARP, traffic inspection slows downloads below that threshold, causing builds to fail. Solution: This script pre-populates the Equo P2 bundle pool using urllib (unaffected by OkHttp timeouts). It:
Developers can run it manually before builds to pre-cache all dependencies, or integrate it into CI/build scripts. Works with SonarSource projects like differential-validation and sonar-iac-enterprise. What reviewers should knowScript entry points and flow: Key implementation decisions to review:
Testing angles: Verify that cache invalidation works correctly, that the script handles missing or malformed P2 repositories gracefully, and that the parallelized downloads don't hammer mirrors with too many concurrent requests.
|
|
![sonar-review-alpha[bot]](https://avatars.githubusercontent.com/in/3025063?s=60&v=4){kind=small}
There was a problem hiding this comment.
The script is well-structured and solves a real problem cleanly. One clear bug needs fixing before merge: the mirror parameter accepted by prefill_bundle_pool is silently discarded and never reaches get_p2_artifacts, meaning the mirror configured in project build scripts has zero effect on this script — exactly the opposite of what a developer behind WARP would expect when they've set up a mirror precisely to work around download failures.
| def prefill_bundle_pool(p2_url: str, mirror: Optional[str] = None) -> None: | ||
| """ | ||
| Download all osgi.bundle JARs from a P2 repository into the Equo bundle pool. | ||
|
|
||
| Always uses the canonical (non-mirrored) URL as the bundle pool key. | ||
| Eclipse P2 mirrors (e.g. ftp.fau.de) only mirror metadata files, not JARs; | ||
| Equo falls back to canonical URLs for JAR downloads regardless of mirror config. | ||
| """ | ||
| print(f"\n P2 URL : {p2_url}") | ||
| _prefill_single_pool(p2_url, p2_url) |
There was a problem hiding this comment.
The mirror parameter is accepted but silently discarded — it is never forwarded to _prefill_single_pool or get_p2_artifacts. The docstring says the canonical URL is always used as the bundle pool key (correct), but that is orthogonal to using the mirror for fetching artifacts.xml.xz. If the canonical download.eclipse.org is slow or unreachable behind WARP, the configured mirror is never tried for metadata, defeating part of the script's purpose.
Either pass mirror through (requires adding it to _prefill_single_pool and forwarding to get_p2_artifacts), or remove the parameter from the signature to make the omission explicit.
- Mark as noise
| missing_fast = [ | ||
| (id_, ver) for id_, ver in cached | ||
| if not (dest_dir / f"{id_}_{ver}.jar").exists() | ||
| or (dest_dir / f"{id_}_{ver}.jar").stat().st_size == 0 |
There was a problem hiding this comment.
Logic duplication: This predicate — not jar.exists() or jar.stat().st_size == 0 — is duplicated verbatim at lines 244–246 for the main-path missing list. If the "is a JAR missing?" check ever changes (e.g. adding a checksum comparison), both places must be updated in sync.
Extract a helper:
def _is_jar_missing(dest_dir: Path, id_: str, ver: str) -> bool:
p = dest_dir / f"{id_}_{ver}.jar"
return not p.exists() or p.stat().st_size == 0Then both list comprehensions can call _is_jar_missing(dest_dir, id_, ver).
- Mark as noise
| python3 scripts/spotless-fix-dependencies.py [project_dir] | ||
|
|
||
| If project_dir is omitted, uses the current directory. | ||
| Works with: differential-validation, sonar-iac-enterprise (and other repos that ise those modules). |
There was a problem hiding this comment.
Typo: "repos that ise those modules" → "repos that use those modules".
| Works with: differential-validation, sonar-iac-enterprise (and other repos that ise those modules). | |
| Works with: differential-validation, sonar-iac-enterprise (and other repos that use those modules). |
- Mark as noise
|
Moved the script to another place: https://github.com/SonarSource/languages-experimental-tooling/pull/594 |
1 participant
No description provided.