Skip to content

[DOP-26758] Include .jar files into worker image#245

Merged
dolfinus merged 1 commit into
developfrom
improvement/DOP-26758
Jul 9, 2025
Merged

[DOP-26758] Include .jar files into worker image#245
dolfinus merged 1 commit into
developfrom
improvement/DOP-26758

Conversation

@dolfinus
Copy link
Copy Markdown
Member

@dolfinus dolfinus commented Jul 7, 2025

Change Summary

Added script which creates empty Spark session with all Maven packages included. Spark download these images into /root/.ivy2 folder, so they can be reused later. This avoids downloading packages from Maven during transfer runs.

Image size increased 933Mb -> 1.67Gb.
This is done only during release image build, because this step is slow, and Github Actions cache size for test images is limited.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus requested a review from TiGrib July 7, 2025 14:21
@dolfinus dolfinus added the ci:skip-changelog Add this label to skip changelog file check label Jul 7, 2025
@dolfinus dolfinus force-pushed the improvement/DOP-26758 branch from b925c0f to 1d57052 Compare July 7, 2025 14:32
@codecov
Copy link
Copy Markdown

codecov Bot commented Jul 7, 2025

Codecov Report

Attention: Patch coverage is 95.45455% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.32%. Comparing base (acc102c) to head (3f5caa6).

Files with missing lines Patch % Lines
syncmaster/worker/spark.py 95.45% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #245      +/-   ##
===========================================
- Coverage    92.35%   92.32%   -0.03%     
===========================================
  Files          192      192              
  Lines         4695     4693       -2     
  Branches       352      351       -1     
===========================================
- Hits          4336     4333       -3     
  Misses         274      274              
- Partials        85       86       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dolfinus dolfinus force-pushed the improvement/DOP-26758 branch from 1d57052 to e0225b8 Compare July 7, 2025 14:54
@dolfinus dolfinus force-pushed the improvement/DOP-26758 branch from e0225b8 to 3f5caa6 Compare July 7, 2025 15:09
@dolfinus dolfinus marked this pull request as ready for review July 7, 2025 15:23
@dolfinus dolfinus force-pushed the improvement/DOP-26758 branch from 3f5caa6 to 17d2c8d Compare July 9, 2025 14:16
@dolfinus dolfinus merged commit ecda51f into develop Jul 9, 2025
7 of 22 checks passed
@dolfinus dolfinus deleted the improvement/DOP-26758 branch July 9, 2025 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:skip-changelog Add this label to skip changelog file check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants