Skip to content

Restructure the pipeline, optimise logs and performance #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 46 commits into
base: clean-up
Choose a base branch
from
Open

Restructure the pipeline, optimise logs and performance #121

wants to merge 46 commits into from

Conversation

marinak-ebi
Copy link
Contributor

@marinak-ebi marinak-ebi commented Jun 27, 2025 •
edited
Loading

Improvements

Optimisation

  • Logs compression changed from console command to sbatch job
  • Added deterministic shuf to make size of each statpacket evenly distributed
  • Reduce number of statspackets by increasing records per file
  • Closes Consider using SSD instead of HDD #57

Performance Notes

  • Statistical pipeline execution time:
    • HDD: 3 days 12 hours
    • SSD: 3 days 5 hours (annotation pipeline failed due to timeout)
  • SSD offers faster read/write operations but slower job submission.
  • Decision: Continue using HDD for now.

marinak-ebi added 30 commits June 26, 2025 13:26
@marinak-ebi
Increase number of rows per file and add shuf
1c2a01f
@marinak-ebi
Create folder for input_parquet_files
c5f5b01
@marinak-ebi
Flatten job folder structure
a4b3c5f
@marinak-ebi
Rename AnnotationExtractorAndHadoopLoader to annotation_extractor
869801d
@marinak-ebi
Flatten annotation_extractor
5af5109
@marinak-ebi
Rename tmp to annotation_pipeline_output
b292c7c
@marinak-ebi
Create folder input_parquet_files
4988caa
@marinak-ebi
Fix paths
808b0bf
@marinak-ebi
Temporally turn off ipdateImpress function
24bb576
@marinak-ebi
Fix path for function_windowed
a9ca8a6
@marinak-ebi
Add additional logs
1613680
@marinak-ebi
Fix path to function_windowed.R
29207ff
@marinak-ebi
Fix mp_chooser path
3e1acfd
@marinak-ebi
Change path to annotation_extractor
59e4b3b
@marinak-ebi
Fix path for indeces
6774f5d
@marinak-ebi
Fix path to mp_chooser
36759c4
@marinak-ebi
Change path of step 2 logs
5ced419
@marinak-ebi
Fix path
30188dd
@marinak-ebi
Compress files in step 2
fed0a03
@marinak-ebi
Change path of step 4 logs
5277621
@marinak-ebi
Add mindepth to fix minor error
8c7d2b8
@marinak-ebi
Fix typo
24dd1bc
@marinak-ebi
Change path of phase II logs
1471b79
@marinak-ebi
Rename slurm jobs
69f2230
@marinak-ebi
Change compression of phase 3 logs
233bc71
@marinak-ebi
Remove intermediate logs
c6769ed
@marinak-ebi
Edit path of annotation pipeline logs
9db9921
@marinak-ebi
Create slurm job for annotation pipeline logs compression
d125559
@marinak-ebi
Change path to annotation_pipeline_output
ab336e7
@marinak-ebi
Fix Typo
e8e17bd
@marinak-ebi marinak-ebi self-assigned this Jun 27, 2025
@marinak-ebi marinak-ebi added bug Something isn't working refactor Restructure code, but not its functionality labels Jun 27, 2025
@marinak-ebi marinak-ebi changed the title Add shuf Restructure the pipeline, optimise logs and performance Aug 12, 2025
@marinak-ebi marinak-ebi requested a review from ficolo August 12, 2025 12:49
This was linked to issues Aug 12, 2025
Fix argument list too long error in orchestration.sh #104
Open
Consider using SSD instead of HDD #57
Open
@marinak-ebi marinak-ebi marked this pull request as ready for review September 5, 2025 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ficolo ficolo Awaiting requested review from ficolo

Assignees

@marinak-ebi marinak-ebi

Labels
bug Something isn't working refactor Restructure code, but not its functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Fix argument list too long error in orchestration.sh Consider using SSD instead of HDD
1 participant
@marinak-ebi