Maestro platform #2235

isimo00 · 2025-03-18T14:25:55Z

Write this prototype workflow for docs/tests
Try to extend the Slurm platform (create a new class, extend slurm platform, override one function and do something different)
Then start checking what you think will be needed for those that want/need to extend a platform like HPE

codecov-commenter · 2025-03-18T14:32:39Z

Codecov Report

Attention: Patch coverage is 46.32353% with 73 lines in your changes missing coverage. Please review.

Project coverage is 55.30%. Comparing base (53b2a14) to head (92ca4a2).
Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
autosubmit/platforms/slurmplatform.py	49.10%	51 Missing and 6 partials ⚠️
autosubmit/platforms/slurm_example.py	0.00%	12 Missing ⚠️
autosubmit/job/job_list.py	57.14%	3 Missing ⚠️
autosubmit/autosubmit.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2235      +/-   ##
==========================================
+ Coverage   53.90%   55.30%   +1.40%     
==========================================
  Files          72       73       +1     
  Lines       17242    17237       -5     
  Branches     3352     3350       -2     
==========================================
+ Hits         9294     9533     +239     
+ Misses       7103     6866     -237     
+ Partials      845      838       -7

Flag	Coverage Δ
fast-tests	`55.30% <46.32%> (+1.40%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

VindeeR · 2025-03-26T10:54:50Z

@kinow we have added quite a bit of things, skeleton to the MaestroPlatform, we removed some redundant and repetitive explanation on the Documentation and add a new session, albeit mostly a draft for now, of the SlurmPlatform test.

Could you double check what we got so far so we know we are going in the right direction?

kinow · 2025-03-28T11:24:12Z

Probably won't have time to review this one (and it's still a draft). Can you have a look at this one later, please, @dbeltrankyl ? You don't need to worry about the Maestro part. The idea here is to provide documentation with enough information (and examples, if possible) to be used by developers outside of the BSC to create new platforms, focused on Slurm as the idea for Maestro is to extend it. So you'd have to just take a look if you find that useful for external developers, and if there's anything wrong in the docs/code. 🙇

Ini docstring fixing SlurmPlatform docstring Add some descriptions (testing) Add skeleton of Maestro platform class adding docstring (#2236) * adding docstring * import failling CI/CD * docstring slurmplatform * linting + docstring Fix exit code in autosubmit commands that return bool 4.1.12 recovery is slow (#2237) * Changelog * WIP DROPME Use a dev version of config parser * WIP DROPME Use a dev version of config parser * rebse * update asconfigparser version * removed update_parameters call --------- Co-authored-by: Bruno P. Kinoshita <[email protected]> specify python version and update command line 4.1.12 release branch (#2042) * Changelog * WIP DROPME Use a dev version of config parser * WIP DROPME Use a dev version of config parser * rebse * update asconfigparser version * 4.1.12 recovery is slow (#2237) * Changelog * WIP DROPME Use a dev version of config parser * WIP DROPME Use a dev version of config parser * rebse * update asconfigparser version * removed update_parameters call --------- Co-authored-by: Bruno P. Kinoshita <[email protected]> * WIP DROPME Use a dev version of config parser * WIP DROPME Use a dev version of config parser * rebse * changelog added * changelog added --------- Co-authored-by: Bruno P. Kinoshita <[email protected]> Add documentation about workflow commit, fix make watch update eb recipe (#2243) Update setuptools requirement from <=76.0.0 to <=77.0.3 Updates the requirements on [setuptools](https://github.com/pypa/setuptools) to permit the latest version. - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@0.6...v77.0.3) --- updated-dependencies: - dependency-name: setuptools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Update setuptools requirement from <=77.0.3 to <=78.0.2 Updates the requirements on [setuptools](https://github.com/pypa/setuptools) to permit the latest version. - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@0.6...v78.0.2) --- updated-dependencies: - dependency-name: setuptools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Use pytestify to convert unittests to pytest Clean unused code Adding slurm_monitor and strategies tests, deleting unused function in slurm_monitor (see BSC-ES/autosubmit-api#175) Add Docker marker, update CONTRIBUTING docs with information for contributors to run integration+docker tests Update portalocker requirement from <=2.7.0 to <=3.1.1 Updates the requirements on [portalocker](https://github.com/wolph/portalocker) to permit the latest version. - [Release notes](https://github.com/wolph/portalocker/releases) - [Changelog](https://github.com/wolph/portalocker/blob/develop/CHANGELOG.rst) - [Commits](wolph/portalocker@v0.5.6...v3.1.1) --- updated-dependencies: - dependency-name: portalocker dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Add DbManager factory function (#2107) * add DbManager factory function * add extra typings * Update autosubmit/job/job_list_persistence.py Co-authored-by: Luiggi Tenorio <[email protected]> --------- Co-authored-by: Bruno P. Kinoshita <[email protected]> verify args to format message (#2113) Documentation improvement maestro (#2247) * fixing some minor documentation problems and removing verbosity that would cause confusion * fixing sphinx identation problem * Creating a step-by-step of slurm tests + reaorganizing existing documentation * refining test * removing file that shouldnt be there * removing duplicated parameters simplifying description and proportions * adjusting docstring of the function improving quality of documentation bettering wording of instruction and giving hints changes to platform to simulate different behaviours between slurm and maestro Change file name Update job.py revert Update ecplatform.py revert Update paramiko_submitter.py revert removing unused imports fixing instructions Fix sphyinx local build and example_platform name change Small fixes adjusting configure platform draft of a extending platform Add some references extra information where might be needed changes Clean Configure Experiments page small fixes document Move autosubmit create syntax doc Move autosubmit create docs Rewrite some snippets Add permalinks and formatting

dbeltrankyl · 2025-04-01T10:15:21Z

I'll take a look at this today

dbeltrankyl

Hello

Thank you for the work and the improvement in code quality!

I've partially revised it and added some comments; I'll finish it tomorrow. I didn't fully read the RST changes

I have one comment about the example_platform.py,.

Do you plan to rename it to maestro_platform.py at some point? Otherwise I'm not sure if we should maintain the example_platform.py inside the code as it will be unused code. Maybe a better place would be to include it as a snippet in the readthedocs? what do you think?

autosubmit/platforms/example_platform.py

dbeltrankyl · 2025-04-01T12:55:34Z

autosubmit/platforms/slurmplatform.py

        return SlurmPlatform(self.expid, self.name, self.config)

-    def get_submit_cmd_x11(self, args, script_name, job):
+    def get_submit_cmd_x11(self, args: str, script_name: str) -> str:


maestro doesn't need to support this

autosubmit/platforms/slurmplatform.py

dbeltrankyl · 2025-04-01T13:15:32Z

autosubmit/platforms/slurmplatform.py

+            return int(job_id)
+        return None
+
+    def submit_Script(self, hold: bool=False) -> Union[List[int], int]:


The name of this function can be confused with the def submit_job one.

This always submits a script that has a bunch of sbatch ... and each line represents one job submission

While the def submit_job

Sometimes, it submits the job directly ( I don't recall the cases; I think that some conditions are not slurm platform or x11 is enabled)

Usually, the slurm platform sends the submission script to the remote and calls later by the submit_script function.

And in other cases, it submits the script

This has to be remade at some point, so we always submit all jobs in batches. The current "solution" was a quick patch due to the necessity of the moment ( before, we were only submitting one by one, and the submission time was a performance bottleneck)

This method is declared in various platforms besides the slurm one, so I created a separate issue to address this, see #2261

autosubmit/platforms/slurmplatform.py

docs/source/userguide/configure/develop_a_project.rst

dbeltrankyl

Hello!

Thanks for the work!

Continuing the feedback... I still miss some files I'm trying to find time to, apologies for the delay

docs/source/userguide/platforms/index.rst

docs/source/userguide/expids.rst

docs/source/userguide/platforms/index.rst

dbeltrankyl · 2025-04-03T07:52:30Z

Hello!

Thanks for the work!

Continuing the feedback... I still miss some files I'm trying to find time to, apologies for the delay

Oh, I think I don't miss files

One general comment since it is meant for developers:

I miss:

Some explanation of the platform attributes and what they do ( some affect to the header setting, other indicates the scheduler that will be used, some affect the path's etc )
How to obtain the yaml configuration and how to add new platform attributes based on that. ex.

How to read this:

PLATFORMS:
  MARENOSTRUM5:
     maestro_attribute: "something"

And Autosubmit understands that maestro_attribute is a platform attribute, not something the user adds for the templates.

How to trigger your new platform code through the yml configuration.

After the feedback resolve, I'll take another general look and see if we can fit something else

Thanks for the work!

EDIT: Also, I found some typos but I'm not good at finding them. Maybe it is worth revising the doc using Grammarly or another language tool

…mple of the platform.type and an extra explanation over ecplatform

isimo00 · 2025-04-04T10:10:35Z

docs/source/devguide/platforms/index.rst


-.. warning::


Why did you remove this? @VindeeR :(

…QoL improvement

dbeltrankyl · 2025-04-04T11:40:24Z

Let me know when you want me to re-revise the .rst files ( maybe @LuiggiTenorioK want also read it and see if it can be followed well )

LuiggiTenorioK

It seems well-structured @VindeeR @isimo00! Congrats!

I didn't go deep into the correctness of the content, but here are some comments to improve it.

docs/source/devguide/index.rst

docs/source/devguide/platforms/index.rst

docs/source/index.rst

Co-authored-by: Luiggi Tenorio <[email protected]>

VindeeR · 2025-04-04T14:31:08Z

Let me know when you want me to re-revise the .rst files ( maybe @LuiggiTenorioK want also read it and see if it can be followed well )

Aside from the last question I've done I guess it's ready to be reviewed again.

Thanks both @dbeltrankyl @LuiggiTenorioK for the big help reviewing the concepts, structure, and bullet proofing the documentation

isimo00 added this to the 4.1.13 milestone Mar 18, 2025

isimo00 assigned isimo00 and VindeeR Mar 18, 2025

isimo00 marked this pull request as draft March 18, 2025 14:26

kinow requested a review from dbeltrankyl March 28, 2025 11:21

isimo00 force-pushed the create-maestro-platform#2165 branch from 1167908 to b9e4eea Compare March 31, 2025 12:21

Merge branch 'master' into create-maestro-platform#2165

2ac4933

isimo00 marked this pull request as ready for review March 31, 2025 12:31

dbeltrankyl reviewed Apr 1, 2025

View reviewed changes

isimo00 and others added 3 commits April 1, 2025 16:30

Correct misresolved conflict

150138a

fixing pointed out observation

ef0436e

rollback adjust requested

9781792

isimo00 mentioned this pull request Apr 2, 2025

Rename submit_Script method in platforms #2261

Open

Rename example platform

dd14dae

dbeltrankyl reviewed Apr 3, 2025

View reviewed changes

isimo00 and others added 7 commits April 3, 2025 10:16

Fix platforms/index.rst typos

8b75916

Fir more platforms/index.rst typos

8132694

Fix experiment id page suggestions

224b238

removing unused import

53714f5

Create Developers Guide and move platform docs

0c5f509

changes suggested by dani fixing comments and detailing more explanation

0738003

addressing comments pointed out by dani

743f744

VindeeR force-pushed the create-maestro-platform#2165 branch from 597dc81 to 743f744 Compare April 3, 2025 15:11

Expanding explanation over deprecated status over hold variable

8ae3f3d

isimo00 and others added 3 commits April 4, 2025 10:32

Add some suggestions

9b419d4

Chane some parts of platform configuration docs

92f05e3

visually organizing documantation and adding a pointer towards an exe…

fb11bed

…mple of the platform.type and an extra explanation over ecplatform

isimo00 commented Apr 4, 2025

View reviewed changes

docs/source/devguide/platforms/index.rst Outdated

.. warning::

Copy link

Contributor Author

isimo00 Apr 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you remove this? @VindeeR :(

VindeeR added 2 commits April 4, 2025 12:14

adjusting order from which batch files shown to be createed and smal …

7475c11

…QoL improvement

adding relative path to bash file

ed69a02

LuiggiTenorioK reviewed Apr 4, 2025

View reviewed changes

docs/source/devguide/index.rst Outdated Show resolved Hide resolved

docs/source/devguide/platforms/index.rst Outdated Show resolved Hide resolved

docs/source/index.rst Outdated Show resolved Hide resolved

VindeeR and others added 6 commits April 4, 2025 14:15

Update docs/source/devguide/index.rst

07f01e0

Co-authored-by: Luiggi Tenorio <[email protected]>

Update docs/source/devguide/platforms/index.rst

9ca326c

Co-authored-by: Luiggi Tenorio <[email protected]>

applying Luiggi's suggestion on consistency of lists

7d66724

changing from commit id to tag

38c5ada

typo seen by Luiggi

391b58c

refining Dani suggestions

92ca4a2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Maestro platform #2235

Maestro platform #2235

isimo00 commented Mar 18, 2025 •

edited

Loading

codecov-commenter commented Mar 18, 2025 •

edited

Loading

VindeeR commented Mar 26, 2025

kinow commented Mar 28, 2025

dbeltrankyl commented Apr 1, 2025

dbeltrankyl left a comment •

edited

Loading

dbeltrankyl Apr 1, 2025

dbeltrankyl Apr 1, 2025

isimo00 Apr 2, 2025

dbeltrankyl left a comment

dbeltrankyl commented Apr 3, 2025 •

edited

Loading

isimo00 Apr 4, 2025

dbeltrankyl commented Apr 4, 2025 •

edited

Loading

LuiggiTenorioK left a comment

VindeeR commented Apr 4, 2025

Maestro platform #2235

Are you sure you want to change the base?

Maestro platform #2235

Conversation

isimo00 commented Mar 18, 2025 • edited Loading

codecov-commenter commented Mar 18, 2025 • edited Loading

Codecov Report

VindeeR commented Mar 26, 2025

kinow commented Mar 28, 2025

dbeltrankyl commented Apr 1, 2025

dbeltrankyl left a comment • edited Loading

Choose a reason for hiding this comment

dbeltrankyl Apr 1, 2025

Choose a reason for hiding this comment

dbeltrankyl Apr 1, 2025

Choose a reason for hiding this comment

isimo00 Apr 2, 2025

Choose a reason for hiding this comment

dbeltrankyl left a comment

Choose a reason for hiding this comment

dbeltrankyl commented Apr 3, 2025 • edited Loading

isimo00 Apr 4, 2025

Choose a reason for hiding this comment

dbeltrankyl commented Apr 4, 2025 • edited Loading

LuiggiTenorioK left a comment

Choose a reason for hiding this comment

VindeeR commented Apr 4, 2025

isimo00 commented Mar 18, 2025 •

edited

Loading

codecov-commenter commented Mar 18, 2025 •

edited

Loading

dbeltrankyl left a comment •

edited

Loading

dbeltrankyl commented Apr 3, 2025 •

edited

Loading

dbeltrankyl commented Apr 4, 2025 •

edited

Loading