Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rationalize invalid resource specification errors #3589

Closed
benclifford opened this issue Aug 16, 2024 · 2 comments · Fixed by #3627
Closed

rationalize invalid resource specification errors #3589

benclifford opened this issue Aug 16, 2024 · 2 comments · Fixed by #3627

Comments

@benclifford
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
HTEX's mpi_prefix_composer defines an InvalidResourceSpecification. This is used by the non-MPI parts of htex too. WorkQueue raises an ExecutorError for the same kind of error. I have not investigated the other executors.

Describe the solution you'd like

This should be rationalised: probably InvalidResourceSpecification should become an error in parsl.executors.error and all executors that validate resource specifications should use it.

Additional context

This came from review of PR #3582

@benclifford
Copy link
Collaborator Author

might be interesting to consider and document the difference in meaning vs MissingResourceSpecification, also in mpi_prefix_composer (and maybe merge?)

@Harichandra-Prasath
Copy link
Contributor

Harichandra-Prasath commented Sep 14, 2024

/There are three places where the executors raise the same or similar error.

  1. when passing the resource_specification parameter to the htex, it raises the InvalidResourceSpecification from validate_resource_spec method. https://github.com/Parsl/parsl/blob/master/parsl/executors/high_throughput/executor.py#L342

  2. ThreadPoolExecutor also raises the UnsupportedFeatureError when passing resource specification to it, but the raised error is in parsl.executors.error. https://github.com/Parsl/parsl/blob/master/parsl/executors/threads.py#L54

  3. WorkQueueExecutor raises two errors similar to this but as a Base ExecutorError, one when there is invalid resource_specification ; https://github.com/Parsl/parsl/blob/master/parsl/executors/workqueue/executor.py#L418 and another one when the autolabel is false and resource_specification is not right, https://github.com/Parsl/parsl/blob/master/parsl/executors/workqueue/executor.py#L429

From the above,

1. As ThreadPoolExecutor raises the UnsupportedFeatureError from the parsl.executors.error, ideally htex should also raise the same instead of raising InvalidResourceSpecification.

  1. If we look at 3, the reason for the second error to be raised is the expected fields in resource_specification are missing. Here, the behaviour is similar to both the MissingResourceSpecification and InvalidResouceSpecification.

What we can do,

  1. We can remove the MissingResourceSpecification and InvalidResourceSpecification from mpi_prefix_compose and merge them as ResourceSpecificationError in the parsl.executors.error. As MissingResourceSpecification is raised only when an empty resource_specification is supplied.

2. htex and threads can raise the UnsupportedFeatureError as that is more reasonable than ResourceSpecificationError.

  1. WorkQueueExecutor and mpi_prefix_compose can raise the ResourceSpecificationError with appropriate Messages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment