Implement GraphOrchestrator Node Retry

[nulinspiratie]

The BasicOrchestrator is an orchestrator that handles the execution of a calibration graph.
When a node encounters an issue with a specific target (e.g., qubit), the available responses are to either proceed by ignoring the issue for that target or to exclude the target from subsequent operations.
We can enhance this by introducing more sophisticated error recovery mechanisms, such as retrying operations, to improve resilience against transient issues.

As an enhancement we propose to introduce the Node Retry functionality:

• Allow the orchestrator to retry a node's execution on failed targets for a configurable number of attempts.
• This would involve adding the following parameter to BasicOrchestrator, such as:
  • retries_on_failure (int): Number of retries on failure, default is zero. (Note: The original PR draft used retry_on_failure (bool) and max_retries (int). We should decide on the final parameter name and type. retries_on_failure as an int seems concise, where 0 means no retries.)
• This will improve the robustness of the calibration process by handling temporary glitches.

As part of the implementation, the BasicOrchestrator.traverse_graph will need to be refactored to accomodate the increased complexity.

• Identify distinct logical blocks within the current traverse_graph (e.g., initialization, fetching next node, executing node, handling results, queueing successors).
• Extract these blocks into private helper methods.
• Ensure the main traverse_graph loop becomes a high-level coordinator of these helper methods.
• The node execution helper method should encapsulate the new retry logic.

Additional context

The goal is to make the BasicOrchestrator more resilient and its core traversal logic easier to understand and extend.
This will be beneficial for more complex calibration routines that might require more intelligent responses to node failures.

How to Test This Feature

1. Setup Development Environment:

• Install the qualibrate package as per its standard documentation and installation instructions.
• Set up the collaborate-core package for development:
  • Create your own fork of the collaborate-core repository.
  • Create a new branch in your fork (e.g., feature/node-retry-orchestrator) where you will implement these changes.
  • Clone the collaborate-core git repository from its source.
  • Ensure your local development environment for collaborate-core is correctly configured to use your forked version (e.g., by installing it in editable mode: pip install -e .).
• Configure qualibrate per the documentation instructions steps 3 & 4
  https://qua-platform.github.io/qualibrate/installation/

2. Implement the Feature:
Implement the node retry functionality and traverse_graph refactoring in your new branch as outlined above.
All changes should be made to basic_orchestrator.py

3. Manual/Integration Testing:

• Add the node and graph below to the calibrations library
• Run the calibration graph from the qualibrate frontend
• Verify in the data viewer that the nodes have been executed in the correct order
• Test edge cases:
  • Graphs with no targets.
  • Nodes that succeed on the first attempt.
  • Nodes that fail consistently even with retries.
  • Interaction with the skip_failed parameter.

Acceptance Criteria:

• The orchestrator can successfully retry a failed node on specific targets according to the retries_on_failure setting. (Note: The original AC mentioned retry_on_failure and max_retries. This should be updated to reflect the chosen parameter name, e.g., retries_on_failure).
• The traverse_graph method is refactored into smaller, logical units.
• Existing functionality (e.g., skip_failed) remains compatible with the new retry mechanism.
• The changes are covered by appropriate tests (initially manual, with automated tests to follow).

Test node & graph files

Basic node
Should be placed in calibrations library with name 01_qubit_random_outcome.py

from qualibrate import QualibrationNode, NodeParameters
import numpy as np
from typing import List


class Parameters(NodeParameters):
    qubits: List[str] = ["q1"]
    success_probability: float = 0.7


node = QualibrationNode(parameters=Parameters())

for qubit in node.parameters.qubits:
    if np.random.rand() < node.parameters.success_probability:
        node.outcomes[qubit] = "successful"
    else:
        node.outcomes[qubit] = "failed"

node.save()

Basic testing graph
Should be saved in same calibrations library

from typing import List, Optional
from qualibrate.orchestration.basic_orchestrator import BasicOrchestrator
from qualibrate import QualibrationGraph, QualibrationLibrary, GraphParameters

library = QualibrationLibrary.get_active_library()


class Parameters(GraphParameters):
    qubits: Optional[List[str]] = ["q1", "q2", "q3", "q4"]
    targets_name: str = "qubits"


g = QualibrationGraph(
    name="test_graph_random_qubit_outcomes",
    parameters=Parameters(),
    nodes={
        "wf1": library.nodes["01_qubit_random_outcome"].copy(),
        "wf2": library.nodes["01_qubit_random_outcome"].copy(),
        "wf3": library.nodes["01_qubit_random_outcome"].copy(),
    },
    connectivity=[("wf1", "wf2"), ("wf1", "wf3")],
    orchestrator=BasicOrchestrator(skip_failed=True),
)

g.run()