Skip to content

Feature/sweep post process#336

Open
natangqm wants to merge 29 commits into
mainfrom
feature/sweep_post_process
Open

Feature/sweep post process#336
natangqm wants to merge 29 commits into
mainfrom
feature/sweep_post_process

Conversation

@natangqm

@natangqm natangqm commented Mar 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds fetch_xarray_data — fetches results from a completed QUA job and returns them as an xarray.Dataset, with each result stream automatically shaped and labelled according to the QUA iterables (sweep) structure. Native iterables are reassembled from their per-index streams, and axes are transposed so the Dataset's dimension ordering matches the original iteration order.

qm-qua compatibility

The feature relies on qm-qua APIs introduced in 1.3.1 (e.g. STREAM_NAME_SEPARATOR, declare_with_stream, the qua_iterators extensions). Rather than raise the package-wide floor and break users on older qm-qua, the requirement is handled gracefully:

  • The qm-qua imports are deferred out of module scope (type hints under TYPE_CHECKING + from __future__ import annotations), so import qualang_tools works on any supported qm-qua.
  • fetch_xarray_data is version-gated: on qm-qua < 1.3.1 it raises a clear ImportError("fetch_xarray_data requires qm-qua>=1.3.1 ...") instead of an opaque import crash.
  • MIN_QM_QUA_VERSION = "1.3.1" is the single source of truth, driving the runtime gate, error message, and test skips.

The qm-qua floor in pyproject.toml stays at >=1.2.3. Because qm-qua 1.3.1 dropped Python 3.9, the lock resolves per-interpreter: 1.3.1 on Python 3.10+, 1.2.6 on Python 3.9 — 3.9 support is preserved.

Changes

  • Add qualang_tools.results.fetch_xarray_data (+ README in qua_iterables_processing/).
  • Deps: xarray promoted from optional extra to a core dependency; packaging added as an explicit core dependency (used for the version comparison).
  • Tests: full coverage of basic/averaging/zip/edge-case behaviour, plus version-guard tests. Feature tests are excluded from collection on qm-qua < 1.3.1 via collect_ignore (their top-level qm-qua 1.3.1 imports can't be collected on older qm-qua, so a pytest.mark.skipif is insufficient).
  • CHANGELOG.md updated.

CI

Green across the matrix: Python 3.10/3.12/3.13 install qm-qua 1.3.1 and run the feature tests; Python 3.9 installs 1.2.6 and cleanly skips them.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Mar 29, 2026

Copy link
Copy Markdown

Unit Test Results

0 tests   0 ✔️  0s ⏱️
0 suites  0 💤
0 files    0

Results for commit 837eaa8.

♻️ This comment has been updated with latest results.

Comment thread qualang_tools/loops/qua_iterable_postprocess.py Outdated
Comment thread qualang_tools/loops/qua_iterable_postprocess.py Outdated
Comment thread qualang_tools/loops/qua_iterable_postprocess.py Outdated
Comment thread qualang_tools/loops/qua_iterable_postprocess.py Outdated
Comment thread qualang_tools/loops/qua_iterable_postprocess.py Outdated
Comment thread qualang_tools/loops/qua_iterable_postprocess.py Outdated
Comment thread tests/tests_qua_utilities/test_fetch_xarray_averaging.py Outdated
Comment thread pyproject.toml Outdated
@natangqm natangqm requested a review from nulinspiratie April 13, 2026 09:13

@nulinspiratie nulinspiratie left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this functionality better belongs in results folder because it is similar to fetching tool, which is located there.
Additionally, there is the XarrayDataFetcher in Qualibration-Libs. I think the behavior of this tool should mimic that one, i.e. be used as a context manager that can also retrieve data while still acquiring. In fact, we should consider merging the functionality of this feature and the XarrayDataFetcher into one tool.

It's fine if the current version doesn't support data fetching while acquiring, but I think we should at least match the UX.

Since XarrayDataFetcher is in QualibrationLibs, we should think together with @OziEgri where the final tool should reside, py-qua-tools or qualibration-libs? I think it makes sense to have one repository for this and related tools

The PR is also missing a README, I'm sending you a draft separately @natangqm

Comment thread tests/tests_qua_utilities/conftest.py
Comment thread qualang_tools/results/sweep_program_processing/qua_iterable_postprocess.py Outdated
Comment thread qualang_tools/results/qua_iterables_processing/qua_iterable_postprocess.py Outdated
Comment thread tests/tests_qua_utilities/test_fetch_xarray_zip.py Outdated
@nulinspiratie

Copy link
Copy Markdown
Contributor

README.md
Here's the proposed readme to accompany it @natangqm

@natangqm natangqm requested a review from nulinspiratie April 16, 2026 09:13
@nulinspiratie

Copy link
Copy Markdown
Contributor

I also propose we ditch the naming "sweep program", it could confuse customers. I propose to instead move the contents to the file qualang_tools.results.xarray_results.py, wdyt @natangqm ?

@natangqm natangqm requested review from yonatanrqm and removed request for yonatanrqm May 13, 2026 05:55

@nulinspiratie nulinspiratie left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but minor docs comments to implement before merging

Comment on lines +103 to +104
wait_until_done: If True, block until the job completes before fetching results.
Defaults to False.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is unclear here what wait_until_done=False actually does. Earlier in the docstring it already says that it retrieves result streams from a completed QUA job, so what happens if it's not complete? Does it acquire whatever is available, or something else?
This could be clarified in the docstring, and also the README.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its clarified in the readme and in the docstring

return stream_data


def fetch_xarray_data(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to consider here is that from a user's perspective they would likely want to keep acquiring until the job is done, then do a final acquisition, and then use that as the final dataset, so something like:

while not job.is_done():  # forgot the exact command here
    ds = fetch_xarray_data(job, iterables)
ds = fetch_xarray_data(job, iterables)  # Acquire one last time

This is a bit clunky, so we should come up with a better solution. For example, in the previous xarray data fetcher, it was done through a for loop, something like:

for ds in fetch_data(job, iterables):  # again, forgot the exact command
    ...
# No need for a final fetch, the `for` loop logic already encapsulates this

I'm not saying this is the right way, but we should have something simple that avoids us having to implement the logic in the first example. We don't need to do it here, but we need a solution in the next phase

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can do that in next version

natangqm and others added 6 commits May 24, 2026 11:58
…m-qua

Version-gate the QUA-iterable post-processing API (added in qm-qua 1.3.1)
via MIN_QM_QUA_VERSION as the single source of truth, raising a clear
ImportError on older qm-qua while keeping qualang_tools importable. Feature
tests skip below the minimum; add packaging as an explicit dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@yomach yomach requested a review from OziEgri June 21, 2026 07:46
@yomach

yomach commented Jun 21, 2026

Copy link
Copy Markdown
Collaborator

@OziEgri Any objections we merge this?

natangqm and others added 2 commits June 21, 2026 11:05
pytestmark runs after module import, so it can't prevent the collection-time
ImportError from the test modules' top-level qm-qua 1.3.1 imports (e.g.
declare_with_stream) on qm-qua 1.2.6 (the Python 3.9 leg). Use collect_ignore
in conftest, gated on qua_iterables_supported(), to avoid importing those
modules entirely when qm-qua is too old.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 1.3.0a1 pre-release lacks STREAM_NAME_SEPARATOR; the API requires 1.3.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread qualang_tools/results/qua_iterables_processing/qua_iterable_postprocess.py Outdated
natangqm and others added 5 commits June 21, 2026 12:17
NativeIterable/NativeIterableRange/NativeIterableBase are deprecated since
qm-qua 1.3.0 (removed in 2.0.0). Use PythonIterable/PythonIterableRange/
PythonIterableBase across the fetch_xarray_data tests and README examples.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@natangqm

Copy link
Copy Markdown
Collaborator Author

@OziEgri please review

@OziEgri OziEgri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, would be good to add Theo or Kevin to the review, as they have more experience here

@natangqm natangqm requested a review from KevinAVR July 1, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants