Skip to content

feat asyncio dicom export#170

Open
fordmcdonald wants to merge 13 commits into
mainfrom
feat-asyncio-dicom-export
Open

feat asyncio dicom export#170
fordmcdonald wants to merge 13 commits into
mainfrom
feat-asyncio-dicom-export

Conversation

@fordmcdonald

@fordmcdonald fordmcdonald commented Oct 8, 2025

Copy link
Copy Markdown
Member

Speed up XNAT DICOM fetch in dicom_export with asyncio/aiohttp

Summary

This PR parallelizes DICOM downloads from XNAT during the dicom_export phase of xnat2bids by using a single-threaded asyncio event loop with aiohttp. Files are streamed to disk concurrently and then BIDS-ified, reducing wall-clock time on exporting session DICOM scans.

I'm recycling the authentication credentials and session cookie from the prior connection set up via the requests module.

Another point of confusion might be the semaphore created in the run_remaining function. This is passed to the Task method download_and_bidsify to limit the number of concurrent requests to 30 per call to xnat2bids with the reason being to soften the load on XNAT. In previous testing, I found XNAT's throughput bottlenecked around 64 concurrent requests.

@fordmcdonald
fordmcdonald marked this pull request as ready for review October 30, 2025 17:02
@fordmcdonald fordmcdonald self-assigned this Oct 30, 2025

@hollandjg hollandjg left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks great! I've a couple of questions, but it's neat and clean and understandable.

My comments are mostly questions about the dependencies, and whether you need all of the ones added, given they aren't imported by bids_utils.py.

Thanks for this, Ford!

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a file which wasn't used anywhere in the repo. What was it doing here?

Comment thread pyproject.toml
Comment thread pyproject.toml
Comment thread pyproject.toml
Comment thread xnat_tools/bids_utils.py
Comment thread pyproject.toml Outdated

@hollandjg hollandjg left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Sorry for the slow re-review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants