Rough code adding some support for manifest

[Shotgunosine]

No description provided.

[yarikoptic]

I am pushing more changes to #13 to get that one usable. I guess conflicts will pile up :-/

But I am not clear from the description of this PR what manifests we are talking about?

[Shotgunosine]

Sorry for the lack of clarity. Manifests are the NDA's way of specifying an image file and some associated non-image files along with a directory structure in which to place them. In other words, manifests are going to be the best way to represent bids data structures in NDA datasets. There is not a lot of overhead to dealing with manifests, but there is some. Would it be better for me to submit this as a PR to mtnhuck:master or just wait for that PR to be closed, rebase my branch, and resubmit?

[yarikoptic]

What is there beast official source to read about them more?
You will just need to merge that branch I guess... Hopefully some feedback will come soon and we improve/merge it

[Shotgunosine]

As far as I know, this is the best documentation for the Manifest feature: https://docs.google.com/document/d/1Qo3etvKgDx1lQH37lbKik5IPuLaif9-LHEaUVIBbiz4/edit#heading=h.wng6mfab9jp2
@ushnaahmad is there more documentation on the feature somewhere else?

[obenshaindw]

@yarikoptic and @Shotgunosine this was early documentation for piloting the feature while it was in development. We have some detail on the NDA site, which links out to https://github.com/NDAR/manifest-data. The GitHub repo is the most recent documentation and provides some helper scripts for creating manifests. @ushnaahmad has left NDA for a new opportunity with with GeneDx. You may also be interested in feedback from @ericearl as he is about to submit all ABCD Study imaging data in BIDS format sing NDA manifests.

[yarikoptic]

@Shotgunosine would you be interested to come back to this PR to facelift it and finalize? I can promise to be active too. ----experiment-id was also proposed in another PR #37 by @ljchang : so it is required only for functional and not anatomicals/dwi? (in #37 it is mandatory)

[Shotgunosine]

@yarikoptic I haven't had to interact with NDA uploading for a few years now, so my memories of all of this are pretty dim. I don't really have the bandwidth to fully re-establish context on this, but I might be able to chip in on small stuff. @obenshaindw, I know you're not and NDA anymore, but do you know anyone there who might be willing to give us some guidance on this?

[ericearl]

experiment-id was also proposed in another PR #37 by @ljchang : so it is required only for functional and not anatomicals/dwi? (in #37 it is mandatory)

In both of these cases, experiment_id is "Conditional" on whether scan_type == 'fMRI':

1. https://nda.nih.gov/data_structure.html?short_name=fmriresults01
2. https://nda.nih.gov/data_structure.html?short_name=imagingcollection01

But I don't know all the cases.

[agt24]

@obenshaindw, I know you're not and NDA anymore, but do you know anyone there who might be willing to give us some guidance on this?

I'd suggest Greg Magdits and/or Karl Kuntzelman. But neither of them seem to be @-able.

[agt24]

I emailed Greg and Karl and got the following helpful info from Greg:

@ericearl 's answer regarding experiment-id’s is correct – they are only required at the moment for fmriresults and imagingcollection data if the associated scan_type is “fMRI”. Information about when specific data-elements need to be specified can be retrieved from NDA’s datadictionary api

https://nda.nih.gov/api/datadictionary/datastructure/fmriresults01
https://nda.nih.gov/api/datadictionary/datastructure/imagingcollection01

Our internal team sometimes make changes to the definitions of the data-structures in NDA so its possible that the condition expression for an element in a data structure ( in the case of experiment-id, “scan_type == ‘fMRI’ ”) to change from one submission cycle to the next.

[agt24]

Also, the Google Doc that @Shotgunosine linked above is dated 2018. The manifest documentation in the NDA github repo is a bit newer, but I'm not sure if it as comprehensive.

https://github.com/NDAR/manifest-data

[ericearl]

@yarikoptic Did you know this was a thing?

https://ndabids.readthedocs.io

It's a set of tools I helped create in the DCAN Labs to upload the ABCD-BIDS Community Collection (ABCC) data to NDA collection 3165. It relies on YAML and a little JSON to map files as-is to NDA data structures and upload in batches of records. I'm happy to answer any questions or pull other people into the conversation.

This was the manifest "code" used there:

https://github.com/DCAN-Labs/nda-bids-upload/blob/main/records.py#L259-L281