image/ID/download_urls json endpoint

[will-moore]

This adds a JSON endpoint to access the same download_urls that are displayed on the image landing pages.

We use the https://raw.githubusercontent.com/IDR/idr.openmicroscopy.org/refs/heads/master/_data/download_urls.tsv table with the same logic as for the landing pages. For a given row (per study) we use the replace_this and with_this columns. See https://proxy-idr-ap1.dundee.ac.uk/download_urls/ where this logic is used to generate the URL for the client_path column.

To test:

• for an image in IDR, go to /image/ID/download_urls/
• JSON response includes client_paths from the server, as well as the download_urls derived from these by replacing replace_this with with_this.

NB: This requires a config change for public url regex. /image is needed (similar to /api etc).

[will-moore]

Deployed on idr-testing.

E.g. https://idr-testing.openmicroscopy.org/image/15154257/download_urls/

cc @jburel @dominikl

[will-moore]

As discussed: idr0168 and idr0170 have no replace_this and with_this columns in the download_urls.tsv. Since these do have client paths, let's see if we can find something useful here...

[pwalczysko]

Edit 2:

Actually, some GitHub links work as expected. The data are never on GitHub
But one must click onto the before replacement column to get to the ftp.

https://github.com/IDR/idr0106-kubota-lunglightsheet/blob/main/experimentA/companions/Mosaic%20Image_Z%5B00001%5D_L%5B1%5D.ome.tiff

[pwalczysko]

Maybe the answer for the Github lines is: Go to EBI when the EBI link is present, otherwise stick to the ftp before replacement with_this ?

[will-moore]

OK, let's document all the GitHub and other studies where:

• this new /image/ID/download_urls/ doesn't work
• Of these which are "zarr" images (we care less about these as the iviewer works - no placeholder page)

Failing studies

• idr0170 (zarr) data/idr-metadata/idr0170-rose-mibitof/experimentA/../metadata/RPM-TMA-1_ROI01_multichannel.ome.xml Images are named like RPM-TMA-1_ROI01_multichannel.zarr which needs to download https://ftp.ebi.ac.uk/biostudies/fire/S-BIAD/961/S-BIAD1961/Files/experimentA/RPM-TMA/RPM-TMA-1_ROI01_multichannel.zarr.zip - Download url is https://ftp.ebi.ac.uk/biostudies/fire/S-BIAD/961/S-BIAD1961/Files/experimentA/RPM-TMA/ + image-name + .zip
• idr0168 (zarr) Image name: A375_6_ATP13A5_Z-stack_HF_01_2024-03-06_YunHao_23.31.23.zarr - Download url https://ftp.ebi.ac.uk/biostudies/fire/S-BIAD/891/S-BIAD1891/Files/A375_6_ATP13A5_Z-stack_HF_01_2024-03-06_YunHao_23.31.23.zarr.tgz is https://ftp.ebi.ac.uk/biostudies/fire/S-BIAD/891/S-BIAD1891/Files/ + name + .tgz
• idr0163 S-BIAD1288 Download url is https://ftp.ebi.ac.uk/biostudies/fire/S-BIAD/288/S-BIAD1288/Files/TONIC/image_data/samples/ + name + .zip
• idr0106 (github) - Single image. LOTs of files in fileset. companion on GitHub (working). Download tiffs from https://ftp.ebi.ac.uk/pub/databases/IDR/idr0106-kubota-lunglightsheet/20210226/Mosaic%20Image_Z%5b00001%5d_L%5b1%5d.ome.tiff - replace data/idr-metadata/idr0106-kubota-lunglightsheet/experimentA/companions/ with https://ftp.ebi.ac.uk/pub/databases/IDR/idr0106-kubota-lunglightsheet/20210226/. e.g.
  • data/idr-metadata/idr0106-kubota-lunglightsheet/experimentA/companions/Mosaic_Image.companion.ome (github)
  • data/idr-metadata/idr0106-kubota-lunglightsheet/experimentA/companions/Mosaic Image_Z[00001]_L[1].ome.tiff
• idr0096: Need to replace # in url e.g. https://ftp.ebi.ac.uk/pub/databases/IDR/idr0096-tratwal-marrowquant/20210609-ftp-ome-tiffs/m4%20pelvic%20(20x%20#2).ome.tiff
• idr0090 (zarr - mkngff) Download multiple (10) zarr.zips per Plate: https://ftp.ebi.ac.uk/biostudies/fire/S-BIAD/882/S-BIAD882/Files/idr0090/ - e.g. https://ftp.ebi.ac.uk/biostudies/fire/S-BIAD/882/S-BIAD882/Files/idr0090/190129.ome.zarr.zip.011 - https://idr-testing.openmicroscopy.org/image/12539701/download_urls/ gives ALL zarr files (except chunks)
• idr0079 (GitHub) companion.ome is OK USE .raw link! uod/idr/metadata/idr0079-hartmann-lateralline/companions/experimentA/056F63395C/056F63395C_8bit_lynEGFP.tif to https://ftp.ebi.ac.uk/pub/databases/IDR/idr0079-hartmann-lateralline/20200220-ftp/experimentA/056F63395C/056F63395C_8bit_lynEGFP.tif Replace uod/idr/metadata/idr0079-hartmann-lateralline/companions/ with https://ftp.ebi.ac.uk/pub/databases/IDR/idr0079-hartmann-lateralline/20200220-ftp/
• id0075 - companion OK. tif uod/idr/metadata/idr0075-cabirol-honeybee/experimentA/companions/MBLipOverviewBee1.tif https://ftp.ebi.ac.uk/pub/databases/IDR/idr0075-cabirol-honeybee/20191129-ftp/Raw%20Images/MBLipOverviewBee1.tif replace uod/idr/metadata/idr0075-cabirol-honeybee/experimentA/companions/ with https://ftp.ebi.ac.uk/pub/databases/IDR/idr0075-cabirol-honeybee/20191129-ftp/Raw%20Images/
• idr0071 - https://ftp.ebi.ac.uk/pub/databases/IDR/idr0071-feldman-crisprko/20201027-companions/companions/experimentA/process/10X_A1_Tile-102.aligned.tif is 404. No tifs in https://ftp.ebi.ac.uk/pub/databases/IDR/idr0071-feldman-crisprko/20201027-companions/companions/experimentA/process/ ???
• idr0065 (GitHub) uod/idr/metadata/idr0065-camsund-crispri/experimentA/companions/subpool-1_run-1_EXP-19-BQ3550/pheno/Pos101/phase/img_000000000.tiff - Too tricky to work this out!!