Skip to content

Conversation

pp-mo
Copy link
Member

@pp-mo pp-mo commented Sep 30, 2025

Closes #6374

This approach uses f3fs-FUSE

Tested: have shown that this can load netcdf and PP files from an S3 bucket.

Much to-do

  • add f3fs-FUSE to deps
  • lots of tests
  • consider saving
  • further explore the "unmount problems"

Copy link

codecov bot commented Sep 30, 2025

Codecov Report

❌ Patch coverage is 5.76923% with 98 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.93%. Comparing base (8ed2570) to head (ada62dd).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
lib/iris/io/s3_mounts.py 0.00% 92 Missing ⚠️
lib/iris/loading.py 50.00% 5 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6731      +/-   ##
==========================================
- Coverage   90.29%   89.93%   -0.36%     
==========================================
  Files          91       92       +1     
  Lines       24472    24577     +105     
  Branches     4569     4583      +14     
==========================================
+ Hits        22097    22104       +7     
- Misses       1607     1704      +97     
- Partials      768      769       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pp-mo
Copy link
Member Author

pp-mo commented Oct 1, 2025

Update: @ukmo-ccbunney pointed out that it could be simpler to leave "s3fs" mounting to the user.
That certainly looks like a simpler approach, and doesn't mire us in some complex issues which this approach raises:

  • testing is not too simple
  • introduces awkward dependencies
  • could be tricky to maintain
  • apparently, works for Linux/macOS/FreeBSD -- but not elsewhere
  • the indeterminate unmount problem ("umount target is busy") is a worry -- especially if it could have security implications

What we might do is provide some simple advice in the Iris docs under "Further Topics" ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Extend Iris loading to accept S3 URLs
1 participant