Description
What is your issue?
Original issue in xarray-contrib/datatree#134 with additional comments.
Intake, catalogs, and datatree
Thanks @TomNicholas and sorry for creating issue noise. I guess I got a bit carried away with these comments in the readme:
- Has functions for mapping user-supplied functions over every node in the tree,
- Automatically dispatches some of xarray.Dataset's API over every node in the tree (such as .isel),
I was thinking that maybe the datatree abstraction could be a more formalised and ultimately 'xarray native' approach to the the problems that have been tackled by e.g. intake-esm and intake-thredds. Leaves in the tree could compositions over netcdf files, which may be aggregated JSON indexes. I guess I was thinking that some sort of formalism over a nested datastructure could help in dask computational graph composition. I have run into issues where the scheduler gets overloaded, or just takes forever to start for calculations across large datasets composed with i.e. mf_opendataset
Originally posted in xarray-contrib/datatree#97 (comment)