Implement a `rex` backend for xarray (rexarray) #192

ppinchuk · 2025-02-03T07:14:01Z

Add a backend for xarray that allows users to read in rex-style NREL data.

The implementation itself closely follows the h5netcdf backend implementation:
https://github.com/pydata/xarray/blob/main/xarray/backends/h5netcdf_.py

Lazy loading is fully supported, which is a big reason why this implementation is so long.

HSDS and S3 (via fsspec) access is explicitly supported.

rex/external/standard_attrs.json

rex/external/encondings.json

grantbuster · 2025-02-19T21:45:31Z

rex/external/rexarray.py

+            else:
+                lock = combine_locks([HDF5_LOCK, get_write_lock(filename)])
+
+        manager = CachingFileManager(h5py.File, filename, mode=mode,


Okay so you might have to walk me through how all of this works, but is this where you define how the file is actually being opened? and with what object handler?

My extra ask (or just naive question) is whether we can open an S3 or HSDS path with xarray.open_dataset() by using h5pyd.File or fsspec.open? It seems like maybe we can open an S3 path but my kernel just hangs when i try. I can't tell if it's latency because its doing a lot behind the scenes or if S3/fsspec is just not supported.

IDK if we can support those but I definitely want to if possible. Adding/validating h5py/fsspec is still on my TODO list (if it works already, its basically by accident). I plan to add tests for these cases as well, assuming it's possible to support these cases out of the box.

Sorry for dragging my feet on this code, I haven't touched it in a while. My plan is to get back into it this week

No worries at all, i haven't had time to review or provide feedback! I'm basically trying to figure out where the h5 file is actually opened and where we specify which object opens it. I think if we can figure that out we could add hsds/fsspec integration? Why don't you reach out when you start work on this again and explain stuff to me and maybe i can help or brainstorm.

So actually, the file is opened pretty much where you left this comment.

Definitely still happy to walk though this code - just let me know when you have time!

As of now, HSDS and S3 are explicitly supported. I have added tests for reading files using xarray in both the hsds and s3 test files.

ppinchuk · 2025-02-28T17:08:51Z

Dare I say this is ready for review? :P

One question for all, but especially @grantbuster : should we make fsspec a base rex dependency? I think this library is used all over the place, including in our other dependencies like pandas. This would mean users can immediately access S3 files without having to specify [s3] during installation. Did we make it an optional dependency intentionally? I know we had to make h5pyd optional because of compatibility issues with the wtk code

bnb32

Will take a deeper dive later but here are just a couple random comments.

rex/external/encondings.json

rex/external/rexarray.py

setup.py

bnb32 · 2025-03-04T16:54:19Z

rex/utilities/utilities.py

+    return data * scale_factor + adder
+
+
+def import_fsspec_or_fail(file_path=None):


Little thing - did you consider just having a combined function import_module_or_fail which accepts either "hsds" or "s3"?

Hmmm, went fast and didn't consider this. I assume you are suggesting an if statement inside a more generic function?

Yeah, exactly. Always find myself with these choices for concision vs simplicity haha.

bnb32 · 2025-03-04T17:03:50Z

rex/external/rexarray.py

+        self.attrs = {}
+
+
+class RexArrayWrapper(BackendArray):


This file mostly follows xarray code, right? Could you point to the main differences I should focus on? Is the main new thing this wrapper class?

Yup, this class and RexStore are the ones that contain most of the rex-specific changes. I would focus on those two

ppinchuk · 2025-03-18T22:55:36Z

@bnb32, @grantbuster, @castelao Friendly bump :)

Curious to hear if anyone has tried using this and/or had success with it

castelao

@ppinchuk , some minor comments.

castelao · 2025-03-19T02:39:18Z

requirements.txt

@@ -1,4 +1,5 @@
 click>=7.0
+dask


It's probably a good idea to set an expected range of versions such as dask = ">=2025.2.0,<2026"

Good call, will add!

requirements.txt

castelao · 2025-03-19T02:45:41Z

rex/external/standard_attrs.json

+        "description": "Global horizontal irradiance is the total amount of solar radiation received per unit area on a horizontal surface."
+    },
+    "inversemoninobukhovlength": {
+        "standard_name": "latitude",


This was probably left behind, but it looks like there is no standard name for the inverse. If that's right, the standard_name should be removed.

Shoot, great catch! Thanks!!

castelao · 2025-03-19T02:54:08Z

rex/external/standard_attrs.json

+        "standard_name": "air_temperature",
+        "units": "C"
+    },
+    "time": {


How do you use this time? A common approach is using f64 and using the metadata to encode/decode it, according to the calendar and units.

This is just a convenience alias for time_index, since people might expect that kind of data in a time variable

bnb32 · 2025-03-19T17:39:06Z

@bnb32, @grantbuster, @castelao Friendly bump :)

Curious to hear if anyone has tried using this and/or had success with it

Digging back in is on the list for next week :)

ppinchuk added 5 commits February 3, 2025 00:12

[WIP] Crude first pass of backend implementation

1ccd031

Add data

8417942

Add init module

6fc9507

Register backend

8b9da95

[WIP] Add tests

f551a05

ppinchuk self-assigned this Feb 3, 2025

ppinchuk added 2 commits February 3, 2025 00:16

Add xarray req

1aae855

Add package data

faa9a23

castelao reviewed Feb 3, 2025

View reviewed changes

rex/external/standard_attrs.json Outdated Show resolved Hide resolved

rex/external/standard_attrs.json Outdated Show resolved Hide resolved

rex/external/standard_attrs.json Outdated Show resolved Hide resolved

rex/external/encondings.json Outdated Show resolved Hide resolved

ppinchuk added 8 commits February 3, 2025 12:17

Add min version to xarray req

de4151b

Update version string

f36bb7c

Add dask as a req

0f899d8

Fix segfault

95a9edf

Better detect dimensions

2b6a7f6

Add test for dims

ef5824f

Add rex_unscale function

b8a019b

Use rex_unscale

f3050a7

grantbuster reviewed Feb 19, 2025

View reviewed changes

ppinchuk added 12 commits February 25, 2025 18:29

Python 3.9 compat

fdafb69

Update reqs

458c0b5

Python 3.9 fix

1b1054a

Custom ds loading so that close method is set properly

1c2022f

Add tests for closing mf ds

952200b

Minor updates

9d86aad

Add test

61a923f

Add docstrings

228a445

Add xarray tests

370d095

Add fsspec (S3) support

348677e

Remove deprecated arg

1184111

Use open_mfdataset

5e204e6

ppinchuk added 14 commits February 27, 2025 16:15

Add some utility funcs

df15362

More utility funcs

62b019d

Use new utility functions

f54b247

Cleanup part 1

1df3009

More cleanup

2d72d85

More cleanup

bb1edc0

More cleanup

b958526

More cleanup

9321598

Add test for overwriting attrs

d1174e2

Linter fix

85e1392

Formatting

490573d

Reformat

233fb1a

Linter fixes

b0a6687

Add nsrdb attrs

305f15e

ppinchuk requested review from castelao, grantbuster and bnb32 and removed request for castelao February 28, 2025 17:04

ppinchuk added enhancement Update to logic or general code improvements p-medium Priority: medium labels Feb 28, 2025

ppinchuk marked this pull request as ready for review February 28, 2025 17:05

Fix test

d31116f

ppinchuk requested a review from castelao February 28, 2025 17:43

ppinchuk changed the title ~~[WIP] Implement a rex backend for xarray (rexarray)~~ Implement a rex backend for xarray (rexarray) Feb 28, 2025

bnb32 reviewed Mar 4, 2025

View reviewed changes

Fix typo

24c501b

castelao reviewed Mar 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a `rex` backend for xarray (rexarray) #192

Implement a `rex` backend for xarray (rexarray) #192

ppinchuk commented Feb 3, 2025 •

edited

Loading

grantbuster Feb 19, 2025 •

edited

Loading

ppinchuk Feb 19, 2025

grantbuster Feb 19, 2025

ppinchuk Feb 28, 2025

ppinchuk commented Feb 28, 2025

bnb32 left a comment

bnb32 Mar 4, 2025

ppinchuk Mar 4, 2025

bnb32 Mar 4, 2025

bnb32 Mar 4, 2025

ppinchuk Mar 4, 2025

ppinchuk commented Mar 18, 2025

castelao left a comment

castelao Mar 19, 2025

ppinchuk Mar 19, 2025

castelao Mar 19, 2025

ppinchuk Mar 19, 2025

castelao Mar 19, 2025

ppinchuk Mar 19, 2025

bnb32 commented Mar 19, 2025

		return data * scale_factor + adder


		def import_fsspec_or_fail(file_path=None):

Implement a rex backend for xarray (rexarray) #192

Are you sure you want to change the base?

Implement a rex backend for xarray (rexarray) #192

Conversation

ppinchuk commented Feb 3, 2025 • edited Loading

grantbuster Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ppinchuk commented Feb 28, 2025

bnb32 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ppinchuk commented Mar 18, 2025

castelao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnb32 commented Mar 19, 2025

Implement a `rex` backend for xarray (rexarray) #192

Implement a `rex` backend for xarray (rexarray) #192

ppinchuk commented Feb 3, 2025 •

edited

Loading

grantbuster Feb 19, 2025 •

edited

Loading