Skip to content

Feat/stream cache data - #57

Merged
tyfolino merged 12 commits into
devfrom
feat/stream-cache-data
Jun 13, 2026
Merged

Feat/stream cache data#57
tyfolino merged 12 commits into
devfrom
feat/stream-cache-data

Conversation

@tyfolino

Copy link
Copy Markdown
Owner

This is a pretty big change. It is no longer sufficient to ask users to download 5 GB of kernels/tutorial data and store it automatically in their package directory; we need to add an option to stream the kernels from Jetstream 2. We have done the following:

  • Created an options.py module that encapsulates global user-defined options, including where to access the kernels (local, stream, or cache) and ties the current version of ClimKern to a specific version of the ClimKern kernels
  • Created a registry.txt that holds all the hashes so that users will be informed if they are using an outdated version of the kernels
  • Updated util.py, frontend.py, and download.py to use the new data access algorithm
  • Added a few tests to make sure our data access is working.

Closes #47 (if it works :P) and built upon @brian-rose's stream branch.

tyfolino and others added 12 commits June 13, 2026 12:53
- Created options.py to host user's decision of where to get the ClimKern kernel data from
- Added hash verification for files

Co-authored-by: Brian Rose <brose@albany.edu>
Co-authored-by: Brian Rose <brose@albany.edu>
…a loading

Co-authored-by: Brian Rose <brose@albany.edu>
…che (not necessarily the package directory)

Co-authored-by: Brian Rose <brose@albany.edu>
Co-authored-by: Brian Rose <brose@albany.edu>
Co-authored-by: Brian Rose <brose@albany.edu>
Co-authored-by: Brian Rose <brose@albany.edu>
@tyfolino
tyfolino merged commit 1fcd6d3 into dev Jun 13, 2026
8 checks passed
@tyfolino
tyfolino deleted the feat/stream-cache-data branch June 13, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Option to stream kernel data from Project Pythia jetstream2 object store

1 participant