Skip to content

Hotfix/stability#10

Open
steffencruz wants to merge 3 commits intomainfrom
hotfix/stability
Open

Hotfix/stability#10
steffencruz wants to merge 3 commits intomainfrom
hotfix/stability

Conversation

@steffencruz
Copy link
Copy Markdown
Contributor

@steffencruz steffencruz commented Aug 21, 2023

  • Adds workaround for known wandb decode error by forcing JSON parsing in non-strict mode
  • Ensures expected metagraph block list is sorted before creating parquet dataframe

@steffencruz steffencruz requested a review from eduardogr August 21, 2023 21:09

def relaxed_json_loads(s, *args, **kwargs):
kwargs.pop("strict", None) # remove the strict argument if it exists
return original_json_loads(s, strict=False, *args, **kwargs)
Copy link
Copy Markdown
Contributor

@eduardogr eduardogr Aug 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not call it explicitly like this in every json.loads call ?? or use this relaxed_json_loads.
Even though the approach is very nice in shape. I think it obscures what's really happening, and that's important for maintenance and readability of the code.

From my point of view, I'd say, let's make the call explicit as it is in this line, or use the function relaxed_json_loads, so we know something different is happening

What do you think about this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea it's not my preferred solution at all, however the issue actual originates inside the wandb codebase - they enforce strict mode when pulling assets from their databases. We never actually call json decide ourselves..
I propose we put it either here or shamefully in a Util file for now

Copy link
Copy Markdown
Contributor Author

@steffencruz steffencruz Aug 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's a way of overriding json globally.
Floppy actually came up with this hack because anyone who wants to analyze wandb data has to do something similar

@eduardogr
Copy link
Copy Markdown
Contributor

very nice catch!

LGTM after solving comments :)

Co-authored-by: Eduardo García <garciaruiz.edu+github@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants