Commit 4bfe00b
refactor/consolidated JSON IO routines (#3998)
* feat: add free functions for JSON document I/O
Add four free functions for moving JSON documents in and out of stores,
plus a thin StorePath.get_json wrapper:
- buffer_to_json / json_to_buffer: convert between a Buffer and a parsed
JSON value. The buffer prototype and the serialization policy
(indent from config, allow_nan=True) live here, in one place.
- get_json / set_json: compose those with Store.get / Store.set.
get_json returns None for a missing key, matching what most callers
want; callers needing presence check for None themselves.
These are free functions, not Store ABC methods, so stores cannot (and
need not) override them and the Store contract gains no dependency on
the buffer prototype or global config. Subsequent commits sweep the
hand-rolled JSON I/O sites onto these and delete the old private
_get_bytes/_get_json store methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: route JSON document I/O through the free functions
Sweep the hand-rolled `json.dumps`/`json.loads` + buffer construction in
the metadata write paths and the metadata/node read paths onto the free
functions added in the previous commit.
Write sites (`to_buffer_dict` in group/metadata) now call `json_to_buffer`,
which centralizes the `json_indent`/`allow_nan=True` serialization policy.
Read sites parse buffers through `buffer_to_json_object`, a new helper that
narrows the parsed `JSON` to `dict[str, JSON]` once (every metadata document
is an object) rather than relying on `json.loads` returning `Any`.
The compact (no-indent) consolidated-metadata blob in `GroupMetadata.to_buffer_dict`
is intentionally left as a raw `json.dumps` to preserve its byte layout, and
`_read_metadata_v2` keeps its `asyncio.gather` of three `store.get` calls.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor: remove the unused private JSON/bytes store methods
`Store._get_bytes`, `_get_bytes_sync`, `_get_json`, and `_get_json_sync`
had no production callers — the metadata read/write paths now go through
the free functions in `zarr.core._json`. Delete the four ABC methods, the
eight per-store overrides on `LocalStore`/`MemoryStore` (which existed only
to make the `prototype` argument optional), and the tests that exercised
them (the shared `StoreTests` methods plus the per-store prototype=None
tests). Drop the now-unused `json`/`sync`/`Any` imports they pulled in.
These methods were always private; removing them is not a public API change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* doc: add changelog fragment for the JSON I/O refactor
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor: make json_to_buffer pure (no config read)
`json_to_buffer` (and `set_json`) now take the JSON encoding parameters
`indent` and `allow_nan` as explicit keyword arguments instead of reading
`config.get("json_indent")` internally. The functions in `zarr.core._json`
are now pure: they depend only on their arguments, not the global config.
Each `to_buffer_dict` caller (group, metadata v2/v3) reads
`config.get("json_indent")` and passes it as `indent=`. This re-localizes
the config read to the metadata layer that owns the policy, and keeps the
I/O helpers free of any config dependency.
With `indent` now explicit, the compact consolidated-metadata blob in
`GroupMetadata.to_buffer_dict` also routes through `json_to_buffer`
(indent defaults to None == compact), removing the last raw `json.dumps`
in that method and the `json` import from group.py. Output bytes are
unchanged (the consolidated/metadata regression tests pass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* doc: rename changelog fragment to PR number 3998
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test: cover the malformed-metadata branches in contains_array/group
The except clauses in `contains_array`, `contains_group`, and
`_contains_node_v3` (widened to catch `TypeError` in the JSON I/O refactor)
had no test exercising them — a stored `zarr.json` that is malformed bytes,
valid-but-non-object JSON, or an object missing `node_type` now has a test
asserting each function reports the node as absent. One test per failure
mode; the two `contains_*` functions are parametrized over `func`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Update src/zarr/core/_json.py
Co-authored-by: Chuck Daniels <cjdaniels4@gmail.com>
* fix: missing import
* doc: address review nits on _json.py docstrings
Add a Parameters section to buffer_to_json_object and drop the
redundant type annotations from the numpydoc Parameters sections,
per review feedback on #3998.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Chuck Daniels <cjdaniels4@gmail.com>1 parent 467dda7 commit 4bfe00b
15 files changed
Lines changed: 378 additions & 967 deletions
File tree
- changes
- src/zarr
- abc
- core
- metadata
- storage
- testing
- tests
- test_store
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | 4 | | |
6 | 5 | | |
7 | 6 | | |
8 | 7 | | |
9 | 8 | | |
10 | 9 | | |
11 | | - | |
12 | | - | |
13 | 10 | | |
14 | 11 | | |
15 | 12 | | |
| |||
219 | 216 | | |
220 | 217 | | |
221 | 218 | | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
247 | | - | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
255 | | - | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
265 | | - | |
266 | | - | |
267 | | - | |
268 | | - | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | | - | |
283 | | - | |
284 | | - | |
285 | | - | |
286 | | - | |
287 | | - | |
288 | | - | |
289 | | - | |
290 | | - | |
291 | | - | |
292 | | - | |
293 | | - | |
294 | | - | |
295 | | - | |
296 | | - | |
297 | | - | |
298 | | - | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | | - | |
303 | | - | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
385 | | - | |
386 | | - | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
397 | | - | |
398 | | - | |
399 | | - | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
406 | | - | |
407 | | - | |
408 | | - | |
409 | | - | |
410 | | - | |
411 | | - | |
412 | | - | |
413 | | - | |
414 | | - | |
415 | | - | |
416 | | - | |
417 | | - | |
418 | | - | |
419 | | - | |
420 | | - | |
421 | | - | |
422 | | - | |
423 | | - | |
424 | | - | |
425 | | - | |
426 | | - | |
427 | | - | |
428 | | - | |
429 | | - | |
430 | | - | |
431 | | - | |
432 | | - | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
440 | | - | |
441 | | - | |
442 | | - | |
443 | | - | |
444 | 219 | | |
445 | 220 | | |
446 | 221 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
0 commit comments