Skip to content

tech-debt: migrate JSON serialization to orjson for performance #1171

Description

@dgarros

Need

The SDK is a foundational library on the hot path of transforms, generators,
checks, exports/imports, and every GraphQL round-trip — JSON encode/decode
throughput directly affects how fast SDK-based automation runs at scale.
Today the SDK relies on ujson (and stdlib json in places). orjson, backed
by Rust (serde_json), is substantially faster than both and stricter on
correctness, so adopting it would give a measurable, no-behaviour-change speedup
for users running large syncs and pipelines.

Context

  • This is a deliberate follow-up to tech-debt: standardize JSON library usage (json vs ujson) across the SDK #1165 (JSON library standardization). That
    issue picks a single library across the codebase; this one proposes making
    that target orjson rather than settling on ujson. Sequencing/merging the
    two is up to whoever picks them up.
  • Adding orjson is a new runtime dependency — flagged here because the repo
    asks to confirm dependency additions.
  • Known migration constraints the implementer will hit (not a design, just
    gotchas): orjson.dumps() returns bytes rather than str, so str(...) /
    .encode() wrappers at call sites like playback.py:52 and utils.py:253
    change; and orjson has no sort_keys= kwarg (it uses
    option=orjson.OPT_SORT_KEYS), which dict_hash (utils.py:253) depends on.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mediumThis issue should be completed in a less than a daytype/tech-debtItem we know we need to improve way it is implemented

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions