from_cbor()/from_msgpack() do not validate UTF-8 in text strings at decode time (only dump() does)

[nlohmann]

Description

from_cbor() (and, by the same code path, from_msgpack()) decodes a CBOR/MessagePack text string by copying its raw bytes into a string_t without validating that they are well-formed UTF-8. RFC 8949 §3.1 requires CBOR major type 3 (text string) to contain "a UTF-8 string", and rejecting ill-formed UTF-8 there is expected decoder behavior — but this library defers that check to dump(), so from_cbor() itself neither throws nor returns a discarded value for invalid UTF-8 payloads. The invalid bytes only surface as an error later, if and when the resulting value is serialized back to JSON text.

Reproduction

#include <nlohmann/json.hpp>
#include <iostream>
using json = nlohmann::json;

int main() {
    // CBOR: 0x62 = text string, length 2; payload 0xC0 0xAE is not valid UTF-8
    std::vector<uint8_t> raw = {0x62, 0xc0, 0xae};

    auto got = json::from_cbor(raw, /*strict*/true, /*allow_exceptions*/false);
    std::cout << "is_discarded: " << got.is_discarded() << "\n";   // false
    std::cout << "type: "        << got.type_name()     << "\n";   // string

    got.dump(); // throws json.exception.type_error.316:
                // "invalid UTF-8 byte at index 0: 0xC0"
}

So from_cbor() reports success (a normal, non-discarded string value) for input the CBOR spec requires to reject; the error only appears later and only if the caller happens to call dump() (or another UTF-8-sensitive operation) on the resulting value.

Traced to binary_reader::get_string(), which forwards straight to get_bytes() — a raw byte copy with no UTF-8 check. Contrast with the text/JSON lexer, which does validate UTF-8 during scanning (scan_string() in lexer.hpp), and with dump()'s own validation (type_error.316).

Why this seems worth a look

• docs/features/binary_formats/bson.md documents comparable BSON leniency explicitly, with its own "Lenient BSON input handling" admonition and an escape hatch ("validate it separately before passing it to from_bson()"). I could not find an equivalent note for CBOR/MessagePack on the CBOR docs page or the from_cbor API docs — so this specific deferred-validation behavior currently isn't documented anywhere I could find.
• It means from_cbor(..., /*allow_exceptions=*/false) — the pattern used specifically to avoid exceptions and get a discarded sentinel on bad input instead — does not actually catch this category of malformed input; the exception can still surface later, from an unrelated call (dump()), if the caller isn't also validating UTF-8 there.

Suggested fix / alternatives

Either (a) validate UTF-8 in get_string()/get_bytes() at CBOR/MessagePack decode time, consistent with the text-JSON lexer, or (b) document the deferred-validation behavior the way BSON's leniency is documented, so it's a known, intentional tradeoff rather than a surprise.

Context

Found while investigating a third-party compliance report (#5525) — most of that report's findings turned out to be a bug in the reporting tool itself, but this particular case survived verification against the current develop header (commit 3bfe2b6da7393af5cfd68c44f58a1058e60499e2).

[xhon-pelushi]

Reproduced on develop (c72f37a), and it is not limited to CBOR and MessagePack — UBJSON has the same gap, and it is undocumented there too.

Feeding the ill-formed payload C0 AE as a text string to each reader, with strict=true, allow_exceptions=false:

format	input	is_discarded()	dump()
CBOR	62 C0 AE	false	throws type_error.316
MessagePack	A2 C0 AE	false	throws type_error.316
UBJSON	53 69 02 C0 AE	false	throws type_error.316
BSON	0F 00 00 00 02 6B 00 03 00 00 00 C0 AE 00 00	false	throws type_error.316

BSON behaving that way is expected given the "Lenient BSON input handling" note, so the finding is that UBJSON belongs in the same bucket as CBOR/MessagePack — accepted silently, nothing in the UBJSON docs mentioning it.

One correction to my own numbers, in case anyone repeats this: my first BSON attempt reported is_discarded() == true, which I nearly wrote up as "BSON validates". It does not — I had hand-written the document with a wrong length prefix (0x10 instead of 0x0F) and it was rejected as malformed for that reason. The row above uses a document built by to_bson() with only the two payload bytes swapped, and it round-trips cleanly before the swap.

On your (a)/(b): I am happy to send either, with tests, but they point in quite different directions for callers who currently rely on the lenient behaviour, so I would rather not guess. If you want (a), is validating in get_string() for all four readers the intent, or only the three that are currently undocumented? If you want (b), I can add the CBOR/MessagePack/UBJSON equivalents of the BSON admonition plus a note on the from_* API pages. Just say which and I will put it together.

[nlohmann]

Thanks for the thorough follow-up — good catch on UBJSON/BJData, and on the underlying reason the bug still reproduces on develop: #5531 was merged onto the still-open bson-sizes branch (#5185) rather than develop, so none of the fix had actually landed on develop yet.

Went with (a): ported the same get_string() fix directly onto develop on branch claude/issue-5529-comment-9f1lck. Since get_string() is the single choke point shared by all five binary readers, this covers CBOR, MessagePack, BSON, UBJSON, and BJData in one place — no per-format special-casing needed. Added decode-time tests for all five formats and the UBJSON/BJData docs admonitions that were missing from #5531's docs updates. No need to send a separate PR for this — happy to fold in anything else you spot, though.

---

Generated by Claude Code

[nlohmann]

One correction to the framing used in this issue and in #5531/the develop port above: none of the affected specifications actually require a decoder to reject ill-formed UTF-8.

Checked the primary sources directly:

• CBOR (RFC 8949 §3.1): text strings are defined as UTF-8, but the RFC explicitly says "a decoder might or might not want to verify that the sequence of bytes in a UTF-8 string is actually valid UTF-8 and react appropriately." It even draws a formal distinction between "well-formed" (structurally valid CBOR) and "valid" (also has proper UTF-8 text) — an item with ill-formed UTF-8 in a text string is well-formed CBOR, just not "valid" under the full data model. Decoder-side validation is explicitly optional.
• MessagePack: the type system section calls str "a UTF-8 string", but the spec's own Limitations section says: "String objects may contain invalid byte sequence and the behavior of a deserializer depends on the actual implementation when it received invalid byte sequence." Leniency is explicitly sanctioned here too.
• BSON (bsonspec.org): defines the string grammar as "the (byte) is zero or more UTF-8 encoded characters"*, but says nothing anywhere about what a decoder should do with malformed UTF-8. It's silent, not permissive-by-name like CBOR/MessagePack, but also not a rejection requirement.
• UBJSON / BJData: same as BSON — both spec texts say the string value "requires UTF-8 encoding" as a type definition, but neither says anything about decoder-side error handling.

So the validation added on claude/issue-5529-comment-9f1lck is a deliberate policy choice by this library (matching what dump() already enforces, and closing a real gap: none of the five to_* writers validate UTF-8 either, so a value with ill-formed UTF-8 could previously be silently decoded from one format and silently re-encoded into another, e.g. CBOR → BSON, with the invalid bytes surviving undetected) — not something any of these specs mandates. Worth being precise about that distinction in the docs/comments rather than saying the specs "require" it, since only the write side and non-dump() consumers were actually silent about it before; the specs themselves never obligated rejection.

---

Generated by Claude Code

[nlohmann]

So let's please discuss: Should we change the behavior of the binary readers to reject non-UTF-8 strings for all formats?

[nlohmann]

I know we just merged #5531, but I would be happy to revert if we agree that these changes were overly strict. /cc @gregmarr

[gregmarr]

Sounds to me that we were following "be strict in what you generate and lenient in what you accept" before. Since this is a non-required validation, I think this would need to be opt-in.