Skip to content

Commit f578245

Browse files
authored
feat: pack() type and family hints (#52) (#25690)
1 parent 1efff3d commit f578245

9 files changed

Lines changed: 1231 additions & 17 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
## [3.1.0] - 2026-09-19
11+
12+
Optional second-argument pack hints force a MessagePack wire type or family
13+
without changing the default mapping. Two or more values still pack as an
14+
array. See `#52`.
15+
16+
### Added
17+
18+
- `pack(value, { type })` writes a fixed MessagePack type (`fixint`,
19+
`uint8`…`uint64`, `int8`…`int64`, `float32`/`float64`, `fixstr`/`str8`…
20+
`str32`, `bin8`…`bin32`, `nil`/`true`/`false`). Out-of-range values throw
21+
`cannot pack value as <type>`.
22+
- `pack(value, { family })` picks a compact encoding in that family (`int`,
23+
`float`, `str`, `bin`). `type` wins if both are set.
24+
- `pack(array, { interpret })` maps each element through `interpret(item)`
25+
which must return `{ data }` and may also set `type` / `family`.
26+
- Detection is last-argument, two-arg only: the object must own-enumerate
27+
only `type`, `family`, and/or `interpret`. Extra keys, one-arg objects, and
28+
`pack(1, 2)` keep the old array packing.
29+
1030
## [3.0.0] - 2026-09-19
1131

1232
Integers whose magnitude is greater than `Number.MAX_SAFE_INTEGER` unpack as
@@ -88,6 +108,7 @@ GitHub Actions tests Node 18/20/22 on Ubuntu, macOS, and Windows 2022.
88108
- Pack throw paths free or return pooled sbuffers on every exit.
89109
- msgpack-c c-7.0.2 includes unpacker buffer-expansion overflow checks.
90110

91-
[Unreleased]: https://github.com/msgpack/msgpack-node/compare/v3.0.0...HEAD
111+
[Unreleased]: https://github.com/msgpack/msgpack-node/compare/v3.1.0...HEAD
112+
[3.1.0]: https://github.com/msgpack/msgpack-node/compare/v3.0.0...v3.1.0
92113
[3.0.0]: https://github.com/msgpack/msgpack-node/compare/e04c9b55f98d64512174d6e859b8294b729659a2...HEAD
93114
[2.0.0]: https://github.com/msgpack/msgpack-node/commit/e04c9b55f98d64512174d6e859b8294b729659a2

‎COVERAGE.md‎

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Coverage — msgpack 3.0.0
1+
# Coverage — msgpack 3.1.0
22

33
`npm run coverage` runs both halves and fails the build under 95%.
44

@@ -8,9 +8,9 @@
88
| `lib/` + `bin/` (c8) | branches | **100%** | ≥ 95% |
99
| `lib/` + `bin/` (c8) | functions | **100%** | ≥ 95% |
1010
| `lib/` + `bin/` (c8) | lines | **100%** | ≥ 95% |
11-
| `src/msgpack.cc` (gcovr) | lines | **95.9%** (473/493) | ≥ 95% |
12-
| `src/msgpack.cc` (gcovr) | branches | **99.5%** (400/402) | ≥ 95% |
13-
| `src/msgpack.cc` (gcovr) | functions | 100% (36/36) | — |
11+
| `src/` (gcovr) | lines | **95.2%** (902/947) | ≥ 95% |
12+
| `src/` (gcovr) | branches | **95.4%** (836/876) | ≥ 95% |
13+
| `src/` (gcovr) | functions | 100% (59/59) | — |
1414

1515
`deps/` is excluded from the native report; the vendored msgpack-c is not our
1616
code. `build/` is rebuilt without instrumentation at the end of
@@ -56,12 +56,12 @@ Every one is an error arm that cannot be entered from JS without stubbing
5656
| 735 | `throw` after `msgpack_pack_array` in `Pack` | Allocation failure only. |
5757
| 809–813, 815 | `MSGPACK_UNPACK_CONTINUE` / parse-error tail of `Unpack` | `ScanOne` walks the same grammar first with limits at or below the vendored library's own (511 vs 512 nested containers, the same 1 000 000 element cap), so once it returns `kScanOk`, `msgpack_unpack_next` can only succeed. The arms stay so a future divergence fails closed instead of reading `result.data` uninitialised. |
5858

59-
## Remaining uncovered native branches (2 of 402)
59+
## Remaining uncovered native branches in `msgpack.cc` (2)
6060

6161
| Line | Code | Why |
6262
| --- | --- | --- |
63-
| 148 | `switch (b)` in `ScanOne` | The `default:` edge — see lines 307–308 above. It cannot be excluded on its own without also dropping the 30 covered case edges on the same line, so it is left in and counted against us. |
64-
| 575 | `switch (mo->type)` in `MsgpackToJs` | Same, for the `default:` edge covering the complete `msgpack_object_type` enum. |
63+
| 154 | `switch (b)` in `ScanOne` | The `default:` edge — see lines 307–308 above. It cannot be excluded on its own without also dropping the 30 covered case edges on the same line, so it is left in and counted against us. |
64+
| 611 | `switch (mo->type)` in `MsgpackToJs` | Same, for the `default:` edge covering the complete `msgpack_object_type` enum. |
6565

6666
## About the native branch number
6767

@@ -139,6 +139,13 @@ gcovr --root . --filter src/ --exclude deps/ --no-markers --txt-metric branch --
139139
failure modes and mark cleanup; and a worker that nests 600 packs deep to
140140
saturate the thread-local sbuffer pool and reach the "pool is full, free it"
141141
arm of `~PackBuffer`.
142+
- `test/pack-hints.test.js` — `pack(value, { type, family, interpret })` wire
143+
types, last-arg options detection (host objects, Proxies, ownKeys throws),
144+
and the reachable miss paths in `src/pack_hints.inc`. Unreachable arms
145+
(empty `info[1]`, `kTypeNone` / `kFamilyNone` defaults, hinted
146+
`kMaxPackDepth`) are marked `GCOVR_EXCL_*`, not deleted. Native overall
147+
stays above the 95% gate (`pack_hints.inc` itself is 91% branches because
148+
switch `default:` edges sit on the same line as covered cases).
142149
- `test/cli.test.js` (12 tests) — the exit-1 paths of both CLIs: invalid JSON,
143150
empty stdin, a pack rejection reachable from real JSON, an unparseable byte,
144151
an oversized header, incomplete input both alone and after a good frame, and

‎README.md‎

Lines changed: 44 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
and de-serializes JavaScript values with [MessagePack](https://msgpack.org).
33
Packed output is a `Buffer` and is typically much smaller than JSON.
44

5-
Version 3.0 requires **Node.js 18+**, vendors **msgpack-c c-7.0.2**, and
6-
unpacks 64-bit integers outside `Number.MAX_SAFE_INTEGER` as `bigint`. See
7-
[`SECURITY.md`](SECURITY.md).
5+
Version 3.1 requires **Node.js 18+**, vendors **msgpack-c c-7.0.2**, unpacks
6+
64-bit integers outside `Number.MAX_SAFE_INTEGER` as `bigint`, and accepts
7+
optional pack type/family hints. See [`SECURITY.md`](SECURITY.md).
88

99
### Usage
1010

@@ -79,6 +79,47 @@ is that same `bigint`.
7979
`unpack.bytes_remaining` is the number of unused trailing bytes after the last
8080
successful (or attempted) unpack. Stream uses that to splice leftover data.
8181

82+
### Pack type hints (3.1)
83+
84+
`pack(value, options)` takes an optional last-argument options object when
85+
there are exactly two arguments and that object own-enumerates only `type`,
86+
`family`, and/or `interpret`. Extra keys, a one-argument `{ type: ... }`
87+
value, and `pack(1, 2)` still pack as values / an array.
88+
89+
```javascript
90+
msgpack.pack(123, { type: 'fixint' }); // 0x7b
91+
msgpack.pack(123, { type: 'uint8' }); // 0xcc 0x7b
92+
msgpack.pack(Math.PI, { type: 'float32' }); // 0xca + 4 bytes
93+
msgpack.pack(buf, { family: 'bin' });
94+
msgpack.pack(1.5, { family: 'int' }); // throws
95+
msgpack.pack(500, { type: 'uint8' }); // throws
96+
97+
msgpack.pack(
98+
[
99+
{ data: Math.PI, type: 'float64' },
100+
{ data: 3.14, type: 'float32' },
101+
],
102+
{
103+
interpret(item) {
104+
return { data: item.data, type: item.type };
105+
},
106+
},
107+
);
108+
```
109+
110+
`type` forces that MessagePack type (`fixint`, `uint8`…`uint64`, `int8`…
111+
`int64`, `float32`/`float64`, `fixstr`/`str8`/`str16`/`str32`, `bin8`/`bin16`/
112+
`bin32`, `nil`/`true`/`false`). `family` (`int`, `float`, `str`, `bin`) picks
113+
a compact encoding in that family. If both are set, `type` wins. Out-of-range
114+
values throw `cannot pack value as <type>`.
115+
116+
`interpret` is used when packing an Array. Each element is passed to
117+
`interpret(item)`, which must return `{ data }` and may also set `type` /
118+
`family` for that element. Nested `interpret` on the returned object is
119+
ignored.
120+
121+
Default packing is unchanged when no recognized options object is passed.
122+
82123
### Limits
83124

84125
* array/map length ≤ 1,000,000

‎index.d.ts‎

Lines changed: 48 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,67 @@
1-
// Type definitions for msgpack 3.0.0
1+
// Type definitions for msgpack 3.1.0
22
// Project: https://github.com/msgpack/msgpack-node
33

44
/// <reference types="node" />
55

66
import { EventEmitter } from 'events';
77

8+
export type PackType =
9+
| 'fixint'
10+
| 'uint8'
11+
| 'uint16'
12+
| 'uint32'
13+
| 'uint64'
14+
| 'int8'
15+
| 'int16'
16+
| 'int32'
17+
| 'int64'
18+
| 'float32'
19+
| 'float64'
20+
| 'fixstr'
21+
| 'str8'
22+
| 'str16'
23+
| 'str32'
24+
| 'bin8'
25+
| 'bin16'
26+
| 'bin32'
27+
| 'nil'
28+
| 'true'
29+
| 'false';
30+
31+
export type PackFamily = 'int' | 'float' | 'str' | 'bin';
32+
33+
export interface PackInterpretResult {
34+
data: any;
35+
type?: PackType;
36+
family?: PackFamily;
37+
}
38+
39+
export interface PackOptions {
40+
type?: PackType;
41+
family?: PackFamily;
42+
interpret?: (item: any) => PackInterpretResult;
43+
}
44+
845
/**
946
* Serialize values to MessagePack.
1047
*
1148
* A single argument is packed as itself; two or more are packed as an array
1249
* of that many elements.
1350
*
51+
* When the second argument own-enumerates only `type`, `family`, and/or
52+
* `interpret`, it is pack options rather than a second value. `type` forces
53+
* a MessagePack wire type; `family` picks a compact encoding in that family
54+
* (`type` wins if both are set). `interpret` is used when packing an Array:
55+
* each element is replaced by `interpret(item)`, which must return `{ data }`
56+
* and may also set `type` / `family`.
57+
*
1458
* `bigint` values in the int64/uint64 range pack as MessagePack integers
1559
* (smallest family that fits). Values outside that range throw. A `number`
1660
* that has already lost bits below 2^53 stays on the Number path; lost bits
17-
* are not recovered.
61+
* are not recovered. BigInt plus an integer `type`/`family` uses the same
62+
* 64-bit path.
1863
*/
64+
export function pack(value: any, options: PackOptions): Buffer;
1965
export function pack(...values: any[]): Buffer;
2066

2167
/**

‎package-lock.json‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "msgpack",
33
"description": "A space-efficient object serialization library for Node.js",
4-
"version": "3.0.0",
4+
"version": "3.1.0",
55
"homepage": "https://github.com/msgpack/msgpack-node",
66
"author": "Peter Griess <pg@std.in>",
77
"contributors": [
@@ -34,7 +34,7 @@
3434
"nan": "^2.23.1"
3535
},
3636
"scripts": {
37-
"test": "node --test test/bigint.test.js test/cli.test.js test/coverage-native.test.js test/msgpack.test.js test/regression.test.js test/security.test.js test/worker.test.js",
37+
"test": "node --test test/bigint.test.js test/cli.test.js test/coverage-native.test.js test/msgpack.test.js test/pack-hints.test.js test/regression.test.js test/security.test.js test/worker.test.js",
3838
"bench": "node test/benchmark/benchmark.js",
3939
"rebuild": "node-gyp rebuild",
4040
"coverage": "npm run coverage:js && npm run coverage:native",

‎src/msgpack.cc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515

1616
#include <cmath>
1717
#include <cstdint>
18+
#include <cstdio>
19+
#include <cstring>
1820

1921
#include <nan.h>
2022
#include <msgpack.h>
@@ -425,6 +427,18 @@ static v8::Local<v8::Array> CheckedOwnNames(v8::Local<v8::Object> obj) {
425427
return r.ToLocalChecked();
426428
}
427429

430+
static v8::Local<v8::Value> CallOneArg(v8::Local<v8::Object> recv,
431+
v8::Local<v8::Function> fn,
432+
v8::Local<v8::Value> arg) {
433+
Nan::TryCatch try_catch;
434+
v8::Local<v8::Value> argv[1] = {arg};
435+
Nan::MaybeLocal<v8::Value> r = Nan::Call(fn, recv, 1, argv);
436+
if (r.IsEmpty()) {
437+
ThrowCaught(try_catch); /* GCOVR_EXCL_BR_LINE: never returns */
438+
}
439+
return r.ToLocalChecked();
440+
}
441+
428442
static void PackArray(msgpack_packer* pk, v8::Local<v8::Array> arr, int depth) {
429443
if (IsMarked(arr)) {
430444
throw MsgpackException(Error("Cowardly refusing to pack circular reference"));
@@ -589,6 +603,8 @@ static void JsToMsgpack(msgpack_packer* pk, v8::Local<v8::Value> o, int depth) {
589603
/* GCOVR_EXCL_BR_STOP */
590604
}
591605

606+
#include "pack_hints.inc"
607+
592608
static v8::Local<v8::Value> MsgpackToJs(const msgpack_object* mo);
593609

594610
static v8::Local<v8::Value> MsgpackToJs(const msgpack_object* mo) {
@@ -755,6 +771,9 @@ NAN_METHOD(Pack) {
755771

756772
if (info.Length() == 1) {
757773
JsToMsgpack(&pk, info[0], 0);
774+
} else if (info.Length() == 2 && IsPackOptionsObject(info[1])) {
775+
PackHint hint = ParsePackOptions(info[1].As<v8::Object>());
776+
JsToMsgpackHinted(&pk, info[0], 0, hint);
758777
} else {
759778
/* GCOVR_EXCL_BR_START: allocation failure only. */
760779
if (msgpack_pack_array(&pk, info.Length())) {

0 commit comments

Comments
 (0)