From fd642063f550e696220471600f4e88d5c36f5241 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 14 May 2025 15:49:57 +0200 Subject: [PATCH 01/25] init --- 23.md | 279 +++++++++++++++++++++++++++++++++++++++++++++++ tests/23-test.md | 23 ++++ 2 files changed, 302 insertions(+) create mode 100644 23.md create mode 100644 tests/23-test.md diff --git a/23.md b/23.md new file mode 100644 index 00000000..89bb0767 --- /dev/null +++ b/23.md @@ -0,0 +1,279 @@ +# NUT-23: Golomb-Coded Set Filters + +`optional` + +`depends on: NUT-06` + +--- + +This NUT describes a structure for compact filters on ecash notes and other information, for client-side use. The filter construction proposed +is an alternative to Bloom filters that minimizes size by using Golomb-Rice coding for compression. + +Clients can query and use this filter to test for set membership, which comes in particularly handy for checking the spent state of ecash notes. + +## Specification + +### Filter + +For each keyset, compact filters are derived containing sets of items associated with the keyset (spent `Y`s or blind signatures `B_`). +A set of such data objects is compressed into a probabilistic structure called a Golomb-Coded set (GCS), which matches all items in the set with +probability $1$, and matches other items with probability $\frac{1}{M}$. The encoding is also parameterized by `P`, the bit length of the remainder code. +Each filter defined specifies values for `P` and `M`, as well as the cardinality of the set it encodes `N`. + +At a high level, a GCS is constructed from a set of $N$ items by: + +1. hashing all items to 64-bit integers in the range $[0, N\cdot M)$ +2. sorting the hashed values in ascending order +3. computing the differences between each value and the previous one +4. writing the differences sequentially, compressed with Golomb-Rice coding + +The following sections describe each step in greater detail. + +### Hashing Data Objects + +The first step in the filter construction is hashing the variable-sized raw items in the set to the range $[0, F)$, where $F = N \cdot +M$. Analysis has shown that `M` and `P` are necessarily bound by $M = 1.497137 \cdot 2^P$. Practically, $M$ can be set to $2^P$ or $2^{P+1}$. +Set membership queries against the hash outputs will have a false positive rate of $\frac{1}{M}$. + +To avoid integer overflow, the number of items `N` **MUST** be <2^32 and `M` **MUST** be <2^32. + +The items are first passed through the pseudorandom function SipHash, which takes a 128-bit key `k` and a variable-sized byte vector and produces a uniformly random 64-bit output. Implementations of this NUT **MUST** use the SipHash parameters `c = 2` and `d = 4`. + +The 64-bit SipHash outputs are then mapped uniformly over the desired range by multiplying with `F` and taking the top 64 bits of the 128-bit result. This algorithm is a faster alternative to modulo reduction, as it avoids the expensive division operation while still retaining a fair mapping property. + +> [!NOTE] +> $F \cdot \text{SipHash}(x) \mod 2^{64} \neq \text{SipHash}(x) \mod F$. +> Make sure your implementation does this right! + +```python +def hash_to_range(item: bytes, f: int, key: bytes) -> int: + """ + Hashes an item to a range using SipHash. + + Args: + item (bytes): The item to hash. + f (int): The maximum range value. + key (bytes): The key used for hashing. + + Returns: + int: The hashed value within the specified range. + """ + return (f * int.from_bytes(siphash24(item, key=key).digest(), 'big')) >> 64 +``` + +### Golomb-Rice Coding + +Instead of writing the items in the hashed set directly to the filter, greater compression is achieved by only writing the differences between successive items in sorted order. Since the items are distributed uniformly, it can be shown that the differences follow a geometric distribution. + +Golomb-Rice Coding is a technique that optimally compresses geometrically distributed values. + +With Golomb-Rice, a value is split into a quotient and remainder modulo $2^P$, which are encoded separately. The quotient $q$ is encoded as unary, with a string of $q$ 1's followed by one 0. The remainder $r$ is represented in big-endian by $P$ bits. + +```python +def golomb_encode(stream: bitarray, x: int, P: int) -> None: + """ + Golomb-encodes a value into a bitarray stream. + + Args: + stream (bitarray): The bitarray to encode into. + x (int): The value to encode. + P (int): The number of bits for the remainder. + """ + + q = x >> P + r = x & (2**P - 1) + + while q > 0: + stream.append(1) + q -= 1 + stream.append(0) + + for i in range(P): + stream.append((r >> (P-1-i)) & 1) + +def golomb_decode(stream: bitarray, offset: int, P: int) -> Tuple[int, int]: + """ + Decodes a Golomb-encoded value from a bitarray stream. + + Args: + stream (bitarray): The bitarray to decode from. + offset (int): The starting offset in the bitarray. + P (int): The number of bits for the remainder. + + Returns: + Tuple[int, int]: The decoded value and the new offset. + """ + q = 0 + while stream[offset] == 1: + q += 1 + offset += 1 + + offset += 1 + + r = 0 + for i in range(P): + r = (r << 1) | stream[offset + i] + + x = (q << P) | r + return x, offset + P +``` + +### Set Construction + +A GCS is constructed from four parameters: + +* `L`, a vector of `N` raw items +* `P`, the bit parameter of the Golomb-Rice coding +* `M`, the inverse of the target false positive rate +* `k`, the 128-bit key used to randomize the SipHash outputs + +The result is a byte vector with a minimum size of $N \cdot (P + 1)$ bits. + +The raw items in $L$ are first hashed to 64-bit unsigned integers as specified above and sorted. The differences between consecutive values, hereafter referred to as deltas, are encoded sequentially to a bit stream with Golomb-Rice coding. Finally, the bit stream is padded with 0's to the nearest byte boundary and serialized to the output byte vector. + +```python +def create(cls, + items: List[bytes], + p: int = 19, + m: int = 784931, + key: bytes = 16 * b'\x00' +) -> bytes: + """ + Turns a list of entries into a Golomb-Coded Set of hashes. + + Args: + items (List[bytes]): The list of items to encode. + p (int): The number of bits for the remainder. + m (int): The inverse of the false positive rate. + key (bytes): The key used for hashing. + + Returns: + bytes: The Golomb-Coded Set as a byte array. + """ + if m.bit_length() > 32: + raise Exception("GCS Error: m parameter must be smaller than 2^32") + if len(items).bit_length() > 32: + raise Exception("GCS Error: number of elements must be smaller than 2^32") + + set_items = create_hashed_set(items, key, m) + + # Sort in non-descending order + sorted_set_items = sorted(set_items) + + output_stream = bitarray() + + last_value = 0 + for item in sorted_set_items: + delta = item - last_value + golomb_encode(output_stream, delta, p) + last_value = item + + # Padded to the next byte boundary + return output_stream.tobytes() +``` + +### Set Querying/Decompression + +To check membership of an item in a compressed GCS, one must reconstruct the hashed set members from the encoded deltas. The procedure to do so is the reverse of the compression: deltas are decoded one by one and added to a cumulative sum. Each intermediate sum represents a hashed value in the original set. The queried item is hashed in the same way as the set members and compared against the reconstructed values. + +> [!Note] +> Querying does not require the entire decompressed set be held in memory at once. + +```python +def match_many( + compressed_set: bytes, + targets: List[bytes], + n: int, + p: int = 19, + m: int = 784931, + key: bytes = 16 * b'\x00', + ) -> Dict[bytes, bool]: + """ + Matches multiple target items against a Golomb-Coded Set. + + Args: + compressed_set (bytes): The Golomb-Coded Set as a byte array. + targets (List[bytes]): The list of target items to match. + n (int): The number of items in the set. + p (int): The number of bits for the remainder. + m (int): The inverse of the false positive rate. + key (bytes): The key used for hashing. + + Returns: + Dict[bytes, bool]: A dictionary indicating which targets are in the set. + """ + if m.bit_length() > 32: + raise Exception("GCS Error: m parameter must be smaller than 2^32") + if n.bit_length() > 32: + raise Exception("GCS Error: number of elements must be smaller than 2^32") + + # Compute the range + f = n * m + + if len(set(targets)) != len(targets): + raise Exception("GCS Error: match targets are not unique entries") + + # Map targets to the same range as the set hashes. + target_hashes: Dict[int, Tuple[bytes, bool]] = {hash_to_range(target, f, key): (target, False) for target in targets} + + input_stream = bitarray() + input_stream.frombytes(compressed_set) + + value = 0 + offset = 0 + for i in range(n): + delta, offset = golomb_decode(input_stream, offset, p) + value += delta + + if value in target_hashes: + target, _ = target_hashes[value] + target_hashes[value] = (target, True) + + return {target: truth_value for target, truth_value in target_hashes.values()} +``` + +## Querying for Spent Ecash Notes + +Users **MAY** -on necessity- query `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. + +The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. + +```json +{ + n: , + p: , + m: , + k: , + content: [], + timestamp: , +} +``` + +Where: +- `n` is the number of items in the filter +- `p` is the bit parameter of the Golomb-Rice coding. If `null`, then the client assumes `p = 19`. +- `m` is the inverse of the false positive rate. If `null`, then the client assume `m = 784931`. +- `k` is the key for the SipHash function. If `null`, then the client assumes `k = "00000000000000000000000000000000"`. + + +`content` is a base-64 string encoding the bytes of the filter. It is typically computed as: +```python +content = b64encode(filter_bytes).decode() +``` + +`timestamp` is the Unix epoch (in seconds) when the filter was created. The Mint might choose not to re-compute the filter upon every request, and instead serve one from cache memory and computed an updated version after arbitrary amount of time. + +## Mint Info Settings + +Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetInfoResponse`. They can do so by updating the response to signal support for NUT-23. + +```json +{ + ..., + "nuts": { + ..., + "23": {"supported": true} + } +} +``` + diff --git a/tests/23-test.md b/tests/23-test.md new file mode 100644 index 00000000..194e4f24 --- /dev/null +++ b/tests/23-test.md @@ -0,0 +1,23 @@ +# NUT-23 Test Vectors + +The following list of items should encode to the target filter `5Ud5NvCtCqXvPaQZe9e6VWmfgAgUdgvVh/A=`, with parameters: +* `p = 19` +* `m = 784931` +* `k = "00000000000000000000000000000000"` + +```json +[ + "c2735796c1d45c68e7f03d3ea3bfcf5d6f10e6eb480e57fc3dccaf8ce66990dfc5", + "3c7ac2a233f8d5439be8cf3109d314e7da476e1ca762dc05f64ca3d5acac2da1fa", + "73e199a811db202ef7fbb1699b0e4859d15735c8f7f838fd9e50b37dc47c0ff4b9", + "02f171db2b577f6d586580651da4951c2e1506454bb9b76077d7a9fdb8606cf2f6", + "106954852453d217ad91e3b14c37bcb6adf62b038cc6a6a281f63edf78de2c7819", + "621e006de8d41b14491933e695985a730179003846b739224316af578fc49c1ee8", + "59b759ecda3c4d9027b9fe549fe6ae33b1bf573b9e9c2d0cdf17d20ea38794f1b7", + "cfcc8745503e9efb67e48b0bee006f6433dec534130707ac23ed4eae911d60eec2", + "f1d57d98f80e528af885e6174f7cd0ef39c31f8436c66b8f27c848a3497c9a7dfb", + "5a21aa11ccd643042f3fe3f0fcc02ccfb51c72419c5eab64a3565aa8499aa64cdf" +] +``` + +Matching every given item using `match_many` should return `True` for every item, while matching any other item should return `False`. \ No newline at end of file From 9ed5126b3489837e840aef9b2721c4bf96432f3e Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 14 May 2025 15:52:50 +0200 Subject: [PATCH 02/25] decode b64 --- 23.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/23.md b/23.md index 89bb0767..f8e52f86 100644 --- a/23.md +++ b/23.md @@ -259,6 +259,8 @@ Where: `content` is a base-64 string encoding the bytes of the filter. It is typically computed as: ```python content = b64encode(filter_bytes).decode() +# And vice-versa +filter_bytes = b64decode(content) ``` `timestamp` is the Unix epoch (in seconds) when the filter was created. The Mint might choose not to re-compute the filter upon every request, and instead serve one from cache memory and computed an updated version after arbitrary amount of time. From a29a9b3dcbe812f8e47be313a19c2615100dd22f Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 14 May 2025 16:08:37 +0200 Subject: [PATCH 03/25] fix note about siphash --- 23.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/23.md b/23.md index f8e52f86..63032edb 100644 --- a/23.md +++ b/23.md @@ -42,7 +42,7 @@ The items are first passed through the pseudorandom function SipHash, which take The 64-bit SipHash outputs are then mapped uniformly over the desired range by multiplying with `F` and taking the top 64 bits of the 128-bit result. This algorithm is a faster alternative to modulo reduction, as it avoids the expensive division operation while still retaining a fair mapping property. > [!NOTE] -> $F \cdot \text{SipHash}(x) \mod 2^{64} \neq \text{SipHash}(x) \mod F$. +> $\lfloor\frac{F \cdot \text{SipHash}(x)}{2^{64}}\rfloor \neq \text{SipHash}(x) \bmod F$. > Make sure your implementation does this right! ```python From 3a5a822d3b17cb13b72215db7bc562200389b878 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 14 May 2025 18:43:47 +0200 Subject: [PATCH 04/25] prettier --- 23.md | 31 +++++++++++++++---------------- tests/23-test.md | 29 +++++++++++++++-------------- 2 files changed, 30 insertions(+), 30 deletions(-) diff --git a/23.md b/23.md index 63032edb..3f9e4239 100644 --- a/23.md +++ b/23.md @@ -35,14 +35,13 @@ The first step in the filter construction is hashing the variable-sized raw item M$. Analysis has shown that `M` and `P` are necessarily bound by $M = 1.497137 \cdot 2^P$. Practically, $M$ can be set to $2^P$ or $2^{P+1}$. Set membership queries against the hash outputs will have a false positive rate of $\frac{1}{M}$. -To avoid integer overflow, the number of items `N` **MUST** be <2^32 and `M` **MUST** be <2^32. +To avoid integer overflow, the number of items `N` **MUST** be <2^32 and `M` **MUST** be <2^32. The items are first passed through the pseudorandom function SipHash, which takes a 128-bit key `k` and a variable-sized byte vector and produces a uniformly random 64-bit output. Implementations of this NUT **MUST** use the SipHash parameters `c = 2` and `d = 4`. The 64-bit SipHash outputs are then mapped uniformly over the desired range by multiplying with `F` and taking the top 64 bits of the 128-bit result. This algorithm is a faster alternative to modulo reduction, as it avoids the expensive division operation while still retaining a fair mapping property. -> [!NOTE] -> $\lfloor\frac{F \cdot \text{SipHash}(x)}{2^{64}}\rfloor \neq \text{SipHash}(x) \bmod F$. +> [!NOTE] > $\lfloor\frac{F \cdot \text{SipHash}(x)}{2^{64}}\rfloor \neq \text{SipHash}(x) \bmod F$. > Make sure your implementation does this right! ```python @@ -109,7 +108,7 @@ def golomb_decode(stream: bitarray, offset: int, P: int) -> Tuple[int, int]: offset += 1 offset += 1 - + r = 0 for i in range(P): r = (r << 1) | stream[offset + i] @@ -122,14 +121,14 @@ def golomb_decode(stream: bitarray, offset: int, P: int) -> Tuple[int, int]: A GCS is constructed from four parameters: -* `L`, a vector of `N` raw items -* `P`, the bit parameter of the Golomb-Rice coding -* `M`, the inverse of the target false positive rate -* `k`, the 128-bit key used to randomize the SipHash outputs +- `L`, a vector of `N` raw items +- `P`, the bit parameter of the Golomb-Rice coding +- `M`, the inverse of the target false positive rate +- `k`, the 128-bit key used to randomize the SipHash outputs The result is a byte vector with a minimum size of $N \cdot (P + 1)$ bits. -The raw items in $L$ are first hashed to 64-bit unsigned integers as specified above and sorted. The differences between consecutive values, hereafter referred to as deltas, are encoded sequentially to a bit stream with Golomb-Rice coding. Finally, the bit stream is padded with 0's to the nearest byte boundary and serialized to the output byte vector. +The raw items in $L$ are first hashed to 64-bit unsigned integers as specified above and sorted. The differences between consecutive values, hereafter referred to as deltas, are encoded sequentially to a bit stream with Golomb-Rice coding. Finally, the bit stream is padded with 0's to the nearest byte boundary and serialized to the output byte vector. ```python def create(cls, @@ -154,7 +153,7 @@ def create(cls, raise Exception("GCS Error: m parameter must be smaller than 2^32") if len(items).bit_length() > 32: raise Exception("GCS Error: number of elements must be smaller than 2^32") - + set_items = create_hashed_set(items, key, m) # Sort in non-descending order @@ -174,10 +173,10 @@ def create(cls, ### Set Querying/Decompression -To check membership of an item in a compressed GCS, one must reconstruct the hashed set members from the encoded deltas. The procedure to do so is the reverse of the compression: deltas are decoded one by one and added to a cumulative sum. Each intermediate sum represents a hashed value in the original set. The queried item is hashed in the same way as the set members and compared against the reconstructed values. +To check membership of an item in a compressed GCS, one must reconstruct the hashed set members from the encoded deltas. The procedure to do so is the reverse of the compression: deltas are decoded one by one and added to a cumulative sum. Each intermediate sum represents a hashed value in the original set. The queried item is hashed in the same way as the set members and compared against the reconstructed values. > [!Note] -> Querying does not require the entire decompressed set be held in memory at once. +> Querying does not require the entire decompressed set be held in memory at once. ```python def match_many( @@ -215,7 +214,7 @@ def match_many( # Map targets to the same range as the set hashes. target_hashes: Dict[int, Tuple[bytes, bool]] = {hash_to_range(target, f, key): (target, False) for target in targets} - + input_stream = bitarray() input_stream.frombytes(compressed_set) @@ -234,7 +233,7 @@ def match_many( ## Querying for Spent Ecash Notes -Users **MAY** -on necessity- query `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. +Users **MAY** -on necessity- query `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. @@ -250,13 +249,14 @@ The Mint **MUST** respond with a `GetFilterResponse`, which has the following st ``` Where: + - `n` is the number of items in the filter - `p` is the bit parameter of the Golomb-Rice coding. If `null`, then the client assumes `p = 19`. - `m` is the inverse of the false positive rate. If `null`, then the client assume `m = 784931`. - `k` is the key for the SipHash function. If `null`, then the client assumes `k = "00000000000000000000000000000000"`. - `content` is a base-64 string encoding the bytes of the filter. It is typically computed as: + ```python content = b64encode(filter_bytes).decode() # And vice-versa @@ -278,4 +278,3 @@ Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetIn } } ``` - diff --git a/tests/23-test.md b/tests/23-test.md index 194e4f24..e50ec730 100644 --- a/tests/23-test.md +++ b/tests/23-test.md @@ -1,23 +1,24 @@ # NUT-23 Test Vectors The following list of items should encode to the target filter `5Ud5NvCtCqXvPaQZe9e6VWmfgAgUdgvVh/A=`, with parameters: -* `p = 19` -* `m = 784931` -* `k = "00000000000000000000000000000000"` + +- `p = 19` +- `m = 784931` +- `k = "00000000000000000000000000000000"` ```json [ - "c2735796c1d45c68e7f03d3ea3bfcf5d6f10e6eb480e57fc3dccaf8ce66990dfc5", - "3c7ac2a233f8d5439be8cf3109d314e7da476e1ca762dc05f64ca3d5acac2da1fa", - "73e199a811db202ef7fbb1699b0e4859d15735c8f7f838fd9e50b37dc47c0ff4b9", - "02f171db2b577f6d586580651da4951c2e1506454bb9b76077d7a9fdb8606cf2f6", - "106954852453d217ad91e3b14c37bcb6adf62b038cc6a6a281f63edf78de2c7819", - "621e006de8d41b14491933e695985a730179003846b739224316af578fc49c1ee8", - "59b759ecda3c4d9027b9fe549fe6ae33b1bf573b9e9c2d0cdf17d20ea38794f1b7", - "cfcc8745503e9efb67e48b0bee006f6433dec534130707ac23ed4eae911d60eec2", - "f1d57d98f80e528af885e6174f7cd0ef39c31f8436c66b8f27c848a3497c9a7dfb", - "5a21aa11ccd643042f3fe3f0fcc02ccfb51c72419c5eab64a3565aa8499aa64cdf" + "c2735796c1d45c68e7f03d3ea3bfcf5d6f10e6eb480e57fc3dccaf8ce66990dfc5", + "3c7ac2a233f8d5439be8cf3109d314e7da476e1ca762dc05f64ca3d5acac2da1fa", + "73e199a811db202ef7fbb1699b0e4859d15735c8f7f838fd9e50b37dc47c0ff4b9", + "02f171db2b577f6d586580651da4951c2e1506454bb9b76077d7a9fdb8606cf2f6", + "106954852453d217ad91e3b14c37bcb6adf62b038cc6a6a281f63edf78de2c7819", + "621e006de8d41b14491933e695985a730179003846b739224316af578fc49c1ee8", + "59b759ecda3c4d9027b9fe549fe6ae33b1bf573b9e9c2d0cdf17d20ea38794f1b7", + "cfcc8745503e9efb67e48b0bee006f6433dec534130707ac23ed4eae911d60eec2", + "f1d57d98f80e528af885e6174f7cd0ef39c31f8436c66b8f27c848a3497c9a7dfb", + "5a21aa11ccd643042f3fe3f0fcc02ccfb51c72419c5eab64a3565aa8499aa64cdf" ] ``` -Matching every given item using `match_many` should return `True` for every item, while matching any other item should return `False`. \ No newline at end of file +Matching every given item using `match_many` should return `True` for every item, while matching any other item should return `False`. From a2889594db64e8ad4c0b1a91dec1983e844d4a1f Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 14 May 2025 23:02:26 +0200 Subject: [PATCH 05/25] fix typos --- 23.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/23.md b/23.md index 3f9e4239..0667ae8a 100644 --- a/23.md +++ b/23.md @@ -41,7 +41,7 @@ The items are first passed through the pseudorandom function SipHash, which take The 64-bit SipHash outputs are then mapped uniformly over the desired range by multiplying with `F` and taking the top 64 bits of the 128-bit result. This algorithm is a faster alternative to modulo reduction, as it avoids the expensive division operation while still retaining a fair mapping property. -> [!NOTE] > $\lfloor\frac{F \cdot \text{SipHash}(x)}{2^{64}}\rfloor \neq \text{SipHash}(x) \bmod F$. +> [!NOTE] > $\biggl\lfloor\frac{F \cdot \text{SipHash}(x)}{2^{64}}\biggr\rfloor \neq \text{SipHash}(x) \bmod F$. > Make sure your implementation does this right! ```python @@ -131,7 +131,7 @@ The result is a byte vector with a minimum size of $N \cdot (P + 1)$ bits. The raw items in $L$ are first hashed to 64-bit unsigned integers as specified above and sorted. The differences between consecutive values, hereafter referred to as deltas, are encoded sequentially to a bit stream with Golomb-Rice coding. Finally, the bit stream is padded with 0's to the nearest byte boundary and serialized to the output byte vector. ```python -def create(cls, +def create( items: List[bytes], p: int = 19, m: int = 784931, From 021dda40afe9ab1adea8394d1791667fe2998245 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Fri, 16 May 2025 09:01:23 +0200 Subject: [PATCH 06/25] replace siphash with murmurhash --- 23.md | 25 +++++++++---------------- tests/23-test.md | 3 +-- 2 files changed, 10 insertions(+), 18 deletions(-) diff --git a/23.md b/23.md index 0667ae8a..eebe1f01 100644 --- a/23.md +++ b/23.md @@ -37,27 +37,27 @@ Set membership queries against the hash outputs will have a false positive rate To avoid integer overflow, the number of items `N` **MUST** be <2^32 and `M` **MUST** be <2^32. -The items are first passed through the pseudorandom function SipHash, which takes a 128-bit key `k` and a variable-sized byte vector and produces a uniformly random 64-bit output. Implementations of this NUT **MUST** use the SipHash parameters `c = 2` and `d = 4`. +The items are first passed through the pseudorandom function MurmurHash128, which takes a variable-sized byte vector and produces a uniformly random 128-bit output. -The 64-bit SipHash outputs are then mapped uniformly over the desired range by multiplying with `F` and taking the top 64 bits of the 128-bit result. This algorithm is a faster alternative to modulo reduction, as it avoids the expensive division operation while still retaining a fair mapping property. +The bottom 64-bit of the MurmurHash output is then mapped uniformly over the desired range by multiplying with `F` and taking the top 64 bits of the 128-bit result. This algorithm is a faster alternative to modulo reduction, as it avoids the expensive division operation while still retaining a fair mapping property. -> [!NOTE] > $\biggl\lfloor\frac{F \cdot \text{SipHash}(x)}{2^{64}}\biggr\rfloor \neq \text{SipHash}(x) \bmod F$. +> [!NOTE] > $\biggl\lfloor\frac{F \cdot \text{Hash}(x)}{2^{64}}\biggr\rfloor \neq \text{Hash}(x) \bmod F$. > Make sure your implementation does this right! ```python -def hash_to_range(item: bytes, f: int, key: bytes) -> int: +def hash_to_range(item: bytes, f: int) -> int: """ - Hashes an item to a range using SipHash. + Hashes an item to a range using Murmurhash. Args: item (bytes): The item to hash. f (int): The maximum range value. - key (bytes): The key used for hashing. Returns: int: The hashed value within the specified range. """ - return (f * int.from_bytes(siphash24(item, key=key).digest(), 'big')) >> 64 + h = mmh3.hash128(item) & (2**64 - 1) + return (f * h) >> 64 ``` ### Golomb-Rice Coding @@ -124,7 +124,6 @@ A GCS is constructed from four parameters: - `L`, a vector of `N` raw items - `P`, the bit parameter of the Golomb-Rice coding - `M`, the inverse of the target false positive rate -- `k`, the 128-bit key used to randomize the SipHash outputs The result is a byte vector with a minimum size of $N \cdot (P + 1)$ bits. @@ -135,7 +134,6 @@ def create( items: List[bytes], p: int = 19, m: int = 784931, - key: bytes = 16 * b'\x00' ) -> bytes: """ Turns a list of entries into a Golomb-Coded Set of hashes. @@ -144,7 +142,6 @@ def create( items (List[bytes]): The list of items to encode. p (int): The number of bits for the remainder. m (int): The inverse of the false positive rate. - key (bytes): The key used for hashing. Returns: bytes: The Golomb-Coded Set as a byte array. @@ -154,7 +151,7 @@ def create( if len(items).bit_length() > 32: raise Exception("GCS Error: number of elements must be smaller than 2^32") - set_items = create_hashed_set(items, key, m) + set_items = create_hashed_set(items, m) # Sort in non-descending order sorted_set_items = sorted(set_items) @@ -185,7 +182,6 @@ def match_many( n: int, p: int = 19, m: int = 784931, - key: bytes = 16 * b'\x00', ) -> Dict[bytes, bool]: """ Matches multiple target items against a Golomb-Coded Set. @@ -196,7 +192,6 @@ def match_many( n (int): The number of items in the set. p (int): The number of bits for the remainder. m (int): The inverse of the false positive rate. - key (bytes): The key used for hashing. Returns: Dict[bytes, bool]: A dictionary indicating which targets are in the set. @@ -213,7 +208,7 @@ def match_many( raise Exception("GCS Error: match targets are not unique entries") # Map targets to the same range as the set hashes. - target_hashes: Dict[int, Tuple[bytes, bool]] = {hash_to_range(target, f, key): (target, False) for target in targets} + target_hashes: Dict[int, Tuple[bytes, bool]] = {hash_to_range(target, f): (target, False) for target in targets} input_stream = bitarray() input_stream.frombytes(compressed_set) @@ -242,7 +237,6 @@ The Mint **MUST** respond with a `GetFilterResponse`, which has the following st n: , p: , m: , - k: , content: [], timestamp: , } @@ -253,7 +247,6 @@ Where: - `n` is the number of items in the filter - `p` is the bit parameter of the Golomb-Rice coding. If `null`, then the client assumes `p = 19`. - `m` is the inverse of the false positive rate. If `null`, then the client assume `m = 784931`. -- `k` is the key for the SipHash function. If `null`, then the client assumes `k = "00000000000000000000000000000000"`. `content` is a base-64 string encoding the bytes of the filter. It is typically computed as: diff --git a/tests/23-test.md b/tests/23-test.md index e50ec730..4d1b1b4c 100644 --- a/tests/23-test.md +++ b/tests/23-test.md @@ -1,10 +1,9 @@ # NUT-23 Test Vectors -The following list of items should encode to the target filter `5Ud5NvCtCqXvPaQZe9e6VWmfgAgUdgvVh/A=`, with parameters: +The following list of items should encode to the target filter `z4fUCDVqdnxWR7Y9+YdT5o0IC9GxiSA2BGyg`, with parameters: - `p = 19` - `m = 784931` -- `k = "00000000000000000000000000000000"` ```json [ From a70b39c0bf77b1a7e0060249b57f6a10e02ecea2 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Fri, 16 May 2025 13:35:49 +0200 Subject: [PATCH 07/25] bulk fpr --- 23.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/23.md b/23.md index eebe1f01..fa3c7d09 100644 --- a/23.md +++ b/23.md @@ -228,7 +228,7 @@ def match_many( ## Querying for Spent Ecash Notes -Users **MAY** -on necessity- query `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. +Wallets **MAY** query `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. @@ -258,6 +258,21 @@ filter_bytes = b64decode(content) `timestamp` is the Unix epoch (in seconds) when the filter was created. The Mint might choose not to re-compute the filter upon every request, and instead serve one from cache memory and computed an updated version after arbitrary amount of time. +### False Positive Rate (FPR) For Bulk Tests + +Each invidivual look-up has $1 - \frac{1}{M}$ chance of being a *true positive*. We can consider a bulk test as $x$ independent look-ups, so the chance that at *all* of them are true positives is $(1 - \frac{1}{M})^x$. + +Therefore, the chance of any one of them being a *false positive* (or equivalently, **not** all of them being *true positives*) is $1 - (1 - \frac{1}{M})^x$. + +For $M = 784931$, this turns out to be: + +| $x$ | $M$ | $P_M(x)$ | +|----------|------------------|---------------------| +| 1 | 784931 | 0.999998726 | +| 10 | 784931 | 0.99998726 | +| 300 | 784931 | 0.999617874 | +| 5000 | 784931 | 0.993650255 | + ## Mint Info Settings Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetInfoResponse`. They can do so by updating the response to signal support for NUT-23. From accde76b91b3de1c0e7ac76a0ebec40f04fe6f56 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Fri, 16 May 2025 14:32:05 +0200 Subject: [PATCH 08/25] prettier --- 23.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/23.md b/23.md index fa3c7d09..a1e37a9c 100644 --- a/23.md +++ b/23.md @@ -260,17 +260,17 @@ filter_bytes = b64decode(content) ### False Positive Rate (FPR) For Bulk Tests -Each invidivual look-up has $1 - \frac{1}{M}$ chance of being a *true positive*. We can consider a bulk test as $x$ independent look-ups, so the chance that at *all* of them are true positives is $(1 - \frac{1}{M})^x$. +Each invidivual look-up has $1 - \frac{1}{M}$ chance of being a _true positive_. We can consider a bulk test as $x$ independent look-ups, so the chance that at _all_ of them are true positives is $(1 - \frac{1}{M})^x$. -Therefore, the chance of any one of them being a *false positive* (or equivalently, **not** all of them being *true positives*) is $1 - (1 - \frac{1}{M})^x$. +Therefore, the chance of any one of them being a _false positive_ (or equivalently, **not** all of them being _true positives_) is $1 - (1 - \frac{1}{M})^x$. For $M = 784931$, this turns out to be: -| $x$ | $M$ | $P_M(x)$ | -|----------|------------------|---------------------| -| 1 | 784931 | 0.999998726 | -| 10 | 784931 | 0.99998726 | -| 300 | 784931 | 0.999617874 | +| $x$ | $M$ | $P_M(x)$ | +| ---- | ------ | ----------- | +| 1 | 784931 | 0.999998726 | +| 10 | 784931 | 0.99998726 | +| 300 | 784931 | 0.999617874 | | 5000 | 784931 | 0.993650255 | ## Mint Info Settings From 694fa6c5a3350e0d5220e2602c657ab95d3f21ea Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Fri, 16 May 2025 14:34:44 +0200 Subject: [PATCH 09/25] big parenthesis --- 23.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/23.md b/23.md index a1e37a9c..7788f28c 100644 --- a/23.md +++ b/23.md @@ -260,9 +260,9 @@ filter_bytes = b64decode(content) ### False Positive Rate (FPR) For Bulk Tests -Each invidivual look-up has $1 - \frac{1}{M}$ chance of being a _true positive_. We can consider a bulk test as $x$ independent look-ups, so the chance that at _all_ of them are true positives is $(1 - \frac{1}{M})^x$. +Each invidivual look-up has $1 - \frac{1}{M}$ chance of being a _true positive_. We can consider a bulk test as $x$ independent look-ups, so the chance that at _all_ of them are true positives is $\bigl(1 - \frac{1}{M}\bigr)^x$. -Therefore, the chance of any one of them being a _false positive_ (or equivalently, **not** all of them being _true positives_) is $1 - (1 - \frac{1}{M})^x$. +Therefore, the chance of any one of them being a _false positive_ (or equivalently, **not** all of them being _true positives_) is $1 - \bigl(1 - \frac{1}{M}\bigr)^x$. For $M = 784931$, this turns out to be: From 28a8ebbafeada1129b6463e3a6a3585df545daec Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 21 May 2025 12:01:50 +0200 Subject: [PATCH 10/25] endpoint for issued blind signatures filter. --- 23.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/23.md b/23.md index 7788f28c..8f701abb 100644 --- a/23.md +++ b/23.md @@ -228,7 +228,9 @@ def match_many( ## Querying for Spent Ecash Notes -Wallets **MAY** query `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. +Wallets **MAY** query the following endpoints: + * `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. + * `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` values of all the blind signatures from `keyset_id`. The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. From 5a8db78ec4cf5be7d95a4811b3de889c70b211fa Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 21 May 2025 12:33:09 +0200 Subject: [PATCH 11/25] prettier --- 23.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/23.md b/23.md index 8f701abb..b631c843 100644 --- a/23.md +++ b/23.md @@ -229,8 +229,9 @@ def match_many( ## Querying for Spent Ecash Notes Wallets **MAY** query the following endpoints: - * `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. - * `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` values of all the blind signatures from `keyset_id`. + +- `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. +- `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` values of all the blind signatures from `keyset_id`. The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. From 4c029cd87ecdba58e7d69e4a607969bb369cca6b Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Mon, 26 May 2025 14:26:50 +0200 Subject: [PATCH 12/25] remove issued filter --- 23.md | 1 - 1 file changed, 1 deletion(-) diff --git a/23.md b/23.md index b631c843..359366e5 100644 --- a/23.md +++ b/23.md @@ -231,7 +231,6 @@ def match_many( Wallets **MAY** query the following endpoints: - `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. -- `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` values of all the blind signatures from `keyset_id`. The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. From feadaa817e72fbbf27700f3c12433479b951f637 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Mon, 26 May 2025 14:28:31 +0200 Subject: [PATCH 13/25] rename to 24 --- 23.md => 24.md | 6 +++--- tests/{23-test.md => 24-test.md} | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) rename 23.md => 24.md (99%) rename tests/{23-test.md => 24-test.md} (97%) diff --git a/23.md b/24.md similarity index 99% rename from 23.md rename to 24.md index 359366e5..67d3793c 100644 --- a/23.md +++ b/24.md @@ -1,4 +1,4 @@ -# NUT-23: Golomb-Coded Set Filters +# NUT-24: Golomb-Coded Set Filters `optional` @@ -277,14 +277,14 @@ For $M = 784931$, this turns out to be: ## Mint Info Settings -Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetInfoResponse`. They can do so by updating the response to signal support for NUT-23. +Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetInfoResponse`. They can do so by updating the response to signal support for NUT-XX. ```json { ..., "nuts": { ..., - "23": {"supported": true} + "XX": {"supported": true} } } ``` diff --git a/tests/23-test.md b/tests/24-test.md similarity index 97% rename from tests/23-test.md rename to tests/24-test.md index 4d1b1b4c..56712efe 100644 --- a/tests/23-test.md +++ b/tests/24-test.md @@ -1,4 +1,4 @@ -# NUT-23 Test Vectors +# NUT-XX Test Vectors The following list of items should encode to the target filter `z4fUCDVqdnxWR7Y9+YdT5o0IC9GxiSA2BGyg`, with parameters: From 3643e326ef32b71f97ee5f3061c5a479d570b873 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 25 Jun 2025 16:59:56 +0200 Subject: [PATCH 14/25] update: include a `issued` filter --- 24.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/24.md b/24.md index 67d3793c..7d6b694e 100644 --- a/24.md +++ b/24.md @@ -230,7 +230,8 @@ def match_many( Wallets **MAY** query the following endpoints: -- `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` values of all the spent ecash from `keyset_id`. +- `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` (nullifiers) values of all the spent ecash from `keyset_id`. +- `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` (blinded_messages) values of all the issued ecash from `keyset_id`. The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. From bf4bbb638de9f9044926d0a115ee798257014658 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Wed, 25 Jun 2025 17:00:54 +0200 Subject: [PATCH 15/25] fix --- 24.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/24.md b/24.md index 7d6b694e..7a84a496 100644 --- a/24.md +++ b/24.md @@ -226,7 +226,7 @@ def match_many( return {target: truth_value for target, truth_value in target_hashes.values()} ``` -## Querying for Spent Ecash Notes +## Querying for Spent or Issued Ecash Notes Wallets **MAY** query the following endpoints: From 3ecb15479cf99b2099da0c5d0adf84178e765553 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Sat, 5 Jul 2025 11:27:08 +0200 Subject: [PATCH 16/25] docs: Refactor NUT-24 to focus on GCS filter specification and recovery flow Co-authored-by: aider (openrouter/google/gemini-2.5-flash) --- 24.md | 283 +++++++++++----------------------------------------------- 1 file changed, 50 insertions(+), 233 deletions(-) diff --git a/24.md b/24.md index 7a84a496..181f1691 100644 --- a/24.md +++ b/24.md @@ -6,274 +6,91 @@ --- -This NUT describes a structure for compact filters on ecash notes and other information, for client-side use. The filter construction proposed -is an alternative to Bloom filters that minimizes size by using Golomb-Rice coding for compression. +This NUT describes a structure for compact filters on ecash notes and other information, for client-side use, primarily for recovery flows. The filter construction proposed uses Golomb-Coded Sets (GCS) for efficient compression. -Clients can query and use this filter to test for set membership, which comes in particularly handy for checking the spent state of ecash notes. +Clients can query and use these filters to test for set membership, which is particularly useful for checking the spent state of ecash notes without revealing which notes are being checked, and for checking if blind signatures have been issued without revealing the specific blind signatures. ## Specification -### Filter +### Golomb-Coded Set (GCS) Filters -For each keyset, compact filters are derived containing sets of items associated with the keyset (spent `Y`s or blind signatures `B_`). -A set of such data objects is compressed into a probabilistic structure called a Golomb-Coded set (GCS), which matches all items in the set with -probability $1$, and matches other items with probability $\frac{1}{M}$. The encoding is also parameterized by `P`, the bit length of the remainder code. -Each filter defined specifies values for `P` and `M`, as well as the cardinality of the set it encodes `N`. +A Golomb-Coded Set (GCS) is a probabilistic data structure that allows for compact representation of a set of items. It enables checking for set membership with a certain false positive rate, but no false negatives. GCS filters are constructed by hashing items to a range, sorting the hashed values, and then encoding the differences between successive values using Golomb-Rice coding. This method provides significant compression, making them suitable for efficient transmission and client-side processing. -At a high level, a GCS is constructed from a set of $N$ items by: +Each filter is defined by: +- `N`: The cardinality of the set it encodes. +- `P`: The bit length of the remainder code in Golomb-Rice coding. +- `M`: The inverse of the target false positive rate. -1. hashing all items to 64-bit integers in the range $[0, N\cdot M)$ -2. sorting the hashed values in ascending order -3. computing the differences between each value and the previous one -4. writing the differences sequentially, compressed with Golomb-Rice coding +### Mint Responsibilities -The following sections describe each step in greater detail. +Mints **MUST** generate GCS filters containing sets of items associated with specific keysets. These filters are generated at self-determined intervals. The mint is responsible for ensuring the filters are available for clients to query. -### Hashing Data Objects +For each keyset, mints generate: +- A filter encoding the `Y` (nullifiers) values of all spent ecash notes. +- A filter encoding the `B_` (blinded_messages) values of all issued ecash notes. -The first step in the filter construction is hashing the variable-sized raw items in the set to the range $[0, F)$, where $F = N \cdot -M$. Analysis has shown that `M` and `P` are necessarily bound by $M = 1.497137 \cdot 2^P$. Practically, $M$ can be set to $2^P$ or $2^{P+1}$. -Set membership queries against the hash outputs will have a false positive rate of $\frac{1}{M}$. +### Wallet Behavior and Recovery Flow -To avoid integer overflow, the number of items `N` **MUST** be <2^32 and `M` **MUST** be <2^32. +Wallets utilize GCS filters during recovery to efficiently determine the status of ecash notes and blind signatures without leaking sensitive information. -The items are first passed through the pseudorandom function MurmurHash128, which takes a variable-sized byte vector and produces a uniformly random 128-bit output. +**Restore Flow for Spent Ecash Notes:** +1. The wallet identifies the `keyset_id` for which it needs to check spent notes. +2. The wallet queries the Mint's `GET v1/filter/spent/{keyset_id}` endpoint to retrieve the GCS filter for spent nullifiers. +3. Upon receiving the `GetFilterResponse`, the wallet extracts the `content` (the GCS filter bytes), `n`, `p`, and `m` parameters. +4. For each ecash note the wallet possesses for that `keyset_id`, it computes the nullifier `Y`. +5. The wallet then queries the GCS filter with its `Y` values. If a `Y` value matches in the filter, the note is considered spent. Due to the probabilistic nature of GCS, a false positive is possible, meaning a note might be marked as spent when it is not. Wallets **SHOULD** handle this by attempting to spend such notes and gracefully handling a `Spent` error from the mint. -The bottom 64-bit of the MurmurHash output is then mapped uniformly over the desired range by multiplying with `F` and taking the top 64 bits of the 128-bit result. This algorithm is a faster alternative to modulo reduction, as it avoids the expensive division operation while still retaining a fair mapping property. +**Restore Flow for Issued Blind Signatures:** +1. The wallet identifies the `keyset_id` for which it needs to check issued blind signatures. +2. The wallet queries the Mint's `GET v1/filter/issued/{keyset_id}` endpoint to retrieve the GCS filter for issued blind signatures. +3. Upon receiving the `GetFilterResponse`, the wallet extracts the `content` (the GCS filter bytes), `n`, `p`, and `m` parameters. +4. For each `blinded_message` (`B_`) the wallet has submitted for signing for that `keyset_id`, it queries the GCS filter with its `B_` value. If a `B_` value matches in the filter, the blind signature is considered issued. Similar to spent notes, false positives are possible. Wallets **SHOULD** handle this by attempting to retrieve the signed blind signature and gracefully handling errors. -> [!NOTE] > $\biggl\lfloor\frac{F \cdot \text{Hash}(x)}{2^{64}}\biggr\rfloor \neq \text{Hash}(x) \bmod F$. -> Make sure your implementation does this right! - -```python -def hash_to_range(item: bytes, f: int) -> int: - """ - Hashes an item to a range using Murmurhash. - - Args: - item (bytes): The item to hash. - f (int): The maximum range value. - - Returns: - int: The hashed value within the specified range. - """ - h = mmh3.hash128(item) & (2**64 - 1) - return (f * h) >> 64 -``` - -### Golomb-Rice Coding - -Instead of writing the items in the hashed set directly to the filter, greater compression is achieved by only writing the differences between successive items in sorted order. Since the items are distributed uniformly, it can be shown that the differences follow a geometric distribution. - -Golomb-Rice Coding is a technique that optimally compresses geometrically distributed values. - -With Golomb-Rice, a value is split into a quotient and remainder modulo $2^P$, which are encoded separately. The quotient $q$ is encoded as unary, with a string of $q$ 1's followed by one 0. The remainder $r$ is represented in big-endian by $P$ bits. - -```python -def golomb_encode(stream: bitarray, x: int, P: int) -> None: - """ - Golomb-encodes a value into a bitarray stream. - - Args: - stream (bitarray): The bitarray to encode into. - x (int): The value to encode. - P (int): The number of bits for the remainder. - """ - - q = x >> P - r = x & (2**P - 1) - - while q > 0: - stream.append(1) - q -= 1 - stream.append(0) - - for i in range(P): - stream.append((r >> (P-1-i)) & 1) - -def golomb_decode(stream: bitarray, offset: int, P: int) -> Tuple[int, int]: - """ - Decodes a Golomb-encoded value from a bitarray stream. - - Args: - stream (bitarray): The bitarray to decode from. - offset (int): The starting offset in the bitarray. - P (int): The number of bits for the remainder. - - Returns: - Tuple[int, int]: The decoded value and the new offset. - """ - q = 0 - while stream[offset] == 1: - q += 1 - offset += 1 - - offset += 1 - - r = 0 - for i in range(P): - r = (r << 1) | stream[offset + i] - - x = (q << P) | r - return x, offset + P -``` - -### Set Construction - -A GCS is constructed from four parameters: - -- `L`, a vector of `N` raw items -- `P`, the bit parameter of the Golomb-Rice coding -- `M`, the inverse of the target false positive rate - -The result is a byte vector with a minimum size of $N \cdot (P + 1)$ bits. - -The raw items in $L$ are first hashed to 64-bit unsigned integers as specified above and sorted. The differences between consecutive values, hereafter referred to as deltas, are encoded sequentially to a bit stream with Golomb-Rice coding. Finally, the bit stream is padded with 0's to the nearest byte boundary and serialized to the output byte vector. - -```python -def create( - items: List[bytes], - p: int = 19, - m: int = 784931, -) -> bytes: - """ - Turns a list of entries into a Golomb-Coded Set of hashes. - - Args: - items (List[bytes]): The list of items to encode. - p (int): The number of bits for the remainder. - m (int): The inverse of the false positive rate. - - Returns: - bytes: The Golomb-Coded Set as a byte array. - """ - if m.bit_length() > 32: - raise Exception("GCS Error: m parameter must be smaller than 2^32") - if len(items).bit_length() > 32: - raise Exception("GCS Error: number of elements must be smaller than 2^32") - - set_items = create_hashed_set(items, m) - - # Sort in non-descending order - sorted_set_items = sorted(set_items) - - output_stream = bitarray() - - last_value = 0 - for item in sorted_set_items: - delta = item - last_value - golomb_encode(output_stream, delta, p) - last_value = item - - # Padded to the next byte boundary - return output_stream.tobytes() -``` - -### Set Querying/Decompression - -To check membership of an item in a compressed GCS, one must reconstruct the hashed set members from the encoded deltas. The procedure to do so is the reverse of the compression: deltas are decoded one by one and added to a cumulative sum. Each intermediate sum represents a hashed value in the original set. The queried item is hashed in the same way as the set members and compared against the reconstructed values. - -> [!Note] -> Querying does not require the entire decompressed set be held in memory at once. - -```python -def match_many( - compressed_set: bytes, - targets: List[bytes], - n: int, - p: int = 19, - m: int = 784931, - ) -> Dict[bytes, bool]: - """ - Matches multiple target items against a Golomb-Coded Set. - - Args: - compressed_set (bytes): The Golomb-Coded Set as a byte array. - targets (List[bytes]): The list of target items to match. - n (int): The number of items in the set. - p (int): The number of bits for the remainder. - m (int): The inverse of the false positive rate. - - Returns: - Dict[bytes, bool]: A dictionary indicating which targets are in the set. - """ - if m.bit_length() > 32: - raise Exception("GCS Error: m parameter must be smaller than 2^32") - if n.bit_length() > 32: - raise Exception("GCS Error: number of elements must be smaller than 2^32") - - # Compute the range - f = n * m - - if len(set(targets)) != len(targets): - raise Exception("GCS Error: match targets are not unique entries") - - # Map targets to the same range as the set hashes. - target_hashes: Dict[int, Tuple[bytes, bool]] = {hash_to_range(target, f): (target, False) for target in targets} - - input_stream = bitarray() - input_stream.frombytes(compressed_set) - - value = 0 - offset = 0 - for i in range(n): - delta, offset = golomb_decode(input_stream, offset, p) - value += delta - - if value in target_hashes: - target, _ = target_hashes[value] - target_hashes[value] = (target, True) - - return {target: truth_value for target, truth_value in target_hashes.values()} -``` - -## Querying for Spent or Issued Ecash Notes +### Querying for Spent or Issued Ecash Notes Wallets **MAY** query the following endpoints: -- `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` (nullifiers) values of all the spent ecash from `keyset_id`. -- `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` (blinded_messages) values of all the issued ecash from `keyset_id`. +- `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` (nullifiers) values of all the spent ecash from `keyset_et_id`. +- `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` (blinded_messages) values of all the issued ecash from `keyset_id`. The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. ```json { - n: , - p: , - m: , - content: [], - timestamp: , + "n": "", + "p": "", + "m": "", + "content": "", + "timestamp": "" } ``` Where: -- `n` is the number of items in the filter -- `p` is the bit parameter of the Golomb-Rice coding. If `null`, then the client assumes `p = 19`. -- `m` is the inverse of the false positive rate. If `null`, then the client assume `m = 784931`. - -`content` is a base-64 string encoding the bytes of the filter. It is typically computed as: - -```python -content = b64encode(filter_bytes).decode() -# And vice-versa -filter_bytes = b64decode(content) -``` - -`timestamp` is the Unix epoch (in seconds) when the filter was created. The Mint might choose not to re-compute the filter upon every request, and instead serve one from cache memory and computed an updated version after arbitrary amount of time. +- `n` is the number of items in the filter. +- `p` is the bit parameter of the Golomb-Rice coding. If `null`, then the client assumes `p = 19`. +- `m` is the inverse of the false positive rate. If `null`, then the client assumes `m = 784931`. +- `content` is a base-64 string encoding the bytes of the filter. It is typically computed as: + ```python + content = b64encode(filter_bytes).decode() + # And vice-versa + filter_bytes = b64decode(content) + ``` +- `timestamp` is the Unix epoch (in seconds) when the filter was created. The Mint might choose not to re-compute the filter upon every request, and instead serve one from cache memory and computed an updated version after arbitrary amount of time. ### False Positive Rate (FPR) For Bulk Tests -Each invidivual look-up has $1 - \frac{1}{M}$ chance of being a _true positive_. We can consider a bulk test as $x$ independent look-ups, so the chance that at _all_ of them are true positives is $\bigl(1 - \frac{1}{M}\bigr)^x$. +Each individual look-up has $1 - \frac{1}{M}$ chance of being a _true positive_. We can consider a bulk test as $x$ independent look-ups, so the chance that at _all_ of them are true positives is $\bigl(1 - \frac{1}{M}\bigr)^x$. Therefore, the chance of any one of them being a _false positive_ (or equivalently, **not** all of them being _true positives_) is $1 - \bigl(1 - \frac{1}{M}\bigr)^x$. For $M = 784931$, this turns out to be: -| $x$ | $M$ | $P_M(x)$ | -| ---- | ------ | ----------- | -| 1 | 784931 | 0.999998726 | -| 10 | 784931 | 0.99998726 | -| 300 | 784931 | 0.999617874 | +| $x$ | $M$ | $P_M(x)$ | +| --- | --- | -------- | +| 1 | 784931 | 0.999998726 | +| 10 | 784931 | 0.99998726 | +| 300 | 784931 | 0.999617874 | | 5000 | 784931 | 0.993650255 | ## Mint Info Settings From 0c4db586fd028a456633e6c061ec607c5cc46c64 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Sat, 5 Jul 2025 11:43:22 +0200 Subject: [PATCH 17/25] docs: Clarify GCS filter behavior and add NUT-13 reference --- 24.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/24.md b/24.md index 181f1691..21fc8d09 100644 --- a/24.md +++ b/24.md @@ -31,20 +31,20 @@ For each keyset, mints generate: ### Wallet Behavior and Recovery Flow -Wallets utilize GCS filters during recovery to efficiently determine the status of ecash notes and blind signatures without leaking sensitive information. +Wallets utilize GCS filters during recovery to determine the status of ecash notes and blind signatures leaking as little sensitive information as possible. **Restore Flow for Spent Ecash Notes:** 1. The wallet identifies the `keyset_id` for which it needs to check spent notes. 2. The wallet queries the Mint's `GET v1/filter/spent/{keyset_id}` endpoint to retrieve the GCS filter for spent nullifiers. 3. Upon receiving the `GetFilterResponse`, the wallet extracts the `content` (the GCS filter bytes), `n`, `p`, and `m` parameters. 4. For each ecash note the wallet possesses for that `keyset_id`, it computes the nullifier `Y`. -5. The wallet then queries the GCS filter with its `Y` values. If a `Y` value matches in the filter, the note is considered spent. Due to the probabilistic nature of GCS, a false positive is possible, meaning a note might be marked as spent when it is not. Wallets **SHOULD** handle this by attempting to spend such notes and gracefully handling a `Spent` error from the mint. +5. The wallet then queries the GCS filter with its `Y` values. If a `Y` value doesn't match in the filter, the note is considered unspent, while it's "maybe" spent otherwise. Due to the probabilistic nature of GCS, false positive are possible, meaning a note might be marked as spent when it is not. For this reason, wallets **SHOULD** check the state of all "maybe" spent notes. **Restore Flow for Issued Blind Signatures:** 1. The wallet identifies the `keyset_id` for which it needs to check issued blind signatures. 2. The wallet queries the Mint's `GET v1/filter/issued/{keyset_id}` endpoint to retrieve the GCS filter for issued blind signatures. 3. Upon receiving the `GetFilterResponse`, the wallet extracts the `content` (the GCS filter bytes), `n`, `p`, and `m` parameters. -4. For each `blinded_message` (`B_`) the wallet has submitted for signing for that `keyset_id`, it queries the GCS filter with its `B_` value. If a `B_` value matches in the filter, the blind signature is considered issued. Similar to spent notes, false positives are possible. Wallets **SHOULD** handle this by attempting to retrieve the signed blind signature and gracefully handling errors. +4. For each deterministically derived ([NUT-13](13)) `blinded_message` (`B_`) of `keyset_id`, it queries the GCS filter with its `B_` value. If a `B_` value doesn't match in the filter, the blind signature was not issued, while it's "maybe" issued otherwise. Similar to spent notes, false positives are possible. Wallets **SHOULD** handle this by attempting to retrieve the signed blind signature and gracefully handling errors. ### Querying for Spent or Issued Ecash Notes @@ -106,3 +106,6 @@ Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetIn } } ``` + + +[13]: 13.md \ No newline at end of file From a2409f60a504a774677d1b4fea9587354e048d49 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Sat, 5 Jul 2025 11:43:27 +0200 Subject: [PATCH 18/25] docs: Update NUT number from 24 to 25 Co-authored-by: aider (openrouter/google/gemini-2.5-flash) --- 24.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/24.md b/24.md index 21fc8d09..03267735 100644 --- a/24.md +++ b/24.md @@ -1,4 +1,4 @@ -# NUT-24: Golomb-Coded Set Filters +# NUT-25: Golomb-Coded Set Filters `optional` @@ -95,17 +95,17 @@ For $M = 784931$, this turns out to be: ## Mint Info Settings -Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetInfoResponse`. They can do so by updating the response to signal support for NUT-XX. +Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetInfoResponse`. They can do so by updating the response to signal support for NUT-25. ```json { ..., "nuts": { ..., - "XX": {"supported": true} + "25": {"supported": true} } } ``` -[13]: 13.md \ No newline at end of file +[13]: 13.md From b5682b83f3151f203f4fbaee6dc062b2d7ee36d5 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Sat, 5 Jul 2025 11:45:14 +0200 Subject: [PATCH 19/25] rename and change file number to 25 --- 24.md => 25.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename 24.md => 25.md (99%) diff --git a/24.md b/25.md similarity index 99% rename from 24.md rename to 25.md index 03267735..e8194783 100644 --- a/24.md +++ b/25.md @@ -1,4 +1,4 @@ -# NUT-25: Golomb-Coded Set Filters +# NUT-25: Compact Nut Filters `optional` From 7a1d0a13509a2b9aeb5cf7e763654168859b70a5 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Sat, 5 Jul 2025 11:54:50 +0200 Subject: [PATCH 20/25] fix fpr 1-(1-/m)^x --- 25.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/25.md b/25.md index e8194783..e7cf5ed5 100644 --- a/25.md +++ b/25.md @@ -88,10 +88,10 @@ For $M = 784931$, this turns out to be: | $x$ | $M$ | $P_M(x)$ | | --- | --- | -------- | -| 1 | 784931 | 0.999998726 | -| 10 | 784931 | 0.99998726 | -| 300 | 784931 | 0.999617874 | -| 5000 | 784931 | 0.993650255 | +| 1 | 784931 | 0.000001274 | +| 10 | 784931 | 0.00001274 | +| 300 | 784931 | 0.000382126 | +| 5000 | 784931 | 0.006349745 | ## Mint Info Settings From fccbd8c192a786c74e6abd58a854525d1e71ee5d Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Sat, 5 Jul 2025 12:09:09 +0200 Subject: [PATCH 21/25] rename to 25-test --- tests/{24-test.md => 25-test.md} | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) rename tests/{24-test.md => 25-test.md} (86%) diff --git a/tests/24-test.md b/tests/25-test.md similarity index 86% rename from tests/24-test.md rename to tests/25-test.md index 56712efe..a24b2cda 100644 --- a/tests/24-test.md +++ b/tests/25-test.md @@ -20,4 +20,8 @@ The following list of items should encode to the target filter `z4fUCDVqdnxWR7Y9 ] ``` -Matching every given item using `match_many` should return `True` for every item, while matching any other item should return `False`. +Matching any given item from this list should return `True`, while matching any item from the following list +should return `False`: +```json + +``` From 2ebec31e22c519a42ef3a063e47c9dca77e54e4a Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Sat, 5 Jul 2025 12:09:32 +0200 Subject: [PATCH 22/25] docs: Add test vectors for NUT-25 Golomb-Coded Set Filters Co-authored-by: aider (openrouter/google/gemini-2.5-flash) --- tests/25-test.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/tests/25-test.md b/tests/25-test.md index a24b2cda..93038f79 100644 --- a/tests/25-test.md +++ b/tests/25-test.md @@ -1,4 +1,4 @@ -# NUT-XX Test Vectors +# NUT-25 Test Vectors The following list of items should encode to the target filter `z4fUCDVqdnxWR7Y9+YdT5o0IC9GxiSA2BGyg`, with parameters: @@ -23,5 +23,11 @@ The following list of items should encode to the target filter `z4fUCDVqdnxWR7Y9 Matching any given item from this list should return `True`, while matching any item from the following list should return `False`: ```json - -``` +[ + "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3", + "d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6", + "00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff00", + "ffeeddccbbaa99887766554433221100ffeeddccbbaa99887766554433221100ffee", + "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b3c" +] +``` From 30929ee8c3dc515bc3d94506dfbdc4ce61a7cd70 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Fri, 11 Jul 2025 12:23:41 +0200 Subject: [PATCH 23/25] updates --- 25.md | 6 +++++- README.md | 3 +++ tests/25-test.md | 2 +- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/25.md b/25.md index e7cf5ed5..81582ae6 100644 --- a/25.md +++ b/25.md @@ -46,7 +46,7 @@ Wallets utilize GCS filters during recovery to determine the status of ecash not 3. Upon receiving the `GetFilterResponse`, the wallet extracts the `content` (the GCS filter bytes), `n`, `p`, and `m` parameters. 4. For each deterministically derived ([NUT-13](13)) `blinded_message` (`B_`) of `keyset_id`, it queries the GCS filter with its `B_` value. If a `B_` value doesn't match in the filter, the blind signature was not issued, while it's "maybe" issued otherwise. Similar to spent notes, false positives are possible. Wallets **SHOULD** handle this by attempting to retrieve the signed blind signature and gracefully handling errors. -### Querying for Spent or Issued Ecash Notes +### Querying for Spent or Issued Ecash Filters Wallets **MAY** query the following endpoints: @@ -78,6 +78,10 @@ Where: ``` - `timestamp` is the Unix epoch (in seconds) when the filter was created. The Mint might choose not to re-compute the filter upon every request, and instead serve one from cache memory and computed an updated version after arbitrary amount of time. +### Implementation Details + +See the reference implementation in [cashu-ts](https://github.com/cashubtc/cashu-ts/blob/aeb85d6b03fa30cc2a2cfa7c3c647ed17cb6501f/src/gcs.ts) for details. + ### False Positive Rate (FPR) For Bulk Tests Each individual look-up has $1 - \frac{1}{M}$ chance of being a _true positive_. We can consider a bulk test as $x$ independent look-ups, so the chance that at _all_ of them are true positives is $\bigl(1 - \frac{1}{M}\bigr)^x$. diff --git a/README.md b/README.md index 626f9339..557b26f9 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,8 @@ Wallets and mints `MUST` implement all mandatory specs and `CAN` implement optio | [20][20] | Signature on Mint Quote | [cdk-cli], [Nutshell][py], [gonuts] | [cdk-mintd], [Nutshell][py], [gonuts] | | [21][21] | Clear authentication | [Nutshell][py], [cdk-cli] | [Nutshell][py], [cdk-mintd], [nutmix] | | [22][22] | Blind authentication | [Nutshell][py], [cdk-cli] | [Nutshell][py], [cdk-mintd], [nutmix] | +| [25][25] | Compact Nut Filters | [Nutshell][py], [cdk-mintd], [cashu-ts][ts] | +[Nutshell][py], [cdk-mintd] #### Wallets: @@ -96,3 +98,4 @@ Wallets and mints `MUST` implement all mandatory specs and `CAN` implement optio [20]: 20.md [21]: 21.md [22]: 22.md +[25]: 25.md diff --git a/tests/25-test.md b/tests/25-test.md index 93038f79..854484aa 100644 --- a/tests/25-test.md +++ b/tests/25-test.md @@ -1,6 +1,6 @@ # NUT-25 Test Vectors -The following list of items should encode to the target filter `z4fUCDVqdnxWR7Y9+YdT5o0IC9GxiSA2BGyg`, with parameters: +The following list of items should encode to the target filter `7sdQJ7OweaujLCqS7KDHzu/3pySZrDsatjQA`, with parameters: - `p = 19` - `m = 784931` From 1bad600b2a3bd1c6e8ee06102b2e4d2136a8b299 Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Fri, 11 Jul 2025 12:26:15 +0200 Subject: [PATCH 24/25] fix --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 557b26f9..d8c69d5d 100644 --- a/README.md +++ b/README.md @@ -38,8 +38,7 @@ Wallets and mints `MUST` implement all mandatory specs and `CAN` implement optio | [20][20] | Signature on Mint Quote | [cdk-cli], [Nutshell][py], [gonuts] | [cdk-mintd], [Nutshell][py], [gonuts] | | [21][21] | Clear authentication | [Nutshell][py], [cdk-cli] | [Nutshell][py], [cdk-mintd], [nutmix] | | [22][22] | Blind authentication | [Nutshell][py], [cdk-cli] | [Nutshell][py], [cdk-mintd], [nutmix] | -| [25][25] | Compact Nut Filters | [Nutshell][py], [cdk-mintd], [cashu-ts][ts] | -[Nutshell][py], [cdk-mintd] +| [25][25] | Compact Nut Filters | [Nutshell][py], [cdk-mintd], [cashu-ts][ts] |[Nutshell][py], [cdk-mintd] | #### Wallets: From e85f2b7c2c08aea9dbd30b22506244a7f6c02e9a Mon Sep 17 00:00:00 2001 From: lollerfirst Date: Tue, 22 Jul 2025 22:47:50 +0200 Subject: [PATCH 25/25] prettier --- 25.md | 49 +++++++++++++++++++++++++----------------------- README.md | 2 +- tests/25-test.md | 1 + 3 files changed, 28 insertions(+), 24 deletions(-) diff --git a/25.md b/25.md index 81582ae6..aa108e85 100644 --- a/25.md +++ b/25.md @@ -17,6 +17,7 @@ Clients can query and use these filters to test for set membership, which is par A Golomb-Coded Set (GCS) is a probabilistic data structure that allows for compact representation of a set of items. It enables checking for set membership with a certain false positive rate, but no false negatives. GCS filters are constructed by hashing items to a range, sorting the hashed values, and then encoding the differences between successive values using Golomb-Rice coding. This method provides significant compression, making them suitable for efficient transmission and client-side processing. Each filter is defined by: + - `N`: The cardinality of the set it encodes. - `P`: The bit length of the remainder code in Golomb-Rice coding. - `M`: The inverse of the target false positive rate. @@ -26,6 +27,7 @@ Each filter is defined by: Mints **MUST** generate GCS filters containing sets of items associated with specific keysets. These filters are generated at self-determined intervals. The mint is responsible for ensuring the filters are available for clients to query. For each keyset, mints generate: + - A filter encoding the `Y` (nullifiers) values of all spent ecash notes. - A filter encoding the `B_` (blinded_messages) values of all issued ecash notes. @@ -34,6 +36,7 @@ For each keyset, mints generate: Wallets utilize GCS filters during recovery to determine the status of ecash notes and blind signatures leaking as little sensitive information as possible. **Restore Flow for Spent Ecash Notes:** + 1. The wallet identifies the `keyset_id` for which it needs to check spent notes. 2. The wallet queries the Mint's `GET v1/filter/spent/{keyset_id}` endpoint to retrieve the GCS filter for spent nullifiers. 3. Upon receiving the `GetFilterResponse`, the wallet extracts the `content` (the GCS filter bytes), `n`, `p`, and `m` parameters. @@ -41,6 +44,7 @@ Wallets utilize GCS filters during recovery to determine the status of ecash not 5. The wallet then queries the GCS filter with its `Y` values. If a `Y` value doesn't match in the filter, the note is considered unspent, while it's "maybe" spent otherwise. Due to the probabilistic nature of GCS, false positive are possible, meaning a note might be marked as spent when it is not. For this reason, wallets **SHOULD** check the state of all "maybe" spent notes. **Restore Flow for Issued Blind Signatures:** + 1. The wallet identifies the `keyset_id` for which it needs to check issued blind signatures. 2. The wallet queries the Mint's `GET v1/filter/issued/{keyset_id}` endpoint to retrieve the GCS filter for issued blind signatures. 3. Upon receiving the `GetFilterResponse`, the wallet extracts the `content` (the GCS filter bytes), `n`, `p`, and `m` parameters. @@ -50,33 +54,33 @@ Wallets utilize GCS filters during recovery to determine the status of ecash not Wallets **MAY** query the following endpoints: -- `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` (nullifiers) values of all the spent ecash from `keyset_et_id`. -- `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` (blinded_messages) values of all the issued ecash from `keyset_id`. +- `GET v1/filter/spent/{keyset_id}` to get the GCS filter that encodes the `Y` (nullifiers) values of all the spent ecash from `keyset_et_id`. +- `GET v1/filter/issued/{keyset_id}` to get the GCS filter that encodes the `B_` (blinded_messages) values of all the issued ecash from `keyset_id`. The Mint **MUST** respond with a `GetFilterResponse`, which has the following structure. ```json { - "n": "", - "p": "", - "m": "", - "content": "", - "timestamp": "" + "n": "", + "p": "", + "m": "", + "content": "", + "timestamp": "" } ``` Where: -- `n` is the number of items in the filter. -- `p` is the bit parameter of the Golomb-Rice coding. If `null`, then the client assumes `p = 19`. -- `m` is the inverse of the false positive rate. If `null`, then the client assumes `m = 784931`. -- `content` is a base-64 string encoding the bytes of the filter. It is typically computed as: - ```python - content = b64encode(filter_bytes).decode() - # And vice-versa - filter_bytes = b64decode(content) - ``` -- `timestamp` is the Unix epoch (in seconds) when the filter was created. The Mint might choose not to re-compute the filter upon every request, and instead serve one from cache memory and computed an updated version after arbitrary amount of time. +- `n` is the number of items in the filter. +- `p` is the bit parameter of the Golomb-Rice coding. If `null`, then the client assumes `p = 19`. +- `m` is the inverse of the false positive rate. If `null`, then the client assumes `m = 784931`. +- `content` is a base-64 string encoding the bytes of the filter. It is typically computed as: + ```python + content = b64encode(filter_bytes).decode() + # And vice-versa + filter_bytes = b64decode(content) + ``` +- `timestamp` is the Unix epoch (in seconds) when the filter was created. The Mint might choose not to re-compute the filter upon every request, and instead serve one from cache memory and computed an updated version after arbitrary amount of time. ### Implementation Details @@ -90,11 +94,11 @@ Therefore, the chance of any one of them being a _false positive_ (or equivalent For $M = 784931$, this turns out to be: -| $x$ | $M$ | $P_M(x)$ | -| --- | --- | -------- | -| 1 | 784931 | 0.000001274 | -| 10 | 784931 | 0.00001274 | -| 300 | 784931 | 0.000382126 | +| $x$ | $M$ | $P_M(x)$ | +| ---- | ------ | ----------- | +| 1 | 784931 | 0.000001274 | +| 10 | 784931 | 0.00001274 | +| 300 | 784931 | 0.000382126 | | 5000 | 784931 | 0.006349745 | ## Mint Info Settings @@ -111,5 +115,4 @@ Mints that support this NUT **MUST** announce it in their [NUT-06](06.md) `GetIn } ``` - [13]: 13.md diff --git a/README.md b/README.md index 8b6cf1a6..b6c9f5d5 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ Wallets and mints `MUST` implement all mandatory specs and `CAN` implement optio | [22][22] | Blind authentication | [Nutshell][py], [cdk-cli] | [Nutshell][py], [cdk-mintd], [nutmix] | | [23][23] | Payment Method: BOLT11 | [Nutshell][py], [cdk-cli] | [Nutshell][py], [cdk-mintd], [nutmix] | | [24][24] | HTTP 402 Payment Required | - | - | -| [25][25] | Compact Nut Filters | [Nutshell][py], [cdk-mintd], [cashu-ts][ts] |[Nutshell][py], [cdk-mintd] | +| [25][25] | Compact Nut Filters | [Nutshell][py], [cdk-mintd], [cashu-ts][ts] | [Nutshell][py], [cdk-mintd] | #### Wallets: diff --git a/tests/25-test.md b/tests/25-test.md index 854484aa..f732de8b 100644 --- a/tests/25-test.md +++ b/tests/25-test.md @@ -22,6 +22,7 @@ The following list of items should encode to the target filter `7sdQJ7OweaujLCqS Matching any given item from this list should return `True`, while matching any item from the following list should return `False`: + ```json [ "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3",