IPIP-484: Opt-in Filtering on Routing V1 HTTP API

src/ipips/ipip-0484.md

@@ -0,0 +1,106 @@
+---
+title: 'IPIP-0484: Opt-in Filtering in Routing V1 HTTP API'
+date: 2024-10-29
+ipip: ratified
+editors:
+  - name: Daniel Norman
+    github: 2color
+    affiliation:
+      name: Shipyard
+      url: https://ipshipyard.com
+relatedIssues:
+  - https://github.com/ipfs/ipfs-check/issues/70
+  - https://github.com/ipfs/boxo/pull/678
+order: 484
+tags: ['ipips']
+---
+
+## Summary
+
+Add opt-in support for filtering specific network transports and/or transfer protocols to the Delegated Routing v1 HTTP endpoints via HTTP GET parameters.
+
+## Motivation
+
+IPFS aims to allow ubiquitous data exchange across different runtimes and platforms. One of the most challenging aspects of this goal is the diversity of network conditions and capabilities across different environments. Web browsers have a very limited network stack, and most web browsers do not support the full range of network transport protocols that are commonly used in other IPFS implementations.
+
+The Delegated Routing v1 API empowers resource constrained clients like web browsers by significantly reducing the number of network connections necessary to fetch content addressed data directly from provider peers.
+
+However, there are many cases where most of the results from the Delegated Routing v1 API are not actionable by clients, because the client does not support either the **network transport protocol** or the **transfer protocol** of the provider.
+
+For instance, web browsers are limited to a specific set of network transport protocols, namely HTTPS, Secure WebSockets, WebTransport (emerging), and WebRTC. Consequently, providing information about providers that exclusively support TCP and/or UDP is not beneficial for browser-based clients, as they are unable to utilize such connections.
+
+Moreover, [Helia](https://github.com/ipfs/helia/), the most actively maintained browser IPFS implementation, supports block retrieval by CID with Bitswap and Trustless Gateways, but does not support Graphsync.
+
+This means that returning providers that only support raw TCP, raw UDP/QUIC, or Graphsync from the Delegated Routing API is not useful for browser clients, and results in unnecessary network traffic for browser clients.
+
+## Note on terminology
+
+The term **"transport"** is overloaded in the IPFS ecosystem.
+
+In the context of this IPIP, we refer to the network layer transport protocol, e.g. TCP, QUIC, WebTransport, as **"network transport protocol"** to avoid ambiguity.
+
+**"Transfer protocol"** refers to data transfer protocols, i.e. content-addressed block retrieval protocols, e.g. Bitswap, Graphsync, HTTP.
+
+## Detailed design
+
+### Network Address Filtering
+
+The proposed change is to add a `?filter-addrs` parameter to the `GET /routing/v1/providers/{cid}` and `GET /routing/v1/peers/{peer-id}` endpoints of :cite[http-routing-v1]:
+
+- Add a `?filter-addrs=<comma-separated-list>` optional parameter to `GET /routing/v1/providers/{CID}` that indicates which network transports to return by filtering the multiaddrs in the `Addrs` field of the [Peer schema].
+- The value of the `filter-addrs` parameter is a comma-separated list of network transport protocol _name strings_ as defined in the [multiaddr protocol registry](https://github.com/multiformats/multiaddr/blob/master/protocols.csv), e.g. `?filter-addrs=webtransport`.
+- `unknown` can be be passed to include providers whose multiaddrs are unknown, e.g. `?filter-addrs=unknown`. This allows filtering providers whose multiaddrs are unknown at the time of filtering.
+- Multiaddrs are filtered by checking if the protocol name appears in any of the multiaddrs (logical OR).
+- Negative filtering is done by prefixing the protocol name with `!`, e.g. to skip IPv6 and QUIC addrs: `?filter-addrs=!ip6,!quic-v1`. Note that negative filtering is done by checking if the protocol name does not appear in any of the multiaddrs (logical AND).
+- If no parameter is passed, the default behavior is to return the original list of addresses unchanged.
+- If only negative filters are provided, addresses not passing any of the negative filters are included.
+- If positive filters are provided, only addresses passing at least one positive filter (and no negative filters) are included.
+- If both positive and negative filters are provided, the address must pass all negative filters and at least one positive filter to be included.
+- If there are no multiaddrs that match the passed transports, the provider is omitted from the response.
+- Filtering is case-insensitive.
+
+### IPFS Protocol Filtering
+
+The proposed change is to add a `?filter-protocols` parameter to the `GET /routing/v1/providers/{cid}` and `GET /routing/v1/peers/{peer-id}` endpoints of :cite[http-routing-v1]:
+
+- Add a `?filter-protocols=<comma-separated-list>` optional parameter to `GET /routing/v1/providers/{CID}` to filter providers based on the `Protocol` field of the [Peer schema].
+- The `filter-protocols` parameter is a comma-separated list of transfer protocol names, e.g. `?filter-protocols=transport-bitswap`.
+- Transfer protocols names should be treated as opaque strings and have a max length of 63 characters. A non-exhaustive list of transfer protocols are defined per convention in the [multicodec registry](https://github.com/multiformats/multicodec/blob/3b7b52deb31481790bc4bae984d8675bda4e0c82/table.csv#L149-L151).
+- Implementations MUST preserve all transfer protocol names when returning a positive result that matches one or more of them.
+- A special `unknown` name can be be passed to include providers whose transfer protocol list is empty (unknown), e.g. `?filter-protocols=unknown`. This allows for including providers returned from the DHT that do not contain explicit transfer protocol information.
+- Providers are filtered by checking if the transfer protocol name appears in the `Protocols` array (logical OR).
+- If the provider doesn't match any of the passed transfer protocols, the provider is omitted from the response.
+- If a provider passes the filter, it is returned unchanged, i.e. the full set of protocols is returned including protocols that not included in the filter. (note that this is different from `filter-addrs` where only the multiaddrs that pass the filter are returned)
+- Filtering is case-insensitive.
+- If no parameter is passed, the default behavior is to not filter by transfer protocol.
+
+:::note
+Even though some of existing IPFS transfer protocol names start with `transport`, e.g. `transport-bitswap`, `transport-graphsync-filecoinv1`, and `transport-ipfs-gateway-http`, they should not to be confused with the network transport protocols used in peer addresses, which are filtered using the `filter-addrs` parameter.
+:::
+
+## Design rationale
+
+- Using these query parameters improves cache efficiency, as the response will be smaller and more specific to the client's needs.
+- Backward compatibility is maintained by not changing the default behavior of the API.
+- Use of protocol name rather than codes makes it easier for human debugging.
+- DHT providers currently do not contain any transfer protocol information. `unknown` can be passed to `filter-protocols` to include such providers.
+- Since provider records are independent of peer records, and it's pretty common to have provider records without up-to-date multiaddrs for that peer, `unknown` can be passed to `filter-addrs` to include such providers.
+- Combining transfer protocol and transport protocol filters is done by ANDing the results of the filters, e.g. `?filter-addrs=webtransport&filter-protocols=transport-bitswap` will return providers that support bitswap and have a webtransport multiaddr.
+
+### User benefit
+
+By filtering out providers that do not support the desired network transport protocol and/or transfer protocol, the client can reduce the traffic necessary in order to fetch the data.
+
+Moreover, it makes it much easier to determine whether there are any browser-usable providers for a given CID, which is a common use case for clients.
+
+### Compatibility
+
+This should not effect existing clients or servers.
+
+The default behavior when `?filter-addrs` and `?filter-protocols` is not passed is left unspecified, this IPIP is limited to opt-in behavior.
+
+### Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
+
+[Peer schema]: https://specs.ipfs.tech/routing/http-routing-v1/#peer-schema

src/routing/http-routing-v1.md

@@ -4,7 +4,7 @@ description: >
   Delegated routing is a mechanism for IPFS implementations to use for offloading
   content routing, peer routing and naming to another process/server. This specification describes
   an HTTP API for delegated routing of content, peers, and IPNS.
-date: 2024-03-22
+date: 2024-10-29
 maturity: reliable
 editors:
   - name: Gus Eggert
@@ -21,14 +21,19 @@ editors:
     url: https://hacdias.com/
     github: hacdias
     affiliation:
-      name: Protocol Labs
-      url: https://protocol.ai/
+      name: Shipyard
+      url: https://ipshipyard.com
   - name: Marcin Rataj
     github: lidel
     url: https://lidel.org/
     affiliation:
-      name: Protocol Labs
-      url: https://protocol.ai/
+      name: Shipyard
+      url: https://ipshipyard.com
+  - name: Daniel Norman
+    github: 2color
+    affiliation:
+      name: Shipyard
+      url: https://ipshipyard.com
 xref:
   - ipns-record
 order: 0
@@ -65,6 +70,38 @@ This API uses a standard version prefix in the path, such as `/v1/...`. If a bac
 
 - `cid` is the [CID](https://github.com/multiformats/cid) to fetch provider records for.
 
+#### Request Query Parameters
+
+##### `filter-addrs` (providers request query parameter)
+
+Optional `?filter-addrs` to apply Network Address Filtering from [IPIP-484](https://specs.ipfs.tech/ipips/ipip-0484/).
+
+- `?filter-addrs=<comma-separated-list>` optional parameter that indicates which network transports to return by filtering the multiaddrs in the `Addrs` field of the [Peer schema](#peer-schema).
+- The value of the `filter-addrs` parameter is a comma-separated (`,` or `%2C`) list of network transport protocol _name strings_ as defined in the [multiaddr protocol registry](https://github.com/multiformats/multiaddr/blob/master/protocols.csv), e.g. `?filter-addrs=tls,webrtc-direct,webtransport`.
+- `unknown` can be be passed to include providers whose multiaddrs are unknown, e.g. `?filter-addrs=unknown`. This allows for not removing providers whose multiaddrs are unknown at the time of filtering (e.g. keeping DHT results that require additional peer lookup).
+- Multiaddrs are filtered by checking if the protocol name appears in any of the multiaddrs (logical OR).
+- Negative filtering is done by prefixing the protocol name with `!`, e.g. to skip IPv6 and QUIC addrs: `?filter-addrs=!ip6,!quic-v1`. Note that negative filtering is done by checking if the protocol name does not appear in any of the multiaddrs (logical AND).
+- If no parameter is passed, the default behavior is to return the original list of addresses unchanged.
+- If only negative filters are provided, addresses not passing any of the negative filters are included.
+- If positive filters are provided, only addresses passing at least one positive filter (and no negative filters) are included.
+- If both positive and negative filters are provided, the address must pass all negative filters and at least one positive filter to be included.
+- If there are no multiaddrs that match the passed transports, the provider is omitted from the response.
+- Filtering is case-insensitive.
+
+##### `filter-protocols` (providers request query parameter)
+
+Optional `?filter-protocols` to apply IPFS Protocol Filtering from [IPIP-484](https://specs.ipfs.tech/ipips/ipip-0484/).
+
+- The `filter-protocols` parameter is a comma-separated (`,` or `%2C`) list of transfer protocol names, e.g. `?filter-protocols=unknown,transport-bitswap,transport-ipfs-gateway-http`.
+- Transfer protocols names should be treated as opaque strings and have a max length of 63 characters. A non-exhaustive list of transfer protocols are defined per convention in the [multicodec registry](https://github.com/multiformats/multicodec/blob/3b7b52deb31481790bc4bae984d8675bda4e0c82/table.csv#L149-L151).
+- Implementations MUST preserve all transfer protocol names when returning a positive result that matches one or more of them.
+- A special `unknown` name can be be passed to include providers whose transfer protocol list is empty (unknown), e.g. `?filter-protocols=unknown`. This allows for including providers returned from the DHT that do not contain explicit transfer protocol information.
+- Providers are filtered by checking if the transfer protocol name appears in the `Protocols` array (logical OR).
+- If the provider doesn't match any of the passed transfer protocols, the provider is omitted from the response.
+- If a provider passes the filter, it is returned unchanged, i.e. the full set of protocols is returned including protocols that not included in the filter. (note that this is different from `filter-addrs` where only the multiaddrs that pass the filter are returned)
+- Filtering is case-insensitive.
+- If no parameter is passed, the default behavior is to not filter by transfer protocol.
+
 #### Response Status Codes
 
 - `200` (OK): the response body contains 0 or more records.
@@ -113,6 +150,16 @@ Each object in the `Providers` list is a record conforming to a schema, usually
 - `peer-id` is the [Peer ID](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md) to fetch peer records for,
 represented as a CIDv1 encoded with `libp2p-key` codec.
 
+#### Request Query Parameters
+
+##### `filter-addrs` (peers request query parameter)
+
+Optional, same rules as [`filter-addrs` providers request query parameter](#filter-addrs-providers-request-query-parameter).
+
+##### `filter-protocols` (peers request query parameter)
+
+Optional, same rules as [`filter-protocols` providers request query parameter](#filter-protocols-providers-request-query-parameter).
+
 #### Response Status Codes
 
 - `200` (OK): the response body contains the peer record.