feat: add dynamic header middleware support for rest namespace

[jtuglu1]

Paired with lance-format/lance-namespace#298.

Changes

• Support dynamic header middlewares for namespace clients. Previously this was possible with something like:

class CustomRestNamespaceConfig(RestNamespaceConfig):
    def __init__(self, properties: Dict[str, str]):
        super().__init__(properties)
        self.e2e = init_e2e_client()
        self.token = None
        self.expiration = 0


    def additional_headers(self) -> Dict[str, str]:
        if self.token is None or time.time() > self.expiration - 60:
            response = self.e2e.get_token()
            self.token = response.token
            self.expiration = response.expires_at_epoch_time
        return {"x-forwarded-authentication": self.token}


class CustomRestNamespace(LanceRestNamespace):
    def __init__(self, **kwargs):
        configuration = Configuration()
        (crt, pkey) = getClientKeystore()
        configuration.cert_file = crt
        configuration.key_file = pkey
        configuration.ssl_ca_cert = getClientTruststore()
        configuration.assert_hostname = False

        configuration.retries = Retry(total=MAX_RETRIES, backoff_factor=HTTP_BACKOFF_FACTOR)
        configuration.host = kwargs["uri"]

        self.api_client = ApiClient(configuration)
        self.namespace_api = NamespaceApi(self.api_client)
        self.table_api = TableApi(self.api_client)
        self.transaction_api = TransactionApi(self.api_client)
        self.config = CustomRestNamespaceConfig(kwargs)


    def namespace_id(self) -> str:
        return "customlance"

With c73c3dc, this behavior was broken as the namespace client pattern was switched to builder and the additional_headers pass-through fields were removed. The idea here is now:

  class CustomBdpLanceRestNamespace(LanceRestNamespace):
      def __init__(self, **kwargs):
          configuration = Configuration()
          (crt, pkey) = getClientKeystore()
          configuration.cert_file = crt
          configuration.key_file = pkey
          configuration.ssl_ca_cert = getClientTruststore()
          configuration.assert_hostname = False
          configuration.retries = Retry(total=MAX_RETRIES, backoff_factor=HTTP_BACKOFF_FACTOR)
          configuration.host = kwargs["uri"]

          # Token state (closure captures these)
          e2e = init_e2e_client()
          token_cache = {"token": None, "expiration": 0}

          def get_headers():
              if token_cache["token"] is None or time.time() > token_cache["expiration"] - 60:
                  response = e2e.get_token()
                  token_cache["token"] = response.token
                  token_cache["expiration"] = response.expires_at_epoch_time
              return {"x-forwarded-authentication": token_cache["token"]}

          # Set the header provider
          configuration.header_provider = get_headers

          self.api_client = ApiClient(configuration)
          self.namespace_api = NamespaceApi(self.api_client)
          self.table_api = TableApi(self.api_client)
          self.transaction_api = TransactionApi(self.api_client)

      def namespace_id(self) -> str:
          return "customlance"

This allows for background propagation of the headers without needing to pass-through the "additionalHeaders" fields in all the API callsites.

Alternatives considered

1. Plumb through the additional_headers param to the rust namespace implementation, then simply pass through headers in rest.rs using the header_provider.

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[jtuglu1]

From the local toml stuff – will be removed.