Skip to content

Forwarded replication operations leak requestId into the operation body, breaking replicated set_configuration #969

Open

Description

@kriszyp

Summary

When a node forwards an operation to a peer over replication, the RPC correlation id requestId stays in the operation body that the peer passes to server.operation. Operations that validate their body strictly then see an unknown requestId key. Replicated set_configuration is the reported case: it is rejected on the peer on current main.

Mechanism (harper-pro main @ 36efd7e4)

  • Sender, sendOperation (replication/replicationConnection.ts ~8572): operation.requestId = requestId; mutates the caller's operation object and sends it as [OPERATION_REQUEST, operation].
  • Receiver, OPERATION_REQUEST handler (~4929): server.operation(data, { user: authorization }, …) receives data with requestId still attached. The reply paths then read data.requestId (~4936, 4940, 4945, 4949) to correlate the response.

So requestId is transport metadata that ends up in the operation payload. Any operation whose validation rejects unknown properties fails on the remote node. Every other forwarded operation silently receives an extra field.

This was found during review of harper-pro#956 (table-drop recovery), and the agent's investigation reported it as pre-existing, independent of that PR. The set_configuration rejection was reported by that investigation and has not been independently reproduced here.

Proposed fix

Treat requestId as envelope, not payload:

  • On send, do not mutate the caller's operation. Send the id beside it, or strip it into the frame.
  • On receive, capture requestId before dispatch, pass server.operation a body without it, and restore it on every reply path (success, encoding failure, operation rejection, synchronous throw) from the captured value instead of data.requestId.

Keep wire compatibility with peers on the old shape. An old sender still embeds requestId in the body, so the receiver should strip it there too. An old receiver still expects it in the body.

Tests

  • Two-node integration test: replicated set_configuration succeeds on the peer and the configuration propagates.
  • Reply/error correlation: concurrent forwarded operations, including one that fails validation, each resolve to their own response.
  • Mixed-version: an old-shape request (id in the body) is still answered and correlated.

Activity

  1. added theissue type on Oct 5, 2026
  2. dawsontoth commented on Oct 22, 2026

    @dawsontoth
    Contributor

    Independently reproduced on stage Fabric, harper-pro 5.3.3, two-node cluster, 2026-10-22.

    A set_configuration sent from Studio to one node at 20:32:06Z applied there. Replicating it to the peer failed, as this issue describes:

    [main/0] [warn]: Replicating set_configuration to <peer> failed: HdbError: Unable to update config, unrecognized config parameter: requestId
    

    The peer logged Error calling operation: setConfiguration / Unable to update config, unrecognized config parameter: requestId at 20:32:15Z.

    Two things this issue doesn't mention yet:

    • The nodes' configs silently diverge. The node that received the call applies the change and the peer rejects it. The only trace is a warn in the receiving node's log, so a Studio config change on a multi-node Fabric cluster can leave the nodes configured differently without anyone noticing.
    • The peer restarts anyway. It started its own restart job and restarted its workers at 20:32:20Z, 5 s after rejecting the config. So it pays for a restart without getting the change. In our case that restart killed an in-flight custom-domain certificate challenge on the peer.

    Given that it ships in 5.3.3 and that Studio triggers it routinely, this may deserve more than P2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:replicationReplication, cluster sync, peer connections

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions