Skip to content
This repository was archived by the owner on Apr 1, 2023. It is now read-only.
This repository was archived by the owner on Apr 1, 2023. It is now read-only.

HTTP proxy assertion fail on shutdown with ongoing txns #310

Description

@iafuture

It seems the heartbeat for a txn is not cancelled when the txn is dropped in the HTTPProxy, causing the hb timer to attempt to use the RPCDispatcher to send a message. The dispatcher has shutdown at this point and asserts that it should not be used, e.g.

[0004:14:18:51.953.116]-http_proxy-[0]-(k2::skv_client) [DEBUG] [/build/src/k2/module/k23si/client/k23si_client.cpp:66 @operator()] send hb for mtr={timestamp={endCount=397131397989158, tsoId=100000, startDelta=100000}, priority=medium}
[0004:14:18:51.953.201]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:88 @partitionRequest] making partition request with deadline=0000:00:00:00.499.993
[0004:14:18:52.009.280]-http_proxy-[0]-(k2::transport) [ERROR] [/build/src/k2/transport/RPCDispatcher.h:189 @operator()] RPC send failed with uncaught exception: dispatcher has shut down
[0004:14:18:52.009.353]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:360 @operator()] partition call completed with status={code=500, message=unknown exception while sending request}
[0004:14:18:52.009.405]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:105 @operator()] call failed with retryable status: {code=503, message=unable to make call at this time}
[0004:14:18:52.009.423]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:110 @operator()] refreshing collection from CPO after status={code=503, message=unable to make call at this time}
[0004:14:18:52.010.636]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:247 @_getAssignedPartitionWithRetry] time remaining=0000:00:00:00.442.563, for coll=HTTPClient, key={schemaName=test_schema, partitionKey=^00^00^01, rangeKey=^00^00^01}, reverse=false, excludedKey=false, retries=1
[0004:14:18:52.013.215]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:273 @_getAssignedPartitionWithRetry] no existing waiter for name=HTTPClient. Creating new one
http_proxy: /usr/local/include/seastar/core/sharded.hh:755: Service& seastar::sharded::local() [with Service = k2::RPCDispatcher]: Assertion `local_is_initialized()' failed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions