"Callback was already called" using wait and capture/match with websocket engine

[xentac]

If your websocket server sends multiple messages while a wait with a capture/match is being processed, you will get a "Callback was already called" stack trace. This is mostly because artillery was written with the assumption that each request will have a response and wait doesn't quite fit into that paradigm. Should a wait consume all messages until it finds a match/capture or should it just consume the next one and the user is responsible for using loop/whileTrue to process all the messages.

Surrounding your wait step with think steps can help clear the message queue before processing a single message, but this will lead to lost messages which could be important. A good load test will need to be able to process every message at one point or another.

My theory is that websocket messages are queued up and multiples are passed to the captureOrMatch method instead of just a single one before processing can happen on the next step.

Whatever the decision is, a stacktrace shouldn't be happening in this case.

Version info:

        ___         __  _ ____
  _____/   |  _____/ /_(_) / /__  _______  __ ___
/____/ /| | / ___/ __/ / / / _ \/ ___/ / / /____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__  /
                                    /____/
 
 
VERSION INFO:
 
Artillery: 2.0.21
Node.js:   v20.18.1
OS:        Darwin

Running this command:

npx artillery run minimal-artillery.yaml

I expected to see this happen:

A successful load test

Instead, this happened:

worker error, id: 1 Error: Callback was already called.
    at …/node_modules/@artilleryio/int-commons/node_modules/async/dist/async.js:969:32
    at captured (…/node_modules/@artilleryio/int-core/lib/engine_ws.js:110:16)
    at …/node_modules/@artilleryio/int-commons/engine_util.js:498:16
    at …/node_modules/@artilleryio/int-commons/node_modules/async/dist/async.js:473:16
    at replenish (…/node_modules/@artilleryio/int-commons/node_modules/async/dist/async.js:1009:25)
    at …/node_modules/@artilleryio/int-commons/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (…/node_modules/@artilleryio/int-commons/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (…/node_modules/@artilleryio/int-commons/node_modules/async/dist/async.js:1049:16)
    at Object.captureOrMatch (…/node_modules/@artilleryio/int-commons/engine_util.js:414:9)
    at WebSocket.messageHandler (…/node_modules/@artilleryio/int-core/lib/engine_ws.js:68:16)

Files being used:

minimal-artillery.yaml

config:
  target: "ws://localhost:8080"
  phases:
    - name: warm up
      duration: 20
      rampTo: 2
  plugins:
    metrics-by-endpoint: {}
scenarios:
  - name: Test
    engine: ws
    flow:
      - wait:
          capture:
            json: "$"
            as: "result"

minimal-websocket.js

import { WebSocketServer } from "ws";
 
const wss = new WebSocketServer({ port: 8080 });
 
wss.on("connection", function connection(ws) {
                for (let i = 0; i < 20; i++) {
                                ws.send("something");
                }
                ws.terminate();
});