Mismatching playwright versions stopping tasks from running

[sam-barker]

Version info:

2.0.21

Running this command:

      artillery run-fargate \
          --region eu-west-1 \
          --dotenv ./.env \
          --cpu $CPU \
          --memory $MEM \
          --max-duration $MAX_DURATION \
          --count $TASK_COUNT \
          --cluster $CLUSTER_NAME \
          --subnet-ids $(aws ssm get-parameter --name /VpcConfig/SubnetIds --region $REGION --output text --query Parameter.Value) \
          --security-group-ids $(aws ssm get-parameter --name /VpcConfig/SecurityGroupIds --region $REGION --output text --query Parameter.Value) \
          ${CONFIG_FILE} \
          --quiet \
          $FLAGS \

I expected to see this happen:

We expected to be able to run the tests as usual

Instead, this happened:

We haven't made any changes or updated any dependencies, but it seems that playwright have published a new version and so it is preventing us from using 1.48.0, even though I have explicitly pinned every version of everything to do with playwright to this version i.e.

  "resolutions": {
    "@playwright/test": "1.48.0",
    "@playwright/browser-chromium": "1.48.0",
    "playwright": "1.48.0",
    "artillery/@playwright/test": "1.48.0",
    "artillery/playwright": "1.48.0",
    "artillery-engine-playwright/@playwright/test": "1.48.0",
    "artillery-engine-playwright/@playwright/browser-chromium": "1.48.0",
    "artillery-engine-playwright/playwright": "1.48.0"
  },

It looks like the docker image is the cause of this. Is there any way around this other than having to wait for a new version of artillery?

[sam-barker]

For some more info: doesn't look like it happens to all the tasks. It just takes one of these to occur and the whole test run stops because of it. Which is even weirder that this isn't happening to all of the tasks.

Also seems to happen on the leader task

[hassy]

Thanks @sam-barker - this looks like a bug in Artillery - v2.0.21 did not pin to a specific version of Playwright - https://github.com/artilleryio/artillery/blob/main/packages/artillery-engine-playwright/package.json#L23-L24 - so a new patch release can break worker containers

[sam-barker]

@hassy I have tried downgrading artillery to v2.0.20 and artillery-engine-playwright to v1.17.0 and I still get the same problem

As far as I'm aware this should use artillery 1.45.3, so I'm not sure why it's complaining about 1.48.1?

[hassy]

Yep, that version of Artillery has an exact pin to v1.45.3. There must be something in the dependency tree that's pulling in a more recent version of Playwright then - what does the table showing all bundled dependencies show when you trigger the test run with run-fargate?

[sam-barker]

I think we've found it. For some reason we were installing artillery and artillery-engine-playwright and that caused the mismatch. All works with artillery 2.0.20, going to go back to 2.0.21 and see if it's all good

[sam-barker]

So 2.0.20 works but 2.0.21 does not, so we're going to stick with the previous version until something is released that pins the versions of playwright