Sinon Chai calledWithMatch causes structuredClone to error in the test runner

[Sanderovich]

See this repo for a reproduction https://github.com/Sanderovich/web-test-runner-called-with-match-error-reproduction.

When an expect(spy).to.be.calledWithMatch() is unsuccesful the expect throws an AssertionError that web test runner can not handle which causes the following exception.

The crash seems to happen because of the actual property inside the AssertionError. The actual property is set to a function called proxy and structuredClone cannot handle functions (https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm).

A possible solution could be to JSON.stringify() and JSON.parse() the AssertionError before the structuredClone.

[CendioOssman]

This is blocking our (@novnc) migration to web test runner, as we make heavy use of sinon. :/

Have you found any way to work around the issue?

[Sanderovich]

@CendioOssman A quick work around would be to delete the actual property from the error thrown by the expect on failure and re-throw the error so web-test-runner can handle it.

it('stops working when errors', () => {
	const sinonSpy = spy();

	sinonSpy();

	try {
		expect(sinonSpy).to.be.calledWithMatch({ foo: 'bar' });
	} catch (e) {
		delete e.actual;
		throw e;
	}
});

[CendioOssman]

Thanks. That doesn't scale terribly well. And we have 765 such assertions. Any workaround would need to be somewhere more central. :/

[Sanderovich]

For a central work around I would implement the work around in a Chai plugin:

import { use, Assertion } from 'chai';
import sinonChai from 'sinon-chai';

function overrideCallWithMatch() {
	Assertion.overwriteMethod('calledWithMatch', _super => {
		return function() {
			try {
				_super.apply(this, arguments)
			} catch (error) {
				delete error.actual;
				throw error;
			}
		}
	});
}

use(sinonChai);
use(overrideCallWithMatch);

Depending on for which Sinon Chai methods you want to implement the work around you can extend the plugin. Would this help you @CendioOssman ?

[lideen]

I have the same issue. This is an issue with any value that structuredClone cannot handle. In my projects tests started failing for HTMLElements as well as they also cannot be cloned.

My current workaround is to lock @web/[email protected]

[rschristian]

I was running into this with sinon-chai but saw no errors in the browser console at all -- seemingly just a failed test that wasn't properly detected. Wasn't using calledWithMatch either.

Both workarounds (wrapper and pinning to @web/[email protected]) worked for me though.

[CendioOssman]

Depending on for which Sinon Chai methods you want to implement the work around you can extend the plugin. Would this help you @CendioOssman ?

Possibly. It is a bit annoying that you have to do it for a bunch of methods, though. But I supposed that can be optimized a bit.

My current workaround is to lock @web/[email protected]

This workaround does not work for me. Is there a regression in 0.7.2 that causes this problem?

[lideen]

Depending on for which Sinon Chai methods you want to implement the work around you can extend the plugin. Would this help you @CendioOssman ?

Possibly. It is a bit annoying that you have to do it for a bunch of methods, though. But I supposed that can be optimized a bit.

My current workaround is to lock @web/[email protected]

This workaround does not work for me. Is there a regression in 0.7.2 that causes this problem?

I think it is this change that causes the problem: 4a4b699. To be more precise, the usage of structuredClone in packages/dev-server-core/src/web-sockets/webSocketsPlugin.ts

[CendioOssman]

Hmm.. Very odd that it didn't resolve the issue here for me in that case.

@lucaelin, is this issue something you are aware of and working on?

[lucaelin]

Hey, thanks for bringing this second issue to my attention, I will see what I can do this weekend,

[lucaelin]

From my initial analysis there are multiple different issues at play here:

1. Circular Objects (which the original implementation solved)
2. Frozen objects, like Redux state (which got fixed in 0.7.2)
3. Objects with getter-only properties, like window (which I don't think ever worked)
4. Non-Transferable objects, like HTMLElement (which broke in 0.7.2)

Fixing 4) should be as simple as catching the clone error and running the original serialization, BUT this would only work if the .actual is not also frozen somewhere within its tree (but I assume that this is an edge case)
Fixing 3) on the other hand, I don't know how to approach, especially since we probably want to avoid mutation on window during serialization, and these objects may very well be mix of all 4 cases simultaneously. Maybe we could detect these and instead replace them with some special keyword, like '[unserializable]', analog to the existing '[Circular]' keyword, but I have no idea on how to reliably detect those.

[bashmish]

@lucaelin that's great analyses, thanks!

do you think you can write some unit tests for each of this use-cases that we need to support? that would be a great way to start a PR, I'd really appreciate it

we can then search for some better algorithms for serialization, the one we use currently is taken from https://github.com/davidmarkclements/fast-safe-stringify which hasn't been update for 3 years now, so probably there are better alternativies in the open source world now
as a last resort, we can also use [unserializable] in place of objects that can't be serialized to anything useful, or at least until there is a better way, so that we can unblock people now

[lucaelin]

@bashmish Just did, see the draft PR I created. I already had those sitting around from my analysis. I wrote a message on the discord back then, asking how to proceed with this issue, let me paste it here for reference:

Original Discord message

I could use some second opinions on this though. As described in my comment on the issue there are 4 cases we need to consider as potential actual values in the message. My problem right now is that the whole serialization implementation makes use of mutation, which in case of objects like window, isn't really ideal and additionally impossible in case of immutability. I previously fixed the immutability by using structuredClone, but that obviously causes problems with non-transferable objects. Looking at this again I think a better way to solve this is to avoid mutation, but this would require replacing the entire algorithm, which may bring other issues to the table. How do you feel about this? I could try to modify the algorithm to shallow clone the problematic objects instead of deep cloning ahead of time? What do you think?

This is the problematic code: https://github.com/modernweb-dev/web/blob/master/packages/dev-server-core/src/web-sockets/webSocketsPlugin.ts#L62

we can also use [unserializable] in place of objects that can't be serialized to anything useful

The issue would be reliably detecting whats serializable and what is not, walking down into each child node and replacing only those that are not. This gets a bit messy, because it causes more mutation on those objects and might cause further problems down the line.