Improvement: End to End tests

[mattheworiordan]

The current test suite provides unit and integration coverage, which is good and appears to work.

However, as we have seen in the recent cursors bug fix PR, race conditions are hard to detect without real network connections and underlying Realtime SDK usage. In addition, it's common practice to have a testing pyramid with some high level e2e tests to ensure the key features work as expected over a real network.

I believe we should improve this SDK with some high level E2E tests for the key features and objects, which will deliver a more robust experience for consumers of this product.

[lawrence-forooghian]

I'm not sure I understand what is the distinction between an E2E test and the integration tests that we currently have, which test the interaction between two clients using the sandbox environment?

[mattheworiordan]

integration tests that we currently have, which test the interaction between two clients using the sandbox environment?

Can you point me at some of those test? I checked Github Actions set up, and it has no environment variables, so assumed it is not testing against actual endpoints. Perhaps I misread that.

[lawrence-forooghian]

I'm referring to the tests in integration.test.ts, which are using the sandbox environment:

spaces/test/integration/integration.test.ts

Lines 27 to 29 in 23f85ac

	[{ spaces: performerSpaces, clientId: performerClientId }, { spaces: observerSpaces }] = await createClients({
	count: 2,
	});

spaces/test/integration/utilities/setup.ts

Lines 6 to 36 in 23f85ac

	/**
	* Fetches a key for the Ably sandbox environment. This key is shared between all callers of this function.
	*/
	const fetchSharedSandboxKey = (() => {
	const sandboxKeyPromise = createSandboxAblyAPIKey();
	return async () => {
	return await sandboxKeyPromise;
	};
	})();
	/**
	* Performs the following part of a test setup:
	*
	* > Given $count Spaces clients, all configured to use the same API key, and each configured to use a different randomly-generated client ID...
	*/
	export async function createClients({ count }: { count: number }) {
	const sandboxKey = await fetchSharedSandboxKey();
	return Array.from({ length: count }, () => {
	const clientId = nanoid();
	const realtime = new Realtime({
	environment: 'sandbox',
	key: sandboxKey,
	clientId: clientId,
	});
	const spaces = new Spaces(realtime);
	return { spaces: spaces, clientId: clientId };
	});
	}

[mattheworiordan]

Oh, oops, thanks @lawrence-forooghian for pointing this out 🙏

I will close this issue 🙃