testing for vs code extension

[felixhaeberle]

Problem

Currently, we lack a proper testing strategy for the vs code extension.

Context: We oftentimes get bugs or have misleading behavior because of changes we do which we can't properly test or it's simply too time-consuming to do so.

Expected behavior

We should implement a basic set of e2e tests that start the extension and test basic functionalities.

The scope of this e2e test should at least cover all the commands & decorations we have so far.

must-have tests for functionality:

• load project
• auto-gen project.inlang.json
• message preview
• edit message
• extract message
• open message in editor
• display lint
• display error

nice-to-have tests for functionality:

• detect properties missing in plugin ("app.inlang.ideExtension")
• display context tooltip

Tasks

• Make E2E run on Windows (Lots of uncertainty, but probably it's not much work left: ~4h): Used 0.75h
• E2E test: load an example project and verify loading status (2h)
• E2E test: edit a message (2h)
• Unit test: auto-gen project.inlang.json (1h)
• Unit test: message preview (hard to estimate, because I don't know how difficult it is to mock into the tooltip rendering and snapshot the HTML there: ~4h)
• Unit test: edit message (0.5h)
• Unit test: extract message (0.5h)
• Unit test: open message in editor (0.5h)
• Unit test: display lint (1h)
• Unit test: display error (1h)

Other information

cc @openscript What do you think? What is your estimate & how would you test the vs code extension?

[openscript]

Everything which can be tested with unit tests should be unit tests. E2E tests are expensive: more work to create, harder to maintain, take longer to execute, ...

I would go for an incomplete set of E2E tests (smoke testing alike):

1. Load a project
2. Edit a message

The rest I would do as unit tests (with quite some vscode mocking). If we find out, that some features we should have tested E2E, we can add it later on.

What do you think @felixhaeberle ?

[felixhaeberle]

@openscript You're right! Let's go for the minimal amount of e2e & test the rest with unit tests 👍

[openscript]

• Make E2E run on Windows (Lots of uncertainty, but probably it's not much work left: ~4h): Used 0.75h
• E2E test: load an example project and verify loading status (2h)
• E2E test: edit a message (2h)
• Unit test: auto-gen project.inlang.json (1h)
• Unit test: message preview (hard to estimate, because I don't know how difficult it is to mock into the tooltip rendering and snapshot the HTML there: ~4h)
• Unit test: edit message (0.5h)
• Unit test: extract message (0.5h)
• Unit test: open message in editor (0.5h)
• Unit test: display lint (1h)
• Unit test: display error (1h)

Worst case estimate is about 16.5 hours.

[felixhaeberle]

@openscript Go for it 👍

[openscript]

ETA: 16th of October

[openscript]

New ETA: 24th of October

[openscript]

There is no public API to access/read decorations, hence it's difficult to test this in an e2e scenario:
https://stackoverflow.com/questions/60989904/how-to-get-current-list-of-decorations-or-handle-decoration-position-change-in-v
microsoft/vscode#54147

[openscript]

Without additional tools it's not possible to write proper E2E tests for Visual Studio Code. We need to access the UI renderer and internal API (like notifications). The guide from the VSCode project itself doesn't cover this: https://code.visualstudio.com/api/working-with-extensions/testing-extension

I've found these tools to achieve this:

• https://github.com/webdriverio-community/wdio-vscode-service
• https://github.com/redhat-developer/vscode-extension-tester

We need this, for the very basic tests like "E2E test: load an example project and verify loading status". The setup of this tools will take 2-3 hours.

[openscript]

Interesting talk about this topic:

https://www.youtube.com/watch?v=PhGNTioBUiU

[openscript]

Status update:

The last two weeks I tried to get E2E testing up and running, but so far I didn't complete this part. Microsoft provides @vscode/test-electron, which helps you to download VSCode and run an instance of it for you tests. Unfortunately @vscode/test-electron doesn't include something, to put VSCode on a workbench and assert its reactions, like when the extension trigger a notification, to assert if the notification really happened.

There are two projects that try to fill this gap:

• https://github.com/webdriverio-community/wdio-vscode-service
• https://github.com/redhat-developer/vscode-extension-tester

I went with wdio-vscode-service. It didn't work behind proxy, so I created a custom version. I got a simple "hello world" test running, but I have strange typing errors, which persist until today:

To proceed with wdio-vscode-service we need to:

• Resolve the typing problems
• Change the build.js script
• Find out how to make it load an example project
• Write tests

[openscript]

Hi @felixhaeberle

This week I've invested 15h into the E2E tests, but I couldn't produce resilient tests, which work with Github Actions. All my tinkering is here #1795.

Along the way there were already several issues:

• Types for custom commands added by wdio-vscode-service webdriverio-community/wdio-vscode-service#95
• selectText() on TextEditor select wrong text when tabs are used for indention webdriverio-community/wdio-vscode-service#96
• ...

How should we proceed? I still think E2E tests are very valuable, as with unit testing we cannot catch things like #1803, but I'm not sure if it's worth to put somebodies main focus on this right now.

[felixhaeberle]

we will introduce high(er) coverage of unit testing with #1876, so closing when merged. going forward, unit testing will be a continuous effort.

[samuelstroschein]

closing as stale / likely has a duplicate in linear