Cypress e2e initial infra

[marcosnav]

Intent

Fixes #2508

PR to have the initial building steps of infra for e2e tests, aiming for consistent reproducibility.

Only includes three tests so far to serve as examples and starting point for more to come:

1. Common checks, basically that the extension is installed and can be opened.
2. Create a new credential and load existing credentials.
3. Deployment of static content.

To try this, just pull the branch and just test/e2e/dev, this command will do the necessary setup and open Cypress.
... and just test/e2e/stop to tear down the containers.

This is not set to run in CI, yet. A separate GH ticket will be created, to determine specifics of adding this e2e process to CI.

Type of Change

• • Bug Fix
• • New Feature
• • Breaking Change
• • Documentation
• • Refactor
• • Tooling

Approach

• Leverage on Docker infra to spin up a Connect and code-server instances.
• Cypress will target these instances for tests.
• Cypress not running in Docker for local runs, a container for Cypress will be needed only for CI, to be addressed in a separate GH issue.

User Impact

There is not a direct impact for the end user.

Automated Tests

Yes

Directions for Reviewers

Try it out with just test/e2e/dev, and just test/e2e/stop to teardown the containers.

Checklist

• I have updated CHANGELOG.md to cover notable changes.

[dotNomad]

Still looking through this, but had a few questions so far

[sagerb]

I attempted to run the three specs, after selecting the Google browser. Looks to be a fundamental issue, any help would be appreciaed.

• common: ran successfullly
• credentials: failed with an error:
  
  (console output is)

Error:       CypressError: `cy.exec('rsconnect bootstrap --raw --jwt-keypath bootstrap-secret.key --server http://localhost:3939')` failed because the command exited with a non-zero code.

Pass `{failOnNonZeroExit: false}` to ignore exit code failures.

Information about the failure:
Code: 1

Stderr:
Traceback (most recent call last):
  File "/Users/billsager/.pyenv/versions/3.8.2/bin/rsconnect", line 5, in <module>
    from rsconnect.main import cli
  File "/Users/billsager/.pyenv/versions/3.8...

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Deployments Section`
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:119310:77)
    at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1777:23)
    at Promise._settlePromiseFromHandler (http://localhost:8080/__cypress/runner/cypress_runner.js:1489:31)
    at Promise._settlePromise (http://localhost:8080/__cypress/runner/cypress_runner.js:1546:18)
    at Promise._settlePromise0 (http://localhost:8080/__cypress/runner/cypress_runner.js:1591:10)
    at Promise._settlePromises (http://localhost:8080/__cypress/runner/cypress_runner.js:1671:18)
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:2362:25)
From previous event:
    at Promise.longStackTracesCaptureStackTrace [as _captureStackTrace] (http://localhost:8080/__cypress/runner/cypress_runner.js:3456:19)
    at Promise._then (http://localhost:8080/__cypress/runner/cypress_runner.js:1209:17)
    at Promise._passThrough (http://localhost:8080/__cypress/runner/cypress_runner.js:4080:17)
    at Promise.lastly.Promise.finally (http://localhost:8080/__cypress/runner/cypress_runner.js:4089:17)
    at Object.onRunnableRun (http://localhost:8080/__cypress/runner/cypress_runner.js:162647:53)
    at $Cypress.action (http://localhost:8080/__cypress/runner/cypress_runner.js:41540:28)
    at Runnable.run (http://localhost:8080/__cypress/runner/cypress_runner.js:145229:13)
    at next (http://localhost:8080/__cypress/runner/cypress_runner.js:155014:10)
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:155058:5)
    at timeslice (http://localhost:8080/__cypress/runner/cypress_runner.js:145569:27)
From Your Spec Code:
    at Context.eval (webpack://publisher-e2e/./support/commands.js:74:5)
    at wrapped (http://localhost:8080/__cypress/runner/cypress_runner.js:137875:43)
index-DwhYwRtt.js:5316 Yielded:     Object

• deployments: failed with an error:
  
  (console output is)

Error:       CypressError: `cy.exec('rsconnect bootstrap --raw --jwt-keypath bootstrap-secret.key --server http://localhost:3939')` failed because the command exited with a non-zero code.

Pass `{failOnNonZeroExit: false}` to ignore exit code failures.

Information about the failure:
Code: 1

Stderr:
Traceback (most recent call last):
  File "/Users/billsager/.pyenv/versions/3.8.2/bin/rsconnect", line 5, in <module>
    from rsconnect.main import cli
  File "/Users/billsager/.pyenv/versions/3.8...

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Deployments Section`
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:119310:77)
    at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1777:23)
    at Promise._settlePromiseFromHandler (http://localhost:8080/__cypress/runner/cypress_runner.js:1489:31)
    at Promise._settlePromise (http://localhost:8080/__cypress/runner/cypress_runner.js:1546:18)
    at Promise._settlePromise0 (http://localhost:8080/__cypress/runner/cypress_runner.js:1591:10)
    at Promise._settlePromises (http://localhost:8080/__cypress/runner/cypress_runner.js:1671:18)
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:2362:25)
From previous event:
    at Promise.longStackTracesCaptureStackTrace [as _captureStackTrace] (http://localhost:8080/__cypress/runner/cypress_runner.js:3456:19)
    at Promise._then (http://localhost:8080/__cypress/runner/cypress_runner.js:1209:17)
    at Promise._passThrough (http://localhost:8080/__cypress/runner/cypress_runner.js:4080:17)
    at Promise.lastly.Promise.finally (http://localhost:8080/__cypress/runner/cypress_runner.js:4089:17)
    at Object.onRunnableRun (http://localhost:8080/__cypress/runner/cypress_runner.js:162647:53)
    at $Cypress.action (http://localhost:8080/__cypress/runner/cypress_runner.js:41540:28)
    at Runnable.run (http://localhost:8080/__cypress/runner/cypress_runner.js:145229:13)
    at next (http://localhost:8080/__cypress/runner/cypress_runner.js:155014:10)
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:155058:5)
    at timeslice (http://localhost:8080/__cypress/runner/cypress_runner.js:145569:27)
From Your Spec Code:
    at Context.eval (webpack://publisher-e2e/./support/commands.js:74:5)
    at wrapped (http://localhost:8080/__cypress/runner/cypress_runner.js:137875:43)
index-DwhYwRtt.js:5316 Yielded:     {stdout: '', stderr: `Traceback (most recent call last):\n  File "/Users/…bssl.1.1.dylib' (no such file, not in dyld cache)`, code: 1}

[marcosnav]

@sagerb The error points to the command used to bootstrap an admin user API key, that is done through rsconnect bootstrap, you may need to update your local version of rsconnect.

Thanks for trying it out!

[dotNomad]

I attempted to run just/e2e/dev and got the Cypress UI.

I first ran into an error where I didn't have rsconnect so I created a Python venv and installed rsconnect-python which got me past that.

Next I ran into an error in the Credentials test:

The text didn't match because of a URL validation error.

The deployments spec also did not pass for me because of Cypress env BOOTSTRAP_ADMIN_API_KEY is empty. Cannot set admin credentials.

Is there additional setup I'm missing?

[marcosnav]

@dotNomad

It could be that not having rsconnect from the beginning left some bad state

Could you try the following?:

just test/e2e/stop
just test/e2e/dev

Also, it could be that the Connect instance of the e2e tests is not running. You may have to double check that there is no other Connect instance running with the same ports that the e2e Connect container wants to pick up.

[dotNomad]

It could be that not having rsconnect from the beginning left some bad state

I didn't run test/e2e/stop in between the failures and the re-test after getting rsconnect working, but this time I did.

No difference - I'm still seeing both of the errors above.

[sagerb]

Validated functionality, and it works! Woo hoo!!!! This is a great start for this functionality. Thanks for working through this.

I recorded some notes through the process:

1. why is it only using connect version rstudio-connect_2024.12.0?
2. I had a conflict with my dev connect server already running, perhaps you should use a different port when you launch the connect server used for the tests?
3. Is there a faster way than starting the just test/e2e/dev all over again, when you have an error? I’ve already built the publisher and seems like I don’t need to do it again.

For posterity... I had some environmental issues locally which I had to resolve. I hit the error once again that I had reported earlier. I narrowed it down to an issue within my environment by being unable to run rsconnect version without a stack trace being displayed.

With this, I concluded that my rsconnect-python installation was invalid. To resolve, I:
- created a virtual environment for a newer version of python, from within VSCode using the command: Python: Select Interpreter and then choosing to create a virtual environment using .venv for a newer version of python.
- I then restarted a terminal from within VSCode, which activated that virtual python environment. I verified this with the command python --version.
- I then uninstalled the previous version of rsconnect-python with pip uninstall rsconnect-python.
- I then installed a fresh copy with pip install rsconnect-python and verified its functionality with the command rsconnect version
- I then retested, using the command just test/e2e/dev

Great Job Marcos!

[marcosnav]

Thanks @sagerb, definitely great feedback to make this better in future PRs 🙇