Test: SSH connection to unsupported distros with custom sysroot

[deepak1556]

Refs: #235232

• anyOS @alexr00

Complexity: 4

Create Issue

---

We plan to provide a path for users on legacy servers to continue using latest version of the remote server if the system contains a sysroot with the required libraries. The necessary steps to achieve this are documented in https://github.com/microsoft/vscode-docs/pull/7953/files. This TPI aims to validate this flow,

Prerequisites:

• Install OpenSSH compatible SSH client, refs https://code.visualstudio.com/docs/remote/troubleshooting#_installing-a-supported-ssh-client
• Unreleased VSCode client from build: remove legacy server support #235232 (comment)
• Latest Pre-release of Remote - SSH extension, version >= 0.117.2025012315
• Sysroot
  • For testing purpose you can use the pre-compiled version that has -glibc-2.28 suffix from https://github.com/microsoft/vscode-linux-build-agent/releases/tag/v20240129-253798
  • If you are feeling adventurous try Build the sysroot step from https://github.com/microsoft/vscode-docs/pull/7953/files
• Patchelf https://github.com/NixOS/patchelf/releases/tag/0.18.0
• Container with SSH server running, you can use the following sample container or spin your own but make sure it is a distro version that used to run legacy servers (Ubuntu <= 18, Debian <= 9, RHEL <= 7)

FROM amazonlinux:2

RUN yum update -y && yum install -y sudo openssh-server tar procps-ng
RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config

RUN useradd -m -s /bin/bash test-ssh-user
RUN usermod -a -G wheel test-ssh-user
RUN echo "test-ssh-user:test123" | chpasswd

EXPOSE 22

Once the container is created, run the following commands to start the ssh server

su test-ssh-user
sudo ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
sudo /usr/sbin/sshd -D

Testing:

• Use the ssh extension to connect to the host, command should end up being ssh test-ssh-user@localhost -p <forwarded port>
• Server connection should fail with error about server missing glibc and libstdc++ prerequisites
• Close the remote connection
• Set the environment variables from step 4) of https://github.com/microsoft/vscode-docs/pull/7953/files to the users ~/.profile or ~/.bash_profile
• Retry connecting to the host via the SSH extension
• Connection should be successful, you should see a prompt about connecting to an unsupported server

[chrisdias]

@deepak1556

Can I get more details on how the set up for this TPI?

• What/where/how to set these prerequisites?

• Sysroot
  • For testing purpose you can use the pre-compiled version from https://github.com/microsoft/vscode-linux-build-agent/releases/tag/v20250107-311735
  • If you are feeling adventurous try Build the sysroot step from https://github.com/microsoft/vscode-docs/pull/7953/files
• Patchelf https://github.com/NixOS/patchelf/releases/tag/0.18.0

• I assume the dockerfile above is the server to connect to?

• Where/how do you set the environment variable, in the container or outside?

[deepak1556]

Yup the docker container mocks a remote with ssh server running in it.

1. Start the container and the ssh server inside the container following the steps above
2. Copy the sysroot archive and the patchelf binary into the container for the architecture you are testing, the architecture can be confirmed with uname -m inside the container
3. Extract the sysroot and patchelf to respective locations in the container
4. Create the environment variables under /home/test-ssh-user/.bash_profile inside the container, an example of how the bash_profile might look

5. Now attempt to connect to this remote via the ssh extension

[chrisdias]

Commands to copy from local file system into container:

docker cp C:\Users\BLAH\Downloads\x86_64-linux-gnu-glibc-2.28.tar.gz bbbe8f:/home/test-ssh-user/x86_64-linux-gnu-glibc-2.28.tar.gz

docker cp C:\Users\BLAH\Downloads\patchelf-0.18.0-x86_64.tar.gz bbbe8f:/home/test-ssh-user/patchelf-0.18.0-x86_64.tar.gz

[deepak1556]

@chrisdias if you use devcontainers, the folders will be mounted automatically into the container.

[chrisdias]

OK I give up :). I've spent a couple of hours trying to get this set up. If you could provide a full devcontainer.json (maybe a "before" and "after") that would speed things up.

Here is how the container is started (container name is 238873:latest).

docker run --rm -it -p 22:22/tcp 238873:latest

Starting the ssh server, I see this (the first two commands seem to work):

[test-ssh-user@c8ce83905e68 /]$ sudo /usr/sbin/sshd -D
Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key

When i try to SSH in (ssh test-ssh-user@localhost -p 22) I see this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:kql8hiT***************DGP0pPvU.
Please contact your system administrator.
Add correct host key in C:\\Users\\cdias/.ssh/known_hosts to get rid of this message.
Offending RSA key in C:\\Users\\cdias/.ssh/known_hosts:1
Host key for localhost has changed and you have requested strict checking.
Host key verification failed.

I was able to copy the tar.gz files in but I not figured out yet where to copy them to (the respective locations). patchelf wasn't installed, so I used yum to install it per Copilot.

Again, I think a couple of devcontainer.json files here might be of great help. I want to test the flow, no so much testing the setup of the containers. any help appreciated.

[dbaeumer]

@deepak1556 pointed out that one tester is enough. Closing the item.

[deepak1556]

Thanks @chrisdias for taking the time to set this up, I could improve on the container setup but for the most part the prerequisites would be deployed to servers. The container was just an attempt to mock the flow.

Given @alexr00 already tested this from windows host and some more additional testing outside this plan item. I think we are at a good place to not spend any additional time on testing it.