| title | High-Performance Computing Cluster (HPC) Remote Development Set-Up Guide | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| output |
|
Attached to accessing superior resources, other benefits of remote development include mobility of access and the ability to use any device to access the remote development (remote-dev). This guide helps you get started with remote development on the EIE cluster with Debian/Ubuntu environment as the host OS. While this tutorial covers VS Code, you can use any editor (or IDE) with remote-dev capabilities. Links to doc sites to some of the common editors are provided at the end of tutorial.
- Login Credentials
- Operating System (OS): Windows 10/Later
- VS Code: Official Site
- OpenSSH Client ( Available in features )
- Local Extensions: Remote-SSH, Remote - SSH: Editing Configuration Files. These can easily be installed by accessing the extensions market place icon on the menu on the left side-pane.
Visit the VSCode Official Site and download the installer. Install the ".exe" file upon successful download.
!!! info Windows Open-SSH Client Features
- Open **Settings**, select **Apps**, then select **Optional Features**
- Scan the list to establish if **OpenSSH*x*server and client), is installed.
- If not, Add a feature, and:
- Find **OpenSSH Client**, then select and Install.
- Find **OpenSSH Server**, then select and install.
Once OpenSSH has been installed, you can connect to OpenSSH Server (On HPC Cluster) from an OpenSSH Client (On your local machine) using windows cmd or PowerShell as follows:-
ssh domain\\username\@servername
cluster Login Node server name is jaguar1.eie.wits.ac.za
Enter yes to add the server to the list of known servers
Enter your password. As a security measure, your password will not be shown. Once successfully connected, you will see the specifications of the cluster.
The server Jaguar1.eie.wits.ac.za is a login node, to access a particular computation node you will need to further establish a connection as per the below example:
ssh domain\\username\@servername
From the above example, the cluster Node server name is: 192.168.1.103.
After entering the password, access to the node will be established as shown below:
Open Visual Studio Code and add the Remote-SSH extension by
selecting extensions (Ctrl+Shift+X), searching Remote-SSH, and
installing the extension as shown below:
After installing the Remote-SSH extension, a Remote Status Bar will appear on the far left as shown below:
Click on the Remote Status Bar, follow this by clicking on Remote-SSH:Connect to Host... as below:
Connect to the cluster using the \username\@servername as detailed in #Section 1.3.1 above and a new window will pop up. On the new window, Visual Studio Code will be connected to the Cluster Login Node with the Remote Status Bar as shown below.
To explore the remote cluster folder structure of your home directory, click Explore on the side panel or press the corresponding shortcut keys (Ctrl+Shift+X). This
option allows the user to drag and drop files between the local
machine and the cluster.
Open cmd prompt and enter the following command:
ssh-keygen -t rsaPress enter to accept the default directory as the location of the keys enter a passphrase or leave the passphrase empty for the default for 'no passphrase'.
This will generate the id_rsa file which contains the private key the id_rsa.pub which contains the public key.
Check if .ssh directory already exists on the cluster by using the following command: ls .ssh .
If it does not exist, create the folder by issuing the following command: mkdir -p .ssh. With this setting, the public key can then be uploaded onto the clusters and authorized. To achieve this, the scp command is issued from the local machine as follows:
scp .ssh/id_rsa.pub server\@somedomain:\~/.ssh/authorized_keysIn the command above, the public key of interest is also authorized in the upload command.
Connecting to the remote cluster and having to deal with password prompts to establish each session may prove to be a tedious process. This can be addressed by setting up SSH configuration settings using a config file. To this end, open VS-Code and create a configuration file by clicking the remote status bar and selecting the configuration file as shown below:
!!! abstract C:\User\username\.ssh> config
```bash
Host jaguar1
HostName jaguar1.eie.wits.ac.za # Login Node server name
user username # your username
IdentityFile C:/users/username/.ssh/id_rsa # Location of private key
Host gpu-node
HostName 192.168.1.103 # gpu-node server name
user username # your username
ProxyCommand ssh -W %h:%p jaguar1
```
!!! info The following are required for remote access dev:
- Login Credentials,
- Operating System (OS): Ubuntu/Debian Distro,
- VS Code: Official Site,
- OpenSSH Client: Installation command ( sudo apt-get install openssh-client ),
- Local Extensions: Remote-SSH, Remote - SSH: Editing Configuration Files. These can easily be installed by accessing the extensions market place icon on the menu on the left side-pane.
To install the SSH-client, open the terminal on your local machine and run the following commands:
sudo apt install openssh-client
The VSCode editor is installed by issuing the following commands:
sudo apt install apt-transport-https
sudo apt update
sudo apt install code # or code-insiders
After installing, you should verify that you can connect to OpenSSH Server (On HPC Cluster) from an OpenSSH Client (On your local machine) the terminal. The cluster Login Node server name is identified by jaguar1.eie.wits.ac.za. To make the first connection, issue the following command, replacing username with your login credential username:
ssh username@jaguar1.eie.wits.ac.za
When prompted to add the login node server the list of known servers, enter yes. You should be prompted for the password associated with your cluster account. Upon successful authentication, you should have access to the terminal on the remote cluster as with the sample below:
The server jaguar1.eie.wits.ac.za is a login node, to access the computation node that is equipped with the development environment, you will need to further establish a connection as shown below (from jagaur1 - the terminal console accessed in the previous step ):
ssh username@192.168.1.103
After entering the password, access to the computation node will be confirmed by a terminal session in GPU-Node server 192.168.1.103, similar to the one displayed above.
To test the SSH connection on VS Code, press (Ctrl+Shift+P) to open the command bar, then search "Remote-SSH" and select "Remote-SSH: Connect to host" from the list that appears. This will prompt for a command; enter the following ssh username@jaguar1.eie.wits.ac.za as shown below:
Enter the password on prompt and connect to the server, this should grant access to the server directory. To explore the remote cluster folder structure of your home directory, click Explore on the side panel or press the corresponding shortcut keys (Ctrl+Shift+E). This option allows the user to drag and drop files between the local machine and the cluster.
So far, setting up an SSH connection via shell or VSCode's command bar requires password authentication for each session. Considering the required frequency and persistence of connections, this approach may prove to be tedious. Authenticating using SSH-Keys introduces the convenience of circumventing password authentication required for every SSH session.
The key pair can be generated on the local machine. This can be achieved using the following commands in the terminal:
# If there is no .ssh/ folder, this can be created using the following command.
mkdir ~/.ssh
# generate keys in the .ssh/ folder
cd ~/.ssh/
~/.ssh-keygen -b 4096 -t rsaAn 'enter a passphrase' will be displayed. This is used to protect your private key in case someone gains access to the machine/account where you have it stored. This can be left blank and it is generally not necessary. The sequence of prompts as shown in the figure below will be displayed:
This will generate the id_rsa file which contains the private key the id_rsa.pub which contains the public key as can be seen in the folder structure.
The public key can be uploaded to the cluster by dragging the '.pub' file to the remote cluster folder structure (specifically the ~/.ssh folder) after a successful ssh connection on VS-code. Alternatively, the upload can be done on the terminal by issuing the following commands:
export USER_AT_HOST="username@jaguar1.eie.wits.ac.za"
export PUBKEYPATH="$HOME/.ssh/id_rsa.pub"
ssh-copy-id -i "$PUBKEYPATH" "$USER_AT_HOST"
Upon successful upload, run the following commands to authorize the public key in the remote server terminal:
cd ~/.ssh # if you are not already in the directory
touch authorized_keys
cp -a authorized_keys authorized_keys.bak # (Optional) Back-up the authorized_keys file
cat ./id_rsa.pub >> authorized_keys
# Correct Folder and File Permissions on the remote server
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keysAn alternatively, the public key can be authorized by copying the contents of the public key into the 'authorized_keys' file using text editors like nano or vim.
After authorizing the public key on the remote server, the next step is to add the remote cluster connection configurations. In VS Code on the local machine, run Remote-SSH: Open Configuration File... in the Command Palette, using (Ctrl+Shift+P), select the SSH config file option, and add (or modify) to add the relevant host entries. Generally, a host entry configuration in the config file is as follows:
!!! example Typical Host Configurations
sh Host name-of-ssh-host-here HostName host-fqdn-or-ip-goes-here User your-user-name-on-host IdentityFile /path/to/private-key or ProxyCommand
For access to the wits cluster, add the following configurations accordingly:
!!! abstract Config File: ~$.ssh/config
```sh
# Login node/server. Functions as proxy server
Host jaguar1
HostName jaguar1.eie.wits.ac.za # Login node/server name
User user # your username
IdentityFile ~/.ssh/id_lin_rsa # Location of private key
# Development Environment Node
Host gpu-node
HostName 192.168.1.103 # dev server name
User user # your username
ProxyCommand ssh -W %h:%p jaguar1 # ssh proxy command through the jaguar1 server
```