Skip to content

Commit 9d1ac25

Browse files
authored
bugfixes for templating (#28)
* bugfixes for templating Signed-off-by: vsoch <[email protected]>
1 parent 5309785 commit 9d1ac25

11 files changed

+23
-38
lines changed

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
1414
The versions coincide with releases on pip. Only major versions will be released as tags on Github.
1515

1616
## [0.0.x](https://github.com/tunel-apps/tunel/tree/main) (0.0.x)
17+
- bugfix template generation (0.0.16)
1718
- support for docker launcher (and intending to test podman) (0.0.15)
1819
- adding support for an app to have multiple launchers (0.0.14)
1920
- adding code server app and commands for apps (0.0.13)

apps/_library/htcondor-job.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ script: "job.sh"
66
maintainer: "@vsoch"
77
github: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/htcondor/job/app.yaml"
88
script_url: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/htcondor/job/job.sh"
9-
updated_at: "2022-08-14 12:59:18.898600"
9+
updated_at: "2022-08-15 17:56:48.522690"
1010
description: "A simple example to launch an HTCondor job (to sleep)"
1111
config: {'launcher': 'htcondor', 'description': 'A simple example to launch an HTCondor job (to sleep)', 'script': 'job.sh', 'args': [{'name': 'cpus', 'description': 'The number of CPUs to allocate for the job (defaults to 1)'}, {'name': 'memory', 'description': 'The memory (in MB, without writing MB) for the job'}, {'name': 'disk', 'description': 'The disk space (also in GB, without the GB suffix) for the job'}, {'name': 'njobs', 'description': 'The number of jobs to launch of this type (defaults to 1)'}]}
1212
---

apps/_library/singularity-socket-code-server.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ script: "app.sh"
66
maintainer: "@vsoch"
77
github: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/singularity/socket/code-server/app.yaml"
88
script_url: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/singularity/socket/code-server/app.sh"
9-
updated_at: "2022-08-14 12:59:18.917422"
9+
updated_at: "2022-08-15 17:56:48.541716"
1010
description: "Work on files on your remote machine via VS Code in the browser, all via unix sockets"
1111
config: {'launcher': 'singularity', 'script': 'app.sh', 'description': 'Work on files on your remote machine via VS Code in the browser, all via unix sockets', 'needs': {'socket': True}, 'examples': '# Run app on login node with singularity, shorter pattern match\ntunel run-app waffles code-server\ntunel run-app waffles singularity/socket/code-server\n', 'commands': {'post': 'cat $socket_dir/home/.config/code-server/config.yaml'}, 'args': [{'name': 'workdir', 'description': 'Working directory for app (and to show file explorer for)'}, {'name': 'container', 'description': 'Change the app container used (default is demo ghcr.io/tunel-apps/tunel-django). Must start with container URI to pull (e.g., docker://)'}, {'name': 'tag', 'description': 'Tag of the container to use (defaults to latest)'}, {'name': 'pull', 'description': 'force a new pull (even if the container already exists).'}]}
1212
---

apps/_library/singularity-socket-jupyter.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ script: "jupyter.sh"
66
maintainer: "@vsoch"
77
github: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/singularity/socket/jupyter/app.yaml"
88
script_url: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/singularity/socket/jupyter/jupyter.sh"
9-
updated_at: "2022-08-14 12:59:18.922122"
9+
updated_at: "2022-08-15 17:56:48.547041"
1010
description: "A singularity jupyter notebook or lab to run directly on a remote (e.g., head node)"
1111
config: {'launcher': 'singularity', 'script': 'jupyter.sh', 'description': 'A singularity jupyter notebook or lab to run directly on a remote (e.g., head node)', 'needs': {'socket': True}, 'examples': '# Run jupyter notebook on login node with custom container\ntunel run-app waffles singularity/socket/jupyter --container=docker://jupyter/datascience-notebook\n# Run Jupyterlab login node (via Singularity container) on open science grid\ntunel run-app osg singularity/socket/jupyter --jupyterlab=true\n', 'args': [{'name': 'container', 'description': 'Change the jupyter container used (default is datascience notebook). Must start with container URI to pull (e.g., docker://)'}, {'name': 'jupyterlab', 'description': 'Try running jupyterlab instead (e,g. set to true to enable)'}, {'name': 'modules', 'description': 'comma separated list of modules to load', 'split': ','}]}
1212
---

apps/_library/slurm-port-jupyter.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ script: "jupyter.sbatch"
66
maintainer: "@vsoch"
77
github: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/slurm/port/jupyter/app.yaml"
88
script_url: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/slurm/port/jupyter/jupyter.sbatch"
9-
updated_at: "2022-08-14 12:59:18.912650"
9+
updated_at: "2022-08-15 17:56:48.536849"
1010
description: "A Jupyter notebook intended to be run with a slurm job, interactive via a port"
1111
config: {'launcher': 'slurm', 'script': 'jupyter.sbatch', 'description': 'A Jupyter notebook intended to be run with a slurm job, interactive via a port', 'args': [{'name': 'workdir', 'description': 'Working directory for the notebook'}, {'name': 'jupyterlab', 'description': 'Try running jupyterlab instead (e,g. set to true to enable)'}, {'name': 'modules', 'description': 'comma separated list of modules to load', 'split': ','}]}
1212
---

apps/_library/slurm-socket-jupyter.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ script: "jupyter.sbatch"
66
maintainer: "@vsoch"
77
github: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/slurm/socket/jupyter/app.yaml"
88
script_url: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/slurm/socket/jupyter/jupyter.sbatch"
9-
updated_at: "2022-08-14 12:59:18.907929"
9+
updated_at: "2022-08-15 17:56:48.532073"
1010
description: "A Jupyter notebook intended to be run with a slurm job, interactive via a socket."
1111
config: {'launcher': 'slurm', 'script': 'jupyter.sbatch', 'description': 'A Jupyter notebook intended to be run with a slurm job, interactive via a socket.', 'args': [{'name': 'workdir', 'description': 'Working directory for the notebook'}, {'name': 'jupyterlab', 'description': 'Try running jupyterlab instead (e,g. set to true to enable)'}, {'name': 'modules', 'description': 'comma separated list of modules to load', 'split': ','}], 'needs': {'socket': True}}
1212
---

apps/_library/slurm-socket-singularity-jupyter.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ script: "jupyter.sbatch"
66
maintainer: "@vsoch"
77
github: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/slurm/socket/singularity-jupyter/app.yaml"
88
script_url: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/slurm/socket/singularity-jupyter/jupyter.sbatch"
9-
updated_at: "2022-08-14 12:59:18.903350"
9+
updated_at: "2022-08-15 17:56:48.527485"
1010
description: "A jupyter notebook (or lab) intended to be run in a Singularity container."
1111
config: {'launcher': 'slurm', 'script': 'jupyter.sbatch', 'description': 'A jupyter notebook (or lab) intended to be run in a Singularity container.', 'args': [{'name': 'jupyterlab', 'description': 'Try running jupyterlab instead (e,g. set to true to enable)'}, {'name': 'workdir', 'description': 'Working directory for the notebook'}, {'name': 'modules', 'description': 'comma separated list of modules to load', 'split': ','}], 'needs': {'socket': True}}
1212
---

apps/_library/socket-tunel-django.md

Lines changed: 6 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ script: "app.sh"
66
maintainer: "@vsoch"
77
github: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/socket/tunel-django/app.yaml"
88
script_url: "https://github.com/tunel-apps/tunel/blob/main/tunel/apps/socket/tunel-django/app.sh"
9-
updated_at: "2022-08-14 12:59:18.893928"
9+
updated_at: "2022-08-15 17:56:48.518220"
1010
description: "An example Django application run via a Singularity container."
11-
config: {'launcher': 'singularity', 'launchers_supported': ['singularity', 'slurm'], 'script': 'app.sh', 'description': 'An example Django application run via a Singularity container.', 'needs': {'socket': True}, 'examples': '# Run app on login node with singularity, custom tag dev\ntunel run-app waffles singularity/socket/tunel-django --tag=dev\n# Force a new pull\ntunel run-app waffles singularity/socket/tunel-django --tag=dev --pull\n# Set a custom user/password (note that the UI is only available to you so this is not for security)\ntunel run-app waffles singularity/socket/tunel-django --tag=dev --user=hello --pass=moto \n', 'args': [{'name': 'user', 'description': 'username for logging into Django app (NOT your cluster username), defaults to tunel-user'}, {'name': 'pass', 'description': 'password for logging in to Django app (NOT your cluster password), defaults to tunel-pass'}, {'name': 'workdir', 'description': 'Working directory for app (and to show file explorer for)'}, {'name': 'container', 'description': 'Change the app container used (default is demo ghcr.io/tunel-apps/tunel-django). Must start with container URI to pull (e.g., docker://)'}, {'name': 'tag', 'description': 'Tag of the container to use (defaults to latest)'}, {'name': 'pull', 'description': 'force a new pull (even if the container already exists).'}]}
11+
config: {'launcher': 'singularity', 'launchers_supported': ['singularity', 'slurm', 'docker', 'podman'], 'script': 'app.sh', 'description': 'An example Django application run via a Singularity container.', 'needs': {'socket': True}, 'examples': '# Run app on login node with singularity, custom tag dev\ntunel run-app waffles singularity/socket/tunel-django --tag=dev\n# Force a new pull\ntunel run-app waffles singularity/socket/tunel-django --tag=dev --pull\n# Set a custom user/password (note that the UI is only available to you so this is not for security)\ntunel run-app waffles singularity/socket/tunel-django --tag=dev --user=hello --pass=moto \n', 'args': [{'name': 'user', 'description': 'username for logging into Django app (NOT your cluster username), defaults to tunel-user'}, {'name': 'pass', 'description': 'password for logging in to Django app (NOT your cluster password), defaults to tunel-pass'}, {'name': 'workdir', 'description': 'Working directory for app (and to show file explorer for)'}, {'name': 'container', 'description': 'Change the app container used (default is demo ghcr.io/tunel-apps/tunel-django). Must start with container URI to pull (e.g., docker://)'}, {'name': 'tag', 'description': 'Tag of the container to use (defaults to latest)'}, {'name': 'pull', 'description': 'force a new pull (even if the container already exists).'}]}
1212
---
1313

1414
### Usage
@@ -93,8 +93,10 @@ tunel run-app waffles singularity/socket/tunel-django --tag=dev --user=hello --p
9393
This app uses the singularity launcher by default, and supports the following:
9494

9595

96-
- singularity
9796
- slurm
97+
- singularity
98+
- docker
99+
- podman
98100

99101
```bash
100102
{% raw %}#!/bin/bash
@@ -138,29 +140,7 @@ STATIC_DIR=${SOCKET_DIR}/static
138140
{% for path in paths %}export PATH={{ path }}:${PATH}
139141
{% endfor %}
140142

141-
# Just pull to tmp for now so cleaned up
142-
SIF="${SINGULARITY_CACHEDIR}/tunel-django.sif"
143-
CONTAINER="{% if args.container %}{{ args.container }}{% else %}docker://ghcr.io/tunel-apps/tunel-django:{% if args.tag %}{{ args.tag }}{% else %}latest{% endif %}{% endif %}"
144-
145-
# First effort
146-
if command -v singularity &> /dev/null
147-
then
148-
printf "singularity pull ${CONTAINER}\n"
149-
150-
mkdir -p ${DB_DIR} ${STATIC_DIR}
151-
152-
# Only pull the container if we do not have it yet, or the user requests it
153-
if [[ ! -f "${SIF}" ]] || [[ "{{ args.pull }}" != "" ]]; then
154-
singularity pull --force ${SIF} ${CONTAINER}
155-
fi
156-
157-
# The false at the end ensures we aren't using nginx, but rather uwsgi just with sockets
158-
printf "singularity exec --bind ${DB_DIR}:/code/db --env TUNEL_PASS=***** --env TUNEL_USER=${TUNEL_USER} --bind ${WORKDIR}:/code/data ${SIF} /bin/bash /code/scripts/run_uwsgi.sh ${SOCKET} false\n"
159-
# The bind for WORKDIR to /var/www/data ensures the filesystem explorer works
160-
singularity exec --bind ${DB_DIR}:/code/db --env TUNEL_PASS=${TUNEL_PASS} --env TUNEL_USER=${TUNEL_USER} --bind ${WORKDIR}:/code/static --bind ${WORKDIR}:/code/data ${SIF} /bin/bash /code/scripts/run_uwsgi.sh ${SOCKET} false
161-
else
162-
printf "Singularity is not available.\n"
163-
fi
143+
{% if docker %}{% include "templates/run_docker.sh" %}{% else %}{% include "templates/run_singularity.sh" %}{% endif %}
164144
{% endraw %}
165145
```
166146

tunel/launcher/slurm.py

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
__license__ = "MPL 2.0"
44

55
import os
6+
import shlex
67
import threading
78
import time
89

@@ -210,12 +211,14 @@ def show_logs_instruction(self, logs_prefix):
210211
logger.info("ssh %s cat %s.err" % (self.ssh.server, logs_prefix))
211212
print()
212213

213-
def print_updated_logs(self, logs_prefix, app):
214+
def print_updated_logs(self, logs_prefix, app, socket):
214215
"""
215216
Start a separate thread that regularly checks and prints logs (when there is an updated line)
216217
"""
217218
logs_thread = threading.Thread(
218-
target=post_commands, name="Logger", args=[self.ssh, app, logs_prefix]
219+
target=post_commands,
220+
name="Logger",
221+
args=[self.ssh, app, logs_prefix, socket],
219222
)
220223
logs_thread.start()
221224

@@ -267,11 +270,11 @@ def run(self, cmd, job_name=None, logs_prefix=None, app=None, socket=None):
267270
)
268271
# Create another process to check logs?
269272
if logs_prefix:
270-
self.print_updated_logs(logs_prefix, app)
273+
self.print_updated_logs(logs_prefix, app, socket=socket)
271274
self.ssh.tunnel(machine, socket=socket, app=app)
272275

273276

274-
def post_commands(ssh, app, logs_prefix):
277+
def post_commands(ssh, app, logs_prefix, socket):
275278
"""
276279
Post commands to show logs and any commands->post defined by the app
277280
"""

tunel/template.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@
1111

1212
# Allow includes from this directory OR providing strings
1313
shared_templates_dir = os.path.join(here, "templates")
14-
env = Environment(loader=FileSystemLoader(shared_templates_dir))
1514

1615

1716
class Template:
@@ -41,5 +40,7 @@ def load(self, template_name, template_dir=None):
4140
env = Environment(
4241
loader=FileSystemLoader([shared_templates_dir, template_dir])
4342
)
43+
else:
44+
env = Environment(loader=FileSystemLoader(shared_templates_dir))
4445
template = env.from_string(temp.read())
4546
return template

tunel/version.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
__copyright__ = "Copyright 2021-2022, Vanessa Sochat"
33
__license__ = "MPL 2.0"
44

5-
__version__ = "0.0.15"
5+
__version__ = "0.0.16"
66
AUTHOR = "Vanessa Sochat"
77
NAME = "tunel"
88
PACKAGE_URL = "https://github.com/tunel-apps/tunel"

0 commit comments

Comments
 (0)