Skip to content

howto.md

Latest commit

 

History

History
237 lines (182 loc) · 10.2 KB

howto.md

File metadata and controls

237 lines (182 loc) · 10.2 KB
Raw

How to port the template to a new implementation

This guide walks through turning template/ into a working PyShowwwcase integration for a new backend pipeline. For the big picture see README.md; for worked examples of the decisions involved, see implementations.md.

Throughout, replace <impl> with your integration's short name (e.g. pipex), lowercase, matching [a-z0-9_].

Prerequisites

  • The backend pipeline already runs as a Docker image driven by docker compose run -d <pipeline>, reading its inputs from a known work directory and writing a LAST_RUN_ID file when it finishes. (Get this working in isolation first — PyShowwwcase only orchestrates it.)
  • A Linux host with Docker + docker compose, cron, and nginx (with TLS if you're serving over HTTPS).

Decide three things up front:

  1. The args contract — what file does the pipeline read its inputs from, and in what format? (a batch CSV, a JSON config, a command line…). This becomes ARGS_FILE and the payload your run.html builds.
  2. The completion signal — the pipeline must write LAST_RUN_ID (containing the run id it read from RUN_ID_FILE) when done. This is how the queue knows a job finished.
  3. The show strategy — self-hosted page, redirect to a viewer, or generate-a-manifest-then-redirect (the three patterns in implementations.md).

Part 1 — Create the implementation from the template

cp -r template implementations/<impl>

You now edit four areas: settings, the run form + start_run, the show page + show_project, and the setup/ wiring.

1.1 Settings (pyshowwwcasesite/pyshowwwcasesite/settings.py)

In the # FROM HERE YOUR CUSTOM PYSHOWWWCASE SETTINGS section, set the values for your integration. Keep the same set of keys as the template (only change values) — every project shares the same settings surface:

Setting What to set
REDIRECT_BASE_URL /pyshowwwcase/<impl>
STATIC_URL /pyshowwwcase/<impl>/pyshowwwcase/static/
BASE_DIRECTORY /opt/pyshowwwcase/files/pyshowwwcase/<impl>/
RUN_DIRECTORY ./work/pyshowwwcase/<impl>/ (in-container path the launcher uses)
ARGS_FILE the filename your pipeline reads its inputs from
RUN_ID_FILE usually run_id.txt
OUTPUT_FILE base name of the download zip (usually results)
SHOW_DIRECTORY subfolder where show writes display assets (show, or output if the pipeline writes there directly)
SHOW_IMAGE_SIZE, ALLOW_GUESTS, DEMO_USERLIST, GUEST_*, USER_* per-deployment policy

CSRF_TRUSTED_ORIGINS / ALLOWED_HOSTS are derived from PYSHOWWWCASE_<IMPL>_SERVER_NAME in .env — don't hardcode them.

1.2 The DB connection token (pyshowwwcasesite/pyshowwwcase/database.py)

database.py reads the MySQL credentials from environment variables named MYSQL_PYSHOWWWCASE_<IMPL>_USER / _PASSWORD / _DATABASE. Make sure the <IMPL> token matches the keys you'll define in .env (this is a classic copy-paste bug — the template historically still pointed at PIPEX).

1.3 The run form (templates/pyshowwwcase/run.html) and start_run

This is where you turn user input into the pipeline's args file.

  • run.html: build the form fields your pipeline needs. The startRun() JS assembles the payload string (the future contents of ARGS_FILE) and puts it in the hidden payload field, then submits. Use placeholders for values the server must fill in — by convention ###dir### for the project's output directory and ###img### for an uploaded-image path. Sanitize any field that becomes a filename on disk.
  • start_run (in the # CUSTOM CODE block of views.py): validate the project, clear stale results (SHOW_DIRECTORY), substitute the placeholders against RUN_DIRECTORY + the project's relative path, enforce the run-count limit, and insert the queue row. Keep a cheap guard that the payload actually contains your expected placeholder/token before enqueuing.

See proticelli for building the payload as JSON via json.dumps (safe against injection from user fields) and hpaseg for the uploaded-file-vs-external-URL substitution.

1.4 The show page (templates/pyshowwwcase/show.html) and show_project

Pick one of the three strategies and adapt the matching reference:

  • Self-hosted page (pipex): in show_project, convert each pipeline output into a display image under SHOW_DIRECTORY, build a context of the URLs, and render show.html. serve_image streams files from SHOW_DIRECTORY (always through safe_path).
  • Redirect to a viewer (proticelli): record the project in the session and redirect to the external viewer URL; add a viewer(request, filename) view + a viewer/<path:filename> route in urls.py to serve the files it needs.
  • Generate a manifest, then redirect (hpaseg): convert outputs to display layers, write the viewer's manifest (e.g. an EllViewer metadata.csv) into SHOW_DIRECTORY, then redirect to the viewer pointed at that manifest; serve both via the viewer view.

If show opens an external viewer, set home.html's showProject() to window.open(...) so it launches in a new tab (see proticelli/hpaseg).

1.5 Keep the generic parts generic

Do not edit anything outside the custom block, the two templates, the settings, and setup/. The base views, queue.py, security.py, forms.py, the shared HTML, and urls.py (apart from any viewer route you add) must stay byte-identical to template/. Fixes to shared behavior belong in template/ and are then propagated to all implementations.

Part 2 — Deploy to a host

This mirrors how the reference implementations are deployed. Assume the install root is /opt.

2.1 Get the pipeline running

Install the backend pipeline (e.g. at /opt/<pipeline>) as a Docker image and confirm docker compose run -d <pipeline> works against its work directory, reading ARGS_FILE and writing LAST_RUN_ID. Set its resource limits in its own docker-compose.yml. Don't proceed until the pipeline works standalone.

2.2 Lay down the web app

# the app itself (without its setup/ subfolder)
cp -r implementations/<impl> /opt/pyshowwwcase/<impl>      # drop the setup/ subfolder here

# the deployment artifacts go to /opt
cp implementations/<impl>/setup/docker-compose.yml /opt/
cp implementations/<impl>/setup/.env.example       /opt/.env
cp implementations/<impl>/setup/pyshowwwcase.sh    /opt/pyshowwwcase/

Edit /opt/.env: set real MariaDB passwords, the SECRET_KEY, the ports, PYSHOWWWCASE_<IMPL>_FILES_ROOT (the host data root), and PYSHOWWWCASE_<IMPL>_SERVER_NAME (your hostname(s)). Review /opt/docker-compose.yml (network name, volume mounts, the *_FILES_ROOT mount).

2.3 Database

cd /opt
docker compose up -d --build maria_db
docker exec -it maria_db bash

Inside the container:

  • create the database pyshowwwcase_<impl> and the user pyshowwwcase_<impl>, granting it all privileges on that database;

  • load the schema from setup/init.sql into the database (creates users and queue);

  • add a PyShowwwcase user. Passwords are unsalted SHA-256. For example, for the password sysadm:

    insert into users (user_id, username, password, settings)
values (1, 'sysadm',
        'd577adc54e95f42f15de2e7c134669888b7d6fb74df97bd62cb4f5b73c281db4', '');

    Generate a hash with: python -c "from hashlib import sha256; print(sha256('sysadm'.encode()).hexdigest())". Users are added directly in the DB like this (there's no signup flow).

2.4 Cron + nginx

  • Add the two lines from setup/crontab to root's crontab. They run every minute:
    • sh /opt/pyshowwwcase/pyshowwwcase.sh — the host launcher (flock-guarded);
    • docker exec -i pyshowwwcase_<impl> sh run_queue.sh — the in-container queue state machine.
  • Add setup/pyshowwwcase_nginx to nginx (e.g. into sites-available and symlink into sites-enabled). It proxies /pyshowwwcase/<impl> to the web container and serves /pyshowwwcase/<impl>/pyshowwwcase/static/ directly. Set server_name and client_max_body_size (large, for uploads). Reload nginx.

2.5 Build & launch the web app

cd /opt
docker compose up -d --build pyshowwwcase_<impl>

Browse to https://<your-host>/pyshowwwcase/<impl> and log in.

Wiring check. pyshowwwcase.sh moves the args file into the pipeline's work dir, so the launcher's paths must agree with the pipeline's work directory (e.g. HPASEG_WORK=/data/<impl> for hpaseg). If runs enqueue but never start, check: the pipeline's work-dir env var, that both cron jobs are firing, and that LAST_RUN_ID is being written/read at the path the queue expects.

Part 3 — Smoke test

  1. Log in (or use guest access), create a project, upload a test input.
  2. Open run, fill the form, submit. The job should appear on the queue page as queued, then running.
  3. Within ~1–2 minutes the launcher starts the pipeline; when it finishes, LAST_RUN_ID is written and the next queue tick clears the job.
  4. Open show — your results should render (or the viewer should open).
  5. download the project to confirm the result zip is produced.

If something stalls, the logs to check are <FILES_ROOT>/queue_log.txt (the state machine), the pipeline's own log.txt in its work dir, and the gunicorn logs in the web container.

Checklist

  • cp -r template implementations/<impl>
  • settings: REDIRECT_BASE_URL, STATIC_URL, BASE_DIRECTORY, RUN_DIRECTORY, ARGS_FILE, SHOW_DIRECTORY (+ policy limits)
  • database.py env token matches <IMPL>
  • run.html + start_run build the args payload and substitute placeholders
  • show.html + show_project implement the chosen show strategy (+ viewer route if redirecting to a viewer)
  • generic files left byte-identical to template/
  • setup/.env.example (copied to /opt/.env), docker-compose.yml, init.sql, pyshowwwcase_nginx, pyshowwwcase.sh, crontab filled in for the host
  • DB created + schema loaded + a user inserted
  • both cron jobs installed; nginx configured & reloaded
  • smoke test passes end to end