fix: build open_terminal from a real fork, remove the dead vendored copy

.github/workflows/docker.yml

@@ -54,6 +54,21 @@ jobs:
             [registry."harbor-core.harbor.svc.cluster.local"]
               ca=["/etc/ssl/certs/ca-certificates.crt"]
 
+      # TEMPORARY, tied to the same fork-build stage in the Dockerfile
+      # (see homelab#822) -- remove once the fork's fixes land upstream
+      # and Dockerfile stage 1 reverts to a plain upstream FROM.
+      # Resolves the fork's CURRENT commit so it's passed as a build-arg
+      # below: Docker's cache keys on RUN command text, not on what a
+      # `git clone --branch main` actually fetches, so without this the
+      # fork-build layer can silently stay cached and stale across runs
+      # even after the fork gets new commits.
+      - name: Resolve open-terminal-app-fork HEAD sha
+        id: fork
+        run: |
+          sha=$(git ls-remote https://github.com/dvystrcil/open-terminal-app-fork.git refs/heads/main | cut -f1)
+          echo "sha=$sha" >> "$GITHUB_OUTPUT"
+          echo "Building against fork main @ $sha"
+
       - name: Build and push
         uses: docker/build-push-action@v7
         with:
@@ -62,6 +77,8 @@ jobs:
           platforms: ${{ matrix.platform }}
           push: true
           provenance: false
+          build-args: |
+            FORK_SHA=${{ steps.fork.outputs.sha }}
           tags: ${{ env.INTERNAL_REGISTRY }}/${{ env.PROJECT }}/${{ env.IMAGE_NAME }}:${{ matrix.platform == 'linux/amd64' && 'dev-amd64' || 'dev-arm64' }}
           cache-from: type=registry,ref=${{ env.INTERNAL_REGISTRY }}/${{ env.PROJECT }}/${{ env.IMAGE_NAME }}:buildcache-${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}
           cache-to: type=registry,ref=${{ env.INTERNAL_REGISTRY }}/${{ env.PROJECT }}/${{ env.IMAGE_NAME }}:buildcache-${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }},mode=max

CHANGELOG.md

@@ -6,6 +6,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Fixed
+
+- 🔒🔌📄 **The `open_terminal/` fixes documented in this CHANGELOG were never actually deployed** — this repo's Dockerfile has always been a thin wrapper around a pre-built upstream image (`FROM .../ghcr-proxy/open-webui/open-terminal:latest`); it never built or installed this repo's own vendored `open_terminal/` package. Every fix this CHANGELOG has described as applied to `open_terminal/main.py` (0.20.40's connection-reset fix, 0.20.41's log-retention fix, and others discovered along the way) was real code, tested, and merged here -- but the running container has always been unmodified upstream code plus wrapper tooling, regardless. See [homelab#822](https://github.com/dvystrcil/homelab/issues/822) for the full incident.
+
+### Changed
+
+- 🍴 **Now builds from `dvystrcil/open-terminal-app-fork`, not upstream `:latest`, temporarily** — Stage 1 of the Dockerfile builds `open_terminal` from source (a real fork, `git clone` + `pip install .`, mirroring upstream's own Dockerfile) instead of pulling the pre-built upstream image. The fork carries the fixes above plus two more found in the same audit (two-tier process-result expiry; new `insert_after`/`append_to_section`/`append` file endpoints with a defensive `replace_file_content` check), submitted upstream as [open-webui/open-terminal#148](https://github.com/open-webui/open-terminal/pull/148), [#149](https://github.com/open-webui/open-terminal/pull/149), [#150](https://github.com/open-webui/open-terminal/pull/150), [#151](https://github.com/open-webui/open-terminal/pull/151). Revert to a plain `FROM .../open-terminal:latest` once all four merge and a release picks them up.
+- 🗑️ **Removed the vendored `open_terminal/` package and its tests** — dead weight now that the actual fixes live in a real fork with a real upstream relationship, not a disconnected local copy that nothing built. `tests/test_actor_env.py` (tests `helpers/bible_bridge.py`, which *is* deployed) is kept; the rest tested only the vendored copy. `pyproject.toml`, `dev.sh`, and `.python-version` (all specific to developing that vendored package) removed too.
+- 🔑 **Ported a homelab-specific fix to the fork** (not submitted upstream -- it's specific to our own GitHub App token-file convention): `refresh_github_token_env()` re-reads the current token from disk into the long-lived Python process's own `os.environ` before every subprocess spawn. Closes a gap `BASH_ENV`-based shell-profile sourcing doesn't cover (the plain-shell and PTY spawn paths never source `/etc/profile.d`). See [dvystrcil/homelab#701](https://github.com/dvystrcil/homelab/issues/701).
+
 ## [0.20.41] - 2026-07-31
 
 ### Fixed

Dockerfile

@@ -1,6 +1,99 @@
-# Wrapper image — extends ghcr.io/open-webui/open-terminal with
-# additional tools and environment configuration.
-FROM harbor-core.harbor.svc.cluster.local/ghcr-proxy/open-webui/open-terminal:latest
+# === STAGE 1: build open_terminal from our fork, not upstream :latest ===
+#
+# TEMPORARY. dvystrcil/open-terminal-app-fork carries fixes not yet in
+# upstream open-webui/open-terminal, submitted as:
+#   - open-webui/open-terminal#148 -- configurable uvicorn keep-alive
+#     timeout (fixes intermittent ConnectionResetError)
+#   - open-webui/open-terminal#149 -- process-log retention security fix
+#     (a log file with no in-memory record was never pruned)
+#   - open-webui/open-terminal#150 -- two-tier process-result expiry
+#     (a slow caller could lose a finished command's result forever)
+#   - open-webui/open-terminal#151 -- insert_after/append_to_section/
+#     append endpoints + a defensive replace_file_content check
+# Plus one homelab-specific commit NOT submitted upstream (GH_TOKEN
+# refresh from disk before every subprocess spawn -- ties into our own
+# entrypoint.sh token-rotation convention, not something upstream has
+# any hook for).
+#
+# Once all four upstream PRs merge and a release picks them up, revert
+# this stage and go back to a plain
+# `FROM harbor-core.../ghcr-proxy/open-webui/open-terminal:latest`
+# (see homelab#822). Mirrors upstream's own Dockerfile build steps
+# exactly, substituting a git clone of our fork for `COPY . .`.
+FROM python:3.12.13 AS fork-build
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        coreutils findutils grep sed gawk diffutils patch \
+        less file tree bc man-db \
+        curl wget net-tools iputils-ping dnsutils netcat-openbsd socat telnet \
+        openssh-client rsync \
+        vim nano \
+        git \
+        build-essential cmake make \
+        perl ruby-full lua5.4 \
+        jq xmlstarlet sqlite3 \
+        ffmpeg pandoc imagemagick texlive-latex-base \
+        zip unzip tar gzip bzip2 xz-utils zstd p7zip-full \
+        procps htop lsof strace sysstat \
+        sudo tmux screen tini iptables ipset dnsmasq \
+        ca-certificates gnupg apt-transport-https \
+        libcap2-bin \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
+    && apt-get install -y --no-install-recommends nodejs \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN curl -fsSL https://get.docker.com | sh
+
+WORKDIR /app
+
+RUN pip install --no-cache-dir \
+    numpy pandas scipy scikit-learn \
+    matplotlib seaborn plotly \
+    jupyter ipython \
+    requests beautifulsoup4 lxml \
+    sqlalchemy psycopg2-binary \
+    pyyaml toml jsonlines \
+    tqdm rich \
+    openpyxl weasyprint \
+    python-docx python-pptx pypdf csvkit
+
+# git clone stands in for upstream's `COPY . .` -- our source lives in a
+# separate fork repo, not this one. FORK_SHA exists purely to bust
+# Docker's build cache: `git clone --branch main` is byte-identical
+# text on every build regardless of what commit main actually points
+# to, so without something that changes per-build in this RUN step,
+# a cached layer silently ships a stale fork clone forever. CI (see
+# docker.yml) resolves the fork's current SHA via `git ls-remote` and
+# passes it explicitly on every run.
+ARG FORK_REF=main
+ARG FORK_SHA=""
+RUN echo "Building open-terminal-app-fork ref=${FORK_REF} sha=${FORK_SHA:-unpinned}" \
+    && git clone --branch "${FORK_REF}" --depth 1 \
+        https://github.com/dvystrcil/open-terminal-app-fork.git /build \
+    && cd /build \
+    && pip install --no-cache-dir . \
+    && cp "$(readlink -f "$(which python3)")" /usr/local/bin/python3-ot \
+    && setcap cap_setgid+ep /usr/local/bin/python3-ot \
+    && sed -i "1s|.*|#!/usr/local/bin/python3-ot|" "$(which open-terminal)" \
+    && rm -rf /build
+
+RUN useradd -m -s /bin/bash user && echo 'user ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
+
+# Matches upstream's own Dockerfile tail exactly -- these are image
+# metadata (ENV/WORKDIR/EXPOSE), not filesystem content, so without
+# restating them here stage 2 (FROM fork-build) would silently lose
+# them. The old single-stage setup got these for free by inheriting
+# straight from the pre-built upstream image; building from source
+# here means restating what upstream's own Dockerfile sets.
+ENV SHELL=/bin/bash
+ENV PATH="/home/user/.local/bin:${PATH}"
+WORKDIR /home/user
+EXPOSE 8000
+
+# === STAGE 2: homelab wrapper -- tools + entrypoint on top of stage 1 ===
+FROM fork-build
 
 USER root
 

README.md

@@ -102,17 +102,19 @@ Plus everything upstream exposes: `OPEN_TERMINAL_PACKAGES`, `OPEN_TERMINAL_PIP_P
 ## Repository layout
 
 ```
-Dockerfile           # FROM ghcr.io/open-webui/open-terminal:latest + tooling
+Dockerfile           # builds open_terminal from dvystrcil/open-terminal-app-fork + tooling
 entrypoint.sh        # secrets resolution, dotfile seeding, helpers, egress, bridge
 helpers/
   bible_bridge.py    # multi-project Story Bible HTTP bridge
   create-pr.sh       # five-step PR workflow
   CONTAINER_TEST_PLAN.md
-open_terminal/       # vendored copy of the upstream Python package (reference)
-dev.sh               # local dev: uv run uvicorn open_terminal.main:app --reload
 ```
 
-> The `open_terminal/` source tree is checked in for reference and local debugging via [dev.sh](dev.sh). The published image runs the upstream `open-terminal` binary from the base image, **not** this local copy — to ship code changes you would need to either pin a custom upstream version or restructure the Dockerfile to install from this tree.
+> **This repo has no vendored copy of `open_terminal`'s Python source.** An earlier attempt at that (checked-in, tested, and documented in this CHANGELOG as if deployed) was never actually built into the image -- the Dockerfile just pulled a pre-built upstream image the whole time. See [homelab#822](https://github.com/dvystrcil/homelab/issues/822).
+>
+> The fix: this repo now genuinely builds `open_terminal` from source, via a real fork, [`dvystrcil/open-terminal-app-fork`](https://github.com/dvystrcil/open-terminal-app-fork) -- Dockerfile stage 1 does `git clone` + `pip install .` against it, mirroring upstream's own build. The fork carries a handful of fixes submitted upstream ([open-webui/open-terminal#148](https://github.com/open-webui/open-terminal/pull/148), [#149](https://github.com/open-webui/open-terminal/pull/149), [#150](https://github.com/open-webui/open-terminal/pull/150), [#151](https://github.com/open-webui/open-terminal/pull/151)) plus one homelab-specific patch (GH_TOKEN refresh, not upstream-appropriate). This is meant to be temporary: once those PRs merge into a real upstream release, switch stage 1 back to a plain `FROM .../open-terminal:latest` and the fork goes away.
+>
+> If you need to change `open_terminal`'s own behavior (not just this wrapper's tooling), make the change in the fork, not here.
 
 ## License