Got it working in customer environment (#14)

* Refactoring docker compose files for easier execution and customer PoC usage of repo-converter
* Moved running process check to beginning of loop to skip earlier
* Better handling and checking of child processes
* Deleting abandoned lockfiles
* Starting on batch sizes (not finished)
* Removing psutils from docker image build to speed up build time
* Updating README to match recent changes
* Updating cloud-agent to latest
* Showing more output for LOG_LEVEL=INFO

README.md

@@ -9,7 +9,7 @@ Running src serve-git and the agent together on the same Docker network allows t
 Docker compose also allows for easier upgrades, troubleshooting, monitoring, logging, flexibility of hosting, etc. than running the binaries directly on the OS.
 
 ## Setup - Sourcegraph Staff Only
-1. Add the needed entries to the sourcegraphConnect targetGroups list in the Cloud instance's config.yaml, get your PR approved and merged
+1. Add the needed entries to the sourcegraphConnect targetGroups list in the Cloud instance's config.yaml, and get your PR approved and merged
 ```yaml
         - dnsName: src-serve-git-ubuntu.local
           listeningAddress: 100.100.100.0
@@ -22,20 +22,25 @@ Docker compose also allows for easier upgrades, troubleshooting, monitoring, log
           ports:
           - 443
 ```
-2. Run the Reload frontend GitHub Action, as this seems to be needed for the frontend pods to start using tunnel connections
+2. Run the "Reload Instance for srcconnect config change" GitHub Action, as many containers need to be restarted to pick up tunnel connection config changes
 3. Clone this repo to a customer's bridge VM, install Docker and Docker's Compose plugin
 4. Copy the `config.yaml` and `service-account-key.json` files using the instructions on the instance's Cloud Ops dashboard
- - Paste them into `./config/cloud-agent-config.yaml` and `./config/cloud-agent-service-account-key.json`
-5. Modify the `./config/cloud-agent-config.yaml` file
- - `serviceAccountKeyFile: /sourcegraph/cloud-agent-service-account-key.json` so that the Go binary inside the agent container finds this file in the path that's mapped via the docker-compose.yaml files
- - Only include the `- dialAddress` entries that this cloud agent instance can reach, remove the others, so the Cloud instance doesn't try using this agent instance for code hosts it can't reach
- - Use extra caution when pasting the config.yaml in Windows, as it may use Windows' line endings or extra spaces, which breaks YAML, as a whitespace-dependent format
-6. Clone the customer's repos into the `repos-to-serve` directory at the root of this repo on the bridge VM, or update the volume mount path for the src-serve-git service in the docker-compose.yaml file
-7. Run `docker compose up -d`
-8. Add a Code Host config to the customer's Cloud instance
- - Type: src serve-git
- - URL: "http://src-serve-git-ubuntu.local:443"
- - or
- - URL: "http://src-serve-git-wsl.local:443"
- - Note that the :443 port may be required, as this seems to default to port 443, even when used with http://
-9. Use the repo-converter to convert SVN, TFVC, or Git repos, to Git format, which will store them in the `repos-to-serve` directory, or use any other means to get the repos into the directory
+    - Paste them into `./config/cloud-agent-config.yaml` and `./config/cloud-agent-service-account-key.json`
+5. Modify the contents of the `./config/cloud-agent-config.yaml` file:
+    - `serviceAccountKeyFile: /sourcegraph/cloud-agent-service-account-key.json` so that the Go binary inside the agent container finds this file in the path that's mapped via the docker-compose.yaml files
+    - Only include the `- dialAddress` entries that this cloud agent instance can reach, remove the others, so the Cloud instance doesn't try using this agent instance for code hosts it can't reach
+    - Use extra caution when pasting the config.yaml in Windows, as it may use Windows' line endings or extra spaces, which breaks YAML, as a whitespace-dependent format
+6. Run `docker compose up -d`
+7. Add a Code Host config to the customer's Cloud instance
+    - Type: src serve-git
+    - `"url": "http://src-serve-git-ubuntu.local:443",`
+    - or
+    - `"url": "http://src-serve-git-wsl.local:443",`
+    - Note the port 443, even when used with http://
+8. Use the repo-converter to convert SVN, TFVC, or Git repos, to Git format, which will store them in the `src-serve-root` directory, or use any other means to get the repos into the directory
+    - There are docker-compose.yaml and override files in a few different directories in this repo, separated by use case, so that each use case only needs to run `docker compose up -d` in one directory, and not fuss around with `-f` paths.
+    - The only difference between the docker-compose-override.yaml files in host-ubuntu vs host-wsl is the src-serve-git container's name, which is how we get a separate `dnsName` for each.
+    - If you're using the repo-converter:
+        - If you're using the pre-built images, `cd ./repo-converter && docker compose up -d`
+        - If you're building the Docker images, `cd ./repo-converter/build && docker compose up -d --build`
+        - Either of these will start all 3 containers: cloud-agent, src-serve-git, and the repo-converter

host-wsl2-ubuntu/docker-compose.yaml → config/docker-compose-common-services.yaml

@@ -4,7 +4,7 @@ services:
 
   cloud-agent:
     container_name: cloud-agent
-    image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-05-22-28-333320ee
+    image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-15-04-02-110c2ea9
     volumes:
       - ../config/cloud-agent-service-account-key.json:/sourcegraph/cloud-agent-service-account-key.json:ro
       - ../config/cloud-agent-config.yaml:/sourcegraph/cloud-agent-config.yaml:ro
@@ -15,7 +15,7 @@ services:
 
   src-serve-git:
     # Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
-    container_name: src-serve-git-wsl.local
+    container_name: src-serve-git-ubuntu.local
     image:  index.docker.io/sourcegraph/src-cli:latest
     volumes:
       - ../src-serve-root/:/sourcegraph/src-serve-root:ro

host-ubuntu/docker-compose-override.yaml

@@ -0,0 +1,5 @@
+services:
+
+  src-serve-git:
+    # Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
+    container_name: src-serve-git-ubuntu.local

host-ubuntu/docker-compose.yaml

@@ -1,27 +1,4 @@
-version: '2.4'
-
-services:
-
-  cloud-agent:
-    container_name: cloud-agent
-    image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-05-22-28-333320ee
-    volumes:
-      - ../config/cloud-agent-service-account-key.json:/sourcegraph/cloud-agent-service-account-key.json:ro
-      - ../config/cloud-agent-config.yaml:/sourcegraph/cloud-agent-config.yaml:ro
-    command: ["-config=/sourcegraph/cloud-agent-config.yaml"]
-    restart: always
-    networks:
-      - sourcegraph
-
-  src-serve-git:
-    # Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
-    container_name: src-serve-git-ubuntu.local
-    image:  index.docker.io/sourcegraph/src-cli:latest
-    volumes:
-      - ../src-serve-root/:/sourcegraph/src-serve-root:ro
-    command: "serve-git -addr :443 /sourcegraph/src-serve-root"
-    networks:
-      - sourcegraph
-
-networks:
-  sourcegraph:
+include:
+  - path:
+    - ../config/docker-compose-common-services.yaml
+    - docker-compose-override.yaml

host-wsl/docker-compose-override.yaml

@@ -0,0 +1,5 @@
+services:
+
+  src-serve-git:
+    # Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
+    container_name: src-serve-git-wsl.local

host-wsl/docker-compose.yaml

@@ -0,0 +1,4 @@
+include:
+  - path:
+    - ../config/docker-compose-common-services.yaml
+    - docker-compose-override.yaml

host-wsl2-ubuntu/wsl2-startup.sh → host-wsl/wsl2-startup.sh

@@ -4,19 +4,20 @@
 log_file="./log"
 git_exit_status=""
 docker_compose_exit_status=""
+repo_build_path="/sourcegraph/implementation-bridges/repo-converter/build"
 
 echo "Starting $0 $@" >> $log_file
 
 # Git pull latest commits to main
-if ! git pull
+if ! git -C $repo_build_path pull
 then
     git_exit_status=$?
     echo "git pull failed, exit code $git_exit_status" >> $log_file
     exit $git_exit_status
 fi
 
 # Start docker compose services
-if ! docker compose up -d
+if ! docker compose -f $repo_build_path/docker-compose.yaml up -d --build
 then
     docker_compose_exit_status=$?
     echo "docker compose up failed, exit code $docker_compose_exit_status" >> $log_file

repo-converter/build/Dockerfile

@@ -18,11 +18,9 @@ RUN apt-get update && \
     apt-get upgrade -y && \
     apt-get install --no-install-recommends -y \
         cron \
-        gcc \
         git \
         git-svn \
         python3 \
-        python3-dev \
         python3-pip \
         python3-wheel \
         subversion \
@@ -33,6 +31,9 @@ RUN apt-get update && \
     systemctl enable cron && \
     systemctl start cron
 
+#        python3-dev \
+#        gcc \
+
 # Download dependencies as a separate step to take advantage of Docker's caching
 # Leverage a cache mount to /root/.cache/pip to speed up subsequent builds
 # Leverage a bind mount to requirements.txt to avoid having to copy them into

repo-converter/build/docker-compose-override.yaml

@@ -0,0 +1,6 @@
+services:
+
+  src-serve-git:
+    # Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
+    container_name: src-serve-git-wsl.local
+    command: "-v serve-git -addr :443 /sourcegraph/src-serve-root"

repo-converter/build/docker-compose.yaml

@@ -1,15 +1,22 @@
 version: '2.4'
 
+include:
+  - path:
+    - ../../config/docker-compose-common-services.yaml
+    - docker-compose-override.yaml
+
 services:
 
   repo-converter:
     container_name: repo-converter
     build:
       context: .
+    image: sourcegraph/implementation-bridge-repo-converter:build
+    restart: always
     volumes:
       - ../../config/repos-to-convert.yaml:/sourcegraph/repos-to-convert.yaml:ro
       - ../../config/toprc:/root/.config/procps/toprc
       - ../../src-serve-root/:/sourcegraph/src-serve-root
     environment:
-      - BRIDGE_REPO_CONVERTER_INTERVAL_SECONDS=10
-      - LOG_LEVEL=DEBUG # DEBUG INFO WARNING ERROR CRITICAL # Default is INFO
+      - BRIDGE_REPO_CONVERTER_INTERVAL_SECONDS=60
+      - LOG_LEVEL=DEBUG # DEBUG INFO WARNING ERROR CRITICAL # Default is INFO

repo-converter/build/pull-build-start.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+# To be used in a cronjob to always pull and build the latest commit to the current branch
+# every 10 minutes
+# so that the running container is only 10 minutes behind the latest commit in the branch
+
+# crontab -e
+# */10 * * * * sudo bash /sourcegraph/implementation-bridges/repo-converter/build/pull-build-start.sh >> /sourcegraph/implementation-bridges/repo-converter/build/pull-build-start.log 2>&1
+
+repo_converter_build_path="/sourcegraph/implementation-bridges/repo-converter/build"
+
+git -C $repo_converter_build_path pull
+
+docker compose -f $repo_converter_build_path/docker-compose.yaml up -d --build

repo-converter/build/requirements.txt

@@ -1,2 +1 @@
-psutil
 PyYAML