Refactoring directory layout, resetting ports to default (#13)

* Refactor directory layout
* Giving up on changing default port 443 to 80, sticking with default

.gitignore

@@ -1,11 +1,13 @@
 # Sourcegraph
-bridge-repo-converter/repos-to-serve
+config/cloud-agent-config.yaml
+config/cloud-agent-service-account-key.json
 config/config.yaml
 config/service-account-key.json
-docker-template/
-repo-stats-svn/*.csv
-repo-stats-svn/repos.txt
-repo-stats-svn/tmp-repo-metadata/
+config/repos-to-convert.yaml
+src-serve-root
+svn-repo-stats/*.csv
+svn-repo-stats/repos.txt
+svn-repo-stats/tmp-repo-metadata/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

README.md

@@ -6,35 +6,36 @@ This repo was created for Sourcegraph Implementation Engineering deployments, an
 ## Why
 Running src serve-git and the agent together on the same Docker network allows the agent to use Docker's DNS to reach src serve-git, and prevents src serve-git's unauthenticated HTTP endpoint from needing to be opened outside of the Docker host.
 
-Docker compose also allows for easier upgrades, troubleshooting, monitoring, logging, flexibility of hosting, etc.
+Docker compose also allows for easier upgrades, troubleshooting, monitoring, logging, flexibility of hosting, etc. than running the binaries directly on the OS.
 
 ## Setup - Sourcegraph Staff Only
-1. Add the entry to the sourcegraphConnect targetGroups list in the Cloud instance's config.yaml, get your PR approved and merged
+1. Add the needed entries to the sourcegraphConnect targetGroups list in the Cloud instance's config.yaml, get your PR approved and merged
 ```yaml
         - dnsName: src-serve-git-ubuntu.local
           listeningAddress: 100.100.100.0
           name: src-serve-git-ubuntu-local
           ports:
-          - 80
+          - 443
         - dnsName: src-serve-git-wsl.local
           listeningAddress: 100.100.100.1
           name: src-serve-git-wsl-local
           ports:
-          - 80
+          - 443
 ```
-2. Clone this repo to a customer's bridge VM, install Docker and Docker's Compose plugin
-3. Copy the config.yaml and service-account-key.json files from the Cloud Ops dashboard, and paste them in the files under the config directory
-4. Modify the config.yaml copied from the Cloud Ops dashboard
- - `serviceAccountKeyFile: /service-account-key.json` so that the Go binary inside the running Docker container finds this file in the path that's mapped via the docker-compose.yaml file
- - Only include the `- dialAddress` entries that this cloud agent can reach, remove the others, so the Cloud instance doesn't try connecting to this instance for code hosts it can't reach
- - Correct open ports from default 443 to 80
- - Careful when pasting the config.yaml into Windows, because it may add weird line endings or extra spaces, which breaks YAML, as a whitespace-dependent format
-5. Clone the customer's repos into the `repos-to-serve` directory at the root of this repo on the bridge VM, or update the volume mount path for the src-serve-git service in the docker-compose.yaml file
-6. docker compose up -d
-7. Add a Code Host config to the customer's instance
+2. Run the Reload frontend GitHub Action, as this seems to be needed for the frontend pods to start using tunnel connections
+3. Clone this repo to a customer's bridge VM, install Docker and Docker's Compose plugin
+4. Copy the `config.yaml` and `service-account-key.json` files using the instructions on the instance's Cloud Ops dashboard
+ - Paste them into `./config/cloud-agent-config.yaml` and `./config/cloud-agent-service-account-key.json`
+5. Modify the `./config/cloud-agent-config.yaml` file
+ - `serviceAccountKeyFile: /sourcegraph/cloud-agent-service-account-key.json` so that the Go binary inside the agent container finds this file in the path that's mapped via the docker-compose.yaml files
+ - Only include the `- dialAddress` entries that this cloud agent instance can reach, remove the others, so the Cloud instance doesn't try using this agent instance for code hosts it can't reach
+ - Use extra caution when pasting the config.yaml in Windows, as it may use Windows' line endings or extra spaces, which breaks YAML, as a whitespace-dependent format
+6. Clone the customer's repos into the `repos-to-serve` directory at the root of this repo on the bridge VM, or update the volume mount path for the src-serve-git service in the docker-compose.yaml file
+7. Run `docker compose up -d`
+8. Add a Code Host config to the customer's Cloud instance
  - Type: src serve-git
- - URL: "http://src-serve-git-ubuntu.local:80"
+ - URL: "http://src-serve-git-ubuntu.local:443"
  - or
- - URL: "http://src-serve-git-wsl.local:80"
- - Note that the :80 port is required, as this seems to default to port 443, even when used with http://
-8. Use the bridge-repo-converter to convert SVN, TFVC, or Git repos, to Git format, which will store them in the `repos-to-serve` directory, or use any other means at your disposal to get the repos into the /`repos-to-serve` directory
+ - URL: "http://src-serve-git-wsl.local:443"
+ - Note that the :443 port may be required, as this seems to default to port 443, even when used with http://
+9. Use the repo-converter to convert SVN, TFVC, or Git repos, to Git format, which will store them in the `repos-to-serve` directory, or use any other means to get the repos into the directory

host-ubuntu/docker-compose.yaml

@@ -0,0 +1,27 @@
+version: '2.4'
+
+services:
+
+  cloud-agent:
+    container_name: cloud-agent
+    image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-05-22-28-333320ee
+    volumes:
+      - ../config/cloud-agent-service-account-key.json:/sourcegraph/cloud-agent-service-account-key.json:ro
+      - ../config/cloud-agent-config.yaml:/sourcegraph/cloud-agent-config.yaml:ro
+    command: ["-config=/sourcegraph/cloud-agent-config.yaml"]
+    restart: always
+    networks:
+      - sourcegraph
+
+  src-serve-git:
+    # Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
+    container_name: src-serve-git-ubuntu.local
+    image:  index.docker.io/sourcegraph/src-cli:latest
+    volumes:
+      - ../src-serve-root/:/sourcegraph/src-serve-root:ro
+    command: "serve-git -addr :443 /sourcegraph/src-serve-root"
+    networks:
+      - sourcegraph
+
+networks:
+  sourcegraph:

host-wsl2-ubuntu/docker-compose.yaml

@@ -0,0 +1,27 @@
+version: '2.4'
+
+services:
+
+  cloud-agent:
+    container_name: cloud-agent
+    image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-05-22-28-333320ee
+    volumes:
+      - ../config/cloud-agent-service-account-key.json:/sourcegraph/cloud-agent-service-account-key.json:ro
+      - ../config/cloud-agent-config.yaml:/sourcegraph/cloud-agent-config.yaml:ro
+    command: ["-config=/sourcegraph/cloud-agent-config.yaml"]
+    restart: always
+    networks:
+      - sourcegraph
+
+  src-serve-git:
+    # Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
+    container_name: src-serve-git-wsl.local
+    image:  index.docker.io/sourcegraph/src-cli:latest
+    volumes:
+      - ../src-serve-root/:/sourcegraph/src-serve-root:ro
+    command: "serve-git -addr :443 /sourcegraph/src-serve-root"
+    networks:
+      - sourcegraph
+
+networks:
+  sourcegraph:

repo-converter/build/docker-compose.yaml

@@ -0,0 +1,15 @@
+version: '2.4'
+
+services:
+
+  repo-converter:
+    container_name: repo-converter
+    build:
+      context: .
+    volumes:
+      - ../../config/repos-to-convert.yaml:/sourcegraph/repos-to-convert.yaml:ro
+      - ../../config/toprc:/root/.config/procps/toprc
+      - ../../src-serve-root/:/sourcegraph/src-serve-root
+    environment:
+      - BRIDGE_REPO_CONVERTER_INTERVAL_SECONDS=10
+      - LOG_LEVEL=DEBUG # DEBUG INFO WARNING ERROR CRITICAL # Default is INFO

bridge-repo-converter/build/run.py → repo-converter/build/run.py

@@ -76,6 +76,7 @@
 import sys                                                  # https://docs.python.org/3/library/sys.html
 import time                                                 # https://docs.python.org/3/library/time.html
 # Third party libraries
+# psutil requires adding gcc to the Docker image build, which adds about 4 minutes to the build time
 import psutil                                               # https://pypi.org/project/psutil/
 import yaml                                                 # https://pyyaml.org/wiki/PyYAMLDocumentation
 
@@ -124,7 +125,7 @@ def parse_args():
     )
     parser.add_argument(
         "--repo-share-path",
-        default = "/repos-to-serve",
+        default = "/sourcegraph/src-serve-root",
         help    = "Root of path to directory to store cloned Git repos",
     )
     parsed = parser.parse_args()

repo-converter/docker-compose.yaml

@@ -0,0 +1,14 @@
+version: '2.4'
+
+services:
+
+  repo-converter:
+    container_name: repo-converter
+    image: ghcr.io/sourcegraph/implementation-bridge-repo-converter:insiders
+    volumes:
+      - ../config/repos-to-convert.yaml:/sourcegraph/repos-to-convert.yaml:ro
+      - ../src-serve-root/:/sourcegraph/src-serve-root"
+    restart: always
+    environment:
+      - BRIDGE_REPO_CONVERTER_INTERVAL_SECONDS=3600
+      - LOG_LEVEL=INFO # DEBUG INFO WARNING ERROR CRITICAL # Default is INFO