Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dynamic gitlab runner registration #137

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Dynamic gitlab runner registration #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ac3af65
Dynamic gitlab runner registration
enggnr Dec 29, 2023
5f38c57
Merge branch 'master' into 134_dynamic_gl_runner_token
enggnr Jan 5, 2024
092e867
Documentation update
enggnr Jan 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions docs/customization/secrets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,9 +193,7 @@ Unless otherwise specified in the description column, all of the variables in th
| `GITHUB_TOKEN` | GitHub personal access token stored in `~/.config/shell/private.sh` so developer API keys can be loaded by running `source "~/.config/shell/private.sh"`. |
| `GITLAB_READ_TOKEN` | GitLab token with repository read permissions. Populates `~/.config/ghorg/conf.yaml` so Ghorg can backup all of your GitLab repositories locally. |
| `GITLAB_TOKEN` | GitLab personal access token stored in `~/.config/shell/private.sh` so developer API keys can be loaded by running `source "~/.config/shell/private.sh"`. |
| `GITLAB_RUNNER_TOKEN_DARWIN` | GitLab runner token to use on macOS machines. |
| `GITLAB_RUNNER_TOKEN_LINUX` | GitLab runner token to use on Linux machines. |
| `GITLAB_RUNNER_TOKEN_WINDOWS` | GitLab runner token to use on Windows machines. |
| `GITLAB_ACCESS_TOKEN` | (Personal) Access Token with `create_runner` access to gitlab.com or the Project Group to configure Runner(s). |
| `GMAIL_PASSWORD` | App password for GMail which is used as an outgoing SMTP service (for `git` e-mail). |
| `GOOGLE_SEARCH_API_KEY` | API key used by `search-gpt` available [here](https://developers.google.com/custom-search/v1/introduction). |
| `GOOGLE_SEARCH_ID` | Search engine ID for custom Google search engine utilized by `search-gpt` available [here](https://programmablesearchengine.google.com/controlpanel/all). |
Expand Down
2 changes: 2 additions & 0 deletions home/.chezmoi.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,10 +134,12 @@ data:
domain: "{{ $domain }}"
gitlabRunners:
- glurl: "https://gitlab.com/"
groupID: 00000000
runnerDescription: "Docker executor"
runnerImage: "alpine:latest"
runnerTags: "alpine,linux"
- glurl: "https://gitlab.com/"
groupID: 00000000
baseVM: "debian"
runnerDescription: "VirtualBox executor - Debian, OpenJDK 20"
runnerTags: "bash,openjdk20,linux"
Expand Down
126 changes: 64 additions & 62 deletions software.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3225,7 +3225,7 @@ softwarePackages:
# @brief Logs into DockerHub for Docker Desktop
# @description
# This script logs into DockerHub so that Docker Desktop is pre-authenticated. This
# functionality requires that the `DOCKERHUB_USER` be passed in as an environment variable (or
# functionality requires that the `DOCKERHUB_USER` be passed in as an environment variable (or
# directly editted in the `~/.config/chezmoi/chezmoi.yaml` file) and that the `DOCKERHUB_TOKEN`
# be passed in as a secret (either via the encrypted secret method or passed in as an environment
# variable).
Expand Down Expand Up @@ -5227,9 +5227,9 @@ softwarePackages:
# @file GitLab Runner Configuration
# @brief Registers GitLab Runner(s) with the given GitLab instance
# @description
# This script registers the runner(s) with the given GitLab instance. SaaS GitLab can also be provided as the GitLab instance to register
# the runners with. The script configures the runners to use either Docker or VirtualBox Executor. Refer to
# [this page](https://docs.gitlab.com/runner/executors/docker.html) and [this page](https://docs.gitlab.com/runner/executors/virtualbox.html)
# This script registers the runner(s) with the given GitLab instance scoped to the Group `.host.gitlabRunners.[].groupID` input in the
# `.chezmoi.yaml.tmpl` file. [SaaS GitLab](https://gitlab.com) can also be provided as the GitLab instance. The script configures the runners to use
# either Docker or VirtualBox Executor. Refer to [this page](https://docs.gitlab.com/runner/executors/docker.html) and [this page](https://docs.gitlab.com/runner/executors/virtualbox.html)
# for details about the available configuration settings.
#
# Runners are always tagged with these 2 values: `hostname` and `docker`/`virtualbox` depending on the type of executor. If a list of tags is provided,
Expand All @@ -5242,11 +5242,11 @@ softwarePackages:
#
# The following chart details the secret(s) that are needed to configure the runner:
#
# | Secret | Description |
# |------------------------|------------------------------------------------------------|
# | `GITLAB_RUNNER_TOKEN` | The token generated when the runner was created in GitLab |
# | Secret | Description |
# |------------------------|--------------------------------------------------------------------------------|
# | `GITLAB_ACCESS_TOKEN` | (Personal) Access Token with `create_runner` access to gitlab.com or the group |
#
# For more information about storing secrets like SSH keys and API keys, refer to our Secrets documentation provided below
# For more information about storing secrets like SSH keys and API keys, refer to our Secrets documentation provided below.
#
# ## Configuration Variables
#
Expand All @@ -5255,6 +5255,7 @@ softwarePackages:
# | Variable | Description |
# |---------------------|----------------------------------------------------------------------------------------------------------|
# | `glurl` | The URL of the Gitlab instance to associate the Runner with |
# | `groupID` | The ID of Group in the Gitlab instance to associate the Runner with |
# | `runnerImage` | Docker image to use to configure the runner. Needed only when configuring `Docker` executor |
# | `runnerDescription` | Description of this runner |
# | `runnerTags` | Comma separated list of tags for this runner. See details in the description for more info |
Expand All @@ -5264,6 +5265,9 @@ softwarePackages:
#
# * [Secrets / Environment variables documentation](https://install.doctor/docs/customization/secrets)

HAS_DOCKER=0
HAS_VIRTUALBOX=0

### Check if Docker is installed and operational so Docker executor(s) can be registered
if command -v docker > /dev/null && docker run --rm hello-world > /dev/null; then
HAS_DOCKER=1
Expand All @@ -5287,55 +5291,53 @@ softwarePackages:
### Run logic if gitlab-runner is installed
if command -v gitlab-runner > /dev/null; then
### Populate appropriate token
case "$OSTYPE" in
solaris*) echo "TODO" ;;
darwin*) GITLAB_RUNNER_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_RUNNER_TOKEN_DARWIN")) }}{{ includeTemplate "secrets/GITLAB_RUNNER_TOKEN_DARWIN" | decrypt | trim }}{{ else }}{{ env "GITLAB_RUNNER_TOKEN_DARWIN" }}{{ end }}" ;;
linux*) GITLAB_RUNNER_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_RUNNER_TOKEN_LINUX")) }}{{ includeTemplate "secrets/GITLAB_RUNNER_TOKEN_LINUX" | decrypt | trim }}{{ else }}{{ env "GITLAB_RUNNER_TOKEN_LINUX" }}{{ end }}" ;;
bsd*) echo "TODO" ;;
msys*) GITLAB_RUNNER_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_RUNNER_TOKEN_WINDOWS")) }}{{ includeTemplate "secrets/GITLAB_RUNNER_TOKEN_WINDOWS" | decrypt | trim }}{{ else }}{{ env "GITLAB_RUNNER_TOKEN_WINDOWS" }}{{ end }}" ;;
cygwin*) GITLAB_RUNNER_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_RUNNER_TOKEN_WINDOWS")) }}{{ includeTemplate "secrets/GITLAB_RUNNER_TOKEN_WINDOWS" | decrypt | trim }}{{ else }}{{ env "GITLAB_RUNNER_TOKEN_WINDOWS" }}{{ end }}" ;;
*) echo "unknown: $OSTYPE" ;;
esac

### Check if Runner Token value is present before attempting to register runner(s)
if [ $GITLAB_RUNNER_TOKEN != "" ]; then
### Registering runners
{{ $cmd := `gitlab-runner register \
--non-interactive \
--token $GITLAB_RUNNER_TOKEN \` }}
### Register Docker based runners if Docker is installed and operational
if [ $HAS_DOCKER -eq 1 ]; then
logg info 'Registering GitLab Runner(s) that use Docker executor'
GITLAB_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_TOKEN")) }}{{ includeTemplate "secrets/GITLAB_TOKEN" | decrypt | trim }}{{ else }}{{ env "GITLAB_TOKEN" }}{{ end }}"

### Check if Gitlab Access Token value is present before attempting to register runner(s)
if [ -n "$GITLAB_TOKEN" ]; then
if command -v jq > /dev/null; then
{{- range .host.gitlabRunners }}
{{- if index . "runnerImage" }}
{{- $cmd }}
--url {{ .glurl }} \
--executor "docker" \
--description "{{ .runnerDescription }} - on {{ $.chezmoi.hostname }}" \
--docker-image {{ .runnerImage }} \
{{ if and .runnerTags (gt (len .runnerTags) 0) }}--tag-list "{{ .runnerTags }},{{ $.chezmoi.hostname }},docker"
{{- else }}--tag-list "{{ $.chezmoi.hostname }},docker" --run-untagged{{ end }} || echo 'Runner registration failed"
{{ end -}}
{{- if or (index . "runnerImage") (index . "baseVM") }}
{{ $cmd := `sudo gitlab-runner register \
--non-interactive \` }}
{{- if index . "runnerImage" }}
### Acquire token
logg info 'Acquiring runner token'
GITLAB_RUNNER_TOKEN=$(curl -sSL -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" -d runner_type=group_type -d group_id={{ .groupID }} \
-d description="{{ .runnerDescription }} - on {{ $.chezmoi.hostname }}" \
{{ if and .runnerTags (gt (len .runnerTags) 0) }}-d tag_list="{{ .runnerTags }},{{ $.chezmoi.hostname }},docker"
{{- else }}-d tag_list="{{ $.chezmoi.hostname }},docker" -d run_untagged=true{{ end }} \
{{ .glurl }}api/v4/user/runners | cat | jq -r .token)
### Register Docker based runners if Docker is installed and operational
logg info 'Registering GitLab Runner(s) that use Docker executor'
{{ $cmd }}
--url "{{ .glurl }}" \
--token $GITLAB_RUNNER_TOKEN \
--executor "docker" \
--docker-image "{{ .runnerImage }}" || echo 'Runner registration failed'
{{- else if index . "baseVM" }}
### Acquire token
logg info 'Acquiring runner token'
GITLAB_RUNNER_TOKEN=$(curl -sSL -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" -d runner_type=group_type -d group_id={{ .groupID }} \
-d description="{{ .runnerDescription }} - on {{ $.chezmoi.hostname }}" \
{{ if and .runnerTags (gt (len .runnerTags) 0) }}-d tag_list="{{ .runnerTags }},{{ $.chezmoi.hostname }},virtualbox"
{{- else }}-d tag_list="{{ $.chezmoi.hostname }},virtualbox" -d run_untagged=true{{ end }} \
{{ .glurl }}api/v4/user/runners | cat | jq -r .token)
### Register VirtualBox based runners if VirtualBox is installed
logg info 'Registering GitLab Runner(s) that use VirtualBox executor'
{{ $cmd }}
--url "{{ .glurl }}" \
--token $GITLAB_RUNNER_TOKEN \
--executor "virtualbox" --ssh-user "{{ $.chezmoi.username }}" --ssh-identity-file "/home/{{ $.chezmoi.username }}/.ssh/id_rsa" \
--virtualbox-base-name "{{ .baseVM }}" || echo 'Runner registration failed'
{{ end }}
{{ end }}
fi

### Register VirtualBox based runners if VirtualBox is installed
if [ $HAS_VIRTUALBOX -eq 1 ]; then
logg info 'Registering GitLab Runner(s) that use VirtualBox executor'
{{- range .host.gitlabRunners }}
{{- if index . "baseVM" }}
{{- $cmd }}
--url {{ .glurl }} \
--executor "virtualbox" \
--description "{{ .runnerDescription }} - on {{ $.chezmoi.hostname }}" \
--virtualbox-base-name "{{ .baseVM }}" \
{{ if and .runnerTags (gt (len .runnerTags) 0) }}--tag-list "{{ .runnerTags }},{{ $.chezmoi.hostname }},virtualbox"
{{- else }}--tag-list "{{ $.chezmoi.hostname }},virtualbox" --run-untagged{{ end }} || echo 'Runner registration failed"
{{ end -}}
{{ end }}
else
logg warn 'jq is required by the Gitlab runner configuration script'
fi
else
logg warn 'GITLAB_RUNNER_TOKEN is not set. Not registering runner(s)'
logg warn 'GITLAB_TOKEN is not set. Not registering runner(s)'
fi
else
logg warn 'gitlab-runner is not installed or is not available in PATH'
Expand Down Expand Up @@ -5392,12 +5394,12 @@ softwarePackages:
if command -v gitomatic > /dev/null; then
### Copy bin to /usr/local/bin
logg info "Copying $HOME/.local/bin/gitomatic-service to /usr/local/bin/gitomatic-service" && sudo cp -f "$HOME/.local/bin/gitomatic-service" /usr/local/bin/gitomatic-service

### Copy gitomatic to global directory
if [ ! -f /usr/local/bin/gitomatic ]; then
logg info 'Copying gitomatic executable to /usr/local/bin/gitomatic' && sudo cp -f "$(which gitomatic)" /usr/local/bin/gitomatic
fi

if [ -d /Applications ] && [ -d /System ]; then
### macOS
logg info 'Copying gitomatic plist file to /Library/LaunchDaemons' && sudo cp -f "${XDG_CONFIG_HOME:-$HOME/.config}/gitomatic/com.github.muesli.gitomatic.plist" /Library/LaunchDaemons/com.github.muesli.gitomatic.plist
Expand Down Expand Up @@ -7937,7 +7939,7 @@ softwarePackages:

### netdata-claim.sh must be run as netdata user
sudo -H -u netdata bash -c 'export NETDATA_ROOM="{{- if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "NETDATA_ROOM")) -}}{{- includeTemplate "secrets/NETDATA_ROOM" | decrypt | trim -}}{{- else -}}{{- env "NETDATA_ROOM" -}}{{- end -}}" && export NETDATA_TOKEN="{{- if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "NETDATA_TOKEN")) -}}{{- includeTemplate "secrets/NETDATA_TOKEN" | decrypt | trim -}}{{- else -}}{{- env "NETDATA_TOKEN" -}}{{- end -}}" && yes | netdata-claim.sh -token="$NETDATA_TOKEN" -rooms="$NETDATA_ROOM" -url="https://app.netdata.cloud"'

### Kernel optimizations
# These are mentioned while installing via the kickstart.sh script method. We are using Homebrew for the installation though.
# Assuming these optimizations do not cause any harm.
Expand Down Expand Up @@ -7969,7 +7971,7 @@ softwarePackages:
logg error 'No etc location found for netdata' && exit 1
fi
logg info "Copying ${XDG_CONFIG_HOME:-$HOME/.config}/netdata/health.d/ to $NETDATA_ETC" && sudo cp -rf "${XDG_CONFIG_HOME:-$HOME/.config}/netdata/health.d/" "$NETDATA_ETC"

if command -v gsed > /dev/null; then
SED_UTIL="gsed"
else
Expand All @@ -7979,7 +7981,7 @@ softwarePackages:
### Blocky
logg info "Adding Blocky metrics collection to $NETDATA_ETC/go.d/prometheus.conf"
sudo "$SED_UTIL" -i "/jobs:/a\ - name: blocky_local \n url: 'http://127.0.0.1:4000/metrics'" "$NETDATA_ETC/go.d/prometheus.conf"

### SFTPGo
logg info "Adding SFTPGo metrics collection to $NETDATA_ETC/go.d/prometheus.conf"
sudo "$SED_UTIL" -i "/jobs:/a\ - name: sftpgo_local \n url: 'http://127.0.0.1:57500/metrics'" "$NETDATA_ETC/go.d/prometheus.conf"
Expand Down Expand Up @@ -10917,7 +10919,7 @@ softwarePackages:
_home: https://github.com/skeeto/endlessh
_name: Endlessh
_service: endlessh
_post: |
_post: |
#!/usr/bin/env bash
# @file Endlessh Configuration
# @brief Applies the Endlessh configuration and starts the service on Linux systems
Expand Down Expand Up @@ -11764,7 +11766,7 @@ softwarePackages:
yay: termius
dagu:
_bin: dagu
_github:
_github:
_name: Dagu
_post: |
#!/usr/bin/env bash
Expand Down Expand Up @@ -12145,7 +12147,7 @@ softwarePackages:
# ## Links
#
# * [Privoxy configuration](https://github.com/megabyte-labs/install.doctor/tree/master/home/dot_config/privoxy/config)

### Define the Privoxy configuration location based on whether system is macOS or Linux
if [ -d /Applications ] && [ -d /System ]; then
### macOS
Expand All @@ -12171,7 +12173,7 @@ softwarePackages:
sudo add-usergroup "$USER" privoxy
fi
sudo chown privoxy:privoxy "$PRIVOXY_CONFIG" 2> /dev/null || sudo chown privoxy:$(id -g -n) "$PRIVOXY_CONFIG"

### Restart Privoxy after configuration is applied
if [ -d /Applications ] && [ -d /System ]; then
### macOS
Expand Down Expand Up @@ -13734,7 +13736,7 @@ softwarePackages:
# ## Links
#
# * [`fail2ban` configuration folder](https://github.com/megabyte-labs/install.doctor/tree/master/home/private_dot_ssh/fail2ban)

### Notify of script start
logg info 'Configuring fail2ban'

Expand Down
Loading