FluxCD EZY

A sanitized, public-safe FluxCD GitOps reference implementation for k0s Kubernetes clusters.

This repository is a comprehensive guide and reference for deploying a production-grade GitOps platform using FluxCD on k0s. It has been sanitized from a private production cluster — all secrets, credentials, IP addresses, and personal data have been replaced with clearly labeled placeholders.

What This Repository Is

This is a production-ready FluxCD GitOps reference implementation demonstrating:

Multi-cluster GitOps — Production (k0s) and Staging (k0s-staging) managed from a single repository
Infrastructure as Code — All controllers and configuration managed via Helm releases and Kustomize overlays
Two secret management patterns — Bitwarden + External Secrets Operator for dynamic app secrets; Sealed Secrets for cluster-specific encrypted secrets
Bare-metal networking — MetalLB for LoadBalancer IPs, Traefik for ingress, Cloudflare Tunnel for public access
Ceph storage — RBD block storage and CephFS with CSI Snapshotter for volume snapshots
TLS automation — cert-manager with Let's Encrypt DNS-01 challenge via Cloudflare
Automated updates — Renovate for Helm chart and Docker image version bumps
Slack notifications — FluxCD notification-controller alerts on reconciliation events
16 self-hosted applications — From monitoring to home automation to AI inference

Repository Structure

.
├── renovate.json               # Automated dependency update configuration
├── .env.example                # Template for all required environment variables
├── apps/
│   ├── base/                   # Base Helm release definitions (shared across clusters)
│   │   ├── capacitor/
│   │   ├── code-server/
│   │   ├── cloudflare-tunnel/
│   │   ├── github-arc/
│   │   ├── home-assistant/
│   │   ├── k10/
│   │   ├── kubernetes-dashboard/
│   │   ├── mealie/
│   │   ├── nextcloud/
│   │   ├── ollama/
│   │   ├── openspeedtest/
│   │   ├── portainer/
│   │   ├── sonarqube/
│   │   ├── speedtest-tracker/
│   │   ├── uptime-kuma/
│   │   └── wordpress/
│   ├── k0s/                    # Production cluster overlays (adds IngressRoutes, patches)
│   │   ├── prod-capacitor/
│   │   ├── prod-home-assistant/
│   │   │   └── dahuavto2mqtt-kustomize.yaml
│   │   └── ...
│   └── k0s-staging/            # Staging cluster overlays
├── clusters/
│   ├── k0s/                    # Production cluster FluxCD entry points
│   │   ├── apps.yaml           # Kustomization pointing to apps/k0s
│   │   ├── infrastructure.yaml # Kustomizations for all infrastructure layers
│   │   └── flux-system/        # FluxCD bootstrap files (generated by flux bootstrap)
│   └── k0s-staging/            # Staging cluster FluxCD entry points
└── infrastructure/
    ├── controllers/             # Helm releases for infrastructure controllers
    │   ├── cert-manager.yaml
    │   ├── metallb.yaml
    │   ├── sealed-secrets.yaml
    │   ├── ceph-csi-rbd.yaml
    │   ├── ceph-csi-cephfs.yaml
    │   └── csi-snapshotter.yaml
    ├── configs/                 # Cluster-level configuration resources
    │   ├── cert-manager-configs.yaml   # ClusterIssuers + wildcard Certificate
    │   ├── metallb-configs.yaml        # IPAddressPool + L2Advertisement
    │   ├── csi-snapshotter/            # VolumeSnapshotClass resources
    │   ├── sealed-secrets/             # SealedSecret for Slack webhook URL
    │   ├── slack-notifications/        # FluxCD Provider + Alert
    │   └── traefik/                    # Middleware + dashboard IngressRoute
    ├── external-secrets/        # External Secrets Operator + Bitwarden backend
    │   ├── external-secrets-operator.yaml
    │   ├── bitwarden-cli.yaml          # Bitwarden CLI sidecar deployment
    │   └── cluster-secret-store.yaml
    └── traefik/                 # Traefik ingress controller Helm release

Reconciliation Order

FluxCD reconciles resources in this dependency order:

infrastructure-controllers  →  infrastructure-configs
                            →  infrastructure-external-secrets
                            →  infrastructure-traefik
                                     ↓
                                   apps

Infrastructure Components

MetalLB — Bare-Metal Load Balancer

MetalLB provides LoadBalancer type services on bare-metal clusters.

File: infrastructure/controllers/metallb.yaml
Config: infrastructure/configs/metallb-configs.yaml
What to configure: Set <YOUR_METALLB_IP_RANGE> to a free IP range in your network (e.g., 192.168.50.50-192.168.50.60)

Cert-Manager — TLS Certificate Management

cert-manager automates TLS certificates from Let's Encrypt using DNS-01 challenges via Cloudflare.

File: infrastructure/controllers/cert-manager.yaml
Config: infrastructure/configs/cert-manager-configs.yaml
What to configure:
- Replace your-email@example.com with your email
- Replace *.local.example.com with your domain
- Create a Cloudflare API token secret (see bootstrap steps)

Traefik — Ingress Controller

Traefik serves as the ingress controller with automatic TLS termination.

File: infrastructure/traefik/helm-release.yaml
Config: infrastructure/configs/traefik/
What to configure: Set <YOUR_TRAEFIK_LOADBALANCER_IP> to an IP from your MetalLB pool
Dashboard: Protected by basic auth middleware (credentials stored in SealedSecret)

Sealed Secrets — Git-Safe Encrypted Secrets

Sealed Secrets encrypts Kubernetes Secrets with a cluster-specific key so they can be safely committed to Git.

File: infrastructure/controllers/sealed-secrets.yaml
Usage: kubeseal --format yaml < secret.yaml > sealedsecret.yaml
Important: Sealed Secrets are cluster-specific — a secret sealed for one cluster cannot be decrypted by another cluster's controller.

External Secrets Operator + Bitwarden

External Secrets Operator dynamically pulls secrets from external stores. This implementation uses Bitwarden as the backend via the Bitwarden CLI.

Files: infrastructure/external-secrets/
Pattern: Each app that needs a secret defines an ExternalSecret resource referencing a Bitwarden item ID
What to configure: Replace all <YOUR_BITWARDEN_ITEM_ID_*> placeholders with your actual Bitwarden vault item IDs

Ceph CSI — Storage Drivers

Ceph CSI provides Kubernetes storage backed by a Ceph cluster.

RBD (ceph-block StorageClass): Block storage for databases and single-node workloads
CephFS (ceph-filesystem StorageClass): Shared filesystem for multi-reader workloads
Files: infrastructure/controllers/ceph-csi-rbd.yaml, infrastructure/controllers/ceph-csi-cephfs.yaml
What to configure: Ceph cluster ID, monitor IPs, and user keys (see Ceph configuration)
k0s note: Uses /var/lib/k0s/kubelet as the kubelet path instead of the default /var/lib/kubelet

CSI Snapshotter — Volume Snapshots

Enables VolumeSnapshot resources for point-in-time storage snapshots.

File: infrastructure/controllers/csi-snapshotter.yaml
Config: infrastructure/configs/csi-snapshotter/volumesnapshotclass.yaml

Application Catalog

Application	Namespace	Description
Capacitor	`capacitor`	FluxCD web UI
Code Server	`code-server`	VS Code in the browser
Cloudflare Tunnel	`cloudflare-tunnel`	Secure public ingress without port forwarding
GitHub ARC	`arc-systems`	Self-hosted GitHub Actions runners
Home Assistant	`home-assistant`	Home automation (+ Dahua VTO MQTT bridge)
Kasten K10	`kasten-io`	Kubernetes backup and disaster recovery
Kubernetes Dashboard	`kubernetes-dashboard`	Official Kubernetes web UI
Mealie	`mealie`	Recipe and meal planning
Nextcloud	`nextcloud`	Self-hosted file storage and collaboration
Ollama / Open WebUI	`ollama`	Local LLM inference with web UI
OpenSpeedTest	`openspeedtest`	LAN network speed testing
Portainer	`portainer`	Container management UI
SonarQube	`sonarqube`	Code quality and security analysis
Speedtest Tracker	`speedtest-tracker`	WAN speed monitoring
Uptime Kuma	`uptime-kuma`	Service uptime monitoring
WordPress	`wordpress`	Content management system

Prerequisites

Before you begin, ensure you have the following:

Required Tools

Tool	Purpose	Install
`kubectl`	Kubernetes CLI	Install
`flux`	FluxCD CLI	Install
`kubeseal`	Sealed Secrets CLI	Install
`helm`	Helm package manager	Install
`git`	Version control	Install

Required Infrastructure

Kubernetes cluster — This was built on k0s but works on any Kubernetes distribution. For k0s, see k0s installation.
GitHub account — With a Personal Access Token that has repo scope
Ceph cluster (optional) — For RBD/CephFS storage; replace with other storage if not available
Cloudflare account (optional) — For DNS-01 TLS challenges; other DNS providers can be configured in cert-manager
Bitwarden account (optional) — For External Secrets; can be replaced with Vault, AWS Secrets Manager, etc.
Slack workspace (optional) — For FluxCD notifications

Verify FluxCD Prerequisites

flux check --pre

How to Install FluxCD

Install the FluxCD CLI:

# Linux/macOS via Homebrew
brew install fluxcd/tap/flux

# Linux via script
curl -s https://fluxcd.io/install.sh | sudo bash

# Verify installation
flux --version

How to Bootstrap and Deploy

1. Fork and Clone

# Fork this repository on GitHub, then:
git clone https://github.com/<YOUR_GITHUB_USERNAME>/<YOUR_REPO_NAME>.git
cd <YOUR_REPO_NAME>

2. Customize Placeholders

Search for all placeholders and replace them with your values:

# Find all placeholders
grep -r "<YOUR_" . --include="*.yaml" | grep -v ".git"

# Key files to update:
# - infrastructure/controllers/ceph-csi-rbd.yaml      (Ceph cluster ID, monitor IPs, user key)
# - infrastructure/controllers/ceph-csi-cephfs.yaml   (Ceph cluster ID, monitor IPs, admin key)
# - infrastructure/configs/cert-manager-configs.yaml  (email, domain)
# - infrastructure/configs/metallb-configs.yaml       (IP range)
# - infrastructure/configs/traefik/dashboard-ingress.yaml (domain)
# - infrastructure/traefik/helm-release.yaml          (Traefik LB IP)
# - apps/base/mealie/flux-resources.yaml              (Postgres IP, Bitwarden IDs)
# - apps/base/nextcloud/flux-resources.yaml           (domain, Postgres IP, S3 config, Bitwarden IDs)
# - apps/base/ollama/flux-resources.yaml              (Ollama server IP, Bitwarden ID)
# - apps/base/home-assistant/flux-resources.yaml      (Home Assistant LB IP)
# - clusters/k0s/flux-system/gotk-sync.yaml           (GitHub username and repo name)

3. Create Pre-Bootstrap Secrets

These secrets must exist before FluxCD bootstraps, as controllers depend on them.

Cloudflare API Token (for cert-manager DNS-01 challenge)

# Create a Cloudflare API token with Zone:DNS:Edit permissions
# https://dash.cloudflare.com/profile/api-tokens
kubectl create secret generic cloudflare-api-token \
  --namespace=cert-manager \
  --from-literal=api-token=<YOUR_CLOUDFLARE_API_TOKEN>

Bitwarden Credentials (for External Secrets Operator)

# Create a Bitwarden API key at https://vault.bitwarden.com/#/settings/security
kubectl create secret generic bitwarden-credentials \
  --namespace=external-secrets \
  --from-literal=clientId=<YOUR_BW_CLIENT_ID> \
  --from-literal=clientSecret=<YOUR_BW_CLIENT_SECRET> \
  --from-literal=password=<YOUR_BW_MASTER_PASSWORD>

Slack Webhook URL (for FluxCD notifications) — Sealed Secret

# 1. Create a Slack Incoming Webhook: https://api.slack.com/messaging/webhooks
# 2. Create the plain secret:
kubectl create secret generic slack-url \
  --namespace=flux-system \
  --from-literal=address=https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  --dry-run=client -o yaml > /tmp/slack-url-secret.yaml

# 3. Seal it (requires sealed-secrets controller to be running):
kubeseal --format yaml < /tmp/slack-url-secret.yaml \
  > infrastructure/configs/sealed-secrets/slack-notification.yaml

# 4. Commit the sealed secret (safe to commit — encrypted with cluster key)
git add infrastructure/configs/sealed-secrets/slack-notification.yaml
git commit -m "Add sealed slack webhook URL"

Traefik Dashboard Credentials — Sealed Secret

# 1. Generate htpasswd credentials:
htpasswd -nb admin yourpassword

# 2. Create the plain secret:
kubectl create secret generic traefik-basic-auth \
  --namespace=traefik \
  --from-literal=users="$(htpasswd -nb admin yourpassword)" \
  --dry-run=client -o yaml > /tmp/traefik-auth-secret.yaml

# 3. Seal it:
kubeseal --format yaml < /tmp/traefik-auth-secret.yaml \
  > infrastructure/configs/traefik/middleware.yaml

# 4. Commit
git add infrastructure/configs/traefik/middleware.yaml
git commit -m "Add sealed traefik dashboard credentials"

4. Bootstrap the Production Cluster

export GITHUB_TOKEN=<YOUR_GITHUB_TOKEN>

flux bootstrap github \
  --owner=<YOUR_GITHUB_USERNAME> \
  --repository=<YOUR_REPO_NAME> \
  --branch=main \
  --path=clusters/k0s \
  --personal

This command will:

Install FluxCD controllers in the flux-system namespace
Create a deploy key in your GitHub repository
Commit and push clusters/k0s/flux-system/gotk-components.yaml and gotk-sync.yaml
Start reconciling everything in clusters/k0s/

5. Bootstrap the Staging Cluster (optional)

flux bootstrap github \
  --owner=<YOUR_GITHUB_USERNAME> \
  --repository=<YOUR_REPO_NAME> \
  --branch=staging \
  --path=clusters/k0s-staging \
  --personal \
  --kubeconfig=~/.kube/staging-config  # Use appropriate kubeconfig

6. Verify the Deployment

# Watch FluxCD reconcile everything
flux get all -A

# Check kustomizations
flux get kustomizations -A

# Check helm releases
flux get helmreleases -A

# Watch events
kubectl get events -n flux-system --sort-by='.lastTimestamp'

How to Configure Required Variables and Secrets

See .env.example for a complete list of all variables. Below is a component-by-component guide.

Ceph Storage

# infrastructure/controllers/ceph-csi-rbd.yaml
clusterID: "<YOUR_CEPH_CLUSTER_ID>"       # ceph fsid
monitors:
  - "<YOUR_CEPH_MONITOR_IP_1>:6789"        # ceph mon dump | grep -oP '\d+\.\d+\.\d+\.\d+'
userID: "<YOUR_CEPH_USER_ID>"              # e.g., kubernetes
userKey: "<YOUR_CEPH_RBD_USER_KEY>"       # ceph auth get-key client.kubernetes

To get your Ceph cluster ID and user key:

# Get cluster ID
ceph fsid

# Create a user for Kubernetes (if not exists)
ceph auth get-or-create client.kubernetes \
  mon 'profile rbd' \
  osd 'profile rbd pool=kubernetes' \
  mgr 'profile rbd pool=kubernetes'

# Get the user key
ceph auth get-key client.kubernetes

MetalLB IP Pool

# infrastructure/configs/metallb-configs.yaml
addresses:
  - <YOUR_METALLB_IP_RANGE>   # e.g., 192.168.50.50-192.168.50.60

Choose a range of IPs in your LAN that are not in your DHCP range.

cert-manager / Let's Encrypt

# infrastructure/configs/cert-manager-configs.yaml
email: your-email@example.com     # Your email for Let's Encrypt notifications
dnsNames:
  - "*.local.example.com"          # Replace with *.local.yourdomain.com
  - "example.com"                  # Replace with yourdomain.com

External Secrets / Bitwarden

Each ExternalSecret references a Bitwarden vault item by its ID:

remoteRef:
  key: "<YOUR_BITWARDEN_ITEM_ID>"     # UUID of the item in Bitwarden
  property: password                   # Field name within the item

To find a Bitwarden item ID:

bw login
bw list items --search "mealie" | jq '.[0].id'

Domain Names

Replace all occurrences of example.com and local.example.com with your actual domain:

# Find all domain placeholders
grep -r "example.com" . --include="*.yaml" -l

# Replace (Linux/macOS):
find . -name "*.yaml" -not -path "./.git/*" \
  -exec sed -i 's/local\.example\.com/local.yourdomain.com/g; s/example\.com/yourdomain.com/g' {} +

How to Maintain and Update

Force Reconciliation

# Reconcile a specific kustomization
flux reconcile kustomization apps --with-source

# Reconcile a specific helm release
flux reconcile helmrelease nextcloud -n nextcloud --with-source

# Reconcile all
flux reconcile source git flux-system

Update Helm Chart Versions

Helm chart versions in this repo use x.x.x as placeholders. Pin to specific versions and use Renovate for automated updates:

# Example: pin to specific version
spec:
  chart:
    spec:
      version: "28.3.0"   # Renovate will open PRs to bump this

Renovate Automated Updates

The renovate.json configuration enables automated dependency updates:

Helm chart minor/patch updates are auto-merged
Docker image major updates require manual review
Flux HelmReleases in clusters/ are automatically detected

To enable Renovate on your fork, install the Renovate GitHub App.

Suspend / Resume Reconciliation

# Suspend an app during maintenance
flux suspend helmrelease nextcloud -n nextcloud

# Resume
flux resume helmrelease nextcloud -n nextcloud

View Logs

# FluxCD controller logs
kubectl logs -n flux-system deployment/source-controller
kubectl logs -n flux-system deployment/kustomize-controller
kubectl logs -n flux-system deployment/helm-controller
kubectl logs -n flux-system deployment/notification-controller

How to Uninstall and Clean Up

Remove a Single Application

# Suspend first to prevent re-reconciliation
flux suspend helmrelease <app-name> -n <namespace>

# Then delete the HelmRelease (Flux will uninstall the Helm chart)
kubectl delete helmrelease <app-name> -n <namespace>

# Remove from git and push
git rm apps/base/<app-name>/ apps/k0s/prod-<app-name>/
git commit -m "Remove <app-name>"
git push

Uninstall FluxCD Completely

# This removes all FluxCD components and CRDs
# WARNING: This will not delete workloads managed by Flux — only the controllers
flux uninstall --namespace=flux-system

Clean Up Namespaces

# List all app namespaces
kubectl get namespaces | grep -E "(nextcloud|mealie|wordpress|..."

# Delete a namespace and all its resources
kubectl delete namespace nextcloud

Security Practices

Two Secret Management Patterns

This repo uses two complementary approaches to secret management:

1. Sealed Secrets (for cluster-specific secrets)

Used for secrets that are tied to a specific cluster's sealing key:

Slack webhook URL (infrastructure/configs/sealed-secrets/slack-notification.yaml)
Traefik dashboard credentials (infrastructure/configs/traefik/middleware.yaml)
WordPress admin password (apps/base/wordpress/flux-resources.yaml)
Dahua VTO / MQTT credentials (apps/k0s/prod-home-assistant/dahuavto2mqtt-kustomize.yaml)

Properties:

Encrypted with cluster-specific public key
Safe to commit to a public Git repository
Cannot be decrypted outside the cluster
Must be re-sealed if the sealing key is rotated

2. External Secrets Operator + Bitwarden (for application secrets)

Used for secrets that are stored in Bitwarden and pulled at runtime:

Database passwords (Mealie, Nextcloud)
API tokens (Cloudflare, GitHub)
Application secret keys (Open WebUI)