Skip to content

rework examples to use kustomize #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Billy99 merged 1 commit into from
May 12, 2026
Merged

rework examples to use kustomize #124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 47 additions & 8 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ else
SED ?= gsed
endif

MAKEFLAGS += --no-print-directory
ARCH=$(shell go env GOARCH)

# Define CONTAINER_FLAGS and include ARCH as an argument
CONTAINER_FLAGS ?= --build-arg TARGETARCH=$(ARCH)

Expand Down Expand Up @@ -339,14 +341,51 @@ undeploy: kustomize delete-webhook-secret-file ## Undeploy operator and agent fr
undeploy-force: ## Same as "make undeploy" but also delete any dependencies.
$(MAKE) undeploy FORCE=--force

.PHONY: deploy-examples
deploy-examples: ## Deploy the examples to the K8s cluster specified in ~/.kube/config.
@echo "Create Namespace based GKMCache"
$(KUBECTL) apply -f examples/namespace/RWO/
$(KUBECTL) apply -f examples/namespace/ROX/
@echo "Create Cluster based ClusterGKMCache"
$(KUBECTL) apply -f examples/cluster/RWO/
$(KUBECTL) apply -f examples/cluster/ROX/
.PHONY: gen-apply-example
gen-apply-example: kustomize
@cd examples; \
if ! EXAMPLE_YAML=$$(DEBUG=false ./generate-files.sh $(EXAMPLE_ACCESS) $(EXAMPLE_SCOPE) $(EXAMPLE_GPU) $(EXAMPLE_VER) $(EXAMPLE_ENV)); then \
echo; \
echo "FAILED: ./generate-files.sh $(EXAMPLE_ACCESS) $(EXAMPLE_SCOPE) $(EXAMPLE_GPU) $(EXAMPLE_VER) $(EXAMPLE_ENV)"; \
echo; \
exit 1; \
fi; \
$(KUBECTL) $$EXAMPLE_CMD -f $$EXAMPLE_YAML
Comment thread
coderabbitai[bot] marked this conversation as resolved.

.PHONY: deploy-examples-kind
deploy-examples-kind: ## Deploy the examples to a KIND K8s cluster
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=namespace EXAMPLE_GPU=rocm EXAMPLE_VER=v2 EXAMPLE_ENV=kind EXAMPLE_CMD=apply
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=cluster EXAMPLE_GPU=rocm EXAMPLE_VER=v3 EXAMPLE_ENV=kind EXAMPLE_CMD=apply
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rox EXAMPLE_SCOPE=namespace EXAMPLE_GPU=rocm EXAMPLE_VER=v3 EXAMPLE_ENV=kind EXAMPLE_CMD=apply
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rox EXAMPLE_SCOPE=cluster EXAMPLE_GPU=rocm EXAMPLE_VER=v2 EXAMPLE_ENV=kind EXAMPLE_CMD=apply

.PHONY: deploy-examples-nfd-cuda
deploy-examples-nfd-cuda: ## Deploy the examples to a K8s cluster running NFD and CUDA, AccessMode=ReadWriteOnce
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=namespace EXAMPLE_GPU=cuda EXAMPLE_VER=v2 EXAMPLE_ENV=nfd EXAMPLE_CMD=apply
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=cluster EXAMPLE_GPU=cuda EXAMPLE_VER=v3 EXAMPLE_ENV=nfd EXAMPLE_CMD=apply

.PHONY: deploy-examples-nfd-rocm
deploy-examples-nfd-rocm: ## Deploy the examples to a K8s cluster running NFD and ROCm, AccessMode=ReadWriteOnce
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=namespace EXAMPLE_GPU=rocm EXAMPLE_VER=v2 EXAMPLE_ENV=nfd EXAMPLE_CMD=apply
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=cluster EXAMPLE_GPU=rocm EXAMPLE_VER=v3 EXAMPLE_ENV=nfd EXAMPLE_CMD=apply

.PHONY: undeploy-examples-kind
undeploy-examples-kind: ## Undeploy the examples to a KIND K8s cluster
@echo "Remove Namespace based GKMCache"
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=namespace EXAMPLE_GPU=rocm EXAMPLE_VER=v2 EXAMPLE_ENV=kind EXAMPLE_CMD="delete --ignore-not-found=$(ignore-not-found)"
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=cluster EXAMPLE_GPU=rocm EXAMPLE_VER=v3 EXAMPLE_ENV=kind EXAMPLE_CMD="delete --ignore-not-found=$(ignore-not-found)"
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rox EXAMPLE_SCOPE=namespace EXAMPLE_GPU=rocm EXAMPLE_VER=v3 EXAMPLE_ENV=kind EXAMPLE_CMD="delete --ignore-not-found=$(ignore-not-found)"
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rox EXAMPLE_SCOPE=cluster EXAMPLE_GPU=rocm EXAMPLE_VER=v2 EXAMPLE_ENV=kind EXAMPLE_CMD="delete --ignore-not-found=$(ignore-not-found)"

.PHONY: undeploy-examples-nfd-cuda
undeploy-examples-nfd-cuda: ## Undeploy the examples to a K8s cluster running NFD and CUDA, AccessMode=ReadWriteOnce
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=namespace EXAMPLE_GPU=cuda EXAMPLE_VER=v2 EXAMPLE_ENV=nfd EXAMPLE_CMD="delete --ignore-not-found=$(ignore-not-found)"
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=cluster EXAMPLE_GPU=cuda EXAMPLE_VER=v3 EXAMPLE_ENV=nfd EXAMPLE_CMD="delete --ignore-not-found=$(ignore-not-found)"

.PHONY: undeploy-examples-nfd-rocm
undeploy-examples-nfd-rocm: ## Undeploy the examples to a K8s cluster running NFD and ROCm, AccessMode=ReadWriteOnce
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=namespace EXAMPLE_GPU=rocm EXAMPLE_VER=v2 EXAMPLE_ENV=nfd EXAMPLE_CMD="delete --ignore-not-found=$(ignore-not-found)"
@$(MAKE) gen-apply-example EXAMPLE_ACCESS=rwo EXAMPLE_SCOPE=cluster EXAMPLE_GPU=rocm EXAMPLE_VER=v3 EXAMPLE_ENV=nfd EXAMPLE_CMD="delete --ignore-not-found=$(ignore-not-found)"

.PHONY: undeploy-examples
undeploy-examples: ## Undeploy the examples from the K8s cluster specified in ~/.kube/config.
Expand Down
3 changes: 3 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,9 @@ topics:
- [Deployment Options](docs/DeploymentOptions.md):
Details on the GKM Custom Resource Definitions, and how to tailor the
different optional fields for different environments.
- [Examples Directory](docs/Examples.md):
Details on the layout of the examples directory and how to apply the examples
to a Kubernetes cluster.
- [MCV Overview](mcv/README.md):
Overview of Model Cache Vault (MCV).
List of prerequisites to build, instructions on building MCV and usage guide.
Expand Down
19 changes: 10 additions & 9 deletions docs/DeploymentOptions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ So the storage backing the PVC dictates a lot of the options.
The most straight forward deployment is the GKM namespace scoped CRD, GKMCache,
and a Kubernetes StorageClass that supports an AccessMode of `ReadOnlyMany`.

![GKM Flowchart](images/GKM_Namespace_ReadOnlyMany.png)
![GKM Namespace ReadOnlyMany](images/GKM_Namespace_ReadOnlyMany.png)

The user creates a GKMCache with the OCI Image.
There is no way to query KubeAPI Server to determine if an AccessMode of
Expand Down Expand Up @@ -105,15 +105,15 @@ The extracted GPU Kernel Cache is then mounted in the application Pod.

Many Kubernetes Clusters are not deployed by default with a Kubernetes
StorageClass that supports an AccessMode of `ReadOnlyMany`.
In AWS this is a special request.
For example, in AWS this is a special request.
A default KIND Cluster does not support it.
If a given cluster doesn't support `ReadOnlyMany`, either to save money or
using a test cluster without the support, GKM will handle the PVC distribution
to Nodes.
However, GKM will be fighting the Kubernetes Scheduler, so some concessions need
to made to allow the Operator to distribute the extracted GPU Kernel Cache.

![GKM Flowchart](images/GKM_Namespace_ReadWriteOnce.png)
![GKM Namespace ReadWriteOnce](images/GKM_Namespace_ReadWriteOnce.png)

Still namespace scoped, the user creates a GKMCache with the OCI Image.
Unlike the previous example, there is no need to provide the optional AccessMode
Expand Down Expand Up @@ -149,7 +149,8 @@ Scheduler is run.
So the Node has not been selected at the time the Mutating Webhook runs, which
is needed to set the correct PVC in the Volume.
To get around this, GKM requires the application to be launched in a Kubernetes
DaemonSet when `ReadOnlyMany` is not supported.
DaemonSet when `ReadOnlyMany` is not supported, because Pods associated with a
DaemonSet do provide the selected Node at the time the Mutating Webhook runs.

So the user then creates a Kubernetes DaemonSet with a `volume:` of type
`persistentVolumeClaim:` and a `claimName:` set to the GKMCache name with the
Expand Down Expand Up @@ -197,14 +198,14 @@ As with GKMCache, a Kubernetes StorageClass that supports an AccessMode of
`ReadOnlyMany` is simpler in deployment because the storage backend is managing
distribution to each Node.

![GKM Flowchart](images/GKM_Cluster_ReadOnlyMany.png)
![GKM Cluster ReadOnlyMany](images/GKM_Cluster_ReadOnlyMany.png)

The user creates a ClusterGKMCache with the OCI Image.
As before, there is no way to query KubeAPI Server to determine if an
AccessMode of `ReadOnlyMany` is supported, so that must be passed in via the
CRD.

PVCs are namespace scoped so must be created in the same namespace and the Pod
PVCs are namespace scoped so must be created in the same namespace as the Pod
they are being mounted.
Since ClusterGKMCache is cluster scoped, the User must specify which Namespaces
the PVCs need to be created in.
Expand Down Expand Up @@ -286,7 +287,7 @@ This scenario covers when a given cluster doesn't support `ReadOnlyMany`, and
the GPU Kernel Cache needs to be loaded in multiple Pods that run in different
Namespaces.

![GKM Flowchart](images/GKM_Cluster_ReadWriteOnce.png)
![GKM Cluster ReadWriteOnce](images/GKM_Cluster_ReadWriteOnce.png)

The user creates a ClusterGKMCache with the OCI Image.
As with the previous ClusterGKMCache, the User must specify which Namespaces the
Expand Down Expand Up @@ -380,7 +381,7 @@ The extracted GPU Kernel Cache is then mounted in the application Pods.

## KIND Clusters

Running GKM in KIND Cluster needs some special consideration.
Running GKM in KIND Cluster needs some special consideration.
For Kubernetes Operator functionality testing, a GPU is not needed, so GPUs are
simulated.
See [Getting Started Guide](docs/GettingStartedGuide.md) for more details on
Expand All @@ -397,7 +398,7 @@ mounted directory are not setup in a way that allows the Pod to access the
mounted directory.
A workaround is to include a init container in each Pod or DaemonSet that is
volume mounting the PVC which adjusts the directory permissions properly.
The following init container is included in all the examples in
The following init container is patched in all the examples in
[./examples/](https://github.com/redhat-et/GKM/tree/main/examples):

```yaml
Expand Down
Loading
Loading