Kubernetes Configuration Guide

Complete reference for deploying madengine workloads on Kubernetes clusters.

📋 Table of Contents

Minimal Configuration (NEW!)
Quick Start
Available Configurations
Decision Matrix
Usage Examples
Data Providers
Configuration Reference
Best Practices
Troubleshooting

🌟 Minimal Configuration (NEW!)

madengine v2.0+ includes built-in presets! You only need to specify what's unique:

Single GPU - Just 1 Field!

{
  "k8s": {
    "gpu_count": 1
  }
}

Note: No "deploy": "k8s" needed - automatically inferred from k8s field presence!

Multi-GPU (2 GPUs)

{
  "k8s": {
    "gpu_count": 2
  },
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 1,
    "nproc_per_node": 2
  }
}

Multi-Node (2 nodes × 2 GPUs)

{
  "k8s": {
    "gpu_count": 2
  },
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 2,
    "nproc_per_node": 2
  }
}

Auto-Applied Defaults:

✅ Deployment type (k8s/slurm/local) inferred from config structure
✅ Resource limits (memory, CPU) based on GPU count
✅ AMD/NVIDIA-specific optimizations
✅ ROCm/CUDA environment variables
✅ NCCL/RCCL configuration
✅ Multi-node settings (host_ipc, etc.)

See: minimal/ directory for more examples and documentation.

🚀 Quick Start

Option 1: Minimal Configuration (Recommended)

# Create minimal config
cat > my-config.json << EOF
{
  "k8s": {
    "gpu_count": 1
  }
}
EOF

# Build and run
MODEL_DIR=tests/fixtures/dummy madengine build \
  --tags my_model \
  --additional-context-file my-config.json \
  --registry dockerhub

MODEL_DIR=tests/fixtures/dummy madengine run \
  --manifest-file build_manifest.json \
  --live-output

Option 2: Full Configuration (Advanced)

1. Choose a Configuration

# For single GPU testing
cp examples/k8s-configs/basic/01-native-single-node-single-gpu.json my-config.json

# For multi-GPU (2 GPUs)
cp examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json my-config.json

# For multi-node distributed (2 nodes × 2 GPUs)
cp examples/k8s-configs/basic/03-torchrun-multi-node-basic.json my-config.json

# For data provider with auto-PVC
cp examples/k8s-configs/basic/06-data-provider-with-pvc.json my-config.json

2. Customize for Your Cluster (Optional)

With built-in defaults, customization is optional. Override only what you need:

{
  "k8s": {
    "namespace": "my-namespace",         // Override default "default"
    "memory": "32Gi",                    // Override auto-calculated memory
    "node_selector": {                   // Optional: target specific nodes
      "node.kubernetes.io/instance-type": "Standard_ND96isr_H100_v5"
    }
  }
}

3. Build and Deploy

# Build container image
MODEL_DIR=tests/fixtures/dummy madengine build \
  --tags my_model \
  --additional-context-file my-config.json \
  --registry dockerhub

# Deploy and run
MODEL_DIR=tests/fixtures/dummy madengine run \
  --manifest-file build_manifest.json \
  --live-output

📁 Available Configurations

Minimal Configs (NEW - Recommended for Most Users)

Located in minimal/ directory:

General Purpose:

File	Description	GPU Count
`minimal/torchrun-single-gpu-minimal.json`	Single GPU with auto-defaults	1
`minimal/torchrun-multi-gpu-minimal.json`	Multi-GPU with auto-defaults	2
`minimal/torchrun-multi-node-minimal.json`	Multi-node with auto-defaults	2×2
`minimal/torchrun-nvidia-gpu-minimal.json`	NVIDIA GPUs with auto-defaults	4
`minimal/custom-namespace-minimal.json`	Shows override examples	1

Distributed Launchers:

File	Launcher	Description	GPUs
`minimal/torchtitan-single-node-minimal.json`	TorchTitan	LLM pre-training (single-node)	8
`minimal/vllm-single-node-minimal.json`	vLLM	LLM inference (single-node)	4
`minimal/sglang-single-node-minimal.json`	SGLang	LLM inference (single-node)	4

See minimal/README.md for detailed documentation and Launchers Guide for launcher details.

Kubernetes Secrets (`credential.json`)

Registry and runtime credentials are not stored in a ConfigMap. By default (k8s.secrets.strategy: from_local_credentials), madengine run creates Kubernetes Secrets from your local credential.json before creating the Job: a kubernetes.io/dockerconfigjson Secret when Docker Hub auth is present, and an opaque Secret with credential.json for in-container use. Mount paths match the previous behavior (/workspace/credential.json).

existing: use only pre-created Secrets; set k8s.secrets.image_pull_secret_names and k8s.secrets.runtime_secret_name (GitOps / CI). The default pull-secret list is dockerhub-rocm, the Docker Hub secret in the default namespace.
omit: no Secret creation from the client; optional extra pull secret names only.

To validate rendered YAML after a debug run, install kubeconform and run ./tests/scripts/k8s_validate_manifests.sh ./k8s_manifests from the madengine repo root (see docs/deployment.md).

Multi-node DNS (PyTorch vs Ray)

For PyTorch-native launchers (torchrun, deepspeed, torchtitan, megatron-lm, primus), multi-node Jobs use a headless Service whose name matches pod.spec.subdomain, per Kubernetes DNS rules, so pods get stable per-pod DNS names for rendezvous.

For Ray-based multi-node (vllm, sglang), a headless Service may still be created for networking, but per-pod DNS via subdomain is not applied the same way as for PyTorch; production multi-node Ray on Kubernetes often uses KubeRay (see upstream vLLM / Ray docs). Treat Job-based multi-node Ray as a best-effort path.

Primus on Kubernetes

Upstream Primus separates two ideas: a universal training driver (examples/run_pretrain.sh, used by all backends) and an optional Kubernetes workload API client (examples/run_k8s_pretrain.sh) that talks to a remote service to create workloads. Those are not the same integration as madengine’s native K8s path.

Approach	Role
Primus `run_k8s_pretrain.sh`	HTTP client to an external workload API (replicas, image, `backend`, `exp`, node labels, etc.). Does not match madengine’s `kubectl`-style Job YAML flow.
madengine `distributed.launcher: "primus"`	Renders a standard Job (and headless Service when `nnodes > 1`), injects cluster env (`MASTER_ADDR`, `NNODES`, `NODE_RANK` / `JOB_COMPLETION_INDEX`, `GPUS_PER_NODE`), `PRIMUS_CONFIG_PATH`, optional `PRIMUS_CLI_EXTRA`, and optional `BACKEND` from `distributed.primus.backend`, then runs your model script (e.g. `scripts/primus_pretrain/run.sh`).

Backends (one madengine launcher; Primus routes inside run_pretrain.sh):

Experiment path (typical)	Backend
`examples/torchtitan/...`	TorchTitan (PyTorch / `torchrun`)
`examples/megatron/...`	Megatron-LM
`examples/maxtext/...`	MaxText (JAX; coordinator env from master)

Set distributed.primus.config_path to your YAML under the Primus repo layout. Use optional distributed.primus.backend (e.g. MaxText, megatron) to emit export BACKEND=... when you want to override inference. If backend is omitted, madengine sets BACKEND from the model name when it matches primus_pretrain/<launcher>_<arch>_... (e.g. primus_pretrain/torchtitan_MI300X_qwen3_4B-pretrain → torchtitan).

MaxText caveat: For multi-node MaxText, Primus run_pretrain.sh may run in-container apt installs (InfiniBand-related packages). Many clusters disallow that unless the image is pre-baked or policy allows it. madengine logs a warning when MaxText is detected (backend or path) and nnodes > 1.

To use Primus, start from any distributed.launcher: "torchrun" example under basic/ or minimal/ and change distributed.launcher to "primus", adding a nested distributed.primus.config_path pointing at your experiment YAML (see Distributed Execution Fields below). This works for TorchTitan, Megatron, and MaxText: set distributed.primus.config_path to your experiment YAML, and rely on madengine's BACKEND inference from the model name (primus_pretrain/<launcher>_<arch>_...) or set distributed.primus.backend only when you need an explicit override. Use docker_env_vars.HF_TOKEN as a placeholder or runtime secrets — do not commit real tokens.

Full Configs (Reference Examples)

Complete configurations showing all available fields:

Training Configs (basic/):

File	GPUs	Nodes	Launcher	Use Case
`basic/01-native-single-node-single-gpu.json`	1	1	None	Basic testing, small models
`basic/01-native-single-node-single-gpu-tools.json`	1	1	None	Single GPU + monitoring
`basic/02-torchrun-single-node-multi-gpu.json`	2	1	torchrun	Multi-GPU training
`basic/02-torchrun-single-node-multi-gpu-tools.json`	2	1	torchrun	Multi-GPU + monitoring
`basic/03-torchrun-multi-node-basic.json`	2/node	2	torchrun	Multi-node basics (4 GPUs total)
`basic/04-torchrun-multi-node-advanced.json`	2/node	4	torchrun	Production multi-node (8 GPUs)
`basic/05-torchrun-nvidia-gpu-example.json`	4	1	torchrun	NVIDIA GPUs (A100, H100)
`basic/06-data-provider-with-pvc.json`	2	1+	torchrun	Data provider with auto-PVC

Distributed Launcher Configs (basic/):

File	GPUs	Nodes	Launcher	Use Case
`basic/torchtitan-multi-node-basic.json`	8/node	4	TorchTitan	Llama 3.1 70B+ training
`basic/vllm-multi-node-basic.json`	4/node	2	vLLM	High-throughput inference
`basic/sglang-multi-node-basic.json`	4/node	2	SGLang	Distributed inference

🎯 Decision Matrix: Which Config to Use?

By GPU Requirements

Scenario	Config File	GPUs	Nodes
Quick test	`basic/01-native-single-node-single-gpu.json`	1	1
Single GPU benchmark	`basic/01-native-single-node-single-gpu-tools.json`	1	1
Multi-GPU (2 GPUs)	`basic/02-torchrun-single-node-multi-gpu.json`	2	1
Multi-GPU + monitoring	`basic/02-torchrun-single-node-multi-gpu-tools.json`	2	1
Multi-node (4 GPUs)	`basic/03-torchrun-multi-node-basic.json`	2×2	2
Multi-node (8 GPUs)	`basic/04-torchrun-multi-node-advanced.json`	2×4	4
NVIDIA GPUs	`basic/05-torchrun-nvidia-gpu-example.json`	4	1
With data download	`basic/06-data-provider-with-pvc.json`	2	1+

By Use Case

Use Case	Recommended Config
Development/Testing	`basic/01-native-single-node-single-gpu.json`
Small models (BERT, ResNet)	`basic/01-native-single-node-single-gpu.json`
Medium models (GPT-2, Stable Diffusion)	`basic/02-torchrun-single-node-multi-gpu.json`
Large models (LLaMA-13B)	`basic/03-torchrun-multi-node-basic.json`
Very large models (LLaMA-70B+)	`basic/04-torchrun-multi-node-advanced.json`
Models requiring datasets	`basic/06-data-provider-with-pvc.json`
Busy/shared clusters	`basic/02-torchrun-single-node-multi-gpu.json` (2 GPUs)

💻 Usage Examples

Example 1: Single GPU Test

MODEL_DIR=tests/fixtures/dummy madengine build \
  --tags dummy \
  --additional-context-file examples/k8s-configs/basic/01-native-single-node-single-gpu.json \
  --registry dockerhub

MODEL_DIR=tests/fixtures/dummy madengine run \
  --manifest-file build_manifest.json \
  --live-output

Example 2: Multi-GPU Training (2 GPUs)

MODEL_DIR=tests/fixtures/dummy madengine build \
  --tags dummy_torchrun \
  --additional-context-file examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json \
  --registry dockerhub

MODEL_DIR=tests/fixtures/dummy madengine run \
  --manifest-file build_manifest.json \
  --live-output

Example 3: Multi-Node Training (2 nodes, 4 GPUs)

MODEL_DIR=tests/fixtures/dummy madengine build \
  --tags dummy_torchrun \
  --additional-context-file examples/k8s-configs/basic/03-torchrun-multi-node-basic.json \
  --registry dockerhub

MODEL_DIR=tests/fixtures/dummy madengine run \
  --manifest-file build_manifest.json \
  --live-output

Example 4: With Data Provider (Auto-PVC)

MODEL_DIR=tests/fixtures/dummy madengine build \
  --tags dummy_torchrun_data_minio \
  --additional-context-file examples/k8s-configs/basic/06-data-provider-with-pvc.json \
  --registry dockerhub

MODEL_DIR=tests/fixtures/dummy madengine run \
  --manifest-file build_manifest.json \
  --live-output

# Verify PVC was auto-created
kubectl get pvc madengine-shared-data

📦 Data Providers with Kubernetes

NEW: madengine automatically handles data provisioning for K8s deployments!

✨ Auto-PVC Feature

No manual PVC creation needed! madengine automatically:

Creates madengine-shared-data PVC if it doesn't exist
Selects appropriate access mode (RWO for single-node, RWX for multi-node)
Downloads data on first run
Reuses data on subsequent runs

Quick Setup

Step 1: Use data provider config

madengine build --tags dummy_torchrun_data_minio \
  --additional-context-file examples/k8s-configs/basic/06-data-provider-with-pvc.json \
  --registry dockerhub

Step 2: Run (PVC auto-created)

madengine run --manifest-file build_manifest.json --live-output

# Output shows:
# 📦 Data provider detected: Will auto-create shared data PVC
#    PVC name: madengine-shared-data (reusable across runs)
#    Access mode: RWO for single-node, RWX for multi-node (auto-selected)

Step 3: Verify (optional)

# Check PVC status
kubectl get pvc madengine-shared-data

# Check PVC contents
kubectl exec -it <pod-name> -- ls -lh /data/

How It Works

┌─────────────────────────────────────────────────────────────┐
│  1. madengine detects data provider in model config          │
├─────────────────────────────────────────────────────────────┤
│  2. Auto-creates madengine-shared-data PVC (if not exists)  │
│     • Single-node: ReadWriteOnce (RWO)                      │
│     • Multi-node: ReadWriteMany (RWX)                       │
├─────────────────────────────────────────────────────────────┤
│  3. Mounts PVC at /data in pod                               │
├─────────────────────────────────────────────────────────────┤
│  4. Downloads data from MinIO/S3/NAS to /data               │
├─────────────────────────────────────────────────────────────┤
│  5. Training starts with data at /data/<filename>           │
├─────────────────────────────────────────────────────────────┤
│  6. PVC persists - subsequent runs skip download! ✅         │
└─────────────────────────────────────────────────────────────┘

Supported Data Providers

Provider	Protocol	Configuration
MinIO	S3-compatible	Automatic (credentials from `credential.json`)
AWS S3	S3	AWS credentials in environment or `credential.json`
NAS	SSH/rsync	NAS credentials in `credential.json`
Local	Filesystem	Pre-mounted PVC

Storage Classes (local-path vs NFS)

madengine separates per-job results from long-lived shared data:

Volume	Typical use	Single-node (`nnodes: 1`)	Multi-node (`nnodes > 1`)
`{job}-results`	Benchmark artifacts (`/results`)	RWO — `single_node_results_storage_class` → `local_path_storage_class` → `storage_class` (e.g. `local-path` or `nfs-banff`)	RWX when that class can serve ReadWriteMany (`multi_node_results_storage_class` → `nfs_storage_class` → `storage_class`). Otherwise one RWO volume per pod (`local_path_storage_class`), and `perf.csv` is recovered from pod logs
`madengine-shared-data`	Dataset cache (`/data`)	RWX when that StorageClass can serve ReadWriteMany; otherwise one local disk per pod	Same rule

Built-in defaults are in presets/k8s/defaults.json: results_layout and data_layout are auto. Multi-node results and the dataset cache use nfs_storage_class / data_storage_class (nfs-banff) only when that StorageClass exists and its provisioner is a shared filesystem, or a PersistentVolume of that class is already ReadWriteMany. Otherwise each pod gets a ReadWriteOnce volume on local-path. Set results_layout or data_layout to shared or per_pod to override. A dataset claim that already exists and is not ReadWriteMany is left in place and not mounted.

Example override for a different cluster:

{
  "k8s": {
    "nfs_storage_class": "nfs-client",
    "local_path_storage_class": "standard",
    "data_storage_class": "nfs-client"
  }
}

nfs_storage_class: RWX class (e.g. nfs-banff) — used for shared-data (with data_storage_class) and multi-node results unless overridden.
local_path_storage_class: RWO class for the single-node results PVC and for a per-pod volume when ReadWriteMany storage is unavailable. New configs can set single_node_results_storage_class for the single-node claim; local_path_storage_class is still the class used for per-pod disks.
storage_class: Generic broad fallback used for both the data PVC and single-node results PVC when no more-specific key is set. Added in 2.0.3.
data_storage_class: Optional override for madengine-shared-data only (defaults to nfs_storage_class then storage_class).
single_node_results_storage_class / multi_node_results_storage_class: Optional fine-grained overrides for results PVCs.
recreate_shared_data_pvc: If true, deletes existing madengine-shared-data before create (destroys data — backup first). Use when migrating from RWO local-path to RWX NFS.

Single-Node (RWO) (results only):

✅ local-path (Rancher)
✅ AWS EBS (gp3, io2)

Multi-node & shared-data (RWX):

✅ NFS (e.g. nfs-banff, nfs-client)
✅ CephFS, GlusterFS, AWS EFS, Azure Files
❌ local-path as a shared multi-node claim (RWO only). It is used for the per-pod disk when ReadWriteMany is unavailable.

Custom PVC (Optional)

To use an existing PVC instead of auto-creation:

{
  "k8s": {
    "data_pvc": "my-existing-pvc"  // Skip auto-creation
  }
}

📖 Configuration Reference

Configuration Structure

{
  "_comment": "Description of this configuration",
  "gpu_vendor": "AMD|NVIDIA",
  "guest_os": "UBUNTU",
  
  "k8s": {
    "kubeconfig": "~/.kube/config",
    "namespace": "default",
    "gpu_count": 2,
    
    "memory": "64Gi",
    "memory_limit": "128Gi",
    "cpu": "16",
    "cpu_limit": "32",
    
    "image_pull_policy": "Always",
    "backoff_limit": 3,
    
    "node_selector": {},
    "tolerations": [],
    
    "data_pvc": null         // Optional: for data providers
  },
  
  "distributed": {
    "enabled": true,
    "backend": "nccl",
    "launcher": "torchrun",
    "nnodes": 1,
    "nproc_per_node": 2,
    "master_port": 29500
  },
  
  "env_vars": {
    "NCCL_DEBUG": "WARN",
    "NCCL_IB_DISABLE": "1",
    "OMP_NUM_THREADS": "8"
  },
  
  "debug": false
}

Field Reference

Top-Level Fields

Field	Type	Required	Description
`gpu_vendor`	string	No (default `"AMD"`)	`"AMD"` or `"NVIDIA"`
`guest_os`	string	No (default `"UBUNTU"`)	`"UBUNTU"` or `"CENTOS"`
`deploy`	string	No	Not needed — deploy target is auto-inferred from presence of the `k8s`/`kubernetes` key (or `slurm`); if set, it must agree with the inferred target
`k8s`	object	Yes	Kubernetes configuration (presence of this key is what triggers K8s deployment)
`distributed`	object	No	Distributed training (for torchrun)
`env_vars`	object	No	Custom environment variables
`debug`	boolean	No	Enable debug mode (saves manifests)

K8s Configuration Fields

Required:

Field	Type	Default	Description
`gpu_count`	integer	-	Number of GPUs per pod

Optional - Basic:

Field	Type	Default	Description
`kubeconfig`	string	`$KUBECONFIG`, else `~/.kube/config`	Path to kubeconfig
`context`	string	file's current context	Context to deploy to; must exist in the kubeconfig
`namespace`	string	`"default"`	Kubernetes namespace
`gpu_resource_name`	string	`"amd.com/gpu"`	GPU resource (`"nvidia.com/gpu"` for NVIDIA)

Optional - Resources:

Field	Type	Default	Description
`memory`	string	`"128Gi"`	Memory request (e.g., `"16Gi"`, `"64Gi"`)
`memory_limit`	string	`"256Gi"`	Memory limit (typically 2× memory)
`cpu`	string	`"32"`	CPU cores request
`cpu_limit`	string	`"64"`	CPU cores limit (typically 2× cpu)

Optional - Job Control:

Field	Type	Default	Description
`image_pull_policy`	string	`"Always"`	`"Always"`, `"IfNotPresent"`, or `"Never"`
`backoff_limit`	integer	`3`	Retry attempts before marking failed
`allow_privileged_profiling`	boolean	`null`	Whether the pod runs with the elevated privileges profiling tools need. If unset (`null`), it's auto-enabled when a `tools` config is present; set explicitly to `true`/`false` to override

Note: host_ipc is not user-configurable — madengine always sets it automatically to nnodes > 1 (enabled for multi-node jobs, disabled otherwise). Setting k8s.host_ipc in your config has no effect; it is not read anywhere.

Optional - Node Selection:

Field	Type	Default	Description
`node_selector`	object	`{}`	Label selectors for pod placement
`tolerations`	array	`[]`	Tolerations for tainted nodes

Optional - Storage:

Field	Type	Default	Description
`data_pvc`	string	`null`	Data PVC name (auto-created if using data provider)
`storage_class`	string	`local-path` (preset)	Broad fallback for the single-node results PVC when no more-specific key is set
`nfs_storage_class`	string	`nfs-banff` (preset)	RWX class for shared-data / multi-node results
`local_path_storage_class`	string	`local-path` (preset)	RWO class for single-node `{job}-results` and for a per-pod volume
`data_storage_class`	string	`nfs-banff` (preset)	Overrides SC for shared-data only
`results_layout`	string	`auto`	`auto` uses a shared results volume when the multi-node class can serve ReadWriteMany; `shared` or `per_pod` force that choice
`data_layout`	string	`auto`	`auto` uses `madengine-shared-data` when that class can serve ReadWriteMany; `shared` or `per_pod` force that choice
`single_node_results_storage_class`	string	`null`	Overrides single-node results SC (falls back to `local_path_storage_class`, then `storage_class`)
`multi_node_results_storage_class`	string	`null`	Overrides multi-node results SC (`nfs_storage_class` if unset)
`results_storage_size`	string	`"10Gi"`	Size of the per-job `{job}-results` PVC
`data_storage_size`	string	`"100Gi"`	Size of the shared `madengine-shared-data` PVC
`recreate_shared_data_pvc`	boolean	`false` (preset)	If `true`, delete `madengine-shared-data` before create (data loss)

Note: results_pvc (the per-job results PVC name) is not user-configurable — madengine always creates and uses {job_name}-results. There is no config key that overrides this name.

Distributed Execution Fields

Configuration for distributed workloads (training and inference):

Field	Type	Default	Description
`launcher`	string	-	Launcher type: `torchrun`, `deepspeed`, `torchtitan`, `megatron-lm`, `primus`, `vllm`, `sglang`
`enabled`	boolean	`false`	Enable distributed execution (legacy, prefer `launcher`)
`backend`	string	`"nccl"`	`"nccl"`, `"gloo"`, or `"mpi"`
`nnodes`	integer	`1`	Number of nodes
`nproc_per_node`	integer	gpu_count	Processes per node (= GPUs per node)
`master_port`	integer	`29500`	Master communication port

When launcher is primus, set nested primus (under distributed):

Field	Type	Description
`primus.config_path`	string	Path to the Primus experiment YAML (e.g. under `examples/torchtitan/...`).
`primus.cli_extra`	string	Optional extra arguments passed to Primus CLI.
`primus.backend`	string	Optional. If set, madengine emits `export BACKEND=...` (e.g. `MaxText`, `megatron`) before your `run.sh`.

Environment Variables

Custom environment variables for containers:

{
  "env_vars": {
    // NCCL/RCCL (AMD distributed execution)
    "NCCL_DEBUG": "WARN",              // "INFO" for debugging, "WARN" for production
    "NCCL_IB_DISABLE": "1",            // Disable InfiniBand (required for K8s)
    "NCCL_SOCKET_IFNAME": "eth0",      // Network interface
    "TORCH_NCCL_HIGH_PRIORITY": "1",   // RCCL optimization for FSDP
    "TORCH_NCCL_ASYNC_ERROR_HANDLING": "1",  // Multi-node error handling
    
    // AMD ROCm optimizations
    "GPU_MAX_HW_QUEUES": "2",          // MI series optimization
    "HSA_ENABLE_SDMA": "0",            // Disable SDMA for multi-GPU
    "HSA_FORCE_FINE_GRAIN_PCIE": "1",  // Multi-node communication
    "RCCL_ENABLE_HIPGRAPH": "0",       // Disable for compatibility
    
    // MIOpen
    "MIOPEN_FIND_MODE": "1",           // Use compiled kernels
    "MIOPEN_USER_DB_PATH": "/tmp/.miopen",  // Writable cache location
    
    // General
    "OMP_NUM_THREADS": "8"             // OpenMP threads
  }
}

🎓 Best Practices

Resource Sizing

Single GPU:

GPUs: 1
Memory: 16Gi (request), 32Gi (limit)
CPU: 8 (request), 16 (limit)

Multi-GPU (2 GPUs):

GPUs: 2
Memory: 64Gi (request), 128Gi (limit)
CPU: 16 (request), 32 (limit)

Multi-Node (2 nodes × 2 GPUs):

GPUs: 2 per node (4 total)
Memory: 64Gi per node
CPU: 16 per node
host_ipc: enabled automatically (nnodes > 1)

Multi-Node Advanced (4 nodes × 2 GPUs):

GPUs: 2 per node (8 total)
Memory: 128Gi per node
CPU: 24 per node
host_ipc: enabled automatically (nnodes > 1)
PVCs: Recommended for data and results

Local `k8s_results` layout (after `madengine run`)

Artifacts are written under ./k8s_results/<job_name>/:

Path	Contents
`<job_name>/<pod_name>/pod.log`	Container log from the Kubernetes API
`<job_name>/<pod_name>/pvc/`	Copy of `/results/<subdir>/` from the results PVC, matched to that pod
`<job_name>/pvc_unmapped/<subdir>/`	PVC folders that could not be matched to a pod name

Write durable outputs under /results/<replica-id>/ in the container so each replica’s files land in a predictable PVC subdir (e.g. hostname or jobname-<index>). Madengine maps that subdir to the full pod name when copying to the host.

Distributed Launchers

Training Launchers:

torchrun: Standard PyTorch DDP/FSDP training
deepspeed: ZeRO optimization for memory efficiency
megatron-lm: Megatron-LM tensor and pipeline parallelism
torchtitan: LLM pre-training with multi-dimensional parallelism (FSDP2+TP+PP)
primus: Unified Primus pretrain (Megatron / TorchTitan / MaxText experiment YAML; see Primus on Kubernetes)

Inference Launchers:

vllm: High-throughput LLM serving with continuous batching
sglang: Fast LLM inference with structured generation

When to use distributed launchers: ✅ Multi-GPU on single node (2+ GPUs) ✅ Multi-node distributed workloads ✅ Large model training or inference ✅ Production-scale deployments

When NOT to use: ❌ Single GPU workloads ❌ Simple benchmarks without distributed execution ❌ Development and testing (use single GPU)

See Launchers Guide for the comprehensive launcher guide.

AMD ROCm Optimizations

Always set in K8s:

NCCL_IB_DISABLE=1 - InfiniBand not available in K8s
NCCL_SOCKET_IFNAME=eth0 - Use Ethernet interface
MIOPEN_FIND_MODE=1 - Avoid MIOpen find-db warnings
MIOPEN_USER_DB_PATH=/tmp/.miopen - Writable cache

For multi-GPU:

TORCH_NCCL_HIGH_PRIORITY=1 - RCCL optimization
GPU_MAX_HW_QUEUES=2 - MI series GPUs
HSA_ENABLE_SDMA=0 - Disable SDMA for better P2P

For multi-node:

Shared memory (host_ipc) is enabled automatically when distributed.nnodes > 1 — no config needed
HSA_FORCE_FINE_GRAIN_PCIE=1 - Cross-node communication
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 - Better error handling

For Busy/Shared Clusters

✅ Recommendations:

Use 1-2 GPUs instead of 8 to avoid scheduling conflicts
Test with single-GPU first, then scale up
Monitor GPU availability: kubectl describe nodes | grep amd.com/gpu
Use node selectors to target specific node types
Consider resource quotas and limits

🐛 Troubleshooting

Pod Stuck in Pending

Symptoms:

kubectl get pods
# NAME                    READY   STATUS    RESTARTS   AGE
# madengine-job-xxxxx     0/1     Pending   0          5m

Solutions:

Check GPU availability:

kubectl describe nodes | grep -A5 "amd.com/gpu\|nvidia.com/gpu"
# Shows: Allocatable vs Allocated

Reduce GPU count:

{
  "k8s": {
    "gpu_count": 1  // Try 1 instead of 2
  }
}

Check node selectors:

kubectl get nodes --show-labels | grep instance-type
# Verify your node_selector matches actual node labels

NCCL/RCCL Errors

Error: "Duplicate GPU detected"

Solution: gpu_count in config must match nproc_per_node in distributed config

Error: "Network connection failed"

Solution: Verify NCCL_SOCKET_IFNAME matches your network interface
Check: kubectl exec <pod> -- ip addr

Error: "NCCL initialization failed"

Solution: Ensure these are set:
  NCCL_IB_DISABLE=1
  NCCL_SOCKET_IFNAME=eth0
Enable debug: NCCL_DEBUG=INFO

Out of Memory (OOM)

Symptoms:

kubectl get pods
# NAME                    READY   STATUS      RESTARTS   AGE
# madengine-job-xxxxx     0/1     OOMKilled   0          2m

Solutions:

Increase memory limit:

{
  "k8s": {
    "memory": "128Gi",      // Increase request
    "memory_limit": "256Gi" // Increase limit (2× request)
  }
}

Reduce batch size (in model config)
Enable gradient checkpointing (model-specific)

Job Failed

Check logs:

kubectl logs <pod-name>
kubectl describe pod <pod-name>

Common issues:

Image pull failed: Check registry credentials
Permission denied: Check security context and PVC permissions
Command not found: Verify scripts are in container
Timeout: Increase backoff_limit or job timeout

Multi-Node Communication Fails

Symptoms:

NCCL WARN ... Connection refused
NCCL WARN ... Unable to find NCCL communicator

Solutions:

Host IPC is automatic — no config needed: madengine always enables shared memory (hostIPC) for the pod whenever distributed.nnodes > 1; there is no host_ipc config key to set, and setting one has no effect. If communication is still failing, confirm nnodes in your distributed config is actually > 1 (a value of 1 disables host IPC).
Verify headless service:

kubectl get svc | grep madengine
# Should show ClusterIP: None (headless)

Check DNS resolution:

kubectl exec <pod> -- nslookup madengine-job-name.default.svc.cluster.local

Increase timeout:

{
  "env_vars": {
    "NCCL_TIMEOUT": "600"  // 10 minutes
  }
}

Data Provider Issues

Error: "Read-only file system"

Solution: Bug in template - should be fixed in latest version
The data PVC mount must have readOnly: false

Error: "Data file not found"

Check:
1. PVC exists: kubectl get pvc madengine-shared-data
2. PVC is Bound: kubectl describe pvc madengine-shared-data
3. Data downloaded: kubectl exec <pod> -- ls -lh /data/
4. MAD_DATAHOME=/data set correctly

Error: "PVC pending"

Solution: Storage class issue
Check: kubectl describe pvc madengine-shared-data
Fix: Ensure your cluster has NFS storage class for RWX
For single-node: Any storage class works (uses RWO)

🔍 Configuration Comparison

Feature	Single GPU	Multi-GPU (2)	Multi-Node (2×2)	Advanced (4×2)
GPUs	1	2	4	8
Nodes	1	1	2	4
Memory	16Gi	64Gi	64Gi/node	128Gi/node
CPU	8	16	16/node	24/node
torchrun	❌	✅	✅	✅
host_ipc	❌	❌	✅	✅
NCCL Vars	Basic	Yes	Full	Advanced
PVCs	No	No	Optional	Recommended
Tolerations	No	No	No	Yes
Complexity	⭐	⭐⭐	⭐⭐⭐	⭐⭐⭐⭐

📚 Advanced Topics

Node Selectors

Target specific node types:

{
  "k8s": {
    "node_selector": {
      "node.kubernetes.io/instance-type": "Standard_ND96isr_H100_v5",
      "gpu-type": "mi300x",
      "zone": "us-west-2a"
    }
  }
}

Check available labels:

kubectl get nodes --show-labels

Tolerations

Schedule on tainted nodes:

{
  "k8s": {
    "tolerations": [
      {
        "key": "gpu",
        "operator": "Equal",
        "value": "true",
        "effect": "NoSchedule"
      }
    ]
  }
}

Custom Storage Classes

For multi-node with custom NFS:

{
  "k8s": {
    "storage_class": "nfs-client"  // Your NFS storage class
  }
}

Check available storage classes:

kubectl get storageclass

Debug Mode

Save rendered K8s manifests for inspection:

{
  "debug": true,
  "k8s": {
    "output_dir": "./debug_manifests"
  }
}

Manifests saved to:

./debug_manifests/job.yaml
./debug_manifests/configmap.yaml
./debug_manifests/service.yaml (multi-node only)

📊 Resource Scaling Guide

Single GPU (Development/Testing)

GPUs: 1
Memory: 16Gi (request), 32Gi (limit)
CPU: 8 (request), 16 (limit)
Use Case: Small models, debugging, cost-effective testing

2 GPUs (Recommended for Shared Clusters)

GPUs: 2
Memory: 64Gi (request), 128Gi (limit)
CPU: 16 (request), 32 (limit)
Use Case: Multi-GPU training, testing on busy clusters

4 GPUs (Multi-Node Testing)

Configuration: 2 nodes × 2 GPUs per node
Memory: 64Gi per node
CPU: 16 per node
host_ipc: enabled automatically (nnodes > 1)
Use Case: Distributed training development

8 GPUs (Production Multi-Node)

Configuration: 4 nodes × 2 GPUs per node
Memory: 128Gi per node
CPU: 24 per node
host_ipc: enabled automatically (nnodes > 1)
PVCs: Recommended
Use Case: Large-scale production training

🎯 Examples by Scenario

Scenario 1: Quick Smoke Test

# Use minimal config (defaults for everything)
madengine build --tags dummy \
  --additional-context-file examples/k8s-configs/basic/01-native-single-node-single-gpu.json \
  --registry dockerhub

madengine run --manifest-file build_manifest.json

Scenario 2: Benchmark on Busy Cluster

# Use 2 GPUs to avoid scheduling conflicts
madengine build --tags resnet50 \
  --additional-context-file examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json \
  --registry dockerhub

madengine run --manifest-file build_manifest.json --live-output

Scenario 3: Large Model Training

# Multi-node for large models
madengine build --tags llama_13b \
  --additional-context-file examples/k8s-configs/basic/03-torchrun-multi-node-basic.json \
  --registry dockerhub

madengine run --manifest-file build_manifest.json --live-output

Scenario 4: Production with Datasets

# Data provider with auto-PVC
madengine build --tags bert_large \
  --additional-context-file examples/k8s-configs/basic/06-data-provider-with-pvc.json \
  --registry dockerhub

madengine run --manifest-file build_manifest.json --live-output

# Verify PVC
kubectl get pvc madengine-shared-data
kubectl exec <pod> -- ls -lh /data/

Scenario 5: GPU Profiling

# Use *-tools.json variant for monitoring
madengine build --tags model \
  --additional-context-file examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu-tools.json \
  --registry dockerhub

madengine run --manifest-file build_manifest.json --live-output

# Profiling results in PVC
kubectl cp <pod>:/results/gpu_info_*.csv ./

🔧 Customization Guide

Start from Example

# Copy closest match
cp examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json my-custom-config.json

# Edit
vim my-custom-config.json

Common Customizations

Change GPU count:

{
  "k8s": {
    "gpu_count": 4  // Change from 2 to 4
  },
  "distributed": {
    "nproc_per_node": 4  // Must match gpu_count
  }
}

Target specific node type:

{
  "k8s": {
    "node_selector": {
      "gpu-type": "mi300x"
    }
  }
}

Increase memory:

{
  "k8s": {
    "memory": "128Gi",
    "memory_limit": "256Gi"  // 2× memory
  }
}

Add custom environment variables:

{
  "env_vars": {
    "MY_CUSTOM_VAR": "value",
    "BATCH_SIZE": "256"
  }
}

📈 Performance Tips

Multi-GPU Scaling

Expected Scaling Efficiency:

2 GPUs: ~95-100% (ideal: 2× single GPU)
4 GPUs: ~85-95% (network overhead)
8 GPUs: ~80-90% (more communication)

Factors affecting scaling:

Model size (larger = better scaling)
Batch size (larger = less communication)
Network bandwidth (faster = better)
NCCL configuration (optimized = better)

NCCL Tuning for AMD

Basic (included in examples):

{
  "NCCL_DEBUG": "WARN",
  "NCCL_IB_DISABLE": "1",
  "TORCH_NCCL_HIGH_PRIORITY": "1",
  "GPU_MAX_HW_QUEUES": "2"
}

Advanced (for production):

{
  "NCCL_DEBUG": "WARN",
  "NCCL_IB_DISABLE": "1",
  "NCCL_SOCKET_IFNAME": "eth0",
  "TORCH_NCCL_HIGH_PRIORITY": "1",
  "TORCH_NCCL_ASYNC_ERROR_HANDLING": "1",
  "GPU_MAX_HW_QUEUES": "2",
  "HSA_ENABLE_SDMA": "0",
  "HSA_FORCE_FINE_GRAIN_PCIE": "1",
  "RCCL_ENABLE_HIPGRAPH": "0",
  "MIOPEN_FIND_MODE": "1"
}

Monitoring During Training

# Watch pod status
kubectl get pods -w

# Monitor resource usage
kubectl top pods

# Stream logs
kubectl logs -f <pod-name>

# Check GPU utilization (from pod)
kubectl exec <pod> -- rocm-smi

# Check NCCL communication (multi-node)
kubectl logs <pod> | grep NCCL

🎓 Learning Path

Level 1: Beginner

Start with basic/01-native-single-node-single-gpu.json
Test on single GPU
Understand basic K8s concepts
Monitor logs and results

Level 2: Intermediate

Try basic/02-torchrun-single-node-multi-gpu.json
Learn distributed execution with torchrun (training workloads)
Understand NCCL configuration
Profile GPU utilization

Level 3: Advanced

Deploy basic/03-torchrun-multi-node-basic.json
Master multi-node networking
Optimize NCCL parameters
Use PVCs for data and results

Level 4: Expert

Customize basic/04-torchrun-multi-node-advanced.json
Fine-tune for your cluster
Implement node affinity and tolerations
Scale to 8+ nodes

📋 Configuration Checklist

Before deploying to production:

🔗 Related Documentation

Main Documentation: ../../README.md
Data Provider Guide: ../../docs/K8S_DATA_PROVIDER_GUIDE.md (if exists)
Deployment Guide: ../../K8S_DEPLOYMENT_GUIDE.md (if exists)
Performance CSV Format: ../../PERF_CSV_UNIFIED_FORMAT.md (if exists)

📝 File Structure

examples/k8s-configs/
├── README.md                                        # This file
├── basic/
│   ├── 01-native-single-node-single-gpu.json        # 1 GPU, basic
│   ├── 01-native-single-node-single-gpu-tools.json  # 1 GPU + monitoring
│   ├── 02-torchrun-single-node-multi-gpu.json       # 2 GPUs, distributed
│   ├── 02-torchrun-single-node-multi-gpu-tools.json # 2 GPUs + monitoring
│   ├── 03-torchrun-multi-node-basic.json            # 2 nodes × 2 GPUs
│   ├── 04-torchrun-multi-node-advanced.json         # 4 nodes × 2 GPUs
│   ├── 05-torchrun-nvidia-gpu-example.json          # NVIDIA GPUs
│   └── 06-data-provider-with-pvc.json               # Data provider + auto-PVC
└── minimal/                                          # Minimal configs (see minimal/README.md)

✅ Summary

8 configuration files covering all common scenarios
Auto-PVC creation for data providers - no manual setup!
Production-ready with best practices
Well-documented with inline comments
Tested on AMD MI300X and NVIDIA clusters
Ready to use - just copy and customize!

Last Updated: December 6, 2025
Status: Production Ready ✅

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Kubernetes Configuration Guide

📋 Table of Contents

🌟 Minimal Configuration (NEW!)

Single GPU - Just 1 Field!

Multi-GPU (2 GPUs)

Multi-Node (2 nodes × 2 GPUs)

🚀 Quick Start

Option 1: Minimal Configuration (Recommended)

Option 2: Full Configuration (Advanced)

1. Choose a Configuration

2. Customize for Your Cluster (Optional)

3. Build and Deploy

📁 Available Configurations

Minimal Configs (NEW - Recommended for Most Users)

Kubernetes Secrets (credential.json)

Multi-node DNS (PyTorch vs Ray)

Primus on Kubernetes

Full Configs (Reference Examples)

🎯 Decision Matrix: Which Config to Use?

By GPU Requirements

By Use Case

💻 Usage Examples

Example 1: Single GPU Test

Example 2: Multi-GPU Training (2 GPUs)

Example 3: Multi-Node Training (2 nodes, 4 GPUs)

Example 4: With Data Provider (Auto-PVC)

📦 Data Providers with Kubernetes

✨ Auto-PVC Feature

Quick Setup

How It Works

Supported Data Providers

Storage Classes (local-path vs NFS)

Custom PVC (Optional)

📖 Configuration Reference

Configuration Structure

Field Reference

Top-Level Fields

K8s Configuration Fields

Distributed Execution Fields

Environment Variables

🎓 Best Practices

Resource Sizing

Local k8s_results layout (after madengine run)

Distributed Launchers

AMD ROCm Optimizations

For Busy/Shared Clusters

🐛 Troubleshooting

Pod Stuck in Pending

NCCL/RCCL Errors

Out of Memory (OOM)

Job Failed

Multi-Node Communication Fails

Data Provider Issues

🔍 Configuration Comparison

📚 Advanced Topics

Node Selectors

Tolerations

Custom Storage Classes

Debug Mode

📊 Resource Scaling Guide

Single GPU (Development/Testing)

2 GPUs (Recommended for Shared Clusters)

4 GPUs (Multi-Node Testing)

8 GPUs (Production Multi-Node)

🎯 Examples by Scenario

Scenario 1: Quick Smoke Test

Scenario 2: Benchmark on Busy Cluster

Scenario 3: Large Model Training

Scenario 4: Production with Datasets

Scenario 5: GPU Profiling

🔧 Customization Guide

Start from Example

Common Customizations

📈 Performance Tips

Kubernetes Secrets (`credential.json`)

Local `k8s_results` layout (after `madengine run`)