Skip to content

gpu

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

gpu

README.md

GPU Workloads on EKS Auto Mode

Table of Contents

Prerequisites

Cluster deployed and kubectl configured per Quick Start.

Overview

NVIDIA GPUs on Amazon EC2 supercharge your workloads with powerful GPU acceleration. Key benefits include:

🚀 High Performance Computing

  • GPU accelerators from g3, g4, g5, g6, p3, and p4 families
  • Optimized for machine learning and graphics workloads
  • Ideal for running large language models

🤖 AI/ML Capabilities

  • Perfect for GenAI model deployment
  • Supports complex deep learning tasks
  • Accelerated model inference

⚙️ Flexible Configuration

  • Customizable instance types
  • Scalable GPU resources
  • EKS Auto Mode integration

This example demonstrates deploying a GenAI model (Qwen 3 32b fp8) on EKS Auto Mode.

⚠️ Prerequisites:

  • You must have a Hugging Face account with an access token!
  • GPU Instance Availability: Many AWS accounts have a default service quota of 0 for p* and g* GPU instance types. You may need to request a quota increase through the AWS Service Quotas console before deploying GPU workloads. This process can take 24-48 hours for approval.

Architecture

This example showcases GPU-accelerated workloads on EKS Auto Mode using the following components:

🖥️ Instance Types

  • Default: G5, G6 or G6e instances (optimized for ML workloads)
  • Customization: Available in gpu-nodepool.yaml.tpl

🔧 Key Components

📦 Infrastructure

  • NodePool and NodeClass for GPU workload management
  • Application Load Balancer (Ingress) for HTTP access — internal-scheme by default, opt-in internet-facing + HTTPS via var.base_domain

🧠 AI Components

  • Hugging Face model deployment (Qwen 3 32b fp8)
  • Interactive Web UI for model interaction

Implementation Steps

1. Get Hugging Face Access Token

Create a Hugging Face account and generate a FINEGRAINED Access Token

2. Deploy GPU NodePool

Deploy the NodePool that will manage our GPU instances:

kubectl apply -f ../../nodepools/gpu-nodepool.yaml

⚠️ The GPU NodePool applies the following taint to ensure only GPU-compatible workloads are scheduled on these nodes:

taints:
  - key: "nvidia.com/gpu"
    value: "true"
    effect: "NoSchedule"   # Prevents non-GPU pods from scheduling

Any pods that need to run on GPU nodes must include matching tolerations in their specifications.

3. Configure Namespace and Secrets

  1. Create Namespace:
kubectl apply -f namespace.yaml
  1. Add Hugging Face Token:
# Replace <your_actual_hugging_face_token> with your token
kubectl create secret generic hf-secret \
  -n vllm-inference \
  --from-literal=hf_api_token=<your_actual_hugging_face_token>

4. Deploy Model and UI

  1. Deploy the Model: Following command will deploy Qwen3 32b (fp8). We also have another manifest file that allows you to deploy Deepseek instead.
kubectl apply -f model-qwen3-32b-fp8.yaml

✅ The model deployment includes the required toleration to run on GPU nodes:

tolerations:
  - key: "nvidia.com/gpu"     # Matches the GPU node taint
    value: "true"
    effect: "NoSchedule"      # Allows scheduling on tainted nodes

This toleration enables the pods to be scheduled on our GPU-enabled instances.

  1. Deploy the Web UI:
kubectl apply -f open-webui.yaml

5. Deploy the Service and Ingress

Apply the ClusterIP Service + ALB Ingress that front the Web UI:

kubectl apply -f lb-service.yaml

📘 The manifest provisions a ClusterIP Service open-webui-service (port 80 → 8080) and an Ingress open-webui-ingress using the cluster-wide alb IngressClass. By default the ALB scheme is internal (VPC-only) — no public endpoint is created.

6. Access the Application

By default, this example exposes its UI via an internal ALB — reachable from inside the VPC only. To access it from your laptop, use kubectl port-forward:

kubectl port-forward -n vllm-inference svc/open-webui-service 8080:80
# then open http://localhost:8080

If you want to inspect the ALB DNS name directly (e.g. from a bastion or VPN):

kubectl get ingress open-webui-ingress \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' \
  -n vllm-inference

To expose the UI publicly over HTTPS, deploy the Terraform stack with var.base_domain set to a public Route53 zone you own (see top-level README). The example will be reachable at https://gpu.<full_domain> once external-dns publishes the record.

⚠️ Select a Model: If you are unable to select a model, it means the model is still being downloaded and is not yet being served by our inferencing server pod. Just refresh until you see a model available.

Clean up

kubectl delete -f lb-service.yaml
kubectl delete -f open-webui.yaml
kubectl delete -f model-qwen3-32b-fp8.yaml
kubectl delete -f namespace.yaml
kubectl delete -f ../../nodepools/gpu-nodepool.yaml

Troubleshooting

🔧 Common issues and their solutions:

🎯 Model Deployment Issues

  1. GPU Node Provisioning

    • Verify nodes are properly labeled for GPU
    • Check node status with kubectl get nodes
    • Ensure GPU drivers are initialized

  2. Model Initialization

    • Check pod logs for startup errors: 
      kubectl logs -n vllm-inference deployment/qwen3-32b-fp8
    • Verify Hugging Face token is valid

🔄 Load Balancer Issues

  1. Ingress / ALB Status
    • Check Ingress provisioning: 
      kubectl describe ingress open-webui-ingress -n vllm-inference
    • Check ALB controller logs: 
      kubectl logs -n kube-system deployment/aws-load-balancer-controller
    • Verify the ALB hostname resolves and target group health is green

💻 Resource Constraints

  1. GPU Capacity
    • Ensure sufficient GPU quota in your AWS account
    • Monitor GPU utilization: 
      kubectl describe node <node-name>
    • Check for pod scheduling events: 
      kubectl get events -n vllm-inference

💡 Tip: Always check pod logs and events first when troubleshooting deployment issues.