Skip to content

troubleshooting.md

Latest commit

 

History

History
277 lines (190 loc) · 14.1 KB

troubleshooting.md

File metadata and controls

277 lines (190 loc) · 14.1 KB
Raw

Troubleshooting

Diagnose from the controller outward. Preserve exact errors, timestamps, and Kubernetes events, but redact tokens, provider credentials, repository names, and public addresses before sharing output.

Fast triage

helm status devboxes -n devboxes
kubectl get deployment,service,pvc,pod -n devboxes -o wide
kubectl logs deployment/devboxes -n devboxes --tail=200
kubectl get events -n devboxes --sort-by=.lastTimestamp
devbox --version

Then inspect one box:

kubectl get deployment,service,pvc,pod -n devboxes \
  -l devboxes.bonalab.org/name=atlas -o wide
kubectl describe pod -n devboxes \
  -l devboxes.bonalab.org/name=atlas

Controller is not ready

Symptoms include /ready returning 503, CLI transport failures, or the dashboard failing to list boxes.

  1. Confirm the controller pod is Ready and inspect its logs.
  2. Verify the controller ServiceAccount exists in the release namespace and a Role plus RoleBinding exists in every configured tenant namespace.
  3. Check Kubernetes API reachability and service-account token mounting on the controller.
  4. Confirm every configured tenant namespace exists and the RoleBinding references the controller ServiceAccount in the release namespace.
  5. Review NetworkPolicy or API-server authorization denials.

The controller needs namespaced access to Deployments, Services, Pods, PVCs, and its generated per-workspace Insights Secrets. It does not need cluster-wide RBAC. Readiness lists Deployments in every configured tenant, so one missing RoleBinding deliberately makes the controller not ready.

If Insights is enabled, inspect the central PVC and controller logs for SQLite open, migration, or locking errors. The volume must support normal filesystem locking and WAL. Avoid NFS-backed claims with unreliable lock semantics.

Authentication fails

For 401 Authentication required, repeat browser authorization for the profile:

DEVBOX_CONFIG=/path/to/profile.toml devbox login --url https://devboxes.example.com

If the controller Secret changed, restart the controller and log in again. Existing sessions are signed with the old token and become invalid.

If authorization times out, confirm the CLI is still running, the browser reached the same controller URL, and the callback uses the exact http://127.0.0.1:PORT/callback shown in the authorization request. Use --no-open when a browser launcher is unavailable. A denial is intentional and issues no token; restart devbox login to make a new request. Never copy a master token into terminal output as a workaround.

For browser 403 Missing or invalid CSRF token, clear the site cookies and log in again. Confirm that an HTTPS installation sets controller.cookieSecure=true. A secure cookie is not sent over plain HTTP.

For 403 Tenant access is forbidden, list the principal's configured memberships, verify X-Devboxes-Tenant, --tenant, or DEVBOX_TENANT, and restart the controller after a principal Secret or Helm-policy change. Unknown tenants and known-but-unauthorized tenants intentionally return the same error.

For 403 The current tenant role is read-only, select a membership with member or admin authority or ask the operator to change the role. Permanent home or Insights purge requires admin; the master identity has operator authority in every tenant.

TLS or URL validation fails

The CLI rejects non-loopback HTTP, embedded credentials, query strings, and fragments. Use a valid certificate for the controller hostname. There is intentionally no skip-verification option.

For local access:

kubectl -n devboxes port-forward service/devboxes 8000:8000
devbox login --url http://127.0.0.1:8000

For remote access, fix DNS, certificate trust, and the ingress route rather than weakening the client.

A box remains starting

The starting state means the controller is waiting for pod readiness or a usable SSH Service address.

Check the pod's scheduling condition, image pull, volume mounts, and readiness probe. Common event reasons include insufficient CPU or memory, an unbound PVC, an unavailable StorageClass, image pull authentication failure, or a missing workspace Secret.

The workspace entrypoint requires SSH_AUTHORIZED_KEYS. Validate the key exists without printing its value:

kubectl get secret devboxes-workspace -n devboxes \
  -o go-template='{{if index .data "SSH_AUTHORIZED_KEYS"}}present{{else}}missing{{end}}{{"\n"}}'

If the pod is Ready but the box still starts, inspect the Service next.

Tenant quota rejects create or start

A direct Kubernetes ResourceQuota admission denial is returned as 409 Conflict. Pod-level enforcement may instead appear asynchronously as a degraded workspace with a quota message from its Deployment. In either case, inspect the active tenant's configured ceiling and current usage:

devbox --tenant platform list
kubectl describe resourcequota -n devboxes-platform
kubectl get events -n devboxes-platform --sort-by=.lastTimestamp

Stopping a box releases pod CPU, memory, and GPU requests but retains its PVC and storage quota. Delete without purge also retains the PVC. Increase quota, stop or purge an approved box, or select a smaller allowed preset based on the constrained resource. Do not bypass quota by moving resources manually to another tenant namespace.

A GPU box remains starting

Start with the user-visible allocation and scheduler message:

devbox status atlas
devbox gpu profiles
kubectl describe pod -n devboxes \
  -l devboxes.bonalab.org/name=atlas

Then verify each layer in order:

  1. The profile's exact extended resource appears under status.allocatable on at least one node.
  2. At least one node matches every profile selector.
  3. The profile tolerates every blocking taint on that node.
  4. The configured RuntimeClass exists, or the vendor runtime is already the node default when the profile omits it.
  5. Enough resource units are free. Device-plugin sharing or partitioning semantics match the profile description.
  6. The profile workspace image pulls successfully and is compatible with the node driver.

Useful cluster checks include:

kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
kubectl get runtimeclass
kubectl get events -n devboxes --sort-by=.lastTimestamp
kubectl get deployment devbox-atlas -n devboxes \
  -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'

Replace nvidia.com/gpu with the profile's resource. Insufficient RESOURCE means Kubernetes has no free advertised units that also satisfy the pod constraints. didn't match Pod's node affinity/selector points to labels. Untolerated taint messages point to profile policy. A missing RuntimeClass is reported during pod admission.

Do not patch the generated Deployment or add privileged mode and host device mounts. The controller owns that resource, and manual changes make the box non-reproducible. Correct the driver, capacity, or Helm profile, then delete and recreate a disposable test box. Existing boxes intentionally retain their creation-time profile snapshot.

A custom image profile fails

First confirm the selected profile and the generated container contract:

devbox status atlas
devbox image profiles
kubectl get deployment devbox-atlas -n devboxes \
  -o jsonpath='{.metadata.annotations.image\.devboxes\.bonalab\.org/profile}{"\n"}{.spec.template.spec.containers[*].name}{"\n"}'
kubectl describe pod -n devboxes \
  -l devboxes.bonalab.org/name=atlas

For a sidecar profile, inspect its own logs and verify the declared pod-local port from the workspace container:

kubectl logs deployment/devbox-atlas -n devboxes -c custom-image --tail=100
kubectl exec deployment/devbox-atlas -n devboxes -c devbox -- \
  curl --fail http://127.0.0.1:8080/

ErrImagePull and ImagePullBackOff indicate a registry, architecture, reference, or pull-credential problem. Validate the approved image on an eligible node before changing the profile. A crash loop after a successful pull is application behavior; check the image's expected environment and command, remembering that Devboxes intentionally does not inject credentials, persistent storage, or a public Service into a sidecar. Sidecars must already run as a non-root user and use a port from 1024 through 65535. For NGINX, use the unprivileged image variant rather than weakening the generated security context.

For a workspace-mode profile, test the complete Devboxes contract instead of treating it like a generic container. The primary container must initialize the persistent home and start SSH on port 2222. Revert the profile to a known compatible Devboxes-derived image if SSH readiness fails. Do not work around this by patching a generated Deployment, adding privilege, or mounting host paths.

SSH address remains pending

For LoadBalancer, inspect .status.loadBalancer.ingress and the load-balancer controller events:

kubectl get service devbox-atlas-ssh -n devboxes -o yaml

Confirm the provider supports one LoadBalancer Service per box, the address quota is not exhausted, and any annotations or loadBalancerClass match the installed controller.

For NodePort, confirm workspace.sshService.host resolves to a reachable node or node load balancer and that the allocated NodePort is allowed from the client network.

SSH connection is refused or times out

Distinguish network reachability from SSH authentication:

devbox status atlas
nc -vz SSH_HOST SSH_PORT
ssh -vvv -p SSH_PORT dev@SSH_HOST

Check the Service endpoints, firewall rules, load-balancer health, externalTrafficPolicy, and pod readiness. With externalTrafficPolicy: Local, the load balancer must route only to the node that currently hosts the workspace pod.

If OpenSSH reports a host-key change after an intentional purge, remove only the alias named in the error from known_hosts. An unexpected host-key change without a purge should be investigated before connecting.

SSH closes after authentication

If public-key authentication succeeds but the connection closes with missing or unsuitable terminal, verify the workspace image matches the current release:

devbox status atlas
kubectl get deployment devbox-atlas -n devboxes \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Current images install broad ncurses terminfo plus the pinned Ghostty xterm-ghostty entry. devbox-shell preserves an installed, safe TERM; otherwise it chooses xterm-256color, screen-256color, xterm, or vt100 and prints one sanitized warning. Do not disable host-key checking or remove an SSH-key passphrase for this error.

Repository clone fails

Public GitHub repositories need no token. Private repositories require a valid GH_TOKEN with access to that repository. Inspect /var/log/devbox-init.log inside the workspace, confirm the repository value matches the accepted owner/repository or HTTPS GitHub form, and verify provider access with gh auth status.

A clone runs only when the target home is new and no existing repository occupies the destination. Reused PVCs intentionally preserve their current working tree.

PVC is pending or cannot expand

kubectl describe pvc devbox-atlas-home -n devboxes
kubectl get storageclass

Confirm a default or configured StorageClass exists, supports ReadWriteOnce, has available capacity, and can provision in the scheduled topology. Expansion requires allowVolumeExpansion: true. Devboxes never shrinks a retained PVC.

Do not delete a PVC as a troubleshooting shortcut. Back it up and confirm the data-retention decision first.

Box is degraded

The controller marks known CrashLoopBackOff, ErrImagePull, ImagePullBackOff, and failed pod states as degraded. Inspect pod events, current and previous logs, workspace Secret mounts, and image credentials:

kubectl logs -n devboxes -l devboxes.bonalab.org/name=atlas
kubectl logs -n devboxes -l devboxes.bonalab.org/name=atlas --previous

Resolve the underlying Kubernetes condition, then allow the Deployment to reconcile. Recreating with the same name can reuse the retained home, but it should not replace diagnosis of a repeatable startup failure.

TTL stopped the box

TTL expiry scales compute to zero and retains storage. Resume normally:

devbox start atlas
devbox ssh atlas

Starting renews the original TTL. Use a longer TTL for sustained work, up to the controller maximum, or stop explicitly when idle.

Insights is empty, stale, or partial

Start with the operator view and one workspace:

devbox metrics status --box atlas
kubectl get deployment devbox-atlas -n devboxes \
  -o jsonpath='{.spec.template.spec.containers[*].name}{"\n"}'
kubectl logs deployment/devbox-atlas -n devboxes -c insights-agent --tail=100

restart_required means an active workspace predates the enabled or current collector template. Stop and start it normally. partial means no first batch has arrived or some selected measurements are unavailable. stale means the last heartbeat is too old. data_loss_detected means the durable outbox exceeded its byte or age bound and dropped oldest data.

The collector is intentionally fail-open for SSH. A sidecar error does not make the main container unready. Check controller reachability from the pod, the generated per-workspace Secret reference, the central PVC, and the queue fields from devbox metrics status.

Codex and Claude Code must run inside an Insights-enabled workspace and send metrics to the injected loopback endpoint. Provider telemetry can change between client versions; Devboxes validates the versions shipped in the workspace image. Git history cloned before the first scan is a baseline and is not imported. Only later reachable commits and current tracked working-tree aggregates appear.

If central data is no longer wanted, use devbox metrics purge --box NAME. Workspace delete --purge removes the home PVC and local outbox but intentionally does not erase central history.

Safe support bundle

Follow SUPPORT.md. Replace names and addresses where needed, never include Secret manifests or decoded values, and report vulnerabilities through the private channel in SECURITY.md.