Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 53 additions & 58 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ Automated validation tool for testing DevWorkspace instances on OpenShift cluste
### Running Tests

```bash
# Basic validation (uses small numbers of entries - 3 images, 5 devfiles)
# Basic validation (uses small lists - images/images.txt and devfiles/devfiles.txt)
./dw-auto-validate.sh

# Verbose mode - shows detailed output
./dw-auto-validate.sh -v

# Full test matrix (uses all entries from -full.txt files - all images, all devfiles. Takes significant time to complete!)
# Full test matrix (uses *-full.txt files - all images, all devfiles. Takes significant time to complete!)
./dw-auto-validate.sh -f

# Debug mode - verbose + runs only first test + no cleanup
Expand Down Expand Up @@ -58,7 +58,7 @@ Automated validation tool for testing DevWorkspace instances on OpenShift cluste
- Iterates through devfiles × images matrix
- For each combination: creates DevWorkspace, waits for Running state, validates, records results
5. **Cleanup**: Deletes DevWorkspace and temporary files (skipped in debug mode)
6. **Summary Report**: Shows test counts, success/failure, elapsed time, and lists failed tests
6. **Summary Report**: Shows test counts, success/failure/excluded, elapsed time, and lists failed tests

### Command-Line Flags

Expand All @@ -81,38 +81,35 @@ Each scenario in `settings/settings-<SCENARIO>.env` exports:
- `PROJECT_URL`: Git repository URL (must include surrounding double quotes)
- `EDITOR_DEFINITION`: URL to the editor definition YAML
- `EDITOR_COMPONENT_NAME`: Component name in the editor definition that contains the editor image (used by `-i`/`-p` to replace the correct image)
- `validate_devworkspace()`: Function that validates the running DevWorkspace
- `EXCLUDED_IMAGE_PATTERNS`: Array of Posix Extended Regular Expressions for images excluded from failure counts
- `LANDING_PAGE_PORT`: Port to curl inside the pod to validate the editor is running

#### Scenario Validation Methods
#### Scenario Validation

**sshd** (settings-sshd.env):
- Timeout: 60s
- Checks `/tmp/sshd.log` for "Server listening on"
- Verifies SSHD server started successfully
All scenarios use the same validation method: curl `localhost:${LANDING_PAGE_PORT}` inside the pod and check for HTTP 200.

**jetbrains** (settings-jetbrains.env):
- Timeout: 120s
- Port-forwards to 3400, curls `127.0.0.1:3400`
- Validates HTTP 200 response from JetBrains landing page
- On failure, outputs `/idea-server/std.out` for debugging

**vscode** (settings-vscode.env):
- Timeout: 60s
- Checks `/checode/entrypoint-logs.txt` for "Extension host agent listening on 3100"
- Verifies VSCode extension host is listening
| Scenario | Timeout | Landing Page Port | Editor Component |
|----------|---------|-------------------|------------------|
| sshd | 60s | 3400 | che-code-sshd-page |
| jetbrains | 120s | 3400 | editor-injector |
| vscode | 90s | 3100 | che-code-injector |

### DevWorkspace Generation

Uses `devworkspace-template.yaml` as base, performs sed substitutions in two stages:
Uses `devworkspace-template.yaml` as base. The template uses ephemeral storage (`controller.devfile.io/storage-type: ephemeral`) to avoid PVC provisioning overhead during tests.

Substitutions are performed in two stages:

**Stage 1** - Metadata and devfile injection:
**Stage 1** - Metadata, devfile, and projects injection:
```bash
cat devworkspace-template.yaml | sed \
-e "/DEVFILE/r ${TMP_DEVFILE}" \ # Inject devfile content
-e '/DEVFILE/ d' \ # Remove DEVFILE placeholder
-e "/DEVFILE/r ${TMP_DEVFILE}" \ # Inject devfile content
-e '/DEVFILE/ d' \ # Remove DEVFILE placeholder
-e "/PROJECTS/r ${TMP_PROJECTS}" \ # Inject projects block
-e '/PROJECTS/ d' \ # Remove PROJECTS placeholder
-e "s|DEVWORKSPACE_NAME|...|" \
-e "s|DEVWORKSPACE_NS|...|" \
-e "s|EDITOR_DEFINITION|...|" \
-e "${EDITOR_SED_EXPR}" \ # Editor definition (uri or kubernetes ref)
-e "s|PROJECT_URL|...|"
```

Expand All @@ -123,6 +120,18 @@ eval "sed \"s|image: .*|image: ${image}|\" > ${TMP_DEVWORKSPACE}"

The two-stage approach ensures devfile content is injected before image replacement.

**Projects handling**: If the devfile contains `starterProjects`, those are extracted and converted into a `projects:` block. Otherwise, the scenario's `PROJECT_URL` is used as a fallback sample project.

**Editor contribution**: When using `-p` or `-i` (override image), the editor contribution switches from `uri:` to `kubernetes: name:` referencing the applied DevWorkspaceTemplate.

### DevWorkspace Lifecycle Management

Between tests, the script handles the workspace depending on its current state:

- **Running / Starting**: Gracefully stops by patching `spec.started: false` and waiting up to `TIMEOUT/4` seconds for `Stopped` state.
- **Failed**: Force-deletes the workspace (`oc delete dw`). A Failed workspace with CrashLoopBackOff containers can take too long to stop gracefully, which would cascade into subsequent test failures. The next `oc apply` recreates it cleanly.
- **Stopped / not found**: Proceeds directly to `oc apply`.

### Logging and Output Control

- `log()`: Outputs only when `VERBOSE=1` (set by `-v` or `-d` flags)
Expand All @@ -140,51 +149,38 @@ Tracks test execution time using bash's `$SECONDS` variable:

```
settings/
settings-sshd.env # SSHD scenario: timeout=60s, validates /tmp/sshd.log
settings-jetbrains.env # JetBrains scenario: timeout=120s, validates port 3400
settings-vscode.env # VSCode scenario: timeout=60s, validates /checode/entrypoint-logs.txt
settings-sshd.env # SSHD scenario: timeout=60s, port 3400
settings-jetbrains.env # JetBrains scenario: timeout=120s, port 3400
settings-vscode.env # VSCode scenario: timeout=90s, port 3100

images/
images.txt # Quick test list (3 UDI images: ubi8, ubi9, ubi10)
images-full.txt # Complete test matrix (UDI + base-developer-image variants)
images.txt # Default test list (3 UDI images: ubi8, ubi9, ubi10)
images-full.txt # Complete test matrix (227 images including UDI, base-developer-image, and UBI variants)

devfiles/
devfiles.txt # Quick test list (nodejs, go, python, php-laravel and java-quarkus devfile)
devfiles.txt # Default test list (nodejs, go, php-laravel, python, java-quarkus)
devfiles-full.txt # Complete devfile list (32 devfiles from devfile registry)

samples/
samples.txt # Sample project URLs (currently unused)
samples-full.txt # Extended sample project list (currently unused)

devworkspace-template.yaml # Base template with placeholders
devworkspace-template.yaml # Base template with placeholders (ephemeral storage)
dw-auto-validate.sh # Main validation orchestrator
verify_images.sh # Skopeo-based image accessibility checker
```

## Implementation Details

### Validation Function Pattern

All `validate_devworkspace()` functions follow this pattern:

```bash
validate_devworkspace() {
devfile_url=$1 # Receives devfile URL as first argument

# Resolve pod and container via shared helper
resolve_devworkspace_pod || return 1
### Validation Function

# Scenario-specific validation logic here
# Return 0 for pass, 1 for fail
}
```
A single `validate_devworkspace()` function in the main script handles all scenarios:

The shared `resolve_devworkspace_pod()` function sets `podName` and `mainContainerName` globals.
1. Calls `resolve_devworkspace_pod()` to set `podName` and `mainContainerName` globals
2. Curls `localhost:${LANDING_PAGE_PORT}` inside the pod container
3. Returns 0 if HTTP 200, 1 otherwise

**Critical details**:
- Has access to `${DEVWORKSPACE_NS}`, `${DEVWORKSPACE_NAME}`, `log()`
- Must return 0 for success, 1 for failure
- Should use `&>/dev/null` on oc exec commands meant only for exit code checking
`resolve_devworkspace_pod()` finds the pod by DevWorkspace label and selects the main container from pod status, filtering out containers whose name starts with `che-`.

### Variable Quoting Requirements

Expand All @@ -203,7 +199,7 @@ git:
```bash
state=""
count=0
while [ "${state}" != "Running" ] && [ ${count} -lt ${TIMEOUT} ]; do
while [ "${state}" != "Running" ] && [ "${state}" != "Failed" ] && [ ${count} -lt ${TIMEOUT} ]; do
state=$(oc get dw ${DEVWORKSPACE_NAME} -o 'jsonpath={.status.phase}')
sleep 1s
count=$((count+1))
Expand All @@ -212,18 +208,17 @@ done

**Finding pod by DevWorkspace label**:
```bash
podNameAndDWName=$(oc get pods -o 'jsonpath={range .items[*]}{.metadata.name}{","}{.metadata.labels.controller\.devfile\.io/devworkspace_name}{end}')
podName=$(echo ${podNameAndDWName} | grep ${DEVWORKSPACE_NAME} | cut -d, -f1)
podNameAndDWName=$(oc get pods -o 'jsonpath={range .items[*]}{.metadata.name}{","}{.metadata.labels.controller\.devfile\.io/devworkspace_name}{"\n"}{end}')
podName=$(echo "${podNameAndDWName}" | grep ${DEVWORKSPACE_NAME} | cut -d, -f1)
```

**Getting main container name**:
**Getting main container name** (from pod status, excluding `che-*` containers):
```bash
mainContainerName=$(oc get devworkspace ${DEVWORKSPACE_NAME} -o json | jq -r '[.spec.template.components[] | select(.container) | .name] | first')
mainContainerName=$(oc get pod "${podName}" -o json | jq -r '[.status.containerStatuses[] | select(.state.running and (.name | test("^che-") | not))] | first | .name // empty')
```

### Adding a New Scenario

1. Create `settings/settings-<name>.env`
2. Export required variables: `TIMEOUT`, `DEVWORKSPACE_NAME`, `PROJECT_URL`, `EDITOR_DEFINITION`
3. Implement `validate_devworkspace()` function that returns 0/1
4. Update scenario selection in dw-auto-validate.sh (add option, update prompts)
2. Export required variables: `TIMEOUT`, `DEVWORKSPACE_NAME`, `PROJECT_URL`, `EDITOR_DEFINITION`, `EDITOR_COMPONENT_NAME`, `EXCLUDED_IMAGE_PATTERNS`, `LANDING_PAGE_PORT`
3. Update scenario selection in dw-auto-validate.sh (add option, update prompts and `-s` validation)
12 changes: 6 additions & 6 deletions devfiles/devfiles-full.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
https://registry.devfile.io/devfiles/dotnet80
https://registry.devfile.io/devfiles/go
https://registry.devfile.io/devfiles/code-with-quarkus
https://registry.devfile.io/devfiles/java-maven
https://registry.devfile.io/devfiles/java-openliberty
https://registry.devfile.io/devfiles/java-openliberty-gradle
Expand All @@ -20,13 +21,12 @@ https://registry.devfile.io/devfiles/nodejs-nuxtjs
https://registry.devfile.io/devfiles/nodejs-react
https://registry.devfile.io/devfiles/nodejs-svelte
https://registry.devfile.io/devfiles/nodejs-vue
https://registry.devfile.io/devfiles/ollama
https://registry.devfile.io/devfiles/php-laravel
https://registry.devfile.io/devfiles/python
https://registry.devfile.io/devfiles/python-django
https://registry.devfile.io/devfiles/udi
https://registry.devfile.io/devfiles/nodejs-basic
https://registry.devfile.io/devfiles/code-with-quarkus
https://registry.devfile.io/devfiles/java-springboot-basic
https://registry.devfile.io/devfiles/python-basic
https://registry.devfile.io/devfiles/go-basic
https://registry.devfile.io/devfiles/hermes
https://registry.devfile.io/devfiles/ollama
https://registry.devfile.io/devfiles/openclaw
https://registry.devfile.io/devfiles/picoclaw
https://registry.devfile.io/devfiles/zeroclaw
1 change: 1 addition & 0 deletions devfiles/devfiles.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ https://registry.devfile.io/devfiles/nodejs
https://registry.devfile.io/devfiles/go
https://registry.devfile.io/devfiles/php-laravel
https://registry.devfile.io/devfiles/python
https://registry.devfile.io/devfiles/java-quarkus
8 changes: 3 additions & 5 deletions devworkspace-template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,10 @@ spec:
started: true
routingClass: che # thanks claude
template:
attributes:
controller.devfile.io/storage-type: ephemeral
DEVFILE
projects:
- name: project-sample
git:
remotes:
origin: PROJECT_URL
PROJECTS

contributions:
- name: editor-definition-component
Expand Down
Loading