Add Torch and Triton kernels for Torch2FlyDSL and Triton2FlyDSL translation

.gitignore

@@ -1,6 +1,7 @@
 .DS_Store
 logs
 __pycache__
+build/
 .vscode
 .idea
 .claude

Makefile

@@ -7,6 +7,7 @@
 SHELL := /bin/bash
 
 .PHONY: help docker-shell docker-check-agents docker-smoke docker-run docker-parallel-run docker-setup-flydsl \
+        check-docker-runner check-evaluator \
         sync-perf-helpers check-perf-helpers materialize-perf-workspace \
         materialize-perf-task cleanup-works install-cursor-agent vllm
 
@@ -21,6 +22,8 @@ help:
 	@echo "make docker-parallel-run CONFIG=config.yaml GPU_IDS=0,1 - Run benchmark across one worker container per GPU"
 	@echo "                         Images: gfx942->mi30x, gfx950->mi35x; override with AKA_DOCKER_IMAGE=..."
 	@echo "make docker-setup-flydsl - Install FlyDSL into the container (needed for flydsl2flydsl tasks)"
+	@echo "make check-docker-runner - Check Docker runner syntax and runtime-specific arguments"
+	@echo "make check-evaluator     - Run centralized evaluator unit tests"
 	@echo ""
 	@echo "Maintenance:"
 	@echo "make sync-perf-helpers   - Refresh committed perf-helper stubs in task sources"
@@ -59,6 +62,12 @@ docker-parallel-run:
 docker-setup-flydsl:
 	@$(DOCKER_RUNNER) setup-flydsl
 
+check-docker-runner:
+	@bash tests/test_docker_benchmark.sh
+
+check-evaluator:
+	@python3 -m unittest discover -s tests -p 'test_evaluator_*.py'
+
 # Refresh committed perf-helper stubs/markers in task sources. Runtime workspaces
 # are materialized from src/tools/perf/ by setup_workspace().
 sync-perf-helpers:

README.md

@@ -99,7 +99,7 @@ runs once after all workers finish.
 ### Prerequisites
 
 - Docker
-- The SGLang Docker image for your GPU arch (`gfx942` uses `lmsysorg/sglang:v0.5.12-rocm720-mi30x`; `gfx950` uses `lmsysorg/sglang:v0.5.12-rocm720-mi35x`)
+- The SGLang Docker image for your GPU arch (`gfx942` uses `lmsysorg/sglang:v0.5.12-rocm720-mi30x`; `gfx950` uses `lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260705`)
 - Git
 - Host-installed agent CLIs for the agents you plan to evaluate
 
@@ -325,7 +325,7 @@ correctness_command:
 performance_command:
   - python3 scripts/task_runner.py --mode performance
 
-task_type: hip2hip   # one of: hip2hip, cuda2hip, triton2triton, torch2hip, instruction2triton, repository, flydsl2flydsl
+task_type: hip2hip   # one of: hip2hip, cuda2hip, triton2triton, triton2flydsl, torch2hip, torch2flydsl, instruction2triton, repository, flydsl2flydsl
 
 prompt:
   source_code: null

agents/task_validator/validation_prompt.py

@@ -159,7 +159,7 @@ def build_validation_prompt(task_config_dir: str, workspace: str, eval_config: d
 - `target_kernel_functions` (list of strings)
 - `compile_command` (list of strings)
 - `correctness_command` (list of strings)
-- `task_type` (string, one of: hip2hip, cuda2hip, triton2triton, torch2hip, instruction2triton, rocprim, flydsl2flydsl, repository)
+- `task_type` (string, one of: hip2hip, cuda2hip, triton2triton, triton2flydsl, torch2hip, torch2flydsl, instruction2triton, rocprim, flydsl2flydsl, repository)
 Also check that optional fields (`performance_command`, `prompt`) are well-formed if present.
 
 **IMPORTANT — `task_type: repository` schema differs.** Repository tasks clone a full upstream
@@ -220,7 +220,27 @@ def build_validation_prompt(task_config_dir: str, workspace: str, eval_config: d
 Capture stdout, stderr, and exit code.
 Check if `build/correctness_report.json` is generated.
 If exit code is non-zero but `eval_result.yaml` clearly records `correctness: true`, treat correctness as PASS and explain the inconsistency.
-Status: PASS if correctness evidence is successful (exit code 0 OR correctness_report status ok OR eval_result correctness=true), FAIL otherwise, TIMEOUT if exceeded {correctness_timeout}s, SKIP if compilation failed OR was skipped (e.g. empty generation-placeholder kernel).
+
+**IMPORTANT — `torch2flydsl` starter-stub contract (task-package validation only).** A shipped
+`torch2flydsl` task may intentionally define a declared top-level target whose body contains only an
+optional docstring / `pass` and a direct, unconditional `raise NotImplementedError(...)`. This is a
+non-empty generation starter, not an optimized implementation. If the correctness harness invokes that
+target, catches that specific `NotImplementedError`, clearly reports the target as unimplemented, and
+still successfully validates the PyTorch/model reference against an independent oracle such as AITER,
+set correctness to `SKIP` and explain that the independent oracle passed. This allowance applies only to
+the as-shipped task validator:
+
+- Do not treat a conditional `NotImplementedError` inside an otherwise implemented target as a starter stub.
+- Missing target symbols/files, `AttributeError`, `ImportError`, `RuntimeError`, and every exception other
+  than the explicit starter `NotImplementedError` are real failures and MUST NOT be converted to `SKIP`,
+  even if a harness prints “SKIP” or exits zero.
+- If the independent reference/oracle check fails, correctness is `FAIL`, not `SKIP`.
+- Performance remains `SKIP` for an accepted starter because there is no implemented target to score.
+- The centralized optimization evaluator performs a static guard before correctness and rejects an agent
+  submission that leaves any declared target as this starter stub. A validator `SKIP` never makes an
+  unimplemented optimization submission eligible for scoring.
+
+Status: PASS if correctness evidence is successful (exit code 0 OR correctness_report status ok OR eval_result correctness=true), FAIL otherwise, TIMEOUT if exceeded {correctness_timeout}s, SKIP if compilation failed/was skipped (e.g. empty generation-placeholder kernel) OR the exact `torch2flydsl` starter-stub contract above is satisfied.
 
 ### Check 6: Performance
 Run the performance command(s) from the workspace directory (if any):
@@ -264,6 +284,10 @@ def build_validation_prompt(task_config_dir: str, workspace: str, eval_config: d
 - Does it use reasonable tolerances (atol, rtol)?
 - Could it trivially pass regardless of kernel output (e.g., always returns 0, no actual comparison)?
 - Does it test with sufficient input shapes/sizes?
+For a `torch2flydsl` starter, catching only the target's explicit `NotImplementedError` while independently
+checking the reference/oracle is acceptable. A broad exception handler or fallback that lets missing
+symbols, import errors, runtime errors, or incorrect implemented targets pass is trivially passing: mark
+this check `FAIL` and set `is_trivially_passing: true`.
 Status: PASS if implementation appears sound, WARN if questionable but functional, FAIL if trivially passing.
 Set `is_trivially_passing: true` if the check would pass even with garbage output.
 
@@ -324,7 +348,8 @@ def build_validation_prompt(task_config_dir: str, workspace: str, eval_config: d
 ```
 
 ### Rules for overall_status:
-- **PASS**: ALL checks passed (no FAIL, no WARN)
+- **PASS**: All applicable checks passed (no FAIL, no WARN); a contract-allowed `SKIP` such as an
+  intentional generation placeholder or confirmed `torch2flydsl` starter does not prevent overall PASS
 - **WARN**: No FAIL checks, but at least one WARN
 - **FAIL**: At least one check has status FAIL
 

docs/how-to/add-task.md

@@ -23,12 +23,15 @@ The `task_type` field declares what kind of optimization the task represents.
 | `triton2triton` | Optimize an existing Triton kernel |
 | `instruction2triton` | Write a Triton kernel from an instruction/spec |
 | `torch2hip` | Replace a PyTorch reference with a HIP kernel |
+| `torch2flydsl` | Replace a PyTorch reference with a FlyDSL kernel |
+| `triton2flydsl` | Translate a Triton kernel to FlyDSL |
 | `flydsl2flydsl` | Optimize a FlyDSL kernel (requires FlyDSL) |
 | `repository` | Repository-level task |
 
 The repository ships task suites including `hip2hip` (gpumode and others),
-`triton2triton` (vLLM and ROCmBench), `torch2hip`, `instruction2triton`, and
-`flydsl2flydsl`, plus repository-level tasks under `tasks/repository/`.
+`triton2triton` (vLLM and ROCmBench), `torch2hip`, `instruction2triton`,
+`torch2flydsl`, `triton2flydsl`, and `flydsl2flydsl`, plus repository-level
+tasks under `tasks/repository/`.
 
 ## Directory layout
 
@@ -67,8 +70,9 @@ compile_command:
 correctness_command:
   - python3 scripts/task_runner.py --mode correctness
 
-# One of: hip2hip, cuda2hip, triton2triton, instruction2triton,
-#         torch2hip, flydsl2flydsl, repository
+# One of: hip2hip, cuda2hip, triton2triton, triton2flydsl,
+#         instruction2triton, torch2hip, torch2flydsl,
+#         flydsl2flydsl, repository
 task_type: hip2hip
 ```
 

docs/install/install.md

@@ -21,7 +21,7 @@ The following prerequisites are required before running AgentKernelArena.
   `/dev/dri`, and `/dev/mem` when present.
 - **SGLang benchmark image** — `gfx942` uses
   `lmsysorg/sglang:v0.5.12-rocm720-mi30x`; `gfx950` uses
-  `lmsysorg/sglang:v0.5.12-rocm720-mi35x`. The runner selects from
+  `lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260705`. The runner selects from
   `target_gpu_model` for benchmark runs and from the visible host GPU for shell
   and smoke commands.
 - **Git**
@@ -84,9 +84,10 @@ npm install -g @anthropic-ai/claude-code
 
 ## FlyDSL tasks (optional)
 
-`flydsl2flydsl` tasks need the `flydsl` package inside the container. Most images
-already ship it (`make docker-smoke` prints `flydsl=ok <version>`). If yours does
-not, install it once into the container's persistent pip user-base:
+`flydsl2flydsl`, `torch2flydsl`, and `triton2flydsl` tasks need the `flydsl`
+package inside the container. Most images already ship it (`make docker-smoke`
+prints `flydsl=ok <version>`). If yours does not, install it once into the
+container's persistent pip user-base:
 
 ```bash
 make docker-setup-flydsl

docs/reference/api-reference.md

@@ -101,15 +101,16 @@ Each task is defined by a `config.yaml` in its directory. Command fields are
 *lists*.
 
 For isolated-kernel tasks (`hip2hip`, `cuda2hip`, `triton2triton`,
-`instruction2triton`, `torch2hip`, and `flydsl2flydsl`):
+`triton2flydsl`, `instruction2triton`, `torch2hip`, `torch2flydsl`, and
+`flydsl2flydsl`):
 
 | Field | Required | Description |
 | --- | --- | --- |
 | `source_file_path` | Yes | Source files containing the kernel, relative to the task root |
 | `target_kernel_functions` | Yes | Kernel function names that must be defined in the source |
 | `compile_command` | Yes | Command(s) to compile or build-check |
 | `correctness_command` | Yes | Command(s) to validate correctness |
-| `task_type` | Yes | One of `hip2hip`, `cuda2hip`, `triton2triton`, `instruction2triton`, `torch2hip`, or `flydsl2flydsl` |
+| `task_type` | Yes | One of `hip2hip`, `cuda2hip`, `triton2triton`, `triton2flydsl`, `instruction2triton`, `torch2hip`, `torch2flydsl`, or `flydsl2flydsl` |
 | `performance_command` | No | Command(s) to measure performance |
 | `task_result_template` | No | Override the result template (`null` = default) |
 | `prompt.source_code` | No | Override the prompt's source-code section |

docs/reference/compatibility-matrix.md

@@ -23,12 +23,13 @@ The following software versions are required or verified.
 | Component | Version | Notes |
 | --- | --- | --- |
 | Docker | Current stable release | Required; serial evaluations run through `make docker-run`; multi-GPU evaluations run through `make docker-parallel-run`. |
-| SGLang benchmark image | `lmsysorg/sglang:v0.5.12-rocm720-mi30x` for `gfx942`; `lmsysorg/sglang:v0.5.12-rocm720-mi35x` for `gfx950` | Override with `AKA_DOCKER_IMAGE`, `AKA_DOCKER_IMAGE_GFX942`, or `AKA_DOCKER_IMAGE_GFX950`. |
+| SGLang benchmark image | `lmsysorg/sglang:v0.5.12-rocm720-mi30x` for `gfx942`; `lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260705` for `gfx950` | The verified `gfx950` digest is `sha256:b435b508b5aa696abb25c909341ce73e41574c4271cf716bed72418dcea86b78`. Override with `AKA_DOCKER_IMAGE`, `AKA_DOCKER_IMAGE_GFX942`, or `AKA_DOCKER_IMAGE_GFX950`. |
 | ROCm | Bundled in the selected SGLang image | The default images are ROCm 7.2 based. |
 | Python | Provided by the image (for example, 3.10) | Bundled in the SGLang image. |
 | PyTorch | ROCm build bundled in the image | Provided by the SGLang Docker image. |
 | Triton | Bundled with the image's ROCm PyTorch | Required for Triton task categories. |
-| FlyDSL | Provided by the image (or `make docker-setup-flydsl`) | Required for `flydsl2flydsl` tasks. |
+| AITER | `0.1.17.dev110+g9127c94a1` in the verified `gfx950` image | Required by AITER-backed task oracles and kernels. |
+| FlyDSL | `0.2.2` in the verified `gfx950` image (or `make docker-setup-flydsl` when absent) | Required for `flydsl2flydsl`, `torch2flydsl`, and `triton2flydsl` tasks. |
 | hipcc | Matches image ROCm | Required for HIP tasks. |
 | rocprof-compute | Matches image ROCm | Required for HIP performance profiling. |
 

docs/what-is-aka.rst

@@ -39,8 +39,9 @@ AgentKernelArena includes the following key features.
 * **Docker-first runtime**: Benchmark runs execute inside pinned ROCm/SGLang
   Docker images selected from the target GPU architecture.
 * **Task categories**: HIP (``hip2hip``), CUDA-to-HIP (``cuda2hip``), Triton
-  (``triton2triton``, ``instruction2triton``), Torch-to-HIP (``torch2hip``), and
-  FlyDSL (``flydsl2flydsl``), plus repository-level tasks.
+  (``triton2triton``, ``instruction2triton``), Torch-to-HIP (``torch2hip``),
+  Torch/Triton-to-FlyDSL (``torch2flydsl``, ``triton2flydsl``), and FlyDSL
+  (``flydsl2flydsl``), plus repository-level tasks.
 * **Objective metrics**: Automated compilation, correctness, and real GPU
   performance speedups.
 * **Benchmark methodology metadata**: Timing method metadata is recorded for

src/evaluator.py

@@ -13,7 +13,7 @@
 from pathlib import Path
 from typing import Dict, Any, Optional, List, Tuple
 
-from .evaluator_utils import run_command
+from .evaluator_utils import inspect_target_definitions, run_command
 from .performance import measure_performance, measure_baseline
 from .testcases import TestCaseResult, save_performance_results, calculate_average_speedup, collect_benchmark_methods
 
@@ -164,6 +164,31 @@ def evaluate_kernel(
 
     # 2. Correctness check
     log.info("Step 2: Checking correctness...")
+    # The as-shipped torch2flydsl starter is allowed during baseline and task
+    # package validation, both of which bypass this optimized-kernel pipeline.
+    # Once an optimization agent has run, however, its declared targets must no
+    # longer be unconditional NotImplementedError stubs; otherwise a harness
+    # could silently time and validate its reference fallback.
+    if task_config.get("task_type") == "torch2flydsl":
+        missing_names, stub_names = inspect_target_definitions(workspace, task_config)
+        target_errors = []
+        if missing_names:
+            target_errors.append(
+                "missing declared top-level target definition(s): "
+                + ", ".join(missing_names)
+            )
+        if stub_names:
+            target_errors.append(
+                "unimplemented target stub(s): " + ", ".join(stub_names)
+            )
+        if target_errors:
+            corr_error = "Invalid torch2flydsl optimization submission: " + "; ".join(
+                target_errors
+            )
+            results['correctness_error_message'] = corr_error
+            log.warning(corr_error)
+            return results
+
     pass_correctness, corr_error = evaluate_correctness(workspace, task_config, logger)
     results['pass_correctness'] = pass_correctness
     results['correctness_error_message'] = corr_error

src/evaluator_utils.py

@@ -2,18 +2,123 @@
 """
 Utilities for evaluator: command execution and file I/O.
 """
+import ast
 import os
 import shutil
 import subprocess
 import logging
 import yaml
 import shlex
 from pathlib import Path
-from typing import Tuple, Optional, List
+from typing import Any, Dict, Tuple, Optional, List
 from .testcases import TestCaseResult
 from .runtime_env import PYTHON_ENV_VAR, build_subprocess_env
 
 
+def _string_list(value: Any) -> List[str]:
+    if isinstance(value, str):
+        return [value]
+    if isinstance(value, (list, tuple)):
+        return [item for item in value if isinstance(item, str)]
+    return []
+
+
+def _is_docstring_statement(statement: ast.stmt) -> bool:
+    return (
+        isinstance(statement, ast.Expr)
+        and isinstance(statement.value, ast.Constant)
+        and isinstance(statement.value.value, str)
+    )
+
+
+def _is_not_implemented_exception(expression: Optional[ast.expr]) -> bool:
+    if isinstance(expression, ast.Call):
+        expression = expression.func
+    if isinstance(expression, ast.Name):
+        return expression.id == "NotImplementedError"
+    return (
+        isinstance(expression, ast.Attribute)
+        and expression.attr == "NotImplementedError"
+        and isinstance(expression.value, ast.Name)
+        and expression.value.id == "builtins"
+    )
+
+
+def _is_unimplemented_target_stub(function: ast.AST) -> bool:
+    """Match only a no-op body followed by a direct NotImplementedError raise."""
+    body = list(getattr(function, "body", []))
+    meaningful_statements: List[ast.stmt] = []
+    for index, statement in enumerate(body):
+        if index == 0 and _is_docstring_statement(statement):
+            continue
+        if isinstance(statement, ast.Pass):
+            continue
+        meaningful_statements.append(statement)
+
+    return (
+        len(meaningful_statements) == 1
+        and isinstance(meaningful_statements[0], ast.Raise)
+        and _is_not_implemented_exception(meaningful_statements[0].exc)
+    )
+
+
+def inspect_target_definitions(
+    workspace: Path,
+    task_config: Dict[str, Any],
+) -> Tuple[List[str], List[str]]:
+    """Return missing and unimplemented declared top-level Python targets.
+
+    This intentionally does not walk into function bodies.  An implemented
+    target may use a conditional ``NotImplementedError`` for an unsupported
+    shape without being classified as an unimplemented submission.  The task
+    contract requires a Python ``def`` for each target; assignment aliases are
+    not treated as target definitions.
+    """
+    target_names = set(_string_list(task_config.get("target_kernel_functions")))
+    if not target_names:
+        return [], []
+
+    found_names = set()
+    stub_names = set()
+    for configured_path in _string_list(task_config.get("source_file_path")):
+        source_path = Path(configured_path)
+        if not source_path.is_absolute():
+            source_path = Path(workspace) / source_path
+        if not source_path.is_file() or source_path.suffix != ".py":
+            continue
+        try:
+            module = ast.parse(source_path.read_text(encoding="utf-8"), filename=str(source_path))
+        except (OSError, UnicodeError, SyntaxError):
+            # Compilation reports missing, unreadable, and invalid source files;
+            # this guard is deliberately limited to recognized starter stubs.
+            continue
+
+        # The last top-level definition is the one bound by the module at run
+        # time.  Nested methods/functions are intentionally excluded.
+        definitions = {}
+        for statement in module.body:
+            if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef)):
+                definitions[statement.name] = statement
+        for target_name in target_names:
+            function = definitions.get(target_name)
+            if function is None:
+                continue
+            found_names.add(target_name)
+            if _is_unimplemented_target_stub(function):
+                stub_names.add(target_name)
+
+    return sorted(target_names - found_names), sorted(stub_names)
+
+
+def find_unimplemented_target_stubs(
+    workspace: Path,
+    task_config: Dict[str, Any],
+) -> List[str]:
+    """Compatibility helper returning only declared starter stubs."""
+    _, stub_names = inspect_target_definitions(workspace, task_config)
+    return stub_names
+
+
 def _replace_leading_token(command: str, token: str, replacement: str) -> str:
     leading_len = len(command) - len(command.lstrip())
     leading = command[:leading_len]