Skip to content

Support external eval fixture files#13

Merged
morganlinton merged 2 commits into
GetSmallAI:mainfrom
HarperZ9:telos-external-fixture-loading
Jun 29, 2026
Merged

Support external eval fixture files#13
morganlinton merged 2 commits into
GetSmallAI:mainfrom
HarperZ9:telos-external-fixture-loading

Conversation

@HarperZ9

Copy link
Copy Markdown
Contributor

Summary

Adds the first narrow slice for external agent eval fixtures:

  • small-harness --eval <path-to-fixture.json> now loads a data-only fixture file.
  • Built-in fixture IDs continue to resolve unchanged.
  • External fixture workspaces resolve relative to the fixture file.
  • Workspace and fileContains.path refs reject absolute paths and parent traversal.
  • README and Quickstart document the external fixture shape.

This is intentionally smaller than full fixture-pack provenance: no executable hooks, no pack manifest loader, no artifact indexing, and no receipt format changes.

Closes part of #12.

Verification

Passed locally on Windows:

cargo test agent_eval
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
changed-file secret scan
stale-language scan

Full cargo test was also run. It passed 424 tests and failed 3 tests that appear unrelated to this change and Windows/environment-specific:

tools::apply_patch_tool::tests::applies_valid_patch: expected LF, got CRLF
shipcheck::tests::collects_real_git_snapshot: path formatting mismatch C:/... vs \\?\C:\...
tools::grep::tests::finds_match_with_correct_fields: expected 1 match, got 0

@HarperZ9

Copy link
Copy Markdown
Contributor Author

Local verification on the PR branch is clean:

  • cargo fmt --all -- --check
  • cargo test agent_eval -> 9 passed, 0 failed
  • cargo clippy --all-targets -- -D warnings

That covers the external fixture loader slice, traversal rejection cases, and the repo's clippy warning gate.

@bennewell35 bennewell35 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the right narrow first slice for external eval fixtures. I found one path-safety gap before merge:

copy_dir_all bounds the workspace directory, but it does not reject symlinked child entries. DirEntry::file_type() does not follow symlinks, so a symlinked file is not treated as a directory and falls into the fs::copy(entry.path(), target) branch. Rust fs::copy accepts a symlink to a regular file as a source, so a fixture workspace can contain workspace/basic/leak.txt -> /tmp/secret.txt; validation passes because both workspace and fileContains.path are syntactically relative, but the copied eval workspace now contains bytes from outside the fixture root.

Smallest fix: reject symlink entries while validating/copying fixture workspaces, or canonicalize every copied child/read target against the fixture root before allowing it. Please add a regression test with a symlinked file under the external workspace. That keeps this PR aligned with the README promise that external workspaces cannot escape the fixture root.

@HarperZ9

Copy link
Copy Markdown
Contributor Author

Addressed the symlink workspace-copy gap in f6097cd.

What changed:

  • copy_dir_all now rejects symlinked fixture workspace entries before fs::copy can dereference them.
  • It only copies regular files and directories; other entry types fail closed.
  • Added a Unix-gated regression test for a workspace child symlink pointing at an outside file.

Local verification on Windows:

  • cargo fmt --all -- --check
  • cargo test agent_eval -> 9 passed, 0 failed (#[cfg(unix)] symlink regression will run on Unix CI)
  • cargo clippy --all-targets -- -D warnings
  • git diff --check

@morganlinton morganlinton merged commit f0387ea into GetSmallAI:main Jun 29, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants