Skip to content

Commit 2a309f2

Browse files
committed
feat(skills): remotion-to-hyperframes eval harness (2/7)
Adds the deterministic eval primitives the skill calls into: scripts/render_diff.sh SSIM diff between two MP4s, JSON summary, configurable threshold scripts/frame_strip.sh side-by-side comparison strip for visual debugging scripts/lint_source.py pre-translation lint over Remotion source — blocks/warnings/infos The harness is decoupled from the render pipeline: it accepts paths to already-rendered MP4s. The skill orchestrator (PR 7) drives both renders and feeds the outputs in. This keeps the harness usable in CI, in sandboxes, and on any machine that has ffmpeg without needing the full Remotion + HyperFrames toolchain. Lint catches the patterns from the skill's out-of-scope list: - useState / useReducer (state-machine driven animation) - useEffect with deps (side effects) - async calculateMetadata (Promise-returning composition metadata) - @remotion/lambda imports - third-party React UI libraries (MUI, Chakra, Mantine, antd, shadcn, Radix, NextUI) - delayRender / useCallback / useMemo (warnings) - staticFile / interpolateColors (info — translatable but flagged) Smoke test (scripts/tests/smoke.sh) exercises all three scripts against synthetic inputs: identical ffmpeg testsrc videos pass at threshold 0.99, different ffmpeg testsrc videos fail at 0.99, frame_strip produces a strip.png, lint produces 0 blockers on a clean fixture and >=3 blockers on a fixture that uses useState + useEffect + MUI + async metadata. Validated locally: smoke.sh exits 0.
1 parent 7ed3d04 commit 2a309f2

6 files changed

Lines changed: 647 additions & 0 deletions

File tree

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#!/usr/bin/env bash
2+
# frame_strip.sh — produce a side-by-side comparison strip from two videos.
3+
#
4+
# Used to debug failing render_diff.sh runs visually: pick a sample timestamp
5+
# range, extract frames from both videos, lay them out as a grid for review.
6+
#
7+
# Usage:
8+
# frame_strip.sh <baseline.mp4> <translated.mp4> [output-dir] [samples]
9+
#
10+
# Defaults: output-dir=./strip-out, samples=8 (evenly spaced across duration).
11+
# Output:
12+
# strip.png — single PNG with `samples` rows, each row is
13+
# (baseline frame | translated frame) at one timestamp
14+
# timestamps.txt — the timestamps sampled
15+
16+
set -euo pipefail
17+
18+
if [[ $# -lt 2 || $# -gt 4 ]]; then
19+
echo "usage: $0 <baseline.mp4> <translated.mp4> [output-dir] [samples]" >&2
20+
exit 2
21+
fi
22+
23+
BASELINE="$1"
24+
TRANSLATED="$2"
25+
OUTDIR="${3:-./strip-out}"
26+
SAMPLES="${4:-8}"
27+
28+
if ! command -v ffmpeg >/dev/null 2>&1 || ! command -v ffprobe >/dev/null 2>&1; then
29+
echo "error: ffmpeg/ffprobe not on PATH" >&2
30+
exit 2
31+
fi
32+
33+
mkdir -p "$OUTDIR"
34+
35+
DURATION=$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$BASELINE")
36+
if [[ -z "$DURATION" ]]; then
37+
echo "error: could not read duration from $BASELINE" >&2
38+
exit 2
39+
fi
40+
41+
# Even-spaced timestamps that avoid both 0.0 (often blank intro) and end-of-video.
42+
python3 - "$DURATION" "$SAMPLES" "$OUTDIR/timestamps.txt" <<'PY'
43+
import sys
44+
duration = float(sys.argv[1])
45+
samples = int(sys.argv[2])
46+
out = sys.argv[3]
47+
# Sample at evenly spaced points starting at 5% into the duration to avoid
48+
# the typical 0-frame transparency / fade-in noise.
49+
start = duration * 0.05
50+
end = duration * 0.95
51+
step = (end - start) / max(samples - 1, 1)
52+
ts = [round(start + i * step, 3) for i in range(samples)]
53+
with open(out, "w") as f:
54+
f.write("\n".join(str(t) for t in ts) + "\n")
55+
PY
56+
57+
# Extract one frame from each video at each timestamp.
58+
i=0
59+
ROW_INPUTS=""
60+
while IFS= read -r ts; do
61+
i=$((i + 1))
62+
ffmpeg -y -hide_banner -loglevel error -ss "$ts" -i "$BASELINE" -frames:v 1 "$OUTDIR/baseline-$i.png"
63+
ffmpeg -y -hide_banner -loglevel error -ss "$ts" -i "$TRANSLATED" -frames:v 1 "$OUTDIR/translated-$i.png"
64+
# hstack each row, then later vstack all rows.
65+
ffmpeg -y -hide_banner -loglevel error \
66+
-i "$OUTDIR/baseline-$i.png" -i "$OUTDIR/translated-$i.png" \
67+
-filter_complex "[0:v][1:v]hstack=inputs=2" \
68+
"$OUTDIR/row-$i.png"
69+
ROW_INPUTS="$ROW_INPUTS -i $OUTDIR/row-$i.png"
70+
done < "$OUTDIR/timestamps.txt"
71+
72+
# vstack all rows.
73+
FILTER=""
74+
COUNT=$(wc -l < "$OUTDIR/timestamps.txt" | tr -d ' ')
75+
for j in $(seq 1 "$COUNT"); do
76+
FILTER="$FILTER[$((j - 1)):v]"
77+
done
78+
FILTER="${FILTER}vstack=inputs=$COUNT"
79+
80+
# shellcheck disable=SC2086
81+
ffmpeg -y -hide_banner -loglevel error \
82+
$ROW_INPUTS \
83+
-filter_complex "$FILTER" \
84+
"$OUTDIR/strip.png"
85+
86+
echo "wrote $OUTDIR/strip.png ($COUNT samples)"
Lines changed: 284 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,284 @@
1+
#!/usr/bin/env python3
2+
"""Lint a Remotion project for patterns that don't translate cleanly to HyperFrames.
3+
4+
The skill should run this *before* attempting a translation. If any blocker
5+
findings come back, the recommendation is to use the runtime interop pattern
6+
from PR #214 instead of producing broken HTML.
7+
8+
Usage:
9+
lint_source.py <path-to-remotion-src> [--json]
10+
11+
Output (default human-readable, --json for machine-readable):
12+
For each .ts/.tsx file, a list of findings with:
13+
- severity: blocker | warning | info
14+
- line, column
15+
- rule id
16+
- message
17+
- recommendation
18+
19+
Blockers (skill should refuse to translate):
20+
- r2hf/use-state React state machine drives animation
21+
- r2hf/use-effect-deps useEffect/useLayoutEffect with non-empty deps (side effects)
22+
- r2hf/use-reducer useReducer drives animation
23+
- r2hf/async-metadata calculateMetadata returns a Promise
24+
- r2hf/third-party-react-ui Imports a React UI library (shadcn, mui, antd, mantine, chakra)
25+
26+
Warnings (translate but flag — drop the construct, keep the rest):
27+
- r2hf/lambda-import @remotion/lambda configuration — drop, HF is single-machine
28+
- r2hf/delay-render delayRender() — HF handles asset loading differently
29+
- r2hf/use-callback useCallback — usually decorative, drop
30+
- r2hf/use-memo useMemo — usually decorative, drop
31+
- r2hf/custom-hook Custom hook (use*) defined locally; may need manual rewrite
32+
33+
Info (translate and document):
34+
- r2hf/static-file staticFile("x") — convert to relative path
35+
- r2hf/interpolate-colors interpolateColors — translate to GSAP color tween
36+
"""
37+
38+
from __future__ import annotations
39+
40+
import argparse
41+
import json
42+
import re
43+
import sys
44+
from dataclasses import dataclass, asdict
45+
from pathlib import Path
46+
47+
BLOCKER = "blocker"
48+
WARNING = "warning"
49+
INFO = "info"
50+
51+
THIRD_PARTY_UI_PACKAGES = {
52+
"@mui/material",
53+
"@mui/icons-material",
54+
"@chakra-ui/react",
55+
"@mantine/core",
56+
"antd",
57+
"@shadcn/ui",
58+
"@radix-ui",
59+
"@nextui-org/react",
60+
}
61+
62+
63+
@dataclass
64+
class Finding:
65+
file: str
66+
line: int
67+
column: int
68+
severity: str
69+
rule: str
70+
message: str
71+
recommendation: str
72+
73+
74+
# Each rule: (pattern, severity, rule_id, message, recommendation).
75+
# Patterns are MULTILINE so ^/$ match line boundaries; we still report the
76+
# line number by re-scanning the line offset.
77+
RULES: list[tuple[re.Pattern[str], str, str, str, str]] = [
78+
(
79+
re.compile(r"\buseState\s*[(<]"),
80+
BLOCKER,
81+
"r2hf/use-state",
82+
"useState detected — Remotion compositions that drive animation via React state are not deterministic frame-capture targets in HyperFrames",
83+
"Use the runtime interop pattern from PR #214 instead of attempting a translation",
84+
),
85+
(
86+
re.compile(r"\buseReducer\s*[(<]"),
87+
BLOCKER,
88+
"r2hf/use-reducer",
89+
"useReducer detected — same issue as useState",
90+
"Use the runtime interop pattern from PR #214",
91+
),
92+
(
93+
# useEffect / useLayoutEffect with deps array that isn't [] —
94+
# i.e. side effects per dep change. Handles four common forms:
95+
# useEffect(() => { ... }, [deps]) (block body, multi-line)
96+
# useEffect(() => fetch(...), [deps]) (expression body, single line)
97+
# useEffect(function () { ... }, [deps]) (function expression)
98+
# useLayoutEffect(...) (same hook semantics)
99+
#
100+
# Anchor on the `, [<non-empty>]` deps signature with a lazy match for
101+
# the function arg. Leading `[\s\S]*?` is greedy-bounded by the deps
102+
# match itself, so we don't over-match across multiple useEffect calls.
103+
re.compile(r"\buse(?:Layout)?Effect\s*\([\s\S]*?,\s*\[[^\]]+\]\s*\)", re.DOTALL),
104+
BLOCKER,
105+
"r2hf/use-effect-deps",
106+
"useEffect/useLayoutEffect with non-empty deps — side effects don't translate to HF's seek-driven model",
107+
"Move the side-effect work into a build step, or use the runtime interop pattern",
108+
),
109+
(
110+
# `async` followed by `calculateMetadata` (with optional whitespace and `:` for type annotations).
111+
re.compile(r"calculateMetadata[^=]*=\s*async\b|async\s+calculateMetadata\b|calculateMetadata\s*:\s*async"),
112+
BLOCKER,
113+
"r2hf/async-metadata",
114+
"calculateMetadata returns a Promise — HF needs composition metadata up front",
115+
"Resolve metadata at build time and pass concrete values, or use runtime interop",
116+
),
117+
(
118+
re.compile(r"from\s+['\"]@remotion/lambda['\"]"),
119+
# Warning, not blocker: lambda config is orthogonal to the rendered
120+
# composition. Treating it as a hard stop would block translation of
121+
# otherwise-clean compositions whose authors happen to also configure
122+
# Lambda. The skill drops the lambda code in step 3 (Generate) and
123+
# writes a TRANSLATION_NOTES.md entry. See escape-hatch.md.
124+
WARNING,
125+
"r2hf/lambda-import",
126+
"@remotion/lambda is Remotion-specific distributed rendering — no HF equivalent today",
127+
"Drop the Lambda config; HF runs single-machine. Document the gap in TRANSLATION_NOTES.md.",
128+
),
129+
(
130+
re.compile(r"\bdelayRender\s*\("),
131+
WARNING,
132+
"r2hf/delay-render",
133+
"delayRender() — HF waits on asset readiness via the Frame Adapter pattern",
134+
"Drop the call; HF handles this transparently",
135+
),
136+
(
137+
re.compile(r"\buseCallback\s*\("),
138+
WARNING,
139+
"r2hf/use-callback",
140+
"useCallback — typically decorative for render performance, no HF equivalent needed",
141+
"Drop the wrapper, inline the function",
142+
),
143+
(
144+
re.compile(r"\buseMemo\s*\("),
145+
WARNING,
146+
"r2hf/use-memo",
147+
"useMemo — typically decorative, no HF equivalent needed",
148+
"Drop the wrapper, compute inline",
149+
),
150+
(
151+
re.compile(r"\bstaticFile\s*\("),
152+
INFO,
153+
"r2hf/static-file",
154+
"staticFile() reference — convert to a relative path in the HF composition",
155+
"Replace `staticFile(\"x.png\")` with `\"x.png\"` and copy the asset alongside the HTML",
156+
),
157+
(
158+
re.compile(r"\binterpolateColors\s*\("),
159+
INFO,
160+
"r2hf/interpolate-colors",
161+
"interpolateColors() — translate to a GSAP color tween",
162+
"See references/timing.md for the GSAP equivalent",
163+
),
164+
]
165+
166+
167+
def lint_file(path: Path) -> list[Finding]:
168+
src = path.read_text()
169+
findings: list[Finding] = []
170+
171+
# Line/column from a string offset.
172+
def loc(offset: int) -> tuple[int, int]:
173+
line = src.count("\n", 0, offset) + 1
174+
col = offset - (src.rfind("\n", 0, offset) + 1) + 1
175+
return line, col
176+
177+
for pattern, severity, rule, message, rec in RULES:
178+
for m in pattern.finditer(src):
179+
line, col = loc(m.start())
180+
findings.append(Finding(str(path), line, col, severity, rule, message, rec))
181+
182+
# Custom hook detection: any `function useXxx(`, `const useXxx = `,
183+
# `export function useXxx(`, or `export const useXxx = ` defined in this file.
184+
# `export default` is captured by the `default` branch.
185+
for m in re.finditer(
186+
r"^\s*(?:export\s+(?:default\s+)?)?(?:function|const|let|var)\s+(use[A-Z]\w+)\b",
187+
src,
188+
re.MULTILINE,
189+
):
190+
name = m.group(1)
191+
# Skip Remotion's own hooks — they're imported, not defined.
192+
if name in {"useCurrentFrame", "useVideoConfig"}:
193+
continue
194+
line, col = loc(m.start())
195+
findings.append(
196+
Finding(
197+
str(path),
198+
line,
199+
col,
200+
WARNING,
201+
"r2hf/custom-hook",
202+
f"Custom hook `{name}` defined locally — may need manual rewrite",
203+
"Inline the hook body if pure; bow out to runtime interop if it uses useState/useEffect",
204+
)
205+
)
206+
207+
# Third-party React UI library imports.
208+
for m in re.finditer(r"from\s+['\"]([^'\"]+)['\"]", src):
209+
pkg = m.group(1)
210+
if any(pkg.startswith(blocker) for blocker in THIRD_PARTY_UI_PACKAGES):
211+
line, col = loc(m.start())
212+
findings.append(
213+
Finding(
214+
str(path),
215+
line,
216+
col,
217+
BLOCKER,
218+
"r2hf/third-party-react-ui",
219+
f"Imports `{pkg}` — third-party React UI library has no HF equivalent",
220+
"Use runtime interop, or rewrite the affected components as HTML+CSS",
221+
)
222+
)
223+
224+
findings.sort(key=lambda f: (f.file, f.line, f.column))
225+
return findings
226+
227+
228+
def main() -> int:
229+
ap = argparse.ArgumentParser()
230+
ap.add_argument("path", type=Path, help="Directory or file to lint")
231+
ap.add_argument("--json", action="store_true", help="Emit JSON instead of human-readable output")
232+
args = ap.parse_args()
233+
234+
if not args.path.exists():
235+
print(f"error: {args.path} does not exist", file=sys.stderr)
236+
return 2
237+
238+
files: list[Path]
239+
if args.path.is_file():
240+
files = [args.path]
241+
else:
242+
files = sorted(
243+
p
244+
for p in args.path.rglob("*")
245+
if p.is_file()
246+
and p.suffix in {".ts", ".tsx", ".jsx", ".js"}
247+
and "node_modules" not in p.parts
248+
)
249+
250+
all_findings: list[Finding] = []
251+
for f in files:
252+
all_findings.extend(lint_file(f))
253+
254+
blockers = sum(1 for f in all_findings if f.severity == BLOCKER)
255+
warnings = sum(1 for f in all_findings if f.severity == WARNING)
256+
infos = sum(1 for f in all_findings if f.severity == INFO)
257+
258+
if args.json:
259+
json.dump(
260+
{
261+
"files_scanned": len(files),
262+
"blockers": blockers,
263+
"warnings": warnings,
264+
"infos": infos,
265+
"findings": [asdict(f) for f in all_findings],
266+
},
267+
sys.stdout,
268+
indent=2,
269+
)
270+
sys.stdout.write("\n")
271+
else:
272+
for f in all_findings:
273+
print(f"{f.file}:{f.line}:{f.column} [{f.severity}] {f.rule}: {f.message}")
274+
print(f" -> {f.recommendation}")
275+
print()
276+
print(f"{len(files)} files scanned · {blockers} blocker · {warnings} warning · {infos} info")
277+
if blockers:
278+
print("RECOMMENDATION: do not attempt translation. Use the runtime interop pattern from PR #214.")
279+
280+
return 1 if blockers else 0
281+
282+
283+
if __name__ == "__main__":
284+
sys.exit(main())

0 commit comments

Comments
 (0)