Repository navigation
Expand file tree
/
Copy pathindex.html
More file actions
154 lines (143 loc) · 10.3 KB
/
Copy pathindex.html
File metadata and controls
154 lines (143 loc) · 10.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="See how Näky turns an AV1 screen recording into a compact, time-ordered screen state for reasoning models.">
<meta name="theme-color" content="#07100f">
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self'; style-src 'self'; img-src 'self' data:; media-src 'self'; connect-src 'self'; object-src 'none'; base-uri 'none'; form-action 'none'">
<title>How Näky sees a screen recording</title>
<link rel="icon" href="assets/favicon.svg" type="image/svg+xml">
<link rel="stylesheet" href="assets/how-it-works.css">
<script type="module" src="assets/how-it-works.mjs"></script>
</head>
<body>
<header class="site-header">
<a class="wordmark" href="./" aria-label="Näky home">Näky<span aria-hidden="true">.</span></a>
<nav aria-label="Main navigation">
<a href="https://github.com/superlinked/naky">GitHub</a>
<a href="README.md#install">Install</a>
<a href="https://github.com/superlinked/naky/releases/tag/v0.1.0">v0.1.0</a>
<a href="evals/gui-world.html">Full evaluation</a>
</nav>
</header>
<main>
<section class="hero" aria-labelledby="page-title">
<p class="eyebrow">Screen perception for reasoning models</p>
<h1 id="page-title">Screen recordings in.<br><span>Structured state out.</span></h1>
<p class="lede">Näky accepts an AV1 screen recording and produces compact, chronological text for a reasoning model: what text was observed, where it was, what persisted, and when the screen changed. The model can follow a task over time without receiving every frame.</p>
</section>
<section class="reading-guide" aria-labelledby="preserves-title">
<p class="section-kicker">From recording to useful context</p>
<h2 id="preserves-title">A small, evolving account of the screen.</h2>
<div class="fact-grid">
<article><span>Input</span><h3>Start with a screen recording</h3><p>Näky reads an AV1 screen recording, whether it captures continuous motion or a timed series of screens.</p></article>
<article><span>Output</span><h3>Give the model chronological text</h3><p>Observed text and screen regions become a compact state that records what persists and what changes over time.</p></article>
<article><span>Honest boundary</span><h3>Keep pixels for visual detail</h3><p>Appearance, icons, color, fine visual state, and anything OCR misses may remain available only in pixels.</p></article>
</div>
</section>
<section class="demo-shell" aria-labelledby="demo-title">
<div class="demo-heading">
<div>
<p class="section-kicker" id="demo-kicker">Loading recording, one shared playhead</p>
<h2 id="demo-title">See the pixels and the state together</h2>
</div>
<div class="mode-control">
<span id="mode-label">What the model receives</span>
<div class="segment" role="group" aria-labelledby="mode-label">
<button type="button" class="mode-button is-active" data-mode="state" aria-pressed="true">Näky state</button>
<button type="button" class="mode-button" data-mode="pixels" aria-pressed="false">Screen pixels</button>
</div>
</div>
</div>
<div class="recording-picker">
<label for="recording-select">Choose a recording</label>
<select id="recording-select" aria-describedby="recording-summary" disabled>
<option>Loading recordings…</option>
</select>
<p id="recording-summary" role="status" aria-live="polite" aria-atomic="true">Loading the public recording gallery…</p>
</div>
<p id="mode-explanation" class="mode-explanation" aria-live="polite" aria-atomic="true">A text model receives the exact state stream on the right. The recording remains visible here only as a reference.</p>
<div class="comparison" data-mode="state">
<article class="screen-column" aria-labelledby="screen-title">
<div class="pane-title">
<div>
<span class="pane-number">01</span>
<h3 id="screen-title">What the screen shows</h3>
</div>
<span class="pane-tag" id="overlay-status">Current-state overlay</span>
</div>
<div class="overlay-legend" aria-label="Overlay legend">
<span><i class="legend-retained" aria-hidden="true"></i>Current retained observation</span>
<span><i class="legend-awaiting" aria-hidden="true"></i>Awaiting structural refresh</span>
<span>Activity and history remain in the exact state stream</span>
</div>
<div class="phone-stage" id="phone-stage">
<video id="demo-video" muted playsinline preload="auto" aria-label="Screen recording used to demonstrate Näky">
Your browser cannot play the selected demo video. <a id="video-fallback-download" href="demo/guicc-001.mp4">Download the MP4</a>.
</video>
<svg id="overlay" class="overlay" viewBox="0 0 1080 2400" preserveAspectRatio="none" aria-hidden="true"></svg>
<div class="screen-caption"><span class="status-dot" aria-hidden="true"></span><span id="moment-label">Loading authentic output…</span></div>
</div>
<div class="playback-controls">
<button id="play-button" class="play-button" type="button" aria-label="Play recording">
<svg viewBox="0 0 16 16" aria-hidden="true"><path class="play-icon" d="M4 2.7 13 8l-9 5.3Z"/><path class="pause-icon" d="M3.5 2.5h3v11h-3Zm6 0h3v11h-3Z"/></svg>
<span id="play-label">Play</span>
</button>
<div class="timeline-wrap">
<input id="timeline" type="range" min="0" max="8000" value="0" step="1" aria-label="Recording position" disabled>
<div id="marker-rail" class="marker-rail" aria-hidden="true"></div>
</div>
<output id="time-output" for="timeline">0:00 / 0:08</output>
</div>
<p id="video-error" class="asset-error" hidden>Video playback is unavailable, but the timeline and state stream remain inspectable. <a id="video-download" href="demo/guicc-001.mp4">Download the selected source MP4.</a></p>
</article>
<article class="state-column" aria-labelledby="state-title">
<div class="pane-title state-title-row">
<div>
<span class="pane-number">02</span>
<h3 id="state-title">What the model receives</h3>
</div>
<span class="raw-links"><a class="raw-link" id="raw-events-link" href="demo/guicc-001.events.ndjson">Events</a><a class="raw-link" id="raw-state-link" href="demo/guicc-001.screen.txt">Raw state</a></span>
</div>
<div class="grammar" aria-label="State syntax legend">
<span><b>@</b> time</span><span><b>S</b> screen</span><span><b>= + ~ > −</b> set · add · change · move · remove</span><span><b>!</b> observed activity</span>
</div>
<div class="state-viewport-wrap">
<div id="state-viewport" class="state-viewport" tabindex="0" aria-label="Time-ordered Näky state stream"></div>
<div class="not-sent" aria-hidden="true"><strong>Not sent</strong><span>The screenshot reader receives sampled pixels instead.</span></div>
</div>
<div class="state-footer">
<span id="state-status" role="status">Loading state stream…</span>
<button id="follow-button" type="button" hidden>Follow playhead</button>
</div>
</article>
</div>
<noscript>
<p class="noscript">JavaScript synchronizes and switches the recordings, overlays, and text. The <a href="demo/gallery.json">public demo catalog</a> links each closed manifest; you can also open the default <a href="demo/guicc-001.screen.txt">exact state stream</a>, <a href="demo/guicc-001.events.ndjson">ScreenEvents</a>, or <a href="demo/guicc-001.mp4">recording</a>.</p>
</noscript>
</section>
<section class="limits" aria-labelledby="limits-title">
<div>
<p class="section-kicker">Read the overlay literally</p>
<h2 id="limits-title">Observations, not inferred actions.</h2>
</div>
<div class="limits-copy">
<p>OCR-derived text and boxes are observations and can be wrong. Geometry and stable IDs show what Näky associated over time.</p>
<p>The live overlay is a clean view of retained observations, not event history. After reported activity, a region awaiting the next structural sample remains location-visible as a faint neutral outline and is counted with the recording; the exact state and raw streams remain unchanged.</p>
<p>Visual activity means pixels changed in a region during an interval. It is not a click, scroll, focus, selection, action, intent, or cause label.</p>
<p>The released comparison found higher quality from screenshots. Näky scored <strong>82.61%</strong> versus <strong>88.21%</strong> for screenshots across 3,468 tasks, while using <strong>50.91% fewer</strong> logical three-trial input tokens and an estimated <strong>46.50% lower</strong> cache-aware reader cost. <a href="evals/gui-world.html">Read the full matched comparison.</a></p>
</div>
</section>
<section class="attribution" aria-labelledby="source-title">
<h2 id="source-title">Demo source</h2>
<p id="source-summary" aria-live="polite">Select a recording to see its source. Third-party names, marks, and interface content belong to their owners. <a href="demo/ATTRIBUTION.md">Full attribution and transformation notes.</a></p>
</section>
</main>
<footer>
<a class="wordmark" href="./">Näky<span aria-hidden="true">.</span></a>
<p>Self-contained Rust screen perception.</p>
<nav aria-label="Footer navigation"><a href="README.md">Documentation</a><a href="evals/gui-world.html">Evaluation</a><a href="https://github.com/superlinked/naky">Source</a></nav>
</footer>
</body>
</html>