Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
5e74b2f
feat: make Call recording audio-first with optional video
nikshilov Aug 14, 2026
5d9792c
fix: qualify exact pre-merge release candidates
nikshilov Aug 14, 2026
fe8c945
fix: harden Call mode and video recovery
nikshilov Aug 14, 2026
7be77b5
ci: add macOS build and test gate
nikshilov Aug 14, 2026
28797ff
ci: make hosted macOS gate deterministic
nikshilov Aug 14, 2026
cb840fb
test: expand installed Call release gate
nikshilov Aug 14, 2026
c4544e4
ci: expose hosted XCTest failures
nikshilov Aug 14, 2026
79eaf71
ci: print concise XCTest failure reasons
nikshilov Aug 14, 2026
d06a3ae
ci: isolate release fixtures from Xcode test paths
nikshilov Aug 14, 2026
7fdc543
fix: preserve final MCP frames after clean exit
nikshilov Aug 14, 2026
1c15317
ci: install call fixture dependency
nikshilov Aug 14, 2026
0afde61
fix: render physical Call checklist literally
nikshilov Aug 14, 2026
50bf28a
fix: detach call audio from screen capture
nikshilov Aug 16, 2026
5ab19b5
fix: confirm Core Audio tap attachment
nikshilov Aug 16, 2026
55c474d
fix: pass complete PID to Core Audio
nikshilov Aug 16, 2026
b0461f6
fix: activate Core Audio tap on aggregate creation
nikshilov Aug 16, 2026
ecc5625
fix: preserve system audio while microphone is active
nikshilov Aug 16, 2026
656463a
diagnostic: measure decoded system audio peak
nikshilov Aug 16, 2026
67aa850
fix: include system audio privacy reason
nikshilov Aug 16, 2026
29ee57b
fix: keep screenshot priority active for calls
nikshilov Aug 16, 2026
0b89cf3
fix: finish call video audio muxing
nikshilov Aug 16, 2026
cb10868
fix: produce verifiable call video mp4
nikshilov Aug 16, 2026
732c1d3
fix: keep call audio alive across app crashes
nikshilov Aug 16, 2026
495f426
fix: yield call video immediately for screenshots
nikshilov Aug 16, 2026
6c13c20
fix: restore continuous call video frames
nikshilov Aug 16, 2026
2f925e3
fix: keep call video stream delivering frames
nikshilov Aug 16, 2026
e8d163c
fix: keep call video stable during native screenshots
nikshilov Aug 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions .github/workflows/macos-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
name: macOS CI

on:
pull_request:
push:
branches:
- main

concurrency:
group: macos-ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read

jobs:
build-and-test:
name: Build and unhosted tests
runs-on: macos-26
timeout-minutes: 60
env:
DEVELOPER_DIR: /Applications/Xcode_26.5.app/Contents/Developer
HOMEBREW_NO_AUTO_UPDATE: "1"

steps:
- name: Check out source
uses: actions/checkout@v4

- name: Install build tools
run: brew install xcodegen ripgrep

- name: Run release preflight fixture contract
run: |
set +e
xcodegen generate
xcodebuild -quiet \
-project ZBSEye.xcodeproj \
-scheme ZBSEyeUnitTests \
-configuration Debug \
-derivedDataPath "$RUNNER_TEMP/ZBSEyeDerivedData" \
-resultBundlePath "$RUNNER_TEMP/ReleaseConfigurationTests.xcresult" \
CODE_SIGNING_ALLOWED=NO \
test \
-only-testing:ZBSEyeTests/ReleaseConfigurationTests
test_status=$?
set -e
if [ "$test_status" -ne 0 ]; then
xcrun xcresulttool get test-results summary \
--path "$RUNNER_TEMP/ReleaseConfigurationTests.xcresult" \
--compact \
| python3 -c 'import json, sys; data = json.load(sys.stdin); [print("{}: {}".format(failure["testName"], failure["failureText"])) for failure in data["testFailures"]]' \
|| true
fi
exit "$test_status"

- name: Run full unhosted suite
run: |
set +e
xcodebuild -quiet \
-project ZBSEye.xcodeproj \
-scheme ZBSEyeUnitTests \
-configuration Debug \
-derivedDataPath "$RUNNER_TEMP/ZBSEyeDerivedData" \
-resultBundlePath "$RUNNER_TEMP/ZBSEyeUnitTests.xcresult" \
CODE_SIGNING_ALLOWED=NO \
test
test_status=$?
set -e
if [ "$test_status" -ne 0 ]; then
xcrun xcresulttool get test-results summary \
--path "$RUNNER_TEMP/ZBSEyeUnitTests.xcresult" \
--compact \
| python3 -c 'import json, sys; data = json.load(sys.stdin); [print("{}: {}".format(failure["testName"], failure["failureText"])) for failure in data["testFailures"]]' \
|| true
fi
exit "$test_status"

- name: Build full GUI target
run: |
xcodebuild -quiet \
-project ZBSEye.xcodeproj \
-scheme ZBSEye \
-configuration Debug \
-derivedDataPath "$RUNNER_TEMP/ZBSEyeDerivedData" \
CODE_SIGNING_ALLOWED=NO \
build

- name: Run Call fixture contract
env:
ZBS_EYE_CALL_DERIVED_DATA_PATH: ${{ runner.temp }}/ZBSEyeDerivedData
run: bash scripts/verify-call-recording.sh --fixtures

- name: Validate capture coexistence protocol
run: bash scripts/verify-capture-coexistence.sh --self-test

- name: Upload test result
if: always()
uses: actions/upload-artifact@v4
with:
name: ZBSEyeUnitTests-${{ github.run_id }}
path: |
${{ runner.temp }}/ReleaseConfigurationTests.xcresult
${{ runner.temp }}/ZBSEyeUnitTests.xcresult
if-no-files-found: warn
retention-days: 7
103 changes: 85 additions & 18 deletions AGENTS.md

Large diffs are not rendered by default.

31 changes: 29 additions & 2 deletions BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,19 @@ bash scripts/verify-call-automation.sh
bash scripts/verify-capture-coexistence.sh --self-test
```

Pull requests and pushes to `main` repeat the full unhosted suite, Debug GUI build, Call fixture contract, and
capture-coexistence protocol self-test on GitHub's explicit `macos-26` runner with Xcode 26.5. CI never launches
Eye, requests TCC, captures media, downloads models, signs, notarizes, installs, or publishes a release. A green
hosted check is necessary but does not replace installed-app Calls and native-screenshot qualification.

## Architecture
```
ZBSEyeApp/
App/ ZBSEyeApp (@main), AppEnvironment (@Observable root)
Capture/ Persistent screen stream, latest-wins FramePipeline, SCKResourceCoordinator, screenshot priority, AXReader
Audio/ AudioCoordinator, mic/system engines, VADSegmenter, TranscriptionService
Meeting/ CoreAudio mic-owner listener, initiator/relay resolution, automatic Call detection
Calls/ CallCoordinator, lifecycle policy, dual-source spool, Calls projection, Whisper/diarization helpers
Calls/ CallCoordinator, launchd audio owner, dual-source spool, Call video, AAC post-process, projection, deletion
Data/ ZBSEyeDatabase, StorageLocation, StorageManager, BackupManager, RetentionManager, IngestService
Search/ SearchService (FTS+vector RRF), EmbeddingService (e5), TimelineService, VectorBackfill
Server/ ZBSEyeHTTPServer (FlyingFox REST, 127.0.0.1, Bearer), KeychainStore
Expand All @@ -57,6 +62,21 @@ See [`AGENTS.md`](AGENTS.md) for the architecture map, invariants, and gotchas.

## Call recorder runtime and model

Calls have an independent persisted mode: `off`, `audio`, or `audio_video`. Video is a separate hardware-only
ScreenCaptureKit/AVFoundation path capped at 1080p and 15 fps. It never owns or backpressures audio; finalized
30-second MP4 fragments receive a background mixed AAC convenience track while the PCM sources remain authoritative.
Starting another Call invalidates that background job without waiting for it; unfinished mux work retries only
after physical Call audio has stopped. If the app dies between MP4 replacement and database publication, bootstrap
uses the segment hash to restore the silent rollback copy or retain the already committed muxed bytes.
The display is locked when authoritative Call audio starts. A native screenshot closes both Timeline and Call-video
admission immediately; video resumes only after the shared helper-aware quiet gate opens, without restarting audio.
The app bundle also contains `Contents/Library/LaunchAgents/gg.zbs.eye.call-audio.plist`. `SMAppService` registers
that same-signed executable as the authoritative mic/Core Audio owner. Its mutually authenticated local XPC surface
accepts only bounded Call lifecycle messages. It appends Call PCM/source-gap evidence through `CallRepository`,
never runs migrations, and never writes Timeline, FTS, vector, or transcript state. A GUI crash leaves it recording;
relaunch adopts the active Call. A helper restart reconciles its open PCM, records the interruption, and resumes in
a new epoch. The installed physical gate must prove both failure paths; an unsigned build cannot qualify them.

The shipping source pins two independent artifacts:

- Runtime: official `whisper.cpp` `v1.9.1` XCFramework in `Packages/ZBSEyeWhisper/Package.swift`, SwiftPM
Expand All @@ -72,7 +92,9 @@ downloaded helper binary. It receives one immutable manifest and writes one boun
GUI remains the only database writer.

`scripts/verify-call-recording.sh --fixtures` is deterministic and does not launch the app, capture media,
request TCC permissions, or download weights. Permission-sensitive qualification is deliberately separate:
request TCC permissions, or download weights. Its DerivedData is temporary and removed on exit; set
`ZBS_EYE_CALL_DERIVED_DATA_PATH` only when a persistent reusable cache is intentional. Permission-sensitive
qualification is deliberately separate:

```bash
ZBS_EYE_CALL_PHYSICAL_GATE=YES scripts/verify-call-recording.sh --physical-preflight
Expand Down Expand Up @@ -103,6 +125,11 @@ Eye fails open and yields through the later screenshot-helper process signal; ma
a required release check. Automated fixtures do not replace the physical shortcut matrix, lifecycle/recovery
matrix, 30-minute churn, or two-hour installed soak.

When physical qualification must happen before merge, pass the exact pushed PR branch to the notarized build:
`ZBSEYE_RELEASE_CANDIDATE_REF=codex/<branch> bash scripts/build-notarized.sh`. This only admits a clean remote
descendant of canonical `main`; publication still requires the same commit to land on `main` unchanged and a
normal `scripts/release-preflight.sh --verify-only` with the variable unset.

### Optional speaker diarization

The app pins FluidAudio `0.15.5` (commit `19600a485baa4998812e4654b70d2bab8f2c9949`) and the
Expand Down
55 changes: 54 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,60 @@

All notable changes to ZBS Eye. The format follows Added / Changed / Fixed sections.

## [0.8.0] — Unreleased
## Unreleased — 0.9.0 (25) candidate source

### Added
- Calls now have **Don't record / Audio only / Audio and video** modes globally and on the active Call.
- Call video records one fixed display at up to 1080p/15 fps in hardware-only 30-second MP4 fragments, with
explicit spans/gaps, sequential playback, authenticated typed references, export, retention, and deletion.

### Changed
- Audio physically starts before video and remains independent during live video toggles, display loss, native
screenshot yields, and background AAC muxing. Existing Mic in use/Always profiles migrate to Audio only;
an explicit old Off remains Don't record, and video is never enabled silently. Timeline audio stays independent.

### Fixed
- Call audio has absolute priority over Timeline capture and video work; native screenshots temporarily stop the
physical Call video stream without stopping either audio source.
- Native screenshot teardown no longer waits behind synchronous HEIC/OCR work, and Call video stays stopped until
the native helper plus the full quiet tail have ended. A newly enabled Call-video stream joins that same gate
instead of starting inside an already-open screenshot window.
- Call video drains every accepted latest-wins frame before finalizing a span, reports separate frame-drop gaps,
and publishes hardware encoder failure without waiting for the Call to end.
- Failed hardware codec attempts and unpublished final segments remove their own bytes, so HEVC failure cannot
block the hardware H.264 fallback or leave unreachable Call video on disk.
- A Call locks its display when audio starts, even when video is enabled later. The current binary recognizes the
v17 schema, and REST/OpenAPI now expose recording modes, video segments, and video gaps.
- Review schedule migration no longer triggers an immediate catch-up, and changing its selected period cannot
leave the panel permanently busy.
- Timeline Review now pins the independently verified official Claude Code 2.1.232 arm64 artifact, while its
installed-binary test remains deterministic when Anthropic's local auto-updater has another version installed.
- Call fixture verification uses disposable DerivedData by default instead of silently growing the repository's
`build/` directory; an explicit cache path remains available for intentional reuse.
- Release preflight can notarize an explicitly named, clean remote PR descendant for the pre-merge physical gate;
publication still requires that exact SHA on canonical main with no candidate override.
- Rapid active-Call mode changes now reconcile to the latest selection in one serial loop, without restarting
Call audio. Audio-only and native-screenshot stops expose honest disabled/gap video state instead of available.
- A failed video start keeps one gap open until a successful restart, an audio-only switch, or Call end, so a
missing locked display can no longer collapse minutes of absent video into a one-millisecond gap.
- Repeated native-screenshot edges during video resume now extend that same pending gap instead of publishing
overlapping intervals in Call Detail, REST/MCP, and export evidence.
- Starting Call audio now invalidates background AAC/MP4 postprocess work without waiting for it. Cancelled mux
work skips degradation, ignores already-muxed segments, and retries remaining segments after audio stops.
- Crash recovery now reconciles the postprocess rollback MP4 against the generation-bound database hash: it
restores uncommitted silent video, keeps committed muxed video, and preserves unknown evidence for inspection.
- Pull requests and `main` now run an independent macOS 26 / Xcode 26.5 CI gate for the full unhosted suite,
Debug GUI build, Call fixture contract, and capture-coexistence protocol self-test.
- Review schedule keys now use the same calendar time zone that selected the day or seven-day period, so a
runner or Mac time-zone difference cannot move a completed Review cursor to the previous date.
- CI fixtures no longer inherit machine-wide Git policy, depend on an installed `/Applications/ZBS Eye.app`, or
assume a loaded hosted runner will dispatch an expired automation lease within a fixed 120 ms window.
- Hosted CI now runs release-preflight fixtures as an early contract and prints structured XCTest failure
messages before uploading result bundles, so runner-only failures stay diagnosable without a large download.
- The installed-candidate Call report now makes every three-mode/video, fixed-display, screenshot-latency,
overflow, recovery, deletion, trimming, and export check explicit instead of reusing the old audio-only gate.

## [0.8.0] — 2026-08-08

### Added
- Timeline now shows a clickable seven-image filmstrip around the selected visual moment and keeps decoded
Expand Down
30 changes: 21 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ Eye records useful evidence and gives it back. It is not a CRM, a calendar manag

- **Timeline** combines screen frames, extracted text, app context, audio density, call spans, and bookmarks
in one scrubbable view. A seven-image filmstrip makes nearby visual moments immediately reachable.
- **Calls** keeps meetings out of the all-day activity stream. Microphone and system audio remain separate;
- **Calls** has three explicit modes, independent from Timeline audio: **Don't record / Audio only / Audio and video**. Microphone and system audio remain separate;
optional Call video fixes the starting display at up to 1080p/15 fps while Timeline images stay paused;
a bookmark schedules a checkpoint transcript without interrupting the recording.
- **Search** combines FTS5 with local multilingual semantic retrieval, so a query in one language can find
evidence written in another.
Expand All @@ -60,7 +61,9 @@ With listen-event access already available, Eye sees native screenshot shortcuts
work, and rejects any in-flight result that crosses the screenshot boundary; those shortcuts never create an Eye
moment. Otherwise the screenshot-helper fallback yields as soon as macOS exposes it. Already-running synchronous
AX or Vision work may finish in the background, but cannot be saved as an Eye frame. Eye never requests a new
permission for this and does not stop or rebuild its stream.
permission for this. The screenshot signal closes visual admission immediately and starts a physical stop of
Eye's screen stream without waiting for AX, OCR, or HEIC; the next ordinary visual intent creates one fresh
stream only after the native helper and its quiet tail have ended. Call audio continues throughout.

Timeline resolves every selected time to the nearest real image at or before that time—never a future frame.
It keeps two earlier, the current, and four later images close at hand, loads at most two files concurrently,
Expand All @@ -78,13 +81,16 @@ name, or keep one alive by itself. Eye also excludes its own processes, a narrow
they never decide whether recording is allowed to start. Detection, recording, and saving do not require
internet access.

Eye requests separate microphone and system-audio tracks. If one track is unavailable, the Call remains
recording and reports the missing track instead of silently hiding the gap. `Audio Off`, privacy pause,
Eye requests separate microphone and system-audio tracks. Settings choose whether automatic Calls are disabled,
audio-only, or audio plus the starting display. The active Call can switch between audio and video without
restarting audio; switching to Don't record ends and saves it. If one track is unavailable, the Call remains
recording and reports the missing track instead of silently hiding the gap. `Don't record`, privacy pause,
critically low disk, and storage relocation remain hard stops.

The main **Record Timeline / Pause Timeline** control governs ordinary screen and Timeline audio capture; it
does not disarm microphone-triggered Calls. Choose **Audio Off** or start a privacy pause when no automatic
Call should begin. Privacy pause remains available from the menu bar even while Timeline capture is paused.
The main **Record Timeline / Pause Timeline** control and Timeline audio settings govern ordinary history; they
do not disarm microphone-triggered Calls. Choose **Don't record** or start a privacy pause when no automatic
Call should begin. **Don't record** disables automatic Calls; a manual Start then asks for one-time audio or
audio-and-video without changing the default. Privacy pause remains available from the menu bar even while Timeline capture is paused.

Eye itself and a narrow list of macOS system daemons are always excluded. **Settings → Audio → Don’t
auto-record these apps** adds user-selected bundle identifiers to a separate audio exclusion list; this does
Expand Down Expand Up @@ -122,16 +128,22 @@ Recording, indexing, Timeline, Calls, search, REST, and MCP do not require an AI
On qualified Apple Silicon hardware, one click can download the pinned ZBS Eye Local model. You may instead
connect a local server, signed-in CLI, or cloud API provider.

That choice changes only generation features such as Ask and Daily Insights. Before any non-local provider
That choice changes only generation features such as Ask and Daily Insights. Timeline **Review** is narrower:
it uses an existing Codex or Claude Code subscription login, never an API key, and shows provider-reported
input/cache/output/reasoning tokens plus Codex credits when the bundled dated rate card knows the model.
Before any non-local provider
receives a prompt excerpt, Eye names the recipient and asks for scoped consent. Raw recordings, the database,
and storage never become provider uploads. API credentials live in the macOS data-protection Keychain.

## Install

1. Open the [current stable release](https://github.com/zbs-gg/eye/releases/latest), download the notarized ZIP,
move **ZBS Eye.app** to `/Applications`, and launch it.
2. Grant Screen Recording and Accessibility. Microphone is optional; system audio has its own switch.
2. Grant Screen Recording and Accessibility. Microphone and System Audio Recording are separate permissions;
Eye asks for each only when the corresponding recording is enabled.
3. Press **Record Timeline**. Open Timeline and change windows once to see the first moments appear.
4. Press **Review** beside Ask to build a saved day or seven-day recap. Scheduling is optional and has a
separate background-consent switch.

ZBS Eye is distributed outside the Mac App Store because cross-app Accessibility and continuous capture are
incompatible with the App Sandbox. Release artifacts use Developer ID, Hardened Runtime, Apple notarization,
Expand Down
Loading
Loading