Skip to content

Commit feca8cb

Browse files
authored
fix(apple): refuse captures whose window or panel identity is unknown (#3761)
* fix(macos): refuse an app screenshot whose front window is not shareable ADR 0031 decides that a native macOS app screenshot captures only the app's front window through ScreenCaptureKit (#3188). When the front window's WindowServer ID was missing from the shareable candidates, the helper captured the app's largest window instead, so the image could show a window the user was not looking at. The helper now selects the front window by ID or refuses with a typed reason: `window-not-found` when the app has no on-screen window, `front-window-not-shareable` when ScreenCaptureKit does not offer the front window. The shared outcomes fixture pins both reasons for Swift and TypeScript, and the host adds a hint for each. Evidence: the audit at bd4fd68 (AppWindowScreenshot.swift:21); re-verified on 4c1c786. Part of #3744. * fix(apple): refuse a foldable capture when CoreDevice names no panel ADR 0025 decides that only CoreDevice facts identify a panel: the sole lit panel, otherwise the `primary` panel, and never array position. When no panel reported `primary` and power did not decide one lit panel, the inventory captured the first lit or first integrated panel in list order. That inventory now names no panel, and capture-target resolution refuses with COMMAND_FAILED, reason `capture-panel-unidentified`, and a hint. The sole-lit-panel path and the primary-panel path are unchanged. The foldable help topic states the refusal. Evidence: the audit at bd4fd68 (display-inventory.ts:261); re-verified on 4c1c786. Part of #3744. * fix(macos): skip invisible windows when selecting a screenshot Choose the first on-screen normal-layer app window with positive alpha and nonempty bounds. Keep interaction window selection unchanged and refuse if an eligible front window is absent from ScreenCaptureKit. Review: #3761 (comment) * fix(macos): clarify screenshot capture exclusion hint Review: #3761 (comment) * docs(foldable): clarify primary panel capture ambiguity Review: #3761 (comment) * fix(macos): share front-window eligibility across capture and actions * docs(macos): define the native front-window target * fix(macos): distinguish windows excluded from screen capture * refactor(macos): centralize front window selection Share the on-screen window-list query while preserving nil versus empty point refusals. Pin layer-8 alert exclusion at the shared selection boundary. Review: #3761 (comment) * docs(capture): clarify alert exclusion and refusal remedy * fix(macos): distinguish unreadable window lists from missing windows Refuse screenshot and scroll with window-list-unreadable when the shared window-list read fails, and preserve a retry hint through the host helper client. Review: #3761 (comment) * refactor(apple): move macOS refusal hints into the helper The helper owns each refusal's reason and remedy, so its details envelope carries the hint and the TypeScript client needs no lookup table, wrapper, or special case. Collapse the one-case WindowListRefusal enum and the screenshot Result mapping into the throw sites that already own them, and keep the front-window selection as one plain function. Press, fill, and their window fallback use the non-throwing path; requireAppWindowPoint has already refused an unreadable list there. The front-window help sentence states the layer-0 rule and the alert exclusion once.
1 parent cf42352 commit feca8cb

10 files changed

Lines changed: 443 additions & 53 deletions

File tree

‎apple/macos-helper/Sources/AgentDeviceMacOSHelper/AppWindowScreenshot.swift‎

Lines changed: 60 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -10,20 +10,15 @@ func captureAppWindowScreenshot(app: NSRunningApplication, outPath: String) thro
1010
throw HelperError.commandFailed("app window screenshots require macOS 14 or newer")
1111
}
1212
let pid = app.processIdentifier
13-
let frontWindowID = frontWindowNumber(pid: pid)
13+
let front = try frontWindow(pid: pid)
1414
let content = try awaitShareableContent()
15-
let candidates = content.windows.filter {
16-
$0.owningApplication?.processID == pid && $0.windowLayer == 0 && $0.frame.width > 0
17-
&& $0.frame.height > 0
18-
}
19-
guard
20-
let window = candidates.first(where: { Int($0.windowID) == frontWindowID })
21-
?? candidates.max(by: { $0.frame.width * $0.frame.height < $1.frame.width * $1.frame.height })
22-
else {
23-
throw HelperError.commandFailed(
24-
"screenshot could not find a window for the app",
25-
details: ["reason": "window-not-found", "bundleId": app.bundleIdentifier ?? ""]
26-
)
15+
let candidates = content.windows.filter { $0.owningApplication?.processID == pid }
16+
let window: SCWindow
17+
switch frontShareableWindow(candidates, front: front, windowID: { Int($0.windowID) }) {
18+
case .success(let shareable):
19+
window = shareable
20+
case .failure(let refusal):
21+
throw refusal.error(bundleId: app.bundleIdentifier)
2722
}
2823

2924
let filter = SCContentFilter(desktopIndependentWindow: window)
@@ -52,6 +47,58 @@ func captureAppWindowScreenshot(app: NSRunningApplication, outPath: String) thro
5247
try writePNG(capturedImage, to: outPath)
5348
}
5449

50+
/// Why an app screenshot refuses, and the error the host reports for it: reason, remedy, and the
51+
/// front window it refused. The capture is the app's front window or nothing; another window of
52+
/// the app would show state the user is not looking at.
53+
enum AppWindowScreenshotRefusal: Error {
54+
case windowNotFound
55+
case frontWindowNotShareable(frontWindowId: Int)
56+
case frontWindowCaptureExcluded(frontWindowId: Int)
57+
58+
func error(bundleId: String?) -> HelperError {
59+
var details = ["bundleId": bundleId ?? ""]
60+
let message: String
61+
switch self {
62+
case .windowNotFound:
63+
details["reason"] = "window-not-found"
64+
details["hint"] =
65+
"Open or unminimize a window of the app on the current Space, then retry the screenshot."
66+
message = "screenshot could not find an on-screen window for the app"
67+
case .frontWindowCaptureExcluded(let frontWindowId):
68+
details["reason"] = "front-window-capture-excluded"
69+
details["frontWindowId"] = String(frontWindowId)
70+
details["hint"] =
71+
"The app excludes the front window from screen capture. Run `snapshot` to read the window's UI tree; retrying the excluded window will not help."
72+
message =
73+
"screenshot could not capture the app's front window: the app excludes it from capture"
74+
case .frontWindowNotShareable(let frontWindowId):
75+
details["reason"] = "front-window-not-shareable"
76+
details["frontWindowId"] = String(frontWindowId)
77+
details["hint"] =
78+
"The front window is unavailable to screen capture. If it is opening or closing, retry after it settles."
79+
message =
80+
"screenshot could not capture the app's front window: ScreenCaptureKit does not offer it"
81+
}
82+
return .commandFailed(message, details: details)
83+
}
84+
}
85+
86+
/// The shareable candidate that is the app's front window, or the reason the screenshot refuses.
87+
func frontShareableWindow<Window>(
88+
_ candidates: [Window],
89+
front: OnScreenWindow?,
90+
windowID: (Window) -> Int
91+
) -> Result<Window, AppWindowScreenshotRefusal> {
92+
guard let front else { return .failure(.windowNotFound) }
93+
guard front.sharingState != 0 else {
94+
return .failure(.frontWindowCaptureExcluded(frontWindowId: front.number))
95+
}
96+
guard let shareable = candidates.first(where: { windowID($0) == front.number }) else {
97+
return .failure(.frontWindowNotShareable(frontWindowId: front.number))
98+
}
99+
return .success(shareable)
100+
}
101+
55102
@available(macOS 14.0, *)
56103
private func awaitShareableContent() throws -> SCShareableContent {
57104
let semaphore = DispatchSemaphore(value: 0)

‎apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift‎

Lines changed: 41 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,7 @@ func scrollInBackground(
206206
guard isVertical || direction == "left" || direction == "right" else {
207207
throw HelperError.invalidArgs("scroll requires --direction <up|down|left|right>")
208208
}
209-
guard let frame = frontWindowFrame(pid: app.processIdentifier) else {
209+
guard let frame = try frontWindowFrame(pid: app.processIdentifier) else {
210210
throw HelperError.commandFailed(
211211
"scroll could not resolve an on-screen window",
212212
details: ["reason": "window-not-found", "bundleId": app.bundleIdentifier ?? ""]
@@ -306,10 +306,7 @@ private func perform(_ target: PressTarget) -> BackgroundDeliveryMechanism? {
306306
/// own on-screen windows, its menus and sheets included, and never on a menu bar element: the app
307307
/// owns its menu bar's window too, so the window check alone does not exclude it.
308308
private func requireAppWindowPoint(_ point: CGPoint, app: NSRunningApplication) throws {
309-
guard
310-
let info = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID)
311-
as? [[String: Any]]
312-
else {
309+
guard let info = appWindowInfo() else {
313310
throw refusal(.noAccessibleTarget, "the app's on-screen windows could not be read", app: app)
314311
}
315312
let inAppWindow = info.contains { entry in
@@ -363,13 +360,14 @@ private func pressSearchChain(from hit: AXUIElement) -> [AXUIElement] {
363360

364361
/// The one window a point action may act in: the window the app's hit test landed in, else the
365362
/// app's front on-screen window. A fallback target never comes from a window behind it, and the
366-
/// response names the window so the host can check it.
363+
/// response names the window so the host can check it. The caller's `requireAppWindowPoint` has
364+
/// already read the window list, so a list that turned unreadable here is treated as no window.
367365
private func actionWindow(app: NSRunningApplication, hit: AXUIElement?) -> AXUIElement? {
368366
if let hit {
369367
if role(of: hit) == "AXWindow" { return hit }
370368
if let window = elementAttribute(hit, attribute: kAXWindowAttribute as String) { return window }
371369
}
372-
guard let front = frontWindowFrame(pid: app.processIdentifier) else { return nil }
370+
guard let front = try? frontWindowFrame(pid: app.processIdentifier) else { return nil }
373371
return windows(of: AXUIElementCreateApplication(app.processIdentifier)).first { window in
374372
guard let rect = rectAttribute(window) else { return false }
375373
return abs(rect.x - front.minX) < 1 && abs(rect.y - front.minY) < 1
@@ -482,39 +480,57 @@ private func numberAttribute(_ element: AXUIElement, attribute: String) -> Doubl
482480
return number.doubleValue
483481
}
484482

485-
private struct OnScreenWindow {
483+
struct OnScreenWindow {
486484
let number: Int
487485
let bounds: CGRect
486+
let sharingState: Int?
488487
}
489488

490-
/// The pid's normal-layer windows, front to back. Bounds and owners need no Screen Recording
491-
/// permission; only titles do.
492-
private func onScreenWindows(pid: pid_t) -> [OnScreenWindow] {
493-
guard
494-
let info = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID)
495-
as? [[String: Any]]
496-
else {
497-
return []
498-
}
499-
return info.compactMap { entry in
489+
/// Only normal-layer windows qualify; app-modal alerts at level 8 are excluded.
490+
let frontWindowLayer = 0
491+
492+
/// The pid's visible, nonempty normal-layer windows, front to back.
493+
func onScreenWindows(pid: pid_t, windowInfo: [[String: Any]]) -> [OnScreenWindow] {
494+
windowInfo.compactMap { entry in
500495
guard (entry[kCGWindowOwnerPID as String] as? Int32) == pid,
501-
(entry[kCGWindowLayer as String] as? Int) == 0,
496+
(entry[kCGWindowLayer as String] as? Int) == frontWindowLayer,
497+
(entry[kCGWindowIsOnscreen as String] as? Bool) == true,
498+
let alpha = entry[kCGWindowAlpha as String] as? Double, alpha > 0,
502499
let number = entry[kCGWindowNumber as String] as? Int,
503500
let boundsDict = entry[kCGWindowBounds as String] as? NSDictionary,
504-
let bounds = CGRect(dictionaryRepresentation: boundsDict as CFDictionary)
501+
let bounds = CGRect(dictionaryRepresentation: boundsDict as CFDictionary),
502+
bounds.width > 0, bounds.height > 0
505503
else {
506504
return nil
507505
}
508-
return OnScreenWindow(number: number, bounds: bounds)
506+
return OnScreenWindow(
507+
number: number, bounds: bounds, sharingState: entry[kCGWindowSharingState as String] as? Int)
509508
}
510509
}
511510

512-
func frontWindowNumber(pid: pid_t) -> Int? {
513-
onScreenWindows(pid: pid).first?.number
511+
func appWindowInfo() -> [[String: Any]]? {
512+
CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID)
513+
as? [[String: Any]]
514+
}
515+
516+
/// The app's front on-screen window: frontmost among its eligible windows, or nil when it has
517+
/// none. A window list that cannot be read is a refusal, not an empty list: the caller must not
518+
/// report "no window" for a WindowServer it could not ask.
519+
func frontWindow(pid: pid_t, windowInfo: [[String: Any]]? = appWindowInfo()) throws -> OnScreenWindow? {
520+
guard let windowInfo else {
521+
throw HelperError.commandFailed(
522+
"the app's on-screen windows could not be read",
523+
details: [
524+
"reason": "window-list-unreadable",
525+
"hint": "The app's on-screen window list could not be read. Retry the command.",
526+
]
527+
)
528+
}
529+
return onScreenWindows(pid: pid, windowInfo: windowInfo).first
514530
}
515531

516-
private func frontWindowFrame(pid: pid_t) -> CGRect? {
517-
onScreenWindows(pid: pid).first?.bounds
532+
func frontWindowFrame(pid: pid_t, windowInfo: [[String: Any]]? = appWindowInfo()) throws -> CGRect? {
533+
try frontWindow(pid: pid, windowInfo: windowInfo)?.bounds
518534
}
519535

520536
private let returnVirtualKey: CGKeyCode = 36
Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
import CoreGraphics
2+
import XCTest
3+
4+
@testable import AgentDeviceMacOSHelper
5+
6+
private struct Candidate: Equatable {
7+
let id: Int
8+
let area: Double
9+
}
10+
11+
/// An app screenshot captures the front window or refuses; it never substitutes another window.
12+
final class AppWindowScreenshotTests: XCTestCase {
13+
func testCaptureIdentityAndScrollAndClickFallbackFrameSelectTheSameEligibleFrontWindow() throws {
14+
let windows = [
15+
windowInfo(id: 1, pid: 99),
16+
windowInfo(id: 2, layer: 1),
17+
windowInfo(id: 3, onScreen: false),
18+
windowInfo(id: 4, alpha: 0),
19+
windowInfo(id: 5, width: 0),
20+
windowInfo(id: 6, height: 0),
21+
windowInfo(id: 7, alpha: 0.5, width: 20, height: 20),
22+
windowInfo(id: 8),
23+
]
24+
25+
let front = try frontWindow(pid: 42, windowInfo: windows)
26+
let selected = frontShareableWindow([7, 8], front: front, windowID: { $0 })
27+
28+
// Scroll and the click actionWindow fallback both consume frontWindowFrame.
29+
XCTAssertEqual(
30+
try frontWindowFrame(pid: 42, windowInfo: windows),
31+
CGRect(x: 10, y: 10, width: 20, height: 20))
32+
XCTAssertEqual(front?.number, 7)
33+
XCTAssertEqual(try selected.get(), 7)
34+
}
35+
36+
func testSkipsAppModalAlertInFrontOfNormalLayerWindow() throws {
37+
let windows = [windowInfo(id: 8, layer: 8), windowInfo(id: 3)]
38+
let front = try frontWindow(pid: 42, windowInfo: windows)
39+
let selected = frontShareableWindow([8, 3], front: front, windowID: { $0 })
40+
41+
XCTAssertEqual(front?.number, 3)
42+
XCTAssertEqual(try selected.get(), 3)
43+
XCTAssertEqual(try frontWindowFrame(pid: 42, windowInfo: windows), front?.bounds)
44+
}
45+
46+
func testRefusesWhenOnlyInvisibleOrEmptyWindowsRemain() throws {
47+
let windows = [
48+
windowInfo(id: 1, onScreen: false),
49+
windowInfo(id: 2, alpha: 0),
50+
windowInfo(id: 3, width: 0),
51+
windowInfo(id: 4, height: 0),
52+
]
53+
let front = try frontWindow(pid: 42, windowInfo: windows)
54+
let selected = frontShareableWindow([5], front: front, windowID: { $0 })
55+
56+
XCTAssertNil(try frontWindowFrame(pid: 42, windowInfo: windows))
57+
XCTAssertNil(front)
58+
XCTAssertThrowsError(try selected.get()) { error in
59+
assertRefusal(error, reason: "window-not-found", hint: "Open or unminimize")
60+
}
61+
}
62+
63+
func testUnreadableWindowListRefusesBeforeCaptureOrScrollCanReportNoWindow() {
64+
XCTAssertThrowsError(try frontWindow(pid: 42, windowInfo: nil)) { error in
65+
assertUnreadableList(error)
66+
}
67+
XCTAssertThrowsError(try frontWindowFrame(pid: 42, windowInfo: nil)) { error in
68+
assertUnreadableList(error)
69+
}
70+
XCTAssertNil(try frontWindow(pid: 42, windowInfo: []))
71+
XCTAssertNil(try frontWindowFrame(pid: 42, windowInfo: []))
72+
}
73+
74+
func testSelectsTheFrontWindowEvenWhenAnotherIsLarger() {
75+
let candidates = [Candidate(id: 7, area: 10_000), Candidate(id: 3, area: 100)]
76+
77+
let selected = frontShareableWindow(candidates, front: onScreenWindow(id: 3), windowID: \.id)
78+
79+
XCTAssertEqual(try selected.get(), Candidate(id: 3, area: 100))
80+
}
81+
82+
func testRefusesWhenTheFrontWindowIsNotShareable() {
83+
let candidates = [Candidate(id: 7, area: 10_000), Candidate(id: 8, area: 100)]
84+
85+
let selected = frontShareableWindow(candidates, front: onScreenWindow(id: 3), windowID: \.id)
86+
87+
XCTAssertThrowsError(try selected.get()) { error in
88+
assertRefusal(
89+
error,
90+
reason: "front-window-not-shareable",
91+
hint: "retry after it settles",
92+
frontWindowId: "3")
93+
}
94+
}
95+
96+
func testCaptureExclusionSurvivesWindowSelectionAndNeverSubstitutesAnotherWindow() throws {
97+
let windows = [windowInfo(id: 3, sharingState: 0), windowInfo(id: 7)]
98+
let front = try frontWindow(pid: 42, windowInfo: windows)
99+
XCTAssertEqual(front?.number, 3)
100+
XCTAssertEqual(front?.sharingState, 0)
101+
102+
for candidates in [[3, 7], [7]] {
103+
let selected = frontShareableWindow(candidates, front: front, windowID: { $0 })
104+
XCTAssertThrowsError(try selected.get()) { error in
105+
assertRefusal(
106+
error,
107+
reason: "front-window-capture-excluded",
108+
hint: "Run `snapshot` to read the window",
109+
frontWindowId: "3")
110+
}
111+
}
112+
113+
var entry = windowInfo(id: 3)
114+
entry.removeValue(forKey: kCGWindowSharingState as String)
115+
let unknown = try frontWindow(pid: 42, windowInfo: [entry])
116+
let selected = frontShareableWindow([3], front: unknown, windowID: { $0 })
117+
XCTAssertEqual(try selected.get(), 3)
118+
}
119+
120+
func testRefusesWhenTheAppHasNoOnScreenWindow() {
121+
let selected = frontShareableWindow(
122+
[Candidate(id: 7, area: 10_000)], front: nil, windowID: \.id)
123+
124+
XCTAssertThrowsError(try selected.get()) { error in
125+
assertRefusal(error, reason: "window-not-found", hint: "Open or unminimize")
126+
}
127+
}
128+
}
129+
130+
/// The refusal carries the reason and the remedy the agent runs next, alongside the front window
131+
/// it refused.
132+
private func assertRefusal(
133+
_ error: Error,
134+
reason: String,
135+
hint: String,
136+
frontWindowId: String? = nil,
137+
file: StaticString = #filePath,
138+
line: UInt = #line
139+
) {
140+
guard let refusal = error as? AppWindowScreenshotRefusal else {
141+
return XCTFail("expected an AppWindowScreenshotRefusal, got \(error)", file: file, line: line)
142+
}
143+
guard
144+
case HelperError.commandFailed(_, let details) = refusal.error(
145+
bundleId: "com.apple.TextEdit")
146+
else {
147+
return XCTFail("expected a commandFailed refusal error", file: file, line: line)
148+
}
149+
XCTAssertEqual(details["reason"], reason, file: file, line: line)
150+
XCTAssertEqual(details["bundleId"], "com.apple.TextEdit", file: file, line: line)
151+
XCTAssertEqual(details["frontWindowId"], frontWindowId, file: file, line: line)
152+
XCTAssertTrue(
153+
(details["hint"] ?? "").contains(hint),
154+
"hint \(details["hint"] ?? "<nil>") lacks \(hint)",
155+
file: file, line: line)
156+
}
157+
158+
private func assertUnreadableList(
159+
_ error: Error, file: StaticString = #filePath, line: UInt = #line
160+
) {
161+
guard case HelperError.commandFailed(_, let details) = error else {
162+
return XCTFail("expected a typed window-list refusal, got \(error)", file: file, line: line)
163+
}
164+
XCTAssertEqual(details["reason"], "window-list-unreadable", file: file, line: line)
165+
XCTAssertEqual(
166+
details["hint"],
167+
"The app's on-screen window list could not be read. Retry the command.",
168+
file: file, line: line)
169+
}
170+
171+
private func onScreenWindow(id: Int) -> OnScreenWindow {
172+
OnScreenWindow(number: id, bounds: .zero, sharingState: 1)
173+
}
174+
175+
private func windowInfo(
176+
id: Int, pid: Int32 = 42, layer: Int = 0, onScreen: Bool = true,
177+
alpha: Double = 1, sharingState: Int = 1, width: Double = 100, height: Double = 100
178+
) -> [String: Any] {
179+
[
180+
kCGWindowNumber as String: id,
181+
kCGWindowOwnerPID as String: pid,
182+
kCGWindowLayer as String: layer,
183+
kCGWindowIsOnscreen as String: onScreen,
184+
kCGWindowAlpha as String: alpha,
185+
kCGWindowSharingState as String: sharingState,
186+
kCGWindowBounds as String: CGRect(x: 10, y: 10, width: width, height: height).dictionaryRepresentation,
187+
]
188+
}

0 commit comments

Comments
 (0)