Starting with macOS 12.2, AltTab only grabs windows in active spaces

[skyzyx]

Describe the bug

• I work with 2 monitors
• I work almost exclusively in full-screen mode

I updated to macOS 12.2 Beta (21D5039d) this morning, and since then, AltTab only grabs the front-most full-screen app window from each of my displays — ignoring the full-screen windows that are not front-most.

I know that the full-screen support is buggy because this app has to use private APIs, etc., but I wanted to bring this situation to the attention of the developers as a known issue.

Usually when AltTab is missing windows, I can quit and re-launch the app to get it to pick up the current state of my environment. However, even this action no longer works as of this morning's macOS beta update.

Screenshots / video

http://s3.ryanparman.com.s3.amazonaws.com/alttab-bug-1.mp4

[dnivi3]

Seeing the same on macOS 12.2 (21D49), the production version of macOS 12.2.

[kaatt]

likely related to kasper/phoenix#289 and ianyh/Amethyst#1192

[dnivi3]

Looks like Amethyst has fixed this in ianyh/Amethyst#1184

[lwouis]

@dnivi3 the code change you linked to is this:

Changing the order remove -> add to add -> remove. AltTab already does add -> remove:

[dnivi3]

@lwouis yeah, I realised my layman non-technical understanding of this isn't helpful. Is there any I can help or contribute towards this getting fixed?

[lwouis]

@dnivi3 i'm afraid only code contributions would help. Please see #1179

[lwouis]

Related: #1351

[stevetodd]

Just to add the datapoint, the issue persists in the newly released macOS 12.3.

[radaczynski]

This behaviour is repeatable. I have 3 desktops. When restarting alt-tab, windows from the active one get detected correctly, the switcher works just as expected, while on the other desktops only the active window is shown in the switcher. Restarting the apps/reopening the windows causes them to be shown in the switcher. Macos 12.3.1 here.

[kaatt]

Allowing screen recording permissions to the app should fix this.

[radaczynski]

@kaatt - were you refering to my comment? It does not fix this issue - AltTab had the permission to record screen all the time. Windows from the current desktop are detected just fine after restart of AltTab, but on other desktops only one window is detected (+ all the windows that are created after AltTab is launched).

[lwouis]

It seems that we could simply replace the call to CGSAddWindowsToSpaces with a call to CGSMoveWindowsToManagedSpace.

Can anyone confirm this would work?

[lwouis]

I experimented with replacing CGSAddWindowsToSpaces with CGSMoveWindowsToManagedSpace. It's not a 1-to-1 replacement. It doesn't move fullscreen windows. It successfully moves regular windows, and moves them back, but it doesn't do anything for fullscreen windows, thus AltTab doesn't see them.

[dnivi3]

@lwouis glad to hear it solves for regular windows, but shame with fullscreen ones. Selfishly, this solves for my use cases. Is there a branch I can build from to test this?

[lwouis]

Is there a branch I can build from to test this?

i'm sorry, there isn't

[lwouis]

It seems that 1Piece is able to do it! I hope we can find out how they do it

[lwouis]

They use the same private method we do:

They also use CGWindowListCreateImageFromArray

I'm struggling to find the place where they somehow grab the AXRef

[decodism]

They seem to get the focused window of each application. I know a trick to get all the windows.

[lwouis]

@decodism Oh! Could you please share more about this process?

[decodism]

The idea is simply to bruteforce the id of the elements, as it is sequential. It's fast enough for a small maxId.

func getWindowElements(_ pid: pid_t, _ maxId: Int) -> [AXUIElement] {
    var elements: [AXUIElement] = []
    for id in 0...maxId {
        var token = Data()
        token.append(contentsOf: withUnsafeBytes(of: pid) { Data($0) })
        token.append(contentsOf: withUnsafeBytes(of: Int32(0)) { Data($0) })
        token.append(contentsOf: withUnsafeBytes(of: Int32(0x636f636f)) { Data($0) })
        token.append(contentsOf: withUnsafeBytes(of: id) { Data($0) })
        var role: CFTypeRef?
        if
            let element = _AXUIElementCreateWithRemoteToken(token as CFData)?.takeUnretainedValue(),
            AXUIElementCopyAttributeValue(element, kAXRoleAttribute as CFString, &role).rawValue == 0,
            role as? String == kAXWindowRole
        {
            elements.append(element)
        }
    }
    return elements
}

[koekeishiya]

If you can just use the pid and windowid to construct the AXUIElementRef on your own like in the above code, you do not need to brute-force the ids. You can get all window ids across all spaces, together with their associated pid from unrestricted private APIs.

int g_connection = SLSMainConnectionID();
int cid = 0;

uint64_t *space_list = ..; 
int space_count = ..;

uint64_t set_tags = 0;
uint64_t clear_tags = 0;
uint32_t options = include_minimized ? 0x7 : 0x2;

CFArrayRef space_list_ref = cfarray_of_cfnumbers(space_list, sizeof(uint64_t), space_count, kCFNumberSInt64Type);
CFArrayRef window_list_ref = SLSCopyWindowsWithOptionsAndTags(g_connection, cid, space_list_ref, options, &set_tags, &clear_tags);

CFTypeRef query = SLSWindowQueryWindows(g_connection, window_list_ref, *count);
CFTypeRef iterator = SLSWindowQueryResultCopyWindows(query);

while (SLSWindowIteratorAdvance(iterator)) {
    pid_t pid = SLSWindowIteratorGetPID(iterator);
    uint32_t wid = SLSWindowIteratorGetWindowID(iterator);

   // construct your own AXUIElement using this info..
}

Of course brute-forcing might as well be the simpler solution depending on how high the values can become.
The windowid is a 32bit unsigned int, but no idea how high they actually go in real usage.

[decodism]

Actually, it's the element id, not the window id, that is needed.

[koekeishiya]

Aight I see, that makes sense yeah; which is why we need to translate the element to the actual window id using a function call instead of just reading the AXUIElementRef data; which we can do to get the pid.

[lwouis]

Thanks a lot @decodism and @koekeishiya!

I've played around with these AXUIElement IDs. They seem to be scoped per process. That is, a process start at 0, and the ID increments with each new element. For instance:

• TextEdit has 1 window
• I brute force this array [10] (10 is the ID of the AXUIElement of that window)
• I create a new window. Array is now [10, 11]
• I remove the window. Array is now [10]
• I create a new window. Array is now [10, 12]

When I looked at Finder, the windows were already at 175. I imagine that for long-running processes, the number can go quite high.

That being said, most people eventually restart their mac, or close their apps. So probably brute forcing the first 1K or 10K IDs may do the trick for most users.

Do you have any better idea on how to refine this approach?

Thank you 🙇

[lwouis]

To everyone following this thread:

Could you please try out this local build? It uses the new trick. On my machine, it fixes this ticket! Could you please let me know if it works for you, and if launch performance is ok?

Thank you!

[metya]

HI!
I've tried the new build and it doesn't work for me. But my problem is slightly different. I described my problem here #2257
1piece perfectly works with these little arc windows in Arc Browser, but the alt-tab is still missing it.
I think that the part of the same problem. Apparently, it does not work with this trick.
However, the app's performance does not vary from the original build.

[dnivi3]

I tried the new build and it works - it now grabs windows from all spaces! Launch performance seems good to me, but I did not measure it in any scientific way or anything.

[lwouis]

Hi,

I found some cases like launching Notes.app where it would miss windows with the new approach. I decided to combine the previous and new approaches.

Could you please try out this local build and let me know?

Thank you 🙇

[koekeishiya]

I implemented a version of this in yabai now. Thanks @decodism; I credited you for the solution.
koekeishiya/yabai@15995ae#diff-13af305c64614b8ece267ef0b496495abcadf8e11ca80df678e64d5abf792b08R1649

I used a combination of the private API I mentioned above to check the window-ids that actually exists.
Then I use the regular MacOS API to get the windows that we can, using the "proper" method.

I then diff the lists to get a list of the window-ids that are missing, and I use the element-id brute-force approach to build the rest of the elements.

Instead of checking the AXRole of the element, I just retrieve the ID of the element using the _AXUIElementGetWindow function, and make sure that the element-id matches one that I expect (from the previous diff).

When all the missing window-ids have been matched I break the brute-force loop and all windows are successfully resolved.
This approach lets selectively brute-force only applications that have windows on inactive spaces.

[decodism]

I see two issues:

• _AXUIElementGetWindow also works for non-window elements It shouldn't be an issue
• I don't think we can be sure that an element exists for a given window id?

Another small potential optimization would be to sort the window ids of a process, find the first window id whose element is not known, take the element of the previous window id, and start bruteforcing from the id of that element + 1.

[lwouis]

Thank you for sharing these insights!

At the moment, I have implement a basic brute-force from 0 to 1000. It takes around 15-20ms on my M2 macbook pro. I think it's reasonably fast given that we do this brute-forcing in a background thread, in an async way. The user will see these windows appear in the switcher, as we discover them.

I'm interested in reducing the compute of course. I've profiled it, and it all goes into the AX calls we make to the OS. For instance to get the subrole, or to get the window ID.

Indeed, it would be nice to limit these AX calls. I think we can use CGWindowListCopyWindowInfo to get a list of all windows across all Spaces and situations. We can filter those by PID, and sort them. Now we have a min CGWindowID and a max CGWindowID.

I think we can then try to reduce this range by checking on both sides if we already known the windows. Once we have unknown windows on both sides, we can't reduce the range further.

An issue I have then is: how do we get the AXUIElement ID from AXUIElement? Do you know how to do this @decodism?

Thank you

[koekeishiya]

@lwouis The element_id is stored inside a CFData which is referenced by the AXUIElementRef.

pid_t = 0x0158ef, element_id = hex 0x2c, decimal 44, window_id = hex 0x30ec, decimal 12524
  0000  59 8d e0 f0 01 00 00 01 80 59 00 20 66 e0 01 00  Y........Y. f...
  0010  ef 58 01 00 00 00 00 00 6f 63 6f 63 00 00 00 00  .X......ococ....
  0020  90 9f 07 01 00 60 00 00 00 00 00 00 00 00 00 00  .....`..........
<CFData 0x600001079f90 [0x1f5f9c8c0]>{length = 8, capacity = 8, bytes = 0x2c00000000000000}
read element_id: 44

Code in C:

static inline CFDataRef ax_window_eid(AXUIElementRef ref)
{
    return *(CFDataRef *)((void *) ref + 0x20);
}

CFDataRef e_data = ax_window_eid(element_ref);
CFShow(e_data);

printf("read element_id: %lld\n", *(uint64_t*)CFDataGetBytePtr(e_data));

Collapsed:

static inline uint64_t ax_window_eid(AXUIElementRef ref)
{
    return *(uint64_t *) CFDataGetBytePtr(*(CFDataRef *)((void *) ref + 0x20));
}

[lwouis]

Thank you @koekeishiya! I'll try this!

[lwouis]

I completely forgot that CGWindowListCopyWindowInfo and CGSCopyWindowsWithOptionsAndTags return lots of non-windows. I've been using AXUIElements for many years now to filter out non-windows. I forgot that the official public APIs return nonsense, and that there is no way to filter it.

Thus I think the optimisation to reduce brute-force is not possible. We simply can't know which windows are missing before we brute force:

• CoreGraphics APIs return unfilterable elements. We can't rely on their listings
• AX APIs don't return other-Spaces windows, which is the whole point of the brute-force workaround

I think I'll just stick with brute-forcing 0 to 1000 for now.

@koekeishiya you mentioned that your approach uses SLSCopyWindowsWithOptionsAndTags. How do you make anything useful from calling this method? It returns hundreds of invisible, nonsense UI elements. Have you found a way to filter those out?

[koekeishiya]

I made a filtering function based on various window properties that I discovered by sampling a ton of window data, and it has been in use in yabai for 1year. The core part has been in use for several years at this point, but the edit 1year ago fixed something with hidden windows I believe it was. The filtering is complex enough that I won't try to explain it here, but you can find the function I use to grab window-ids here: https://github.com/koekeishiya/yabai/blob/master/src/space.c#L17

Before adding the brute-force method discussed in this issue, I used this primarily to grab a count of how many windows I expected to be returned by the AX API, and I had a setup to track the applications that had yet to be resolved, and I would attempt to resolve these specific applications upon a space switch event from macOS, until the applications all had matched their window count. As far as I can tell this has been working properly for yabai users for a long time; I don't recall anyone reporting missing windows since that change was implemented. (I also used this information when the user is querying windows, to output information about them even though yabai did not have an AX-reference, so users relying on yabai to list their windows would not wonder why some widnows were completely missing).

I may just have been lucky of course; I cannot prove without a doubt that this solution is flawless.

[decodism]

These functions can also be used (not quite sure of the signatures):

AXError _AXUIElementGetData(AXUIElementRef element, CFDataRef *data, int *type);
AXUIElementRef _AXUIElementCreateWithDataAndPid(CFDataRef data, int type, pid_t pid, pid_t presenterPid);

[koekeishiya]

The brute force method doesn't actually work after a clean reboot. There appears to be some state that is missing. After interacting with windows of applications, the brute force method does begin to work for those applications. I don't know exactly what is going on and have not had time to go deeper into it. However, that means the code needs to assume that it will fail and be failsafe. I will change how I use it in yabai to combine it with the approach I used before, instead of replacing what i did before.

[lwouis]

Thank you for sharing this @koekeishiya! It's surprising. For this project, it should be okay since we either grab windows with the method, or we don't. We don't keep an exact count of which windows are known and which have an AXRef. We can get away with that for now, I think.

By the way, I noticed that you've changed the way you focus/make_key the windows in yabai:

I've been using your your technique for years now. Recently, people have complained about focus issues with some apps (#4192). Is that what you fixed in yabai? I see many follow-up commits. Are you satisfied with the current implementation?

If it's working well, I'll update it in AltTab. Hopefully, it will improve focus and fix #4192!

Thank you 🙇

[koekeishiya]

I haven't heard anything about specific apps refusing to become focused by yabai in the issue tracker, but I have to admit I don't use the applications mentioned in #4192 myself, so cannot say for sure. I would assume that I would get reports by yabai users if it didn't work though.