WinUI: Mouse tracking fails in applications using Windows App SDK 1.6.

[hwf1324]

Steps to reproduce:

In a WinUI application using the Windows App SDK 1.6.
Perform mouse tracking.

Actual behavior:

Does not report what is under the mouse.

Expected behavior:

Report what's under the mouse

NVDA logs, crash dumps and other attachments:

System configuration

NVDA installed/portable/running from source:

install

NVDA version:

alpha-34581,32245d46 x86

Windows version:

Windows 11 22H2 (10.0.22621) workstation AMD64

Name and version of other software in use when reproducing the issue:

WinUI application using Windows App SDK 1.6.
For example, PowerToys v0.86.0
Or create a blank WinUI app in VS 2022

Other information about your system:

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

If NVDA add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Yes

[hwf1324]

Hi, CC: @josephsl , @michaelDCurran , Maybe you guys will have some thoughts on this issue.

It's strange that NVDA considers the external window and pane objects of these apps to be IAccessible, while internally they are normal UIA objects, perhaps this is the cause of the problem?

But I don't know what it's because of, and both the narrator and Accessibility Insights For Windows are able to get the internal controls with the mouse.

[seanbudd]

Has this been reported to microsoft? Have you tested this with the latest release 1.6.3?

It seems like this SDK is new and under active development. Bugs should be reported to the authors primarily.

[hwf1324]

The issue has been reported to the Windows App SDK team, but there has been no response yet.

There are already official versions of some apps that have encountered this problem, such as PowerToys v0.86.0, UniGetUI, and others.

[josephsl]

Hi, also do let Microsoft know that at least PowerToys 0.86 does not expose control labels at all to screen reading technologies. Thanks.

[hwf1324]

Hi, also do let Microsoft know that at least PowerToys 0.86 does not expose control labels at all to screen reading technologies. Thanks.

Hi,

This seems to only appear in the dashboard controls.
I created microsoft/PowerToys#36252.

Thanks!

[hwf1324]

It's strange that NVDA considers the external window and pane objects of these apps to be IAccessible, while internally they are normal UIA objects, perhaps this is the cause of the problem?

But I don't know what it's because of, and both the narrator and Accessibility Insights For Windows are able to get the internal controls with the mouse.

CC: @codeofdusk Can you provide some suggestions?

I'm not sure if this is due to NVDA thinking that the NVDAObjects in the outer pane are IAccessible objects, thus interrupting NVDA from getting the correct UIA objects?

I'd like to try to get NVDA to think that these IAccessible objects are UIA objects and see if that solves the problem. But I'm not sure how to do that.

[codeofdusk]

I'm the primary maintainer of Accessibility Insights for Windows. That app uses the UIA bridge found in Windows 1903 and later. NVDA opts out of this bridge, only using UIA where a native provider is available.

[josephsl]

Hi Bill, thanks for this insight. Would it be possible for you to offer an NVDA PR to let it use UIA bridge in addition to letting it rely on native UIA elements? CC @michaelDCurran on his thoughts.

[codeofdusk]

@josephsl I'm assuming you're looking for a toggle (in advanced settings) to allow use of the bridge to ease testing?

[codeofdusk]

You can do this now with nav.appModule.isGoodUIAWindow = lambda hwnd: True in the Python console

[hwf1324]

You can do this now with nav.appModule.isGoodUIAWindow = lambda hwnd: True in the Python console

This is great! That does get mouse tracking back to work.

[josephsl]

HI, let’s start with that along with a base implementation for UIA bridge in NVDA. Thanks for looking into this.

[hwf1324]

You can do this now with nav.appModule.isGoodUIAWindow = lambda hwnd: True in the Python console

This also seems to work for fixing mouse tracking in Notepad or Windows Terminal.