UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 20 Sep 2022, 10:22

@jekko1976, with UIA you will need to first expand the combobox and then select the correct item. For example, I pasted your html into this online editor and then this code worked:

Code: Select all

#include <UIA_Interface>

UIA := UIA_Interface()
browserId := "ahk_exe chrome.exe"
WinActivate, %browserId%
WinWaitActive, %browserId%
cbEl := UIA.ElementFromHandle(browserId).FindFirst("Type=ComboBox")
if (cbEl.Value != "25") {
    cbEl.ExpandCollapsePattern.Expand()
    cbEl.WaitElementExist("Name=25").Click()
}

jekko1976
Posts: 97
Joined: 10 Oct 2014, 07:03

Re: UIAutomation with a focus on Chrome

Post by jekko1976 » 21 Sep 2022, 07:27

@Descolada
Hi Descolada and thank you so much.
I don't know why, but your snippet injected to my script "as is" wasn't working, so I tried other solutions and this one seems to work:

Code: Select all

UIA := UIA_Interface()
browserId := "ahk_exe chrome.exe"
WinActivate, %browserId%
WinWaitActive, %browserId%
cbEl := UIA.ElementFromHandle(browserId).WaitElementExistByName("Visualizza elementi")
cbEl.ExpandCollapsePattern.Expand()
cbEl.WaitElementExistByNameAndType("100","ListItem").click()
Thank you for the help with this lifesaver "ExpandCollapsePattern" method!

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 21 Sep 2022, 16:26

Any idea how to get npEl := ElementFromHandleBuild() to work with ahk_class notepad++ ? It looks like if I do a dump of the resulting object, I get ALL of the text in any visible files I have open within Notepad++ plus the regular tree of nodes. When I try to search the npEl with FindFirstBy("Name=Close AND ControlType=button") it never seems to return a valid result even though I can see the button in the tree dump?

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 22 Sep 2022, 07:33

@Skrell, as far as I know, ElementFromHandleBuild doesn't exist in UIA. Either you mean ElementFromHandle or ElementFromHandleBuildCache. The following code found the Close button and closed Notepad++:

Code: Select all

#include <UIA_Interface>
UIA := UIA_Interface()
WinActivate, ahk_exe notepad++.exe
WinWaitActive, ahk_exe notepad++.exe
npEl := UIA.ElementFromHandle("ahk_exe notepad++.exe")
npEl.FindFirstBy("Name=Close AND ControlType=button").Highlight().Click()
ExitApp
Though perhaps I didn't quite understand your goal - are you trying to close a specific tab? In that case it seems that trying to Invoke/Click the small X button threw an error, but using LegacyIAccessible worked: npEl.FindFirstBy("Name=X AND ControlType=MenuItem").Highlight().LegacyIAccessiblePattern.DoDefaultAction() closed the current tab.
Or maybe you could send a screenshot of how your Notepad++ is set up so I can test it?

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 22 Sep 2022, 07:57

Descolada wrote:
22 Sep 2022, 07:33
@Skrell, as far as I know, ElementFromHandleBuild doesn't exist in UIA. Either you mean ElementFromHandle or ElementFromHandleBuildCache. The following code found the Close button and closed Notepad++:

Code: Select all

#include <UIA_Interface>
UIA := UIA_Interface()
WinActivate, ahk_exe notepad++.exe
WinWaitActive, ahk_exe notepad++.exe
npEl := UIA.ElementFromHandle("ahk_exe notepad++.exe")
npEl.FindFirstBy("Name=Close AND ControlType=button").Highlight().Click()
ExitApp
Though perhaps I didn't quite understand your goal - are you trying to close a specific tab? In that case it seems that trying to Invoke/Click the small X button threw an error, but using LegacyIAccessible worked: npEl.FindFirstBy("Name=X AND ControlType=MenuItem").Highlight().LegacyIAccessiblePattern.DoDefaultAction() closed the current tab.
Or maybe you could send a screenshot of how your Notepad++ is set up so I can test it?
My apologies for the typo I mean ElementFromHandle. The code you wrote is exactly what I'm doing... I wonder if though when you tested it you had no text in any tabs. I'm simply trying to access the min/max/close buttons in the frame of notepad++ btw. If you type any text into a tab, I THINK you'll find that it also gets built into the npEl object but outside the tree. Does your code still work in this case?

Below is an example of what happens if I write npEl.DumpAll() to a log file.
image.png
image.png (160.37 KiB) Viewed 1979 times

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 23 Sep 2022, 02:53

@Skrell, the UIA tree for Notepad++ contains Pane elements for which the CurrentName property contains the Pane content itself. Why Notepad++ has implemented this in such a way (instead of using the Value property), I don't know. But in any case this shouldn't interfere with finding elements unless you are using matchmode 2, in which case the search might match erroneously the Pane element if you are not careful.
But to answer your question, I do get the same result as you in the DumpAll output, but my provided code still works. Although clicking the small X button to close a code tab doesn't, and I have no idea why (some error in Notepad++ implementation for UIA perhaps?), even the LegacyIAccessible method didn't work with in split view mode (two panes open side by side as in your screenshot), only clicking with Click("left") worked reliably. Though the best way to click it isn't with UIA: WinMenuSelectItem, ahk_exe notepad++.exe, , X

jekko1976
Posts: 97
Joined: 10 Oct 2014, 07:03

Re: UIAutomation with a focus on Chrome

Post by jekko1976 » 23 Sep 2022, 10:36

I haven't really clear how "treewalker" works.
I think that it would be fantastic if a "treewalker" example will be added to the existing ones.
I didn't find it.
BTW: I think that UIA is a really awesome library!
Thank you

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 23 Sep 2022, 11:07

@jekko1976, thanks for the kind words. About the TreeWalker - have you taken a look at the Wiki?

jekko1976
Posts: 97
Joined: 10 Oct 2014, 07:03

Re: UIAutomation with a focus on Chrome

Post by jekko1976 » 23 Sep 2022, 14:24

@Descolada
oh...thank you! In the wiki I found the findbypath too that should do the trick in an easier way :)

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 25 Sep 2022, 11:05

I've looked through the documentation, but can't seem to figure out how to do the following:

Can someone please post an example of using FindAllBy and then iterating through the results looking for a GetCurrentPos of a specific dimension?

In a related post, can anyone figure out why no tooltip EVER prints after "tooltip, trying.... xxxxx" ? Almost like it's hung?

#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn ; Enable warnings to assist with detecting common errors.
SendMode Input ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.

Code: Select all

#Include %A_ScriptDir%\WinHook.ahk 
#include %A_ScriptDir%\UIA_Interface.ahk
lastGoodHwnd        := ""
lastGoodCapture     := ""
lastGoodExe         := ""

UIA := UIA_Interface()

WinHook.Shell.Add("Created",,, ,1) 
Return

Created(Win_Hwnd, Win_Title, Win_Class, Win_Exe, Win_Event)
{
   global lastGoodCapture, lastGoodExe, lastGoodHwnd
   WinGetTitle, currentTitle, ahk_id %Win_Hwnd%
   tooltip, %currentTitle%
   If (currentTitle != "")
   {
       WinGetClass, currentClass, ahk_id %Win_Hwnd%
       ; WinGet, currentHwnd, ID, ahk_id %Win_Hwnd%
       WinGet, currentExe, ProcessName, ahk_id %Win_Hwnd%
       ; tooltip, % currentClass "-" currentExe "-" lastGoodCapture
       If ( WinExist(ahk_id %Win_Hwnd%)
            &&  (currentClass != "tooltips_class32")
            &&  (currentClass != "ApplicationManager_DesktopShellWindow" )
            &&  (currentClass != "TaskListThumbnailWnd" )
            &&  (currentClass != "MSO_BORDEREFFECT_WINDOW_CLASS" )
            &&  (currentClass != "MultitaskingViewFrame"         )
            &&  (currentClass != "#32768"                        )
            &&  (currentClass != "#32770"                        )
            &&  (currentClass != "Shell_TrayWnd")
            &&  (currentClass != "WorkerW"      ))
        {
            lastGoodHwnd    := Format("{:#x}", Win_Hwnd)
            lastGoodCapture := currentClass
            lastGoodExe     := currentExe
            GoSub, ButCaptureCached
        }
   }
}

ButCaptureCached:
    Critical
    sleep 200
   If (!PrintButton && lastGoodExe && lastGoodCapture)
    {
        mWinIdbc2 = ahk_id %lastGoodHwnd%
        try {
            prevCached := False
            
            for shwnds, c in scannedAhkIds
            {
                If (mWinIdbc2 == shwnds)
                {
                    prevCached := True
                    break
                }
            }
            ; tooltip, 00
            If !prevCached
            {
            }
            Else
            {
                Tooltip, exiting!
                Return
            }
           ; tooltip, 0
            If (true)
            {
                tooltip, trying.... %mWinIdbc2% %lastGoodCapture% %lastGoodExe%
                
                If !prevCached
                    npEl := UIA.ElementFromHandle(mWinIdbc2) ; Get element and also build the cache
                
                regexMin := "Name=Minimize AND ControlType=button AND ClassName=NetUIAppFrameHelper)"
                regexMax := "Name=Maximize AND ControlType=button AND ClassName=NetUIAppFrameHelper)"
                regexClo := "Name=Close AND (ControlType=button OR ControlType=ListItem) AND ClassName=NetUIAppFrameHelper"
                minimizeElAr := npEl.FindAllBy(regexMin, 0x4, 2, False)
                maximizeElAr := npEl.FindAllBy(regexMax, 0x4, 2, False)
                closeElAr    := npEl.FindAllBy(regexClo, 0x4, 2, False)
                for result in minimizeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        minimizeEl := result
                        break
                    }
                }
                for result in maximizeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        maximizeEl := result
                        break
                    }
                }
                for result in closeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        closeEl := result
                        break
                    }
                }
                tooltip, done1!
            }
            ; tooltip, 1
            
            If (!minimizeEl && !maximizeEl && !closeEl)
            {
                regexMin := "Name=Minimize AND ControlType=button AND (AutomationId=Minimize OR AutomationId=view_2)"
                regexMax := "Name=Maximize AND ControlType=button AND (AutomationId=Maximize OR AutomationId=view_2)"
                regexClo := "Name=Close AND (ControlType=button OR ControlType=ListItem) AND (AutomationId=Close OR AutomationId=view_2)"
                minimizeElAr := npEl.FindAllBy(regexMin, 0x4, 2, False)
                maximizeElAr := npEl.FindAllBy(regexMax, 0x4, 2, False)
                closeElAr    := npEl.FindAllBy(regexClo, 0x4, 2, False)
                
                for result in minimizeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        minimizeEl := result
                        break
                    }
                }
                for result in maximizeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        maximizeEl := result
                        break
                    }
                }
                for result in closeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        closeEl := result
                        break
                    }
                }
                tooltip, done2!
            }
            Else If (!minimizeEl && !maximizeEl && !closeEl)
            {
                regexMin := "Name=Minimize AND ControlType=button"
                regexMax := "Name=Maximize AND ControlType=button"
                regexClo := "Name=Close AND (ControlType=button OR ControlType=ListItem)"
                minimizeElAr := npEl.FindAllBy(regexMin, 0x4, 2, False)
                maximizeElAr := npEl.FindAllBy(regexMax, 0x4, 2, False)
                closeElAr    := npEl.FindAllBy(regexClo, 0x4, 2, False)
                for result in minimizeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        minimizeEl := result
                        break
                    }
                }
                for result in maximizeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        maximizeEl := result
                        break
                    }
                }
                for result in closeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        closeEl := result
                        break
                    }
                }
                tooltip, done3!
            }
            ; tooltip, 2
            If (!minimizeEl && !maximizeEl && !closeEl)
            {
                tooltip, dammit! %lastGoodCapture% %lastGoodExe%
                FileAppend, % npEl.DumpAll() "`n", C:\Users\vmb11\Desktop\log2.txt 
                scannedAhkIds.remove(mWinIdbc2)
            }
            Else
            {
                minimizePos := minimizeEl.GetCurrentPos()
                maximizePos := maximizeEl.GetCurrentPos()
                closePos    := closeEl.GetCurrentPos()
                
                minX        := minimizePos.x
                minXW       := minimizePos.x+minimizePos.w
                minY        := minimizePos.y
                minYH       := minimizePos.y+minimizePos.h
                
                maxX        := maximizePos.x
                maxXW       := maximizePos.x+maximizePos.w
                maxY        := maximizePos.y
                maxYH       := maximizePos.y+maximizePos.h
                
                closeX      := closePos.x
                closeXW     := closePos.x+closePos.w
                closeY      := closePos.y
                closeYH     := closePos.y+closePos.h
                
                tooltip, % minX "-" minXW "-" minY "-" minYH 
                scannedAhkIds[mWinIdbc2] := 1
                
                Array := {"X": minX, "XW": minXW, "Y": minY, "YH": minYH}
                minDimsAhkId[mWinIdbc2] := Array
                Array := {"X": maxX, "XW": maxXW, "Y": maxY, "YH": maxYH}
                maxDimsAhkId[mWinIdbc2] := Array
                Array := {"X": closeX, "XW": closeXW, "Y": closeY, "YH": closeYH}
                closeDimsAhkId[mWinIdbc2] := Array
                
                minimizeEl   := {}
                maximizeEl   := {}
                closeEl      := {}
                tooltip, stored!
            }
            
        } catch e {
        
        }
    }
Return

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 25 Sep 2022, 23:21

@Skrell, I'm not exactly sure what your provided code should be doing, but I can say that you are using FindAllBy correctly and iterating through the results properly with GetCurrentPos as well. Perhaps you could provide a simpler example?
If your code gets stuck, then try adding some debugging lines (MsgBox, OutputDebug, or write to a log file) to see exactly where (or use a debugger) it gets stuck.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 26 Sep 2022, 08:17

Descolada wrote:
25 Sep 2022, 23:21
@Skrell, I'm not exactly sure what your provided code should be doing, but I can say that you are using FindAllBy correctly and iterating through the results properly with GetCurrentPos as well. Perhaps you could provide a simpler example?
If your code gets stuck, then try adding some debugging lines (MsgBox, OutputDebug, or write to a log file) to see exactly where (or use a debugger) it gets stuck.
Glad to know I'm at least on the right track! I'm starting to wondering if there could be a security setting or update in Windows 10x64 that could actively be messing with your UIA library? Bc I tried this script on another PC and it seemed to be fine.... Any thoughts on how I could configure something in windows to negatively impact UIA?

PS:
My for loops are NOT correct:
They need to be this for some reason:

Code: Select all

for idx, result in minimizeElAr
                {
                    If (minimizeElAr[idx].GetCurrentPos().w > 10)
                    {
                        minimizeEl := result
                        break
                    }
                }

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 26 Sep 2022, 09:06

@Skrell, first of all thank you for helping me find a bug in UIA_Interface: namely ElementFromHandle silently threw an error when a ControlGet call failed to find a Chromium element. This is why your tooltip got stuck: actually an error was thrown and everything after ElementFromHandle was skipped. You can fix this by updating UIA_Interface.ahk or disabling the activateChromiumAccessibility flag: npEl := UIA.ElementFromHandle(mWinIdbc2, False)

Another thing caught my eye:

Code: Select all

                regexMax := "Name=Maximize AND ControlType=button AND ClassName=NetUIAppFrameHelper)"
                regexClo := "Name=Close AND (ControlType=button OR ControlType=ListItem) AND ClassName=NetUIAppFrameHelper"
Should ClassName be "NetUIAppFrameHelper" or "NetUIAppFrameHelper)" with a parenthesis at the end?

There shouldn't be any differences between the latest Window updates, though the element names and classes may change when programs are updated. What might mess you up is GetCurrentPos(): namely by default it uses the current CoordMode, and by default that is Window. So if you are using GetCurrentPos() on an unfocused window then it might give different results. You can use GetCurrentPos("Window") to always use the same CoordMode.

Edit: And yes, your loops weren't correct. I forgot that for some reason AHK v1 defaults to the key, not the value in a for loop. So this works:

Code: Select all

                for _, result in minimizeElAr
                {
                    If (result.GetCurrentPos().w > 10)
                    {
                        minimizeEl := result
                        break
                    }
                }
The "_" is just a dummy variable to ignore the index.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 26 Sep 2022, 09:24

@Descolada feels like my last series of stupid questions did some good! :) (missed that ")" typo!!!! )
Next stupid question, can you please discuss a bit in terms of speed the difference between SmallestElementFromPoint and ElementFromPoint? I've read the descriptions for SmallestElementFromPoint several times but don't really understand what it means vs what ElementFromPoint does ?

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 26 Sep 2022, 10:27

@Skrell, ElementFromPoint is the original Microsoft implementation, but it has the drawback of sometimes not returning the actual smallest element at the point. For example it might return a container element (Pane, Group etc) instead, while there might be a CheckBox or Button under the point (which are contained in the container element). The same issue is described here: https://arstechnica.com/civis/viewtopic.php?f=20&t=1467202.
SmallestElementFromPoint is a workaround for this issue: it first uses ElementFromPoint, and then drills down until it finds the actual smallest element at that point. This is what UIAViewer uses internally.
SmallestElementFromPoint also has the windowEl argument, which if provided checks the whole window for elements other than ElementFromPoint element that contain the point. This is what UIAViewer uses if the "Deep search (slower)" checkbox is checked. To see why this is needed, try to inspect Chrome's min-max buttons: without "Deep search" you cannot find them (because ElementFromPoint returns an element that doesn't contain those buttons...), but with "Deep search" they are found.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 27 Sep 2022, 07:22

How is it possible that sometimes I can have UIAViewer detect correctly a caption button, but then nothing shows up in the tree window? And then, if I push the "Construct tree for whole window" the button doesn't show up at ALL anywhere in the tree?
image.png
image.png (49.08 KiB) Viewed 1630 times

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 27 Sep 2022, 10:31

@Skrell, I've had that happen when the window that contains the element is destroyed when clicking away from the window, for example menus that close when focus goes away. Try hovering over the element and then constructing the tree with shortcuts: first press F1, then F2. If that doesn't work then perhaps you could share the program that is causing it, I'll take a look then.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 27 Sep 2022, 13:07

Descolada wrote:
27 Sep 2022, 10:31
@Skrell, I've had that happen when the window that contains the element is destroyed when clicking away from the window, for example menus that close when focus goes away. Try hovering over the element and then constructing the tree with shortcuts: first press F1, then F2. If that doesn't work then perhaps you could share the program that is causing it, I'll take a look then.
So F2 did work and it populated the tree, but the min/max/close buttons were no where in the tree. I even tried doing a deep search and still nothing. So ElementFromPoint is doing something different than whatever getting the whole tree does. The program in question is Nitro Pro. Hopefully you can try like a trial version or something as it isn't a free pdf reader.

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 27 Sep 2022, 14:15

@Skrell, UIAViewer isn't doing anything differently, it's just ElementFromPoint gets the proper element, but the same element is not contained in the window element... Also the min/max/close buttons don't have a parent element (which should be impossible, since every element should be a descendant of the root element), so I think this is an UIA implementation problem in Nitro Pro, or a bug in Microsofts' UIA itself.
Though I don't really understand why you even need to find the min/max/close buttons: AHK functions (WinClose etc) should work fine, and to detect if the user is hovering over one of those buttons you can still use ElementFromPoint (or just calculate relative coordinates of the mouse to the upper right corner of the window to figure out which button is being hovered).

Loop
Posts: 155
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 28 Sep 2022, 16:48

Hi,
What does this error message mean
I am on a web page, click on buttons, then I scroll a bit and new buttons appear and then this error message appears


---------------------------
Test.ahk
---------------------------
Error in #include file "C:\PORT\ahk\Lib\UIA_Interface.ahk":
0x80131509 - UIA_E_INVALIDOPERATION

Specifically: DoDefaultAction (UIA_LegacyIAccessiblePattern)

Line#
1268: {
1269: Sleep,%sleepTime%
1270: Return,1
1271: }
1272: }
1273: if (this.GetCurrentPropertyValue(UIA_Enum.UIA_IsLegacyIAccessiblePatternAvailablePropertyId))
1273: {
---> 1274: this.GetCurrentPatternAs("LegacyIAccessible").DoDefaultAction()
1275: Sleep,%sleepTime%
1276: Return,1
1277: }
1278: Return,0
1279: }
1279: Else
1279:


Thanks

Post Reply

Return to “Scripts and Functions (v1)”