UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 30 Jan 2023, 20:42

Descolada wrote:
30 Jan 2023, 00:26
@mora145, I guess if FindByPath isn't working out at all, then you could use as a workaround hello.FindByPath("p.10")? Or something like this might work as well:

Code: Select all

TextTW := cUIA.CreateTreeWalker("Type=Text")
MsgBox, % TextTW.GetLastChildElement(hello.FindByPath("p")).Dump()
I make that

Code: Select all

phoneButton:="דברו איתי"
otherText := otherText.GetCurrentPropertyValueEx("Name")
        
                    Loop, 10 
                        {            
                        walker := A_Index + 7
                        otherText := father.FindByPath(walker)
                        otherText := otherText.GetCurrentPropertyValueEx("Name")
        
                            if (otherText=phoneButton)
                                {
                                    otherText := father.FindByPath(walker+2)
                                    otherText := otherText.GetCurrentPropertyValueEx("Name")
                                    break
                                }
                        }

TheBeginner
Posts: 91
Joined: 19 Oct 2018, 12:05

Re: UIAutomation with a focus on Chrome

Post by TheBeginner » 31 Jan 2023, 01:59

Hi Descolada,, you've done an amazing work on this, something the community was really missing. and I also appreciate the time you invested in documenting everything in the wiki

I do have a question I haven't been able to find an answer for
if I have an chrome interface element that I can access and invoke through the UIAViewer, but it does not appear in the dump all
how do I interact with it?
image.png
image.png (130.07 KiB) Viewed 3195 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 31 Jan 2023, 02:15

@TheBeginner, if you stop capturing and then build the tree for the whole window, are you able to find the element for the Image or the Group which has Invoke method available?
It would help if you provided the result of DumpAll (perhaps upload it to pastebin or something).

TheBeginner
Posts: 91
Joined: 19 Oct 2018, 12:05

Re: UIAutomation with a focus on Chrome

Post by TheBeginner » 31 Jan 2023, 03:00

this is what it captures when I use build the tree out of the window, it's the same thing that dump all fines
but it is different then what hovering over the object fines

you can see that the tree structure when hovering (previous image), and dumping gives different results (this one)
image.png
image.png (132.31 KiB) Viewed 3154 times
here is a dump of that section from dump all

Code: Select all

291.1 Type: 50006 (Image) LocalizedControlType: "image"
292 Type: 50002 (CheckBox) Name: "Clip content" LocalizedControlType: "check box" AutomationId: "frame-mask-disabled-checkbox"
293 Type: 50006 (Image) LocalizedControlType: "image"
294 Type: 50020 (Text) Name: "Auto layout" LocalizedControlType: "text"
295 Type: 50000 (Button) Name: "Remove" LocalizedControlType: "button"
295.1 Type: 50006 (Image) LocalizedControlType: "image"
296 Type: 50004 (Edit) LocalizedControlType: "edit"
297 Type: 50006 (Image) LocalizedControlType: "image"
298 Type: 50006 (Image) LocalizedControlType: "image" <====== this 
299 Type: 50026 (Group) LocalizedControlType: "group"
299.1 Type: 50006 (Image) LocalizedControlType: "image"
300 Type: 50000 (Button) Name: "Advanced layout settings" LocalizedControlType: "button"
300.1 Type: 50006 (Image) LocalizedControlType: "image"
301 Type: 50006 (Image) LocalizedControlType: "image"
302 Type: 50004 (Edit) Name: "Spacing between items" Value: "8" LocalizedControlType: "edit"
303 Type: 50006 (Image) LocalizedControlType: "image"

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 31 Jan 2023, 06:08

@TheBeginner, yes unfortunately due to UIAutomation implementation quirks which I have no control over, starting with ElementFromPoint and going upwards in the tree (which UIAViewer capturing starts) results in a different layout than starting from ElementFromHandle and going downwards (which "Construct the tree", DumpAll, and Find methods use). This inconsistency is fixed in UIA v2 (for AHK v2)
I think your best bet to access it in this case is FindFirst the "Auto layout" element and then travel to the correct image:

Code: Select all

layoutEl := element.FindFirstByNameAndType("Auto layout", "Text")
layoutEl.FindByPath("+3", "Type=Image").Highlight().Click() ; if this doesn't work, try "+2" instead.

TheBeginner
Posts: 91
Joined: 19 Oct 2018, 12:05

Re: UIAutomation with a focus on Chrome

Post by TheBeginner » 01 Feb 2023, 07:19

@Descolada, thanks
the approach with the path works, I still don't understand how the progression in the path works (how you count objects along the path, as a sub-object counted? skipped?)
nevertheless it works 👍

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 01 Feb 2023, 10:25

@TheBeginner, FindByPath uses a TreeWalker to walk the UIA tree, and if you use "+n" it will use GetNextSiblingElement. If no condition is provided then the True condition is used and it will return the next sibling that is displayed in UIAViewer or DumpAll as the sibling. But if a condition IS provided, then I'm pretty sure that UIAutomation "flattens" the tree so that instead of

Code: Select all

295 Type: 50000 (Button) Name: "Remove" LocalizedControlType: "button"
295.1 Type: 50006 (Image) LocalizedControlType: "image"
296 Type: 50004 (Edit) LocalizedControlType: "edit"
297 Type: 50006 (Image) LocalizedControlType: "image"
298 Type: 50006 (Image) LocalizedControlType: "image"
it will look something like this

Code: Select all

11 Type: 50000 (Button) Name: "Remove" LocalizedControlType: "button"
12 Type: 50006 (Image) LocalizedControlType: "image"
13 Type: 50006 (Image) LocalizedControlType: "image"
14 Type: 50006 (Image) LocalizedControlType: "image"
Why this would make sense, you would have to ask the Microsoft team that is developing UIAutomation. :)

TheBeginner
Posts: 91
Joined: 19 Oct 2018, 12:05

Re: UIAutomation with a focus on Chrome

Post by TheBeginner » 01 Feb 2023, 13:52

@Descolada 👍

phenylalmine
Posts: 2
Joined: 05 Feb 2023, 22:03

Re: UIAutomation with a focus on Chrome

Post by phenylalmine » 05 Feb 2023, 22:06

Code: Select all

#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.

#Include %A_ScriptDir%\Lib\UIA_Interface.ahk
 ; SetTitleMatchMode, 2

el := WinExist("Microsoft Store ahk_exe ApplicationFrameHost.exe ahk_class ApplicationFrameWindow")
WinActivate, ahk_id %el%
WinWaitActive, ahk_id %el%
el := UIA.ElementFromHandle(el)

el.WaitElementExist("ControlType=Button AND AutomationId='AppIdentityBuyButton'").Click("left")
I can't seem to get this to click the "GET" button on a Microsoft Store app. What am I doing wrong? I've ran as administrator and also changed the AutomationId if needed.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Feb 2023, 00:14

@phenylalmine, I don't know if it was a copy-paste mistake, but you haven't initiated UIA with UIA_Interface(), so your code is doing nothing but activating the window.
I would recommend using Highlight() and DumpAll for debugging, so you know you've actually got the elements you want.

Code: Select all

#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetBatchLines, -1

#Include <UIA_Interface>
 ; SetTitleMatchMode, 2

UIA := UIA_Interface()

el := WinExist("Microsoft Store ahk_exe ApplicationFrameHost.exe ahk_class ApplicationFrameWindow")
WinActivate, ahk_id %el%
WinWaitActive, ahk_id %el%
el := UIA.ElementFromHandle(el)

el.WaitElementExist("AutomationId='AppIdentityBuyButton' OR AutomationId='AppIdentityInstallButton' AND ControlType=Button").Highlight().Click()
Notice that in this code I chained two AutomationId's together. The expression is evaluated left to right (no parenthesis are allowed unfortunately), so this will match for a button that has either AutomationId 'AppIdentityBuyButton' or 'AppIdentityInstallButton'.

You can use Click("left") instead of Click() if you want, but in the case of Microsoft Store, the regular Click() should work because the button responds to UIA calls normally. Click("left") will cause the mouse to move on the button and click it with MouseClick function.

phenylalmine
Posts: 2
Joined: 05 Feb 2023, 22:03

Re: UIAutomation with a focus on Chrome

Post by phenylalmine » 07 Feb 2023, 21:12

Thanks.

Code: Select all

el.WaitElementExist("ControlType=Button AND AutomationId='AppIdentityBuyButton'", 30000).Click("") ; timeout 30 seconds
Is this wrong for a timeout? It doesn't seem to proceed even after the 30 seconds have passed.

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 07 Feb 2023, 23:28

phenylalmine wrote:
07 Feb 2023, 21:12
Thanks.

Code: Select all

el.WaitElementExist("ControlType=Button AND AutomationId='AppIdentityBuyButton'", 30000).Click("") ; timeout 30 seconds
Is this wrong for a timeout? It doesn't seem to proceed even after the 30 seconds have passed.
You should write like that:
el.WaitElementExist("ControlType='Button' AND AutomationId='AppIdentityBuyButton'",,,,30000).Click() ; timeout 30 seconds

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Feb 2023, 00:04

@phenylalmine, as leosouza85 mentioned, the arguments for WaitElementExist are WaitElementExist(expression, scope:=4, matchMode:=3, caseSensitive:=True, timeOut:=-1, cacheRequest:=""). This will be changed in UIA v2 (for AHK v2), where timeOut is the second argument, as naturally expected.

daeinai
Posts: 3
Joined: 08 Feb 2023, 06:42

Re: UIAutomation with a focus on Chrome

Post by daeinai » 08 Feb 2023, 08:16

Hello all, I'm newbie in this forum and autohotkey, and I need a guidance on how to grab the new value when changed happened using UI Automation library by @Descolada.
Stock.jpg
Stock.jpg (3.88 KiB) Viewed 2800 times
Actually I want to add a kind of listener to the bid price above which will trigger a function handler if the price changed.

After reading all the post in this forum I try it without success using :

Code: Select all

#NoEnv
#SingleInstance force
#Include Lib\UIA_Interface.ahk
SetTitleMatchMode,2

UIA := UIA_Interface()
mg := UIA.ElementFromHandle(WinExist("My Broker")).FindFirstBy("Name=My Stock")
mg1 := mg.FindFirstBy("Name=Bid row 0")

MsgBox, % mg1.Value

handler := UIA_CreateEventHandler("Hdl", "PropertyChanged")

UIA.AddPropertyChangedEventHandler(mg1,,,handler, [UIA_Enum.UIA_ValueValuePropertyId])

OnExit("Cleanup")
return
Cleanup() {
    UIA_Interface().RemoveAllEventHandlers()
}

Hdl(sender, propertyId, newValue)
{
    MsgBox, % "Sender: " sender.Dump()
        . "`nPropertyId: " propertyId
        . "`nNew value: " newValue
}
F5::ExitApp
The "MsgBox, % mg1.Value" will show me the bid price value when the script started but still the script fail to trigger the Hdl (the handler function).

Below also fail to trigger the Hdl function:
UIA.AddPropertyChangedEventHandler(mg1,,,handler, [UIA_Enum.UIA_ValuePatternId])
UIA.AddPropertyChangedEventHandler(mg1,,,handler, [UIA_Enum.UIA_LegacyIAccessibleValuePropertyId])


Can some one please... guide me on how to make it work but I prefer not to use timer or periodic checking because next I will add more listener for each ticker in my watchlist's stocks.


Below is data from Accessibility Insight and Inspect :
Accessibility.jpg
Accessibility.jpg (78.54 KiB) Viewed 2800 times
Inspect.jpg
Inspect.jpg (143.23 KiB) Viewed 2800 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Feb 2023, 11:08

@daeinai, unfortunately I am unable to test that program out myself, so I can only give general advice. My first suggestion is to inspect the element with the Accessibility Insights tool and check which events it's capable of firing. You can listen to events by pressing the "..." button next to the element:
image.png
image.png (83.81 KiB) Viewed 2780 times
But unfortunately it's rather common that PropertyChangedEvent doesn't work, because implementing it is left to the developer of the program... It's more common that StructureChanged event is implemented though, so perhaps that could work? An example in Chrome, press the "Toggle Text" button once the website has opened:

Code: Select all

#NoEnv
#Warn
#SingleInstance force
SetTitleMatchMode, 2
SetBatchLines, -1
#include ..\Lib\UIA_Interface.ahk

UIA := UIA_Interface()
Run, % "chrome.exe https://www.w3schools.com/howto/howto_js_toggle_text.asp"
WinWaitActive, How To Toggle Text
targetEl := UIA.ElementFromHandle().WaitElementExist("Name='Hello' AND Type=Text").Highlight()
parentEl := UIA.TreeWalkerTrue.GetParentElement(targetEl)
handler := UIA_CreateEventHandler("StructureChangedEventHandler", "StructureChanged")
UIA.AddStructureChangedEventHandler(parentEl,,, handler)
OnExit("ExitFunc") ; Set up an OnExit call to clean up the handler when exiting the script
return

StructureChangedEventHandler(sender, changeType, runtimeId) {
    try ToolTip, % "Sender: " sender.Dump() 
        . "`nChange type: " changeType
        . "`nRuntime Id: " PrintArray(runtimeId)
	SetTimer, RemoveToolTip, -3000
}
PrintArray(arr) {
	ret := ""
	for k, v in arr
		ret .= "Key: " k " Value: " (IsFunc(v)? v.name:IsObject(v)?PrintArray(v):v) "`n"
	return ret
}
ExitFunc() {
	UIA_Interface().RemoveAllEventHandlers()
}
RemoveToolTip:
	ToolTip
	return
F5::ExitApp
With StructureChanged the element itself usually doesn't send out the event, so you need to register the elements parent (or the window).

daeinai
Posts: 3
Joined: 08 Feb 2023, 06:42

Re: UIAutomation with a focus on Chrome

Post by daeinai » 08 Feb 2023, 20:16

Thank you very much for your kindness and quick response @Descolada , I'll check it soon :)

daeinai
Posts: 3
Joined: 08 Feb 2023, 06:42

Re: UIAutomation with a focus on Chrome

Post by daeinai » 09 Feb 2023, 02:16

@Descolada here is the screenshoot from the Accessibility Insight :


Accessibility Event Detail.jpg
Accessibility Event Detail.jpg (55.81 KiB) Viewed 2704 times


In configuration tab I only tick the ValuePattern.Value, else leave un-ticked. It seems that there's no event raised/detected when the value has changed.

The screenshoot from AccEvent tools below gives another same confirmation of no event raised when the value has changed :


AccEvent setting.jpg
AccEvent setting.jpg (59.03 KiB) Viewed 2704 times
AccEvent result.jpg
AccEvent result.jpg (45.81 KiB) Viewed 2704 times


***In the AccEvent tools event type setting, I also ticked the ValueValue and IsValuePatternAvailable property.

Anyway, thank you very much @Descolada :)

malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIAutomation with a focus on Chrome

Post by malcev » 09 Feb 2023, 12:51

Descolada, why do You use for click() method firstly invoke method, but not DoDefaultAction?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 09 Feb 2023, 13:41

@malcev
1) Often both LegacyIAccessiblePattern and some other "click"-type pattern (InvokePattern, TogglePattern etc) are available for the same element, but real life testing has shown that DoDefaultAction is less reliable: it sometimes shows as available, but actually doesn't trigger an action, whereas the UIA pattern does. And since I know of no way to verify whether the "click" was received, I went with the more reliable option. If I remember correctly, this problem was most pronounced in browser content elements.
2) DoDefaultAction doesn't obey UIA.AutoSetFocus property: at least every time I tested it, the window was activated regardless of AutoSetFocus value.

malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIAutomation with a focus on Chrome

Post by malcev » 09 Feb 2023, 14:10

Interesting, for me DoDefaultAction does not activate chrome window at all.

Post Reply

Return to “Scripts and Functions (v1)”