UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
r2997790
Posts: 71
Joined: 02 Feb 2017, 02:46

Re: UIAutomation with a focus on Chrome

Post by r2997790 » 21 Dec 2022, 08:06

This is wonderful. Thank you!

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 11 Jan 2023, 09:57

Hi @Descolada

I am trying to use FindFirstWithOptions but I can't get it to work. Do you have an example in AHK that can help me?

Descolada
Posts: 1117
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 11 Jan 2023, 11:30

@mora145, currently FindFirstWithOptions is only useful for the TreeTraversalOptions_LastToFirstOrder option:
Element.FindFirstWithOptions(4,"Type=Button", UIA.TreeTraversalOptions_LastToFirstOrder) would find the last element of type "Button" (scope is TreeScope_Descendants). Unfortunately the "root" argument is currently broken and I'm querying Microsofts Q&A as to why (possibly a buggy implementation). Also my testing has shown that if not using LastToFirstOrder, then the TreeScope gets completely ignored (seems like another bug). So the example I provided is the only useful way of using it unfortunately :(

EDIT: I'm completely mistaken, FindFirstWithOptions is working exactly as it should be, it's just that its documentation is horrible. It seems that the root argument element must be a *parent* of the starting point element, otherwise it won't work properly. If root is omitted, then it will start from the Root Element (this will be fixed in a later UIA_Interface.ahk release). And this should explain PostOrder travel. I've written some examples (ran on Windows 10) you can study:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
#Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2
#include Lib\UIA_Interface.ahk

Run notepad.exe
WinWaitActive ahk_exe notepad.exe

UIA := UIA_Interface()
npEl := UIA.ElementFromHandle("ahk_exe notepad.exe")
npEl.FindFirst("Type=Document or Type=Edit").Value := npEl.DumpAll()

postOrder := (npEl.FindFirst("Type=MenuBar")).FindAllWithOptions(2, "", UIA.TreeTraversalOptions_PostOrder, npEl)
out := ""
for i, el in postOrder
 out .= i ": " el.Dump() "`n"
MsgBox, % "FindAll with PostOrder. `nThe order of elements is exactly how the tree was traversed.`nNote that TreeScope_Children contains now both the starting element and Notepad element children.`n" out

MsgBox, % "Looking for first MenuItem: " npEl.FindFirstWithOptions(, "Type=MenuItem").Dump() "`n" ; Works like regular FindFirst
MsgBox, % "Looking for first MenuItem starting from the end: " npEl.FindFirstWithOptions(, "Type=MenuItem", UIA.TreeTraversalOptions_LastToFirstOrder).Dump() "`n" ; Returns LAST MenuItem

DownButton := npEl.FindFirst("AutomationId=DownButton")
MsgBox, % "Looking for UpButton starting from DownButton and looking forwards: " DownButton.FindFirstWithOptions(4, "AutomationId=UpButton",, npEl).Dump() "`n" ; Should be empty, because "Line up" button is before "Line down"
MsgBox, % "Looking for UpButton starting from DownButton and looking backwards: " DownButton.FindFirstWithOptions(4, "AutomationId=UpButton", UIA.TreeTraversalOptions_LastToFirstOrder, npEl).Dump() "`n"
MsgBox, % "Looking for UpButton starting from DownButton and looking backwards, TreeScope_Element: " DownButton.FindFirstWithOptions(1, "AutomationId=UpButton", UIA.TreeTraversalOptions_LastToFirstOrder, npEl).Dump() "`n" ; TreeScope seems to be ignored
MsgBox, % "Looking for UpButton starting from DownButton and looking backwards, TreeScope_Descendants: " DownButton.FindFirstWithOptions(4, "AutomationId=NonClientVerticalScrollBar", UIA.TreeTraversalOptions_LastToFirstOrder, npEl).Dump() "`n" ; Starts from DownButton, but doesn't find the parent ScrollBar because the tree node has already been "visited"
MsgBox, % "Looking for UpButton starting from DownButton and looking backwards, TreeScope_Descendants: " DownButton.FindFirstWithOptions(4, "AutomationId=NonClientVerticalScrollBar", UIA.TreeTraversalOptions_LastToFirstOrder | UIA.TreeTraversalOptions_PostOrder, npEl).Dump() "`n" ; Finds the parent ScrollBar, because it's traveling in PostOrder

ExitApp

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 11 Jan 2023, 15:12

Excelentísimo! Gracias @Descolada. Prometo que haré una donación cuando reciba algún dinero. Gran trabajo con tu librería. Eres lo máximo explicando, gracias por tu tiempo.

ludamo
Posts: 44
Joined: 25 Mar 2015, 02:21

Re: UIAutomation with a focus on Chrome

Post by ludamo » 11 Jan 2023, 18:03

I have read with interest this whole thread (thank you so much Descolada and others) and checked out other topics in V1 (e.g. viewtopic.php?p=391568 - getting the URL from Firefox, and also another one getting the video links from a Youtube page) to try and solve the thing I am trying to do. But I have been unable to successfully translate their code into v2. I have been trying to construct an independent, v2 code, to scroll the Win10 clipboard when Excel 2010 is active. The problem being that Excel captures all the scroll messages when it is active. The following code using the UIAutomation.ahk class of thqby https://github.com/thqby/ahk2_lib/blob/master/UIAutomation/UIAutomation.ahk works well as can be demonstrated with my code below, But I would very much appreciate it if someone could help me construct some self-contained code in v2 presumably utilizing ComObject(CLSID_CUIAutomation := "{FF48DBA4-60EF-4201-AA87-54103EEF594E}", IID_IUIAutomation := "{30CBE57D-D9D0-452A-AB13-7AC5AC4825EE}") and preferably ComCall's (or DllCall's).

Code: Select all

#Warn
#SingleInstance Force
SetTitleMatchMode 2
#Requires AutoHotkey v2.0+ 64-bit

#Include <UIAutomation>

oUIA := UIA() ; Initialize UIA interface

; cv_UIA := ComObject(CLSID_CUIAutomation := "{FF48DBA4-60EF-4201-AA87-54103EEF594E}", IID_IUIAutomation := "{30CBE57D-D9D0-452A-AB13-7AC5AC4825EE}")
; p_cv_UIA := Format("0x{:X}", ComObjValue(cv_UIA))

WheelDown::
WheelUp:: {
MouseGetPos ,,&hWndM

cbEl := UIA.ElementFromHandle(hWndM) ; Get the element for the Clipboard
cbLV := cbEl.FindControl("List")
cbScrollPat := cbLV.GetCurrentPattern("Scroll")

if (A_ThisHotkey = "WheelDown")
	cbScrollPat.Scroll(2,4)				; 2 = no horizontal scroll; 4 = small scroll down
else
	cbScrollPat.Scroll(2,1)				; 2 = no horizontal scroll; 1 = small scroll up

}

Esc:: {
ExitApp
}

Descolada
Posts: 1117
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 12 Jan 2023, 00:18

@ludamo, perhaps somebody else can help you more thoroughly, but me myself have decided not to help with translations of UIA libraries code to raw Dll/ComCalls, because it's annoyingly time intensive. But you could take a look at this Reddit post, where there is raw UIA v2 code, you could base yours on that. UIA_ListControlTypeId = 50008, and ScrollPatternId = 10004.

ludamo
Posts: 44
Joined: 25 Mar 2015, 02:21

Re: UIAutomation with a focus on Chrome

Post by ludamo » 12 Jan 2023, 01:32

Thanks for the reply and that link. That example works in Notepad (doesn't work in Notepad++ or Word 2010) so I shall try and adapt the code to my goal and hopefully report back.

ludamo
Posts: 44
Joined: 25 Mar 2015, 02:21

Re: UIAutomation with a focus on Chrome

Post by ludamo » 12 Jan 2023, 03:45

It works! Many thanks again. Would you care to comment if I need to do ObjRelease or something like that in the code?

Code: Select all

#Requires AutoHotkey v2.0+ 64-bit
#SingleInstance Force
IUIA := ComObject("{E22AD333-B25F-460C-83D0-0581107395C9}", "{34723AFF-0C9D-49D0-9896-7AB52DF8CD8A}")
VT_I4 := 3	; identifies a 32-bit signed int

WheelDown::
WheelUp:: {

	MouseGetPos( ,, &hWndM)
	ComCall(6, IUIA, "ptr", hWndM, "ptr*", &ElfromHnd:=0) 			                 ; ElementFromHandle

	var := Buffer(24,0), NumPut("short",VT_I4,var,0), NumPut("ptr", 50008, var, 8)	; UIA_ListControlTypeId = 50008
	ComCall(23, IUIA, "int", 30003, "ptr", var, "ptr*", &listCond:=0)				; CreatePropertyCondition
	ComCall(5, ElfromHnd, "int", 4, "ptr", listCond, "ptr*", &listCB:=0) 			; FindFirst
	ComCall(16, listCB, "int", 10004, "ptr*", &patternObject := 0)					; get Pattern ScrollPatternId = 10004
	
if (A_ThisHotkey = "WheelDown")
	ComCall(3, patternObject, "int", 2, "int", 4)				; 2 = no horizontal scroll; 4 = small scroll down
else
	ComCall(3, patternObject, "int", 2, "int", 1)				; 2 = no horizontal scroll; 4 = small scroll up
}

Descolada
Posts: 1117
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 12 Jan 2023, 04:06

@ludamo, AFAIK you have to call ObjRelease on every object you get from UIA, because UIA calls AddRef on objects before sending them over to AHK, but the task of releasing it is left to the AHK side because UIA can't know when you are done with using the object, so you need to Release it manually when done with it. Otherwise the garbage collector can't delete it and you'll get a memory leak.

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 13 Jan 2023, 08:38

Hi, @Descolada

I've noticed, the function SmallestElementFromPoint, only works when the mouse cursor is an arrow, when hovering a link (the arrow turns into a hand and when hovering a edit, the mouse turns into a caret) the code will not work:

Example:
MouseGetPos, OutputVarX, OutputVarY
Msgbox % cUIA.SmallestElementFromPoint(OutputVarX, OutputVarY)

Edit: The problem is the MouseGetPos function, it is hanging when the mouse is a hand! Sorry

only works on chrome when the cursor is in the normal state, a arrow. (tested on chrome)

Descolada
Posts: 1117
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 13 Jan 2023, 15:29

@leosouza85, works for me using AHK 1.1.36.02. When the cursor is a "hand" then it hangs for a second before returning the correct element: this is because the normal ElementFromPoint is returning the main document element, so the script has to sort through all the elements in the document until it finds the correct smallest element under the cursor. This unfortunately takes some time and makes the code laggy, but in my setup it still works...

Code: Select all

SetTitleMatchMode, 2
#include Lib\UIA_Interface.ahk
#include Lib\UIA_Browser.ahk
CoordMode, Mouse, Screen
cUIA := new UIA_Browser("ahk_exe chrome.exe")
Loop {
    MouseGetPos, OutputVarX, OutputVarY
    ToolTip % "Mouse X: " OutputVarX " Y: " OutputVarY 
        . "`nElement under mouse: " cUIA.SmallestElementFromPoint(OutputVarX, OutputVarY).Dump()
        . "`nAHK version: " A_AhkVersion
;    try ToolTip % "Mouse X: " OutputVarX " Y: " OutputVarY 
;        . "`nElement under mouse: " cUIA.ElementFromPoint(OutputVarX, OutputVarY).Dump()
}

swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: UIAutomation with a focus on Chrome

Post by swagfag » 13 Jan 2023, 15:55

ludamo wrote:
12 Jan 2023, 03:45
if I need to do ObjRelease or something like that in the code?

Code: Select all

......"ptr*", &ElfromHnd:=0)

if ure working with raw pointers, yes
if ure wrapping them in comvalues, no - the comvalue will release it automatically whenever it goes out of scope

Code: Select all

.."ptr*", ElfromHnd := ComValue(9, 0))

ludamo
Posts: 44
Joined: 25 Mar 2015, 02:21

Re: UIAutomation with a focus on Chrome

Post by ludamo » 13 Jan 2023, 18:28

Thanks swagfag, I have added that to my code.

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 21 Jan 2023, 14:45

hi @Descolada

In UIA Tree, we have a lot of Elements without any Name, usually children of something that have a name... lets suppose we have the following:
Group (name = television)
-Button (name = "")
--Text (name = "")

Maybe there will be a good thing if UIA Interface gives the name of the parent to the nameless children, so if the tree construction saw the above structure, it could read as:
Group (name = television)
-Button (name = television)
--Text (name = television)

Maybe with a code to infer it is an auto naming, like

Group (name = television)
-Button (name = television c1)
--Text (name = television c1c1)

c1 meaning child 1

What do you think? I think it would be great to speed up developing when having a lot of nameless childs... and also I think it is not difficult to implement.

So UIA Viewer and TreeInspector could show this names, and the findfunctions will translate thet to find the correspondent nameless element

Descolada
Posts: 1117
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 23 Jan 2023, 00:30

@leosouza85, I think I understood the problem you are describing and I've thought about your proposed solution for a bit.

Code: Select all

Group (name = television)
-Button (name = television c1)
--Text (name = television c1c1)
Essentially what you want from this is for nameless elements to have some unique identifier to more easily access them. Something similar to a path, but also filtering for only nameless elements. So a wrapper for something like El.FindFirstBy("Name=television").FindAllByName(,2)[1].FindAllByName(,2)[1] which would return the "television c1c1" element.

Now, this proposed solution has similar kind of issues like the numeric path from DumpAll: it can get shifted/changed when elements are added to the tree. So if you were to add an element:

Code: Select all

Group (name = television)
-List (name = television c1)
-Button (name = television c2)
--Text (name = television c2c1)
You see that the "name-path" for Text element changed and would not be found now. So why not also consider the type of the element? For example:

Code: Select all

Group (name = television)
-List (name = television List1)
-Button (name = television Button1)
--Text (name = television Button1Text1)
This would avoid such collisions.. But why stop at that?

What I've been thinking is why not try to create as unique a path as possible for every element? For example we could consider Type, Name, and AutomationId:

Code: Select all

Group (name = television) (GeneratedId = television and type=group and automationid=)
-List (name = television c1) (GeneratedId = name= and type=List and automationid=)
-Button (name = television c2) (GeneratedId = name= and type=button and automationid=)
--Text (name = television c2c1) (GeneratedId = name= and type=Text and automationid=)
Then perhaps we could chain those GeneratedId's together to create a more reliable "path": El.FindByGeneratedPath("name=television and type=group and automationid=-->GeneratedId = name= and type=button and automationid=-->GeneratedId = name= and type=Text and automationid=").
But I don't like how long paths it creates... I'm gonna think about this a bit more ;)

EDIT: If we are always using the same properties, then the path could be shortened to a certain order of properties, and the type can be replaced with the typeId minus 50000 (because they are all 50000 something). So it could be shortened to something like "television|26|-->|0|-->|20|"

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 23 Jan 2023, 07:06

Descolada wrote:
23 Jan 2023, 00:30
@leosouza85, I think I understood the problem you are describing and I've thought about your proposed solution for a bit.

Code: Select all

Group (name = television)
-Button (name = television c1)
--Text (name = television c1c1)
Essentially what you want from this is for nameless elements to have some unique identifier to more easily access them. Something similar to a path, but also filtering for only nameless elements. So a wrapper for something like El.FindFirstBy("Name=television").FindAllByName(,2)[1].FindAllByName(,2)[1] which would return the "television c1c1" element.

Now, this proposed solution has similar kind of issues like the numeric path from DumpAll: it can get shifted/changed when elements are added to the tree. So if you were to add an element:

Code: Select all

Group (name = television)
-List (name = television c1)
-Button (name = television c2)
--Text (name = television c2c1)
You see that the "name-path" for Text element changed and would not be found now. So why not also consider the type of the element? For example:

Code: Select all

Group (name = television)
-List (name = television List1)
-Button (name = television Button1)
--Text (name = television Button1Text1)
This would avoid such collisions.. But why stop at that?

What I've been thinking is why not try to create as unique a path as possible for every element? For example we could consider Type, Name, and AutomationId:

Code: Select all

Group (name = television) (GeneratedId = television and type=group and automationid=)
-List (name = television c1) (GeneratedId = name= and type=List and automationid=)
-Button (name = television c2) (GeneratedId = name= and type=button and automationid=)
--Text (name = television c2c1) (GeneratedId = name= and type=Text and automationid=)
Then perhaps we could chain those GeneratedId's together to create a more reliable "path": El.FindByGeneratedPath("name=television and type=group and automationid=-->GeneratedId = name= and type=button and automationid=-->GeneratedId = name= and type=Text and automationid=").
But I don't like how long paths it creates... I'm gonna think about this a bit more ;)

EDIT: If we are always using the same properties, then the path could be shortened to a certain order of properties, and the type can be replaced with the typeId minus 50000 (because they are all 50000 something). So it could be shortened to something like "television|26|-->|0|-->|20|"
Thank you! I think this is the best format, more friendly:

You see that the "name-path" for Text element changed and would not be found now. So why not also consider the type of the element? For example:

Code: Select all

Group (name = television)
-List (name = television List1)
-Button (name = television Button1)
--Text (name = television Button1Text1)

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 29 Jan 2023, 04:20

Hi Descolada, I have a problem getting an element. I spent all day trying to get the text from path 1.2.16, but I keep getting an empty element like this:
Screenshot_29.jpg
Screenshot_29.jpg (5.6 KiB) Viewed 2504 times
This is what my DumpAll returns when I get the window.

Code: Select all

1.2.15 Type: 50005 (Hyperlink) Name: "דברו איתי" Value: "tel:072-3301774" LocalizedControlType: "link"
1.2.15.1 Type: 50020 (Text) Name: "דברו איתי" LocalizedControlType: "text"
1.2.16 Type: 50020 (Text) Name: "טשרנחובסקי 10 , באר שבע" LocalizedControlType: "text"
1.2.17 Type: 50020 (Text) Name: "פנו לטכנאי" LocalizedControlType: "text"
.

And to get the text I'm doing first.

hello := chrome.FindFirstBy("Name="phoneButton) get the 1.2.15 element.

That gets the path 1.2.15
Then with test := hello.FindByPath("+1",cUIA.CreateCondition("ControlType", "Text")) I try to get the 1.2.16 element but it appears empty, as in the image.

Descolada
Posts: 1117
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 29 Jan 2023, 04:50

@mora145, if you Dump hello := chrome.FindFirstBy("Name="phoneButton), do you get the correct output of Type: 50005 (Hyperlink) Name: "דברו איתי" Value: "tel:072-3301774" LocalizedControlType: "link"?

You could try hello.FindByPath("p").DumpAll() to verify that the text element is found and get it by path from there, or try hello.FindByPath("+1") or hello.FindByPath("+1") to see what they return. Sometimes TreeWalker and FindAll (which DumpAll uses) results differ, so it might need some experimentation...

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 29 Jan 2023, 05:21

Descolada wrote:
29 Jan 2023, 04:50
@mora145, if you Dump hello := chrome.FindFirstBy("Name="phoneButton), do you get the correct output of Type: 50005 (Hyperlink) Name: "דברו איתי" Value: "tel:072-3301774" LocalizedControlType: "link"?
Ye, I obtain the correct.
You could try hello.FindByPath("p").DumpAll() to verify that the text element is found and get it by path from there, or try hello.FindByPath("+1") or hello.FindByPath("+1") to see what they return. Sometimes TreeWalker and FindAll (which DumpAll uses) results differ, so it might need some experimentation...
I obtain that:
Screenshot_30.jpg
Screenshot_30.jpg (38 KiB) Viewed 2475 times
n this case, the text I am looking for is number 10.

Descolada
Posts: 1117
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 30 Jan 2023, 00:26

@mora145, I guess if FindByPath isn't working out at all, then you could use as a workaround hello.FindByPath("p.10")? Or something like this might work as well:

Code: Select all

TextTW := cUIA.CreateTreeWalker("Type=Text")
MsgBox, % TextTW.GetLastChildElement(hello.FindByPath("p")).Dump()

Post Reply

Return to “Scripts and Functions (v1)”