UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 05 Jul 2022, 16:21

Descolada wrote:
02 Jul 2022, 09:43
@william_ahk, I am not at all opposed to GitHub pull requests for such changes :) For me, I decided that manually testing n browsers for every change I make in UIA_Browser is a real pain, so limiting to the 1-2 most popular ones seemed logical. I do have plans to expose the element names to the user (or at least give a way to change the default action), just haven't gotten around to it yet.

That JS CSS selector scripts seems very cool, I'll check it out soon. You probably could switch Click to ControlClick for a more reliable result ;) And soon there might be a real need for UIA_Browser_JS.ahk extension!
In general controlclick is more reliable? im having trouble with some old guis...

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 05 Jul 2022, 17:52

Thank you, excellent work.
You have some examples with Paths
How can you work with them?

william_ahk
Posts: 482
Joined: 03 Dec 2018, 20:02

Re: UIAutomation with a focus on Chrome

Post by william_ahk » 05 Jul 2022, 22:40

@Descolada Good :trollface:
BTW I actually tested BlockInput and it doesn't help with the modifier keys issue, on my test the Alt key was somehow already stuck and needed release no matter the physical state of the key. But anyway it's better to add BlockInput during the SetURL process in case the user presses any key physically.

@leosouza85 Using ControlClick for coordinates is much less reliable than Click, we usually only use it to keep our mouse from moving around.

Descolada
Posts: 1100
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 05 Jul 2022, 23:21

@leosouza85, more reliable in the sense that ControlClick isn't interfered by the user and works even when the window is hidden behind other windows. In some of the older GUIs it might not work at all, but if it does work then in my experience it is quite reliable in the sense of "the click actually happens". Your milage might vary though :D And getting it to work properly might be tricky sometimes because of the coordinate system and the click not being visible (makes it harder to debug).

@william_ahk, yeah, currently I decided not to block input, because I've noticed sometimes UIA gets slow/hangs a bit in Chrome, and I don't want to block input for a longer period of time than a couple hundred ms. I'll do some more testing and if needed, add it in later (or with an optional flag for it) :)

Descolada
Posts: 1100
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Jul 2022, 07:32

@mora145, I assume you are asking about the FindByPath method? You need to provide it the "path" from DumpAll(), and you need to use FindByPath also on the same element (in my example, it is the Notepad window element, using it on a child element of the Notepad window wouldn't work). Example of it:

Code: Select all

SetTitleMatchMode, 2
#Include <UIA_Interface>

UIA := UIA_Interface()
Run, notepad.exe
WinWaitActive, ahk_exe notepad.exe
npEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe"))
MsgBox, % npEl.DumpAll()
MsgBox, % "File menuitem by path: " npEl.FindByPath("4.1").Dump()
MsgBox, % "File menuitem by name: " npEl.FindFirstByName("File").Dump()
MsgBox, % "File menuitem by type using FindAll: " npEl.FindAllByType("MenuItem")[2].Dump()
MsgBox, % "File menuitem using TreeWalker: " UIA.TreeWalkerTrue.GetFirstChildElement(npEl.FindFirstBy("AutomationId=MenuBar")).Dump()
ExitApp
The thing is, I haven't ever actually needed that method. I only added it "just in case" if it is ever needed, but in every app so far I have been able to find any element using either FindFirst, FindAll or TreeWalker (as my example demonstrates)...

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 06 Jul 2022, 12:50

Descolada wrote:
06 Jul 2022, 07:32
@mora145, I assume you are asking about the FindByPath method? You need to provide it the "path" from DumpAll(), and you need to use FindByPath also on the same element (in my example, it is the Notepad window element, using it on a child element of the Notepad window wouldn't work). Example of it:

Code: Select all

SetTitleMatchMode, 2
#Include <UIA_Interface>

UIA := UIA_Interface()
Run, notepad.exe
WinWaitActive, ahk_exe notepad.exe
npEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe"))
MsgBox, % npEl.DumpAll()
MsgBox, % "File menuitem by path: " npEl.FindByPath("4.1").Dump()
MsgBox, % "File menuitem by name: " npEl.FindFirstByName("File").Dump()
MsgBox, % "File menuitem by type using FindAll: " npEl.FindAllByType("MenuItem")[2].Dump()
MsgBox, % "File menuitem using TreeWalker: " UIA.TreeWalkerTrue.GetFirstChildElement(npEl.FindFirstBy("AutomationId=MenuBar")).Dump()
ExitApp
The thing is, I haven't ever actually needed that method. I only added it "just in case" if it is ever needed, but in every app so far I have been able to find any element using either FindFirst, FindAll or TreeWalker (as my example demonstrates)...
I don't know how else I could click on this text box. I just need to click on it, but I don't have names or anything like that to be able to point it to. That's when I saw the Path route.
Thanks for the help :) Sorry, I'm just very new and learning how to automate.
Imagen1.png
Imagen1.png (131.13 KiB) Viewed 3465 times

Descolada
Posts: 1100
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Jul 2022, 13:18

@mora145, that path is for use with the Acc library and won't work with UIA, but you should be able to click it with Acc as well :)
But with UIA it's hard to say what the best way to get that element would be without knowing the whole tree (you could hover over the element and press Esc, then press "Construct tree for the whole window" and take a screenshot of the relevant part of the tree). Perhaps you could instead find the "Save Translation" button and then get the previous element with a TreeWalker?

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 06 Jul 2022, 14:12

Descolada wrote:
06 Jul 2022, 13:18
@mora145, that path is for use with the Acc library and won't work with UIA, but you should be able to click it with Acc as well :)
But with UIA it's hard to say what the best way to get that element would be without knowing the whole tree (you could hover over the element and press Esc, then press "Construct tree for the whole window" and take a screenshot of the relevant part of the tree). Perhaps you could instead find the "Save Translation" button and then get the previous element with a TreeWalker?
Thanks, anyway I think I've made it.
I made

Code: Select all

cEl := UIA.ElementFromHandle(WinExist("Cataclysm-DDA | Transifex Editor - Google Chrome"))
cEl.FindByPath("4.1.1.2.2.2.1.5.2.1.2.1.1.1.1.1.4.1").Click(3)
I had to do 3 clicks, I don't know why, but only 1 does nothing. 2 just directs the cursor to the element, and 3 is that it does the click.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 06 Jul 2022, 14:29

I have also noticed that the Path that appears in UIAViewer does not correspond to the one that should be used in the path.
So I had to do:

Code: Select all

cEl := UIA.ElementFromHandle(WinExist("Cataclysm-DDA | Transifex Editor - Google Chrome")) 
Clipboard := cEl.DumpAll()
With this I was able to copy the paths to the Clipboard and then look for the text Type your translation here to find the path, in my case it changed to '1.198.10'.

I put it in

Code: Select all

cEl.FindByPath("1.198.10").Click(3)
and it worked perfectly, making the necessary clicks, no need to use the UIA_Browser.ahk

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 06 Jul 2022, 14:54

image.png
image.png (53.59 KiB) Viewed 3309 times
let's say, i want to read the date published ("Jul 6, 2022" as shown) of a youtube video page. and from the screenshot, i can tell that there is no direct way to get to it. what i may do is to identify a reliable nearby element as an anchor. findfirst to it, and maneuver by treewalker (sibling, child, parent, etc.) so, in this case, the "•" is a good (enough) anchor.

another attempt, for instance, what if i want to read everything below the "complementary" group and until the date published? the "complementary" group therefore would be a good anchor, find first to it, maneuver, and loop until "•" (stop sign).

there we may not have a definite path sometimes, as the number of elements around may vary. so, anchor and stop sign could be the last resort, i guess.

my two cents.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 06 Jul 2022, 17:20

SundayProgrammer wrote:
06 Jul 2022, 14:54
image.png

let's say, i want to read the date published ("Jul 6, 2022" as shown) of a youtube video page. and from the screenshot, i can tell that there is no direct way to get to it. what i may do is to identify a reliable nearby element as an anchor. findfirst to it, and maneuver by treewalker (sibling, child, parent, etc.) so, in this case, the "•" is a good (enough) anchor.

another attempt, for instance, what if i want to read everything below the "complementary" group and until the date published? the "complementary" group therefore would be a good anchor, find first to it, maneuver, and loop until "•" (stop sign).

there we may not have a definite path sometimes, as the number of elements around may vary. so, anchor and stop sign could be the last resort, i guess.

my two cents.
Excellent idea. Thank you, I am very new and have doubts with almost everything. So I would have to ask: How can I 'maneuver' to select different elements?
Do you have a tutorial on how to do it?

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 06 Jul 2022, 19:12

mora145 wrote:
06 Jul 2022, 17:20
Excellent idea. Thank you, I am very new and have doubts with almost everything. So I would have to ask: How can I 'maneuver' to select different elements?
Do you have a tutorial on how to do it?
for the examples i brought up, just use GetNextSiblingElement of treewalker should be sufficient. you may need, for other cases, some more methods (such as GetFirstChildElement or GetParentElement), nonetheless.

i'm incapable of providing tutorial, i'm afraid. me very new too, may be just a few weeks more than you, i guess. and never tried to pretend expert anything. just excited about uiautomation, and a bit (too? (maybe)) enthusiastic to share what i've learned along the way.

cheers.

Descolada
Posts: 1100
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Jul 2022, 23:49

@mora145, there isn't a tutorial on it as far as I know. But I created some examples for finding the Youtube "date published" text using TreeWalkers:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

UIA := UIA_Interface()
chromeId := WinExist("ahk_exe chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%
chrome := UIA.ElementFromHandle(chromeId, True)

; Find the date using the bullet character text element
bulletText := chrome.FindFirstByName("•") ; Find the bullet text element
TextWalker := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType", "Text")) ; Create a TreeWalker that only searches Text elements. If we used UIA.TreeWalkerTrue here, then it would return some invisible element that UIAViewer doesn't show
dateText := TextWalker.GetNextSiblingElement(bulletText) ; Get the next element over
MsgBox, % "Date: " dateText.Dump()
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%

; Find the date using the Like button as a reference
likeButton := chrome.FindFirstByNameAndType("like this video", "Button",,1) ; Find the "Like" button
dateText2 := TextWalker.GetPreviousSiblingElement(likeButton) ; Get the previous element. Notice that we don't have to climb the tree upwards here (use GetParentElement)
MsgBox, % "Date2: " dateText2.Dump()
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%

; Find the date using the player element and bullet text as references
nextText := chrome.FindFirstBy("AutomationId=player-container") ; Find the Youtube player-container element (the "complementary """ element)
while (nextText := TextWalker.GetNextSiblingElement(nextText)) { ; Loop until we find the bullet Text element
	if (nextText.CurrentName == "•") {
		dateText3 := TextWalker.GetNextSiblingElement(nextText) ; Then the next one over should be the date
		MsgBox, % "Date3: " dateText3.Dump()
		break
	}
}

ExitApp

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 07 Jul 2022, 01:11

Descolada wrote:
06 Jul 2022, 23:49
:
:
TextWalker := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType", "Text")) ; Create a TreeWalker that only searches Text elements. If we used UIA.TreeWalkerTrue here, then it would return some invisible element that UIAViewer doesn't show
:
:
dateText2 := TextWalker.GetPreviousSiblingElement(likeButton) ; Get the previous element. Notice that we don't have to climb the tree upwards here (use GetParentElement)
:
:
good ideas :thumbup: thanks.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 07 Jul 2022, 01:45

Descolada wrote:
06 Jul 2022, 23:49
@mora145, there isn't a tutorial on it as far as I know. But I created some examples for finding the Youtube "date published" text using TreeWalkers:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

UIA := UIA_Interface()
chromeId := WinExist("ahk_exe chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%
chrome := UIA.ElementFromHandle(chromeId, True)

; Find the date using the bullet character text element
bulletText := chrome.FindFirstByName("•") ; Find the bullet text element
TextWalker := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType", "Text")) ; Create a TreeWalker that only searches Text elements. If we used UIA.TreeWalkerTrue here, then it would return some invisible element that UIAViewer doesn't show
dateText := TextWalker.GetNextSiblingElement(bulletText) ; Get the next element over
MsgBox, % "Date: " dateText.Dump()
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%

; Find the date using the Like button as a reference
likeButton := chrome.FindFirstByNameAndType("like this video", "Button",,1) ; Find the "Like" button
dateText2 := TextWalker.GetPreviousSiblingElement(likeButton) ; Get the previous element. Notice that we don't have to climb the tree upwards here (use GetParentElement)
MsgBox, % "Date2: " dateText2.Dump()
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%

; Find the date using the player element and bullet text as references
nextText := chrome.FindFirstBy("AutomationId=player-container") ; Find the Youtube player-container element (the "complementary """ element)
while (nextText := TextWalker.GetNextSiblingElement(nextText)) { ; Loop until we find the bullet Text element
	if (nextText.CurrentName == "•") {
		dateText3 := TextWalker.GetNextSiblingElement(nextText) ; Then the next one over should be the date
		MsgBox, % "Date3: " dateText3.Dump()
		break
	}
}

ExitApp
This is excellent :) Thank you for commenting so precisely on all the examples. I just saw this one and the other examples you have.
Thanks again.

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 07 Jul 2022, 07:39

@Descolada
I have a sugestion... It would be great if in UIAviewer we could when inspecting a element, copy a function to the clipboard to click or to wait for the element. :geek:

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 08 Jul 2022, 01:08

I was playing around a bit and made this. Find the date of a video in an open Youtube window in the Chrome browser.
I don't know why but the above examples didn't work for me. The text - not recognized.
I would like to know if there is a way to 'walk' through the tree indicating only the number of times it should jump? I did the process twice because I needed to get to the date.

[Pardon my English].

Code: Select all

SetTitleMatchMode, 2
#Include <UIA_Interface>

UIA := UIA_Interface()

chromeId := WinExist("ahk_exe Chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%
chrome := UIA.ElementFromHandle(chromeId, True)

bulletText := chrome.FindFirstBy("Name=DISLIKE")
TextWalker := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType", "Text"))
Likes := TextWalker.GetPreviousSiblingElement(bulletText)
dateText1 := TextWalker.GetPreviousSiblingElement(Likes)
MsgBox, % dateText1.GetCurrentPropertyValueEx("Name")

Descolada
Posts: 1100
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Jul 2022, 09:40

@mora145, your version of the code works as well :) I have two thoughts why my examples might not have worked for you:
1) Since the code contains special characters (the bullet character), perhaps you haven't saved your script in the correct encoding "UTF-8-BOM". I tried with regular "UTF-8" and it threw an error.
2) You have an older UIA_Interface.ahk version

As for your question: no, currently there isn't such a method available and you do need to call "GetPreviousSiblingElement" twice. I think I will write a helper function for that soon enough though.

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 08 Jul 2022, 11:59

I'm having trouble with a website:

I Have an element that is nested by 9 levels.

UIAViewer and TreeInspector find this element with no trouble.
But when Im using FindFirst or WaitElement, those functions dont go deeply enough.

How can I set the scope to find this element?

panel
panel
document
group
group
group
group
group
button name: Anexar

Edit: I found something, if the button have a illegal character (like a font awesome icon) the uiaviewer and the treeinspector find the illegal character and the text, but the functions to find or wait only sees the illegal character ignoring the text after... @Descolada any tips to solve that?
Last edited by leosouza85 on 08 Jul 2022, 12:25, edited 1 time in total.

Descolada
Posts: 1100
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Jul 2022, 12:15

@leosouza85, have you tried element.FindFirstByNameAndType("Anexar","Button",,2,False)? This should make sure that you are not accidentally looking for "Anexar" if the element is actually named "Anexar " with a space at the end, and also turns off case-sensitivity.
I haven't encountered a situation where an element can't be found if its too deep, can you bring a testable example on some webpage?

Post Reply

Return to “Scripts and Functions (v1)”