UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 31 Aug 2022, 16:04

Anyone know/understand how/when to use the "cached" functions?

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 01 Sep 2022, 00:10

Check the wiki

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 01 Sep 2022, 01:51

@Descolada , in UIA_Browser->WaitPageLoad(), since i couln't use the function (locale) i had to debug and found the culprit ;)

Code: Select all

stopButtonText := this.CustomNames.StopButtonName ? this.CustomNames.StopButtonName : stopButtonText
instead of

Code: Select all

stopButtonText := this.CustomNames.StopButtonText ? this.CustomNames.StopButtonText: stopButtonText
Edit:
Now reloadBut.CurrentFullDescription raises error.

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 01 Sep 2022, 09:32

Hi @Descolada I was thinking about the necessity of the sleep before clicking something that exists but is not clickable yet.

Did you tested the following hipothesys?

1 - waitelementexist

2 - then a loop trying to focus the element and test if it is focused

3 - and then click

I dont know if it is true but maybe the element is clickable and focusable at the same time, and the focus is possuble to test.

If this works, incorporate a wait focus inside the click...or as a parameter...

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 01 Sep 2022, 11:02

pto wrote:
01 Sep 2022, 01:51
@Descolada , in UIA_Browser->WaitPageLoad(), since i couln't use the function (locale) i had to debug and found the culprit ;)

Code: Select all

stopButtonText := this.CustomNames.StopButtonName ? this.CustomNames.StopButtonName : stopButtonText
instead of

Code: Select all

stopButtonText := this.CustomNames.StopButtonText ? this.CustomNames.StopButtonText: stopButtonText
Edit:
Now reloadBut.CurrentFullDescription raises error.
For the meantime, this resolves. By replacing:

Code: Select all

if (!InStr(reloadBut.CurrentName, stopButtonText) && !InStr(legacyPattern.CurrentDescription, stopButtonText) && !InStr(reloadBut.CurrentFullDescription, stopButtonText))
with

Code: Select all

if !(InStr(reloadBut.CurrentName, stopButtonText) && !(InStr(legacyPattern.CurrentDescription, stopButtonText)) && !(winEl.BrowserType="Chrome") ? !(InStr(reloadBut.CurrentFullDescription, stopButtonText)) : "")
UIA_Browser.WaitPageLoad() works.

@leosouza85, i guess you can loop while refreshing you main element and checking PollForPotentialSupportedPatterns on the element in question, or passing IAccessible_Enum class flags to WaitElementExistByPath condition.

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 01 Sep 2022, 13:20

@pto, I finally got around to updating UIA_Browser, so I removed language-specific elements altogether (among other things, for example timeouts were changed from 10s to indefinite, because it was causing problems with slow internet...). It means CustomNames shouldn't be necessary anymore and Reload button is detected in any language. Because I haven't tested it a ton, I uploaded it only to Github at the moment, but check it out.

@leosouza85, no I haven't tested such method. So far I've just checked whether the action of the button happened (waiting for some element with a short timeout). If you try it out, post the results here as well :) Detecting whether an element is "ready" or not would be very useful!

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 01 Sep 2022, 13:34

Descolada wrote:
01 Sep 2022, 13:20
@pto, I finally got around to updating UIA_Browser, so I removed language-specific elements altogether (among other things, for example timeouts were changed from 10s to indefinite, because it was causing problems with slow internet...). It means CustomNames shouldn't be necessary anymore and Reload button is detected in any language. Because I haven't tested it a ton, I uploaded it only to Github at the moment, but check it out.

@leosouza85, no I haven't tested such method. So far I've just checked whether the action of the button happened (waiting for some element with a short timeout). If you try it out, post the results here as well :) Detecting whether an element is "ready" or not would be very useful!

I want to test it! But I don't remember right now any any element that need a sleep before clicking

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 01 Sep 2022, 14:55

Descolada wrote:
01 Sep 2022, 13:20
@pto, I finally got around to updating UIA_Browser, so I removed language-specific elements altogether (among other things, for example timeouts were changed from 10s to indefinite, because it was causing problems with slow internet...). It means CustomNames shouldn't be necessary anymore and Reload button is detected in any language. Because I haven't tested it a ton, I uploaded it only to Github at the moment, but check it out.
I guess it will have support for the three majors browsers :crazy:
Much cleaner, well done! I haven't got the time to dig into, but for the moment i get a pointer not valid for UIA_Browser.ReloadButton at line 452, just by initiating the class.
Thank for your work man. :clap:
Win 8.1 Chrome 95.0.4638.69 AutoHotkey_1.1.33.10
leosouza85 wrote: I want to test it! But I don't remember right now any any element that need a sleep before clicking
:lol:

Edit:
pointer not valid for UIA_Browser.ReloadButton at line 452, specifically GetFirstChildElement (UIATreeWalker). Latest UIA_Interface and UIA_Browser.

Edit:
Actually, i can initialize the class, but calling the new non locale dependent functions raises the error, even if i set the browser lang to en-US (updated Chrome to 104 just in case):

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

exeApp := "ahk_exe chrome.exe"
WinActivate, % exeApp
WinWaitActive, % exeApp

chromeWin := new UIA_Browser()

chromeWin.Back() ; or
chromeWin.Reload() ; throws pointer not valid exception

fenchai
Posts: 290
Joined: 28 Mar 2016, 07:57

Re: UIAutomation with a focus on Chrome

Post by fenchai » 01 Sep 2022, 20:30

I am trying to check if the element I am hovering at is a button or the name, etc but I do not know how to access it?
I only know that I can find the whole tree with:

- FindFirstByName
- FindFirstByNameAndType
- FindFirstByType
- FindAllBy

but I don't know how to access the element I am hovering at. How can I access it?

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 01 Sep 2022, 21:08

Cant you tell if the element has ControlType and/or a Name? There are more properties ids but these are the most common.

Wich application?

By hovering, do you mean your using UIAViewer.ahk?

Anyway, a common way to get a button called "myButton" is

Code: Select all

#include <UIA_Interface>

exeApp := "ahk_exe chrome.exe" ; chrome is set for example and must be changed to suit needs
WinActivate, % exeApp
WinWaitActive, % exeApp

UIA := new UIA_Interface()

windowEl := UIA.ElementFromHandle()

foundEl := windowEl.FindFirstByNameAndType("myButton", "Button")
MsgBox % "button found: " foundEl.Dump()

foundEl := windowEl.FindFirstByName("myButton")
MsgBox % "button found: " foundEl.Dump()

foundEl := windowEl.FindFirstBy("Name=myButton AND ControlType=Button")
MsgBox % "button found: " foundEl.Dump()


fenchai
Posts: 290
Joined: 28 Mar 2016, 07:57

Re: UIAutomation with a focus on Chrome

Post by fenchai » 01 Sep 2022, 21:34

@pto

no, what I mean is I want to check if the element my mouse is hovering over is the name and control type I want.

Code: Select all

MButton::
check if element over mouse is of name="Running applications" & controltype="Toolbar".
if element matches then:
do something
else
send MButton

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 01 Sep 2022, 22:40

I understood. Have you tried MouseGetPos?

Code: Select all

Loop
{
	MouseGetPos, , , , control

	if (control = "MSTaskListWClass1")
	{
		; found, do stuff
		Break
	}
	Sleep, 50
}

ExitApp

F5::ExitApp
Just make sure that class name MSTaskListWClass1 matches your windows version

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 01 Sep 2022, 23:13

@fenchai, you are looking for ElementFromPoint.

Code: Select all

UIA := UIA_Interface()
Loop {
    ToolTip, % UIA.ElementFromPoint().Dump()
}
By default it gets the element from under the mouse, but you can also provide X and Y coordinates to get an element from any point. Note that this doesn't always return the "deepest" element, so I also created the SmallestElementFromPoint which sometimes might be useful.

@pto, what is reported for cUIA.URLEditElement.Dump()? I did use the newer FindFirstWithOptions which might have been added in Windows 10, so I'm just guessing that might be the problem (and is used to find the URLEditElement).

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 02 Sep 2022, 00:13

Hi @Descolada, URLEditElement.Dump() is empty, no error raised.

Here's the simple test case i'm using, in case there's some typo or something (it starts to be that hour :D )

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

browser := "ahk_exe chrome.exe" ; v104 en-US
WinActivate, %browser%
WinWaitActive, %browser%

browserEl := new UIA_Browser()
MsgBox % browserEl.URLEditElement.Dump()
Descolada wrote: I did use the newer FindFirstWithOptions which might have been added in Windows 10
I see what you mean. If it's the case the previous version works for me, at least for the meantime (i will get rid of win 8.1 :D).

Mind the loop

Code: Select all

UIA := UIA_Interface()
Loop {
    ToolTip, % UIA.ElementFromPoint().Dump()
}
; always good idea
F1::ExitApp
Edit:
ElementFromPoint works on Win 8.1. I'll keep posting if i find anything else.

fenchai
Posts: 290
Joined: 28 Mar 2016, 07:57

Re: UIAutomation with a focus on Chrome

Post by fenchai » 02 Sep 2022, 10:31

thanks @Descolada

Code: Select all

UIA.ElementFromPoint().Dump()
the above works perfectly, but how do you extract the values?

Code: Select all

Type: 50033 (Pane) Name: "Taskbar" LocalizedControlType: "pane" ClassName: "Shell_TrayWnd"
it seeems to be a string and not an array nor map, do I need to parse it myself?

guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: UIAutomation with a focus on Chrome

Post by guest3456 » 02 Sep 2022, 12:34

Descolada wrote:
01 Sep 2022, 23:13
By default it gets the element from under the mouse, but you can also provide X and Y coordinates to get an element from any point. Note that [ElementFromPoint] doesn't always return the "deepest" element, so I also created the SmallestElementFromPoint which sometimes might be useful.
i've noticed this too sometimes... why might this be?


Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 02 Sep 2022, 13:11

@fenchai, yes, Dump() returns a string with some relevant information. You can extract those values by getting the property of interest:

Code: Select all

element := UIA.ElementFromPoint()
element.Name ; returns the Name property
element.Type ; Type property
A more complete list of element and control pattern properties is in the wiki: https://github.com/Descolada/UIAutomation/wiki/04.-Elements

@guest3456, you would need to ask Microsoft that :) It appears that some Win32 functions have similarly odd behavior as discussed in this forum post.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 02 Sep 2022, 18:03

Hello.
I am trying to wait for a text on screen using

Code: Select all

if (chrome.WaitElementExist("Name=11 suggestions available",,,,8000)){
    MsgBox, Estoy
}
The problem is that the number 11 changes, so I should wait only for suggestions available but I try to do "Name=*suggestions available" and it doesn't detect the * as a wildcard. What can I do?

EDIT:
In the wiky says: Escape ' with \, escape \ with \ but I don't quite understand. My English is limited.

Solved: (Sorry my noob) I had forgotten that there are several search modes, I just changed this to mode 2.

1 = name must must start with the specified name
2 = can contain anywhere
3 = exact match (Default)
RegEx=using regular expression. In this case the Name can't be empty.

Code: Select all

chrome.WaitElementExist("Name=suggestions available",,2,,8000)

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 03 Sep 2022, 02:15

@Descolada , i'm using the latest version, though i'm making minor adjustments:

Code: Select all

; had to change UIA_Browser.NewTab() to
TabBarButtons := this.TabBarElement.FindAll(this.ButtonCondition)
TabBarButtons[TabBarButtons.MaxIndex()].Click() ; reasonnable

; and UIA_Browser.GetCurrentReloadButton() to
TabBarButtons := this.NavigationBarElement.FindAll(this.ButtonCondition)
this.ReloadButton := TabBarButtons[3] ; closest i can get without localization
Obviously this is not ideal for i'll have to apply whatever i may change to the following updates. :crazy:
FindAllWithOptions is an optimal solution, so i'm not giving up yet. :)


Hi @mora145 , aren't we all noobs somehow? :)
The important is that you resolved your issue.
Also, providing multiple element properties, if they exist in the element you search for, can avoid other elements to correspond to the condition supplied, specially in search mode 2. Check your element and see if you can find one static property, something like:

Code: Select all

chrome.WaitElementExist("Name='suggestions available' AND ControlType=Text",,2,,8000)
Check Descolada's previous post for more info.

Cheers

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 03 Sep 2022, 15:27

@pto That's true :)

Yes, then I added a ControlType for text, but I did a search with regular expressions: It was better for the purpose I needed.

Code: Select all

chrome.FindFirstBy("Name='\d+ suggestions available' AND ControlType=Text",,"RegEx")
Thank you for your recommendation.

Post Reply

Return to “Scripts and Functions (v1)”