UIAutomation with a focus on Chrome

[Skrell]

Anyone know/understand how/when to use the "cached" functions?

[pto]

Check the wiki

[pto]

@Descolada , in UIA_Browser->WaitPageLoad(), since i couln't use the function (locale) i had to debug and found the culprit

Code: Select all

stopButtonText := this.CustomNames.StopButtonName ? this.CustomNames.StopButtonName : stopButtonText

instead of

Code: Select all

stopButtonText := this.CustomNames.StopButtonText ? this.CustomNames.StopButtonText: stopButtonText

Edit:
Now reloadBut.CurrentFullDescription raises error.

[leosouza85]

Hi @Descolada I was thinking about the necessity of the sleep before clicking something that exists but is not clickable yet.

Did you tested the following hipothesys?

1 - waitelementexist

2 - then a loop trying to focus the element and test if it is focused

3 - and then click

I dont know if it is true but maybe the element is clickable and focusable at the same time, and the focus is possuble to test.

If this works, incorporate a wait focus inside the click...or as a parameter...

[pto]

@Descolada , in UIA_Browser->WaitPageLoad(), since i couln't use the function (locale) i had to debug and found the culprit

Code: Select all

stopButtonText := this.CustomNames.StopButtonName ? this.CustomNames.StopButtonName : stopButtonText

instead of

Code: Select all

stopButtonText := this.CustomNames.StopButtonText ? this.CustomNames.StopButtonText: stopButtonText

Edit:
Now reloadBut.CurrentFullDescription raises error.

For the meantime, this resolves. By replacing:

Code: Select all

if (!InStr(reloadBut.CurrentName, stopButtonText) && !InStr(legacyPattern.CurrentDescription, stopButtonText) && !InStr(reloadBut.CurrentFullDescription, stopButtonText))

with

Code: Select all

if !(InStr(reloadBut.CurrentName, stopButtonText) && !(InStr(legacyPattern.CurrentDescription, stopButtonText)) && !(winEl.BrowserType="Chrome") ? !(InStr(reloadBut.CurrentFullDescription, stopButtonText)) : "")

UIA_Browser.WaitPageLoad() works.

@leosouza85, i guess you can loop while refreshing you main element and checking PollForPotentialSupportedPatterns on the element in question, or passing IAccessible_Enum class flags to WaitElementExistByPath condition.

[Descolada]

@pto, I finally got around to updating UIA_Browser, so I removed language-specific elements altogether (among other things, for example timeouts were changed from 10s to indefinite, because it was causing problems with slow internet...). It means CustomNames shouldn't be necessary anymore and Reload button is detected in any language. Because I haven't tested it a ton, I uploaded it only to Github at the moment, but check it out.

@leosouza85, no I haven't tested such method. So far I've just checked whether the action of the button happened (waiting for some element with a short timeout). If you try it out, post the results here as well Detecting whether an element is "ready" or not would be very useful!

[leosouza85]

@pto, I finally got around to updating UIA_Browser, so I removed language-specific elements altogether (among other things, for example timeouts were changed from 10s to indefinite, because it was causing problems with slow internet...). It means CustomNames shouldn't be necessary anymore and Reload button is detected in any language. Because I haven't tested it a ton, I uploaded it only to Github at the moment, but check it out.

@leosouza85, no I haven't tested such method. So far I've just checked whether the action of the button happened (waiting for some element with a short timeout). If you try it out, post the results here as well Detecting whether an element is "ready" or not would be very useful!

I want to test it! But I don't remember right now any any element that need a sleep before clicking

[pto]

@pto, I finally got around to updating UIA_Browser, so I removed language-specific elements altogether (among other things, for example timeouts were changed from 10s to indefinite, because it was causing problems with slow internet...). It means CustomNames shouldn't be necessary anymore and Reload button is detected in any language. Because I haven't tested it a ton, I uploaded it only to Github at the moment, but check it out.

I guess it will have support for the three majors browsers
Much cleaner, well done! I haven't got the time to dig into, but for the moment i get a pointer not valid for UIA_Browser.ReloadButton at line 452, just by initiating the class.
Thank for your work man.
Win 8.1 Chrome 95.0.4638.69 AutoHotkey_1.1.33.10

I want to test it! But I don't remember right now any any element that need a sleep before clicking

Edit:
pointer not valid for UIA_Browser.ReloadButton at line 452, specifically GetFirstChildElement (UIATreeWalker). Latest UIA_Interface and UIA_Browser.

Edit:
Actually, i can initialize the class, but calling the new non locale dependent functions raises the error, even if i set the browser lang to en-US (updated Chrome to 104 just in case):

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

exeApp := "ahk_exe chrome.exe"
WinActivate, % exeApp
WinWaitActive, % exeApp

chromeWin := new UIA_Browser()

chromeWin.Back() ; or
chromeWin.Reload() ; throws pointer not valid exception

[fenchai]

I am trying to check if the element I am hovering at is a button or the name, etc but I do not know how to access it?
I only know that I can find the whole tree with:

- FindFirstByName
- FindFirstByNameAndType
- FindFirstByType
- FindAllBy

but I don't know how to access the element I am hovering at. How can I access it?

[pto]

Cant you tell if the element has ControlType and/or a Name? There are more properties ids but these are the most common.

Wich application?

By hovering, do you mean your using UIAViewer.ahk?

Anyway, a common way to get a button called "myButton" is

Code: Select all

#include <UIA_Interface>

exeApp := "ahk_exe chrome.exe" ; chrome is set for example and must be changed to suit needs
WinActivate, % exeApp
WinWaitActive, % exeApp

UIA := new UIA_Interface()

windowEl := UIA.ElementFromHandle()

foundEl := windowEl.FindFirstByNameAndType("myButton", "Button")
MsgBox % "button found: " foundEl.Dump()

foundEl := windowEl.FindFirstByName("myButton")
MsgBox % "button found: " foundEl.Dump()

foundEl := windowEl.FindFirstBy("Name=myButton AND ControlType=Button")
MsgBox % "button found: " foundEl.Dump()

[fenchai]

@pto

no, what I mean is I want to check if the element my mouse is hovering over is the name and control type I want.

Code: Select all

MButton::
check if element over mouse is of name="Running applications" & controltype="Toolbar".
if element matches then:
do something
else
send MButton

[pto]

I understood. Have you tried MouseGetPos?

Code: Select all

Loop
{
	MouseGetPos, , , , control

	if (control = "MSTaskListWClass1")
	{
		; found, do stuff
		Break
	}
	Sleep, 50
}

ExitApp

F5::ExitApp

Just make sure that class name MSTaskListWClass1 matches your windows version

[Descolada]

@fenchai, you are looking for ElementFromPoint.

Code: Select all

UIA := UIA_Interface()
Loop {
    ToolTip, % UIA.ElementFromPoint().Dump()
}

By default it gets the element from under the mouse, but you can also provide X and Y coordinates to get an element from any point. Note that this doesn't always return the "deepest" element, so I also created the SmallestElementFromPoint which sometimes might be useful.

@pto, what is reported for cUIA.URLEditElement.Dump()? I did use the newer FindFirstWithOptions which might have been added in Windows 10, so I'm just guessing that might be the problem (and is used to find the URLEditElement).

[pto]

Hi @Descolada, URLEditElement.Dump() is empty, no error raised.

Here's the simple test case i'm using, in case there's some typo or something (it starts to be that hour )

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

browser := "ahk_exe chrome.exe" ; v104 en-US
WinActivate, %browser%
WinWaitActive, %browser%

browserEl := new UIA_Browser()
MsgBox % browserEl.URLEditElement.Dump()

I did use the newer FindFirstWithOptions which might have been added in Windows 10

I see what you mean. If it's the case the previous version works for me, at least for the meantime (i will get rid of win 8.1 ).

Mind the loop

Code: Select all

UIA := UIA_Interface()
Loop {
    ToolTip, % UIA.ElementFromPoint().Dump()
}
; always good idea
F1::ExitApp

Edit:
ElementFromPoint works on Win 8.1. I'll keep posting if i find anything else.

[fenchai]

thanks @Descolada

Code: Select all

UIA.ElementFromPoint().Dump()

the above works perfectly, but how do you extract the values?

Code: Select all

Type: 50033 (Pane) Name: "Taskbar" LocalizedControlType: "pane" ClassName: "Shell_TrayWnd"

it seeems to be a string and not an array nor map, do I need to parse it myself?

[guest3456]

By default it gets the element from under the mouse, but you can also provide X and Y coordinates to get an element from any point. Note that [ElementFromPoint] doesn't always return the "deepest" element, so I also created the SmallestElementFromPoint which sometimes might be useful.

i've noticed this too sometimes... why might this be?

[Descolada]

@fenchai, yes, Dump() returns a string with some relevant information. You can extract those values by getting the property of interest:

Code: Select all

element := UIA.ElementFromPoint()
element.Name ; returns the Name property
element.Type ; Type property

A more complete list of element and control pattern properties is in the wiki: https://github.com/Descolada/UIAutomation/wiki/04.-Elements

@guest3456, you would need to ask Microsoft that It appears that some Win32 functions have similarly odd behavior as discussed in this forum post.

[mora145]

Hello.
I am trying to wait for a text on screen using

Code: Select all

if (chrome.WaitElementExist("Name=11 suggestions available",,,,8000)){
    MsgBox, Estoy
}

The problem is that the number 11 changes, so I should wait only for suggestions available but I try to do "Name=*suggestions available" and it doesn't detect the * as a wildcard. What can I do?

EDIT:
In the wiky says: Escape ' with \, escape \ with \ but I don't quite understand. My English is limited.

Solved: (Sorry my noob) I had forgotten that there are several search modes, I just changed this to mode 2.

1 = name must must start with the specified name
2 = can contain anywhere
3 = exact match (Default)
RegEx=using regular expression. In this case the Name can't be empty.

Code: Select all

chrome.WaitElementExist("Name=suggestions available",,2,,8000)

[pto]

@Descolada , i'm using the latest version, though i'm making minor adjustments:

Code: Select all

; had to change UIA_Browser.NewTab() to
TabBarButtons := this.TabBarElement.FindAll(this.ButtonCondition)
TabBarButtons[TabBarButtons.MaxIndex()].Click() ; reasonnable

; and UIA_Browser.GetCurrentReloadButton() to
TabBarButtons := this.NavigationBarElement.FindAll(this.ButtonCondition)
this.ReloadButton := TabBarButtons[3] ; closest i can get without localization

Obviously this is not ideal for i'll have to apply whatever i may change to the following updates.
FindAllWithOptions is an optimal solution, so i'm not giving up yet.


Hi @mora145 , aren't we all noobs somehow?
The important is that you resolved your issue.
Also, providing multiple element properties, if they exist in the element you search for, can avoid other elements to correspond to the condition supplied, specially in search mode 2. Check your element and see if you can find one static property, something like:

Code: Select all

chrome.WaitElementExist("Name='suggestions available' AND ControlType=Text",,2,,8000)

Check Descolada's previous post for more info.

Cheers

[mora145]

@pto That's true

Yes, then I added a ControlType for text, but I did a search with regular expressions: It was better for the purpose I needed.

Code: Select all

chrome.FindFirstBy("Name='\d+ suggestions available' AND ControlType=Text",,"RegEx")

Thank you for your recommendation.