UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: UIAutomation with a focus on Chrome

Post by guest3456 » 26 Aug 2022, 00:15

wondering if anyone has run into this issue

- i can do ElementFromPoint fine, and it retrieves an Edit element
- i can then do Element.GetCurrentPatternAs("Value") and successfully get the Value pattern
- but when i try to do ValuePattern.SetValue(xxx) i get an exception thrown:

error 0x80020003 which googling says DISP_E_MEMBERNOTFOUND

i know that pattern works because i can do ValuePattern.CurrentValue and it returns the correct value

i get the same exception if i try with LegacyIaccessible pattern too. error thrown when attempting SetValue

same SetValue code works in most places. only fails in this on particular desktop app


Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 26 Aug 2022, 00:26

@guest3456, this most likely means that either:
1) The element you got isn't the correct element. For example, sometimes Edit elements have Edit subelements, so try DumpAll() with the found element. Also make sure that once the focus is in the element, you still have the correct element. In some browsers, the URL bar for example is made of nested Edits, where you can access the deeper one once you've set focus to the topmost Edit...
2) The program hasn't implemented ValuePattern correctly. Unfortunately this sometimes happens, I've had a similar problem with Vivaldi. As a workaround you can use SetFocus along with Send/ControlSend.

guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: UIAutomation with a focus on Chrome

Post by guest3456 » 26 Aug 2022, 08:26

Descolada wrote:
26 Aug 2022, 00:26
@guest3456, this most likely means that either:
1) The element you got isn't the correct element. For example, sometimes Edit elements have Edit subelements, so try DumpAll() with the found element. Also make sure that once the focus is in the element, you still have the correct element. In some browsers, the URL bar for example is made of nested Edits, where you can access the deeper one once you've set focus to the topmost Edit...
fairly sure i got the correct element, because both of these work correctly and return the existing edit value:

Code: Select all

   msgbox % element.GetCurrentPropertyValue(30045)
   ;or
   valueptrn := element.GetCurrentPatternAs("Value")
   msgbox % valueptrn.CurrentValue
   ; valueptrn.SetValue(0.23) ;EXCEPTION 0x80020003 
havent tested DumpAll
Descolada wrote:
26 Aug 2022, 00:26
2) The program hasn't implemented ValuePattern correctly. Unfortunately this sometimes happens, I've had a similar problem with Vivaldi. As a workaround you can use SetFocus along with Send/ControlSend.
yes this is what i'm doing as a fallback. i wasn't aware that a program even had a choice about not implementing a pattern. i figured once they enabled the UIA interface, then the rest would all be handled by windows. its strange because i get the same exception with the LegacyIAccessible pattern too. but yeah i originally concluded this too, since it only fails on this one app


pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 28 Aug 2022, 01:24

Hello world :D

I´m not able to use WaitPageLoad() in UIA_Browser.ahk accordingly.

I can't get values for legacyPattern.CurrentDescription nor reloadBut.CurrentFullDescription properties.

It seems to me (no expert) that the exit function occurs at the timer and will never break in the loop, or am i missing something?

If someone has a hint i would very much appreciate.

Cheers

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 28 Aug 2022, 01:50

Nevermind the loop (must go to bed :yawn: ), it's just the property error.

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 28 Aug 2022, 02:12

@pto, the loop will break either on the timeout, or once the "Stop" button text isn't found in the descriptions of the reload button.
The way I got the text for the stop button is with something like this:

Code: Select all

#Include <UIA_Interface>

browser := "ahk_exe chrome.exe"
WinActivate, %browser%
WinWaitActive, %browser%
UIA := UIA_Interface()
chrome := UIA.ElementFromHandle(browser)
reloadBut := chrome.FindFirstByNameAndType("Reload", "Button")
Loop {
    ToolTip, % reloadBut.DumpAll() " FullDescription: " reloadBut.FullDescription " Description: " reloadBut.LegacyIAccessiblePattern.Description
}
F5::ExitApp
Chrome's stop button is identifiable by the LegacyIAccessiblePattern.Description, and unfortunately I haven't been able to find a better way of checking for "page load complete" other this or waiting for an identifying element with WaitElementExist.
For Edge the FullDescription property is used, it is empty in Chrome. To test with Edge, change browser := "ahk_exe msedge.exe" and reloadBut := chrome.FindFirstByNameAndType("Refresh", "Button").

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 28 Aug 2022, 13:49

Descolada wrote: @pto, the loop will break either on the timeout, or once the "Stop" button text isn't found in the descriptions of the reload button.
Sorry for the brain chrash, it's was late (note to myself: it doesn't make a difference triple checking if your brain is malfunctioning). :D
Descolada wrote: Chrome's stop button is identifiable by the LegacyIAccessiblePattern
Thank you @Descolada , LegacyIAccessiblePattern is indeed the property i was loking for.
I haven't been able to find a better way of checking for "page load complete" other this or waiting for an identifying element with WaitElementExist.
Neither did i, your way beats, as far as I'm concerned, a number of methods found on the web (pixel search, etc.). I'm already using WaitElementExist for specific pages, but a general approach it's always welcome, so i appreciate the share.

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 28 Aug 2022, 14:27

Descolada wrote: For Edge the FullDescription property is used, it is empty in Chrome
I still haven't explained my resolved issue properly and for the sake of completeness, thats where i kept trying without suspicion of an empty property from Chrome (although i should have checked)
legacyPattern.CurrentFullDescription instead of legacyPattern.CurrentDescription

tester
Posts: 84
Joined: 10 Jun 2021, 23:03

Re: UIAutomation with a focus on Chrome

Post by tester » 28 Aug 2022, 21:42

I'm trying to check if a certain DOM element exists in a web page using UIA_Browser through JS rather than checking UIA elements as the page can be in different languages with the same structure.

What I'm currently doing is something like this.

Code: Select all

cUIA.SetURL("https://www.google.com", True)
cUIA.WaitPageLoad()
sSelector := "#gbqfbb"
js =
(LTrim
	(() => {
		return document.querySelector("%sSelector%") !== null ? 1 : 0
	})()
)
msgbox % "The element, " sSelector ", exists: " (cUIA.JSReturnThroughClipboard(js))
Is there a better way, especially by not using the clipboard?

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 28 Aug 2022, 23:15

@tester, currently there are only two ways to return info from JS functions: either through the clipboard, or the window title. Neither are good ways, but browsers are sandboxed in such a way that communication with the "outside world" is very difficult - of course I wouldn't want a random website sending messages to my computer or executing random commands through cmd.exe for example! So these limitations make sense.
UIA isn't meant for executing JS anyway, what you are trying to do is better done with Chrome.ahk (using Websocket to communicate with external programs) or Rufaydium.

Perhaps some language-independent elements can be identified on the page, or a certain order of elements? Eg "the third image element needs to be followed by a button and the second checkbox by an edit element" or something.

tester
Posts: 84
Joined: 10 Jun 2021, 23:03

Re: UIAutomation with a focus on Chrome

Post by tester » 29 Aug 2022, 02:14

@DescoladaOK, thank you.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 29 Aug 2022, 13:29

Anyone know how to find the number of found elements that meet a specific condition using something like FindAll?

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 29 Aug 2022, 13:33

@Skrell, FindAll returns an array, so Array.MaxIndex() should give you the number of returned elements.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 29 Aug 2022, 14:00

Descolada wrote:
29 Aug 2022, 13:33
@Skrell, FindAll returns an array, so Array.MaxIndex() should give you the number of returned elements.
tytyty!

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 29 Aug 2022, 15:55

Descolada wrote:
29 Aug 2022, 13:33
@Skrell, FindAll returns an array, so Array.MaxIndex() should give you the number of returned elements.
I'm trying to count all the running buttons in the taskbar, any idea why this isn't working?

Code: Select all

tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
c := UIA.CreateCondition("Name==running AND ControlType==Button",,2)
winCount := (tbEl.FindAll(c, 0x4)).MaxIndex()

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 29 Aug 2022, 19:06

Skrell, this works for me

Code: Select all

UIA := UIA_Interface()
tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
toolbarEl := tbEl.FindFirstBy("ClassName=MSTaskSwWClass AND ControlType=Pane")
winCount := (toolbarEl.FindAll("ControlType=Button")).MaxIndex()
Wath out for the == in the condition since it's a comparison operator, instead of = to assign a value to a property Id.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 30 Aug 2022, 09:35

pto wrote:
29 Aug 2022, 19:06
Skrell, this works for me

Code: Select all

UIA := UIA_Interface()
tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
toolbarEl := tbEl.FindFirstBy("ClassName=MSTaskSwWClass AND ControlType=Pane")
winCount := (toolbarEl.FindAll("ControlType=Button")).MaxIndex()
Wath out for the == in the condition since it's a comparison operator, instead of = to assign a value to a property Id.
Unfortunately this code does not work for me. When I printout the results of winCount I don't even get 0, I get literally nothing. Next, I changed the code to the following (as I wasn't sure why you were looking for ClassName=MSTaskSwWClass ?) and this also does not work:

Code: Select all

    tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
    toolbarEl := tbEl.FindFirstBy("Name=Taskbar AND ControlType=Pane")
    winCount := (toolbarEl.FindAll("ControlType=Button")).MaxIndex()
    tooltip % winCount 

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 30 Aug 2022, 10:37

@Skrell, try

Code: Select all

UIA := UIA_Interface()
tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
toolbarEl := tbEl.FindFirstBy("ClassName=MSTaskListWClass AND ControlType=Toolbar")
windowEls := toolbarEl ? toolbarEl.FindAll() : tbEl.FindAllBy("ClassName=Taskbar.TaskListButtonAutomationPeer")

result := "Found " windowEls.MaxIndex() " buttons`n`n"
for i, el in windowEls
    result .= i ": " el.Dump() "`n"

MsgBox, % result
ExitApp
Worked on Win 10 (using ClassName MSTaskListWClass) and Win 11 (finding all ClassName=Taskbar.TaskListButtonAutomationPeer buttons).

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 30 Aug 2022, 12:18

Descolada wrote:
30 Aug 2022, 10:37
@Skrell, try

Code: Select all

UIA := UIA_Interface()
tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
toolbarEl := tbEl.FindFirstBy("ClassName=MSTaskListWClass AND ControlType=Toolbar")
windowEls := toolbarEl ? toolbarEl.FindAll() : tbEl.FindAllBy("ClassName=Taskbar.TaskListButtonAutomationPeer")

result := "Found " windowEls.MaxIndex() " buttons`n`n"
for i, el in windowEls
    result .= i ": " el.Dump() "`n"

MsgBox, % result
ExitApp
Worked on Win 10 (using ClassName MSTaskListWClass) and Win 11 (finding all ClassName=Taskbar.TaskListButtonAutomationPeer buttons).
Beautiful thank you! Works!

pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 30 Aug 2022, 12:28

Yes, it's worth mentionning that i tried on Win 8.1 (don't ask :D ).

@Skrell , i'm glad you sort that out.

I also take the opportunity to thank you mods, for all the work you do.

Post Reply

Return to “Scripts and Functions (v1)”