UIAutomation with a focus on Chrome
Re: UIAutomation with a focus on Chrome
wondering if anyone has run into this issue
- i can do ElementFromPoint fine, and it retrieves an Edit element
- i can then do Element.GetCurrentPatternAs("Value") and successfully get the Value pattern
- but when i try to do ValuePattern.SetValue(xxx) i get an exception thrown:
error 0x80020003 which googling says DISP_E_MEMBERNOTFOUND
i know that pattern works because i can do ValuePattern.CurrentValue and it returns the correct value
i get the same exception if i try with LegacyIaccessible pattern too. error thrown when attempting SetValue
same SetValue code works in most places. only fails in this on particular desktop app
- i can do ElementFromPoint fine, and it retrieves an Edit element
- i can then do Element.GetCurrentPatternAs("Value") and successfully get the Value pattern
- but when i try to do ValuePattern.SetValue(xxx) i get an exception thrown:
error 0x80020003 which googling says DISP_E_MEMBERNOTFOUND
i know that pattern works because i can do ValuePattern.CurrentValue and it returns the correct value
i get the same exception if i try with LegacyIaccessible pattern too. error thrown when attempting SetValue
same SetValue code works in most places. only fails in this on particular desktop app
Re: UIAutomation with a focus on Chrome
@guest3456, this most likely means that either:
1) The element you got isn't the correct element. For example, sometimes Edit elements have Edit subelements, so try DumpAll() with the found element. Also make sure that once the focus is in the element, you still have the correct element. In some browsers, the URL bar for example is made of nested Edits, where you can access the deeper one once you've set focus to the topmost Edit...
2) The program hasn't implemented ValuePattern correctly. Unfortunately this sometimes happens, I've had a similar problem with Vivaldi. As a workaround you can use SetFocus along with Send/ControlSend.
1) The element you got isn't the correct element. For example, sometimes Edit elements have Edit subelements, so try DumpAll() with the found element. Also make sure that once the focus is in the element, you still have the correct element. In some browsers, the URL bar for example is made of nested Edits, where you can access the deeper one once you've set focus to the topmost Edit...
2) The program hasn't implemented ValuePattern correctly. Unfortunately this sometimes happens, I've had a similar problem with Vivaldi. As a workaround you can use SetFocus along with Send/ControlSend.
Re: UIAutomation with a focus on Chrome
fairly sure i got the correct element, because both of these work correctly and return the existing edit value:Descolada wrote: ↑26 Aug 2022, 00:26@guest3456, this most likely means that either:
1) The element you got isn't the correct element. For example, sometimes Edit elements have Edit subelements, so try DumpAll() with the found element. Also make sure that once the focus is in the element, you still have the correct element. In some browsers, the URL bar for example is made of nested Edits, where you can access the deeper one once you've set focus to the topmost Edit...
Code: Select all
msgbox % element.GetCurrentPropertyValue(30045)
;or
valueptrn := element.GetCurrentPatternAs("Value")
msgbox % valueptrn.CurrentValue
; valueptrn.SetValue(0.23) ;EXCEPTION 0x80020003
yes this is what i'm doing as a fallback. i wasn't aware that a program even had a choice about not implementing a pattern. i figured once they enabled the UIA interface, then the rest would all be handled by windows. its strange because i get the same exception with the LegacyIAccessible pattern too. but yeah i originally concluded this too, since it only fails on this one app
Re: UIAutomation with a focus on Chrome
Hello world
I´m not able to use WaitPageLoad() in UIA_Browser.ahk accordingly.
I can't get values for legacyPattern.CurrentDescription nor reloadBut.CurrentFullDescription properties.
It seems to me (no expert) that the exit function occurs at the timer and will never break in the loop, or am i missing something?
If someone has a hint i would very much appreciate.
Cheers
I´m not able to use WaitPageLoad() in UIA_Browser.ahk accordingly.
I can't get values for legacyPattern.CurrentDescription nor reloadBut.CurrentFullDescription properties.
It seems to me (no expert) that the exit function occurs at the timer and will never break in the loop, or am i missing something?
If someone has a hint i would very much appreciate.
Cheers
Re: UIAutomation with a focus on Chrome
Nevermind the loop (must go to bed ), it's just the property error.
Re: UIAutomation with a focus on Chrome
@pto, the loop will break either on the timeout, or once the "Stop" button text isn't found in the descriptions of the reload button.
The way I got the text for the stop button is with something like this:
Chrome's stop button is identifiable by the LegacyIAccessiblePattern.Description, and unfortunately I haven't been able to find a better way of checking for "page load complete" other this or waiting for an identifying element with WaitElementExist.
For Edge the FullDescription property is used, it is empty in Chrome. To test with Edge, change browser := "ahk_exe msedge.exe" and reloadBut := chrome.FindFirstByNameAndType("Refresh", "Button").
The way I got the text for the stop button is with something like this:
Code: Select all
#Include <UIA_Interface>
browser := "ahk_exe chrome.exe"
WinActivate, %browser%
WinWaitActive, %browser%
UIA := UIA_Interface()
chrome := UIA.ElementFromHandle(browser)
reloadBut := chrome.FindFirstByNameAndType("Reload", "Button")
Loop {
ToolTip, % reloadBut.DumpAll() " FullDescription: " reloadBut.FullDescription " Description: " reloadBut.LegacyIAccessiblePattern.Description
}
F5::ExitApp
For Edge the FullDescription property is used, it is empty in Chrome. To test with Edge, change browser := "ahk_exe msedge.exe" and reloadBut := chrome.FindFirstByNameAndType("Refresh", "Button").
Re: UIAutomation with a focus on Chrome
Sorry for the brain chrash, it's was late (note to myself: it doesn't make a difference triple checking if your brain is malfunctioning).Descolada wrote: @pto, the loop will break either on the timeout, or once the "Stop" button text isn't found in the descriptions of the reload button.
Thank you @Descolada , LegacyIAccessiblePattern is indeed the property i was loking for.Descolada wrote: Chrome's stop button is identifiable by the LegacyIAccessiblePattern
Neither did i, your way beats, as far as I'm concerned, a number of methods found on the web (pixel search, etc.). I'm already using WaitElementExist for specific pages, but a general approach it's always welcome, so i appreciate the share.I haven't been able to find a better way of checking for "page load complete" other this or waiting for an identifying element with WaitElementExist.
Re: UIAutomation with a focus on Chrome
I still haven't explained my resolved issue properly and for the sake of completeness, thats where i kept trying without suspicion of an empty property from Chrome (although i should have checked)Descolada wrote: For Edge the FullDescription property is used, it is empty in Chrome
legacyPattern.CurrentFullDescription instead of legacyPattern.CurrentDescription
Re: UIAutomation with a focus on Chrome
I'm trying to check if a certain DOM element exists in a web page using UIA_Browser through JS rather than checking UIA elements as the page can be in different languages with the same structure.
What I'm currently doing is something like this.
Is there a better way, especially by not using the clipboard?
What I'm currently doing is something like this.
Code: Select all
cUIA.SetURL("https://www.google.com", True)
cUIA.WaitPageLoad()
sSelector := "#gbqfbb"
js =
(LTrim
(() => {
return document.querySelector("%sSelector%") !== null ? 1 : 0
})()
)
msgbox % "The element, " sSelector ", exists: " (cUIA.JSReturnThroughClipboard(js))
Re: UIAutomation with a focus on Chrome
@tester, currently there are only two ways to return info from JS functions: either through the clipboard, or the window title. Neither are good ways, but browsers are sandboxed in such a way that communication with the "outside world" is very difficult - of course I wouldn't want a random website sending messages to my computer or executing random commands through cmd.exe for example! So these limitations make sense.
UIA isn't meant for executing JS anyway, what you are trying to do is better done with Chrome.ahk (using Websocket to communicate with external programs) or Rufaydium.
Perhaps some language-independent elements can be identified on the page, or a certain order of elements? Eg "the third image element needs to be followed by a button and the second checkbox by an edit element" or something.
UIA isn't meant for executing JS anyway, what you are trying to do is better done with Chrome.ahk (using Websocket to communicate with external programs) or Rufaydium.
Perhaps some language-independent elements can be identified on the page, or a certain order of elements? Eg "the third image element needs to be followed by a button and the second checkbox by an edit element" or something.
Re: UIAutomation with a focus on Chrome
@DescoladaOK, thank you.
Re: UIAutomation with a focus on Chrome
Anyone know how to find the number of found elements that meet a specific condition using something like FindAll?
Re: UIAutomation with a focus on Chrome
@Skrell, FindAll returns an array, so Array.MaxIndex() should give you the number of returned elements.
Re: UIAutomation with a focus on Chrome
I'm trying to count all the running buttons in the taskbar, any idea why this isn't working?
Code: Select all
tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
c := UIA.CreateCondition("Name==running AND ControlType==Button",,2)
winCount := (tbEl.FindAll(c, 0x4)).MaxIndex()
Re: UIAutomation with a focus on Chrome
Skrell, this works for me
Wath out for the == in the condition since it's a comparison operator, instead of = to assign a value to a property Id.
Code: Select all
UIA := UIA_Interface()
tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
toolbarEl := tbEl.FindFirstBy("ClassName=MSTaskSwWClass AND ControlType=Pane")
winCount := (toolbarEl.FindAll("ControlType=Button")).MaxIndex()
Re: UIAutomation with a focus on Chrome
Unfortunately this code does not work for me. When I printout the results of winCount I don't even get 0, I get literally nothing. Next, I changed the code to the following (as I wasn't sure why you were looking for ClassName=MSTaskSwWClass ?) and this also does not work:pto wrote: ↑29 Aug 2022, 19:06Skrell, this works for meWath out for the == in the condition since it's a comparison operator, instead of = to assign a value to a property Id.Code: Select all
UIA := UIA_Interface() tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd") toolbarEl := tbEl.FindFirstBy("ClassName=MSTaskSwWClass AND ControlType=Pane") winCount := (toolbarEl.FindAll("ControlType=Button")).MaxIndex()
Code: Select all
tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
toolbarEl := tbEl.FindFirstBy("Name=Taskbar AND ControlType=Pane")
winCount := (toolbarEl.FindAll("ControlType=Button")).MaxIndex()
tooltip % winCount
Re: UIAutomation with a focus on Chrome
@Skrell, try
Worked on Win 10 (using ClassName MSTaskListWClass) and Win 11 (finding all ClassName=Taskbar.TaskListButtonAutomationPeer buttons).
Code: Select all
UIA := UIA_Interface()
tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd")
toolbarEl := tbEl.FindFirstBy("ClassName=MSTaskListWClass AND ControlType=Toolbar")
windowEls := toolbarEl ? toolbarEl.FindAll() : tbEl.FindAllBy("ClassName=Taskbar.TaskListButtonAutomationPeer")
result := "Found " windowEls.MaxIndex() " buttons`n`n"
for i, el in windowEls
result .= i ": " el.Dump() "`n"
MsgBox, % result
ExitApp
Re: UIAutomation with a focus on Chrome
Beautiful thank you! Works!Descolada wrote: ↑30 Aug 2022, 10:37@Skrell, tryWorked on Win 10 (using ClassName MSTaskListWClass) and Win 11 (finding all ClassName=Taskbar.TaskListButtonAutomationPeer buttons).Code: Select all
UIA := UIA_Interface() tbEl := UIA.ElementFromHandle("ahk_class Shell_TrayWnd") toolbarEl := tbEl.FindFirstBy("ClassName=MSTaskListWClass AND ControlType=Toolbar") windowEls := toolbarEl ? toolbarEl.FindAll() : tbEl.FindAllBy("ClassName=Taskbar.TaskListButtonAutomationPeer") result := "Found " windowEls.MaxIndex() " buttons`n`n" for i, el in windowEls result .= i ": " el.Dump() "`n" MsgBox, % result ExitApp
Re: UIAutomation with a focus on Chrome
Yes, it's worth mentionning that i tried on Win 8.1 (don't ask ).
@Skrell , i'm glad you sort that out.
I also take the opportunity to thank you mods, for all the work you do.
@Skrell , i'm glad you sort that out.
I also take the opportunity to thank you mods, for all the work you do.