UIAutomation with a focus on Chrome
Re: UIAutomation with a focus on Chrome
@Loop, I've usually had that happen when the element exists but isn't actually ready yet for user input. Try putting a small Sleep before the Click and after finding the element. When I get some time then I'll try adding an automatic retry after like 40-100ms so this wouldn't be necessary...
Re: UIAutomation with a focus on Chrome
Thank you, but unfortunately it does not work.
Here is my script, do you have any suggestions for improvement?
First loop works, then error message appears
Here is my script, do you have any suggestions for improvement?
First loop works, then error message appears
Code: Select all
F8::
UIA := UIA_Interface()
el := WinExist("A")
WinActivate, ahk_id %el%
WinWaitActive, ahk_id %el%
While !(el := UIA.ElementFromHandle(el)){
el := UIA.ElementFromHandle(el)
Sleep, 1000
}
Loop, 3
{
AllLinks := el.FindAllBy("ControlType=MenuItem AND Name='Share'")
ToolTip, % HoMaLinks := AllLinks.MaxIndex()
For _, Link in AllLinks
{
ToolTip, % A_index " - " HoMaLinks
While(Link.CurrentIsOffscreen){
Click, WD
Sleep, 150
}
Sleep, 1500
Link.Highlight(200).Click()
Sleep, 1500
Clipboard := ""
el.FindFirstBy("ControlType=Text AND Name='Copy Link'").Click()
ClipWait,2
if !InStr(CopyLinks, Clipboard)
CopyLinks .= Clipboard "`r`n"
}
Sleep, 10000
}
ToolTip
MsgBox, % CopyLinks
ExitApp
Re: UIAutomation with a focus on Chrome
@Loop, your While loop is essentially calling UIA.ElementFromHandle("") on its second try (if the element isn't found) and that is throwing the error. el := UIA.ElementFromHandle(el) overwrites your window handle with the result of UIA.ElementFromHandle(el) which is "" if no element is found...
Also,
can probably replaced with
Also,
Code: Select all
While(Link.CurrentIsOffscreen){
Click, WD
Sleep, 150
}
Code: Select all
if Link.CurrentIsOffscreen
Link.ScrollItemPattern.ScrollIntoView()
Re: UIAutomation with a focus on Chrome
Hi! Any idea what could cause an error like this:
My best guess is that somehow the program takes longer than usual after the Save and while the button for opening the next menu exists, it isn't ready to receive inputs. I now added a slightly longer sleep after clicking Save. The bug is so rare that I'll have to wait and see if this fixes it. Any other ideas?
This happens rarely and seemingly randomly after my program has finished inputting data in a field and clicks a Save-button (StatisticsContentDataSaveButton). This click registered correctly but right after I got the error, before clicking the next button (StatisticsContactDataOpenButton), which opens another menu. Here's my code:Code: Select all
eKäynninYhteenveto := UIA.ElementFromHandle(WinExist("Käynnin yhteenveto"))
eSisältömerkinnät := eKäynninYhteenveto.WaitElementExist("ClassName=ContentMarkDataControl")
; ... some code for inputting stuff
eSisältömerkinnät.WaitElementExist("AutomationId=StatisticsContentDataSaveButton").Click()
Sleep, 300
eKontaktitiedot := eKäynninYhteenveto.WaitElementExist("ClassName=ContactDataControl")
eKontaktitiedot.WaitElementExist("AutomationId=StatisticsContactDataOpenButton").Click()
Re: UIAutomation with a focus on Chrome
@Youhoney, I think you are right about the cause of the error, but I would recommend adding the sleep after the eKontaktitiedot := eKäynninYhteenveto.WaitElementExist("ClassName=ContactDataControl") line: that one appears to wait for the control/window that contains the save button, so this would give the save button some time to load after the parent element ContactDataControl has appeared. Even better would be to add it after eKontaktitiedot.WaitElementExist("AutomationId=StatisticsContactDataOpenButton") to make sure the Save button is ready for a click.
Re: UIAutomation with a focus on Chrome
@Descolada Hi.
First of all, I'd like to congratulate you on the great work you've done. I feel like you've improved the utility of AHK enormously.
I've been trying to automate Calibre (the book management software) using UIA, and I've been mostly successful, except that I have not been able to add any working event handlers. I've added a few of those according the examples given, but I receive no events when I expect them.
Calibre is implemented using Qt - I wonder if you had any experience with implementing event handlers on Qt applications, and if you have any suggestions.
First of all, I'd like to congratulate you on the great work you've done. I feel like you've improved the utility of AHK enormously.
I've been trying to automate Calibre (the book management software) using UIA, and I've been mostly successful, except that I have not been able to add any working event handlers. I've added a few of those according the examples given, but I receive no events when I expect them.
Calibre is implemented using Qt - I wonder if you had any experience with implementing event handlers on Qt applications, and if you have any suggestions.
Re: UIAutomation with a focus on Chrome
@Descolada, thanks a lot! I'll try that.
Re: UIAutomation with a focus on Chrome
@mikiher, I have very little experience with Qt apps, but the experience I've had has been quite bad accessibility-wise, that is Qt apps don't support UIAutomation framework well. Inspecting the Calibre window with Accessibility Insights event listener, I seem to be able to mostly capture FocusChanged and StructureChanged events, also for the status bar text change events, and some list item selection events. What you would need to use of these really depends on what you are trying to achieve. For example, detecting clicks could be done with FocusChanged events (which actually listens to every window, so it's necessary to check if Calibre window is active):
Code: Select all
#Include <UIA_Interface>
UIA := UIA_Interface()
handler := UIA_CreateEventHandler("FocusChangedEvent", "FocusChanged")
UIA.AddFocusChangedEventHandler(handler)
OnExit("Cleanup")
return
Cleanup() {
UIA_Interface().RemoveAllEventHandlers()
}
FocusChangedEvent(sender) {
if !WinActive("ahk_exe calibre.exe")
Return
ToolTip, % sender.Dump()
}
Re: UIAutomation with a focus on Chrome
@Descolada many thanks for the tips! I will try to capture the events you mentioned.
Re: UIAutomation with a focus on Chrome
What can I do about this error in order to keep the script going?
Re: UIAutomation with a focus on Chrome
How do we use FindAllBy? When I use FindFirstByNameAndType, script finds the first element on chrome page with it and clicks
However, I want the 2nd element
Works! but clicks the first link in page "Billing"
Doesn't work. Throws Error
However, I want the 2nd element
Works! but clicks the first link in page "Billing"
Code: Select all
cUIA.FindFirstByNameAndType("Stage","MenuItem",,2).click(1000)
cUIA.FindFirstByNameAndType("Billing","ListItem").click()
Code: Select all
cUIA.FindFirstByNameAndType("Stage","MenuItem",,2).click(1000)
cUIA.FindAllByNameAndType("Billing","ListItem").2.click()
Re: UIAutomation with a focus on Chrome
@Skrell, use Try-Catch to handle the error. Or you can totally ignore the error by only using Try: try UIA.FindFirst().Click() ; something that throws an error
@jsong55, it was a bug in FindAllByNameAndType which is fixed now (download new file from GitHub)
@jsong55, it was a bug in FindAllByNameAndType which is fixed now (download new file from GitHub)
Re: UIAutomation with a focus on Chrome
@Descolada
Fantastic! Works like a charm now. I love your class and still finding ways to use it more extensively. I know you probably explained somewhere but mind linking the page that shows what goes on behind the scenes? Like I know the chromeAHK uses the DevTools protocol. What does this use?
Fantastic! Works like a charm now. I love your class and still finding ways to use it more extensively. I know you probably explained somewhere but mind linking the page that shows what goes on behind the scenes? Like I know the chromeAHK uses the DevTools protocol. What does this use?
Re: UIAutomation with a focus on Chrome
@jsong55, it's a wrapper for Microsoft UIAutomation framework with additional methods added by me (eg Click, GetCurrentPos, Highlight). But to understand what happens behind the scenes, you will need to dig into the source codes
Re: UIAutomation with a focus on Chrome
In WhatsApp
This works to trigger a search but
this doesn't work
any idea how to do it better
This works to trigger a search but
Code: Select all
wEl.FindFirstByNameAndType("Search input textbox", "Edit").Click()
send Google
Code: Select all
wEl.FindFirstByNameAndType("Search input textbox", "Edit").value:="Google"
Re: UIAutomation with a focus on Chrome
@jsong55, WhatsApp begins a search when the user types something, but setting the value directly isn't counted as "typing", so it doesn't do the search. So you need to use either Send or ControlSend...
Code: Select all
#include <UIA_Interface>
winTitle := "WhatsApp"
WinActivate, % winTitle
WinWaitActive, % winTitle
UIA := UIA_Interface()
waEl := UIA.ElementFromHandle(winTitle)
if UIA.GetFocusedElement().Name != "Search input textbox"
waEl.FindFirstByNameAndType("Search or start new chat", "Button").Click()
ClipControlSend(,"Google", winTitle)
;ControlSend,, Google, WhatsApp ; Simpler (albeit slower) alternative
ClipControlSend(Control:="", Keys:="", WinTitle:="", WinText:="", ExcludeTitle:="", ExcludeText:="") {
ClipSave := ClipboardAll
Clipboard := ""
Clipboard := Keys
ClipWait, 1
ControlSend, %Control%, {Ctrl down}v{Ctrl up}, %WinTitle%, %WinText%, %ExcludeTitle%, %ExcludeText%
Clipboard := ClipSave
}
Re: UIAutomation with a focus on Chrome
@Descolada
Ah that is a great idea. Thanks
Ah that is a great idea. Thanks
Re: UIAutomation with a focus on Chrome
Hi again,
In UIA_CreateEventHandler() there's a check for IsFunc.
I wonder if you could remove or broaden the check so it also accepts user-defined function objects as well.
I tried removing the check and passing a function object, and it works as expected (as long as the function object implements the required __call, of course).
In UIA_CreateEventHandler() there's a check for IsFunc.
Code: Select all
UIA_CreateEventHandler(funcName, handlerType="") { ; Possible handlerType values: empty, FocusChanged, StructureChanged, TextEditTextChanged, Changes, Notification.
if !IsFunc(funcName){
throw, % funcName "is not a function!"
return
}
...
I tried removing the check and passing a function object, and it works as expected (as long as the function object implements the required __call, of course).
Re: UIAutomation with a focus on Chrome
@mikiher, I broadened it to either Func or Object, though the check will fail if the object isn't callable. If anyone has ideas on how to implement that check (allowing BoundFuncs, and any object that has __Call/Call implemented) with no more than a few lines of code, I will be glad to include it in the function.
Re: UIAutomation with a focus on Chrome
@Descolada,
Thanks a lot for the quick fix!
Thanks a lot for the quick fix!