UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 28 Sep 2022, 23:58

@Loop, I've usually had that happen when the element exists but isn't actually ready yet for user input. Try putting a small Sleep before the Click and after finding the element. When I get some time then I'll try adding an automatic retry after like 40-100ms so this wouldn't be necessary...

Loop
Posts: 160
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 29 Sep 2022, 07:49

Thank you, but unfortunately it does not work.
Here is my script, do you have any suggestions for improvement?
First loop works, then error message appears

Code: Select all

F8::
UIA := UIA_Interface()
el := WinExist("A")
WinActivate, ahk_id %el%
WinWaitActive, ahk_id %el%

While !(el := UIA.ElementFromHandle(el)){
el := UIA.ElementFromHandle(el)
Sleep, 1000
}

Loop, 3
{
     AllLinks := el.FindAllBy("ControlType=MenuItem AND Name='Share'")
     ToolTip, % HoMaLinks := AllLinks.MaxIndex()

	For _, Link in AllLinks
	{
	ToolTip, % A_index " - " HoMaLinks
			While(Link.CurrentIsOffscreen){
		     	     Click, WD
			     Sleep, 150
			}
	    Sleep, 1500
	    Link.Highlight(200).Click()
	    Sleep, 1500
	     Clipboard := ""
	     el.FindFirstBy("ControlType=Text AND Name='Copy Link'").Click()
	     ClipWait,2
		if !InStr(CopyLinks, Clipboard)
		     CopyLinks .= Clipboard "`r`n"
	}

     Sleep, 10000

}
ToolTip
MsgBox, % CopyLinks
ExitApp

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 29 Sep 2022, 13:27

@Loop, your While loop is essentially calling UIA.ElementFromHandle("") on its second try (if the element isn't found) and that is throwing the error. el := UIA.ElementFromHandle(el) overwrites your window handle with the result of UIA.ElementFromHandle(el) which is "" if no element is found...

Also,

Code: Select all

			While(Link.CurrentIsOffscreen){
		     	     Click, WD
			     Sleep, 150
			}
can probably replaced with

Code: Select all

if Link.CurrentIsOffscreen
    Link.ScrollItemPattern.ScrollIntoView()

Youhoney
Posts: 8
Joined: 16 Jul 2022, 03:30

Re: UIAutomation with a focus on Chrome

Post by Youhoney » 05 Oct 2022, 04:16

Hi! Any idea what could cause an error like this:
error.png
error.png (38.08 KiB) Viewed 2308 times
This happens rarely and seemingly randomly after my program has finished inputting data in a field and clicks a Save-button (StatisticsContentDataSaveButton). This click registered correctly but right after I got the error, before clicking the next button (StatisticsContactDataOpenButton), which opens another menu. Here's my code:

Code: Select all

eKäynninYhteenveto := UIA.ElementFromHandle(WinExist("Käynnin yhteenveto"))
eSisältömerkinnät := eKäynninYhteenveto.WaitElementExist("ClassName=ContentMarkDataControl")
; ... some code for inputting stuff
eSisältömerkinnät.WaitElementExist("AutomationId=StatisticsContentDataSaveButton").Click()
Sleep, 300
eKontaktitiedot := eKäynninYhteenveto.WaitElementExist("ClassName=ContactDataControl")
eKontaktitiedot.WaitElementExist("AutomationId=StatisticsContactDataOpenButton").Click()
My best guess is that somehow the program takes longer than usual after the Save and while the button for opening the next menu exists, it isn't ready to receive inputs. I now added a slightly longer sleep after clicking Save. The bug is so rare that I'll have to wait and see if this fixes it. Any other ideas?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 05 Oct 2022, 08:16

@Youhoney, I think you are right about the cause of the error, but I would recommend adding the sleep after the eKontaktitiedot := eKäynninYhteenveto.WaitElementExist("ClassName=ContactDataControl") line: that one appears to wait for the control/window that contains the save button, so this would give the save button some time to load after the parent element ContactDataControl has appeared. Even better would be to add it after eKontaktitiedot.WaitElementExist("AutomationId=StatisticsContactDataOpenButton") to make sure the Save button is ready for a click.

mikiher
Posts: 6
Joined: 23 Aug 2022, 00:26

Re: UIAutomation with a focus on Chrome

Post by mikiher » 05 Oct 2022, 12:57

@Descolada Hi.

First of all, I'd like to congratulate you on the great work you've done. I feel like you've improved the utility of AHK enormously.
I've been trying to automate Calibre (the book management software) using UIA, and I've been mostly successful, except that I have not been able to add any working event handlers. I've added a few of those according the examples given, but I receive no events when I expect them.

Calibre is implemented using Qt - I wonder if you had any experience with implementing event handlers on Qt applications, and if you have any suggestions.

Youhoney
Posts: 8
Joined: 16 Jul 2022, 03:30

Re: UIAutomation with a focus on Chrome

Post by Youhoney » 06 Oct 2022, 03:53

@Descolada, thanks a lot! I'll try that.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Oct 2022, 08:11

@mikiher, I have very little experience with Qt apps, but the experience I've had has been quite bad accessibility-wise, that is Qt apps don't support UIAutomation framework well. Inspecting the Calibre window with Accessibility Insights event listener, I seem to be able to mostly capture FocusChanged and StructureChanged events, also for the status bar text change events, and some list item selection events. What you would need to use of these really depends on what you are trying to achieve. For example, detecting clicks could be done with FocusChanged events (which actually listens to every window, so it's necessary to check if Calibre window is active):

Code: Select all

#Include <UIA_Interface>

UIA := UIA_Interface()
handler := UIA_CreateEventHandler("FocusChangedEvent", "FocusChanged")
UIA.AddFocusChangedEventHandler(handler)
OnExit("Cleanup")
return
Cleanup() {
    UIA_Interface().RemoveAllEventHandlers()
}
FocusChangedEvent(sender) {
    if !WinActive("ahk_exe calibre.exe")
        Return
    ToolTip, % sender.Dump()
}

mikiher
Posts: 6
Joined: 23 Aug 2022, 00:26

Re: UIAutomation with a focus on Chrome

Post by mikiher » 06 Oct 2022, 14:30

@Descolada many thanks for the tips! I will try to capture the events you mentioned.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 07 Oct 2022, 20:54

What can I do about this error in order to keep the script going?
image.png
image.png (5.6 KiB) Viewed 2135 times

jsong55
Posts: 221
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 08 Oct 2022, 00:14

How do we use FindAllBy? When I use FindFirstByNameAndType, script finds the first element on chrome page with it and clicks

However, I want the 2nd element

Works! but clicks the first link in page "Billing"

Code: Select all

cUIA.FindFirstByNameAndType("Stage","MenuItem",,2).click(1000)
cUIA.FindFirstByNameAndType("Billing","ListItem").click()
Doesn't work. Throws Error

Code: Select all

cUIA.FindFirstByNameAndType("Stage","MenuItem",,2).click(1000)
cUIA.FindAllByNameAndType("Billing","ListItem").2.click()

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Oct 2022, 00:28

@Skrell, use Try-Catch to handle the error. Or you can totally ignore the error by only using Try: try UIA.FindFirst().Click() ; something that throws an error

@jsong55, it was a bug in FindAllByNameAndType which is fixed now (download new file from GitHub)

jsong55
Posts: 221
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 08 Oct 2022, 00:35

@Descolada

Fantastic! Works like a charm now. I love your class and still finding ways to use it more extensively. I know you probably explained somewhere but mind linking the page that shows what goes on behind the scenes? Like I know the chromeAHK uses the DevTools protocol. What does this use?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Oct 2022, 00:46

@jsong55, it's a wrapper for Microsoft UIAutomation framework with additional methods added by me (eg Click, GetCurrentPos, Highlight). But to understand what happens behind the scenes, you will need to dig into the source codes :)

jsong55
Posts: 221
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 08 Oct 2022, 00:57

In WhatsApp

This works to trigger a search but

Code: Select all

wEl.FindFirstByNameAndType("Search input textbox", "Edit").Click()
send Google
this doesn't work

Code: Select all

wEl.FindFirstByNameAndType("Search input textbox", "Edit").value:="Google"
any idea how to do it better

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Oct 2022, 01:30

@jsong55, WhatsApp begins a search when the user types something, but setting the value directly isn't counted as "typing", so it doesn't do the search. So you need to use either Send or ControlSend...

Code: Select all

#include <UIA_Interface>
winTitle := "WhatsApp"
WinActivate, % winTitle
WinWaitActive, % winTitle
UIA := UIA_Interface()
waEl := UIA.ElementFromHandle(winTitle)
if UIA.GetFocusedElement().Name != "Search input textbox"
    waEl.FindFirstByNameAndType("Search or start new chat", "Button").Click()
ClipControlSend(,"Google", winTitle)
;ControlSend,, Google, WhatsApp ; Simpler (albeit slower) alternative

ClipControlSend(Control:="", Keys:="", WinTitle:="", WinText:="", ExcludeTitle:="", ExcludeText:="") {
    ClipSave := ClipboardAll
    Clipboard := ""
    Clipboard := Keys
    ClipWait, 1
    ControlSend, %Control%, {Ctrl down}v{Ctrl up}, %WinTitle%, %WinText%, %ExcludeTitle%, %ExcludeText%
    Clipboard := ClipSave
}

jsong55
Posts: 221
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 08 Oct 2022, 04:05

@Descolada
Ah that is a great idea. Thanks

mikiher
Posts: 6
Joined: 23 Aug 2022, 00:26

Re: UIAutomation with a focus on Chrome

Post by mikiher » 08 Oct 2022, 04:32

Hi again,
In UIA_CreateEventHandler() there's a check for IsFunc.

Code: Select all

	UIA_CreateEventHandler(funcName, handlerType="") { ; Possible handlerType values: empty, FocusChanged, StructureChanged, TextEditTextChanged, Changes, Notification.
		if !IsFunc(funcName){
			throw, % funcName "is not a function!"
			return
		}
	...
I wonder if you could remove or broaden the check so it also accepts user-defined function objects as well.
I tried removing the check and passing a function object, and it works as expected (as long as the function object implements the required __call, of course).

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Oct 2022, 06:09

@mikiher, I broadened it to either Func or Object, though the check will fail if the object isn't callable. If anyone has ideas on how to implement that check (allowing BoundFuncs, and any object that has __Call/Call implemented) with no more than a few lines of code, I will be glad to include it in the function.

mikiher
Posts: 6
Joined: 23 Aug 2022, 00:26

Re: UIAutomation with a focus on Chrome

Post by mikiher » 08 Oct 2022, 09:58

@Descolada,
Thanks a lot for the quick fix!

Post Reply

Return to “Scripts and Functions (v1)”