UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIAutomation with a focus on Chrome

Post by malcev » 18 Oct 2022, 12:56

Descolada wrote:
16 Oct 2022, 23:20
how to access the unmanaged UIA2 I don't know
UIA2 is managed. I think You can run it with clr.
@Shield
You can look here
viewtopic.php?t=93892

fenchai
Posts: 292
Joined: 28 Mar 2016, 07:57

Re: UIAutomation with a focus on Chrome

Post by fenchai » 19 Oct 2022, 15:46

Hi guys, I have this automation I am testing, basically it goes to a webpage, searches for the input element and pastes the number I want to paste.

issue: My current issue is that this seems to be a abit slower than manual input, I can see that the element has already appeared but it seems UIA is still trying to find it. (this is just me trying to min-max the code) not sure if there is another faster method?

here is the code:

Code: Select all

#NoEnv
#SingleInstance, Force
SendMode, Input
SetBatchLines, -1
SetWorkingDir, %A_ScriptDir%

#Include, %A_Scriptdir%\lib\UIA_Interface.ahk
#Include, %A_Scriptdir%\lib\UIA_Browser.ahk

webAutoItemInput()
return

webAutoItemInput() {

    ; example variable set webpage & item
    webTextInfo := "https://www.js-filter.com/catalogue,17801-10030"

    ; analize and split string
    info := StrSplit(webTextInfo, ",")
    webpage := info[1], item := info[2]

    ; move mouse to center of screen to prevent any weird popups or change in UI
    CoordMode, Mouse, Screen
    MouseMove, (A_ScreenWidth // 2), (A_ScreenHeight // 2), 0

    ; runs chrome if not active, activates chrome if exists
    browserExe := "chrome.exe"
    if !WinExist("ahk_exe " browserExe)
        Run, %browserExe%
    else
        WinActivate, % "ahk_exe " browserExe

    WinWaitActive, ahk_exe %browserExe%
    WinMaximize, ahk_exe %browserExe%

    ; create UIA instance
    UIA := new UIA_Browser("ahk_exe " browserExe)

    ; check if UIA instance exists
    if !(IsObject(UIA)) {
        msgbox UIA is not loaded
        return
    }

    ; start UIA work
    Try {
        if !(UIA.GetCurrentURL() == "chrome://new-tab-page/") {
            UIA.NewTab()
            ; tooltip, waiting for new tab to load
            UIA.WaitPageLoad("New Tab", 1000)
            ; tooltip, navigating to webpage
        }
        UIA.Navigate(webpage)
        ; tooltip, waiting for webpage to load
        UIA.WaitPageLoad()

        if (webpage == "https://www.jnbk-brakes.com/catalogue/cars" || webpage == "https://www.js-filter.com/catalogue") {
            retries := 0
            checkEl:
                ; THIS IS THE ISSUE
                ; THE FindFirstBy Method is a bit slow
                ; I WOULD SAY IT'S AROUND 1 SEC SLOWER THAN MANUAL INPUT
                ; NOT SURE IF THERE IS A FASTER WAY?
                editEl := UIA.FindFirstBy("AutomationId=txtPartNo")
                ; check if input element exists
                if !(IsObject(editEl)) {
                    ; retry if input doesn't exist
                    ; maybe there is no internet
                    retries := retries + 1
                    if (retries > 3)
                        return
                    sleep 500
                    Gosub, checkEl
                    return
                }

                ; Automate inputs and clicks
                editEl.SetValue(item)
                UIA.FindFirstBy("AutomationId=btnProductSearch").Click()
            }
        }
    }

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 19 Oct 2022, 23:10

@fenchai, Navigate already calls WaitPageLoad as well, so UIA.WaitPageLoad() isn't necessary here. Also both Navigate and WaitPageLoad have a default sleep of 500ms after each call to ensure the webpage has loaded (which means that your code is sleeping in total for 1 seconds for the navigation). In your case ensuring the page has loaded might not be that important: you could use UIA.Navigate(webpage,,,0) instead and replace editEl := UIA.FindFirstBy("AutomationId=txtPartNo") with editEl := UIA.WaitElementExist("AutomationId=txtPartNo") (perhaps also with some timeout for a retry: in case there is no internet, or something went wrong).

fenchai
Posts: 292
Joined: 28 Mar 2016, 07:57

Re: UIAutomation with a focus on Chrome

Post by fenchai » 20 Oct 2022, 20:21

@Descolada

thanks for the help and the tips, removing the UIA.WaitPageLoad() made the best improvement, not sure about the UIA.waitElementExist part.

malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIAutomation with a focus on Chrome

Post by malcev » 30 Oct 2022, 05:56

@Descolada, here I wrote 2 examples with 3-party libraries and without
viewtopic.php?p=488634#p488634

perfect1151986
Posts: 13
Joined: 30 Mar 2022, 21:06

Re: UIAutomation with a focus on Chrome

Post by perfect1151986 » 30 Oct 2022, 20:19

i have question. I use findbyname search Button Tìm. I use UIAviewer read Button Tìm, it's Name = "T×m". But when i run scripts. it don'n find it. Everyone help me.
This is my'code

Code: Select all

SendMode Input
SetWorkingDir, %A_ScriptDir%
#include C:\Users\Ha Noi\Desktop\UIAutomation-main\Lib\UIA_Interface.ahk
#include C:\Users\Ha Noi\Desktop\UIAutomation-main\Lib\UIA_Constants.ahk
F3::
UIA := UIA_Interface() ; Initialize UIA interface
B1 := UIA.ElementFromHandle(WinExist("ahk_exe Nghiep vu Bao hiem.exe")) ;
Nuttim := B1.FindFirstByName("T×m")
    While IsObject(Nuttim) = 0
        {
        Nuttim := B1.FindFirstByName("T×m")
        }
    Nuttim.Highlight()
[Mod edit: [code][/code] tags added.]
Attachments
z3842320897632_ee9c907710d6ce2b036f8d5d288537ea.jpg
z3842320897632_ee9c907710d6ce2b036f8d5d288537ea.jpg (28.55 KiB) Viewed 3285 times

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 31 Oct 2022, 01:43

@perfect1151986
1) Check that your code is saved in UTF-8-BOM encoding
2) Make sure that you copied the name from UIAViewer by rightclicking on it and then pasting it in your code, not by typing the "×" character out yourself
3) If it still doesn't work, then since the problem is most likely in the unicode character "×" which UIA seems to sometimes have problems with, you could FindAll buttons starting with "T" and then filter through the results to find the button with the correct name.
4) In my experience ThunderRT6CommandButton type buttons are clickable with regular AHK ControlClick, which usually is a better option for clicking buttons.

Edit: the "×" character appears to be found properly in an AHK GUI. The following code works in both UTF-8 and UTF-8-BOM, but in the former the "×" will be displayed incorrectly. Running the GUI in a different script that is saved in UTF-8-BOM, but running the UIA part in regular UTF-8 will cause this to fail (button not clicked). So make sure your script is saved in UTF-8-BOM...

Code: Select all

#Include <UIA_Interface>
#Include <UIA_Browser>

Gui, New,, MyTest
Gui, Add, Button, gTest, T×m
Gui, Show

WinWaitActive, MyTest
UIA := UIA_Interface()
wEl := UIA.ElementFromHandle("MyTest")
wEl.FindFirstByNameAndType("T×m", "Button").Click()
ExitApp

Test:
    MsgBox, Test
    return

jekko1976
Posts: 97
Joined: 10 Oct 2014, 07:03

Re: UIAutomation with a focus on Chrome

Post by jekko1976 » 03 Nov 2022, 05:23

When I send a click on an element nothing happens.
I have inspected the element with Accessibility insights and I have seen that, when I click my element, a "AutomationFocusChanged" Event is sent.
So I reviewed the Example number 7 but unfortunately it can only "Detect" a "AutomationFocusChanged" Event.
What I need to to (I guess) is to FIRE a "AutomationFocusChanged" Event on my element.
How could I reach this goal?
Thank you so much

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 03 Nov 2022, 13:04

@jekko1976, if AutomationFocusChanged event is sent then it means that the click is focusing the element, but apparently not actually "clicking" it. AutomationFocusChanged event is triggered when an element changes focus: the element is activated, but might not be clicked. For example you can focus a button by left-pressing on it, but you can prevent a click by dragging the cursor away from the button, which will cause a FocusChanged event to trigger but the button isn't clicked.
In your case, the failure of the click usually means that the click mechanism that UIA_Interface uses (for buttons this is Invoke or DoDefaultAction) is not working correctly. You could try using SetFocus before the Click, sometimes this works. Check what kind of pattern your button supports for clicks (Invoke?) and try triggering it with Accessibility Insights - if that works, you can use that pattern in your code.
Alternatively use Click("left") to MouseClick the element, or ControlClick the element.

jsong55
Posts: 222
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 05 Nov 2022, 20:44

@Descolada

I found that

Code: Select all

cUIA.WaitPageLoad("Page Name")
was a tad too quick and the page at times had not fully loaded.

So I used

Code: Select all

WinGetTitle, PageTitle, A
while !instr(PageTitle,row.name)
{
    sleep,500
    ; tooltip,% PageTitle
    WinGetTitle, PageTitle, A
}
instead. Works for me. FYI

adamaze
Posts: 2
Joined: 16 Nov 2022, 12:57

Re: UIAutomation with a focus on Chrome

Post by adamaze » 16 Nov 2022, 13:13

Hello,

Although it took me a while to figure it out, this is working great for a need I have inside firefox. (except for one issue)

In my current script, a keypress opens a URL in a new tab in firefox, waits for it to load, and then executes something. I can do this over and over again just fine, but if firefox is closed, and then re-opened, the next run of

Code: Select all

cUIA := new UIA_Browser("ahk_exe " browserExe)
fails with the following message:

Code: Select all

Error in #include file "UIA_Browser.ahk":
     0x80004003 - Pointer not valid.

Specifically: GetFirstChildElement  (UIA_TreeWalker)

	Line#
	434: {
	435: Try
	435: {
	436: if this.ReloadButton && this.ReloadButton.CurrentName  
	437: Return,this.ReloadButton
	438: }
	439: ButtonWalker := this.UIA.CreateTreeWalker(this.ButtonCondition)
--->	440: this.ReloadButton := ButtonWalker.GetNextSiblingElement(ButtonWalker.GetNextSiblingElement(ButtonWalker.GetFirstChildElement(this.NavigationBarElement)))  
	441: Return,this.ReloadButton
	442: }
	445: {
	446: this.JSExecute("document.title=""" newTitle """; void(0);")  
	447: }
	449: {
	450: this.SetURL("javascript:" js, True)  

The current thread will exit.
originally the keypress only worked the first time, and then i would get this issue every time, as it fails on the second initialization, so i put a check to look for the browser PID, and then attempt to re-initialize if the PID changes (firefox is closed), but it would seem that doesnt work.

am i missing something here? will the "UIA_Browser" initialization only work once per script run?

any help would be greatly appreciated

thanks!


[Mod edit: Changed type of code box for the error message from an AHK script type to a text type]

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Nov 2022, 14:38

@adamaze, please try the latest UIA_Browser.ahk from GitHub, I think the latest update fixed this problem. Thanks for finding this bug :)

adamaze
Posts: 2
Joined: 16 Nov 2022, 12:57

Re: UIAutomation with a focus on Chrome

Post by adamaze » 16 Nov 2022, 15:21

wow! thanks for fixing this so fast! I have spent hours trying to troubleshoot and figure this out. i should have just posted sooner!

you should set up Github Sponsors so i can show some more appreciation!

lx930129
Posts: 10
Joined: 13 Sep 2016, 03:41

Re: UIAutomation with a focus on Chrome

Post by lx930129 » 17 Nov 2022, 07:25

excellent work! Descolada. Thanks for sharing. :thumbup:

TomDonovan
Posts: 4
Joined: 25 Dec 2020, 11:57

Re: UIAutomation with a focus on Chrome

Post by TomDonovan » 17 Nov 2022, 21:52

How do I create the code to monitor for a UIA_AutomationFocusChanged Event. I've played with AddFocusChangedEventHandler, but can't seem to get the right syntax.

Thanks in advance for any and all help.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 18 Nov 2022, 00:02

@TomDonovan, example 7 implements FocusChangedEvent with UIA_Browser, which shares syntax with regular UIA_Interface as well so you can use that as a base. Or a more compact example:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.

#include <UIA_Interface>

EventHandler(el) {
	ToolTip, % "Caught event!`nElement name: " el.CurrentName
}

ExitFunc() {
	UIA.RemoveFocusChangedEventHandler(h) ; Remove the event handler. Alternatively use UIA.RemoveAllEventHandlers() to remove all handlers
}

global UIA := UIA_Interface() ; Defined as a global variable, so it is accessible in ExitFunc as well

h := UIA_CreateEventHandler("EventHandler", "FocusChanged") ; Create a new FocusChanged event handler that calls the function EventHandler (required arguments: element)
UIA.AddFocusChangedEventHandler(h) ; Add a new FocusChangedEventHandler
OnExit("ExitFunc") ; Set up an OnExit call to clean up the handler when exiting the script
return

F5::ExitApp

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 28 Nov 2022, 15:32

Hi Descolada, I'm trying to click loop, but sometimes I get this error:
Screenshot_76.jpg
Screenshot_76.jpg (32.48 KiB) Viewed 2754 times

Sometimes the button does not exist, so it should skip that click. So is there any way to avoid this error or at least create an exception when it appears?

Code: Select all

#Include <UIA_Interface>
#include <UIA_Browser>

#NoEnv
#SingleInstance, Force
SendMode, Input
SetBatchLines, -1
SetWorkingDir, %A_ScriptDir%

UIA := UIA_Interface()
browserExe := "chrome.exe"

chromeId := WinExist("ahk_exe Chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%

Sleep, 2000
cUIA := new UIA_Browser("ahk_exe " browserExe)
chrome := cUIA.ElementFromHandle(chromeId, True)


Sleep, 2000

Loop, 20
{
    Loop, 4
    {
    chrome.FindFirstBy("Name=FOLLOW AND isOffscreen=0").Click()
    Sleep, 400
    }
    Send, {WheelDown 3}
}

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 29 Nov 2022, 00:03

@mora145, you can either check whether the button exists before clicking it:

Code: Select all

if (but := chrome.FindFirstBy("Name=FOLLOW AND isOffscreen=0"))
    but.Click()
or you can wrap the whole line in a try...catch which will ignore the error: try chrome.FindFirstBy("Name=FOLLOW AND isOffscreen=0").Click()
Of course you can also handle the error with the "catch" part:

Code: Select all

try {
    chrome.FindFirstBy("Name=FOLLOW AND isOffscreen=0").Click()
} catch {
    ; something went wrong, an exception was thrown or ErrorLevel is raised
}
Though I don't see why you couldn't replace that Loop altogether with WaitElementExist: try chrome.WaitElementExist("Name=FOLLOW AND isOffscreen=0",,,,2000).Click() ; Waits 2 seconds for the element to appear and then tries to click it

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 30 Nov 2022, 02:19

@Descolada
Ty man.!

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 02 Dec 2022, 02:28

I am trying to learn about Caching, as I have a small automation that takes a long time to check the elements. I took the example of the note block, left only the "Name" property and removed the others, but I still get the other properties in the DumpAll(). I would like my Caché to have only the Name property. Is this part wrong or am I doing it wrong (Sorry for the Noob)

https://github.com/Descolada/UIAutomation/wiki/10.-Advanced:-Caching-(to-do)

Code: Select all

#Include <UIA_Interface>

UIA := UIA_Interface()
cacheRequest := UIA.CreateCacheRequest()
cacheRequest.TreeScope := 5 ; Set TreeScope to include the starting element and all descendants as well
cacheRequest.AddProperty("Name")

cacheRequest.AddPattern("Window") ; To use cached patterns, first add the pattern
cacheRequest.AddProperty("WindowCanMaximize") ; Also need to add any pattern properties we wish to use

Run, notepad.exe
WinWaitActive, ahk_exe notepad.exe

npEl:= UIA.ElementFromHandleBuildCache("ahk_exe notepad.exe", cacheRequest) ; Get element and also build the cache
MsgBox, % npEl.DumpAll()
MsgBox, % npEl.CachedWindowPattern.CachedCanMaximize

Post Reply

Return to “Scripts and Functions (v1)”