UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 02 Dec 2022, 03:00

@mora145, I have to warn you that I also have very little experience with using caching, because personally I haven't had the need for it. As far as I know though, caching will still allow you to access "live" values when calling the normal properties/patterns, but you can't access cached properties/patterns if they haven't been defined in the CacheRequest.

My example in GitHub was accidentally a bit misleading. DumpAll will work fine (since it accessed live values), but try your example with MsgBox, % npEl.CachedDumpAll() - it will throw an error. Same if you comment out the line cacheRequest.AddPattern("Window"), which will also cause an error.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 02 Dec 2022, 12:21

I still don't really understand how this works. Let's say I have a page with constant changes in its elements and it grows every so often, exponentially. I suppose that in these cases I could not use a cache because it needs to be constantly updated, right?

I could tell FindFirstByName to start searching, let's say, from the middle of the elements obtained onwards. This is so that I don't have to go through the whole page again.c

mikiher
Posts: 6
Joined: 23 Aug 2022, 00:26

Re: UIAutomation with a focus on Chrome

Post by mikiher » 03 Dec 2022, 17:11

Hi @Descolada,

I recently started getting this error without anything changing in the code calling it (see screenshot). Any ideas what's happening?
Attachments
Screenshot 2022-12-04 000546.png
Screenshot 2022-12-04 000546.png (28.04 KiB) Viewed 2848 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 04 Dec 2022, 01:32

@mora145, I don't think caching will help you in your case, because as you said the site is constantly updating and thus the cache would need to be constantly updated anyway. If you want to start your search from the middle, then you might have more luck with FindFirstWithOptions. Specifically you can provide a starting point element with the root argument (perhaps your previously found element?), which would start the search with that element. At least that is how its documented, I haven't used FindFirstWithOptions with that particular option before :)

@mikiher, General access denied might be thrown by UIA if the element isn't accessible for some reason (is the element destroyed by the time the function is being called?), or sometimes if the accessed program is ran as Admin but the AHK script is not. If this error is occurring only sometimes then Admin access isn't the problem. You can suppress that error by using try...catch and perhaps try fetching the element again if it fails on the first try?

mikiher
Posts: 6
Joined: 23 Aug 2022, 00:26

Re: UIAutomation with a focus on Chrome

Post by mikiher » 04 Dec 2022, 02:17

Thanks very much, I'll try what you suggested (although it didn't look like the element was destroyed, nor were there any admin access issues, to the best of my knowledge).

Also, it looks like you made some related fix to the code - I just downloaded it and the problem seemed to have gone away.

Thanks again for the quick response!

r2997790
Posts: 71
Joined: 02 Feb 2017, 02:46

Re: UIAutomation with a focus on Chrome

Post by r2997790 » 05 Dec 2022, 06:54

Firstly, I have to give huge thanks to @Descolada for making UIA available to us all. Thank you!

I'm having real trouble selecting/assigning values with the Facebook Ads environment because they seem to be either dynamically assigning or conditionally assigning a lot of UI names, IDs etc... so... I was going to use the 'dot notation' and FindbyPath... but I'm also finding that flaky... probably because I am not using it properly... and I wanted to ask a couple of questions and ask for some help please.

Thank you.

Do I need to some how 'attach' a TreeCrawler function to my script to accurate crawl by Path?

My script starts like this...

Code: Select all

#include ..\Lib\UIA_Interface.ahk
#include ..\Lib\UIA_Browser.ahk

browserExe := "chrome.exe"
Run, %browserExe% --force-renderer-accessibility
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) 
cUIA.Navigate("URL IS HERE") ; 
cUIA.WaitPageLoad("Ads Manager - Manage ads") 
tooltip, Loaded...
And here's how I'm trying to access via FindbyPath

Code: Select all

conv := cUIA.FindbyPath("1.39.1")
conv.click("left")
Sometimes this works, but other times not. And that's why I am confused, I thought this was a failsafe way to connect to items.

Here's a few more examples...
image.png
image.png (4 KiB) Viewed 2766 times

Code: Select all

StartDateValue := "11/06/2023"
StartDate1 := cUIA.FindbyPath("1.55.1").SetFocus()
StartDate1.SetValue(StartDateValue)

;Here's the paths:
;1.53 Type: 50020 (Text) Name: "Schedule" LocalizedControlType: "text"
;1.54 Type: 50020 (Text) Name: "Start date" LocalizedControlType: "text"
;1.55 Type: 50026 (Group) LocalizedControlType: "group"
;1.55.1 Type: 50020 (Text) Name: "18/6/2023" LocalizedControlType: "text"
;1.55.2 Type: 50026 (Group) LocalizedControlType: "group"
;1.55.2.1 Type: 50020 (Text) LocalizedControlType: "text" AutomationId: "js_3o"
;1.55.2.1.1 Type: 50020 (Text) LocalizedControlType: "heading"
And this one I don't understand either and doesn't work... to set a dropdown value...
image.png
image.png (19.94 KiB) Viewed 2766 times

Code: Select all

SetBudgetType := cUIA.FindbyPath("1.46").SetFocus()
SetBudgetType.SetValue("Daily budget")

;1.43 Type: 50020 (Text) Name: "Off" LocalizedControlType: "text"
;1.44 Type: 50000 (Button) Name: "Off" LocalizedControlType: "button" AutomationId: "js_s"
;1.45 Type: 50020 (Text) Name: "Budget" LocalizedControlType: "text"
;1.46 Type: 50000 (Button) Name: "Daily budget" LocalizedControlType: "button"
This one doesn't work clicking the button either at 1.46.

Thank you in advance for any helpful advice on how to solve this. It's driving me crazy.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 05 Dec 2022, 09:41

@r2997790, FindByPath is only useful if/when the relative path between the elements doesn't change: that is the page won't change in a way that elements will be added inbetween the starting element and target element. I almost never use FindByPath, because usually better alternatives are available.

I'm not sure what cUIA.FindbyPath("1.39.1") is supposed to find and click, so I can't help with that.

But for example, the "Start date" value. When I inspected it in the Facebook Ads page, then I found that it has an AutomationId with value "js_9x". This means that you could just do cUIA.FindFirstBy("AutomationId=js_9x").Value := "6/12/2022" to change the value (or use SetValue - same thing). If it appears that the AutomationId doesn't work (for example it turns out there are multiple elements with Id "js_9x") then I would use the Text element with value "Start date" as a reference point and then use a TreeWalker to get the next Edit element (which would be the date edit):

Code: Select all

txtStart := cUIA.FindFirstByNameAndType("Start date", "Text")
cUIA.CreateTreeWalker("Type=Edit").GetNextSiblingElement(txtStart).Value := "6/12/2022"
FindByPath would be more of a last resort and I would still use "Start date" as a starting point: cUIA.FindFirstByNameAndType("Start date", "Text").FindByPath("p2.2.2.2.1").Value := "6/12/2022". The path "p2.2.2.2.1" I found more or less by trial and error, because often the paths in UIAViewer aren't the same as shown by DumpAll. I just tested out what FindByPath("+1").DumpAll(), FindByPath("p1").DumpAll() etc showed until DumpAll showed the desired element and then I got the final part of the path from there (FindByPath("p2") finally showed the Edit element).

Now for the Budget dropdown... If you look at the type of the element with path 1.46 you see that it's a button, which means that you can't change its value, you can just click it and the dropdown will show. This also means that the path for elements after the button/dropdown will change, since new elements will be added (the dropdown with choices "Daily budget" and "Lifetime budget"). Here I would use again a TreeWalker to get the next Button element, then WaitElementExist to click the desired choice.

Code: Select all

budgetTxt := cUIA.FindFirstByNameAndType("Budget", "Text")
cUIA.CreateTreeWalker("Type=Button").GetNextSiblingElement(budgetTxt).Click()
cUIA.WaitElementExistByNameAndType("Lifetime Budget", "MenuItem").Click()

r2997790
Posts: 71
Joined: 02 Feb 2017, 02:46

Re: UIAutomation with a focus on Chrome

Post by r2997790 » 05 Dec 2022, 11:08

Descolada wrote:
05 Dec 2022, 09:41
Budget dropdown... If you look at the type of the element with path 1.46 you see that it's a button, which means that you can't change its value, you can just click it and the dropdown will show. This also means that the path for elements after the button/dropdown will change, since new elements will be added (the dropdown with choices "Daily budget" and "Lifetime budget"). Here I would use again a TreeWalker to get the next Button element, then WaitElementExist to click the desired choice.

Code: Select all

budgetTxt := cUIA.FindFirstByNameAndType("Budget", "Text")
cUIA.CreateTreeWalker("Type=Button").GetNextSiblingElement(budgetTxt).Click()
cUIA.WaitElementExistByNameAndType("Lifetime Budget", "MenuItem").Click()
@Descolada you've been so incredibly helpful and generous in your reply — giving me the missing piece to the puzzle and explaining it so well. Thank you!

I love the TreeWalker approach — fixing on a 'known element' I can target and then 'walking' the tree up or down to find the element(s) adjacent.

I'll get working on fixing my script tonight and thanks again for your wonderful explanation. I'm very grateful.

Many thanks
R

r2997790
Posts: 71
Joined: 02 Feb 2017, 02:46

Re: UIAutomation with a focus on Chrome

Post by r2997790 » 06 Dec 2022, 07:16

HI @Descolada

Sorry for another couple of questions... but I'm close to solved this UIA conundrum now... and your input was so helpful.. Thank you!!

Unfortunately I am getting this error: is there a way to prevent or suspress it? It means the UIA routine doesn't run:
image.png
image.png (28.86 KiB) Viewed 2678 times
Also, is there a simple way to look up siblings (or children) further down the TreeWalker... ie. not GetNextSiblingElement but perhaps two siblings later... in the context of the Facebook ad platform you so helpfully gave an example of...

Code: Select all

StartDate := cUIA.FindFirstByNameAndType("Start date", "Text")
cUIA.CreateTreeWalker("Type=Edit").GetNextSiblingElement(StartDate).Value := "17/12/2022"

; how would I then select the NEXT sibling after the date, reference from the StartDate
And on a related point... is this allowed:

Code: Select all

StartDate := cUIA.FindFirstByNameAndType("Start date", "Text")
cUIA.CreateTreeWalker("Type=Edit").GetNextSiblingElement(StartDate).Value := "17/12/2022"

EndDate := cUIA.FindFirstByNameAndType("End", "Text")
cUIA.CreateTreeWalker("Type=Checkbox").GetNextSiblingElement(EndDate).click("Left")
Many thanks for any advice...
R

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Dec 2022, 10:29

@r2997790, that error means that you are passing an element that doesn't exist, which means budgetTxt wasn't found. Make sure that the name is exactly "Budget", or if you want to match partial names then specify the match mode (by default exact match is required). I would recommend checking whether elements were found anyway, because otherwise you won't know why and where your code is failing.

Code: Select all

if (!budgetTxt) {
    ; was not found...
}
You can use a treewalkers GetNextSiblingElement multiple times as well, or FindByPath which uses treewalkers internally.

Code: Select all

editWalker := cUIA.CreateTreeWalker("Type=Edit")
secondEdit := editWalker.GetNextSiblingElement(editWalker.GetNextSiblingElement(StartDate))
which is equal to StartDate.FindByPath("+2", "Type=Edit")
And on a related point... is this allowed:
I'm not sure what you are asking "is allowed". Creating multiple treewalkers is indeed allowed.

r2997790
Posts: 71
Joined: 02 Feb 2017, 02:46

Re: UIAutomation with a focus on Chrome

Post by r2997790 » 07 Dec 2022, 09:22

HI @Descolada

I think I'm winning now... but I had two last hiccups I wonder if you could steer me in the right direction with.... both related to the Facebook ad site you know, which is a bit tricky to apply UIA on... here's my challenges...

I can't see to do anything to get to this edit box:
image.png
image.png (72.96 KiB) Viewed 2562 times
I've tried a bunch, including...

Code: Select all

tooltip, Toggling on End date...
cUIA.WaitElementExistByNameAndType("Set an end date", "Text").click()
sleep, 1500
endTxt := cUIA.WaitElementExistByNameAndType("Set an end date", "Text")
cUIA.CreateTreeWalker("Type=Edit").GetNextSiblingElement(endTxt).SetValue("24/1/2023")
Also, the 'spinner' ControlType (which sets time HH:MM) or you can past 12:23 etc I think also seems elusive to target.
image.png
image.png (11.75 KiB) Viewed 2562 times
Unfortunately the AutomationId is not reliable so having to use the TreeWalker and/or FindPath but can't see to inject values into these spinners.

Would you be able to suggest any approach and example to help me finish this off — the finish line is in site now, but these two examples I can't see to 'connect to'.

Thank you so much!
R
Attachments
image.png
image.png (7.65 KiB) Viewed 2562 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 07 Dec 2022, 13:11

r2997790 wrote:
07 Dec 2022, 09:22
I've tried a bunch, including...

Code: Select all

tooltip, Toggling on End date...
cUIA.WaitElementExistByNameAndType("Set an end date", "Text").click()
sleep, 1500
endTxt := cUIA.WaitElementExistByNameAndType("Set an end date", "Text")
cUIA.CreateTreeWalker("Type=Edit").GetNextSiblingElement(endTxt).SetValue("24/1/2023")
The second part of your code (after Sleep) works fine in my browser after removing the cUIA.WaitElementExistByNameAndType("Set an end date", "Text").click() - it unchecked the checkbox and then the date Edit disappeared. You can avoid that by setting the toggle state straight to toggled ("checked").
Setting the time with UIA seems to be impossible, because apparently Chrome hasn't implemented the RangeValuePattern properly (this isn't the first time I've run into this... blame Chrome developers :D ). You can get around that by using Send or ControlSend. This is what finally worked:

Code: Select all

cUIA.WaitElementExistByNameAndType("Set an end date", "Checkbox").TogglePattern.ToggleState := 1
cUIA.WaitElementExistByNameAndType("Set an end date", "Text").FindByPath("+1", "Type=Edit").SetValue("21/1/2023")
ControlSend,, {Tab}12{Tab}23, % "ahk_id " cUIA.BrowserId

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 07 Dec 2022, 17:48

Just curious if there are any plans to add timeouts for functions like ElementFromPoint? I've had my script hang up from time to time when using this and I wonder if it's just taking a long time to build and search the tree? I wouldn't mind having it just fail after say 2 seconds or whatever.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Dec 2022, 00:13

@Skrell, there is ConnectionTimeout that you can lower from the default of 2 seconds. From my implementations the only thing you could change is to turn off activateChromiumAccessibility, which might cause a maximum delay of 500ms if the window in question has a Chromium element that hasn't yet been activated, but this shouldn't cause recurring delays. Otherwise I don't see how to reduce delays further, because they will occur either in Microsofts implementation of UIA (which is known to be a bit sluggish) or the window itself (for example I've seen Chrome freeze sometimes for a few seconds when trying to find elements, I believe this is because they have implemented UIA sorta lazily).

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 08 Dec 2022, 08:36

Descolada wrote:
08 Dec 2022, 00:13
@Skrell, there is ConnectionTimeout that you can lower from the default of 2 seconds. From my implementations the only thing you could change is to turn off activateChromiumAccessibility, which might cause a maximum delay of 500ms if the window in question has a Chromium element that hasn't yet been activated, but this shouldn't cause recurring delays. Otherwise I don't see how to reduce delays further, because they will occur either in Microsofts implementation of UIA (which is known to be a bit sluggish) or the window itself (for example I've seen Chrome freeze sometimes for a few seconds when trying to find elements, I believe this is because they have implemented UIA sorta lazily).
From this explanation I don't think what I'm seeing is related. My script literally will hang up for 10-30 seconds before giving me back control and I had assumed it was your library, but now I'll go back and dig I in further.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Dec 2022, 09:32

@Skrell, it seems I had some false assumptions about your problem (for some reason I thought you were having small few-second delays). Such big delays can be caused by large transactions - iterating over a lot of elements with FindFirst/FindAll. You could try stopping that from happening by changing TransactionTimeout, but that might just make some of your Find functions fail unexpectedly. One way would be to run the task in another script so your main one wouldn't freeze?

In any case I'm pretty sure the problem isn't in the UIA_Interface.ahk library, though you could verify that by trying it with jethrow's original library or perhaps the AHK2 version.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 08 Dec 2022, 14:11

Descolada wrote:
08 Dec 2022, 09:32
@Skrell, it seems I had some false assumptions about your problem (for some reason I thought you were having small few-second delays). Such big delays can be caused by large transactions - iterating over a lot of elements with FindFirst/FindAll. You could try stopping that from happening by changing TransactionTimeout, but that might just make some of your Find functions fail unexpectedly. One way would be to run the task in another script so your main one wouldn't freeze?

In any case I'm pretty sure the problem isn't in the UIA_Interface.ahk library, though you could verify that by trying it with jethrow's original library or perhaps the AHK2 version.
Thank you for the feedback! But what is "jethrow's original library"?

Also, I'm sorry for being daft, but how do you actually change TransactionTimeout?
I think I somehow use __Set(member, value) ?
Last edited by Skrell on 08 Dec 2022, 14:21, edited 2 times in total.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Dec 2022, 14:19

@Skrell, it's the one UIA_Interface.ahk is based on, available here. Neptercn has wrote one as well, though I haven't tried it personally.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 08 Dec 2022, 15:37

Descolada wrote:
08 Dec 2022, 14:19
@Skrell, it's the one UIA_Interface.ahk is based on, available here. Neptercn has wrote one as well, though I haven't tried it personally.
Also, I'm sorry for being daft, but how do you actually change TransactionTimeout?
I think I somehow use __Set(member, value) ?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 09 Dec 2022, 00:05

@Skrell, both ConnectionTimeout and TransactionTimeout are part of the main UIA_Interface properties, so you can get the current value with UIA.TransactionTimeout and change it with UIA.TransactionTimeout := 20.

Post Reply

Return to “Scripts and Functions (v1)”