UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Youhoney
Posts: 8
Joined: 16 Jul 2022, 03:30

Re: UIAutomation with a focus on Chrome

Post by Youhoney » 16 Jul 2022, 12:10

@Descolada, wow, thanks, your code fixed the problem! :clap: I can now click the button without any error message. However, it created another mysterious problem. After clicking the button the program opens a new window with a different handle. Seemingly from nowhere, I'm now getting an error when I try to get the element from the handle of the new window. This did not cause any problems before when I used the ControlGet-function. Any idea why this might happen?

So the following code works without problems:

Code: Select all

KäynninYhteenveto := UIA.ElementFromHandle(WinExist("Käynnin yhteenveto"))
Kontaktitiedot := KäynninYhteenveto.FindFirstBy("ClassName=ContactDataControl")
Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink").ControlClick()
Send, {Enter} ; For some reason, ControlClick only sets the focus to the button, but doesn't open the link. That's why I send an Enter here.
WinWaitActive, Laskutettava palvelu, , 10
LaskutettavaPalvelu := UIA.ElementFromHandle(WinExist("Laskutettava palvelu"))
And with the change you suggested it gives an error:

Code: Select all

KäynninYhteenveto := UIA.ElementFromHandle(WinExist("Käynnin yhteenveto"))
Kontaktitiedot := KäynninYhteenveto.FindFirstBy("ClassName=ContactDataControl")
saveConnectionTimeout := UIA.ConnectionTimeout, saveTransactionTimeout := UIA.TransactionTimeout ; Save previous values for timeouts
UIA.ConnectionTimeout := 500, UIA.TransactionTimeout := 200 ; Set shorter timeouts (could experiment with even shorter ones, minimum is 50)
try Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink").Click() ; Use "try" to avoid the error being thrown
UIA.ConnectionTimeout := saveConnectionTimeout, UIA.TransactionTimeout := saveTransactionTimeout ; Restore previous timeout values
WinWaitActive, Laskutettava palvelu, , 10
LaskutettavaPalvelu := UIA.ElementFromHandle(WinExist("Laskutettava palvelu"))
Attachments
error.png
error.png (61.49 KiB) Viewed 3581 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Jul 2022, 13:05

@Youhoney, we are in uncharted territories right now, since I have never found a good use for changing the timeout values before. So I have zero experience with it ;) I have a theory though: that error is a timeout error. This means that either the new changes haven't been applied yet; the last action (Invoke) hasn't timed out yet (we know that the problem is that it gets stuck somehow); or your program is still busy handling the invoke event.
We could try to let the last action properly time out before continuing? So add a sleep after the Click method. If that doesn't work, then I think your program might be stuck for some time (10-20 seconds) processing the Invoke and you can't use any UIA commands during that time. In that case the Click("left") and ControlClick ways are left...

So try this:

Code: Select all

KäynninYhteenveto := UIA.ElementFromHandle(WinExist("Käynnin yhteenveto"))
Kontaktitiedot := KäynninYhteenveto.FindFirstBy("ClassName=ContactDataControl")
saveConnectionTimeout := UIA.ConnectionTimeout, saveTransactionTimeout := UIA.TransactionTimeout ; Save previous values for timeouts
UIA.ConnectionTimeout := 500, UIA.TransactionTimeout := 200 ; Set shorter timeouts (could experiment with even shorter ones, minimum is 50)
try Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink").Click() ; Use "try" to avoid the error being thrown
Sleep, 500 ; Let it definitely time out after the click
UIA.ConnectionTimeout := saveConnectionTimeout, UIA.TransactionTimeout := saveTransactionTimeout ; Restore previous timeout values
WinWaitActive, Laskutettava palvelu, , 10
For the ControlClick version, perhaps using element.SetFocus() would work instead of sending Enter?

Code: Select all

KäynninYhteenveto := UIA.ElementFromHandle(WinExist("Käynnin yhteenveto"))
Kontaktitiedot := KäynninYhteenveto.FindFirstBy("ClassName=ContactDataControl")
hyperlinkEl := Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink")
hyperlinkEl.SetFocus()
hyperlinkEl.ControlClick()
WinWaitActive, Laskutettava palvelu, , 10
LaskutettavaPalvelu := UIA.ElementFromHandle(WinExist("Laskutettava palvelu"))

Youhoney
Posts: 8
Joined: 16 Jul 2022, 03:30

Re: UIAutomation with a focus on Chrome

Post by Youhoney » 16 Jul 2022, 13:23

@Descolada, thanks again for your efforts, I tried both of those. Unfortunately adding the time out didn't change the behaviour. Must be something weird with how the program implements UIA.

I also tried SetFocus, didn't work. Perhaps I'll use my old Acc library method, which works, albeit a bit slow.

Code: Select all

WinActivate, Käynnin yhteenveto
WinGet, hWnd, ID, A
oAcc := Acc_Get("Object", "4.1.3.2.2.2.36.1", 0, "ahk_id " hWnd)
oAcc.accDoDefaultAction(0)
Let me know if there's a way to buy you a coffee, I'd be happy to. :thumbup:

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Jul 2022, 13:50

@Youhoney, unfortunate that it didn't work :( Though I am surprised that Acc works, since the element you showed in the UIAViewer screenshot doesn't contain the LegacyIAccessible pattern. Could it be that the hyperlink element is not actually the clickable one, but instead the text is? In that case it should easily be clickable with UIA...

Code: Select all

Kontaktitiedot.FindFirstByName("Laskutettavat tuotteet...").Click()
or

Code: Select all

Kontaktitiedot.FindFirstByName("Laskutettavat tuotteet...").GetCurrentPatternAs("LegacyIAccessible").DoDefaultAction() ; Specifically use Acc
Let me know if there's a way to buy you a coffee, I'd be happy to. :thumbup:
Thanks, perhaps when I happen to go to Finland ;)

Youhoney
Posts: 8
Joined: 16 Jul 2022, 03:30

Re: UIAutomation with a focus on Chrome

Post by Youhoney » 16 Jul 2022, 14:52

@Descolada, beats me! Couldn't click the text either, it really seems only to be a text. Well, I do have fairly reliable ways to access it so no worries.
Thanks, perhaps when I happen to go to Finland ;)
Hit me with a PM if you do! :)

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 18 Jul 2022, 23:39

hi @Descolada, is there a way to invoke an item on the context menu without having to bring it up first? or a closest alternative if there is not?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 19 Jul 2022, 00:48

SundayProgrammer wrote:
18 Jul 2022, 23:39
hi @Descolada, is there a way to invoke an item on the context menu without having to bring it up first? or a closest alternative if there is not?
As far as I know, if the context menu item does not appear in the UIA tree before being visible, then it's not possible to invoke before making it visible.
I actually haven't heard of a way of doing that with other methods ( such as SendMessage) either, except if the program has implemented COM or win32 message systems to interact with it.

For alternative methods, knowing more about what you are trying to achieve would help :)

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 19 Jul 2022, 03:12

Descolada wrote:
19 Jul 2022, 00:48
For alternative methods, knowing more about what you are trying to achieve would help :)
thanks.

scope "youtube + msedge": when i rightclick on a video thumbnail, the context menu is supposed to have additional items related to image "processing". however, i noticed that, when the thumbnail has preview on hover, the rightclick becomes less reliable - it may not come with those image related items. i have to sometimes (more than 50% to my impression) rightclick the 2nd time (or even the 3rd time) to get those items shown. that's why i was thinking to bypass the context menu altogether to get around this "bug". any ideas?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 19 Jul 2022, 09:05

@SundayProgrammer, I think those image related context menu elements are only shown if you actually have a image underneath your cursor. When hovering over the video thumbnail on Youtube main page, then at the beginning it is an image, but eventually becomes larger and starts playing the preview: it is now video, not an image, so it can't have image-related menu items.

This should detect if there is an image underneath the cursor, though it sometimes still fails (probably because the UIA tree doesn't change as fast as the image turns into a video...)

Code: Select all

SetBatchLines, -1
global UIA := UIA_Interface()
RButton::
    if UIA.ElementFromPoint().FindFirstByType("Image", 0x5)
            Click, right
    return
Probably a safer bet would be to use Rufaydium/Chrome.ahk to extract the image URL from the source code :) (Could probably do it with UIA_Browser.JSExecute() as well, but less reliably)

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 19 Jul 2022, 10:15

@Descolada, thank you for spending time on my question. yep, javascript is an alternative that i did think of. the problem is, i don't know the function/method name behind the menu items. and have no clue how to get such information.

i wanted to dig into rufaydium, yet months gone by, still not started though. probably not until i have something big enough to achieve, i guess. and that something is relatively easy by using rufaydium.

for chrome.ahk, it requires the browser to afaik start with a particular option (debug mode something i can't recall) which is less favorable to me.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 19 Jul 2022, 16:18

Hola @Descolada I hope you are feeling well.
I am new to AutoHotKey :)
I am learning about scope in FindFirstBy. Could you explain me a little bit how it works?

I'm trying to click on this element, but as I told you before, it seems that pages that have a lot of text take a long time to search and during that time Chrome delay a lot of time, sometimes crashes. So I have this idea of telling the library to only search in a specific place. In this case the group that this element belongs to. Do you think I should change something in scope?

EDIT: One thing I noticed is that the first few times the slowness happens, but then it works fine and takes almost no time at all. Until I change pages and browse for a while. Then I have the same problem again.
ggg.jpg
ggg.jpg (168.94 KiB) Viewed 2613 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 19 Jul 2022, 23:20

@mora145, hey!
Scope simply tells the function how deep to search. The default is 4, which means TreeScope_Descendants: scope includes children, and children of the children all the way down. You could try going down layer by layer by setting scope to 2 (TreeScope_Children), which would search only the direct children. Judging from the tree you provided, you would probably have to do this 2-3 times.

Another thing to try would be to try Edge instead of Chrome. I've noticed that Microsoft has implemented UIAutomation in Edge quite well (probably better than Chrome), and there are speed differences as well.

The slowness on first times might be related to caching: either UIA caches the tree internally for faster use later, or Chrome builds the tree in response to you trying to search it, in either case the work is done on the first search and subsequently things should be faster. This also explains why it's slow again when you refresh the page.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 20 Jul 2022, 12:19

I understand. Thanks for clarifying :)
I also have a doubt about this: The text I am looking for is within a specific group (the only one in the Tree). So if I tell the interface to first search for the Group and then search for the text, this would be done in the same way or two separate searches will be done anyway?

Code: Select all

Group1 := cE2.FindFirstBy("ControlType=Group")
TypeText := Group1.FindFirstBy("Name=Type your translation here").Click()

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 20 Jul 2022, 12:27

@mora145, I think that will not speed it up, because the scope of the first search is still Treescope_Descendants... The first searches should be FindFirstBy("...", Treescope_Children := 2), only the last one (hopefully with a small subtree) with scope Treescope_Descendants.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 20 Jul 2022, 14:31

I see. Thank you again @Descolada

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 21 Jul 2022, 21:30

I just saw that you had already posted the tutorial. It looks very good and explanatory. I am starting to study it.

Thanks for the time you took to do it.

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 25 Jul 2022, 18:17

Hi.
I think FindFirstByName does not accept 'OR'. Should it accept it?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 25 Jul 2022, 23:05

@mora145, you are correct, FindFirstByName accepts exactly one name. To match multiple criteria you need to use FindFirstBy("Name=name1 OR Name=name2").
Also I'm glad that someone found the Wiki already, I hope its helpful :) And if you find any mistakes/bugs there, let me know!

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 26 Jul 2022, 01:26

Descolada, I have not seen that there is a way to verify that X action was completed. In AutoHotKey normally Errorlevel is used.
I have a script that even though it doesn't click it still works, and it should not.

Code: Select all

   WinActivate, ahk_exe chrome.exe
            Group1 := cE2.FindFirstBy("ControlType=Group AND NOT ControlType=List")
            TypeText := Group1.FindFirstBy("Name=Type your translation here OR Name=Escriba aquí su traducción", 0x4).Click(3)
            Sleep 500
            Send {LControl down}{v}
            Send {LControl up}
            Clipboard_save2 := Clipboard
            Sleep 500
            Send {End}
            Send {Down}
            Send {BackSpace}
            Send {Delete 3}
            MouseMove, OutputVarX, OutputVarY
If this Name is not found, then wait 10 seconds and try again.

Code: Select all

TypeText := Group1.FindFirstBy("Name=Type your translation here OR Name=Escriba aquí su traducción", 0x4).Click(3)
EDIT: Yes, I'll keep an eye on everything I see (although I'm still a newbie) :D

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 26 Jul 2022, 05:07

@mora145, Click doesn't throw an error, because it's not possible to determine whether the program responded to it or not. Also if the element isn't found then calling click won't throw an error, because apparently AHK allows doing "".Click() without an error.

But you CAN check whether the element you want to click exists: either use WaitElementExist instead if FindFirstBy with an appropriate timeout value (-1 for indefinite wait), or use

Code: Select all

if (el := Group1.FindFirstBy("Name=..."))
    el.Click()

Post Reply

Return to “Scripts and Functions (v1)”