UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
pto
Posts: 29
Joined: 24 Aug 2022, 17:21

Re: UIAutomation with a focus on Chrome

Post by pto » 03 Sep 2022, 18:49

You are very welcome my friend.

Loop
Posts: 155
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 06 Sep 2022, 14:35

Hello,
i want to move mouse on an element without clicking, how can I do that?
Thx

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Sep 2022, 23:03

@Loop, currently there is no built-in method to do that. Easiest way would be to use

Code: Select all

pos := Element.GetCurrentPos()
MouseMouse, % pos.x + pos.w//2, % pos.y+pos.h//2
Though be careful if your coordmode is not set to Screen that the window is active...

Loop
Posts: 155
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 07 Sep 2022, 09:36

many thanks @Descolada

I have the following problem, I move the mouse over a button after 1sec, then a Pic appears to the right of the button, on this picture I must make a left click.

My script works:

Code: Select all

AudioWalker := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType=Button AND Name='Voicemail'"))
NextAudio := AudioWalker.GetNextSiblingElement(FirstAudio)
CurrPos := NextAudio.GetCurrentPos()
MouseMove, % CurrPos.x + CurrPos.w//2 , % CurrPos.y + CurrPos.h//2
Sleep, 1500
Click, % CurrPos.x + 280 " " CurrPos.y " left"
but 280 is the problem, on different computers the distance is different, is there a reliable way.
So click left on an image that is located to the right of a button.

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 07 Sep 2022, 09:41

@Loop, if the image is an element identifiable by UIA then you could use that to position the cursor (you could probably use a TreeWalker to get it). Otherwise if it always appears right next to the button then perhaps Click, % CurrPos.x + CurrPos.w + some value " " CurrPos.y " left" would work more reliably.

Loop
Posts: 155
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 07 Sep 2022, 15:58

if the image is an element identifiable by UIA
No, unfortunately not! The only distinguishing factor is ControlType=image
Thanks, for the suggestion

jekko1976
Posts: 97
Joined: 10 Oct 2014, 07:03

Re: UIAutomation with a focus on Chrome

Post by jekko1976 » 08 Sep 2022, 01:37

Is it possible with UIA to "connect" to an already open Chrome session/page?
In other words, I want to start Chrome manually, then navigate some pages, then make my script "get" this page and finish the job navigating/clicking/ecc...
Thank you

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Sep 2022, 02:36

@jekko1976, yes you can "connect" to a Chrome window at any point. If you are using UIA_Browser then just use cUIA := new UIA_Browser("browser title or ahk_id/ahk_exe").

jekko1976
Posts: 97
Joined: 10 Oct 2014, 07:03

Re: UIAutomation with a focus on Chrome

Post by jekko1976 » 08 Sep 2022, 03:10

@Descolada
Yes!
I have written a test code with this exact page and it seems to work!
Thank you so much!

Code: Select all

#include Lib\UIA_Interface.ahk
#include Lib\UIA_Browser.ahk
#SingleInstance Force
ch := new UIA_Browser("UIAutomation with a focus on Chrome - Page 16 - AutoHotkey Community - Google Chrome")
ch.WaitElementExistByName("Login").Click()

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

UIA for Chromium windows (Skype, Discord etc)

Post by Descolada » 09 Sep 2022, 12:36

I think I've had a breakthrough for the annoying problem of getting UIA access to Chromium windows such as Skype and Discord! The secret appears to be to get the Chrome_RenderWidgetHostHWND1 control and then use that hwnd to get the element! I've added a new method UIA_Interface.ElementFromChromium(winTitle="A", activateChromiumAccessibility=True), and made all other activateChromiumAccessibility flags (eg for ElementFromHandle) ByRef so they will also return that element on the first run.

Example:

Code: Select all

#include <UIA_Interface>
UIA := UIA_Interface()
skypeEl := UIA.ElementFromChromium("ahk_exe skype.exe")
MsgBox, % skypeEl.DumpAll()
Example2:

Code: Select all

#include <UIA_Interface>
UIA := UIA_Interface()
winEl := UIA.ElementFromHandle("ahk_exe skype.exe", skypeEl)
MsgBox, % winEl.DumpAll() ; will not show the content
MsgBox, % skypeEl.DumpAll() ; WILL show the content, but only on the first run (since UIA will be activated only once per run)

; The following will NOT work, since it would be ran for a second time. Use ElementFromChromium for multiple accesses
; winEl := UIA.ElementFromHandle("ahk_exe skype.exe", skypeEl)
; MsgBox, % skypeEl.DumpAll()

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIA for Chromium windows (Skype, Discord etc)

Post by leosouza85 » 10 Sep 2022, 10:41

Descolada wrote:
09 Sep 2022, 12:36
I think I've had a breakthrough for the annoying problem of getting UIA access to Chromium windows such as Skype and Discord! The secret appears to be to get the Chrome_RenderWidgetHostHWND1 control and then use that hwnd to get the element! I've added a new method UIA_Interface.ElementFromChromium(winTitle="A", activateChromiumAccessibility=True), and made all other activateChromiumAccessibility flags (eg for ElementFromHandle) ByRef so they will also return that element on the first run.

Example:

Code: Select all

#include <UIA_Interface>
UIA := UIA_Interface()
skypeEl := UIA.ElementFromChromium("ahk_exe skype.exe")
MsgBox, % skypeEl.DumpAll()
Example2:

Code: Select all

#include <UIA_Interface>
UIA := UIA_Interface()
winEl := UIA.ElementFromHandle("ahk_exe skype.exe", skypeEl)
MsgBox, % winEl.DumpAll() ; will not show the content
MsgBox, % skypeEl.DumpAll() ; WILL show the content, but only on the first run (since UIA will be activated only once per run)

; The following will NOT work, since it would be ran for a second time. Use ElementFromChromium for multiple accesses
; winEl := UIA.ElementFromHandle("ahk_exe skype.exe", skypeEl)
; MsgBox, % skypeEl.DumpAll()
Wow congratulations!! You are the best ahk developer right now!

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 12 Sep 2022, 03:29

@Descolada - stumbled over this, (probably outdated but worth reading): https://www.autoitscript.com/forum/topic/196833-uiaspy-ui-automation-spy-tool/
Hell, and that started in 2013 with now > 40 pages :lol: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/
...where fellow @malcev has provided an AHK script (@ page44) without having been tortured by AU3 fundamentalist afterward :mrgreen: :shifty:

8-)

To whom it may concern: a bunch of "viewers" kindly recommended by malcev: https://github.com/blackrosezy/gui-inspect-tool

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 15 Sep 2022, 20:19

Any way to create a uia element for ahk_class Framework::CFrame? Seems like UIA_Interface isn't a fan of modern Microsoft Office apps?

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 15 Sep 2022, 23:00

@Skrell, it should be possible to use ANY window identifier you use with other AHK functions to also get UIA elements. That means ElementFromHandle("ahk_class Framework::CFrame") is valid, and even using control hwnds is valid (this can be gotten with ControlGet Hwnd and then used with "ahk_id ...").
I haven't played much with Office apps - is there any specific problem you are having trouble with?

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 16 Sep 2022, 08:13

Descolada wrote:
15 Sep 2022, 23:00
@Skrell, it should be possible to use ANY window identifier you use with other AHK functions to also get UIA elements. That means ElementFromHandle("ahk_class Framework::CFrame") is valid, and even using control hwnds is valid (this can be gotten with ControlGet Hwnd and then used with "ahk_id ...").
I haven't played much with Office apps - is there any specific problem you are having trouble with?
My apologies, it was user error. Other topic, is there a way to limit how far down each node the tree builds when using UIA.ElementFromHandle? So for instance, if I print out the results using npEl.DumpAll(), there are a lot of sub-sub-sub nodes displayed. ANy way to limit how deep of these nodes gets built?

Code: Select all

1 Type: 50033 (Pane) LocalizedControlType: "Pane" ClassName: "NetUIPanViewer"
1.1 Type: 50026 (Group) LocalizedControlType: "Group" ClassName: "NetUITWMenuItemGroup"
1.1.1 Type: 50011 (MenuItem) Name: "Copy" LocalizedControlType: "Menu Item" ClassName: "NetUITWBtnMenuItem"
1.1.2 Type: 50011 (MenuItem) Name: "Quick Print" LocalizedControlType: "Menu Item" ClassName: "NetUITWBtnMenuItem"
1.2 Type: 50026 (Group) LocalizedControlType: "Group" ClassName: "NetUITWMenuItemGroup"
1.2.1 Type: 50011 (MenuItem) Name: "Reply" LocalizedControlType: "Menu Item" ClassName: "NetUITWBtnMenuItem"
1.2.2 Type: 50011 (MenuItem) Name: "Reply All" LocalizedControlType: "Menu Item" ClassName: "NetUITWBtnMenuItem"
1.2.3 Type: 50011 (MenuItem) Name: "Forward" LocalizedControlType: "Menu Item" ClassName: "NetUITWBtnMenuItem"

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Sep 2022, 10:36

@Skrell, you can limit DumpAll with the maxDepth argument. el.DumpAll(0) will only "build" once layer, el.DumpAll(1) two layers etc. Though I just noticed this is weird notation (1 should show 1 layer...), so I just fixed this and el.DumpAll(1) will show one layer and el.DumpAll(0) the node itself (the updated UIA_Interface.ahk is in Github).

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 16 Sep 2022, 12:19

Descolada wrote:
16 Sep 2022, 10:36
@Skrell, you can limit DumpAll with the maxDepth argument. el.DumpAll(0) will only "build" once layer, el.DumpAll(1) two layers etc. Though I just noticed this is weird notation (1 should show 1 layer...), so I just fixed this and el.DumpAll(1) will show one layer and el.DumpAll(0) the node itself (the updated UIA_Interface.ahk is in Github).
Sorry, I wasn't clear, I want to speed up UIA.ElementFromHandle by having it build an element with only part of the tree by limiting its depth... is that a possibility?

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Sep 2022, 12:44

@Skrell, ElementFromHandle doesn't "build" anything, it only gets an element from an already existing UIA tree. Building the tree is handled by Microsoft UIAutomation library itself and the application. The only way to speed up ElementFromHandle is to set the "activateChromiumAccessibility" flag to False, which otherwise might cause slowdowns if the app is a Chromium app (Skype, VSCode etc) and doesn't require activating.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 16 Sep 2022, 13:02

Descolada wrote:
16 Sep 2022, 12:44
@Skrell, ElementFromHandle doesn't "build" anything, it only gets an element from an already existing UIA tree. Building the tree is handled by Microsoft UIAutomation library itself and the application. The only way to speed up ElementFromHandle is to set the "activateChromiumAccessibility" flag to False, which otherwise might cause slowdowns if the app is a Chromium app (Skype, VSCode etc) and doesn't require activating.
Got it ty!

jekko1976
Posts: 97
Joined: 10 Oct 2014, 07:03

Re: UIAutomation with a focus on Chrome

Post by jekko1976 » 20 Sep 2022, 09:36

i'd need to fire a select option on a site, on a html like:

Code: Select all

<select name="pagamenti_table_length" aria-controls="pagamenti_table" class="custom-select custom-select-sm form-control form-control-sm">
<option value="10">10</option>
<option value="25">25</option>
<option value="50">50</option>
<option value="100">100</option>
</select>
and eventually fire an change event.
in javascript i would use Setattribute('selected','selected')
Is there a way to do this action with UIAutomation?
Thank you

Post Reply

Return to “Scripts and Functions (v1)”