UIAutomation with a focus on Chrome

[Ralf_Reddings200244]

@Descolada

Thanks for the help Descolada, especially the last example.

>If you don't mind me asking: what is the library that isn't ported over?

I don't mind, It is called MouseGestureL. Its a mouse mouse gesture program and its the best one out there on any platform, not just AHK (trust me, I have looked). The developer has been asked multiple times about plans for AHKV2, THIS person was the most recent person to ask.

The version I use is somewhat modified from the public release to fit my needs and the fact that mouse gestures are a large part of how I use my PC, makes MouseGestureL deeply embedded in many parts of my AHK codebase.

There was another script library that I also make extensive use of and its developer abandoned it but luckily a kind fellow by the name of ntepa ported it over to AHKV2, HERE. So pretty much MouseGestureL is holding the only one me back.

I have been slowly figuring out how its codebase and design and I am confident as my know how increases I will be able to port it over, finger crossed lol.

>What do you mean by "saving the entire application and all of its tree structure"? You can query for multiple elements at a time with FindAllBy, and/or you can cache properties via caching. I can't say whether you can prefetch the AIL object beforehand and interact with it even if it's closed, because it depends on how Illustrator handles its UIA implementations. Usually the UIA object would be invalid

I mean when you do AIL := UIA.ElementFromHandle(WinId), isn't the retrieved UIA data, in this case the illustrator window element and all of its sub elements, being dumped to the AIL variable?

I am thinking, if indeed the retrieved data is dumped to the variable, then what is contained in AIL is just raw data dump, that can be queried with general AHK code such as AIL[1].name or AIL[3].WindowID.

The main reason why I am trying to find about this is mostly to do with your note from Page 1:

> Firstly, every UIA call to interact with elements is a cross-application call from AutoHotkey to the UIA framework, which takes a relatively long time. This means that it is recommended to set tight conditions for searches (like FindAll) and to do as little calls as possible. Doing something like FindAll with the True condition and looping over all the elements to find one specific one is strongly not recommended

Its safe to assume that AIL := UIA.ElementFromHandle(WinId) is a query from AHK to the external UIA interface.

Are further queries to the AIL object such as IDs:= AIL.FindAllByName("ControlType=Pane").GetParentHwnd() still further UIA queries and not just accessing the data that was saved to AIL through pure AHK facilities?

[Gary-Atlan82]

I am looking for a functionality similar to the native ControlGetFocus command in UIA. But I am not sure if its possible.

VsCode currently does not have a hotkey command for just showing the 'debug console' but rather it will show it and also activate it.

I just want to show the panel but not activate it. I am thinking to use UIA to:

• Store the currently active control/panel in a variable

• Trigger the VsCode hotkey to bring up the panel

• Then finally, use UIA to restore focus back to the control/panel that was stored in the variable

Being able to determine and store the currently active control/panel in chromium based windows is something I have not been able to do with native AHK, is this something that is generally achievable in UIA with regards to chromium window?

[Descolada]

@Gary-Atlan82 try something like

Code: Select all

#include UIA_Interface.ahk
UIA := UIA_Interface()
focusedEl := UIA.GetFocusedElement()
; Send hotkey to bring up the panel
; Probably Sleep for a bit to give the panel time to appear/be focused
focusedEl.SetFocus() ; restore the focus to whatever was focused before

[Gary-Atlan82]

@Descolada
That did it, it was that easy. This will make working with chroimium windows a lot more fun now. Thanks so much for the answer and for providing this cool library.

[Gary-Atlan82]

@Descolada

How do I get all of the child items of a element?



I have a simple window that belongs to a programme that acts as a "window manager" or "virtual desktop manager". The programme is really cool and all but has no support for querying information from it "programmatically".

Such as get all the window names assigned to a particular virtual desktop, I discovered I can get all of this information from it using UIA.

I want to build simple arrays of window titles that corresponds to the state of the above screenshot, if I were to do it manually (using the above screenshots data) it would look like this:

Code: Select all

virtDesktop1 := []
virtDesktop1.Push("my firefox window")
virtDesktop1.Push("Document 1 - Painter")

virtDesktop2 := []
virtDesktop2.Push("Document 2 - Painter")
virtDesktop2.Push("media player")
virtDesktop2.Push("document 2 - Markdown reader")

virtDesktop3 := []
virtDesktop3.Push("Document 3 - Painter")
virtDesktop3.Push("document 3  - Markdown reader")

virtDesktop4 := []
virtDesktop4.Push("Document 4 - Painter")
virtDesktop4.Push("image viewer")
virtDesktop4.Push("document 4  - Markdown reader")

I am having trouble achieving this in UIA though. I tried getting all of the groups "Toolbar" control, then using loop through them, storing each "toolbar" child item into a corresponding array:

Code: Select all

UIA          := UIA_Interface()
swticherId	 := WinExist("Virtual Desktops Switcher")
avd	         := UIA.ElementFromHandle(swticherId).FindAllBy("ControlType=ToolBar")

virtDesktop1 := []
virtDesktop2 := []
virtDesktop3 := []
virtDesktop4 := []

loop,% avd.MaxIndex(){
	console(avd[A_Index].dump())
	console(avd[A_Index].FindAllBy("ControlType=Button").Name)
}

But inside the loop I am not able to get the child items of a "toolbar". The only thing that works inside the loop is console(avd[A_Index].dump()).

How do I get all of the child items of a element inside the loop? Also is there a better way to go about this? I am thinking there is a "UIA way" to do this that I am not aware of.

Thanks!

[jchestnut]

I've been using this UIAutomation for long time now and I love it! However, sometime last week when I tried to start it up, it would open a new Chrome window but not navigate to the website.

Not sure what changed or why it won't go to the website.

Code: Select all

#NoEnv
#Warn
#SingleInstance force
SetTitleMatchMode, 2
SetBatchLines, -1
;#include <UIA_Interface> ; Uncomment if you have moved UIA_Interface.ahk to your main Lib folder
#include ..\Lib\UIA_Interface.ahk
;#include <UIA_Browser> ; Uncomment if you have moved UIA_Browser.ahk to your main Lib folder
#include ..\Lib\UIA_Browser.ahk

browserExe := "chrome.exe"
Run, %browserExe% --force-renderer-accessibility ;-incognito ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.Navigate("https://managedprintview.com/#!/dashboard") ; Set the URL and navigate to it. WaitPageLoad is not necessary with Navigate.

[Descolada]

@jchestnut try adding a Sleep 1000 (or perhaps an ever longer sleep for testing purposes) after WinWaitActive, ahk_exe %browserExe%. Namely it waits for Chrome window to exist and be the window with focus, but it doesn't wait for it to actually have any UI elements that UIA_Browser uses and requires. Sleep ensures that Chrome has enough time to load those elements, though it's not 100% reliable.

[dox]

Greetings. I have tried to use UIA Viewer either in v1 or v2 to get at the status of the Zoom Mute/Unmute toggle button. References to others' attempts are below:

viewtopic.php?t=77105

https://github.com/oldjohngalt/ZoomStatus/blob/master/getZoomStatus.py

However, both ZoomEl.ElementFromPath("XXY0YY") from v2 UIA and el.WaitElementExist("ControlType=Pane",,2) or el.FindFirstBy("ControlType=Pane",,2) from the v1 UIAViewer return errors.

I would like to achieve the same goal as these guys (https://devforum.zoom.us/t/easiest-way-to-get-mute-status/18462/10), which is to query the status text of Zoom mic toggle button, so that I do not accidentally unmute it when it is already muted (i.e. getting the toggle "out-of-sync").

Any ideas would be much appreciated in advance!

[Descolada]

@dox since you mention trying v2 as well, I'm gonna provide an answer for that one (AHK v1 has reached end-of-line and I'm not developing UIA for v1 any more):

Code: Select all

#Requires AutoHotkey v2
#include UIA.ahk

zoom := UIA.ElementFromHandle("Zoom Meeting ahk_exe zoom.exe")
MsgBox zoom.FindElement({Type:"Button", Name:"Alt+A", matchmode:"Substring", casesense:False}).Name

[dox]

@Descolada , thanks so much! I see that the macro recorder can be a bit misleading.

I adapted your answer for v2 to using UIAViewer on v1. Basically, in v1 the rough equivalent would be MsgBox % zoom.FindFirstBy("ControlType=Button AND Name='currently unmuted'",,2,False).Name

I know the last call on v1 has ended. But the inertia is hard to overcome.

BTW, greatly appreciate your work on migrating both FindText and UIA to v2. I understand that you took a huge initiative that spurred feiyue (obviously ++ gratitude to @feiyue as well) into completing the v2 port on FindText! I am a Citrix user (not that I have any choice in such matter) and FindText has been frankly indispensable. Your introduction and tutorial have made it accessible to me personally. Else I wouldn't even have known.

At least I can feel reassured that the most crucial libraries for work are solidly on v2 ground. I just gotta get cracking on moving my own stuff ...


P.S., as an aside, if anyone can point to a good "introductory guide" on converting GUI stuff from v1 to v2 it would be kindly appreciated. From cursory reading, it might seem that this area could be among the most daunting for beginners.

[mankvl]

Could someone help to make UIA work when edge tab is not active? If I do:

Code: Select all

#left::
cUIA := new UIA_Browser("ahk_exe msedge.exe") ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.FindFirstByName("Account Identification").ControlClick()
return

its ok when window is active but not when other tab is selected. If I try to invoke with UIAViewer all works ok even when tab is not active.

[Descolada]

@mankvl, no it is not possible as far as I know. Edge only provides UIA access for the active tab, although if you get an element and then switch tabs, the element might continue to exist and work for a while (as it appears to do with UIAViewer).

The only browser I know that exposes elements for all tabs is Firefox.