UIAutomation with a focus on Chrome

[Ralf_Reddings200244]

@Descolada

Thanks for the help Descolada, especially the last example.

>If you don't mind me asking: what is the library that isn't ported over?

I don't mind, It is called MouseGestureL. Its a mouse mouse gesture program and its the best one out there on any platform, not just AHK (trust me, I have looked). The developer has been asked multiple times about plans for AHKV2, THIS person was the most recent person to ask.

The version I use is somewhat modified from the public release to fit my needs and the fact that mouse gestures are a large part of how I use my PC, makes MouseGestureL deeply embedded in many parts of my AHK codebase.

There was another script library that I also make extensive use of and its developer abandoned it but luckily a kind fellow by the name of ntepa ported it over to AHKV2, HERE. So pretty much MouseGestureL is holding the only one me back.

I have been slowly figuring out how its codebase and design and I am confident as my know how increases I will be able to port it over, finger crossed lol.

>What do you mean by "saving the entire application and all of its tree structure"? You can query for multiple elements at a time with FindAllBy, and/or you can cache properties via caching. I can't say whether you can prefetch the AIL object beforehand and interact with it even if it's closed, because it depends on how Illustrator handles its UIA implementations. Usually the UIA object would be invalid

I mean when you do AIL := UIA.ElementFromHandle(WinId), isn't the retrieved UIA data, in this case the illustrator window element and all of its sub elements, being dumped to the AIL variable?

I am thinking, if indeed the retrieved data is dumped to the variable, then what is contained in AIL is just raw data dump, that can be queried with general AHK code such as AIL[1].name or AIL[3].WindowID.

The main reason why I am trying to find about this is mostly to do with your note from Page 1:

> Firstly, every UIA call to interact with elements is a cross-application call from AutoHotkey to the UIA framework, which takes a relatively long time. This means that it is recommended to set tight conditions for searches (like FindAll) and to do as little calls as possible. Doing something like FindAll with the True condition and looping over all the elements to find one specific one is strongly not recommended

Its safe to assume that AIL := UIA.ElementFromHandle(WinId) is a query from AHK to the external UIA interface.

Are further queries to the AIL object such as IDs:= AIL.FindAllByName("ControlType=Pane").GetParentHwnd() still further UIA queries and not just accessing the data that was saved to AIL through pure AHK facilities?

[Gary-Atlan82]

I am looking for a functionality similar to the native ControlGetFocus command in UIA. But I am not sure if its possible.

VsCode currently does not have a hotkey command for just showing the 'debug console' but rather it will show it and also activate it.

I just want to show the panel but not activate it. I am thinking to use UIA to:

• Store the currently active control/panel in a variable

• Trigger the VsCode hotkey to bring up the panel

• Then finally, use UIA to restore focus back to the control/panel that was stored in the variable

Being able to determine and store the currently active control/panel in chromium based windows is something I have not been able to do with native AHK, is this something that is generally achievable in UIA with regards to chromium window?

[Descolada]

@Gary-Atlan82 try something like

Code: Select all

#include UIA_Interface.ahk
UIA := UIA_Interface()
focusedEl := UIA.GetFocusedElement()
; Send hotkey to bring up the panel
; Probably Sleep for a bit to give the panel time to appear/be focused
focusedEl.SetFocus() ; restore the focus to whatever was focused before

[Gary-Atlan82]

@Descolada
That did it, it was that easy. This will make working with chroimium windows a lot more fun now. Thanks so much for the answer and for providing this cool library.

[Gary-Atlan82]

@Descolada

How do I get all of the child items of a element?



I have a simple window that belongs to a programme that acts as a "window manager" or "virtual desktop manager". The programme is really cool and all but has no support for querying information from it "programmatically".

Such as get all the window names assigned to a particular virtual desktop, I discovered I can get all of this information from it using UIA.

I want to build simple arrays of window titles that corresponds to the state of the above screenshot, if I were to do it manually (using the above screenshots data) it would look like this:

Code: Select all

virtDesktop1 := []
virtDesktop1.Push("my firefox window")
virtDesktop1.Push("Document 1 - Painter")

virtDesktop2 := []
virtDesktop2.Push("Document 2 - Painter")
virtDesktop2.Push("media player")
virtDesktop2.Push("document 2 - Markdown reader")

virtDesktop3 := []
virtDesktop3.Push("Document 3 - Painter")
virtDesktop3.Push("document 3  - Markdown reader")

virtDesktop4 := []
virtDesktop4.Push("Document 4 - Painter")
virtDesktop4.Push("image viewer")
virtDesktop4.Push("document 4  - Markdown reader")

I am having trouble achieving this in UIA though. I tried getting all of the groups "Toolbar" control, then using loop through them, storing each "toolbar" child item into a corresponding array:

Code: Select all

UIA          := UIA_Interface()
swticherId	 := WinExist("Virtual Desktops Switcher")
avd	         := UIA.ElementFromHandle(swticherId).FindAllBy("ControlType=ToolBar")

virtDesktop1 := []
virtDesktop2 := []
virtDesktop3 := []
virtDesktop4 := []

loop,% avd.MaxIndex(){
	console(avd[A_Index].dump())
	console(avd[A_Index].FindAllBy("ControlType=Button").Name)
}

But inside the loop I am not able to get the child items of a "toolbar". The only thing that works inside the loop is console(avd[A_Index].dump()).

How do I get all of the child items of a element inside the loop? Also is there a better way to go about this? I am thinking there is a "UIA way" to do this that I am not aware of.

Thanks!

[jchestnut]

I've been using this UIAutomation for long time now and I love it! However, sometime last week when I tried to start it up, it would open a new Chrome window but not navigate to the website.

Not sure what changed or why it won't go to the website.

Code: Select all

#NoEnv
#Warn
#SingleInstance force
SetTitleMatchMode, 2
SetBatchLines, -1
;#include <UIA_Interface> ; Uncomment if you have moved UIA_Interface.ahk to your main Lib folder
#include ..\Lib\UIA_Interface.ahk
;#include <UIA_Browser> ; Uncomment if you have moved UIA_Browser.ahk to your main Lib folder
#include ..\Lib\UIA_Browser.ahk

browserExe := "chrome.exe"
Run, %browserExe% --force-renderer-accessibility ;-incognito ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.Navigate("https://managedprintview.com/#!/dashboard") ; Set the URL and navigate to it. WaitPageLoad is not necessary with Navigate.

[Descolada]

@jchestnut try adding a Sleep 1000 (or perhaps an ever longer sleep for testing purposes) after WinWaitActive, ahk_exe %browserExe%. Namely it waits for Chrome window to exist and be the window with focus, but it doesn't wait for it to actually have any UI elements that UIA_Browser uses and requires. Sleep ensures that Chrome has enough time to load those elements, though it's not 100% reliable.

[dox]

Greetings. I have tried to use UIA Viewer either in v1 or v2 to get at the status of the Zoom Mute/Unmute toggle button. References to others' attempts are below:

viewtopic.php?t=77105

https://github.com/oldjohngalt/ZoomStatus/blob/master/getZoomStatus.py

However, both ZoomEl.ElementFromPath("XXY0YY") from v2 UIA and el.WaitElementExist("ControlType=Pane",,2) or el.FindFirstBy("ControlType=Pane",,2) from the v1 UIAViewer return errors.

I would like to achieve the same goal as these guys (https://devforum.zoom.us/t/easiest-way-to-get-mute-status/18462/10), which is to query the status text of Zoom mic toggle button, so that I do not accidentally unmute it when it is already muted (i.e. getting the toggle "out-of-sync").

Any ideas would be much appreciated in advance!

[Descolada]

@dox since you mention trying v2 as well, I'm gonna provide an answer for that one (AHK v1 has reached end-of-line and I'm not developing UIA for v1 any more):

Code: Select all

#Requires AutoHotkey v2
#include UIA.ahk

zoom := UIA.ElementFromHandle("Zoom Meeting ahk_exe zoom.exe")
MsgBox zoom.FindElement({Type:"Button", Name:"Alt+A", matchmode:"Substring", casesense:False}).Name

[dox]

@Descolada , thanks so much! I see that the macro recorder can be a bit misleading.

I adapted your answer for v2 to using UIAViewer on v1. Basically, in v1 the rough equivalent would be MsgBox % zoom.FindFirstBy("ControlType=Button AND Name='currently unmuted'",,2,False).Name

I know the last call on v1 has ended. But the inertia is hard to overcome.

BTW, greatly appreciate your work on migrating both FindText and UIA to v2. I understand that you took a huge initiative that spurred feiyue (obviously ++ gratitude to @feiyue as well) into completing the v2 port on FindText! I am a Citrix user (not that I have any choice in such matter) and FindText has been frankly indispensable. Your introduction and tutorial have made it accessible to me personally. Else I wouldn't even have known.

At least I can feel reassured that the most crucial libraries for work are solidly on v2 ground. I just gotta get cracking on moving my own stuff ...


P.S., as an aside, if anyone can point to a good "introductory guide" on converting GUI stuff from v1 to v2 it would be kindly appreciated. From cursory reading, it might seem that this area could be among the most daunting for beginners.

[mankvl]

Could someone help to make UIA work when edge tab is not active? If I do:

Code: Select all

#left::
cUIA := new UIA_Browser("ahk_exe msedge.exe") ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.FindFirstByName("Account Identification").ControlClick()
return

its ok when window is active but not when other tab is selected. If I try to invoke with UIAViewer all works ok even when tab is not active.

[Descolada]

@mankvl, no it is not possible as far as I know. Edge only provides UIA access for the active tab, although if you get an element and then switch tabs, the element might continue to exist and work for a while (as it appears to do with UIAViewer).

The only browser I know that exposes elements for all tabs is Firefox.

[Ralf_Reddings200244]

Help with selecting a item from a drop down control please

I have this control on this window that uses QT 5 framework, much to my surprise, almost all of the controls on this window, has full UIA support, with click/get/set etc.

I am struggling to work with drop down type of controls though, I was recently able to figure out how to get the current value it has, but I cant seem to set its value.
Using the UIA viewer, I can see the complete list of items available.

In the viewer when I select one of the list item nodes, a method called select() becomes available on the left side panel but double clicking does nothing, I thought maybe a dialog would open.

Looking at the docs on Github, there is a select() mentioned but could not find a section explaining what it is or how to use it.






Edit1:

Reading some of the past posts on here and the examples over at github, I tried to figure it out.

I tried to store an element that consists of one of the selectable items in the drop down list control to listEl, then convert that to a SelectionItemPattern:

Code: Select all

UIA		:= UIA_Interface()
hwnd		:= WinExist("Lazy Nezumi Pro ahk_exe LazyNezumiPro.exe")
el			:= UIA.ElementFromHandle(hwnd)
listEl	:= el.FindFirstBy("name='Parallel Lines [list]R45'")[/list]
select	:= listEl.GetCurrentPatternAs("SelectionItemPattern")
select.select()

but when select.select() is invoked, the current item in the control does not change, Its also not reflected on the functionality of the window, so nothing has actually changed.

I am going about this the wrong way?

[Descolada]

@Ralf_Reddings200244 Your example looks correctly writte, but just in case - have you tried Click?

Code: Select all

listEl	:= el.FindFirstBy("name='Parallel Lines - R45'")
listEl.Click()

Although I've found that with Qt windows it's easier to get information than actually interact with them. It may very well be that SelectionItemPattern.Select isn't implemented, or some other method such as InvokePattern.Invoke() or LegacyIAccesiblePattern.DoDefaultAction() works. If none of them select the item then you'd need to use either listEl.ControlClick() or listEl.Click("left").

[Ralf_Reddings200244]

@Descolada
No I did not try click() but I tried it after your suggestion I did, it seems to only work if the drop down panel is open but I think I will go with an alternative design using FindText, at least for setting the state of the widnow.

Cheers!

[bearbear]

Just to preface this post, I'm a complete coding novice. I've been trying to use UIA, but it's been hit or miss and I'm not entirely sure what's going wrong.

What I'm trying to achieve:
I have several Chrome windows open on different monitors, each with their own tabs. I want to go through all the tabs and check if a title exists, but only check the Chrome window on my 2nd monitor.

Current code:

Code: Select all

#NoEnv
#SingleInstance force
DetectHiddenWindows, On
SetTitleMatchMode, 2
#Include C:\Program Files\AutoHotkey\Lib\UIA_Browser.ahk

WinGet, chrome_id, List, ahk_exe chrome.exe

Loop, %chrome_id%
{
    hwnd := chrome_id%A_Index%
    WinGetPos, winX,,,, ahk_id %hwnd%
    if (winX >= 1912) ; Window Spy reports the X position of the Chrome window on 2nd monitor as 1912
    {
        WinActivate, ahk_id %hwnd%
        WinWaitActive, ahk_id %hwnd%
        cUIA := new UIA_Browser("ahk_exe chrome.exe")
        SearchTab := cUIA.GetTabs("puppyrescuestream - Twitch", 2, False)
        FoundTab := SearchTab.Count()

        if (FoundTab == 0 && !WinExist("Streamlink ahk_exe cmd.exe")){
            MsgBox, "Window found"
            break
        }
    MsgBox, "Window found but conditions not met"
    break
    } 
    MsgBox, "No Window found"
    break
}

ExitApp

This code only sometimes works (MsgBox, Window found), but if you change focus of the Chrome windows or to another program and run the script again, it can sometimes stop working (MsgBox, No Window found) despite not changing anything other than the focus of some windows.

I cannot understand why this is happening, can anyone shed any light on this?

Also as a side note, I believe the cUIA.GetTabs function breaks if any tab has the Chrome find option open.

[Descolada]

@bearbear if you only need to check for a certain tab and not do anything else with the browser, then you can omit the UIA_Browser call which will slow down your script, because it fetches a lot of elements for navigation etc. Try this instead:

Code: Select all

#NoEnv
#SingleInstance force
DetectHiddenWindows, On
SetTitleMatchMode, 2
#Include <UIA_Interface>

UIA := UIA_Interface()

WinGet, chrome_id, List, ahk_exe chrome.exe

Loop, %chrome_id%
{
    hwnd := chrome_id%A_Index%
    WinGetPos, winX,,,, ahk_id %hwnd%
    if (winX >= 1912) ; Window Spy reports the X position of the Chrome window on 2nd monitor as 1912
    {
        WinActivate, ahk_id %hwnd%
        WinWaitActive, ahk_id %hwnd%

        if (!WinExist("Streamlink ahk_exe cmd.exe") && !UIA.ElementFromHandle("ahk_exe chrome.exe").FindFirstBy("ClassName=TabContainerImpl",, 2).FindFirstBy("Name=puppyrescuestream - Twitch",,2, false)){
            MsgBox, "Window found"
            break
        }
    MsgBox, "Window found but conditions not met"
    break
    } 
    MsgBox, "No Window found"
    break
}

ExitApp

However, if you are getting the message "No Window found" then the problem shouldn't be with UIA at all because otherwise you'd get the message "Window found but conditions not met". I think the problem is in how you are handing the loop. Namely the part

Code: Select all

    MsgBox, "No Window found"
    break

will be executed once any Chrome window is found that is not on the second screen. So if you have two windows open, one on the primary screen and one on the secondary screen, and your loop tries the first one first, then it exits the loop before trying the second one.

Perhaps this will work:

Code: Select all

#NoEnv
#SingleInstance force
DetectHiddenWindows, On
SetTitleMatchMode, 2
#Include <UIA_Interface>

UIA := UIA_Interface()

WinGet, chrome_id, List, ahk_exe chrome.exe

windowFound := false
Loop, %chrome_id%
{
    hwnd := chrome_id%A_Index%
    WinGetPos, winX,,,, ahk_id %hwnd%
    if (winX >= 1912) ; Window Spy reports the X position of the Chrome window on 2nd monitor as 1912
    {
        WinActivate, ahk_id %hwnd%
        WinWaitActive, ahk_id %hwnd%

        if (!WinExist("Streamlink ahk_exe cmd.exe") && !UIA.ElementFromHandle("ahk_exe chrome.exe").FindFirstBy("ClassName=TabContainerImpl",, 2).FindFirstBy("Name=puppyrescuestream - Twitch",,2, false)){
            MsgBox, "Window found"
            windowFound := true
            break
        }
        MsgBox, "Window found but conditions not met, continuing search"
    }
}

if (!windowFound)
    MsgBox, "No Window found"

ExitApp

Also, if you are just starting out then I recommend using AHK v2 with the UIA.ahk library instead. It makes finding bugs much easier, because you get errors if something goes wrong (eg a window isn't found, or an element was not found etc).
The same script in v2:

Code: Select all

#Requires AutoHotkey v2
#include <UIA>

hwnds := WinGetList("ahk_exe chrome.exe")
windowFound := false

for hwnd in hwnds {
    WinGetPos(&X,,,, hwnd)
    if x >= 1912 {
        WinActivate hwnd
        WinWaitActive hwnd

        if (!WinExist("Streamlink ahk_exe cmd.exe") && UIA.ElementFromHandle(hwnd).FindElement({ClassName:"TabContainerImpl", mm:2}).ElementExist({Name:"puppyrescuestream - Twitch", mm:2})) {
            MsgBox "Window found"
            windowFound := true
            break
        }
        MsgBox "Window found but conditions not met, continuing search"
    }
}

if !windowFound
    MsgBox "No window found"

Additionally, I don't know why but in the latest Chrome version searching for a specific tab is quite slow: takes almost a second in my setup. Yet it's much faster in Edge. Idk...

[bearbear]

if you only need to check for a certain tab and not do anything else with the browser, then you can omit the UIA_Browser call

@Descolada I did plan in the future to develop this script further, but I may just revert to using traditional AHK methods of sending keypresses and using WinGetActiveTitle since all this seems to be a bit beyond my capabilities.

I had a feeling the loop logic was causing problems as well as the possibility that Chrome's ahk_ids weren't reliable as unique identifiers, but after trying a variety of things, I still couldn't get it to work the way I wanted.

I've tested out both your revisions, however I don't think the lines !UIA.ElementFromHandle("ahk_exe chrome.exe").FindFirstBy("ClassName=TabContainerImpl",, 2).FindFirstBy("Name=puppyrescuestream - Twitch",,2, false) or UIA.ElementFromHandle(hwnd).FindElement({ClassName:"TabContainerImpl", mm:2}).ElementExist({Name:"puppyrescuestream - Twitch", mm:2}) for v1 and v2 respectively are working.

No matter if I use a ! before that line of code to explicitly include it or not in the if statement, the script seems to ignore it, this is testing with or without that site open. As an additional test, if I remove !WinExist("Streamlink ahk_exe cmd.exe") from the v2 code, then it always gives the error Error: An element matching the condition was not found.

Also, if you are just starting out then I recommend using AHK v2

I've had a look at AHK v2 and while it may seem like a minor difference to experienced coders, but to me as a novice it's quite a step up and less readable/understandable, although I do realise it has far greater coding potential.

Thanks for the help so far.

[Sabestian Caine]

I am also facing a problem related to edge browser....please look this link-
viewtopic.php?p=583840#p583840

[Sabestian Caine]

I am also facing a problem related to edge browser....please look this link-
viewtopic.php?p=583840#p583840