UIAutomation with a focus on Chrome

[Loop]

Hello, I always want to click on the last entry. The problem is, AutomationId='one-column-item-22 is sometimes 0 and sometimes 22. So it keeps changing. How can I find out the last AutomationId?

el.FindFirstBy("ControlType=Button AND AutomationId='one-column-item-22'").Click()

Thanks!

[Descolada]

@Loop, one way would be to use a TreeWalker to get the last element with a condition that matches for substrings (flag=2):

Code: Select all

foundEl := UIA.CreateTreeWalker("ControlType=Button AND AutomationId='one-column-item' FLAGS=2").GetLastChildElement(el)

Another way would be to use that same condition but with FindFirstWithOptions, using LastToFirst traversal option:

Code: Select all

foundEl := el.FindFirstWithOptions(, "ControlType=Button AND AutomationId='one-column-item' FLAGS=2", 2, el)

If you were to use AHK v2 with UIA-v2, then it would be foundEl := el.FindElement({Type:"Button", AutomationId:"one-column-item", matchmode:"SubString", index:-1})

[Gary-Atlan82]

@Descolada
UIAutomation SetFocus() is not working for me

In the following example, the .Highlight() line works fine but the .SetFocus() line does not. The window get activated but nothing else happens.
UIA cant set the values in this program so I am trying to do it manually myself but I first need to activate the edit field:

Code: Select all

UIA := UIA_Interface()
el := WinExist("Pencil Tool Properties ahk_exe PDFXEdit.exe ahk_class UIX:Window")
WinActivate, ahk_id %el%
WinWaitActive, ahk_id %el%
el := UIA.ElementFromHandle(el)
; el.FindFirstBy("ControlType=Text AND Name='6 pt' AND AutomationId='1@'").Highlight()
el.FindFirstBy("ControlType=Text AND Name='6 pt' AND AutomationId='1@'").SetFocus()

Note: the width property has to have "6" for UIA to find it.

Thanks for any help.

[Descolada]

@Gary-Atlan82, yeah, for some reason SetFocus doesn't work. I changed the code to target the parent of the Text element (a ListItem type element) and instead Clicked it, which seems to focus it. Then after clicking the UIA tree updates with a new Combobox element that has an "Open" Button element to reveal the pencil size selection menu.
I also changed the way the pencil element is found by using Name=' pt' along with matchmode=2, so the pencil size element can be found independent of the currently selected value.

Code: Select all

el.FindFirstBy("ControlType=ListItem AND Name=' pt' AND AutomationId='2@'",,2).Click()
; el.FindFirstBy("Type=Button AND Name='Open'").Click() ; opens the combo-box

[Gary-Atlan82]

Aah! even better. Thanks so much man

[Gary-Atlan82]

@Descolada
UIAutomation: How to perform Invoke() on a tray icon while the taskbar is hidden?

I am trying to write a script using UIAutomation, that will click on a icon present in the tray bar (right side of the taskbar).
Most of the time, my taskbar is hidden but I found the Invoke() button in the UIA Viewer window (Found in "UIAutomation Element Patterns" sidebar) works even when the taskbar is hidden!



I am having issue actually invoking this Inokve() pattern in a AHK script, I have tried:

Code: Select all

UIA		      := UIA_Interface()
WinID		      := WinExist("ahk_class Shell_TrayWnd ahk_exe Explorer.EXE")
TaskBar	      := UIA.ElementFromHandle(WinID)
; TransMisson  := TaskBar.FindFirstBy("ControlType=Button AND Name=' Idle' AND AutomationId='NotifyItemIcon'").ControlClick()                                ;This seems to be a standard AHK mousemove with Click function?
TransMisson		:= TaskBar.FindFirstBy("ControlType=Button AND Name=' Idle' AND AutomationId='NotifyItemIcon'").LegacyIAccessiblePattern.DoDefaultAction()    ;Does Nothing, no mouse movement observed
; TransMisson	:= TaskBar.FindFirstBy("ControlType=Button AND Name=' Idle' AND AutomationId='NotifyItemIcon'").Click()                                       ;Does Nothing either

What I am expecting to happen is the window being brought to focus, same thing as clicking on the icon.

Also when .Click() is used without any parameters is it performing one of the AHK mouse clicks or is it doing a UIA method? What method is actually doing the Invoke() pattern?
Any help would be greatly appreciated!

[Descolada]

@Gary-Atlan82, Element.Click() will use one of the UIA methods and the first one it tries is Invoke. You can also specifically call Invoke from the InvokePattern by using Element.InvokePattern.Invoke(). Element.ControlClick() will use AHK ControlClick function to click it, and Element.Click("left") will use AHK Click function.

If you were to switch to AHK v2 and UIA-v2 then Element.Click() works the same, but for InvokePattern you could simply use Element.Invoke() because the pattern is auto-detected.

[Gary-Atlan82]

@Descolada

I was not able to get Element.InvokePattern.Invoke() to work but this mostly likely due to me hiding the taskbar though, the following works:

Code: Select all

#SingleInstance, force
SetBatchLines, -1
Process, Exist, transmission-qt.exe
If !(ErrorLevel){
		Run, "C:\Program Files\Transmission\transmission-qt.exe"
		WinWait , ahk_exe transmission-qt.exe,, 1
		WinActivate, ahk_exe transmission-qt.exe,, 1
		WinWaitActive, ahk_exe transmission-qt.exe,, 1
	}
Else If WinExist("ahk_exe transmission-qt.exe"){
	WinActivate, ahk_exe transmission-qt.exe,, 1
	WinWaitActive, ahk_exe transmission-qt.exe,, 1
	}
Else{
	If !WinExist("ahk_class Shell_TrayWnd"){
		WinShow, ahk_class Shell_TrayWnd
		WinShow, ahk_class Shell_SecondaryTrayWnd
		TaskbarIsShown := 1
		}
	UIA		:= UIA_Interface()
	WinID		:= WinExist("ahk_class Shell_TrayWnd ahk_exe Explorer.EXE")
	WinID		:= WinExist(" ahk_exe Explorer.EXE ahk_class Shell_TrayWnd")
	TaskBar	:= UIA.ElementFromHandle(WinID)
	Trans		:= TaskBar.FindFirstBy("ControlType=Button AND Name=' Idle' AND AutomationId='NotifyItemIcon'").Click()
	If (TaskbarIsShown){
		WinHide, ahk_class Shell_TrayWnd
		WinHide, ahk_class Shell_SecondaryTrayWnd
	}
	WinActivate, ahk_exe transmission-qt.exe,, 1
	WinWaitActive, ahk_exe transmission-qt.exe,, 1
	}
ExitApp

If you are thinking "All of this to bring up a window", sadly yes. Its a broken program (maybe broken by the Win11 taskbar) and I have made many thread asking for help about it. Until now I have not been able to solve it.

With UIA it was a piece of work. Thank you for writing this library, I plan to do so much with it! and your clarification with Invoke() will help allot going forward. Cheers!

[Gary-Atlan82]

@Descolada
How can I use the RegEX option to solve this problem?
So the program I wrote in the above post yesterday broke today. The issue when there is a file being downloaded, it will show the download speed in the tray, such as 598 kB/s ▾ 0 kB/s ▴. This also apllies to the control name in UIA.






I want to build a script that will send a click to the tray Icon whether its name is Idle or something like 598 kB/s ▾ 0 kB/s ▴. I have tried the following but its not working:

Code: Select all

UIA		:= UIA_Interface()
WinID		:= WinExist(" ahk_exe Explorer.EXE ahk_class Shell_TrayWnd")
MsgBox, % WinId
TaskBar	:= UIA.ElementFromHandle(WinID)
; TransMission := TaskBar.FindFirstBy("ControlType=Image").Click()
; TransMission := TaskBar.FindFirstBy("ControlType=Button AND Name=' Idle' AND AutomationId='NotifyItemIcon'").Click()
; TransMission := TaskBar.FindFirstBy("ControlType=Button AND Name='▾| Idle' AND AutomationId='NotifyItemIcon'",, RegEx).Click()
; TransMission := TaskBar.FindFirstBy("ControlType=Button AND Name='kb| Idle' AND AutomationId='NotifyItemIcon'",, RegEx).Click()
TransMission	:= TaskBar.FindFirstBy("'▾| Idle'",, RegEx).Click()

I am using the RegEx option the right way?

Thanks for any help.

[Descolada]

@Gary-Atlan82, you almost got it, but you are using RegEx as a variable, not as a string-type option. TransMission := TaskBar.FindFirstBy("'▾| Idle'",, "RegEx").Click() is correct, or if you do actually want to use a variable then:

Code: Select all

RegEx := "RegEx"
TransMission	:= TaskBar.FindFirstBy("'▾| Idle'",, RegEx).Click()

[Gary-Atlan82]

@Descolada Thanks for the help!

I got skewered by such a Noobie mistake. Well I am Noob but AHK has one acomplishing feats so fast, one starts to feel like a level 99 hacker.

I appreciate the varied examples of Var vs String.

Anyways, I want to confirm following works for me:

Code: Select all

	UIA		:= UIA_Interface()
	WinID		:= WinExist(" ahk_exe Explorer.EXE ahk_class Shell_TrayWnd")
	TaskBar	:= UIA.ElementFromHandle(WinID)
	Trans		:= TaskBar.FindFirstBy("ControlType=Button AND Name='/s|Idle' AND AutomationId='NotifyItemIcon'",, "RegEx").Click()

[songdg]

Thanks again, can UIA_Browser connect to an already existing Chrome tab?

[Descolada]

@songdg, yes and no. With UIA_Browser you can connect to a *window* with new UIA_Browser(WinTitle), which might be the same thing as the tab you want to connect to. If that Chrome window has the specific tab you wish to connect to, then you can use cUIA.SelectTab to switch to it.

But, I am not actively updating UIA_Browser nor UIA_Inteface for v1, and since browser automation with UIA is tricky and browsers tend to be updated, then it might stop working at some time (if it hasn't already). I recommend switching to AHK v2 and UIA.ahk if at all possible.

[songdg]

@Descolada, thanks for your reply, I currently use Chrome.ahk to scrape some information from a website, but with little knowledge of javascript, sometime it's very difficult for me to use Chrome.ahk to click a button or fill an edit, yet use UIAutomation is quite easy to do that, so I want to combine them to do my job. Unfortunately Chrome.ahk no longer update, I wonder if UIA.ahk@V2 will be easy to perform simple scrape jobs if possible.

[Gary-Atlan82]

@Descolada
Search for a element and return the Hwnd of its Window?
I want to heavily utilise AutoHotkey with Adobe Illustrator, unfortunately, aside from the main window and document windows, Illustrator "Panels" windows do not have window titles at all, every "Panel" window has only the following information:

Code: Select all

ahk_class OWL.Dock
ahk_exe Illustrator.exe
ahk_pid 27784

I cant use any of the cool Functions/Libraries I have, If I cant tell what's what.

What I am trying to do
At first launch of the program or any change of "WorkSpace", run AHK script, that will look for windows that do not have titles and set them, via WinsetTitle, so the Layers window when first created goes from:

Code: Select all

ahk_class OWL.Dock
ahk_exe Illustrator.exe
ahk_pid 27784

to

Code: Select all

Layers
ahk_class OWL.Dock
ahk_exe Illustrator.exe
ahk_pid 27784

As for the problem of identifying which window is which, I think UIA can help with this, because it can actually get allot of information from Illustrator:


I want UIA to go through the Illustrator application tree and find all the tab item "Layers" or tab item "Navigator" and return a series of Hwnd for the windows that elements belongs to.

If two or more tab item "..." belong to one window, such as in THIS example, that is fine as I want to do something like this anyways:

Code: Select all

Layers | Transparency | Document Info | Info
ahk_class OWL.Dock
ahk_exe Illustrator.exe
ahk_pid 27784

I only know abit about UIA, certainly not enough to recurse a series of windows. Any help or pointers would be really appreciated. Thanks.

[Descolada]

@Gary-Atlan82, if you are dealing with separate windows (not controls), then looping over them shouldn't be too difficult, something like this:

Code: Select all

#include <UIA_Interface>

UIA := UIA_Interface()
WinGet, wList, List, ahk_class OWL.Dock ahk_exe Illustrator.exe

Loop %wList% 
{
    wId := wList%A_index%
    WinGetTitle, wTitle, ahk_id %wId%
    if (wTitle)
        continue
    winEl := UIA.ElementFromHandle("ahk_id" wId)
    if (foundEl := winEl.FindFirst("Name='Layers' OR Name='Navigator' AND Type=TabItem")) { ; multiple and/or statements are interpreted left-to-right, so this is (Name='Layers' OR Name='Navigator') AND Type=TabItem
        WinActivate, ahk_id %wId%
        foundEl.Highlight()
        MsgBox, % "Window with winId " wId " contains TabItem Layers or Navigator"
        ;WinSetTitle, ahk_id %wId%,, Layers
    }
}

If you need access to a control that is embedded in a window, then you'd need to loop the windows and for every window loop over WinGet ControlList for all controls, and run an UIA search on those.

[Gary-Atlan82]

@Descolada

Amazing, you make it look so easy, that OR operator(?) looks super handy, I will cherish your example and apply it to Photoshop next. Cheers as always!

[Gary-Atlan82]

@Descolada
Retreive all the "name" values of a control type in one query?

I got your solution working!

I am trying to emulate a feature I use all the time in FindText, Ok.[1].id but I cant seem to figure it out in UIA.

Given a illustrator window like the one in the picture, that has three tabs (Layers, Info and Artboards), is it possible to make a single query to UIA like:

Code: Select all

UIA := UIA_Interface()
AI  := UIA.ElementFromHandle(0x4b1b18)   ;Lets say its the panels Window ID
Var := AI.FindAllByName("ControlType=TabItem")

Somewhere in Var, there should be Artboards , Layers and Info. Var can be an array or even one long string.

I have tried to see what is actually inside the AI/Var object in VsCode debugger, its just one deeply nested rabbit hole.

If I can do this it would immensely simplify many tasks I have in mind for UIA

[Descolada]

@Gary-Atlan82, you can do that in almost the same way as in the last example, but instead of FindFirst use FindAll:

Code: Select all

    winEl := UIA.ElementFromHandle("ahk_id" wId)
    foundEls := winEl.FindAll("Name='Layers' OR Name='Info' OR Name='Artboards' AND Type=TabItem")
    if (foundEls.MaxIndex()) {
        WinActivate, ahk_id %wId%
        foundEls[1].Highlight()
        MsgBox, % "Window with winId " wId " contains element " foundEls[1].Dump()
        ;WinSetTitle, ahk_id %wId%,, Layers
    }

[Gary-Atlan82]

@Descolada
Amazing, thank you. Cannot wait to test this as soon as I get home.