UIAutomation with a focus on Chrome

[d_romeo]

I made some personal modifications to UIAViewer.ahk, in this version I added:
- "Test Macro" button that runs the recorded macro
- Macro creator is now the default window
- press esc twice to exit the app: prevents exiting the gui if you press esc twice while recording
- use only window class and exe for recognition, in case there are different titles for windows, as with some text editors
- the print button now does not call the Windows screenshot

Change line 9 with your path to "UIA_Interface.ahk"

image.png (53.51 KiB) Viewed 2333 times

[mora145]

Hi @Descolada

Im trying to find a ListItem using the FindByPath() function, but not work.

image.png (123.97 KiB) Viewed 2172 times

Here is my code:

element := cUIA.FindFirstBy("Name='ds4' AND ControlType=Text")
hola := element.FindByPath("-1", "ControlType=ListItem")


Element pointer is finded correctly but the list not. The debugger error level say: "Step 1 with ""-1"" was out of bounds (GetPreviousSiblingElement failed)"

image.png (7.13 KiB) Viewed 2172 times

[Descolada]

@mora145, I think it's because the ListItem is not a sibling element, but is a parent. Try instead hola := element.FindByPath("p", "ControlType=ListItem")

[mora145]

@mora145, I think it's because the ListItem is not a sibling element, but is a parent. Try instead hola := element.FindByPath("p", "ControlType=ListItem")

I see. So just work with 'brothers' elements.

[chamber_wang]

Hi @Descolada
I appreciate for the powerful UIA library and make all automation flow come true,
However, I am still struggle in pick up (select) a specific list item by it.
May please guide me how to get all available list item and select on of them by name?

I try following, but could only ExpandCollapse this drop down menu and get error message as attached.

ERROR: Method Call not support by the UIA_interface
My AHK version is V1.1.37, Should I upgrade to v2?
Appreciate for your time in advance.

OBS dropdown list error message

Capture.JPG (96.37 KiB) Viewed 1575 times

UIA Tree of drop down list

Capture1.JPG (91.43 KiB) Viewed 1575 times

Code: Select all

	expandCollapsePattern := VcdEl.FindAllByType("ComboBox")[3].GetCurrentPatternAs("ExpandCollapse")
	expandCollapsePattern.Expand()
	sleep 500

	UIA.WaitElementExistByNameAndType("320x240" "ListItem").Click()

[Mod edit: Removed redundant codebox tags inside of code tags.]

[Descolada]

@chamber_wang, WaitElementExist is not an UIA method, but an UIA element method: you probably need to use VcdEl.WaitElementExistByNameAndType("320x240", "ListItem").Click() instead. (By the way, make sure there is a comma between "320x240" and "ListItem").

You don't necessarily need to upgrade to AHK v2, but I recommend it for new scripts or if your script isn't too large either since overall I think v2 is much easier to use. Additionally, I don't actively develop UIA_Interface for AHK v1, so any updates will come to UIA-v2.
For example, the code snippet in UIA-v2 would look something like this:

Code: Select all

VcdEl.FindElement({Type:"Combobox", i:3}).Expand()
Sleep 500
VcdEl.WaitElement({Name:"320x240", Type:"ListItem"}).Click()

[chamber_wang]

Hi @Descolada
Thanks for reply and point my error.
However, after doing so, the script exit without error but still no item be selected.

Any better idea? Please help.

Script exit without error

Capture2.JPG (76.83 KiB) Viewed 1509 times

Click not work any better idea?

Capture3.JPG (216.92 KiB) Viewed 1509 times

[chamber_wang]

My bad not post code here,

Code: Select all

	expandCollapsePattern := VcdEl.FindAllByType("ComboBox")[3].GetCurrentPatternAs("ExpandCollapse")
	expandCollapsePattern.Expand()
	Sleep 100
	VcdEl.WaitElementExistByNameAndType("640x480", "ListItem").Click()

[Descolada]

@chamber_wang, I took a look at OBS and for some reason that list item doesn't respond to Click() nor SelectionItemPattern.Select(). However, ControlClick() appears to work:

Code: Select all

#Requires AutoHotkey v1
#include UIA_Interface.ahk

SetTitleMatchMode 2
WinActivate Properties for
WinWaitActive Properties for
UIA := UIA_Interface()
prop := UIA.ElementFromHandle("A")

cb := prop.FindFirstByNameAndType("Resolution", "Text").FindByPath("+1")
cb.ExpandCollapsePattern.Expand()
sel := cb.WaitElementExistByName("640x360").ControlClick()

A faster and more reliable way is to set the combobox's value directly, because it also has an edit element as a child:

Code: Select all

#Requires AutoHotkey v1
#include UIA_Interface.ahk

SetTitleMatchMode 2
WinActivate Properties for
WinWaitActive Properties for
UIA := UIA_Interface()
prop := UIA.ElementFromHandle("A")

cb := prop.FindFirstByNameAndType("Resolution", "Text").FindByPath("+1", "Type=Edit")
cb.Value := "640x360"

[chamber_wang]

Hi @Descolada

Appreciate for your reply and guide me this solution. I have tried both code work perfectly
Now I am more confident to upgrade AHK to v2 as you mentioned will be more advantages by previous thread.

@chamber_wang, I took a look at OBS and for some reason that list item doesn't respond to Click() nor SelectionItemPattern.Select(). However, ControlClick() appears to work:

[chamber_wang]

Hi @Descolada ,

I suffer a list item does not interact with invoke pattern and workaround it by click button as Figure
I understand this may cause by OBS does not implement Invoke method.
However, this has been an entry barrier for beginners (as me) to get things done.
May please let me know any better insight to trial more efficiently?

I am confused the variety ways to access elements and associated methods.

Path, Control, Elements ... @.@

Code: Select all

UIA := UIA_Interface() ; Initialize UIA interface
WinWaitActive, ahk_exe obs64.exe
obsEl := UIA.ElementFromHandle("ahk_exe obs64.exe") ; Get the element for the Notepad window

WinEl := obsEl.FindFirstBy("Name=Sources")
VideoSrc := WinEl.FindFirstBy("Name=Video Capture Device")	; Find Video Src list item
; Select
SelPattern := VideoSrc.GetCurrentPatternAs("SelectionItemPattern")
sel := SelPattern.Select()

; Wait properties button exist and click, invoke pattern seems not work
Prop := obsEl.FindFirstByNameAndType("Properties", "Button")
PropinvokePattern := Prop.InvokePattern
PropinvokePattern.Invoke()

Instead click Properties button to activate

Capture.PNG (134.81 KiB) Viewed 1416 times

[Descolada]

@chamber_wang, I'm not sure that I understood your question (if there was one?). The code you posted works fine on my computer: it selects Video Capture Device and then clicks Properties. Perhaps you have some kind of timing issue, meaning that after sel := SelPattern.Select() the Properties button may not be active yet before Prop := obsEl.FindFirstByNameAndType("Properties", "Button") and thus Invoke fails? Try adding a small Sleep after selecting.

But yes, UIA is complicated. The Wiki might be of some help, and for UIA-v2 I've also created a small video tutorial series.

[chamber_wang]

Hi @Descolada ,

Thanks for reply. Let me explain more clear my question.
Please refer to following picture.

First, I would like to invoke the yellow blocked element but it does not work by Invoke pattern or Controlclick. (By double click the yellow block directly)
Therefore, I workaround it to click Properties button which marked by cobalt blue arrow.

Question more in detail

Capture.PNG (138.36 KiB) Viewed 1358 times

[Descolada]

@chamber_wang, UIA doesn't have a native built-in method to double-click an element (unfortunately...). But we can use ControlClick to double-click it:

Code: Select all

VideoSrc := WinEl.FindFirstBy("Name=Video Capture Device")	; Find Video Src list item
VideoSrc.ControlClick(,,,2)

[chamber_wang]

Hi @Descolada ,

Thanks for reply, It works.
Now I understand the ControlClick() parameters by this real practice.
And also get better insight by refer to wiki page you mentioned.


https://github.com/Descolada/UIAutomation/wiki/04.-Elements#controlclick

Happy New Year!

[Youhoney]

I made some personal modifications to UIAViewer.ahk, in this version I added:
- "Test Macro" button that runs the recorded macro
- Macro creator is now the default window
- press esc twice to exit the app: prevents exiting the gui if you press esc twice while recording
- use only window class and exe for recognition, in case there are different titles for windows, as with some text editors
- the print button now does not call the Windows screenshot

Change line 9 with your path to "UIA_Interface.ahk"

Thanks, macro testing straight from the viewer is a welcome addition.

[Ralf_Reddings200244]

@Descolada

What are the purposes for the UIA_Browser,UIA_Constants and UIA_Interface .ahk files?

I have been using this library for some time now and I always get tripped up with the script provided in the /lib/ directory on Github. Which one am I supposed to include in my script and which one I am meant to use for purely discovering UI elements, their names etc during development.

This confusion especially becomes pronounced when I come back to programming for UIA after a long hiatus. I skimmed the Wiki, I was not able to get an answer.

Any help would be greatly appreciated!

[Descolada]

@Ralf_Reddings200244 the purposes of the files are described on the GitHub main page:

1. UIA_Interface.ahk, which is based on jethrow's project, and contains wrapper functions for the UIAutomation framework. In addition it has some helper functions for easier use of UIA.
2. UIA_Browser.ahk, which contains helper functions for Chrome (and Edge mostly too) automation (fetching URLs, switching tabs etc).
3. OPTIONAL: UIA_Constants.ahk, which contains most necessary constants to use UIA. It is mostly based on an Autoit3 project by LarsJ. Note that this file creates a lot of global variables, so make sure your script doesn't change any of these during runtime. The preferred option is to use constants/enumerations from the UIA_Interface.ahk UIA_Enum class (details are available in the AHK forum thread).

Meaning the only one you need to include is UIA_Interface.ahk, either with #include <UIA_Interface> if it's in the Lib directory, or #include UIA_Interface.ahk if in the same folder as the script.

Also, I highly recommend switching to AHK v2 and UIA-v2 library instead, because I am not actively supporting UIA_Interface.ahk any more, and UIA-v2 is overall much more pleasant to use.

[Ralf_Reddings200244]

Ahh how did I miss that... forgive me. That answers what I wanted to know. Thank you.

Also, I highly recommend switching to AHK v2 and UIA-v2 library instead, because I am not actively supporting UIA_Interface.ahk any more, and UIA-v2 is overall much more pleasant to use.

I have been following the project and its progression into AHKV2, I am really excited to give it go along with AHK V2, unfortunetly there are a few libraries that still have not transitioned/ported to V2, one that is critical to me and so until some time I am stuck on AHK V1. I am so glad FindText and UIA have made smooth transitions though, soon enough I will make a decision to move forward.


Could I ask one other follow up question please?

I am trying to querry a UIA object saved to a variable, to get one or more window IDs for the trees child windows. I am not sure if this possible, I have tried to do it and was not successful. I am hoping to get some clarification.

Lets say I have a application that has a few child windows, amongst other stuff and I want to save the entire applications and all of its tree structure to a variable:

Code: Select all

WinId		:= WinExist("ahk_exe Illustrator.exe")
UIA		:= UIA_Interface()           
AIL	   := UIA.ElementFromHandle(WinId)

My reasoning for wanting to do this could be to avoid making successive queries to UIA, maybe the application is very expensive to query with UIA and so can only afford to make a single query, or I just want to work with the information at the time of saving the tree to the variable and that may have changed since then.

Is it possible to query this AIL object independent of the current/live state of the Adobe Illustrator tree structure? For example after running the above code, if I closed the Illustrator window, can I still work with AIL using UIA methods?

This may seem very obvious, so pardon me for asking if it is. I tried to do this a few times on my own and did not succeed:

Code: Select all

WinId		:= WinExist("ahk_exe Illustrator.exe")
UIA		:= UIA_Interface()           
AIL		:= UIA.ElementFromHandle(WinId)  
IDs		:= AIL.FindAllByName("ControlType=Pane").GetParentHwnd()

MsgBox,% IDs[1]  ;Nothing is shown

I am thinking FindAllByName("ControlType=Pane") should match a number of child windows that have this windows Pane control type. so I am expecting IDs to have a series of window IDs that correspond to the number of illustrator child windows but in my MsgBox I get nothing.

[Descolada]

I have been following the project and its progression into AHKV2, I am really excited to give it go along with AHK V2, unfortunetly there are a few libraries that still have not transitioned/ported to V2, one that is critical to me and so until some time I am stuck on AHK V1. I am so glad FindText and UIA have made smooth transitions though, soon enough I will make a decision to move forward.

If you don't mind me asking: what is the library that isn't ported over?

I am trying to querry a UIA object saved to a variable, to get one or more window IDs for the trees child windows. I am not sure if this possible, I have tried to do it and was not successful. I am hoping to get some clarification.

If an element that an UIA object represents is destroyed (for example it usually happens if a child window is closed) then the UIA object is invalid and won't respond to queries any more, including querying for window ID.

Lets say I have a application that has a few child windows, amongst other stuff and I want to save the entire applications and all of its tree structure to a variable:

What do you mean by "saving the entire application and all of its tree structure"? You can query for multiple elements at a time with FindAllBy, and/or you can cache properties via caching. I can't say whether you can prefetch the AIL object beforehand and interact with it even if it's closed, because it depends on how Illustrator handles its UIA implementations. Usually the UIA object would be invalid, but in some applications I've seen that the object remains valid for some time. You'd need to experiment with it and find out.

This may seem very obvious, so pardon me for asking if it is. I tried to do this a few times on my own and did not succeed:

Code: Select all

WinId		:= WinExist("ahk_exe Illustrator.exe")
UIA		:= UIA_Interface()           
AIL		:= UIA.ElementFromHandle(WinId)  
IDs		:= AIL.FindAllByName("ControlType=Pane").GetParentHwnd()

MsgBox,% IDs[1]  ;Nothing is shown

First of all you are trying to use GetParentHwnd on an array which will not work. Second, GetParentHwnd returns the hWnd for the main Illustrator window, not the controls/subwindows (so it should be the same as the WinId variable in your code). If the Pane elements are implemented as win32 controls (meaning inspection with WindowSpy shows control names) then you can get the control hWnd by normalizing to the NativeWindowHandle property, something like UIA.CreateTreeWalker(UIA.CreateNotCondition(UIA.CreatePropertyCondition(UIA_Enum.UIA_PropertyId("NativeWindowHandle"), 0))).NormalizeElement(Element).GetCurrentPropertyValue(UIA_Enum.UIA_PropertyId("NativeWindowHandle"))
How useful that is depends on what you are planning to do with the panes/subwindows...