UIA v2

[Descolada]

This is an UIAutomation library for AHK v2. Credit to thqby's work who's library I based mine on.

The library is available here: https://github.com/Descolada/UIA-v2

Learn more about UIAutomation from the Wiki

Videos: Tutorial Series, Joe the Automator, Axlefublr

Quick Start Guide

1. Run UIA.ahk as a stand-alone script to display UIAViewer. Press the "Start capturing" button to start inspecting elements: then you can see element properties, patterns (ways to interact), the UIA tree (if you hold the mouse still for a bit), and the UIA path for elements.
2. Include UIA.ahk in your script to use it as a library. UIA doesn't have to be initialized in any way.
3. Get a starting point element with UIA.ElementFromHandle(WinTitle), UIA.ElementFromPoint(X, Y), or other methods described in the Wiki.
4. Traverse to the element of interest with the UIA path using Element[path] such as Element[{T:10}, {T0,N:"Save"}], or Element.FindElement(condition). Condition is an object determining what properties need to be filled: for example {Name:"Save"} requires the Name property to be "Save". Read more about Conditions in the Wiki: multiple conditions can be combined with "and" or "or" conditions, for negation there is the "not" condition.
5. Get property values displayed in UIAViewer with Element.PropertyName. Some properties such as Value can sometimes be assigned with Element.Value := "newvalue"
6. Use pattern methods with Element.Method() such as the Toggle method from the TogglePattern: Element.Toggle(); or use it through the pattern object itself: Element.TogglePattern.Toggle. For clicking an element there is the method Element.Click().
7. Some methods require special values (constants, or otherwise known as enumerations), which can be accessed with UIA.EnumerationClass.EnumerationName such as UIA.Type.Button. This you can use whenever a method requires a constant, or a property returns one: Element.Type returns such a value, which you can either compare with Element.Type == UIA.Type.Button or get the enumeration name with UIA.Type[Element.Type] => "Button". If a method requires an enumeration, then it can usually be used only by the name: Element.FindElement(condition, "Children") which uses the value of UIA.TreeScope.Children.
8. Debug with Element.Highlight() which highlights the element, or Element.Dump() or Element.DumpAll() which creates a string containing some properties of the element (display it with MsgBox for example).


Main changes from UIA_Interface.ahk

1) UIA doesn't need to be initialized, just start using it
2) The main Find methods are FindElement and FindElements. Errors are thrown if an element is not found.
3) Conditions have been replaced by condition object notation such as {Name:"Something"}. {} creates and-condition, [] creates or-condition, index, matchmode and casesense keys can be set.
4) Elements can accessed with array notation: Element[2,1,4...] or Element[condition1, condition2...]; alternatively ElementFromPath can be used
5) Arguments have been switched around for many methods, such as for cache-related methods the cacheRequest argument is now the first
6) UIA constants/enumerations are included in the main UIA object. For example UIA.Type.Button == 50000
7) UIAViewer is built into the library: when ran directly the UIAViewer will show, when included in another script then it won't show (but can be opened by calling UIA.Viewer())
8) UIAViewer creates a "UIA path" that can be used with the array notation to get elements
Read from the Wiki about details for all of these, also take a look at the included examples.

If you have any suggestions or comments about what should be changed or improved, please leave a comment. Also feel free to create Pull requests in GitHub.

[AHK_user]

Great work!

The lack of V2 was actually holding me back from using UIA.

It would be cool to also include into the viewer some test possibilities to for example test clicking on a specific element, and a button to copy the code for this.

[Descolada]

@AHK_user, as per your request I added a small macro creation tool to UIAViewer. Currently it supports only a few actions, but that should be enough for debugging purposes.

[AHK_user]

@AHK_user, as per your request I added a small macro creation tool to UIAViewer. Currently it supports only a few actions, but that should be enough for debugging purposes.

@Descolada: Thanks a lot
Amazing (and quick) work!

This makes it easier to quickly generate some usefull code examples that get people started with this library.

[neogna2]

Exciting! I like how your v2 UIA and ACC libs now have more similar naming and format. Makes it easier to use both.

I worked through the examples, very helpful!

Example09 is currently implicitly dependent on Explorer being set to always show the ribbon (Ctrl+F1 or little grey arrow next to right corner questionmark icon). If not the script errors because both MsgBox steps and toggle steps autohides the not-always-shown ribbon.

Several of the Explorer example scripts depend on the Explorer opening with enough width to show the menu and ribbon elements acted on. That's not guaranteed if the Explorer window is set to reuse size and position from previous instances. Adding WinMove(100, 200, 1000, , "A") after the WinWaitActive step would ensure enough width.

Glancing through the lib I noticed that L1625 comment
; Get the parent window hwnd from the element
should be something like
; Get the control hwnd from the element

Here's a suggestion for an extra example to illustrate that elements need to be in view to be accessible

Code: Select all

;#include <UIA> ; Uncomment if you have moved UIA.ahk to your main Lib folder
#include ..\Lib\UIA.ahk

Run "explorer C:\Windows"
if !explorerHwnd := WinWaitActive("C:\Windows",,1)
    ExitApp
; Decrease window height so that folder "System32" is out of view
WinMove( , , 400, 400, explorerHwnd)
explorerEl := UIA.ElementFromHandle("A")
if !explorerEl
    ExitApp
listEl := explorerEl.FindElement({Type:"List"})
if !listItem := explorerEl.FindElement({Name:"System32", Type:"ListItem"})
{
    if "OK" != MsgBox("Press OK to scroll until 'System32' is in view and then select it.")
        ExitApp
    Loop
    {
        if listItem := explorerEl.FindElement({Name:"System32", Type:"ListItem"})
            break
        listEl.Scroll(, UIA.ScrollAmount.LargeIncrement)
        if Round(listEl.VerticalScrollPercent) = 100
            break
    }
    if listItem
          listItem.AddToSelection()
}
ExitApp

F5:: ExitApp

[Descolada]

@neogna2, thanks for the suggestions, I modified by previous examples and added your proposed example (though I changed it a bit).

I decided to make a breaking change as well: ScrollPattern.Scroll and ScrollPattern.SetScrollPercent now take the vertical scroll percent as the first argument. I believe that horizontal scrolling is rare (if not non-existant?), so it makes sense for vertical to be first...

[Descolada]

Another major-ish update, and some more breaking changes:
1) FindByPath has been replaced by ElementFromPath, which is a more descriptive name for it (since it's not actually doing any "finding", rather traveling down the tree by path).
2) FindByPath previously combined both moving "downwards" in the UIA tree, as well as using TreeWalkers to travel "laterally" and "upwards" to parents. This led to some confusing behaviours, because TreeWalkers behave differently that Find methods. This is why it's now split into ElementFromPath which goes only downwards, and WalkTree which uses TreeWalkers to travel step by step.
3) UIAViewer got added an "UIA path", which is a compressed version of a path consisting of Type and index information. This can be used in the format of Element["abc"] or Element.ElementFromPath("abc"). Right-clicking the UIAViewer statusbar now shows a menu where one can choose between the "UIA path", integer path, or a condition path (which is most robust to UI changes, yet can get very long). Additionally UIAViewer now supports testing out pattern methods by double-clicking them in the patterns listview.

[ImFeyzBtw]

Amazing work as always

[Descolada]

Since I think there aren't any more breaking changes coming, I have moved the library from alpha to beta stage. This means that mostly there will be bug fixes and minor changes, although if somebody proposes a very good idea that would be a breaking change, I will still consider it.

Notable changes since my last post:
1) FindElement throws an error if no matching element is found. ElementExist can be used to check if an element exist (the found element is returned if yes).
2) Cache-related methods have been added: FindCachedElement, WalkCachedTree
3) Better error messages, and most methods will throw errors if no match is found (eg. TreeWalker.GetNextSiblingElement).

[wpb]

I'm very grateful for this library, as I'm migrating to AHKv2, and I use UIA a lot. So thank you!

When trying to get the "LabeledBy" property of an element, I got this error from the library:

Error: Invalid IUnknown interface pointer

Specifically: UIA.IUIAutomationElement

---- ...\AutoHotkey\Lib\v2\UIA.ahk
2624: }
2631: {
▶ 2633: Return (ComCall(44, this, "ptr*", &retVal := 0), UIA.IUIAutomationElement(retVal))
2634: }
2639: {

The current thread will exit.

I suspect it's because the element didn't have a "LabeledBy" property, but I imagine the library should just return null (false) in that case?

Is this a bug perhaps?

[Descolada]

@wpb, it does most likely mean that it just doesn't have "LabeledBy" property. The error is deliberate, because it causes the same behavior as other AHK objects. For example

Code: Select all

test := {prop:"hello"}
MsgBox test.LabeledBy

will throw an error, because the object doesn't contain a property with the key "LabeledBy", and it doesn't simply return null or false. This allows you to differentiate whether you have an object without the key "LabeledBy", or it's the case of {prop:"hello", LabeledBy:0}

Unfortunately the error message is rather nondescript as it doesn't say anything about the property missing. I will look into providing better error messages when that error is thrown, since it usually means that either the property is undefined, or the interface doesn't support it (e.g. the user is running an older Windows where it hasn't been added yet).

[wpb]

@Descolada, thanks for the response. All understood. I'll wrap it in a try block! Keep up the excellent work on this library.

[fenchai]

So I was testing and so far it works pretty well!

I have had issues with:

ComCall(4, this)

when trying to .Click() it seems the path cannot be found? but weirdly enough, I am using the test tool from UIAViewer which should contain the correct path?
any solution to this issue? it only happened on 1 specific element, I am testing it with the UIAViewer.

[Descolada]

@fenchai, thanks for the input
The ComCall(4, this) could be many different methods, so with this info I can't say what's going wrong. Could you provide the error message with the stack trace? And if possible, provide the program you are trying to get working along with the path that is giving the error.

[fenchai]

@Descolada, this is what I am doing:

- I am running edge in private mode.
- this is the code I am running in the test zone in UIAViewer:

Code: Select all

#include UIA.ahk

msedgeEl := UIA.ElementFromHandle("Some Website - [InPrivate] - Microsoft​ Edge ahk_exe msedge.exe")
msedgeEl.ElementFromPath("Y/YYY/YqYYYYVRR/RqRRR/RRRqRK").Click()

Code: Select all

this is the error I am getting:
	Error: (0x80131509) 

	---- just_some_path_\lib\UIA.ahk
	5124: }
	5127: {
▶	5127: Return ComCall(4, this)
	5127: }
	5130: {

The current thread will exit.

Call stack:
just_some_path_\lib\UIA.ahk (5127) : [ComCall] Return ComCall(4, this)
just_some_path_\lib\UIA.ahk (5127) : [UIA.IUIAutomationLegacyIAccessiblePattern.Prototype.DoDefaultAction] Return ComCall(4, this)
just_some_path_\lib\UIA.ahk (2235) : [UIA.IUIAutomationElement.Prototype.Click] this.LegacyIAccessiblePattern.DoDefaultAction()
just_some_path_\lib\~UIAViewerMacro.tmp (4) : [] msedgeEl.ElementFromPath("Y/YYY/YqYYYYVRR/RqRRR/RRRqRK").Click()
> Auto-execute

for whatever reason, I do the same on Chrome and it works (path changes for obvious reasons) but no errors and it clicks properly.

[Descolada]

@fenchai, that error most likely means that no "click-like" method (InvokePattern.Invoke(), TogglePattern.Toggle(), ExpandCollapsePattern.Expand(), SelectionItemPattern.Select(), LegacyIAccessiblePattern.DoDefaultAction()) was found for the element. Are you sure it's actually clickable? You can test it out by inspecting with UIAViewer and taking a look at the available patterns. If it has Invoke for example, then you can test it by double-clicking Invoke(). If there are no actionable patterns, then perhaps you are trying to interact with a wrong element. For example, sometimes a Button-type element has the description text as a Text-type element, and the Text might not be clickable whereas the Button would be.

[fenchai]

@Descolada, for some weird reason...

- edge has no invoke methods, only LegacyIAccessible, ScrollItem,TextChild,TextEdit and Text. does it means it cannot do click actions? then why does it show up on the Macro creator Action menu? mmm maybe it's best to disable it?

- chrome shows Invoke, LegacyIAccessible and ScrollItem, it is able to click() while edge cannot.

For this reason, I sadly have to use Chrome.

[Descolada]

@fenchai, if LegacyIAccessible is available and the element is clickable with the mouse, then DoDefaultAction() usually works (but here it's throwing an error). Without testing the website and the element myself, I can't comment much more on it.

If you still decide you want to use Edge, then you could use the non-UIA ways of clicking: Element.ControlClick() to send a ControlClick to the elements location, or Element.Click("left") which moves the mouse to the element and left-clicks.

[elbitjusticiero]

Hi! I'm trying to interact with a toast notification from Skype, and from what I could gather searching the forums, UIA would be the current best way to achieve it. I installed the library and tried to make it work, but I can't quite grasp how to do it.

What I want to do is intercept the notification when it appears, and move it to the opposite side of the screen, so that it doesn't get in the way.

Is it possible with UIA? I'm not sure whether I can get an id I can use with WinMove() or I should use a UIA method for moving the window. I've read the documentation, but I'm not sure what piece of information given by UIAViewer I should put where, haha. Whatever I tried hasn't worked.

[LAPIII]

I don't think UIA can help you here, but you should first try with WinMove. When the Skype window appears can you bring up Window Spy, it's in C:\Program Files\AutoHotkey?. I'd like to know how to solve this too.