The library is available here: https://github.com/Descolada/UIAutomation
Learn more about UIAutomation from the Wiki
Instructional videos from Joe the Automator
It consists of two (three?) main files:
1) UIA_Interface.ahk, which is based on jethrow's project, and contains wrapper functions for the UIAutomation framework. In addition it has some helper functions for easier use of UIA.
2) UIA_Browser.ahk, which contains helper functions for Chrome (and Edge mostly too) automation (fetching URLs, switching tabs etc).
3) OPTIONAL UIA_Constants.ahk, which contains constants to use UIA. All the constants are also available in UIA_Interface.ahk, but the format of constants in UIA_Constants.ahk might be preferrable to some people. Note that this file creates a lot of global variables, so make sure your script doesn't change any of these during runtime. Credit: UIA_Constants file is mostly based on an Autoit3 project by LarsJ.
Short intro for UIAutomation:
https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-overview
UIAutomation is a Microsoft accessibility framework that allows interacting with user-accessible elements (clicking buttons, sending text etc) in a programmatic way. It is the successor to the Acc/MSAA framework.
UIAutomation framework holds the entire desktop with all the windows and controls inside a tree of AutomationElement objects. The root element is the Desktop itself, its children are all the windows (Notepad, Chrome, etc), then in turn the windows have child elements (toolbars, main content, buttons, etc) and so on.
To interact with windows and controls, usually we need to first get the AutomationElement to that window/control. This can be done with functions from UIA_Interface.ahk UIA_Interface class, such as getting a window element from an ahk_id handle with ElementFromHandle, or from a point with ElementFromPoint. We can also get the root element (Desktop) with GetRootElement.
Once an element is found, one way that sub-elements can be looked for is using FindFirst or FindAll functions, that take a specific condition (such as the control type needs to be "edit", or name must be "something") and return the first matching element (FindFirst) or all matching elements (FindAll). Conditions can be created by using Create...Condition methods (CreatePropertyCondition, CreateTrueCondition, etc) and combined with using CreateAndCondition, CreateOrCondition. One of the more commonly used condition method is CreatePropertyCondition, which takes a property id (these can be found under UIA_Enum class under PropertyIds, or in UIA_Constants.ahk file under UIA_PropertyIds) and the desired property value. For example, to create a condition that only looks for button elements, first look up the corresponding PropertyId in the UIA_Enum class ("button" is a control type, so it would be UIA_ControlTypePropertyId), then under UIA_ControlTypeIds to find the value for button (UIA_ButtonControlTypeId).
Alternatively the tree can be walked with the TreeWalker class. This can be done with any element (for example after ElementFromHandle or FindFirst methods) and use the TreeWalker class methods such as GetFirstChildElement or GetNextSiblingElement. TreeWalkers can be created with conditions as well to only return certain elements filtered by type, name etc.
Once an element is found, some basic info is accessible with properties such as CurrentName (returns the name of the element), CurrentControlType (returns the type of the element), CurrentIsEnabled etc. These can be found in UIA_Interface.ahk in the UIA_Element class (properties end with []).
Elements can be interacted with using control patterns. Patterns contain the actual methods to interact with the element, such as InvokePattern to Invoke (click) a button, or ScrollPattern to scroll a control, ValuePattern to get or set the value (such as an edit box). Patterns can be fetched by calling the GetCurrentPatternAs method, then using the methods available in the pattern (for example after calling GetCurrentPatternAs("Invoke") to get a UIA_InvokePattern object, this would be object.Invoke()).
Elements and the element tree can be inspected with various tools. A good tool to get started with is Accessibility Insights which also lets you test out methods of patterns (such as SetValue for ValuePattern). I have also written a AccViewer analog for UIA, the UIAViewer, which can be ran without the UIA_Interface library:
(due to AHK forum constraints this is an older version, please get the latest one from GitHub)
Code: Select all
#SingleInstance Force
#NoEnv
SetWorkingDir %A_ScriptDir%
SetBatchLines -1
CoordMode, Mouse, Screen
global DeepSearchFromPoint := False ; When set to True, UIAViewer iterates through the whole UIA tree to find the smallest element from mouse point. This might be very slow with large trees.
global UIA := UIA_Interface(), IsCapturing := False, Stored := {}, Acc, EnableAccTree := False
Stored.TreeView := {}
Acc_Init()
Acc_Error(1)
_xoffsetfirst := 8
_xoffset := 5
_yoffset := 20
_ysoffset := 2
Gui Main: New, AlwaysOnTop Resize, UIAViewer
Gui Main: Default
Gui Add, GroupBox, x8 y10 w302 h160, Window/Control Info
Gui Add, Text, xm+%_xoffsetfirst% yp+%_yoffset% w30 Section, WinTitle:
Gui Add, Text,, Text:
Gui Add, Edit, ys-%_ysoffset% w235 vEditWinTitle,
Gui Add, Edit, w235 vEditWinText,
Gui Add, Text, x18 yp+30 Section, Hwnd:
Gui Add, Text,, Position:
Gui Add, Text,, Size:
Gui Add, Edit, ys-%_ysoffset% w80 vEditWinHwnd,
Gui Add, Edit, w80 vEditWinPosition,
Gui Add, Edit, w80 vEditWinSize,
Gui Add, Text, ys, ClassNN:
Gui Add, Text,, Process:
Gui Add, Text,, Process ID:
Gui Add, Edit, ys-%_ysoffset% w80 vEditWinClass,
Gui Add, Edit, w80 vEditWinProcess,
Gui Add, Edit, w80 vEditWinProcessID,
Gui Add, GroupBox, x%_xoffsetfirst% y180 w302 h295, UIAutomation Element Info
Gui Add, Text, xm+%_xoffsetfirst% yp+%_yoffset%, ControlType:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w216 vEditControlType,
Gui Add, Text, xm+%_xoffsetfirst% yp+30 Section, Name:
Gui Add, Text,, Value:
Gui Add, Text,, Patterns:
Gui Add, Edit, ys-%_ysoffset% w228 vEditName,
Gui Add, Edit, w228 vEditValue,
Gui Add, Edit, w228 vEditPatterns,
Gui Add, Text, xm+%_xoffsetfirst% yp+30, AutomationId:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w208 vEditAutomationId,
Gui Add, Text, xm+%_xoffsetfirst% yp+30 Section, BoundingRectangle:
Gui Add, Edit, ys-%_ysoffset% w174 vEditBoundingRectangle,
Gui Add, Text, xm+%_xoffsetfirst% yp+30, ClassName:
Gui Add, Edit, x+3 yp-%_ysoffset% w68 vEditClassName,
Gui Add, Text, x+%_xoffset% yp+%_ysoffset%, HelpText:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w97 vEditHelpText,
Gui Add, Text, xm+%_xoffsetfirst% yp+30, AccessKey:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w68 vEditAccessKey,
Gui Add, Text, x+%_xoffset% yp+%_ysoffset%, AcceleratorKey:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w68 vEditAcceleratorKey,
Gui Add, CheckBox, x18 yp+30 Section +Disabled vCBIsKeyboardFocusable, IsKeyboardFocusable
Gui Add, CheckBox, ys +Disabled vCBIsEnabled, IsEnabled
Gui Add, Button, xm+60 yp+30 w150 gButCapture vButCapture, Start capturing
Gui Add, Button, xp+300 yp w192 vButRefreshTreeView gButRefreshTreeView +Disabled, Start capturing to show tree
Gui Add, TreeView, x320 y8 w300 h435 hwndhMainTreeView vMainTreeView gMainTreeView
Gui, Font, Bold
Gui, Add, StatusBar, gMainSB vMainSB
SB_SetText("`tClick here to enable Acc path capturing (can't be used with UIA!)")
Gui, Font
SB_SetParts(380)
SB_SetText("`tCurrent UIA Interface version: " UIA.__Version,2)
Gui Show,, UIAViewer
Return
MainGuiEscape:
MainGuiClose:
IsCapturing := False
ExitApp
MainGuiSize(GuiHwnd, EventInfo, Width, Height){
GuiControlGet, Pos, Pos , MainTreeView
GuiControl, Move, MainTreeView, % "w" Width -Posx-10 " h" Height -Posy-60
GuiControl, Move, ButRefreshTreeView, % "y" Height -50
WinSet, Redraw,,UIAViewer
}
MainSB:
GuiControlGet, SBText,, MainSB
if (SBText == "`tClick here to enable Acc path capturing (can't be used with UIA!)") {
EnableAccTree := True
SB_SetText("",1)
SB_SetText("`tClick on path to copy to Clipboard",2)
} else if SBText {
Clipboard := SubStr(SBText, 8)
ToolTip, % "Successfully copied """ SubStr(SBText, 8) """ to Clipboard!"
SetTimer, RemoveToolTip, -2000
}
return
ButCapture:
if IsCapturing {
RangeTip()
IsCapturing := False
GuiControl, Main:, ButCapture, Start capturing
GuiControl, Main: Enable, ButCapture
GuiControl, Main: Enable, ButRefreshTreeView
GuiControl, Main:, ButRefreshTreeView, Construct tree for whole Window
} else {
IsCapturing := True
GuiControl, Main:, ButCapture, Press Esc to stop capturing
GuiControl, Main: Disable, ButCapture
GuiControl, Main: Disable, ButRefreshTreeView
GuiControl, Main:, ButRefreshTreeView, Hold cursor still to construct tree
Stored := {}
While (IsCapturing) {
MouseGetPos, mX, mY, mHwnd, mCtrl
try {
mEl := UIA.SmallestElementFromPoint(mX, mY,, DeepSearchFromPoint ? UIA.ElementFromHandle(mHwnd) : "")
} catch e {
UpdateElementFields()
GuiControl, Main:, EditName, % "ERROR: " e.Message
if InStr(e.Message, "0x80070005")
GuiControl, Main:, EditValue, Try running UIAViewer with Admin privileges
}
if (mHwnd != Stored.Hwnd) {
; In some setups Chromium-based renderers don't react to UIA calls by enabling accessibility, so we need to send the WM_GETOBJECT message to the first renderer control for the application to enable accessibility. Thanks to users malcev and rommmcek for this tip. Explanation why this works: https://www.chromium.org/developers/design-documents/accessibility/#TOC-How-Chrome-detects-the-presence-of-Assistive-Technology
WinGet, cList, ControlList, ahk_id %mHwnd%
if InStr(cList, "Chrome_RenderWidgetHostHWND1")
SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Chrome_RenderWidgetHostHWND1, ahk_id %mHwnd%
WinGetTitle, wTitle, ahk_id %mHwnd%
WinGetPos, wX, wY, wW, wH, ahk_id %mHwnd%
WinGetClass, wClass, ahk_id %mHwnd%
WinGetText, wText, ahk_id %mHwnd%
WinGet, wProc, ProcessName, ahk_id %mHwnd%
WinGet, wProcID, PID, ahk_id %mHwnd%
GuiControl, Main:, EditWinTitle, %wTitle%
GuiControl, Main:, EditWinText, %wText%
GuiControl, Main:, EditWinHwnd, ahk_id %mHwnd%
GuiControl, Main:, EditWinPosition, X: %wX% Y: %wY%
GuiControl, Main:, EditWinSize, W: %wW% H: %wH%
GuiControl, Main:, EditWinClass, %wClass%
GuiControl, Main:, EditWinProcess, %wProc%
GuiControl, Main:, EditWinProcessID, %wProcID%
}
if IsObject(Stored.Element) {
try {
if !UIA.CompareElements(mEl, Stored.Element) {
UpdateElementFields(mEl)
Stored.TickCount := A_TickCount
if EnableAccTree {
oAcc := Acc_ObjectFromPoint(childId, mX, mY), Acc_Location(oAcc,childId,vAccLoc)
SB_SetText(" Path: " GetAccPathTopDown(mHwnd, vAccLoc), 1)
}
} else if (Stored.TickCount && (A_TickCount - Stored.TickCount > 1000)) { ; Wait for mouse to be stable for a second
Stored.TickCount := 0
RedrawTreeView(mEl, False)
for k,v in Stored.TreeView {
if (v == mEl)
TV_Modify(k)
}
}
}
}
Stored.Hwnd := mHwnd, Stored.Element := mEl
Sleep, 200
}
}
return
ButRefreshTreeView:
if (Stored.Hwnd && WinExist("ahk_id" Stored.Hwnd))
RedrawTreeView(UIA.ElementFromHandle(Stored.Hwnd), True)
if IsObject(Stored.Element) {
for k,v in Stored.TreeView
if UIA.CompareElements(v, Stored.Element)
TV_Modify(k)
if EnableAccTree {
br := Stored.Element.CurrentBoundingRectangle
GetAccPathTopDown(Stored.Hwnd, "x" br.l " y" br.t " w" (br.r-br.l) " h" (br.b-br.t), True)
}
}
return
MainTreeView:
if (A_GuiEvent == "S") {
UpdateElementFields(Stored.Treeview[A_EventInfo])
if EnableAccTree {
br := Stored.Treeview[A_EventInfo].CurrentBoundingRectangle
SB_SetText(" Path: " GetAccPathTopDown(Stored.Hwnd, "x" br.l " y" br.t " w" (br.r-br.l) " h" (br.b-br.t)), 1)
}
}
return
RemoveToolTip:
ToolTip
return
UpdateElementFields(mEl="") {
if !IsObject(mEl) {
GuiControl, Main:, EditControlType,
GuiControl, Main:, EditName,
GuiControl, Main:, EditValue,
GuiControl, Main:, EditPatterns,
GuiControl, Main:, EditAutomationId,
GuiControl, Main:, EditBoundingRectangle,
GuiControl, Main:, EditAccessKey,
GuiControl, Main:, EditAcceleratorKey,
GuiControl, Main:, EditClassName,
GuiControl, Main:, EditHelpText,
GuiControl, Main:, CBIsKeyboardFocusable,
GuiControl, Main:, CBIsEnabled,
return
}
try {
mElPos := mEl.CurrentBoundingRectangle
RangeTip(mElPos.l, mElPos.t, mElPos.r-mElPos.l, mElPos.b-mElPos.t, "Blue", 4)
GuiControl, Main:, EditControlType, % (ctrlType := mEl.CurrentControlType) " (" UIA_ControlTypeId(ctrlType) ")`t[Localized: " mEl.CurrentLocalizedControlType "]"
GuiControl, Main:, EditName, % mEl.CurrentName
GuiControl, Main:, EditValue, % mEl.GetCurrentPropertyValue(UIA_ValueValuePropertyId := 30045)
patterns := ""
for k, v in UIA.PollForPotentialSupportedPatterns(mEl)
patterns .= ", " RegexReplace(k, "Pattern$")
GuiControl, Main:, EditPatterns, % SubStr(patterns, 3)
GuiControl, Main:, EditAutomationId, % mEl.CurrentAutomationId
GuiControl, Main:, EditBoundingRectangle, % "l: " mElPos.l " t: " mElPos.t " r: " mElPos.r " b: " mElPos.b
GuiControl, Main:, EditAccessKey, % mEl.CurrentAccessKey
GuiControl, Main:, EditAcceleratorKey, % mEl.CurrentAcceleratorKey
GuiControl, Main:, EditClassName, % mEl.CurrentClassName
GuiControl, Main:, EditHelpText, % mEl.CurrentHelpText
GuiControl, Main:, CBIsKeyboardFocusable, % mEl.CurrentIsKeyboardFocusable
GuiControl, Main:, CBIsEnabled, % mEl.CurrentIsEnabled
}
return
}
RedrawTreeView(el, noAncestors=True) {
global MainTreeView, hMainTreeView
TV_Delete()
TV_Add("Constructing TreeView, do not move the mouse...")
GuiControl, Main: -Redraw, MainTreeView
Gui, TreeView, MainTreeView
TV_Delete()
Stored.TreeView := {}
if noAncestors {
ConstructTreeView(el)
} else {
; Get all ancestors
ancestors := [], parent := el
while IsObject(parent) {
try {
if IsObject(parent := UIA.TreeWalkerTrue.GetParentElement(parent))
ancestors.Push(parent)
} catch {
break
}
}
; Loop backwards through ancestors to create the TreeView
maxInd := ancestors.MaxIndex(), parent := ""
while (--maxInd) {
if !IsObject(ancestors[maxInd])
return
try {
elDesc := ancestors[maxInd].CurrentLocalizedControlType " """ ancestors[maxInd].CurrentName """"
if (elDesc == " """"")
break
Stored.TreeView[parent := TV_Add(elDesc, parent)] := ancestors[maxInd]
}
}
; Add child elements to TreeView also
ConstructTreeView(el, parent)
}
for k,v in Stored.TreeView
TV_Modify(k, "Expand")
SendMessage, 0x115, 6, 0,, ahk_id %hMainTreeView% ; scroll to top
GuiControl, Main: +Redraw, MainTreeView
}
ConstructTreeView(el, parent="") {
if !IsObject(el)
return
try {
elDesc := el.CurrentLocalizedControlType " """ el.CurrentName """"
if (elDesc == " """"")
return
Stored.TreeView[TWEl := TV_Add(elDesc, parent)] := el
if !(children := el.FindAll(UIA.TrueCondition, 0x2))
return
for k, v in children
ConstructTreeView(v, TWEl)
}
}
; UIA FUNCTIONS
class UIA_Base {
__New(p="", flag=1, ver=1) {
ObjInsert(this,"__Type","IUIAutomation" SubStr(thisClass,5))
,ObjInsert(this,"__Value",p)
,ObjInsert(this,"__Flag",flag)
,ObjInsert(this,"__Version",ver)
}
__Get(member) {
if member not in base,__UIA,TreeWalkerTrue,TrueCondition ; base & __UIA should act as normal
{ if raw:=SubStr(member,0)="*" ; return raw data - user should know what they are doing
member:=SubStr(member,1,-1)
if RegExMatch(this.__properties, "im)^" member ",(\d+),(\w+)", m) { ; if the member is in the properties. if not - give error message
if (m2="VARIANT") ; return VARIANT data - DllCall output param different
return UIA_Hr(DllCall(this.__Vt(m1), "ptr",this.__Value, "ptr",UIA_Variant(out)))? (raw?out:UIA_VariantData(out)):
else if (m2="RECT") ; return RECT struct - DllCall output param different
return UIA_Hr(DllCall(this.__Vt(m1), "ptr",this.__Value, "ptr",&(rect,VarSetCapacity(rect,16))))? (raw?out:UIA_RectToObject(rect)):
else if UIA_Hr(DllCall(this.__Vt(m1), "ptr",this.__Value, "ptr*",out))
return raw?out:m2="BSTR"?StrGet(out):RegExMatch(m2,"i)IUIAutomation\K\w+",n)?new UIA_%n%(out):out ; Bool, int, DWORD, HWND, CONTROLTYPEID, OrientationType?
}
else throw Exception("Property not supported by the " this.__Class " Class.",-1,member)
}
}
__Delete() {
this.__Flag ? ObjRelease(this.__Value):
}
__Vt(n) {
return NumGet(NumGet(this.__Value+0,"ptr")+n*A_PtrSize,"ptr")
}
}
class UIA_Interface extends UIA_Base {
;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671406(v=vs.85).aspx
static __IID := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}"
, __properties := ""
CompareElements(e1,e2) {
return UIA_Hr(DllCall(this.__Vt(3), "ptr",this.__Value, "ptr",e1.__Value, "ptr",e2.__Value, "int*",out))? out:
}
ElementFromHandle(hwnd) {
return UIA_Hr(DllCall(this.__Vt(6), "ptr",this.__Value, "ptr",hwnd, "ptr*",out))? new UIA_Element(out):
}
ElementFromPoint(x="", y="", activateChromiumAccessibility=False) {
return UIA_Hr(DllCall(this.__Vt(7), "ptr",this.__Value, "UInt64",x==""||y==""?DllCall("GetCursorPos","Int64*",pt)*0+pt:x&0xFFFFFFFF|(y&0xFFFFFFFF)<<32, "ptr*",out))? new UIA_Element(out):
}
CreateTreeWalker(condition) {
return UIA_Hr(DllCall(this.__Vt(13), "ptr",this.__Value, "ptr",Condition.__Value, "ptr*",out))? new UIA_TreeWalker(out):
}
CreateTrueCondition() {
return UIA_Hr(DllCall(this.__Vt(21), "ptr",this.__Value, "ptr*",out))? new UIA_BoolCondition(out):
}
CreatePropertyCondition(propertyId, var, type="Variant") {
if (type!="Variant")
UIA_Variant(var,type,var)
return UIA_Hr((A_PtrSize == 4) ? DllCall(this.__Vt(23), "ptr",this.__Value, "int",propertyId, "int64", NumGet(var, 0, "int64"), "int64", NumGet(var, 8, "int64"), "ptr*",out) : DllCall(this.__Vt(23), "ptr",this.__Value, "int",propertyId, "ptr",&var, "ptr*",out))? new UIA_PropertyCondition(out):
}
PollForPotentialSupportedPatterns(e, Byref Ids="", Byref Names="") { ; Returns an object where keys are the names and values are the Ids
return UIA_Hr(DllCall(this.__Vt(51), "ptr",this.__Value, "ptr",e.__Value, "ptr*",Ids, "ptr*",Names))? UIA_SafeArraysToObject(Names:=ComObj(0x2008,Names,1),Ids:=ComObj(0x2003,Ids,1)):
}
; Gets ElementFromPoint and filters out the smallest subelement that is under the specified point. If windowEl (window under the point) is provided, then a deep search is performed for the smallest element (this might be very slow in large trees).
SmallestElementFromPoint(x="", y="", activateChromiumAccessibility=False, windowEl="") {
if IsObject(windowEl) {
element := this.ElementFromPoint(x, y, activateChromiumAccessibility)
bound := element.CurrentBoundingRectangle, elementSize := (bound.r-bound.l)*(bound.b-bound.t), prevElementSize := 0, stack := [windowEl]
Loop
{
bound := stack[1].CurrentBoundingRectangle
if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b)) { ; If parent is not in bounds, then children arent either
if ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize)
element := stack[1], elementSize := newSize
for _, childEl in stack[1].FindAll(this.__UIA.TrueCondition, 0x2) {
bound := childEl.CurrentBoundingRectangle
if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b)) {
stack.Push(childEl)
if ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize)
elementSize := newSize, element := childEl
}
}
}
stack.RemoveAt(1)
} Until !stack.MaxIndex()
return element
} else {
element := this.ElementFromPoint(x, y, activateChromiumAccessibility)
bound := element.CurrentBoundingRectangle, elementSize := (bound.r-bound.l)*(bound.b-bound.t), prevElementSize := 0
for k, v in element.FindAll(this.__UIA.TrueCondition) {
bound := v.CurrentBoundingRectangle
if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b) && ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize))
element := v, elementSize := newSize
}
return element
}
}
}
class UIA_Element extends UIA_Base {
;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671425(v=vs.85).aspx
static __IID := "{d22108aa-8ac5-49a5-837b-37bbb3d7591e}"
, __properties := "CurrentProcessId,20,int`r`nCurrentControlType,21,CONTROLTYPEID`r`nCurrentLocalizedControlType,22,BSTR`r`nCurrentName,23,BSTR`r`nCurrentAcceleratorKey,24,BSTR`r`nCurrentAccessKey,25,BSTR`r`nCurrentHasKeyboardFocus,26,BOOL`r`nCurrentIsKeyboardFocusable,27,BOOL`r`nCurrentIsEnabled,28,BOOL`r`nCurrentAutomationId,29,BSTR`r`nCurrentClassName,30,BSTR`r`nCurrentHelpText,31,BSTR`r`nCurrentCulture,32,int`r`nCurrentIsControlElement,33,BOOL`r`nCurrentIsContentElement,34,BOOL`r`nCurrentIsPassword,35,BOOL`r`nCurrentNativeWindowHandle,36,UIA_HWND`r`nCurrentItemType,37,BSTR`r`nCurrentIsOffscreen,38,BOOL`r`nCurrentOrientation,39,OrientationType`r`nCurrentFrameworkId,40,BSTR`r`nCurrentIsRequiredForForm,41,BOOL`r`nCurrentItemStatus,42,BSTR`r`nCurrentBoundingRectangle,43,RECT`r`nCurrentLabeledBy,44,IUIAutomationElement`r`nCurrentAriaRole,45,BSTR`r`nCurrentAriaProperties,46,BSTR`r`nCurrentIsDataValidForForm,47,BOOL`r`nCurrentControllerFor,48,IUIAutomationElementArray`r`nCurrentDescribedBy,49,IUIAutomationElementArray`r`nCurrentFlowsTo,50,IUIAutomationElementArray`r`nCurrentProviderDescription,51,BSTR"
FindAll(c="", scope=0x4) {
return UIA_Hr(DllCall(this.__Vt(6), "ptr",this.__Value, "uint",scope, "ptr",(c=""?this.TrueCondition:c).__Value, "ptr*",out))&&out? UIA_ElementArray(out):
}
GetCurrentPropertyValue(propertyId, ByRef out="") {
return UIA_Hr(DllCall(this.__Vt(10), "ptr",this.__Value, "uint", propertyId, "ptr",UIA_Variant(out)))? UIA_VariantData(out):
}
GetSupportedPatterns() { ; Get all available patterns for the element. Use of this should be avoided, since it calls GetCurrentPatternAs for every possible pattern.
result := []
patterns := "Invoke,Selection,Value,RangeValue,Scroll,ExpandCollapse,Grid,GridItem,MultipleView,Window,SelectionItem,Dock,Table,TableItem,Text,Toggle,Transform,ScrollItem,ItemContainer,VirtualizedItem,SyncronizedInput,LegacyIAccessible"
Loop, Parse, patterns, `,
{
try {
if this.GetCurrentPropertyValue(UIA_PropertyId("Is" A_LoopField "PatternAvailable")) {
result.Push(A_LoopField)
}
}
}
return result
}
}
class UIA_ElementArray extends UIA_Base {
static __IID := "{14314595-b4bc-4055-95f2-58f2e42c9855}"
, __properties := "Length,3,int"
GetElement(i) {
return UIA_Hr(DllCall(this.__Vt(4), "ptr",this.__Value, "int",i, "ptr*",out))? new UIA_Element(out):
}
}
class UIA_TreeWalker extends UIA_Base {
;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671470(v=vs.85).aspx
static __IID := "{4042c624-389c-4afc-a630-9df854a541fc}"
, __properties := "Condition,15,IUIAutomationCondition"
GetParentElement(e) {
return UIA_Hr(DllCall(this.__Vt(3), "ptr",this.__Value, "ptr",e.__Value, "ptr*",out))&&out? new UIA_Element(out):
}
GetFirstChildElement(e) {
return UIA_Hr(DllCall(this.__Vt(4), "ptr",this.__Value, "ptr",e.__Value, "ptr*",out))&&out? new UIA_Element(out):
}
GetLastChildElement(e) {
return UIA_Hr(DllCall(this.__Vt(5), "ptr",this.__Value, "ptr",e.__Value, "ptr*",out))&&out? new UIA_Element(out):
}
GetNextSiblingElement(e) {
return UIA_Hr(DllCall(this.__Vt(6), "ptr",this.__Value, "ptr",e.__Value, "ptr*",out))&&out? new UIA_Element(out):
}
}
class UIA_Condition extends UIA_Base {
;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671420(v=vs.85).aspx
static __IID := "{352ffba8-0973-437c-a61f-f64cafd81df9}"
, __properties := ""
}
class UIA_PropertyCondition extends UIA_Condition {
;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee696121(v=vs.85).aspx
static __IID := "{99ebf2cb-5578-4267-9ad4-afd6ea77e94b}"
, __properties := "PropertyId,3,PROPERTYID`r`nPropertyValue,4,VARIANT`r`nPropertyConditionFlags,5,PropertyConditionFlags"
}
class UIA_BoolCondition extends UIA_Condition {
;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671411(v=vs.85).aspx
static __IID := "{1B4E1F2E-75EB-4D0B-8952-5A69988E2307}"
, __properties := "BooleanValue,3,boolVal"
}
UIA_Interface() {
max := 7+1
for k, v in ["{29de312e-83c6-4309-8808-e8dfcb46c3c2}","{aae072da-29e3-413d-87a7-192dbf81ed10}","{25f700c8-d816-4057-a9dc-3cbdee77e256}","{1189c02a-05f8-4319-8e21-e817e3db2860}","{73d768da-9b51-4b89-936e-c209290973e7}","{34723aff-0c9d-49d0-9896-7ab52df8cd8a}"] {
try {
if uia:=ComObjCreate("{e22ad333-b25f-460c-83d0-0581107395c9}",v)
return uia:=new UIA_Interface(uia, 1, max-k), uia.base.base.__UIA:=uia, uia.base.base.TrueCondition:=uia.CreateTrueCondition(), uia.base.base.TreeWalkerTrue := uia.CreateTreeWalker(uia.base.base.TrueCondition)
}
}
; If all else fails, try the first UIAutomation version
try {
if uia:=ComObjCreate("{ff48dba4-60ef-4201-aa87-54103eef594e}","{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
return uia:=new UIA_Interface(uia, 1), uia.base.base.__UIA:=uia, uia.base.base.TrueCondition:=uia.CreateTrueCondition(), uia.base.base.TreeWalkerTrue := uia.CreateTreeWalker(uia.base.base.TrueCondition)
throw "UIAutomation Interface failed to initialize."
} catch e
MsgBox, 262160, UIA Startup Error, % IsObject(e)?"IUIAutomation Interface is not registered.":e.Message
return
}
UIA_Hr(hr) {
;~ http://blogs.msdn.com/b/eldar/archive/2007/04/03/a-lot-of-hresult-codes.aspx
static err:={0x8000FFFF:"Catastrophic failure.",0x80004001:"Not implemented.",0x8007000E:"Out of memory.",0x80070057:"One or more arguments are not valid.",0x80004002:"Interface not supported.",0x80004003:"Pointer not valid.",0x80070006:"Handle not valid.",0x80004004:"Operation aborted.",0x80004005:"Unspecified error.",0x80070005:"General access denied.",0x800401E5:"The object identified by this moniker could not be found.",0x80040201:"UIA_E_ELEMENTNOTAVAILABLE",0x80040200:"UIA_E_ELEMENTNOTENABLED",0x80131509:"UIA_E_INVALIDOPERATION",0x80040202:"UIA_E_NOCLICKABLEPOINT",0x80040204:"UIA_E_NOTSUPPORTED",0x80040203:"UIA_E_PROXYASSEMBLYNOTLOADED"} ; //not completed
if hr&&(hr&=0xFFFFFFFF) {
RegExMatch(Exception("",-2).what,"(\w+).(\w+)",i)
throw Exception(UIA_Hex(hr) " - " err[hr], -2, i2 " (" i1 ")")
}
return !hr
}
UIA_Hex(p) {
setting:=A_FormatInteger
SetFormat,IntegerFast,H
out:=p+0 ""
SetFormat,IntegerFast,%setting%
return out
}
UIA_GUID(ByRef GUID, sGUID) { ;~ Converts a string to a binary GUID and returns its address.
VarSetCapacity(GUID,16,0)
return DllCall("ole32\CLSIDFromString", "wstr",sGUID, "ptr",&GUID)>=0?&GUID:""
}
UIA_ElementArray(p, uia="") { ; should AHK Object be 0 or 1 based? /// answer: 1 based ///
if !p
return
a:=new UIA_ElementArray(p),out:=[]
Loop % a.Length
out[A_Index]:=a.GetElement(A_Index-1)
return out, out.base:={UIA_ElementArray:a}
}
UIA_Variant(ByRef var,type=0,val=0) {
; Does a variant need to be cleared? If it uses SysAllocString?
return (VarSetCapacity(var,8+2*A_PtrSize)+NumPut(type,var,0,"short")+NumPut(type=8? DllCall("oleaut32\SysAllocString", "ptr",&val):val,var,8,"ptr"))*0+&var
}
UIA_IsVariant(ByRef vt, ByRef type="") {
size:=VarSetCapacity(vt),type:=NumGet(vt,"UShort")
return size>=16&&size<=24&&type>=0&&(type<=23||type|0x2000)
}
UIA_VariantData(ByRef p, flag=1) {
return !UIA_IsVariant(p,vt)?"Invalid Variant"
:vt=3?NumGet(p,8,"int")
:vt=8?StrGet(NumGet(p,8))
:vt=9||vt=13||vt&0x2000?ComObj(vt,NumGet(p,8),flag)
:vt<0x1000&&UIA_VariantChangeType(&p,&p)=0?StrGet(NumGet(p,8)) UIA_VariantClear(&p)
:NumGet(p,8)
}
UIA_VariantChangeType(pvarDst, pvarSrc, vt=8) { ; written by Sean
return DllCall("oleaut32\VariantChangeTypeEx", "ptr",pvarDst, "ptr",pvarSrc, "Uint",1024, "Ushort",0, "Ushort",vt)
}
UIA_VariantClear(pvar) { ; Written by Sean
DllCall("oleaut32\VariantClear", "ptr",pvar)
}
UIA_SafeArraysToObject(keys,values) {
;~ 1 dim safearrays w/ same # of elements
out:={}
for key in keys
out[key]:=values[A_Index-1]
return out
}
UIA_RectToObject(ByRef r) { ; rect.__Value work with DllCalls?
static b:={__Class:"object",__Type:"RECT",Struct:Func("UIA_RectStructure")}
return {l:NumGet(r,0,"Int"),t:NumGet(r,4,"Int"),r:NumGet(r,8,"Int"),b:NumGet(r,12,"Int"),base:b}
}
UIA_RectStructure(this, ByRef r) {
static sides:="ltrb"
VarSetCapacity(r,16)
Loop Parse, sides
NumPut(this[A_LoopField],r,(A_Index-1)*4,"Int")
}
UIA_PropertyId(n="") {
static ids:="RuntimeId:30000,BoundingRectangle:30001,ProcessId:30002,ControlType:30003,LocalizedControlType:30004,Name:30005,AcceleratorKey:30006,AccessKey:30007,HasKeyboardFocus:30008,IsKeyboardFocusable:30009,IsEnabled:30010,AutomationId:30011,ClassName:30012,HelpText:30013,ClickablePoint:30014,Culture:30015,IsControlElement:30016,IsContentElement:30017,LabeledBy:30018,IsPassword:30019,NativeWindowHandle:30020,ItemType:30021,IsOffscreen:30022,Orientation:30023,FrameworkId:30024,IsRequiredForForm:30025,ItemStatus:30026,IsDockPatternAvailable:30027,IsExpandCollapsePatternAvailable:30028,IsGridItemPatternAvailable:30029,IsGridPatternAvailable:30030,IsInvokePatternAvailable:30031,IsMultipleViewPatternAvailable:30032,IsRangeValuePatternAvailable:30033,IsScrollPatternAvailable:30034,IsScrollItemPatternAvailable:30035,IsSelectionItemPatternAvailable:30036,IsSelectionPatternAvailable:30037,IsTablePatternAvailable:30038,IsTableItemPatternAvailable:30039,IsTextPatternAvailable:30040,IsTogglePatternAvailable:30041,IsTransformPatternAvailable:30042,IsValuePatternAvailable:30043,IsWindowPatternAvailable:30044,ValueValue:30045,ValueIsReadOnly:30046,RangeValueValue:30047,RangeValueIsReadOnly:30048,RangeValueMinimum:30049,RangeValueMaximum:30050,RangeValueLargeChange:30051,RangeValueSmallChange:30052,ScrollHorizontalScrollPercent:30053,ScrollHorizontalViewSize:30054,ScrollVerticalScrollPercent:30055,ScrollVerticalViewSize:30056,ScrollHorizontallyScrollable:30057,ScrollVerticallyScrollable:30058,SelectionSelection:30059,SelectionCanSelectMultiple:30060,SelectionIsSelectionRequired:30061,GridRowCount:30062,GridColumnCount:30063,GridItemRow:30064,GridItemColumn:30065,GridItemRowSpan:30066,GridItemColumnSpan:30067,GridItemContainingGrid:30068,DockDockPosition:30069,ExpandCollapseExpandCollapseState:30070,MultipleViewCurrentView:30071,MultipleViewSupportedViews:30072,WindowCanMaximize:30073,WindowCanMinimize:30074,WindowWindowVisualState:30075,WindowWindowInteractionState:30076,WindowIsModal:30077,WindowIsTopmost:30078,SelectionItemIsSelected:30079,SelectionItemSelectionContainer:30080,TableRowHeaders:30081,TableColumnHeaders:30082,TableRowOrColumnMajor:30083,TableItemRowHeaderItems:30084,TableItemColumnHeaderItems:30085,ToggleToggleState:30086,TransformCanMove:30087,TransformCanResize:30088,TransformCanRotate:30089,IsLegacyIAccessiblePatternAvailable:30090,LegacyIAccessibleChildId:30091,LegacyIAccessibleName:30092,LegacyIAccessibleValue:30093,LegacyIAccessibleDescription:30094,LegacyIAccessibleRole:30095,LegacyIAccessibleState:30096,LegacyIAccessibleHelp:30097,LegacyIAccessibleKeyboardShortcut:30098,LegacyIAccessibleSelection:30099,LegacyIAccessibleDefaultAction:30100,AriaRole:30101,AriaProperties:30102,IsDataValidForForm:30103,ControllerFor:30104,DescribedBy:30105,FlowsTo:30106,ProviderDescription:30107,IsItemContainerPatternAvailable:30108,IsVirtualizedItemPatternAvailable:30109,IsSynchronizedInputPatternAvailable:30110,OptimizeForVisualContent:30111,IsObjectModelPatternAvailable:30112,AnnotationAnnotationTypeId:30113,AnnotationAnnotationTypeName:30114,AnnotationAuthor:30115,AnnotationDateTime:30116,AnnotationTarget:30117,IsAnnotationPatternAvailable:30118,IsTextPattern2Available:30119,StylesStyleId:30120,StylesStyleName:30121,StylesFillColor:30122,StylesFillPatternStyle:30123,StylesShape:30124,StylesFillPatternColor:30125,StylesExtendedProperties:30126,IsStylesPatternAvailable:30127,IsSpreadsheetPatternAvailable:30128,SpreadsheetItemFormula:30129,SpreadsheetItemAnnotationObjects:30130,SpreadsheetItemAnnotationTypes:30131,IsSpreadsheetItemPatternAvailable:30132,Transform2CanZoom:30133,IsTransformPattern2Available:30134,LiveSetting:30135,IsTextChildPatternAvailable:30136,IsDragPatternAvailable:30137,DragIsGrabbed:30138,DragDropEffect:30139,DragDropEffects:30140,IsDropTargetPatternAvailable:30141,DropTargetDropTargetEffect:30142,DropTargetDropTargetEffects:30143,DragGrabbedItems:30144,Transform2ZoomLevel:30145,Transform2ZoomMinimum:30146,Transform2ZoomMaximum:30147,FlowsFrom:30148,IsTextEditPatternAvailable:30149,IsPeripheral:30150,IsCustomNavigationPatternAvailable:30151,PositionInSet:30152,SizeOfSet:30153,Level:30154,AnnotationTypes:30155,AnnotationObjects:30156,LandmarkType:30157,LocalizedLandmarkType:30158,FullDescription:30159,FillColor:30160,OutlineColor:30161,FillType:30162,VisualEffects:30163,OutlineThickness:30164,CenterPoint:30165,Rotation:30166,Size:30167,IsSelectionPattern2Available:30168,Selection2FirstSelectedItem:30169,Selection2LastSelectedItem:30170,Selection2CurrentSelectedItem:30171,Selection2ItemCount:30173,IsDialog:30174"
if !n
return ids
if n is integer
{
RegexMatch(ids, "([^,]+):" n, m)
return m1
}
n := StrReplace(StrReplace(n, "UIA_"), "PropertyId")
RegexMatch(ids, "(?:^|,)" n "(?:" n ")?(?:Id)?:(\d+)", m)
return m1
}
UIA_ControlTypeId(n="") {
static id:={Button:50000,Calendar:50001,CheckBox:50002,ComboBox:50003,Edit:50004,Hyperlink:50005,Image:50006,ListItem:50007,List:50008,Menu:50009,MenuBar:50010,MenuItem:50011,ProgressBar:50012,RadioButton:50013,ScrollBar:50014,Slider:50015,Spinner:50016,StatusBar:50017,Tab:50018,TabItem:50019,Text:50020,ToolBar:50021,ToolTip:50022,Tree:50023,TreeItem:50024,Custom:50025,Group:50026,Thumb:50027,DataGrid:50028,DataItem:50029,Document:50030,SplitButton:50031,Window:50032,Pane:50033,Header:50034,HeaderItem:50035,Table:50036,TitleBar:50037,Separator:50038,SemanticZoom:50039,AppBar:50040}, name:={50000:"Button",50001:"Calendar",50002:"CheckBox",50003:"ComboBox",50004:"Edit",50005:"Hyperlink",50006:"Image",50007:"ListItem",50008:"List",50009:"Menu",50010:"MenuBar",50011:"MenuItem",50012:"ProgressBar",50013:"RadioButton",50014:"ScrollBar",50015:"Slider",50016:"Spinner",50017:"StatusBar",50018:"Tab",50019:"TabItem",50020:"Text",50021:"ToolBar",50022:"ToolTip",50023:"Tree",50024:"TreeItem",50025:"Custom",50026:"Group",50027:"Thumb",50028:"DataGrid",50029:"DataItem",50030:"Document",50031:"SplitButton",50032:"Window",50033:"Pane",50034:"Header",50035:"HeaderItem",50036:"Table",50037:"TitleBar",50038:"Separator",50039:"SemanticZoom",50040:"AppBar"}
if !n
return id
if n is integer
return name[n]
if ObjHasKey(id, n)
return id[n]
return id[StrReplace(StrReplace(n, "ControlTypeId"), "UIA_")]
}
; Acc functions
GetAccPathTopDown(hwnd, vAccPos, updateTree=False) {
static accTree
if !IsObject(accTree)
accTree := {}
if (!IsObject(accTree[hwnd]) || updateTree)
accTree[hwnd] := BuildAccTreeRecursive(Acc_ObjectFromWindow(hwnd, 0), {})
for k, v in accTree[hwnd] {
if (v == vAccPos)
return k
}
}
BuildAccTreeRecursive(oAcc, tree, path="") {
if !IsObject(oAcc)
return tree
try
oAcc.accChildCount
catch
return tree
For i, oChild in Acc_Children(oAcc) {
if IsObject(oChild)
Acc_Location(oChild,,vChildPos)
else
Acc_Location(oAcc,oChild,vChildPos)
tree[path (path?(IsObject(oChild)?".":" c"):"") i] := vChildPos
tree := BuildAccTreeRecursive(oChild, tree, path (path?".":"") i)
}
return tree
}
Acc_Init()
{
Static h
If Not h
h:=DllCall("LoadLibrary","Str","oleacc","Ptr")
}
Acc_ObjectFromEvent(ByRef _idChild_, hWnd, idObject, idChild)
{
Acc_Init()
If DllCall("oleacc\AccessibleObjectFromEvent", "Ptr", hWnd, "UInt", idObject, "UInt", idChild, "Ptr*", pacc, "Ptr", VarSetCapacity(varChild,8+2*A_PtrSize,0)*0+&varChild)=0
Return ComObjEnwrap(9,pacc,1), _idChild_:=NumGet(varChild,8,"UInt")
}
Acc_ObjectFromPoint(ByRef _idChild_ = "", x = "", y = "")
{
Acc_Init()
If DllCall("oleacc\AccessibleObjectFromPoint", "Int64", x==""||y==""?0*DllCall("GetCursorPos","Int64*",pt)+pt:x&0xFFFFFFFF|y<<32, "Ptr*", pacc, "Ptr", VarSetCapacity(varChild,8+2*A_PtrSize,0)*0+&varChild)=0
Return ComObjEnwrap(9,pacc,1), _idChild_:=NumGet(varChild,8,"UInt")
}
Acc_ObjectFromWindow(hWnd, idObject = -4)
{
Acc_Init()
If DllCall("oleacc\AccessibleObjectFromWindow", "Ptr", hWnd, "UInt", idObject&=0xFFFFFFFF, "Ptr", -VarSetCapacity(IID,16)+NumPut(idObject==0xFFFFFFF0?0x46000000000000C0:0x719B3800AA000C81,NumPut(idObject==0xFFFFFFF0?0x0000000000020400:0x11CF3C3D618736E0,IID,"Int64"),"Int64"), "Ptr*", pacc)=0
Return ComObjEnwrap(9,pacc,1)
}
Acc_WindowFromObject(pacc)
{
If DllCall("oleacc\WindowFromAccessibleObject", "Ptr", IsObject(pacc)?ComObjValue(pacc):pacc, "Ptr*", hWnd)=0
Return hWnd
}
Acc_Error(p="") {
static setting:=0
return p=""?setting:setting:=p
}
Acc_Children(Acc) {
if ComObjType(Acc,"Name") != "IAccessible"
ErrorLevel := "Invalid IAccessible Object"
else {
Acc_Init(), cChildren:=Acc.accChildCount, Children:=[]
if DllCall("oleacc\AccessibleChildren", "Ptr",ComObjValue(Acc), "Int",0, "Int",cChildren, "Ptr",VarSetCapacity(varChildren,cChildren*(8+2*A_PtrSize),0)*0+&varChildren, "Int*",cChildren)=0 {
Loop %cChildren%
i:=(A_Index-1)*(A_PtrSize*2+8)+8, child:=NumGet(varChildren,i), Children.Push(NumGet(varChildren,i-8)=9?Acc_Query(child):child), NumGet(varChildren,i-8)=9?ObjRelease(child):
return Children.MaxIndex()?Children:
} else
ErrorLevel := "AccessibleChildren DllCall Failed"
}
if Acc_Error()
throw Exception(ErrorLevel,-1)
}
Acc_Location(Acc, ChildId=0, byref Position="") { ; adapted from Sean's code
try Acc.accLocation(ComObj(0x4003,&x:=0), ComObj(0x4003,&y:=0), ComObj(0x4003,&w:=0), ComObj(0x4003,&h:=0), ChildId)
catch
return
Position := "x" NumGet(x,0,"int") " y" NumGet(y,0,"int") " w" NumGet(w,0,"int") " h" NumGet(h,0,"int")
return {x:NumGet(x,0,"int"), y:NumGet(y,0,"int"), w:NumGet(w,0,"int"), h:NumGet(h,0,"int")}
}
Acc_Parent(Acc)
{
try parent:=Acc.accParent
return parent?Acc_Query(parent):
}
Acc_Child(Acc, ChildId=0)
{
try child:=Acc.accChild(ChildId)
return child?Acc_Query(child):
}
Acc_Query(Acc)
{
try return ComObj(9, ComObjQuery(Acc,"{618736e0-3c3d-11cf-810c-00aa00389b71}"), 1)
}
RangeTip(x:="", y:="", w:="", h:="", color:="Red", d:=2) ; from the FindText library, credit goes to feiyue
{
local
static id:=0
if (x="")
{
id:=0
Loop 4
Gui, Range_%A_Index%: Destroy
return
}
if (!id)
{
Loop 4
Gui, Range_%A_Index%: +Hwndid +AlwaysOnTop -Caption +ToolWindow
-DPIScale +E0x08000000
}
x:=Floor(x), y:=Floor(y), w:=Floor(w), h:=Floor(h), d:=Floor(d)
Loop 4
{
i:=A_Index
, x1:=(i=2 ? x+w : x-d)
, y1:=(i=3 ? y+h : y-d)
, w1:=(i=1 or i=3 ? w+2*d : d)
, h1:=(i=2 or i=4 ? h+2*d : d)
Gui, Range_%i%: Color, %color%
Gui, Range_%i%: Show, NA x%x1% y%y1% w%w1% h%h1%
}
}
#If IsCapturing
Esc::gosub ButCapture
NOTE: if inspecting browsers, it is good to make sure that accessibility is turned on: either run the browser with --force-renderer-accessibility flag (eg chrome.exe --force-renderer-accessibility) or use browser-specific user interfaces (for example in Chrome navigate to chrome://accessibility/ and turn on accessibility). Usually this isn't necessary, but if you have trouble inspecting browser content elements then I suggest trying it.
Some notes and caveats on UIA:
1) Firstly, every UIA call to interact with elements is a cross-application call from AutoHotkey to the UIA framework, which takes a relatively long time. This means that it is recommended to set tight conditions for searches (like FindAll) and to do as little calls as possible. Doing something like FindAll with the True condition and looping over all the elements to find one specific one is strongly not recommended, and usually a better solution exists anyway.
2) Like Acc, UIA also depends on the implementation of accessibility methods of specific programs. If a program doesn't implement UIA support for custom controls (such as Chrome scrollbar, or Sublime Text status bar), then unfortunately they simply can't be accessed with UIA. Fortunately most common controls (buttons, edits etc) and website elements ARE supported.
3) In some Windows builds Chromium-based apps (web browsers, Skype etc) might not automatically allow viewing content. This appears to be a bug that was eventually fixed (in my tests Windows 11 didn't have this prblem), but it can also be worked around by sending the WM_GETOBJECT message to the renderer element before making any UIA calls, for example with the following function. This doesn't need to be used if accessibility is already enabled (eg in chrome://accessibility) or browser is ran with the --force-renderer-accessibility flag.
Code: Select all
ChromiumActivateUIA(wTitle) {
WinGet, cList, ControlList, %wTitle%
if InStr(cList, "Chrome_RenderWidgetHostHWND1")
SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Chrome_RenderWidgetHostHWND1, %wTitle%
}
Now for some examples using the UIA library.
1) Setting Notepad text. Opens up Notepad, then sets the text field to contain the text "Lorem ipsum"
Code: Select all
#NoEnv
#SingleInstance force
SetTitleMatchMode, 2
#include <UIA_Interface>
Run, notepad.exe
UIA := UIA_Interface() ; Initialize UIA interface
WinWaitActive, ahk_exe notepad.exe
npEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe")) ; Get the element for the Notepad window
documentEl := npEl.FindFirstByType("Document") ; Find the first Document control (in Notepad there is only one). This assumes the user is running a relatively recent Windows and UIA interface version 2+ is available. In UIA interface v1 this control was Edit, so an alternative option instead of "Document" would be "UIA.__Version > 1 ? "Document" : "Edit""
documentEl.SetValue("Lorem ipsum") ; Set the value for the document control
ExitApp
Code: Select all
#NoEnv
#SingleInstance force
SetTitleMatchMode, 2
#include <UIA_Interface>
Run, notepad.exe
UIA := UIA_Interface() ; Initialize UIA interface
WinWaitActive, ahk_exe notepad.exe
npEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe")) ; Get the element for the Notepad window
MsgBox, % npEl.DumpAll() ; Display all the sub-elements for the Notepad window. Press OK to continue
documentEl := npEl.FindFirstByType("Document") ; Find the first Document control (in Notepad there is only one). This assumes the user is running a relatively recent Windows and UIA interface version 2+ is available. In UIA interface v1 this control was Edit, so an alternative option instead of "Document" would be "UIA.__Version > 1 ? "Document" : "Edit""
documentEl.SetValue("Lorem ipsum") ; Set the value of the document control
MsgBox, Press OK to test saving. ; Wait for the user to press OK
fileEl := npEl.FindFirstByNameAndType("File", "MenuItem").Click() ; Click the "File" menu item
saveEl := npEl.WaitElementExistByName("Save",,2) ; Wait for the "Save" menu item to exist
saveEl.Click() ; And now click Save
ExitApp
Code: Select all
#NoEnv
#SingleInstance force
#include <UIA_Interface>
Run, calc.exe
UIA := UIA_Interface() ; Initialize UIA interface
WinWaitActive, Calculator
cEl := UIA.ElementFromHandle(WinExist("Calculator")) ; Get the element for the Calculator window
; All the calculator buttons are of "Button" ControlType, and if the system language is English then the Name of the elements are the English words for the buttons (eg button 5 is named "Five", = sign is named "Equals")
cEl.FindFirstBy("Name=Six").Click() ; Find the first "Six" button by name and click it
cEl.FindFirstBy("Name=Five AND ControlType=Button").Click() ; Specify both name "Five" and control type "Button"
cEl.FindFirstByName("Plus").Click() ; An alternative method to FindFirstBy("Name=Plus")
cEl.FindFirstByNameAndType("Four", UIA.ButtonControlTypeId).Click() ; The type can be specified as "Button", UIA.ButtonControlTypeId, UIA_Enum.UIA_ButtonControlTypeId, or 50000 (which is the value of UIA.ButtonControlTypeId)
cEl.FindFirstByNameAndType("Equals", "Button").Click()
ExitApp
4) First a simple test script to dump all the page information to the clipboard, after pressing F1 use Ctrl+V to paste it somewhere such as Notepad. Chrome is ran in incognito mode to avoid any extensions or previous logins on webpages from interfering.
Code: Select all
#NoEnv
#SingleInstance force
SetTitleMatchMode, 2
#include <UIA_Interface>
#include <UIA_Browser>
browserExe := "chrome.exe"
Run, %browserExe% -incognito --force-renderer-accessibility ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.WaitPageLoad("New Tab", 3000) ; Wait the New Tab page to load with a timeout of 3 seconds
Clipboard=
Clipboard := cUIA.GetCurrentDocumentElement().DumpAll() ; Get the current document element (this excludes the URL bar, navigation buttons etc) and dump all the information about it in the clipboard. Use Ctrl+V to paste it somewhere, such as in Notepad.
ClipWait, 1
if Clipboard
MsgBox, Page information successfully dumped. Use Ctrl+V to paste the info somewhere, such as in Notepad.
else
MsgBox, Something went wrong and nothing was dumped in the clipboard!
return
Code: Select all
#NoEnv
#SingleInstance force
SetTitleMatchMode, 2
#include <UIA_Interface>
#include <UIA_Browser>
browserExe := "chrome.exe"
Run, %browserExe% -incognito --force-renderer-accessibility ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
; Before doing a translate, lets first set Google services language to English to ensure that locale-specific words are in English (if Google is in German for example, "English" would be "Englisch"
cUIA.WaitPageLoad("New Tab", 5000) ; Wait the New Tab page to load with a timeout of 5 seconds
cUIA.SetURL("https://www.google.com/preferences#languages", True) ; Set the URL and navigate to it
cUIA.WaitPageLoad() ; Wait the page to load
cUIA.FindFirstByName("Show more").Click() ; Display all languages to ensure English is visible
cUIA.WaitElementExistByName("English").Click() ; Select English
cUIA.FindFirstByName("Save").Click() ; Click "Save"
Sleep, 1000
cUIA.FindFirstByName("OK").Click() ; Sometimes a dialog pops up that confirms the save, in that case press "OK"
cUIA.WaitPageLoad("Google") ; Wait for Google main page to load, default timeout of 10 seconds
cUIA.SetURL("https://translate.google.com/", True) ; Navigate to Google Translate
cUIA.WaitPageLoad()
cUIA.FindFirstByName("More source languages").Click() ; Click source languages selection
cUIA.WaitElementExistByName("Spanish").Click() ; Select Spanish
Sleep, 500
cUIA.FindFirstByName("More target languages").Click() ; Open target languages selection
Sleep, 500
allEnglishEls := cUIA.FindAllByName("English") ; Find all elements with name "English"
allEnglishEls[allEnglishEls.MaxIndex()].Click() ; Select the last element with the name English (because English might also be an option in source languages, in which case it would be found first)
cUIA.WaitElementExistByName("Source text").SetValue("Este es un texto de muestra") ; Set some text to translate
return
There are some caveats with Edge: sometimes after loading a document or webpage, it takes some time and multiple calls of FindFirst or FindAll to find elements. This means that after navigation or clicking, it is recommended to use WaitElementExist functions to make sure the element is found.
Code: Select all
#NoEnv
#SingleInstance force
SetTitleMatchMode, 2
#include <UIA_Interface>
#include <UIA_Browser>
browserExe := "msedge.exe"
Run, %browserExe% -inprivate --force-renderer-accessibility ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.WaitPageLoad("New Tab", 5000) ; Wait the New Tab page to load with a timeout of 5 seconds
cUIA.SetURL("google.com", True) ; Set the URL to google and navigate
cUIA.WaitPageLoad()
langBut := cUIA.WaitElementExist("ClassName=neDYw tHlp8d AND ControlType=Button OR ControlType=MenuItem") ; First lets make sure the selected language is correct. The expression is evaluated left to right: finds an element where ClassName is "neDYw tHlp8d" and ControlType is button OR an element with a MenuItem controltype.
expandCollapsePattern := langBut.GetCurrentPatternAs("ExpandCollapse") ; Since this isn't a normal button, we need to get the ExpandCollapse pattern for it and then call the correct method.
if (expandCollapsePattern.CurrentExpandCollapseState == cUIA.ExpandCollapseState_Collapsed) ; Check that it is collapsed
expandCollapsePattern.Expand() ; Expand the menu
cUIA.WaitElementExist("Name=English AND ControlType=MenuItem").Click() ; Select the English language
cUIA.WaitElementExist("Name=Accept all OR Name=I agree AND ControlType=Button").Click() ; If the "I agree" or "Accept all" button exists, then click it to get rid of the consent form
searchBox := cUIA.WaitElementExist("Name=Searc AND ControlType=ComboBox",,2) ; Looking for a partial name match "Searc" using matchMode=2. FindFirstByNameAndType is not used here, because if the "I agree" button was clicked then this element might not exist right away, so lets first wait for it to exist.
searchBox.SetValue("autohotkey forums") ; Set the search box text to "autohotkey forums"
cUIA.FindFirstByName("Google Search").Click() ; Click the search button to search
cUIA.WaitPageLoad()
; Now that the Google search results have loaded, lets scroll the page to the end.
docEl := cUIA.GetCurrentDocumentElement() ; Get the document element
sbPat := docEl.GetCurrentPatternAs("Scroll") ; Get the Scroll pattern to access scrolling methods and properties
hPer := sbPat.CurrentHorizontalScrollPercent ; Horizontal scroll percent doesn't actually interest us, but the SetScrollPercent method requires both horizontal and vertical scroll percents...
ToolTip, % "Current scroll percent: " sbPat.CurrentVerticalScrollPercent
for k, v in [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] { ; Lets scroll down in steps of 10%
sbPat.SetScrollPercent(hPer, v)
Sleep, 500
ToolTip, % "Current scroll percent: " sbPat.CurrentVerticalScrollPercent
}
Sleep, 3000
ToolTip
ExitApp
Code: Select all
#SingleInstance, force
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
SetTitleMatchMode, 2
#include <UIA_Interface>
#include <UIA_Browser>
EventHandler(el) {
global cUIA
ToolTip, % "Caught event!`nElement name: " el.CurrentName
if cUIA.CompareElements(cUIA.URLEditElement, el) ; Check if the focused element is the same as Chrome's address bar element (comparison using == won't work)
el.SetValue("") ; If the Address bar was focused, clear it
}
ExitFunc() {
global cUIA, h
UIA.RemoveFocusChangedEventHandler(h) ; Remove the event handler. Alternatively use cUIA.RemoveAllEventHandlers() to remove all handlers
}
browserExe := "chrome.exe"
Run, %browserExe% -incognito
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe)
h := UIA_CreateEventHandler("EventHandler", "FocusChanged") ; Create a new FocusChanged event handler that calls the function EventHandler (required arguments: element)
cUIA.AddFocusChangedEventHandler(h) ; Add a new FocusChangedEventHandler
OnExit("ExitFunc") ; Set up an OnExit call to clean up the handler when exiting the script
return
F5::ExitApp
Code: Select all
#SingleInstance, force
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
SetTitleMatchMode, 2
#include <UIA_Interface>
TextSelectionChangedEventHandler(el, eventId) {
textPattern := el.GetCurrentPatternAs("TextPattern")
selectionArray := textPattern.GetSelection() ; Gets the currently selected text in Notepad as an array of TextRanges (some elements support selecting multiple pieces of text at the same time, thats why an array is returned)
selectedRange := selectionArray[1] ; Our range of interest is the first selection (TextRange)
wholeRange := textPattern.DocumentRange ; For comparison, get the whole range (TextRange) of the document
selectionStart := selectedRange.CompareEndpoints(UIA_Enum.TextPatternRangeEndpoint_Start, wholeRange, UIA_Enum.TextPatternRangeEndpoint_Start) ; Compare the start point of the selection to the start point of the whole document
selectionEnd := selectedRange.CompareEndpoints(UIA_Enum.TextPatternRangeEndpoint_End, wholeRange, UIA_Enum.TextPatternRangeEndpoint_Start) ; Compare the end point of the selection to the start point of the whole document
ToolTip, % "Selected text: " selectedRange.GetText() "`nSelection start location: " selectionStart "`nSelection end location: " selectionEnd ; Display the selected text and locations of selection
}
ExitFunc() {
global UIA, handler, NotepadEl
try UIA.RemoveAutomationEventHandler(UIA.Text_TextSelectionChangedEventId, NotepadEl, handler) ; Remove the event handler. Alternatively use UIA.RemoveAllEventHandlers() to remove all handlers. If the Notepad window doesn't exist any more, this throws an error.
}
lorem =
(
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
) ; Some sample text to play around with
Run, notepad.exe
WinActivate, ahk_exe notepad.exe
WinWaitActive, ahk_exe notepad.exe
UIA := UIA_Interface()
NotepadEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe"))
DocumentControlCondition := UIA.CreatePropertyCondition(UIA.ControlTypePropertyId, UIA.DocumentControlTypeId) ; If UIA Interface version is 1, then the ControlType is Edit instead of Document!
DocumentControl := NotepadEl.FindFirst(DocumentControlCondition)
DocumentControl.SetValue(lorem) ; Set the value to our sample text
handler := UIA_CreateEventHandler("TextSelectionChangedEventHandler") ; Create a new event handler that points to the function TextSelectionChangedEventHandler, which must accept two arguments: element and eventId.
UIA.AddAutomationEventHandler(UIA.Text_TextSelectionChangedEventId, NotepadEl,,, handler) ; Add a new automation handler for the TextSelectionChanged event
OnExit("ExitFunc") ; Set up an OnExit call to clean up the handler when exiting the script
return
F5::ExitApp
UIA_Interface, UIA_Browser properties and methods
Properties and methods for UIA_Browser.ahk:
Code: Select all
;When initializing UIA_Browser it takes as an argument the browsers title. It follows the same syntax as ahk WinTitle, and SetTitleMatchMode also works the same. Example:
cUIA := new UIA_Browser("ahk_exe chrome.exe")
; UIA_Browser properties:
; UIA_Browser can return both UIA_Interface and the browser window elements properties and methods, depending on where the property exists.
; Example: cUIA.CurrentProcessId will return the ProcessId for the browser window element by calling cUIA.BrowserElement.CurrentProcessId.
UIA_Browser.UIA ; The UIA_Interface object itself, which can be more readily accessed by just calling a UIA_Interface method/property from UIA_Browser (eg cUIA.CreateNotCondition() will actually call cUIA.UIA.CreateNotCondition())
UIA_Browser.BrowserId ; ahk_id of the browser window
UIA_Browser.BrowserType ; "Chrome", "Edge", or "Unknown"
UIA_Browser.BrowserElement ; The browser window element, which can also be accessed by just calling an element method from UIA_Browser (cUIA.FindFirst would call FindFirst method on the BrowserElement, is equal to cUIA.BrowserElement.FindFirst)
UIA_Browser.MainPaneElement ; Element for the upper part of the browser containing the URL bar, tabs, extensions etc
UIA_Browser.URLEditElement ; Element for the address bar
UIA_Browser.TWT ; Contains a TreeWalker that is created with TrueCondition
; All UIA enumerations/constants are also available in the UIA_Browser object.
cUIA.UIA_ControlTypePropertyId ; Returns UIA_ControlTypePropertyId from UIA_Enum, returning value 30003
cUIA.ButtonControlTypeId ; "UIA_" can be omitted. Returns UIA_ButtonControlTypeId from UIA_Enum class, returning value 50000
; UIA_Browser methods:
; UIA_Browser can return both UIA_Interface and the browser window elements properties and methods, depending on where the method exists.
Example: cUIA.CreateTreeWalker(condition) accesses the CreateTreeWalker method from UIA_Interface class; cUIA.FindFirst(condition) calls FindFirst on the browser window element (because FindFirst doesn't exist inside UIA_Interface).
UIA_Browser.GetCurrentMainPaneElement() ; Refreshes UIA_Browser.MainPaneElement and also returns it
UIA_Browser.GetCurrentDocumentElement() ; Returns the current document/content element of the browser
UIA_Browser.GetAllText() ; Gets all text from the browser element (CurrentName properties for all child elements)
UIA_Browser.GetAllLinks() ; Gets all link elements from the browser
UIA_Browser.WaitTitleChange(targetTitle="", timeOut=10000) ; Waits the browser title to change to targetTitle (by default just waits for the title to change), timeOut is in milliseconds (default is 10 seconds)
UIA_Browser.WaitPageLoad(targetTitle="", timeOut=10000, sleepAfter=500) ; Waits the browser page to load to targetTitle, default timeOut is 10 seconds, sleepAfter additionally sleeps for 500ms after the page has loaded.
UIA_Browser.Back() ; Presses the Back button
UIA_Browser.Forward() ; Presses the Forward button
UIA_Browser.Reload() ; Presses the Reload button
UIA_Browser.Home(butName="Home") ; Presses the Home button if it exists. If the browser language is not set to English, the correct butName can be specified.
UIA_Browser.GetCurrentURL(fromAddressBar=False) ; Gets the current URL. fromAddressBar=True gets it straight from the URL bar element, which is not a very good method, because the text might be changed by the user and doesn't start with "http(s)://". Default of fromAddressBar=False will cause the real URL to be fetched, but the browser must be visible for it to work (if is not visible, it will be automatically activated).
UIA_Browser.GetCurrentDocumentElement() ; Gets the current element for the content of the website
UIA_Browser.SetURL(newUrl, navigateToNewUrl = False) ; Sets the URL bar to newUrl, optionally also navigates to it if navigateToNewUrl=True
UIA_Browser.Navigate(url, targetTitle="", waitLoadTimeOut=10000, sleepAfter=500, stopButtonText="Stop") ; Navigates to URL and waits page to load
UIA_Browser.NewTab(butName="New tab", matchMode=2, caseSensitive=True) ; Presses the New tab button. The button name might differ if the browser language is not set to English and can be specified with butName (with options for matchMode and caseSensitivity for the search)
UIA_Browser.GetAllTabs() ; Gets all tab elements
UIA_Browser.GetAllTabNames() ; Gets all the titles of tabs
UIA_Browser.SelectTab(tabName, matchMode=3, caseSensitive=True) ; Selects a tab with the text of tabName. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
UIA_Browser.GetTab(searchPhrase="", matchMode=3, caseSensitive=True) ; Returns a tab element with text of searchPhrase, or if empty then the currently selected tab. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
UIA_Browser.CloseTab(tabElementOrName="", matchMode=3, caseSensitive=True) ; Close tab by either providing the tab element or the name of the tab. If tabElementOrName is left empty, the current tab will be closed.
UIA_Browser.IsBrowserVisible() ; Returns True if any of the 4 corners of the browser are visible.
UIA_Browser.GetAlertText() ; Gets the text from an alert box
UIA_Browser.CloseAlert() ; Closes an alert box
UIA_Browser.JSExecute(js) ; Executes Javascript code using the address bar
UIA_Browser.JSReturnThroughClipboard(js) ; Executes Javascript code using the address bar and returns the return value of the code using the clipboard (resetting it back afterwards)
UIA_Browser.JSReturnThroughTitle(js, timeOut=500) ; Executes Javascript code using the address bar and returns the return value of the code using the browsers title (resetting it back afterwards). This might be unreliable, so the clipboard method is recommended instead.
UIA_Browser.JSSetTitle(newTitle) ; Uses Javascript through the address bar to change the title of the browser
UIA_Browser.JSGetElementPos(selector, useRenderWidgetPos=False) ; Uses Javascript's querySelector to get a Javascript element and then its position. useRenderWidgetPos=True uses position of the Chrome_RenderWidgetHostHWND1 control to locate the position element relative to the window, otherwise it uses UIA_Browsers CurrentDocumentElement position.
UIA_Browser.JSClickElement(selector) ; Uses Javascript's querySelector to get and click a Javascript element
UIA_Browser.ControlClickJSElement(selector, WhichButton="", ClickCount="", Options="", useRenderWidgetPos=False) ; Uses Javascript's querySelector to get a Javascript element and then ControlClicks that position. useRenderWidgetPos=True uses position of the Chrome_RenderWidgetHostHWND1 control to locate the position element relative to the window, otherwise it uses UIA_Browsers CurrentDocumentElement position.
UIA_Browser.ClickJSElement(selector, WhichButton="", ClickCount=1, DownOrUp="", Relative="", useRenderWidgetPos=False) ; Uses Javascript's querySelector to get a Javascript element and then Clicks that position. useRenderWidgetPos=True uses position of the Chrome_RenderWidgetHostHWND1 control to locate the position element relative to the window, otherwise it uses UIA_Browsers CurrentDocumentElement position.
Code: Select all
UIA_Interface functions:
UIA_Interface(maxVersion="") ; Creates a new UIAutomation object, or returns a previously created one. maxVersion can be specified to limit the highest used IUIAutomation version (for Windows 10 this currently is 7). Most applications can use IUIAutomation2, but since higher IUIAutomation versions can use all the methods and properties of the lower ones, usually the highest version can be used.
UIA_Enum(e) ; Returns a constant/enumeration from the UIA_Enum class (which can also be used directly). If a constant starts with "UIA_", that can be omitted. Example: UIA_Enum("ButtonControlTypeId") returns 50000.
UIA_CreateEventHandler(funcName, handlerType="") ; Creates a new event handler object that calls funcName when the appropriate event is received (after calling a Add...EventHandler method). handlerType should be left empty when using AddAutomationEventHandler, in other cases specify the EventHandler name (possible options: FocusChanged, StructureChanged, TextEditTextChanged, Changes, Notification), eg for AddFocusChangedEventHandler specify "FocusChanged".
UIA_Base class properties (apply for the UIA_Interface object, and for all sub-objects suchs as elements etc):
UIA_Base.TrueCondition ; Contains the return value for CreateTrueCondition, for more convenient use
UIA_Base.TreeWalkerTrue ; Contains a TreeWalker created with TrueCondition, for more convenient use
class UIA_Interface
UIA_Interface.CreateCondition(property, val) ; Wrapper function for CreatePropertyCondition. Property can be either the number or text from UIA_PropertyIds (PROPERTY part of UIA_PROPERTYPropertyId).
; cUIA.CreateCondition("Name", "Username") ; Creates a new property condition using UIA_NamePropertyId with the value of "Username"
UIA_Interface.ElementFromChromium(winTitle="A", activateChromiumAccessibility=True) ; Usually gets the Document element for Chromium apps (such as Skype, Discord etc) if it is inaccessible with ElementFromHandle
UIA_Interface.SmallestElementFromPoint(x="", y="", activateChromiumAccessibility=True) ; Gets ElementFromPoint and filters out the smallest subelement that is under the specified point.
class UIA_Element
UIA_Element.CurrentValue ; This property gets or sets the current value of the element. This is a wrapper for GetCurrentPropertyValue("Value") and SetValue()
UIA_Element.CurrentExists ; Checks if the element stills exists
UIA_Element.WaitNotExist(timeOut=10000) ; Wait until the element doesn't exist, with a default timeOut of 10000ms (10 seconds). Returns 1 if the element doesn't exist, otherwise 0.
UIA_Element.GetClickablePointRelativeTo(relativeTo="") { ; Wrapped for the GetClickablePoint function, that converts the coordinates according to the mode selected with relativeTo. relativeTo can be window, screen or client, default is A_CoordModeMouse.
UIA_Element.GetParentHwnd() ; Gets the hwnd for the parent window. A more reliable way would be to use CurrentProcessId with ahk_pid.
UIA_Element.SetValue(val, pattern="") ; Set element value using Value pattern, or as a fall-back using LegacyIAccessible pattern. If a pattern is specified then that is used instead.
/*
UIA_Element.Click
Click using one of the available click-like methods (InvokePattern Invoke(), TogglePattern Toggle(), ExpandCollapsePattern Expand() or Collapse() (depending on the state of the element), SelectionItemPattern Select(), or LegacyIAccessible DoDefaultAction()), in which case ClickCount is ignored. If WhichButton is specified (for example "left", "right") then the native mouse Click function will be used to click the center of the element.
If WhichButton is a number, then Sleep will be called with that number. Eg Click(200) will sleep 200ms after clicking
If ClickCountAndSleepTime is a number >=10, then Sleep will be called with that number. To click 10+ times and sleep after, specify "ClickCount SleepTime". Ex1: Click("left", 200) will sleep 200ms after clicking. Ex2: Click("left", "20 200") will left-click 20 times and then sleep 200ms.
If Relative is "Rel" or "Relative" then X and Y coordinates are treated as offsets from the current mouse position. Otherwise it expects offset values for both X and Y (eg "-5 10" would offset X by -5 and Y by +10).
*/
UIA_Element.Click(WhichButtonOrSleepTime="", ClickCountAndSleepTime=1, DownOrUp="", Relative="")
; ControlClicks the element after getting relative coordinates with GetClickablePointRelativeTo("window"). Specifying WinTitle makes the function faster, since it bypasses getting the Hwnd from the element.
; If WinTitle or WinText is a number, then Sleep will be called with that number of milliseconds. Ex: ControlClick(200) will sleep 200ms after clicking. Same for ControlClick("ahk_id 12345", 200)
UIA_Element.ControlClick(WinTitleOrSleepTime="", WinTextOrSleepTime="", WhichButton="", ClickCount="", Options="", ExcludeTitle="", ExcludeText="")
UIA_Element.GetSupportedPatterns() ; Get available patterns for the element. Use of this should be avoided, since it calls GetCurrentPatternAs for every pattern. A better option is PollForPotentialSupportedPatterns
UIA_Element.GetCurrentPos(relativeTo="") ; Returns an object containing the x, y coordinates and width and height: {x:x coordinate, y:y coordinate, w:width, h:height}. relativeTo can be client, window or screen, default is A_CoordModeMouse.
UIA_Element.GetChildren(scope=0x2) ; By default get only direct children (TreeScope_Children)
UIA_Element.Dump() { ; Returns info about the element: ControlType, Name, Value, LocalizedControlType, AcceleratorKey, AutomationId.
UIA_Element.DumpAll(maxDepth=20) ; Returns info (ControlType, Name etc) for all descendants of the element. maxDepth is the allowed depth of recursion, by default 20 layers. DO NOT call this on the root element!
/*
FindFirst using search criteria.
expr:
Takes a value in the form of "PropertyId=matchvalue" to match a specific property with the value matchValue. PropertyId can be most properties from UIA_Enum.UIA_PropertyId method (for example Name, ControlType, AutomationId etc).
Example1: "Name=Username:" would use FindFirst with UIA_Enum.UIA_NamePropertyId matching the name "Username:"
Example2: "ControlType=Button would FindFirst using UIA_Enum.UIA_ControlTypePropertyId and matching for UIA_Enum.UIA_ButtonControlTypeId. Alternatively "ControlType=50000" can be used (direct value for UIA_ButtonControlTypeId which is 50000)
Criteria can be combined with AND, OR, &&, ||:
Example3: "Name=Username: AND ControlType=Button" would FindFirst an element with the name property of "Username:" and control type of button.
Flags can be modified for each individual condition by specifying FLAGS=n after the condition (and before and/or operator). 0=no flags; 1=ignore case (case insensitive matching); 2=match substring; 3=ignore case and match substring
If matchMode==3 or matching substrings is supported (Windows 10 17763 and above) and matchMode==2, then parentheses are supported.
Otherwise parenthesis are not supported, and criteria are evaluated left to right, so "a AND b OR c" would be evaluated as "(a and b) or c".
Negation can be specified with NOT:
Example4: "NOT ControlType=Edit" would return the first element that is not an edit element
scope:
Scope by default is UIA_TreeScope_Descendants.
matchMode:
If using Name PropertyId as a criteria, this follows the SetTitleMatchMode scheme:
1=name must must start with the specified name
2=can contain anywhere
3=exact match
RegEx=using regular expression. In this case the Name can't be empty.
caseSensitive:
If matching for a string, this will specify case-sensitivity.
*/
UIA_Element.FindFirstBy(expr, scope=0x4, matchMode=3)
UIA_Element.FindFirstByName(name, scope=0x4, matchMode=3) ; Wrapper for FindFirst using UIA_NamePropertyId. Scope by default is UIA_TreeScope_Descendants. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
UIA_Element.FindFirstByType(controlType, scope=0x4) ; Wrapper for FindFirst using UIA_ControlTypePropertyId
UIA_Element.FindFirstByNameAndType(name, controlType, scope=0x4, matchMode=3) ; Wrapper for FindFirst using UIA_NamePropertyId, UIA_ControlTypePropertyId and creating an AND condition for the two.
UIA_Element.FindAllByName(name, scope=0x4, matchMode=3) ; Wrapper for FindAll using UIA_NamePropertyId. Scope by default is UIA_TreeScope_Descendants. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
UIA_Element.FindAllByType(controlType, scope=0x4) ; Wrapper for FindAll using UIA_ControlTypePropertyId.
UIA_Element.FindAllByNameAndType(name, controlType, scope=0x4, matchMode=3) ; Wrapper for FindAll using UIA_NamePropertyId, UIA_ControlTypePropertyId and creating an AND condition for the two.
UIA_Element.FindByPath(path) ; Gets an element by the "path" that is displayed in the UIA_Element.DumpAll() result. This is similar to the Acc path, but for UIA (they are not compatible). Use of this is discouraged, because usually there are more reliable methods available. Also can be used to get parent elements ("Pn") and sibling elements (+n or -n): "P2.+2.1" would get parent of the parent, and then the second siblings first child.
UIA_Element.WaitElementExist(expr, scope=0x4, matchMode=3, timeOut=10000) ; Calls UIA_Element.FindFirstBy until the element is found and then returns it, with a timeOut of 10000ms (10 seconds)
UIA_Element.WaitElementNotExist(expr, scope=0x4, matchMode=3, caseSensitive=True, timeOut=10000) ; Tries to FindFirstBy the element and if it is found then waits until the element doesn't exist (using WaitNotExist()), with a timeOut of 10000ms (10 seconds). For explanations of the other arguments, see FindFirstBy
UIA_Element.WaitElementExistByName(name, scope=0x4, matchMode=3, timeOut=10000) ; Calls UIA_Element.FindFirstByName until the element is found and then returns it, with a timeOut of 10000ms (10 seconds)
UIA_Element.WaitElementExistByType(controlType, scope=0x4, timeOut=10000) ; Calls UIA_Element.FindFirstByType until the element is found and then returns it, with a timeOut of 10000ms (10 seconds)
UIA_Element.WaitElementExistByNameAndType(name, controlType, scope=0x4, timeOut=10000) ; Calls UIA_Element.FindFirstByNameAndType until the element is found and then returns it, with a timeOut of 10000ms (10 seconds)
UIA_Element.Highlight(displayTime:=2000, color:="Red", d:=4) ; Highlights the element borders for 2 seconds by default.
The use of UIA is dependent on many constants/enumerations: all properties, control types, patterns, scopes etc are defined by constants. These can be found under the UIA_Enum class in the UIA_Interface.ahk library, or alternatively inside the UIA_Constants.ahk file. Accessing the constants can be done in multiple different ways:
1) Use the returned object of UIA_Interface:
Code: Select all
UIA := UIA_Interface()
UIA.UIA_InvokePatternId ; returns the corresponding ID of 10000.
UIA.TreeScope_Descendants ; returns 0x4
Calling UIA_Enum methods is also supported:
Code: Select all
UIA.UIA_EventId(20000) ; returns "ToolTipOpened"
Code: Select all
UIA_Enum.UIA_InvokePatternId
UIA_Enum.UIA_PatternId(10000)
3) Use a UIA_Enum object:
Code: Select all
UIAc := new UIA_Enum
UIAc.InvokePatternId ; returns 10000
UIAc.PatternId(10000) ; returns "InvokePattern"
4) Use the UIA_Enum function:
Code: Select all
UIA_Enum("ButtonControlTypeId") would return 50000
Code: Select all
#include <UIA_Constants>
UIA_InvokePatternId ; equal to 10000
Limiting the version of patterns can be done by specifying the exact name of the pattern when creating it. If a full pattern name is specified then that exact version will be used (eg "TextPattern" will return a UIA_TextPattern object), otherwise the highest version will be used (eg "Text" might return UIA_TextPattern2 if it is available).
Downloadable libraries
Libraries themselves if Github is not available:
UIA_Interface.ahk UIA_Browser.ahk OPTIONAL: UIA_Constants.ahk
Edit history:
Code: Select all
06.06.22: Fixed typos in examples. Added example files to Github.
10.06.22: Fixed UIAViewer inconsistencies. Added Acc Path finding option to UIAViewer (note that Acc Path cannot be used with the UIA library, only with Acc library).
11.06.22: Fixed UIAViewer bugs: Esc isn't blocked anymore when capturing. Created a workaround for ElementFromPoint not returning the deepest element. Added error handling when looking at elevated windows (eg Task Manager).
12.06.22: Fixed UIA_Browser.ahk bugs: typo in UIA_Browser.ahk SelectTab method; rewrote GetCurrentMainPaneElement method to avoid looking for the first toolbar (might fail in Chrome).
14.06.22: UIAViewer.ahk now sends WM_GETOBJECT message when appropriate to make Chromium apps content available. Thanks to users malcev and rommmcek for this tip.
16.02.22: modified example 5 to be in accordance with forum rules
19.06.22: MAJOR UPDATE.
1) UIA_Interface was updated to use the latest UIA interface available (currently that is IUIAutomation7), falling back to the highest available version. Notably most applications do not implement methods above IUIAutomation2/CUIAutomation8. To limit the version, call UIA_Interface(maxVersion) (eg UIA_Interface(2) would use IUIAutomation2, or if that fails then IUIAutomation version 1). The current version of UIA_Interface, elements and patterns is available under __Version property. To limit the maximum version of elements, modify UIA_Enum.UIA_CurrentVersion_Element property accordingly.
2) UIA_Constants.ahk was made optional, all constants and enumerations are now available under UIA_Enum class.
3) Examples were updated to according to the changes.
4) Most patterns are now implemented.
5) Event handlers support was added (with two examples: Example 7 and 8).
6) ElementFromHandle and ElementFromPoint were added arguments to send the WM_GETOBJECT message to Chromium apps, which otherwise might not show any content.
7) GetCurrentValue() was replaced with CurrentValue property (also supports setting the value).
8) Many smaller bugfixes and improvements.
21.06.22 fixed UIA_Interface FindAllByType. Improved UIA_Browser error-handling.
23.06.22 updated UIAViewer to show AutomationId, and ControlType is shown in text form as well. Fixed a bug in UIA_Browser where GetCurrentMainPaneElement didn't get URL edit element if the URL bar was not found by name. Fixed bugs in UIA_Interface: case-sensitivity with matchMode=1 now works; ScrollPattern CurrentHorizontalViewSize and CurrentHorizontallyScrollable now work. Added Pattern examples 9-16.
29.06.22 many bug fixes: FindFirstBy with NOT condition now works properly when matchMode!=3. Fixed CompareRuntimeIds() and GetRuntimeId(). Fixed Click() methods use of ExpandCollapsePattern. Improved comments with references to Microsoft documentation. Added SmallestElementFromPoint() method to UIA_Interface class. Added new properties/methods to UIA_Element: CurrentExists, WaitNotExist(), WaitElementNotExist(). Added TextPattern, TextRange, and TextChangedEvent examples 17-18.
04.07.22 UIA_Interface fixes: fixed memory leaks (objects and BSTR-s not being released), rewrote variant handling, fixed VT_BOOL to be interpreted correctly (before fix: -1=True, 0=False, 1=False).
UIA_Browser changes: FindTabByName() was named to GetTab() and it can additionally return the currently selected tab. Added JSGetElementPos, JSClickElement, ControlClickJSElement and ClickJSElement methods. Changed caseSensitive default values to True to be more consistent with UIA_Interface. UIA_Browser can now be created with a custom object customNames, that defines custom CurrentName values for locale-specific elements (such as the name of the URL bar)
10.07.22 UIA_Interface updates: UIA_Element.FindByPath can now get parent and siblings as well; UIA_Element.Dump() displays also the ControlType in English, and values of "0" are displayed; improved activateChromiumAccessibility flag for ElementFromHandle and ElementFromPoint; UIA_Element.GetClickablePoint now returns a value.
UIA_Browser updates: WaitPageLoad better implementation; ControlSend releases modifier keys
UIA_Viewer updates: listview sizes can now be changed by dragging
15.07.22 UIA_Interface updates: UIA_Interface.CreateCondition now supports expressions (parenthesis are allowed). UIA_Element.FindFirstBy and FindAllBy support parenthesis if matchMode is 2 or 3. UIA_Element.Highlight() highlights the element borders (for debugging purposes). Properties have been rewritten, now Current... properties can be accessed without the "Current" part (eg UIA_Element.Value). For some properties, setting values is supported. Patterns can be accessed like properties: UIA_Element.ValuePattern gets UIA_ValuePattern (note that the highest version of the pattern won't be selected). UIA_Element.Click() and UIA_Element.ControlClick() first or second arguments can be a "sleep after" value.
UIA_Browser updates: Added Navigate() method.
24.07.22 UIA_Interface updates: fixed EventHandlerGroups, and multiple EventHandlers can be created at the same time. enableChromiumAccessibility is now by default True. SPI_SCREENREADER flag is set on UIA_Interface() calls. Multiple bugfixes to caching. UIA_Browser updates: closing tabs is no longer locale-dependent.
UIAViewer updates: fixed bug where custom elements with empty names are not shown.
09.09.22 UIA_Interface updates: Added ElementFromChromium. ActivateChromiumAccessibility is now byref (which will return the Chromium element on the first call of ElementFrom... methods).
UIA_Browser updates: added some support for Firefox. Separated classes for different browsers for easier add-ons.
11.09.22 UIA_Interface update: Fixed ElementFromChromium crashing Chrome. UIA_Browser update: now waits for browser to fully load before hooking elements.
08.10.22 UIA_Interface update: Fixed FindAllByNameAndType. UIA_CreateEventHandler accepts objects as funcName.