UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

UIAutomation with a focus on Chrome

Post by Descolada » 06 Jun 2022, 09:11

Preamble: Recently I've been working on a project involving browser (specifically Chrome) automation, with the limitation that external programs are not allowed (such as Rufaydium). This led me to the UIAutomation framework, for which I discovered a few AHK libraries already existed. The most popular one for AHK1 seemed to be jethrow's UIA_Interface ( https://github.com/jethrow/UIA_Interface/blob/master/UIA_Interface.ahk ), which unfortunately seems to be abandoned at the moment. Still, after getting over some initial hurdles (such as it not working in 32-bit mode), I got it working and bit-by-bit added some additional methods and helper functions. Perhaps somebody else is also interested in using or improving it. Please don't hate too much on the code, since I am quite unfamiliar with dll calls, variants etc... :)

The library is available here: https://github.com/Descolada/UIAutomation

Learn more about UIAutomation from the Wiki

Instructional videos from Joe the Automator

It consists of two (three?) main files:
1) UIA_Interface.ahk, which is based on jethrow's project, and contains wrapper functions for the UIAutomation framework. In addition it has some helper functions for easier use of UIA.
2) UIA_Browser.ahk, which contains helper functions for Chrome (and Edge mostly too) automation (fetching URLs, switching tabs etc).
3) OPTIONAL UIA_Constants.ahk, which contains constants to use UIA. All the constants are also available in UIA_Interface.ahk, but the format of constants in UIA_Constants.ahk might be preferrable to some people. Note that this file creates a lot of global variables, so make sure your script doesn't change any of these during runtime. Credit: UIA_Constants file is mostly based on an Autoit3 project by LarsJ.

Short intro for UIAutomation:
https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-overview
UIAutomation is a Microsoft accessibility framework that allows interacting with user-accessible elements (clicking buttons, sending text etc) in a programmatic way. It is the successor to the Acc/MSAA framework.

UIAutomation framework holds the entire desktop with all the windows and controls inside a tree of AutomationElement objects. The root element is the Desktop itself, its children are all the windows (Notepad, Chrome, etc), then in turn the windows have child elements (toolbars, main content, buttons, etc) and so on.

To interact with windows and controls, usually we need to first get the AutomationElement to that window/control. This can be done with functions from UIA_Interface.ahk UIA_Interface class, such as getting a window element from an ahk_id handle with ElementFromHandle, or from a point with ElementFromPoint. We can also get the root element (Desktop) with GetRootElement.
Once an element is found, one way that sub-elements can be looked for is using FindFirst or FindAll functions, that take a specific condition (such as the control type needs to be "edit", or name must be "something") and return the first matching element (FindFirst) or all matching elements (FindAll). Conditions can be created by using Create...Condition methods (CreatePropertyCondition, CreateTrueCondition, etc) and combined with using CreateAndCondition, CreateOrCondition. One of the more commonly used condition method is CreatePropertyCondition, which takes a property id (these can be found under UIA_Enum class under PropertyIds, or in UIA_Constants.ahk file under UIA_PropertyIds) and the desired property value. For example, to create a condition that only looks for button elements, first look up the corresponding PropertyId in the UIA_Enum class ("button" is a control type, so it would be UIA_ControlTypePropertyId), then under UIA_ControlTypeIds to find the value for button (UIA_ButtonControlTypeId).
Alternatively the tree can be walked with the TreeWalker class. This can be done with any element (for example after ElementFromHandle or FindFirst methods) and use the TreeWalker class methods such as GetFirstChildElement or GetNextSiblingElement. TreeWalkers can be created with conditions as well to only return certain elements filtered by type, name etc.

Once an element is found, some basic info is accessible with properties such as CurrentName (returns the name of the element), CurrentControlType (returns the type of the element), CurrentIsEnabled etc. These can be found in UIA_Interface.ahk in the UIA_Element class (properties end with []).
Elements can be interacted with using control patterns. Patterns contain the actual methods to interact with the element, such as InvokePattern to Invoke (click) a button, or ScrollPattern to scroll a control, ValuePattern to get or set the value (such as an edit box). Patterns can be fetched by calling the GetCurrentPatternAs method, then using the methods available in the pattern (for example after calling GetCurrentPatternAs("Invoke") to get a UIA_InvokePattern object, this would be object.Invoke()).

Elements and the element tree can be inspected with various tools. A good tool to get started with is Accessibility Insights which also lets you test out methods of patterns (such as SetValue for ValuePattern). I have also written a AccViewer analog for UIA, the UIAViewer, which can be ran without the UIA_Interface library:
(due to AHK forum constraints this is an older version, please get the latest one from GitHub)

Code: Select all

#SingleInstance Force
#NoEnv
SetWorkingDir %A_ScriptDir%
SetBatchLines -1
CoordMode, Mouse, Screen

global DeepSearchFromPoint := False ; When set to True, UIAViewer iterates through the whole UIA tree to find the smallest element from mouse point. This might be very slow with large trees.

global UIA := UIA_Interface(), IsCapturing := False, Stored := {}, Acc, EnableAccTree := False
Stored.TreeView := {}
Acc_Init()
Acc_Error(1)

_xoffsetfirst := 8
_xoffset := 5
_yoffset := 20
_ysoffset := 2

Gui Main: New, AlwaysOnTop Resize, UIAViewer
Gui Main: Default

Gui Add, GroupBox, x8 y10 w302 h160, Window/Control Info
Gui Add, Text, xm+%_xoffsetfirst% yp+%_yoffset% w30 Section, WinTitle:
Gui Add, Text,, Text:
Gui Add, Edit, ys-%_ysoffset% w235 vEditWinTitle, 
Gui Add, Edit, w235 vEditWinText, 
Gui Add, Text, x18 yp+30 Section, Hwnd:
Gui Add, Text,, Position:
Gui Add, Text,, Size:
Gui Add, Edit, ys-%_ysoffset% w80 vEditWinHwnd, 
Gui Add, Edit, w80 vEditWinPosition,
Gui Add, Edit, w80 vEditWinSize,
Gui Add, Text, ys, ClassNN:
Gui Add, Text,, Process:
Gui Add, Text,, Process ID:
Gui Add, Edit, ys-%_ysoffset% w80 vEditWinClass,
Gui Add, Edit, w80 vEditWinProcess,
Gui Add, Edit, w80 vEditWinProcessID,

Gui Add, GroupBox, x%_xoffsetfirst% y180 w302 h295, UIAutomation Element Info
Gui Add, Text, xm+%_xoffsetfirst% yp+%_yoffset%, ControlType:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w216 vEditControlType,

Gui Add, Text, xm+%_xoffsetfirst% yp+30 Section, Name:
Gui Add, Text,, Value:

Gui Add, Text,, Patterns:
Gui Add, Edit, ys-%_ysoffset% w228 vEditName,
Gui Add, Edit, w228 vEditValue,
Gui Add, Edit, w228 vEditPatterns,

Gui Add, Text, xm+%_xoffsetfirst% yp+30, AutomationId:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w208 vEditAutomationId,

Gui Add, Text, xm+%_xoffsetfirst% yp+30 Section, BoundingRectangle:
Gui Add, Edit, ys-%_ysoffset% w174 vEditBoundingRectangle,

Gui Add, Text, xm+%_xoffsetfirst% yp+30, ClassName:
Gui Add, Edit, x+3 yp-%_ysoffset% w68 vEditClassName,
Gui Add, Text, x+%_xoffset% yp+%_ysoffset%, HelpText:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w97 vEditHelpText,

Gui Add, Text, xm+%_xoffsetfirst% yp+30, AccessKey:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w68 vEditAccessKey,
Gui Add, Text, x+%_xoffset% yp+%_ysoffset%, AcceleratorKey:
Gui Add, Edit, x+%_xoffset% yp-%_ysoffset% w68 vEditAcceleratorKey,

Gui Add, CheckBox, x18 yp+30 Section +Disabled vCBIsKeyboardFocusable, IsKeyboardFocusable
Gui Add, CheckBox, ys +Disabled vCBIsEnabled, IsEnabled

Gui Add, Button, xm+60 yp+30 w150 gButCapture vButCapture, Start capturing
Gui Add, Button, xp+300 yp w192 vButRefreshTreeView gButRefreshTreeView +Disabled, Start capturing to show tree

Gui Add, TreeView, x320 y8 w300 h435 hwndhMainTreeView vMainTreeView gMainTreeView
Gui, Font, Bold
Gui, Add, StatusBar, gMainSB vMainSB
SB_SetText("`tClick here to enable Acc path capturing (can't be used with UIA!)")
Gui, Font
SB_SetParts(380)
SB_SetText("`tCurrent UIA Interface version: " UIA.__Version,2)

Gui Show,, UIAViewer
Return

MainGuiEscape:
MainGuiClose:
	IsCapturing := False
    ExitApp

MainGuiSize(GuiHwnd, EventInfo, Width, Height){
	GuiControlGet, Pos, Pos , MainTreeView
	GuiControl, Move, MainTreeView, % "w" Width -Posx-10 " h" Height -Posy-60
	GuiControl, Move, ButRefreshTreeView, % "y" Height -50
	WinSet, Redraw,,UIAViewer
}

MainSB:
	GuiControlGet, SBText,, MainSB
	if (SBText == "`tClick here to enable Acc path capturing (can't be used with UIA!)") {
		EnableAccTree := True
		SB_SetText("",1)
		SB_SetText("`tClick on path to copy to Clipboard",2)
	} else if SBText {
		Clipboard := SubStr(SBText, 8)
		ToolTip, % "Successfully copied """ SubStr(SBText, 8) """ to Clipboard!"
		SetTimer, RemoveToolTip, -2000
	}
	return

ButCapture:
	if IsCapturing {
		RangeTip()
		IsCapturing := False
		GuiControl, Main:, ButCapture, Start capturing
		GuiControl, Main: Enable, ButCapture
		GuiControl, Main: Enable, ButRefreshTreeView
		GuiControl, Main:, ButRefreshTreeView, Construct tree for whole Window
	} else {
		IsCapturing := True
		GuiControl, Main:, ButCapture, Press Esc to stop capturing
		GuiControl, Main: Disable, ButCapture
		GuiControl, Main: Disable, ButRefreshTreeView
		GuiControl, Main:, ButRefreshTreeView, Hold cursor still to construct tree
		Stored := {}
		
		While (IsCapturing) {
			MouseGetPos, mX, mY, mHwnd, mCtrl
					
			try {
				mEl := UIA.SmallestElementFromPoint(mX, mY,, DeepSearchFromPoint ? UIA.ElementFromHandle(mHwnd) : "")
			} catch e {
				UpdateElementFields()
				GuiControl, Main:, EditName, % "ERROR: " e.Message
				if InStr(e.Message, "0x80070005")
					GuiControl, Main:, EditValue, Try running UIAViewer with Admin privileges
			}
		
			if (mHwnd != Stored.Hwnd) {
				; In some setups Chromium-based renderers don't react to UIA calls by enabling accessibility, so we need to send the WM_GETOBJECT message to the first renderer control for the application to enable accessibility. Thanks to users malcev and rommmcek for this tip. Explanation why this works: https://www.chromium.org/developers/design-documents/accessibility/#TOC-How-Chrome-detects-the-presence-of-Assistive-Technology 
				WinGet, cList, ControlList, ahk_id %mHwnd%
				if InStr(cList, "Chrome_RenderWidgetHostHWND1")
					SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Chrome_RenderWidgetHostHWND1, ahk_id %mHwnd%
				WinGetTitle, wTitle, ahk_id %mHwnd%
				WinGetPos, wX, wY, wW, wH, ahk_id %mHwnd%
				WinGetClass, wClass, ahk_id %mHwnd%
				WinGetText, wText, ahk_id %mHwnd%
				WinGet, wProc, ProcessName, ahk_id %mHwnd%
				WinGet, wProcID, PID, ahk_id %mHwnd%
			
				GuiControl, Main:, EditWinTitle, %wTitle%
				GuiControl, Main:, EditWinText, %wText%
				GuiControl, Main:, EditWinHwnd, ahk_id %mHwnd%
				GuiControl, Main:, EditWinPosition, X: %wX% Y: %wY%
				GuiControl, Main:, EditWinSize, W: %wW% H: %wH%
				GuiControl, Main:, EditWinClass, %wClass%
				GuiControl, Main:, EditWinProcess, %wProc%
				GuiControl, Main:, EditWinProcessID, %wProcID%
			}

			if IsObject(Stored.Element) {
				try {
					if !UIA.CompareElements(mEl, Stored.Element) {
						UpdateElementFields(mEl)
						Stored.TickCount := A_TickCount
						if EnableAccTree {
							oAcc := Acc_ObjectFromPoint(childId, mX, mY), Acc_Location(oAcc,childId,vAccLoc)
							SB_SetText(" Path: " GetAccPathTopDown(mHwnd, vAccLoc), 1)
						}
					} else if (Stored.TickCount && (A_TickCount - Stored.TickCount > 1000)) { ; Wait for mouse to be stable for a second
						Stored.TickCount := 0
						RedrawTreeView(mEl, False)
						for k,v in Stored.TreeView {
							if (v == mEl)
								TV_Modify(k)
						}
					}
				} 
			}

			Stored.Hwnd := mHwnd, Stored.Element := mEl
			Sleep, 200
		}
		
	}
	return

ButRefreshTreeView:
	if (Stored.Hwnd && WinExist("ahk_id" Stored.Hwnd))
		RedrawTreeView(UIA.ElementFromHandle(Stored.Hwnd), True)
	if IsObject(Stored.Element) {
		for k,v in Stored.TreeView
			if UIA.CompareElements(v, Stored.Element)
				TV_Modify(k)
		if EnableAccTree {
			br := Stored.Element.CurrentBoundingRectangle
			GetAccPathTopDown(Stored.Hwnd, "x" br.l " y" br.t " w" (br.r-br.l) " h" (br.b-br.t), True)
		}
	}
	return

MainTreeView:
	if (A_GuiEvent == "S") {
		UpdateElementFields(Stored.Treeview[A_EventInfo])
		if EnableAccTree {
			br := Stored.Treeview[A_EventInfo].CurrentBoundingRectangle
			SB_SetText(" Path: " GetAccPathTopDown(Stored.Hwnd, "x" br.l " y" br.t " w" (br.r-br.l) " h" (br.b-br.t)), 1)
		}
	}
	return

RemoveToolTip:
	ToolTip
	return

UpdateElementFields(mEl="") {
	if !IsObject(mEl) {
		GuiControl, Main:, EditControlType, 
		GuiControl, Main:, EditName,
		GuiControl, Main:, EditValue,
		GuiControl, Main:, EditPatterns,
		GuiControl, Main:, EditAutomationId,
		GuiControl, Main:, EditBoundingRectangle,
		GuiControl, Main:, EditAccessKey,
		GuiControl, Main:, EditAcceleratorKey,
		GuiControl, Main:, EditClassName,
		GuiControl, Main:, EditHelpText,
		GuiControl, Main:, CBIsKeyboardFocusable,
		GuiControl, Main:, CBIsEnabled,
		return
	}
	try {
		mElPos := mEl.CurrentBoundingRectangle
		RangeTip(mElPos.l, mElPos.t, mElPos.r-mElPos.l, mElPos.b-mElPos.t, "Blue", 4)
		GuiControl, Main:, EditControlType, % (ctrlType := mEl.CurrentControlType) " (" UIA_ControlTypeId(ctrlType) ")`t[Localized: " mEl.CurrentLocalizedControlType "]"
		GuiControl, Main:, EditName, % mEl.CurrentName
		GuiControl, Main:, EditValue, % mEl.GetCurrentPropertyValue(UIA_ValueValuePropertyId := 30045)
		patterns := ""
		for k, v in UIA.PollForPotentialSupportedPatterns(mEl)
			patterns .= ", " RegexReplace(k, "Pattern$")
		GuiControl, Main:, EditPatterns, % SubStr(patterns, 3)
		GuiControl, Main:, EditAutomationId, % mEl.CurrentAutomationId
		GuiControl, Main:, EditBoundingRectangle, % "l: " mElPos.l " t: " mElPos.t " r: " mElPos.r " b: " mElPos.b
		GuiControl, Main:, EditAccessKey, % mEl.CurrentAccessKey
		GuiControl, Main:, EditAcceleratorKey, % mEl.CurrentAcceleratorKey
		GuiControl, Main:, EditClassName, % mEl.CurrentClassName
		GuiControl, Main:, EditHelpText, % mEl.CurrentHelpText
		GuiControl, Main:, CBIsKeyboardFocusable, % mEl.CurrentIsKeyboardFocusable
		GuiControl, Main:, CBIsEnabled, % mEl.CurrentIsEnabled
	}
	return
}

RedrawTreeView(el, noAncestors=True) {
	global MainTreeView, hMainTreeView
	TV_Delete()
	TV_Add("Constructing TreeView, do not move the mouse...")
	GuiControl, Main: -Redraw, MainTreeView
	Gui, TreeView, MainTreeView
	TV_Delete()
	Stored.TreeView := {}
	if noAncestors {
		ConstructTreeView(el)
	} else {
		; Get all ancestors
		ancestors := [], parent := el
		while IsObject(parent) {
			try {
				if IsObject(parent := UIA.TreeWalkerTrue.GetParentElement(parent))
					ancestors.Push(parent)
			} catch {
				break
			}
		}
		
		; Loop backwards through ancestors to create the TreeView
		maxInd := ancestors.MaxIndex(), parent := ""
		while (--maxInd) {
			if !IsObject(ancestors[maxInd])
				return
			try {
				elDesc := ancestors[maxInd].CurrentLocalizedControlType " """ ancestors[maxInd].CurrentName """"
				if (elDesc == " """"")
					break
				Stored.TreeView[parent := TV_Add(elDesc, parent)] := ancestors[maxInd]
			}
		}
		; Add child elements to TreeView also
		ConstructTreeView(el, parent)
	}
	for k,v in Stored.TreeView
		TV_Modify(k, "Expand")
	
	SendMessage, 0x115, 6, 0,, ahk_id %hMainTreeView% ; scroll to top
	GuiControl, Main: +Redraw, MainTreeView
}

ConstructTreeView(el, parent="") {
	if !IsObject(el)
		return
	try {
		elDesc := el.CurrentLocalizedControlType " """ el.CurrentName """"
		if (elDesc == " """"")
			return
		Stored.TreeView[TWEl := TV_Add(elDesc, parent)] := el
		if !(children := el.FindAll(UIA.TrueCondition, 0x2))
			return
		for k, v in children
			ConstructTreeView(v, TWEl)
	}
}

; UIA FUNCTIONS
class UIA_Base {
	__New(p="", flag=1, ver=1) {
		ObjInsert(this,"__Type","IUIAutomation" SubStr(thisClass,5))
		,ObjInsert(this,"__Value",p)
		,ObjInsert(this,"__Flag",flag)
		,ObjInsert(this,"__Version",ver)
	}
	__Get(member) {
		if member not in base,__UIA,TreeWalkerTrue,TrueCondition ; base & __UIA should act as normal
		{	if raw:=SubStr(member,0)="*" ; return raw data - user should know what they are doing
				member:=SubStr(member,1,-1)
			if RegExMatch(this.__properties, "im)^" member ",(\d+),(\w+)", m) { ; if the member is in the properties. if not - give error message
				if (m2="VARIANT")	; return VARIANT data - DllCall output param different
					return UIA_Hr(DllCall(this.__Vt(m1), "ptr",this.__Value, "ptr",UIA_Variant(out)))? (raw?out:UIA_VariantData(out)):
				else if (m2="RECT") ; return RECT struct - DllCall output param different
					return UIA_Hr(DllCall(this.__Vt(m1), "ptr",this.__Value, "ptr",&(rect,VarSetCapacity(rect,16))))? (raw?out:UIA_RectToObject(rect)):
				else if UIA_Hr(DllCall(this.__Vt(m1), "ptr",this.__Value, "ptr*",out))
					return raw?out:m2="BSTR"?StrGet(out):RegExMatch(m2,"i)IUIAutomation\K\w+",n)?new UIA_%n%(out):out ; Bool, int, DWORD, HWND, CONTROLTYPEID, OrientationType?
			}
			else throw Exception("Property not supported by the " this.__Class " Class.",-1,member)
		}
	}
	__Delete() {
		this.__Flag ? ObjRelease(this.__Value):
	}
	__Vt(n) {
		return NumGet(NumGet(this.__Value+0,"ptr")+n*A_PtrSize,"ptr")
	}
}

class UIA_Interface extends UIA_Base {
	;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671406(v=vs.85).aspx
	static __IID := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}"
		,  __properties := ""
	CompareElements(e1,e2) {
		return UIA_Hr(DllCall(this.__Vt(3), "ptr",this.__Value, "ptr",e1.__Value, "ptr",e2.__Value, "int*",out))? out:
	}
	ElementFromHandle(hwnd) {
		return UIA_Hr(DllCall(this.__Vt(6), "ptr",this.__Value, "ptr",hwnd, "ptr*",out))? new UIA_Element(out):
	}
	ElementFromPoint(x="", y="", activateChromiumAccessibility=False) {
		return UIA_Hr(DllCall(this.__Vt(7), "ptr",this.__Value, "UInt64",x==""||y==""?DllCall("GetCursorPos","Int64*",pt)*0+pt:x&0xFFFFFFFF|(y&0xFFFFFFFF)<<32, "ptr*",out))? new UIA_Element(out):
	}	
	CreateTreeWalker(condition) {
		return UIA_Hr(DllCall(this.__Vt(13), "ptr",this.__Value, "ptr",Condition.__Value, "ptr*",out))? new UIA_TreeWalker(out):
	}
	CreateTrueCondition() {
		return UIA_Hr(DllCall(this.__Vt(21), "ptr",this.__Value, "ptr*",out))? new UIA_BoolCondition(out):
	}
	CreatePropertyCondition(propertyId, var, type="Variant") {
		if (type!="Variant")
			UIA_Variant(var,type,var)
		return UIA_Hr((A_PtrSize == 4) ? DllCall(this.__Vt(23), "ptr",this.__Value, "int",propertyId, "int64", NumGet(var, 0, "int64"), "int64", NumGet(var, 8, "int64"), "ptr*",out) : DllCall(this.__Vt(23), "ptr",this.__Value, "int",propertyId, "ptr",&var, "ptr*",out))? new UIA_PropertyCondition(out):
	}
	PollForPotentialSupportedPatterns(e, Byref Ids="", Byref Names="") { ; Returns an object where keys are the names and values are the Ids
		return UIA_Hr(DllCall(this.__Vt(51), "ptr",this.__Value, "ptr",e.__Value, "ptr*",Ids, "ptr*",Names))? UIA_SafeArraysToObject(Names:=ComObj(0x2008,Names,1),Ids:=ComObj(0x2003,Ids,1)):
	}
	; Gets ElementFromPoint and filters out the smallest subelement that is under the specified point. If windowEl (window under the point) is provided, then a deep search is performed for the smallest element (this might be very slow in large trees).
	SmallestElementFromPoint(x="", y="", activateChromiumAccessibility=False, windowEl="") { 
		if IsObject(windowEl) {
			element := this.ElementFromPoint(x, y, activateChromiumAccessibility)
			bound := element.CurrentBoundingRectangle, elementSize := (bound.r-bound.l)*(bound.b-bound.t), prevElementSize := 0, stack := [windowEl]
			Loop 
			{
				bound := stack[1].CurrentBoundingRectangle
				if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b)) { ; If parent is not in bounds, then children arent either
					if ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize)
						element := stack[1], elementSize := newSize
					for _, childEl in stack[1].FindAll(this.__UIA.TrueCondition, 0x2) {
						bound := childEl.CurrentBoundingRectangle
						if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b)) {
							stack.Push(childEl)
							if ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize)
								elementSize := newSize, element := childEl
						}
					}
				}
				stack.RemoveAt(1)
			} Until !stack.MaxIndex()
			return element
		} else {
			element := this.ElementFromPoint(x, y, activateChromiumAccessibility)
			bound := element.CurrentBoundingRectangle, elementSize := (bound.r-bound.l)*(bound.b-bound.t), prevElementSize := 0
			for k, v in element.FindAll(this.__UIA.TrueCondition) {
				bound := v.CurrentBoundingRectangle
				if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b) && ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize))
					element := v, elementSize := newSize
			}
			return element
		}
	}
}
class UIA_Element extends UIA_Base {
	;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671425(v=vs.85).aspx
	static __IID := "{d22108aa-8ac5-49a5-837b-37bbb3d7591e}"
		,  __properties := "CurrentProcessId,20,int`r`nCurrentControlType,21,CONTROLTYPEID`r`nCurrentLocalizedControlType,22,BSTR`r`nCurrentName,23,BSTR`r`nCurrentAcceleratorKey,24,BSTR`r`nCurrentAccessKey,25,BSTR`r`nCurrentHasKeyboardFocus,26,BOOL`r`nCurrentIsKeyboardFocusable,27,BOOL`r`nCurrentIsEnabled,28,BOOL`r`nCurrentAutomationId,29,BSTR`r`nCurrentClassName,30,BSTR`r`nCurrentHelpText,31,BSTR`r`nCurrentCulture,32,int`r`nCurrentIsControlElement,33,BOOL`r`nCurrentIsContentElement,34,BOOL`r`nCurrentIsPassword,35,BOOL`r`nCurrentNativeWindowHandle,36,UIA_HWND`r`nCurrentItemType,37,BSTR`r`nCurrentIsOffscreen,38,BOOL`r`nCurrentOrientation,39,OrientationType`r`nCurrentFrameworkId,40,BSTR`r`nCurrentIsRequiredForForm,41,BOOL`r`nCurrentItemStatus,42,BSTR`r`nCurrentBoundingRectangle,43,RECT`r`nCurrentLabeledBy,44,IUIAutomationElement`r`nCurrentAriaRole,45,BSTR`r`nCurrentAriaProperties,46,BSTR`r`nCurrentIsDataValidForForm,47,BOOL`r`nCurrentControllerFor,48,IUIAutomationElementArray`r`nCurrentDescribedBy,49,IUIAutomationElementArray`r`nCurrentFlowsTo,50,IUIAutomationElementArray`r`nCurrentProviderDescription,51,BSTR"
	FindAll(c="", scope=0x4) {
		return UIA_Hr(DllCall(this.__Vt(6), "ptr",this.__Value, "uint",scope, "ptr",(c=""?this.TrueCondition:c).__Value, "ptr*",out))&&out? UIA_ElementArray(out):
	}
	GetCurrentPropertyValue(propertyId, ByRef out="") {
		return UIA_Hr(DllCall(this.__Vt(10), "ptr",this.__Value, "uint", propertyId, "ptr",UIA_Variant(out)))? UIA_VariantData(out):
		
	}
	GetSupportedPatterns() { ; Get all available patterns for the element. Use of this should be avoided, since it calls GetCurrentPatternAs for every possible pattern.
		result := []
		patterns := "Invoke,Selection,Value,RangeValue,Scroll,ExpandCollapse,Grid,GridItem,MultipleView,Window,SelectionItem,Dock,Table,TableItem,Text,Toggle,Transform,ScrollItem,ItemContainer,VirtualizedItem,SyncronizedInput,LegacyIAccessible"

		Loop, Parse, patterns, `,
		{
			try {
				if this.GetCurrentPropertyValue(UIA_PropertyId("Is" A_LoopField "PatternAvailable")) {
					result.Push(A_LoopField)
				}
			}
		}
		return result
	}
}
class UIA_ElementArray extends UIA_Base {
	static __IID := "{14314595-b4bc-4055-95f2-58f2e42c9855}"
		,  __properties := "Length,3,int"
	
	GetElement(i) {
		return UIA_Hr(DllCall(this.__Vt(4), "ptr",this.__Value, "int",i, "ptr*",out))? new UIA_Element(out):
	}
}

class UIA_TreeWalker extends UIA_Base {
	;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671470(v=vs.85).aspx
	static __IID := "{4042c624-389c-4afc-a630-9df854a541fc}"
		,  __properties := "Condition,15,IUIAutomationCondition"
	
	GetParentElement(e) {
		return UIA_Hr(DllCall(this.__Vt(3), "ptr",this.__Value, "ptr",e.__Value, "ptr*",out))&&out? new UIA_Element(out):
	}
	GetFirstChildElement(e) {
		return UIA_Hr(DllCall(this.__Vt(4), "ptr",this.__Value, "ptr",e.__Value, "ptr*",out))&&out? new UIA_Element(out):
	}
	GetLastChildElement(e) {
		return UIA_Hr(DllCall(this.__Vt(5), "ptr",this.__Value, "ptr",e.__Value, "ptr*",out))&&out? new UIA_Element(out):
	}
	GetNextSiblingElement(e) {
		return UIA_Hr(DllCall(this.__Vt(6), "ptr",this.__Value, "ptr",e.__Value, "ptr*",out))&&out? new UIA_Element(out):
	}
}

class UIA_Condition extends UIA_Base {
	;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671420(v=vs.85).aspx
	static __IID := "{352ffba8-0973-437c-a61f-f64cafd81df9}"
		,  __properties := ""
}
class UIA_PropertyCondition extends UIA_Condition {
	;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee696121(v=vs.85).aspx
	static __IID := "{99ebf2cb-5578-4267-9ad4-afd6ea77e94b}"
		,  __properties := "PropertyId,3,PROPERTYID`r`nPropertyValue,4,VARIANT`r`nPropertyConditionFlags,5,PropertyConditionFlags"
}
class UIA_BoolCondition extends UIA_Condition {
	;~ http://msdn.microsoft.com/en-us/library/windows/desktop/ee671411(v=vs.85).aspx
	static __IID := "{1B4E1F2E-75EB-4D0B-8952-5A69988E2307}"
		,  __properties := "BooleanValue,3,boolVal"
}
UIA_Interface() {
	max := 7+1
	for k, v in ["{29de312e-83c6-4309-8808-e8dfcb46c3c2}","{aae072da-29e3-413d-87a7-192dbf81ed10}","{25f700c8-d816-4057-a9dc-3cbdee77e256}","{1189c02a-05f8-4319-8e21-e817e3db2860}","{73d768da-9b51-4b89-936e-c209290973e7}","{34723aff-0c9d-49d0-9896-7ab52df8cd8a}"] {
		try {
			if uia:=ComObjCreate("{e22ad333-b25f-460c-83d0-0581107395c9}",v)
				return uia:=new UIA_Interface(uia, 1, max-k), uia.base.base.__UIA:=uia, uia.base.base.TrueCondition:=uia.CreateTrueCondition(), uia.base.base.TreeWalkerTrue := uia.CreateTreeWalker(uia.base.base.TrueCondition)
		}
	}
	; If all else fails, try the first UIAutomation version
	try {
		if uia:=ComObjCreate("{ff48dba4-60ef-4201-aa87-54103eef594e}","{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
			return uia:=new UIA_Interface(uia, 1), uia.base.base.__UIA:=uia, uia.base.base.TrueCondition:=uia.CreateTrueCondition(), uia.base.base.TreeWalkerTrue := uia.CreateTreeWalker(uia.base.base.TrueCondition)
		throw "UIAutomation Interface failed to initialize."
	} catch e
		MsgBox, 262160, UIA Startup Error, % IsObject(e)?"IUIAutomation Interface is not registered.":e.Message
	return
}
UIA_Hr(hr) {
	;~ http://blogs.msdn.com/b/eldar/archive/2007/04/03/a-lot-of-hresult-codes.aspx
	static err:={0x8000FFFF:"Catastrophic failure.",0x80004001:"Not implemented.",0x8007000E:"Out of memory.",0x80070057:"One or more arguments are not valid.",0x80004002:"Interface not supported.",0x80004003:"Pointer not valid.",0x80070006:"Handle not valid.",0x80004004:"Operation aborted.",0x80004005:"Unspecified error.",0x80070005:"General access denied.",0x800401E5:"The object identified by this moniker could not be found.",0x80040201:"UIA_E_ELEMENTNOTAVAILABLE",0x80040200:"UIA_E_ELEMENTNOTENABLED",0x80131509:"UIA_E_INVALIDOPERATION",0x80040202:"UIA_E_NOCLICKABLEPOINT",0x80040204:"UIA_E_NOTSUPPORTED",0x80040203:"UIA_E_PROXYASSEMBLYNOTLOADED"} ; //not completed
	if hr&&(hr&=0xFFFFFFFF) {
		RegExMatch(Exception("",-2).what,"(\w+).(\w+)",i)
		throw Exception(UIA_Hex(hr) " - " err[hr], -2, i2 "  (" i1 ")")
	}
	return !hr
}
UIA_Hex(p) {
	setting:=A_FormatInteger
	SetFormat,IntegerFast,H
	out:=p+0 ""
	SetFormat,IntegerFast,%setting%
	return out
}
UIA_GUID(ByRef GUID, sGUID) { ;~ Converts a string to a binary GUID and returns its address.
	VarSetCapacity(GUID,16,0)
	return DllCall("ole32\CLSIDFromString", "wstr",sGUID, "ptr",&GUID)>=0?&GUID:""
}
UIA_ElementArray(p, uia="") { ; should AHK Object be 0 or 1 based? /// answer: 1 based ///
	if !p
		return
	a:=new UIA_ElementArray(p),out:=[]
	Loop % a.Length
		out[A_Index]:=a.GetElement(A_Index-1)
	return out, out.base:={UIA_ElementArray:a}
}
UIA_Variant(ByRef var,type=0,val=0) {
	; Does a variant need to be cleared? If it uses SysAllocString? 
	return (VarSetCapacity(var,8+2*A_PtrSize)+NumPut(type,var,0,"short")+NumPut(type=8? DllCall("oleaut32\SysAllocString", "ptr",&val):val,var,8,"ptr"))*0+&var
}
UIA_IsVariant(ByRef vt, ByRef type="") {
	size:=VarSetCapacity(vt),type:=NumGet(vt,"UShort")
	return size>=16&&size<=24&&type>=0&&(type<=23||type|0x2000)
}
UIA_VariantData(ByRef p, flag=1) {
	return !UIA_IsVariant(p,vt)?"Invalid Variant"
			:vt=3?NumGet(p,8,"int")
			:vt=8?StrGet(NumGet(p,8))
			:vt=9||vt=13||vt&0x2000?ComObj(vt,NumGet(p,8),flag)
			:vt<0x1000&&UIA_VariantChangeType(&p,&p)=0?StrGet(NumGet(p,8)) UIA_VariantClear(&p)
			:NumGet(p,8)
}
UIA_VariantChangeType(pvarDst, pvarSrc, vt=8) { ; written by Sean
	return DllCall("oleaut32\VariantChangeTypeEx", "ptr",pvarDst, "ptr",pvarSrc, "Uint",1024, "Ushort",0, "Ushort",vt)
}
UIA_VariantClear(pvar) { ; Written by Sean
	DllCall("oleaut32\VariantClear", "ptr",pvar)
}
UIA_SafeArraysToObject(keys,values) {
;~	1 dim safearrays w/ same # of elements
	out:={}
	for key in keys
		out[key]:=values[A_Index-1]
	return out
}
UIA_RectToObject(ByRef r) { ; rect.__Value work with DllCalls?
	static b:={__Class:"object",__Type:"RECT",Struct:Func("UIA_RectStructure")}
	return {l:NumGet(r,0,"Int"),t:NumGet(r,4,"Int"),r:NumGet(r,8,"Int"),b:NumGet(r,12,"Int"),base:b}
}
UIA_RectStructure(this, ByRef r) {
	static sides:="ltrb"
	VarSetCapacity(r,16)
	Loop Parse, sides
		NumPut(this[A_LoopField],r,(A_Index-1)*4,"Int")
}
UIA_PropertyId(n="") {
	static ids:="RuntimeId:30000,BoundingRectangle:30001,ProcessId:30002,ControlType:30003,LocalizedControlType:30004,Name:30005,AcceleratorKey:30006,AccessKey:30007,HasKeyboardFocus:30008,IsKeyboardFocusable:30009,IsEnabled:30010,AutomationId:30011,ClassName:30012,HelpText:30013,ClickablePoint:30014,Culture:30015,IsControlElement:30016,IsContentElement:30017,LabeledBy:30018,IsPassword:30019,NativeWindowHandle:30020,ItemType:30021,IsOffscreen:30022,Orientation:30023,FrameworkId:30024,IsRequiredForForm:30025,ItemStatus:30026,IsDockPatternAvailable:30027,IsExpandCollapsePatternAvailable:30028,IsGridItemPatternAvailable:30029,IsGridPatternAvailable:30030,IsInvokePatternAvailable:30031,IsMultipleViewPatternAvailable:30032,IsRangeValuePatternAvailable:30033,IsScrollPatternAvailable:30034,IsScrollItemPatternAvailable:30035,IsSelectionItemPatternAvailable:30036,IsSelectionPatternAvailable:30037,IsTablePatternAvailable:30038,IsTableItemPatternAvailable:30039,IsTextPatternAvailable:30040,IsTogglePatternAvailable:30041,IsTransformPatternAvailable:30042,IsValuePatternAvailable:30043,IsWindowPatternAvailable:30044,ValueValue:30045,ValueIsReadOnly:30046,RangeValueValue:30047,RangeValueIsReadOnly:30048,RangeValueMinimum:30049,RangeValueMaximum:30050,RangeValueLargeChange:30051,RangeValueSmallChange:30052,ScrollHorizontalScrollPercent:30053,ScrollHorizontalViewSize:30054,ScrollVerticalScrollPercent:30055,ScrollVerticalViewSize:30056,ScrollHorizontallyScrollable:30057,ScrollVerticallyScrollable:30058,SelectionSelection:30059,SelectionCanSelectMultiple:30060,SelectionIsSelectionRequired:30061,GridRowCount:30062,GridColumnCount:30063,GridItemRow:30064,GridItemColumn:30065,GridItemRowSpan:30066,GridItemColumnSpan:30067,GridItemContainingGrid:30068,DockDockPosition:30069,ExpandCollapseExpandCollapseState:30070,MultipleViewCurrentView:30071,MultipleViewSupportedViews:30072,WindowCanMaximize:30073,WindowCanMinimize:30074,WindowWindowVisualState:30075,WindowWindowInteractionState:30076,WindowIsModal:30077,WindowIsTopmost:30078,SelectionItemIsSelected:30079,SelectionItemSelectionContainer:30080,TableRowHeaders:30081,TableColumnHeaders:30082,TableRowOrColumnMajor:30083,TableItemRowHeaderItems:30084,TableItemColumnHeaderItems:30085,ToggleToggleState:30086,TransformCanMove:30087,TransformCanResize:30088,TransformCanRotate:30089,IsLegacyIAccessiblePatternAvailable:30090,LegacyIAccessibleChildId:30091,LegacyIAccessibleName:30092,LegacyIAccessibleValue:30093,LegacyIAccessibleDescription:30094,LegacyIAccessibleRole:30095,LegacyIAccessibleState:30096,LegacyIAccessibleHelp:30097,LegacyIAccessibleKeyboardShortcut:30098,LegacyIAccessibleSelection:30099,LegacyIAccessibleDefaultAction:30100,AriaRole:30101,AriaProperties:30102,IsDataValidForForm:30103,ControllerFor:30104,DescribedBy:30105,FlowsTo:30106,ProviderDescription:30107,IsItemContainerPatternAvailable:30108,IsVirtualizedItemPatternAvailable:30109,IsSynchronizedInputPatternAvailable:30110,OptimizeForVisualContent:30111,IsObjectModelPatternAvailable:30112,AnnotationAnnotationTypeId:30113,AnnotationAnnotationTypeName:30114,AnnotationAuthor:30115,AnnotationDateTime:30116,AnnotationTarget:30117,IsAnnotationPatternAvailable:30118,IsTextPattern2Available:30119,StylesStyleId:30120,StylesStyleName:30121,StylesFillColor:30122,StylesFillPatternStyle:30123,StylesShape:30124,StylesFillPatternColor:30125,StylesExtendedProperties:30126,IsStylesPatternAvailable:30127,IsSpreadsheetPatternAvailable:30128,SpreadsheetItemFormula:30129,SpreadsheetItemAnnotationObjects:30130,SpreadsheetItemAnnotationTypes:30131,IsSpreadsheetItemPatternAvailable:30132,Transform2CanZoom:30133,IsTransformPattern2Available:30134,LiveSetting:30135,IsTextChildPatternAvailable:30136,IsDragPatternAvailable:30137,DragIsGrabbed:30138,DragDropEffect:30139,DragDropEffects:30140,IsDropTargetPatternAvailable:30141,DropTargetDropTargetEffect:30142,DropTargetDropTargetEffects:30143,DragGrabbedItems:30144,Transform2ZoomLevel:30145,Transform2ZoomMinimum:30146,Transform2ZoomMaximum:30147,FlowsFrom:30148,IsTextEditPatternAvailable:30149,IsPeripheral:30150,IsCustomNavigationPatternAvailable:30151,PositionInSet:30152,SizeOfSet:30153,Level:30154,AnnotationTypes:30155,AnnotationObjects:30156,LandmarkType:30157,LocalizedLandmarkType:30158,FullDescription:30159,FillColor:30160,OutlineColor:30161,FillType:30162,VisualEffects:30163,OutlineThickness:30164,CenterPoint:30165,Rotation:30166,Size:30167,IsSelectionPattern2Available:30168,Selection2FirstSelectedItem:30169,Selection2LastSelectedItem:30170,Selection2CurrentSelectedItem:30171,Selection2ItemCount:30173,IsDialog:30174"
	if !n
		return ids		
	if n is integer 
	{
		RegexMatch(ids, "([^,]+):" n, m)
		return m1
	}
	
	n := StrReplace(StrReplace(n, "UIA_"), "PropertyId")
	RegexMatch(ids, "(?:^|,)" n "(?:" n ")?(?:Id)?:(\d+)", m)
	return m1
}

UIA_ControlTypeId(n="") {
	static id:={Button:50000,Calendar:50001,CheckBox:50002,ComboBox:50003,Edit:50004,Hyperlink:50005,Image:50006,ListItem:50007,List:50008,Menu:50009,MenuBar:50010,MenuItem:50011,ProgressBar:50012,RadioButton:50013,ScrollBar:50014,Slider:50015,Spinner:50016,StatusBar:50017,Tab:50018,TabItem:50019,Text:50020,ToolBar:50021,ToolTip:50022,Tree:50023,TreeItem:50024,Custom:50025,Group:50026,Thumb:50027,DataGrid:50028,DataItem:50029,Document:50030,SplitButton:50031,Window:50032,Pane:50033,Header:50034,HeaderItem:50035,Table:50036,TitleBar:50037,Separator:50038,SemanticZoom:50039,AppBar:50040}, name:={50000:"Button",50001:"Calendar",50002:"CheckBox",50003:"ComboBox",50004:"Edit",50005:"Hyperlink",50006:"Image",50007:"ListItem",50008:"List",50009:"Menu",50010:"MenuBar",50011:"MenuItem",50012:"ProgressBar",50013:"RadioButton",50014:"ScrollBar",50015:"Slider",50016:"Spinner",50017:"StatusBar",50018:"Tab",50019:"TabItem",50020:"Text",50021:"ToolBar",50022:"ToolTip",50023:"Tree",50024:"TreeItem",50025:"Custom",50026:"Group",50027:"Thumb",50028:"DataGrid",50029:"DataItem",50030:"Document",50031:"SplitButton",50032:"Window",50033:"Pane",50034:"Header",50035:"HeaderItem",50036:"Table",50037:"TitleBar",50038:"Separator",50039:"SemanticZoom",50040:"AppBar"}
	if !n
		return id
	if n is integer
		return name[n]
	if ObjHasKey(id, n)
		return id[n]
	return id[StrReplace(StrReplace(n, "ControlTypeId"), "UIA_")]
}

; Acc functions

GetAccPathTopDown(hwnd, vAccPos, updateTree=False) {
	static accTree
	if !IsObject(accTree)
		accTree := {}
	if (!IsObject(accTree[hwnd]) || updateTree)
		accTree[hwnd] := BuildAccTreeRecursive(Acc_ObjectFromWindow(hwnd, 0), {})
	for k, v in accTree[hwnd] {
		if (v == vAccPos)
			return k
	}
}

BuildAccTreeRecursive(oAcc, tree, path="") {
	if !IsObject(oAcc)
		return tree
	try 
		oAcc.accChildCount
	catch
		return tree
	For i, oChild in Acc_Children(oAcc) {
		if IsObject(oChild)
			Acc_Location(oChild,,vChildPos)
		else
			Acc_Location(oAcc,oChild,vChildPos)
		tree[path (path?(IsObject(oChild)?".":" c"):"") i] := vChildPos
		tree := BuildAccTreeRecursive(oChild, tree, path (path?".":"") i)
	}
	return tree
}
Acc_Init()
{
	Static	h
	If Not	h
	h:=DllCall("LoadLibrary","Str","oleacc","Ptr")
}
Acc_ObjectFromEvent(ByRef _idChild_, hWnd, idObject, idChild)
{
	Acc_Init()
	If	DllCall("oleacc\AccessibleObjectFromEvent", "Ptr", hWnd, "UInt", idObject, "UInt", idChild, "Ptr*", pacc, "Ptr", VarSetCapacity(varChild,8+2*A_PtrSize,0)*0+&varChild)=0
	Return	ComObjEnwrap(9,pacc,1), _idChild_:=NumGet(varChild,8,"UInt")
}

Acc_ObjectFromPoint(ByRef _idChild_ = "", x = "", y = "")
{
	Acc_Init()
	If	DllCall("oleacc\AccessibleObjectFromPoint", "Int64", x==""||y==""?0*DllCall("GetCursorPos","Int64*",pt)+pt:x&0xFFFFFFFF|y<<32, "Ptr*", pacc, "Ptr", VarSetCapacity(varChild,8+2*A_PtrSize,0)*0+&varChild)=0
	Return	ComObjEnwrap(9,pacc,1), _idChild_:=NumGet(varChild,8,"UInt")
}

Acc_ObjectFromWindow(hWnd, idObject = -4)
{
	Acc_Init()
	If	DllCall("oleacc\AccessibleObjectFromWindow", "Ptr", hWnd, "UInt", idObject&=0xFFFFFFFF, "Ptr", -VarSetCapacity(IID,16)+NumPut(idObject==0xFFFFFFF0?0x46000000000000C0:0x719B3800AA000C81,NumPut(idObject==0xFFFFFFF0?0x0000000000020400:0x11CF3C3D618736E0,IID,"Int64"),"Int64"), "Ptr*", pacc)=0
	Return	ComObjEnwrap(9,pacc,1)
}

Acc_WindowFromObject(pacc)
{
	If	DllCall("oleacc\WindowFromAccessibleObject", "Ptr", IsObject(pacc)?ComObjValue(pacc):pacc, "Ptr*", hWnd)=0
	Return	hWnd
}
Acc_Error(p="") {
	static setting:=0
	return p=""?setting:setting:=p
}
Acc_Children(Acc) {
	if ComObjType(Acc,"Name") != "IAccessible"
		ErrorLevel := "Invalid IAccessible Object"
	else {
		Acc_Init(), cChildren:=Acc.accChildCount, Children:=[]
		if DllCall("oleacc\AccessibleChildren", "Ptr",ComObjValue(Acc), "Int",0, "Int",cChildren, "Ptr",VarSetCapacity(varChildren,cChildren*(8+2*A_PtrSize),0)*0+&varChildren, "Int*",cChildren)=0 {
			Loop %cChildren%
				i:=(A_Index-1)*(A_PtrSize*2+8)+8, child:=NumGet(varChildren,i), Children.Push(NumGet(varChildren,i-8)=9?Acc_Query(child):child), NumGet(varChildren,i-8)=9?ObjRelease(child):
			return Children.MaxIndex()?Children:
		} else
			ErrorLevel := "AccessibleChildren DllCall Failed"
	}
	if Acc_Error()
		throw Exception(ErrorLevel,-1)
}
Acc_Location(Acc, ChildId=0, byref Position="") { ; adapted from Sean's code
	try Acc.accLocation(ComObj(0x4003,&x:=0), ComObj(0x4003,&y:=0), ComObj(0x4003,&w:=0), ComObj(0x4003,&h:=0), ChildId)
	catch
		return
	Position := "x" NumGet(x,0,"int") " y" NumGet(y,0,"int") " w" NumGet(w,0,"int") " h" NumGet(h,0,"int")
	return	{x:NumGet(x,0,"int"), y:NumGet(y,0,"int"), w:NumGet(w,0,"int"), h:NumGet(h,0,"int")}
}
Acc_Parent(Acc)
{
	try parent:=Acc.accParent
	return parent?Acc_Query(parent):
}
Acc_Child(Acc, ChildId=0)
{
	try child:=Acc.accChild(ChildId)
	return child?Acc_Query(child):
}
Acc_Query(Acc)
{
	try return ComObj(9, ComObjQuery(Acc,"{618736e0-3c3d-11cf-810c-00aa00389b71}"), 1)
}

RangeTip(x:="", y:="", w:="", h:="", color:="Red", d:=2) ; from the FindText library, credit goes to feiyue
{
  local
  static id:=0
  if (x="")
  {
    id:=0
    Loop 4
      Gui, Range_%A_Index%: Destroy
    return
  }
  if (!id)
  {
    Loop 4
      Gui, Range_%A_Index%: +Hwndid +AlwaysOnTop -Caption +ToolWindow
        -DPIScale +E0x08000000
  }
  x:=Floor(x), y:=Floor(y), w:=Floor(w), h:=Floor(h), d:=Floor(d)
  Loop 4
  {
    i:=A_Index
    , x1:=(i=2 ? x+w : x-d)
    , y1:=(i=3 ? y+h : y-d)
    , w1:=(i=1 or i=3 ? w+2*d : d)
    , h1:=(i=2 or i=4 ? h+2*d : d)
    Gui, Range_%i%: Color, %color%
    Gui, Range_%i%: Show, NA x%x1% y%y1% w%w1% h%h1%
  }
}

#If IsCapturing
Esc::gosub ButCapture
UIA can also be set up to capture different events such as focus changed to a different element, menu opened, text selected/modified etc. For some examples of this, see Example 7 and 8 below.

NOTE: if inspecting browsers, it is good to make sure that accessibility is turned on: either run the browser with --force-renderer-accessibility flag (eg chrome.exe --force-renderer-accessibility) or use browser-specific user interfaces (for example in Chrome navigate to chrome://accessibility/ and turn on accessibility). Usually this isn't necessary, but if you have trouble inspecting browser content elements then I suggest trying it.

Some notes and caveats on UIA:
1) Firstly, every UIA call to interact with elements is a cross-application call from AutoHotkey to the UIA framework, which takes a relatively long time. This means that it is recommended to set tight conditions for searches (like FindAll) and to do as little calls as possible. Doing something like FindAll with the True condition and looping over all the elements to find one specific one is strongly not recommended, and usually a better solution exists anyway.
2) Like Acc, UIA also depends on the implementation of accessibility methods of specific programs. If a program doesn't implement UIA support for custom controls (such as Chrome scrollbar, or Sublime Text status bar), then unfortunately they simply can't be accessed with UIA. Fortunately most common controls (buttons, edits etc) and website elements ARE supported.
3) In some Windows builds Chromium-based apps (web browsers, Skype etc) might not automatically allow viewing content. This appears to be a bug that was eventually fixed (in my tests Windows 11 didn't have this prblem), but it can also be worked around by sending the WM_GETOBJECT message to the renderer element before making any UIA calls, for example with the following function. This doesn't need to be used if accessibility is already enabled (eg in chrome://accessibility) or browser is ran with the --force-renderer-accessibility flag.

Code: Select all

ChromiumActivateUIA(wTitle) {
	WinGet, cList, ControlList, %wTitle%
	if InStr(cList, "Chrome_RenderWidgetHostHWND1")
		SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Chrome_RenderWidgetHostHWND1, %wTitle%
}
Examples
Now for some examples using the UIA library.
1) Setting Notepad text. Opens up Notepad, then sets the text field to contain the text "Lorem ipsum"

Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, notepad.exe
UIA := UIA_Interface() ; Initialize UIA interface
WinWaitActive, ahk_exe notepad.exe
npEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe")) ; Get the element for the Notepad window
documentEl := npEl.FindFirstByType("Document") ; Find the first Document control (in Notepad there is only one). This assumes the user is running a relatively recent Windows and UIA interface version 2+ is available. In UIA interface v1 this control was Edit, so an alternative option instead of "Document" would be "UIA.__Version > 1 ? "Document" : "Edit""
documentEl.SetValue("Lorem ipsum") ; Set the value for the document control
ExitApp
2) A more complicated Notepad example, navigating the menu bar

Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, notepad.exe
UIA := UIA_Interface() ; Initialize UIA interface
WinWaitActive, ahk_exe notepad.exe
npEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe")) ; Get the element for the Notepad window
MsgBox, % npEl.DumpAll() ; Display all the sub-elements for the Notepad window. Press OK to continue
documentEl := npEl.FindFirstByType("Document") ; Find the first Document control (in Notepad there is only one). This assumes the user is running a relatively recent Windows and UIA interface version 2+ is available. In UIA interface v1 this control was Edit, so an alternative option instead of "Document" would be "UIA.__Version > 1 ? "Document" : "Edit""
documentEl.SetValue("Lorem ipsum") ; Set the value of the document control
MsgBox, Press OK to test saving. ; Wait for the user to press OK
fileEl := npEl.FindFirstByNameAndType("File", "MenuItem").Click() ; Click the "File" menu item
saveEl := npEl.WaitElementExistByName("Save",,2) ; Wait for the "Save" menu item to exist
saveEl.Click() ; And now click Save
ExitApp
3) Windows 10 Calculator example:

Code: Select all

#NoEnv
#SingleInstance force
#include <UIA_Interface>

Run, calc.exe
UIA := UIA_Interface() ; Initialize UIA interface
WinWaitActive, Calculator
cEl := UIA.ElementFromHandle(WinExist("Calculator")) ; Get the element for the Calculator window
; All the calculator buttons are of "Button" ControlType, and if the system language is English then the Name of the elements are the English words for the buttons (eg button 5 is named "Five", = sign is named "Equals")
cEl.FindFirstBy("Name=Six").Click() ; Find the first "Six" button by name and click it
cEl.FindFirstBy("Name=Five AND ControlType=Button").Click() ; Specify both name "Five" and control type "Button"
cEl.FindFirstByName("Plus").Click() ; An alternative method to FindFirstBy("Name=Plus")
cEl.FindFirstByNameAndType("Four", UIA.ButtonControlTypeId).Click() ; The type can be specified as "Button", UIA.ButtonControlTypeId, UIA_Enum.UIA_ButtonControlTypeId, or 50000 (which is the value of UIA.ButtonControlTypeId)
cEl.FindFirstByNameAndType("Equals", "Button").Click()
ExitApp
Now some examples using the UIA_Browser library.
4) First a simple test script to dump all the page information to the clipboard, after pressing F1 use Ctrl+V to paste it somewhere such as Notepad. Chrome is ran in incognito mode to avoid any extensions or previous logins on webpages from interfering.

Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2
#include <UIA_Interface>
#include <UIA_Browser>

browserExe := "chrome.exe"
Run, %browserExe% -incognito --force-renderer-accessibility ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.WaitPageLoad("New Tab", 3000) ; Wait the New Tab page to load with a timeout of 3 seconds
Clipboard=
Clipboard := cUIA.GetCurrentDocumentElement().DumpAll() ; Get the current document element (this excludes the URL bar, navigation buttons etc) and dump all the information about it in the clipboard. Use Ctrl+V to paste it somewhere, such as in Notepad.
ClipWait, 1
if Clipboard
	MsgBox, Page information successfully dumped. Use Ctrl+V to paste the info somewhere, such as in Notepad.
else
	MsgBox, Something went wrong and nothing was dumped in the clipboard!
return
5) Translate some text with Google Translate (there are a lot more direct options of doing this, take this only as an example)

Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2
#include <UIA_Interface>
#include <UIA_Browser>

browserExe := "chrome.exe"
Run, %browserExe% -incognito --force-renderer-accessibility ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface

; Before doing a translate, lets first set Google services language to English to ensure that locale-specific words are in English (if Google is in German for example, "English" would be "Englisch"
cUIA.WaitPageLoad("New Tab", 5000) ; Wait the New Tab page to load with a timeout of 5 seconds
cUIA.SetURL("https://www.google.com/preferences#languages", True) ; Set the URL and navigate to it
cUIA.WaitPageLoad() ; Wait the page to load
cUIA.FindFirstByName("Show more").Click() ; Display all languages to ensure English is visible
cUIA.WaitElementExistByName("English").Click() ; Select English
cUIA.FindFirstByName("Save").Click() ; Click "Save"
Sleep, 1000
cUIA.FindFirstByName("OK").Click() ; Sometimes a dialog pops up that confirms the save, in that case press "OK"
cUIA.WaitPageLoad("Google") ; Wait for Google main page to load, default timeout of 10 seconds

cUIA.SetURL("https://translate.google.com/", True) ; Navigate to Google Translate
cUIA.WaitPageLoad()
cUIA.FindFirstByName("More source languages").Click() ; Click source languages selection
cUIA.WaitElementExistByName("Spanish").Click() ; Select Spanish
Sleep, 500
cUIA.FindFirstByName("More target languages").Click() ; Open target languages selection
Sleep, 500
allEnglishEls := cUIA.FindAllByName("English") ; Find all elements with name "English"
allEnglishEls[allEnglishEls.MaxIndex()].Click() ; Select the last element with the name English (because English might also be an option in source languages, in which case it would be found first)

cUIA.WaitElementExistByName("Source text").SetValue("Este es un texto de muestra") ; Set some text to translate
return
6) An example using Microsoft Edge to do a simple Google search and some scrolling. This example DOES NOT work with Chrome, because unfortunately Chrome browsers scrollbar can't be accessed with UIA.
There are some caveats with Edge: sometimes after loading a document or webpage, it takes some time and multiple calls of FindFirst or FindAll to find elements. This means that after navigation or clicking, it is recommended to use WaitElementExist functions to make sure the element is found.

Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2
#include <UIA_Interface>
#include <UIA_Browser>

browserExe := "msedge.exe"
Run, %browserExe% -inprivate --force-renderer-accessibility ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.WaitPageLoad("New Tab", 5000) ; Wait the New Tab page to load with a timeout of 5 seconds
cUIA.SetURL("google.com", True) ; Set the URL to google and navigate
cUIA.WaitPageLoad()
langBut := cUIA.WaitElementExist("ClassName=neDYw tHlp8d AND ControlType=Button OR ControlType=MenuItem") ; First lets make sure the selected language is correct. The expression is evaluated left to right: finds an element where ClassName is "neDYw tHlp8d" and ControlType is button OR an element with a MenuItem controltype.

expandCollapsePattern := langBut.GetCurrentPatternAs("ExpandCollapse") ; Since this isn't a normal button, we need to get the ExpandCollapse pattern for it and then call the correct method.
if (expandCollapsePattern.CurrentExpandCollapseState == cUIA.ExpandCollapseState_Collapsed) ; Check that it is collapsed
	expandCollapsePattern.Expand() ; Expand the menu
cUIA.WaitElementExist("Name=English AND ControlType=MenuItem").Click() ; Select the English language

cUIA.WaitElementExist("Name=Accept all OR Name=I agree AND ControlType=Button").Click() ; If the "I agree" or "Accept all" button exists, then click it to get rid of the consent form
searchBox := cUIA.WaitElementExist("Name=Searc AND ControlType=ComboBox",,2) ; Looking for a partial name match "Searc" using matchMode=2. FindFirstByNameAndType is not used here, because if the "I agree" button was clicked then this element might not exist right away, so lets first wait for it to exist.
searchBox.SetValue("autohotkey forums") ; Set the search box text to "autohotkey forums"
cUIA.FindFirstByName("Google Search").Click() ; Click the search button to search
cUIA.WaitPageLoad()
; Now that the Google search results have loaded, lets scroll the page to the end.
docEl := cUIA.GetCurrentDocumentElement() ; Get the document element
sbPat := docEl.GetCurrentPatternAs("Scroll") ; Get the Scroll pattern to access scrolling methods and properties
hPer := sbPat.CurrentHorizontalScrollPercent ; Horizontal scroll percent doesn't actually interest us, but the SetScrollPercent method requires both horizontal and vertical scroll percents...
ToolTip, % "Current scroll percent: " sbPat.CurrentVerticalScrollPercent
for k, v in [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] { ; Lets scroll down in steps of 10%
	sbPat.SetScrollPercent(hPer, v)
	Sleep, 500
	ToolTip, % "Current scroll percent: " sbPat.CurrentVerticalScrollPercent
}
Sleep, 3000
ToolTip
ExitApp
7) The following script sets up an event handler for a "focus changed" event, and if the user focuses Chrome address bar then it clears it. To test it, run the script, type something in the address bar, click away and then click into the address bar. Press F5 to exit.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>
#include <UIA_Browser>

EventHandler(el) {
	global cUIA
	ToolTip, % "Caught event!`nElement name: " el.CurrentName
	if cUIA.CompareElements(cUIA.URLEditElement, el) ; Check if the focused element is the same as Chrome's address bar element (comparison using == won't work)
		el.SetValue("") ; If the Address bar was focused, clear it
}

ExitFunc() {
	global cUIA, h
	UIA.RemoveFocusChangedEventHandler(h) ; Remove the event handler. Alternatively use cUIA.RemoveAllEventHandlers() to remove all handlers
}

browserExe := "chrome.exe"
Run, %browserExe%  -incognito
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe)

h := UIA_CreateEventHandler("EventHandler", "FocusChanged") ; Create a new FocusChanged event handler that calls the function EventHandler (required arguments: element)
cUIA.AddFocusChangedEventHandler(h) ; Add a new FocusChangedEventHandler
OnExit("ExitFunc") ; Set up an OnExit call to clean up the handler when exiting the script
return

F5::ExitApp
8) Another event handler example, this time activating when text is selected. A new Notepad window is opened, try selecting some text in it.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

TextSelectionChangedEventHandler(el, eventId) {
	textPattern := el.GetCurrentPatternAs("TextPattern")
	selectionArray := textPattern.GetSelection() ; Gets the currently selected text in Notepad as an array of TextRanges (some elements support selecting multiple pieces of text at the same time, thats why an array is returned)
	selectedRange := selectionArray[1] ; Our range of interest is the first selection (TextRange)
	wholeRange := textPattern.DocumentRange ; For comparison, get the whole range (TextRange) of the document
	selectionStart := selectedRange.CompareEndpoints(UIA_Enum.TextPatternRangeEndpoint_Start, wholeRange, UIA_Enum.TextPatternRangeEndpoint_Start) ; Compare the start point of the selection to the start point of the whole document
	selectionEnd := selectedRange.CompareEndpoints(UIA_Enum.TextPatternRangeEndpoint_End, wholeRange, UIA_Enum.TextPatternRangeEndpoint_Start) ; Compare the end point of the selection to the start point of the whole document

	ToolTip, % "Selected text: " selectedRange.GetText() "`nSelection start location: " selectionStart "`nSelection end location: " selectionEnd  ; Display the selected text and locations of selection
}

ExitFunc() {
	global UIA, handler, NotepadEl
	try UIA.RemoveAutomationEventHandler(UIA.Text_TextSelectionChangedEventId, NotepadEl, handler) ; Remove the event handler. Alternatively use UIA.RemoveAllEventHandlers() to remove all handlers. If the Notepad window doesn't exist any more, this throws an error.
}
	
lorem =
(
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
) ; Some sample text to play around with
Run, notepad.exe
WinActivate, ahk_exe notepad.exe
WinWaitActive, ahk_exe notepad.exe
UIA := UIA_Interface()
	
NotepadEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe"))
DocumentControlCondition := UIA.CreatePropertyCondition(UIA.ControlTypePropertyId, UIA.DocumentControlTypeId) ; If UIA Interface version is 1, then the ControlType is Edit instead of Document!
DocumentControl := NotepadEl.FindFirst(DocumentControlCondition)
DocumentControl.SetValue(lorem) ; Set the value to our sample text

handler := UIA_CreateEventHandler("TextSelectionChangedEventHandler") ; Create a new event handler that points to the function TextSelectionChangedEventHandler, which must accept two arguments: element and eventId.
UIA.AddAutomationEventHandler(UIA.Text_TextSelectionChangedEventId, NotepadEl,,, handler) ; Add a new automation handler for the TextSelectionChanged event
OnExit("ExitFunc") ; Set up an OnExit call to clean up the handler when exiting the script
return

F5::ExitApp
Pattern examples 9-16 are available here

UIA_Interface, UIA_Browser properties and methods
Properties and methods for UIA_Browser.ahk:

Code: Select all

;When initializing UIA_Browser it takes as an argument the browsers title. It follows the same syntax as ahk WinTitle, and SetTitleMatchMode also works the same. Example: 
cUIA := new UIA_Browser("ahk_exe chrome.exe")

; UIA_Browser properties:
; UIA_Browser can return both UIA_Interface and the browser window elements properties and methods, depending on where the property exists.
; Example: cUIA.CurrentProcessId will return the ProcessId for the browser window element by calling cUIA.BrowserElement.CurrentProcessId.
UIA_Browser.UIA ; The UIA_Interface object itself, which can be more readily accessed by just calling a UIA_Interface method/property from UIA_Browser (eg cUIA.CreateNotCondition() will actually call cUIA.UIA.CreateNotCondition())
UIA_Browser.BrowserId ; ahk_id of the browser window
UIA_Browser.BrowserType ; "Chrome", "Edge", or "Unknown"
UIA_Browser.BrowserElement ; The browser window element, which can also be accessed by just calling an element method from UIA_Browser (cUIA.FindFirst would call FindFirst method on the BrowserElement, is equal to cUIA.BrowserElement.FindFirst)
UIA_Browser.MainPaneElement ; Element for the upper part of the browser containing the URL bar, tabs, extensions etc
UIA_Browser.URLEditElement ; Element for the address bar
UIA_Browser.TWT ; Contains a TreeWalker that is created with TrueCondition
; All UIA enumerations/constants are also available in the UIA_Browser object. 
cUIA.UIA_ControlTypePropertyId ; Returns UIA_ControlTypePropertyId from UIA_Enum, returning value 30003
cUIA.ButtonControlTypeId ; "UIA_" can be omitted. Returns UIA_ButtonControlTypeId from UIA_Enum class, returning value 50000

; UIA_Browser methods:
; UIA_Browser can return both UIA_Interface and the browser window elements properties and methods, depending on where the method exists. 
Example: cUIA.CreateTreeWalker(condition) accesses the CreateTreeWalker method from UIA_Interface class; cUIA.FindFirst(condition) calls FindFirst on the browser window element (because FindFirst doesn't exist inside UIA_Interface).
UIA_Browser.GetCurrentMainPaneElement() ; Refreshes UIA_Browser.MainPaneElement and also returns it
UIA_Browser.GetCurrentDocumentElement() ; Returns the current document/content element of the browser
UIA_Browser.GetAllText() ; Gets all text from the browser element (CurrentName properties for all child elements)
UIA_Browser.GetAllLinks() ; Gets all link elements from the browser
UIA_Browser.WaitTitleChange(targetTitle="", timeOut=10000) ; Waits the browser title to change to targetTitle (by default just waits for the title to change), timeOut is in milliseconds (default is 10 seconds)
UIA_Browser.WaitPageLoad(targetTitle="", timeOut=10000, sleepAfter=500) ; Waits the browser page to load to targetTitle, default timeOut is 10 seconds, sleepAfter additionally sleeps for 500ms after the page has loaded.
UIA_Browser.Back() ; Presses the Back button
UIA_Browser.Forward() ; Presses the Forward button
UIA_Browser.Reload() ; Presses the Reload button
UIA_Browser.Home(butName="Home") ; Presses the Home button if it exists. If the browser language is not set to English, the correct butName can be specified.
UIA_Browser.GetCurrentURL(fromAddressBar=False) ; Gets the current URL. fromAddressBar=True gets it straight from the URL bar element, which is not a very good method, because the text might be changed by the user and doesn't start with "http(s)://". Default of fromAddressBar=False will cause the real URL to be fetched, but the browser must be visible for it to work (if is not visible, it will be automatically activated).
UIA_Browser.GetCurrentDocumentElement() ; Gets the current element for the content of the website
UIA_Browser.SetURL(newUrl, navigateToNewUrl = False) ; Sets the URL bar to newUrl, optionally also navigates to it if navigateToNewUrl=True
UIA_Browser.Navigate(url, targetTitle="", waitLoadTimeOut=10000, sleepAfter=500, stopButtonText="Stop") ; Navigates to URL and waits page to load
UIA_Browser.NewTab(butName="New tab", matchMode=2, caseSensitive=True) ; Presses the New tab button. The button name might differ if the browser language is not set to English  and can be specified with butName (with options for matchMode and caseSensitivity for the search)
UIA_Browser.GetAllTabs() ; Gets all tab elements
UIA_Browser.GetAllTabNames() ; Gets all the titles of tabs
UIA_Browser.SelectTab(tabName, matchMode=3, caseSensitive=True) ; Selects a tab with the text of tabName. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
UIA_Browser.GetTab(searchPhrase="", matchMode=3, caseSensitive=True) ; Returns a tab element with text of searchPhrase, or if empty then the currently selected tab. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
UIA_Browser.CloseTab(tabElementOrName="", matchMode=3, caseSensitive=True) ; Close tab by either providing the tab element or the name of the tab. If tabElementOrName is left empty, the current tab will be closed.
UIA_Browser.IsBrowserVisible() ; Returns True if any of the 4 corners of the browser are visible.
UIA_Browser.GetAlertText() ; Gets the text from an alert box
UIA_Browser.CloseAlert() ; Closes an alert box
UIA_Browser.JSExecute(js) ; Executes Javascript code using the address bar
UIA_Browser.JSReturnThroughClipboard(js) ; Executes Javascript code using the address bar and returns the return value of the code using the clipboard (resetting it back afterwards)
UIA_Browser.JSReturnThroughTitle(js, timeOut=500) ; Executes Javascript code using the address bar and returns the return value of the code using the browsers title (resetting it back afterwards). This might be unreliable, so the clipboard method is recommended instead.
UIA_Browser.JSSetTitle(newTitle) ; Uses Javascript through the address bar to change the title of the browser
UIA_Browser.JSGetElementPos(selector, useRenderWidgetPos=False) ; Uses Javascript's querySelector to get a Javascript element and then its position. useRenderWidgetPos=True uses position of the Chrome_RenderWidgetHostHWND1 control to locate the position element relative to the window, otherwise it uses UIA_Browsers CurrentDocumentElement position.
UIA_Browser.JSClickElement(selector) ; Uses Javascript's querySelector to get and click a Javascript element
UIA_Browser.ControlClickJSElement(selector, WhichButton="", ClickCount="", Options="", useRenderWidgetPos=False) ; Uses Javascript's querySelector to get a Javascript element and then ControlClicks that position. useRenderWidgetPos=True uses position of the Chrome_RenderWidgetHostHWND1 control to locate the position element relative to the window, otherwise it uses UIA_Browsers CurrentDocumentElement position.
UIA_Browser.ClickJSElement(selector, WhichButton="", ClickCount=1, DownOrUp="", Relative="", useRenderWidgetPos=False) ; Uses Javascript's querySelector to get a Javascript element and then Clicks that position. useRenderWidgetPos=True uses position of the Chrome_RenderWidgetHostHWND1 control to locate the position element relative to the window, otherwise it uses UIA_Browsers CurrentDocumentElement position.
Extra properties and methods for UIA_Interface.ahk:

Code: Select all

UIA_Interface functions:
UIA_Interface(maxVersion="") ; Creates a new UIAutomation object, or returns a previously created one. maxVersion can be specified to limit the highest used IUIAutomation version (for Windows 10 this currently is 7). Most applications can use IUIAutomation2, but since higher IUIAutomation versions can use all the methods and properties of the lower ones, usually the highest version can be used.
UIA_Enum(e) ; Returns a constant/enumeration from the UIA_Enum class (which can also be used directly). If a constant starts with "UIA_", that can be omitted. Example: UIA_Enum("ButtonControlTypeId") returns 50000.
UIA_CreateEventHandler(funcName, handlerType="") ; Creates a new event handler object that calls funcName when the appropriate event is received (after calling a Add...EventHandler method). handlerType should be left empty when using AddAutomationEventHandler, in other cases specify the EventHandler name (possible options: FocusChanged, StructureChanged, TextEditTextChanged, Changes, Notification), eg for AddFocusChangedEventHandler specify "FocusChanged".

UIA_Base class properties (apply for the UIA_Interface object, and for all sub-objects suchs as elements etc):
UIA_Base.TrueCondition ; Contains the return value for CreateTrueCondition, for more convenient use
UIA_Base.TreeWalkerTrue ; Contains a TreeWalker created with TrueCondition, for more convenient use

class UIA_Interface
UIA_Interface.CreateCondition(property, val) ; Wrapper function for CreatePropertyCondition. Property can be either the number or text from UIA_PropertyIds (PROPERTY part of UIA_PROPERTYPropertyId). 
; cUIA.CreateCondition("Name", "Username") ; Creates a new property condition using UIA_NamePropertyId with the value of "Username"
UIA_Interface.ElementFromChromium(winTitle="A", activateChromiumAccessibility=True) ; Usually gets the Document element for Chromium apps (such as Skype, Discord etc) if it is inaccessible with ElementFromHandle
UIA_Interface.SmallestElementFromPoint(x="", y="", activateChromiumAccessibility=True) ; Gets ElementFromPoint and filters out the smallest subelement that is under the specified point.

class UIA_Element
UIA_Element.CurrentValue ; This property gets or sets the current value of the element. This is a wrapper for GetCurrentPropertyValue("Value") and SetValue()
UIA_Element.CurrentExists ; Checks if the element stills exists
UIA_Element.WaitNotExist(timeOut=10000) ; Wait until the element doesn't exist, with a default timeOut of 10000ms (10 seconds). Returns 1 if the element doesn't exist, otherwise 0.
UIA_Element.GetClickablePointRelativeTo(relativeTo="") { ; Wrapped for the GetClickablePoint function, that converts the coordinates according to the mode selected with relativeTo. relativeTo can be window, screen or client, default is A_CoordModeMouse.
UIA_Element.GetParentHwnd() ; Gets the hwnd for the parent window. A more reliable way would be to use CurrentProcessId with ahk_pid.
UIA_Element.SetValue(val, pattern="") ; Set element value using Value pattern, or as a fall-back using LegacyIAccessible pattern. If a pattern is specified then that is used instead.
/*
	UIA_Element.Click
	Click using one of the available click-like methods (InvokePattern Invoke(), TogglePattern Toggle(), ExpandCollapsePattern Expand() or Collapse() (depending on the state of the element), SelectionItemPattern Select(), or LegacyIAccessible DoDefaultAction()), in which case ClickCount is ignored. If WhichButton is specified (for example "left", "right") then the native mouse Click function will be used to click the center of the element.
	If WhichButton is a number, then Sleep will be called with that number. Eg Click(200) will sleep 200ms after clicking
	If ClickCountAndSleepTime is a number >=10, then Sleep will be called with that number. To click 10+ times and sleep after, specify "ClickCount SleepTime". Ex1: Click("left", 200) will sleep 200ms after clicking. Ex2: Click("left", "20 200") will left-click 20 times and then sleep 200ms.
	If Relative is "Rel" or "Relative" then X and Y coordinates are treated as offsets from the current mouse position. Otherwise it expects offset values for both X and Y (eg "-5 10" would offset X by -5 and Y by +10).
*/
UIA_Element.Click(WhichButtonOrSleepTime="", ClickCountAndSleepTime=1, DownOrUp="", Relative="")
; ControlClicks the element after getting relative coordinates with GetClickablePointRelativeTo("window"). Specifying WinTitle makes the function faster, since it bypasses getting the Hwnd from the element.
; If WinTitle or WinText is a number, then Sleep will be called with that number of milliseconds. Ex: ControlClick(200) will sleep 200ms after clicking. Same for ControlClick("ahk_id 12345", 200)
UIA_Element.ControlClick(WinTitleOrSleepTime="", WinTextOrSleepTime="", WhichButton="", ClickCount="", Options="", ExcludeTitle="", ExcludeText="")
UIA_Element.GetSupportedPatterns() ; Get available patterns for the element. Use of this should be avoided, since it calls GetCurrentPatternAs for every pattern. A better option is PollForPotentialSupportedPatterns
UIA_Element.GetCurrentPos(relativeTo="") ; Returns an object containing the x, y coordinates and width and height: {x:x coordinate, y:y coordinate, w:width, h:height}. relativeTo can be client, window or screen, default is A_CoordModeMouse.
UIA_Element.GetChildren(scope=0x2) ; By default get only direct children (TreeScope_Children)
UIA_Element.Dump() { ; Returns info about the element: ControlType, Name, Value, LocalizedControlType, AcceleratorKey, AutomationId. 
UIA_Element.DumpAll(maxDepth=20) ; Returns info (ControlType, Name etc) for all descendants of the element. maxDepth is the allowed depth of recursion, by default 20 layers. DO NOT call this on the root element!
/*
	FindFirst using search criteria. 
	expr: 
		Takes a value in the form of "PropertyId=matchvalue" to match a specific property with the value matchValue. PropertyId can be most properties from UIA_Enum.UIA_PropertyId method (for example Name, ControlType, AutomationId etc). 
		
		Example1: "Name=Username:" would use FindFirst with UIA_Enum.UIA_NamePropertyId matching the name "Username:"
		Example2: "ControlType=Button would FindFirst using UIA_Enum.UIA_ControlTypePropertyId and matching for UIA_Enum.UIA_ButtonControlTypeId. Alternatively "ControlType=50000" can be used (direct value for UIA_ButtonControlTypeId which is 50000)
		
		Criteria can be combined with AND, OR, &&, ||:
		Example3: "Name=Username: AND ControlType=Button" would FindFirst an element with the name property of "Username:" and control type of button.

		Flags can be modified for each individual condition by specifying FLAGS=n after the condition (and before and/or operator). 0=no flags; 1=ignore case (case insensitive matching); 2=match substring; 3=ignore case and match substring

		If matchMode==3 or matching substrings is supported (Windows 10 17763 and above) and matchMode==2, then parentheses are supported. 
		Otherwise parenthesis are not supported, and criteria are evaluated left to right, so "a AND b OR c" would be evaluated as "(a and b) or c".
		
		Negation can be specified with NOT:
		Example4: "NOT ControlType=Edit" would return the first element that is not an edit element
	
	scope:
		Scope by default is UIA_TreeScope_Descendants. 
		
	matchMode:
		If using Name PropertyId as a criteria, this follows the SetTitleMatchMode scheme: 
			1=name must must start with the specified name
			2=can contain anywhere
			3=exact match
			RegEx=using regular expression. In this case the Name can't be empty.
	
	caseSensitive:
		If matching for a string, this will specify case-sensitivity.

*/
UIA_Element.FindFirstBy(expr, scope=0x4, matchMode=3)
UIA_Element.FindFirstByName(name, scope=0x4, matchMode=3) ; Wrapper for FindFirst using UIA_NamePropertyId. Scope by default is UIA_TreeScope_Descendants. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
UIA_Element.FindFirstByType(controlType, scope=0x4) ; Wrapper for FindFirst using UIA_ControlTypePropertyId
UIA_Element.FindFirstByNameAndType(name, controlType, scope=0x4, matchMode=3) ; Wrapper for FindFirst using UIA_NamePropertyId, UIA_ControlTypePropertyId and creating an AND condition for the two.
UIA_Element.FindAllByName(name, scope=0x4, matchMode=3) ; Wrapper for FindAll using UIA_NamePropertyId. Scope by default is UIA_TreeScope_Descendants. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
UIA_Element.FindAllByType(controlType, scope=0x4) ; Wrapper for FindAll using UIA_ControlTypePropertyId.
UIA_Element.FindAllByNameAndType(name, controlType, scope=0x4, matchMode=3) ; Wrapper for FindAll using UIA_NamePropertyId, UIA_ControlTypePropertyId and creating an AND condition for the two.
UIA_Element.FindByPath(path) ; Gets an element by the "path" that is displayed in the UIA_Element.DumpAll() result. This is similar to the Acc path, but for UIA (they are not compatible). Use of this is discouraged, because usually there are more reliable methods available. Also can be used to get parent elements ("Pn") and sibling elements (+n or -n): "P2.+2.1" would get parent of the parent, and then the second siblings first child.
UIA_Element.WaitElementExist(expr, scope=0x4, matchMode=3, timeOut=10000) ; Calls UIA_Element.FindFirstBy until the element is found and then returns it, with a timeOut of 10000ms (10 seconds)
UIA_Element.WaitElementNotExist(expr, scope=0x4, matchMode=3, caseSensitive=True, timeOut=10000) ; Tries to FindFirstBy the element and if it is found then waits until the element doesn't exist (using WaitNotExist()), with a timeOut of 10000ms (10 seconds). For explanations of the other arguments, see FindFirstBy
UIA_Element.WaitElementExistByName(name, scope=0x4, matchMode=3, timeOut=10000) ; Calls UIA_Element.FindFirstByName until the element is found and then returns it, with a timeOut of 10000ms (10 seconds)
UIA_Element.WaitElementExistByType(controlType, scope=0x4, timeOut=10000) ; Calls UIA_Element.FindFirstByType until the element is found and then returns it, with a timeOut of 10000ms (10 seconds)
UIA_Element.WaitElementExistByNameAndType(name, controlType, scope=0x4, timeOut=10000) ; Calls UIA_Element.FindFirstByNameAndType until the element is found and then returns it, with a timeOut of 10000ms (10 seconds)
UIA_Element.Highlight(displayTime:=2000, color:="Red", d:=4) ; Highlights the element borders for 2 seconds by default.
Constants
The use of UIA is dependent on many constants/enumerations: all properties, control types, patterns, scopes etc are defined by constants. These can be found under the UIA_Enum class in the UIA_Interface.ahk library, or alternatively inside the UIA_Constants.ahk file. Accessing the constants can be done in multiple different ways:
1) Use the returned object of UIA_Interface:

Code: Select all

UIA := UIA_Interface()
	UIA.UIA_InvokePatternId ; returns the corresponding ID of 10000. 
	UIA.TreeScope_Descendants ; returns 0x4
If a property starts with "UIA_" then that part may be omitted: UIA.ButtonControlTypeId would return 50000.

Calling UIA_Enum methods is also supported:

Code: Select all

UIA.UIA_EventId(20000) ; returns "ToolTipOpened"
2) Use UIA_Enum class directly. This requires using exact property and method names:

Code: Select all

UIA_Enum.UIA_InvokePatternId
	UIA_Enum.UIA_PatternId(10000)

3) Use a UIA_Enum object:

Code: Select all

UIAc := new UIA_Enum
	UIAc.InvokePatternId ; returns 10000
	UIAc.PatternId(10000) ; returns "InvokePattern"

4) Use the UIA_Enum function:

Code: Select all

UIA_Enum("ButtonControlTypeId") would return 50000
5) Alternatively the optional UIA_Constants.ahk file can be included and the constants from there used. This creates a lot of global variables and might not be preferred because of that.

Code: Select all

#include <UIA_Constants>
	UIA_InvokePatternId ; equal to 10000
The UIA_Enum class can also be used to limit the maximum version of used elements or objects. By default, the highest available version is used, so if a new element is created it is UIA_Element7 (if that is available on the current setup). By changing UIA_Enum.UIA_CurrentVersion_Element value, all new elements will be created with a maximum of that version number.
Limiting the version of patterns can be done by specifying the exact name of the pattern when creating it. If a full pattern name is specified then that exact version will be used (eg "TextPattern" will return a UIA_TextPattern object), otherwise the highest version will be used (eg "Text" might return UIA_TextPattern2 if it is available).

Downloadable libraries
Libraries themselves if Github is not available:
UIA_Interface.ahk
UIA_Interface.ahk
(256.41 KiB) Downloaded 384 times
UIA_Browser.ahk
UIA_Browser.ahk
(38.39 KiB) Downloaded 397 times
OPTIONAL: UIA_Constants.ahk
UIA_Constants.ahk
(44.13 KiB) Downloaded 466 times

Edit history:

Code: Select all

06.06.22: Fixed typos in examples. Added example files to Github.
10.06.22: Fixed UIAViewer inconsistencies. Added Acc Path finding option to UIAViewer (note that Acc Path cannot be used with the UIA library, only with Acc library).
11.06.22: Fixed UIAViewer bugs: Esc isn't blocked anymore when capturing. Created a workaround for ElementFromPoint not returning the deepest element. Added error handling when looking at elevated windows (eg Task Manager).
12.06.22: Fixed UIA_Browser.ahk bugs: typo in UIA_Browser.ahk SelectTab method; rewrote GetCurrentMainPaneElement method to avoid looking for the first toolbar (might fail in Chrome).
14.06.22: UIAViewer.ahk now sends WM_GETOBJECT message when appropriate to make Chromium apps content available. Thanks to users malcev and rommmcek for this tip.
16.02.22: modified example 5 to be in accordance with forum rules
19.06.22: MAJOR UPDATE. 
1) UIA_Interface was updated to use the latest UIA interface available (currently that is IUIAutomation7), falling back to the highest available version. Notably most applications do not implement methods above IUIAutomation2/CUIAutomation8. To limit the version, call UIA_Interface(maxVersion) (eg UIA_Interface(2) would use IUIAutomation2, or if that fails then IUIAutomation version 1). The current version of UIA_Interface, elements and patterns is available under __Version property. To limit the maximum version of elements, modify UIA_Enum.UIA_CurrentVersion_Element property accordingly.
2) UIA_Constants.ahk was made optional, all constants and enumerations are now available under UIA_Enum class.
3) Examples were updated to according to the changes.
4) Most patterns are now implemented.
5) Event handlers support was added (with two examples: Example 7 and 8).
6) ElementFromHandle and ElementFromPoint were added arguments to send the WM_GETOBJECT message to Chromium apps, which otherwise might not show any content.
7) GetCurrentValue() was replaced with CurrentValue property (also supports setting the value).
8) Many smaller bugfixes and improvements.
21.06.22 fixed UIA_Interface FindAllByType. Improved UIA_Browser error-handling.
23.06.22 updated UIAViewer to show AutomationId, and ControlType is shown in text form as well. Fixed a bug in UIA_Browser where GetCurrentMainPaneElement didn't get URL edit element if the URL bar was not found by name. Fixed bugs in UIA_Interface: case-sensitivity with matchMode=1 now works; ScrollPattern CurrentHorizontalViewSize and CurrentHorizontallyScrollable now work. Added Pattern examples 9-16.
29.06.22 many bug fixes: FindFirstBy with NOT condition now works properly when matchMode!=3. Fixed CompareRuntimeIds() and GetRuntimeId(). Fixed Click() methods use of ExpandCollapsePattern. Improved comments with references to Microsoft documentation. Added SmallestElementFromPoint() method to UIA_Interface class. Added new properties/methods to UIA_Element: CurrentExists, WaitNotExist(), WaitElementNotExist(). Added TextPattern, TextRange, and TextChangedEvent examples 17-18.
04.07.22 UIA_Interface fixes: fixed memory leaks (objects and BSTR-s not being released), rewrote variant handling, fixed VT_BOOL to be interpreted correctly (before fix: -1=True, 0=False, 1=False).
UIA_Browser changes: FindTabByName() was named to GetTab() and it can additionally return the currently selected tab. Added JSGetElementPos, JSClickElement, ControlClickJSElement and ClickJSElement methods. Changed caseSensitive default values to True to be more consistent with UIA_Interface. UIA_Browser can now be created with a custom object customNames, that defines custom CurrentName values for locale-specific elements (such as the name of the URL bar)
10.07.22 UIA_Interface updates: UIA_Element.FindByPath can now get parent and siblings as well; UIA_Element.Dump() displays also the ControlType in English, and values of "0" are displayed; improved activateChromiumAccessibility flag for ElementFromHandle and ElementFromPoint; UIA_Element.GetClickablePoint now returns a value. 
UIA_Browser updates: WaitPageLoad better implementation; ControlSend releases modifier keys
UIA_Viewer updates: listview sizes can now be changed by dragging
15.07.22 UIA_Interface updates: UIA_Interface.CreateCondition now supports expressions (parenthesis are allowed). UIA_Element.FindFirstBy and FindAllBy support parenthesis if matchMode is 2 or 3. UIA_Element.Highlight() highlights the element borders (for debugging purposes). Properties have been rewritten, now Current... properties can be accessed without the "Current" part (eg UIA_Element.Value). For some properties, setting values is supported. Patterns can be accessed like properties: UIA_Element.ValuePattern gets UIA_ValuePattern (note that the highest version of the pattern won't be selected). UIA_Element.Click() and UIA_Element.ControlClick() first or second arguments can be a "sleep after" value.
UIA_Browser updates: Added Navigate() method.
24.07.22 UIA_Interface updates: fixed EventHandlerGroups, and multiple EventHandlers can be created at the same time. enableChromiumAccessibility is now by default True. SPI_SCREENREADER flag is set on UIA_Interface() calls. Multiple bugfixes to caching. UIA_Browser updates: closing tabs is no longer locale-dependent.
UIAViewer updates: fixed bug where custom elements with empty names are not shown.
09.09.22 UIA_Interface updates: Added ElementFromChromium. ActivateChromiumAccessibility is now byref (which will return the Chromium element on the first call of ElementFrom... methods). 
UIA_Browser updates: added some support for Firefox. Separated classes for different browsers for easier add-ons.
11.09.22 UIA_Interface update: Fixed ElementFromChromium crashing Chrome. UIA_Browser update: now waits for browser to fully load before hooking elements.
08.10.22 UIA_Interface update: Fixed FindAllByNameAndType. UIA_CreateEventHandler accepts objects as funcName.
Last edited by Descolada on 30 Jan 2023, 13:04, edited 40 times in total.

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Jun 2022, 09:11

Taking notes from this AutoIt3 post about UIA patterns, I wrote some similar examples to try with UIA_Interface.ahk. All examples are also available at GitHub. The reason I am creating a new post for this is that the main post is reaching its character limit...
Note: these examples do not look at all available UIA_Interface patterns. Missing are some of the rarely used patterns like VirtualizedItemPattern. Also LegacyIAccessiblePattern is missing, because the Acc library handles that functionality.

Example 9: InvokePattern and TogglePattern
InvokePattern can be used to invoke the action of a control: this is usually a singular action where no other actions are possible (eg clicking a button). This won't work with controls which have multiple actions available (eg toggling or untoggling a checkbox).
TogglePattern can be used to cycle through a set of states of a control, which usually is checking or unchecking a checkbox.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, explore C:\
UIA := UIA_Interface()
WinWaitActive, (C:)
explorerEl := UIA.ElementFromHandle(WinActive("A"))
fileEl := explorerEl.FindFirstByNameAndType("File tab", "Button")
invokePattern := fileEl.GetCurrentPatternAs("Invoke")
MsgBox, % "Invoke pattern doesn't have any properties. Press OK to call Invoke on the ""File"" button..."
invokePattern.Invoke()

Sleep, 1000
MsgBox, Press OK to navigate to the View tab to test TogglePattern... ; Not part of this demonstration
explorerEl.FindFirstByNameAndType("View", "TabItem").GetCurrentPatternAs("SelectionItem").Select() ; Not part of this demonstration

hiddenItemsCB := explorerEl.FindFirstByNameAndType("Hidden items", "CheckBox")
togglePattern := hiddenItemsCB.GetCurrentPatternAs("Toggle")
Sleep, 500
MsgBox, % "TogglePattern properties for ""Hidden items"" checkbox: "
	. "`nCurrentToggleState: " togglePattern.CurrentToggleState

MsgBox, % "Press OK to toggle"
togglePattern.Toggle()
Sleep, 500
MsgBox, % "Press OK to toggle again"
togglePattern.Toggle()

ExitApp
Example 10: ExpandCollapsePattern
ExpandCollapsePattern can be used to visually expand a controls content to display it, or collapse the content (eg treeviews).

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, explore C:\
UIA := UIA_Interface()
DriveGet, CDriveName, Label, C:
CDriveName := CDriveName " (C:)"
WinWaitActive, %CDriveName%,,1
explorerEl := UIA.ElementFromHandle(WinActive("A"))
CDriveEl := explorerEl.FindFirstByNameAndType(CDriveName, "TreeItem")
if !CDriveEl {
	MsgBox, Drive C: element not found! Exiting app...
	ExitApp
}

expColPattern := CDriveEl.GetCurrentPatternAs("ExpandCollapse")
Sleep, 500
MsgBox, % "ExpandCollapsePattern properties: "
	. "`nCurrentExpandCollapseState: " (state := expColPattern.CurrentExpandCollapseState) " (" UIA_Enum.ExpandCollapseState(state) ")"

MsgBox, Press OK to expand drive C: element
expColPattern.Expand()
Sleep, 500
MsgBox, Press OK to collapse drive C: element
expColPattern.Collapse()

ExitApp
Example 11: ScrollPattern and ScrollItemPattern
ScrollPattern can be used to scroll an element.
ScrollItemPattern can be used to scroll a subelement of a scrollable element (eg combobox item) into view.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, explore C:\
UIA := UIA_Interface()
DriveGet, CDriveName, Label, C:
CDriveName := CDriveName " (C:)"
WinWaitActive, %CDriveName%,,1
explorerEl := UIA.ElementFromHandle(WinActive("A"))
treeEl := explorerEl.FindFirstByType("Tree")

MsgBox, % "For this example, make sure that the folder tree on the left side in File Explorer has some scrollable elements (make the window small enough)."
scrollPattern := treeEl.GetCurrentPatternAs("Scroll")
Sleep, 500
MsgBox, % "ScrollPattern properties: "
	. "`nCurrentHorizontalScrollPercent: " scrollPattern.CurrentHorizontalScrollPercent ; If this returns an error about not existing, make sure you have the latest UIA_Interface.ahk
	. "`nCurrentVerticalScrollPercent: " scrollPattern.CurrentVerticalScrollPercent
	. "`nCurrentHorizontalViewSize: " scrollPattern.CurrentHorizontalViewSize
	. "`nCurrentHorizontallyScrollable: " scrollPattern.CurrentHorizontallyScrollable
	. "`nCurrentVerticallyScrollable: " scrollPattern.CurrentVerticallyScrollable
Sleep, 50
MsgBox, % "Press OK to set scroll percent to 50% vertically and 0% horizontally."
scrollPattern.SetScrollPercent(,50)
Sleep, 500
MsgBox, % "Press OK to scroll a Page Up equivalent upwards vertically."
scrollPattern.Scroll(, UIA.ScrollAmount_LargeDecrement) ; LargeDecrement is equivalent to pressing the PAGE UP key or clicking on a blank part of a scroll bar. SmallDecrement is equivalent to pressing an arrow key or clicking the arrow button on a scroll bar.

Sleep, 500
MsgBox, Press OK to scroll drive C: into view.
CDriveEl := explorerEl.FindFirstByNameAndType(CDriveName, "TreeItem")
if !CDriveEl {
	MsgBox, C: drive element not found! Exiting app...
	ExitApp
}
scrollItemPattern := CDriveEl.GetCurrentPatternAs("ScrollItem")
scrollItemPattern.ScrollIntoView()

ExitApp
Example 12: GridPattern and GridItemPattern
From Microsoft documentation:
GridPattern provides access to a control that acts as a container for a collection of child controls that are organized in a two-dimensional logical coordinate system that can be traversed by row and column. The children of this element support the IUIAutomationGridItemPattern interface.
GridItemPattern provides access to a child control in a grid-style container that supports the IUIAutomationGridPattern interface.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, explore C:\
UIA := UIA_Interface()
DriveGet, CDriveName, Label, C:
CDriveName := CDriveName " (C:)"
WinWaitActive, %CDriveName%
explorerEl := UIA.ElementFromHandle(WinActive("A"))
listEl := explorerEl.FindFirstByType("List")

gridPattern := listEl.GetCurrentPatternAs("Grid")
Sleep, 500
MsgBox, % "GridPattern properties: "
	. "`nCurrentRowCount: " gridPattern.CurrentRowCount
	. "`nCurrentColumnCount: " gridPattern.CurrentColumnCount
Sleep, 50
MsgBox, % "Getting grid item from row 4, column 1 (0-based indexing)"
editEl := gridPattern.GetItem(3,0)
MsgBox, % "Got this element: `n" editEl.Dump()

gridItemPattern := editEl.GetCurrentPatternAs("GridItem")
MsgBox, % "GridItemPattern properties: "
	. "`nCurrentRow: " gridItemPattern.CurrentRow
	. "`nCurrentColumn: " gridItemPattern.CurrentColumn
	. "`nCurrentRowSpan: " gridItemPattern.CurrentRowSpan
	. "`nCurrentColumnSpan: " gridItemPattern.CurrentColumnSpan
	; gridItemPattern.CurrentContainingGrid should return listEl

ExitApp
Example 13: TablePattern and TableItemPattern
From Microsoft documentation:
TablePattern provides access to a control that acts as a container for a collection of child elements. The children of this element support IUIAutomationTableItemPattern and are organized in a two-dimensional logical coordinate system that can be traversed by row and column.
TableItemPattern provides access to a child element in a container that supports IUIAutomationTablePattern.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, explore C:\
UIA := UIA_Interface()
DriveGet, CDriveName, Label, C:
CDriveName := CDriveName " (C:)"
WinWaitActive, %CDriveName%
explorerEl := UIA.ElementFromHandle(WinActive("A"))
listEl := explorerEl.FindFirstByType("List")

tablePattern := listEl.GetCurrentPatternAs("Table")
MsgBox, % "TablePattern properties: "
	. "`nCurrentRowOrColumnMajor: " tablePattern.CurrentRowOrColumnMajor


rowHeaders := tablePattern.GetCurrentRowHeaders()
rowHeadersDump := ""
for _,header in rowHeaders
	rowHeadersDump .= header.Dump() "`n"
MsgBox, % "TablePattern elements from GetCurrentRowHeaders:`n" rowHeadersDump ; Should be empty, there aren't any row headers
columnHeaders := tablePattern.GetCurrentColumnHeaders()
columnHeadersDump := ""
for _,header in columnHeaders
	columnHeadersDump .= header.Dump() "`n"
MsgBox, % "TablePattern elements from GetCurrentColumnHeaders:`n" columnHeadersDump

editEl := listEl.GetCurrentPatternAs("Grid").GetItem(3,0) ; To test the TableItem pattern, we need to get an element supporting that using Grid pattern...
tableItemPattern := editEl.GetCurrentPatternAs("TableItem")
rowHeaderItems := tableItemPattern.GetCurrentRowHeaderItems()
rowHeaderItemsDump := ""
for _,headerItem in rowHeaderItems
	rowHeaderItemsDump .= headerItem.Dump() "`n"
MsgBox, % "TableItemPattern elements from GetCurrentRowHeaderItems:`n" rowHeaderItemsDump ; Should be empty, there aren't any row headers
columnHeaderItems := tableItemPattern.GetCurrentColumnHeaderItems()
columnHeaderItemsDump := ""
for _,headerItem in columnHeaderItems
	columnHeaderItemsDump .= headerItem.Dump() "`n"
MsgBox, % "TableItemPattern elements from GetCurrentColumnHeaderItems:`n" columnHeaderItemsDump
ExitApp
Example 14: MultipleViewPattern
From Microsoft documentation:
MultipleViewPattern provides access to a control that can switch between multiple representations of the same information or set of child controls.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, explore C:\
UIA := UIA_Interface()
DriveGet, CDriveName, Label, C:
CDriveName := CDriveName " (C:)"
WinWaitActive, %CDriveName%
explorerEl := UIA.ElementFromHandle(WinActive("A"))
listEl := explorerEl.FindFirstByType("List")

mvPattern := listEl.GetCurrentPatternAs("MultipleView")
MsgBox, % "MultipleView properties: "
	. "`nCurrentCurrentView: " (currentView := mvPattern.CurrentCurrentView)

supportedViews := mvPattern.GetCurrentSupportedViews()
viewNames := ""
for _, view in supportedViews {
	viewNames .= mvPattern.GetViewName(view) " (" view ")`n"
}
MsgBox, % "This MultipleView supported views:`n" viewNames
MsgBox, % "Press OK to set MultipleView to view 4."
mvPattern.SetCurrentView(4)

Sleep, 500
MsgBox, % "Press OK to reset back to view " currentView "."
mvPattern.SetCurrentView(currentView)

ExitApp
Example 15: SelectionPattern and SelectionItemPattern
From Microsoft documentation:
SelectionPattern provides access to a control that contains selectable child items. The children of this element support IUIAutomationSelectionItemPattern.
SelectionItemPattern provides access to the selectable child items of a container control that supports IUIAutomationSelectionPattern.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, explore C:\
UIA := UIA_Interface()
DriveGet, CDriveName, Label, C:
CDriveName := CDriveName " (C:)"
WinWaitActive, %CDriveName%
explorerEl := UIA.ElementFromHandle(WinActive("A"))
listEl := explorerEl.FindFirstByType("List")

selectionPattern := listEl.GetCurrentPatternAs("Selection")
MsgBox, % "SelectionPattern properties: "
	. "`nCurrentCanSelectMultiple: " selectionPattern.CurrentCanSelectMultiple
	. "`nCurrentIsSelectionRequired: " selectionPattern.CurrentIsSelectionRequired

currentSelectionEls := selectionPattern.GetCurrentSelection()
currentSelections := ""
for index,selection in currentSelectionEls
	currentSelections .= index ": " selection.Dump() "`n"

windowsListItem := explorerEl.FindFirstByNameAndType("Windows", "ListItem")
selectionItemPattern := windowsListItem.GetCurrentPatternAs("SelectionItem")
MsgBox, % "ListItemPattern properties for Windows folder list item:"
	. "`nCurrentIsSelected: " selectionItemPattern.CurrentIsSelected
	. "`nCurrentSelectionContainer: " selectionItemPattern.CurrentSelectionContainer.Dump()

MsgBox, % "Press OK to select ""Windows"" folder list item."
selectionItemPattern.Select()
MsgBox, % "Press OK to add to selection ""Program Files"" folder list item."
explorerEl.FindFirstByNameAndType("Program Files", "ListItem").GetCurrentPatternAs("SelectionItem").AddToSelection()
MsgBox, % "Press OK to remove selection from ""Windows"" folder list item."
selectionItemPattern.RemoveFromSelection()

ExitApp
Example 16: WindowPattern and TransformPattern
From Microsoft documentation:
WindowPattern provides access to the fundamental functionality of a window.
TransformPattern provides access to a control that can be moved, resized, or rotated.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

Run, explore C:\
UIA := UIA_Interface()
DriveGet, CDriveName, Label, C:
CDriveName := CDriveName " (C:)"
WinWaitActive, %CDriveName%
explorerEl := UIA.ElementFromHandle(WinActive("A"))
windowPattern := explorerEl.GetCurrentPatternAs("Window")
Sleep, 500
MsgBox, % "WindowPattern properties: "
	. "`nCurrentCanMaximize: " windowPattern.CurrentCanMaximize
	. "`nCurrentCanMinimize: " windowPattern.CurrentCanMinimize
	. "`nCurrentIsModal: " windowPattern.CurrentIsModal
	. "`nCurrentIsTopmost: " windowPattern.CurrentIsTopmost
	. "`nCurrentWindowVisualState: " (visualState := windowPattern.CurrentWindowVisualState) " (" UIA_Enum.WindowVisualState(visualState) ")"
	. "`nCurrentWindowInteractionState: " (interactionState := windowPattern.CurrentWindowInteractionState) " (" UIA_Enum.WindowInteractionState(interactionState) ")"
Sleep, 50
MsgBox, Press OK to try minimizing
windowPattern.SetWindowVisualState(UIA_Enum.WindowVisualState_Minimized)

Sleep, 500
MsgBox, Press OK to bring window back to normal
windowPattern.SetWindowVisualState(UIA_Enum.WindowVisualState_Normal)

transformPattern := explorerEl.GetCurrentPatternAs("TransformPattern") ; Note: for some reason TransformPattern2 doesn't extend TransformPattern properties/methods. If we called GetCurrentPatternAs("Transform"), we would get TransformPattern2 and wouldn't be able to access these properties and methods. Thats why previously we could use GetCurrentPatternAs("Window") instead of GetCurrentPatternAs("WindowPattern"), but here we need GetCurrentPatternAs("TransformPattern") to get TransformPattern explicitly.
Sleep, 500
MsgBox, % "TransformPattern properties: "
	. "`nCurrentCanMove: " transformPattern.CurrentCanMove
	. "`nCurrentCanResize: " transformPattern.CurrentCanResize
	. "`nCurrentCanRotate: " transformPattern.CurrentCanRotate

MsgBox, Press OK to move to coordinates x100 y200
transformPattern.Move(100,200)

Sleep, 500
MsgBox, Press OK to resize to w600 h400
transformPattern.Resize(600,400)

Sleep, 500
MsgBox, Press OK to close window
windowPattern.Close()

TextPattern and TextRange
TextPattern provides access to a control that contains text.

Code: Select all

TextPattern properties:
DocumentRange ; Returns the TextRange for all the text inside the element
SupportedTextSelection ; Whether selecting text is supported

TextPattern methods:
RangeFromPoint(x,y) ; Retrieves an empty TextRange nearest to the specified screen coordinates
RangeFromChild(child) ; Retrieves a text range enclosing a child element such as an image, hyperlink, Microsoft Excel spreadsheet, or other embedded object.
GetSelection() ; Returns the currently selected text
GetVisibleRanges() ; Retrieves an array of disjoint text ranges from a text-based control where each text range represents a contiguous span of visible text
From Microsoft documentation:
TextRange provides access to a span of continuous text in a container that supports the TextPattern interface. TextRange can be used to select, compare, and retrieve embedded objects from the text span. The interface uses two endpoints to delimit where the text span starts and ends. Disjoint spans of text are represented by a TextRangeArray, which is an array of TextRange interfaces.

Code: Select all

TextRange doesn't have any properties.

TextRange methods:
Clone() ; Returns a copy of the TextRange (retrieves a new IUIAutomationTextRange identical to the original and inheriting all properties of the original).
Compare(comparisonTextRange) ; Compares whether this TextRange has the same endpoints as comparisonTextRange
CompareEndPoints(srcEndPoint, comparisonTextRange, targetEndPoint) ; Retrieves a value that specifies whether the start or end endpoint of this text range is the same as the start or end endpoint of comparisonTextRange. Returns a negative value if the caller's endpoint occurs earlier in the text than the target endpoint; 0 if the caller's endpoint is at the same location as the target endpoint; or a positive value if the caller's endpoint occurs later in the text than the target endpoint. srcEndPoint and targetEndPoint need to be TextPatternRangeEndpoint enums.
ExpandToEnclosingUnit(unit=6) ; Normalizes the text range by the specified text unit. The range is expanded if it is smaller than the specified unit, or shortened if it is longer than the specified unit. unit needs to be a TextUnit enum (default is TextUnit_Document == 6)
FindAttribute(attr, val, backward=False) ; Retrieves a text range subset that has the specified text attribute value. attr needs to be a UIA_TextAttributeId enum, and val the desired value (some can be strings, others text attribute enums such as BulletStyle enum)
FindText(text, backward=False, ignoreCase=False) ; Retrieves a text range subset that contains the specified text.		GetAttributeValue(attr) ; Retrieves the value of the specified text attribute across the entire text range. attr needs to be a UIA_TextAttributeId enum.
GetBoundingRectangles() { ; Returns an array of bounding rectangle objects {x:top left X-coord,y:top left Y-coord,w:width,h:height} for each fully or partially visible line of text in a text range.
GetEnclosingElement() ; Returns the innermost UI Automation element that encloses the text range.
GetText(maxLength=-1) ; Returns the plain text of the text range. maxLength is the maximum length of the string to return, or -1 if no limit is required.
Move(unit, count) ; Moves the text range forward or backward by the specified number of text units. unit needs to be a TextUnit enum.
MoveEndpointByUnit(endpoint, unit, count) ; Moves one endpoint of the text range the specified number of text units within the document range. endpoint needs to be TextPatternRangeEndpoint enum. unit needs to be a TextUnit enum.
MoveEndpointByRange(srcEndPoint, range, targetEndPoint) ; Moves one endpoint of the current text range to the specified endpoint of a second text range. srcEndPoint and targetEndPoint need to be TextPatternRangeEndpoint enums.
Select() ; Selects the span of text that corresponds to this text range, and removes any previous selection.
AddToSelection() ; Adds the text range to the collection of selected text ranges in a control that supports multiple, disjoint spans of selected text.
RemoveFromSelection() ; Removes the text range from an existing collection of selected text in a text container that supports multiple, disjoint selections.
ScrollIntoView(alignToTop) ; Causes the text control to scroll until the text range is visible in the viewport. alignToTop is a boolean value.
GetChildren() { ; Retrieves a collection of all embedded objects that fall within the text range.
Note that TextPattern nor TextRange can't be used to change the text itself, only to get information about the text or select text. To change the text, UIA_Element.SetValue(val) can be used.

Example17. Using TextPattern and TextRange

Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2

#include <UIA_Interface>

lorem =
(
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
)

Run notepad.exe
;WinActivate, ahk_exe notepad.exe
UIA := UIA_Interface()
WinWaitActive, ahk_exe notepad.exe
;MsgBox, % UIA.TextPatternRangeEndpoint_Start " " UIA.TextPatternRangeEndpoint_End " " UIA.TextUnit_Character
NotepadEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe"))
editEl := NotepadEl.FindFirstBy("ControlType=Document OR ControlType=Edit") ; Get the Edit or Document element (differs between UIAutomation versions)
editEl.CurrentValue := lorem ; Set the text to our sample text
textPattern := editEl.GetCurrentPatternAs("Text") ; Get the TextPattern

MsgBox, % "TextPattern properties:"
	. "`nDocumentRange: returns the TextRange for all the text inside the element"
	. "`nSupportedTextSelection: " textPattern.SupportedTextSelection
	. "`n`nTextPattern methods:"
	. "`nRangeFromPoint(x,y): retrieves an empty TextRange nearest to the specified screen coordinates"
	. "`nRangeFromChild(child): retrieves a text range enclosing a child element such as an image, hyperlink, Microsoft Excel spreadsheet, or other embedded object."
	. "`nGetSelection(): returns the currently selected text"
	. "`nGetVisibleRanges(): retrieves an array of disjoint text ranges from a text-based control where each text range represents a contiguous span of visible text"

wholeRange := textPattern.DocumentRange ; Get the TextRange for all the text inside the Edit element

MsgBox, % "To select a certain phrase inside the text, use FindText() method to get the corresponding TextRange, then Select() to select it.`n`nPress OK to select the text ""dolor sit amet"""
WinActivate, ahk_exe notepad.exe
wholeRange.FindText("dolor sit amet").Select()
Sleep, 1000

; For the next example we need to clone the TextRange, because some methods change the supplied TextRange directly (here we don't want to change our original wholeRange TextRange). An alternative would be to use wholeRange, and after moving the endpoints and selecting the new range, we could call ExpandToEnclosingUnit() to reset the endpoints and get the whole TextRange back 
textSpan := wholeRange.Clone()

MsgBox, % "To select a span of text, we need to move the endpoints of the TextRange. This can be done with MoveEndpointByUnit.`n`nPress OK to select the text with startpoint of 28 characters from start`nand 390 characters from the end of the sample text"
WinActivate, ahk_exe notepad.exe
textSpan.MoveEndpointByUnit(UIA.TextPatternRangeEndpoint_Start, UIA.TextUnit_Character, 28) ; Move 28 characters from the start of the sample text
textSpan.MoveEndpointByUnit(UIA.TextPatternRangeEndpoint_End, UIA.TextUnit_Character, -390) ; Move 390 characters backwards from the end of the sample text
textSpan.Select()
Sleep, 1000

MsgBox, % "We can also get the location of texts. Press OK to test it"
br := wholeRange.GetBoundingRectangles()
;MsgBox, % br.MaxIndex() " " br[1].l " " br[1].t " " br[1].r " " br[1].b
for _, v in br {
	RangeTip(v.x, v.y, v.w, v.h)
	Sleep, 1000
}
RangeTip()

ExitApp

RangeTip(x:="", y:="", w:="", h:="", color:="Red", d:=2) ; from the FindText library, credit goes to feiyue
{
  local
  static id:=0
  if (x="")
  {
    id:=0
    Loop 4
      Gui, Range_%A_Index%: Destroy
    return
  }
  if (!id)
  {
    Loop 4
      Gui, Range_%A_Index%: +Hwndid +AlwaysOnTop -Caption +ToolWindow
        -DPIScale +E0x08000000
  }
  x:=Floor(x), y:=Floor(y), w:=Floor(w), h:=Floor(h), d:=Floor(d)
  Loop 4
  {
    i:=A_Index
    , x1:=(i=2 ? x+w : x-d)
    , y1:=(i=3 ? y+h : y-d)
    , w1:=(i=1 or i=3 ? w+2*d : d)
    , h1:=(i=2 or i=4 ? h+2*d : d)
    Gui, Range_%i%: Color, %color%
    Gui, Range_%i%: Show, NA x%x1% y%y1% w%w1% h%h1%
  }
}
Example 18. TextRange and TextChangedEventHandler example
For this example, open up Word and create a new document (the window name should be "Document1 - Word"). Then type some text into the main body of the document: use at least two different fonts, and create a bullet list (filled bullets). After doing that, run the following example.

Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2

#include <UIA_Interface>

UIA := UIA_Interface() ; Initialize UIA interface
program := "Document1 - Word"
WinActivate, %program%
WinWaitActive, %program%
wordEl := UIA.ElementFromHandle(WinExist(program))
bodyEl := wordEl.FindFirstBy("AutomationId=Body") ; First get the body element of Word
textPattern := bodyEl.GetCurrentPatternAs("Text") ; Get TextPattern for the body element
document := textPattern.DocumentRange ; Get the TextRange for the whole document

MsgBox, % "Current text inside Word body element:`n" document.GetText() ; Display the text from the TextRange
WinActivate, %program%

MsgBox, % "We can get text from a specific attribute, such as text within a ""bullet list"".`nTo test this, create a bullet list (with filled bullets) in Word and press OK."
MsgBox, % "Found the following text in bullet list:`n" document.FindAttribute(UIA_Enum.UIA_BulletStyleAttributeId, UIA_Enum.BulletStyle_FilledRoundBullet).GetText()

Loop {
	InputBox, font, Find, % "Search text in Word by font. Type some example text in Word.`nThen write a font (such as ""Calibri"") and press OK`n`nNote that this is case-sensitive, and fonts start with a capital letter`n(""calibri"" is not the same as ""Calibri"")"
	if ErrorLevel
		break
	else if !font
		MsgBox, You need to type a font to search!
	else if (found := document.FindAttribute(UIA_Enum.UIA_FontNameAttributeId, font).GetText())
		MsgBox, % "Found the following text:`n" found
	else
		MsgBox, No text with the font %font% found!
}

MsgBox, % "Press OK to create a new EventHandler for the TextChangedEvent.`nTo test this, type some new text inside Word, and a tooltip should pop up.`n`nTo exit the script, press F5."
handler := UIA_CreateEventHandler("TextChangedEventHandler") ; Create a new event handler that points to the function TextChangedEventHandler, which must accept two arguments: element and eventId.
UIA.AddAutomationEventHandler(UIA.Text_TextChangedEventId, wordEl,,, handler) ; Add a new automation handler for the TextChanged event. Note that we can only use wordEl here, not bodyEd, because the event is handled for the whole window.
OnExit("ExitFunc") ; Set up an OnExit call to clean up the handler when exiting the script

return

TextChangedEventHandler(el, eventId) {
	try {
		textPattern := el.GetCurrentPatternAs("Text")
		ToolTip, % "You changed text in Word:`n`n" textPattern.DocumentRange.GetText()
		SetTimer, RemoveToolTip, -2000
	}
}

ExitFunc() {
	global UIA, handler, bodyEl
	try UIA.RemoveAutomationEventHandler(UIA.Text_ChangedEventId, bodyEl, handler) ; Remove the event handler. Alternatively use UIA.RemoveAllEventHandlers() to remove all handlers
}

RemoveToolTip:
	ToolTip
	return

F5::ExitApp
Last edited by Descolada on 04 Jul 2022, 14:30, edited 3 times in total.

ahk7
Posts: 572
Joined: 06 Nov 2013, 16:35

Re: UIAutomation with a focus on Chrome

Post by ahk7 » 06 Jun 2022, 09:44

:clap:

guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: UIAutomation with a focus on Chrome

Post by guest3456 » 06 Jun 2022, 09:44

good work


User avatar
joedf
Posts: 8937
Joined: 29 Sep 2013, 17:08
Location: Canada
Contact:

Re: UIAutomation with a focus on Chrome

Post by joedf » 06 Jun 2022, 10:29

Wow, this is handy and neat! :+1:
Image Image Image Image Image
Windows 10 x64 Professional, Intel i5-8500, NVIDIA GTX 1060 6GB, 2x16GB Kingston FURY Beast - DDR4 3200 MHz | [About Me] | [About the AHK Foundation] | [Courses on AutoHotkey]
[ASPDM - StdLib Distribution] | [Qonsole - Quake-like console emulator] | [LibCon - Autohotkey Console Library]

robodesign
Posts: 932
Joined: 30 Sep 2017, 03:59
Location: Romania
Contact:

Re: UIAutomation with a focus on Chrome

Post by robodesign » 06 Jun 2022, 10:36

Awesomeness! Thank you very much for taking the time to do this work! Excellent!

I'm going to test this asap.

Please publish on the repository the examples provided as well.

Best regards, Marius.
-------------------------
KeyPress OSD v4: GitHub or forum. (presentation video)
Quick Picto Viewer: GitHub or forum.
AHK GDI+ expanded / compilation library (on GitHub)
My home page.

viv
Posts: 217
Joined: 09 Dec 2020, 17:48

Re: UIAutomation with a focus on Chrome

Post by viv » 08 Jun 2022, 09:57

I would like to ask a question
In the ACC
It have path that can be accurately located
For example, 4.1.1.4.1.2.5.4
You can use this to locate the control instantly

Is there a similar way for UIA?
I've looked at it and it seems to enumerate and find the control that matches?

Let's say GetCurrentURL
Using path in ACC can be done between 0-16ms
In uia I use this method
It takes more than 600-700ms ....

Code: Select all

 
 browserExe := "msedge.exe"
 cUIA := new UIA_Browser("ahk_exe " browserExe) 
 url :=  cUIA.GetCurrentURL()

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 08 Jun 2022, 13:54

viv wrote:
08 Jun 2022, 09:57
I would like to ask a question
In the ACC
It have path that can be accurately located
For example, 4.1.1.4.1.2.5.4
You can use this to locate the control instantly

Is there a similar way for UIA?
I've looked at it and it seems to enumerate and find the control that matches?

Let's say GetCurrentURL
Using path in ACC can be done between 0-16ms
In uia I use this method
It takes more than 600-700ms ....

Code: Select all

 
 browserExe := "msedge.exe"
 cUIA := new UIA_Browser("ahk_exe " browserExe) 
 url :=  cUIA.GetCurrentURL()
Hello,
I think it's possible to implement the same method using TreeViewer, but it is inadvisable, because the path can change at any time due to UI updates, changes by the user etc. The problem here actually is that Edge doesn't play nice with UIA: finding elements sometimes fails, takes much longer than in other windows, and I haven't figured out the reason for it yet. In Chrome the GetCurrentURL takes 0-32ms...

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 08 Jun 2022, 17:40

@Descolada - Is there a chance that your UIAViewer.ahk (probably as an opt-in/Easter-egg) would deliver all acc-paths too? Unfortunately @jeeswg's 'JEE_AccGetTextAll.ahk' hasn't made it into AccViewer.ahk... :|
...and YES, your scripts are AMAZING! :clap: :thumbup:

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 10 Jun 2022, 02:04

This post is unrelated to UIAutomation, but instead takes a little detour to Acc.ahk
BoBo wrote:
08 Jun 2022, 17:40
@Descolada - Is there a chance that your UIAViewer.ahk (probably as an opt-in/Easter-egg) would deliver all acc-paths too? Unfortunately @jeeswg's 'JEE_AccGetTextAll.ahk' hasn't made it into AccViewer.ahk... :|
...and YES, your scripts are AMAZING! :clap: :thumbup:
@BoBo, I added your requested feature to UIAViewer. This was actually quite difficult, because it seems the AccViewer bottom-to-top style path finding actually doesn't work properly, so I had to implement a top-to-bottom approach. Also learned a few new things about Acc:
1) Acc tree and UIA trees differ in important ways, so one cannot be substituted for the other but instead generated separately. For UIAViewer this means that if the Acc path option is enabled, the whole Acc tree will also be generated for any windows the mouse is hovering over. This tree will be generated only once though, and updated when either "Start capturing" or "Construct tree for whole Window" buttons are pressed. So if the window content changes while "capturing", the Acc path might not be reliable.

2) AccViewer implements a bottom-to-top approach of generating the Acc path. This means it starts from the node under the cursor, then iterates over Acc_Children(Acc_Parent(node)) to find the node location, and so on to the main window itself. But it seems that Acc_Parent or Acc_Children sometimes fail to return the correct node, so the original nodes location cannot be found! For example in Notepad++, when looking at tabs with AccViewer, the reported path is "4", but the actual path (gotten with JEE_AccGetTextAll or UIAViewer) is "4.8.4". This often happens in deep trees like browser windows, making AccViewer very unreliable. The top-to-bottom approach works fine, but involves generating the tree for the whole window beforehand, then finding the node of interest and reporting the path.

3) Acc tree nodes can be OBJECTS or VARIANTS. This is an important difference, because object nodes have methods available (accFocus, accValue, accDoDefaultAction etc). Variant type nodes are usually the final child nodes that do NOT have methods, but instead rely on the methods of their parents. An example involving Notepad++: if looking for a Notepad++ tab, then JEE_AccGetTextAll and UIAViewer reports a path of "4.8.4 cN" where N is the number of the tab. The "c" denotes that the node in question is a child (variant) type, so not an object. This path cannot be directly copied into Acc_Get, instead the "4.8.4" part needs to be copied, and then the "cN" part manually applied. So for example if we are interested in activating the second tab with Acc path of "4.8.4 c2", we need to do the following:

Code: Select all

oAcc := Acc_Get("Object", "4.8.4", 0, "ahk_id " WinExist("ahk_exe notepad++.exe"))
oAcc.accDoDefaultAction(Acc_Children(oAcc)[2]) ; The "2" part of "c2" was used here to get the second child
A more generalized Notepad++ example to look for a tab with a specific name:

Code: Select all

#include <Acc>
NotepadPPActivateTab("test")

NotepadPPActivateTab(tabName, hwnd=0) {
	hwnd := hwnd?hwnd:WinExist("ahk_exe notepad++.exe")
	WinActivate, ahk_id %hwnd% ; Activate the Notepad++ window, otherwise accDoDefaultAction won't work
	WinWaitActive, ahk_id %hwnd%
	oAcc := Acc_Get("Object", "4.8.4", 0, "ahk_id " hwnd) ; Get the object of the tab bar, because tabs themselves are not objects and accDoDefaultAction wouldn't work
	for _, child in Acc_Children(oAcc) { ; Get all tabs from the tab bar
		if InStr(oAcc.accName(child), tabName) { ; oAcc.accName(child) calls the accName method from the tab bar object and as an argument passes the tab variant. This means that the tab bar object itself returns the name of the tab!
			try
				return oAcc.accDoDefaultAction(child) ; If the name of the tab matched, then now try to do the default action on the tab element, again using the tab bar object
			Catch
				MsgBox, Error, Something went wrong! Does a tab with the text "%tabName%" exist and is the tab visible on the screen?
		}
	}
	MsgBox, Error, Something went wrong! Does a tab with the text "%tabName%" actually exist?
}
Perhaps somebody is interested in fixing AccViewer.ahk with a top-to-bottom approach, but I think I don't want to work with legacy code any more (Oleacc/MSAA is not actively being developed anymore, but UIA is)... This was quite the detour. :)

EDIT: User @rommmcek has implemented the top-to-bottom approach in AccViewer: viewtopic.php?p=382050#p382050
Last edited by Descolada on 11 Jun 2022, 00:35, edited 2 times in total.

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 10 Jun 2022, 02:32

@Descolada - I'm smashed by the effort you've put into this request. Thx a million <kowtow> :)

gya
Posts: 25
Joined: 04 Nov 2021, 01:22

Re: UIAutomation with a focus on Chrome

Post by gya » 10 Jun 2022, 09:20

@Descolada
I have also written a AccViewer analog for UIA, the UIAViewer
Thank you very much. With "UIAViewer.ahk" I could understand your examples and use NotePad in French.
Sincerely yours.

User avatar
rommmcek
Posts: 1470
Joined: 15 Aug 2014, 15:18

Re: UIAutomation with a focus on Chrome

Post by rommmcek » 10 Jun 2022, 23:17

Descolada wrote:
10 Jun 2022, 02:04
Perhaps somebody is interested in fixing AccViewer.ahk with a top-to-bottom approach
Somebody already did, not because he was an expert, just noticed some patterns.

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 11 Jun 2022, 00:33

rommmcek wrote:
10 Jun 2022, 23:17
Descolada wrote:
10 Jun 2022, 02:04
Perhaps somebody is interested in fixing AccViewer.ahk with a top-to-bottom approach
Somebody already did, not because he was an expert, just noticed some patterns.
Nice! Somehow I totally missed that in the main Acc thread, modified my previous post to reflect that.
Tried it out and AccViewer RP3.5 works fine in most windows, but in some windows it returns multiple paths? For example Notepad++: when selecting a tab, it shows 6 different paths, the same when looking at toolbar items.

newbie007
Posts: 20
Joined: 21 Jan 2022, 06:34

Re: UIAutomation with a focus on Chrome

Post by newbie007 » 11 Jun 2022, 04:46

Hello,

I wanted to test some examples but when I try it I get an error even tho the libraries are in the same folder as my ahk script.
C:\Users\feyz\Downloads\AHK-Studio-master\Projects\test.ahk (5) : ==> Function library not found.
Specifically: #include <UIA_Interface>

I used this code:

Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2

#include <UIA_Interface>

F5::ExitApp
F1::
Run, notepad.exe
UIA := UIA_Interface() ; Initialize UIA interface
WinWaitActive, ahk_exe notepad.exe
npEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe")) ; Get the element for the Notepad window
editEl := npEl.FindFirstByType("Edit") ; Find the first Edit control (in Notepad there is only one)
editEl.SetValue("Lorem ipsum") ; Set the value for the edit control
return
What do I do wrong?

User avatar
rommmcek
Posts: 1470
Joined: 15 Aug 2014, 15:18

Re: UIAutomation with a focus on Chrome

Post by rommmcek » 11 Jun 2022, 04:50

@Descolada:
Spoiler
Last edited by rommmcek on 11 Jun 2022, 07:37, edited 1 time in total.

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 11 Jun 2022, 05:15

newbie007 wrote:
11 Jun 2022, 04:46
Hello,

I wanted to test some examples but when I try it I get an error even tho the libraries are in the same folder as my ahk script.
C:\Users\feyz\Downloads\AHK-Studio-master\Projects\test.ahk (5) : ==> Function library not found.
Specifically: #include <UIA_Interface>

What do I do wrong?
I refer to the AHK documentation #include page. To fix this, create a "Lib" folder in your script directory and put the library files (UIA_Interface.ahk, UIA_Constants.ahk, UIA_Browser.ahk) in there. Other way would be to change #include <UIA_Interface> to #include UIA_Interface.ahk.

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: UIAutomation with a focus on Chrome

Post by Xeo786 » 11 Jun 2022, 05:41

Thanks for your Great work, :D
Descolada wrote:
11 Jun 2022, 05:15
Other way would be to change #include <UIA_Interface> to #include UIA_Interface.ahk.
I faced #include issues and I changed something like this and everything in the Example folder start working
image.png
image.png (99.27 KiB) Viewed 21321 times
UIAViewer.ahk do not highlights element in browser for me, its just highlights Documents frame,
I have a AccViewer.ahk version that does highlight every element inside Chrome, and I might make few changes to UIAViewer.ahk and enable the ability to highlight chrome elements
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

Descolada
Posts: 1077
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 11 Jun 2022, 06:08

@Xeo786, I'll include a fix for the UIA_Constants.ahk include in the next update.

Highlighting document elements works fine here:
UIAViewer.png
UIAViewer.png (238.08 KiB) Viewed 21265 times
Does the problem exist using the Accessibility Insights tool as well? What if you run Chrome forcefully in UIAccessibility mode (Run chrome.exe --force-renderer-accessibility) or you enable UIA under chrome://accessibility tab?

If you find the bug, be sure to share the fixed code ;)

newbie007
Posts: 20
Joined: 21 Jan 2022, 06:34

Re: UIAutomation with a focus on Chrome

Post by newbie007 » 11 Jun 2022, 06:12

Descolada wrote:
11 Jun 2022, 05:15
newbie007 wrote:
11 Jun 2022, 04:46
Hello,

I wanted to test some examples but when I try it I get an error even tho the libraries are in the same folder as my ahk script.
C:\Users\feyz\Downloads\AHK-Studio-master\Projects\test.ahk (5) : ==> Function library not found.
Specifically: #include <UIA_Interface>

What do I do wrong?
I refer to the AHK documentation #include page. To fix this, create a "Lib" folder in your script directory and put the library files (UIA_Interface.ahk, UIA_Constants.ahk, UIA_Browser.ahk) in there. Other way would be to change #include <UIA_Interface> to #include UIA_Interface.ahk.
not worked with

Code: Select all

#Include UIA_Interface.ahk
but when I did

Code: Select all

#Include Lib\UIA_Interface.ahk
it worked.
Thanks alot! very powerful library :)

Post Reply

Return to “Scripts and Functions (v1)”