Clicking on an object using UIA? (UI automation) Topic is solved

[TheBeginner]

I found this function in the forum as an Self-contained example Of working with UIA,
Can anybody help me adjust it to target and click an element If I already know its Name?

Also if the approach to this can be done differently/better that can also help me out

Basically I'm trying to click on a Windows 10 application such as the calculator, the regular ways don't work that well with that

Code: Select all

getControlNameByHwnd(_, controlHwnd){
	UIAutomation := ComObjCreate("{ff48dba4-60ef-4201-aa87-54103eef594e}", "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	DllCall(NumGet(NumGet(UIAutomation+0)+6*A_PtrSize), "Ptr", UIAutomation, "Ptr", controlHwnd, "Ptr*", IUIAutomationElement)
	DllCall(NumGet(NumGet(IUIAutomationElement+0)+29*A_PtrSize), "Ptr", IUIAutomationElement, "Ptr*", automationId)
	ret := StrGet(automationId,, "UTF-16")
	DllCall("oleaut32\SysFreeString", "Ptr", automationId)
	ObjRelease(IUIAutomationElement)
	ObjRelease(UIAutomation)
	return ret
}

https://www.autohotkey.com/boards/viewtopic.php?t=25017

[nitrofenix]

Have only played with UIA fleetingly, so might be barking up the wrong tree, but have you tried accessing either UIA_BoundingRectanglePropertyId or UIA_CenterPointPropertyId for the IUIAutomationElement?

Assuming you can process/access the returned coordinates, it should allow you to then feed the X and Y into MouseClick()?

[TheBeginner]

It's above my knowledge, you know how to do that?
I can partially understand the code but not good enough yet, I don't exactly know what each thing does there
and how it gets triggered in general

[nitrofenix]

Haven't had time to play with it in-depth, but would first suggest checking out this post and checking out both Inspect.ahk and UIA.ahk from the linked GitHub, since it will shortcut your way around looking-up/coding vtable offsets.

This is how they've achieved it in Inspect.ahk as part of an IUIAutomationElement class imported from UIA.ahk -- propertyId for a bounding box in this case would be 30001, as listed in the MS docs:

Code: Select all

; Retrieves a property value from the cache for this UI Automation element.
GetCachedPropertyValue(propertyId){
static _,_v:=variant(_)
_Error(DllCall(vt(this._p,12),"ptr",this._p,"int",propertyId,"ptr",_v),"GetCachedPropertyValue")
return GetVariantValue(_v)
}
; ---- snip ----
GetVariantValue(v){
if (type:=NumGet(v+0,"ushort"))&0x2000
	return GetSafeArrayValue(NumGet(v+8),type&0xFF)
else return NumGet(v+8,varianttype(type).2)
}

And the value is accessed in Inspect.ahk using:

Code: Select all

global $e:=new IUIAutomationElement
; ---- snip ----
"BoundingRectangle:                  " BoundingRectangle($e.GetCachedPropertyValue(UIA_Property("BoundingRectangle"))) 		"`n" ;
; ---- snip ----
BoundingRectangle(t){
	return (IsObject(t)?"l:" SubStr(t.1,1,InStr(t.1,".")-1) " t:" SubStr(t.2,1,InStr(t.2,".")-1) " w:" SubStr(t.3,1,InStr(t.3,".")-1) " h:" SubStr(t.4,1,InStr(t.4,".")-1):)
}

And the UIA_Property function from UIA.ahk:

Code: Select all

UIA_Property(n){
	static id:={RuntimeId:30000,BoundingRectangle:30001,ProcessId:30002,ControlType:30003,LocalizedControlType:30004,Name:30005,AcceleratorKey:30006,AccessKey:30007,HasKeyboardFocus:30008,IsKeyboardFocusable:30009,IsEnabled:30010,AutomationId:30011,ClassName:30012,HelpText:30013,ClickablePoint:30014,Culture:30015,IsControlElement:30016,IsContentElement:30017,LabeledBy:30018,IsPassword:30019,NativeWindowHandle:30020,ItemType:30021,IsOffscreen:30022,Orientation:30023,FrameworkId:30024,IsRequiredForForm:30025,ItemStatus:30026,IsDockPatternAvailable:30027,IsExpandCollapsePatternAvailable:30028,IsGridItemPatternAvailable:30029,IsGridPatternAvailable:30030,IsInvokePatternAvailable:30031,IsMultipleViewPatternAvailable:30032,IsRangeValuePatternAvailable:30033,IsScrollPatternAvailable:30034,IsScrollItemPatternAvailable:30035,IsSelectionItemPatternAvailable:30036,IsSelectionPatternAvailable:30037,IsTablePatternAvailable:30038,IsTableItemPatternAvailable:30039,IsTextPatternAvailable:30040,IsTogglePatternAvailable:30041,IsTransformPatternAvailable:30042,IsValuePatternAvailable:30043,IsWindowPatternAvailable:30044,ValueValue:30045,ValueIsReadOnly:30046,RangeValueValue:30047,RangeValueIsReadOnly:30048,RangeValueMinimum:30049,RangeValueMaximum:30050,RangeValueLargeChange:30051,RangeValueSmallChange:30052,ScrollHorizontalScrollPercent:30053,ScrollHorizontalViewSize:30054,ScrollVerticalScrollPercent:30055,ScrollVerticalViewSize:30056,ScrollHorizontallyScrollable:30057,ScrollVerticallyScrollable:30058,SelectionSelection:30059,SelectionCanSelectMultiple:30060,SelectionIsSelectionRequired:30061,GridRowCount:30062,GridColumnCount:30063,GridItemRow:30064,GridItemColumn:30065,GridItemRowSpan:30066,GridItemColumnSpan:30067,GridItemContainingGrid:30068,DockDockPosition:30069,ExpandCollapseExpandCollapseState:30070,MultipleViewCurrentView:30071,MultipleViewSupportedViews:30072,WindowCanMaximize:30073,WindowCanMinimize:30074,WindowWindowVisualState:30075,WindowWindowInteractionState:30076,WindowIsModal:30077,WindowIsTopmost:30078,SelectionItemIsSelected:30079,SelectionItemSelectionContainer:30080,TableRowHeaders:30081,TableColumnHeaders:30082,TableRowOrColumnMajor:30083,TableItemRowHeaderItems:30084,TableItemColumnHeaderItems:30085,ToggleToggleState:30086,TransformCanMove:30087,TransformCanResize:30088,TransformCanRotate:30089,IsLegacyIAccessiblePatternAvailable:30090,LegacyIAccessibleChildId:30091,LegacyIAccessibleName:30092,LegacyIAccessibleValue:30093,LegacyIAccessibleDescription:30094,LegacyIAccessibleRole:30095,LegacyIAccessibleState:30096,LegacyIAccessibleHelp:30097,LegacyIAccessibleKeyboardShortcut:30098,LegacyIAccessibleSelection:30099,LegacyIAccessibleDefaultAction:30100,AriaRole:30101,AriaProperties:30102,IsDataValidForForm:30103,ControllerFor:30104,DescribedBy:30105,FlowsTo:30106,ProviderDescription:30107,IsItemContainerPatternAvailable:30108,IsVirtualizedItemPatternAvailable:30109,IsSynchronizedInputPatternAvailable:30110}
	, name:={30000:"UIA_RuntimeIdPropertyId",30001:"UIA_BoundingRectanglePropertyId",30002:"UIA_ProcessIdPropertyId",30003:"UIA_ControlTypePropertyId",30004:"UIA_LocalizedControlTypePropertyId",30005:"UIA_NamePropertyId",30006:"UIA_AcceleratorKeyPropertyId",30007:"UIA_AccessKeyPropertyId",30008:"UIA_HasKeyboardFocusPropertyId",30009:"UIA_IsKeyboardFocusablePropertyId",30010:"UIA_IsEnabledPropertyId",30011:"UIA_AutomationIdPropertyId",30012:"UIA_ClassNamePropertyId",30013:"UIA_HelpTextPropertyId",30014:"UIA_ClickablePointPropertyId",30015:"UIA_CulturePropertyId",30016:"UIA_IsControlElementPropertyId",30017:"UIA_IsContentElementPropertyId",30018:"UIA_LabeledByPropertyId",30019:"UIA_IsPasswordPropertyId",30020:"UIA_NativeWindowHandlePropertyId",30021:"UIA_ItemTypePropertyId",30022:"UIA_IsOffscreenPropertyId",30023:"UIA_OrientationPropertyId",30024:"UIA_FrameworkIdPropertyId",30025:"UIA_IsRequiredForFormPropertyId",30026:"UIA_ItemStatusPropertyId",30027:"UIA_IsDockPatternAvailablePropertyId",30028:"UIA_IsExpandCollapsePatternAvailablePropertyId",30029:"UIA_IsGridItemPatternAvailablePropertyId",30030:"UIA_IsGridPatternAvailablePropertyId",30031:"UIA_IsInvokePatternAvailablePropertyId",30032:"UIA_IsMultipleViewPatternAvailablePropertyId",30033:"UIA_IsRangeValuePatternAvailablePropertyId",30034:"UIA_IsScrollPatternAvailablePropertyId",30035:"UIA_IsScrollItemPatternAvailablePropertyId",30036:"UIA_IsSelectionItemPatternAvailablePropertyId",30037:"UIA_IsSelectionPatternAvailablePropertyId",30038:"UIA_IsTablePatternAvailablePropertyId",30039:"UIA_IsTableItemPatternAvailablePropertyId",30040:"UIA_IsTextPatternAvailablePropertyId",30041:"UIA_IsTogglePatternAvailablePropertyId",30042:"UIA_IsTransformPatternAvailablePropertyId",30043:"UIA_IsValuePatternAvailablePropertyId",30044:"UIA_IsWindowPatternAvailablePropertyId",30045:"UIA_ValueValuePropertyId",30046:"UIA_ValueIsReadOnlyPropertyId",30047:"UIA_RangeValueValuePropertyId",30048:"UIA_RangeValueIsReadOnlyPropertyId",30049:"UIA_RangeValueMinimumPropertyId",30050:"UIA_RangeValueMaximumPropertyId",30051:"UIA_RangeValueLargeChangePropertyId",30052:"UIA_RangeValueSmallChangePropertyId",30053:"UIA_ScrollHorizontalScrollPercentPropertyId",30054:"UIA_ScrollHorizontalViewSizePropertyId",30055:"UIA_ScrollVerticalScrollPercentPropertyId",30056:"UIA_ScrollVerticalViewSizePropertyId",30057:"UIA_ScrollHorizontallyScrollablePropertyId",30058:"UIA_ScrollVerticallyScrollablePropertyId",30059:"UIA_SelectionSelectionPropertyId",30060:"UIA_SelectionCanSelectMultiplePropertyId",30061:"UIA_SelectionIsSelectionRequiredPropertyId",30062:"UIA_GridRowCountPropertyId",30063:"UIA_GridColumnCountPropertyId",30064:"UIA_GridItemRowPropertyId",30065:"UIA_GridItemColumnPropertyId",30066:"UIA_GridItemRowSpanPropertyId",30067:"UIA_GridItemColumnSpanPropertyId",30068:"UIA_GridItemContainingGridPropertyId",30069:"UIA_DockDockPositionPropertyId",30070:"UIA_ExpandCollapseExpandCollapseStatePropertyId",30071:"UIA_MultipleViewCurrentViewPropertyId",30072:"UIA_MultipleViewSupportedViewsPropertyId",30073:"UIA_WindowCanMaximizePropertyId",30074:"UIA_WindowCanMinimizePropertyId",30075:"UIA_WindowWindowVisualStatePropertyId",30076:"UIA_WindowWindowInteractionStatePropertyId",30077:"UIA_WindowIsModalPropertyId",30078:"UIA_WindowIsTopmostPropertyId",30079:"UIA_SelectionItemIsSelectedPropertyId",30080:"UIA_SelectionItemSelectionContainerPropertyId",30081:"UIA_TableRowHeadersPropertyId",30082:"UIA_TableColumnHeadersPropertyId",30083:"UIA_TableRowOrColumnMajorPropertyId",30084:"UIA_TableItemRowHeaderItemsPropertyId",30085:"UIA_TableItemColumnHeaderItemsPropertyId",30086:"UIA_ToggleToggleStatePropertyId",30087:"UIA_TransformCanMovePropertyId",30088:"UIA_TransformCanResizePropertyId",30089:"UIA_TransformCanRotatePropertyId",30090:"UIA_IsLegacyIAccessiblePatternAvailablePropertyId",30091:"UIA_LegacyIAccessibleChildIdPropertyId",30092:"UIA_LegacyIAccessibleNamePropertyId",30093:"UIA_LegacyIAccessibleValuePropertyId",30094:"UIA_LegacyIAccessibleDescriptionPropertyId",30095:"UIA_LegacyIAccessibleRolePropertyId",30096:"UIA_LegacyIAccessibleStatePropertyId",30097:"UIA_LegacyIAccessibleHelpPropertyId",30098:"UIA_LegacyIAccessibleKeyboardShortcutPropertyId",30099:"UIA_LegacyIAccessibleSelectionPropertyId",30100:"UIA_LegacyIAccessibleDefaultActionPropertyId",30101:"UIA_AriaRolePropertyId",30102:"UIA_AriaPropertiesPropertyId",30103:"UIA_IsDataValidForFormPropertyId",30104:"UIA_ControllerForPropertyId",30105:"UIA_DescribedByPropertyId",30106:"UIA_FlowsToPropertyId",30107:"UIA_ProviderDescriptionPropertyId",30108:"UIA_IsItemContainerPatternAvailablePropertyId",30109:"UIA_IsVirtualizedItemPatternAvailablePropertyId",30110:"UIA_IsSynchronizedInputPatternAvailablePropertyId"}
	if n is integer
		return name[n]
	else return id[n]
}

Using Include.ahk on a Calculator window resulted in much more information than I needed, but after stepping through the tree, I find the following BoundingRectangle for the "8" button:

Code: Select all

BoundingRectangle:                  l:545 t:709 w:69 h:51

From that, you should then be able to calculate a clickable point for MouseClick().

[TheBeginner]

wow.. sorry I didn't know anybody replied to the post, it seems that to get notified by email you need to manually turn it on in the settings

also rather not movie the mouse to the element, I was hoping more to be using something native like Invoke(); But I don't know how

[malcev]

I recommend You to use this library:
https://www.autohotkey.com/boards/viewtopic.php?t=2216
And learn examples how it works here:
https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/

[TheBeginner]

one of the links points to AutoIt, I don't know much about it other than it's another automation language which if I remember correctly has some common roots with autohotkey.

I believe I found the two functions I need to call to make it work with UIA, but I don't know how to invoke them/structure them correctly that it would work
I'm trying to start with something simple like clicking something on the calculator with this, before I moved to the other program (XD), tried it with aac in more complex application (Adobe XD) but acc doesn't have an ID as far as I know, and names don't always work. hoping that if I get an example of a simple version of it working I can take it from there, since most of it is very complicated

but all I really want to do is just find the object --> and invoke its action

these are the functions, do you know how to use them?

from - https://github.com/neptercn/UIAutomation/blob/master/UIA2.ahk

Code: Select all

; Retrieves the Microsoft UI Automation identifier of the element. 
	; The identifier is unique among sibling elements in a container, and is the same in all instances of the application.
	CurrentAutomationId(){
	_Error(DllCall(vt(this._p,29),"ptr",this._p,"ptr*",retVal),"get_CurrentAutomationId")
	return retVal
	}
	
....

	; Invokes the action of a control, such as a button click. 
	; Calls to this method should return immediately without blocking. However, this behavior depends on the implementation.
	Invoke_Invoke(){
	return _Error(DllCall(vt(this._p,3),"ptr",this._p),"Invoke")
	}

[malcev]

As I said I do not recommend use this wrapper.
If You go to my first link You will see example how to activate IE tab.
From Autoit link You can understand how this technology works and which methods better to use.
Adobe XD has API:
https://adobexdplatform.com/plugin-docs/reference/core/apis.html

[TheBeginner]

Yes, I know it has an API. it can't perform what I need it to
I'll check the link again

[malcev]

I Installed this software.
As I understand MSAA default action does not work with toggle buttons.
Therefore to push such buttons You have to use UIA.
For example to toggle design button we can do like this:

Code: Select all

uia := UIA_Interface()
xd := uia.ElementFromHandle(hwnd)
condition := uia.CreateAndConditionFromArray([uia.CreatePropertyCondition(UIA_ControlTypePropertyId := 30003, button := 50000, VT_I4 := 3), uia.CreatePropertyCondition(UIA_NamePropertyId := 30005, "Design", VT_BSTR := 8)])
element := xd.FindFirst(condition, TreeScope_Descendants := 0x4)
toggle := element.GetCurrentPatternAs("Toggle")
toggle.toggle()

jethrow did not finish his library, therefore You have to add this class to his wrapper
https://github.com/jethrow/UIA_Interface/blob/master/UIA_Interface.ahk

Code: Select all

class UIA_TogglePattern extends UIA_Base
{
   ; https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/nn-uiautomationclient-iuiautomationtogglepattern
   static __IID := "{94cf8058-9b8d-4ab9-8bfd-4cd0a33c8c70}", __PatternID := 10015, __Properties := "CurrentToggleState,4,int`r`nCachedToggleState,5,int"
   Toggle()
   {
      return UIA_Hr(DllCall(this.__Vt(3), "ptr",this.__Value))
   }
}

And use inspect.exe in UI Automation mode for getting info about elements of window and for detecting actions that You need.
https://github.com/guolaok/Python-UIAutomation-for-Windows/tree/master/inspect

[TheBeginner]

your code level is amazing, thanks for helping out, it works !

[guest3456]

jethrow did not finish his library, therefore You have to add this class to his wrapper
https://github.com/jethrow/UIA_Interface/blob/master/UIA_Interface.ahk

Code: Select all

class UIA_TogglePattern extends UIA_Base
{
   ; https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/nn-uiautomationclient-iuiautomationtogglepattern
   static __IID := "{94cf8058-9b8d-4ab9-8bfd-4cd0a33c8c70}", __PatternID := 10015, __Properties := "CurrentToggleState,4,int`r`nCachedToggleState,5,int"
   Toggle()
   {
      return UIA_Hr(DllCall(this.__Vt(3), "ptr",this.__Value))
   }
}

submit a pull request?

[malcev]

I do not know how to do it.
May be You can do it?

[guest3456]

I do not know how to do it.
May be You can do it?

done:
https://github.com/jethrow/UIA_Interface/pull/2