Clicking on an object using UIA? (UI automation) Topic is solved

Get help with using AutoHotkey and its commands and hotkeys
TheBeginner
Posts: 40
Joined: 19 Oct 2018, 12:05

Clicking on an object using UIA? (UI automation)

07 Nov 2019, 14:54

I found this function in the forum as an Self-contained example Of working with UIA,
Can anybody help me adjust it to target and click an element If I already know its Name?

Also if the approach to this can be done differently/better that can also help me out

Basically I'm trying to click on a Windows 10 application such as the calculator, the regular ways don't work that well with that

Code: Select all

getControlNameByHwnd(_, controlHwnd){
	UIAutomation := ComObjCreate("{ff48dba4-60ef-4201-aa87-54103eef594e}", "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	DllCall(NumGet(NumGet(UIAutomation+0)+6*A_PtrSize), "Ptr", UIAutomation, "Ptr", controlHwnd, "Ptr*", IUIAutomationElement)
	DllCall(NumGet(NumGet(IUIAutomationElement+0)+29*A_PtrSize), "Ptr", IUIAutomationElement, "Ptr*", automationId)
	ret := StrGet(automationId,, "UTF-16")
	DllCall("oleaut32\SysFreeString", "Ptr", automationId)
	ObjRelease(IUIAutomationElement)
	ObjRelease(UIAutomation)
	return ret
}
https://www.autohotkey.com/boards/viewtopic.php?t=25017
nitrofenix
Posts: 16
Joined: 20 Sep 2019, 03:26

Re: Clicking on an object using UIA? (UI automation)

07 Nov 2019, 23:05

Have only played with UIA fleetingly, so might be barking up the wrong tree, but have you tried accessing either UIA_BoundingRectanglePropertyId or UIA_CenterPointPropertyId for the IUIAutomationElement?

Assuming you can process/access the returned coordinates, it should allow you to then feed the X and Y into MouseClick()?
TheBeginner
Posts: 40
Joined: 19 Oct 2018, 12:05

Re: Clicking on an object using UIA? (UI automation)

08 Nov 2019, 04:27

It's above my knowledge, you know how to do that?
I can partially understand the code but not good enough yet, I don't exactly know what each thing does there
and how it gets triggered in general
nitrofenix
Posts: 16
Joined: 20 Sep 2019, 03:26

Re: Clicking on an object using UIA? (UI automation)

10 Nov 2019, 22:16

Haven't had time to play with it in-depth, but would first suggest checking out this post and checking out both Inspect.ahk and UIA.ahk from the linked GitHub, since it will shortcut your way around looking-up/coding vtable offsets.

This is how they've achieved it in Inspect.ahk as part of an IUIAutomationElement class imported from UIA.ahk -- propertyId for a bounding box in this case would be 30001, as listed in the MS docs:

Code: Select all

; Retrieves a property value from the cache for this UI Automation element.
GetCachedPropertyValue(propertyId){
static _,_v:=variant(_)
_Error(DllCall(vt(this._p,12),"ptr",this._p,"int",propertyId,"ptr",_v),"GetCachedPropertyValue")
return GetVariantValue(_v)
}
; ---- snip ----
GetVariantValue(v){
if (type:=NumGet(v+0,"ushort"))&0x2000
	return GetSafeArrayValue(NumGet(v+8),type&0xFF)
else return NumGet(v+8,varianttype(type).2)
}
And the value is accessed in Inspect.ahk using:

Code: Select all

global $e:=new IUIAutomationElement
; ---- snip ----
"BoundingRectangle:                  " BoundingRectangle($e.GetCachedPropertyValue(UIA_Property("BoundingRectangle"))) 		"`n" ;
; ---- snip ----
BoundingRectangle(t){
	return (IsObject(t)?"l:" SubStr(t.1,1,InStr(t.1,".")-1) " t:" SubStr(t.2,1,InStr(t.2,".")-1) " w:" SubStr(t.3,1,InStr(t.3,".")-1) " h:" SubStr(t.4,1,InStr(t.4,".")-1):)
}
And the UIA_Property function from UIA.ahk:

Code: Select all

UIA_Property(n){
	static id:={RuntimeId:30000,BoundingRectangle:30001,ProcessId:30002,ControlType:30003,LocalizedControlType:30004,Name:30005,AcceleratorKey:30006,AccessKey:30007,HasKeyboardFocus:30008,IsKeyboardFocusable:30009,IsEnabled:30010,AutomationId:30011,ClassName:30012,HelpText:30013,ClickablePoint:30014,Culture:30015,IsControlElement:30016,IsContentElement:30017,LabeledBy:30018,IsPassword:30019,NativeWindowHandle:30020,ItemType:30021,IsOffscreen:30022,Orientation:30023,FrameworkId:30024,IsRequiredForForm:30025,ItemStatus:30026,IsDockPatternAvailable:30027,IsExpandCollapsePatternAvailable:30028,IsGridItemPatternAvailable:30029,IsGridPatternAvailable:30030,IsInvokePatternAvailable:30031,IsMultipleViewPatternAvailable:30032,IsRangeValuePatternAvailable:30033,IsScrollPatternAvailable:30034,IsScrollItemPatternAvailable:30035,IsSelectionItemPatternAvailable:30036,IsSelectionPatternAvailable:30037,IsTablePatternAvailable:30038,IsTableItemPatternAvailable:30039,IsTextPatternAvailable:30040,IsTogglePatternAvailable:30041,IsTransformPatternAvailable:30042,IsValuePatternAvailable:30043,IsWindowPatternAvailable:30044,ValueValue:30045,ValueIsReadOnly:30046,RangeValueValue:30047,RangeValueIsReadOnly:30048,RangeValueMinimum:30049,RangeValueMaximum:30050,RangeValueLargeChange:30051,RangeValueSmallChange:30052,ScrollHorizontalScrollPercent:30053,ScrollHorizontalViewSize:30054,ScrollVerticalScrollPercent:30055,ScrollVerticalViewSize:30056,ScrollHorizontallyScrollable:30057,ScrollVerticallyScrollable:30058,SelectionSelection:30059,SelectionCanSelectMultiple:30060,SelectionIsSelectionRequired:30061,GridRowCount:30062,GridColumnCount:30063,GridItemRow:30064,GridItemColumn:30065,GridItemRowSpan:30066,GridItemColumnSpan:30067,GridItemContainingGrid:30068,DockDockPosition:30069,ExpandCollapseExpandCollapseState:30070,MultipleViewCurrentView:30071,MultipleViewSupportedViews:30072,WindowCanMaximize:30073,WindowCanMinimize:30074,WindowWindowVisualState:30075,WindowWindowInteractionState:30076,WindowIsModal:30077,WindowIsTopmost:30078,SelectionItemIsSelected:30079,SelectionItemSelectionContainer:30080,TableRowHeaders:30081,TableColumnHeaders:30082,TableRowOrColumnMajor:30083,TableItemRowHeaderItems:30084,TableItemColumnHeaderItems:30085,ToggleToggleState:30086,TransformCanMove:30087,TransformCanResize:30088,TransformCanRotate:30089,IsLegacyIAccessiblePatternAvailable:30090,LegacyIAccessibleChildId:30091,LegacyIAccessibleName:30092,LegacyIAccessibleValue:30093,LegacyIAccessibleDescription:30094,LegacyIAccessibleRole:30095,LegacyIAccessibleState:30096,LegacyIAccessibleHelp:30097,LegacyIAccessibleKeyboardShortcut:30098,LegacyIAccessibleSelection:30099,LegacyIAccessibleDefaultAction:30100,AriaRole:30101,AriaProperties:30102,IsDataValidForForm:30103,ControllerFor:30104,DescribedBy:30105,FlowsTo:30106,ProviderDescription:30107,IsItemContainerPatternAvailable:30108,IsVirtualizedItemPatternAvailable:30109,IsSynchronizedInputPatternAvailable:30110}
	, name:={30000:"UIA_RuntimeIdPropertyId",30001:"UIA_BoundingRectanglePropertyId",30002:"UIA_ProcessIdPropertyId",30003:"UIA_ControlTypePropertyId",30004:"UIA_LocalizedControlTypePropertyId",30005:"UIA_NamePropertyId",30006:"UIA_AcceleratorKeyPropertyId",30007:"UIA_AccessKeyPropertyId",30008:"UIA_HasKeyboardFocusPropertyId",30009:"UIA_IsKeyboardFocusablePropertyId",30010:"UIA_IsEnabledPropertyId",30011:"UIA_AutomationIdPropertyId",30012:"UIA_ClassNamePropertyId",30013:"UIA_HelpTextPropertyId",30014:"UIA_ClickablePointPropertyId",30015:"UIA_CulturePropertyId",30016:"UIA_IsControlElementPropertyId",30017:"UIA_IsContentElementPropertyId",30018:"UIA_LabeledByPropertyId",30019:"UIA_IsPasswordPropertyId",30020:"UIA_NativeWindowHandlePropertyId",30021:"UIA_ItemTypePropertyId",30022:"UIA_IsOffscreenPropertyId",30023:"UIA_OrientationPropertyId",30024:"UIA_FrameworkIdPropertyId",30025:"UIA_IsRequiredForFormPropertyId",30026:"UIA_ItemStatusPropertyId",30027:"UIA_IsDockPatternAvailablePropertyId",30028:"UIA_IsExpandCollapsePatternAvailablePropertyId",30029:"UIA_IsGridItemPatternAvailablePropertyId",30030:"UIA_IsGridPatternAvailablePropertyId",30031:"UIA_IsInvokePatternAvailablePropertyId",30032:"UIA_IsMultipleViewPatternAvailablePropertyId",30033:"UIA_IsRangeValuePatternAvailablePropertyId",30034:"UIA_IsScrollPatternAvailablePropertyId",30035:"UIA_IsScrollItemPatternAvailablePropertyId",30036:"UIA_IsSelectionItemPatternAvailablePropertyId",30037:"UIA_IsSelectionPatternAvailablePropertyId",30038:"UIA_IsTablePatternAvailablePropertyId",30039:"UIA_IsTableItemPatternAvailablePropertyId",30040:"UIA_IsTextPatternAvailablePropertyId",30041:"UIA_IsTogglePatternAvailablePropertyId",30042:"UIA_IsTransformPatternAvailablePropertyId",30043:"UIA_IsValuePatternAvailablePropertyId",30044:"UIA_IsWindowPatternAvailablePropertyId",30045:"UIA_ValueValuePropertyId",30046:"UIA_ValueIsReadOnlyPropertyId",30047:"UIA_RangeValueValuePropertyId",30048:"UIA_RangeValueIsReadOnlyPropertyId",30049:"UIA_RangeValueMinimumPropertyId",30050:"UIA_RangeValueMaximumPropertyId",30051:"UIA_RangeValueLargeChangePropertyId",30052:"UIA_RangeValueSmallChangePropertyId",30053:"UIA_ScrollHorizontalScrollPercentPropertyId",30054:"UIA_ScrollHorizontalViewSizePropertyId",30055:"UIA_ScrollVerticalScrollPercentPropertyId",30056:"UIA_ScrollVerticalViewSizePropertyId",30057:"UIA_ScrollHorizontallyScrollablePropertyId",30058:"UIA_ScrollVerticallyScrollablePropertyId",30059:"UIA_SelectionSelectionPropertyId",30060:"UIA_SelectionCanSelectMultiplePropertyId",30061:"UIA_SelectionIsSelectionRequiredPropertyId",30062:"UIA_GridRowCountPropertyId",30063:"UIA_GridColumnCountPropertyId",30064:"UIA_GridItemRowPropertyId",30065:"UIA_GridItemColumnPropertyId",30066:"UIA_GridItemRowSpanPropertyId",30067:"UIA_GridItemColumnSpanPropertyId",30068:"UIA_GridItemContainingGridPropertyId",30069:"UIA_DockDockPositionPropertyId",30070:"UIA_ExpandCollapseExpandCollapseStatePropertyId",30071:"UIA_MultipleViewCurrentViewPropertyId",30072:"UIA_MultipleViewSupportedViewsPropertyId",30073:"UIA_WindowCanMaximizePropertyId",30074:"UIA_WindowCanMinimizePropertyId",30075:"UIA_WindowWindowVisualStatePropertyId",30076:"UIA_WindowWindowInteractionStatePropertyId",30077:"UIA_WindowIsModalPropertyId",30078:"UIA_WindowIsTopmostPropertyId",30079:"UIA_SelectionItemIsSelectedPropertyId",30080:"UIA_SelectionItemSelectionContainerPropertyId",30081:"UIA_TableRowHeadersPropertyId",30082:"UIA_TableColumnHeadersPropertyId",30083:"UIA_TableRowOrColumnMajorPropertyId",30084:"UIA_TableItemRowHeaderItemsPropertyId",30085:"UIA_TableItemColumnHeaderItemsPropertyId",30086:"UIA_ToggleToggleStatePropertyId",30087:"UIA_TransformCanMovePropertyId",30088:"UIA_TransformCanResizePropertyId",30089:"UIA_TransformCanRotatePropertyId",30090:"UIA_IsLegacyIAccessiblePatternAvailablePropertyId",30091:"UIA_LegacyIAccessibleChildIdPropertyId",30092:"UIA_LegacyIAccessibleNamePropertyId",30093:"UIA_LegacyIAccessibleValuePropertyId",30094:"UIA_LegacyIAccessibleDescriptionPropertyId",30095:"UIA_LegacyIAccessibleRolePropertyId",30096:"UIA_LegacyIAccessibleStatePropertyId",30097:"UIA_LegacyIAccessibleHelpPropertyId",30098:"UIA_LegacyIAccessibleKeyboardShortcutPropertyId",30099:"UIA_LegacyIAccessibleSelectionPropertyId",30100:"UIA_LegacyIAccessibleDefaultActionPropertyId",30101:"UIA_AriaRolePropertyId",30102:"UIA_AriaPropertiesPropertyId",30103:"UIA_IsDataValidForFormPropertyId",30104:"UIA_ControllerForPropertyId",30105:"UIA_DescribedByPropertyId",30106:"UIA_FlowsToPropertyId",30107:"UIA_ProviderDescriptionPropertyId",30108:"UIA_IsItemContainerPatternAvailablePropertyId",30109:"UIA_IsVirtualizedItemPatternAvailablePropertyId",30110:"UIA_IsSynchronizedInputPatternAvailablePropertyId"}
	if n is integer
		return name[n]
	else return id[n]
}
Using Include.ahk on a Calculator window resulted in much more information than I needed, but after stepping through the tree, I find the following BoundingRectangle for the "8" button:

Code: Select all

BoundingRectangle:                  l:545 t:709 w:69 h:51
From that, you should then be able to calculate a clickable point for MouseClick().
TheBeginner
Posts: 40
Joined: 19 Oct 2018, 12:05

Re: Clicking on an object using UIA? (UI automation)

11 Nov 2019, 01:57

wow.. sorry I didn't know anybody replied to the post, it seems that to get notified by email you need to manually turn it on in the settings

also rather not movie the mouse to the element, I was hoping more to be using something native like Invoke(); But I don't know how
TheBeginner
Posts: 40
Joined: 19 Oct 2018, 12:05

Re: Clicking on an object using UIA? (UI automation)

11 Nov 2019, 14:20

one of the links points to AutoIt, I don't know much about it other than it's another automation language which if I remember correctly has some common roots with autohotkey.

I believe I found the two functions I need to call to make it work with UIA, but I don't know how to invoke them/structure them correctly that it would work
I'm trying to start with something simple like clicking something on the calculator with this, before I moved to the other program (XD), tried it with aac in more complex application (Adobe XD) but acc doesn't have an ID as far as I know, and names don't always work. hoping that if I get an example of a simple version of it working I can take it from there, since most of it is very complicated

but all I really want to do is just find the object --> and invoke its action

these are the functions, do you know how to use them?

from - https://github.com/neptercn/UIAutomation/blob/master/UIA2.ahk

Code: Select all


; Retrieves the Microsoft UI Automation identifier of the element. 
	; The identifier is unique among sibling elements in a container, and is the same in all instances of the application.
	CurrentAutomationId(){
	_Error(DllCall(vt(this._p,29),"ptr",this._p,"ptr*",retVal),"get_CurrentAutomationId")
	return retVal
	}
	
....

	; Invokes the action of a control, such as a button click. 
	; Calls to this method should return immediately without blocking. However, this behavior depends on the implementation.
	Invoke_Invoke(){
	return _Error(DllCall(vt(this._p,3),"ptr",this._p),"Invoke")
	}
malcev
Posts: 262
Joined: 12 Aug 2014, 12:37

Re: Clicking on an object using UIA? (UI automation)

11 Nov 2019, 14:37

As I said I do not recommend use this wrapper.
If You go to my first link You will see example how to activate IE tab.
From Autoit link You can understand how this technology works and which methods better to use.
Adobe XD has API:
https://adobexdplatform.com/plugin-docs/reference/core/apis.html
TheBeginner
Posts: 40
Joined: 19 Oct 2018, 12:05

Re: Clicking on an object using UIA? (UI automation)

11 Nov 2019, 14:47

Yes, I know it has an API. it can't perform what I need it to
I'll check the link again
malcev
Posts: 262
Joined: 12 Aug 2014, 12:37

Re: Clicking on an object using UIA? (UI automation)  Topic is solved

12 Nov 2019, 07:22

I Installed this software.
As I understand MSAA default action does not work with toggle buttons.
Therefore to push such buttons You have to use UIA.
For example to toggle design button we can do like this:

Code: Select all

uia := UIA_Interface()
xd := uia.ElementFromHandle(hwnd)
condition := uia.CreateAndConditionFromArray([uia.CreatePropertyCondition(UIA_ControlTypePropertyId := 30003, button := 50000, VT_I4 := 3), uia.CreatePropertyCondition(UIA_NamePropertyId := 30005, "Design", VT_BSTR := 8)])
element := xd.FindFirst(condition, TreeScope_Descendants := 0x4)
toggle := element.GetCurrentPatternAs("Toggle")
toggle.toggle()
jethrow did not finish his library, therefore You have to add this class to his wrapper
https://github.com/jethrow/UIA_Interface/blob/master/UIA_Interface.ahk

Code: Select all

class UIA_TogglePattern extends UIA_Base
{
   ; https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/nn-uiautomationclient-iuiautomationtogglepattern
   static __IID := "{94cf8058-9b8d-4ab9-8bfd-4cd0a33c8c70}", __PatternID := 10015, __Properties := "CurrentToggleState,4,int`r`nCachedToggleState,5,int"
   Toggle()
   {
      return UIA_Hr(DllCall(this.__Vt(3), "ptr",this.__Value))
   }
}
And use inspect.exe in UI Automation mode for getting info about elements of window and for detecting actions that You need.
https://github.com/guolaok/Python-UIAutomation-for-Windows/tree/master/inspect
TheBeginner
Posts: 40
Joined: 19 Oct 2018, 12:05

Re: Clicking on an object using UIA? (UI automation)

12 Nov 2019, 16:20

:dance: :dance: your code level is amazing, thanks for helping out, it works !
guest3456
Posts: 2684
Joined: 09 Oct 2013, 10:31

Re: Clicking on an object using UIA? (UI automation)

12 Nov 2019, 23:38

malcev wrote:
12 Nov 2019, 07:22
jethrow did not finish his library, therefore You have to add this class to his wrapper
https://github.com/jethrow/UIA_Interface/blob/master/UIA_Interface.ahk

Code: Select all

class UIA_TogglePattern extends UIA_Base
{
   ; https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/nn-uiautomationclient-iuiautomationtogglepattern
   static __IID := "{94cf8058-9b8d-4ab9-8bfd-4cd0a33c8c70}", __PatternID := 10015, __Properties := "CurrentToggleState,4,int`r`nCachedToggleState,5,int"
   Toggle()
   {
      return UIA_Hr(DllCall(this.__Vt(3), "ptr",this.__Value))
   }
}
submit a pull request?

malcev
Posts: 262
Joined: 12 Aug 2014, 12:37

Re: Clicking on an object using UIA? (UI automation)

13 Nov 2019, 06:02

I do not know how to do it.
May be You can do it?
guest3456
Posts: 2684
Joined: 09 Oct 2013, 10:31

Re: Clicking on an object using UIA? (UI automation)

13 Nov 2019, 09:08

malcev wrote:
13 Nov 2019, 06:02
I do not know how to do it.
May be You can do it?
done:
https://github.com/jethrow/UIA_Interface/pull/2


Return to “Ask For Help”

Who is online

Users browsing this forum: just me, mehrez1989, MrGrayline, QuestVBA, ulraf and 206 guests