UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
User avatar
rommmcek
Posts: 1470
Joined: 15 Aug 2014, 15:18

Re: UIAutomation with a focus on Chrome

Post by rommmcek » 11 Jun 2022, 07:55

Xeo786 wrote:
11 Jun 2022, 05:41
Thanks for your Great work,
newbie007 wrote:
11 Jun 2022, 06:12
Thanks alot! very powerful library
Thank you alot from me too, it was much needed!

Xeo786 wrote:
11 Jun 2022, 05:41
UIAViewer.ahk do not highlights element in browser
For Acc malcev proposed sending WM_GETOBJECT.
[Edit]: And it works! After viewing elements with Acc Viewer using WM_GETOBJECT UIAViewer starts to frame elements in Chrome too.

UIAViewer highjacks the Esc Keyboard key. To fix this I use:

Code: Select all

#If IsCapturing
Esc:: gosub ButCapture
#If

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 11 Jun 2022, 10:16

@Xeo786, on further testing it appears you are right. For some reason ElementFromPoint doesn't always return the deepest element, so I added a workaround for that in UIAViewer. Please test it out and see if it worked. :)

@rommmcek, I fixed the Esc highjacking, thanks for the tip. As for WM_GETOBJECT, I was under the impression that it is called automatically by the UIAutomation framework. Could you try the latest UIAViewer and see if my ElementFromPoint workaround fixed this too? Otherwise I might consider adding the SendMessage as well (btw, does SendMessage, WM_GETOBJECT := 0x003D, 0, 1,, % "ahk_id " WinExist("A") also work?).

User avatar
rommmcek
Posts: 1470
Joined: 15 Aug 2014, 15:18

Re: UIAutomation with a focus on Chrome

Post by rommmcek » 11 Jun 2022, 12:07

Latest UIA Viewer does not frame elements in Chrom for me neither. Sending WM_GETOBJECT w/o specifying Control does not work. So we have to send:
SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Chrome_RenderWidgetHostHWND1, % "ahk_id " WinExist("A") and
SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Windows.UI.Core.CoreWindow1, % "ahk_id " WinExist("A") for Chrome and Edge respectively.
See explanation by malcev too.

[Edit]: Running Chrom w/ --force-renderer-accessibility switch works too.

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 11 Jun 2022, 13:10

rommmcek wrote:
11 Jun 2022, 12:07
Latest UIA Viewer does not frame elements in Chrom for me neither. Sending WM_GETOBJECT w/o specifying Control does not work. So we have to send:
SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Chrome_RenderWidgetHostHWND1, % "ahk_id " WinExist("A") and
SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Windows.UI.Core.CoreWindow1, % "ahk_id " WinExist("A") for Chrome and Edge respectively.
See explanation by malcev too.

[Edit]: Running Chrom w/ --force-renderer-accessibility switch works too.
The SendMessage approach didn't work for me at all. When I ran Chrome and Vivaldi with --disable-renderer-accessibility then indeed I couldn't inspect any of the content elements, but sending WM_GETOBJECT didn't re-enable it. Also the documentation for WM_GETOBJECT implies that UIA interface sends the message itself anyway, so I have no idea why sending that message works for you. I think the safer bet is to run browsers with --force-renderer-accessibility (or use browser-specific ways of enabling accessibility)...

aifritz
Posts: 301
Joined: 29 Jul 2018, 11:30
Location: Germany

Re: UIAutomation with a focus on Chrome

Post by aifritz » 12 Jun 2022, 12:11

@Descolada, thank you for this wonderful work! :bravo: :bravo: :bravo:

I've tested the SelectTab() function and found some typos -> The var Name must be tabName

Furthermore didn't get the function working with matchMode=3. It seems so as this.BrowserElement.FindFirst(AndCondition).Click() is not working, but I don't know why...

Code: Select all

	SelectTab(tabName, matchMode=3) { ; Selects a tab with the text of tabName. matchMode follows SetTitleMatchMode scheme: 1=tab name must must start with tabName; 2=can contain anywhere; 3=exact match; RegEx
		if (matchMode==3) {
			TabItemControlCondition := this.UIA.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_TabItemControlTypeId, VT_I4 := 3)
      TabNameCondition := this.UIA.CreatePropertyCondition(UIA_NamePropertyId, tabName, VT_BSTR := 8)
			AndCondition := this.UIA.CreateAndCondition(TabItemControlCondition,TabNameCondition)
			return this.BrowserElement.FindFirst(AndCondition).Click()
		}
		for k, v in this.GetAllTabs() {
			curName := v.CurrentName
			if (((matchMode == 1) && (SubStr(curName, 1, StrLen(tabName)) == tabName)) || ((matchMode == 2) && InStr(curName, tabName)) || ((matchMode == "RegEx") && RegExMatch(curName, tabName)))
				return v.Click()
		}
	}

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 12 Jun 2022, 15:24

@aifritz, thank you for your feedback, fixed that typo. Also fixed a bug in GetCurrentMainPaneElement which was using FindFirst to look for a ToolBar element, but in Chrome UIA tree the document element is located before the actual toolbar, so if the website content contained a toolbar then that was found instead.

If SelectTab worked after fixing the "tabName" typo, then Click() must be working properly. Are you sure you are providing the exact name (case-sensitive, not missing any spaces etc) for the tab when using matchMode=3?
Try pasting the result of this code to somewhere and try using a tab name from there.

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

browserName := "ahk_exe chrome.exe"
WinActivate, %browserName%
WinWaitActive, %browserName%
cUIA := new UIA_Browser(browserName)
Clipboard := cUIA.PrintArray(cUIA.GetAllTabNames())

Loop
Posts: 160
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 12 Jun 2022, 16:24

Hi,
you have an example for GetAllLinks
Thx

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

browserName := "ahk_exe chrome.exe"
WinActivate, %browserName%
WinWaitActive, %browserName%
cUIA := new UIA_Browser(browserName)

links := cUIA.GetAllLinks()
For i, link in links
MsgBox, % link



malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIAutomation with a focus on Chrome

Post by malcev » 12 Jun 2022, 16:28

The SendMessage approach didn't work for me at all. When I ran Chrome and Vivaldi with --disable-renderer-accessibility then indeed I couldn't inspect any of the content elements, but sending WM_GETOBJECT didn't re-enable it. Also the documentation for WM_GETOBJECT implies that UIA interface sends the message itself anyway, so I have no idea why sending that message works for you. I think the safer bet is to run browsers with --force-renderer-accessibility (or use browser-specific ways of enabling accessibility)...
Try to run chrome or vivaldi not from shortcut (where You add parameters), but from real exe and You see that without sending WM_GETOBJECT You will not get deepest element.
Also with vivaldi or with skype You need call acc.accName(0) after sending WM_GETOBJECT to rebuild tree.

Code: Select all

SendMessage, WM_GETOBJECT := 0x003D, 0, 1, Chrome_RenderWidgetHostHWND1, % "ahk_id " WinExist("A")		
acc := Acc_ObjectFromWindow(WinExist("A"))
acc.accName(0)   ; needed for rebuilding acc tree

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 12 Jun 2022, 17:01

@Loop, GetAllLinks returns all link elements, so you need to call some method to access them:

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

browserName := "ahk_exe chrome.exe"
WinActivate, %browserName%
WinWaitActive, %browserName%
cUIA := new UIA_Browser(browserName)

links := cUIA.GetAllLinks()
For i, link in links {
	ToolTip, % link.GetCurrentValue() ; Alternatively use link.Dump()
	Sleep, 500
}
links[1].Click() ; Click on the first link
@malcev, opening both Chrome and Vivaldi straight from the exe worked just fine, could access the deepest elements. Accessing Skype worked as expected too (I didn't use SendMessage nor inspect with AccViewer before using UIAViewer).

malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIAutomation with a focus on Chrome

Post by malcev » 12 Jun 2022, 17:22

What OS do You have?
I tested on 3 different clean OS win10, and if I run any chrominium software without --force-renderer-accessibility, then I need to send WM_GETOBJECT.

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 12 Jun 2022, 17:53

malcev wrote:
12 Jun 2022, 17:22
What OS do You have?
I tested on 3 different clean OS win10, and if I run any chrominium software without --force-renderer-accessibility, then I need to send WM_GETOBJECT.
Running Windows10 Pro version 21H2 build 19044.1706. When I have some time I will try to reproduce this on a VM with a fresh install.

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 14 Jun 2022, 13:14

@malcev, I tested WM_GETOBJECT on 3 builds:
1) Windows 10 Pro (build 10240) fresh install: all Chromium-based apps needed the WM_GETOBJECT call (I tested Edge, Chrome, Vivaldi, Skype) if not started with --force-renderer-accessibility.
2) Windows 10 Pro version 21H2 (build 19044.1706), my current build: intermittently needed WM_GETOBJECT (but usually not, only Edge needed it sometimes), perhaps this explains why Edge is/was finnicky with UIA (didn't always respond to calls properly)?
3) Windows 11 Pro fresh install: none on the Chromium-based apps needed the WM_GETOBJECT call
Notably when the WM_GETOBJECT call was needed, sending it to only Chrome_RenderWidgetHostHWND1 worked and Windows.UI.Core.CoreWindow1 was not needed in Edge. And in Vivaldi nor Skype I couldn't establish a clear link to accName(0) being needed to enable Acc path finding. Skype worked fine after WM_GETOBJECT, and in Vivaldi it only started to work after starting Vivaldi with --force-renderer-accessibility. So for Acc paths, probaly the safest bet still is the --force-renderer-accessibility.

So, I updated UIAViewer.ahk to send WM_GETOBJECT if a Chromium rendered control is detected, hopefully that is enough to fix the problem for most people. And since Chromium apps are quite common, I am also considering adding it to UIA.ElementFromHandle and ElementFromPoint as an optional flag...

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 14 Jun 2022, 17:38

@Descolada, this is awesome, your work is very useful, thank you.

im thinking of writing some scripts with it. for instance, when i drag a link from browser and drop it to wordpad, only the link itself will be brought up, i mean, the title is lost. so, with uia and your fantastic work, i suppose i can intercept the operation and bring also the title as well. (even better, bring the thumbnail too.)

any chance, you could provide a drag&drop example for me to study?

besides, the example of retrieving of all of the links from a webpage is wonderful. it would be very handy if you could also provide a strip down version of it? something like the geturl() that you wrote before.

thank you.

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 15 Jun 2022, 08:58

@SundayProgrammer, it sounds like an interesting project but I think I can't help you much there. The UIA part seems easy enough: you can get the text of the element under the mouse using ElementFromPoint() and then CurrentName and/or GetCurrentValue(). But detecting the dragging part is more complicated and I don't have an example to provide. UIA's Microsoft documentation does have a section on DragPatterns, but I tested it and couldn't get it working: I think its meant only for draggable elements, not drag-dropping text/images/whatnot. Perhaps you could set up a LButton Down hotkey to detect clicks and record the coordinate, then if the user holds the button down long enough and then releases it (and you implement a way of detecting that a drag-drop was done?), you can call ElementFromPoint on the coordinate where the drag was initiated.
Other option would be to detect EVENT_OBJECT_CREATE and DESTROY for SysDragImage with a hook.

For the webpage links part: I don't think a stripped-down version would be very useful here, because the GetAllLinks() method returns elements, so if you wanted to actually do anything with the links (eg clicking, getting text, getting URL etc), then it would require additional methods and would get messy and complicated quite fast. A better way would be to extract the pertinent parts from UIA_Interface.ahk and include them in your script, something like user teadrinker did in one example (although not using UIA_Interface.ahk).

aifritz
Posts: 301
Joined: 29 Jul 2018, 11:30
Location: Germany

Re: UIAutomation with a focus on Chrome

Post by aifritz » 15 Jun 2022, 10:37

Descolada wrote:
12 Jun 2022, 15:24
@aifritz, thank you for your feedback, fixed that typo. Also fixed a bug in GetCurrentMainPaneElement which was using FindFirst to look for a ToolBar element, but in Chrome UIA tree the document element is located before the actual toolbar, so if the website content contained a toolbar then that was found instead.

If SelectTab worked after fixing the "tabName" typo, then Click() must be working properly. Are you sure you are providing the exact name (case-sensitive, not missing any spaces etc) for the tab when using matchMode=3?
Try pasting the result of this code to somewhere and try using a tab name from there.

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

browserName := "ahk_exe chrome.exe"
WinActivate, %browserName%
WinWaitActive, %browserName%
cUIA := new UIA_Browser(browserName)
Clipboard := cUIA.PrintArray(cUIA.GetAllTabNames())
Hi Descolada,
matchMode=3 is working now for me. Thank you

Another thing I want to ask, concerns filling a webform:

In the form I can see with UIAViewer the name of the description. Beside is an input field, where value/name is blank. I only see ControlType: 50004 LocalizedControlType: Edit
My idea was to find the description and address the edit field next to it, to populate it. Can you give me a hint how to accomplish this. Is the treewalker intended for this purpose? I've no clue how to use it...many thanks

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 15 Jun 2022, 13:35

@aifritz, you can try inspecting it with Accessibility Insights as well, because it shows more properties than UIAViewer and you might find something that will separate it (though this is unlikely, because usually Edit elements only have different names or values). Other than that, using a TreeWalker would be the best bet. You need to locate an element before the Edit and then use GetNextSiblingElement to find your element of interest. Something like this:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>
#include <UIA_Browser>
return

F1::
	browserExe := "chrome.exe"
	Run, %browserExe% -incognito --force-renderer-accessibility
	WinWaitActive, ahk_exe %browserExe%
	cUIA := new UIA_Browser("ahk_exe " browserExe)
	
	cUIA.WaitPageLoad("New Tab",5000)
	cUIA.SetURL("google.com", True)
	cUIA.WaitPageLoad()
	
	; First get rid of the tracking notice
	langBut := cUIA.WaitElementExist("ControlType=MenuItem") ; First lets make sure the selected language is correct. First lets get the language menu button using its ClassName
	expandCollapsePattern := langBut.GetCurrentPatternAs("ExpandCollapse") ; Since this isn't a normal button, we need to get the ExpandCollapse pattern for it and then call the correct method.
	if (expandCollapsePattern.CurrentExpandCollapseState == UIA_ExpandCollapseState_Collapsed) ; Check that it is collapsed
		expandCollapsePattern.Expand() ; Expand the menu
	cUIA.WaitElementExist("Name=English AND ControlType=MenuItem").Click() ; Select the English language
	cUIA.WaitElementExistByNameAndType("I agree", UIA_ButtonControlTypeId).Click() ; If the "I agree" button exists, then click it to get rid of the consent form
	cUIA.WaitPageLoad()
	
	googleImage := cUIA.WaitElementExistByNameAndType("Google", "Image") ; Find the Google image
	CBCondition := cUIA.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_ComboBoxControlTypeId) ; Create a TreeWalker to only look for ComboBox elements in the tree
	CBTW := cUIA.CreateTreeWalker(CBCondition) ; Create the TreeWalker with the ComboBox condition
	searchCB := CBTW.GetNextSiblingElement(googleImage) ; Find the first ComboBox after the Google image
	searchCB.SetValue("search term")
	return
EDIT: code modified to be in accordance with forum rules
Last edited by Descolada on 16 Jun 2022, 09:04, edited 1 time in total.

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 16 Jun 2022, 02:34

@Descolada, thank you for your time and effort on this. the information is very helpful, i would try "EVENT_OBJECT_CREATE and DESTROY for SysDragImage" first, it seems like a good alternative. thank you.

regarding the stripped-down version of "getalllinks", it would be very handy when all i needed is just a list of the links in return.

thank you.

AHK_user
Posts: 515
Joined: 04 Dec 2015, 14:52
Location: Belgium

Re: UIAutomation with a focus on Chrome

Post by AHK_user » 16 Jun 2022, 05:43

Very interesting, cool to see that we can explore the office ribbons now, (But of course com stays the way to go.)

I am getting an error when trying Example4_ChromeTest.ahk that UIA_PropertyId function is missing, is this replaced by the values in UIA_Constants?

@AHK_user
Edited posting regarding (previously available) code examples. For details, feel free to get in touch with staff/moderators.
Moderators: viewtopic.php?f=21&t=105077
:eh:

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Jun 2022, 07:07

@AHK_user, I'm so sorry, it seems during updating files I mixed up UIA_Browser.ahk files: I am preparing a big update and uploaded the newer version. :D It should work now.

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 16 Jun 2022, 10:13

SundayProgrammer wrote:
16 Jun 2022, 02:34
regarding the stripped-down version of "getalllinks", it would be very handy when all i needed is just a list of the links in return.
@Descolada, here is my attempt, please kindly see if there is any way to improve. thank you.

Code: Select all

F1::GetAllYtvLinks("A")
GetAllYtvLinks(wTitle*) {	;based on the script by Descolada (https://www.autohotkey.com/boards/viewtopic.php?p=459451#p459451)
	ErrorLevel := 0
	if !(wId := WinExist(wTitle*)) {
		ErrorLevel := 1
		return
	}
	IUIAutomation := ComObjCreate(CLSID_CUIAutomation := "{ff48dba4-60ef-4201-aa87-54103eef594e}", IID_IUIAutomation := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	DllCall(NumGet(NumGet(IUIAutomation+0)+21*A_PtrSize), "ptr", IUIAutomation, "ptr*", TrueCondition)   ; IUIAutomation::CreateTrueCondition
	DllCall(NumGet(NumGet(IUIAutomation+0)+6*A_PtrSize), "ptr", IUIAutomation, "ptr", wId, "ptr*", elementMain)   ; IUIAutomation::ElementFromHandle
	DllCall(NumGet(NumGet(elementMain+0)+6*A_PtrSize), "ptr", elementMain, "int", TreeScope_Descendants := 0x4, "ptr", TrueCondition, "ptr*", elementArray)   ; IUIAutomationElement::FindAll
	DllCall(NumGet(NumGet(elementArray+0)+3*A_PtrSize), "ptr", elementArray, "ptr*", aLen)   ; IUIAutomationElementArray::get_Length
	Loop, %aLen%
		DllCall(NumGet(NumGet(elementArray+0)+4*A_PtrSize), "ptr", elementArray, "int", A_Index-1, "ptr*", currentElement)   ; IUIAutomationElementArray::GetElement
		,DllCall(NumGet(NumGet(currentElement+0)+10*A_PtrSize),"ptr",currentElement,"uint",30045,"ptr",(VarSetCapacity(currentValue,8+2*A_PtrSize)+NumPut(0,currentValue,0,"short")+NumPut(0,currentValue,8,"ptr"))*0+&currentValue)   ;IUIAutomationElement::GetCurrentPropertyValue
		,log.=InStr(v:=StrGet(NumGet(currentValue,8,"ptr"),"utf-16"),".youtube.com/watch?v=") ? v "`n" : ""
	ObjRelease(currentElement)
	ObjRelease(elementArray)
	ObjRelease(elementMain)
	ObjRelease(IUIAutomation)
	If RegExMatch(log, "i)\.youtube\.com/watch\?v=.+?&index=")
		Clipboard := RegExReplace(RegExReplace(log, "im`a)^(?!.+?\.youtube\.com/watch\?v=.+?&index=).+?(\R|$)"), "im`a)&(list|pageId)=.+?(?=\R|$)")
	Else Clipboard := log
	If StrLen(Clipboard)
		MsgBox,, YouTube Video Links, Retrieved, 1
	Else MsgBox, Not Any YouTube Video Link Is Found
	return Clipboard
}

Post Reply

Return to “Scripts and Functions (v1)”