UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Jun 2022, 11:57

@SundayProgrammer, seems good :) I would've used CreatePropertyCondition hyperlink instead of TrueCondition, but you might indeed have situations where the element isn't a hyperlink but contains an URL as the value. Also your RegExReplace doesn't replace "&list=" and "&pageId=" when the RegExMatch doesn't match - I'm not sure if this is by design or not.

Code: Select all

#NoEnv
SetWorkingDir %A_ScriptDir%

GetAllYtvLinks(wTitle*) {
	ErrorLevel := 0
	if !(wId := WinExist(wTitle*)) {
		ErrorLevel := 1
		return
	}
	IUIAutomation := ComObjCreate(CLSID_CUIAutomation := "{ff48dba4-60ef-4201-aa87-54103eef594e}", IID_IUIAutomation := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	VarSetCapacity(var,8+2*A_PtrSize), NumPut(3,var,0,"short"), NumPut(UIA_HyperlinkControlTypeId := 50005,var,8,"ptr")
	if (A_PtrSize == 4) 
		DllCall(NumGet(NumGet(IUIAutomation+0)+23*A_PtrSize), "ptr",IUIAutomation, "int", UIA_ControlTypePropertyId := 30003, "int64", NumGet(var, 0, "int64"), "int64", NumGet(var, 8, "int64"), "ptr*",linkCondition) ; IUIAutomation::CreatePropertyCondition
	else 
		DllCall(NumGet(NumGet(IUIAutomation+0)+23*A_PtrSize), "ptr",IUIAutomation, "int", UIA_ControlTypePropertyId := 30003, "ptr",&var, "ptr*",linkCondition) ; IUIAutomation::CreatePropertyCondition
	DllCall(NumGet(NumGet(IUIAutomation+0)+6*A_PtrSize), "ptr", IUIAutomation, "ptr", wId, "ptr*", elementMain)   ; IUIAutomation::ElementFromHandle
	DllCall(NumGet(NumGet(elementMain+0)+6*A_PtrSize), "ptr", elementMain, "int", TreeScope_Descendants := 0x4, "ptr", linkCondition, "ptr*", elementArray)   ; IUIAutomationElement::FindAll
	DllCall(NumGet(NumGet(elementArray+0)+3*A_PtrSize), "ptr", elementArray, "ptr*", aLen)   ; IUIAutomationElementArray::get_Length
	Loop, %aLen%
		DllCall(NumGet(NumGet(elementArray+0)+4*A_PtrSize), "ptr", elementArray, "int", A_Index-1, "ptr*", currentElement)   ; IUIAutomationElementArray::GetElement
		,DllCall(NumGet(NumGet(currentElement+0)+10*A_PtrSize),"ptr",currentElement,"uint",30045,"ptr",(VarSetCapacity(currentValue,8+2*A_PtrSize)+NumPut(0,currentValue,0,"short")+NumPut(0,currentValue,8,"ptr"))*0+&currentValue)   ;IUIAutomationElement::GetCurrentPropertyValue
		,log.=InStr(v:=StrGet(NumGet(currentValue,8,"ptr"),"utf-16"),".youtube.com/watch?v=") ? v "`n" : ""
	ObjRelease(IUIAutomation)
	Clipboard := RegExReplace(RegExMatch(log, "i)\.youtube\.com/watch\?v=.+?&index=") ? RegExReplace(log, "im`a)^(?!.+?\.youtube\.com/watch\?v=.+?&index=).+?(\R|$)") : log, "im`a)&(list|pageId)=.+?(?=\R|$)")
	If StrLen(Clipboard)
		MsgBox,, YouTube Video Links, Retrieved, 1
	Else MsgBox, Not Any YouTube Video Link Is Found
	return Clipboard
}

F1::GetAllYtvLinks("A")
The same, but using UIA_Interface:

Code: Select all

#NoEnv
SetWorkingDir %A_ScriptDir%

#include <UIA_Interface>

UIA := UIA_Interface()
GetAllYtvLinks(wTitle*) {
	ErrorLevel := 0
	if !(wId := WinExist(wTitle*)) {
		ErrorLevel := 1
		return
	}
	for _, value in % UIA.ElementFromHandle(wId).FindAll(UIA.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_HyperlinkControlTypeId))
		log.=InStr(v:=value.GetCurrentValue(),".youtube.com/watch?v=") ? v "`n" : ""
	Clipboard := RegExReplace(RegExMatch(log, "i)\.youtube\.com/watch\?v=.+?&index=") ? RegExReplace(log, "im`a)^(?!.+?\.youtube\.com/watch\?v=.+?&index=).+?(\R|$)") : log, "im`a)&(list|pageId)=.+?(?=\R|$)")
	If StrLen(Clipboard)
		MsgBox,, YouTube Video Links, Retrieved, 1
	Else MsgBox, Not Any YouTube Video Link Is Found
	return Clipboard
}
F1::GetAllYtvLinks("A")

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 16 Jun 2022, 13:17

Descolada wrote:
16 Jun 2022, 11:57
I would've used CreatePropertyCondition hyperlink instead of TrueCondition, but you might indeed have situations where the element isn't a hyperlink but contains an URL as the value.
i tried to use CreatePropertyCondition hyperlink, but failed to get it work. that's why.
Descolada wrote:
16 Jun 2022, 11:57
Also your RegExReplace doesn't replace "&list=" and "&pageId=" when the RegExMatch doesn't match - I'm not sure if this is by design or not.
yes, you caught me. it's an omission.

thank you.

AHK_user
Posts: 515
Joined: 04 Dec 2015, 14:52
Location: Belgium

Re: UIAutomation with a focus on Chrome

Post by AHK_user » 16 Jun 2022, 14:26

Descolada wrote:
16 Jun 2022, 07:07
@AHK_user, I'm so sorry, it seems during updating files I mixed up UIA_Browser.ahk files: I am preparing a big update and uploaded the newer version. :D It should work now.
The examples seem to work now, amazing stuff :bravo: :bravo: :bravo:
Unfortunatley I encountered some issues with the websites itselfs. :think:
Let me explain: in Europe, you get annoying requests to allow the cookies if you access specific websites with a lot of tracking cookies.
Also, my setting of chrome are in my native language...
So the chosen websites are not optimal for displaying examples for users inside europe... I suggest to use websites with only one language...

A question:
- Is it possible to run some javascript on the browser and get the return value, using the name is not always a usefull method when working with flexible webpages?
(Trick: cUIA.SetURL("javascript:alert(""Hello World"");", True) seems to work, but it is not the most optimal way)

I have created an chrome browser example that I think that should work for more users:
For the examples, I would reccommend to change them with the following mindset:
- The example may activate immediately.
- The example may close when finisched.
- It would be nice to be able to run the examples immediately, but unfortunately, one library refers to another one. If you put UIA_Constants directly inside UIA_Interface, you could run the examples without moving the libraries and by using #Include,

- Removed code example -
Last edited by AHK_user on 17 Jun 2022, 13:19, edited 1 time in total.

AHK_user
Posts: 515
Joined: 04 Dec 2015, 14:52
Location: Belgium

Re: UIAutomation with a focus on Chrome

Post by AHK_user » 16 Jun 2022, 16:31

I found the solution for the library referral of the examples:

=> Change in UIA_Interface the include line to:
#include %A_LineFile%\..\UIA_Constants.ahk ; %A_LineFile%\.. refers to the directory where the file is located

=> Change the includes in the examples to:
#include ..\lib\UIA_Interface.ahk
#include ..\lib\UIA_Browser.ahk

This way, users can just download the script and test the examples.

william_ahk
Posts: 481
Joined: 03 Dec 2018, 20:02

Re: UIAutomation with a focus on Chrome

Post by william_ahk » 17 Jun 2022, 01:04

Now this is groundbreaking work! Simply getting text from Chrome is a great feat. Previously the simplest way apart from going CDP/WebDriver would be evaluating Javascript in the address bar, and that was in no way reliable and convenient. Now with this library we can finally unleash the power of AHK upon the web!

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 17 Jun 2022, 10:38

@AHK_user, I'm also from Europe and also have problems with the GDPR popups. So far I have either:
1) used the browser in normal mode (not incognito), accepted the cookies, then later automated with UIA
2) added a check for the pop-up with FindFirst or WaitElementExist and if necessary then accepted it
And the chosen websites were meant to simulate real situations to show how one might get around these problems. My Google doesn't start up in English, so in incognito mode the example script switches the language to English to make everything constant (so hopefully run mostly the same on all computers).

About the Javascript question... As far as I know, UIA can't interact with windows in a "deeper" way than users can, meaning injecting JS can probably only be done through the address bar. To inject it without the address bar, perhaps look into Rufaydium or Selenium AHK libraries. But using UIA, perhaps a better way would be to get the return data through the window title? Try putting this in the UIA_Browser class:

Code: Select all

	JSReturnThroughTitle(js, timeOut=500) {
		WinGetTitle, origTitle, % "ahk_id " this.BrowserId
		this.SetURL("javascript:origTitle=document.title;document.title=(" js ");void(0);setTimeout(function() {document.title=origTitle;void(0);}, " timeOut ")", True)
		startTime := A_TickCount
		Loop {
			WinGetTitle, newTitle, % "ahk_id " this.BrowserId
			Sleep, 40
		} Until ((origTitle != newTitle) || (A_TickCount - startTime > timeOut))
		return (origTitle == newTitle) ? "" : RegexReplace(newTitle, "(?: - Personal)? - [^-]+$")
	}
Test script for the snippet:

Code: Select all

	browserExe := "chrome.exe"
	Run, %browserExe% -incognito
	WinWaitActive, ahk_exe %browserExe%
	cUIA := new UIA_Browser("ahk_exe " browserExe)
	cUIA.WaitPageLoad()
	cUIA.SetURL("autohotkey.com ", True)
	cUIA.WaitPageLoad()
	MsgBox, % cUIA.JSReturnThroughTitle("document.getElementById('downloadLink').textContent", 1000) ; Should show "Download Forums"
	return
As for the examples... I will make your suggested changes to activate and close immediately. And UIA_Constants.ahk will be optional after the next update.
I suggest you remove your suggested example script because it involves automating the AHK forum, which might be subject to harsh punishment without warning. This is the same reason I removed my Example5, thankfully didn't get insta-banned... :)

About the library referrals. On GitHub I accepted the pull request for the UIA_Constants include change. But the other suggested changes I don't know about... From AHK documentations:

Code: Select all

A script may call a function in an external file without having to use #Include. For this to work, a file of the same name as the function must exist in one of the following library directories:

%A_ScriptDir%\Lib\  ; Local library - requires [v1.0.90+].
%A_MyDocuments%\AutoHotkey\Lib\  ; User library.
directory-of-the-currently-running-AutoHotkey.exe\Lib\  ; Standard library.
So if the includes are in the Lib folder relative to the script itself, the examples should work. Perhaps you can determine why it doesn't work in your computer? Are you running installed AHK or stand-alone? Are you passing any other arguments into AHK when executing the script?

@william_ahk, thank you! And the JS option is still available, and can be made more reliable with UIA. :)

AHK_user
Posts: 515
Joined: 04 Dec 2015, 14:52
Location: Belgium

Re: UIAutomation with a focus on Chrome

Post by AHK_user » 17 Jun 2022, 17:32

@Descolada

I removed the example, it was not my intention to break the rules. Thanks for the warning, I will keep it in mind in the future.

Cool and out of the box idea to use the wintitle to return a value :bravo:
The only limitation would be the string size and some characters are not allowed, I am not sure how large the wintitle can be to be returned. But kind of clean way to get small returns.
I would more think of using a temporary file, or the clipboard is maybe also an interesting way.


Regarding the includes in the examples, my idea is that examples should be able to run with minimal effort, so just download and try to run it.
The current examples also require to put the files already in a library folder, or move the lib folder to the examples folder.

With a line like #include ..\lib\UIA_Interface.ahk in your examples you can just download the script and run it.
Using #include %A_LineFile%\.. also is related to the problem if you use library files who are not in a default library folders.
But this is just my opinion to make the examples as user-friendly, the script works fine and is amazing. :thumbup:

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 19 Jun 2022, 01:51

SundayProgrammer wrote:
16 Jun 2022, 02:34
@Descolada, thank you for your time and effort on this. the information is very helpful, i would try "EVENT_OBJECT_CREATE and DESTROY for SysDragImage" first, it seems like a good alternative. thank you.
here is the very first draft of the finished product. it requires winclip.ahk.

Code: Select all

; materials referenced mainly: https://stackoverflow.com/questions/2947139/how-to-know-user-is-dragging-something-when-the-cursor-is-not-in-my-window, https://www.autohotkey.com/board/topic/85231-lib-hook-winevents-to-catch-windows-creationdestruction/, https://stackoverflow.com/questions/13222689/how-do-i-insert-formatted-text-using-autohotkey, https://www.autohotkey.com/board/topic/74670-class-winclip-direct-clipboard-manipulations/, https://stackoverflow.com/questions/7894772/writing-unicode-to-rtf-file, https://www.autohotkey.com/boards/viewtopic.php?p=459451&sid=dde6b407875aadff59bc35e0864f63e4#p459451, https://www.autohotkey.com/boards/viewtopic.php?p=468059&sid=702a6e5619dce3f22bda11a2ec84c4eb#p468059
#Persistent
DetectHiddenWindows, On
OnExit, QUIT
DllCall("CoInitialize", UInt,0)
hWinEventHook := DllCall("SetWinEventHook", UInt,0x8000, UInt,0x8001, UInt,0, UInt,RegisterCallback("WinEventHandler"), UInt,0, UInt,0, UInt,0, UInt)	; EVENT_OBJECT_CREATE:=0x8000, EVENT_OBJECT_DESTROY:=0x8001
Return
QUIT:
UnhookWinEvent(hWinEventHook)
ExitApp

WinEventHandler(hWinEventHook, event, hWnd, idObject, idChild, dwEventThread, dwmsEventTime) {
	static LinkTitle, url, beg := 0
	WinGetClass, wClass, ahk_id %hWnd%
	If wClass = SysDragImage
		If (event == 0x8000)	;drag started
			If (A_TickCount - beg > 9999) and (o := GetFocusedElement()) and not (o.Value = GetURL("A")) and SubStr(o.Value, 1, 8) = "https://"
			{	LinkTitle := o.Name, url := o.Value, beg := A_TickCount, ClipSave(True), Clipboard := "Drag & Drop Operation In Effect:`n`nTitle`t" LinkTitle "`n`nURL`t" url
				ToolTip, Drag & Drop Operation In Effect
				SetTimer, MonitorLButton, -1
			}Else{}
		Else	;dropped
		{	MouseGetPos,,, wHwnd
			WinGetTitle, wt, ahk_id %wHwnd%
			If InStr(wt, " - WordPad")
			{	While, GetKeyState("LButton", "P")
					Sleep, 100
				Sleep, 1000
				Send, ^z
				WinClip.Clear(), WinClip.SetRTF(s := "{\rtf>>>{\field{\*\fldinst{HYPERLINK " url " }}{\fldrslt{" Conv4RTF(LinkTitle) "}}}<<<}"), WinClip.Paste()
				MsgBox,, Manipulation, Done, 1
			}
		}
}
ClipSave(Yes) {
	static ClipSave
	If Yes
		ClipSave := ClipboardAll
	Else Clipboard := ClipSave
}
MonitorLButton() {
	static pX, pY, pT
	If GetKeyState("LButton", "P")
	{	MouseGetPos, x, y
		If not (x "," y == pX "," pY)
			pX := x, pY := y, pT := A_TickCount
		Else If A_TickCount - pT > 500
		{	ToolTip, %Clipboard%
			pT := A_TickCount
		}SetTimer, MonitorLButton, -100
	}Else SetTimer, ExitDragDrop, -2500
}
ExitDragDrop() {
	ToolTip
	ClipSave(False)
}
UnhookWinEvent(hWinEventHook) {
	DllCall("UnhookWinEvent", UInt,hWinEventHook)
	DllCall("CoUninitialize")
}
GetURL(wTitle*) {	;written by Descolada (https://www.autohotkey.com/boards/viewtopic.php?p=459451&sid=dde6b407875aadff59bc35e0864f63e4#p459451)
	ErrorLevel := 0
	if !(wId := WinExist(wTitle*)) {
		ErrorLevel := 1
		return
	}
	IUIAutomation := ComObjCreate(CLSID_CUIAutomation := "{ff48dba4-60ef-4201-aa87-54103eef594e}", IID_IUIAutomation := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	DllCall(NumGet(NumGet(IUIAutomation+0)+6*A_PtrSize), "ptr", IUIAutomation, "ptr", wId, "ptr*", elementMain)   ; IUIAutomation::ElementFromHandle
	NumPut(addressbarStrPtr := DllCall("oleaut32\SysAllocString", "wstr", "Address and search bar", "ptr"),(VarSetCapacity(addressbar,8+2*A_PtrSize)+NumPut(8,addressbar,0,"short"))*0+&addressbar,8,"ptr")
	DllCall("oleaut32\SysFreeString", "ptr", addressbarStrPtr)
	if (A_PtrSize = 4) {
		DllCall(NumGet(NumGet(IUIAutomation+0)+23*A_PtrSize), "ptr", IUIAutomation, "int", 30005, "int64", NumGet(addressbar, 0, "int64"), "int64", NumGet(addressbar, 8, "int64"), "ptr*", addressbarCondition)   ; IUIAutomation::CreatePropertyCondition
	} else {
		DllCall(NumGet(NumGet(IUIAutomation+0)+23*A_PtrSize), "ptr", IUIAutomation, "int", 30005, "ptr", &addressbar, "ptr*", addressbarCondition)   ; IUIAutomation::CreatePropertyCondition
	}
	DllCall(NumGet(NumGet(elementMain+0)+5*A_PtrSize), "ptr", elementMain, "int", TreeScope_Descendants := 0x4, "ptr", addressbarCondition, "ptr*", currentURLElement) ; IUIAutomationElement::FindFirst
	DllCall(NumGet(NumGet(currentURLElement+0)+10*A_PtrSize),"ptr",currentURLElement,"uint",30045,"ptr",(VarSetCapacity(currentURL,8+2*A_PtrSize)+NumPut(0,currentURL,0,"short")+NumPut(0,currentURL,8,"ptr"))*0+&currentURL) ;IUIAutomationElement::GetCurrentPropertyValue
	ObjRelease(currentURLElement)
	ObjRelease(elementMain)
	ObjRelease(IUIAutomation)
	return StrGet(NumGet(currentURL,8,"ptr"),"utf-16")
}
GetFocusedElement() {	;based on the script by Descolada (https://www.autohotkey.com/boards/viewtopic.php?p=459451&sid=dde6b407875aadff59bc35e0864f63e4#p459451)
	IUIAutomation := ComObjCreate(CLSID_CUIAutomation := "{ff48dba4-60ef-4201-aa87-54103eef594e}", IID_IUIAutomation := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	DllCall(NumGet(NumGet(IUIAutomation+0)+8*A_PtrSize), "ptr",IUIAutomation, "ptr*",element)	;IUIAutomation::GetFocusedElement
	DllCall(NumGet(NumGet(element+0)+23*A_PtrSize), "ptr", element, "ptr*", name)	;IUIAutomationElement::CurrentName
	DllCall(NumGet(NumGet(element+0)+10*A_PtrSize), "ptr", element, "uint",30045, "ptr", (VarSetCapacity(val,8+2*A_PtrSize)+NumPut(0,val,0,"short")+NumPut(0,val,8,"ptr"))*0+&val)	;IUIAutomationElement::GetCurrentPropertyValue
	ObjRelease(element)
	ObjRelease(IUIAutomation)
	Return {"Name":StrGet(name,"utf-16"), "Value":StrGet(NumGet(val,8,"ptr"),"utf-16")}
}
Conv4RTF(t) {
	Loop, Parse, t
		Result .= Asc(A_LoopField) > 127 ? "\u" Asc(A_LoopField) "?" : A_LoopField
	Return Result
}
#Include WinClipAPI.ahk
#Include WinClip.ahk
one thing to note though, depending on the speed of your machine, the retrieving of information thru uia may not come as quickly as the drag movement you can make. if you encounter any missing of "interception", try to slow down your drag movement a bit (especially at the starting part (and better only speed up until you see the tooltip is shown) but don't stay holding down lbutton without any movement at all (coz it by definition won't trigger drag&drop)) to see if it works again.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 19 Jun 2022, 04:25

@SundayProgrammer, awesome work, man! You might perhaps be the first one to capture drag-drop events globally like this :D

Regarding the information speed from UIA and needing to slow down drag movement... I tried tweaking your code a bit and I think it got better... Note that this uses the freshly-updated UIA_Interface.ahk.

Code: Select all

; materials referenced mainly: https://stackoverflow.com/questions/2947139/how-to-know-user-is-dragging-something-when-the-cursor-is-not-in-my-window, https://www.autohotkey.com/board/topic/85231-lib-hook-winevents-to-catch-windows-creationdestruction/, https://stackoverflow.com/questions/13222689/how-do-i-insert-formatted-text-using-autohotkey, https://www.autohotkey.com/board/topic/74670-class-winclip-direct-clipboard-manipulations/, https://stackoverflow.com/questions/7894772/writing-unicode-to-rtf-file, https://www.autohotkey.com/boards/viewtopic.php?p=459451&sid=dde6b407875aadff59bc35e0864f63e4#p459451, https://www.autohotkey.com/boards/viewtopic.php?p=468059&sid=702a6e5619dce3f22bda11a2ec84c4eb#p468059
#Persistent
DetectHiddenWindows, On

#Include <WinClipAPI>
#Include <WinClip>
#include <UIA_Interface>

global UIA := UIA_Interface(), currentlyDragging := False
OnExit, QUIT
DllCall("CoInitialize", UInt,0)
hWinEventHook := DllCall("SetWinEventHook", UInt,0x8000, UInt,0x8001, UInt,0, UInt,RegisterCallback("WinEventHandler"), UInt,0, UInt,0, UInt,0, UInt)	; EVENT_OBJECT_CREATE:=0x8000, EVENT_OBJECT_DESTROY:=0x8001
Return
QUIT:
UnhookWinEvent(hWinEventHook)
ExitApp

WinEventHandler(hWinEventHook, event, hWnd, idObject, idChild, dwEventThread, dwmsEventTime) {
	static LinkTitle, url, beg := 0
	WinGetClass, wClass, ahk_id %hWnd%
	If wClass = SysDragImage
		If ((event == 0x8000) && !currentlyDragging) {	; drag started
			If (IsObject(o := UIA.GetFocusedElement()) && IsObject(URLEditEl := GetURLEditElement("A")) && !UIA.CompareElements(o, URLEditEl)) { ; Also make sure that we are in a browser?
				currentlyDragging := True, LinkTitle := o.CurrentName, url := o.CurrentValue, beg := A_TickCount, ClipSave(True), Clipboard := "Drag & Drop Operation In Effect:`n`nTitle`t" LinkTitle "`n`nURL`t" url
				SetTimer, MonitorLButton, -1
			}
		} Else { ; dropped
			MouseGetPos,,, wHwnd
			WinGetTitle, wt, ahk_id %wHwnd%
			If InStr(wt, " - WordPad")
			{	While, GetKeyState("LButton", "P")
					Sleep, 100
				Sleep, 500
				Send, ^z
				WinClip.Clear(), WinClip.SetRTF(s := "{\rtf>>>{\field{\*\fldinst{HYPERLINK " url " }}{\fldrslt{" Conv4RTF(LinkTitle) "}}}<<<}"), WinClip.Paste()
			}
		}
}
ClipSave(Yes) {
	static ClipSave
	If Yes
		ClipSave := ClipboardAll
	Else Clipboard := ClipSave
}
MonitorLButton() {
	static pX, pY, pT
	If GetKeyState("LButton", "P")
	{	ToolTip, %Clipboard%
		SetTimer, MonitorLButton, -100
	} else ExitDragDrop()
}
ExitDragDrop() {
	ToolTip
	ClipSave(currentlyDragging := False)
}
UnhookWinEvent(hWinEventHook) {
	DllCall("UnhookWinEvent", UInt,hWinEventHook)
	DllCall("CoUninitialize")
}
GetURLEditElement(wTitle*) {
	ErrorLevel := 0
	if !(wId := WinExist(wTitle*)) {
		ErrorLevel := 1
		return
	}
	BrowserElement := UIA.ElementFromHandle(WinExist(wTitle*))
	static EditControlCondition, EditNameCondition, EditAndCondition, ToolbarControlCondition, TWT
	if !EditControlCondition
		EditControlCondition := UIA.CreatePropertyCondition(UIA.ControlTypePropertyId, UIA.EditControlTypeId), EditNameCondition := UIA.CreateOrCondition(UIA.CreatePropertyCondition(UIA.NamePropertyId, "Address and search bar"), UIA.CreatePropertyCondition(UIA.NamePropertyId, "Search or enter an address")), EditAndCondition := UIA.CreateAndCondition(EditControlCondition, EditNameCondition), ToolbarControlCondition := UIA.CreatePropertyCondition(UIA.ControlTypePropertyId, UIA.ToolBarControlTypeId), TWT := UIA.CreateTreeWalker(UIA.CreateTrueCondition())
	if !(URLEditElement := BrowserElement.FindFirst(EditAndCondition)) {
		ToolbarElements := BrowserElement.FindAll(ToolbarControlCondition), topCoord := 10000000
		for k, v in ToolbarElements {
			if ((bT := v.CurrentBoundingRectangle.t) && (bT < topCoord))
				topCoord := bT, NavigationBarElement := v
		}
	}
	return URLEditElement
}
Conv4RTF(t) {
	Loop, Parse, t
		Result .= Asc(A_LoopField) > 127 ? "\u" Asc(A_LoopField) "?" : A_LoopField
	Return Result
}

aifritz
Posts: 301
Joined: 29 Jul 2018, 11:30
Location: Germany

Re: UIAutomation with a focus on Chrome

Post by aifritz » 19 Jun 2022, 10:53

Descolada wrote:
15 Jun 2022, 13:35
@aifritz, you can try inspecting it with Accessibility Insights as well, because it shows more properties than UIAViewer and you might find something that will separate it (though this is unlikely, because usually Edit elements only have different names or values). Other than that, using a TreeWalker would be the best bet. You need to locate an element before the Edit and then use GetNextSiblingElement to find your element of interest. Something like this:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>
#include <UIA_Browser>
return

F1::
	browserExe := "chrome.exe"
	Run, %browserExe% -incognito --force-renderer-accessibility
	WinWaitActive, ahk_exe %browserExe%
	cUIA := new UIA_Browser("ahk_exe " browserExe)
	
	cUIA.WaitPageLoad("New Tab",5000)
	cUIA.SetURL("google.com", True)
	cUIA.WaitPageLoad()
	
	; First get rid of the tracking notice
	langBut := cUIA.WaitElementExist("ControlType=MenuItem") ; First lets make sure the selected language is correct. First lets get the language menu button using its ClassName
	expandCollapsePattern := langBut.GetCurrentPatternAs("ExpandCollapse") ; Since this isn't a normal button, we need to get the ExpandCollapse pattern for it and then call the correct method.
	if (expandCollapsePattern.CurrentExpandCollapseState == UIA_ExpandCollapseState_Collapsed) ; Check that it is collapsed
		expandCollapsePattern.Expand() ; Expand the menu
	cUIA.WaitElementExist("Name=English AND ControlType=MenuItem").Click() ; Select the English language
	cUIA.WaitElementExistByNameAndType("I agree", UIA_ButtonControlTypeId).Click() ; If the "I agree" button exists, then click it to get rid of the consent form
	cUIA.WaitPageLoad()
	
	googleImage := cUIA.WaitElementExistByNameAndType("Google", "Image") ; Find the Google image
	CBCondition := cUIA.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_ComboBoxControlTypeId) ; Create a TreeWalker to only look for ComboBox elements in the tree
	CBTW := cUIA.CreateTreeWalker(CBCondition) ; Create the TreeWalker with the ComboBox condition
	searchCB := CBTW.GetNextSiblingElement(googleImage) ; Find the first ComboBox after the Google image
	searchCB.SetValue("search term")
	return
EDIT: code modified to be in accordance with forum rules
Hi Descolada,

I've problems to create the PropertyCondition. Running your example, it stops allways at this point...any ideas what might be wrong here?

Code: Select all

---------------------------
TEST_UIA_Google.ahk
---------------------------
Error:  Method call not supported by UIA_Browser nor UIA_Interface or UIA_Element class.

Specifically: CreatePropertyCondition

	Line#
	013: Run,%browserExe% -incognito --force-renderer-accessibility
	014: WinWaitActive,ahk_exe %browserExe%
	015: cUIA := new UIA_Browser("ahk_exe " browserExe)
	017: cUIA.WaitPageLoad("Neuer Tab",5000)  
	018: cUIA.SetURL("google.com", True)  
	019: cUIA.WaitPageLoad()  
	031: googleImage := cUIA.WaitElementExistByNameAndType("Google", "Image")
--->	032: CBCondition := cUIA.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_ComboBoxControlTypeId)
	033: CBTW := cUIA.CreateTreeWalker(CBCondition)
	034: searchCB := CBTW.GetNextSiblingElement(googleImage)
	035: searchCB.SetValue("search term")  
	036: Return
	039: Reload
	039: Return
	040: Exit

The current thread will exit.
---------------------------
OK   
---------------------------
The vars, have the following values:
UIA_ControlTypePropertyId = 30003
UIA_ComboBoxControlTypeId = 50003

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 19 Jun 2022, 12:36

@aifritz, there was some kind of bug with a previous version of UIA_Browser where that type of error popped up. In addition the Google consent form has changed a little bit, the button being "I accept" or "Accept all" (previously it was only "I accept"). Please try downloading the latest UIA_Interface and UIA_Browser and try this:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>
#include <UIA_Browser>

browserExe := "chrome.exe"
Run, %browserExe% -incognito --force-renderer-accessibility
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe)

cUIA.WaitPageLoad("New Tab",5000)
cUIA.SetURL("google.com", True)
cUIA.WaitPageLoad()

; First get rid of the tracking notice
langBut := cUIA.WaitElementExist("ControlType=MenuItem") ; First lets make sure the selected language is correct. First lets get the language menu button using its ClassName
expandCollapsePattern := langBut.GetCurrentPatternAs("ExpandCollapse") ; Since this isn't a normal button, we need to get the ExpandCollapse pattern for it and then call the correct method.
if (expandCollapsePattern.CurrentExpandCollapseState == cUIA.ExpandCollapseState_Collapsed) ; Check that it is collapsed
	expandCollapsePattern.Expand() ; Expand the menu
cUIA.WaitElementExist("Name=English AND ControlType=MenuItem").Click() ; Select the English language
cUIA.WaitElementExist("Name=I agree OR Name=Accept all AND ControlType=Button").Click() ; If the "I agree" or "Accept all" button exists, then click it to get rid of the consent form
cUIA.WaitPageLoad()

googleImage := cUIA.WaitElementExistByNameAndType("Google", "Image") ; Find the Google image
CBCondition := cUIA.CreatePropertyCondition(cUIA.ControlTypePropertyId, cUIA.ComboBoxControlTypeId) ; Create a TreeWalker to only look for ComboBox elements in the tree
CBTW := cUIA.CreateTreeWalker(CBCondition) ; Create the TreeWalker with the ComboBox condition
searchCB := CBTW.GetNextSiblingElement(googleImage) ; Find the first ComboBox after the Google image
searchCB.SetValue("search term")
return
If you want to use the constants in the old form (eg UIA_ControlTypePropertyId instead of cUIA.ControlTypePropertyId), add #include <UIA_Constants>.

EDIT: alternatively, using cUIA.UIA.CreatePropertyCondition(...) should bypass the cause of the bug.

Loop
Posts: 160
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 19 Jun 2022, 16:58

Hello, can you also make a example with GetAllTabs()

Code: Select all

F6::
cUIA := new UIA_Browser("A")
Tabs := cUIA.GetAllTabs()
For i, Tab in Tabs
Msgbox, % i ":" Tab.GetCurrentUrl() " or " Tab.GetCurrentValue() " or " Tab.GetCurrentName()

Thx

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 19 Jun 2022, 23:47

Descolada wrote:
19 Jun 2022, 04:25
Regarding the information speed from UIA and needing to slow down drag movement... I tried tweaking your code a bit and I think it got better... Note that this uses the freshly-updated UIA_Interface.ahk.
@Descolada, i wanted to try your tweak, but i got this below.

Exception thrown!

what: UIA_Base.__Get
file: testing.ahk
line: 69
message: Property not supported by the UIA_Interface Class.
extra: ControlTypePropertyId


Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 20 Jun 2022, 01:29

@SundayProgrammer, make sure you have the latest UIA_Interface.ahk which has all constants in UIA_Enum class, and such error will be thrown if you try to run newer code with an old version of UIA_Interface.

@Loop, this code assumes you have Chrome open with at least 2 tabs:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>
#include <UIA_Browser>

browserExe := "chrome.exe"
WinActivate, ahk_exe %browserExe%
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe)
tabs := cUIA.GetAllTabs()
tabDump := ""
for i, tabEl in tabs {
	tabDump .= "Name: " tabEl.CurrentName " Value: " tabEl.CurrentValue " ControlType: " tabEl.CurrentControlType  "`n" ; In the old UIA_Interface.ahk version CurrentValue was GetCurrentValue(), so if it throws an error there then you are running an older version
}
MsgBox, % tabDump
MsgBox, Clicking the second tab
WinActivate, ahk_exe %browserExe%
tabs[2].Click()
Sleep, 500
MsgBox, % "A tabs current URL can be accessed only when the tab is active: " cUIA.GetCurrentURL()
MsgBox, Clicking the first tab
WinActivate, ahk_exe %browserExe%
tabs[1].Click()
MsgBox, 4, Closing, Try closing the first tab?
IfMsgBox, Yes 
{
	tabs[1].FindFirstByType("Button").Click()
}
InputBox, output , Tab search, Try entering a partial or full tab name to look for
if !ErrorLevel {
	if (found := cUIA.FindFirstByNameAndType(output, "TabItem",,2, False)) {
		MsgBox, % "Found this tab: " found.CurrentName
		found.Click()
	} Else
		MsgBox, Unfortunately didn't find any tabs with that name!
}
ExitApp
If it throws an error saying "CurrentValue does not exist" or something, either update your UIA_Interface to the newest version, or replace tabEl.CurrentValue with tabEl.GetCurrentValue()

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 20 Jun 2022, 04:55

Descolada wrote:
20 Jun 2022, 01:29
@SundayProgrammer, make sure you have the latest UIA_Interface.ahk which has all constants in UIA_Enum class, and such error will be thrown if you try to run newer code with an old version of UIA_Interface.
@Descolada, your tweak may have made the problem somewhat lesser, but still not significant per my testing. and after further investigation, i think i have found the main cause of it. and attained hundred percent success rate (as far as the tooltip is shown.) the uia + the slowness (of my own machine) are just the small part of the problem, the bigger part of the problem (i just discovered) is that the SysDragImage Destroy Event doesn't get thru sometimes, somehow. (and hence, a possibility is, neither uia nor the slowness are involved at all, probably just the SysDragImage Create Event also doesn't get thru sometimes.)

so, here is the revised version of my code.

Code: Select all

; materials referenced mainly: https://stackoverflow.com/questions/2947139/how-to-know-user-is-dragging-something-when-the-cursor-is-not-in-my-window, https://www.autohotkey.com/board/topic/85231-lib-hook-winevents-to-catch-windows-creationdestruction/, https://stackoverflow.com/questions/13222689/how-do-i-insert-formatted-text-using-autohotkey, https://www.autohotkey.com/board/topic/74670-class-winclip-direct-clipboard-manipulations/, https://stackoverflow.com/questions/7894772/writing-unicode-to-rtf-file, https://www.autohotkey.com/boards/viewtopic.php?p=459451&sid=dde6b407875aadff59bc35e0864f63e4#p459451, https://www.autohotkey.com/boards/viewtopic.php?p=468059&sid=702a6e5619dce3f22bda11a2ec84c4eb#p468059
#Persistent
DetectHiddenWindows, On
OnExit, QUIT
global DragStamp := False
DllCall("CoInitialize", UInt,0)
hWinEventHook := DllCall("SetWinEventHook", UInt,0x8000, UInt,0x8001, UInt,0, UInt,RegisterCallback("WinEventHandler"), UInt,0, UInt,0, UInt,0, UInt)	; EVENT_OBJECT_CREATE:=0x8000, EVENT_OBJECT_DESTROY:=0x8001
Return
QUIT:
UnhookWinEvent(hWinEventHook)
ExitApp

WinEventHandler(hWinEventHook, event, hWnd, idObject, idChild, dwEventThread, dwmsEventTime) {
	static LinkTitle, url
	WinGetClass, wClass, ahk_id %hWnd%
	If wClass = SysDragImage
		If (event == 0x8000)	;drag started
			If (A_TickCount - DragStamp > 29999) and (o := GetFocusedElement()) and not (o.Value = GetURL("A")) and SubStr(o.Value, 1, 8) = "https://"
			{	LinkTitle := o.Name, url := o.Value, DragStamp := A_TickCount, ClipSave(True), Clipboard := "Drag & Drop Operation In Effect:`n`nTitle`t" LinkTitle "`n`nURL`t" url
				SetTimer, MonitorLButton, -1
			}Else{}
		Else	;dropped
		{	While, GetKeyState("LButton", "P")
				Sleep, 100
			Sleep, 500
			ReplaceTheLink(LinkTitle, url)
		}
	Else If wClass = Static
	{	WinGetTitle, wt, ahk_id %hWnd%
		If SubStr(wt, 1, 29) = "Do you want to open this link"
			If (event == 0x8000)
			{	Send, n
				If RegExMatch(wt, "https://.+$", url)
					SetURL(StrReplace(url, "`n"), True, "ahk_exe msedge.exe")
			}
	}
}
ClipSave(Yes) {
	static ClipSave
	If Yes
		ClipSave := ClipboardAll
	Else Clipboard := ClipSave
}
MonitorLButton() {
	static pX, pY, pT
	If GetKeyState("LButton", "P")
	{	MouseGetPos, x, y
		If not (x "," y == pX "," pY)
		{	ToolTip, Drag & Drop Operation In Effect
			pX := x, pY := y, pT := A_TickCount
		}Else If A_TickCount - pT > 500
		{	ToolTip, %Clipboard%
			pT := A_TickCount
		}SetTimer, MonitorLButton, -100
	}Else
	{	Sleep, 500
		RegExMatch(Clipboard, "Title\t\K[^\n]*(?=\n)", LinkTitle), RegExMatch(Clipboard, "URL\t\K[^\n]*$", url), ReplaceTheLink(LinkTitle, url)
		ExitDragDrop()
	}
}
ExitDragDrop() {
	ToolTip
	ClipSave(DragStamp := False)
}
ReplaceTheLink(LinkTitle, url) {
	static priorStamp := False
	If A_TickCount - priorStamp < 3000
		Return
	priorStamp := A_TickCount
	MouseGetPos,,, wHwnd
	WinGetTitle, wt, ahk_id %wHwnd%
	If InStr(wt, " - WordPad")
	{	Send, ^z
		WinClip.Clear(), WinClip.SetRTF("{\rtf>>>{\field{\*\fldinst{HYPERLINK " url " }}{\fldrslt{" Conv4RTF(LinkTitle) "}}}<<<}"), WinClip.Paste()
		Send, {Enter}
		MsgBox,, Manipulation, Done, 1
	}
}
UnhookWinEvent(hWinEventHook) {
	DllCall("UnhookWinEvent", UInt,hWinEventHook)
	DllCall("CoUninitialize")
}
GetURL(wTitle*) {	;written by Descolada (https://www.autohotkey.com/boards/viewtopic.php?p=459451&sid=dde6b407875aadff59bc35e0864f63e4#p459451)
	ErrorLevel := 0
	if !(wId := WinExist(wTitle*)) {
		ErrorLevel := 1
		return
	}
	IUIAutomation := ComObjCreate(CLSID_CUIAutomation := "{ff48dba4-60ef-4201-aa87-54103eef594e}", IID_IUIAutomation := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	DllCall(NumGet(NumGet(IUIAutomation+0)+6*A_PtrSize), "ptr", IUIAutomation, "ptr", wId, "ptr*", elementMain)   ; IUIAutomation::ElementFromHandle
	NumPut(addressbarStrPtr := DllCall("oleaut32\SysAllocString", "wstr", "Address and search bar", "ptr"),(VarSetCapacity(addressbar,8+2*A_PtrSize)+NumPut(8,addressbar,0,"short"))*0+&addressbar,8,"ptr")
	DllCall("oleaut32\SysFreeString", "ptr", addressbarStrPtr)
	if (A_PtrSize = 4) {
		DllCall(NumGet(NumGet(IUIAutomation+0)+23*A_PtrSize), "ptr", IUIAutomation, "int", 30005, "int64", NumGet(addressbar, 0, "int64"), "int64", NumGet(addressbar, 8, "int64"), "ptr*", addressbarCondition)   ; IUIAutomation::CreatePropertyCondition
	} else {
		DllCall(NumGet(NumGet(IUIAutomation+0)+23*A_PtrSize), "ptr", IUIAutomation, "int", 30005, "ptr", &addressbar, "ptr*", addressbarCondition)   ; IUIAutomation::CreatePropertyCondition
	}
	DllCall(NumGet(NumGet(elementMain+0)+5*A_PtrSize), "ptr", elementMain, "int", TreeScope_Descendants := 0x4, "ptr", addressbarCondition, "ptr*", currentURLElement) ; IUIAutomationElement::FindFirst
	DllCall(NumGet(NumGet(currentURLElement+0)+10*A_PtrSize),"ptr",currentURLElement,"uint",30045,"ptr",(VarSetCapacity(currentURL,8+2*A_PtrSize)+NumPut(0,currentURL,0,"short")+NumPut(0,currentURL,8,"ptr"))*0+&currentURL) ;IUIAutomationElement::GetCurrentPropertyValue
	ObjRelease(currentURLElement)
	ObjRelease(elementMain)
	ObjRelease(IUIAutomation)
	return StrGet(NumGet(currentURL,8,"ptr"),"utf-16")
}
GetFocusedElement() {	;based on the script by Descolada (https://www.autohotkey.com/boards/viewtopic.php?p=459451&sid=dde6b407875aadff59bc35e0864f63e4#p459451)
	IUIAutomation := ComObjCreate(CLSID_CUIAutomation := "{ff48dba4-60ef-4201-aa87-54103eef594e}", IID_IUIAutomation := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	DllCall(NumGet(NumGet(IUIAutomation+0)+8*A_PtrSize), "ptr",IUIAutomation, "ptr*",element)	;IUIAutomation::GetFocusedElement
	DllCall(NumGet(NumGet(element+0)+23*A_PtrSize), "ptr", element, "ptr*", name)	;IUIAutomationElement::CurrentName
	DllCall(NumGet(NumGet(element+0)+10*A_PtrSize), "ptr", element, "uint",30045, "ptr", (VarSetCapacity(val,8+2*A_PtrSize)+NumPut(0,val,0,"short")+NumPut(0,val,8,"ptr"))*0+&val)	;IUIAutomationElement::GetCurrentPropertyValue
	ObjRelease(element)
	ObjRelease(IUIAutomation)
	Return {"Name":StrGet(name,"utf-16"), "Value":StrGet(NumGet(val,8,"ptr"),"utf-16")}
}
Conv4RTF(t) {
	Loop, Parse, t
		Result .= Asc(A_LoopField) > 127 ? "\u" Asc(A_LoopField) "?" : A_LoopField
	Return Result
}
SetURL(theURL, NewTab, wTitle*) {	;based on the script written by Descolada (https://www.autohotkey.com/boards/viewtopic.php?p=459451&sid=dde6b407875aadff59bc35e0864f63e4#p459451)
	ErrorLevel := 0
	if !(wId := WinExist(wTitle*)) {
		ErrorLevel := 1
		return
	}
	IUIAutomation := ComObjCreate(CLSID_CUIAutomation := "{ff48dba4-60ef-4201-aa87-54103eef594e}", IID_IUIAutomation := "{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
	DllCall(NumGet(NumGet(IUIAutomation+0)+6*A_PtrSize), "ptr", IUIAutomation, "ptr", wId, "ptr*", elementMain)   ; IUIAutomation::ElementFromHandle
	NumPut(addressbarStrPtr := DllCall("oleaut32\SysAllocString", "wstr", "Address and search bar", "ptr"),(VarSetCapacity(addressbar,8+2*A_PtrSize)+NumPut(8,addressbar,0,"short"))*0+&addressbar,8,"ptr")
	DllCall("oleaut32\SysFreeString", "ptr", addressbarStrPtr)
	if (A_PtrSize = 4) {
		DllCall(NumGet(NumGet(IUIAutomation+0)+23*A_PtrSize), "ptr", IUIAutomation, "int", 30005, "int64", NumGet(addressbar, 0, "int64"), "int64", NumGet(addressbar, 8, "int64"), "ptr*", addressbarCondition)   ; IUIAutomation::CreatePropertyCondition
	} else {
		DllCall(NumGet(NumGet(IUIAutomation+0)+23*A_PtrSize), "ptr", IUIAutomation, "int", 30005, "ptr", &addressbar, "ptr*", addressbarCondition)   ; IUIAutomation::CreatePropertyCondition
	}
	DllCall(NumGet(NumGet(elementMain+0)+5*A_PtrSize), "ptr", elementMain, "int", TreeScope_Descendants := 0x4, "ptr", addressbarCondition, "ptr*", currentURLElement)   ; IUIAutomationElement::FindFirst
	DllCall(NumGet(NumGet(currentURLElement+0)+16*A_PtrSize),"ptr",currentURLElement,"int",10002,"ptr*",URLValuePattern)   ; IUIAutomationElement::GetCurrentPattern
	If NewTab
	{	jswo := "javascript:window.open()"
		DllCall(NumGet(NumGet(URLValuePattern+0)+3*A_PtrSize),"ptr",URLValuePattern,"ptr",&jswo)   ; IUIAutomationValuePattern::SetValue
		Send, {Enter}
		Sleep, 500
	}DllCall(NumGet(NumGet(URLValuePattern+0)+3*A_PtrSize),"ptr",URLValuePattern,"ptr",&theURL)   ; IUIAutomationValuePattern::SetValue
	Send, {Enter}
	DllCall(NumGet(NumGet(currentURLElement+0)+10*A_PtrSize),"ptr",currentURLElement,"uint",30045,"ptr",(VarSetCapacity(currentURL,8+2*A_PtrSize)+NumPut(0,currentURL,0,"short")+NumPut(0,currentURL,8,"ptr"))*0+&currentURL)   ; IUIAutomationElement::GetCurrentPropertyValue
	ObjRelease(URLValuePattern)
	ObjRelease(currentURLElement)
	ObjRelease(elementMain)
	ObjRelease(IUIAutomation)
	return StrGet(NumGet(currentURL,8,"ptr"),"utf-16")
}
#Include WinClipAPI.ahk
#Include WinClip.ahk
2022-06-20 fixed a careless bug, and refined the method for handling "sysdragimage destroy event sometimes failed to callback".
2022-06-23 added a feature to manipulate the reaction when a link is clicked in wordpad.
Last edited by SundayProgrammer on 23 Jun 2022, 06:24, edited 2 times in total.

aifritz
Posts: 301
Joined: 29 Jul 2018, 11:30
Location: Germany

Re: UIAutomation with a focus on Chrome

Post by aifritz » 20 Jun 2022, 12:15

@Descolada, thank you very much for your update. It's working now amazingly good! 8-)

The next issue I am stuck on is how to select a specific value from a web combo box.
Do you have an example for this?

Code: Select all

MyCombo := cUIA.WaitElementExist("ControlType=ComboBox") ;working
expandCollapsePattern := MyCombo.GetCurrentPatternAs("ExpandCollapse") ;working
expandCollapsePattern.Expand() ;working, but do I need to expand the menu for setting a value?
cUIA.WaitElementExist("ControlType=ComboBox").SetValue("My Special Value to set") ;Not working or do I need to click for setting a value? Can I walk perhaps through all CB values?


Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 20 Jun 2022, 14:57

@aifritz, usually a ComboBox has subelements with type MenuItem, so you would need to look for that subelement and call Click() on it. Here is a part of example 6, where the Expand() method is first called to open a Combobox (although for some reason its called MenuItem on the Google page), then the correct element is looked for and clicked:

Code: Select all

langBut := cUIA.WaitElementExist("ClassName=neDYw tHlp8d AND ControlType=Button OR ControlType=MenuItem") ; First lets make sure the selected language is correct. The expression is evaluated left to right: finds an element where ClassName is "neDYw tHlp8d" and ControlType is button OR an element with a MenuItem controltype.

expandCollapsePattern := langBut.GetCurrentPatternAs("ExpandCollapse") ; Since this isn't a normal button, we need to get the ExpandCollapse pattern for it and then call the correct method.
if (expandCollapsePattern.CurrentExpandCollapseState == cUIA.ExpandCollapseState_Collapsed) ; Check that it is collapsed
	expandCollapsePattern.Expand() ; Expand the menu
cUIA.WaitElementExist("Name=English AND ControlType=MenuItem").Click() ; Select the English language
But in your example, you are first looking for the ComboBox and expanding it, but then with the second WaitElementExist you are again looking for a ComboBox, which should be the same as MyCombo... Try inspecting the expanded ComboBox with UIAViewer and search for the desired selection, then call Click on it. If Click doesn't work, then you could try getting the SelectionItemPattern on the desired element and call Select.

Loop
Posts: 160
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 20 Jun 2022, 16:10

Hi, SetUrl does not work with the latest version.
The same script works with version from 9 Jun. But here GetAllTabs does not work
Has it changed anything?


Code: Select all

#NoEnv
#SingleInstance force
SetTitleMatchMode, 2

#include <UIA_Interface>
#include <UIA_Browser>
#include <UIA_Constants>

browserExe := "chrome.exe"
RunChromeExe := "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"
Run, %RunChromeExe% -incognito --force-renderer-accessibility ; Run in Incognito mode to avoid any extensions interfering. Force accessibility in case its disabled by default.
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
cUIA.WaitPageLoad("New Tab", 5000) ; Wait the New Tab page to load with a timeout of 5 seconds
cUIA.SetURL("https://www.google.com/", True) ; Set the URL and navigate to it
cUIA.WaitPageLoad()

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 20 Jun 2022, 23:38

@Loop, I'm unable to reproduce this, your test script worked fine (I did have to change RunChromeExe's "Program Files (x86)" -> "Program Files"). Is the URL not set at all?

But yes, because UIA_Interface was updated, UIA_Browser was as well. Are you running the latest versions of both?

aifritz
Posts: 301
Joined: 29 Jul 2018, 11:30
Location: Germany

Re: UIAutomation with a focus on Chrome

Post by aifritz » 21 Jun 2022, 03:45

@Descolada

Thanks for your support :) ...finally this code does the job. Instead of "MenuItem" I needed "ListItem".

Code: Select all

MyCombo := cUIA.WaitElementExist("ControlType=ComboBox")
expandCollapsePattern := MyCombo.GetCurrentPatternAs("ExpandCollapse")
expandCollapsePattern.Expand() 
cUIA.WaitElementExist("Name=T012 AND ControlType=ListItem",,2).Click() 
One thing i have observed, that UIAViewer shows the names of the Localized Control Types in the language you are currently logged in with, although the script expects the English identifiers. Is there any trick how to always see the English names? (For e.g. I'm logged in as German User, it shows me "Listenobjekt" instead of "list item")

Post Reply

Return to “Scripts and Functions (v1)”