UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 21 Jun 2022, 04:41

@aifritz, glad to hear that :)
LocalizedControlType does mean that it uses the local language (eg German if your system is using German as the main language), so you can set your system language to English to always see English names... Though I'll consider adding an extra field to UIAViewer to also display the English equivalent from UIA_Enum.UIA_ControlTypeIds method.

aifritz
Posts: 301
Joined: 29 Jul 2018, 11:30
Location: Germany

Re: UIAutomation with a focus on Chrome

Post by aifritz » 21 Jun 2022, 07:23

Hi @Descolada it seems so, as something is broken in your last update.

In the previous version this code was working fine for me:

Code: Select all

cUIA := new UIA_Browser("A")
msgbox,4096,, % cUIA.FindAllByType(50000).length()
In the lastest version, it gives me this error :roll:
---------------------------
ahkit.ahk
---------------------------
Error in
Method call not supported by UIA_Browser nor UIA_Interface or UIA_Element class or an error was encountered.

Specifically: FindAllByType

Line#
1899: cUIA := new UIA_Browser("A")
---> 1900: MsgBox,4096,,cUIA.FindAllByType(50000).length()

The current thread will exit.
---------------------------
OK
---------------------------

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 21 Jun 2022, 10:37

@aifritz, thanks for notifying, it has been fixed. The bug only involved FindAllByType method, no other methods nor UIA_Browser was involved. Removing UIA_Constants involved changing a lot of constant names, it seems at least one got away from me :)

@Loop, I also tested your code in fresh Windows 10 and Windows 11 installs, and SetURL worked fine. Perhaps you have an old version of UIA_Browser or UIA_Interface somewhere in one of the Lib folders that AHK automatically checks, and it uses the wrong one?

Loop
Posts: 160
Joined: 07 Jan 2019, 14:51

Re: UIAutomation with a focus on Chrome

Post by Loop » 21 Jun 2022, 17:22

Please excuse the inconvenience!
I have translated the string "Address and search bar" into local language then it works for me too

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 22 Jun 2022, 01:22

@Descolada

do you know where to find the information "behind" the back button? i want to get the url of the webpage prior to the current one without actually going back to it nor right clicking the back button for a list of the recent webpages. any ideas?

another one: are you aware of a method in uia which can create a new tab next to the current one (not appending it at the end) and behaves a bit like a subordinate which goes back to its "mother" tab instead of focusing the next one (on its righthand side) when it's been closed. any clues?

newbie007
Posts: 20
Joined: 21 Jan 2022, 06:34

Re: UIAutomation with a focus on Chrome

Post by newbie007 » 22 Jun 2022, 08:11

Hello @Descolada ,

I don't know if you've received my private message but I wanted to ask it here also:

I have a question, for example I have found an Element by Name (for example: Link) and got it's rectangles, but I want to click 30 or 50 pixels to the right from the Link box, because the link text is changing so I cannot use the link text, I need to use the Link label and click 30 or 50 pixels to right.

How do I do that?

Code: Select all


element := chrome.FindFirstByName("Link")

link := element + 50

Click, link


Would this work like this?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 22 Jun 2022, 16:06

@newbie007, it seems I have missed your PM. :/ One way would be to get element coordinates from CurrentBoundingRectangle, and then link element with ElementFromPoint. You could also just use regular Click to click:

Code: Select all

element := chrome.FindFirstByName("Link")
bounds := element.CurrentBoundingRectangle ; Returns screen coordinates: l: left, t: top, r: right, b: bottom
;One way:
link := chrome.ElementFromPoint(bounds.r + 50, bounds.t + ((bound.b - bounds.t) // 2))
link.Click()
;Other way:
CoordMode, Mouse, Screen
Click, % (bounds.r + 50) " " (bounds.t + ((bound.b - bounds.t) // 2)) ; Click 50px right from the right border of element, and y coordinate is the middle of element y-coord
Other way would be to create a TreeWalker with the link property and get the next sibling element, then click it.

Code: Select all

tw := chrome.CreateTreeWalker(chrome.CreateCondition("ControlType", "Link"))
link := tw.GetNextSiblingElement(element)
link.Click()
@SundayProgrammer, I don't know how to get the information from the Back button, nor how to long-click it to display the history, but I will investigate it sometime in the near future. But sometimes you can get the previous website URL with javascript document.referrer: MsgBox, % cUIA.JSReturnThroughClipboard("document.referrer;")
And I am not aware of any UIA method of opening a new tab next to the current one. If you have a link on the webpage, then it might be possible to simulate a middle click, but it's probably easier to use AHKs Click function to middle click (you can get the coordinates with CurrentBoundingRectangle or CurrentPos()). Javascript might work also, for example cUIA.JSExecute("window.open('https://google.com');").

User avatar
Joe Glines
Posts: 770
Joined: 30 Sep 2013, 20:49
Location: Dallas
Contact:

Re: UIAutomation with a focus on Chrome

Post by Joe Glines » 22 Jun 2022, 18:57

Descolada wrote:
16 Jun 2022, 11:57
Thank you so much for this code! It's just amazing!!!
We made a short video demonstrating it but have much, much more plans!
Sign-up for the 🅰️HK Newsletter

ImageImageImageImage:clap:
AHK Tutorials:Web Scraping | | Webservice APIs | AHK and Excel | Chrome | RegEx | Functions
Training: AHK Webinars Courses on AutoHotkey :ugeek:
YouTube

:thumbup: Quick Access Popup, the powerful Windows folders, apps and documents launcher!

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 23 Jun 2022, 06:08

Descolada wrote:
22 Jun 2022, 16:06
... I will investigate it sometime in the near future

...cUIA.JSExecute("window.open('https://google.com');")
:thumbup:

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 23 Jun 2022, 13:24

@Joe Glines, glad to see UIAutomation getting more popular! Hopefully more people can start enjoying it :) (and motivates me to fix important bugs faster :D)

nt-_-ts
Posts: 12
Joined: 27 Apr 2022, 08:02

Re: UIAutomation with a focus on Chrome

Post by nt-_-ts » 23 Jun 2022, 23:23

Learned about this by getting an email from Joe, really looking forward to testing this, my main engineering program is losing old controls in favour of UIA each and every update.

newbie007
Posts: 20
Joined: 21 Jan 2022, 06:34

Re: UIAutomation with a focus on Chrome

Post by newbie007 » 24 Jun 2022, 05:44

Descolada wrote:
22 Jun 2022, 16:06
@newbie007, it seems I have missed your PM. :/ One way would be to get element coordinates from CurrentBoundingRectangle, and then link element with ElementFromPoint. You could also just use regular Click to click:

Code: Select all

element := chrome.FindFirstByName("Link")
bounds := element.CurrentBoundingRectangle ; Returns screen coordinates: l: left, t: top, r: right, b: bottom
;One way:
link := chrome.ElementFromPoint(bounds.r + 50, bounds.t + ((bound.b - bounds.t) // 2))
link.Click()
;Other way:
CoordMode, Mouse, Screen
Click, % (bounds.r + 50) " " (bounds.t + ((bound.b - bounds.t) // 2)) ; Click 50px right from the right border of element, and y coordinate is the middle of element y-coord
Other way would be to create a TreeWalker with the link property and get the next sibling element, then click it.

Code: Select all

tw := chrome.CreateTreeWalker(chrome.CreateCondition("ControlType", "Link"))
link := tw.GetNextSiblingElement(element)
link.Click()
@SundayProgrammer, I don't know how to get the information from the Back button, nor how to long-click it to display the history, but I will investigate it sometime in the near future. But sometimes you can get the previous website URL with javascript document.referrer: MsgBox, % cUIA.JSReturnThroughClipboard("document.referrer;")
And I am not aware of any UIA method of opening a new tab next to the current one. If you have a link on the webpage, then it might be possible to simulate a middle click, but it's probably easier to use AHKs Click function to middle click (you can get the coordinates with CurrentBoundingRectangle or CurrentPos()). Javascript might work also, for example cUIA.JSExecute("window.open('https://google.com');").

Thanks for your answer

I have tried both methods.. both didn't work.
I tried to output what CurrentBoundingRectangle returns, but I get nothing returned, so it doesn't send any clicks.

the second method is clicking but relative to the mouse, it doesn't take the rectangle as offset.

I would preffer to use the first method, since it react directly on chrome, but idk what I do wrong.

code:

Code: Select all

object1 := chrome.WaitElementExistByName("Link")
	bounds := object1.CurrentBoundingRectangle()   ; <------------- I have tried both with and without brackets ()
	link := chrome.ElementFromPoint(bounds.r + 30, bounds.t + ((bound.b - bounds.t) // 2))
	link.Click()

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 24 Jun 2022, 07:21

Descolada wrote:
23 Jun 2022, 14:09
All examples are also available at GitHub. The reason I am creating a new post for this is that the main post is reaching its character limit...
Not sure if that setting can be changed for this thread (feel free to request it with my fellow moderators/forum admins), but I’d recommend to create the posting as a list of consecutive spoilered sections. That way the whole thread appears less bloated :arrow: ("tl:dr") :shifty:
For that, the forum offers the spoiler2= feature that allows to label them differently ie like their headlines.
Thx for your amazing work/effort.

Bo 8-)

PS. meanwhile a moderator can move your example section right under your initial posting, if requested. ;)

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 24 Jun 2022, 10:44

@newbie007, right now its hard to guess what is going wrong here, needs some debugging. :)
First of all make sure you are getting the element you want to get with Dump().
Once you know you have the correct element, use CurrentBoundingRectangle (without the parenthesis, because its a property not a method) and display its output, making sure the numbers are correct (check with UIAViewer or Accessibility Insights).
Then try ElementFromPoint and display the found elements information with Dump(), making sure it found the right element. For example, sometimes UIA doesn't find the deepest element (this is not related to my library, UIA is just implemented that way) so you might need to do some additional filtering on child elements.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>
#include <UIA_Browser>

browserExe := "chrome.exe"
Run, %browserExe% -incognito --force-renderer-accessibility "https://www.autohotkey.com/"
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe)
cUIA.WaitPageLoad("AutoHotkey",,1000)

dlLink := cUIA.FindFirstByName("Download")
MsgBox, % dlLink.DumpAll()
dlLinkLoc := dlLink.CurrentBoundingRectangle
MsgBox, % cUIA.PrintArray(dlLinkLoc)

forumsLink := SmallestElementFromPoint(cUIA, dlLinkLoc.r+100, dlLinkLoc.t + ((dlLinkLoc.b-dlLinkLoc.t) // 2))
MouseMove, % (dlLinkLoc.r+100), % (dlLinkLoc.t + ((dlLinkLoc.b-dlLinkLoc.t) // 2)) ; Moves the mouse cursor to the point that was used to look for the element.
MsgBox, % forumsLink.Dump()
forumsLink.Click()

SmallestElementFromPoint(IUIA, x="", y="", activateChromiumAccessibility=False) {
	element := IUIA.ElementFromPoint(x, y, activateChromiumAccessibility)
	bound := element.CurrentBoundingRectangle, elementSize := (bound.r-bound.l)*(bound.b-bound.t), prevElementSize := 0
	for k, v in element.FindAll(IUIA.TrueCondition) {
		bound := v.CurrentBoundingRectangle
		if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b) && ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize))
			element := v, elementSize := newSize
	}
	return element
}
The TreeWalker example might not be finding the correct element because your desired link might not be the next one (NextSibling), perhaps its the one over or inside a different container element altogether. This is hard to tell exactly without knowing what page you are working with... Inspect the elements with UIAViewer and display the tree for the whole window, then see where your elements are located relative to each other and modify your code accordingly.

@BoBo, thanks for the tip, working on it :)
Last edited by Descolada on 24 Jun 2022, 11:00, edited 2 times in total.

gregster
Posts: 8916
Joined: 30 Sep 2013, 06:48

Re: UIAutomation with a focus on Chrome

Post by gregster » 24 Jun 2022, 10:46

BoBo wrote:
24 Jun 2022, 07:21
PS. meanwhile a moderator can move your example section right under your initial posting, if requested. ;)
I doubt that a moderator can do that, sounds more like admin stuff, but perhaps I misunderstand something.
Edit: Looks like it's done already.

newbie007
Posts: 20
Joined: 21 Jan 2022, 06:34

Re: UIAutomation with a focus on Chrome

Post by newbie007 » 24 Jun 2022, 14:49

Descolada wrote:
24 Jun 2022, 10:44
@newbie007, right now its hard to guess what is going wrong here, needs some debugging. :)
First of all make sure you are getting the element you want to get with Dump().
Once you know you have the correct element, use CurrentBoundingRectangle (without the parenthesis, because its a property not a method) and display its output, making sure the numbers are correct (check with UIAViewer or Accessibility Insights).
Then try ElementFromPoint and display the found elements information with Dump(), making sure it found the right element. For example, sometimes UIA doesn't find the deepest element (this is not related to my library, UIA is just implemented that way) so you might need to do some additional filtering on child elements.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>
#include <UIA_Browser>

browserExe := "chrome.exe"
Run, %browserExe% -incognito --force-renderer-accessibility "https://www.autohotkey.com/"
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe)
cUIA.WaitPageLoad("AutoHotkey",,1000)

dlLink := cUIA.FindFirstByName("Download")
MsgBox, % dlLink.DumpAll()
dlLinkLoc := dlLink.CurrentBoundingRectangle
MsgBox, % cUIA.PrintArray(dlLinkLoc)

forumsLink := SmallestElementFromPoint(cUIA, dlLinkLoc.r+100, dlLinkLoc.t + ((dlLinkLoc.b-dlLinkLoc.t) // 2))
MouseMove, % (dlLinkLoc.r+100), % (dlLinkLoc.t + ((dlLinkLoc.b-dlLinkLoc.t) // 2)) ; Moves the mouse cursor to the point that was used to look for the element.
MsgBox, % forumsLink.Dump()
forumsLink.Click()

SmallestElementFromPoint(IUIA, x="", y="", activateChromiumAccessibility=False) {
	element := IUIA.ElementFromPoint(x, y, activateChromiumAccessibility)
	bound := element.CurrentBoundingRectangle, elementSize := (bound.r-bound.l)*(bound.b-bound.t), prevElementSize := 0
	for k, v in element.FindAll(IUIA.TrueCondition) {
		bound := v.CurrentBoundingRectangle
		if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b) && ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize))
			element := v, elementSize := newSize
	}
	return element
}
The TreeWalker example might not be finding the correct element because your desired link might not be the next one (NextSibling), perhaps its the one over or inside a different container element altogether. This is hard to tell exactly without knowing what page you are working with... Inspect the elements with UIAViewer and display the tree for the whole window, then see where your elements are located relative to each other and modify your code accordingly.

@BoBo, thanks for the tip, working on it :)
I could tell you and maybe we could work together on it. Do you have Discord? If yes, I would be happy if you private message me your addy.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 27 Jun 2022, 07:43

Is there anything you can do to fix the UIA_Viewer's ability to detect the min/max/close buttons in Chrome?
image.png
image.png (73.31 KiB) Viewed 4027 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 27 Jun 2022, 08:12

@Skrell, there is something about the buttons that they can't be directly visualized by hovering over, this is not only for UIAViewer but for other tools (Accessibility Insights, Inspect.exe) as well. However when you click "Construct tree for whole Window" and scroll all the way down, the buttons are somewhere under "pane "Google Chrome"". Then when you click on them, the blue box will show the correct place.

Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 27 Jun 2022, 09:13

Descolada wrote:
27 Jun 2022, 08:12
@Skrell, there is something about the buttons that they can't be directly visualized by hovering over, this is not only for UIAViewer but for other tools (Accessibility Insights, Inspect.exe) as well. However when you click "Construct tree for whole Window" and scroll all the way down, the buttons are somewhere under "pane "Google Chrome"". Then when you click on them, the blue box will show the correct place.
Is there any way roll this functionality automatically into the default UIViewer? This is literally the ONLY thing I've found not working 100% correctly in this amazing library!

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 29 Jun 2022, 13:26

Added some TextPattern and TextRange examples to the second post in the thread. Also updated UIA_Interface.ahk with many bugfixes and some new features (eg UIA_Element.CurrentExists, WaitElementNotExist, ControlClick elements).

@Skrell, I updated UIAViewer.ahk to do what you wished. There is now a global DeepSearchFromPoint := False line at the top, when set to True will also find Chromes min-max buttons. This will impact the performance though...

Post Reply

Return to “Scripts and Functions (v1)”