UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
aifritz
Posts: 301
Joined: 29 Jul 2018, 11:30
Location: Germany

Re: UIAutomation with a focus on Chrome

Post by aifritz » 01 Jul 2022, 01:13

@Descolada thank you for your updates!

If I may turn on wish mode....I would have the following ideas to the UIAViewer:

- For faster code creation, I would find it helpful, if one could open a context menu in the UIAViewer in the TreeView by right clicking on an element, which shows you all access commands including properties and methods to copy this to the clipboard.

- A record mode functionality

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 01 Jul 2022, 04:37

@Descolada, is there a way to get the location of the current tab on chrome's tab bar? i'm imagining something like currenttabboundingrectangle, if it makes sense.

nt-_-ts
Posts: 12
Joined: 27 Apr 2022, 08:02

Re: UIAutomation with a focus on Chrome

Post by nt-_-ts » 01 Jul 2022, 08:09

Is it possible that a software doesn't expose any of it's controls? I am trying to use it on 3DEXPERIENCE from Dassault and it seems that controls are not accessible by any means. I've tried the UIA viewer
UIA 1.png
UIA 1.png (227.09 KiB) Viewed 4162 times
and it highlights basically the entire window. Building the tree looks like pic attached
UIA tree for whole window.png
UIA tree for whole window.png (46.77 KiB) Viewed 4162 times
ACC viewer is similar
ACC 1.png
ACC 1.png (225.97 KiB) Viewed 4162 times
And the regular window spy also doesn't show anything.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 01 Jul 2022, 08:23

@aifritz, thanks for the suggestions! I'm not sure what you mean by "recording mode" though?

@SundayProgrammer, cUIA.MainPaneElement.FindFirstBy("ControlType=TabItem AND SelectionItemIsSelected=1") should return the current tab element, then you can use CurrentBoundingRectangle on that :)

@nt-_-ts, yes, unfortunately it is totally possible since implementing UIAutomation is usually left to the developers of the program. For most regular windows the UIA elements are automatically generated (for example old Win32 windows, most .NET windows etc), but when dealing with custom elements like in your program, then its up to the developer of the program whether to support accessibility or not. So you might just be out of luck here :( You could also check if that program is creating some hidden windows with separate GUIs (which is unlikely), you can use UIATreeInspector.ahk in my GitHub for that.

william_ahk
Posts: 481
Joined: 03 Dec 2018, 20:02

Re: UIAutomation with a focus on Chrome

Post by william_ahk » 01 Jul 2022, 08:40

When I use it for Opera the address bar errors, any idea?
Error in #include file "D:\UIATest\Lib\UIA_Browser.ahk":
0x80004005 - Unspecified error.

Specifically: SetValue (UIA_ValuePattern)

Line#
270: WinActivate,"ahk_id" this.BrowserId
272: Return,this.GetCurrentDocumentElement().CurrentValue
273: }
274: }
276: {
277: this.URLEditElement.SetFocus()
278: valuePattern := this.URLEditElement.GetCurrentPatternAs("Value")
---> 279: valuePattern.SetValue(newUrl " ")
280: if !InStr(this.URLEditElement.CurrentValue, newUrl)
280: {
281: legacyPattern := this.URLEditElement.GetCurrentPatternAs("LegacyIAccessible")
282: legacyPattern.SetValue(newUrl " ")
283: legacyPattern.Select()
284: }
285: if (navigateToNewUrl&&InStr(this.URLEditElement.CurrentValue, newUrl))

The current thread will exit.
When I check it with UIAViewer the control's name is indeed different, but even after I change the name it still has the same error.
AutoHotkey_2022-07-01_21-39-27.png
AutoHotkey_2022-07-01_21-39-27.png (47.04 KiB) Viewed 4141 times

leosouza85
Posts: 90
Joined: 22 Jul 2016, 16:28

Re: UIAutomation with a focus on Chrome

Post by leosouza85 » 01 Jul 2022, 11:53

@Descolada , congratulations for the excelent work... I have a sugestion for simplyfing the code writing, could you add a parametter to the find* functions, so if we put a flag like "1" it wait the element to exist?

Like var.FindFirstbyName("name", 1)

So it will wait for the name to exist and then return the first result.

aifritz
Posts: 301
Joined: 29 Jul 2018, 11:30
Location: Germany

Re: UIAutomation with a focus on Chrome

Post by aifritz » 01 Jul 2022, 12:31

Descolada wrote:
01 Jul 2022, 08:23
@aifritz, thanks for the suggestions! I'm not sure what you mean by "recording mode" though?
I mean a kind of macro recorder, which records all the steps you do... similar to the excel vba recorder.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 01 Jul 2022, 13:28

@leosouza85, I prefer the existing WaitElementExist functions, the name is more descriptive :)

@william_ahk, you can fix that by changing in GetCurrentMainPaneElement() method "Address and search bar" to "Address field". That said, UIA_Browser.ahk is mostly advertised for Chrome (as the thread title suggests) and I test it with Edge too, but I have no plans on supporting other browsers as well. There would be way too many browsers to test and exceptions to add...

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 01 Jul 2022, 22:28

Descolada wrote:
01 Jul 2022, 08:23
@SundayProgrammer, cUIA.MainPaneElement.FindFirstBy("ControlType=TabItem AND SelectionItemIsSelected=1") should return the current tab element, then you can use CurrentBoundingRectangle on that :)
@Descolada, when the current tab is the first tab, it gave me l:171 t:0 r:308 b:33. and when the current tab goes for the rest, all gave me l:43 t:0 r:180 b:33.
and none of the above seems reflect the correct locations.

for vertical tab bar, all gave me blank.

i doesn't have a chrome actually, these came from my msedge :P . i assumed they work the same. probably not though :think: .

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 01 Jul 2022, 23:22

SundayProgrammer wrote:
01 Jul 2022, 22:28
for vertical tab bar, all gave me blank.
I didn't know that Edge even had vertical tabs, cool :D But it looks like it moves the tabs away from the main pane (eg where the URL bar is), and that's why it fails. I'll look into it in the next UIA_Browser update.. In the meanwhile cUIA.FindFirstBy("ControlType=TabItem AND SelectionItemIsSelected=1") is a temporary workaround, which might fail if the content has tab items (like Facebook has for example).
SundayProgrammer wrote:
01 Jul 2022, 22:28
@Descolada, when the current tab is the first tab, it gave me l:171 t:0 r:308 b:33. and when the current tab goes for the rest, all gave me l:43 t:0 r:180 b:33.
and none of the above seems reflect the correct locations.
I'm not sure why this fails, it worked in my Edge. Could you call Dump() on the returned element and see if it's actually the tab or something else?

william_ahk
Posts: 481
Joined: 03 Dec 2018, 20:02

Re: UIAutomation with a focus on Chrome

Post by william_ahk » 02 Jul 2022, 00:33

Descolada wrote:
01 Jul 2022, 13:28
@william_ahk, you can fix that by changing in GetCurrentMainPaneElement() method "Address and search bar" to "Address field". That said, UIA_Browser.ahk is mostly advertised for Chrome (as the thread title suggests) and I test it with Edge too, but I have no plans on supporting other browsers as well. There would be way too many browsers to test and exceptions to add...
Thanks, that did work :thumbup:
I do think we should support as many Chromium-based browsers as possible no? Since the difference is minimal. Maybe we can expose the crucial parts like address bar names to the user so that they could define them accordingly for any browser, and language as well.

Also may I present you a way to select and click elements by css selectors:
(A JSON library will be required)

Code: Select all

    JSGetElementPos(selector) {
        js =
        (
(() => {
    let bounds = document.querySelector("%selector%").getBoundingClientRect().toJSON();
    let zoom = window.devicePixelRatio.toFixed(2);
    for (const key in bounds) {
        bounds[key] = bounds[key] * zoom;
    }
    return JSON.stringify(bounds);
})()
        )
        bounds_str := this.JSReturnThroughClipboard(js)
        bounds := JSON.Load(bounds_str)
        ControlGetPos, win_x, win_y, win_w, win_h, Chrome_RenderWidgetHostHWND1, % "ahk_id " this.BrowserId
        bounds.x += win_x
        bounds.y += win_y
        return bounds
    }
    
    ClickElement(selector, emulateMouse=False) {
        try
            bounds := this.JSGetElementPos(selector)
        catch e
            throw Exception(e.Message, -1, e.What)
        x := (bounds.x + bounds.width / 2), y := (bounds.y + bounds.height / 2)
        if emulateMouse
            Click % x " " y
        else
            ControlClick, % "x" x " y" y, % "ahk_id " this.BrowserId
    }
Last edited by william_ahk on 04 Jul 2022, 04:11, edited 1 time in total.

SundayProgrammer
Posts: 143
Joined: 25 Dec 2020, 12:26

Re: UIAutomation with a focus on Chrome

Post by SundayProgrammer » 02 Jul 2022, 02:10

Descolada wrote:
01 Jul 2022, 23:22
... it looks like it moves the tabs away from the main pane (eg where the URL bar is), and that's why it fails. I'll look into it in the next UIA_Browser update
:thumbup:
Descolada wrote:
01 Jul 2022, 23:22
I'm not sure why this fails, it worked in my Edge. Could you call Dump() on the returned element and see if it's actually the tab or something else?
when the current tab is the first tab, this below dumped (which actually is the information of the second tab :shock: , so strange :? )
Type: 50019 Name: "Scripts and Functions - AutoHotkey Community" LocalizedControlType: "tab item" AutomationId: "view_24"

when the current tab goes for the rest, one by one, this below dumped (which actually is the information of the first tab :shock: , so strange :crazy: ) for every one of them
Type: 50019 Name: "UIAutomation with a focus on Chrome - Page 5 - AutoHotkey Community" LocalizedControlType: "tab item" AutomationId: "view_24"

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 02 Jul 2022, 09:43

@william_ahk, I am not at all opposed to GitHub pull requests for such changes :) For me, I decided that manually testing n browsers for every change I make in UIA_Browser is a real pain, so limiting to the 1-2 most popular ones seemed logical. I do have plans to expose the element names to the user (or at least give a way to change the default action), just haven't gotten around to it yet.

That JS CSS selector scripts seems very cool, I'll check it out soon. You probably could switch Click to ControlClick for a more reliable result ;) And soon there might be a real need for UIA_Browser_JS.ahk extension!

william_ahk
Posts: 481
Joined: 03 Dec 2018, 20:02

Re: UIAutomation with a focus on Chrome

Post by william_ahk » 02 Jul 2022, 20:46

@Descolada Yes and most importantly, users who are displaying the browser in languages other than English will have different element names despite using the same Chrome. I would suggest to make it a language dictionary (associative array) property which enables the user to provide translations for the original English names.

Good idea, I'll extend it to support both methods of click. :trollface:

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 04 Jul 2022, 09:51

Some private chat debugging with @SundayProgrammer resulted in finding a bug in the implementation of UIA_Variant method in the original UIA_Interface.ahk library by jethrow, so I have rewrote some variant handling methods and updated UIA_Interface.ahk. Also added a way to get the current tab element with cUIA.GetTab()

@william_ahk, I've implemented your Click suggestion along with a version for ControlClick into UIA_Browser.ahk (and no need for a JSON file).

william_ahk
Posts: 481
Joined: 03 Dec 2018, 20:02

Re: UIAutomation with a focus on Chrome

Post by william_ahk » 05 Jul 2022, 01:08

@Descolada Awesome :clap:

crocodile
Posts: 98
Joined: 28 Dec 2020, 13:41

Re: UIAutomation with a focus on Chrome

Post by crocodile » 05 Jul 2022, 03:34

Can this tool get the current page source code from Chrome? Thanks.

william_ahk
Posts: 481
Joined: 03 Dec 2018, 20:02

Re: UIAutomation with a focus on Chrome

Post by william_ahk » 05 Jul 2022, 03:41

I was visiting some pages repeatedly and I ran into this problem that .WaitPageLoad() won't wait but instantly skips waiting. Then I discovered window titles were used in the process. Wouldn't it be more reliable to only check the reload button? For example, something like this? (but again, I'm no expert in UIAutomation)

Code: Select all

	WaitPageLoad(targetTitle="", timeOut=10000, sleepAfter=500, stopButtonText="Stop") { 
		startTime := A_TickCount
        isLoading := false
		loop {
            reloadBut := this.UIA.TreeWalkerTrue.GetNextSiblingElement(this.UIA.TreeWalkerTrue.GetNextSiblingElement(this.UIA.TreeWalkerTrue.GetFirstChildElement(this.NavigationBarElement)))
            if (A_TickCount - startTime) >= timeOut
                break
			if (InStr(reloadBut.CurrentName, stopButtonText)) {
                if !isLoading
                    isLoading := true
            } else {
                if isLoading
                    break
            }
			Sleep, 200
		}
	}

When I'm switching between windows using Alt-Tab it appears the modifier keys were interfering with the Enter key in the address bar, causing it to be missed, prefixed with Alt (thus opening the url in new tab), etc. So I added the modifier release keys to it and solved the issue:

Code: Select all

	SetURL(newUrl, navigateToNewUrl = False) { ; Sets the URL bar to newUrl, optionally also navigates to it if navigateToNewUrl=True
		this.URLEditElement.SetFocus()
		valuePattern := this.URLEditElement.GetCurrentPatternAs("Value")
		valuePattern.SetValue(newUrl " ")
		if !InStr(this.URLEditElement.CurrentValue, newUrl) {
			legacyPattern := this.URLEditElement.GetCurrentPatternAs("LegacyIAccessible")
			legacyPattern.SetValue(newUrl " ")
			legacyPattern.Select()
		}
		if (navigateToNewUrl&&InStr(this.URLEditElement.CurrentValue, newUrl))
			ControlSend,, {LCtrl up}{LAlt up}{LShift up}{RCtrl up}{RAlt up}{RShift up}{Enter}, % "ahk_id" this.BrowserId
	}
Last edited by william_ahk on 05 Jul 2022, 04:08, edited 1 time in total.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 05 Jul 2022, 03:48

crocodile wrote:
05 Jul 2022, 03:34
Can this tool get the current page source code from Chrome? Thanks.
Not directly with UIA, but UIA_Browser has the method JSReturnThroughClipboard that can execute Javascript through the URL bar and returns the value through the Clipboard, and Javascript can get the source code.

Untested:

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>
cUIA := new UIA_Browser("A")
MsgBox, % cUIA.JSReturnThroughClipboard("document.documentElement.outerHTML")
Though it's probably better to use Selenium/Rufaydium, since they are probably more reliable in this case :)

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 05 Jul 2022, 08:03

@william_ahk, thanks for the suggestions.
I implemented a better UIA_Browser.WaitPageLoad() method, which doesn't rely on the window title, and also added the ControlSend modifier-key releasing as you suggested. The updated UIA_Browser.ahk is currently available only on GitHub, I'll update the main post when more changes have accumulated.

Post Reply

Return to “Scripts and Functions (v1)”