UIA v2

Post your working scripts, libraries and tools.
emp00
Posts: 145
Joined: 15 Apr 2023, 12:02

Re: UIA v2 beta

Post by emp00 » 06 May 2023, 10:19

@Descolada : I'm trying to grab the url of the currently running active browser tab (e.g. chrome) with UIA v2. You posted the solution for UIA v1 here.

Automatically converting this script v1->v2 does (of course) not work because UIA-v2 has somewhat different element names etcpp. Anybody having an idea how implement this url-grab functionality with UIA-v2? I think @mockroot just asked the same question in the v1 thread... Many thanks and chapeau! :rainbow:

I tried this (using Vivaldi) --> this returns a cryptic string like "chrome-extension://mpfksdfwasflkkdfgasdsdf/browser.html" but not the url ?!

Code: Select all

#include <UIA>
#include <UIA_Browser>

If !WinWaitActive("ahk_exe vivaldi.exe",, 1)
    ExitApp()
; Initialize UIA_Browser, use Last Found Window (returned by WinWaitActive)
cUIA := UIA_Browser() 
; Grab the browser url and display it in a MsgBox
BrowserURL := cUIA.GetCurrentURL()
MsgBox BrowserURL

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIA v2 beta

Post by Descolada » 06 May 2023, 12:11

@emp00, UIA_Browser unfortunately doesn't officially support Vivaldi, but perhaps I'll take a look at that problem in the future. Try this function instead:

Code: Select all

GetURL(winTitle?, fromDocument:=1) {
    static savedHwnd := 0, saved := 0, edgeTitle := "- Microsoft" Chr(0x200b) " Edge" ; Zero-width space
    if !(hWnd := WinExist(winTitle?))
        throw TargetError("Window not found")
    if !saved || (hWnd != savedHwnd) {
        savedHwnd := hWnd, saved := {}, saved.Window := UIA.ElementFromHandle(hWnd)
        title := WinGetTitle()
        saved.Type := InStr(title, edgeTitle) ? 2 : title ~= "- Vivaldi$" ? 3 :  (InStr(WinGetClass(), "Mozilla") ? 1 : 0)
    }
    if fromDocument = 1 {
        if saved.Type = 1 { ; Mozilla
            offscreen := 1
            try offscreen := saved.Panel.IsOffscreen
            if offscreen { ; Update elements if the panel is offscreen (another tab is selected)
                saved.Panel := saved.Window.FindElement({AutomationId:"panel", matchmode:"Substring", IsOffscreen:0},2)
                saved.Document := saved.Panel.FindElement({Type:"Document"})
            }
        } else if !saved.HasOwnProp("Document") {
            if saved.Type = 3 ; Vivaldi
                saved.Document := saved.Window.FindElement({Type:"Document", i:2})
            else
                saved.Document := UIA.ElementFromChromium(hWnd).FindFirst(UIA.TrueCondition,1) ; The FindFirst part is mostly necessary for Edge
        }
        return saved.Document.Value
    } else {
        if !saved.HasOwnProp("AddressBar")
            saved.AddressBar := saved.Window.FindElement({Type:"Edit"},, saved.Type = 0 ? -1 : 1)
        return saved.AddressBar.Value
    }
}
This beauty should get the URL either from the document element or the address bar. Tested in Edge, Chrome, Firefox, Vivaldi, Brave, and some Chromium apps (eg Discord).

emp00
Posts: 145
Joined: 15 Apr 2023, 12:02

Re: UIA v2 beta

Post by emp00 » 06 May 2023, 13:33

@Descolada Cool, this GetURL beauty indeed works with Vivaldi, Chrome and Edge! Now my script grabs the currently active URL and opens another tab with the same URL -> this works fine with Chrome and Edge -> but not Vivaldi :-( -- actually Vivaldi is pulling 100% CPU already when executing NewTab() and all I can do is terminate / kill my complete Vivaldi session :-(

Does UIA-v2 really not support Vivaldi for this simple task --> opening a new tab? Vivaldi is just a chromium variant and should work fine with UIA (I was hoping...) How can I circumvent this issue?

Code: Select all

#include <UIA>
#include <UIA_Browser> 
GroupAdd("mygroup", "ahk_exe vivaldi.exe")
GroupAdd("mygroup", "ahk_exe chrome.exe")
GroupAdd("mygroup", "ahk_exe msedge.exe") ; Combine all 3 browsers to be detected in "mygroup"
If !WinWaitActive("ahk_group mygroup",, 1) ; This detects all browsers in mygroup
    ExitApp()
cUIA := UIA_Browser() ; Initialize UIA_Browser, use Last Found Window (returned by WinWaitActive)
BrowserURL := GetURL()  ; Grab the current active browser URL -- Script thanks to Descolada!
cUIA.NewTab() ; Open a new tab with the grabbed URL
cUIA.SetURL(BrowserURL, True) ; Set the URL and navigate to it

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIA v2 beta

Post by Descolada » 06 May 2023, 14:15

@emp00, Vivaldi works fine with UIA, just not with UIA_Browser. All browser layouts are different, so pretty much all of them require different approaches. UIA_Browser supports Chrome, Edge, and Firefox, I haven't implemented proper support for anything else (eg Brave, Vivaldi). You can automate it anyway, just without the helper functions of UIA_Browser (until I've had time to implement support for them).

Code: Select all

#include <UIA>

winTitle := "ahk_exe vivaldi.exe"
WinActivate winTitle
WinWaitActive winTitle,,1

vivaldi := UIA.ElementFromHandle(winTitle)
BrowserURL := vivaldi.FindElement({Type:"Document", i:2}).Value
vivaldi.FindElement({Name:"New Tab", Type:"Button", matchmode:"Substring"}).Click(100) ; Clicks "New Tab" button, then sleeps for 100ms to give time for tab to open
vivaldi.WaitElement({Type:"Edit"}).Value := BrowserURL
ControlSend("{LCtrl up}{LAlt up}{LShift up}{RCtrl up}{RAlt up}{RShift up}{LCtrl down}{Enter}{LCtrl up}",,winTitle)

emp00
Posts: 145
Joined: 15 Apr 2023, 12:02

Re: UIA v2 beta

Post by emp00 » 06 May 2023, 16:26

Confirmed working! You're a genius 100% --> thank you!

fenchai
Posts: 292
Joined: 28 Mar 2016, 07:57

Re: UIA v2 beta

Post by fenchai » 17 May 2023, 10:54

it looks like WaitElementFromPath() has no arguments? is there a way to set the timeout? or do I have to implement it myself? WaitElement() has timeout argument.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIA v2 beta

Post by Descolada » 17 May 2023, 12:45

@fenchai, it seems I haven't adequately documented that method. You can set an integer timeout value as the last argument. Eg. Element.WaitElementFromPath("Button1", 2000) uses timeout 2 seconds.

emp00
Posts: 145
Joined: 15 Apr 2023, 12:02

Re: UIA v2 beta

Post by emp00 » 17 May 2023, 14:02

By pure chance, I just noticed this comment on github "Descolada / UIA-v2": Added basic support for Vivaldi and Brave

That's great news! Just tried to find some "HOWTO" and stepped by the UIA-v2-Wiki, however the wiki still mentions "BrowserType: "Chrome", "Edge", "Mozilla" or "Unknown" - so I guess documentation for Vivaldi/Brave is still work in progress? @Descolada: Could you shortly comment what the recent changes mean? I'm a Vivaldi-heavy-user and I'd happily test anything that needs to be tested... Can we now use more-or-less all features available for Chrome 1:1 also with Vivaldi (see my example above, you helped me with a dedicated script for Vivaldi, this may now not be required any more)? Any pitfalls? Are you still working on improvements so it may be too early to communicate/summarize?

Thanks for your incredible efforts! :rainbow: :bravo:

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIA v2 beta

Post by Descolada » 17 May 2023, 23:18

@emp00, yeah, I haven't gotten around to updating the wiki for it and it's a bit of a work-in-progress still. The features should be more or less the same, but I haven't tested it with Vivaldi thoroughly. Be sure to report any bugs you find :) One thing I know is that detecting page loading is more difficult, because if I launch a new Vivaldi window then it takes a couple of seconds to properly load, much slower than Chrome. So I needed to add a Sleep 3000 or something like that after launching Vivaldi and before the UIA_Browser startup call.
The syntax is the same as for any other browser:

Code: Select all

cUIA := UIA_Browser("ahk_exe vivaldi.exe") ; Example, can also target by window name
cUIA.Navigate("google.com")

emp00
Posts: 145
Joined: 15 Apr 2023, 12:02

Re: UIA v2 beta

Post by emp00 » 18 May 2023, 06:20

Dear @Descolada - happily testing! I'm using an AHK script which detects if Chrome, Firefox, Edge or Vivaldi are running (WinWaitActive) and then grabs the currently active URL in order to open this URL in a new tab with Google Translate. I have updated the UIA-v2 libs to your latest revision and added "vivaldi.exe" to the browser group.

Symptoms:
- Works still flawlessly with Chrome and Edge!
- Firefox: No errors thrown, but NewTab() has no effect, the "translate URL" is opened again in the currenly active Tab. thus overwritten!
- Vivaldi: NewTab() fails with this error:
---- C:\XXXXX\AutoHotkey_v2\Libs\UIA-v2-main\Lib\UIA_Browser.ahk
158: If !this.HasOwnProp("NewTabButton")
158: {
▶ 159: lastTab := this.MainPaneElement.FindElement({AutomationId:"tab-", matchmode:"Substring", i:-1}, UIA.TreeScope.Children)
160: this.NewTabButton := this.MainPaneElement.FindElement({Type:"Button", startingElement:lastTab},2)
161: }
Here's my complete script. Thanks so much for looking into this!

Code: Select all

#Requires AutoHotkey v2
#include Libs\UIA-v2-main\Lib\UIA.ahk
#include Libs\UIA-v2-main\Lib\UIA_Browser.ahk
#SingleInstance Force

; Combine all browsers to be detected in "mygroup" in order to make this script work with "all browsers" !
GroupAdd("mygroup", "ahk_exe vivaldi.exe") ; Problem: Vivaldi UIA-v2 support added in May'23 but NewTab throws error!
GroupAdd("mygroup", "ahk_exe firefox.exe") ; Problem: NewTab still does not work with Firefox!
GroupAdd("mygroup", "ahk_exe chrome.exe")
GroupAdd("mygroup", "ahk_exe msedge.exe")

; This detects all browsers in mygroup 
If !WinWaitActive("ahk_group mygroup",, 0.1)
	ExitWithToolTip("No browser window detected within 100ms timeout!", 2000, 310, 3)

Try cUIA := UIA_Browser() ; Initialize UIA_Browser, use Last Found Window (returned by WinWaitActive)
Catch
	ExitWithToolTip("UIA_Browser initialization failed, please try again.", 2000, 310, 3)

Sleep 200 ; This seems to help avoiding a error here! No more problems ever since.
Try BrowserURL := cUIA.GetCurrentURL() ; Hopefully this now also works with all browsers? At least, no errors are thrown with Firefox/Vivaldi!
Catch
	ExitWithToolTip("GetCurrentURL() failed, please try again.", 2000, 310, 3)

; Encode the url with "URLencode / URIencode" so that complex urls are passed correctly to Google translate!
BrowserURLencoded := EncodeDecodeURI(BrowserURL) 

Try cUIA.NewTab() ; <---------- ### This currently fails with Vivaldi and Firefox!!!
Catch
	ExitWithToolTip("NewTab() failed.", 2000, 310, 3)
	
Try cUIA.SetURL("https://translate.google.com/website?op=translate&sl=auto&tl=en&hl=en&u=" . BrowserURLencoded, True) ; Set the URL and navigate to it
Catch
	ExitWithToolTip("SetURL() failed calling translate.google.com.", 2000, 310, 3)
			
ExitWithToolTip("Script successfully executed, done.", 2500, 200, 60)

; Terminate script with a ToolTip & SoundBeep
ExitWithToolTip(msg, timer, soundfreq, soundmillisecs) 
{
	ToolTip msg
	; #Hides the tooltip after 2s without having to use Sleep (which would stop the current thread)
	SetTimer () => ToolTip(), -timer
	SoundBeep soundfreq, soundmillisecs
	Sleep timer
	ExitApp()
}

; URLencode required to correctly pass "complicated" urls with spaces and special characters to Google translate !
EncodeDecodeURI(str, encode := true, component := true) {
    ; Adapted from teadrinker: https://www.autohotkey.com/boards/viewtopic.php?p=372134#p372134
    static Doc, JS
    if !IsSet(Doc) {
        Doc := ComObject("htmlfile")
        Doc.write('<meta http-equiv="X-UA-Compatible" content="IE=9">')
        JS := Doc.parentWindow
        ( Doc.documentMode < 9 && JS.execScript() )
    }
    Return JS.%( (encode ? "en" : "de") . "codeURI" . (component ? "Component" : "") )%(str)
}

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: UIA v2 beta

Post by k0stell0 » 20 May 2023, 02:28

Hi @Descolada
Thanks a lot for your work, I switched to UIA v2 and everything works good so far.

I'm having a bit of a problem accessing an element that has no identifiers I can use, so after searching for some time and checking your GitHub Wiki, I found that there's another way to access an element by using ElementFromPath.

It looks like UIAViewer should have a path parameter:
8) UIAViewer creates a "UIA path" that can be used with the array notation to get elements
Unfortunately, I can't find anything in mine.

I would highly appreciate if you can explain how to use it.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 20 May 2023, 08:08

@k0stell0, have you taken a look at the Getting started section of UIA-v2 wiki? The UIA path is displayed in the bottom left of the v2 UIAViewer, and alternatively may be changed to display either the condition path or numeric path (also explained in the article). UIAViewer can be opened by running UIA.ahk directly.

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: UIA v2

Post by k0stell0 » 20 May 2023, 19:44

Thanks a lot, @Descolada
I didn't realise there's a new UIAViewer if I launch UIA.ahk directly.

So, I was able to get access to the required element by path, but it only has LegacyIAccessible patterns.
There's a "LegacyIAccessible pattern" section in your GitHub Wiki, but after digging into it I still fail to understand how to use those methods.

I have a toggle button, so I was hoping to check its toggle status first and then click it if needed.
I know how to do it with GetCurrentPatternAs("Toggle"), but this one only has the following patterns available:

Code: Select all

LegacyIAccessiblePattern
- ChildId: 0
- DefaultAction: 
- Description: 
- Help: 
- KeyboardShorcut: 
- Name:  (Bridge View)
- Role: ROLE_SYSTEM_CLIENT(10)
- State: 1048576
- Value: 
Any help would be highly appreciated 🙏

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIA v2

Post by thaihoa3189 » 20 May 2023, 21:42

@Descolada
Excuse me, i use JSReturnThroughClipboard method of UIA_Browser but can't get the return value. Please show me specifically how to use the JSReturnThroughClipboard method and get the return value and display it on a MsgBox?

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 21 May 2023, 01:47

@k0stell0, you can access it the same as any other pattern. You don't have to use Element.GetCurrentPatternAs("Toggle"), it's more efficient (and simpler) to use Element.TogglePattern. Similarly you can use Element.LegacyIAccessiblePattern.State to get the State value (in your example it's 1048576). The state value is a bitmask of the possible LegacyIAccessible.State enumerations, so we can find out what they are for example like this:

Code: Select all

#include UIA.ahk

state := 1048576
allStates := "State value: " state "`nIt consists of:`n"
for StateName, StateValue in UIA.LegacyIAccessible.State.OwnProps()
    if state > StateValue && state & StateValue
        allStates .= StateName " (" StateValue ")`n"

MsgBox allStates
So, for example if the LegacyIAccessible state of the element sometimes has UIA.LegacyIAccessible.State.Focused in it's state, you can check in your code whether it's Focused or not by using if Element.LegacyIAccessiblePattern.State & UIA.LegacyIAccessible.State.Focused { ... }.

You can read more about these kind of "bitmask enumerations" from this blog post or Google it.

@thaihoa3189,

Code: Select all

MsgBox cUIA.JSReturnThroughClipboard("2+6")

js := "
(
	function getCurrentURL () {
		return window.location.href;
	}
	getCurrentURL();
)"
MsgBox cUIA.JSReturnThroughClipboard("window.location.href")

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIA v2

Post by thaihoa3189 » 21 May 2023, 02:27

@Descolada
I use this code:

Code: Select all

  cUIA:=UIA_Browser("ahk_exe chrome.exe")
	MsgBox cUIA.JSReturnThroughClipboard("document.querySelector('#profile522273 > dt > a').outerText;")
To return the string "Descolada" from here:
image.png
image.png (64.56 KiB) Viewed 2278 times
But it returns nothing:
image.png
image.png (2 KiB) Viewed 2278 times

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 21 May 2023, 02:38

@thaihoa3189, remove the semicolon from the end and it works.
You can test out the javascript syntax from the browser addressbar by using javascript:alert(code).

For example, typing javascript:alert(document.querySelector('#profile522273 > dt > a').outerText) in the address bar and pressing Enter shows a alert dialog with "Descolada" in it, but javascript:alert(document.querySelector('#profile522273 > dt > a').outerText;) does nothing.

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIA v2

Post by thaihoa3189 » 21 May 2023, 03:25

@Descolada
Thank you very much! Descolada :thumbup:

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: UIA v2

Post by k0stell0 » 21 May 2023, 19:17

Thanks a lot, @Descolada 🙏

Using your code I found that state := 1048576 is part of 1048580 and consists of: Focusable(1048576) and Focused (4)
I tried using if Element.LegacyIAccessiblePattern.State & UIA.LegacyIAccessible.State.Focused but it doesn't determine whether it's checked or not. I assume this is because it doesn't have "Checked" state.
After inspecting it using "Accessibility Insights" I found that it never changes any of LegacyIAccessible patterns. From the user interface, it behaves like it has "Checked" state.

Do you happen to know any other way to check if that button is checked? Perhaps there's another parameter it can store that value.

P.S.
I have another legacy button that has "Checked" state, so I'm using If Element.LegacyIAccessiblePattern.State & UIA.LegacyIAccessible.State.Checked and it works fine.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 22 May 2023, 12:57

@k0stell0, unfortunately besides TogglePattern and LegacyIAccessiblePattern.State I don't know any other ways besides checking for image or pixels at the checkbox location, which you can get with the BoundingRectangle/Location property. Or you can file a feature request with the program developers to implement TogglePattern (or better UIA support in general) :)

Post Reply

Return to “Scripts and Functions (v2)”