UIA v2

elbitjusticiero · Post by **elbitjusticiero** » 12 Mar 2023, 11:21

LAPIII wrote: ↑
12 Mar 2023, 10:47
I think UIA can help you here, but you should first try with WinMove. When the Skype window appears can you bring up Window Spy, it's in C:\Program Files\AutoHotkey?. I'd like to know how to solve this too.

That's the first thing I tried, but, alas, it doesn't work. There are several threads in the old and new forums about this. Toast notifications are apparently not tractable with vanilla AHK.

Descolada · Post by **Descolada** » 12 Mar 2023, 12:35

@elbitjusticiero, I wouldn't use UIA here, but instead preferrably SetWinEventHook, or alternatively set up a ShellHook, or a timer for FindWindowEx (like malcev suggested). Since I don't know how to trigger Skype notifications, I did my testing with the built-in TrayTip command:

Code: Select all

WinEventProc := CallbackCreate(CallbackFunc, "F", 7)
hHook := DllCall("SetWinEventHook", "Int", 32792, "Int", 32792, "Ptr", 0, "Ptr", WinEventProc, "UInt", 0, "UInt", 0, "UInt", 0x2) ; EVENT_OBJECT_UNCLOAKED
OnExit((*) => DllCall("UnhookWinEvent", "Ptr", hHook))
Sleep 1000
TrayTip "#1", "This is TrayTip #1"
Sleep 1000
ExitApp

CallbackFunc(hWinEventHook, event, hWnd, *) {
	wTitle := WinGetTitle(hWnd)
	wClass := WinGetClass(hWnd)
	if wClass = "Windows.UI.Core.CoreWindow" && InStr(wTitle, "notification") {
		WinMove(0,,,,hWnd)
	}
}

ShellHook version:

Code: Select all

DllCall("RegisterShellHookWindow", "UInt", A_ScriptHwnd)
MsgNum := DllCall( "RegisterWindowMessage", "Str", "SHELLHOOK")
OnMessage(MsgNum, ShellMessage)
Sleep 1000
TrayTip "#1", "This is TrayTip #1"
Sleep 1000

ShellMessage(msg, hWnd, *) {
	wTitle := WinGetTitle(hWnd)
	wClass := WinGetClass(hWnd)
	if msg == 6 { ; HSHELL_REDRAW
		if wClass = "Windows.UI.Core.CoreWindow" && InStr(wTitle, "notification") {
			WinMove(0,,,,hWnd)
		}
	}
}

FindWindowEx to find the notification:

Code: Select all

TrayTip "#1", "This is TrayTip #1"
Sleep 1000
hwnd := DllCall("FindWindowEx", "ptr", 0, "ptr", 0, "str", "Windows.UI.Core.CoreWindow", "str", "New notification")
WinMove(0,,,,hwnd)

elbitjusticiero · Post by **elbitjusticiero** » 12 Mar 2023, 17:05

Ah, you see, DllCalls to internal Windows functions are like Chinese to me. That's why I was hoping for UIA to work here. ツ

But thanks for your suggestion! I'll see if I can make it work. Also, I'm hooked with UIA now, hoping to be able to convert some messy browser automation code I use everyday to use your neat library.

Thank you again!

Descolada · Post by **Descolada** » 13 Mar 2023, 01:24

@elbitjusticiero, if you really want to then you can detect Window-opened events with UIA as well, but it isn't much simpler than the DllCalls (and is slower and probably less reliable:

Code: Select all

#include UIA.ahk

h := UIA.CreateEventHandler(WindowOpened)
UIA.AddAutomationEventHandler(h, UIA.GetRootElement(), UIA.Event.Window_WindowOpened)
Sleep 1000
TrayTip "#1", "This is TrayTip #1"
Sleep 20000
ExitApp

WindowOpened(sender, eventId) {
	try {
		if sender.AutomationId = "NormalToastView" {
			WinMove(0,,,,sender.GetWinId())
		}
	}
}

elbitjusticiero · Post by **elbitjusticiero** » 13 Mar 2023, 06:58

Thank you, Descolada! I was actually able to manage it by setting a timer to call that FindWindowEx line at short intervals. I absolutely have no idea what I'm doing there, but it's working. ¯\_(ツ)_/¯

Joefango · Post by **Joefango** » 22 Mar 2023, 08:14

Hello,
I just discover this UIA thing and it looks awesome thank you so much for doing that. I'm a middle noob at programming but with your examples I managed to make (small) progress. For now I want to use UIA lib mainly to perform clicks on toolbar's button in programs.
I have an issue when I try to click on a toolbar's button in a program.
It works with

Code: Select all

Click("left")

but it don't works with only

Code: Select all

Click() or ControlClick()

The problem is that Click("Left") perform the click but also move the mouse cursor and I don't want that.
Any idea why it don't work with just click() ? Here is the code I'm using:

Code: Select all

DesEl := UIA.ElementFromHandle("ahk_exe Designer.exe")
ToolEl := DesEl.FindElement({Name:"Serif.Affinity.UI.Controls.Tools.ToolTray"})
ToolEl.ElementFromPath("2").Highlight().Click()

Descolada · Post by **Descolada** » 22 Mar 2023, 08:24

@Joefango, Click() tries to use a UIAutomation "clicky" method to click the element (eg InvokePattern.Invoke(), TogglePattern.Toggle(), LegacyIAccessiblePattern.DoDefaultAction()), which depends on whether such a method is implemented by the program or not. One possibility is that although you are getting an element, it isn't actually the element you can interact with. Eg. a tooltray might have ListItem elements inside of it, which could be triggered with Click(), but Click("left") will just click the location where the found element is at and triggers the actually clickable element. You might have better luck by inspecting the element with UIAViewer and taking a look at the available patterns: if it has Invoke for example, then you can test it by double-clicking Invoke(). If there are no actionable patterns, then perhaps you are trying to interact with a wrong element, or it just can't be clicked with UIA.
If you share the name of the program and the element you are trying to click, I could take a look at it... Screenshot of UIAViewer while inspecting the target element also helps.

Joefango · Post by **Joefango** » 22 Mar 2023, 09:19

Thanks for your reply I will read it carefully, the program is called Affinity designer from Serif.

perhaps you are trying to interact with a wrong element

The highlight function seems to target the correct button that lets me think it have the correct information

if it has Invoke for example, then you can test it by double-clicking Invoke()

I tried it with no success..
I continue my tests

Descolada · Post by **Descolada** » 22 Mar 2023, 11:44

@Joefango, I checked that application out and those buttons are unfortunately not accessible with UIAutomation. Some other elements are (buttons, menuitems), but for some reason not that toolbars' ones, so unfortunately the only way left is Click("left"). You can save the mouse position before the click and move it back there after the click, so it will be less perceptible.

Example:

Code: Select all

WinActivate "ahk_exe Designer.exe"
WinWaitActive "ahk_exe Designer.exe"
DesEl := UIA.ElementFromHandle("ahk_exe Designer.exe")
ToolEl := DesEl.FindElement({Name:"Serif.Affinity.UI.Controls.Tools.ToolTray"})[2]
ClickRestore(ToolEl)

ClickRestore(element) {
    MouseGetPos(&x, &y)
    element.Click("left")
    MouseMove(x, y)
}

Edit: Arguably this might be an even faster option:

Code: Select all

ClickRestore(element) {
    static SysX := 65535.0/A_ScreenWidth, SysY := 65535.0/A_ScreenHeight, StructSize := 4*4 + A_PtrSize*3
    loc := element.GetClickablePoint()
    DllCall("GetCursorPos", "int64P", &pt64:=0)
    pInput := Buffer(StructSize*2, 0)
    NumPut("ptr", 0, "int", Integer((loc.x)*SysX), "int", Integer((loc.y)*SysY), "int", 0, "uint", 0x8003, "ptr", 0, "ptr", 0
    , "ptr", 0, "int", 0, "int", 0, "int", 0, "uint", 5, "ptr", 0, "ptr", 0, pInput)
    DllCall("SendInput", "uint", 2, "ptr", pInput, "int", StructSize)
    Sleep 1
    DllCall("SetCursorPos", "int", pt64 & 0xFFFFFFFF, "int", pt64 >> 32)
}

Joefango · Post by **Joefango** » 22 Mar 2023, 12:04

No luck this time! I can stick with Click("left") for the moment. Thanks a lot taking from your time to answer me and do some tests

emp00 · Post by **emp00** » 16 Apr 2023, 09:04

Dear Team - I'm using UIA-v2-main\Lib\UIA_Browser.ahk (latest April 12th release) trying to automate chrome. So far working quite nicely with regular chrome browser windows.

However, today I wanted to modify my script by starting chrome in "App mode" via the command line option "--app". Unfortunately UIH_Browser.ahk throws an error, see below. Any ideas? How can I automate a "chrome browser app" instance on my desktop?

Code: Select all

Run "C:\XXXXX\GoogleChromePortable\GoogleChromePortable.exe --app=https:\google.com"

Error: An element matching the condition was not found
---- XXXXX\UIA-v2-main\Lib\UIA_Browser.ahk
127: Loop 2
127: {
▶ 128: this.URLEditElement := this.BrowserElement.FindFirstWithOptions(this.EditControlCondition, 2, this.BrowserElement)
129: Try
129: {
The current thread will exit.

Descolada · Post by **Descolada** » 16 Apr 2023, 10:06

@emp00, this is because UIA_Browser doesn't support Chrome in app mode. It wouldn't be of much use: the biggest point of UIA_Browser is to simplify getting the address bar and navigation buttons, both of which are missing in app mode.
Also it seems that Chrome doesn't play nice with UIA when in app mode. ElementFromHandle doesn't work, ElementFromChromium doesn't work. Currently the only way I know is with ElementFromPoint:

Code: Select all

#include UIA.ahk
hWnd := WinExist("AppModeWindow ahk_exe chrome.exe")
WinActivate hWnd
WinWaitActive hWnd
WinGetPos(&X, &Y, &W, &H)
chrome := UIA.ElementFromPoint(X+W//2, Y+H//2).WalkTree("n", {Type:"Document"})
MsgBox "Current URL: " chrome.Value

Then the whole page is accessible from the chrome element. Though for serious browser automation I'd consider using Chrome.ahk or Rufaydium instead - more reliable, faster, etc...

emp00 · Post by **emp00** » 17 Apr 2023, 15:16

Thanks @Descolada for pointing me here, my posting in the v1 thread was indeed my mistake, sorry for that.

I have further tweaked my script which is supposed to start the Snipping Tool and then a new Chrome browser window with the Google Lens page. See my latest code below.

Problem: It works most of the time, but it's maybe only 90% reliable. How can I make this script more robust? See my comments in the code...

For example, is this While loop a good idea? I copied it from another AHK forum discussion... Do I need such While loops for the other commands as well? What's best practice to ensure none of the commands give undesired "hanging script" situations or error messages? I need to run this script on a few different computers - fast and more slow ones, and I noticed only changing to a different PC makes a big difference requiring additional "Sleep" commands or other tweaks. I don't like that - would like to generate a more reliable script! Any comments how to improve my code would be appreciated.

Code: Select all

#include <UIA>
#include <UIA_Browser>

RunWait "SnippingTool.exe /clip"
; This runs the good old windows snipping tool starting in clipping mode; RunWait will wait until the program finishes before continuing.

Run "C:\XXXXX\GoogleChromePortable\GoogleChromePortable.exe https:\lens.google.com\search?p=" 
; Task: Startup Chrome with new Google Lens tab and wait until it's ready
; This While loop is supposed to help on slow machines, first browser start sometimes takes a few seconds, not yet 100 sure if this really works as designed?
; At least after adding this I had no more errors at the below WinActivate command which sometimes did not work on the slow machine...
While !WinExist("ahk_exe chrome.exe") && !(A_Index=50){ ; waiting 50 times, means 5 secs, max
	Sleep 100
}

if !WinExist(){
	MsgBox "Chrome startup failed, no window was found within 5 seconds timeout!"
	ExitApp
}

WinActivate "ahk_exe chrome.exe"
WinWaitActive "ahk_exe chrome.exe"
; Added this sequence to ensure the window is active, otherwise the following step sometimes hangs because "window not active" ! 
; Is there a better way?

; Initialize UIA_Browser, use Last Found Window (returned by WinWaitActive)
cUIA := UIA_Browser()
cUIA.WaitPageLoad("Google Lens", 1000)
; Wait until the Google Lens page has fully loaded, at least 1000ms

ControlClick "x200 y250", "ahk_exe chrome.exe",,,, "Pos"
; Clicking into the Lens window required to be able to paste the snipped image from clipboard to Google Lens
; No other way found to activate the page, no button or other element found which can be activated or clicked directly
; This workaround is working OK most of the time, but every 5th to 10th try the paste command is too fast or is not accepted ?!

Send "^v"
Sleep 1500
; Paste the snipped image from clipboard to the Google Lens window by sending CTRL-V and wait 1500ms as precaution if WaitElement does not work reliably

; Good Hint from Descolada: "So in UIA-v2 it would be cUIA.WaitElement({Name:"Switch to Translate mode"}).Click(). The condition syntax is explained in the UIA-v2 wiki."
; cUIA.WaitElement({Name:"Switch to Translate mode"},5000).Click()
; This does not work, needs Type:Button !
cUIA.WaitElement({Name:"Switch to Translate mode", Type:"Button"},5000).Click()

; MsgBox "We're done!"

Descolada · Post by **Descolada** » 17 Apr 2023, 18:16

@emp00, for best reliability you should probably look into other browser automation methods such as with Chrome.ahk (which recently got AHK v2 beta version), or Rufaydium. I should also warn you that Google is able to detect browser automation in pretty much any way, and they might not like you automating this process. Instead they provide the Google Vision API which you could use to read the text from the image.

But if using UIA, I would do something like this:

Code: Select all

#include <UIA>
#include <UIA_Browser>

RunWait "SnippingTool.exe /clip"
; This runs the good old windows snipping tool starting in clipping mode; RunWait will wait until the program finishes before continuing.

Run "chrome.exe google.com" 

if !WinWaitActive("Google ahk_exe chrome.exe",, 5)
	ExitWithMsg "Chrome startup failed, no window was found within 5 seconds timeout!"

; Put a small sleep, because if the page is still loading when UIA_Browser is called, 
; then it might get the "Reload page" button incorrect, and assume that page loading means page loaded
; and vice-versa.
Sleep 1000

; Initialize UIA_Browser, use Last Found Window (returned by WinWaitActive)
try cUIA := UIA_Browser()
catch
    ExitWithMsg "Something went wrong with loading UIA_Browser"

; Click Google Lens button
lens := cUIA.WaitElement({Name:"Search by image", Type:"Button"}, 5000)
if !lens
    ExitWithMsg "Google Lens button wasn't found"
else 
    lens.Click()
; Wait for the dialog to open
if !cUIA.WaitElement({Name:"Search", Type:"Button"}, 5000)
    ExitWithMsg "Google Lens dialog didn't load"

Send "^v"
if !WinWaitActive("Google Lens ahk_exe chrome.exe",, 5)
    ExitWithMsg "Google Lens failed to open, no window found within 5 second timeout!"
cUIA.WaitPageLoad()

; Paste the snipped image from clipboard to the Google Lens window by sending CTRL-V and wait 1500ms as precaution if WaitElement does not work reliably
if !(translate := cUIA.WaitElement({Name:"Switch to Translate mode", Type:"Button"}, 5000))
    ExitWithMsg "Google Lens translate button wasn't found"
else
    translate.Click()

; MsgBox "We're done!"

ExitWithMsg(msg) {
    MsgBox msg
    ExitApp
}

I had to change the loaded webpage URL, because I couldn't use Google Lens from lens.google address, instead I had to open it from the Google main page with the Lens button.
The While loops aren't necessary, because WinWaitActive with a timeout value can be used instead.
Avoid infinite loops by providing reasonable time-out values and check whether necessary elements were found or not.
All-in-all, most likely it'll be quite tricky to get it working across different computers reliably

emp00 · Post by **emp00** » 18 Apr 2023, 15:49

@Descolada I am missing words - THANK YOU sooo much, not only for the improved code but even more for the very constructive comments and explanations!! I have learned so much! Works flawlessly on computer1, will test on computer 2+3 tomorrow.

emp00 · Post by **emp00** » 21 Apr 2023, 15:04

Dear Team, I'm now using UIA-v2 for my next automation project. This time I need to use Edge browser and would like to automate a few clicks which the user has to execute every time he uses a special intranet application. The website is delivered by a database system with rather old technical background, initially developed for IE and now ported to Edge with minor changes. Very user-unfriendly, so I want to automate many repetitive clicks.

I managed to automate the first step: Clicking on "Groups & Folders", this unfolds the next level starting with "1. Library" and several folders, see screenshot No.1. But now I need to click on one of the small triangles and I captured the Accessibility Insights output for this small clickable object: "Unlabeled graphic, not focusable", more details see also in the second UIAViewer screenshot. Unfortunately this "Unlabeled graphic" has no name which I could use in UIA! However, the next item in the table is the folder name (which itself cannot be clicked! see screenshot marked "X only this has a name") . So cUIA can find this name but as explained, I need to Click on the "Unlabeled graphic" just above this table/row item.

Question: Is there any chance how I can click this yellow marked "Unlabeled graphic" triangle which does not even seem to be a "button" but something else? As explained, I hope that we can make use of the following named object "X only this has a name" but I need help how to access this preceding triangle element. Maybe you can also identify other characteristics of this triangle element that I can use to access and click it directly with UIA?

My code would look similar to this, but so far it fails because Name & Type both are not visible to cUIA ...

Many thanks for your help. I'm 99% sure there is solution.

Code: Select all

; Click the correct folder triangle
TriangleClicker := cUIA.WaitElement({Name:"????????", Type:"???????"}, 5000)
If !TriangleClicker
    ExitWithMsg "Step2: Clickable triangle object wasn't found within 5 seconds timeout! Please try again or increase timeout."
Else
    TriangleClicker.Click()

------

Descolada · Post by **Descolada** » 22 Apr 2023, 00:12

@emp00, you can use WalkTree or TreeWalkers to get to that element. First get the named element, then walk 2 steps backwards in the tree with a Type:Image condition (filtering only for Image-type elements): TriangleClicker := cUIA.WaitElement({Name:"the named element blacked out on the screenshot"}).WalkTree("-2", {Type:"Image"})

Though I should note that for browser automation I recommend more direct methods (through the CDP protocol), although I'm not sure whether Edge.ahk has been ported over to v2 already. This is because the UI might vary significantly between computers, so using UIA to automate it might be tricky business.

emp00 · Post by **emp00** » 22 Apr 2023, 11:48

@Descolada Unfortunately this results in the following error message, pointing into the UIA library...

Error: (0x80131509)
---- C:\Users\XXXXXXXX\UIA-v2-main\Lib\UIA.ahk
5182: }
5185: {
▶ 5185: Return ComCall(4, this)
5185: }
5188: {
The current thread will exit.

Descolada · Post by **Descolada** » 22 Apr 2023, 12:08

@emp00, without more info, I'm guessing you got that error when you tried to click the image element? Have you checked whether it's clickable by UIA at all (with UIAViewer or Accessibility Insights?
If it's not, then you can use Element.Click("left") or Element.ControlClick() instead.

emp00 · Post by **emp00** » 22 Apr 2023, 12:30

Hey, I found the solution! Type:Image was wrong --> Type:DataItem works! Now I'm happy, my script works very reliably as designed

THANK you again!

TriangleClicker := cUIA.WaitElement({Name:"the named element blacked out on the screenshot"}, 5000).WalkTree("-2", {Type:"DataItem"}))

UIA v2

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta

Re: UIA v2 beta