UIA v2

Post your working scripts, libraries and tools.
arcylix
Posts: 54
Joined: 27 Sep 2022, 14:43

Re: UIA v2

Post by arcylix » 26 May 2023, 12:18

I am so appreciative of UIA. It has allowed me to migrate from Power Automate to a sleeker (and remarkably faster) option.

I do have a question, though. In using ahk_exe, is there any way to avoid needing to have the window active to populate UIA_Browser()? It would be nice if I could do something like:

Code: Select all

cEdge := UIA_Browser("ahk_exe msedge.exe")
rather than

Code: Select all

WinActivate "ahk_exe msedge.exe"
WinWaitActive "ahk_exe msedge.exe"
cEdge := UIA_Browser()
Then again, I'm still relatively new to AHK despite using it for the better part of almost a year now, so it's possible that there are cleaner ways to do what I'm trying to do. I just want to avoid window focus stealing because of the need to pop up the window.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 27 May 2023, 01:12

@arcylix, the safest way is to make the window active, but usually the window can also be at least partially visible and will still work. If the window is totally obstructed by another window, minimized, or off-screen, then UIA can't be used to automate Chromium windows (it might still work with other more standard windows, albeit nothing works on *minimized* windows).

You can check whether a window is visible on the screen with something like this:

Code: Select all

#include UIA.ahk

IsWindowVisible(WinTitle) {
    local X, Y, W, H
    if !IsInteger(WinTitle)
        WinTitle := WinExist(WinTitle)
    if !WinTitle
        throw TargetError("Window not found", -1)
    WinGetPos &X, &Y, &W, &H, WinTitle
    if ((WinTitle == UIA.WindowFromPoint(X, Y)) || (WinTitle == UIA.WindowFromPoint(X, Y+H-1)) || (WinTitle == UIA.WindowFromPoint(X+W-1, Y)) || (WinTitle == UIA.WindowFromPoint(X+W-1, Y+H-1)))
        return True
    return False
}

arcylix
Posts: 54
Joined: 27 Sep 2022, 14:43

Re: UIA v2

Post by arcylix » 28 May 2023, 09:39

@Descolada, I may look into that. Regardless, I absolutely love this and it's been working great other than my one "gripe".

LAPIII
Posts: 667
Joined: 01 Aug 2021, 06:01

Re: UIA v2

Post by LAPIII » 29 May 2023, 14:22

There is a glitch in UIA Browser, at least with ElementFromPath where you can't just make a single-line hotkey with script, or else an error will read that the veritable has not been assigned a value, you must first add braces. You can then run the script and afterwards delete the braces to put the hotkey with script on a single-line.

EDIT: It's a bit inconvenient that the UIA Viewer is always on top as I switch programs all the time when I'm using it.

EDIT: Here is the latest script, could you please check just in case if I'm doing this right? The hotkey z it works after I went through the process above and g doesn't seem to work even if I tried that process:

Code: Select all

#Requires Autohotkey v2.0+
#NoTrayIcon
#Include <UIA>
#Include <UIA_Browser>

WinWaitActive "YouTube ahk_exe msedge.exe"
cUIA := UIA_Browser()
ytEl := UIA.ElementFromHandle("YouTube ahk_exe msedge.exe")
#HotIf WinActive("YouTube ahk_exe msedge.exe")

z::ytEl.ElementFromPath("Y/YYY/YqYYYYVRRRR").Click() ; Guide Buttton

g::
{
ytEl.FindElement({Type:"Button", Name:"Guide"}).Click()
}
Sometime when reloading the script concerning this problem, I got:

YouTube.ahk 29_05_23 03⦂41⦂52⦂674 PM.jpg
YouTube.ahk 29_05_23 03⦂41⦂52⦂674 PM.jpg (45.21 KiB) Viewed 3437 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 29 May 2023, 14:57

@LAPIII, could you show the call stack from the error message as well? Also did the error show when reloading the script, or using one of the hotkeys?

LAPIII
Posts: 667
Joined: 01 Aug 2021, 06:01

Re: UIA v2

Post by LAPIII » 29 May 2023, 15:00

The error showed after reloading and using then using a hotkey. Here it is:

YouTube.ahk 29_05_23 04⦂03⦂47⦂956 PM.jpg
YouTube.ahk 29_05_23 04⦂03⦂47⦂956 PM.jpg (66.58 KiB) Viewed 3424 times
Here's a related one that got later:

YouTube.ahk 29_05_23 04⦂07⦂31⦂584 PM.jpg
YouTube.ahk 29_05_23 04⦂07⦂31⦂584 PM.jpg (44.38 KiB) Viewed 3420 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 29 May 2023, 23:46

@LAPIII, I haven't managed to reproduce the error with your provided code, perhaps your original code differs in significant ways? I recommend updating AHK, UIA, and UIA_Browser to the latest version if you haven't already (I also just pushed an update to UIA_Browser to make cUIA start faster in Edge).

I modified your code a bit. Firstly, IsSet(ytEl) makes sure that ytEl is actually set by the time you try the hotkey. Sometimes UIA_Browser initiates slowly, so this makes sure ytEl is actually set.
Then I had to change the path in the z hotkey. I think the Youtube or Edge layout changes from time to time, so it's usually better to avoid paths in browsers anyway.
Also I sped up the g hotkey with two optimizations: cUIA.DocumentElement limits the search to only the document, which means tab elements, address bar etc won't be searched. order:"PostOrder" makes the search happen from deepest element to the top, not from top to bottom.

Code: Select all

#Requires Autohotkey v2.0+
#NoTrayIcon
#Include <UIA>
#Include <UIA_Browser>

WinWaitActive "YouTube ahk_exe msedge.exe"
cUIA := UIA_Browser()
ytEl := UIA.ElementFromHandle("YouTube ahk_exe msedge.exe")

#HotIf WinActive("YouTube ahk_exe msedge.exe") && IsSet(ytEl)

z::ytEl.ElementFromPath("Y/YYY/YqYYYYVRRR0").Click() ; Guide Buttton

g::
{
    cUIA.DocumentElement.FindElement({Type:"Button", Name:"Guide", order:"PostOrder"}).Click()
}
NOTE: You can disable UIAViewer "always on top" mode by right-clicking the status bar where the UIA paths are, and disabling "UIAViewer always on top" setting.

arcylix
Posts: 54
Joined: 27 Sep 2022, 14:43

Re: UIA v2

Post by arcylix » 30 May 2023, 14:52

@Descolada

Using the following code:

Code: Select all

^+e::
{
   getDemos()
   cEdge.SelectTab("HealtheConnections", 1)
   hecUrl := "https://hecprodmrui.myhie.com/mirthresults/Patient.action?filter.firstName=" firstName "&filter.lastName=" lastName "&filter.dateOfBirth=" dateVar
   cEdge.Navigate(hecUrl, "HealtheConnections - Patient List")
   cEdge.JSClickElement("#listpage_tablecontent_mbisubjectslist tbody tr:first-child td:not(.listCheckbox, .noClick)")
   cEdge.WaitPageLoad("HealtheConnections - Edit")
   nDaysURL := cEdge.GetCurrentURL() . "&numberOfDays=730"
   cEdge.Navigate(nDaysURL, "HealtheConnections - Edit")
}
your latest update to the Browser returns this error for me:

Image

Using the previous version, I have no complications. getDemos() is not relevant to this, but merely gathers name, date of birth, etc.

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 31 May 2023, 14:42

@LAPIII, @arcylix, thanks for the bug report, I think I've now fixed a critical bug in the UIA_Browser startup sequence concerning Edge browser so please make sure you update UIA_Browser.ahk to the latest version :)

emp00
Posts: 145
Joined: 15 Apr 2023, 12:02

Re: UIA v2

Post by emp00 » 01 Jun 2023, 14:42

@Descolada : I also had this error with Edge and some specific UIA_Browser routines (after the Vivaldi-improvements had been introduced) - sounds like this was the cause. Cool, I will download and test the latest UIA_Browser library tomorrow... No more time today. Thanks for these constant improvements and the hard work fixing bugs!!!

EDIT 2023-06-02 --> confirmed, the latest UIA_Browser bugfix enabled my Edge-script again: error messages gone, all good!! Thanks!
Last edited by emp00 on 02 Jun 2023, 13:35, edited 2 times in total.

Joefango
Posts: 30
Joined: 05 Mar 2016, 07:36

Re: UIA v2

Post by Joefango » 02 Jun 2023, 12:17

Hi,
I'm trying to click an (hidden) button in the following application Adobe Substance 3D Painter. I managed to get the first step but I'm locked into the second.
Here is what work for the first step:

Code: Select all

AdobeEl := UIA.ElementFromHandle("ahk_exe Adobe Substance 3D Painter.exe")
AdobeEl.ElementFromPath("XrQR").ControlClick(500)
Image

then I would like to click this element of the dropdown menu:
Image
here is some code I tried (among others) but after hours of tries I'm stuck

Code: Select all

AdobeEl.FindElement({Name:"Height"}).ControlClick()
And also tried these
AdobeEl.ElementFromPath("87q").ControlClick()
AdobeEl.FindElement({LocalizedType:"élément de liste", Name:"Height"}).ControlClick()
I'm pretty sure I'm close but I can't get it to work! :cry: any help is welcome

----------

Ps: for the first step I tried to use AutomationId as alternative without success:

Code: Select all

AdobeEl.FindElement({AutomationId:"S4MainWindow.LayersStackView.LayersStackView.channelSelector"}).Click()
Ps bis: I noticed that for some reason UIA inspector throw an error and can't found the program window. That being said it should have work at least once because of the path I found for the first step
Image

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 02 Jun 2023, 15:18

@Joefango, the problem is that the menu you are trying to select the "Height" item from is actually a new window. This means that AdobeEl will not contain it, and neither ElementFromPath nor FindElement will find it. The hint that it's a separate window is that when you inspect it then in UIAViewer you see only the list items and none of the main window elements.

In my setup, the following code worked:

Code: Select all

WinActivate "ahk_exe Adobe Substance 3D Painter.exe"
AdobeEl := UIA.ElementFromHandle("ahk_exe Adobe Substance 3D Painter.exe")
AdobeEl.FindElement({AutomationId:"LayersStackView.LayersStackView.channelSelector"}).ControlClick() ; If the AutomationId doesn't work, just use ElementFromPath instead... Or alternatively {AutomationId:"channelSelector", ClassName:"QComboBox", matchmode:"Substring"} works too
WinWait "ahk_class Qt5158QWindowPopupDropShadowSaveBits" ; Waits for the menu window to appear
UIA.ElementFromHandle().FindElement({Name:"Height"}).ControlClick() ; Gets the element for the appeared window and selects the item
Click() doesn't work in this particular application probably because it's a Qt application which usually implements UIA quite badly. This means that ControlClick() and Click("left") are the leftover options... Nothing to do about that unfortunately.

I couldn't reproduce the "window not found" error. Are you able to reproduce it somehow and tell me the steps? (Or perhaps record a short clip of it happening?)

Joefango
Posts: 30
Joined: 05 Mar 2016, 07:36

Re: UIA v2

Post by Joefango » 04 Jun 2023, 09:46

Hi Descolada, thank you very much for your help (comments are appreciate)
the problem is that the menu you are trying to select the "Height" item from is actually a new window. This means that AdobeEl will not contain it, and neither ElementFromPath nor FindElement will find it. The hint that it's a separate window is that when you inspect it then in UIAViewer you see only the list items and none of the main window elements.
I thought about that but had no idea how to solve it! :crazy:

I think the code is working now but I'm facing issues because sometimes it doesn't work, and when it doesn't work UIA inspector don't work too. If UIA works and find the element then it works in the application... I recorded a video to show you that in some case it works and in some case no. I can't put my finger on why it fails so I made a few tries.
I noticed that when it fails and I restart the application I have a popup that say the application didn't shutdown correctly... sometimes it even crash the application (didn't happened in the video record unfortunately)

ps: (I send you the video link in private)

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 04 Jun 2023, 13:03

@Joefango, I took a look at the issue you're having and unfortunately I have no idea what's causing it. Though it's very likely not related to the UIA library, because if it were then the error would pop up more consistently. I've tried replicating the bug with 3D painter v8.3.0.2094 (non-Steam) with no luck.
The next time it happens, try changing line 7192 of UIA.ahk:

Code: Select all

        try this.RecurseTreeView(UIA.ElementFromHandle(this.Stored.mwId, this.cacheRequest))
        catch {
            this.Stored.TreeView := []
            this.TVUIA.Add("Error: window not found")
        }
to

Code: Select all

this.RecurseTreeView(UIA.ElementFromHandle(this.Stored.mwId, this.cacheRequest))
As you can see, I've unfortunately mislabeled the error as a general "window not found" error, where it actually might be a bunch of other stuff. Without the try-catch block you can see the actual error which you can PM me or post here.

Googling the error code that you got with your main script it seems like the cause might be on the UIAutomation server side (implemented by Microsoft). If possible I'd recommend updating to the latest update of your Windows and run some of the repair commands for example as demonstrated in the following video: https://www.youtube.com/watch?v=XAuqCbmtDRE. But as per this forum post it might even be unsolvable...

Spitzi
Posts: 309
Joined: 24 Feb 2022, 03:45

Re: UIA v2

Post by Spitzi » 05 Jun 2023, 02:44

@Descolada, thanks for this very promising framework.

I was just running UIA.ahk on it's own with AHK2.0.2 on a win10 system, but the UIA-Viewer did not show up on the display. Also if I call UIA.Viewer() from another script, no window pops up. What am I doing wrong?

Thanks ang Greets Simon

Joefango
Posts: 30
Joined: 05 Mar 2016, 07:36

Re: UIA v2

Post by Joefango » 05 Jun 2023, 05:08

Hi, Descolada thank for your feedback
Though it's very likely not related to the UIA
One thing to notice is that when I use the windows inspector there is no error with the detection... (just saying)
otherwise
- I tried all videos fixes (no success)
- I changed the line code in UIA and I got an other error
Image
- My application is on D: drive and maybe that cause a problem of rights
- Something is wrong in my windows installation (need windows reinstall on last resort)
- The Steam version is supposed to be the same but it may have something different than the regular one
I continue my investigation..

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 05 Jun 2023, 09:00

@Joefango, I meant that you should replace this whole bit:

Code: Select all

        try this.RecurseTreeView(UIA.ElementFromHandle(this.Stored.mwId, this.cacheRequest))
        catch {
            this.Stored.TreeView := []
            this.TVUIA.Add("Error: window not found")
        }
with
this.RecurseTreeView(UIA.ElementFromHandle(this.Stored.mwId, this.cacheRequest))
Not just one line but all 5.

I do have a theory that this might be related to caching. Namely Accessibility Insights and inspect.exe don't use caching, but UIAViewer uses it. But this doesn't explain why your code stops working - unless you are also using caching, or FindElement with matchmode StartsWith or RegEx (which also uses caching internally). Other than that I don't know what's causing it, especially if it's not reproducable.

@Spitzi, all errors of UIAViewer are outputted to the debugging console, so could you try running it in VSCode or something similar, and copy the error messages from the debug console (if there are any)?

Spitzi
Posts: 309
Joined: 24 Feb 2022, 03:45

Re: UIA v2

Post by Spitzi » 05 Jun 2023, 09:55

Thanks @Descolada, I am using Scite, so I will look into that.

I had the following uncaught error:
Spoiler
when running the first line of this part of my code:

Code: Select all

				wEl := UIA.ElementFromHandle(activeWinHwnd)
				wEl.FindElement({Name:"Passwort:", Type:"Edit"}).Value := merlinPwd
				Sleep(100)
				wEl.FindElement({Name:"OK", Type:"Button"}).Click()
could it be due to a window not existing anymore?

Thanks a lot Spitzi

Spitzi
Posts: 309
Joined: 24 Feb 2022, 03:45

Re: UIA v2

Post by Spitzi » 05 Jun 2023, 16:50

I am really loving this UIA-Stuff!! Awesome, how easy it is to interact with programs that I could not automate before!

Another question @Descolada, if you don't mind :

is it possible to send an element a DropFiles-Event? As in https://www.autohotkey.com/docs/v2/lib/GuiOnEvent.htm#DropFiles? Or any other method to simulate a drag/drop action of a file on an element?

I did not find anyting about it in the wiki

Thanks again. Spitzi

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 05 Jun 2023, 23:16

@Spitzi, the first error you got usually means that either the window doesn't exist or is inaccessible for some reason. Use WinExist to make sure that it exists, and WinWait with a small sleep to give it time to be ready. Usually that fixes it :)

AFAIK it's not possible to send drag-drop events with UIA, only to detect them. You'd have to use other interfaces for that. Note that it will be quite complicated, so it might be easier to just use MouseClickDrag instead.

Post Reply

Return to “Scripts and Functions (v2)”