UIA v2

[Grigorij]

Hello,

I have an old and UI-wise severely messed up application to automate.
As you can see, on the left I have all corresponding elements with unique controls that theoretically could be automated. But I don't have a tree so I can't figure out how to access an element. When taking the application window as a root element, the tree that is built skips all the controls(?) that are visible on the left.

Help would be appreciated.

[Descolada]

@Grigorij, that looks like a .NET application. If you select the main window, then does it display any kind of tree? Or if you inspect with UIAViewer (run UIA.ahk alone), then is there a tree?

The left-hand side treeview is just all the controls the window has, which means that you can access them with normal Control methods of AHK. You can use WindowSpy to get the names of the controls, and if you wish then you can get the handle to the control with ControlGetHwnd and then use UIA.ElementFromHandle to get the corresponding UIA element.

[Grigorij]

@Grigorij, that looks like a .NET application. If you select the main window, then does it display any kind of tree? Or if you inspect with UIAViewer (run UIA.ahk alone), then is there a tree?

The left-hand side treeview is just all the controls the window has, which means that you can access them with normal Control methods of AHK. You can use WindowSpy to get the names of the controls, and if you wish then you can get the handle to the control with ControlGetHwnd and then use UIA.ElementFromHandle to get the corresponding UIA element.

That worked well, thanks!

[songdg]

cUIA.SelectTab("AutoHotkey", matchMode:=3, caseSensitive:=True)
I try to use the SelectTab function to select the AutoHotkey page, but it throw that error.
Error: This class does not contain property "TabBarElement"
---- D:\UIA-v2-main\Lib\UIA_Browser.ahk
823: }
826: {
▶ 827: Return (searchPhrase == "") ? this.TabBarElement.FindElement({Type:"TabItem", SelectionItemIsSelected:1}) : this.TabBarElement.FindElement({Name:searchPhrase, Type:"TabItem", mm:matchMode, cs:caseSense})
828: }
830: {

[Descolada]

@songdg, I've updated UIA_Browser.ahk, please see if that fixes the problem. If it doesn't, please provide which browser you are using and its version.

[songdg]

@Descolada
Now it only work at the first time when I launch the browser(chrome@120.0.6099.109) and open several pages, after that it fail, showing the same error message.

[Virlix]

@Descolada Hello. Thanks for your help. how can I use your script in Backgound please.

[Descolada]

@songdg, I ran the following code in Chrome 120.0.6099.110 without any problems:

Code: Select all

#Requires AutoHotkey v2
#include <UIA>
#include <UIA_Browser>

browserExe := "chrome.exe"
Run browserExe (browserExe = "msedge.exe" ? " -inprivate" : " -incognito")
WinTitle := "ahk_exe " browserExe
WinWaitActive WinTitle
cUIA := UIA_Browser()
cUIA.NewTab()
cUIA.Navigate("autohotkey.com")
Loop 10 {
    cUIA.NewTab()
    cUIA.Navigate("google.com")
}
cUIA.SelectTab("AutoHotkey", matchMode:=2, caseSensitive:=True)

Can you provide code that reproduces the problem?

@Virlix, you can try adding UIA.AutoSetFocus := False to the beginning of your script which prevents UIA from auto-activating the element before interacting with it. If you are automating a Chromium window then background automation isn't possible by any easy means, because it rejects most input from UIA and ControlSend/ControlClick as well unless the window is active. One possible work-around is automating in a virtual machine, or if you are doing browser automation then I'd recommend Chrome.ahk instead.

[songdg]

@Descolada
Here is a video I recording.https://www.bilibili.com/video/BV1gj411W7vN?share_source=copy_web

Code: Select all

#Requires AutoHotkey v2.0
#SingleInstance Force
#include <UIA_Interface>
#include <UIA_Browser>
WinActivate "ahk_exe chrome.exe"
WinWaitActive "ahk_exe chrome.exe"
cUIA := UIA_Browser()
chrome := UIA.ElementFromHandle("ahk_exe chrome.exe")
Sleep 1000
cUIA.SelectTab("AutoHotkey", 2, True)
ExitApp

[Robt800]

Hi @Descolada

First off - thank you what a cracking piece of software - it has been really helpful. First of all I tried to use the UIA Viewer to get the path from the 3rd party app (Allen Bradley LogixDesigner 5000) which gave me for e.g.

LogixDesignerEl.ElementFromPath("Y/XY/YQ4YE0q").Highlight()

However I quickly realised this path wasn't fixed and was liable to change depending upon what items appeared on screen etc.

Hence I started using the .FindElement which I have working great with fixed names.

However I now have a situation like the following:


The 4 edit lines at the bottom - these are dynamic and could potentially be 100s of lines. I wanted to find the last one so I used the following:

Code: Select all

Last_Line_In_Inputs_Window := LogixDesignerEl.FindElements({Name:"Line", Type:50004, MatchMode:1, index:-1},,,,Inputs_Window_Top_Tab)

Once I add in the MatchMode (or index - but index can only really be used with MatchMode in this instance) - I get errors like the following:

Error: CacheRequest argument requires parameter with type UIA.IUIAutomationCacheRequest, but received UIA.IUIAutomationElement

I'll be able to work around these issues, but if you could point me in the direction to finding dynamic elements - that would be great.

Thanks
Rob

[Descolada]

@Robt800, firstly you are using FindElements which will find all the elements, but you probably wanted to use FindElement instead.

Second, as the error message suggests you are trying to use an element in the argument slot which expects a CacheRequest. In FindElements the 5th argument is a CacheRequest, whereas for FindElement it's StartingElement. I'm not sure what Inputs_Window_Top_Tab represents in the code and where it's located in the tree, but perhaps you meant instead Last_Line_In_Inputs_Window := LogixDesignerEl.FindElement({Name:"Line", Type:50004, MatchMode:1, index:-1},,,,Inputs_Window_Top_Tab) which can also be written as Last_Line_In_Inputs_Window := LogixDesignerEl.FindElements({Name:"Line", Type:50004, MatchMode:1, index:-1, startingElement:Inputs_Window_Top_Tab})

Another option is to find "DragTextBlock" element and then walk one step backwards in the tree using an Edit condition as a filter. Eg
Last_Line_In_Inputs_Window := LogixDesignerEl.FindElement({Name:"DragTextBlock", Type:"Text"}).WalkTree("-1", {Type:"Edit"})

[Robt800]

@Descolada thanks for taking the time to reply. I am sorry for the confusion - I did use

.findelement

to start with, but then began playing when that didnt work. My full function is:

Code: Select all

#Requires AutoHotkey v2.0

#include lib\UIA.ahk


;{#region Funtions
Open_Inputs_Ladder_LogixDesigner()
{
    LogixDesignerEl := UIA.ElementFromHandle(Logix5000_WinTitle . " ahk_exe LogixDesigner.Exe")
    Controller_Organiser_Pane := LogixDesignerEl.FindElement({Name:"Controller Organizer", Type:50033})
    Inputs_Routine := LogixDesignerEl.findelement({Name: "Inputs", Type:50024},,,,Controller_Organiser_Pane)
    Inputs_Routine.Highlight()
    Inputs_Routine.click("right")
    Sleep(200)
    Send("{Down}")
    sleep(100)
    Send("{Enter}")

    loop ; Loop until 'Inputs' window pops up
    {
        try
        {
            Inputs_Window_Top_Tab := LogixDesignerEl.findelement({Name: "MainProgram - Inputs", Type: 50032})
            Sleep(200)
        }
        catch
        {
            ; put this in just for completeness
        }
        else
        {
            break
        }
    }

    First_Line_In_Inputs_Window := LogixDesignerEl.FindElement({Name:"Line 0", Type:50004},,,,Inputs_Window_Top_Tab)
    First_Line_In_Inputs_Window.Highlight()
    Send("{PgDn}")
    Sleep(100)
    Send("^{Right}")
    Sleep(200)
    Send("{Enter}")
    Last_Line_In_Inputs_Window := LogixDesignerEl.FindElement({Name:"Line", Type:50004, MatchMode:1, index:-1},,,,Inputs_Window_Top_Tab)
    Last_Line_In_Inputs_Window.click()
    Last_Line_In_Inputs_Window.value := "Gotcha"
    MsgBox("This completed")
}

Only thing missing from above is the

Logix5000_WinTitle

which is simply pulled from the selected window.
This:

Last_Line_In_Inputs_Window := LogixDesignerEl.FindElement({Name:"Line", Type:50004, MatchMode:1, index:-1},,,,Inputs_Window_Top_Tab)

still throws an error - this time being:

▶ 2678: found := cache.FindCachedElement(condition, scope, index, order, startingElement)

.

I'm going to now play with the 'DragTextBlock' you recommended.
Thanks again

[Descolada]

@Robt800, the problem is you are using StartingElement wrong.

Code: Select all

First_Line_In_Inputs_Window := LogixDesignerEl.FindElement({Name:"Line 0", Type:50004},,,,Inputs_Window_Top_Tab)

This starts the search from Inputs_Window_Top_Tab and moves downwards in the tree until it finds "Line 0", like so:

image.png (8.12 KiB) Viewed 1469 times

Code: Select all

Last_Line_In_Inputs_Window := LogixDesignerEl.FindElement({Name:"Line", Type:50004, MatchMode:1, index:-1},,,,Inputs_Window_Top_Tab)

This also starts the search from Inputs_Window_Top_Tab, but moves upwards in the tree because you are using index -1 (which reverses the search direction to find the first element from the bottom):

image.png (7.91 KiB) Viewed 1469 times

As you can see, this can't find the element you want to find...

[songdg]

@Descolada
It seems this forum has a glitch and lost some recent posts, have you ever seen my last reply?
There's no problem when switch from a blank tab, here is the vedio. https://www.bilibili.com/video/BV1QG411k7bT?share_source=copy_web

[gregster]

It seems this forum has a glitch and lost some recent posts

Unfortunately, we lost a day (or a little more) of posts due to technical problems. See viewtopic.php?f=3&t=124181
We can only invite you to re-post your question.

[cgx5871]

A simple question stumped me.
How to get the path of an ei

Code: Select all

ei:=UIA.SmallestElementFromPoint()
MsgBox ei.ConditionPath

[Descolada]

@cgx5871, you can use Element.GetConditionPath(targetEl), Element.GetUIAPath(targetEl), or Element.GetNumericPath(targetEl) to generate a path from one element to another. In this case you could use

Code: Select all

MouseGetPos(,,&hWnd)
MsgBox UIA.ElementFromHandle(hWnd).GetUIAPath(UIA.SmallestElementFromPoint())

Also, I've just now updated the libraries GetConditionPath method to be more consistent with UIAViewers condition path, because previously GetConditionPath only used Type and index, but now it also includes AutomationId, ClassName and Name as well (like UIAViewer-generated condition path).

[cgx5871]

@Descolada
* will be used. In this case if ClickCount is a number <10, then that number of clicks will be performed.
Is line 2519 wrong? Func Click()
} else if ClickCount > 9 {
SleepTime := cCount, cCount := 1
}
Should SleepTime := ClickCount ?
In addition, similar to the obtained ei, or other SmallestElementFromPoint ei

NotionEl := UIA.ElementFromHandle("ahk_exe Notion.exe")

Should we cache hwnd directly?
Then use "ei.ConditionPath" to directly retrieve the path. Would it be more convenient?
Element.GetConditionPath(targetEl)

Element---targetEl a little confused

[Descolada]

@cgx5871

Is line 2519 wrong? Func Click()
} else if ClickCount > 9 {
SleepTime := cCount, cCount := 1
}
Should SleepTime := ClickCount ?

It is not wrong, cCount == ClickCount. Although using the variable cCount is superfluous, so I'll remove it in the next update. Since Click is almost never used to click more than twice (a double-click) then a ClickCount greater than 9 will be treated as a sleep duration instead, meaning El.Click("left", 10) will sleep 10ms after clicking.

Element---targetEl a little confused

path := Element.GetConditionPath(targetEl) outputs a path such that Element[path*] == targetEl. UIAViewer outputs paths starting from the window element, but a path can actually start from any element and reach any of its descendants.

NotionEl := UIA.ElementFromHandle("ahk_exe Notion.exe")

Should we cache hwnd directly?
Then use "ei.ConditionPath" to directly retrieve the path. Would it be more convenient?

I don't understand the question. You can use NotionEl as the starting point if you are sure that UIA.SmallestElementFromPoint() will return an element that is inside Notion window, meaning that you have validated that the cursor is inside Notion, otherwise no path can be generated.

[cgx5871]

@Descolada
Got it, thank you.

There are some special interfaces.
Use Click.
It cannot be activated using invoke, etc.
I can only implement it by adding a func Visable above.
Do you have any better suggestions?

Code: Select all

Visable(ei,Delay:=0) {      
	p := this.Location, pos := ei.Location
	saveCoordMode := A_CoordModeMouse
	CoordMode("Mouse", "Screen")
		While (p.y+p.h>pos.y+pos.h){
			ei.ControlClick("WheelDown")
			Sleep Delay 
			p := this.Location
		}
		While (p.y<pos.y){
			ei.ControlClick("WheelUp")
			Sleep Delay       
			p := this.Location
		}
	CoordMode("Mouse", saveCoordMode)
}

Finally, I hope ahk click adds a parameter that returns the original mouse position.
and
Add a ControlSend function