UIA v2

[reddyshyam]

@reddyshyam generally it's not possible, because Chromium-based apps (like Brave) require the window to be at least partially visible for UIAutomation to work. This means you can't use UIA to read contents from a non-selected tab, nor a minimized window, nor a window completely obscured by another windows. There are workarounds for the last case, eg hiding the window, bringing it to front, and then sending the window back again, but it isn't 100% reliable. I'd recommend going the Chrome.ahk or Rufaydium approach, as they can be used to automate the browser more reliably and in the background.

Thanks @Descolada! I like to take advantage of brave sessions hence my original request. I am aware of Chrome.ahk or Rufaydium approach but wanted everything to happen via the browser I use and non intrusive way. I guess no easy go. Thanks again for your time, really appreciate it!

[gdqb521]

How to move the mouse to the object position without clicking, and offsetting the center coordinates a little, the following code always clicks

Code: Select all

WeChatEl := UIA.ElementFromHandle("ahk_exe WeChat.exe")
WeChatEl.ElementFromPath("Y/YL").Click("left","0",,"0 -80")

[Descolada]

@gdqb521 you can use Element.Location to get the {x,y,w,h} properties of the element with coordinates relative to the screen, or Element.GetPos() to get the properties with coordinates relative to the setting determined by CoordMode Mouse.

Untested:

Code: Select all

WeChatEl := UIA.ElementFromHandle("ahk_exe WeChat.exe")
MoveMouseToElement(WeChatEl.ElementFromPath("Y/YL"), , -80)

MoveMouseToElement(element, offsetX:=0, offsetY:=0, speed?) {
    SavedCoordMode := A_CoordModeMouse, loc := element.Location
    CoordMode "Mouse", "Screen"
    MouseMove(loc.x+loc.w//2+offsetX, loc.y+loc.h//2+offsetY, speed?)
    CoordMode "Mouse", SavedCoordMode
}

[buster25]

Hi @Descolada I can't find the cause of this error.
My code

Code: Select all

npEl.FindElement({AutomationId:"btnSeleccionar"}).Click()
; --------------------------------------------------------------------------------------------- >  THE SCRIPT STOPS HERE AND ERROR
FileAppend "CLICK EN SELECCIONAR EN FUNC VARIABLES. " A_Hour ":" A_Min ":" A_Sec "`n", "log_treeview.txt"

image.png (27.23 KiB) Viewed 2176 times

image.png (17.55 KiB) Viewed 2176 times

[Descolada]

@buster25, it means that the program you are trying to automate is reporting that the InvokePattern (used to click an element) is available, but in actuality it is not. You could try instead Element.DoDefaultAction() or Element.ControlClick(), perhaps one of those will work.

[buster25]

Thanks @Descolada I'll try it on Monday. It's curious, the script does execute the click, but it stops where I indicated and after a few seconds the error appears.

[altersu]

Hello @Descolada

I want to retrieve some data from a webpage.
The data is student's ID and there are many of them in one webpage.
I checked with UIA viewer and founded that they have same Type and different value.

I can get one ID by using ElementFromPath.
But I cannot get every ID at once.

I tried to use Loop to do it, it worked but I thought maybe there is a better solution?
The elements I need to retrieve has similar path.
For example, the path is ("2,1,1,2,2,1,1,1,1,1,1,1,2,1,a,7") a = 1 to 5
Can I get the elements from path:
("2,1,1,2,2,1,1,1,1,1,1,1,2,1,1,7")
("2,1,1,2,2,1,1,1,1,1,1,1,2,1,2,7")
("2,1,1,2,2,1,1,1,1,1,1,1,2,1,3,7")
("2,1,1,2,2,1,1,1,1,1,1,1,2,1,4,7")
("2,1,1,2,2,1,1,1,1,1,1,1,2,1,5,7")

My code right now:

Code: Select all

WinWaitActive "ahk_exe msedge.exe"
cUIA := UIA_Browser()

!1::
{
    a := 1
    npEl := cUIA.ElementFromHandle("ahk_exe msedge.exe")
    link := Array()
    Loop 5
        {
            msedgeEl := npEl.ElementFromPath("2,1,1,2,2,1,1,1,1,1,1,1,2,1," . a . ",7").Value
            link.Push msedgeEl
            a := a+1
        }
        MsgBox "Done"
        return
}

How to save those elements to an array with a quick and simple way?
Or I have to use Loop?

Any tips or suggestions will be helpful, thanks.

[Descolada]

@altersu first of all, it seems you don't need to use UIA_Browser in this case since you aren't using any browser-specific methods.

Second, using numeric paths to get elements is very unreliable and prone to breaking. Usually there are better ways, eg using FindElement to get the container element for the student IDs.

You can get all the child elements in one go like this:

Code: Select all

!1::
{
    WinWaitActive "ahk_exe msedge.exe"
    msedgeEl := UIA.ElementFromHandle("ahk_exe msedge.exe")
    students := msedgeEl.ElementFromPath("2,1,1,2,2,1,1,1,1,1,1,1,2,1")
    links := []
    for student in students.Children
        links.Push(student[7].Value)

    MsgBox "Done"
}

[altersu]

Thanks for your reply.
I use the numeric path because I don't know how to get the IDs by other method.
Maybe you can check the attachments and show me how to do it in a better way?

I copied the tree from UIAviewer, you can see the path and the ID in red column.
The IDs path are listed below:
2,1,1,2,2,1,1,1,1,1,1,1,2,1,2,8
2,1,1,2,2,1,1,1,1,1,1,1,2,1,3,8
2,1,1,2,2,1,1,1,1,1,1,1,2,1,4,8
2,1,1,2,2,1,1,1,1,1,1,1,2,1,5,8

I tried with your code but the result is empty.
I just started to use UIA v2, so maybe I did something wrong.

[Descolada]

@altersu okay, perhaps something like this?

Code: Select all

!1::
{
    WinWaitActive "ahk_exe msedge.exe"
    msedgeEl := UIA.ElementFromHandle("ahk_exe msedge.exe")
    students := msedgeEl.FindElement({AutomationId:"form1"}).FindElement({Type:"Table"})
    links := []
    for student in students.Children
        try links.Push(student[8].Name)

    MsgBox "Done! Found " links.Length " names"
}

EDIT: removed incorrect line (see next post)

[altersu]

@Descolada
I fixed the code in your first reply.
The second reply showed error:

Error: Method RemoveAt not found in UIA.IUIAutomationElement Class.

Specifically: RemoveAt

037: msedgeEl := UIA.ElementFromHandle("ahk_exe msedge.exe")
038: students := msedgeEl.FindElement({AutomationId:"form1"}).FindElement({Type:"Table"})
▶ 039: students := students.RemoveAt(1)
040: links := []
041: For student in students.Children

The current thread will exit.

So I removed line 39 and tried again.
This time it worked, but it seems slower than the numeric path method.
Why?

Code: Select all

!1::
{
    WinWaitActive "ahk_exe msedge.exe"
    msedgeEl := UIA.ElementFromHandle("ahk_exe msedge.exe")
    students := msedgeEl.ElementFromPath("2,1,1,2,2,1,1,1,1,1,1,1,2,1")
    links := []
    for student in students.Children
        links.Push(student[8].Name)

    MsgBox "Done! Found " links.Length " names"
}



!2::
{
    WinWaitActive "ahk_exe msedge.exe"
    msedgeEl := UIA.ElementFromHandle("ahk_exe msedge.exe")
    students := msedgeEl.FindElement({AutomationId:"form1"}).FindElement({Type:"Table"})
    ;students := students.RemoveAt(1) ; remove first element since it doesn't contain info
    links := []
    for student in students.Children
        try links.Push(student[8].Name)

    MsgBox "Done! Found " links.Length " names"
}

[Descolada]

@altersu it's slower because ElementFromPath "knows" exactly where to look, but FindElement needs to search all the elements in the tree before the match. You could probably optimize it for performance (have you taken a look at my YouTube video tutorials? it has some info on how to do that), or even using the UIA path or condition path should be more reliable than the numeric path.

[altersu]

@Descolada
Got it.
I will check you video tutorial and try to optimize it, thanks!!

[altersu]

@Descolada
I checked the wiki in uia github page and found patterns.
The attachment is the webpage that I need to retrieve IDs.

Is that possibel to get all IDs in red column by using Grid pattern?

[Descolada]

@altersu GridPattern can be used if the element supports it, which isn't always true for tables. However, if it does support it then you could use TableElement.GetItem(row, column) to access the cells.

Alternatively, I think a better performing solution would involve the use of caching. Something like

Code: Select all

!1::
{
    WinWaitActive "ahk_exe msedge.exe"
    msedgeEl := UIA.ElementFromHandle("ahk_exe msedge.exe")
    cacheRequest := UIA.CreateCacheRequest(["Name"],, UIA.TreeScope.Subtree, UIA.AutomationElementMode.None)
    students := msedgeEl.FindElement({AutomationId:"form1"}).FindElement({Type:"Table"},,,,, cacheRequest)
    links := []
    for student in students.CachedChildren
        try links.Push(student.CachedChildren[8].CachedName)

    MsgBox "Done! Found " links.Length " names"
}

If you want to use ElementFromPath instead of FindElement, then you can build the cached element with EdgeEl.ElementFromPath(path).BuildUpdatedCache(cacheRequest)

[gdqb521]

@Descolada
Thank you for writing the function “MoveMouseToElement(element, offsetX := 0, offsetY := 0, speed?)”，it can work。

Hello, I have 44 DataIetms in a DataGrid, { type: “DataItem” ClassName: “Datagridrow”} , but the screen can only see two DataIetms , UIA can see five, how to make him turn the page, let the items one by one on the screen, and follow up? I tried to use loop , scroll ... ,can not work,
What am I supposed to do?

Code: Select all

ProgenesisEl := UIA.ElementFromHandle("Progenesis QI Tutorial HDMSe - Progenesis QI ahk_exe Progenesis QI.exe")
; FilteredCompoundsNum is 44
FilteredCompoundsNum := StrSplit(ProgenesisEl.ElementFromPath("QqKr").Name, " ")[2]
MsgBox FilteredCompoundsNum

myDataGrid := ProgenesisEl.ElementFromPath({ T: 25, CN: "IdentifyCompoundsView" }, { T: 28 })
mydataItemlen := myDataGrid.FindElements({ Type: "DataItem" }, 2).Length
; mydataItemlen is 5
MsgBox mydataItemlen

firstItem := myDataGrid.FindElement({ Type: "DataItem" }, 2)
; how can i do
; myDataGrid.SetScrollPercent(30)
; Error
loop FilteredCompoundsNum
{
	firstItem.WalkTree("+" A_Index).ScrollIntoView()
	firstItem.WalkTree("+" A_Index).Highlight(200).Click()
}

[eugenesv]

Would you please explain why I get an error if I use a cache request (this is similar to what

@Descolada Hi.
I am using UIA to do stuff in an app, which removes the focus from the element it was before. I was thinking I could use UIA.GetFocusedElement() to find out the focused element and when done doing stuff, to give the focus back to the element that had it before.
But the method throws an error: Error.png.

reported, but that conversation resolved without a solution).

It seems to me that using cache for some reason prevents using regular Properties without giving a clear error of such. Just discovered it by accident, so at least a helpful error would be nice, though I don't understand why this is an erorr in the first place.

Code: Select all

#Requires AutoHotKey 2.1-alpha.4
#Include <UIA>
+1::reload()
+2:: {
  WinWaitActive "ahk_exe wordpad.exe"
  appEl := UIA.ElementFromHandle("ahk_exe wordpad.exe")
  cacheReq := UIA.CreateCacheRequest(["Name"],, UIA.TreeScope.Subtree, UIA.AutomationElementMode.None)

  qat0 := appEl.FindElement({Name:"Quick Access Toolbar"},,,,,cacheRequest:=0       ) ; FindElement(condition, scope:=4, index:=1, order:=0, startingElement:=0, cacheRequest:=0)
  qatC := appEl.FindElement({Name:"Quick Access Toolbar"},,,,,cacheRequest:=cacheReq)
  ; Not Cached: WORKS
  msgbox(type(qat0) '`nName=' qat0.Name)
  ; msgbox(type(qat0) '`nName=' qat0.CachedName) ; legible error: Error: (0x80070057) The parameter is incorrect.
  ; Cached: WORKS with cached properties
  msgbox(type(qatC) '`nName=' qatC.CachedName)
  ; Cached: FAILS with regular properties
  msgbox(type(qatC) '`nName=' qatC.Name) ; illegible error: Error: (0x80004005) Unspecified error
}

[Descolada]

@gdqb521 usually only visible UI elements are displayed in the UIA tree so as to not waste computer resources, which explains why only a few elements out of 44 are displayed. If Element.ScrollIntoView() doesn't work then it's unlikely that ScrollPattern would work, but you could still try: in the screenshot the selected element has the Scroll pattern, which means you could try scrolling it with myDataGrid.Scroll(UIA.ScrollAmount.SmallIncrement). If that scrolls it by one element then you could probably reliably extract all the elements (supposing that the UIA tree updates with new elements once you've scrolled down).
Other than that, unfortunately there aren't any good options besides trying to scroll with the keyboard. Perhaps if you Send/ControlSend the Down arrow key it might select a new element, and then you could maybe get the selected element by using the tables Selection pattern: selectedElement := myDataGrid.GetSelection().
Alternatively maybe UIA.GetFocusedElement() would work, or ElementFromPoint (since you know the location of the parent element, you can calculate the coordinates of the first item).

@eugenesv qat0 is created without a cache request, so it doesn't contain any cached properties and that is why qat0.CachedName throws an error.
qatC is created with a cache request that will cache the Name property (accessed with qatC.CachedName), and will not reference the live element at all because UIA.AutomationElementMode.None was used when creating the cache request. If you instead used cacheReq := UIA.CreateCacheRequest(["Name"],, UIA.TreeScope.Subtree) then qatC.Name should also work in addition to qatC.CachedName (and also access to patterns remains), but it is considerably slower.

As to the illegible error, I haven't thought of a good way for UIA-v2 to figure out whether an element has access to live elements or not, without creating significant overhead in the UIA-v2 library... So unfortunately for now the illegible errors remain.

[eugenesv]

Ah, I see, the meaning of AutomationElementMode.None is what I was missing, unforunately the Enum, which I looked at, doesn't document what each variant means, but now I've found a reference in another place that there is no live element with None. Thanks for the clarification

I think it might've been less confusing if the property names remained the same and instead you would have "earlier" mental separation between live and cache variants of the same element by accessing one variante explicitly as Element.Live (or Element.Cache)

[Descolada]

@eugenesv there is another way to do the "mental separation": instead of using Element.Name you could always specify "Current" as in Element.CurrentName. That is the naming convention in the win32 implementation, but I decided to make the "Current" part optional as it is the most commonly used and I was getting tired of typing it so many times. Note that for full effect you could use it with pattern names and methods as well: Element.CurrentSelectionPattern.GetCurrentSelection() instead of Element.SelectionPattern.GetSelection().