UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
jsong55
Posts: 219
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 08 Oct 2022, 21:45

Hi @Descolada is there a way to select tab by URL rather than by name?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 09 Oct 2022, 00:16

@jsong55, in most browsers no, because the URL will be available only once the tab is selected (the URL bar value and the Document element value will both then update). So you would need to loop through the tabs to find the correct one.
Firefox might be an exception because there all tabs' content (including the URL) is available without selecting them and I think that all the documents elements are in the same order as the tabs, so you could perhaps implement a function to find the correct document elements number and then select the corresponding tab.
Doing that with Chrome.ahk/Rufaydium should be possible though.

jsong55
Posts: 219
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 09 Oct 2022, 01:47

@Descolada
well put

I answer my own question here for the record then, in case anyone wants to "SelectTab" via the URL instead of the title

Code: Select all

cUIA := new UIA_Browser("ahk_exe chrome.exe")
page:=chrome.getpagebyURL("mail.google.com","contains")
title:=page.evaluate("document.title").value
cUIA.SelectTab(title)

safetycar
Posts: 435
Joined: 12 Aug 2017, 04:27

Re: UIAutomation with a focus on Chrome

Post by safetycar » 09 Oct 2022, 03:02

Descolada wrote:
09 Oct 2022, 00:16
Firefox might be an exception because there all tabs' content (including the URL) is available without selecting them and I think that all the documents elements are in the same order as the tabs
It would take a bit more effort than it seems at first sight, tabs appear to show in order until some tab is reordered, that kind of change doesn't reflect in UIA or Acc.
EDIT:
Clarifying: There are 2 ways to look at tabs, one is the elements that contain the url and document, and other are the tab bar buttons.
The ones with url don't reflect order changes. The tab buttons should reflect changes but it has to be searched to what url belongs (independent trees).
This can be observed with "UIATreeInspector.ahk".

jsong55
Posts: 219
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 09 Oct 2022, 21:29

I'm trying to scrap a table from a webpage where the control type is "Table" and there are items called header, and item.

How do I extract all the info in the table as an object?

I used DumpAll excessively. This sort of works.

Code: Select all

contact_name:=cUIA.FindAllByType("Hyperlink")
for i,v in contact_name
{
    dumpvalue:=v.dumpall()
    if instr(dumpvalue,"contact details")
        output.="Contact - " dumpvalue "`n"
    else if instr(dumpvalue,"mailto:")
        output.="Email - " dumpvalue "`n---`n"
}
clipboard:=output
Then after this some Regex to extract all the values nicely

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 10 Oct 2022, 00:44

@jsong55, you need to get all the elements from the table with either FindAll or GetChildren, and then check the Name or Value property to get the content. Using this CSS page as an example:

Code: Select all

#Include <UIA_Interface>

wExe := "ahk_exe chrome.exe"
UIA := UIA_Interface()
WinActivate, %wExe%
WinWaitActive, %wExe%
chrome := UIA.ElementFromHandle(wExe)
table1 := chrome.FindFirstByType("Table")
out := ""
for i, rowEl in table1.GetChildren() {
    out .= "Row " i ": "
    for j, item in rowEl.GetChildren()
        out .= item.Name ", "
    out := SubStr(out, 1, StrLen(out)-2) "`n"
}
MsgBox, %out%
ExitApp
At least on that page (and running the latest version of Chrome), the Table element has Custom type children that are the rows, and then every child of the Custom element is a DataItem type column element, where the Name property of the item contains the content.

jsong55
Posts: 219
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 10 Oct 2022, 10:09

@Descolada
Thanks! That's a great solution. I was trying to follow the TreeWalker Wiki and kinda got lost

dbareis
Posts: 39
Joined: 04 Feb 2022, 17:57

Re: UIAutomation with a focus on Chrome

Post by dbareis » 13 Oct 2022, 02:31

OK, I'm completely new to this and have now spent many hours trying to read the value of a control in Chrome.

The PAGE is: https://www.findagrave.com/memorial/244311993/brandon-fraser

I am trying to get the BIRTH value (20 Jul 1967). UIAViewer shows that the "Name" & "BoundingRectangle" exist but I can only seem to get the control type (text). I also have to use "P2.+1" instead of "P1.+1" as the output from "Example4_ChromeTest.ahk" implies, it's tree differs from the one "UIAViewer.ahk" generates (which I don't understand).

The Highlight() does highlight the birth date as expected.

Code: Select all

...
cUIA := new UIA_Browser("ahk_exe " BrowserExe) ; Initialize UIA_Browser, which also initializes UIA_Interface
        X := "TEXT:" .  cUIA.GetAllText() ;WORKS


Element := cUIA.FindFirstByNameAndType("BIRTH", "Text")
    ;Element.Highlight()

NextParentSibling = "P3"
NextParentSibling = "P2.+1"
Element := Element.FindByPath(NextParentSibling)
    ;Element.CurrentValue := "FRED"
    ;Element.SetValue("FRED")
    Element.Highlight()
    ;VP := Element.GetCurrentPatternAs("Value")
    ;X := "VALUE: " . VP.Value
    ;X := "VALUE: " . Element.GetCurrentPatternAs("Value")
    ;X := "VALUE: " . Element.GetCurrentPattern("Value").Value
    ;X := "VALUE: " . Element.Name . Element.name
    X := "VALUE: " . Element.ControlType . ", NAME=" . Element.Name . ", BR=" . Element.CurrentBoundingRectangle
    MsgBox, % X


ExitApp
Any help appreciated :-)

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 13 Oct 2022, 03:51

@dbareis, yeah that is some weird behaviour. I know that the top-down approach (eg getting children with FindAll or TreeWalker) and bottom-up approach (getting parent with TreeWalker) will give different results, and that is also why UIAViewer is showing different trees at times. The first tree in UIAViewer is generated with a bottom-up approach, but the full tree is made with top-down. Anyway, for some reason the birth date seems to be inaccessible from within a bottom-up approach (probably another bug in UIAutomation or Chrome, since the same problems occur with Acc too), but it is accessible with top-down:

Code: Select all

#Include <UIA_Interface>

wExe := "ahk_exe chrome.exe"
UIA := UIA_Interface()
WinActivate, %wExe%
WinWaitActive, %wExe%
chrome := UIA.ElementFromHandle(wExe)
MsgBox, % chrome.FindFirstByNameAndType("BIRTH", "Text").FindByPath("p3.2").Name
; MsgBox, % cUIA.FindFirstByNameAndType("BIRTH", "Text").FindByPath("p3.2").Name ; If using UIA_Chrome

dbareis
Posts: 39
Joined: 04 Feb 2022, 17:57

Re: UIAutomation with a focus on Chrome

Post by dbareis » 13 Oct 2022, 17:36

Descolada wrote:
13 Oct 2022, 03:51
@dbareis, yeah that is some weird behaviour. I know that the top-down approach (eg getting children with FindAll or TreeWalker) and bottom-up approach (getting parent with TreeWalker) will give different results, and that is also why UIAViewer is showing different trees at times. The first tree in UIAViewer is generated with a bottom-up approach, but the full tree is made with top-down. Anyway, for some reason the birth date seems to be inaccessible from within a bottom-up approach (probably another bug in UIAutomation or Chrome, since the same problems occur with Acc too), but it is accessible with top-down:

Code: Select all

#Include <UIA_Interface>

MsgBox, % cUIA.FindFirstByNameAndType("BIRTH", "Text").FindByPath("p3.2").Name ; If using UIA_Chrome
Thanks @Descolada for the quick response, explaining the top-down vs bottom-up issues and the working example. I don't fully understand what that means but I expect that is why the code below fails (your working code commented out), is this documented anywhere?

The birth is one of many items I will grab. Am I better to use your tree walker interface to load everything into memory? If so how do I work out the paths for that, or can the tree be dumped for diagnostics?

Code: Select all

    ;--- Works ---
    ;cUIA.FindFirstByNameAndType("BIRTH", "Text").FindByPath("p3.2").Highlight()
    ;X := "VALUE: " . cUIA.FindFirstByNameAndType("BIRTH", "Text").FindByPath("p3.2").Name

    ;--- Fails ---
    OBJ = cUIA.FindFirstByNameAndType("BIRTH", "Text").FindByPath("p3.2")
    OBJ.Highlight()
    X := "VALUE: " . OBJ.Name
    MsgBox, % X

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 13 Oct 2022, 23:06

@dbareis, why your previously posted code fails is not related to UIA at all. OBJ = cUIA... is using legacy syntax with the =, so OBJ is being assigned the string literal value of "cUIA.FindFirstByNameAndType("BIRTH", "Text").FindByPath("p3.2")", without actually executing that line (try printing out the value of OBJ with MsgBox...). You need to use OBJ := cUIA... instead.

You can figure out the tree by using the DumpAll method, which *should* have the "correct" path in it, and FindByPath should work correctly then (except the pN options for getting parents, that is still bottom-up). You could also use DumpAll to "load" everything into memory and parse it, or you can use FindAll and then loop over the gotten elements and get the Name/Value properties, or use TreeWalker. Speed-wise using the TreeWalker is probably slowest.

sanmaodo
Posts: 45
Joined: 28 Aug 2020, 01:39

Re: UIAutomation with a focus on Chrome

Post by sanmaodo » 15 Oct 2022, 13:04

There are 2 tabs with the same name. How to select the second TAB ?


@Descolada Thank you very much. The script helped me a lot .

User avatar
Shield
Posts: 9
Joined: 07 Dec 2021, 17:55

Re: UIAutomation with a focus on Chrome

Post by Shield » 15 Oct 2022, 13:06

Hi Descolada! I really hope that you can help me. I'm struggling for days now getting access to a custom ListView control which has the "ListViewWndClass". It's part of a fairly old, internal application which I believe has been written in VB6. I need the full content of specific rows, and thanks to jeeswg I was able to do so by means of MSAA's Description property. However, the Description turned out to be faulty: in some cases the text pulled from the last column misses the final character. After digging out several accessibility viewers I finally found UISpy.exe being the only one capable of accessing the individual cells themselves, reporting back the complete content.

So with UIA it's possible. But unfortunately, the UIA_Interface.ahk doesn't seem to support this. Possible reasons seem to be:
  • UISpy identifies the ControlType as DataGrid, specifying the number of columns and rows, while UIA_Interface identifies the ControlType as List, stating a number of entries.
  • UISpy reports the following supported patterns: Selection / Grid / MultipleView / Table, while UIA_Interface names Selection / LegacyIAccessible. I assume this to be the decisive factor.
Also (not sure if this is important) on my Windows 7 system the UIA_Interface() function indicates maxVersion 7, but falls back to Version 1 for whatever reason. That ListViewWndClass-application however runs on Windows 10.

Below I provided a modified version of the Example12_GridPatternGridItemPattern.ahk. It expects a running instance of 'xplorer2 lite', a free file manager which I currently use for testing purposes (not sure If I am allowed to post a link here). The file panes of xplorer2 are also custom controls and seem to be very similar to that ListViewWndClass. Just make sure that the view of the file panes are set to 'Details'.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

;#include <UIA_Interface> ; Uncomment if you have moved UIA_Interface.ahk to your main Lib folder
#include ..\Lib\UIA_Interface.ahk

UIA := UIA_Interface()

IfWinNotExist, ahk_exe xplorer2_lite.exe
	{	MsgBox Please run xplorer2 lite first.
		ExitApp
	}

WinActivate
ControlGet, Handle, Hwnd,, ATL:BrowserListView1
If !Handle
	{	MsgBox File pane not found
		ExitApp
	}


x2El := UIA.ElementFromHandle(handle)

AllPatterns := x2El.GetSupportedPatterns()		; UIA_Interface: Selection LegacyIAccessible / UISpy: Selection Grid MultipleView Table
For Each, Pattern in AllPatterns
	Patterns .= "  " Pattern "`n"

gridPattern := x2El.GetCurrentPatternAs("Grid")

If gridPattern.CurrentRowCount
	{	MsgBox, % "Supported Patterns:`n" Patterns "`nGridPattern found! Properties: "
		. "`n CurrentRowCount: " gridPattern.CurrentRowCount
		. "`n CurrentColumnCount: " gridPattern.CurrentColumnCount
	}
else
	{	MsgBox, % "Supported Patterns of the fIle pane:`n" Patterns "`nNo Grid found! Exiting."
		ExitApp
	}

editEl := gridPattern.GetItem(1,2)

MsgBox, % "Getting grid item from row 2, column 3 (0-based indexing).`n"
		. "`nGot this element: `n`n" editEl.Dump()

gridItemPattern := editEl.GetCurrentPatternAs("GridItem")
MsgBox, % "GridItemPattern properties:`n"
	. "`nCurrentRow: " gridItemPattern.CurrentRow
	. "`nCurrentColumn: " gridItemPattern.CurrentColumn
	. "`nCurrentRowSpan: " gridItemPattern.CurrentRowSpan
	. "`nCurrentColumnSpan: " gridItemPattern.CurrentColumnSpan
	; gridItemPattern.CurrentContainingGrid should return listEl

ExitApp

Esc::
ExitApp
I'm pretty confident that getting this script to work with xplorer2 will solve the ListViewWndClass-issue as well.

In any case, thank you for paying attention! :thumbup:

jsong55
Posts: 219
Joined: 30 Mar 2021, 22:02

Re: UIAutomation with a focus on Chrome

Post by jsong55 » 15 Oct 2022, 20:23

@Descolada cUIA.SelectTab(TabName) is not able to select something that is in the background. E.g. if there are multiple chrome windows. Only when the chrome window with the said tab exists does it work.

I'm using this in conjunction with chrome.ahk to select tab by URL.

Code: Select all

chromeapp:="ahk_exe Chrome.exe"
if !WinActive(chromeapp)
    WinActivate,% chromeapp
cUIA := new UIA_Browser(chromeapp) ; Initialize UIA_Browser, which also initializes UIA_Interface 
page_list:=chrome.GetPageList()
found:=0
for i, pageData in page_list
{

    if instr(pageData.url,"cloud.google.com/text-to-speech")
    {
        cUIA.SelectTab(pageData.title)
        if !WinActive(pageData.title)
            WinActivate,% pageData.title
        found:=1
        return
        
    }
}

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Oct 2022, 11:33

@Shield, I've inspected xplorer Lite with UIAViewer, Accessibility Insights and Inspect.exe, and none of these show GridPattern to be available. I am not sure what UISpy is reporting you, maybe it's using some other methods besides UIA as well? Accessibility Insights and Inspect.exe are both Microsofts' "official" tools, so if those don't show GridPattern, then it simply isn't available.
For xplorer2 Lite I was able to get the information with LegacyIAccessiblePattern: the Name column was under LegacyIAccessible.Name, other columns were under LegacyIAccessible.Value and comma-separated (Extension, Size etc) so you can just parse that with StrSplit. Though I'm not sure if your actual program is accessible this way as well :)

If UIA version falls back to 1, then that is the highest available version (which makes your Windows 7 quite old... perhaps an update is in order?). By default UIA_Interface tries every version from 7->1 until one succeeds.

@jsong55, UIA often cannot access windows that aren't at least partially directly visible, and even calling new UIA_Browser("...") might hang until the window is visible. But you can get a list of all Chrome windows and then loop through their ahk_id's, activating the window and then accessing it with UIA or UIA_Browser. Btw, is it not possible to use Chrome.ahk to select the tab?

@sanmaodo, so far there wasn't a good method to do that besides cUIA.GetAllTabs(), then loop through the elements until you find the second occurence. I added a new method cUIA.GetTabs(searchPhrase, matchMode=3, caseSensitive=True) which you can use to get all tabs with the same name and then get the second one with [2] like for example cUIA.GetTabs("Google", 2, False)[2]. You can do the same without updating UIA_Browser with cUIA.TabBarElement.FindAllByNameAndType(searchPhrase, "TabItem",, matchMode, caseSensitive).

User avatar
Shield
Posts: 9
Joined: 07 Dec 2021, 17:55

Re: UIAutomation with a focus on Chrome

Post by Shield » 16 Oct 2022, 17:24

@Descolada: Thank you for taking a look into this matter. Unfortunately I seem to be unable to reproduce your findings. I do get the Name, but the Value shows no comma-separated (or any other) content. What am I doing wrong here?

Code: Select all

 
#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include ..\Lib\UIA_Interface.ahk

UIA := UIA_Interface()

Loop {

	MouseGetPos,,,Window
	WinGet,PtcName,ProcessName,ahk_id %Window%
	If PtcName != xplorer2_lite.exe
		{	Tooltip Please point at x2 Lite file pane`n(Esc to Exit)
			Sleep 500
			Continue
		}

	x2El := UIA.SmallestElementFromPoint()

	AllPatterns := x2El.GetSupportedPatterns()		; UIA_Interface: Selection LegacyIAccessible / UISpy: Selection Grid MultipleView Table
	Patterns =
	For each, Pattern in AllPatterns
		Patterns .= "  " Pattern "`n"

	LegacyIAccessiblePattern := x2El.GetCurrentPatternAs("LegacyIAccessible")
	TheName := LegacyIAccessiblePattern.Name
	TheValue := LegacyIAccessiblePattern.Value
	TheDescription := LegacyIAccessiblePattern.Description

	Tooltip % "Supported Patterns:`n" Patterns "`n"
			. "`nName: " TheName
			. "`nValue: " TheValue
			. "`nDescription: " TheDescription
	Sleep 100
}

Esc::
ExitApp

There's that semicolon-separated Description, but as mentioned, within the actual application it sometimes misses the final character,

Regarding Inspect vs USpy: Actually both of them are offical MS tools. Here at https://stackoverflow.com/questions/40496048/ somebody shed some light on the differences - which are a bit over my head. As far as I understand it, both use UIAutomationCore.dll, but in UISpy it's included and outdated. Assuming that this outdated version of UIAutomationCore.dll provided way more functionality than the current one - which I doubt - would it be possible to "link" an external, old version of this dll to the UIA_Interface.ahk?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 16 Oct 2022, 23:20

@Shield, sorry, I meant LegacyIAccessible.Description (not Value).

I looked a bit into UISpy, which seems to be a deprecated .NET version of Inspect. But it does seem to be able to access more information that Inspect. I didn't use UISpy (had some difficulty running it), but instead used FlaUI inspect tool: in managed UIA3 mode it showed pretty much the same stuff as Inspect, but in unmanaged UIA2 mode it showed the Grid pattern. Why this is and how to access the unmanaged UIA2 I don't know, but when I get some time I'll try to look into it. Also you could try asking for help in the general section about how to access the unmanaged UIA2 version - perhaps somebody has encountered this problem before.

sanmaodo
Posts: 45
Joined: 28 Aug 2020, 01:39

Re: UIAutomation with a focus on Chrome

Post by sanmaodo » 17 Oct 2022, 04:16

@Descolada thank you very much! :D
Following your instructions, I did it, is there any room for improvement in the code please?

Example: Select the last tab with the same name.

Code: Select all

#include <UIA_Interface>
#include <UIA_Browser>

cUIA := new UIA_Browser("ahk_exe msedge.exe")
n := cUIA.GetTabs(" | AutoHotkey", 2, False).Count()
if n
cUIA.TabBarElement.FindAllByNameAndType(" | AutoHotkey", "TabItem",, 2)[n].Click()

ExitApp

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 17 Oct 2022, 07:30

@sanmaodo, you can also just do

Code: Select all

cUIA := new UIA_Browser("ahk_exe msedge.exe")
cUIA.GetTabs(" | AutoHotkey", 2, False)[2].Click() ; will click the second tab
(tabs := cUIA.GetTabs(" | AutoHotkey", 2, False))[tabs.Count()].Click() ; clicks the last tab
Reusing the output from GetTabs makes the code twice as fast, since your code is searching for all the tabs twice.

sanmaodo
Posts: 45
Joined: 28 Aug 2020, 01:39

Re: UIAutomation with a focus on Chrome

Post by sanmaodo » 17 Oct 2022, 08:17

@Descolada Thanks! The script works perfectly now. :D

Post Reply

Return to “Scripts and Functions (v1)”