UIAutomation with a focus on Chrome

[Descolada]

@malcev, interesting, now it doesn't activate any window at all, regardless of the AutoSetFocus property. I'm guessing a Windows update in the last year has changed something, because it used to auto-activate every window it was called on. Chromium has gotten a lot of UIAutomation improvements as well in the meanwhile. I'll play around with it a bit more when I get time, and see if they have fixed the reliability issue as well.
Although since LegacyIAccessible is a legacy pattern, I'm not sure how wise it would be to continue using it...

[songdg]

"Continuation section too long." What's the problem when I include the UIA_Interface library?

[Descolada]

@songdg, not exactly sure what the problem is.
1) Try downloading it again from GitHub by pressing the green Code button -> Download ZIP
2) Make sure you are running a relatively recent AHK v1 version (preferably 1.1.36.02)

[songdg]

@songdg, not exactly sure what the problem is.
1) Try downloading it again from GitHub by pressing the green Code button -> Download ZIP
2) Make sure you are running a relatively recent AHK v1 version (preferably 1.1.36.02)

Thanks for answer my question, now I can use UIAutomation. I found your liberary can select sub_menu item of context menu from the forumviewtopic.php?f=76&t=108384&p=484315&hilit=context+menu+submenu#p484315, but when I try to create a new folder on the desktop it doesn't work.

Code: Select all

Click Right
Sleep, 2000
WinWait, ahk_class #32768
WinGet, hwnd, ID, ahk_class #32768
contextmenu := UIA.ElementFromHandle(hwnd)
item := "New"
subitem := "Folder"
subMenu := contextmenu.FindFirstByNameAndType(item, "menuitem")
expandMenu  := subMenu.GetCurrentPatternAs("ExpandCollapse")
expandMenu.Expand()
subMenu.FindFirstByNameAndType(subitem, "menuitem").click()
Winclose

[Descolada]

@songdg, rather unfortunately every new context menu is its own window, so you need to wait for it and get the handle individually.

F1 is your code with a slight modification to get the correct window element for the second context menu. F2 is similar, but I added some modifications of my own: WaitElementExist usually helps a bit with timing issues, and accessing patterns can be done without GetCurrentPatternAs.

Code: Select all

#Requires AutoHotkey v1.1
#SingleInstance Force
#Persistent
#include <UIA_Interface>

UIA := UIA_Interface()
return

F1::
    Click Right
    WinWait, ahk_class #32768
    WinGet, hwnd, ID, ahk_class #32768
    contextmenu := UIA.ElementFromHandle(hwnd)
    item := "New"
    subitem := "Folder"
    subMenu := contextmenu.FindFirstByNameAndType(item, "menuitem")
    expandMenu := subMenu.GetCurrentPatternAs("ExpandCollapse")
    expandMenu.Expand()
    WinWait, ahk_class #32768,,, ahk_id %hwnd%
    WinGet, hwnd, ID, ahk_class #32768,, ahk_id %hwnd%
    UIA.ElementFromHandle(hwnd).FindFirstByNameAndType(subitem, "menuitem").click()
    return

F2::
    Click Right
    WinWait, ahk_class #32768
    contextmenu := UIA.ElementFromHandle(hwnd := WinExist("ahk_class #32768"))
    item := "New"
    subitem := "Folder"
    subMenu := contextmenu.WaitElementExistByNameAndType(item, "menuitem")
    expandMenu := subMenu.ExpandCollapsePattern.Expand()
    WinWait, ahk_class #32768,,, ahk_id %hwnd%
    hwnd := WinExist("ahk_class #32768",, "ahk_id " hwnd)
    UIA.ElementFromHandle(hwnd).WaitElementExistByNameAndType(subitem, "menuitem").click()
    return

[songdg]

@songdg, rather unfortunately every new context menu is its own window, so you need to wait for it and get the handle individually.

F1 is your code with a slight modification to get the correct window element for the second context menu. F2 is similar, but I added some modifications of my own: WaitElementExist usually helps a bit with timing issues, and accessing patterns can be done without GetCurrentPatternAs.

Code: Select all

#Requires AutoHotkey v1.1
#SingleInstance Force
#Persistent
#include <UIA_Interface>

UIA := UIA_Interface()
return

F1::
    Click Right
    WinWait, ahk_class #32768
    WinGet, hwnd, ID, ahk_class #32768
    contextmenu := UIA.ElementFromHandle(hwnd)
    item := "New"
    subitem := "Folder"
    subMenu := contextmenu.FindFirstByNameAndType(item, "menuitem")
    expandMenu := subMenu.GetCurrentPatternAs("ExpandCollapse")
    expandMenu.Expand()
    WinWait, ahk_class #32768,,, ahk_id %hwnd%
    WinGet, hwnd, ID, ahk_class #32768,, ahk_id %hwnd%
    UIA.ElementFromHandle(hwnd).FindFirstByNameAndType(subitem, "menuitem").click()
    return

F2::
    Click Right
    WinWait, ahk_class #32768
    contextmenu := UIA.ElementFromHandle(hwnd := WinExist("ahk_class #32768"))
    item := "New"
    subitem := "Folder"
    subMenu := contextmenu.WaitElementExistByNameAndType(item, "menuitem")
    expandMenu := subMenu.ExpandCollapsePattern.Expand()
    WinWait, ahk_class #32768,,, ahk_id %hwnd%
    hwnd := WinExist("ahk_class #32768",, "ahk_id " hwnd)
    UIA.ElementFromHandle(hwnd).WaitElementExistByNameAndType(subitem, "menuitem").click()
    return

Thanks, much appreciated I should further explore what else this amazing tool can do.

[d_romeo]

Is it possible with this library to call context menu functions? For example to add shortcut keys to the context menu of explorer or better still for some software, such as 7zip, and avoid doing "RButton>compress and send by email..." every time Maybe without necessarily opening the context menu if possible.

[Descolada]

@d_romeo, with this library you can only navigate the context menu, not bypass it completely. For that you would need task-specific commands, such as for compressing files and then call a command to send it...

[d_romeo]

Ok, well, it's not a problem if you see for a while. What I'm looking for is a more refined method of calling specific commands from the menu, not using:

Code: Select all

Sleep, 300
        Send, {AppsKey}
        Sleep, 20
        WinSet, Transparent, 0, ahk_class #32768
        Sleep, 20
        Send, {Down 6}
        Sleep, 20
        Send, {Enter}
        Sleep, 20
        Send, {Right 1}
        Sleep, 20
        Send, {Enter}

Can this library automate similar behavior?

[Descolada]

@d_romeo, see a few posts up for my reply to user songdg, where we discussed exactly what you are asking (automating a context menu, in his case the one which appears after right-clicking the Desktop).

[jekko1976]

in my program, when I click on the element of my interest the only UIA parameter that changes is this (seen with accessibility insights)

LegacyIAccessiblePattern.State 1048576

to

LegacyIAccessiblePattern.State 1048580


How could I ask to UIA to change this property?

Thank you

[Descolada]

@jekko1976, LegacyIAccessiblePattern.State is a read-only property which you can't change programatically. If you need to change it then you need a click the element of interest (perhaps with LegacyIAccessiblePattern.DoDefaultAction()?), or do some other action that will change the state (toggling etc).

[jekko1976]

@Descolada
Unfortunately the element is an EDIT element, but never change for any value that you insert.
DoDefaultAction does nothing.
I will do like I always did. Searching for the bounding area and clicking in the middle of it.
I Leave herewith the report of Accessibility Insights if you have some different idea.
Thank you

Properties:
AcceleratorKey Property does not exist
AccessKey Property does not exist
AutomationId Property does not exist
BoundingRectangle [l=50,t=373,r=324,b=395]
ControlType Edit(50004)
Culture 0
FillType 0
HasKeyboardFocus True
HeadingLevel HeadingLevel_None (80050)
HelpText Property does not exist
IsContentElement True
IsControlElement True
IsDataValidForForm False
IsDialog False
IsEnabled True
IsKeyboardFocusable True
IsOffscreen False
IsPassword False
IsPeripheral False
IsRequiredForForm False
LegacyIAccessiblePattern.ChildId 1
LegacyIAccessiblePattern.Name Row 6, Column 1
LegacyIAccessiblePattern.Role ROLE_SYSTEM_TEXT(42)
LegacyIAccessiblePattern.State 1048580
LegacyIAccessiblePattern.Value Ritenute e Contributo
LiveSetting 0
LocalizedControlType Modifica
Name Row 6, Column 1
NativeWindowHandle 0
OptimizeForVisualContent False
Orientation None(0)
ProcessId 18228
ProviderDescription [pid:18228,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:UIAutomationCore.dll)]
ValuePattern.IsReadOnly False
ValuePattern.Value Ritenute e Contributo
VisualEffects 0

Available Patterns:
ValuePattern
- IsReadOnly: False
- Value: Ritenute e Contributo
LegacyIAccessiblePattern
- ChildId: 1
- DefaultAction:
- Description:
- Help:
- KeyboardShorcut:
- Name: Row 6, Column 1
- Role: ROLE_SYSTEM_TEXT(42)
- State: 1048580
- Value: Ritenute e Contributo

[Descolada]

@jekko1976, yeah, an edit element isn't "clickable" in the traditional sense. You can try Element.SetFocus(), but otherwise I don't have better ideas than Click("left") or ControlClick() to focus the element.

[emp00]

I'm trying to automate Google Lens on chrome using UIA_Browser.ahk, more specifically "image translation":

• Step 1: Starting SnippingTool.exe and let user select a screenshot
• Step 2: Starting chrome.exe, open the Google Lens page and paste the screenshot with CTRL-V

until here everything's working fine, but this final Step 3 does not work...

• Step 3: Wait for the upload to finish and then click the "Translate" button at the bottom of the Google Lens window

Using "cUIA.WaitElementExistByName("Switch to Translate mode").Click()" gives this error:
Error: This class does not contain method "WaitElementExistByName"

Can you help my why and how to succeed in automating this last click?

Here's my complete AHK code - many thanks for looking into this!

Code: Select all

#include <UIA> 
#include <UIA_Browser>

RunWait "SnippingTool.exe /clip"
; This runs the good old windows snipping tool starting in clipping mode; RunWait will wait until the program finishes before continuing.

Run "GoogleChromePortable.exe https:\lens.google.com\search?p=" 
; Startup Chrome with new Google Lens tab and wait until it's ready
WinWaitActive "ahk_exe chrome.exe"
; Initialize UIA_Browser, use Last Found Window (returned by WinWaitActive)
cUIA := UIA_Browser()
cUIA.WaitPageLoad("Google Lens", 1000)
; Wait until the Google Lens pages has fully loaded
ControlClick "x100 y100", "ahk_exe chrome.exe",,,, "Pos"
; Click left mouse button at coordinates relative to active Chrome window, this is important -> before sending CTRL-V !
Send "^v" 
Sleep 500
; Paste the snipped image from clipboard to the Google Lens window by sending CTRL-V and wait 500ms as precaution if WaitPageLoad does not work reliably
cUIA.WaitPageLoad("Google Lens", 1000)
; Wait again until Google Lens upload has finished and the new pages has fully loaded, not sure if this really does anything
cUIA.WaitElementExistByName("Switch to Translate mode").Click()
; THIS SHOULD wait for the Translate button to appear and then click it.... But it throws an error :-(
MsgBox "Now were here and should be done!"

ExitApp

Here's the UIAViewer output showing the Google Lens browser window with the Translate-button (marked yellow). I concluded the button name must be "Switch to Translate mode" and thus I'm trying to use the above WaitElementExistByName.Click() method. But somehow it does not work.

[Descolada]

@emp00, it appears that you are using AHK v2 code and UIA-v2? UIA-v2 and UIA_Interface (which is a AHK v1 library) have some differences, in this case that all UIA_Interface WaitElementExist methods have been replaced by WaitElement. So in UIA-v2 it would be cUIA.WaitElement({Name:"Switch to Translate mode"}).Click(). The condition syntax is explained in the UIA-v2 wiki.
Also I recommend posting v2 code in the v2 section (perhaps the UIA v2 main thread?), otherwise readers might get confused

[leosouza85]

Hi @Descolada !

I have an Idea:

Why not make the functions after and before? that will search the first object after or before a specified one

Example:
cUIA.WaitElementExistByName("Submit").after("Full Editor & Preview").click()
cUIA.WaitElementExistByName("Submit").before("Post Reply").click()

Its noob friendly and helps to better identify something when there are no names, duplicate names, and etc.

[Loop]

Hi, @Descolada,
The Windows Editor is now a UWP app and offers tabs. My problem is that the Window Spy tool doesn't display the window title of the current tab. However, I can access the window titles using your UIA class. But I need to get the title of the current tab when multiple tabs are open. Do you have any ideas on how to get only the title of the active tab?

Code: Select all

#NoEnv
#Singleinstance Force

 #Include <UIA_Interface>
UIA := UIA_Interface()
el := WinExist("Editor ahk_exe Notepad.exe ahk_class Notepad")
el := UIA.ElementFromHandle(el)

Tabs := el.FindAllBy("ControlType=TabItem AND ClassName=ListViewItem")


For _,Tab in Tabs
   MsgBox,% Tab.Name

[Descolada]

@Loop, I can't test this at the moment (running Windows 10), but usually tabs have "SelectionItemIsSelected" property. This means you can either do selectedTab := el.FindFirstBy("ControlType=TabItem AND ClassName=ListViewItem AND SelectionItemIsSelected=1"), or loop over the tabs and test if Tab.SelectionItemIsSelected == 1.

[Loop]

Thank you, that worked!

Tab.SelectionItemIsSelected == 1