UIAutomation with a focus on Chrome

[Descolada]

@TomDonovan, I tried listening for the UIA_SelectionItem_ElementSelectedEventId with Accessibility Insights, but apparently in Outlook 365 it doesn't trigger that event, nor does it trigger FocusChanged event. But if a program/element were to trigger it, then this is how you would listen to it:

Code: Select all

UIA := UIA_Interface()
OutlookMainID := WinExist("ahk_exe outlook.exe")
elementMain := UIA.ElementFromHandle(OutlookMainID)
homeTab := elementMain.FindFirstByNameAndType("Home","TabItem")
handler := UIA_CreateEventHandler("TabItemSelectedEventHandler")
UIA.AddAutomationEventHandler(UIA_Enum.UIA_SelectionItem_ElementSelectedEventId, homeTab,,,handler) ; or if it doesn't work, try using elementMain here
OnExit("Cleanup")
return

Cleanup() {
    UIA_Interface().RemoveAllEventHandlers()
}

TabItemSelectedEventHandler(sender, event) {
    ToolTip, Selected!
}

I tested in File Explorer, where there is a quirk that even if you hook to a specific tab item button, the event is thrown anyway when clicking on any tab item, so you would need to filter out the correct element by checking sender.CurrentName in the event handler...

Alternative options:
1) Add a FocusedChanged event listener (though this didn't work in Outlook 365 either...)
2) Create a ~LButton hotkey and check if the element under the cursor is the Home tab item element.

EDIT: PropertyChangedEvent seems to work:

Code: Select all

#Include <UIA_Interface>

UIA := UIA_Interface()
OutlookMainID := WinExist("ahk_exe outlook.exe")
elementMain := UIA.ElementFromHandle(OutlookMainID)
homeTab := elementMain.FindFirstByNameAndType("Home","TabItem")
handler := UIA_CreateEventHandler("HomeTabSelected", "PropertyChanged")
UIA.AddPropertyChangedEventHandler(homeTab,,,handler, [UIA_Enum.UIA_SelectionItemIsSelectedPropertyId])
OnExit("Cleanup")
return
Cleanup() {
    UIA_Interface().RemoveAllEventHandlers()
}
HomeTabSelected(sender, propertyId, newValue) {
    ToolTip, % "Sender: " sender.Dump() 
        . "`nPropertyId: " propertyId
        . "`nNew value: " newValue
}

Though newValue is reported as "Invalid variant", this bug will be fixed in the next UIA_Interface update.

[leosouza85]

@Descolada

I have an idea for a function on UIAViewer

Press F2 to copy current hovered element name and type to the clipboard (so we dont need to pause and then go on UIAViewer to copy it

Press F3 and it will copy the following text to clipboard:
el.WaitElementExistByNameAndType("%Name%#,%Type%").Click()

Or even accumulate this to a variable, so we can make a sort of not much sophisticated step recorder:
var .= el.WaitElementExistByNameAndType("%Name%#,%Type%").Click()

then F3 again

var .= el.WaitElementExistByNameAndType("%Name%#,%Type%").Click() and etc.

Then press F4:

clipboard := var

[Descolada]

@leosouza85, I added a very basic macro creating tool to UIAViewer. I hope it does what you wanted

[leosouza85]

@leosouza85, I added a very basic macro creating tool to UIAViewer. I hope it does what you wanted

Thank you so much! This will speed up code so much!!!

[wetware05]

Hi.

How to know the coordinates and size of a browser's client window? I don't know if I'm using the right language. I am referring to the window where the web page is seen, if the additions of the scroll bar, the favorites buttons, the address bar, the tabs and the status bar...

I've been looking all over the place, using different functions, but none of them give me this information. I need it to take a screenshot, but all the grabbers I've used take it from the whole browser, not just from the client window (I know it can be cropped afterwards, but I need it for an automatic process. I also know I can crop the image knowing what it looks like in my browser and screen, but I need it for any browser and screen size, so I can share the script. Not just that it works for me).

I've been looking at the UIAutomation information, but I can't find that it provides the coordinates of the different objects or windows (well, maybe in the elementary, in applications).

[Descolada]

@wetware05, you can do it both with and without UIA.
Without:

Code: Select all

browserId := WinExist("ahk_exe chrome.exe")
ControlGetPos, x, y, w, h, Chrome_RenderWidgetHostHWND1, ahk_id %browserId%
WinGetPos, wx, wy, ww, wh, ahk_id %browserId%
MsgBox, % "X: " wx+ww-w-11 " Y: " wy+wh-h-11 " W: " w " H: " h

With UIA:

Code: Select all

#include <UIA_Interface>
browserId := WinExist("ahk_exe chrome.exe")
WinActivate, ahk_id %browserId%
WinWaitActive, ahk_id %browserId%
pos := UIA_Interface().ElementFromHandle("ahk_id " browserId).FindFirstByType("Document").GetCurrentPos("screen")
MsgBox, % "X: " pos.x " Y: " pos.y " W: " pos.w " H: " pos.h

[wetware05]

Thank you very much @Descolada. I was going crazy trying to find such seemingly "basic" information. The case was to know the name of the control, and sometimes it becomes impossible to know something like that. Although from what I see UIA uses a totally different method.

I see in the first method that you take into account the size of the browser and that of the control, to find the result of the position of the control... where does the subtraction of -11 come from?

In the second method I get the following error:

Error at line 46 in #include file Interface.ahk".
Line Text: _ New(p flag=0, version "") Error: Parameters of hotkey functions must be optional.
I interpret that it happens because I have assigned a keyboard shortcut, I will try if it. Yes, it was, it was because I had put a keyboard shortcut, without the shortcut it works fine.

Thanks again.

Reissued.

I have verified that changing the browser, to vivaldi, which is based on chrome, both systems fail: they give the total size of the browser, not the client window. From what I see there isn't a "universal method" and you have to do some research to make it work in every browser... it's frustrating!

(Previously change the corresponding parameter in the line browserId := WinExist("ahk_exe chrome.exe") to vivaldi.exe.)

[leosouza85]

@Descolada

Hi, on some instances of Chrome, independent of the website I'm visiting, UIAViwer only catches the document window, it will not go bellow that. But TreeInspector works fine. At my work this happens always, and at home sometimes. Maybe there is some bug?! I'm using the last version with macro creator, but on older versions it happens too.

[Descolada]

@leosouza85, I think this has happened to me too for a few times, but it has went away after moving the mouse a while and hasn't been reproducable. Since this hasn't happened in other apps, I think this might be a Chrome bug. Are you able to reproduce it?

[leosouza85]

@leosouza85, I think this has happened to me too for a few times, but it has went away after moving the mouse a while and hasn't been reproducable. Since this hasn't happened in other apps, I think this might be a Chrome bug. Are you able to reproduce it?

Yes, right now, with chrome in this right forum, it only shows the document. I moved the mouse a lot to see if it would solve and nothing.... what could I do or send to you to debug it?

[Loop]

Hello, I would like to click on an image that is next to String "Next page". The place is highlighted correctly but right click made on coordinate 0,0 and Dimension.t is 0.
Is my Code wrong?

Code: Select all

SearchText := chrome.FindFirstBy("ControlType=Text AND Name='Next page'")
PicAndTextWalker := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType=Image OR ControlType=Text"))
NextPic := PicAndTextWalker.GetNextSiblingElement(SearchText)
NextPic.HighLight(2000)
Dimension := NextPic.BoundingRectangle
ToolTip, % Dimension.t
Sleep, 3000
ToolTip
NextPic.Click("right")

Thx

[Descolada]

@Loop, I can't see anything wrong with your code. I also tested out a similar code in multiple programs and couldn't reproduce the issue. Could you perhaps share the webpage (in a private message) so I could test it out?
Internally HighLight is using NextPic.CurrentBoundingRectangle with t, l, b, r, so if the highlighting works then your code should as well... Unless the element is being destroyed in the time the highlight is active (for example the webpage is refreshing the layout), in which case all the coordinates will be 0.

[Loop]

Thanks!

A small gif animation appeared, that was the reason, i was able to solve the problem.
I want to get values from a ComboBox, how can I do that?
My idea takes too long time and not reliable

Code: Select all

AllVal := ""
Loop
{
Chrome.FindFirstBy("ControlType=ComboBox AND AutomationId='SalesSelection'").SetFocus()
Send, {Down}
CurrVal := Chrome.FindFirstBy("ControlType=ComboBox AND AutomationId='SalesSelection'").Value
AllVal .= CurrVal "`n"

if InStr(AllVal, CurrVal)
break
}

[Descolada]

@Loop, glad to hear you found the bug

I've usually gotten ComboBox values by first expanding it, then using FindAll to get all list items (don't even have to find by type, because a ComboBox's children are only list items):

Code: Select all

CBEl := Chrome.FindFirstBy("ControlType=ComboBox AND AutomationId='SalesSelection'")
CBEl.ExpandCollapsePattern.Expand()
Sleep, 200 ; Give some time for the list items to show up. Alternatively use WaitElementExist
CBchildren := CBEl.FindAll(UIA.TrueCondition)
for _, child in CBchildren
    MsgBox, % child.Value

Oh, and unfortunately UIA can't see the items without first expanding the ComboBox...

[Loop]

Thank you, works perfectly!

[Charon]

I have been trying to figure out how to connect to an existing chrome window and specific tab. I figured out how to connect to the existing window but how do I get to a specific chrome tab?

Code: Select all

browserExe := "chrome.exe"
WinActivate, ahk_exe %browserExe%
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe)
Clipboard=
Clipboard := cUIA.GetCurrentDocumentElement().DumpAll() ; Get the current document element (this excludes the URL bar, navigation buttons etc) and dump all the information about it in 
ExitApp

[Descolada]

@Charon, welcome to the AHK forums!
UIA can usually only "see" visible elements on the screen, so to get a specific Chrome tab you first need to select it (for example with the cUIA.SelectTab method).
To get information from a tab without activating it first, perhaps look into Chrome.ahk or Rufaydium libraries.

[Loop]

Hello, today tried to expand all the Replies under a youtube video, only the first works, my treewalker does not work.
Thanks in advance

Code: Select all

FirstComment := el.FindFirstBy("ControlType=Text AND Name='\d+ REPLIES'",,"RegEx")
FirstComment.HighLight(2000)
CommentWalker := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType=Text AND Name='\d+ REPLIES'",,"RegEx"))
NextComment := CommentWalker.GetNextSiblingElement(FirstComment).HighLight(2000)

[Descolada]

@Loop, the treewalker doesn't work, because UIA.CreateCondition doesn't support RegEx (because Microsoft hasn't implemented it). From UIA.CreateCondition documentation for the flags argument: Flags: 0=no flags; 1=ignore case (case insensitive); 2=match substring; 3=ignore case and match substring. No Regex, and no matching by start of string (only by substring).

For FindFirstBy I have implemented a RegEx option, which in your case will in the backend call FindAllByType("Text") and then filter out the first match using RegexMatch. I have done the same for FindAllBy, so perhaps you could use

Code: Select all

AllComments := el.FindAllBy("ControlType=Text AND Name='\d+ REPLIES'",,"RegEx")
for _, comment in AllComments
    comment.Highlight(2000)

[Skrell]

If I call a function like mEl.FindFirstByNameAndType("Minimize", "Button") how can I check the result to know it either found something or didn't?