UIAutomation with a focus on Chrome

[Descolada]

@mora145, I didn't find any problems with it, just uploaded the wrong file I think. Corrected it.

Also if anybody else has the same problem with SmallestElementFromPoint as user Skrell, please PM me I am unable to reproduce the issue, so the more people testing it out the better

[mora145]

@mora145, I didn't find any problems with it, just uploaded the wrong file I think. Corrected it.

Also if anybody else has the same problem with SmallestElementFromPoint as user Skrell, please PM me I am unable to reproduce the issue, so the more people testing it out the better

Excellent. I can see it in the Git
Thank you. You will become famous with your library.

[mora145]

Code: Select all

SetTitleMatchMode, 2
#Include <UIA_Interface>

UIA := UIA_Interface()

chromeId := WinExist("ahk_exe Chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%
chrome := UIA.ElementFromHandle(chromeId, True)
bulletText := chrome.FindFirstBy("Name=NO ME GUSTA OR Name=DISLIKE")
Sleep, 1000
dateUpload := bulletText.FindByPath("-2", UIA.CreateCondition("ControlType", "Text"))
MsgBox, % dateUpload.GetCurrentPropertyValueEx("Name")

For those who are interested, this is how my little test has turned out. You must have Youtube open and active in Chrome.

Screenshot_7.jpg (160.01 KiB) Viewed 4212 times

[leosouza85]

@Descolada Thank you again for your support... after days of debugging I found this:

In my home the code works ok
At my work, the findfirst or the waitelement only find the first branch of the tree, but when I go with find by path I can find the other branches..

Setup at home:
Windows 10
Last version of Chrome
Last version of AHK 1.1
Last version of the library from your github

Setup at work:
Windows 10
Last version of Chrome
Last version of AHK 1.1
Last version of the library from your github

Both at home and at work UIAViewer and Treeinspector works identically... only the find and wait functions work different...

No idea how to solve... I've already checked the encoding so many times...

But maybe is that what is givving some different results on skype automation on similar setups... something that we dont identify yet, but causes in some setups to only search the first branch....

[Descolada]

@leosouza85, if the code between the computers is the same, then either
1) One of your Windows 10 builds is too old (FindFirst can search partial matches only after Windows 10 October 2018 Update (version 1809)). I will add a note about this in my main post at some point...
2) Microsofts UIAutomation could have a bug in FindFirst
3) My library might have some kind of bug that has eluded me.

I recommend trying bypassing the wrapper functions and use FindFirst directly:

Code: Select all

MsgBox, Windows version: %A_OSVersion%
UIA := UIA_Interface()
condition := UIA.CreatePropertyConditionEx(30005, "Anexar",,0x2) ; 30005=UIA_NamePropertyId. Flag 0x2 == PropertyConditionFlags_MatchSubstring (works on Windows 10 October 2018 Update (version 1809) and newer)
MsgBox, % UIA.ElementFromHandle(WinExist("ahk_exe chrome.exe"), True).FindFirst(condition, 0x4).Dump()

If it turns out you have an older Windows 10 build, then you either need to look for an exact match, or use FindAll and loop over the elements, checking with InStr whether element.CurrentName contains your search phrase.

[Joe Glines]

We had another epic call with @Descolada
Descolada shows how to work with patterns and a ton of other amazing tips! Here's a link to the UIA Playlist
Thank you for your time Descolada! You've done an amazing job on this class!

[leosouza85]

@Descolada Hi!!

After some debugging of that problem with the button name I can click this way:

Code: Select all

buttons := cUIA.FindAllByType("Button")
counter := buttons.length()
Loop, %counter%
{
msg := buttons[A_Index].Dump()
if msg contains Anexar
{
buttons[A_Index].Click()
break
}
}

[Mod edit: [code][/code] tags added.]

But not that way:

Code: Select all

cUIA.FindFirstByName("Anexar",,2,False).Click()

So, I think this problem only affects names with illegal characters...
The dump function reads the name, but the find functions do not work.

But at home I can click by name normally... I think if there is a way to make illegal character proof, it will improve the library... thank you so much for your work!

I can't use a fixed index on this website because it varies according to the number of files attached.

[Skrell]

If I parse the whole Window, are there any tricks that would allow me to more quickly find the element I'm interested in? Are all nodes searched every time sequentially? Perhaps by using the Create__Condition() functions?
Also, what exactly is the definition of a propertyId? I assume a propertyId can be ANY of the 5 digit values (10,000->90,000) specified in UIA_Constants.ahk?

[Descolada]

@Skrell, I am not sure of this, but I think FindFirst methods visit all the nodes recursively and sequentially. Eg in a tree of

Code: Select all

1
1.1
1.1.1
1.2

FindFirst would visit the nodes from top to bottom.

If you want speed increases:
1) Use FindFirst instead of FindAll, since FindAll will always have to go through the whole UIA tree, but FindFirst can finish early
2) Use smaller scope. The default is TreeScope_Descendants=0x4, but in some cases this might be very slow. For example if the first branch of the tree is huge but the second branch very small, and your element of interest is in the second branch, it might be better to first get that second branch with FindFirst TreeScope_Children=0x2 or TreeWalker, and then use FindFirst on that.
3) Sometimes you might have uses for the newer FindFirstWithOptions method, where you can specify the search direction with TreeTraversalOptions (so you can pretty much create a FindLast method).
4) If you plan on running multiple searches on the same tree or access properties multiple times, you might get a speed improvement by using caching. I haven't discussed this topic yet though, but you could probably piece it together from Google searches.
Using Create__Condition() instead of FindFirstBy(expression) will probably not give you a big speed increase, since FindFirstBy will use those same exact condition creation methods

PropertyIds are the digits in the 30000's, which are also listed in the enumerations in class UIA_Enum under UIA_PropertyIds. A PropertyId will end with "PropertyId". These can be used for example with the GetCurrentPropertyValue method.
10000's for example are PatternIds, which can be used with GetCurrentPattern method.

After I've pushed out the big UIA_Interface update that is coming soon, I'll start working on a more comprehensive tutorial

[mora145]

@Joe Glines Incredible! Thank you very much

@Descolada that last message is super interesting. Thank you.

[Descolada]

Updated UIA_Interface, UIA_Browser, example files, UIAViewer and UIATreeInspector.

UIA_Interface updates:
1) UIA_Interface.CreateCondition now supports expressions (parenthesis are allowed). Example: UIA.CreateCondition("(type=button && name='SomeName' flags=2) or type=menuitem") would create a condition that matches either a button with "SomeName" anywhere in its name, or a menuitem.
2) UIA_Element.FindFirstBy and FindAllBy support parenthesis if matchMode is 3 or 2 (and if matching substrings is supported in the currently running Windows: at least Windows 10 October 2018 Update, version 1809). MatchMode Regex and 1 do not support parenthesis, but now the first "string" type property will have these matchmodes applied (eg in RegEx mode "Type=Button and AutomationId='.*test?'" would search for button elements with RegexMatch for ".*test?"
3) UIA_Element.Highlight() highlights the element borders (for debugging purposes).
4) Properties have been rewritten, now Current... properties can be accessed without the "Current" part (eg UIA_Element.Value, UIA_Element.BoundingRectangle). For some properties, setting values is supported (UIA_Element.Value := "test").
4) Patterns can be accessed like properties: UIA_Element.ValuePattern gets UIA_ValuePattern (note that the highest version of the pattern won't be selected).
5) UIA_Element.Click() and UIA_Element.ControlClick() first or second arguments can be a "sleep after" value.

UIA_Browser updates: Added Navigate method for easier navigation. Fixed bug in JSSetTitle.
Example files have been updated to run "out of the box" without changing #include.
UIAViewer and UIATreeInspector have been updated with multiple bug-fixes.

[SundayProgrammer]

very nice

[mora145]

Hi @Descolada

I am trying to do this, to find the file name of the image, using Microsoft Office 2010 (Picture manager).
It quickly finds the 'Next' button, but when I use the Path to search for the next item (Image), the program crashes for overload. It doesn't find anything either.
In general I have noticed that FindByPath takes a long time to find any item using the "+n".

Can you help me?

Code: Select all

#SingleInstance, Force
SendMode Input
SetWorkingDir, %A_ScriptDir%

#Include <UIA_Interface>

UIA := UIA_Interface()
WinActivate, Microsoft Office 2010
cEl := UIA.ElementFromHandle(WinExist("Microsoft Office 2010"))

^<::
Begins:
next := cEl.FindFirstBy("Name=Siguiente")
MsgBox, % next.FindByPath("+1", UIA.CreateCondition("ControlType", "Image")).Dump()

vch.png (648.23 KiB) Viewed 3141 times

[Descolada]

@mora145, you code is fine, but there is something broken with Microsofts implementation of the TreeWalker in that program (inspect.exe treewalking doesn't work properly either). I'm filing a bug report, but it probably won't be fixed very soon :/
A workaround to get the image element:

Code: Select all

next := cEl.FindFirstBy("NOT Name= AND ControlType=Image")
MsgBox, % next.Dump()

[mora145]

@mora145, you code is fine, but there is something broken with Microsofts implementation of the TreeWalker in that program (inspect.exe treewalking doesn't work properly either). I'm filing a bug report, but it probably won't be fixed very soon :/
A workaround to get the image element:

Code: Select all

next := cEl.FindFirstBy("NOT Name= AND ControlType=Image")
MsgBox, % next.Dump()

It has also happened to me with Chrome, on pages with many elements. I thought TreeWalker consumed a lot of resources very fast and crashed the program.
Your implementation is excellent. It works much faster. I guess it's not always a good idea to walk through the Element Tree.

[Youhoney]

Hi all! It's my first post here. Many thanks to Descolada and all others for this highly useful UIA library! I'm currently trying to automate a work related program using UIA. Some tasks I have automated successfully, but I've come across a problem I can't make sense of.

After correctly identifying a button and clicking it using .click(), the script waits for 10 seconds or so and gives an error message. The click script seems to be using Invoke(). The same error happens if I identify the button using UIAViewer and manually Invoke the element. The error message is usually on UIA_Interface.ahk line 1174 or 147, it seems. Sometimes the error contains "UIA_E_ELEMENTNOTAVAILABLE" and "Specifically: Invoke (UIA_InvokePattern)". Some of the dysfunctioning buttons contain DoDefaultAction in addition to Invoke pattern. I get an error regardless of which I am using.

This is puzzling, since the script does in fact find and is able to click the element before the error. Also the error doesn't replicate for all buttons in the software. Any ideas how I could solve this problem or is this perhaps a bug?

Attached are some screenshots and my code.

Code: Select all

#NoEnv
#SingleInstance force
#include <UIA_Interface>

UIA := UIA_Interface()

q::
Asiakastuote_lisäys:
WinActivate, Käynnin yhteenveto
Sleep, 100
;asiakastuote := % A_Args[2]
KäynninYhteenveto := UIA.ElementFromHandle(WinExist("Käynnin yhteenveto"))
Kontaktitiedot := KäynninYhteenveto.FindFirstBy("ClassName=ContactDataControl")
Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink").Click()
Return

error4.png (150.46 KiB) Viewed 2765 times

error3.png (49.38 KiB) Viewed 2765 times

error2.png (140.59 KiB) Viewed 2765 times

[mora145]

Spoiler

@Youhoney

I think it is related to that line, don't trust me, I'm new

Code: Select all

Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink").Click()

This should be:

Code: Select all

Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicing AND ControlType=Hyperlink").Click()

My mistake, I just saw that using an automazation ID

[Descolada]

@Youhoney, quite interesting. If Invoke was called successfully, then the Click function should return immediately. This might mean that the implementation for UIA in that program might be the cause of this? Especially if other programs don't have this problem.
Could you try Accessibility Insights or inspect.exe tools as well - do they cause the same error?

As a workaround you could use Click("left") or ControlClick()...

[Youhoney]

@Descolada thanks for the swift reply! It might very well be the case that the UIA implementation is the problem. I haven't yet tried other programs. However, some buttons in this software work fine. I'm unfortunately using AHK on a work computer and I don't have admin rights and couldn't therefore install Accessibility Insights. I tried viewtopic.php?t=28866 Inspect.ahk (not sure if this is what you meant, but I didn't find any way to use Invoke through that.

Thanks for the tips, I did get ControlClick working, which hopefully gets the job done reliably enough.

[Descolada]

@Youhoney, inspect.exe is a separate program (not an AHK script), you can download it for example here.

I had another though in the meanwhile: if Invoke clicks your element but then stops for a while and throws an error, perhaps you could try temporarily changing the timeout values:

Code: Select all

saveConnectionTimeout := UIA.ConnectionTimeout, saveTransactionTimeout := UIA.TransactionTimeout ; Save previous values for timeouts
UIA.ConnectionTimeout := 500, UIA.TransactionTimeout := 200 ; Set shorter timeouts (could experiment with even shorter ones, minimum is 50)
try Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink").Click() ; Use "try" to avoid the error being thrown
UIA.ConnectionTimeout := saveConnectionTimeout, UIA.TransactionTimeout := saveTransactionTimeout ; Restore previous timeout values

Glad to head you got it working