@mora145, I didn't find any problems with it, just uploaded the wrong file I think. Corrected it.
Also if anybody else has the same problem with SmallestElementFromPoint as user Skrell, please PM me I am unable to reproduce the issue, so the more people testing it out the better
UIAutomation with a focus on Chrome
Re: UIAutomation with a focus on Chrome
Excellent. I can see it in the GitDescolada wrote: ↑11 Jul 2022, 23:13@mora145, I didn't find any problems with it, just uploaded the wrong file I think. Corrected it.
Also if anybody else has the same problem with SmallestElementFromPoint as user Skrell, please PM me I am unable to reproduce the issue, so the more people testing it out the better
Thank you. You will become famous with your library.
Re: UIAutomation with a focus on Chrome
Code: Select all
SetTitleMatchMode, 2
#Include <UIA_Interface>
UIA := UIA_Interface()
chromeId := WinExist("ahk_exe Chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%
chrome := UIA.ElementFromHandle(chromeId, True)
bulletText := chrome.FindFirstBy("Name=NO ME GUSTA OR Name=DISLIKE")
Sleep, 1000
dateUpload := bulletText.FindByPath("-2", UIA.CreateCondition("ControlType", "Text"))
MsgBox, % dateUpload.GetCurrentPropertyValueEx("Name")
-
- Posts: 90
- Joined: 22 Jul 2016, 16:28
Re: UIAutomation with a focus on Chrome
@Descolada Thank you again for your support... after days of debugging I found this:
In my home the code works ok
At my work, the findfirst or the waitelement only find the first branch of the tree, but when I go with find by path I can find the other branches..
Setup at home:
Windows 10
Last version of Chrome
Last version of AHK 1.1
Last version of the library from your github
Setup at work:
Windows 10
Last version of Chrome
Last version of AHK 1.1
Last version of the library from your github
Both at home and at work UIAViewer and Treeinspector works identically... only the find and wait functions work different...
No idea how to solve... I've already checked the encoding so many times...
But maybe is that what is givving some different results on skype automation on similar setups... something that we dont identify yet, but causes in some setups to only search the first branch....
In my home the code works ok
At my work, the findfirst or the waitelement only find the first branch of the tree, but when I go with find by path I can find the other branches..
Setup at home:
Windows 10
Last version of Chrome
Last version of AHK 1.1
Last version of the library from your github
Setup at work:
Windows 10
Last version of Chrome
Last version of AHK 1.1
Last version of the library from your github
Both at home and at work UIAViewer and Treeinspector works identically... only the find and wait functions work different...
No idea how to solve... I've already checked the encoding so many times...
But maybe is that what is givving some different results on skype automation on similar setups... something that we dont identify yet, but causes in some setups to only search the first branch....
Re: UIAutomation with a focus on Chrome
@leosouza85, if the code between the computers is the same, then either
1) One of your Windows 10 builds is too old (FindFirst can search partial matches only after Windows 10 October 2018 Update (version 1809)). I will add a note about this in my main post at some point...
2) Microsofts UIAutomation could have a bug in FindFirst
3) My library might have some kind of bug that has eluded me.
I recommend trying bypassing the wrapper functions and use FindFirst directly:
If it turns out you have an older Windows 10 build, then you either need to look for an exact match, or use FindAll and loop over the elements, checking with InStr whether element.CurrentName contains your search phrase.
1) One of your Windows 10 builds is too old (FindFirst can search partial matches only after Windows 10 October 2018 Update (version 1809)). I will add a note about this in my main post at some point...
2) Microsofts UIAutomation could have a bug in FindFirst
3) My library might have some kind of bug that has eluded me.
I recommend trying bypassing the wrapper functions and use FindFirst directly:
Code: Select all
MsgBox, Windows version: %A_OSVersion%
UIA := UIA_Interface()
condition := UIA.CreatePropertyConditionEx(30005, "Anexar",,0x2) ; 30005=UIA_NamePropertyId. Flag 0x2 == PropertyConditionFlags_MatchSubstring (works on Windows 10 October 2018 Update (version 1809) and newer)
MsgBox, % UIA.ElementFromHandle(WinExist("ahk_exe chrome.exe"), True).FindFirst(condition, 0x4).Dump()
- Joe Glines
- Posts: 771
- Joined: 30 Sep 2013, 20:49
- Location: Dallas
- Contact:
Re: UIAutomation with a focus on Chrome
We had another epic call with @Descolada
Descolada shows how to work with patterns and a ton of other amazing tips! Here's a link to the UIA Playlist
Thank you for your time Descolada! You've done an amazing job on this class!
Descolada shows how to work with patterns and a ton of other amazing tips! Here's a link to the UIA Playlist
Thank you for your time Descolada! You've done an amazing job on this class!
Sign-up for the HK Newsletter
AHK Tutorials:Web Scraping | | Webservice APIs | AHK and Excel | Chrome | RegEx | Functions
Training: AHK Webinars Courses on AutoHotkey
YouTube
Quick Access Popup, the powerful Windows folders, apps and documents launcher!
AHK Tutorials:Web Scraping | | Webservice APIs | AHK and Excel | Chrome | RegEx | Functions
Training: AHK Webinars Courses on AutoHotkey
YouTube
Quick Access Popup, the powerful Windows folders, apps and documents launcher!
-
- Posts: 90
- Joined: 22 Jul 2016, 16:28
Re: UIAutomation with a focus on Chrome
@Descolada Hi!!
After some debugging of that problem with the button name I can click this way:
[Mod edit: [code][/code] tags added.]
But not that way:
So, I think this problem only affects names with illegal characters...
The dump function reads the name, but the find functions do not work.
But at home I can click by name normally... I think if there is a way to make illegal character proof, it will improve the library... thank you so much for your work!
I can't use a fixed index on this website because it varies according to the number of files attached.
After some debugging of that problem with the button name I can click this way:
Code: Select all
buttons := cUIA.FindAllByType("Button")
counter := buttons.length()
Loop, %counter%
{
msg := buttons[A_Index].Dump()
if msg contains Anexar
{
buttons[A_Index].Click()
break
}
}
But not that way:
Code: Select all
cUIA.FindFirstByName("Anexar",,2,False).Click()
The dump function reads the name, but the find functions do not work.
But at home I can click by name normally... I think if there is a way to make illegal character proof, it will improve the library... thank you so much for your work!
I can't use a fixed index on this website because it varies according to the number of files attached.
Re: UIAutomation with a focus on Chrome
If I parse the whole Window, are there any tricks that would allow me to more quickly find the element I'm interested in? Are all nodes searched every time sequentially? Perhaps by using the Create__Condition() functions?
Also, what exactly is the definition of a propertyId? I assume a propertyId can be ANY of the 5 digit values (10,000->90,000) specified in UIA_Constants.ahk?
Also, what exactly is the definition of a propertyId? I assume a propertyId can be ANY of the 5 digit values (10,000->90,000) specified in UIA_Constants.ahk?
Re: UIAutomation with a focus on Chrome
@Skrell, I am not sure of this, but I think FindFirst methods visit all the nodes recursively and sequentially. Eg in a tree of
FindFirst would visit the nodes from top to bottom.
If you want speed increases:
1) Use FindFirst instead of FindAll, since FindAll will always have to go through the whole UIA tree, but FindFirst can finish early
2) Use smaller scope. The default is TreeScope_Descendants=0x4, but in some cases this might be very slow. For example if the first branch of the tree is huge but the second branch very small, and your element of interest is in the second branch, it might be better to first get that second branch with FindFirst TreeScope_Children=0x2 or TreeWalker, and then use FindFirst on that.
3) Sometimes you might have uses for the newer FindFirstWithOptions method, where you can specify the search direction with TreeTraversalOptions (so you can pretty much create a FindLast method).
4) If you plan on running multiple searches on the same tree or access properties multiple times, you might get a speed improvement by using caching. I haven't discussed this topic yet though, but you could probably piece it together from Google searches.
Using Create__Condition() instead of FindFirstBy(expression) will probably not give you a big speed increase, since FindFirstBy will use those same exact condition creation methods
PropertyIds are the digits in the 30000's, which are also listed in the enumerations in class UIA_Enum under UIA_PropertyIds. A PropertyId will end with "PropertyId". These can be used for example with the GetCurrentPropertyValue method.
10000's for example are PatternIds, which can be used with GetCurrentPattern method.
After I've pushed out the big UIA_Interface update that is coming soon, I'll start working on a more comprehensive tutorial
Code: Select all
1
1.1
1.1.1
1.2
If you want speed increases:
1) Use FindFirst instead of FindAll, since FindAll will always have to go through the whole UIA tree, but FindFirst can finish early
2) Use smaller scope. The default is TreeScope_Descendants=0x4, but in some cases this might be very slow. For example if the first branch of the tree is huge but the second branch very small, and your element of interest is in the second branch, it might be better to first get that second branch with FindFirst TreeScope_Children=0x2 or TreeWalker, and then use FindFirst on that.
3) Sometimes you might have uses for the newer FindFirstWithOptions method, where you can specify the search direction with TreeTraversalOptions (so you can pretty much create a FindLast method).
4) If you plan on running multiple searches on the same tree or access properties multiple times, you might get a speed improvement by using caching. I haven't discussed this topic yet though, but you could probably piece it together from Google searches.
Using Create__Condition() instead of FindFirstBy(expression) will probably not give you a big speed increase, since FindFirstBy will use those same exact condition creation methods
PropertyIds are the digits in the 30000's, which are also listed in the enumerations in class UIA_Enum under UIA_PropertyIds. A PropertyId will end with "PropertyId". These can be used for example with the GetCurrentPropertyValue method.
10000's for example are PatternIds, which can be used with GetCurrentPattern method.
After I've pushed out the big UIA_Interface update that is coming soon, I'll start working on a more comprehensive tutorial
Last edited by Descolada on 15 Jul 2022, 09:00, edited 1 time in total.
Re: UIAutomation with a focus on Chrome
@Joe Glines Incredible! Thank you very much
@Descolada that last message is super interesting. Thank you.
@Descolada that last message is super interesting. Thank you.
Re: UIAutomation with a focus on Chrome
Updated UIA_Interface, UIA_Browser, example files, UIAViewer and UIATreeInspector.
UIA_Interface updates:
1) UIA_Interface.CreateCondition now supports expressions (parenthesis are allowed). Example: UIA.CreateCondition("(type=button && name='SomeName' flags=2) or type=menuitem") would create a condition that matches either a button with "SomeName" anywhere in its name, or a menuitem.
2) UIA_Element.FindFirstBy and FindAllBy support parenthesis if matchMode is 3 or 2 (and if matching substrings is supported in the currently running Windows: at least Windows 10 October 2018 Update, version 1809). MatchMode Regex and 1 do not support parenthesis, but now the first "string" type property will have these matchmodes applied (eg in RegEx mode "Type=Button and AutomationId='.*test?'" would search for button elements with RegexMatch for ".*test?"
3) UIA_Element.Highlight() highlights the element borders (for debugging purposes).
4) Properties have been rewritten, now Current... properties can be accessed without the "Current" part (eg UIA_Element.Value, UIA_Element.BoundingRectangle). For some properties, setting values is supported (UIA_Element.Value := "test").
4) Patterns can be accessed like properties: UIA_Element.ValuePattern gets UIA_ValuePattern (note that the highest version of the pattern won't be selected).
5) UIA_Element.Click() and UIA_Element.ControlClick() first or second arguments can be a "sleep after" value.
UIA_Browser updates: Added Navigate method for easier navigation. Fixed bug in JSSetTitle.
Example files have been updated to run "out of the box" without changing #include.
UIAViewer and UIATreeInspector have been updated with multiple bug-fixes.
UIA_Interface updates:
1) UIA_Interface.CreateCondition now supports expressions (parenthesis are allowed). Example: UIA.CreateCondition("(type=button && name='SomeName' flags=2) or type=menuitem") would create a condition that matches either a button with "SomeName" anywhere in its name, or a menuitem.
2) UIA_Element.FindFirstBy and FindAllBy support parenthesis if matchMode is 3 or 2 (and if matching substrings is supported in the currently running Windows: at least Windows 10 October 2018 Update, version 1809). MatchMode Regex and 1 do not support parenthesis, but now the first "string" type property will have these matchmodes applied (eg in RegEx mode "Type=Button and AutomationId='.*test?'" would search for button elements with RegexMatch for ".*test?"
3) UIA_Element.Highlight() highlights the element borders (for debugging purposes).
4) Properties have been rewritten, now Current... properties can be accessed without the "Current" part (eg UIA_Element.Value, UIA_Element.BoundingRectangle). For some properties, setting values is supported (UIA_Element.Value := "test").
4) Patterns can be accessed like properties: UIA_Element.ValuePattern gets UIA_ValuePattern (note that the highest version of the pattern won't be selected).
5) UIA_Element.Click() and UIA_Element.ControlClick() first or second arguments can be a "sleep after" value.
UIA_Browser updates: Added Navigate method for easier navigation. Fixed bug in JSSetTitle.
Example files have been updated to run "out of the box" without changing #include.
UIAViewer and UIATreeInspector have been updated with multiple bug-fixes.
Last edited by Descolada on 16 Jul 2022, 07:23, edited 1 time in total.
-
- Posts: 143
- Joined: 25 Dec 2020, 12:26
Re: UIAutomation with a focus on Chrome
very nice
Re: UIAutomation with a focus on Chrome
Hi @Descolada
I am trying to do this, to find the file name of the image, using Microsoft Office 2010 (Picture manager).
It quickly finds the 'Next' button, but when I use the Path to search for the next item (Image), the program crashes for overload. It doesn't find anything either.
In general I have noticed that FindByPath takes a long time to find any item using the "+n".
Can you help me?
I am trying to do this, to find the file name of the image, using Microsoft Office 2010 (Picture manager).
It quickly finds the 'Next' button, but when I use the Path to search for the next item (Image), the program crashes for overload. It doesn't find anything either.
In general I have noticed that FindByPath takes a long time to find any item using the "+n".
Can you help me?
Code: Select all
#SingleInstance, Force
SendMode Input
SetWorkingDir, %A_ScriptDir%
#Include <UIA_Interface>
UIA := UIA_Interface()
WinActivate, Microsoft Office 2010
cEl := UIA.ElementFromHandle(WinExist("Microsoft Office 2010"))
^<::
Begins:
next := cEl.FindFirstBy("Name=Siguiente")
MsgBox, % next.FindByPath("+1", UIA.CreateCondition("ControlType", "Image")).Dump()
Re: UIAutomation with a focus on Chrome
@mora145, you code is fine, but there is something broken with Microsofts implementation of the TreeWalker in that program (inspect.exe treewalking doesn't work properly either). I'm filing a bug report, but it probably won't be fixed very soon :/
A workaround to get the image element:
A workaround to get the image element:
Code: Select all
next := cEl.FindFirstBy("NOT Name= AND ControlType=Image")
MsgBox, % next.Dump()
Re: UIAutomation with a focus on Chrome
It has also happened to me with Chrome, on pages with many elements. I thought TreeWalker consumed a lot of resources very fast and crashed the program.Descolada wrote: ↑16 Jul 2022, 00:23@mora145, you code is fine, but there is something broken with Microsofts implementation of the TreeWalker in that program (inspect.exe treewalking doesn't work properly either). I'm filing a bug report, but it probably won't be fixed very soon :/
A workaround to get the image element:Code: Select all
next := cEl.FindFirstBy("NOT Name= AND ControlType=Image") MsgBox, % next.Dump()
Your implementation is excellent. It works much faster. I guess it's not always a good idea to walk through the Element Tree.
Re: UIAutomation with a focus on Chrome
Hi all! It's my first post here. Many thanks to Descolada and all others for this highly useful UIA library! I'm currently trying to automate a work related program using UIA. Some tasks I have automated successfully, but I've come across a problem I can't make sense of.
After correctly identifying a button and clicking it using .click(), the script waits for 10 seconds or so and gives an error message. The click script seems to be using Invoke(). The same error happens if I identify the button using UIAViewer and manually Invoke the element. The error message is usually on UIA_Interface.ahk line 1174 or 147, it seems. Sometimes the error contains "UIA_E_ELEMENTNOTAVAILABLE" and "Specifically: Invoke (UIA_InvokePattern)". Some of the dysfunctioning buttons contain DoDefaultAction in addition to Invoke pattern. I get an error regardless of which I am using.
This is puzzling, since the script does in fact find and is able to click the element before the error. Also the error doesn't replicate for all buttons in the software. Any ideas how I could solve this problem or is this perhaps a bug?
Attached are some screenshots and my code.
After correctly identifying a button and clicking it using .click(), the script waits for 10 seconds or so and gives an error message. The click script seems to be using Invoke(). The same error happens if I identify the button using UIAViewer and manually Invoke the element. The error message is usually on UIA_Interface.ahk line 1174 or 147, it seems. Sometimes the error contains "UIA_E_ELEMENTNOTAVAILABLE" and "Specifically: Invoke (UIA_InvokePattern)". Some of the dysfunctioning buttons contain DoDefaultAction in addition to Invoke pattern. I get an error regardless of which I am using.
This is puzzling, since the script does in fact find and is able to click the element before the error. Also the error doesn't replicate for all buttons in the software. Any ideas how I could solve this problem or is this perhaps a bug?
Attached are some screenshots and my code.
Code: Select all
#NoEnv
#SingleInstance force
#include <UIA_Interface>
UIA := UIA_Interface()
q::
Asiakastuote_lisäys:
WinActivate, Käynnin yhteenveto
Sleep, 100
;asiakastuote := % A_Args[2]
KäynninYhteenveto := UIA.ElementFromHandle(WinExist("Käynnin yhteenveto"))
Kontaktitiedot := KäynninYhteenveto.FindFirstBy("ClassName=ContactDataControl")
Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink").Click()
Return
Re: UIAutomation with a focus on Chrome
Spoiler
My mistake, I just saw that using an automazation ID
Last edited by mora145 on 16 Jul 2022, 08:02, edited 2 times in total.
Re: UIAutomation with a focus on Chrome
@Youhoney, quite interesting. If Invoke was called successfully, then the Click function should return immediately. This might mean that the implementation for UIA in that program might be the cause of this? Especially if other programs don't have this problem.
Could you try Accessibility Insights or inspect.exe tools as well - do they cause the same error?
As a workaround you could use Click("left") or ControlClick()...
Could you try Accessibility Insights or inspect.exe tools as well - do they cause the same error?
As a workaround you could use Click("left") or ControlClick()...
Re: UIAutomation with a focus on Chrome
@Descolada thanks for the swift reply! It might very well be the case that the UIA implementation is the problem. I haven't yet tried other programs. However, some buttons in this software work fine. I'm unfortunately using AHK on a work computer and I don't have admin rights and couldn't therefore install Accessibility Insights. I tried viewtopic.php?t=28866 Inspect.ahk (not sure if this is what you meant, but I didn't find any way to use Invoke through that.
Thanks for the tips, I did get ControlClick working, which hopefully gets the job done reliably enough.
Thanks for the tips, I did get ControlClick working, which hopefully gets the job done reliably enough.
Re: UIAutomation with a focus on Chrome
@Youhoney, inspect.exe is a separate program (not an AHK script), you can download it for example here.
I had another though in the meanwhile: if Invoke clicks your element but then stops for a while and throws an error, perhaps you could try temporarily changing the timeout values:
Glad to head you got it working
I had another though in the meanwhile: if Invoke clicks your element but then stops for a while and throws an error, perhaps you could try temporarily changing the timeout values:
Code: Select all
saveConnectionTimeout := UIA.ConnectionTimeout, saveTransactionTimeout := UIA.TransactionTimeout ; Save previous values for timeouts
UIA.ConnectionTimeout := 500, UIA.TransactionTimeout := 200 ; Set shorter timeouts (could experiment with even shorter ones, minimum is 50)
try Kontaktitiedot.FindFirstBy("AutomationId=StatisticsContactDataInvoicingHyperlink").Click() ; Use "try" to avoid the error being thrown
UIA.ConnectionTimeout := saveConnectionTimeout, UIA.TransactionTimeout := saveTransactionTimeout ; Restore previous timeout values
Return to “Scripts and Functions (v1)”
Who is online
Users browsing this forum: No registered users and 95 guests