UIAutomation with a focus on Chrome

[leosouza85]

@leosouza85, have you tried element.FindFirstByNameAndType("Anexar","Button",,2,False)? This should make sure that you are not accidentally looking for "Anexar" if the element is actually named "Anexar " with a space at the end, and also turns off case-sensitivity.
I haven't encountered a situation where an element can't be found if its too deep, can you bring a testable example on some webpage?

Hi! I will try that, I've edited my previous post, and detected that the element is found, but the element has an illegal character, a font awesome icon... and the functions wait and find dont find the text only the illegal character, but uia viewer and treeinspector show the illegal character and the text...

I dont have a site to show because it is a government site that only works internally

The name is detected by the function dumpAll as Name: "" and on UIAViwer it detects as: Name: " Anexar mais arquivos".

I think the other functions are detecting as the DumpAll function does and do not work on this page

[Descolada]

@leosouza85, UIAViewer uses the same methods as DumpAll to display the CurrentName value, so I think this might be an encoding issue. Make sure your script is saved in the proper encoding UTF-8-BOM (not regular UTF-8 or something similar) and try again.

For example the following code works only in UTF-8-BOM, open this site to test: https://codepen.io/vivinantony/pen/bGzdxW

Code: Select all

SetTitleMatchMode, 2
#include <UIA_Interface>
UIA := UIA_Interface()
chromeId := WinExist("ahk_exe chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%
chrome := UIA.ElementFromHandle(chromeId, True)
chrome.FindFirstByName("Нравится2 тыс.").Click("left")

[leosouza85]

@leosouza85, UIAViewer uses the same methods as DumpAll to display the CurrentName value, so I think this might be an encoding issue. Make sure your script is saved in the proper encoding UTF-8-BOM (not regular UTF-8 or something similar) and try again.

For example the following code works only in UTF-8-BOM, open this site to test: https://codepen.io/vivinantony/pen/bGzdxW

Code: Select all

SetTitleMatchMode, 2
#include <UIA_Interface>
UIA := UIA_Interface()
chromeId := WinExist("ahk_exe chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%
chrome := UIA.ElementFromHandle(chromeId, True)
chrome.FindFirstByName("Нравится2 тыс.").Click("left")

The script is saved on that encoding, UTF-8-BOM, I can click the element with find by path, but with this code it is not clicking

cUIA.FindByPath("1.2.2.3.1.9.3.1.1").Click("left") ;Works
cUIA.WaitElementExistByNameAndType("Anexar", 50020,,2,False).Click("left") ;Not work

Im using Click("left") just to see the mouse and making sure that the element is not found

[Descolada]

@leosouza85, just to confirm - you ran my provided example and it worked?

I don't think I can help you much more without a test case. At least you got it working with FindByPath, so all is not lost

[leosouza85]

@leosouza85, just to confirm - you ran my provided example and it worked?

I don't think I can help you much more without a test case. At least you got it working with FindByPath, so all is not lost

No your sugestion not work here... but thankyou so much anyway...

But what if you modify the functions to ready only legal characters?

[Descolada]

@leosouza85, it seems that the code I provided doesn't work, because AHK forums has apparently removed the illegal invisible character in it What I did: navigate to that test site and inspect the Facebook like button. It has a "item" or "button" element, which has some text: I copied it into my script. That text contains an invisible U+00a0 character and also some cyrillic, good for testing encoding problems... If your script is UTF-8-BOM and UIA_Interface.ahk is also saved in UTF-8-BOM, and you have made sure you are running the latest UIA_Interface.ahk from my Github, then perhaps try a suggestion by user @SundayProgrammer:

Code: Select all

Conv4uChar(t) {
	s := StrPut(t, "CP0")
	VarSetCapacity(u, s)
	s := StrPut(t, &u, s, "CP0")
	Return StrGet(&u, "UTF-8")
}

It should remove all illegal characters from your search phrase, but I don't think it will work here, because it doesn't affect the actual search function (FindFirst) which is a UIA native method (not added by me) and can't be altered by us.

"" and on UIAViwer it detects as: Name: " Anexar mais arquivos"

The thing is, that UIAViewer uses element.CurrentName and DumpAll() uses element.CurrentName as well, no other processing inbetween. So they should return the same value... This highly suggests that something is up with the encoding - what encoding is you UIAViewer saved as?

[leosouza85]

@leosouza85, it seems that the code I provided doesn't work, because AHK forums has apparently removed the illegal invisible character in it What I did: navigate to that test site and inspect the Facebook like button. It has a "item" or "button" element, which has some text: I copied it into my script. That text contains an invisible U+00a0 character and also some cyrillic, good for testing encoding problems... If your script is UTF-8-BOM and UIA_Interface.ahk is also saved in UTF-8-BOM, and you have made sure you are running the latest UIA_Interface.ahk from my Github, then perhaps try a suggestion by user @SundayProgrammer:

Code: Select all

Conv4uChar(t) {
	s := StrPut(t, "CP0")
	VarSetCapacity(u, s)
	s := StrPut(t, &u, s, "CP0")
	Return StrGet(&u, "UTF-8")
}

It should remove all illegal characters from your search phrase, but I don't think it will work here, because it doesn't affect the actual search function (FindFirst) which is a UIA native method (not added by me) and can't be altered by us.

"" and on UIAViwer it detects as: Name: " Anexar mais arquivos"

The thing is, that UIAViewer uses element.CurrentName and DumpAll() uses element.CurrentName as well, no other processing inbetween. So they should return the same value... This highly suggests that something is up with the encoding - what encoding is you UIAViewer saved as?

Everything is saved in UTF-8-BOM

I've made a mistake.... both functions and uiaviwer are finding Name: " Anexar mais arquivos"... Im trying other things to see if i can debug that... I will keep in touch in case i solve this

The Strange thing is... its something on computer or in chrome version... the code works from home, but not work at work... at work i can click in other elements... but on that element only by path

[marynofear]

UIAViewer-1.jpg (35.42 KiB) Viewed 3505 times

First, thanks for the great work.

I have a smaller problem. Needs to captures and evaluate some numbers from a webpage. (ui.soc.eu-central-1.mcafee.com)
Everything seems to be working but for some reason i cant get one "0" value.
The value shows in UIAViewer and in UIATreeInspector. I can even copy the number from the webpage. But no get it by code.
Code is working getting all other values but not the "0". Im using latest version of the libraries. Have not tried with older versions.
Pictures of Msg-boxes are just for info. One finds the correct value. The other finds the "0" but its not in the OneText.Dump()
Is it a problem in the lib or is it something I do wrong ?

Code: Select all

UIA := UIA_Interface()
chromeId := WinExist("ahk_exe chrome.exe")
WinActivate, ahk_id %chromeId%
WinWaitActive, ahk_id %chromeId%
chrome := UIA.ElementFromHandle(chromeId, True)

nameText := chrome.FindFirstByName("Total Threats") 
TextWalker := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType", "Text"))
oneText := TextWalker.GetNextSiblingElement(nameText) 

MsgBox, % "Date: " oneText.Dump()

[Descolada]

@marynofear, it is a problem in the Dump() method. Your screenshot means that the element is found properly, and element.CurrentName should display "0" for you. But the Dump() method only displays values if they exist (eg if CurrentName is "", then the name won't be displayed at all), and "0" was evaluated as "not existing". I fixed this and the new UIA_Interface.ahk is available at GitHub.
I also upgraded the FindByPath method, so now you could use oneText := nameText.FindByPath("+1", UIA.CreateCondition("ControlType", "Text"))

@mora145, I added your desired functionality to FindByPath (available at GitHub): dateText1 := bulletText.FindByPath("-2", UIA.CreateCondition("ControlType", "Text")) calls GetPreviousSiblingElement twice.

[Joe Glines]

@Joe Glines, glad to see UIAutomation getting more popular! Hopefully more people can start enjoying it (and motivates me to fix important bugs faster )

Would you be up for joining us on Zoom sometime to record a video discussing the class & uses? If so, you can email me at [email protected]...

[marynofear]

@marynofear, it is a problem in the Dump() method. Your screenshot means that the element is found properly, and element.CurrentName should display "0" for you. But the Dump() method only displays values if they exist (eg if CurrentName is "", then the name won't be displayed at all), and "0" was evaluated as "not existing". I fixed this and the new UIA_Interface.ahk is available at GitHub.
I also upgraded the FindByPath method, so now you could use oneText := nameText.FindByPath("+1", UIA.CreateCondition("ControlType", "Text"))

@mora145, I added your desired functionality to FindByPath (available at GitHub): dateText1 := bulletText.FindByPath("-2", UIA.CreateCondition("ControlType", "Text")) calls GetPreviousSiblingElement twice.

@Descolada What a great support - Your one of the best around.
Now everything I needs, works perfectly. And I can finally monitor for changes in this value.
This is way more simple that having to use Rufaydium WebDriver and deal chromedriver needing to be update together with every Chrome update.
- Million Thanks - This is crazy good programming and support -

[leosouza85]

@Joe Glines, glad to see UIAutomation getting more popular! Hopefully more people can start enjoying it (and motivates me to fix important bugs faster )

Would you be up for joining us on Zoom sometime to record a video discussing the class & uses? If so, you can email me at [email protected]...

Thanks for the chat with descolada on youtube... I've learned so much... tomorow I will try to apply the knoledge at work, to some elements that are in other windows with root element. Any progress in skype automation with that?

also you could change the scope of the search to see if it will work like
test.WaitElementToExistByName("name",0x7,2,false)

the default scope is 0x4, descendants, the function search all descendants (from the handle). The scope 0x7, subtree, includes the root of the search and all descendants

Ancestors 16
Specifies that the search include the element's ancestors, including the parent. Not supported.

Children 2
Specifies that the search include the element's immediate children.

Descendants 4
Specifies that the search include the element's descendants, including children.

Element 1
Specifies that the search include the element itself.

Parent 8
Specifies that the search include the element's parent. Not supported.

Subtree 7
Specifies that the search include the root of the search and all descendants.

[Descolada]

@leosouza85, unfortunately not much progress with Skype. I tested it on three setups:
1) My main setup: cannot detect elements with ElementFromHandle, but ElementFromPoint works
2) Windows 10 VM && Windows 11 VM: both work after activating accessibility (by setting activateChromiumAccessibility to True)

I haven't figured out what the difference between the setups are yet. Skype versions should be the same, so perhaps a Windows update changes the behaviour? Perhaps somebody will figure it out
In the meantime it's still possible to access all the content of Skype with UIA, but the window must be visible for this:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

UIA := UIA_Interface()
WinActivate, ahk_exe skype.exe
WinWaitActive, ahk_exe skype.exe

skypeEl := UIA.ElementFromHandle(WinExist("ahk_exe skype.exe"), True)
MsgBox, % skypeEl.DumpAll() ; Try with ElementFromHandle first
WinActivate, ahk_exe skype.exe ; Activate again after the MsgBox (sometimes loses focus)
WinWaitActive, ahk_exe skype.exe

WinGetPos, X, Y, W, H, ahk_exe skype.exe ; Get Skype window location
chromiumTW := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType","Document")) ; Create a treewalker to find the Document element
el := UIA.ElementFromPoint(x+w//2, y+h//2) ; Use ElementFromPoint on the center of the window (the actual coordinate doesn't really matter, it just needs to be inside the window)
try el := chromiumTW.NormalizeElement(el) ; Get the first parent that is a Window element
MsgBox, % el.DumpAll()

[TomDonovan]

Has anyone tried UIA with BitWarden. Using UIAViewer in the main BitWarden window, if I click on my saved items, I can get the viewer to show the item, but I have been unsuccessful finding that item thru code and UIATreeInspector doesn't see the items. Also in UIAViewer if I press F2 after having selected my item, and pressed esc so it stays, the window structure that it builds no longer show my item (just like UIATreeInspector).

It would be nice if UIAViewer could show same information as Dumpall as maybe if I could have the path I could find the item, but I haven't been able to find anything that could show a path?

[Skrell]

Your latest UIAViewer is still not detecting Chrome's caption buttons correctly unfortunately...

image.png (6.15 KiB) Viewed 2523 times

Forgive me if I'm wrong, but I thought this is why you introduced the activateChromiumAccessibility parameter? So then if I'm hovering over Chrome I can set this to "true" and force it to dive deeper into the tree to find the correct button?

ElementFromPoint(x="", y="", activateChromiumAccessibility=False)

[Descolada]

@TomDonovan, yes this same issue exists with a lot of Electron/Chromium web-based apps. I still haven't figured out the exact cause of this unfortunately (I know it isn't related to the UIA_Interface library, its caused by the apps themselves), but the workaround is the same as for Skype:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

UIA := UIA_Interface()
WinActivate, ahk_exe bitwarden.exe
WinWaitActive, ahk_exe bitwarden.exe
WinGetPos, X, Y, W, H, ahk_exe bitwarden.exe ; Get window location
el := UIA.ElementFromPoint(x+w//2, y+h//2) ; Use ElementFromPoint on the center of the window (the actual coordinate doesn't really matter, it just needs to be inside the window)
chromiumTW := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType","Window")) ; Create a treewalker to find the Window element
try el := chromiumTW.NormalizeElement(el) ; Get the first parent that is a Window element
MsgBox, % el.DumpAll()

@Skrell, activateChromiumAccessibility triggers accessibility for Chrome when it is turned off, and if it is off then you shouldn't be able to access any of the elements (tabs, text etc). To deal with the problem of getting min-max buttons I introduced the global DeepSearchFromPoint variable at the top of UIAViewer source code, which if turned to True will detect those elements. Internally it uses the method SmallestElementFromPoint.

[Skrell]

@Descolada So would you expect this to work then?

Code: Select all

    
    MouseGetPos, mX, mY, mHwnd, mCtrl
    WinGetClass, wClass, ahk_id %mHwnd%
            
    try {
        If wClass == "Chrome_WidgetWin_1"
            mEl := UIA.SmallestElementFromPoint(mX, mY, True, UIA.ElementFromHandle(mHwnd))
        Else
            mEl := UIA.ElementFromPoint(mX, mY)
    }
    
        If (mEl.CurrentName == "Close")
        {
            Tooltip, %wClass% " Closed!"
        }
    

[Descolada]

@Skrell, I wouldn't expect that to work. UIA isn't defined; wClass == "Chrome_WidgetWin_1" is probably evaluated as legacy syntax; MouseGetPos returns relative coordinates but UIA uses absolute. This should work though:

Code: Select all

CoordMode, Mouse, Screen

#include <UIA_Interface>
UIA := UIA_Interface()
Loop {
    MouseGetPos, mX, mY, mHwnd, mCtrl
    WinGetClass, wClass, ahk_id %mHwnd%
            
    try {
        If (wClass == "Chrome_WidgetWin_1")
            mEl := UIA.SmallestElementFromPoint(mX, mY, True, UIA.ElementFromHandle(mHwnd))
        Else
            mEl := UIA.ElementFromPoint(mX, mY, True)
		
    }
        If (mEl.CurrentName == "Close")
        {
            Tooltip, %wClass% " Closed!"
        }
}

[Skrell]

@Skrell, I wouldn't expect that to work. UIA isn't defined; wClass == "Chrome_WidgetWin_1" is probably evaluated as legacy syntax; MouseGetPos returns relative coordinates but UIA uses absolute. This should work though:

Code: Select all

CoordMode, Mouse, Screen

#include <UIA_Interface>
UIA := UIA_Interface()
Loop {
    MouseGetPos, mX, mY, mHwnd, mCtrl
    WinGetClass, wClass, ahk_id %mHwnd%
            
    try {
        If (wClass == "Chrome_WidgetWin_1")
            mEl := UIA.SmallestElementFromPoint(mX, mY, True, UIA.ElementFromHandle(mHwnd))
        Else
            mEl := UIA.ElementFromPoint(mX, mY, True)
		
    }
        If (mEl.CurrentName == "Close")
        {
            Tooltip, %wClass% " Closed!"
        }
}

My apologies... That is exactly what I had I just failed to include everything. This code did not work for me... Maybe I need admin?

FYI: If I do this:

Code: Select all

        
        If (wClass == "Chrome_WidgetWin_1")
        {
        	Tooltip, "1"
        	mEl := UIA.SmallestElementFromPoint(mX, mY, True, UIA.ElementFromHandle(mHwnd))
        	Tooltip, "2"
        }
        Else
            mEl := UIA.ElementFromPoint(mX, mY, True)
            

I never see "2" but I do see "1"

[mora145]

@marynofear, it is a problem in the Dump() method. Your screenshot means that the element is found properly, and element.CurrentName should display "0" for you. But the Dump() method only displays values if they exist (eg if CurrentName is "", then the name won't be displayed at all), and "0" was evaluated as "not existing". I fixed this and the new UIA_Interface.ahk is available at GitHub.
I also upgraded the FindByPath method, so now you could use oneText := nameText.FindByPath("+1", UIA.CreateCondition("ControlType", "Text"))

@mora145, I added your desired functionality to FindByPath (available at GitHub): dateText1 := bulletText.FindByPath("-2", UIA.CreateCondition("ControlType", "Text")) calls GetPreviousSiblingElement twice.

Were there any problems with this? I think you removed it the next day.