UIAutomation with a focus on Chrome

[Descolada]

@Skrell, if mEl.FindFirstByNameAndType("Minimize", "Button") ... should work

[Skrell]

@Skrell, if mEl.FindFirstByNameAndType("Minimize", "Button") ... should work

but isn't the result of this function an object? How does the if statement know what to do with it?

[Descolada]

@Skrell, yes, if it is successful then an object is returned, and if unsuccessful then an empty string is returned. Since if Object evaluates as True, and if "" evaluates as False, you can use this to test whether something was found or not.

[Skrell]

@Skrell, yes, if it is successful then an object is returned, and if unsuccessful then an empty string is returned. Since if Object evaluates as True, and if "" evaluates as False, you can use this to test whether something was found or not.

What about checking the return UIA.ElementFromPoint() ?

[Descolada]

@Skrell, UIA.ElementFromPoint() should always return an element, which at the very least is the root element (Desktop). It can happen that the element is removed from the screen by the time the function has ended, in which case UIA_E_ELEMENTNOTAVAILABLE error is thrown.

[Skrell]

@Skrell, UIA.ElementFromPoint() should always return an element, which at the very least is the root element (Desktop). It can happen that the element is removed from the screen by the time the function has ended, in which case UIA_E_ELEMENTNOTAVAILABLE error is thrown.

So you can't then simply write If (UIA.ElementFromPoint()){} ? You'd have to first call UIA.ElementFromPoint() THEN write something after like

Code: Select all

If (errorlevel == UIA_E_ELEMENTNOTAVAILABLE ) 

?

[guest3456]

@Skrell, UIA.ElementFromPoint() should always return an element, which at the very least is the root element (Desktop). It can happen that the element is removed from the screen by the time the function has ended, in which case UIA_E_ELEMENTNOTAVAILABLE error is thrown.

So you can't then simply write If (UIA.ElementFromPoint()){} ? You'd have to first call UIA.ElementFromPoint() THEN write something after like

Code: Select all

If (errorlevel == UIA_E_ELEMENTNOTAVAILABLE )

?

he already told you that if object evals to true

[Descolada]

@Skrell, in most cases you can just call UIA.ElementFromPoint() and expect an element, no "if" necessary. If you want to be extra sure and careful, you can do

Code: Select all

try UIA.ElementFromPoint()
catch { ; UIA_E_ELEMENTNOTAVAILABLE was thrown, handle this somehow
    ; try UIA.ElementFromPoint again perhaps?
}

[Skrell]

@Skrell, in most cases you can just call UIA.ElementFromPoint() and expect an element, no "if" necessary. If you want to be extra sure and careful, you can do

Code: Select all

try UIA.ElementFromPoint()
catch { ; UIA_E_ELEMENTNOTAVAILABLE was thrown, handle this somehow
    ; try UIA.ElementFromPoint again perhaps?
}

Got it thank you!

[leosouza85]

@Descolada
HI! found a possible bug in UIA Viewer.

When inspecting elements, uia viewer uses the resolution of the first monitor, so if i have a second monitor on diferent resolution it does not show the bounding rectangle in the correct place.

[Descolada]

@leosouza85, this is a known problem with Acc/UIA for which I don't know the fix. Adding DllCall("SetThreadDpiAwarenessContext", "ptr", -4, "ptr") to the beginning of the script seems to *partially* fix the problem, at least for the Taskbar and Win32 programs it works. But other programs (like most Chromium apps - Chrome, VSCode etc...) are still broken - the main window location is reported correctly, but elements are not. Also I am not sure what side-effects SetThreadDpiAwarenessContext might have, so I am a bit wary of changing it :/ I've tried converting logical coordinates to physical, but in Windows10 this does nothing. So I'm afraid this will stay broken for now...

[pto]

Thanks to @jethrow for this piece of code.
@Descolada, thank you for maintaining and upgrading it.
I've also been reading this thread and finding your support very thorough. Kudos.

I have a request, wich i believe that the "not so experts" like me would beneficiate.

In UIA_Browser.ahk, which contains helper functions, is it possible to have a GetAllUrls()?
Or maybe you could assist me, getting all urls without activate every tab?

[pto]

Sorry for the double post (didn't find edit button).

Does anyone knows how to find Chrome's localized (pt-pt) string for "StopButtonText"?

[tester]

@Descolada Thanks for such a useful library and thorough examples for new users. For some reasons, UIA_Interface.ControlClick() method doesn't seem to work. Could you take a look at this topic if you get a chance?

[Descolada]

@pto, UIA_Browser.ahk already has the GetAllLinks method, which returns all HyperLink elements. To get the URLs from them, loop through the elements and get the Value property.
Unfortunately getting all the URLs without activating tabs isn't possible in Chrome, since elements for a given tab are only accessible when the tab is active. Though you could use URLDownloadToFile or get the page's source code with COM, and then extract the URLs from there (which would be much faster and probably more reliable anyway).

And if anyone figures out how to get Chrome's localized string for the Stop button, please share it in this thread

@tester, ControlClick doesn't necessarily work in every program. Does sending 1 click instead of 2 do anything in the program (or in other elements)? You could try the reliability tips from the documentation. Unfortunately I don't have access to Kindle so I can't test this out, but in the free part of the app clicking the "Add to collections button" worked fine:

Code: Select all

SetTitleMatchMode, 2
UIA_Interface().ElementFromHandle("Kindle for PC").FindFirstByName("Create or Import new Collections").ControlClick("Kindle for PC",,"left", 2)

Other ideas:
1) Perhaps AHK ControlClick is double-clicking too fast for the program to respond? Try calling it twice with a small sleep inbetween, or set SetControlDelay to a higher value (60ms?).
2) Sometimes WaitElementExist returns the element because the element does exist, but isn't necessarily accessible yet. Try a small sleep between WaitElementExist and ControlClick and see if that changes anything?

[pto]

In UIA_Browser class, at Navigate() (line 348), you have this:

Code: Select all

this.WaitPageLoad(targetTitle, waitLoadTimeOut, sleepAfter, stopButtonText="Stop")

Maybe you could upgrade to something like:

Code: Select all

this.WaitPageLoad(targetTitle, waitLoadTimeOut, sleepAfter, this.CustomNames.stopButtonText ? this.CustomNames.stopButtonText : stopButtonText)

[pto]

@pto, UIA_Browser.ahk already has the GetAllLinks method, which returns all HyperLink elements. To get the URLs from them, loop through the elements and get the Value property.
Unfortunately getting all the URLs without activating tabs isn't possible in Chrome, since elements for a given tab are only accessible when the tab is active. Though you could use URLDownloadToFile or get the page's source code with COM, and then extract the URLs from there (which would be much faster and probably more reliable anyway).

Sorry for the lack of info on my end. What i intend is a backend way to match the address bar url agaisnt title as an extra security measure.

And if anyone figures out how to get Chrome's localized string for the Stop button, please share it in this thread

Thank's for the appeal

[tester]

Unfortunately I don't have access to Kindle so I can't test this out, but in the free part of the app clicking the "Add to collections button" worked fine:

Code: Select all

SetTitleMatchMode, 2
UIA_Interface().ElementFromHandle("Kindle for PC").FindFirstByName("Create or Import new Collections").ControlClick("Kindle for PC",,"left", 2)

Thanks for your insights and even testing some code. I tried your suggestions but to no avail. The quoted code doesn't work if the window is not active (at least in my environment). For example, while making the Kindle app window behind a Notepad window and if you run that code, the click doesn't occur.

To see if ControlClick works to added books, you can manually add some PDF files to that app library without logging-in from File -> Import PDF... and check if you can double-click an item while the window is inactive or minimized. If an item is double-clicked, the app opens an UI element to show its contents.

[Descolada]

@tester, I added some PDF files and tried it out. In my setup the following code selects the book and opens it even when behind other windows, but unfortunately both ControlSend and ControlClick appear to activate the window, so it can't be done without eventually activating it.

Code: Select all

SetTitleMatchMode, 2
sBookTitle := "mybook"
UIA := UIA_Interface()
UIA.AutoSetFocus := False
KindleEl := UIA.ElementFromHandle("Kindle for PC")
KindleEl.FindFirstBy("AutomationId=ReaderMainWindow.centralWidget.ToolbarReading.homeButton").Click()
KindleEl.WaitElementExist("ControlType=ListItem AND Name='" sBookTitle "'",,2).SelectionItemPattern.Select()
ControlSend, Qt5QWindowIcon6, {Enter}, Kindle for PC
; KindleEl.WaitElementExist("ControlType=ListItem AND Name='" sBookTitle "'",,2).ControlClick("Kindle for PC",,"left", 2) ; Alternative way

Note that I needed to change ControlType=DataItem to ControlType=ListItem.

I thought perhaps navigating through the menu with File -> Open Book would avoid activating the window, but no such luck: the File button can't be invoked for some reason. Also minimized windows can't be interacted with UIA, nor with ControlClick/ControlSend. A workaround could be to restore the window, move it offscreen, open the book, then move onscreen again and minimize.

Maybe you could upgrade to something like:

Code: Select all

this.WaitPageLoad(targetTitle, waitLoadTimeOut, sleepAfter, this.CustomNames.stopButtonText ? this.CustomNames.stopButtonText : stopButtonText)

@pto, fixed, thanks.

[pto]

@Descolada , i'm using it so i'm glad to contibute, i'll keep on posting if i find anything else.