UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Skrell
Posts: 302
Joined: 23 Jan 2014, 12:05

Re: UIAutomation with a focus on Chrome

Post by Skrell » 09 Dec 2022, 08:56

Descolada wrote:
09 Dec 2022, 00:05
@Skrell, both ConnectionTimeout and TransactionTimeout are part of the main UIA_Interface properties, so you can get the current value with UIA.TransactionTimeout and change it with UIA.TransactionTimeout := 20.
THANK YOU ALWAYS! :bravo:

User avatar
marynofear
Posts: 4
Joined: 20 Oct 2018, 19:03

Re: UIAutomation with a focus on Chrome

Post by marynofear » 09 Dec 2022, 20:25

@Descolada Do you have any plan for converting to AHK V2 ?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 10 Dec 2022, 06:28

@marynofear, there are already UIA libraries available for AHK v2, for example thqby's UIAutomation.ahk.
I do have plans of using his library as a base and convert all the "helper functions" (such as Click and Highlight) to AHK v2 as well, but progress is slow at the moment. I hope to have an usable version ready in the first half of next year, but no promises.

r2997790
Posts: 71
Joined: 02 Feb 2017, 02:46

Re: UIAutomation with a focus on Chrome

Post by r2997790 » 11 Dec 2022, 05:55

Hello friends and @Descolada in particular,

I'm making steady progress with UIA and discovering a lot on the way... I was wondering if it was possible to use RegEx with UIA elements...

for example:

Code: Select all

findStartAge := cUIA.FindFirstByNameAndType("18", "Button")
findStartAge.click()
... the button I need to check in fact could have the name from any 2 digits, from 13..64 or the string "65+" ... it is possible some how to use RegEx to allow a variety of conditions?

ALSO... is there an elegant way to capture errors / skip elements when they don't exist so the script doesn't pause or crash... I'm trying to make it as 'full proof' as possible.

Many thanks everyone.
R

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 11 Dec 2022, 07:06

@r2997790, you can set the matchMode argument to "RegEx": findStartAge := cUIA.FindFirstByNameAndType("\d+\+?", "Button",, "RegEx"). Performance-wise it will be similar to FindAll though, because RegEx matching isn't provided natively by Microsofts UIA.

You can skip elements by checking whether they were found or not:

Code: Select all

if !(findStartAge := cUIA.FindFirstByNameAndType("18", "Button"))
    return ; element wasn't found, so skip the rest
findStartAge.click()
Alternatively use try...catch:

Code: Select all

try {
    findStartAge := cUIA.FindFirstByNameAndType("18", "Button")
    findStartAge.click()
} catch {
    ; something went wrong: an exception was thrown or ErrorLevel is raised
}

r2997790
Posts: 71
Joined: 02 Feb 2017, 02:46

Re: UIAutomation with a focus on Chrome

Post by r2997790 » 12 Dec 2022, 08:10

@Descolada thank you so much for your very generous interventions and examples which have helped nudge me in the right direction.

I know you've been following my attempt to automate parts of the Facebook ads page... I find an odd example here...
image.png
image.png (12.89 KiB) Viewed 3012 times
These 'cells' are clickable DIVS which are not accessible so I've accessed them via Bounded Rectangles and then surrended to mousemoves to click them. It works. But interesting sometimes UIA doesn't actually make some thing accessible.

I did have one question for you please...

Is it possible to attach to an **existing** webpage, ie. rather then launch a new UIA instance rather attach to a current session and page?

Many thanks
R

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 12 Dec 2022, 10:34

@r2997790, I'm not sure what you mean when you say the "DIVS are not accessible". You mean you can't get the individual elements for the cells, or you mean that they can't be clicked with Click()? If you can get BoundingRectangles for the DIVS, then you can try using Click("left") to use AHK-s MouseClick to click (which it seems you already did), or ControlClick(). Why some elements are "accessible" and others are not is usually up to the program developers, and Chrome hasn't implemented UIA that well. I've found that Microsoft Edge has the best UIA support - perhaps try it there?

You can attach UIA_Browser to any open browser with any webpage open, you can use multiple UIA_Browsers, and you can use regular UIA_Interface at the same time as UIA_Browser. So if you had Facebook open in one window, Google open in a second window, and Notepad also opened, then this should work:

Code: Select all

WinActivate, Facebook
FB := new UIA_Browser("Facebook")
WinActivate, Google - Google
Google := new UIA_Browser("Google - Google")
UIA := UIA_Interface()

MsgBox, % FB.GetCurrentURL()
MsgBox, % Google.GetCurrentURL()
MsgBox, % UIA.ElementFromHandle("ahk_exe notepad.exe").Dump()

User avatar
marynofear
Posts: 4
Joined: 20 Oct 2018, 19:03

Re: UIAutomation with a focus on Chrome

Post by marynofear » 18 Dec 2022, 18:30

Descolada wrote:
10 Dec 2022, 06:28
@marynofear, there are already UIA libraries available for AHK v2, for example thqby's UIAutomation.ahk.
I do have plans of using his library as a base and convert all the "helper functions" (such as Click and Highlight) to AHK v2 as well, but progress is slow at the moment. I hope to have an usable version ready in the first half of next year, but no promises.
@Descolada
Thanks. Have now moved on to Acc library for AHK v2 viewtopic.php?f=83&t=107857

You forgot to mention this one.. :xmas:

Again here I'm thanksfull for all the great work your doing. :salute: >>>keep going - your one of the best<<<< :salute:
Had to make some small change to my V1 code with UIA, but got everything running with V2 rc3 and Acc v2.

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIAutomation with a focus on Chrome

Post by thaihoa3189 » 18 Dec 2022, 20:24

@Descolada how to paste or send Ctrl+V (keystroke) to an element please ?

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 19 Dec 2022, 01:18

@marynofear, I didn't mention it because Acc is not UIA 😉 Acc v2 does have Find methods and ways of clicking, but UIA is still much more powerful and the Find methods are faster. But if it solved your problem, great!

@thaihoa3189, UIA doesn't have a method for pasting. You can use element.SetFocus() to focus the element and then use ControlSend or Send to send Ctrl+v.
But usually you should be able to achieve the same with using the Value property to change the content of the element, and if needed you can even find the caret position. So a "pasting method" would use element.Value to get the current content, then TextPattern to find out the caret location, next you can insert the clipboard text into the caret location, and finally update the content with element.Value := newvalue. But easier to use Ctrl+v 😀

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIAutomation with a focus on Chrome

Post by thaihoa3189 » 19 Dec 2022, 04:51

@Descolada
Thank you for your tutorial. You're so nice. I only dont understand what is caret location and how to insert the clipboard text into the caret location, tell me please

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 19 Dec 2022, 11:00

I meant something like this:
Press F1 in Notepad to "paste" the current clipboard content. This may or may not work in other programs (for example it fails in Chrome, because TextPattern isn't properly implemented there).

Code: Select all

#include <UIA_Interface>
SetBatchLines, -1

lorem =
(
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
) ; Some sample text to play around with
Run, notepad.exe
WinWaitActive, ahk_exe notepad.exe
UIA := UIA_Interface()
	
NotepadEl := UIA.ElementFromHandle("ahk_exe notepad.exe")
NotepadEl.FindFirstByType("Document").Value := lorem

F1::
  el := UIA.GetFocusedElement()
  try {
  textPattern := el.GetCurrentPatternAs("TextPattern")
  selectedRange := textPattern.GetSelection()[1] ; Get the first TextRange (may be multiple if multiple selections are allowed, though this is rare)
  wholeRange := textPattern.DocumentRange ; For comparison, get the whole range (TextRange) of the document
  selectionStart := selectedRange.CompareEndpoints(UIA_Enum.TextPatternRangeEndpoint_Start, wholeRange, UIA_Enum.TextPatternRangeEndpoint_Start) ; Compare the start point of the selection to the start point of the whole document
  selectionEnd := selectedRange.CompareEndpoints(UIA_Enum.TextPatternRangeEndpoint_End, wholeRange, UIA_Enum.TextPatternRangeEndpoint_Start) ; Compare the end point of the selection to the start point of the whole document

  value := el.Value
  el.Value := SubStr(value, 1, selectionStart) Clipboard SubStr(value, selectionEnd+1)
  } catch
    MsgBox Sending the clipboard failed!
  return
Esc::ExitApp

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIAutomation with a focus on Chrome

Post by thaihoa3189 » 19 Dec 2022, 20:39

@Descolada
OK thank you very much. I understand it now.
One more issue with the UIA_Browser.ahk. That is when connecting to already existing Chrome tabs through remote debugging port 9222, it seems to only successfully connect to the active tab. When I try to set wtitle argument as a title of a inactive tab, it says failed to find the browser. See this picture:
image.png
image.png (71.91 KiB) Viewed 2739 times

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 20 Dec 2022, 00:05

@thaihoa3189, I think you might be confusing UIA with Chrome.ahk:
1) Chrome.ahk uses debugging port 9222 to connect to a new instance of Chrome that is running in debugging mode. With Chrome.ahk you can also run tasks in the background (when the window is inactive or invisible/headless), usually faster and more reliably since you can for example directly execute Javascript.
2) UIA_Browser.ahk is using UIAutomation to help automate Chrome and other browsers. It doesn't require Chrome in debugging mode (like Chrome.ahk), nor installing anything (like Rufaydium). Injecting Javascript in possible, but not the most reliable - it needs to send it through the URL bar, just like an user could. But it can't access elements that aren't visible - that means Chrome window has to be at least partially visible on the screen, and you can't connect to inactive tabs like shown in your attached image. You would need to attach to the active window/tab, and then switch tabs to the inactive one.

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIAutomation with a focus on Chrome

Post by thaihoa3189 » 20 Dec 2022, 09:21

@Descolada
Wow, your explanation made everything clear, your explanation was easy to understand, thank you very much.

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIAutomation with a focus on Chrome

Post by thaihoa3189 » 20 Dec 2022, 23:09

@Descolada
Descolada wrote: You would need to attach to the active window/tab, and then switch tabs to the inactive one.
Hey. I cant find the tab switch function in your UI Automation lib. How do i switch tabs to inactive one? Or i use command: Send, ^{Tab}

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 21 Dec 2022, 00:02

@thaihoa3189, you can use cUIA.SelectTab(tabName, matchMode:=3, caseSensitive:=True) for that, for example cUIA.SelectTab("Facebook", 2) which will select the first tab containing the word "Facebook". Of course you can use Send, ^{Tab} as well if you want.

thaihoa3189
Posts: 35
Joined: 23 Mar 2022, 22:10

Re: UIAutomation with a focus on Chrome

Post by thaihoa3189 » 21 Dec 2022, 05:28

@Descolada
Nice. Great library. Thank you.

r2997790
Posts: 71
Joined: 02 Feb 2017, 02:46

Re: UIAutomation with a focus on Chrome

Post by r2997790 » 21 Dec 2022, 07:42

Happy holidays and Merry Christmas to everyone!

I'm continuing to explore UIA and finally getting the hang of it, in huge part to the great insight shared by @Descolada Thank you!

Code: Select all

activebrowser.WaitElementExistByNameAndType(team, "HyperLink",, "RegEx").click()
I've hit a new interesting snag... I want to click the item which contains the string stored in the value 'team'. This works great UNLESS the element doesn't exist.

Is there a way, does anyone know, to use RegEx to see if a variable in included in a Control (Hyperlink) in this case BUT not Wait?
My issue is my script hangs/waits at this point expectantly, but sometimes the element doesn't exist (which is fine).

Ive tried added try {} and catch {} but it doesn't 'error' per se, it just does what it says it should ... Wait..and wait... and wait :D

I hope that makes sense. Can I use another UIA command which doesn't include Wait, to do a similar RegEx match against a string variable contained in a HyperLink control?

Many thanks
R

Descolada
Posts: 1098
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 21 Dec 2022, 07:59

@r2997790, one option is to set a timeout for the WaitElementExist and/or then check whether it was found or not:

Code: Select all

activebrowser.WaitElementExistByNameAndType(team, "HyperLink",, "RegEx", 500).Click() ; timeout 500ms, tries to click if found

If (el := activebrowser.WaitElementExistByNameAndType(team, "HyperLink",, "RegEx", 500))
    el.Click()
Or you can just look for the element once with FindFirstByNameAndType. Though WaitElementExist with a very short timeout (like 1ms) will effectively be the same.

Post Reply

Return to “Scripts and Functions (v1)”