UIAutomation with a focus on Chrome

Post your working scripts, libraries and tools for AHK v1.1 and older
Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 31 Jul 2022, 13:55

@mora145, leosouza85's answer should work. Perhaps you aren't setting your variable LanguagueTarget correctly?

mora145
Posts: 57
Joined: 25 Jun 2022, 15:31

Re: UIAutomation with a focus on Chrome

Post by mora145 » 31 Jul 2022, 14:04

I just tested, and yes, it works :) After testing for about 24 hours. It was the capital letter at the beginning that didn't match. Thanks @Descolada and @leosouza85

crocodile
Posts: 98
Joined: 28 Dec 2020, 13:41

Re: UIAutomation with a focus on Chrome

Post by crocodile » 02 Aug 2022, 08:59

If there is an image under the mouse, but it has no image menu to save. Can I use UIAutomation to save it locally? Thanks.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 02 Aug 2022, 10:06

@crocodile, AFAIK you cannot save UIAutomation image elements into files using UIAutomation. But you can get the exact position and size of the element with UIA, then use another library to take a screenshot of that area and save it. For example the Gdip library function Gdip_BitmapFromScreen does that.

crocodile
Posts: 98
Joined: 28 Dec 2020, 13:41

Re: UIAutomation with a focus on Chrome

Post by crocodile » 02 Aug 2022, 11:14

thank you very much.

kasandra
Posts: 2
Joined: 20 May 2022, 01:37

Re: UIAutomation with a focus on Chrome

Post by kasandra » 03 Aug 2022, 07:44

Hi,
I want to use this UIA library to automate some functions in MSPaint. I noted that it doesn't run from certain depth of this app, e.g. element by name "Obrót w prawo o 90°" (eng. "90° right rotation"). Do you have any solution for that case?

Code: Select all

#SingleInstance, Force
SendMode Input
SetWorkingDir, %A_ScriptDir%

#include %A_ScriptDir%\Lib\UIA_Interface.ahk
UIA := UIA_Interface() ; Initialize UIA interface
pI0 := UIA.ElementFromHandle(WinExist("ahk_exe mspaint.exe")) 
firststepE1 := pI0.FindFirstByNameAndType("Obróć", "SplitButton").Click()
secondstepE1 := pI0.WaitElementExistByName("Obrót w prawo o 90°")
secondstepE1.Click()
ExitApp
For your convenience, below is result of DumpAll method applied to UIA object. Note the "Obróć" name is visible.
Type: 50032 (Window) Name: "Bez tytułu - Paint" LocalizedControlType: "okno" ClassName: "MSPaintApp"
1 Type: 50033 (Pane) Name: "UIRibbonDockTop" LocalizedControlType: "okienko" ClassName: "UIRibbonCommandBarDock"
1.1 Type: 50033 (Pane) Name: "Ribbon" LocalizedControlType: "okienko" ClassName: "UIRibbonCommandBar"
1.1.1 Type: 50033 (Pane) Name: "Ribbon" LocalizedControlType: "okienko" ClassName: "UIRibbonWorkPane"
1.1.1.1 Type: 50033 (Pane) LocalizedControlType: "okienko" ClassName: "NUIPane"

1.1.1.1.1.7.1.1.2.3 Type: 50000 (Button) Name: "Zmień rozmiar" LocalizedControlType: "przycisk"
1.1.1.1.1.7.1.1.2.4 Type: 50031 (SplitButton) Name: "Obróć" LocalizedControlType: "przycisk podziału"
1.1.1.1.1.7.1.1.3 Type: 50021 (ToolBar) Name: "Narzędzia" Value: "Group" LocalizedControlType: "pasek narzędzi"
Kind regards, Kasandra

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 03 Aug 2022, 08:59

@kasandra, the menu item "Obrót w prawo o 90°" will not show up in DumpAll until it is actually visible: you need to put DumpAll after the line firststepE1 := pI0.FindFirstByNameAndType("Obróć", "SplitButton").Click(), or inspect it with an inspect tool like UIAViewer.

Your provided code worked perfectly in my computer, though I did have to use English names for buttons:

Code: Select all

firststepE1 := pI0.FindFirstByNameAndType("Rotate", "SplitButton").Click()
secondstepE1 := pI0.WaitElementExistByName("Rotate right 90°")
Note that you need to save the code in UTF-8-BOM encoding, otherwise the ° sign won't be recognized. Or alternatively avoid using ° in your code: secondstepE1 := pI0.WaitElementExistByName("Obrót w prawo o 90",,2)

kasandra
Posts: 2
Joined: 20 May 2022, 01:37

Re: UIAutomation with a focus on Chrome

Post by kasandra » 03 Aug 2022, 09:09

@Descolada Thanks for quick reply, it works correctly now 🙂

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 05 Aug 2022, 10:59

Here another challenge with probably a billion people desperately waiting in line to get it solved :mrgreen:

There are several command line tools that can cast local media files on a Google Chromecast enabled device, but I haven’t seen any that will establish the connection to simply display the PC's desktop instead (ie to show a livestream within a fullscreen browser session).
Here's an attempt to do this manually, but I'd guess it'll be a hell smarter using UIA… :)

https://stackoverflow.com/a/67202498

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 05 Aug 2022, 13:42

@BoBo, I don't have experience with Chromecasting, but gave it a shot anyway. Couldn't debug it thoroughly (only the "Cast tab" option was available), also I have no idea why it has to be done in a new Chrome window.

Code: Select all

#SingleInstance, Force
SetBatchLines, -1

#include <UIA_Interface>
UIA := UIA_Interface()

RunNewChrome := False
CastMode := "Cast tab" ; Cast tab, Cast screen, Cast file
DeviceName := "My device (case sensitive, matching partial names)"

!#c::
    if RunNewChrome
        Run, chrome.exe --fullscreen --start-maximized
    WinWaitActive, ahk_exe chrome.exe
    chrome := UIA.ElementFromHandle("ahk_exe chrome.exe")
    chrome.FindFirstBynameAndType("Chrome", "MenuItem").Click()
    chrome.WaitElementExistByNameAndType("Cast…", "MenuItem",,2).Click()
    castTab := chrome.WaitElementExistByNameAndType("Cast ", "Pane",,1)
    castTab.FindFirstByNameAndType("Sources", "Button").Click()
    castTab.WaitElementExistByNameAndType(CastMode, "MenuItem").Click()
    castTab.FindFirstByNameAndType(DeviceName, "Button",,2).Click()
    if RunNewChrome
        WinClose, ahk_exe chrome.exe
    return
Last edited by Descolada on 07 Aug 2022, 01:38, edited 1 time in total.

malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIAutomation with a focus on Chrome

Post by malcev » 05 Aug 2022, 14:51

Need to write chrome extension, that informs autohotkey about needed sites and then from autohotkey You can start cast for example with this api:
https://chromedevtools.github.io/devtools-protocol/tot/Cast/

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 06 Aug 2022, 05:20

@Descolada - unfortunately it's throwing an error.
I've updated Chrome to its latest release: Version 104.0.5112.81...
executed the script with an active Chrome instance - and no instance.
Ran the script with RunNewChrome := false ;and true...
to no avail *sigh*

Spoiler

@malcev would love to see that in action, BUT I wouldn't be able to sort this out on my own, so ending up torturing a fellow supporter with a bunch of stupid questions. Not that attractive perspective, right? :shifty:


Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 06 Aug 2022, 05:55

@BoBo, I was using a relatively new syntax of UIA_Interface/UIA_Browser: chrome := UIA.ElementFromHandle("ahk_exe chrome.exe") instead of chrome := UIA.ElementFromHandle(WinExist("ahk_exe chrome.exe")). Try changing that line, or updating UIA_Interface (and UIA_Browser).

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 06 Aug 2022, 10:06

@Descolada - got it. Pushed it into a function.
Question: is there a chance to extract the current "cast mode" setting/label from this pane for further validation?

castTab := chrome.WaitElementExistByNameAndType("Cast ", "Pane",,1)

... because it's possible using that same function to toggle the connection to a device, but only if that label hasn't changed meanwhile (ie by manual intervention).

Code: Select all

#SingleInstance, Force
SetBatchLines, -1
#include <UIA_Interface>

Global en, de

en := ["Cast…","Sources","Cast tab","Cast screen","Cast file"]
de := ["Streamen…","Quellen","Tab streamen","Bildschirm streamen","Datei streamen"]

!c::cast("kitchen")														   ;press ALT+c to connect/cast to an active chromecast device

cast(myDevice:="living room",mode:="en",RunNewChrome:=false) {             ;mode:="de" ;myDevice (case sensitive, matching partial names)
   UIA := UIA_Interface()
   if RunNewChrome
      Run, chrome.exe --fullscreen --start-maximized                       ;--lang=en-US,en
   WinWaitActive, ahk_exe chrome.exe
   chrome := UIA.ElementFromHandle("ahk_exe chrome.exe")
   chrome.FindFirstBynameAndType("Chrome", "MenuItem").Click()
   chrome.WaitElementExistByNameAndType(%mode%[1], "MenuItem",,2).Click()  ;selecting Chrome's streaming option

   castTab := chrome.WaitElementExistByNameAndType(%mode%[3], "Pane",,1)   ;waiting till selection pane occurs
   castTab.FindFirstByNameAndType(%mode%[2], "Button").Click()             ;activating 'Sources' drop down list
   castTab.FindFirstByNameAndType(%mode%[3], "MenuItem",,2).Click()        ;selecting casting mode 3=tab/4=screen/5=file
   castTab.FindFirstByNameAndType(myDevice, "Button",,2).Click()           ;selecting device
   if RunNewChrome
      WinClose, ahk_exe chrome.exe
   return
   }
@malcev - I'll have a look at it. Thx a lot (and no, I wouldn't interfere if you would play with it meanwhile) :shh: :lol:

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 07 Aug 2022, 01:41

@BoBo, I think you can check whether the device is available or not by the second line of its name: if InStr(StrSplit(castTab.FindFirstByNameAndType(DeviceName, "Button",,2).CurrentName, "`n")[2], "Available") { ... }.

Edit: forgot to get the property of "Name" ;)
Last edited by Descolada on 07 Aug 2022, 11:39, edited 1 time in total.

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 07 Aug 2022, 06:08

@Descolada - not working (at least for me)

Code: Select all

myDevice:="living room"
;
;
;
MsgBox % castTab.FindFirstByNameAndType(myDevice, "Button",,2)                                           ;empty
MsgBox % StrSplit(castTab.FindFirstByNameAndType(myDevice, "Button",,2), "`n")[2]                        ;spliting emptyness
MsgBox % InStr(StrSplit(castTab.FindFirstByNameAndType("myDevice", "Button",,2), "`n")[2], "Available")  ;0 not changing bc of the above
:think:

The challenge, I need to identify that cast mode to be able to toggle/switch off a currently running session in case I wanna change the cast mode from 'screen' to 'tab' or vice versa :arrow: if done for the same device.
If I change the cast mode and the device at the same time it'll simply terminate the running session.

Interesting side fact: you can cast a Chrome "tab" but not a "screen" to a screenless device (like the Google Home Mini) :wtf:

Descolada:
Couldn't debug it thoroughly (only the "Cast tab" option was available),...
Isn't that depending on the source application that has the focus at the time you initialize the cast??
...also I have no idea why it has to be done in a new Chrome window.
Weird, as I can use an already YT/AHK/whatever loaded Chrome tab and it connects fine.

-----

Edit: Looks like I can access the above mentioned details using this... :o :thumbup:
castTab.GetCurrentPropertyValue("Name") ;we can provide the property name to get the value
Source: https://github.com/Descolada/UIAutomation/wiki/04.-Elements

PS. do I remember it correctly that all of these UIA events can't be executed 'hidden'-style?

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: UIAutomation with a focus on Chrome

Post by Descolada » 07 Aug 2022, 11:52

@BoBo, I'm so sorry, in a hurry it seems I omitted the Name property of the element :crazy: I changed my last post to reflect that: InStr(StrSplit(castTab.FindFirstByNameAndType(DeviceName, "Button",,2).CurrentName, "`n")[2], "Available") is correct.

Regarding the current cast mode:
1) Your option of castTab.GetCurrentPropertyValue("Name") works, gets the pane name. This is equivalent to castTab.CurrentName, and in newer UIA_Interface versions also castTab.Name
2) Second option would be to get the TitleBar name: castTab.FindFirstByType("TitleBar").CurrentName
3) Third option is to get the toggle state of the menu item button:

Code: Select all

castTab.FindFirstByNameAndType("Sources", "Button").Click()
castTab.WaitElementExistByNameAndType(CastMode, "MenuItem").GetCurrentPropertyValue("ToggleToggleState")

results in 1 for checked, 0 for not checked.
Isn't that depending on the source application that has the focus at the time you initialize the cast??
Yeah, I have no idea. :D As I said, I haven't used Chromecast and I only managed to get it working in YouTube...
Weird, as I can use an already YT/AHK/whatever loaded Chrome tab and it connects fine.
Yes, it works for me too. I was referring to the StackOverflow post you originally posted (https://stackoverflow.com/a/67202498) which did that for whatever reason.
do I remember it correctly that all of these UIA events can't be executed 'hidden'-style?
Best to do it when Chrome is active, but you can try it out in unfocused mode. For that remove the WinWaitActive line and set AutoSetFocus to False:

Code: Select all

UIA := UIA_Interface()
UIA.AutoSetFocus := False
Might not be reliable though, so caution is advised ;)

BoBo
Posts: 6564
Joined: 13 May 2014, 17:15

Re: UIAutomation with a focus on Chrome

Post by BoBo » 07 Aug 2022, 12:32

@Descolada - Thx a lot for your explanations, much appreciated. Stay tuned :)

TomDonovan
Posts: 4
Joined: 25 Dec 2020, 11:57

Re: UIAutomation with a focus on Chrome

Post by TomDonovan » 09 Aug 2022, 15:26

Does anyone have a sample of creating an event to monitor for a change in a UIA Property

In Outlook desktop, there are tabs across the top. Home tab is the one I am interested in. I am interested in the IsSelected Property

A real simplification would be

OutlookMainID := WinExist("ahk_exe outlook.exe")
elementMain := uia.ElementFromHandle(OutlookMainID)
homeTab := elementMain.FindFirstByNameAndType("Home","TabItem")
SelectionPattern := HomeTab.GetCurrentPatternAs("SelectionItemPattern")
if SelectionPattern.IsSelected
make my gui visible
else
hide my gui

I can do this thru a timer, but think it would be more efficient to have an event handler notify me with IsSelected changes. From Accessibility Insights, I know the event is raised - Property ID 30079, SelectionItemPattern.IsSelected is what shows in Accessibility Insights.

But I haven't figure out how to connect to the even.

Any and all help (or kicks in the butt if this is more obvious than I am seeing) appreciated.

Post Reply

Return to “Scripts and Functions (v1)”