UIAutomation with a focus on Chrome
Re: UIAutomation with a focus on Chrome
@mora145, leosouza85's answer should work. Perhaps you aren't setting your variable LanguagueTarget correctly?
Re: UIAutomation with a focus on Chrome
I just tested, and yes, it works After testing for about 24 hours. It was the capital letter at the beginning that didn't match. Thanks @Descolada and @leosouza85
Re: UIAutomation with a focus on Chrome
If there is an image under the mouse, but it has no image menu to save. Can I use UIAutomation to save it locally? Thanks.
Re: UIAutomation with a focus on Chrome
@crocodile, AFAIK you cannot save UIAutomation image elements into files using UIAutomation. But you can get the exact position and size of the element with UIA, then use another library to take a screenshot of that area and save it. For example the Gdip library function Gdip_BitmapFromScreen does that.
Re: UIAutomation with a focus on Chrome
thank you very much.
Re: UIAutomation with a focus on Chrome
Hi,
I want to use this UIA library to automate some functions in MSPaint. I noted that it doesn't run from certain depth of this app, e.g. element by name "Obrót w prawo o 90°" (eng. "90° right rotation"). Do you have any solution for that case?
For your convenience, below is result of DumpAll method applied to UIA object. Note the "Obróć" name is visible.
I want to use this UIA library to automate some functions in MSPaint. I noted that it doesn't run from certain depth of this app, e.g. element by name "Obrót w prawo o 90°" (eng. "90° right rotation"). Do you have any solution for that case?
Code: Select all
#SingleInstance, Force
SendMode Input
SetWorkingDir, %A_ScriptDir%
#include %A_ScriptDir%\Lib\UIA_Interface.ahk
UIA := UIA_Interface() ; Initialize UIA interface
pI0 := UIA.ElementFromHandle(WinExist("ahk_exe mspaint.exe"))
firststepE1 := pI0.FindFirstByNameAndType("Obróć", "SplitButton").Click()
secondstepE1 := pI0.WaitElementExistByName("Obrót w prawo o 90°")
secondstepE1.Click()
ExitApp
Kind regards, KasandraType: 50032 (Window) Name: "Bez tytułu - Paint" LocalizedControlType: "okno" ClassName: "MSPaintApp"
1 Type: 50033 (Pane) Name: "UIRibbonDockTop" LocalizedControlType: "okienko" ClassName: "UIRibbonCommandBarDock"
1.1 Type: 50033 (Pane) Name: "Ribbon" LocalizedControlType: "okienko" ClassName: "UIRibbonCommandBar"
1.1.1 Type: 50033 (Pane) Name: "Ribbon" LocalizedControlType: "okienko" ClassName: "UIRibbonWorkPane"
1.1.1.1 Type: 50033 (Pane) LocalizedControlType: "okienko" ClassName: "NUIPane"
…
1.1.1.1.1.7.1.1.2.3 Type: 50000 (Button) Name: "Zmień rozmiar" LocalizedControlType: "przycisk"
1.1.1.1.1.7.1.1.2.4 Type: 50031 (SplitButton) Name: "Obróć" LocalizedControlType: "przycisk podziału"
1.1.1.1.1.7.1.1.3 Type: 50021 (ToolBar) Name: "Narzędzia" Value: "Group" LocalizedControlType: "pasek narzędzi"
…
Re: UIAutomation with a focus on Chrome
@kasandra, the menu item "Obrót w prawo o 90°" will not show up in DumpAll until it is actually visible: you need to put DumpAll after the line firststepE1 := pI0.FindFirstByNameAndType("Obróć", "SplitButton").Click(), or inspect it with an inspect tool like UIAViewer.
Your provided code worked perfectly in my computer, though I did have to use English names for buttons:
Note that you need to save the code in UTF-8-BOM encoding, otherwise the ° sign won't be recognized. Or alternatively avoid using ° in your code: secondstepE1 := pI0.WaitElementExistByName("Obrót w prawo o 90",,2)
Your provided code worked perfectly in my computer, though I did have to use English names for buttons:
Code: Select all
firststepE1 := pI0.FindFirstByNameAndType("Rotate", "SplitButton").Click()
secondstepE1 := pI0.WaitElementExistByName("Rotate right 90°")
Re: UIAutomation with a focus on Chrome
@Descolada Thanks for quick reply, it works correctly now
Re: UIAutomation with a focus on Chrome
Here another challenge with probably a billion people desperately waiting in line to get it solved
There are several command line tools that can cast local media files on a Google Chromecast enabled device, but I haven’t seen any that will establish the connection to simply display the PC's desktop instead (ie to show a livestream within a fullscreen browser session).
Here's an attempt to do this manually, but I'd guess it'll be a hell smarter using UIA…
https://stackoverflow.com/a/67202498
There are several command line tools that can cast local media files on a Google Chromecast enabled device, but I haven’t seen any that will establish the connection to simply display the PC's desktop instead (ie to show a livestream within a fullscreen browser session).
Here's an attempt to do this manually, but I'd guess it'll be a hell smarter using UIA…
https://stackoverflow.com/a/67202498
Re: UIAutomation with a focus on Chrome
@BoBo, I don't have experience with Chromecasting, but gave it a shot anyway. Couldn't debug it thoroughly (only the "Cast tab" option was available), also I have no idea why it has to be done in a new Chrome window.
Code: Select all
#SingleInstance, Force
SetBatchLines, -1
#include <UIA_Interface>
UIA := UIA_Interface()
RunNewChrome := False
CastMode := "Cast tab" ; Cast tab, Cast screen, Cast file
DeviceName := "My device (case sensitive, matching partial names)"
!#c::
if RunNewChrome
Run, chrome.exe --fullscreen --start-maximized
WinWaitActive, ahk_exe chrome.exe
chrome := UIA.ElementFromHandle("ahk_exe chrome.exe")
chrome.FindFirstBynameAndType("Chrome", "MenuItem").Click()
chrome.WaitElementExistByNameAndType("Cast…", "MenuItem",,2).Click()
castTab := chrome.WaitElementExistByNameAndType("Cast ", "Pane",,1)
castTab.FindFirstByNameAndType("Sources", "Button").Click()
castTab.WaitElementExistByNameAndType(CastMode, "MenuItem").Click()
castTab.FindFirstByNameAndType(DeviceName, "Button",,2).Click()
if RunNewChrome
WinClose, ahk_exe chrome.exe
return
Last edited by Descolada on 07 Aug 2022, 01:38, edited 1 time in total.
Re: UIAutomation with a focus on Chrome
Need to write chrome extension, that informs autohotkey about needed sites and then from autohotkey You can start cast for example with this api:
https://chromedevtools.github.io/devtools-protocol/tot/Cast/
https://chromedevtools.github.io/devtools-protocol/tot/Cast/
Re: UIAutomation with a focus on Chrome
@Descolada - unfortunately it's throwing an error.
I've updated Chrome to its latest release: Version 104.0.5112.81...
executed the script with an active Chrome instance - and no instance.
Ran the script with RunNewChrome := false ;and true...
to no avail *sigh*
@malcev would love to see that in action, BUT I wouldn't be able to sort this out on my own, so ending up torturing a fellow supporter with a bunch of stupid questions. Not that attractive perspective, right?
I've updated Chrome to its latest release: Version 104.0.5112.81...
executed the script with an active Chrome instance - and no instance.
Ran the script with RunNewChrome := false ;and true...
to no avail *sigh*
Spoiler
@malcev would love to see that in action, BUT I wouldn't be able to sort this out on my own, so ending up torturing a fellow supporter with a bunch of stupid questions. Not that attractive perspective, right?
Re: UIAutomation with a focus on Chrome
@BoBo, I was using a relatively new syntax of UIA_Interface/UIA_Browser: chrome := UIA.ElementFromHandle("ahk_exe chrome.exe") instead of chrome := UIA.ElementFromHandle(WinExist("ahk_exe chrome.exe")). Try changing that line, or updating UIA_Interface (and UIA_Browser).
Re: UIAutomation with a focus on Chrome
@Descolada - got it. Pushed it into a function.
Question: is there a chance to extract the current "cast mode" setting/label from this pane for further validation?
castTab := chrome.WaitElementExistByNameAndType("Cast ", "Pane",,1)
... because it's possible using that same function to toggle the connection to a device, but only if that label hasn't changed meanwhile (ie by manual intervention).
@malcev - I'll have a look at it. Thx a lot (and no, I wouldn't interfere if you would play with it meanwhile)
Question: is there a chance to extract the current "cast mode" setting/label from this pane for further validation?
castTab := chrome.WaitElementExistByNameAndType("Cast ", "Pane",,1)
... because it's possible using that same function to toggle the connection to a device, but only if that label hasn't changed meanwhile (ie by manual intervention).
Code: Select all
#SingleInstance, Force
SetBatchLines, -1
#include <UIA_Interface>
Global en, de
en := ["Cast…","Sources","Cast tab","Cast screen","Cast file"]
de := ["Streamen…","Quellen","Tab streamen","Bildschirm streamen","Datei streamen"]
!c::cast("kitchen") ;press ALT+c to connect/cast to an active chromecast device
cast(myDevice:="living room",mode:="en",RunNewChrome:=false) { ;mode:="de" ;myDevice (case sensitive, matching partial names)
UIA := UIA_Interface()
if RunNewChrome
Run, chrome.exe --fullscreen --start-maximized ;--lang=en-US,en
WinWaitActive, ahk_exe chrome.exe
chrome := UIA.ElementFromHandle("ahk_exe chrome.exe")
chrome.FindFirstBynameAndType("Chrome", "MenuItem").Click()
chrome.WaitElementExistByNameAndType(%mode%[1], "MenuItem",,2).Click() ;selecting Chrome's streaming option
castTab := chrome.WaitElementExistByNameAndType(%mode%[3], "Pane",,1) ;waiting till selection pane occurs
castTab.FindFirstByNameAndType(%mode%[2], "Button").Click() ;activating 'Sources' drop down list
castTab.FindFirstByNameAndType(%mode%[3], "MenuItem",,2).Click() ;selecting casting mode 3=tab/4=screen/5=file
castTab.FindFirstByNameAndType(myDevice, "Button",,2).Click() ;selecting device
if RunNewChrome
WinClose, ahk_exe chrome.exe
return
}
Re: UIAutomation with a focus on Chrome
@BoBo, I think you can check whether the device is available or not by the second line of its name: if InStr(StrSplit(castTab.FindFirstByNameAndType(DeviceName, "Button",,2).CurrentName, "`n")[2], "Available") { ... }.
Edit: forgot to get the property of "Name"
Edit: forgot to get the property of "Name"
Last edited by Descolada on 07 Aug 2022, 11:39, edited 1 time in total.
Re: UIAutomation with a focus on Chrome
@Descolada - not working (at least for me)
The challenge, I need to identify that cast mode to be able to toggle/switch off a currently running session in case I wanna change the cast mode from 'screen' to 'tab' or vice versa if done for the same device.
If I change the cast mode and the device at the same time it'll simply terminate the running session.
Interesting side fact: you can cast a Chrome "tab" but not a "screen" to a screenless device (like the Google Home Mini)
Descolada:
-----
Edit: Looks like I can access the above mentioned details using this...
castTab.GetCurrentPropertyValue("Name") ;we can provide the property name to get the value
Source: https://github.com/Descolada/UIAutomation/wiki/04.-Elements
PS. do I remember it correctly that all of these UIA events can't be executed 'hidden'-style?
Code: Select all
myDevice:="living room"
;
;
;
MsgBox % castTab.FindFirstByNameAndType(myDevice, "Button",,2) ;empty
MsgBox % StrSplit(castTab.FindFirstByNameAndType(myDevice, "Button",,2), "`n")[2] ;spliting emptyness
MsgBox % InStr(StrSplit(castTab.FindFirstByNameAndType("myDevice", "Button",,2), "`n")[2], "Available") ;0 not changing bc of the above
The challenge, I need to identify that cast mode to be able to toggle/switch off a currently running session in case I wanna change the cast mode from 'screen' to 'tab' or vice versa if done for the same device.
If I change the cast mode and the device at the same time it'll simply terminate the running session.
Interesting side fact: you can cast a Chrome "tab" but not a "screen" to a screenless device (like the Google Home Mini)
Descolada:
Isn't that depending on the source application that has the focus at the time you initialize the cast??Couldn't debug it thoroughly (only the "Cast tab" option was available),...
Weird, as I can use an already YT/AHK/whatever loaded Chrome tab and it connects fine....also I have no idea why it has to be done in a new Chrome window.
-----
Edit: Looks like I can access the above mentioned details using this...
castTab.GetCurrentPropertyValue("Name") ;we can provide the property name to get the value
Source: https://github.com/Descolada/UIAutomation/wiki/04.-Elements
PS. do I remember it correctly that all of these UIA events can't be executed 'hidden'-style?
Re: UIAutomation with a focus on Chrome
@BoBo, I'm so sorry, in a hurry it seems I omitted the Name property of the element I changed my last post to reflect that: InStr(StrSplit(castTab.FindFirstByNameAndType(DeviceName, "Button",,2).CurrentName, "`n")[2], "Available") is correct.
Regarding the current cast mode:
1) Your option of castTab.GetCurrentPropertyValue("Name") works, gets the pane name. This is equivalent to castTab.CurrentName, and in newer UIA_Interface versions also castTab.Name
2) Second option would be to get the TitleBar name: castTab.FindFirstByType("TitleBar").CurrentName
3) Third option is to get the toggle state of the menu item button:
results in 1 for checked, 0 for not checked.
Might not be reliable though, so caution is advised
Regarding the current cast mode:
1) Your option of castTab.GetCurrentPropertyValue("Name") works, gets the pane name. This is equivalent to castTab.CurrentName, and in newer UIA_Interface versions also castTab.Name
2) Second option would be to get the TitleBar name: castTab.FindFirstByType("TitleBar").CurrentName
3) Third option is to get the toggle state of the menu item button:
Code: Select all
castTab.FindFirstByNameAndType("Sources", "Button").Click()
castTab.WaitElementExistByNameAndType(CastMode, "MenuItem").GetCurrentPropertyValue("ToggleToggleState")
results in 1 for checked, 0 for not checked.
Yeah, I have no idea. As I said, I haven't used Chromecast and I only managed to get it working in YouTube...Isn't that depending on the source application that has the focus at the time you initialize the cast??
Yes, it works for me too. I was referring to the StackOverflow post you originally posted (https://stackoverflow.com/a/67202498) which did that for whatever reason.Weird, as I can use an already YT/AHK/whatever loaded Chrome tab and it connects fine.
Best to do it when Chrome is active, but you can try it out in unfocused mode. For that remove the WinWaitActive line and set AutoSetFocus to False:do I remember it correctly that all of these UIA events can't be executed 'hidden'-style?
Code: Select all
UIA := UIA_Interface()
UIA.AutoSetFocus := False
Re: UIAutomation with a focus on Chrome
@Descolada - Thx a lot for your explanations, much appreciated. Stay tuned
-
- Posts: 4
- Joined: 25 Dec 2020, 11:57
Re: UIAutomation with a focus on Chrome
Does anyone have a sample of creating an event to monitor for a change in a UIA Property
In Outlook desktop, there are tabs across the top. Home tab is the one I am interested in. I am interested in the IsSelected Property
A real simplification would be
OutlookMainID := WinExist("ahk_exe outlook.exe")
elementMain := uia.ElementFromHandle(OutlookMainID)
homeTab := elementMain.FindFirstByNameAndType("Home","TabItem")
SelectionPattern := HomeTab.GetCurrentPatternAs("SelectionItemPattern")
if SelectionPattern.IsSelected
make my gui visible
else
hide my gui
I can do this thru a timer, but think it would be more efficient to have an event handler notify me with IsSelected changes. From Accessibility Insights, I know the event is raised - Property ID 30079, SelectionItemPattern.IsSelected is what shows in Accessibility Insights.
But I haven't figure out how to connect to the even.
Any and all help (or kicks in the butt if this is more obvious than I am seeing) appreciated.
In Outlook desktop, there are tabs across the top. Home tab is the one I am interested in. I am interested in the IsSelected Property
A real simplification would be
OutlookMainID := WinExist("ahk_exe outlook.exe")
elementMain := uia.ElementFromHandle(OutlookMainID)
homeTab := elementMain.FindFirstByNameAndType("Home","TabItem")
SelectionPattern := HomeTab.GetCurrentPatternAs("SelectionItemPattern")
if SelectionPattern.IsSelected
make my gui visible
else
hide my gui
I can do this thru a timer, but think it would be more efficient to have an event handler notify me with IsSelected changes. From Accessibility Insights, I know the event is raised - Property ID 30079, SelectionItemPattern.IsSelected is what shows in Accessibility Insights.
But I haven't figure out how to connect to the even.
Any and all help (or kicks in the butt if this is more obvious than I am seeing) appreciated.