Interacting with DaVinci using UIAutomation

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 07 Nov 2022, 02:08

Thanks a lot for your help guys :thumbup: It's finally working for me :dance:

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 04 Dec 2022, 20:29

@Descolada Hey there,
Is there a way to get rid of Error Message Box's everytime UIA can't find the window?
Maybe If I adjust the funciton to turn off error message?
Cause I use it for DaVinci Resolve I press about 30 hotkeys a minute and I get error messages from time to time even though DaVinci Resaolve Still EXISTS.
Image
Attachments
2022_12_05@11-34AM.PNG
[email protected] (26.6 KiB) Viewed 2200 times

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 05 Dec 2022, 00:29

@SteveMylo, as to my knowledge COR_E_TIMEOUT will be thrown when ConnectionTimeout or TransactionTimeout happens. This likely means that Davinci is busy processing something else and not ready for user input.

One thing you could do is increase ConnectionTimeout (by default is 2 seconds) and see whether that fixes your error. Other way would be to catch that error and try again.

If using multiple hotkeys that need to interact with Davinci, I would try to avoid calling the slow ElementFromHandle multiple times. Instead I would for example create a function that saves and returns the element:

Code: Select all

GetDavinciElement() {
    static davinciId, davinciElement
    
    if (!(dvId := WinExist("ahk_exe davinci.exe")))
    	return
    if ((davinciId == dvId) && davinciElement)
        return davinciElement
    davinciElement := "", davinciId := dvId
    while (!davinciElement && WinExist("ahk_exe davinci.exe")) {
        try davinciElement := UIA_Interface().ElementFromHandle(dvId)
    }
    return davinciElement
}
This way you know that if GetDavinciElement returns something, then the Davinci window actually exists and ElementFromHandle was successful, so you don't need to check these things every time. Also this will speed up your code by avoiding getting the window element all over again every time you use a hotkey.

07.12.22 EDIT: fixed mistake of "ahk_id davinci.exe" instead of "ahk_exe davinci.exe"
Last edited by Descolada on 07 Dec 2022, 00:06, edited 1 time in total.

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 05 Dec 2022, 08:01

Thanks. Before I asked you, I was playing around with the timeout feature but I couldn’t get it to work.

So I just tried your Function on Wordpad.exe as i’m away till Wednesday & it seems to be working great!
The function seems simple & just what I need.
Thank you.

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 06 Dec 2022, 16:01

Hey @Descolada
Thanks a lot for sharing that function. I also use multiple hotkeys in Davinci.
Could you please explain a bit about how to use it?
Do I need to call it in each hotkey? Or just once at the beginning of my script file?
Sorry for dummy questions, I just recently started to use AHK.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 07 Dec 2022, 00:15

@k0stell0, that is a (untested) function that simply checks whether DaVinci window exists, and if it does then it either returns a previously found and saved UIA element for the DaVinci window (because the element doesn't change or get destroyed until the window does, that is until the ahk_id of the window changes) or finds, saves and returns the UIA element for DaVinci. You could do this for any program in fact, and for example UIA_Browser does pretty much the same internally.

You can use it the same as you would've ElementFromHandle. Instead of

Code: Select all

if (!WinExist("ahk_exe davinci.exe"))
    ; DaVinci window doesn't exist, run it?
if !(dvEl := UIA.ElementFromHandle("ahk_exe davinci.exe"))
    ; DaVinci element wasn't returned - something went wrong? Window closed in the meanwhile?
; Otherwise the UIA element was found and is ready to be used
you can do

Code: Select all

if !(dvEl := GetDavinciElement())
    ; DaVinci window UIA element wasn't returned, which probably means that the window doesn't exist - run it?
; Element found, do stuff

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 07 Dec 2022, 04:57

k0stell0 wrote:
06 Dec 2022, 16:01
Do I need to call it in each hotkey? Or just once at the beginning of my script file?
To save Descolada some time I'll reply. It works for me but Descolada is the expert here not me.
To answer your question, Only once for EACH script only.
Here is an example of how to click on Media Pool and Inspector.
If you're going to play around in Davinci , not everything lets you click, so with the 'Inspector tab' , I show you how to manually MOUSEclick on something with UIA

Code: Select all

#NoEnv
SetWorkingDir %A_ScriptDir%
CoordMode, Mouse, Screen
SendMode Input
#SingleInstance, Force
SetTitleMatchMode 2
SetTitleMatchMode, Fast
SetControlDelay -1
SetWinDelay -1
SetKeyDelay -1
SetMouseDelay 1
SetBatchLines -1
ListLines, Off
#Persistent
#include <UIA_Interface>

GetDavinciElement()  ; this runs just once as soon script is loaded
if !(dvEl := GetDavinciElement())  ; if Davinci not exist, do nothing
return

d::
;~ UIA := UIA_Interface()   ; I don't think we need this
dvEl.FindFirstByName("Media Pool").Click()
return

b::
Inspector := dvEl.FindFirstByName("Inspector")
; not everything lets you click, so this is how you MOUSEclick on something with UIA
Coords := Inspector.CurrentBoundingRectangle
MouseClick, Left, (Coords.l+((Coords.r-Coords.l)/ 2)) , (Coords.t + ((Coords.b-Coords.t)/2)), 1, 9
return

~Esc::ExitApp

;;=======================   FUNCTION   ===============================

GetDavinciElement() {
    static davinciId, davinciElement

    if (!(dvId := WinExist("ahk_exe Resolve.exe")))
    	return
    if ((davinciId == dvId) && davinciElement)
        return davinciElement
    else {
        davinciElement := "", davinciId := dvId
        while (!davinciElement && WinExist("ahk_exe Resolve.exe")) {
            try davinciElement := UIA_Interface().ElementFromHandle(dvId)
        }
        return davinciElement
    }
}







Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 07 Dec 2022, 05:37

@SteveMylo, @k0stell0, actually you can, and probably should, use it on every call of the hotkey. This is because if you close Davinci in the meanwhile and run it again, the script will still work of you call it with every hotkey call. If you call it again, then the old element will still be in dvEl and will not work. Also yes, calling UIA_interface is then also not needed.

This is untested code, but should work even if you close and reopen Davinci:

Code: Select all

#NoEnv
SetWorkingDir %A_ScriptDir%
CoordMode, Mouse, Screen
SendMode Input
#SingleInstance, Force
SetTitleMatchMode 2
SetTitleMatchMode, Fast
SetControlDelay -1
SetWinDelay -1
SetKeyDelay -1
SetMouseDelay 1
SetBatchLines -1
ListLines, Off
#Persistent
#include <UIA_Interface>

d::
if !(dvEl := GetDavinciElement())  ; if Davinci not exist, do nothing
return

dvEl.FindFirstByName("Media Pool").Click() 
dvEl.FindFirstByName("Inspector").Click("left")
; not everything lets you click, so this is how you MOUSEclick on something with UIA
return
~Esc::ExitApp

;;=======================   FUNCTION   ===============================

GetDavinciElement() {
    static davinciId, davinciElement

    if (!(dvId := WinExist("ahk_exe Resolve.exe")))
    	return
    if ((davinciId == dvId) && davinciElement)
        return davinciElement
    else {
        davinciElement := "", davinciId := dvId
        while (!davinciElement && WinExist("ahk_exe Resolve.exe")) {
            try davinciElement := UIA_Interface().ElementFromHandle(dvId)
        }
        return davinciElement
}

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 07 Dec 2022, 05:51

@Descolada That works a treat. Well done and thanks. :dance:

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 08 Dec 2022, 03:17

Thanks a lot for your help guys. I learned so much from you.

If it's not too much trouble, can you please help me with the code below?
Davinci has a lot of checkboxes I need to click on, so I was hoping to check their current state before clicking. Otherwise, I can disable something instead of enabling it.
I tried googling it, but without any success. Here's the code I came up with, but it's obviously not working:

Code: Select all

MediaTab := UIA_Interface().ElementFromHandle(WinExist("ahk_exe Resolve.exe"))
Metadata := MediaTab.FindFirstByNameAndType("Metadata", "checkbox").ToggleState

If (Matadata == True) {
MsgBox, Matadata panel is visible
}
else {
MsgBox,  Matadata panel isn't visible
}
Is this even possible? If so, what's the right way of doing it?

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 08 Dec 2022, 20:41

k0stell0 wrote:Davinci has a lot of checkboxes I need to click on,
G'day....It's complex cause there are sooo many checkboxes .
But the good news is, as long as your METADATA panel is open, you can toggle a particular check box on! It doesn't have to be visible cause it's toggleable. But as long as it exists in the panel.
-------So for the code below, I manually clicked on the metadata Panel first in the EDIT page, So I search that area with it's Automation Id.
This narrows the entire checkbox confined to the MetadataPanel area, making your life so much easier.
If your metadata Panel is on the MEDIA page or elsewhere then of course you will need a different Automation Id.
You kinda have to do this 1st cause there are probably a thousand Checkboxes in DaVinci Resolve
I also clicked on all groups to show ALL the checkboxes for the purposes of testing my code.

--------Then you FindFirstByName what ever you want to toggle, in my example it's "Good Take" See pic attached
Good take doesn't exist in 'Sort By Clip Details' but it exists in 'ALL GROUPS' , so be careful, the code below only works if Good Take exists in the panel. But doesn't have to be visible.
So for every checkbox you want to check, you need to know the name to the left of the check box and copy and paste the script again and repeat.
-------But the important part is creating a "TreeWalker". you don't have to understand them just copy my code. But they help find the next 'sibling' next to something.
Where I have written "tw" means TreeWalker
------Then below the TW code is a TOGGLE code for checking and toggling things on/Off

Work hard at knowing why the code is written the way it is, and try coding another toggle checkbox below it, and you'll be on your way.

Image

Code: Select all

#include <UIA_Interface>
d::
UIA := UIA_Interface()
dvEl := UIA.ElementFromHandle(WinExist("DaVinci Resolve"))
;This searches the area confined to the MetadataPanel area
MetadataPanel := dvEl.FindFirstBy("AutomationId=UiMainWindow.bigBossWidget.widgetStack_Panel.WStackPage_Conform.m_pConformPanel.frameEditPageWidgetsContainer.frameEditPageMetadata")


Find1 := MetadataPanel.FindFirstByName("Good Take") ,
tw := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType", "checkbox")) ; creates a treewalker
CheckBoxGoodTake := tw.GetNextSiblingElement(Find1) ; Checks next Sibling from Good Take

if CheckBoxGoodTake  ; If not toggled then it will toggle checkbox on.
	togglePattern := CheckBoxGoodTake.GetCurrentPatternAs("Toggle")
		if !togglePattern.CurrentToggleState  ; remove the  "!"  if you want opposite (to toggle Off) or ....remove the whole line to keep toggling on and Off without checking
		togglePattern.Toggle()
	return

~Esc::ExitApp
Attachments
image.png
image.png (30.96 KiB) Viewed 1960 times

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 08 Dec 2022, 22:33

Hey @Descolada , With my above code above helping out user: k0stell0 , is there a way to use the new Function we were talking about if !(dvEl := GetDavinciElement()) when creating TreeWalkers?
if !(dvEl := GetDavinciElement()) works so great I've replaced all my scripts with, except for TreeWalkers.

It seems it needs the UIA := UIA_Interface() Function to create TreeWalkers and hence if it's not visible it'll throw an Error message which I'm trying to avoid.

Here is my code below trying to use the if !(dvEl := GetDavinciElement()) function

Code: Select all

#include <UIA_Interface>
d::
if !(dvEl := GetDavinciElement())
    return
UIA := UIA_Interface() ; Treewalker needs this which negates the use of the GetDavinciElement()) funciton

MetadataPanel := dvEl.FindFirstBy("AutomationId=UiMainWindow.bigBossWidget.widgetStack_Panel.WStackPage_Conform.m_pConformPanel.frameEditPageWidgetsContainer.frameEditPageMetadata")

Find1 := MetadataPanel.FindFirstByName("Good Take")

tw := UIA.CreateTreeWalker(UIA.CreateCondition("ControlType", "checkbox"))  I tried replacing UIA with dvEl but it didn't work

CheckBoxGoodTake := tw.GetNextSiblingElement(Find1)
if CheckBoxGoodTake
	togglePattern := CheckBoxGoodTake.GetCurrentPatternAs("Toggle")

		togglePattern.Toggle()
	return

~Esc::ExitApp
;;=======================   FUNCTION   ===============================

GetDavinciElement() {
    static davinciId, davinciElement

    if (!(dvId := WinExist("ahk_exe Resolve.exe")))
    	return
    if ((davinciId == dvId) && davinciElement)
        return davinciElement
    else {
        davinciElement := "", davinciId := dvId
        while (!davinciElement && WinExist("ahk_exe Resolve.exe")) {
            try davinciElement := UIA_Interface().ElementFromHandle(dvId)
        }
        return davinciElement
    }
}

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 09 Dec 2022, 00:15

@SteveMylo, to use a TreeWalker you can still use UIA := UIA_Interface() and you can do it as a global variable as well. GetDavinciElement should still work and you should be able to use TreeWalkers with it normally. This is because the interface returned by UIA_Interface() will always be the same - in fact you don't necessarily need to define UIA := UIA_Interface(), it's possible to use UIA_Interface() in place of UIA. Eg tw := UIA_Interface().CreateTreeWalker(UIA_Interface().CreateCondition("ControlType", "checkbox")). Does your provided code not work?
GetDavinciElement wasn't meant to replace the UIA variable completely: I simply used UIA_Interface() instead of UIA inside the function so that it would work in any script. This way it will still work when you don't use UIA as the UIA_Interface() variable, but for example cUIA as in UIA_Browser.

Also you can actually avoid using TreeWalker here, you can use FindByPath as well:

Code: Select all

Find1 := MetadataPanel.FindFirstByName("Good Take")
CheckBoxGoodTake := Find1.FindByPath("+1", "Type=CheckBox")
FindByPath will use the same TreeWalker internally. This works because at some point I updated CreateTreeWalker to accept the simplified conditions in text form (eg Type=Checkbox) instead of CreateCondition: UIA.CreateTreeWalker(UIA.CreateCondition("ControlType", "checkbox")) == UIA.CreateTreeWalker("Type=CheckBox").

As a small side-note: when toggling checkboxes it's sometimes happened that the checkbox I want to check is already checked (by the user perhaps) and Toggle() will uncheck it. This is why I usually specifically set the checkbox to a desired state:

Code: Select all

CheckBoxGoodTake := tw.GetNextSiblingElement(Find1)
if CheckBoxGoodTake
	CheckBoxGoodTake.TogglePattern.ToggleState := 1 ; 1 is checked, 0 is unchecked
@k0stell0, your code snippet seems okay except that you defined Metadata but then checked for Matadata - typo? In this case you are looking for a element with name "Metadata" and type "CheckBox", so if such an element exists then it should work:

Code: Select all

Metadata := MediaTab.FindFirstByNameAndType("Metadata", "checkbox").ToggleState

If (Metadata == True) { ; or try "if Metadata {" instead

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 09 Dec 2022, 07:13

Descolada wrote:
09 Dec 2022, 00:15
Does your provided code not work?
Sorry for any confusion, YES my code works perfectly.
But what I need which was my original question was, when something isn't visible cause I'm not in the correct PAGE in Davinci, how do I avoid the error messages?

You seemed to have solved it by giving me the function GetDavinciElement() but since I need treewalkers now, it's back to using UIA_Interface() function again and now I'm getting Error Messages when my page isn't open.
With using GetDavinciElement() only, I don't get error messages when the particular page isn't open when I need to use UIA_Interface().

So my question is, how can I avoid any error message if something isn't found or if the window isn't open.?
====================================================================
Oh and also, I tried CheckBoxGoodTake := Find1.FindByPath("+1", "Type=checkbox") but it doesn't work.
I was thinking that I havn't updated the latest #include file - UIA_Interface.ahk in a while.
but when I downloaded your latest UIA_Interface.ahk file from the github, It displays errors. Any ideas what is happening?
See Pic attached.
Attachments
image.png
image.png (38.95 KiB) Viewed 1847 times

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 09 Dec 2022, 10:10

@SteveMylo, sorry about that error, it should be fixed now. IsSet() was implemented in AHK 1.1.35.00, so you getting the error just meant that you are running an older version :)

If you want to avoid that error with TreeWalkers then check that you are passing in an actual element, not an empty string (which is returned when Find doesn't find a matching element).

Code: Select all

if (!Find1) {
    ; The first element wasn't found, so you can't proceed with the TreeWalker
    ; Switch tabs and try again? 
    return
}
CheckBoxGoodTake := tw.GetNextSiblingElement(Find1)
Or you could wrap it in a try...catch block:

Code: Select all

try {
    Find1 := MetadataPanel.FindFirstByName("Good Take")
    CheckBoxGoodTake := tw.GetNextSiblingElement(Find1)
} catch {
    ; Something went wrong
}

When using FindByPath, that error won't be thrown but you won't know where the search was unsuccessful. 
[code]CheckBoxGoodTake := MetadataPanel.FindFirstByName("Good Take").FindByPath("+1", "Type=CheckBox") ; Was MetadataPanel not found? Or "Good Take"? Or a checkbox wasn't found? Hard to tell here

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 09 Dec 2022, 13:39

@Descolada Thankyou it's all working now :bravo:

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 11 Dec 2022, 16:05

Thanks a lot, guys. :clap: Your help has been invaluable to me. I got my script working exactly the way I wanted.

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 14 Dec 2022, 19:14

Hi guys,

Is there a reliable way to click on bins located in Media Tab? Please see the image attached.
Ideally, I would prefer to find them by name or automation ID. Unfortunately, UIAViever can't see anything in the "Media Folder" panel. Maybe I'm doing something wrong.

I also tried to use FindText, but couldn't make it work reliably. Sometimes it finds, other times it doesn't. Plus it works a bit slowly.
Attachments
Media Tab - Bins.jpg
Media Tab - Bins.jpg (10.29 KiB) Viewed 1725 times

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 15 Dec 2022, 11:29

@k0stell0, it seems that particular part of DaVinci is not accessible with UIA, so you need to normal AHK methods for that (Send, MouseClick etc) or FindText.
FindText might need multiple Text for the bins, since the Text probably changes depending on if the bin is selected or not. You can also speed the search up a lot by narrowing the search area: with UIA you can get the "Master" element (the parent element which contains the bins) location and limit the search to that.
Another way might be to use UIA to set focus to the "Master" element with el.SetFocus(), use Send {Up 20} to select the topmost bin (Master), and then Send {Down n} to select the nth bin. Not sure if SetFocus will work though, haven't tested it out.

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 18 Dec 2022, 17:30

Thanks for your help @Descolada. I played with FindText over the weekend, but still couldn't make it work every time. So, I created a loop that goes through all folders in that panel, checks the name of each folder and does what I need if the name is right. Maybe it's not the best way to do it, but works better than FindText for me.

Post Reply

Return to “Ask for Help (v1)”