Interacting with DaVinci using UIAutomation

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Interacting with DaVinci using UIAutomation

Post by Descolada » 29 Jun 2022, 05:34

As per request of moderator @BoBo, I am continuing the thread from here: viewtopic.php?p=469345#p469345 (mods, feel free to move the pertinent posts to here :)).

@SteveMylo,
SteveMylo wrote:
28 Jun 2022, 17:42
1. When you created the TreeWalker, Can you go to the 2nd or 3rd sibling? not just the next or previous?
Once you have created a TreeWalker with a certain condition, you can feed it any element you want and call the TreeWalker methods on that element. So you can just feed the element gotten by GetNextSiblingElement into the TreeWalker again, like this:

Code: Select all

XValue := tw.GetNextSiblingElement(XText) ; First sibling
nextFromXValue := tw.GetNextSiblingElement(XValue) ; Second sibling, and so on...
2. In the inspector.exe tool when you click on the Rectangle icon 'watch focus' button only and not the 'watch cursor', it gives you access to a few more spots (in Davinci for example) that the watch Cursor can't.
But I can't use the data from it, for example, the AutomationID.
If you see the pictures below. I used it to separate the Video and Audion panels, normally they are one big Panel. But any data given by the 'watch focus' method doesn't do anything even though it shows you a unique AutomationID.
Is it possible? It would be a great help for me in automating.
Notice that the element you got with the inspector.exe tool is of type "Window". This means that DaVinci is generating a custom window (GUI) for the Audio/Video panel, and getting the information from these proved tricky... I ended up creating a custom tool (for now named UIATreeInspector), where if you uncheck Visible and click though all the Resolve.exe files you should eventually find the correct window for the Audio and Video panels. Unfortunately it still doesn't display the buttons inside (for example Lock Track), so its of not much use. So I think interacting with those buttons is pretty much hopeless (though if you really need it, you could request the creators of DaVinci to implement UIAutomation a bit better in their programs). :(

You can also get all the windows' info like this (paste it somewhere):

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2
DetectHiddenWindows, On

#include <UIA_Interface>
UIA := UIA_Interface()

WinActivate, DaVinci Resolve
WinGet, wList, List, ahk_exe Resolve.exe
Clipboard := ""
Loop, %wList%
{
    hwnd := wList%A_Index%
	winEl := UIA.ElementFromHandle(hwnd)
	Clipboard .=  "Window with hwnd " hwnd "`n" winEl.DumpAll() "`n`n"
	/*
	automationIds := winEl.FindAll(UIA.CreateNotCondition(UIA.CreatePropertyCondition(UIA.AutomationIdPropertyId, "")))
	for k, automationEl in automationIds {
		if InStr(automationEl.CurrentAutomationId, "AudioTracks") {
			; Do something
		}
	}
	*/
}

MsgBox, Finished.
ExitApp

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 30 Jun 2022, 19:30

This has been great. Your examples have all worked perfectly.
I've even started automating my DOPUS.Lister File explorer with this just for fun.
And well if you like a challenge I have another question about Davinci. You don't have to answer unless you want to but it may help others in complex situations.

If you see my image below, The UIAutomation always can see hidden' Panel Groups'. Unless of course, the VIDEO toggle isn't activated but It always will be in my case.
image.png
Transform Panel
image.png (35.16 KiB) Viewed 3727 times
So what I mean is, I want to know if the 'Transform' panel is maximized or minimized.
I'm trying to just Automate everything with all the checks in place and everything toggles open.
It's fun trying to work this one out but I've given up after many attempts.
So If I wanted to enter a 'Zoom' value by double-clicking on it, it won't work if the 'Transform Panel' is minimised. Maybe it can't be done. (well it can with 'SetValue' probably... but I'm just trying to find unique and different checks to do this)
I can easily do a Pixel Search and see if it's Open or Not but I want to figure it out with UIAutomation.
When it's minimized, my code will still send a mouse there to double click on the Zoom Parametre, but if it's minimized, it'll click in the same place but on something else because it still sees the hidden panel behind it which I don't want.

Even though setValue will probably work for this, AND....just now realised that I Can check if the Bounding Box of the 'Cropping' Panel is at a certain height
BUT.......... I want to know if my initial way can work, (to check if the Transform Panel is maximized) cause you never know when you need it.
See my code below: Even when Transform Panel is minimized, it still finds 'Zoom' and it Beeps.

Code: Select all

#NoEnv
SetWorkingDir %A_ScriptDir%
CoordMode, Mouse, Screen
SendMode Event
#SingleInstance, Force
SetTitleMatchMode 2
SetTitleMatchMode Fast
#Persistent
#include <UIA_Interface>

v::
UIA := UIA_Interface()
WinActivate, ahk_exe Resolve.exe
WinWaitActive, ahk_exe Resolve.exe
davinci := UIA.ElementFromHandle(WinExist("ahk_exe Resolve.exe"))
Zoom := davinci.FindFirstByNameAndType("Zoom", "text")

if Zoom {
	soundbeep
}
Return

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 01 Jul 2022, 00:57

@SteveMylo, I'm always up for a challenge ;)
This case is tricky indeed. Usually elements like these should support the ExpandCollapsePattern, which clearly isn't the case here. And usually when elements aren't visible then they don't exist or have CurrentOffscreen set to 0/1, but this also isn't the case.
Now, I didn't inspect all the properties of the Zoom text element and Transform panel element so there possible might be some obscure property that allows differentiating these, but I resorted to this instead:

Code: Select all

if Zoom {
	br := Zoom.CurrentBoundingRectangle
	pointEl := UIA.ElementFromPoint(br.l, br.t)
	if UIA.CompareElements(Zoom, pointEl)
		SoundBeep
}
Kind of hacky and and this point ImageSearch/FindText would probably work just as fine, but it works. :P

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 01 Jul 2022, 08:51

Descolada wrote:
01 Jul 2022, 00:57
Kind of hacky and and this point ImageSearch/FindText would probably work just as fine, but it works. :P
yeah I use FINDTEXT a lot.

Yes your hacky way worked well, although I had my own hacky way (see below) by assessing the height of the Cropping Panel compared to the transform panel.
So what exactly is your code doing? I kinda get it.
Also is there any documentation where i can learn what these functions do and their names?
e.g. UIA.CompareElements and... UIA.ElementFromPoint
Or are descriptions just written in the Script its self?

Code: Select all

UIA := UIA_Interface()    
davinci := UIA.ElementFromHandle(WinExist("DaVinci Resolve"))
1a := davinci.FindFirstByNameAndType("Transform", "Text")
2a := davinci.FindFirstByNameAndType("Cropping", "Text")

CoordsTR := 1a.CurrentBoundingRectangle
CoordsCR := 2a.CurrentBoundingRectangle
TrCrDistance := CoordsCR.t - CoordsTR.t   ; ; Check if Transform Panel  Is Maxamized. Checks height of Cropping Panel

if TrCrDistance < 150
{
Soundbeep
}

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 01 Jul 2022, 10:23

The best place for UIA documentation is probably Microsoft's own documentation. Some info is also in the library itself: methods where you use "UIA." are in the UIA_Interface class, methods/properties for elements are in UIA_Element class etc.

My versioon gets the screen coordinates for the Zoom elements, uses ElementFromPoint to get an element from that screen point, and then compares these two elements with CompareElements: if Zoom is hidden, then ElementFromPoint will get some other visible element from that point (eg. for the Cropping panel).
Direct comparison with == between these objects would not work, because we only have references to the elements, not the element objects itself, that's why the need for ElementCompare.

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 13 Jul 2022, 17:55

Hey @Descolada , thanks.
Another Query.
After watching the Automators tutorial. using FindAllByType() is fantastic.
And then clicking click() on the actual index works perfectly too.
For example, clicking on the bottom Right 'House icon' (Project manager) in DaVinci.
image.png
image.png (1.38 KiB) Viewed 3406 times

Code: Select all

UIA := UIA_Interface()
davinci := UIA.ElementFromHandle(WinExist("ahk_exe Resolve.exe"))
buttons := davinci.FindAllByType("button")
buttons[4].click()  ;  Works perfect!
What I need to Learn is mousemove instead, because not everything is clickable. But I still want to use the FindAllByType("button") and index[] method.
I could use other methods yes of course, but the FindAll and index method is a really nice short code. :angel:

Isaias Baez Mentioned to me that this could work but it didn't.

Code: Select all

UIA := UIA_Interface()
davinci := UIA.ElementFromHandle(WinExist("ahk_exe Resolve.exe"))
buttons := davinci.FindAllByType("button")
btnPos := buttons[4].GetClickablePoint()
MouseMove, btnPos.x, btnPos.y, 1

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 13 Jul 2022, 22:45

@SteveMylo, I haven't tested the snippet you provided, but what stands out to me is MouseMove, btnPos.x, btnPos.y, 1: I think it should be MouseMove, % btnPos.x, % btnPos.y, 1.

Otherwise buttons[4].Click("left") should also use AHKs Click function to click. In some programs ControlClick also works (buttons[4].ControlClick(WinExist("ahk_exe Resolve.exe"))).

gregster
Posts: 8921
Joined: 30 Sep 2013, 06:48

Re: Interacting with DaVinci using UIAutomation

Post by gregster » 13 Jul 2022, 22:56

Descolada wrote:
13 Jul 2022, 22:45
what stands out to me is MouseMove, btnPos.x, btnPos.y, 1: I think it should be MouseMove, % btnPos.x, % btnPos.y, 1.
Generally, this syntax should be okay, since the docs say that the first two (actually three) parameters of MouseMove allow expressions.
In AHK v1, this is usually the case for command input parameters which are strictly numerical. Another random example: Sleep and its delay parameter.
But parameters which also deal with literal strings won't allow this (the docs will always tell you, if expressions are allowed).

On the other hand, forcing an expression, like you recommended, doesn't hurt either when dealing with these strictly numerical params. There's no difference.

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 13 Jul 2022, 23:00

@Descolada Thanks
MouseMove, % btnPos.x, % btnPos.y, 1 Does not work. :(

Code: Select all

davinci := UIA.ElementFromHandle(WinExist("ahk_exe Resolve.exe"))
buttons := davinci.FindAllByType("button")
btnPos := buttons[4].GetClickablePoint()
MouseMove, % btnPos.x, % btnPos.y, 2    ; doesn't work  :-(


But your code below works well. buttons[4].Click("Left") :D
Although it'll be nice if we can work out the above Mousemove to work.... then I could offset the X & Y coordinates for various different ideas I have.

Code: Select all

davinci := UIA.ElementFromHandle(WinExist("ahk_exe Resolve.exe"))
buttons := davinci.FindAllByType("button")
buttons[4].Click("Left")      ; this moves the mouse and works :-)
When you're at your desk, let me know if you find time to have a looksy! :angel:

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 14 Jul 2022, 01:53

@SteveMylo, so it appears that GetClickablePoint doesn't work when the Davinci window isn't visible: did your code also include a part for activating the window or making sure it is in view?
Also it's safer to use GetClickablePointRelativeTo(), since it automatically corrects for CoordMode Mouse (Microsofts original GetClickablePoint returns absolute screen coordinates).

Code: Select all

#include <UIA_Interface>
UIA := UIA_Interface()
WinActivate, ahk_exe Resolve.exe
WinWaitActive, ahk_exe Resolve.exe,,1
davinci := UIA.ElementFromHandle(WinExist("ahk_exe Resolve.exe"))
buttons := davinci.FindAllByType("button")
btnPos := buttons[4].GetClickablePointRelativeTo()
MouseMove, btnPos.x, btnPos.y, 2
But Click("left") internally uses GetCurrentPos and calculates the middle of the element, so you can alternatively use that instead:

Code: Select all

pos := buttons[4].GetCurrentPos()
Click, % pos.x+pos.w//2 " " pos.y+pos.h//2

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 14 Jul 2022, 02:20

Descolada wrote:
14 Jul 2022, 01:53
did your code also include a part for activating the window or making sure it is in view?
Yes I did although your last code didn't work either. :cry:
I'm also not interested in any control clicks at the moment but very useful to know.
So, to get it WORKING I resorted back to the good old faithful XYCoords := buttons[4].CurrentBoundingRectangle and it WORKED! I can now offset my Coords if I want to.
Thanks for helping again, hopefully others will learn from this too:

Code: Select all

v::
UIA := UIA_Interface()
WinActivate, ahk_exe Resolve.exe
WinWaitActive, ahk_exe Resolve.exe,,1
davinci := UIA.ElementFromHandle(WinExist("ahk_exe Resolve.exe"))
buttons := davinci.FindAllByType("button")

XYCoords := buttons[4].CurrentBoundingRectangle
mousemove, (XYCoords.l+((XYCoords.r-XYCoords.l)/2)) ,(XYCoords.t + ((XYCoords.b-XYCoords.t)/2)) , 3

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 30 Oct 2022, 18:27

Hi guys,

I've been trying to automate changing of frame rate for selected clips in Davinci Resolve and can't figure out how to choose an item by name from ListItems (a.k.a ComboBox).
Clip Attributes.jpg
Clip Attributes.jpg (21.72 KiB) Viewed 2711 times
Please see my code below:

Code: Select all

UIA := UIA_Interface()
WinWaitActive, Clip Attributes
ClipAttributes := UIA.ElementFromHandle(WinExist("Clip Attributes"))
ClipAttributes.FindFirstBy("AutomationId=UiClipAttributesDialog.clipAttributesHolderFrame.m_ClipAttributesView.frameClipAttributesBody.frame.frame_3.m_StackedWidgetClipAttributes.m_ClipAttributesVideoView.caVideoFrameRateFrame.frameFPSContainer.m_cbVideoFrameRate").Click()
FrameRates := ClipAttributes.FindAllByType("ListItem")
FrameRates.FindFirstByName("25").Click()
It opens the Video Frame Rate field, but the last line doesn't click on 25.
I know that I could possibly use the top arrow to move the cursor 5 lines up and click enter, but since I use clips with different frame rates, I would prefer to choose it by name.

Could anyone please point me in the right direction to solve this one?

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 30 Oct 2022, 21:45

k0stell0 wrote:I've been trying to automate changing of frame rate for
UIAutomation physically has to see the 25. Otherwise it will never find it. So yes you have to scroll up or down

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 30 Oct 2022, 23:19

SteveMylo wrote:
30 Oct 2022, 21:45
UIAutomation physically has to see the 25. Otherwise it will never find it. So yes you have to scroll up or down
Hi Steve,
I inspected the Clip Attributes windows and from my understanding UIAutomation can see all values in that ComboBox. Please see the screenshot below.
Spoiler
I just can't find a way to click on 25.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 31 Oct 2022, 01:46

@k0stell0, FrameRates := ClipAttributes.FindAllByType("ListItem") will return you an array of elements, but an array doesn't have a method FindFirstByName which you use in FrameRates.FindFirstByName("25").Click(). Try using ClipAttributes.WaitElementExistByNameAndType("25", "ListItem").Click() instead.

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 31 Oct 2022, 17:49

k0stell0 wrote:
30 Oct 2022, 23:19
I just can't find a way to click on 25.
Ok I stand by what I said, .Click() does not work on menu items in DaVinci Resolve it seems.
You have to find the.CurrentBoundingRectangle for the item then MouseClick on the coordinates.
This is also why I mentioned you have to physically see the number '25'.
I've run into this problem before, and so if you can't see '25' , then you have to Code a (mousemove and scroll) to the VideoFramerateComboBox until UIA see's '25', then MouseClick

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 31 Oct 2022, 18:34

k0stell0 wrote:
30 Oct 2022, 23:19
I just can't find a way to click on 25.
Here is the code below which is the only way I find it can work for any (Menu or List Items)
For example, the code ClipAttributes.WaitElementExistByNameAndType("25", "ListItem") won't work if you set the frame rate to 120 beforehand, and then try My code below. (without pressing Up 16 times that is)
UIA cannot find 25 when the dropdown menu appears... 25 is hidden up the top.
But if you set it to 30, UIA can see the hidden 25. So you have to press up 16 times ish for UIA to even know 25 exists.

Code: Select all

UIA := UIA_Interface()
WinWaitActive, Clip Attributes
ClipAttributes := UIA.ElementFromHandle(WinExist("Clip Attributes"))
ClipAttributes.FindFirstBy("AutomationId=UiClipAttributesDialog.clipAttributesHolderFrame.m_ClipAttributesView.frameClipAttributesBody.frame.frame_3.m_StackedWidgetClipAttributes.m_ClipAttributesVideoView.caVideoFrameRateFrame.frameFPSContainer.m_cbVideoFrameRate").Click()  ; click works here cause it's not a menu item

send, {Up 16} ; UIA has to physically see the number 25, so this send it close to the top.
Sleep, 55
ClipAttributes.WaitElementExistByNameAndType("25", "ListItem")
FrameRates := ClipAttributes.FindFirstByNameAndType("25", "ListItem")
Sleep, 55

XYCoords := FrameRates.CurrentBoundingRectangle
MouseClick,, (XYCoords.l+((XYCoords.r-XYCoords.l) // 2)) , (XYCoords.t + ((XYCoords.b-XYCoords.t)//2)), 1, 9
return

k0stell0
Posts: 14
Joined: 30 Oct 2022, 18:01

Re: Interacting with DaVinci using UIAutomation

Post by k0stell0 » 06 Nov 2022, 22:15

Hi @SteveMylo,

Thanks a lot for your time and help. I tried your code over the weekend, unfortunately, it still doesn't click on 25 for me.
It just moves to the top of the drop-down list and then closes it without changing the frame rate.

I'm trying to change from 50 to 25 fps, so I assume UIA should see 25 even without moving to the top of the list.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: Interacting with DaVinci using UIAutomation

Post by Descolada » 07 Nov 2022, 00:26

@k0stell0, this finally worked for me:

Code: Select all

UIA := UIA_Interface()
WinActivate, Clip Attributes
WinWaitActive, Clip Attributes
ClipAttributes := UIA.ElementFromHandle("Clip Attributes")
fpsCombo := ClipAttributes.FindFirstBy("AutomationId=UiClipAttributesDialog.clipAttributesHolderFrame.m_ClipAttributesView.frameClipAttributesBody.frame.frame_3.m_StackedWidgetClipAttributes.m_ClipAttributesVideoView.caVideoFrameRateFrame.frameFPSContainer.m_cbVideoFrameRate")
fpsCombo.Click()
fps25 := fpsCombo.WaitElementExistByName("25")
fps25.ControlClick("Clip Attributes")
ExitApp
You can combine lines to make it shorter, at the cost of readability:

Code: Select all

(fpsCombo := ClipAttributes.FindFirstBy("AutomationId=UiClipAttributesDialog.clipAttributesHolderFrame.m_ClipAttributesView.frameClipAttributesBody.frame.frame_3.m_StackedWidgetClipAttributes.m_ClipAttributesVideoView.caVideoFrameRateFrame.frameFPSContainer.m_cbVideoFrameRate")).Click()
(fps25 := fpsCombo.WaitElementExistByName("25")).ControlClick("Clip Attributes")
It seems that for the ListItem elements the normal Invoke doesn't work: fps25.InvokePattern.Invoke() usually should click/select the element, didn't work here. SelectionItemPattern Select does highlight the item, but doesn't actually select it: fps25.SelectionItemPattern.Select() makes it active, but sending Enter after that doesn't actually select it, so I think that as well isn't working properly. So I think there are two solutions: either use fps25.Click("left") to use MouseClick, or my proposed solution with ControlClick. Theoretically you could also use SelectionItem Select and then ControlSend {Up}{Down}{Enter}, which will re-highlight the desired item and selects it. As @SteveMylo pointed out the item isn't accessible if it isn't visible, so you might need to send Home/End before selecting the item ControlSend,, {Home}, Clip Attributes (does the same as SteveMylo's Send {Up 16}).
Why SteveMylo's solution didn't work for you - I think your CoordMode for Mouse might not be set to Screen: CurrentBoundingRectangle reports Screen/absolute values.

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: Interacting with DaVinci using UIAutomation

Post by SteveMylo » 07 Nov 2022, 01:57

Descolada fabulous as always

Post Reply

Return to “Ask for Help (v1)”