UIA v2

Post your working scripts, libraries and tools.
LAPIII
Posts: 676
Joined: 01 Aug 2021, 06:01

Re: UIA v2

Post by LAPIII » 07 Jun 2023, 13:57

Today I'm getting the following error and all my scripts that use UIA Browser:

Warning: This variable appears to never be assigned a value.
Specifically: global UIA_Browser


I start all my scripts like this:

Code: Select all

#SingleInstance Force
#Requires Autohotkey v2.0+
#Include <UIA>
#Include <UIA_Browser>
Run "https://www.google.com/"
WinWaitActive "ahk_exe msedge.exe"
cUIA := UIA_Browser()
cUIA.ElementFromHandle("ahk_exe msedge.exe")
I had tried changing the Wintitle to ahk_exe waterfox.exe, but get the same error.

EDIT: I keep going back with your updates to UIA_Browser, i.e. trying many of your old commits, and I'm getting:

UIA_Browser.ahk 09_06_23 08⦂04⦂41⦂624 AM.jpg
UIA_Browser.ahk 09_06_23 08⦂04⦂41⦂624 AM.jpg (47.71 KiB) Viewed 4113 times
Spitzi
Posts: 352
Joined: 24 Feb 2022, 03:45

Re: UIA v2

Post by Spitzi » 12 Jun 2023, 09:43

Hi @Descolada

If I search an element using

Code: Select all

widgetEl := UIA.ElementFromHandle(merlinWinTitle).FindElement({AutomationID:"SerienFenster.DicomDraw_OpenGL_0", matchmode:"Substring"})
, it does not find anything. But if i omit the substring als in

Code: Select all

widgetEl := UIA.ElementFromHandle(merlinWinTitle).FindElement({AutomationID:"SerienFenster.DicomDraw_OpenGL_0"})
, it works. Is there a bug in the substring matchmode?
Spitzi
Posts: 352
Joined: 24 Feb 2022, 03:45

Re: UIA v2

Post by Spitzi » 12 Jun 2023, 09:51

also, can I look for an element, that has a non-zero width?

widgetEl := UIA.ElementFromHandle(merlinWinTitle).FindElement({AutomationID:"SerienFenster.DicomDraw_OpenGL_0", matchmode:"Substring", Location.w>0})

or

widgetEl := UIA.ElementFromHandle(merlinWinTitle).FindElement({AutomationID:"SerienFenster.DicomDraw_OpenGL_0", matchmode:"Substring", not:{Location.w:"0"}})

is something like this possible?
Descolada
Posts: 1431
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 12 Jun 2023, 14:22

@Spitzi, no, specifically looking for an element with a non-zero width is not possible at the moment. You can instead use FindElements in conjunction with UIA.Filter to find such elements:

Code: Select all

matches := UIA.Filter(UIA.ElementFromHandle(merlinWinTitle).FindElements({AutomationID:"SerienFenster.DicomDraw_OpenGL_0", matchmode:"Substring"}), (el) => el.Location.w != 0) ; Returns an array of elements with the specified AutomationId and a non-zero width
widgetEl := matches[1]
Spitzi
Posts: 352
Joined: 24 Feb 2022, 03:45

Re: UIA v2

Post by Spitzi » 13 Jun 2023, 02:12

@Descolada , thanks for your help.

I managed to find the element I want by looping through the children of the parent element. But your input surely will come in handy in some other way. Good to know about UIA.Filter

On another note: if I have an element, how can I find it's previous sibling? There is a parent property, and a children property, but no siblings property.
LAPIII
Posts: 676
Joined: 01 Aug 2021, 06:01

Re: UIA v2

Post by LAPIII » 14 Jun 2023, 12:52

@Descolada Your second commit on May 31, 2023 for UIA Browser works perfectly and I don't get the error in my last post.
emp00
Posts: 197
Joined: 15 Apr 2023, 12:02

Re: UIA v2

Post by emp00 » 18 Jun 2023, 05:27

Dear Team, I have already automated a few browser tasks - all of them running in a vivaldi/chrome/edge/firefox browser window.

Question: How can I display a specific website in a simple AHK gui window with UI_Browser? It should be a barebone gui window allowing to browse a website -> without URL bar, no tab bar, no browser buttons etcpp. What I have in mind is a gadget for my outlook calendar - the gui should display and allow interaction with "https://outlook.office.com/calendar/" (with my saved login credentials). Is this possible?
Descolada
Posts: 1431
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 18 Jun 2023, 06:31

@emp00, embedding a website in a GUI is out of the scope of this library, so it's not possible. There are other methods to do this though, you can for example Google "AHK embedding webpage in GUI".
Charon
Posts: 12
Joined: 20 Aug 2022, 07:34

Re: UIA v2

Post by Charon » 23 Jun 2023, 08:15

How do I select a value in a combo box. From what I understand I have to expand the drop down box and then select a value. I can't seem to select anything in the combo box

Code: Select all

#include Lib\UIA.ahk
#include Lib\UIA_Browser.ahk

cUIA := UIA_Browser()
langBut := cUIA.FindElement({AutomationId:"00N120000066oX1", Type:"ComboBox"})
	if (langBut.ExpandCollapseState == UIA.ExpandCollapseState.Collapsed)
		langBut.Expand()

cUIA.WaitElement({Name:"test", Type:"ComboBox"}).SetValue := "green"



HTML code

Code: Select all

<form>
  <select id="00N120000066oX1" name = "test">
            <option value = "red" selected>red</option>
            <option value = "green">green</option>
			<option value = "blue">blue</option>
			<option value = "yellow">yellow</option>
         </select>
	  
	  </form>
Descolada
Posts: 1431
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 23 Jun 2023, 09:42

@Charon, UIA element AutomationId and Name might not always correspond to the id or name properties in the HTML, so I would recommend verifying those with UIAViewer. Also, to select combobox items with UIA, you usually need to find the selection element (e.g. element with Type="ListItem" and Name="green", though verify this with UIAViewer) and then Click() it.

So it might be something like this:

Code: Select all

comboboxEl := cUIA.FindElement({AutomationId:"00N120000066oX1", Type:"ComboBox"})
	if (comboboxEl.ExpandCollapseState == UIA.ExpandCollapseState.Collapsed)
		comboboxEl.Expand()
cUIA.WaitElement({Name:"green", Type:"ListItem"}).Click()
Also, sometimes (usually?) the combobox items are direct children to the combobox element, so you could improve the search speed by using comboboxEl.WaitElement.
Charon
Posts: 12
Joined: 20 Aug 2022, 07:34

Re: UIA v2

Post by Charon » 24 Jun 2023, 06:11

@Descolada, That worked. Thank you
nt-_-ts
Posts: 12
Joined: 27 Apr 2022, 08:02

Re: UIA v2

Post by nt-_-ts » 09 Jul 2023, 08:16

Hi @Descolada ,

With the following code:

Code: Select all

                    try
                        {
                            cadmaticWindow := UIA.ElementFromHandle("Project ahk_exe UIManager.exe")
                            cadmaticWindow.WaitElement({Type:"Button",Name:buttonName},timeOut := 1000)
                            cadmaticWindow.FindElement({Type:"Button",Name:buttonName}).Click()
                            MsgBox("This line is never reached")
                            DetectSaveWindow
                            return succes := true
                        }
The MsgBox line is not reached until the window that is opened by the click action is closed. If I replace the click action with AHK's Send and just send the hotkey for it, the MsgBox is reached. Similarly, with the same lines of code but for an Excel button it works. Is UIA waiting for something to finish based on the Click() action? I tried Invoke(), doesn't work either. It might have something to do with the software itself (Cadmatic Outfitting) as it is a strange engineering software.

I will use a workaround and still want to thank you for the amazing work you've done.
Descolada
Posts: 1431
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 09 Jul 2023, 09:07

@nt-_-ts, I've had that happen in a program that I can't remember the name of as well, where Invoke froze AHK and the program itself for a while. If using Invoke() causes the same problem then obviously it's the culprit. Possible workarounds might be DoDefaultAction(), ControlClick(), or Click('left').
malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIA v2

Post by malcev » 09 Jul 2023, 11:06

https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-implementinginvoke
Invoke is an asynchronous call and must return immediately without blocking.
This behavior is particularly critical for controls that, directly or indirectly, launch a modal dialog when invoked. Any UI Automation client that instigated the event will remain blocked until the modal dialog is closed.
https://learn.microsoft.com/en-us/windows/win32/api/oleacc/nf-oleacc-iaccessible-accdodefaultaction
Also, while accDoDefaultAction is supposed to return immediately, some implementations block the return. For example, if clicking a link displays a dialog, some implementations will block the return until the dialog is dismissed. Such delays can prevent client applications from processing a dialog box.
You can run invoke method from new ahk thread:
viewtopic.php?p=247253#p247253
Descolada
Posts: 1431
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 09 Jul 2023, 23:28

@malcev, ah yes, THAT'S where I remember that problem from - modal windows blocking code and user actions! I'll try to test running through a separate thread at some point when I have more time, it seems like it should work. I do wish the original author had documented his cryptic numbers a bit better though: for example NumPut(0x20EC8348, ptr + 17, "UInt"), NumPut(0xBACB8948, ptr + 21, "UInt") is fun to decipher. :)
malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: UIA v2

Post by malcev » 10 Jul 2023, 00:54

I think that original author Александр_ will not answer on this question because his last post on the forum was about 10 years ago.
But lexikos explains it here:
viewtopic.php?p=497537#p497537
BKnight
Posts: 1
Joined: 14 Jul 2023, 21:42

Re: UIA v2 beta

Post by BKnight » 14 Jul 2023, 22:06

emp00 wrote:
18 Apr 2023, 15:49
@Descolada I am missing words - THANK YOU sooo much, not only for the improved code but even more for the very constructive comments and explanations!! I have learned so much! Works flawlessly on computer1, will test on computer 2+3 tomorrow.

:bravo: :dance: :bravo:
Hi @emp00 , I was working on a similar project, can you guide me through the same
Descolada
Posts: 1431
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 15 Jul 2023, 01:55

@nt-_-ts, using the function suggested by @malcev, try something like the following:

Code: Select all

cadmaticWindow := UIA.ElementFromHandle("Project ahk_exe UIManager.exe")
cadmaticWindow.WaitElement({Type:"Button",Name:buttonName},timeOut := 1000)
UnFreeze(NextStuff, 200)
cadmaticWindow.FindElement({Type:"Button",Name:buttonName}).Click()
return

NextStuff() {
    MsgBox("This line is never reached")
    DetectSaveWindow
    return succes := true
}

UnFreeze(userFunc, timeOut) {
    static SYNCHRONIZE := 0x100000, hGui, oInfo := {}
    OnMessage( msg := DllCall("RegisterWindowMessage", "Str", "WM_INFO"), WM_INFO.Bind(oInfo) )
    if !IsSet(hGui)
        hGui := Gui()
    hProc := DllCall("OpenProcess", "UInt", SYNCHRONIZE, "UInt", 0, "UInt", DllCall("GetCurrentProcessId"), "Ptr")
    pPtr := GetProcAddr(hProc, hGui.Hwnd, msg, timeOut)
    oInfo.func := userFunc, oInfo.hProc := hProc
    DllCall("CreateThread", "Ptr", 0, "Ptr", 0, "Ptr", pPtr, "Ptr", 0, "UInt", 0, "Ptr", 0)
}
 
WM_INFO(oInfo, *) {
    DllCall("CloseHandle", "Ptr", oInfo.hProc)
    oInfo.func.Call()
}
 
GetProcAddr(Handle, hWnd, Msg, Timeout:=-1) { ; на основе http://forum.script-coding.com/viewtopic.php?pid=56073#p56073
    static MEM_COMMIT := 0x1000, PAGE_EXECUTE_READWRITE := 0x40
    ptr := DllCall("VirtualAlloc", "Ptr", 0, "Ptr", A_PtrSize = 4 ? 49 : 85, "UInt", MEM_COMMIT, "UInt", PAGE_EXECUTE_READWRITE, "Ptr")
    , hModule := DllCall("GetModuleHandle", "Str", "kernel32.dll", "Ptr")
    , pWaitForSingleObject := DllCall("GetProcAddress", "Ptr", hModule, "AStr", "WaitForSingleObject", "Ptr")
    , hModule := DllCall("GetModuleHandle", "Str", "user32.dll", "Ptr")
    , pSendMessageW := DllCall("GetProcAddress", "Ptr", hModule, "AStr", "SendMessageW", "Ptr")
    , NumPut("ptr", pWaitForSingleObject, ptr*1)
    , NumPut("ptr", pSendMessageW, ptr + A_PtrSize)

    if (A_PtrSize = 4)  {
        NumPut("UChar", 0x68, ptr + 8)
        , NumPut("Uint", Timeout, ptr + 9), NumPut("UChar", 0x68, ptr + 13)
        , NumPut("ptr", Handle, ptr + 14), NumPut("UShort", 0x15FF, ptr + 18)
        , NumPut("ptr", ptr, ptr + 20), NumPut("UShort", 0x6850, ptr + 24)
        , NumPut("ptr", Handle, ptr + 26), NumPut("UChar", 0x68, ptr + 30)
        , NumPut("UInt", Msg, ptr + 31), NumPut("UChar", 0x68, ptr + 35)
        , NumPut("ptr", hWnd, ptr + 36), NumPut("UShort", 0x15FF, ptr + 40)
        , NumPut("ptr", ptr+4, ptr + 42), NumPut("UChar", 0xC2, ptr + 46), NumPut("UShort", 4, ptr + 47)
    } else  {
        NumPut("UChar", 0x53, ptr + 16)
        , NumPut("UInt", 0x20EC8348, ptr + 17), NumPut("UInt", 0xBACB8948, ptr + 21)
        , NumPut("UInt", Timeout, ptr + 25), NumPut("UShort", 0xB948, ptr + 29)
        , NumPut("UPtr", Handle, ptr + 31), NumPut("UShort", 0x15FF, ptr + 39)
        , NumPut("UInt", -45, ptr + 41), NumPut("UShort", 0xB849, ptr + 45)
        , NumPut("UPtr", Handle, ptr + 47), NumPut("UChar", 0xBA, ptr + 55)
        , NumPut("UInt", Msg, ptr + 56), NumPut("UShort", 0xB948, ptr + 60)
        , NumPut("UPtr", hWnd, ptr + 62), NumPut("UInt", 0xC18941, ptr + 70)
        , NumPut("UShort", 0x15FF, ptr + 73), NumPut("UInt", -71, ptr + 75)
        , NumPut("UInt", 0x20C48348, ptr + 79), NumPut("UShort", 0xC35B, ptr + 83)
    }
    return ptr + A_PtrSize*2
}
If that works, you could try changing UnFreeze(NextStuff, 200) to SetTimer NextStuff, -200 to figure out whether UnFreeze is necessary at all.

EDIT: you could also try running Invoke in an actually new thread. Something like this:

Code: Select all

cadmaticWindow := UIA.ElementFromHandle("Project ahk_exe UIManager.exe")
el := cadmaticWindow.WaitElement({Type:"Button",Name:buttonName},timeOut := 1000)

hProc := DllCall("OpenProcess", "UInt", 0x100000, "UInt", 0, "UInt", DllCall("GetCurrentProcessId"), "Ptr")
hThread := DllCall("CreateThread", "Ptr", 0, "Ptr", 0, "Ptr", NumGet(NumGet(el.InvokePattern.ptr, "Ptr"), 3*A_PtrSize, "ptr"), "Ptr", el.InvokePattern.ptr, "UInt", 0, "Ptr", 0) ; Run Invoke in a new thread
timeOut := A_TickCount+3000
While (DllCall("WaitForSingleObject", "Ptr", hThread, "Int", 200) != 0 && A_TickCount < timeOut) ; Wait maximum three seconds for thread to terminate
    Sleep 40
DllCall("CloseHandle", "Ptr", hProc) ; Maybe should terminate thread beforehand?
fortpierre
Posts: 4
Joined: 18 Jul 2023, 17:01

Re: UIA v2

Post by fortpierre » 22 Jul 2023, 15:27

Hello @Descolada , thanks you for your very usefull liibrary !

I want to use it to automate some of my work and share it with my collegues, but working in a law environment I was wondering under what license it falls into ? I saw that some of your others works are under MIT 2 or GPL but it isn't mark for this one.

Thans you very much !
Descolada
Posts: 1431
Joined: 23 Dec 2021, 02:30

Re: UIA v2

Post by Descolada » 23 Jul 2023, 00:19

@fortpierre, thanks for bringing my attention to this - I've now updated my libraries to fall under the MIT license.
Post Reply

Return to “Scripts and Functions (v2)”