UIA v2

[LAPIII]

Today I'm getting the following error and all my scripts that use UIA Browser:

Warning: This variable appears to never be assigned a value.
Specifically: global UIA_Browser

I start all my scripts like this:

Code: Select all

#SingleInstance Force
#Requires Autohotkey v2.0+
#Include <UIA>
#Include <UIA_Browser>
Run "https://www.google.com/"
WinWaitActive "ahk_exe msedge.exe"
cUIA := UIA_Browser()
cUIA.ElementFromHandle("ahk_exe msedge.exe")

I had tried changing the Wintitle to ahk_exe waterfox.exe, but get the same error.

EDIT: I keep going back with your updates to UIA_Browser, i.e. trying many of your old commits, and I'm getting:

UIA_Browser.ahk 09_06_23 08⦂04⦂41⦂624 AM.jpg (47.71 KiB) Viewed 3111 times

[Spitzi]

Hi @Descolada

If I search an element using

Code: Select all

widgetEl := UIA.ElementFromHandle(merlinWinTitle).FindElement({AutomationID:"SerienFenster.DicomDraw_OpenGL_0", matchmode:"Substring"})

, it does not find anything. But if i omit the substring als in

Code: Select all

widgetEl := UIA.ElementFromHandle(merlinWinTitle).FindElement({AutomationID:"SerienFenster.DicomDraw_OpenGL_0"})

, it works. Is there a bug in the substring matchmode?

[Spitzi]

also, can I look for an element, that has a non-zero width?

widgetEl := UIA.ElementFromHandle(merlinWinTitle).FindElement({AutomationID:"SerienFenster.DicomDraw_OpenGL_0", matchmode:"Substring", Location.w>0})

or

widgetEl := UIA.ElementFromHandle(merlinWinTitle).FindElement({AutomationID:"SerienFenster.DicomDraw_OpenGL_0", matchmode:"Substring", not:{Location.w:"0"}})

is something like this possible?

[Descolada]

@Spitzi, no, specifically looking for an element with a non-zero width is not possible at the moment. You can instead use FindElements in conjunction with UIA.Filter to find such elements:

Code: Select all

matches := UIA.Filter(UIA.ElementFromHandle(merlinWinTitle).FindElements({AutomationID:"SerienFenster.DicomDraw_OpenGL_0", matchmode:"Substring"}), (el) => el.Location.w != 0) ; Returns an array of elements with the specified AutomationId and a non-zero width
widgetEl := matches[1]

[Spitzi]

@Descolada , thanks for your help.

I managed to find the element I want by looping through the children of the parent element. But your input surely will come in handy in some other way. Good to know about UIA.Filter

On another note: if I have an element, how can I find it's previous sibling? There is a parent property, and a children property, but no siblings property.

[LAPIII]

@Descolada Your second commit on May 31, 2023 for UIA Browser works perfectly and I don't get the error in my last post.

[emp00]

Dear Team, I have already automated a few browser tasks - all of them running in a vivaldi/chrome/edge/firefox browser window.

Question: How can I display a specific website in a simple AHK gui window with UI_Browser? It should be a barebone gui window allowing to browse a website -> without URL bar, no tab bar, no browser buttons etcpp. What I have in mind is a gadget for my outlook calendar - the gui should display and allow interaction with "https://outlook.office.com/calendar/" (with my saved login credentials). Is this possible?

[Descolada]

@emp00, embedding a website in a GUI is out of the scope of this library, so it's not possible. There are other methods to do this though, you can for example Google "AHK embedding webpage in GUI".

[Charon]

How do I select a value in a combo box. From what I understand I have to expand the drop down box and then select a value. I can't seem to select anything in the combo box

Code: Select all

#include Lib\UIA.ahk
#include Lib\UIA_Browser.ahk

cUIA := UIA_Browser()
langBut := cUIA.FindElement({AutomationId:"00N120000066oX1", Type:"ComboBox"})
	if (langBut.ExpandCollapseState == UIA.ExpandCollapseState.Collapsed)
		langBut.Expand()

cUIA.WaitElement({Name:"test", Type:"ComboBox"}).SetValue := "green"

HTML code

Code: Select all

<form>
  <select id="00N120000066oX1" name = "test">
            <option value = "red" selected>red</option>
            <option value = "green">green</option>
			<option value = "blue">blue</option>
			<option value = "yellow">yellow</option>
         </select>
	  
	  </form>

[Descolada]

@Charon, UIA element AutomationId and Name might not always correspond to the id or name properties in the HTML, so I would recommend verifying those with UIAViewer. Also, to select combobox items with UIA, you usually need to find the selection element (e.g. element with Type="ListItem" and Name="green", though verify this with UIAViewer) and then Click() it.

So it might be something like this:

Code: Select all

comboboxEl := cUIA.FindElement({AutomationId:"00N120000066oX1", Type:"ComboBox"})
	if (comboboxEl.ExpandCollapseState == UIA.ExpandCollapseState.Collapsed)
		comboboxEl.Expand()
cUIA.WaitElement({Name:"green", Type:"ListItem"}).Click()

Also, sometimes (usually?) the combobox items are direct children to the combobox element, so you could improve the search speed by using comboboxEl.WaitElement.

[Charon]

@Descolada, That worked. Thank you

[nt-_-ts]

Hi @Descolada ,

With the following code:

Code: Select all

                    try
                        {
                            cadmaticWindow := UIA.ElementFromHandle("Project ahk_exe UIManager.exe")
                            cadmaticWindow.WaitElement({Type:"Button",Name:buttonName},timeOut := 1000)
                            cadmaticWindow.FindElement({Type:"Button",Name:buttonName}).Click()
                            MsgBox("This line is never reached")
                            DetectSaveWindow
                            return succes := true
                        }

The MsgBox line is not reached until the window that is opened by the click action is closed. If I replace the click action with AHK's Send and just send the hotkey for it, the MsgBox is reached. Similarly, with the same lines of code but for an Excel button it works. Is UIA waiting for something to finish based on the Click() action? I tried Invoke(), doesn't work either. It might have something to do with the software itself (Cadmatic Outfitting) as it is a strange engineering software.

I will use a workaround and still want to thank you for the amazing work you've done.

[Descolada]

@nt-_-ts, I've had that happen in a program that I can't remember the name of as well, where Invoke froze AHK and the program itself for a while. If using Invoke() causes the same problem then obviously it's the culprit. Possible workarounds might be DoDefaultAction(), ControlClick(), or Click('left').

[malcev]

https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-implementinginvoke

Invoke is an asynchronous call and must return immediately without blocking.
This behavior is particularly critical for controls that, directly or indirectly, launch a modal dialog when invoked. Any UI Automation client that instigated the event will remain blocked until the modal dialog is closed.

https://learn.microsoft.com/en-us/windows/win32/api/oleacc/nf-oleacc-iaccessible-accdodefaultaction

Also, while accDoDefaultAction is supposed to return immediately, some implementations block the return. For example, if clicking a link displays a dialog, some implementations will block the return until the dialog is dismissed. Such delays can prevent client applications from processing a dialog box.

You can run invoke method from new ahk thread:
viewtopic.php?p=247253#p247253

[Descolada]

@malcev, ah yes, THAT'S where I remember that problem from - modal windows blocking code and user actions! I'll try to test running through a separate thread at some point when I have more time, it seems like it should work. I do wish the original author had documented his cryptic numbers a bit better though: for example NumPut(0x20EC8348, ptr + 17, "UInt"), NumPut(0xBACB8948, ptr + 21, "UInt") is fun to decipher.

[malcev]

I think that original author Александр_ will not answer on this question because his last post on the forum was about 10 years ago.
But lexikos explains it here:
viewtopic.php?p=497537#p497537

[BKnight]

@Descolada I am missing words - THANK YOU sooo much, not only for the improved code but even more for the very constructive comments and explanations!! I have learned so much! Works flawlessly on computer1, will test on computer 2+3 tomorrow.

Hi @emp00 , I was working on a similar project, can you guide me through the same

[Descolada]

@nt-_-ts, using the function suggested by @malcev, try something like the following:

Code: Select all

cadmaticWindow := UIA.ElementFromHandle("Project ahk_exe UIManager.exe")
cadmaticWindow.WaitElement({Type:"Button",Name:buttonName},timeOut := 1000)
UnFreeze(NextStuff, 200)
cadmaticWindow.FindElement({Type:"Button",Name:buttonName}).Click()
return

NextStuff() {
    MsgBox("This line is never reached")
    DetectSaveWindow
    return succes := true
}

UnFreeze(userFunc, timeOut) {
    static SYNCHRONIZE := 0x100000, hGui, oInfo := {}
    OnMessage( msg := DllCall("RegisterWindowMessage", "Str", "WM_INFO"), WM_INFO.Bind(oInfo) )
    if !IsSet(hGui)
        hGui := Gui()
    hProc := DllCall("OpenProcess", "UInt", SYNCHRONIZE, "UInt", 0, "UInt", DllCall("GetCurrentProcessId"), "Ptr")
    pPtr := GetProcAddr(hProc, hGui.Hwnd, msg, timeOut)
    oInfo.func := userFunc, oInfo.hProc := hProc
    DllCall("CreateThread", "Ptr", 0, "Ptr", 0, "Ptr", pPtr, "Ptr", 0, "UInt", 0, "Ptr", 0)
}
 
WM_INFO(oInfo, *) {
    DllCall("CloseHandle", "Ptr", oInfo.hProc)
    oInfo.func.Call()
}
 
GetProcAddr(Handle, hWnd, Msg, Timeout:=-1) { ; на основе http://forum.script-coding.com/viewtopic.php?pid=56073#p56073
    static MEM_COMMIT := 0x1000, PAGE_EXECUTE_READWRITE := 0x40
    ptr := DllCall("VirtualAlloc", "Ptr", 0, "Ptr", A_PtrSize = 4 ? 49 : 85, "UInt", MEM_COMMIT, "UInt", PAGE_EXECUTE_READWRITE, "Ptr")
    , hModule := DllCall("GetModuleHandle", "Str", "kernel32.dll", "Ptr")
    , pWaitForSingleObject := DllCall("GetProcAddress", "Ptr", hModule, "AStr", "WaitForSingleObject", "Ptr")
    , hModule := DllCall("GetModuleHandle", "Str", "user32.dll", "Ptr")
    , pSendMessageW := DllCall("GetProcAddress", "Ptr", hModule, "AStr", "SendMessageW", "Ptr")
    , NumPut("ptr", pWaitForSingleObject, ptr*1)
    , NumPut("ptr", pSendMessageW, ptr + A_PtrSize)

    if (A_PtrSize = 4)  {
        NumPut("UChar", 0x68, ptr + 8)
        , NumPut("Uint", Timeout, ptr + 9), NumPut("UChar", 0x68, ptr + 13)
        , NumPut("ptr", Handle, ptr + 14), NumPut("UShort", 0x15FF, ptr + 18)
        , NumPut("ptr", ptr, ptr + 20), NumPut("UShort", 0x6850, ptr + 24)
        , NumPut("ptr", Handle, ptr + 26), NumPut("UChar", 0x68, ptr + 30)
        , NumPut("UInt", Msg, ptr + 31), NumPut("UChar", 0x68, ptr + 35)
        , NumPut("ptr", hWnd, ptr + 36), NumPut("UShort", 0x15FF, ptr + 40)
        , NumPut("ptr", ptr+4, ptr + 42), NumPut("UChar", 0xC2, ptr + 46), NumPut("UShort", 4, ptr + 47)
    } else  {
        NumPut("UChar", 0x53, ptr + 16)
        , NumPut("UInt", 0x20EC8348, ptr + 17), NumPut("UInt", 0xBACB8948, ptr + 21)
        , NumPut("UInt", Timeout, ptr + 25), NumPut("UShort", 0xB948, ptr + 29)
        , NumPut("UPtr", Handle, ptr + 31), NumPut("UShort", 0x15FF, ptr + 39)
        , NumPut("UInt", -45, ptr + 41), NumPut("UShort", 0xB849, ptr + 45)
        , NumPut("UPtr", Handle, ptr + 47), NumPut("UChar", 0xBA, ptr + 55)
        , NumPut("UInt", Msg, ptr + 56), NumPut("UShort", 0xB948, ptr + 60)
        , NumPut("UPtr", hWnd, ptr + 62), NumPut("UInt", 0xC18941, ptr + 70)
        , NumPut("UShort", 0x15FF, ptr + 73), NumPut("UInt", -71, ptr + 75)
        , NumPut("UInt", 0x20C48348, ptr + 79), NumPut("UShort", 0xC35B, ptr + 83)
    }
    return ptr + A_PtrSize*2
}

If that works, you could try changing UnFreeze(NextStuff, 200) to SetTimer NextStuff, -200 to figure out whether UnFreeze is necessary at all.

EDIT: you could also try running Invoke in an actually new thread. Something like this:

Code: Select all

cadmaticWindow := UIA.ElementFromHandle("Project ahk_exe UIManager.exe")
el := cadmaticWindow.WaitElement({Type:"Button",Name:buttonName},timeOut := 1000)

hProc := DllCall("OpenProcess", "UInt", 0x100000, "UInt", 0, "UInt", DllCall("GetCurrentProcessId"), "Ptr")
hThread := DllCall("CreateThread", "Ptr", 0, "Ptr", 0, "Ptr", NumGet(NumGet(el.InvokePattern.ptr, "Ptr"), 3*A_PtrSize, "ptr"), "Ptr", el.InvokePattern.ptr, "UInt", 0, "Ptr", 0) ; Run Invoke in a new thread
timeOut := A_TickCount+3000
While (DllCall("WaitForSingleObject", "Ptr", hThread, "Int", 200) != 0 && A_TickCount < timeOut) ; Wait maximum three seconds for thread to terminate
    Sleep 40
DllCall("CloseHandle", "Ptr", hProc) ; Maybe should terminate thread beforehand?

[fortpierre]

Hello @Descolada , thanks you for your very usefull liibrary !

I want to use it to automate some of my work and share it with my collegues, but working in a law environment I was wondering under what license it falls into ? I saw that some of your others works are under MIT 2 or GPL but it isn't mark for this one.

Thans you very much !

[Descolada]

@fortpierre, thanks for bringing my attention to this - I've now updated my libraries to fall under the MIT license.