UIAutomation with a focus on Chrome
Re: UIAutomation with a focus on Chrome
This is wonderful. Thank you!
Re: UIAutomation with a focus on Chrome
Hi @Descolada
I am trying to use FindFirstWithOptions but I can't get it to work. Do you have an example in AHK that can help me?
I am trying to use FindFirstWithOptions but I can't get it to work. Do you have an example in AHK that can help me?
Re: UIAutomation with a focus on Chrome
@mora145, currently FindFirstWithOptions is only useful for the TreeTraversalOptions_LastToFirstOrder option:
Element.FindFirstWithOptions(4,"Type=Button", UIA.TreeTraversalOptions_LastToFirstOrder) would find the last element of type "Button" (scope is TreeScope_Descendants). Unfortunately the "root" argument is currently broken and I'm querying Microsofts Q&A as to why (possibly a buggy implementation). Also my testing has shown that if not using LastToFirstOrder, then the TreeScope gets completely ignored (seems like another bug). So the example I provided is the only useful way of using it unfortunately
EDIT: I'm completely mistaken, FindFirstWithOptions is working exactly as it should be, it's just that its documentation is horrible. It seems that the root argument element must be a *parent* of the starting point element, otherwise it won't work properly. If root is omitted, then it will start from the Root Element (this will be fixed in a later UIA_Interface.ahk release). And this should explain PostOrder travel. I've written some examples (ran on Windows 10) you can study:
Element.FindFirstWithOptions(4,"Type=Button", UIA.TreeTraversalOptions_LastToFirstOrder) would find the last element of type "Button" (scope is TreeScope_Descendants). Unfortunately the "root" argument is currently broken and I'm querying Microsofts Q&A as to why (possibly a buggy implementation). Also my testing has shown that if not using LastToFirstOrder, then the TreeScope gets completely ignored (seems like another bug). So the example I provided is the only useful way of using it unfortunately
EDIT: I'm completely mistaken, FindFirstWithOptions is working exactly as it should be, it's just that its documentation is horrible. It seems that the root argument element must be a *parent* of the starting point element, otherwise it won't work properly. If root is omitted, then it will start from the Root Element (this will be fixed in a later UIA_Interface.ahk release). And this should explain PostOrder travel. I've written some examples (ran on Windows 10) you can study:
Code: Select all
#SingleInstance, force
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
#Warn
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
SetTitleMatchMode, 2
#include Lib\UIA_Interface.ahk
Run notepad.exe
WinWaitActive ahk_exe notepad.exe
UIA := UIA_Interface()
npEl := UIA.ElementFromHandle("ahk_exe notepad.exe")
npEl.FindFirst("Type=Document or Type=Edit").Value := npEl.DumpAll()
postOrder := (npEl.FindFirst("Type=MenuBar")).FindAllWithOptions(2, "", UIA.TreeTraversalOptions_PostOrder, npEl)
out := ""
for i, el in postOrder
out .= i ": " el.Dump() "`n"
MsgBox, % "FindAll with PostOrder. `nThe order of elements is exactly how the tree was traversed.`nNote that TreeScope_Children contains now both the starting element and Notepad element children.`n" out
MsgBox, % "Looking for first MenuItem: " npEl.FindFirstWithOptions(, "Type=MenuItem").Dump() "`n" ; Works like regular FindFirst
MsgBox, % "Looking for first MenuItem starting from the end: " npEl.FindFirstWithOptions(, "Type=MenuItem", UIA.TreeTraversalOptions_LastToFirstOrder).Dump() "`n" ; Returns LAST MenuItem
DownButton := npEl.FindFirst("AutomationId=DownButton")
MsgBox, % "Looking for UpButton starting from DownButton and looking forwards: " DownButton.FindFirstWithOptions(4, "AutomationId=UpButton",, npEl).Dump() "`n" ; Should be empty, because "Line up" button is before "Line down"
MsgBox, % "Looking for UpButton starting from DownButton and looking backwards: " DownButton.FindFirstWithOptions(4, "AutomationId=UpButton", UIA.TreeTraversalOptions_LastToFirstOrder, npEl).Dump() "`n"
MsgBox, % "Looking for UpButton starting from DownButton and looking backwards, TreeScope_Element: " DownButton.FindFirstWithOptions(1, "AutomationId=UpButton", UIA.TreeTraversalOptions_LastToFirstOrder, npEl).Dump() "`n" ; TreeScope seems to be ignored
MsgBox, % "Looking for UpButton starting from DownButton and looking backwards, TreeScope_Descendants: " DownButton.FindFirstWithOptions(4, "AutomationId=NonClientVerticalScrollBar", UIA.TreeTraversalOptions_LastToFirstOrder, npEl).Dump() "`n" ; Starts from DownButton, but doesn't find the parent ScrollBar because the tree node has already been "visited"
MsgBox, % "Looking for UpButton starting from DownButton and looking backwards, TreeScope_Descendants: " DownButton.FindFirstWithOptions(4, "AutomationId=NonClientVerticalScrollBar", UIA.TreeTraversalOptions_LastToFirstOrder | UIA.TreeTraversalOptions_PostOrder, npEl).Dump() "`n" ; Finds the parent ScrollBar, because it's traveling in PostOrder
ExitApp
Re: UIAutomation with a focus on Chrome
Excelentísimo! Gracias @Descolada. Prometo que haré una donación cuando reciba algún dinero. Gran trabajo con tu librería. Eres lo máximo explicando, gracias por tu tiempo.
Re: UIAutomation with a focus on Chrome
I have read with interest this whole thread (thank you so much Descolada and others) and checked out other topics in V1 (e.g. viewtopic.php?p=391568 - getting the URL from Firefox, and also another one getting the video links from a Youtube page) to try and solve the thing I am trying to do. But I have been unable to successfully translate their code into v2. I have been trying to construct an independent, v2 code, to scroll the Win10 clipboard when Excel 2010 is active. The problem being that Excel captures all the scroll messages when it is active. The following code using the UIAutomation.ahk class of thqby https://github.com/thqby/ahk2_lib/blob/master/UIAutomation/UIAutomation.ahk works well as can be demonstrated with my code below, But I would very much appreciate it if someone could help me construct some self-contained code in v2 presumably utilizing ComObject(CLSID_CUIAutomation := "{FF48DBA4-60EF-4201-AA87-54103EEF594E}", IID_IUIAutomation := "{30CBE57D-D9D0-452A-AB13-7AC5AC4825EE}") and preferably ComCall's (or DllCall's).
Code: Select all
#Warn
#SingleInstance Force
SetTitleMatchMode 2
#Requires AutoHotkey v2.0+ 64-bit
#Include <UIAutomation>
oUIA := UIA() ; Initialize UIA interface
; cv_UIA := ComObject(CLSID_CUIAutomation := "{FF48DBA4-60EF-4201-AA87-54103EEF594E}", IID_IUIAutomation := "{30CBE57D-D9D0-452A-AB13-7AC5AC4825EE}")
; p_cv_UIA := Format("0x{:X}", ComObjValue(cv_UIA))
WheelDown::
WheelUp:: {
MouseGetPos ,,&hWndM
cbEl := UIA.ElementFromHandle(hWndM) ; Get the element for the Clipboard
cbLV := cbEl.FindControl("List")
cbScrollPat := cbLV.GetCurrentPattern("Scroll")
if (A_ThisHotkey = "WheelDown")
cbScrollPat.Scroll(2,4) ; 2 = no horizontal scroll; 4 = small scroll down
else
cbScrollPat.Scroll(2,1) ; 2 = no horizontal scroll; 1 = small scroll up
}
Esc:: {
ExitApp
}
Re: UIAutomation with a focus on Chrome
@ludamo, perhaps somebody else can help you more thoroughly, but me myself have decided not to help with translations of UIA libraries code to raw Dll/ComCalls, because it's annoyingly time intensive. But you could take a look at this Reddit post, where there is raw UIA v2 code, you could base yours on that. UIA_ListControlTypeId = 50008, and ScrollPatternId = 10004.
Re: UIAutomation with a focus on Chrome
Thanks for the reply and that link. That example works in Notepad (doesn't work in Notepad++ or Word 2010) so I shall try and adapt the code to my goal and hopefully report back.
Re: UIAutomation with a focus on Chrome
It works! Many thanks again. Would you care to comment if I need to do ObjRelease or something like that in the code?
Code: Select all
#Requires AutoHotkey v2.0+ 64-bit
#SingleInstance Force
IUIA := ComObject("{E22AD333-B25F-460C-83D0-0581107395C9}", "{34723AFF-0C9D-49D0-9896-7AB52DF8CD8A}")
VT_I4 := 3 ; identifies a 32-bit signed int
WheelDown::
WheelUp:: {
MouseGetPos( ,, &hWndM)
ComCall(6, IUIA, "ptr", hWndM, "ptr*", &ElfromHnd:=0) ; ElementFromHandle
var := Buffer(24,0), NumPut("short",VT_I4,var,0), NumPut("ptr", 50008, var, 8) ; UIA_ListControlTypeId = 50008
ComCall(23, IUIA, "int", 30003, "ptr", var, "ptr*", &listCond:=0) ; CreatePropertyCondition
ComCall(5, ElfromHnd, "int", 4, "ptr", listCond, "ptr*", &listCB:=0) ; FindFirst
ComCall(16, listCB, "int", 10004, "ptr*", &patternObject := 0) ; get Pattern ScrollPatternId = 10004
if (A_ThisHotkey = "WheelDown")
ComCall(3, patternObject, "int", 2, "int", 4) ; 2 = no horizontal scroll; 4 = small scroll down
else
ComCall(3, patternObject, "int", 2, "int", 1) ; 2 = no horizontal scroll; 4 = small scroll up
}
Re: UIAutomation with a focus on Chrome
@ludamo, AFAIK you have to call ObjRelease on every object you get from UIA, because UIA calls AddRef on objects before sending them over to AHK, but the task of releasing it is left to the AHK side because UIA can't know when you are done with using the object, so you need to Release it manually when done with it. Otherwise the garbage collector can't delete it and you'll get a memory leak.
-
- Posts: 90
- Joined: 22 Jul 2016, 16:28
Re: UIAutomation with a focus on Chrome
Hi, @Descolada
I've noticed, the function SmallestElementFromPoint, only works when the mouse cursor is an arrow, when hovering a link (the arrow turns into a hand and when hovering a edit, the mouse turns into a caret) the code will not work:
Example:
MouseGetPos, OutputVarX, OutputVarY
Msgbox % cUIA.SmallestElementFromPoint(OutputVarX, OutputVarY)
Edit: The problem is the MouseGetPos function, it is hanging when the mouse is a hand! Sorry
only works on chrome when the cursor is in the normal state, a arrow. (tested on chrome)
I've noticed, the function SmallestElementFromPoint, only works when the mouse cursor is an arrow, when hovering a link (the arrow turns into a hand and when hovering a edit, the mouse turns into a caret) the code will not work:
Example:
MouseGetPos, OutputVarX, OutputVarY
Msgbox % cUIA.SmallestElementFromPoint(OutputVarX, OutputVarY)
Edit: The problem is the MouseGetPos function, it is hanging when the mouse is a hand! Sorry
only works on chrome when the cursor is in the normal state, a arrow. (tested on chrome)
Re: UIAutomation with a focus on Chrome
@leosouza85, works for me using AHK 1.1.36.02. When the cursor is a "hand" then it hangs for a second before returning the correct element: this is because the normal ElementFromPoint is returning the main document element, so the script has to sort through all the elements in the document until it finds the correct smallest element under the cursor. This unfortunately takes some time and makes the code laggy, but in my setup it still works...
Code: Select all
SetTitleMatchMode, 2
#include Lib\UIA_Interface.ahk
#include Lib\UIA_Browser.ahk
CoordMode, Mouse, Screen
cUIA := new UIA_Browser("ahk_exe chrome.exe")
Loop {
MouseGetPos, OutputVarX, OutputVarY
ToolTip % "Mouse X: " OutputVarX " Y: " OutputVarY
. "`nElement under mouse: " cUIA.SmallestElementFromPoint(OutputVarX, OutputVarY).Dump()
. "`nAHK version: " A_AhkVersion
; try ToolTip % "Mouse X: " OutputVarX " Y: " OutputVarY
; . "`nElement under mouse: " cUIA.ElementFromPoint(OutputVarX, OutputVarY).Dump()
}
Re: UIAutomation with a focus on Chrome
Code: Select all
......"ptr*", &ElfromHnd:=0)
if ure working with raw pointers, yes
if ure wrapping them in comvalues, no - the comvalue will release it automatically whenever it goes out of scope
Code: Select all
.."ptr*", ElfromHnd := ComValue(9, 0))
Re: UIAutomation with a focus on Chrome
Thanks swagfag, I have added that to my code.
-
- Posts: 90
- Joined: 22 Jul 2016, 16:28
Re: UIAutomation with a focus on Chrome
hi @Descolada
In UIA Tree, we have a lot of Elements without any Name, usually children of something that have a name... lets suppose we have the following:
Group (name = television)
-Button (name = "")
--Text (name = "")
Maybe there will be a good thing if UIA Interface gives the name of the parent to the nameless children, so if the tree construction saw the above structure, it could read as:
Group (name = television)
-Button (name = television)
--Text (name = television)
Maybe with a code to infer it is an auto naming, like
Group (name = television)
-Button (name = television c1)
--Text (name = television c1c1)
c1 meaning child 1
What do you think? I think it would be great to speed up developing when having a lot of nameless childs... and also I think it is not difficult to implement.
So UIA Viewer and TreeInspector could show this names, and the findfunctions will translate thet to find the correspondent nameless element
In UIA Tree, we have a lot of Elements without any Name, usually children of something that have a name... lets suppose we have the following:
Group (name = television)
-Button (name = "")
--Text (name = "")
Maybe there will be a good thing if UIA Interface gives the name of the parent to the nameless children, so if the tree construction saw the above structure, it could read as:
Group (name = television)
-Button (name = television)
--Text (name = television)
Maybe with a code to infer it is an auto naming, like
Group (name = television)
-Button (name = television c1)
--Text (name = television c1c1)
c1 meaning child 1
What do you think? I think it would be great to speed up developing when having a lot of nameless childs... and also I think it is not difficult to implement.
So UIA Viewer and TreeInspector could show this names, and the findfunctions will translate thet to find the correspondent nameless element
Re: UIAutomation with a focus on Chrome
@leosouza85, I think I understood the problem you are describing and I've thought about your proposed solution for a bit.
Essentially what you want from this is for nameless elements to have some unique identifier to more easily access them. Something similar to a path, but also filtering for only nameless elements. So a wrapper for something like El.FindFirstBy("Name=television").FindAllByName(,2)[1].FindAllByName(,2)[1] which would return the "television c1c1" element.
Now, this proposed solution has similar kind of issues like the numeric path from DumpAll: it can get shifted/changed when elements are added to the tree. So if you were to add an element:
You see that the "name-path" for Text element changed and would not be found now. So why not also consider the type of the element? For example:
This would avoid such collisions.. But why stop at that?
What I've been thinking is why not try to create as unique a path as possible for every element? For example we could consider Type, Name, and AutomationId:
Then perhaps we could chain those GeneratedId's together to create a more reliable "path": El.FindByGeneratedPath("name=television and type=group and automationid=-->GeneratedId = name= and type=button and automationid=-->GeneratedId = name= and type=Text and automationid=").
But I don't like how long paths it creates... I'm gonna think about this a bit more
EDIT: If we are always using the same properties, then the path could be shortened to a certain order of properties, and the type can be replaced with the typeId minus 50000 (because they are all 50000 something). So it could be shortened to something like "television|26|-->|0|-->|20|"
Code: Select all
Group (name = television)
-Button (name = television c1)
--Text (name = television c1c1)
Now, this proposed solution has similar kind of issues like the numeric path from DumpAll: it can get shifted/changed when elements are added to the tree. So if you were to add an element:
Code: Select all
Group (name = television)
-List (name = television c1)
-Button (name = television c2)
--Text (name = television c2c1)
Code: Select all
Group (name = television)
-List (name = television List1)
-Button (name = television Button1)
--Text (name = television Button1Text1)
What I've been thinking is why not try to create as unique a path as possible for every element? For example we could consider Type, Name, and AutomationId:
Code: Select all
Group (name = television) (GeneratedId = television and type=group and automationid=)
-List (name = television c1) (GeneratedId = name= and type=List and automationid=)
-Button (name = television c2) (GeneratedId = name= and type=button and automationid=)
--Text (name = television c2c1) (GeneratedId = name= and type=Text and automationid=)
But I don't like how long paths it creates... I'm gonna think about this a bit more
EDIT: If we are always using the same properties, then the path could be shortened to a certain order of properties, and the type can be replaced with the typeId minus 50000 (because they are all 50000 something). So it could be shortened to something like "television|26|-->|0|-->|20|"
-
- Posts: 90
- Joined: 22 Jul 2016, 16:28
Re: UIAutomation with a focus on Chrome
Thank you! I think this is the best format, more friendly:Descolada wrote: ↑23 Jan 2023, 00:30@leosouza85, I think I understood the problem you are describing and I've thought about your proposed solution for a bit.Essentially what you want from this is for nameless elements to have some unique identifier to more easily access them. Something similar to a path, but also filtering for only nameless elements. So a wrapper for something like El.FindFirstBy("Name=television").FindAllByName(,2)[1].FindAllByName(,2)[1] which would return the "television c1c1" element.Code: Select all
Group (name = television) -Button (name = television c1) --Text (name = television c1c1)
Now, this proposed solution has similar kind of issues like the numeric path from DumpAll: it can get shifted/changed when elements are added to the tree. So if you were to add an element:You see that the "name-path" for Text element changed and would not be found now. So why not also consider the type of the element? For example:Code: Select all
Group (name = television) -List (name = television c1) -Button (name = television c2) --Text (name = television c2c1)
This would avoid such collisions.. But why stop at that?Code: Select all
Group (name = television) -List (name = television List1) -Button (name = television Button1) --Text (name = television Button1Text1)
What I've been thinking is why not try to create as unique a path as possible for every element? For example we could consider Type, Name, and AutomationId:Then perhaps we could chain those GeneratedId's together to create a more reliable "path": El.FindByGeneratedPath("name=television and type=group and automationid=-->GeneratedId = name= and type=button and automationid=-->GeneratedId = name= and type=Text and automationid=").Code: Select all
Group (name = television) (GeneratedId = television and type=group and automationid=) -List (name = television c1) (GeneratedId = name= and type=List and automationid=) -Button (name = television c2) (GeneratedId = name= and type=button and automationid=) --Text (name = television c2c1) (GeneratedId = name= and type=Text and automationid=)
But I don't like how long paths it creates... I'm gonna think about this a bit more
EDIT: If we are always using the same properties, then the path could be shortened to a certain order of properties, and the type can be replaced with the typeId minus 50000 (because they are all 50000 something). So it could be shortened to something like "television|26|-->|0|-->|20|"
You see that the "name-path" for Text element changed and would not be found now. So why not also consider the type of the element? For example:
Code: Select all
Group (name = television)
-List (name = television List1)
-Button (name = television Button1)
--Text (name = television Button1Text1)
Re: UIAutomation with a focus on Chrome
Hi Descolada, I have a problem getting an element. I spent all day trying to get the text from path 1.2.16, but I keep getting an empty element like this:
This is what my DumpAll returns when I get the window.
.
And to get the text I'm doing first.
hello := chrome.FindFirstBy("Name="phoneButton) get the 1.2.15 element.
That gets the path 1.2.15
Then with test := hello.FindByPath("+1",cUIA.CreateCondition("ControlType", "Text")) I try to get the 1.2.16 element but it appears empty, as in the image.
This is what my DumpAll returns when I get the window.
Code: Select all
1.2.15 Type: 50005 (Hyperlink) Name: "דברו איתי" Value: "tel:072-3301774" LocalizedControlType: "link"
1.2.15.1 Type: 50020 (Text) Name: "דברו איתי" LocalizedControlType: "text"
1.2.16 Type: 50020 (Text) Name: "טשרנחובסקי 10 , באר שבע" LocalizedControlType: "text"
1.2.17 Type: 50020 (Text) Name: "פנו לטכנאי" LocalizedControlType: "text"
And to get the text I'm doing first.
hello := chrome.FindFirstBy("Name="phoneButton) get the 1.2.15 element.
That gets the path 1.2.15
Then with test := hello.FindByPath("+1",cUIA.CreateCondition("ControlType", "Text")) I try to get the 1.2.16 element but it appears empty, as in the image.
Re: UIAutomation with a focus on Chrome
@mora145, if you Dump hello := chrome.FindFirstBy("Name="phoneButton), do you get the correct output of Type: 50005 (Hyperlink) Name: "דברו איתי" Value: "tel:072-3301774" LocalizedControlType: "link"?
You could try hello.FindByPath("p").DumpAll() to verify that the text element is found and get it by path from there, or try hello.FindByPath("+1") or hello.FindByPath("+1") to see what they return. Sometimes TreeWalker and FindAll (which DumpAll uses) results differ, so it might need some experimentation...
You could try hello.FindByPath("p").DumpAll() to verify that the text element is found and get it by path from there, or try hello.FindByPath("+1") or hello.FindByPath("+1") to see what they return. Sometimes TreeWalker and FindAll (which DumpAll uses) results differ, so it might need some experimentation...
Re: UIAutomation with a focus on Chrome
Ye, I obtain the correct.
I obtain that:You could try hello.FindByPath("p").DumpAll() to verify that the text element is found and get it by path from there, or try hello.FindByPath("+1") or hello.FindByPath("+1") to see what they return. Sometimes TreeWalker and FindAll (which DumpAll uses) results differ, so it might need some experimentation...
n this case, the text I am looking for is number 10.
Re: UIAutomation with a focus on Chrome
@mora145, I guess if FindByPath isn't working out at all, then you could use as a workaround hello.FindByPath("p.10")? Or something like this might work as well:
Code: Select all
TextTW := cUIA.CreateTreeWalker("Type=Text")
MsgBox, % TextTW.GetLastChildElement(hello.FindByPath("p")).Dump()
Return to “Scripts and Functions (v1)”
Who is online
Users browsing this forum: No registered users and 133 guests