@aifritz, glad to hear that
LocalizedControlType does mean that it uses the local language (eg German if your system is using German as the main language), so you can set your system language to English to always see English names... Though I'll consider adding an extra field to UIAViewer to also display the English equivalent from UIA_Enum.UIA_ControlTypeIds method.
UIAutomation with a focus on Chrome
Re: UIAutomation with a focus on Chrome
Hi @Descolada it seems so, as something is broken in your last update.
In the previous version this code was working fine for me:
In the lastest version, it gives me this error
---------------------------
ahkit.ahk
---------------------------
Error in
Method call not supported by UIA_Browser nor UIA_Interface or UIA_Element class or an error was encountered.
Specifically: FindAllByType
Line#
1899: cUIA := new UIA_Browser("A")
---> 1900: MsgBox,4096,,cUIA.FindAllByType(50000).length()
The current thread will exit.
---------------------------
OK
---------------------------
In the previous version this code was working fine for me:
Code: Select all
cUIA := new UIA_Browser("A")
msgbox,4096,, % cUIA.FindAllByType(50000).length()
---------------------------
ahkit.ahk
---------------------------
Error in
Method call not supported by UIA_Browser nor UIA_Interface or UIA_Element class or an error was encountered.
Specifically: FindAllByType
Line#
1899: cUIA := new UIA_Browser("A")
---> 1900: MsgBox,4096,,cUIA.FindAllByType(50000).length()
The current thread will exit.
---------------------------
OK
---------------------------
Re: UIAutomation with a focus on Chrome
@aifritz, thanks for notifying, it has been fixed. The bug only involved FindAllByType method, no other methods nor UIA_Browser was involved. Removing UIA_Constants involved changing a lot of constant names, it seems at least one got away from me
@Loop, I also tested your code in fresh Windows 10 and Windows 11 installs, and SetURL worked fine. Perhaps you have an old version of UIA_Browser or UIA_Interface somewhere in one of the Lib folders that AHK automatically checks, and it uses the wrong one?
@Loop, I also tested your code in fresh Windows 10 and Windows 11 installs, and SetURL worked fine. Perhaps you have an old version of UIA_Browser or UIA_Interface somewhere in one of the Lib folders that AHK automatically checks, and it uses the wrong one?
Re: UIAutomation with a focus on Chrome
Please excuse the inconvenience!
I have translated the string "Address and search bar" into local language then it works for me too
I have translated the string "Address and search bar" into local language then it works for me too
-
- Posts: 143
- Joined: 25 Dec 2020, 12:26
Re: UIAutomation with a focus on Chrome
@Descolada
do you know where to find the information "behind" the back button? i want to get the url of the webpage prior to the current one without actually going back to it nor right clicking the back button for a list of the recent webpages. any ideas?
another one: are you aware of a method in uia which can create a new tab next to the current one (not appending it at the end) and behaves a bit like a subordinate which goes back to its "mother" tab instead of focusing the next one (on its righthand side) when it's been closed. any clues?
do you know where to find the information "behind" the back button? i want to get the url of the webpage prior to the current one without actually going back to it nor right clicking the back button for a list of the recent webpages. any ideas?
another one: are you aware of a method in uia which can create a new tab next to the current one (not appending it at the end) and behaves a bit like a subordinate which goes back to its "mother" tab instead of focusing the next one (on its righthand side) when it's been closed. any clues?
Re: UIAutomation with a focus on Chrome
Hello @Descolada ,
I don't know if you've received my private message but I wanted to ask it here also:
I have a question, for example I have found an Element by Name (for example: Link) and got it's rectangles, but I want to click 30 or 50 pixels to the right from the Link box, because the link text is changing so I cannot use the link text, I need to use the Link label and click 30 or 50 pixels to right.
How do I do that?
Would this work like this?
I don't know if you've received my private message but I wanted to ask it here also:
I have a question, for example I have found an Element by Name (for example: Link) and got it's rectangles, but I want to click 30 or 50 pixels to the right from the Link box, because the link text is changing so I cannot use the link text, I need to use the Link label and click 30 or 50 pixels to right.
How do I do that?
Code: Select all
element := chrome.FindFirstByName("Link")
link := element + 50
Click, link
Would this work like this?
Re: UIAutomation with a focus on Chrome
@newbie007, it seems I have missed your PM. :/ One way would be to get element coordinates from CurrentBoundingRectangle, and then link element with ElementFromPoint. You could also just use regular Click to click:
Other way would be to create a TreeWalker with the link property and get the next sibling element, then click it.
@SundayProgrammer, I don't know how to get the information from the Back button, nor how to long-click it to display the history, but I will investigate it sometime in the near future. But sometimes you can get the previous website URL with javascript document.referrer: MsgBox, % cUIA.JSReturnThroughClipboard("document.referrer;")
And I am not aware of any UIA method of opening a new tab next to the current one. If you have a link on the webpage, then it might be possible to simulate a middle click, but it's probably easier to use AHKs Click function to middle click (you can get the coordinates with CurrentBoundingRectangle or CurrentPos()). Javascript might work also, for example cUIA.JSExecute("window.open('https://google.com');").
Code: Select all
element := chrome.FindFirstByName("Link")
bounds := element.CurrentBoundingRectangle ; Returns screen coordinates: l: left, t: top, r: right, b: bottom
;One way:
link := chrome.ElementFromPoint(bounds.r + 50, bounds.t + ((bound.b - bounds.t) // 2))
link.Click()
;Other way:
CoordMode, Mouse, Screen
Click, % (bounds.r + 50) " " (bounds.t + ((bound.b - bounds.t) // 2)) ; Click 50px right from the right border of element, and y coordinate is the middle of element y-coord
Code: Select all
tw := chrome.CreateTreeWalker(chrome.CreateCondition("ControlType", "Link"))
link := tw.GetNextSiblingElement(element)
link.Click()
And I am not aware of any UIA method of opening a new tab next to the current one. If you have a link on the webpage, then it might be possible to simulate a middle click, but it's probably easier to use AHKs Click function to middle click (you can get the coordinates with CurrentBoundingRectangle or CurrentPos()). Javascript might work also, for example cUIA.JSExecute("window.open('https://google.com');").
- Joe Glines
- Posts: 771
- Joined: 30 Sep 2013, 20:49
- Location: Dallas
- Contact:
Re: UIAutomation with a focus on Chrome
Sign-up for the HK Newsletter
AHK Tutorials:Web Scraping | | Webservice APIs | AHK and Excel | Chrome | RegEx | Functions
Training: AHK Webinars Courses on AutoHotkey
YouTube
Quick Access Popup, the powerful Windows folders, apps and documents launcher!
AHK Tutorials:Web Scraping | | Webservice APIs | AHK and Excel | Chrome | RegEx | Functions
Training: AHK Webinars Courses on AutoHotkey
YouTube
Quick Access Popup, the powerful Windows folders, apps and documents launcher!
-
- Posts: 143
- Joined: 25 Dec 2020, 12:26
Re: UIAutomation with a focus on Chrome
Descolada wrote: ↑22 Jun 2022, 16:06... I will investigate it sometime in the near future
...cUIA.JSExecute("window.open('https://google.com');")
Re: UIAutomation with a focus on Chrome
@Joe Glines, glad to see UIAutomation getting more popular! Hopefully more people can start enjoying it (and motivates me to fix important bugs faster )
Re: UIAutomation with a focus on Chrome
Learned about this by getting an email from Joe, really looking forward to testing this, my main engineering program is losing old controls in favour of UIA each and every update.
Re: UIAutomation with a focus on Chrome
Descolada wrote: ↑22 Jun 2022, 16:06@newbie007, it seems I have missed your PM. :/ One way would be to get element coordinates from CurrentBoundingRectangle, and then link element with ElementFromPoint. You could also just use regular Click to click:Other way would be to create a TreeWalker with the link property and get the next sibling element, then click it.Code: Select all
element := chrome.FindFirstByName("Link") bounds := element.CurrentBoundingRectangle ; Returns screen coordinates: l: left, t: top, r: right, b: bottom ;One way: link := chrome.ElementFromPoint(bounds.r + 50, bounds.t + ((bound.b - bounds.t) // 2)) link.Click() ;Other way: CoordMode, Mouse, Screen Click, % (bounds.r + 50) " " (bounds.t + ((bound.b - bounds.t) // 2)) ; Click 50px right from the right border of element, and y coordinate is the middle of element y-coord
@SundayProgrammer, I don't know how to get the information from the Back button, nor how to long-click it to display the history, but I will investigate it sometime in the near future. But sometimes you can get the previous website URL with javascript document.referrer: MsgBox, % cUIA.JSReturnThroughClipboard("document.referrer;")Code: Select all
tw := chrome.CreateTreeWalker(chrome.CreateCondition("ControlType", "Link")) link := tw.GetNextSiblingElement(element) link.Click()
And I am not aware of any UIA method of opening a new tab next to the current one. If you have a link on the webpage, then it might be possible to simulate a middle click, but it's probably easier to use AHKs Click function to middle click (you can get the coordinates with CurrentBoundingRectangle or CurrentPos()). Javascript might work also, for example cUIA.JSExecute("window.open('https://google.com');").
Thanks for your answer
I have tried both methods.. both didn't work.
I tried to output what CurrentBoundingRectangle returns, but I get nothing returned, so it doesn't send any clicks.
the second method is clicking but relative to the mouse, it doesn't take the rectangle as offset.
I would preffer to use the first method, since it react directly on chrome, but idk what I do wrong.
code:
Code: Select all
object1 := chrome.WaitElementExistByName("Link")
bounds := object1.CurrentBoundingRectangle() ; <------------- I have tried both with and without brackets ()
link := chrome.ElementFromPoint(bounds.r + 30, bounds.t + ((bound.b - bounds.t) // 2))
link.Click()
Re: UIAutomation with a focus on Chrome
Not sure if that setting can be changed for this thread (feel free to request it with my fellow moderators/forum admins), but I’d recommend to create the posting as a list of consecutive spoilered sections. That way the whole thread appears less bloated ("tl:dr")
For that, the forum offers the spoiler2= feature that allows to label them differently ie like their headlines.
Thx for your amazing work/effort.
Bo
PS. meanwhile a moderator can move your example section right under your initial posting, if requested.
Re: UIAutomation with a focus on Chrome
@newbie007, right now its hard to guess what is going wrong here, needs some debugging.
First of all make sure you are getting the element you want to get with Dump().
Once you know you have the correct element, use CurrentBoundingRectangle (without the parenthesis, because its a property not a method) and display its output, making sure the numbers are correct (check with UIAViewer or Accessibility Insights).
Then try ElementFromPoint and display the found elements information with Dump(), making sure it found the right element. For example, sometimes UIA doesn't find the deepest element (this is not related to my library, UIA is just implemented that way) so you might need to do some additional filtering on child elements.
The TreeWalker example might not be finding the correct element because your desired link might not be the next one (NextSibling), perhaps its the one over or inside a different container element altogether. This is hard to tell exactly without knowing what page you are working with... Inspect the elements with UIAViewer and display the tree for the whole window, then see where your elements are located relative to each other and modify your code accordingly.
@BoBo, thanks for the tip, working on it
First of all make sure you are getting the element you want to get with Dump().
Once you know you have the correct element, use CurrentBoundingRectangle (without the parenthesis, because its a property not a method) and display its output, making sure the numbers are correct (check with UIAViewer or Accessibility Insights).
Then try ElementFromPoint and display the found elements information with Dump(), making sure it found the right element. For example, sometimes UIA doesn't find the deepest element (this is not related to my library, UIA is just implemented that way) so you might need to do some additional filtering on child elements.
Code: Select all
#SingleInstance, force
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
SetTitleMatchMode, 2
#include <UIA_Interface>
#include <UIA_Browser>
browserExe := "chrome.exe"
Run, %browserExe% -incognito --force-renderer-accessibility "https://www.autohotkey.com/"
WinWaitActive, ahk_exe %browserExe%
cUIA := new UIA_Browser("ahk_exe " browserExe)
cUIA.WaitPageLoad("AutoHotkey",,1000)
dlLink := cUIA.FindFirstByName("Download")
MsgBox, % dlLink.DumpAll()
dlLinkLoc := dlLink.CurrentBoundingRectangle
MsgBox, % cUIA.PrintArray(dlLinkLoc)
forumsLink := SmallestElementFromPoint(cUIA, dlLinkLoc.r+100, dlLinkLoc.t + ((dlLinkLoc.b-dlLinkLoc.t) // 2))
MouseMove, % (dlLinkLoc.r+100), % (dlLinkLoc.t + ((dlLinkLoc.b-dlLinkLoc.t) // 2)) ; Moves the mouse cursor to the point that was used to look for the element.
MsgBox, % forumsLink.Dump()
forumsLink.Click()
SmallestElementFromPoint(IUIA, x="", y="", activateChromiumAccessibility=False) {
element := IUIA.ElementFromPoint(x, y, activateChromiumAccessibility)
bound := element.CurrentBoundingRectangle, elementSize := (bound.r-bound.l)*(bound.b-bound.t), prevElementSize := 0
for k, v in element.FindAll(IUIA.TrueCondition) {
bound := v.CurrentBoundingRectangle
if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b) && ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize))
element := v, elementSize := newSize
}
return element
}
@BoBo, thanks for the tip, working on it
Last edited by Descolada on 24 Jun 2022, 11:00, edited 2 times in total.
Re: UIAutomation with a focus on Chrome
I could tell you and maybe we could work together on it. Do you have Discord? If yes, I would be happy if you private message me your addy.Descolada wrote: ↑24 Jun 2022, 10:44@newbie007, right now its hard to guess what is going wrong here, needs some debugging.
First of all make sure you are getting the element you want to get with Dump().
Once you know you have the correct element, use CurrentBoundingRectangle (without the parenthesis, because its a property not a method) and display its output, making sure the numbers are correct (check with UIAViewer or Accessibility Insights).
Then try ElementFromPoint and display the found elements information with Dump(), making sure it found the right element. For example, sometimes UIA doesn't find the deepest element (this is not related to my library, UIA is just implemented that way) so you might need to do some additional filtering on child elements.The TreeWalker example might not be finding the correct element because your desired link might not be the next one (NextSibling), perhaps its the one over or inside a different container element altogether. This is hard to tell exactly without knowing what page you are working with... Inspect the elements with UIAViewer and display the tree for the whole window, then see where your elements are located relative to each other and modify your code accordingly.Code: Select all
#SingleInstance, force #NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases. ; #Warn SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory. SetTitleMatchMode, 2 #include <UIA_Interface> #include <UIA_Browser> browserExe := "chrome.exe" Run, %browserExe% -incognito --force-renderer-accessibility "https://www.autohotkey.com/" WinWaitActive, ahk_exe %browserExe% cUIA := new UIA_Browser("ahk_exe " browserExe) cUIA.WaitPageLoad("AutoHotkey",,1000) dlLink := cUIA.FindFirstByName("Download") MsgBox, % dlLink.DumpAll() dlLinkLoc := dlLink.CurrentBoundingRectangle MsgBox, % cUIA.PrintArray(dlLinkLoc) forumsLink := SmallestElementFromPoint(cUIA, dlLinkLoc.r+100, dlLinkLoc.t + ((dlLinkLoc.b-dlLinkLoc.t) // 2)) MouseMove, % (dlLinkLoc.r+100), % (dlLinkLoc.t + ((dlLinkLoc.b-dlLinkLoc.t) // 2)) ; Moves the mouse cursor to the point that was used to look for the element. MsgBox, % forumsLink.Dump() forumsLink.Click() SmallestElementFromPoint(IUIA, x="", y="", activateChromiumAccessibility=False) { element := IUIA.ElementFromPoint(x, y, activateChromiumAccessibility) bound := element.CurrentBoundingRectangle, elementSize := (bound.r-bound.l)*(bound.b-bound.t), prevElementSize := 0 for k, v in element.FindAll(IUIA.TrueCondition) { bound := v.CurrentBoundingRectangle if ((x >= bound.l) && (x <= bound.r) && (y >= bound.t) && (y <= bound.b) && ((newSize := (bound.r-bound.l)*(bound.b-bound.t)) < elementSize)) element := v, elementSize := newSize } return element }
@BoBo, thanks for the tip, working on it
Re: UIAutomation with a focus on Chrome
Is there anything you can do to fix the UIA_Viewer's ability to detect the min/max/close buttons in Chrome?
Re: UIAutomation with a focus on Chrome
@Skrell, there is something about the buttons that they can't be directly visualized by hovering over, this is not only for UIAViewer but for other tools (Accessibility Insights, Inspect.exe) as well. However when you click "Construct tree for whole Window" and scroll all the way down, the buttons are somewhere under "pane "Google Chrome"". Then when you click on them, the blue box will show the correct place.
Re: UIAutomation with a focus on Chrome
Is there any way roll this functionality automatically into the default UIViewer? This is literally the ONLY thing I've found not working 100% correctly in this amazing library!Descolada wrote: ↑27 Jun 2022, 08:12@Skrell, there is something about the buttons that they can't be directly visualized by hovering over, this is not only for UIAViewer but for other tools (Accessibility Insights, Inspect.exe) as well. However when you click "Construct tree for whole Window" and scroll all the way down, the buttons are somewhere under "pane "Google Chrome"". Then when you click on them, the blue box will show the correct place.
Re: UIAutomation with a focus on Chrome
Added some TextPattern and TextRange examples to the second post in the thread. Also updated UIA_Interface.ahk with many bugfixes and some new features (eg UIA_Element.CurrentExists, WaitElementNotExist, ControlClick elements).
@Skrell, I updated UIAViewer.ahk to do what you wished. There is now a global DeepSearchFromPoint := False line at the top, when set to True will also find Chromes min-max buttons. This will impact the performance though...
@Skrell, I updated UIAViewer.ahk to do what you wished. There is now a global DeepSearchFromPoint := False line at the top, when set to True will also find Chromes min-max buttons. This will impact the performance though...
Return to “Scripts and Functions (v1)”
Who is online
Users browsing this forum: No registered users and 88 guests