UIAutomation with a focus on Chrome
Re: UIAutomation with a focus on Chrome
@Descolada thank you for your updates!
If I may turn on wish mode....I would have the following ideas to the UIAViewer:
- For faster code creation, I would find it helpful, if one could open a context menu in the UIAViewer in the TreeView by right clicking on an element, which shows you all access commands including properties and methods to copy this to the clipboard.
- A record mode functionality
If I may turn on wish mode....I would have the following ideas to the UIAViewer:
- For faster code creation, I would find it helpful, if one could open a context menu in the UIAViewer in the TreeView by right clicking on an element, which shows you all access commands including properties and methods to copy this to the clipboard.
- A record mode functionality
-
- Posts: 143
- Joined: 25 Dec 2020, 12:26
Re: UIAutomation with a focus on Chrome
@Descolada, is there a way to get the location of the current tab on chrome's tab bar? i'm imagining something like currenttabboundingrectangle, if it makes sense.
Re: UIAutomation with a focus on Chrome
Is it possible that a software doesn't expose any of it's controls? I am trying to use it on 3DEXPERIENCE from Dassault and it seems that controls are not accessible by any means. I've tried the UIA viewer
and it highlights basically the entire window. Building the tree looks like pic attached ACC viewer is similar And the regular window spy also doesn't show anything.Re: UIAutomation with a focus on Chrome
@aifritz, thanks for the suggestions! I'm not sure what you mean by "recording mode" though?
@SundayProgrammer, cUIA.MainPaneElement.FindFirstBy("ControlType=TabItem AND SelectionItemIsSelected=1") should return the current tab element, then you can use CurrentBoundingRectangle on that
@nt-_-ts, yes, unfortunately it is totally possible since implementing UIAutomation is usually left to the developers of the program. For most regular windows the UIA elements are automatically generated (for example old Win32 windows, most .NET windows etc), but when dealing with custom elements like in your program, then its up to the developer of the program whether to support accessibility or not. So you might just be out of luck here You could also check if that program is creating some hidden windows with separate GUIs (which is unlikely), you can use UIATreeInspector.ahk in my GitHub for that.
@SundayProgrammer, cUIA.MainPaneElement.FindFirstBy("ControlType=TabItem AND SelectionItemIsSelected=1") should return the current tab element, then you can use CurrentBoundingRectangle on that
@nt-_-ts, yes, unfortunately it is totally possible since implementing UIAutomation is usually left to the developers of the program. For most regular windows the UIA elements are automatically generated (for example old Win32 windows, most .NET windows etc), but when dealing with custom elements like in your program, then its up to the developer of the program whether to support accessibility or not. So you might just be out of luck here You could also check if that program is creating some hidden windows with separate GUIs (which is unlikely), you can use UIATreeInspector.ahk in my GitHub for that.
-
- Posts: 502
- Joined: 03 Dec 2018, 20:02
Re: UIAutomation with a focus on Chrome
When I use it for Opera the address bar errors, any idea?
When I check it with UIAViewer the control's name is indeed different, but even after I change the name it still has the same error.Error in #include file "D:\UIATest\Lib\UIA_Browser.ahk":
0x80004005 - Unspecified error.
Specifically: SetValue (UIA_ValuePattern)
Line#
270: WinActivate,"ahk_id" this.BrowserId
272: Return,this.GetCurrentDocumentElement().CurrentValue
273: }
274: }
276: {
277: this.URLEditElement.SetFocus()
278: valuePattern := this.URLEditElement.GetCurrentPatternAs("Value")
---> 279: valuePattern.SetValue(newUrl " ")
280: if !InStr(this.URLEditElement.CurrentValue, newUrl)
280: {
281: legacyPattern := this.URLEditElement.GetCurrentPatternAs("LegacyIAccessible")
282: legacyPattern.SetValue(newUrl " ")
283: legacyPattern.Select()
284: }
285: if (navigateToNewUrl&&InStr(this.URLEditElement.CurrentValue, newUrl))
The current thread will exit.
-
- Posts: 90
- Joined: 22 Jul 2016, 16:28
Re: UIAutomation with a focus on Chrome
@Descolada , congratulations for the excelent work... I have a sugestion for simplyfing the code writing, could you add a parametter to the find* functions, so if we put a flag like "1" it wait the element to exist?
Like var.FindFirstbyName("name", 1)
So it will wait for the name to exist and then return the first result.
Like var.FindFirstbyName("name", 1)
So it will wait for the name to exist and then return the first result.
Re: UIAutomation with a focus on Chrome
@leosouza85, I prefer the existing WaitElementExist functions, the name is more descriptive
@william_ahk, you can fix that by changing in GetCurrentMainPaneElement() method "Address and search bar" to "Address field". That said, UIA_Browser.ahk is mostly advertised for Chrome (as the thread title suggests) and I test it with Edge too, but I have no plans on supporting other browsers as well. There would be way too many browsers to test and exceptions to add...
@william_ahk, you can fix that by changing in GetCurrentMainPaneElement() method "Address and search bar" to "Address field". That said, UIA_Browser.ahk is mostly advertised for Chrome (as the thread title suggests) and I test it with Edge too, but I have no plans on supporting other browsers as well. There would be way too many browsers to test and exceptions to add...
-
- Posts: 143
- Joined: 25 Dec 2020, 12:26
Re: UIAutomation with a focus on Chrome
@Descolada, when the current tab is the first tab, it gave me l:171 t:0 r:308 b:33. and when the current tab goes for the rest, all gave me l:43 t:0 r:180 b:33.
and none of the above seems reflect the correct locations.
for vertical tab bar, all gave me blank.
i doesn't have a chrome actually, these came from my msedge . i assumed they work the same. probably not though .
Re: UIAutomation with a focus on Chrome
I didn't know that Edge even had vertical tabs, cool But it looks like it moves the tabs away from the main pane (eg where the URL bar is), and that's why it fails. I'll look into it in the next UIA_Browser update.. In the meanwhile cUIA.FindFirstBy("ControlType=TabItem AND SelectionItemIsSelected=1") is a temporary workaround, which might fail if the content has tab items (like Facebook has for example).
I'm not sure why this fails, it worked in my Edge. Could you call Dump() on the returned element and see if it's actually the tab or something else?SundayProgrammer wrote: ↑01 Jul 2022, 22:28@Descolada, when the current tab is the first tab, it gave me l:171 t:0 r:308 b:33. and when the current tab goes for the rest, all gave me l:43 t:0 r:180 b:33.
and none of the above seems reflect the correct locations.
-
- Posts: 502
- Joined: 03 Dec 2018, 20:02
Re: UIAutomation with a focus on Chrome
Thanks, that did workDescolada wrote: ↑01 Jul 2022, 13:28@william_ahk, you can fix that by changing in GetCurrentMainPaneElement() method "Address and search bar" to "Address field". That said, UIA_Browser.ahk is mostly advertised for Chrome (as the thread title suggests) and I test it with Edge too, but I have no plans on supporting other browsers as well. There would be way too many browsers to test and exceptions to add...
I do think we should support as many Chromium-based browsers as possible no? Since the difference is minimal. Maybe we can expose the crucial parts like address bar names to the user so that they could define them accordingly for any browser, and language as well.
Also may I present you a way to select and click elements by css selectors:
(A JSON library will be required)
Code: Select all
JSGetElementPos(selector) {
js =
(
(() => {
let bounds = document.querySelector("%selector%").getBoundingClientRect().toJSON();
let zoom = window.devicePixelRatio.toFixed(2);
for (const key in bounds) {
bounds[key] = bounds[key] * zoom;
}
return JSON.stringify(bounds);
})()
)
bounds_str := this.JSReturnThroughClipboard(js)
bounds := JSON.Load(bounds_str)
ControlGetPos, win_x, win_y, win_w, win_h, Chrome_RenderWidgetHostHWND1, % "ahk_id " this.BrowserId
bounds.x += win_x
bounds.y += win_y
return bounds
}
ClickElement(selector, emulateMouse=False) {
try
bounds := this.JSGetElementPos(selector)
catch e
throw Exception(e.Message, -1, e.What)
x := (bounds.x + bounds.width / 2), y := (bounds.y + bounds.height / 2)
if emulateMouse
Click % x " " y
else
ControlClick, % "x" x " y" y, % "ahk_id " this.BrowserId
}
Last edited by william_ahk on 04 Jul 2022, 04:11, edited 1 time in total.
-
- Posts: 143
- Joined: 25 Dec 2020, 12:26
Re: UIAutomation with a focus on Chrome
when the current tab is the first tab, this below dumped (which actually is the information of the second tab , so strange )
Type: 50019 Name: "Scripts and Functions - AutoHotkey Community" LocalizedControlType: "tab item" AutomationId: "view_24"
when the current tab goes for the rest, one by one, this below dumped (which actually is the information of the first tab , so strange ) for every one of them
Type: 50019 Name: "UIAutomation with a focus on Chrome - Page 5 - AutoHotkey Community" LocalizedControlType: "tab item" AutomationId: "view_24"
Re: UIAutomation with a focus on Chrome
@william_ahk, I am not at all opposed to GitHub pull requests for such changes For me, I decided that manually testing n browsers for every change I make in UIA_Browser is a real pain, so limiting to the 1-2 most popular ones seemed logical. I do have plans to expose the element names to the user (or at least give a way to change the default action), just haven't gotten around to it yet.
That JS CSS selector scripts seems very cool, I'll check it out soon. You probably could switch Click to ControlClick for a more reliable result And soon there might be a real need for UIA_Browser_JS.ahk extension!
That JS CSS selector scripts seems very cool, I'll check it out soon. You probably could switch Click to ControlClick for a more reliable result And soon there might be a real need for UIA_Browser_JS.ahk extension!
-
- Posts: 502
- Joined: 03 Dec 2018, 20:02
Re: UIAutomation with a focus on Chrome
@Descolada Yes and most importantly, users who are displaying the browser in languages other than English will have different element names despite using the same Chrome. I would suggest to make it a language dictionary (associative array) property which enables the user to provide translations for the original English names.
Good idea, I'll extend it to support both methods of click.
Good idea, I'll extend it to support both methods of click.
Re: UIAutomation with a focus on Chrome
Some private chat debugging with @SundayProgrammer resulted in finding a bug in the implementation of UIA_Variant method in the original UIA_Interface.ahk library by jethrow, so I have rewrote some variant handling methods and updated UIA_Interface.ahk. Also added a way to get the current tab element with cUIA.GetTab()
@william_ahk, I've implemented your Click suggestion along with a version for ControlClick into UIA_Browser.ahk (and no need for a JSON file).
@william_ahk, I've implemented your Click suggestion along with a version for ControlClick into UIA_Browser.ahk (and no need for a JSON file).
-
- Posts: 502
- Joined: 03 Dec 2018, 20:02
Re: UIAutomation with a focus on Chrome
@Descolada Awesome
Re: UIAutomation with a focus on Chrome
Can this tool get the current page source code from Chrome? Thanks.
-
- Posts: 502
- Joined: 03 Dec 2018, 20:02
Re: UIAutomation with a focus on Chrome
I was visiting some pages repeatedly and I ran into this problem that .WaitPageLoad() won't wait but instantly skips waiting. Then I discovered window titles were used in the process. Wouldn't it be more reliable to only check the reload button? For example, something like this? (but again, I'm no expert in UIAutomation)
When I'm switching between windows using Alt-Tab it appears the modifier keys were interfering with the Enter key in the address bar, causing it to be missed, prefixed with Alt (thus opening the url in new tab), etc. So I added the modifier release keys to it and solved the issue:
Code: Select all
WaitPageLoad(targetTitle="", timeOut=10000, sleepAfter=500, stopButtonText="Stop") {
startTime := A_TickCount
isLoading := false
loop {
reloadBut := this.UIA.TreeWalkerTrue.GetNextSiblingElement(this.UIA.TreeWalkerTrue.GetNextSiblingElement(this.UIA.TreeWalkerTrue.GetFirstChildElement(this.NavigationBarElement)))
if (A_TickCount - startTime) >= timeOut
break
if (InStr(reloadBut.CurrentName, stopButtonText)) {
if !isLoading
isLoading := true
} else {
if isLoading
break
}
Sleep, 200
}
}
When I'm switching between windows using Alt-Tab it appears the modifier keys were interfering with the Enter key in the address bar, causing it to be missed, prefixed with Alt (thus opening the url in new tab), etc. So I added the modifier release keys to it and solved the issue:
Code: Select all
SetURL(newUrl, navigateToNewUrl = False) { ; Sets the URL bar to newUrl, optionally also navigates to it if navigateToNewUrl=True
this.URLEditElement.SetFocus()
valuePattern := this.URLEditElement.GetCurrentPatternAs("Value")
valuePattern.SetValue(newUrl " ")
if !InStr(this.URLEditElement.CurrentValue, newUrl) {
legacyPattern := this.URLEditElement.GetCurrentPatternAs("LegacyIAccessible")
legacyPattern.SetValue(newUrl " ")
legacyPattern.Select()
}
if (navigateToNewUrl&&InStr(this.URLEditElement.CurrentValue, newUrl))
ControlSend,, {LCtrl up}{LAlt up}{LShift up}{RCtrl up}{RAlt up}{RShift up}{Enter}, % "ahk_id" this.BrowserId
}
Last edited by william_ahk on 05 Jul 2022, 04:08, edited 1 time in total.
Re: UIAutomation with a focus on Chrome
Not directly with UIA, but UIA_Browser has the method JSReturnThroughClipboard that can execute Javascript through the URL bar and returns the value through the Clipboard, and Javascript can get the source code.
Untested:
Code: Select all
#include <UIA_Interface>
#include <UIA_Browser>
cUIA := new UIA_Browser("A")
MsgBox, % cUIA.JSReturnThroughClipboard("document.documentElement.outerHTML")
Re: UIAutomation with a focus on Chrome
@william_ahk, thanks for the suggestions.
I implemented a better UIA_Browser.WaitPageLoad() method, which doesn't rely on the window title, and also added the ControlSend modifier-key releasing as you suggested. The updated UIA_Browser.ahk is currently available only on GitHub, I'll update the main post when more changes have accumulated.
I implemented a better UIA_Browser.WaitPageLoad() method, which doesn't rely on the window title, and also added the ControlSend modifier-key releasing as you suggested. The updated UIA_Browser.ahk is currently available only on GitHub, I'll update the main post when more changes have accumulated.