Version 2 of some Accessibility code.

Post your working scripts, libraries and tools.
neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 12 Jan 2022, 09:22

@ludamo
I got errors in Acc_GetRoleText(nRole) when using your v2 Acc version on a Firefox window.
The cause: the nRole parameter expects an integer (an object role constant) but in Firefox it was sometimes a string like "div", "span", "h2" (HTML tag name). The nRole value is set by Acc.accRole(ChildId) in Acc_Role(). I don't know what causes it to be a string sometimes but a workaround for your v2 Acc is to add these two lines. The role text in such cases then becomes "Unknown object", which in my tests is also what v1 Acc.ahk outputs in this case.

Code: Select all

Acc_GetRoleText(nRole)
{
	if !IsInteger(nRole)
		nRole := 0
    ;MsgBox(nRole), nRole := 0    ;notify if nRole is string (for testing)
https://docs.microsoft.com/en-us/windows/win32/api/oleacc/nf-oleacc-iaccessible-get_accrole
https://docs.microsoft.com/en-us/windows/win32/winauto/object-roles

If further investigation into this shows that "Unknown object" is a plausible role text in case nRole is a string then we can simply return it early

Code: Select all

	if !IsInteger(nRole)
		return "Unknown object"
Last edited by neogna2 on 12 Jan 2022, 09:43, edited 3 times in total.

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 12 Jan 2022, 09:31

@eugenesv Your v2 version has errors in the Acc_Children() function, specifically the line 177 loop currently only covers the next line that starts with i := but needs to brace enclose multiple lines. I suspect this happened when you split the multi expression oneliner from v1 into multiple lines. Compare with the v1 Acc code. Also, I notice that the the =3 check in that loop logic differs from that in v1 Acc and ludamo's v2 Acc, though not sure yet if that's an error or only a different implementation choice.

eugenesv
Posts: 171
Joined: 21 Dec 2015, 10:11

Re: Version 2 of some Accessibility code.

Post by eugenesv » 12 Jan 2022, 11:11

neogna2 wrote:
12 Jan 2022, 09:31
@eugenesv Your v2 version has errors in the Acc_Children() function, specifically the line 177 loop currently only covers the next line that starts with i := but needs to brace enclose multiple lines. I suspect this happened when you split the multi expression oneliner from v1 into multiple lines. Compare with the v1 Acc code. Also, I notice that the the =3 check in that loop logic differs from that in v1 Acc and ludamo's v2 Acc, though not sure yet if that's an error or only a different implementation choice.
Thanks, missed embracing :) that sneaky one-liner expansion, updated the code!

The second one is likely just a "reverse" implementation detail (notice the swapped positions of Acc_Query and child) due to a difference in this:

=3 (VT_I4:=3 ; 32-bit signed int)

=9 (VT_DISPATCH:=9 ; COM object)

So if it's 9, then it's a COM object, it gets via Acc_Query, otherwise (=3) you pass a child

but could easily be a mistake as well, let me know if you find out :)

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 12 Jan 2022, 11:49

eugenesv wrote:
12 Jan 2022, 11:11
Thanks, missed embracing :) that sneaky one-liner expansion, updated the code!
Thanks, working. After that hurdle your v2 Acc now shows the same issue with Acc_GetRoleText(nRole) as in my post to ludamo above. Same workaround works.

BTW I've noticed these issues when working on a v2 port of jeeswg's JEE_AccGetTextAll which traverses all Acc paths in a window and outputs a list.

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 14 Jan 2022, 07:28

@eugenesv
Another issue I encountered is that v2 Acc_Children(), specifically the line DllCall("oleacc\AccessibleChildren" ..., throws an error in some cases where v1 Acc doesn't.

Here is code to trigger the issue in Windows 10 File Explorer.

v2 (errors)

Code: Select all

#Include Acc.ahk2
Run "C:\"	     ; open File Explorer to C:\ for Acc test
Sleep(500)
If !WinActive("C:\ ahk_class CabinetWClass")
    ExitApp
A := Acc_ObjectFromWindow(hWnd := WinExist("A"), 0x0)  ; get root Acc object
MsgBox Acc_GetRoleText(A.accRole(0))      ; "window"
C := Acc_Children(A)                                 ; get level1 children
MsgBox Acc_GetRoleText(C[1].accRole(0))   ; "menu bar"
CC := Acc_Children(C[1])                            ; (fail to) get level2 children of level1 node1
; "Error in #include Acc.ahk2", "An exception was thrown.", "Specifically: 0xc0000005"
v1 (working)

Code: Select all

#Include Acc.ahk
Run C:\	     ; open File Explorer to C:\ for Acc test
Sleep 500
If !WinActive("C:\ ahk_class CabinetWClass")
    ExitApp
A := Acc_ObjectFromWindow(hWnd := WinExist("A"), 0x0)     ; get root Acc object
MsgBox % Acc_GetRoleText(A.accRole(0))      ; "window"
C := Acc_Children(A)                                    ; get level1 children
MsgBox % Acc_GetRoleText(C[1].accRole(0))   ; "menu bar"
CC := Acc_Children(C[1])                               ; get level2 children of level1 node1
MsgBox % Acc_GetRoleText(CC[1].accRole(0))   ; "menu item"
Here for comparison is Acc_Children() from your v2 for and the v1 original (I split the long oneliner for readability)

(edit 2022-08-17: the below v2 eugenesv Acc snippet is old, see here for latest version. But as of today the issue showcased by above script reproduces also with most recent v2 Acc.)

v2 eugenesv

Code: Select all

Acc_Children(Acc) {
  Acc_Init()
  cChildren := Acc.accChildCount
  Children	:= []
  vChildren := Buffer(cChildren*(8+2*A_PtrSize), 0)

  gotChildren := DllCall("oleacc\AccessibleChildren"
    , "Ptr" 	, ComObjValue(Acc)
    , "Int" 	, 0
    , "Int" 	, cChildren
    , "Ptr" 	, vChildren.Ptr
    , "Int*"	, &cChildren
    )
  if (gotChildren = 0) {
    Loop cChildren {
      i    	:= (A_Index-1)*(A_PtrSize*2+8)+8
      child	:= NumGet(vChildren, i, "Int64")
      Children.Push(NumGet(vChildren,i-8,"Int64")=3 ? child : Acc_Query(child))
      ObjRelease(child)
    }
    return Children
  }
  NewError := Error("" , -1)
  msgResult:=MsgBox("File:  " NewError.file "`nLine: " NewError.line "`n`nContinue Script?", , 262420)
  if (msgResult = "No")
    ExitApp()
}
v1 original

Code: Select all

Acc_Children(Acc) {
	if ComObjType(Acc,"Name") != "IAccessible"
		ErrorLevel := "Invalid IAccessible Object"
	else {
		Acc_Init()
		cChildren:=Acc.accChildCount
		Children:=[]
		if DllCall("oleacc\AccessibleChildren"
      , "Ptr",ComObjValue(Acc)
      , "Int",0
      , "Int",cChildren
      , "Ptr",VarSetCapacity(varChildren,cChildren*(8+2*A_PtrSize),0)*0+&varChildren
      , "Int*",cChildren)=0 {
			Loop %cChildren%
				i:=(A_Index-1)*(A_PtrSize*2+8)+8, child:=NumGet(varChildren,i), Children.Insert(NumGet(varChildren,i-8)=9?Acc_Query(child):child), NumGet(varChildren,i-8)=9?ObjRelease(child):
			return Children.MaxIndex()?Children:
		} else
			ErrorLevel := "AccessibleChildren DllCall Failed"
	}
	if Acc_Error()
		throw Exception(ErrorLevel,-1)
}
Last edited by neogna2 on 17 Aug 2022, 02:40, edited 2 times in total.

eugenesv
Posts: 171
Joined: 21 Dec 2015, 10:11

Re: Version 2 of some Accessibility code.

Post by eugenesv » 15 Jan 2022, 08:07

neogna2 wrote:
14 Jan 2022, 07:28
Another issue I encountered is that v2 Acc_Children(), specifically the line DllCall("oleacc\AccessibleChildren" ..., throws an error in some cases where v1 Acc doesn't.

Here is code to trigger the issue in Windows 10 File Explorer.
Thanks for the example, I've adjusted the functions to be formatted in exactly the same way (changed 3 to 9 and swapped the args, added an extra condition at the beginning and before ObjRelease) and they do look line-by-line identical, yet v1 doesn't error, while v2 does (but only in Explorer, other apps seem to be just fine and allow execution of your sample code ok, have you found any other apps that bug?)

So I don't know what's going on. Could it be some AHK v2 beta bug or something? Otherwise we'd need some much deeper understanding of the COM objects/interfaces or more luck and testing to uncover the issue without it :)

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 15 Jan 2022, 11:20

eugenesv wrote:
15 Jan 2022, 08:07
only in Explorer, other apps seem to be just fine and allow execution of your sample code ok, have you found any other apps that bug?
Yes I've seen it in other applications too, Notepad++ was one I think. I picked Explorer for the example because easy to reproduce. I will report back with more details.
eugenesv wrote:
15 Jan 2022, 08:07
Could it be some AHK v2 beta bug or something? Otherwise we'd need some much deeper understanding of the COM objects/interfaces or more luck and testing to uncover the issue without it :)
Possibly. I've tried searching the AHK forums, StackOverflow etcetera for the error code and AccessibleChildren. This old jethrow post talks about the same error code in v1 Acc code back in 2017, but I don't know if relevant to this v2 issue.

eugenesv
Posts: 171
Joined: 21 Dec 2015, 10:11

Re: Version 2 of some Accessibility code.

Post by eugenesv » 15 Jan 2022, 11:34

neogna2 wrote:
15 Jan 2022, 11:20
Yes I've seen it in other applications too, Notepad++ was one I think. I picked Explorer for the example because easy to reproduce. I will report back with more details.
Hm, I don't get it in Notepad++, nor in Word, nor in a few other apps (file managers, browsers), only the default Explorer, so if it's the only one, no big deal, no one should us It anyway ;)
Though still curious as to what the root cause is, seem so strange

eugenesv
Posts: 171
Joined: 21 Dec 2015, 10:11

Re: Version 2 of some Accessibility code.

Post by eugenesv » 15 Jan 2022, 11:49

neogna2 wrote:
15 Jan 2022, 11:20
This old jethrow post talks about the same error code in v1 Acc code back in 2017, but I don't know if relevant to this v2 issue.
Well, it says "I've found, however, if I access the Parent Object via the AccessibleObjectFromWindow function, I don't get this error"

This is how we're accessing the parent object as well — via the "Acc_ObjectFromWindow" function :)

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 16 Jan 2022, 03:33

eugenesv wrote:
15 Jan 2022, 11:34
Hm, I don't get it in Notepad++

Code: Select all

#Include Acc.ahk2
; open notepad++ first
A := Acc_ObjectFromWindow(hWnd := WinExist("ahk_exe notepad++.exe"), 0x0)  ; root
A := Acc_Children(A)
A := Acc_Children(A[4])       ; client  4
A := Acc_Children(A[18])      ; window 4.18
A := Acc_Children(A[4])       ; client 4.18.4
MsgBox "errors on next line"
A := Acc_Children(A[1])       ; window 4.18.4.1
relevant parts of Acc tree from JEE_AccGetTextAll with v1 Acc

Code: Select all

	window [new 1 - Notepad++][]
4	  client [new 1 - Notepad++][]
4.18	    window [][]
4.18.4	      client [][]
4.18.4.1	        window [][]
4.18.4.1.1	          menu bar [System][]
4.18.4.1.2	          title bar [][]
4.18.4.1.2 c1	            push button [IME][]
4.18.4.1.2 c2	            push button [Minimize][]
4.18.4.1.2 c3	            push button [Maximize][]
...

eugenesv
Posts: 171
Joined: 21 Dec 2015, 10:11

Re: Version 2 of some Accessibility code.

Post by eugenesv » 16 Jan 2022, 06:55

I've tested a bit more with the following convenient loop that follows a path you set in an array, so no need to manually copy&paste:

Code: Select all

AccPath	:= [4,18,4,1,1]	; Set children path here from the root element (can disable MsgBox in the loop to test faster or enable it to compare to another tool that gives you the tree without bugs)
T      	:= "T0"
p      	:= ""
Cci    	:= Acc_ObjectFromWindow(hWnd := WinExist("ahk_exe notepad++.exe"), 0x0)  ; root
Loop AccPath.Length {
  Cc     	:= Acc_Children(Cci)	;
  i      	:= AccPath[A_Index]
  p      	.= "." i                            	;
  Cci    	:= Cc[i]                            	;
  try Nm 	:=                  Cci.accName( 0) 	;
  try Val	:=                  Cci.accValue(0) 	;
  Role   	:= Acc_GetRoleText( Cci.accRole( 0))	;
  State  	:= Acc_GetStateText(Cci.accState(0))	;
  MsgBox("Rc" p "`n" Nm "`n" Val "`n" Role "`n" State,, T)
}
and got the following results:

Code: Select all

AccPath	:= [4,18,4,1]  	;k
AccPath	:= [1,1,1]     	;k
AccPath	:= [2]         	;k
AccPath	:= [3,1,1,3,1] 	;k
AccPath	:= [3,1,1,14,1]	;k
AccPath	:= [3,2,1,20,1]	;k 11–20
AccPath	:= [3,13,1]    	;k
AccPath	:= [5]         	;k
AccPath	:= [6]         	;k
AccPath	:= [4,1,2]     	;k
AccPath	:= [4,1,4,1,2] 	;k
AccPath	:= [4,1,4,1,5] 	;k
AccPath	:= [4,1,4,2,2] 	;k
AccPath	:= [4,1,4,3,2] 	;k
AccPath	:= [4,1,5]     	;k
AccPath	:= [4,2,2]     	;k
AccPath	:= [4,2,4,1,2] 	;k
AccPath	:= [4,5,2]     	;k
AccPath	:= [4,17,2]    	;k
AccPath	:= [4,17,5]    	;k
AccPath	:= [4,12,4]    	;k
AccPath	:= [4,18,3]    	;k
AccPath	:= [4,18,4,2]  	;k
AccPath	:= [4,19,2]    	;k
AccPath	:= [4,12,4]    	;k
AccPath	:= [4,20,5]    	;k
AccPath	:= [4,18,4,3]  	; ok, but err on children
AccPath	:= [4,18,4,3,2]	;   err on any last i
AccPath	:= [4,18,4,1]  	; ok, but err on children
AccPath	:= [4,18,4,1,1]	;   err on any last i
AccPath	:= [4,19,4,1]  	; ok, but err on children
AccPath	:= [4,19,4,1,1]	;   err
Still no idea why those are special :(
Tried to look at the difference between the fails and the successes in VSCode with an AHK debuggin plugin, but also don't see anything immediately obvious

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 18 Jan 2022, 07:07

@eugenesv I'm also stumped on what causes the error in the v2 code but not v1.
The Explorer error case uses these parts of v1/v2 Acc:
Acc_Init()
Acc_ObjectFromWindow()
Acc_GetRoleText()
Acc_Query()
Acc_Children()

Either the v2 code has some error there or the cause is in some change or issue with how v2.beta3 deals with COM with regard to Acc.

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 21 Feb 2022, 05:18

@eugenesv
Found an issue in your v2 Acc_Location() code. Change from this

Code: Select all

Acc_Location(Acc, ChildId:=0) { ; adapted from Sean's code
  global com
  try {
    Acc.accLocation(
      ComValue(com.pi32,&x:=0)
    , ComValue(com.pi32,&y:=0)
    , ComValue(com.pi32,&w:=0)
    , ComValue(com.pi32,&h:=0)
    , ChildId)
  }  catch {
    return
  }
To this (adapted from Ludamo's Acc v2 version)

Code: Select all

Acc_Location(Acc, ChildId:=0) { ; adapted from Sean's code
  global com
  x:=Buffer(4), y:=Buffer(4), w:=Buffer(4), h:=Buffer(4)
  try {
    Acc.accLocation(
      ComValue(com.pi32, x.ptr)
    , ComValue(com.pi32, y.ptr)
    , ComValue(com.pi32, w.ptr)
    , ComValue(com.pi32, h.ptr)
    , ChildId)
  } catch {
    return
  }
Same change is also needed after line AccObj.accLocation( in your Acc_Get()


Also, and this is a nitpick, your Acc_Location() returns a Map

Code: Select all

  retMap := Map()
    retMap["x"] := (x := NumGet(x, 0, "Int"))
  , retMap["y"] := (y := NumGet(y, 0, "Int"))
  , retMap["w"] := (w := NumGet(w, 0, "Int"))
  , retMap["h"] := (h := NumGet(h, 0, "Int"))
  , retMap["pos"] := "x" x " y" y " w" w " h" h
  return retMap
Which means outside code would access values with OutsideMap["x"]
Compare: v1 Acc and Ludamo's v2 Acc returns an Object and outside code can access values with OutsideObj.x
My nitpicky point is that existing outside code requires slightly less v1 to v2 changes if you instead use

Code: Select all

  retObj := Object()
    retObj.x := (x := NumGet(x, 0, "Int"))
  , retObj.y := (y := NumGet(y, 0, "Int"))
  , retObj.w := (w := NumGet(w, 0, "Int"))
  , retObj.h := (h := NumGet(h, 0, "Int"))
  , retObj.pos := "x" x " y" y " w" w " h" h
  return retObj

eugenesv
Posts: 171
Joined: 21 Dec 2015, 10:11

Re: Version 2 of some Accessibility code.

Post by eugenesv » 22 Feb 2022, 10:36

Thanks! Updated the code (on page 1) with your fixes, Ptr and retObj (also used this opportunity to remove code duplication from Acc_Get, now it points to Acc_Location instead of reimplementing the same thing), tested both to get Explorer's window position, seems to work
Now if only we could get that nasty access error fixed...

AHK_user
Posts: 515
Joined: 04 Dec 2015, 14:52
Location: Belgium

Re: Version 2 of some Accessibility code.

Post by AHK_user » 24 Feb 2022, 15:00

I was trying out this code to see if I could get the position of the caret in Visual Studio Code.
But I never get an Object from Acc_Get
What am I doing wrong?

I confirmed the correct acc path with a v1 script that I wrote that makes it easy to visualize all acc paths viewtopic.php?f=6&t=81059&p=353038&hilit=windowlist#p353038

This path can be different depending on your settings of vs code.

Code: Select all

#Requires AutoHotKey v2.0-beta.3
#SingleInstance force

#Include "acc.ahk"

Caret := VsCodeLineColumn()
Msgbox Caret.Ln "," Caret.Col
return

VsCodeLineColumn() {
    hwnd := WinExist("ahk_exe Code.exe")
    ;hwnd := ControlGetHwnd("Chrome_RenderWidgetHostHWND1", "ahk_exe Code.exe")
    oAcc := Acc_Get("Object", "4.1.1.1.1.2.3.1.11", 0, "ahk_id " hwnd)

    msgbox(IsObject(oAcc))
    RegExMatch(oAcc.accName(0), "Ln (\d+), Col (\d+)", &m)
    MsgBox(m[1])
    return { Ln: m[1], Col: m[2] }
}

Escape::{
    ExitApp 
}

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 24 Feb 2022, 15:56

AHK_user wrote:
24 Feb 2022, 15:00
I was trying out this code to see if I could get the position of the caret in Visual Studio Code.
But I never get an Object from Acc_Get
What am I doing wrong?
Don't know what the cause is. My install of VS Code doesn't play nice with Acc, I fail to get the caret position with that kind of approach with v2 and v1 alike.

But worth mentioning that we can get the VS Code caret screen position another way which does work for me with v2 Acc.

Code: Select all

;v2 get VS Code caret X Y screen position with Acc
#Include Acc.ahk  ; eugene's v2 Acc
WinActivate("ahk_class Chrome_WidgetWin_1 ahk_exe Code.exe")
ActiveHwnd := WinExist("A")
Acc_Caret := Acc_ObjectFromWindow(ActiveHwnd, OBJID_CARET := 0xFFFFFFF8)
ObjCaretPos := Acc_Location(Acc_Caret)
MsgBox ObjCaretPos.x "|" ObjCaretPos.y
;https://docs.microsoft.com/en-us/windows/win32/winauto/caret
;https://docs.microsoft.com/en-us/windows/win32/winauto/object-identifiers
;OBJID_CARET := -8 ; 0xFFFFFFFFFFFFFFF8 (uint32: 0xFFFFFFF8)

AHK_user
Posts: 515
Joined: 04 Dec 2015, 14:52
Location: Belgium

Re: Version 2 of some Accessibility code.

Post by AHK_user » 25 Feb 2022, 02:29

neogna2 wrote:
24 Feb 2022, 15:56
AHK_user wrote:
24 Feb 2022, 15:00
I was trying out this code to see if I could get the position of the caret in Visual Studio Code.
But I never get an Object from Acc_Get
What am I doing wrong?
Don't know what the cause is. My install of VS Code doesn't play nice with Acc, I fail to get the caret position with that kind of approach with v2 and v1 alike.

But worth mentioning that we can get the VS Code caret screen position another way which does work for me with v2 Acc.

Code: Select all

;v2 get VS Code caret X Y screen position with Acc
#Include Acc.ahk  ; eugene's v2 Acc
WinActivate("ahk_class Chrome_WidgetWin_1 ahk_exe Code.exe")
ActiveHwnd := WinExist("A")
Acc_Caret := Acc_ObjectFromWindow(ActiveHwnd, OBJID_CARET := 0xFFFFFFF8)
ObjCaretPos := Acc_Location(Acc_Caret)
MsgBox ObjCaretPos.x "|" ObjCaretPos.y
;https://docs.microsoft.com/en-us/windows/win32/winauto/caret
;https://docs.microsoft.com/en-us/windows/win32/winauto/object-identifiers
;OBJID_CARET := -8 ; 0xFFFFFFFFFFFFFFF8 (uint32: 0xFFFFFFF8)
Your code indeed seems to work to get x and y, but I want try to retrieve position in ln and col.
in post viewtopic.php?f=60&t=77464&start=20#p364385 there is a working example of v1 code.
The idea is to create a hotkey to lookup the help file of the function where the charet is positioned. The filepath can be retrieved by changing the setting of vsCode so it displays the filepath inside the wintitle.

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 27 Feb 2022, 05:50

AHK_user wrote:
25 Feb 2022, 02:29
The idea is to create a hotkey to lookup the help file of the function where the charet is positioned.
I'd like to have that too! Hopefully we'll get enough people poking around the v2 Acc lib conversions to figure out what still needs to be fixed.

neogna2
Posts: 586
Joined: 15 Sep 2016, 15:44

Re: Version 2 of some Accessibility code.

Post by neogna2 » 16 Aug 2022, 06:51

Let's revive this thread. I mostly use v2 now but issues with v2 Acc (discussed above, with yet unknown causes) make me fall back to v1 for scripts that use Acc. It would be splendid if we could get v2 Acc on par with v1 Acc. Since those of use active in this thread seem stuck I wonder if @lexikos has any thoughts on how to move forward with this?

lexikos
Posts: 9559
Joined: 30 Sep 2013, 04:07
Contact:

Re: Version 2 of some Accessibility code.

Post by lexikos » 16 Aug 2022, 21:59

I suggest posting code and details about a specific issue in the Ask for Help v2 forum, showing the non-working code and working v1 code.

Post Reply

Return to “Scripts and Functions (v2)”