Jump to content


Photo

Acc Library [AHK_L] (updated 09/27/2012)

Started by jethrow , Feb 19 2012 07:03 PM

  • Please log in to reply
35 replies to this topic

#1 jethrow

jethrow
  • Fellows
  • 2549 posts

Posted 19 February 2012 - 07:03 PM

Microsoft Active Accessibility (MSAA) is an Application Programming Interface (API) for user interface accessibility ... The programmatic goal of MSAA is to allow Windows controls to expose basic information, such as name, location on screen, or type of control, and state information such as visibility, enabled, or selected. - Wikipedia

>>>Download Acc Library<<<

NOTE: This library is an AHK version of some of the Active Accessibility Client Functions & IAccessible interface members. Statements in this post may be direct or slightly modified quotes from the MSDN Active Accessibility User Interface Services page.

Function List:

  • Acc_ObjectFromEvent(OutputChildId, hWnd, idObject, idChild)

    Acc_ObjectFromPoint([OutputChildId, x, y])

    Acc_ObjectFromWindow(hWnd [, idObject=-4])
    - Retrieves the specified interface for the object associated with the specified window.
    • Parameters
      hWnd [in]
      • Specifies the handle of a window for which an object is to be retrieved.

      idObject [in]
      • Specifies the object ID. This value is one of the standard object identifier constants (defaults to OBJID_CLIENT) or a custom object ID such as OBJID_NATIVEOM.

      Return value
      If successful, returns an AHK wrapped IAccessible object.

    Acc_WindowFromObject(Acc)
    - Retrieves the window handle that corresponds to an IAccessible object.
    • Parameters
      Acc [in]
      • IAccessible object whose corresponding window handle will be retrieved.

      Return value
      If successful, returns a handle to the window containing the object specified in Acc.

    Acc_GetRoleText(Role)
    - Retrieves the localized string that describes the object's role for the specified role value.
    • Parameters
      Role [in]
      Return value
      If successful, returns the role text string.

    Acc_GetStateText(State)
    - Retrieves a localized string that describes an object's state.
    • Parameters
      State [in]
      Return value
      If successful, returns the state text string.


    Acc_SetWinEventHook(eventMin, eventMax, pCallback)

    Acc_UnhookWinEvent(hHook)

    Acc_Role(Acc [, ChildId])

    Acc_State(Acc [, ChildId])

    Acc_Children(Acc)

    Acc_Location(Acc [, ChildId])

    Acc_Parent(Acc)

    Acc_Child(Acc [, ChildId])

    Acc_Query(Acc)

 

Object Constants:

Spoiler

 

Event Constants:

Spoiler

 

 

Accessible Info Viewer
*Note - thread history can be found here


#2 sinkfaze

sinkfaze
  • Moderators
  • 6077 posts

Posted 28 February 2012 - 03:59 AM

I'm trying to sort out the different control roles in Firefox 10.0.2 and when I use Acc_Children like this:

win :=	Acc_ObjectFromWindow(WinExist("Ask for Help ahk_class MozillaWindowClass"))
For page in Acc_Children(win)
	res .=	A_Index "`t" page.accChildCount "`n"
MsgBox, %	res

I get this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


But when I run this:

win :=	Acc_ObjectFromWindow(WinExist("Ask for Help ahk_class MozillaWindowClass"))
Loop %	win.accChildCount
	res .=	A_Index "`t" win.accChild(A_Index).accChildCount "`n"
MsgBox %	Trim(res)

I get this:

1 0
2 0
3 0
4 0
5 9
6 0
7 0
8 17
9 9
10 3
11 0
12 0
13 0
14 0
15 0
16 0
17 0
18 0
19 0
20 9
21 0
22 0
23 0
24 0


Shouldn 't these produce the same results, or am I not understanding how Acc_Children should work?

#3 jethrow

jethrow
  • Fellows
  • 2549 posts

Posted 28 February 2012 - 07:12 AM

Quick Answer - Change your for-loop to:
For Each, page in Acc_Children(win) ; Acc_Children() returns an array

Details:
It looks like you're trying to use the form For Variant,variant_type in ComObj, similar to how Com Object enumerations work. This is how the deprecated function Acc_EnumChildren worked, but I decided to have Acc_Children operate more like the actual AccessibleChildren function - specifically by returning an AHK version of the rgvarChildren [out] param:

rgvarChildren [out]
Type: VARIANT*

Pointer to an array of VARIANT structures that receives information about the container's children. If the vt member of an array element is VT_I4 (number), then the lVal member for that element is the child ID. If the vt member of an array element is VT_DISPATCH (object), then the pdispVal member for that element is the address of the child object's IDispatch interface.

The Acc_Children function loops through this structure & inserts each vt member into an array, which is then returned. NOTE - if the vt member is VT_DISPATCH, the Acc_Children function queries for the IAccessible interface & enwraps it. So, to loop through & get info about the children, you could simply do this:
t := "Child #`tType"

For Each, child in Acc_Children(win)
	if IsObject(child)
		t .= "`n" A_Index "`tobject"  ; var child contains an IAccessible object
	else
		t .= "`n" A_Index "`telement"  ; var child contains the ChildId (integer)

MsgBox %t%
NOTE - a ChildId doesn't always equal A_Index.

#4 sinkfaze

sinkfaze
  • Moderators
  • 6077 posts

Posted 28 February 2012 - 03:52 PM

Hmm...now when I try the method as you suggested it on an external listview, this:

ControlGet, hWnd, hWnd, , SysListView321, ahk_class nunya_bidness
Acc :=   Acc_ObjectFromWindow(hWnd)
For i, child in Acc_Children(Acc)
   	MsgBox %	child.accName(0)

I get all blank messages boxes except for the last item, whereas if I run this:

ControlGet, hWnd, hWnd, , SysListView321, ahk_class nunya_bidness
Acc :=   Acc_ObjectFromWindow(hWnd)
Loop %	Acc.accChildCount
	MsgBox %	Acc.accName(A_Index)

It returns data for everything but the last item (which throws an error then a blank message box). Am I again misunderstanding how Acc_Children should work?

#5 jethrow

jethrow
  • Fellows
  • 2549 posts

Posted 28 February 2012 - 04:01 PM

Child elements & child objects are different:

For i, child in Acc_Children(Acc)
	if IsObject(child)
		MsgBox % child.accName(0) ; var child is an IAccessible object
	else
		MsgBox % Acc.accName(child) ; var child is a ChildId (integer)

#6 kenn

kenn
  • Members
  • 407 posts

Posted 28 February 2012 - 04:03 PM

Very nice library, thanks for your efforts. I have a question, is it possible to grab text under mouse with this library?

#7 jethrow

jethrow
  • Fellows
  • 2549 posts

Posted 28 February 2012 - 04:16 PM

Thanks - though remember that it's mostly Sean's work.

... is it possible to grab text under mouse ...

Not exactly - though you can grab the text from the IAccessible object under the mouse:
f1:: MsgBox % GetTextUnderMouse()

GetTextUnderMouse() {
	Acc := Acc_ObjectFromPoint(child)
	try value := Acc.accValue(child)
	if Not value
		try value := Acc.accName(child)
	return value
}

#8 sinkfaze

sinkfaze
  • Moderators
  • 6077 posts

Posted 28 February 2012 - 05:24 PM

I've already used the Accessibility functions in places where it was easy to use them and have made some marked process improvements, so thanks for that.

But if you could, help walk me through a situation. Say I want to expose the main menu bar in Notepad (File > Edit > Format, etc.). I already know by running this code:

WinGet, hWnd, id, ahk_class Notepad
win :=	Acc_ObjectFromWindow(hWnd)
For each, child in Acc_Children(win)
{
	res :=	""
	For each, obj in Acc_Children(child)
		res .=	A_Index "`t" (IsObject(obj)
		 ? "object`t" Acc_GetRoleText(obj.accRole(0))
		 : "element`t" Acc_GetRoleText(child.accRole(obj)))
	MsgBox %	Trim(res)
}

That I can expose the main structure of the Notepad window:

1	object	menu bar
2	object	title bar
3	object	menu bar
4	object	editable text
5	object	scroll bar
6	object	scroll bar
7	object	grip

Logic would lead me to infer that the first menu bar is the one on the title bar, so I should try to get the second menu bar. But doing this:

For each, child in Acc_Children(win)
	For each, obj in Acc_Children(child)
		if	(A_Index=3)
			MsgBox %	Acc_Children(obj).MaxIndex()

Indicates that there are no children.

I know I can capture a menu item by first invoking the menu and connecting to the object through the menu window, but is this the only way? Am I missing a step to get from the menu bar to a menu item, or am I looking in the wrong place?

#9 jethrow

jethrow
  • Fellows
  • 2549 posts

Posted 29 February 2012 - 07:43 AM

Am I missing a step to get from the menu bar to a menu item, or am I looking in the wrong place?

Thanks for the question. It seems Acc_ObjectFromWindow is accessing the client object, rather than the window object. So, you would need to access the client objects parent (which is the window object), & then get the menu bar object:
WinGet, hWnd, id, ahk_class Notepad

client := Acc_ObjectFromWindow(hWnd)
MsgBox % Acc_Role(client) ; client

window := Acc_Parent(client)
MsgBox % Acc_Role(window) ; window

menu_bar := Acc_Children(window)[3]

For Each, menu_item in Acc_Children(menu_bar)
	t .= menu_item.accName "`n" ; children are objects
MsgBox %t%

#10 sinkfaze

sinkfaze
  • Moderators
  • 6077 posts

Posted 29 February 2012 - 06:07 PM

Thank you for your help to this point, I would've never figured out the issue of client object versus window object on my own. Using Accessibility obviously represents a world of possibilities for users here at the forums but the level of understanding required to work your way through things is ridiculously high compared to using COM with Internet Explorer or Excel or what have you.

So I've gotten to the point where I've "silently" navigated to the 'Open...' menu in Notepad:

WinGet, hWnd, id, ahk_class Notepad
window :=	Acc_Parent(Acc_ObjectFromWindow(hWnd))
For each, object in Acc_Children(window)
	if	(object.accName="Application")
		break
For each, mainMenu in Acc_Children(object)
	if	(mainMenu.accName="File")
		break
For each, item in Acc_Children((menuItem :=	mainMenu.accChild(1)))
	if	InStr(menuItem.accName(item),"Open")
		break	; MsgBox, Found it!!!

The problem now is that 'Open...' item is an element, not an object, so I can't use accDoDefaultAction as it expects an object. Is this a shortcut to execution that's just not allowed? Am I forced to invoke the menu popup at every step?

#11 sinkfaze

sinkfaze
  • Moderators
  • 6077 posts

Posted 29 February 2012 - 10:48 PM

I'm guessing this library will inevitably be frustrating since it has been superceded by the UI Automation framework. I believe Ranorex Spy uses this framework.

It looks like there is an intermediate interface between IAccessible and UI Automation, IAccessibleEx, but there doesn't seem to be a good deal fo information on how to implement it in the way we would use it.

#12 jethrow

jethrow
  • Fellows
  • 2549 posts

Posted 01 March 2012 - 02:11 AM

... but the level of understanding required to work your way through things is ridiculously high ...

It would be way easier if there were an IAccessible Viewer.

The problem now is that 'Open...' item is an element, not an object, so I can't use accDoDefaultAction as it expects an object.

You would call accDoDefaultAction from the IAccessible Object, and use the ChildId as the parameter. For some reason this is saying Member Not Found though:
WinGet, hWnd, id, ahk_class Notepad
client := Acc_ObjectFromWindow(hWnd)
popup_menu := Acc_GetChild(client.accParent, "3,1,1")

MsgBox %	"Name:`t" popup_menu.accName(1)
		.	"`nRole:`t" Acc_Role(popup_menu, 1)
		.	"`nAction:`t" popup_menu.accDefaultAction(1)

if IsMemberOf(popup_menu, "accDoDefaultAction")
	popup_menu.accDoDefaultAction(1) ; <- member not found?
		

Acc_GetChild(Acc, path="") {
	Loop Parse, path, csv
		Acc := Acc_Children(Acc)[A_LoopField]
	return Acc
}
IsMemberOf(obj, name) {
   return, DllCall(NumGet(NumGet(1*p:=ComObjValue(obj))+A_PtrSize*5), "Ptr",p, "Ptr",VarSetCapacity(iid,16,0)*0+&iid, "Ptr*",&name, "UInt",1, "UInt",1024, "Int*",dispID)=0 && dispID+1
}

#13 sinkfaze

sinkfaze
  • Moderators
  • 6077 posts

Posted 01 March 2012 - 05:41 AM

Same error I received, which is when I checked MSDN and saw this:

accDoDefaultAction[/url">

":iz29qbxe]A menu item represents a particular item in a menu bar or pop-up menu. For example, Microsoft Active Accessibility creates a menu item object for the File menu in the menu bar. Similarly, Microsoft Active Accessibility creates a menu item object for the Open menu item from the File pop-up menu.

The window class name for a menu item is "#32768".


Which leaves me to believe that

[*:iz29qbxe]a menu item must necessarily be an object to utilize accDoDefaultAction, and
[*:iz29qbxe]you can only access the object by invoking the menu and passing the window handle through Acc_ObjectFromWindow.

It would be way easier if there were an IAccessible Viewer.


I actually took a look at the source for your viewer when I started having problems but it was calling a non-existent function and even though I know which one to replace it with I'm not positive that will necessarily fix things.

#14 jethrow

jethrow
  • Fellows
  • 2549 posts

Posted 01 March 2012 - 07:35 AM

accDoDefaultAction[/url">

":1i6ja6sx]For menu items from the menu bar, accDoDefaultAction either displays or closes the menu depending on the state of the menu. For menu items from a pop-up menu, accDoDefaultAction clicks the menu item to execute the menu command.

I believe the pop-up menu window would have to exist in order to access it's children. I haven't been access the pop-up menu items via the IAccessible interface unless the pop-up window is visible.

#15 maul.esel

maul.esel
  • Members
  • 790 posts

Posted 01 March 2012 - 02:55 PM

I'm guessing this library will inevitably be frustrating since it has been superceded by the UI Automation framework.

For anyone interested, I started integrating UIAutomation in CCF. Links:[*:32fteim4]request
[*:32fteim4]UIAutomation branchCode is written in AHK v2 right now, will be ported to AHK_L when finished. Any help is appreciated, just fork & add.
Back to Scripts
  1. AutoHotkey Community
  2. → AutoHotkey
  3. → Scripts
  4. Privacy Policy