Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

[AHK_L] Screen Reader -- a tool to get text anywhere


  • Please log in to reply
46 replies to this topic
jethrow
  • Moderators
  • 2854 posts
  • Last active: May 17 2017 01:57 AM
  • Joined: 24 May 2009

... If you have interests, it is appreciated that you develop it. My work is here for your reference , not completed. https://github.com/n...cn/UIAutomation


Excellent - will do. Here's the work thus far: https://github.com/j...w/UIA_Interface. I am kinda learning/testing & determining how I want to implement overall functionality as I add each method. Hence why so many are still Commented-Out.
 
For anyone else reading this - this code is extremely not completed. Here, however, would be example usage:

uia := UIA_Interface()

f1::
	ElementUnderMouse := uia.ElementFromPoint()
	t := "Type/Role: " ElementUnderMouse.CurrentLocalizedControlType "`n"
	t .= "Name:`t " ElementUnderMouse.CurrentName "`n"
	
	rect := ElementUnderMouse.CurrentBoundingRectangle ;// ltrb
	t .= "Location:`t L" rect.l " T" rect.t " R" rect.r " B" rect.b "`n"
	
	rect.Struct(r)
	SArray := uia.RectToVariant(r)	;// xywh
	t .= "Dims:`t w" Round(SArray[2]) " h" Round(SArray[3]) "`n"
	
	MsgBox %t%


segalion
  • Members
  • 50 posts
  • Last active: Oct 16 2014 09:20 AM
  • Joined: 02 Feb 2011

amazing... promising...

 

Please (when you can do) could be great a "text_under_mouse" example usage.



nepter
  • Members
  • 53 posts
  • Last active: Jun 05 2019 07:59 AM
  • Joined: 29 Jun 2012

 

Excellent - will do. Here's the work thus far: https://github.com/j...w/UIA_Interface. I am kinda learning/testing & determining how I want to implement overall functionality as I add each method. Hence why so many are still Commented-Out.

Learn a lot coding skills. Building framework and some basic infrastructure is not easy and costs much time. If it succeed, other interface such as direct2d also could be wrapped in same framework.

 

amazing... promising...

Amazing will come in continuously.cool.png



Guest10
  • Members
  • 1216 posts
  • Last active: Oct 30 2015 05:12 PM
  • Joined: 27 Oct 2012

when i run...

error: 0x80040154 - Class not registered

if uia:=ComObjCreate("{ff48dba4-60ef-4201-aa87-54103eef594e}","{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")

 

Excellent - will do. Here's the work thus far: https://github.com/j...w/UIA_Interface. I am kinda learning/testing & determining how I want to implement overall functionality as I add each method. Hence why so many are still Commented-Out.
 
For anyone else reading this - this code is extremely not completed. Here, however, would be example usage:

uia := UIA_Interface()

f1::
	ElementUnderMouse := uia.ElementFromPoint()
	t := "Type/Role: " ElementUnderMouse.CurrentLocalizedControlType "`n"
	t .= "Name:`t " ElementUnderMouse.CurrentName "`n"
	
	rect := ElementUnderMouse.CurrentBoundingRectangle ;// ltrb
	t .= "Location:`t L" rect.l " T" rect.t " R" rect.r " B" rect.b "`n"
	
	rect.Struct(r)
	SArray := uia.RectToVariant(r)	;// xywh
	t .= "Dims:`t w" Round(SArray[2]) " h" Round(SArray[3]) "`n"
	
	MsgBox %t%


Learning one
  • Members
  • 1483 posts
  • Last active: Jan 02 2016 02:30 PM
  • Joined: 04 Apr 2009

Great stuff. Thanks! cool.png


My Website • Recommended: AutoHotkey Unicode 32-bit • Join DropBox, Copy


jethrow
  • Moderators
  • 2854 posts
  • Last active: May 17 2017 01:57 AM
  • Joined: 24 May 2009
I've gotten stuck on a SafeArray issue. I'm implementing CreateAndConditionFromArray, but it's not working when I manually create the array:
uia := UIA_Interface()
root := uia.GetRootElement()
c1 := uia.CreatePropertyCondition(30004, "pane", 8) ;// condition 1
c2 := uia.CreatePropertyCondition(30005, "Program Manager", 8) ;// condition 2

;// SafeArray from IUIAutomationAndCondition::GetChildren - WORKS
c3 := uia.CreateAndCondition(c1,c2)
SafeArray := c3.GetChildren() ;// returns ComObj(0x2003,out,1)
c4 := uia.CreateAndConditionFromArray(SafeArray)
GoSub, Enum

;// Manually Create SafeArray - Does not Work
SArray := ComObjArray(3,2)
SArray[0]:=c1.__Value
SArray[1]:=c2.__Value
MsgBox,, Verify SafeArray Values, % SArray[0] " = " SafeArray[0] "`n" SArray[1] " = " SafeArray[1]
c4 := uia.CreateAndConditionFromArray(SArray) ;// <-- One or more args are not valid.

Enum:
	for each, element in root.FindAll(c4)
		t .= element.CurrentName "`n"
	MsgBox %t%
	t := ""
	return

https://github.com/j...w/UIA_Interface
 
 
Edit - I found creating SArray like this works:
SArray:=ComObj(0x2003,DllCall("oleaut32\SafeArrayCreateVector", "uint",13, "uint",0, "uint",2),1)
SArray[0]:=c1.__Value, ObjAddRef(c1.__Value) ;// SafeArrayDestory releases objects stored in the array
SArray[1]:=c2.__Value, ObjAddRef(c2.__Value)


Cattleya
  • Members
  • 90 posts
  • Last active: Sep 13 2013 05:11 AM
  • Joined: 28 Sep 2011

Very useful script, finally it works on my system today, don't know why but other day it just not work :D



Clouds
  • Members
  • 1 posts
  • Last active: Aug 09 2013 01:33 PM
  • Joined: 05 Aug 2013

Nice script



BeLO
  • Members
  • 6 posts
  • Last active: Sep 29 2015 04:34 PM
  • Joined: 28 Apr 2013

What a work !

I combined it with the standard Ctrl-C/Ctrl-Insert / Ctrl-X/Ctrl-Delete, so that when there's nothing copied with the first system operation (using onClipboardChange label functionality), I try to copy the text read with the tool - if there's only one line. If there are more lines, so I display the GUI.

 

If you want the code, just tell me, cos' it's quite long, for I also extended the possibilities of the clipboard by using an alternate clipboard, by alllowing plain text operations and by copying or appending the text (with some trimming extra operations).

 

Thanks !



bpdgu
  • Members
  • 3 posts
  • Last active: Aug 12 2013 11:25 PM
  • Joined: 29 Nov 2012

It doesn't work at all in Skype. Either nothing happens or I get ContProf_#####



US91
  • Members
  • 1 posts
  • Last active: Sep 15 2013 10:00 AM
  • Joined: 15 Sep 2013

Hi, i need to search a string text in screen and click on it.

Your script working very well. I modified it to realize this function.

; !!!   -----------------------   !!!
;-----> Script modified by US91 <-----
; !!!   -----------------------   !!!

intro:="
(
The script is written by nepter, administrator of <a href=""http://bbs.ahk8.com"">Chinese autohotkey forum</a>.

Function: Press a hotkey, Capslock as default, to retrieve text under cursor and show it. It could get text in gui, html, instant messager and etc. The original Capslock is replaced by Shift+Capslock.  A tray icon in notification area could change the hotkey.

You have to run it above WinXp sp3 and by Autohotkey_L 

Version: ScreenReader 0.1.1a
)"

CoordMode, Mouse, screen
SetTitleMatchMode, 2
WinMaximize, Mozilla Firefox
WinActivate, Mozilla Firefox

if !uia:=ComObjCreate("{ff48dba4-60ef-4201-aa87-54103eef594e}","{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}")
  {
    msgbox UI Automation Failed.
    ExitApp
  }
;------------------------------------------------------------------
StringSearch=Google
;------------------------------------------------------------------
X_Pos=0
Y_Pos=0
Loop
  {
    If X_Pos > %A_ScreenWidth%
      {
        X_Pos=0
        Y_Pos+=5
      }
    If Y_Pos > %A_ScreenHeight%
      Return
;   DllCall(vt)
    item:=GetElementItem(X_Pos,Y_Pos)
;   if !item.1
;     return
    for k,v in item
      FoundString=%v%
    If FoundString = %StringSearch%
      {
        MouseClick, Left, %X_Pos%, %Y_Pos%
        return
      }
    X_Pos+=50
  }

vas(obj,ByRef txt){
	for k,v in obj
		if (v=txt)
			return 0
	return 1
}

GetPatternName(id){
	global uia
	DllCall(vt(uia,50),"ptr",uia,"uint",id,"ptr*",name)
	return StrGet(name)
}
GetPropertyName(id){
	global uia
	DllCall(vt(uia,49),"ptr",uia,"uint",id,"ptr*",name)
	return StrGet(name)
}
GetElementItem(X_Pos,Y_Pos){
	global uia
	item:={}
	DllCall(vt(uia,7),"ptr",uia,"int64",X_Pos|Y_Pos<<32,"ptr*",element) ;IUIAutomation::ElementFromPoint
        if !element
            return
	DllCall(vt(element,23),"ptr",element,"ptr*",name) ;IUIAutomationElement::CurrentName
	DllCall(vt(element,10),"ptr",element,"uint",30045,"ptr",variant(val)) ;IUIAutomationElement::GetCurrentPropertyValue::value
	DllCall(vt(element,10),"ptr",element,"uint",30092,"ptr",variant(lname)) ;IUIAutomationElement::GetCurrentPropertyValue::lname
	DllCall(vt(element,10),"ptr",element,"uint",30093,"ptr",variant(lval)) ;IUIAutomationElement::GetCurrentPropertyValue::lvalue
	a:=StrGet(name,"utf-16"),b:=StrGet(NumGet(val,8,"ptr"),"utf-16"),c:=StrGet(NumGet(lname,8,"ptr"),"utf-16"),d:=StrGet(NumGet(lval,8,"ptr"),"utf-16")
	a?item.Insert(a):0
	b&&vas(item,b)?item.Insert(b):0
	c&&vas(item,c)?item.Insert(c):0
	d&&vas(item,d)?item.Insert(d):0
	DllCall(vt(element,21),"ptr",element,"uint*",type) ;IUIAutomationElement::CurrentControlType
	if (type=50004)
		e:=GetElementWhole(element),e&&vas(item,e)?item.Insert(e):0
	ObjRelease(element)
	return item
}
GetElementWhole(element){
	global uia
	static init:=1,trueCondition,walker
	if init
		init:=DllCall(vt(uia,21),"ptr",uia,"ptr*",trueCondition) ;IUIAutomation::CreateTrueCondition
		,init+=DllCall(vt(uia,14),"ptr",uia,"ptr*",walker) ;IUIAutomation::ControlViewWalker
	DllCall(vt(uia,5),"ptr",uia,"ptr*",root) ;IUIAutomation::GetRootElement
	DllCall(vt(uia,3),"ptr",uia,"ptr",element,"ptr",root,"int*",same) ;IUIAutomation::CompareElements
	ObjRelease(root)
	if same {
		return
	}
	hr:=DllCall(vt(walker,3),"ptr",walker,"ptr",element,"ptr*",parent) ;IUIAutomationTreeWalker::GetParentElement
	if parent {
		e:=""
		DllCall(vt(parent,6),"ptr",parent,"uint",2,"ptr",trueCondition,"ptr*",array) ;IUIAutomationElement::FindAll
		DllCall(vt(array,3),"ptr",array,"int*",length) ;IUIAutomationElementArray::Length
		loop % length {
			DllCall(vt(array,4),"ptr",array,"int",A_Index-1,"ptr*",newElement) ;IUIAutomationElementArray::GetElement
			DllCall(vt(newElement,23),"ptr",newElement,"ptr*",name) ;IUIAutomationElement::CurrentName
			e.=StrGet(name,"utf-16")
			ObjRelease(newElement)
		}
                ObjRelease(array)
		ObjRelease(parent)
		return e
	}
}
variant(ByRef var,type=0,val=0){
	return (VarSetCapacity(var,8+2*A_PtrSize)+NumPut(type,var,0,"short")+NumPut(val,var,8,"ptr"))*0+&var
}
vt(p,n){
	return NumGet(NumGet(p+0,"ptr")+n*A_PtrSize,"ptr")
}

text to search must be changed in StringSearch Var.

 

it work, never errors, but it is slow, i have a quad core machine but this script is slow.

do you know causes? do you know a solution to speed up?

thank you !!!



Coldblackice
  • Members
  • 19 posts
  • Last active:
  • Joined: 30 Oct 2013

What a work !

I combined it with the standard Ctrl-C/Ctrl-Insert / Ctrl-X/Ctrl-Delete, so that when there's nothing copied with the first system operation (using onClipboardChange label functionality), I try to copy the text read with the tool - if there's only one line. If there are more lines, so I display the GUI.

 

If you want the code, just tell me, cos' it's quite long, for I also extended the possibilities of the clipboard by using an alternate clipboard, by alllowing plain text operations and by copying or appending the text (with some trimming extra operations).

 

Thanks !

Sounds awesome! Mind if I get the code?



SpankZor
  • Members
  • 46 posts
  • Last active: Oct 06 2015 08:35 PM
  • Joined: 08 Feb 2013

Looks nice, reminds me of a script that i got from here too. But don't know what i have to change to get this script working, i have the latest AUTOHOTKEY version and win 7 x32. But i get this error:

 

Error at line 95.

Line Text:         if !element
Error: This line does not contain a recognized action.

The program will exit.



rani
  • Members
  • 217 posts
  • Last active: Jul 21 2016 12:53 PM
  • Joined: 18 Mar 2008

mouse on GUI grid(table)

 

1.

on mouse on grid elements, the screen reader cannot 'see' any element(cell) of a table

I tried it, for example, on the total commander grid table.

may be the grid is not standard MS control ?

can be done something about ?

 

2. can the screen reader , read non -standard MS controls ?

 

3. screen reader under MS Internet explorer, work ok,

 but not under chrome.



rvphoenix
  • Members
  • 4 posts
  • Last active: Nov 19 2014 05:28 PM
  • Joined: 19 Nov 2014

 

The script is written by nepter, administrator of Chinese autohotkey forum.

Function: Press a hotkey, Capslock as default, to retrieve text under cursor and show it. It could get text in gui, html, instant messager and etc. The original Capslock is replaced by Shift+Capslock.  A tray in notification area could change the hotkey.

You have to run it above WinXp sp3 and by Autohotkey_L

Version: ScreenReader 0.1.1b

 

Video on Screenr

 

You could move mouse over the script below and press Capslock. What you see would excite you, I believe.

#SingleInstance,force
#InstallKeybdHook
;#NoTrayIcon
intro:="
(
The script is written by nepter, administrator of <a href=""http://bbs.ahk8.com"">Chinese autohotkey forum</a>.

Function: Press a hotkey, Capslock as default, to retrieve text under cursor and show it. It could get text in gui, html, instant messager and etc. The original Capslock is replaced by Shift+Capslock.  A tray icon in notification area could change the hotkey.

You have to run it above WinXp sp3 and by Autohotkey_L 

Version: ScreenReader 0.1.1a
)"
menu,tray,NoStandard
menu,tray,add,Hotkey,hotkey
menu,tray,add,About...,about
menu,tray,add,Exit,exit
CoordMode,mouse,screen
KeyName:="capslock"
Hotkey,%KeyName%,main
if !uia:=ComObjCreate("{ff48dba4-60ef-4201-aa87-54103eef594e}","{30cbe57d-d9d0-452a-ab13-7ac5ac4825ee}"){
	msgbox UI Automation Failed.
	ExitApp
}
return
+CapsLock::CapsLock
exit:
	exitapp
hotkey:
	gui,1:Destroy
	gui,1:add,text,,New Hotkey
	gui,1:add,Hotkey,vChosenHotkey,%KeyName%
	gui,1:add,button,Default gbtnHK,Confirm
	gui,1:show,,%A_Space%
	return
about:
	gui,2:Destroy
	gui,2:add,link,,%intro%
	gui,2:show,,About ScreenReader 0.1.1a
	return
btnHK:
	gui,1:submit
	if (ChosenHotkey!=KeyName){
		Hotkey,%KeyName%,,off
		KeyName:=ChosenHotkey
		Hotkey,%KeyName%,main,on
	}
	gui,1:Destroy
	return
main:
MouseGetPos,x,y
;DllCall(vt)
item:=GetElementItem(x,y)
if !item.1
	return
gui,3:Destroy
;gui,3:new,ToolWindow
for k,v in item
{
	gui,3:add,edit,x5 w480 -Tabstop vedit%k%,%v%
	gui,3:add,button,X+5 yp-2 vbtn%k% gcp2cb,To Clipboard
}
	gui,3:show,,You Get
return
vas(obj,ByRef txt){
	for k,v in obj
		if (v=txt)
			return 0
	return 1
}
cp2cb:
	n:=SubStr(A_GuiControl,4)
	GuiControlGet,txt,,edit%n%
	if txt
		Clipboard:=txt
	gui,3:Destroy
return
3GuiEscape:
	gui,3:Destroy
	return
GetPatternName(id){
	global uia
	DllCall(vt(uia,50),"ptr",uia,"uint",id,"ptr*",name)
	return StrGet(name)
}
GetPropertyName(id){
	global uia
	DllCall(vt(uia,49),"ptr",uia,"uint",id,"ptr*",name)
	return StrGet(name)
}
GetElementItem(x,y){
	global uia
	item:={}
	DllCall(vt(uia,7),"ptr",uia,"int64",x|y<<32,"ptr*",element) ;IUIAutomation::ElementFromPoint
        if !element
            return
	DllCall(vt(element,23),"ptr",element,"ptr*",name) ;IUIAutomationElement::CurrentName
	DllCall(vt(element,10),"ptr",element,"uint",30045,"ptr",variant(val)) ;IUIAutomationElement::GetCurrentPropertyValue::value
	DllCall(vt(element,10),"ptr",element,"uint",30092,"ptr",variant(lname)) ;IUIAutomationElement::GetCurrentPropertyValue::lname
	DllCall(vt(element,10),"ptr",element,"uint",30093,"ptr",variant(lval)) ;IUIAutomationElement::GetCurrentPropertyValue::lvalue
	a:=StrGet(name,"utf-16"),b:=StrGet(NumGet(val,8,"ptr"),"utf-16"),c:=StrGet(NumGet(lname,8,"ptr"),"utf-16"),d:=StrGet(NumGet(lval,8,"ptr"),"utf-16")
	a?item.Insert(a):0
	b&&vas(item,b)?item.Insert(b):0
	c&&vas(item,c)?item.Insert(c):0
	d&&vas(item,d)?item.Insert(d):0
	DllCall(vt(element,21),"ptr",element,"uint*",type) ;IUIAutomationElement::CurrentControlType
	if (type=50004)
		e:=GetElementWhole(element),e&&vas(item,e)?item.Insert(e):0
	ObjRelease(element)
	return item
}
GetElementWhole(element){
	global uia
	static init:=1,trueCondition,walker
	if init
		init:=DllCall(vt(uia,21),"ptr",uia,"ptr*",trueCondition) ;IUIAutomation::CreateTrueCondition
		,init+=DllCall(vt(uia,14),"ptr",uia,"ptr*",walker) ;IUIAutomation::ControlViewWalker
	DllCall(vt(uia,5),"ptr",uia,"ptr*",root) ;IUIAutomation::GetRootElement
	DllCall(vt(uia,3),"ptr",uia,"ptr",element,"ptr",root,"int*",same) ;IUIAutomation::CompareElements
	ObjRelease(root)
	if same {
		return
	}
	hr:=DllCall(vt(walker,3),"ptr",walker,"ptr",element,"ptr*",parent) ;IUIAutomationTreeWalker::GetParentElement
	if parent {
		e:=""
		DllCall(vt(parent,6),"ptr",parent,"uint",2,"ptr",trueCondition,"ptr*",array) ;IUIAutomationElement::FindAll
		DllCall(vt(array,3),"ptr",array,"int*",length) ;IUIAutomationElementArray::Length
		loop % length {
			DllCall(vt(array,4),"ptr",array,"int",A_Index-1,"ptr*",newElement) ;IUIAutomationElementArray::GetElement
			DllCall(vt(newElement,23),"ptr",newElement,"ptr*",name) ;IUIAutomationElement::CurrentName
			e.=StrGet(name,"utf-16")
			ObjRelease(newElement)
		}
                ObjRelease(array)
		ObjRelease(parent)
		return e
	}
}
variant(ByRef var,type=0,val=0){
	return (VarSetCapacity(var,8+2*A_PtrSize)+NumPut(type,var,0,"short")+NumPut(val,var,8,"ptr"))*0+&var
}
vt(p,n){
	return NumGet(NumGet(p+0,"ptr")+n*A_PtrSize,"ptr")
}

 

Hey i tried running your code.. when i hover onto something and press caps a gui shows up but no second field.. do i need to add something to it..? Newbie here.. help me out please.. :)