FindText - Capture screen image into text and then find it

[feiyue]

Updated to 6.7 version - 2019/04/13
.... 1. Merge FindText() and FindText2() functions.
.... 2. The color difference mode is added to adjust the matching range accurately.
.... 3. Add a parameter "FindAll" that allows you to find only one result and return it immediately.

[SL5]

Updated to 6.7 version - 2019/04/13 ...

Wow your

Code: Select all

for i,v in ok
	if i<=12
		MouseTip(v.1+v.3//2, v.2+v.4//2)

really works much better! Compliment and heartfelt Thanks

19-04-14_10-45 some fun with it:

Code: Select all

 findTextTool_demo_backwardsLoop_bottomTooTop(ok)
findTextTool_demo_backwardsLoop_bottomTooTop(ok){ 
i := ok.MaxIndex()
while(ok[i] && i){ 
	X:=ok[i].1+ok[i].3//2, Y:=ok[i].2+ok[i].4//2, W:=ok.1.3, H:=ok.1.4, Comment:=ok.1.5, X+=W//2, Y+=H//2
	MouseMove,% X+50 + mod(A_Index,2)*40,% Y ; MouseClick,left,% X+50,% Y
	i--
}
Return
}

[songdg]

Updated to 6.6 version - 2019/04/11
.... 1. FindText(), FindText2() share the same machine code, reduces the code size.

For a single text or image, the effects of these two functions are similar,
but for multiple text or images, FindText() loops through each text or image,
while FindText2() looks at them as a continuous combination of text for a one-time search.

Sometimes the text of a web page will be deformed and the spacing will be increased.
You can automatically cut a text into more than one text with PicX() and use FindText2() to find it.
FindText(..., Text ) ==> FindText2(..., PicX(Text) )

That's the problem i currently facing, but how to use PicX(), can you detail a bit more.

[pedroabs]

Hi feiyue,

I am new to autohotkey and I found out about your script a month ago. I have been working on my games in Bluestack, but I have 1 issue. I am unable to pass the parameter of EmulatorArea variable to the function. It gives me this error. I have tried with and without percentage sign. Please help me thank you.

test.png

Code: Select all

#SingleInstance Force
EmulatorArea=145,645,244,751

t1:=A_TickCount


Text:="|<>*122$41.zVzzzzzytzzzzzvvzzzzzrq3X4DzjgquTTzTPg5szyyrPvhzytanrPzyD3ki7zzyzzzzzzxzzzzzzvzzzzzzzzzzzzzzzzzzzzzzzzsTzzzzzrTzzzzzizzzzzzRX6vcsk7BqmLiZqvhJb1PhrOfayrPil7YxVrXrQQ/U"

if (ok:=FindText(EmulatorArea, 0, 0, Text))
{
  CoordMode, Mouse
  X:=ok.1.1, Y:=ok.1.2, W:=ok.1.3, H:=ok.1.4, Comment:=ok.1.5, X+=W//2, Y+=H//2
  ; Click, %X%, %Y%
}

MsgBox, 4096,, % "Time:`t" (A_TickCount-t1) " ms`n`n"
  . "Pos:`t" X ", " Y "`n`n"
  . "Result:`t" (ok ? "Success !":"Failed !"), 3
MouseMove, X, Y
(ok) && MouseTip()

Hi. Instead of putting all coordinates in 1 variable, split them into 4. So you only need to update 4 variables instead of several lines in your script.

[pedroabs]

I'm sorry, I'm from China, so I can't go to www.youtube.com or https youtu.be Broken Link for safety

ups. @feiyue, thanks for answering and again for all your great work on this script.(now inserted it as gif without audio now).

Full screen can be found and window range can not be found, indicating that window range acquisition error.
Your problem can only be debugged by yourself, others can not help, because it is difficult to reproduce your situation.

in other words:
How would you do it if you wrote that yourself?
1. So first search for the window and save the area. how?
2. then use this area in FindText. how?
whey is facotr 1.88 nedet? (see examples below). do you use factor 1.88? is everybody using factor 1.88? only me? How would you do it if you did not use a factor?

Code: Select all

needle=Action List Appears Here ahk_class AutoHotkeyGUI ; mouseWindowTitle=0x100b50  ; 
WinGetPos, wX, wY, wW, wH, % needle
xxx:=floor(wX/1.88)
yyy:=floor(wY/1.88)
if(ok:=FindText(xxx,yyy,xxx+wW,yyy+wH, 0, 0, Text)){
	X:=ok.1.1, Y:=ok.1.2, W:=ok.1.3, H:=ok.1.4, Comment:=ok.1.5, X+=W//2, Y+=H//2
	MouseMove,% X+50,% Y
	msgbox,found
}else{
	Clipboard = found nothing  in %wX%`, %wY%`, %wW%`, %wH% (19-04-09_01-29)
	msgbox, % Clipboard
}

Maybe you have multiple window matching conditions, but WinGet return an inappropriate window.

at least that sounds like I'm not wrong, not totally wrong. that helps me a lot already! thanks for the feedback.

You can check how many window matching conditions:
WinGet, list, List, % needle
MsgBox, % list

i will check that.

snowiscold1 , Here's a small wrapper function that calls FindText.

Code: Select all

MyFindText(EmulatorArea, arg*) {
  a:=IsObject(EmulatorArea) ? EmulatorArea : StrSplit(EmulatorArea, ",")
  return, FindText(a[1]+0, a[2]+0, a[3]+0, a[4]+0, arg*)
}
EmulatoraArea:="x,y,w,h"  ...

^---- I think it has nothing to do with my question but thanks anyway (i dont use a string)

There is no problem with the script. The problem is with windows and AHK and how the DPI scaling is implemented.

I would guess your screen has a DPI of 180. A normal screen has a DPI of 96 (scaling is just 1). Newer screens has DPI of 120 ( recommended scaling is 125%). Yours is 180 (I guess when you go to display settings it says 187.5% scaling recommended).

you can workout a formula using this values to find the correct coordinates to search. But if you will use the recommended screen resolution and scaling, you should not see any problem (aside from using GUIs in AHK).

[feiyue]

@songdg , Hello !
First, let me introduce the role of FindText2(). It is used for combination lookup.
For example, a 0-9 text library has been generated.
You can search for any ID number that these text libraries make up.
For example:
Text.= t1:= "|<1>**50$5.9kV248EVDk" ;==> ASCII Search Text
(
__0__
000__
__0__
__0__
__0__
__0__
__0__
__0__
__0__
__0__
00000
)
Text.= t2:= "|<2>**50$6.SVVV2248EVzU" ;==> ASCII Search Text
(
_0000_
0____0
0____0
0____0
____0_
____0_
___0__
__0___
_0____
0____0
000000
)
Text.="|<3>**50$6.SVV2A211VWQU"
Text.="|<4>**50$6.26+GGWWz22DU"
Text.="|<5>**50$6.zUUUgm11VWQU"
Text.="|<6>**50$6.CGUUgmVVVGAU"
Text.="|<7>**50$6.zWW44888888U"
Text.="|<8>**50$6.SVVVGAGVVVSU"
Text.="|<9>**50$5.Aa489A02LE"
Text.="|<0>**50$6.AGVVVVVVVGAU"
Pic(Text, 1) ;==> Store text data in a text library
;==>If you want to find the text "21"
ok:=FindText2(x, y, w, h, 0, 0, PicN("21") )

; The function of PicN() is to extract the data of each character from the text library
; according to the comment text (<1>...<0>) and connect them with "|". ==> PicN("21") == t2 . "|" . t1

Second, FindText2() has been merged into FindText() due to script upgrades,
So it should be used as follows:
ok:=FindText(x, y, w, h, 0, 0, PicN("21"), 1, 1, JoinText:=1)
;==> the JoinText parameter for setting combination lookup

Thirdly, let me introduce the role of PicX(). It is used to decompose the whole text data
into a combination of individual character data.
For example: The original code for Find "12" is :
Text_12:="|<12>**50$14.87i24UV88G08U28120UUE88TXy" ;==> ASCII Search Text
(
__0______0000_
000_____0____0
__0_____0____0
__0_____0____0
__0_________0_
__0_________0_
__0________0__
__0_______0___
__0______0____
__0_____0____0
00000___000000
)
ok:=FindText(x, y, w, h, 0, 0, Text_12)

If the interval between 1,2 increases when the page is displayed, the text will not be found.
At this point, use multiple individual character combination lookup can adapt to different intervals.
PicX(Text_12) == t1 . "|" . t2 ;==> Use blank vertical bars to segment individual characters

;==> So you can use the following code to combine lookups :
ok:=FindText(x, y, w, h, 0, 0, PicX(Text_12), 1, 1, JoinText:=1)

I use English automatic translation, I hope you can understand what I mean.

[feiyue]

For beginners, since coordinate parameters are easily misunderstood,
the following wrapping functions are recommended:

Code: Select all

; @FindText() ---- << Copy this function into your script >> ----
; (x1, y1) is the upper-left coordinate of the search range,
; (x2, y2) is the lower-right coordinate, Other parameters are the same as FindText()
@FindText(x1, y1, x2, y2, @*) {
    return, FindText(x1, y1, x2-x1+1, y2-y1+1, @*)
}

For example:
If you get the coordinates of two points on the screen (100,100) and (500,500), you can use:
ok:=@FindText(100,100, 500,500, 0,0, Text)

[songdg]

Thanks, I've been trying Text:="|<d>*165$7.1UkMAzXVksQD6zU" ;character "d", Text.="|<a>*164$7.SFUHv56XSs" ;character "a", ok:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, PicN("da"), 1, 1, JoinText:=1) to locate "data", but it is fail, do i miss something?

[feiyue]

@songdg
You haven't looked carefully:
Pic(Text, 1) ;==> Store text data in a text library

[songdg]

Code: Select all

Text:="|<d>*165$7.1UkMAzXVksQD6zU" 
Text.="|<a>*164$7.SFUHv56XSs" 
Pic(Text, d)
Pic(Text, a)
if (ok:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, PicN("da"), 1, 1, JoinText:=1))
{
  CoordMode, Mouse
  X:=ok.1.1, Y:=ok.1.2, W:=ok.1.3, H:=ok.1.4, Comment:=ok.1.5, X+=W//2, Y+=H//2
  ; Click, %X%, %Y%
}

Thanks for answer my question, I add Pic(Text, d) and Pic(Text, a), but still fail to find "data".

[feiyue]

Please look at the instructions of the function for use.
If you don't understand what it means, You can copy it, but don't change it indiscriminately.
Instead of using variables, you only need to use 1
Pic(Text, 1) ;==> Store text data in a text library

Code: Select all

; You can put the text library at the beginning of the script,
; and Use Pic(Text,1) to add the text library to Pic()'s Lib,
; Use Pic("comment1|comment2|...") to get text images from Lib

Pic(comments, add_to_Lib=0)
{
  static Lib:=[]
  if (add_to_Lib)
  {
    re:="<([^>]*)>[^$]+\$\d+\.[\w+/]+"
    Loop, Parse, comments, |
      if RegExMatch(A_LoopField,re,r)
        Lib[Trim(r1)]:=r
    Lib[""]:=""
  }
  else
  {
    Text:=""
    Loop, Parse, comments, |
      Text.="|" . Lib[Trim(A_LoopField)]
    return, Text
  }
}


PicN(Number)
{
  return, Pic( RegExReplace(Number, ".", "|$0") )
}

The following four forms are equivalent：
1:
Text:="|<d>*165$7.1UkMAzXVksQD6zU"
Text.="|<a>*164$7.SFUHv56XSs"
Pic(Text, 1)
ok:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, PicN("da"), 1, 1, JoinText:=1)

2:
Text:="|<d>*165$7.1UkMAzXVksQD6zU"
Text.="|<a>*164$7.SFUHv56XSs"
Pic(Text, 1)
; PicN("abc")==Pic("a|b|c") , but Pic("aaa|bbb|ccc") != PicN("aaabbbccc")
ok:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, Pic("d|a"), 1, 1, JoinText:=1)

3:
Text:="|<d>*165$7.1UkMAzXVksQD6zU"
Text.="|<a>*164$7.SFUHv56XSs"
; the Text == Pic("d|a") == PicN("da")
ok:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, Text, 1, 1, JoinText:=1)

4:
Text:="|<d>*165$7.1UkMAzXVksQD6zU"
Pic(Text, 1)

Text:="|<a>*164$7.SFUHv56XSs"
Pic(Text, 1)

ok:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, PicN("da"), 1, 1, JoinText:=1)

[songdg]

@feiyueThanks for your patience, this tool is wonderful and with increasing improvement.

[feiyue]

Text1:="|<pic1>..."
Text2:="|<pic2>..."
Text3:="|<pic3>..."
Text4:="|<pic4>..."

TimeOut:=A_TickCount+5000, findok:=0

Loop {

check:=[]
; When multiple images are searched and the results of each image are important, the fault tolerance cannot be 0.
For k, v in ok:=FindText(0,0,150000,150000, 0.1, 0.1, Text1 . Text2 . Text3 . Text4)
{
x:=v.1+v.3//2, y:=v.2+v.4//2, comment:=v.5 ; v == ok[k]
check[comment]:=[x, y] ; comment is XXX in "|<XXX>..."
}

check_ok_num:=(!!check["pic1"])+(!!check["pic2"])+(!!check["pic3"])+(!!check["pic4"])

/*
; If the number of images to be searched is small, a simple method can be used
check_ok_num:=0
if ok1:=FindText(0,0,150000,150000, 0, 0, Text1, ScreenShot:=1)
check_ok_num++
if ok2:=FindText(0,0,150000,150000, 0, 0, Text2, ScreenShot:=0)
check_ok_num++
if ok3:=FindText(0,0,150000,150000, 0, 0, Text3, ScreenShot:=0)
check_ok_num++
if ok4:=FindText(0,0,150000,150000, 0, 0, Text4, ScreenShot:=0)
check_ok_num++
*/

; check if four different images are present on the screen at a given time. if yes than click one of the image
if (check_ok_num=4)
{
findok:=1
MouseClick,, ok.1.1+ok.1.3//2, ok.1.2+ok.1.4//2 ; This is the first result found
Break
}

; check if any one of the four different images is present on the screen at a given time. if yes than click that image
if (ok)
{
findok:=1
MouseClick,, ok.1.1+ok.1.3//2, ok.1.2+ok.1.4//2 ; This is the first result found
Break
}

; check if any one of the four different images is present on the screen AND one other different image is present on the screen at a given time
if (check_ok_num>=2)
{
findok:=1
Break
}

Sleep, 100

} Until (A_TickCount>TimeOut)

if (findok=0)
MsgBox, Failure !

if (check["pic1"])
MouseClick,, check["pic1"].1, check["pic1"].2

if (check.pic2)
MouseClick,, check.pic2[1], check.pic2[2]

[kazhafeizhale]

hi feiyue,Thank you very much for the script,
I have a question:
What's the difference between "splitadd" and "addall"? If I click on these two buttons, it's the same thing.

[feiyue]

When using the "SplitAdd" button, you need to click on the splitting position on the yellow tag line.

Because automatic segmentation using "SplitAdd" can reduce background blanks on all sides,
So if you want to make a text library with blank edges, such as punctuation marks,
It needs to be manually produced using "AllAdd".

[feiyue]

@Ipot , You can use this function flexibly only if you understand the meaning of the parameter.

; Introduction of function parameters:
;
; returnArray := FindText(
; X --> the search scope's upper left corner X coordinates
; , Y --> the search scope's upper left corner Y coordinates
; , W --> the search scope's Width
; , H --> the search scope's Height
; , Character "0" fault-tolerant in percentage --> 0.1=10%
; , Character "_" fault-tolerant in percentage --> 0.1=10%
; , Text --> can be a lot of text parsed into images, separated by "|"
; , ScreenShot --> if the value is 0, the last screenshot will be used
; , FindAll --> if the value is 0, Just find one result and return
; , JoinText --> if the value is 1, Join all Text for combination lookup
; , offsetX --> Set the max text offset for combination lookup
; , offsetY --> Set the max text offset for combination lookup
; )

So if you want to speed up, you should set ScreenShot and FindAll to 0.
You can use the ScreenShot() function to get screenshot once,
and then use the last screenshot for all subsequent searches. For example:

Code: Select all

CheckIfFarmersBusy:

ScreenShot()

okfb:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, farmerbusy, 0,0)
okfb1:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, farmerbusy1, 0,0)
okrss:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, searchRSS, 0,0)
okcl:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, cropland, 0,0)
okfl:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, foodlabel, 0,0)
oksrb:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, searchRSSButton, 0,0)
okf:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, farm, 0,0)
okf1:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, farm1, 0,0)
okf2:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, farm2, 0,0)
okf3:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, farm3, 0,0)
okf4:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, farm4, 0,0)
okf5:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, farm5, 0,0)
okgr:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, gatherRSS, 0,0)
okua:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, userAccount, 0,0)
okizl:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, izzy01label, 0,0)
okizzy01:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, izzy01, 0,0)
okshanewat:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, shanewat, 0,0)
okshanewatlabel:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, shanewatlabel, 0,0)
okuserSetting:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, userSetting, 0,0)
okcharacterManagement:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, characterManagement, 0,0)
okcharacterLogin:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, characterLogin, 0,0)
okuserAllAccount:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, userAllAccount, 0,0)
okchooseAnAccount:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, chooseAnAccount, 0,0)
okcandicecoldgmail:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, candicecoldgmail, 0,0)
okgoogleButton:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, googleButton, 0,0)
okswitchButton:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, switchButton, 0,0)
okfightfairlabel:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, fightfairlabel, 0,0)
oklionsloverlabel:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, lionsloverlabel, 0,0)
okfightfair:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, fightfair, 0,0)
oklionslover:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, lionslover, 0,0)
okcharismajoygmail:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, charismajoygmail, 0,0)
okdannypopolabel:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, dannypopolabel, 0,0)
okcharismajoylabel:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, charismajoylabel, 0,0)
okdannypopo:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, dannypopo, 0,0)
okcharismajoy:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, charismajoy, 0,0)
okmalikneevedgmail:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0, 0, malikneevedgmail, 0,0)

In addition, your click function is time-consuming, which can be simplified as follows:

Code: Select all

clickText(o){
      X:=o[1].1, Y:=o[1].2, W:=o[1].3, H:=o[1].4, X+=W//2, Y+=H//2
      Click, %X%, %Y%
      Sleep, 500
}

If you can't find the image, you may need to recapture the text
or adjust the fault-tolerant size. Please debug it yourself.

Important tips:
1. In order to improve reliability, try to capture the unique image or text on the screen.
2. Use regular graphics as much as possible, and cluttered graphics can easily lead to mismatching.

[SL5]

I am running this code in an emulator window which is slightly more than half of my screen size.. the title of the emulator window is bluestacks_5 and this is the working code that i have . is there any way to make it quicker ...

did you tried:

Code: Select all

SetTitleMatchMode, 2
WinGetPos, wX, wY, wW, wH, bluestacks_5
okfb:=FindText(wX, wY,wW,wH, tolerant1:=0, tolerant2:=0,  farmerbusy)

? works?

Code: Select all

rectangleTip(wX,wY,wW,wH) 
;/¯¯¯¯ rectangleTip ¯¯ 190424041852 ¯¯ 24.04.2019 04:18:52 ¯¯\
rectangleTip(x,y,w,h){
	Gui, _rectangleTip_: +AlwaysOnTop -Caption +ToolWindow +Hwndmyid +E0x08000000
	Gui, _rectangleTip_: Show, Hide w%w% h%h%

	dhw:=A_DetectHiddenWindows
	DetectHiddenWindows, On
	d:=8, i:=w-d, j:=h-d
	s=0-0 %w%-0 %w%-%h% 0-%h% 0-0
	s=%s%  %d%-%d% %i%-%d% %i%-%j% %d%-%j% %d%-%d%
	WinSet, Region, %s%, ahk_id %myid%
	DetectHiddenWindows, %dhw%
	Gui, _rectangleTip_: Show, NA x%x% y%y%
	Loop, 8 {
		Gui, _rectangleTip_: Color, % A_Index & 1 ? "Red" : "Green"
		Sleep, 500
	}
}
;\____ rectangleTip __ 190424041908 __ 24.04.2019 04:19:08 __/

[xypie]

Hi,

Awesome! Your script works flawlessly.

[Vaklev]

Hey the gui is not loading for me once I launch, is there a hotkey to reload or something?

[xypie]

Just want to know, will it be possible to search the text on an inactive window using hwnd?