FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 22 Apr 2021, 16:30

@c7aesa7r You can do this:
1. Click the button: CaptureS, OpenDir, Cancel.
2. Copy the image to the open directory ( %A_Temp%\Ahk_ScreenShot ).
3. Click the button: CaptureS, (select your picture), OK.
4. Your image is displayed in the upper left corner of the screen, you can capture it.
User avatar
rommmcek
Posts: 1313
Joined: 15 Aug 2014, 15:18

Re: FindText - Capture screen image into text and then find it

Post by rommmcek » 23 Apr 2021, 19:13

Since saved image is displayed exactly in the upper left corner, it has one additional positive side effect i.e. capturing position is correct even after screen is being magnified. Thank you feiyue!
jsong55
Posts: 42
Joined: 30 Mar 2021, 22:02

Re: FindText - Capture screen image into text and then find it

Post by jsong55 » 28 Apr 2021, 22:10

How does one tell the function to click a certain occurrence of the found text?

I wrote this function and here's the use case

The use of function here

Code: Select all

ftc(Text:="|<>*159$23.0zw0TzzUzzz1zzy7y7yDs7wTU7sz0Dly0Tbw0zjw3zTwDyzzzxzzzvzzzs0000000E",,,,"146, 79, 922, 569","Y",1)
This works

Code: Select all

ftc(text:="",offX:=0,offY:=0,LR:="Left",range:="780, 272, 1870, 1008",show:="N",occurrence:=1) {
	rangeA:=[]
	rangeA:=StrSplit(range, ", ")
	While !ok:=FindText(rangeA[1],rangeA[2],rangeA[3],range[A], 0, 0, Text)
		sleep, 100
	If occurrence=1
		MouseClick, Left, % ok.1.x+offX, % ok.1.y+offY
	If occurrence=2
		MouseClick, Left, % ok.2.x+offX, % ok.2.y+offY
	If (show="Y") {
		for i,v in ok
		if (i<=2)
		FindText.MouseTip(ok[i].x+offX, ok[i].y+offY)
		}
	}
This doesn't work

Code: Select all

ftc(text:="",offX:=0,offY:=0,LR:="Left",range:="780, 272, 1870, 1008",show:="N",occurrence:=1) {
	rangeA:=[]
	rangeA:=StrSplit(range, ", ")
	While !ok:=FindText(rangeA[1],rangeA[2],rangeA[3],range[A], 0, 0, Text)
		sleep, 100
	MouseClick, Left, % ok.occurrence.x+offX, % ok.occurrence.y+offY
	If (show="Y") {
		for i,v in ok
		if (i<=2)
		FindText.MouseTip(ok[i].x+offX, ok[i].y+offY)
		}
	}
jsong55
Posts: 42
Joined: 30 Mar 2021, 22:02

Re: FindText - Capture screen image into text and then find it

Post by jsong55 » 28 Apr 2021, 23:05

I solved it lel

Code: Select all

ftc(text:="",offX:=0,offY:=0,LR:="Left",range:="780, 272, 1870, 1008",show:="N",occurrence:=1) {
	rangeA:=[]
	rangeA:=StrSplit(range, ", ")
	While !ok:=FindText(rangeA[1],rangeA[2],rangeA[3],range[A], 0, 0, Text)
		sleep, 100
	MouseClick, Left, % ok[occurrence].x+offX, % ok[occurrence].y+offY
	If (show="Y") {
		for i,v in ok
		if (i<=10)
		FindText.MouseTip(ok[i].x+offX, ok[i].y+offY)
		}
	}
c7aesa7r
Posts: 149
Joined: 02 Jun 2016, 21:09

Re: FindText - Capture screen image into text and then find it

Post by c7aesa7r » 04 May 2021, 11:30

@feiyue

If i have various text with the same comment for example:

Code: Select all

text := "|<1>*.......
text  .="|<1>*.......
text  .="|<1>*.......
text  .="|<1>*.......
text  .="|<1>*.......
text  .="|<1>*.......
And I have set:

Code: Select all

if (ok:=findtext(x, y, x2 , y2,0.1,0.1, text,1,0)) {
}
As the last parameter is 0, it means it will return at the first match.
I know I can see the comment matched inside:

Code: Select all

comment :=ok.1.id
But as all comment have the same name it would not help to know which one was caught, is possible somehow to differentiate it? Maybe the array index of the text caught or ?
User avatar
arimania
Posts: 8
Joined: 26 Sep 2019, 06:16

Re: FindText - Capture screen image into text and then find it

Post by arimania » 04 May 2021, 12:33

feiyue wrote:
01 Apr 2021, 16:18
Updated to 8.4 version - 2021/04/02 :dance: :beer:
.... 1. Improvement: in multicolor mode,You can find multiple colors and adapt to different resolutions.
....... Now it also supports searching images, for example: Text:="|<>##10$ c:\a.bmp"
....... It's better to limit the search range to a small one.
.... 2. Modify: the return value of OCR identification function is changed.
....... eg: MsgBox, % FindText.Ocr(ok).ocr --> Change to: MsgBox, % FindText.Ocr(ok).text
.... 3. Add: FindText.QPC() and FindText.SetColor() is added.
.... 4. Modify: The search direction (dir = 1-9), dir = 9: From the Center to all around.
Sorry I dont really understand about what you're saying. Hi Feiyue, i try to OCR this bmp but not work (link: https drive.google.com /file/d/1tbtUisFVqi7h-__brb1qJz1BYmrIH0Pm/view?usp=sharing). Broken Link for safety

The aim is want to OCR multi line and get the value 1 as as variable1 secondline as a variable 2, etc

with this code but not work: is that something wrong?

Code: Select all

#NoEnv
CoordMode, Mouse ; Required: change coord mode to screen vs relative.

F2::
; using findtext8.1 status not working
; This use the monitor size 1280x1028 (show only on monitor 1).
;universal (grey or white background) 
;ini minusnya sama area putih supaya nggak cari ke atas menu
;~ Text:="|<->*150$13.0000000000000000000000000C000000000000000E"
;~ Text:="|<1>*132$4./W8W8WU"
;~ Text:="|<2>*143$6.S2226AEEzU"
;~ Text:="|<3>*121$5.s8Ea24/Y"
;~ Text:="|<4>*134$6.66+GGz22U"
;~ Text:="|<5>*131$5.R27V34/Y"
;~ Text:="|<6>*136$6.CMESHFFGCU"
;~ Text:="|<7>*132$5.y4EW40V4"
;~ Text:="|<8>*131$6.CGGGAGFHCU"
;~ ;GrayDiff 50 baru bisa universal
;~ Text:="|<9>**50$6.SHnnnT36QU"
;~ Text:="|<0>**50$6.SHnnnnHHSU"
;~ Text:="|<.>**50$3.000003M04"
;===use this code also not works then trying on monitor 2 only.


;using findtext 8.5 also not working
; This use the monitor size 1280x1028 (show only on monitor 2) not using the left monitor.
Text:="|<->*121$10.00000000000000000000007U000000000000000002"
Text:="|<1>*171$6.6SK66666666U"
Text:="|<2>*134$7.D8k846266621z"
Text:="|<3>*137$8.D4E61UEs1U821bm"
Text:="|<4>*127$8.10kA52FYFDw410G"
Text:="|<5>*132$7.0000000T8421s610UIPk000000000E"
Text:="|<7>*135$7.z0UUEM844211V"
Text:="|<6>*128$8.7X0UE5lWEoB28Vm"
Text:="|<8>*135$8.0000000003kaEY8a7693Eq8w0000000008"
Text:="|<9>*128$8.D6N2EY9aCU8217W"
Text:="|<0>*133$8.D2N2EY93EY92NXm"
;using GrayDiff 50, and then only after that i can get the dot (.) of full stop, in the white and grey area.
Text:="|<.>**50$3.s0000000P0007U"

;MouseGetPos, x, y
t1:=A_TickCount
;------------------------------
;====1128,662, 1253,761
;ok:=FindText(x-150, y-20, x+150, y+20, 0.2, 0.1, Text)
ok:=FindText(1128, 662, 1253, 761, 0.2, 0.1, Text)

arr:=FindText.OCR(ok), OCR:=arr.text
;------------------------------
t1:=A_TickCount-t1
MsgBox, 4096, OCR, OCR Result: [%OCR%] in %t1% ms.
Return

below is findtext copied code.
Giving output :
OCR Result: [...] in 47 ms.

How to fix this, is that something wrong?
and the second question. How to make it showing in red box before searching, just to make sure the program not seeking at the wrong area?

(sorry for my bad english)
kauan014
Posts: 19
Joined: 18 Feb 2021, 20:03

Re: FindText - Capture screen image into text and then find it

Post by kauan014 » 04 May 2021, 22:58

feiyue wrote:
16 Jul 2020, 22:16
c7aesa7r wrote:
16 Jul 2020, 05:54
do someone know if is possible to search inside a pbitmap instead searching for the text on screen?

i tried edit here, but did not work :cry:

Code: Select all

    if (id) and !(w<1 or h<1)
    {
      ;hDC2:=DllCall("GetDCEx", Ptr,id, Ptr,0, "int",3, Ptr)
      ;hDC2	    := Gdip_BitmapFromHWND(id)
      hDC2 := Gdip_CreateBitmapFromFile("C:\Users\User\Documents\x.png") ;
      
      DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
        , Ptr,hDC2, "int",x-wx, "int",y-wy, "uint",0x00CC0020|0x40000000)
      DllCall("ReleaseDC", Ptr,id, Ptr,hDC2)
    }
I think you can replace screen data with image files like this, but I haven't tested:

Code: Select all

/*
  if (hBM) and !(w<1 or h<1)
  {
    win:=DllCall("GetDesktopWindow", Ptr)
    hDC:=DllCall("GetWindowDC", Ptr,win, Ptr)
    mDC:=DllCall("CreateCompatibleDC", Ptr,hDC, Ptr)
    oBM:=DllCall("SelectObject", Ptr,mDC, Ptr,hBM, Ptr)
    DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
      , Ptr,hDC, "int",x, "int",y, "uint",0x00CC0020|0x40000000)
    DllCall("ReleaseDC", Ptr,win, Ptr,hDC)
    if (id:=this.BindWindow(0,0,1))
      WinGet, id, ID, ahk_id %id%
    if (id)
    {
      WinGetPos, wx, wy, ww, wh, ahk_id %id%
      left:=x, right:=x+w-1, up:=y, down:=y+h-1
      left:=left<wx ? wx:left, right:=right>wx+ww-1 ? wx+ww-1:right
      up:=up<wy ? wy:up, down:=down>wy+wh-1 ? wy+wh-1:down
      x:=left, y:=up, w:=right-left+1, h:=down-up+1
    }
    if (id) and !(w<1 or h<1)
    {
      hDC2:=DllCall("GetDCEx", Ptr,id, Ptr,0, "int",3, Ptr)
      DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
        , Ptr,hDC2, "int",x-wx, "int",y-wy, "uint",0x00CC0020|0x40000000)
      DllCall("ReleaseDC", Ptr,id, Ptr,hDC2)
    }
    DllCall("SelectObject", Ptr,mDC, Ptr,oBM)
    DllCall("DeleteDC", Ptr,mDC)
  }
*/
;-----------------
  if FileExist(file:="C:\Users\User\Documents\x.png")
  {
    pBitmap := Gdip_CreateBitmapFromFile(file)
    w:=Gdip_GetImageWidth(pBitmap), h:=Gdip_GetImageHeight(pBitmap)
    w:=w>zw ? zw:w, h:=h>zh ? zh:h
    hBM2:=Gdip_CreateHBITMAPFromBitmap(pBitmap), Gdip_DisposeImage(pBitmap)
    mDC:=DllCall("CreateCompatibleDC", Ptr,0, Ptr)
    oBM:=DllCall("SelectObject", Ptr,mDC, Ptr,hBM, Ptr)
    mDC2:=DllCall("CreateCompatibleDC", Ptr,0, Ptr)
    oBM2:=DllCall("SelectObject", Ptr,mDC2, Ptr,hBM2, Ptr)
    DllCall("BitBlt",Ptr,mDC,"int",0,"int",0,"int",w,"int",h
      , Ptr,mDC2, "int",0, "int",0, "uint",0x00CC0020|0x40000000)
    DllCall("SelectObject", Ptr,mDC2, Ptr,oBM2)
    DllCall("DeleteDC", Ptr,mDC2)
    DllCall("SelectObject", Ptr,mDC, Ptr,oBM)
    DllCall("DeleteDC", Ptr,mDC)
    DllCall("DeleteObject", Ptr,hBM2)
  }
;-----------------
@feiyue i tried to do this modification but it does not work, maybe because you updated the function a lot? do you know how to edit to work with the newer version?
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 05 May 2021, 03:58

c7aesa7r wrote:
04 May 2021, 11:30

Code: Select all

text := "|<1>*.......
text  .="|<1>*.......
text  .="|<1>*.......
text  .="|<1>*.......
text  .="|<1>*.......
text  .="|<1>*.......
But as all comment have the same name it would not help to know which one was caught, is possible somehow to differentiate it? Maybe the array index of the text caught or ?
Comment is used to distinguish between the results.

Code: Select all

text := "|<1.1>*.......
text  .="|<1.2>*.......
text  .="|<1.3>*.......
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 05 May 2021, 04:08

@arimania
1. It's a connection, not a replacement.

Code: Select all

Text := "|<->*121$10.00000000000000000000007U000000000000000002"
Text .= "|<1>*171$6.6SK66666666U"
Text .= "|<2>*134$7.D8k846266621z"
2. The decimal point should include the surrounding white space.
Or it can be judged by setting a smaller interval to make the result generate a * symbol.

Code: Select all

Text:="|<.>**50$3.s0000000P0007U"
3. The range of OCR can be obtained by clicking the "GetRange" button.
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 05 May 2021, 07:19

c7aesa7r wrote:
16 Jul 2020, 05:54
do someone know if is possible to search inside a pbitmap instead searching for the text on screen?
@kauan014 You can do it like this: :dance: :superhappy:

Code: Select all


#Include <FindText>

f1 := "d:\a.bmp"
f2 := "d:\b.bmp"
FindText.SavePic(f1, 0, 0, 0, 0, ScreenShot:=1)  ; This line is not needed if the image already exists
FindText.ShowPic(f1, show:=0)
FindText.SavePic(f2, 0, 0, 100, 100, ScreenShot:=0)  ; This line is not needed if the image already exists
Text:="|<>##0$" . f2
if (ok:=FindText(0, 0, 0, 0, 0, 0, Text, ScreenShot:=0))
{
   CoordMode, Mouse
   X:=ok.1.1, Y:=ok.1.2, Comment:=ok.1.id
   MsgBox, Found Pos: %X%`, %Y%
}
else
   MsgBox, Not Found !
return

Last edited by feiyue on 06 May 2021, 04:52, edited 1 time in total.
kauan014
Posts: 19
Joined: 18 Feb 2021, 20:03

Re: FindText - Capture screen image into text and then find it

Post by kauan014 » 05 May 2021, 10:11

feiyue wrote:
05 May 2021, 07:19
@kauan014 You can do it like this: :dance: :superhappy:

Code: Select all


#Include <FindText>

f1 := "d:\a.bmp"
f2 := "d:\b.bmp"
FindText.SavePic(f1, 0, 0, 0, 0, ScreenShot:=1)  ; This line is not needed if the image already exists
FindText.ShowPic(f1, show:=0)
FindText.SavePic(f2, 0, 0, 100, 100, ScreenShot:=0)  ; This line is not needed if the image already exists
Text:="|<>##0$" . f2
if (ok:=FindText(0, 0, 0, 0, 0, 0, Text, ScreenShot:=0))
{
   CoordMode, Mouse
   X:=ok.1.1, Y:=ok.1.1, Comment:=ok.1.id
   MsgBox, Found Pos: %X%`, %Y%
}
else
   MsgBox, Not Found !
return


The text inside the variable text is the icon of VSCode

But the code does not find it, am I doing something wrong?
Last edited by kauan014 on 05 May 2021, 14:06, edited 2 times in total.
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 05 May 2021, 11:31

@kauan014 You don't understand the meaning of each line of code, you just need to:

Code: Select all


#Include <FindText>

Text:="|<>*77$......"

; Add this line to load the image into memory. Note that show:=0 and ScreenShot:=0
FindText.ShowPic("c:\b.bmp", show:=0)
if (ok:=FindText(0, 0, 0, 0, 0, 0, Text, ScreenShot:=0))
{
   X:=ok.1.1, Y:=ok.1.2, Comment:=ok.1.id
   MsgBox, Found Pos: %X%`, %Y%
}
else
   MsgBox, Not Found !
return

Last edited by feiyue on 06 May 2021, 04:52, edited 1 time in total.
kauan014
Posts: 19
Joined: 18 Feb 2021, 20:03

Re: FindText - Capture screen image into text and then find it

Post by kauan014 » 05 May 2021, 14:28

@feiyue thanks! now it finds the image.

In this case, the x1 y1 x2 y2 parameters are not relative to the image loaded or they are ignored and is searched inside the entire image?
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 06 May 2021, 04:06

@kauan014
Because all the coordinates of FindText are relative to the screen coordinates,
it is a bit troublesome to search for images instead of the screen.
The images are loaded into the upper left corner of the screen,
and the upper left corner of the screen is not necessarily located in (0,0).
You can refer to the following code: (Use the files that FindText updated today) :beer: :dance:

Code: Select all


#Include <FindText>

Text:="|<>*77$......"

; Add this line to load the image into memory. Note that show:=0 and ScreenShot:=0
FindText.ShowPic("c:\b.bmp", show:=0, zx, zy, w, h)

; Search range of relative pictures: (x1,y1,x2,y2)
; if (ok:=FindText(x1+zx, y1+zy, x2+zx, y2+zy, 0, 0, Text, ScreenShot:=0))

; Search range of relative pictures: (0,0,w,h)
if (ok:=FindText(zx+0, zy+0, zx+w, zy+h, 0, 0, Text, ScreenShot:=0))
{
   ; Results of relative pictures:
   X:=ok.1.1-zx, Y:=ok.1.2-zy, Comment:=ok.1.id
   MsgBox, Found Pos: %X%`, %Y%
}
else
   MsgBox, Not Found !
return

kauan014
Posts: 19
Joined: 18 Feb 2021, 20:03

Re: FindText - Capture screen image into text and then find it

Post by kauan014 » 06 May 2021, 12:22

@feiyue Thanks amazing! I have tested it, working :clap:
I got some questions if you don't mind.
Does searching inside images compared to searching on the screen have less performance?

I mean:

1- does it use more CPU power compared to searching on screen in a loop with a very low interval?
2- does the time to find a match is higher compared to when searching on-screen.
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 06 May 2021, 13:07

A screen size memory image is reserved inside the FindText function, All searches are performed in this memory image.
When searching the screen, use BitBlt() to copy the screen content to this memory image,
When searching the disk image, use LoadPicture() to copy the disk image content to this memory image,
I don't know which copy process is faster, but the search speed of memory image is the same. :D
kauan014
Posts: 19
Joined: 18 Feb 2021, 20:03

Re: FindText - Capture screen image into text and then find it

Post by kauan014 » 06 May 2021, 14:12

@feiyue
Is possible to search for a combination of a specific text?
Example:

I captured just one pink pixel from the image above:

Code: Select all

text:="|<1>[email protected]$1.k" ;
pic.png
pic.png (93 Bytes) Viewed 158 times
Then I would like to search inside a region if it has 10? of these pixels, combining 10x the text it would look like:
image.png
image.png (308 Bytes) Viewed 158 times
Theres such option? :)
Last edited by kauan014 on 06 May 2021, 23:14, edited 3 times in total.
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 06 May 2021, 22:01

@kauan014 Please look at the function description carefully and try it yourself.
kauan014
Posts: 19
Joined: 18 Feb 2021, 20:03

Re: FindText - Capture screen image into text and then find it

Post by kauan014 » 06 May 2021, 23:12

@feiyue Trust me i have look inside the function the entire day :?

Code: Select all

text:="|<1>[email protected]$1.k" 

FindText.PicLib(text, add_to_Lib:=1)

text:= FindText.PicN("11111111111")
fileappend %text% `n,*

if (ok:=FindText(0, 0, 0, 0, 0,0, text,1,1))
{

  fileappend found! `n,*
  for i,v in ok
    FindText.MouseTip(ok[i].x, ok[i].y)

}
I discovered this thing of Lib, I thought it was what I'm looking for, but looks like it just combines a pre-saved text and search for each of it and not a combination of all of they.
feiyue
Posts: 297
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 07 May 2021, 00:34

look at the function description :JoinText
Post Reply

Return to “Scripts and Functions”