FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools for AHK v1.1 and older
User avatar
iilabs
Posts: 296
Joined: 07 Jun 2020, 16:57

Re: FindText - Capture screen image into text and then find it

Post by iilabs » 07 Aug 2022, 14:21

Thank you, the above suggestions worked!

As far as library, I was referring to this in the posts? Is this something necessary for OCR and findtext()?

Code: Select all


FindTextOCR(nX, nY, nW, nH, err1, err0, Text, Interval=20)
{
  OCR:="", RightX:=nX+nW-1
  While (ok:=FindText(nX, nY, nW, nH, err1, err0, Text))
  {
    ; For multi text search, This is the number of text images found
    For k,v in ok
    {
      ; X is the X coordinates of the upper left corner
      ; and W is the width of the image have been found
      x:=v.1, y:=v.2, w:=v.3, h:=v.4, comment:=v.5
      ; We need the leftmost X coordinates
      if (A_Index=1 or x<LeftX)
        LeftX:=x, LeftY:=y, LeftW:=w, LeftH:=h, LeftOCR:=comment
      else if (x=LeftX)
      {
        Loop, 100
        {
          err:=A_Index/100
          if FindText(x, y, w, h, err, err, Text)
          {
            LeftX:=x, LeftY:=y, LeftW:=w, LeftH:=h, LeftOCR:=comment
            Break
          }
          if FindText(LeftX, LeftY, LeftW, LeftH, err, err, Text)
            Break
        }
      }
    }
    ; If the interval exceeds the set value, add "*" to the result
    OCR.=(A_Index>1 and LeftX-nX-1>Interval ? "*":"") . LeftOCR
    ; Update nX and nW for next search
    nX:=LeftX+LeftW-1, nW:=RightX-nX+1
  }
  return, OCR
}

User avatar
iilabs
Posts: 296
Joined: 07 Jun 2020, 16:57

Re: FindText - Capture screen image into text and then find it

Post by iilabs » 07 Aug 2022, 14:29

Is it possible to display multiple search items in the array found? In the above corrected example, the last text is CBD, when the findtext() finds CBD (Common bile duct) and then 0.2 it displays the text and index 1 for CBD and index 2 for 0.2? What I am trying to do is combine the two items found to build a string. Maybe need to loop but thought the Text fields were all combined with |? Should I do a loop parse with delinator as "** seems like that may work but figured there is a way to build as found?

Code: Select all


!j::
t1:=A_TickCount
t1:=A_TickCount, Text:=X:=Y:=""

Text.="|<0.2>*120$69.zzzzzzzzzzzzzzjzzzzzzzzzzzzzzs3zzU3zzzzzy0Dzw0DzzzzzU0zy00zzzzzsT3zkz3zzzzy7wDy7wDzzzzkzVztzVzzzzy7wDzzwC007zkzVzzzVk00zy7wDzzsDzzzzkzVzzz7zzzzy7wDzzVzzzzzkzVzzwDzzzzy7wDzy7y007zkzVzzUzk00zy7wDzwDzzzzzkzVly7zzzzzy3sCDUzzzzzzwD7lwDzzzzzzk1yC00Dzzzzz0Dlk01zzzzzzzzzzzw"
Text.="|<0.2>*127$68.zzzzzzzzzzzzzzzzw1zzk1zzzzzy0Dzw0Dzzzzz01zw7VzzzzzVwDz3wDzzzzkzVzkzVzzzzwDsTyTsTzzzz3y7zzy6007zkzVzzzVU01zwDsTzzkTzzzz3y7zzwTzzzzkzVzzwDzzzzwDsTzz3zzzzz3y7zzXy007zkzVzzVzU01zwDsTzszzzzzz3y77sTzzzzzsT1lw7zzzzzz3lwT3zzzzzzs0z7007zzzzz0Dlk01zzzzzzzzzzzs"
Text.="|<0.>*132$23.zzzzzzzzzzzzk7zz07zw07zky7z3y7y7wDwDsTsTkzkzVzVz3z3y7y7wDwDsTsTkzkzVzVz3X1w77VsyDU3wTU7szzzzzzzz"
Text.="|<CBD>*110$55.zzzzzzzzzzzzzzzzzzzk0s0Dk0DzU0A03s01zUz2DkwDkTkzV7yC7z7kzlXz33zVkzzlzXVzksTzszVkzwADzwDVsTz67zy00wDzX3zz00C7zlVzzXz33zskzzlzkVzwMTzszwEzwADzwTy8TwD3z6Dz4Dy7kzX7z27z7sDkXz33w7w00k03U03zk0s03s0Dzzzzzzzzzzzzzzzzzzz"
ok:=FindText(0, 0, A_ScreenWidth, A_ScreenHeight, 0.2,0.1,Text)
;FindText().MouseTip(X, Y) 
arr:=FindText().OCR(ok), OCR:=arr.text
msgbox, %OCR%
msgbox, %  ok[A_index].id  ;Did not work
msgbox, % "The" OCR.1 "measures " OCR.2 " in diameter."   ;Did not work
msgbox, % "The" OCR[1] "measures " OCR[2] " in diameter."   ;Did not work
Return

ksh
Posts: 4
Joined: 29 Mar 2022, 08:56

Re: FindText - Capture screen image into text and then find it

Post by ksh » 07 Aug 2022, 22:12

When I run the attached FindText.exe v 2022-05-28 resulted from Ahk3Exe v 1.1.33.10, its GUI is not displayed. Am I doing something wrong?
Attachments
FindText.rar
(517.12 KiB) Downloaded 216 times

gregster
Posts: 8921
Joined: 30 Sep 2013, 06:48

Re: FindText - Capture screen image into text and then find it

Post by gregster » 07 Aug 2022, 22:32

ksh wrote:
07 Aug 2022, 22:12
When I run the attached FindText.exe v 2022-05-28 resulted from Ahk3Exe v 1.1.33.10, its GUI is not displayed. Am I doing something wrong?
I would recommend to post the source code you used, since many people - for good reasons - won't run exe files from unknown sources.
(Not that running your exe will tell them what went wrong anyway.)
Then they can try to reproduce the problem.

Did your code work uncompiled?

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: FindText - Capture screen image into text and then find it

Post by Descolada » 07 Aug 2022, 23:15

@iilabs, as far as I know you do need to split the string by the "*" character. Why the results aren't returned in an array I do not know, but if you really want to, you could modify the OCR method yourself (which actually shouldn't be too difficult).

FindTextOCR doesn't seem to be necessary, you can achieve the same result with FindText + FindText().OCR.

ksh
Posts: 4
Joined: 29 Mar 2022, 08:56

Re: FindText - Capture screen image into text and then find it

Post by ksh » 08 Aug 2022, 01:35

gregster wrote:
07 Aug 2022, 22:32
ksh wrote:
07 Aug 2022, 22:12
When I run the attached FindText.exe v 2022-05-28 resulted from Ahk3Exe v 1.1.33.10, its GUI is not displayed. Am I doing something wrong?
I would recommend to post the source code you used, since many people - for good reasons - won't run exe files from unknown sources.
(Not that running your exe will tell them what went wrong anyway.)
Then they can try to reproduce the problem.

Did your code work uncompiled?
Please find attached the code that I am compiling. I had downloaded the source code provided at page one of the topic and then compiled it but it was not clearly understood from my previous message.

Yes, it displays the message "Successfully compiled as: ...".
Attachments
FindText.ahk
(107.23 KiB) Downloaded 182 times

BaBa44
Posts: 5
Joined: 18 Feb 2022, 05:33

Re: FindText - Capture screen image into text and then find it

Post by BaBa44 » 10 Aug 2022, 11:09

Hello, it's me again ^.^

even though I tried using the 2nd SearchDirection (Right to Left, Top to Bottom), I wasn't able to get the rightmost, first match of a 1x1 pixel with a given colour with a range like 0.96.

Code: Select all

ok := FindText(X, Y, X1, Y1, X2, Y2, 0.0005, 0.0005, Format("|<{1}>0x{1}@{2}$1.k", ColorID, Round(1.00 - Ceil(Variation / 2.55) * 0.01 , 2)), , 0, , , , 2)
It does return the leftmost result.

I want that snip to return the rightmost result as fast as possible, because the target behind that in general is to measure the distance between two FindText-Queries.
Maybe I'm missing something or there's an even better approach?

Any help is highly appreciated!

Best regards
BaBa

User avatar
iilabs
Posts: 296
Joined: 07 Jun 2020, 16:57

Re: FindText - Capture screen image into text and then find it

Post by iilabs » 10 Aug 2022, 11:25

Descolada wrote:
07 Aug 2022, 23:15
@iilabs, as far as I know you do need to split the string by the "*" character. Why the results aren't returned in an array I do not know, but if you really want to, you could modify the OCR method yourself (which actually shouldn't be too difficult).

FindTextOCR doesn't seem to be necessary, you can achieve the same result with FindText + FindText().OCR.
Would you mind at an attempt or example? Will this involve editing the class?
Also, do you recommend for OCR to capture each letter or number without periods (3.14) and make one for the period and capture thinnest as possible?

BaBa44
Posts: 5
Joined: 18 Feb 2022, 05:33

Re: FindText - Capture screen image into text and then find it

Post by BaBa44 » 10 Aug 2022, 15:10

BaBa44 wrote:
10 Aug 2022, 11:09
Hello, it's me again ^.^

even though I tried using the 2nd SearchDirection (Right to Left, Top to Bottom), I wasn't able to get the rightmost, first match of a 1x1 pixel with a given colour with a range like 0.96.

Code: Select all

ok := FindText(X, Y, X1, Y1, X2, Y2, 0.0005, 0.0005, Format("|<{1}>0x{1}@{2}$1.k", ColorID, Round(1.00 - Ceil(Variation / 2.55) * 0.01 , 2)), , 0, , , , 2)
It does return the leftmost result.

I want that snip to return the rightmost result as fast as possible, because the target behind that in general is to measure the distance between two FindText-Queries.
Maybe I'm missing something or there's an even better approach?

Any help is highly appreciated!

Best regards
BaBa
Nevermind.
The problem seems to be that I've used BindWindow with a value of 4, because 0 didn't work.
After I disabled the line with that BindWindow-Call it all worked as expected.

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: FindText - Capture screen image into text and then find it

Post by Descolada » 11 Aug 2022, 00:55

@iilabs, easiest to just do StrSplit(result, "*") to split up the results into an array. Also my tutorial has some OCR examples, so take a look at those if you already haven't.

I don't have very much experience with using FindText OCR to read text from screen, but here are my experiences with it:
1) To avoid many irrelevant results and false positives, use as small search area as possible. Sometimes this can be done by searching for a reference point and then constraining the area to like 50x20 pixels area. For example if your numbers are located near some specific text, or near the cursor, then use that location to constrain the search area.
2) Small symbols are tricky to OCR correctly. For example the period might be confused with the dot on an "i" letter, or a comma, or plenty of other symbols. When I've had this problem, I tried to capture each character with as similar height as possible, and then constrain OCR method offsetX and offsetY to very small values. For example if my characters are all 10 pixels tall, then I might use offsetX:=5 (to capture only characters close together), and offsetY:=12 or 15. This way I don't get a lot of irrelevant matches... As you mentioned, the other way would be to capture the letters with and without periods, but this might also prove to be tricky, since sometimes the distance between characters varies, so you might need to capture a LOT of images. I haven't used this method myself.
3) To avoid false matches, I usually capture characters with at least 1 row of empty pixels to each side. This helps to avoid some problems like "q" being matched as "o", since if you don't leave any room underneath "o" then FindText might also find "q", "p" etc.
4) The order of characters in Text also matters: you might need to put less common characters in the beginning of Text and more common ones in the end. For example "l" might match the front part of "p", so put "p" before "l". FindText will try to match in the order that the characters appear in the Text, and this way it will match "p" first.

And sometimes FindText isn't the best option at all, you might be better served with dedicated OCR libraries (Vis2 for example).

User avatar
iilabs
Posts: 296
Joined: 07 Jun 2020, 16:57

Re: FindText - Capture screen image into text and then find it

Post by iilabs » 11 Aug 2022, 09:46

Thank you very much for the detailed explanation. Yes I choose findtext() function to be extremely fast compared to other OCRs and just needed really for numbers. But I’ve noticed that even after all the nuances you mentioned it differs from pc to pc so going to try the vis2. There are so many out there. If you have a saved link to your favorite one that will be much appreciated. I’ll review your tutorials. I’m assuming on the link to your tutorial on findtext(). Thanks again for your time. I’m the meantime I’ll parse out the data as you suggested to overcome that hurdle.

Descolada
Posts: 1101
Joined: 23 Dec 2021, 02:30

Re: FindText - Capture screen image into text and then find it

Post by Descolada » 13 Aug 2022, 13:11

Working on porting FindText to AHK v2: I ran it through the v1 -> v2 converter which sped up the process but also created a lot of problems... Now I've gotten it to the phase where the main GUI is mostly functional and simple searches work kinda reliably.
Also probably some new design choices need to be made. For example FindText v2 returns an empty array if the search failed, because otherwise in AHK2 for k,v in ok would throw an error if not checked before for 0 value.

If anybody wants to chime in and help the process, please submit pull requests on the GitHub page (or report issues): https://github.com/Descolada/FindText-v2

Balan
Posts: 2
Joined: 14 Aug 2022, 07:23

Re: FindText - Capture screen image into text and then find it

Post by Balan » 14 Aug 2022, 07:29

Hello. I need a script that, after pressing a key, will start pressing the Down button until it finds the text that I need. I also need to not appear gui. I don't know how to do that so I'm asking for your help.

User avatar
boiler
Posts: 16772
Joined: 21 Dec 2014, 02:44

Re: FindText - Capture screen image into text and then find it

Post by boiler » 14 Aug 2022, 07:35

@Balan - You can ask here specifically about how to use FindText, but you apparently are asking for help in writing your script in general. That will not be answered here. You can post those kinds of questions in Ask For Help.

chaoscreater
Posts: 59
Joined: 12 Sep 2019, 21:15

Re: FindText - Capture screen image into text and then find it

Post by chaoscreater » 19 Aug 2022, 23:51

In the older versions of FindText, you would first have to click "Capture" and then text will be generated inside the text box for you to copy into your script. If you didn't click on Capture, then the edit box would be blank.

For some reason, the newer version of FindText pre-populates the edit text box with whatever was there in the previous session and I don't want that. Is it possible to revert this to the old behaviour?

User avatar
rommmcek
Posts: 1470
Joined: 15 Aug 2014, 15:18

Re: FindText - Capture screen image into text and then find it

Post by rommmcek » 20 Aug 2022, 05:20

Try to comment %Gui_%("LoadScr") out (in my case line 2106 for v8.9).

grego90
Posts: 3
Joined: 20 Aug 2022, 06:20

Re: FindText - Capture screen image into text and then find it

Post by grego90 » 20 Aug 2022, 12:29

Used FindText for the first time. I capture my text, then test and obtained code to use in my script.
I'm trying to check the screen every 30 seconds and if findtext detected then open a .WAV file. How can I do?

c7aesa7r
Posts: 209
Joined: 02 Jun 2016, 21:09

Re: FindText - Capture screen image into text and then find it

Post by c7aesa7r » 24 Aug 2022, 00:12

@feiyue I'm converting the entire findtext class to c++, I wonder if you don't already have it there?

c7aesa7r
Posts: 209
Joined: 02 Jun 2016, 21:09

Re: FindText - Capture screen image into text and then find it

Post by c7aesa7r » 25 Aug 2022, 17:32

Code: Select all

     if IsFunc(k:="GetBitsFromScreen2")
       and %k%(bits, x-zx, y-zy, w, h)
     {
       zx:=bits.zx, zy:=bits.zy, zw:=bits.zw, zh:=bits.zh
       Critical, %cri%
       SetBatchLines, %bch%
       return bits
     }
feiyue is this used somewhere? i couldn't find a definition for GetBitsFromScreen2

feiyue
Posts: 349
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 26 Aug 2022, 04:38

This optional function is used to test other screen capture modes other than BitBlt:
Other screen capture modes are from @malcev research : :thumbup:
Screen capture to video with Directx9 / Directx11 and Microsoft Media Foundation
Finding image with IDXGIOutputDuplication + Gdip_FastImageSearch()

Code: Select all

GetBitsFromScreen2(bits, x, y, w, h)
{
  global dxgi_mode
  Try
  {
    if (dxgi_mode=1)
      DXGI(bits, x, y, w, h)
    else if (dxgi_mode=2)
      D3D9(bits, x, y, w, h)
    else if (dxgi_mode=3)
      ; DwmGetDxSharedSurface or DwmpDxGetWindowSharedSurface
      DWM(bits, x, y, w, h)
    else if (dxgi_mode=4)
      OBS(bits, x, y, w, h)
    else  ; dxgi_mode=0
      return 0  ; Use Bitblt in FindText()
  }
  return 1
}

Post Reply

Return to “Scripts and Functions (v1)”