FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools for AHK v1.1 and older
Kei[bab]
Posts: 1
Joined: 28 Jul 2021, 03:33

Re: FindText - Capture screen image into text and then find it

28 Jul 2021, 03:42

hi, feihye
it works super fast, which i have never seen in any image searcher before, thanks for your great work, it's even neater without having to store images like image search before, well done
byzod
Posts: 87
Joined: 21 Jun 2021, 06:46

Re: FindText - Capture screen image into text and then find it

30 Jul 2021, 20:43

不知道是否能支持透明背景模式,指定特定像素为透明背景,可以匹配白点或黑点
类似于ImageSearch的*TransN模式
*TransN: 此选项通过指定图像内的某种颜色能匹配屏幕上的任何颜色, 使得更容易找到匹配. 它常用于寻找含有透明区域的 PNG, GIF 和 TIF 文件(然而, 对于图标则不需要此选项, 因为它们的透明度是自动支持的). 对于 GIF 文件, *TransWhite 很可能会有用. 对于 PNG 和 TIF 文件, *TransBlack 可能是最佳的. 否则, 指定 N 为其他颜色名称或 RGB 值(请参阅颜色图表进行了解或使用 PixelGetColor 的 RGB 模式). 例如: *TransBlack, *TransFFFFAA, *Trans0xFFFFAA.
例如用*表示透明背景

Code: Select all

_0_   
_*_
_0_
_0_
可以匹配以下目标

Code: Select all

_0_		_0_
___		_0_
_0_		_0_
_0_		_0_
但是不会匹配以下目标

Code: Select all

00_		_0_
___		_0_
_0_		_0_
_0_		000
这是目前仅通过设置err0为1做不到的


一个实际的例子是,匹配某游戏道具图标,它在右下角有数量显示,并且仅通过裁剪不包含数量的矩形区域化,无法与其他类似道具区分开
比如道具ABCD分别为(假设右下角4个像素为数量显示)

Code: Select all

__0000_		__000_0		__000_0		__00000
__0____		__0____		__0____		__0__0_
_0___00		__0__00		_0___00		__00_00
__0__00		__0__00		__0__00		__0__00
显然如果不包含数量区域,无论截取哪个部分区域,都可能无法区分道具AB或者道具BC;如果截取整个图形,把数量区域改为背景,并且取err0=1,又会匹配到道具D
但如果支持透明区域(以*表示),事情就简单了,道具ABCD的匹配图形分别为

Code: Select all

__0000_		__000_0		__000_0		__00000
__0____		__0____		__0____		__0__0_
_0___**		__0__**		_0___**		__00_**
__0__**		__0__**		__0__**		__0__**


https://translate.google.com/

I don’t know if it can support the transparent background mode, specify a specific pixel as a transparent background, you can match the white point or black point
*TransN mode similar to ImageSearch

*TransN: This option can match any color on the screen by specifying a certain color in the image, making it easier to find a match. It is often used to find PNG, GIF and TIF files that contain transparent areas (however, this is not required for icons Option, because their transparency is automatically supported). For GIF files, *TransWhite is likely to be useful. For PNG and TIF files, *TransBlack may be the best. Otherwise, specify N as another color name or RGB value (please Refer to the color chart for understanding or use the RGB mode of PixelGetColor). For example: *TransBlack, *TransFFFFAA, *Trans0xFFFFAA.

For example, use * to indicate a transparent background

Code: Select all-Toggle Line numbers

_0_
_*_
_0_
_0_

Can match the following goals

Code: Select all-Toggle Line numbers

_0_ _0_
___ _0_
_0_ _0_
_0_ _0_

But will not match the following targets

Code: Select all-Toggle Line numbers

00_ _0_
___ _0_
_0_ _0_
_0_ 000

This is currently impossible to do only by setting err0 to 1.


A practical example is to match a certain game prop icon, which has a quantity displayed in the lower right corner, and only by cropping a rectangular area that does not contain the quantity, it cannot be distinguished from other similar props.
For example, the props ABCD are respectively (assuming that 4 pixels in the lower right corner are the number display)

Code: Select all-Toggle Line numbers

__0000_ __000_0 __000_0 __00000
__0____ __0____ __0____ __0__0_
_0___00 __0__00 _0___00 __00_00
__0__00 __0__00 __0__00 __0__00

Obviously if the quantity area is not included, no matter which part of the area is intercepted, it may not be able to distinguish between the props AB and the props BC; if the whole figure is intercepted, the quantity area is changed to the background, and err0=1, it will be matched to the prop D
But if it supports transparent areas (indicated by *), things are simple, the matching graphics of props ABCD are respectively

Code: Select all-Toggle Line numbers

__0000_ __000_0 __000_0 __00000
__0____ __0____ __0____ __0__0_
_0___** __0__** _0___** __00_**
__0__** __0__** __0__** __0__**
User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: FindText - Capture screen image into text and then find it

31 Jul 2021, 06:57

SteveMylo wrote:
28 Jul 2021, 02:35
STOPING a LOOP doesn't seem to work. I know how to use FindText, it's great but I have this Loop Problem.
Ok so after a couple of days I figured it out. You have to use a gosub which seems to solve a lot of issues with functions and FindTEXT.
The last (gosub, doit ) you see on the script is the only way I can get a loop to perform normally and not forever with this Function. Normal LOOP can't be broken or stopped.
I'm still only an intermediate Autohotkey user so there may be a better way.
Here is my script. I searched for two images and when either was found it would perform a MouseClickDrag which you'll see at the bottom of the script. It's quite daunting that you have to have the 'enter' values perfect otherwise the whole thing doesn't work properly but FindText has been such an amazing tool for my business.

Code: Select all

#Include C:\Users\Mylo\Documents\AutoHotkey\lib\Functions\FindText.ahk	
#NoEnv
SetWorkingDir %A_ScriptDir%
CoordMode, Mouse, Screen
#SingleInstance, Force
SetTitleMatchMode 2
SetTitleMatchMode Fast
SetControlDelay 1
SetWinDelay 0
SetKeyDelay -1
SetMouseDelay -1
#Persistent

n::

as:="|<HistO>##0$0/0/A88726,1/7/A88726,1/1/2D240A,0/6/211B08,-1/3/876D1F,2/4/876D1F,0/14/463810,0/21/463810,0/10/000000,0/13/000000,0/8/000000,2/14/000000,2/21/000000,4/4/000000,13/4/000000,22/4/000000,29/4/000000,-4/4/000000,-7/4/000000,-11/4/000000,-14/4/000000"
If ErrorLevel
	Return
if (ok:=FindText(1217, 640, 1919, 1078, 0, 0, as))
	gosub, whiteC 
return




whiteC:
ty:="|<whiteC>##0$0/0/D9D4D4,1/0/D9D4D4,0/5/D9D4D4,1/5/D9D4D4,3/2/D9D4D4,3/3/D9D4D4,-2/3/D9D4D4,-2/2/D9D4D4,1/2/D9D4D4,5/3/131212,17/3/272727"
If ErrorLevel
	return

if (ok:=FindText(745-150000, 966-150000, 745+150000, 966+150000, 0, 0, ty))
	{
		SoundBeep
		CoordMode, Mouse
		X:=ok.1.x, Y:=ok.1.y, Comment:=ok.1.id
		MouseMove, X-7, Y-2, 1
		gosub, doit
		return
	}

qw:="|<WhiteC_hole>**77$9.7vVM6SbwnaQznn0Q8zU"
If ErrorLevel
	return

if (ok:=FindText(170, 816, 1912, 1038, 0, 0, qw))
	{
		SoundBeep
		CoordMode, Mouse
		hX:=ok.1.x, hY:=ok.1.y, Comment:=ok.1.id
		MouseMove, hX, hY, 1
		gosub, doit
	}
return




doit:
as:="|<HistO>##0$0/0/A88726,1/7/A88726,1/1/2D240A,0/6/211B08,-1/3/876D1F,2/4/876D1F,0/14/463810,0/21/463810,0/10/000000,0/13/000000,0/8/000000,2/14/000000,2/21/000000,4/4/000000,13/4/000000,22/4/000000,29/4/000000,-4/4/000000,-7/4/000000,-11/4/000000,-14/4/000000"
If ErrorLevel
	Return
if (ok:=FindText(1217, 640, 1919, 1078, 0, 0, as))
	{
		SetMouseDelay, 1
		MouseClickDrag, Left, 0, 0, 7, 0, 1, R
		gosub, doit
	}
return
feiyue
Posts: 349
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

31 Jul 2021, 21:01

@SteveMylo ok:=FindText() Do not modify the value of ErrorLevel.
If you want to judge whether it is successful, you can use if (ok) to judge.
The use of FindText.ImageSearch() is similar to the built-in function ImageSearch. It will modify the value of ErrorLevel.

Code: Select all

; eg:
ImageSearch, rx, ry, 0, 0, 500, 500, c:\a.bmp
if (ErrorLevel=0)
  MouseMove, rx, ry
; ==>
#Include <FindText>
text:="|<>*50$10.asdasd"
FindText.ImageSearch(rx, ry, 0, 0, 500, 500, text)
if (ErrorLevel=0)
  MouseMove, rx, ry
feiyue
Posts: 349
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

31 Jul 2021, 21:38

@byzod If you want to use FindText for games, I recommend using MultiColor search mode. :beer: :dance:

@thebbandit FindText supports multiple displays, so the image captured by BitBlt is the image of the virtual desktop (from GetDesktopWindow()). I don't know why.

@byzod The default fault tolerance of FindText is two zeros.
For a single text to be found, if it is not found, it will use 5% fault tolerance to find it again. For multiple text to be found, it will directly use 5% fault tolerance to find it.
If you don't want to use fault tolerance, you can set the fault tolerance to a very small number (don't use two zeros), such as 0.00001
I still recommend carefully choosing to search for a unique image to avoid not using fault tolerance.
byzod
Posts: 87
Joined: 21 Jun 2021, 06:46

Re: FindText - Capture screen image into text and then find it

01 Aug 2021, 00:43

@feiyue
那么这个MultiColor模式,或者说除了灰度阈值以外的所有其他模式,究竟要咋用?所有的模式都没有教程,也没有文档,只有语焉不详的几个说明单词,摸索了几天完全没有头绪
还是说,只可意会,不可言传,是我悟性不够



关于之前的这个bug,也就是说,要完成无偏差搜索,不能用0作为err1 err0参数,否则其实设为0就等于0.05?只能用0.00001之类的奇怪数值来获得真正的0的效果?

我只能说老哥的设计思路非常地……富有创意
如果还有其他类似的隐藏魔数,能不能提前告知,或者写到文档里,谢谢

So how should this MultiColor mode, or all other modes except grayscale threshold, be used? There are no tutorials or documents for all modes, only a few vague explanations, and I have no clue after a few days of groping.
Or, I can only understand it, not tell it, it’s because I don’t have enough savvy



Regarding the previous bug, that is to say, in order to complete an unbiased search, you cannot use 0 as the err1 err0 parameter, otherwise, if it is set to 0, it is equal to 0.05? Can only use strange values like 0.00001 to get the real 0 effect?

I can only say that my design idea is very... creative
If there are other similar hidden magic numbers, can you tell them in advance or write them in the document, thank you
User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: FindText - Capture screen image into text and then find it

06 Aug 2021, 06:35

Ignore below post: I'm astonished to find that one cannot delete their own posts. Wow.


;//////
Would the FindText tool have a way to Find the Darkest part of an area?
So, the Darkest R or G or B value of an area?
It could be the darkest black, or maybe there is no black but the darkest part would be Red.
I would like to play around with it in my Colour grading software Davinci Resolve.
Last edited by SteveMylo on 11 Aug 2021, 21:47, edited 1 time in total.
necomerx
Posts: 57
Joined: 01 Jan 2021, 09:09

Re: FindText - Capture screen image into text and then find it

07 Aug 2021, 00:52

Hi,
Is it possible to find text or more precisely match text found on 2 image captures with Findtext?
I capture 2 images of a text or numbers and i want to write some actions if for exemple the word "monday" or "52" is matched. Is that possible ?
Gamesiarz
Posts: 4
Joined: 04 Jan 2020, 15:20

Re: FindText - Capture screen image into text and then find it

09 Aug 2021, 09:27

Hi I want to say thank you for this amazing script. It's so fast and accurate :)
ppapkor
Posts: 14
Joined: 30 Jan 2019, 03:22

Re: FindText - Capture screen image into text and then find it

13 Aug 2021, 23:39

Thank you for your nice script.
I've been using your script for over 4 years and I really can't thank you enough.

anyway today I've experienced a error.
I always use findtext tolerance to 0 but even if it's not exact match, like 1or 2 pixels error, findtext alert me it found a match.
when I use findtext() for small area like 5x5 or 6x6, it doesn't match if area has 1 or 2 pixels small error.
but when I use findtext() for large area like 20x7 or more, 1 or 2 pixel error is acceptable.
Is it intended? or just my script error?


+) and i've just noticed looking around previous post, Tolerance 0,0 option becomes 0.05,0.05 if it's failed once.
so......... if I want only exact match, 0.000001,0.000001 is my best chance right?
Thank you for your sharing . your code is very fancy and you are genious i guess :bravo:
felixcatto
Posts: 6
Joined: 15 Aug 2021, 05:25

Re: FindText - Capture screen image into text and then find it

15 Aug 2021, 08:04

feiyue, big thanks for this function! Works perfectly!
Also thanks to ed1chandler and c4p for explanation how to use it.
User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

a handy Script for Loop 'multiple' images until found :-)

16 Aug 2021, 07:47

image.png
AllAdd Button
image.png (665 Bytes) Viewed 4686 times
Here is an easy way for FindText to search multiple images Until Found and MouseMove to the 1st one it sees.
Note: Every time you do a 'Capture' just click the 'ALLAdd' button Image

Code: Select all

 x::
Text:="|<start>**50$34.zzyTzz00B20NiQobDbdVGIWCa79GASPQB8QtAkoXtY39m/C7Yr8UsKHAXyzDzy8"
Text.="|<kill>**50$22.zznmN999AYYZmGG7998oYYVmGGL999AbbaG67zzzs"
Text.="|<chrome>**50$44.Dz0000062E00003CY00000btzzzzjd23662CCEaH9aN9Y9Yn9aENyNAmNYzCaH9aNDsNYsNaMLzzvzzzy"
Text.="|<youtube>**50$52.xzxvzyzzyQsQc0+EMRnAmbAdAbmNv+ImYmFdZYdH+GNqCGGZAd1VAtN+ImYmQmbgtHCHV1/AnBAnAb4i7VYsQ64STnySzzzs"

Loop
	{
		if (ok := FindText(20, 29, 1918, 999, 0, 0, Text))
			{
				X:=ok.1.x, Y:=ok.1.y, Comment:=ok.1.id
				CoordMode, Mouse, Screen
				Sleep, 111
				SoundBeep
				Sleep, 333
				MouseMove, X, Y, 1
			} 
	}
until ((ok != 0)(a_timesincethishotkey > 3000))  ; 3 second timer on the loop. 

;if you don't want a timer, then just use -  Until (ok != 0)  and it will loop until found. 

;~ Until (ok != 0)
return
Last edited by SteveMylo on 18 Aug 2021, 19:14, edited 2 times in total.
fabricio234
Posts: 122
Joined: 06 Mar 2020, 21:48

Re: FindText - Capture screen image into text and then find it

17 Aug 2021, 23:05

@feiyue I'm trying to understand how to properly use the option MultiColor, in the input of R/G/B it says allowed error (0-255), is this value something like a tolerance?
image.png
image.png (7.04 KiB) Viewed 4624 times
My doubts:

1) How do i calculate what % of tolerance represents the value insert in R/G/B (0-255)?
Like you specified for err1 and err0:

Code: Select all

;    , err1 --> Fault tolerance percentage of text       (0.1=10%)
;    , err0 --> Fault tolerance percentage of background (0.1=10%)
2) When using a MultiColor Text, it does ignore the fault tolerance in the function header? I mean the 0.1, 0.1 here:

Code: Select all

if (ok:=FindText(x, y, w, h, 0.1, 0.1, Text, 1, 1)) {
	Msgbox OK
}
or it does combine the tolerance added in the R/G/B plus the tolerance (0.1, 0.1)?
feiyue
Posts: 349
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

18 Aug 2021, 01:57

The allowable error of the color in the Text string is whether a single color matches.
The fault tolerance in parameters ( 0.1, 0.1 ) is aimed at how many points can not match. :)
fabricio234
Posts: 122
Joined: 06 Mar 2020, 21:48

Re: FindText - Capture screen image into text and then find it

18 Aug 2021, 09:52

Sorry i still dont understand, i forgot to mention that the use case is, MultiColor plus the FindMultiColor, and not just one single color.
feiyue
Posts: 349
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

18 Aug 2021, 12:30

Because I use automatic translation, I don't know if you can understand the following explanation. :)

In the multi-color search mode, you can find the image by finding the colors of multiple relative positions,
which turns the image search into finding several points of different colors on the image, which can omit the image file and speed up the speed.

Just as the built-in ImageSearch command allows gradient values,
the code generated by the multi-color search mode allows gradient values,
such as color 0x505050, if the allowed gradient value is 0x10 (or 16), 0x404040 ~ 0x606060 are equivalent to successfully matching the color 0x505050.

What happens if a color still doesn't match after setting the gradient value ?
The built-in ImageSearch command will directly return not found,
but FindText allows certain fault tolerance. For example, set 30% (err1 := 0.3) to allow 3 mismatches in 10 color sequences.

Of course, the colors captured by the multi-color search mode usually do not exceed 10 points,
so using the default fault tolerance (err1: = 0) is equivalent to not allowing fault tolerance,
Therefore, it is recommended to use a certain gradient value (such as 16) instead of the default 0.
For example: Text:="|<>##16$0/0/505050,......" instead of Text:="|<>##0$0/0/505050,......"
Humbug
Posts: 18
Joined: 14 Jul 2017, 14:11

Re: FindText - Capture screen image into text and then find it

31 Aug 2021, 14:40

Hi there,
I am a first time user of this script.
I'd like to create a hotkey that right clicks and picks a specific menu item from the popup menu that shows up.
The popup menu might be anywhere on screen, so I use:

Code: Select all

ok:=FindText(0, 0, 2560, 1600, 0, 0, Text)
However, it would seem that the menu item is not always found.
The background is black and the text is white and I guess there is some edge smoothing that might interfere with the scanning(?).
I attached some sample pics taken at different locations on the screen.
Here's the text as shown on the UI:
0_______0000000000000000000000000000000___000000_____________0000000000
__0000__00000000000000000000000000000000__000000___00___00___0000000000
__00000_00000000000000000000000000000000__000000__000___000__0000000000
__00000_00000000000000000000000000000000__00000000000___000000000000000
___0000000000_____000___0____000000_______00000000000___00000000______0
____00000000__00___00_________0000__000___00000000000___0000000___00___
0_____00000__0000__000__0000__000___000___00000000000___000000___0000__
000____0000________000__0000__000__00000__00000000000___000000__00000__
0000____000_______0000__0000__000__00000__00000000000___000000__00000__
000000__00___000000000__0000__000__00000__00000000000___000000__00000__
_00000___0___000000000__0000__000__00000__00000000000___000000__00000__
_00000__000__000000000__0000__000__00000__00000000000___000000__00000__
__0000__000___000__00___0000__000___000___00000000000___000000___000__0
____0__00000______000____00____000_________00000000_______00000_______0
0_____0000000____0000____00____0000___00__000000000_______000000____000


In the earlier versions of the FindText script there used to be an 'invert' function - I wonder if that would help.
Why was that feature removed?

Not sure how else I could improve accuracy.
Additional fault tolerance did not help:

Code: Select all

ok:=FindText(0, 0, 2560, 1600, 0.25, 0.25, Text)
Any thoughts?
I am thinking, I need the inverse mode back because the role of the foreground and background pixels are swapped in my case.

Something else.
The hotkey does not seem to work so I can't use CaptureS.
I tried running it as admin, still no luck.
May have something to with trying to use it with a fullscreen borderless DirectX window.
Can the hotkey method be tweaked somehow?

I am also having problems scrolling the image - the bars become grayed out when I start dragging them, so I can't see what I trim off.
Is there any reason why the main window is not resizable?
I have a pretty big screen ;-)

Thank you!
Attachments
Item3.png
Item3.png (2.21 KiB) Viewed 4273 times
Item2.png
Item2.png (2.23 KiB) Viewed 4273 times
Item1.png
Item1.png (4.16 KiB) Viewed 4273 times
User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: FindText - Capture screen image into text and then find it

02 Sep 2021, 01:31

Humbug wrote:
31 Aug 2021, 14:40
I need the inverse mode back because the role of the foreground and background pixels are swapped in my case.
Can't you just click on the black around the text for inverse mode? That's what i do
Humbug
Posts: 18
Joined: 14 Jul 2017, 14:11

Re: FindText - Capture screen image into text and then find it

02 Sep 2021, 02:30

SteveMylo wrote:
02 Sep 2021, 01:31
Humbug wrote:
31 Aug 2021, 14:40
I need the inverse mode back because the role of the foreground and background pixels are swapped in my case.
Can't you just click on the black around the text for inverse mode? That's what i do
Hi SteveMylo,
I could, yes, except the text is not found.
I figured inverting the picture first and then using the grayscale conversion would yield better results.
Except the invert feature has been removed, it seems.
feiyue
Posts: 349
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

02 Sep 2021, 20:16

@Humbug :beer:
1. The search image should be as small as possible, so it is faster and more reliable.
2. For images that can switch the foreground and background colors,
I suggest capturing the image twice, and then connecting it to find it at one time (Text := Text1 . Text2).
3. For the default fault tolerance (0,0), you often need to adjust it yourself in actual use, such as (0.1,0.1).

Return to “Scripts and Functions (v1)”

Who is online

Users browsing this forum: gwarble and 124 guests