[v2] FindText - Capture screen image into text and then find it

[feiyue]

Updated to 9.6 version - 2024/05/25
1. Modify: X:="wait/wait1/wait0" After running once, X will be cleared without affecting subsequent calls.
2. Modify: The color similarity and color difference binarization mode can mix multiple colors for binarization.
Text:="|<>RRGGBB1@0.8/RRGGBB2-DRDGDB2/... $ ..."
3. Add: ColorBlock() function, which can generate color blocks with specified area and color, and limit the number of matching colors.
4. Add: Add the GuiCreate Class in V1 to make Gui related code compatible with V2 and reduce the difficulty of code maintenance.

[feiyue]

Updated to 9.7 version - 2024/06/18
1. Modify: The use of the left and right dilation algorithm greatly improves the success rate of searching for slightly deformed images.
You need to set at least one fault tolerance to be less than 0 in order to enable the left and right dilation algorithm.
In this case, please set the fault tolerance to be very small, otherwise it may lead to incorrect matching.
2. Add: Use FindText().GetTextFromScreen() quickly generates Text to the clipboard.
3. Modify: Skip searching for overlapping areas around the image (FindMultiColor).

[redrum]

Nevermind, I think I figured it out. I'm new to FindText and didn't realize that the version I was using was Descolada's port of FindText to AHK v2, and that feiyue's port of FindText to AHK v2 was different, so it wasn't a simple change of libraries, as I had to make some minor script modifications since the two libraries have slightly different returns and default search behavior.

[qomph]

In Fortnite, Lumberjack Heroes, I was searching for x1 Charged using v9.6, and that worked fine.

Now using same Text code, it finds x3 Charged when searching for x1 Charged.

I can try recapture (or just continue using v9.6), but what changed?

https://github.com/tallpeak/AHK/tree/main/fortnite/lumberjack
(using 9.6 from latest checkin)

[feiyue]

@qomph
Compared to the old 9.6 version, the new 9.7 version does have both advantages and disadvantages. The advantage is that it uses left and right fault-tolerant matching internally, greatly improving the success rate of matching. The disadvantage is that it cannot accurately distinguish subtle image differences.
Because after binarization of the screen image, the left and right dilation algorithm expands each black and white point to the left and right, and after overlapping with each other, blurs the details of the image, which is equivalent to a large internal fault tolerance. Therefore, the external fault tolerance should not be too large (it is recommended not to exceed 0.1), otherwise it is easy to make incorrect matches.

For previous scripts, the new version may result in incorrect search results because the previous script may have set a larger fault tolerance value, and the new version needs to reduce the fault tolerance value.

[qomph]

This found inaccurate results using v9.7 ("x4 Charged" instead of "x2 Charged")

Code: Select all

ok:=FindText(&X, &Y, 1344-xtrax, 242-xtray, 1344+xtrax, 242+xtray, 0, 0, Text)

This one works correctly:

Code: Select all

ok:=FindText(&X, &Y, 1344-xtrax, 242-xtray, 1344+xtrax, 242+xtray, 0.01, 0.01, Text)

So it seems err tolerance arguments (7th and 8th args) default to a nonzero value when 0 is passed?
==
FYI:

This is the pattern decoded from below Text by FindText, which is clearly "x2 Charge":

Code: Select all

Text:="|<x2Charge>*240$52.zPynTzzjxhjvxSRFtzyzjqynPPbryzPvRhxwzvBhhzrqkzlymrTbU"

000000_00_0000000_00__00_000000000000000000_00000000
_00_00_00_0000000_000000_0_0000__000_0_0___0000__000
000000000_0000000_000000_00_00000_00__00_00_00_00_00
0__00000_00000000_000000_00_00000_00_000_00_00_00000
_00000__000000000_00__00_00_00_00_00_000000000_00000
_00_00____00000000___000000_00__0_00_000_000000__000

This shows 9.7 finding the wrong match using the 0 arguments:

Screenshot 2024-06-20 201048.png (7.94 KiB) Viewed 1237 times

[Mod edit: Added [code][/code] tags. Please use them yourself when posting code!]

[feiyue]

@qomph
I have updated version 9.7 to no longer default to using the left and right dilation algorithm.
Now you need to set a fault tolerance to less than 0 to enable the left and right dilation algorithm, which will not affect the previous code.

[kashmirLZ]

Usage: (required AHK v2.02)

The latest download is 2.0.17
Did you mean 2.0.2 ?

[gregster]

Usage: (required AHK v2.02)

The latest download is 2.0.17
Did you mean 2.0.2 ?

I would guess that this is just the minimum requirement, but I might be wrong.

[kashmirLZ]

Look at the decimal places closely to see the issue

[gregster]

I see. Since there is no 2.02 yet (or rather 2.2 according to the used version system; currently, there are only 2.1 alpha versions available), you can - with a high likelihood - assume that just a dot is missing.
Or, you could just try it.

[kashmirLZ]

Fun addition: RangeTip (the red/blue flashing box) but with a crosshair!
I use FindText primarily for clicking on precise locations, so this works great for me when selecting and testing. This shows a crosshair as you test and capture!
note: will also affect mousetip()

Code: Select all

; Shows a range of the borders (with a centered crosshair)
RangeTip(x:="", y:="", w:="", h:="", color:="Red", d:=3)
{
  ListLines (lls:=A_ListLines)?0:0
  static Range:=[0,0,0,0,0,0]  ; Increase array size to accommodate 6 GUIs
  if (x="")
  {
    Loop 6
      if (Range[i:=A_Index])
        Range[i].Destroy(), Range[i]:=""
    ListLines lls
    return
  }
  if (!Range[1])
  {
    Loop 6
      Range[A_Index]:=Gui("+AlwaysOnTop -Caption +ToolWindow -DPIScale +E0x08000000")
  }
  x:=this.Floor(x), y:=this.Floor(y), w:=this.Floor(w), h:=this.Floor(h)
  , d:=this.Floor(d)
  thin_d := d / 2  ; Define a thinner thickness for the crosshair lines
  Loop 6
  {
    i:=A_Index
    if (i <= 4)  ; For the rectangle borders
    {
        x1:=(i=2 ? x+w : x-d)
      , y1:=(i=3 ? y+h : y-d)
      , w1:=(i=1 || i=3 ? w+2*d : d)
      , h1:=(i=2 || i=4 ? h+2*d : d)
    }
    else  ; For the crosshair lines
    {
        if (i = 5)  ; Horizontal line
        {
            x1 := x
            y1 := y + (h / 2) - (thin_d / 2)
            w1 := w
            h1 := thin_d
        }
        else if (i = 6)  ; Vertical line
        {
            x1 := x + (w / 2) - (thin_d / 2)
            y1 := y
            w1 := thin_d
            h1 := h
        }
    }
    Range[i].BackColor:=color
    Range[i].Show("NA x" x1 " y" y1 " w" w1 " h" h1)
  }
  ListLines lls
}

[kashmirLZ]

@feiyue
There is a significant slow-down in searching when using concatenated search-text-strings even if the first string matches!. example:

Code: Select all

"|<one>*137$75.sTztzy0....kn1U|<two>*140$80.jfIAfnajvz....zzU"

Here "one" matches. So why does the script continue to search for all the others?

[Descolada]

@kashmirLZ the default is to find all matches in the specified region. Set the FindAll argument to 0 to stop at the first match.

[qomph]

Note: In the Click() function, I inserted Sleep(33) between MouseMove and Click. It seems quite necessary; I "upgraded" from 9.6 to 9.7 and some old glitchy behaviors returned (sometimes missing the button.)

rg -C2 "MouseMove x" .\findtextv2_v9.7.ahk
...
2048: MouseMove x, y, 0
2049- Sleep(33) ; AWW:trying to fix my Lumberjack Heroes frenzyloop macro that glitches at low wattage (when CPU is slow)
2050- Click x "," y "," other1 "," other2

[redrum]

I have a situation where the same table column header in 2 instances of the same app on the same monitor are not being seen as the same by FindText. Using Grey2Two, I've tried grey thresholds between 80 and 180 (~130 is the default), and FindText will only find one of these (whichever I captured the findtext from). I am cropping out the color difference at the bottom of the images. Running FindText v9.7.

What else should I try?

image1.png (1.18 KiB) Viewed 489 times

image2.png (1.32 KiB) Viewed 489 times

[feiyue]

you can set err1:=-0.05, err0:=0.05

FindText(&x,&y,0,0,0,0,err1:=-0.05,err0:=0.05,Text)

[redrum]

you can set err1:=-0.05, err0:=0.05

FindText(&x,&y,0,0,0,0,err1:=-0.05,err0:=0.05,Text)

Thanks!

[rockitdontstopit]

you can set err1:=-0.05, err0:=0.05

From @Descolada ahk v1 tutorial.
"If err1 and err0 are 0 and no match is found, then FindText will automatically try again with 0.05=5% error margins."

Is this true for FindText v2?

[qomph]

Note, again referencing latest AHK code at:
https://github.com/tallpeak/AHK/tree/main/fortnite/lumberjack

I've returned to v9.6, because I had a problem with AHK crashing, and my macros run for many hours since with no crashes.

The crashing started after installing "2024-07 Cumulative Update for Windows 11 Version 23H2 for x64-based Systems (KB5040442)". (I don't know how that's relevant, but a system restore caused other problems and I ended up accidentally letting the update reinstall.)

WinDbg says this on my minidump file:
FAILURE_BUCKET_ID: NULL_POINTER_WRITE_c0000005_AutoHotkey64.exe!unknown_error_in_process
and .. FRAME_ONE_INVALID: 1
Seems like the stack is smashed?

Could you have introduced a null-pointer exception somehow in v9.7? Buffer overflow somewhere on the stack?

I ran AHK main from github (AutoHotKeyx) in the latest version of Visual Studio 2022 to try to debug it further, but still seemed to have a mess on the stack when it crashed. (I guess I should have copied some disassembly, at least...)

I'm thinking it's likely some MCode...

I know I should debug further but reverting to 9.6 fixed my problem... if I were to debug further, how would I do that? The only thing I can think of is "printf debugging", eg. create a wrapper for each FindText function call that logs its arguments so I can narrow down the problem a bit. Log exact nanoseconds too, so that I can have some confidence in whether the crash occurred during an MCode call (DllCall).

Do you have an example of a wrapper? I know it should be trivial to write one...