FindText - Capture screen image into text and then find it

[Epoch]

Is there a way (maybe even built-in within the function?) to set a search range related to the mouse cursor position at any given time?
Meaning, search for and only execute the part of the code within the “if (ok:=” brackets if the image is under the cursor or within a predetermined range related to it i.e X-5 until X+5 and Y-5 t until Y+5? In other words, pretty much replace the predetermined search range with one related to the cursor? That would be an excellent and instant (since the search area would be so small) way to make context sensitive hotkeys, so I am sure I can’t be the only one who thought about it?

[SteveMylo]

@Epoch I think what you are after is just getting the mouse position at the beginning yeah?

Code: Select all

MouseGetPos, aX, aY

if (ok:=FindText(X, Y, aX-50, aY-50, aX+50, aY+50, 0, 0, Text))
{
  FindText().Click(ok[1].x, ok[1].y, "L", 1)
}

[Epoch]

@Epoch I think what you are after is just getting the mouse position at the beginning yeah?

Code: Select all

MouseGetPos, aX, aY

if (ok:=FindText(X, Y, aX-50, aY-50, aX+50, aY+50, 0, 0, Text))
{
  FindText().Click(ok[1].x, ok[1].y, "L", 1)
}

Always fast and to the point, thanks mate. I figured it'd most likely have to do with MouseGetPos, but I didn't expect you could just put the X and Y from that into the Findtext function. Dunno, I think this could be really useful with audio or video editing programs, you can have the same mouse button doing different things depending where your cursor is (toolbars, video/audio timeline, piano rolls etc.). Cheers!

[SteveMylo]

@Epoch Yeah It is, I do it, It just isn't reliable on other peoples computers cause of different screen resolutions and scales. But yeah sure make your own it's great

[pirik33]

is this encodes image to base64?

[Descolada]

@pirik33, it encodes a pixelated ASCII style representation of the image to base64, yes.

[chaoscreater]

Hi guys, I use an app called Authy Desktop and it got updated automatically today (it's possible to downgrade, but the upgrade happens automatically). After the upgrade, it broke my scripts. Turns out that this app uses Electron and the app can now hide itself from screenshot apps, including FindText. I've tried Window's native screenshot and also a 3rd party tool called PicPick and they all cannot take screenshots of Authy Desktop, it'll just hide itself. For apps that have a similar issue, how can we use FindText to scan the images?

[feiyue]

Updated to 9.0 version - 2023/04/15
1. Modify: Fixed the bug that the returned width and height were incorrect if the image scaling parameter (zoomW, zoomH) was used. In version 8.9.
2. Modify: Fixed a bug in the compiled program where the generated code does not contain library function content.
3. Modify: When searching for image files, a transparent color can be specified without the need to create transparent images yourself.
You can use Text:="##10-RRGGBB $ d:\a.bmp", then the 0xRRGGBB(+/-10) as transparent color.
You can use FindText().ImageSearch(x,y,0,0,0,0, "*10 *Trans112233 d:\a.bmp")
4. Add: Added WaitNotChange() function.

[Descolada]

@chaoscreater, setting display affinity to 0 for Authy Desktop seems to fix that problem. I'm not sure whether it's necessary to call SetWindowDisplayAffinity before every FindText call or just periodically, but the following code seems to fix it. Unfortunately it's an incomplete solution, because I only got it working with Authy Desktop, and calling it on other programs will most likely crash the program. Perhaps somebody else can figure out the cause (I suspect it's because LoadLibrary isn't called from the external thread?).

Code: Select all

hWnd := WinExist("Twilio")
OutputDebug, % GetWindowDisplayAffinity(hWnd) "`n"
SetWindowDisplayAffinity(hWnd, 0)
OutputDebug, % GetWindowDisplayAffinity(hWnd) "`n"

GetWindowDisplayAffinity(hWnd) {
    DllCall("GetWindowDisplayAffinity", "int", hWnd, "int*", out)
    return out
}

SetWindowDisplayAffinity(hWnd, dwAffinity:=0) {
    static pGlobal, mcode:="2,x86:g+wci0QkIItQCIlUJATB+h+JVCQIi1AEiRQk/xCDxBzD,x64:TItBCEiLURBIiwFMicFI/+A=", e := {1:4, 2:1}, c := (A_PtrSize=8) ? "x64" : "x86"
    , hFunc := DllCall("GetProcAddress", "ptr", DllCall("GetModuleHandle", "str", "user32.dll", "ptr"), "astr", "SetWindowDisplayAffinity", "ptr")
    /*
        #include <windows.h>
        BOOL SetAffinity(void* args[]) {
            BOOL(*func)(HWND, long long) = (BOOL(*)(HWND, long long))args[0];
            return func((HWND)args[1], (long long)args[2]);
        }
    */
    ; Load MCode into local memory 
    if (!pGlobal) {
        if (!regexmatch(mcode, "^([0-9]+),(" c ":|.*?," c ":)([^,]+)", m))
            return
        if (!DllCall("crypt32\CryptStringToBinary", "str", m3, "uint", 0, "uint", e[m1], "ptr", 0, "uint*", s, "ptr", 0, "ptr", 0))
            return
        pGlobal := DllCall("GlobalAlloc", "uint", 0, "ptr", s, "ptr")
        if (c="x64")
            DllCall("VirtualProtect", "ptr", pGlobal, "ptr", s, "uint", 0x40, "uint*", op)
        if (!DllCall("crypt32\CryptStringToBinary", "str", m3, "uint", 0, "uint", e[m1], "ptr", pGlobal, "uint*", s, "ptr", 0, "ptr", 0))
            DllCall("GlobalFree", "ptr", pGlobal)
    }

    ; Write MCode function pointer, window hWnd, and dwAffinity parameters into a buffer to be copied over
    WinGet, pid, PID, ahk_id %hWnd%
    VarSetCapacity(buffer, A_PtrSize*3)
    NumPut(hFunc, buffer, 0, "ptr")
    NumPut(hWnd, buffer, A_PtrSize, "ptr")
    NumPut(dwAffinity, buffer, A_PtrSize*2, "ptr")

    ; Get process handle for process
    hProc := DllCall("OpenProcess", "int", 0x000F0000 | 0x00100000 | 0xFFFF, "int", 0, "int", pid) ; PROCESS_ALL_ACCESS
    ; Allocate memory for the process, and then copy the parameters from buffer
    remotebuffer := DllCall("VirtualAllocEx", "ptr", hProc, "ptr", 0, "int", A_PtrSize*3, "int", 0x00001000 | 0x00002000, "int", 0x04, "ptr")
    DllCall("WriteProcessMemory", "ptr", hProc, "ptr", remotebuffer, "ptr", &buffer, "int", A_PtrSize*3, "int*", written)

    ; Write the MCode function in the process memory
    pRemoteFunc := DllCall("VirtualAllocEx", "ptr", hProc, "ptr", 0, "int", s, "int", 0x00001000 | 0x00002000, "int", 0x40, "ptr")
    DllCall("WriteProcessMemory", "ptr", hProc, "ptr", pRemoteFunc, "ptr", pGlobal, "int", s, "int*", written)
    
    ; Execute new thread using MCode function and allocated parameters
    hThread := DllCall("CreateRemoteThread", "ptr", hProc, "uint", 0, "uint", 0, "ptr", pRemoteFunc, "ptr", remotebuffer, "uint", 0, "uint", 0, "ptr")

    DllCall("WaitForSingleObject", "ptr", hThread, "UInt", 0xFFFFFFFF)
    DllCall("VirtualFree", "ptr", pRemoteFunc, "int", 0, "int", 0x00008000) ; MEM_DECOMMIT
    DllCall("VirtualFree", "ptr", remotebuffer, "int", 0, "int", 0x00008000)
    DllCall("CloseHandle", "ptr", hThread)
    DllCall("CloseHandle", "ptr", hProc)
}

[Leox23]

Hi guys, is it possible to take a new function for find text that allows searching within an image file instead of the screen?

[feiyue]

@Leox23 You can try this:

Code: Select all

pic1:="d:\big.bmp"
pic2:="d:\small.jpg"

if FindTextInPic(rx, ry, pic1,  0, 0, Text:="##10$" . pic2)
  MsgBox % "Found small pic in big pic : " rx ", " ry
else
  MsgBox % "Can't Found small pic in big pic !"
return

FindTextInPic(ByRef rx, ByRef ry, pic, err1, err0, Text, args*)
{
  FindText().ShowPic( pic, show:=0, x, y, w, h ), args[1]:=0
  if (ok:=FindText( rx, ry, x, y, x+w-1, y+h-1, err1, err0, Text, args*))
  {
    For i,v in ok
      v[1]-=x, v[2]-=y, v.x-=x, v.y-=y
    rx:=ok[1][1], ry:=ok[1][2]
  }
  return ok
}

[pirik33]

Will this work on v2?

[ogib]

Will it continue to be updated to v1?

[Galakrond]

I have a few questions for its use,

I might have misunderstood. If so, I'm sorry.

For example, I will search for an image or pixel at the coordinates 200,200,200,200, and when it is found, I want the 1 key to be pressed.

I have 10 different images that I want to be searched at coordinates 200,200,200,200. Can I specify these images in the script and press the keys I want quickly when the pictures appear at the specified coordinate?

Thanks for any help fei

[feiyue]

@Galakrond I think you can do it like this:

Code: Select all

Text:=""
Text.="|<pic1>..."     ; Text.="|<color1>##10 $ 0/0/RRGGBB"  ; Search for the first color
Text.="|<pic2>..."     ; Text.="|<color2>##10 $ 0/0/RRGGBB"  ; Search for the second color
Text.="|<pic3>..."     ; Text.="|<color3>##10 $ 0/0/RRGGBB"  ; Search for the third color

Loop {
  if (ok:=FindText(x,y, 200,200,300,300, 0.1,0.1,Text,, FindAll:=0))
  {
    id:=ok[1].id
    if (id="pic1")
      Send a
    else if (id="pic2")
      Send b
    else if (id="pic3")
      Send c
    Sleep 200
  }
}

; Or use arrays to simplify code

Key:={ pic1:"a", pic2:"b", pic3:"c" }

Loop {
  if (ok:=FindText(x,y, 200,200,300,300, 0.1,0.1,Text,, FindAll:=0))
  {
    Send % Key[ ok[1].id ]
    Sleep 200
  }
}

[garbanian]

Ive been struggling with tryng to get variable inputs to work for the range that would be loaded in through iniread. If anyone would be able to give ANY direction or hint or any help I'd ,much appreciate it

[chaoscreater]

@chaoscreater, setting display affinity to 0 for Authy Desktop seems to fix that problem. I'm not sure whether it's necessary to call SetWindowDisplayAffinity before every FindText call or just periodically, but the following code seems to fix it. Unfortunately it's an incomplete solution, because I only got it working with Authy Desktop, and calling it on other programs will most likely crash the program. Perhaps somebody else can figure out the cause (I suspect it's because LoadLibrary isn't called from the external thread?).

Code: Select all

hWnd := WinExist("Twilio")
OutputDebug, % GetWindowDisplayAffinity(hWnd) "`n"
SetWindowDisplayAffinity(hWnd, 0)
OutputDebug, % GetWindowDisplayAffinity(hWnd) "`n"

GetWindowDisplayAffinity(hWnd) {
    DllCall("GetWindowDisplayAffinity", "int", hWnd, "int*", out)
    return out
}

SetWindowDisplayAffinity(hWnd, dwAffinity:=0) {
    static pGlobal, mcode:="2,x86:g+wci0QkIItQCIlUJATB+h+JVCQIi1AEiRQk/xCDxBzD,x64:TItBCEiLURBIiwFMicFI/+A=", e := {1:4, 2:1}, c := (A_PtrSize=8) ? "x64" : "x86"
    , hFunc := DllCall("GetProcAddress", "ptr", DllCall("GetModuleHandle", "str", "user32.dll", "ptr"), "astr", "SetWindowDisplayAffinity", "ptr")
    /*
        #include <windows.h>
        BOOL SetAffinity(void* args[]) {
            BOOL(*func)(HWND, long long) = (BOOL(*)(HWND, long long))args[0];
            return func((HWND)args[1], (long long)args[2]);
        }
    */
    ; Load MCode into local memory 
    if (!pGlobal) {
        if (!regexmatch(mcode, "^([0-9]+),(" c ":|.*?," c ":)([^,]+)", m))
            return
        if (!DllCall("crypt32\CryptStringToBinary", "str", m3, "uint", 0, "uint", e[m1], "ptr", 0, "uint*", s, "ptr", 0, "ptr", 0))
            return
        pGlobal := DllCall("GlobalAlloc", "uint", 0, "ptr", s, "ptr")
        if (c="x64")
            DllCall("VirtualProtect", "ptr", pGlobal, "ptr", s, "uint", 0x40, "uint*", op)
        if (!DllCall("crypt32\CryptStringToBinary", "str", m3, "uint", 0, "uint", e[m1], "ptr", pGlobal, "uint*", s, "ptr", 0, "ptr", 0))
            DllCall("GlobalFree", "ptr", pGlobal)
    }

    ; Write MCode function pointer, window hWnd, and dwAffinity parameters into a buffer to be copied over
    WinGet, pid, PID, ahk_id %hWnd%
    VarSetCapacity(buffer, A_PtrSize*3)
    NumPut(hFunc, buffer, 0, "ptr")
    NumPut(hWnd, buffer, A_PtrSize, "ptr")
    NumPut(dwAffinity, buffer, A_PtrSize*2, "ptr")

    ; Get process handle for process
    hProc := DllCall("OpenProcess", "int", 0x000F0000 | 0x00100000 | 0xFFFF, "int", 0, "int", pid) ; PROCESS_ALL_ACCESS
    ; Allocate memory for the process, and then copy the parameters from buffer
    remotebuffer := DllCall("VirtualAllocEx", "ptr", hProc, "ptr", 0, "int", A_PtrSize*3, "int", 0x00001000 | 0x00002000, "int", 0x04, "ptr")
    DllCall("WriteProcessMemory", "ptr", hProc, "ptr", remotebuffer, "ptr", &buffer, "int", A_PtrSize*3, "int*", written)

    ; Write the MCode function in the process memory
    pRemoteFunc := DllCall("VirtualAllocEx", "ptr", hProc, "ptr", 0, "int", s, "int", 0x00001000 | 0x00002000, "int", 0x40, "ptr")
    DllCall("WriteProcessMemory", "ptr", hProc, "ptr", pRemoteFunc, "ptr", pGlobal, "int", s, "int*", written)
    
    ; Execute new thread using MCode function and allocated parameters
    hThread := DllCall("CreateRemoteThread", "ptr", hProc, "uint", 0, "uint", 0, "ptr", pRemoteFunc, "ptr", remotebuffer, "uint", 0, "uint", 0, "ptr")

    DllCall("WaitForSingleObject", "ptr", hThread, "UInt", 0xFFFFFFFF)
    DllCall("VirtualFree", "ptr", pRemoteFunc, "int", 0, "int", 0x00008000) ; MEM_DECOMMIT
    DllCall("VirtualFree", "ptr", remotebuffer, "int", 0, "int", 0x00008000)
    DllCall("CloseHandle", "ptr", hThread)
    DllCall("CloseHandle", "ptr", hProc)
}

thank you, that worked!

[terryys]

Is there a way to search an image file or a pointer to bitmap instead of the screen?
Also, how can I convert many image files to text form without using the gui?

[KennethQuicy]

DEear T.Magenheimer

I have also faced similar problem 3rd time or 4th.
I waited for caluclation finished for about 10-20 minutes, however the calculation was not stopping or seems not to process, or stuck.

Finally I restarted PC.

[feiyue]

Updated to 9.1 version - 2023/07/30
1. Modify: Omitting the first two parameters is no longer allowed (improvements made in v8.9 have been removed).
Because considering compatibility with code before v8.5 would add a lot of complexity.
The first and second parameters can be default values, as long as the subsequent parameters are not all empty.
eg: ok:=FindText(,,,,,,,, Text)
2. Modify: Improved the speed of ImageSearch() in finding large image files.
ImageSearch()、ShowPic() can now use image handles. eg: HBITMAP:*%handle%
3. Add: Added PixelSearch(), PixelCount(), ControlClick(), GetRange() function.