FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools
feiyue
Posts: 237
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

16 Jul 2020, 22:16

c7aesa7r wrote:
16 Jul 2020, 05:54
do someone know if is possible to search inside a pbitmap instead searching for the text on screen?

i tried edit here, but did not work :cry:

Code: Select all

    if (id) and !(w<1 or h<1)
    {
      ;hDC2:=DllCall("GetDCEx", Ptr,id, Ptr,0, "int",3, Ptr)
      ;hDC2	    := Gdip_BitmapFromHWND(id)
      hDC2 := Gdip_CreateBitmapFromFile("C:\Users\User\Documents\x.png") ;
      
      DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
        , Ptr,hDC2, "int",x-wx, "int",y-wy, "uint",0x00CC0020|0x40000000)
      DllCall("ReleaseDC", Ptr,id, Ptr,hDC2)
    }
I think you can replace screen data with image files like this, but I haven't tested:

Code: Select all

/*
  if (hBM) and !(w<1 or h<1)
  {
    win:=DllCall("GetDesktopWindow", Ptr)
    hDC:=DllCall("GetWindowDC", Ptr,win, Ptr)
    mDC:=DllCall("CreateCompatibleDC", Ptr,hDC, Ptr)
    oBM:=DllCall("SelectObject", Ptr,mDC, Ptr,hBM, Ptr)
    DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
      , Ptr,hDC, "int",x, "int",y, "uint",0x00CC0020|0x40000000)
    DllCall("ReleaseDC", Ptr,win, Ptr,hDC)
    if (id:=this.BindWindow(0,0,1))
      WinGet, id, ID, ahk_id %id%
    if (id)
    {
      WinGetPos, wx, wy, ww, wh, ahk_id %id%
      left:=x, right:=x+w-1, up:=y, down:=y+h-1
      left:=left<wx ? wx:left, right:=right>wx+ww-1 ? wx+ww-1:right
      up:=up<wy ? wy:up, down:=down>wy+wh-1 ? wy+wh-1:down
      x:=left, y:=up, w:=right-left+1, h:=down-up+1
    }
    if (id) and !(w<1 or h<1)
    {
      hDC2:=DllCall("GetDCEx", Ptr,id, Ptr,0, "int",3, Ptr)
      DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
        , Ptr,hDC2, "int",x-wx, "int",y-wy, "uint",0x00CC0020|0x40000000)
      DllCall("ReleaseDC", Ptr,id, Ptr,hDC2)
    }
    DllCall("SelectObject", Ptr,mDC, Ptr,oBM)
    DllCall("DeleteDC", Ptr,mDC)
  }
*/
;-----------------
  if FileExist(file:="C:\Users\User\Documents\x.png")
  {
    pBitmap := Gdip_CreateBitmapFromFile(file)
    w:=Gdip_GetImageWidth(pBitmap), h:=Gdip_GetImageHeight(pBitmap)
    w:=w>zw ? zw:w, h:=h>zh ? zh:h
    hBM2:=Gdip_CreateHBITMAPFromBitmap(pBitmap), Gdip_DisposeImage(pBitmap)
    mDC:=DllCall("CreateCompatibleDC", Ptr,0, Ptr)
    oBM:=DllCall("SelectObject", Ptr,mDC, Ptr,hBM, Ptr)
    mDC2:=DllCall("CreateCompatibleDC", Ptr,0, Ptr)
    oBM2:=DllCall("SelectObject", Ptr,mDC2, Ptr,hBM2, Ptr)
    DllCall("BitBlt",Ptr,mDC,"int",0,"int",0,"int",w,"int",h
      , Ptr,mDC2, "int",0, "int",0, "uint",0x00CC0020|0x40000000)
    DllCall("SelectObject", Ptr,mDC2, Ptr,oBM2)
    DllCall("DeleteDC", Ptr,mDC2)
    DllCall("SelectObject", Ptr,mDC, Ptr,oBM)
    DllCall("DeleteDC", Ptr,mDC)
    DllCall("DeleteObject", Ptr,hBM2)
  }
;-----------------
wysnxzm
Posts: 2
Joined: 16 Jul 2020, 22:51

Re: FindText - Capture screen image into text and then find it

16 Jul 2020, 22:57

This is really cool, a meaningful thing, four years, praise you
c7aesa7r
Posts: 62
Joined: 02 Jun 2016, 21:09

Re: FindText - Capture screen image into text and then find it

16 Jul 2020, 23:11

ty @feiyue it worked

also, where does the actual code store the value of the image being printed?
Last edited by c7aesa7r on 17 Jul 2020, 01:41, edited 1 time in total.
wysnxzm
Posts: 2
Joined: 16 Jul 2020, 22:51

Re: FindText - Capture screen image into text and then find it

16 Jul 2020, 23:30

有看到你在CSDN发的教程,但是很久没更新
建议放上github,让更多的人知道
srooo
Posts: 1
Joined: 05 Jun 2019, 14:50

Re: FindText - Capture screen image into text and then find it

22 Jul 2020, 00:57

srooo wrote:
21 Jul 2020, 15:05
How to use FindText_BindWindow to find the color of x, y in the full-screen temporary image of the background
If it can be combined with getcolor(), from this arcticir article
https://www.autohotkey.com/boards/viewtopic.php?f=6&t=7451
I think I should solve it, thank you @feiyue for creating this program

Code: Select all

    WinGet, OutputVar, ID, A
    FindText_BindWindow(OutputVar)
    FindText_ScreenShot(0,0,1920,1080)
    color := FindText_ScreenShot_GetColor(1615,562)
    msgbox % "color=" . color
ras
Posts: 13
Joined: 25 Dec 2016, 11:52
Facebook: rastinder

Re: FindText - Capture screen image into text and then find it

23 Jul 2020, 09:39

how to make a database to search???
i mean i will give specific xy coordinates
if found from database
write it down
else
auto generate string from that coordinates and add it to database with comment "unidentified"

or just tell me the command which will auto generate text string
thanks
feiyue
Posts: 237
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

23 Jul 2020, 22:40

You can use FindText_GetTextFromScreen(x1, y1, x2, y2)
songdg
Posts: 171
Joined: 04 Oct 2017, 20:04

Re: FindText - Capture screen image into text and then find it

29 Jul 2020, 04:11

Can this useful tool be adapted to 4K display on 200% scale.
arimania
Posts: 3
Joined: 26 Sep 2019, 06:16

Re: FindText - Capture screen image into text and then find it

07 Aug 2020, 11:34

Hi Feiyue

Great, that You are doing a very good work at this findtext. :bravo:

I try to detect or wait change in some area first i try RegionWaitChange.ahk but not works.

the error message:

Code: Select all

RegionWaitChange.ahk (63) : ==> Call to nonexistent function.
     Specifically: Hash_GetCode(bits, sz)
Maybe some library are missing since i can't get fully the zip file from https ahknet.autohotkey.com /~Titan/dl/RegionWaitChange.zip Broken Link for safety (broken link) so i find another way.

Then from found your post, i try use your code from your answer to post https://www.autohotkey.com/boards/viewtopic.php?t=17834&start=138 or 137, but also got error. but as i remember i ever run well this code. But now it fail, but not sure what's happen. :facepalm:

the error message:

Code: Select all

Check_the_regionchange.ahk (22) : ==> Call to nonexistent function.
     Specifically: xywh2xywh(x,y,w,h, x,y,w,h)
I don't know what's wrong with this, hopefully you or someone here could help. :)

system:
The autohotkey 1.1.33.01 64bit (just run it today, formerly used 32bit, change to 64 in order to fix the error but no luck, still not working)
Windows 10 Pro v 1909
list of lib

Code: Select all

Directory of d:\aarun2\AutohotkeyArea\AutoHotkey_1.1.33.01\Lib
08/15/2018  07:13 AM             4,067 Acc.ahk
04/15/2018  12:05 AM            25,730 AccViewer.ahk
08/28/2012  06:56 PM             1,105 ADO.ahk
04/15/2018  12:05 AM             3,287 Anchor.ahk
08/28/2012  06:56 PM               563 ArchLogger.ahk
03/11/2003  06:47 AM            20,480 ASWHook.dll
08/28/2012  06:56 PM             2,586 Base.ahk
02/03/2020  03:17 PM            64,415 Class_SQLiteDB.ahk
08/28/2012  06:56 PM                 9 clean.bat
08/28/2012  06:56 PM             1,920 Collection.ahk
08/21/2015  12:55 AM            21,155 com.ahk
08/28/2012  06:56 PM             5,948 DataBaseAbstract.ahk
08/28/2012  06:56 PM             3,721 DataBaseADO.ahk
08/28/2012  06:56 PM             1,168 DataBaseFactory.ahk
08/28/2012  06:56 PM             6,232 DataBaseMySQL.ahk
08/28/2012  06:56 PM             9,321 DataBaseSQLLite.ahk
08/28/2012  06:56 PM               651 DBA.ahk
12/03/2017  01:39 PM             1,481 Excel_Get.ahk
12/03/2017  01:39 PM             1,925 Find.ahk
11/18/2016  01:28 AM            88,789 Gdip.ahk
11/18/2016  01:28 AM         1,748,992 gdiplus.dll
09/18/2018  01:04 PM            97,743 Gdip_All.ahk
08/24/2018  11:16 AM             6,048 iMacros.ahk
08/24/2018  11:11 AM             6,048 iMacros.ahk.bak
04/15/2018  12:07 AM            23,802 iWB2 Learner UPDATED.ahk
04/05/2018  08:30 AM            11,733 JSON.ahk
08/28/2012  06:56 PM         3,987,456 libmysql.dll
08/28/2012  06:56 PM             4,510 MemoryBuffer.ahk
11/29/2011  02:45 PM             5,176 mysql-demo.ahk
11/29/2011  02:54 PM             3,088 mysql-queries.ahk
08/28/2012  06:56 PM            11,232 mysql.ahk
08/28/2012  06:56 PM             1,952 RecordSetADO.ahk
08/28/2012  06:56 PM             1,929 RecordSetMySQL.ahk
08/28/2012  06:56 PM             3,439 RecordSetSqlLite.ahk
09/18/2018  12:59 PM             9,319 ScreenCapture.ahk
08/28/2012  06:56 PM             4,263 sqlite3.def
01/28/2020  03:29 AM           944,840 sqlite3.dll
08/28/2012  06:56 PM            42,226 SQLite_L.ahk
04/17/2018  03:45 PM             4,442 Tesseract.ahk
08/10/2018  02:45 PM            47,438 TF.ahk
08/29/2018  11:20 PM             2,729 ToolTipOpt.ahk
04/05/2018  08:30 AM           124,013 Vis2.ahk
              42 File(s)      7,356,971 bytes
this i repost the code:

Code: Select all

;it looks like the coordinat been using is Relative 
xcoord := 265
ycoord := 71
width := 160
height := 72
Loop {
  pic1:=GetPicArr(xcoord, ycoord, width, height)
  Sleep, 1000
  pic2:=GetPicArr(xcoord, ycoord, width, height)
  if ComparePicArr(pic1, pic2, n:=0.3)
  {
    MsgBox, The screen image changed !
    Send z
  }
}

;==============This is the function need for use in above function======
  ; Get the RGB value of all the points of the picture to the array

GetPicArr(x, y, w, h)
{
  xywh2xywh(x,y,w,h, x,y,w,h)
  if (w<1 or h<1)
    return, 0
  ListLines, % "Off" (lls:=A_ListLines=0?"Off":"On")/0
  SetBatchLines, % "-1" (bch:=A_BatchLines)/0
  GetBitsFromScreen(x,y,w,h,Scan0,Stride,bits)
  arr:=[], i:=-4
  Loop, % w*h
    arr.Push(NumGet(bits,i+=4,"uint")&0xFFFFFF)
  SetBatchLines, %bch%
  ListLines, %lls%
  return, arr
}
(sorry for my bad english)
feiyue
Posts: 237
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

07 Aug 2020, 21:05

feiyue wrote:
10 Apr 2020, 12:28
Updated to 7.5 version - 2020/04/13
.... 2. Modify: in order to reduce the conflict of function names,
....... Use "FindText_" as the function prefix. However, this creates some incompatibility.
....... For example: MouseTip() must be changed to FindText_MouseTip()

Code: Select all

GetPicArr(x, y, w, h)
{
  ListLines, % (lls:=A_ListLines=0?"Off":"On") ? "Off":"Off"
  SetBatchLines, % (bch:=A_BatchLines) ? "-1":"-1"
  FindText_ScreenShot(x, y, x+w-1, y+h-1)
  arr:=[], i:=1
  Loop, % h {
    yy:=y+A_Index-1
    Loop, % w {
      xx:=x+A_Index-1
      , arr[i++]:=FindText_ScreenShot_GetColor(xx, yy)
    }
  }
  SetBatchLines, %bch%
  ListLines, %lls%
  return arr
}
arimania
Posts: 3
Joined: 26 Sep 2019, 06:16

Re: FindText - Capture screen image into text and then find it

09 Aug 2020, 17:15

Thanks Feiyue the your code now works,

But now, i want to try your code but it cannot get my code work to detect changing, it always get Tooltip status: Wait the screen image still the same.
Image
[spoiler3]https 1.bp.blogspot.com /-PXY996U8SJE/XzBzb7xlYDI/AAAAAAAAHiQ/oyOxyqPx1GMzG4HyZQhdT32LtSiINVv-wCLcBGAsYHQ/s0/findtext%2Bobject03.png[/spoiler3]



I still do not know how to make it in one ahk script to test it.

1) send text to find what inputbox area {that the area i want to detect the changing

Code: Select all

#SingleInstance Force
wbk := ComObjGet("e:\web duplikat\fixing kerjaan ariel pada webbaru\link error\temp\replace_data.ods")

^1::
if WinExist("ahk_class Notepad++") or WinExist("ahk_class" . ClassName)
    WinActivate  ; Uses the last found window.

if WinExist("ahk_class #32770") or WinExist("ahk_class" . ClassName)
    WinActivate  ; Uses the last found window.

loop {

;this click is to make sure the window of find and replace (Ctrl+H) of notepad plus is activate not the "writing area of notepadplus" , because it usually go back activating the parent instead of the child window(find replace).
Text:="|<klik_biar_diem_aje>*139$71.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000D0040000E000F0080yU0U000WCSHV00100014WWUW2wS0003lx5D7p540004W2+X"
if (ok:=FindText(607, 222, 150000, 150000, 0, 0, Text))
  {
    CoordMode, Mouse
    X:=ok.1.1, Y:=ok.1.2, W:=ok.1.3, H:=ok.1.4, Comment:=ok.1.5, X+=W//2, Y+=H//2
    Y+= %Y% - 5
    MouseMove, %X%, %Y%
	Click, %X%, %Y%
	Break
	}
	if (!ok) {
		send, ^h
		sleep, 500
 Return
}
}


;sending the inputbox long and many text to make sure it always changeing/not stopping with label name "Find What"
;the detecting changing to make sure the sending text all done, before ask the script to run another process
Loop, 20 {
; Gets the value of cell 1, 1 (also known as cell A1) from Sheet1
A1Val := wbk.Sheets("Sheet1").Cells(A_Index, 1).Value
A2Val := wbk.Sheets("Sheet1").Cells(A_Index, 2).Value
SendInput, % A1Val
Sleep, 1000
SendInput, % A2Val
Sleep, 1000
}

^2::Pause
^3::Exit

;===== Copy The Following Functions To Your Own Code Just once using v5.9 still works=====

2) the second script is to check detect the screen is changing.

Code: Select all

#SingleInstance Force
#Persistent
CoordMode, Mouse

if WinExist("ahk_class Notepad++") or WinExist("ahk_class" . ClassName)
    WinActivate  ; Uses the last found window.

if WinExist("ahk_class #32770") or WinExist("ahk_class" . ClassName)
    WinActivate  ; Uses the last found window.

loop {
Text:="|<klik_biar_diem_aje>*139$71.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000D0040000E000F0080yU0U000WCSHV00100014WWUW2wS0003lx5D7p540004W2+X"
if (ok:=FindText(476-150000, 391-150000, 476+150000, 391+150000, 0, 0, Text))
  {
    CoordMode, Mouse
    X:=ok.1.1, Y:=ok.1.2, W:=ok.1.3, H:=ok.1.4, Comment:=ok.1.5, X+=W//2, Y+=H//2
    Y+= %Y% - 5
    MouseMove, %X%, %Y%,
	Click, %X%, %Y%
	ToolTip, found, 300, 5
	Sleep, 1000
    Tooltip
	Break
	}
	if (!ok) {
		send, ^h
		sleep, 500
 Return
}
}

;This is the startting point of area to for checking (top left)
Text:="|<titik_mulai_A>*147$71.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001zzzzzy00000200000000000400000000000800000000000E00000020000U000000402010000000804020000019SCS44000002GW2E88000005J4wU0E000002W+900U00000AYIG11000001"
 if (ok:=FindText(553-150000, 413-150000, 553+150000, 413+150000, 0, 0, Text))
 {
   CoordMode, Mouse
   X1:=ok.1.x, Y1:=ok.1.y, Comment:=ok.1.id
   ; Click, %X%, %Y%
 }

;This is the ending point of area to for checking (bottom right)
 Text:="|<Titik_selesai_A>*164$71.000000000000000000000000000000000007z00000000000203zzzzzzzzU407zzzzzzzz080A000000000E0M000000000U0k00000000101U000000002030000000004060000001x480A00000020EE0M000000450U0k000000Dg101U000000EE2030000000UU40600000011080A000000220E0M000000000U0k00000000101U0000000020300000000Tw07zzzzzzzz000Dzzzzzzzz"
 if (ok:=FindText(763-150000, 425-150000, 763+150000, 425+150000, 0, 0, Text))
 {
   CoordMode, Mouse
   X:=ok.1.x, Y:=ok.1.y, Comment:=ok.1.id
   X2 := X - 50
   Y2 := Y + 8
	}

xcoord := X1 + (X2 - X1)/2
ycoord := Y1 + (Y2 - Y1)/2
half_width := (X2 - X1)/2
half_height := (Y2 - Y1)/2

;msgbox, %xcoord%, %ycoord%
;mousemove, %xcoord%, %ycoord%, 50

;checking the center of area selection is on the right coordinate
FindText_MouseTip(xcoord, ycoord)


Loop {
  pic1:=GetPicArr(xcoord, ycoord, half_width, half_height)
  Sleep,1000
  pic2:=GetPicArr(xcoord, ycoord, half_width, half_height)
  if ComparePicArr(pic1, pic2, n:=0.3)
  {
    ToolTip, The screen area image still changing, 300, 5
    sleep, 800
    ToolTip
  } else {
	ToolTip, Wait the screen image still the same, 300, 5
    sleep, 800
    Tooltip
}
}
; Get the RGB value of all the points of the picture to the array

GetPicArr(x, y, w, h)
{
  ListLines, % (lls:=A_ListLines=0?"Off":"On") ? "Off":"Off"
  SetBatchLines, % (bch:=A_BatchLines) ? "-1":"-1"
  FindText_ScreenShot(x, y, x+w-1, y+h-1)
  arr:=[], i:=1
  Loop, % h {
    yy:=y+A_Index-1
    Loop, % w {
      xx:=x+A_Index-1
      , arr[i++]:=FindText_ScreenShot_GetColor(xx, yy)
    }
  }
  SetBatchLines, %bch%
  ListLines, %lls%
  return arr
}

; Check whether all change points in the picture exceed the percentage

ComparePicArr(arr1, arr2, n:=0.3)
{
  diff:=0, diffMax:=Round(arr1.MaxIndex()*n)
  For k,v in arr1
    diff+=(arr2[k]!=v)
  if (diff>diffMax)
    return, 1
  return, 0
}


^2::Pause
^3::Exit
The center coordinate of the detecting changing area is get from function
feiyue
Posts: 237
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

09 Aug 2020, 22:37

I think the default comparison threshold is too large. You can set it smaller.
I recommend that the range used to detect screen changes is as small as possible.

Code: Select all

#Include <FindText>

WinGetPos, x, y, w, h, A
pic1:=GetPicArr(x, y+50, w, h-50)
Loop {
  Sleep, 500
  WinGetPos, x, y, w, h, A
  pic2:=GetPicArr(x, y+50, w, h-50)
  if ComparePicArr(pic1, pic2, n:=0.001)
  {
    Tip("Change")
  }
  else
  {
    Tip("Not Change")
  }
  pic1:=pic2
}

; Get the RGB value of all the points of the picture to the array

GetPicArr(x, y, w, h)
{
  ListLines, % (lls:=A_ListLines=0?"Off":"On") ? "Off":"Off"
  SetBatchLines, % (bch:=A_BatchLines) ? "-1":"-1"
  FindText_ScreenShot(x, y, x+w-1, y+h-1)
  arr:=[], i:=1
  Loop, % h {
    yy:=y+A_Index-1
    Loop, % w {
      xx:=x+A_Index-1
      , arr[i++]:=FindText_ScreenShot_GetColor(xx, yy)
    }
  }
  SetBatchLines, %bch%
  ListLines, %lls%
  return arr
}

; Check whether all change points in the picture exceed the percentage

ComparePicArr(arr1, arr2, n:=0.3)
{
  diff:=0, diffMax:=Round(arr1.MaxIndex()*n)
  For k,v in arr1
    if (diff+=(arr2[k]!=v))>diffMax
      return, 1
  return, 0
}

Tip(s:="") {
  SetTimer, %A_ThisFunc%, % s="" ? "Off" : -2000
  ToolTip, %s%, 0, 0
}

arimania
Posts: 3
Joined: 26 Sep 2019, 06:16

Re: FindText - Capture screen image into text and then find it

11 Aug 2020, 02:53

Wow Great feiyue your code works like a charm :bravo: :clap:

Note: my comment is just to make me understand, hope it not wrongly understood of Feiyue code.

Now then i try to modify your tips to show the area for visual checking before compare

Code: Select all

WinGetPos, x, y, w, h, A
xcoord := x
ycoord := y
width := Round(w*0.5)
height := Round(h*0.2)
MsgBox, %xcoord%,%ycoord% w = %width%, h = %height%
FindText_ShowArea(xcoord, ycoord, width, height)

; Prompt Area for assuring the area with visual, just incase need to resize to more specific area. 
FindText_ShowArea(x:="", y:="", Active_width:="", Active_height:="")
{
  local
  if (x="")
  {
    VarSetCapacity(pt,16,0), DllCall("GetCursorPos","ptr",&pt)
    x:=NumGet(pt,0,"uint"), y:=NumGet(pt,4,"uint")
  }
  x:=Round(x), y:=Round(y), w:=Round(Active_width)  h:=Round(Active_height)
  ;-------------------------
  Gui, _ShowArea_: +AlwaysOnTop -Caption +ToolWindow +Hwndmyid +E0x08000000
  Gui, _ShowArea_: Show, Hide w%w% h%h%
  ;-------------------------
  dhw:=A_DetectHiddenWindows
  DetectHiddenWindows, On
  d:=4, i:=w-d, j:=h-d
  s=0-0 %w%-0 %w%-%h% 0-%h% 0-0
  s=%s%  %d%-%d% %i%-%d% %i%-%j% %d%-%j% %d%-%d%
  WinSet, Region, %s%, ahk_id %myid%
  DetectHiddenWindows, %dhw%
  ;-------------------------
  Gui, _ShowArea_: Show, NA x%x% y%y%
  Loop, 4
  {
    Gui, _ShowArea_: Color, % A_Index & 1 ? "Red" : "Blue"
    Sleep, 500
  }
  Gui, _ShowArea_: Destroy
}

This is the full code i use

Code: Select all

#Include <FindText>
#SingleInstance Force
#Persistent
CoordMode, Mouse

Run, notepad ; open notepad
WinWait, Untitled - Notepad ; wait for window to exist

ControlSetText, Edit1 ; delete any current text
MsgBox, Notepad is empty`, press F12 to fill it with random text.

; monitor the area of active window, 50% of width and 20% of height, from and offset the coordinate (5, 10) :
; get top_left of last active windows
WinGetPos, x, y, w, h, A
xcoord := x + 5
ycoord := y + 10
width := Round(w*0.5)
height := Round(h*0.2)
MsgBox, %xcoord%,%ycoord% w = %width%, h = %height%
FindText_ShowArea(xcoord, ycoord, width, height)

pic1:=GetPicArr(xcoord, ycoord, width, height)
Loop {
  Sleep, 500
  WinGetPos, x, y, w, h, A
  pic2:=GetPicArr(xcoord, ycoord, width, height)
  ; the default comparison threshold if too large, you can change to n:=0.001
  if ComparePicArr(pic1, pic2, n:=0.01)
  {
    ToolTip, "Change" . %A_index%
    Sleep, 500
    MsgBox, "the last active windows already change"
    Break
  }
  else
  {
    ToolTip, "Not Change" . %A_index%
  }
  pic1:=pic2
}

WinClose
ExitApp

F12::
VarSetCapacity(dots, 5000, Asc(".")) ; fill a variable with something
ControlSetText, Edit1, %dots%, Untitled - Notepad ; change the text on notepad artificially
dots = ; free variable
Return


; Get the RGB value of all the points of the picture to the array

GetPicArr(x, y, w, h)
{
  ListLines, % (lls:=A_ListLines=0?"Off":"On") ? "Off":"Off"
  SetBatchLines, % (bch:=A_BatchLines) ? "-1":"-1"
  FindText_ScreenShot(x, y, x+w-1, y+h-1)
  arr:=[], i:=1
  Loop, % h {
    yy:=y+A_Index-1
    Loop, % w {
      xx:=x+A_Index-1
      , arr[i++]:=FindText_ScreenShot_GetColor(xx, yy)
    }
  }
  SetBatchLines, %bch%
  ListLines, %lls%
  return arr
}

; Check whether all change points in the picture exceed the percentage
ComparePicArr(arr1, arr2, n:=0.3)
{
  diff:=0, diffMax:=Round(arr1.MaxIndex()*n)
  For k,v in arr1
    diff+=(arr2[k]!=v)
  if (diff>diffMax)
    return, 1
  return, 0
}

RemoveToolTip:
ToolTip
return

^2::Pause
^3::Exit

; Prompt Area in remote assistance
FindText_ShowArea(x:="", y:="", Active_width:="", Active_height:="")
{
  local
  if (x="")
  {
    VarSetCapacity(pt,16,0), DllCall("GetCursorPos","ptr",&pt)
    x:=NumGet(pt,0,"uint"), y:=NumGet(pt,4,"uint")
  }
  x:=Round(x), y:=Round(y), w:=Round(Active_width)  h:=Round(Active_height)
  ;-------------------------
  Gui, _ShowArea_: +AlwaysOnTop -Caption +ToolWindow +Hwndmyid +E0x08000000
  Gui, _ShowArea_: Show, Hide w%w% h%h%
  ;-------------------------
  dhw:=A_DetectHiddenWindows
  DetectHiddenWindows, On
  d:=4, i:=w-d, j:=h-d
  s=0-0 %w%-0 %w%-%h% 0-%h% 0-0
  s=%s%  %d%-%d% %i%-%d% %i%-%j% %d%-%j% %d%-%d%
  WinSet, Region, %s%, ahk_id %myid%
  DetectHiddenWindows, %dhw%
  ;-------------------------
  Gui, _ShowArea_: Show, NA x%x% y%y%
  Loop, 4
  {
    Gui, _ShowArea_: Color, % A_Index & 1 ? "Red" : "Blue"
    Sleep, 1000
  }
  Gui, _ShowArea_: Destroy
}

The Result, the area created for height is right but the but the problem Width is wrong, showing very long red paralel line from the left and to the right, even i already change the width to 50 percent or 10 percent of width active window.
is use secondary monitor as primary, is it matter??. after trying change the monitor to extended monitor, or primary only at laptop, the problem still the same.

Can you fixed and give the more comments to understand whats your code doing, if you don't mind.
or may be there is another better code is welcome.

Thanks
feiyue
Posts: 237
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

11 Aug 2020, 10:47

You just need to debug to find the problem.

Code: Select all

  x:=Round(x), y:=Round(y), w:=Round(Active_width)  h:=Round(Active_height)
  ; MsgBox, % w  ; Debug w --> w:=Round(Active_width), h:=Round(Active_height)
Netmano
Posts: 48
Joined: 17 Jun 2020, 16:24

Re: FindText - Capture screen image into text and then find it

16 Aug 2020, 11:37

First time using this library here, I am trying to see how it works for myself by creating my first example.
but am stuck, when I click on "Capture" Findtext is capturing 15 centermetres beneath where my cursor actually is.
I have not modified any of the code. I copied exerything as is and pasted into an empty .ahk file that I named FindText.ahk
Any idea what I may be doing wrong? Thanks!



N5mUoHAUwo.gif
N5mUoHAUwo.gif (1.68 MiB) Viewed 1271 times
feiyue
Posts: 237
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

17 Aug 2020, 05:50

Please try the latest version. Recently, I fixed a small bug, but there is no updated version.
This bug is about the virtual screen of the system, it may not start from (0,0), as in your case.. :beer:

Code: Select all

    DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
      , Ptr,hDC, "int",x, "int",y, "uint",0x00CC0020|0x40000000)
Change to -->

Code: Select all

    DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
      , Ptr,hDC, "int",x-zx, "int",y-zy, "uint",0x00CC0020|0x40000000)
Reniazas
Posts: 3
Joined: 29 Jul 2020, 05:01

Re: FindText - Capture screen image into text and then find it

19 Aug 2020, 22:40

feiyue wrote:
17 Aug 2020, 05:50
Please try the latest version. Recently, I fixed a small bug, but there is no updated version.
This bug is about the virtual screen of the system, it may not start from (0,0), as in your case.. :beer:

Code: Select all

    DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
      , Ptr,hDC, "int",x, "int",y, "uint",0x00CC0020|0x40000000)
Change to -->

Code: Select all

    DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
      , Ptr,hDC, "int",x-zx, "int",y-zy, "uint",0x00CC0020|0x40000000)
I have the similar issue
Capture on Monitor1 is fine
but Monitor2 have offest
monitor1:: 27-inch 4K(3480*2160)
monitor2:: 24-inch FHD(1920*1080)
Use Windows 10 Function(scale and layout) scale 4k monitor1 to 150% FHD montior2 is default 100%
Snip_2020-08-20_11-08-19.jpg
Snip_2020-08-20_11-08-19.jpg (5.88 KiB) Viewed 522 times

Code: Select all

  SysGet, Mon1, Monitor, 1
  SysGet, Mon2, Monitor, 2
  MsgBox, Monitor1`nLeft: %Mon1Left% -- Top: %Mon1Top% -- Right: %Mon1Right% -- Bottom %Mon1Bottom%.`nMonitor2`nLeft: %Mon2Left% -- Top: %Mon2Top% -- Right: %Mon2Right% -- Bottom %Mon2Bottom%.
Snip_2020-08-20_11-08-48.jpg
Snip_2020-08-20_11-08-48.jpg (19.11 KiB) Viewed 522 times

Code: Select all

  DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
  , Ptr,hDC, "int",x-zx, "int",y-zy, "uint",0x00CC0020|0x40000000)
x=-2880
y=0
zx=-2880
zy=0
Try to modify zx and zy but every position is different

By the way
I need use this code to move the same position on my daily use

Code: Select all

  CoordMode, Mouse , Screen
  MouseGetPos, Xpos, Ypos
  If (Xpos < 0)
    MouseMove, Xpos/(2880/1920), Ypos/(1620/1080)
  Else
    MouseMove, Xpos, Ypos
chaoscreater
Posts: 20
Joined: 12 Sep 2019, 21:15

Re: FindText - Capture screen image into text and then find it

24 Aug 2020, 23:13

Hi there,

I'm having this really strange issue and I can't figure this out. Hopefully, someone will be able to help me :)

I have 2 laptops and they both have the same resolution (1920 x 1080). They do not have any scaling enabled (scaling in Display settings is set to 100% for both).

The script below only works on one laptop, but not the other.

Code: Select all

		 t1:=A_TickCount, X:=Y:=""
		
		Text:="|<>*194$62.zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzDtzzzzzzzznyTzzzzzzzwzbzzzzzzzzDtzzzzzzzznyS1w7zzzzwzb0Q0zzzzzDtXz77zzzznyMzXlzzzzwzb7swTzzzzDtsS07zzzznyT3U1zzzzwz7wMzzzzzz7lzaDzzzzzssztlzzzzzy0D0Q0Tzzzzk7kDkDzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzU"
		
		 if (ok:=FindText(1125, 754, 1247, 818, 0, 0, Text))
		 {
		   CoordMode, Mouse
		   X:=ok.1.x, Y:=ok.1.y, Comment:=ok.1.id
		   Click, %X%, %Y%
		 }

		 for i,v in ok
		   if (i<=2)
		     FindText_MouseTip(ok[i].x, ok[i].y)	

If I use the FindText to capture the same image in the same location and compare the differences, I can see that they are slightly different.

My question is, why would the captured image be different? Both laptops have the same resolution, so the FindText capture should be identical for both machines?

My 2nd question is, is it possible to do a fuzzy search for the text?
chaoscreater
Posts: 20
Joined: 12 Sep 2019, 21:15

Re: FindText - Capture screen image into text and then find it

28 Aug 2020, 06:40

I'm running into a really weird issue...

i'm trying to check if icon A exist. If it does, I would then want to check whether icon B exists. Icon B doesn't always appear, but when it does, I don't want the script to do anything. If only icon A appears, then just click it. This doesn't work for some reason. The clicked location is not correct. It clicks outside of the window/program.

Code: Select all

   Text:="|<>*184$68.zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz0zzzzwzzzzzU3zzzzDzzzzssTzzzzzzzzyT7zzzzzzzzzbtwDjrCzT1ztyS0lsn7XUDyT76CSQttlnzU3nlbbCSQwTs1swMlnX6D7y8S06AwsnU1zb7U3nDDAs0ztsszwnnnCDzySDDz0ww3nzzblkXsTDVw0ztyS0y7nsTUDzzzsTnzzDy7zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzs"

            SpiralIcon:="|<>*150$28.zzk1zzz03zzs0DzzU0Tzy01zUk07s0U0T0101w0007k080T30U1s0207lU80S31U1w6A0DUTU0yUE0Ly002zk00HzU03Tz00/zy04zzy0DzzzzzzU"
            
             if (ok:=FindText(889, 675, 1059, 723, 0, 0, Text))
             {
                
                sleep, 500

                 if (ok:=FindText(924, 617, 1024, 665, 0, 0, SpiralIcon))
                 {             
                    return
                    
                 }else{
                 
                   CoordMode, Mouse
                   MouseGetPos, save123, save321
                   X:=ok.1.x, Y:=ok.1.y, Comment:=ok.1.id
                   Click, %X%, %Y%
                    sleep, 100
                    
                   MouseMove, save123, save321                                
                 }
             }


However, if I take out the check for icon B, the click for icon A works perfectly fine and clicks in the correct location:

Code: Select all

    Text:="|<>*184$68.zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz0zzzzwzzzzzU3zzzzDzzzzssTzzzzzzzzyT7zzzzzzzzzbtwDjrCzT1ztyS0lsn7XUDyT76CSQttlnzU3nlbbCSQwTs1swMlnX6D7y8S06AwsnU1zb7U3nDDAs0ztsszwnnnCDzySDDz0ww3nzzblkXsTDVw0ztyS0y7nsTUDzzzsTnzzDy7zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzs"

            SpiralIcon:="|<>*150$28.zzk1zzz03zzs0DzzU0Tzy01zUk07s0U0T0101w0007k080T30U1s0207lU80S31U1w6A0DUTU0yUE0Ly002zk00HzU03Tz00/zy04zzy0DzzzzzzU"
            
             if (ok:=FindText(889, 675, 1059, 723, 0, 0, Text))
             {

    
                   CoordMode, Mouse
                   MouseGetPos, save123, save321
                   X:=ok.1.x, Y:=ok.1.y, Comment:=ok.1.id
                   Click, %X%, %Y%
                    sleep, 100
                    
                   MouseMove, save123, save321                                
                 
             }
User avatar
rommmcek
Posts: 1101
Joined: 15 Aug 2014, 15:18

Re: FindText - Capture screen image into text and then find it

29 Aug 2020, 08:03

Try:

Code: Select all

    if (ok:=FindText(924, 617, 1024, 665, 0, 0, SpiralIcon))
    {             
        return
    }
     
    sleep, 500
    if (ok:=FindText(889, 675, 1059, 723, 0, 0, Text))
    {
        CoordMode, Mouse
        MouseGetPos, save123, save321
        X:=ok.1.x, Y:=ok.1.y, Comment:=ok.1.id
        Click, %X%, %Y%
        sleep, 100

        MouseMove, save123, save321                                
    }
or to be more consistent with your code:

Code: Select all

             if (ok:=FindText(889, 675, 1059, 723, 0, 0, Text))
             {
                
                sleep, 500

                 if FindText(924, 617, 1024, 665, 0, 0, SpiralIcon)
                 {             
                    return
                    
                 }else{
                 
                   CoordMode, Mouse
                   MouseGetPos, save123, save321
                   X:=ok.1.x, Y:=ok.1.y, Comment:=ok.1.id
                   Click, %X%, %Y%
                    sleep, 100
                    
                   MouseMove, save123, save321                                
                 }
             }
P.s.: For your previous question try: if (ok:=FindText(1125, 754, 1247, 818, 0.15, 0.15, Text)) or the like.

Return to “Scripts and Functions”

Who is online

Users browsing this forum: Ben and 16 guests