FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools for AHK v1.1 and older
c4p
Posts: 21
Joined: 18 Jan 2017, 18:38

Re: FindText - Capture screen image into text and then find it

23 May 2017, 09:46

Luffy wrote: It doesn't Click , only selects the browser. Sometimes it brings the desired window "This is a new title" window to the top , but doesn't do anything else.
Updated https://autohotkey.com/boards/search.ph ... 7&sr=posts to correctly handle controlclicks in internet explorer.
hal123

Re: FindText - Capture screen image into text and then find it

08 Jun 2017, 09:15

Hi,

This is a really cool script :-)

Would it somehow be possible to make it work on dynamic text (Taking the text to find from a file)

My idea is somehow to generate the image to find. First I capture all the letters in the alphabet, based on the text and captured letters I then generate the image.

Code: Select all

Pseudo code:
LetterA.="|<A>*144$10.60Q1EBUW28Tl1A6kO"
LetterB.="|<B>*144$8.z8O2Vjm6Uc+6z8"
LetterC.="|<C>*144$10.7UX44E1040E112A7W"

TextToFindDynamic = "ABC"

dynImage = generateImage(TextToFindDynamic)

ok:=FindText(0,0,150000,150000,0,0,dynImage)

generateImage(txt)
{
	return = LetterA + LetterB + LetterC
}
How can I merge the letters into an "image word" that I can search for?

Thanks a lot.
stealzy
Posts: 91
Joined: 01 Nov 2015, 13:43

Re: FindText - Capture screen image into text and then find it

09 Jun 2017, 00:34

hal123, it was possible in version 4.7. Example.
But I think this is not best idea, you can use Tesseract OCR for such goals.
rupeshrulzz
Posts: 3
Joined: 07 Jun 2017, 02:23
Contact:

Re: FindText - Capture screen image into text and then find it

10 Jun 2017, 09:34

I am writing a script to automate some tasks in an online game I play. I am using this image search tool to make my script recognize images in my game and click at right places.
There are some images which are searched at multiple places in my script. I want to put the details for all the required images in the beginning of the script and then call the desired image search wherever I need it in the script. How could I do it?
For example I want to search for image x, image y and image z at multiple points in my script. So I'll write the details for those 3 images which I got from find text tool at the beginning of the script. So now whenever I want to search for any image say image x I'll write a function to make the script search for image x instead of writing the detail for image x again. I don't know how to make a function to search for find text image.
feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

10 Jun 2017, 22:57

rupeshrulzz wrote: I want to put the details for all the required images in the beginning of the script and then call the desired image search wherever I need it in the script. How could I do it?
This problem is easy to solve, you can do it like this:

Code: Select all


; This function import text images through the comments in < >
Pic(comments, add_new=0) {
  static Lib:=[]
  if (add_new)
  {
    re:="<([^>]*)>[^$]+\$\d+\.[\w+/]{3,}"
    Loop, Parse, comments, |
      if RegExMatch(A_LoopField,re,r)
        Lib[Trim(r1)]:=r
  }
  else
  {
    text:=""
    Loop, Parse, comments, |
      text.="|" . Lib[Trim(A_LoopField)]
    return, text
  }
}

; Usage For example:

; You can put the text library at the beginning of the script
CoordMode, Mouse, Screen
Text:="|<comment1>*138$24.10480U98zz9884Nz48O848u82Ec81U/z1U88CQ88k788U"
Text.="|<comment2>*138$24.10480U98zz9884Nz48O848u82Ec81U/z1U88CQ88k788U"
Text.="|<comment3>*138$24.10480U98zz9884Nz48O848u82Ec81U/z1U88CQ88k788U"

;-- Use Pic(Text,1) to add the text library to Pic()'s Lib
Pic(Text,1)

;.....................

;.....................

;.....................

;-- Find one picture at a time
if ok:=FindText(27,1061,150000,150000,0,0,Pic("comment1"))
{
  X:=ok.1, Y:=ok.2, W:=ok.3, H:=ok.4, X+=W//2, Y+=H//2
  ;------------------------------
  Click, X, Y
}

;-- Find more pictures at a time
if ok:=FindText(27,1061,150000,150000,0,0,Pic("comment1|comment2|comment3"))
{
  Loop, % ok.MaxIndex()//5
  {
    i:=(A_Index-1)*5, X:=ok[i+1], Y:=ok[i+2]
    W:=ok[i+3], H:=ok[i+4], Comment:=ok[i+5], X+=W//2, Y+=H//2
    ;------------------------------
    if (Comment="comment1")
    {
      Click, X, Y
    }
    else if (Comment="comment2")
    {
      Click, X, Y
    }
    else if (Comment="comment3")
    {
      Click, X, Y
    }
  }
}

20170201225639
Posts: 144
Joined: 01 Feb 2017, 22:57

Re: FindText - Capture screen image into text and then find it

11 Jun 2017, 13:04

edit: please ignore —— i misunderstood what the script does
it just occurred to me there's an excellent use case for this script.

i've been working outside in the park the last couple of days (thanks to a really capacious laptop power bank i just got recently) and noticed that the fresh air and sun made a huge difference to productivity. i doubt i'll ever (voluntarily) go back to my office again!

anyway, since using mouse on a park bench or grass is inconvenient (and i don't like touchpad either), i've been trying to use autohotkey to make everything i needed to do on my computer, doable via keyboard

i find that although defined hotkeys are an important part of the workflow, it's also really important to have a convenient way to move mouse pointer with your keyboard, because a lot of applications these days are not designed to be in principle operable with keyboard only.

i had some rudimentary scripts to do this, and also tried a couple of scripts i found on the board. but none of them can really move mouse pointer as fast as with an actual mouse.

then it occurred to me .. you know how many vim users become accustomed to move caret not by hjkl, but by search (typing 'f' then type some relevant char)? and i realized that with FindText it may be possible to do that on a system-wide level! so that, say, i might be able to move mouse pointer to the OK button, just by typing 'f' followed by OK, then the pointer will jump straight to that button. This will save you the trouble of having to literally simulated any kind of mouse 'movement' with your keyboard.

this is just an idea of course (just popped into my head half an hour ago actually)... just throwing it out there in case anyone else is interested in using the script for this kind of purpose.
Last edited by 20170201225639 on 13 Jun 2017, 10:00, edited 1 time in total.
stealzy
Posts: 91
Joined: 01 Nov 2015, 13:43

Re: FindText - Capture screen image into text and then find it

13 Jun 2017, 02:58

20170201225639, I use VimFx firefox addon and understand your idea, but it won't work because different windows have different fonts/sizes. Even in one window may be dozens combinations.
In fact, the function should be called FindMonochromePattern, because this absolutely not about OCR.
Last edited by stealzy on 13 Jun 2017, 15:17, edited 1 time in total.
20170201225639
Posts: 144
Joined: 01 Feb 2017, 22:57

Re: FindText - Capture screen image into text and then find it

13 Jun 2017, 09:57

aack! my bad for not reading the discussions closely. thanks stealzy for pointing that out.
i realized another poster also warned against this misunderstanding earlier (https://autohotkey.com/boards/viewtopic ... 319#p94319)
guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: FindText - Capture screen image into text and then find it

13 Jun 2017, 10:04

stealzy wrote: In fact, the function should be called FindMonochromePattern, because this absolutely not about OCR.
+1

ozyank
Posts: 3
Joined: 25 Aug 2017, 00:01

Re: FindText - Capture screen image into text and then find it

25 Aug 2017, 16:08

First, thank you feiyue for the really nice script. I'm very new to scripting and AHK, but I've been able to use your example below and it works great. Much faster than trying to do the same thing in imagesearch.

As I said, I'm new to all of this so looking for advice on how to modify your script to conduct a nested search within the initial loop. The flow would look something like this:

1. Search for images using your original multi-image search loop.
2. If Image 1 found, click ;; I have this working
3. If Image 2 found, click ;; I have this working
4. If Image 3 found, click And
4a. While Image 3a, 3b and 3c are on the screen, click each until all have been clicked. ;; When Image 3 is clicked, it brings up another screen with 3 images on it. When an image is clicked, it is removed from the screen. I want this nested loop to click each of these until none are remaining. There may be 1, 2 or 3 images present so it needs to click until all of them are gone.

Images in initial loop:
Image 1
Image 2
Image 3

Images in nested loop initiated when Image 3 is clicked:
Image 3a
Image 3b
Image 3c

Pseudocode:
If Image 1 is found, Click
If Image 2 is found, Click
If Image 3 is found, Click
While (Image 3a and/or 3b and/or 3c) are on screen, Click each until none remain

I'm not sure I've explained this very well so please let me know if I can clarify. Thanks for any assistance!

feiyue wrote:
rupeshrulzz wrote: I want to put the details for all the required images in the beginning of the script and then call the desired image search wherever I need it in the script. How could I do it?
This problem is easy to solve, you can do it like this:

Code: Select all


; This function import text images through the comments in < >
Pic(comments, add_new=0) {
  static Lib:=[]
  if (add_new)
  {
    re:="<([^>]*)>[^$]+\$\d+\.[\w+/]{3,}"
    Loop, Parse, comments, |
      if RegExMatch(A_LoopField,re,r)
        Lib[Trim(r1)]:=r
  }
  else
  {
    text:=""
    Loop, Parse, comments, |
      text.="|" . Lib[Trim(A_LoopField)]
    return, text
  }
}

; Usage For example:

; You can put the text library at the beginning of the script
CoordMode, Mouse, Screen
Text:="|<comment1>*138$24.10480U98zz9884Nz48O848u82Ec81U/z1U88CQ88k788U"
Text.="|<comment2>*138$24.10480U98zz9884Nz48O848u82Ec81U/z1U88CQ88k788U"
Text.="|<comment3>*138$24.10480U98zz9884Nz48O848u82Ec81U/z1U88CQ88k788U"

;-- Use Pic(Text,1) to add the text library to Pic()'s Lib
Pic(Text,1)

;.....................

;.....................

;.....................

;-- Find one picture at a time
if ok:=FindText(27,1061,150000,150000,0,0,Pic("comment1"))
{
  X:=ok.1, Y:=ok.2, W:=ok.3, H:=ok.4, X+=W//2, Y+=H//2
  ;------------------------------
  Click, X, Y
}

;-- Find more pictures at a time
if ok:=FindText(27,1061,150000,150000,0,0,Pic("comment1|comment2|comment3"))
{
  Loop, % ok.MaxIndex()//5
  {
    i:=(A_Index-1)*5, X:=ok[i+1], Y:=ok[i+2]
    W:=ok[i+3], H:=ok[i+4], Comment:=ok[i+5], X+=W//2, Y+=H//2
    ;------------------------------
    if (Comment="comment1")
    {
      Click, X, Y
    }
    else if (Comment="comment2")
    {
      Click, X, Y
    }
    else if (Comment="comment3")
    {
      Click, X, Y
    }
  }
}

feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

26 Aug 2017, 02:10

The following code is for your reference.

Code: Select all

CoordMode, Mouse

Loop {

;-- Find more pictures at a time
if ok:=FindText(27,1061,150000,150000,0,0,Pic("pic3a|pic3b|pic3c"))
{
  Loop, % ok.MaxIndex()//5
  {
    i:=(A_Index-1)*5, X:=ok[i+1], Y:=ok[i+2]
    W:=ok[i+3], H:=ok[i+4], Comment:=ok[i+5], X+=W//2, Y+=H//2
    ;------------------------------
    Click, % X, % Y
    Sleep, 500
  }
  Continue
}

;-- Find more pictures at a time
if ok:=FindText(27,1061,150000,150000,0,0
,Pic("pic1|pic2|pic3"))
{
  Loop, % ok.MaxIndex()//5
  {
    i:=(A_Index-1)*5, X:=ok[i+1], Y:=ok[i+2]
    W:=ok[i+3], H:=ok[i+4], Comment:=ok[i+5], X+=W//2, Y+=H//2
    ;------------------------------
    Click, % X, % Y
    Sleep, 500
  }
}

}
User avatar
Joe Glines
Posts: 770
Joined: 30 Sep 2013, 20:49
Location: Dallas
Contact:

Re: FindText - Capture screen image into text and then find it

28 Aug 2017, 12:40

Great job feiyue! This script rocks!
I created a tutorial showing how to use the basics of the script.


I also took a stab at writing some functions to click/send text, etc. to the image once it was found.

Code: Select all

FindText_Left_Click(Obj,X_Adj=0,Y_Adj=0,Move_Back=0){
CoordMode, Mouse
MouseGetPos,Orig_X,Orig_Y
x:=(obj.1+obj.3//2)+X_Adj
y:=(obj.2+obj.4//2)+Y_Adj
Click, %X%,%Y%
if (Move_Back){
  Sleep, 30
  MouseMove, %Orig_X%,%Orig_Y%
}
Return
}

;***********Right click*******************
FindText_Right_Click(Obj,X_Adj=0,Y_Adj=0,Move_Back=0){
CoordMode, Mouse
MouseGetPos,Orig_X,Orig_Y
x:=(obj.1+obj.3//2)+X_Adj
y:=(obj.2+obj.4//2)+Y_Adj
Click Right, %X%,%Y%
if (Move_Back){
  Sleep, 30
  MouseMove, %Orig_X%,%Orig_Y%
}
Return
}

;***********Double click*******************
FindText_Double_Click(Obj,X_Adj=0,Y_Adj=0,Move_Back=0){
CoordMode, Mouse
MouseGetPos,Orig_X,Orig_Y
x:=(obj.1+obj.3//2)+X_Adj
y:=(obj.2+obj.4//2)+Y_Adj
MouseMove, %X%,%Y%
Click 2
if (Move_Back){
  Sleep, 30
  MouseMove, %Orig_X%,%Orig_Y%
}
}

;***********Move mouse to location*******************
FindText_Move_Mouse(Obj,X_Adj=0,Y_Adj=0){
  CoordMode, Mouse
x:=(obj.1+obj.3//2)+X_Adj
y:=(obj.2+obj.4//2)+Y_Adj
MouseMove, %X%,%Y%
}

;***********Send text to location*******************
FindText_Send_Text(Obj,Text,Enter=0,X_Adj=0,Y_Adj=0,Move_Back=0){
CoordMode, Mouse
MouseGetPos,Orig_X,Orig_Y
x:=(obj.1+obj.3//2)+X_Adj
y:=(obj.2+obj.4//2)+Y_Adj
MouseMove, %X%,%Y%
Sleep 30
Click, %X%,%Y%
Sleep 30
SendInput , %Text%
if (Enter=1)
  SendInput , {Enter}
if (Move_Back){
  Sleep, 30
  MouseMove, %Orig_X%,%Orig_Y%
}
}
You can check out a video I did using the custom scripts and more on my page here
Sign-up for the 🅰️HK Newsletter

ImageImageImageImage:clap:
AHK Tutorials:Web Scraping | | Webservice APIs | AHK and Excel | Chrome | RegEx | Functions
Training: AHK Webinars Courses on AutoHotkey :ugeek:
YouTube

:thumbup: Quick Access Popup, the powerful Windows folders, apps and documents launcher!
User avatar
Reloaded
Posts: 283
Joined: 25 Aug 2017, 08:48

Re: FindText - Capture screen image into text and then find it

28 Aug 2017, 15:17

Wow best Script i ever see :D
vava

Re: FindText - Capture screen image into text and then find it

28 Sep 2017, 17:19

hello All...
I like this program. but have no luck to test all the functions.
each time click on the "test" function get an error "Can't run AHK script!" .
Running windows 7 64bit with the latest Autohotkey. hope some can help.
Helgef
Posts: 4709
Joined: 17 Jul 2016, 01:02
Contact:

Re: FindText - Capture screen image into text and then find it

29 Sep 2017, 01:50

Joe Glines wrote:Great job feiyue!
Indeed it seems very nice.
Joe Glines wrote:I created a tutorial showing how to use the basics of the script.
Hello Joe Glines :wave:
I watched your video when you posted it, but forgot to comment. It is very nice and helpful. I hope feiyue links to it in the first post so others can find.

Cheers.
feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

29 Sep 2017, 11:41

Upgrade to version 5.3, changing the returned value of the object.
Previously, when looking for multiple images,
it would be difficult to extract each matching information if multiple images were found.
Now, a two level array is used to improve the situation. :beer:
But the original:
X:=ok.1, Y:=ok.2, W:=ok.3, H:=ok.4, Comment:=ok.5
It has now been changed to:
X:=ok.1.1, Y:=ok.1.2, W:=ok.1.3, H:=ok.1.4, Comment:=ok.1.5
Or (variable): i:=A_Index
X:=ok.1, Y:=ok.2, W:=ok.3, H:=ok.4, Comment:=ok.5
Last edited by feiyue on 03 Oct 2017, 05:17, edited 2 times in total.
ali80
Posts: 4
Joined: 23 Jul 2017, 02:04

Re: FindText - Capture screen image into text and then find it

01 Oct 2017, 17:57

Hi feiyue, thanks for the great work
I have a minor problem, I include nearly all my scripts in one main script, and in that script I have #Warn flag on which gives useful warnings, when I include your code I get three warnings from it, all of them uninitialized variable usage warning, I tries fixing your code by initializing the variables first but it stopped working,
the three warnings are associated with following variables:
ppvBits
myFunc
ozzii
Posts: 481
Joined: 30 Oct 2013, 06:04

Re: FindText - Capture screen image into text and then find it

02 Oct 2017, 04:15

@Joe Glines
Do you need to update your custom scripts for the latest update ???
Frosty Fridge

Re: FindText - Capture screen image into text and then find it

02 Oct 2017, 20:06

Hello feiyue,

First off I'd like to say that this script is amazing! It's both powerful and easy to use.

I noticed that you added support for returning info in an array for multiple different images
feiyue wrote:Upgrade to version 5.3, changing the returned value of the object.
Previously, when looking for multiple images,
it would be difficult to extract each matching information if multiple images were found.
Now, a two level array is used to improve the situation. :beer:
But the original:
X:=ok.1, Y:=ok.2, W:=ok.3, H:=ok.4, Comment:=ok.5
It has now been changed to:
X:=ok.1.1, Y:=ok.1.2, W:=ok.1.3, H:=ok.1.4, Comment:=ok.1.5
Or (variable): i:=A_Index
X:=ok.1, Y:=ok.2, W:=ok.3, H:=ok.4, Comment:=ok.5


I was wondering if it was possible to have the function find and return data for multiple instances of the same image. For example, if the user's screen contains two identical pictures that are being searched for, could the function return info on both of them?

Thanks for making this script!
feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

03 Oct 2017, 04:47

Frosty Fridge wrote:I was wondering if it was possible to have the function find and return data for multiple instances of the same image. For example, if the user's screen contains two identical pictures that are being searched for, could the function return info on both of them?
Updated to the v5.4 version, Realized your idea. :superhappy:
However, even if the returned object contains multiple locations,
You also need loop processing to find the top or leftmost one.

Return to “Scripts and Functions (v1)”

Who is online

Users browsing this forum: chinagreenelvis, MrDoge, TheNaviator and 85 guests