FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools for AHK v1.1 and older
feiyue
Posts: 349
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 24 Apr 2022, 19:51

@Descolada
This is a good suggestion and solves a long-standing problem. :D :thumbup:

goldfish
Posts: 2
Joined: 03 May 2022, 13:16

Re: FindText - Capture screen image into text and then find it

Post by goldfish » 03 May 2022, 13:35

Thanks feiyue for your awesome work. And thanks descolada, that fix worked like a charm.

I was wondering if it was possible for a script to take a screenshot, process it through findtext, and save it in a piclib.
Say I have this: FindText().Screenshot(0, 100, 100, 100). Would it be possible to save the text abstraction of that image into a piclib?

I'm fairly new to all of this, so apologies if I'm not explaining myself clearly.

Edit:
Think I found my answer on page 28.
FindText_GetTextFromScreen(x1, y1, x2, y2)
However, I can't seem to change the comment(id) on the results

feiyue
Posts: 349
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

Post by feiyue » 03 May 2022, 21:45

Text:=FindText().GetTextFromScreen(x1, y1, x2, y2)
Text:=StrReplace(Text, "<>", "<" id ">")

goldfish
Posts: 2
Joined: 03 May 2022, 13:16

Re: FindText - Capture screen image into text and then find it

Post by goldfish » 03 May 2022, 21:59

@feiyue
Thanks for the quick response. That worked!
Whole time I was trying to assign an item into the array and it wasn't working :headwall:

cgx5871
Posts: 315
Joined: 26 Jul 2018, 14:02

Re: FindText - Capture screen image into text and then find it

Post by cgx5871 » 09 May 2022, 06:53

Can there be a detailed instruction manual?
Newbie, don't know what FindText().Click is.
So can you make a list.
Which functions are supported. Description of what to return.
for example:
FindText().Click ; Description
FindText().RangeTip ; Description
FindText().GetTextFromScreen ; Description

gregster
Posts: 8921
Joined: 30 Sep 2013, 06:48

Re: FindText - Capture screen image into text and then find it

Post by gregster » 09 May 2022, 09:52

cgx5871 wrote:
09 May 2022, 06:53
Can there be a detailed instruction manual?
Did you already look at this new tutorial?
viewtopic.php?f=7&t=102806

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: FindText - Capture screen image into text and then find it

Post by SteveMylo » 09 May 2022, 16:26

@gregster i’ve been using FindText for a year & even teaching others. & I could never find that tutorial no matter how hard I searched :-). Thanks.

gregster
Posts: 8921
Joined: 30 Sep 2013, 06:48

Re: FindText - Capture screen image into text and then find it

Post by gregster » 09 May 2022, 16:37

No problem. Thanks to @Descolada for making it. :thumbup:
(Edit: I was thinking that a link to it could be added to the OP, but I see that feiyue already did that.)

Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: FindText - Capture screen image into text and then find it

Post by Epoch » 14 May 2022, 14:35

I'm trying to set F4 and F5 keys to change presets by finding and clicking the next/previous preset icons in different virtual instruments on an Audio Production program.
For what I am currently trying to do, I am using a plugin host so the instruments open within a common GUI so unfortunately Wintitle, exe, process, etc. is the same, which is why I resort in searching icons. These of course look different and they are located in different positions in each instrument

Code: Select all

f4:: ; Patch -

Text:="|<>[email protected]$17.zvnzjDytzzjDytzrjzzzzzzzzzzzzzzk" ; Try to find the "Previous Preset"  icon for Instrument A

if (ok:=FindText(X, Y, 508, 153, 2278, 963, 0, 0, Text))     ; If it's Instrument A
{
MouseGetPos, CoordXRec, CoordYRec
FindText().Click(X-912, Y+473, "L")                         ; Click the "Previous Preset"  icon and return to the original cursor position.
MouseMove, %CoordXRec%, %CoordYRec%, 0
return
}

else                                            ; If it is not found...

Text:="|<>*120$17.U00000000zzlzzXzz006007zw7zwDzk00000200DzzzzzzzzU00001"    ; ...Then perhaps it is instrument B?
if (ok:=FindText(X, Y, 266, 95, 2264, 982, 0, 0, Text))
{
MouseGetPos, CoordXRec, CoordYRec	
FindText().Click(X+655, Y-20, "L")
MouseMove, %CoordXRec%, %CoordYRec%, 0
return
}

else...                                   ;... check whether it's Instrument C etc. etc.


f5:: ; Patch +   ; Similar approach for the "Next Preset" icon...

Now, while this seems to work fine for the first 3-4 instruments and Findtext says it takes it no more than 50 ms approx. for finding each given icon, by the time it reaches Instrument 6, somehow it all adds up to about 2-3 seconds between moving to the previous or the next preset, which is less than ideal...
Provided there are ways to speed up the process, my questions are:

1). When it comes to the approach I am following, is there anything I can do to make Findtext process it faster? Maybe there's something wrong or something missing on my code that would improve things? Things I've tried:

a). Using the Method4::

Code: Select all

 FindText_Control_Click(ok, ok[1].x, ok[1].y,
found on this guide by SteveMylo viewtopic.php?p=441482#p441482. Controlclick doesn't seem to work in this case despite I could have sworn I was previously able to make it work when using Joe Glines "AutomateMyTask". The reason I don't use that one for this (and most other tasks) is the fact that unlike FindText, once it can't detect an image, it returns an error message popup so it ruins the chance to build code around it, an "if not found, search for the next icon" conditional in this case. If by any chance, anyone knows a way to get around that, please let me know.

b). Using "GetRange". While it introduces the danger of missing the icon completely if it's out of range, it doesn't seem to reduce time that much to make a difference in the whole process (Findtext test reports 10 ms less at best).




2). While I'd still be very interested to know about possible mistakes or things I could do to improve the time my process takes (I use Findtext a lot so learning new things is always useful) I couldn't help but realize that there's probably a much better way (faster process - less/slicker code) to deal with this than the one I follow.
My AHK skills just aren't advanced enough to find it though, the only alternative I was able to think was somehow using Timers to keep searching for the icons in the background so the script always know at least what instrument is in focus then, when I press F4 or F5 to browse presets, it will already know where to look.
Something like:

Code: Select all

SetTimer, GoThroughAllInstruments
{
;keep rotating findtext searches here and be ready to recall them when I press F4 or F5. Once pressed, stop rotating and take the appropriate (preset + or -) action. When the icon isn't found/in focus anymore, restart the process etc.
}
I can imagine something like that would at least save me the time when browsing presets from the same instrument but I am not that skilled with Timers to set that up or be in the position to tell if there's an even better way, so any help is welcome. Thanks!

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: FindText - Capture screen image into text and then find it

Post by Descolada » 15 May 2022, 07:37

Epoch wrote:
14 May 2022, 14:35
I'm trying to set F4 and F5 keys to change presets by finding and clicking the next/previous preset icons in different virtual instruments on an Audio Production program.
For what I am currently trying to do, I am using a plugin host so the instruments open within a common GUI so unfortunately Wintitle, exe, process, etc. is the same, which is why I resort in searching icons. These of course look different and they are located in different positions in each instrument

....
Hello,
The slow speed is due to multiple reasons. First is that you are calling FindText multiple times, which means it takes a screenshot and goes through the screenshot all over again for every search. To get around it you could give your Text an id and combine them into one, then you could use ok[1].id in the if statements (an example is below). Another way would be to set Screenshot argument to 0 for all but the first search.
Second reason is that your err1 and err0 are set to 0. In this default case if FindText fails the search, then it tries again with 0.05 values, thus doubling the search time. Try either using 0.05 (5% error margin) or if you need exact matches then a very low number like 0.00001.

Maybe something like this:

Code: Select all

f4:: ; Patch -

Text:="|<instrumentA>[email protected]$17.zvnzjDytzzjDytzrjzzzzzzzzzzzzzzk"
Text.="|<instrumentB>*120$17.U00000000zzlzzXzz006007zw7zwDzk00000200DzzzzzzzzU00001"  
if !(ok:=FindText(X, Y, 508, 153, 2278, 963, 0.05, 0.05, Text))
    return
MouseGetPos, CoordXRec, CoordYRec
if (ok[1].id == "instrumentA") {
FindText().Click(X-912, Y+473, "L") 
} else if (ok[1].id == "instrumentB") {	
FindText().Click(X+655, Y-20, "L")
}
MouseMove, %CoordXRec%, %CoordYRec%, 0
return
f5:: ; Patch +   ; Similar approach for the "Next Preset" icon...
The ControlClick problem might be because of differences in coordinate handling. FindText returns coordinates relative to the screen (0,0 = top left corner of screen), but ControlClick takes coordinates relative to the window (0,0 = top left corner of window). Try using FindText().ScreenToWindow function to get the correct coordinates and try again with those.

EDIT: changed my explanation about the ControlClick problem to not mention CoordMode, which is unrelated here and might be confusing.

Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: FindText - Capture screen image into text and then find it

Post by Epoch » 15 May 2022, 19:44

Thanks a lot for the reply, I really appreciate it! Your guide on the function has also helped in general while first trying to get my head around it and combining the searches under the same id definitely improved things, but in a true "Teach a man how to fish..." fashion, I do have some questions, in order to understand the whole thing better.
Descolada wrote:
15 May 2022, 07:37
First is that you are calling FindText multiple times, which means it takes a screenshot and goes through the screenshot all over again for every search. To get around it you could give your Text an id and combine them into one, then you could use ok[1].id in the if statements (an example is below).
That did make a considerable difference indeed. So, what's the difference behind the scenes, does it skip the taking the screenshot part in this case or...?
Another way would be to set Screenshot argument to 0 for all but the first search.
Where should I put that in our example? I saw that argument mentioned after "Text" so I suppose it goes right after that, like this?

Code: Select all

if !(ok:=FindText(X, Y, 508, 153, 2278, 963, 0.05, 0.05, Text, Screenshot=0)
Also, why is there an !/not equal on the FindText function now?
Second reason is that your err1 and err0 are set to 0. In this default case if FindText fails the search, then it tries again with 0.05 values, thus doubling the search time. Try either using 0.05 (5% error margin) or if you need exact matches then a very low number like 0.00001.
That didn't seem to make a difference with the process speed but it's good to know anyway. Does a wider error margin always mean a faster search? For example, If hypothetically 0.05, 0.1, 0.3 values were all able to find the image, would the process with the 0.3 value be the fastest one or anything other than 0 is equally fast?
So err1 and err0 seem to be the equivalent of Imagesearch's * n (variation) ? Because I tried to use them in order to get colour variations before and I've tried again now, with some other icons and they were not found. These for example:
Image
Image
No matter how much I changed the value, I wasn't able to catch both icons with the same search.
The ControlClick problem might be because of differences in coordinate handling. FindText returns coordinates relative to the screen (0,0 = top left corner of screen), but ControlClick takes coordinates relative to the window (0,0 = top left corner of window). Try using FindText().ScreenToWindow function to get the correct coordinates and try again with those.
EDIT: changed my explanation about the ControlClick problem to not mention CoordMode, which is unrelated here and might be confusing.
Yes, I was about to ask whether simply putting CoordMode before my search would have the same effect. But, again, I'm not sure where to put the ScreenToWindow function. I tried to put it on several different positions inside the code, but the coordinates wouldn't change and/or I'd get a "Call to a nonexistent function".
Here's what I came up with by basically trying to copy what you say on section 6 on your tutorial about "WindowToScreen" (figured the syntax would be similar, is it correct?)

Code: Select all

    
Text:="|<>*112$10.TwTszUy1w1k30A1kS3sz7xzs"

if (ok:=FindText(X, Y, 1420-150000, 431-150000, 1420+150000, 431+150000, 0, 0, Text))
{
  FindText().Click(X, Y, "L")
  ScreenToWindow(1420, 431, 200, 300) 
  FindText_Control_Click(ok, ok[1].x, ok[1].y, ahkEXE:="ahk_exe Studio One.exe")
  }
return
ScreenToWindow seems to return a "Call to a nonexistent function"

Thanks again.

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: FindText - Capture screen image into text and then find it

Post by SteveMylo » 15 May 2022, 21:59

]@EpochHi Epoch.
Here is a super simple script I use that may be what you want. Not sure.
I use an Editing program, I have 3 icons that I want to click when I push a hotkey. So I just create a loop.
The reason the loop works is because once the icon is clicked, the icons 'Change Colour'. therefor becomes invisible to FindText which is good, so then the function moves to the next icon to find.
Is this the case for you?
Here is the simple loop script. I don't bother with Tolerances at all. IF you need two or more variations then I just put in more search lines.
This script below is lightning fast.

Code: Select all

d::
Text:="|<Icon1>FFFFFF-585858$20.00401UU8080003zzwU01800G404XU1cEAG05YU2B9l0GXU4U01800Hzzy"
Text.="|<Icon2>FFFFFF-585858$19.00E00Q00T0870C10200000070U7UE7kw7k46k22E02M82MC3823801800s00E"
Text.="|<Icon3>FFFFFF-585858$16.U01U670kC60QMEsn3Xs430200E0AS1lwD7tsTr1y83m"
Loop
{
	if !(ok:=FindText(X, Y, 0, 0, 0, 0, 0, 0, Text,,,,,,1))    ; put in your search range if you want it even faster. 
		break       ;  The '!'  and break' is needed to break the loop if nothing is found. 
	  MouseClick, Left, X, Y, 1, 1
}
return 

~Esc::ExitApp
Last edited by SteveMylo on 16 May 2022, 00:12, edited 2 times in total.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: FindText - Capture screen image into text and then find it

Post by Descolada » 15 May 2022, 23:31

Epoch wrote:
15 May 2022, 19:44
That did make a considerable difference indeed. So, what's the difference behind the scenes, does it skip the taking the screenshot part in this case or...?
Yes, in that case it takes the screenshot just once and then goes through all the different Text one by one. Speedwise it would probably be the same as using FindText separately for each Text, but setting Screenshot to 1 for the first call.
Epoch wrote:
15 May 2022, 19:44
Where should I put that in our example? I saw that argument mentioned after "Text" so I suppose it goes right after that, like this?
Yes, it goes after Text, like so: (ok:=FindText(X, Y, 508, 153, 2278, 963, 0.05, 0.05, Text, 0) (without the "Screenshot=")
Epoch wrote:
15 May 2022, 19:44

Code: Select all

if !(ok:=FindText(X, Y, 508, 153, 2278, 963, 0.05, 0.05, Text, Screenshot=0)
Also, why is there an !/not equal on the FindText function now?
This is because down the line I am using ok[1].id for the comparisons, and this makes sure that ok contains something (otherwise would return). If FindText didn't find anything and then I would use ok[1].id in a comparison, then most likely everything would work anyway, because AHK knows that ok is equal to 0 (not an array!) and then if I try to get the first element of 0 it would return 0 anyway. But since in other languages it might result in errors or weird behavior, I like to check first. If you want you can skip the check:

Code: Select all

ok:=FindText(X, Y, 508, 153, 2278, 963, 0.05, 0.05, Text)
if (ok[1].id == .....
Epoch wrote:
15 May 2022, 19:44
That didn't seem to make a difference with the process speed but it's good to know anyway. Does a wider error margin always mean a faster search? For example, If hypothetically 0.05, 0.1, 0.3 values were all able to find the image, would the process with the 0.3 value be the fastest one or anything other than 0 is equally fast?
So err1 and err0 seem to be the equivalent of Imagesearch's * n (variation) ? Because I tried to use them in order to get colour variations before and I've tried again now, with some other icons and they were not found. These for example:
Image
Image
No matter how much I changed the value, I wasn't able to catch both icons with the same search.
The wider the error margin, the slower the search. 0.3 would process the image much slower than 0.05. My example wasn't the best for speed comparisons, because I wasn't sure how much your code relied on the default error margin, so I used 0.05 which should be a tiny bit faster. If you are looking for exact matches, try 0.00001 for both err1 and err0, which should be a bit faster still.

Err1 and err0 are similar to the variation number, yes. I tried to capture your images in one Text, but its difficult indeed (without using wide error margins). In this case its probably better to capture each separately and then combine them in the search (as in my example).
Epoch wrote:
15 May 2022, 19:44

Code: Select all

  ScreenToWindow(1420, 431, 200, 300) 
  FindText_Control_Click(ok, ok[1].x, ok[1].y, ahkEXE:="ahk_exe Studio One.exe")
should be

Code: Select all

  FindText().ScreenToWindow(newX, newY, X, Y, WinExist("ahk_exe Studio One.exe"))
  ControlClick,x%newX% y%newY%,ahk_exe Studio One.exe,,,,NA
Further speed improvements could be had by reducing the search range: for example only in the desired window, then maybe further cropping out the toolbar or other spots where the image cannot appear.
Also using a smaller Text should be faster too.

Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: FindText - Capture screen image into text and then find it

Post by Epoch » 16 May 2022, 13:03

Thanks again for the heads up!
Descolada wrote:
15 May 2022, 23:31
The wider the error margin, the slower the search. 0.3 would process the image much slower than 0.05. My example wasn't the best for speed comparisons, because I wasn't sure how much your code relied on the default error margin, so I used 0.05 which should be a tiny bit faster. If you are looking for exact matches, try 0.00001 for both err1 and err0, which should be a bit faster still.
I was initially a bit confused thinking these two are contradicting statements:
your err1 and err0 are set to 0. In this default case if FindText fails the search, then it tries again with 0.05 values, thus doubling the search time.
The wider the error margin, the slower the search. 0.3 would process the image much slower than 0.05.
I mean, it sounds like 0 is the smallest value possible so it should result in the fastest process. But then again, you've said it'd try again with 0.05 values (will it keep trying with more values? does that happen only 0 as an initial value?) if I set it to 0, so I guess that explains 0 being slower than 0.00001...
So, in any case I am looking for the exact image and not variances I should use 0.00001 rather than 0?
Further speed improvements could be had by reducing the search range: for example only in the desired window, then maybe further cropping out the toolbar or other spots where the image cannot appear.
What tool bar? I take screenshots of just the back/forward arrows, use the L3/R3/L/R/Auto etc to get rid of any unnecessary pixels only keeping those strictly necessary, Gray2Two/Colour - OK, at times using the "GetRange" to limit the area it needs to search (in this case, since I am looking for many different icons on different GUIs I chose not to limit it that much) and... that's it. Is there something I could use to limit it even more / make it even faster I am missing? Btw It doesn't search in "other" monitors by default, does it? Because I also have my TV permanently connected to the PC too, albeit usually switched off and only use "PC Sceen only" on Win Project manager (no extend or duplicate).
Also using a smaller Text should be faster too.
How? Wouldn't randomly remove characters from the text line that was auto-generated mess with the whole process?

should be

Code: Select all - Download - Toggle Line numbers

FindText().ScreenToWindow(newX, newY, X, Y, WinExist("ahk_exe Studio One.exe"))
ControlClick,x%newX% y%newY%,ahk_exe Studio One.exe,,,,NA
That did the trick! Now it finds everything using Controlclick. I am not sure if it speeds up the process significantly yet, but it's definitely slicker as it gets rid of setting and returning to the original cursor position, so I am keeping it.
Is there a way to permanently attach code lines I use a lot to be included in the generated code for each and every search? For example, I'd like to uncomment the findclick line and add the screentowindow/controlclick or the getmouseposition lines (commented or not) by defaut.
I guess it should be possible by editing the function placed on the Autokey "lib" folder, somewhere, but that's where my guesses stop helping...

I know passing params correctly, unlike the facepalming examples I've used must look really obvious to the advanced user, but for the beginner/intermediate user that's not always the case... ;)
After 6). in your tutorial I find it considerably harder to follow and it makes me wonder what other features of this great function I miss...

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: FindText - Capture screen image into text and then find it

Post by Descolada » 16 May 2022, 14:17

Epoch wrote:
16 May 2022, 13:03
I mean, it sounds like 0 is the smallest value possible so it should result in the fastest process. But then again, you've said it'd try again with 0.05 values (will it keep trying with more values? does that happen only 0 as an initial value?) if I set it to 0, so I guess that explains 0 being slower than 0.00001...
So, in any case I am looking for the exact image and not variances I should use 0.00001 rather than 0?
If err1 and err0 are both set to 0 (which is the default value for both), then FindText will first try to find an exact match (because the error margins are 0), but if it fails it will try one more time with both err1 and err0 set to 0.05, thus a maximum of two tries. This means if your Text is not visible on the screen, then every search happens twice and pretty much doubles the search time (not exactly so, because 0.05 takes just a tiny bit longer). But this only happens when err1 and err0 are exactly 0, which means if you put 0.000001 (effectively 0 but does not equal 0) for err1, err0, or both (just one needs to be non-zero), then only one search will happen (with effective error margins of 0.000001≈0).
So yes, if you are looking for the exact image, then you should use 0.00001 (or some other very small number).
Epoch wrote:
16 May 2022, 13:03
What tool bar? I take screenshots of just the back/forward arrows, use the L3/R3/L/R/Auto etc to get rid of any unnecessary pixels only keeping those strictly necessary, Gray2Two/Colour - OK, at times using the "GetRange" to limit the area it needs to search (in this case, since I am looking for many different icons on different GUIs I chose not to limit it that much) and... that's it. Is there something I could use to limit it even more / make it even faster I am missing? Btw It doesn't search in "other" monitors by default, does it? Because I also have my TV permanently connected to the PC too, albeit usually switched off and only use "PC Sceen only" on Win Project manager (no extend or duplicate).
It seems I was typing in a hurry: I didn't mean toolbar, but menubar. I usually use WinGetPos to get the search range for the window of interest, but it also includes areas that are usually not needed: titlebar, menubar (and perhaps also toolbars in some windows?), so I usually adjust the range parameters by some margin to not include these. The smaller the search area, the faster the search... If you have multiple text that might appear in different regions, then you can use different search ranges for the searches, but set Screenshot argument to 0 after the first search of the batch.

Code: Select all

if (ok:=FindText(X, Y, 0, 0, 500, 500, 0.00001,, Text)) ; search inside the area (0;0) (500;500)
	; do something
else if (ok:=FindText(X, Y, 500, 500, 1000, 1000, 0.00001,, Text, 0)) ; use the previously taken screenshot but search inside the area (500;500) (1000;1000)
	; do something else
And it seems you are already reducing the size of the Text as well which increases the speed: back-forward arrows, L3, Auto etc crop the screenshot, which results in a smaller Text.
Epoch wrote:
16 May 2022, 13:03
How? Wouldn't randomly remove characters from the text line that was auto-generated mess with the whole process?
Don't remove characters from the Text manually, the screenshot cropping buttons allow you to do that.
Epoch wrote:
16 May 2022, 13:03
Is there a way to permanently attach code lines I use a lot to be included in the generated code for each and every search? For example, I'd like to uncomment the findclick line and add the screentowindow/controlclick or the getmouseposition lines (commented or not) by defaut.
I guess it should be possible by editing the function placed on the Autokey "lib" folder, somewhere, but that's where my guesses stop helping...
Ctrl+F the FindText library code and search for "; FindText().Click" (without the quotes of course), should be easy to replace it with your desired function name. ;)

Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: FindText - Capture screen image into text and then find it

Post by Epoch » 16 May 2022, 18:38

SteveMylo wrote:
15 May 2022, 21:59
]@EpochHi Epoch.
Here is a super simple script I use that may be what you want. Not sure.
I use an Editing program, I have 3 icons that I want to click when I push a hotkey. So I just create a loop.
The reason the loop works is because once the icon is clicked, the icons 'Change Colour'. therefor becomes invisible to FindText which is good, so then the function moves to the next icon to find.
Is this the case for you?
Here is the simple loop script. I don't bother with Tolerances at all. IF you need two or more variations then I just put in more search lines.
This script below is lightning fast.

Code: Select all

d::
Text:="|<Icon1>FFFFFF-585858$20.00401UU8080003zzwU01800G404XU1cEAG05YU2B9l0GXU4U01800Hzzy"
Text.="|<Icon2>FFFFFF-585858$19.00E00Q00T0870C10200000070U7UE7kw7k46k22E02M82MC3823801800s00E"
Text.="|<Icon3>FFFFFF-585858$16.U01U670kC60QMEsn3Xs430200E0AS1lwD7tsTr1y83m"
Loop
{
	if !(ok:=FindText(X, Y, 0, 0, 0, 0, 0, 0, Text,,,,,,1))    ; put in your search range if you want it even faster. 
		break       ;  The '!'  and break' is needed to break the loop if nothing is found. 
	  MouseClick, Left, X, Y, 1, 1
}
return 

~Esc::ExitApp
Hi. Thanks for the reply. I am not sure the arrows in the virtual instruments change colour (at least not all of them), but I am saving your script because it could definitely be useful at some point. I think Adobe programs icons are like that and I plan to give them the AHK treatment, further down the line.
I often consult your viewtopic.php?p=441482#p441482 post, along with Descolada's (and the others') tutorials. Thanks for helping out people. And while I am at it, thanks to feiyue for developing this great function as well!
AHK and functions like this have been almost life-altering when it comes to pretty much doing anything on a PC, including Productivity, I've been using it for a few months now and I wouldn't be able going back.
Since you've said you use editing programs I have a recommendation for you and others, "Radial Menu" which is my latest AHK-related discovery and looks really powerful. I haven't started using it properly yet as it needs some setup, but once that's done I can imagine it will only make things better.
Makes me wonder what other awesome functions/scripts are waiting to be discovered... To be honest, I guess most people know it already but it just came up as a genuine way to say thank you to people like you and Descolada.

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: FindText - Capture screen image into text and then find it

Post by SteveMylo » 16 May 2022, 19:39

@Epoch Great. I'll be the 1st to admit I'm only an intermediate user and coder. But I'm really good at using Findtext to find images really fast and that's it haha.
If your icons/buttons don't change colour/brightness after clicking then the 'Loop' won't work. You'll just have to restructure the code.
As I mentioned, I don't use tolerances, I just add more searches if needed. Tolerance values will slow it down. Two monitors gets tricky but can be done as long as you don't keep changing screen resolution and DPI etc.
I found ColorDiff capture method to be the most consistent. MultiColor is perfect too for tricky situations.

Oh and yess... I use Radial Menu for all things I need quick in my PC. I don't use it for editing but maybe I should :-)
as for changing the Function to add in new lines of code. I have done that so I have a default ready for me to copy and paste.
BEWARE, make a full copy of FINDTEXT fucniotn 1st. One little mistake will ruin the function.
Below I will paste what I have done to add my own defaults in the GUI. It's starts around the line 3000

Code: Select all

. "`nt1:=A_TickCount, X:=Y:=""""`n" s
    . "`nif (ok:=FindText(X, Y, 0, 0, 0, 0, 0, 0, Text,,,,,,1))"
    . "`n{"
    . "`n  `FindText().Click(ok[1].x, ok[1].y, ""U"", 1)"
    . "`n}`n"
    . "`n `;~1 = TL - Scan R"
    . "`n `;~2 = TR - Scan L"
    . "`n `;~3= BL Scan R"
    . "`n `;~4= BR scan L"
    . "`n `;~5= TL Scan D"
    . "`n `;~6= BL - Scan U"
    . "`n `;~7= TR Scan D"
    . "`n `;~8= BR Scan U"
    . "`n `;~9= Starts at the Centre`n"
    . "`n  `;~ Click(1)"
    . "`n  `;~ FindText_Control_Click(ok,X,Y,ahkEXE:=""Your .exe program here"")"
    . "`n`; ok:=FindText(X:=""wait"", Y:=3, 0,0,0,0,0,0,Text)    `; Wait 3 seconds for appear"
    . "`n`; ok:=FindText(X:=""wait0"", Y:=-1, 0,0,0,0,0,0,Text)  `; Wait indefinitely for disappear`n"
    . "`nMsgBox, 4096, Tip, `% """ r[1] ":``t"" Round(ok.Length())"
    . "`n  . ""``n``n" r[2] ":``t"" (A_TickCount-t1) "" " r[3] """"
    . "`n  . ""``n``n" r[4] ":``t"" X "", "" Y"
    . "`n  . ""``n``n" r[5] ":``t<"" (Comment:=ok[1].id) "">""`n"
    . "`nfor i,v in ok  `; ok value can be get from ok:=FindText().ok"
    . "`n  if (i<=4)"
    . "`n    FindText().MouseTip(ok[i].x, ok[i].y)`n"

Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: FindText - Capture screen image into text and then find it

Post by Epoch » 19 May 2022, 10:20

SteveMylo wrote: @Epoch
I found ColorDiff capture method to be the most consistent. MultiColor is perfect too for tricky situations.

To be honest, the differences between all the modes are bit confusing to me, I am not sure which fits each situation best, I just use "Gray2two" when colours aren't important and I just need to get the shape and "Colour2two" for those cases where colours are important and it seems to be going well for the most part.
Are the modes you mention more accurate or faster?
BEWARE, make a full copy of FINDTEXT fucniotn 1st. One little mistake will ruin the function.
Below I will paste what I have done to add my own defaults in the GUI. It's starts around the line 3000

Code: Select all

. "`nt1:=A_TickCount, X:=Y:=""""`n" s
    . "`nif (ok:=FindText(X, Y, 0, 0, 0, 0, 0, 0, Text,,,,,,1))"
    . "`n{"
    . "`n  `FindText().Click(ok[1].x, ok[1].y, ""U"", 1)"
    . "`n}`n"
    . "`n `;~1 = TL - Scan R"
    . "`n `;~2 = TR - Scan L"
    . "`n `;~3= BL Scan R"
    . "`n `;~4= BR scan L"
    . "`n `;~5= TL Scan D"
    . "`n `;~6= BL - Scan U"
    . "`n `;~7= TR Scan D"
    . "`n `;~8= BR Scan U"
    . "`n `;~9= Starts at the Centre`n"
    . "`n  `;~ Click(1)"
    . "`n  `;~ FindText_Control_Click(ok,X,Y,ahkEXE:=""Your .exe program here"")"
    . "`n`; ok:=FindText(X:=""wait"", Y:=3, 0,0,0,0,0,0,Text)    `; Wait 3 seconds for appear"
    . "`n`; ok:=FindText(X:=""wait0"", Y:=-1, 0,0,0,0,0,0,Text)  `; Wait indefinitely for disappear`n"
    . "`nMsgBox, 4096, Tip, `% """ r[1] ":``t"" Round(ok.Length())"
    . "`n  . ""``n``n" r[2] ":``t"" (A_TickCount-t1) "" " r[3] """"
    . "`n  . ""``n``n" r[4] ":``t"" X "", "" Y"
    . "`n  . ""``n``n" r[5] ":``t<"" (Comment:=ok[1].id) "">""`n"
    . "`nfor i,v in ok  `; ok value can be get from ok:=FindText().ok"
    . "`n  if (i<=4)"
    . "`n    FindText().MouseTip(ok[i].x, ok[i].y)`n"
Cheers, it was more complicated than I thought for some reason, but eventually I made it work. What do these scan code lines do? If they are useful I might adopt them. ;)

Descolada wrote: Ctrl+F the FindText library code and search for "; FindText().Click" (without the quotes of course), should be easy to replace it with your desired function name. ;)
For some reason, even though I was making sure to copy the quotation marks, the line breaks etc. AHK would give me a hard time, complaining for illegal characters et. (usually for brackets and equation marks) but eventually it let me copy what I wanted.
Not sure why it wasn't as straightforward as copying the lines I wanted under the "; FindText().Click" one, but I think it's alright now. I use AHK Studio for writing scripts btw. Thanks again!


I've been experimenting with the BindWin3 function (couldn't get the rest to work) lately, pretty cool you can reach unfocused windows, you can do some useful stuff with it.
I've been trying to create Rewind/Fast Forward hotkeys for Deezer UI, loaded on a browser in the background (Opera, using it only for this reason) by clicking near (3 or 5 secs) the playhead dot in the timeline, while a track is playing.
This works fine when the window is in focus but the problem is, as you can see in the gif, that playhead dot only appears when the mouse cursor goes near the timeline area, which complicates the BindWin operation.
I've tried using a FindText().Click at some spot where it triggers it, THEN search for the dot and click close to it, X-10/X+10 for Rewind/FF, but unless the window is in focus, it will only find the first spot.
Also tried to see if I can get it to click at the point where the horizontal line changes colour (remaining time line is always gray, elapsed time line is usually a different colour depending on the track/artist btw), which is pretty much the current position in the current track, but it seems it creates a pretty generic screenshot, finding many random spots. Any ideas guys?

Image

User avatar
SteveMylo
Posts: 233
Joined: 22 Jun 2021, 00:50
Location: Australia
Contact:

Re: FindText - Capture screen image into text and then find it

Post by SteveMylo » 19 May 2022, 16:00

Epoch wrote:
19 May 2022, 10:20
Are the modes you mention more accurate or faster?
No mode is faster than the other, it only depends on how big the image is which makes the difference in speed. always try to do the tiniest
Grey or Grey Diff works most of the time, but I found that once i had to search for a Play Button icon and it found something that looks like a play button in my actual footage & not in my softwares GUI haha. Colour doesn't seem to matter with grey. So the search was incorrect.
Also..... I transferred my script to another computer, ran my software and didn't get good results. Only ColorDiff was the perfect one. But if you have simple images and staying on one computer then choose any I guess.
Epoch wrote:What do these scan code lines do? If they are useful I might adopt them. ;)
I mentioned it in this thread viewtopic.php?p=441482#p441482 you referenced it. It is the direction FindText searches. Sometimes you need it to start the search from somewhere else because there is a duplicate image somewhere that you don't want, maybe you only need findtext to find only one of the duplicate images. Also the closer the image to the search direction the quicker the search. I have placed them in my 'FindText GUI' so I remember what direction they go according to the number.
The number is written at the end, after the 6 commas ,,,,,, that is after the variable word "Text", ..... e.g. if (ok:=FindText(X, Y, 0, 0, 1920, 1080, 0, 0, Text,,,,,,6))
Below are the Directions to add to your code:
1 = Starts Top Left & Scans Right. then it scans again from 1 Pixel down from the Top Left and it scans Right again etc. etc.
2 = TR - Scan L
3 = BL Scan R
4 = BR scan L
5 = TL Scan D
6 = BL - Scan U
7 = TR Scan D
8 = BR scan U
9 = Starts at the Centre and scans outward in a circle

AHKharry
Posts: 7
Joined: 15 May 2022, 07:40

Re: FindText - Capture screen image into text and then find it

Post by AHKharry » 22 May 2022, 14:21

Would there be an easy way to change this from a standard Click to a ControlClick?

Code: Select all

If ok:=FindText(outX, outY,574,1092,1107,1387,,,Text)
		{
		CoordMode, Mouse
		X:=ok.1, Y:=ok.2, W:=ok.3, H:=ok.4, Comment:=ok.5
		Click, %outX% %outY%
        }
    }

Post Reply

Return to “Scripts and Functions (v1)”