Help with ImageSearch Topic is solved

[tomstuber]

I have enjoyed learning autohotkey. I found the tutorials and references to be excellent.
However, I have hit a stumbling block with one instance of ImageSearch (others work fine). There is a screen with times arranged in a two dimensional array as shown in TestScreen.png. A given time can be in different positions. I have attached two times which I clipped using the Window snipping tool. The script will find test1.png but not test2.png. But, this can change at various times it may find test2 and not test1 or both or neither. I thought maybe the iamges were different based on which column they were in so I was able to find an instance of a given time in each of the different columns and clipped them and looped through the search. This helped but it was still unreliable. I also tried setting the variation to 2 (*2) but that didn't help. I tried only clipping the numbers in the time clipping (e.g. 12:58) and search for that, but that wasn't more reliable. I may have an issue with there being transparency in the images. Once when I put one of the images in paint and tried to save it, it gave me a warning that it couldn't save the transparency. (the examples I have given were not processed on Paint))
I am enclosing a snippet of the code that I use for testing for reference.

Thank any and all for your help. I am anxious to hear the resolution

#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases. #Warn ; Enable warnings to assist with detecting common errors.
;SendMode Input ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
#SingleInstance Force
SetTitleMatchMode, 2
CoordMode Pixel,Screen
CoordMode Mouse,Screen
Monitor := "ImagesSurface"
SetWorkingDir %Monitor%

^j:: Gosub, FindTime

FindTime:
ImageName := "Test1.png"
loop 2
{
ImageSearch, Px, Py, 0, 0, A_ScreenWidth, A_ScreenHeight, %ImageName%

;ImageSearch, Px, Py, 790, 440, 2400, 1400, %ImageName%
If ErrorLevel=0
{
If ErrorLevel=0
Px += 200
Py += 275
MouseMove, %Px%, %Py%
msgbox % ImageName . "found"
}
else If ErrorLevel=2
msgbox % "error level 2" . ImageName

else If ErrorLevel=1
msgbox, % ImageName . "not found"
ImageName := "Test2.png"
}
return

[boiler]

Searching for anti-aliased text (also known as font smoothing) is one of the biggest headaches in using ImageSearch to find images containing text. This is very likely the problem in this case. If you zoom in on the edges of the text, you'll probably find that it is rendered differently if it appears in a different location on the screen, especially if it's in a different horizontal location, as the horizontal position is the main driver in why it will smooth the edges differently in different instances. The edges of the text (or almost all of the text for smaller font sizes) will be made up of some different colors or sometimes just shades of the main color (black in this case), but it's not made up of purely black pixels. This is what makes text look smooth to the naked eye.

Luckily, there are things you can do to work around this issue:

1. Sounds like you already tried this, but it's important (and must be coupled with part 2 below): Keep the image you are searching for to a minimum. In your case, the tee time seems to be all you need to search for, so crop your image to just that area. That will also help with the next part because you can ignore all the smaller text that would be an even bigger problem.

2. You can use the *Trans option in ImageSearch to have it ignore problem areas of your image. This is different than making part of your images transparent. You are making them a certain color that ImageSearch will skip over. So you would use MS Paint to paint over the edges of the tee time text (all the parts that aren't always solid black) in your reference images, and make that the color to be ignored. Liberally paint over any pixels that could possibly be different than pure black or white. Paint over even more pixels than you think is necessary just to be sure. If you paint over them with red (make sure it's pure red, not something that just looks red like the default red in MS Paint), then include *TransRed or *TransFF0000 in your ImageSearch options.

3. You can increase the variation. A variation of 2 like you tried is very small and will rarely help. Oftentimes, variations of 20, 40, or even higher are needed. But even variation alone can't get the job done in the case where the pixels are fundamentally different as will happen with anti-aliased text, but it can help in addition to using the *Trans option.

This should take care of the problem. If it doesn't, it is likely either because the Trans color code isn't really matching what you used to paint over the pixels or because you need to paint over more pixels. I didn't look over your code in detail because it sounds like it's good if it's finding them some of the time. The problem is bound to be what I described above.

[tomstuber]

I really appreciate the reply and I understand in theory what you are saying. I will need to do some more studying and experimentation to put it into practice.

Thank you again

[tomstuber]

You said not to use the color red in Paint because it isn't a true red that is expected in Autohotkey. Can you recommend a color that would be compatible.
I am also looking at the script FindText that I found in the forum. Do you have any experience with it?
Thank you again?

[boiler]

I usually try to use pure white or black as the Trans color, but that's not really an option in your case because the text you're trying to find is black and the background is white, so you need another color. The best thing to do is to use Edit Colors in Paint to make a special "pure" custom color where the value of red is 255 and green and blue are both 0. And when you paint with it, use the pencil tool because it paints pure pixels of the selected color, while the brushes will produced blended colors at the brush edges that will not work. Whatever color you use, even if you use one that's not pure such as FF0000 red, just make sure you match it with its RGB hex values in the *Trans option of the ImageSearch command.

I haven't done much with that FindText tool, but it may do exactly what you need for your case. I would think it's susceptible to not finding certain text if it's rendered differently because of anti-aliasing, but it's worth a try.

[tomstuber]

When I try to smooth the edges of a number I have clipped from the screen, my ImageSearch will not find it.I tried the color green. I chose the code 00FF00. I verified this was the color with Window spy. I chose just one digit from the screen and saved it into a file (Test1). I then brought the Test1 file into Paint and put a small line of green on part of one edge and saved it as Test2. I then ran put the following line into a loop and tested for each of the two images. The search finds the image Test1 but not Test2.
ImageSearch, Px, Py, 0, 0, A_ScreenWidth, A_ScreenHeight, *Trans0x00FF00 %ImageName%

I am not sure where to try next. I appreciate any suggestions.

Thank you in advance

[Masonjar13]

It appears this is for a website, correct? I would use ImageSearch only as a last resort. Primarily, I would suggest JavaScript injection, dropping AHK entirely. Alternatively, there are many ways to connect to browsers in AHK: IE COM, Chrome.ahk (library by GeekDude), Selenium.. you could even use a headless browser.

If it's a window that isn't a browser, consider using ControlClick/Send.

When I try to smooth the edges of a number I have clipped from the screen, my ImageSearch will not find it.

Modifying the image that's being searched, besides cropping and coloring for transcolor, almost entirely defeats the purpose. It's suppose to be identical. The problem is when the image isn't consistent, which is what seems to be alluded to here. You might try cutting off all edges of the text, replacing it with transcolor. As long as the distance between them is consistent, should be fine. Alternatively, if the distance also changes, you could search for each number individually in this way as well, but.. likely more hassle than it's worth.

Edit: another idea would be to use a browser extension, such as Stylish, to remove AA or any font styling that could be creating the issue.

[boiler]

When I try to smooth the edges of a number I have clipped from the screen

Based on your wording, I'm not sure you're painting the correct pixels green. You shouldn't be trying to smooth the edges of a number, you should be painting over all of those in-between colors that are (or possibly could be) other than pure black or the background grey color. All of the in-between colors should be painted over in green. And even paint over some of the grey and some of the black pixels so that what is left behind is sure to be black or background grey pixels no matter how the text is rendered in a particular instance. In fact you should paint way over the edges in green so there is no doubt that you have covered all of the pixels that could potentially change.

I suggest that you post one of the images as you've modified them so we can see how you've cropped and painted over them. I also suggest using an allowable variation of about *20 or *30 in your ImageSearch command because it may be that the number or background isn't completely exactly the same color for some reason.

[boiler]

Modifying the image that's being searched, besides cropping and coloring for transcolor, almost entirely defeats the purpose. It's suppose to be identical. The problem is when the image isn't consistent, which is what seems to be alluded to here. You might try cutting off all edges of the text, replacing it with transcolor.

This is exactly why he is modifying it. It's not identical because of anti-aliasing, so he has to modify it to find it. Modifying the image for the Trans option of ImageSearch is exactly what he should be doing here. It will work as long as he paints over the image in his Trans color adequately.

[tomstuber]

I tried shading various times to try to get ImageSearch to find the altered image but I couldn't get it to match. So, I simplified it to take a single digit from one of the times (Test1 attached). I then painted a small section along one edge of the digit (Test2 attached) to see if I could get ImageSearch to find it on the page. I have included the test code below where I loop through and it finds Test1 but not Test2. My understanding that any color that matches TransN is ignore. If that is correct, then it seems my small modification should be found. Thanks for the tip about putting in *30. You had mentioned that before and I forgot to add it. I did this time but to no avail.
Thanks again
ImageName := "Test1.png"
loop 2
{
ImageSearch, Px, Py, 0, 0, A_ScreenWidth, A_ScreenHeight, *30 *Trans0x00FF00 %ImageName%

;ImageSearch, Px, Py, 790, 440, 2400, 1300, %ImageName%
If ErrorLevel=0
{
If ErrorLevel=0
Px += 200
Py += 275
MouseMove, %Px%, %Py%
msgbox % ImageName . " found"
}
else If ErrorLevel=2
msgbox % "error level 2" . ImageName

else If ErrorLevel=1
msgbox, % ImageName . " not found"
ImageName := "Test2.png"
}
return

[Xtra]

Your image should look like this:

[boiler]

It can’t find the image if you’ve painted just one small section. The point of it is to cover up ALL of the anti-aliased edges of the text that appears in your reference image. Covering up one small portion of it has no chance of working since you’ve left most of the rest in place.

Xtra shows one way you can do it, although I would leave some of the background color around it as well because when all you have left are black pixels, that could be found wherever there is solid black or within another number. That all-black 1 could be found within the number 4, for example. I’ll post an image of what I’m describing a bit later.

[tomstuber]

Thanks for you help. I am trying to understand the logic. I knew what I was trying in this experiment wasn't going to solve the problem of my image not being recognized when it shows up on different places on the screen. I was testing to see if I was using the correct color. My understanding was that using ImageSearch with *Trans00FF00 anything with that shade of green would be ignored. So, when I put that small green line on my image I thought it would be ignored and since the rest of the image is the same and it would be found. Where is my logic amiss. Thanks again

[boiler]

Oh, I’m sorry. If you’re trying to find the exact same image, then you’re right that adding some green should work when that is the trans color. I’ll have some time later to look at it in more detail to see what else might be an issue. In the meantime, you might try searching for Xtra’s image.

[boiler]

I see what's wrong with your Test2.png image where you painted some green pixels. The problem is what I warned you about. You can't use the brush tool in MS Paint because it paints some non-pure pixels around the edges of your brush stroke. It does so to make the brush stroke look smoother to the naked eye. But since it's introducing some new colors, there are now colors that are neither in the original image nor the pure green pixels you added. So ImageSearch can't find that image. If you use MS Paint, you must use the pencil tool because it paints individual pure pixels of your chosen color, not a stroke of the pure color surrounded by some other colors to make it look smoother.

[tomstuber]

Ugh! How embarrassing. I used the brush for this test because I couldn't wanted to be sure I was getting true green. I was using Window Spy and I couldn't get the cursor over the correct pixels.
I ran the test tonight with the pencil and it worked fine.

Is there a better tool to use than Paint to change the non-black colors to another color (green in my case). I have quite a few times I may want to use, and I had difficulty outlining just the four digits of the one time I was using. The image that Xtra posted didn't appear to be hand drawn. Either that, or he has a steadier hand than I.

Thank you again for all your help. I was stumped

[boiler]

Well, that's good since we now have a good idea that it all should work. But don't worry about just painting over the non-black pixels. Like I was suggesting, you should not only paint over all the anti-aliased pixels (non-black and non-background grey) but over a good amount of the black and grey as well. In other words, you can and should sloppily paint over a wider path because the anti-aliasing could occur in a slightly different position in different instances, so you need to cover more than just what is indicated in your reference image.

In using MS Paint for this purpose, one thing that helps is to zoom in as much as possible. You should be able to use Photoshop, but you have to disable brush smoothing or it will be even worse in terms of creating non-pure pixels with its brushes. There probably is some other image manipulation program that would be well-suited for this purpose, but I haven't looked very hard for it yet.

You're right that Xtra didn't hand-draw the green on there. I happen to know that he wrote a script that replaces all pixels not of a certain color with another, so he replaced all non-black with green.