How to find "similar" images on Screen Topic is solved

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
rc76
Posts: 144
Joined: 07 Nov 2020, 01:45

How to find "similar" images on Screen

Post by rc76 » 17 Sep 2021, 13:15

I have a sort of different problem scenario would like to seek your help.

I have a program that will generate several types of notifications.
- Each has different background colors
- Each has symbols (i.e. !, X, ?)

Is it possible to:
(1) Capture any type of notifications (either Yellow or Pink background ones) that appears on screen, for any type of contents in the notification (i.e. images are not exactly the same every time).
(2) Recognize the content of the image (i.e. text recognition or find text).

Any hint on the direction how I may achieve the above will be super appreciated. Is there any existing function can achieve it, as ImageSearch needs exact match…
Last edited by rc76 on 24 Sep 2021, 00:17, edited 2 times in total.
User avatar
mikeyww
Posts: 26437
Joined: 09 Sep 2014, 18:38

Re: How to find "similar" images on Screen

Post by mikeyww » 17 Sep 2021, 14:07

If the yellow or red icon is common to them, you can always ImageSearch for them. If it is a window, then a shell hook might be able to detect it. There are some OCR scripts such as Vis2 (and a few others) posted on the forum. Some of them work reasonably well. If the window has text instead of an image, then you can try WinGetText.
rc76
Posts: 144
Joined: 07 Nov 2020, 01:45

Re: How to find "similar" images on Screen

Post by rc76 » 18 Sep 2021, 20:50

Thank you so much @mikeyww !

A quick clarification before I jump into it. After I use imagesearch to identify the icons for the yellow/red text, can I than determine the boundary of the yellow / red textbox?

Because once I determined the yellow/red textbox coordinates, I can than perform OCR on the textbox area to retrieve text?

Is my above understanding correct?
User avatar
mikeyww
Posts: 26437
Joined: 09 Sep 2014, 18:38

Re: How to find "similar" images on Screen

Post by mikeyww » 18 Sep 2021, 21:24

That sounds correct. FindText is another popular script on the forum.

A couple of ideas:

You could use a specified x- and y-distance from the icon to define your region of interest.
Or: You could do a sort of loop and PixelGetColor, to find the boundaries of the colored fill.

The first method is probably faster. Another method could be to look for an image representing just the corner of the box. That is likely to be fast. An example is below.

corner.png
corner.png (277 Bytes) Viewed 2224 times
User avatar
Xtra
Posts: 2744
Joined: 02 Oct 2015, 12:15

Re: How to find "similar" images on Screen

Post by Xtra » 18 Sep 2021, 22:05

With your error notification on screen use the windowspy tool that comes with AHK and mouse over the notification. Capture the windowspy window and post it here. This will help narrow down your options.
User avatar
mikeyww
Posts: 26437
Joined: 09 Sep 2014, 18:38

Re: How to find "similar" images on Screen

Post by mikeyww » 19 Sep 2021, 05:29

A good idea-- as perhaps you can just identify the window and its dimensions directly.
rc76
Posts: 144
Joined: 07 Nov 2020, 01:45

Re: How to find "similar" images on Screen

Post by rc76 » 19 Sep 2021, 11:16

Thank you so much @mikeyww and @Xtra

I have tried the Window Spy and I have noticed a few things:

(1) It uses ahk_exe MessageCenter.exe
(2) The list of error messages pop out along y-axis, with x=2700.
(3) The size of the error message window can vary based on length of the message itself. But I believe the x=2700 will stay the same no matter how lengthy the message will be. It will always just making the message box bigger by changing height (y-axis) of the box, than width (x-axis).

So:

(1) Maybe we can just scan for the features with x>=2700, y=0 to y=max?
(2) But than the box dimension height (y-axis) still cannot be predetermined with Window Spy. How may we solve this part?

Also about the behaviour of the error message itself:

(1) There can be multiple error messages, each with different contents and length, pop-ing out at the same time, from bottom of the window fill all the way up to the top of the window Desktop.
Last edited by rc76 on 24 Sep 2021, 00:18, edited 1 time in total.
AHKStudent
Posts: 1472
Joined: 05 May 2018, 12:23

Re: How to find "similar" images on Screen

Post by AHKStudent » 19 Sep 2021, 11:45

are these error messages posted to a local log file?
rc76
Posts: 144
Joined: 07 Nov 2020, 01:45

Re: How to find "similar" images on Screen

Post by rc76 » 19 Sep 2021, 12:24

@AHKStudent Good point. I tried to look for it, seems not saved to any log file for these alert messages.
rc76
Posts: 144
Joined: 07 Nov 2020, 01:45

Re: How to find "similar" images on Screen

Post by rc76 » 22 Sep 2021, 19:24

@mikeyww, @Xtra

Any feedbacks/guidance/recommendations on the above will be so great, thank you!
User avatar
mikeyww
Posts: 26437
Joined: 09 Sep 2014, 18:38

Re: How to find "similar" images on Screen

Post by mikeyww » 22 Sep 2021, 20:16

WinWait can wait for a window to exist. How do you know that the window contains no text? This would seem odd, since the messages in the windows vary.

Since you have the class, process name, and dimensions of the window, if the windows contain no text, you could then use one of the OCR routines.
User avatar
Xtra
Posts: 2744
Joined: 02 Oct 2015, 12:15

Re: How to find "similar" images on Screen

Post by Xtra » 22 Sep 2021, 20:49

viewtopic.php?t=76103&start=20#p331863
viewtopic.php?t=41273#p188087

You should post in the 1st link asking if getting the text is possible with their method.
It would also be worth checking if Acc can get the text.

If none of the above pan out you could take a look at this newly posted OCR
viewtopic.php?f=6&t=94856#p421258
rc76
Posts: 144
Joined: 07 Nov 2020, 01:45

Re: How to find "similar" images on Screen  Topic is solved

Post by rc76 » 23 Sep 2021, 00:07

Thank you so much @mikeyww , @Xtra

viewtopic.php?f=6&t=94856#p421258
The new OCR seems very powerful, will definitely give it a try!
Post Reply

Return to “Ask for Help (v1)”