Click on specific area with variable distance from reference image? Topic is solved

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Click on specific area with variable distance from reference image?

03 Aug 2021, 17:12

Taking these images as example, I want to click on the centre of ANY these numbers (E.g. 237, -1.12%, B:237).

However,
1. the numbers can be different colors and different numbers
2. the company name can be very long or short

How do I click on it reliably then? P.S. there is no other spot (including top and bottom) I can click on to activate my function.
image.png
(6.1 KiB) Downloaded 275 times
image.png
(7.7 KiB) Downloaded 275 times
image.png
(7.28 KiB) Downloaded 275 times
Last edited by milkygirl90 on 03 Aug 2021, 17:13, edited 1 time in total.
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Re: Click on specific area with variable distance from reference image?

03 Aug 2021, 17:13

image.png
(7.47 KiB) Downloaded 274 times
image.png
(7.98 KiB) Downloaded 274 times
User avatar
boiler
Posts: 16767
Joined: 21 Dec 2014, 02:44

Re: Click on specific area with variable distance from reference image?

03 Aug 2021, 17:18

The little vertical bars separating the number fields look like reliably repeatable reference points. Find the two of them surrounding the numbers you want to click on and click on the midway point.
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Re: Click on specific area with variable distance from reference image?

03 Aug 2021, 21:53

boiler wrote:
03 Aug 2021, 17:18
The little vertical bars separating the number fields look like reliably repeatable reference points. Find the two of them surrounding the numbers you want to click on and click on the midway point.
I'm using Findclick here. Few questions:

1. How do I get Findclick to ignore the numbers and colors in the middle, so that I can target the middle of the 2 vertical lines?
1b. The distance between both vertical lines can vary too since the number can be 10.2, 3990.17 etc.
2. If this can't be done, then it may not be possible since there are many similar vertical lines with no distinct shapes around..
User avatar
boiler
Posts: 16767
Joined: 21 Dec 2014, 02:44

Re: Click on specific area with variable distance from reference image?

03 Aug 2021, 22:15

milkygirl90 wrote: 1. How do I get Findclick to ignore the numbers and colors in the middle, so that I can target the middle of the 2 vertical lines?
I'm not sure what you mean by ignoring the numbers in the middle. It has nothing to do with the numbers. You find the location of one vertical line, then you find the location of the next vertical line, then you click halfway in between them in the x direction. The y direction would just be part way down the vertical line, like halfway or whatever.

milkygirl90 wrote: 1b. The distance between both vertical lines can vary too since the number can be 10.2, 3990.17 etc.
Why is that a problem? Find both and take the average of the found x values to click halfway between them. If the numbers aren't always spanning the center point, then click on a point a little left of center.

milkygirl90 wrote: 2. If this can't be done, then it may not be possible since there are many similar vertical lines with no distinct shapes around..
I doubt that's the case. What other vertical line would be found if you took the image below as the reference image (open it in MS Paint and zoom in to see the details), with that exact length, that blue color for the line itself, that other blue color on either side of it, yet another blue color above and below it, and that brown color above and below that? It looks like there are a certain number of those lines, and you would know which ones you want to click between, like halfway between the third and the fourth vertical line or whatever.
vertical line.png
(119 Bytes) Downloaded 215 times
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Re: Click on specific area with variable distance from reference image?

03 Aug 2021, 22:48

I guess my question is: how do I take in Two reference images (which look the same) using findclick, and then divide by 2?

sample here:
image.png
(157 Bytes) Downloaded 199 times

Code: Select all

FindClick("E:\Screenshots\AutoHotkey_2021-09-13_18-57-42.png", "x30 o40")
these 4 lines are identical:
image.png
(23.59 KiB) Downloaded 198 times
Attachments
image.png
(157 Bytes) Downloaded 199 times
User avatar
boiler
Posts: 16767
Joined: 21 Dec 2014, 02:44

Re: Click on specific area with variable distance from reference image?  Topic is solved

03 Aug 2021, 23:05

You don't use different reference images. You find it once, then based on that, you search again for another one with the search area adjusted so you don't find the same one again, etc.

Alternatively, you could use this function to return all the instances of it within the search area, then you can use the 3rd and 4th results (or whatever) from the array that is returned:

Code: Select all

ImageSearchAll(imageFile, x1 := 0, y1 := 0, x2 := "Screen", y2 := "Screen", var := 0) {
	; found coordinates are returned as a simple array of coordinate pairs
	; each coordinate pair is an associative array with keys "x" and "y"

	x2 := x2 = "Screen" ? A_ScreenWidth : x2
	y2 := y2 = "Screen" ? A_ScreenHeight : y2
	found := []
	y := y1
	loop {
		x := x1
		loop {
			ImageSearch, foundX, foundY, x, y, x2, y2, % "*" var " " imageFile
			if (ErrorLevel = 2)
				return -1
			if !ErrorLevel {
				found.Push({x: foundX, y: foundY})
				x := foundX + 1
				lastFoundY := foundY
			}
		} until ErrorLevel
		Y := lastFoundY + 1
	} until (x = x1) && ErrorLevel
	return found
}

See the comments for how to use the returned value. For example, if Lines := ImageSearchAll(...) found 4 lines, then the results would be in Lines.1, Lines.2, Lines.3, and Lines.4. The x values of lines 3 and 4 would be Lines.3.x and Lines.4.x, so the halfway point would be (Lines.3.x + Lines.4.x) / 2.

By the way, I wouldn't use the image you attached as the reference image. There is too much blue space around it, which could cause it not to be found in each case. If you zoom in on the one I attached, it captures the same colors, but just wide enough to capture all the colors. That's how your reference images should always be. As small as possible that still ensures it's uniquely identifying the image of interest.
User avatar
boiler
Posts: 16767
Joined: 21 Dec 2014, 02:44

Re: Click on specific area with variable distance from reference image?

03 Aug 2021, 23:10

(corrected a couple things in the above post, so re-read it after seeing this message)
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Re: Click on specific area with variable distance from reference image?

03 Aug 2021, 23:43

This is quite complex for me.. I can't really understand most of the lines.

Tried adding these but it didn't do anything for me..

Code: Select all

	imageFile := "E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png"
		ImageSearchAll(imageFile, x1 := 0, y1 := 0, x2 := "Screen", y2 := "Screen", var := 0) {
	; found coordinates are returned as a simple array of coordinate pairs
	; each coordinate pair is an associative array with keys "x" and "y"

	x2 := x2 = "Screen" ? A_ScreenWidth : x2
	y2 := y2 = "Screen" ? A_ScreenHeight : y2
	found := []
	y := y1
	loop {
		x := x1
		loop {
			ImageSearch, foundX, foundY, x, y, x2, y2, % "*" var " " imageFile
			if (ErrorLevel = 2)
				return -1
			if !ErrorLevel {
				found.Push({x: foundX, y: foundY})
				x := foundX + 1
				lastFoundY := foundY
			}
		} until ErrorLevel
		Y := lastFoundY + 1
	} until (x = x1) && ErrorLevel
	return found
}
MidX := (Found.1.x + Found.2.x)/2
MidY := (Found.1.y + Found.2.y)/2
Click, MidX, MidY, Right
return 
User avatar
boiler
Posts: 16767
Joined: 21 Dec 2014, 02:44

Re: Click on specific area with variable distance from reference image?

04 Aug 2021, 05:34

You don't need to understand any of the lines, and it's not meant for you to work with the lines within it, just like you don't need to know how FindClick() works. You just use it. It's a function. You put the whole thing in your code (like at the bottom, out of the way or in a separate file that you #Include), and you call it and use the returned result like any other function. Example:

Code: Select all

Lines := ImageSearchAll("E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png", 0, 0, A_ScreenWidth, A_ScreenHeight, 0)
...and the above case of searching the whole screen with 0 variation is the default, so the call can just be:

Code: Select all

Lines := ImageSearchAll("E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png")
Then as I mentioned, the result of the above call would make Lines an array with each found instance. The location of the first line would be Lines.1.x and Lines.1.y. And the x position halfway between the third and fourth line would be (Lines.3.x + Lines.4.x) / 2.

I don't think you want to search the whole screen at once, though because you have lines in several places on the screen. I would think you'd want to search one area of the screen at a time, like for the stock or fund of interest.
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Re: Click on specific area with variable distance from reference image?

04 Aug 2021, 08:23

boiler wrote:
04 Aug 2021, 05:34
You don't need to understand any of the lines, and it's not meant for you to work with the lines within it, just like you don't need to know how FindClick() works. You just use it. It's a function. You put the whole thing in your code (like at the bottom, out of the way or in a separate file that you #Include), and you call it and use the returned result like any other function. Example:

Code: Select all

Lines := ImageSearchAll("E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png", 0, 0, A_ScreenWidth, A_ScreenHeight, 0)
...and the above case of searching the whole screen with 0 variation is the default, so the call can just be:

Code: Select all

Lines := ImageSearchAll("E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png")
Then as I mentioned, the result of the above call would make Lines an array with each found instance. The location of the first line would be Lines.1.x and Lines.1.y. And the x position halfway between the third and fourth line would be (Lines.3.x + Lines.4.x) / 2.

I don't think you want to search the whole screen at once, though because you have lines in several places on the screen. I would think you'd want to search one area of the screen at a time, like for the stock or fund of interest.
Please pardon my silly questions again:

Say the red arrow is the area I want to click:
Image
Image



So I will write this?

Code: Select all

Lines := ImageSearchAll("E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png", 100, 100, 300, 200, 0) ; 100,100 is top left region I want to search and I restrict it to 300,200 by a rectangle 
Click, Lines, Right
return

;function copied below
User avatar
boiler
Posts: 16767
Joined: 21 Dec 2014, 02:44

Re: Click on specific area with variable distance from reference image?

04 Aug 2021, 08:44

I don't understand what you mean by "100,100 is top left region I want to search and I restrict it to 300,200 by a rectangle." How does a single coordinate 100,100 define a region? And what do you mean by restricting it to 300,200 by a rectangle? By using the rectangle defined by 100, 100, 300, 200 as you did in your function call, you would be looking for your reference image to be found somewhere within the screen area bound by the upper-left coordinate 100,100 and the lower-right coordinate of 300,200. Your reference image must be found wholly within that area of the screen.

And no, that's not how you would click on it. As I described, the results are in an array, and each element of that array has a two-element array with x and y keys. Assuming the result is the first or only result (remember, this function is meant for when multiple instances are expected to be found, or else you might as well use regular ImageSearch), the way you would click on it with the right mouse button would be one of the following:

Using regular Click, it can look complex because you have to force an expression since Click really only has one parameter that gets parsed by the interpreter even though the various parts can be separated by commas (don't worry if you don't know what that means):

Code: Select all

Lines := ImageSearchAll("E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png", 100, 100, 300, 200, 0)
Click, % Lines.1.x "," Lines.1.y " " Right
It may be easier to understand using MouseClick since it has separate parameters for x and y that can be expressions:

Code: Select all

Lines := ImageSearchAll("E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png", 100, 100, 300, 200, 0)
MouseClick, Right, Lines.1.x, Lines.1.y
Using built-in ImageSearch, which is what you would do when you only expect one result or when you want to use the first instance that would be found:

Code: Select all

ImageSearch, LineX, LineY, 100, 100, 300, 200, E:\Dropbox\Screenshots\AutoHotkey_2021-08-04_12-10-21.png
Click, %LineX%, %LineY%, Right
None of the above include checking that something was actually found before attempting to click on it, but that's easy to add.
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Re: Click on specific area with variable distance from reference image?

04 Aug 2021, 16:44

I finally got it working! thank you so much!

Just 2 last questions - is there a performance difference between Click vs Mouseclick?

How should I write the proper syntax with Click? Here's mine for MouseClick:

Code: Select all

MouseClick, Right, (Lines.1.x + Lines.2.x) / 2, (Lines.1.y + Lines.2.y) / 2 + 10
Also, is there a modifier to ignore colors (i.e. just search for the pattern) in this function?
User avatar
boiler
Posts: 16767
Joined: 21 Dec 2014, 02:44

Re: Click on specific area with variable distance from reference image?

04 Aug 2021, 17:18

milkygirl90 wrote: is there a performance difference between Click vs Mouseclick?
No. The documentation says that using Click is preferred over MouseClick because it will handle if the user has swapped left and right mouse buttons in their Windows settings, so if that's not an issue for you, use either one.


milkygirl90 wrote: How should I write the proper syntax with Click? Here's mine for MouseClick:

Code: Select all

MouseClick, Right, (Lines.1.x + Lines.2.x) / 2, (Lines.1.y + Lines.2.y) / 2 + 10
To use expressions with click, you force an expression for its one parameter that is expressed in several parts, and you simulate what the legacy syntax would look like if you put a number in place of your expression (I realize that's not easy to understand). In this case, it would be like this:

Code: Select all

Click, % ((Lines.1.x + Lines.2.x) / 2) . "," .  ((Lines.1.y + Lines.2.y) / 2 + 10) . ", Right"
By the way, I would think that the y position of the pair of lines is the same, so you don't have to average them because the average of two equivalent numbers is simply either of those numbers. So it can be:

Code: Select all

Click, % ((Lines.1.x + Lines.2.x) / 2) . "," .  (Lines.1.y + 10) . ", Right"


milkygirl90 wrote: Also, is there a modifier to ignore colors (i.e. just search for the pattern) in this function?
The function makes use of ImageSearch, and there is nothing in ImageSearch that has it ignore all colors and just looks for a pattern. You can have it ignore a particular color (i.e., match a pixel of that color no matter what color it is using the *TransN option (e.g., *TransBlack to ignore all black pixels in the reference image), but that doesn't sound like what you want to do.
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Re: Click on specific area with variable distance from reference image?

05 Aug 2021, 17:07

Thank you for patiently answering my queries. Appreciate it!
User avatar
milkygirl90
Posts: 565
Joined: 10 Nov 2020, 21:22

Re: Click on specific area with variable distance from reference image?

15 Aug 2021, 22:05

Hi @boiler, I've been testing this script for awhile and realised recently there are plenty of false positives. It clicks on colored lines even though the reference image is simply a grey line on a darker grey background.

How do I improve the script by defining the absolute search area as the top 20% (screen height) of the screen only?
User avatar
boiler
Posts: 16767
Joined: 21 Dec 2014, 02:44

Re: Click on specific area with variable distance from reference image?

16 Aug 2021, 06:50

The only way it would be clicking on different colors is if you are allowing a variation in the color values. Reduce that allowable variation to 0 and it won’t do that.

To search the top 20% of the screen, your search rectangle would be defined by:
0, 0, A_ScreenWidth, 0.2 * A_ScreenHeight

Return to “Ask for Help (v1)”

Who is online

Users browsing this forum: Descolada, Google [Bot] and 89 guests