AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

Post your working scripts, libraries and tools
GavinPen
Posts: 22
Joined: 24 Oct 2017, 16:49
GitHub: GavinPen

AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

26 Jun 2018, 17:29

AHKCoordGrid - overlay grid of buttons to provide (some level of) mouseless screen navigation

(I finally got round to a solution for the question I posted last year at https://autohotkey.com/boards/viewtopic ... 17&t=38867)

When you press NumKeyEnter, this script builds then displays a 26*26 grid of buttons (actually AHK edit controls), labelled AA-ZZ, overlaying the whole screen.
You can then emulate a mouse-click (or a touchscreen 'touch') at any of those 676 locations by entering the coordinates, as displayed on each button, as keyboard shortcuts.
Pressing NumKeyEnter thereafter toggles the grid (it displays much faster after the first time, as it has already been built)

Inspired by the cVim chrome plugin, which provides shortcut keys to most links / buttons on web pages.
Includes code around timing of key presses which 'scriptor' (thanks!) posted here:
https://autohotkey.com/board/topic/1165 ... -question/

This is quite a blunt instrument:
- you won't always be able to click on everything you need to
- on any given screen, most buttons will probably not be useful
But as somebody who likes to avoid using the mouse, I already find it useful - I hope others will too.
You could even look at it as an accessibility tool.

There are a few things I'd like to improve:
- There's a trade-off between minimizing the size of the buttons so as not to obscure what's behind them, and keeping them big enough to read. This may take more tweaking
- I chose 'Edit' controls as buttons as their white background made them easy to read.
But this means they can be typed over.
- I might add coordinates 0-9 for screens with more pixels
- As most clicking activity tends to happen round the edges of the screen and not in the middle, I might look at reducing the row / column spacing round the edges only
- I've only tried this on 2 monitors. I might need to tweak the small values added and subtracted from the coordinates to ensure that the click is sent as close to the centre of each button as possible, and also the colSpacing constant.

I've attached a screenshot showing the grid overlaying a Notepad++ window, part of a Chrome window, and part of my desktop.

Code: Select all

#SingleInstance
DetectHiddenWindows, On
VerticalScale := 1
numberOfRows := 26
numberOfCols := 26
GridHeight := VerticalScale * A_ScreenHeight
GridWidth := A_ScreenWidth
rowSpacing := GridHeight / numberOfRows	
colSpacing := 1.025 * GridWidth / numberOfCols
AscA := 97
KeyArray := ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"]

#IfWinNotActive CoordGrid
	#IfWinNotExist CoordGrid
		global VerticalScale, numberOfRows, numberOfCols, GridHeight, GridWidth, rowSpacing, colSpacing, KeyArray

		NumpadEnter::
	 		; Display Coordinates 
			rowCounter := 0
			Loop {
				rowYCoord := (numberOfRows - 1 - rowCounter) * rowSpacing ;
				rowYCoordAlpha := KeyArray[rowCounter+1]
				StringUpper, rowYCoordAlpha, rowYCoordAlpha
				colCounter := 0
				Loop {
					colXCoord := colCounter * colSpacing
					colXCoordAlpha := KeyArray[colCounter+1]
					StringUpper, colXCoordAlpha, colXCoordAlpha
					; gui, add, edit, x%colXCoord% y%rowYCoord% ReadOnly cRed, %colXCoordAlpha%%rowYCoordAlpha%
					gui, add, edit, x%colXCoord% y%rowYCoord%, %colXCoordAlpha%%rowYCoordAlpha%
					colCounter := colCounter + 1
				}    
				Until colCounter = numberOfcols			
				rowCounter := rowCounter + 1
			}    
			Until rowCounter = numberOfRows		

			Gui, Color, 000115
			Gui, Show, W%GridWidth% H%GridHeight%, CoordGrid
			Gui -Caption +AlwaysOnTop
			Gui, maximize
			WinSet, Transcolor, 000115, CoordGrid
			
			Return
	#IfWinNotExist	
#IfWinNotActive	

#IfWinExist CoordGrid
	#IfWinNotActive CoordGrid
		NumpadEnter::
			Gui Show
			Return
	#IfWinNotActive	
#IfWinExist

#IfWinActive CoordGrid
	NumpadEnter::
		Gui Hide
		Return

	~a:: gosub, RunKey
	~b:: gosub, RunKey
	~c:: gosub, RunKey
	~d:: gosub, RunKey
	~e:: gosub, RunKey
	~f:: gosub, RunKey
	~g:: gosub, RunKey
	~h:: gosub, RunKey
	~i:: gosub, RunKey
	~j:: gosub, RunKey
	~k:: gosub, RunKey
	~l:: gosub, RunKey
	~m:: gosub, RunKey
	~n:: gosub, RunKey
	~o:: gosub, RunKey
	~p:: gosub, RunKey
	~q:: gosub, RunKey
	~r:: gosub, RunKey
	~s:: gosub, RunKey
	~t:: gosub, RunKey
	~u:: gosub, RunKey
	~v:: gosub, RunKey
	~w:: gosub, RunKey
	~x:: gosub, RunKey
	~y:: gosub, RunKey
	~z:: gosub, RunKey

	Runkey:
		if winc_presses > 0  
		{
		    winc_presses += 1
		    Return
		} 
		winc_presses = 1
		SetTimer, TheKey, 1000
		Return

	TheKey:
		SetTimer, TheKey, off
		if winc_presses = 2  
		{
			NavigateToCoord()
		}

		winc_presses = 0
		Return	

	NavigateToCoord()
	{
		CoordMode, Mouse, Window
		global VerticalScale, numberOfRows, numberOfCols, GridHeight, GridWidth, rowSpacing, colSpacing

		XCoordInput := SubStr(A_PriorHotkey,2,1)
		YCoordInput := SubStr(A_ThisHotkey,2,1)
		XCoordToUse := ConvertInputCoord(XcoordInput, "X")
		YCoordToUse := ConvertInputCoord(YcoordInput, "Y")

		XCoord := (XCoordToUse+0.2) * colSpacing
		YCoord := (YCoordToUse-0.7) * rowSpacing

		MouseMove, %XCoord%, %YCoord%
		sleep, 100
		Gui Hide
		Click
		Return
	}

	ConvertInputCoord(coordInput, XorY)
	{
		global AscA
		coordAsc := Asc(coordInput)
		if (XorY = "X") {
			coordToUse := coordAsc - AscA
		}
		else {
			coordToUse := numberOfRows - (coordAsc - AscA)
		}
		coordToUse := floor(coordToUse) 
		Return coordToUse
	}
#IfWinActive
Attachments
AHKCoordGrid example screenshot.png
AHKCoordGrid example screenshot.png (254.99 KiB) Viewed 2093 times
gwarble
Posts: 376
Joined: 30 Sep 2013, 15:01

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

27 Jun 2018, 18:53

Very cool idea, you could play with gdip to make a pretty and semi transparent version... Also to get higher precision, you could have a usage like:

Typing MM.MM would click the center of the center square, typing MM.AA would click the bottom left corner of MM
EitherMouse - Multiple mice, individual settings . . . . www.EitherMouse.com . . . . forum . . . .
GavinPen
Posts: 22
Joined: 24 Oct 2017, 16:49
GitHub: GavinPen

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

28 Jun 2018, 06:28

That's a good ides, @gwarble.

https://en.wikipedia.org/wiki/Display_r ... esolutions
says that nearly 30% of users use resolution 1366*768 = 1,049,088 pixels.
Using 4 alphabetic coordinates (AAAA-ZZZZ) would give 26*26*26*26 = 456,976 clickable points, which is not far off 1 for every 2 pixels at this resolution!

I could easily make the grid much more useful by instead just introducing a 3*3 numeric grid
(along the same lines as Dragon NaturallySpeaking's mousegrid, with a unique digit 1-9 assigned to each of the 9 cells)
say above and to the right of each existing point AA-ZZ.
Space congestion would mean that it wouldn't be practical to actually display these 9 coordinates.
It might be possible to display just the gridlines, but even if not, users could probably guess most of the time which of the 9 to use.

This would mean that you could choose from AA1 through ZZ9, giving 6084 (=26*26*9) clickable points - probably enough for most everyday use.
Hope to get round to this soon!
GavinPen
Posts: 22
Joined: 24 Oct 2017, 16:49
GitHub: GavinPen

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

21 Aug 2018, 09:00

referencing the attachment within img tags here so it can be viewed directly:

Image
User avatar
TheDewd
Posts: 1323
Joined: 19 Dec 2013, 11:16
Location: USA

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

22 Aug 2018, 13:37

Reminds me of the Blade Runner "Enhance" scene.

Combine this with speech recognition to navigate the screen.
Deckard wrote:Deckard: Enhance 224 to 176.
[a man's arm becomes visible]

Deckard: Enhance. Stop.
[the man's shoulder and wrist are visible]

Deckard: Move in. Stop.
[close-up of man's wrist]

Deckard: Pull out, track right. Stop.
[writing is visible]

Deckard: Center and pull back. Stop.
[arm and door are visible]

Deckard: Track 45 right. Stop. Center and stop.
[doorway and mirror are visible]

Deckard: Enhance 34 to 36.
[dresser top is visible]

Deckard: Pan right or-and pull back. Stop.
[mirror is visible]

Deckard: Enhance 34 to 46.
[blurred white object in mirror becomes visible]

Deckard: Pull back. Wait a minute. Go right. Stop.
[Zhora's arm becomes visible]

Deckard: Enhance 57 to 19. Track 45 left. Stop.
[Zhora is visible]

Deckard: Enhance 15 to 23.
[marks on Zhora's face become visible]

Deckard: Gimme a hard copy right there.
Image Bulldozer - Sokoban inspired game from 1994 recreated in AutoHotkey.
User avatar
Nextron
Posts: 1364
Joined: 01 Oct 2013, 08:23
Location: Netherlands OS: Win7 x64 AHK: Unicode x32

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

22 Aug 2018, 16:30

I could easily make the grid much more useful by instead just introducing a 3*3 numeric grid
(...)
This would mean that you could choose from AA1 through ZZ9, giving 6084 (=26*26*9) clickable points
Why not ditch the letters all together? Two letters gives about the same precision as three digits: 26^2=676 vs 9^3=729, with added benefit you can limit/add layers as needed. Sure its an additional button press, but the benefit of spatial similarity of the numpad layout probably makes up for in speed. You could even make depth variable. Display a level 1 9*9 grid in color X, within each of those a level 2 9*9 grid in color Y. As soon as you press the first digit, the non-relevant level 2 grids are hidden, but a third layer grid shows in each of the relevant level 2 grids in color Z, etc. If you hit 0 for example, it clicks the center of the selected grid, regardless of depth you are at. So clicking Start, would just be 1110. But activating a window in the top right of the screen would suffice with 90. Opposed to having to think, "right is Z, which is on the left of my keyboard, but, I don't want full right because of the close/minimize buttons, so I need U or something and then A for the top of the window which is in the middle left of the keyboard.
GavinPen
Posts: 22
Joined: 24 Oct 2017, 16:49
GitHub: GavinPen

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

04 Oct 2018, 06:34

@Nextron That does sound like a good alternative.

I've realized that cVim calls its dynamic shortcuts 'hints'
(as do the other extensions which provide similar ones, eg pentadactyl, vrome, vimperator, and vimium)

https://github.com/zsims/hunt-and-peck also looks useful - it provides similar hints for navigation in the currently active application on Windows.
User avatar
SpeedMaster
Posts: 324
Joined: 12 Nov 2016, 16:09

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

04 Feb 2019, 14:22

Hi GavinPen, I really like your script. :thumbup:
If you make a few small changes you can solve a lot of problems. :think:
For example you can use the cursor keys to position the grid before clicking.
For the visual rendering you can use a "progress" control along with a "text" control to have a text with a colored background.
It's an old trick that still works. That way anyone can choose the color they like... 8-)
GavinPen wrote:
26 Jun 2018, 17:29
- you won't always be able to click on everything you need to....
Modify this part of your script with this and you will be able to adjust the grid position with the cursors before choosing a target. :idea:

(lines 48 to 59)

Code: Select all

#IfWinExist CoordGrid
	#IfWinNotActive CoordGrid
		NumpadEnter::
			winmove , CoordGrid,,0,0
			Gui Show
			Return
	#IfWinNotActive	
#IfWinExist

#IfWinActive CoordGrid
	NumpadEnter::
		winmove , CoordGrid,,0,0
		Gui Hide
		Return
		
	left:: 
	WinGetPos ,  currentposX, currentposY,,, CoordGrid
	winmove, CoordGrid,, % currentposX-10
	return
	right:: 
	WinGetPos ,  currentposX, currentposY,,, CoordGrid
	winmove, CoordGrid,, % currentposX+10
	return
	Up:: 
	WinGetPos ,  currentposX, currentposY,,, CoordGrid
	winmove, CoordGrid,, , % currentposY-10
	return
	Down:: 
	WinGetPos ,  currentposX, currentposY,,, CoordGrid
	winmove, CoordGrid,, , % currentposY+10
	return	
GavinPen wrote:
26 Jun 2018, 17:29
- I chose 'Edit' controls as buttons as their white background made them easy to read.
But this means they can be typed over.
As I suggested above. Use the progress control trick to see a text with a colored background. :thumbup:
Replace the loop of your script by this one...

(lines 20 to 36)

Code: Select all

Loop {
				rowYCoord := (numberOfRows - 1 - rowCounter) * rowSpacing ;
				rowYCoordAlpha := KeyArray[rowCounter+1]
				StringUpper, rowYCoordAlpha, rowYCoordAlpha
				colCounter := 0
				Loop {
					colXCoord := colCounter * colSpacing
					colXCoordAlpha := KeyArray[colCounter+1]
					StringUpper, colXCoordAlpha, colXCoordAlpha
					;gui, font, bold
					Gui, Add, Progress, w21 h21 x%colXCoord% y%rowYCoord% BackgroundFFFFFF disabled
					gui, add, text, w21 h21 x%colXCoord% y%rowYCoord% border 0x201 readonly backgroundtrans cBlack, %colXCoordAlpha%%rowYCoordAlpha%
					colCounter := colCounter + 1
				}    
				Until colCounter = numberOfcols			
				rowCounter := rowCounter + 1
			}    
			Until rowCounter = numberOfRows
Cheers
Last edited by SpeedMaster on 06 Mar 2019, 16:41, edited 1 time in total.
GavinPen
Posts: 22
Joined: 24 Oct 2017, 16:49
GitHub: GavinPen

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

02 Mar 2019, 17:06

Thanks @SpeedMaster, the script is much more useful now!
User avatar
DataLife
Posts: 321
Joined: 29 Sep 2013, 19:52

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

03 Mar 2019, 13:30

This is really cool. I added SpeedMasters suggestions on moving the grid before clicking. Now you can click on everything on the screen.

I would like to make a couple changes, but due to time restraints at the moment I can not.

Change 1...Not every keyboard has NumPadEnter. Need different hotkey to open the grid.

Change 2...Some dropdown menus drop down when the mouse is hovering over the dropdown then you click on something in the dropdown. Using this script it moves the cursor and clicks. This causes the dropdown to open and close due to the click. I think that when pressing the 2 letter button the script could look for a modifier key and only move the mouse and not click if the modifier key is pressed may work. Then the user would press the 2 letter button to click on an item in the dropdown.

Here is an example of this type of dropdown list. https://fritzsadventure.com/
At this webpage the click does not cause the dropdown to go away but the clicking on a dropdown item does not work.

thanks
DataLife
Check out my scripts. (MyIpChanger) (ClipBoard Manager) (SavePictureAs)
All my scripts are tested on Windows 10, AutoHotkey 32 bit Ansi unless otherwise stated.
GavinPen
Posts: 22
Joined: 24 Oct 2017, 16:49
GitHub: GavinPen

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

05 Mar 2019, 17:47

@DataLife
Change 1 is straightforward
Change 2 not so much! But I don't tend to use the grid in my browser, because cVim (in Chrome) is more convenient and allows you to display shortcuts (hints) for most of the clickable elements on webpages.
I tried cVim on https://fritzsadventure.com/ and it does work - though the hints appear well above each dropdown
User avatar
SpeedMaster
Posts: 324
Joined: 12 Nov 2016, 16:09

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

06 Mar 2019, 16:49

GavinPen wrote:
05 Mar 2019, 17:47
Change 1 is straightforward
Change 2 not so much!
Maybe you don't like colored gui. :?
I changed my above script respect to your theme (black and white). ;)
User avatar
SpeedMaster
Posts: 324
Joined: 12 Nov 2016, 16:09

Re: AHKCoordGrid - overlay grid of buttons to provide (a degree of) mouseless screen navigation

06 Mar 2019, 16:54

DataLife wrote:
03 Mar 2019, 13:30
Here is an example of this type of dropdown list. https://fritzsadventure.com/
At this webpage the click does not cause the dropdown to go away but the cli
You made a good point. :shock:
I hadn't noticed this issue. :)
Find this line Gui -Caption +AlwaysOnTop and replaces it with this one Gui -Caption +AlwaysOnTop +E0x20 to fix the problem

Cheers

Return to “Scripts and Functions”

Who is online

Users browsing this forum: Frosti, panex0845, robodesign and 46 guests