How to automate arrow buttons overlaid on different pictures?

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
newcod3r
Posts: 505
Joined: 30 Sep 2021, 02:16

How to automate arrow buttons overlaid on different pictures?

Post by newcod3r » 26 Jun 2022, 17:10

Trying to assign keyboard shortcuts on instagram, so that I can scroll through in-post photos (i.e. 1 & 2), instead of through posts only (3 & 4). How do I automate 1 & 2 since the arrow buttons are overlaid on different images AND also have different sizes depending on how big the original image is?

5 is the indicator which shows whether there are multiple images to scroll through within 1 instagram post.S
image.png
image.png (257.12 KiB) Viewed 949 times

User avatar
mikeyww
Posts: 26600
Joined: 09 Sep 2014, 18:38

Re: How to automate arrow buttons overlaid on different pictures?

Post by mikeyww » 26 Jun 2022, 17:21

You might be able to look for the edge of the circle, trying a few different colors, or possibly just look for the colors in the circles.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: How to automate arrow buttons overlaid on different pictures?

Post by Descolada » 27 Jun 2022, 02:32

My attempt at this using UIAutomation library. Requires Instagram to be in English (or get the name of the buttons with UIAViewer or Accessibility Insights, and change variables nextButtonName and prevButtonName accordingly), and seems to work smoother in Edge.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

global UIA := UIA_Interface(), prevHwnd, TW_Next, TW_Prev, nextButtonName := "Next", prevButtonName := "Go Back"

TW_Next := UIA.CreateTreeWalker(UIA.CreateAndCondition(UIA.CreateCondition("ControlType", "Button"), UIA.CreateCondition("Name", nextButtonName)))
TW_Prev := UIA.CreateTreeWalker(UIA.CreateAndCondition(UIA.CreateCondition("ControlType", "Button"), UIA.CreateCondition("Name", prevButtonName)))

#IfWinActive Instagram	
$Right::
$Left::
	if (prevHwnd != (newHwnd := WinActive("Instagram")))
		window := UIA.ElementFromHandle(prevHwnd := newHwnd, True)
	if ((UIA.GetFocusedElement().CurrentControlType != 50004) && (but := (InStr(A_ThisHotkey, "Right") ? TW_Next : TW_Prev).GetLastChildElement(window)))
		but.Click()
	else
		Send, % "{" (InStr(A_ThisHotkey, "Right") ? "Right" : "Left") "}"
	return
#If

~F1::ExitApp

newcod3r
Posts: 505
Joined: 30 Sep 2021, 02:16

Re: How to automate arrow buttons overlaid on different pictures?

Post by newcod3r » 28 Jun 2022, 16:56

Descolada wrote:
27 Jun 2022, 02:32
My attempt at this using UIAutomation library. Requires Instagram to be in English (or get the name of the buttons with UIAViewer or Accessibility Insights, and change variables nextButtonName and prevButtonName accordingly), and seems to work smoother in Edge.

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

global UIA := UIA_Interface(), prevHwnd, TW_Next, TW_Prev, nextButtonName := "Next", prevButtonName := "Go Back"

TW_Next := UIA.CreateTreeWalker(UIA.CreateAndCondition(UIA.CreateCondition("ControlType", "Button"), UIA.CreateCondition("Name", nextButtonName)))
TW_Prev := UIA.CreateTreeWalker(UIA.CreateAndCondition(UIA.CreateCondition("ControlType", "Button"), UIA.CreateCondition("Name", prevButtonName)))

#IfWinActive Instagram	
$Right::
$Left::
	if (prevHwnd != (newHwnd := WinActive("Instagram")))
		window := UIA.ElementFromHandle(prevHwnd := newHwnd, True)
	if ((UIA.GetFocusedElement().CurrentControlType != 50004) && (but := (InStr(A_ThisHotkey, "Right") ? TW_Next : TW_Prev).GetLastChildElement(window)))
		but.Click()
	else
		Send, % "{" (InStr(A_ThisHotkey, "Right") ? "Right" : "Left") "}"
	return
#If

~F1::ExitApp
Thanks! I just checked using UIViewer and the button names are correct. However nothing seems to happen after using the hotkeys, no errors as well. tested on a fresh script as is (with the UIAInterface.ahk directory updated)
image.png
image.png (71.3 KiB) Viewed 743 times

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: How to automate arrow buttons overlaid on different pictures?

Post by Descolada » 29 Jun 2022, 02:33

Interesting... I tried it on three different computers and it worked everywhere. So lets try some debugging: first, the hotkey only works if the window is active, so are you sure Instagram is active?
If yes, try running this with a Instagram window open where there are some Next and Go Back buttons available:

Code: Select all

#SingleInstance, force
#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetTitleMatchMode, 2

#include <UIA_Interface>

global UIA := UIA_Interface(), prevHwnd, TW_Next, TW_Prev, nextButtonName := "Next", prevButtonName := "Go Back"

instagramHwnd := WinExist("Instagram")

TW_Next := UIA.CreateTreeWalker(UIA.CreateAndCondition(UIA.CreateCondition("ControlType", "Button"), UIA.CreateCondition("Name", nextButtonName)))
TW_Prev := UIA.CreateTreeWalker(UIA.CreateAndCondition(UIA.CreateCondition("ControlType", "Button"), UIA.CreateCondition("Name", prevButtonName)))

WinActivate, ahk_id %instagramHwnd%
WinWaitActive, ahk_id %instagramHwnd%
window := UIA.ElementFromHandle(instagramHwnd, True)

nextEl := TW_Next.GetLastChildElement(window)
MsgBox, % "Found: " nextEl.Dump() "`n`nPress OK to display found ""Next"" button"
WinActivate, ahk_id %instagramHwnd%
br := nextEl.CurrentBoundingRectangle
RangeTip(br.l, br.t, br.r-br.l, br.b-br.t)

MsgBox, % "Press OK to try clicking ""Next"""
nextEl.Click()
Sleep, 1000
RangeTip()

prevEl := TW_Prev.GetLastChildElement(window)
MsgBox, % "Found: " prevEl.Dump() "`n`nPress OK to display found ""Go Back"" button"
WinActivate, ahk_id %instagramHwnd%
br := prevEl.CurrentBoundingRectangle
RangeTip(br.l, br.t, br.r-br.l, br.b-br.t)

MsgBox, % "Press OK to try clicking ""Go Back"""
prevEl.Click()
Sleep, 1000
RangeTip()

MsgBox, Press OK to Exit
ExitApp

RangeTip(x:="", y:="", w:="", h:="", color:="Red", d:=2) ; from the FindText library, credit goes to feiyue
{
  local
  static id:=0
  if (x="")
  {
    id:=0
    Loop 4
      Gui, Range_%A_Index%: Destroy
    return
  }
  if (!id)
  {
    Loop 4
      Gui, Range_%A_Index%: +Hwndid +AlwaysOnTop -Caption +ToolWindow
        -DPIScale +E0x08000000
  }
  x:=Floor(x), y:=Floor(y), w:=Floor(w), h:=Floor(h), d:=Floor(d)
  Loop 4
  {
    i:=A_Index
    , x1:=(i=2 ? x+w : x-d)
    , y1:=(i=3 ? y+h : y-d)
    , w1:=(i=1 or i=3 ? w+2*d : d)
    , h1:=(i=2 or i=4 ? h+2*d : d)
    Gui, Range_%i%: Color, %color%
    Gui, Range_%i%: Show, NA x%x1% y%y1% w%w1% h%h1%
  }
}
It should go through every step one-by-one and might give a hint as to where the problem lies.

newcod3r
Posts: 505
Joined: 30 Sep 2021, 02:16

Re: How to automate arrow buttons overlaid on different pictures?

Post by newcod3r » 29 Jun 2022, 03:33

I tried again on this link and msgbox says Found, yet it did not move through the images...
https://www.instagram.com/p/CfW4IKgpk-Z/?hl=en
image.png
image.png (620.64 KiB) Viewed 637 times
image.png
image.png (65.09 KiB) Viewed 637 times

newcod3r
Posts: 505
Joined: 30 Sep 2021, 02:16

Re: How to automate arrow buttons overlaid on different pictures?

Post by newcod3r » 29 Jun 2022, 03:36

mikeyww wrote:
26 Jun 2022, 17:21
You might be able to look for the edge of the circle, trying a few different colors, or possibly just look for the colors in the circles.
the colors are different, but the shape is very similar to the bigger navigation buttons on the extreme corners.. hence the issue

User avatar
mikeyww
Posts: 26600
Joined: 09 Sep 2014, 18:38

Re: How to automate arrow buttons overlaid on different pictures?

Post by mikeyww » 29 Jun 2022, 04:53

The search can exclude areas by coordinates.

Descolada
Posts: 1099
Joined: 23 Dec 2021, 02:30

Re: How to automate arrow buttons overlaid on different pictures?

Post by Descolada » 29 Jun 2022, 05:20

@newcod3r, please post an image of the first MsgBox shown. The posted Instagram page doesn't contain a Go Back button, so "Found:" was empty. The first MsgBox should contain this:

Code: Select all

Found: Type: 50000 Name: "Next" LocalizedControlType: "button"

Press OK to display found "Next" button
If it doesn't (displays just "Found:" without the Type and Name after it), then the element wasn't found. Since UIAViewer found the correct ControlType and Name, then check again that your UIA_Interface.ahk is uptodate (is an older version getting loaded from another Lib folder perhaps?).

Post Reply

Return to “Ask for Help (v1)”