Ascertaining Names of Buttons in the System

Alexander2 · 23 Jun 2021, 09:23

Window Spy does not differentiate between the different buttons in the system’s settings. The buttons are in the same category: Windows.UI.Core.CoreWindow1. Is it possible to ascertain the names of the buttons to be able to click them through the Control Click command?

mikeyww · 23 Jun 2021, 09:38

Consider that you could do this by Run instead. It might be more reliable.

https://classjoo.wordpress.com/2016/12/30/the-windows-control-panel-commands/

WalkerOfTheDay · 23 Jun 2021, 09:53

If you really want to push the buttons, I think you might need to resort to doing ImageSearches

Alexander2 · 23 Jun 2021, 10:41

Thank you. I mentioned the Settings buttons as an example in order to learn how to ascertain the names of any buttons. There are a lot of buttons in the system menus and various programs. Is it possible to ascertain the names of buttons and automatically click on them instead of doing image searches, which are less reliable and slower?

doubledave22 · 23 Jun 2021, 11:30

The ahk ACC library seems good for this task.
https://www.autohotkey.com/boards/viewtopic.php?t=26201

after including the ACC library run this function to evaluate a window to see if the elements you are interested in are found
https://www.autohotkey.com/boards/viewtopic.php?f=6&t=40615

Code: Select all


JEE_StrRept(vText, vNum)
{
	if (vNum <= 0)
		return
	return StrReplace(Format("{:" vNum "}", ""), " ", vText)
	;return StrReplace(Format("{:0" vNum "}", 0), 0, vText)
}

JEE_AccGetTextAll(hWnd:=0, vSep:="`n", vIndent:="`t", vOpt:="")
{
	vLimN := 20, vLimV := 20
	Loop, Parse, vOpt, % " "
	{
		vTemp := A_LoopField
		if (SubStr(vTemp, 1, 1) = "n")
			vLimN := SubStr(vTemp, 2)
		else if (SubStr(vTemp, 1, 1) = "v")
			vLimV := SubStr(vTemp, 2)
	}

	oMem := {}, oPos := {}
	;OBJID_WINDOW := 0x0
	oMem[1, 1] := Acc_ObjectFromWindow(hWnd, 0x0)
	oPos[1] := 1, vLevel := 1
	VarSetCapacity(vOutput, 1000000*2)

	Loop
	{
		if !vLevel
			break
		if !oMem[vLevel].HasKey(oPos[vLevel])
		{
			oMem.Delete(vLevel)
			oPos.Delete(vLevel)
			vLevelLast := vLevel, vLevel -= 1
			oPos[vLevel]++
			continue
		}
		oKey := oMem[vLevel, oPos[vLevel]]

		vName := "", vValue := ""
		if IsObject(oKey)
		{
			vRoleText := Acc_GetRoleText(oKey.accRole(0))
			try vName := oKey.accName(0)
			try vValue := oKey.accValue(0)
		}
		else
		{
			oParent := oMem[vLevel-1,oPos[vLevel-1]]
			vChildId := IsObject(oKey) ? 0 : oPos[vLevel]
			vRoleText := Acc_GetRoleText(oParent.accRole(vChildID))
			try vName := oParent.accName(vChildID)
			try vValue := oParent.accValue(vChildID)
		}
		if (StrLen(vName) > vLimN)
			vName := SubStr(vName, 1, vLimN) "..."
		if (StrLen(vValue) > vLimV)
			vValue := SubStr(vValue, 1, vLimV) "..."
		vName := RegExReplace(vName, "[`r`n]", " ")
		vValue := RegExReplace(vValue, "[`r`n]", " ")

		vAccPath := ""
		if IsObject(oKey)
		{
			Loop, % oPos.Length() - 1
				vAccPath .= (A_Index=1?"":".") oPos[A_Index+1]
		}
		else
		{
			Loop, % oPos.Length() - 2
				vAccPath .= (A_Index=1?"":".") oPos[A_Index+1]
			vAccPath .= " c" oPos[oPos.Length()]
		}
		vOutput .= vAccPath "`t" JEE_StrRept(vIndent, vLevel-1) vRoleText " [" vName "][" vValue "]" vSep

		oChildren := Acc_Children(oKey)
		if !oChildren.Length()
			oPos[vLevel]++
		else
		{
			vLevelLast := vLevel, vLevel += 1
			oMem[vLevel] := oChildren
			oPos[vLevel] := 1
		}
	}
	return SubStr(vOutput, 1, -StrLen(vSep))
}

example usage. Click on your settings window that you showed us and hit F10. Once Done appears paste the results to a notepad window.

Code: Select all

F10::
Clipboard := JEE_AccGetTextAll(Winexist("A"), "`r`n")
MsgBox, 262208, , Done
return

It should look something like this:

Spoiler

These are the paths that can be used in the acc_get function. An example of how to click lets say devices:

Code: Select all


^t::
hwnd := Winexist("A")
loop, 5 ; the path sometimes changes so do a quick loop to cover a few more paths (you'll need to do this often with ACC)
{
	try oAcc := Acc_Get("Object", "4.3.4.2.6." a_index, 0, "ahk_id " hwnd) ; here we use the path we found above for "Devices"
	try vText := oAcc.accName(0)

	if instr(vText,"Devices")
	{
		try vLocation := Acc_Location(oAcc) ; returns array with .X .Y .W and .H
			
		; get the centerpoints 
		Button_X := vLocation.X + ( vLocation.W / 2 )
		Button_Y := vLocation.Y + ( vLocation.H / 2 )
			
		; convert screen to relative
		ScreenToWin(hWnd, Button_X, Button_Y, wX, wY)
		
		MouseClick,,wX, wY
	}
}
return

ScreenToWin(WindowID, sX, sY, ByRef wX, ByRef wY)
{
	WinGetPos,x,y,,,ahk_id %WindowID%
	wX:=sX-x
	wY:=sY-y
	return
}
return

I tried dodefaultaction but it was acting very strangely on this window. Also controlclick didn't seem to work either so just leaving you with mouseclick hopefully you wouldnt care about the mouse moving.

Rohwedder · 23 Jun 2021, 12:14

Hallo,
https://docs.microsoft.com/en-us/windows/uwp/launch-resume/launch-settings-app#system
Use ms-settings:… in the Run-Window.

Alexander2 · 24 Jun 2021, 09:26

doubledave22 wrote: ↑
23 Jun 2021, 11:30
The ahk ACC library seems good for this task.
https://www.autohotkey.com/boards/viewtopic.php?t=26201

after including the ACC library run this function to evaluate a window to see if the elements you are interested in are found
https://www.autohotkey.com/boards/viewtopic.php?f=6&t=40615
Code: Select all
JEE_StrRept(vText, vNum)
{
	if (vNum <= 0)
		return
	return StrReplace(Format("{:" vNum "}", ""), " ", vText)
	;return StrReplace(Format("{:0" vNum "}", 0), 0, vText)
}

JEE_AccGetTextAll(hWnd:=0, vSep:="`n", vIndent:="`t", vOpt:="")
{
	vLimN := 20, vLimV := 20
	Loop, Parse, vOpt, % " "
	{
		vTemp := A_LoopField
		if (SubStr(vTemp, 1, 1) = "n")
			vLimN := SubStr(vTemp, 2)
		else if (SubStr(vTemp, 1, 1) = "v")
			vLimV := SubStr(vTemp, 2)
	}

	oMem := {}, oPos := {}
	;OBJID_WINDOW := 0x0
	oMem[1, 1] := Acc_ObjectFromWindow(hWnd, 0x0)
	oPos[1] := 1, vLevel := 1
	VarSetCapacity(vOutput, 1000000*2)

	Loop
	{
		if !vLevel
			break
		if !oMem[vLevel].HasKey(oPos[vLevel])
		{
			oMem.Delete(vLevel)
			oPos.Delete(vLevel)
			vLevelLast := vLevel, vLevel -= 1
			oPos[vLevel]++
			continue
		}
		oKey := oMem[vLevel, oPos[vLevel]]

		vName := "", vValue := ""
		if IsObject(oKey)
		{
			vRoleText := Acc_GetRoleText(oKey.accRole(0))
			try vName := oKey.accName(0)
			try vValue := oKey.accValue(0)
		}
		else
		{
			oParent := oMem[vLevel-1,oPos[vLevel-1]]
			vChildId := IsObject(oKey) ? 0 : oPos[vLevel]
			vRoleText := Acc_GetRoleText(oParent.accRole(vChildID))
			try vName := oParent.accName(vChildID)
			try vValue := oParent.accValue(vChildID)
		}
		if (StrLen(vName) > vLimN)
			vName := SubStr(vName, 1, vLimN) "..."
		if (StrLen(vValue) > vLimV)
			vValue := SubStr(vValue, 1, vLimV) "..."
		vName := RegExReplace(vName, "[`r`n]", " ")
		vValue := RegExReplace(vValue, "[`r`n]", " ")

		vAccPath := ""
		if IsObject(oKey)
		{
			Loop, % oPos.Length() - 1
				vAccPath .= (A_Index=1?"":".") oPos[A_Index+1]
		}
		else
		{
			Loop, % oPos.Length() - 2
				vAccPath .= (A_Index=1?"":".") oPos[A_Index+1]
			vAccPath .= " c" oPos[oPos.Length()]
		}
		vOutput .= vAccPath "`t" JEE_StrRept(vIndent, vLevel-1) vRoleText " [" vName "][" vValue "]" vSep

		oChildren := Acc_Children(oKey)
		if !oChildren.Length()
			oPos[vLevel]++
		else
		{
			vLevelLast := vLevel, vLevel += 1
			oMem[vLevel] := oChildren
			oPos[vLevel] := 1
		}
	}
	return SubStr(vOutput, 1, -StrLen(vSep))
}
example usage. Click on your settings window that you showed us and hit F10. Once Done appears paste the results to a notepad window.
Code: Select all
F10::
Clipboard := JEE_AccGetTextAll(Winexist("A"), "`r`n")
MsgBox, 262208, , Done
return
It should look something like this:
Spoiler

4.2.4.2.5 editable text [Search box, Find a s...][]
4.2.4.2.5.1 text [Find a setting][]
4.2.4.2.6 list [][]
4.2.4.2.6.1 list item [System][]
4.2.4.2.6.1.1 text [System][]
4.2.4.2.6.1.2 text [Display, sound, noti...][]
4.2.4.2.6.2 list item [Devices][]
4.2.4.2.6.2.1 text [Devices][]
4.2.4.2.6.2.2 text [Bluetooth, printers,...][]
4.2.4.2.6.3 list item [Phone][]
4.2.4.2.6.3.1 text [Phone][]
4.2.4.2.6.3.2 text [Link your Android, i...][]
4.2.4.2.6.4 list item [Network & Internet][]
4.2.4.2.6.4.1 text [Network & Internet][]
4.2.4.2.6.4.2 text [Wi-Fi, airplane mode...][]
4.2.4.2.6.5 list item [Personalization][]
4.2.4.2.6.5.1 text [Personalization][]
4.2.4.2.6.5.2 text [Background, lock scr...][]
4.2.4.2.6.6 list item [Apps][]
4.2.4.2.6.6.1 text [Apps][]
4.2.4.2.6.6.2 text [Uninstall, defaults,...][]
4.2.4.2.6.7 list item [Accounts][]
4.2.4.2.6.7.1 text [Accounts][]
4.2.4.2.6.7.2 text [Your accounts, email...][]
4.2.4.2.6.8 list item [Time & Language][]
4.2.4.2.6.8.1 text [Time & Language][]
4.2.4.2.6.8.2 text [Speech, region, date][]
4.2.4.2.6.9 list item [Gaming][]
4.2.4.2.6.9.1 text [Gaming][]
4.2.4.2.6.9.2 text [Game bar, captures, ...][]
4.2.4.2.6.10 list item [Ease of Access][]
4.2.4.2.6.10.1 text [Ease of Access][]
4.2.4.2.6.10.2 text [Narrator, magnifier,...][]
4.2.4.2.6.11 list item [Search][]
4.2.4.2.6.11.1 text [Search][]
4.2.4.2.6.11.2 text [Find my files, permi...][]
4.2.4.2.6.12 list item [Cortana][]
4.2.4.2.6.12.1 text [Cortana][]
4.2.4.2.6.12.2 text [Cortana language, pe...][]
4.2.4.2.6.13 list item [Privacy][]
4.2.4.2.6.13.1 text [Privacy][]
4.2.4.2.6.13.2 text [Location, camera, mi...][]
4.2.4.2.6.14 list item [Update & Security][]
4.2.4.2.6.14.1 text [Update & Security][]
4.2.4.2.6.14.2 text [Windows Update, reco...][]
These are the paths that can be used in the acc_get function. An example of how to click lets say devices:
Code: Select all
^t::
hwnd := Winexist("A")
loop, 5 ; the path sometimes changes so do a quick loop to cover a few more paths (you'll need to do this often with ACC)
{
	try oAcc := Acc_Get("Object", "4.3.4.2.6." a_index, 0, "ahk_id " hwnd) ; here we use the path we found above for "Devices"
	try vText := oAcc.accName(0)

	if instr(vText,"Devices")
	{
		try vLocation := Acc_Location(oAcc) ; returns array with .X .Y .W and .H
			
		; get the centerpoints 
		Button_X := vLocation.X + ( vLocation.W / 2 )
		Button_Y := vLocation.Y + ( vLocation.H / 2 )
			
		; convert screen to relative
		ScreenToWin(hWnd, Button_X, Button_Y, wX, wY)
		
		MouseClick,,wX, wY
	}
}
return

ScreenToWin(WindowID, sX, sY, ByRef wX, ByRef wY)
{
	WinGetPos,x,y,,,ahk_id %WindowID%
	wX:=sX-x
	wY:=sY-y
	return
}
return
I tried dodefaultaction but it was acting very strangely on this window. Also controlclick didn't seem to work either so just leaving you with mouseclick hopefully you wouldnt care about the mouse moving.

Thank you. AccViewer is a useful application which does what I needed. It was able to retrieve all the information which I needed to obtain. I will have to learn how to use the retrieved data in order to click on buttons from which the data has been obtained.

Alexander2 · 01 Jul 2021, 12:46

Can you also explain what is the role of the additional application iWB2 Learner given in the link?

doubledave22 · 01 Jul 2021, 17:58

Alexander2 wrote: ↑
01 Jul 2021, 12:46
Can you also explain what is the role of the additional application iWB2 Learner given in the link?

Hi, sorry I don't have any experience with iWB2. I think it might just be for internet explorer.

Ascertaining Names of Buttons in the System Topic is solved

Ascertaining Names of Buttons in the System

Re: Ascertaining Names of Buttons in the System

Re: Ascertaining Names of Buttons in the System

Re: Ascertaining Names of Buttons in the System

Re: Ascertaining Names of Buttons in the System Topic is solved

Re: Ascertaining Names of Buttons in the System

Re: Ascertaining Names of Buttons in the System

Re: Ascertaining Names of Buttons in the System

Re: Ascertaining Names of Buttons in the System

Who is online