Is afx: the doom of Autohotkey ? Does Acc.ahk help ?

[Hotte]

Hi,

it seems to me that more and more modern applications are all but impossible to handle with Autohotkey. If you spy an explorer like Opera or Edge little shows up that could be used by ahk. And with every new version of my video-editor, nasty afx-controls show up I cannot handle.

In an old thread I read:

All the controls begining with "Afx" come froms progs developped in VC++ with the MFC. They are not standard controls and can't be managed directly with AHK.

Looking for a solution I found Acc.ahk, but I did not manage to withdraw any texts from an Afx with acc_get(). I am not a programmer, so I have not quite understood if Acc is helful at all with this matter.

Any ideas or help to save the future of my ahk-project ? Thanks!

[swagfag]

its not. switch from MSAA to UIA, provided the developers bothered supporting it

[gregster]

If you spy an explorer like Opera or Edge little shows up that could be used by ahk.

All common browsers always implemented their own controls - this is nothing new.

Opera and Edge Beta (the future release) are both Chromium-based, like Chrome, and can be automated via the chrome debugging protocol and AHK. IE has COM and there is Selenium...

The old (current) Edge has another protocol that probably could have been automated, too, but it will be gone soon anyway.

Edit: Topic moved to 'Ask For Help'.

[Hotte]

Ah, thanks. Seems a complicated matter.
In fact I am not interested in automating browsers but my video-application (EDIUS).
How can I find out which library or tool they are using to get hold of their afx-controls ?

Is swagfag's approach an idea to go for ? And what would I need to implement (sorry for not being so technically experienced as to know right away what to do next).

[Elgin]

Hi there!

Out of interest in video editing I just had a quick look around in their trial version with UIA (should be equal to MSAA in almost all cases).

Only about one third of the controls seems to be (halfway) correctly registered and can be controlled by UIA, some more show up as empty panes where you could use relative coordinates to send mouse clicks and the rest is just not there at all and can thus only be controlled by blind clicks or image searching.

At first glance I couldn't find a publicly available API and the application doesn't seem to register a COM object (at least nothing that has a catchy name).

Depending on what you want to do that might get tough...

If you're serious about it, it might be worth it to contact the manufacturer and ask them to step up their support for users with disabilities by putting in correct accessibility information for the controls. Some companies are simply not aware that exists and are quite interested and helpful.

Good luck!

[guest3456]

its not. switch from MSAA to UIA, provided the developers bothered supporting it

exactly its the same problem with Acc or UIA.. it all depends on what the app supports or not

Any ideas or help to save the future of my ahk-project ? Thanks!

last case resort is to go the graphics route. you could click when you detect a certain pixel or image at a location. you won't be able to read text unless you do OCR

[Hotte]

@Elgin, thank you very much for you having a look right into the EDIUS application. Too bad, what you found out.

My video application is already more or less finished, more than 95% of the code adresses dialog-controls, only a few annoying ones need surface-controls. Somehow I worked around . But EDIUS is coming up with a completely new interface soon. Windows and controls could be easily adapted since I've got a very structured approach, but only if the dialog-controls are still adressable...worrying.

2 more questions please:
What do you mean by blind clicks ?
Could you give me a hint for non programmers how to implement and use UIA ?

Thanks again.

[Elgin]

Hi!

With blind clicks I mean clicking some position in a window without being able to know or confirm if what you want to click is actually (still) there.

For a non-programmer I would honestly not bother with UIA, as there is AFAIK no user friendly package out there for AHK (can't publish my own wrapper for it so far unfortunately). By itself the library is a bit unfriendly; especially to non-programmers. You can have a look at some simple example code here:
https://github.com/Elgin1/TypeLib2AHK

ACC.ahk will give you the same information and the only big thing missing in my opinion is the powerful search function of UIA. Since you only need it for one application that should not be a big issue and there is plenty of example code out there.

[guest3456]

For a non-programmer I would honestly not bother with UIA, as there is AFAIK no user friendly package out there for AHK

this thread:
https://www.autohotkey.com/boards/viewtopic.php?f=76&t=69627

links to this wrapper:
https://www.autohotkey.com/boards/viewtopic.php?t=2216

may be worth a look

[Hotte]

Thanks guys for being so helpful.

To be honest: This is horrible. I do not understand what those guys in the threads are talking about and the code examples look Chinese to me although I have written some 15.000 lines of ahk-Code.

To anwser my own questions: Yes, these UIA-applications seem to herald the decline of Autohotkey

[AHKStudent]

Thanks guys for being so helpful.

To be honest: This is horrible. I do not understand what those guys in the threads are talking about and the code examples look Chinese to me although I have written some 15.000 lines of ahk-Code.

To anwser my own questions: Yes, these UIA-applications seem to herald the decline of Autohotkey

take a look here https://www.autohotkey.com/boards/viewforum.php?f=6 great ahk scripts that have nothing to do with accessing some programs windows

almost all programs can be controlled using either standard ahk methods or acc / UIA.

[guest3456]

To anwser my own questions: Yes, these UIA-applications seem to herald the decline of Autohotkey

i work mostly with game windows... very few of these windows actually expose control classNNs for me to work with. i havent even tried UIA or Acc because i highly doubt those interfaces are supported either. yet, there is no decline for me, i just use the graphics methods i mentioned earlier, searching for pixels or images and responding appropriately.

[Hotte]

@AHKStudent : I have been looking around quite a lot for UIA stuff (I tried ACC_Get but didn`t manage to pull additional information). I have difficulties to understand the approach and the code examples in general. If there would be some tutorial / non-expert introduction...could help. I am just looking for a replacement for ControlGet and ControlClick. Anyway in your link I found an interesting script to decode text with ImageSearch. Will try that one.

@guest3456 :
Very interesting, please let´s exchange some experience on this: I have tried ImageSearch and it basically works but I found there are three very heavy drawbacks - please tell me, when you think I am wrong or you found better solutions:

1. The control has to be visible. Due to user interaction this is never save. ControlSend also works, when the window is in the background, covered, not in sight or even when the TabView is not activated. Crazy. Even more windows can be set to transparency 0 and the user doesn't even notice anything of what the script ist doing in the background, but it works smooth and save if you put some effort into it. You only have to make sure, that the window is there. Do I have to say goodbye to such power ?
2. With the ImageSearch approach I found that to operate on the controls you would have to send MouseClicks, which I have avoided whenever I could. The reason is they interupt and irritate user interaction heavily and so the operation is never save. You'd basically have to tell the user to stop doing anything or disable the mouse. I found that the script becomes completely unreliable if you are doing more complex things
3. The images of the controls change: The user or the app changes GUI-colors, shape, position, font or size of elements I hoped to find with ImageSearch

I have difficulties to imagine how to build something reliable with it apart from some simple surface activity. But you seem to have more experience with its so I am happy to learn from your ideas.

[guest3456]

1. The control has to be visible. Due to user interaction this is never save. ControlSend also works, when the window is in the background, covered, not in sight or even when the TabView is not activated. Crazy. Even more windows can be set to transparency 0 and the user doesn't even notice anything of what the script ist doing in the background, but it works smooth and save if you put some effort into it. You only have to make sure, that the window is there. Do I have to say goodbye to such power ?
2. With the ImageSearch approach I found that to operate on the controls you would have to send MouseClicks, which I have avoided whenever I could. The reason is they interupt and irritate user interaction heavily and so the operation is never save. You'd basically have to tell the user to stop doing anything or disable the mouse. I found that the script becomes completely unreliable if you are doing more complex things

Gdip_ImageSearch can search windows in the background. But if no contrls are exposed, then obviously controlSend won't work, and you'll have to first focus the window, send a click to the box, and then send your string. No other good way. Sometimes the app will still respond to ControlClicks (you can just specify x/y coords), which means that the user's mouse wont be interrupted, and the clicks will be programmatically posted to the window.

3. The images of the controls change: The user or the app changes GUI-colors, shape, position, font or size of elements I hoped to find with ImageSearch

if your users are going to be messing with the GUI stuff, then this is a problem. You can always require your users to go through a setup wizard or the like, and during that time, you take screenshots of their app that you will use in subsequent imagesearches. it all depends on what and how you plan on automating

for example the blind clicks:

With blind clicks I mean clicking some position in a window without being able to know or confirm if what you want to click is actually (still) there.

what i do for one game is, i grab the pixel at a certain location first, then i send a (blind?) click to a coordinate, and then i check the pixel color again. i'm expecting a popup window to appear, so i'm checking a coordinate that the popup covers. if the pixel color is different, then i can assume that my click was successful, and that the popup is visible. if the color is the same as originally, then i know that my click failed to trigger the popup for whatever reason. of course, if i know my users won't change the graphics, then i can simply hard code my expected popup window color. but some of my users will change the GUI colors etc and "skin" the game to their liking, and so you just have to try to find clever approaches

yes all of this is way more unreliable than having the controls exposed. but what are you gonna do? either solve the problem, or give up. As you've noticed, more and more apps are choosing to not use default microsoft controls, and instead either building their own custom controls, or even more recently, some apps are using the ElectronJS platform, where they can write their apps once using HTML/JS, and the apps will work cross platform on windows/linux/apple. Obviously those apps dont expose any windows controls, since the app is basically a webbrowser wrapped in window borders and titlebars.

[Hotte]

@guest3456 Can't believe it! Although I have been working with ahk for so many years, I have completely overseen the posibilty to use ControlClick x123 y456 to send Clicks without moving the mouse as is the case with click,123,456 that I have been using so far. Indeed that will of course much less obstruct user interaction. Good one!

You made the situation pretty clear : If I am going for a certain UI-technology I might end up to meet another one in the next application. If I look e.g. at "darktable" which is a photo raw editor there are absolutely no controls to spy and ImageSearch would be the only way to go. Understood.

Could you just tell me how you would solve the following situations:

Case 1:
On the darktable main window there may be more than 100 graphic controls, 10 of which I consider being special buttons that I would like to do 10 different things. The user may have completely rearranged the window to his needs. So there is no chance to find the buttons at certain locations. Would you then loop ImageSearch on a database of 10 tiny bmp-Files you captured from screen and compare the mouse area the user clicked onto until you got the right button ? Isn`t that very slow ? Or Is there a smarter way to do it ?

Case 2:
Say I would like to automate something when the user tyes Alt+N. I then need to find an entry named "Noise reduction" in the main window and perform a number of controlClicks to get that done. I would have to ImageSearch the full window (which can be 6 Megapixels or larger on an UHD monitor) for this noise_reduction.bmp which is slow. Or is there a smarter way to do it ?

Thanks

[guest3456]

@guest3456 Can't believe it! Although I have been working with ahk for so many years, I have completely overseen the posibilty to use ControlClick x123 y456 to send Clicks without moving the mouse as is the case with click,123,456 that I have been using so far. Indeed that will of course much less obstruct user interaction. Good one!

some apps won't respond to ControlClicks and you have no choice but to use normal Clicks. its a case by case basis

Case 1:
On the darktable main window there may be more than 100 graphic controls, 10 of which I consider being special buttons that I would like to do 10 different things. The user may have completely rearranged the window to his needs. So there is no chance to find the buttons at certain locations. Would you then loop ImageSearch on a database of 10 tiny bmp-Files you captured from screen and compare the mouse area the user clicked onto until you got the right button ? Isn`t that very slow ? Or Is there a smarter way to do it ?

well what are the 10 different things you are looking to do? that would dictate how you would construct the program logic. you dont compare the area the user clicked.. i mean i guess you could just intercept every Left click, but then what if the user clicks on the extreme edge of your tiny image, then how are you going to determine what happened? you need to be more specific on what the controls look like, and what you're trying to accomplish. do they change after being clicked? sometimes controls will be "depressed" and so that would change their image, and so you could just loop imagesearch, and when you find the button depressed, then you know its been activated.

Case 2:
Say I would like to automate something when the user tyes Alt+N. I then need to find an entry named "Noise reduction" in the main window and perform a number of controlClicks to get that done. I would have to ImageSearch the full window (which can be 6 Megapixels or larger on an UHD monitor) for this noise_reduction.bmp which is slow. Or is there a smarter way to do it ?

what is an "entry" named Noise Reduction? is it a popup window? if it cannot be limited to a certain area of the screen, then yes you have to search the full screen. yes it could be slow. it all depends if you can figure out ways to optimize it. sorry i cant give you more help without knowing more specifics

[Hotte]

Intercepting leftclick on graphic button-like areas should work if the image of the button holds upper left corner at pos 0,0 of the button. If the button has a width of 80 pixels and a height of 20 the ImageSearch is conducted only in the area of ClickPosX-80, ClickPosY-20 to ClickPosX+80, ClickPosY+20 This is a tiny area and the search will also be very fast. Color change of the button during the click is not a problem because the interception takes place before the click is effective. This can be used e.g. to do additional things when the user presses the button. I tried this already and it works.
My question was more aiming at if there is another solution rather than comparing the clickarea with 10 images which could be slow. And YES I just found the absolutely brilliant FindText() https://www.autohotkey.com/boards/viewtopic.php?p=146096#p94352
This speeds up and eases ImageFinding a whole lot since the images are being transformed into relatively short textstrings and picked up easily and flexibly with the integrated picking and testing tool. Highly interesting approach!

So a first idea could be to locate and store all the 10 Buttons of my example using FindText() which should only take around 1 second and then only compare the clickpositions in memory. This could be lightning fast as long as the user does not rearrange the screen.

You really inspired me to rethink my prejudices against ImageSearching, and it is getting even more attractive with FindText()

Thanks!

[guest3456]

Intercepting leftclick on graphic button-like areas should work if the image of the button holds upper left corner at pos 0,0 of the button. If the button has a width of 80 pixels and a height of 20 the ImageSearch is conducted only in the area of ClickPosX-80, ClickPosY-20 to ClickPosX+80, ClickPosY+20 This is a tiny area and the search will also be very fast.

yes i understand now, very good idea, that should work i believe

Color change of the button during the click is not a problem because the interception takes place before the click is effective. This can be used e.g. to do additional things when the user presses the button. I tried this already and it works.

well are you intercepting all clicks and doing your own processing? or are you using ~ LButton hotkey which will pass the click to both AHK and also to the underlying window? if the latter, i dont think you can guarantee that AHK will complete the imagesearch BEFORE the app responds to the click

And YES I just found the absolutely brilliant FindText() https://www.autohotkey.com/boards/viewtopic.php?p=146096#p94352
This speeds up and eases ImageFinding a whole lot since the images are being transformed into relatively short textstrings and picked up easily and flexibly with the integrated picking and testing tool. Highly interesting approach!

i have not tried FindText yet.. it looks really good. i'd be surprised if it was much faster than Gdip_ImageSearch but either solution would be fine

So a first idea could be to locate and store all the 10 Buttons of my example using FindText() which should only take around 1 second and then only compare the clickpositions in memory. This could be lightning fast as long as the user does not rearrange the screen.

You really inspired me to rethink my prejudices against ImageSearching, and it is getting even more attractive with FindText()

yep, i mean i agree with your initial premise.. obviously we all would prefer to have exposed controls and be much more confident in our automations.. but as apps evolve sometimes we have no choice but to hack away any way we can.

[Hotte]

well are you intercepting all clicks and doing your own processing? or are you using ~ LButton hotkey which will pass the click to both AHK and also to the underlying window? if the latter, i dont think you can guarantee that AHK will complete the imagesearch BEFORE the app responds to the click

I block all clicks. If the context of the click is relevant (say the user hoovers over a button or ImageSearch yielded a relevant icon according to the cursors position), then I first do all what I want to do, and after that I pass on the click (or not). If the context is not relevant I bypass. This is a matter of sequence, not timing. Very simple and effective if you know how. See https://www.autohotkey.com/boards/viewtopic.php?f=76&t=42375&p=192581#p192581 for details.

i have not tried FindText yet.. it looks really good. i'd be surprised if it was much faster than Gdip_ImageSearch but either solution would be fine

With FindText() you get a tool to pick an area from the screen which you crop to your needs. That is being transformed into a simple string that you copy into your ahk code to call FindText() with. My tests yielded about 35-50ms for a FHD fullscreen scan if the item exists and 60 to 80ms if the item does not exist. If you reduce the search area to an average dialog window you might double the speed but with no lower than 17ms. Seems very fast to me. will have a look into Gdip_ImageSearch at the next opportunity because you seed it also works on covered windows, which is interesting and not possible with findText()

[guest3456]

I block all clicks. If the context of the click is relevant (say the user hoovers over a button or ImageSearch yielded a relevant icon according to the cursors position), then I first do all what I want to do, and after that I pass on the click (or not). If the context is not relevant I bypass. This is a matter of sequence, not timing. Very simple and effective if you know how. See https://www.autohotkey.com/boards/viewtopic.php?f=76&t=42375&p=192581#p192581 for details.

understood. that would work fine.

With FindText() you get a tool to pick an area from the screen which you crop to your needs. That is being transformed into a simple string that you copy into your ahk code to call FindText() with. My tests yielded about 35-50ms for a FHD fullscreen scan if the item exists and 60 to 80ms if the item does not exist. If you reduce the search area to an average dialog window you might double the speed but with no lower than 17ms. Seems very fast to me. will have a look into Gdip_ImageSearch at the next opportunity because you seed it also works on covered windows, which is interesting and not possible with findText()

i'm sure findText can be modified to work with windows that are behind others. it depends on how you get the source graphics. with the gdip lib, you can use Gdip_BitmapFromHWND which will use PrintWindow (which is a little slower), or you can use Gdip_BitmapFromScreen("Hwnd:" hwnd) and that will grab the graphics straight from the GetDC, which requires the DWM to be running (always running on win8.1+, needs Aero theme on Win7). i'm sure findText can be modified to use one of these