Help Request: Motion Tracking script

[Rvralph803]

Objective & Intent:
I would like to create a script that tracks the motion of pixels from an area on the screen, within a set radius of those selected pixels. This will be used as a method to stabilize the aim within a shooter -- The intent is not an 'anti recoil' or 'anti-sway' binding, but rather to moderately stabilize the gun on a moving tank. There is no access to the fundamental actor rotation and translation data to assist. This method must rely solely on extracting and comparing frame data from the screen to accomplish this task.

My Approaches so far
So far I have attempted to use GDIP to extract a small token (that's what I'm calling it) of approximately 7x7 pixels from the screen. On the next frame (16ms later, eg 60fps) I used GDIP's imagesearch function to search a 101x101 pixel box centered around where that token was extracted to search for the same set of pixels with some variation. This then outputs the x and y position at which this the token was found in this larger search space on the next frame. Eventually the hope is to feed these x and y offsets into a method that inputs up, down, left and right controls to return that token back towards its initial position -- exact positioning is likely impossible but keeping it within the vicinity is really the objective.

Problems
I feel like this method is both dirty and slow. And quite frankly I haven't gotten it to work. I think this is for multiple reasons but here are a few:
- The algorithm is sampling random patterns from the terrain that are subject to per frame pixel shifts as the player moves in any axis and represent an inherently dirty raw data to draw from. This is expected, but my expectation was that these shifts would be minor enough between individual frames so as to be overcome.
- The method probably requires refinement: There is no image processing occurring on these bitmap tokens or search area data so as to increase contrast or clarity

What I really want is an algorithm that works much like that of an optical mouse. It looks at a small patch of pixels, quickly determines the difference between a previously saved state, and calculates the vector of acceleration based off of those two. I just have no idea how to implement this and cannot find the sort of algorithm expressed in terms I can understand in my limited coding experience to enact a solution like this.

[Rvralph803]

The more I research this topic the more It seems like what I'm attempting to do is called image registration, or Phase Correlation, and these methods take a fast fourier transform of two images, compare the outputs and then determine the XY displacement.

An example can be found here:
https://sthoduka.github.io/imreg_fmt/docs/phase-correlation/

AHK does not have native support for FFT, and so some library or other method would have to be brought in for this to be successful. Unless FFT can be coded mathematically into some function (way beyond my ability).

Though I could see how having FFT in AHK could be useful in multiple different ways -- for example having an inexact image searching function.

[Noitalommi_2]

Hi.

Just out of curiosity, you want to stabilize the tank's cannon while driving and for this you want to use the crosshair as a reference?

[Rvralph803]

Hi.

Just out of curiosity, you want to stabilize the tank's cannon while driving and for this you want to use the crosshair as a reference?

I want to stabilize the tank's cannon through the main viewport, but not using a static asset like the crosshair at all. The region of the screen that would be sampled to determine if motion is occurring would be in an area offset from any static screen elements, and only show the background terrain close to, but not on the crosshairs.

[Noitalommi_2]

And what game is it?

[Rvralph803]

Why does the game matter? It's Post scriptum, a WWII shooter.

It's based on unreal4.7, but the player aim variables are not screen accessible, and I'm not looking for a solution that is an actual and functional hack from game memory pointers.

[Noitalommi_2]

[...] and I'm not looking for a solution that is an actual and functional hack from game memory pointers.

That would have been my suggestion. Anything else is probably unlikely to work in time.
But since the game uses anti-cheat, reading the memory is probably not a good idea.

[Rvralph803]

Yep. That's why I was looking for some sort of basic algorithm like those used in optical mice that could be run on a small patch of screen real estate. 16ms is a rather long pause for modern CPUs and as long as bitmaps are pulled into memory and not swapped from files I was fairly confident that the speed could be under 10ms.

I just need some sort of optical flow algorithm to do that work. Phase correlation seems like the ideal candidate, and there are many libraries out there for python, C or other languages that could be included into AHK... but it's beyond my limited understanding on how to do those includes.

At this point my Idea setup would grab a small patch -- apparently optical mice only look at about 10x10 pixels total -- and then do an optical flow algorithm on two successive images over two successive frames. Output x-y translation data could be averaged over, say the last 3 frames and used with some form of hysteresis to feedback controls to alter the view of the actor to fight those gross movements.

This requires access to fourier transform math, which AHK apparently doesn't have.

[MrDodel]

No matter how you word this, you're aiming (no pun intended) for an anti-recoil "moderately stabilize" script, no assistance will be provided, see the forum rules.

[gregster]

@MrDodel, which rule are you referring to exactly?

Apart from that, I doubt that (pure) AHK is fast enough for a task like this.

[MrDodel]

@MrDodel, which rule are you referring to exactly?

I thought I'd read it here: viewtopic.php?f=2&t=29155 but as it's marked as draft, I've put 2 and 2 together to get 5.

""2. Any kind of AutoHotkey script that removes intended drawbacks from the game or its elements, by automating, hacking or modifying the game in general, is not allowed."