After about 40 hours of research, I believe I finally understand how to actually control windows DPI scaling!
I will try and summarise what I have learned so that it may help other people who may have become as confused as I have been.
First thing I will note is that if windows dpi scaling is set to anything other than 100%, you will not be able to force your app to display as if dpi scaling was set to 100% just for your app. I thought this feature existed, but it doesn't. You can attempt to fudge it that way, but it will look ugly and many elements will not display right (more about this later).
So applications can contain a "dpi aware" flag which tells Windows whether they are aware of the current Windows dpi setting. The flag can be set in multiple places and different things happen depending on where you set it:
1. In the app's manifest (editable with Resource Hacker). The flag looks like this and goes inside the <assembly></assembly> tags:
<application xmlns="urn:schemas-microsoft-com:asm.v3">
<windowsSettings>
<dpiAware xmlns="
http://schemas.microsoft.com/SMI/2005/W ... </dpiAware>
</windowsSettings>
</application>
A reboot is required for the manifest changes to take effect.
2. By calling SetProcessDpiAwareness (8.1+), or SetProcessDpiAwarenessContext (10+), or SetThreadDpiAwarenessContext (10+) function inside the program itself. The first 2 are global settings and do the same thing, the last is a "per thread" setting which allows you to use different setting for different GUI's. The first 2 will be overridden by the app manifest setting, and can otherwise only be set once. There is also an older SetProcessDPIAware function which is compatible with Vista+, however on my Windows 7 system it does't work properly as it doesn't cause A_ScreenDPI to receive current setting. Note: dllcalls are case-sensitive.
3. By setting the following reg key
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\YourScript.exe]
"dpiAwareness"=dword:00000001
On my system this overrides (1) and (2), however the setting is only compatible with Windows 10.
Prior to Windows 10, there is only one flag setting (true/false). Not setting the flag has the same effect as setting it to false, but the reverse is not true. After Windows 10 there are 2 extra flag settings (per monitor/per monitor v2).
For now I will ignore the per monitor settings and just explain what happens with the true/false setting.
If the flag is false (or not present), windows will use bitmap upscaling. This looks like upscaling standard definition video to high definition. Blurry and soft. With better upscaling filters it could have looked half decent, but I digress. In this mode you will not need to reposition any of your GUI elements, everything will be rendered at 1:1 100% zoom level and then finally the entire window raster will receive the bitmap upscale, so everything gets enlarged by exactly the same proportion relative to eachother. But it looks so soft and blurry I doubt you will want to use it.
If the flag is true, windows will not do bitmap scaling and leave it up to the app itself to get all the positioning right, since now all the gui elements such as text controls, check boxes, drop down lists etc. have all gotten bigger due to the windows dpi setting, and many controls will need to be adjusted manually by you, and elements cannot be made smaller, as reducing the controls below their default size results in horrible artefacts like nearest-neighbour type aliasing or bits of controls just not getting rendered at all. This is why the Autohotkey -DPIScale setting fails, it tries to shrink things back down for you, but because Windows doesn't play nice with lower than default dimensions for many controls it ends up looking nasty. You will get the same result using that mentioned dpi() library as it basically works the same way. To get a properly drawn GUI you will need to use A_ScreenDPI to adjust your control position/dimensions. This can be tricky because some things like window positions will be in absolute pixel values, others like width and height will be in relative dimensions so you will need to either multiply or divide by (A_ScreenDPI/96). Also you will need to check all levels of windows dpi scaling, as for example some of my text controls are cut off only at a windows dpi setting of 175%.
Autohotkey's executable is set to true in its app manifest.
The value can be checked with Process Explorer, but only in Windows 10.
Now we know what the true/false setting does, there are still other places where it will be overridden:
a. The exe's properties' compatibility tab's "disable display scaling on high dpi settings". Its regkey is in HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers in case you need to set it in your script. This setting confused me for so long, but here's what it does when ticked: if the app's dpi awareness flag is false or not present, Windows will force the app to a true setting. On Windows 10 it actually forces it to "per monitor", but the effect on a single monitor is the same. Since this checkbox is unticked by default, the default behaviour is that apps which have their flag set to false or unset , i.e lots of apps, will receive the crappy bitmap upscaling.
b. On Windows 7, there is a setting in the Control Panel called "use Windows XP style DPI scaling". Default on my system was ticked. This appears to effectively tick "disable display scaling on high dpi settings" for all apps. Its regkey appears to be HKEY_CURRENT_USER\Software\Microsoft\Windows\DWM] "UseDpiScaling"=dword:00000000
So what is the best option for your ahk gui?
The answer to me is simple: bitmap upscaling looks like crap, -dpiscale looks like crap, therefore I just leave ahk's app manifest as-is (flag=true) and reposition my controls manually using A_ScreenDPI. It took a lot of effort but in the end it was worth it.
edit: forgot to mention, on 4k displays a dpi scaling setting that seems to be anecdotally popular is 200%. If windows bitmap upscaling is used, this can look better because at whole integer levels of scaling (200%, 300% etc) windows uses a different "nearest-neighbour" scaling mode which results in 1 pixel becoming exactly 4 pixels and the original look and sharpness is preserved. Therefore it may be desirable to allow the user to manually set the app's dpi flag to false to achieve this. Achieving this on windows 10 is easy - just use (3) and restart your app. Alternatively, you could delete the flag in the app manifest (not false - delete) and then control it with (2) or (a). On windows 7, because (3) doesn't work, you will have to delete the flag in the app manifest (not false - delete) which then opens up the possibility of controlling the flag by (a) or (b), preferably (a) as it doesn't require a restart.