From MSDDN:
https://msdn.microsoft.com/en-us/librar ... s.85).aspx
App modal:
The user must respond to the message box before continuing work in the window identified by the hWnd parameter. However, the user can move to the windows of other threads and work in those windows.
Depending on the hierarchy of windows in the application, the user may be able to move to other windows within the thread. All child windows of the parent of the message box are automatically disabled, but pop-up windows are not.
System Modal:
Same as MB_APPLMODAL except that the message box has the WS_EX_TOPMOST style. Use system-modal message boxes to notify the user of serious, potentially damaging errors that require immediate attention (for example, running out of memory). This flag has no effect on the user's ability to interact with windows other than those associated with hWnd.
Task Modal:
Same as MB_APPLMODAL except that all the top-level windows belonging to the current thread are disabled if the hWnd parameter is NULL. Use this flag when the calling application or library does not have a window handle available but still needs to prevent input to other windows in the calling thread without suspending other threads.
Since Autohotkey is an interpreted language, and depends on Windows functionality in the background, and since from what I can tell, Windows message boxes do not block other input or threads from running or being interacted with... you've got to be more creative with your solution.
Perhaps begin by looking into
https://autohotkey.com/docs/commands/BlockInput.htm.