Excessive page faults with Process,Exist

Report problems with documented functionality
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Excessive page faults with Process,Exist

24 May 2017, 23:38

When calling "Process,Exist" many times per second, such as when trying to detect if a process is running, there are many soft page faults generated (on my system, 34,000 per second).

Here is an example.

Code: Select all

#SingleInstance FORCE
#Persistent
SetWorkingDir %A_ScriptDir%
SetBatchLines, -1
SetTimer, DetectProcess, 10
return

DetectProcess:
If (processExist("NonexistentProcess.exe"))
	msgbox Found the process.	
return


processExist(Name){
Process,Exist,%Name%
return Errorlevel
}	
Page faults on my system while this loop is running (Win7 x64)

Image



Here is a solution I have adapted from the dllCall example in the help file

Code: Select all

#SingleInstance FORCE
#Persistent
SetWorkingDir %A_ScriptDir%
SetBatchLines, -1
SetTimer, DetectProcess, 10
return


DetectProcess:
If (processExist("NonexistentProcess.exe"))
	msgbox Found the process.	
return


ProcessExist(Name){
global HasPrivilege

	If (HasPrivilege = 1){
		
		l := ""
		s := 4096 
		s := VarSetCapacity(a, s, 0)  
		DllCall("Psapi.dll\EnumProcesses", "Ptr", &a, "UInt", s, "UIntP", r)
		sleep 10

			Loop, % r // 4 
			{
				id := NumGet(a, A_Index * 4, "UInt")    
				h := DllCall("OpenProcess", "UInt", 0x0010 | 0x0400, "Int", false, "UInt", id, "Ptr")
			   
				if !h
				  continue	  
			   
			   VarSetCapacity(n, s, 0)  
			   e := DllCall("Psapi.dll\GetModuleBaseName", "Ptr", h, "Ptr", 0, "Str", n, "UInt", A_IsUnicode ? s//2 : s)
			   
			   if !e   
				  if e := DllCall("Psapi.dll\GetProcessImageFileName", "Ptr", h, "Str", n, "UInt", A_IsUnicode ? s//2 : s)
					 SplitPath n, n
			  
			  ;DllCall("CloseHandle", "Ptr", h) 
			   
			   if (n && e)  
				  if (n = Name)
					 return id			
			}
	}
	
	Else
		SetDetectionPrivileges()
	
	
}




SetDetectionPrivileges(){
Process, Exist  
h := DllCall("OpenProcess", "UInt", 0x0400, "Int", false, "UInt", ErrorLevel, "Ptr")
DllCall("Advapi32.dll\OpenProcessToken", "Ptr", h, "UInt", 32, "PtrP", t)
VarSetCapacity(ti, 16, 0)  
NumPut(1, ti, 0, "UInt") 
DllCall("Advapi32.dll\LookupPrivilegeValue", "Ptr", 0, "Str", "SeDebugPrivilege", "Int64P", luid)
NumPut(luid, ti, 4, "Int64")
NumPut(2, ti, 12, "UInt")  
r := DllCall("Advapi32.dll\AdjustTokenPrivileges", "Ptr", t, "Int", false, "Ptr", &ti, "UInt", 0, "Ptr", 0, "Ptr", 0)
DllCall("CloseHandle", "Ptr", t)  
DllCall("CloseHandle", "Ptr", h) 
hModule := DllCall("LoadLibrary", "Str", "Psapi.dll")
global HasPrivilege := 1 
return
}
Page faults while the above loop is running, i.e no page faults.

Image

Although soft page faults don't necessarily affect performance, I still think something is wrong with the Process,Exist function, at least on my system, considering the dllCall method is pretty much flawless in that regard. Maybe it has something to do with handles not being released, or being released too often?

Calling Process,Exist at a much slower rate such as 1000ms brings the page faults down to a reasonable level, however if I have a big list of processes (such as games) then I need to iterate quickly through them at the 10ms interval, checking whether each one exists, in order to execute some lines of code immediately on launch. So while a 1000ms loop might work fine for casual use, it's not a proper solution for advanced use.

Side question: are there are any windows API messages that are automatically sent whenever a new process is launched? In this case a loop would not be needed, saving a lot of CPU and other problems related to multiple threads running under the Timer scheme which can cause some problems such as thread deadlock when looping at faster speeds such as 10ms and sleeping within one of them, such as when waiting for a process to terminate.
lexikos
Posts: 6653
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: Excessive page faults with Process,Exist

25 May 2017, 03:40

This is not a bug. I don't see any evidence that it's even a problem.

Process Exist uses CreateToolhelp32Snapshot, which most likely includes information in the snapshot which is not needed, accessing the memory space of each process to get it.

Your solution fails to detect processes running at a higher (UAC) integrity level. For example, it does not detect programs which were run as administrator without being run as administrator itself. Script processes started with the Run with UI access option sit somewhere in between, and are not detected by your script unless it runs as administrator or with UI access.

There may be other situations where it fails. The original AutoHotkey was designed to use EnumProcesses only on Windows NT4, and CreateToolhelp32Snapshot on everything else. I'm not sure what the reasons were. None of the compilers I've used support NT4 (nor have I ever supported or even used NT4 myself), so I omit the code for it via pre-processor directives.
Maybe it has something to do with handles not being released, or being released too often?
You should add the "Handle Count" column in Process Explorer. If you do, you will see that your script leaks handles at an alarming rate. CloseHandle is not some optional thing that you can omit to improve performance. Calling it or not does not appear to affect the page fault delta, as far as I could see.

Process Exist only deals directly with one handle; the one returned by CreateToolhelp32Snapshot. If the API was failing to release handles, that would be a Windows bug.
When calling "Process,Exist" many times per second, such as when trying to detect if a process is running,
Even unaware of this page fault "problem", I would have considered that to be an inefficient solution and a bad idea in general.

WMI can notify you of new processes. See New Process Notifier for example.

If the process creates windows in the current window station ("user session" if you want to oversimplify), your script can be notified of them by registering a shell hook (search the forum) or SetWinEventHook. The shell hook is probably the lightest on resources and most responsive.
lexikos
Posts: 6653
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: Excessive page faults with Process,Exist

25 May 2017, 04:15

Someone on Stack Overflow wrote that the EnumProcesses method failed in cases like the one I mentioned, and between 32-bit and 64-bit processes.

After posting, I remembered fixing similar issues with WinGet ProcessName/ProcessPath:
1.1.02.01
...
Fixed process name/path retrieval in certain cases, including:
•Retrieving name/path of a 64-bit process from a 32-bit script.
•Retrieving name/path of an elevated process from a non-elevated process (UAC).
The key was that while GetModuleBaseName requires PROCESS_QUERY_INFORMATION and PROCESS_VM_READ, GetProcessImageFileName only requires PROCESS_QUERY_LIMITED_INFORMATION. So WinGet opens the process like this:

Code: Select all

	if (  !(hproc = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, aProcessID))  )
		// OpenProcess failed, so try fallback access; this will probably cause the
		// first method below to fail and fall back to GetProcessImageFileName.
		if (  !(hproc = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, aProcessID))  )
			return 0;
Under h := DllCall("OpenProcess", ... in your script, you can just add this:

Code: Select all

if !h
    h := DllCall("OpenProcess", "UInt", 0x1000, "Int", false, "UInt", id, "Ptr")
However, I believe the original AutoHotkey code works as far back as Windows 95. AutoHotkey doesn't run on 95 anymore, but it does run on 2000, where GetProcessImageFileName does not exist. Putting that aside, I'm not convinced that 1) the page fault delta is a problem, and 2) switching the implementation won't cause other problems.

Edit: PROCESS_QUERY_LIMITED_INFORMATION is not supported on Windows XP.
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

25 May 2017, 08:35

Thank you for all the info.
pneumatic

Re: Excessive page faults with Process,Exist

25 May 2017, 09:43

On Windows 7 my script is currently able to detect the difference between both 32-bit and 64-bit processes, including those of the same .exe name (using MD5) regardless of any Run As Administrator settings and UAC settings that I set. From what I can gather, windows 10 is much stricter in that regard and my script would probably be broken on it. Actually a few months ago I tested it briefly on a windows 10 machine and found it did not even have privileges to read and write to its own folder nor the c:\program files folder :crazy: however the process detection was working, albeit I didn't get the chance to check under all scenarios such as "Run as Administrator" processes and both 32/64-bit .exe's of the same name, which as you have said will probably not work.

I was thinking about the COM method you mentioned ("New Process Notifier") and although it would save CPU cycles at idle, still, every time a new process launches, I would still have to be looping through my entire game .exe list checking each and every one in a very small amount of time to see if it matches the newly detected process, which for me to be comfortable with in terms of CPU footprint would need at least a 10ms sleep (15.6ms apparently) in between each iteration of the loop, in which case if that is allowed to happen sporadically (eg. with windows-scheduled background processes popping in and out of existence, or opening a new browser tab spawning a new chrome.exe) then the loop will be running unpredictably, and on some systems could be a lot. So the loop really has to be CPU friendly at all times anyway, in which case it may not be so much worse to just have it running all the time. On the other hand ,maybe there is a better way to do the string matchin such as concatenating the whole process list and game .exe list to a string and using "IfInStr", but I don't know how CPU intensive it is compared to "for string in Array" method.
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

25 May 2017, 10:17

Oops somehow I had been logged out and posted as a guest. Anyway thanks again for all the great info. I will have to spend some time thinking about which way is best : COM method, shell hook or just process,Exist loop type method with sleeps. Perhaps the soft page faults are not that bad after all, I certainly can't detect any performance degradation from it. Still I feel like I don't want to release my script into the wild onto peoples machines if it's going to cause page faults when all my other processes dont have that issue. I dont want someone using my script and then thinking "what the hell is this crap causing all these page faults" and think that I am some negligent programmer. Then again, if I was to pick a typical Dell laptop off the shelf with all its bloatware it probably has millions of soft page faults at idle :D
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

25 May 2017, 11:59

Wow your COM method in the post you linked is really good, so light on CPU usage and very quick too (with reduced interval). Thanks a lot! :superhappy:

edit: must be careful with the interval though, at interval=0.1 I thought was low CPU usage but I didn't see WmiPrvSE.exe now has all the CPU usage. Looks like interval=0.75 might be a good compromise.
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

25 May 2017, 12:48

Playing around with it a bit more, I find the detection speed to CPU usage ratio is just not quite there sadly. For me I think it's going to be the Shell method, or just stick with the Process ,Exist loop for overall best speed and lowest CPU, and if the latter, then require the user runs as administrator. Actually I notice most games don't run as administrator anyway so they will be detected just fine. It appears to only be "run as administrator" programs that the dllcall method cannot detect. I really don't like UAC....was tricked into thinking it wasn't an issue but it takes a reboot before the dll method stops detecting "run as administrator" processes.
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

25 May 2017, 22:23

lexikos wrote: The key was that while GetModuleBaseName requires PROCESS_QUERY_INFORMATION and PROCESS_VM_READ, GetProcessImageFileName only requires PROCESS_QUERY_LIMITED_INFORMATION.
Unfortunately it looks like the GetProcessImageFileName only retrieves the file name and not the full path, which I need to tell 2 .exe's of the same name apart from each other. For example a few of the source engine games are sharing hl2.exe so I need to get the path of the running process to do a MD5 check on which game it actually is.

But, it's not such a big problem because it's highly unlikely that anyone would be running games in administrator mode with UAC active since every time they launch the game they would have to bypass the annoying popup screen. I see there is a Task Scheduler workaround though which some users may be using though. The Shell method looks good but seems like it would have the same limitation.
Last edited by pneumatic on 25 May 2017, 23:03, edited 1 time in total.
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

25 May 2017, 22:31

Nope! It seems you are somehow getting the path of the .exe belonging to the window in WinGet despite it that .exe being in an elevated state.

May I ask how you are doing this? I have tried using GetModuleFileNameEx but it doesn't work , presumably because that requires PROCESS_QUERY_INFORMATION and PROCESS_VM_READ
lexikos
Posts: 6653
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: Excessive page faults with Process,Exist

25 May 2017, 22:37

I would still have to be looping through my entire game .exe list checking each and every one in a very small amount of time to see if it matches the newly detected process
I think your concern is misplaced. A single call to Process Exist takes roughly 1000 times longer than if var in MatchList with a list of 10 items (or 1 item). According to my rough benchmarking, your script takes about 3 times longer than Process Exist, so roughly 3000 times longer than if var in MatchList.

If the list is very large, using an associative array (processes := {"A.exe": 1, "B.exe": 1, ...} ... if processes[name]) may be faster, but the difference is nothing compared to the cost of enumerating processes.
the loop will be running unpredictably, and on some systems could be a lot
Realistically, do new processes start on average more than once a second, or even close to that? Surely the notifications would come in much less often than you would be polling with Process Exist or EnumProcesses.

The tough question is whether the WMI method is actually more efficient (maybe it uses polling internally).
Unfortunately it looks like the GetProcessImageFileName only retrieves the file name and not the full path
You are mistaken. My guess is that you have forgotten that you are using SplitPath to strip away the directory.
The GetProcessImageFileName function returns the path in device form, rather than drive letters. For example, the file name C:\Windows\System32\Ctype.nls would look as follows in device form:
\Device\Harddisk0\Partition1\Windows\System32\Ctype.nls
[...]
To retrieve the name of the main executable module for a remote process in win32 path format, use the QueryFullProcessImageName function.
Source: GetProcessImageFileName function (Windows)
(I think QueryFullProcessImageName requires Vista or later.)

Was the somewhat large quote really necessary or relevant?
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

25 May 2017, 22:59

Thank you, that explains a lot. Your help has been absolutely invaluable here!

So in fact I should read in the whole process list to a single memory object such as a string or an array of elements, and then search through it for a match to any of my game .exe's, which are also already in memory. That way I will not have to keep calling "process exist" (or the dll equivalent) on each iteration through the array of game .exe's. That sounds a lot better. And I will use GetProcessImageFileName since it works on elevated .exe's too. And then finally I will use some MD5 hashing to tell apart .exe's of the same file name. It's all coming together now! :dance:

lexikos wrote:Realistically, do new processes start on average more than once a second, or even close to that?
Surprisingly they do! I was running the COM script "Process Notifier" you linked to and just watching various processes come in and out of existence at idle. So to be safe, the detection loop should be CPU friendly at all times. If you reduce the interval value to something like 0.1 then you should see WmiPrvSE.exe getting high CPU usage.
Last edited by pneumatic on 25 May 2017, 23:22, edited 1 time in total.
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

25 May 2017, 23:14

lexikos wrote: My guess is that you have forgotten that you are using SplitPath to strip away the directory.
Actually I was mistaken because the GetProcessImageFileName article says "Retrieves the name of the executable file for the specified process" whereas the GetModuleFileNameEx article says "Retrieves the fully qualified path" so I thought one was specifically for retrieving the path :oops:

But I probably would have forgotten about the splitpath anyway :lol:
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

26 May 2017, 01:46

lexikos wrote: The GetProcessImageFileName function returns the path in device form, rather than drive letters. For example, the file name C:\Windows\System32\Ctype.nls would look as follows in device form:
\Device\Harddisk0\Partition1\Windows\System32\Ctype.nls
[...]
To retrieve the name of the main executable module for a remote process in win32 path format, use the QueryFullProcessImageName function.
Source: GetProcessImageFileName function (Windows)
(I think QueryFullProcessImageName requires Vista or later.)
May I ask whether you did the path format conversion manually or did you end up using QueryFullProcessImageName and losing pre-Vista compatibility with WinGet ,,ProcessPath?
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

26 May 2017, 03:08

pneumatic wrote: then finally I will use some MD5 hashing to tell apart .exe's of the same file name.
lol I just realised it's not needed as the paths can be compared :oops:
pneumatic
Posts: 233
Joined: 05 Dec 2016, 01:51

Re: Excessive page faults with Process,Exist

26 May 2017, 04:58

Ok here is my final solution for fast reliable process detection with low CPU usage, no pagefaults and no leaking handles!

I have tested this script under all combinations of the following conditions:

-UAC enabled and target process running in Administrator mode
-Target processes of 32/64bit
-Target process inside c:\program files , which with UAC enabled is set to "read only" by default
-Script running from Autohotkey Unicode 32bit and 64bit
-Different target processes of the same name (eg. c:\games\halflife2\hl2.exe and c:\games\portal\hl2.exe)

In all scenarios it is working on my system! (Win7 x64)

Cons:
-Requires Vista or above
-Untested on Windows 10
-Untested on a system where the user is logged in as a guest
-With UAC enabled, Windows services' processes are not detected. However they seem to be the only exception to the rule; I have not found any processes other than windows services that fail to be detected.

Code: Select all

#SingleInstance FORCE
#Persistent
SetWorkingDir %A_ScriptDir%
SetBatchLines, -1
SetDetectionPrivileges()
SetTimer, DetectProcess, 100
return


DetectProcess:
GetRunningProcesses()

for index, element in RunningProcesses
{
	if (element = "C:\Program Files\Program\Program.exe")
		msgbox Program is running
}

return






GetRunningProcesses(){

global RunningProcesses := Object()
l := ""
s := 4096 
s := VarSetCapacity(a, s, 0)  
DllCall("Psapi.dll\EnumProcesses", "Ptr", &a, "UInt", s, "UIntP", r)


	Loop, % r // 4 
	{
		id := NumGet(a, A_Index * 4, "UInt")
	   
		h := DllCall("OpenProcess", "UInt", 0x1000, "Int", false, "UInt", id, "Ptr")
		if !h
			continue
		  
		VarSetCapacity(n, s, 0)  
		e := DllCall("QueryFullProcessImageName", "Ptr", h, "UInt", 0, "Ptr", &n, "UintP", A_IsUnicode ? s//2 : s )

		DllCall("CloseHandle", "Ptr", h) 
	   
		if (n && e)  
			RunningProcesses.Push(n)
			 
	}

}







SetDetectionPrivileges(){
Process, Exist  
h := DllCall("OpenProcess", "UInt", 0x0400, "Int", false, "UInt", ErrorLevel, "Ptr")
DllCall("Advapi32.dll\OpenProcessToken", "Ptr", h, "UInt", 32, "PtrP", t)
VarSetCapacity(ti, 16, 0)  
NumPut(1, ti, 0, "UInt") 
DllCall("Advapi32.dll\LookupPrivilegeValue", "Ptr", 0, "Str", "SeDebugPrivilege", "Int64P", luid)
NumPut(luid, ti, 4, "Int64")
NumPut(2, ti, 12, "UInt")  
r := DllCall("Advapi32.dll\AdjustTokenPrivileges", "Ptr", t, "Int", false, "Ptr", &ti, "UInt", 0, "Ptr", 0, "Ptr", 0)
DllCall("CloseHandle", "Ptr", t)  
DllCall("CloseHandle", "Ptr", h) 
hModule := DllCall("LoadLibrary", "Str", "Psapi.dll")
}

lexikos
Posts: 6653
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: Excessive page faults with Process,Exist

21 Oct 2019, 16:32

Having revisited this and tested an implementation of ProcessExist that uses EnumProcesses, I found that it was maybe 20% slower (if the process did not exist, slower by a smaller amount if it did) than the current implementation using CreateToolhelp32Snapshot. Evidently the page fault count is no indication of a performance problem. It also fails to detect services (as mentioned above). I thought it might at least decrease code size, but the compiler has surprised me by doing the opposite. So I don't think there's a single reason to use EnumProcesses.

From a script might be a little different, since the overhead of DllCall and the script itself plays a bigger role.
User avatar
jNizM
Posts: 2509
Joined: 30 Sep 2013, 01:33
GitHub: jNizM
Contact:

Re: Excessive page faults with Process,Exist

22 Oct 2019, 01:22

Whats with WTSEnumerateProcesses(Ex) function and the WTS_PROCESS_INFO(Ex) structure
How it looks like in AHK: https://www.autohotkey.com/boards/viewtopic.php?p=139983#p139983



Or with NtQuerySystemInformation function and SYSTEM_PROCESS_INFORMATION structure.
How it looks like in AHK: https://www.autohotkey.com/boards/viewtopic.php?f=6&t=62989&p=268985#p268985



Some other old code I found. Not sure if usefull

Code: Select all

DllCall("RegisterShellHookWindow", "ptr", A_ScriptHwnd)
MsgID := DllCall("RegisterWindowMessage", "str", "SHELLHOOK")
OnMessage(MsgID, "ShellMessage")
return

ShellMessage(nCode, wParam)
{
	static HSHELL_WINDOWCREATED   := 1
	static HSHELL_WINDOWDESTROYED := 2
	if (nCode = HSHELL_WINDOWCREATED) {
		;MsgBox % wParam
		if (EnumProcesses(GetWindowThreadProcessId(wParam)) = "firefox.exe")
			MsgBox % "firefox started"
	}
}

GetWindowThreadProcessId(handle)
{
	DllCall("GetWindowThreadProcessId", "ptr", handle, "uint*", ProcessID)
	return ProcessID
}

EnumProcesses(ProcessID)
{
	static hWTSAPI := DllCall("LoadLibrary", "str", "wtsapi32.dll", "ptr")

	static WTS_CURRENT_SERVER_HANDLE := 0
	static WTS_PROCESS_INFO_EX       := 1
	static WTS_ANY_SESSION           := -2

	if !(DllCall("wtsapi32\WTSEnumerateProcessesEx", "ptr",    WTS_CURRENT_SERVER_HANDLE
													, "uint*", WTS_PROCESS_INFO_EX
													, "uint",  WTS_ANY_SESSION
													, "ptr*",  buf
													, "uint*", ttl))
		throw Exception("Get information about the active processes failed", -1)

	addr := buf
	loop % ttl {
		if (NumGet(addr + 4, "uint") = ProcessID) {
			ProcessName := StrGet(NumGet(addr + 8, "ptr"))
			break
		}
		addr += 48 + (A_PtrSize * 2)
	}
	DllCall("wtsapi32\WTSFreeMemoryEx", "int", WTS_PROCESS_INFO_EX, "ptr", buf, "uint", ttl)
	return ProcessName, VarSetCapacity(addr, 0)
}
[AHK] 1.1.30.03 x64 Unicode | [WIN] 10 Pro (Version 1903) x64 | [GitHub] Profile
Donations are appreciated if I could help you
lexikos
Posts: 6653
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: Excessive page faults with Process,Exist

24 Oct 2019, 03:42

Is there some reason to consider either of those functions over the existing implementation? I just got through restating that there's nothing to fix.

Return to “Bug Reports”

Who is online

Users browsing this forum: No registered users and 6 guests