Loop, Files SPEED UP

Get help with using AutoHotkey and its commands and hotkeys
euras
Posts: 411
Joined: 05 Nov 2015, 12:56

Loop, Files SPEED UP

28 Jan 2019, 02:10

Hi, the request is to loop through all folders and their subfolders into sharedrive to find a folders, which contains at least a part of search string.
There is 4257 folders/subfolders there. And using my method it takes 410-430 seconds to find those folders. It's a long time to wait. I'm wondering is it possible to speed up the loop somehow?

Code: Select all

SetBatchLines,-1
Process, Priority,, High
Full_Path := "I:\Dat\Inbox\Customer_Folders"
Loop, Files, %Full_Path%\*.*, DR
{
	for q,w in Search ; contains the search string divided by words
	{
		if (InStr(A_LoopFileName, w)) ; a word matches with the folder name
		{
			for e,r in Results ; checks if the path is already added to Results
			{
				if (A_LoopFileName = r) {
					continue, 2
				}
			}
			Results.Push(A_LoopFileName)
			Results_FullPath.Push(A_LoopFileFullPath)
		}
	}
}
EDIT: I have reduced the code by following, but it still takes 400 seconds to perform the search..

Code: Select all

Full_Path := "I:\Dat\Inbox\Customer_Folders"
our_word := ""
for i,o in Search
	our_word .= o ","
Loop, Files, %Full_Path%\*.*, DR
{
	if A_LoopFileName contains %our_word%
	{
		Results.Push(A_LoopFileName)
		Results_FullPath.Push(A_LoopFileFullPath)
	}
}
User avatar
Xtra
Posts: 1999
Joined: 02 Oct 2015, 12:15

Re: Loop, Files SPEED UP

28 Jan 2019, 03:16

Run this:

Code: Select all

Run, %comspec% /q /c dir /s /b > print.txt, I:\Dat\Inbox\Customer_Folders
And then parse the print.txt file.

HTH
GEV
Posts: 844
Joined: 25 Feb 2014, 00:50

Re: Loop, Files SPEED UP

28 Jan 2019, 03:19

You could create an index file of those folders/files e.g. every 500 seconds using SetTimer
or start an efficient program for searching and filtering results (Windows search, Everything, UltraSearch, ...).
IMEime
Posts: 750
Joined: 20 Sep 2014, 06:15

Re: Loop, Files SPEED UP

28 Jan 2019, 03:21

your code has 10 lines
if it takes 400 sec
not all lines take 40 sec evenly
you'd better find the slowest line (command/logic) firstly - bottleneck
and change it faster
and do it again for the 2nd bottleneck....
IMEime
Posts: 750
Joined: 20 Sep 2014, 06:15

Re: Loop, Files SPEED UP

28 Jan 2019, 03:24

I did not compared though

your 2nd For could be changed into Dictionary
because they say this kind of search it should be faster
I do not know it clearly
you just test it
euras
Posts: 411
Joined: 05 Nov 2015, 12:56

Re: Loop, Files SPEED UP

28 Jan 2019, 04:05

Xtra wrote:
28 Jan 2019, 03:16
Run this:

Code: Select all

Run, %comspec% /q /c dir /s /b > print.txt, I:\Dat\Inbox\Customer_Folders
And then parse the print.txt file.
HTH
I have tried this:

Code: Select all

Run, %comspec% /q /c dir /s /b > C:\Users\Desktop\Projects\print.txt, I:\Dat\Customer_Folders
But it doesn't give me I:\Dat\Customer_Folders folders/subfolders. Instead of that, it gives me a list of folders in I:\App
hd0202
Posts: 167
Joined: 04 Oct 2013, 03:07
Location: Germany near Cologne

Re: Loop, Files SPEED UP

28 Jan 2019, 05:21

I think, in the proposed code "I:\Dat\Customer_Folders" is defined as working directory, but it must be part of "dir"

Code: Select all

Run, %comspec% /q /c dir /s /b I:\Dat\Customer_Folders\*.* > C:\Users\Desktop\Projects\print.txt
Hubert
DRocks
Posts: 559
Joined: 08 May 2018, 10:20

Re: Loop, Files SPEED UP

28 Jan 2019, 06:30

Could you create a big global variable that lies in your computers memory on script start-up?
This variable/object could contain the complete list of folders that seems to loop from afresh each time in the current state.
Of course the first run of the script would take more time, but all the further requests could probably be faster.

...After this would be loaded into memory, I think you could use another parsing method but directly comparing your search field query with the existing variable that contains the folders info?

Let me know if this sounds an idea.
euras
Posts: 411
Joined: 05 Nov 2015, 12:56

Re: Loop, Files SPEED UP

28 Jan 2019, 06:36

hd0202 wrote:
28 Jan 2019, 05:21
I think, in the proposed code "I:\Dat\Customer_Folders" is defined as working directory, but it must be part of "dir"

Code: Select all

Run, %comspec% /q /c dir /s /b I:\Dat\Customer_Folders\*.* > C:\Users\Desktop\Projects\print.txt
Hubert
As I mentioned, there is 4257 directories. If I want to get a .txt list of all folders/subfolders and I use the code bellow, it takes 380 seconds for cmd to complete the list... Can I speed up that cmd somehow? why it takes so long time to complete?

Code: Select all

RunWait, %comspec% /q /c dir /s /b /o:n /a:d I:\Dat\Inbox\Customer_Folders\*.* > C:\Users\Desktop\Projects\print.txt
SOTE
Posts: 1129
Joined: 15 Jun 2015, 06:21

Re: Loop, Files SPEED UP

28 Jan 2019, 06:58

euras wrote:
28 Jan 2019, 06:36
hd0202 wrote:
28 Jan 2019, 05:21
I think, in the proposed code "I:\Dat\Customer_Folders" is defined as working directory, but it must be part of "dir"

Code: Select all

Run, %comspec% /q /c dir /s /b I:\Dat\Customer_Folders\*.* > C:\Users\Desktop\Projects\print.txt
Hubert
As I mentioned, there is 4257 directories. If I want to get a .txt list of all folders/subfolders and I use the code bellow, it takes 380 seconds for cmd to complete the list... Can I speed up that cmd somehow? why it takes so long time to complete?

Code: Select all

RunWait, %comspec% /q /c dir /s /b /o:n /a:d I:\Dat\Inbox\Customer_Folders\*.* > C:\Users\Desktop\Projects\print.txt
Have you ever used an Anti-Virus scanner or did a file search with File Explorer in Windows? Things on your computer don't all happen instantly.

You could try putting this at the top of the script and see what happens.

Code: Select all

#MaxMem 512  ; that's a large chunk of memory for a script
SetBatchLines, -1  ; affects CPU utilization... script will run at max speed
ListLines Off  ; helps with speed
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: Loop, Files SPEED UP

28 Jan 2019, 07:15

- You could use SetCapacity on the objects, although that might not make too much difference.
- I've noticed that if I loop through files, it can take a long time, but that if I do the same search again immediately afterwards, it's much faster. [EDIT: When looping through files and using FileRead, to search for text inside files, tested on Windows 7.]
- So, periodically looping through the files, that might make future loops faster.
- Also, I've noticed that copying lots of small files is slower than copying a few big files. So, if possible, using a smaller number of bigger files could make things faster.
Last edited by jeeswg on 28 Jan 2019, 07:30, edited 1 time in total.
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
pneumatic
Posts: 299
Joined: 05 Dec 2016, 01:51

Re: Loop, Files SPEED UP

28 Jan 2019, 07:25

I have noticed that on my Windows 7 machine with User Account Control disabled, looping through files is very fast. On Windows 10 I have found that looping through files is extremely slow. In my case I could only loop about 8 files before the "slow down" kicks in. Also I noticed the same thing as jeeswg where after the first time looping through the files, it suddenly gets fast again. But on Windows 7 it's fast the first time. imo something changed in Windows 10 which produces this behaviour and it has caused me a lot of headache in my app where I had to rewrite a lot of stuff to avoid it.
euras
Posts: 411
Joined: 05 Nov 2015, 12:56

Re: Loop, Files SPEED UP

28 Jan 2019, 07:38

I have played with many methods to loop faster with AHK or CMD, but the best result I got was about 380 seconds. So my last hope is to make a DLL call for Windows Search, but I don't know how. Maybe someone have it in their librarys and can post it?
DRocks
Posts: 559
Joined: 08 May 2018, 10:20

Re: Loop, Files SPEED UP

28 Jan 2019, 08:12

In my opinion windows search is slow too.

So I wonder how can google give 1 million results in few seconds over the internet?
IMEime
Posts: 750
Joined: 20 Sep 2014, 06:15

Re: Loop, Files SPEED UP

28 Jan 2019, 08:23

Google..
Maybe..
Thousands of smart engineers
Thousands of expensive computers
Nothing else..
You (Including me), cheap computers and spoonful of knowledge.
I guess, you are making better performance than Google, Maybe (or not)
safetycar
Posts: 125
Joined: 12 Aug 2017, 04:27

Re: Loop, Files SPEED UP

29 Jan 2019, 00:33

Is your search as slow on C:\ ? I suspect of the config of th I:\ drive.
SOTE
Posts: 1129
Joined: 15 Jun 2015, 06:21

Re: Loop, Files SPEED UP

29 Jan 2019, 01:16

safetycar wrote:
29 Jan 2019, 00:33
Is your search as slow on C:\ ? I suspect of the config of th I:\ drive.
Good point. Before saying your AutoHotkey script is "slow", look at your computer. Do you have some low spec CPU and little RAM? Do you have 15 other programs open, all taking CPU and RAM resources? Is the I:\ drive some external hard drive using the old and slow USB 2.0 standard, versus say USB 3.0? And are searches on your computer slow, in general? Say if you use some other program or used a portable Anti-Virus scanner program, does that take a long time? Etc...
User avatar
hasantr
Posts: 678
Joined: 05 Apr 2016, 14:18
Location: İstanbul

Re: Loop, Files SPEED UP

29 May 2020, 07:42

deletet....

Return to “Ask For Help”

Who is online

Users browsing this forum: Albireo, ananthuthilakan, c7aesa7r, Google [Bot], mikeyww, SteveTranslator, teadrinker, Tulevik and 67 guests