AutoHotkey Community

It is currently May 26th, 2012, 11:54 pm

All times are UTC [ DST ]




Post new topic Reply to topic  [ 10 posts ] 
Author Message
PostPosted: September 25th, 2009, 3:56 am 
Offline

Joined: August 12th, 2009, 6:07 pm
Posts: 26
Would anyone have any suggestions on why a script would run extremely slow on one computer (XP, 1.6 Ghz Duo) and fast on the other (vista)? The program does some basic text file manipulation, and runs extremely slow when Loop reading a text file with an inner Loop parse routine.

The computer with Vista takes 27 seconds to complete, the XP computer is taking over 20 minutes operating on the same text file, which is seemingly impossible!

I have never noticed a difference between the two before.

The longest part of the code...

Code:
Loop 150
      {
         Loop,parse,fieldlist,`,   
         {
            Num+=%A_loopfield%
            Field:=pcsSplit%Num%
            Fileappend,%field%`,,%A_programfiles%\WIA\Data\MainData1.txt
            Num-=%A_loopfield%
         }
         
         num+=1434
      }


Report this post
Top
 Profile  
Reply with quote  
PostPosted: September 25th, 2009, 6:03 am 
Offline

Joined: August 13th, 2006, 6:45 am
Posts: 355
Location: Germany
Uminnsky wrote:
The program does some basic text file manipulation
As I know from an earlier post your program does some hundred thousend basic text file manipulation. Your problem on the very slow computer is the disk and the filesystem. Have a look at my post "Performance of file ouput (FileAppend)" to find a better solution.

Hubert


Report this post
Top
 Profile  
Reply with quote  
PostPosted: September 25th, 2009, 6:56 am 
hd0202 wrote:
Uminnsky wrote:
Have a look at my post "Performance of file ouput (FileAppend)" to find a better solution.

You could/should give a link when refering to another thread.

For the op, http://www.autohotkey.com/forum/topic45771.html


Report this post
Top
  
Reply with quote  
 Post subject:
PostPosted: September 25th, 2009, 1:38 pm 
Offline
User avatar

Joined: December 21st, 2007, 3:14 pm
Posts: 3826
Location: Louisville KY USA
like EVERY application ahk relies on available interupts for cpu time
like EVERY application that relies on data written to a disk the speed of the hard drive is a factor
Like EVERY application currently running processes affect the above

troubleshoot what has changed and you will know why performance is affected

_________________
No matter what your oppinion Please join this discussion
Formal request to Polyethene
Image


Report this post
Top
 Profile  
Reply with quote  
PostPosted: September 25th, 2009, 2:55 pm 
Offline

Joined: July 6th, 2009, 9:58 pm
Posts: 678
Uminnsky wrote:
Would anyone have any suggestions on why a script would run extremely slow on one computer (XP, 1.6 Ghz Duo) and fast on the other (vista)?


Drive fragmentation and available space.


Report this post
Top
 Profile  
Reply with quote  
PostPosted: September 30th, 2009, 2:48 am 
Offline

Joined: August 12th, 2009, 6:07 pm
Posts: 26
hd0202 wrote:
Uminnsky wrote:
The program does some basic text file manipulation
As I know from an earlier post your program does some hundred thousend basic text file manipulation. Your problem on the very slow computer is the disk and the filesystem. Have a look at my post "Performance of file ouput (FileAppend)" to find a better solution.

Hubert


Thanks for referencing this.

I made this switch, the commented was my old code, the speed difference in my situation was 47 seconds down to 18 seconds on the data tested:

Code:
Loop,read, % fflist, % output
         {
            Num+=%A_loopreadline%
            Field:=pcsSplit%Num%
            Fileappend, % field "`,"
            Num-=%A_loopreadline%
         }
         
;~          Loop,parse,fieldlist,`,   
;~          {
;~             Num+=%A_loopfield%
;~             Field:=pcsSplit%Num%
;~             Fileappend,%field%`,,%A_programfiles%\WIA\Data\MainData1.csv
;~             Num-=%A_loopfield%
;~          }


This is pretty huge considering I plan for my program to build a database on thousands of files which would take anywhere from 8-12 hours. Just so I understand better what is going on, by switching from loop parse to loop read I am preventing the file from opening when it appends? Any suggestions on how I should or coudl speed up my next section of code...

Code:
Loop, read, %A_programfiles%\WIA\Data\MainData.csv
   {
      Bob:=A_loopreadline
      Stringsplit, fish, A_loopreadline, `,
      Loop, read, %A_programfiles%\WIA\Data\results.csv               ; Import the results into the Data file
      {
         ifinstring,A_loopreadline, %fish1%`,%fish2%`,%fish9%
         fileappend,%bob%`,%A_loopreadline%`n,%A_programfiles%\WIA\Data\Data.csv
      }   
   }


Report this post
Top
 Profile  
Reply with quote  
PostPosted: September 30th, 2009, 8:34 pm 
Offline

Joined: August 13th, 2006, 6:45 am
Posts: 355
Location: Germany
For youzr first problem: did you get the same data as before you changed the code? I say NO without seeing any of your data!
Here is my suggestion with some comments:
Code:
; dummyfile.txt must contain minimal 1 record of any content, it is not the data to be processed
; here the output file is opened
Loop,read, duumyfile.txt, %A_programfiles%\WIA\Data\MainData1.txt
(
   Loop 150
   {
      Loop,parse,fieldlist,`,   
      {
         Num+=%A_loopfield%
         Field:=pcsSplit%Num%

;        this command opens, (positions to the end and) appends and closes
;        150 times <the number of fields> the field data
;        ------------------------------------------------------------
;        Fileappend,%field%`,,%A_programfiles%\WIA\Data\MainData1.txt
;        ------------------------------------------------------------
;
;        this command really only appends 150 time <the number of fields>
;        the field data as the file is kept open all the time
;        --------------------
         Fileappend,%field%`,
;        --------------------
;
         Num-=%A_loopfield%
      }
     
      num+=1434
   }
   break  ; the dummyfile loop after the only or first record
}
; here the ouput file is closed

I hope that helps.

Hubert


Report this post
Top
 Profile  
Reply with quote  
PostPosted: September 30th, 2009, 8:54 pm 
Offline

Joined: August 13th, 2006, 6:45 am
Posts: 355
Location: Germany
And here is my suggestion for your second problem (also not tested as I do not have the data):
Code:
FileRead, results, %A_programfiles%\WIA\Data\results.csv  ; get results into variable with one read

Loop,read, dummyfile.txt, %A_programfiles%\WIA\Data\Data.csv
(
   FileRead, maindata, %A_programfiles%\WIA\Data\MainData.csv  ; get data into variable with one read
   Loop, parse, maindata, `n, `r
   {
      if a_loopfield =
         continue
      Bob:=A_loopfield
      Stringsplit, fish, A_loopfield, `,
      Loop, parse, results, `n, `r
      {
         if a_loopfield =
            continue
         ifinstring,A_loopfield, %fish1%`,%fish2%`,%fish9%
            fileappend,%bob%`,%A_loopfield%`n
      }   
   }
   break
}

It also uses the dummyfile to hold the output file open.

I hope this also helps.

Hubert


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: September 30th, 2009, 11:30 pm 
Offline

Joined: December 21st, 2008, 7:29 pm
Posts: 181
Why use FileAppend in the loop? It would be faster if you kept the output in a variable during the loop and write it out in one go afterwards.

Code:
FullText =
Loop 150
      {
         Loop,parse,fieldlist,`,   
         {
            Num += %A_loopfield%
            Field := pcsSplit%Num%
            FullText := FullText . Field . ","
            Num -= %A_loopfield%
         }
         
         num+=1434
      }
FileAppend, %FullText%, %A_programfiles%\WIA\Data\MainData1.txt


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: October 2nd, 2009, 4:25 am 
entropic wrote:
Why use FileAppend in the loop? It would be faster if you kept the output in a variable during the loop and write it out in one go afterwards.

Code:
FullText =
Loop 150
      {
         Loop,parse,fieldlist,`,   
         {
            Num += %A_loopfield%
            Field := pcsSplit%Num%
            FullText := FullText . Field . ","
            Num -= %A_loopfield%
         }
         
         num+=1434
      }
FileAppend, %FullText%, %A_programfiles%\WIA\Data\MainData1.txt


Thanks, this will help me a lot. I have been meaning to research on how the syntax for this would look, but just never got around to it. From some initial testing it is significantly faster, and there are a lot of places I append during loops. Thanks!


Report this post
Top
  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

All times are UTC [ DST ]


Who is online

Users browsing this forum: hyper_, Leef_me, patgenn123, Pulover, RaptorX, rbrtryn, XstatyK, Yahoo [Bot] and 20 guests


You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Group