AutoHotkey Community
It is currently May 26th, 2012, 8:47 pm

View unanswered posts | View active topics


Board index » AutoHotkey » Support

All times are UTC [ DST ]



Post new topic Reply to topic  Page 1 of 1
  [ 10 posts ] 
  Print view | E-mail friend Previous topic | Next topic 
Author Message
Costas
PostPosted: August 28th, 2009, 6:27 pm 
Offline

Joined: March 31st, 2009, 12:28 pm
Posts: 70
I love AHK and have found many uses for it. Recently, I decided to use it to add a header record to a number of HTML files that are automatically generated by another software tool.

AHK works great for this. The catch is that I need to read about 25 input files and some of these input files contain thousands of records.

I quickly learned that my AHK script would work, but it takes a long time to complete (over 30 minutes for all files).

I then did some performance tests.

If I have an input file of 3,000 records, I can read the entire file with AHK in less than one second (just read, no writes)

When I add the FileAppend command to write the records to a new file (to add my 1 new header record) the run time jumps to 30 seconds.

Is there a way to programmatically reduce this time? Am I missing something? Adding faster hardware is not an option due to budget constraints.

Thanks for your ideas,
Costas


Report this post
Top
 Profile  
Reply with quote  
Leef_me
 Post subject:
PostPosted: August 28th, 2009, 7:13 pm 
Online

Joined: April 8th, 2009, 7:49 pm
Posts: 6065
Location: San Diego, California
Quote:
Is there a way to programmatically reduce this time?
Can you post your script? Perhaps the group can spot an improvement. :!:

If that proves fruitless, you might be able to use a DOS level command (I haven't tried - - try seaching for 'comspec' in help)
Quote:
COPY [/D] [/V] [/N] [/Y | /-Y] [/Z] [/L] [/A | /B ] source [+ source [/A | /B] [+ ...]] [destination [/A | /B]]

To append files, specify a single file for destination, but multiple files for source (using wildcards or file1+file2+file3 format).


Report this post
Top
 Profile  
Reply with quote  
[VxE]
 Post subject:
PostPosted: August 28th, 2009, 8:09 pm 
Offline
User avatar

Joined: October 7th, 2006, 8:45 am
Posts: 3330
Location: Simi Valley, CA
Using a Loop, Read, InputFile, OutputFile and using FileAppend inside the loop to append to the OutputFile will dramatically increase the speed of multiple FileAppends. See the (linked) docs for details.
edit:] link fix, sry about that.

_________________
Ternary (a ? b : c) guide     TSV Table Manipulation Library
Post code inside [code][/code] tags!


Last edited by [VxE] on August 29th, 2009, 7:19 pm, edited 1 time in total.

Report this post
Top
 Profile  
Reply with quote  
Costas
 Post subject:
PostPosted: August 29th, 2009, 2:42 am 
Offline

Joined: March 31st, 2009, 12:28 pm
Posts: 70
Here is the code that I trimmed down for performance testing. After several tests, it appears that the FileAppend is the command that is very time consuming. Any ideas on how to speed this up would be most appreciated.

Thanks,
Costas

;~~~~~~~~~~~ START ~~~~~~~~~~~~~~~~~~~~~~~~

startTime:=A_TickCount

Input_File = C:\Input_File.TXT
Output_File = C:\Output_File.TXT

Filedelete %Output_File%

;~~~~~~~~
; Read input file and add a new record before the <HEAD> record as output file is created

loop, read, %Input_File%

{

IfInString, A_LoopReadLine, <HEAD>
{
FileAppend , Newly-Added-Test-Record `n, %Output_File%
}

FileAppend , %A_LoopReadLine% `n, %Output_File%

}


;~~~~~~~


EndTime:=A_TickCount

TimeInScript:=(A_TickCount-startTime)/1000
Msgbox Time in Script = %TimeInScript% seconds `n`n

EXITAPP

;~~~~~~~~~~~ END ~~~~~~~~~~~~~~~~~~~~~~~~


Report this post
Top
 Profile  
Reply with quote  
horntail
 Post subject:
PostPosted: August 29th, 2009, 2:56 am 
Offline

Joined: August 3rd, 2009, 4:44 pm
Posts: 69
Location: UK
test it again with this code, it should be faster and its what [VxE] meant in his post above:

Code:
startTime:=A_TickCount

Input_File = C:\Input_File.TXT
Output_File = C:\Output_File.TXT

Filedelete %Output_File%

;~~~~~~~~
; Read input file and add a new record before the <HEAD> record as output file is created

loop, read, %Input_File%, %Output_File%

{

IfInString, A_LoopReadLine, <HEAD>
{
FileAppend , Newly-Added-Test-Record `n
}

FileAppend , %A_LoopReadLine% `n

}


;~~~~~~~


EndTime:=A_TickCount

TimeInScript:=(A_TickCount-startTime)/1000
Msgbox Time in Script = %TimeInScript% seconds `n`n

EXITAPP


Report this post
Top
 Profile  
Reply with quote  
Costas
 Post subject:
PostPosted: August 29th, 2009, 1:24 pm 
Offline

Joined: March 31st, 2009, 12:28 pm
Posts: 70
Horntail,

Thanks for the help, I really appreciate it.

I made this minor change as you suggested and the run time was reduced significantly.

Being a curious person, I would like to understand why this minor change made such a large impact.

Thanks again,
Costas


Report this post
Top
 Profile  
Reply with quote  
why_bother_helping
 Post subject:
PostPosted: August 29th, 2009, 1:46 pm 
Costas wrote:
Being a curious person, I would like to understand why this minor change made such a large impact.

Did you even read your own thread? The answer was provided by VxE here . Read the links he provided!! :evil:


Report this post
Top
  
Reply with quote  
Costas
 Post subject:
PostPosted: August 29th, 2009, 2:23 pm 
Offline

Joined: March 31st, 2009, 12:28 pm
Posts: 70
Sorry...

I did read what VXE posted, but I just didn't understand. I am still fairly new to AHK. The example that Horntail posted better helped me to understand what to try.

Now I would like to understand the underlying reason why this minor change makes such a big difference.

Thanks,
Costas


Report this post
Top
 Profile  
Reply with quote  
Guest
 Post subject:
PostPosted: August 29th, 2009, 2:52 pm 
Costas wrote:
Sorry...

I did read what VXE posted, but I just didn't understand. I am still fairly new to AHK. The example that Horntail posted better helped me to understand what to try.

Now I would like to understand the underlying reason why this minor change makes such a big difference.

Thanks,
Costas

You read this???
Loop (read file contents)
Quote:
Within the loop's body, use the FileAppend command with only one parameter (the text to be written) to append to this special file. Appending to a file in this manner performs better than using FileAppend in its 2-parameter mode because the file does not need to be closed and re-opened for each operation.


Report this post
Top
  
Reply with quote  
Costas
 Post subject:
PostPosted: August 29th, 2009, 5:41 pm 
Offline

Joined: March 31st, 2009, 12:28 pm
Posts: 70
No, I had not read this. When I click on the first of the four links in the post that VXE added, I receive a "404 Page Not Found" error message.

The link that you show works fine and leads to the explanation of why this minor change is going to have such a large impact.

Thanks for the help, I really appreciate it.

Costas


Report this post
Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 1
  [ 10 posts ] 

Board index » AutoHotkey » Support

All times are UTC [ DST ]

Who is online

Users browsing this forum: coinman, joetazz, Leef_me, tidbit, Yahoo [Bot] and 64 guests

You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Group