Find string in a very large file

Get help with using AutoHotkey and its commands and hotkeys
Elaphe666
Posts: 122
Joined: 28 Nov 2015, 11:39

Find string in a very large file

03 Jun 2017, 05:17

I need to search for a string in a text file which has 60.000 lines. Is there a better way than using fileread, I mean, that is fast and doesn't load the entire file into memory?
neomulemi6
Posts: 216
Joined: 30 Jun 2016, 06:01

Re: Find string in a very large file

03 Jun 2017, 06:34

You could use a parsing loop and search each line individually.
https://autohotkey.com/docs/commands/LoopParse.htm
User avatar
MilesAhead
Posts: 232
Joined: 03 Oct 2013, 09:44

Re: Find string in a very large file

03 Jun 2017, 08:13

How long are the lines? When I calculate 60000 lines of 256 chars each it is still less than 15 MB. So call it 30 MB for double byte characters. Seems like it may be fine to load the entire file into a variable and search it. At least worth a benchmark to compare it to reading the file in a loop.
"My plan is to ghostwrite my biography. Then hire another writer to put his
name on it and take the blame."

- MilesAhead
User avatar
jeeswg
Posts: 6469
Joined: 19 Dec 2016, 01:58
Location: UK

Re: Find string in a very large file

03 Jun 2017, 09:24

Ideas:
- read file in chunks using a File object (be aware that your search term, could be split across 2 chunks) (also useful if you know for example that the search term is not in the first half of the file)
- read the whole file using FileRead, but set the process priority to low first to take up less CPU
- split the file into smaller files if you use it a lot (or have it in alphabetical order or some other order to allow you to only read part of it)
- try the InFile function (case sensitive) here, (I haven't tested it,) 32-bit only:
Machine code binary buffer searching regardless of NULL - Scripts and Functions - AutoHotkey Community
https://autohotkey.com/board/topic/2362 ... s-of-null/

Btw as was said above 60,000 lines isn't necessarily that much.
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
User avatar
MilesAhead
Posts: 232
Joined: 03 Oct 2013, 09:44

Re: Find string in a very large file

03 Jun 2017, 09:46

jeeswg wrote: Btw as was said above 60,000 lines isn't necessarily that much.
Just call FileGetSize as a sanity check. If it is on the large side use the loop method. Or you could amaze your friends and do API calls to memory map the file(for those with too much time on their hands.) :)
"My plan is to ghostwrite my biography. Then hire another writer to put his
name on it and take the blame."

- MilesAhead

Return to “Ask For Help”

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], vsub and 210 guests