Efficiently read file into byte array Topic is solved

[pneumatic]

Hello, I would like to read a 6MB binary file into an array of raw bytes.

I am currently doing it like so, but it is very slow (around 5 minutes) and uses 467MB of RAM.

Code: Select all

file := FileOpen( myfile , "r")
filebytes := []
loop
{
	if file.RawRead(byte,1) == 1
		filebytes[A_Index] := Format("{:02X}", NumGet(byte, "int"))
	else 
	{
		msgbox % "Reached EOF. Length = " . filebytes.length() . " bytes"
		break
	}	
}

Is there a more efficient way?

Thanks!

[teadrinker]

Try this:

Code: Select all

file := FileOpen(myfile , "r")
file.Pos := 0
SafeArray := ComObjArray(VT_UI1 := 0x11, file.Length)
file.RawRead(NumGet(ComObjValue(SafeArray) + 8 + A_PtrSize), file.Length)
file.Close()
MsgBox, % "The first byte: " . Format("{:#x}", SafeArray[0]) ; <-- null based array

[pneumatic]

Try this:

Code: Select all

file := FileOpen(myfile , "r")
file.Pos := 0
SafeArray := ComObjArray(VT_UI1 := 0x11, file.Length)
file.RawRead(NumGet(ComObjValue(SafeArray) + 8 + A_PtrSize), file.Length)
file.Close()
MsgBox, % "The first byte: " . Format("{:#x}", SafeArray[0]) ; <-- null based array

Thanks, that's very fast, but ultimately not useful to me as I need the array to be compatible with all the standard ahk array/object funcs.

Copying it into an ahk array takes around 7 seconds with setbatchlines -1, which is not too bad, but then my original code does it in 14 seconds with setbatchlines -1.

Copying it to an ahk array causes memory usage to increase from 7MB to 470MB. It seems that can be reduced to 280MB with filebytes.SetCapacity(A_Index, 4). But if I go lower than 4, the byte becomes truncated. I guess Format("{:02X}", NumGet(byte, "int") is increasing the size to 4 bytes or something?

edit: I adapted my script to use ComObjArray's funcs, so it's all good

I am still curious to know if there is a way to do it in ahk's standard library.

[pneumatic]

SafeArray := ComObjArray(VT_UI1 := 0x11, file.Length)

edit: I adapted my script to use ComObjArray's funcs, so it's all good

Nope, AHK doing silly things internally with numbers again:

Code: Select all

filebytes := ComObjArray(VT_UI1 := 0x11, 8)

;works -- says looped 8 times
loop % filebytes.maxindex()+1
	count++
msgbox % "looped " . count . " times" 


;fails -- says 9 <= 8
if 9 > filebytes.maxindex()+1
	msgbox 9 > 8
else msgbox 9 <= 8

[teadrinker]

Use parentheses.
https://www.autohotkey.com/docs/commands/IfExpression.htm

[pneumatic]

Use parentheses.
https://www.autohotkey.com/docs/commands/IfExpression.htm

Thanks, but that doesn't work.

However if I use parentheses and percent signs, then it works.

[pneumatic]

Use parentheses.
https://www.autohotkey.com/docs/commands/IfExpression.htm

Doesn't work... replace 9 with 7 and now the problem is reversed (thinks 7 > 8).

[teadrinker]

Show your code.

[pneumatic]

Code: Select all

filebytes := ComObjArray(VT_UI1 := 0x11, 8)

;works -- says looped 8 times
loop % filebytes.maxindex()+1
	count++
msgbox % "looped " . count . " times" 


;fails -- says 7 > 8
maxindex := filebytes.maxindex()+1
if (7 > %maxindex%)
	msgbox ,  7 > 8
else
	msgbox , 7 <= 8

[pneumatic]

Ok now it's working, no need for the percent signs.

[pneumatic]

Just for completeness, I found that FileRead with the *C option is about twice as fast as reading it into a ComObjArray. Then using NumGet to retrieve the values.