 |
AutoHotkey Community
Let's help each other out
|
| View previous topic :: View next topic |
| Author |
Message |
PhiLho
Joined: 27 Dec 2005
Posts: 6721
Location: France (near Paris)
|
 Posted: Mon Jan 23, 2006 2:39 pm Post subject: Binary file reading and writing |
 |
|
Laszlo already provided some nice code to do this, and actually I "stole" (with due credit) a lot of code from his work, thanks!
So, why does I publish this? I plan to parse some binary headers, so I need to do several operations on the same file. Laszlo's routines open the file, perform the operation and close the file. Mine open the file, keep the file handle around for various operations, and close the file.
Laszlo's routines are perfectly in the spirit of most AHK commands, like FileAppend: you don't have to remember to open or close the file.
My routines seek a little more performance, even if I am not sure if this is really noticeable.
Well, at least, I improved my DllCall knowledge, and I have some little bonus like allowing relative moves, calling SetFilePointer only if requested (great for sequential reading/writing), no call to GetFileSize if not needed, etc. Plus I had fun making some tricks like improving the decimal to 2-digit hex conversion... 
Caveat: by design, these functions may malfunction with files larger than 2GB... (videos, DVD ISO images, server logs...)
Remark 1: it can still work if you only want to read the first few bytes to parse the header.
Remark 2: when dealing with such files, you should better switch to a compiled language, or perhaps to Perl...
You can download BinReadWrite.ahk and TestBinReadWrite.ahk to avoid selecting the code, copying it and pasting it in an editor, loosing tabs...
BinReadWrite.ahk
| Code: | /*
BinReadWrite.ahk
Routines to read and write binary data from/to files.
Based on original functions written by Laszlo
http://www.autohotkey.com/forum/viewtopic.php?t=4604
TODO: Perhaps set a lastError variable to explicit the errors.
// by Philippe Lhoste <PhiLho(a)GMX.net> http://Phi.Lho.free.fr
// File/Project history:
1.03.000 -- 2006/02/15 (PL) -- Moved Bin2Hex & Hex2Bin to DllCallStruct, apply code rules.
1.02.000 -- 2006/01/24 (PL) -- Slight change of the API: all functions return -1 if error.
Integrated Laszlo suggestions on improving Bin2Hex and Hex2Bin.
1.01.000 -- 2006/01/23 (PL) -- Declaration of the local variables, to get access to global
WinAPI constants, and for consistency.
1.00.000 -- 2006/01/19 (PL) -- Rewrote the functions to separate opening and closing,
allowing efficient multiple operations.
*/
; WinAPI constants
INVALID_HANDLE_VALUE = -1
INVALID_FILE_SIZE = 0xFFFFFFFF
FILE_BEGIN = 0
FILE_CURRENT = 1
FILE_END = 2
/*
// Open the file for reading.
// Return the file handle to provide in further read operations and in the final close operation,
// or INVALID_HANDLE_VALUE if an error was found.
*/
OpenFileForRead(_filename)
{
local handle
handle := DllCall("CreateFile"
, "Str", _filename ; lpFileName
, "UInt", 0x80000000 ; dwDesiredAccess (GENERIC_READ)
, "UInt", 3 ; dwShareMode (FILE_SHARE_READ|FILE_SHARE_WRITE)
, "UInt", 0 ; lpSecurityAttributes
, "UInt", 3 ; dwCreationDisposition (OPEN_EXISTING)
, "UInt", 0 ; dwFlagsAndAttributes
, "UInt", 0) ; hTemplateFile
If (handle = INVALID_HANDLE_VALUE or handle = 0)
{
ErrorLevel = -1
}
IfNotEqual ErrorLevel, 0, Return INVALID_HANDLE_VALUE ; Couldn't open the file
Return handle
}
/*
// Open the file for writing.
// Return the file handle to provide in further write operations and in the final close operation,
// or INVALID_HANDLE_VALUE if an error was found.
*/
OpenFileForWrite(_filename)
{
local handle
handle := DllCall("CreateFile"
, "Str", _filename ; lpFileName
, "UInt", 0x40000000 ; dwDesiredAccess (GENERIC_WRITE)
, "UInt", 3 ; dwShareMode (FILE_SHARE_READ|FILE_SHARE_WRITE)
, "UInt", 0 ; lpSecurityAttributes
, "UInt", 4 ; dwCreationDisposition (OPEN_ALWAYS: create if not exists)
, "UInt", 0 ; dwFlagsAndAttributes
, "UInt", 0) ; hTemplateFile
If (handle = INVALID_HANDLE_VALUE or handle = 0)
{
ErrorLevel = -1
}
IfNotEqual ErrorLevel, 0, Return INVALID_HANDLE_VALUE ; Couldn't open the file
Return handle
}
/*
// Close the file.
*/
CloseFile(_handle)
{
local result
result := DllCall("CloseHandle"
, "UInt", _handle)
If (result = 0)
{
ErrorLevel = -1
}
}
/*
// Get the size of the opened file, in bytes.
// Limited to 4GB, so it is more limited that AHK's FileGetSize.
// It is here for consistency, and because it accepts a file handle instead of a path.
//
// Return the size in bytes, -1 if there was an error.
*/
GetFileSize(_handle)
{
local fileSize
fileSize := DllCall("GetFileSize"
, "UInt", _handle
, "UInt", 0)
If (fileSize = INVALID_FILE_SIZE)
{
ErrorLevel = -1
}
IfNotEqual ErrorLevel, 0, Return -1
Return fileSize
}
/*
// Move the file pointer in the file to the given offset relative to moveMethod.
//
// moveMethod can be FILE_BEGIN, FILE_CURRENT or FILE_END.
// If moveMethod is -1, nothing is done (default, for operations at current position).
// To get the current position, call this function with just FILE_CURRENT (null offset).
// offset can be positive (toward end of the file) or negative (toward start of the file).
//
// Return -1 if there was an error, the new file pointer position if OK.
// Note: Currently it doesn't work for files larger than 2GB...
*/
MoveInFile(_handle, _moveMethod=-1, _offset=0)
{
local result
result = %INVALID_FILE_SIZE%
if (_moveMethod != -1)
{
result := DllCall("SetFilePointer"
, "UInt", _handle ; hFile
, "Int", _offset ; lDistanceToMove
, "UInt", 0 ; lpDistanceToMoveHigh
, "UInt", _moveMethod) ; dwMoveMethod
if (result = -1) ; INVALID_SET_FILE_POINTER
{
ErrorLevel = -1
}
IfNotEqual ErrorLevel, 0, Return -1 ; Couldn't make the move
}
Return result
}
/*
// Write in a file opened for writing.
//
// Move to position given by moveMethod and offset
// (by default stand at current position) and
// write byteNb bytes from data (all data if byteNb = 0;
// data contains binary bytes that can be a string or
// raw bytes generated from hexa data with the Hex2Bin routine).
//
// moveMethod, defaulting to -1 (no move, write at current position),
// can also be FILE_BEGIN, FILE_CURRENT or FILE_END.
// offset can be positive (toward end of file) or negative (toward beginning of file).
//
// Return the number of bytes written (-1 if there was an error).
*/
WriteInFile(_handle, ByRef @data, _byteNb=0, _moveMethod=-1, _offset=0)
{
local dataSize, result, written
_offset := MoveInFile(_handle, _moveMethod, _offset)
IfNotEqual ErrorLevel, 0, Return -1 ; Couldn't make the move
dataSize := VarSetCapacity(@data) ; Get the capacity (>= used length!)
If (_byteNb < 1 or _byteNb > dataSize)
{
byteNb := dataSize
}
result := DllCall("WriteFile"
, "UInt", _handle ; hFile
, "Str", @data ; lpBuffer
, "UInt", _byteNb ; nNumberOfBytesToWrite
, "UInt *", written ; lpNumberOfBytesWritten
, "UInt", 0) ; lpOverlapped
if (result = 0 or written < _byteNb)
{
ErrorLevel = -2
}
IfNotEqual ErrorLevel, 0, Return -1 ; Couldn't write in the file
Return written
}
/*
// Read from a file opened for reading.
//
// Move to position given by moveMethod and offset
// (by default stand at current position) and
// read byteNb bytes in data (the whole file if byteNb = 0;
// data contains binary bytes that can be a string or
// raw bytes that can be converted to hex digits with the Bin2Hex routine).
//
// moveMethod, defaulting to -1 (no move, read at current position),
// can also be FILE_BEGIN, FILE_CURRENT or FILE_END.
// offset can be positive (toward end of file) or negative (toward beginning of file).
//
// Return the number of bytes read (-1 if there was an error), which can be less
// than requested if end-of-file is meet.
*/
ReadFromFile(_handle, ByRef @data, _byteNb=0, _moveMethod=-1, _offset=0)
{
local fileSize, granted, result, read
_offset := MoveInFile(_handle, _moveMethod, _offset)
IfNotEqual ErrorLevel, 0, Return -1 ; Couldn't make the move
if (_byteNb = 0)
{
; Read whole file (or less if file pointer isn't at start)
fileSize := GetFileSize(_handle)
IfNotEqual ErrorLevel, 0, Return -1 ; Couldn't get the file size
_byteNb := fileSize
}
granted := VarSetCapacity(@data, _byteNb, 0)
if (granted < _byteNb)
{
; Cannot allocate enough memory
ErrorLevel = Mem=%granted%
Return -1
}
result := DllCall("ReadFile"
, "UInt", _handle ; hFile
, "Str", @data ; lpBuffer
, "UInt", _byteNb ; nNumberOfBytesToRead
, "UInt *", read ; lpNumberOfBytesRead
, "UInt", 0) ; lpOverlapped
if (result = 0)
{
ErrorLevel = -2
}
;~ MsgBox fileSize: %fileSize% - offset: %_offset% - byteNb: %_byteNb% - granted: %granted% - read: %read%
IfNotEqual ErrorLevel, 0, Return -1 ; Couldn't read the file
; Note that we can have read less data than requested,
; eg. if end of file has been meet
Return read
}
|
Note (obsolete, see links above): providing large sources on this forum is inconvenient, I would prefer to offer a file. I can, on my site, but it would need to set up a AHK page, to avoid direct download, as host doesn't like much them...
I know the inconveniences of allowing to upload files to a forum (lot of dead files, spam, virus...), so I won't ask for that.
[Edit 1: 1.01.000] I updated the file, as I was bitten by the particular variable scoping in AHK (a bit like PHP): I was trying to access FILE_CURRENT from a function and I was wondering why it didn't work... Instead of declaring the global variables as global, I prefered to declare my local variables as such. Seemed cleaner and more Lua-like... For consistency, I did that even in functions not accessing global WinAPI constants.
[Edit 2: 1.02.000] Slight change of API, some more comments, integration of Laszlo suggestions.
I uploaded the files in AutoHotkey.net for easier grabbing. I left them there, as I like to take a look at code before downloading it
[Edit 3: 1.03.000] Moved Bin2Hex & Hex2Bin to DllCallStruct.ahk.
I also applied my new coding conventions on variable prefixes.
DllCallStruct.ahk (relevant extract, real file also has Set/GetInteger and DumpDWORD...)
| Code: | /*
// Convert raw bytes stored in a variable to a string of hexa digit pairs.
// Convert either byteNb bytes or, if null, the whole content of the variable.
//
// Return the number of converted bytes, or -1 if error (memory allocation)
*/
Bin2Hex(ByRef @hex, ByRef @bin, _byteNb=0)
{
local intFormat, dataSize, dataAddress, granted, x
; Save original integer format
intFormat = %A_FormatInteger%
; For converting bytes to hex
SetFormat Integer, Hex
; Get size of data
dataSize := VarSetCapacity(@bin)
If (_byteNb < 1 or _byteNb > dataSize)
{
_byteNb := dataSize
}
dataAddress := &@bin
; Make enough room (faster)
granted := VarSetCapacity(@hex, _byteNb * 2)
if (granted < _byteNb * 2)
{
; Cannot allocate enough memory
ErrorLevel = Mem=%granted%
Return -1
}
Loop %_byteNb%
{
; Get byte in hexa
x := *dataAddress + 0x100
StringRight x, x, 2 ; 2 hex digits
StringUpper x, x
@hex = %@hex%%x%
dataAddress++ ; Next byte
}
; Restore original integer format
SetFormat Integer, %intFormat%
Return _byteNb
}
/*
// Convert a string of hexa digit pairs to raw bytes stored in a variable.
// Convert either byteNb bytes or, if null, the whole content of the variable.
//
// Return the number of converted bytes, or -1 if error (memory allocation)
*/
Hex2Bin(ByRef @bin, _hex, _byteNb=0)
{
local dataSize, granted, dataAddress, x
; Get size of data
x := StrLen(_hex)
dataSize := Ceil(x / 2)
if (x = 0 or dataSize * 2 != x)
{
; Invalid string, empty or odd number of digits
ErrorLevel = Param
Return -1
}
If (_byteNb < 1 or _byteNb > dataSize)
{
_byteNb := dataSize
}
; Make enough room
granted := VarSetCapacity(@bin, _byteNb, 0)
if (granted < _byteNb)
{
; Cannot allocate enough memory
ErrorLevel = Mem=%granted%
Return -1
}
dataAddress := &@bin
Loop Parse, _hex
{
if (A_Index & 1) ; Odd
{
x = %A_LoopField% ; Odd digit
}
else
{
; Concatenate previous x and even digit, converted to hex
x := "0x" . x . A_LoopField
; Store integer in memory
DllCall("RtlFillMemory"
, "UInt", dataAddress
, "UInt", 1
, "UChar", x)
dataAddress++
}
}
Return _byteNb
}
|
_________________
 vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")
Last edited by PhiLho on Fri Mar 17, 2006 10:25 am; edited 7 times in total |
|
| Back to top |
|
 |
Chris
Site Admin
Joined: 02 Mar 2004
Posts: 10480
|
| Posted: Mon Jan 23, 2006 2:48 pm Post subject: Re: Binary file reading and writing |
|
|
Looks great. Thanks for posting it.
| PhiLho wrote: | | I would prefer to offer a file. |
If you ever need to host a zip file, etc., try www.autohotkey.net |
|
| Back to top |
|
|
Laszlo
Joined: 14 Feb 2005
Posts: 4078
Location: Pittsburgh
|
| Posted: Mon Jan 23, 2006 9:51 pm Post subject: |
|
|
Impressive! I am sure it will prove to be useful.
| PhiLho wrote: | | ...improving the decimal to 2-digit hex conversion... |
Well, in Bin2Hex you can save one more line with | Code: | x := *dataAddress + 0x100
StringRight x, x, 2 |
With huge hex strings Hex2Bin could be made faster with a Parse loop, which does not create many copies of "hex", each 2 characters shorter. Something like this (untested) | Code: | Loop Parse, hex
{
If (A_Index & 1)
x = %A_LoopField%
else {
DllCall("RtlFillMemory"
, "UInt", dataAddress
, "UInt", 1
, "UChar", "0x" x A_LoopField)
dataAddress++
}
} |
|
|
| Back to top |
|
|
PhiLho
Joined: 27 Dec 2005
Posts: 6721
Location: France (near Paris)
|
| Posted: Tue Jan 24, 2006 10:04 am Post subject: |
|
|
@Chris: thanks, I didn't understood the related sticky post...
I have uploaded the files so they can be retreived in an easier way, less prone to error than select/copy/paste.
@Laszlo: thanks again, you beat me on the clever ideas, I integrated them in the new release...
I didn't tried the parse loop before, you make a smart use of it. |
|
| Back to top |
|
|
PhiLho
Joined: 27 Dec 2005
Posts: 6721
Location: France (near Paris)
|
| Posted: Wed Jan 25, 2006 4:26 pm Post subject: Return in functions |
|
|
I had to update my script when I saw that the opening functions returned a string in case of error instead of a numerical value.
The trick is that if a raw Return in a function does interpret its parameter as an expression, a Return in a 'classical' If is still using the 'classical' syntax:
| Code: | TestFunction()
{
[snip]
IfNotEqual ErrorLevel, 0, Return %ERROR_CODE%
Return functionResult
}
|
Perhaps I missed a bit of information somewhere. AHK documentation is very rich and detailled, and it is easy to skip or miss some parts. |
|
| Back to top |
|
|
Chris
Site Admin
Joined: 02 Mar 2004
Posts: 10480
|
| Posted: Wed Jan 25, 2006 5:01 pm Post subject: |
|
|
| PhiLho wrote: | | a Return in a 'classical' If is still using the 'classical' [%Var%] syntax |
Return supports the classical syntax as a side-effect of allowing commands like "Sleep X" to be treated the same as "Sleep %X%". Thus, I believe the following produce identical behavior:
return Var
return %Var%
However, I don't believe return's behavior varies just because you use it on the same line as another command (I tested it and couldn't reproduce it). If you can post a short test script that proves otherwise, I'll definitely try to fix it. Thanks. |
|
| Back to top |
|
|
PhiLho
Joined: 27 Dec 2005
Posts: 6721
Location: France (near Paris)
|
| Posted: Thu Jan 26, 2006 1:51 pm Post subject: |
|
|
Doh, I tried to reproduce it, with no result... Whatever combination I tried. So I am totally confused (I really can remember having seen the variable name instead of its value), but you can see this as a false alert...
Oh, so if I have:
bar = 1000
foo = bar
and I write:
x := %foo%
x will be 1000.
But if I write:
Sleep %foo%
the command will be ignored because Sleep gets "bar" as value.
The problem is the versatility of Sleep (and Return) which accepts either a variable name (classical or naked) or an expression (or a value).
Gosh, this is still confusing...
Well, once I know that, I suppose I can get used to it... |
|
| Back to top |
|
|
kapege.de
Joined: 07 Feb 2005
Posts: 188
Location: Munich, Germany
|
| Posted: Fri Jan 27, 2006 10:16 am Post subject: |
|
|
| PhiLho wrote: | bar = 1000
foo = bar
and I write:
x := %foo% |
With this line ( x := %foo%) you process the content of the content. Content of "foo" are the letters B, A and R = "bar". This letters are processed again with ":=". So you now get the content of "bar" which is "1000".
This is nothing special to AHK, but hardly to see through it.
_________________
Peter
Wisenheiming for beginners: KaPeGe (German only, sorry) |
|
| Back to top |
|
|
Thalon
Joined: 12 Jul 2005
Posts: 640
|
| Posted: Fri Jan 27, 2006 10:49 am Post subject: |
|
|
| PhiLho wrote: | bar = 1000
foo = bar
and I write:
x := %foo% |
To do what you want you have 2 ways:
| Code: | bar = 1000
foo = bar
x = %foo% |
or | Code: | bar = 1000
foo = bar
x := foo |
I prefer the first method, even if it is more to write, because I have a clear difference to strings..
Thalon
_________________
AHK-Icon-Changer
AHK-IRC
deutsches Forum
SacredVault |
|
| Back to top |
|
|
PhiLho
Joined: 27 Dec 2005
Posts: 6721
Location: France (near Paris)
|
| Posted: Fri Jan 27, 2006 12:28 pm Post subject: |
|
|
I fear Thalon and kapege.de didn't understood what I meant (but thanks for wanting to help).
I understand the x := %foo% mechanism, which is powerful.
I wanted to point out that if Sleep was like MsgBox, it would accept only %bar% to get the value of the variable. Sleep foo would be illegal.
If Sleep accepted only expressions, Sleep bar would sleep 1000ms and so would do Sleep %foo%.
Sleep is more flexible than that and accepts both variables and expressions, hence the ambiguity:
Sleep bar ; Wait 1s
Sleep %bar% ; Also wait 1s
Sleep %foo% ; Don't wait (non numerical)
Sleep %foo% + 1 ; Wait 1s
Sleep %bar% + 1 ; Don't wait (invalid expression)
As often in AHK, context is paramount in interpreting the code... |
|
| Back to top |
|
|
kiu
Joined: 18 Dec 2005
Posts: 229
Location: Italy - Galatro(RC)
|
| Posted: Sun Jan 29, 2006 10:50 am Post subject: |
|
|
I'm trying to write an image contained in clipboard with this code, but it doesn't work
| Code: |
file = e:\temp3.bmp
IfExist, %file%
FileDelete, %file%
fh := OpenFileForWrite(file)
l := Hex2Bin(data, ClipboardAll)
WriteInFile(fh,data )
CloseFile(fh) |
What's wrong?
_________________
____________________
______________________
kiu |
|
| Back to top |
|
|
Chris
Site Admin
Joined: 02 Mar 2004
Posts: 10480
|
| Posted: Sun Jan 29, 2006 12:22 pm Post subject: |
|
|
| The format of ClipboardAll is probably not suitable for becoming an on-disk BMP file. Its data format is described in the documentation. By contrast, the format of a BMP file is something I can't tell you much about. |
|
| Back to top |
|
|
PhiLho
Joined: 27 Dec 2005
Posts: 6721
Location: France (near Paris)
|
| Posted: Mon Jan 30, 2006 11:05 am Post subject: |
|
|
Perhaps running IrfanView (for example) and pasting the content of the clipboard, then saving the file may be easier than transforming the content of the clipboard.
You can also use Paint for this, but I remember a bug in the early versions of Paint where pasting only painted to the visible size of the image. I can't remember if pasting twice was a workaround or not...
You can also use ImageMagick: convert clipboard: foo.bmp. |
|
| Back to top |
|
|
toralf
Joined: 31 Jan 2005
Posts: 3841
Location: Bremen, Germany
|
| Posted: Mon Jan 30, 2006 11:24 am Post subject: |
|
|
You can use the command line options of IrfanView to convert the clipboard to any image file. You do not have to start IrfanView for it. Which is much better, since you do not have the flickering of windows poping up and disappearing.
_________________
Ciao
toralf  |
|
| Back to top |
|
|
kiu
Joined: 18 Dec 2005
Posts: 229
Location: Italy - Galatro(RC)
|
| Posted: Mon Jan 30, 2006 5:15 pm Post subject: |
|
|
thanks
_________________
____________________
______________________
kiu |
|
| Back to top |
|
|
|
|
You can post new topics in this forum
You can reply to topics in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Register
Profile
Log in to check your private messages
Log in
