Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

simple binary file read/write functions


  • Please log in to reply
61 replies to this topic
Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
Chris keeps saying that someone has to write simple binary file read/write functions based on the example he provided. Here is mine. They open a file of the given name, read/write hex data and finally close the file. The returned value is the actual number of bytes processed. If there was an error, Errorlevel is set. The data is a long stream of 2-digit hex numbers, without the 0x prefixes.

The script below starts with calling examples, followed by the functions which could be included in other scripts.
file = c:\z.dat
FileDelete %file%
IfNotEqual ErrorLevel,0, MsgBox Can't delete file "%file%"`nErrorLevel = "%ErrorLevel%"

res := BinWrite(file,"000102030405060708090a0b0c0d0e0f00")
MsgBox ErrorLevel = %ErrorLevel%`nBytes Written = %res%
res := BinRead(file,data)
MsgBox ErrorLevel = %ErrorLevel%`nBytes Read = %res%`nData = "%data%"

res := BinWrite(file,"aa00bb",0,2)
MsgBox ErrorLevel = %ErrorLevel%`nBytes Written = %res%
res := BinRead(file,data)
MsgBox ErrorLevel = %ErrorLevel%`nBytes Read = %res%`nData = "%data%"

res := BinRead(file,data,3,-2)
MsgBox ErrorLevel = %ErrorLevel%`nBytes Read = %res%`nData = "%data%"
ExitApp

/* ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; BinWrite ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|  - Open binary file
|  - (Over)Write n bytes (n = 0: all)
|  - From offset (offset < 0: counted from end)
|  - Close file
|  data -> file[offset + 0..n-1], rest of file unchanged
|  Return #bytes actually written
*/ ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

BinWrite(file, data, n=0, offset=0)
{
   ; Open file for WRITE (0x40..), OPEN_ALWAYS (4): creates only if it does not exists
   h := DllCall("CreateFile","str",file,"Uint",0x40000000,"Uint",0,"UInt",0,"UInt",4,"Uint",0,"UInt",0)
   IfEqual h,-1, SetEnv, ErrorLevel, -1
   IfNotEqual ErrorLevel,0,Return,0 ; couldn't create the file

   m = 0                            ; seek to offset
   IfLess offset,0, SetEnv,m,2
   r := DllCall("SetFilePointerEx","Uint",h,"Int64",offset,"UInt *",p,"Int",m)
   IfEqual r,0, SetEnv, ErrorLevel, -3
   IfNotEqual ErrorLevel,0, {
      t = %ErrorLevel%              ; save ErrorLevel to be returned
      DllCall("CloseHandle", "Uint", h)
      ErrorLevel = %t%              ; return seek error
      Return 0
   }

   TotalWritten = 0
   m := Ceil(StrLen(data)/2)
   If (n <= 0 or n > m)
       n := m
   Loop %n%
   {
      StringLeft c, data, 2         ; extract next byte
      StringTrimLeft data, data, 2  ; remove  used byte
      c = 0x%c%                     ; make it number
      result := DllCall("WriteFile","UInt",h,"UChar *",c,"UInt",1,"UInt *",Written,"UInt",0)
      TotalWritten += Written       ; count written
      if (!result or Written < 1 or ErrorLevel)
         break
   }

   IfNotEqual ErrorLevel,0, SetEnv,t,%ErrorLevel%

   h := DllCall("CloseHandle", "Uint", h)
   IfEqual h,-1, SetEnv, ErrorLevel, -2
   IfNotEqual t,,SetEnv, ErrorLevel, %t%

   Return TotalWritten
}

/* ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; BinRead ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|  - Open binary file
|  - Read n bytes (n = 0: all)
|  - From offset (offset < 0: counted from end)
|  - Close file
|  data (replaced) <- file[offset + 0..n-1]
|  Return #bytes actually read
*/ ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

BinRead(file, ByRef data, n=0, offset=0)
{
   h := DllCall("CreateFile","Str",file,"Uint",0x80000000,"Uint",3,"UInt",0,"UInt",3,"Uint",0,"UInt",0)
   IfEqual h,-1, SetEnv, ErrorLevel, -1
   IfNotEqual ErrorLevel,0,Return,0 ; couldn't open the file

   m = 0                            ; seek to offset
   IfLess offset,0, SetEnv,m,2
   r := DllCall("SetFilePointerEx","Uint",h,"Int64",offset,"UInt *",p,"Int",m)
   IfEqual r,0, SetEnv, ErrorLevel, -3
   IfNotEqual ErrorLevel,0, {
      t = %ErrorLevel%              ; save ErrorLevel to be returned
      DllCall("CloseHandle", "Uint", h)
      ErrorLevel = %t%              ; return seek error
      Return 0
   }

   TotalRead = 0
   data =
   IfEqual n,0, SetEnv n,0xffffffff ; almost infinite

   format = %A_FormatInteger%       ; save original integer format
   SetFormat Integer, Hex           ; for converting bytes to hex

   Loop %n%
   {
      result := DllCall("ReadFile","UInt",h,"UChar *",c,"UInt",1,"UInt *",Read,"UInt",0)
      if (!result or Read < 1 or ErrorLevel)
         break
      TotalRead += Read             ; count read
      c += 0                        ; convert to hex
      StringTrimLeft c, c, 2        ; remove 0x
      c = 0%c%                      ; pad left with 0
      StringRight c, c, 2           ; always 2 digits
      data = %data%%c%              ; append 2 hex digits
   }

   IfNotEqual ErrorLevel,0, SetEnv,t,%ErrorLevel%

   h := DllCall("CloseHandle", "Uint", h)
   IfEqual h,-1, SetEnv, ErrorLevel, -2
   IfNotEqual t,,SetEnv, ErrorLevel, %t%

   SetFormat Integer, %format%      ; restore original format
   Totalread += 0                   ; convert to original format
   Return TotalRead
}
Edit 2005.07.20: With Chris’ tip I was able to update the functions. Now BinRead can read n characters (all if n = 0) starting from an offset ( offset < 0: counting from the end of the file). It allows, for example, getting MP3 tags or icons from a file, without any external SW.

BinWrite now overwrites n bytes of the file (if it exists) starting from the offset (offset < 0 counting from the end of the file) taken from the beginning of data. If n = 0 all of data is written. It does not change the length of the file unless the written data expands over the original end of file. If the file does not exist, it gets created anew. Un-initialized data could be left in the beginning of a newly created file when non-zero offset is specified. Also, writing at an offset beyond the end of the file also leaves un-initialized data between the old end and the offset.

Edit 2005.07.21: Minor clarifications

Rajat
  • Members
  • 1904 posts
  • Last active: Jul 17 2015 07:45 AM
  • Joined: 28 Mar 2004
aw crap! i was going to post those! ;) for binread i was tinkering to see if there's a way to not read 1 byte in a loop and maybe read the whole file or atleast 1kb a time.
thanx for posting 'em! :)

MIA

CleanNews.in : Bite sized latest news headlines from India with zero bloat


Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
Rajat, I was thinking about the larger block read speedup, too. Windows probably keeps a whole block of data (512 bytes at ATA drives) in a buffer, and the DLL-calls only access this buffer, not fetching the disk blocks repeatedly. It means, the speedup only eliminates some subroutine calls and returns. They are not drastic.

For me the other issues look more interesting: to read/write in the middle of the file. I have no clue...

Rajat
  • Members
  • 1904 posts
  • Last active: Jul 17 2015 07:45 AM
  • Joined: 28 Mar 2004

Windows probably keeps a whole block of data (512 bytes at ATA drives) in a buffer, and the DLL-calls only access this buffer, not fetching the disk blocks repeatedly. It means, the speedup only eliminates some subroutine calls and returns. They are not drastic.

i didn't know that... so it probably already runs at a near optimal rate.

MIA

CleanNews.in : Bite sized latest news headlines from India with zero bloat


Rajat
  • Members
  • 1904 posts
  • Last active: Jul 17 2015 07:45 AM
  • Joined: 28 Mar 2004
i tried the following, but the file wasn't recreated.

file = e:\stuff\water.exe

data =
res := BinRead(file,data)
BinWrite("new.exe",res)

the result was just a 2 byte file... where did i mess up?

MIA

CleanNews.in : Bite sized latest news headlines from India with zero bloat


Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
BinWrite() expects as the second parameter a sequence of hex digits. res contains only the number of bytes read by the preceeding Binread(), data is containing the read bytes.

Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004
Thanks for writing these functions. I feel a little guilty for pressuring you to do it because your time would probably have been better spent proving that P != NP. :)

It would be nice to be able to read from a specified offset in a file, or overwrite certain bytes, insert new ones, delete a few.

I Googled it and it seems SetFilePointer is most likely the API call used for random access of a file's data.

Rajat
  • Members
  • 1904 posts
  • Last active: Jul 17 2015 07:45 AM
  • Joined: 28 Mar 2004
doh!
thanx laszlo!

MIA

CleanNews.in : Bite sized latest news headlines from India with zero bloat


Rajat
  • Members
  • 1904 posts
  • Last active: Jul 17 2015 07:45 AM
  • Joined: 28 Mar 2004
i made a few minor changes to suit me:
- binread returns the read data
- errorlevel contains bytes read/written for both functions

if function didn't work out smoothly then errorlevel will still have error code

BinRead(file)				; return #bytes read 
{ 
	format = %A_FormatInteger%		 ; save original integer format 
	SetFormat Integer, Hex			  ; need hex in here 

	h := DllCall("CreateFile","str",file,"Uint",0x80000000,"Uint",3,"UInt",0,"UInt",3,"Uint",0,"UInt", 0) 
	IfEqual h,-1, SetEnv, ErrorLevel, -1 
	IfNotEqual ErrorLevel,0,Return,0 ; couldn't open the file 
	TotalRead = 0 
	data = 
	Loop 
	{ 
		result := DllCall("ReadFile","UInt",h,"UChar *",c,"UInt",1,"UInt *",Read,"UInt",0) 
		if (!result or Read < 1 or ErrorLevel) 
			break 
		TotalRead += Read				 ; count read 
		c += 0								; convert to hex 
		StringTrimLeft c, c, 2		  ; remove 0x 
		c = 0%c%							 ; pad left with 0 
		StringRight c, c, 2			  ; always 2 digits 
		StringUpper, c, c
		data = %data%%c% 
	} 

	SetFormat Integer, %format%		; restore original format 
	Totalread += 0						 ; convert to original format 
	ErrorLevel = %TotalRead%

	h := DllCall("CloseHandle", "Uint", h) 
	IfEqual h,-1, SetEnv, ErrorLevel, -2 

	Return Data
}


BinWrite(file,data)					  ; return #bytes written 
{ 
	h := DllCall("CreateFile","str",file,"Uint",0x40000000,"Uint",0,"UInt",0,"UInt",2,"Uint",0,"UInt",0) 
	IfEqual h,-1, SetEnv, ErrorLevel, -1 
	IfNotEqual ErrorLevel,0,Return,0 ; couldn't create the file 
	TotalWritten = 0 

	Loop % Ceil(StrLen(data)/2) 
	{ 
		StringLeft c, data, 2			; extract next byte 
		StringTrimLeft data, data, 2  ; remove  used byte 
		c = 0x%c%							; make it number 
		result := DllCall("WriteFile","UInt",h,"UChar *",c,"UInt",1,"UInt *",Written,"UInt",0) 
		TotalWritten += Written		 ; count written 
		if (!result or Written < 1 or ErrorLevel) 
			break 
	} 
 
	ErrorLevel = %TotalWritten%

	h := DllCall("CloseHandle", "Uint", h) 
	IfEqual h,-1, SetEnv, ErrorLevel, -2 
} 

MIA

CleanNews.in : Bite sized latest news headlines from India with zero bloat


Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
The original functions had the same calling syntax result:=func(file, data), which I like for their symmetry. Also, if something went wrong ErrorLevel used to tell a lot about what had happened. Now you see less information. The last dll calls might clear ErrorLevel, so you should move the assignment to it after the dll calls, under an Else, but still it is not a clean solution. You have a different design, so some more changes are necessary (besides updating the comments).

Thalon
  • Members
  • 641 posts
  • Last active: Jan 02 2017 12:17 PM
  • Joined: 12 Jul 2005
I'll have a look over it, maybe I will overtake it in some piece of software (with little changes)!

Thx Laszlo!

Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
With Chris’ tip I was able to update the functions (see edit in the original post). Now BinRead can read n characters (all if n = 0) starting from an offset ( < 0 counting from the end of the file). It allows, for example, getting MP3 tags or icons from a file, without any external SW.

BinWrite now overwrites n bytes of the file (if it exists) starting from the offset (< 0 counting from the end of the file) taken from the beginning of data. It does not change the length of the file. If the file does not exist, it gets created anew. In this case un-initialized data could be left in the beginning of the file when non-zero offset is specified.

There could be a further generalization of the BinWrite function with somewhat different syntax: BinWrite(file, data, insert_offset, continue_offset), meaning that the file remains unchanged before the insert_offset, the new data comes after it, but the bytes until continue_offset are dropped. The end of the file remains unchanged.

Changing the length of a binary file looks dangerous, though. If it has code, we can easily mess it up. Still, using the posted functions this data insertion can be easily programmed: read the beginning of the file, concatenate the new data followed by the end of the file also read. The resulting data can be written to a new file.

Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
The original post is further edited with minor clarifications an cleanup.

These functions keep the data in hex form, such that the powerful string manipulating and search functions of AHK could be applied. If only pieces need to be cut and pasted, there is no need for hex conversions. Large chunk of data can be manipulated in binary buffers. The details are left as an exercise for the reader. ;-)

Rabiator
  • Members
  • 292 posts
  • Last active: Aug 29 2016 09:29 PM
  • Joined: 17 Apr 2005
I always get errorlevel -4 when I run the script on Win 98. The reason is, SetFilePointerEx doesn't exist in this OS.
Is there a similar function?

Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
Look here