Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

Binary file reading and writing


  • Please log in to reply
28 replies to this topic
PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005
Laszlo already provided some nice code to do this, and actually I "stole" (with due credit) a lot of code from his work, thanks!

So, why does I publish this? I plan to parse some binary headers, so I need to do several operations on the same file. Laszlo's routines open the file, perform the operation and close the file. Mine open the file, keep the file handle around for various operations, and close the file.

Laszlo's routines are perfectly in the spirit of most AHK commands, like FileAppend: you don't have to remember to open or close the file.
My routines seek a little more performance, even if I am not sure if this is really noticeable.

Well, at least, I improved my DllCall knowledge, and I have some little bonus like allowing relative moves, calling SetFilePointer only if requested (great for sequential reading/writing), no call to GetFileSize if not needed, etc. Plus I had fun making some tricks like improving the decimal to 2-digit hex conversion... :-)

Caveat: by design, these functions may malfunction with files larger than 2GB... (videos, DVD ISO images, server logs...)
Remark 1: it can still work if you only want to read the first few bytes to parse the header.
Remark 2: when dealing with such files, you should better switch to a compiled language, or perhaps to Perl...

You can download BinReadWrite.ahk and TestBinReadWrite.ahk to avoid selecting the code, copying it and pasting it in an editor, loosing tabs...

BinReadWrite.ahk
/*
BinReadWrite.ahk

Routines to read and write binary data from/to files.
Based on original functions written by Laszlo
http://www.autohotkey.com/forum/viewtopic.php?t=4604

TODO: Perhaps set a lastError variable to explicit the errors.

// by Philippe Lhoste <PhiLho(a)GMX.net> http://Phi.Lho.free.fr
// File/Project history:
 1.03.000 -- 2006/02/15 (PL) -- Moved Bin2Hex & Hex2Bin to DllCallStruct, apply code rules.
 1.02.000 -- 2006/01/24 (PL) -- Slight change of the API: all functions return -1 if error.
			  Integrated Laszlo suggestions on improving Bin2Hex and Hex2Bin.
 1.01.000 -- 2006/01/23 (PL) -- Declaration of the local variables, to get access to global
			 WinAPI constants, and for consistency.
 1.00.000 -- 2006/01/19 (PL) -- Rewrote the functions to separate opening and closing,
			 allowing efficient multiple operations.
*/

; WinAPI constants
INVALID_HANDLE_VALUE = -1
INVALID_FILE_SIZE = 0xFFFFFFFF
FILE_BEGIN = 0
FILE_CURRENT = 1
FILE_END = 2

/*
// Open the file for reading.
// Return the file handle to provide in further read operations and in the final close operation,
// or INVALID_HANDLE_VALUE if an error was found.
*/
OpenFileForRead(_filename)
{
	local handle

	handle := DllCall("CreateFile"
			, "Str", _filename		; lpFileName
			, "UInt", 0x80000000		; dwDesiredAccess (GENERIC_READ)
			, "UInt", 3		; dwShareMode (FILE_SHARE_READ|FILE_SHARE_WRITE)
			, "UInt", 0		; lpSecurityAttributes
			, "UInt", 3		; dwCreationDisposition (OPEN_EXISTING)
			, "UInt", 0		; dwFlagsAndAttributes
			, "UInt", 0)	; hTemplateFile
	If (handle = INVALID_HANDLE_VALUE or handle = 0)
	{
		ErrorLevel = -1
	}
	IfNotEqual ErrorLevel, 0, Return INVALID_HANDLE_VALUE	; Couldn't open the file
	Return handle
}

/*
// Open the file for writing.
// Return the file handle to provide in further write operations and in the final close operation,
// or INVALID_HANDLE_VALUE if an error was found.
*/
OpenFileForWrite(_filename)
{
	local handle

	handle := DllCall("CreateFile"
			, "Str", _filename		; lpFileName
			, "UInt", 0x40000000	; dwDesiredAccess (GENERIC_WRITE)
			, "UInt", 3		; dwShareMode (FILE_SHARE_READ|FILE_SHARE_WRITE)
			, "UInt", 0		; lpSecurityAttributes
			, "UInt", 4		; dwCreationDisposition (OPEN_ALWAYS: create if not exists)
			, "UInt", 0		; dwFlagsAndAttributes
			, "UInt", 0)	; hTemplateFile
	If (handle = INVALID_HANDLE_VALUE or handle = 0)
	{
		ErrorLevel = -1
	}
	IfNotEqual ErrorLevel, 0, Return INVALID_HANDLE_VALUE	; Couldn't open the file
	Return handle
}

/*
// Close the file.
*/
CloseFile(_handle)
{
	local result

	result := DllCall("CloseHandle"
			, "UInt", _handle)
	If (result = 0)
	{
		ErrorLevel = -1
	}
}

/*
// Get the size of the opened file, in bytes.
// Limited to 4GB, so it is more limited that AHK's FileGetSize.
// It is here for consistency, and because it accepts a file handle instead of a path.
//
// Return the size in bytes, -1 if there was an error.
*/
GetFileSize(_handle)
{
	local fileSize

	fileSize := DllCall("GetFileSize"
			, "UInt", _handle
			, "UInt", 0)
	If (fileSize = INVALID_FILE_SIZE)
	{
		ErrorLevel = -1
	}
	IfNotEqual ErrorLevel, 0, Return -1
	Return fileSize
}

/*
// Move the file pointer in the file to the given offset relative to moveMethod.
//
// moveMethod can be FILE_BEGIN, FILE_CURRENT or FILE_END.
// If moveMethod is -1, nothing is done (default, for operations at current position).
// To get the current position, call this function with just FILE_CURRENT (null offset).
// offset can be positive (toward end of the file) or negative (toward start of the file).
//
// Return -1 if there was an error, the new file pointer position if OK.
// Note: Currently it doesn't work for files larger than 2GB...
*/
MoveInFile(_handle, _moveMethod=-1, _offset=0)
{
	local result

	result = %INVALID_FILE_SIZE%
	if (_moveMethod != -1)
	{
		result := DllCall("SetFilePointer"
				, "UInt", _handle		; hFile
				, "Int", _offset		; lDistanceToMove
				, "UInt", 0			; lpDistanceToMoveHigh
				, "UInt", _moveMethod)	; dwMoveMethod
		if (result = -1)	; INVALID_SET_FILE_POINTER
		{
			ErrorLevel = -1
		}
		IfNotEqual ErrorLevel, 0, Return -1	; Couldn't make the move
	}
	Return result
}

/*
// Write in a file opened for writing.
//
// Move to position given by moveMethod and offset
// (by default stand at current position) and
// write byteNb bytes from data (all data if byteNb = 0;
// data contains binary bytes that can be a string or
// raw bytes generated from hexa data with the Hex2Bin routine).
//
// moveMethod, defaulting to -1 (no move, write at current position),
// can also be FILE_BEGIN, FILE_CURRENT or FILE_END.
// offset can be positive (toward end of file) or negative (toward beginning of file).
//
// Return the number of bytes written (-1 if there was an error).
*/
WriteInFile(_handle, ByRef @data, _byteNb=0, _moveMethod=-1, _offset=0)
{
	local dataSize, result, written

	_offset := MoveInFile(_handle, _moveMethod, _offset)
	IfNotEqual ErrorLevel, 0, Return -1	; Couldn't make the move

	dataSize := VarSetCapacity(@data)        ; Get the capacity (>= used length!)
	If (_byteNb < 1 or _byteNb > dataSize)
	{
		byteNb := dataSize
	}
	result := DllCall("WriteFile"
			, "UInt", _handle	; hFile
			, "Str", @data		; lpBuffer
			, "UInt", _byteNb	; nNumberOfBytesToWrite
			, "UInt *", written	; lpNumberOfBytesWritten
			, "UInt", 0)		; lpOverlapped
	if (result = 0 or written < _byteNb)
	{
		ErrorLevel = -2
	}
	IfNotEqual ErrorLevel, 0, Return -1	; Couldn't write in the file
	Return written
}

/*
// Read from a file opened for reading.
//
// Move to position given by moveMethod and offset
// (by default stand at current position) and
// read byteNb bytes in data (the whole file if byteNb = 0;
// data contains binary bytes that can be a string or
// raw bytes that can be converted to hex digits with the Bin2Hex routine).
//
// moveMethod, defaulting to -1 (no move, read at current position),
// can also be FILE_BEGIN, FILE_CURRENT or FILE_END.
// offset can be positive (toward end of file) or negative (toward beginning of file).
//
// Return the number of bytes read (-1 if there was an error), which can be less
// than requested if end-of-file is meet.
*/
ReadFromFile(_handle, ByRef @data, _byteNb=0, _moveMethod=-1, _offset=0)
{
	local fileSize, granted, result, read

	_offset := MoveInFile(_handle, _moveMethod, _offset)
	IfNotEqual ErrorLevel, 0, Return -1	; Couldn't make the move

	if (_byteNb = 0)
	{
		; Read whole file (or less if file pointer isn't at start)
		fileSize := GetFileSize(_handle)
		IfNotEqual ErrorLevel, 0, Return -1	; Couldn't get the file size
		_byteNb := fileSize
	}
	granted := VarSetCapacity(@data, _byteNb, 0)
	if (granted < _byteNb)
	{
		; Cannot allocate enough memory
		ErrorLevel = Mem=%granted%
		Return -1
	}

	result := DllCall("ReadFile"
			, "UInt", _handle	; hFile
			, "Str", @data		; lpBuffer
			, "UInt", _byteNb	; nNumberOfBytesToRead
			, "UInt *", read	; lpNumberOfBytesRead
			, "UInt", 0)		; lpOverlapped
	if (result = 0)
	{
		ErrorLevel = -2
	}
;~ MsgBox fileSize: %fileSize% - offset: %_offset% - byteNb: %_byteNb% - granted: %granted% - read: %read%
	IfNotEqual ErrorLevel, 0, Return -1	; Couldn't read the file

	; Note that we can have read less data than requested,
	; eg. if end of file has been meet
	Return read
}

Note (obsolete, see links above): providing large sources on this forum is inconvenient, I would prefer to offer a file. I can, on my site, but it would need to set up a AHK page, to avoid direct download, as host doesn't like much them...
I know the inconveniences of allowing to upload files to a forum (lot of dead files, spam, virus...), so I won't ask for that.

[Edit 1: 1.01.000] I updated the file, as I was bitten by the particular variable scoping in AHK (a bit like PHP): I was trying to access FILE_CURRENT from a function and I was wondering why it didn't work... Instead of declaring the global variables as global, I prefered to declare my local variables as such. Seemed cleaner and more Lua-like... For consistency, I did that even in functions not accessing global WinAPI constants.

[Edit 2: 1.02.000] Slight change of API, some more comments, integration of Laszlo suggestions.
I uploaded the files in AutoHotkey.net for easier grabbing. I left them there, as I like to take a look at code before downloading it :-)

[Edit 3: 1.03.000] Moved Bin2Hex & Hex2Bin to DllCallStruct.ahk.
I also applied my new coding conventions on variable prefixes.

DllCallStruct.ahk (relevant extract, real file also has Set/GetInteger and DumpDWORD...)
/*
// Convert raw bytes stored in a variable to a string of hexa digit pairs.
// Convert either byteNb bytes or, if null, the whole content of the variable.
//
// Return the number of converted bytes, or -1 if error (memory allocation)
*/
Bin2Hex(ByRef @hex, ByRef @bin, _byteNb=0)
{
	local intFormat, dataSize, dataAddress, granted, x

	; Save original integer format
	intFormat = %A_FormatInteger%
	; For converting bytes to hex
	SetFormat Integer, Hex

	; Get size of data
	dataSize := VarSetCapacity(@bin)
	If (_byteNb < 1 or _byteNb > dataSize)
	{
		_byteNb := dataSize
	}
	dataAddress := [email protected]
	; Make enough room (faster)
	granted := VarSetCapacity(@hex, _byteNb * 2)
	if (granted < _byteNb * 2)
	{
		; Cannot allocate enough memory
		ErrorLevel = Mem=%granted%
		Return -1
	}
	Loop %_byteNb%
	{
		; Get byte in hexa
		x := *dataAddress + 0x100
		StringRight x, x, 2	; 2 hex digits
		StringUpper x, x
		@hex = %@hex%%x%
		dataAddress++	; Next byte
	}
	; Restore original integer format
	SetFormat Integer, %intFormat%

	Return _byteNb
}

/*
// Convert a string of hexa digit pairs to raw bytes stored in a variable.
// Convert either byteNb bytes or, if null, the whole content of the variable.
//
// Return the number of converted bytes, or -1 if error (memory allocation)
*/
Hex2Bin(ByRef @bin, _hex, _byteNb=0)
{
	local dataSize, granted, dataAddress, x

	; Get size of data
	x := StrLen(_hex)
	dataSize := Ceil(x / 2)
	if (x = 0 or dataSize * 2 != x)
	{
		; Invalid string, empty or odd number of digits
		ErrorLevel = Param
		Return -1
	}
	If (_byteNb < 1 or _byteNb > dataSize)
	{
		_byteNb := dataSize
	}
	; Make enough room
	granted := VarSetCapacity(@bin, _byteNb, 0)
	if (granted < _byteNb)
	{
		; Cannot allocate enough memory
		ErrorLevel = Mem=%granted%
		Return -1
	}
	dataAddress := [email protected]

	Loop Parse, _hex
	{
		if (A_Index & 1)	; Odd
		{
			x = %A_LoopField%	; Odd digit
		}
		else
		{
			; Concatenate previous x and even digit, converted to hex
			x := "0x" . x . A_LoopField
			; Store integer in memory
			DllCall("RtlFillMemory"
					, "UInt", dataAddress
					, "UInt", 1
					, "UChar", x)
			dataAddress++
		}
	}

	Return _byteNb
}

Posted Image vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")

Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004
Looks great. Thanks for posting it.

I would prefer to offer a file.

If you ever need to host a zip file, etc., try <!-- w -->www.autohotkey.net<!-- w -->

Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
Impressive! I am sure it will prove to be useful.

...improving the decimal to 2-digit hex conversion...

Well, in Bin2Hex you can save one more line with
x := *dataAddress + 0x100
StringRight x, x, 2
With huge hex strings Hex2Bin could be made faster with a Parse loop, which does not create many copies of "hex", each 2 characters shorter. Something like this (untested)
Loop Parse, hex
{
   If (A_Index & 1)
      x = %A_LoopField%
   else {
      DllCall("RtlFillMemory"
            , "UInt", dataAddress
            , "UInt", 1
            , "UChar", "0x" x A_LoopField)
      dataAddress++
   }
}


PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005
@Chris: thanks, I didn't understood the related sticky post...
I have uploaded the files so they can be retreived in an easier way, less prone to error than select/copy/paste.

@Laszlo: thanks again, you beat me on the clever ideas, I integrated them in the new release...
I didn't tried the parse loop before, you make a smart use of it.

PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005
I had to update my script when I saw that the opening functions returned a string in case of error instead of a numerical value.
The trick is that if a raw Return in a function does interpret its parameter as an expression, a Return in a 'classical' If is still using the 'classical' syntax:
TestFunction()
{
[snip]
    IfNotEqual ErrorLevel, 0, Return %ERROR_CODE%
    Return functionResult
}
Perhaps I missed a bit of information somewhere. AHK documentation is very rich and detailled, and it is easy to skip or miss some parts.

Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004

a Return in a 'classical' If is still using the 'classical' [%Var%] syntax

Return supports the classical syntax as a side-effect of allowing commands like "Sleep X" to be treated the same as "Sleep %X%". Thus, I believe the following produce identical behavior:
return Var
return %Var%

However, I don't believe return's behavior varies just because you use it on the same line as another command (I tested it and couldn't reproduce it). If you can post a short test script that proves otherwise, I'll definitely try to fix it. Thanks.

PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005
Doh, I tried to reproduce it, with no result... Whatever combination I tried. So I am totally confused (I really can remember having seen the variable name instead of its value), but you can see this as a false alert...

Oh, so if I have:
bar = 1000
foo = bar

and I write:
x := %foo%
x will be 1000.
But if I write:
Sleep %foo%
the command will be ignored because Sleep gets "bar" as value.
The problem is the versatility of Sleep (and Return) which accepts either a variable name (classical or naked) or an expression (or a value).
Gosh, this is still confusing...
Well, once I know that, I suppose I can get used to it...

kapege.de
  • Members
  • 192 posts
  • Last active: Jan 16 2012 12:34 PM
  • Joined: 07 Feb 2005

bar = 1000
foo = bar

and I write:
x := %foo%

With this line ( x := %foo%) you process the content of the content. Content of "foo" are the letters B, A and R = "bar". This letters are processed again with ":=". So you now get the content of "bar" which is "1000".
This is nothing special to AHK, but hardly to see through it.
Peter

Wisenheiming for beginners: KaPeGe (German only, sorry)

Thalon
  • Members
  • 641 posts
  • Last active: Jan 02 2017 12:17 PM
  • Joined: 12 Jul 2005

bar = 1000
foo = bar
and I write:
x := %foo%

To do what you want you have 2 ways:
bar = 1000
foo = bar
x = %foo%
or
bar = 1000
foo = bar
x := foo
I prefer the first method, even if it is more to write, because I have a clear difference to strings..

Thalon

PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005
I fear Thalon and kapege.de didn't understood what I meant (but thanks for wanting to help).

I understand the x := %foo% mechanism, which is powerful.

I wanted to point out that if Sleep was like MsgBox, it would accept only %bar% to get the value of the variable. Sleep foo would be illegal.
If Sleep accepted only expressions, Sleep bar would sleep 1000ms and so would do Sleep %foo%.

Sleep is more flexible than that and accepts both variables and expressions, hence the ambiguity:
Sleep bar ; Wait 1s
Sleep %bar% ; Also wait 1s
Sleep %foo% ; Don't wait (non numerical)
Sleep %foo% + 1 ; Wait 1s
Sleep %bar% + 1 ; Don't wait (invalid expression)


As often in AHK, context is paramount in interpreting the code...

kiu
  • Members
  • 234 posts
  • Last active: Oct 10 2010 07:30 PM
  • Joined: 18 Dec 2005
I'm trying to write an image contained in clipboard with this code, but it doesn't work

file = e:\temp3.bmp
IfExist, %file%
FileDelete, %file%
fh := OpenFileForWrite(file)
l := Hex2Bin(data, ClipboardAll)
WriteInFile(fh,data )
CloseFile(fh)

What's wrong?
____________________
______________________
kiu - www.romeosa.com

Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004
The format of ClipboardAll is probably not suitable for becoming an on-disk BMP file. Its data format is described in the documentation. By contrast, the format of a BMP file is something I can't tell you much about.

PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005
Perhaps running IrfanView (for example) and pasting the content of the clipboard, then saving the file may be easier than transforming the content of the clipboard.

You can also use Paint for this, but I remember a bug in the early versions of Paint where pasting only painted to the visible size of the image. I can't remember if pasting twice was a workaround or not...

You can also use ImageMagick: convert clipboard: foo.bmp.

toralf
  • Moderators
  • 4035 posts
  • Last active: Aug 20 2014 04:23 PM
  • Joined: 31 Jan 2005
You can use the command line options of IrfanView to convert the clipboard to any image file. You do not have to start IrfanView for it. Which is much better, since you do not have the flickering of windows poping up and disappearing.
Ciao
toralf
 
I use the latest AHK version (1.1.15+)
Please ask questions in forum on ahkscript.org. Why?
For online reference please use these Docs.

kiu
  • Members
  • 234 posts
  • Last active: Oct 10 2010 07:30 PM
  • Joined: 18 Dec 2005
thanks
____________________
______________________
kiu - www.romeosa.com