Some data is missing when using FileRead/FileAppend

[pkey]

[Moderator's note: Topic moved from Bug Reports to Ask for Help.]

Hello!
I'm using it to read from text file and write to new one:

Code: Select all

#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
#Warn  ; Recommended for catching common errors.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.

MsgBox, Go!

FileRead, fileContent, Test.txt
FileDelete, Test2.txt
FileAppend, %fileContent%, Test2.txt

MsgBox, Done!
ExitApp

But the size of new file is different! (Original: 142KB vs New: 112KB)
Something wrong in FileRead/FileAppend?

Attached the sample file here:
https://dl.dropboxusercontent.com/u/241 ... stFile.zip

Environment:
Windows 7 SP1
AutoHotkey v1.1.22.07

Thank you!

[unix-v-windows]

Don't think it is a bug.

Pretty sure your source file is a unix file so you only have `n for a newline, see https://en.wikipedia.org/wiki/End-of-li ... sentations

As you can read here http://ahkscript.org/docs/commands/FileAppend.htm for windows it is `r`n so that would add extra `r characters in there e.g. bigger file. If you add a * infront of your filename it should save it with `n only and it should be the same size.

Not likely but another cause could be that the source is ansi and you write it in unicode so some characters will take up two bytes (if I'm not mistaken)
See the FileEcoding command of FileEcoding options of FileAppen.

[wolf_II]

@pkey: It looks like all information after the first NULL character got lost. You could use option *c with FileRead on a file that includes binary data:

Code: Select all

...
FileRead, fileContent, *c Test.txt
...

Tested, both files are the same size, and the same content.

[lexikos]

@pkey: There is something wrong with your so-called "text" file. Text files should not include null characters. If I view the file in Notepad, there is a whole lot of empty space below this line:

Code: Select all

<script src="https://wwwimages2.adobe.com/ubi/globalnav/_all/ims.js"></script>

The first null character immediately follows the "</script>".

Why are you reading and then writing the file anyway? Why not just FileCopy? Is your actual script more complicated than what you've posted?

@wolf_II: FileRead already reads the entire file, without *c. The truncation occurs in FileAppend, which is designed to append text. Binary zero is not valid inside text in AutoHotkey (but it works in some very limited contexts).


For non-text binary files, you do not want to use FileRead (without *c) or FileAppend, as they interpret the file as text according to the default FileEncoding setting or the *P/Encoding parameter. They are for text only. For binary files, use FileOpen() and the File object.

[pkey]

Hello!
Thanks all of your replies!

@lexikos:

Sorry for my limited English writing ability.
Yes, you are right!
It's a simplified version which has the same (I think) issue I met actually.
The real one does the following things:

Code: Select all

TempFile = zzz.txt
UpdateVersionKeyword = <td>19.0.0.226</td>
FlashUpdateUrl = https://www.adobe.com/software/flash/about/
UrlDownloadToFile, %FlashUpdateUrl%, %TempFile%

; Try to get version info from server response
FileRead, fileContent, %TempFile%
len := StrLen(fileContent)
if (0 != InStr(fileContent, UpdateVersionKeyword)) {
    ; Your Flash player is up-to-date!
    MsgBox, UPD
} else {
    ; Notify user
    MsgBox, NOOP
}
ListVars
Pause
ExitApp

It get webpage from that URL and try to search for specific keyword I want and then do something.
Several NULL characters are there for unknown reason. It's normal before 11/02.
In this case, InStr() will not search texts after those NULL characters.
It will always return 0(string not found) and then reached "; Notify user" section.
If FileAppend/InStr() are designed to process text, it's not a bug since NULL character is invalid and may be treated as the end of string (C-style string).
Using StrLen() to get the length, you will get "146030" which is the same as I see when running ListVars window: fileContent[146030 of 146030] but you are not able to search some text in it! Maybe it should have the same behavior as FileAppend/InStr(), count total string length just until NULL characters (truncate remaining data)?

I can do some workaround on this but I'd also like to know if there is other way to locate string/data in the whole variable data in faster way? (Assume it don't have encoding/linefeed conversion issue)

[lexikos]

FileRead knows the length of the text, as it read the text from file and converted the entire data with WideCharToMultiByte (on Unicode versions). The length of the text is stored in the variable for performance, but only a select few commands/functions handle null characters safely. You can call VarSetCapacity(fileContent, -1) to update the variable's length to reflect its zero-terminated content, but in this case I think this solution would be more suitable for you:

Code: Select all

FileRead text, Test.txt
text := RegExReplace(text, "\0", "")  ; Strip out null characters.

This only works if the variable's length is set to include the null characters. For FileRead, it will only work correctly with text files, as other binary data can still be corrupted by ANSI->Unicode conversion.

If you don't need the actual text other than to search for a single substring, you can just use RegExMatch instead of InStr:

Code: Select all

FileRead text, Test.txt
MsgBox % RegExMatch(text, "</html>")

[pkey]

Ok, I will try those functions you mentioned!
Thanks for your detailed reply

[art]

@pkey
You can also find the latest flash version in: https://fpdownload.macromedia.com/pub/f ... ersion.xml

Code: Select all

<?xml version="1.0"?>
<version>
	<ActiveX major="19" minor="0" buildMajor="0" buildMinor="226"/>
	<Plugin major="19" minor="0" buildMajor="0" buildMinor="226"/>
	<Pepper major="19" minor="0" buildMajor="0" buildMinor="226"/>
	<MacPlugin major="19" minor="0" buildMajor="0" buildMinor="226"/>
	<MacPepper major="19" minor="0" buildMajor="0" buildMinor="226"/>
	<SAUConfig checkFrequency="1"/>
</version>