Possible A_LoopFileLongPath bug

[Builder]

I will ask in advance for forgiveness if this is not actually a bug, and should have been put elsewhere.... I was really unsure where to put it.

Apparently, the use of A_LoopFileLongPath in File Loops is Orders of Magnitude slower than using other File Loop variables...

Here is the test code I used, and the results:

Code: Select all

SetBatchLines -1

Source:="D:"				; My System Root Directory
SetWorkingDir, % Source
Target_File:="R:\Test.txt"

counter=
StartTime:=A_TickCount
loop, files, *.*, FR
	{
	counter++
	tempvar:=Source . A_LoopFileFullPath
	}
gosub, Stopwatch							;Calculates difference in TickCount, and formats the result
FileAppend, Number of Files Found:`t %counter%`n, %Target_File%
FileAppend, Elapsed Time:`t%ElapsedTime%, %Target_File%

Run, %Target_File%
ExitApp

As written, using the LoopFileFullPath added to the working directory, the results were pretty good, I think. 45 seconds to complete.

Results:
Number of Files Found: 1281076
Elapsed Time: 0:45

Number of Files Found: 1281076
Elapsed Time: 0:45

Number of Files Found: 1281076
Elapsed Time: 0:45


However, the same program substituting A_LoopFileLongPath were incredibly slow:

Code: Select all

SetBatchLines -1

Source:="D:"							; My System Root 
SetWorkingDir, % Source
Target_File:="R:\Test.txt"

counter=
StartTime:=A_TickCount
loop, files, *.*, FR
	{
	counter++
	tempvar:=A_LoopFileLongPath
	}
gosub, Stopwatch							;Calculates difference in TickCount, and formats the result
FileAppend, Number of Files Found:`t %counter%`n, %Target_File%
FileAppend, Elapsed Time:`t%ElapsedTime%, %Target_File%

Run, %Target_File%
ExitApp

Results:
Number of Files Found: 1281076
Elapsed Time: 0:57:38

Number of Files Found: 1281076
Elapsed Time: 0:57:34

Number of Files Found: 1281076
Elapsed Time: 0:57:32


My system drive is an SSD bottlenecked by a SATA 2 Connection:
Windows 7 Ultimate

The script was run from my "scratch drive" (R:) which is a RAM Drive.
The AutoHotKey Directory is in my Program Files Directory Located on Drive F: which is a different SSD
The version of Hotkey is 1.1.27.04

The computer was idle except for the test scripts during testing.

If I have missed listing a parameter you need, I will happily supply it, but these are the only ones I can think of that you would find relevant.

[Osprey]

According to the help file entry for A_LoopFileLongPath:

A_LoopFileLongPath
This is different than A_LoopFileFullPath in the following ways: 1) It always contains the absolute/complete path of the file even if FilePattern contains a relative path; 2) Any short (8.3) folder names in FilePattern itself are converted to their long names; 3) Characters in FilePattern are converted to uppercase or lowercase to match the case stored in the file system. This is useful for converting file names -- such as those passed into a script as command line parameters -- to their exact path names as shown by Explorer.

I guess that all of that conversion takes a long time, especially when you have over a million files. Even if not every path is converted, simply checking if each path needs to be converted could add overhead. Perhaps it could be a lot faster than it is, and you may have a point on that, but it may not be a bug, per se.

[Pacheco]

@Osprey:
If I'm reading it correctly, going from 45 seconds to 57 minutes is way too different...

[Osprey]

@Osprey:
If I'm reading it correctly, going from 45 seconds to 57 minutes is way too different...

Oh, I know. That's why I wasn't making excuses for the length of time. It's just that there's a difference between code being horribly unoptimized and it being buggy. Just the other day, I had some code that I managed to make run in 1/10th the time that it did before. It wasn't that it was initially buggy. I just found a 10x more efficient way to do it.

[Builder]

Well, I have no idea how the functions are implemented, so I can't say what conversions are being made behind the scenes.

However, a quick and dirty test (ie, not a specifically controlled enviroment for testing) suggests that other variables, ie. FileSize and Date are equally as quick as the A_LoopFileFullPath.

The speed difference appears to be limited to only the A_LoopFileLongPath.

As for the number of files on the drive... The Windows OS does seem to bloat itself...

I have also noticed that AHK's File Loops find a lot of files that you can not see using Windows Explorer, even if the "show hidden files" setting is active. It also will loop over a file twice if Windows has more than one way of referencing the address....

example...
D:\Documents and Settings\Thomas\Desktop\STARTUP.ahk
D:\Users\Thomas\Desktop\STARTUP.ahk

[jeeswg]

Oh dear, any way to prevent the files being listed twice?

Code: Select all

q:: ;list files (different names for the same folder)
Loop, 2
{
	if (A_Index = 1)
		vDir1 := "C:\Documents and Settings\" A_UserName "\Desktop"
	else
		vDir1 := "C:\Users\" A_UserName "\Desktop"
	vOutput := "", vCount := 0
	Loop, Files, % vDir1 "\*", F
	{
		vPath := A_LoopFileFullPath
		SplitPath, vPath, vName, vDir, vExt, vNameNoExt, vDrive
		vOutput .= vPath "`r`n", vCount += 1
	}
	MsgBox, % vCount "`r`n" vOutput
}
return

[Builder]

I am not sure if there is a way to exclude the links (the name of the alternative references to the same file/folder) directly.

However, it does not effect me personally much because I don't cycle through an entire drive as I have done for the demonstration purpose here very often.

One thing that I do use in some of the stuff I do I call exclusions...

An example would be:

Code: Select all

Exclude:="Folder Not Processed"

Loop, Files, C:\*.*, DR
	{
	IfInString, A_LoopFileFullPath, %Exclude%
		Continue
	; Place Code here to process the rest of the files
	}

In this case the "Folder Not Processed" would simply be bypassed.

There are obviously any number of ways to test for the file(s)/folder(s) you wish to exclude, but you get the idea.

[just me]

Most of the A_LoopFile... variables are taken from the values returned by the Win-API calls FindFirstFile() / FindNextFile() with minor modifications. A_LoopFileLongPath needs at least one extra API call GetFullPathName(). If that fails, an internal function ConvertFilespecToCorrectCase() is called. This function calls FindFirstFile() several times in many cases. It might cost some time. I don't know whether this can be improved.

[Builder]

@just me....
That is very interesting. Thanks for the information.

[Noesis]

Oh dear, any way to prevent the files being listed twice?

I am not sure if there is a way to exclude the links (the name of the alternative references to the same file/folder) directly.
...
There are obviously any number of ways to test for the file(s)/folder(s) you wish to exclude, but you get the idea.

In case you're interested, to test if a folder is a link:

Code: Select all

attrib := DllCall("GetFileAttributes","str", folderName)
if (attrib != -1 && (attrib & 0x400)) ;0x400 FILE_ATTRIBUTE_REPARSE_POINT (i.e. link)
	Msgbox, %folderName% is a link

file attribute constants can be found here:https://msdn.microsoft.com/en-us/librar ... s.85).aspx

Anyway sorry for going

[jeeswg]

Thank you Noesis, although I didn't see any difference when I did this.

Code: Select all

q:: ;get file attributes
vDir1 := "C:\Documents and Settings\" A_UserName "\Desktop"
vDir2 := "C:\Users\" A_UserName "\Desktop"
;MsgBox, % FileGetAttrib(vDir1) "`r`n" FileGetAttrib(vDir2)
MsgBox, % FileExist(vDir1) "`r`n" FileExist(vDir2)
MsgBox, % DllCall("kernel32\GetFileAttributes", Str,vDir1, UInt) "`r`n" DllCall("kernel32\GetFileAttributes", Str,vDir2, UInt)
return

[Noesis]

Yeah, that's because neither of those folders (i.e. C:\...\Desktop) are links. They are in fact exactly the same folder. The link occurs at an earlier level, and once the link is resolved (to the folder it points to), it's no longer distinguishable from the actual folder it points to. Hence you have to check at the appropriate level. e.g. change the paths like so and they will be different:
vDir1 := "C:\Documents and Settings"
vDir2 := "C:\Users"
This isn't something that generally matters, it's rarely an issue, the main exceptions are right here at the root folder level and there are some links within each user's profile as well. Documents vs My Documents for example.

[jeeswg]

- @Noesis: Thanks very much for the information, I've started a new thread here:
file loop listing the same files twice (identify redirected folders) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=42975
- @Builder: Thanks for mentioning the problem.

[Builder]

@Noesis:
Thank you very much for that!

Since jeeswg asked it there was a way around this problem, I have been playing with trying to use the command prompt to identify the links, (which it can do) and use that information to influence the loop.

Your solution is far superior!

I know that dll's have been the subject of many threads, but I have still not managed to wrap my head around them. One of these days I am going to have to find someone to take me by the hand like a three year old, and teach me how to use them.

[Noesis]

No worries Bulder, FYI, It's not really MY solution, I was working on something a little while ago which needed to be able to identify links so it could skip them, and I found a post on the old forums, mentioning this method (so thanks Lexicos).

Anyway if you're having trouble with dll calls, first thing to do is look up the function being called on MSDN (Microsoft Developer Network). You can read about it's functionality there, but the bit you're interested in for the dll call is the function definition at the top. It tells you what is returned and what types of arguments it accepts. It can be a bit annoying to decipher, since it uses types like DWORD or LPCSTR, but over time you learn what these mean, with the help of web searches etc.

The thing is, to take a look at dllcalls you've seen (esp those you already use from others), even though they work and you don't really need to, but just to start seeing/learning what is going on for yourself, some may be over your head, others perhaps not but come back to the hard ones after you learn more about the easy ones. Without that step, ie looking at them in detail, it's always going to be more daunting than it needs to be, and you'll likely always feel like you don't "get" them yet. Personally I feel I understand them, but implementing them from scratch ... well that depends on what it is, some sure, while others remind me I haven't finished learning.