long path support

Discuss the future of the AutoHotkey language
User avatar
jeeswg
Posts: 4793
Joined: 19 Dec 2016, 01:58
Location: UK

long path support

21 Apr 2018, 07:16

- In response to:
Test build - Obj.Count(), OnError(), long paths, experimental switch-case - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=24&t=47682&p=213868#p213868

TERMINOLOGY
- I will distinguish between '259-' and '260+' files, files of 259 characters or fewer, and files of 260 characters or more.
- Note: 'short', 'long', and 'full' have special meanings, hence why I'm avoiding those terms. See the MSDN pages for: GetShortPathName, GetLongPathName, GetFullPathName.

INTRO
- I'm thankful for the new functionality. Also, the summary of long path findings, in the link, has been most interesting.
- In a file loop, the A_LoopXXX variables present you with info for each file in a folder. Previously for a file loop, any '260+' files were ignored, they are now included, without any changes needing to be made to existing scripts. [EDIT: Some '260+' files will appear without changing existing loops. But to ensure that *all* '260+' files are listed (and that all '260+' folders are recused into) use the '\\?\' prefix.]
- Apart from 'PROBLEM 1 OF 2: FOLDERS', all of the statements about functions, and the example code relate to v1.1.28.02 (the version before the test build).

PROBLEM 1 OF 2: FOLDERS
- The file loop currently handles '260+' files in full, but not folders. It will list '260+' folders, but it will not recurse into them, and it will not accept '260+' folders as the starting directory. [EDIT: It depends on whether '\\?\' is used, see 'EDIT' above.]
- In this script I present methods for listing every file on a drive, including '260+' files and folders.
259-char path limit workarounds - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=26170

SOURCE CODE
- FileCopy(Dir) and FileMove(Dir) have not been changed at present. Here is some info from the source code re. which Winapi functions they use:
ACT_FILECOPY: Util_CopyFile (uses CopyFile)
ACT_FILEMOVE: Util_CopyFile (uses MoveFile)
ACT_FILECOPYDIR: Util_CopyDir (uses SHFileOperation)
ACT_FILEMOVEDIR: MoveFile
- Curiously it seems that MoveFile can handle files *and* folders, but that CopyFile can only handle files.

PROBLEM 2 OF 2: MOVE/COPY
- I did some tests with MoveFile and CopyFile (with DllCall) on Windows 7. Unless I'm missing something, it seems that the functions can handle '260+' files as input files, but not as output files. So if you want to copy a '260+' file, 'C:\...\MyFile.txt' to E:, I'm not sure how you're supposed to do that. [EDIT: They *can* handle '260+' files as output files, what I was missing was that each substring has a limit of 255 characters (the MaximumComponentLength) e.g. C:\substring2\substring3\substring4.]
- I have a workaround at present, although it's not great. CreateDirectory appeared to be able to create '260+' folders, and FileOpen was able to create '260+' files. So, you could open an existing file in FileOpen, and create a new file via FileOpen, and copy the data over. I would be interested in the most efficient FileOpen script to do this, even if a better method becomes apparent. [EDIT: Workaround not needed, MoveFile/CopyFile *can* be used.]

GOOD NEWS
- Some functions can already handle long files. FileExist, DirExist (AHK v2) and FileOpen. Just prepend the path with '\\?\'.
- Sometimes you can use the short-form of a path, with existing commands e.g. FileDelete. Use GetShortPathName (with DllCall) to get the short-form path.
- Quick replacements for certain functionality just need a one-line DllCall to CopyFile/CreateDirectory/DeleteFile/MoveFile.

FILE OBJECTS: GET PATH/DELETE FILE
- Unless I've missed something, it has surprised me that for FileOpen there are no methods to delete the file or get the path. However, while the file is open, on Vista/newer OSes, you can use the handle (hFile) with GetFinalPathNameByHandle to get the path of the file, including for '260+' files.
- Link:
Obtaining a File Name From a File Handle (Windows)
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366789(v=vs.85).aspx

TEST CODE
- Here is a collection of test code for v1.1.28.02 (the version before the test build). Cheers.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


CONVERTING BETWEEN SHORT/LONG FORMS
- I have found being able to convert to and from short-form useful for different reasons, it's part of the functionality that I've added here:
file get part (SplitPath alternative/short-form/long-form/correct case) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=6&t=47709

NOTES ON FUNCTIONALITY RELATED TO FILE LOOPS/LONG PATHS
- A file loop with * currently checks the short-form name as well as the long-form name. [I don't particularly mind if this behaviour is kept.]
- Since various bits of file-related functionality are touched by 'long' path issues, I thought it would be useful to collect ideas from my wish list that overlap with affected code.
- A_LoopFileAttribNum (or A_LoopFileAttribValue).
- A_LoopFileNameNoExt. (Useful in it's own right, plus it would be consistent with A_AhkNameNoExt and A_ScriptNameNoExt which would be very useful with #Include.)
- A_LoopFileTimeCreatedUTC/A_LoopFileTimeModifiedUTC (and A_LoopFileTimeAccessedUTC).
- A_Recent (or A_RecentItems) (Recent folder).
- FileInstall function for AHK v1.
Last edited by jeeswg on 14 May 2018, 15:18, edited 4 times in total.
User avatar
nnnik
Posts: 3072
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: long path support

21 Apr 2018, 15:07

The file object is meant for interacting with the file content not with the container.
Controlling both the container and the content with a single Object could cause issues if the data container is not a normal file - which could happen if it is not a normal file object but rather an object which implements the same Interface.
E.g. if you implement an object that can be used just Luke a file object but Handels the content of a file inside a .zip
Recommends AHK Studio
lexikos
Posts: 6045
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: long path support

21 Apr 2018, 21:11

Does this topic have some relevance to AutoHotkey v2 Development that I'm not seeing?
The file loop currently handles '260+' files in full, but not folders. It will list '260+' folders, but it will not recurse into them, and it will not accept '260+' folders as the starting directory.
I did some tests with MoveFile and CopyFile on Windows 7. Unless I'm missing something, it seems that the functions can handle '260+' files as input files, but not as output files.
You must be using them incorrectly.


You should already know the path of the file, since you open it. If you will need to recall that path later, then store it somewhere. If you have a specific need for GetFinalPathNameByHandle, you can call it. (For instance, to "normalize" a path which contains symbolic links or junctions.)
User avatar
jeeswg
Posts: 4793
Joined: 19 Dec 2016, 01:58
Location: UK

Re: long path support

22 Apr 2018, 01:26

- Based on my tests, there may actually be some issues with the file loop at present. Maybe your OSes, or your settings don't cause the problems. I'm using Windows 7.

- Re. the first quote. I tried a basic file loop, and it seems that it is recursing into some long-path folders, but not all of them, omitting some files. E.g. long paths within long paths within long paths.
- I used some code based on the script here, and it is listing all files:
259-char path limit workarounds - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=26170

- Some tests (below) suggested that, with FileAppend, the test build can handle files up to 285 chars long.

- Re. the second quote. Small/big to small works, small/big to big doesn't work. (To be clear I'm referring to the Winapi functions not the AHK built-in functions.) There's a MoveFile example at the top of the test script below.

- (Re. FileOpen, I was doing tests re. creating files, and short-form to long-form conversion. I thought that the object might have had more file properties available, but they're not strictly needed.)

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

lexikos
Posts: 6045
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: long path support

22 Apr 2018, 02:43

vNum := 300 ;didn't work

Your destination path is invalid, not too long. No individual component within the path can exceed the maximum length of a name supported by the file system.
The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters). To specify an extended-length path, use the "\\?\" prefix.
Source: Naming Files, Paths, and Namespaces (Windows)
User avatar
jeeswg
Posts: 4793
Joined: 19 Dec 2016, 01:58
Location: UK

Re: long path support

22 Apr 2018, 03:09

- Thanks so much lexikos. I'll clear up my earlier posts at some point, now that I know about this issue. It's just the question of folder recursion left.
- Oh dear, with a maximum component length of 255, there's virtually no benefit in having long paths. (Although I notice it says 'commonly'.)
- New test code that works to create files/folders with long paths:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

lexikos
Posts: 6045
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: long path support

22 Apr 2018, 05:15

Oh dear, with a maximum component length of 255, there's virtually no benefit in having long paths.
That's ridiculous. Of course, the purpose of long paths is to create files with absurdly long names in the root of the drive. MAX_PATH is exactly enough for the drive letter (C:\), a name of 255 characters and a null-terminator, so there's no point allowing for paths longer than that. :roll:
(Although I notice it says 'commonly'.)
Don't get your hopes up. The exception is UDF, which only supports 127 Unicode or 254 ASCII characters. FindFirstFile cannot possibly support names longer than 259 characters, because _countof(WIN32_FIND_DATA::cFileName) == MAX_PATH.
User avatar
jeeswg
Posts: 4793
Joined: 19 Dec 2016, 01:58
Location: UK

Re: long path support

22 Apr 2018, 05:48

- The file loop is working if the \\?\ prefix is used [in both test builds: 1.1.28.02-7+ga1186b3 and 1.1.28.02-10+g1a41623], curiously, some but not all long paths are retrieved if you omit it (which is what confused me). Also, the list you get with \\?\ omitted is slightly different to the one you get in the previous official release (not that I mind particularly). [Files retrieved in a file loop in the previous official release, did not appear to exceed 259 characters.]
- With the \\?\ prefix included, the loop gave exactly the same results as my 2016 script, I'm most grateful.
- I'm not sure whether you stated anywhere that '\\?\' must be used (I'm happy either way).
- Here's a working example script:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


- Re. the 255 limit.
- I don't particularly like or want to use long paths.
- Experience dictates that it's better to avoid using them.
- Limits used to be 8 (well, 8+1+3 = 12), now 255, I thought the idea was to go one better.
- Long paths normally only occur for me when webpages are saved.
- I need to be able to list/hash/move/copy them when backing up files.
lexikos
Posts: 6045
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: long path support

22 Apr 2018, 06:05

... curiously, some but not all long paths are retrieved if you omit it.
I already explained this behaviour in the test build topic.
I'm not sure whether you stated anywhere that '\\?\' must be used
I did, in the test build topic.
User avatar
jeeswg
Posts: 4793
Joined: 19 Dec 2016, 01:58
Location: UK

Re: long path support

09 May 2018, 10:23

- Thanks. These are the closest mentions I've found:
Without long path awareness, using a long path requires the \\?\ prefix, as in \\?\C:\.
...
Now, if a directory is able to be searched by the API (that is, if FindFirstFile succeeds), all files in that directory are returned.

- The test build thread doesn't mention getting long paths via the Clipboard variable. After doing some testing, I'm wondering whether it's sometimes actually impossible.
long paths: get paths from clipboard - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=48661
- My one use case is basically this: I open a folder, I want to do something with the selected file, but the path is too long, I try to get the path into AHK via the clipboard (fails), or I get the selected file path via an object (which appears to work but still has complications, i.e. you can't get the extension, code is in the link).
- Anyhow, I don't mind if this functionality isn't put into AHK, but I am trying to answer the inevitable question: 'why does the Clipboard variable only list shorter paths and not long paths?'
lexikos
Posts: 6045
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: long path support

09 May 2018, 21:10

Explorer cannot view a folder with a long path, or generally do anything with long paths. It fakes such by using short path names.

Return to “AutoHotkey v2 Development”

Who is online

Users browsing this forum: No registered users and 2 guests