Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

Faster searching (Loop) of files in a folder?


  • Please log in to reply
7 replies to this topic
scriptmonkey
  • Members
  • 176 posts
  • Last active: Jan 08 2015 07:42 PM
  • Joined: 19 May 2006

I have a script that the user enters a text to search in a particular folder.    This folder contains about 20,000 PDF files. 

Example, the user types in "7773".   The script searches that folder for any file that starts with 7773.

I loop through the files in that folder, then add any matching files to a list view.  The user then can double-click on a item in the list view to open the PDF file.

 

Loop, %Item%*.pdf,0,1
{
LV_Add("", A_LoopFileFullPath,CTime(A_LoopFileTimeModified),A_LoopFileLongPath)
LV_ModifyCol(1,309)
LV_ModifyCol(2,220)
}

Using this Loop, it takes 25 seconds to search for a particular string (641 files found).   Doing a standard windows search in that same folder, it takes 4 seconds to get a list of the the same files.
 
Is there a better method to use in Autohotkey for searching these files?

Zak M.

sinkfaze
  • Moderators
  • 6367 posts
  • Last active: Nov 30 2018 08:50 PM
  • Joined: 18 Mar 2008
Why shouldn't your system perform the search faster? It indexes files for faster searching. Try looping through all of the files in advance and save the file details to an array, then later search the array for your queries.

pdfFiles :=	[]
Loop, *.pdf,0,1
	pdfFiles.Insert({name:A_LoopFileName,path:A_LoopFilePath,tmod:A_LoopFileTimeModified,lpath:A_LoopFileLongPath}
For each, file in pdfFiles
	if	InStr(file.name,Item)
	{
		LV_Add("",file.path,CTime(file.tmod),file.lpath)
		LV_ModifyCol(1,309)
		LV_ModifyCol(2,220)
	}


scriptmonkey
  • Members
  • 176 posts
  • Last active: Jan 08 2015 07:42 PM
  • Joined: 19 May 2006

Why shouldn't your system perform the search faster? It indexes files for faster searching. Try looping through all of the files in advance and save the file details to an array, then later search the array for your queries.
 

 

 

The PDF folder is on a network drive, which isnt indexed.  PDFs are added to this folder throughout the day, so an index would have to be constantly updated.   Are you suggesting that I create an index each time the user runs the script, then the script searches the index?


Zak M.

hd0202
  • Members
  • 709 posts
  • Last active: Feb 14 2016 08:05 PM
  • Joined: 13 Aug 2006
GuiControl, -Redraw, <YourListView>  ; Improve performance by disabling redrawing during load.
Loop, %Item%*.pdf,0,1
{
  LV_Add("", A_LoopFileFullPath,CTime(A_LoopFileTimeModified),A_LoopFileLongPath)
}
GuiControl, +Redraw, <YourListView>  ; Re-enable redrawing (it was disabled above).
LV_ModifyCol(1,309)
LV_ModifyCol(2,220)

Try this and see, how much time it saves.

 

Is your loop much faster if you do not use the "CTime" function and use the time without modification? Then check this function!

 

Hubert



scriptmonkey
  • Members
  • 176 posts
  • Last active: Jan 08 2015 07:42 PM
  • Joined: 19 May 2006
GuiControl, -Redraw, <YourListView>  ; Improve performance by disabling redrawing during load.
Loop, %Item%*.pdf,0,1
{
  LV_Add("", A_LoopFileFullPath,CTime(A_LoopFileTimeModified),A_LoopFileLongPath)
}
GuiControl, +Redraw, <YourListView>  ; Re-enable redrawing (it was disabled above).
LV_ModifyCol(1,309)
LV_ModifyCol(2,220)

Try this and see, how much time it saves.

 

Is your loop much faster if you do not use the "CTime" function and use the time without modification? Then check this function!

 

Hubert

 

 

Thanks for the reply Hubert.   I tested the code above, it saves a couple of seconds when searching a short wildcard.

Regarding the Ctime function, what are you suggesting I change?


Zak M.

hd0202
  • Members
  • 709 posts
  • Last active: Feb 14 2016 08:05 PM
  • Joined: 13 Aug 2006

I cannot make any suggestions regarding the Ctime function, as I do not know it (it is not contained in your post).

 

For testing purpose you should change one line:

LV_Add("", A_LoopFileFullPath,A_LoopFileTimeModified,A_LoopFileLongPath)

If you then get a significant difference in runtime, the Ctime function must be the cause and you should check (and change) it.

 

Hubert



scriptmonkey
  • Members
  • 176 posts
  • Last active: Jan 08 2015 07:42 PM
  • Joined: 19 May 2006

I cannot make any suggestions regarding the Ctime function, as I do not know it (it is not contained in your post).

 

For testing purpose you should change one line:

LV_Add("", A_LoopFileFullPath,A_LoopFileTimeModified,A_LoopFileLongPath)

If you then get a significant difference in runtime, the Ctime function must be the cause and you should check (and change) it.

 

Hubert

 

Saved a couple seconds.


Zak M.

steventaitinger
  • Members
  • 17 posts
  • Last active: May 02 2018 08:12 PM
  • Joined: 05 Feb 2013
;build folder list takes takes 200-300 ms
word_files :=[]
loop, M:\* , 0 , 1 
{
if instr(A_LoopFileName, "~");this part skips recording the short names
continue
word_files.Insert({path:A_LoopFileFullPath})
}

I implemented the solution of building an index of the files and then doing the for each, object in array type search through the list of filenames. It took my script from 10 seconds to less than a second because I need to do about 240 matches each time I run the script.
 
Something that is worth mentioning is that if you want your indexing loop to go faster or if you don't want to match the shortname of the file name (annoying!) then do the index loop like below. Also, don't use A_loopfilelongpath as it has to do extra conversions than A_LoopFileLongPath!