Efficient files and folders treeview population

[zcooler]

Hi!

Im getting increasingly concerned with how long it takes to populate a 30 row treeview if the folder im populating the treeview with contains large amounts of files and/or folders. So I decided to try to find something that would do this more efficient. It got tricky rather fast though. I would imagine the big load time stealer is the looping through files and folders part and not the actual writing/population of the treeview, yes? So since im already using AHK_H I came to think about the read part could be multi-thredded in order to speed up the total load time. Never done the multi-thredding stuff though so that might be a future endeavour. Instead I was focusing on the write part (populating the treeview) which gets very tricky after a while. What I had in mind was to populate the visible treeview area in a first stage and show the treeview. The second stage begins with assigning all unpopulated files and folders to an associative array and then populate the rest in the already showed (and workable) treeview (in other words background population). The only sign of the background population the user should see is how the scrollbar nob decreases in height (pretty much how for example MS Word works when loading a very large document).

The hard part begins when to take height for IsDir or not and if stage 1 should add child nodes or if those are better added in stage 2. Folders with many childnodes containing large videofiles takes longer to populate, but as said I do think that should be more related to the slowness of looping through the large files in those folders.

Anyway, anyone with expereince handling these kind of things are welcome to chip in

Here is a script by Pulover and Learning one which have the base to build on (I scribbled a bit on it but nothing substansial):

Code: Select all

;========================================================================
;
; 				TreeView Browser
;
; Author:		Pulover [Rodolfo U. Batista]
;				[email protected]
;
; Requires CreateTreeView.ahk (Thanks Learning one for this function)
; http://www.autohotkey.com/board/topic/92863-function-createtreeview/
;
;========================================================================

#NoEnv
SetWorkingDir %A_ScriptDir%
SetBatchLines, -1

; Customize File Extension (optional)
CustomExt = tvb

ImageListID := IL_Create(10)
Gui, +Resize +MinSize260x60
Gui, Add, Button, Default w70 gLoadFolder, Load Folder
Gui, Add, Button, w70 yp xp+85 gLoadFile, Load File
Gui, Add, Button, w70 yp xp+85 gSaveFile, Save File
Gui, Add, Checkbox, Checked yp+5 xp+85 vLoadIcons gLoadIcons, Load Icons
Gui, Add, TreeView, xm h400 w300 ImageList%ImageListID% vTreeView
Gui, Show,, TreeView Browser
return

LoadFolder:
Gui, Submit, NoHide
Gui, +OwnDialogs
FileSelectFolder, TreeRoot, *%A_MyDocuments%
if !TreeRoot
	return
TV_Delete()
Progress, M H80 W600 FS10,, Loading folders & files..., TreeView Browser
TVString := CreateString(TreeRoot)
Progress, Off
CreateTreeView(TVString)
return

LoadFile:
Gui, Submit, NoHide
Gui, +OwnDialogs
FileSelectFile, StrFile,,, Load File, TreeView (*.%CustomExt%)
if !StrFile
	return
TV_Delete()
TVString := ""
Progress, M H80 W600 FS10,, % "Loading tree" (LoadIcons ? " & icons" : "") "...", TreeView Browser
Loop, Read, %StrFile%
{
	RegExMatch(A_LoopReadLine, "\t+([^\t]+)", Line)
	TVString .= RegExReplace(A_LoopReadLine, "\tIcon\d+")
	. (LoadIcons ? "`tIcon" GetIcon(Line1) : "") "`n"
	Progress, %A_Index%, %Line1%
}
Progress, Off
CreateTreeView(TVString)
return

SaveFile:
Gui, +OwnDialogs
FormatTime, Today,, yyyy-MM-dd
FileSelectFile, SaveFile, S16, %Today%.%CustomExt%, Save File, TreeView (*.%CustomExt%)
If !SaveFile
	return
SplitPath, SaveFile,,, ext
If (ext <> CustomExt)
	SaveFile .= "." CustomExt
IfExist %SaveFile%
	FileDelete %SaveFile%
FileAppend, %TVString%, %SaveFile%, UTF-8
return

LoadIcons:
Gui, Submit, NoHide
If LoadIcons
	TV_SetImageList(ImageListID)
Else
	TV_SetImageList(0)
return

GuiSize:
if A_EventInfo = 1
    return
GuiControl, Move, TreeView, % "W" . (A_GuiWidth - 20) . " H" . (A_GuiHeight - 40)
return

GuiClose:
ExitApp

CreateString(Folder, Call=0)
{
	global LoadIcons
	
	Call++
	Loop, %Folder%\*.*, 1
	{
		Progress, %A_Index%, %A_LoopFileDir%
		If LoadIcons
			Icon := "`tIcon" GetIcon(A_LoopFileFullPath)
		If InStr(FileExist(A_LoopFileFullPath), "D")
		{
			Loop, %Call%
				String .= "`t"
			String .= A_LoopFileName . Icon "`n"
			String .= CreateString(A_LoopFileFullPath, Call)
		}
		Else
		{
			Loop, %Call%
				Files .= "`t"
			Files .= A_LoopFileName . Icon "`n"
		}
	}
	String .= Files
	Call--
	return String
}

GetIcon(FileName)
{
	global ImageListID

	sfi_size := A_PtrSize + 8 + (A_IsUnicode ? 680 : 340)
	VarSetCapacity(sfi, sfi_size)
	SplitPath, FileName,,, FileExt
	if FileExt in EXE,ICO,ANI,CUR
	{
		ExtID := FileExt
		IconNumber = 0
	}
	else
	{
		ExtID = 0
		Loop 7
		{
			StringMid, ExtChar, FileExt, A_Index, 1
			If not ExtChar
				break
			ExtID := ExtID | (Asc(ExtChar) << (8 * (A_Index - 1)))
		}
		IconNumber := IconArray%ExtID%
	}
	If not IconNumber
	{
		if not DllCall("Shell32\SHGetFileInfo" . (A_IsUnicode ? "W":"A"), "str",  "." FileExt
			, "uint", (FileExt ? 0x80 : 0), "ptr", &sfi, "uint", sfi_size, "uint", (FileExt ? 0x111 : 0x101))
			IconNumber = 9999999
		else
		{
			hIcon := NumGet(sfi, 0)
			IconNumber := DllCall("ImageList_ReplaceIcon", "ptr", ImageListID, "int", -1, "ptr", hIcon) + 1
			DllCall("DestroyIcon", "ptr", hIcon)
			IconArray%ExtID% := IconNumber
		}
	}
	return IconNumber
}
CreateTreeView(TreeViewDefinitionString) {	; by Learning one
	IDs := {} 
	Loop, parse, TreeViewDefinitionString, `n, `r
	{
		if A_LoopField is space
			continue
		Item := RTrim(A_LoopField, A_Space A_Tab), Item := LTrim(Item, A_Space), Level := 0
		While (SubStr(Item,1,1) = A_Tab)
			Level += 1,	Item := SubStr(Item, 2)
		RegExMatch(Item, "([^`t]*)([`t]*)([^`t]*)", match)	; match1 = ItemName, match3 = Options
		if (Level=0) { ; Level 0 = Folders
		  if (a_index < 4)  {
		    IDs["Level0"] := TV_Add(match1, 0, match3)
          }
		  else {
			IDs["Level0"] := TV_Add(match1, 0, match3)
		  }
		}
		else {         ; Level 1 = Files
		  if (a_index < 4) {
		      IDs["Level" Level] := TV_Add(match1, IDs["Level" Level-1], match3)
		    ;tooltip % a_index
			;sleep, 1000
          }
		  else {
		    sleep, 1000
		  	IDs["Level" Level] := TV_Add(match1, IDs["Level" Level-1], match3)
		  }
		}		
	}
}	; http://www.autohotkey.com/board/topic/92863-function-createtreeview/

/*
Hi! 

I am lately getting more and more annoyed with the extensive load time of filling my treeview with a +15.000 file music library and are contemplating some kind of soultion, but does not find any particular coding attemps doing this when searching. So I was thinking about a two stage load cycle where only files and folders (without child nodes) that are visible in the treeview area are loaded in a first stage. 
, trying to mimicing the MS Word 
understand it first step would be to  

[wolf_II]

You have not mentioned how long it takes? At least I have not seen it.
21,150 files + 2,533 folders in less than 2 seconds:

Code: Select all

SetBatchLines, -1
Gui, Add, TreeView, r30
Add_Tree("C:\Program Files (x86)")
Gui, Show
Return
GuiClose:
ExitApp
Add_Tree(Folder, ParentID := 0) {
    Loop, Files, %Folder%\*.*, DF
        Add_Tree(A_LoopFileFullPath, TV_Add(A_LoopFileName, ParentID, "expand"))
}

[zcooler]

You have not mentioned how long it takes?

Yes, I left that out intentionally to avoid getting caught up in such discussions. The timings might be dependent on which kind of drive you have those files and folders stored on. For me +6000 files and folders gives approximately a 3 second load time (sometimes it can take up to 5-6 seconds depending on CPU load and how long the treeview have been running without restart and no drive is allowed going to energy saving state). Im interested in all approaches to improve on these timings if possible.