[solved] UrlDownloadToFile slows down GUI response

[Azevedo]

Hi,

I wrote this script to download files from a website.
However, using UrlDownloadToFile will slow down the GUI, which may stop responding.
For example, if UrlDownloadToFile is in progress, dragging the window by the title bar pauses the download while the window is being dragged.

Code: Select all

 ; GUI becomes irresponsive 
 ; UrlDownloadToFile, % url, % filepath
 
 ; GUI smooth as butter
RunWait, %A_ScriptDir%\download.vbs "%url%" "%filepath%"

To fix that I wrote a standalone VBScript that solves the problem:

download.vbs:

Code: Select all

	Dim xHttp		: Set xHttp = CreateObject("WinHttp.WinHttpRequest.5.1")
	Dim bitStream	: Set bitStream = CreateObject("Adodb.Stream")
	
	xHttp.Open "GET", WScript.Arguments(0), False
	xHttp.Send
	If xHttp.status = 200 Then
		With bitStream
			.type = 1 'binary
			.Open
			.Write xHttp.responseBody
			.SaveToFile WScript.Arguments(1) , 2
			.Close
		End With
	End If

It works, the GUI doesn't hang anymore as the download is being made in a different proccess (wscript.exe).

Is there a way to download *binary* (not text) files in AHK without hanging the GUI?

[gilliduck]

Maybe Run a second script that does the actual UrlDownloadToFile instead of it being in the main script? Depending on the amount of data being downloaded it can seriously impede the progress of the script. If it's running in an unrelated script, then it won't bother your main GUI. Don't #Include it, otherwise it's the same as putting it in the main script, just Run it.

[Azevedo]

I thought of that but downloading with vbscript (WSH) is way lighter than calling AutoHotkeyU64.exe (900Kb) again which will load a thousand things into memory before actually downloading the file.

Imagine a cycle of 100 files. poor cpu..

[gilliduck]

How many UrlDownloadToFile are you doing? Specifically how many unique downloads are you doing (are any being repeated and get you the same data as the first time it was run)?

[Azevedo]

It is a queue loop. may be 10 or 200.

Code: Select all

For index, image in this.images {

			RegExMatch(image, "imgur\.com\/(.*)", current) ; current file name
			current := current1
			filepath := this.destination . "\" . current
			
			if ( inStr(this.skip,  current) == 0 ) { ; file was not downloaded yet
				
				try {
					;UrlDownloadToFile, % image, % filepath ; GUI becomes irresponsive 
					RunWait, %A_ScriptDir%\download.vbs "%image%" "%filepath%"
				} catch e {
					MsgBox, 16, , % "Error downloading file `n" . image
				}
				
				if ( FileExist(filepath) != "A") {
					this.iFail++
					MsgBox, 16, , % "Error downloading file `n" . image
				} else {
					this.iDone++
				}
				
			} else {  ; file already exists
				this.iDone++
			}
			
			if this.paused 
				return
		}

[lexikos]

There are two examples in the help file for HTTP requests which do not make the script unresponsive. To get binary data, you can use ResponseBody instead of ResponseText.

I thought of that but downloading with vbscript (WSH) is way lighter than calling AutoHotkeyU64.exe (900Kb) again

How do you figure? If file size is relevant (doubtful), consider that wscript.exe+vbscript.dll = 735KB, and who knows what else vbscript.dll loads.

Either way it's irrelevant if a single child process does all the downloading - you only need to initialize once. For instance, you can write the list of files to a script and have it download everything, or you can have the script read filenames from stdin one at a time.

Also, I think your assumption that VBScript will load faster is false. My benchmarks with a single-line script showed that RunWait test.vbs and Run test.ahk performed exactly the same (for 100 iterations). If you were to rewrite your vbs script in ahk, it would probably perform much the same. VBScript may or may not be more optimized, but it's likely that any difference in language performance will be greatly outweighed by network latency/download time and the overhead of running a new process.

[Azevedo]

There are two examples in the help file for HTTP requests which do not make the script unresponsive. To get binary data, you can use ResponseBody instead of ResponseText.

I've seen all these examples. None of them show how to save a binary stream to a file (like I did in vbscript in the 1st post). Do you know how to do that in AHK?
So I am a dumb to come here and ask a question that is in the help file? Where is in the help file "save a binary stream to a file using HTTPRequest"? Please don't imply that I'm lazy and I didn't looked in the help file.

PS: If AHK performs better than VBS/WSH in your computer then good for you! I'm not the one to prove contrary.

[tmplinshi]

Code: Select all

WinHttp_DownloadToFile("http://ahkscript.org/static/ahk_logo.png", "logo.png")

WinHttp_DownloadToFile(Url, FileName) {
	whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
	whr.Open("GET", Url, True)
	whr.Send()
	If whr.WaitForResponse()
	{
		ado := ComObjCreate("ADODB.Stream")
		ado.Type := 1 ; adTypeBinary
		ado.Open
		ado.Write(whr.ResponseBody)
		ado.SaveToFile(FileName, 2)
		ado.Close
	}
}

[Azevedo]

Thanks tmplinshi!
Worked perfectly.
And the GUI didn't froze.

[just me]

Seemingly there's no need to use ADO in this case:

Code: Select all

WinHttp_DownloadToFileBin(Url, FileName) {
   WHR := ComObjCreate("WinHttp.WinHttpRequest.5.1")
   WHR.Open("GET", Url, True)
   WHR.Send()
   WHR.WaitForResponse()
   ResponseLength := WHR.ResponseBody.MaxIndex()
   FileLength := 0
   VarSetCapacity(Index, 4, 0)
   If !DllCall("OleAut32.dll\SafeArrayPtrOfIndex", "Ptr", ComObjValue(WHR.ResponseBody), "Ptr", &Index, "PtrP", PtrOfIndex) {
      hFile := FileOpen(FileName, "w")
      hFile.RawWrite(PtrOfIndex + 0, ResponseLength)
      FileLength := hFile.Pos
      hFile.Close()
   }
   Return FileLength
}

[Azevedo]

hmm interesting. I'll try that too!
Danke

[tmplinshi]

Seemingly there's no need to use ADO in this case:

Code: Select all

WinHttp_DownloadToFileBin(Url, FileName) {
   WHR := ComObjCreate("WinHttp.WinHttpRequest.5.1")
   WHR.Open("GET", Url, True)
   WHR.Send()
   WHR.WaitForResponse()
   ResponseLength := WHR.ResponseBody.MaxIndex()
   FileLength := 0
   VarSetCapacity(Index, 4, 0)
   If !DllCall("OleAut32.dll\SafeArrayPtrOfIndex", "Ptr", ComObjValue(WHR.ResponseBody), "Ptr", &Index, "PtrP", PtrOfIndex) {
      hFile := FileOpen(FileName, "w")
      hFile.RawWrite(PtrOfIndex + 0, ResponseLength)
      FileLength := hFile.Pos
      hFile.Close()
   }
   Return FileLength
}

nice

[lexikos]

I've seen all these examples.

You didn't give any indication that you had, so I assumed you had not. Why would I tell you something I think you already know?

None of them show how to save a binary stream to a file (like I did in vbscript in the 1st post). Do you know how to do that in AHK?

Yes, using ResponseBody as I indicated. ResponseBody is an array of bytes, so you can do whatever you want with the data, including writing to a file. In fact, you already had code to deal with ResponseBody in your first post - it would just need minimal translation for use in AutoHotkey.

PS: If AHK performs better than VBS/WSH in your computer then good for you!

My implication was the opposite, but that it was irrelevant. It was based on assumptions and not real benchmarks.

[Azevedo]

Ok. It was a misunderstanding. My bad.
Thanks buddy!

[Taurus]

Hi, thats a great script. I am interested, why UrlDownloadToFile is making the script unresponseable. Where is the difference to WinHttpRequest? (heavier load?)

At the moment i am developing an app, which downloads a 50 MB file. So i am interested, which has the best performance?

UrlDownloadToFile vs WinHttpRequest?

Does WinHttpRequest: ResponseBody + ADODB.Stream vs ResponseStream + omObjQuery(STREAM, "{0000000c-0000-0000-C000-000000000046}") make a difference?

Thanks for your answers/ideas!

[jeeswg]

I was doing some testing recently and I believe that:
ResponseLength := WHR.ResponseBody.MaxIndex()
should be:
ResponseLength := 1+WHR.ResponseBody.MaxIndex()
because the array is 0-based.

For a 0-byte file, oHTTP.ResponseBody.MaxIndex() returns blank.
For a 1-byte file, oHTTP.ResponseBody.MaxIndex() returns 0.

So, some code might be something like this:

Code: Select all

;vUrl := "https://raw.githubusercontent.com/f5devcentral/f5-irule-editor/master/iRuler/Templates/Blank.txt" ;1 byte
;vUrl := "https://raw.githubusercontent.com/SparkPost/java-sparkpost/master/src/main/resources/empty.txt" ;0 bytes

if IsObject(oHTTP.ResponseBody)
	vSize := 1+oHTTP.ResponseBody.MaxIndex()
else
	vSize := 0