download urls to vars, partially/fully, via WinHttpRequest Topic is solved

[Sam_]

- download htm to variable
- download file to variable e.g. image/icon
- download htm to variable (partial) - get webpage title

Code: Select all

; [...]

JEE_UrlDownloadToVar(vUrl, ByRef vData, ByRef vSize)
{
	oHTTP := ComObjCreate("WinHttp.WinHttpRequest.5.1")
	oHTTP.Open("GET", vUrl)
	oHTTP.Send()

	oHTTP.WaitForResponse()
	;vSize := oHTTP.ResponseBody.MaxIndex()
	vSize := IsObject(oHTTP.ResponseBody) ? 1+oHTTP.ResponseBody.MaxIndex() : 0
	VarSetCapacity(vData, vSize, 0)

	VarSetCapacity(vIndex, 4, 0)
	if !DllCall("oleaut32\SafeArrayPtrOfIndex", Ptr,ComObjValue(oHTTP.ResponseBody), Ptr,&vIndex, PtrP,vPtrOfIndex)
		DllCall("kernel32\RtlMoveMemory", Ptr,&vData, Ptr,vPtrOfIndex, UPtr,vSize)
	oHTTP := ""
}

;==================================================

I'm having some trouble getting this function to perform reliably. About half the files I try to download come out corrupted: the first bit of the file is good data, but the rest is nulls. After a couple of tries, I can usually get a good/full version. The final filesize is sometimes a few bytes different. Any idea what could be going wrong? It's like WaitForResponse() is sometimes returning before the data is fully there.

Here's my code:
Warning: It will try to download some PDFs to your computer

Code: Select all

Loop, 250
	{
	TryAgain:
	Name:=SubStr(000 A_Index,-2)
	URL:="https://babel.hathitrust.org/cgi/imgsrv/download/pdf?id=uva.x001010237&attachment=1&seq=" A_Index
	OutName:=A_ScriptDir "\uva-x001010237-" Name ".pdf"
	If FileExist(OutName)
		Continue
	Data:=Size:=""
	JEE_UrlDownloadToVar(URL,Data,Size)
	If (Size>6000)
		{
		Magic:=StrGet(&Data,4,"CP0")
		If (Magic="%PDF")
			{
			File:=FileOpen(OutName,"w-d")
				File.RawWrite(&Data,Size)
			File.Close()
			}
		Else
			{
			Sleep, 60000 ; 1 min
			GoTo, TryAgain
			}
		}
	Else
		{
		Sleep, 60000 ; 1 min
		GoTo, TryAgain
		}
	}
ExitApp