AutoHotkey Community

It is currently May 27th, 2012, 8:54 am

All times are UTC [ DST ]




Post new topic Reply to topic  [ 82 posts ]  Go to page 1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: UrlDownloadToVar
PostPosted: June 13th, 2006, 7:48 pm 
Offline

Joined: June 4th, 2005, 1:30 am
Posts: 113
Location: Stuttgart, Germany
I recommend using the WinHTTP COM alternative from page 2 of this thread. It doesn't suffer from some of the unresolved issues in the WinInet script.

Supplementary to UrlDownloadToFile:
Not yet fully functional!
Code:
msgbox % UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt")

UrlDownloadToVar(URL, Proxy="", ProxyBypass="") {
AutoTrim, Off
hModule := DllCall("LoadLibrary", "str", "wininet.dll")

If (Proxy != "")
AccessType=3
Else
AccessType=1
;INTERNET_OPEN_TYPE_PRECONFIG                    0   // use registry configuration
;INTERNET_OPEN_TYPE_DIRECT                       1   // direct to net
;INTERNET_OPEN_TYPE_PROXY                        3   // via named proxy
;INTERNET_OPEN_TYPE_PRECONFIG_WITH_NO_AUTOPROXY  4   // prevent using java/script/INS

io_hInternet := DllCall("wininet\InternetOpenA"
, "str", "" ;lpszAgent
, "uint", AccessType
, "str", Proxy
, "str", ProxyBypass
, "uint", 0) ;dwFlags

iou := DllCall("wininet\InternetOpenUrlA"
, "uint", io_hInternet
, "str", url
, "str", "" ;lpszHeaders
, "uint", 0 ;dwHeadersLength
, "uint", 0x80000000 ;dwFlags: INTERNET_FLAG_RELOAD = 0x80000000 // retrieve the original item
, "uint", 0) ;dwContext

If (ErrorLevel != 0 or iou = 0) {
DllCall("FreeLibrary", "uint", hModule)
return 0
}

VarSetCapacity(buffer, 512, 0)
VarSetCapacity(NumberOfBytesRead, 4, 0)
Loop
{
  irf := DllCall("wininet\InternetReadFile", "uint", iou, "uint", &buffer, "uint", 512, "uint", &NumberOfBytesRead)
  NOBR = 0
  Loop 4  ; Build the integer by adding up its bytes. - ExtractInteger
    NOBR += *(&NumberOfBytesRead + A_Index-1) << 8*(A_Index-1)
  IfEqual, NOBR, 0, break
  ;BytesReadTotal += NOBR
  DllCall("lstrcpy", "str", buffer, "uint", &buffer)
  res = %res%%buffer%
}
StringTrimRight, res, res, 2

DllCall("wininet\InternetCloseHandle",  "uint", iou)
DllCall("wininet\InternetCloseHandle",  "uint", io_hInternet)
DllCall("FreeLibrary", "uint", hModule)
AutoTrim, on
return, res
}


Edit:
Replaced "res := res . buffer" with "res = %res%%buffer%" for better performance, as Chris recommended.
Also removed A_ScriptName as lpszAgent
Edit: Added AutoTrim, off for correct concatenation (thanks PhiLho)


Last edited by olfen on June 23rd, 2007, 8:42 am, edited 4 times in total.

Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 14th, 2006, 9:45 am 
Offline

Joined: December 27th, 2005, 1:46 pm
Posts: 6837
Location: France (near Paris)
Nice, it has been asked some times!

_________________
Image vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 15th, 2006, 1:56 am 
Offline

Joined: March 2nd, 2004, 3:36 pm
Posts: 10720
I think I may use this for server uptime monitoring. Thanks.


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 17th, 2006, 5:10 am 
Offline

Joined: March 2nd, 2004, 3:36 pm
Posts: 10720
One minor improvement to this that might help performance when downloading huge webpages:

Replace: res := res . buffer
With: res = %res%%buffer%

The latter performs better due to an optimization.

To get even more performance out of it (once again, for large web pages or files), you could call VarSetCapacity(res, ExpectedSize) beforehand. But this requires that you have some idea of how large the file will be.


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 17th, 2006, 7:51 am 
Offline

Joined: June 4th, 2005, 1:30 am
Posts: 113
Location: Stuttgart, Germany
Chris wrote:
One minor improvement to this that might help performance when downloading huge webpages:

Replace: res := res . buffer
With: res = %res%%buffer%

The latter performs better due to an optimization.

Thanks, I changed the script accordingly.
Chris wrote:
To get even more performance out of it (once again, for large web pages or files), you could call VarSetCapacity(res, ExpectedSize) beforehand. But this requires that you have some idea of how large the file will be.

I will look into adding a call to fetch HTTP_QUERY_CONTENT_LENGTH via HttpQueryInfo as demonstrated here.
This would also allow to give feedback on download progress. Many thanks for the idea.

Also, do you think increasing the buffer size would improve performance?


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 17th, 2006, 12:23 pm 
Offline

Joined: December 27th, 2005, 1:46 pm
Posts: 6837
Location: France (near Paris)
Chris wrote:
Replace: res := res . buffer
With: res = %res%%buffer%
Maybe he should switch off AutoTrim, in this case, no?

_________________
Image vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 17th, 2006, 3:18 pm 
Offline

Joined: March 2nd, 2004, 3:36 pm
Posts: 10720
It might help, but it would have to be tested to find out if it's worth it.

olfen wrote:
do you think increasing the buffer size would improve performance?
That's a good question. I know it tends to do so with FTP, but these API Internet functions are probably pretty high above the network layer, so it might not make much of a difference. However, a smaller buffer size would probably make the script more responsive/granular since it would be likely to spend less time inside the call (especially when the download is slow, such as dial-up).


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 18th, 2006, 11:23 am 
Offline

Joined: December 27th, 2005, 1:46 pm
Posts: 6837
Location: France (near Paris)
Chris wrote:
It might help, but it would have to be tested to find out if it's worth it.
I didn't meant it as speed improvement (althought skipping trimming might be beneficial) but, if I understood correctly the code portion, he fills the buffer with 512 bytes of data from the server, likely to be HTML, so it is no unlikely to have spaces or tabs at the end of buffer.
res := res . buffer
will preserves these spaces while
res = %res%%buffer%
will remove them, thus alter the final result, if AutoTrim is On (default).

BTW, is it usable for binary data (image)? Althought currently we can't do much use of such image in memory.

_________________
Image vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 18th, 2006, 1:27 pm 
Offline

Joined: June 4th, 2005, 1:30 am
Posts: 113
Location: Stuttgart, Germany
Chris, thanks for your considerations regarding the buffer size.
I will do some automated testing to see if it makes a difference.
But first, I will have to fix this buggy thing: Files that are larger than the buffer size are getting a queer appendage...
Code:
a := UrlDownloadToVar("http://jm.greatnow.com/ahk_test_20060618140900_517b.txt")
b := UrlDownloadToVar("http://jm.greatnow.com/ahk_spaces_20060618140900_224b.txt")
fileappend, %a%, a.txt
fileappend, %b%, b.txt

@PhiLho
AutoTrim, Off really is neccessary to prevent loss of whitespace. The second URL in the code above demonstates it. Thanks.

I'll check if it can be used to fetch binary data, as soon as the other problems are solved...


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: July 28th, 2006, 9:14 am 
Offline

Joined: July 2nd, 2006, 10:39 am
Posts: 7
Autotrim should not be set to "on" at the end of the function if it wasn't befor.

Is there any way to make the function binary save to downlaod images or zips and save them to file? I'd like to use the proxy login of this function for downlaods, which UrlDownloadToFile can't do.


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: January 7th, 2007, 4:06 am 
Offline

Joined: June 4th, 2005, 1:30 am
Posts: 113
Location: Stuttgart, Germany
Here's an update of the function:
Code:
UrlDownloadToVar(URL, ByRef Result, UserAgent = "", Proxy = "", ProxyBypass = "") {
  ; Requires Windows Vista, Windows XP, Windows 2000 Professional, Windows NT Workstation 4.0,
  ; Windows Me, Windows 98, or Windows 95.
  ; Requires Internet Explorer 3.0 or later.
 
  hModule := DllCall("LoadLibrary", "Str", "wininet.dll")

  AccessType := Proxy != "" ? 3 : 1
  ;INTERNET_OPEN_TYPE_PRECONFIG                    0   // use registry configuration
  ;INTERNET_OPEN_TYPE_DIRECT                       1   // direct to net
  ;INTERNET_OPEN_TYPE_PROXY                        3   // via named proxy
  ;INTERNET_OPEN_TYPE_PRECONFIG_WITH_NO_AUTOPROXY  4   // prevent using java/script/INS

   io := DllCall("wininet\InternetOpenA"
   , "Str", UserAgent ;lpszAgent
   , "UInt", AccessType
   , "Str", Proxy
   , "Str", ProxyBypass
   , "UInt", 0) ;dwFlags
   
   iou := DllCall("wininet\InternetOpenUrlA"
   , "UInt", io
   , "Str", url
   , "Str", "" ;lpszHeaders
   , "UInt", 0 ;dwHeadersLength
   , "UInt", 0x80000000 ;dwFlags: INTERNET_FLAG_RELOAD = 0x80000000 // retrieve the original item
   , "UInt", 0) ;dwContext
   
   If (ErrorLevel != 0 or iou = 0) {
     DllCall("FreeLibrary", "UInt", hModule)
     return 0
   }
   
   VarSetCapacity(buffer, 10240, 0)
   VarSetCapacity(BytesRead, 4, 0)
   
   Loop
   {
     ;http://msdn.microsoft.com/library/en-us/wininet/wininet/internetreadfile.asp
     irf := DllCall("wininet\InternetReadFile", "UInt", iou, "UInt", &buffer, "UInt", 10240, "UInt", &BytesRead)
     VarSetCapacity(buffer, -1) ;to update the variable's internally-stored length
    
    BytesRead_ = 0 ; reset
     Loop, 4  ; Build the integer by adding up its bytes. (From ExtractInteger-function)
      BytesRead_ += *(&BytesRead + A_Index-1) << 8*(A_Index-1) ;Bytes read in this very DllCall
   
    ; To ensure all data is retrieved, an application must continue to call the
    ; InternetReadFile function until the function returns TRUE and the lpdwNumberOfBytesRead parameter equals zero.
    If (irf = 1 and BytesRead_ = 0)
       break
     Else ; append the buffer's contents
       Result .= SubStr(buffer, 1, BytesRead_)
   
    /* optional: retrieve only a part of the file
    BytesReadTotal += BytesRead_
    If (BytesReadTotal >= 30000) ; only read the first x bytes
       break                      ; (will be a multiple of the buffer size, if the file is not smaller; trim if neccessary)
     */
   }
   
   DllCall("wininet\InternetCloseHandle",  "UInt", iou)
   DllCall("wininet\InternetCloseHandle",  "UInt", io)
   DllCall("FreeLibrary", "UInt", hModule)
}


@jared: I'll try if it can be modified to handle binary data, but I can't promise anything. It won't be soon either.


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: January 8th, 2007, 11:36 am 
Offline

Joined: January 18th, 2006, 7:39 am
Posts: 274
Location: Conway, Arkansas
Would you mind providing an example call using the update? I can't seem to make it work at all. From my tests, it's spinning its wheels within the function's loop. Thanks.


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: January 8th, 2007, 1:06 pm 
skwire: worked fine for me with this code:
Code:
#SingleInstance, force

UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)
filedelete, urldata.txt
fileappend, %urldata%,urldata.txt
msgbox, urldata:`n%urldata%
exitapp


@olfen any time you get around to handling binary data, it will be appreciated :D


Report this post
Top
  
Reply with quote  
 Post subject:
PostPosted: January 8th, 2007, 4:15 pm 
Noobish/BoBoish question: (how) would it be possible to process an already loaded page (the repeat of a previously POSTed form) that way?
(I guess it will be a 'localhost' kinda thing ...:?)


Report this post
Top
  
Reply with quote  
 Post subject:
PostPosted: January 8th, 2007, 10:00 pm 
Offline

Joined: January 18th, 2006, 7:39 am
Posts: 274
Location: Conway, Arkansas
Gah...my fault. I was trying to test it like this:

Code:
clipboard := UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)


But since the function is using a ByRef return variable, that will never work. I should have been using it like this:

Code:
UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)
clipboard := urldata


Report this post
Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 82 posts ]  Go to page 1, 2, 3, 4, 5, 6  Next

All times are UTC [ DST ]


Who is online

Users browsing this forum: Bing [Bot], IsNull, tomoe_uehara, Xx7 and 12 guests


You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Group