 |
AutoHotkey Community
Let's help each other out
|
| View previous topic :: View next topic |
| Author |
Message |
olfen
Joined: 04 Jun 2005
Posts: 99
Location: Stuttgart, Germany
|
 Posted: Tue Jun 13, 2006 7:48 pm Post subject: UrlDownloadToVar |
 |
|
I recommend using the WinHTTP COM alternative from page 2 of this thread. It doesn't suffer from some of the unresolved issues in the WinInet script.
Supplementary to UrlDownloadToFile:
Not yet fully functional!
| Code: |
msgbox % UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt")
UrlDownloadToVar(URL, Proxy="", ProxyBypass="") {
AutoTrim, Off
hModule := DllCall("LoadLibrary", "str", "wininet.dll")
If (Proxy != "")
AccessType=3
Else
AccessType=1
;INTERNET_OPEN_TYPE_PRECONFIG 0 // use registry configuration
;INTERNET_OPEN_TYPE_DIRECT 1 // direct to net
;INTERNET_OPEN_TYPE_PROXY 3 // via named proxy
;INTERNET_OPEN_TYPE_PRECONFIG_WITH_NO_AUTOPROXY 4 // prevent using java/script/INS
io_hInternet := DllCall("wininet\InternetOpenA"
, "str", "" ;lpszAgent
, "uint", AccessType
, "str", Proxy
, "str", ProxyBypass
, "uint", 0) ;dwFlags
iou := DllCall("wininet\InternetOpenUrlA"
, "uint", io_hInternet
, "str", url
, "str", "" ;lpszHeaders
, "uint", 0 ;dwHeadersLength
, "uint", 0x80000000 ;dwFlags: INTERNET_FLAG_RELOAD = 0x80000000 // retrieve the original item
, "uint", 0) ;dwContext
If (ErrorLevel != 0 or iou = 0) {
DllCall("FreeLibrary", "uint", hModule)
return 0
}
VarSetCapacity(buffer, 512, 0)
VarSetCapacity(NumberOfBytesRead, 4, 0)
Loop
{
irf := DllCall("wininet\InternetReadFile", "uint", iou, "uint", &buffer, "uint", 512, "uint", &NumberOfBytesRead)
NOBR = 0
Loop 4 ; Build the integer by adding up its bytes. - ExtractInteger
NOBR += *(&NumberOfBytesRead + A_Index-1) << 8*(A_Index-1)
IfEqual, NOBR, 0, break
;BytesReadTotal += NOBR
DllCall("lstrcpy", "str", buffer, "uint", &buffer)
res = %res%%buffer%
}
StringTrimRight, res, res, 2
DllCall("wininet\InternetCloseHandle", "uint", iou)
DllCall("wininet\InternetCloseHandle", "uint", io_hInternet)
DllCall("FreeLibrary", "uint", hModule)
AutoTrim, on
return, res
}
|
Edit:
Replaced "res := res . buffer" with "res = %res%%buffer%" for better performance, as Chris recommended.
Also removed A_ScriptName as lpszAgent
Edit: Added AutoTrim, off for correct concatenation (thanks PhiLho)
Last edited by olfen on Sat Jun 23, 2007 8:42 am; edited 4 times in total |
|
| Back to top |
|
 |
PhiLho
Joined: 27 Dec 2005
Posts: 6702
Location: France (near Paris)
|
| Posted: Wed Jun 14, 2006 9:45 am Post subject: |
|
|
Nice, it has been asked some times!
_________________
 vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2") |
|
| Back to top |
|
|
Chris
Site Admin
Joined: 02 Mar 2004
Posts: 10450
|
| Posted: Thu Jun 15, 2006 1:56 am Post subject: |
|
|
| I think I may use this for server uptime monitoring. Thanks. |
|
| Back to top |
|
|
Chris
Site Admin
Joined: 02 Mar 2004
Posts: 10450
|
| Posted: Sat Jun 17, 2006 5:10 am Post subject: |
|
|
One minor improvement to this that might help performance when downloading huge webpages:
Replace: res := res . buffer
With: res = %res%%buffer%
The latter performs better due to an optimization.
To get even more performance out of it (once again, for large web pages or files), you could call VarSetCapacity(res, ExpectedSize) beforehand. But this requires that you have some idea of how large the file will be. |
|
| Back to top |
|
|
olfen
Joined: 04 Jun 2005
Posts: 99
Location: Stuttgart, Germany
|
| Posted: Sat Jun 17, 2006 7:51 am Post subject: |
|
|
| Chris wrote: | One minor improvement to this that might help performance when downloading huge webpages:
Replace: res := res . buffer
With: res = %res%%buffer%
The latter performs better due to an optimization. |
Thanks, I changed the script accordingly.
| Chris wrote: |
To get even more performance out of it (once again, for large web pages or files), you could call VarSetCapacity(res, ExpectedSize) beforehand. But this requires that you have some idea of how large the file will be. |
I will look into adding a call to fetch HTTP_QUERY_CONTENT_LENGTH via HttpQueryInfo as demonstrated here.
This would also allow to give feedback on download progress. Many thanks for the idea.
Also, do you think increasing the buffer size would improve performance? |
|
| Back to top |
|
|
PhiLho
Joined: 27 Dec 2005
Posts: 6702
Location: France (near Paris)
|
| Posted: Sat Jun 17, 2006 12:23 pm Post subject: |
|
|
| Chris wrote: | Replace: res := res . buffer
With: res = %res%%buffer% |
Maybe he should switch off AutoTrim, in this case, no?
_________________
vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2") |
|
| Back to top |
|
|
Chris
Site Admin
Joined: 02 Mar 2004
Posts: 10450
|
| Posted: Sat Jun 17, 2006 3:18 pm Post subject: |
|
|
It might help, but it would have to be tested to find out if it's worth it.
| olfen wrote: | | do you think increasing the buffer size would improve performance? |
That's a good question. I know it tends to do so with FTP, but these API Internet functions are probably pretty high above the network layer, so it might not make much of a difference. However, a smaller buffer size would probably make the script more responsive/granular since it would be likely to spend less time inside the call (especially when the download is slow, such as dial-up). |
|
| Back to top |
|
|
PhiLho
Joined: 27 Dec 2005
Posts: 6702
Location: France (near Paris)
|
| Posted: Sun Jun 18, 2006 11:23 am Post subject: |
|
|
| Chris wrote: | | It might help, but it would have to be tested to find out if it's worth it. |
I didn't meant it as speed improvement (althought skipping trimming might be beneficial) but, if I understood correctly the code portion, he fills the buffer with 512 bytes of data from the server, likely to be HTML, so it is no unlikely to have spaces or tabs at the end of buffer.
res := res . buffer
will preserves these spaces while
res = %res%%buffer%
will remove them, thus alter the final result, if AutoTrim is On (default).
BTW, is it usable for binary data (image)? Althought currently we can't do much use of such image in memory.
_________________
vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2") |
|
| Back to top |
|
|
olfen
Joined: 04 Jun 2005
Posts: 99
Location: Stuttgart, Germany
|
| Posted: Sun Jun 18, 2006 1:27 pm Post subject: |
|
|
Chris, thanks for your considerations regarding the buffer size.
I will do some automated testing to see if it makes a difference.
But first, I will have to fix this buggy thing: Files that are larger than the buffer size are getting a queer appendage... | Code: | a := UrlDownloadToVar("http://jm.greatnow.com/ahk_test_20060618140900_517b.txt")
b := UrlDownloadToVar("http://jm.greatnow.com/ahk_spaces_20060618140900_224b.txt")
fileappend, %a%, a.txt
fileappend, %b%, b.txt |
@PhiLho
AutoTrim, Off really is neccessary to prevent loss of whitespace. The second URL in the code above demonstates it. Thanks.
I'll check if it can be used to fetch binary data, as soon as the other problems are solved... |
|
| Back to top |
|
|
jared
Joined: 02 Jul 2006
Posts: 7
|
| Posted: Fri Jul 28, 2006 9:14 am Post subject: |
|
|
Autotrim should not be set to "on" at the end of the function if it wasn't befor.
Is there any way to make the function binary save to downlaod images or zips and save them to file? I'd like to use the proxy login of this function for downlaods, which UrlDownloadToFile can't do. |
|
| Back to top |
|
|
olfen
Joined: 04 Jun 2005
Posts: 99
Location: Stuttgart, Germany
|
| Posted: Sun Jan 07, 2007 4:06 am Post subject: |
|
|
Here's an update of the function:
| Code: |
UrlDownloadToVar(URL, ByRef Result, UserAgent = "", Proxy = "", ProxyBypass = "") {
; Requires Windows Vista, Windows XP, Windows 2000 Professional, Windows NT Workstation 4.0,
; Windows Me, Windows 98, or Windows 95.
; Requires Internet Explorer 3.0 or later.
hModule := DllCall("LoadLibrary", "Str", "wininet.dll")
AccessType := Proxy != "" ? 3 : 1
;INTERNET_OPEN_TYPE_PRECONFIG 0 // use registry configuration
;INTERNET_OPEN_TYPE_DIRECT 1 // direct to net
;INTERNET_OPEN_TYPE_PROXY 3 // via named proxy
;INTERNET_OPEN_TYPE_PRECONFIG_WITH_NO_AUTOPROXY 4 // prevent using java/script/INS
io := DllCall("wininet\InternetOpenA"
, "Str", UserAgent ;lpszAgent
, "UInt", AccessType
, "Str", Proxy
, "Str", ProxyBypass
, "UInt", 0) ;dwFlags
iou := DllCall("wininet\InternetOpenUrlA"
, "UInt", io
, "Str", url
, "Str", "" ;lpszHeaders
, "UInt", 0 ;dwHeadersLength
, "UInt", 0x80000000 ;dwFlags: INTERNET_FLAG_RELOAD = 0x80000000 // retrieve the original item
, "UInt", 0) ;dwContext
If (ErrorLevel != 0 or iou = 0) {
DllCall("FreeLibrary", "UInt", hModule)
return 0
}
VarSetCapacity(buffer, 10240, 0)
VarSetCapacity(BytesRead, 4, 0)
Loop
{
;http://msdn.microsoft.com/library/en-us/wininet/wininet/internetreadfile.asp
irf := DllCall("wininet\InternetReadFile", "UInt", iou, "UInt", &buffer, "UInt", 10240, "UInt", &BytesRead)
VarSetCapacity(buffer, -1) ;to update the variable's internally-stored length
BytesRead_ = 0 ; reset
Loop, 4 ; Build the integer by adding up its bytes. (From ExtractInteger-function)
BytesRead_ += *(&BytesRead + A_Index-1) << 8*(A_Index-1) ;Bytes read in this very DllCall
; To ensure all data is retrieved, an application must continue to call the
; InternetReadFile function until the function returns TRUE and the lpdwNumberOfBytesRead parameter equals zero.
If (irf = 1 and BytesRead_ = 0)
break
Else ; append the buffer's contents
Result .= SubStr(buffer, 1, BytesRead_)
/* optional: retrieve only a part of the file
BytesReadTotal += BytesRead_
If (BytesReadTotal >= 30000) ; only read the first x bytes
break ; (will be a multiple of the buffer size, if the file is not smaller; trim if neccessary)
*/
}
DllCall("wininet\InternetCloseHandle", "UInt", iou)
DllCall("wininet\InternetCloseHandle", "UInt", io)
DllCall("FreeLibrary", "UInt", hModule)
} |
@jared: I'll try if it can be modified to handle binary data, but I can't promise anything. It won't be soon either. |
|
| Back to top |
|
|
skwire
Joined: 18 Jan 2006
Posts: 129
Location: Conway, Arkansas
|
| Posted: Mon Jan 08, 2007 11:36 am Post subject: |
|
|
| Would you mind providing an example call using the update? I can't seem to make it work at all. From my tests, it's spinning its wheels within the function's loop. Thanks. |
|
| Back to top |
|
|
rogerr (as guest)
Guest
|
| Posted: Mon Jan 08, 2007 1:06 pm Post subject: |
|
|
skwire: worked fine for me with this code:
| Code: | #SingleInstance, force
UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)
filedelete, urldata.txt
fileappend, %urldata%,urldata.txt
msgbox, urldata:`n%urldata%
exitapp
|
@olfen any time you get around to handling binary data, it will be appreciated  |
|
| Back to top |
|
|
BoBo
Guest
|
| Posted: Mon Jan 08, 2007 4:15 pm Post subject: |
|
|
Noobish/BoBoish question: (how) would it be possible to process an already loaded page (the repeat of a previously POSTed form) that way?
(I guess it will be a 'localhost' kinda thing ... ) |
|
| Back to top |
|
|
skwire
Joined: 18 Jan 2006
Posts: 129
Location: Conway, Arkansas
|
| Posted: Mon Jan 08, 2007 10:00 pm Post subject: |
|
|
Gah...my fault. I was trying to test it like this:
| Code: | | clipboard := UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata) |
But since the function is using a ByRef return variable, that will never work. I should have been using it like this:
| Code: | UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)
clipboard := urldata
|
|
|
| Back to top |
|
|
|
|
You can post new topics in this forum
You can reply to topics in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Register
Profile
Log in to check your private messages
Log in
