AutoHotkey Community
It is currently May 27th, 2012, 1:40 pm

View unanswered posts | View active topics


Board index » AutoHotkey » Scripts

All times are UTC [ DST ]



Post new topic Reply to topic  Page 1 of 11
  [ 152 posts ]  Go to page 1, 2, 3, 4, 5 ... 11  Next
  Print view | E-mail friend Previous topic | Next topic 
Author Message
SKAN
PostPosted: June 25th, 2009, 5:38 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
    InternetFileRead()

    As the name suggests, this function reads a file from internet (passed to it as URL).
      It is similar to AHK FileRead command in following ways:
      1) Reads the content into Memory Variable
      2) Can read the specified number of bytes from the leading part of the file.
      3) Data retrieved will be in binary ( but to process it, we are on our own )

      Parameters :

    • V : Variable name passed as ByRef
    • URL : A valid existing URL from which File will be read from
    • RB : Requested Bytes. Pass 0 to read the whole file or request the number of bytes to read.
    • bSz : Buffer Size. Default value is 1024 ( bytes ). File will be read in small chunks as specified with this parameter. You can increase it for faster downloads but should result in less responsive GUI.
    • DLP : The dynamic function name of the function that will handle the "Download Progress". I have included one .. You can add more in the same script and call them as per your criteria. A simple one would look like:
      Code:
      DLPCE( WP=0, LP=0, MSG="" ) {
       Tooltip, Downloading : %WP% / %LP%
      }
    • F : Flags. Default is INTERNET_FLAG_RELOAD|INTERNET_FLAG_DONT_CACHE which makes sure file is not read from/written into cache.


      Return Value & ErrorLevel:

      If there are no errors from API functions, the function will return the Bytes Read, which will be exactly equal to the Variable's capacity.
      If Bytes Read is lesser than the File/Content Length or If Bytes Read is equal to Requested Bytes, AHK ErrorLevel will be set to 1
      On an API error, the function will return a negative number which can interpreted as follows:

      -1 Internet Connection Problem
      -2 Internet Connection Problem
      -4 Length of file could not be ascertained
      -5 The above and one of the Internet handles could not closed
      -6 The above one more of the Internet handles could not closed
      -7 One of the Internet handles could not closed
      -8 Both of the the Internet handles could not closed

    The included dynamic function DLP() will display a progress-bar
    in the right-bottom for the screen:
    Image


Code:
/*
       ___       _                       _   _____ _ _      ____                _
      |_ _|_ __ | |_ ___ _ __ _ __   ___| |_|  ___(_) | ___|  _ \ ___  __ _  __| |
       | || '_ \| __/ _ \ '__| '_ \ / _ \ __| |_  | | |/ _ \ |_) / _ \/ _` |/ _` |
       | || | | | ||  __/ |  | | | |  __/ |_|  _| | | |  __/  _ <  __/ (_| | (_| |
      |___|_| |_|\__\___|_|  |_| |_|\___|\__|_|   |_|_|\___|_| \_\___|\__,_|\__,_|

                              by SKAN ( Suresh Kumar A N, arian.suresh@gmail.com )
                                      Created: 24-Jun-2009 | LastEdit: 15-Dec-2009
                       Forum Topic: www.autohotkey.com/forum/viewtopic.php?t=45718
           Included : InternetFileRead(), DLP() Progress Bar, VarZ_Save(),Examples
      ____________________________________________________________________________
      Credit:

      Olfen for his topics:

                          DllCall: HttpQueryInfo - Get HTTP headers
                          - www.autohotkey.com/forum/topic10510.html
                          UrlDownloadToVar()
                          - www.autohotkey.com/forum/topic10466.html
      Lexikos for:
                          For supporting this project with valuable info,
                          especially:

                          code/method to support FTP read
                          - www.autohotkey.com/forum/viewtopic.php?p=277646#277646

                          clarifying alternative parameter for proxy issues
                          - www.autohotkey.com/forum/viewtopic.php?p=279205#279205
                          - www.autohotkey.com/forum/viewtopic.php?p=279210#279210

      jballi for fixing an important bug
                          - www.autohotkey.com/forum/viewtopic.php?p=496321#496321

      Thanks to all the replies in Forum Topic which motivates me to perfect this.
*/

#SingleInstance Force

; Example 1: Download the leading 100 bytes of default HTML and extract a part of text.
URL := "http://www.formyip.com/"
If ( InternetFileRead( IP, URL, 100, 100, "No-Progress" ) > 0 )
  MsgBox, 64, Your External IP Address
        , % IP := SubStr( IP,SP:=InStr(IP,"My ip address ")+17,InStr(IP," ",0,SP+1)-SP )


; Example 2, Download a binary file ( AHK Script Decompiler ) and save it.
URL := "http://www.autohotkey.com/download/Exe2Ahk.exe"
If ( InternetFileRead( binData, URL, False, 10240, "No-Progress" ) > 0 && !ErrorLevel )
  If VarZ_Save( binData, A_ScriptDir "\Exe2Ahk.exe" )
     MsgBox, 64, AHK Script Compiler Downloaded and Saved, % A_ScriptDir "\Exe2Ahk.exe"


; Example 3, Download a FTP file: EditPlus 3.11 Evaluation Version (1 MB) and save it.
URL := "ftp://ftp.editplus.com/epp311_en.exe"
If ( InternetFileRead( binData, URL ) > 0 && !ErrorLevel )
    If VarZ_Save( binData, A_Temp "\epp311_en.exe" ) {
         Sleep 500
         DLP( False ) ; or use Progress, off
         Run %A_Temp%\epp311_en.exe
       }

; AHK will automatically unload libraries on exit. If you are particular, here is a method
; to unload Wininet library without a handle.
DllCall( "FreeLibrary", UInt,DllCall( "GetModuleHandle", Str,"wininet.dll") )
Return ;                                                 // end of auto-execute section //


InternetFileRead( ByRef V, URL="", RB=0, bSz=1024, DLP="DLP", F=0x84000000 ) {
 Static LIB="WININET\", CL="00000000000000", N=""
 QRL := 16
 If ! DllCall( "GetModuleHandle", Str,"wininet.dll" )
      DllCall( "LoadLibrary", Str,"wininet.dll" )
 If ! hIO:=DllCall( LIB "InternetOpenA", Str,N, UInt,4, Str,N, Str,N, UInt,0 )
   Return -1
 If ! (( hIU:=DllCall( LIB "InternetOpenUrlA", UInt,hIO, Str,URL, Str,N, Int,0, UInt,F
                                                            , UInt,0 ) ) || ErrorLevel )
   Return 0 - ( !DllCall( LIB "InternetCloseHandle", UInt,hIO ) ) - 2
 If ! ( RB  )
 If ( SubStr(URL,1,4) = "ftp:" )
    CL := DllCall( LIB "FtpGetFileSize", UInt,hIU, UIntP,0 )
 Else If ! DllCall( LIB "HttpQueryInfoA", UInt,hIU, Int,5, Str,CL, UIntP,QRL, UInt,0 )
   Return 0 - ( !DllCall( LIB "InternetCloseHandle", UInt,hIU ) )
            - ( !DllCall( LIB "InternetCloseHandle", UInt,hIO ) ) - 4
 VarSetCapacity( V,64 ), VarSetCapacity( V,0 )
 SplitPath, URL, FN,,,, DN
 FN:=(FN ? FN : DN), CL:=(RB ? RB : CL), VarSetCapacity( V,CL,32 ), P:=&V,
 B:=(bSz>CL ? CL : bSz), TtlB:=0, LP := RB ? "Unknown" : CL,  %DLP%( True,CL,FN )
 Loop {
       If ( DllCall( LIB "InternetReadFile", UInt,hIU, UInt,P, UInt,B, UIntP,R ) && !R )
       Break
       P:=(P+R), TtlB:=(TtlB+R), RemB:=(CL-TtlB), B:=(RemB<B ? RemB : B), %DLP%( TtlB,LP )
       Sleep -1
 } TtlB<>CL ? VarSetCapacity( T,TtlB ) DllCall( "RtlMoveMemory", Str,T, Str,V, UInt,TtlB )
  . VarSetCapacity( V,0 ) . VarSetCapacity( V,TtlB,32 ) . DllCall( "RtlMoveMemory", Str,V
  , Str,T, UInt,TtlB ) . %DLP%( TtlB, TtlB ) : N
 If ( !DllCall( LIB "InternetCloseHandle", UInt,hIU ) )
  + ( !DllCall( LIB "InternetCloseHandle", UInt,hIO ) )
   Return -6
Return, VarSetCapacity(V)+((ErrorLevel:=(RB>0 && TtlB<RB)||(RB=0 && TtlB=CL) ? 0 : 1)<<64)
}
;  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
; The following function is an add-on to provide "Download Progress" to InternetFileRead()
; InternetFileRead() calls DLP() dynamically, i.e., will not error-out if DLP() is missing
;  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

DLP( WP=0, LP=0, Msg="" ) {
 If ( WP=1 ) {
 SysGet, m, MonitorWorkArea, 1
 y:=(mBottom-46-2), x:=(mRight-370-2), VarSetCapacity( Size,16,0 )
 DllCall( "shlwapi.dll\StrFormatByteSize64A", Int64,LP, Str,Size, UInt,16 )
 Size := ( Size="0 bytes" ) ? N : "«" Size "»"
 Progress, CWE6E3E4 CT000020 CBF73D00 x%x% y%y% w370 h46 B1 FS8 WM700 WS400 FM8 ZH8 ZY3
         ,, %Msg%  %Size%, InternetFileRead(), Tahoma
 WinSet, Transparent, 210, InternetFileRead()
 } Progress,% (P:=Round(WP/LP*100)),% "Memory Download: " wp " / " lp " [ " P "`% ]"
 IfEqual,wP,0, Progress, Off
}
;  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
; The following function is a part of: VarZ 46L - Native Data Compression
; View topic : www.autohotkey.com/forum/viewtopic.php?t=45559
;  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

VarZ_Save( byRef V, File="" ) { ;   www.autohotkey.net/~Skan/wrapper/FileIO16/FileIO16.ahk
Return ( ( hFile :=  DllCall( "_lcreat", Str,File, UInt,0 ) ) > 0 )
                 ?   DllCall( "_lwrite", UInt,hFile, Str,V, UInt,VarSetCapacity(V) )
                 + ( DllCall( "_lclose", UInt,hFile ) << 64 ) : 0
}


Last edited by SKAN on December 15th, 2011, 5:56 pm, edited 11 times in total.

Report this post
Top
 Profile  
Reply with quote  
HotKeyIt
 Post subject:
PostPosted: June 25th, 2009, 8:51 pm 
Offline

Joined: June 18th, 2008, 8:36 am
Posts: 4923
Location: AHK Forum
Thank you, looks cool 8) and a nice progress bar :)
Can you give more info for parameters?

_________________
AHK_H (2alpha) AHF TT _Struct WatchDir Yaml _Input ObjTree RapidHotkey DynaRun :wink:


Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: June 25th, 2009, 10:08 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
HotKeyIt wrote:
Thank you, looks cool 8) and a nice progress bar :)


Thanks for trying my code. :)

HotKeyIt wrote:
Can you give more info for parameters?


Sure.. I have updated the title post.


Report this post
Top
 Profile  
Reply with quote  
Lexikos
 Post subject:
PostPosted: June 26th, 2009, 9:46 am 
Offline

Joined: October 17th, 2006, 4:15 pm
Posts: 7503
Location: Australia
Nice work. I especially like the progress window.

I was a little surprised that this part works:
Code:
 VarSetCapacity(CL,14,32)
 If ! ( RB  )
 If ! DllCall( LIB "HttpQueryInfoA", UInt,hIU, Int,5, UInt,&CL, UIntP,QRL, UInt,0 )
   Return 0 - ( !DllCall( LIB "InternetCloseHandle", UInt,hIU ) )
            - ( !DllCall( LIB "InternetCloseHandle", UInt,hIO ) ) - 4
StrLen(CL) shows 14 regardless of the apparent length of the string. The code still works because AutoHotkey doesn't fully support binary zero/null-terminator in strings - HttpQueryInfoA null-terminates the string and although the internal length field of CL remains incorrect, most uses of it in an expression would stop at the null-terminator.

Anyway, Str,CL would properly update the variable's length.

It's simple to support FTP in addition to HTTP; for instance, something like this should work:
Code:
 If SubStr(URL,1,4) = "ftp:"
    CL := DllCall( LIB "FtpGetFileSize", UInt,hIU, UIntP,0 )
 Else If ! DllCall( LIB "HttpQueryInfoA", UInt,hIU, Int,5, Str,CL, UIntP,QRL, UInt,0 )
   Return 0 - ( !DllCall( LIB "InternetCloseHandle", UInt,hIU ) )
            - ( !DllCall( LIB "InternetCloseHandle", UInt,hIO ) ) - 4

URLDownloadToFile uses InternetReadFileEx, which apparently doesn't support FTP.

I tested using a URL like ftp://user:password@host/path.


Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: June 26th, 2009, 3:53 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
Lexikos wrote:
Nice work. I especially like the progress window.


I am glad you liked it. :D

Quote:
I was a little surprised that this part works:
...
StrLen(CL) shows 14 regardless of the apparent length of the string.


That was the intended effect... VarSetCapacity() was inserted at the last minute of posting!
Original code used Static CL="00000000000000" and first example reset CL StrLen to 3 bytes ( Requested Bytes 100 ), and so, for the seconds call ( autohotkeyinstall.exe ), the size was 204!

Quote:
Anyway, Str,CL would properly update the variable's length.


Yes.. Thank you! That was the mistake. I have reverted back to Static CL="00000000000000" and changed the Type,Arg accordingly.

Quote:
It's simple to support FTP in addition to HTTP


Oh! Thank you very much Lexikos. I had tested the function for FTP with Requested Bytes and it worked... I had the wrong idea that it would require a FtpOpenFile() handle to call FtpFileGetSize() :oops:

BTW, do you know a simple way to ascertain the size of default html ( like http://www.autohotkey.com/ ) when HTTP_QUERY_CONTENT_LENGTH fails. I am looking at ahklerner's UrlGetContents(), but that looks pretty tough for me..

Thanks again for all the help. :)


Report this post
Top
 Profile  
Reply with quote  
Lexikos
 Post subject:
PostPosted: June 26th, 2009, 4:27 pm 
Offline

Joined: October 17th, 2006, 4:15 pm
Posts: 7503
Location: Australia
SKAN wrote:
BTW, do you know a simple way to ascertain the size of default html ( like http://www.autohotkey.com/ ) when HTTP_QUERY_CONTENT_LENGTH fails.
No, but UrlDownloadToFile works without knowing the size in advance - it reads into a fixed-size buffer, writes out to file, and repeats until there's no more to read. If you know the file is text, you can simply append to a variable. Otherwise you can use GlobalAlloc to allocate memory and GlobalReAlloc to resize it, preserving data... or use some other method.


Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: June 28th, 2009, 2:36 am 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
Lexikos wrote:
No, but UrlDownloadToFile works without knowing the size in advance - it reads into a fixed-size buffer, writes out to file, and repeats until there's no more to read. If you know the file is text, you can simply append to a variable.


I had thought on similar lines.. but I cannot generate a progress bar!

Quote:
Otherwise you can use GlobalAlloc to allocate memory and GlobalReAlloc to resize it, preserving data...


Cool!.. I never knew this!

Quote:
or use some other method.


I would have used VarSetCapacityEx(), but would be very slow, I guess


Report this post
Top
 Profile  
Reply with quote  
Lexikos
 Post subject:
PostPosted: June 28th, 2009, 9:49 am 
Offline

Joined: October 17th, 2006, 4:15 pm
Posts: 7503
Location: Australia
SKAN wrote:
I would have used VarSetCapacityEx(), but would be very slow, I guess
Yes, it basically has to do twice the work, and in a semi-interpreted scripting language.

Another advantage of Global(Re)Alloc is that because you reference the data by pointer, it is easily passed to or returned from a function, referenced in another structure, etc. On the other hand, string operations are more difficult and costly.

Note that there are numerous (similar) alternatives to the Global memory management functions such as the Heap, Virtual, Local or C run-time functions. See Memory Management Functions and malloc/free.


Report this post
Top
 Profile  
Reply with quote  
Sean
 Post subject:
PostPosted: June 28th, 2009, 11:16 am 
Offline

Joined: February 12th, 2007, 7:54 am
Posts: 2462
Lexikos wrote:
See Memory Management Functions and malloc/free.
Those omit another important function which is actually my recommendation: CoTaskMem.... Although it's not mentioned explicitly in the page, I think it's thread-safe which is not that important with current AHK though.
http://msdn.microsoft.com/en-us/library/aa366533.aspx


Report this post
Top
 Profile  
Reply with quote  
Drugwash
 Post subject:
PostPosted: June 28th, 2009, 5:02 pm 
Offline
User avatar

Joined: September 8th, 2008, 12:26 am
Posts: 1048
Location: Ploieşti, RO
All 3 examples work perfectly on my 98SE system. 8)

If I may: first thought when I saw the function name was: why so long? I'd make it WebFileRead - easy to use, compact.

Nice job SKAN, thanks! :wink:


Report this post
Top
 Profile  
Reply with quote  
tank
 Post subject:
PostPosted: June 28th, 2009, 6:26 pm 
Offline
User avatar

Joined: December 21st, 2007, 3:14 pm
Posts: 3826
Location: Louisville KY USA
damnit this is brutifule
this is the coolest f$%^& thing done in ahk since COM

_________________
No matter what your oppinion Please join this discussion
Formal request to Polyethene
Image


Report this post
Top
 Profile  
Reply with quote  
DeWild1
 Post subject:
PostPosted: June 29th, 2009, 7:27 am 
Offline

Joined: April 30th, 2006, 6:23 pm
Posts: 358
Location: Shigle Springs
holly fricken cow, you just cut out so many lines of my old code, in so many programs with this.
8) 8) 8)
Thank you!

_________________
CPULOCK.com
virusSWAT.com
Computer Repair Computer Service.com
911PCFIX.com


Report this post
Top
 Profile  
Reply with quote  
Drugwash
 Post subject:
PostPosted: June 30th, 2009, 4:12 pm 
Offline
User avatar

Joined: September 8th, 2008, 12:26 am
Posts: 1048
Location: Ploieşti, RO
I've already stumbled into an issue with this function: can't download anything that does not end the URL in an explicit file type. :(

Testing was performed using URL := "http://www.pcidatabase.com/reports.php?type=csv" . This should download a file called reports.php whose format depends on the trailing parameter ?type=csv.

Any idea on how this can be fixed?


Report this post
Top
 Profile  
Reply with quote  
Sean
 Post subject:
PostPosted: July 1st, 2009, 1:55 am 
Offline

Joined: February 12th, 2007, 7:54 am
Posts: 2462
Drugwash wrote:
Any idea on how this can be fixed?
In analogy with ReadFile, it appears to me similar as you're asking ReadFile to format the data in the file to Unicode. Does it make a sense? I highly doubt InternetReadFile knows anything about MIME type.

PS. BTW, you can use this in that case, although it's no longer maintained.
http://www.autohotkey.com/forum/topic19475.html

PS2. I just tested InternetReadFile with the following script, and it worked! So, I reckon it's a problem of HttpQueryInfo failing to retrieve the size.
http://www.autohotkey.com/forum/topic19608.html


Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: July 1st, 2009, 5:32 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
Drugwash wrote:
I've already stumbled into an issue with this function: can't download anything that does not end the URL in an explicit file type.


As Sean said, this happens because HttpQueryInfo fails to retrieve the file size... and that was what I have been discussing with Lexikos a few posts above.

Quote:
Any idea on how this can be fixed?


You may force download the file if you can guesstimate the file size:

Code:
URL := "http://www.pcidatabase.com/reports.php?type=csv"
If ( InternetFileRead( CSV, URL, 2**19 ) > 0 ) && !ErrorLevel ; 2**19 is 512K
  If VarZ_Save( CSV, "reports.csv" )
     Run Notepad Reports.csv


Note: AHK ErrorLevel will be 0 if complete file was downloaded else it will be 1


Refer the first example on title post... http://www.formyip.com/ would also fail if I had not specified 100 bytes..


Report this post
Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 11
  [ 152 posts ]  Go to page 1, 2, 3, 4, 5 ... 11  Next

Board index » AutoHotkey » Scripts

All times are UTC [ DST ]

Who is online

Users browsing this forum: No registered users and 21 guests

You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Group