AutoHotkey Community
It is currently May 27th, 2012, 12:58 pm

View unanswered posts | View active topics


Board index » General » Utilities

All times are UTC [ DST ]



Post new topic Reply to topic  Page 1 of 2
  [ 26 posts ]  Go to page 1, 2  Next
  Print view | E-mail friend Previous topic | Next topic 
Author Message
SKAN
PostPosted: October 1st, 2011, 4:51 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
Download URLGet.exe 64 KiB
See virusTotal results: http://goo.gl/0CPAU

    Features ( 0.9.0.0 ):
  • IE based ( urlmon/URLDownloadToCacheFileW ). Tested fine for me in: XP SP3 IE6
  • Small footprint 64 KiB, low processor usage ( but memory around 4 MiB per instance ).
  • No need to specify target filename. Auto-Increments target filename.
  • Automatically decodes content encoding.
  • Progress in TitleBar.
  • IPC via WM.

Credit: Many thanks to Sean for URLDownloadToCacheFile

Author: SKAN ( Suresh Kumar A N ).
License: Public Domain
FileType: 32bit Console
FileVersion: 0.9.00 ( MD5: 6a529f13ca4261cab359a0333b9572dd )

Usage Example:

URLGet "http://google.co.in/"
will download into IE cache and copy the file as google.co.htm into your current folder ( A_WorkingDir ) and returns the fullpath of the copied file.

URLGet "http://google.co.in/" 1
will omit the FileCopy and instead, will return the fullpath to the cache file.

________________________________________________________________________________________

Source code

This utility was written and compiled in Gentee: Open Source Free Programming Language

Source code released: http://gentee.com/phpbb/viewtopic.php?p=9822#p9822
View the code: http://dl.dropbox.com/u/6428211/Gentee/ ... 0/URLGet.g


Report this post
Top
 Profile  
Reply with quote  
SKAN
PostPosted: October 5th, 2011, 3:37 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
AutoHotkey and URLGet 0.9.0.0

URLGet can report the progress status as 'Window Messages' when called in Four Parameter Mode.

    The parameters:
  • URL : A valid url wrapped in double quotes
  • NoFileCopy : 0 or 1. 1 will omit file copy. By default, downloaded file is copied from cache into the current folder ( A_WorkingDir )
  • CallerWnd : A handle to Window. ( hWnd of GUI )
  • CallerMsg : OnMessage Num, like 0x7FFF ( refer Application defined Message Identifier )



    URLGet will Send a combination of UGM ( URLGet Message ) and redirects of BINDSTATUS callback.
  • When wParam of OnMessage() is greater than 0x400 ( WM_USER ), it is UGM - otherwise, a redirect of BINDSTATUS
  • Use lParam to recieve Numbers. For Text, you will have to retrieve it from the Console Window Titlebar.
  • For each BINDSTATUS Message, OnMessage() should return 0 for URLGet to proceed, or may return 0x80004004 ( E_ABORT ) to cancel the download

Code:
; URLGet Messages:
UGM_INIT               := 0x401 ; lParam = hwnd of Console
UGM_CACHEFILE          := 0x404 ; use WinGetTitle on Console Window
UGM_TARGETFILE         := 0x405 ; use WinGetTitle on Console Window
UGM_DEINIT             := 0x406 ; lParam = result from URLDownloadToCacheFile
UGM_CANCEL             := 0x410 ; lParam = 1 for acknowledgement

; BINDSTATUS

BINDSTATUS_FINDINGRESOURCE                     :=  1
BINDSTATUS_CONNECTING                          :=  2
BINDSTATUS_REDIRECTING                         :=  3
BINDSTATUS_BEGINDOWNLOADDATA                   :=  4
BINDSTATUS_DOWNLOADINGDATA                     :=  5
BINDSTATUS_ENDDOWNLOADDATA                     :=  6
BINDSTATUS_BEGINDOWNLOADCOMPONENTS             :=  7
BINDSTATUS_INSTALLINGCOMPONENTS                :=  8
BINDSTATUS_ENDDOWNLOADCOMPONENTS               :=  9
BINDSTATUS_USINGCACHEDCOPY                     := 10
BINDSTATUS_SENDINGREQUEST                      := 11
BINDSTATUS_CLASSIDAVAILABLE                    := 12
BINDSTATUS_MIMETYPEAVAILABLE                   := 13
BINDSTATUS_CACHEFILENAMEAVAILABLE              := 14
BINDSTATUS_BEGINSYNCOPERATION                  := 15
BINDSTATUS_ENDSYNCOPERATION                    := 16
BINDSTATUS_BEGINUPLOADDATA                     := 17
BINDSTATUS_UPLOADINGDATA                       := 18
BINDSTATUS_ENDUPLOADINGDATA                    := 19
BINDSTATUS_PROTOCOLCLASSID                     := 20
BINDSTATUS_ENCODING                            := 21
BINDSTATUS_VERFIEDMIMETYPEAVAILABLE            := 22
BINDSTATUS_CLASSINSTALLLOCATION                := 23
BINDSTATUS_DECODING                            := 24
BINDSTATUS_LOADINGMIMEHANDLER                  := 25
BINDSTATUS_CONTENTDISPOSITIONATTACH            := 26
BINDSTATUS_FILTERREPORTMIMETYPE                := 27
BINDSTATUS_CLSIDCANINSTANTIATE                 := 28
BINDSTATUS_IUNKNOWNAVAILABLE                   := 29
BINDSTATUS_DIRECTBIND                          := 30
BINDSTATUS_RAWMIMETYPE                         := 31
BINDSTATUS_PROXYDETECTING                      := 32
BINDSTATUS_ACCEPTRANGES                        := 33
BINDSTATUS_COOKIE_SENT                         := 34
BINDSTATUS_COMPACT_POLICY_RECEIVED             := 35
BINDSTATUS_COOKIE_SUPPRESSED                   := 36
BINDSTATUS_COOKIE_STATE_UNKNOWN                := 37
BINDSTATUS_COOKIE_STATE_ACCEPT                 := 38
BINDSTATUS_COOKIE_STATE_REJECT                 := 39
BINDSTATUS_COOKIE_STATE_PROMPT                 := 40
BINDSTATUS_COOKIE_STATE_LEASH                  := 41
BINDSTATUS_COOKIE_STATE_DOWNGRADE              := 42
BINDSTATUS_POLICY_HREF                         := 43
BINDSTATUS_P3P_HEADER                          := 44
BINDSTATUS_SESSION_COOKIE_RECEIVED             := 45
BINDSTATUS_PERSISTENT_COOKIE_RECEIVED          := 46
BINDSTATUS_SESSION_COOKIES_ALLOWED             := 47
BINDSTATUS_CACHECONTROL                        := 48
BINDSTATUS_CONTENTDISPOSITIONFILENAME          := 49
BINDSTATUS_MIMETEXTPLAINMISMATCH               := 50
BINDSTATUS_PUBLISHERAVAILABLE                  := 51
BINDSTATUS_DISPLAYNAMEAVAILABLE                := 52
BINDSTATUS_SSLUX_NAVBLOCKED                    := 53
BINDSTATUS_SERVER_MIMETYPEAVAILABLE            := 54
BINDSTATUS_SNIFFED_CLASSIDAVAILABLE            := 55
BINDSTATUS_64BIT_PROGRESS                      := 56
BINDSTATUS_LAST = BINDSTATUS_64BIT_PROGRESS    := 57
BINDSTATUS_RESERVED_0                          := 58
BINDSTATUS_RESERVED_1                          := 59
BINDSTATUS_RESERVED_2                          := 60
BINDSTATUS_RESERVED_3                          := 61
BINDSTATUS_RESERVED_4                          := 62
BINDSTATUS_RESERVED_5                          := 63
BINDSTATUS_RESERVED_6                          := 64
BINDSTATUS_RESERVED_7                          := 65
BINDSTATUS_RESERVED_8                          := 66
BINDSTATUS_RESERVED_9                          := 67


Terminating download:
URLGet process can be terminated by ControlSending ^c to the Console Window, however:
When simultaneously running multiple instances of URLGet with FileCopy in effect ( NoFileCopy=0 ), every instance will compete for filename in current folder. Providing for such cases, every copy of URLGet increments* filename to find an available one and immediately creates a zero-byte-file to reserve it for FileCopy, before the download can even commence. Cancelling a download will delete the empty file, whereas, terminating URLGet will leave orphaned ones.

*Filename will be incremented in following pattern: filename.ext, filename.1.ext, filename.2.ext ... filename.100.ext


DEMONSTRATION:

    Image

Code:
SetWorkingDir %A_ScriptDir%
URL := "http://www.autohotkey.net/~Lexikos/AutoHotkey_L/AutoHotkey_L_Install.exe"

Gui +LastFound +AlwaysOnTop
hGUI := WinExist()

Gui, Font, S10, Tahoma
Gui, Add, Text,    w270 h20 0x200, Initiating..                 ; Static1
Gui, Add, Text,    x+3 w42 hp 0x201,                            ; Static2
Gui, Add, Progress,xm y+3 w315 h10 cF73D00 h8, 1                ; msctls_progress321
Gui, Add, Text,    xm y+3 w42 h25 y+5 c808080 0x200, Bytes:     ; Static3
Gui, Add, Text,    x+2 w80 h25 Right 0x200,                     ; Static4
Gui, Add, Text,    x+2 w42 h25 Center c808080 0x200             ; Static5
Gui, Add, Text,    x+2 w80 h25 Left 0x200                       ; Static6
Gui, Add, Button,  x+5 w60 hp gCancelDownload,       &Cancel    ; Button1
Gui, Show,, URLGet Demo

OnMessage( 0x7FFF, "URLGet_Callback" )
Run URLGet "%URL%" 0 %hGUI% 0x7FFF,, Hide
Return ;                                                 // end of auto-execute section //


GuiClose:
GuiEscape:
 SendMessage, 0x7FFF, 0x410, False,, ahk_id %hGUI% ; UGM_CANCEL before ExitApp
 hwndConsole := ErrorLevel
 While DllCall( "IsWindow", UInt,hwndConsole )     ; Instead of WinWaitClose to avoid
   Sleep 100                                       ; 'DetectHiddenWindows, On'
 ExitApp
Return


CancelDownload:
 SendMessage, 0x7FFF, 0x410, False,, ahk_id %hGUI%
 IfEqual, A_GuiControl, &Close, ExitApp
Return


WinGetTitle( hWnd ) { ; Instead of WinGetTitle command, to avoid 'DetectHiddenWindows, On'
 VarSetcapacity( Title, 4096 ), DllCall( "GetWindowText", UInt,hwnd, Str,Title, Int,4096 )
Return Title
}


UrlGet_Callback( wParam, lParam, Msg, hGUI ) {
 static hwndConsole, dataSz, RetVal=0

 If ( wParam = 0x401 ) {     ; UGM_INIT
   hwndConsole := lparam
 }

 If ( wParam = 0x405 ) {     ; UGM_TARGETFILE
   TargetFileName := WinGetTitle( hwndConsole )
   SplitPath, TargetFileName, TargetFileName
   GuiControl,,Static1, %TargetFilename%
 }

 If ( wParam = 0x406 ) {     ; UGM_DEINIT
   hwndConsole := dataSz := Retval := 0
   GuiControl, ,  Button1, &Close
 }

 If ( wParam = 0x410 && DllCall( "IsWindow", UInt,hwndConsole ) ) {   ; UGM_CANCEL
   If ( lParam = False ) {
       RetVal := 0x80004004 ; E_ABORT
       GuiControl,,Static1, Cancelling download..
       GuiControl, Disable, Button1
       Return hwndConsole
   } Else {
       RetVal := 0
       GuiControl,,Static1, Download cancelled!
       GuiControl, Enable,  Button1
       GuiControl, ,  Button1, &Close
   }
 }

 If ( wParam = 5 || wParam = 6 ) ; BINDSTATUS_DOWNLOADINGDATA
   {                             ; BINDSTATUS_ENDDOWNLOADDATA
   GuiControl,, Static4, %lParam%
   If ( dataSz > 0 ) {
     prc := Round( (lParam/dataSz)*100 )
     GuiControl,,Static2,           %prc%`%
     GuiControl,,msctls_progress321, %prc%
   }
 }

 If ( wParam = 1 )  {            ; BINDSTATUS_FINDINGRESOURCE
   Domain := WinGetTitle( hwndConsole )
   GuiControl,,Static1, Finding %Domain%
 }

 If ( wParam = 4 )  {            ; BINDSTATUS_BEGINDOWNLOADDATA
   If ( dataSz := lParam ) {     ; Download data size
   GuiControl,,Static5, out of
   GuiControl,,Static6, %dataSz%
   }
}
Return RetVal
}



Report this post
Top
 Profile  
Reply with quote  
TheGreatSwami Woo
 Post subject:
PostPosted: October 17th, 2011, 12:01 pm 
Offline

Joined: May 26th, 2011, 7:53 am
Posts: 237
Location: uk
just wondering could you use this to download a video from youtube if you gave it the correct url ?


Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: October 18th, 2011, 6:59 am 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
URLGet can download any URL.
In case of large files ( > 10 MiB ), it is better that URLGet does not handle the FileCopy.

Calling it with 2nd parameter as 1 ( eg. URLGet "<url>" 1 ) will download the file to cache and return the fullpath which can be used with FileMove command.


Report this post
Top
 Profile  
Reply with quote  
garry
 Post subject:
PostPosted: October 18th, 2011, 12:46 pm 
Offline

Joined: April 19th, 2005, 10:26 am
Posts: 2250
Location: switzerland
interesting
urldownloadtofile not worked in this case (can't see played songs) :
Code:
urldownloadtofile,http://109.169.26.78:9246/played.html,test.txt




script with urlget.exe :

Code:
;-------- http://www.autohotkey.com/forum/viewtopic.php?p=482932#482932 ---
SetWorkingDir %A_ScriptDir%

URL=http://109.169.26.78:9246/played.html

;-- used this because URLGet.exe makes extension with 3 characters ( html > htm ) 
    SplitPath,url, name, dir, ext, name_no_ext, drive
    stringmid,ext,ext,1,3
F1=%name_no_ext%.%ext%
ifexist,%f1%
  filedelete,%f1%
;----------------------------------------------------

Runwait, URLGet.exe "%URL%",,hide

aaa=
FileRead,aaa,%f1%
xx4=Current Song
StringReplace,aaa,aaa,</tr>,$, All
Loop,parse,aaa,$,
      {
      if A_loopfield contains %xx4%
            {
            aa:=RegExReplace( A_loopfield, "<.*?>" )
            stringreplace,aa,aa,Current Song,,all
            stringmid,aa,aa,9,200
            break
            }
      }
msgbox,%aa%
exitapp



script from IsNull , AHK_L
urldownloadToVAR /RAM

Code:
;-- MODIFIED =20110909
;-- tested with AHK_L    1.1.03.00
;-- IsNull
;-- http://de.autohotkey.com/forum/post-71924.html#71924
;-- UrlDownloadToVar
;========================================================

F1=http://109.169.26.78:9246/played.html
;F1:=http://130.166.82.14:8006/played.html

httpQuery(aaa,F1)
xx4=Current Song
StringReplace,aaa,aaa,</tr>,$, All
Loop,parse,aaa,$,
      {
      if A_loopfield contains %xx4%
            {
            aa:=RegExReplace( A_loopfield, "<.*?>" )
            stringreplace,aa,aa,Current Song,,all
            stringmid,aa,aa,9,200
            break
            }
      }
msgbox,%aa%
exitapp


httpQuery(byref Result, lpszUrl, POSTDATA="", HEADERS="")
{
   WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")
   WebRequest.Open("GET", lpszUrl)
   WebRequest.setRequestHeader("Content-Type", "application/x-www-form-urlencoded")
   WebRequest.Send(POSTDATA)
   Result := WebRequest.ResponseText
   WebRequest := ""
}

/*
<tr><td>11:10:18</td><td>Florante Aguilar and Lori Abucayan - Lahat Ng Araw (Silayan) <td><b>Current Song</b></td></tr>
*/


Last edited by garry on October 19th, 2011, 8:05 am, edited 1 time in total.

Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: October 18th, 2011, 4:07 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
Dear garry, thanks for testing :)

garry wrote:
script with urlget.exe :

Code:
;-------- http://www.autohotkey.com/forum/viewtopic.php?p=482932#482932 ---
SetWorkingDir %A_ScriptDir%

URL=http://109.169.26.78:9246/played.html

    SplitPath,url, name, dir, ext, name_no_ext, drive
    stringmid,ext,ext,1,3
F1=%name_no_ext%.%ext%
ifexist,%f1%
  filedelete,%f1%

Runwait,%comspec% /c URLGet.exe "%URL%",,hide

aaa=
FileRead,aaa,%f1%
xx4=Current Song
StringReplace,aaa,aaa,</tr>,$, All
Loop,parse,aaa,$,
      {
      if A_loopfield contains %xx4%
            {
            aa:=RegExReplace( A_loopfield, "<.*?>" )
            stringreplace,aa,aa,Current Song,,all
            stringmid,aa,aa,9,200
            break
            }
      }
msgbox,%aa%
exitapp


comspec will be required only if you want to redirect stdout to a file, in order to capture the downloaded filename, like in:
Runwait, %COMSPEC% /c URLGet.exe "%URL%" > filename.txt,, Hide
In your example script, we know for sure that downloaded file is played.htm
The following should suffice:

Code:
SetWorkingDir %A_ScriptDir%
FileDelete, played.htm
Runwait, URLGet.exe "http://109.169.26.78:9246/played.html",, Hide


PS:
I am able to reboot my 'beetel 220 BXI ADSL2 Modem' with a desktop shortcut to:
URLGet.exe "http://username:password@192.168.1.1/rebootinfo.cgi"


Report this post
Top
 Profile  
Reply with quote  
garry
 Post subject:
PostPosted: October 19th, 2011, 8:13 am 
Offline

Joined: April 19th, 2005, 10:26 am
Posts: 2250
Location: switzerland
thank you SKAN for comments, script, examples
I removed %comspec%
your suggestion filedelete is enough for this example, but I used often URLGet so I have the correct filename with splitpath / stringmid ( variable F1 )


Report this post
Top
 Profile  
Reply with quote  
UberNoob
 Post subject:
PostPosted: December 15th, 2011, 2:49 pm 
Hi SKAN,

I took URLGet for a spin and I like it, but there is one small change I would like to make and I can't figure out how to go about it.

Currently, when the download is complete, the 'Cancel' button changes to 'Close.' It would make more sense to me, personally, if the ''Cancel' button changed to 'Done.'

I found two instances of "GuiControl, , Button1, &Close" in your code. Changing the 2nd instance of &Close to &Done made no difference. Changing the 1st instance of &Close to &Done worked, but clicking the new 'Done' button failed to exit the application.

This is no big deal, SKAN, but if you could kindly point out to me where in your code I need to make changes, I'd appreciate it.

As you can see, I tested URLGet with a small file. The download completed so quickly that the progress bar never left the starting gate and no percentage appears above it. That's why having the ''Cancel' button change to 'Done' makes more sense to me.

Image

Win XP Sp2
ahk 1.0.48.5


Report this post
Top
  
Reply with quote  
SKAN
 Post subject:
PostPosted: December 15th, 2011, 5:52 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
Quote:
The download completed so quickly that the progress bar never left the starting gate and no percentage appears above it.


Usually, we cannot retrieve filesize for webpages and hence progress will not be available.

UberNoob wrote:
point out to me where in your code I need to make changes


Replace all &Close with &Done
Add the following line

Code:
 If ( wParam = 0x406 ) {     ; UGM_DEINIT
   hwndConsole := dataSz := Retval := 0
   GuiControl, ,  Button1, &Done
 }


Report this post
Top
 Profile  
Reply with quote  
polyethene
 Post subject:
PostPosted: December 15th, 2011, 7:37 pm 
Offline
User avatar

Joined: August 11th, 2004, 1:47 am
Posts: 5347
Location: UK
How is this advantageous to curl or wget?

_________________
GitHub • Scripts • IronAHK • Contact by email not private message.


Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: December 15th, 2011, 9:48 pm 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
polyethene wrote:
How is this advantageous to curl or wget?


I do not see any advantage.
I wrote this to overcome a few limitations of URLDownloadToFile:

1) As good as a browser download. No 'User-agent' problems with certain sites that requires it.
2) Decodes 'Content Encoded' data ( like the torrent files from torcache )
3) Does not block or hang the calling script, and can be simply killed from taskbar.

As for anybody wanting to distribute it:
1) It is light weight
2) Public Domain


Report this post
Top
 Profile  
Reply with quote  
UberNoob
 Post subject:
PostPosted: December 15th, 2011, 10:10 pm 
SKAN wrote:
Replace all &Close with &Done
Add the following line

Code:
 If ( wParam = 0x406 ) {     ; UGM_DEINIT
   hwndConsole := dataSz := Retval := 0
   GuiControl, ,  Button1, &Done
 }



It worked great, SKAN. Thank you!


Report this post
Top
  
Reply with quote  
ahknby
 Post subject:
PostPosted: December 23rd, 2011, 5:51 pm 
Offline

Joined: August 31st, 2011, 1:13 am
Posts: 6
Very nice work SKAN!

Working perfect here!

btw, win7 x64 IE9

CUmps!


Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: December 24th, 2011, 8:02 am 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
ahknby wrote:
btw, win7 x64 IE9


Glad to know it. Thanks for the feedback :)


Report this post
Top
 Profile  
Reply with quote  
Guest
 Post subject:
PostPosted: March 1st, 2012, 12:19 pm 
Hi SKAN,

I'm interested in this tool.

Is it possible with this tool to do something similar to the IE menu item, "Save As Webpage, complete(*.htm, *.html)" so that the images and the css file is downloaded to a separate folder other than the source html? It enables the user to view the web page offline.


Report this post
Top
  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 2
  [ 26 posts ]  Go to page 1, 2  Next

Board index » General » Utilities

All times are UTC [ DST ]

Who is online

Users browsing this forum: No registered users and 13 guests

You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Group