Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

URLGet - Internet Explorer based Downloader


  • Please log in to reply
43 replies to this topic
SKAN
  • Administrators
  • 9105 posts
  • Last active:
  • Joined: 26 Dec 2005
Download URLGet.exe 64 KiB
See virusTotal results: <!-- m -->http://goo.gl/0CPAU<!-- m -->

Features ( 0.9.0.0 ):
[*:1zt2us2i] IE based ( urlmon/URLDownloadToCacheFileW ). Tested fine for me in: XP SP3 IE6
[*:1zt2us2i] Small footprint 64 KiB, low processor usage ( but memory around 4 MiB per instance ).
[*:1zt2us2i] No need to specify target filename. Auto-Increments target filename.
[*:1zt2us2i] Automatically decodes content encoding.
[*:1zt2us2i] Progress in TitleBar.
[*:1zt2us2i] IPC via WM.
Credit: Many thanks to Sean for URLDownloadToCacheFile

Author: SKAN ( Suresh Kumar A N ).
License: Public Domain
FileType: 32bit Console
FileVersion: 0.9.00 ( MD5: 6a529f13ca4261cab359a0333b9572dd )

Usage Example:

URLGet "http://google.co.in/"
will download into IE cache and copy the file as google.co.htm into your current folder ( A_WorkingDir ) and returns the fullpath of the copied file.

URLGet "http://google.co.in/" 1
will omit the FileCopy and instead, will return the fullpath to the cache file.

________________________________________________________________________________________

Source code

This utility was written and compiled in Gentee: Open Source Free Programming Language

Source code released: <!-- m -->http://gentee.com/ph...hp?p=9822#p9822<!-- m -->
View the code: <!-- m -->http://dl.dropbox.co... ... 0/URLGet.g<!-- m -->

SKAN
  • Administrators
  • 9105 posts
  • Last active:
  • Joined: 26 Dec 2005
AutoHotkey and URLGet 0.9.0.0

URLGet can report the progress status as 'Window Messages' when called in Four Parameter Mode.

The parameters:
[*:3bjenlk3]URL : A valid url wrapped in double quotes
[*:3bjenlk3]NoFileCopy : 0 or 1. 1 will omit file copy. By default, downloaded file is copied from cache into the current folder ( A_WorkingDir )
[*:3bjenlk3]CallerWnd : A handle to Window. ( hWnd of GUI )
[*:3bjenlk3]CallerMsg : OnMessage Num, like 0x7FFF ( refer Application defined Message Identifier )



URLGet will Send a combination of UGM ( URLGet Message ) and redirects of BINDSTATUS callback.
[*:3bjenlk3]When wParam of OnMessage() is greater than 0x400 ( WM_USER ), it is UGM - otherwise, a redirect of BINDSTATUS
[*:3bjenlk3]Use lParam to recieve Numbers. For Text, you will have to retrieve it from the Console Window Titlebar.
[*:3bjenlk3]For each BINDSTATUS Message, OnMessage() should return 0 for URLGet to proceed, or may return 0x80004004 ( E_ABORT ) to cancel the download
[color=darkred]; URLGet Messages:[/color]
UGM_INIT               := 0x401 ; lParam = hwnd of Console
UGM_CACHEFILE          := 0x404 ; use WinGetTitle on Console Window
UGM_TARGETFILE         := 0x405 ; use WinGetTitle on Console Window
UGM_DEINIT             := 0x406 ; lParam = result from URLDownloadToCacheFile
UGM_CANCEL             := 0x410 ; lParam = 1 for acknowledgement

[color=darkred]; BINDSTATUS[/color]

[color=indigo]BINDSTATUS_FINDINGRESOURCE                     :=  1
BINDSTATUS_CONNECTING                          :=  2
BINDSTATUS_REDIRECTING                         :=  3
BINDSTATUS_BEGINDOWNLOADDATA                   :=  4
BINDSTATUS_DOWNLOADINGDATA                     :=  5
BINDSTATUS_ENDDOWNLOADDATA                     :=  6
[/color]BINDSTATUS_BEGINDOWNLOADCOMPONENTS             :=  7
BINDSTATUS_INSTALLINGCOMPONENTS                :=  8
BINDSTATUS_ENDDOWNLOADCOMPONENTS               :=  9
BINDSTATUS_USINGCACHEDCOPY                     := 10
[color=indigo]BINDSTATUS_SENDINGREQUEST                      := 11[/color]
BINDSTATUS_CLASSIDAVAILABLE                    := 12
[color=indigo]BINDSTATUS_MIMETYPEAVAILABLE                   := 13
BINDSTATUS_CACHEFILENAMEAVAILABLE              := 14[/color]
BINDSTATUS_BEGINSYNCOPERATION                  := 15
BINDSTATUS_ENDSYNCOPERATION                    := 16
BINDSTATUS_BEGINUPLOADDATA                     := 17
BINDSTATUS_UPLOADINGDATA                       := 18
BINDSTATUS_ENDUPLOADINGDATA                    := 19
BINDSTATUS_PROTOCOLCLASSID                     := 20
BINDSTATUS_ENCODING                            := 21
BINDSTATUS_VERFIEDMIMETYPEAVAILABLE            := 22
BINDSTATUS_CLASSINSTALLLOCATION                := 23
[color=indigo]BINDSTATUS_DECODING                            := 24[/color]
BINDSTATUS_LOADINGMIMEHANDLER                  := 25
BINDSTATUS_CONTENTDISPOSITIONATTACH            := 26
BINDSTATUS_FILTERREPORTMIMETYPE                := 27
BINDSTATUS_CLSIDCANINSTANTIATE                 := 28
BINDSTATUS_IUNKNOWNAVAILABLE                   := 29
BINDSTATUS_DIRECTBIND                          := 30
BINDSTATUS_RAWMIMETYPE                         := 31
BINDSTATUS_PROXYDETECTING                      := 32
BINDSTATUS_ACCEPTRANGES                        := 33
BINDSTATUS_COOKIE_SENT                         := 34
BINDSTATUS_COMPACT_POLICY_RECEIVED             := 35
BINDSTATUS_COOKIE_SUPPRESSED                   := 36
BINDSTATUS_COOKIE_STATE_UNKNOWN                := 37
[color=indigo]BINDSTATUS_COOKIE_STATE_ACCEPT                 := 38
BINDSTATUS_COOKIE_STATE_REJECT                 := 39
[/color]BINDSTATUS_COOKIE_STATE_PROMPT                 := 40
BINDSTATUS_COOKIE_STATE_LEASH                  := 41
BINDSTATUS_COOKIE_STATE_DOWNGRADE              := 42
BINDSTATUS_POLICY_HREF                         := 43
BINDSTATUS_P3P_HEADER                          := 44
BINDSTATUS_SESSION_COOKIE_RECEIVED             := 45
BINDSTATUS_PERSISTENT_COOKIE_RECEIVED          := 46
BINDSTATUS_SESSION_COOKIES_ALLOWED             := 47
BINDSTATUS_CACHECONTROL                        := 48
BINDSTATUS_CONTENTDISPOSITIONFILENAME          := 49
BINDSTATUS_MIMETEXTPLAINMISMATCH               := 50
BINDSTATUS_PUBLISHERAVAILABLE                  := 51
BINDSTATUS_DISPLAYNAMEAVAILABLE                := 52
BINDSTATUS_SSLUX_NAVBLOCKED                    := 53
BINDSTATUS_SERVER_MIMETYPEAVAILABLE            := 54
BINDSTATUS_SNIFFED_CLASSIDAVAILABLE            := 55
BINDSTATUS_64BIT_PROGRESS                      := 56
BINDSTATUS_LAST = BINDSTATUS_64BIT_PROGRESS    := 57
BINDSTATUS_RESERVED_0                          := 58
BINDSTATUS_RESERVED_1                          := 59
BINDSTATUS_RESERVED_2                          := 60
BINDSTATUS_RESERVED_3                          := 61
BINDSTATUS_RESERVED_4                          := 62
BINDSTATUS_RESERVED_5                          := 63
BINDSTATUS_RESERVED_6                          := 64
BINDSTATUS_RESERVED_7                          := 65
BINDSTATUS_RESERVED_8                          := 66
BINDSTATUS_RESERVED_9                          := 67

Terminating download:
URLGet process can be terminated by ControlSending ^c to the Console Window, however:
When simultaneously running multiple instances of URLGet with FileCopy in effect ( NoFileCopy=0 ), every instance will compete for filename in current folder. Providing for such cases, every copy of URLGet increments* filename to find an available one and immediately creates a zero-byte-file to reserve it for FileCopy, before the download can even commence. Cancelling a download will delete the empty file, whereas, terminating URLGet will leave orphaned ones.

*Filename will be incremented in following pattern: filename.ext, filename.1.ext, filename.2.ext ... filename.100.ext


DEMONSTRATION:

Posted Image
SetWorkingDir %A_ScriptDir%
URL := "http://www.autohotkey.net/~Lexikos/AutoHotkey_L/AutoHotkey_L_Install.exe"

Gui +LastFound +AlwaysOnTop
hGUI := WinExist()

Gui, Font, S10, Tahoma
Gui, Add, Text,    w270 h20 0x200, Initiating..                 ; Static1
Gui, Add, Text,    x+3 w42 hp 0x201,                            ; Static2
Gui, Add, Progress,xm y+3 w315 h10 cF73D00 h8, 1                ; msctls_progress321
Gui, Add, Text,    xm y+3 w42 h25 y+5 c808080 0x200, Bytes:     ; Static3
Gui, Add, Text,    x+2 w80 h25 Right 0x200,                     ; Static4
Gui, Add, Text,    x+2 w42 h25 Center c808080 0x200             ; Static5
Gui, Add, Text,    x+2 w80 h25 Left 0x200                       ; Static6
Gui, Add, Button,  x+5 w60 hp gCancelDownload,       &Cancel    ; Button1
Gui, Show,, URLGet Demo

OnMessage( 0x7FFF, "URLGet_Callback" )
[color=#CC0000]Run URLGet "%URL%" 0 %hGUI% 0x7FFF,, Hide[/color]
Return ;                                                 // end of auto-execute section //


[color=darkred]GuiClose:
GuiEscape:
[/color] SendMessage, 0x7FFF, 0x410, False,, ahk_id %hGUI% ; UGM_CANCEL before ExitApp
 hwndConsole := ErrorLevel
 While DllCall( "IsWindow", UInt,hwndConsole )     ; Instead of WinWaitClose to avoid
   Sleep 100                                       ; 'DetectHiddenWindows, On'
 ExitApp
Return


[color=darkred]CancelDownload[/color]:
 SendMessage, 0x7FFF, 0x410, False,, ahk_id %hGUI%
 IfEqual, A_GuiControl, &Close, ExitApp
Return


[color=darkred]WinGetTitle[/color]( hWnd ) { ; Instead of WinGetTitle command, to avoid 'DetectHiddenWindows, On'
 VarSetcapacity( Title, 4096 ), DllCall( "GetWindowText", UInt,hwnd, Str,Title, Int,4096 )
Return Title
}


[color=darkred]UrlGet_Callback[/color]( wParam, lParam, Msg, hGUI ) {
 static hwndConsole, dataSz, RetVal=0

 If ( wParam = 0x401 ) {     ; UGM_INIT
   hwndConsole := lparam
 }

 If ( wParam = 0x405 ) {     ; UGM_TARGETFILE
   TargetFileName := WinGetTitle( hwndConsole )
   SplitPath, TargetFileName, TargetFileName
   GuiControl,,Static1, %TargetFilename%
 }

 If ( wParam = 0x406 ) {     ; UGM_DEINIT
   hwndConsole := dataSz := Retval := 0
   GuiControl, ,  Button1, &Close
 }

 If ( wParam = 0x410 && DllCall( "IsWindow", UInt,hwndConsole ) ) {   ; UGM_CANCEL
   If ( lParam = False ) {
       RetVal := 0x80004004 ; E_ABORT
       GuiControl,,Static1, Cancelling download..
       GuiControl, Disable, Button1
       Return hwndConsole
   } Else {
       RetVal := 0
       GuiControl,,Static1, Download cancelled!
       GuiControl, Enable,  Button1
       GuiControl, ,  Button1, &Close
   }
 }

 If ( wParam = 5 || wParam = 6 ) ; BINDSTATUS_DOWNLOADINGDATA
   {                             ; BINDSTATUS_ENDDOWNLOADDATA
   GuiControl,, Static4, %lParam%
   If ( dataSz > 0 ) {
     prc := Round( (lParam/dataSz)*100 )
     GuiControl,,Static2,           %prc%`%
     GuiControl,,msctls_progress321, %prc%
   }
 }

 If ( wParam = 1 )  {            ; BINDSTATUS_FINDINGRESOURCE
   Domain := WinGetTitle( hwndConsole )
   GuiControl,,Static1, Finding %Domain%
 }

 If ( wParam = 4 )  {            ; BINDSTATUS_BEGINDOWNLOADDATA
   If ( dataSz := lParam ) {     ; Download data size
   GuiControl,,Static5, out of
   GuiControl,,Static6, %dataSz%
   }
}
Return RetVal
}



TheGreatSwami Woo
  • Members
  • 237 posts
  • Last active: Jan 22 2012 03:31 PM
  • Joined: 26 May 2011
just wondering could you use this to download a video from youtube if you gave it the correct url ?

SKAN
  • Administrators
  • 9105 posts
  • Last active:
  • Joined: 26 Dec 2005
URLGet can download any URL.
In case of large files ( > 10 MiB ), it is better that URLGet does not handle the FileCopy.

Calling it with 2nd parameter as 1 ( eg. URLGet "<url>" 1 ) will download the file to cache and return the fullpath which can be used with FileMove command.

garry
  • Members
  • 2845 posts
  • Last active: Apr 21 2014 04:25 PM
  • Joined: 19 Apr 2005
interesting
urldownloadtofile not worked in this case (can't see played songs) :
urldownloadtofile,http://109.169.26.78:9246/played.html,test.txt



script with urlget.exe :

;-------- http://www.autohotkey.com/forum/viewtopic.php?p=482932#482932 ---
SetWorkingDir %A_ScriptDir%

URL=http://109.169.26.78:9246/played.html

;-- used this because URLGet.exe makes extension with 3 characters ( html > htm )  
    SplitPath,url, name, dir, ext, name_no_ext, drive
    stringmid,ext,ext,1,3
F1=%name_no_ext%.%ext%
ifexist,%f1%
  filedelete,%f1%
;----------------------------------------------------

Runwait, URLGet.exe "%URL%",,hide

aaa=
FileRead,aaa,%f1%
xx4=Current Song
StringReplace,aaa,aaa,</tr>,$, All
Loop,parse,aaa,$,
      {
      if A_loopfield contains %xx4%
            {
            aa:=RegExReplace( A_loopfield, "<.*?>" )
            stringreplace,aa,aa,Current Song,,all
            stringmid,aa,aa,9,200
            break
            }
      }
msgbox,%aa%
exitapp


script from IsNull , AHK_L
urldownloadToVAR /RAM

;-- MODIFIED =20110909
;-- tested with AHK_L    1.1.03.00
;-- IsNull
;-- http://de.autohotkey.com/forum/post-71924.html#71924
;-- UrlDownloadToVar
;========================================================

F1=http://109.169.26.78:9246/played.html
;F1:=http://130.166.82.14:8006/played.html

httpQuery(aaa,F1)
xx4=Current Song
StringReplace,aaa,aaa,</tr>,$, All
Loop,parse,aaa,$,
      {
      if A_loopfield contains %xx4%
            {
            aa:=RegExReplace( A_loopfield, "<.*?>" )
            stringreplace,aa,aa,Current Song,,all
            stringmid,aa,aa,9,200
            break
            }
      }
msgbox,%aa%
exitapp


httpQuery(byref Result, lpszUrl, POSTDATA="", HEADERS="")
{
   WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")
   WebRequest.Open("GET", lpszUrl)
   WebRequest.setRequestHeader("Content-Type", "application/x-www-form-urlencoded")
   WebRequest.Send(POSTDATA)
   Result := WebRequest.ResponseText
   WebRequest := ""
}

/*
<tr><td>11:10:18</td><td>Florante Aguilar and Lori Abucayan - Lahat Ng Araw (Silayan) <td><b>Current Song</b></td></tr>
*/


SKAN
  • Administrators
  • 9105 posts
  • Last active:
  • Joined: 26 Dec 2005
Dear garry, thanks for testing :)

script with urlget.exe :

;-------- http://www.autohotkey.com/forum/viewtopic.php?p=482932#482932 ---
SetWorkingDir %A_ScriptDir%

URL=http://109.169.26.78:9246/played.html

    SplitPath,url, name, dir, ext, name_no_ext, drive
    stringmid,ext,ext,1,3
F1=%name_no_ext%.%ext%
ifexist,%f1%
  filedelete,%f1%

Runwait,%comspec% /c URLGet.exe "%URL%",,hide

aaa=
FileRead,aaa,%f1%
xx4=Current Song
StringReplace,aaa,aaa,</tr>,$, All
Loop,parse,aaa,$,
      {
      if A_loopfield contains %xx4%
            {
            aa:=RegExReplace( A_loopfield, "<.*?>" )
            stringreplace,aa,aa,Current Song,,all
            stringmid,aa,aa,9,200
            break
            }
      }
msgbox,%aa%
exitapp


comspec will be required only if you want to redirect stdout to a file, in order to capture the downloaded filename, like in:
Runwait, %COMSPEC% /c URLGet.exe "%URL%" > filename.txt,, Hide
In your example script, we know for sure that downloaded file is played.htm
The following should suffice:

SetWorkingDir %A_ScriptDir%
FileDelete, played.htm
Runwait, URLGet.exe "http://109.169.26.78:9246/played.htm[color=darkred]l[/color]",, Hide

PS:
I am able to reboot my 'beetel 220 BXI ADSL2 Modem' with a desktop shortcut to:
URLGet.exe "http://username:password@192.168.1.1/rebootinfo.cgi"

garry
  • Members
  • 2845 posts
  • Last active: Apr 21 2014 04:25 PM
  • Joined: 19 Apr 2005
thank you SKAN for comments, script, examples
I removed %comspec%
your suggestion filedelete is enough for this example, but I used often URLGet so I have the correct filename with splitpath / stringmid ( variable F1 )

UberNoob
  • Guests
  • Last active:
  • Joined: --
Hi SKAN,

I took URLGet for a spin and I like it, but there is one small change I would like to make and I can't figure out how to go about it.

Currently, when the download is complete, the 'Cancel' button changes to 'Close.' It would make more sense to me, personally, if the ''Cancel' button changed to 'Done.'

I found two instances of "GuiControl, , Button1, &Close" in your code. Changing the 2nd instance of &Close to &Done made no difference. Changing the 1st instance of &Close to &Done worked, but clicking the new 'Done' button failed to exit the application.

This is no big deal, SKAN, but if you could kindly point out to me where in your code I need to make changes, I'd appreciate it.

As you can see, I tested URLGet with a small file. The download completed so quickly that the progress bar never left the starting gate and no percentage appears above it. That's why having the ''Cancel' button change to 'Done' makes more sense to me.

Posted Image

Win XP Sp2
ahk 1.0.48.5

SKAN
  • Administrators
  • 9105 posts
  • Last active:
  • Joined: 26 Dec 2005

The download completed so quickly that the progress bar never left the starting gate and no percentage appears above it.


Usually, we cannot retrieve filesize for webpages and hence progress will not be available.

point out to me where in your code I need to make changes


Replace all &Close with &Done
Add the following line

If ( wParam = 0x406 ) {     ; UGM_DEINIT
   hwndConsole := dataSz := Retval := 0
[color=red]   GuiControl, ,  Button1, &Done[/color]
 }


polyethene
  • Administrators
  • 5511 posts
  • Last active: Today, 11:40 AM
  • Joined: 26 Oct 2012
How is this advantageous to curl or wget?

SKAN
  • Administrators
  • 9105 posts
  • Last active:
  • Joined: 26 Dec 2005

How is this advantageous to curl or wget?


I do not see any advantage.
I wrote this to overcome a few limitations of URLDownloadToFile:

1) As good as a browser download. No 'User-agent' problems with certain sites that requires it.
2) Decodes 'Content Encoded' data ( like the torrent files from torcache )
3) Does not block or hang the calling script, and can be simply killed from taskbar.

As for anybody wanting to distribute it:
1) It is light weight
2) Public Domain

UberNoob
  • Guests
  • Last active:
  • Joined: --

Replace all &Close with &Done
Add the following line

If ( wParam = 0x406 ) {     ; UGM_DEINIT
   hwndConsole := dataSz := Retval := 0
[color=red]   GuiControl, ,  Button1, &Done[/color]
 }



It worked great, SKAN. Thank you!

ahknby
  • Members
  • 6 posts
  • Last active: Dec 22 2011 12:09 AM
  • Joined: 31 Aug 2011
Very nice work SKAN!

Working perfect here!

btw, win7 x64 IE9

CUmps!

SKAN
  • Administrators
  • 9105 posts
  • Last active:
  • Joined: 26 Dec 2005

btw, win7 x64 IE9


Glad to know it. Thanks for the feedback :)

  • Guests
  • Last active:
  • Joined: --
Hi SKAN,

I'm interested in this tool.

Is it possible with this tool to do something similar to the IE menu item, "Save As Webpage, complete(*.htm, *.html)" so that the images and the css file is downloaded to a separate folder other than the source html? It enables the user to view the web page offline.