AutoHotkey Homepage AutoHotkey Community
Let's help each other out
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

[function] httpQuery GET and POST requests - update 0.3.3
Goto page 1, 2, 3  Next
 
Post new topic   Reply to topic    AutoHotkey Community Forum Index -> Scripts & Functions
View previous topic :: View next topic  

Should a more generic function be released which will include the already available web functions, such as header queries, uri encoding, base encoding, etc ?
yes, i'd like to have one function to get all neccessary http functionalities in one instead of collecting each for my own
93%
 93%  [ 14 ]
no, i prefer collecting the functions i need
0%
 0%  [ 0 ]
neither nor ... explained in post
6%
 6%  [ 1 ]
Total Votes : 15

Author Message
DerRaphael



Joined: 23 Nov 2007
Posts: 445
Location: Heidelberg, Germany

PostPosted: Tue Jul 08, 2008 11:58 am    Post subject: [function] httpQuery GET and POST requests - update 0.3.3 Reply with quote

Introduction
    hi all
    This is a small function for handling POST and GET queries natively in AHK so external utilities to handle it are not needed any more. It can be used where no additional utilities, such as cURL or wget are wanted.
    The latest version handles following features:

    • ports in URL
    • Username:Password@domain style URLs (see this answer to avoid common problems)
    • SSL aka https
    • HeaderInformations / Dumping / Storing
    • DownloadSizes / CurrentSize (for download progressbars)
    • Flags for InternetConnection handling (auto follow feature etc)
    • Referrer for Queries
    • Accepttypes for Queries
    • Proxy handling
    • timeout handling
    • custom UserAgent


Usage

    Usage is pretty simple:
    You need to define a Variable which will receive the returned databuffer.
    When only Strings are expected as returnvalues, autohotkey needs to learn the new size, so a VarSetCapacity(buffer,-1) is needed. This has been modified to support binary downloads aswell.
    httpQUERY(buffer:="",URL) will result the length of the standard HTML which is usually retrieved. The html itself is stored in the variable buffer. The query method is GET as the length of Postparams is zero.
    httpQUERY(buffer:="",URL,POSTDATA) will work as the GET above just with POSTDATA being transmitted. The query method is POST as the length of Postparams is non zero.
    The URL now supports following scheme:
    http://username:password@server.tld:80/path/to/target.ext?queryops#fragment

    since httpQuery has been updated to use InternetCrackURL from winINet, all essential parts will be recognized. so there is no need to set up any additional parameters. Attention: When u need to authetificate in the Website the username / password attempt will not work. u have to submit those parameters via POST or GET method.

    Additional Parameters:
    To see a dump of the received httpHeaders, there is buildIn support for a global Variable named httpQueryOps. It may consist of one or more Verbs. For now "showHeader", "storeHeader", and "updateSize" verbs are supported. If You use storeHeader the complete Header will be saved in a variable named HttpQueryHeader which will be made global at runtime. The verb updateSize will make two variables globally Available: httpQueryFullSize and httpQueryCurrentSize. An usage example to show a download status indicator is included

    Following Variables are evaluated globally:
    httpAgent: can hold an individual UserAgent Code. The default is AutoHotkeyScript
    httpProxy: the name of a proxy server to use. default = 0
    httpProxyByPass: list of domains which will not be used with the proxy. default = 0
    httpQueryReferer: an URL which is recognized by QueriedServer, as the location where the Request was generated mind though its "Referrer" the name of the variable is httpQueryReferer [sic]
    httpQueryAcceptType: this variable lets u specify your accepted stream formats for the results of the query
    httpQueryDwFlags: if in need for any special flags for the current connection this is the variable to set (example V shows an useCase for this feat)

    I think these are pretty self explaining.

Examples

    Example I: Showing the raw HTML result of a search for httpQuery in AutoHotkey's forum in an edit control

Code:
; exmpl.searchAHKforum.httpQuery.ahk
; Searches the forum for a given Phrase: in this case httpQuery
#noenv
html     := ""
URL      := "http://www.autohotkey.com/forum/search.php?mode=results"
POSTData := "search_keywords=httpQuery&search_terms=all&search_forum=-1&"
          . "search_time=0&search_fields=all&show_results=topics&return_chars=500"
          . "&sort_by=0&sort_dir=DESC"

length := httpQuery(html,URL,POSTdata)
varSetCapacity(html,-1)

Gui,Add,Edit,w600 +Wrap r25,% html
Gui,Show
Return

GuiClose:
GuiEscape:
   ExitApp
   
#include httpQuery-0-3-3.ahk

    Remarks: To make a function work with POST parameters, it is essential to know which parameters are actually sent. These can be obtained by studying the HTML source of the target URL or by using analyzing tools such as the HttpFox Addon for Firefox. You might also want to use the following example to learn the nature of the form in a HTML document


    Example II: Showing a dump of a HTML form to help making valid POST parameters

Code:
; exmpl.formdump.httpquery.ahk
; Form Dumper v0.1b (w) 9th July 2008 by derRaphael
#NoEnv
InputBox
 ,URL                                                                  ; OutputVariable
 ,Enter URL to analyze                                                 ; Title of box
 ,Please enter the complete URL starting with http:// to be analysed   ; Descriptive text
 ,,,,,,,,http://www.autohotkey.com/forum/search.php                    ; default value

   html := ""
   htmlLength := httpQuery(html,URL)
   VarSetCapacity(html,-1)

; The Complete Form Node from given URL's HTML
   RegExMatch(html,"i)<form.+?</form>",formNode)
; Just the formtag for further analyzing
   RegExMatch(formNode,"i)<form[^>]+?>",formTag)
; The name of the form
   RegExMatch(formTag,"i)NAME=""?(?P<Name>.+?)""?>",form)
; The method used to process Data
   RegExMatch(formTag,"i)METHOD=""?(?P<Method>.+?)""?\s",form)
; The complete address used to send data to
   RegExMatch(formTag,"i)ACTION=""?(?P<Action>.+?)""?\s",form)
; just the url
   RegExMatch(formAction,"i)(?P<URL>[^\?]+)",formA_)
; any existing GET parameters
   RegExMatch(formAction,"i)\?(?P<GET>.*)",formA_)
   ; Fix &amp; to & as delimiter
   StringReplace,formA_GET,formA_GET,&amp`;,&,All

   startPosI := startPosS := startPosT := 0
   inpCount := selCount := txtCount := 0

   Loop,
   {
      If (startPosI:=RegExMatch(formNode,"i)<input[^>]+>",inputTag,startPosI+1)) {
         inpCount++
         formInput%InpCount% := inputTag
      } else {
         noMoreInput := 1
      }
      If (startPosS:=RegExMatch(formNode,"i)<select.*?</select>",selectNode,startPosS+1)) {
         selCount++
         formSelect%selCount% := selectNode
      } else {
         noMoreSelectOptions := 1
      }
      If (startPosT:=RegExMatch(formNode,"i)<textarea.*?</textarea>",textareaNode,startPosT+1)) {
         txtCount++
         formTextarea%txtCount% := textareaNode
      } else {
         noMoreTextArea := 1
      }
      if (NoMoreInput) && (NoMoreTextarea) && (NoMoreSelectOptions) {
         break
      }
   }
   Loop,% inpCount
      Inputs .= formInput%A_Index% "`n"
   StringReplace,inputs,inputs,<,%A_Tab%<,All
   Loop,% selCount
      Selects .= formSelect%A_Index% "`n"
   StringReplace,Selects,Selects,</option>,</option>`n,All
   StringReplace,Selects,Selects,<option,%A_Tab%<option,All
   Selects := RegExReplace(selects,"i)<select[^>]+>","$0`n")
      
   Loop,% txtCount
      txtAreas .= formTextArea%A_Index% "`n"
   StringReplace,txtAreas,txtAreas,</textarea>,</textarea>`n,All
   StringReplace,txtAreas,txtAreas,<textarea,%A_Tab%<textarea,All

; dump results to Gui
Gui,Add,Tab2, w800 h20,Analyzed Content|found Form|raw HTML
Gui,Tab,1
Gui,Add,Edit,w800 h600 yp+20 xp , % ""
      . "formTag:`n" formTag "`n`n"
        . "formMethod:`t" formMethod "`n"
        . "formAction:`t" formAction "`n"
        . "formActionURL:`t" formA_URL "`n"
        . "formActionGET:`t" formA_GET "`n`n"
      . "Total Inputs:`t" inpCount "`n"
      . inputs
      . "`nTotal Selects:`t" selCount "`n"
      . selects "`n"
      . "`nTotal TextAreas:`t" txtCount "`n"
      . txtAreas "`n"
Gui,Tab,2
Gui,Add,Edit,w800 h600 yp xp , % formNode
Gui,Tab,3
Gui,Add,Edit,w800 h600 yp xp , % html
Gui,Show,,Statistics for Form: "%formName%" (%URL%)
return

GuiClose:
GuiEscape:
   ExitApp
   
#include httpQuery-0-3-3.ahk

    Remarks: Using this function provides informations of which elements are used in a form which will be submitted. It neither catches modded JavaScript content nor more than one form. It has been tested with google.com, autohotkey.com's searchform and autohotkey.net's pastebin


    Example III: Downloading binary data and saving it to a local file.

Code:
; exmpl.downloadBinary.httpQuery.ahk
; This example downloads the latest AHK environment and stores
; the received binary data to a file.
#noenv
data     := ""
URL      := "http://www.autohotkey.net/programs/AutoHotkey104706.zip"
httpQueryOps := "updateSize"
SetTimer,showSize,10
length   := httpQuery(data,URL)
Tooltip
if (write_bin(data,"ahk.exe",length)!=1)
   MsgBox "There was an Error!"
else
   MsgBox AHK Source downloaded and saved as "ahk.zip"!
Return

showSize:
   Tooltip,% HttpQueryCurrentSize "/" HttpQueryFullSize
return

GuiClose:
GuiEscape:
   ExitApp

write_bin(byref bin,filename,size){
   h := DllCall("CreateFile","str",filename,"Uint",0x40000000
            ,"Uint",0,"UInt",0,"UInt",4,"Uint",0,"UInt",0)
   IfEqual h,-1, SetEnv, ErrorLevel, -1
   IfNotEqual ErrorLevel,0,ExitApp ; couldn't create the file
   r := DllCall("SetFilePointerEx","Uint",h,"Int64",0,"UInt *",p,"Int",0)
   IfEqual r,0, SetEnv, ErrorLevel, -3
   IfNotEqual ErrorLevel,0, {
      t = %ErrorLevel%              ; save ErrorLevel to be returned
      DllCall("CloseHandle", "Uint", h)
      ErrorLevel = %t%              ; return seek error
   }
   result := DllCall("WriteFile","UInt",h,"Str",bin,"UInt"
               ,size,"UInt *",Written,"UInt",0)
   h := DllCall("CloseHandle", "Uint", h)
   return, 1
}

#include httpQuery-0-3-3.ahk

    Remarks: This example downloads the current Autohotkey Source and saves it to local disk as ahk.zip in the same path as the script. While it does so, it shows the current progress of the download



    Example IV: Uploading an image to Imageshack using the official (free) API

Code:
; exmpl.imageshack.httpQuery.ahk
; This example uploads an image and constructs a multipart/form-data Type
; for fileuploading and returns the XML which is returned to show the stored Imagepath
   FileSelectFile,image
   FileGetSize,size,%image%
   SplitPath,image,OFN
   FileRead,img,%image%
   VarSetCapacity(placeholder,size,32)
   boundary := makeProperBoundary()
   post:="--" boundary "`ncontent-disposition: form-data; name=""MAX_FILE_SIZE""`n`n"
      . "1048576`n--" boundary "`ncontent-disposition: form-data; name=""xml""`n`nyes`n--"
      . boundary "`ncontent-disposition: form-data; name=""fileupload""; filename="""
      . ofn """`nContent-type: " MimeType(img) "`nContent-Transfer-Encoding: binary`n`n"
      . placeholder "`n--" boundary "--"
   headers:="Content-type: multipart/form-data, boundary=" boundary "`nContent-Length: " strlen(post)
   DllCall("RtlMoveMemory","uInt",(offset:=&post+strlen(post)-strlen(Boundary)-size-5)
         ,"uInt",&img,"uInt",size)
   size := httpQuery(result:="","http://www.imageshack.us/index.php",post,headers)
   VarSetCapacity(result,-1)
   Gui,Add,Edit,w800 h600, % result
   Gui,Show
return

GuiClose:
GuiEscape:
   ExitApp

makeProperBoundary(){
   Loop,26
      n .= chr(64+a_index)
   n .= "0123456789"
   Loop,% StrLen(A_Now) {
      Random,rnd,1,% StrLen(n)
      Random,UL,0,1
      b .= RegExReplace(SubStr(n,rnd,1),".$","$" (round(UL)? "U":"L") "0")
   }
   Return b
}

MimeType(ByRef Binary) {
   MimeTypes:="424d image/bmp|4749463 image/gif|ffd8ffe image/jpeg|89504e4 image/png|4657530"
          . " application/x-shockwave-flash|49492a0 image/tiff"
   @:="0123456789abcdef"
   Loop,8
      hex .= substr(@,(*(a:=&Binary-1+a_index)>>4)+1,1) substr(@,((*a)&15)+1,1)
   Loop,Parse,MimeTypes,|
      if ((substr(hex,1,strlen(n:=RegExReplace(A_Loopfield,"\s.*"))))=n)
         Mime := RegExReplace(A_LoopField,".*?\s")
   Return (Mime!="") ? Mime : "application/octet-stream"
}

#include httpQuery-0-3-3.ahk

    Remarks: Uploading binary data where an encodingtype "multipart/form-data" was needed, didnt work with classic POST usage. this example shows how to build the complete POST request in a different manner and to include raw binary data with no encoding such as base64 or ascii85



    Example V: Modifying dwFlags for special useCases where a 302 is the returncode such as with dynamic created content urls

Code:
#NoEnv
   Gui,add,edit,w800 h400 vPasteBin, this is a testtext!
   Gui,Add,Button,gPaste wp, Upload this text to http://pastebin.com
   Gui,Show
return

Paste:
   Gui,Submit,Nohide
   URL := "http://pastebin.com/pastebin.php"
   POSTDATA := "parent_pid=&format=text&code2=" uriEncode(PasteBin) "&"
           . "poster=&paste=Send&expiry=d&email=&"
   httpQueryOps := "showHeader storeHeader"
   httpQueryDwFlags := (INTERNET_FLAG_NO_AUTO_REDIRECT:= 0x00200000)
   length := httpQuery(HTML:="",URL,POSTDATA)
   VarSetCapacity(HTML,-1)
   gui,destroy
   Gui,Add,Edit,w800 h600,% HttpQueryHeader "`n`n" html
   Gui,Show
Return

GuiClose:
GuiEscape:
   ExitApp

uriEncode(str)
{ ; v 0.3 / (w) 24.06.2008 by derRaphael / zLib-Style release
   b_Format := A_FormatInteger
   data := ""
   SetFormat,Integer,H
   Loop,Parse,str
      if ((Asc(A_LoopField)>0x7f) || (Asc(A_LoopField)<0x30) || (asc(A_LoopField)=0x3d))
         data .= "%" . ((StrLen(c:=SubStr(ASC(A_LoopField),3))<2) ? "0" . c : c)
      Else
         data .= A_LoopField
   SetFormat,Integer,%b_format%
   return data
}

#include httpQUERY-0-3-3.ahk

    Remarks: Some WebServices such as PasteBin.com generate Dynamic Location Urls in order to show processed content. Using httpQuery in a default manner, results in a 404 Not found error when the script tries to access the location to fast. Specifying the NO_FOLLOW dwFlag solves this problem but needs a 2nd request to grab the fresh received Location Header.


httpQuery 0.3.3
Code:
; httpQuery-0-3-3.ahk
httpQuery(byref Result, lpszUrl, POSTDATA="", HEADERS="")
{   ; v0.3.3 (w) 27.07.2008 by Heresy & derRaphael / zLib-Style release   
   ; currently the verbs showHeader, storeHeader, and updateSize are supported in httpQueryOps
   ; in case u need a different UserAgent, Proxy, ProxyByPass, Referrer, and AcceptType just
   ; specify them as global variables - mind the varname for referrer is httpQueryReferer [sic].
   ; Also if any special dwFlags are needed such as INTERNET_FLAG_NO_AUTO_REDIRECT or cache
   ; handling this might be set using the httpQueryDwFlags variable as global
   global httpQueryOps, httpAgent, httpProxy, httpProxyByPass, httpQueryReferer, httpQueryAcceptType
       , httpQueryDwFlags
   ; Get any missing default Values
   defaultOps =
   (LTrim Join|
      httpAgent=AutoHotkeyScript|httpProxy=0|httpProxyByPass=0|INTERNET_FLAG_SECURE=0x00800000
      SECURITY_FLAG_IGNORE_UNKNOWN_CA=0x00000100|SECURITY_FLAG_IGNORE_CERT_CN_INVALID=0x00001000
      SECURITY_FLAG_IGNORE_CERT_DATE_INVALID=0x00002000|SECURITY_FLAG_IGNORE_CERT_WRONG_USAGE=0x00000200
      INTERNET_OPEN_TYPE_PROXY=3|INTERNET_OPEN_TYPE_DIRECT=1|INTERNET_SERVICE_HTTP=3
   )
   Loop,Parse,defaultOps,|
   {
      RegExMatch(A_LoopField,"(?P<Option>[^=]+)=(?P<Default>.*)",http)
      if StrLen(%httpOption%)=0
         %httpOption% := httpDefault
   }

   ; Load Library
   hModule := DllCall("LoadLibrary", "Str", "WinINet.Dll")

   ; SetUpStructures for URL_COMPONENTS / needed for InternetCrackURL
   ; http://msdn.microsoft.com/en-us/library/aa385420(VS.85).aspx
   offset_name_length:= "4-lpszScheme-255|16-lpszHostName-1024|28-lpszUserName-1024|"
                  . "36-lpszPassword-1024|44-lpszUrlPath-1024|52-lpszExtrainfo-1024"
   VarSetCapacity(URL_COMPONENTS,60,0)
   ; Struc Size               ; Scheme Size                  ; Max Port Number
   NumPut(60,URL_COMPONENTS,0), NumPut(255,URL_COMPONENTS,12), NumPut(0xffff,URL_COMPONENTS,24)
   
   Loop,Parse,offset_name_length,|
   {
      RegExMatch(A_LoopField,"(?P<Offset>\d+)-(?P<Name>[a-zA-Z]+)-(?P<Size>\d+)",iCU_)
      VarSetCapacity(%iCU_Name%,iCU_Size,0)
      NumPut(&%iCU_Name%,URL_COMPONENTS,iCU_Offset)
      NumPut(iCU_Size,URL_COMPONENTS,iCU_Offset+4)
   }

   ; Split the given URL; extract scheme, user, pass, authotity (host), port, path, and query (extrainfo)
   ; http://msdn.microsoft.com/en-us/library/aa384376(VS.85).aspx
   DllCall("WinINet\InternetCrackUrlA","Str",lpszUrl,"uInt",StrLen(lpszUrl),"uInt",0,"uInt",&URL_COMPONENTS)

   ; Update variables to retrieve results
   Loop,Parse,offset_name_length,|
   {
      RegExMatch(A_LoopField,"-(?P<Name>[a-zA-Z]+)-",iCU_)
      VarSetCapacity(%iCU_Name%,-1)
   }
   nPort:=NumGet(URL_COMPONENTS,24,"uInt")
   
   ; Import any set dwFlags
   dwFlags := httpQueryDwFlags
   ; For some reasons using a selfsigned https certificates doesnt work
   ; such as an own webmin service - even though every security is turned off
   ; https with valid certificates works when
   if (lpszScheme = "https")
      dwFlags |= (INTERNET_FLAG_SECURE|SECURITY_FLAG_IGNORE_CERT_CN_INVALID
               |SECURITY_FLAG_IGNORE_CERT_DATE_INVALID|SECURITY_FLAG_IGNORE_UNKNOWN_CA
               |SECURITY_FLAG_IGNORE_CERT_WRONG_USAGE)
               

   ; Check for Header and drop exception if unknown or invalid URL
   if (lpszScheme="unknown") {
      Result := "ERR: No Valid URL supplied."
      Return StrLen(Result)
   }

   ; Initialise httpQuery's use of the WinINet functions.
   ; http://msdn.microsoft.com/en-us/library/aa385096(VS.85).aspx
   hInternet := DllCall("WinINet\InternetOpenA"
                  ,"Str",httpAgent,"UInt"
                  ,(httpProxy != 0 ?  INTERNET_OPEN_TYPE_PROXY : INTERNET_OPEN_TYPE_DIRECT )
                  ,"Str",httpProxy,"Str",httpProxyBypass,"Uint",0)

   ; Open HTTP session for the given URL
   ; http://msdn.microsoft.com/en-us/library/aa384363(VS.85).aspx
   hConnect := DllCall("WinINet\InternetConnectA"
                  ,"uInt",hInternet,"Str",lpszHostname, "Int",nPort
                  ,"Str",lpszUserName, "Str",lpszPassword,"uInt",INTERNET_SERVICE_HTTP
                  ,"uInt",0,"uInt*",0)

   ; Do we POST? If so, check for header handling and set default
   if (Strlen(POSTDATA)>0) {
      HTTPVerb:="POST"
      if StrLen(Headers)=0
         Headers:="Content-Type: application/x-www-form-urlencoded"
   } else ; otherwise mode must be GET - no header defaults needed
      HTTPVerb:="GET"   

   ; Form the request with proper HTTP protocol version and create the request handle
   ; http://msdn.microsoft.com/en-us/library/aa384233(VS.85).aspx
   hRequest := DllCall("WinINet\HttpOpenRequestA"
                  ,"uInt",hConnect,"Str",HTTPVerb,"Str",lpszUrlPath . lpszExtrainfo
                  ,"Str",ProVer := "HTTP/1.1", "Str",httpQueryReferer,"Str",httpQueryAcceptTypes
                  ,"uInt",dwFlags,"uInt",Context:=0 )

   ; Send the specified request to the server
   ; http://msdn.microsoft.com/en-us/library/aa384247(VS.85).aspx
   sRequest := DllCall("WinINet\HttpSendRequestA"
                  , "uInt",hRequest,"Str",Headers, "uInt",Strlen(Headers)
                  , "Str",POSTData,"uInt",Strlen(POSTData))

   VarSetCapacity(header, 2048, 0)  ; max 2K header data for httpResponseHeader
   VarSetCapacity(header_len, 4, 0)
   
   ; Check for returned server response-header (works only _after_ request been sent)
   ; http://msdn.microsoft.com/en-us/library/aa384238.aspx
   Loop, 5
     if ((headerRequest:=DllCall("WinINet\HttpQueryInfoA","uint",hRequest
      ,"uint",21,"uint",&header,"uint",&header_len,"uint",0))=1)
      break

   If (headerRequest=1) {
      VarSetCapacity(res,headerLength:=NumGet(header_len),32)
      DllCall("RtlMoveMemory","uInt",&res,"uInt",&header,"uInt",headerLength)
      Loop,% headerLength
         if (*(&res-1+a_index)=0) ; Change binary zero to linefeed
            NumPut(Asc("`n"),res,a_index-1,"uChar")
      VarSetCapacity(res,-1)
   } else
      res := "timeout"

   ; Get 1st Line of Full Response
   Loop,Parse,res,`n,`r
   {
      RetValue := A_LoopField
      break
   }
   
   ; No Connection established - drop exception
   If (RetValue="timeout") {
      html := "Error: timeout"
      return -1
   }
   ; Strip protocol version from return value
   RetValue := RegExReplace(RetValue,"\Q" ProVer "\E\s+")
   
    ; List taken from http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
   HttpRetCodes := "100=Continue|101=Switching Protocols|102=Processing (WebDAV) (RFC 2518)|"
              . "200=OK|201=Created|202=Accepted|203=Non-Authoritative Information|204=No"
              . " Content|205=Reset Content|206=Partial Content|207=Multi-Status (WebDAV)"
              . "|300=Multiple Choices|301=Moved Permanently|302=Found|303=See Other|304="
              . "Not Modified|305=Use Proxy|306=Switch Proxy|307=Temporary Redirect|400=B"
              . "ad Request|401=Unauthorized|402=Payment Required|403=Forbidden|404=Not F"
              . "ound|405=Method Not Allowed|406=Not Acceptable|407=Proxy Authentication "
              . "Required|408=Request Timeout|409=Conflict|410=Gone|411=Length Required|4"
              . "12=Precondition Failed|413=Request Entity Too Large|414=Request-URI Too "
              . "Long|415=Unsupported Media Type|416=Requested Range Not Satisfiable|417="
              . "Expectation Failed|418=I'm a teapot (RFC 2324)|422=Unprocessable Entity "
              . "(WebDAV) (RFC 4918)|423=Locked (WebDAV) (RFC 4918)|424=Failed Dependency"
              . " (WebDAV) (RFC 4918)|425=Unordered Collection (RFC 3648)|426=Upgrade Req"
              . "uired (RFC 2817)|449=Retry With|500=Internal Server Error|501=Not Implem"
              . "ented|502=Bad Gateway|503=Service Unavailable|504=Gateway Timeout|505=HT"
              . "TP Version Not Supported|506=Variant Also Negotiates (RFC 2295)|507=Insu"
              . "fficient Storage (WebDAV) (RFC 4918)|509=Bandwidth Limit Exceeded|510=No"
              . "t Extended (RFC 2774)"
   
   ; Gather numeric response value
   RetValue := SubStr(RetValue,1,3)
   
   ; Parse through return codes and set according informations
   Loop,Parse,HttpRetCodes,|
   {
      HttpReturnCode := SubStr(A_LoopField,1,3)    ; Numeric return value see above
      HttpReturnMsg  := SubStr(A_LoopField,5)      ; link for additional information
      if (RetValue=HttpReturnCode) {
         RetMsg := HttpReturnMsg
         break
      }
   }

   ; Global HttpQueryOps handling
   if strlen(HTTPQueryOps)>0 {
      ; Show full Header response (usefull for debugging)
      if (instr(HTTPQueryOps,"showHeader"))
         MsgBox % res
      ; Save the full Header response in a global Variable
      if (instr(HTTPQueryOps,"storeHeader"))
         global HttpQueryHeader := res
      ; Check for size updates to export to a global Var
      if (instr(HTTPQueryOps,"updateSize")) {
         Loop,Parse,res,`n
            If RegExMatch(A_LoopField,"Content-Length:\s+?(?P<Size>\d+)",full) {
               global HttpQueryFullSize := fullSize
               break
            }
         if (fullSize+0=0)
            HttpQueryFullSize := "size unavailable"
      }
   }

   ; Check for valid codes and drop exception if suspicious
   if !(InStr("100 200 201 202 302",RetValue)) {
      Result := RetValue " " RetMsg
      return StrLen(Result)
   }

   VarSetCapacity(BytesRead,4,0)
   fsize := 0
   Loop            ; the receiver loop - rewritten in the need to enable
   {               ; support for larger file downloads
      bc := A_Index
      VarSetCapacity(buffer%bc%,1024,0) ; setup new chunk for this receive round
      ReadFile := DllCall("wininet\InternetReadFile"
                  ,"uInt",hRequest,"uInt",&buffer%bc%,"uInt",1024,"uInt",&BytesRead)
      ReadBytes := NumGet(BytesRead)    ; how many bytes were received?
      If ((ReadFile!=0)&&(!ReadBytes))  ; we have had no error yet and received no more bytes
         break                         ; we must be done! so lets break the receiver loop
      Else {
         fsize += ReadBytes            ; sum up all chunk sizes for correct return size
         sizeArray .= ReadBytes "|"
      }
      if (instr(HTTPQueryOps,"updateSize"))
         Global HttpQueryCurrentSize := fsize
   }
   sizeArray := SubStr(sizeArray,1,-1)   ; trim last PipeChar
   
   VarSetCapacity(result,fSize)          ; reconstruct the result from above generated chunkblocks
   Dest := &result                       ; to a our ByRef result variable
   Loop,Parse,SizeArray,|
      DllCall("RtlMoveMemory","uInt",Dest,"uInt",&buffer%A_Index%,"uInt",A_LoopField)
      , Dest += A_LoopField
      
   DllCall("WinINet\InternetCloseHandle", "uInt", hRequest)   ; close all opened
   DllCall("WinINet\InternetCloseHandle", "uInt", hInternet)
   DllCall("WinINet\InternetCloseHandle", "uInt", hConnect)
   DllCall("FreeLibrary", "UInt", hModule)                    ; unload the library
   return fSize                          ; return the size - strings need update via VarSetCapacity(res,-1)
}

Additional Informations / Needs to be done

  • For now the SSL functionality (aka https or secured communication) works but just when using trusted SSL servers. Using servers with self-signed certificates such as WebMin wont work and needs further research.

Version History

    v0.1 - Initial Version (Tue Jul 08, 2008)
    v0.1b - Fixed Typo, thx (Thu Jul 10, 2008)
    v0.3.1 - added Features such as InternetCrackURL, SSL, header Return Values, changed function call.
    v0.3.2 - added new httpQueryOps Verb to update full and current download Size
    v0.3.3 - added new global Vars to handle 302 answers properly by using custom dwFlags (thx skrommel for pointing this out).

    Remarks: Thanks to Heresy who helped a big deal in collecting all needed winINet calls to make httpPOST happen, and to ahklerner, who collected the neccessary SSL parameters in his version of this script.
    Also a big thanks to all who took their time to review this lil function Smile


greets
DerRaphael
_________________


Last edited by DerRaphael on Sun Jul 27, 2008 1:16 pm; edited 9 times in total
Back to top
View user's profile Send private message
Guest






PostPosted: Tue Jul 08, 2008 12:06 pm    Post subject: Reply with quote

Thanks god we have native http function now
Back to top
Joy2DWorld



Joined: 04 Dec 2006
Posts: 422
Location: Galil, Israel

PostPosted: Tue Jul 08, 2008 3:41 pm    Post subject: Reply with quote

clean!


the IF structure
Code:
if ((InStr(Server,"http://")) or (InStr(Server,"ftp://"))) {
      Protocoll := (RegExMatch(Server,"(^http://|^ftp://)",pr)) ? pr1 : "unknown"
   }

seems redundant...


(although have not tested time to do two INSTR vs. one regexmatch)


and would suggested
Code:
 if !instr(url,":")
   url = http://%url%
or such.


cf. http://www.autohotkey.com/forum/topic10393.html

and http://www.autohotkey.com/forum/viewtopic.php?t=10466&start=30 (which Sean, using COM, shows how to have url request work as 'callback' so you don't have to wait for page//script continues even if download does not!)
_________________
Joyce Jamce


Last edited by Joy2DWorld on Wed Jul 09, 2008 1:33 am; edited 1 time in total
Back to top
View user's profile Send private message
engunneer



Joined: 30 Aug 2005
Posts: 6772
Location: Pacific Northwest, US

PostPosted: Tue Jul 08, 2008 5:52 pm    Post subject: Reply with quote

the regex is needed due to odd urls such as: http://go.fark.com/cgi/fark/go.pl?i=3720009&l=http://ydr.inyork.com/ci_9797611
_________________
Unless otherwise noted, all code is untested.
Common Answers: 1.(Loops, Viruses, etc.) 2. Search 3.RTFM
Back to top
View user's profile Send private message Visit poster's website
Joy2DWorld



Joined: 04 Dec 2006
Posts: 422
Location: Galil, Israel

PostPosted: Tue Jul 08, 2008 6:51 pm    Post subject: Reply with quote

and the "IF" structure is needed for ?
_________________
Joyce Jamce
Back to top
View user's profile Send private message
neXt



Joined: 19 Mar 2007
Posts: 461

PostPosted: Wed Jul 09, 2008 12:29 am    Post subject: Reply with quote

so this function can be used in auto login scripts? Can someone post an example if yes please.

P.S. Nice work DerRaphael!
_________________
simplified csv - easy way to handle csv files.
Back to top
View user's profile Send private message
heresy



Joined: 11 Mar 2008
Posts: 269

PostPosted: Wed Jul 09, 2008 10:28 am    Post subject: Reply with quote

random xkcd comics getter Smile

LButton : Drag GUI
LButton Double Click : change image
RButton : Exitapp

Code:
;WinXP+ & needs HttpQuery()
GetPic()
OnMessage(0x203, "GetPic")
OnMessage(0x200, "ShowAlt")
Return

GuiContextMenu:
Exitapp

Move:
PostMessage,0xA1,2,,,A
Return

GetPic(){
    WinGetPos, X, Y,,, RandXKCD
    Gui, Destroy
    HttpQuery(raw, "http://dynamic.xkcd.com/comic/random/")
    VarSetCapacity(raw, -1)
    RegExMatch(raw,"Sm)<img src=""(?P<pic>http://imgs.xkcd.com/comics.+?)"" title=""(?P<title>.+?)"" alt=(?P<alt>"".+?)"" /><br/>", _)
    Passer(_Alt,_Title)
    SplitPath, _pic, PicFile
    picloc := A_Temp "\" PicFile
    If !FileExist(picloc)
        URLDownloadToFile, %_pic%, %picloc%
    Gui, Margin, 0,0
    Gui, +ToolWindow -Caption
    Gui, Add, Picture, gMove h-1, %picloc%
    Gui, Show, NA AutoSize x%x% y%y%, RandXKCD

}

ShowAlt(){
    Global a, b
    If A_Gui=1
        ToolTip, %a% "`n" %b%
    Else
        ToolTip
}

Passer(Byref _Alt, Byref _Title){
    Global a:=_Alt, b:=_Title
}

_________________
Easy WinAPI - Dive into Windows API World
Benchmark your AutoHotkey skills at PlayAHK.com


Last edited by heresy on Fri Jul 18, 2008 5:53 pm; edited 4 times in total
Back to top
View user's profile Send private message
DerRaphael



Joined: 23 Nov 2007
Posts: 445
Location: Heidelberg, Germany

PostPosted: Wed Jul 09, 2008 10:50 am    Post subject: Reply with quote

updated example section for a form dumper

derRaphael
Back to top
View user's profile Send private message
n-l-i-d
Guest





PostPosted: Wed Jul 09, 2008 12:02 pm    Post subject: Reply with quote

Nice! Smile

But isn't there a typo in your code?
Code:
httpQUERY(URL, POSTDATA="", AGENT="AutohotkeyScript", HTTP_POST=80)

Code:
, "uInt",hInternet,"Str",ServerName, "Int",HTTP_PORT
Back to top
heresy



Joined: 11 Mar 2008
Posts: 269

PostPosted: Wed Jul 09, 2008 4:21 pm    Post subject: Reply with quote

Getting first hit from google search.
looping the regex part will get the whole result

Code:
MsgBox % Google("AutoHotkey")

Google(Query){
    Fp=1
    StringReplace, Query, Query, ", `%22, All
    StringReplace, Query, Query,%A_Space%,+, All
    URL := "http://www.google.com/search?q=" . Query
    httpQUERY(raw, URL)
    VarSetCapacity(raw, -1)   
        StringReplace, Raw, Raw,<b>,, All
        StringReplace, Raw, Raw,</b>,, All
    Fp := RegExMatch(Raw,"Sm)<div class=g><a href=""(?P<Link>.+?)""", _, Fp)
    Fp := RegExMatch(Raw,"Sm)class=l>(?P<Title>.+?)</a>", _, Fp)
    Fp := RegExMatch(Raw,"Sm)<div class=std>(?P<description>.+?)<br><span", _, Fp)
    Return _Link "`n" _Title "`n" _Description
}

_________________
Easy WinAPI - Dive into Windows API World
Benchmark your AutoHotkey skills at PlayAHK.com


Last edited by heresy on Fri Jul 18, 2008 5:56 pm; edited 1 time in total
Back to top
View user's profile Send private message
tank



Joined: 21 Dec 2007
Posts: 616

PostPosted: Wed Jul 09, 2008 9:33 pm    Post subject: Reply with quote

Very interesting DerRaphael
very very interesting
_________________
Read this
Com
Automate IE7 with Tabs
Back to top
View user's profile Send private message
tank



Joined: 21 Dec 2007
Posts: 616

PostPosted: Wed Jul 09, 2008 9:43 pm    Post subject: Reply with quote

wow been toying a bit here might end up canning my own library this is impressive
_________________
Read this
Com
Automate IE7 with Tabs
Back to top
View user's profile Send private message
DerRaphael



Joined: 23 Nov 2007
Posts: 445
Location: Heidelberg, Germany

PostPosted: Thu Jul 10, 2008 3:25 am    Post subject: Reply with quote

fixed typo, thx n-l-i-d
_________________
Back to top
View user's profile Send private message
ahklerner



Joined: 26 Jun 2006
Posts: 1202
Location: USA

PostPosted: Thu Jul 10, 2008 5:46 pm    Post subject: Reply with quote

I wrapped the WinInet.dll functions to be StdLib compatible. It can be found here:
http://www.autohotkey.com/forum/viewtopic.php?t=33561

The WININET lib includes the functionality above as well as the FTP functions that were spread across the forum. This allows a single include that can be utilized for HTTP and FTP communication.
_________________
Back to top
View user's profile Send private message
bigsalgo911



Joined: 02 Jul 2008
Posts: 21

PostPosted: Sat Jul 12, 2008 2:37 am    Post subject: Reply with quote

So this works great...however. I am having a problem getting this to work with any variables in the POSTdata field.

Here is the code I'm using:

Code:
URL := "http://ecase.xxx.yyy.net/login.asp"
POSTdata := "logonid=%login%&password=%pass%"
httpQUERY(URL,POSTDATA)


the variables are defined earlier. I have tried many combinations of using and not using quotes and I cannot figure it out. It works perfectly when I enter my login information, but it can't seem to read the variable. What am I doing wrong?
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    AutoHotkey Community Forum Index -> Scripts & Functions All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Jump to:  
You can post new topics in this forum
You can reply to topics in this forum


Powered by phpBB © 2001, 2005 phpBB Group