AutoHotkey Community
It is currently May 27th, 2012, 4:05 am

View unanswered posts | View active topics


Board index » AutoHotkey » Scripts

All times are UTC [ DST ]



Post new topic Reply to topic  Page 1 of 16
  [ 228 posts ]  Go to page 1, 2, 3, 4, 5 ... 16  Next
  Print view | E-mail friend Previous topic | Next topic 
Author Message
[VxE]
PostPosted: June 14th, 2011, 6:19 am 
Offline
User avatar

Joined: October 7th, 2006, 8:45 am
Posts: 3330
Location: Simi Valley, CA
 
HTTPRequest - A function for submitting http(s) requests to the web and handling the responses.

Version: 2.48 - Tested compatible with AHK v1.0.48.05 and AHK v1.1.00.00 (Unicode); Presumed compatible with later versions, ANSI, and x64.

              Download HTTPRequest.ahk      or      Download HTTPRequest.zip

Overview: I wrote HTTPRequest specifically to service client-authenticated web APIs (like Amazon and eBay), which it does to my satisfaction. It can also upload data and download resources (e.g: HTML, images, etc...) from the internet, optionally saving them as a file on disk. The core of HTTPRequest is WinINet.dll, which comes with Internet Explorer.

The Basics: HTTPRequest has 4 parameters, the second and third are BYREF and used for both input and output. HTTPRequest returns either the number of bytes downloaded OR the length of the text downloaded, depending on how it treats the downloaded data.
Code:
; This is the order of HTTPRequest's parameters
BytesDownloaded := HTTPRequest( URL, In_POST__Out_Data, In_Out_HEADERS, Options )
  • 1). URL (required) - The URL of the internet resource you wish to access.
  • 2). In_POST__Out_Data (optional) - A variable. Before the function is called, this variable should hold any data you wish to upload. If you don't wish to upload data, make this variable blank (empty). A simple way to do this is to assign an empty string to it. After the function completes successfully, this variable holds the response data.
    HTTPRequest does not convert uploaded text to utf-8 automatically. If you use unicode AHK, see the "charset" option below.
  • 3). In_Out_HEADERS (recommended) - A variable. Prior to calling the function, put a (`n) newline delimited list of custom request headers into this variable (or make it blank, if you don't need to supply any extra headers). After the function completes, this variable holds the HTTP response headers and/or messages about any errors the function encountered. Do not ignore this parameter, it often contains very useful debugging info.
  • 4). Options (optional) - A newline delimited list of keywords that modify how HTTPRequest works.

Headers: are a collection of information about the client/server, the request/response, and about the data being transferred. Even if parameter 3 is empty, HTTPRequest still submits request headers. There are a few request headers you should take special note of:
  • Content-Length: 12345 - HTTPRequest will add the "Content-Length" header to the request headers if it detects data to upload. If HTTPRequest can't determine the content length automatically (e.g: you are uploading non-text data), you must provide the "Content-Length" header with the total number of bytes you wish to upload. NOTE for UNICODE AHK: if you are uploading text, the number of bytes in text is twice the length of the text.
  • Content-Type: application/x-www-form-urlencoded - If the "Content-Length" is greater than zero, HTTPRequest will add the "Content-Type" header unless you already provided it. HTTPRequest's default values for this header are "application/octet-stream" when uploading from a file and the file's extension is neither "txt" nor "xml"; "text/xml" when the string "<?xml" appears near the beginning of the data; or "application/x-www-form-urlencoded" in all other cases.
  • Content-MD5: - Some web APIs require this header because they need to verify the integrity of the uploaded data. HTTPRequest can fill in the value automatically if you leave it blank (i.e: supply the header name and colon without the value like "Content-MD5:"). Also, if this appears in the response headers, HTTPRequest will add a "Computed-MD5" pseudo-header for your convenience.
  • User-Agent: MyScriptName/1.0 (Language=AutoHotkey/1.0.48.05; Platform=WIN_XP) - HTTPRequest will add the "User-Agent" header to every request. By default, your script name, AHK version, and OS are used. If this is unacceptable to you, provide your own "User-Agent" header.

Options: modify how HTTPRequest behaves. Text in gray is only a syntax example.
  • AutoProxy - This option tells HTTPRequest to use Internet Exlporer's proxy configuration. If your Internet Explorer is not configured to use a proxy, this option has no effect.
  • Binary - Normally, after downloading text, HTTPRequest will try to convert its codepage to the script's codepage, which is essential if you're using unicode AHK. The "binary" option tells HTTPRequest NOT to do the conversion (i.e: HTTPRequest will treat the downloaded data as raw binary data).
  • Callback: MyFunctionName [, Third Parameter Value] - Tells HTTPRequest to call "MyFunctionName" at regular intervals during the data transfer. "MyFunctionName" must NOT require more than 3 parameters. When HTTPRequest calls the function, the first parameter's value ranges from -1 to +1, where values less than 0 indicate an upload in progress and values greater than 0 indicate a download. The second parameter's value is the number of bytes anticipated in the transfer (i.e: the value of the "Content-Length" header). The third parameter's value is whatever you put as "Third Parameter Value". The callback function may return the word "CANCEL" to tell HTTPRequest to cancel the transaction.
  • Charset: utf-8 - Tells HTTPRequest to convert the text in "In_POST__Out_Data" from the script's codepage to the character set specified. HTTPRequest contains a lookup table with the supported charset names, which was copied from http://msdn.microsoft.com/en-us/library/dd317756%28v=vs.85%29.aspx.
  • Codepage: 65001 - The more 'technical' counterpart to the "charset" option, where the number on the right indicates the desired codepage number.
  • +NO_COOKIES - Tells WinINet to NOT automatically handle cookies (by default, WinINet uses Internet Explorer's cookie pool). This is required if you want your script to manage cookies with HTTPRequest. If your request headers contain any "Cookie" headers, this option is applied automatically.
  • +NO_AUTO_REDIRECT - Internet flags may be added or removed from a partiular request by putting a plus (+) or minus (-) sign in front of the flag's name, respectively (with or without the preceding "INTERNET_FLAG_"). Some flags are added automatically, depending on whether the connection uses SSL.
  • +IGNORE_UNKNOWN_CA - Security flags may be added or removed from a partiular request by putting a plus (+) or minus (-) sign in front of the flag's name, respectively (use the full name with "SECURITY_FLAG_" in front when the flag name would otherwise be ambiguous).
  • Method: POST - Tells HTTPRequest to use the indicated verb for the request. Typically, HTTPRequest uses "POST" if it is uploading data or "GET" otherwise. Use this option for web API functions that require a different verb. Supported verbs are GET, HEAD, POST, PUT, DELETE, OPTIONS, TRACE.
  • Proxy: http://www.exampleproxy.com:1234 - Tells HTTPRequest to use a proxy for the request
  • ProxyBypass: http://www.autohotkey.com - Tells HTTPRequest NOT to use a proxy to access [www.autohotkey.com]. This has no effect unless HTTPRequest has been told to use a proxy via the "proxy" option.
  • Resume: C:\My Partially Downloaded File.jpg - Tells HTTPRequest to continue downloading the specified file. It is up to you to make sure that the URL points to the same resource as the one used to download the file in the first place. When the download succeeds, "In_POST__Out_Data" is made blank.
  • SaveAs: C:\Some File.htm - Tells HTTPRequest to save the data downloaded to your hard disk. This option simulates UrlDownloadToFile. When the download succeeds, "In_POST__Out_Data" is made blank. You may use FileRead to read the data into memory later.
  • Upload: C:\My Awesome Pic.png - Tells HTTPRequest to use "C:\My Awesome Pic.png" as the source of data to upload. Please consult your webservice's documentation for any other conditions for uploading data. When using this option, I highly recommend supplying the appropriate "Content-Type" header.

Finally, this is a work in progress and features may be added/removed. Old versions will not remain available and I don't keep a change log. If you have questions, comments, bug reports, or requests for features, post them in this thread.

Special thanks to all the guineapi... er, contributors who helped spot bugs and suggested new features. Extra special thanks to derRaphael, whose work on httpQuery inspired me to start this project.

_________________
Ternary (a ? b : c) guide     TSV Table Manipulation Library
Post code inside [code][/code] tags!


Last edited by [VxE] on May 25th, 2012, 8:41 pm, edited 19 times in total.

Report this post
Top
 Profile  
Reply with quote  
[VxE]
 Post subject:
PostPosted: June 14th, 2011, 6:19 am 
Offline
User avatar

Joined: October 7th, 2006, 8:45 am
Posts: 3330
Location: Simi Valley, CA
Example: binary POST data. Upload Images to Imgur.com

This script requires HTTPRequest (7-1-2011) and your Imgur Anonymous API key. Running this script will prompt you to select an image file, upload the file to Imgur.com, and return the URL of the image.
The response XML is saved to a file in the same folder as the chosen image, so you can delete the image if you wish.
Code:
; Select an image and upload it to Imgur.com via the anonymous API.
Anonymous_API_Key := "< Your Imgur Anonymous API Key>"
FileSelectFile, image_file, 2, ::{450d8fba-ad25-11d0-98a8-0800361b1103}
, Upload Image To Imgur, Image Files (*.jpg; *.jpeg; *.gif; *.png; *.bmp)
If !( ErrorLevel )
{
   image_url := Imgur_Upload( image_file, Anonymous_API_Key, xml )
   FileAppend, % XML_MakePretty( xml ), % info_file := RegexReplace( image_file, ".*\K\..*", "-Imgur Info.txt" )
   If ( image_url )
      MsgBox, 0, Upload Successful!, % ( Clipboard := image_url ) "`n`nHas been put on the clipboard."
   Else
      MsgBox, 0, Upload Failed!, Something went wrong. For details, please see %info_file%
}
Exitapp

Imgur_Upload( image_file, Anonymous_API_Key, byref output_XML="" ) { ; -----------------------------
; Uploads one image file to Imgur via the anonymous API and returns the URL to the image.
; To acquire an anonymous API key, please register at http://imgur.com/register/api_anon.
; This function was written by [VxE] and relies on the HTTPRequest function, also by [VxE].
; HTTPRequest can be found at http://www.autohotkey.com/forum/viewtopic.php?t=73040
   Static Imgur_Upload_Endpoint := "http://api.imgur.com/2/upload.xml"
   FileGetSize, size, % image_file
   FileRead, output_XML, % "*c " image_file
   If HTTPRequest( Imgur_Upload_Endpoint "?key=" Anonymous_API_Key, output_XML
      , Response_Headers := "Content-Type: application/octet-stream`nContent-Length: " size
      , "Callback: CustomProgress" )
   && ( pos := InStr( output_XML, "<original>" ) )
      Return SubStr( output_XML, pos + 10, Instr( output_XML, "</original>", 0, pos ) - pos - 10 )
   Else Return "" ; error: see response
} ; Imgur_Upload( image_path, Anonymous_API_Key, byref output_XML="" ) -----------------------------

CustomProgress( pct, total ) {
If ( pct = "" )
   Tooltip
Else If ( pct < 0 )
   Tooltip, % "Uploading " Round( 100 * ( pct + 1 ), 1 ) "%. "
   . Round( ( pct + 1 ) * total, 0 ) " of " total " bytes."
Else If ( 0 <= pct )
   Tooltip, % "Downloading " Round( 100 * pct, 1 ) "%. "
   . Round( pct * total, 0 ) " of " total " bytes."
}

XML_MakePretty( XML, Tab="`t" ) { ; ----------------------------------------------------------------
; Function by [VxE]. Adds newlines and tabs between XML tags to give human-friendly arrangement to
; an XML stream. 'Tab' contains the string to use as an indentation unit (it may be more readable to
; use 2 or 3 spaces instead of a full tab... so it's up to you!).
   oel := ErrorLevel, PrevCloseTag := 0, tabs := "", tablen := StrLen( tab )
   StringLen, pos, XML
   Loop, Parse, XML, <, % "`t`r`n "
      If ( A_Index = 1 )
         VarSetCapacity( XML, pos, 0 )
      Else
      {
         StringGetPos, pos, A_LoopField, >
         StringMid, b, A_LoopField, pos, 1
         StringLeft, a, A_LoopField, 1
         If !( OpenTag := a != "/" ) * ( CloseTag := a = "/" || a = "!" || a = "?" || b = "/" )
            StringTrimRight, tabs, tabs, tablen
         XML .= ( OpenTag || PrevCloseTag ? tabs : "" ) "<" A_LoopField
         If !( PrevCloseTag := CloseTag ) * OpenTag
            tabs := ( tabs = "" ? "`n" : tabs ) tab
      }
   Return XML, ErrorLevel := oel
} ; XML_MakePretty( XML, Tab="`t" ) ----------------------------------------------------------------

                                                                        Image

_________________
Ternary (a ? b : c) guide     TSV Table Manipulation Library
Post code inside [code][/code] tags!


Last edited by [VxE] on July 2nd, 2011, 4:09 am, edited 5 times in total.

Report this post
Top
 Profile  
Reply with quote  
sumon
 Post subject:
PostPosted: June 14th, 2011, 6:39 am 
Offline
User avatar

Joined: May 18th, 2010, 3:10 pm
Posts: 1179
Location: Sweden
Let me be the first to be impressed. It seems well structured and functioning, and I can't wait to try it out. I also like your way of handling input -> output, as for me I'll make it easy and name the variables "POST" in the case of a post.

I have two requests/questions:

1) An example of transferring raw binary data, since that's something that I know I will find useful, and probably others too. For example for [img] imgur's API (see "upload"), or later, to upload apps to Appifyer.com.

2) The main differences between this and httpQuery. Their main functionality is the same, if I am not majorly mistaken?

_________________
~sumon • Appifyer• AHK Nova halted • Recommended: AHK_L (Why?)


Report this post
Top
 Profile  
Reply with quote  
[VxE]
 Post subject:
PostPosted: June 14th, 2011, 7:53 am 
Offline
User avatar

Joined: October 7th, 2006, 8:45 am
Posts: 3330
Location: Simi Valley, CA
sumon wrote:
2) The main differences between this and httpQuery. Their main functionality is the same, if I am not majorly mistaken?

Indeed, though my function dares to differ in several ways. I don't want to enumerate the points (out of respect for derRaphael), but I wrote HTTPRequest because httpQuery did not satisfy some of my requirements.


The Imgur anonymous API looks very useful (and it looks like a good demo for OAuth). I'll look into it in the next few weeks.

_________________
Ternary (a ? b : c) guide     TSV Table Manipulation Library
Post code inside [code][/code] tags!


Report this post
Top
 Profile  
Reply with quote  
sumon
 Post subject:
PostPosted: June 14th, 2011, 9:52 am 
Offline
User avatar

Joined: May 18th, 2010, 3:10 pm
Posts: 1179
Location: Sweden
[VxE] wrote:
sumon wrote:
2) The main differences between this and httpQuery. Their main functionality is the same, if I am not majorly mistaken?

Indeed, though my function dares to differ in several ways. I don't want to enumerate the points (out of respect for derRaphael), but I wrote HTTPRequest because httpQuery did not satisfy some of my requirements.


The Imgur anonymous API looks very useful (and it looks like a good demo for OAuth). I'll look into it in the next few weeks.


Oh, indeed, but I was referring to the anonymous/open API (where you just need an API key and a binary image file - getting an API key is easy), in general a guide purely on how to post raw binary data, since it's (like you said) a bit more advanced.

On the topic of oAuth, I think you would be a hero if you managed to (using your slightly more "convenient" HTTPRequest function) provide an example for authenticating with oAuth, since that authentication is becoming more and more common. Read more about it, and Octal's start, here.

_________________
~sumon • Appifyer• AHK Nova halted • Recommended: AHK_L (Why?)


Report this post
Top
 Profile  
Reply with quote  
derRaphael
 Post subject:
PostPosted: June 14th, 2011, 6:48 pm 
Offline

Joined: November 23rd, 2007, 10:23 am
Posts: 841
Location: ~/.
nice one ;)

good to know that httpQuery has now a worthy successor.

when talking of OAuth, it might be worth an idea to have build in cookie support (ideally with keeping session cookie data while following request relocations). some services (OAuth2, OpenSocial) have tendencies to move tokens and auhtorization information into cookie parts - so it will ease using these services (like facebook)

8)
dR

_________________
Image
    All scripts, unless otherwise noted, are hereby released under CC-BY


Report this post
Top
 Profile  
Reply with quote  
guest3456
 Post subject:
PostPosted: June 15th, 2011, 4:40 am 
Offline

Joined: March 10th, 2011, 7:17 pm
Posts: 374
whats the difference between this (and httpquery) compared to InternetFileRead?
http://www.autohotkey.com/forum/viewtopic.php?t=45718


Report this post
Top
 Profile  
Reply with quote  
tank
 Post subject:
PostPosted: June 15th, 2011, 4:55 am 
Offline
User avatar

Joined: December 21st, 2007, 3:14 pm
Posts: 3826
Location: Louisville KY USA
support for AHK_l and unicode for one

_________________
No matter what your oppinion Please join this discussion
Formal request to Polyethene
Image


Report this post
Top
 Profile  
Reply with quote  
[VxE]
 Post subject:
PostPosted: June 15th, 2011, 10:54 am 
Offline
User avatar

Joined: October 7th, 2006, 8:45 am
Posts: 3330
Location: Simi Valley, CA
@guest3456: I hadn't looked at InternetFileRead before, but from a cursory inspection, it seems that the main difference is in the function premises. InternetFileRead is meant to download files from the internet (with special attention to large files) whereas HTTPRequest is meant to handle requests to, and responses from, web APIs. I think httpQuery's premise is closest to 'mimic a browser's ability to submit POST data and retrieve results from those transactions'.

I don't think it's constructive for me to enumerate which functions have the shiniest bells and whistles.... Every user should select the appropriate tool for their needs (or write their own).


@dR: Thank you for your blessing on this project, I truly appreciate it.

Cookie support was part of some of the earlier iterations of this project. I didn't include it in this release because i hadn't thought of a graceful and intuitive way to integrate the feature.

As it stands, cookies should appear in the response as 'Set-Cookie' headers, and the user can submit cookies using the 'Cookie' header (see RFC 2965). It's possible for the user to manage cookies if needed.

I took a similar approach to redirects and response codes other than 2xx, leaving it up to the user to decide what to do based on the response headers.

_________________
Ternary (a ? b : c) guide     TSV Table Manipulation Library
Post code inside [code][/code] tags!


Report this post
Top
 Profile  
Reply with quote  
SKAN
 Post subject:
PostPosted: June 19th, 2011, 6:07 am 
Offline
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8776
[VxE] wrote:
The Imgur anonymous API looks very useful (and it looks like a good demo for OAuth). I'll look into it in the next few weeks.


It will be very helpful [VxE].
I currently use http://imguruploader.codeplex.com/releases/view/40965 but it is .net and requires installation.


Report this post
Top
 Profile  
Reply with quote  
Uberi
 Post subject:
PostPosted: June 20th, 2011, 10:39 pm 
Offline
User avatar

Joined: August 23rd, 2010, 6:22 pm
Posts: 781
Location: Ontario, Canada
Nice work! I'm glad that there's finally a valid alternative to my HTTPQuery version, as I don't have to maintain it any more :D. Also, HTTPS support is awesome, couldn't get it to work with any other methods.

_________________
AutoHotkey.net | GitHub

My default license.


Report this post
Top
 Profile  
Reply with quote  
[VxE]
 Post subject:
PostPosted: June 30th, 2011, 5:13 am 
Offline
User avatar

Joined: October 7th, 2006, 8:45 am
Posts: 3330
Location: Simi Valley, CA
Thanks, SKAN and Uberi.

BUGFIX: 6-20-2011. If you have an older version, please re-download from the link in the OP. The bug was in the call to HttpOpenRequest with the verb parameter for 'POST' operations. Also, there were typos in some of the auto-headers.

I'll keep a running commentary on bugfixes as I find/fix them. The function date (version?) can be found on the first comment line within the function.

[Edit: 6-29-2011] Well, it turns out that Imgur's API has two different modes and the anonymous API does not require OAuth. So, instead of demonstrating OAuth, I posted a demo of uploading binary data via POST (the demo script is just below the top post).

TWEAK: 6-29-2011: I expanded the automatic recognition of text-based 'Content-Type' responses. This version is required for the Imgur upload example to work correctly. If you don't trust the auto text recognition, and you know your API responses will be some sort of text, then by all means use VarSetCapacity( output_var, -1 ) after calling HTTPRequest.

_________________
Ternary (a ? b : c) guide     TSV Table Manipulation Library
Post code inside [code][/code] tags!


Report this post
Top
 Profile  
Reply with quote  
sumon
 Post subject:
PostPosted: June 30th, 2011, 10:05 am 
Offline
User avatar

Joined: May 18th, 2010, 3:10 pm
Posts: 1179
Location: Sweden
Yeah, imgur has two modes, but oAuth is something that in general is a widely used method, so that's why I requested that. imgur is just a decent example of oAuth.

Anyway, thanks for sharing the imgur upload example. Basically you just supply the binary data as POST data I take it from the FileRead as I understand it. For some reason, I get this error message all the time:

Quote:
<?xml version="1.0" encoding="utf-8"?><error>
<message>Image format not supported, or image is corrupt.</message>
<request>/2/upload.xml</request>
<method>post</method>
<format>xml</format>
<parameters></parameters>
</error>


AHK_L x32 Unicode, my API key is not the problem afaik.

_________________
~sumon • Appifyer• AHK Nova halted • Recommended: AHK_L (Why?)


Report this post
Top
 Profile  
Reply with quote  
[VxE]
 Post subject:
PostPosted: June 30th, 2011, 5:44 pm 
Offline
User avatar

Joined: October 7th, 2006, 8:45 am
Posts: 3330
Location: Simi Valley, CA
sumon wrote:
Yeah, imgur has two modes, but oAuth is something that in general is a widely used method, so that's why I requested that. imgur is just a decent example of oAuth.

Lol, I'm still working on OAuth.. just the different Imgur modes had me a bit confused at first. It will take a little while longer though. (OAuth is for accessing an account through the Imgur API... but I had started coding for the anonymous mode...).

Anyways, that error happens because I didn't test the Imgur example with AHK-L and forgot to add the '*c' option to FileRead. I have edited the example (after testing it with AHK-L U 32b) and it should work now.

_________________
Ternary (a ? b : c) guide     TSV Table Manipulation Library
Post code inside [code][/code] tags!


Report this post
Top
 Profile  
Reply with quote  
sumon
 Post subject:
PostPosted: June 30th, 2011, 5:50 pm 
Offline
User avatar

Joined: May 18th, 2010, 3:10 pm
Posts: 1179
Location: Sweden
Oh you are, nice to hear. Want contact with another who is? Contact octalmage. Sorry about the confusion!

Also, thanks alot for the bug fix, will be great. Gonna improve Zizorz alot soon...

_________________
~sumon • Appifyer• AHK Nova halted • Recommended: AHK_L (Why?)


Report this post
Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 16
  [ 228 posts ]  Go to page 1, 2, 3, 4, 5 ... 16  Next

Board index » AutoHotkey » Scripts

All times are UTC [ DST ]

Who is online

Users browsing this forum: No registered users and 19 guests

You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Group