 |
AutoHotkey Community
Let's help each other out
|
| View previous topic :: View next topic |
| Author |
Message |
wakewatcher
Joined: 15 Jul 2006
Posts: 200
|
 Posted: Tue Jun 02, 2009 12:31 am Post subject: DerRaphael/Ahklerner httpQuery/UrlGC and Amazon S3 Service |
 |
|
DerRaphael's httpQuery (http://www.autohotkey.com/forum/topic33506.html) has been a godsend and I've used it for a couple of applications. Now I've got a use which I think maybe stretching the envelope. I've got an application which uses Amazon's S3 on-line storage facility. Amazon has provided the capability for folks to upload via a posted web form. With my application a browser isn't used but instead an ahk script which processes files (videos) and then uploads them to S3. By using the Amazon post capability I do not have to share my Amazon secret keys with my users. The problem is I can't get it to work. And truthfully I'm messing with stuff I don't totally understand fully. (But having said that I've made some progress and at least have it to the point where perhaps I can reasonably and hopefully get some help.) I really think a lot of folks could use this capability if it can be made to work.
This effort is based on the amazon service as described here. (http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1434)
What happens is a page generates a properly constructed "policy document" and an encoded "signature value" which is used to authorize an upload to an S3 account within a certain time period. (The signature value is generated based on the policy document and the account owners private 'secret key') These two items populate hidden fields in the form. Then, in the browser usage the users selects a file and then clicks submit and the post is pointed to amazon which then validates the documents and accepts the file and stores it in the specified account 'bucket'. On success amazon returns with a redirected landing page. (In the example it is set to the same page.)
So this is how far that I've got. I built the test php page which works fine with a browser. That page has two entry points. One is the typical, first use, page where one is invited to pick a file to upload to my account. The second is when it is redirected to on success. In the later case I simply share the url with the last uploaded file so that it can be verified that it got there and that it can be retrieved. So far so good. Now for ahk (non browser application) I added a third entry point. That is when httpQuery hits the page with a POSTdata variable of 'policy' set. Then the page simply returns the policy document and signature value as a result. Then (the idea anyway) is that my ahk program uses those values to populate POSTdata to the query to be sent to the amazon url. Something at this point breaks. I don't know if its because I needed to do something different for the https use for the amazon page or something else. I'm assume that the redirect is not an issue as that would just come back as an html string that I would ignore (other than using it to verify success.)
As I mentioned I really think something like this could be really useful for storing data on S3 without the necessity of sharing private secret keys. If anyone is interested my page is available to play with. It is located here. (http://wakewatcher.redirect.hm/sandbox/SendVidS3.php)
You can upload test files up to 50k.
The web page code is:
| Code: |
<?PHP
// Example of how to send a file to the Amazon S3 service with PHP
// Inspired by: http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=139&externalID=1434&fromSearchPage=true
//
// Puts up a page which allows the user to select a file and sendit directly to S3, and calls this same page with the results whencompleted
// Change the following to correspond to your system:
$AWS_ACCESS_KEY = 'my public key';
$AWS_SECRET_KEY = 'its a secret thats the point';
$S3_BUCKET = 'autohotkey';
$S3_FOLDER = 'folder/'; // folder within bucket
$MAX_FILE_SIZE = 50 * 1024; // KB size limit
$SUCCESS_REDIRECT = ' http://wakewatcher.redirect.hm/sandbox/SendVidS3.php';
// setup transfer form
$expTime = time() + (1 * 60 * 60); // now plus one hour (1 hour; 60 mins; 60secs)
$expTimeStr = gmdate('Y-m-d\TH:i:s\Z', $expTime);
// create policy document
$policyDoc = '
{"expiration": "' . $expTimeStr . '",
"conditions": [
{"bucket": "' . $S3_BUCKET . '"},
["starts-with", "$key", "' . $S3_FOLDER . '"],
{"acl": "public-read"},
{"success_action_redirect": "' . $SUCCESS_REDIRECT . '"},
["starts-with", "$Content-Type", ""],
["starts-with", "$Content-Disposition", ""],
["content-length-range", 0, ' . $MAX_FILE_SIZE . ']
]
}
';
//echo "policyDoc: " . $policyDoc . '<BR/>';
// remove CRLFs from policy document
$policyDoc = implode(explode('\r', $policyDoc));
$policyDoc = implode(explode('\n', $policyDoc));
$policyDoc64 = base64_encode($policyDoc); // encode to base 64
// create policy document signature
$sigPolicyDoc = base64_encode(hash_hmac("sha1", $policyDoc64, $AWS_SECRET_KEY, TRUE/*raw_output*/));
if(isset($_POST['policy']))
die($policyDoc64 . " " . $sigPolicyDoc);
// create document header
echo '
<html>
<head>
<title>S3 POST Form</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
';
if(isset($_GET['bucket'])) {
$Bucket = $_GET['bucket'];
$Key = $_GET['key'];
echo 'URL (public read) : http://s3.amazonaws.com/' . $Bucket . '/' . $Key . ' or Click <a href="http://s3.amazonaws.com/' . $Bucket . '/' . $Key . '">HERE</a> <br>';
}
Echo "<br>policyDoc64:<br>$policyDoc64<br><br>sigPolicyDoc:<br>$sigPolicyDoc<br><br>";
// create file transfer form
echo '
<form action=" https://' . $S3_BUCKET . '.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
<input type="hidden" name="key" value="' . $S3_FOLDER . '${filename}">
<input type="hidden" name="AWSAccessKeyId" value="' . $AWS_ACCESS_KEY . '">
<input type="hidden" name="acl" value="public-read">
<input type="hidden" name="success_action_redirect" value="' . $SUCCESS_REDIRECT . '">
<input type="hidden" name="policy" value="' . $policyDoc64 . '">
<input type="hidden" name="signature" value="' . $sigPolicyDoc . '">
<input type="hidden" name="Content-Disposition" value="attachment; filename=${filename}">
<input type="hidden" name="Content-Type" id="Content-Type" value="video/mp4">
File to upload to S3 (size limit 50kb):
<input name="file" id="file" type="file">
<br/><br/>
<input type="submit" value="Upload File to S3">
</form>
';
// create document footer
echo '
</body>
</html>
';
?>
|
The ahk prototype script is:
| Code: |
loop:
FileSelectFile,filename,,Pick a File which is less than 50K
POSTdata := "policy=Y"
URL = http://www.wakewatcher.redirect.hm/sandbox/SendVidS3.php
length := httpQuery(result:="",URL,POSTdata)
varSetCapacity(result,-1)
stringgetpos,pos,result,%A_Space%
stringtrimright,policyDoc64,result,% length - pos
stringtrimleft,sigPolicyDoc,result,% Pos + 1
;msgbox policyDoc64:`n%policyDoc64%`n`nsigPolicyDoc:`n%sigPolicyDoc%
POSTdataString = key=folder/${filename}
POSTdataString = %POSTdataString%&AWSAccessKeyId=19SPCQ42SZ55Y0GM3JG2
POSTdataString = %POSTdataString%&success_action_redirect=http://www.wakewatcher.redirect.hm/sandbox/SendVidS3.php
POSTdataString = %POSTdataString%&policy=%policyDoc64%
POSTdataString = %POSTdataString%&signature=%sigPolicyDoc%
POSTdataString = %POSTdataString%&Content-Disposition=attachment`; filename=${filename}
POSTdataString = %POSTdataString%&Content-Type=Y
POSTdataString = %POSTdataString%&file=%filename%
URL = https://autohotkey.s3.amazonaws.com
length := httpQuery(result:="",URL,POSTdata)
varSetCapacity(result,-1)
msgbox Returned Result: %result%
goto loop |
One thing the original php script did was use some java script to determine the mime type. Of course this wouldn't work in the ahk implementation since there isn't a browser involved. I expected that I would have to determine it using ahk but as a test I just set it static to a character and it doesn't complain.
Anyway I'm hoping that I can use this somehow to auto-magically upload files to S3 using this post form method.
Last edited by wakewatcher on Mon Jun 08, 2009 8:02 am; edited 3 times in total |
|
| Back to top |
|
 |
wakewatcher
Joined: 15 Jul 2006
Posts: 200
|
| Posted: Tue Jun 02, 2009 7:37 am Post subject: |
|
|
Making some progress. Upon closer examination it looks like DerRaphael's code is for http and the variant that ahklerner made for https. So now using UrlGetContents() I'm making some progress. Here is my test script.
| Code: | loop:
FileSelectFile,filename,,Pick a File which is less than 50K
POSTdata := "policy=Y"
URL = http://www.wakewatcher.redirect.hm/sandbox/SendVidS3.php
;length := httpQuery(result:="",URL,POSTdata)
WININET_Init()
result := UrlGetContents(URL,"","",POSTdata)
;varSetCapacity(result,-1)
stringgetpos,pos,result,%A_Space%
stringtrimright,policyDoc64,result,% length - pos
stringtrimleft,sigPolicyDoc,result,% Pos + 1
;msgbox policyDoc64:`n%policyDoc64%`n`nsigPolicyDoc:`n%sigPolicyDoc%
POSTdataString = key=folder/${filename}
POSTdataString = %POSTdataString%&AWSAccessKeyId=19SPCQ42SZ55Y0GM3JG2
POSTdataString = %POSTdataString%&success_action_redirect=http://www.wakewatcher.redirect.hm/sandbox/SendVidS3.php
POSTdataString = %POSTdataString%&policy=%policyDoc64%
POSTdataString = %POSTdataString%&signature=%sigPolicyDoc%
POSTdataString = %POSTdataString%&Content-Disposition=attachment`; filename=${filename}
POSTdataString = %POSTdataString%&Content-Type=Y
POSTdataString = %POSTdataString%&file=%filename%
URL = https://autohotkey.s3.amazonaws.com
msgbox % UrlGetContents(URL,"","",POSTdataString)
WININET_UnInit()
goto Loop
Return
#include ahklerner stuff |
Now at least I get the page to return something. I get:
| Code: | <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PreconditionFailed</Code>
<Message>At least one of the pre-conditions you specified did not hold</Message>
<Condition>Bucket POST must be of the enclosure-type multipart/form-data</Condition>
<RequestId>1CCAD8504FC1AA1B</RequestId>
<HostId>y6LH5etxARspSASjNTGzPFjluNZIerRpoVwlFE8/euGoIoA2oWLH2twd0mSeuONb</HostId>
</Error> |
So I edited UrlGetContents:
| Code: | if sPostData
; sHTTPVerb:="POST", sHeaders:="Content-Type: application/x-www-form-urlencoded"
sHTTPVerb:="POST", sHeaders:="Content-Type: application/x-www-form-urlencoded`; Enclosure-Type: multipart/form-data" |
But it didn't make any difference. However at least I'm now getting to the site. I think I will also have wrong stuff on the parameter list but right now I'm trying to get past this error. Help anyone? -thx-
Last edited by wakewatcher on Sun Jun 07, 2009 2:42 am; edited 1 time in total |
|
| Back to top |
|
|
wakewatcher
Joined: 15 Jul 2006
Posts: 200
|
| Posted: Sat Jun 06, 2009 10:18 pm Post subject: |
|
|
| This must be a tough one. (Or more likely very little interest from others.) Ok plan C. How about cURL? I've been looking at that but it appears that its a quagmire for https. Needs cURL, OpenSSl, and a bunch of MS stuff and some of ti needs to be installed. My need is to provide an executable (compiled script) so I'm hoping not to have to have my users install a bunch of stuff other than through FileInstall. Anyway is there anyone that has used cURL with SSL in a simple-ish way? -thx |
|
| Back to top |
|
|
wakewatcher
Joined: 15 Jul 2006
Posts: 200
|
| Posted: Sun Jun 07, 2009 11:27 pm Post subject: |
|
|
Got a great suggestion/pointer/idea from "tank" on another thread. The idea is to instead of formulating the https post request in emulating a browser just to actually populate the fields on hidden IE browser. This then is the "simple" form I'm interacting with. (What it does is let a user visiting that page upload a file of <50Meg to my S3 account bucket named "autohotkey".)
| Code: | <html>
<head>
<title>S3 POST Form</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<form action=" https://autohotkey.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
<input type="hidden" name="key" value="folder/${filename}">
<input type="hidden" name="AWSAccessKeyId" value="public access key">
<input type="hidden" name="acl" value="public-read">
<input type="hidden" name="success_action_redirect" value=" http://wakewatcher.redirect.hm/sandbox/SendVidS3.php">
<input type="hidden" name="policy" value="big long encoded string generated at run time">
<input type="hidden" name="signature" value="signature generated at run time">
<input type="hidden" name="Content-Disposition" value="attachment; filename=${filename}">
<input type="hidden" name="Content-Type" id="Content-Type" value="video/mp4">
File to upload to S3 (size limit 50kb):
<input name="file" id="file" type="file">
<br/><br/>
<input type="submit" value="Upload File to S3">
</form>
</body>
</html> |
Here I've pulled out the generated encoded ascii strings just for readability and forced the content-type to video/mp4 since that is what I'm uploading to S3 also on success I just have it redirect to the same page. So now off to see what I can find on this idea. |
|
| Back to top |
|
|
wakewatcher
Joined: 15 Jul 2006
Posts: 200
|
| Posted: Mon Jun 08, 2009 4:44 am Post subject: |
|
|
OK making "some" progress. Thought I'd take baby steps and clean it up when/if I get it working. So with this code (unabashedly lifted from tank and sean.) I can open my ie instance, bring up my website and display the value (after pausing to set it) in the file name field. I used tank's nifty DOM viewer to find the item numbers. However I'm struggling with 'setting' the 'file' name field. (Then I will need to auto-magically 'click' the "upload File to S3" button.) I've commented out one of my attempts. I'm not sure if this is something I need or if I can set it with a javascript injection or something. Help?
| Code: | COM_Init()
COM_Error(0)
pwb := COM_CreateObject("InternetExplorer.Application")
COM_Invoke(pwb , "Visible=", "True") ;"False" ;"True" ;
url:="http://www.wakewatcher.redirect.hm/sandbox/SendVidS3.php"
COM_Invoke(pwb, "Navigate", url)
loop
If (rdy:=COM_Invoke(pwb,"readyState") = 4)
break
msgbox Pick File to Upload then click OK
pageSearched:="S3 POST Form"
Loop, % COM_Invoke(psw := COM_Invoke(psh:=COM_CreateObject("Shell.Application"), "Windows"), "Count")
{
LocationName:=COM_Invoke(pwb:=COM_Invoke(psw, "Item", A_Index-1), "LocationName")
IfInString,LocationName,%pageSearched%
Break
COM_Release(pwb)
}
item14:=COM_Invoke(all:=COM_Invoke(document:=COM_Invoke(pwb,"Document"),"All"),"Item",14)
MsgBox % value:=COM_Invoke(item14,"value")
file := "C:\zappa.txt"
;COM_Invoke(file:=COM_Invoke(all,"Item","file"),"value",file)
;MsgBox % value:=COM_Invoke(item14,"value")
COM_Release(item14),COM_Release(all),COM_Release(document),COM_Release(pwb)
COM_Invoke(pwb, "Quit")
COM_Term() |
|
|
| Back to top |
|
|
wakewatcher
Joined: 15 Jul 2006
Posts: 200
|
| Posted: Mon Jun 08, 2009 7:32 am Post subject: |
|
|
I keep hitting road blocks. I couldn't figure out the COM routines to set the file name in the form so I tried the brute force method with ControlSetText injecting some javascript in the address bar ala:
| Code: | SetTitleMatchMode,2
COM_Init()
COM_Error(0)
pwb := COM_CreateObject("InternetExplorer.Application")
COM_Invoke(pwb , "Visible=", "True") ;"False" ;"True" ;
url:="http://www.wakewatcher.redirect.hm/sandbox/SendVidS3.php"
COM_Invoke(pwb, "Navigate", url)
loop
If (rdy:=COM_Invoke(pwb,"readyState") = 4)
break
ControlFocus, Edit1,S3 POST Form
ControlSetText, Edit1, javascript:void(document.uploadform.file.value="C:\dynomohum.php"),S3 POST Form
ControlSend, Edit1, {ENTER},S3 POST Form
msgbox wait
COM_Invoke(pwb, "Quit")
COM_Term()
return |
However it doesn't work and worse yet apparently isn't allowed. | Quote: | | It is a security feature in browsers. You cannot change the value of an <input type="file"> using JavaScript. |
(Tested the js by changing the type to 'text' and it set the field fine.)
So I assume I can't do it with a COM function either. Any Ideas? |
|
| Back to top |
|
|
JoeSchmoe
Joined: 17 Feb 2008
Posts: 167
|
| Posted: Fri Jul 24, 2009 6:37 pm Post subject: Tips on using S3 |
|
|
Hi wakewatcher,
It looks like you've put a lot of work into trying to figure out how to use Amazon S3. I would love to use it, but, like you, am intimidated by the complexity. It seems like it could be a terrific resource. I'm bummed that you didn't get much of a response from the forum, but my hunch is that your requests required highly specialized knowledge and that you are probably the only person with all of the relevant knowledge at this point.
Originally, I had written out my question in this post, but it now seems more appropriate to transfer it to the following post:
http://www.autohotkey.com/forum/viewtopic.php?p=283774#283774 |
|
| Back to top |
|
|
|
|
You can post new topics in this forum
You can reply to topics in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Register
Profile
Log in to check your private messages
Log in
