The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Tigerlily · 27 Feb 2019, 16:22

Hi Folks,

Can anyone tell me why when web scraping some websites with WinHttpRequest5.1 API, this error is thrown and won't retreive any data from this site?

Two URLs that I tested and found did not work:

http://www.artodia.com/
https://www.phpbb.com/ideas/

these are both return 200 OK status codes in the browser.

This is the error message that it throws:

0x80072EF1 -
Source: WinHttp.WinHttpRequest
Description: The operation has been canceled

Which correlates to this error in the WinHttpRequest5.1 MSDN docs:

ERROR_WINHTTP_OPERATION_CANCELLED

12017

The operation was canceled, usually because the handle on which the request was operating was closed before the operation completed.

https://docs.microsoft.com/en-us/windows/desktop/winhttp/error-messages

How do I keep the handle open? Why is the handle closing when I did not specify it to? Is this the website blocking me? Is there any way around this? I don't really understand what a "handle" is in this scenario..

Thank you for your replies.

Tigerlily · 04 Mar 2019, 03:57

*bump*

Anyone???

Thanks!

teadrinker · 04 Mar 2019, 08:58

Hi

With http://www.artodia.com/ the script works for me like this:

Code: Select all

oWhr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
oWhr.Open("GET", "http://www.artodia.com/", false)
oWhr.Send()
MsgBox, % oWhr.responseText

With https://www.phpbb.com/ideas/ data is loaded, if you specify the request header oWhr.SetRequestHeader("Accept-Encoding", "gzip, deflate, br")

In this case, the data will be loaded in gzip encoding. To decode such data actually you need zlib1.dll, but you could using the online js library.

Code: Select all

oWhr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
oWhr.Open("GET", "https://www.phpbb.com/ideas/", false)
oWhr.SetRequestHeader("Accept-Encoding", "gzip, deflate, br")
oWhr.Send()

MsgBox, % oWhr.getAllResponseHeaders()
MsgBox, % oWhr.status

arr := oWhr.ResponseBody
length := arr.MaxIndex() + 1
pData := NumGet( ComObjValue(arr) + 8 + A_PtrSize )
MsgBox, % res := GzipDecode(pData, length)

GzipDecode(pData, size) {
   arr := "["
   Loop % size
      arr .= (arr = "[" ? "" : ",") . NumGet(pData + A_Index - 1, "UChar")
   arr .= "]"

   html =
   (
   <!DOCTYPE html>
   <html>
   <head>
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <script src="https://cdnjs.cloudflare.com/ajax/libs/pako/1.0.8/pako.min.js"></script>
     <script>
       var charData = %arr%;
       var binData  = new Uint8Array(charData);
       var inflated = pako.inflate(binData);
       var arr16    = new Uint16Array(inflated);
       var str      = arr16.length;
       for (var i = 0; i < arr16.length; i++)
         str += "|" + arr16[i];
       var _str     = str;
     </script>
   </head>
   </html>
   )
   oDoc := ComObjCreate("htmlfile")
   oDoc.write(html)
   Loop {
      Sleep, 10
      try res := oDoc.parentWindow._str
      catch
        continue
      break
   }
   RegExMatch(res, "(?<len>\d+)\|(?<arr>.*)", _)
   VarSetCapacity(data, _len*2, 0)
   Loop, parse, _arr, |
      NumPut(A_LoopField, data, (A_Index - 1)*2, "UShort")
   Return StrGet(&data, "UTF-16")
}

Tigerlily · 20 Mar 2019, 13:35

teadrinker wrote: ↑

04 Mar 2019, 08:58

Hi

With http://www.artodia.com/ the script works for me like this:

Code: Select all

oWhr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
oWhr.Open("GET", "http://www.artodia.com/", false)
oWhr.Send()
MsgBox, % oWhr.responseText

With https://www.phpbb.com/ideas/ data is loaded, if you specify the request header oWhr.SetRequestHeader("Accept-Encoding", "gzip, deflate, br")

In this case, the data will be loaded in gzip encoding. To decode such data actually you need zlib1.dll, but you could using the online js library.

Code: Select all

oWhr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
oWhr.Open("GET", "https://www.phpbb.com/ideas/", false)
oWhr.SetRequestHeader("Accept-Encoding", "gzip, deflate, br")
oWhr.Send()

MsgBox, % oWhr.getAllResponseHeaders()
MsgBox, % oWhr.status

arr := oWhr.ResponseBody
length := arr.MaxIndex() + 1
pData := NumGet( ComObjValue(arr) + 8 + A_PtrSize )
MsgBox, % res := GzipDecode(pData, length)

GzipDecode(pData, size) {
   arr := "["
   Loop % size
      arr .= (arr = "[" ? "" : ",") . NumGet(pData + A_Index - 1, "UChar")
   arr .= "]"

   html =
   (
   <!DOCTYPE html>
   <html>
   <head>
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <script src="https://cdnjs.cloudflare.com/ajax/libs/pako/1.0.8/pako.min.js"></script>
     <script>
       var charData = %arr%;
       var binData  = new Uint8Array(charData);
       var inflated = pako.inflate(binData);
       var arr16    = new Uint16Array(inflated);
       var str      = arr16.length;
       for (var i = 0; i < arr16.length; i++)
         str += "|" + arr16[i];
       var _str     = str;
     </script>
   </head>
   </html>
   )
   oDoc := ComObjCreate("htmlfile")
   oDoc.write(html)
   Loop {
      Sleep, 10
      try res := oDoc.parentWindow._str
      catch
        continue
      break
   }
   RegExMatch(res, "(?<len>\d+)\|(?<arr>.*)", _)
   VarSetCapacity(data, _len*2, 0)
   Loop, parse, _arr, |
      NumPut(A_LoopField, data, (A_Index - 1)*2, "UShort")
   Return StrGet(&data, "UTF-16")
}

Thank you very much for your insightful reply teadrinker - I always appreciate your insight in these forums. How come for artodia.com must I make the request synchronous? I would need to create some sort of event handler to check for these cases since I will not know when I will encounter a URL that requires this. How can I discern between a URL that requires a synchronous request vs. asynchronous request?

Is there any way around the gzip encoding - why must I request it to be gzip as opposed to some other format? Also, how do I spot when this may be a problem? I would never have figured this out if you didn't show me. From my understanding gzip compression is helpful for resource loading efficiency on a webpage, so I understand that I also may encounter this more frequently as is actively Google recommending this to conserve resources. I will need to create an event handler for this too at some point.. but looks like some work..

teadrinker · 20 Mar 2019, 14:45

Urls don't require synchronous or asynchronous requests, this is only your choice. You choose one of them, which is convenient for you.

Tigerlily wrote: ↑ Is there any way around the gzip encoding - why must I request it to be gzip as opposed to some other format?

I don't have enough experience to answer for sure. As I can see in my firefox developer tools, the browser always sets the header Accept-Encoding to "gzip, deflate, br". Then, as I understand, it cheks if the response header "Content-Encoding" contains "gzip" or "inflate", and if yes uses the decoding algorithm to read the response body.

20 Mar 2019, 15:16

bored at work today

teadrinker wrote: ↑
20 Mar 2019, 14:45
synchronous or asynchronous

asynchronous you need an event handler
synchronous you do not

Tigerlily wrote: ↑
20 Mar 2019, 13:35
Is there any way around the gzip encoding

Nope its dictated by the site. You have to check the headers of the web site your requesting Content-Encoding: gzip means you need to accept and decode gzip

teadrinker · 20 Mar 2019, 15:53

tank wrote: ↑Nope its dictated by the site.

Yes, but in most cases you can choose whether to use compression or not:

Code: Select all

whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
whr.Open("GET", "https://www.autohotkey.com/boards/viewtopic.php?f=76&t=62405&p=268688/", false)
whr.Send()
try MsgBox, % whr.GetResponseHeader("Content-Encoding")
catch
   MsgBox, No Content-Encoding header
MsgBox, % whr.ResponseText

whr.Open("GET", "https://www.autohotkey.com/boards/viewtopic.php?f=76&t=62405&p=268688/", false)
whr.SetRequestHeader("Accept-Encoding", "gzip, deflate")
whr.Send()
try MsgBox, % contentEncoding := whr.GetResponseHeader("Content-Encoding")
if !(contentEncoding ~= "i)gzip|inflate")
   MsgBox, % whr.ResponseText
else {
   arr := whr.ResponseBody
   length := arr.MaxIndex() + 1
   pData := NumGet( ComObjValue(arr) + 8 + A_PtrSize )
   MsgBox, % GzipDecode(pData, length)
}
Return

GzipDecode(pData, size) {
  if !FileExist(A_ScriptDir . "\pako.min.js") {
    URLDownloadToFile, https://cdnjs.cloudflare.com/ajax/libs/pako/1.0.8/pako.min.js, %A_ScriptDir%\pako.min.js
    if ErrorLevel
      throw Exception("Can't download script")
  }
   arr := "["
   Loop % size
      arr .= (arr = "[" ? "" : ",") . NumGet(pData + A_Index - 1, "UChar")
   arr .= "]"

   html =
   (
   <!DOCTYPE html>
   <html>
   <head>
     <meta http-equiv='X-UA-Compatible' content='IE=edge'>
     <script src="file://%A_ScriptDir%\pako.min.js"></script>
     <script>
       var charData = %arr%;
       var binData  = new Uint8Array(charData);
       var inflated = pako.inflate(binData, {to:'string'});
     </script>
   </head>
   </html>
   )
   doc := ComObjCreate("htmlfile")
   doc.write(html)
   Loop {
      Sleep, 10
      try res := doc.parentWindow.inflated
      catch
        continue
      break
   }
   Return res
}

On the other hand, if you do not set "Accept-Encoding" to "gzip, deflate", you never (?) get comressed body.

teadrinker · 21 Mar 2019, 13:37

I fixed the GzipDecode() function.

The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Re: The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Re: The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Re: The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Re: The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Re: The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Re: The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Re: The operation has been canceled - error 12017 - 0x80072EF1 - WinHttpRequest5.1 COM

Who is online