Cloudflare and COM! Topic is solved

[Bugz000]

Hey guys,
Before i start, i am an advanced AHK user, but absolutely horrendous at any form of web systems
the lack of any standardization, and the requirement to know HTML, CSS, JS, PHP, SQL, and f knows what else just to code a website has resulted in me having ZERO desire to be any part of that absolute mess!
as such, my knowledge on how websites and even moreso the security surrounding websites is completely lost on me, i know as much as your grandmother does

Anywho, with that out of the way;
i'm trying to pull information from a site that is using Cloudflart, ofcourse it's checking to see if it's a browser or not, and with COM controlling an IE8 instance, cloudflare should pass

i have scoured the forums and found ONE guy who appears to have got it working but only posted vague details on why and how (also was horrendously salty that nobody was helping him!)
here's the code he posted (renamed WHO to ie because it was bugging me)

Code: Select all

		hdr := "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)"
		url1 := "http://google.com/"  ; Referrer URL, in case it's required
		ie := ComObjCreate("WinHttp.WinHttpRequest.5.1")
		ie.Option(0) := hdr				; UserAgentString
		ie.Option(4) := 0x3300		; SslErrorIgnoreFlags (ignore all certificate errors)
		ie.Option(6) := 1			    ; EnableRedirects
		ie.Option(9) := 0xA8		  	; SecureProtocols: TLS 1.0=0x80, SSL 3.0=0x20, SSL 2.0=0x8
		ie.Option(11) := 0		     	; RevertImpersonationOverSsl
		ie.Option(12) := 1			  	; EnableHttpsToHttpRedirects
		ie.Option(17) := 1			    ; EnableHttp1_1
		ie.Option(18) := 0			    ; EnableCertificateRevocationCheck
		ie.Open("GET", url, true)	; async mode=true
		ie.SetRequestHeader("Referer", url1)
		ie.SetRequestHeader("User-Agent", hdr) ; (this appears to be separate from the WinHTTP option 0)
		ie.SetTimeouts(0, 60000, 30000, 120000)	; Resolve, Connect, Send, Receive.Timeout
		ie.Send()
		ie.WaitForResponse(120)	; only in async mode
		response := ie.ResponseText
		; url2 is the prepared reply URL containing the elements of the solved challenge
		ie.Open("GET", url2, true)	; async mode=true
		ie.SetRequestHeader("Referer", url)  ; set original URL as referrer
		ie.SetRequestHeader("User-Agent", hdr)
		ie.SetTimeouts(0, 60000, 30000, 120000)	; Resolve, Connect, Send, Receive.Timeout
		ie.Send()
		ie.WaitForResponse(120)	; only in async mode
		finalResponse := ie.ResponseText
		ie.quit()

this errors out in a million different ways, which is to be expected!

specifically:

Code: Select all

Error:  0x80070057 - The parameter is incorrect.
Specifically: Open
	025: ie.Open("GET", url, true)  
	026: ie.SetRequestHeader("Referer", url1)  
	027: ie.SetRequestHeader("User-Agent", hdr)  
	028: ie.SetTimeouts(0, 60000, 30000, 120000)  
	029: ie.Send()  
	030: ie.WaitForResponse(120)  
	031: response := ie.ResponseText
--->	033: ie.Open("GET", url2, true)  
	034: ie.SetRequestHeader("Referer", url)  
	035: ie.SetRequestHeader("User-Agent", hdr)  
	036: ie.SetTimeouts(0, 60000, 30000, 120000)  
	037: ie.Send()  
	038: ie.WaitForResponse(120)  
	039: finalResponse := ie.ResponseText
	040: ie.quit()  

however the data from

Code: Select all

response := ie.ResponseText

appears to contain captcha

Code: Select all

            <h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>
            
            <p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>

i cannot find much information on how cloudflare works (though a billion different sites that will check if your website's cloudflare is working or not!... not much use )

I run Brave as a main browser, and it disables all JS iirc, this leads to me being captcha'd at every check, which is fine for me, ofcourse not fine for automation
now ofcourse COM uses IE8, so is it possible there's JS code not runnning or something?

https://www.autohotkey.com/boards/viewtopic.php?t=14218

certainly seems to be a JS issue!

https://autohotkey.com/board/topic/111944-winhttprequest-and-ddos-protection/

Progress has been made in the form of this;
https://github.com/Stickymaddness/Procurement/wiki/SessionID#chrome

seems procurement got by the issue by informing people to plug in their session ID, which isn't too bad!!

i'll look to getting this working, ofcourse any more info would be fantastic!

[malcev]

What is the site?

[Bugz000]

What is the site?

Curseforge

sadly their API is lacking significantly regarding pulling information etc

[kczx3]

Brave does not block all JS. You wouldn’t be able to use any websites honestly if that were the case.

Dumb question but in your posted code I don’t see url2 defined

[Bugz000]

(it is on by default for all URLs for me)

i wouldn't pay too much attention to the code above, though URL2 is defined as a referral URL, and i think he said was only "if required"

either way it seems the solution to my problem lies within POE Trademacro, i'm in the process of pulling that code apart atm

[malcev]

Because of captcha You have to use Msxml2.XMLHTTP.6.0.
If You want to use WinHttp.WinHttpRequest.5.1 You have to send cookie manually.

Code: Select all

HTTP := ComObjCreate("Msxml2.XMLHTTP.6.0")
HTTP.Open("GET", "https://www.curseforge.com/", false)
HTTP.SetRequestHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko")
HTTP.SetRequestHeader("Pragma", "no-cache")
HTTP.SetRequestHeader("Cache-Control", "no-cache, no-store")
HTTP.SetRequestHeader("If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT")
HTTP.SetRequestHeader("Host", "www.curseforge.com")
HTTP.SetRequestHeader("Accept-Language", "en-US, en; q=0.8, ru; q=0.5, lv; q=0.3")
HTTP.SetRequestHeader("Accept", "text/html, application/xhtml+xml, image/jxr, */*")
HTTP.Send()
msgbox % HTTP.ResponseText

It works for me with Win10 and IE11.

[Bugz000]

Because of captcha You have to use Msxml2.XMLHTTP.6.0.
If You want to use WinHttp.WinHttpRequest.5.1 You have to send cookie manually.

Code: Select all

HTTP := ComObjCreate("Msxml2.XMLHTTP.6.0")
HTTP.Open("GET", "https://www.curseforge.com/", false)
HTTP.SetRequestHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko")
HTTP.SetRequestHeader("Pragma", "no-cache")
HTTP.SetRequestHeader("Cache-Control", "no-cache, no-store")
HTTP.SetRequestHeader("If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT")
HTTP.SetRequestHeader("Host", "www.curseforge.com")
HTTP.SetRequestHeader("Accept-Language", "en-US, en; q=0.8, ru; q=0.5, lv; q=0.3")
HTTP.SetRequestHeader("Accept", "text/html, application/xhtml+xml, image/jxr, */*")
HTTP.Send()
msgbox % HTTP.ResponseText

It works for me with Win10 and IE11.

Fantastic work dude!

i found the required code in poe trademacro and was just about to start stripping it down but you've saved me a lot of work <3 thanks dude!

[malcev]

I was wrong. It also can be done with WinHttp.WinHttpRequest.5.1.
And if You do not use proxy, then there is no need to send cache related headers: with WinHttp.WinHttpRequest.5.1.

HTTP.SetRequestHeader("Pragma", "no-cache")
HTTP.SetRequestHeader("Cache-Control", "no-cache, no-store")
HTTP.SetRequestHeader("If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT")

Actually this "Cloudflare protection" requires TLS connection and checks for referer and for cookie with domain hash (by my test, they can be any value).

Code: Select all

HTTP := ComObjCreate("WinHttp.WinHttpRequest.5.1")
HTTP.Option(9) := 0x80   ; TLS 1.0
HTTP.Open("GET", "https://www.curseforge.com/", true)
HTTP.SetRequestHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko")
HTTP.SetRequestHeader("Referer", "https://www.curseforge.com/")
HTTP.SetRequestHeader("Cookie", "__utmc=94490894")
HTTP.Send()
HTTP.WaitForResponse()
msgbox % HTTP.ResponseText