Recommended way to get a webpage and parse it later

[menteith]

Hi all,

I am trying to get a web page and parse some text from it (with document.getElementById and so on). What would be best way to do it?

I could use:

First solution:

Code: Select all

url = www.google.pl ; an example

	whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
	whr.Open("GET", url, false ), whr.Send()
	www := whr.ResponseText

It works fine, but if a webpage is in charset UTF-8 it sometimes throws and error (0x80070459 - No mapping for the Unicode character exists in the target multi-byte code page.). A possible workaround would be:

Second solution:

Code: Select all

url = www.google.pl ; an example

	whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
	whr.Open("GET", url, false ), whr.Send()
	www := whr.ResponseText
	if A_LastError
		www := URLDownloadToVar(sURL, "UTF-8")
		
; found at https://autohotkey.com/board/topic/101686-objectresponsetext-error/
URLDownloadToVar(url, Encoding = ""){
	hObject:=ComObjCreate("WinHttp.WinHttpRequest.5.1")
	hObject.Open("GET",url)
	hObject.Send()

	If Encoding {
		oADO          := ComObjCreate("adodb.stream")
		oADO.Type     := 1
		oADO.Mode     := 3
		oADO.Open()
		oADO.Write( hObject.ResponseBody )
		oADO.Position := 0
		oADO.Type     := 2
		oADO.Charset  := Encoding
		return oADO.ReadText(), oADO.Close()
	}
	return hObject.ResponseText
}

However, this workaround sometimes fails to work even thought I think it should – if A_LastError should work fine. I also tried if ErrorLevel.

Third solution:

Then I tried to use URLDownloadToFile but the URL should be encoded and more importantly the downloaded webpage could be in different charset and (I could be wrong) reading files in different charset could be challenging at times in AHK.

The following used to work but when I installed Internet Explorer 11 (I had had IE 8 earlier) it fails to work: It works fine after I have restarted my computer once again.

Fourth solution:

Code: Select all

IE := ComObjCreate("InternetExplorer.Application")
IE.Visible := false
url = http://www.google.pl
IE.Navigate(url)

While IE.Busy
Sleep, 10

contents := IE.document.body.outerhtml

Any ideas?

[Capn Odin]

How about requesting a specific charset ?

Code: Select all

url := "http://www.google.pl" ; an example

whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
whr.Open("GET", url)
whr.SetRequestHeader("Accept-Charset", "iso-8859-1")
whr.Send()
www := whr.ResponseText

[menteith]

I will work. Basically my second solution does that but only when I download a page I will know the charset:)

[Joe Glines]

I was helping a friend work with an API and getting a weird error on the ResponseText.

The error was "No mapping for the Unicode character exists in the target multi-byte code page"

I asked Tank to give me a hand and he showed me how to iterate over the ResponseBody and convert the array of bytes to thext