[Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No IE!

[Stavencross]

Stavencross

Really been digging in to this Lib, its so helpful! Just want to see if ya'll have a better way on how to scrape the JSON I've been grabbing from these pages. Here is my code:

*snip*

If you check jsonGrabber(), you'll see I'm grabbing the JSON from the page DOM. I grab the innerHTML of the page, and run strReplace to get rid of the <pre> tags. I thought I'd see if ya'll had a better way to accomplish this as it feels dirty. I'm also open to any other code optimizations you could throw my way. I've been working on my coding skills a lot in the last 6 months and I know I really need to start pushing myself to write better & more modular code

It's hard to say without access to the actual page, but I would bet that grabbing from document.body.innerText would work better for you than document.body.innerHTML

I really love this library! Just a couple questions as I'd like to work it in deeper into my webapp.

One of the things I have to do is navigate to a url, and which triggers a pdf download. Is there a way for the lib to know when the download is complete? Right now I use a function called watch folder to loop through my downloads folder until a new file is present and grab it.

https://developer.chrome.com/extensions ... -onCreated

Something like this here would be perfect, but it's part of the extensions Api so I don't know that you have access to it.

Secondly, is it possible to launch chrome without the ui buttons, or similar to how in IE we can launch a shell as a gui browser?

[geek]

It will be very powerful instrument!

In "Source code (zip)" - link => in "Chrome.ahk" file => in 72 code-line have a one missing letter "t" (...\App Pahs\chrome.exe).

My bad, I'll fix it in the next release. Thanks for pointing it out!

Can I ask you to add the name of the function to the constructor of class Page by the third parameter, which will be called when the connection is broken?
For example:

*snip*

I make a simple wrapper for use in the program and concluded that it would be convenient to remove data about an unused connection from the object of this wrapper because of a disconnection that may occur if the user closes the tab.
Example:

*snip*

I hope this request does not look like impudence.
And thank you very much for your hard work!
Beautiful code and maximum efficiency!

Good idea, I'll try to add it to the next release. Thanks!

And another question. Is it possible to replace this exception:

Code: Select all

Call(DomainAndMethod, Params:="", WaitForResponse:=True)
{
 if !this.Connected
 throw Exception("Not connected to tab")
 
 ; ... other code
}

To return a connection to the state "disabled"?

I think that this is logical, if all the tabs are closed, then you need to create a new connection, reinitializing the instance of the class Chrome. Instead, there is a constant call "Runtime.evaluate" without parameters, and the program terminates its work after the exception message.

Because it throws an exception you can put multiple calls into a single try-catch block, and use the catch to define something to happen instead of the script terminating. If Call returned a value to indicate an error instead, you would have to check the result of every Call individually, and it would make error propagation through wrappers like Evaluate much more difficult.

Sometimes, when creating an instance of a class Chrome, passing an address to the constructor, the method WaitForLoad() returns flow control before the page loads:

*snip*

This can be circumvented by creating an instance of the class without an address, followed by a transition PageInst.Call("Page.navigate", {"url": address}) and PageInst.WaitForLoad(), but is there any way to do without it?

Creating an instance without an address (or with an address like about:blank then later using Page.navigate is the recommended way right now. It's likely that, in the future, the library will use this method internally instead of passing in the URLs directly.

Is it possible to connect to events such as Target.targetCreated? This feature would reduce the possibility of errors, telling the ahk-code what him need to create a new connection.

I haven't tested it, but I think if use the Target.setDiscoverTargets method that you can receive those events using the new callback system.

I really love this library! Just a couple questions as I'd like to work it in deeper into my webapp.

One of the things I have to do is navigate to a url, and which triggers a pdf download. Is there a way for the lib to know when the download is complete? Right now I use a function called watch folder to loop through my downloads folder until a new file is present and grab it.

https://developer.chrome.com/extensions ... -onCreated

Something like this here would be perfect, but it's part of the extensions Api so I don't know that you have access to it.

Secondly, is it possible to launch chrome without the ui buttons, or similar to how in IE we can launch a shell as a gui browser?

I don't think there's a way to do that directly, but there may be a somewhat indirect way to go about it. When using Chrome.ahk you can connect to regular tabs, but also to the pages that make up extensions. If there's an extension with the proper permissions already in place, or if you can get an extension of your own creation installed, you should be able to connect to the extension and manipulate things from there.

Another possibility would be using one of the more hidden targets/pages that you can connect to. There's one listed on this page http://localhost:9222/json/version separate from all the others (which are listed on just http://localhost:9222/json) which has some slightly different functionality to it. What that functionality is, though, I'm not sure.

One method you may be interested in, although it's not what your'e asking for, is Page.setDownloadBehavior which lets you change the folder that the files are saved to.

Final thing on this question. Your method of watching the folder sounds pretty robust, given how chrome downloads as a separate file then renames it to be the final downloaded file once it's complete. There are very efficient methods of watching folders demonstrated elsewhere on the forum. For example, https://autohotkey.com/boards/viewtopic.php?t=8384 . On the other hand, you may find that complex functions for such a simple problem are unnecessary . I'm all about informing people of their options, but you're more than free to make your own opinion.

For your second question, I'm pretty sure the answer is not with this library. You could look into something like the Chrome Embedded Framework for Python and try to copy what it does, but it will be a lot of work.

[malcev]

Why does it take a lot of time to get page source?
For example if I use IE or Selenium it takes about 100ms.
IE:

Code: Select all

oIE := ComObjCreate("InternetExplorer.Application")
oIE.visible := True
oIE.navigate("http://delfi.lv")
While oIE.readyState != 4 || oIE.document.readyState != "complete" || oIE.busy  
   sleep 10
a := a_tickcount
OuterHTML := oIE.Document.documentElement.OuterHtml
msgbox % a_tickcount - a

Selenium:

Code: Select all

driver:= ComObjCreate("Selenium.ChromeDriver")
driver.Get("http://delfi.lv")
a := A_TickCount
OuterHtml := driver.PageSource()
msgbox % A_TickCount - a

But with Chrome DevTools Protocol about 3500ms.
I tried with CHROME DOM API and with JavaScript injection - the time is the same.
Chrome DevTools Protocol:

Code: Select all

ChromeInst := new Chrome()
PageInst := ChromeInst.GetPage()
PageInst.Call("Page.navigate", {"url": "http://delfi.lv"})
PageInst.WaitForLoad()

a := a_tickcount
RootNode := PageInst.Call("DOM.getDocument").root
OuterHTML := PageInst.Call("DOM.getOuterHTML", {"nodeId": RootNode.nodeId}).OuterHTML
msgbox % a_tickcount - a

a := a_tickcount
OuterHTML := PageInst.Evaluate("document.documentElement.outerHTML;").Value
msgbox % a_tickcount - a

[geek]

The delay is caused by decoding chrome's response from a JSON message into an AHK object. Like with IE, the browser responds with the source immediately. However, IE responds with the text directly, and Chrome responds with a JSON blob that contains the text but must be decoded into a useful form. Because AutoHotkey does not support JSON encode/decode natively, this conversion must be done with a user-made script. That page's source is roughly 200kb, so the decoding process takes a while.

One alternative may be to use the file api to make chrome download the source as a file, then use AHK to read in that file. This takes about ~700ms on my system.

Code: Select all

FileRemoveDir, DownloadDir, 1
FileCreateDir, DownloadDir

; Allow automatic downloads without prompt
PageInst.Call("Page.setDownloadBehavior"
, {"behavior": "allow", "downloadPath": A_WorkingDir "\DownloadDir"})

a := a_tickcount

; Run some javascript to download the page source as a file
ExpectedSize := PageInst.Evaluate("
( LTrim Join
(function(){
	var a = document.createElement('a');
	var blob = new Blob([document.documentElement.outerHTML]);
	a.href = window.URL.createObjectURL(blob);
	a.download = 'Source.txt';
	a.click();
	return blob.size;
})();
)").value

; Wait for the file to exist then read and remove it.
; Chrome may report a download error because we remove
; the file so quickly, but that is not important
while !FileExist("DownloadDir\Source.txt")
	Sleep, 10
Source := FileOpen("DownloadDir\Source.txt", "r").Read()
FileDelete, DownloadDir\Source.html

msgbox % a_tickcount - a
MsgBox, % "Expected: " ExpectedSize "`nReceived: " StrLen(Source)

ExitApp
return

Some other options could be to find a more performant way of reading JSON and extend/modify the class to use that instead. You could look for another way of doing interprocess communication with chrome, such as fiddling with the clipboard (I tried this with little to no success) or saving to the browser's local storage and reading that with an sqlitedb script.

Another potential option is to try to avoid transferring so much data across. If there's some kind of processing that needs to be done to the page, could that processing be done in JavaScript and only the results be returned to AHK?

Finally, you could use something other than this library, like the Selenium example you used to demonstrate.

Good luck!

[Stavencross]

To start chrome without all the UI buttons, instantiate the class like this
ChromeInst := new Chrome("ChromeProfile", "--app=https://autohotkey.com/")

[KusochekDobra]

Is it possible to connect to events such as Target.targetCreated? This feature would reduce the possibility of errors, telling the ahk-code what him need to create a new connection.

I haven't tested it, but I think if use the Target.setDiscoverTargets method that you can receive those events using the new callback system.

Thank you very much! Indeed, it works! And I did not think about this possibility.

[PoorInRichfield]

Not an earth-shattering change, but I added the -incognito flag to the Run command in attempt to get Chrome to stop showing all sorts of annoying alerts and pop-ups...

Code: Select all

	Run, % this.CliEscape(ChromePath)
		. " -incognito --remote-debugging-port=" this.DebugPort
		. (ProfilePath ? " --user-data-dir=" this.CliEscape(ProfilePath) : "")
		. (Flags ? " " Flags : "")
		. URLString
		,,, OutputVarPID
		this.PID := OutputVarPID

[geek]

Not an earth-shattering change, but I added the -incognito flag to the Run command in attempt to get Chrome to stop showing all sorts of annoying alerts and pop-ups...

Some other good flags I've seen around the net are --no-first-run and --disable-extensions. You don't necessarily have to modify the Chrome class to add flags, since there's a parameter in the constructor you can pass them to.

[Xtra]

A nice list here:
chromium-command-line-switches

[KusochekDobra]

Help me please.
I do not understand, how to use Chrome browser with "Chrome.ahk" in "Headless" mod.
Can you give an example?

[geek]

Help me please.
I do not understand, how to use Chrome browser with "Chrome.ahk" in "Headless" mod.
Can you give an example?

Check out the ExportPDF.ahk example

[Guest]

After spending the last 2 hours trying to figure out GetElement and GetTab, I finally realized GetTab doesn't exist lol.

Anyways, thank you SO MUCH for adding this!

I have one really dumb question - how do I extract the URL from the Address Bar?

Also, is there documentation on what functions are available with Chrome? I'm very, very new to programming and don't know any COM syntax. But I'm eager to learn!

Thanks,
Bryan

[Xtra]

how do I extract the URL from the Address Bar?

PageInstance.Evaluate("document.URL;").Value
or
PageInstance.Evaluate("window.location.href;").Value

HTH

[taapo]

How do you "type" text?

I'm trying to create a status update for a Facebook page using your script, and I'm almost there, but I'm running into problems.

So'm I'm unsure if I should do this:

Code: Select all

PageInst.Evaluate("document.querySelector('div._1mf._1mj span').click();")
Send This is a status update
PageInst.Evaluate("document.querySelector('._1mf7._69g1._4jy0._4jy3._4jy1._51sy.selected._42ft').click();")

... which doesn't seem to work properly half of the time (not sure why).

or more something like this (which gives a script error):

Code: Select all

PageInst.Evaluate("document.querySelector('div._1mf._1mj span span').value('This is a status update');")
PageInst.Evaluate("document.querySelector('._1mf7._69g1._4jy0._4jy3._4jy1._51sy.selected._42ft').click();")

Any pointers?

[geek]

The syntax to set the value of a control is (IIRC) document.querySelector('abc').value = 'This is some text', but many websites don't like it when you set the value directly.

In that case, you would want to make a bunch of calls to the dispatchKeyEvent endpoint documented here: https://chromedevtools.github.io/devtoo ... /tot/Input

I don't have any examples of using it though, sorry!

[malcev]

In selenium You can post in facebook like this:

Code: Select all

driver:= ComObjCreate("Selenium.ChromeDriver")
driver.Get("https://www.facebook.com/")
element := driver.findElementByCss("textarea[name=""xhpc_message""]")
element.Click()
element.sendKeys("test")
element := driver.findElementByCss("button[data-testid=""react-composer-post-button""]")
element.Click()

[KusochekDobra]

Help me please.
I do not understand, how to use Chrome browser with "Chrome.ahk" in "Headless" mod.
Can you give an example?

Check out the ExportPDF.ahk example

When I run example, MsgBox tells me "Could not retrieve page!". This is exactly the problem, because of which I can not understand how to use "HeadLess Mod".

[malcev]

You have to run it like this:

Code: Select all

ChromeInst := new Chrome("", "https://example.com", "--headless")

[KusochekDobra]

It work!
Thank U very much!!!

[Manuf]

Hi sirs. Can you help me with this simple task

i want to write a script to detect
- can check if webpage full loaded
- can check if webpage loaded error
- can check if an element display after an action. Ie: after upload a file, a button will turn from gray to yellow. i want to check that state of the button.

Thanks you