[Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

Post your working scripts, libraries and tools
ars4l4n
Posts: 5
Joined: 26 Aug 2020, 07:27

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

13 Sep 2020, 04:40

@Tre4shunter
in my usecase I want to just select the textbox and not have the script type something into it. I just don't like that they don't immediately activate the textbox, I think it's a bad UI decision by them
@burque505
But knowing about inserting a text into the textbox via the script also seems interesting. How did you find out it has to be this

Code: Select all

("document.querySelector(""#twotabsearchtextbox"")
EDIT: ah, you just rightclick->copy js path(ed) on the portion of the textbox
hm you know, I just kept watching Geekdudes tutorial and I think he should've shown the general Javascript example before the specific api example to make it less confusing
____________________

EDIT2: I tried

Code: Select all

document.querySelector("#twotabsearchtextbox").click
inside the devtool console but it didn't click the searchbox >.<
burque505
Posts: 1398
Joined: 22 Jan 2017, 19:37

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

13 Sep 2020, 07:57

Hi, @ars4l4n,

Code: Select all

document.querySelector("#twotabsearchtextbox")click();
runs without error for me too, but there's no event listener yet. Try this:

Code: Select all

inputbox = document.querySelector("#twotabsearchtextbox");
inputbox.addEventListener('click', function (event) {
  alert('Element clicked through function!');
});
Then run this in the console:

Code: Select all

inputbox.click();
You should get the alert as shown.
After that it's just a matter of figuring out what real stuff you want the event listener to actually do! :D
(I'll try here in a bit to see if I can get the cursor to blink inside the input field, if that's the general idea.)

EDIT: This code works for me now to get the cursor blinking in the search field so you can just start typing. (It drove me nuts for half an hour, because Google was "offering" to translate the page to English for me. :problem: When I finally clicked on the settings for that popup and told Google to NEVER translate the page it worked fine.)

Code: Select all

#Include Chrome.ahk
ChromeInst := new Chrome("ChromeProfile")
PageInstance := ChromeInst.GetPage()
PageInstance.Call("Page.navigate", {"url": "https://www.amazon.de/"})
PageInstance.WaitForLoad()
Send, {Tab} ; Needed to get focus away from address bar for me;
Sleep, 1000
PageInstance.Evaluate("searchfield = document.querySelector(""#twotabsearchtextbox"");")
PageInstance.Evaluate("searchfield.focus();")
PageInstance.Evaluate("searchfield.select();")
Sleep, 10000
PageInstance.Call("Browser.close")
ExitApp
Regards,
burque505
Last edited by burque505 on 13 Sep 2020, 09:53, edited 1 time in total.
blue83
Posts: 126
Joined: 11 Apr 2018, 06:38

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

13 Sep 2020, 08:41

Hi,

How to get whole cokies from page using javascript, not just part of it?

Thanks,
Blue
burque505
Posts: 1398
Joined: 22 Jan 2017, 19:37

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

13 Sep 2020, 14:54

Example table scraper using @GeekDude's Chrome.ahk and @teadrinker's JS injection code here. This is a work in progress.
It opens the well-known and much-abused w3schools html tables page here .
It displays a MsgBox with the scraped table results, then writes them into the file 'results.tsv' in the script directory (YMMV, but for me Excel can open a .tsv file directly with all information correctly displayed).

Code: Select all

#Include Chrome.ahk
; Thanks to @teadrinker for the javascript.
; Regards, burque505
SetTitleMatchMode, 2

js = 
(
var items = document.querySelectorAll('#customers > tbody > tr');
var docStr = "";
items.forEach(function(element){
    docStr += element.innerText + "\n";
    });
   (() => {
      if (window.location.protocol === 'https:') {
         document.documentElement.focus();
         const timer = setInterval(() => {
            if (document.hasFocus()) {
               clearInterval(timer);
               navigator.clipboard.writeText(docStr);
            }
         }, 1000);
      }
      else {
         const textArea = document.createElement('textarea');
         textArea.value = document.documentElement.outerHTML;
         textArea.wrap = 'off';
         textArea.rows = 100000;
         textArea.style.position = 'fixed';
         document.documentElement.appendChild(textArea);
         textArea.focus();
         textArea.select();
         document.execCommand('copy');
         textArea.parentNode.removeChild(textArea);
      }
   })();
)

ChromeInst := new Chrome("ChromeProfile")
PageInstance := ChromeInst.GetPage()
PageInstance.Call("Page.navigate", {"url": "http://www.w3schools.com/html/html_tables.asp"})
PageInstance.WaitForLoad()
Send, {Tab} ; Needed to get focus away from address bar for me;
Sleep, 1000
PageInstance.Evaluate(js)
Sleep, 5000
results := Clipboard
msgbox, , Table Results, % ErrorLevel ? "Failed to get value" : results
If FileExist("results.tsv")
	FileDelete, results.tsv
FileAppend, %results%, results.tsv

Sleep, 1000
PageInstance.Call("Browser.close")
WinKill, chrome.exe
ExitApp

/*
; **********************************************************************************
; You may need this at the end of javascript on an https page. Maybe not ...
; See these two pages:
; https://stackoverflow.com/questions/19389200/javascript-sleep-delay-wait-function
; https://stackoverflow.com/questions/951021/what-is-the-javascript-version-of-sleep
; **********************************************************************************
function startTimer () {
   timer.start();
   setTimeout(stopTimer,);
}

function stopTimer () {
    timer.stop();
}
startTimer();
*/
For a snippet of JS to convert other formatting to tab delimited, you might take a look at the post about DropCatch this link.

Regards,
burque505
burque505
Posts: 1398
Joined: 22 Jan 2017, 19:37

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

13 Sep 2020, 15:51

Bloomberg scraper. Be careful with this one. Bloomberg has anti-scraping measures in place and may present you with a captcha. If you run it and it tanks, please be sure to kill all the orphaned chrome.exe running instances. Look at the JS for replacing newlines with tabs. Also writes a 'results.tsv' file in the script directory, which Excel opens painlessly with more-or-less correct formatting.

Code: Select all

#Include Chrome.ahk
; Thanks to @teadrinker for the javascript.
; Regards, burque505
SetTitleMatchMode, 2

js = 
(
var items = document.querySelectorAll('.data-table-row'	);
var docStr = "";
var tmp = "";
var res = "";
items.forEach(function(element){
   tmp = element.innerText;
   res = tmp.replace(/(\r\n|\n|\r)/gm, "\t");
    docStr += res + "\n";
    });
   (() => {
      if (window.location.protocol === 'https:') {
         document.documentElement.focus();
         const timer = setInterval(() => {
            if (document.hasFocus()) {
               clearInterval(timer);
               navigator.clipboard.writeText(docStr);
            }
         }, 1000);
      }
      else {
         const textArea = document.createElement('textarea');
         textArea.value = document.documentElement.outerHTML;
         textArea.wrap = 'off';
         textArea.rows = 100000;
         textArea.style.position = 'fixed';
         document.documentElement.appendChild(textArea);
         textArea.focus();
         textArea.select();
         document.execCommand('copy');
         textArea.parentNode.removeChild(textArea);
      }
   })();
)

ChromeInst := new Chrome("ChromeProfile")
PageInstance := ChromeInst.GetPage()
PageInstance.Call("Page.navigate", {"url": "https://www.bloomberg.com/markets/stocks/world-indexes/americas"})
PageInstance.WaitForLoad()
sleep, 5000
Send, {Tab} ; Needed to get focus away from address bar for me;
Sleep, 1000
PageInstance.Evaluate(js)
Sleep, 10000
results := Clipboard
msgbox, , Table Results, % ErrorLevel ? "Failed to get value" : results
If FileExist("results.tsv")
	FileDelete, results.tsv
FileAppend, %results%, results.tsv

Sleep, 1000
PageInstance.Call("Browser.close")
WinKill, chrome.exe
ExitApp
Bloomberg-2.jpg
Bloomberg-2.jpg (78.09 KiB) Viewed 847 times

Regards,
burque505
ars4l4n
Posts: 5
Joined: 26 Aug 2020, 07:27

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

14 Sep 2020, 07:42

@burque505
How did you know to use this code?
When I google for "search field java", "java ahk" etc. I don't find any useful resources for automating java pages with ahk
Besides that, is there like an addressbar unfocus function to use instead of Send, {Tab}? (Something that doesn't make stuff move on the screen like Tab does)
burque505
Posts: 1398
Joined: 22 Jan 2017, 19:37

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

14 Sep 2020, 08:32

@ars4l4n, almost everything in my code was found either in this topic or elsewhere on the forum, except most javascript snippets (besides @teadrinker's, that is), which come from all over. And search for "javascript", not "java", you may have better results.
Now that you mention it, you could try something focusing on another element instead of sending a tab. Haven't tried it yet, but will later today or tomorrow.
Regards,
burque505
Tre4shunter
Posts: 73
Joined: 26 Jan 2016, 16:05

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

14 Sep 2020, 09:57

Edit - ahh @burque505 beat me to it, didn't mean to hijack your response!

Its Javascript, not java...and really you aren't automating with 'ahk + java' all you are doing is sending javascript to the page through a websocket connection via ahk.

So, you are just better off searching for how to do this directly in Javascript, and then simply execute that on the page with PageInst.Evaluate(JS)

For example - iterate though all text edit fields on a page:

Code: Select all

JS = 
(
var editfields = document.querySelectorAll('input[type="text"');

for(i=0;i<editfields.length;i++)
{
alert(editfields[i].outerHTML)
}
)
PageInst.Evaluate(JS)
burque505
Posts: 1398
Joined: 22 Jan 2017, 19:37

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

14 Sep 2020, 12:00

Nice example, @Tre4shunter.
Regards,
burque505
ars4l4n
Posts: 5
Joined: 26 Aug 2020, 07:27

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

14 Sep 2020, 19:55

@Tre4shunter
I can't figure out what to do with your codesnippet in regards to my script though

Edit:
I hope you're not reluctant to answer because you got the impression I didn't try properly figuring this out on my own

So if I understand correctly this is some method for cycling through searchfields
I tried inserting it into the devconsole to see what it does. Okay, it pops up 3 notifications with what I think are supposed to be all the searchfields in that website. (Only the first popup is information about the actual searchfield on the website, the second and third show the same popup about some other element the search criteria of your code snippet applies to but since I don't see another searchfield on the website they're something else I think. I guess I have to add more search parameters besides ('input[type="text"') to narrow down the cycling to the actual searchfield of that site if this is supposed target it. Don't know how in terms of syntax though)

If I then proceed to enter the previous lines of javascript into the commandline, the searchfield can be focused even when I previously had the addressbar in focus but it may be so only because the messagepopups shifted the focus away from the addressbar
I put it into my autoit script, tried editing out the messagebox to see if that's the case but I got errors and that's basically where my ability to work with this on my own ends because I don't know anything about javascript and can't find the infos I need online in a reasonable amount of time even with the tips on where/how to search provided in this thread.
Tre4shunter
Posts: 73
Joined: 26 Jan 2016, 16:05

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

16 Sep 2020, 19:28

My example was simply to show how you would use javascript to search through all of the input elements which are of type 'text' reference here on css selectors:

https://www.w3schools.com/cssref/css_selectors.asp

You can use them to great specificity to extract very specific parts of a page based on classnames, id's etc.

My example wasn't really a specific answer to your question - more of a generic explanation of what you can do with JS.
burque505
Posts: 1398
Joined: 22 Jan 2017, 19:37

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

19 Sep 2020, 13:31

This may be of some use at the end of your Chrome.ahk scripts while debugging, or even in a finished script, with the caveat that it is going to kill conhost.exe.

(You may very well NOT want to kill all conhost.exe instances, so please use with care.)

Edit: Code below may create a crash report on next startup, I've had it happen. But I am still using it, as it appears to work almost all the time.

Code: Select all

; Kill conhost.exe so it doesn't hang around.
; 
DetectHiddenWindows, On
; From GeekDude's tips and tricks: https://www.autohotkey.com/boards/viewtopic.php?f=7&t=7190
Run, cmd,, Hide, PID
WinWait, ahk_pid %PID%
DllCall("AttachConsole", "UInt", PID)

; Run another process that would normally
; make a command prompt pop up, such as
; RunWait, cmd /c dir > PingOutput.txt
Runwait, taskkill /im conhost.exe /f 
; Thanks to @flyingDman post from 2010, 
; https://autohotkey.com/board/topic/49732-kill-process/

; Close the hidden command prompt process
Process, Close, %PID%
; EDIT - removing loops to remove orphaned chrome instances. They generate crash reports.
EDIT: As long as Chrome is not running headless from Chrome.ahk AND remains visible and in focus,

Code: Select all

Send, ^w
works better, apparently guaranteeing a clean shutdown of Chrome.

Regards,
burque505
malcev
Posts: 803
Joined: 12 Aug 2014, 12:37

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

12 Oct 2020, 17:26

Nowhere is said that to run this library successfully should be enabled active scripting in IE.
It can be checked/modified here:
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Zones\3
GeekDude
Posts: 887
Joined: 02 Oct 2013, 22:13

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

14 Oct 2020, 21:43

Unfortunately true. If you mess around with the IE installation too much you can break Chrome.ahk. The websocket library it uses relies on IE/Javascript components to function, so depending on how you lock down, remove, or otherwise configure IE, your mileage may vary. If I ever make it to Chrome.ahk version 2.0 that dependency should go away, as I'll have moved onto a different websocket API.

You may ask why I didn't use the different websocket API to start with. And to that, all I can say is that I wrote the library before Microsoft wrote the documentation for the alternative API.
malcev
Posts: 803
Joined: 12 Aug 2014, 12:37

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

16 Oct 2020, 22:36

I mean that when chrome.ahk starts it can check if javascript is enabled in IE.
And if js is disabled then enable js, coonnect to chrome and then disable js.
Also it will be good to write about it in limitations in the 1 post.
fatodubs
Posts: 4
Joined: 20 Sep 2017, 18:53

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

19 Oct 2020, 09:18

I'm running into the same scripting SecurityError as detailed early on. The stackoverflow page was not helpful. I'm trying to do this on a work computer with the IE settings limited by group policy, and an apparent restriction on connecting to localhost (I can't access http localhost:9222 /json/version Broken Link for safety or the same with 127.0.0.1 in a Chrome window)

Is anyone aware of security risks of adding localhost to the intranet sites? Would that solve the issue?

malcev mentioned enabling javascript. I can apparently access and edit parts of the registry, but I don't understand the specifics enough to play with it.

Anyone have any ideas of other things to look at?
gregster
Posts: 5390
Joined: 30 Sep 2013, 06:48

Re: [Library] Chrome.ahk - Automate Google Chrome using native AutoHotkey. No Selenium!

19 Oct 2020, 09:35

fatodubs wrote:
19 Oct 2020, 09:18
I'm running into the same scripting SecurityError as detailed early on.
Afaik, there is a limit of 6 connections for IE (and IE component) websocket connections - as I don't know your code, I don't know if that's the problem here (but it seems like a typical error message for that scenario). Edit: I guess, there are more reasons for this rather generic error message.

If you are trying to use more than six websocket instances (tabs or Chrome instances), you'll need to edit the registry (at your own risk), afaik, to allow the AutoHotkey.exe process to use more than 6 connections (because that process - via Chrome.ahk - uses an IE component to establish these websockets). You could change it manually, or via AHK script (run as admin, if you get Errorlevel = 1 and A_LastError = 5):

Code: Select all

limit := 10 	; values between 2 and 128 are allowed
RegWrite, REG_DWORD, HKEY_CURRENT_USER\Software\Policies\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_WEBSOCKET_MAXCONNECTIONSPERSERVER, AutoHotkey.exe, % limit
; RegWrite, REG_DWORD, HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_WEBSOCKET_MAXCONNECTIONSPERSERVER, AutoHotkey.exe, % limit
msgbox % "Errorlevel: " errorlevel "`nA_LastError: " A_LastError		; 0 means success
You could change it for the current user or the 'local machine' - but both might be prohibited on a work computer, I suppose.
Usually admin rights are required to create or change these keys (at least for the 'local machine' key).

There will probably be ways around this specific problem when/if GeekDude introduces the new Websocket API, he mentioned above.
Please anyone correct me, if I am wrong.

Edit: It seems you are not able to establish any websocket connection at all...? Then that is an even more fundamental obstacle, I guess, and this won't help you at all. Sorry.
I am not aware of a solution (other than what is outlined in the StackOverflow thread, or by tmplinshi here), but I think there have been questions about the IE websocket limit (of 6) before (so I hope this is at least helpful for other people).

Return to “Scripts and Functions”

Who is online

Users browsing this forum: No registered users and 29 guests