Rufaydium WebDriver 1.7.2 (no selenium/websocket)

Post your working scripts, libraries and tools for AHK v1.1 and older
ahk7
Posts: 574
Joined: 06 Nov 2013, 16:35

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by ahk7 » 24 May 2022, 14:42

What could be the cause for a script to work with Rufaydium 1.6.0 on Windows 7 32bit + 32bit chrome (both 101.0.4951.67 and 102.0.5005.63 after updating) but fail under Windows 10, 64bit?

The script signs into a CMS (works for both win7/10), after that it continues to select a menu option which works under Win7, but under win10 it seems to loose the session and become slow/unresponsive and when I do a MsgBox % IsObject(MyPage) the answer is 0 under win10 but 1 under win7.
(no point in posting the script as it works under win7 so as such is correct).

When I set Chrome.Driver.visible:=true I do see an error along the lines of "can't create c:\program files\google\updater" or something like that (forgot to copy the text and I'm not in win10 at the moment)

User avatar
hotcheesesoup
Posts: 34
Joined: 08 May 2022, 01:41

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by hotcheesesoup » 25 May 2022, 00:19

@Milchmann - I tested this out with just normal Javascript and got values. I like doing it this way since I guess I'm used to it after messing with Chrome.ahk so much.

Code: Select all

Chrome := new Rufaydium("chromedriver.exe")
Session := Chrome.getSession(1)
Session.CDP.Document()
MsgBox, % Session.CDP.Evaluate("document.getElementsByTagName('body').length").Value 	; This returned 1.
MsgBox, % Session.CDP.Evaluate("document.querySelectorAll('body').length").Value 		; This returned 1.

MsgBox, % Session.CDP.Evaluate("document.getElementById('keywords').name").Value 		; This returned "keywords."
MsgBox, % Session.CDP.Evaluate("document.querySelectorAll('#keywords')[0].name").Value 	; This returned "keywords."

MsgBox, % Session.CDP.Evaluate("document.querySelectorAll('#keywords').length").Value	; This returned 1.
Hope that helps!
-HCS

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Milchmann » 25 May 2022, 06:19

@hotcheesesoup ,
thanks

MoUse_G
Posts: 10
Joined: 07 Oct 2021, 00:52

Re: Rufaydium WebDriver 1.5.1 (no selenium/websocket)

Post by MoUse_G » 26 May 2022, 09:06

globalzen wrote:
21 May 2022, 09:47
Can anyone tell me why I get the error message from the Line 3? Line one seems to work ok I can see the chromedriver task running and monitoring the port.

One thing I notice is that I have Chrome Version 101.0.4951.67 (Official Build) (64-bit) while the download is ChromeDriver 101.0.4951.41.

As soon as line 3 tries to exec it gets the errror message shown.
Close the running webdriver.

ffalcon
Posts: 2
Joined: 26 May 2022, 18:57

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by ffalcon » 26 May 2022, 19:02

This is fantastic thank you Xeo786 for this!

How can I wait for an element to be clickable?
In Selenium I use:

Code: Select all

element = WebDriverWait(driver, delay).until(ec.element_to_be_clickable((by.ID, my_id))
element.click()
Is there an equivalent in Rufaydium? I haven't found it. Thanks!

User avatar
hotcheesesoup
Posts: 34
Joined: 08 May 2022, 01:41

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by hotcheesesoup » 26 May 2022, 22:28

@ffalcon
What do you mean by "clickable"? I haven't used Selenium so that might be something specific to that. Do you mean a disabled element like a grayed out button that will be enabled so you can click it? Maybe you're just waiting for an element to exist so you can click it?

I've had good success using timers:

Code: Select all

#Persistent
#Include Rufaydium.ahk

Chrome := new Rufaydium("chromedriver.exe")
Session := Chrome.getSession(1)
Session.CDP.Document()
SetTimer, WaitForButton, 1000
return

WaitForButton:
; I'm just waiting for the button to become enabled so I can click it.
temp := Session.CDP.Evaluate("document.getElementById('testbutton').disabled").Value
if (temp == 0)
{
	MsgBox, Button is enabled! Let's click it!
	Session.CDP.Evaluate("document.getElementById('testbutton').click()")
}
return
You could also try using a short loop:

Code: Select all

F1::
Loop 60
{
	temp := Session.CDP.Evaluate("document.getElementById('testbutton').disabled").Value 	; This returned 1.
	if (temp == 0)
	{
		MsgBox, Button is enabled! Let's click it!
		Session.CDP.Evaluate("document.getElementById('testbutton').click()")
		break
	}
	else
		Sleep 1000
}
return
I'm not sure if this is what you're looking for, but maybe it will point you in the right direction!
-HCS

ffalcon
Posts: 2
Joined: 26 May 2022, 18:57

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by ffalcon » 27 May 2022, 11:19

hotcheesesoup wrote:
26 May 2022, 22:28
@ffalcon
What do you mean by "clickable"? I haven't used Selenium so that might be something specific to that. Do you mean a disabled element like a grayed out button that will be enabled so you can click it? Maybe you're just waiting for an element to exist so you can click it?
Thanks HCS! I was able to get it working with your input plus some stackoverflow help with the javascript.
Clickable seems to mean visible and not disabled.

This is what's working for me:

Code: Select all

clickable_js_code = document.querySelector(%myselector%).offsetWidth > 0 && document.querySelector(%myselector%).offsetHeight > 0 && !document.querySelector(%myselector%).disabled;

Loop {
		clickable := Session.CDP.Evaluate(clickable_js_code).value	
		if clickable = 1
			break
		else
			sleep 1000
}
msgbox, done

User avatar
hotcheesesoup
Posts: 34
Joined: 08 May 2022, 01:41

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by hotcheesesoup » 27 May 2022, 13:53

@ffalcon - Glad I could help!

docterry
Posts: 4
Joined: 13 Jun 2014, 15:51

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by docterry » 31 May 2022, 00:11

Thanks for this Xeo786. This gives a breath of life to scripts that depended on IE COM with the impending end-of-life for IE.

I have an AHK script which downloads recent files from a webpage list using session.element[0].click(). It then renames that file and copies to a share folder.

Is there a way for the script to determine the users downloaded files path? Or to change the download path just for that session (I wouldn't want to change the users personal settings in Chrome/Edge).

Thanks!

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Xeo786 » 31 May 2022, 02:58

docterry wrote:
31 May 2022, 00:11
Thanks for this Xeo786. This gives a breath of life to scripts that depended on IE COM with the impending end-of-life for IE.

I have an AHK script which downloads recent files from a webpage list using session.element[0].click(). It then renames that file and copies to a share folder.

Is there a way for the script to determine the users downloaded files path? Or to change the download path just for that session (I wouldn't want to change the users personal settings in Chrome/Edge).

Thanks!
You can use following code, to change default download path at any point after creating session. Make sure to use / instead \ for download path

Code: Select all

;here Page is session 
Page.CDP.Call("Browser.setDownloadBehavior", { "behavior" : "allow", "downloadPath" : "D:/newpath"}) 
for edge you can simply use MsEdgedriver and setuserprofile() method from Session.Capabilities look into readme.md on git
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Xeo786 » 31 May 2022, 03:12

ffalcon wrote:
26 May 2022, 19:02
This is fantastic thank you Xeo786 for this!

How can I wait for an element to be clickable?
In Selenium I use:

Code: Select all

element = WebDriverWait(driver, delay).until(ec.element_to_be_clickable((by.ID, my_id))
element.click()
Is there an equivalent in Rufaydium? I haven't found it. Thanks!
element.enabled() , element.Displayed() can be utilized to wait

Code: Select all

while !element.enabled() 
	sleep, 200
element.click()
if that does not works, please share Html of that element,
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Milchmann » 31 May 2022, 06:01

@Xeo786

What else stands out. If I restart the script with newsession, I get no values with page.cdp. With evulate it worked. Then with an existing session with getSessionByUrl it works with both variants with outerhtml .

Code: Select all

. "`nkeywords outerHTML from CDP : `n" Page.cdp.QuerySelector("#keywords").outerHTML "`n`n"
. "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').outerHTML;").value

Is this also an error?

And on we go.

@Xeo786 wrote:
page.CDP.Evaluate(js) simply evaluate JS is same as page.ExecuteSync()
for example both method do same thing with different approach using CDP
Why do I get different values here?

Code: Select all

Chrome := new Rufaydium("chromedriver.exe")
Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
if !isobject(page)
{
	Page := Chrome.NewSession()
	Page.Navigate("https://www.autohotkey.com/boards/")
}
Page.QuerySelector("#keywords").sendkey("ABC/")

msgbox %  "`nkeywords value from Basic : " Page.QuerySelector("#keywords").value
. "`nkeywords value from CDP: " Page.cdp.QuerySelector("#keywords").value
. "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').value;").value
And for me the rightclick into an element doesn't work, even better would be a command for the content menu.
Example right click in QuerySelector("#keywords")
grafik.png
grafik.png (10.74 KiB) Viewed 3096 times
Can you show me an example of how to access the contentmenu?

thanks

Thanks

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Xeo786 » 01 Jun 2022, 02:20

Milchmann wrote:
31 May 2022, 06:01
What else stands out. If I restart the script with newsession, I get no values with page.cdp. With evulate it worked. Then with an existing session with getSessionByUrl it works with both variants with outerhtml .

Code: Select all

. "`nkeywords outerHTML from CDP : `n" Page.cdp.QuerySelector("#keywords").outerHTML "`n`n"
. "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').outerHTML;").value
Is this also an error?
Both following methods are working as intended, I find no issue.

Code: Select all

Page.cdp.QuerySelector("#keywords").outerHTML
Page.CDP.Evaluate("document.querySelector('#keywords').outerHTML;").value
Milchmann wrote:
31 May 2022, 06:01
And for me the rightclick into an element doesn't work, even better would be a command for the content menu.
Example right click in QuerySelector("#keywords")
grafik.png
Can you show me an example of how to access the contentmenu?
Sending right click will not bring context menu, page will realized right click and if there is some JS event listener that waiting for right click might have trigger some programmed action, but context menu is not Rufaydium thing.

with Rufaydium, you can get Window Coordinates and you can also get element Coordinates use them to calculate coordinate on screen and send ahk's right click there to get context menu

and with CDP.Evaluate you can use JS to calculate screen coordinates where you want context menu send AHK's right click

Code: Select all

x :=  Page.CDP.Evaluate("document.querySelector('#keywords').clientHeight").value
offsetx :=  Page.CDP.Evaluate("document.querySelector('#keywords').offsetHeight").value
windowX :=  Page.CDP.Evaluate("window.screenX").value
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium is already outdated for most advanced web scraping

Post by Xeo786 » 01 Jun 2022, 04:17

Chance451 wrote:
24 May 2022, 12:42
It's very simple, as rufaydium is based upon Selenium
btw we are not using selenium, webdriver is written by browsers makers, webdriver just creates the HTTP server, which can be used by any driver handler and Rufaydium doing the same, communicating with webdriver. there are plenty of other platforms using webdrivers Appium, Webdriver.io, Postman, Protractor, Robot Framework etc idk how many out there...
Chance451 wrote:
24 May 2022, 12:42
it's functionally useless for most advanced sites. When I was attempting to use it to scrape sites that used a little bit of client side javascript it typically failed and got identified.
I am curious what website you are referring to
this webpage can be used to detect webdriver with following method one can bypass webdriver detection,

Code: Select all

Chrome := new Rufaydium()
Chrome.Capabilities.addArg("--disable-blink-features=AutomationControlled") ; just disable this line webpage will detect webdriver
Page := Chrome.NewSession()
Page.Navigate("https://bot.sannysoft.com/")
from here we can use Page.CDP.methods() to bypass further detection, if webpage do not detect basic methods than we can continue with Page.methods()
here selenium detection parameters on bot.sannysoft
Image
Chance451 wrote:
24 May 2022, 12:42
In the end of the day whilst I appreciate the work done on rufaydium
Thank You
Chance451 wrote:
24 May 2022, 12:42
if I was you I would already be attempting to make it Playwright compatible as Selenium is basically 2017 technology and is effectively useless for most advanced webscraping of sites that really don't want you there.
Rufaydium covers both sides Basic + CDP. Everything can be done with Webdriver + Devtools protocls can be done with AHK using Rufaydium. do you know what is the best part for this Rufaydium? You just need to install AHK, nothing else.
Last edited by Xeo786 on 31 Jan 2023, 02:46, edited 1 time in total.
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Milchmann » 01 Jun 2022, 04:59

Both following methods are working as intended, I find no issue.
really, i don't understand.
Chrome version 101.0.4951.67 and Chrome driver from 04/19/22.
Display when restarting Chromedriver
grafik.png
grafik.png (5.85 KiB) Viewed 2988 times
and
grafik.png
grafik.png (8.19 KiB) Viewed 2988 times


and then with

Code: Select all

Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
grafik.png
grafik.png (3.98 KiB) Viewed 2988 times
And I don't get any values for the 2nd point either

Code: Select all

msgbox %  "`nkeywords value from Basic : " Page.QuerySelector("#keywords").value
. "`nkeywords value from CDP: " Page.cdp.QuerySelector("#keywords").value
but with

Code: Select all

msgbox % "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').value;").value
the value is there.

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Xeo786 » 01 Jun 2022, 06:23

Milchmann wrote:
01 Jun 2022, 04:59
Both following methods are working as intended, I find no issue.
really, i don't understand.
Chrome version 101.0.4951.67 and Chrome driver from 04/19/22.
Display when restarting Chromedriver
grafik.png and grafik.png

and then with

Code: Select all

Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
grafik.png

And I don't get any values for the 2nd point either

Code: Select all

msgbox %  "`nkeywords value from Basic : " Page.QuerySelector("#keywords").value
. "`nkeywords value from CDP: " Page.cdp.QuerySelector("#keywords").value
but with

Code: Select all

msgbox % "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').value;").value
the value is there.
You might be using old release, I fixed few issues in latest release including this CDP issue
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.1 (no selenium/websocket)

Post by Milchmann » 01 Jun 2022, 11:00

You might be using old release, I fixed few issues in latest release including this CDP issue
No, I have the latest release ( Added notes for incognito mode ) . And the two errors are still there.
I quit Chrome and Rufaydium completely and then restart. Then I don't get any values from cdp ( Value 0 ) on the first start. Only when I access the existing page for the second time do I get the correct values from cdp. I don't get any values for basic and cdp when I query the values for #keywords. I only get this with driver.CDP.Evaluate . Why?
Please try it on you. Thanks

Code: Select all

Chrome := new Rufaydium("chromedriver.exe")
Chrome.Driver.visible := true
Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
if !isobject(page)
{
	Page := Chrome.NewSession()
	Page.Navigate("https://www.autohotkey.com/boards/")
}

MsgBox % Page.url
. "`nbody len from Basic "	Page.QuerySelectorAll("body").Count()
. "`nbody len from CDP "		Page.cdp.QuerySelectorAll("body").Count()
. "`nkeywords len from Basic "	Page.QuerySelectorAll("#keywords").count() ; autohotkey search
. "`nkeywords len from CDP "	Page.cdp.QuerySelectorAll("#keywords").count()
. "`nkeywords len from Evaluate :  `n" Page.CDP.Evaluate("document.querySelectorAll('#keywords').length;").value


Page.QuerySelector("#keywords").sendkey("ABC/")
Page.cdp.QuerySelector("#keywords").sendkey("CDP")
msgbox %  "`nkeywords value from Basic : " Page.QuerySelector("#keywords").value
. "`nkeywords value from CDP: " Page.cdp.QuerySelector("#keywords").value
. "`nkeywords value from Evaluate: " Page.CDP.Evaluate("document.querySelector('#keywords').value;").value  "`n"
. "`nkeywords outerHTML from CDP : `n" Page.cdp.QuerySelector("#keywords").outerHTML "`n`n"
. "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').outerHTML;").value
 
 

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.1 (no selenium/websocket)

Post by Xeo786 » 01 Jun 2022, 13:52

Milchmann wrote:
01 Jun 2022, 11:00
You might be using old release, I fixed few issues in latest release including this CDP issue
No, I have the latest release ( Added notes for incognito mode ) . And the two errors are still there.
I quit Chrome and Rufaydium completely and then restart. Then I don't get any values from cdp ( Value 0 ) on the first start. Only when I access the existing page for the second time do I get the correct values from cdp. I don't get any values for basic and cdp when I query the values for #keywords. I only get this with driver.CDP.Evaluate . Why?
Please try it on you. Thanks

Code: Select all

Chrome := new Rufaydium("chromedriver.exe")
Chrome.Driver.visible := true
Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
if !isobject(page)
{
	Page := Chrome.NewSession()
	Page.Navigate("https://www.autohotkey.com/boards/")
}

MsgBox % Page.url
. "`nbody len from Basic "	Page.QuerySelectorAll("body").Count()
. "`nbody len from CDP "		Page.cdp.QuerySelectorAll("body").Count()
. "`nkeywords len from Basic "	Page.QuerySelectorAll("#keywords").count() ; autohotkey search
. "`nkeywords len from CDP "	Page.cdp.QuerySelectorAll("#keywords").count()
. "`nkeywords len from Evaluate :  `n" Page.CDP.Evaluate("document.querySelectorAll('#keywords').length;").value


Page.QuerySelector("#keywords").sendkey("ABC/")
Page.cdp.QuerySelector("#keywords").sendkey("CDP")
msgbox %  "`nkeywords value from Basic : " Page.QuerySelector("#keywords").value
. "`nkeywords value from CDP: " Page.cdp.QuerySelector("#keywords").value
. "`nkeywords value from Evaluate: " Page.CDP.Evaluate("document.querySelector('#keywords').value;").value  "`n"
. "`nkeywords outerHTML from CDP : `n" Page.cdp.QuerySelector("#keywords").outerHTML "`n`n"
. "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').outerHTML;").value
 
 
Everything works fine form me .. I just want to make sure ... Driver.exe runs and creates http server, I wonder what if driver takes time to create http server and Rufaydium creates newsession without driver properly created server .. 🤔 then it should give winhttp error, I am clueless why It does not work 1st time for you.
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.1 (no selenium/websocket)

Post by Milchmann » 01 Jun 2022, 15:12

I have now tried this with 2 different computers, same effect.
Regardless of point 1, why do I not get any values from #keywords, only with

Code: Select all

age.CDP.Evaluate("document.querySelector('#keywords').value;").value
Can this have something to do with :
viewtopic.php?f=6&t=102616&start=140#p463693 have to do with it?

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.1 (no selenium/websocket)

Post by Xeo786 » 02 Jun 2022, 00:29

Milchmann wrote:
01 Jun 2022, 15:12
I have now tried this with 2 different computers, same effect.
Regardless of point 1, why do I not get any values from #keywords, only with

Code: Select all

age.CDP.Evaluate("document.querySelector('#keywords').value;").value
Can this have something to do with :
viewtopic.php?f=6&t=102616&start=140#p463693 have to do with it?
Try look into html if element HTML has Value attribute then it should bring it to you. if there is no attribute.
Last edited by Xeo786 on 02 Jun 2022, 01:27, edited 1 time in total.
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

Post Reply

Return to “Scripts and Functions (v1)”