Rufaydium WebDriver 1.7.2 (no selenium/websocket)

Post your working scripts, libraries and tools for AHK v1.1 and older
User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.5.1 (no selenium/websocket)

Post by Xeo786 » 21 May 2022, 06:54

ahk7 wrote:
21 May 2022, 02:01
@Xeo786 cheers for the update
Thanks :D
but I haven't made a release as I haven't tested Opera
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

globalzen
Posts: 2
Joined: 19 Feb 2018, 20:37

Re: Rufaydium WebDriver 1.5.1 (no selenium/websocket)

Post by globalzen » 21 May 2022, 09:47

Can anyone tell me why I get the error message from the Line 3? Line one seems to work ok I can see the chromedriver task running and monitoring the port.

One thing I notice is that I have Chrome Version 101.0.4951.67 (Official Build) (64-bit) while the download is ChromeDriver 101.0.4951.41.

As soon as line 3 tries to exec it gets the errror message shown.
Attachments
Screenshot 2022-05-22 004243.png
Screenshot 2022-05-22 004243.png (26.9 KiB) Viewed 3208 times

burque505
Posts: 1731
Joined: 22 Jan 2017, 19:37

Re: Rufaydium WebDriver 1.5.1 (no selenium/websocket)

Post by burque505 » 21 May 2022, 09:48

@Xeo786, thanks once again for all your work on this extremely useful lib. I have a question about line 117 of v1.5.1:

Code: Select all

			chromeOptions := Se["capabilities",This.driver.options]
What does "Se" refer to?
Regards,
burque505

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.5.1 (no selenium/websocket)

Post by Xeo786 » 21 May 2022, 11:17

globalzen wrote:
21 May 2022, 09:47
Can anyone tell me why I get the error message from the Line 3? Line one seems to work ok I can see the chromedriver task running and monitoring the port.

One thing I notice is that I have Chrome Version 101.0.4951.67 (Official Build) (64-bit) while the download is ChromeDriver 101.0.4951.41.

As soon as line 3 tries to exec it gets the errror message shown.

Code: Select all

; Chrome := new Rufaydium(A_ScriptDir ".\chromedriver.exe") ;path is absolute because of "."
 Chrome := new Rufaydium(A_ScriptDir "\chromedriver.exe") ; would work
 ; but chromedriver.exe is default driver therefore
 Chrome := new Rufaydium() ; will also load chromedriver
Last edited by Xeo786 on 21 May 2022, 11:56, edited 1 time in total.
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.5.1 (no selenium/websocket)

Post by Xeo786 » 21 May 2022, 11:25

burque505 wrote:
21 May 2022, 09:48
@Xeo786, thanks once again for all your work on this extremely useful lib. I have a question about line 117 of v1.5.1:

Code: Select all

			chromeOptions := Se["capabilities",This.driver.options]
What does "Se" refer to?
Regards,
burque505
line 113 returns with all the sessions and their capabilities, and we need to Recreate all sessions again

Code: Select all

Sessions := this.send(this.DriverUrl "/sessions","GET")
I using for loop to get every session id and from capabilities getting chromeOptions.debuggerAddress
now that line 113 is 151 for 1.6.0 I am going update release.
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

User avatar
hotcheesesoup
Posts: 34
Joined: 08 May 2022, 01:41

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by hotcheesesoup » 21 May 2022, 16:34

@globalzen - I think I'm having the same issue after downloading the newest release on a different computer. Chrome is Version 101.0.4951.67 (Official Build) (64-bit), and I don't know if chromedriver.exe is available for this release yet?

https://chromedriver.chromium.org/downloads says:
If you are using Chrome version 101, please download ChromeDriver 101.0.4951.41
Maybe we need to downgrade?

@Xeo786 - What do you think?

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Xeo786 » 21 May 2022, 17:53

hotcheesesoup wrote:
21 May 2022, 16:34
@globalzen - I think I'm having the same issue after downloading the newest release on a different computer. Chrome is Version 101.0.4951.67 (Official Build) (64-bit), and I don't know if chromedriver.exe is available for this release yet?

https://chromedriver.chromium.org/downloads says:
If you are using Chrome version 101, please download ChromeDriver 101.0.4951.41
Maybe we need to downgrade?

@Xeo786 - What do you think?
Chrome driver 101.0.xxxx driver will work for every Chrome browser 101.0.xxxx unless 102 browser arrives
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

User avatar
hotcheesesoup
Posts: 34
Joined: 08 May 2022, 01:41

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by hotcheesesoup » 21 May 2022, 18:46

@Xeo786 - I can't seem to get it to work with Chrome now, even manually adding the chromedriver.exe won't work. Manually adding msedgedriver.exe DID work though.
---------------------------
Rufaydium WebDriver Support
---------------------------
session not created: No matching capabilities found

Please Press Yes to download latest driver
---------------------------
Yes No
---------------------------
This will download chromedriver.exe and replace my manual version, but this error just loops.

User avatar
hotcheesesoup
Posts: 34
Joined: 08 May 2022, 01:41

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by hotcheesesoup » 21 May 2022, 19:00

@Xeo786
@globalzen
Well, I changed nothing and it started working again. No clue what changed.

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Milchmann » 23 May 2022, 02:09

@Xeo786

Please excuse me for being persistent here.
But I still don't get any values returned since the version from 17.05.22 and also with version 1.6.0.
viewtopic.php?f=6&t=102616&start=120#p463055

Code: Select all

Chrome := new Rufaydium()
Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
if !isobject(page)
{
	Page := Chrome.NewSession()
	Page.Navigate("https://www.autohotkey.com/boards/")
}
MsgBox % Page.url
. "`nbody len from Basic "	Page.QuerySelectorAll("body").length()
. "`nbody len from CDP "		Page.cdp.QuerySelectorAll("body").length()
. "`nkeywords len from Basic "	Page.QuerySelectorAll("#keywords").length() ; autohkey search
. "`nkeywords len from CDP "	Page.cdp.QuerySelectorAll("#keywords").length()
return
@ahk7 and @hotcheesesoup Do you get values greater than 0 here ?

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Xeo786 » 23 May 2022, 02:49

Milchmann wrote:
23 May 2022, 02:09
@Xeo786

Please excuse me for being persistent here.
But I still don't get any values returned since the version from 17.05.22 and also with version 1.6.0.
viewtopic.php?f=6&t=102616&start=120#p463055

Code: Select all

Chrome := new Rufaydium()
Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
if !isobject(page)
{
	Page := Chrome.NewSession()
	Page.Navigate("https://www.autohotkey.com/boards/")
}
MsgBox % Page.url
. "`nbody len from Basic "	Page.QuerySelectorAll("body").length()
. "`nbody len from CDP "		Page.cdp.QuerySelectorAll("body").length()
. "`nkeywords len from Basic "	Page.QuerySelectorAll("#keywords").length() ; autohkey search
. "`nkeywords len from CDP "	Page.cdp.QuerySelectorAll("#keywords").length()
return
@ahk7 and @hotcheesesoup Do you get values greater than 0 here ?
Rufaydium working fine.....! just use .count() instead .length() for the time being.
here is the issue
JS array starts with 0 and AHK array starts with 1, Rufaydium return AHK array of elements, to make it more like JS DOM Rufaydium return with element array
starts from zero and thats why this issue starts, just try following examples

Code: Select all

; using AHK Version 1.1.34.02
a := []
msgbox, % "Len:" a.length() "`nKeys:0: " a[0] ; showing Len 0 
a[0] := "a"
msgbox, % "Len:" a.length() "`nKeys:0: " a[0] ; showing Len 0 >>> should be 1  
a[1] := "b"
msgbox, % "Len:" a.length() "`nKeys:`n0: " a[0] "`n1: " a[1] ; ; showing Len 1 >>> should be 2
a[2] := "c"
msgbox, % "Len:" a.length() "`nKeys:`n0: " a[0] "`n1: " a[1] "`n2: " a[2] ; ; showing Len 2 >>> should be 3
return
above behavior must be some bug.
@RaptorX
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Milchmann » 23 May 2022, 04:00

Rufaydium working fine.....! just use .count() instead .length() for the time being.
Works with basic, not cdp.
Is .count now permanent or just an interim solution?
here is the issue
In my opinion it has always been with Autohotkey, start with one. Only Javascript started with 0.
https://www.autohotkey.com/docs/Objects.htm#Usage_Simple_Arrays
I myself, also set A_index-1 in Autohotkey when working with javascript.

Code: Select all

oCells := oRows[A_Index-1].querySelectorAll("td")

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Xeo786 » 23 May 2022, 23:29

Milchmann wrote:
23 May 2022, 04:00
Rufaydium working fine.....! just use .count() instead .length() for the time being.
Works with basic, not cdp.
Is .count now permanent or just an interim solution?
here is the issue
In my opinion it has always been with Autohotkey, start with one. Only Javascript started with 0.
https://www.autohotkey.com/docs/Objects.htm#Usage_Simple_Arrays
I myself, also set A_index-1 in Autohotkey when working with javascript.

Code: Select all

oCells := oRows[A_Index-1].querySelectorAll("td")
If you getting elements not element then .count() should work, please show me the code.
.count() works fine as it counts total index, unlike .length()
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

User avatar
hotcheesesoup
Posts: 34
Joined: 08 May 2022, 01:41

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by hotcheesesoup » 23 May 2022, 23:42

@Milchmann
Sorry, I didn't get the notification for this for some reason.

I've just been using Javascript with Session.CDP.Evaluate() instead of the Rufaydium ones and it's been working well.

I'll test it out tomorrow and let you know!

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Milchmann » 24 May 2022, 01:52

@Xeo786
If you getting elements not element then .count() should work, please show me the code.

Code: Select all

Chrome := new Rufaydium("chromedriver.exe")
Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
if !isobject(page)
{
	Page := Chrome.NewSession()
	Page.Navigate("https://www.autohotkey.com/boards/")
}
Page.CDP.Document()
MsgBox % Page.url
. "`nbody len from Basic "	Page.QuerySelectorAll("body").count()
. "`nbody len from CDP "		Page.cdp.QuerySelectorAll("body").count()
. "`nkeywords len from Basic "	Page.QuerySelectorAll("#keywords").count() ; autohotkey search
. "`nkeywords len from CDP "	Page.cdp.QuerySelectorAll("#keywords").count()

Page.QuerySelector("#keywords").sendkey("ABC/")
Page.cdp.QuerySelector("#keywords").sendkey("CDP") ; error
msgbox %  "`nkeywords value from Basic : " Page.QuerySelector("#keywords").value
. "`nkeywords value from CDP: " Page.cdp.QuerySelector("#keywords").value
. "`nkeywords value from Evaluate: " Page.CDP.Evaluate("document.querySelector('#keywords').value;").value  "`n"
. "`nkeywords outerHTML from CDP : `n" Page.cdp.QuerySelector("#keywords").outerHTML "`n`n"
. "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').outerHTML;").value
Tested with your version from yesterday. Please test yourself. Before 17.05.22 I got the values ( incl. . length() and except basic.value and cdp.value ).

User avatar
Xeo786
Posts: 759
Joined: 09 Nov 2015, 02:43
Location: Karachi, Pakistan

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Xeo786 » 24 May 2022, 02:06

Milchmann wrote:
24 May 2022, 01:52
@Xeo786
If you getting elements not element then .count() should work, please show me the code.

Code: Select all

Chrome := new Rufaydium("chromedriver.exe")
Page := Chrome.getSessionByUrl("https://www.autohotkey.com/boards/")
if !isobject(page)
{
	Page := Chrome.NewSession()
	Page.Navigate("https://www.autohotkey.com/boards/")
}
Page.CDP.Document()
MsgBox % Page.url
. "`nbody len from Basic "	Page.QuerySelectorAll("body").count()
. "`nbody len from CDP "		Page.cdp.QuerySelectorAll("body").count()
. "`nkeywords len from Basic "	Page.QuerySelectorAll("#keywords").count() ; autohotkey search
. "`nkeywords len from CDP "	Page.cdp.QuerySelectorAll("#keywords").count()

Page.QuerySelector("#keywords").sendkey("ABC/")
Page.cdp.QuerySelector("#keywords").sendkey("CDP") ; error
msgbox %  "`nkeywords value from Basic : " Page.QuerySelector("#keywords").value
. "`nkeywords value from CDP: " Page.cdp.QuerySelector("#keywords").value
. "`nkeywords value from Evaluate: " Page.CDP.Evaluate("document.querySelector('#keywords').value;").value  "`n"
. "`nkeywords outerHTML from CDP : `n" Page.cdp.QuerySelector("#keywords").outerHTML "`n`n"
. "`nkeywords outerHTML from Evaluate :  `n" Page.CDP.Evaluate("document.querySelector('#keywords').outerHTML;").value
Tested with your version from yesterday. Please test yourself. Before 17.05.22 I got the values ( incl. . length() and except basic.value and cdp.value ).
Oh CDP was not working just because of this bug
Thanks for pointing out, line 223 in rufaydium.ahk was saying switch this.name it should be switch i.name, I just fixed Main Repo.
"When there is no gravity, there is absolute vacuum and light travel with no time" -Game changer theory

Milchmann
Posts: 112
Joined: 05 Nov 2016, 08:50

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by Milchmann » 24 May 2022, 02:41

@Xeo786 ?
Oh CDP was not working just because of this bug
Thanks for pointing out, line 223 in rufaydium.ahk was saying switch this.name it should be switch i.name, I just fixed Main Repo.
Thanks.


Bert
Last edited by Milchmann on 31 May 2022, 06:02, edited 3 times in total.

User avatar
RaptorX
Posts: 371
Joined: 06 Dec 2014, 14:27
Contact:

Re: Rufaydium WebDriver 1.6.0 (no selenium/websocket)

Post by RaptorX » 24 May 2022, 09:14

Xeo786 wrote:
23 May 2022, 02:49

Code: Select all

; using AHK Version 1.1.34.02
a := []
msgbox, % "Len:" a.length() "`nKeys:0: " a[0] ; showing Len 0 
a[0] := "a"
msgbox, % "Len:" a.length() "`nKeys:0: " a[0] ; showing Len 0 >>> should be 1  
a[1] := "b"
msgbox, % "Len:" a.length() "`nKeys:`n0: " a[0] "`n1: " a[1] ; ; showing Len 1 >>> should be 2
a[2] := "c"
msgbox, % "Len:" a.length() "`nKeys:`n0: " a[0] "`n1: " a[1] "`n2: " a[2] ; ; showing Len 2 >>> should be 3
return
above behavior must be some bug.
@RaptorX
I think this is documented behavior, you can find a bit more in the documentation.
Projects:
AHK-ToolKit

Chance451
Posts: 1
Joined: 24 May 2022, 12:35

Rufaydium is already outdated for most advanced web scraping

Post by Chance451 » 24 May 2022, 12:42

It's very simple, as rufaydium is based upon Selenium it's functionally useless for most advanced sites. When I was attempting to use it to scrape sites that used a little bit of client side javascript it typically failed and got identified. The main issue is that quite simply rufaydium is selenium based (Since basically 2021 even selenium stealth has been busted and is traceable now) if you truly want to be able to engage with most sites the only methods that actually work are Playwright and Puppeteer (I would recommend Playwright python as I know fuck all javascript).

For older sites selenium could be useful but after spending a few days attempting to design selenium in such a way that it would be functionally identical to a human being I've thrown in the towel and retreated to Playwright. Playwright enables much more javascript functionality and so most websites are forced to reduce themselves to advance mouse tracking etc in order to actually find playwright bots (especially ones on firefox). (Google firefox's settings on webdrivers)

In the end of the day whilst I appreciate the work done on rufaydium, if I was you I would already be attempting to make it Playwright compatible as Selenium is basically 2017 technology and is effectively useless for most advanced webscraping of sites that really don't want you there.


[Mod action: Moved this post and the reply to it from its own thread in "Scripts and Functions" to this thread. This post contains no script or function and thus does not meet the criteria for a thread in this sub-forum. It is discussion regarding this thread.]


Post Reply

Return to “Scripts and Functions (v1)”