Can you do web scraping with AHK in the background?

[Nixcalo]

Hi people!

A quick question. Does anybody know of a library, a script or anything that performs web scraping in the background?

I have managed to perform some web scraping with UIA libraries but it requires an active browser window. I can't work on the PC while the script is running (it reads text from there, it gathers data, etc.).

Is it possible, even in principle? If you could point me to a resource, I'll do my best to look it up but I don' t know where to start. I do believe that Chromium. Selenium, etc (not that I know much about those) are not helpful for webscraping in the background... but again, perhaps I am mistaken.


I'd appreciate it! Thank you!

[lmstearn]

Hi.
Take it you've seen this Web Scraping thread?
Otherwise, the first port of call is Wget, had some success with it, find a script using it back here. Since then, Metacognition posted two scripts here and here.
There is also curl, there is an AHK library for it, some of its commands can be replicated in AHK, like here for example.

[Nixcalo]

Thank you for your answer, but the Web Scraping thread seems to be about Internet Explorer and COM, which is dead now. I have tried to use Wget and I get a Unable to establish the SSL connection I can't bypass, and curl .. I have to look onto that, but the last link "curl to ahk" seems to be about Internet Explorer as well, so it's out of the question.


I need something to work in the background with Microsoft Edge... and I need to access a https:// page. I don't know why but wget does not work and UrlDownloadToFile does nothing I need (doesn't seem to create a file).

[lmstearn]

COM is very much alive (as is Internet Options), there's a thread dealing with SSL here.
Otherwise, can't help much then if there is no code posted, it might be something going on with the TLS with the site, - there are other workarounds in answers to this question. (The latest WGet binaries are found at EternallyBored.)