Scraping then pasting multiple variables

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Scraping then pasting multiple variables

04 Jun 2021, 11:36

I want to scrape game information from one or multiple (whatever is simpler/faster) sites, then using it to fill fields on a game collection program (Collectorz Game Collector - It only fetches info from its own database, which seems to lack many games, especially indies).

The approach I came up with (I am pretty new to AHK so, again, if there's a better/easier way to deal with this let me know) is using getElementById commands to grab various parts (game description, url of the trailer on Youtube, developer) from their page on sites such as Steam, igdb.com and https rawg.io (AFAIK these seem to be the best video game database sites), store them as variables then use them to fill the corresponding fields in the Collectorz program. I do use Firefox/Waterfox btw but from what I understand the COM/GetElementById wizardry needs Internet Explorer, so be it.

By researching and adapting code found on the second post of this thread: https://www.autohotkey.com/boards/viewtopic.php?t=68923
I can go as far as opening a specific game STEAM page (this for example: https store.steampowered.com /app/1097200/Twelve_Minutes/), successfully getting the description field then launch a msgbox popup with it.

Breaking down things I know and things I have a problem with:

1. How do I scrape data from any game page rather than "Twelve Minutes" in particular? I suppose a good start would be to have the script reading my clipboard or launch an input box so I type a game title then performing a search on Steam and/or igbd.com etc THEN do the scraping. I don't know how to do that though.

2. Rather than get the description on a messagebox pop up, how do I save it as a variable to be used later and fill the appropriate Collectorz program field? (I know how to use mouse events to move to specific points/fields in the program, I don't know how to store then paste the necessary variable).

3. How do I add more variables? For example, I figured

Code: Select all

pwb.document.getElementById("developers_list").innertext
grabs the name of the developer and I want the script to scrape this too.

4. How do I grab the video url behind the trailer on youtube found here: https www.igdb.com /games/twelve-minutes and store it along the other variables for filling the corresponding trailer field on Collectorz (needs to be a youtube url). For this example, it is https youtu.be/qQ2vsnapBhU btw.

5. Once I grab the necessary info from the sites I suppose I merely have to:

Code: Select all

WinActivate, ahk_exe GameCollector.exe
then use absolute mouse positions, but I am not sure how to recall/paste the variables grabbed earlier and what else I should do to make sure the script does its job without errors. Thank you!
swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Scraping then pasting multiple variables

05 Jun 2021, 06:14

  1. assuming u know how the Clipboard variable and InputBox command work in ahk(if u dont, read the docs), for most search pages u can pass the search params as part of the URL. so string manipulate(StrReplace, RegExReplace) whatever it is u copied from the clipboard or typed into the inputbox into the appropriate format, construct a URL out of it and navigate to that page, eg: then locate the first element of inside the results list and .click() it, this opens that particular game's page, then proceed to scrape whatever u need scraped.
    as to how to do all that? this has very little to do with ahk, and everything to do with javascript. each page will have its own uniquely named and placed elements, so the procedure will vary. best get to know .querySelector() and CSS selector syntax well, since its more flexible than .getElement(s)ByXXX()
  2. with the example uve given:

    Code: Select all

    myVariable := pwb.document.getElementById("developers_list").innertext ; myVariable now contains the string "Luis Antonio"
    learn about variable/function scope in the ahk docs
  3. see point in 1. about webpages varying. for each page, ull have to come up with new ways. use Chrome DevTools to inspect the DOM(or whatever mozilla's built-in equivalent is)
  4. on that page there is one particular div that has a JSON dump of pretty much everything on the page

    Code: Select all

    pwb.document.querySelector("div[data-react-class='GamePageHeader']").getAttribute('data-react-props') ; returns see below
    
    /*
    {"id":16612,"slug":"twelve-minutes","name":"Twelve Minutes","screenshots":[{"title":"Screenshot","image_width":1024,"image_height":506,"id":19839,"cloudinary_id":"kywt4dprxyi4x7iqhkya"},
    {"title":"Screenshot","image_width":1024,"image_height":671,"id":19840,"cloudinary_id":"zkdha2uodde4ozf1w856"},{"title":"Screenshot","image_width":1024,"image_height":505,"id":19841,"cloudinary_id":"lqy0ejak38kvizvsm1gi"},
    {"title":"Screenshot","image_width":1024,"image_height":640,"id":19842,"cloudinary_id":"k3dqr9gk6bn0aqfap9ns"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322754,"cloudinary_id":"sc6x1e"},
    {"title":"Screenshot","image_width":3840,"image_height":2160,"id":322755,"cloudinary_id":"sc6x1f"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322752,"cloudinary_id":"sc6x1c"},
    {"title":"Screenshot","image_width":3840,"image_height":2160,"id":322753,"cloudinary_id":"sc6x1d"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322751,"cloudinary_id":"sc6x1b"},
    {"title":"Screenshot","image_width":3840,"image_height":2160,"id":322750,"cloudinary_id":"sc6x1a"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322749,"cloudinary_id":"sc6x19"}],"artwork":null,"cover":
    {"image_width":680,"image_height":907,"id":74971,"cloudinary_id":"co1luj"},"status":null,"release_date":"2021","release_date_time_ago":null,"edit_url":"/games/twelve-minutes/edit","genres":[{"to_param":"adventure","name":"Adventure"},
    {"to_param":"indie","name":"Indie"},{"to_param":"point-and-click","name":"Point-and-click"},{"to_param":"puzzle","name":"Puzzle"}],"external_reviews":0.0,"external_reviews_count":1,"rating":null,"ratings_count":0,"summary":"\u003cp\u003eA romantic evening with your wife turns into a violent invasion, as a man breaks into your home, accuses your wife of murder and beats you to death. Only for you to wake up and find yourself stuck in a twelve-minute time loop, doomed to relive the same terror again and again.\u003c/p\u003e","websites":[{"url":"https://www.youtube.com/channel/UCoztP3NSW03Kd69KjQEIxeg","site":9},{"url":"https://twitter.com/12minutesgame","site":5},{"url":"http://twelveminutesgame.com/","site":1},
    {"url":"https://www.facebook.com/twelveminutesgame","site":4},{"url":"https://store.steampowered.com/app/1097200","site":13}],"website_links":[{"id":13,"slug":"steam","type":"store","name":"Steam","domains":["steampowered.com"],"icon":"fab fa-steam","svg":"steam"},{"id":17,"slug":"gog","type":"store","name":"GOG","domains":["gog.com"],"svg":"gog"},{"id":16,"slug":"epic","type":"store","name":"Epic","domains":["epicgames.com"],"svg":"epic"},{"id":12,"slug":"android","type":"store","name":"Google Play","domains":["google.com"],"svg":"play-store"},{"id":10,"slug":"iphone","type":"store","name":"App Store (iPhone)","domains":["apple.com"],"icon":"fab fa-apple","svg":"ios-iphone"},{"id":11,"slug":"ipad","type":"store","name":"App Store (iPad)","domains":["apple.com"],"icon":"fab fa-apple","svg":"ios-ipad"},{"id":15,"slug":"itch","type":"store","name":"Itch","domains":["itch.io"],"svg":"itch"},{"id":1,"slug":"official","type":"social","name":"Official Website","icon":"fa fa-external-link-alt","svg":"official-site"},{"id":6,"slug":"twitch","type":"social","name":"Twitch","domains":["twitch.tv"],"svg":"twitch"},{"id":9,"slug":"youtube","type":"social","name":"YouTube","domains":["youtube.com","youtu.be"],"prefix":"/user/","svg":"youtube"},
    {"id":4,"slug":"facebook","type":"social","name":"Facebook","domains":["facebook.com"],"svg":"facebook"},{"id":5,"slug":"twitter","type":"social","name":"Twitter","domains":["twitter.com"],"svg":"twitter"},
    {"id":8,"slug":"instagram","type":"social","name":"Instagram","domains":["instagram.com"],"svg":"instagram"},{"id":18,"slug":"discord","type":"social","name":"Discord","domains":["discord.gg","discordapp.com"],"svg":"discord"},
    {"id":14,"slug":"reddit","type":"social","name":"Subreddit","domains":["reddit.com"],"svg":"reddit"},{"id":2,"slug":"wikia","type":"social","name":"Wikia","icon":"far fa-file-alt","svg":"wikia"},
    {"id":3,"slug":"wikipedia","type":"social","name":"Wikipedia","domains":["wikipedia.org"],"icon":"fab fa-wikipedia-w","svg":"wikipedia"}],"videos":[{"title":"Announcement Trailer","url":"qQ2vsnapBhU","thumb":
    ["//img.youtube.com/vi/qQ2vsnapBhU/0.jpg","//img.youtube.com/vi/qQ2vsnapBhU/1.jpg","//img.youtube.com/vi/qQ2vsnapBhU/2.jpg","//img.youtube.com/vi/qQ2vsnapBhU/3.jpg"],"id":38842},{"title":"Gamescom 2020 Cast Reveal 
    Trailer","url":"E6CmZN9TYdI","thumb":["//img.youtube.com/vi/E6CmZN9TYdI/0.jpg","//img.youtube.com/vi/E6CmZN9TYdI/1.jpg","//img.youtube.com/vi/E6CmZN9TYdI/2.jpg","//img.youtube.com/vi/E6CmZN9TYdI/3.jpg"],"id":40096}],"developers":
    [["Luis Antonio","/companies/luis-antonio"]],"can_rate":false,"rating_path":"/games/16612/rates","rating_delete_url":"/games/16612/rates/delete_rating","new_list_path":"/users/list/new?
    game_id=16612","hype_path":"/games/16612/hypes","hypes_count":4,"changes_path":"/games/twelve-minutes/changes","loader_path":"/assets/ajax-loading-response-
    86bb92c6295652dc51afc6ce33b0074683688bc10f44287f5622e650823f92b9.gif","short_url":"https://www.igdb.com/g/ctg","category":0,"extension":null,"platforms":[["Xbox One","xboxone",49],["PC (Microsoft Windows)","win",6],["Xbox Series","series-
    x",169]],"versions_count":0,"versions_slug":"twelve-minutes"}
    */
    u can parse this string or convert it to an AHK object using some JSON parsing library, then grab the URLs of entries containing "Trailer"
    • if an edit box is given focus in ur target app, u can type something in it

      Code: Select all

      Send % "{Text}" myVariable
    • or u can paste something in it

      Code: Select all

      Clipboard := myVaraible
      Send ^v ; ctrl + v
Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: Scraping then pasting multiple variables

05 Jun 2021, 08:11

swagfag wrote:
05 Jun 2021, 06:14
Thanks a lot for your reply as well as taking the time to answer my questions. I have already started studying some of the variables and commands you mentioned. And while I am looking forward to learn more stuff about AHK because I understand it's such a powerful tool and I am all about "teach a man to learn how to fish", I hope you understand how my task is relatively advanced/complicated for a complete beginner. I am fairly familiar with Inputbox (I kinda learnt using it (on a basic level of course) which led me to GUIs then I've figured I can actually make AHK scrape the info for me, rather than inputting it manually!) but using regex to manipulate strings, "JSON parsing libraries" and Javascript get into DEV territory and it's stuff certainly out of my league. I am sure they might seem simple for you or other experienced users or programmers/DEVS but it'd take ages for me to comprehend all that, especially since I'd have to put them all together to come up with a working script.I mean I've already checked the RegExReplace() section and I can't really follow.
I could be way off, but it appears like you are in the position to do the same in a few minutes therefore I'd really appreciate it if you could do that. Then apart from being able to get AHK to do what I want, I'd also have a reference point to study and adapt on other things/AHK related tasks.
If, however, it's a hard/time consuming task for you to put together, then I wouldn't want you to waste your time. Thanks again for your input! :)
swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Scraping then pasting multiple variables

05 Jun 2021, 14:06

Code: Select all

#NoEnv
#Warn All
#SingleInstance Force
#Requires AutoHotkey v1.1.33+
SetBatchLines -1

igdbSearchURL := "https://www.igdb.com/search?type=1&q="

InputBox gameToSearchFor, type a game name
gameSearchURL := igdbSearchURL . StrReplace(gameToSearchFor, A_Space, "+") ; replace spaces with +. other special characters, ull have to URL encode

IE := ComObjCreate("InternetExplorer.Application")
IE.Visible := true
IE.Navigate(gameSearchURL)

Sleep 4000 ; wait for page to load

IE.document.querySelector("div.block :first-child a").click()

Sleep 4000 ; wait for page to load

gamePageHeaderJSON := IE.document.querySelector("div[data-react-class='GamePageHeader']").getAttribute("data-react-props")
GamePageHeader := JsonToAHK(gamePageHeaderJSON)

ytBaseURL := "https://www.youtube.com/watch?v="
foundTrailers := ""
for each, Video in GamePageHeader.Videos
	if (Video.Title ~= "i)trailer") ; if contains case insensitive "trailer"
		foundTrailers .= Video.Title . ": " . ytBaseURL . Video.URL "`n"

MsgBox % "found the following trailers:`n`n" foundTrailers "`n`nbye"
ExitApp

; by teadrinker
JsonToAHK(json, rec := false) {
   static doc := ComObjCreate("htmlfile")
         , __ := doc.write("<meta http-equiv=""X-UA-Compatible"" content=""IE=9"">")
         , JS := doc.parentWindow
   if !rec
      obj := %A_ThisFunc%(JS.eval("(" . json . ")"), true)
   else if !IsObject(json)
      obj := json
   else if JS.Object.prototype.toString.call(json) == "[object Array]" {
      obj := []
      Loop % json.length
         obj.Push( %A_ThisFunc%(json[A_Index - 1], true) )
   }
   else {
      obj := {}
      keys := JS.Object.keys(json)
      Loop % keys.length {
         k := keys[A_Index - 1]
         obj[k] := %A_ThisFunc%(json[k], true)
      }
   }
   Return obj
}
Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: Scraping then pasting multiple variables

12 Jun 2021, 14:15

swagfag wrote:
05 Jun 2021, 14:06

Code: Select all

#NoEnv
#Warn All
#SingleInstance Force
#Requires AutoHotkey v1.1.33+
SetBatchLines -1

igdbSearchURL := "https www.igdb.com /search?type=1&q="  Broken Link for safety

InputBox gameToSearchFor, type a game name
gameSearchURL := igdbSearchURL . StrReplace(gameToSearchFor, A_Space, "+") ; replace spaces with +. other special characters, ull have to URL encode

IE := ComObjCreate("InternetExplorer.Application")
IE.Visible := true
IE.Navigate(gameSearchURL)

Sleep 4000 ; wait for page to load

IE.document.querySelector("div.block :first-child a").click()

Sleep 4000 ; wait for page to load

gamePageHeaderJSON := IE.document.querySelector("div[data-react-class='GamePageHeader']").getAttribute("data-react-props")
GamePageHeader := JsonToAHK(gamePageHeaderJSON)

ytBaseURL := "https www.youtube.com /watch?v="  Broken Link for safety
foundTrailers := ""
for each, Video in GamePageHeader.Videos
	if (Video.Title ~= "i)trailer") ; if contains case insensitive "trailer"
		foundTrailers .= Video.Title . ": " . ytBaseURL . Video.URL "`n"

MsgBox % "found the following trailers:`n`n" foundTrailers "`n`nbye"
ExitApp

; by teadrinker
JsonToAHK(json, rec := false) {
   static doc := ComObjCreate("htmlfile")
         , __ := doc.write("<meta http-equiv=""X-UA-Compatible"" content=""IE=9"">")
         , JS := doc.parentWindow
   if !rec
      obj := %A_ThisFunc%(JS.eval("(" . json . ")"), true)
   else if !IsObject(json)
      obj := json
   else if JS.Object.prototype.toString.call(json) == "[object Array]" {
      obj := []
      Loop % json.length
         obj.Push( %A_ThisFunc%(json[A_Index - 1], true) )
   }
   else {
      obj := {}
      keys := JS.Object.keys(json)
      Loop % keys.length {
         k := keys[A_Index - 1]
         obj[k] := %A_ThisFunc%(json[k], true)
      }
   }
   Return obj
}
Thanks a lot for your help, I really appreciate it. Instead of keep asking stuff, I wanted to give it some time to study the script in conjunction with your initial reply, test it, adapt things I can adapt, research those I don't get, then ask.

Code: Select all

IE.document.querySelector("div.block :first-child a").click()
I was about to ask you "Why the code in question in this case picks the first result?" then I read your first reply again and I figured it's all about Javascript and getting to know .querySelector() and CSS selector syntax as you've said. Since I guess there is no "one fits all - or even the most" magical trick I could ask you for, are you aware of any browser addon / greasemonkey script / third party program or even AHK script that would allow me to click an element to reveal the code behind? I mean AFAIK and unlike .getElement(s)ByXXX(), this isn't possible by using the browser "inspect" function, is it? And if there isn't any, and because as much as I'd like to, I am not sure learning Javascript at this point in time/life is possible, perhaps you could direct me to an "explain me like I am 5" source that would at least teach me how to figure the code behind each element, to do stuff like this?
on that page there is one particular div that has a JSON dump of pretty much everything on the page/u can parse this string or convert it to an AHK object using some JSON parsing library, then grab the URLs of entries containing "Trailer"
Could you please elaborate by any chance? Do most pages contain a div like that? How do I figure whether that's the case and what is it? And by using the current example, how do I parse the string using AHK to grab the URL behind the "trailer" entry? I did check the related docs and although things like the "replace" function made sense, I still can't get my head around the regex/loop over/parsing function, but I think that, unlike the querySelector() part, an example (with the trailer in our case) would help me to understand, then apply the same rule in other things too.
About the "JSON parsing library", which I suppose it's the "by teadrinker" part, at the end of your script, I am not sure how to use it. I mean, the script searches IGDB for the game title, grabs the url trailer(s) from Youtube then after the text box launches, I don't see how it utilizes that part of the code?
swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Scraping then pasting multiple variables

12 Jun 2021, 15:26

Why the code in question in this case picks the first result?
because i just so decided it ought to do that. the search page list a bunch of results https://www.igdb.com/search?type=1&q=gta+5. chances are the first one is gonna be what ure after. feel free to come up with more robust heuristics
the selector reads: "look for a div with the className 'block', get its first child, look into all of first child's descendants until u find the first a, return it"
image.png
image.png (220.67 KiB) Viewed 1539 times
any browser addon / greasemonkey script / third party program or even AHK script that would allow me to click an element to reveal the code behind?
yes, see:
i wrote:use Chrome DevTools to inspect the DOM(or whatever mozilla's built-in equivalent is)
"explain me like I am 5" source that would at least teach me how to figure the code behind each element, to do stuff like this?
https://www.w3.org/TR/selectors-3/
Do most pages contain a div like that?
probably not. everyone is free to put whatever they like in whichever place they like on their webpages
How do I figure whether that's the case and what is it?
look at the page source with, u guessed it, "Chrome DevTools(or whatever mozilla's built-in equivalent is)". see if u can find something interesting, usable
how do I parse the string using AHK to grab the URL behind the "trailer" entry?
thats too broad a question. u learn the tools u have available to u(ie string functions), u learn what they do, how to call them, what to call them with, what they return. u look at the string, u identify which part of it u need. then u figure out a way of combining said functions to achieve that goal. eg if u know u always want to ignore the first 5 chars of a string, find out what function allows u to do that and call it.
I am not sure how to use it. I don't see how it utilizes that part of the code?
its a function. u call it with some arguments(a JSON-formatted string), u get something back from it(a representation of ur JSON-encoded string as a normal AHK object, u can interact with using normal AHK facilities, eg pass it to for-loops, member access dot operator, etc)

Code: Select all

GamePageHeader := JsonToAHK(gamePageHeaderJSON)
Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: Scraping then pasting multiple variables

13 Jun 2021, 14:53

swagfag wrote:
12 Jun 2021, 15:26
Why the code in question in this case picks the first result?
because i just so decided it ought to do that. the search page list a bunch of results https www.igdb.com /search?type=1&q=gta+5. chances are the first one is gonna be what ure after?
No, no, I got that. I wasn't trying to figure why you were after the first result, but how did you come up with the code you did to pick it. Which you pretty much cover with your next line, as well as the screencap. Of course it doesn't mean this example will apply everywhere, but seeing the logic behind the code is useful nonetheless. Cheers.

how do I parse the string using AHK to grab the URL behind the "trailer" entry?
thats too broad a question. u learn the tools u have available to u(ie string functions), u learn what they do, how to call them, what to call them with, what they return. u look at the string, u identify which part of it u need. then u figure out a way of combining said functions to achieve that goal.
I do get what you are saying and well, technically, it's the truth and I agree. The thing is, as you can understand, you can apply this logic to pretty much anything (without that necessarily offering much to the uninitiated...). I understand it might be crystal clear to you, since it's obviously your area of expertise, but to use an analogy, in theory, all you need to write a song from scratch, record it and mix it, as well as flying a plane is to learn the tools you have available and properly utilize them. But just the principle alone, doesn't do much to help us becoming musicians or pilots. Furthermore, in our case, I am not exactly looking to score a #1 hit or become a pilot.
I mean, when it comes to this particular question, I really think I was quite specific. I've talked about a very specific string (one you told me about), asking how to grab a certain thing (the trailer). And that's because, rather than let's say asking "How I do make AHK parse anything in any page", which would be very abstract indeed, I thought asking for a specific example would maybe, hopefully, help me get a better understanding of the process.
swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Scraping then pasting multiple variables

13 Jun 2021, 16:35

the point is, ure asking how to compose a song, when u dont yet know how to pluck the strings(no pun intended lol). a musician's gotta practice
I've talked about a very specific string (one you told me about), asking how to grab a certain thing (the trailer).
many ways to skin a cat. heres one using regex, good luck

Code: Select all

JSONString = {"id":16612,"slug":"twelve-minutes","name":"Twelve Minutes","screenshots":[{"title":"Screenshot","image_width":1024,"image_height":506,"id":19839,"cloudinary_id":"kywt4dprxyi4x7iqhkya"},{"title":"Screenshot","image_width":1024,"image_height":671,"id":19840,"cloudinary_id":"zkdha2uodde4ozf1w856"},{"title":"Screenshot","image_width":1024,"image_height":505,"id":19841,"cloudinary_id":"lqy0ejak38kvizvsm1gi"},{"title":"Screenshot","image_width":1024,"image_height":640,"id":19842,"cloudinary_id":"k3dqr9gk6bn0aqfap9ns"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322754,"cloudinary_id":"sc6x1e"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322755,"cloudinary_id":"sc6x1f"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322752,"cloudinary_id":"sc6x1c"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322753,"cloudinary_id":"sc6x1d"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322751,"cloudinary_id":"sc6x1b"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322750,"cloudinary_id":"sc6x1a"},{"title":"Screenshot","image_width":3840,"image_height":2160,"id":322749,"cloudinary_id":"sc6x19"}],"artwork":null,"cover":{"image_width":680,"image_height":907,"id":74971,"cloudinary_id":"co1luj"},"status":null,"release_date":"Aug 19, 2021","release_date_time_ago":1629331200000,"edit_url":"/games/twelve-minutes/edit","genres":[{"to_param":"adventure","name":"Adventure"},{"to_param":"indie","name":"Indie"},{"to_param":"point-and-click","name":"Point-and-click"},{"to_param":"puzzle","name":"Puzzle"}],"external_reviews":0.0,"external_reviews_count":1,"rating":null,"ratings_count":0,"summary":"\u003cp\u003eA romantic evening with your wife turns into a violent invasion, as a man breaks into your home, accuses your wife of murder and beats you to death. Only for you to wake up and find yourself stuck in a twelve-minute time loop, doomed to relive the same terror again and again.\u003c/p\u003e","websites":[{"url":"https://www.youtube.com/channel/UCoztP3NSW03Kd69KjQEIxeg","site":9},{"url":"https://twitter.com/12minutesgame","site":5},{"url":"http://twelveminutesgame.com/","site":1},{"url":"https://www.facebook.com/twelveminutesgame","site":4},{"url":"https://store.steampowered.com/app/1097200","site":13}],"website_links":[{"id":13,"slug":"steam","type":"store","name":"Steam","domains":["steampowered.com"],"icon":"fab fa-steam","svg":"steam"},{"id":17,"slug":"gog","type":"store","name":"GOG","domains":["gog.com"],"svg":"gog"},{"id":16,"slug":"epic","type":"store","name":"Epic","domains":["epicgames.com"],"svg":"epic"},{"id":12,"slug":"android","type":"store","name":"Google Play","domains":["google.com"],"svg":"play-store"},{"id":10,"slug":"iphone","type":"store","name":"App Store (iPhone)","domains":["apple.com"],"icon":"fab fa-apple","svg":"ios-iphone"},{"id":11,"slug":"ipad","type":"store","name":"App Store (iPad)","domains":["apple.com"],"icon":"fab fa-apple","svg":"ios-ipad"},{"id":15,"slug":"itch","type":"store","name":"Itch","domains":["itch.io"],"svg":"itch"},{"id":1,"slug":"official","type":"social","name":"Official Website","icon":"fa fa-external-link-alt","svg":"official-site"},{"id":6,"slug":"twitch","type":"social","name":"Twitch","domains":["twitch.tv"],"svg":"twitch"},{"id":9,"slug":"youtube","type":"social","name":"YouTube","domains":["youtube.com","youtu.be"],"prefix":"/user/","svg":"youtube"},{"id":4,"slug":"facebook","type":"social","name":"Facebook","domains":["facebook.com"],"svg":"facebook"},{"id":5,"slug":"twitter","type":"social","name":"Twitter","domains":["twitter.com"],"svg":"twitter"},{"id":8,"slug":"instagram","type":"social","name":"Instagram","domains":["instagram.com"],"svg":"instagram"},{"id":18,"slug":"discord","type":"social","name":"Discord","domains":["discord.gg","discordapp.com"],"svg":"discord"},{"id":14,"slug":"reddit","type":"social","name":"Subreddit","domains":["reddit.com"],"svg":"reddit"},{"id":2,"slug":"wikia","type":"social","name":"Wikia","icon":"far fa-file-alt","svg":"wikia"},{"id":3,"slug":"wikipedia","type":"social","name":"Wikipedia","domains":["wikipedia.org"],"icon":"fab fa-wikipedia-w","svg":"wikipedia"}],"videos":[{"title":"Release Date Trailer","url":"G6I2rP3-B9k","thumb":["//img.youtube.com/vi/G6I2rP3-B9k/0.jpg","//img.youtube.com/vi/G6I2rP3-B9k/1.jpg","//img.youtube.com/vi/G6I2rP3-B9k/2.jpg","//img.youtube.com/vi/G6I2rP3-B9k/3.jpg"],"id":50896},{"title":"Gameplay Trailer","url":"1siS7ilp8gE","thumb":["//img.youtube.com/vi/1siS7ilp8gE/0.jpg","//img.youtube.com/vi/1siS7ilp8gE/1.jpg","//img.youtube.com/vi/1siS7ilp8gE/2.jpg","//img.youtube.com/vi/1siS7ilp8gE/3.jpg"],"id":50690},{"title":"Announcement Trailer","url":"qQ2vsnapBhU","thumb":["//img.youtube.com/vi/qQ2vsnapBhU/0.jpg","//img.youtube.com/vi/qQ2vsnapBhU/1.jpg","//img.youtube.com/vi/qQ2vsnapBhU/2.jpg","//img.youtube.com/vi/qQ2vsnapBhU/3.jpg"],"id":38842},{"title":"Gamescom 2020 Cast Reveal Trailer","url":"E6CmZN9TYdI","thumb":["//img.youtube.com/vi/E6CmZN9TYdI/0.jpg","//img.youtube.com/vi/E6CmZN9TYdI/1.jpg","//img.youtube.com/vi/E6CmZN9TYdI/2.jpg","//img.youtube.com/vi/E6CmZN9TYdI/3.jpg"],"id":40096}],"developers":[["Luis Antonio","/companies/luis-antonio"]],"can_rate":false,"rating_path":"/games/16612/rates","rating_delete_url":"/games/16612/rates/delete_rating","new_list_path":"/users/list/new?game_id=16612","hype_path":"/games/16612/hypes","hypes_count":4,"changes_path":"/games/twelve-minutes/changes","loader_path":"/assets/ajax-loading-response-86bb92c6295652dc51afc6ce33b0074683688bc10f44287f5622e650823f92b9.gif","short_url":"https://www.igdb.com/g/ctg","category":0,"extension":null,"platforms":[["Xbox One","xboxone",49],["PC (Microsoft Windows)","win",6],["Xbox Series","series-x",169]],"versions_count":0,"versions_slug":"twelve-minutes"}

; are there videos at all? if so, where, at which character?
if (pos := RegExMatch(JSONString, """videos"":\[\{"))
;        matches the literal string "videos":[{     note the need for both AHK and PCRE character escaping
{
	; AHK's RegExMatch can only match ONCE, then it stops. because of that we need to use a loop 
	while pos := RegExMatch(JSONString, """title"":""(?P<Title>.*?)"",""url"":""(?P<URL>[^""]+)""", Match, pos)
;                     matches the pattern "title":"whatever-the-title-is","url":"whatever-the-url-is"
;                     u aint learning the whole of regex from a comment, so why bother explaining...
	{
		MsgBox % "overall match: " Match "`ntitle: " MatchTitle "`nURL: " MatchURL 

		; advance forward as many chars as there are in whatever had been matched
		; to skip past it and continue searching for more strings
		pos += StrLen(Match)
	}
}
Epoch
Posts: 41
Joined: 04 Jun 2021, 11:09

Re: Scraping then pasting multiple variables

14 Jun 2021, 08:37

swagfag wrote:
13 Jun 2021, 16:35
the point is, ure asking how to compose a song, when u dont yet know how to pluck the strings(no pun intended lol). a musician's gotta practice
You do have a point there. I guess you could say I am trying to buildup calluses, however, since I need to play some chords right now, I also use the quick & dirty superglue trick... Oh well, being a musician using a real life example, I definitely agree about practicing. No previous pilot experience though, maybe I should start posting here: https www.airlinepilotforums.com /career-questions/118827-how-become-airline-pilot.html
many ways to skin a cat. heres one using regex, good luck
Thank you. However, this only works for the page of the specific game. My understanding was that a similar JSONString wold probably exist for every game page in the site, (I mean we haven't checked of course, but chances are that's the case). So the idea was to have AHK using variables which would work for all of them (again, if they use a similar JSONstring scheme of course), maybe by grabbing the JSONstring in question beforehand, if it has to. Because, as you can imagine, writing manually a script for each game (page), pretty much cancels the whole point of automating the task in the first place, copy pasting the necessary info would be quicker.
swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Scraping then pasting multiple variables

14 Jun 2021, 09:04

theyre probably is a similar json string for every game page on that particular game site. it might not have the same format as the one in our example(eg some attributes may be missing, some additional ones might have been added, there might be other edge cases) but thats something for someone willing to research to figure out, and im not that guy
theres nothing hardcoded about the regex pattern. if u apply it to other similarly formatted strings, it will yield titles and URLs. i guess the only thing u can consider hardcoded about it is the pattern itself, which has to be, since thats what encodes the instructions regarding how to interpret the string

Return to “Ask for Help (v1)”

Who is online

Users browsing this forum: No registered users and 241 guests