Download an online folder and its contents

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
Jose Hidalgo
Posts: 222
Joined: 07 Mar 2021, 07:44

Download an online folder and its contents

Post by Jose Hidalgo » 25 Jul 2021, 14:41

Hi all,

I know how to download an online file with UrlDownloadToFile.
How could I download a whole online folder and all its files ?
Just in case, no need to be recursive on various hierarchy levels: in my case it's an online folder with a variable number of .txt files in it, depending on the folder.

Thanks.

Jose Hidalgo
Posts: 222
Joined: 07 Mar 2021, 07:44

Re: Download an online folder and its contents

Post by Jose Hidalgo » 28 Jul 2021, 17:20

Just a slight bump :angel:

teadrinker
Posts: 4309
Joined: 29 Mar 2015, 09:41
Contact:

Re: Download an online folder and its contents

Post by teadrinker » 28 Jul 2021, 17:30

Jose Hidalgo wrote: How could I download a whole online folder
What do you mean by "online folder", can you provide an example? Is this folder on FTP?

Jose Hidalgo
Posts: 222
Joined: 07 Mar 2021, 07:44

Re: Download an online folder and its contents

Post by Jose Hidalgo » 29 Jul 2021, 05:47

Sure :) Here's the story.

I have published an AHK app on SourceForge. There are already some files and folders online as you can see.
I can download the files via this url.
Apparently there's also FTP access, which I haven't tried yet.

However what I'd like to do would be to download a given directory and all its files. No need to be recursive, only all the .txt files that would be directly under a given directory (it can go from 1 to 20 files approximately).
Those files are not online yet, but if they were, they would be in the same place as the other files you can already see.

I'd like to include that feature in a future update of my app, to allow users to do selective downloads of a part of an online database, instead of having to download the whole database with 15.000 files or so which is what they're doing currently.
The database itself is currently part of the whole app download (you can check, it's the "EQ Presets" folder). It's not very big (8 MB), but 15.000 files is a lot when you only need 10 or 50 of them.

teadrinker
Posts: 4309
Joined: 29 Mar 2015, 09:41
Contact:

Re: Download an online folder and its contents

Post by teadrinker » 29 Jul 2021, 07:03

Nice to see detailed explanations. :)
If you use the https protocol you can't consider these "folders"
 
 Image
 
as real folders (independing on their structure on the server). Consider them as links to web pages.
But in any case (https or ftp) there is no such feature as «download all files in the directory». You have to enumerate files and download them one by one, but in each case using different approaches.

In this case, you have to extract the links to the files you need from the "folder" page. Taking into account that the link to download the text file ends with ".txt/download", you can use this approach:

Code: Select all

url := "https://sourceforge.net/projects/mega-switcher/files/"

; get html code of the page:
Whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
Whr.Open("GET", url, true)
Whr.Send()
Whr.WaitForResponse()
status := Whr.status
if (status != 200)
   throw "HttpRequest error, status: " . status
html := Whr.ResponseText

; create the document object
Doc := ComObjCreate("htmlfile")
Doc.write(html)

; extract links to text files:
links := Doc.links
urls := []
Loop % links.length {
   link := links[A_Index - 1]
   if (link.href ~= "\.txt/download$")
      urls.Push(link.href)
}

; your links:
for k, link in urls
   MsgBox, % link

Jose Hidalgo
Posts: 222
Joined: 07 Mar 2021, 07:44

Re: Download an online folder and its contents

Post by Jose Hidalgo » 30 Jul 2021, 18:46

Thank you very much :) I've been swamped because I'm going on vacation tomorrow. I'll make sure to take a look in a week when I'm back, and I'll report back. Thanks again !

Post Reply

Return to “Ask for Help (v1)”