Asynchronous API data obtaining

[MannyKSoSo]

Hello all!
I have come to an issue with obtaining multiple sets of data from an API (sadly I cannot provide the data on the api as this is a private api, but I will describe it as best I can). So working with this api is no different from any other and I have no problem obtaining the data, but the issue comes in with the amount of data I wish to gather. So in order to gain the data I need I have to go through a two step process, the first is to obtain ID's of a specific request, which in my case returns over 2,000 ID's (which will only grow), then I parse the ID's to obtain specific data from each ID via a second search (which is where the problem comes in). I have used the method to go each one in a loop, but this takes an extremely long time as each search takes a bit before the data is obtained (usually 1-3 minutes as there is a lot of data the api has to go through), so it would take hours to get the data in this method, the second way I have tried is by creating a second script using the Send_WM_COPYDATA method which involves starting a script for each ID (which is over 2,000 scripts running!) and returning the data, but as you can probably guess does not work well as by the time some of the data gets collected the original script is still trying to create each instance it needs. So then data is being ignored and creates more issues with conflicts of data and the loading of data. So I am hoping someone has a good Idea on how to help.

Also I know it will still take some time for the script to fully grab all the data, which is fine, but I would rather it take a max of 30 mins compared to several hours!

Thanks All!

[buttshark]

The only suggestion I can think of is using curl. You can try to run 2000 commands via

Code: Select all

loop 2000
{
	command := "curl " yourCommand " > " tmpFile%A_Index%
	Run, %ComSpec% /c %command%,, Hide
}

If you use Run instead of RunWait, your script can run the 2000 commands fairly quickly, and each one may take the 1-3 minutes each. You can then regularly check if the files were written to and go from there.

Code: Select all

SetTimer, lbl, 1000
return

lbl:
loop 2000
{
	FileRead, response, % tmpFile%A_Index%
	if(response) {
		;do something
	}
}

Or you could instead directly send the response to your main script, as you tried earlier.

None of this was tested, so I have no idea how viable this is. Good luck!

[MannyKSoSo]

Going with the curl method I happen to run across a problem where it will end after 40 secs of obtaining the data, even after I have attempted to allow for an increased timer, this fails. Would you happen to know how to get it to work with a longer timer? (Also I can confirm it works with the api as if I put something in wrong it gives me the proper response).

Edit: just as a note, it will make the file with the run command (from the first code block), but will not put any information in it

[buttshark]

I'm confused, are you saying that the curl command will crap out after 40 seconds? If so, I'm not totally sure. Looking at the documentation, [url=https://curl.haxx.se/docs/manpage.html]documentation], you could try to set a timeout with -m with some large number (assuming there is a default max), or use the --retry <num> to retry the call upon failure. Besides that, I've got nothing.

curl also has exit codes, though I'm not sure how to get them in ahk. If you can figure that out, that may also help.

And yes, for whatever reason it makes the file long before it ever writes to it. That's why I have

Code: Select all

FileRead, response, % tmpFile%A_Index%
if(response) {
	...
}

and not

Code: Select all

if(FileExist(tmpFile%A_Index%)) {
	FileRead, response, % tmpFile%A_Index%
	...
}

So assume it makes it, aborts, and doesn't ever write anything to the file.

[MannyKSoSo]

Apologies for the confusion, so using -m --connect-timeout and --retry had no effect on the time it takes. The specific timing is under Time Spent, and it never reaches over 40 seconds (even with setting a time to cut out), which may be the fact that its expecting the result to come rather quickly. Even looking around I don't see anyone having similar issues, but its probably because the connection time is extremely long so it looks like its doing nothing.

Edit: I have tested curl with another free api and have no problems, its only with the one I cannot share I am having issues with

Edit: Using a smaller instance I get this as the reference, so the time it takes to find it is correct (eg the 9 seconds), but the Received is zero. I still haven't found a way to get the data yet. I think what is happening is that the page turns blank for a split second and throws off the whole thing (this is only a guess) since I can input incorrect data or use another api and get resutls.

Code: Select all

%	Total	%	Received	%	Xferd	Average Dload	Speed Upload	Time Total	Time Spent	Time Left	Current Speed
0	0	0	0		0	0	0		0		--:--:--	0:00:09		--:--:--	0

[MannyKSoSo]

Alright so I managed to get it to work finally, I ended up looking at the network page under the devtools for Chrome and copied the url as a curl command. With some fixing of the url I managed to obtain the data! So thank you so far for this. Now I get the joy of parsing each response for specific data, yay