Multiple scripts with same variables

[scriptomundo]

Hi all,

I am using multiple scripts which refer to huge variable sets.

Rather than running the scripts and saving variables for each instance (which seems really inefficient), is there a way to store the variables in memory and have all scripts refer to their values?

Note: ini's/file storage is too slow for my use case.

thanks everyone!

[flyingDman]

What is huge? Hundreds? Millions? Example?

[scriptomundo]

probably 60k variables per script...I would rather have 60k total that 3 scripts refer to than 180k.

Not sure how I would go about posting examples... just heaps of var := val essentially.

[swagfag]

• write the variables' contents to a common region in memory https://docs.microsoft.com/en-us/windows/win32/memory/creating-named-shared-memory
• write the variables' contents to an object, then use viewtopic.php?t=6148

if ur script requires using 60k variables, ure probably doing something wrong

[scriptomundo]

Yeah I probably am doing something wrong.

I'm using an api to save price data and other information for thousands of companies. I use arrays where I can, but found that searching arrays was a little too slow for my needs. So I put all data in variables so I can at least access them quickly.

[boiler]

I’m not seeing why you would have to search arrays but you can instantly access one of thousands of simple variables. How (and why) were you searching your array? Were you looping through it or something? How are you referencing the desired variable out of 60,000 variables? Does each variable have the company name as well as the name of property (such as price) that it contains? Did you know that you can use an associative array to identify the desired element directly with that same information instead of searching through a simple array?

[scriptomundo]

I’m not seeing why you would have to search arrays but you can instantly access one of thousands of simple variables. How (and why) were you searching your array? Were you looping through it or something? How are you referencing the desired variable out of 60,000 variables? Does each variable have the company name as well as the name of property (such as price) that it contains? Did you know that you can use an associative array to identify the desired element directly with that same information instead of searching through a simple array?

I definitely don't have to, think I'm just not familiar with what a better option is.

Currently I use a script that builds arrays/simple variable lists and saves to file.

These are loaded by a 'processor' script, which reads these variables at startup and then handles decision logic.

Code: Select all

;============================================================================================
 GetCoinIDs()	{						; get coingecko ids
;============================================================================================
global
Loop 6 
	{
command=\K -H ""Content-Type: application/json"" -X GET https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=250&page=1&sparkline=false"" ",," hide
stringreplace, command, command, page=1, page=%A_index%
;msgbox command: %command%
out := RunGetOutput("curl",command)
;clipboard := out
;pause

stringreplace, out, out, ",,A
stringreplace, out, out, },,A
stringreplace, out, out, {,,A
stringreplace, out, out, ],,A
stringreplace, out, out, [,,A
stringsplit, spl, out, `,
coincount=1
StartTime := A_TickCount
loop %spl0%
	{
	thisspl := spl%A_index%
	stringsplit, subspl, thisspl, :

	label := subspl1
	value := subspl2
	if instr(label,"symbol")
		{
		symbolarray=%symbolarray%,%value%
		symbolcount++
		lastsymbol=%value%
		bigarray=%bigarray%,[%lastsymbol%]%lastid%

		}
	if instr(label,"id")
		{
		idarray=%idarray%,%value%
		idcount++
		lastid=%value%
		}	
	if instr(label,"name")
		{
		namearray=%namearray%,%value%
		CoinList1=%coinlist1%%lastsymbol%`r`n%value%`r`n
		namecount++
		lastname=%value%
		}		
	}
}

;======================================
; housekeeping name array
;======================================
stringreplace, namearray, namearray,%A_space%Protocol,,A
stringreplace, namearray, namearray,%A_space%Token,,A 
stringreplace, namearray, namearray,%A_space%Network,,A 
stringreplace, namearray, namearray,%A_space%Coin,,A

stringreplace, coinlist1, coinlist1,%A_space%Protocol,,A
stringreplace, coinlist1, coinlist1,%A_space%Token,,A 
stringreplace, coinlist1, coinlist1,%A_space%Network,,A 
stringreplace, coinlist1, coinlist1,%A_space%Coin,,A

stringreplace, coinlist1, coinlist1,the-,,A
stringreplace, coinlist1, coinlist1, -network,,A
stringreplace, coinlist1, coinlist1, -usd,,A
stringreplace, coinlist1, coinlist1, -protocol,,A
stringreplace, coinlist1, coinlist1, -coin,,A
stringreplace, coinlist1, coinlist1, -token,,A
stringreplace, coinlist1, coinlist1, -standard,,A
stringreplace, coinlist1, coinlist1, -governance,,A
stringreplace, coinlist1, coinlist1, -,%A_space%,A

fileappend,%symbolarray%,E:\Trade\ref\symbols.txt
fileappend,%namearray%,E:\Trade\ref\names.txt
fileappend,%idarray%,E:\TradeBot\ref\ids.txt
fileappend,%bigarray%,E:\Trade\ref\bigarray.txt
fileappend,%coinlist1%,E:\Trade\ref\CoinListNEW.txt
symbolarray=
namearray=
idarray=
bigarray=
}
clipboard := coinlist1
msgbox idcount: %idcount%`nsymbolcount: %symbolcount%
return

[boiler]

OK, well you're basically building "pseudo-arrays" in a sense, but it looks even worse in the sense that apparently you're making a huge string that has a ton of keys and values strung together. You would be somewhat better off using psuedo-arrays, where at least they're separate variables that have different names that you could key off of like you would with a true array's keys. But the best would be to use real object-based arrays. Like I was saying, there would be no searching involved if you used associative arrays where you could instantly obtain the value of any element based on the key(s).

[scriptomundo]

thanks boiler, i'll look into this and wb!

[scriptomundo]

@boiler one question - would it be better/is there a need to create associative arrays when building datasets, and having child scripts that can refer to their location (rather than building text files and then using associative arrays from child scripts like I am currently thinking of doing)?

Objects/Classes etc have

[boiler]

It is easy to share your main script's array object with your other scripts using ObjRegisterActive as swagfag recommended. Once you've shared it, accessing the object from the child scripts would be exactly the same (with some relatively unimportant limitations) as you would access it from the main script. And in general, using objects is a better approach than strings of text.

[flyingDman]

So I decided to test this for speed. I made a look-up application. I created a 60,000 row text file of 2 columns. First column contains strings of 8 random letters and the second contains random 8 digit integers. The 2 columns are tab separated. Looks like this:

Code: Select all

IGPGHHTT	50323898
WOOXKZFB	86657532
NYSJVTMB	56869807
FTGEELLG	04950696
ABRHWNHX	51560344
FBNZMAXR	88955001
SWJSETSL	87320915
GJSFYBWU	50882448

I changed one random line to AAAAAAAA 99999999. Goal is to find AAAAAAAA and return 99999999.
I have a server script:

Code: Select all

#Persistent

OnExit Revoke

fileread, var, twocolumn.txt
AnObject := strsplit(var,"`n")
ObjRegisterActive(AnObject, "{072736AA-BC5F-4AB3-8E6A-1311B2ED444F}")
return

Revoke:
ObjRegisterActive(ActiveObject, "")
ExitApp

ObjRegisterActive(Object, CLSID, Flags:=0) {
    static cookieJar := {}
    if (!CLSID) {
        if (cookie := cookieJar.Remove(Object)) != ""
            DllCall("oleaut32\RevokeActiveObject", "uint", cookie, "ptr", 0)
        return
    }
    if cookieJar[Object]
        throw Exception("Object is already registered", -1)
    VarSetCapacity(_clsid, 16, 0)
    if (hr := DllCall("ole32\CLSIDFromString", "wstr", CLSID, "ptr", &_clsid)) < 0
        throw Exception("Invalid CLSID", -1, CLSID)
    hr := DllCall("oleaut32\RegisterActiveObject"
        , "ptr", &Object, "ptr", &_clsid, "uint", Flags, "uint*", cookie
        , "uint")
    if hr < 0
        throw Exception(format("Error 0x{:x}", hr), -1)
    cookieJar[Object] := cookie
}

and a receiving script:

Code: Select all

starttime := a_tickcount
x := ComObjActive("{072736AA-BC5F-4AB3-8E6A-1311B2ED444F}")
loop, % x.maxindex()
	if (strsplit(x[a_index],"`t")[1] = "AAAAAAAA")
		msgbox % strsplit(x[a_index],"`t")[2] "`n" a_tickcount - starttime

It gets the result in a lousy ~3.2 seconds
Compared with a fileread script like this:

Code: Select all

starttime := a_tickcount
fileread, var, twocolumn.txt
for x,y in strsplit(var,"`n")
	if (strsplit(y,"`t").1 = "AAAAAAAA")
		msgbox % strsplit(y,"`t").2 "`n" a_tickcount - starttime

which does it 100 times faster, i.e. in ~32 ms!
What am I doing wrong? Or is this it? (in which case ObjRegisterActive should not be used for this type of application....)

twocolumn.txt

(1.09 MiB) Downloaded 34 times

[boiler]

But what if with the object approach, you made it an associative array like I was recommending so it would eliminate the need to do any searching? Then instead of looping until you find it, it would just be simply:

Code: Select all

MsgBox, % MyArray["AAAAAAAA"]

[flyingDman]

Thank you. Big difference! and faster than fileread.
Serving script:

Code: Select all

#Persistent
OnExit Revoke
fileread, var, twocolumn.txt
AnObject := {}
for x,y in strsplit(var,"`n")
    AnObject[strsplit(y,"`t").1] := strsplit(y,"`t").2

ObjRegisterActive(AnObject, "{072736AA-BC5F-4AB3-8E6A-1311B2ED444F}")
return

Revoke:
ObjRegisterActive(ActiveObject, "")
ExitApp

ObjRegisterActive(Object, CLSID, Flags:=0) {
    static cookieJar := {}
    if (!CLSID) {
        if (cookie := cookieJar.Remove(Object)) != ""
            DllCall("oleaut32\RevokeActiveObject", "uint", cookie, "ptr", 0)
        return
    }
    if cookieJar[Object]
        throw Exception("Object is already registered", -1)
    VarSetCapacity(_clsid, 16, 0)
    if (hr := DllCall("ole32\CLSIDFromString", "wstr", CLSID, "ptr", &_clsid)) < 0
        throw Exception("Invalid CLSID", -1, CLSID)
    hr := DllCall("oleaut32\RegisterActiveObject"
        , "ptr", &Object, "ptr", &_clsid, "uint", Flags, "uint*", cookie
        , "uint")
    if hr < 0
        throw Exception(format("Error 0x{:x}", hr), -1)
    cookieJar[Object] := cookie
}

receiving script:

Code: Select all

starttime := a_tickcount
x := ComObjActive("{072736AA-BC5F-4AB3-8E6A-1311B2ED444F}")
msgbox % x["AAAAAAAA"] "`n" a_tickcount - starttime

[boiler]

Big difference! and faster than fileread.

Nice. Thanks for running that.