XA_Load/Save - Saving/Loading Arrays in XML - AHK v1.1.33

[TheArkive]

AHK v2 version can be found here.

Thanks to @trueski for posting this way back when, and thanks to @ahk7 for his update with XA_CleanInvalidChars().

This library converts linear and associative arrays into XML text, and takes XML text to convert into linear and associative arrays of any depth and nesting. Saving text to file is done with FileAppend and Loading text from file is done with FileRead. The user must do this in their code.

====================================================

Updates

2020/07/15 Updates:

• linear arrays are now properly supported - output XML also includes data types
• now when a duplicate node is detected an error is thrown and the associated XML is displayed
• created Type() function to try and detect as many types as possible (so far so good)
• Known Issue: Empty arrays/maps can't be detected as a specific type in AHK v1, it's only detected as an object. Default for empty array is linear array.

2020/02/28 Update:

• @ahk7 contributed XA_CleanInvalidChars()

2020/02/26:

• Initial Post

====================================================

Code: Select all

; AHK v1
; Example ===================================================================================
; ===========================================================================================
; a := Object(), b := Object(), c := Object(), d := Object(), e := Object(), f := Object() ; Object() is more technically correct than {} but both will work.

; d["g"] := 1, d["h"] := 2, d["i"] := ["purple","pink","pippy red"]
; e["g"] := 1, e["h"] := 2, e["i"] := Object("1","test1","2","test2","3",["x","y","z"])
; f["g"] := 1, f["h"] := 2, f["i"] := [1,2,Object("a",1.0009,"b",2.0003,"c",3.0001)]

; a["test1"] := "test11", a["d"] := d
; b["test3"] := "test33", b["e"] := e
; c["test5"] := "test55", c["f"] := f

; myObj := Object()
; myObj["a"] := a, myObj["b"] := b, myObj["c"] := c, myObj["test7"] := "test77", myObj["test8"] := "test88"

; g := ["blue","green","red"], myObj["h"] := g ; add linear array for testing

; q := Chr(34)
; textData2 := XA_Save(myObj) ; ===> convert array to XML
; msgbox % "XML Breakdown:`r`n(Should match second breakdown.)`r`n`r`n" textData2


; ===========================================================================================
; Error Test - Duplicate Node ; =============================================================
; ===========================================================================================
; textData2 := StrReplace(textData2,"</Base>","<test8 type=" q "String" q ">test88</test8>`r`n</Base>") ; <--- test error, duplicate node
; ===========================================================================================
; ===========================================================================================


; newObj := XA_Load(textData2) ; ===> convert XML back to array

; textData3 := XA_Save(newObj) ; ===> break down array into 2D layout again, should be identical
; msgbox % "Second Breakdown:`r`n(should be identical to first breakdown)`r`n`r`n" textData3
; msgbox % Jxon_Dump(newObj,4)

; ExitApp
; ===========================================================================================
; End Example ; =============================================================================
; ===========================================================================================


; =============================================================
; XA_Save(Array,BaseName := "Base")		>>> dumps XML text from associative array var
; XA_Load(XMLText)						>>> converts XML text to associative array
;		Originally posted by trueski
;		in collaberation with user ahk7 contributing adding functions XA_XMLEncode() and XA_CleanInvalidChars()
;		https://autohotkey.com/board/topic/85461-ahk-l-saveload-arrays/
;		Modified by TheArkive
; - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
; Root node by default is called "Base".  It doesn't matter because it isn't
; used and is not part of the array after using XA_Load().
;
; User must choose how to handle output, ie.
;		1) Save XML to file after ... var := XA_Save()
;		2) Load XML text from file before converting with ... myArray := XA_Load()
; =============================================================
XA_Save(arr,BaseName := "Base") { ; XA_Save(Arr, BaseName := "Base")
	q := Chr(34)
	outVar := "<?xml version=" q "1.0" q " encoding=" q "UTF-8" q "?>`n<" BaseName " type=" q Type(arr) q ">`n" XA_ArrayToXML(arr) "`n</" BaseName ">"
	return OutVar
}
XA_Load(XMLText) { ; orig param was ... XA_Load(Path) ... Path = XML file location
	Local XMLObj, XMLRoot, Root1, Root2 ; XMLText
	
	XMLObj    := XA_LoadXML(XMLText)
	XMLObj    := XMLObj.selectSingleNode("/*")
	If (IsObject(XMLObj)) { ; check if settings are blank
		XMLRoot   := XMLObj.nodeName
		curType := ""
		Try curType := XMLObj.getAttribute("type")
		outputArray := XA_XMLToArray(XMLObj.childNodes,,curType)
		return outputArray
	} Else
		return "" ; if settings are blank return a blank Map()
}
XA_XMLToArray(nodes, NodeName:="", curType:="") {
	If (curType = "Array")
		Obj := []
	Else If (curType = "Map")
		Obj := Object()
	
	for node in nodes {
		if (node.nodeName != "#text") { ;NAME
			If (node.nodeName == "Invalid_Name" && node.getAttribute("ahk") == "True")
				NodeName := node.getAttribute("id")
			Else
				NodeName := node.nodeName
		} else { ;VALUE
			Obj := node.nodeValue
		}
		
		if node.hasChildNodes { 
			prevSib := ""
			Try prevSib := node.previousSibling.nodeName
			
			nextSib := ""
			Try  nextSib := node.nextSibling.nodeName
			
			nextType := ""
			Try nextType := node.getAttribute("type")
			
			If ((nextSib = node.nodeName || node.nodeName = prevSib) && node.nodeName != "Invalid_Name" && node.getAttribute("ahk") != "True") { ;Same node name was used for multiple nodes
				pN := "", cN := "", nN := ""
				Try pN := node.previousSibling.xml
				Try cN := node.xml
				Try nN := node.nextSibling.xml
				
				Throw "Duplicate node:`r`n`r`nPrev:`r`n" pN "`r`n`r`nCurrent:`r`n" cN "`r`n`r`nNext:`r`n" nN
				; If (!prevSib) { ;Create object - previous -> !node.previousSibling.nodeName
					; Obj[NodeName] := {}
					; ItemCount := 0
				; }
			  
				; ItemCount++
			  
				; If (node.getAttribute("id") != "") ;Use the supplied ID if available
					; Obj[NodeName][node.getAttribute("id")] := XA_XMLToArray(node.childNodes, node.getAttribute("id"))
				; Else ;Use ItemCount if no ID was provided
					; Obj[NodeName][ItemCount] := XA_XMLToArray(node.childNodes, ItemCount)
			} Else {
				If (curType = "Map") {
					Obj[NodeName] := XA_XMLToArray(node.childNodes, NodeName, nextType)
				} Else If (curType = "Array") {
					Obj.InsertAt(NodeName,XA_XMLToArray(node.childNodes, NodeName, nextType))
				}
			}
		}
	}
	
	return Obj
}
XA_LoadXML(ByRef data){
	o := ComObjCreate("MSXML2.DOMDocument.6.0")
	o.async := false
	o.LoadXML(data)
	return o
}
XA_ArrayToXML(theArray, tabCount:=1, NodeName:="") {
    Local tabSpace, extraTabSpace, tag, val, theXML, root, iTag, eTag, aXML, eVal, curType
	q := Chr(34)
	tabCount++
    tabSpace := "", extraTabSpace := "", curType := ""
	
	if (!IsObject(theArray)) {
		root := theArray
		theArray := %theArray%
    }
	
	While (A_Index < tabCount) {
		tabSpace .= "`t" 
		extraTabSpace := tabSpace . "`t"
    } 
     
	for tag, val in theArray {
        If (!IsObject(val)) { ; items/values
			iTag := "Invalid_Name"
			eTag := XA_XMLEncode(tag) ; xml encoded tag
			eVal := XA_XMLEncode(val)
			If (XA_InvalidTag(tag))
				theXML .= "`n" tabSpace "<" iTag " id=" q eTag q " ahk=" q "True" q " type=" q Type(val) q ">" eVal "</" iTag ">"
			Else
				theXML .= "`n" tabSpace "<" tag " type=" q Type(val) q ">" eVal "</" tag ">"
		} Else {
			iTag := "Invalid_Name"
			eTag := XA_XMLEncode(tag) ; xml encoded tag
			aXML := XA_ArrayToXML(val, tabCount, "")
			If (XA_InvalidTag(tag))
				theXML .= "`n" tabSpace "<" iTag " id=" q eTag q " ahk=" q "True" q " type=" q Type(val) q ">`n" aXML "`n" tabSpace "</" iTag ">"
			Else
				theXML .= "`n" tabSpace "<" tag " type=" q Type(val) q ">" "`n" aXML "`n" tabSpace "</" tag ">"
	    }
    } 
	
	theXML := SubStr(theXML, 2)
	Return theXML
} 
XA_InvalidTag(Tag) {
	q := Chr(34)
	Char1      := SubStr(Tag, 1, 1) 
	Chars3     := SubStr(Tag, 1, 3)
	StartChars := "~``!@#$%^&*()_-+={[}]|\:;" q "'<,>.?/1234567890 `t`n`r"
	Chars := q "'<>=/ `t`n`r"
	
	Loop, Parse, StartChars
	{
		If (Char1 = A_LoopField)
		  Return 1
	}
	
	Loop, Parse, Chars
	{
		If (InStr(Tag, A_LoopField))
			Return 1
	}
	
	If (Chars3 = "xml")
		Return 1
	Else
		Return 0
}

Type(arr) {
	If (arr + 0 = arr) { ; is number
		If (arr = Format("{:d}",arr))
			return "Integer"
		Else
			return "Float"
	} If (arr.CaseSensitive != "" And arr.BackspaceIsUndo != "")
		return "InputHook"
	Else If (arr.AtEOF != "" And f.__Handle > 0)
		return "File"
	Else If (arr.IsBuiltIn != "" And arr.IsVariadic != "")
		return "Function"
	Else If (arr.__Class != "")
		return "Class"
	Else If (IsObject(arr)) {
		sequential := true
		For i, val in arr {
			
			If (i != A_Index) {
				sequential := false
				Break
			}
		}
		
		equal := false
		If (arr.Length() = arr.Count())
			equal := true
		
		If (sequential And equal)
			return "Array"
		Else If (!equal)
			return "Map"
		Else If (arr.Length() = 0 And arr.Count() > 0)
			return "Map"
		Else
			return ""
	} Else If (arr + 0 = "")
		return "String"
}


; XA_XMLEncode references:
; https://en.wikipedia.org/wiki/XML#Escaping
; https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references#Predefined_entities_in_XML
; added again as original source code posted at forum was lost due to forum upgrade 

XA_XMLEncode(Text) {
	Text := StrReplace(Text,"&","&amp;")
	Text := StrReplace(Text,"<","&lt;")
	Text := StrReplace(Text,">","&gt;")
	Text := StrReplace(Text,Chr(34),"&quot;")
	Text := StrReplace(Text,"'","&apos;")
	Return XA_CleanInvalidChars(Text)
}

XA_CleanInvalidChars(text, replace:="") {
	re := "[^\x09\x0A\x0D\x20-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]"
	Return RegExReplace(text, re, replace)
	
	/*
		Source: http://stackoverflow.com/questions/730133/invalid-characters-in-xml
		public static string CleanInvalidXmlChars(string text) 
		{ 
		// From xml spec valid chars: 
		// #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]     
		// any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. 
		string re = @"[^\x09\x0A\x0D\x20-\uD7FF\uE000-\uFFFD\u10000-u10FFFF]"; 
		return Regex.Replace(text, re, ""); 
		}
	*/
}

XML uses a lot of overhead. If you need to save/load arrays/maps I recommend JSON format.

Suggested scripts:

[v2 only] Yaml and JSON parser written by HotKeyIt
YEJP - Yet Another Json Parser - AHK v1/2 written by me

[elModo7]

Seems useful for some projects, I have been saving arrays into json for later use.
The one I really liked was the treeview one, man that saved so much time for a web rom indexer I did a few weeks ago.

Spoiler

[youtube]https://youtu.be/v_qLkJbJeu0?t=37[/youtube]

I did an sqlite one, but wanted to have a listview for each system, instead of saving that on the bbdd or query again for the web, I had your treeview script keep all the info on disk and then load it from there.
It saved quite a few seconds and worked great!

[TheArkive]

@elModo7 which library do you use for saving/loading arrays in JSON format? Sounds like something that will definitely come in handy...

I found this one initially.

[elModo7]

@elModo7 which library do you use for saving/loading arrays in JSON format? Sounds like something that will definitely come in handy...

I found this one initially.

That one exactly, for saving it, I export the object to json, then I fileappend it.
For loading to object I then do fileread and json parse.

Been using it a lot for league of legends data for faster testing instead of querying the api over and over.

[ahk7]

I added XA_CleanInvalidChars() to XA https://github.com/hi5/XA/blob/master/xa.ahk#L177 - I use XA often as it is just so easy to load/save settings.

[TheArkive]

thanks @ahk7 ... not sure how I missed your reply, sorry about that.

The old forums actually replaced all the HTML encoded stuff with their text counterpart. So all I saw was:

StringReplace, Text, Text, &, &, All

... for the entire function *laughs*... I was wondering what the XML_Encode() function did. Now I know what it was supposed to be.

I'll update the OP soon. I noticed you had your version posted already a few years ago. If you want go ahead and take this code in the OP and I'll link the OP to your repo.

[ahk7]

I added links to the GH repo and forum post to this thread for a AHK v2 compatible version, having multiple versions can't hurt

[TheArkive]

@ahk7 sounds good man!

[songdg]

Does not work on AHKv1? a problem occurs saying :call to nonexistent function of "return Map()".

[TheArkive]

@songdg
You'll need to check the comments and make the indicated changes for it to work on AHK v1. Map() is for AHK v2 only. Check comments. Should be just a few lines.

Let me know if you have issues, I'll help more.

[songdg]

@songdg
You'll need to check the comments and make the indicated changes for it to work on AHK v1. Map() is for AHK v2 only. Check comments. Should be just a few lines.

Let me know if you have issues, I'll help more.

Thank you very much!

[songdg]

another problem, MyArray:=[111111,222222,8888888], I use XA_Save(MyArray, "test.xml") to save the array, no error occurs but also can't find the file which I saved.And the use of BaseName := "Base"?

[TheArkive]

@songdg
These functions don't save to file, they convert to text so that you can save to file. Try this first:

(ahk v1 right?)

var := XA_Save(MyArray)
MsgBox % var ; <--- just to test
FileAppend %var%, test.xml

edit: I'll run these functions again to see if AHK v1.1.33 and AHK v2 alphas have broken anything

EDIT: I found an error that doesn't convert a linear array back properly... trying to fix that now.

[daywalker]

Check post #3 for AHK v1 version.

Just a proposal for AHK V1 & 2: Would it not make sense to have in AHK a #define/#ifdef-Compiler-Directives like in C ? So Code could be easier maintained?

[TheArkive]

2020/07/15 Updates:

• linear arrays are now properly supported - output XML also includes data types
• now when a duplicate node is detected an error is thrown and the associated XML is displayed
• created Type() function to try and detect as many types as possible (so far so good)
• Known Issue: Empty arrays/maps can't be detected as a specific type in AHK v1, it's only detected as an object. Default for empty array is linear array.

@songdg
Updated, your array should load AND save now using these functions.

[songdg]

2020/07/15 Updates:

• linear arrays are now properly supported - output XML also includes data types
• now when a duplicate node is detected an error is thrown and the associated XML is displayed
• created Type() function to try and detect as many types as possible (so far so good)
• Known Issue: Empty arrays/maps can't be detected as a specific type in AHK v1, it's only detected as an object. Default for empty array is linear array.

@songdg
Updated, your array should load AND save now using these functions.

Thanks! Something need to be change? I have a problem running this version saying "call to nonexistent function." on "DebugMsg("this shouldn't happen")".

[TheArkive]

@songdg
Dang I'm having a heck of a time with this one.
Just comment out that line. I updated the OP too.

[elModo7]

Glad to see updates on this!