Delete Object key - How? Topic is solved

[Albireo]

I have a string that I create an array from. (like this) .:

Code: Select all

InputCSV = 
( LTrim
	`n
	"201","Freddy","4,50"
	"302","Emelie","174,50"
	"203","Clas","9,90"
	
	"904","Johanna","1400,00"
	"205","Erica","1,75"
	
)
InputData := StrSplit(InputCSV, "`n")

MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % InputCSV "`n`n" InputData.Count() "`n- " InputData[1] "`n- " InputData[2] "`n- " InputData[3] "`n- " InputData[4] "`n- " InputData[5] "`n- " InputData[6]
Count = 0
Loop % InputData.Count()
{	If ( StrLen(InputData[A_Index]) < 1 )
		InputData.Delete(A_Index)
	else
		Count += 1
}
MsgBox ,, Rad %A_LineNumber% -> %A_ScriptName%, % InputCSV "`n`n" InputData.Count() "`n- " InputData[1] "`n- " InputData[2] "`n- " InputData[3] "`n- " InputData[4] "`n- " InputData[5] "`n- " InputData[6]

There are empty rows in that string above (InputCSV). These should be deleted - along with the object keys.
But… InputData[1] is empty before InputData.Delete(1) and even after - why?
InputData.Count() is "9" before InputData.Delete() and "5" after (it's correct)

I have made an attempt with what I believe the AHK manual say about Object.delete()
But it doesn't work, as I want.
How can this be solved?

[Hellbent]

Is this what you're trying to do?

Code: Select all

Index:=InputData.Length()
Loop,% InputData.Length()	{
	If( StrLen(InputData[Index]) < 1 )
		InputData.RemoveAt(Index)
	
	Index--
}

Or if you want to save space.

Code: Select all

Loop,% Index:=InputData.Length()
	(StrLen(InputData[Index])< 1)?(InputData.RemoveAt(Index--)):(Index--)

[Albireo]

Thank you!

... Or if you want to save space...

Save Space? (In the array or AHK-program?)
I get the result I want, in both examples.

What does this instruction do? Index--?

[Hellbent]

Thank you!

No problem.

... Or if you want to save space...

Save Space? (In the array or AHK-program?)
I get the result I want, in both examples.

Just reduce the number of lines of code.
If it didn't add that in someone else would surly be along shortly to do it lol.

What does this instruction do? Index--?

That is so that I can loop through the array backwards.
If you don't, when you remove a element, the one ahead of it gets shifted and fills that now vacant spot and ends up getting missed.

[just me]

If you don't want to store empty lines in the array, don't store them:

Code: Select all

#NoEnv
InputCSV =
(LTrim
	`n
	"201","Freddy","4,50"
	"302","Emelie","174,50"
	"203","Clas","9,90"

	"904","Johanna","1400,00"
	"205","Erica","1,75"

)

; --------------------------------------
InputData := []
DataIndex := 0
For LineIndex, Line In StrSplit(InputCSV, "`n")
{
   LineCount := LineIndex
   If StrLen(Line)
   {
      DataIndex += 1
      InputData[DataIndex] := Line
   }
}
; --------------------------------------

DataLines := ""
For Each, Line In InputData
   DataLines .= "`n" Line
MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % LineCount . "`n" . InputCSV . "`n`n`n" . InputData.Count() . DataLines

[Albireo]

Thanks for the suggestion. All tips give the same results
Have done an analysis on all three examples.
I can't find any difference in performance with the suggestions from @Hellbent

Two CSV-files are used in the test,
- File1 with 25,622 rows (20 columns and 6172kB) and
- File2 with 34,658 rows (72 columns 6638kB)

The test was done as follows:
- Start the clock
- Load a CSV file
- Create an array
- Stop the clock

The result .: File1.: File2.:
@Helbent .: (94ms) (125ms)
@just me .: (2937ms) (3297ms)

Was very surprised that there was such a big difference.
I don't understand why!

[just me]

The test was done as follows:
- Start the clock
- Load a CSV file
- Create an array
- Stop the clock
...
Was very surprised that there was such a big difference.
I don't understand why!

Nice, but I'm not able to run your 'script'. So I cannot help you.

[swagfag]

reserve enough capacity for the array to hold 35k rows
push into the array with .Push, not via array index access
if strlen(line) to check whether a line is empty(assuming that was the intention) is wasteful

[SpeedMaster]

Have done an analysis on all three examples.

How about this one ?

Code: Select all

Loop, % InputData.Length()
	(z:=InputData.Pop()) ? (InputData.InsertAt(1,z))

[Albireo]

reserve enough capacity for the array to hold 35k rows...

How do you do it? Something like that? InputData.SetCapacity(35000)

...push into the array with .Push, not via array index access
if strlen(line) to check whether a line is empty(assuming that was the intention) is wasteful.

If you do not use StrLen(),how do you know which line is "empty"?
(Even a line that contains only field delimiters is empty and should be removed.)

---

The suggestion from @SpeedMaster give the following run times .:
File1 .: 3391ms
File2 .: 6313ms

[just me]

Your benchmarking results are less meaningful if you don't show your complete benchmark script.

[swagfag]

an empty line is "". anything else is by definition not empty. if u have redefined empty to mean something else, then simply checking against "" won't work. a different discriminator is required.

[just me]

I just ran my own benchmark. Reading a CSV file with ~20000 rows of 18 columns (~3 MB) and converting it to an array using my code needs < 100 ms here.

[Albireo]

Below is the AHK code I used.
To perform the tests I chose the input file, and configured Goto to different test codes

Unfortunately, I will not share the test files, but create your own files. The time differences will also depend on the type of computer the test takes place.

Code: Select all

#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
#Persistent
#SingleInstance Force
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.

StartTime := A_TickCount
; FileRead InputCSV, c:\temp\FDT_ART_100.txt	; File1
FileRead InputCSV, c:\temp\EXPOART.txt		; File2

Goto Example4	; Example 1-4
ExitApp

Example1:	; Test1 @Helbent
	InputData := StrSplit(InputCSV, "`n")
	Loop % Index := InputData.Length()
		( StrLen(InputData[Index]) < 1 )?(InputData.RemoveAt(Index--)):(Index--)

	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex1 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp


Example2:	; Test @Helbent
	InputData := StrSplit(InputCSV, "`n")
	Index:=InputData.Length()
	Loop,% InputData.Length()	{
		If( StrLen(InputData[Index]) < 1 )
			InputData.RemoveAt(Index)
	Index--
	}
	
	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex2 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp


Example3:	; Test @just me
	InputData := []
	DataIndex := 0
	For LineIndex, Line In StrSplit(InputCSV, "`n")
	{	LineCount := LineIndex
		If StrLen(Line)
		{	DataIndex += 1
			InputData[DataIndex] := Line
		}
	}
	DataLines := ""
	For Each, Line In InputData
		DataLines .= "`n" Line
	
	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex3 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp


Example4:	; Test @SpeedMaster
	InputData := StrSplit(InputCSV, "`n")
	Loop, % InputData.Length()
		(z:=InputData.Pop()) ? (InputData.InsertAt(1,z))

	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex4 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp

[Hellbent]

Below is the AHK code I used.
To perform the tests I chose the input file, and configured Goto to different test codes

I ran your test. Here is my script.

Code: Select all

/*

#SingleInstance,Force
SetBatchLines,-1
FileDelete,%A_ScriptDir%\Test File.txt
temp:=""
Loop, 10000	{
	if(Mod(A_Index,4)=0){
		temp.= "`n"
	}else	{
		Temp .= "Some Random Text`n"
		
	}

}
FileAppend,%Temp%,%A_ScriptDir%\Test File.txt
SoundBeep, 500
return
*ESC::ExitApp

*/


#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
#Persistent
#SingleInstance Force
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.

StartTime := A_TickCount
; FileRead InputCSV, c:\temp\FDT_ART_100.txt	; File1
FileRead InputCSV, %A_ScriptDir%\Test File.txt		; File2

Goto Example4	; Example 1-4     ;<--------- ran each 10 times to get fastest and longest times
ExitApp

Example1:	; Test1 @Helbent   ;<-------------------- 31 - 63 ms
	InputData := StrSplit(InputCSV, "`n")
	Loop % Index := InputData.Length()
		( StrLen(InputData[Index]) < 1 )?(InputData.RemoveAt(Index--)):(Index--)

	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex1 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp


Example2:	; Test @Helbent    ;<-------------------- 31 - 93 ms
	InputData := StrSplit(InputCSV, "`n")
	Index:=InputData.Length()
	Loop,% InputData.Length()	{
		If( StrLen(InputData[Index]) < 1 )
			InputData.RemoveAt(Index)
	Index--
	}
	
	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex2 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp


Example3:	; Test @just me       ;<-------------------- 125 - 157 ms
	InputData := []
	DataIndex := 0
	For LineIndex, Line In StrSplit(InputCSV, "`n")
	{	LineCount := LineIndex
		If StrLen(Line)
		{	DataIndex += 1
			InputData[DataIndex] := Line
		}
	}
	DataLines := ""
	For Each, Line In InputData
		DataLines .= "`n" Line
	
	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex3 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp


Example4:	; Test @SpeedMaster     ;<-------------------- 1938 - 2063    ms
	InputData := StrSplit(InputCSV, "`n")
	Loop, % InputData.Length()
		(z:=InputData.Pop()) ? (InputData.InsertAt(1,z))

	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex4 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp

I ran each one 10 times and recorded the fastest and slowest times.


Example 1: 31 - 63 ms
Example 2: 31 - 93 ms
Example 3: 125 - 157 ms
Example 4: 1938 - 2063 ms

***Edit***
Closer inspection of the strlen showed that min length was 1.

New results:
Example 1: average 250 ms
Example 2: average 250 ms
Example 3: average 125 ms

[just me]

@Albireo:

Code: Select all

Example3:	; Test just me
	InputData := []
	DataIndex := 0
	For LineIndex, Line In StrSplit(InputCSV, "`n")
	{	LineCount := LineIndex
		If StrLen(Line)
		{	DataIndex += 1
			InputData[DataIndex] := Line
		}
	}
	DataLines := ""
	For Each, Line In InputData
		DataLines .= "`n" Line

	StopTime := A_TickCount
	CountTime := StopTime - StartTime
	MsgBox ,, Row %A_LineNumber% -> %A_ScriptName%, % "Ex3 `nNumber of rows .: " InputData.Count()"`nRunTime .: " CountTime " ms"
ExitApp

You kept the test code rebuilding a string from the array.

Code: Select all

	DataLines := ""
	For Each, Line In InputData
		DataLines .= "`n" Line

It consumes most of the time if the string gets really large.

[Albireo]

an empty line is "". anything else is by definition not empty. if u have redefined empty to mean something else, then simply checking against "" won't work. a different discriminator is required.

I agree with you, but in my world even a row with only field dividers can be empty. These "empty" lines occur when a tab with data in Excel, OpenOffice calc / LibreOffice calc, is processed with formulas into another tab and this tab is exported to a file with CSV structure.
All cells that lack content in the first tab will be empty cells in the second tab and only the field delimiters is exported for this rows.

e.g. Tab 1 contains 3 columns and 200 rows of data. Tab 2 retrieves data from Flik1 with formulas and performs something.
But in Tab 2, there are 500 rows of formulas (with data from Tab 1).
When Tab 2 is exported to a CSV file, the first 200 rows will contain data, while the 300 following rows will only contain field separators. The same thing happens if empty rows are inserted in the data on Tab 1
Because this happens quite often to me, I see these lines as empty and should be handled and skipped in the CSV-file, when the file is read by AHK.

[swagfag]

Code: Select all

Cleaned := []
Cleaned.SetCapacity(BIG NUMBER)
emptyLineOrJustCommas := "^(?:,*)$"

Loop Parse, csvFile, `n
{
	if !(A_LoopField ~= emptyLineOrJustCommas)
		Cleaned.Push(A_LoopField)
}

[Albireo]

Code: Select all

Cleaned := []
Cleaned.SetCapacity(BIG NUMBER)
emptyLineOrJustCommas := "^(?:,*)$"

Loop Parse, csvFile, `n
{
	if !(A_LoopField ~= emptyLineOrJustCommas)
		Cleaned.Push(A_LoopField)
}

Thanks for your tip, but I don't understand what happens
Specially on line .: emptyLineOrJustCommas := "^(?:,*)$" and maybe if !(A_LoopField ~= emptyLineOrJustCommas)

If I understand correctly, all rows that are empty or just consist of "," (comma) are ignored
But how to control other separators? (like ";" "TAB" or ...)?
Can these be checked in the same code?