SetFormat inconsistent behaviour

[pneumatic]

MsgBox tells me my vars are equivalent, but behind the scenes they aren't.
Can't debug my script with MsgBox because it's telling me lies

Code: Select all

#SingleInstance Force
SetFormat , Float , 0.3  ;also, removing this makes both tests not equal,
						 ;but only if I don't put it in any functions
Test1()
Test2()
return


Test1(){
	
	;Result: equal
	
	Var1 := ((17/255)**(1/2))**(2)
	Var2 := (17/255)
	
	If ( var1 = var2 )
		MsgBox Comparing vars`n%Var1% is equal to %Var2%
	Else
		MsgBox Comparing vars`n%Var1% is not equal to %Var2%
		
}


Test2(){
	
	;Result: not equal
	
	Array1 := [ ((17/255)**(1/2))**(2) ]	
	Array2 := [ (17/255) ]
	
	If ( Array1[1] = Array2[1] )
		MsgBox % "Comparing arrays`n" . Array1[1] . " is equal to " . Array2[1]
	Else
		MsgBox % "Comparing arrays`n" . Array1[1] . " is not equal to " . Array2[1]
	
}

[just me]

[v1.0.48+]: Floating point variables have about 15 digits of precision internally unless SetFormat Float (i.e. the slow mode) is present anywhere in the script. In that case, the stored precision of floating point numbers is determined by DecimalPlaces (like it was in pre-1.0.48 versions). In other words, once a floating point result is stored in a variable, the extra precision is lost and cannot be reclaimed without redoing the calculation with something like SetFormat, Float, 0.15. To avoid this loss of precision, avoid using SetFormat Float anywhere in the script, or use SetFormat FloatFast instead.

I think the 'loss of precision' only happens if 'a floating point result is stored in a variable'.

OTOH:

Regardless of whether slow or fast mode is in effect, floating point results and variables are rounded off to DecimalPlaces whenever they are displayed or converted to a string of text (e.g. MsgBox or FileAppend). To see the full precision, use something like SetFormat, FloatFast, 0.15.

[pneumatic]

If I SetFormat to 15 or 16 decimal places, the result is the same, but 17 decimal places and the numbers are no longer identical in the MsgBox. Putting SetFormat 15 or 16 decimal places directly before the calculation and MsgBox, doesn't change this behaviour.