SetFormat inconsistent behaviour

Report problems with documented functionality
Posts: 221
Joined: 05 Dec 2016, 01:51

SetFormat inconsistent behaviour

03 May 2019, 13:26

MsgBox tells me my vars are equivalent, but behind the scenes they aren't.
Can't debug my script with MsgBox because it's telling me lies :(

Code: Select all

#SingleInstance Force
SetFormat , Float , 0.3  ;also, removing this makes both tests not equal,
						 ;but only if I don't put it in any functions

	;Result: equal
	Var1 := ((17/255)**(1/2))**(2)
	Var2 := (17/255)
	If ( var1 = var2 )
		MsgBox Comparing vars`n%Var1% is equal to %Var2%
		MsgBox Comparing vars`n%Var1% is not equal to %Var2%

	;Result: not equal
	Array1 := [ ((17/255)**(1/2))**(2) ]	
	Array2 := [ (17/255) ]
	If ( Array1[1] = Array2[1] )
		MsgBox % "Comparing arrays`n" . Array1[1] . " is equal to " . Array2[1]
		MsgBox % "Comparing arrays`n" . Array1[1] . " is not equal to " . Array2[1]
just me
Posts: 6260
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: SetFormat inconsistent behaviour

04 May 2019, 03:52

[v1.0.48+]: Floating point variables have about 15 digits of precision internally unless SetFormat Float (i.e. the slow mode) is present anywhere in the script. In that case, the stored precision of floating point numbers is determined by DecimalPlaces (like it was in pre-1.0.48 versions). In other words, once a floating point result is stored in a variable, the extra precision is lost and cannot be reclaimed without redoing the calculation with something like SetFormat, Float, 0.15. To avoid this loss of precision, avoid using SetFormat Float anywhere in the script, or use SetFormat FloatFast instead.
I think the 'loss of precision' only happens if 'a floating point result is stored in a variable'.

Regardless of whether slow or fast mode is in effect, floating point results and variables are rounded off to DecimalPlaces whenever they are displayed or converted to a string of text (e.g. MsgBox or FileAppend). To see the full precision, use something like SetFormat, FloatFast, 0.15.
Posts: 221
Joined: 05 Dec 2016, 01:51

Re: SetFormat inconsistent behaviour

04 May 2019, 11:36

If I SetFormat to 15 or 16 decimal places, the result is the same, but 17 decimal places and the numbers are no longer identical in the MsgBox. Putting SetFormat 15 or 16 decimal places directly before the calculation and MsgBox, doesn't change this behaviour.

Return to “Bug Reports”

Who is online

Users browsing this forum: Google [Bot] and 4 guests