Format not enough precise
Posted: 03 Dec 2022, 10:49
This example
fNum := 100.22
sTest2 := Format( "{:03.15f}", fNum )
returns sTest = 100.21999999
fNum := 100.22
sTest2 := Format( "{:03.15f}", fNum )
returns sTest = 100.21999999
Let's help each other out
https://www.autohotkey.com/boards/
https://www.autohotkey.com/boards/viewtopic.php?f=14&t=111140
Code: Select all
fNum := 100.22
sTest2 := Format( "{:03.15f}", fNum )
MsgBox % sTest2 ; 100.220000000000000
Also maybe worth mentioning, I get different results from different ahk versions.https://lexikos.github.io/v2/docs/Concepts.htm#pure-numbers
Note: There are some decimal fractions which the binary floating-point format cannot precisely represent, so a number is rounded to the closest representable number. This may lead to unexpected results. For example:One strategy for dealing with this is to avoid direct comparison, instead comparing the difference. For example:Code: Select all
MsgBox 0.1 + 0 ; 0.10000000000000001 MsgBox 0.1 + 0.2 ; 0.30000000000000004 MsgBox 0.3 + 0 ; 0.29999999999999999 MsgBox 0.1 + 0.2 = 0.3 ; 0 (not equal)
Code: Select all
MsgBox Abs((0.1 + 0.2) - (0.3)) < 0.0000000000000001