Round() inconsistency

[pneumatic]

Code: Select all

n := 1.05
MsgBox % Round(n,1)

Code: Select all

n := 1.005
MsgBox % Round(n,2)

[swagfag]

Code: Select all

MsgBox % Format("1.05`t{:.52f}`n1.005`t{:.52f}", 1.05, 1.005)

[Helgef]

This is no bug, the result is because the number 1.005 cannot be exactly represented as a float. Instead, 1.005 is stored as something sligthly less. 1.05 cannot either but is stored as something sligthly larger. You can use format to display the float with as many decimal places you like. Note that in v1 a := floatliteral is equivalent to f := "floatliteral", this is a common source of confusion and has been fixed in v2 .

[Klarion]

Code: Select all

For Each, x In [1.005, 1.006]
	r .= Round(x, 2) " " Format("{:.52f}", x) "`n"
For Each, x In [1.0005, 1.0006]
	r .= Round(x, 3) " " Format("{:.52f}", x) "`n"
MsgBox % r

1.00 1.0049999999999998934185896359849721193313598632812500
1.01 1.0060000000000000053290705182007513940334320068359375
1.001 1.0004999999999999449329379785922355949878692626953125
1.001 1.0005999999999999339195255743106827139854431152343750

[pneumatic]

This is no bug, the result is because the number 1.005 cannot be exactly represented as a float. Instead, 1.005 is stored as something sligthly less. 1.05 cannot either but is stored as something sligthly larger. You can use format to display the float with as many decimal places you like.

Thanks, I wasn't aware of this. I had assumed it was stored something like an integer 1005 with a decimal place location. Somehow had it in my mind that precision was only a problem when storing really large or really small numbers. But no, it seems floating point has this inherent precision problem even when storing relatively simple numbers like 1.005. Is there any way around it? Round(Round(1.005,3),2) doesn't work.

edit: some context

I've got a list of numbers spaced 0.05 apart like this: 1.00, 1.05, 1.10, 1.15, 1.20 etc.
The user inputs a decimal number, and I want to snap it to the nearest number in the above list.
So I'm going

Code: Select all

UserInput := 1.075
ClosestValue := Round(UserInput/0.05) * 0.05

Result: 1.05 is the closest number, when really it should be 1.1, to be consistent with the direction of rounding always being upwards (or just consistently in any direction). Is floor/ceiling the only way to achieve a consistent rounding rule for any UserInput?

1.001 1.0004999999999999449329379785922355949878692626953125

Weird, this converter is giving totally different numbers

http://www.binaryconvert.com/convert_float.html
https://www.binaryconvert.com/convert_double.html

Not sure if ahk/C++/Windows strictly uses IEEE754 though.

[swagfag]

@Klarion
that is weird. maybe format 52 is wrong. or round could be wrong.

[Klarion]

something is wrong