Round() inconsistency

Get help with using AutoHotkey and its commands and hotkeys
pneumatic
Posts: 289
Joined: 05 Dec 2016, 01:51

Round() inconsistency

08 May 2019, 11:14

Code: Select all

n := 1.05
MsgBox % Round(n,1)
Image



Code: Select all

n := 1.005
MsgBox % Round(n,2)
Image
Last edited by pneumatic on 11 Nov 2019, 23:55, edited 1 time in total.
swagfag
Posts: 3092
Joined: 11 Jan 2017, 17:59

Re: Round() inconsistency

08 May 2019, 11:47

Code: Select all

MsgBox % Format("1.05`t{:.52f}`n1.005`t{:.52f}", 1.05, 1.005)
Helgef
Posts: 4031
Joined: 17 Jul 2016, 01:02
Contact:

Re: Round() inconsistency

08 May 2019, 11:50

This is no bug, the result is because the number 1.005 cannot be exactly represented as a float. Instead, 1.005 is stored as something sligthly less. 1.05 cannot either but is stored as something sligthly larger. You can use format to display the float with as many decimal places you like. Note that in v1 a := floatliteral is equivalent to f := "floatliteral", this is a common source of confusion and has been fixed in v2 :thumbup: .
Klarion
Posts: 176
Joined: 26 Mar 2019, 10:02

Re: Round() inconsistency

08 May 2019, 13:03

Code: Select all

For Each, x In [1.005, 1.006]
	r .= Round(x, 2) " " Format("{:.52f}", x) "`n"
For Each, x In [1.0005, 1.0006]
	r .= Round(x, 3) " " Format("{:.52f}", x) "`n"
MsgBox % r
1.00 1.0049999999999998934185896359849721193313598632812500
1.01 1.0060000000000000053290705182007513940334320068359375
1.001 1.0004999999999999449329379785922355949878692626953125
1.001 1.0005999999999999339195255743106827139854431152343750
pneumatic
Posts: 289
Joined: 05 Dec 2016, 01:51

Re: Round() inconsistency

08 May 2019, 17:39

Helgef wrote:
08 May 2019, 11:50
This is no bug, the result is because the number 1.005 cannot be exactly represented as a float. Instead, 1.005 is stored as something sligthly less. 1.05 cannot either but is stored as something sligthly larger. You can use format to display the float with as many decimal places you like.

Thanks, I wasn't aware of this. I had assumed it was stored something like an integer 1005 with a decimal place location. Somehow had it in my mind that precision was only a problem when storing really large or really small numbers. But no, it seems floating point has this inherent precision problem even when storing relatively simple numbers like 1.005. Is there any way around it? Round(Round(1.005,3),2) doesn't work.

edit: some context

I've got a list of numbers spaced 0.05 apart like this: 1.00, 1.05, 1.10, 1.15, 1.20 etc.
The user inputs a decimal number, and I want to snap it to the nearest number in the above list.
So I'm going

Code: Select all

UserInput := 1.075
ClosestValue := Round(UserInput/0.05) * 0.05
Result: 1.05 is the closest number, when really it should be 1.1, to be consistent with the direction of rounding always being upwards (or just consistently in any direction). Is floor/ceiling the only way to achieve a consistent rounding rule for any UserInput?

Klarion wrote:
08 May 2019, 13:03
1.001 1.0004999999999999449329379785922355949878692626953125
Weird, this converter is giving totally different numbers

http://www.binaryconvert.com/convert_float.html
https://www.binaryconvert.com/convert_double.html

Not sure if ahk/C++/Windows strictly uses IEEE754 though.
swagfag
Posts: 3092
Joined: 11 Jan 2017, 17:59

Re: Round() inconsistency

09 May 2019, 05:02

@Klarion
that is weird. maybe format 52 is wrong. or round could be wrong.
Klarion
Posts: 176
Joined: 26 Mar 2019, 10:02

Re: Round() inconsistency

09 May 2019, 11:16

something is wrong

Return to “Ask For Help”

Who is online

Users browsing this forum: Hotte, MIRKOSOFT, nitrofenix, Scr1pter, SeparateQuestion4 and 238 guests