Helgef wrote: ↑
08 May 2019, 11:50
This is no bug, the result is because the number 1.005
cannot be exactly represented as a float. Instead, 1.005
is stored as something sligthly less. 1.05 cannot either but is stored as something sligthly larger. You can use format to display the float with as many decimal places you like.
Thanks, I wasn't aware of this. I had assumed it was stored something like an integer 1005 with a decimal place location. Somehow had it in my mind that precision was only a problem when storing really large or really small numbers. But no, it seems floating point has this inherent precision problem even when storing relatively simple numbers like 1.005. Is there any way around it? Round(Round(1.005,3),2) doesn't work.
edit: some context
I've got a list of numbers spaced 0.05 apart like this: 1.00, 1.05, 1.10, 1.15, 1.20 etc.
The user inputs a decimal number, and I want to snap it to the nearest number in the above list.
So I'm going
Code: Select all
UserInput := 1.075
ClosestValue := Round(UserInput/0.05) * 0.05
Result: 1.05 is the closest number, when really it should be 1.1, to be consistent with the direction of rounding always being upwards (or just consistently in any direction). Is floor/ceiling the only way to achieve a consistent rounding rule for any UserInput?
Klarion wrote: ↑
08 May 2019, 13:03
Weird, this converter is giving totally different numbers
Not sure if ahk/C++/Windows strictly uses IEEE754 though.