Page 1 of 1

Why is there such a huge difference in numerical calculations?

Posted: 11 Aug 2018, 03:19
by arcticir

Code: Select all

MsgBox (0-0.25) "`n" (0-0.2)
;-0.25
;-0.20000000000000001

Re: Why is there such a huge difference in numerical calculations?

Posted: 11 Aug 2018, 05:09
by nnnik
Numbers in our PC are represented as finite binary numbers. As finite numbers they have a maximum precision and length.
Integers represent all numbers that are not floating point values in the range from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.
Doubles are a bit more difficult to describe. What I can tell you is that there is a float that perfectly describes the number 0.25 and it's value is 2**-2.
0.2 can not be represented like that. That seems confusing but keep in mind that our PC uses the binary system rather than the decimal system.

Re: Why is there such a huge difference in numerical calculations?

Posted: 11 Aug 2018, 20:33
by lexikos
0.00000000000000001 is not a "huge" difference. It is very small. :P

AutoHotkey v1 hides this imprecision from you by default, by showing only 6 decimal places. This leads to confusion like:

Code: Select all

MsgBox % 0.3 - 0.1  ; 0.200000
MsgBox % 0.2 + 0    ; 0.200000
MsgBox % 0.3 - 0.1 = 0.2 + 0  ; False!
This topic has been covered multiple times on these forums.

Re: Why is there such a huge difference in numerical calculations?

Posted: 12 Aug 2018, 00:56
by arcticir
Sorry, I should search the forum.

Thank you two.