Rounding error of Format()

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
slur
Posts: 10
Joined: 13 Jun 2014, 07:05

Rounding error of Format()

12 Jan 2016, 13:07

Format() shows somewhat arbitrary yet consistent errors
What's happening here?

Code: Select all

;SetFormat, FloatFast, 0.6 ; Default 74-6-1
;SetFormat, Float, 0.6 ; 74-386-386
;SetFormat, Float, 0.15 ; 74-18-18
show_rounding_diff(0.05, 1)
show_rounding_diff(0.05, 0.1)
show_rounding_diff(0.05, 0.01)

ExitApp

show_rounding_diff(num, inc) {
	While, ( num < 100 )
		{
		F := Format("{:.1f}", num)
		R := Round(num, 1)
		if ( F <> R )
			{
			output .= num . "`t F: " . F . " `tR: " . R . "`n"
			count++
			}
		num += inc
		}
	msgbox, , %count%, %output%
}

Return to “Ask for Help (v1)”

Who is online

Users browsing this forum: Descolada and 232 guests