How Computers Work
rounding error
a value not exactly representable in binary (0.1)
Source board
This definition comes from the board below. Open it to see the term in context.
Numbers for RealHow Computers WorkMore terms from this board
- epsilon compare|a-b| < tiny instead of ==
- floating pointbinary scientific notation (sign
- IEEE 754the standard (float32 = 1/8/23, float64 = 1/11/52)
- Infoverflowed-to-infinity
- mantissathe significant digits
- modular arithmeticclock/wheel wraparound
- NaN'not a number', ≠ itself
- overflowa result too big for the width, wraps around (mod 2^n)
- signedallows negatives
- the 2038 problem32-bit Unix time overflow
- two's complementnegate by flip-bits-then-+1; one adder does both; range -2^(n-1)..2^(n-1)-1
- unsignednon-negative, 0..2^n-1