Numbers for Real (signed ints, overflow, floating point)

A byte’s worth of switches, and the three sharp edges hiding inside it: a biggest number it can hold, a trick it needs to fake a negative one, and a limit on how exactly it can write a fraction.

Before you start. You need what Everything Is a Number (bits & bytes) gave you: a computer stores everything as a fixed row of on-or-off switches, bits, and eight of them make a byte, which holds exactly 256 different patterns. This lesson is about what happens right at the edge of that fixed row - when a count needs a 257th pattern, or a fraction needs more precision than the row has room for.

4 June 1996: the number that didn’t fit

At 12:34:06 UTC (world standard time) on 4 June 1996, the Ariane 5 rocket lifted off from French Guiana carrying Cluster, four European research satellites. Thirty-seven seconds later it veered off course and its own safety system blew it up. The loss: more than $370 million, one of the most expensive software bugs on record.

Engineers traced it to a single line of leftover code, copied from the previous rocket, Ariane 4, which had never flown this fast. That line took a measurement of the rocket’s sideways speed - a number with room for fractions and a huge range - and squeezed it into a slot built for a much smaller kind of number. It did not fit. No alarm went off. The rocket kept flying, steering itself using its own garbage.

That bug is this whole lesson in miniature. A computer stores every number in a fixed number of switches, the way a cheap calculator has a fixed number of digit slots. Overflow a calculator and it shows an error. Overflow a computer and something worse happens: it keeps the bits that fit, throws the rest away, and calls the wrong answer a result.

The wheel that runs out of room

Picture the trip counter in a car, a row of wheels that count upward as you drive. Each wheel has ten positions, so the counter only has as many total readings as its wheels allow. A computer’s counter uses the same idea with two positions per wheel - on or off - so eight wheels, a byte, have exactly 256 positions, 0 through 255.

Run the counter past its last position and it does not stop. It goes 255, then 0 again, the same way a car’s trip odometer rolls from 999.9 back to 000.0. This silent wrap is called overflow.

flowchart LR
  A["253"] -->|"+1"| B["254"]
  B -->|"+1"| C["255"]
  C -->|"+1, wheel turns"| D["0"]

It happens on real machines, watched live. On 1 December 2014, YouTube’s view counter for “Gangnam Style” ran past 2,147,483,647 views - the largest number a very common wheel size, 32 positions with one spent on a sign, can hold - and YouTube had to move the count onto a bigger wheel before it could keep climbing.

The same wheel, counting seconds instead of views, is why a lot of computers will misbehave one second after 03:14:07 UTC on 19 January 2038. That instant is 2,147,483,647 seconds after clocks started counting from 1 January 1970 - a year is about 31.5 million seconds, so 2,147,483,647 ÷ 31,500,000 ≈ 68 years, 1970 + 68 = 2038 - and the next second the wheel rolls over to a date in 1901.

Half the wheel, relabelled: two’s complement

A counting wheel is fine for counting up, but what about negative numbers? A calculator has a spare key for the minus sign. A fixed row of on/off switches has no spare switch to spend on one - every switch is already busy holding digits. So relabel the top half of the wheel’s positions as negative instead. That trick has a name: two’s complement.

To find the negative of a number: flip every switch, then add one.

flowchart LR
  X["+5 = 00000101"] -->|"flip every bit"| F["11111010"]
  F -->|"add 1"| R["11111011 = -5"]

+5   =  0000 0101
flip =  1111 1010
+1   =  1111 1011   =  -5

Why it works, in one byte: flipping 5 gives 255 − 5 = 250, and adding one gives 251. Now 5 + 251 = 256, one full turn of the wheel, which a byte shows as 0. So a number plus its own negative always lands on zero, which is the one thing “negative” has to mean.

On a byte-sized wheel that split gives a range from -128 to +127:

1000 0000 = -128 (most negative)   0111 1111 = +127 (max)   1111 1111 = -1

That’s the whole payoff: one relabelled wheel does the job of two separate machines, because the switches never encode which kind of number they hold - only the label at the end does.

There is one wart: one more negative position than positive, because zero eats a slot on the positive side and no room is left for a matching “positive 128”. Flip and add one to -128 and you land right back on -128.

That last edge is exactly where Ariane’s software broke. The converted speed number had to fit inside a 16-bit wheel - a signed range of -32,768 to +32,767 - and the rocket, flying faster than Ariane 4 ever had, produced a value bigger than that. Same trick, same silent overflow, a $370 million bill.

Numbers that don’t end: what 0.1 really is

A calculator with, say, ten digit slots has a trick for numbers too big or small to fit: scientific notation. Instead of writing every zero, it shows a handful of digits and a separate number saying how far to slide the decimal point - 6.022, then × 10²³. A computer’s fractions use exactly this idea, in base 2. The format is called floating point.

The everyday kind, float64, spends its switches on three fields: a sign, eleven bits for how far to slide the point (the exponent), and fifty-two for the digits themselves (the mantissa).

flowchart LR
  V["0.1, the true value"] -->|"cut the pattern to 52 mantissa bits, round"| S["0.1000000000000000055..., stored"]

More mantissa switches buy precision; more exponent switches buy range; the total is fixed, so a computer trades one against the other and never gets more of both.

Here is the edge that bites almost every program. Write 0.1 in binary the way you’d write 1/3 in decimal - by doubling and reading off the whole part each time:

 0.1 × 2 = 0.2   → 0
 0.2 × 2 = 0.4   → 0
 0.4 × 2 = 0.8   → 0
 0.8 × 2 = 1.6   → 1   (keep 0.6)
 0.6 × 2 = 1.2   → 1   (keep 0.2)
 0.2 × 2 = 0.4   → 0   ← back at 0.2: the cycle 0011 repeats forever

It never ends. So the mantissa’s fifty-two switches cut the pattern off and round, storing a value a hair above one tenth. 0.2 gets the same treatment, and the exact sum of the two rounded values lands a hair above 0.3 - while stored 0.3 itself rounds down. That gap is why, on every computer built to this standard:

0.1 + 0.2 = 0.30000000000000004

Nothing is broken. Three roundings simply didn’t cancel out. The fix, whenever code compares two fractions, is to check that they’re close rather than identical, and to never use floating point for money, where sums must match to the cent.

25 February 1991: a third of a second, twenty-eight soldiers

A Patriot missile battery, a launcher unit, tracks an incoming target using a clock, and the battery at Dhahran, Saudi Arabia, stored its clock’s tenths-of-a-second ticks as a rounded fraction, the same kind of rounding you just watched happen to 0.1. Each tick the error was tiny, about a ten-millionth of a second. But the battery had been running continuously for 100 hours by 25 February 1991, and a tiny error times enough ticks stops being tiny: by then the clock had drifted by about a third of a second.

The battery’s radar used that clock to predict where an incoming Iraqi Scud missile would be, and a third of a second of drift moved the prediction about 600 metres from the missile’s real position - far enough that the system decided there was nothing there, and never fired. The Scud that got through killed 28 American soldiers. The fix, once found, was two lines of code correcting the drift before it built up; the flaw had been sitting there for years.

You now know

• A computer stores every number in a fixed number of switches, so there is always a biggest number, and the machine does not warn you when a calculation goes past it.
• Numbers with no minus sign overflow by wrapping back to zero, the way a car’s odometer rolls from its highest reading back to its lowest.
• Two’s complement stores a negative number by relabelling half the range as negative, using the same switches and adding circuit as positive numbers, with no separate minus sign.
• Floating point trades range for precision by splitting its switches between a sign, an exponent, and a mantissa, which is why 0.1 + 0.2 does not come out exactly 0.3.
• Both failures are silent: the machine hands back a wrong number and keeps going; a rocket and a missile defence have both been lost that way.

Stop here if you want. What follows is the same edges, in the exact bits, for when you want to watch one of them break.

Second read: watching the edges break, in the exact bits

JavaScript’s typed arrays (Uint8Array, Int8Array, Int32Array) store real fixed-width wheels and wrap exactly the way hardware does, which makes them the cheapest way to watch every edge above actually happen, including the two’s complement flip done bit by bit and the two 32-bit edges (overflow, and 2038) computed rather than quoted:

const u = new Uint8Array(1), s = new Int8Array(1);
u[0] = 255; u[0] += 1;            console.log("unsigned 255 + 1  =", u[0]);
s[0] = 127; s[0] += 1;            console.log("signed   127 + 1  =", s[0]);
s[0] = -128; s[0] = -s[0];        console.log("signed  -(-128)   =", s[0]);
s[0] = -128; s[0] = Math.abs(s[0]); console.log("signed  abs(-128) =", s[0]);
// two's complement by hand: flip, add 1
const bits8 = n => (n & 0xff).toString(2).padStart(8, "0");
const five = 5, flipped = ~five & 0xff, negated = (flipped + 1) & 0xff;
console.log("+5      ", bits8(five));
console.log("flip    ", bits8(flipped));
console.log("+1      ", bits8(negated), "= as int8:", new Int8Array([negated])[0]);
// the same adder, both readings: 250 + 10 as unsigned and as signed
const sum = (250 + 10) & 0xff;
console.log("250+10 -> byte", bits8(sum), "= unsigned", sum, "= signed", new Int8Array([sum])[0], "(-6 + 10 = 4)");
// the two famous 32-bit edges
const i32 = new Int32Array(1);
i32[0] = 2147483647; i32[0] += 1;  console.log("int32 max + 1     =", i32[0]);
i32[0] = 2147483647;               console.log("2038 problem: int32 max as a unix time =", new Date(i32[0] * 1000).toISOString());

unsigned 255 + 1  = 0
signed   127 + 1  = -128
signed  -(-128)   = -128
signed  abs(-128) = -128
+5       00000101
flip     11111010
+1       11111011 = as int8: -5
250+10 -> byte 00000100 = unsigned 4 = signed 4 (-6 + 10 = 4)
int32 max + 1     = -2147483648
2038 problem: int32 max as a unix time = 2038-01-19T03:14:07.000Z

Four lines worth reading twice. Wraparound is silent every time - no exception, just the next-door number. abs(-128) returning -128 is the surprise, and it’s the wart from the story: flip 1000 0000 and you get 0111 1111, add one and you’re back at 1000 0000, because there’s no +128 slot to land on. 250 + 10 on one byte reads as 4 whether the byte is unsigned or signed - 250 + 10 = 260 = 4 mod 256, and -6 + 10 = 4 - because the adder circuit never knew which kind of number it was holding; only the label at the end decides. And the last line is 2038 computed, not quoted: the largest signed 32-bit value read as seconds since 1970 lands on 03:14:07 UTC, 19 January 2038, to the second.

Now the exact bits of a float, decoded the way the board does it by hand. A DataView lets code write a number into raw bytes and read the same bytes back as one big integer, split into the three fields floating point promised: sign, exponent, mantissa.

const f32bits = x => { const dv = new DataView(new ArrayBuffer(4)); dv.setFloat32(0, x); return dv.getUint32(0); };
const decode32 = u => {
  const sign = u >>> 31, exp = (u >>> 23) & 0xff, man = u & 0x7fffff;
  const value = (sign ? -1 : 1) * (1 + man / 2 ** 23) * 2 ** (exp - 127);
  return { hex: "0x" + u.toString(16).padStart(8, "0").toUpperCase(), sign, exp, expUnbiased: exp - 127, mantissa: "1." + man.toString(2).padStart(23, "0"), value };
};
console.log(decode32(0xC0C80000));           // the x-ray board's example
console.log(decode32(f32bits(0.1)));         // 0.1 as float32
console.log("0.1 as float32 exactly:", new Float32Array([0.1])[0].toFixed(30));
console.log("0.1 as float64 exactly:", (0.1).toFixed(55));
console.log("0.2 as float64 exactly:", (0.2).toFixed(55));
console.log("0.3 as float64 exactly:", (0.3).toFixed(55));
console.log("0.1 + 0.2 =", 0.1 + 0.2, " misses 0.3 by", 0.1 + 0.2 - 0.3);
// the gap between neighbours (1 ULP) grows with magnitude
const next = x => { const dv = new DataView(new ArrayBuffer(8)); dv.setFloat64(0, x); dv.setBigUint64(0, dv.getBigUint64(0) + 1n); return dv.getFloat64(0); };
for (const x of [1, 1000, 1e6, 1e9, 1e15, 2 ** 53, 1e18]) console.log(`gap after ${x.toExponential(0).padStart(6)} = ${(next(x) - x).toExponential(2)}`);
console.log("2**53 + 1 === 2**53 ?", 2 ** 53 + 1 === 2 ** 53);
// the Patriot drift, from the rounding of 0.1 in a 24-bit register (23 bits after the point)
const tenth24 = Math.floor(0.1 * 2 ** 23) / 2 ** 23;
const err = 0.1 - tenth24, ticks = 100 * 3600 * 10;
console.log("0.1 in 23 fraction bits =", tenth24, " error per tick =", err.toExponential(3), "s");
console.log("after 100 h =", ticks, "ticks, drift =", (err * ticks).toFixed(4), "s");
// the corners
console.log("1/0 =", 1 / 0, " 0/0 =", 0 / 0, " NaN === NaN ?", NaN === NaN, " -0 === 0 ?", -0 === 0, " 1/-0 =", 1 / -0);

{
  hex: '0xC0C80000',
  sign: 1,
  exp: 129,
  expUnbiased: 2,
  mantissa: '1.10010000000000000000000',
  value: -6.25
}
{
  hex: '0x3DCCCCCD',
  sign: 0,
  exp: 123,
  expUnbiased: -4,
  mantissa: '1.10011001100110011001101',
  value: 0.10000000149011612
}
0.1 as float32 exactly: 0.100000001490116119384765625000
0.1 as float64 exactly: 0.1000000000000000055511151231257827021181583404541015625
0.2 as float64 exactly: 0.2000000000000000111022302462515654042363166809082031250
0.3 as float64 exactly: 0.2999999999999999888977697537484345957636833190917968750
0.1 + 0.2 = 0.30000000000000004  misses 0.3 by 5.551115123125783e-17
gap after   1e+0 = 2.22e-16
gap after   1e+3 = 1.14e-13
gap after   1e+6 = 1.16e-10
gap after   1e+9 = 1.19e-7
gap after  1e+15 = 1.25e-1
gap after  9e+15 = 2.00e+0
gap after  1e+18 = 1.28e+2
2**53 + 1 === 2**53 ? true
0.1 in 23 fraction bits = 0.09999990463256836  error per tick = 9.537e-8 s
after 100 h = 3600000 ticks, drift = 0.3433 s
1/0 = Infinity  0/0 = NaN  NaN === NaN ? false  -0 === 0 ? true  1/-0 = -Infinity

Take it one block at a time. The board’s own example, 0xC0C80000, splits into sign 1, exponent field 129, mantissa 1.1001.... Subtract the bias of 127 to get 2. Read the mantissa as 1 + 1/2 + 1/16 = 1.5625. Multiply: -1.5625 × 2² = -6.25, the same answer the board reaches by hand:

flowchart LR
  H["hex 0xC0C80000"] --> B["32 bits"]
  B --> S["sign bit: 1"]
  B --> E["8 exponent bits: 129"]
  B --> M["23 mantissa bits: 1001..."]
  S --> F["value = (-1)^sign x 1.mantissa x 2^(exp - 127)"]
  E -->|129 - 127 = 2| F
  M -->|1 + 1/2 + 1/16 = 1.5625| F
  F --> V["-6.25"]

Stored 0.1, 0x3DCCCCCD, is a hair above the true value; stored 0.2 is a hair above its true value too; their exact sum lands above 0.3, while stored 0.3 itself is the nearest float below 0.3. The sum rounds to the float above, one step away - that’s the 5.55e-17 miss, one ULP (the gap to the neighbouring float) at that size. These exact bit layouts, sign, exponent, mantissa, are also what crosses the wire and gets stored on disk - the number encodings Encoding and Schema Change assumes when it talks about schemas surviving a format change.

The same gap widens everywhere: 2.2 × 10⁻¹⁶ near 1, 0.125 by 10¹⁵, and exactly 2 at 2⁵³ (about 9 quadrillion) - which is why 2**53 + 1 === 2**53 prints true. Integers above that line silently lose their odd values, which is why large ids travel as BigInt or as strings in JSON rather than as ordinary numbers.

The last two lines recompute the Patriot number instead of quoting it. 0.1 stored in the missile’s 23-bit register was really 0.09999990463256836, an error of 9.537 × 10⁻⁸ seconds per tick. A tick every tenth of a second for 100 hours is 3,600,000 ticks, and 3,600,000 × 9.537e-8 = 0.3433 seconds, the drift in the incident report.

The final console line shows the reserved corners: dividing by zero gives Infinity, not a crash; 0/0 is NaN; and NaN is never equal to itself, even to itself, which is why code checks Number.isNaN(x) rather than x === NaN.

Every rule above collapses into one decision, the one every schema and every API has to make:

flowchart TD
  Q{What is the number?} -->|money, ids, counts| I[integer, 64 bits wide]
  Q -->|a measurement or a ratio| F[float64]
  Q -->|a weight in a neural net| B[bf16 or float16, range over precision]
  I --> C{Will it ever exceed 9.2e18, the 64-bit signed max?}
  C -->|no| OK[done]
  C -->|yes| BI[BigInt or a decimal type]
  F --> E{Comparing two of them?}
  E -->|yes| EPS[compare within an epsilon, never ==]
  E -->|no| OK

If the value is counted rather than measured, a float is the wrong type before any arithmetic even happens.

Try it

import struct
print(0.1 + 0.2)                       # 0.30000000000000004
print((0.1 + 0.2) == 0.3)              # False
print(struct.pack(">f", 0.1).hex())    # 3dcccccd - see the rounded bits
import numpy as np
x = np.int8(127); print(x + np.int8(1))  # -128  (signed overflow, with a warning)
print(bin(np.uint8(256)))              # wraps to 0

Without numpy, node does the same in one line each:

node -e 'const s=new Int8Array([127]); s[0]+=1; console.log(s[0])'                 # -128
node -e 'console.log((0.1).toFixed(55))'                                              # the real 0.1
node -e 'console.log(2**53+1===2**53, Number.MAX_SAFE_INTEGER)'                       # true 9007199254740991

Then change the 0xC0C80000 in the decoder listing to 0x3F800000 and predict the answer before you run it (sign 0, exponent 127, mantissa all zeros). Then try 0x7F800000 and 0x7FC00000, the two reserved corners with every exponent bit set. The naive decoder prints 3.4e38 and 5.1e38 for them, and both answers are wrong: read the same patterns back with dv.getFloat32(0) and the hardware says Infinity and NaN. The formula stops applying when the exponent field is all ones, and a real decoder needs that special case - the listing leaves it out so you can see the corner from the outside.

Go deeper

• What Every Computer Scientist Should Know About Floating-Point - Goldberg (the canonical paper)
• float.exposed & 0.30000000000000004.com (interactive)
• Two’s complement (Ben Eater, video)
• The Ariane 5 failure report.

Related

Byte-level companion: IEEE-754 Decoder - decode 0xC0C80000 → -6.25 by hand, why 0.1+0.2≠0.3 as a number-line grid, the reserved corners (±0, ∞, NaN), and float64/float32/bf16 budgets. Move the dial yourself: the lab on that x-ray board shows what is actually in the 64 bits your program stores for 0.1, and where integers start going missing past 2^53.

The full How Computers Work list is in CURRICULUM.

Retrieval practice

10 questions. Answer each one before you open it.