Vectors, Dot Products and Matrix Multiply

The What a Neural Network Is board. Every arrow on it is a multiply and every box is an add. This lesson is the arithmetic under that drawing, done by hand, slowly enough to see it.

Before you start. You need school arithmetic and nothing more: multiply two numbers, add a short column, and remember that a minus times a minus is a plus. This is the maths floor under What a Neural Network Is, Tokens & Embeddings and Attention & the Transformer, and each of those will feel lighter once the one sum in this lesson is a habit.

2013: a chip built for one sum

In 2013, engineers at Google did a sum on the back of an envelope and did not like the answer. Their speech recognition ran on neural networks, and it worked. So they asked: what if people used voice search for three minutes a day? The answer, in their own words later, was that Google’s datacenters would have to double to keep up.

Until then, the company had run neural networks on ordinary processors, the general-purpose chips in every server. Those chips can do anything, and so they do any one thing only so fast. Spare capacity had made that free. A doubling was not free.

So Google started a rush project to build a chip that could do almost nothing. It could not run a web server or a database. It did one sum, over and over: multiply two numbers, add the result to a running total. At its heart was a grid of 256 by 256 little calculators, 65,536 of them, all doing that one sum at the same moment. They called it the Tensor Processing Unit, designed, built and running in datacenters within 15 months, in service from 2015. The paper describing it came out in 2017.

A whole chip for “multiply, then add”. This lesson is what that sum is, what question it answers, and how a few of them make one step of a neural network.

A vector: a film described as a short list of numbers

Start with something a computer cannot eat: a film. It eats numbers only. So describe the film with numbers.

Pick three things you care about, always in the same order: how much action, how much romance, how much comedy, each out of 5. These numbers are made up for this lesson. A heist film might score 4 for action, 1 for romance and 3 for comedy. Write that as one list in square brackets:

heist  = [4, 1, 3]      action, romance, comedy
caper  = [5, 0, 4]
weepie = [0, 5, 1]

The order is the whole trick: the second slot always means romance.

A list of numbers like this, where each position has a fixed meaning, is a vector. That is all the word means.

There is a second way to see one, and both matter. Drop romance for a moment and keep two numbers, action and comedy. Now the heist film is a point on graph paper: 4 along, 3 up. Draw an arrow from the corner to that point. The caper, at 5 along and 4 up, is an arrow pointing almost the same way. The weepie, at 0 along and 1 up, points straight up the page, off in its own direction.

flowchart LR
  O["corner: 0 action, 0 comedy"] -->|"4 along, 3 up"| H["heist"]
  O -->|"5 along, 4 up"| C["caper"]
  O -->|"0 along, 1 up"| W["weepie"]

Films of the same kind point the same way. Keep that picture. A vector is a description when you read it, and an arrow when you draw it, and the next sum works on both.

The dot product: multiply the pairs, add them up

Now you. Say one point of action is worth 2 to you, one point of romance costs you 1 (you get bored), and one point of comedy is worth 1. Your taste is also a vector, in the same order:

you = [2, -1, 1]

How much will you enjoy the heist film? Line the two lists up, multiply each pair, add:

action:   4 × 2    =  8
romance:  1 × (-1) = -1
comedy:   3 × 1    =  3
                    ----
                     10

That number is a dot product: two vectors of the same length, multiplied slot by slot, then added. It is the sum from the Google chip, done three times and totalled.

The weepie gives 0 × 2 + 5 × (-1) + 1 × 1 = -4. The caper gives 5 × 2 + 0 + 4 × 1 = 14. You would pick the caper.

Read what the number means. Every slot where the film has a lot of something you like adds a lot. A slot where one of you has a zero adds nothing. A slot where they pull opposite ways, the film full of romance and you against it, subtracts. So the dot product is how much two lists agree. Big and positive: they point the same way. Near zero: they have nothing to do with each other. Negative: they point apart.

It works between two films as well. Heist against caper is 4×5 + 1×0 + 3×4 = 32. Heist against weepie is 0 + 5 + 3 = 8. The heist agrees with the caper four times as much.

A matrix: many dot products at once

One viewer is not a cinema. Add two more, each a taste vector, and stack all three as rows of one table:

            action  romance  comedy
you       [   2      -1        1  ]
romantic  [  -1       3        0  ]
comic     [   0       0        2  ]

A table of numbers in rows and columns is a matrix. This one holds three tastes.

Multiplying the matrix by the heist vector means: take each row, dot it with the film, write the answer down.

flowchart LR
  F["heist [4, 1, 3]"] -->|"row 1: 8 - 1 + 3"| A["you: 10"]
  F -->|"row 2: -4 + 3 + 0"| B["romantic: -1"]
  F -->|"row 3: 0 + 0 + 6"| C["comic: 6"]

In: one vector of three numbers, a film. Out: another vector of three numbers, [10, -1, 6]. That is matrix multiply. Nothing new, only the dot product done once per row.

Look at what just happened to the film. It went in described by what is in it (action, romance, comedy) and came out described by who likes it (you, the romantic, the comic). Same film, new description. A matrix is a machine for re-describing a thing.

Put three films side by side and you get nine dot products, three viewers by three films, one grid out. Google’s chip does 65,536 multiply-and-adds in a single tick: this grid, far larger.

One layer: y = Wx + b, then the zero rule

Two small additions turn this into one step of a neural network.

First, some viewers are harder to please before the film even starts. Give each one a fixed amount added to their score: you start at -4 (a long day), the romantic at 0, the comic at +1. That fixed extra per row is the bias.

scores  [10, -1, 6]
bias  + [-4,  0, 1]
        -----------
        [ 6, -1, 7]

Second, a rule: any score below zero becomes zero. A viewer who would hate the film does not report hating it. They just do not show up. The romantic’s -1 becomes 0. The rule is an activation, and this one, “negative becomes zero”, is the one What a Neural Network Is calls ReLU.

after the zero rule: [6, 0, 7]

Written the short way, one layer is:

y = activation(W x + b)

In English: take the input list x, dot it with every row of the table W, add each row’s fixed extra b, then zero anything negative. The numbers in W are the network’s weights, the dials What a Neural Network Is shows being turned. The film became a new vector, and that new vector can be the input to the next layer, which re-describes it again.

That is all a neural network computes, layer after layer. A large language model does it with tables thousands of numbers wide, for every word, which is why Google built a chip that does nothing else.

You now know

• A vector is a list of numbers where each position has a fixed meaning, and it can be read as a description or drawn as an arrow.
• A dot product multiplies two same-length lists slot by slot and adds the results; the total says how much the two agree.
• A positive dot product means two vectors point the same way, near zero means unrelated, negative means they point apart.
• Multiplying a matrix by a vector is one dot product per row, and it turns one description of a thing into another.
• One neural-network layer is y = activation(W x + b): a matrix multiply, a fixed extra per row, then a rule such as “negative becomes zero”.
• Chips like Google’s TPU exist because nearly all of that work is multiply-then-add, done in huge grids at once.

Stop here if you want. What follows is the same arithmetic in notation and in code, and where it turns up again in embeddings and attention.

Second read: the sum in notation, in code, and inside a model

Every number about films and viewers in this lesson is a teaching value, chosen so the arithmetic fits in your head; real embeddings are learned, not typed in, and their slots have no names. The chip numbers are from Google’s 2017 paper (linked below). Every code output here was run with python3 and numpy 2.4.6 on 2026-09-27.

The dot product, written out. For two vectors of length n:

a · b = a1·b1 + a2·b2 + ... + an·bn

In English: pair up the slots, multiply each pair, add everything. The two vectors must be the same length, or some slot has no partner.

No magic in it: plain Python. A dot product is a loop with a running total, and a matrix times a vector is that loop once per row.

def dot(a, b):
    total = 0
    for x, y in zip(a, b):      # multiply each pair...
        total += x * y          # ...and keep a running sum
    return total

def matvec(W, x):
    return [dot(row, x) for row in W]   # one dot product per row

W = [[2, -1, 1], [-1, 3, 0], [0, 0, 2]]
print(matvec(W, [4, 1, 3]))

[10, -1, 6]

The same, in numpy. The @ sign is matrix multiply. Here is the whole story half in one script:

import numpy as np
# films described as [action, romance, comedy], scores 0-5 (made up)
heist  = np.array([4, 1, 3])
caper  = np.array([5, 0, 4])
weepie = np.array([0, 5, 1])
you    = np.array([2, -1, 1])          # your taste: what one point of each is worth to you

print("you . heist :", you @ heist)
print("you . caper :", you @ caper)
print("you . weepie:", you @ weepie)

W = np.array([[ 2, -1, 1],     # you
              [-1,  3, 0],     # the romantic
              [ 0,  0, 2]])    # the comedy fan
print("W @ heist   :", W @ heist)

films = np.column_stack([heist, caper, weepie])   # one film per column
print("W @ films   :\n", W @ films)

b = np.array([-4, 0, 1])
z = W @ heist + b
print("W @ heist + b      :", z)
print("after the zero rule:", np.maximum(0, z))

you . heist : 10
you . caper : 14
you . weepie: -4
W @ heist   : [10 -1  6]
W @ films   :
 [[10 14 -4]
 [-1 -5 15]
 [ 6  8  2]]
W @ heist + b      : [ 6 -1  7]
after the zero rule: [6 0 7]

Read the 3 by 3 grid one way and it is three viewers (rows) meeting three films (columns). The romantic’s row, [-1, -5, 15], says the weepie is the only film for them. Row i, column j of the answer is the dot product of row i of W with column j of films.

The shape rule. A table with m rows and n columns times a vector of length n gives a vector of length m. In English: the width of the table must match the length of the input, and the output has one number per row. It costs m × n multiply-and-adds. Our table: 3 × 3 = 9. A layer that takes 768 numbers in and gives 3,072 out, the size What a Neural Network Is uses for GPT-2 small, is 768 × 3,072 = 2,359,296 multiply-and-adds for every single token.

Rows or columns: the same thing turned over. Each row of W here is what What a Neural Network Is calls a neuron: one dot product, one output. You will meet two ways of laying the same neurons out, so it is worth seeing both once. That lesson writes a layer as x @ W, with one neuron per column. This lesson wrote W @ x, one neuron per row. They are the same numbers read the other way, and they agree:

W = np.array([[2,-1,1],[-1,3,0],[0,0,2]]); x = np.array([4,1,3])
print('W @ x   :', W @ x); print('x @ W.T :', x @ W.T)

W @ x   : [10 -1  6]
x @ W.T : [10 -1  6]

W.T is W flipped so rows become columns (the transpose). Libraries pick whichever way round is faster for them; the sums inside are identical.

Length, and why agreement needs dividing. The dot product grows with how big the vectors are, not only with where they point. A film with every score doubled would agree “twice as much” with everything. To measure direction alone, divide by both lengths. A vector’s length is the square root of its dot product with itself, the distance from the corner to its point:

|a| = sqrt(a · a)
cos(a, b) = (a · b) / (|a| × |b|)

In English: dot the two, then divide out their sizes, and what is left runs from 1 (same direction) through 0 (unrelated) to -1 (opposite). This is the cosine similarity.

heist, caper, weepie = np.array([4,1,3]), np.array([5,0,4]), np.array([0,5,1])
length = lambda v: float(np.sqrt(v @ v))
cos    = lambda a, b: float(a @ b) / (length(a) * length(b))
print("heist . caper :", heist @ caper,  " cosine", round(cos(heist, caper), 3))
print("heist . weepie:", heist @ weepie, " cosine", round(cos(heist, weepie), 3))
print("lengths:", round(length(heist),3), round(length(caper),3), round(length(weepie),3))

heist . caper : 32  cosine 0.98
heist . weepie: 8  cosine 0.308
lengths: 5.099 6.403 5.099

By hand for the first line: 32 / (5.099 × 6.403) = 32 / 32.65 = 0.98. The heist and the caper point almost exactly the same way.

Why a chip is a grid. The TPU paper describes a matrix unit of 256 × 256 multiply-and-add cells, 65,536 of them, clocked at 700 MHz. Count each multiply-and-add as two operations: 65,536 × 2 × 700,000,000 = 91.75 trillion per second, which is the paper’s quoted peak of 92 trillion (for 8-bit whole numbers, not decimals). The grid shape is the matrix shape: weights sit in the cells, inputs flow through, and one row of 256 partial sums comes out per tick.

flowchart LR
  X["input vector, 256 numbers"] -->|"flows across"| G["256 x 256 grid of weights: 65,536 multiply-adds per tick"]
  G -->|"one row of 256 sums per tick"| A["running totals"]
  A -->|"add bias, apply the zero rule"| Y["output vector"]

Where you meet the dot product next.

• Embeddings. Tokens & Embeddings gives every token a vector, like the film vectors here but learned, with hundreds of unlabelled slots. “Which words are similar” is the cosine above. Its toy vectors give king · queen = 0.81 − 0.09 + 0.64 = 1.36 and king · banana = 0: banana shares no slot with king, so nothing adds up. Search over documents, the heart of RAG (Retrieval-Augmented Generation), is the same cosine run against a whole library.
• Attention scores. In Attention & the Transformer, each word has a query (what it is looking for) and a key (what it offers), both vectors. The score of one word for another is the dot product of the first’s query with the second’s key. Stack all the queries as rows of Q and all the keys as rows of K, and every score at once is one matrix multiply, Q @ K.T. On that lesson’s hand-picked vectors:

Q = np.array([[1,0],[0,1],[2,1],[1,1]])   # the, cat, sat, down
K = np.array([[0,1],[2,0],[1,1],[0,0]])
print(Q @ K.T)

[[0 2 1 0]
 [1 0 1 0]
 [1 4 3 0]
 [1 2 2 0]]

The third row is sat asking: 1, 4, 3, 0, with its biggest agreement on cat. That is the raw row Attention & the Transformer divides by √2 and turns into shares of 7%, 62% and 31%. Turning agreement scores into shares that add up to 1 is softmax, and it is the next lesson, The Maths Floor, Part 2: Probability, Softmax and Temperature.

Try it

Change your own taste and see which film wins, then give the cinema a fourth viewer:

import numpy as np
films = np.column_stack([[4,1,3], [5,0,4], [0,5,1]])   # heist, caper, weepie
W = np.array([[ 2, -1, 1],
              [-1,  3, 0],
              [ 0,  0, 2],
              [ 1,  1, 1]])      # a fourth viewer who likes everything equally
print(W @ films)

[[10 14 -4]
 [-1 -5 15]
 [ 6  8  2]
 [ 8  9  6]]

A fourth row in the table is a fourth number out: the output got longer because the table got taller, and the input did not change. Now set that viewer to [0, 0, 0] and predict the row before you run it.

Go deeper

• 3Blue1Brown - Essence of linear algebra (the visual series)
• 3Blue1Brown - Dot products and duality
• Jouppi et al., In-Datacenter Performance Analysis of a Tensor Processing Unit (2017), the paper with the voice-search projection and the 256 × 256 grid.

Related

• What a Neural Network Is builds the neuron and the zero rule from history; every row of W here is one of its neurons.
• Tokens & Embeddings is the vector as a description, learned instead of typed in, and cosine as the measure of “near”.
• Attention & the Transformer is the dot product as a question: every query against every key, in one matrix multiply.
• The Maths Floor, Part 2: Probability, Softmax and Temperature is part 2 of the maths floor: how a row of scores becomes shares that add to 1.

Retrieval practice

8 questions. Answer each one before you open it.