Build Your Own Redis - From Scratch
The nine boards of this build, one wall at a time, in the order they showed up.
Before you start. You need to know that a running program holds data in its own memory, and that memory empties the moment the program stops. That gap is what this lesson bridges.
Boards
- The Map
- The Socket That Answers
- RESP The Wire Protocol
- The Map SET GET DEL
- One Thread Many Clients
- Keys That Die (Expiry)
- Eviction Memory Is Full
- Surviving A Crash (Persistence)
- Everything
Code: teaching/code/from-scratch/redis/ - one Go module, each stage runs alone:
go run ./01-socket … go run ./07-persistence. Poke it with redis-cli -p 6379 ….
Then go run ./08-whole runs all seven as one server, with expiry, eviction and persistence live at once.
2007, Sicily: a notepad that could not keep up
In 2007 a programmer in Sicily, Salvatore Sanfilippo, known online as antirez, built a small tool called LLOOGG. It showed a website’s owner who was on the site right now, refreshing as visitors clicked and moved - a feature Google’s own visitor-counting service would not add for another four years. LLOOGG had to record every click the instant it happened and answer “what just happened” in a fraction of a second.
He built it the ordinary way first: every click, a row in a database on disk, the same kind of database (MySQL, PostgreSQL) running most of the rest of the web. It worked, for a handful of requests a second. Then real traffic arrived, and the same question that used to answer instantly took longer than the click it described. The database wasn’t doing its job badly; it was the wrong job. A disk file is a filing cabinet down the hall, built to hold everything forever. A live “who’s here right now” count doesn’t want a filing cabinet - it wants a notepad on the desk, scribbled on and read back before the next visitor arrives.
2009: skip the filing cabinet, write the notepad
Sanfilippo’s fix was direct: stop asking a filing-cabinet database to behave like a notepad, and write the notepad instead. He prototyped a program that kept everything in the computer’s own memory, never touched a disk unless told to, and answered questions over the network rather than through a query language. He called it Redis, short for REmote DIctionary Server - a dictionary in the sense of pages you look up by name, not one you read cover to cover.
He posted it to Hacker News, a news site programmers read, in February 2009; a handful of people replied. A year later the software company VMware hired him to work on it full time, and over the next five years the notepad he wrote for one struggling side project answered two billion page views for LLOOGG alone, before it sat inside most of the internet. This lesson builds that notepad from an empty file, hitting the same walls, in the order he hit them.
The walls, in the order they showed up
flowchart LR M["The plan"] --> S["A line to talk on"] S --> R["Bytes that agree where a word ends"] R --> D["A page that finds itself"] D --> E["One clerk, many voices"] E --> X["Notes that expire"] X --> V["A desk that fills up"] V --> P["An office that could lose power"] P --> A["The assembled desk"]
A phone call needs a line open before it needs a conversation, so that comes first. The moment bytes can flow, a stream has no idea where one message stops and the next starts, so the next problem is agreeing on that boundary. Only then can a word actually reach a page to answer it, and one page is fine for a single visitor but not for a thousand. A desk that gets used then starts collecting notes that should not outlive their moment, notes nobody should have kept once room runs low, and finally: the power goes, and every note on an unplugged desk disappears unless something wrote it down twice.
The line to talk on: a socket
Two computers that want to talk open a connection first: one listens on a numbered address, the other dials in, and bytes go each way. Nothing more. Send PING down that connection and it might arrive whole, in two pieces, or stuck to the next thing sent right after - a stream has no idea where one message stops and the next starts. Engineers call this open connection a socket, and that punctuation problem is the first wall this build hits.
Saying the length before the word: RESP
The fix is a rule both ends agree to follow: before every word, say how many bytes it is. “Three letters, S-E-T” leaves nothing to guess. Redis’s version is RESP: how many pieces are coming, then each piece’s length, then its bytes, then a fixed break. SET name chalk has 12 letters of actual meaning. Add a count at the front (three pieces), a length before each piece (3, 4 and 5), and a two-character break after every count, length and piece, and it goes out as 34 characters: 22 of pure bookkeeping, the price of never guessing where a value ends.
A page that finds itself: the map
Once words arrive framed, they need somewhere to land. A key-value map is the notepad itself: hand it a name and it computes which page that name lives on, rather than flipping through every page in turn. That one property is why looking a key up costs about the same whether the notepad holds ten entries or ten million.
One clerk, a thousand voices: the event loop
One visitor talking to one page is easy. The moment a second, third and thousandth connect at once, something must decide whose turn it is without one slow question blocking everyone. Real Redis’s answer is one clerk who never waits: a single thread that asks the operating system which open line has something to say, answers it, and moves on - engineers call that an event loop. No second clerk means no two hands ever reach for the same page, so nothing needs a lock (the “wait, I’m using this” flag that many clerks would have to take turns holding). Go does the opposite by habit: a lightweight helper per visitor, and a lock to keep two helpers off the same page. Same problem, a different hand doing the guarding.
Notes that expire, a desk that fills, an office that could lose power
Some notes shouldn’t outlive their moment - a visitor’s “online right now” flag - so instead of re-reading the whole desk every second, the notepad checks a random few and drops whichever have gone stale. When the desk runs out of room, something must leave to make space; a perfect memory of which note was touched longest ago costs bookkeeping on every read, so the trick is to glance at a handful and drop whichever has sat quietest - close enough to the true oldest, for nearly nothing.
And a desk is only as safe as the room it sits in. The fix for a power cut is two things: a running record of every note as it is written - engineers call this a log - so replaying it rebuilds the desk from nothing, plus a full photocopy taken every so often, so a restart doesn’t replay years of notes to catch up. Real Redis writes that log in the same length-then-word shape RESP already speaks, so replaying it reuses the very reader that took commands off the wire.
You now know
- A live count wants a notepad in memory, not a filing cabinet on disk. That is what Redis is, and why it exists.
- A socket hands you a stream, not messages; you must invent where one command ends and the next begins.
- RESP fixes that by saying each word’s length before the word, so the reader never guesses.
- A hash map finds its page directly from the key, instead of searching for it.
- One event loop with no locks, and many workers that each need a lock, are two answers to the same problem: many voices, one desk.
- Surviving a crash needs a log of every write, replayed from the top, plus a periodic snapshot of the whole desk.
Stop here if you want. What follows is the same nine boards in bytes, milliseconds and real terminal output, for when you want to watch the notepad run.
Second read: the numbers behind each wall
RESP, in bytes. SET name chalk is 12 bytes of meaning; on the wire:
*3\r\n 4 bytes "an array of three"
$3\r\nSET\r\n 9 bytes length, then the word, then the terminator
$4\r\nname\r\n 10 bytes
$5\r\nchalk\r\n 11 bytes
──
34 bytes on the wire, 12 of payload, 22 of framing
The event loop, in bytes. A thread’s stack is ~1 MB; a goroutine’s starts at 2 KB. The 10,000 open connections cancel out between the two designs, so what’s left is the per-client cost:
10,000 threads × 1 MB = 10 GB of stacks (the machine is gone)
10,000 goroutines × 2 KB = 20 MB (a rounding error)
ratio = 500×
That is the C10k problem: not “too many connections” but “too many bytes per idle one”. Real Redis sidesteps it by never spawning a thread per client - one thread, epoll saying which lines are ready.
Expiry, in keys checked. Sampling ~20 keys every 100 ms inspects 20 × 10 = 200 a second, against a map that might hold a million - a full rescan would be 1,000,000 ÷ 200 = 5,000× more work. At 1% expired, one pass finds 20 × 1% = 0.2 dead on average; the 25% rule (sweep again if 5+ of 20 came back dead) catches a denser patch fast.
Eviction, in numbers. Sample K = 5 random keys, evict the oldest of the sample, tracked by a 24-bit access clock per key - within a few percent of a true oldest-first list, for close to no bookkeeping. Policies range from allkeys-lru (approximate oldest) to noeviction (refuse new writes instead).
Persistence, named. The log is AOF (append-only file); the snapshot is RDB, taken by calling fork() so the parent keeps serving while the child writes a frozen 10 GB dataset to disk at almost no extra memory. Boot loads RDB first for a fast baseline, then replays AOF for whatever came after.
Everything. Real Redis layers more on the exact same core, none of it changing the hinge above: replication, a cluster of up to 16,384 hash slots, pub/sub, streams, more data types, scripting, transactions, modules.
One SET, end to end
One write through every layer, each hop one of the walls above:
sequenceDiagram participant C as client participant S as socket participant R as RESP reader participant M as the map participant A as AOF file C->>S: *3 $3 SET $4 name $5 chalk S->>R: byte stream, no boundaries R->>M: args = [SET, name, chalk] M->>M: lock, store[name] = chalk, unlock M->>A: append the same RESP bytes, fsync A-->>M: on disk M-->>C: +OK
The AOF write happens before +OK returns - the whole durability promise in one arrow. A plain SET never touches expiry or eviction; those run on their own timers around the map.
What you can run today
Run the assembled build in one terminal (it prints the keys it loaded - zero, the first time):
go run ./07-persistence
listening on :6379 (loaded 0 keys)
Speak RESP by hand with nc, no redis-cli needed: one SET, GET, SAVE, and a GET of a missing key, in the exact bytes a real client would send:
printf '*3\r\n$3\r\nSET\r\n$4\r\nname\r\n$5\r\nchalk\r\n*2\r\n$3\r\nGET\r\n$4\r\nname\r\n*1\r\n$4\r\nSAVE\r\n*2\r\n$3\r\nGET\r\n$7\r\nmissing\r\n' \
| nc -w 1 127.0.0.1 6379 | od -c
Real output:
0000000 + O K \r \n $ 5 \r \n c h a l k \r \n
0000020 + O K \r \n $ - 1 \r \n
Four replies, no separator but RESP’s own framing: +OK for the SET, $5\r\nchalk for the GET, +OK for the SAVE, $-1 for the missing key - RESP’s nil, a first-class answer, not an error.
Two files appeared next to the binary. The AOF holds the write, byte for byte:
$ od -c appendonly.aof
0000000 * 3 \r \n $ 3 \r \n S E T \r \n $ 4 \r
0000020 \n n a m e \r \n $ 5 \r \n c h a l k
0000040 \r \n
and the snapshot holds key length, value length, then the bytes:
$ cat dump.rdb
4,5,namechalk
Now stop the server (Ctrl-C) and start it again:
go run ./07-persistence
listening on :6379 (loaded 1 keys)
printf '*2\r\n$3\r\nGET\r\n$4\r\nname\r\n' | nc -w 1 127.0.0.1 6379 | od -c
0000000 $ 5 \r \n c h a l k \r \n
The key survived a process death: RDB loaded first, then AOF replayed on top, the same order real Redis uses. Delete dump.rdb and restart: still 1 key, from the AOF alone. Delete both: 0 keys.
Go, as you go (new-to-Go markers)
net.Listen/ln.Accept()return(value, error)- errors are values, not thrown.defer conn.Close()runs on function return - the cleanup idiom.go handle(conn)starts a goroutine; the scheduler + netpoller do the rest.sync.RWMutex: manyRLockreaders or oneLockwriter.for k, v := range someMapiterates in randomized order - free random sampling.
Try it
cd teaching/code/from-scratch/redis
go run ./01-socket # then: nc localhost 6379 -> type PING -> +PONG
# redis-cli -p 6379 ping -> ERR (RESP not spoken yet!)
go run ./02-resp # redis-cli -p 6379 ping -> PONG ; echo hello -> "hello"
go run ./03-store # set name garv ; get name ; del name
go run -race ./04-eventloop # open several redis-cli at once; -race proves the mutex earns its keep
go run ./05-expiry # set s abc EX 3 ; ttl s ; wait 3s ; get s -> (nil)
go run ./06-eviction # for i in $(seq 1 20); do redis-cli set k$i v$i; done ; dbsize stays capped
go run ./07-persistence # set name garv ; save ; Ctrl-C ; re-run -> name survives
Run it yourself: all 7 source files
- Install Go from
go.dev/dl. Check it withgo version. - Make the folder:
mkdir -p teaching/code/from-scratch/redis. Save each of the 7 files below at the path shown above it, including any subfolders. - Inside that folder, run
go mod init lessononce. Go needs a module name to build. - Run the commands in the block above from the folder where you ran mkdir.
teaching/code/from-scratch/redis/01-socket/main.gopackage main
// v01 — the socket that answers.
//
// Goal: prove a client can connect and get an answer. No protocol yet.
//
// Run: go run ./01-socket
// Poke: nc localhost 6379 -> type PING, enter -> +PONG
// Wall: redis-cli -p 6379 ping -> (error) ERR, because redis-cli sends
// *1\r\n$4\r\nPING\r\n (RESP), not the raw bytes "PING". -> v02.
//
// Go notes: net.Listen opens the socket; err is a value, not an exception.
// `defer conn.Close()` runs on return — Go's cleanup idiom.
import (
"bufio"
"fmt"
"net"
"strings"
)
func main() {
ln, err := net.Listen("tcp", "127.0.0.1:6379")
if err != nil {
panic(err)
}
fmt.Println("listening on :6379 (Ctrl-C to stop)")
for {
conn, err := ln.Accept() // blocks until a client knocks
if err != nil {
continue
}
handle(conn) // v01: ONE client at a time, on purpose (see v04)
}
}
func handle(conn net.Conn) {
defer conn.Close()
r := bufio.NewReader(conn)
for {
line, err := r.ReadString('\n') // we PRETEND one line == one command
if err != nil {
return
}
if strings.ToUpper(strings.TrimSpace(line)) == "PING" {
conn.Write([]byte("+PONG\r\n"))
} else {
conn.Write([]byte("-ERR unknown command\r\n"))
}
}
}
teaching/code/from-scratch/redis/02-resp/main.gopackage main
// v02 — RESP, the wire protocol. Now redis-cli itself can talk to us.
//
// A client command is ALWAYS an array of bulk strings:
// *2\r\n $4\r\nECHO\r\n $5\r\nhello\r\n == ["ECHO", "hello"]
// The first byte names the type: * array $ bulk string.
// Replies: + simple - error : integer $ bulk $-1 nil.
//
// Run: go run ./02-resp
// Now: redis-cli -p 6379 ping -> PONG
// redis-cli -p 6379 echo hello -> "hello"
//
// The point: length prefixes solve framing. `$4` says "the next 4 bytes are
// the value" — no guessing where a command ends. That's why RESP beats a
// line protocol.
import (
"bufio"
"fmt"
"net"
"strconv"
"strings"
)
func main() {
ln, _ := net.Listen("tcp", "127.0.0.1:6379")
fmt.Println("listening on :6379")
for {
conn, err := ln.Accept()
if err != nil {
continue
}
handle(conn)
}
}
func handle(conn net.Conn) {
defer conn.Close()
r := bufio.NewReader(conn)
for {
args, err := readCommand(r) // <- framing lives here now
if err != nil {
return
}
if len(args) == 0 {
continue
}
switch strings.ToUpper(args[0]) {
case "PING":
conn.Write([]byte("+PONG\r\n"))
case "ECHO":
writeBulk(conn, args[1])
default:
conn.Write([]byte("-ERR unknown command '" + args[0] + "'\r\n"))
}
}
}
// readCommand parses one request: *N\r\n then N bulk strings.
func readCommand(r *bufio.Reader) ([]string, error) {
line, err := r.ReadString('\n')
if err != nil {
return nil, err
}
line = strings.TrimRight(line, "\r\n")
if len(line) == 0 || line[0] != '*' {
return nil, fmt.Errorf("expected array, got %q", line)
}
n, _ := strconv.Atoi(line[1:])
args := make([]string, n)
for i := 0; i < n; i++ {
hdr, err := r.ReadString('\n') // $<len>
if err != nil {
return nil, err
}
hdr = strings.TrimRight(hdr, "\r\n")
length, _ := strconv.Atoi(hdr[1:])
buf := make([]byte, length+2) // value + trailing \r\n
if _, err := readFull(r, buf); err != nil {
return nil, err
}
args[i] = string(buf[:length])
}
return args, nil
}
func readFull(r *bufio.Reader, buf []byte) (int, error) {
got := 0
for got < len(buf) {
n, err := r.Read(buf[got:])
got += n
if err != nil {
return got, err
}
}
return got, nil
}
func writeBulk(conn net.Conn, s string) {
fmt.Fprintf(conn, "$%d\r\n%s\r\n", len(s), s)
}
teaching/code/from-scratch/redis/03-store/main.gopackage main
// v03 — the map. SET / GET / DEL. Now it's a real key-value server.
//
// Run: go run ./03-store
// redis-cli -p 6379 set name garv -> OK
// redis-cli -p 6379 get name -> "garv"
// redis-cli -p 6379 del name -> (integer) 1
//
// The store IS just a Go map. That's the whole "database". Everything hard
// about Redis is what surrounds this map, not the map itself.
//
// Note: NO lock here — v03 still serves one connection at a time (handle is
// called synchronously). The moment we add `go handle(conn)` in v04, this
// map becomes a data race and needs a mutex. Hold that thought.
import (
"bufio"
"fmt"
"net"
"strconv"
"strings"
)
var store = map[string]string{} // <- the entire database
func main() {
ln, _ := net.Listen("tcp", "127.0.0.1:6379")
fmt.Println("listening on :6379")
for {
conn, err := ln.Accept()
if err != nil {
continue
}
handle(conn)
}
}
func handle(conn net.Conn) {
defer conn.Close()
r := bufio.NewReader(conn)
for {
args, err := readCommand(r)
if err != nil {
return
}
if len(args) == 0 {
continue
}
switch strings.ToUpper(args[0]) {
case "PING":
conn.Write([]byte("+PONG\r\n"))
case "ECHO":
writeBulk(conn, args[1])
case "SET":
store[args[1]] = args[2]
conn.Write([]byte("+OK\r\n"))
case "GET":
if v, ok := store[args[1]]; ok {
writeBulk(conn, v)
} else {
conn.Write([]byte("$-1\r\n")) // nil
}
case "DEL":
n := 0
for _, k := range args[1:] {
if _, ok := store[k]; ok {
delete(store, k)
n++
}
}
fmt.Fprintf(conn, ":%d\r\n", n)
default:
conn.Write([]byte("-ERR unknown command '" + args[0] + "'\r\n"))
}
}
}
func readCommand(r *bufio.Reader) ([]string, error) {
line, err := r.ReadString('\n')
if err != nil {
return nil, err
}
line = strings.TrimRight(line, "\r\n")
if len(line) == 0 || line[0] != '*' {
return nil, fmt.Errorf("bad array %q", line)
}
n, _ := strconv.Atoi(line[1:])
args := make([]string, n)
for i := 0; i < n; i++ {
hdr, err := r.ReadString('\n')
if err != nil {
return nil, err
}
length, _ := strconv.Atoi(strings.TrimRight(hdr, "\r\n")[1:])
buf := make([]byte, length+2)
if _, err := readFull(r, buf); err != nil {
return nil, err
}
args[i] = string(buf[:length])
}
return args, nil
}
func readFull(r *bufio.Reader, buf []byte) (int, error) {
got := 0
for got < len(buf) {
n, err := r.Read(buf[got:])
got += n
if err != nil {
return got, err
}
}
return got, nil
}
func writeBulk(conn net.Conn, s string) { fmt.Fprintf(conn, "$%d\r\n%s\r\n", len(s), s) }
teaching/code/from-scratch/redis/04-eventloop/main.gopackage main
// v04 — one thread, many clients.
//
// v03 served ONE client at a time: while Alice's connection is open, Bob
// waits. Add one word — `go handle(conn)` — and every client runs at once.
//
// But now two goroutines touch `store` at the same time => a data race
// (run with `go run -race ./04-eventloop` and watch it scream). We add a
// sync.RWMutex: many readers OR one writer.
//
// THE REVEAL: goroutines are NOT one-OS-thread-each. Go's runtime multiplexes
// thousands of goroutines onto a handful of OS threads using an epoll/kqueue
// "netpoller" — the exact reactor pattern real Redis writes BY HAND in C.
// - Real Redis: single thread + explicit epoll, so NO lock is ever needed.
// - Our Go: many goroutines + hidden epoll, so we PAY with a mutex.
// Same machine underneath; two philosophies about who holds the lock.
//
// Run: go run ./04-eventloop (open several redis-cli at once)
// Race: go run -race ./04-eventloop
import (
"bufio"
"fmt"
"net"
"strconv"
"strings"
"sync"
)
var (
mu sync.RWMutex
store = map[string]string{}
)
func main() {
ln, _ := net.Listen("tcp", "127.0.0.1:6379")
fmt.Println("listening on :6379 (concurrent)")
for {
conn, err := ln.Accept()
if err != nil {
continue
}
go handle(conn) // <- the one word that changes everything
}
}
func handle(conn net.Conn) {
defer conn.Close()
r := bufio.NewReader(conn)
for {
args, err := readCommand(r)
if err != nil {
return
}
if len(args) == 0 {
continue
}
switch strings.ToUpper(args[0]) {
case "PING":
conn.Write([]byte("+PONG\r\n"))
case "SET":
mu.Lock() // one writer
store[args[1]] = args[2]
mu.Unlock()
conn.Write([]byte("+OK\r\n"))
case "GET":
mu.RLock() // many readers
v, ok := store[args[1]]
mu.RUnlock()
if ok {
writeBulk(conn, v)
} else {
conn.Write([]byte("$-1\r\n"))
}
case "DEL":
mu.Lock()
n := 0
for _, k := range args[1:] {
if _, ok := store[k]; ok {
delete(store, k)
n++
}
}
mu.Unlock()
fmt.Fprintf(conn, ":%d\r\n", n)
default:
conn.Write([]byte("-ERR unknown command '" + args[0] + "'\r\n"))
}
}
}
func readCommand(r *bufio.Reader) ([]string, error) {
line, err := r.ReadString('\n')
if err != nil {
return nil, err
}
line = strings.TrimRight(line, "\r\n")
if len(line) == 0 || line[0] != '*' {
return nil, fmt.Errorf("bad array %q", line)
}
n, _ := strconv.Atoi(line[1:])
args := make([]string, n)
for i := 0; i < n; i++ {
hdr, err := r.ReadString('\n')
if err != nil {
return nil, err
}
length, _ := strconv.Atoi(strings.TrimRight(hdr, "\r\n")[1:])
buf := make([]byte, length+2)
if _, err := readFull(r, buf); err != nil {
return nil, err
}
args[i] = string(buf[:length])
}
return args, nil
}
func readFull(r *bufio.Reader, buf []byte) (int, error) {
got := 0
for got < len(buf) {
n, err := r.Read(buf[got:])
got += n
if err != nil {
return got, err
}
}
return got, nil
}
func writeBulk(conn net.Conn, s string) { fmt.Fprintf(conn, "$%d\r\n%s\r\n", len(s), s) }
teaching/code/from-scratch/redis/05-expiry/main.gopackage main
// v05 — keys that die. TTL, and two ways to reap expired keys.
//
// Run: go run ./05-expiry
// redis-cli -p 6379 set session abc EX 5
// redis-cli -p 6379 ttl session -> (integer) 5, 4, 3...
// (wait 5s) redis-cli -p 6379 get session -> (nil)
//
// Two reapers, exactly like real Redis:
// LAZY — on GET, if the key is past its expiry, delete it then. Cheap,
// but a key nobody reads never gets cleaned up (memory leak).
// ACTIVE — a background loop every 100ms samples ~20 keys that HAVE a TTL,
// deletes the dead ones. If >25% were dead, it loops again right
// away (the set is "hot" with expiries). Probabilistic, not a scan.
//
// Why sampling and not "check every key every second"? A million keys * 10/s
// = 10M checks/s of mostly-alive keys. Sampling keeps expiry ~O(dead keys),
// not O(all keys). You trade a little staleness for a huge CPU win.
import (
"bufio"
"fmt"
"net"
"strconv"
"strings"
"sync"
"time"
)
type entry struct {
val string
expireAt time.Time // zero value = never expires
}
var (
mu sync.RWMutex
store = map[string]entry{}
)
func main() {
ln, _ := net.Listen("tcp", "127.0.0.1:6379")
fmt.Println("listening on :6379 (with expiry)")
go activeExpiryLoop() // the background reaper
for {
conn, err := ln.Accept()
if err != nil {
continue
}
go handle(conn)
}
}
// getLive is the LAZY reaper: expired-on-read keys die here.
func getLive(key string) (string, bool) {
mu.RLock()
e, ok := store[key]
mu.RUnlock()
if !ok {
return "", false
}
if !e.expireAt.IsZero() && time.Now().After(e.expireAt) {
mu.Lock()
delete(store, key)
mu.Unlock()
return "", false
}
return e.val, true
}
// activeExpiryLoop is Redis's adaptive sampler.
func activeExpiryLoop() {
for {
time.Sleep(100 * time.Millisecond)
for {
dead := sweepSample(20)
if dead < 5 { // <25% of 20 -> this batch is mostly alive, rest
break
}
}
}
}
func sweepSample(n int) int {
mu.Lock()
defer mu.Unlock()
now := time.Now()
checked, dead := 0, 0
for k, e := range store { // Go randomizes map order -> a free random sample
if checked >= n {
break
}
if e.expireAt.IsZero() {
continue // only sample keys that HAVE a TTL
}
checked++
if now.After(e.expireAt) {
delete(store, k)
dead++
}
}
return dead
}
func handle(conn net.Conn) {
defer conn.Close()
r := bufio.NewReader(conn)
for {
args, err := readCommand(r)
if err != nil {
return
}
if len(args) == 0 {
continue
}
switch strings.ToUpper(args[0]) {
case "PING":
conn.Write([]byte("+PONG\r\n"))
case "SET":
e := entry{val: args[2]}
if len(args) >= 5 { // SET key val EX secs | PX ms
n, _ := strconv.Atoi(args[4])
switch strings.ToUpper(args[3]) {
case "EX":
e.expireAt = time.Now().Add(time.Duration(n) * time.Second)
case "PX":
e.expireAt = time.Now().Add(time.Duration(n) * time.Millisecond)
}
}
mu.Lock()
store[args[1]] = e
mu.Unlock()
conn.Write([]byte("+OK\r\n"))
case "GET":
if v, ok := getLive(args[1]); ok {
writeBulk(conn, v)
} else {
conn.Write([]byte("$-1\r\n"))
}
case "TTL":
mu.RLock()
e, ok := store[args[1]]
mu.RUnlock()
switch {
case !ok:
conn.Write([]byte(":-2\r\n")) // no such key
case e.expireAt.IsZero():
conn.Write([]byte(":-1\r\n")) // exists, no TTL
default:
fmt.Fprintf(conn, ":%d\r\n", int(time.Until(e.expireAt).Seconds()))
}
default:
conn.Write([]byte("-ERR unknown command '" + args[0] + "'\r\n"))
}
}
}
func readCommand(r *bufio.Reader) ([]string, error) {
line, err := r.ReadString('\n')
if err != nil {
return nil, err
}
line = strings.TrimRight(line, "\r\n")
if len(line) == 0 || line[0] != '*' {
return nil, fmt.Errorf("bad array %q", line)
}
n, _ := strconv.Atoi(line[1:])
args := make([]string, n)
for i := 0; i < n; i++ {
hdr, err := r.ReadString('\n')
if err != nil {
return nil, err
}
length, _ := strconv.Atoi(strings.TrimRight(hdr, "\r\n")[1:])
buf := make([]byte, length+2)
if _, err := readFull(r, buf); err != nil {
return nil, err
}
args[i] = string(buf[:length])
}
return args, nil
}
func readFull(r *bufio.Reader, buf []byte) (int, error) {
got := 0
for got < len(buf) {
n, err := r.Read(buf[got:])
got += n
if err != nil {
return got, err
}
}
return got, nil
}
func writeBulk(conn net.Conn, s string) { fmt.Fprintf(conn, "$%d\r\n%s\r\n", len(s), s) }
teaching/code/from-scratch/redis/06-eviction/main.gopackage main
// v06 — eviction. Memory is finite; when it fills, WHICH key dies?
//
// Run: go run ./06-eviction
// for i in $(seq 1 20); do redis-cli -p 6379 set k$i v$i; done
// redis-cli -p 6379 dbsize -> stays capped at maxKeys
//
// Real Redis caps BYTES (maxmemory). Here we cap KEY COUNT (maxKeys) to keep
// the idea visible — same algorithm, simpler yardstick.
// ponytail: key-count cap stands in for maxmemory; swap in a byte counter if
// you want true faithfulness.
//
// THE TRICK: real Redis does NOT keep a perfectly-ordered LRU list (that costs
// a pointer-shuffle on every access). It APPROXIMATES: sample K random keys,
// evict the least-recently-used of the sample. K=5 gets you within a few % of
// true LRU for ~free. Each entry carries a logical "access clock" instead of a
// real timestamp (real Redis uses a 24-bit clock in the object header).
import (
"bufio"
"fmt"
"net"
"strconv"
"strings"
"sync"
)
type entry struct {
val string
lastClock uint64 // logical access time; bigger = more recent
}
var (
mu sync.Mutex
store = map[string]entry{}
clock uint64
maxKeys = 8 // small so you can SEE eviction happen
)
const sampleK = 5
func main() {
ln, _ := net.Listen("tcp", "127.0.0.1:6379")
fmt.Printf("listening on :6379 (maxKeys=%d, allkeys-lru by sampling)\n", maxKeys)
for {
conn, err := ln.Accept()
if err != nil {
continue
}
go handle(conn)
}
}
func touch(e entry) entry { // stamp an access
clock++
e.lastClock = clock
return e
}
// evictOne: sample K keys, drop the least-recently-used. This IS approximated LRU.
func evictOne() {
var victim string
var oldest uint64 = ^uint64(0)
n := 0
for k, e := range store { // random map order = our random sample
if e.lastClock < oldest {
oldest, victim = e.lastClock, k
}
n++
if n >= sampleK {
break
}
}
if victim != "" {
delete(store, victim)
fmt.Println("evicted (LRU):", victim)
}
}
func handle(conn net.Conn) {
defer conn.Close()
r := bufio.NewReader(conn)
for {
args, err := readCommand(r)
if err != nil {
return
}
if len(args) == 0 {
continue
}
switch strings.ToUpper(args[0]) {
case "PING":
conn.Write([]byte("+PONG\r\n"))
case "SET":
mu.Lock()
if _, exists := store[args[1]]; !exists {
for len(store) >= maxKeys { // make room BEFORE inserting
evictOne()
}
}
store[args[1]] = touch(entry{val: args[2]})
mu.Unlock()
conn.Write([]byte("+OK\r\n"))
case "GET":
mu.Lock()
e, ok := store[args[1]]
if ok {
store[args[1]] = touch(e) // a read counts as recent use
}
mu.Unlock()
if ok {
writeBulk(conn, e.val)
} else {
conn.Write([]byte("$-1\r\n"))
}
case "DBSIZE":
mu.Lock()
n := len(store)
mu.Unlock()
fmt.Fprintf(conn, ":%d\r\n", n)
default:
conn.Write([]byte("-ERR unknown command '" + args[0] + "'\r\n"))
}
}
}
func readCommand(r *bufio.Reader) ([]string, error) {
line, err := r.ReadString('\n')
if err != nil {
return nil, err
}
line = strings.TrimRight(line, "\r\n")
if len(line) == 0 || line[0] != '*' {
return nil, fmt.Errorf("bad array %q", line)
}
n, _ := strconv.Atoi(line[1:])
args := make([]string, n)
for i := 0; i < n; i++ {
hdr, err := r.ReadString('\n')
if err != nil {
return nil, err
}
length, _ := strconv.Atoi(strings.TrimRight(hdr, "\r\n")[1:])
buf := make([]byte, length+2)
if _, err := readFull(r, buf); err != nil {
return nil, err
}
args[i] = string(buf[:length])
}
return args, nil
}
func readFull(r *bufio.Reader, buf []byte) (int, error) {
got := 0
for got < len(buf) {
n, err := r.Read(buf[got:])
got += n
if err != nil {
return got, err
}
}
return got, nil
}
func writeBulk(conn net.Conn, s string) { fmt.Fprintf(conn, "$%d\r\n%s\r\n", len(s), s) }
teaching/code/from-scratch/redis/07-persistence/main.gopackage main
// v07 — surviving a crash. RAM is gone the instant the power blinks.
//
// Run: go run ./07-persistence
// redis-cli -p 6379 set name garv
// redis-cli -p 6379 save -> writes dump.rdb
// Ctrl-C, then run again -> "name" is still there.
//
// TWO durability strategies, exactly like real Redis:
// AOF (Append-Only File): log EVERY write command to disk as it happens.
// Recovery = replay the log. Durable to the last fsync, but the file
// grows forever (until a rewrite compacts it). Slower writes.
// RDB (snapshot): dump the whole dataset to one file, point-in-time.
// Real Redis fork()s a child and copy-on-write lets the parent keep
// serving while the child writes the snapshot. Fast, compact, but you
// lose everything since the last snapshot.
//
// The fork+COW trick is the clever bit: the child sees a frozen copy of memory
// for free (the OS only copies a page when the parent writes it), so a snapshot
// of a 10GB dataset costs almost no extra RAM and never blocks reads.
//
// Boot order (real Redis too): load RDB first (the baseline), THEN replay the
// AOF (the writes since that snapshot). Here we do both.
import (
"bufio"
"fmt"
"net"
"os"
"strconv"
"strings"
"sync"
)
var (
mu sync.Mutex
store = map[string]string{}
aof *os.File
)
const rdbPath = "dump.rdb"
const aofPath = "appendonly.aof"
func main() {
loadRDB() // baseline snapshot
replayAOF() // writes since the snapshot
var err error
aof, err = os.OpenFile(aofPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
if err != nil {
panic(err)
}
ln, _ := net.Listen("tcp", "127.0.0.1:6379")
fmt.Printf("listening on :6379 (loaded %d keys)\n", len(store))
for {
conn, err := ln.Accept()
if err != nil {
continue
}
go handle(conn)
}
}
// appendAOF logs a write command in RESP so replay reuses the same parser.
func appendAOF(args []string) {
fmt.Fprintf(aof, "*%d\r\n", len(args))
for _, a := range args {
fmt.Fprintf(aof, "$%d\r\n%s\r\n", len(a), a)
}
aof.Sync() // fsync policy "always" — durable, slowest. everysec = looser.
}
func replayAOF() {
f, err := os.Open(aofPath)
if err != nil {
return // no AOF yet
}
defer f.Close()
r := bufio.NewReader(f)
for {
args, err := readCommand(r)
if err != nil {
break
}
apply(args) // rebuild state, but do NOT re-log it
}
}
// apply mutates the store. Caller decides whether to also log/persist.
func apply(args []string) {
switch strings.ToUpper(args[0]) {
case "SET":
store[args[1]] = args[2]
case "DEL":
for _, k := range args[1:] {
delete(store, k)
}
}
}
func saveRDB() error {
tmp := rdbPath + ".tmp"
f, err := os.Create(tmp)
if err != nil {
return err
}
w := bufio.NewWriter(f)
mu.Lock()
for k, v := range store { // length-prefixed so values can hold anything
fmt.Fprintf(w, "%d,%d,%s%s", len(k), len(v), k, v)
}
mu.Unlock()
w.Flush()
f.Close()
return os.Rename(tmp, rdbPath) // atomic swap: a crash never leaves a torn file
}
func loadRDB() {
data, err := os.ReadFile(rdbPath)
if err != nil {
return
}
s := string(data)
for len(s) > 0 {
c1 := strings.IndexByte(s, ',')
c2 := strings.IndexByte(s[c1+1:], ',') + c1 + 1
klen, _ := strconv.Atoi(s[:c1])
vlen, _ := strconv.Atoi(s[c1+1 : c2])
body := s[c2+1:]
store[body[:klen]] = body[klen : klen+vlen]
s = body[klen+vlen:]
}
}
func handle(conn net.Conn) {
defer conn.Close()
r := bufio.NewReader(conn)
for {
args, err := readCommand(r)
if err != nil {
return
}
if len(args) == 0 {
continue
}
switch strings.ToUpper(args[0]) {
case "PING":
conn.Write([]byte("+PONG\r\n"))
case "SET":
mu.Lock()
store[args[1]] = args[2]
mu.Unlock()
appendAOF(args) // durability: log before we ack
conn.Write([]byte("+OK\r\n"))
case "GET":
mu.Lock()
v, ok := store[args[1]]
mu.Unlock()
if ok {
writeBulk(conn, v)
} else {
conn.Write([]byte("$-1\r\n"))
}
case "DEL":
mu.Lock()
n := 0
for _, k := range args[1:] {
if _, ok := store[k]; ok {
delete(store, k)
n++
}
}
mu.Unlock()
appendAOF(args)
fmt.Fprintf(conn, ":%d\r\n", n)
case "SAVE":
if err := saveRDB(); err != nil {
conn.Write([]byte("-ERR save failed\r\n"))
} else {
conn.Write([]byte("+OK\r\n"))
}
default:
conn.Write([]byte("-ERR unknown command '" + args[0] + "'\r\n"))
}
}
}
func readCommand(r *bufio.Reader) ([]string, error) {
line, err := r.ReadString('\n')
if err != nil {
return nil, err
}
line = strings.TrimRight(line, "\r\n")
if len(line) == 0 || line[0] != '*' {
return nil, fmt.Errorf("bad array %q", line)
}
n, _ := strconv.Atoi(line[1:])
args := make([]string, n)
for i := 0; i < n; i++ {
hdr, err := r.ReadString('\n')
if err != nil {
return nil, err
}
length, _ := strconv.Atoi(strings.TrimRight(hdr, "\r\n")[1:])
buf := make([]byte, length+2)
if _, err := readFull(r, buf); err != nil {
return nil, err
}
args[i] = string(buf[:length])
}
return args, nil
}
func readFull(r *bufio.Reader, buf []byte) (int, error) {
got := 0
for got < len(buf) {
n, err := r.Read(buf[got:])
got += n
if err != nil {
return got, err
}
}
return got, nil
}
func writeBulk(conn net.Conn, s string) { fmt.Fprintf(conn, "$%d\r\n%s\r\n", len(s), s) }
Break it on purpose: in 04-eventloop, delete the mu.Lock()/Unlock() and run
with -race under load - watch Go detect the corruption the single-threaded design
would never have.
No redis-cli installed? Every stage from 02 onward speaks RESP, and nc is enough. A PING is the array *1\r\n$4\r\nPING\r\n:
printf '*1\r\n$4\r\nPING\r\n' | nc -w 1 127.0.0.1 6379
prints +PONG. Writing the bytes yourself once is the fastest way to stop thinking of RESP as a protocol and start seeing it as a length in front of every string.
Go deeper (primary sources)
- CodeCrafters - “Build Your Own Redis”: https://codecrafters.io/challenges/redis
- build-your-own.org - “Build Your Own Redis with C/C++”: https://build-your-own.org/redis/
- Redis source:
src/ae.c,src/expire.c,src/evict.c,src/rdb.c,src/aof.c- https://github.com/redis/redis - RESP spec and persistence docs: https://redis.io/docs/latest/develop/reference/protocol-spec/ , https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/
- Eviction / LRU approximation: https://redis.io/docs/latest/develop/reference/eviction/
Related
- Redis 01: The Socket That Answers - the socket board, in full
- Redis 02: RESP, The Wire Protocol - RESP, byte by byte
- Redis: The Map (SET / GET / DEL) - the map as the database
- Redis: One Thread, Many Clients - goroutine per client, netpoller underneath
- Keys That Die (Expiry) - lazy and active expiry
- Eviction: Memory Is Full - approximated LRU, measured
- Surviving a Crash (Persistence) - AOF and RDB, timed
- Capstone: Redis, Everything - the assembled server, what real Redis adds
- The slate next to the stove (caching) - what a memory notepad is for
- Build Your Own HTTP Server - From Scratch - the same framing wall, words not lengths
- Build Your Own Kafka - From Scratch - the append-only log, on its own
- S3 · Build a Redis Server (that real clients can talk to) - ship it as a portfolio project: the same server as one repo, with stage tests, CI and a resume line
Retrieval practice
10 questions. Answer each one before you open it.
Why can’t the reader just check whether the bytes that arrived spell PING?
A socket hands you a stream, not messages. One read can return half a command or several glued together, so both ends must agree in advance where one word ends - exactly what RESP’s length prefixes give you.
What does the $4 in $4\r\nECHO\r\n mean, and why does it matter?
“The next 4 bytes are this word.” The length comes first, so the reader copies exactly that many bytes and never guesses where it stops - what turns a stream into unambiguous commands.
Real Redis answers every client from one thread. Why is that fast, not slow?
One clerk means no two hands reach for the same page, so nothing needs a lock. I/O never blocks it either, since the OS says which lines are ready. The cost: one slow command blocks everyone, so Redis keeps commands small.
This build gives every visitor its own goroutine. How is that not “one clerk per visitor” again?
A goroutine starts at ~2 KB against a thread’s ~1 MB, and Go’s runtime multiplexes many onto a few real threads itself - the same “ask what’s ready” trick Redis writes by hand. The price is a lock on the shared map, which one true clerk never needs.
Why does the notepad sample a few keys for expiry and eviction instead of checking every page?
Checking everything costs work proportional to the whole desk, and most notes are fine. Sampling a handful tracks only the stale or evictable ones, at the price of a little slack. Close-to-right for nearly nothing beats exactly-right for a lot.
What is the fork-and-copy-on-write trick behind the snapshot, and what problem does it solve?
A full photocopy shouldn’t stop the desk being used. Redis fork()s a copy of itself; copy-on-write means the OS only duplicates a memory page once the original writes to it afterward - so the copy writes a frozen desk to disk while the original keeps serving, at almost no extra memory.
The log and the snapshot: one line each, and why keep both?
The log (AOF) is every write, replayed in order: durable, but slower to restart from. The snapshot (RDB) is the whole desk reloaded whole: fast, but loses whatever came after it was taken. Keep both - the snapshot for speed, the log for what it missed.
Why do the walls arrive in this order - could notes that expire have been the first problem?
Each wall exists only because the one before it got solved. Bytes must arrive before they’re framed into words; words must be framed before a page can answer them; one page must serve many visitors before it’s worth filling with real data. Only a desk full of live notes has an expiry, eviction or crash problem.
For a SET on the assembled server, which happens first - the write to disk or the +OK - and why is the log written in RESP?
Disk first: the log is appended and confirmed on disk, then +OK is sent - the reply never promises more than the disk holds. The log uses RESP, the same bytes the client sent, so replay at startup reuses the very reader that takes commands off the socket.
What does the wire say when a key is missing, and what does the snapshot file hold after SET name chalk then SAVE?
A missing key comes back as $-1\r\n, RESP’s own nil. The snapshot holds 4,5,namechalk - key length, value length, then the raw bytes of both, so a value can hold commas or line breaks without breaking the format.