How this drag-and-drop library works, traced through one tree

Making a <div> draggable is solved everywhere. That is not the hard part.

The hard part is what happens after you let go.

Here is a document tree. Keep it in your head — the whole post follows this one tree, and every number below is traced from the real implementation, not invented for the example.

Handbook          depth 0
  Onboarding      depth 1
  Engineering     depth 1
    Style guide   depth 2
    On-call       depth 2
Roadmap           depth 0
  Q3              depth 1
Meeting notes     depth 0

That indentation isn't decoration — it is the data. The tree is an array of nodes, and every node has the same shape: an id, and an optional children array of more nodes. (Your nodes carry their own fields too — a title, an icon — but the library only ever reads id and children.)

type TreeItem = { id: string; children?: TreeItem[] }

So the outline above is really this:

const tree = [
  { id: 'Handbook', children: [
    { id: 'Onboarding' },
    { id: 'Engineering', children: [
      { id: 'Style guide' },
      { id: 'On-call' },
    ] },
  ] },
  { id: 'Roadmap', children: [{ id: 'Q3' }] },
  { id: 'Meeting notes' },
]

Every step of indentation on the left is one more children array deep on the right. And that is the only address a node ever has: walk down children by index. On-call is tree[0].children[1].children[1] — the root's second child (Engineering), then its second child (On-call). Read any such path left to right: item.children[0].children[2].children[1] means "first child, then its third child, then that one's second child." Each [index] picks one sibling inside one parent — so how many children hops you took is the node's depth, and the last index you picked is its index among its siblings. That depth-vs-index distinction is the one the whole library turns on; keep it in view.

Pick up Onboarding. Drag it down. Release it in the gap just under On-call.

Where did it go?

There are four honest answers, and they occupy the same pixels:

    On-call (depth 2)
──────────────  ← you released here
    Roadmap (depth 0)

First child of On-call — depth 3.

Next sibling of On-call — depth 2.

Next sibling of Engineering — depth 1.

A new root document — depth 0.

Same gap. Same pixel row. Four different trees afterwards. The only thing separating them is how far right your cursor was.

That is the question this library answers. Not "what gesture did you make" but where does the thing land.

1 · The state lives outside React

Start with where the drag state is kept, because everything else follows from it.

It is a plain JavaScript object, in a plain closure. Not useState, not context. Its whole shape is four fields:

type DragStoreState = {
  readonly origin: DragOrigin | null          // who you picked up — stable for the whole drag
  readonly overId: DndId | null               // an id, not an object
  readonly translate: Translate               // the only part that changes as you move
  readonly measuredRects: ReadonlyMap<DndId, Rect>
}

Created once per <DndProvider>, never at module scope. That is not fussiness: a module-level store is shared across requests on a server, and two providers on one page would fight over a single drag. Per-provider is SSR-safe and multi-provider-safe by construction.

React reads this object through useSyncExternalStore — the hook that exists precisely for state React does not own.

Now follow one pointermove from the browser to the pixel. This is the spine of the whole library:

pointermove                          the browser, on the document
↓
sensor turns it into a translate     client code, reads no DOM
↓
store.move(translate)                updates the store's `translate` — a plain object, outside React
↓
collision against CACHED rects       arithmetic — no getBoundingClientRect
↓
a brand-new immutable state object   minted on every move
↓
subscribers are notified             every mounted row gave the store a callback — each is called
↓
each subscriber's selector runs, its result is compared
↓
only the slices that CHANGED reach React

Two of those steps do the real work.

Rects are measured once. When the drag begins, every droppable's rectangle is read in one batched pass and cached in measuredRects. After that, every move is arithmetic against the cache — the active rect is origin.rect + translate, and collision compares numbers. Nothing calls getBoundingClientRect in the move path. Reading layout mid-move — read, write, read, write — is the classic drag-and-drop performance trap; this library reads once and never again, until a scroll or resize marks the cache dirty and the next move re-measures lazily.

A new state object is minted every move. The store never mutates state; it replaces it wholesale:

state = {
  origin,
  overId: detectOver(origin, translate, measuredRects),
  translate,
  measuredRects,
}

That immutability looks optional but it's actually essential. When I talk below about re-renders, you will find out why.

So the answer to "where does the work run" is: all of it on the client, none of it in React's render loop. React is a reader of a store it does not drive.

2 · Turning a gesture into a position

A flat sortable list has an easy mapping: dropping "on item 3" means item 3 is an element, it has a rectangle, the pointer is inside it. Geometry answers the whole question.

A tree breaks that. As the four answers above showed, several outcomes share the same pixels, told apart by horizontal position — which corresponds to no element's box. So a tree drop is not an element you hit — it is a position you compute. Picture the movement first. You pick up Onboarding and release it right below Engineering's children — under On-call, the last row of Engineering's subtree — but held at Engineering's own indent, not pushed deeper:

Handbook
  Onboarding            ← picked up
  Engineering
    Style guide
    On-call
  ────────────          ← lands here: below Engineering's subtree, at Engineering's indent
Roadmap
Meeting notes

It comes to rest below Engineering's whole subtree, but at the same level as Engineering — so Onboarding becomes Engineering's sibling, not its child. That movement is exactly this position:

{
  parentId: 'handbook',                        // depth 1 sits under a depth-0 parent
  index: 2,                                    // Engineering's next sibling — still a Handbook child
  depth: 1,
  mode: 'between',
  afterId: 'engineering', beforeId: null,      // neighbours, by id
  indicator: { rowId: 'on-call', edge: 'below', depth: 1 },   // draw below Engineering's subtree
}

That tuple is the thing this library returns and general-purpose libraries do not. It is built in three moves, all pure functions over plain data — no React, no DOM, testable under plain Node in milliseconds.

Move 1 — flatten, and cycles disappear here

You can't do geometry on a nested structure. The rows on screen are a flat list, so the math runs on a flat list. flattenTree turns the nested array into one row per visible line, each carrying its depth and its sibling index:

screen row                                    sibling index
 1 │ Handbook          depth 0   parent null          index 0
 2 │   Onboarding      depth 1   parent handbook      index 0
 3 │   Engineering     depth 1   parent handbook      index 1
 4 │     Style guide   depth 2   parent engineering   index 0
 5 │     On-call       depth 2   parent engineering   index 1
 6 │ Roadmap           depth 0   parent null          index 1
 7 │   Q3              depth 1   parent roadmap       index 0
 8 │ Meeting notes     depth 0   parent null          index 2

The number on the left is the screen row; index on the right is the sibling index — and they are not the same. Meeting notes is index 2 (the root's third child) but screen row 8. Mixing the two up is a silent off-by-one; a TreeRow carries the sibling index, never a row number.

Now the elegant part. Drag Engineering onto its own child Style guide. That must be impossible — a node cannot become its own grandparent.

The bad implementation is: compute the drop, then check for a cycle, then reject it. Drag all the way there, get refused.

This library does something better. flattenTree takes the id you are dragging and leaves that node and its whole subtree out of the rows:

Handbook          ← dragging Engineering
  Onboarding
                  ← Engineering, Style guide, On-call: not rows at all
Roadmap
  ...

Style guide is no longer a row. It can't be aimed at. "Inside your own subtree" is never a position the math can produce.

A cycle you cannot express beats a cycle you catch. (There is still a defensive guard in applyTreeDrop for a hand-built projection — but the library's own path has never reached it, because the illegal state is unreachable.)

Move 2 — aim at a row, into or between

Every visible row has a cached rectangle. A row that will accept children is cut into three bands:

┌────────────────────────────┐
│  top 30%                   │  → the gap ABOVE this row     (between)
├────────────────────────────┤
│  middle 40%   Engineering  │  → INSIDE this row            (into)
├────────────────────────────┤
│  bottom 30%                │  → the gap BELOW this row     (between)
└────────────────────────────┘

That 30 / 40 / 30 split is a tunable option — 0.3 of the row's height to each outer band by default. A row that refuses children has no middle — two bands, split down the half. That one choice is mode: into a node, or between two of them.

Move 3 — choose the depth, and let people un-nest

You are in a gap. Which of the four answers did you mean? Your horizontal offset decides, in indent-sized steps (24 px by default):

Style guide (depth 2)
On-call (depth 2)
────┬────────┬────────┬────────┬─────
    │        │        │        └─ depth 3 → first child of On-call
    │        │        └────────── depth 2 → sibling of On-call
    │        └─────────────────── depth 1 → sibling of Engineering
    └──────────────────────────── depth 0 → sibling of Handbook
Roadmap (depth 0)

Not every depth is legal in every gap, though — the four answers above are the most any gap ever offers, and some gaps allow fewer. To see which, first name the two rows the gap sits between: the one just above it is rowAbove, the one just below it is rowBelow. That is the whole definition — the names are what they mean.

On-call     depth 2    ← rowAbove   (the row just above the gap)
──────────             ← the gap you're hovering in
Roadmap     depth 0    ← rowBelow   (the row just below the gap)

The legal depths for a gap form a range, [minDepth … maxDepth], and both ends are read straight off those two rows:

maxDepth = rowAbove.depth + 1   →   On-call is depth 2, so 2 + 1 = 3   →  deepest:    become On-call's first child
minDepth = rowBelow.depth       →   Roadmap is depth 0, so         0   →  shallowest: sit beside Roadmap

• maxDepth (deepest) = rowAbove.depth + 1. The deepest this gap can reach is to make the dragged row the direct child of the row above — and a direct child sits exactly one level in, at rowAbove.depth + 1. It can't go deeper: rowAbove.depth + 2 would be a grandchild, and a grandchild has to hang off one of rowAbove's own children — but sitting in the gap right below rowAbove, there is no child row here to hang off yet. You can only nest one level into a row you can actually see, so a row's children always live at its depth + 1 — and that is the deepest this position offers. (To reach a grandchild you'd drop in a gap below one of those children, not below rowAbove.)

• minDepth (shallowest) = rowBelow.depth. To land right here — immediately before rowBelow — you have to be at least as deep as rowBelow. Go shallower and you no longer belong beside it: you belong to an outer group, and you'd land after its whole subtree, not before it.

So in this gap you may drop anywhere from depth 0 to depth 3 — exactly the four answers from the top of the post. And minDepth is 0 here only because Roadmap happens to be a root; in a gap where the row below is nested, minDepth is that row's depth, not 0 — which is where things get interesting.

That maxDepth is the uncontroversial half — every tree implementation clamps it the same way. It also respects the "will you accept children?" rule: if rowAbove refuses children, there is nothing to nest into, so maxDepth collapses to rowAbove's own depth (no + 1).

minDepth is the half where this library differs from most implementations, on purpose. In the On-call gap above, rowBelow (Roadmap) is a root, so minDepth was 0 — you could drop anywhere, out to a brand-new root. But in a gap where the row below is nested, the same rule (minDepth = rowBelow.depth) stops you from un-nesting. The next example shows the problem, and why this library does not use it.

Say you're dragging Onboarding, and you want to lift it out of Handbook to become its own top-level document — depth 0. You hold it near the top and pull left, into the gap between Handbook and Engineering:

Handbook        depth 0     ← rowAbove  (the row above the gap)
─────────────               ← the gap you're pulling into
  Engineering   depth 1     ← rowBelow  (the row below the gap)
    Style guide depth 2

Trace that bad minDepth for this exact gap:

minDepth 1, maxDepth 1. The only legal depth is 1 — a child of Handbook. Under this rule you can pull left as hard as you like; the minDepth is stuck at 1, depth 0 is never offered, and the 0 you asked for is clamped straight back to 1. Onboarding cannot leave Handbook from this gap.

With that bad minDepth, un-nesting works in exactly one place: the very last gap of the group — below On-call, above Roadmap — where rowBelow is finally a shallow row (Roadmap at depth 0) and drags the minDepth to 0. So you'd have to drag Onboarding all the way to the bottom of Handbook first, and only then left. Nobody does that. It feels like the tree is ignoring your hand — which is exactly why this library doesn't stop at that minDepth.

What this library does instead: a deliberate pull to the left drops the minDepth all the way to the root.

const minDepth = isPullingLeft ? 0 : rowBelow.depth       // ← pulling left ⇒ minDepth drops to 0
const maxDepth = canNest(rowAbove, active) ? rowAbove.depth + 1 : rowAbove.depth

(isPullingLeft just means your cursor has moved left of where the row started — the same horizontal offset that already picks the depth.)

Back in that Handbook↔Engineering gap, pulling left now gives [0 … 1] instead of [1 … 1]. One indent-step left takes the requested depth to 0, and Onboarding does become a top-level document.

Where does it land? Not floating between Handbook's children — a row can't sit at root level in the middle of Handbook's subtree. It lands after Handbook's whole subtree, as the next root item. Pulling left slides the row down and out. Coming out of a group is a downward move.

And the asymmetry — minDepth 0 only when pulling left — is the whole design, not a hack. If the minDepth were always 0, you could never drop anything into a group: a row dragged in from elsewhere already asks for depth 0, so an always-0 minDepth would grant it and drop the row past the group instead of inside it. The leftward pull is the one signal that tells "put this inside" apart from "take this out."

The keyboard reuses all of it — ArrowLeft/ArrowRight produce the same horizontal translate a pointer does, so the same clamp runs. Nothing downstream can tell which sensor drove the drag.

Applying the drop shares structure

applyTreeDrop returns a new tree, but nearly every node in it is the same object you passed in. Only two ancestor spines are rebuilt — root-to-where-it-left, root-to-where-it-landed. Everything else keeps its identity. A drop allocates about 2 × depth nodes whether the tree has three nodes or ten thousand.

The bad version here is a deep clone — structuredClone, a JSON round-trip, any recursive copy. It is correct in every value and new in every reference, so a drop rebuilds the entire tree and every memoised row re-renders. It looks fine in review and stutters when the tree grows. The sharing is pinned by tests that assert untouched nodes come back === identical.

Where the general libraries stop

To be fair and precise: the ecosystem does not ignore trees. Atlassian's Pragmatic drag-and-drop ships a real tree hitbox with five instructions — reorder-above, reorder-below, make-child, reparent, instruction-blocked. It knows about nesting bands and indentation.

But it stops at the gesture relative to one row. The function you call for it, attachInstruction, runs on a single row's drop data: it looks at where the pointer sits inside that one row and tags it with one of those instructions (make-child, reorder-above, …). It can't hand back the parent id, the index, or a depth clamped against the neighbours, because it never sees the neighbours — the real position depends on three rows (rowAbove, rowBelow, and the one in your hand), and a hitbox scoped to a single element sees only that element. That boundary is where every general-purpose React DnD library stops: it gives you the gesture, and turning the gesture into a position is left to you, every time. This library is that step.

3 · Fewer re-renders than dnd-kit, and the map that makes it possible

Now the performance claim, measured.

React <Profiler> around each row, in Chrome, on a page that renders the same 24-item list twice — once with each library, identical counters. The drag itself is scripted, not done by hand: the pointer events are created in code — a pointerdown, 40 pointermoves along a fixed path, a pointerup — and the identical sequence is dispatched to each list in turn. A hand cannot repeat a drag pixel-for-pixel; a script can, which is what makes the two sides comparable and the runs repeatable. The path drags item 1 down onto item 4: 154 px, 3 boundary crossings, 40 pointer moves. Three runs, identical each time.

Four is the row in your hand plus the three it displaces. dnd-kit re-renders every row in the list on every change.

Then you let go — and the full list re-renders in both libraries, because the drop is your own state update: it fires onSortEnd, your setState replaces the array, and a list whose data just changed re-renders. That is why every row's counter ends at 1 or more even on the fc-react-dnd side. The drop was measured too, and the two sides are not equal even here:

Counting rows that re-rendered is the measure that matters, because it is the architecture made visible: when the drag state changes, how many rows have to find out about it? Here, four. That is the whole claim, and the rest of this section is why it is four and not twenty-four.

Why context re-renders everything

This is not a knock on dnd-kit — it is a property of React.

Put drag state in context, and every consumer re-renders when the value changes. There is no "subscribe to just the slice I care about." memo does not help; context propagation goes straight through it. So when you cross a boundary and the "who is over what" value changes, every row that reads the context re-renders — and each then works out, inside its own render, whether it moved. Most find out they did not, after they already rendered.

useSyncExternalStore is the escape — and since this post says "subscribe" and "subscriber" constantly, follow one row through the act, because it is concrete enough to show in full.

Row item-7 renders and calls useSortable({ id: 'item-7', … }). Inside, that hook calls useSyncExternalStore(store.subscribe, getSnapshot) — and here is the entire store.subscribe, not an excerpt:

const listeners = new Set<() => void>()
// ...
subscribe: (listener) => {
  listeners.add(listener)
  return () => listeners.delete(listener)
}

That is the entire act of subscribing. The store keeps a Set of callback functions; item-7's render adds one — a callback React itself hands in, whose meaning is "something may have changed, re-check your snapshot." The returned function removes it again, and React calls that when the row unmounts. Everywhere this post says "a subscribed row," it means exactly this: a row whose callback is currently in that Set. Twenty-four rows mounted = twenty-four callbacks in the set.

Notifying is just as plain. After minting the new state, the store loops the set:

for (const listener of [...listeners]) listener()

Each call sends React back to that row's selector — the second half of a subscription: a function from the whole store state to the small piece this row cares about, its slice. item-7's selector ignores almost everything and answers one question — is a drag active, and how far am I displaced?

;(state) => {
  if (state.origin === null) return NO_DRAG_SLICE   // no drag → one shared idle object
  // the dragged row follows the pointer; every other row reads its own
  // shift out of the projection
  return { translate: /* my shift */, isDragActive: true }
}

React runs the selector, compares the fresh slice with the previous one, and — this is the guarantee — if the slice is unchanged, nothing happens. No lane, no scheduling, no render. The comparison happens before React is involved. The whole mechanism is this:

const next = select(state)
if (cached && areEqual(cached.slice, next)) {
  cache.current = { state, slice: cached.slice, select }
  return cached.slice        // ← the SAME reference, on purpose
}

Returning the same reference is what makes React schedule nothing. (It is also correctness, not just speed: a selector that built a fresh object every call would look changed every time, React would call it again, and you'd have an infinite loop — exactly what React's DEV "getSnapshot should be cached" warning detects.)

The cost you can't avoid, and the map that keeps it cheap

First, name the thing being computed. On every move, something has to answer: if you dropped right now, where would the dragged item land, and how far does every other row shift to make room? Section 2 built exactly that answer for the tree — the position tuple with parentId, index, depth, and the indicator. The sortable list has its own version of the same answer: which rows translate up or down, and by how much. That per-move answer is called the projection — it projects what the drop would produce while you are still dragging, and it is what the indicator line and the sliding rows are drawn from. (The types are literally named for it: TreeDropProjection, ListProjection.)

The goal: compute that projection once per move, no matter how many rows are listening.

The previous subsection got the renders down to the four rows that move. What it cannot reduce is the selector calls: to decide that a row's slice did not change, React has to run that row's selector and compare — the comparison that prevents the render requires the call. So every subscriber's selector runs on every store notification. Twenty-four subscribed rows × 40 moves in the measured drag is 960 selector calls, almost all of which conclude "nothing changed, render nothing."

960 calls is fine while each one is a cheap read. But the projection is not cheap to compute: it reads every row's cached rect, sorts them along the axis, and builds the per-row shifts. That is O(N) work. If all 24 selectors computed it themselves, every move would repeat that work 24 times. So the projection must be computed once, stored where every selector can reach it, and the other 23 calls must read it instead of recompute it.

The storage for that is one module-level cache, the same shape in both projection files:

const projectionCache = new WeakMap<DragStoreState, WeakMap<object, CacheEntry>>()

Two WeakMaps, one inside the other. Build it up from what it has to do.

How a selector knows "this move already has a projection": the state object's identity. Section 1 set this up — the store never mutates its state; every move mints a brand-new DragStoreState object. So the object's identity works as a version number for the drag. Two selector calls that receive the same state object are on the same move, and the projection cannot differ between them. A call that receives a new object is on a new move, and every older projection is stale. That turns "once per move" into something checkable: cache the projection under the state object itself. The first reader of a given state object computes and stores; every later reader of that same object finds the entry and returns it; the next move brings a new object, whose lookup misses, so it computes exactly once. (This is also what keeps two <DndProvider>s on one page from colliding in the shared cache: each provider has its own store, so their state objects are never the same object, and their entries never overwrite each other.)

What a WeakMap is, and why a plain Map would leak. The cache lives at module level for the life of the app, and its keys are state objects the store replaces at a rate of one per move — a few seconds of dragging at pointer frequency makes hundreds. In a plain Map, an entry keeps its key alive: long after the store has replaced a state object and nothing else in the program references it, the Map still would, holding the dead object and its projection until someone writes eviction code and picks a moment to run it. Nothing here ever would — the cache would only grow.

A WeakMap is a key→value map with one different rule: it does not keep its keys alive. When the only remaining reference to a key object is the WeakMap itself, the garbage collector is allowed to collect that object — and the entry, key and stored value together, disappears from the map with it. Applied here: the moment the store replaces state, the old state object's last reference is the cache, the collector reclaims it, and its projection goes too. Entries expire exactly when their move stops existing, with no eviction code anywhere. That is the entire reason this is a WeakMap and not a Map. The restriction that comes with it — a WeakMap cannot be iterated or counted, only asked "what is stored under this exact object?" — costs nothing here, because that lookup is the only operation this cache performs.

Why two of them, nested. One WeakMap keyed by state would be enough if each move had exactly one projection. It does not, because one provider can hold several lists: two <SortableList>s under the same <DndProvider> share one store, so on any given move they read the same state object — while each needs its own projection, computed from its own rows. Keyed by state alone, whichever list computed first would win, and the second list's lookup would return the first list's projection.

So under each state object the cache holds another map, one entry per list, keyed by the thing that identifies a list: the itemIds array the consumer passed in (the tree projection keys by its items array the same way) — each list has its own array, and its reference is stable across a move. The two keys split the two questions:

outer key: the state object       → which move is this?
inner key: the itemIds array      → which list is asking?

The inner map is a WeakMap for the same reason as the outer one: unmount a list and its array becomes unreachable, so its entries are collected instead of accumulating. And the inner key covers one more case — when the rows themselves change mid-drag (a tree auto-expanding under the cursor mounts new rows), the consumer passes a new array, so the changed list gets a fresh entry instead of a stale projection even while the state object is momentarily the same.

The read is a two-level lookup with one real computation on a miss:

let byList = projectionCache.get(state)          // this move's bucket
if (!byList) { byList = new WeakMap(); projectionCache.set(state, byList) }

const cached = byList.get(args.itemIds)          // this list's entry
if (cached && cached.direction === args.direction) return cached.projection  // O(1) hit

const projection = computeProjection(state, args)   // the one real computation
byList.set(args.itemIds, { direction: args.direction, projection })
return projection

Now — and this is the part I got wrong at first, so it is worth stating carefully — the payoff is different on the two paths, and they should not be confused.

On the sortable list, each of the 24 rows has its own useSortable, and each calls projectList(state, …). On one move, the first row to run computes the projection; the other 23 get an O(1) WeakMap hit off the same state. One computation, twenty-three cheap reads.

On the tree, useTreeDrop subscribes once, in the parent — tree rows are measure-only and never subscribe to a projection slice. So there is only one projection reader. But React reads a snapshot several times per move — the notify-time compare, the render read afterward (with a fresh selector closure each render), plus React's own tearing and DEV re-reads — and without the map, each would re-walk flattenTree + projectTreeDrop. The map collapses them all to one compute per move.

Different mechanism, same rule:

The projection is computed once per store-state version — no matter how many rows read it, or how many times React asks.

Get that wrong — put the computation inside the selector — and the render counts still look perfect while the move path quietly goes O(N²) and the frame is gone. That is why this is the architecture's binding constraint, not a tuning detail.

4 · The limits

Honesty about the edges.

v0.1 ships one collision strategy (closestCenter), no cross-list moves, no cross-tree moves, and no drop animation. The tree is indicator-only — rows do not reflow live during a tree drag; one indicator line moves. Deliberate scope cuts, not oversights.

It targets React 19+. That is the only version tested, so it is the only version claimed.

Very large trees have a ceiling. The selector floor scales with row count — every subscriber's selector runs on every notification whether or not it renders — so past the low tens of thousands of rows the honest answer is virtualization, which is genuinely harder because rects exist only for mounted rows.

And the plain trade: if your list is flat and small, a library that re-renders every row on every move feels identical to this one. This architecture makes a measurable difference only when the list is long, or the structure is a tree, or both.

The store holds the truth, outside React.

The pure tree math turns a gesture into a position.

The map computes that position once, so React barely notices the drag at all.

Let's say you have your myFn function and inside it you have some logic (an algorithm) that takes some inputs (n elements).

One concept you need to understand is that a machine can roughly do ~10⁹ operations per second.

An operation can be a comparison, an addition, an arr[i], an i++. So if you want to check for yourself, paste this code in your browser console:

const myFn = (n) => {
  let i = 0
  while (i < n) {
    i++
  }
}

const myCalls = 1_000_000_000 // billion, with a "B" or 10⁹

const s = performance.now();
myFn(myCalls);
console.log(`run ${(performance.now() - s).toFixed(0)} ms`);
run 956 ms // roughly 1 second

As you see, it's a run 956 ms. Careful though — this number is not stable: run it a few times, or on a different machine, and you'll get anywhere from ~250ms to ~1.1s for the exact same code. It depends on your machine's state (other tabs, memory pressure) and how aggressively the engine optimises the loop.
Also note, this is just an example to give you a feel of what an "operation", which in this case is i++.

And now, the fun part. The above loop is O(n) — it does one operation per element, so n operations in total. And since the machine does ~10⁹ operations per second, the most it can get through in one second is about 10⁹ elements.

So if you do myFn(5 * myCalls) — that's 5× the operations — it takes roughly 5× as long. That's exactly what linear (O(n)) means. Don't count on an exact number though: the wall-clock swings a lot with your machine and environment (on native Node it's ~2s; a browser-based playground running Node in WebAssembly can be 5s+). The shape is the point — 5× the work, ~5× the time.

And here is why algorithm complexity matters. It basically answers you this question:

For a given complexity, how many items can this algorithm process in one second?

or if you want it more academic sounding

Given a rough computation budget, how large can n get for an algorithm of this complexity?

Bigger complexity means fewer items. Which also means, depending on how big your n is, it can take from a few ms to hours to process.

Here is a table which tells you how many n elements can your algorithm process in 1 second with ~10⁹ simple operations as our one-second budget.

Note that these numbers are an approximation, not fixed numbers, but it's a starting point that can help you better visualize what these complexities are all about.

Let me know if it clicked for you as it clicked for me.

Here is the fast and costly implementation — the one almost everyone ships first:

the model read the whole document
the model re-typed the whole document
we saved the whole document

Re-typed, literally: the edit tool's output is the complete new version of the page.

The document I used in the measurements below — is ~7.4 KB of markdown, which is roughly 1,550 tokens. So changing one sentence means the model generates ~1,560 tokens: the one sentence you asked for, plus every paragraph you didn't.

One changed sentence. ~1,560 generated tokens. And if you add new content, those output tokens start growing fast.

To see why that's the worst possible shape, look at how LLM pricing is split. Every call has two lanes:

input  — the tokens you send: the conversation so far, the document
output — the tokens the model generates

The lanes are not priced alike. On a typical frontier model today, input runs on the order of $1–3 per million tokens while output runs $10–15 per million — roughly 8× more expensive. And the real gap is wider than the list price, because the two lanes cache differently: providers discount repeated input heavily — a prompt prefix the model has recently seen costs around a tenth of the normal input price — so re-sending the same document every turn is surprisingly cheap. Output gets no such discount. There is no cache for generation; every output token is full price, every time.

So: input is the cheap, cached lane. Output is the expensive, uncached one. And the implementation above answers a one-sentence request by pushing the entire document through the expensive lane.

Where the idea came from. Not from the LLM world at all — from the internals of Slate, the framework many block-based rich-text editors are built on. When you type into a Slate document, the page does not re-render. Slate keeps WeakMaps tracking which paths of the document actually changed — the dirty paths — and only the components for those blocks re-render. The clean blocks aren't re-rendered cheaply. They aren't re-rendered at all.

Staring at that, the question asked itself: my UI refuses to re-render blocks that didn't change — so why is my agent re-emitting them? Output tokens are the agent's render budget, and it was re-rendering the entire page on every keystroke.

That analogy is what led to the ~10× output cut below. (One honest note: the mechanism I ended up shipping looks less like React and more like git — content fingerprints compared at write time, rather than live dirty-tracking — but the question that started everything came straight from the renderer.)

This post is the method, with A/B numbers from a production agent: give every block of the document a stable id, let the agent address blocks instead of re-emitting documents, and verify every edit against a content fingerprint taken when the agent read the page.

Across a real five-edit session, output dropped from 8,128 tokens to 775 — about 10× — and edits came back ~3× faster. A second, independent technique (history compaction) cut multi-turn input by ~60% on top.

None of the individual ingredients here is my invention — I'll point at the neighbours later. What I can give you is the full composition for structured documents, traced end to end, with measurements.

First, how a writing agent actually works

If you already build agents, skip ahead. But the two facts in this section are load-bearing for everything after.

An agent is a model in a loop. You send a prompt; the model either answers or asks to call a tool — a function you exposed, like read_document or search. Your server runs the tool, appends the result to the conversation, and calls the model again. Repeat until the model answers without asking for a tool. This loop (reason → act → observe) is usually called ReAct.

user prompt
↓
model → "call edit tool with these args"
↓
server runs the tool, appends the result
↓
model → "done, here's my summary"

Fact one: the model has no hands. It never touches your database. It only describes operations; your server performs them. Like SQL — the client sends UPDATE … WHERE id = …, the database executes it.

Fact two: the model has no memory. Every turn, you re-send the entire conversation so far — the transcript. A follow-up tomorrow is a new HTTP request; your server loads the stored history (for example a JSON on a per-session database row), replays it to the model, and appends the new message. Anything you want the model to "remember" is something you are paying to re-send, every single turn.

Those two facts define the two costs this post attacks:

• Output — everything the model generates, including tool arguments. Expensive, never cached.

• Input — the replayed transcript. Cheaper, cache-discounted, but re-sent every turn, so it compounds.

The prerequisite: a stable id on every block

Here is the entire architectural bet, and it's one sentence:

A block of text (data) must have an identity that survives its content changing.

Most block-based editors already assign every block a short random id at creation time — for deep links, for drag-and-drop, for anchoring comments. If yours does, the prerequisite is free. That id is the handle the agent will use to say which block it means.

Why must it be an id? Why not "the third paragraph", or a hash of the block's content? Because those are the three possible addressing schemes, and two of them die on contact with a live document:

• Position ("paragraph 3", "line 42") dies when anyone inserts or deletes above the target. Every address below the change shifts.

• Content (a hash of the block's text) dies when anyone edits the block — the address was the content. Worse, it can't distinguish "this block moved" from "this block was deleted and a similar one created".

• Identity (an id stored on the block) survives both. The block can move, the text can change; the id rides along.

The system needs to express one specific sentence: "the same block, changed." Only identity-addressing can say it — position and content addressing each collapse "same block" into the thing that changed. Hold that sentence; the fingerprints in the next section exist to finish it.

(If some blocks lack ids — legacy content — you can fall back to content+position handles. It works, degraded: addresses die on any edit, so every conflict looks like "block not found" instead of "block changed", and the follow-up optimization below gives up more often. Our system assigns real ids to everything the agent writes, so documents heal toward fully-addressed as they're edited.)

Markdown is the language of the agent

The document lives in the database as a JSON tree. The model never sees that.

Everything the model reads, and everything it writes, is markdown. On read, the server renders each block to markdown; on write, the model sends markdown back and the server parses just that fragment into blocks.

Two reasons, one obvious and one that took me longer:

The obvious one: models are dramatically better at markdown than at your internal JSON. They've read a lifetime of it. Emitting a paragraph of markdown is cheap and reliable; emitting your nested block schema, with every property right, is neither — and it spends tokens on syntax instead of content.

The less obvious one is about the fingerprints. The system needs to answer "has this block changed since the agent read it?" — which basically means:

"Would the model see different text at response time than it saw at read time?"

So we don't fingerprint the stored JSON. We fingerprint the rendered markdown — a SHA-1 of exactly the text the model was shown, produced by the same renderer at read time and at check time. That buys precision in both directions:

• Internal properties can churn (metadata, layout props, id reassignments) without triggering a false conflict — if the rendered text is identical, the block is not stale for the model's purposes.

• Any change the model would see, however small, flips the hash completely.

The fingerprint is a statement about the model's view, not about storage bytes. Hash the view, because the view is what the model's decisions were based on.

One more property matters: the fingerprints are cheap. ~50 bytes per block to store, microseconds to compute. A 200-block document snapshots to ~10 KB regardless of how much text the blocks hold.

The read: addresses to the model, fingerprints to the server

Take a five-block document — a webhooks guide for example:

## Webhooks
Deliveries are sent whenever a document is published.
Every request carries a signature header.
Failed deliveries are retried three times.
You can replay any delivery from the dashboard.

When the agent reads it, one walk over the blocks produces two artifacts with two destinations:

stored blocks (JSON tree, database)
↓  render each block to markdown, hash each rendering
┌──────────────────────────────┬──────────────────────────────┐
│ anchored markdown            │ snapshot                     │
│ → goes TO the model          │ → stays on the SERVER        │
└──────────────────────────────┴──────────────────────────────┘

The model receives the full document — every block, complete — with one anchor line naming each block's id:

<!-- block:h1K2p9 -->
## Webhooks

<!-- block:aB3xK9 -->
Deliveries are sent whenever a document is published.

<!-- block:pQ7mN2 -->
Every request carries a signature header.

<!-- block:zT5rW8 -->
Failed deliveries are retried three times.

<!-- block:qF4dR7 -->
You can replay any delivery from the dashboard.

The server keeps the snapshot: for each block, the handle and the hash of its rendered markdown (without the anchor lines, just the actual text from the document).

{ "docId": "doc_web1",
  "blocks": [
    { "handle": "h1K2p9", "hash": "6a5362c0…" },
    { "handle": "aB3xK9", "hash": "80c7da16…" },
    { "handle": "pQ7mN2", "hash": "0db39487…" },
    { "handle": "zT5rW8", "hash": "01e5960c…" },
    { "handle": "qF4dR7", "hash": "27788d86…" }
  ] }

The snapshot is the server's record of what the model saw. During a turn it's an in-memory map; between turns it's persisted as a JSON column on the same database row that stores the session's conversation history. The model never sees a hash. It doesn't know hashes exist.

anchors      → the model   ("which block do you mean?")
fingerprints → the server  ("is that block still what you read?")

The edit: change the third paragraph from X to Y

Now the running example. You tell the agent:

In the third paragraph, change "retried three times" to "retried five times".

What the model does. It finds the third paragraph by reading — the full document is in its input, so "third paragraph" is a comprehension task, not a lookup API. That's block zT5rW8. It emits one tool call:

{ "tool": "edit_blocks", "docId": "doc_web1",
  "operations": [
    { "op": "replace", "blockId": "zT5rW8",
      "markdown": "Failed deliveries are retried five times." }
  ] }

That is the model's entire output for this edit!!! An address, and the new text for one block. In our logs: 80–170 output tokens per edit, against ~1,550 for the full rewrite. The other four blocks were read, understood, and never retyped.

(The tool speaks five operations: replace, insert_before, insert_after, delete, append. "Add a note under the heading" is an insert_after; "remove that paragraph" is a delete. Same machine throughout.)

What the server does. Before touching anything, it asks one question — is the block the model wants to change still the same as when the model read it?

The server does not hash the markdown the model just sent. Of course that hash would differ — new content hashing differently is the point of new content.

Instead, three versions of the target block are on the table:

the PAST     — the snapshot hash: what the model READ
the PRESENT  — the block as stored RIGHT NOW, re-rendered, re-hashed
the FUTURE   — the op's markdown: what the model WANTS

The check compares past against present. The future is never hashed — it's the payload, not the evidence. In words:

"Since the model read this block, has anyone changed it?
 No  → the edit was decided while looking at current text. Apply it.
 Yes → the edit was decided on a stale picture. Refuse — and show the model the present."

For our edit: the present renders to Failed deliveries are retried three times., which hashes to 01e5960c… — equal to the snapshot. Not stale. Proceed.

The apply is array surgery, not document reassembly. The server parses the model's one line of markdown into one new block (assigning it a fresh id), finds the target's position, and splices:

[ h1K2p9, aB3xK9, pQ7mN2, zT5rW8, qF4dR7 ]
                          ↑ replace
[ h1K2p9, aB3xK9, pQ7mN2, NEW,    qF4dR7 ]

Look at the four other blocks. They are the same stored objects. Not re-rendered, not re-parsed, not re-created — spliced around. This is the property no text-file edit format can give you, because a text file has no richer layer to protect: in a structured store, an untouched block is not a faithful reproduction of itself. It's the same bytes. It cannot drift, because it never moved.

Then one save, and the server refreshes its snapshot so the model's own edit counts as "seen" — otherwise the agent's next operation would collide with its own work.

And if someone else edited that paragraph first? Say a teammate changed it to Failed deliveries are retried with exponential backoff. while the model was thinking. The present now hashes to d80a9820…; the snapshot still says 01e5960c…. Mismatch → the whole call is rejected atomically (every operation validated before any is applied — a failed call means a byte-identical document), and the error is a targeted re-read:

Error: block "zT5rW8" was modified after you read it.
Current content of this block:
Failed deliveries are retried with exponential backoff.
Re-issue the operation against this content.

~100 tokens, and the model retries against reality instead of its recollection. The same style of self-correcting error covers a hallucinated block id (the error returns the real outline: every current handle plus the first 60 characters of its block). The failure mode is always "no edit — try again, informed", never "wrong edit". This is per-block optimistic concurrency, the same idea as HTTP's ETag/If-Match, applied per paragraph: a teammate editing block 12 never blocks the agent's edit of block 3.

The second turn: what the agent receives tomorrow

Next day, same session, you ask: "actually, make that paragraph mention the backoff delay too."

Look at what the model's input actually is tomorrow:

[replayed history]
  turn 1: the FULL anchored document        ← still here, re-sent verbatim
  turn 1: the model's own edit_blocks call  ← its record of what it changed
  turn 1: "Edited: replaced block zT5rW8"
[fresh]
  [Document unchanged since your last read]  ← the delta line your new message

The model re-reads yesterday's full document every turn — that's what replaying history means. The one-line marker isn't the document. It's a certificate: "that copy of the document sitting earlier in your transcript, plus your own edits — still accurate. Trust it."

That reframing answers the cost question too, which is the part that makes the design clever rather than just correct. The full document is being paid for every turn — but in the input lane, as part of a byte-stable replayed prefix, which is exactly the lane that's cheap and cache-discounted.

This is a fresh HTTP request. The server replays the stored conversation — and here the doc snapshot earns its keep a second time. Instead of stapling a fresh full copy of the document onto the transcript (what we used to do, every turn), the server re-renders the current blocks, diffs their hashes against the snapshot, and injects one of three things:

nothing changed        → one line:
                         [Document unchanged since your last read]

a few blocks changed   → only those blocks, with their anchors:
                         --- Changed blocks ---
                         <!-- block:zT5rW8 -->
                         Failed deliveries are retried five times.
                         --- End of changed blocks ---

a lot changed (>30%)   → give up, re-inject the whole document

The model's picture of the document is now three layers, all cheap: the original full injection (still in the replayed history), its own edit operations (also in history — the record of what it changed), and this delta. Full context, delta price. In our runs a typical delta injection was ~220 tokens where a full re-read was thousands.

If you know rsync, this is rsync — the snapshot plays the role of the receiver's checksum list, except the "receiver" being synchronized is the model's context window.

The other lever: compact the replayed history

Block editing cuts what the model writes. The transcript it reads every turn has its own disease, and a different cure.

By turn five of a session, the replayed history was carrying: the turn-1 document injection, a fresh full injection stapled to every user message, every full-document rewrite the model ever emitted as tool arguments, and every raw search-result dump. The same document, over and over, in slightly different wrappers — we measured 71,812 input tokens re-sent at the start of a single turn.

Compaction is a pure function that runs when history is replayed. No summariser model, no AI call. It walks the stored messages and swaps known bulky payloads for one-line placeholders:

Read the placeholders again — they aren't apologies, they're directions. Each one tells the model where the authoritative version now lives: the fresh injection, or a tool it can call again. That's the rule the whole elision list was built from: remove bytes only when their authoritative version still reaches the model somewhere else. The user's words, the model's own replies and decisions, tool names, error messages — never touched.

The numbers

Everything below is from A/B runs of a production agent (GPT-5.1) against the same ~7.4 KB document with the same five small, surgical edit prompts — one variable toggled per experiment, token counts read straight from per-iteration logs.

Output — the headline. The edit call itself, per turn:

Session totals: 775 vs 8,128 output tokens — ~10× less. And the saving is flat: the rewrite pays full document price for a one-word change on turn five exactly as on turn one. There is no session length at which it catches up.

Speed. Median edit latency 5.3 s vs 16.9 s (~3.2×); averages 6.8 s vs 16.7 s. Generating ~1,550 tokens takes real wall-clock time; generating ~130 doesn't. This is the difference users feel on every single edit.

Input, as a side effect. Because block editing never deposits a full-document rewrite into the history, the replayed transcript grows ~294 tokens/turn instead of ~3,176 — 11× slower. Over five edits: 78,034 vs 180,032 input tokens (−57%) — without compaction even enabled.

Compaction, measured separately (block editing off in both runs, so the only variable is compaction):

Turn one is a tie — nothing to elide yet. By turn five, −71%; across the session, −60%, and the gap widens every turn. With compaction on, the transcript grows roughly as fast as the document itself; without it, by about a whole extra document copy per turn.

Correctness — the result that surprised me. Both modes applied 5/5 edits correctly, including two deliberate traps: a decoy (the target phrase appeared twice; only the asked-for instance was to change) and a two-mention update ("change every place it says 24 hours"). So I won't tell you block editing is more accurate — on well-specified edits it measurably isn't. The honest claim is different: it's equally correct while re-generating ~60× less text per edit, which means a far smaller surface for silent drift, and a ~3× shorter window in which a dropped connection can kill a half-finished rewrite. Same outcome, smaller blast radius.

How the two levers stack. They overlap on input (both attack the same history bloat from different ends), so the savings don't add — but they own different lanes, and the lanes are clean:

block editing owns OUTPUT  (~10× less, ~3× faster — compaction can't touch output)
compaction    owns INPUT   (~60% less replayed history, compounding with session length)

One honest asterisk on compaction: rewriting the transcript prefix disturbs prompt-cache locality, so its effect on provider-side cost depends on your traffic pattern (with warm caches it's roughly neutral; with cold caches it wins outright). Its effect on raw token counts — which is also what usage quotas typically meter — is the clean ~60%.

Prior art, honestly

I arrived at this independently, and then discovered — as you should assume about any good idea under economic pressure — that the field was converging on the pieces. The pricing asymmetry (expensive uncached output, cheap cached input) pushes everyone toward the same shape:

• Aider's search/replace edit formats attack the same retyping problem for code: emit only the diff, never the who le file (their unified-diffs write-up is the classic argument against whole-file rewrites).

• antirez sketched line edits carrying a small content checksum, checked before apply — verify-before-write at line granularity, pos itional addressing.

• OMP's "hashline" edits give every line a content-hash the model references instead of r etyping — content addressing, with the hashes travelling through the model's context.

• Anthropic's context editing clears stale tool results from the transcript and leaves placeholders — the same family as our compaction.

• Notion's API exposes block-id-targeted updates — stable ids on blocks, commercially at scale.

What I haven't seen assembled anywhere public is the composition this post describes, and each piece of the composition does specific work: identity addressing rather than position or content (addresses that survive both movement and editing); fingerprints of the rendered view rather than the storage bytes (staleness defined as "would the model see different text", eliminating false conflicts); fingerprints held server-side rather than routed through the model (no per-line token tax, nothing for the model to mangle); the persisted snapshot doing double duty — guarding writes and powering the delta re-injection that keeps follow-up turns cheap; and, available only because the document store is ours, untouched blocks that are never serialized at all. The deepest ancestry is older than all of us: per-block ETag/If-Match, git's content hashing, rsync's checksum lists — pointed at a context window instead of a cache or a network link.

If you're building an agent over any block-structured store — a CMS, a notes product, a site builder, an email composer — that composition transfers whole.

The honest limits

Input still scales with document size. The model reads everything, every session. Deliberate: input is the cheap, cacheable lane, and full context is what keeps edits grounded. This method saves where it's expensive and spends where it's safe.

One giant block is still one giant block. replace re-emits the whole target block, so nothing is saved inside a 20k-token table. If your users build monster blocks, you'll eventually want sub-block addressing.

Cross-block renames are the wrong job for it. An 18-occurrence rename cost 855 output tokens in our runs, because the model re-emitted every affected block. A plain server-side find-and-replace tool does it for ~40.

The concurrency is optimistic and per-block, not a global transaction. It catches the overwhelmingly common conflicts — and its failure mode is refusal, not corruption — but if you need multi-writer guarantees across blocks, that's CRDT territory, a different (and far heavier) machine.

Blocks without ids degrade the experience, not the safety. Content+position fallback handles keep edits sound but turn "block changed" conflicts into blunt "block not found" errors and make the delta optimization give up more often.

If you want to build this

The checklist is short:

1. You own a structured document store — a tree of typed blocks, not opaque text files.

2. Every block carries a stable id. If your editor already assigns ids for deep links or drag-and-drop, this is free. This is the prerequisite everything else stands on.

3. One deterministic per-block markdown renderer, used at read time and check time — the hashes are only comparable because both come from the same function.

4. Somewhere to persist the per-session snapshot — a JSON column next to your conversation history is plenty; ~50 bytes per block.

5. Errors that teach. Every rejection should hand the model the current truth it was missing — the block's present content, the real outline. The error channel is your sync channel.

Then the flow is the one you've just read:

The read hands the model addresses and keeps the fingerprints.
The edit names an address, and the fingerprints judge it — past against present — before anything moves.
The follow-up diffs the fingerprints, so the model's picture stays current at delta price.

The model reads everything, addresses one block, and retypes nothing else. That single change moved our bill's most expensive line item by 10× — and made the agent faster and less destructive at the same time. The rare alignment of cheaper, faster, and safer is usually a sign the previous design was paying for something nobody needed. Re-typed text nobody asked for was exactly that.

One rule to keep in your head the whole way down:

TypeScript has two universes: the value space (runtime) and the type space (compile time). const, let, functions, objects live in the value space. type, interface, keyof live in the type space. typeof is the door from value space into type space. Almost everything below is a consequence of that.

1. Interfaces (and the fact you can declare one twice)

Let's say you're modeling a user, and you declare it twice in the same file:

// user.ts
interface User {
  id: string;
  email: string;
}

interface User {
  plan: 'free' | 'pro';
}

// User is now { id: string; email: string; plan: 'free' | 'pro' }

Nobody overwrote anything. Two declarations of the same interface name merge their members. This is called declaration merging, and it is the one thing interface can do that type cannot.

The rule that everyone gets wrong

Merging happens within a declaration space, not within a project. And every file with a top-level import or export is its own module — its own declaration space.

So this does not merge the interface:

// user.ts
export interface User { id: string; email: string }

// plugin.ts
export interface User { plan: 'free' | 'pro' }

These are two unrelated types that happen to share a name. import { User } from './user' still gives you { id, email }. And if plugin.ts tries to import the original alongside its own declaration, you don't get a merge — you get Import declaration conflicts with local declaration of 'User'.

A file only merges into another file's types in two situations.

1. Both files are global. A file with no top-level import/export isn't a module — it's a script, and its declarations land in the global space:

// globals.d.ts — no imports, no exports, deliberately
interface Window {
  __APP_STATE__: AppState; // ✅ merges into lib.dom's Window
}

// app.ts
window.__APP_STATE__; // ✅ compiles

Now add one line to the bottom of globals.d.ts:

// globals.d.ts - adds an export in the file
export {};

// app.ts
window.__APP_STATE__;
// ❌ Property '__APP_STATE__' does not exist on type 'Window & typeof globalThis'.

Nine characters with no semantic content, and the merge is gone. globals.d.ts is now a module, so its Window is a private local interface that happens to share a name with the real one.

Note where the error lands: in app.ts — the file you didn't touch. The file you actually broke reports nothing at all. This is why the bug is so expensive: you go hunting in the consumer, and the cause is a stray export (or an editor-inserted auto-import) in a declaration file you weren't even looking at.

The escape hatch is declare global, which pushes you back out into global scope from inside a module:

// globals.d.ts
import type { User } from './types'; // ← this alone makes the file a module

declare global {
  interface Window {
    __APP_STATE__: { user: User }; // ✅ merges again
  }
}

export {};

declare global is the global-scope sibling of declare module 'specifier'. Same job: name the declaration space you want to open.

2. Module augmentation. The declaring file explicitly names the module scope it wants to reopen:

// express.d.ts — you never touch Express' source
import 'express'; // this file is a module...

declare module 'express-serve-static-core' {
  interface Request {
    currentUser?: User; // ...but this block lands in Express' declaration space
  }
}

// now req.currentUser type-checks in every handler in your app

declare module 'specifier' is the actual mechanism. Without it, a file cannot reach into another module's types — which is the right design. Otherwise any file in your repo could silently mutate any type in any other file.

This is how you extend next-auth's Session, add a field to Express' Request, or teach fastify about your decorators. It's also why "just declare it again" never works when people try it in a normal .ts file.

The trade-off: an interface is open. Anyone who can name your module can add to it, and you'll only find out at the call site. That openness is the exact thing type gives up.

2. Types (and the fact they can't be re-declared)

type User = {
  id: string;
  email: string;
};

type User = {
  plan: 'free' | 'pro';
};
// Error: Duplicate identifier 'User'.

A type is a name for an expression. Like const in the value space, the name is bound exactly once. You can't reopen it, which means when you read type User = ..., that is the whole truth about User in that file.

And because it's just a name for an expression, it can name things an interface can't. Here are the six you'll actually use — each one explained properly, because the names are useless until you see the logic.

| — a union: "one of these"

Read | out loud as "or".

type Status = 'draft' | 'published' | 'archived';

This says: a Status is the string 'draft', or the string 'published', or the string 'archived'. Nothing else.

Not "a string." Those three strings. That's the point.

const a: Status = 'draft';     // ✅
const b: Status = 'published'; // ✅
const c: Status = 'deleted';   // ❌ Type '"deleted"' is not assignable to type 'Status'.
const d: Status = 'Draft';     // ❌ capital D — not one of the three

You can union anything, not just strings:

type Id = string | number;              // could be either
type MaybeUser = User | null;           // a user, or nothing
type Input = string | string[];         // one item or many

An interface cannot do this. There's no way to write "an interface that is one of three strings" — an interface describes an object, and 'draft' is not an object. That is the single biggest reason type exists.

[a, b] — a tuple: an array where each position has a meaning

A normal array says "many things, all the same type, any number of them":

type Scores = number[];
const s: Scores = [1, 2, 3, 4, 5]; // ✅ any length is fine

A tuple says "exactly this many things, in this order, and each slot has its own type":

type Point = [number, number];

const p: Point = [12, 40];     // ✅ x and y
const q: Point = [12];         // ❌ Source has 1 element but target requires 2.
const r: Point = [12, 40, 99]; // ❌ too many
const t: Point = [12, 'a'];    // ❌ second slot must be a number

The slots can have different types, which is where it gets useful:

type Result = [Error, null] | [null, User];

That reads: either an error and no user, or no error and a user. Never both, never neither. (That's a union of two tuples — you've now stacked two of these ideas.)

You already use tuples every day without noticing. This is what useState returns:

const [count, setCount] = useState(0);
// useState returns [number, (n: number) => void]
// slot 0 is the value, slot 1 is the setter — different types, fixed order

That's why destructuring useState gives you correctly-typed variables even though you invented the names count and setCount yourself. TypeScript isn't matching names, it's matching positions.

<T> — a generic: a type with a hole in it

This is the one that scares people, and it shouldn't.

A generic is a type that takes a type as an argument. It's a function — but instead of taking a value and returning a value, it takes a type and returns a type.

Let's say you keep writing this:

type NullableUser = User | null;
type NullablePost = Post | null;
type NullableComment = Comment | null;

Same shape three times. The only thing that changes is the type on the left. So punch a hole in it and give the hole a name:

type Nullable<T> = T | null;
//            ↑        ↑
//     the hole    the hole, used

Now you call it, the same way you'd call a function — but with angle brackets instead of parens:

type NullableUser = Nullable<User>;    // → User | null
type NullablePost = Nullable<Post>;    // → Post | null
type MaybeName = Nullable<string>;     // → string | null

T is just a parameter name. It has no special meaning. You could write Nullable<Whatever> = Whatever | null and it would work identically. T is convention, short for "Type", the way i is convention in a for-loop.

Substitution is the whole mental model. Nullable<User> means: take the definition, and wherever you see T, write User instead.

Nullable<T>    =  T    | null
Nullable<User> =  User | null

That's it. That's generics. Every scary-looking generic in the wild is that same substitution, just nested.

You've been using them already:

Array<string>       // an array whose items are strings — same as string[]
Promise<User>       // a promise that will produce a User
Record<string, number>  // an object with string keys and number values
Pick<User, 'email'>     // (section 5) — it takes TWO type arguments

(x) => y — a function type: the shape of a callable

You can name the signature of a function, without writing the function.

type Handler = (event: MouseEvent) => void;

That reads: a thing you can call with a MouseEvent, which gives you back nothing. (void = "returns nothing useful, ignore the return value.")

Let's say five components all take an onClick. Name the shape once:

type Handler = (event: MouseEvent) => void;

interface ButtonProps {
  onClick: Handler;
  onHover: Handler;
}

const handleClick: Handler = (event) => {
  console.log(event.clientX); // ✅ TypeScript knows `event` is a MouseEvent
};

Notice you never annotated event in handleClick. You didn't have to — the type flowed in from Handler. That's called contextual typing, and it's most of why annotating your props properly is worth the effort: you write the type once, and every callback beneath it becomes free.

keyof — give me the key names, as a union

keyof takes an object type and hands you back the list of its property names, as a union of strings.

interface User {
  id: string;
  email: string;
  createdAt: Date;
}

type UserKey = keyof User;
// → 'id' | 'email' | 'createdAt'

That's the whole operation. keyof in, union of key names out.

Compare it to something you already know:

Object.keys(user)  // value space, at runtime  → ['id', 'email', 'createdAt']
keyof User         // type space, at compile time → 'id' | 'email' | 'createdAt'

Same idea, different universe. One gives you an array of strings you can loop over. The other gives you a union of strings the compiler can check against.

Why you'd want it — let's say you're writing a sort function:

const sortBy = (users: User[], key: keyof User) => { /* ... */ };

sortBy(list, 'email');     // ✅
sortBy(list, 'emial');     // ❌ Argument of type '"emial"' is not assignable
                           //    to parameter of type 'keyof User'.
sortBy(list, 'password');  // ❌ that field doesn't exist on User

If you'd typed key: string, all three would compile and two would silently sort by nothing. keyof means "a real key of this thing, not just any old string." And you get autocomplete on the argument.

User['id'] — indexed access: give me the type of that property

Square brackets, but in the type space.

interface User {
  id: string;
  createdAt: Date;
}

type Id = User['id'];               // → string
type Timestamp = User['createdAt']; // → Date

Read it exactly like property access, because it is property access — just one universe up:

user['id']   // value space → the actual id, "abc-123"
User['id']   // type space  → the TYPE of that field, string

Why bother, when you could just write string?

Let's say six months from now you brand your IDs for safety (section 15):

interface User {
  id: UserId;  // ← changed from string
  createdAt: Date;
}

Every place that wrote type Id = User['id'] updates itself. Every place that hand-wrote string is now quietly, invisibly wrong. That's the whole game: derive, don't duplicate.

And now the part that pays off later — you can index with a union of keys, and you get back a union of the types:

type Values = User['id' | 'createdAt'];  // → string | Date

Feed it every key at once, and you get every value type:

type AllValues = User[keyof User];  // → string | Date
//                    ↑
//            'id' | 'createdAt'

Sit with that line for a second, because it is section 4 — the (typeof X)[keyof typeof X] monster — with the scary part removed. Same two operators, same order. You already understand it.

So: an interface can only ever describe the shape of an object. A type can describe any type — a union, a tuple, a function, or something computed from another type entirely.

3. interface vs type — the actual decision

That last row is the one worth internalizing.

interface A { id: string }
interface B extends A { id: number }
// Error: Interface 'B' incorrectly extends 'A'.
//        Type 'number' is not assignable to type 'string'.

TypeScript stops you. Now the same thing with an intersection:

type A = { id: string };
type B = A & { id: number };

const b: B = { id: ??? }; // id is `string & number` → never. Nothing satisfies it.

No error at the declaration. The bug is deferred to whoever tries to construct a B, and the error they get ("Type 'string' is not assignable to type 'never'") points at their code, not at yours.

4. (typeof CustomType)[keyof typeof CustomType] — the one that makes it click

Read it inside out, and keep the two worlds separate: CUSTOM_TYPE the value (a real object at runtime) and the type it happens to have. Every operator here just moves between those worlds.

Start with the value:

const CUSTOM_TYPE = {
  FIRST: 'first',
  SECOND: 'second',
  THIRD: 'third',
} as const

typeof CUSTOM_TYPE — "give me the type of that value". This is the type-level typeof, nothing to do with the JS typeof that returns 'string'. It's the bridge from value-world to type-world.

// typeof CUSTOM_TYPE
{
  readonly FIRST: 'first'
  readonly SECOND: 'second'
  readonly THIRD: 'third'
}

keyof typeof CUSTOM_TYPE — keyof takes a type and hands back a union of its keys:

'FIRST' | 'SECOND' | 'THIRD'

(typeof X)[keyof typeof X] — an indexed access. Same syntax as obj[key] in JS, but at the type level: index a type with a key, get the value type at that key.

I like to imagine typeof X as an object with keys. and keyof that object provides those keys. this reads like obj[key]

type A = (typeof CUSTOM_TYPE)['FIRST']  // 'first'

The trick is that indexing with a union of keys gives you a union of the value types:

(typeof CUSTOM_TYPE)['FIRST' | 'SECOND' | 'THIRD']
// → 'first' | 'second' | 'third'

So the whole thing reads: "the union of all value types in CUSTOM_TYPE." In plain JS terms, it's Object.values(), at the type level.

The parens around typeof CUSTOM_TYPE are only there for precedence — without them typeof X[...] parses as typeof (X[...]), which is a different thing entirely. That's a big part of why the line looks so dense: you're reading X twice, once inside parens and once inside keyof.

If it helps, break it up — the meaning survives intact:

type CustomTypeMap = typeof CUSTOM_TYPE
type CustomTypeKey = keyof CustomTypeMap              // 'FIRST' | 'SECOND' | 'THIRD'
export type CustomType = CustomTypeMap[CustomTypeKey] // 'first' | 'second' | 'third'

I'd argue that's better code anyway — the intermediate names are free and they document the two halves.

Why bother at all: it keeps the type derived from the object. Add a fourth member to CUSTOM_TYPE and the union updates itself, and every switch that was exhaustive now fails to compile until you handle the new case. Write the union by hand and it drifts.

There's also a generic helper worth stashing, since you'll hit this pattern constantly:

type ValueOf<T> = T[keyof T]

export type CustomType = ValueOf<typeof CUSTOM_TYPE>

Same thing, reads like English.

Same trick, other shapes

const ROUTES = ['/home', '/settings', '/billing'] as const;
type Route = (typeof ROUTES)[number]; // "/home" | "/settings" | "/billing"

An array is an object whose keys are numbers, so indexing its type with number gives you the union of its elements. Same mechanism.

5. Pick, Omit

Let's say you have this:

interface User {
  id: string;
  email: string;
  passwordHash: string;
  createdAt: Date;
}

Your API route must never leak passwordHash. Don't hand-write a second interface — it will drift from the first one within a week.

type PublicUser = Omit<User, 'passwordHash'>;
// { id: string; email: string; createdAt: Date }

type UserCredentials = Pick<User, 'email' | 'passwordHash'>;
// { email: string; passwordHash: string }

Pick = keep these keys. Omit = drop these keys. They're derivations: add a field to User tomorrow and PublicUser gets it for free, while UserCredentials stays exactly as narrow as it was.

The subtle difference: Pick<User, 'emial'> is a compile error (the key must exist), while Omit<User, 'emial'> silently does nothing — Omit's key parameter isn't constrained to keyof T. So typos in Omit are invisible. Prefer Pick when the safe list is short; use Omit when the removal list is short and review it.

Neighbours you'll reach for constantly:

Partial<User>            // every field optional  — patch payloads
Required<User>           // every field required
Readonly<User>           // every field readonly
Record<Status, string[]> // { draft: string[]; published: string[]; ... }
ReturnType<typeof fn>    // what fn returns
Parameters<typeof fn>    // its args as a tuple
Awaited<ReturnType<typeof fetchUser>> // unwrap the Promise
Exclude<Status, 'archived'>           // remove members from a union
Extract<Status, 'draft' | 'x'>        // keep members of a union
NonNullable<T>                        // T minus null and undefined

Note Omit/Pick work on object types; Exclude/Extract work on unions. Same idea, different universe of "members".

6. interface extends otherInterface

Let's say every entity in your DB has the same audit columns:

interface Entity {
  id: string;
  createdAt: Date;
  updatedAt: Date;
}

interface User extends Entity {
  email: string;
}

interface Document extends Entity {
  title: string;
  ownerId: User['id']; // indexed access — if id becomes a branded type, this follows
}

extends here means "has at least everything Entity has, plus these." It's not inheritance in the OOP sense — there's no runtime, no prototype chain. It's a subtyping assertion, checked at compile time and erased.

You can extend multiple interfaces, and you can extend a type as long as it's object-shaped:

type Timestamped = { createdAt: Date };
interface Post extends Timestamped, Entity { title: string }

And remember section 3: if Post declares a member that conflicts with Entity, you get an error at the declaration — which is exactly what you want.

7. A & B and A | B

type Draggable = { onDrag: () => void };
type Resizable = { onResize: () => void };

type Widget = Draggable & Resizable; // must have BOTH methods
type Event = Draggable | Resizable;  // has at LEAST one of them

Here's the part that confuses people: & (AND) gives you a bigger object; | (OR) gives you a smaller usable object.

That's because a type is a set of possible values.

• A & B = the set of values that are simultaneously an A and a B. To qualify, a value needs more properties. More constraints, fewer values.

• A | B = the set of values that are an A or a B. More values qualify — so TypeScript knows less about any given one.

const handle = (e: Draggable | Resizable) => {
  e.onDrag(); // Error: Property 'onDrag' does not exist on type 'Resizable'.
};

Of course it doesn't. TypeScript doesn't know which one you have. You only get to touch the properties that exist on every member of the union — until you narrow it (section 10).

8. The mutually-exclusive props pattern (never as a "you may not pass this")

Let's say you have a delete function that takes either one id, or many. Naively:

interface DeleteArgs {
  id?: string;
  ids?: string[];
}

deleteItems({});                          // 🙃 valid, deletes nothing
deleteItems({ id: '1', ids: ['2', '3'] }); // 🙃 valid, which wins?

The type says "both optional" when you meant "exactly one." Encode the logic:

type DeleteArgs =
  | { id: string;  ids?: never }
  | { id?: never;  ids: string[] };

deleteItems({ id: '1' });              // ✅
deleteItems({ ids: ['1', '2'] });      // ✅
deleteItems({ id: '1', ids: ['2'] });  // ❌ Type 'string[]' is not assignable to type 'never'.
deleteItems({});                       // ❌ neither branch satisfied

Why does this work? never is the type with zero possible values (section 12). So ids?: never reads as: this key is optional, and if you do provide it, there is no value on earth that will type-check. It's a compile-time "this key must be absent."

Important detail: the ? is load-bearing. { id: string; ids: never } would make the branch impossible to satisfy, because ids would be a required property that can never be given a value. Optional-never = "must be absent." Required-never = "this object cannot exist."

Inside the function, narrow with a truthiness check or a discriminant:

const deleteItems = (args: DeleteArgs) => {
  const idList = args.ids ?? [args.id];
  // ...
};

If you find yourself writing many of these, add an explicit discriminant instead — it's cheaper for the compiler and clearer for humans:

type DeleteArgs =
  | { mode: 'single'; id: string }
  | { mode: 'bulk';   ids: string[] };

Now mode is a tag. TypeScript builds an internal map from 'single' → first member, 'bulk' → second member, so narrowing becomes a lookup rather than an assignability check against each shape. That's the "cheaper for the compiler" part.

Then check like so:

const remove = (args: DeleteArgs) => {
  if (args.mode === 'single') {
    return deleteOne(args.id);
  }
  return deleteMany(args.ids);  // narrowed to 'bulk' by elimination
};

9. as const — and what happens without it

Let's say you write a config object:

const config = {
  env: 'production',
  retries: 3,
  hosts: ['a.com', 'b.com'],
};

What did TypeScript infer?

// {
//   env: string;          ← not "production"
//   retries: number;      ← not 3
//   hosts: string[];      ← mutable, element type string
// }

This is widening. Because object properties are mutable, TypeScript assumes you might later write config.env = 'staging', so it widens the literal 'production' up to string. Which means this fails:

type Env = 'production' | 'staging';
const env: Env = config.env; // Error: Type 'string' is not assignable to type 'Env'.

Now add as const:

const config = {
  env: 'production',
  retries: 3,
  hosts: ['a.com', 'b.com'],
} as const;

// {
//   readonly env: "production";
//   readonly retries: 3;
//   readonly hosts: readonly ["a.com", "b.com"];
// }

as const says: "nothing here will ever be reassigned, so don't widen anything." Every property becomes readonly, every literal stays literal, every array becomes a readonly tuple.

Two consequences worth knowing:

1. It's what makes section 4's (typeof X)[keyof typeof X] produce "draft" | "published" instead of a useless string.

2. readonly string[] is not assignable to string[]. If a function takes string[], passing config.hosts is an error. Widen the parameter to readonly string[] — you almost never actually needed a mutable array.

Its better half: satisfies

as const and a type annotation each do half the job, and they get in each other's way.

Annotate, and you get checking — but you lose the literals:

const ROUTES: Record<string, `/${string}`> = {
  home: '/home',
  billing: 'billing', // ❌ caught: no leading slash. Good.
};

ROUTES.home; // `/${string}` — not '/home'. You were told the type; TS forgot the value.

An annotation is a command: you are this type now. TypeScript takes you at your word and stops looking at what you actually wrote.

Use as const, and you keep the literals — but nothing is checked:

const ROUTES = {
  home: '/home',
  billing: 'billing', // ✅ compiles. Nobody asked whether this was a valid path.
} as const;

ROUTES.home; // '/home'. Exactly what you wrote.

satisfies is the missing piece. It checks the value against a constraint and then throws the constraint away — it never becomes the variable's type:

const ROUTES = {
  home: '/home',
  billing: '/billing',
} as const satisfies Record<string, `/${string}`>;

Two jobs, one line:

• as const → freezes the literals. ROUTES.home is '/home', and keyof typeof ROUTES is 'home' | 'billing', not string.

• satisfies → audits them. Drop the leading slash and the build fails.

Which means this now works, and is trustworthy:

type Route = (typeof ROUTES)[keyof typeof ROUTES];
// '/home' | '/billing'

With the annotation, Route would have been `/${string}` — useless as a union. With as const alone, a typo would have quietly become a member of it.

Rule of thumb: an annotation declares a type. as const preserves one. satisfies verifies one without replacing it. You want the last two together.

One caveat worth knowing: satisfies on its own sometimes preserves literals too, depending on the constraint. Don't rely on it. If you want literals, say as const — then the behavior is a rule, not a coincidence.

10. Type guards (value is Type) — teaching the compiler what you know

Let's say you're handling something from a catch or a JSON.parse. You know it's a User because you checked. TypeScript doesn't.

A type predicate is a function whose return type is arg is Type. When it returns true, the compiler narrows the argument in the calling scope.

const isUser = (value: unknown): value is User =>
  typeof value === 'object' &&
  value !== null &&
  'id' in value &&
  typeof (value as User).id === 'string';

const handle = (payload: unknown) => {
  if (!isUser(payload)) return;
  payload.email; // ✅ payload is User in here
};

The is is a promise you are making to the compiler. If your check is wrong, TypeScript will happily believe you and you'll crash at runtime. It's a controlled as, not a proof.

The narrowings you get for free

if (typeof x === 'string')      // primitives
if (Array.isArray(x))           // arrays
if (x instanceof Error)         // classes
if ('id' in x)                  // the `in` operator, for object unions
if (x !== null)                 // truthiness / null checks

Discriminated unions — the pattern to reach for first

Give every member of a union a shared literal field. TypeScript narrows on it automatically, and you write zero guards:

type Result =
  | { status: 'loading' }
  | { status: 'success'; data: User }
  | { status: 'error'; error: Error };

const render = (result: Result) => {
  switch (result.status) {
    case 'loading': return spinner();
    case 'success': return view(result.data);   // ✅ .data exists only here
    case 'error':   return alert(result.error); // ✅ .error exists only here
  }
};

This is 90% of the value of TypeScript in one construct. It makes illegal states unrepresentable — you cannot have a Result that is both loading and holding data.

11. Recursive types

For example, you have a comment tree, a nav menu, a file system:

interface Comment {
  id: string;
  body: string;
  replies: Comment[]; // fine — self-reference through a property
}

Recursion gets powerful when combined with mapped types.

Here's a shape you've written a hundred times:

type User = {
  id: string;
  profile: {
    name: string;
    address: {
      city: string;
      zip: string;
    };
  };
};

Now write the type for a PATCH payload — the same shape, but every field optional, because the client only sends what changed.

Your first instinct is the built-in:

type PatchUser = Partial<User>;

Hover it, and you get this:

{
  id?: string;
  profile?: {
    name: string;                            // ← still required
    address: { city: string; zip: string };  // ← still required
  };
}

It made the top-level keys optional and stopped. Which means the payload you actually want to send doesn't compile:

const patch: PatchUser = {
  profile: { name: 'Ada' },
};
// ❌ Property 'address' is missing

To rename a user you'd have to resend their entire address. That's the problem. Now let's look at why Partial behaves this way, because the fix falls out of it.

Partial is one line, and it only loops once

This is the whole thing, straight from the standard library:

type Partial<T> = { [K in keyof T]?: T[K] };

That's a mapped type. Read it as a for-loop that builds a new object type:

for each key K in keyof T:
  emit key K, but optional (?)
  give it the type T[K] — whatever it was before

Applied to User, the loop runs twice:

Look at the second row. It made the profile key optional. But the value — that nested object — was copied across verbatim. T[K] hands it over untouched. Nobody ever went inside it.

That's the bug in one sentence: the loop runs once, at one level. The nested object is just a value being copied, not a thing being processed.

The fix: process the value instead of copying it

So don't copy T[K]. Run it through the same transformation again:

type DeepPartial<T> = {[K in keyof T]?: DeepPartial<T[K]>};
//                                       ^^^^^^^^^^^^^^^^^
//  instead of T[K], it's DeepPartial<T[K]>

One token changed. T[K] became DeepPartial<T[K]>.

That's the recursion, and it's not a clever trick — it's the obvious move once you see that T[K] was the place the descent should have happened and didn't.

But now it never stops

Run DeepPartial<User> with the definition above and follow the id key:

• id's type is string

• so we call DeepPartial<string>

• which expands to { [K in keyof string]?: DeepPartial<string[K]> }

• and keyof string is... 'charAt' | 'slice' | 'length' | 'toUpperCase' | ...

You get an object with optional charAt and toUpperCase properties. Which is not a string, and not what anyone wanted.

The loop needs a floor. Primitives are the floor — there's nothing inside a string to make optional, so when you hit one, hand it back and stop:

type DeepPartial<T> = T extends object
  ? { [K in keyof T]?: DeepPartial<T[K]> }   // it's an object — go in
  : T;                                        // it's a primitive — done

That conditional is the base case. Every recursive type needs one, and it's almost always "am I still looking at something with keys?"

Trace it end to end

DeepPartial<User>. User is an object, so loop over its keys:

id → DeepPartial<string> → string is not an object → returns string. Emits id?: string. ✅ floor reached.

profile → DeepPartial<{ name; address }> → that is an object → loop again:

    name → DeepPartial<string> → string. ✅ floor.

    address → DeepPartial<{ city; zip }> → object → loop again:

        city → string. ✅ floor.         zip → string. ✅ floor.

Every branch bottoms out at a primitive. Unwind, and every ? that got emitted on the way down is still there:

type PatchUser = DeepPartial<User>;
// {
//   id?: string;
//   profile?: {
//     name?: string;
//     address?: {
//       city?: string;
//       zip?: string;
//     };
//   };
// }

And the payload from the top now compiles:

const patch: PatchUser = {
  profile: { name: 'Ada' },   // ✅
};

const deep: PatchUser = {
  profile: { address: { city: 'Bucharest' } },  // ✅ two levels down, one field
};

Add a fifth level of nesting to User tomorrow and you change nothing. The type follows the shape wherever it goes — that's what you bought.

The gotcha nobody warns you about

T extends object is broader than "plain object." It's true for arrays, functions, Date, Map, class instances — everything that isn't a primitive. So DeepPartial marches straight into them:

type T1 = DeepPartial<Date>;
// { toISOString?: () => string; getTime?: () => number; ... }
// A Date with optional methods. It is no longer a Date.

type T2 = DeepPartial<string[]>;
// { length?: number; push?: ...; map?: ... }
// It is no longer an array.

The floor is too low. Raise it — anything you don't want to descend into becomes part of the base case:

type Primitive = string | number | boolean | bigint | symbol | null | undefined;

type DeepPartial<T> = T extends Primitive | Date | RegExp
  ? T                                          // atomic — return as-is
  : T extends ReadonlyArray<infer U>
    ? ReadonlyArray<DeepPartial<U>>            // array — recurse on the element type
    : T extends object
      ? { [K in keyof T]?: DeepPartial<T[K]> } // plain object — recurse on values
      : T;

Order matters: conditionals resolve top-down, first match wins, so the escape hatches sit above the general object case.

Whether arrays should be descended into at all is a judgment call. For a PATCH payload, arrays are usually atomic — you replace the whole list or you omit it, you don't partially patch element 3. In that case:

: T extends ReadonlyArray<unknown>
  ? T   // leave it alone

Same skeleton, different transformation

Once the pattern is in your hands — conditional for the floor, mapped type for the descent, T[K] for the recursion — you get the whole family by changing one token:

type DeepReadonly<T> = T extends object
  ? { readonly [K in keyof T]: DeepReadonly<T[K]> }  // add readonly
  : T;

type DeepRequired<T> = T extends object
  ? { [K in keyof T]-?: DeepRequired<T[K]> }         // -? strips optionality
  : T;

type DeepNullable<T> = T extends object
  ? { [K in keyof T]: DeepNullable<T[K]> | null }    // union in a null
  : T;

What you need to know before you start applying this everywhere:

• Recursion is depth-limited (~50 levels) — deep recursive conditional types can hit "Type instantiation is excessively deep and possibly infinite."

• Heavy recursive types are a real compile-time cost. If your IDE gets sluggish, look here first.

12. any vs unknown vs never

Think of a type as a set of values:

any is BAD

const data: any = await res.json();

data.user.profile.nmae;  // no error (typo)
data.foo();              // no error
data + 1;                // no error

const email: string = data.user;  // no error — `any` just became `string`

That last line is the damage. any doesn't stay put. It flows through assignments and return values into modules that never mentioned any. One any at an API boundary can hollow out a whole feature.

unknown is the honest version

const data: unknown = await res.json();

data.user;  // ❌ 'data' is of type 'unknown'

Anything can go into unknown. Nothing comes out until you prove what it is — because as far as TypeScript knows, data could be null, or 42, and neither has a .user.

Narrowing is that proof. You write a runtime check; TypeScript watches it and gives you a better type inside the branch:

const value: unknown = JSON.parse(raw);

if (typeof value === "string") {
  value.toUpperCase();  // ✅ inside here, value is `string`
}

value.toUpperCase();    // ❌ outside, it's `unknown` again

The check is what buys the type. No check, no access.

For object shapes, write the check once and label it with is:

type User = { id: string; email: string };

const isUser = (v: unknown): v is User =>
  typeof v === "object" && v !== null &&
  typeof (v as User).id === "string" &&
  typeof (v as User).email === "string";

const data: unknown = await res.json();

if (!isUser(data)) throw new Error("Bad payload");

data.email;  // ✅ data is `User` from here on

any deletes the check. unknown demands it. Same runtime risk either way — opposite compile-time posture. That's why unknown belongs at every boundary: fetch, JSON.parse, localStorage, postMessage, catch (e: unknown).

unknown in, guard, User out. (Which is exactly what Zod automates — section 15.)

never is a proof, not a mistake

never is where a value is impossible:

const fail = (msg: string): never => { throw new Error(msg); };  // never returns
type Impossible = string & number;                               // empty set

Its killer app is exhaustiveness. Because never is the empty set, nothing is assignable to it — so a function that demands a never argument only compiles when you've genuinely run out of cases:

type Result =
  | { status: "loading" }
  | { status: "success"; data: Data }
  | { status: "error"; error: string };

const assertNever = (value: never): never => {
  throw new Error(`Unhandled: ${JSON.stringify(value)}`);
};

const render = (result: Result) => {
  switch (result.status) {
    case "loading": return spinner();
    case "success": return view(result.data);
    case "error":   return alert(result.error);
    default:        return assertNever(result);  // ✅ compiles
  }
};

Each case narrows one variant away. By default, all three are gone and result is never — which is the only thing assertNever accepts.

Now add { status: "idle" } to Result. In default, result is { status: "idle" }, not never, and the build breaks — right at the switch you forgot to update.

That's the whole point: a compile error at every place you need to think.

13. Why enums are bad

enum Status {
  Draft = 'draft',
  Published = 'published',
}

They don't disappear

Every other type construct is erased at compile time. Enums are not. They emit runtime JavaScript:

var Status;
(function (Status) {
    Status["Draft"] = "draft";
    Status["Published"] = "published";
})(Status || (Status = {}));

That's an IIFE that mutates an object. Bundlers see a function call with side effects on a shared binding and, conservatively, keep it — even if you imported the enum only to reference one member in a type position. You now ship runtime code for something you used as a type. Multiply by every enum in a shared types.ts and it adds up.

Numeric enums are worse: they emit a reverse mapping too (Status[0] === "Draft"), so the object is twice the size.

They're nominal, in a structural language

TypeScript is structural — a thing that looks like a User is a User. Enums break that rule:

const publish = (status: Status) => { /* ... */ };

publish('draft');        // ❌ Argument of type '"draft"' is not assignable to 'Status'.
publish(Status.Draft);   // ✅ only this works

Your enum is now viral: every caller, every test fixture, every mock has to import it. You can't just write the string you can plainly see in the database.

const enum is not the fix

const enum inlines and emits nothing — but it requires whole-program type information, so it breaks under isolatedModules, which means it breaks under Babel, esbuild, SWC, and anything Vite-adjacent. TS 5.0 added preserveConstEnums/erasable-syntax pressure precisely because this feature doesn't fit modern build pipelines. And --erasableSyntaxOnly (TS 5.8, for Node's native type-stripping) bans enums outright.

Just use as const + the section-4 trick

export const StatusEnum = {
  Draft: 'draft',
  Published: 'published',
} as const;

export type Status = (typeof StatusEnum)[keyof typeof StatusEnum]; // "draft" | "published"

You get:

publish(StatusEnum.Draft); // ✅ autocomplete, single source of truth
publish('draft');      // ✅ also fine — it's just a string

• Zero enum machinery. A plain object — tree-shakeable, inlinable, JSON-serializable.

• Structural, so raw strings from your DB/API just work.

• Same DX: Status. still autocompletes.

• The type and the value share a name, so consumers import one symbol.

If you don't even need the runtime object, a bare union is enough: type Status = 'draft' | 'published';

14. Zod, and z.infer

Everything above happens at compile time. At runtime, an API response is a lie until you check it.

const res = await fetch('/api/user');
const user: User = await res.json(); // ← this annotation is a wish, not a check

res.json() returns any. You annotated it User and TypeScript relaxed. If the backend renamed email to emailAddress last night, you find out in production.

Zod flips the direction: define the schema once, derive the type from it.

import { z } from 'zod';

const UserSchema = z.object({
  id: z.string().uuid(),
  email: z.string().email(),
  plan: z.enum(['free', 'pro']),
  createdAt: z.coerce.date(),
  posts: z.array(z.object({ id: z.string(), title: z.string() })).default([]),
});

type User = z.infer<typeof UserSchema>;
// {
//   id: string;
//   email: string;
//   plan: "free" | "pro";
//   createdAt: Date;
//   posts: { id: string; title: string }[];
// }

Look at what z.infer<typeof UserSchema> is doing — it's exactly section 4. UserSchema is a value. typeof UserSchema walks it into the type space. z.infer is a conditional type that reaches inside and pulls out the shape it describes. Same door, same move.

Now the type can't drift from the validation, because the validation is the source of truth:

const fetchUser = async (id: string): Promise<User> => {
  const res = await fetch(`/api/users/${id}`);
  return UserSchema.parse(await res.json()); // throws on the boundary, not 3 layers deep
};

parse throws. safeParse doesn't — it returns a discriminated union, which you already know how to narrow:

const result = UserSchema.safeParse(payload);

if (!result.success) {
  return { error: result.error.flatten() }; // ✅ .error only exists here
}

result.data; // ✅ fully typed User, guaranteed to match at runtime

One more thing worth knowing: when a schema transforms (.coerce, .default, .transform), input and output types differ.

type UserInput  = z.input<typeof UserSchema>;  // createdAt: string | Date, posts optional
type UserOutput = z.output<typeof UserSchema>; // createdAt: Date, posts required
// z.infer === z.output

Use it at every boundary where data enters your program: HTTP responses, form submissions, process.env, webhook payloads, localStorage. unknown in → parse → typed value out.

15. Some "nice to know"s

Generics are just parameters for types

If a function's return type depends on its input type, that's a generic:

const first = <T,>(items: T[]): T | undefined => items[0];

first([1, 2, 3]);       // number | undefined
first(['a', 'b']);      // string | undefined

Constrain them with extends — read it as "T must be at least this":

const getId = <T extends { id: string }>(entity: T) => entity.id;

keyof + generics = type-safe property access

const prop = <T, K extends keyof T>(obj: T, key: K): T[K] => obj[key];

prop(user, 'email'); // string
prop(user, 'emial'); // ❌ Argument of type '"emial"' is not assignable to 'keyof User'

K extends keyof T constrains the key to actually exist; T[K] returns whatever that specific key's type is. The compiler tracks the relationship between two arguments.

Conditional types + infer

type Unwrap<T> = T extends Promise<infer U> ? U : T;

type A = Unwrap<Promise<User>>; // User
type B = Unwrap<string>;        // string

T extends X ? Y : Z is a ternary in the type space. infer U is "pattern-match here and give the captured piece a name." That's Awaited, ReturnType, and z.infer — all the same machinery.

Template literal types

type Route = `/${string}`;
type EventName = `on${Capitalize<'click' | 'focus'>}`; // "onClick" | "onFocus"

Strings you can compute with. Combine with key remapping for things like generating getters:

type Getters<T> = {
  [K in keyof T & string as `get${Capitalize<K>}`]: () => T[K];
};
// Getters<{ id: string }> → { getId: () => string }

Branded types (nominal typing, when you actually want it)

Let's say userId and postId are both string, and one day you pass the wrong one. TypeScript can't help — structurally, they're identical. So make them different:

type Brand<T, B extends string> = T & { readonly __brand: B };

type UserId = Brand<string, 'UserId'>;
type PostId = Brand<string, 'PostId'>;

const findUser = (id: UserId) => { /* ... */ };

const postId = 'abc' as PostId;
findUser(postId); // ❌ 'PostId' is not assignable to 'UserId'

The __brand property doesn't exist at runtime — it's a phantom, purely to make the two types structurally different. Zero cost, real safety at boundaries where IDs get swapped.

! (non-null assertion) is any's little brother

const user = users.find((u) => u.id === id)!; // "trust me"

You've silenced the compiler, and you get a TypeError: Cannot read properties of undefined instead of a helpful error. Handle the undefined, or throw explicitly so the failure has a message. Reserve ! for cases you can prove locally in the next line.

Prettify — the debug helper

Intersections and mapped types show up in tooltips as unreadable soup. This forces the compiler to flatten them:

type Prettify<T> = { [K in keyof T]: T[K] } & {};

type Ugly = Omit<User, 'id'> & { role: string };  // hover: a mess
type Nice = Prettify<Ugly>;                       // hover: the actual flat object

Does nothing at runtime, changes nothing semantically, saves your eyes.

Turn on strict

If strict: true isn't in your tsconfig.json, most of this document is decorative. Also worth adding:

{
  "compilerOptions": {
    "strict": true,
    "noUncheckedIndexedAccess": true, // arr[0] is T | undefined — because it is
    "exactOptionalPropertyTypes": true, // `{ a?: string }` ≠ `{ a: undefined }`
    "noImplicitOverride": true,
    "isolatedModules": true
  }
}

noUncheckedIndexedAccess is the one people fight and then thank you for.

The whole thing, compressed

1. Two universes. typeof is the door from value space into type space; keyof and indexed access let you walk around once you're through.

2. Types are sets. & shrinks the set (bigger objects), | grows it (less you can do). unknown is everything, never is nothing, any is a surrender.

3. Derive, never duplicate. Pick, Omit, typeof, z.infer — one source of truth, everything else follows from it.

4. Make illegal states unrepresentable. Discriminated unions and optional-never beat a bag of optional booleans, every time.

5. Validate at the boundary, trust inside it. unknown in, Zod parse, typed value out.

6. Prefer things that vanish. If a construct emits runtime JavaScript for a compile-time concern (looking at you, enum), there's usually a plain object that does it better.

WHERE — which rows to keep

NULL — the "no value" trap

Sorting & paging

INSERT / UPDATE / DELETE

JOINs — combining two tables (multiple readings)

Example: users and their documents (documents.owner_id points at users.id; a user can own many documents, or none).

Every join answers one question: a row found no match — does it survive? The join word names who survives. A survivor with no partner gets NULL in the other table's columns.

Memory hook: ON decides who matches · the join word decides who survives without a match · NULL fills the missing side.

Reminder: OUTER is an optional word — LEFT JOIN = LEFT OUTER JOIN.

GROUP BY & aggregates (multiple examples)

GROUP BY collapses rows that share a value into one row per group; the aggregate describes each group.

CREATE TABLE — column rules (constraints)

Foreign keys — ON DELETE (multiple readings)

A foreign key lives on the row that points. Example: a team_members row points at a teams row through team_id. ON DELETE decides what happens to me, the pointing row, when the row I point to is deleted.

Indexes

Transactions

And now for the meat of the article. If you want a more in depth explanation, below is a blog post where I get into the details of it.

How SQL works

SQL looks like a wall of shouting keywords.

SELECT ... FROM ... WHERE ... JOIN ... ON ... GROUP BY ... FOREIGN KEY ... CASCADE

The usual tutorial hands you that wall and starts defining the bricks. Which is not an explanation. It's a glossary.

So let me give you the one idea the whole thing rests on, and then everything above turns into plain sentences.

Here it is:

A database is a pile of grids.
SQL is how you talk to the grids.

That's it. That is the whole mental model. A grid is a table — rows and columns, like a spreadsheet with rules. SQL is the language for making grids, putting rows in them, asking questions about them, and connecting one grid to another.

Every keyword below is just a word in that conversation.

I'll use one running example the whole way: a small app with users, teams, and the memberships that connect them. Three people — Mara, Sam, Theo. One team — Design crew. Watch them move through every idea.

A table is a grid

Picture the users table as a literal grid.

id       | email             | name  | email_verified
---------+-------------------+-------+---------------
u_mara   | mara@example.com  | Mara  | true
u_sam    | sam@example.com   | Sam   | false
u_theo   | theo@example.com  | Theo  | true

Columns are the headings: id, email, name, email_verified.

Rows are the entries: one per person.

A cell is where a row meets a column — Mara's email is one cell.

(Real ids are long random strings like cc512230-de30-44c7-a197-fd6a88cb3f3c. I'm writing u_mara so the examples stay readable. More on why ids look like that later.)

That's the whole shape of a database. Grids of rows. Everything else is talking to them.

Making a grid: CREATE TABLE

Before a grid can hold rows, you declare its columns. That's CREATE TABLE.

CREATE TABLE users (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  email          text NOT NULL,
  name           text,
  email_verified boolean NOT NULL DEFAULT false,
  created_at     timestamp with time zone NOT NULL DEFAULT now()
);

Read it as a sentence: "make a grid called users, with these columns."

Each line inside the parentheses is one column, and it has three parts:

email          text          NOT NULL
^ the name     ^ the type    ^ the rules

The name is the heading. The type is what kind of value the cell may hold. The rules are promises the database will enforce.

Let me name the types you'll actually use:

• text — a string of any length. 'mara@example.com'.

• boolean — true or false.

• integer — a whole number. 42.

• timestamp with time zone — a moment in time that knows its timezone.

• uuid — a long random identifier.

And the rules:

• NOT NULL — this cell can never be empty. (NULL is SQL's word for "no value at all." NOT NULL forbids it. More on NULL soon — it's sneakier than it looks.)

• DEFAULT <value> — if nobody supplies this cell, fill it with this. email_verified defaults to false; created_at defaults to now() (the current time, stamped automatically).

• PRIMARY KEY — this column is the row's unique name-tag. No two rows may share one, and it's the fast way to find a single row.

One line that trips people up:

id uuid PRIMARY KEY DEFAULT gen_random_uuid()

uuid is the type — what the cell holds. gen_random_uuid() is a function — it produces a fresh random id each time it runs. So the column holds a uuid, and its default is "call this function to make one." That's why we never invent ids by hand — insert a row, and the database mints its id. Random uuids almost never collide, so two servers can both create rows and never clash. That's the whole reason to prefer them over a counter like 1, 2, 3.

Putting a row in: INSERT

The grid exists but it's empty. INSERT adds a row.

INSERT INTO users (email, name) VALUES ('mara@example.com', 'Mara');

"Into the users grid, in the email and name columns, put these values."

Notice what I didn't write: no id, no email_verified, no created_at. I left them out on purpose, and the defaults filled them in — a fresh uuid, false, and the current time. The row that lands is complete:

id       | email             | name  | email_verified | created_at
---------+-------------------+-------+----------------+---------------------
u_mara   | mara@example.com  | Mara  | false          | 2026-07-08 09:14:...

You can insert several rows at once:

INSERT INTO users (email, name) VALUES
  ('sam@example.com',  'Sam'),
  ('theo@example.com', 'Theo');

Now the grid has three rows. Let's ask for them back.

Asking for it back: SELECT ... FROM ... WHERE

This is the sentence you'll write more than any other. It has three parts, and each answers one question.

SELECT email, name          -- which COLUMNS do I want?
FROM users                  -- from which GRID?
WHERE email_verified = true -- which ROWS?

Read it top to bottom:

SELECT   →  which columns
FROM     →  which grid
WHERE    →  which rows

SELECT * means "every column" (the * is "all"). SELECT email, name means just those two.

FROM users picks the grid.

WHERE email_verified = true is the filter. Only rows where that's true come back.

So that query returns Mara and Theo (verified), not Sam (not verified):

email             | name
------------------+------
mara@example.com  | Mara
theo@example.com  | Theo

Change the question by changing the WHERE. Want just Sam?

SELECT * FROM users WHERE email = 'sam@example.com';

The database walks the grid, keeps the rows the WHERE approves of, and hands back the columns the SELECT asked for. That loop — filter rows, pick columns — is 80% of SQL.

WHERE is where the thinking happens

The WHERE clause is a yes/no test run against every row. The row stays if the test is true.

You have the comparisons you'd expect:

WHERE created_at > '2026-01-01'          -- after a date
WHERE name = 'Mara'                      -- exactly equal
WHERE name != 'Mara'                     -- not equal

And you can combine tests with AND and OR:

SELECT * FROM users
WHERE email_verified = true
  AND created_at > '2026-01-01';

"Verified and created this year." A row must pass both.

OR means either is enough:

WHERE name = 'Mara' OR name = 'Sam';

Now, the sneaky one. NULL — the "no value at all" from earlier — does not behave like a value.

Here's the trap:

-- ✗ this returns NOTHING, even for rows where name really is empty
SELECT * FROM users WHERE name = NULL;

You'd expect it to find the rows with no name. It finds none.

Why? Because NULL means unknown, and "is this unknown thing equal to unknown?" isn't true — it's itself unknown. So the row fails the test. = NULL can never be true for anyone.

The fix is a special operator that asks the question directly:

-- ✓ the right way to ask "is this cell empty?"
SELECT * FROM users WHERE name IS NULL;

IS NULL and IS NOT NULL are how you test for emptiness. Reach for = and you'll silently get nothing back. This bites everyone once.

Sorting and limiting: ORDER BY, LIMIT

Rows come back in no guaranteed order unless you ask for one. ORDER BY sorts them.

SELECT name, created_at FROM users
ORDER BY created_at DESC;

DESC = descending, newest first. ASC = ascending, oldest first (and it's the default).

LIMIT caps how many rows come back:

SELECT name FROM users
ORDER BY created_at DESC
LIMIT 1;

"The single most recently created user." Sort newest-first, then take one.

That pairing — ORDER BY then LIMIT — is how you get "the latest," "the top 10," "the most recent 5."

Changing a row: UPDATE (and the WHERE you must never forget)

UPDATE changes cells in rows that already exist.

UPDATE users
SET email_verified = true
WHERE email = 'sam@example.com';

"In the users grid, set email_verified to true, for the row where email is Sam's."

The SET says what to change. The WHERE says which rows — and it is the most important word in the statement.

Here's the mistake that has ruined real production databases:

-- ✗ NO WHERE — this verifies EVERY user in the table
UPDATE users SET email_verified = true;

No WHERE means "every row." You meant to update Sam. You just marked all three million users as verified, in one keystroke, with no undo.

The rule burns itself into you fast: an UPDATE without a WHERE hits everything. Write the WHERE first.

Removing a row: DELETE

DELETE throws rows away. Same lesson, sharper.

DELETE FROM users WHERE email = 'theo@example.com';

"Remove the row where email is Theo's." Theo is gone.

And the same landmine:

-- ✗ NO WHERE — this empties the entire table
DELETE FROM users;

No WHERE, no survivors. Every row, gone.

So for both UPDATE and DELETE, the WHERE is not optional decoration. It's the difference between "change one thing" and "change everything." A good habit: write the WHERE before you write the SET or the DELETE, so the target exists before the action does.

Rows that point at other rows: the foreign key

So far, one grid at a time. But real data connects.

A membership connects a user to a team. So there's a third grid, team_members, that sits between users and teams. Each row says "this user is in this team, with this role":

CREATE TABLE team_members (
  id      uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  team_id uuid NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
  user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  role    text NOT NULL DEFAULT 'member'
);

Look at team_id. On its own it's just a uuid sitting in a cell. Nothing yet forces it to match a real team. The REFERENCES teams(id) is what forces it. That is a foreign key.

Read it in plain English:

the team_members.team_id column
must always REFERENCE a real teams.id
— you can't have a membership pointing at a team that doesn't exist.

That's the whole idea of a foreign key: a column whose value must exist as a real row in another grid. Try to insert a membership for a team id that isn't there, and the database refuses. It's the rule that keeps the grids honest with each other — no memberships floating in space, pointing at nothing.

A foreign key is what turns a pile of separate grids into a connected web.

When the row I point at is deleted, what happens to me?

A foreign key raises a question the moment you try to delete something.

Say you delete the Design crew team. What should happen to the membership rows that point at it? They can't keep pointing at a team that's gone — that's the exact thing the foreign key forbids.

So every foreign key must answer one question:

When the row I point at is deleted, what happens to me?

You answer it with ON DELETE. There are two answers you'll reach for.

ON DELETE CASCADE — "delete me too."

team_id uuid NOT NULL REFERENCES teams(id) ON DELETE CASCADE

Delete a team, and all its membership rows vanish with it. Delete a user, and all their membership rows vanish. No orphaned memberships pointing at a deleted team or a deleted user. Clean. The deletion cascades — it flows down the chain from the parent to the rows that depend on it.

ON DELETE SET NULL — "don't delete me, just blank the pointer."

Here's a different case. A team remembers who created it:

CREATE TABLE teams (
  id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  name          text NOT NULL,
  created_by_id uuid REFERENCES users(id) ON DELETE SET NULL
);

Now delete the user who created Design crew.

You do not want the team to vanish — other people are still in it. So ON DELETE SET NULL says: keep the team, just set created_by_id to NULL. The pointer goes blank; the row lives on. The team outlives its creator.

That single word — CASCADE versus SET NULL — is the entire difference between "the child dies with the parent" and "the child survives, forgetting the parent." It's worth pausing on, because getting it wrong means either orphaned rows or accidental mass-deletion.

parent row deleted
↓
CASCADE   → the rows pointing at it are deleted too
SET NULL  → the rows survive, their pointer set to NULL

(You may notice created_by_id has no NOT NULL. It can't — SET NULL needs to be allowed to write a blank there. The two rules go together.)

Two grids, one question: JOIN

Now the payoff. Data lives in separate grids, but questions cross them.

"Which teams is Sam in?" The answer needs team_members (who's in what) and teams (the team's name). One grid can't answer it alone.

A JOIN stitches two grids together on a matching column.

SELECT teams.name, team_members.role
FROM team_members
JOIN teams ON team_members.team_id = teams.id
WHERE team_members.user_id = 'u_sam';

Read the JOIN ... ON as: "glue each team_members row to the teams row where their ids match." The ON is the matching rule — team_members.team_id = teams.id.

For each membership row, the database finds the team it points at and lays the two rows side by side, into one wider row:

teams.name    | team_members.role
--------------+------------------
Design crew   | member

Sam's membership pointed at Design crew; the join pulled in that team's name. Two grids, one combined answer.

You qualify columns with their grid — teams.name, team_members.role — because once two grids are joined, a bare name could be ambiguous. Say which grid you mean.

The many-to-many, and why it needs a table in the middle

Why does team_members exist at all? Why not just... put the members on the team?

Because a user can be in many teams, and a team has many members. That's a many-to-many relationship, and neither grid can hold it alone. A teams row can't list an unbounded number of members in one cell. A users row can't list an unbounded number of teams.

So the relationship gets its own grid. Each row is one "this user is in this team" fact. That middle grid is a join table, and you read it both directions:

"Which teams is Sam in?"    → team_members rows where user_id = Sam
"Who is in Design crew?"    → team_members rows where team_id = Design crew

One grid, both questions. This pattern — two things that relate many-to-many, joined by a table in the middle — is everywhere once you see it. Students and classes. Orders and products. Users and teams.

INNER JOIN vs LEFT JOIN — the difference is who gets dropped

There's a fork in JOIN that matters, and the names hide it. Let me show the bug first.

You want a roster: every user, and their role if they're on a team.

-- looks right...
SELECT users.name, team_members.role
FROM users
JOIN team_members ON team_members.user_id = users.id;

Run it and Theo is missing.

name  | role
------+--------
Mara  | owner
Sam   | member

Theo is in no team. A plain JOIN — an inner join — only keeps rows where both sides match. Theo has no team_members row, so he has nothing to match, so he falls out entirely. The join silently dropped him.

Sometimes that's what you want. Here it isn't — you wanted every user.

The fix is LEFT JOIN. It keeps every row from the left grid (users), matched or not. Where there's no match, it fills the right side with NULL:

SELECT users.name, team_members.role
FROM users
LEFT JOIN team_members ON team_members.user_id = users.id;

name  | role
------+--------
Mara  | owner
Sam   | member
Theo  | NULL     ← kept, with a blank role

Now Theo is there, his role NULL because he has none.

That's the whole distinction:

INNER JOIN  → keep only rows that matched on both sides
LEFT JOIN   → keep ALL left rows; NULL-fill the right where nothing matched

"Where did that row go?" in a report is, nine times out of ten, an inner join that should have been a left join.

The other two joins: RIGHT and FULL

INNER and LEFT are the two you'll write almost every day. But there are four in total — so let me close the set, and clear up a word while I'm at it.

"OUTER" is noise. You'll see LEFT OUTER JOIN, RIGHT OUTER JOIN, FULL OUTER JOIN. The OUTER adds nothing — LEFT JOIN and LEFT OUTER JOIN are the exact same thing. So "right join" and "outer join" aren't two separate answers; OUTER is just the formal middle name of joins you already have. Four joins, one word you can ignore.

Picture two overlapping circles — users on the left, team_members on the right. The overlap is the rows that match on the ON condition. Each join keeps a different region:

INNER JOIN  → just the overlap             (only rows matched on both sides)
LEFT JOIN   → whole LEFT circle + overlap   (all users; NULL where no membership)
RIGHT JOIN  → whole RIGHT circle + overlap  (all memberships; NULL where no user)
FULL JOIN   → both circles, whole           (everything; NULL wherever either side is missing)

RIGHT JOIN is the mirror of LEFT: keep every row from the right grid, NULL-fill the left. FULL JOIN keeps everything from both — a user with no membership comes back with a NULL role, and a membership with no matching user comes back with a NULL name.

Now the honest part, and it's the lesson worth keeping. On this pair of grids, RIGHT and FULL are secretly the same as joins you've already seen — and the foreign key is why.

Remember: team_members.user_id REFERENCES users.id. A membership can never point at a user who doesn't exist — the database forbids it. So the part of the right circle sticking out past the overlap — memberships with no user — is always empty.

Watch what that does. RIGHT JOIN keeps all memberships:

SELECT users.name, team_members.role
FROM users
RIGHT JOIN team_members ON team_members.user_id = users.id;

But every membership already has a real user, so nothing gets NULL-filled — you get exactly the matched rows. RIGHT JOIN here collapses to INNER JOIN. Same two rows, Mara and Sam:

name | role
-----+--------
Sam  | member
Mara | owner

And FULL JOIN here collapses to LEFT JOIN. The only unmatched rows that exist are users without a membership (Theo, and a nameless account), so FULL adds nothing beyond what LEFT already kept — the same four rows, Theo and the nameless one with a NULL role.

That's the real reason I reached for LEFT and skipped RIGHT earlier. On this schema the gap only runs one way:

a user can have no team          → LEFT JOIN surfaces it
a membership can't have no user  → the foreign key already made that impossible

LEFT shows the gap that can actually happen. RIGHT would go looking for a gap the foreign key has ruled out.

RIGHT and FULL earn their keep when both sides can have unmatched rows — two independent lists you're reconciling, wanting to see what each has that the other is missing. With a foreign key in play, that's usually not your situation, so you'll live in INNER and LEFT.

And a practical note: almost nobody writes RIGHT JOIN anyway. A RIGHT JOIN B is just B LEFT JOIN A with the grids swapped, and since we read left-to-right, people flip it so the "keep them all" grid comes first. So the family quietly narrows back down to the two you started with.

(There's also CROSS JOIN — every left row paired with every right row, no ON at all. It builds combinations rather than matching rows, a different job entirely; you'll rarely reach for it.)

The one model that covers all of them:

a JOIN matches rows across two grids;
the word in front only decides which UNMATCHED rows survive —
left, right, both, or neither.

Counting: GROUP BY

One more kind of question: not "which rows" but "how many," per something.

"How many members does each team have?"

You don't want the rows themselves — you want a count per team. That's GROUP BY. It collapses rows that share a value into one group, and lets you count (or sum, or average) each group.

SELECT team_id, COUNT(*) AS member_count
FROM team_members
GROUP BY team_id;

GROUP BY team_id gathers all the rows with the same team_id into one bucket. COUNT(*) counts the rows in each bucket. AS member_count just names the output column.

team_id       | member_count
--------------+-------------
t_design_crew | 2

COUNT(*) is the common one, but the same shape works with SUM(...), AVG(...), MAX(...). The mental model: GROUP BY turns many rows into one row per group, and the count/sum describes each group.

Making it fast: the index

Everything so far is about correctness. This one is about speed.

Ask "which teams is Sam in?" and, by default, the database reads every single row in team_members and checks each one's user_id. Three rows, fine. Three million rows, slow — every time.

An index fixes that. It's a lookup shortcut the database maintains on the side:

CREATE INDEX team_members_user_idx ON team_members (user_id);

Now "find Sam's memberships" jumps straight to his rows instead of scanning the whole grid — the same way the index at the back of a book beats flipping every page.

An index costs a little: it takes space, and it must be updated on every write. So you don't index every column — you index the ones you look things up by.

There's a second kind that does double duty:

CREATE UNIQUE INDEX team_members_team_user ON team_members (team_id, user_id);

A UNIQUE index is a shortcut and a rule: the pair (team_id, user_id) must be unique across the whole grid. Meaning the same user can't be in the same team twice. If two "add Sam to Design crew" requests arrive at the same instant, the database lets the first win and rejects the second — the uniqueness is enforced by the database, not by fragile "check, then insert" code that two requests can both slip through.

All-or-nothing: the transaction

Last idea, and it's the one that separates toy SQL from real SQL.

Creating a team is really two writes: insert the team, then insert the membership that makes the creator its owner. Both have to happen, or the team is born broken — a team with no owner that nobody can manage.

Here's the danger, written the naive way:

INSERT INTO teams (name, created_by_id) VALUES ('Design crew', 'u_mara');
-- ... what if the process crashes RIGHT HERE? ...
INSERT INTO team_members (team_id, user_id, role) VALUES ('t_design_crew', 'u_mara', 'owner');

If the process dies between the two lines, the first row committed and the second never ran. Now there's a team with no members. A ghost.

A transaction fixes this. You wrap the statements so the database treats them as one indivisible unit — all of them commit, or none of them do.

BEGIN;
  INSERT INTO teams (name, created_by_id) VALUES ('Design crew', 'u_mara');
  INSERT INTO team_members (team_id, user_id, role) VALUES ('t_design_crew', 'u_mara', 'owner');
COMMIT;

BEGIN opens the transaction. COMMIT seals it — both rows land at the same instant. If anything between them fails, you ROLLBACK (or the database does it for you), and it's as if neither statement ever ran. The team can never exist without its owner.

BEGIN
↓
insert the team
insert the owner membership
↓
COMMIT  → both land together
   or
ROLLBACK → neither happened, no ghost

That property — several changes that must be true together — is what transactions are for. Any time "do X" really means "do X and Y, and half of it would be a mess," wrap them.

What SQL is not, and when to reach for something else

Two honest notes to end on.

SQL is not a general programming language you write loops in. It's declarative: you describe the rows you want, and the database figures out how to get them. You don't tell it "scan this grid, check each row" — you say WHERE email_verified = true, and it decides whether to use an index or scan. That flip — describe the result, don't script the steps — is the mental adjustment that makes SQL click.

And it's not always the right tool. If your data is a bag of loosely-shaped documents with no relationships — logs, a cache, a blob of JSON you always read whole — a relational database's grids and joins are overhead you don't need. SQL earns its keep exactly when your data has structure and has relationships: users who belong to teams that own documents, where a question crosses all three and the answer must stay consistent. That's the case these grids were built for.

The whole thing in three beats:

A table is a grid, and SELECT ... WHERE is how you ask it questions.
A foreign key connects two grids, and ON DELETE decides what a deletion drags with it.
A transaction makes several writes land together, so your grids are never caught half-changed.

This is the log. The what.

Every command, in order, with the traps I hit and how I got out of them.

Same site as before: produsebaby.ro, a Next.js content site. Same request the whole way through:

A parent opens their phone, types produsebaby.ro/carucioare/cele-mai-bune-carucioare, and wants the strollers article.

By the end, that request travels through a stack I built by hand — and a git push rebuilds the whole thing on its own.

One note on secrets: everywhere you see <VPS_IP> or <user>, that's a placeholder. The real server IP, the GitHub username, the private keys — those never go in a blog post, and they never go in git.

The order matters. Each phase leans on the one before. Don't skip ahead.

0. Prepare the code (laptop)
1. Rent the server (Hetzner)
2. First deploy — build on the VPS
3. Go live — Caddy + DNS
4. Lock it down — firewall
5. Automate — GitHub Actions
6. Put Cloudflare in front

Phase 0 — Prepare the code, on the laptop

Three changes, one local test. Nothing touches a server yet.

1. Tell Next to build a self-contained bundle. In next.config.ts:

output: 'standalone',

This makes next build emit .next/standalone — the app plus only the node_modules it actually traced. That's what the container will run.

2. Write the Dockerfile. Multi-stage: a heavy builder, a lean runner.

# ---- builder ----
FROM oven/bun:1 AS builder
WORKDIR /app
COPY . .
RUN bun install --frozen-lockfile
RUN bun run build

# ---- runner ----
FROM oven/bun:1 AS runner
WORKDIR /app
ENV NODE_ENV=production
ENV PORT=3000
ENV HOSTNAME=0.0.0.0
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public
EXPOSE 3000
CMD ["bun", "server.js"]

Trap: COPY . . comes before bun install. The obvious order is to copy package.json first and install, for better caching. It breaks here. My package.json has a prepare script that runs panda codegen, and codegen needs the config and source present. Install before the source exists → install fails. So copy everything first.

3. Write .dockerignore. Keep the image lean and secret-free:

node_modules
.next
.git
.env*
*.log
explanatory-docs
public/pagefind

.git and .env* are the important lines — no history, no secrets in the image.

4. Test locally. This is the whole point of Docker:

docker build -t produsebaby:latest .
docker run -d -p 3000:3000 --name test produsebaby:latest
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3000/   # 200

If it works in a container on the laptop,
it works in the same container on the server.

That's the contract. Delete the local container once you've seen 200 — it was just proof.

Phase 1 — Rent the server

I bought the cheapest tier that builds comfortably: 2 vCPU / 4 GB RAM / ~40 GB SSD, Ubuntu LTS, a datacenter in Germany. A few euros a month, fixed.

The SSH key. One rule, and it's the whole security model:

private key  → stays on the laptop, never shared
public key   → goes on the server

Paste the public key into the provider's panel when creating the server. You get an IP back.

Connect:

ssh root@<VPS_IP>

First connection asks you to trust the host fingerprint — yes. Then you're in, no password, because the key did the work.

Update the machine:

apt update && apt upgrade -y

If the upgrade pulls a new kernel, it says so. A new kernel only loads after a reboot:

reboot

The SSH session drops (normal). Wait ~30s, reconnect, confirm the new kernel:

uname -r

Install Docker. On a brand-new Ubuntu, the distro packages are the safe bet:

apt install -y docker.io docker-compose-v2
docker run hello-world   # "Hello from Docker!" → engine works

docker.io is the engine + docker CLI. docker-compose-v2 is the docker compose plugin. That's everything the server needs.

Phase 2 — First deploy: build on the VPS

The plan for the first deploy: get the code onto the server, build the image there, run it, curl it. Prove the container serves before adding HTTPS or a domain.

1. Add swap. A 4 GB box can run out of memory mid-build and get the process Killed. A swap file is the safety net:

fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile

2. Sync the code up with rsync — copy only what changed, delete what's gone, skip the junk:

rsync -avz --delete \
  --exclude node_modules --exclude .next --exclude .git \
  --exclude '.env*' \
  ./ root@<VPS_IP>:/srv/produsebaby/

The excludes matter: node_modules and .next get rebuilt in the image, .git and .env* must never leave the laptop.

3. Build, run, test — on the server:

cd /srv/produsebaby
docker build -t produsebaby:latest .
docker run -d -p 3000:3000 --name produsebaby produsebaby:latest
curl -s -o /dev/null -w 'status: %{http_code}\n' http://localhost:3000/   # status: 200

status: 200 means the site runs on the VPS — but only on localhost:3000. No HTTPS, no domain yet. That's the next phase.

Phase 3 — Go live: Caddy + DNS

Two containers, one config file each, then flip the domain.

Internet (443, HTTPS)
↓
caddy    ← the only thing exposed to the internet
↓  app:3000 (private Docker network)
app      ← Next, hidden behind Caddy

1. docker-compose.yml describes both containers:

services:
  app:
    build: .
    restart: unless-stopped
    volumes:
      - nextjs_cache:/app/.next/cache   # ISR + optimized images survive redeploys
    expose:
      - "3000"                          # visible ONLY to other containers

  caddy:
    image: caddy:2
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"                       # the only ports open to the internet
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data                # issued TLS certs live here
      - caddy_config:/config
    depends_on:
      - app

volumes:
  nextjs_cache:
  caddy_data:
  caddy_config:

expose vs ports is the security line: app is only reachable inside the private network; only caddy faces the internet.

2. Caddyfile — four lines gets you automatic HTTPS:

produsebaby.ro {
	encode gzip zstd
	reverse_proxy app:3000
}

www.produsebaby.ro {
	redir https://produsebaby.ro{uri} permanent
}

Naming the domain is what makes Caddy request and auto-renew a Let's Encrypt certificate for it. You configure nothing else for HTTPS.

3. Bring the stack up:

docker rm -f produsebaby        # remove the manual test container
docker compose up -d --build
docker compose ps               # app + caddy both "Up"

Caddy starts but can't get a certificate yet — the domain still points at the old host. It retries in the background. Fine.

4. Flip DNS. In the DNS panel, the two records that carry web traffic move to the VPS:

produsebaby.ro   A → <VPS_IP>   (grey cloud / DNS only)
www              A → <VPS_IP>   (grey cloud / DNS only)

Trap: you can't change a record's type. Mine were CNAMEs pointing at the old host. The edit form won't let you switch CNAME → A. Delete each one, add a fresh A record.

Trap: leave it GREY (DNS only) for now. Let's Encrypt validates over port 80. A proxy in front can interfere with that first handshake. Grey now, colored later (Phase 6).

Also: do not touch the MX and TXT records. Those run email. Only the two web records change.

5. Force the certificate, verify:

docker compose restart caddy         # try the cert now that DNS points here
curl -sI https://produsebaby.ro/      # HTTP/2 200

The response headers tell the whole story:

via: 1.1 Caddy        → traffic goes through my reverse proxy
x-nextjs-cache: HIT   → ISR is serving from cache

The chain works: DNS → VPS → Caddy (HTTPS) → app:3000 → Next. The site is live on my own server.

Phase 4 — Lock it down: the firewall

Trap: don't reach for ufw. Docker writes its own iptables rules and bypasses ufw for published ports. You'd think 80/443 are filtered; Docker quietly leaves them open behind ufw's back.

Use the provider's network-level firewall instead. It sits in front of the machine, so Docker can't route around it.

Internet
↓
Cloud firewall   ← filters here, before the VM
↓  (only 22 / 80 / 443 pass)
VM (Docker, Caddy, app)

Rules — inbound only, everything else dropped:

22/TCP    SSH    source: anywhere
80/TCP    HTTP   source: anywhere
443/TCP   HTTPS  source: anywhere
ICMP      ping   source: anywhere
outbound: allow all

Trap: include port 22 before you apply, or you lock yourself out. And after applying, test SSH in a new terminal while the old one is still open — so you can undo if you got it wrong.

Verify from the outside — allowed ports answer instantly, blocked ones hang until timeout:

for p in 22 80 443 3000 8080; do nc -z -G 5 -w 5 <VPS_IP> $p && echo "$p open" || echo "$p blocked"; done

22 open   80 open   443 open        ← the doors you want
3000 blocked   8080 blocked         ← ~5s each: dropped, not refused

That delay is the point. A refused port replies instantly with "no." A dropped port stays silent until the client gives up — the machine is invisible to a scanner on that port.

Phase 5 — Automate: GitHub Actions

Until now every deploy was manual: rsync, build, restart. This phase turns git push into the whole thing.

git push
↓
GitHub runner: build image → push to registry (tagged with the commit)
↓
runner SSHes into the VPS → pull that image → restart
↓
the runner disappears

The heavy build runs on GitHub's machine, not the little VPS. The VPS just pulls a finished image.

1. Point compose at the registry image. Change the app service so it can pull a tagged image, but keep build: . as a manual fallback:

  app:
    image: ghcr.io/<user>/produsebaby:${IMAGE_TAG:-latest}
    build: .
    restart: unless-stopped
    volumes:
      - nextjs_cache:/app/.next/cache
    expose:
      - "3000"

2. Write .github/workflows/deploy.yml:

name: Deploy
on:
  push:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read
  packages: write

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    env:
      IMAGE: ghcr.io/${{ github.repository }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: |
            ${{ env.IMAGE }}:${{ github.sha }}
            ${{ env.IMAGE }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max
      - uses: appleboy/ssh-action@v1
        env:
          GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          host: ${{ secrets.VPS_HOST }}
          username: root
          key: ${{ secrets.VPS_SSH_KEY }}
          envs: GHCR_TOKEN
          script: |
            echo "$GHCR_TOKEN" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
            cd /srv/produsebaby
            export IMAGE_TAG=${{ github.sha }}
            docker compose pull
            docker compose up -d
            docker image prune -f

Three production habits are hiding in there: tag with github.sha (traceable, rollback-able — never rely on :latest), type=gha layer cache, and cancel-in-progress so a second push kills the first deploy.

Trap: permissions: packages: write. Without it, GITHUB_TOKEN is read-only and the push to the registry 403s. This one line is the most common first-run failure.

Trap: pulling a private image on the VPS. The image is private by default. The deploy step logs in on the VPS using GITHUB_TOKEN — but you must forward it with envs: (not just env:), and log in with --password-stdin.

3. Make a dedicated deploy key — not your personal key, so it's revocable on its own:

ssh-keygen -t ed25519 -f ~/.ssh/produsebaby_deploy -N "" -C "deploy"
ssh-copy-id -f -i ~/.ssh/produsebaby_deploy.pub root@<VPS_IP>

Trap: ssh-copy-id without -f. It first tries to log in "to filter out keys already installed." Since you already have SSH access via your personal key, that login succeeds, and it wrongly concludes the new key is already there — "All keys were skipped." -f skips the check and installs it.

4. Put the two secrets in GitHub (Settings → Secrets and variables → Actions):

VPS_HOST      → <VPS_IP>
VPS_SSH_KEY   → the deploy PRIVATE key (~/.ssh/produsebaby_deploy)

For the multi-line private key, copy it exactly with the clipboard:

pbcopy < ~/.ssh/produsebaby_deploy

Trap: the clipboard clobber. pbcopy puts the key on the clipboard — but any Cmd+C afterwards overwrites it, including copying a verification command out of a chat window. So pbcopy must be the last thing that touches the clipboard before you paste into GitHub. After it, go straight to the browser and paste. Nothing in between.

Trap: know which machine you're on. Read the prompt. you@laptop % is local; root@server # is the VPS. pbcopy is a macOS command and the private key only exists on the laptop — it cannot run on the server. The private key never leaves your machine; only its public half is on the VPS.

5. Sync the updated compose to the VPS. The deploy step runs docker compose pull on the server, so the server's compose file must have the new image: line:

scp docker-compose.yml root@<VPS_IP>:/srv/produsebaby/docker-compose.yml

Trap: a stale compose on the server is a silent no-op. If the server still has the old build:-only compose, pull has nothing to pull and the deploy just restarts the old container — "green," but nothing changed. (My earlier rsync had run before I edited the compose, so the server had the old one. This scp fixes it.)

6. Push, and watch:

git push

Open the repo's Actions tab. The steps run green: checkout, login, build-push (the slow one, first run has no cache), then the SSH deploy. Confirm the server is running the exact commit:

ssh root@<VPS_IP> 'cd /srv/produsebaby && docker compose images app'
# ghcr.io/<user>/produsebaby   <the pushed commit sha>

From now on: git push is the deploy. Rollback is IMAGE_TAG=<older-sha> docker compose up -d, because every image is tagged with its commit.

Phase 6 — Put Cloudflare in front

The last piece: a CDN and DDoS shield in front of the origin. Two settings, in this order.

Order matters. Set the SSL mode before turning the proxy on.

1. SSL/TLS mode → Full (strict). This makes Cloudflare talk to the origin over HTTPS and validate the origin's certificate. Caddy's Let's Encrypt cert is valid and trusted, so it passes immediately.

Flexible  → Cloudflare↔origin is plain HTTP. With Caddy's HTTP→HTTPS redirect,
            this loops forever ("too many redirects"). Never use it here.
Full (strict) → both legs encrypted, origin cert validated. Correct.

2. Turn the two A records ORANGE (Proxied). MX/TXT stay grey.

Now the request path changes:

phone → DNS → Cloudflare's edge → VPS → Caddy → app

Verify from the outside:

dig +short produsebaby.ro                                   # Cloudflare IPs, not <VPS_IP>
curl -sI https://produsebaby.ro/ | grep -i server           # server: cloudflare
curl -sI --resolve produsebaby.ro:443:<VPS_IP> https://produsebaby.ro/ | grep -i via  # via: 1.1 Caddy

DNS now returns Cloudflare's IPs, not the server's.
Public traffic shows server: cloudflare.
The origin, hit directly, still answers via Caddy — Full strict validated its cert.

The renewal fear was wrong. I worried the proxy would break Caddy's cert renewal (the TLS-based challenge does die behind a proxy). But Caddy falls back to the HTTP challenge, and Cloudflare lets /.well-known/acme-challenge/ pass through to the origin. Renewal keeps working.

What orange does not do: hide the origin. <VPS_IP> can still be hit directly, bypassing Cloudflare — I just did it with --resolve. To force all traffic through Cloudflare, restrict the firewall's 80/443 to Cloudflare's IP ranges only. That's real hardening, but it couples you to keeping the proxy on. Optional.

What I ended up with

Follow the request now — every hop is a piece I placed by hand:

the parent's phone
↓
DNS → Cloudflare's edge (CDN, DDoS, caches images)
↓
my VPS, port 443  (firewall: only 22/80/443 get in)
↓
caddy   → verifies TLS, decrypts, reverse_proxy app:3000
↓
app     → Next; ISR serves from cache or regenerates; sharp optimizes photos once
↓
volumes remember cache + certs across deploys
↓
the response goes back up → Caddy → Cloudflare (caches) → the phone

And the whole thing rebuilds itself:

git push → GitHub builds the image → pushes it tagged with the commit
         → SSHes into the VPS → pulls that exact image → restarts

The five files that made it happen:

next.config.ts              → output: 'standalone'
Dockerfile                  → the image recipe
.dockerignore               → what stays out of the image
docker-compose.yml          → app + caddy + volumes
Caddyfile                   → domain + auto HTTPS + proxy
.github/workflows/deploy.yml → git push = deploy

What it costs to be your own platform. Vercel is on call so you're not. Here, the admin is you: you watch the cert renew (Caddy does it, you verify), you keep the machine patched, you own the backups. Not hard. Yours.

The trade, said plainly:

Vercel glued six services into one button.
I pulled them apart and rebuilt each on a bare computer — build, hosting, HTTPS, CDN, images, ISR.
I gave up the convenience and took the control, the fixed cost, and the understanding.

Do the manual deploy first. Understand every hop. Then automate it — because now you know exactly what the automation is doing.

You press "deploy" on Vercel and the site works.

Domain, HTTPS, fast images, pages that update themselves.

Everything "just works."

That is the trap. It looks like Vercel is one thing.

It is not.

Vercel is about six different services, packaged so well that you never felt the seam between them. When you move to your own server, the seams become yours. You have to rebuild them, one at a time.

This post is the rebuild. It teaches DevOps the way I actually learned it — not as a list of definitions, but as one concrete job: taking a real Next.js content site, produsebaby.ro, off Vercel and onto a rented Linux box running Docker.

And it carries one example the whole way through.

A parent in Romania opens their phone.

They type produsebaby.ro/carucioare/cele-mai-bune-carucioare.

They want to read your article about the best baby strollers.

That page exists right now in the code: src/app/[category]/[slug]/page.tsx, with revalidate = 86400. It has text. It has stroller photos. It has affiliate links.

Every time I say "the request," that is the request.

You follow it with your finger. From the parent's phone all the way to the pixels on the screen. Through every single piece.

By the end you should be able to close this page and say the whole flow out loud. If you can, it worked. If you only learned some new vocabulary, read it again.

Chapter 1 — What Vercel was actually doing for you

Let's follow that request today, while you are still on Vercel.

Here is its path:

The parent's phone
↓
DNS: "where is produsebaby.ro?" → Cloudflare → Vercel
↓
Vercel's network receives the request (nearest city to the user)
↓
HTTPS: Vercel already installed the certificate, the connection is encrypted
↓
Vercel looks for the page in its cache (ISR)
↓
Page is fresh → send it directly
Page is stale → regenerate it, then send it
↓
The images on the page → optimized and served from Vercel's CDN
↓
The parent sees the article

Six things happened there. Each one done by Vercel, free for your attention. Name them, because each one becomes a chapter — and each one is really the same question in disguise: once Vercel is gone, who does this work now?

1. It ran the build. When you did git push, Vercel ran the exact command from your package.json:

panda codegen && next build && pagefind --site .next/server/app --output-path public/pagefind

On a VPS, the build becomes your job.

2. It hosted the app. The built code has to run somewhere — a live next start process, waiting for requests. Vercel started it and kept it alive. On a VPS, you start the process and you keep it alive.

3. It put HTTPS on. The padlock in the address bar. The TLS certificate, auto-renewed. You never touched this subject. Vercel made it invisible. On a VPS, you get the certificate — and the good news is this can be nearly invisible too.

4. It served static files from a CDN. The 597MB in public/ — 2844 images plus the Pagefind search index — were never served by your app. Vercel put them on its CDN: copies of the files in cities all over the world, close to users. That is why this odd hack exists in next.config.ts:

outputFileTracingExcludes: {
  '*': ['public/**'],
}

It tells Vercel: "do not stuff the 600MB of images into the function that runs the code — the images are served separately, by the CDN." Without it, the function blew past Vercel's 300MB limit and failed. On a VPS there is no 300MB limit, so the hack becomes irrelevant. But a new, very practical question appears: who serves those 597MB now?

5. It optimized the images. When the page asks for a stroller photo, you do not get the original 2MB file. next/image delivers a resized WebP, exactly as wide as the phone screen — maybe 40KB instead of 2MB. That optimization ran on Vercel's servers. On a VPS, it runs inside your own Node process, with a library called sharp (already installed).

6. It regenerated the pages by itself (ISR). revalidate = 86400 means "this page is good for 24 hours; after that, regenerate it from fresh content." Vercel kept track of which page was stale, when to remake it, where to put it. This is the piece you are least sure about, and rightly so — it is the subtlest. The big relief: ISR is not Vercel magic. It runs on your server too.

So what is a VPS, really? Now you can see it clearly.

Vercel gives you:  build + hosting + HTTPS + CDN + image optimization + ISR
                   (all already running)

A VPS gives you:   a bare computer
                   (you add the rest)

It is not a loss. It is a trade. You lose convenience. You gain control, and usually a small fixed monthly cost instead of a bill that grows with traffic.

And the pieces are not many. There are exactly six, and you already have their names:

build              → Docker + GitHub Actions
hosting            → VPS + container + compose
HTTPS              → Caddy (reverse proxy)
CDN + static files → Caddy + Cloudflare
image optimization → sharp, in your container
ISR                → cache on disk, in a volume

One honest thing so you don't worry needlessly: DNS is already solved. produsebaby.ro already lives on Cloudflare, and Cloudflare already gives you a free CDN in front. So of the six pieces, one is half-done before you start. You are not starting from zero. You are starting from Cloudflare already standing up.

Chapter 2 — The empty computer: your VPS

VPS stands for Virtual Private Server. Three words that sound important and explain nothing.

Here is what it is, no detours:

A VPS is a computer running in a data center, which you rent by the month, and talk to over the internet instead of with a keyboard.

That's it.

No monitor. No keyboard. It sits in a room full of other computers, somewhere in Germany or Finland, powered on around the clock. You speak to it over the internet.

Why "virtual." You could rent a whole physical machine. It is expensive and far too much for one site. Instead, the provider takes one big physical server and slices it in software:

One huge physical server
↓
sliced in software into 20 smaller computers
↓
each slice = one VPS
↓
you rent one slice

Each slice behaves like a complete, separate computer. Its own sliver of CPU, its own memory, its own disk. Neighbors on the same physical box cannot see your files. That is why it is "virtual" (cut in software, not a physical box) and "private" (your slice is yours).

"You have root." You will hear "you have root access" and "full control." root is the boss user on Linux. It can do anything: install programs, start services, delete everything by accident. On Vercel you never had this — you ran inside their box, by their rules.

Good:  you can install Docker, run anything, nobody says no.
Bad:   if you break something, you fix it. Nobody is on call.

The reassuring part: with Docker (the next chapters), you touch "root" very rarely. You install Docker once, and after that you work almost entirely through it.

How you talk to it: SSH. No monitor, so how do you give it commands? Through SSH — Secure Shell. An encrypted tunnel from the terminal on your laptop to the command line on the server.

Your laptop (terminal)
↓  ssh, encrypted
The internet
↓
The VPS, its command line

It looks like this:

ssh root@188.34.xxx.xxx

root is who you log in as. 188.34.xxx.xxx is the server's IP address (the provider gives it to you when you create the VPS). You press Enter, and suddenly your terminal is typing commands on the server in Germany.

You do not log in with a password. You log in with an SSH key: a pair of files, one secret on your laptop, one public on the server. They fit together like a key in a lock. Remember this shape — something secret with you, its public pair on the server. It comes back at deploy time.

How big a VPS do you need? Three numbers matter.

• CPU. How many "vCPU." For you it matters for two things: building the site and optimizing images with sharp. Both eat CPU in short bursts.

• RAM. How much it holds in mind at once. next build on a content-heavy site wants memory. Below a threshold, the build dies with "out of memory."

• Disk. Where the images live. You have 597MB in public/, plus the system, plus the Docker images. It adds up.

Translated for this site — static-first, ISR, no database, small-to-medium traffic — the hard part is not serving visitors. It is building and optimizing images. A reasonable starting target:

2 vCPU
4 GB RAM   (so the build doesn't die on your rich content)
~40–80 GB disk (images + Docker need room to breathe)

Do not take the smallest plan "to save money." A build that dies for lack of RAM costs you more hours than the few euros a month. And you can always grow — most providers let you resize from a panel in minutes.

Common providers for small VPSes: Hetzner (cheap, data centers in Germany/Finland — close to Romanian users, so low latency), DigitalOcean (friendly panel, lots of beginner docs), Vultr, Linode/Akamai, Contabo (a lot for a little, but more variable). The order of magnitude for the specs above is "a few euros up to ~15–20 €/month" — a fixed cost, unlike a Vercel bill that grows with traffic.

Here is the mistake everyone makes at the start: "VPS means I have to learn Linux administration." Partly true, but less than you think. Without Docker, yes — you install Node by hand, configure services, fight versions. Painful. With Docker, the VPS becomes almost empty inside. You install Docker once. After that, everything that runs — the app, the proxy — comes packaged. The server barely knows what Next.js is. It just runs containers.

And when is a VPS not the answer? If you want zero hassle and don't mind a cost that grows with traffic, Vercel is hard to beat — you pay precisely so you never do any of this. A VPS makes sense when you want a fixed, predictable cost, when you need control the platform won't give you, or when your reason is I want to learn DevOps. That last one is a very good reason, and it is the one driving this whole post.

Chapter 3 — Packaging the app: images and containers

A Docker image is a sealed box with everything your app needs to run: the built code, node_modules, the runtime version, even the system files.

It solves one problem, but a big one: "works on my machine."

Without Docker:  your laptop has version 1.3.5, the server has another → it breaks.
With Docker:     YOU package the exact versions → it runs the same everywhere.

An image is a cooked meal, sealed. Not instructions — the food, already made. You reheat it anywhere and it is identical.

Do not confuse the image with the container:

The image     = the template, frozen, on disk. Does not run.
The container = an instance started from the image. Runs.

The image is the class, the container is the object. One image → any number of containers.

How an image is built: the Dockerfile. An image is born from a file called Dockerfile: a list of steps, top to bottom. Each step is a layer, which Docker caches.

FROM ...     ← start from a ready-made image (e.g. bun installed)
COPY ...     ← bring your files in
RUN ...      ← run commands (install, build)
CMD ...      ← the command that starts when it becomes a container

Layer caching is why order matters: if package.json hasn't changed, Docker reuses the bun install layer and skips it. That is why you copy package.json first, then the rest of the code.

The one config change: output: 'standalone'. A normal next build leaves a .next that needs all of node_modules to run — hundreds of useless MB in the image. output: 'standalone' tells Next: "figure out exactly which files from node_modules are actually used at runtime, and put them in a small folder, .next/standalone, with its own server.js." So you add to next.config.ts:

const nextConfig: NextConfig = {
  output: 'standalone',   // ← ADD this
  images: {
    minimumCacheTTL: 2678400,
    remotePatterns: [{ protocol: 'https', hostname: '**.akamaized.net' }],
  },
  // outputFileTracingExcludes: no longer needed on a VPS (it was for Vercel's 300MB limit).
}

Now the trap everyone falls into:

standalone does NOT copy public/ or .next/static. It leaves them out on purpose. If you don't copy them into the Dockerfile yourself, ALL your images and CSS return 404.

Hold that thought. We fix it below.

The real Dockerfile. Your build has three steps: panda codegen, next build, then pagefind. The Dockerfile does all three, in two stages (multi-stage):

Stage 1 (builder):  installs everything, builds, generates Pagefind.  Heavy.
Stage 2 (runner):   copies ONLY the result. Light. This is what ships.

# ---- Stage 1: builder — has everything needed to build ----
FROM oven/bun:1 AS builder
WORKDIR /app

# Manifests first → cache bun install as long as they don't change
COPY package.json bun.lock ./
RUN bun install --frozen-lockfile

# Then the rest of the code
COPY . .

# Your full build: panda + next build + pagefind (exactly package.json)
RUN bun run build

# ---- Stage 2: runner — only what's needed to RUN ----
FROM oven/bun:1 AS runner
WORKDIR /app
ENV NODE_ENV=production
ENV PORT=3000
ENV HOSTNAME=0.0.0.0

# 1) the standalone server (small, node_modules already triaged)
COPY --from=builder /app/.next/standalone ./
# 2) the static CSS/JS — standalone omits it, you copy it (the trap)
COPY --from=builder /app/.next/static ./.next/static
# 3) the files in public/ (images + the Pagefind index built at build time)
COPY --from=builder /app/public ./public

EXPOSE 3000
CMD ["bun", "server.js"]

Those three COPY --from=builder lines are the heart. The first brings the server; the second and third fix exactly the 404 trap.

And a .dockerignore so COPY . . doesn't drag in your local node_modules, .next, .git:

node_modules
.next
.git
.env*
*.log

You build the image from the project root:

docker build -t produsebaby:latest .

-t produsebaby:latest is the name and tag. The . means "the Dockerfile is here." At the end you have an image on disk, ready to start. (:latest is fine here, local, to learn. In automated deploys we'll tag with the commit instead — traceable and easy to roll back.)

Now start it. The image sits on disk and does nothing. A container is that image started — a live process.

docker run produsebaby:latest
↓
Docker takes the image (frozen)
↓
starts it → now it's a container running `bun server.js`
↓
your app listens on port 3000, inside

Here is the rule you have to feel in your stomach:

Whatever a container writes to its own disk disappears when the container is deleted or rebuilt.

A container is disposable. Stop it, start another from the same image, and it is clean, like new. It remembers nothing. Why that matters directly:

ISR writes regenerated pages to disk (.next/cache).
next/image writes optimized images to disk (.next/cache/images).

Next deploy → new container → its disk is empty → all that cache is lost.

The fix has a name: a volume. A volume is a folder on the VPS (which stays) linked into a folder inside the container. What the container writes there lives on the VPS, not in the container. New container → same volume → the cache is still there. Remember the shape: the container is disposable; the volume is the memory that survives. We use it seriously in the ISR and image chapters.

To reach the app from outside, you punch a port through — the app listens on 3000 inside a closed box:

docker run -p 3000:3000 produsebaby:latest

-p 3000:3000 means "port 3000 on the VPS → port 3000 in the container." And to change settings without rebuilding the image, you pass environment variables (-e KEY=value), read in code as process.env.KEY. Secrets come in this way too — never baked into the image.

Two commands you will lean on constantly:

docker ps          # what containers are running now
docker logs <id>   # what the app printed (your errors live here)

On Vercel you looked in a dashboard. Here you look in docker logs.

One honest tradeoff before we move on: the COPY /app/public line stuffs those ~597MB (images + Pagefind) straight into the image. Simple, one artifact, works first try. But every deploy then ships ~600MB, and you rebuild even when you add just a few products. For now, that's fine — you want something that works, so you can learn. We decouple public/images from the image later, so deploys get fast again. First simple that works, then fast.

Chapter 4 — The whole stack in one file: Compose and Caddy

The command above already carries -p, several -e flags, soon volumes, plus a second container. That becomes a long, fragile line you don't want to type correctly every time.

So you stop typing it. Your site needs two containers working together:

app    → your Next app (bun server.js, on 3000)
caddy  → the reverse proxy, which takes the public traffic and the HTTPS

Docker Compose writes all of it once, in a file, and starts everything with one command:

docker compose up -d

-d means "detached," runs in the background. One command, the whole stack up.

Here is the complete stack, the starter version:

services:
  app:
    build: .                      # builds from the Dockerfile
    restart: unless-stopped       # if it crashes / the VPS reboots → it comes back
    environment:
      - NODE_ENV=production
      - NEXT_PUBLIC_SITE_URL=https://produsebaby.ro
    volumes:
      - nextjs_cache:/app/.next/cache   # ISR + optimized images survive redeploys
    expose:
      - "3000"                    # visible ONLY to other containers, not the internet

  caddy:
    image: caddy:2                # ready-made image, you build nothing
    restart: unless-stopped
    ports:
      - "80:80"                   # HTTP  → comes in from the internet
      - "443:443"                 # HTTPS → comes in from the internet
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro   # the proxy's config
      - caddy_data:/data          # where Caddy keeps TLS certificates (so it doesn't re-fetch)
      - caddy_config:/config
    depends_on:
      - app                       # start app before caddy

volumes:
  nextjs_cache:
  caddy_data:
  caddy_config:

Three ideas, that's all.

1. expose vs ports — the security difference.

ports:  "443:443"  → open to the INTERNET. Anyone can reach it.
expose: "3000"     → open only to the other containers in the stack.

Only caddy has ports. Your app is hidden behind, on expose. Nobody outside hits Next directly — everyone goes through Caddy. That is exactly what you want.

Internet → caddy :443 → app :3000
            (public)     (hidden)

2. Volumes are the memory that stays. nextjs_cache links /app/.next/cache inside the container to space that lives on the VPS. New container at deploy → same volume → ISR and optimized images are still there, not from scratch. caddy_data does the same for the TLS certificates.

3. restart: unless-stopped is self-restart. App crashes? VPS reboots? The containers come back automatically. On Vercel you never had this worry; here you solve it with one line.

And there is a hidden network: Compose automatically puts both containers on the same private network and names them. That is why, in the Caddy config next, you just write app:3000 — app is the service name, and Docker translates it to the right container. No IPs, no headache.

Now, what is Caddy actually doing at the front? Your app listens on port 3000, plain HTTP, hidden on the private network. The internet speaks on 443 (HTTPS) and 80 (HTTP), and demands a certificate. Something has to stand at the border and translate. That is the reverse proxy.

Internet (443, HTTPS, wants a certificate)
↓
REVERSE PROXY  ← stands at the door
↓
app:3000 (plain HTTP, behind)

"Reverse" because it guards the servers, not the clients. It is the building's receptionist: it takes everyone off the street and sends them to the right office inside. Concretely it does four jobs — terminates HTTPS (puts the padlock on, decrypts, sends plain HTTP behind), fetches the certificate automatically and free from Let's Encrypt and renews it, routes everyone to app:3000, and compresses responses.

That second job is why we pick Caddy over the classic Nginx. With Nginx, HTTPS means installing certbot, writing config, adding a renewal cron — steps you can get wrong. Caddy does HTTPS automatically, first try. You give it the domain name; it requests the certificate itself, installs it, renews it.

Nginx:  maximum power, you configure the certificates yourself.
Caddy:  90% of cases, HTTPS free and automatic, four lines of config.

Here is the entire Caddyfile. Four lines that matter:

produsebaby.ro {
    encode gzip zstd
    reverse_proxy app:3000
}

www.produsebaby.ro {
    redir https://produsebaby.ro{uri} permanent
}

The mere fact that you wrote the domain name in produsebaby.ro { ... } makes Caddy request the TLS certificate for it automatically. reverse_proxy app:3000 forwards everything to the app container on the private network. The www block sends www visitors to the canonical no-www address (good for SEO).

So here is the back half of our request flow — the first arrows of a diagram we'll keep adding to:

Internet
↓  :443, encrypted
caddy: checks the certificate, decrypts
↓  app:3000, plain HTTP, private network
app (Next) receives the request and starts working

Your app never knew about HTTPS. It got clean HTTP. Caddy carried all the encryption weight.

Chapter 5 — The pages that refresh themselves: ISR

This is your big fear: "ISR is something Vercel does. If I leave, do I lose it?"

Short, reassuring answer:

ISR is not Vercel magic. It is a Next.js feature. It runs just as well on your server. You only have to understand where it keeps its cache — and put that on a volume.

ISR is Incremental Static Regeneration. Your stroller page goes through three states:

revalidate = 86400   (from src/app/[category]/[slug]/page.tsx)

That means "this page is good for 24 hours; then, on the first request after, regenerate it in the background from fresh content." The three states:

FRESH   (under 24h)  → serve from cache instantly. Zero work.
STALE   (over 24h)   → still serve from cache instantly (the user does NOT wait),
                       AND kick off regeneration in the background. The next user
                       gets the new version.
MISSING (never built) → generate now, put it in cache, serve it.

The key idea: the user almost never waits. They always get something from cache; the new version bakes in the background. Static speed plus content that refreshes.

Where does the cache live? On disk, in .next/cache. This is the whole secret, and it is more boring than you feared. When Next generates the stroller page, it writes it as files, on disk, in .next/cache/... — an HTML file plus some metadata. A folder on disk.

Request for /carucioare/cele-mai-bune-carucioare
↓
Next looks in .next/cache
↓
found + fresh?  → send the file. done.
found + stale?  → send the file NOW, regenerate in background, overwrite on disk
missing?        → generate, write to disk, send

On Vercel, that "disk" was their infrastructure, spread across the whole CDN. On your VPS, that "disk" is the container's disk. Same mechanism, different place.

Who triggers the regeneration? A request — not a cron. A common confusion: you think a clock somewhere remakes the pages every 24 hours. No. Nothing happens until a request comes in for an expired page. That request is the trigger. It finds the stale page and starts the regeneration.

No request = no regeneration. The stale page sits quietly on disk.
First request after expiry = pulls the new version in the background.

So generateStaticParams (which you already use on three routes) pre-generates the pages at build time, so the first visit is already fast. ISR refreshes them afterward, on demand.

Now the bad version first. Here is the naive setup, with no volume:

Deploy → new container from the new image → its disk is EMPTY
↓
.next/cache is empty
↓
every page is "MISSING" → first visit regenerates it from scratch
↓
every deploy, the entire ISR cache is lost

Nothing breaks — but on every deploy, your first visitors pay to regenerate every page. On a content-heavy site, you feel it.

The fix you already know: a volume. It's one line in the Compose file:

    volumes:
      - nextjs_cache:/app/.next/cache

It links .next/cache inside the container to space that lives on the VPS, outside the container.

Deploy → new container → but .next/cache = the same volume as before
↓
the regenerated pages are still there
↓
ISR survives the deploy

One line. That is the whole difference between "ISR runs poorly on a VPS" and "ISR runs like it did on Vercel."

One hidden trap: stay at a single container. The on-disk-files mechanism assumes one container writing to that .next/cache. If one day you put two app containers behind Caddy for more traffic, each has its own volume. Container A regenerates the page; container B still sees the old version. They desync.

For you, now: stay at one container. Static-first, small-to-medium traffic — one container carries it easily. Just so you know it exists: if you ever scale to several containers, Next 16 gives you a cacheHandler in next.config.ts that keeps the cache in a shared place (usually Redis) instead of each container's disk. Remember only the rule: one container → volume on disk (simple, what you have). Several containers → shared cache (Redis, later, if ever).

Now the request flow grows — add ISR to the arrows from the last chapter:

caddy hands the request to app:3000
↓
Next: /carucioare/cele-mai-bune-carucioare — I look in .next/cache (on the volume)
↓
fresh → instant HTML.  stale → instant HTML + regenerate in background.
↓
caddy encrypts the response, sends it back
↓
the parent sees the article — and now the images enter the scene

Chapter 6 — Images, sharp, and whether you need a CDN

You have 2844 images. 404MB in public/images. All local — published products serve their photos from public/, not from an external CDN.

Two questions nag at you: who optimizes them now and do I need a CDN? Same running example, one thread.

What "optimized" means. The stroller page asks for a photo. The original file might be 2MB, 2000px wide. next/image does not send it as is:

Original:  carucior.jpg, 2000px, 2MB
Delivered to phone:  WebP, 640px, ~45KB

Resized to the screen, in a modern format, compressed. Forty times smaller.

Where the optimization runs now — and this is the change. On Vercel, it ran on their servers. On a VPS, it runs in your own Node process, with the sharp library (already installed).

Request: /_next/image?url=/images/carucior.jpg&w=640&q=75
↓
Next reads the source file from public/images
↓
sharp resizes it + converts to WebP   ← THIS eats CPU
↓
the result is written to .next/cache/images (on the volume!)
↓
sent to the user

Two things to keep from this. First, sharp eats CPU — the first request for a new size does real work (which is why the VPS chapter told you not to take the weakest box). Second, it happens ONCE. The result lands in .next/cache/images. The second visitor asking for the same photo at the same size gets it from cache, no work. And because .next/cache is on a volume, that cache survives deploys too. minimumCacheTTL: 2678400 in your config (31 days) reinforces it: it tells the browser and CDN "this optimized photo is good for a month, don't ask again."

Do you need a CDN? What it is, really. A CDN keeps copies of files in cities all over the world, close to users, so the photo comes from Frankfurt, not from your server.

Without a CDN:  every visitor → hits your VPS → the VPS sends the photo
With a CDN:     first visitor → VPS. Everyone else → from the CDN cache, near them.

Good news: you already have a CDN. produsebaby.ro is on Cloudflare, and Cloudflare is a free CDN. You don't buy anything. You just let it do its job over your images.

Which is the single most important optimization here — Cloudflare caching the optimized photos:

Without Cloudflare caching:
  every visitor → /_next/image → sharp works (or reads from disk)
  → your VPS works on every photo, for every visitor

With Cloudflare caching /_next/image:
  first visitor → sharp makes the photo once
  → Cloudflare holds it at the edge
  → thousands of visitors → straight from Cloudflare, the VPS never hears about it

So sharp runs once per (photo, size), then Cloudflare serves everyone. Your VPS barely breathes. You set this in Cloudflare with a Cache Rule that caches the paths /_next/image* and /images/*. Cloudflare already respects the Cache-Control header Next sets (from minimumCacheTTL), so it often works from the start.

The 600MB problem: don't bake images into the Docker image. In the packaging chapter we stuffed all of public/ into the Docker image. Simple, but painful:

The Docker image becomes ~600MB.
You add 5 new products → you rebuild and ship 600MB.
The images have nothing to do with the code, but you haul them on every code deploy.

The upgrade: keep the images on the VPS, outside the Docker image. They don't change on every code deploy, so they don't belong in the code artifact. Three moves.

First, put the images folder on the VPS once (e.g. /srv/produsebaby/images) and update it separately. Second, mount it into the container read-only, so the optimizer can read the photos:

  app:
    volumes:
      - nextjs_cache:/app/.next/cache
      - /srv/produsebaby/images:/app/public/images:ro   # ← photos live on the VPS, not in the Docker image

That line is a bind mount: path_on_VPS : path_in_container : options.

/srv/produsebaby/images   real folder, on the VPS disk (the host)
/app/public/images        where that folder appears INSIDE the container
:ro                       read-only — the container only READS, never writes

The container sees the VPS folder as if it were its own. Not a copy — the same folder, viewed from two places. Third, drop the images from the Docker image so you stop baking them, in .dockerignore:

public/images

Note that .next/static and the Pagefind index (public/pagefind) STAY in the image — those genuinely change with every build and must stay in sync with the code. Only public/images moves out, because it's large and independent of the code.

How the images get onto the VPS: rsync. You update the images "separately." With what? Here is the bad version first — copying the whole folder every time with scp:

scp -r ./public/images root@188.34.xxx.xxx:/srv/produsebaby/images

The problem: scp doesn't look at what's already there. It sends all 404MB, from scratch, even if you added just 8 new stroller photos. 404MB on the wire, for 8 photos. Every time. Slow, and pointless.

rsync looks at both sides first — what you have locally and what's already on the VPS — compares, and sends only the difference.

Your laptop: 2852 images
VPS:         2844 images
↓ rsync compares
sends only the 8 new ones
↓
VPS: 2852 images

The real command:

rsync -avz --delete ./public/images/ root@188.34.xxx.xxx:/srv/produsebaby/images/

-a        "archive": keep folder structure, dates, permissions
-v        "verbose": show which files it sends
-z        "zip": compress on the way, fewer bits on the wire
--delete  what you deleted locally, delete on the VPS too (exact mirror)
./public/images/   source (local) — the trailing / matters: "the folder's contents"
root@...:/srv/...  destination (on the VPS, over SSH)

You run it whenever you add or change product images locally. Only the new bits fly — no Docker rebuild, no code deploy. Images and code become two separate streams, exactly what you wanted. (Be careful with --delete: it makes the VPS an exact mirror of your local folder. The first time, run it without --delete, or add --dry-run, which shows what it would do without touching anything.)

Now the flow grows again — the image path folded in:

The stroller page asks for the photos
↓
/_next/image?url=/images/carucior.jpg&w=640
↓
Cloudflare has the optimized copy?  → sends it. The VPS is not even bothered.
↓ (if not)
Next + sharp make it once, put it in .next/cache/images (volume), send it
↓
Cloudflare holds it for everyone else
↓
the parent sees the strollers, fast

Chapter 7 — Getting the domain to the server: DNS and HTTPS

Until now, the request started magically heading for the server. This chapter explains the first step, the one that was missing: how the phone knows where "produsebaby.ro" is.

DNS is the phone book of the internet. Computers don't find each other by name, but by number (IP address). DNS translates.

"produsebaby.ro" = ?
↓ DNS question
"188.34.xxx.xxx"  ← your VPS's IP
↓
the phone connects to that number

That translation lives in an A record: name → IP.

A record:  produsebaby.ro → 188.34.xxx.xxx

Right now your A record points to Vercel. The migration, in essence, is changing the IP in the A record to point to your VPS. That's it. The rest of the internet picks up the change on its own.

You make the change on Cloudflare, which you already use. You don't add a new provider — you just edit the A record:

produsebaby.ro       A     188.34.xxx.xxx   (the VPS IP)
www.produsebaby.ro   A     188.34.xxx.xxx   (same, Caddy redirects it)

The change propagates in minutes to hours (usually minutes on Cloudflare).

The cloud: DNS-only (grey) vs Proxied (orange). Next to each record, Cloudflare has a little cloud. This is the whole nuance, and it is simple once you see it.

Grey (DNS-only): Cloudflare just answers "the IP is X." The visitor connects directly to your VPS.

Phone → Cloudflare (only says the IP) → straight to the VPS

Orange (Proxied): the traffic goes through Cloudflare. This is where you get the CDN and the image cache from the last chapter.

Phone → Cloudflare (CDN, cache) → VPS

You want orange in the end, so it caches the images. But at first startup, orange complicates Caddy's certificate. That's why the order matters.

HTTPS: who puts the padlock on, and the two-layer trap. Recall that Caddy requests the certificate itself from Let's Encrypt. To grant it, Let's Encrypt verifies by hitting port 80 of the domain. The problem with orange on from the start:

Orange ON → Let's Encrypt hits port 80, but lands on Cloudflare, not Caddy
→ Caddy can't prove it owns the domain → no certificate → HTTPS fails

Plus, with orange you have two TLS layers (visitor↔Cloudflare and Cloudflare↔VPS) which, badly configured, fight each other ("too many redirects"). So do exactly this order:

1. In Cloudflare: A record → the VPS IP. Leave the cloud GREY (DNS-only).
2. Start the stack: docker compose up -d
   → Caddy hits Let's Encrypt unobstructed (port 80 reaches it directly)
   → gets the certificate, HTTPS works
3. Verify: open https://produsebaby.ro — padlock, site OK, straight from the VPS.
4. NOW turn the cloud ORANGE on the records.
5. In Cloudflare → SSL/TLS mode → "Full (strict)".
   → visitor↔Cloudflare encrypted, Cloudflare↔VPS encrypted, no quarrels.
6. Verify again. Now you also have Cloudflare's CDN in front (image cache).

Once fetched, Caddy keeps the certificate on the caddy_data volume, so it doesn't re-request it on every start, and it handles renewal by itself.

Now the flow finally grows a front — the first step is no longer magic:

Phone: "where is produsebaby.ro?"
↓  DNS (Cloudflare) → the VPS IP
↓  (orange) through Cloudflare — image cache
↓  :443
Caddy on the VPS: certificate valid, decrypt
↓  app:3000
Next: ISR from .next/cache
↓
Cloudflare caches, the parent sees the page

Chapter 8 — Shipping code: manual deploy, then GitHub Actions

You wrote new code on your laptop. How does it go live?

On Vercel: git push, done. Vercel caught the push, built, started. Invisible. On a VPS you have to know what was actually happening there, to rebuild it. Every deploy, however fancy, is the same four steps:

1. The new CODE reaches the server        (git pull, or a ready-made image)
2. The new image is BUILT                  (the Dockerfile)
3. The old container is REPLACED           with one from the new image
4. The volumes STAY                        (ISR, images, certificates — intact)

Remember these four and you won't get lost.

Method 1: build on the server (the simplest — start here). The code lives on the VPS as an ordinary git repo. Once, at the start, you clone it. Then every deploy is:

ssh root@188.34.xxx.xxx
cd /srv/produsebaby
git pull                        # step 1: the new code
docker compose up -d --build    # steps 2+3: rebuild the image, replace the container

--build rebuilds the image, then Compose stops the old container and starts a new one from the fresh image. The volumes pass through untouched.

Good:  simple, nothing to configure, you see everything happen.
Bad:   the build runs ON the VPS → eats its RAM (that's why 4GB).
       And you type commands by hand on every deploy.

Secrets never go in git. On the VPS you put them in a .env file next to docker-compose.yml (a file that is NOT in git), and Compose reads it automatically.

Method 2: build elsewhere, ship a ready-made image (registry). Method 1 has a flaw: the heavy build runs on your small VPS, while it is serving visitors. The alternative: build the image somewhere else, push it to a registry (a Docker image warehouse — e.g. GitHub Container Registry, ghcr.io), and on the VPS just pull it ready-made.

Somewhere powerful (laptop / GitHub)     Your VPS
────────────────────────────────────     ─────────────
build the image
push to ghcr.io          ───────►         pull from ghcr.io
                                          docker compose up -d
                                          (no local build!)

On the VPS, the deploy becomes:

docker compose pull    # pull the new, already-built image
docker compose up -d   # replace the container. Zero build on the VPS.

In docker-compose.yml, the difference is one line: instead of build: . you point to the registry image, tagged with the commit, not :latest:

  app:
    image: ghcr.io/your-user/produsebaby:${IMAGE_TAG:-latest}   # instead of "build: ."

${IMAGE_TAG:-latest} means "use the IMAGE_TAG variable, and if it's not set, fall back to latest." Why bother? Rollback in 10 seconds. Something broke? On the VPS:

IMAGE_TAG=<yesterday's-good-commit> docker compose up -d

and you're back to the version that worked, no rebuild, no git. Each commit has its own image in the registry, ready to restart. With :latest you'd have nothing to go back to — every version wore the same tag.

Now automate it: GitHub Actions. Doing the deploy by hand works, but it's tiring and easy to get wrong at 2 a.m. GitHub Actions does those steps by itself, on every git push — exactly what Vercel did, except now you wrote the recipe once. That's CI/CD:

CI (Continuous Integration): on every push, build and check automatically.
CD (Continuous Deployment):  if it's OK, ship to the server automatically.

What runs the recipe: a "runner." When you push, GitHub starts a temporary, free computer of theirs — a clean Ubuntu that lives for the ~3 minutes your job runs, then vanishes.

git push
↓
GitHub starts a runner (clean Ubuntu, on their side, free)
↓
the runner: builds the image, pushes it to the registry, calls the VPS to pull the new version
↓
the runner self-destructs

The beauty: the heavy build happens on their computer, not your small VPS. The VPS just pulls the ready image (method 2). Here is the recipe, .github/workflows/deploy.yml:

name: Deploy

on:
  push:
    branches: [main]        # triggers on push to main

# If you push twice fast, cancel the old run and keep only the latest.
concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest  # the runner: a clean Ubuntu from GitHub
    steps:
      # 1. Get the code into the runner
      - uses: actions/checkout@v4

      # 2. Log in to the registry (GitHub Container Registry)
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}   # given automatically by GitHub

      # 3. Prepare buildx (the build engine with layer cache)
      - uses: docker/setup-buildx-action@v3

      # 4. Build the image and push it to the registry.
      #    tag = the commit (github.sha), not :latest → traceable + easy rollback.
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      # 5. Call the VPS over SSH and restart with that commit's image
      - uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.VPS_HOST }}
          username: root
          key: ${{ secrets.VPS_SSH_KEY }}
          script: |
            cd /srv/produsebaby
            export IMAGE_TAG=${{ github.sha }}   # exactly the commit built now
            docker compose pull                  # pull that commit's image
            docker compose up -d                 # replace the container

Read the steps like a story: get the code → log in to the warehouse → prepare buildx → build and push the image (tagged with the commit) → call the server to pull exactly that commit. Exactly the manual deploy, written once. Three small lines do a lot of work:

1. Tag with github.sha, not :latest. github.sha is the hash of the pushed commit. :latest means "the latest" — which latest? You don't know what code is inside. :<commit-sha> means this image is this code. Traceable (you can always say exactly what runs in production) and easy to roll back. That's why the deploy step does export IMAGE_TAG=${{ github.sha }} before docker compose pull.

2. Docker layer cache → fast builds. cache-from / cache-to: type=gha save the Docker image layers in the GitHub Actions cache, between runs. First run: bun install runs from scratch. Later runs: if bun.lock hasn't changed, the bun install layer is reused from cache. Every Dockerfile line is a layer; the cache reuses the unchanged ones, on clean runners that would otherwise start from zero every time.

3. concurrency: cancel-in-progress → no overlapping deploys. You push, spot a mistake, push again fast. Without this, you'd have two deploys running in parallel, racing to reach the VPS last. With it, the new run kills the old one. Only the latest push goes live.

Where the passwords live. You never put passwords in the .yml (it goes into git, everyone sees it). You put them in GitHub → repo → Settings → Secrets, and call them with ${{ secrets.NAME }}. You need two, plus one free:

VPS_HOST      = the VPS IP
VPS_SSH_KEY   = the private SSH key the runner uses to enter the VPS
GITHUB_TOKEN  = GitHub gives it to you automatically, you don't set it

And there is the SSH key shape from the VPS chapter — something secret with you, its public pair on the server — except now the runner plays the role of "you":

The PRIVATE key  → in GitHub Secrets (VPS_SSH_KEY). Only the runner uses it.
The PUBLIC key   → placed on the VPS (in ~/.ssh/authorized_keys).

That is how the temporary runner logs into the VPS with no password, exactly like you, but automatically. What you gained, concretely:

Before:  git push → ssh VPS → cd → git pull → docker compose up --build → wait
With Actions:  git push → (go make a coffee) → it's live

You rebuilt the Vercel experience. You push, the rest happens on its own. Except now you know what happens, on every line.

Chapter 9 — The whole picture

You've seen every piece separately. Now put them in one picture and follow the parent's request from end to end, without skipping a single link.

The parent types produsebaby.ro/carucioare/cele-mai-bune-carucioare
│
▼  1. DNS (Cloudflare): "produsebaby.ro" → the VPS IP
│
▼  2. Cloudflare (orange cloud): CDN + image cache
│       └─ has the page/photo at the edge? → sends it, done.
│
▼  3. Your VPS, port 443
│
▼  4. Container `caddy`: verify the TLS certificate, decrypt
│       └─ reverse_proxy app:3000
│
▼  5. Container `app` (Next, bun server.js, port 3000)
│       ├─ page in .next/cache?  ISR: fresh→instant, stale→instant+regenerate
│       └─ photos: /_next/image → sharp once → .next/cache/images
│
▼  6. The volumes remember between deploys: nextjs_cache, caddy_data
│
▼  the response goes back up through Caddy → Cloudflare (caches) → phone
│
▼  the parent sees the strollers article. Fast.

All six services Vercel had glued together are there, now yours, each with its own name.

The whole migration fits in five new files plus one changed line:

produsebaby.ro/
├── next.config.ts              ← ADD: output: 'standalone'
├── Dockerfile                  ← NEW: the image recipe (bun, multi-stage)
├── .dockerignore               ← NEW: what not to put in the image
├── docker-compose.yml          ← NEW: app + caddy + volumes
├── Caddyfile                   ← NEW: domain + HTTPS + proxy
└── .github/workflows/deploy.yml ← NEW: auto-deploy on push

Plus, on the VPS (not in git): a .env with secrets and the /srv/produsebaby/images folder. And the plan, in order — each step leans on the one before:

Prepare (on the laptop)
1. Add output: 'standalone' to next.config.ts
2. Create Dockerfile, .dockerignore, docker-compose.yml, Caddyfile
3. Test LOCALLY: docker compose up --build → open localhost
   (if it works locally, it works on the VPS — that's the whole point of Docker)

The server (once)
4. Rent a VPS (2 vCPU / 4GB / ~40-80GB), get an IP
5. Connect: ssh root@IP, install Docker
6. Put your public SSH key on the VPS
7. git clone into /srv/produsebaby, create .env with the secrets
8. Upload the images to /srv/produsebaby/images (rsync)

Startup + domain
9.  docker compose up -d  → Caddy gets the certificate, the site comes up
10. In Cloudflare: A record → the VPS IP, GREY cloud
11. Verify https://produsebaby.ro directly from the VPS
12. Turn the cloud ORANGE + SSL "Full (strict)"
13. Cache Rule on Cloudflare for /_next/image and /images

Automation (last, once manual works)
14. Add .github/workflows/deploy.yml
15. Put the secrets in GitHub: VPS_HOST, VPS_SSH_KEY
16. git push → automatic deploy. You've rebuilt Vercel.

Don't jump to step 14 until the manual deploy works. First understand what you're automating, then automate it.

What you gain and lose versus Vercel. Honestly, so you decide with open eyes:

It is not universally "better." It is a trade: convenience for control, fixed cost, and understanding. And when should you not do this? If your site might explode in traffic tomorrow and you have no time for any of it, Vercel carries you without touching any of these files. You pay precisely so you're never on call. A VPS asks you to be the admin: certificate renewal (Caddy does it, but you check), server security, backups. Not hard, but yours.

Here is the final test. If you read all of this, you should be able to say, without looking:

First, DNS on Cloudflare translates my domain into the VPS's IP.

Cloudflare sits in front and caches the images, then sends the request to the VPS on 443.

Caddy receives it, puts HTTPS on, and hands it to the Next container on 3000.

Next looks in .next/cache (on a volume) and either serves the ready page or regenerates it in the background — that's ISR, and it runs on my server, not just on Vercel.

The photos are optimized by sharp once, kept in cache, and Cloudflare serves them to everyone else.

And when I git push, GitHub Actions rebuilds the image and restarts the container, by itself.

If you can say that, you learned the DevOps you needed. Not some words. The whole flow.

Because that was the trade all along:

Vercel glued six services into one button.
You pulled them apart and put them back on a bare computer, each with its name.
You gave up the convenience, and took the control, the fixed cost, and the understanding.

First of all, you can play with the layout trashing here.

Here is a number that should not be possible.

Two loops. Same work. The same 2,000 elements.

One takes 4.9 milliseconds.

The other takes 2,504 milliseconds.

That is 511 times slower. For the same result.

The slow one didn't do more work. It did the same work in the wrong order. It read a value at the wrong moment.

That is layout thrashing.

By the end of this you'll be able to look at a loop and say, out loud, exactly when the browser is about to freeze — and why.

The one question underneath

You write this:

const height = box.offsetHeight;

You just read a number. A property. It looks free.

It is not free.

In the wrong loop, that one line is slower than a network call.

So here is the question this whole post answers:

How can *reading* a value off an element
be the slowest thing your page does?

To answer it, you have to know what the browser is doing behind that read. So let's follow a frame.

The pipeline — what happens between your code and a pixel

Your JavaScript does not draw pixels.

It changes a description of the page. The browser turns that description into pixels, in stages, in order:

Your JavaScript
↓
Style      (which CSS rules apply to which elements?)
↓
Layout     (where is every box, and how big?)
↓
Paint      (what color is every pixel — as a list of draw commands)
↓
Composite  (stitch the painted layers together on the GPU)

Four stages after your code runs. Each one takes the output of the last.

Now the part most explanations skip. Where does each stage run?

Your JavaScript    → main thread
Style              → main thread
Layout             → main thread
Paint (record)     → main thread
Raster + Composite → compositor thread + GPU

The first four share one thread. The same thread that runs your code. The same thread that handles clicks and keystrokes.

If that thread is busy, nothing else happens. No frame. No response to input. The page is frozen.

Hold onto that line. It is the whole story.

What "Layout" actually computes

Style decides which rules apply.

Layout decides where everything goes.

Given the tree of boxes and their styles, Layout computes the geometry: for every element, its x, y, width, and height. Where it sits. How big it is.

The data going in: a tree of boxes plus their styles.

The data coming out: the same tree, now with a number for every position and size.

This is expensive, and it is not local. Boxes affect each other. Make one wider and its siblings shift. Its parent grows. On a page with a thousand boxes, Layout has to place a thousand boxes, together.

So the browser does something clever to avoid running it too often.

The browser is lazy on purpose

The browser does not recompute Layout every time you touch the DOM.

That would be madness. You might change ten things in one function.

So when you write something that could change geometry:

box.style.width = "200px";

the browser does not recompute layout.

It makes a note: layout is dirty now.

Then it goes back to running your code.

It is waiting. It wants to batch. The plan is to recompute layout once, later — right before it paints the next frame, after your code is done making its mess.

write, write, write   → "dirty, dirty, dirty"   (no layout yet)
...your code finishes...
↓
Layout runs once
↓
Paint

One layout per frame, no matter how many writes. That is the plan.

The up-to-date geometry is clean, and the browser keeps it cached. A write marks it dirty. Clean layout is stored and reused. Dirty layout is a promise to recompute — later.

Later. Not now.

That word is doing all the work.

The trap: when "later" becomes "right now"

Here is the naive loop. It looks completely reasonable.

You have a list of boxes. You want to double each one's height.

for (const box of boxes) {
  const height = box.offsetHeight;       // READ
  box.style.height = height * 2 + "px";  // WRITE
}

Read the height. Write double. Next box.

You might think reading offsetHeight is cheap. It's just a property.

But watch what it does to the browser's plan.

The browser wanted to batch layout for later. Then you ask for offsetHeight.

offsetHeight is a geometry value. To hand you a truthful answer, the browser needs layout to be clean. But you dirtied it one line ago, when you wrote style.height on the previous box.

So the browser has no choice. It will not give you a stale number. It stops, and recomputes layout right now — synchronously, in the middle of your loop — just to answer your read.

That is a forced synchronous layout. A forced reflow.

Then the next line writes style.height again. Dirty again.

Next iteration reads offsetHeight. Forced reflow again.

write → read → reflow → write → read → reflow → ...

For a thousand boxes, that is a thousand full layouts, in a single frame. The browser is running the most expensive stage of its pipeline over and over — because you keep asking it a question it can only answer by doing the work.

That is the thrash.

The fix is just reordering

You don't need to do less work. You need to stop interleaving.

Do all the reads first. Then all the writes.

// read everything while layout is still clean
const heights = boxes.map((box) => box.offsetHeight);

// now write everything — this dirties layout, but nobody reads it back
boxes.forEach((box, i) => {
  box.style.height = heights[i] * 2 + "px";
});

The reads happen while layout is clean, so they're free — the browser hands back cached numbers.

Then the writes happen together. They dirty layout. But nothing reads it back before the loop ends. So the browser keeps its promise: it recomputes layout once, afterward, before the next paint.

A thousand forced reflows became one.

Same boxes. Same result. These are the two numbers from the top:

thrash  (read, write, read, write):   2,504 ms
batched (read all, then write all):       4.9 ms

511 times faster. By moving the reads above the writes.

So, the definition

Layout thrashing is reading a layout value after you've written one, in a loop, forcing the browser to recompute layout every time.

That is all it is.

It is not a browser bug. The browser is doing exactly what you asked: handing you a truthful, up-to-date number. Every single time you ask.

When a read after a write is fine

Now don't over-correct.

You could read all this and start fearing every offsetHeight.

Here is the auto-growing textarea you have probably written:

const handleInput = (e) => {
  const el = textareaRef.current;
  if (!el) return;
  el.style.height = "auto";                  // WRITE — reset so it can shrink
  el.style.height = el.scrollHeight + "px";  // READ (one flush) + WRITE
};

There is a write, then a read, then a write. By the letter of the definition, that read forces a reflow.

And it is completely fine.

It forces layout once. On one element. When the user types a character.

Thrashing is not "a read after a write." It is a read after a write repeated — in a loop, across many elements, many times per frame. One flush on one textarea is a rounding error. A thousand flushes in one loop is a frozen page.

The scale is the whole difference.

If you want it tidy, do the read-once move anyway. It costs nothing and it reads better:

el.style.height = "auto";       // write
const next = el.scrollHeight;   // read once
el.style.height = next + "px";  // write

The rule that actually matters: don't go on to read more layout — offsetHeight, getBoundingClientRect() — later in the same handler, especially in a loop. One localized read-write pair is not the enemy. Interleaving is.

The reads that spring the trap

The dangerous read isn't only offsetHeight.

It's any value that depends on layout. Ask for one while layout is dirty, and you force a reflow.

There's a canonical list (Paul Irish maintains it). The ones you actually hit:

• offsetTop, offsetLeft, offsetWidth, offsetHeight

• clientTop, clientWidth, clientHeight

• scrollTop, scrollWidth, scrollHeight

• getBoundingClientRect()

• window.innerWidth, window.innerHeight — yes, even these

• getComputedStyle(el) — when you read a layout-dependent value off it

That last one is the sneaky one.

getComputedStyle looks like it just reads CSS. Passive. Harmless. But ask it for .height while layout is dirty, and it forces the exact same reflow offsetHeight does.

People sprinkle getComputedStyle through render loops thinking it's a free lookup.

It is not free.

It is not a passive CSS read.

It forces layout, same as offsetHeight — just wearing a different coat.

The expensive part is the call, not the value

Here is the part that saves you.

getBoundingClientRect() forces layout when you call it. Once, and only if layout was dirty.

What it hands back is a DOMRect — a plain object, frozen at that moment:

const rect = container.getBoundingClientRect(); // may force layout — once

That rect is a snapshot. It is detached from the live page. Reading from it later is just reading an object in memory:

rect.height; // free — no layout
rect.top;    // free
rect.width;  // free

You might think every rect.height re-measures the element.

It does not.

The measuring already happened, at the call. The object does not update itself when the page changes underneath it.

So the pattern that avoids thrashing is not "never read geometry." It is:

Call once. Store the rect. Read from the snapshot.

One getBoundingClientRect() at the top of your frame, cached in a variable, can feed a hundred later reads for free. The trap is calling it a hundred times — once per element, interleaved with writes.

Where thrashing loves to hide: scroll handlers

The loop doesn't have to be a for loop you wrote.

The worst one fires on scroll.

scroller.addEventListener("scroll", () => {
  for (const box of boxes) {
    const rect = box.getBoundingClientRect(); // forced reflow, ×N
    // ...work out whether this box is on screen
  }
});

Scroll events fire fast — many times a second, in the gaps between frames. Each one reads getBoundingClientRect() for every box. Each read forces layout.

So the user drags, and every scroll tick is doing hundreds of reflows. The scroll stutters. The thing they're touching is the thing that's frozen.

The fix is the same shape: read once and cache, or throttle to one read per frame.

But this handler at least contains its own crime — the reads and the loop are right there to review. The worst version of this bug doesn't.

The write you can't see: thrashing across frames

Here is a real one, from a production site.

The table of contents highlights the heading you're currently reading. A scroll handler finds it:

const handleScroll = () => {
  for (const heading of headings) {
    const rect = heading.getBoundingClientRect(); // READ — clean, right?
    // ...find the heading nearest the top of the viewport
  }
  setActiveHeadingId(nearest); // React state update. Not a DOM write... yet.
};

Now look for the write→read interleave.

It is not there.

All the reads happen first. Then one setState. Inside this function, the ordering is textbook-correct.

But setState is a deferred write. React re-renders and commits the new highlight class to the DOM after the handler returns — in the gap between this scroll event and the next one.

So the timeline across events looks like this:

scroll event 1:  reads (layout clean — cheap) → setState
↓
React commits the highlight class   → layout is dirty now
↓
scroll event 2:  first read         → FORCED REFLOW
↓
React commits the next highlight    → dirty again
↓
scroll event 3:  first read         → forced reflow again…

No single function contains the anti-pattern. The write and the read live in different frames. The interleaving only exists across time — which is exactly why it survives code review. Every piece, reviewed alone, is correct.

Measured on that TOC, with a heading list about twenty items long: 25 layout reads per frame while scrolling, and 60% of frames doing forced-layout work.

The fix didn't reorder the reads. It deleted them.

IntersectionObserver lets you describe the zone you care about once — and then the browser tells you when things cross it:

const observer = new IntersectionObserver(onHeadingsCrossed, {
  rootMargin: "0px 0px -66% 0px", // only the top third counts as "being read"
});
headings.forEach((heading) => observer.observe(heading));

The observer computes intersections off the hot path and calls back with ready-made entries — each one already carrying a boundingClientRect, precomputed, free. The handler flips a class and never asks the DOM a geometry question.

Same TOC, after the refactor: 3.6 reads per frame. 6% thrashing frames instead of 60%. Same highlight, same UX.

And notice what the fix did not do: it did not stop writing. The highlight still moves, the class flip still dirties layout, and the browser still reflows once before the next paint — exactly as designed.

Writes are unavoidable. The page has to change; that is the point of the page.

What you can avoid is asking geometry questions while the answer is being recomputed. Stop the asking, and the writes go back to being what the browser always wanted them to be: batched, once, right before paint.

The real fix, generalized: reads now, writes later

The loop fix was "all the reads, then all the writes."

The same idea scales up to your whole app. The tool for it is requestAnimationFrame.

requestAnimationFrame runs your callback once, right before the browser paints the next frame — after the current JavaScript has finished.

So you can split a frame into two phases:

during the event:    read everything you need — layout is clean
↓
requestAnimationFrame
↓
just before paint:   write everything — layout dirties, then flushes once

Reads now. Writes later. Never interleaved.

Here it is on a scroll-to-top after some change:

// before — a synchronous write, in the middle of everything else
container.scrollTo({ top: 0 });

// after — deferred to just before the next paint
requestAnimationFrame(() => {
  container.scrollTo({ top: 0 });
});

Wrapping the write in requestAnimationFrame lifts it out of the read phase. If you had already read some geometry earlier in the same event, this stops the write from forcing a reflow in the middle of it.

This is the seed of what a library like FastDOM does: a measure() queue and a mutate() queue, both flushed in one rAF, reads always before writes.

But be honest about what you changed.

The write now happens one frame later.

You might think that is free.

It is not always.

requestAnimationFrame(() => {
  container.scrollTo({ top: 0 });
});
const top = container.scrollTop; // reads the OLD position — the scroll hasn't run yet

Any code right after — anything that assumes the scroll already happened — now sees the old value. If something depends on "we are at the top now," either keep the write synchronous, or move that dependent code into the same requestAnimationFrame callback.

Deferring a write buys you clean read/write ordering. It costs you a frame of "not yet." Know which one you need.

The sibling problem: animating the wrong property

Layout thrashing is reads forcing layout. There's a mirror problem: writes that force layout every frame.

Animate an element's position with left and top:

box.style.left = x + "px"; // layout property
box.style.top = y + "px";

left and top are layout properties. Change them and the browser re-runs Layout — then Paint, then Composite — every frame of the animation. On many elements, it starts missing frames.

Animate with transform instead:

box.style.transform = `translate(${x}px, ${y}px)`;

transform touches neither Layout nor Paint. It goes straight to Composite — the GPU stage, off the main thread. The browser just moves a layer it already painted. Cheap.

Two properties. The same motion on screen. One re-runs the whole pipeline; the other skips to the last stage.

This is also why transition: all is a quiet trap. It opts every animatable property into transitions — including the layout ones. You meant to fade a color. You also signed up to animate margin, and now the browser runs Layout through the entire transition.

Name the properties you actually animate. Never all.

The twist: a smooth FPS counter can be lying

Here is where the mental model earns its keep.

I built a small lab for all of this — a grid of boxes, a checkbox per anti-pattern, and a live FPS meter. Tick "forced reflow loop" and the meter craters from 120 to 4. You can watch the frame time explode from 8 ms to 235 ms. The thrash is right there in the number.

Then I added a "paint bomb": heavy blurred shadows, re-painted every frame, across a thousand boxes. Genuinely expensive.

I ticked it, expecting the meter to crater again.

It didn't move. 120 FPS. 8.3 ms. Flat.

The animation was visibly stuttering — and the number said everything was fine.

Go back to the pipeline. Where does Paint run? The main thread records the paint, but the heavy part — rasterizing all that blur — happens on the compositor thread and the GPU. My FPS meter measures the main thread, by timing requestAnimationFrame. It is blind to compositor work by construction.

You might think a green FPS counter means a fast page.

It means a free main thread. That is not the same thing.

Layout thrashing shows up there, because Layout is on the main thread. Paint jank doesn't, because raster isn't. To see paint cost you need a different instrument — DevTools' rendering stats — not a rAF counter.

The blind spot is the lesson. Always know which thread your metric is watching.

When this isn't your problem

Reordering reads and writes is the fix for thrashing. But be honest about the ceiling.

If you're thrashing over ten thousand DOM nodes, the real problem isn't the ordering — it's that you have ten thousand nodes. The fix is to render fewer of them (virtualization), not to reflow all of them more politely.

And not every janky page is thrashing. If your reads and writes are already batched and it still stutters, you might be paint-bound, or running too much JavaScript, or animating the wrong property. Layout thrashing is one specific cause with one specific fix. Reach for it when you see reads and writes interleaved — not as a cure-all.

The whole thing, in three lines

The browser batches layout, so it computes geometry once per frame.

A read after a write forces it to compute now, in the middle of your loop.

Move every read above every write, and a thousand reflows collapse into one.

Cheat sheet: how layout thrashing happens

It always takes two ingredients in the same frame:

a write that dirties layout   +   a geometry read that needs it clean= the browser recomputes layout right now, to answer the read

Everything below is a variation on that one recipe.

The reads that force layout (asking any of these while layout is dirty triggers a reflow):

The shapes it takes, and the fix for each:

When it's not thrashing (don't over-correct):

The one rule under all of it: reads now, writes later — never interleaved.

I wanted a personal knowledge base that could actually answer questions across my notes, with citations, and I wanted it to run entirely on my laptop. No OpenAI key. No Anthropic key. No cloud database. Nothing about what I write leaving the machine.

That is what this walkthrough builds: gbrain wired so that a local model does the thinking, a local model does the remembering, and a file on disk does the storing.

Along the way I hit one trap that every "just use Ollama" guide gets wrong. I will show you the trap, because understanding why it fails is the fastest way to understand how the whole thing fits together.

First: what is gbrain, and why bother

gbrain is a searchable memory that answers questions about your own notes — and cites the note it used.

Not a note app. Not a generic chatbot. Something in the gap between them.

You feed it decisions, people, meetings, whole documents. Later you ask a question, and it finds the relevant notes and writes an answer built from your pages.

Why that gap matters: the two tools you already have both fail, in opposite ways.

Search gives you ten links and makes you do the reading.

A chatbot writes a confident answer, but it has never seen your notes — ask it "what did we decide about pricing?" and it guesses.

gbrain reads your actual pages and answers from them:

you:     "Why was the base tier price raised?"
gbrain:  "Raised from $19 to $29 because support costs per seat grew and the
          $19 tier was unprofitable below 50 seats [pricing-decision]."

The answer is built from your note, and it names the page, so you can trust it. And when it does not know, it says so — an honest "I don't have that" instead of a confident wrong answer.

So the reasons to run it: your context stops evaporating, you can ask across everything you have ever written at once, and — wired into a coding agent — it gives that agent a memory so it stops re-asking what it could look up. In this build, all of that stays on your laptop.

One caveat worth saying up front: an empty brain answers nothing, so day one feels broken. It earns its keep once capturing becomes a habit.

Everything below was built and verified on an Apple M5 with 32 GB. One running example carries through the whole post: a folder with two notes in it.

pricing-decision.md   "raise the base tier from $19 to $29 in July..."
hiring.md             "freeze backend hiring until the pricing change lands..."

By the end, I can ask "why was the price raised, and what hiring change followed?" and get a cited answer, with zero network calls.

The shape of the machine

Before any commands, here is the whole system on one screen. Follow the arrows.

your notes (markdown)
      ↓
Ollama · nomic-embed-text   turns each note into 768 numbers      (embeddings)
      ↓
PGLite · a file on disk     stores the notes and the numbers      (memory)
      ↓   ← your question comes in here, gets turned into numbers too
find the closest notes by comparing numbers
      ↓
Ollama · qwen2.5:14b        reads those notes, writes the answer   (generation)

Three actors. Ollama runs the models. PGLite holds everything. gbrain is the CLI that moves data between them.

Keep that picture. Every section below is just one arrow in it.

Two models, because there are two different jobs

The first thing that trips people up: you need two models, not one, and they are not interchangeable.

One model turns text into numbers.

You hand it a sentence. It hands back a list of 768 numbers.

That list is called an embedding. Similar sentences produce similar lists. That is the entire reason search will work later.

The other model turns notes into an answer.

You hand it your question plus the notes that matched. It writes prose back.

An embedding model cannot write you a paragraph. A chat model could produce an embedding, but it is huge and slow for a job that runs on every single note. So gbrain uses one of each:

ollama pull nomic-embed-text     # embeddings · 768 dims · ~274 MB
ollama pull qwen2.5:14b          # generation · ~9 GB

ollama pull downloads the weights to ~/.ollama/models once. After that they load from local disk, and nothing about them touches the network again.

An embedding is not an AI conversation. It is not something the chat model does. It is a separate, tiny, constant job — and it needs its own small model.

Ingestion: how a note becomes a memory

Here is the first arrow. It runs entirely on your machine.

gbrain import ~/notes
# Found 2 markdown files
# 2 pages imported, 2 chunks created

Walk through what "2 chunks created" actually means.

gbrain reads each file, splits it into chunks, and sends each chunk to Ollama to be embedded. Ollama returns 768 numbers per chunk. gbrain writes the chunk text and its 768 numbers into the database.

The naive version skips the chunking and embeds the whole file as one blob. That breaks the moment a file covers two topics: the pricing note and a paragraph about, say, office snacks would share one averaged vector, and neither would be findable on its own. Splitting first keeps each idea searchable.

Where does the database live? In a single folder:

~/.gbrain/brain.pglite

That is PGLite. It is Postgres, compiled to run inside gbrain instead of as a separate server you install and start. No port. No password. No postgres process. You could copy that folder to another Mac and it would just work.

So after ingestion:

• Where it ran: your machine. gbrain locally, Ollama locally.

• What got stored: the note text and its 768-number vector, in the PGLite file.

• What was computed once: the embeddings. They are not recomputed when you search.

• What it hands to the next step: a database full of (text, vector) rows, ready to be matched.

This is worth saying plainly: embedding happens at import time, once. Your question, later, is the only thing embedded fresh.

Asking: how a question finds its answer

Now the interesting arrow.

gbrain think "What did we decide about pricing and why?"

Four things happen, in order:

your question
   ↓  Ollama embeds it into 768 numbers        (fresh, every time)
   ↓  PGLite finds the note-vectors closest to it
   ↓  gbrain hands those notes + your question to qwen2.5:14b
   ↓  qwen writes an answer that cites the notes

The search step is pure math. "Closest" means the note vectors whose 768 numbers point in nearly the same direction as your question's 768 numbers. No AI runs during the search — it is comparing lists of numbers the embedding model already produced.

Only the last step is generation. The model does not know your notes; it is handed them, in the prompt, and told to answer using only those. That is why the answer can cite [pricing-decision]: the note was in front of it.

Here is the real answer it gave:

We decided to raise the base tier from $19 to $29/month starting July because support costs per seat increased and the $19 tier was unprofitable below 50 seats [pricing-decision]. Existing customers are grandfathered at $19 for 12 months.

Retrieval prepared the knowledge. Search found the knowledge. The model explained the knowledge. Now the only question left is: how do we make that last model local?

The trap: ollama: does not work for chat

This is the part every shortcut guide gets wrong, so slow down here.

Embeddings used ollama:nomic-embed-text. The obvious next move is to point generation at Ollama the same way:

gbrain config set chat_model ollama:qwen2.5:14b
gbrain think "Why was the price raised?"

And you get this:

(no LLM available — set ANTHROPIC_API_KEY or pass `client`)
Model: ollama:qwen2.5:14b | Pages: 2 | Warnings: NO_ANTHROPIC_API_KEY

Look closely, because the output is almost taunting you. It knows the model is ollama:qwen2.5:14b. It found 2 pages. Then it refused to write anything and asked for a cloud key.

The retrieval ran locally. The synthesis quit.

Why it quits: recipes and touchpoints

gbrain reaches every provider through a recipe — a small description of one provider: its URL, how it authenticates, and which jobs it can do.

Those jobs are called touchpoints. There are three: embedding, chat, reranker.

Here is the actual Ollama recipe shipped inside gbrain, trimmed to the point:

ollama recipe
  touchpoints:
    embedding: ✓   (nomic-embed-text, mxbai-embed-large, all-minilm)
    chat:      ✗   (not declared)

That is the whole explanation. The Ollama recipe declares embedding and nothing else. When you ask gbrain to chat with Ollama, it checks the recipe, finds no chat touchpoint, and silently falls back to its built-in default — a cloud model that needs a key.

This is not your fault, and it is not a bug you can config your way out of. It is a known, open gap: gbrain's own COMMUNITY_IDEAS.md lists "local-first chat parity" and "litellm proxy unusable for chat" as open issues. Today, Ollama is wired for embeddings only.

The fix: borrow a recipe that has chat

We need a recipe that does declare chat and that lets us change its URL to point at Ollama. Of the chat-capable recipes, the OpenAI-compatible ones can be repointed with an environment variable. openrouter is one.

The move:

openrouter recipe (has chat ✓, speaks the OpenAI API)
        ↓  change its base URL to...
http://localhost:11434/v1   (Ollama's OpenAI-compatible endpoint)

Ollama already speaks the OpenAI API. So gbrain believes it is calling OpenRouter, and the request actually lands on Ollama, on your laptop.

export OPENROUTER_BASE_URL=http://localhost:11434/v1   # send openrouter calls to Ollama
export OPENROUTER_API_KEY=ollama                       # dummy; Ollama ignores auth
gbrain config set chat_model     openrouter:qwen2.5:14b
gbrain config set models.default openrouter:qwen2.5:14b

The dummy key exists only because gbrain checks that a key is present before it will try. Ollama throws it away.

Two things I learned the hard way:

• The base URL must be an environment variable. gbrain config set provider_base_urls.openrouter ... looks like it works — config get even reads it back — but the gateway never reads that value. It reads OPENROUTER_BASE_URL from the environment. So it lives in a env.sh you source, not in the brain config.

• A cold LLM can starve the query embedding. The first question after a restart timed out its embedding at 6 seconds (Ollama was busy loading the 9 GB model) and returned an empty answer. Two exports fix it:

  export GBRAIN_QUERY_EMBED_TIMEOUT_MS=30000   # let the embed wait out the load
  export OLLAMA_KEEP_ALIVE=30m                  # keep the model warm
  

Proof: watch the traffic, do not trust the claim

"Fully local" is a claim. The only way to trust it is to watch every model call and confirm it landed on localhost.

So ask a real question and tail Ollama's log at the same time:

gbrain think "What did we decide about pricing and why?"

The answer comes back cited, Model: openrouter:qwen2.5:14b | Citations: 1. And in Ollama's own log, at that exact moment:

200   25ms    POST  /v1/embeddings         ← the question became a vector
200   10.6s   POST  /v1/chat/completions   ← qwen wrote the answer

Two requests. Both to 127.0.0.1. No Anthropic. No OpenAI. No key that reaches the internet. That log is the whole point of the exercise — it is the difference between "I think it is local" and "I watched it be local."

Making it survive a reboot

Everything so far works in this shell. Close the terminal and two things evaporate: the Ollama daemon (I started it by hand) and the four environment variables (they only live in a file called env.sh). A real install has to outlive the session. Two steps do that, and it is worth knowing what each one actually produces.

Step 1 — turn Ollama into a background service.

brew services start ollama

This does not just run Ollama. It writes a launchd agent — a small file at ~/Library/LaunchAgents/homebrew.mxcl.ollama.plist — and hands it to macOS. launchd is the thing that starts programs at login and restarts them if they die.

So what this produces is a daemon that is always there: it comes up when you log in, relaunches itself if it crashes, and writes its log to /opt/homebrew/var/log/ollama.log. You never type ollama serve again. brew services list shows it as started.

ollama   started   ~/Library/LaunchAgents/homebrew.mxcl.ollama.plist

Step 2 — load the environment in every shell.

The four exports live in env.sh. To make every new terminal pick them up, add one line to ~/.zshrc (the file your shell runs on startup):

source ~/Projects/gbrain-ollama/env.sh

What this produces is that OPENROUTER_BASE_URL, OPENROUTER_API_KEY, and the two timeout/keep-alive vars are set in every interactive shell you open. gbrain's CLI reads them from the environment, so gbrain think finds the local route without you sourcing anything by hand.

The test that proves both worked: open a brand-new terminal, type nothing but gbrain think "...", and watch /opt/homebrew/var/log/ollama.log. You should see two 127.0.0.1 requests and a cited answer.

One honest caveat. The launchd daemon starts with a clean environment — it does not read env.sh — so OLLAMA_KEEP_ALIVE from your shell does not reach it, and it falls back to unloading an idle model after a few minutes. That is exactly why GBRAIN_QUERY_EMBED_TIMEOUT_MS=30000 matters: it lets the query embedding wait out the occasional cold reload instead of timing out.

Now actually use it: locally and globally

You have a running brain. Here is how you talk to it.

Everything you do is one of two things: putting knowledge in, or getting it out.

   PUT IN                          GET OUT
   capture / import / put / sync   search / query / think
        ↓                                    ↑
              ~/.gbrain/brain.pglite

And one rule governs all of it: gbrain only saves what it is told to save. Nothing is captured automatically — not your terminal, not your chats. Text enters the brain when a write command runs, and only then.

Locally, from the terminal

Putting things in:

gbrain capture "Decided to run gbrain fully local: Ollama + PGLite, no cloud."
gbrain import ~/notes/                 # a folder of markdown you already have
gbrain sync --repo ~/code/myproject    # a whole git repo

Getting things out — and the three commands are not the same:

gbrain search "pricing"                    # ranked pages, keyword — fast, no LLM
gbrain query  "why was the price raised?"  # hybrid (meaning + keywords), ranked pages
gbrain think  "why was the price raised?"  # a written, cited answer — uses the LLM

The practical difference, which I checked by watching Ollama's log: search and query only ever hit /v1/embeddings — they hand you matching pages and need no chat model. think is the only one that calls qwen to write an answer. So most of your day-to-day retrieval does not even touch the generation model.

Globally, from your coding agent

This is where it stops being a CLI and becomes a memory your agent shares. Wire it in over MCP, globally:

claude mcp add gbrain -s user \
  -e OPENROUTER_BASE_URL=http://localhost:11434/v1 \
  -e OPENROUTER_API_KEY=ollama \
  -e GBRAIN_QUERY_EMBED_TIMEOUT_MS=30000 \
  -- gbrain serve

-s user is what makes it global. It writes the registration to ~/.claude.json, so the brain is available in every project on this machine — and because that file belongs to your Mac user, not your Anthropic login, it stays available no matter which Claude account you sign in with. Drop -s user and it registers for the current project only. (If the agent later reports it can't find gbrain, an MCP subprocess may not have your shell's PATH — swap gbrain serve for the absolute path from command -v gbrain.)

Those -e flags are not optional on a local setup, and the reason ties back to everything above. A coding agent launches gbrain serve with a clean environment — it does not read your ~/.zshrc. Skip the flags and brain search still works (embeddings need no env), but synthesis over MCP falls back to "no LLM available." The flags hand the local route straight to the subprocess.

Then teach the agent to use it. This is the step people skip — and without it the tools just sit there. The agent can see them, but it will not reach for them on its own; you would have to say "search my brain" every single time. The fix is a few lines in a CLAUDE.md file, which Claude Code reads at the start of every session.

Where you put the file decides how far it reaches:

• ~/.claude/CLAUDE.md — your user file, loaded in every project. Best for a personal brain, and it matches a user-scope MCP.

• ./CLAUDE.md in a repo — scoped to just that project.

Create the file if it does not exist, and paste in the protocol. Here is exactly how it looks:

## gbrain — brain-first protocol
You have a personal knowledge brain connected over MCP (tools: search, query, put_page). Use it:
1. Search first. Before answering about people, companies, decisions, projects, or past context, call search/query against the brain. If it has the answer, use it and cite the page — don't ask what you can retrieve.
2. Write back. When I make a decision or mention a new person/company/idea worth keeping, save it with put_page.
3. Cite. Name the page you used.

That is the whole thing — one markdown heading and three rules. On the next session start, Claude reads it and begins reaching for the brain on its own: searching before it asks you, writing decisions back as you work. (You can go further and add note-quality rules to the same file — that is the "Teaching Claude to write good notes" section below.) If you use gstack, two skills automate the whole thing: /setup-gbrain does the wire-up and writes that protocol for you, and /sync-gbrain indexes a code repo so gbrain search works semantically across the codebase.

And to close the loop on the obvious worry: connecting your agent does not record the conversation. The agent saves something only when it calls put_page — decisions and new ideas, nothing else. You stay in control of what your brain remembers.

Is it fast enough for coding?

The honest first reaction to a local setup is: I asked a question and waited ten seconds. Won't that wreck my flow?

It won't, and the reason is in the numbers. Measured on this brain:

Read that with one question: where does the time go?

Not the embedding. That is 3 milliseconds.

Not the database. That is instant.

The 8.6 seconds is entirely qwen2.5:14b writing the answer, a token at a time, on your laptop. A 14B local model is slower than a cloud one. That is the price of nothing leaving the machine.

But look at which command pays it. Only think.

And your coding agent does not call think.

When Claude uses the brain while you code, it calls search and query — it fetches your notes and reasons over them with its own fast model. It does not ask the local 14B to write prose. Fetching is sub-second.

It is even faster than the table looks. Most of that 0.6–1.1s is the gbrain CLI booting a fresh process. Over MCP, gbrain serve stays resident, so that cost is paid once at session start, not per call. The agent's real lookups are an embedding plus a database query — tens of milliseconds.

So the picture during development: a brain lookup returns in well under a second, a rounding error next to the agent's own thinking time. The eight-second wait only happens when you deliberately ask the brain to compose an answer with think — not something the agent's loop does.

If you want think snappier too, three levers:

• A smaller chat model. gbrain config set chat_model openrouter:qwen2.5:7b (after ollama pull qwen2.5:7b) roughly halves generation time, for slightly weaker answers.

• Keep it warm. The cold runs earlier were 17–21s because qwen had to load into memory first; warm runs are 7–9s. An idle model unloads after a few minutes, so the first think after a break pays that reload. It never touches search/query.

• Cloud chat for think only. Point just the chat model at a real key if you want fast, high-quality synthesis and don't mind that one step leaving the machine. Retrieval stays fully local.

Want a better brain, for free?

qwen2.5:14b is fine, but it is not Claude. So the obvious wish: can I just use the good model I am already using in Claude Code, without a separate API key?

Directly, no — for two concrete reasons.

gbrain never asks the host to think for it. Even the think tool, called over MCP, runs gbrain's own configured model, not Claude Code's.

And a Claude subscription is not a reusable API key. gbrain's anthropic recipe wants a standalone ANTHROPIC_API_KEY, which is pay-per-token — a separate, paid thing, not the login you already have.

But the outcome you want is free, and it is what the MCP wiring is actually for. You flip who does the writing.

gbrain think makes gbrain synthesize, on the slow local model. Instead, let Claude Code be the reasoning layer:

gbrain retrieves   — search / query, local, fast, free
Claude Code writes — with the frontier model you already use

So inside Claude Code you say "search my brain for X and explain it." Claude calls the fast search tool, pulls your pages into its own context, and writes the answer itself — Claude quality, no extra key, no local 14B. That is "use the Claude Code model over my brain, for free," and the brain-first protocol already nudges the agent toward search/query instead of the local think.

The honest caveat: those retrieved snippets go into Claude Code's context, so that one step reaches Anthropic. No new key, the same trust boundary you already accepted by using Claude Code — but not "nothing leaves the machine." Storage and retrieval stay local.

And if you want gbrain's own think better, with no host in the loop:

• Better, still fully local, still free — a bigger local model. gpt-oss:20b (13GB) sits comfortably on 32GB and beats 14B; qwen2.5:32b (20GB) is higher quality still, but tight on RAM and slower.

  ollama pull gpt-oss:20b
  gbrain config set chat_model     openrouter:gpt-oss:20b
  gbrain config set models.default openrouter:gpt-oss:20b
  

• Better, free, but not local — a free cloud tier. Google Gemini (free key from AI Studio) or Groq (free tier); gbrain ships google and groq recipes. Higher quality than qwen, free within rate limits, but your notes go to that provider. Grab a key and set chat_model to google:<model> or groq:<model> (exact names via gbrain providers env google).

For coding, the best option is not in that last list at all — it is the one from a few paragraphs up: let Claude Code do the writing. You are already sitting inside the strong model, so let gbrain fetch and let Claude synthesize. Best quality, no extra key. That is what connecting the brain was for in the first place — the brain is Claude's memory to look things up in, not a second, weaker model trying to answer in its place.

How Claude actually talks to your brain

This is the part that surprised me, and it changes how you use the whole thing.

You do not run gbrain think.

You do not even type the word gbrain.

You ask Claude a question in plain English — "when's my birthday?" — and it answers from your notes. No command. So how does that work?

Claude learns your brain exists, at startup

When you ran claude mcp add gbrain … -- gbrain serve, you did not just save a line of config. You told Claude Code: "there is a tool server here — launch it and ask it what it can do."

So every time Claude Code starts, it:

1. Spawns gbrain serve as a background subprocess.

2. Asks it, over MCP, "what tools do you offer?"

3. Gets back a list — 92 of them: search, query, put_page, get_page, and so on, each with a one-line description of what it does.

Now your brain is just there, in Claude's toolbox, the same way it knows it can read a file or run a command.

What happens when you ask

Watch the birthday question flow through:

you (in Claude Code):  "when's my birthday?"
      ↓
Claude decides a brain lookup would help, and calls a tool:
      search({ query: "birthday" })              ← Claude → gbrain
      ↓
gbrain runs it locally: embed the query (Ollama, ~3ms), search PGLite
      ↓
      returns the page text: "on 20th of june… born 1989"   ← gbrain → Claude
      ↓
Claude reads that in its own context and writes:
      "June 20th; born 1989, so you'll turn 37 — in 14 days."

Two things to notice.

Claude chose to call the tool. You did not tell it to. It saw a question about you, remembered it has a brain tool, and reached for it — partly because the tool's description says it searches your knowledge, and partly because the brain-first protocol you pasted into CLAUDE.md tells it to look there first.

gbrain never wrote a sentence. It embedded, searched, and handed back a raw page. The reasoning — the age math, "in 14 days" — is Claude's. gbrain was the memory; Claude was the mind.

Why it works: MCP is just a tool protocol

There is no magic here. MCP (Model Context Protocol) is a standard way for a program to expose tools to an LLM. gbrain speaks it; Claude Code speaks it. When they connect, Claude gets a menu of gbrain's tools with descriptions, and from then on it can call any of them, read the result, and continue — the same tool-use loop it runs for reading files or executing shell commands.

So "talking to your brain" is not a special mode. Your brain simply became one more tool Claude picks up when a question calls for it.

The practical upshot

Stop thinking in gbrain commands. Once it is wired in:

• Ask Claude questions in plain language; it retrieves and reasons for you.

• Tell it to remember things — "note that we decided X" — and it calls put_page.

• The terminal gbrain commands are still there for when Claude is not in the loop.

You built a filing cabinet and handed Claude the key.

What actually makes the answers better

Once Claude is the one writing, the chat model stops being the lever. Two other things take over, and it is worth being clear about which.

Your notes are the first lever, and the biggest by far.

Retrieval can only surface what you actually wrote down. A thin brain gives thin answers, no matter how strong the models are. So the highest-return habit is simply capturing — a decision here, a fact there — until the pages exist to be found. A fuller brain beats a fancier model every time.

Retrieval quality is the second lever — and that is the embedding model, not the chat model.

Keep the split in mind: search/query find pages with nomic-embed-text, and that choice decides which pages Claude ever sees. If the right note is not in the results, Claude cannot use it — it never reached the prompt. So making Claude smarter about your brain is really about helping it find the right page, which is the embedding model's job.

nomic-embed-text (768 dimensions) is a solid default. When you outgrow it, the usual step up is bge-m3 (1024 dimensions): multilingual, and noticeably better at pulling the right page out of a large or messy brain. (mxbai-embed-large is a middle option; bge-m3 is the one worth knowing.)

The catch: this is not a config toggle. The vector width — 768 for nomic — is baked into the database column at init time, so moving to a 1024-dimension model means wiping and re-embedding. Export first so nothing is lost:

gbrain export --dir ~/brain-backup     # save your pages as markdown
ollama pull bge-m3
mv ~/.gbrain/brain.pglite ~/.gbrain/brain.pglite.bak
gbrain init --pglite --embedding-model ollama:bge-m3 --embedding-dimensions 1024
gbrain import ~/brain-backup           # re-embed every page with the new model

When is it worth it? When you have thousands of notes, or you write in more than one language, or search starts missing things you know are in there. Not on day one — on day one, nomic-embed-text plus the habit of capturing is the whole game.

(gbrain can also add a reranker — a second pass that re-orders the top hits for precision — but that is a later refinement. The embedding model and your notes are where the real gains live.)

Teaching Claude to write good notes

If notes are the biggest lever, the obvious next question is: will Claude write good ones on its own?

Partly. Left to the three-line brain-first protocol, it will save things — but the quality drifts. It might dump a whole conversation into one page, invent inconsistent slugs, or forget to link anything. Serviceable, not great. It does not magically write clean notes just because the tool is there.

Three things shape note quality — one you add, two gbrain already does.

What you add: note conventions in CLAUDE.md.

The same file that tells Claude to search first can tell it how to write. Extend the protocol:

## Writing to the brain
When you save a page with put_page:
- One idea per page. Capture the specific decision or fact and the *why* — not a
  whole conversation.
- Namespace the slug by kind: people/<name>, companies/<name>, decisions/<slug>,
  notes/<slug>.
- Set an accurate type (note, person, company, decision) so it fits the schema.
- Link related pages with [[slug]] — a decision links to the people and projects it
  touches.
- Search first and update an existing page instead of creating a duplicate.
- Keep my exact wording for decisions and quotes; don't paraphrase away the specifics.

Now the agent files things the way you would, and the brain stays navigable instead of turning into a pile.

What gbrain already does #1: it publishes its own filing rules.

Because mcp.publish_skills is on, gbrain exposes list_skills / get_skill over MCP — the agent can ask the brain how it wants pages filed, and gbrain answers with its schema conventions. So you are mostly reinforcing habits the brain already advertises, not inventing them from scratch.

What gbrain already does #2: it improves the notes over time, on its own.

gbrain has an overnight maintenance pass — the "dream cycle" (gbrain dream once, or gbrain autopilot --install to run it continuously). It dedupes people pages, fixes broken citations, and wires up links you never made by hand. So even notes you wrote sloppily get tidied while you sleep. This one does use the chat model — so the dream cycle is exactly where a local qwen, or a bigger model, actually earns its keep (unlike live retrieval, which never touches it).

So the recipe for good notes: add a few conventions to CLAUDE.md for quality as they are written, lean on gbrain's published rules, and let the dream cycle polish the pile over time.

The honest tradeoffs

Real understanding includes the limits, so here they are.

The simple path is not this one. The sanctioned, fewer-moving-parts setup is local embeddings + cloud synthesis: keep nomic-embed-text local and let a real Anthropic or OpenAI key write the answer. It is less setup and the prose is better. I chose full-local on purpose, and paid for it with a workaround and a smaller model.

Local retrieval quality is lower. nomic-embed-text is good, but a hosted embedding model will find relevant notes more reliably on a large, messy brain. For a few thousand notes you will not notice. For a hundred thousand, you might.

You are borrowing the OpenRouter namespace machine-wide. Putting OPENROUTER_BASE_URL and OPENROUTER_API_KEY=ollama in ~/.zshrc sets them for every program in every shell. The day you install something that genuinely uses OpenRouter, it will quietly point at your Ollama with a dummy key and fail in a confusing way. If that day comes, move these exports out of ~/.zshrc and source env.sh only when you use gbrain.

Agentic features run hot. Pointing models.default at a local model means gbrain's agent loops (gbrain agent run, autopilot) run on qwen, which has no prompt caching — so long loops cost more time. gbrain doctor warns about exactly this and suggests keeping just that one tier on a cloud model. For plain think over your notes, it never comes up.

When should you not do this? If you have an API key you are comfortable using, and you value answer quality over privacy, go local-embeddings-plus-cloud-chat and skip the openrouter dance entirely. Full-local is for when "nothing leaves the machine" is the actual requirement, not a nice-to-have.

The whole thing in three beats

If you remember nothing else, remember the shape:

Ingestion turns your notes into vectors, once, on your machine.
PGLite finds the right vectors when you ask, on your machine.
A local model reads them and writes the answer, on your machine.

One embedding model to remember. One chat model to explain. One file to hold it all. And a log full of 127.0.0.1 to prove it.

Appendix: every command

The whole surface of gbrain 0.42.53.0, grouped the way gbrain --help groups it. Prefix each with gbrain; run gbrain <command> --help for details. The ones you actually reach for day-to-day are marked ★.

Setup & health

init [--pglite|--supabase|--url]      create a brain (PGLite = local, no server)
migrate --to <supabase|pglite>        move a brain between engines
upgrade                               self-update gbrain
check-update [--json]                 check for a new version
doctor [--json] [--fast]           ★  health check (embeddings, pgvector, skills…)
integrations [subcommand]             manage integration recipes (senses + reflexes)

Get content in

capture [content] [--file P] [--stdin]  ★ single entrypoint to add content (→ inbox/)
import <dir> [--no-embed]               ★ bulk-import a markdown directory
put <slug> [< file.md]                    write/update one page
sync [--repo P] [--watch] [--install-cron] git repo → brain, incremental
embed [<slug>|--all|--stale]              (re)generate embeddings

Read, search, ask

search <query>                     ★  keyword search — ranked pages, no LLM
query <question> [--no-expand]     ★  hybrid search, meaning + keywords (alias: ask)
think <question>                   ★  synthesized, cited answer (uses the LLM)
get <slug>                            read one page
list [--type T] [--tag T] [-n N]      list pages

Pages & versions

delete <slug>                         delete a page
history <slug>                        page version history
revert <slug> <version-id>            revert to a prior version

The graph: links, tags, timeline

link <from> <to> [--link-type T]      create a typed link (alias: link-add)
unlink <from> <to>                    remove a link (alias: link-rm)
link-sources                          list link provenances + edge counts
backlinks <slug>                      incoming links
graph <slug> [--depth N]              traverse the link graph
graph-query <slug> [--type T] [--direction in|out|both]  edge-filtered traversal
tags <slug> / tag <slug> <t> / untag <slug> <t>          list / add / remove tags
timeline [<slug>]                     view timeline
timeline-add <slug> <date> <text>     add a timeline entry

Ideate (brainstorming over your brain)

brainstorm <question> [--json]        bisociation idea generator (hybrid + far-set + judge)
lsd <question> [--json]               Lateral Synaptic Drift — far-from-obvious ideas

Code indexing (for a synced codebase)

code-def <symbol> [--lang l]          find a symbol's definition
code-refs <symbol> [--lang l]         find references to a symbol
code-callers <symbol>                 who calls this symbol
code-callees <symbol>                 what this symbol calls
query <q> --lang <l> | --symbol-kind <k>  filter hybrid search by language / symbol type
reconcile-links [--dry-run]           recompute doc↔impl edges
reindex-code [--source id] [--yes]    reindex code pages
sync --strategy code                  sync code files into the brain

Multiple sources / repos

sources list                          show registered sources
sources add <id> --path <p>           register a source
sources remove <id>                   remove a source + its pages
sync --all | --source <id>            sync all sources / one source

Export & files

export [--dir ./out/]                 export the brain to markdown
files list [slug]                     list stored files
files upload <file> --page <slug>     attach a file to a page
files upload-raw <file> --page <s>    smart upload (size routing + redirect)
files signed-url <path>               1-hour signed URL
files sync <dir> / files verify       bulk upload / verify uploads

Maintenance & tools

extract <links|timeline|all>          extract links/timeline (idempotent)
lint <dir|file> [--fix]               catch LLM artifacts, bad frontmatter, placeholder dates
orphans [--json] [--count]            pages with no inbound links
check-backlinks <check|fix> [dir]     find/fix missing backlinks
salience [--days N] [--kind P]        pages ranked by emotional + activity salience
anomalies [--since D] [--sigma N]     cohort-based statistical anomalies
transcripts recent [--days N]         recent raw local transcripts
dream [--dry-run] [--json]            run the overnight maintenance cycle once
publish <page.md> [--password]        shareable HTML (strips private data, optional AES-256)
report --type <name> --content ...    save a timestamped report to the brain
check-resolvable [--json] [--fix]     validate the skill tree (reachability/MECE/DRY)

Background jobs (Minions — Postgres/Supabase only)

jobs submit <name> [--params JSON]    submit a background job [--follow]
jobs list | get <id> | cancel <id> | retry <id>   manage jobs
jobs prune [--older-than 30d] | stats | work      clean / dashboard / worker daemon

Serve & connect an agent

serve                              ★  MCP server over stdio (what `claude mcp add` runs)
serve --http [--port N]               HTTP MCP server with OAuth 2.1
connect <mcp-url> --token <t> [--install]  wire this machine to a remote gbrain
watch [--json]                        pipe conversation turns in, stream brain pages out
call <tool> '<json>'                  raw tool invocation
--tools-json                          tool discovery (JSON)

Admin

stats                                 brain statistics
health                                brain health dashboard
features [--json] [--auto-fix]        scan usage, recommend unused features
autopilot [--repo] [--interval N]     self-maintaining brain daemon
config [show|get|set] <key> [val]  ★  brain config (e.g. config set chat_model …)
storage status [--json]               storage tier status and health
version                               version info

think is the odd one out: it is not printed in gbrain --help's summary, but it is real and it is the command that writes a synthesized, cited answer. search, query, and ask return ranked pages; think writes the prose.

One walkthrough of the whole real-time layer of a collaborative document editor — the merge, the transport, the editor, and the persistence — built by hand on Y.js to actually understand it. Code references are real; every worked CRDT example below was run through Yjs before it was written down.

Two people open the same document.

They both start typing into the same paragraph, at the same instant.

Neither has seen the other's keystrokes yet.

And a second later, both screens show the exact same text — with nobody's edit lost, and no central server deciding who won.

That last clause is the hard part.

No referee.

The usual way this gets explained is a pile of vocabulary:

CRDT. Operational transform. State vector. Tombstone. WebSocket upgrade. Snapshot. Compaction.

Which is not an explanation. It is a list of things you are now supposed to understand.

I built the whole real-time layer of a collaborative review tool by hand — on Yjs for the merge itself, but the transport, the reconnect, and the persistence all hand-rolled — specifically so I could explain every piece instead of importing a black box. So here is the box, opened.

One example runs through the entire post. Mara and Theo, editing a document called the Q3 strategy memo. Mara is at the top fixing a heading. Theo is three paragraphs down rewriting a sentence. Keep them in mind; every part below is really about them.

The shape of the whole thing, before we start:

keystroke → a CRDT update → over a WebSocket → to a relay → to everyone else → and onto disk

We will go in dependency order: first how edits merge, then how they travel, then how the editor hooks in, then how they are saved.

Part 1 — The merge: how two keystrokes don't clobber

Start with the naive model, because it is the one everyone reaches for, and watch it break.

Model the document as a string. An edit names a position:

"insert 'X' at position 5"
"delete position 3"

Now run Mara and Theo concurrently. The document is base. Neither has seen the other:

Mara:  insert 'X' at position 0
Theo:  delete position 2          (he means the 's')

Apply Mara's, then Theo's:

"base"  --insert 'X' at 0--> "Xbase"
"Xbase" --delete position 2--> "Xbse" ← deleted 'a', not 's'

Theo meant to delete the s — index 2 in base. But Mara's insert shifted every position right by one, so in Xbase index 2 is now a. His delete hit the wrong character.

A position is not stable. It means one thing before a concurrent edit and another thing after.

Operational Transformation (OT) patches this by transforming one operation against the other — "Theo's index 2 has to become 3, because Mara inserted to its left." It works, and it is notoriously hard to get right; every pair of operation types needs its own transform.

A CRDT — Conflict-free Replicated Data Type — takes the other road. It builds positions that never shift.

Give every character a permanent name

Stop saying "position 5." Say "the character between this one and that one." If characters have permanent names, "between X and Y" means the same thing forever.

Three pieces make that work. This is YATA (Yet Another Transformation Approach), the algorithm inside Yjs:

1. Every character is an item with a permanent id — a pair (client, clock). The client is unique per editor (Mara is 1, Theo is 2). The clock ticks up as that client creates items. Mara's first character is (1,0), her next (1,1). The id never changes.

2. The document is a doubly-linked list of items, not a string. To read the text you walk the list and concatenate. No character stores a position number — a character's position is simply where it falls as you walk the list.

(start) ⇄ [ H (1,0) ] ⇄ [ i (1,1) ] ⇄ (end)
          each item = { id, origin, content, deleted }

3. Each item remembers its origin — the id of the item to its left at the moment it was inserted. That is its anchor: "I went in right after (1,0)." Because ids never change, that anchor is permanent.

Inserting is now just: make an item, point its origin at the left neighbor's id, splice it in. No offsets exist, so nothing can shift out from under anyone.

The hard case, and the tiebreak that saves it

Empty document. Mara and Theo both type at the very start, same instant:

Mara inserts 'A'  →  (1,0), origin = (start)
Theo inserts 'B'  →  (2,0), origin = (start)

Both items have the same origin. Both want to be first. Every replica has to pick an order — and it must be the same order everywhere, or Mara sees AB, Theo sees BA, and they have diverged for good.

YATA's rule: items sharing an origin are ordered by client id — the same comparison on every machine. Lower id to the left. (The full rule has a few more cases, for inserts whose origins interleave, but they all resolve the same way: a fixed comparison on immutable ids, so every replica agrees.)

So (1,0) 'A' lands left of (2,0) 'B', on every replica, because the client ids ride inside the items themselves. I ran exactly this through Yjs:

apply Mara's update then Theo's  →  "AB"
apply Theo's update then Mara's  →  "AB"     (same answer, either order)

Both edits survived. The tiebreak only chose their order, and chose it identically for everyone. That is the whole trick: a conflict becomes a deterministic ordering, never a winner and a loser.

If the two inserts had different origins, there is nothing to resolve. Theo inserts X before b (origin = the start), Mara inserts Y after e (origin = e), and the merge is just "XbaseY" (verified) — both survive, because they were never competing for the same anchor.

And notice what is not happening: Theo has the higher client id, yet his X still lands on the left. No client-id comparison runs here at all — the order falls straight out of the origins. The tiebreak from the last section only fires when two items fight over the same origin. This is Mara's heading and Theo's sentence: different places, no conflict.

Deletes leave a tombstone

One subtlety keeps it all consistent. If you truly spliced a deleted character out of the list, any item anchored to it would lose its origin. So you never remove — you mark the item deleted, a tombstone, and skip it when rendering.

Document is XY. Mara deletes X; Theo, concurrently, inserts Z right after X:

[ X (deleted) ] ⇄ [ Z ] ⇄ [ Y ]
   skip            keep    keep
render → "ZY"     (verified)

X is invisible but still present as an anchor, so Z (origin X) lands exactly where X was. The tombstone keeps the map intact.

Why it always converges

Three facts together are the entire guarantee:

• ids are immutable,

• origins point at ids, so they are stable,

• the tiebreak is a fixed function of immutable data (client ids).

So applying the same updates in any order builds the same list, which renders the same text. "Yjs updates are commutative" is not magic — it is every decision is a fixed function of data every replica already has.

That commutativity is the property everything else in this post leans on.

Part 2 — Getting every keystroke to everyone (the sync layer)

Yjs gives us mergeable updates. It does not move them between people. That is the part I hand-built.

When Mara types, Yjs emits a small binary update — the diff of that one change, encoding where it belongs logically (an item with an origin), not "characters 40–58." Our job is only to get every update to every participant, exactly enough times.

The flow of one keystroke, watching where each step runs:

Mara's browser (client)
↓  Yjs emits an update
send it over the WebSocket            (client → server)
↓
server applies it to the room's Y.Doc  (server, in memory)
↓
├─► append the update to Postgres       (durable immediately — Part 4)
└─► broadcast to every OTHER client      (server → Theo, NOT back to Mara)
↓
Theo's browser applies it, and sees Mara's edit

The server is a relay with a memory. It is not a participant and has no opinion about the document. It holds the current Y.Doc for one document in RAM so it can answer "what's the state?", it forwards updates between the people connected, and it writes every update down. One document's relay is a room, keyed by document id — everyone editing that document shares one room, one in-memory Y.Doc, one set of connections.

How a newcomer catches up

When Theo opens the memo, his browser's Y.Doc is empty and the server's is full — and Theo might also have offline edits the server has never seen. So catch-up has to go both ways.

This is what state vectors are for. A state vector is a compact summary of "how much of each author's changes I already have" — not the content, just the watermarks: a map of client → the highest clock I've seen from them. (Now you can see why Part 1's ids matter — the ids are the bookmarks.)

The exchange is two steps, each direction:

Theo connects
↓
server → Theo:  SyncStep1 = "here is my state vector"
↓
Theo → server:  SyncStep2 = "based on yours, here's everything you're missing from me"
              + SyncStep1 = "and here is MY state vector"
↓
server → Theo:  SyncStep2 = "based on yours, here's everything you're missing from me"

After that both sides have everything, and they stay in sync by streaming updates as they happen. In code it is almost anticlimactic, because Yjs's y-protocols does the vector math — the server just sends the opening message the instant a connection joins:

// src/sync/doc-room.ts — addConnection
conn.send(encodeSyncStep1(doc))   // "here is my state vector; tell me what you lack"

The beautiful part: reconnect is not a special case. A returning client is just a newcomer whose Y.Doc happens to be half-full. SyncStep1/Step2 figures out the difference and replays exactly the missing updates. There is no separate "reconnect" code path — reconnect is the handshake.

The bug every hand-built relay has: the echo

Here is the trap. The server receives Mara's update and broadcasts it to everyone in the room. If "everyone" includes Mara, she receives her own edit back, applies it, re-emits it, sends it again... an echo, then a loop.

The fix is the self-echo guard: broadcast to everyone except the connection the update came from. The mechanism is Yjs's transaction origin — when the server applies Mara's update it tags the transaction with Mara's connection as the origin, and the room's single update handler fans out to everyone but that origin:

// src/sync/doc-room.ts
doc.on('update', (update, origin) => {
  broadcast(encodeUpdate(update), senderOf(origin))   // senderOf(origin) is skipped
  void appendUpdate(documentId, update)               // and persisted
})

There is a second, quieter guard Yjs gives you for free, and it matters most on reconnect. When Theo comes back and his SyncStep2 replays updates the server already has, applying them changes nothing — and Yjs only fires the update event for changes that actually mutate the doc. So a redundant replay triggers neither a re-broadcast nor a duplicate row in the log. Convergence stays correct and persistence stays clean with zero lines of dedup code.

Why it is all bytes (lib0, and why we encode and decode)

How does an update actually get onto the socket? Three words in the code look like noise — lib0, encoder, varUint — and they are exactly the answer. Start with the data.

Mara types one character. Yjs hands us an update. It is not text. It is a Uint8Array — a handful of raw bytes:

[ 0x01 0x01 0x84 … ]   // a Yjs update; the exact bytes are Yjs's business, not ours

And the same socket also has to carry presence — who is in the room, where each cursor is. Two kinds of traffic, one socket.

The bad version is JSON. The reflex is ws.send(JSON.stringify({ type: 'sync', update })). It breaks on the first line: the update is binary, and JSON cannot hold raw bytes. You would have to convert it — base64 (bigger, and still opaque text you decode again), or a JSON array of numbers like [1, 1, 132, 47, 120, 1]. A 6-byte update just became ~50 bytes of text, on every keystroke, and the receiver has to parse it back into bytes before Yjs can even look at it. You pay to turn bytes into text, send more of it, then turn it back into bytes. For nothing. So we stay in bytes the whole way.

lib0 is the toolkit Yjs is built on. The binary primitives live there — a growable byte buffer, variable-length integers, the matching read/write pair. Yjs uses them internally, and so does y-protocols, the helper that gives us writeSyncStep1, writeUpdate, and readSyncMessage. That last fact is the whole reason we use lib0 and nothing else: the update we are wrapping was produced by lib0, and the helpers that read it expect a lib0 buffer. Pick any other serializer and you cannot compose with the very functions doing the sync.

An encoder is a buffer you append into. You write typed values in order, then ask for the finished bytes; a decoder is the exact mirror, reading them back in the same order they were written:

createEncoder()              start an empty buffer
writeVarUint(enc, n)         append a small number
writeVarUint8Array(enc, b)   append a chunk of bytes, length-prefixed
toUint8Array(enc)            the finished buffer to send

That ordering is the protocol: write tag, then payload; read tag, then payload. Read them out of order and you get garbage, with no error to tell you. A varUint (variable-length unsigned integer) costs one byte for a small number — our channel tag is 0 or 1, where a fixed 32-bit int would spend four to say "0." It is the same encoding Yjs uses internally, so it is already the dialect the buffer speaks.

The framing, end to end:

// src/sync/sync-protocol.ts
export const encodeUpdate = (update: Uint8Array): Uint8Array => {
  const encoder = encoding.createEncoder()
  encoding.writeVarUint(encoder, SYNC_MESSAGE.sync)   // 1 byte: "channel 0 — a document update"
  writeUpdate(encoder, update)                         // append the payload, length-prefixed
  return encoding.toUint8Array(encoder)
}

Mara's update:   « a few opaque bytes from Yjs »
↓  writeVarUint(sync)   prepends the tag byte 0x00
↓  writeUpdate          appends the payload, length-prefixed
framed message:  [ 0x00 | «length» | « the update bytes » ]

The other end reads the tag first, and the tag decides who handles the rest:

tag 0  →  sync       the document CRDT (the handshake + ongoing updates)
tag 1  →  awareness  ephemeral presence (who's here, cursor positions)

That one byte is how a single socket multiplexes both channels — the cheapest multiplexer there is. A wrong byte is a silent disaster (presence bytes parsed as a document update), so the tag values live in exactly one place both ends import, never as bare 0/1:

// src/sync/sync-protocol.ts
export const SYNC_MESSAGE = { sync: 0, awareness: 1 } as const

The price of going binary is debuggability — you cannot eyeball a byte buffer in the network tab. We pay it because the payload is already binary and we must interoperate with y-protocols; JSON would only add cost and a translation layer.

The door: authorize before the socket opens

A WebSocket starts life as a normal HTTP request asking to "upgrade" to a socket. That request carries cookies — so authorization happens there, before any socket exists:

// src/sync/ws-routes.ts — preValidation, before the upgrade completes
const active = rawToken === undefined ? null : await getSessionUser(rawToken)
if (active === null) return reply.code(401).send({ error: 'not_authenticated' })
const document = await getDocumentForOwner({ documentId, ownerId: active.userId })
if (document === null) return reply.code(404).send({ error: 'document_not_found' })

An unauthorized client never gets a live connection — just a plain HTTP error on the upgrade. And it is the same owner check the REST routes use, so "can read this over REST" and "can join its live editing room" can never drift apart.

That 404, not 403, is deliberate. A 403 ("forbidden") would confirm the document exists — a stranger now knows there is a real document at that id, they just can't have it. A 404 says nothing: yours or imaginary, you get the same answer. The endpoint is not an existence oracle. (For now the authorization rule is the simplest possible one — you can see a document only if you own it. Teams and roles come later.)

What is honest about this, and what is deferred

It is single-instance. One server process holds each room. Run two processes behind a load balancer and Mara could land on process A, Theo on process B, and B's room never hears A's updates. The fix is cross-instance fan-out: publish each update to a Redis channel and have every process relay what it hears, ignoring its own echoes by an instance id. It slots into the exact same doc.on('update') handler — which is why the persistence and broadcast already live there.

The durable write is fire-and-forget. The update is broadcast and appendUpdate is issued, but not awaited before the client moves on. The data-loss window is tiny, not zero — and, as Part 4 shows, it is exactly this fire-and-forget concurrency that makes compaction's bookkeeping subtle. (One operational gotcha worth recording: on Bun, app.close() waits forever on an open WebSocket, so graceful shutdown has to close the sockets first.)

Part 3 — From keystroke to pixels (the browser)

That covered the server. The browser is the other end — what happens when Mara presses a key, and how three pieces hand work to each other. Keep them separate; it is easy to blur them:

TipTap        the editor UI — toolbar behavior, what a heading is, where the cursor blinks
Yjs           the CRDT — the mergeable document state, emits an "update" on every change
our provider  the network — carries Yjs updates over the WebSocket to the server and back

TipTap does not know about the network. Yjs does not know about the network. The provider is the only piece that touches the socket. That separation is the whole design: swap the provider and nothing else changes.

The glue between TipTap and Yjs is one TipTap extension, Collaboration. It replaces TipTap's normal "store the document in the editor" with "store the document in this Y.Doc." After that, every edit Mara makes is a Yjs update, automatically:

// web/src/editor/CollaborativeEditor.tsx
extensions: [
  ...documentExtensions,
  Collaboration.configure({ document: doc }),                        // editor edits → Y.Doc updates
  CollaborationCaret.configure({ provider, user: { name, color } }), // others' cursors from awareness
]

The flow of one keystroke, in the browser:

Mara types a character (TipTap)
↓
Collaboration writes it into the Y.Doc
↓
the Y.Doc emits an "update" event
↓
our provider's doc.on('update') fires → encode as a 'sync' message, ws.send → (server)
... meanwhile, a message arrives from the server ...
ws.onmessage → readSyncMessage applies it to the Y.Doc (origin = remote)
↓
Collaboration sees the Y.Doc change and re-renders the editor
↓
Theo's edit appears in Mara's editor

Notice Mara never re-sends what she just received. The provider applies remote updates with a sentinel origin and skips sending anything tagged with it — the same self-echo guard as the server, on the client side:

// web/src/editor/sync-provider.ts
const remoteOrigin = { remote: true }
const onDocUpdate = (update, origin) => {
  if (origin === remoteOrigin) return   // came FROM the server — don't bounce it back
  send(encodeUpdate(update))            // a genuine local edit — send it up
}

Why the schema has to be shared

Here is the fact that makes this section click, because the name "extensions" hides it.

A schema is the grammar of a document: which node types may exist (paragraph, heading, bulletList, codeBlock…), which marks (bold, italic…), and how they nest. A document containing a node the schema doesn't define is invalid — there is nowhere to put it.

In raw ProseMirror you write that grammar by hand. In TipTap you do not: TipTap derives the schema from your list of extensions. The array is the schema:

// web/src/editor/document-extensions.ts
export const documentExtensions = [StarterKit.configure({ undoRedo: false })]

Each extension contributes a piece of the grammar — Heading declares the heading node, Bold declares the bold mark; StarterKit bundles ~15 of them. TipTap runs getSchema(documentExtensions) and assembles one concrete schema. Same list in, byte-identical schema out. (undoRedo: false is the exception that proves the rule — undo/redo is a behavior plugin, adding no node or mark type, so it doesn't touch the schema. We turn it off because Yjs owns undo for a shared document.)

Now the trap. A Yjs update names node types by their schema name:

update says:  insert a node of type "heading", level 2, containing the text "Budget"

To apply that, the receiving client hands it to the ProseMirror↔Yjs binding, which looks up "heading" in its own schema to rebuild the node. So picture two clients with different extension lists:

Client A's extensions include Heading  →  A's schema knows "heading"
Client B's extensions do not            →  B's schema has no "heading"

Mara inserts a heading on A. The update reaches B. The binding on B tries to build a "heading" node, finds nothing in its schema, and cannot map it — it throws or drops the content. Desync. The two ends never disagreed about the text; they disagreed about which node types exist.

So the extension list lives in one shared module with no React and no networking — the single source of truth for "what a document may contain." Every browser editor imports it (so all clients decode each other's updates identically), and a server-side import pipeline imports it too (it builds documents without a browser, and must seed them with the exact same schema, or the first client to open one receives nodes its schema can't read). It must be shared, never copied — the day someone adds an extension to one copy and not the other, documents silently break.

Reconnect: the provider survives a bad network

Mara's wifi drops. The socket closes. What should happen? Nothing visible — she keeps typing.

That works because of the layering. Her edits still flow into the Y.Doc — Yjs doesn't care the socket is gone — and queue up as document state. Meanwhile the provider notices the close and reconnects with exponential backoff plus jitter:

// web/src/editor/sync-provider.ts
const backoff = Math.min(MAX_BACKOFF_MS, 1000 * 2 ** (reconnectAttempts - 1))  // 1s, 2s, 4s … cap 15s
const delay = backoff + backoff * 0.3 * Math.random()                          // jitter: no thundering herd

When the socket reopens, the provider sends SyncStep1 again, and the handshake replays exactly what each side missed. Mara's offline edits go up; everything that happened while she was gone comes down. The "reconnect" path is just the "connect" path — the CRDT handshake is the catch-up. The jitter matters at scale: if a server restarts and 200 clients all reconnect on the same 1-second timer, they hammer it in lockstep; a random 0–30% spread turns a spike into a smear. (True half-open detection — a socket dead but never firing close — needs an application heartbeat, still on the list; today we reconnect on the close event, which covers the common cases.)

The React lifecycle gotcha

There is one non-obvious bug this code is shaped to avoid, and the fix only makes sense once you've seen the broken version.

The instinct is useMemo. The Y.Doc and provider are expensive and you want them built once per document, not rebuilt on every render — that is exactly what useMemo is for. You memoize them, and tear them down in an effect cleanup:

// THE BROKEN VERSION — do not ship this
const doc = useMemo(() => new Y.Doc(), [documentId])
const provider = useMemo(
  () => createSyncProvider({ documentId, doc, onStatusChange: setStatus }),
  [documentId, doc],
)

useEffect(() => {
  // ◄── THIS returned function is the cleanup. It tears down BOTH memoized resources:
  return () => {
    provider.destroy()   // closes the WebSocket and stops syncing  → dead socket
    doc.destroy()        // tears down the CRDT state TipTap is bound to → dead document
  }
}, [provider, doc])

Both lines matter equally — neither resource is rebuilt after the cycle below, so the editor ends up bound to a dead socket and a dead Y.Doc. (If anything doc.destroy() is the worse one: a dead socket only stops network sync, but a destroyed Y.Doc is the editor's whole data model, gone.)

It reads correctly. It even works in production. It is broken in development, and the cause is React StrictMode.

In dev, StrictMode deliberately mounts every component, immediately unmounts it, then mounts it again — once — to surface effects that don't survive a remount. The subtlety is what it does across that cycle: it re-runs your effects (cleanup, then setup again), but it does not re-run the component's useMemo factories. There is no fresh render in between, and the deps (documentId) haven't changed — so the memo keeps handing back the instance it already built.

Trace it:

1. Mount
   useMemo builds doc D1 + provider P1        (P1 opens the WebSocket)
   effect setup runs                           (no-op — the effect is only a cleanup)
2. Simulated unmount
   effect CLEANUP runs → provider.destroy() AND doc.destroy()    ◄── both torn down here
                                              (P1's socket closed, D1's CRDT state gone)
3. Simulated remount
   NO re-render, deps unchanged →
     useMemo returns the SAME D1, P1           (the factory does NOT run again)
   effect setup runs again                      (still a no-op)
4. The component now holds D1 and P1 — both destroyed in step 2, neither rebuilt.
   Dead socket AND dead document. The editor renders against a corpse.

The bug is a split between creation and destruction. Creation ran once, in the memo, and survives the remount. Destruction runs on every unmount, in the effect cleanup. One StrictMode cycle gives you one destroy and zero rebuilds — and they are now out of sync.

The fix is to put creation and destruction in the same place — the effect — so every teardown is paired with a fresh build:

// web/src/pages/DocumentEditorPage.tsx — the real version
useEffect(() => {
  const doc = new Y.Doc()
  const provider = createSyncProvider({ documentId, doc, onStatusChange: setStatus })
  setSession({ doc, provider })            // hand the fresh pair to render
  return () => {
    provider.destroy()
    doc.destroy()
    setSession(null)
  }
}, [documentId])

Now the StrictMode remount re-runs the effect setup, which builds a fresh D2 + P2 — so every destroy is matched by a build, and the editor renders against a live pair. In production, where the double-mount doesn't happen, the effect just runs once. (Because the pair is built inside the effect, after the first render, the first render has no provider yet — which is why it lives in session state with a Loading editor… fallback.)

Part 4 — Never losing a word (durable state + compaction)

The document's live state is a CRDT in memory. Memory is gone the instant the process restarts. So how do we save it without saving too much, or losing the last few seconds of typing?

Where the content actually lives

Here is the surprising part. The documents row does not hold the text of the memo:

// src/db/schema.ts
export const documents = pgTable('documents', {
  id: uuid('id').defaultRandom().primaryKey(),
  ownerId: uuid('owner_id').notNull().references(() => users.id, { onDelete: 'cascade' }),
  title: text('title').notNull().default('Untitled document'),
  snapshot: bytea('snapshot'),            // <- the content, but NULL right now
  createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
  updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
})

When Mara creates the memo, snapshot is NULL. The document is real — it has an id, a title, an owner — it just has no content yet. The content is the CRDT, and it is not written here the way you write a string to a column; it gets seeded the first time someone opens the memo and types. So the interesting design is not the columns. It is how the content gets saved once it exists.

The naive save loses work

The obvious way to persist an editor: every time the document changes, write the whole thing back.

Mara types a word  →  serialize the entire document  →  UPDATE documents SET snapshot = <whole doc>

Bad two ways. It rewrites the entire document on every keystroke — a 40-page memo, re-serialized because she fixed a typo. So people debounce it: "only save every 2 seconds." Which trades one problem for a worse one. Picture the process dying at second 1.9 — a redeploy, a crash, a reboot. The last ~2 seconds of everyone's typing are gone, because they only ever lived in memory, waiting for a timer that never fired. For a tool whose whole point is collaborative editing, silently dropping the last edit is the cardinal sin.

Two tiers: append now, fold later

A CRDT does not force you to choose between "save everything" and "save rarely." Every change produces a small binary update — the diff of just that change. So we keep two tiers.

Tier one — the append-only log. The instant the sync server receives an update, it appends it as a row. Durable now, no timer:

// src/db/schema.ts
export const documentUpdates = pgTable('document_updates', {
  seq: bigserial('seq', { mode: 'number' }).primaryKey(),
  documentId: uuid('document_id').notNull().references(() => documents.id, { onDelete: 'cascade' }),
  update: bytea('update').notNull(),     // one Yjs update — the diff of a single change
  createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
})

Tier two — the snapshot. A background worker periodically folds the log into one compacted blob. To fold is to take the snapshot plus the loose update rows, replay them into one Y.Doc, re-encode that single state, write it back as the new snapshot, and delete the rows it folded — many small diffs collapse into one. Loading a document back is the same idea in reverse:

snapshot  +  replay every update row still in the log  =  the current document

The append is what kills the data-loss window: the change is on disk before the editor repaints. The snapshot is just an optimization on top, so the log never has to be replayed from the beginning of time.

The log only grows

Why bother folding at all? Because the append log has a second cost. Each keystroke is a row, and the log only grows:

day 1:   snapshot = ∅   +   replay 200 rows      → fast
day 7:   snapshot = ∅   +   replay 50,000 rows   → slow, and getting slower

The append bought durability and charges for it in load time. Nothing trims the log, so the bill grows forever. Compaction is the housekeeping that pays it: every so often, fold the pile of little updates into one snapshot blob and delete the rows it merged. After a fold, loading reads one blob and replays nothing.

It helps to say what compaction is not. It is not required for correctness — the append already made every edit durable; if the worker never ran, nothing would be lost, loads would just keep getting slower. It is not on the editing path — Mara never waits for it; it runs in a separate worker process. And it is not a debounce — the durable write already happened; compaction only consolidates what is already on disk.

The fold, in code

// src/sync/compaction.ts — compactDocument, all inside one transaction
const pending = await tx
  .select({ seq: documentUpdatesTable.seq, update: documentUpdatesTable.update })
  .from(documentUpdatesTable)
  .where(eq(documentUpdatesTable.documentId, documentId))
  .orderBy(asc(documentUpdatesTable.seq))

const doc = new Y.Doc()
if (meta.snapshot !== null) Y.applyUpdate(doc, meta.snapshot)   // start from the old snapshot
for (const row of pending) Y.applyUpdate(doc, row.update)       // replay the loose updates
const foldedSnapshot = Y.encodeStateAsUpdate(doc)              // re-encode: many → one

await tx.update(documentsTable).set({ snapshot: foldedSnapshot }).where(/* this doc */)
await tx.delete(documentUpdatesTable).where(/* exactly the rows we just folded */)

When it runs: the sweep

Folding on every keystroke would be absurd — re-encoding a 40-page memo because Mara typed one letter. Never folding makes loads crawl. So we fold on a schedule, and only the documents that need it. A repeatable job fires every 60 seconds; each tick is one sweep: find the documents whose un-folded log has crossed a threshold, and fold each.

// src/sync/compaction.ts
export const COMPACTION_THRESHOLD = 200   // fold a document once it has ≥ 200 loose updates

Because folded rows are deleted, the rows left in document_updates are the un-folded tail — so a plain COUNT per document is exactly "how far behind is this one." The threshold is a tuning knob, not a correctness one: fold more often for a smaller tail and cheaper loads at the cost of more background work, or less often for the reverse. Durability does not depend on it at all.

Now the two subtleties — both real bugs I had to design around.

The watermark that loses edits

How does a fold know which rows it folded, so it deletes those and not others? The tempting answer is a high-water mark: store a number snapshotThrough = N meaning "the snapshot folds in every update through seq N." Then load replays seq > N, and a fold deletes seq <= N. One number, clean — and it even looks race-proof, since a new append during a fold gets a seq above N.

It loses edits. seq is a bigserial — Postgres hands out the number at the moment of insert, but a row stays invisible to everyone else until its transaction commits, and commits do not finish in the order the numbers were handed out. The appends are fire-and-forget, so several are in flight at once:

1. Append X grabs seq = 11 — has NOT committed yet
2. Append Y grabs seq = 12 — commits fast
3. Fold reads "seq > snapshotThrough" → sees 12, NOT 11 (uncommitted = invisible). Folds it, sets N = 12
4. Fold deletes seq <= 12. Row 11 is still invisible, so it survives the delete
5. X finally commits seq = 11
6. Next load replays seq > 12 → row 11 is never replayed

Row 11 is gone — Mara's edit, dropped silently, by the exact layer whose whole job is to never drop one. The bug hides inside the word watermark: seq <= N does not mean "I have seen everything up to N." A counter that hands out numbers before transactions commit cannot promise that.

The fix: delete what you fold, replay what remains

Throw the watermark away. Two rules, and the loss becomes impossible.

Delete exactly the rows you folded. The fold reads the loose rows, folds the ones it can see, and remembers their exact seq list. It deletes that list — never seq <= max:

// src/sync/compaction.ts
const foldedSeqs = pending.map((row) => row.seq)
await tx.delete(documentUpdatesTable).where(
  and(eq(documentUpdatesTable.documentId, documentId), inArray(documentUpdatesTable.seq, foldedSeqs)),
)

A row that commits after the read is simply not in the list, so it is never deleted.

Load replays every remaining row — no seq > N filter. Folded rows are gone, so whatever is still in document_updates is the un-folded tail, a late-committing row 11 included. Run the race again: row 11 commits late, the fold never deleted it (not in the list), and the next load replays every remaining row. Commutativity (Part 1) means replay order doesn't matter, so row 11 lands correctly on top of the snapshot. Nothing lost. "Which rows did I fold?" is answered by holding the actual list for one transaction, not by trusting a single stored number.

One fold at a time, without blocking writers

One race is left. Two sweeps fold the same document at the same moment:

Sweep A reads loose rows {1, 2, 3}
Sweep B reads loose rows {1, 2, 3, 4}
B writes its snapshot (covers 1–4), deletes rows 1–4
A writes ITS snapshot (covers only 1–3)   ← clobbers B's, and row 4 is already gone

So two folds of one document must take turns. The tool is a lock — a way for one operation to say "I'm using this, wait for me." The catch is that the obvious lock breaks the thing we care about most, and the cleanest way to see why is an analogy.

Picture the documents row as a house. It has an address — its id, the permanent identity other things point at — and contents — the snapshot, the stuff inside that changes. A fold rewrites the contents; it never changes the address. An append barely touches the house at all: it drops a row in the log that points back at the house by its address (a foreign key), and before it attaches, the database places a light hold meaning "don't demolish this or change its address while I'm linking to it." Many appends can hold that at once.

The reflex lock for the fold grabs the house exclusively — a sign reading "Renovation in progress. Keep out. The address itself might change." That makes folds take turns, yes, but now no append can attach, because every append needs the address to stay put. Every keystroke-save stalls behind a background cleanup job — exactly backwards.

But a fold never changes the address, only the contents. So it can hang a more honest sign: "Rearranging the contents. The address stays exactly the same." Now a second fold still can't start (two renovators rewriting the same contents would collide), but appends keep flowing — they only ever needed the address to stay put.

Those three signs are real Postgres row-level locks, requested by adding a FOR … clause to a read:

FOR KEY SHARE       "don't change this row's identity"       ← the append's automatic hold (from the foreign key)
FOR NO KEY UPDATE   "I'll change its contents, not its id"   ← the fold
FOR UPDATE          "I might change anything about it"       ← the greedy one we avoid

FOR NO KEY UPDATE conflicts with itself (two folds serialize) but not with FOR KEY SHARE (appends sail past). In code, that is one method on the locking read:

// src/sync/compaction.ts
const [meta] = await tx
  .select({ snapshot: documentsTable.snapshot })
  .from(documentsTable)
  .where(eq(documentsTable.id, documentId))
  .for('no key update')                      //  → SELECT … FOR NO KEY UPDATE

The append's FOR KEY SHARE you never write — Postgres takes it automatically because document_updates.document_id is a foreign key referencing documents(id), so inserting a child row holds the parent's identity still until the link is made. The lock lasts until the transaction's COMMIT. The whole point in one line: the only thing a fold ever waits on — or makes anything wait on — is another fold.

It works (the proof)

Two layers of evidence, because "it should converge" is not "it converges."

Automated — 16 integration tests against the real ws server and real Postgres: bidirectional convergence, same-tick concurrent merge, reconnect → resync → replay, durable cold-reload from Postgres, auth-on-upgrade, and compaction (including a staged commit-reordering race that proves the watermark fix).

Live, in two browser tabs on one document, the real editor:

• type in Tab A → it renders in Tab B, and back — the full TipTap ↔ Yjs ↔ provider ↔ ws round trip

• each tab shows the other's named caret (presence)

• reload a tab → content returns and reconnects

• I pushed the document to 247 updates; ~30 seconds later the background sweep folded it live — 247 rows → 0, snapshot written (442 bytes) — exactly the two-tier persistence, end to end

• zero app-level console errors

It is roughly 600 lines I own and have debugged.

What this can't do yet: a second server

One limitation is worth stating plainly, because it is the first thing that breaks under real load.

Everything above assumes one server process. Each room — the in-memory Y.Doc for a document — lives in that single process's heap. As long as everyone editing a document connects to the same process, they share the same room, and real-time sync just works.

Now scale out. Put two server processes behind a load balancer. Mara opens the memo and lands on process A; Theo opens the same memo and lands on process B. Each process builds its own room for that document — its own in-memory Y.Doc, its own set of connections — and neither knows the other exists.

Mara ──ws──► process A ──► room (Y.Doc) for the memo
Theo ──ws──► process B ──► a DIFFERENT room (Y.Doc) for the same memo

They are both "connected" and editing the same document, but their keystrokes never reach each other live. A's room broadcasts only to A's connections; B's only to B's. Mara would see Theo's edits only later — on a reload, rebuilt from the shared Postgres log they both append to. For a tool whose entire promise is live collaboration, that is a real hole.

The fix is a later step: cross-instance fan-out over Redis pub/sub. Whenever a room produces an update, its process publishes that update to a Redis channel keyed by the document id — and every process subscribed to that channel relays what it hears into its own local room, ignoring its own echoes by an instance id (the self-echo guard, one level up). Redis becomes the bus that makes N separate rooms behave as one.

The shape of the code makes this an addition, not a rewrite: it slots into the exact same doc.on('update') handler where broadcast-and-persist already live —

doc.on('update'):
  ├─► broadcast to this process's own connections   (today)
  ├─► append to Postgres                             (today)
  └─► publish to the Redis channel                   (the later step)

— and the inbound side subscribes to that channel and feeds what it receives back through the same apply-and-broadcast path. Until that lands, the honest statement is this: the implementation is single-instance. It is correct and complete for one process, and incomplete the moment you run two.

When I'd reach for a library instead

I hand-built this to understand it, and that goal is the whole justification. If you don't want to learn the sync layer, Hocuspocus or Liveblocks give you the relay, reconnect, and awareness already debugged — and you should use them.

I'd reach for one the moment the parts I deferred turn expensive: multi-instance fan-out (one process holds each room today; sharing rooms across processes is a Redis pub/sub layer), acknowledged-not-fire-and-forget persistence, a heartbeat for half-open sockets. None of those are hard to see coming; none change the fact that, for learning, hand-rolling was the point.

The whole system in three beats:

The CRDT gives every character a permanent name, so concurrent edits merge with no referee.
The relay forwards every update to everyone else and writes down every word.
The append log saves it the instant it happens; compaction folds it so loading stays cheap.

And the property that ties all three together is the same one: apply the updates in any order, and everyone lands on the same document.

Find the app deployed here and the code here.

The app is two panes.

On the left, you edit a tree of key: value rows.

On the right, the JSON updates as you type.

Drag a row to reorder it. The JSON follows. Change a field's type. The JSON follows. Import a file. The whole tree rebuilds.

So here is the question underneath all of it:

How does the right side always match the left,
even while you are dragging rows around
and renaming the very keys used to find them?

There is one more thing to say up front, because it changes how you read everything below.

There is no server.

No backend. No database. No network call anywhere in the flow. Every step in this post runs in your browser, in JavaScript, on the data sitting in memory. When I say "where does this run," the answer is always the same: the client. So the interesting question is not where the work happens — it is what data is canonical, and what is derived from it.

That distinction is the whole post.

One tree is the truth. Everything else is computed from it.

Start with the data, before any of the words.

Here is a small JSON object. We will follow it the whole way through.

{
  "name": "Ada",
  "age": 36,
  "address": {
    "city": "London"
  }
}

Inside the app, that JSON does not exist as JSON.

It exists as a list of nodes. One node per row. Here is the shape of a node:

interface TreeItem {
  id: string          // a stable handle — never appears in the JSON
  key: string
  type: 'string' | 'number' | 'boolean' | 'object'
  value: string | boolean
  children: TreeItem[] // used when type === 'object'
}

So our example becomes four nodes:

TreeItem  id:"a1"  key:"name"     type:"string"  value:"Ada"
TreeItem  id:"a2"  key:"age"      type:"number"  value:"36"
TreeItem  id:"a3"  key:"address"  type:"object"  children:[
  TreeItem  id:"a4"  key:"city"   type:"string"  value:"London"
]

This list — items: TreeItem[] — lives in a single store.

It is the source of truth.

The JSON on the right is not stored anywhere. It is recomputed from this list, fresh, every time the list changes. Hold onto that. It is the answer to the opening question, and we will earn it properly in a minute.

The data flow, top to bottom:

items: TreeItem[]   ← the one source of truth (in memory)
↓
itemsToJson(items)  ← runs on every change
↓
the JSON text you see on the right

Left is canonical. Right is derived. Nothing on the right is ever the truth.

The id you never see

Look back at that node shape. The first field is id. It is not part of your JSON. It will never be written to the output. So why is it there?

Because the app constantly has to answer one question:

Which node did the user just touch?

You type in the age field. Something has to find that exact node among all of them and update it. You drag the address row. Something has to know which node moved.

You might reach for the obvious handle: the position in the array. The age node is index 1.

This is bad.

The moment you drag age above name, it is index 0. Its identity changed out from under you, just because it moved.

So maybe use the key. The node is "age", after all.

This is also bad.

The key is editable — the user can rename it mid-edit. And nothing stops two rows from both being called "age". An identity that can be renamed or duplicated is not an identity.

So the fix is a separate, stable id that is never shown and never reused for anything else:

export const createItem = (key: string, type: ItemType): TreeItem => ({
  id: crypto.randomUUID(),
  key,
  type,
  value: DEFAULT_VALUE_FOR_TYPE[type],
  children: [],
})

Every node gets a crypto.randomUUID() at birth. "a1", "a2" above are stand-ins for those.

That id does three jobs, all of them about tracking:

It is the React key, so React can tell rows apart across re-renders.

It is the drag identity, so dnd-kit knows which row you grabbed.

It is the lookup key, so an update can find the right node.

The id is the thread the whole app holds onto. The key and the value can change freely; the id never does.

An update, traced end to end

Let's change age from 36 to 37.

You type in the field. The input fires onChange. The row calls one store action:

updateValue(item.id, event.target.value)   // updateValue("a2", "37")

Notice what it passes: the id, not the position, not the key. The id we just set up. That is the handle in action.

Now the store has to find node "a2" and change it. The nodes are nested — address has a child — so finding one means walking the tree:

const findItem = (items: TreeItem[], id: string): TreeItem | undefined => {
  for (const item of items) {
    if (item.id === id) return item
    const found = findItem(item.children, id)   // recurse into children
    if (found) return found
  }
  return undefined
}

Depth-first. Check each node. If it is not a match, descend into its children. Return the first node whose id matches.

What it returns is not a copy. It is the live node, and the action mutates it in place:

updateValue: (id, rawValue) =>
  set((state) => {
    const item = findItem(state.items, id)
    if (item) item.value = rawValue
  }),

item.value = rawValue looks like it breaks React's "never mutate state" rule.

It does not, because the store runs every change through Immer. You mutate a draft. Immer watches what you touched and produces a new immutable tree with only those nodes replaced. The age node and its ancestors get fresh object references; everything else is reused untouched.

So the update flow is:

you type "37"
↓
onChange → updateValue("a2", "37")
↓
findItem walks the tree, returns node a2
↓
mutate the Immer draft: item.value = "37"
↓
Immer produces a new items tree
↓
the store notifies React, the JSON recomputes

Every store action follows this exact shape. Rename a key, change a type, remove a node — find by id, mutate the draft, let Immer rebuild. The id is always the entry point.

One detail that looks like a bug and is not

The age field holds the number 36. But look again at the node:

TreeItem  id:"a2"  key:"age"  type:"number"  value:"36"

value:"36". A string. With quotes. For a number.

This is on purpose.

You might think a number field should store a number. Bind a <input type="number"> to it and be done.

But a controlled number input fights you mid-typing. Try to type 51.5. The instant you have typed 51., the value is not yet a valid number, so it gets coerced back to 51, and the decimal you were about to type is gone.

So number rows store the user's literal edit text — "51." and all — exactly as typed. It stays text the entire time it lives in the tree.

It becomes a real number in exactly one place: the boundary where JSON is generated.

const primitiveJsonValue = (type, value) => {
  if (type === 'number') {
    const parsed = Number(value)
    return Number.isFinite(parsed) ? parsed : 0   // "37" → 37 here, and only here
  }
  if (type === 'boolean') return value === true
  return typeof value === 'string' ? value : String(value)
}

So the tree carries text, and the JSON carries a real number. The coercion is a single line, at a single boundary. Everywhere else, "37" is just text, and your typing is never snapped back.

The JSON pane is computed, never stored

Now we can pay off the promise from the top.

The JSON on the right is not a second copy of your data that the app keeps in sync. Keeping two copies in sync is exactly the bug that makes these things break. There is only one copy — the tree — and the JSON is a pure function of it:

export const itemsToJson = (items: TreeItem[]): JsonObject => {
  const json = {}
  for (const item of items) {
    json[item.key] = isObjectType(item.type)
      ? itemsToJson(item.children)   // recurse for objects
      : primitiveJsonValue(item.type, item.value)
  }
  return json
}

Walk the list in order. Each node contributes one key: value pair. Objects recurse into their children. Then JSON.stringify(result, null, 2) turns it into the indented text you read.

The right pane component does only this:

const items = useTreeStore((state) => state.items)
const jsonText = useMemo(() => itemsToJsonText(items), [items])

It subscribes to items. When items changes — and after any update, Immer hands back a new items — useMemo reruns and the text is regenerated.

So the right side cannot drift from the left. It is not synced to the left. It is the left, run through a function.

That is also why expand/collapse never touches the JSON. Which rows are open is view state, kept in a separate collapsedIds map, deliberately outside the tree. itemsToJson never looks at it. You can collapse address to tidy the view, and the output is byte-for-byte identical. Drawing state and data state never cross.

How a row renders itself

The tree on screen has the same recursive shape as the data, because the component renders itself.

TreeItemRow draws one node: its drag handle, its key input, its type dropdown, and its value editor. Then, if the node is an object, it renders a TreeItemRow for each child — which renders its children, and so on, as deep as the data goes.

{isObject && isExpanded && (
  <ul className="tree-list">
    {item.children.map((child) => (
      <TreeItemRow key={child.id} item={child} parentId={item.id} />
    ))}
  </ul>
)}

Two things ride along in that one line, and both matter later.

key={child.id} — the id again, now as React's reconciliation key. It is how React keeps each DOM row matched to its node across every re-render and reorder.

parentId={item.id} — every row is told who its parent is. A root row gets parentId={null}. This breadcrumb is what makes drag-and-drop safe, which is the next section.

Each row reaches into the store for only the slices it needs:

const updateKey = useTreeStore((state) => state.updateKey)
const removeItem = useTreeStore((state) => state.removeItem)
const isCollapsed = useTreeStore((state) => Boolean(state.collapsedIds[item.id]))

It does not receive a giant prop bundle from its parent and it does not hold the whole tree. It takes its one item, subscribes to the actions and the sliver of state it uses, and renders. The recursion gives you the shape; the store gives every row direct access to the truth.

App
↓
JsonTreeEditor (left)        JsonCodeView (right)
↓                            ↓
TreeItemRow (name)           itemsToJson(items)
TreeItemRow (age)            ↓
TreeItemRow (address)        the JSON text
  └ TreeItemRow (city)

Drag and drop: dnd-kit moves nothing

Here is the part that surprises people.

You drag the address row to a new spot. You might think dnd-kit picked up the node and moved it in your data.

It did not. dnd-kit does not touch your state at all.

All it does is watch the pointer and, when you let go, hand you two ids: the row you picked up, and the row you dropped it over.

const handleDragEnd = (event: DragEndEvent) => {
  const { active, over } = event
  if (!over || active.id === over.id) return
  // ...
}

active.id is the row you grabbed. over.id is the row you released onto. That is the entire gift from dnd-kit. Moving the node in your own state is your job — and because the truth is one tree keyed by id, you already have everything you need to do it.

But first, a guardrail. Remember the parentId breadcrumb every row was given? dnd-kit carries it in the drag payload, so when a drag ends, you can read the parent of both rows:

const activeParentId = (active.data.current?.parentId ?? null)
const overParentId = (over.data.current?.parentId ?? null)
if (activeParentId !== overParentId) return     // different parents → ignore

If you try to drag city (inside address) out to sit next to name (at the root), their parents differ, and the drag is dropped. Reordering is allowed only among true siblings — rows that share a parent. That keeps the operation a simple reshuffle of one list, never a re-parenting that would have to splice a node out of one place and into another.

When the parents match, the move is one store action:

reorderSiblings: (parentId, activeId, overId) =>
  set((state) => {
    const list = childListFor(state.items, parentId)   // the sibling array
    const from = list.findIndex((item) => item.id === activeId)
    const to = list.findIndex((item) => item.id === overId)
    const nothingToMove = from === -1 || to === -1 || from === to
    if (nothingToMove) return
    const reordered = arrayMove(list, from, to)
    list.splice(0, list.length, ...reordered)          // write order back into the draft
  }),

Find the right sibling list — the root list if parentId is null, otherwise that object's children. Find where the dragged node is and where it landed, by id. arrayMove returns the list in the new order. Splice that order back into the Immer draft.

Immer rebuilds the tree. The JSON recomputes. The key order in the output now matches the new row order, because itemsToJson walks the list in exactly the order the list is in.

The full drag flow:

you drop the row
↓
dnd-kit: "active=a3 ended over=a1"   (ids only — no data changed)
↓
same parent? if not, stop
↓
reorderSiblings(parent, "a3", "a1")
↓
arrayMove on the sibling list, written into the Immer draft
↓
new items tree → JSON recomputes in the new order

dnd-kit ran the gesture. The id told you which node it was. Your store did the move. The same division of labor as every other update.

Import: untrusted text becomes a trusted tree

Last flow. You paste JSON, or pick a .json file. The text has to become a tree of nodes — with ids, with types — before the editor can show it.

You might think you can hand the parsed object straight to the components. You cannot. Parsed JSON has no ids, so nothing can be tracked or dragged. And it is untrusted — it might not even be an object.

So import runs through one careful door:

export const parseJsonToItems = (text: string): ParseResult => {
  let parsed: unknown
  try {
    parsed = JSON.parse(text)
  } catch (error: unknown) {
    return { ok: false, error: error instanceof Error ? error.message : 'Invalid JSON' }
  }
  const isJsonObject =
    parsed !== null && typeof parsed === 'object' && !Array.isArray(parsed)
  if (!isJsonObject) {
    return { ok: false, error: 'Top level must be a JSON object.' }
  }
  return { ok: true, items: jsonToItems(parsed) }
}

Three gates, in order:

Parse it. If JSON.parse throws, return the error message — never let it crash the app.

Check the top level is a plain object. Not null, not an array. The editor renders key: value rows, and only an object has those.

Hand the clean object to jsonToItems, which builds the tree.

Note the return type: a ParseResult that is either { ok: true, items } or { ok: false, error }. Errors come back as values, not exceptions, so the import panel can show "Top level must be a JSON object." in red instead of blowing up.

Building the tree is the mirror image of generating the JSON. Walk the object's entries; turn each into a node:

export const jsonToItems = (json: JsonObject): TreeItem[] =>
  Object.entries(json).map(([key, value]) => itemFromEntry(key, value))

const itemFromEntry = (key: string, value: JsonValue): TreeItem => {
  const type = itemTypeForValue(value)
  const item = createItem(key, type)          // ← fresh id minted here
  if (isPlainObject(value)) {
    item.children = jsonToItems(value)         // recurse for nested objects
    return item
  }
  if (type === 'boolean') { item.value = value; return item }
  item.value = primitiveTextFor(value)         // numbers land as text, e.g. 36 → "36"
  return item
}

Three things worth catching here:

createItem mints a fresh crypto.randomUUID() for every node. The imported data is fully tracked and draggable the instant it loads — same id machinery as everything else.

Nested objects recurse, so an imported address rebuilds its city child to any depth.

A number like 36 is stored as the text "36", exactly the convention from earlier — so an imported number behaves identically to one you typed.

And the edges? null and arrays are not among the four supported types, so rather than throw, they are kept as readable text via JSON.stringify and shown as string rows. Import never crashes on a value it does not model.

Once parseJsonToItems returns ok, the import control hands the new tree to the store:

setItems: (items) =>
  set((state) => {
    state.items = items
    state.collapsedIds = {}   // a new document starts fully expanded
  }),

The truth is replaced wholesale. The JSON pane, being derived, redraws itself the next render. No syncing. The new tree simply is the new truth.

pasted text
↓
JSON.parse           (reject on syntax error)
↓
top-level object?    (reject arrays, null, primitives)
↓
jsonToItems          (mint ids, recurse, numbers → text)
↓
setItems             (replace the truth)
↓
JSON pane recomputes from the new tree

The tradeoff, honestly

One source of truth that everything derives from is simple, and it is why nothing drifts. But it is not free, and it is not always what you want.

The JSON regenerates on every keystroke. The whole tree is walked each time. For a human-scale document that is instant and you will never feel it. For a 10,000-node tree, recomputing all of it on every character would be the first thing to fix — you would memoize per-subtree, or debounce the text generation.

And the model is deliberately small. Four types: string, number, boolean, object. No arrays. No null as a first-class value. That keeps the node shape and every conversion trivial to reason about. If you needed real arrays and round-trip-perfect editing of every JSON value, the tree would need a richer type, and several of the clean one-liners above would grow.

So this design is the right answer for an editor you can hold in your head. It is the wrong answer for a million-node document or a faithful JSON-spec editor. Knowing which you are building is the actual decision.

The id tracks every node, no matter how it is renamed or moved.

Immer rebuilds the one tree on every change.

And the JSON is never stored — only ever computed back from the truth.

memo looks simple until you wrap a component in it and it re-renders anyway.

So here is the question underneath this whole pattern:

How do you give a memoized child function props that never change identity,
when the parent rebuilds those functions on every render
and you can't refactor the parent to stop?

Let me trace it with one example, all the way through.

The running example

A <DataGrid> that renders thousands of cells. Expensive. A perfect memo candidate.

Its parent owns the state and passes handlers down:

const Page = () => {
  const [query, setQuery] = useState('')
  const [selected, setSelected] = useState<Row[]>([])

  return (
    <>
      <input value={query} onChange={(e) => setQuery(e.target.value)} />

      <DataGrid
        rows={filterRows(query)}
        selected={selected}
        onSelect={(row) => setSelected([...selected, row])}
        onClearSelection={() => setSelected([])}
        onDeleteSelected={() => deleteRows(selected)}
        onSortColumn={(col) => sortBy(col)}
      />
    </>
  )
}

const DataGrid = memo((props: Props) => {
  // thousands of cells
})

The handlers are not independent.

onSelect reads and replaces selected.

onClearSelection replaces selected.

onDeleteSelected reads selected.

They are interconnected. They all close over the same state. Hold onto that — it is what breaks the easy fix.

Watch memo fail

memo compares the new props to the old props before re-rendering.

Shallowly. Key by key. With Object.is.

Same reference for every prop → it skips the render.

One new reference → it renders.

Type a single character in the search box:

keystroke
↓
setQuery
↓
Page re-renders
↓
every inline arrow is created fresh: onSelect, onClearSelection, onDeleteSelected, onSortColumn
↓
memo compares: onSelect !== last onSelect
↓
DataGrid re-renders all of its cells

A function created inline is a new object every render.

(row) => ... today is not Object.is-equal to (row) => ... yesterday.

The shallow compare fails on the first callback it checks.

The expensive render runs anyway.

Why useCallback can't save this one

The textbook fix is useCallback: it hands back the same function object between renders, so the comparison passes.

Wrap onSelect:

const onSelect = useCallback(
  (row) => setSelected([...selected, row]),
  [selected]
)

The callback closes over selected, so selected goes in the dependency array.

Which means onSelect becomes a new object every time selected changes.

Which is every time you select a row.

memo breaks again, for the exact reason it was breaking before.

Drop the dependency instead:

const onSelect = useCallback(
  (row) => setSelected([...selected, row]),
  [] // stable, but now wrong
)

The reference is stable now. But the closure is frozen at the first render.

selected is always the empty array it was on mount.

Selecting a second row throws away the first.

That is a bug.

This is the trap:

Put the deps in   → the reference changes → memo breaks.
Leave the deps out → the closure goes stale → bug.

useCallback alone cannot give you a stable reference and fresh values.

These are just some shallow examples to make a point, but imagine that you have some thousands of lines of code and over thirty interconnected handlers with lots of props themselves. To fix the grid you'd have to get the dependency arrays right across all of them, keep them in sync as the code changes, and probably lift state or thread it through reducers.

That is a real refactor of the parent which I'm not against, but sometime in the real world, you don't have the time to do this.

The escape is a ref

You need one thing the trap denies you: a value that is stable in identity but reads fresh logic.

A function whose reference never changes.

That, when called, runs the newest version of the code.

A plain variable can't do that. useCallback can't do that.

A ref can.

A ref is a box with a stable identity whose contents you can swap.

So:

Hand the child a function that never changes.
Have that function read the real callback out of a ref at call time.
Update the ref to the latest props on every render.

The child sees a frozen reference.

The ref always holds the latest logic.

The two are decoupled.

The hook

useStablePropsCallbacks does this for every function prop at once:

import { useRef } from 'react'

type AnyCallback = (...args: unknown[]) => unknown

// A function of any shape — only used to detect which props are callbacks.
type AnyFunction = (...args: any[]) => any

// The prop names in P whose value is a function.
// Map each key to itself-or-never, then `[keyof P]` collapses it to the union of survivors.
type CallbackKeys<P> = {
  [K in keyof P]: P[K] extends AnyFunction ? K : never
}[keyof P]

// P with only those props kept.
type CallbacksOf<P> = Pick<P, CallbackKeys<P>>

const isCallbackProp = <P extends object>(
  propKey: string,
  props: P
): propKey is string & keyof CallbacksOf<P> =>
  typeof (props as Record<string, unknown>)[propKey] === 'function'

export const useStablePropsCallbacks = <P extends object>(
  props: P
): CallbacksOf<P> => {
  // Refreshed every render so the proxies reach the newest callbacks.
  const latestPropsRef = useRef(props)
  latestPropsRef.current = props

  // The proxy object. Built once, never rebuilt.
  const stableCallbacksRef = useRef<CallbacksOf<P> | null>(null)

  if (!stableCallbacksRef.current) {
    const proxies = {} as Record<string, AnyCallback>

    for (const propKey of Object.keys(props)) {
      if (!isCallbackProp(propKey, props)) continue

      proxies[propKey] = (...args: unknown[]) => {
        const latestProps = latestPropsRef.current as Record<string, unknown>
        const latestCallback = latestProps[propKey] as AnyCallback
        return latestCallback(...args)
      }
    }

    stableCallbacksRef.current = proxies as CallbacksOf<P>
  }

  return stableCallbacksRef.current
}

CallbacksOf<P> is just "P with the non-function props removed" — it types the return value as only the callbacks. The runtime hook is three moving parts.

latestPropsRef holds the latest props. latestPropsRef.current = props runs on every render, so the box always contains the newest callbacks. (Writing a ref during render is normally a yellow flag; it is safe here because the proxies only ever read "whatever is latest" — they never snapshot a value at a particular render.)

stableCallbacksRef holds the proxy object. The if (!stableCallbacksRef.current) block runs only on the first render. After that the same object comes back forever.

Each proxy is, for one function prop, a wrapper that resolves the real callback at call time: read the latest props, pick the callback for this key, forward the args. Its identity is fixed for the life of the component, but the function it forwards to is always the current one.

Wiring it in

The hook returns stable versions of the function props. Spread them over the real props so they win:

const StableDataGrid = (props: Props) => {
  const stableCallbacks = useStablePropsCallbacks(props)
  return <DataGrid {...props} {...stableCallbacks} />
}

Order matters. {...props} puts the raw, unstable callbacks in first. {...stableCallbacks} then overrides each function prop with its stable proxy.

A thin outer wrapper stabilizes the references. The memoized inner component receives the stable ones.

The flow, end to end

Type a character in the search box again. This time:

keystroke
↓
setQuery → Page re-renders → new inline callbacks (as before)
↓
StableDataGrid runs
↓
latestPropsRef.current = props          (every render — ref now holds the new callbacks)
↓
proxy object returned                   (same object as last render — built once at mount)
↓
DataGrid receives the same proxy references it saw last time
↓
memo shallow compare passes → DataGrid skips its render
↓
later, user clicks a row → proxy onSelect fires
↓
proxy reads latestPropsRef.current.onSelect → runs the newest closure → fresh `selected`

The expensive grid stayed put through every keystroke.

When a real interaction happened, the call hit the current logic, with the current state.

Stable identity for memo. Fresh closure for correctness. No useCallback, no dependency arrays, no parent refactor.

What this gives up

The callback set is frozen at mount. Object.keys(props) runs once, inside the if block. A handler that is undefined on the first render and supplied later never gets a proxy. If your function props are all present from the start, you are fine. If they appear conditionally, they slip through.

Only function props are stabilized. Object and array props still get new references when the parent recreates them, and will still break memo. Those need their own handling.

The indirection costs you some tooling. Stack traces pass through the proxy. And because the wiring is dynamic, the react-hooks/exhaustive-deps lint can't see it — you trade a class of dependency-array bugs for a class of "invisible to the linter" ones.

When not to reach for it

One or two callbacks? Just use useCallback. It is clearer, and the linter helps you.

This pattern earns its keep in one narrow situation: many interconnected, stateful callbacks, on a genuinely expensive child, where rewriting the parent to hand-stabilize each one isn't worth it.

Inside that window, it is the difference between memo working and memo being decoration. Outside it, it is indirection you don't need.

The whole thing in three beats:

The proxy holds a stable identity.
The ref holds the latest logic.
The memo holds its ground.

Appendix: the types, traced one step at a time

Three lines of types do the heavy lifting in that hook, and they are where most eyes glaze over. Mine did.

type CallbackKeys<P> = {
  [K in keyof P]: P[K] extends AnyFunction ? K : never
}[keyof P]

type CallbacksOf<P> = Pick<P, CallbackKeys<P>>

const isCallbackProp = <P extends object>(
  propKey: string,
  props: P
): propKey is string & keyof CallbacksOf<P> =>
  typeof (props as Record<string, unknown>)[propKey] === 'function'

The P is the part that makes it abstract. But P is not a real type — it is a placeholder. A fill-in-the-blank.

It works exactly like a function parameter, one level up. A function parameter stands for whatever value you pass in. P is a type parameter: it stands for whatever type you pass in.

And you pass a type in by writing it in the angle brackets. CallbacksOf<Props> hands Props to the blank called P. From that point on, TypeScript substitutes Props for P everywhere in the definition — keyof P becomes keyof Props, P[K] becomes Props[K], Pick<P, …> becomes Pick<Props, …>.

Here is the type we fill it with:

type Props = {
  selected: Row[]
  onSelect: (row: Row) => void
  onClearSelection: () => void
  count: number
}

Two functions. Two non-functions. We want a type that keeps only the two functions.

For every example below, the rule is simple: wherever a definition says P, read Props.

One thing to hold in your head before we start: none of this runs.

These types are erased before your code executes. There is no CallbackKeys object at runtime, no loop, no work. This is the compiler reasoning about shapes — all of it happens in your editor and at build time, then vanishes. So when I say "the value becomes a name," I mean in the type, on the compiler's scratch pad. Nothing is computed when the app runs.

CallbackKeys: filtering keys when the language has no filter

You'd think you could just write keyof Props.

keyof Props gives you the names of every prop, as a union:

"selected" | "onSelect" | "onClearSelection" | "count"

But that is all four. We only want the two that are functions.

And here is the problem: TypeScript has no .filter() for keys. You cannot say "keep the keys where the value is a function." There is no such operator.

So you do it sideways. In three moves.

Move 1 — walk every key and replace its value.

{
  [K in keyof P]: P[K] extends AnyFunction ? K : never
}

[K in keyof P] means "for each key K in Props, make an entry."

The value of each entry is not the original type. It is P[K] extends AnyFunction ? K : never — a question asked per key:

Is this prop's value a function?
Yes → put the key's own name as the value.
No  → put `never` as the value.

So the object type you get is:

{
  selected: never                          // Row[] is not a function
  onSelect: "onSelect"                      // a function → its own name
  onClearSelection: "onClearSelection"      // a function → its own name
  count: never                              // number is not a function
}

Read that again. The function keys now hold their own name as a value. The rest hold never.

Move 2 — read all the values at once.

[keyof P] on the end looks like indexing an array. It is indexing a type.

When you index an object type by a single key, you get that key's value. When you index it by a union of keys, you get the union of all those values.

keyof Props is the union of all four keys, so:

{…}["selected" | "onSelect" | "onClearSelection" | "count"]

hands back the union of the four values:

never | "onSelect" | "onClearSelection" | never

Move 3 — never falls out for free.

never is the empty type. In a union it just disappears — never | "onSelect" is "onSelect".

So the whole thing collapses to:

"onSelect" | "onClearSelection"

That is CallbackKeys<Props>. The names of the function props, and nothing else.

The flow, end to end:

Props
↓ keyof P
"selected" | "onSelect" | "onClearSelection" | "count"
↓ map each key → (its own name if a function, else never)
{ selected: never, onSelect: "onSelect", onClearSelection: "onClearSelection", count: never }
↓ [keyof P] — read every value as one union
never | "onSelect" | "onClearSelection" | never
↓ never drops out of a union
"onSelect" | "onClearSelection"

The map-to-name-then-index flow is the missing filter. That is the only reason it exists.

CallbacksOf: turning the names back into an object

Now you have the names. You want the object.

type CallbacksOf<P> = Pick<P, CallbackKeys<P>>

Pick<T, K> is built into TypeScript. It means exactly what it says: pick keys K out of object T, keep their types, drop everything else.

So Pick<Props, "onSelect" | "onClearSelection"> is:

{
  onSelect: (row: Row) => void
  onClearSelection: () => void
}

Props with the non-functions gone. The values come through untouched.

That is CallbacksOf<Props>. One readable line, because CallbackKeys already did the hard part.

isCallbackProp: a runtime check that teaches the compiler something

This one mixes two worlds, which is what makes it look strange.

const isCallbackProp = <P extends object>(
  propKey: string,
  props: P
): propKey is string & keyof CallbacksOf<P> =>
  typeof (props as Record<string, unknown>)[propKey] === 'function'

The two worlds are both in the signature, and they are easy to mix up because they look alike.

props — lowercase — is the real object your code holds at runtime: the actual { selected, onSelect, … }.

P — uppercase — is its type: Props, the same thing we filled in earlier.

props: P just says "this value has that shape." Lowercase is the value, uppercase is the shape of the value. So in our example, props is an object and P is Props.

Ignore the return type for a second. The body is trivial:

typeof props[propKey] === 'function'

"Is the value at this key a function?" A plain boolean. That part runs at runtime, in the browser, while the hook builds its proxies.

Now the strange part: the return type is not boolean. It is propKey is string & keyof CallbacksOf<P>.

That is a type predicate. It changes what the compiler knows about propKey after the call.

Here is why you need it. Inside the hook:

for (const propKey of Object.keys(props)) {
  if (!isCallbackProp(propKey, props)) continue
  // ...
}

Object.keys returns string[]. So propKey starts as a plain string. The compiler has no idea which key it is — it could be "count", which is not a callback.

A naive version returning : boolean would check the value at runtime, but the compiler would still see propKey as a plain string afterward. It would not let you treat it as a callback key.

The is predicate fixes that. It tells the compiler:

If this function returns true, then for the code that follows,
you may treat `propKey` as `string & keyof CallbacksOf<P>`.

For our Props, keyof CallbacksOf<Props> is "onSelect" | "onClearSelection". So after the if, the compiler narrows propKey from "some string" to "one of the callback names." Now indexing the callbacks object with it is safe.

The string & part is a small guard. keyof can in general yield string | number | symbol, but Object.keys only ever gives strings — intersecting with string keeps the result a string key. For our Props the keys are already strings, so it changes nothing here; it matters only for the fully generic P.

Now the honest part, because a predicate has a sharp edge.

The body runs. The predicate does not.

typeof … === 'function' is real work the browser does. propKey is … is a promise you make to the compiler, and the compiler believes it without checking. If you wrote a boolean that did not actually match the promise, TypeScript would trust the lie and narrow anyway.

Here the boolean and the promise agree — a prop whose value is a function genuinely is a callback key — so it is sound. But that is on you to get right, not something the compiler proves.

The three types in three beats:

keyof P lists every prop name.
CallbackKeys filters those names down to the callbacks.
isCallbackProp proves, at runtime, that one loose string is really one of them.

This post is about the second factor. The thing where Mara scans her face to finish logging in. It's built on one more vetted package — @simplewebauthn for the passkey crypto — and everything around it is hand-rolled so you can see the seams.

Same method as last time: trace one person through the whole machine.

Mara already has an account. She signs up, logs in, signs in with Google, resets her password — all of that is the first post. Now she wants a second lock on her account. She turns it on. She logs in with it. She loses her phone and gets back in with a recovery code. She buys a laptop and adds a second key. Eventually she turns the whole thing off.

Follow Mara, and the rest falls into place.

Three themes carry the whole post, so watch for them:

• Nothing secret ever travels. A password is a shared secret — it has to cross the wire to be checked. A passkey isn't. The thing that proves Mara's identity never leaves her phone, not even to the server. That single property is what makes it un-phishable, and it reshapes everything downstream.

• The server always aims the second factor. The dangerous bug in 2FA isn't a weak signature — it's letting the client say whose second factor this is. Get that wrong and the first factor becomes decorative. The fix is one rule, enforced in four places.

• The best defenses are holes you never built. Like last time. A whole class of attack keeps turning out to be impossible because of the shape of the thing, not because of a check we added.

Let's go.

First: what a passkey actually is

Here are the scary words, all at once:

Passkey.   WebAuthn.   Relying Party.   Challenge.   Attestation.   Assertion.

And the usual explanation: "the authenticator signs a challenge with a private key, and the server verifies it with the public key."

That is not an explanation. That is a list of things you are now supposed to understand.

So here is the real question underneath all of it:

How can Mara prove it's really her using her face —
when her face never leaves her phone, and in a way a fake website can't copy and replay?

The naive second factor, and why it's phishable

The obvious way to add a second factor is a shared secret.

A password is a shared secret. A TOTP code — the 6 digits in Google Authenticator — is a shared secret too. The server and the phone both know it, and the phone just shows the current code to Mara to type in.

Mara's phone:   secret = 7Q2F...  →  shows code  →  482913
redline server: secret = 7Q2F...  →  expects     →  482913

It works. It is also phishable.

A secret only works if it travels. Mara reads the code off her screen and types it into a page. So picture a fake site — red1ine.com as a pixel-perfect copy. Mara lands there by mistake and logs in. It asks for her code. She types 482913. The fake site forwards it to the real redline within the 30 seconds it's valid.

The thief is in.

The secret crossed the wire. It crossed Mara's eyes. Anything that travels can be intercepted or relayed.

A passkey never shares a secret at all.

Collapse the definition: a passkey is a key pair

A passkey is two keys that belong together.

A private key. And a public key.

That is all it is.

The private key stays inside the phone's secure hardware. Forever. It never leaves. The public key is the half you are allowed to hand out.

The whole trick is what each half can do:

The private key can SIGN a message.
The public key can CHECK that signature — but can never produce one.

So Mara's phone proves it holds the private key by signing something. The server, holding only the public key, confirms the signature is genuine. But the server — or a thief who steals the server's entire database — can never sign anything as Mara. The public key doesn't let you. That's the asymmetry the whole feature stands on.

Nothing secret ever travels. Only the public key (safe to share) and signatures (useless to replay, as we'll see).

Your face is not the second factor

You might think the phone scans Mara's face and sends "yes, it's Mara's face" to the server.

It does not. Her face never leaves the phone.

Face ID is a local lock. It unlocks the private key sitting in the phone's secure chip. That is its only job.

Face ID  →  unlocks the private key  →  the key signs

The server never sees a face. It never sees the private key. It sees a signature, and checks it against the public key it stored. Touch ID, a Windows Hello PIN, a YubiKey tap — same shape. A local gate that releases a local key.

The challenge: why a signature can't be replayed

If the phone just signed the word "redline" every time, a thief who captured one signature could resend it forever.

So the server never asks for a signature over something fixed. It sends a challenge — a fresh random number — and asks the phone to sign that.

server: here is a random challenge → 9f2a7c...e1
phone:  sign(9f2a7c...e1) with the private key → <signature>
server: does <signature> check out against Mara's public key? yes.

Next login, a different random challenge. A captured signature is worthless — it answers a question the server will never ask again. A challenge is a one-time question; the signature is the one-time answer.

RP ID vs origin: the part that actually kills phishing

Two identifiers trip everyone up. They sound alike. They are not.

RP ID — the domain the passkey belongs to. "Relying Party" is just jargon for "the site." Ours is localhost in dev, the real host in production.

Origin — the exact URL the browser is actually at, like http://localhost:3000.

Here is the load-bearing rule, and it's enforced by the browser itself, not by our code:

The browser binds every signature to the origin it is running on and refuses to use a redline passkey anywhere but redline's origin.

So go back to the fake red1ine.com. Mara's phone has a passkey for redline. The fake site asks for a signature. The browser checks: this passkey is bound to localhost / redline.app, and the page is red1ine.com. Mismatch. The browser will not even offer the key.

There's no code on screen for Mara to mistype. There's no secret to forward. The one thing that proves her identity is locked to the real origin by the browser. That is why a passkey is phishing-resistant and a typed code is not — and it's the whole reason I reached for passkeys instead of TOTP, even though TOTP is friendlier to build.

This is a SECOND factor, not passwordless

A passkey can be your entire login — no password at all. That's "passwordless," and it's a different design. We are not doing that here.

Here the password (or Google sign-in) is factor one. The passkey is factor two. Both must pass. This choice isn't cosmetic — it changes the knobs we set when we build the options, and you'll see exactly where in a moment.

Second: how we store and verify a passkey

The idea becomes rows and functions. Three concrete questions: where does the public key live, how do we verify a signature without writing crypto ourselves, and how does the server look the right key back up?

The data model, and what's deliberately not in it

One passkey is one row.

// src/db/schema.ts
export const webauthnCredentials = pgTable('webauthn_credentials', {
  id: text('id').primaryKey(),              // the credential id from the authenticator (base64url)
  userId: uuid('user_id').notNull().references(() => users.id, { onDelete: 'cascade' }),
  publicKey: text('public_key').notNull(),  // base64url COSE public key — the only key we store
  counter: bigint('counter', { mode: 'number' }).notNull().default(0),
  transports: jsonb('transports').$type<AuthenticatorTransportFuture[]>(),
  deviceType: text('device_type'),          // 'singleDevice' | 'multiDevice'
  backedUp: boolean('backed_up'),
  name: text('name'),                       // "iPhone 15" — a user label
  createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
  lastUsedAt: timestamp('last_used_at', { withTimezone: true }),
})

Notice what is not there. There's no private-key column. There's no "secret." The server never had Mara's private key and never will. The only key material here is public_key — the half that can check a signature but can't make one.

And notice the primary key. id is not a fresh uuid we generate; it's the credential id the authenticator handed back at enrollment. That string is how the browser refers to the key on later logins, so it is the natural key — we look the row up by exactly the value an assertion reports.

There's also no twoFactorEnabled boolean, and that's deliberate. A boolean would be a second copy of the truth, free to drift out of sync with reality. Instead, 2FA is on when Mara has at least one credential row:

// the login gate — a COUNT, not a flag
export const hasEnrolledPasskey = async (userId: string): Promise<boolean> => {
  const [row] = await db
    .select({ total: count() })
    .from(webauthnCredentials)
    .where(eq(webauthnCredentials.userId, userId))
  return (row?.total ?? 0) > 0
}

Delete her last credential and she's back to single-factor — automatically, with no flag to remember to flip. (This pays off later: "disable 2FA" becomes "delete the rows," nothing more.)

Do NOT write the crypto

Here is the part it's tempting to build yourself.

Verifying a registration means taking the blob the browser returned and pulling the public key out of it. That blob is a CBOR-encoded attestation object, wrapping a COSE-encoded public key, wrapping signature flags and counters. Verifying a login means decoding authenticator data, re-hashing client data, and checking an ECDSA or RSA signature with exactly the right parameters.

This is the bad version:

// do not do this
const attestation = decodeCbor(response.attestationObject)
const publicKey = parseCoseKey(attestation.authData.slice(/* ...offsets... */))
// ...and now you are one off-by-one away from a CVE

Get one offset or one flag wrong and you either reject every honest user, or — worse — accept a forged one.

So we don't. We use @simplewebauthn/server for the parsing and signature math. We orchestrate; the library does the dangerous bytes. That's the entire division of labor in src/auth/webauthn.ts — the same reasoning that made argon2id and arctic library choices in the first post. Crypto is the thing you stand on a vetted primitive for, never hand-roll. (Its return shape changed across major versions, so the field names in our code were read off the installed .d.ts, not from memory.)

The two option flags that are the design

Before the phone can do anything, the server hands the browser an "options" blob — the parameters for navigator.credentials.create() (enroll) or .get() (login). This is where second-factor-not-passwordless stops being a sentence and becomes two flags:

// src/auth/webauthn.ts — enrollment options
generateRegistrationOptions({
  rpID: env.RP_ID,
  rpName: env.RP_NAME,
  userID: isoUint8Array.fromUTF8String(input.userId),
  userName: input.userName,
  attestationType: 'none',
  excludeCredentials: input.existingCredentials.map(toCredentialDescriptor),
  authenticatorSelection: { residentKey: 'discouraged', userVerification: 'preferred' },
})

Read the last line, because it is the policy.

residentKey: 'discouraged' — do not ask the phone to store a discoverable credential. A discoverable credential is the passwordless feature: it lets you log in with no username at all. We don't want that. The password is factor one; the passkey only confirms it.

userVerification: 'preferred' — ask for the biometric (the Face ID Mara wanted) where the device can do it, but don't hard-fail an authenticator that can't. A bare security key with no fingerprint reader still proves possession, and possession is the second factor.

excludeCredentials is the list of keys Mara already enrolled — the browser greys those out so she can't register the same iPhone twice. The login side is the mirror image: instead of excluding, it passes allowCredentials (her credential ids), because the password already told us who she is, so we can name exactly which keys may answer.

Both builders generate a fresh random challenge inside the returned options. The caller's job is to stash that challenge somewhere short-lived so the verify step can demand the same one back — which is the next two sections.

The public key: base64url, decoded only at the edge

The library hands us a public key as raw bytes — a Uint8Array. Postgres columns are text. So we encode to base64url on the way in:

// after a registration verifies
publicKey: isoBase64URL.fromBuffer(credential.publicKey),  // bytes → "pQECAyYgASFY..."

When Mara logs in later, the library needs the bytes back to check her signature. So we decode at exactly one place — the verify boundary — and nowhere else:

// inside verifyPasskeyAuthentication
credential: {
  id: input.credential.credentialId,
  publicKey: isoBase64URL.toBuffer(input.credential.publicKey),  // "pQECAyYgASFY..." → bytes
  counter: input.credential.counter,
  transports: input.credential.transports ?? undefined,
}

Text at rest. Bytes for one function call. The conversion never leaks into the rest of the code.

And the phishing defense from the first section becomes two arguments on that same verify call:

const EXPECTED_ORIGIN = new URL(env.APP_URL).origin  // strips a stray trailing slash

await verify({
  response: input.response,
  expectedChallenge: input.expectedChallenge,
  expectedOrigin: EXPECTED_ORIGIN,   // the URL the browser must have been at
  expectedRPID: env.RP_ID,           // the domain the key must be bound to
  requireUserVerification: false,    // honors 'preferred' — don't reject UV-less authenticators
})

new URL(env.APP_URL).origin is not fussiness. WebAuthn compares the origin byte-for-byte. A trailing slash in APP_URL would silently fail every verification, and you'd spend an afternoon wondering why valid passkeys are rejected. Deriving .origin strips it.

Why we never judge the counter ourselves

Every authenticator keeps a signature counter — how many times it has signed. The idea is clone detection: if a counter ever goes backwards, two copies of the key might exist.

So the naive rule is: store the counter, reject any login where the new counter isn't strictly greater.

That rule locks Mara out.

Modern synced passkeys — iCloud Keychain, Google Password Manager — report a counter of 0. Forever. They live in the cloud, not one chip, so the count is meaningless and they just send zero. Enforce "must increase" and every iPhone passkey fails on its second use.

So we don't enforce it. We hand the stored counter to the library, let it apply the spec-correct check, and simply persist whatever new value comes back. The counter is the library's to interpret; we're just its storage.

This module is deliberately dumb. It builds options, verifies responses, reads and writes rows. It does not know about challenges-in-Redis, login state, or who's allowed to call it. That's on purpose — it stays a pure "given a challenge, prove the key" layer, so the stateful, security-critical orchestration lives in one place you can audit. That place is the next three sections. If you ever find auth-flow logic creeping into this file, that's the smell; push it back out.

Third: the gap between "password OK" and "you're in"

Mara has a passkey now. So her login has two steps, not one.

Step one: email and password. Correct. Step two: her phone signs a challenge. Between those two steps, something has to remember her.

Without 2FA, login was one moment:

password correct → mint a session → set the cookie → done

With 2FA, there's a gap in the middle:

password correct → ??? → second factor → mint a session → done

What lives in the ???. Mara has proven one factor and not the second. She is half-authenticated — more than a stranger, less than logged in. You can't give her a session yet; a session is "fully logged in." But you have to remember something across her next few requests, or step two has no idea who's knocking.

The naive version, and why it quietly defeats 2FA

Here's the tempting shortcut. The /login handler just checked Mara's password, so it knows her userId. Hand that to the browser and let the second-factor requests send it back:

POST /login            
{ mfaRequired: true, userId: "mara-uuid" }   ← BAD
POST /2fa/verify         
{ userId: "mara-uuid", assertion: ... }       ← BAD

Read that second line again. The userId is coming from the client.

Now picture an attacker. They have their own account and their own passkey. They know Mara's userId (it leaked, or it's in a URL somewhere). They call the verify endpoint with Mara's userId and their own assertion — signed by their own phone, which they can Face-ID all day. If the server trusts that userId and just checks "is this a valid assertion for some registered credential," the attacker walks into Mara's account.

The password — the first factor — just became decorative. The second factor authenticated the attacker's finger against the attacker's key, but logged them into Mara's account. The bug is one word: the userId came from the request.

The fix: the second factor's identity comes only from server state

The rule that makes 2FA real, and the second of our three themes:

After the password passes, the userId for the second factor
comes ONLY from server-side state — never from the request.

So at /login, when the password is right and Mara has a passkey, we don't hand her a userId. We mint a pending-MFA token and keep the userId on our side, in Redis, under the token's hash:

// src/auth/mfa.ts
export const createPendingMfa = async (input: { userId: string }) => {
  const rawToken = generateToken()                          // 256-bit random
  const value = { userId: input.userId, challenge: null }
  await redis.set(pendingKey(rawToken), JSON.stringify(value), 'EX', PENDING_MFA_TTL_SECONDS)
  return { rawToken, expiresAt: new Date(Date.now() + PENDING_MFA_TTL_MS) }
}

The raw token goes to Mara in a cookie. The userId stays in Redis. When step two arrives, the server reads the userId back from Redis, keyed by the token in her cookie. The attacker can send any userId they like in the body — nobody reads it. The only userId that matters is the one we wrote, that only Mara's cookie can point at. The first factor is load-bearing again.

Why it's a different cookie from the session

Mara's pending token rides a cookie called redline_mfa. Not the session cookie. On purpose.

Here's the trap: if the half-auth token were accepted by getSessionUser, then "password correct, second factor still pending" would already be "logged in" — and the second factor would be skipped entirely. So the pending token lives in its own cookie, resolved only by loadPendingMfa, and the session resolver never looks at it.

// src/auth/cookies.ts
export const MFA_COOKIE_NAME = 'redline_mfa'   // separate from SESSION_COOKIE_NAME

Two cookies. One means "logged in." The other means "halfway there." They never cross.

Same hash-at-rest trick, single-use, bounded

The pending token is a key — whoever holds it can finish Mara's login — so we treat it exactly like the session token from the first post. The cookie holds the raw token; Redis stores only its sha256 hash as the key:

const pendingKey = (rawToken: string) => `mfa:pending:${hashToken(rawToken)}`

A Redis leak hands the attacker a list of hashes that key nothing without the raw token, which only ever lived in Mara's cookie. (Why sha256 and not argon2? Same split as the first post: the token is 256 bits of randomness, not a guessable password — nothing to brute-force, so the fast hash is correct.)

Two more properties keep the gap small. It's single-use — deleted the instant the second factor succeeds, so a finished login can't be replayed (we consume only on success, so a failed Face ID leaves the token alive for a retry). And it's bounded — a 10-minute TTL, so the half-authenticated state can't linger. Plenty of time to glance at a prompt and tap; after that it evaporates and Mara starts over. The challenge gets written into this same entry later, and even that write re-reads the userId from Redis rather than trusting the caller — the invariant holds at every step.

This whole flow exists because we chose 2FA. A passwordless design has no "half" state to hold — the assertion either logs you in or it doesn't. The gap is the price of a first factor, and the pending token is how you guard it.

Fourth: recovery codes, for when the phone goes in the lake

Mara enrolled a passkey. Her account is safer. It's also more fragile in exactly one way: the passkey lives in one device's secure hardware. Drop that phone in a lake, and the private key is at the bottom of the lake with it.

Why "reset my passkey" cannot exist

With a password, losing it is fine. You click "forgot password," prove you own the email, set a new one — the server always could set a new password, because the server controls it. A passkey is the opposite. The server never had the private key. It can't reset what it never held. There is no "email me a new passkey."

So the naive recoveries don't work:

"Let support flip 2FA off."           → now a support social-engineer is your second factor.
"Email a magic link to disable 2FA."  → now your email is your second factor, and email is phishable.

Each quietly hands the second factor back to something weaker. The whole point was to not depend on a phishable channel. We need a backup as strong as the passkey, handed to Mara up front, while she still has the device. That's a recovery code.

A recovery code is a one-time password you were given in advance. We generate ten recovery codes the moment Mara enrols her first passkey, show them once, and she saves them somewhere safe. Months later, phone in the lake, she types one in instead of using her passkey. It logs her in. Then it's dead — used up, never again.

What a code looks like, and why

A7KM-9QR3-FXP2

Three groups of four, from a deliberately small alphabet:

const RECOVERY_CODE_ALPHABET = 'ABCDEFGHJKMNPQRSTUVWXYZ23456789' // no 0 O 1 I L

Look at what's missing: no 0 or O, no 1 or I or L. Those are the characters people mistype when reading off paper. Leave them out and a misread can't accidentally land on a different valid code. Each code is 12 characters from 32 symbols — about 60 bits of entropy, unguessable, and single-use and rate-limited on top, so there's nothing to brute-force.

The dashes are just for the eye. When Mara types it back, we don't care about her dashes or her caps lock:

const normalizeRecoveryCode = (rawCode: string): string =>
  rawCode.toUpperCase().replace(/[\s-]/g, '')

a7km9qr3fxp2, A7KM-9QR3-FXP2, A7KM 9QR3 FXP2 all canonicalize to the same thing before we hash it. The stored hash and the typed-back code always meet in one shape.

Stored hashed, salted with the userId

We never store the codes themselves — a database leak would otherwise hand the attacker ten working 2FA-bypasses per user. So we store only a hash, salted with the userId:

const recoveryCodeId = (userId: string, normalizedCode: string): string =>
  hashToken(`\({userId}:\){normalizedCode}`)   // sha256(userId + ':' + code)

Why salt with the userId? A bare sha256(code) would be global — two users who happened to get the same code would collide on the primary key, and identical codes would produce identical hashes, leaking that they match. Salting per-user scopes each hash to its owner. And a code is only ever looked up in the context of a known user — at login, the one from the pending-MFA token, never the request. Same invariant, carried through.

Single-use, enforced by the database

"Each code works exactly once" sounds like a rule you check in code. It isn't. It's enforced in one query:

export const consumeRecoveryCode = async (userId: string, rawCode: string): Promise<boolean> => {
  const id = recoveryCodeId(userId, normalizeRecoveryCode(rawCode))
  const consumed = await db
    .update(recoveryCodes)
    .set({ usedAt: new Date() })
    .where(and(eq(recoveryCodes.id, id), eq(recoveryCodes.userId, userId), isNull(recoveryCodes.usedAt)))
    .returning({ id: recoveryCodes.id })
  return consumed.length > 0
}

The isNull(usedAt) in the WHERE is the whole trick. It says: mark this code used only if it is currently unused. The database does the check and the write in one atomic step. So picture a double-submit — Mara fat-fingers the button twice, two requests race. Both compute the same code id, both try to update WHERE used_at IS NULL. The database lets exactly one match: the first sets used_at, the second now finds nothing matching and updates zero rows. .returning() tells us which we were — one row means we consumed it, zero means unknown-or-already-spent. No read-then-write gap for a race to slip through. (It's the same shape as the 23505 unique-constraint trick from the first post: let the database be the atomic judge instead of checking-then-acting.)

The codes are shown exactly once. generateRecoveryCodes returns the raw codes for the route to display and keeps only the hashes — there is no "show me my codes again" endpoint, because there's nothing to show; we threw the originals away on purpose. Regenerating is the same call: it deletes the old batch in a transaction and writes a fresh ten, so an old printout stops working the moment a new one is issued.

The honest cost: a recovery code is a bearer secret. Whoever holds it passes the second factor — that's the entire job — so a leaked code is as dangerous as a leaked password, minus the username. That's the deal we accept for not locking Mara out forever, and we blunt it the obvious ways (single-use, rate-limited, regeneration burns the old set, and we tell her to store them like passwords). If you could guarantee every user a second hardware key in a safe, you'd sidestep the printable bypass entirely — but most people don't have a spare YubiKey in a drawer. Mara doesn't. So she gets codes.

Fifth: wiring the two-factor login

We have all the parts — the service that verifies a passkey, the pending-login token, recovery codes. This is the assembly: the actual endpoints Mara's browser calls. Two journeys, turning 2FA on and then logging in with it.

Login now has two endings

Before 2FA, loginWithPassword had one outcome: a session. Now it has two, and the type says so out loud:

// src/auth/password-auth.ts — the two outcomes named once, so no bare 'mfa_required' string drifts around
export const PASSWORD_LOGIN_STATUS = {
  authenticated: 'authenticated',
  mfaRequired: 'mfa_required',
} as const

export type PasswordLoginResult =
  | { status: typeof PASSWORD_LOGIN_STATUS.authenticated; user: User; session: CreatedSession }
  | { status: typeof PASSWORD_LOGIN_STATUS.mfaRequired; userId: string }

A correct password is no longer the end of the story. If the user has a passkey, the function stops and returns mfaRequired with only the userId — no session:

if (await hasEnrolledPasskey(account.userId)) {
  return { status: PASSWORD_LOGIN_STATUS.mfaRequired, userId: account.userId }
}

One detail matters for security: this branch is reached only after the password verified. So mfaRequired can never tell an attacker "this email has 2FA" — they'd have needed the right password to see it. (The status values are a named const object rather than bare strings on purpose — one source of truth, so a typo at any call site is a compile error instead of a silent miss.)

The /login route reads the discriminated result and forks:

// src/auth/routes.ts
if (result.status === PASSWORD_LOGIN_STATUS.mfaRequired) {
  const pending = await createPendingMfa({ userId: result.userId })
  setMfaCookie(reply, pending.rawToken, pending.expiresAt)
  return reply.send({ mfaRequired: true })   // ← no session cookie on this path
}
setSessionCookie(reply, result.session.rawToken, result.session.expiresAt)
return reply.send({ user: publicUser(result.user, linkedProviders) })

The fork is the whole point: one branch hands out a session, the other hands out a pending token and asks for more.

Turning 2FA on: the enrollment round trip

Mara is already signed in. She clicks "add a passkey." Two requests:

POST /auth/2fa/register/options   (her session cookie proves who she is)
  → build options { challenge, excludeCredentials, rp, ... }
  → store the challenge in Redis, keyed by her userId
  → return options
browser: navigator.credentials.create(options)   ← Face ID; makes the key pair
POST /auth/2fa/register/verify    { response, name? }
  → take the stored challenge back (GETDEL — single use)
  → verifyPasskeyRegistration(response, challenge)
  → store the public key
  → if this is her FIRST passkey: generate 10 recovery codes, return them ONCE

Notice where the enrollment challenge lives. Mara has a session here, so we key it by her userId, and read-and-delete it atomically so it can only be spent once:

// src/auth/webauthn-challenge.ts
const registrationChallengeKey = (userId: string) => `webauthn:challenge:reg:${userId}`

export const takeRegistrationChallenge = (userId: string) =>
  redis.getdel(registrationChallengeKey(userId))   // read-and-delete, atomic, single-use

That's different from the login challenge, which has no session to key on and rides the pending-MFA entry instead. Same idea — a server-issued challenge the verify step demands back — keyed differently because enrolment knows the user and login doesn't yet.

The recovery codes are minted here, at the first passkey:

const isFirstPasskey = !(await hasEnrolledPasskey(active.userId))

await storePasskey({ userId: active.userId, registration: verified, name: body.name ?? null })

const recoveryCodes = isFirstPasskey ? await generateRecoveryCodes(active.userId) : undefined

return { credentialId: verified.credentialId, recoveryCodes }

The moment 2FA turns on is the moment Mara could get locked out — so that's the moment she gets her backup. Check isFirstPasskey before storing, or the new row makes every enrollment look like the first.

Logging in with 2FA: the second round trip

Now the everyday path:

POST /auth/login           → password OK, has passkey → { mfaRequired: true }
                             + redline_mfa cookie, NO session
POST /auth/2fa/authenticate/options   (reads redline_mfa cookie)
  → load the pending login → her userId
  → build options { challenge, allowCredentials: her keys }
  → attach the challenge to the pending entry
  → return options
browser: navigator.credentials.get(options)   ← Face ID; signs the challenge
POST /auth/2fa/authenticate/verify    { response }
  → load pending → { userId, challenge }
  → the asserted credential must belong to THIS user        ← the guard
  → verifyPasskeyAuthentication(response, challenge, storedKey)
  → update the counter
  → finishMfaLogin: burn pending, clear cookie, MINT SESSION

Every one of those /2fa/authenticate handlers begins the same way — by resolving the cookie to a pending login, server-side, with the userId coming out of Redis and never out of the request body. That's the invariant from the third section, now enforced at every route that finishes a login.

The guard that makes the second factor real

Here's the line the whole feature hinges on:

const pending = await requirePendingMfa(req.cookies[MFA_COOKIE_NAME])
// ...
const credential = await getPasskey(body.response.id)
const isOwnedByPendingUser = credential !== null && credential.userId === pending.userId
      
if (!isOwnedByPendingUser) {
   throw badRequest('webauthn_unknown_credential', 'That passkey is not registered here.')
}

Picture the attack one more time. An attacker has their own account and their own passkey. They get hold of the victim's pending cookie but sign the challenge with their own Face ID, their own key. Without this check, the server might think "this is a valid assertion for a real, registered passkey" and let them in — as the victim.

The check stops it cold. The asserted credential's userId must equal the pending login's userId. The attacker's key belongs to the attacker, not the victim. Mismatch. Rejected before we even verify the signature. A valid assertion for the wrong account is worth nothing.

Session creation has exactly one home

In the whole 2FA flow, createSession runs in one place — the shared tail both success paths call:

const finishMfaLogin = async (reply, req, rawMfaToken, userId) => {
  await consumePendingMfa(rawMfaToken)   // burn the pending token (single-use)
  clearMfaCookie(reply)                  // drop the half-auth cookie
  const session = await createSession({ userId, ip: req.ip, userAgent: ... })
  setSessionCookie(reply, session.rawToken, session.expiresAt)
  // ...return the public user
}

Passkey login calls it after a verified assertion. Recovery-code login calls it after a consumed code (same shape, minus the signature: consumeRecoveryCode instead of verifyPasskeyAuthentication, then the very same tail). Nowhere else in the flow is a session minted. One door to "you're fully in," and both factors have to walk through it.

Five routes for "log in" is a lot of surface, and it's tempting to read it as over-engineering. But WebAuthn is a challenge/response handshake: the server issues a challenge, the device signs it, the server verifies that exact challenge. That's inherently two round trips per phase — options, then verify — and you can't collapse them without throwing away the replay protection the challenge buys. The endpoint count isn't accidental complexity; it's the protocol's shape made honest. (Putting them in their own twofa-routes.ts plugin instead of piling onto routes.ts is just housekeeping, so the core auth file stays about passwords and sessions.)

Sixth: managing passkeys, and the step-up that guards the off switch

Mara turned on 2FA. Now she lives with it. She buys a laptop and enrolls a second key. She names her phone "iPhone." She sells the laptop and removes its key. One day she wants 2FA off entirely. Every one of those is a management action. Most are ordinary. One is dangerous. This section is about telling them apart.

The ordinary actions

Listing, renaming, removing-one. Mara is signed in; her session proves who she is; the server edits her own rows:

GET    /auth/2fa/credentials       → her passkeys + how many recovery codes are left
PATCH  /auth/2fa/credentials/:id   → rename one
DELETE /auth/2fa/credentials/:id   → remove one (but see below)

Two small things keep these safe. The list never leaks key material — the Security page sees a projection (id, name, backedUp, timestamps), never the public key, counter, or userId, the same discipline as publicUser for accounts. And edits are scoped to the owner: rename and delete both filter by userId, so a credential id alone can't touch someone else's key. If the row doesn't belong to Mara, nothing matches, and she gets a clean "no such passkey" — never a peek at whether that id exists for someone else.

The one dangerous action: dropping to zero

Here's the asymmetry that matters. Removing a passkey when Mara has two is fine — she still has one, 2FA is still on. Removing her last passkey is different: that turns 2FA off. So does the explicit "disable 2FA." Those two — remove-the-last and disable — are the dangerous ones, because they take the account from protected to unprotected.

So DELETE simply refuses to be the off switch:

if ((await countPasskeys(active.userId)) <= 1) {
  throw conflict('last_passkey', 'This is your last passkey. Disable 2FA to remove it.')
}

You cannot quietly delete your way to zero. The last step has to go through /disable, where there's a stronger gate.

Why a valid session is not enough

Picture the attack this gate exists for. Someone steals Mara's live session — a cookie lifted off an unlocked laptop, a hijacked tab. As far as the server can tell, they're signed in as Mara. If "disable 2FA" only needed a session, they'd just turn it off, then change her password, and the second factor that was supposed to protect her is gone — removed by the very session it was meant to backstop.

So disabling 2FA demands something the session-thief doesn't have: a fresh factor, proven right now.

const proven = await proveFreshFactor({ userId, sessionId, proof })
if (!proven) {
  throw forbidden('step_up_failed', 'Confirm a passkey or a recovery code to disable 2FA.')
}
await disableTwoFactor(active.userId)

This is step-up: a valid session gets you to the door, but a sensitive action makes you prove a factor again before it opens. A fresh factor is one of two things, because Mara might be in either situation. A passkey assertion is the clean path — she taps Face ID, signs a fresh challenge, nothing spent; the challenge for it is issued by /stepup/options and keyed to her session, so a challenge minted for one session can't be redeemed by another. A recovery code is the fallback, for when she's disabling 2FA because she lost the device. The session-thief has neither — they hold a cookie, not the phone and not the printed codes. And the assertion path reuses the exact ownership check from the login flow: a fresh factor has to be Mara's fresh factor.

When step-up passes, disableTwoFactor wipes 2FA in one transaction — both webauthnCredentials and recoveryCodes for that user, together, so there's never a half state where the keys are gone but stale codes linger. And because "2FA is enabled" is derived from the credential count, deleting the keys is all it takes to flip the login gate back to single-factor. There's no separate flag to forget. (That design choice from the second section paying off, exactly as promised.)

Step-up is a deliberate speed bump, and the art is putting it only where it earns its friction. Renaming a passkey doesn't get it — relabeling "iPhone" to "Work phone" changes no security posture, so demanding Face ID would be theater. Removing a non-last key doesn't either: 2FA stays on, the blast radius is small. We spend the friction only where the account goes from protected to unprotected. Guard the off switch; leave the light switches alone.

Seventh: the browser's half of the handshake

Everything so far has been the server. Now the part Mara actually touches. She types her password, a passkey prompt appears, she looks at her phone, she's in. Three button-presses of UI on top of all that backend — and one genuinely tricky moment.

The fork that sends her to /2fa

The API can now answer login two ways, and the type says so:

// web/src/lib/api.ts
export type LoginResult = { user: PublicUser } | { mfaRequired: true }

So the login page reads which branch it got and routes accordingly:

// web/src/pages/LoginPage.tsx
const result = await API_login({ email, password })
if ('mfaRequired' in result) {
  navigate('/2fa')   // password was right; the second factor is next
  return
}
await refresh()
navigate('/')         // no 2FA — straight in

Notice what the client does not receive on the mfaRequired branch: no token, no userId, nothing. The pending-MFA cookie was set by the server, httpOnly, invisible to JavaScript. The browser just knows "go to /2fa," and the cookie rides along automatically on the next request. (The invariant again, seen from the client's side: the browser literally cannot name whose login this is, because it was never told.)

The three-step flow

On /2fa, Mara taps "Verify with passkey." That kicks off a handshake the page orchestrates in three moves:

// web/src/pages/TwoFactorPage.tsx
const options = await API_2faAuthenticateOptions()                  // 1. ask the server (network)
const assertion = await startAuthentication({ optionsJSON: options }) // 2. hand to the device (Face ID)
await API_2faAuthenticateVerify(assertion)                          // 3. send the signed result (network)
await finishLogin()

Step 2 is the only line that isn't a network call. startAuthentication is from @simplewebauthn/browser — the wrapper around navigator.credentials.get(), the thing that actually makes the OS show the Face ID sheet, unlock the private key, and sign the server's challenge. That's why the device step lives in the page, not in api.ts: the API_ functions are network calls and nothing else (that prefix is a promise), while the browser ceremony is a different kind of operation, so it sits where the user gesture is.

And it is a gesture — a button, not an auto-run. The page could fire startAuthentication on mount; it doesn't. Browsers gate the WebAuthn prompt behind a real click, partly so a page can't silently pop a credential request the instant you land on it. So /2fa shows a button, and the handshake starts when Mara presses it. The gesture is both a browser requirement and the honest UX: she chose to authenticate.

When the phone is in the lake

Mara might not have her passkey — lost phone, wiped laptop. So the page has a second door: "Use a recovery code instead" swaps the passkey button for a text field, and one of the codes goes to the same finish line, minus the device. Both paths end in finishLogin, which does the one thing the client can do once the cookie is set: re-ask the server who it is.

const finishLogin = async () => {
  await refresh()   // GET /auth/me — the server is the source of truth
  navigate('/')
}

The session cookie is httpOnly — the client can't read it to know "am I logged in now?" So it asks. refresh() pulls /auth/me, the auth context flips to authenticated, and Mara lands on the dashboard.

A passkey login can fail two very different ways, and the page flattens them into one human sentence. An ApiError means the server said no (expired pending login, rejected assertion) — show its message. A WebAuthnError means the browser ceremony broke — Mara hit cancel, the sensor timed out, there's no matching credential on this device. That never reached the server, so there's no server message; the page supplies its own and points her at the recovery-code escape hatch.

The temptation here is to make the frontend smarter — cache the user, decode something, track "2FA pending" in React state. Resist it. The client holds no secret and no authority: it can't read the httpOnly cookies, it can't verify a passkey, it can't decide who's logged in. Its whole job is to route to the right screen, trigger the device prompt on a click, and ask the server what's true afterward. Every time the frontend is tempted to know something about auth, the right move is to ask /auth/me instead. A dumb client is the secure client.

The one place that rule breaks: the show-once problem

Enrolling is the mirror of logging in — same three-step shape, different verb in the middle. startRegistration (the create-time twin of startAuthentication) wraps navigator.credentials.create(): the OS prompts for Face ID, the device generates a fresh key pair in its secure hardware, and hands back the public key. The server stores it. Mara has a passkey.

But when she enrolls her first one, the server turns 2FA on and mints ten recovery codes — and returns them in that one verify response, never again:

const result = await API_2faRegisterVerify({ response, name })
if (result.recoveryCodes !== undefined) {
  setNewCodes(result.recoveryCodes)   // present them NOW; there is no second chance
}

Why only once? Because the server stores codes hashed — it cannot show them again because it threw the originals away on purpose. There is no "resend my codes" endpoint, because there is nothing to resend.

So the UI has a duty the rest of the page doesn't: it must make Mara stop and save them. The codes live in component state, shown in a loud amber box, dismissed only by an explicit "I've saved them." Reload the page and they're gone — exactly as they should be.

This is the one screen where the "dumb client" rule inverts. Almost everywhere in this system, losing a client-side value is harmless — the server is the truth, just ask again. The recovery codes are the single exception: for one render, the client is the only holder of the plaintext, and if the page fails to make Mara save them, nothing else can recover them. So here UX is security. The loud box, the explicit acknowledge, the refusal to tuck the codes behind a reload — those aren't polish, they're the safeguard. Get the show-once moment wrong and you've built a 2FA that quietly locks people out the first time they lose a phone. Everywhere else the client can be dumb; here it has to be insistent.

(Disabling, from the client side, is just the login handshake pointed at a destructive action: stepup/options → startAuthentication → disable, or a recovery code instead. The server enforces that a session alone isn't enough; the page only collects the fresh factor and passes it along. Add, remove, rename, disable all run through one busy/error wrapper, so the page has many actions but one way to be busy and one way to fail.)

The gap I didn't close: Google sign-in doesn't enforce 2FA

Honesty, the same as the first post earned. There's a hole here I left open on purpose, and it's worth naming plainly rather than hoping you don't notice.

A user with a passkey can still log in through "Continue with Google" without the second factor.

The reason is structural. The password path runs through our server — loginWithPassword returns mfaRequired, and the route withholds the session. The Google path is a redirect flow that mints a session directly in the callback (that's the whole OAuth flow from the first post). Gating it means redirecting out of the callback to a frontend /2fa step, carrying the pending cookie, before the session is set — real extra work that I scoped out of this slice.

So the threat model this 2FA actually defends is the password one: a leaked or reused password no longer earns an account on its own. A determined attacker who can complete Mara's Google OAuth is outside what this slice stops. That's a real limitation, not an oversight — and it's exactly the kind of thing that belongs in a post, not buried in a commit. Closing it is a Google-callback interstitial, and it's the natural next increment.

Also deliberately out of scope: passwordless passkey login (this is second-factor only), TOTP as an alternate factor, and admin-mandated enforcement.

An honest word on testing

You cannot drive a real authenticator from a test. The biometric and the secure chip aren't scriptable — there is no way to make Face ID say yes inside bun test. So there is no real end-to-end passkey test, and nothing in this codebase claims to be one.

What is tested is the state machine we own: pending-token issuance, expiry, and single-use; the ownership guard; counter updates; recovery-code consumption; step-up gating; and — added after a reviewer pointed out it had never actually executed — that the rate limit really binds on the doubly-nested /auth/2fa routes. The @simplewebauthn verify is mocked at exactly one boundary (it's injected as a default argument precisely so a test can replace it), and the recovery-code path is the honest no-mock end-to-end for both login and disable — a full round trip through the route machine without a real authenticator.

The passkey-assertion success path is covered at the service layer and named honestly everywhere it appears. The thing left for a human is a five-minute manual smoke on localhost (a WebAuthn secure context): real Face ID enroll → log out → log in with the passkey → disable with step-up. That's the one part the mocks can't prove, and it's flagged as such.

I'd rather tell you the seam exists than paper over it with a test that pretends to drive a sensor it can't.

What you can learn from this

Trace Mara through the second factor and a few ideas keep doing the work.

Nothing secret has to travel. A password is checked by sending it; a passkey is checked without ever moving the private key off the phone. That one difference is the entire reason a passkey survives a phishing site and a typed code doesn't — the browser binds the signature to the origin, and a fake site has nothing to relay.

The first factor is only load-bearing if the server aims the second. The userId for the second factor comes from server-side state, never the request — at the pending token, at every login route, at step-up. Get that one rule wrong and the password becomes decoration; an attacker passes their factor against your account. Four places, one invariant.

The best defenses are still holes you never built. There's no "reset my passkey" because the server never held the key — so there's no reset endpoint to abuse. "2FA is on" is derived from a row count, so there's no boolean to drift. A stolen session can't strip 2FA, because the off switch demands a fresh factor the session-thief doesn't have. You don't add those defenses; you build a shape with no room for the attack.

And the whole machine, in three beats:

The password proves who you are, once — and a passkey-holder gets no session for it alone.
The device signs a fresh challenge your face unlocked but never sent, and the server checks it aims at the right account.
Lose the phone and ten one-time codes get you back; steal the session and the lock still won't come off.

This will take you through the auth core — sessions, login, account linking, password reset — it's built on top of a few vetted packages: argon2id for hashing, arctic for the Google OAuth (Open Authorization) flow, BullMQ for the email queue.

So follow along by tracing one person through the whole machine.

Her name is Mara. She signs up with an email and a password. She logs in. She signs in with Google instead. She verifies her email. Weeks later she forgets her password and resets it. Eventually she links her Google login to her existing account.

Follow Mara, and the rest falls into place.

Two themes will keep coming back, so watch for them:

• The interesting question is almost never "can this be stolen?" — usually the answer is yes. It is "once it is stolen, what can the thief actually do, and can you turn it off?"

• The best defenses are not features you add. They are holes you never built. Several times in this post, a whole class of attack just cannot happen because of the shape of the thing — not because of a check.

Let's go.

First: never store the password

Mara types hunter2-but-longer.

The naive version stores exactly that:

users
  email           password
  mara@work.test  hunter2-but-longer

This is bad.

The day that table leaks — and tables leak — every password is just sitting there. In plain text. Reusable on every other site Mara owns.

So we never store the password.

We store a one-way transformation of it. A hash.

A hash is a function you can run forwards but not backwards.

You can turn hunter2-but-longer into a hash.

You cannot turn the hash back into hunter2-but-longer.

That is the whole point.

// src/auth/password.ts
export const hashPassword = (plainPassword: string): Promise<string> =>
  hash(plainPassword, ARGON2_OPTIONS)

Why argon2id, and not just any hash

Not every hash is safe for passwords.

sha256(password) is a hash (SHA-256, the Secure Hash Algorithm). It is also a terrible password hash.

Because it is fast. A modern GPU (Graphics Processing Unit) computes billions of sha256 per second. An attacker with your leaked table just tries billions of guesses until one matches.

A password hash should be slow. On purpose. Slow enough that a billion guesses becomes impractical.

argon2id is slow on purpose. It is also memory-hard — each guess must allocate real memory, which GPUs hate:

const ARGON2_OPTIONS = {
  memoryCost: 19456, // 19 MiB per hash
  timeCost: 2,
  parallelism: 1,
} as const

19 MiB per attempt. That is nothing for one honest login. It is a wall for an attacker doing billions.

Why a fresh salt matters

Run the same password through the same hash twice and you would normally get the same output.

That is a problem.

If two users both pick password123, identical hashes give it away. And an attacker can precompute a giant table of hash → password once and reuse it against everyone — a "rainbow table."

A salt kills both.

A salt is a random value mixed into each hash, so the same password hashes differently every time.

argon2 generates a fresh random salt per call and stores it inside the hash string. You do not manage it. You can watch it work:

const first = await hashPassword('same input')
const second = await hashPassword('same input')
expect(first).not.toBe(second) // same password, different hash

Same input. Different output. Every time.

To check Mara's password at login, argon2 reads the salt and parameters back out of the stored hash and recomputes:

export const isPasswordCorrect = (storedHash: string, plainPassword: string): Promise<boolean> =>
  verify(storedHash, plainPassword)

The comparison is constant-time, so an attacker cannot learn the password by measuring how long the check takes.

So, where are we: the password runs through the server and never further than the login handler. A string goes in, a longer string comes out — the hash, with salt and params baked in. The hash gets stored; the password never does. A new salt is computed on every signup, a verify on every login. And what gets handed to the next step is a verified identity: this really is Mara.

Second: the cookie is a key, so store the lock, not the key

Mara is who she says she is. Now the server needs to remember that on her next request.

It hands her a token. A long random string. She stores it in a cookie and sends it back every time.

// src/auth/tokens.ts
export const generateToken = (): string => randomBytes(32).toString('base64url')

32 random bytes. 256 bits of entropy. Nobody guesses that.

Here is the tempting, naive version:

sessions
  token                         userId
  k9f2...rawTokenInTheClear...  mara-uuid

Store the token. Mara sends it, you look it up, match found, she's in.

This is bad for the same reason plaintext passwords are bad.

That token is the key to Mara's account. Anyone holding it is Mara. And here you have copied every key into a database — the one thing most likely to leak.

So we do the same trick as passwords.

We store the hash of the token, not the token.

export const hashToken = (rawToken: string): string =>
  createHash('sha256').update(rawToken).digest('hex')

The raw token lives in exactly one place: Mara's cookie.

The database stores only sha256(token):

// src/db/schema.ts — sessions.id is the HASH, not the raw token
id: text('id').primaryKey(), // sha256(rawToken)

Now think about a database leak.

The attacker gets a column of sha256 hashes.

To turn one back into a working cookie, they would have to reverse sha256. They cannot.

A leaked sessions table is a list of useless fingerprints.

This "store the hash, never the secret" move is the spine of this whole project. We will do it again for email-verification tokens and again for password-reset tokens. Same shape every time: the raw secret rides in the link or the cookie; the database keeps only its sha256.

Why sha256 here, but argon2id for passwords

This looks like a contradiction. We just said sha256 is a bad password hash. Now we use it for tokens. On purpose.

The difference is what is being hashed.

A password is low-entropy. Humans pick summer2024. It is guessable, so the hash must be slow to make guessing expensive.

A token is 256 bits of pure randomness. It is not guessable — there is nothing to brute-force. So a fast hash is fine, and fast is good, because we verify tokens far more often than passwords.

Slow hash for the guessable thing.

Fast hash for the unguessable thing.

Why a server session, and not a JWT

Mara has a cookie. The server checks it on every request. But what is actually in that cookie? There are two designs, and they differ in one thing: where the identity facts live, and how the server trusts the cookie each time.

Think of it as a coat-check ticket versus a signed ID card.

Our design — a coat-check ticket. Mara's cookie holds a random, meaningless string:

k9f2x7q...   (43 random chars — says nothing on its own)

It is a ticket number. To learn who it belongs to, the server takes it to the back room and looks up what it points to:

cookie = k9f2x7q...
↓  hash it: sha256(k9f2x7q...)
↓  look that hash up in the sessions table
Postgres:  sha256(k9f2...) → { userId: mara, expiresAt: ... }

The facts live in the database. The cookie is just a pointer to them.

The alternative — a JWT (JSON Web Token), a signed ID card. A JWT cookie holds the facts themselves, encoded, with a signature:

eyJhbGci...  .  eyJ1c2VySWQiOiJtYXJhIn0  .  3aF9c...sig
   header              payload                signature

Base64-decode that middle part and it literally reads:

{ "userId": "mara", "exp": 1699999999 }

Anyone can read it. A JWT is not encrypted — only signed. The server wrote userId: mara and stamped it with a signature only its secret can produce. So on each request the server does no lookup. It recomputes the signature — an HMAC, a Hash-based Message Authentication Code — over header.payload with its secret and checks it matches:

cookie = eyJ...payload...sig
↓  HMAC(secret, header.payload) == sig ?
↓  yes → trust the payload. userId is mara.
(no database — the proof rides inside the card.)

The facts live inside the token. The server stores nothing per session, only its one secret.

One thing to be explicit about, because it is easy to conflate: hashing our session token at rest has nothing to do with JWT. We hash because we store the token, and a stored copy should be useless if the database leaks. A JWT is never stored — there is nothing to hash. Different concern.

Ours:  random token  → STORE its hash, then look it up    (stateful)
JWT:   signed facts   → STORE nothing, just verify math     (stateless)

But can't someone just steal the cookie?

Yes. And here is the honest part: a stolen cookie and a stolen JWT are exactly as bad as each other.

Both are bearer tokens. Whoever holds it, is Mara. Steal her session cookie, or steal her JWT, and the thief is Mara until something stops them.

Our design is not more theft-resistant. A stolen cookie is a stolen cookie.

Can it be stolen?             ours: yes        JWT: yes      (identical)
Can you kill it once stolen?  ours: instantly  JWT: not until it expires

So the interesting question is not whether the token can be stolen — both can. It is whether, once it is stolen, you can turn it off. (Remember the theme.)

One word: revocation.

You might assume a JWT can be logged out. It mostly cannot.

A signed JWT is valid until it expires, because nothing is checking a list. The proof is self-contained. To kill it early you have to bolt a denylist back on — which quietly re-adds the database lookup you went stateless to avoid.

A server session is the opposite. Logout is a DELETE:

// src/auth/session.ts
export const revokeSession = (rawToken: string): Promise<void> =>
  revokeBySessionId(hashToken(rawToken))

The row is gone. The next request with that cookie finds nothing. Mara is out. Instantly. Everywhere that token was used.

For a document-review tool where people share access and need to really be removed, instant revocation is worth one lookup.

Here is the honest tradeoff, stated once so I don't have to keep restating it: if you were building a fleet of stateless services that must verify identity with zero shared database — many microservices, edge functions — a signed JWT earns its keep, and you accept weaker logout. We are a single app with one Postgres and one Redis. Instant revocation matters more than shaving a lookup. So: server sessions. Pick the one that fits.

Third: Redis sits in front, but Postgres is the truth

One lookup per request sounds cheap. At scale it is not — it is a database round-trip on every call, including every WebSocket message later.

So we cache it.

But caching identity is dangerous if you do it naively.

The naive version: store sessions only in Redis.

Fast. Also fragile. Redis is memory; restart it and everyone is logged out. And it makes revocation murky — which copy is the truth?

We avoid that by being strict about one thing:

Postgres is the source of truth.
Redis is only a fast shortcut in front of it.

This is a read-through cache. The flow:

request with cookie
↓
check Redis for this session          (fast path, in-memory)
↓ miss
read Postgres                          (source of truth)
↓ found + not expired
copy it into Redis with a short TTL    (time-to-live, so the next read is fast)
↓
return the user

// src/auth/session.ts — the miss path repopulates the cache
const [row] = await db.select().from(sessions).where(eq(sessions.id, sessionId)).limit(1)
// ...
await redis.set(key, JSON.stringify(cacheValue), 'EX', ttl)

If Redis dies, nothing breaks. Every request just becomes a Postgres read again. Slower. Not broken.

The one honest tradeoff: revoke-all vs the cache

Single logout is instant. We hold the raw token, so we delete the row and delete that exact Redis key:

const revokeBySessionId = async (sessionId: string): Promise<void> => {
  await db.delete(sessions).where(eq(sessions.id, sessionId))
  await redis.del(cacheKey(sessionId))
}

"Log out everywhere" is different. We delete all of Mara's rows in Postgres immediately. But we do not hold every raw token, so we cannot bust every Redis key by hand.

A session cached a second before the revoke could still validate — until its cache entry expires.

So we keep that window short on purpose:

const CACHE_TTL_SECONDS = 60 // bounds how long a cached session can outlive a revoke-all

Sixty seconds. That is the tradeoff, stated plainly: single logout is instant; logout-everywhere is instant in Postgres and lags at most a minute in cache. We chose that over tracking every session key per user. For this app, a one-minute tail on "log out all devices" is fine. If it were a bank, it would not be. (This sixty-second lag comes back later, when a password reset tries to log out a thief.)

Fourth: a free defense you get just from the shape

There is a subtle attack called session fixation.

The attacker plants a known session id on a victim before they log in, then rides that same id after the victim authenticates.

It only works if the session id stays the same across the login boundary.

Ours cannot.

We do not have a session before login to reuse. We mint a brand-new one at the moment login succeeds:

// createSession runs on successful login — fresh random token every time
const rawToken = generateToken()

A fresh token on every login means there is no pre-login id to fix. The defense falls out of the design. We did not add a feature for it; we just never created the hole.

The good ones are often like that. The vulnerability is the thing you didn't build.

Assembling it: one request, four files, one round trip

We have the parts. Now the assembly.

Mara clicks Sign in. Between that click and Welcome back there is one HTTP request, four files, and exactly one round trip.

Everything here runs on the server. The browser's whole job is to send a little JSON and, later, to hold a cookie.

Four steps, always in this order: validate → authenticate → mint → set cookie.

Each hands the next a smaller, more-trusted thing. The body comes in as unknown. The validator turns it into a typed { email, password }. The service turns that into a userId. The session turns the userId into a raw token. The cookie carries the token back to the browser.

The rest of this section is why each step is shaped the way it is.

Step one: the body is guilty until proven typed

The request body is whatever the client sent. It could be anything.

So the first thing the route does is refuse to trust it:

const signupBody = z.object({
  email: z.string().email(),
  password: z.string().min(8, 'Password must be at least 8 characters.').max(200),
  name: z.string().trim().min(1).max(100).optional(),
})

If the email is malformed or the password is too short, the request dies here with a 400 and a field message. It never reaches the database.

The naive version skips this and lets the database be the validator — a NOT NULL blows up somewhere deep, and the client gets a 500 with a stack trace.

A 500 is a bug.

A 400 is an answer.

We want answers. After parseOrThrow(signupBody, req.body), the unknown is gone. Everything past it is typed.

Step two, signup: two rows, or none

Mara is new. Signup has to create two things:

users      who Mara is        (id, email, emailVerified)
accounts   how Mara logs in   (provider='password', password_hash)

One identity. One credential. They only make sense together.

The bad version:

await db.insert(users)...      // succeeds
await db.insert(accounts)...   // throws

Now there is a user with no way to log in. A ghost. And the email is taken, so Mara can't sign up again either.

The fix is a transaction — both rows commit, or neither does:

user = await db.transaction(async (tx) => {
  const [createdUser] = await tx.insert(users).values({ email, name }).returning()
  await tx.insert(accounts).values({ userId: createdUser.id, provider: 'password', passwordHash })
  return createdUser
})

All-or-nothing. No ghosts.

The duplicate-email race

Two browser tabs. Mara double-clicks. Two identical signups arrive at almost the same instant.

The tempting guard is check, then insert:

const existing = await db.select()...where(eq(users.email, email))   // both see "no one"
if (existing) throw conflict()
await db.insert(users)...                                            // both insert

Both requests check before either inserts. Both see an empty table. Both proceed. Now there are two Maras.

This is a race, and you cannot win it with a check — there is always a gap between looking and acting.

So we don't look. We let the database's UNIQUE index on email be the judge. The first insert wins; the second violates the constraint and Postgres raises SQLSTATE 23505 — the SQL (Structured Query Language) standard's code for a unique-constraint violation. We catch exactly that:

const isUniqueViolation = (error: unknown): boolean => {
  if (typeof error !== 'object' || error === null) return false
  const { code } = error as { code?: unknown }
  return code === '23505'
}

The constraint is atomic in a way a check-then-insert can never be. One Mara, always — even under a double-click.

(What happens after we catch that 23505 is a story in itself — a duplicate signup is a privacy leak waiting to happen. Hold that thought; we close it once the email channel exists.)

Step three: the cookie is the only thing that goes to the browser

Authentication succeeded. We mint a session and write the cookie:

reply.setCookie('redline_session', rawToken, {
  httpOnly: true,                              // JavaScript can't read it — XSS (Cross-Site Scripting) can't steal it
  secure: env.NODE_ENV === 'production',       // https-only in prod, but http in local dev
  sameSite: 'lax',                             // not sent on cross-site POSTs — blunts CSRF (Cross-Site Request Forgery)
  path: '/',
  expires: session.expiresAt,                  // dies exactly when the session does
})

Three flags, three different attacks.

httpOnly is for the cross-site script that tries to read document.cookie.

sameSite: 'lax' is for the cross-site form that tries to POST as Mara.

secure is for the network eavesdropper — off in dev only because there's no https there.

The raw token is the only secret that ever leaves the server. Everything else — the hash, the user row, the session row — stays put.

Step four, later: the cookie becomes a user

Mara loads a page. The browser attaches the cookie. GET /auth/me runs:

const rawToken = req.cookies[SESSION_COOKIE_NAME]
if (rawToken === undefined) throw unauthorized(...)      // no cookie → not signed in

const active = await getSessionUser(rawToken)            // Redis → Postgres read-through
if (active === null) throw unauthorized(...)             // expired or revoked → not signed in

It is the inverse of login. Login turned a userId into a token. /me turns the token back into a userId.

And logout shrugs:

if (rawToken !== undefined) await revokeSession(rawToken)   // kill the row + bust the cache
clearSessionCookie(reply)                                   // always
return reply.code(204).send()                               // always

Logout is idempotent. Log out twice, or log out with no session at all, and the answer is still 204. Logout is a cleanup action, and a cleanup action that can fail is a worse experience than one that can't. There's no security in making "log out when you're already logged out" an error. So it isn't one.

The leak hiding in the error message

Now Mara comes back a week later and logs in. This is where it gets interesting.

A login can fail two ways:

the email isn't registered
the password is wrong

The obvious thing is to say which:

404  no account with that email
401  wrong password

That is user enumeration, and it is a real leak.

Watch what an attacker does with it. They don't know who banks here, or whose documents are on redline. So they probe:

POST /auth/login  ceo@bigco.com  / anything   →  404   (not a user — move on)
POST /auth/login  mara@work.test / anything   →  401   (a user! now brute-force her)

The 404 vs 401 is the answer to "does this person have an account here?" — which on a private tool is itself sensitive.

So login gives one answer for both cases:

const invalidCredentials = (): never => {
  throw unauthorized('invalid_credentials', 'Email or password is incorrect.')
}

"Email or password." Same status, same body, whether the email is unknown or the password is wrong. The response stops being an oracle.

The leak you can't see: timing

Make the bodies identical and you've closed the visible leak. But there's a second one, and it hides in the clock.

Look at the honest two-branch code:

no such account   →  return immediately
wrong password    →  run argon2id verify (~50ms), then return

The bodies match. The timing doesn't.

A 401 that comes back in 2ms means "no such account." A 401 that takes 50ms means "real account, wrong password." The attacker times the response and reads the difference. Same leak, through a side door.

The fix is to make the missing-account path do the same expensive work:

if (account === undefined || account.passwordHash === null) {
  await verifyAgainstDecoy(input.password)   // burn ~50ms against a throwaway hash
  return invalidCredentials()
}

verifyAgainstDecoy runs a real argon2id verify against a hash of a dummy password. It always fails, but that's not the point — the point is it takes the same time. Now both paths cost ~50ms, and the clock says nothing.

This timing-equalization with a decoy hash is a move we'll make again at signup. Burn the same work on the path that has no real work to do, so the duration carries no signal.

Notice account.passwordHash === null sits in the same branch as "no account." A Google-only user has a row, but no password. The naive code finds the row, calls verify(null, password), and argon2id throws on a null hash — a 500. Which, to an attacker, is a third distinct answer. Folding null into the decoy branch keeps the three cases — no email, no password, wrong password — indistinguishable.

One honest limit while we're here: timing equalization is approximate, not perfect. A determined attacker with a clean network path and thousands of samples can still tease out a statistical difference. The strong mitigation against that isn't timing — it's the rate limiter, which caps how many samples they can take. We build that later.

Google sign-in, and the link you must not make

There are three parties in this story, not two.

Mara's browser. Our server. And Google.

Password login was a conversation between two of them — the browser tells our server a secret, our server checks it. Google sign-in is a flow between all three, where our server never sees Mara's Google password at all.

The interesting part isn't the flow. arctic handles the steps. The interesting part is the very last decision: Mara shows up with a verified Google email, and there's already an account here with that email. Do we merge them?

Get that wrong and you hand one person's account to another.

Let's earn our way to that decision.

The base flow: what actually happens

Forget security for a minute and just watch Mara sign in.

The goal is simple: Google vouches for Mara to us, and we never see her Google password.

But Google can't call our server out of the blue. The only thing that touches both Google and our server is Mara's browser, bouncing between them. So the whole flow is built out of browser redirects.

Three beats:

1. Mara clicks "Sign in with Google."
   Our server bounces her browser over to Google.
   ↓
2. Mara logs in at Google and approves.
   Google bounces her browser back to us — carrying a code.
   ↓
3. Our server takes that code and calls Google directly:
   "who does this code stand for?"
   Google answers with Mara's identity.

That code is the pivot of the whole thing.

It's a one-time ticket Google hands out that means "a real user just approved." Step 3 redeems the ticket for Mara's actual identity — and that redemption is a server-to-server call, our server straight to Google, with no browser in the middle.

That's the entire flow. It works. It's also broken in two ways — and the two cookies we set on the way out (state and code_verifier, which the code spells codeVerifier) are the two patches. Everything else in this section is just those two patches.

OUT     (our redirect  → Google):  state + code_challenge
BACK    (Google        → us):      code + state   (state echoed)
DIRECT  (our server    → Google):  code + code_verifier

state makes a round trip. code comes back. code_verifier only ever leaves at the very end, and never through the browser. The two patches below are just why each value travels the way it does.

The same flow, with real values

Names blur together. Values stick. So let's give all four concrete (made-up but realistic) values and watch where each one physically is at every hop.

state          = "x7Kp9w"             — random, WE make it
code_verifier  = "tZ7n_K2pQ9xR4mL8"   — random secret, WE make it
code_challenge = "E9f2b7c1aZ…"        — SHA-256 of the verifier, WE make it
code           = (doesn't exist yet)  — GOOGLE mints it in Hop 2

Three of the four exist before we ever contact Google — we make them. code is the odd one out. This is the bit that trips people up: we never send Google a code. We send state and code_challenge; Google hands a code back.

Hop 1 — we send Mara's browser to Google.

Not to our callback (that's the trip back, in Hop 3). We build a URL to Google's sign-in endpoint and redirect her there:

302 → https://accounts.google.com/o/oauth2/v2/auth
        ?client_id=redline.apps.googleusercontent.com
        &redirect_uri=https://redline.app/auth/google/callback
        &response_type=code
        &scope=openid email profile
        &state=x7Kp9w               ← in the URL
        &code_challenge=E9f2b7c1aZ…  ← in the URL (the HASH)
        &code_challenge_method=S256

In that same response, we set two cookies on Mara's browser:

Set-Cookie: g_state=x7Kp9w;              HttpOnly
Set-Cookie: g_verifier=tZ7n_K2pQ9xR4mL8; HttpOnly

So right now, here's where everything is:

URL → Google (public):    state, code_challenge(the hash)
Mara's cookies (httpOnly): state, code_verifier(the secret for that hash)

code_verifier (the secret) sits in a cookie and appears in no URL. Only its hash travelled. That one split is all of PKCE (Proof Key for Code Exchange — is pronounced "pixie.").

Hop 2 — Google's turn.

Google shows Mara the consent screen; she clicks Allow. Google now:

holds  state          = x7Kp9w        → will echo it back
files  code_challenge = E9f2b7c1aZ…   → pinned to the code below
mints  code           = 4/0AfJ…       → a fresh one-time ticket

What Google never received: code_verifier. It only has the hash.

Hop 3 — Google sends Mara's browser back. This is the callback:

302 → https://redline.app/auth/google/callback
        ?code=4/0AfJ…       ← Google's ticket
        &state=x7Kp9w       ← our state, echoed back unchanged

Mara's browser still carries the Hop-1 cookies, so our server now holds both:

from the URL:     code = 4/0AfJ…     state = x7Kp9w
from the cookies: verifier = tZ7n…   state = x7Kp9w

Gate 1 — our server checks:

state in URL (x7Kp9w)  ===  state in cookie (x7Kp9w) ?
yes → this callback answers the sign-in WE started. Continue.

Hop 4 — we redeem the code, server-to-server (no browser):

POST https://oauth2.googleapis.com/token
        code=4/0AfJ…
        code_verifier=tZ7n_K2pQ9xR4mL8   ← secret's first time out
        client_id=…  client_secret=…
        grant_type=authorization_code

Gate 2 — Google checks:

SHA-256("tZ7n_K2pQ9xR4mL8")          = E9f2b7c1aZ…   (recomputed now)
the challenge Google filed in Hop 2  = E9f2b7c1aZ…
match → whoever holds the verifier started this. Here's the id token.

Where each value lived, all at once:

With those values in hand, both attacks below are easy to see:

Thief reads the Hop-1 URL → gets state=x7Kp9w, code_challenge=E9f2b7c1aZ…
   state: useless without Mara's httpOnly cookie (can't be written).
   challenge: a hash — won't reverse to tZ7n_K2pQ9xR4mL8.

Thief steals code=4/0AfJ… from a leaked Hop-3 URL → stuck at Gate 2:
   it demands code_verifier=tZ7n_K2pQ9xR4mL8, which only ever lived
   in Mara's httpOnly cookie and never rode any URL.

The next two sections are just those two attacks, spelled out.

state: proving the callback answers the request we started

Look at step 2 again. Google sends Mara's browser back to us with a code in the URL (Uniform Resource Locator):

GET /auth/google/callback?code=…

The hole: anyone can point Mara's browser at that callback URL.

So an attacker starts their own Google sign-in and gets a valid code for their account. They don't redeem it. They trick Mara's browser into hitting our callback with their code:

/auth/google/callback?code=THE-ATTACKERS-CODE

If our server redeems whatever code shows up, it logs Mara into the attacker's account — and everything she uploads or comments now sits in an account the attacker can read. (This is CSRF — Cross-Site Request Forgery — a forged request riding in on the victim's browser.)

The fix: refuse any callback that can't prove it's finishing the sign-in we started for this browser. That is state's whole job:

if (query.data.state !== cookieState) {
  throw badRequest('oauth_state_mismatch', 'Google sign-in could not be verified.')
}

The attacker's forged callback carries the attacker's state — but Mara's browser never received a cookie for that sign-in. The two don't match. Rejected.

state is the thread tying who started this to who's finishing it.

PKCE: making a stolen code worthless

state proved the callback is ours. PKCE protects the code itself.

Remember how the code travels: Google → browser → us, sitting in a URL. URLs leak — server logs, browser history, a shoulder-surfed address bar. So assume the code can be stolen in transit. If a stolen code were enough to redeem, the whole thing falls over.

So redeeming a code has to require a second thing — a secret that only the browser which started this sign-in could hold. That second thing is code_verifier.

The trick is that there are two different roads to Google — one through Mara's browser, one a direct server-to-server HTTPS (HTTP Secure) call:

Front channel — through the browser (redirect URLs — these leak)
Back channel  — our server → Google directly (never via browser)

PKCE keeps the secret off the leaky road. We generate the secret, send Google only a hash of it up front, and reveal the secret itself only on the private road, at the very end:

code_verifier   = a long random secret   (stays in our cookie)
code_challenge  = SHA-256(code_verifier) (just a hash — shareable)

The verifier is the key. The challenge is a photo of the key's shape — enough to check a match, not enough to cut a copy.

Up front, as the sign-in starts, we compute the challenge locally and put it in the redirect URL:

const url = google.createAuthorizationURL(state, codeVerifier, GOOGLE_SCOPES)

That URL carries the challenge, never the verifier. Mara's browser delivers it to Google (front channel), and Google files it against this request before handing back a code.

At the end, to redeem the code, our server makes its one direct call — sending the verifier for the first and only time:

const tokens = await google.validateAuthorizationCode(code, codeVerifier)

This is the back channel. Google computes SHA-256(verifier) and checks it equals the challenge it filed earlier. Match → tokens. Mismatch → rejected.

So the two halves reach Google by two different roads, and only the harmless one ever rides the leaky channel:

challenge (hash)    → browser → Google   — up front
verifier  (secret)  → server  → Google   — only at the very end

Now the attacker who scraped the code from a leaked URL tries to redeem it. Google asks for the verifier:

Not in a URL   — it was never put in one.
Not in reach   — the cookie is httpOnly, on Mara's machine.
Not reversible — SHA-256 doesn't run backwards.

No verifier, no exchange. The stolen code is dead. A code without its verifier is a key without its cut.

What comes back: an id token, not a password

Look closer at that id token. It's a JWT whose payload is a small bag of facts about Mara, signed by Google:

{ "sub": "11029...", "email": "mara@work.test", "email_verified": true, "name": "Mara" }

sub is Google's permanent, unique id for Mara — stable even if she changes her email. That's what we store as her Google identity, not the email.

We got this token directly from Google's token endpoint, over TLS (Transport Layer Security). So we decode the claims rather than re-verifying the signature — the secure channel already proved where it came from:

rawClaims = decodeIdToken(tokens.idToken())

Then we parse it strictly. email_verified must be a real boolean. If Google ever sent something surprising, the parse fails and the whole sign-in fails — closed:

const googleIdTokenClaims = z.object({
  sub: z.string().min(1),
  email: z.string().email(),
  email_verified: z.boolean().optional(),
  // …
})

Fail-closed is the rule whenever the question is "is this email proven?" A wrong "no" is an inconvenience. A wrong "yes" is the bug in the next section.

The decision: create, link, or refuse

Now we have a verified identity in hand. Three cases.

Case 1 — a returning Google user. The sub already maps to one of our users. Found it, open a session, done. The common path.

Case 3 — a brand-new person. No Google identity, and no local account owns this email. Create the user and the Google credential together, in one transaction. Google already verified the email, so the new user inherits that.

I'm skipping Case 2 on purpose. Case 2 is the whole point.

Case 2: the email is already taken

Mara signed up weeks ago with mara@work.test and a password. Today she clicks "Sign in with Google," and her Google email is the same address.

No Google identity yet. But a user row already owns mara@work.test.

The tempting move:

Google says the email is verified, and it matches. Link them. Log her in.

Here's why that's a door you must not open.

mara@work.test being in our users table does not mean Mara put it there.

Our signup lets anyone type any email. It doesn't prove ownership. So picture this, before Mara ever shows up:

1. Attacker signs up with email = mara@work.test, password = (their own).
   Our users table now has a row for mara@work.test. emailVerified = false.
   (It was never proven — nobody clicked a link.)

2. The real Mara later clicks "Sign in with Google."
   Google says: mara@work.test, email_verified = true. It matches!

3. Naive rule auto-links Google to that existing row.
   Mara is now sharing an account with the attacker —
   who still knows the password.

The attacker is now inside Mara's account. They planted the email; the verified Google login walked right into the trap.

So matching-and-Google-verified is not enough. The fix is to demand proof from both sides:

const bothEmailsVerified = claims.emailVerified && existingUser.emailVerified
if (!bothEmailsVerified) {
  throw conflict('account_exists', 'An account with this email already exists. Sign in with your password to link Google.')
}

Google-verified and the local account already verified. The attacker's planted row was never verified, so the guard refuses to link and the takeover never happens.

You might assume "verified email matches" is the linking rule. It is half of it. The other half is whose verified — and a local row you can't trust doesn't count.

There's a catch, and it's worth being honest about: when this guard first landed, nothing could produce a verified local account. There was no way to verify a password account yet, so every "I have a password account, now I'm adding Google" attempt hit the 409. The guard was correct, but in practice refuse-only.

That's fine. Refusing safely while the verified state is unreachable beats linking unsafely. And it's exactly the gap the next section fills — the guard doesn't move; the world catches up to it.

A few limits worth naming before we move on. This is one provider: the whole module is Google-shaped — one sub, one id token, Google's verified-email semantics. A second provider (GitHub) doesn't hand you a reliable email_verified, and the linking rule would change. The refusal is a JSON 409 — fine for a backend, but a browser mid-redirect deserves a real "you already have an account, here's how to link" page; that's frontend work. And decode-don't-verify rests on one assumption: the id token came straight from Google over TLS. It does here. Accept an id token from somewhere you didn't fetch yourself and you must verify the signature.

Email verification: a claim is not a proof

Signup asked Mara for an email. It never asked the email whether it wanted Mara.

Anyone can type anyone's address into a signup form. A row in our users table that reads mara@work.test is a claim — "someone said they own this" — not a proof. We just saw the damage when you treat the claim as proof: an attacker registers a victim's address, and a later real Google login links straight into the attacker's account.

Verification turns the claim into a proof. The mechanism is one link, clicked once.

users.emailVerified is a single boolean. The job of this whole piece is to make sure it only ever flips to true for someone who can actually read mail at that address.

So we send something to the address and check whether it comes back. That something is a token — and it's the session token in a different hat. The moves are identical to the ones we already built:

generate a long random token   — 32 bytes, unguessable
store only its sha256 hash      — the id column of email_verification_tokens
put the RAW token in the link   — the only copy that leaves the server
give it an expiry               — 24 hours

const rawToken = generateToken()
await db.insert(emailVerificationTokens).values({ id: hashToken(rawToken), userId, expiresAt })

The raw token rides in the email. The database keeps only sha256(rawToken). A leak of that table hands an attacker a column of hashes that open nothing — the same reason session tokens are stored hashed.

The bad version makes the point: email a link like ?verify=user_42. Now anyone verifies anyone by editing the URL. The token is long and random because it has to be the proof — unguessable and unforgeable, not just an id.

The flow

Signup                                                          (server)
↓
issue token: store sha256(token), email the raw token in a link (server)
↓
Mara opens her inbox and clicks the link                        (client)
↓
GET /auth/verify?token=…                                        (server)
↓
sha256 the token, look it up, check expiry
↓
flip emailVerified = true, delete the token, redirect to the app

Server-side at every step except the click. The only thing stored is the hash, plus the flag once it flips.

Single-use, and why the row is deleted

await db.transaction(async (tx) => {
  await tx.update(users).set({ emailVerified: true }).where(eq(users.id, row.userId))
  await tx.delete(emailVerificationTokens).where(eq(emailVerificationTokens.id, id))
})

The flip and the delete commit together. A token that already verified someone no longer exists, so the same link can't be replayed into a second verification.

That deletion shapes the error messages. A verify fails two ways the user can tell apart:

expired   — the link is older than 24 hours
invalid   — unknown, or already used

It cannot separate "already used" from "never existed" — a used token was deleted, so there's no row left to distinguish it. Both return invalid_verification_token. Keeping a tombstone that said "this one was used" would be an oracle: a way to probe which tokens once existed. Forgetting is safer.

What this makes possible

The flag isn't decoration.

The Google guard from the last section becomes reachable. A verified password account can now merge with a matching verified Google login — the exact precondition the takeover guard was waiting on. The guard didn't change; verification is what made its success path possible.

And a uniform signup response becomes possible — answering "check your email" identically whether the address is new or already registered, moving the real signal into the inbox instead of the HTTP status. That's the next section, and it's how we finally close the duplicate-signup leak I flagged earlier.

When this is the wrong shape

A GET that changes state will be clicked by robots. The verify link is a plain URL in an email, so the browser fetches it with a GET — and so does every link scanner that touches the message first: corporate mail filters, antivirus, chat-app link previews. That bot's GET hits /auth/verify, flips the flag, and deletes the token before Mara clicks. She ends up verified, but her real click then shows "invalid or already used." It's confusing, not a hole — the scanner can only verify the address it was already trusted to handle. The standard fix is a two-step page: the GET renders a "Confirm" button and a POST does the mutation, which scanners don't follow. That's frontend work, and the wart is inherent to a link that both arrives by GET and changes state.

Unverified users can still log in. We don't gate basic access on verification; an unverified Mara can sign in and look around. Verification gates linking, not the front door. That's a policy choice — an app moving money would gate more.

A token sent is a question: can you read this? A token returned is the answer: yes. The flag it flips is the difference between a claim and a proof.

How mail actually leaves the process

Twice now I've said "we email the user." Never how an email leaves the process and lands in an inbox.

This is that part. And the interesting thing is that "send an email" means three completely different things depending on where the code is running.

Start with the bad version

The obvious way to send mail is to call the provider right where you need it:

// in the signup handler
await resend.emails.send({ from, to, subject, html })

Three things break.

Your tests now hit the network — or you mock the Resend SDK (Software Development Kit) in every test that touches signup.

A slow provider stalls the signup request, because the user is waiting on an API (Application Programming Interface) call to a third party.

And you've welded the signup flow to one vendor. Swap Resend for SES (Amazon's Simple Email Service) and you edit every call site.

The fix is a seam. One function, sendEmail, that every caller uses — and behind it, a transport chosen by environment.

The seam

Everything that sends mail calls exactly this:

sendEmail({ to, subject, text })

A recipient, a subject, a body. No provider, no SMTP (Simple Mail Transfer Protocol), no SDK in sight.

Who calls it? Never the route directly. Two small templates do, each owning one message:

// src/auth/verify.ts — these decide WHAT to say
sendVerificationEmail(to, rawToken)   // "Confirm your email: <APP_URL>/auth/verify?token=…"
sendAccountExistsEmail(to)            // "You already have an account — just log in."

So the layers are clean:

signupWithPassword        WHEN to send
  ↓
sendVerificationEmail     WHAT to say    (subject + body + the link)
  ↓
sendEmail                 WHERE to send  (picks the transport)
  ↓
the transport             HOW it travels (memory, or an SMTP socket)

Each layer knows only the one below it. The template doesn't know about SMTP. The route doesn't know about either.

Three transports, chosen by NODE_ENV

Here is the part that surprises people. sendEmail does three different things, and the only input that decides which is NODE_ENV:

sendEmail(message)
├─ test         → push onto sentEmails[]        (no network at all)
├─ development  → SMTP → Mailpit                (localhost:1025, viewable at :8025)
└─ production   → SMTP → a real provider        (SMTP_HOST / USER / PASS from env)

export const sendEmail = async (message) => {
  if (env.NODE_ENV === 'test') {
    sentEmails.push(message)   // an array in memory
    return
  }
  await mailer().sendMail({ from: env.EMAIL_FROM, to: message.to, ... })
}

Under test, there is no email. sentEmails is a plain array. A test signs up, then reads the array back to find the link that was "sent":

const link = sentEmails.find((m) => m.to === email)?.text.match(/token=(\S+)/)?.[1]

That's how the verification end-to-end test gets the token without a mail server. No network, no flake, instant.

Under dev and prod, it's the same code — mailer().sendMail(...) over SMTP. The only difference is where the socket points, and that's pure config, not code. Dev points at Mailpit: a docker container that speaks SMTP like a real server but never delivers anything — it just catches every message and shows it at http://localhost:8025. You sign up, switch tabs, and there's the verification email — click the link, you're verified. A real round trip, nothing leaving your laptop. Prod points those same env vars at a provider that does deliver.

The connection is a singleton

mailer() doesn't build a new connection per email:

const mailer = () => {
  globalForMailer.mailer ??= createTransport({ host: env.SMTP_HOST, port: env.SMTP_PORT, ... })
  return globalForMailer.mailer
}

One transporter per process, cached on globalThis so dev hot-reload reuses it instead of leaking a socket each reload — the same pattern as the DB (database) pool and the Redis client. It's also lazy: createTransport opens no socket; the connection happens on the first sendMail. So importing the module costs nothing — the import stays side-effect-free even though the module owns a connection. (Remember this singleton-on-globalThis shape; the queue uses it next.)

A failed send must not fail the signup

One rule the pipeline has to honor: the user row is already committed by the time we send. If the mail throws, the signup must still succeed.

Two reasons. First, it would be absurd to destroy an account because a third-party mail API hiccupped. Second — and sharper — on the duplicate path a thrown send would change the response, and that would reopen the enumeration oracle we're about to close: success returns 200, a throw returns 500, and now new-vs-taken is distinguishable again.

So both sends go through a best-effort wrapper:

const sendSignupEmail = async (send) => {
  try { await send() }
  catch (error) { console.error('[signup] email failed to send; signup still succeeds', error) }
}

Both call sites use it identically, so their failure behavior can't diverge. sendEmail itself still throws honestly; it's the signup flow that chooses to swallow, because only it knows the row is already there.

The problem this creates

The send is synchronous. Mara's signup waits for sendMail to finish. Mailpit is instant and the fail-soft wrapper means a slow provider won't fail the request — but it can still slow it.

Worse, look again at what fail-soft actually does on a failure: it logs and moves on. Signup succeeds — good. But the verification link is now gone. One attempt, it failed, nothing tries again. Mara sits at "check your email" forever, staring at an inbox that will never receive anything.

That wrapper protected the signup by sacrificing the email. The fix is a queue. That's the next section.

The email queue: a failed send must not lose the link

The whole idea fits in one swap. Instead of sending the email at signup time, enqueue it:

BEFORE   signup → sendEmail() → SMTP → (provider hiccups) → log + drop. Link lost.

AFTER    signup → enqueueEmail() → Redis. Returns in ~1ms.
                                     ↓
         worker → sendEmail() → SMTP → (hiccups) → BullMQ retries with backoff
                                     ↓ (eventually succeeds)
                                   delivered

Two properties fall out.

Fast. enqueueEmail touches only Redis (local, ~1ms) and returns. Signup no longer waits on a third party at all — not even the instant Mailpit case.

Durable. The job lives in Redis. If the first SMTP attempt fails, BullMQ re-runs it later — 5 attempts, exponential backoff. A transient provider outage costs minutes of delay, not a lost account.

The producer side

enqueueEmail is the new seam the app calls. It replaces the direct sendEmail at the two template sites:

export const enqueueEmail = async (message) => {
  if (env.NODE_ENV === 'test') {
    await sendEmail(message)   // no worker under test — deliver inline (see below)
    return
  }
  await emailQueue().add(EMAIL_JOB_NAME, message)
}

emailQueue() is the same singleton pattern as the DB pool, the Redis client, and the mailer — one Queue per process, cached on globalThis, built lazily so the import stays free. The job payload is just the { to, subject, text } from before; the template still decides what to say, the queue is purely how it travels now.

The consumer side — and the one bug that matters

The worker drains the queue and delivers each job. The handler is three lines, and one of them is load-bearing:

export const createEmailJobHandler =
  (deliver) =>
  async (job) => {
    await deliver(job.data)   // throws → BullMQ retries. Do NOT catch here.
  }

Here is the trap. Your instinct, fresh off the fail-soft wrapper, is to wrap this in a try/catch — "emails shouldn't crash things." That instinct breaks the entire feature.

A BullMQ job is retried only if it rejects. If the handler catches the SMTP error and returns normally, BullMQ sees a job that completed successfully, drops it, and never retries. You'd have built a queue that swallows failures exactly as silently as the thing it replaced — except now it looks robust. The whole point is that sendEmail throws, and that throw must travel all the way up to BullMQ.

So fail-soft does NOT live in the worker. It moved to wrap only the enqueue:

producer (signup):   try { await enqueueEmail(...) } catch { log }   ← guards a rare Redis hiccup
worker (delivery):   await deliver(job.data)                          ← MUST reject so BullMQ retries

The producer still can't fail signup or reopen the enumeration oracle — but now it's guarding a local Redis write (almost never fails), not a flaky third-party SMTP call. The flaky part got moved to where retries live.

This is the one thing worth a unit test, and it's the cheapest test in the suite:

const handle = createEmailJobHandler(async () => { throw new Error('smtp down') })
await expect(handle({ data: message })).rejects.toThrow('smtp down')   // proves: rejects → will retry

Delivery is injected into the handler precisely so this test can drive a failure without a real mail server. We don't test BullMQ's backoff — that's the library's job. We test our wiring: a failed delivery rejects.

Test mode still has no worker

Under the test runner there's no worker process running, so a queued job would sit in Redis forever and the auth end-to-end test — which reads the verification link back out of that in-memory sentEmails array — would hang.

So enqueueEmail keeps the same NODE_ENV fork: under test it delivers inline, straight to sentEmails. The queue is real in dev and prod; in test it's transparent. The test never knew the queue arrived.

Two ways to run the worker

In production the worker is its own process — a separate container from the API. That's the entire point of the deploy constraint I set for this project: a persistent worker, not serverless, so a slow mail provider ties up a worker, never an API request handler.

bun run src/worker.ts        # prod: the standalone consumer

But forcing two terminals in dev would silently break the signup→Mailpit flow the moment you forget the second one. So in development the API boots the same worker in-process:

// server.ts, after listen()
if (env.NODE_ENV === 'development') {
  createEmailWorker()   // one `bun run api` still delivers email end to end
}

Same worker module either way. Dev co-locates for convenience; prod splits for isolation. The split is config-shaped, not a code fork.

Pass connection config, not a client

BullMQ's connection accepts either a live ioredis instance or a plain options object. Hand it an object:

new Queue(EMAIL_QUEUE_NAME, { connection: queueConnectionConfig() })  // { host, port, maxRetriesPerRequest: null, … }

…and BullMQ builds and owns its own connections from it. Two payoffs.

First, a blocking worker needs its own connection — a BRPOPLPUSH that waits indefinitely for the next job would stall anything sharing the socket — so letting BullMQ create them means you never accidentally share one.

Second, and subtler: BullMQ bundles its own copy of ioredis, often a different version than the app's. A Redis instance from one copy is a different class than the other's, so passing your app's client across that line is a type error (and an instanceof hazard at runtime). A plain options object has no class identity — it crosses cleanly.

The one option that matters is maxRetriesPerRequest: null, and it lives only on BullMQ's connections. The app's shared client keeps ioredis's default, so an ordinary cache GET fails fast instead of hanging forever waiting on a dead Redis.

BullMQ itself is the deliberate choice here, same reasoning as argon2id and arctic: a reliable queue (atomic claim, visibility timeout, backoff, dead-lettering) is genuinely hard to get right, so you stand on a vetted primitive rather than hand-roll the core.

When this is the wrong shape

The commit-and-enqueue gap. Signup commits the user row, then enqueues. If the process dies in the microsecond between, the row exists and no email job does — the same lost-link symptom, just far rarer. Closing it needs a transactional outbox: write the job to Postgres in the same transaction as the user, and a relay moves it to Redis. That's the next tier of durability and deliberately not built here — the failure window went from "any SMTP hiccup" to "a crash in a one-instruction window," which is the right amount of hardening for this slice.

The raw token rides in the job. Only the token's hash is in the DB, but the queued job carries the raw token in its payload, briefly, in Redis. Acceptable — Redis is trusted infra and the job is dropped on delivery — but it's a wider blast radius than the DB, worth a note for a stricter threat model.

Uniform signup: the same answer, whether you exist or not

Now I can pay off the debt I've flagged twice.

Login is enumeration-proof: one answer whether or not the email exists. Signup was not. It answered two different ways:

POST /auth/signup  { new@x.com }     → 201 Created   + a session cookie
POST /auth/signup  { taken@x.com }   → 409 Conflict  { "error": "email_taken" }

The 409 says, out loud: someone already has this email here.

That's the same leak we closed on login — just on the other door. An attacker with a list of ten thousand addresses POSTs each one and writes down the status code. Every 409 is a confirmed member.

On a public site, membership is no secret. On redline — a private tool for reviewing confidential documents — who is on it is the client list. The attacker reads it off the status codes, no password-guessing required.

The fix: one answer for everyone

Both paths now return the identical response:

POST /auth/signup  { new@x.com }     → 200  "Check your email to finish signing up."
POST /auth/signup  { taken@x.com }   → 200  "Check your email to finish signing up."
                                        ↑ byte-for-byte the same

Same status, same body, no error field. The attacker's whole list comes back as 200s. Nothing to write down.

The response can't carry the new-vs-taken signal now — so it goes where the inbox already is, a channel only the address owner can read:

new email    → the verification email          ("Confirm your email: <link>")
taken email  → one new message                  ("You already have an account — just log in.")

In code, the duplicate that used to throw now sends instead:

} catch (error: unknown) {
  // Was: throw conflict('email_taken'). Now: tell the real owner by email, return the uniform 200.
  if (isUniqueViolation(error)) {
    await sendAccountExistsEmail(email)
    return
  }
  throw error
}

The attacker can hit the endpoint ten thousand times and can't read one of those inboxes. The signal is real; it just travels where they can't follow.

The consequence: signup stops logging you in

This is the part that isn't obvious, and it's the actual cost.

The old new-email response set a session cookie — signup logged you straight in. A uniform response can't.

Because the taken path has no session to grant. That account isn't yours; you proved nothing by typing its address. So watch what happens if only the new path keeps its cookie:

new email    → 200 + Set-Cookie: session=...
taken email  → 200   (no cookie)
                ↑ the cookie is the tell again

The presence of the cookie becomes the difference the attacker was hunting. We'd have moved the leak from the status code into the Set-Cookie header and called it fixed.

So neither path sets one.

before:  sign up ──────────────────► you're in
after:   sign up → check email → log in → you're in

Signup says "check your email," cookie-free, every time. One extra step on the happy path, paid because "sometimes a session, sometimes not" is exactly the oracle we're closing.

Then why does Google signup still log you in?

It does — and that looks like a contradiction. It isn't, because the two reasons password signup had to give up auto-login don't exist in the Google flow.

No enumeration vector. To even reach the "do you already have an account?" branch, you must first complete the OAuth handshake — prove to Google you control that account. An attacker can't probe alice@gmail.com without being alice at Google. The password leak was an unauthenticated request revealing membership; by the time Google's callback runs, the request is already authenticated. No anonymous oracle, so withholding the session would buy nothing.

The email is already proven. Password signup also couldn't log you in because the address was an unverified claim. Google sends email_verified, and a new user inherits it. Ownership is already proven, so there's no "check your email" to wait on.

So password signup withheld the session because the request was anonymous and the email unproven. Google signup is neither — so it logs you in, and that's correct, not an exception.

The second tell: time

Identical bodies aren't enough; the clock can still leak — the same timing problem login had.

argon2id hashing is deliberately slow (~50ms) and is the dominant cost of a signup. If the taken path skipped it — hashing only when it's about to create the account — it would reply ~50ms sooner, and the attacker just stops reading the body and starts timing:

fast reply  → email is taken
slow reply  → email is new

So we hash on both paths, before the insert is even attempted. The taken path hashes a password it throws away — the same timing-equalizer move login makes with its decoy hash. A future reader who "optimizes" the hash to after the insert would silently reopen the oracle, which is why the code says so at that line.

The whole flow

POST /auth/signup
↓
hashPassword(password)  ~50ms            (BOTH paths — timing equalizer)
↓
try: insert users + accounts in one tx
│
├── success ──► verification email ───────────┐
│                                             │
└── 23505  ──► sendAccountExistsEmail() ──────┤
                                              ↓
                200 "Check your email to finish signing up."
                (identical body, no cookie, either way)

Two paths in, one answer out. This is worth keeping only where who is registered is itself confidential — a private review tool, a medical portal, an internal admin. If membership isn't a secret — a forum, a game — a plain 409 "email taken" is friendlier and costs new users no extra step. redline is the confidential kind, so it pays the step.

The endpoint tells everyone the same thing. The inbox tells the one person allowed to know. The clock tells no one anything.

Manual account linking: when you've already proven both ends

Earlier, Mara's Google email matched her password email exactly, and the question was whether to auto-link. Now a different scenario.

Mara signed up with a password under mara@work.test. She wants to also sign in with Google — but her personal Google address is mara@gmail.com. A different email.

She has two ways to prove she's Mara. The system sees two strangers. Linking is making them one user: two accounts rows — one password, one google — pointing at one users row.

Why the auto-link can't do this

The auto-link rule was: when a Google sign-in arrives, link it to an existing local account only if the Google email is verified and matches. Run Mara through it:

Google says: mara@gmail.com
Local has:   mara@work.test

The emails don't match. The auto-link looks for a local user with mara@gmail.com, finds none, and does the only safe thing it can for an anonymous caller: it creates a brand-new, separate account for mara@gmail.com. Now Mara has two accounts and no way to merge them.

That's not a bug. It's the auto-link being careful. Watch why.

The asymmetry — this is the whole idea

The auto-link runs for an anonymous caller. Someone just showed up with a Google login. The server cannot tell Mara from an attacker who registered mara@work.test first. So it trusts almost nothing: it links only when both sides carry a verified, matching email — proof that can't be faked by typing an address.

Manual linking runs for a caller who has already proven both ends:

a valid session   → they own mara@work.test    (they're logged in)
a finished OAuth   → they control mara@gmail.com (Google just confirmed it)

Both halves are proven before the link function is even called. So it requires no email match at all. That's not a relaxed-security shortcut — it's the entire reason manual linking exists. The case the auto-link refuses (different email, or unverified) is exactly the case only a signed-in user should be allowed to resolve, by hand.

Anonymous caller → trust almost nothing → match verified emails.
Authenticated caller who finished OAuth → both ends already proven → no match needed.

The flow, and the one new cookie

Mara is signed in. She clicks "Connect Google."

GET /auth/google/link        (server: she has a session? if not → 401)
↓
mint state + PKCE, set handshake cookies   ← same as sign-in
set the link marker cookie                 ← the one new thing
↓
redirect to Google's consent screen        (client → google.com)
↓
Google redirects back: GET /auth/google/callback?code=…&state=…
↓
verify state (CSRF), exchange code for claims   ← same as sign-in
↓
marker present? ── yes ──► re-check session ─► linkGoogleAccount(userId, claims) ─► redirect ?linked=google
              └─ no  ──► sign in (the normal path)

Google only lets us register one callback URL, so sign-in and link come back to the same /auth/google/callback. The callback has to know which one it is. That's what the marker cookie is for: the link route sets it; the callback reads it to pick the branch.

Notice the re-check session step. The marker only says "this was a link attempt." It does not say who. The session cookie says who — and it might have expired while Mara was on Google's consent screen. So the callback reads the session again, right then, and links to whoever is actually still signed in. The marker is a mode flag, never a trust token.

The trap: a stale marker

Here's a bug that hides in the one path no test can reach (the callback needs a live Google to exercise).

state and verifier are written by both start routes, so they're always fresh. If the marker were written by only the link route, it could go stale:

Mara clicks "Connect Google"   → marker set (lives 10 min)
Mara abandons it at Google
Mara clicks "Sign in with Google" → sets new state + verifier…
                                     …but the old marker is still in the jar
↓
callback sees the marker → treats a plain sign-in as a link → mislink or a spurious 401

The fix is to make both start routes write a definitive mode. The link route sets the marker; the plain sign-in route clears it. Now there's exactly one source of truth per flow, and the callback can never read a marker left over from an abandoned flow.

/auth/google (sign-in)         → clearOAuthLinkCookie(reply)
/auth/google/link (linking)    → setOAuthLinkCookie(reply)

Because no test covers that branch, it has to be correct by construction. Two routes, two definitive writes, no leftover state.

What linkGoogleAccount refuses

No email match — but not no rules. A Google identity belongs to exactly one user, and that's enforced:

already linked to THIS user      → idempotent no-op (a double-click is success, not an error)
already linked to ANOTHER user   → 409: you haven't proven you own that one
this user already has a Google   → 409: one Google identity per account
otherwise                        → insert; UNIQUE(provider, provider_uid) is the atomic backstop

The takeover-critical guarantee is that last index: even if two requests race past the in-code checks, the database lets exactly one (google, sub) row exist.

And the classic link-CSRF attack — an attacker gets an auth code for their own Google, then tricks a signed-in victim into hitting the callback with it, linking the attacker's Google into the victim's account — is already dead, from the same state check that stops sign-in CSRF. The attacker's code carries the attacker's state; the victim's browser never held it. The link never happens.

When this is the wrong shape

Only one direction is built: this links Google to a signed-in account. The reverse — setting a password on a Google-only account — is a separate small flow, not built yet. Errors are JSON mid-redirect: a link conflict returns a 409 body in the browser, not a friendly page — frontend work, same as the sign-in errors. And unlinking — removing a provider, and refusing to remove the last one, which would lock the user out — is its own feature with its own guard. Not here.

The auto-link trusts the math, because the caller is a stranger. The manual link trusts the session, because the caller already signed in. Both end at the same place: two proofs, one person.

Rate limiting: a leak somewhere else is a password list for here

Attackers rarely guess one password ten thousand times.

They take ten thousand email/password pairs leaked from some other site and try each one once against your login. People reuse passwords, so a small fraction work. This is credential stuffing, and a login endpoint that answers as fast as you can ask is its ideal target.

The defense isn't a better password check. It's a budget: a client gets N tries per minute, then the door stops opening.

What we count, and where

One counter, per client IP (Internet Protocol address), per endpoint, per time window.

key:    rate-limit for POST /auth/login from 203.0.113.7
value:  a number, starting at 0
expiry: 1 minute

Every request does two things in Redis: INCR the counter, and set EXPIRE to the window on the first hit. When the counter passes the limit, the next request gets a 429 Too Many Requests before it ever reaches the handler. A minute later the key expires and the budget resets.

The limits, tuned to how often a real person does each thing:

signup            5  / minute
login            10  / minute
verify           10  / minute
forgot-password   5  / minute
google (+ cb)    10  / minute

Why the counter lives in Redis, not memory

The app will be deployed with two app instances behind a load balancer.

If each instance kept its counts in its own memory, an attacker capped at 10/min on instance A would get a fresh 10/min the moment the balancer sent them to instance B — 20/min across two, 30 across three. The limit would dissolve the more you scaled.

So the counter lives in one shared Redis. Both instances INCR the same key, so the attacker's 11th request is the 11th no matter which instance handles it. It's the same shared-Redis property that makes sessions work across instances: one source of truth, many readers.

The limiter only touches routes that opt in. /health and /ready are left out deliberately — the load balancer hits them every few seconds to decide whether to route traffic; throttling them would make the platform think the app is down and pull it from rotation. The limiter guards the doors attackers push on, not the ones the infrastructure knocks on.

The whole thing rests on one value: req.ip

The counter keys on req.ip. Every request that shares a req.ip shares a bucket. So per-IP limiting is only as correct as req.ip being the real client. Get that wrong and the limiter doesn't bend — it breaks, in one of two opposite directions.

Here's the break. In production the request never reaches you directly:

client ──TCP──> load balancer ──new TCP──> your instance

The balancer doesn't pass the client's connection through. It opens its own TCP (Transmission Control Protocol) connection to your instance. So the socket your process sees belongs to the balancer, not the client.

If req.ip were the raw socket address, then in prod every request — from every user on earth — would arrive wearing the same IP. Now signup at 5/min is no longer 5 per user — it's 5 total, shared by the whole internet. The sixth signup anywhere trips it. The limiter has become a self-inflicted outage.

So how does the real client IP survive the hop? The balancer writes it into a header:

X-Forwarded-For: <original client>, <next hop>, ...

And trustProxy is the switch that decides which value Fastify reads into req.ip:

• trustProxy: false → req.ip is the raw socket address. Whoever physically connected.

• trustProxy: <n> → req.ip is taken from X-Forwarded-For instead — but only across the proxies you declare trustworthy.

It adds no security by itself. It only tells Fastify which value to believe. Locally, nothing sits in front of the app, so the socket address already is the real client — trust no proxy. That's why TRUST_PROXY is empty in local config. Empty isn't a missing value; it's the right value for an environment with no proxy.

Why you count proxies instead of naming them

On a managed host (Railway, Render, Fly) there's no single balancer IP to point at. The balancer is a fleet — a pool of edge proxies that scale up and down and rotate addresses you're never handed. "Set trustProxy to the balancer's address" is a trap: pin one IP and it works until the platform adds a node next Tuesday.

The way out is to stop naming addresses and describe trust by position. Two pieces of "who sent this" arrive together, and they disagree:

The socket address — the IP at the far end of the real TCP connection. Nobody can forge it (you can't finish a handshake while pretending to hold an IP you don't), but behind a balancer it's the balancer's address: honest, identical on every request, useless for telling clients apart.

The X-Forwarded-For header — text that names the actual client. The right machine, but only as trustworthy as whoever wrote it.

trustProxy's real job is to turn the header (right machine, maybe lying) into req.ip safely, using the socket address (honest, but the balancer) as its anchor. The rule every proxy obeys makes that possible:

Forwarding a request, a proxy appends the address it received the request from — the socket address it saw — onto the end of X-Forwarded-For.