Rolling Back a Version
In this section
A rollback is a newer store opened by an older binary. It is the reverse of the direction every release is gated on (an older store opened by the newer binary), and it has a different law, because the older binary was finished before the newer store's shape existed.
The three tiers of a store, and what each promises backwards
Tier | Written by | Backwards promise |
|---|---|---|
Canonical (entities, verbs, the generation log) | every version, one shape | Exact. Every |
Arenas (id mappers, verb endpoints, rotating logs) | base + delta + head, swapped by generation | Opens. A newer binary only sizes the files; the layout is the same. An older binary preallocates at its own sizes on its next grow, which costs apparent bytes and nothing else. |
Projections (the metadata index, columns, the vector and graph indexes) | versioned by header flags | Opens, then may read wrong. A flag an older binary documents as reserved is ignored, not refused. The table loads; lookups that depend on the new shape miss. |
The third row is the whole subject. A projection is a derived artifact and the engine treats it as one: it can always be rebuilt from the canonical. But a rebuild happens when it is asked for, and an older binary cannot know to ask for it.
The 11.2.0 → 11.1.0 case, exactly
11.2.0 writes kind-tagged posting keys in the metadata index. Every entity written, updated or converged under 11.2.0 has its postings under the tagged key. The table carries a header flag saying so; 11.1.0's format documents that byte as reserved and ignores it.
So 11.1.0 opens an 11.2.0 store without a complaint, and then:
getNoun(id)andgetVerb(id): exact, from the canonical.Field queries (
where,in, ranges) and text queries: silently miss every row touched under 11.2.0. The lookup uses the untagged key; the posting sits under the tagged one. Rows never touched under 11.2.0 still answer, which makes the miss look like data loss rather than a shape difference.
That is a wrong answer, not a crash, and a wrong answer is the worse of the two.
The procedure
There is one exact rollback and one repairing rollback. Prefer the first.
Exact: restore a snapshot. Before the swap to the newer version, take a sparse-preserving copy of each store:
rsync -aS --delete "$STORE/" "$STORE.pre-11.2.0/"
# or: cp -a --sparse=always "$STORE" "$STORE.pre-11.2.0"BASHThe arenas are sparse; a plain cp, scp or tar without its sparse flag inflates a store to its apparent size and can fill the disk (see brainy/disk-footprint once it lands with 11.2.1). A rollback is then: stop the newer binary with a clean close, restore the copy over the store, start the older binary. Zero wrong reads, no rebuild, and it costs disk for as long as you keep the copy. Rows written after the snapshot and before the rollback are in the newer store only; keep it until you have decided about them.
Repairing: roll back, then rebuild the projection. If a snapshot was not taken, start the older binary and run:
await brain.repairIndex({ rebuild: ['metadata'] })TSThe rebuild is the online generational one: the doors stay open under the serving law while a shadow index is built from the canonical in the older binary's shape, then swapped. Until it completes, field and text queries under the older binary are the wrong answers described above. Its cost is the metadata shadow build for the store (minutes on a store of tens of thousands of rows; measured per store in the rehearsal lane's r.rebuild rows).
What the engine will do about it
An older binary cannot be changed after it ships, so the law forward is:
A format change that an older reader would misread sets a must-understand flag, so the older reader refuses with a named error instead of serving a miss. 11.2.0's typed-keys flag was placed in the reserved byte; from 11.2.1 the reader treats any unknown must-understand flag as a refusal, and every later format change is declared there.
Every release note names the rollback class of the release: exact (no projection shape changed), rebuild (a projection shape changed; rollback needs the rebuild above) or snapshot (a canonical or arena shape changed; rollback needs the restore above). 11.2.0 is rebuild.