Rolling Back a Version

A rollback is a newer store opened by an older binary. It is the reverse of the direction every release is gated on (an older store opened by the newer binary), and it has a different law, because the older binary was finished before the newer store's shape existed.

The three tiers of a store, and what each promises backwards

The third row is the whole subject. A projection is a derived artifact and the engine treats it as one: it can always be rebuilt from the canonical. But a rebuild happens when it is asked for, and an older binary cannot know to ask for it.

The 11.2.0 → 11.1.0 case, exactly

11.2.0 writes kind-tagged posting keys in the metadata index. Every entity written, updated or converged under 11.2.0 has its postings under the tagged key. The table carries a header flag saying so; 11.1.0's format documents that byte as reserved and ignores it.

So 11.1.0 opens an 11.2.0 store without a complaint, and then:

• getNoun(id) and getVerb(id): exact, from the canonical.

• Field queries (where, in, ranges) and text queries: silently miss every row touched under 11.2.0. The lookup uses the untagged key; the posting sits under the tagged one. Rows never touched under 11.2.0 still answer, which makes the miss look like data loss rather than a shape difference.

That is a wrong answer, not a crash, and a wrong answer is the worse of the two.

The procedure

There is one exact rollback and one repairing rollback. Prefer the first.

Exact: restore a snapshot. Before the swap to the newer version, take a sparse-preserving copy of each store:

rsync -aS --delete "$STORE/" "$STORE.pre-11.2.0/"
# or: cp -a --sparse=always "$STORE" "$STORE.pre-11.2.0"BASH

The arenas are sparse; a plain cp, scp or tar without its sparse flag inflates a store to its apparent size and can fill the disk (see brainy/disk-footprint once it lands with 11.2.1). A rollback is then: stop the newer binary with a clean close, restore the copy over the store, start the older binary. Zero wrong reads, no rebuild, and it costs disk for as long as you keep the copy. Rows written after the snapshot and before the rollback are in the newer store only; keep it until you have decided about them.

Repairing: roll back, then rebuild the projection. If a snapshot was not taken, start the older binary and run:

await brain.repairIndex({ rebuild: ['metadata'] })TS

The rebuild is the online generational one: the doors stay open under the serving law while a shadow index is built from the canonical in the older binary's shape, then swapped. Until it completes, field and text queries under the older binary are the wrong answers described above. Its cost is the metadata shadow build for the store (minutes on a store of tens of thousands of rows; measured per store in the rehearsal lane's r.rebuild rows).

What the engine will do about it

An older binary cannot be changed after it ships, so the law forward is:

• A format change that an older reader would misread sets a must-understand flag, so the older reader refuses with a named error instead of serving a miss. 11.2.0's typed-keys flag was placed in the reserved byte; from 11.2.1 the reader treats any unknown must-understand flag as a refusal, and every later format change is declared there.

• Every release note names the rollback class of the release: exact (no projection shape changed), rebuild (a projection shape changed; rollback needs the rebuild above) or snapshot (a canonical or arena shape changed, or an older binary refuses the store; rollback needs the restore above). 11.2.0 is rebuild.

The 13.1 → 13.0.x case

13.1 keeps each field's value index (the list of distinct values that exists, missing, ranges and value listings read) as a starting copy plus small change records written at each save, instead of one whole copy rewritten at every save. 13.0.x reads only the whole copy, so on a store 13.1 has saved it would load each field's values as of the last full copy and miss every value written since: the "opens, then reads wrong" case above.

So 13.1 declares the change where an older reader is made to stop. Every metadata index table it writes carries the first must-understand header flag, and 13.0.x refuses such a table by name when it opens the store, naming the flag and this page. Because 13.0.x cannot open the store at all, it cannot run the repairing rebuild either. 13.1 is rollback class snapshot: take the snapshot before the upgrade, and restore it to roll back.

A rebuild that a 13.1 close parked keeps its value counts in that same format, under the rebuild's own generation, so the same rule covers it: the snapshot taken before the upgrade holds no park, and nothing about a park needs its own rollback step.

The 13.1.1 → 13.1.0 case

13.1.1 writes each field's starting copy in a new layout that is read in place instead of being parsed whole when the field is first used. 13.1.0 does not know that layout. Every metadata index table 13.1.1 writes carries a second must-understand header flag, so 13.1.0 refuses the store by name when it opens it, as above; and the starting copy itself carries a flag 13.1.0 refuses, so even a store with no new table yet is never read as if the copy were missing. 13.1.1 is rollback class snapshot: take the snapshot before the upgrade, and restore it to roll back.

The 13.1.2 → 13.1.1 case (the column revival record)

A sorted index (one per field) keeps two records of what changed: which items were deleted, and which deleted items were written again (an update) together with the point from which their new values count. Through 13.1.1 the second record was rewritten whole at every save. 13.1.2 writes only the items updated since the field's last save, inside the change file the first record already keeps, and rewrites the whole record only when those change files are folded together.

An older binary does not know that a change file can carry these entries. So a change file that does carries a new leading tag, and 13.1.1 (and every build back to 12.13.4) refuses it as damaged: that one field is refused by name, its queries fail with a typed error instead of answering, and the brain rebuilds the field from the stored items, as it does for any damaged sorted index. A field whose change files carry no such entries reads exactly as before. This change alone is rollback class rebuild; 13.1.2 as a release keeps the snapshot class of the 13.1.1 change above it.

The relationship tier key (after 13.2.0)

After 13.2.0 a relationship's visibility tier is indexed under system.visibility instead of the bare visibility key (see the field-addressing page). An older binary reads such a store without refusing anything, but relationships it writes or updates during a rollback are indexed the old way while the store's marker still says every relationship is keyed the new way, so the newer engine's open does not warn about them — after rolling forward again, run repairIndex({ rebuild: ['metadata'] }) once to re-post them. related() reads each relationship's own record and is right in both directions.