The maintenance hold

A brain does more than answer calls. Behind its doors it runs jobs: it merges files that pile up, counts, checks, repacks old history, measures what it holds and keeps its indexes tidy. Each one is bounded and yields to the calls in front of it, and on one brain that is invisible. On a process that holds many brains open at once, every brain's jobs run in the same process, and a busy stretch can keep the process's main thread near full for a long time — folder listings and reads then wait their turn behind work that could have waited.

The maintenance hold is the operator's switch for that. While it is up, the maintenance jobs stand down. Nothing is lost and nothing is cancelled for good: what a job was owed stays owed, and the release gives it back.

What a hold does

• A maintenance job that is running stops at its next natural pause. Its progress marker stays where its last checkpoint left it, nothing is counted against it as a failure, and no retry is armed.

• A maintenance job that is asked for while the hold is up (an open, a cadence, a call) is deferred, never dropped. The hold remembers it, once per job, with the delay it was asked for.

• A brain that is opened during the hold starts held too. A process started with the environment switch starts held, so even its first open is held.

• The release replays what was deferred, exactly once per job. A job a hold stopped mid-run resumes from its marker when it keeps one, and starts over when it does not; a job that was only waiting is started, or armed again at the delay it was asked for.

What a hold never holds

Some jobs are not tidying. A serving job is work a door waits on: while it is owed, some call answers nothing, or one field refuses by name. Holding it would turn a hold into an outage, so a hold never touches one. The engine ranks them in one table, and anything not in it is maintenance.

Every job the engine runs, and what the hold does with it:

Durability, never held. A hold may let files pile up, but a write must still be accepted inside its normal time for as long as the hold lasts. Five jobs are classed as maintenance by the engine and are still never held, because without them a write waits or is refused:

• vector-l0-flush — held, the in-memory vector delta would grow to its hard cap and the next write would wait for a whole flush.

• vector-consolidation — held, vector segments pile up to their cap and every vector write is then refused.

• vector-mapper-maintenance, metadata-mapper-maintenance — held, the shared id table's debt crosses its high-water mark; the vector write path then waits for a compaction pass, and past the ceiling it refuses.

• policy-mirror-rebuild — held, a store with no policy mirror refuses every write while its metadata index is dark, because the rule check cannot be answered.

The engine's own class for each is unchanged (status() still says maintenance); the exemption lives in the hold, by name. The other write paths were read and need no exemption: the graph flushes its in-memory trees inline when they fill, the metadata index's memory gate is drained by the engine's own flusher rather than a job, and everything else held only costs reads or disk. An acknowledged write is durable in the canonical log, never in a derived index, so a held flush cannot lose one.

Calls that need a held job refuse at once. Three calls can only be answered by a maintenance job: the first query of an aggregate that has never been built (aggregate-backfill; a warm aggregate answers from its current state and is never refused), brain.language.summarize (language-summarize-cluster), and whenServing({ families: ['pending-embeds'] }) (pending-embed-fold). Under a hold each throws MaintenanceHeldError immediately, naming the job and the hold's reason, instead of waiting for a release that may be hours away. Nothing is queued by the refused call; release the hold and call again. The engine's own table decides the class, not this page: status() reports each job's class, and a test keeps this table in step with the engine's.

Three ways to reach it

The environment. Start the process with BRAINY_MAINTENANCE_HOLD=1. The variable is read once, when the library loads, so the first brain the process opens is held. 0 (or unset) leaves maintenance running; any other value is refused by name when the library loads, so a typo cannot leave maintenance running quietly.

In the process. The library exports three names:

import { holdMaintenance, releaseMaintenance, maintenanceHoldStatus } from '@soulcraft/brainy'

const hold = holdMaintenance('migrating the venue brains')   // a reason is required
maintenanceHoldStatus()   // { held, reason, since, holds, heldRunners, deferredJobs, deferredCount }
hold.release()            // drops this hold's reference; returns whether the process is still held
releaseMaintenance()      // drops the most recent hold, for a caller with no handleTS

Holds are counted. Two holds and one release is still held; the process is free only when every hold has been released. Releasing a hold twice, or releasing a free process, changes nothing. maintenanceDebt() carries the same state under held, so a debt that is growing because of a hold reads differently from one that is growing because a job is stuck.

On the host. brainy host answers three doors on the service key (a principal's key is refused by name), outside the contract's route table because brainy serve runs no background jobs of its own:

The host's doors and the process's are one switch: a hold raised in the process shows in the GET, and one raised over the wire shows in maintenanceHoldStatus().

What holding for a day costs

Nothing is skipped, so everything that was deferred is debt that grows while the hold is up and is paid when it is released. What that looks like, by kind, without a measured figure for any of it (none has been taken for a day-long hold):

• Merges (metadata-lsm-compaction, metadata-column-segment-compaction, graph-tree-compaction): a writer keeps adding files and nothing merges them, so reads have more files to look through the longer the hold lasts.

• Reclaims (metadata-lsm-reclaim, metadata-column-reclaim): files that earlier merges retired stay on disk until the release.

• The value index's change records (metadata-field-index-compaction): each save that touches a field adds one small record to that field's chain, and an open reads the chain behind the field's base, so opens read a little more the longer the hold lasts. A field has room for 64 records; when a hold has kept the job away that long, the save that would add the 65th writes the field's base itself instead (once per 64 saves to that field) and says so in its line.

• Small deferred seeds and measurements (the counts, tallies and size measurements): the numbers they feed are late or approximate until they run.

• Memory and language jobs: derived results stop updating.

The release does not run all of it in one breath: the deferred jobs come back through the same scheduler as ever, one claim at a time, yielding to the calls in front of them. On a process with many brains, release the hold when the load is low, and expect the catch-up to take a while.

A hold is a tool for getting a struggling process to breathe, not a setting to leave on. Read held and deferredJobs while it is up, and release it.