Pictures described in the engine

A picture is invisible to search. Its bytes say nothing a text query can match, and the only text a file write embeds for it is its name and size. So brainy describes pictures the way it embeds text: through a describe provider that sits beside the embed provider, reached through one door on a GPU box with one token. Every brain gets it with no code change.

When a picture lands in a brain, the engine sends it to the describe service once, in the background, and stores on the picture's record:

• a one-paragraph description,

• a short tag list — the things in it, the roles they play, the places it shows, and any words visible in it,

• what skin the picture shows, when the describe service says so,

• how the picture reads — its mood, colours, light, setting, style, how many people are in it, how good the photograph is, and a short alt text — and a second, closer look at how revealing it is, when the service says so,

• the description's embedding, as the record's own vector.

After that the picture turns up in an ordinary find({ query }), its tags can be filtered like any other field, and the change feed announces them. Nothing blocks the save: the write returns before the describe service has seen the picture.

What counts as a picture

A VFS file (brain.vfs.writeFile(path, bytes)) whose MIME type is image/* and whose bytes live in the blob store. The MIME type comes from the file name's extension. A working-state file (retention: 'latest-only') is never described: it never embeds at all, by design.

What the record carries

metadata.vision = {
  description: 'A photograph of a lighthouse standing alone under a grey sky.',
  tags: {
    things: ['lighthouse'],   // lower-case, at most 12 per list
    roles: ['landmark'],
    places: ['coast'],
    text: [],                 // words visible in the picture
  },
  exposed: [{ part: 'breast', score: 0.97 }],  // only when the service answered it
  exposedParts: ['breast'],                    // the same parts, as a plain list
  reading: {                                   // only when the service answered it; every member optional
    mood: { words: ['calm', 'warm'], confidence: 0.8 },
    palette: { colours: [{ hex: '#c8a165', name: 'sand' }], confidence: 0.7 },
    light: { value: 'golden-hour', confidence: 0.9 },
    setting: { text: 'a harbour wall at dusk', confidence: 0.6 },
    style: { value: 'candid', confidence: 0.5 },
    people: { count: 2, confidence: 0.95 },
    quality: { sharpness: 0.8, noise: 0.1, word: 'sharp', confidence: 0.85 },
    alt: { text: 'Two people on a harbour wall at golden hour.', confidence: 0.7 },
  },
  readingTerms: ['calm', 'warm', 'sand', 'golden-hour', 'a harbour wall at dusk', 'candid'],  // the engine's own plain list
  secondLook: { verdict: 'artistic-nude', confidence: 0.64, crops: 9, at: '2026-10-14T16:20:00.000Z' },
  model: 'the-model-name',
  version: '1.0',
  describedAt: '2026-10-14T16:20:00.000Z',
  sourceHash: '<the blob hash of the bytes this describes>',
  confidence: 0.92,
  served: ['description', 'tags', 'text', 'exposed', 'reading', 'secondLook'],
}TS

exposed and exposedParts say what skin the picture shows, so a product can keep nude pictures off public pages, prints and social posts. An empty list means the service looked and found none; when the two fields are absent, the service did not say, and the picture needs a look. served lists what the engine asked the service for when it made this description. See Exposed skin and The richer reading and the second look.

It is written through the engine's ordinary update path, by the engine's own housekeeping actor:

• The change feed carries it as an update event whose metadata holds the tags — a subscriber learns a picture's tags without reading the record.

• The metadata index serves every part of it as a field: find({ where: { 'vision.tags.things': 'lighthouse' } }) returns the pictures tagged so (a tag list indexes one entry per tag).

• The vector. The record's embedded text becomes the description followed by every tag, joined, and is embedded through the same background embedding path every file write uses. find({ query: 'lighthouse on a cliff' }) finds the picture.

sourceHash is the blob hash of the bytes the description was made from. When a file is written again with new bytes, its record still carries the old vision until the job describes the new bytes; a reader can tell by comparing vision.sourceHash with metadata.storage.hash. The new description replaces the old one whole.

A file written again with the same bytes is not sent again. The write puts the record's embedded text back to the file's name, so the job puts the description back as the embedded text, without a call to the service.

The door

The engine speaks to the describe service through one fixed contract.

POST /v1/describe — Authorization: Bearer <token>

{
  "requestId": "…",
  "items": [ { "id": "…", "mime": "image/png", "bytes": "<base64>", "locale": "en" } ],
  "want": ["description", "tags", "text", "exposed", "reading", "secondLook"]
}JSON

want always holds description, tags and text; exposed, reading and secondLook are each added only when the service's health lists it in serves, so a service is never asked for something it has not declared. At most 8 items per call and 20 MB in total. An item may carry url instead of bytes; the engine always sends bytes.

It answers 200:

{
  "model": { "name": "…", "version": "…" },
  "results": [
    { "id": "…", "description": "≤ 600 characters, plain",
      "tags": { "things": [], "roles": [], "places": [], "text": [] },
      "confidence": 0.0, "ms": 0,
      "exposed": [ { "part": "…", "score": 0.0 } ],
      "reading": { "mood": { "words": [], "confidence": 0.0 }, "…": "…" },
      "secondLook": { "verdict": "…", "confidence": 0.0, "crops": 9, "at": "…" } }
  ],
  "refused": [ { "id": "…", "code": "…", "message": "…" } ]
}JSON

Tags are lower-case, at most 12 per list; text is the words visible in the picture. exposed is optional — a service that does not answer it is still in contract; when it is there it is a list of at most 32 entries, each a lower-case part and a score from 0 to 1. reading and secondLook are optional the same way and are bounded in The richer reading and the second look. One bad item never fails the batch — it comes back in refused.

Refusals: 400 unsupported-mime | too-large | bad-image (per item, in refused); 413 batch too large; 429 busy (with Retry-After); 503 model unavailable; 401 bad token.

GET /v1/health answers { model, version, device, queue, msPerImageP50, serves }. serves is optional: the wants the service answers, e.g. ["description", "tags", "text", "exposed", "reading", "secondLook"]. A service that leaves it out is taken to answer the first three.

The job

The work runs on the engine's background job runtime as the job media-describe — a maintenance job, so it never holds a door, it stands down at close, and the operator's maintenance hold holds it.

1. On landing. After a picture's write returns, the engine drops a small queue entry in the store (_brainy_describe/queue/<id>.json) and asks for a run. The entry is durable: a process stopped before the run finds it again at its next open. A picture written twice before the run has one entry.

2. The drain. A run reads up to 32 queued pictures, plans calls of at most 8 pictures and 20 MB, and makes at most 4 calls before handing the runtime back and asking to run again — so a large backlog drains in short runs other jobs can interleave with. A queue entry is removed only after its record is written.

3. The pace. Before every call the job reads GET /v1/health. After a call carrying n pictures, the next call is not sent until n × msPerImageP50 milliseconds after it — the job never sends faster than the service says it describes. A 429 is waited out for its Retry-After (at most 30 s) and the call tried again — at most 3 tries per call.

4. The receipt. Every call prints one line: how many pictures were described and refused, the wall it took, the service's median per picture, and the model.

The daily pass

The job is a cadence with a 24-hour interval: once a day while the brain is open it walks every picture in the brain and describes the ones that were never described, or whose bytes changed since (vision.sourceHash differs from the current blob hash). An open on a brain that holds pictures and whose last complete walk is more than a day old runs one, so a brain that was closed for a week catches up when it opens. Pictures that landed while no service was configured are found this way.

The walk is durable: _brainy_describe/sweep.json records the last picture a walk finished, and a process stopped mid-walk resumes after it.

Exposed skin

The describe service can also say what skin a picture shows. The engine asks for it ("exposed" in want) on every call to a service whose health lists exposed in serves, and writes what comes back onto the record as vision.exposed (each part with its score, in the service's order) and vision.exposedParts (the part names, each once). The plain list is what a filter reads:

const exposed = await brain.find({ where: { 'vision.exposedParts': 'breast' } })TS

• The parts the engine knows are breast, genitals, buttocks and anus (exported as DESCRIBE_EXPOSED_PARTS). The service may name others; a part the engine does not know is kept as the service said it, never dropped.

• Absent means the service did not say. A service that does not answer exposed writes neither field. A product should treat such a picture as one that needs a look, not as a clean one.

• One re-pass when the service learns it. A picture described before its service answered exposed is described again once, by the daily pass, after the service's health starts listing exposed in serves. The new record's served names exposed, so it is never sent again for that reason. A service that does not list it is sent nothing extra, and is never asked for exposed.

• A service that was asked and did not say is named, not retried. When a service that lists exposed answers a picture without it, the picture keeps its description, carries no exposed (so it reads as needing a look), and served still names exposed, so it is not sent again for it. Each pass that saw this prints one line with the count — the service must answer exposed on every result, an empty list when it finds none.

• Never in search. The words for exposed parts never enter the picture's embedded text: its vector is still made from the description and its tags only.

• Out of contract is a refusal. An exposed that is not a list, holds more than 32 entries, or has a part that is not lower-case or a score outside 0..1 makes that picture's answer answer-out-of-contract, retried by the next daily pass.

The richer reading and the second look

Two more things the describe service can say about a picture. The engine asks for each ("reading", "secondLook" in want) only on a call to a service whose health lists it in serves, and writes what comes back onto the record. They follow the same rules as exposed skin: absent means the service did not say, one re-pass when a service starts declaring them, a want the service was asked for and left off is named once per pass and never re-sent, and neither ever enters the picture's embedded text.

vision.reading is how the picture reads, in plain words. Every member is optional — a service may answer only some — and every member carries its own confidence, a number from 0 to 1:

A member the contract does not name is ignored, not stored.

vision.secondLook is the service's second, closer opinion on how revealing the picture is: { verdict, confidence, crops, at }. verdict is one of safe, suggestive, artistic-nude, exposed, explicit; confidence is 0 to 1; crops is the whole number of crops of the picture the service looked at; at is the ISO 8601 instant it looked.

vision.readingTerms is the engine's own plain list, made from the reading and present exactly when reading is: the mood words, the palette colour names, the light, the setting and the style, lower-cased, each once, at most 32. It is what a filter reads for a colour name:

const warm = await brain.find({ where: { 'vision.readingTerms': 'golden-hour' } })
const sandy = await brain.find({ where: { 'vision.readingTerms': 'sand' } })TS

The reading's own members are also fields the metadata index serves: vision.reading.light.value, vision.reading.style.value, vision.reading.people.count and vision.reading.mood.words (a list of words, one entry each). A list of objects' sub-field is not in the index, so vision.reading.palette.colours.name finds nothing; use vision.readingTerms for colour names.

• Out of contract is a refusal of the whole picture's answer. A present member outside its bound — six palette colours, an alt text of 121 characters, a confidence of 1.2, a light that is not on the list, a secondLook without a whole number of crops — makes that picture's answer answer-out-of-contract, naming the fault. Nothing of that answer is stored partially, and nothing is trimmed into shape. A picture already described well keeps its description, tags and vector, with the refusal beside them; the next daily pass tries again.

• Absent means the service did not say. A service that does not answer reading writes neither reading nor readingTerms; one that does not answer secondLook writes no secondLook. A service that answers only some of the reading's members writes only those.

• New bytes replace all of it. A file written again with new bytes is described again and its reading, readingTerms and secondLook are replaced with the new answer, or dropped if the service no longer answers them.

Refusals are rows, never silence

A picture the service refuses gets a refused row instead of a description:

metadata.vision = {
  sourceHash: '<the blob hash of the refused bytes>',
  refused: { code: 'bad-image', message: 'the bytes are not an image', at: '2026-10-14T…' },
}TS

• unsupported-mime, too-large, bad-image cannot change while the bytes do not, so the same bytes are not sent again. New bytes are tried again.

• A picture too large to fit in one call even alone is not sent at all; it gets too-large from the engine with that reason in its message.

• answer-out-of-contract — the service answered this picture outside its own contract (a description over 600 characters, a tag list over 12 or not lower-case, a confidence outside 0..1, an exposed list, a reading or a secondLook out of shape). The answer is recorded as a refusal, never trimmed into shape. Retried by the next daily pass.

• unanswered — the service answered neither a result nor a refusal for it. Retried by the next daily pass.

• detector-error — the service's skin detector failed on this one picture. Retried by the next daily pass.

A picture already described well and sent again only to learn its exposed skin, its reading or its second look keeps its description, tags and vector when the service refuses it: the refusal is recorded beside them in vision.refused, never in their place.

When the whole call is refused, nothing is written and the pictures stay queued. The job's run fails with DescribeProviderUnavailableError, whose reason says which:

Configure it

Zero-config is one name: run the describe service where brainy-describe resolves (a hosts line or a DNS record) on port 8201, and set BRAINY_DESCRIBE_TOKEN to its token. This mirrors the embedding service (brainy-embed, port 8200, BRAINY_EMBED_TOKEN).

Discovery runs once per process and is shared by every brain in it. A name that did not resolve is asked again after ten minutes, so a service that comes up later is found without a restart.

With no service configured, nothing is described and no record is touched — the engine never writes a description a model did not make. The first picture to land prints one line saying it was not described and how to configure the service; later pictures print nothing more.

Cost

• A picture landing: one small queue object written after the save has returned. A write that is not a picture pays one check of its metadata and nothing else.

• A brain with no pictures pays nothing: no job runtime, no queue, no cadence. When a describe service is configured, the open lists the brain's media files once, from the metadata index, to decide whether to start the job.

• Describing: one service call per 8 pictures (fewer for large ones), one health read before each, and one record update per picture — the same update any caller's update() costs, plus the background embedding of the description.

• The daily pass: one metadata-index read listing the brain's media files, then one record read per picture, in runs of at most 512. Only owed pictures reach the service.

• Wall time is the service's, not the engine's. MEASURED on the hosted describe service (2026-10-14): 1.5–4.6 s per picture sent alone, about 2.2 s per picture in a batch, and a declared msPerImageP50 of 3,538 ms. Because the job paces to that median, a batch of 8 owns the service for about 28 s. PROJECTED from that median: a first daily pass over 1,000 pictures takes about an hour. Each pass that moves the walk prints where it stands ("daily pass at N of M pictures"), and a stopped process resumes after the last picture it finished.

• The index: each description is a near-unique string, indexed like any other field. A store that never filters on it can declare vision.description unindexed and keep the stored value.

In code

import { DescribeProviderUnavailableError, resolveDescribeProvider } from '@soulcraft/brainy'

const { provider } = await resolveDescribeProvider()
if (provider.configured) {
  const health = await provider.health()
  console.log(`${health.model} ${health.version}, ${health.msPerImageP50} ms per picture`)
}

await brain.vfs.writeFile('/photos/harbour.jpg', bytes) // returns at once
// … later, once the job has run:
const hits = await brain.find({ where: { 'vision.tags.places': 'harbour' } })TS