Status: draft
A document record is the archival unit of a Diaryx workspace: everything the workspace knows about one gathered document. It is the target shape that every importer emits, the structure that annotation lives in, and the referent that identifiers point at.
The design follows the PHILOSOPHY: gather, transcribe, annotate, publish — the same structure the Joseph Smith Papers Project uses for each document (artifact, transcript layers, source note, editorial annotation). Everything is plain text — Markdown with frontmatter for prose, YAML for the card that stands in for a file that is not prose — and a document record is fully legible in any text editor.
Components #
A document record consists of:
- Artifact — the original file, byte-for-byte (a scan, photo, PDF, audio file, or born-digital original). Never modified.
- Attachment sidecar (the catalog card) — prov's
<payload>.<ext>.yaml, minted beside the artifact byattach. It carries the artifact's title, itscontent:pointer, typed links, and provenance metadata. This is the record: there is exactly one document per artifact, and it is the sidecar. - Transcription(s) — zero or more Markdown files containing the artifact's textual content. Each is a document in its own right: independently editable, publishable, audience-filterable, and addressable.
- Annotation — footnotes within a transcription, plus sibling note files for document-level commentary, linked from the catalog card.
Only the artifact and its sidecar are required. A born-digital Markdown import needs no transcription; a photo of a letter usually wants one.
One document per artifact #
An earlier draft of this spec gave the artifact a Markdown catalog card of its own. That card was a second document about the same file, since attach had already minted the sidecar — two nodes in the tree, two rows in a lens, and no answer to which of them the record is. A transcription link would have had two places to hang off.
So the card is the sidecar, and there is no Markdown card. The cost is real and is accepted: a whole-file YAML document has no prose body, so the free-form source note is a frontmatter string (source_note:) rather than paragraphs under a fence. Anything longer than a note belongs in a sibling commentary file (see Annotation), which is where document-level prose was always meant to live.
Storage layout #
An artifact and its sidecar are siblings in the directory of the index that contains them — the placement import_attachment writes and prov's reverse index requires (photo.jpg ⇄ photo.jpg.yaml is found by name alone, so the pair cannot be split). There is no attachments/ subdirectory: an attachment is a child, so the first one promotes a leaf into its own folder exactly as a first child page does, and the pair lands there beside it. The containing index is resolved by the same rules a story written that day would use, so an artifact is filed under the period index the library already files writing into, rather than into a folder of its own:
Daily/2026/
index.md (the containing index; contents: links the sidecar, then the transcription, by id)
letter-from-may-1943.eml (artifact, byte-for-byte)
letter-from-may-1943.eml.yaml (catalog card: content:, attachment: true, transcription:, provenance)
letter-from-may-1943.transcript.md (transcription: transcription_of back to the card, part_of the index)
There is no record folder and no date prefix. A record is a link neighbourhood, not a directory: the sidecar names its payload through content:, its transcriptions through transcription:, and its container through the index's spanning field. Tools must follow those links rather than assume any path — the index's directory is where writes land, not a location anything may rely on when reading, and libraries written before this rule still carry pairs in an attachments/ subdirectory.
Records are not confined to Documents/. That directory was this spec's earlier convention; placement now follows the library's declared date view, and a user may move a record anywhere.
Born-digital text imports (a Day One journal, a folder of Markdown) have no artifact, and so no card: they import as plain entries, filed into the same date structure by the same placement rules. A record is what an artifact gets; an entry is what text that was never anything else gets.
The attachment sidecar (catalog card) #
Extends prov's sidecar — which attach mints carrying title, content: and attachment: true — with transcription links and provenance metadata. The whole file is the metadata; there are no fences:
title: Letter from Ruth Harris, May 1943
content: letter-1943.jpg
attachment: true
created: 2026-08-16
transcription:
- '[Letter from Ruth Harris, May 1943 (transcript)](letter-1943.transcript.md)'
source: Grandma's shoebox, box 2
source_note: >-
Water-stained but legible. Ruth describes the week Dad was born.
This is the only surviving letter from this period.
date_of_document: 1943-05-12
people:
- Ruth Harris
places:
- Ogden, Utah
Two keys are prov's and are not this spec's to redefine: content: is the pointer to the payload, resolved relative to the sidecar's own folder, and attachment: true is what stops prov reading the payload as prose. Containment is not a key here at all — the index gains the spanning link, in id-form, so the relationship survives the sidecar being moved.
The two dates answer different questions and must not be conflated. created is when Diaryx/prov made the document — an act of the software, stamped like any other document's. date_of_document is when the original was made, which is a fact about the world that often only its owner knows. A scan made today of a letter from 1943 has created: 2026-08-16 and date_of_document: 1943-05-12.
This matters most to importers, which are the tempting place to blur it: an EXIF origin date, an email Date: header, and a Day One entry's timestamp are all statements about the original, so they belong in date_of_document. created is never inferred from the source — it is when the record entered this archive, and nothing else.
Frontmatter keys #
| Key | Type | Meaning |
|---|---|---|
content |
path | The artifact this card represents, relative to the card (prov's, not this spec's) |
attachment |
bool | true — marks the card a sidecar rather than prose (prov's) |
transcription |
link list | Transcription files for this artifact |
source_note |
string | Free-form note: condition, context, why it matters. The body a whole-file card cannot have |
source |
string | Provenance: where the artifact came from (see Provenance below) |
date_of_document |
date, or unknown |
When the original was made (distinct from created, which is when Diaryx made the document). See below |
people, places |
string lists | Entities appearing in or associated with the document |
All provenance keys are optional. Unknown keys remain permitted, as everywhere in Diaryx frontmatter.
date_of_document: unknown #
A date lens groups by a chain — [date_of_document, created, updated], declared per-library as a view's group: — and the chain is first key present wins, not first key parseable. A grain then cuts the winning value, and validates rather than slices: banana at year grain is not the group bana, it is no group at all.
Those two rules compose into the marker. Writing
date_of_document: unknown
stops the chain at the first link and yields a value no grain can cut, so the document files as undated instead of inheriting the day it was scanned. Verified against prov: a document carrying date_of_document: unknown alongside created: 2026-01-01 lands in (ungrouped), while the same document without the marker lands in 2026.
This is what a shoebox needs. A folder of 200 undated scans imported on one afternoon would otherwise all inherit that afternoon through created, burying a year of the reader's real archive under a day that means nothing about any of the documents. Undated is the honest answer, and the marker is how a record gives it.
Two consequences worth stating plainly:
- The marker is a convention, not a mechanism. Nothing privileges the literal string
unknown— any uncuttable value stops the chain identically, so a typo'dMay 1943silently files as undated too.unknownearns its place by saying deliberately unknown to the next human, not by doing anything special to the engine. (Nor does anything catch the typo: prov'scheckdoes not validatetype: datevalues, which is a real gap and not this spec's to close.) date_of_documentis already load-bearing for placement. Period indexes are resolved by the date they carry rather than by title — real ones are titled2025-03 index,December 2025,March 2026and07, and no title rule matches all four — so the index-creating path stampsdate_of_documenton each index it makes. The key is structural, not merely descriptive, and a tool writing it is participating in filing.
Transcription files #
A transcription is an ordinary Markdown file — fenced frontmatter and a prose body, unlike the card — with a back-link to that card:
---
title: Letter from Ruth Harris, May 1943 (transcript)
part_of: '[Daily 2026](../index.md)'
transcription_of: '[Letter from Ruth Harris, May 1943](letter-1943.jpg.yaml)'
transcription_status: needs_review
generated:
by: agent:gpt-5-vision
at: 2026-08-16T19:02:11.000000Z
how: transcribed
audience:
- family
---
Dear Mother,
We are all well here, though the rain has not let up since Tuesday...[^1]
[^1]: Spring 1943 was unusually wet in northern Utah.
Keys #
| Key | Type | Meaning |
|---|---|---|
transcription_of |
link | Back-link to the record — the attachment sidecar, or the spanning index of a document of several pages (see below). Pairs with transcription, following the link/link_of convention. It names the card, not the payload — the card is the record, and a payload prov cannot read as text has nothing to link back from |
transcription_status |
enum | draft → needs_review → verified |
generated |
mapping | How the text came to exist, in prov's provenance vocabulary: by is the actor, at the instant, how the act. A machine transcript carries by: agent:<model> and how: transcribed, so a reader with cat can tell the draft from a person's words. Written once, by whatever made the file, and never maintained afterward; a transcript a person typed carries none |
Where a transcription is filed #
A transcription is a document in its own right — editable, publishable, audience-filterable — so it lives in the containment tree, as a spanning child of the same index that contains its record, listed in that index's contents: after the card. That is what makes it a row in a lens, a page in a book, and a file prov check can see; a file reached only through transcription: would be an orphan to any reader that did not know the key. On disk it sits beside its card: letter-1943.jpg.yaml and letter-1943.transcript.md share a directory.
transcription/transcription_of are a relation pair Diaryx declares over prov's default vocabulary, not two carried strings — cardinality: many both ways, each the other's inverse. A workspace Diaryx writes carries the declaration in its relations: config, so prov maintains the inverse, rewrites both ends when either document moves, and reports a card whose transcript has gone. A library written by hand that omits the declaration loses only that maintenance; the keys mean the same thing.
Writing a draft therefore touches three documents: the transcript is created, the containing index gains a contents: entry, and the card gains a transcription: entry. Nothing else changes, and in particular the card's own transcription_status is not a thing — status is the transcript's, since each layer is reviewed separately.
Status lifecycle #
Machine-produced text (OCR, speech-to-text) is a draft pending human review, never silently treated as truth:
draft— machine-generated or in-progress, not yet human-reviewedneeds_review— complete but awaiting verification against the artifactverified— a human has checked it against the artifact
This gives users the archivist workflow: gather → transcribe → verify → annotate → publish. verified is a position in that workflow, not an attributed claim about particular bytes — see Provenance below for what it does not yet say, and what prov will eventually let it say.
Multiple transcription layers #
transcription is a list. A record may carry multiple layers — e.g. a verbatim transcript (original spelling, strikethroughs preserved) and a normalized reading copy — as separate files. No layer-type key is defined yet; distinguish layers by title until real usage motivates one.
A document of several pages #
A four-page letter photographed as four images is one document, and a two-hundred-page diary photographed page by page is one document, but a sidecar stands for exactly one payload — prov's content: is a path, not a list, and that is not this spec's to change. So a document whose artifact is several files in order is a spanning index above one card per page:
Daily/2026/letter-from-ruth/
letter-from-ruth.md (the record: an index whose contents: are the pages, in order)
letter-from-ruth.transcript.md (one transcription of the whole document)
001.jpg (page 1, byte-for-byte)
001.jpg.yaml (its card)
002.jpg
002.jpg.yaml
...
---
title: Letter from Ruth Harris, May 1943
part_of: '[Daily 2026](../index.md)'
contents:
- '[001](001.jpg.yaml)'
- '[002](002.jpg.yaml)'
- '[003](003.jpg.yaml)'
- '[004](004.jpg.yaml)'
- '[Letter from Ruth Harris, May 1943 (transcript)](letter-from-ruth.transcript.md)'
transcription:
- '[Letter from Ruth Harris, May 1943 (transcript)](letter-from-ruth.transcript.md)'
date_of_document: 1943-05-12
source: Grandma's shoebox, box 2
---
Water-stained but legible. Ruth describes the week Dad was born.
The rules that follow from the shape:
-
The index is the record. Provenance keys —
source,date_of_document,people,places— go on the index, and its Markdown body is the source note, so a document of several pages gets the prose a whole-file card cannot hold. A page's card carries only what is true of that page: a diary page may have its owndate_of_document; a letter's pages do not. -
Page order is
contents:order. Nothing is read off the file names.001.jpgis a courtesy to a reader in the Finder, not a rule, and a page inserted later goes where the list puts it. -
A page card is an ordinary card. It may carry its own
transcription:— a single page transcribed on its own, or a layer that exists only for that page — and everything that works on a card works on it. -
Two pages or more. An index is read as a document of several pages when every child it contains is a page card (or a transcription of it) and there are at least two. An entry with one photograph attached is the commonest shape in a journal, and it is a page with a picture in it, not a document of one page; the picture's own card is where a single page is transcribed.
-
The document's transcription hangs off the index, through the same
transcription/transcription_ofpair, and is filed beside it as the index's last spanning child. A transcription of a document of several pages is one file, with the page boundaries marked so the text can be checked against the pictures:::page[Page 1 of 4]{card="id:j7hmq39"} Dear Mother, We are all well here, though the rain has not let up since Tuesday... ::page[Page 2 of 4]{card="id:k2p8w4d"} ...and Dad says the new calf is doing fine.Each page's text opens with a
pageleaf directive on a line of its own. Its label,Page 2 of 4, is for the reader: an editor draws it as a labelled mark in the text, and a site renders it as the page's heading. Its attributes are for a program:cardpoints at the page's card the way a config pointer does —id:<id>, or a root-anchored path (/letter/002.jpg.yaml) in a vault that registers no ids — so it survives the card being renamed or moved, which a file name in a comment did not. A blank page keeps its marker with nothing under it, so page 3's text is never found under picture 2. A reader who edits the draft keeps the markers, moves them, or removes them as the text warrants — they are a convenience for review, not a structure anything depends on.A leaf directive rather than a fenced
:::pagecontainer, because a transcript a person is about to type starts with every page empty, and an empty container has nowhere for a caret to go. Transcripts written before this carried the page as an HTML comment (<!-- page 1 of 4: 001.jpg -->) followed by a thematic break between pages; those still read correctly as text, and nothing rewrites them. -
A PDF is the same document with its pages already bound. It is one artifact and one card, and a transcription of it is one file with the same markers, each naming the one card and the page within the file:
::page[Page 2 of 4]{card="id:j7hmq39" page=2}. A single photograph's transcript has no marker at all — there is only one page to tell apart.
This is the shape rather than a manifest node over a directory of pages because a page is worth a card: it can be dated, linked, and transcribed on its own, and every tool that reads a card already reads it. What the shape costs — a row per page in a lens — is the honest count of what was photographed. A shoebox of unrelated scans is not this case and should not be forced into it: those are separate records, as import writes them.
Annotation #
Two forms, both plain Markdown:
- In-text annotation: standard Markdown footnotes inside the transcription file.
- Document-level commentary: sibling Markdown files linked from the catalog card (via
linksorcontents), for essays, research notes, or historical introductions about the document.
Explicitly out of scope: standoff annotation. No character-offset or range anchors into transcripts — offsets break on every edit. If in-text anchoring beyond footnotes is ever needed, anchor to quoted text snippets (fuzzy-matched), not positions.
Provenance #
This spec's source key and transcription_status lifecycle predate prov's provenance proposal, which reaches for the same goals with more mechanism behind it. That proposal is an early draft — eight open questions, three of which change the shape of the rest, and a Phase 0 that is carry only — so nothing here changes today. What this section fixes is the layering, so that neither side grows a duplicate of the other.
transcription_status is not prov's verified, and should not become it #
prov proposes verified: [{by, at, of}] — an append-only list of dated, attributed confirmations, each bound to the fixity digest it was made against — with trust tiers derived from the entries rather than stored. Diaryx's transcription_status looks like a smaller version of that, but the two answer different questions and both are worth keeping:
transcription_statusis a position in the transcription workflow.needs_reviewhas no prov equivalent and will not get one: prov's proposal lists workflow as a non-goal ("no review queues, no assignment, no approval states").verifiedis an attributed, dated, digest-bound claim. It says who checked, when, and against which bytes.
The intended relationship is that the second backs the first, rather than replacing it: transcription_status: verified is legitimate only when a verified entry exists whose of matches the transcription's current digest.
The gap this leaves open today #
Diaryx's verified is a bare enum, so it survives an edit to the transcription it describes. Someone checks a transcript against the artifact, someone else fixes a line, and the frontmatter still claims the transcript is verified — prov's proposal names this exact failure and calls it worse than having no vocabulary at all, "because it launders staleness as assurance."
Diaryx stamps fixity over attachments by default, and a library can widen it to fixity: all in config/config.yaml, which is what puts a content_hash on a transcript's own body — so the ingredient is available, but a library that wants checkable transcription claims has to ask for it. When prov's Phase 2 lands, the demotion rule comes with it: a transcript whose bytes no longer match the digest its verification was made against falls back to needs_review, and prov reports a VerificationStale finding. Until then this is a known, unenforced gap, not a solved problem — and transcription_status: verified should be read as "a human said so at some point," not as a checkable claim.
Two rules from the proposal apply to Diaryx as written, and are worth stating here because importers are the tempting place to break them:
- Nothing auto-promotes a transcription to
verified. A clean OCR run, a confident speech-to-text score, and a passingprov checkare all attestations about state; verification is a claim about meaning, and only a person or a named process makes one. - A stored verification never suppresses a finding. You cannot verify your way out of a fixity mismatch.
source and prov's sources #
source stays a free-form human string — Grandma's shoebox, box 2 is not a path and never will be. It is the narrow case of prov's proposed sources list.
The part worth tracking is §6 of the proposal: extending prov's fields mechanism from term-valued to path-valued fields, so that a frontmatter string prov knows is a path gets rewrite-on-move, locator handling, and dangling-reference reporting for free. It is Phase 1, independently valuable, and fixes a live bug rather than adding a capability. When a document record's provenance needs to name another document in the workspace — a finding aid, an accession record, a parent collection — that belongs in a path-valued sources field, not in source, and not in a new Diaryx-specific key.
Identifiers #
Identity is prov's, not Diaryx's. This spec earlier described a hand-rolled ID index (Meta/Ids.md, reached through a Config indirection); prov settled the question first and differently, and that design was never built. What follows is the layer Diaryx actually runs on.
prov treats identity as a strictly-additive layer over a workspace that already works on paths alone: the graph, traversal, and mutation layers never dereference an ID, and turning identity off changes nothing about how a record is stored or linked. IDs are minted at registration rather than derived — an unregistered document has no opaque ID at all and is addressed by path. Two config axes govern it, both set in the library's config/config.yaml:
identity: none | lazy | eager— when an ID is minted.lazy(prov's default) registers on link-by-id or publish, keeping the registry to the minimal set something external depends on. Diaryx runseager, which also registers on create: stable identity from birth, at the cost of a registry entry per document.id_storage: registry | frontmatter | both— where a registered ID lives. Diaryx runsboth: each document carries its ownidfield and the registry document is a rebuildable cache. Frontmatter storage is what makes identity move- and copy-robust — the ID travels with the file, which is exactly what an artifact and its card copied out of the workspace need.
The registry is a first-class member of the workspace, discovered through prov's registry pointer relation off the root. It is not a Meta/ convention, and nothing in this spec should assume its path.
What remains Diaryx's is what publication means. Publication turns a registered ID into a permanent promise: at publish time the file receives a permalink frontmatter key holding the full URL with the ARK inside, in the canonical bare-ark form served by the site proxy (e.g. permalink: https://diaryx.org/ark/dxg6h4ncm/4k2ph9t, blades illustrative; the ark:99999 prefix form is accepted as an alias). The ARK's workspace blade is the library's prov workspace_id, and its file blade is the document's prov id — prov and diaryx_ark mint the same 6-random-plus-check shape over the same alphabet, so publication mints nothing per file and stores no second identifier. The ID is thereby recorded redundantly — in the document, in the registry, and in the server-side publish record — and must never be reused or reassigned.
The ARK envelope itself (NAAN, dx shoulder, check character, .<FILE>/?<QUERY>/#<CALLOUT> suffixes) is specified in the ROADMAP; the alphabet and blade lengths are diaryx_ark's, which follows prov's minter.
Import contract #
Every importer emits document records. Import is a first-class workspace crate (import), not a plugin — the Extism stack that once held it was removed in archive/tauri-web-1.6.1, and the crate archived alongside it targeted the since-deleted diaryx_core. Its parsers are pure (bytes -> records) and port forward; its writer does not, because it emits the Markdown card this spec no longer describes.
The contract:
ImportedEntry→ an entry file (title, date, body, extra frontmatter), for born-digital text with no artifactImportedAttachment→ an artifact written throughWorkspace::import_attachment, which places the payload and lets prov mint the sidecar; the importer then stamps provenance onto that sidecar- Importers that recover text from an artifact (OCR, speech-to-text) emit a transcription file with
transcription_status: draft, linked from the sidecar'stranscription:and carrying ageneratedmapping that names the process and the act (how: transcribedfor text read off the artifact,how: convertedfor a text part decoded from a born-digital original). The writer is the workspace's (DiaryxWorkspace::write_transcription), which files it as Where a transcription is filed says - Importers populate provenance keys when the source format provides them (email headers →
people, EXIF origin date →date_of_document, etc.), and writedate_of_document: unknownwhen the source offers no date for the original — never today's date, and never a date inferred fromcreated
Nothing an importer writes may claim more than it knows. Deterministic decodes of born-digital originals (e.g. the text part of an email) may enter as transcription_status: verified; OCR and speech-to-text must enter as draft, per the lifecycle above.
Future directions (deliberately deferred) #
- Entity pages:
people/placesshould eventually be links to entity pages rather than bare strings. Bare strings are forward-compatible with this (a string can be upgraded to a link), so start simple. - Transcription layer typing (verbatim vs. normalized) — needs real usage first.
- Finer page-level mapping than the page markers above — a transcript section addressable per page, for a viewer that shows the picture beside its text. The markers are the coarse form of it, and are all that is defined.