PROP-005: Optional package index — per-org metadata + standalone index server
01Milestone: retrofits into M2.10 (vibe search) and M1.10 (vibe outdated) from ROADMAP.md. Slices land independently; index is opt-in everywhere.
02Status: implemented; folded into the workspace 2026-05-22. Slices 1–8 (the vibe-index server + CLI) and slices 9–10 (publisher hook + consumer fast path) are shipped, plus M2.10 vibe search. vibe-index lives at crates/vibe-index/ as a workspace member (§6) and parses through vibe-core::Manifest. See §9 item 11 for the de-rot and fold that got it there.
03Related: PROP-001 (git backend), PROP-002 ([[registry]] / [[mirror]] / [[override]] / content-hashed identity), PROP-003 (features / subskills / describes / conditional deps), PROP-004 §5.x (gap analysis), spec://org.vibevm.core/vibevm/common/PROP-000 (especially §15 dep weight, §16 JTD, §17 production architecture, §18 complexity ≥ RPM, §20 token secrecy).
04Out-of-band research summary (2026-05-06, prior session). Comparative inventory of indexing strategies in production package managers — Maven Central (Lucene + directory layout), npm (CouchDB replicated DB), PyPI (PEP 503/691 simple API), RPM/DNF (repodata/primary.xml.gz + libsolv), Deb/APT (Packages.gz RFC822), Cargo (git index → sparse HTTP), Go modules (proxy + Merkle sumdb), Nix flakes (per-flake flake.lock, no global index), Homebrew (mono-repo formula), OCI registries (/v2/_catalog). Three candidate paths surfaced for vibevm: (a) Cargo-sparse-style per-package JSON files in an org-level index; (b) DNF-style single repodata directory with full SAT-ready dep graph; (c) Nix-flake-style indexless live-resolve (current state) with optional flake-registry-shape short-name mapping. PROP-005 picks (a) augmented by (b)'s integrity-manifest pattern (repomd.json).
1. Motivation
05vibevm today resolves packages live against the host's git API:
- 06
vibe install flow:waltranslates togit ls-remote <org-url>/flow-wal.gitto enumerate tags, thengit archive(orgit fetch && git checkout) to read the manifest at the candidate ref. One pkgref = at least one round-trip per registry walked, two if the package is not in the first registry. vibe outdated(M1.10) callsMultiRegistryResolver::resolve(<pkgref>@Latest)per locked package — one round-trip each.vibe search(M2.10, not yet shipped) cannot work at all without enumerating an org. GitHub'sGET /orgs/<org>/reposis rate-limited to 60 req/h unauth or 5000 req/h with a token; GitVerse exposes no org-scoped repo listing in its public API.
07This works at the M0 / M1 demonstration scale (3 packages in vibespecs). It does not work at v1 shipping scale (target: hundreds of packages per org, multiple orgs configured per project).
08Failure modes the live-resolve path produces in practice:
- 09Cold-cache install latency grows linearly with the dep graph. A project with 20 transitive deps spread across two registries spends 30–60 s in
git ls-remotealone before any actual content fetch. - Rate-limit visibility for
vibe outdated. Polling N packages at refresh time burns N requests; against an unauthenticated GitHub registry, this exhausts the quota at 60 packages. vibe searchis impossible. Even with an authenticated org-listing endpoint, parsing every repo'svibe.tomlat every search would be intractable.- Discovery story is silent. A consumer with a fresh checkout of an unknown vibevm org has no way to enumerate "what packages live here?" without scraping the host UI.
- Mirror-driven offline workflows degrade silently. PROP-002 §2.3 makes mirror dispatch invisible to the lockfile, but
ls-remoteagainst a mirror still leaks live host calls when the resolver wants to know "what versions exist?" — there's no offline catalog.
10What every other production package manager does — and what vibevm now needs — is an index: a small set of files, regenerable from authoritative package state, that lets consumers perform list, search, outdated, and resolve-version-shortlist operations against cached / mirror-able metadata instead of live git. RPM (primary.xml.gz), Cargo (sparse index), Deb (Packages.gz), npm (CouchDB document per package) all converge on the same shape: derived metadata files alongside or near the artefact storage, regenerated by a tool, served as plain HTTP.
2. Decisions
2.1 Index is OPTIONAL — zero impact when absent
11req r1
12Decision. The index layer is strictly additive. Every existing vibevm code path keeps working exactly as today when no index is present. No registry is required to have an index. No project is required to consume one; a consumer that finds none falls back to the live git ls-remote path that exists today.
13Discovery is a ladder, and the eternal handshake is its first rung. A consumer probes two candidate bases in order — <index-url>/v1/index, then <index-url> — and at each one asks hello.json before any repomd.json. A 200 whose body parses as a handshake and carries a world of the epoch this build reads settles the probe: that world's path refines the base every later file is fetched from. Only «no handshake here» (404, 5xx, connect failure) moves on to the next candidate and, when neither answers, to the repomd.json probe at the same two bases — the compatibility surface for indexes published before the handshake existed. Nothing at all → the live path, silently, exactly as before.
14Asked first, not asked beside — because successor is the in-band forwarding pointer for an index that MOVED, and it is readable exactly when the old address no longer serves a catalog. A handshake sought only next to a repomd.json that answered would be read in every case except the one it exists for. The price is up to two extra GETs, and it is paid only by indexes that have no handshake.
15A probe answers found, absent, or refused, and the third is what keeps this section honest. «Absent» is the only outcome that falls through quietly, because it is the one that means what the fall-through assumes: nothing is published here. An index that IS there and cannot serve this consumer — it refused us (401/403), its body does not parse as a handshake, its handshake format is one this build does not read, or it publishes no world of this build's epoch — answers refused, carrying the offered epochs, this build's epoch, a recipe, and whatever the document said in min_client / notice / successor. Collapsing that into «absent» would make a private, broken or newer-than-us index indistinguishable from a missing one, which is the silence PROP-044 §2 forbids: a break that announces itself is normal life, a riddle is what strands users.
16A successor is named to the operator, never followed by the client. Automatic following needs a cycle watchman and a trust rule for the address it lands on, and neither is decided; naming the address costs the human one command and invents no policy.
17Rationale.
- 18Backward compat for existing
vibespecs(GitHub) +vibespecs-gitverse(GitVerse) registries — they keep working unchanged. Index is something the org owner opts into. - Decouples the index design from Phase A of vibevm — operators with three packages do not need an index; operators with three hundred do.
- Removes the "central server is now load-bearing" failure mode: if the index disappears, the live path is still there.
- 19Consequence. All optimisations the index unlocks (cold-cache install speed,
vibe search, fastervibe outdated) are opportunistic. - The integrity story (
content_hashper PROP-002 §2.1) does not change — content_hash is verified at fetch time regardless of whether the resolve path went through the index.
2.2 Form factor: per-org index living in a separate git repository
20req r2
21Decision. Each vibevm registry org that opts in maintains a dedicated git repository named index (configurable; default name index) under the same org root:
- 22
https://github.com/vibespecs/index git@gitverse.ru:vibespecs/index.git
23Inside this repository sits a fixed file layout (§2.4) holding the org's catalog. The repository is:
- 24Cloneable like any other — same auth model as the package repos (HTTPS public read; SSH or token push for the maintainer).
- HTTP-fetchable at raw URLs without cloning —
https://raw.githubusercontent.com/vibespecs/index/main/repomd.json(GitHub) /https://gitverse.ru/api/v1/repos/vibespecs/index/raw/main/repomd.json(GitVerse). Consumers default to raw HTTP (one GET, no clone); falling back to git clone only when the host's raw-HTTP shape is unknown. - Mirror-able trivially — the
[[mirror]]machinery from PROP-002 §2.3 applies unchanged: a mirror athttps://mirror.internal/vibespecs/indexis a drop-in.
25Why a dedicated repo, not files-in-package-repos.
- 26Discovery. A single
<org>/indexrepo answers "what's in this org?" with one HTTP GET. Per-package metadata files would still require enumerating the org first — chicken-and-egg. - Atomicity of catalog state. A single index repo can be replaced as a whole, signed as a whole, mirrored as a whole. Per-package files leave catalog consistency to the consumer to reconstruct.
- Decoupling. Index regeneration does not touch package repos. Authors do not need to run any utility. The org owner runs
vibe-indexon a cadence; package repos stay pristine. - Mirror parity with PROP-002 §2.3. The same mirror chain that covers package repos covers the index. Operators who already understand
[[mirror]]get the index covered for free.
27Why not a hosted central HTTP service (npm-style registry.npmjs.org):
- 28Requires running infra. vibevm's deliberate posture per PROP-000 §17 is "every org self-hosts on hosting they already use" — git platforms are that hosting.
- Single-vendor. We rejected this shape in PROP-002 §1 (the "Nix's failure pattern"). Centralising the index is the same anti-pattern at one layer up.
- HTTP service is available (§2.5) — the
vibe-index servemode lets an operator run one — but it is not the default consumption path. Most consumers go through static raw-HTTP files in the index git repo.
29Configurable but defaulted. A [[registry]] block pins a custom index location — the key exists in RegistrySection (one type serving both the project vibe.toml and the machine-global ~/.vibe/registry.toml, so the columns share one vocabulary), and this exact block parses (pinned by test prop005_index_url_example_parses, which carries it verbatim):
30[[registry]]
name = "vibespecs"
url = "https://github.com/vibespecs"
naming = "fqdn"
index_url = "https://raw.githubusercontent.com/vibespecs/index/main" # explicit override
# or, to point at a hosted server:
# index_url = "https://index.vibespecs.dev"
# or, to disable index lookup entirely:
# index_url = "none"
31The bottom rung is host-aware. Canonical public https://github.com/<org>
maps to https://raw.githubusercontent.com/<org>/index/<registry-ref>;
other hosts retain <registry-url>/index. This makes both fresh and already-
seeded GitHub configurations whose index_url field is absent consume the static
repository rather than its HTML page. The full ladder remains env override →
manifest key → host-aware default; exact none on either explicit rung
disables lookup. Lookalike hosts, nested paths, userinfo, unsafe owners/refs,
queries and fragments are never rewritten.
32For three months this block was a parse refusal, and saying so was the point — until 2026-08-20, when the key landed (B-083, owner-ruled «построить ключ»). RegistrySection carried deny_unknown_fields without index_url, so a reader copying the example got an error; the fact above honestly said plan while the tree disagreed with the document. The strictness itself was never the defect and stays: the section still refuses unknown fields, now with index_url among the known ones — pinned red-then-green by the landing (the pre-landing parse error is quoted in the worker report; registry_section_still_refuses_unknown_fields_alongside_index_url keeps the refusal honest).
33The environment variable VIBEVM_INDEX_URL_<REGISTRY> is the ladder's top rung — the operator's per-run re-point, no longer the only source. Until 2026-08-20 it was the sole locator, deliberately weaker than the manifest field it stood in for (per-shell, per-run, travelling with neither project nor lockfile) — which is why it never closed the requirement above. Now it overrides the key: env beats index_url beats the default, and none at either explicit rung disables the index. The name normalization (ASCII alphanumerics upper-cased, the rest to _``) is unchanged.
34404 / connect-failure on the index → silent fallback to live ls-remote: no error message, because the operator never promised an index. This half is built, and it is the absent outcome of ##A-PROBE-HAS-THREE-OUTCOMES-NOT-TWO — the other two outcomes are never silent.
2.3 Source of truth: package repos remain authoritative; index is a hot cache
35req r1
36Decision. Package repositories are the source of truth for content (manifests, files, tags). The index is a derived hot cache, regeneratable from the authoritative package state.
- 37This matters because it disambiguates the failure mode: if the index disagrees with reality, reality wins.
- A consumer that resolves a package through the index still verifies
content_hashagainst the actually-fetched bytes per PROP-002 §2.1. A mismatch between an index-recordedcontent_hashand the actually-fetched one is a hardIntegrityError, surfaced with both values and a hint to refresh the index. No silent acceptance.
38Server-mode wrinkle. The vibe-index serve mode (§2.5) is described in the prompt as "the only writer; source of truth". This is true for the index data. The server holds the canonical RAM copy, persists it to disk on every mutation, and refuses writes from other processes (file lock + mtime checks at startup). But the server's writes are derived from package-repo state — the ground truth for "is this version published?" is still the actual git tag in the actual package repo. A divergence (index says v0.1.0 exists; the package repo's tag was force-deleted) surfaces at consumer install time as an integrity failure on cross-source content_hash verification, and the operator has the diagnostic clear: "your index lies; reindex".
2.4 File layout inside the index repo
39req r2
40Decision. The index repo's working tree (and equivalently its raw-HTTP-served file tree) carries:
41<index-root>/
├── hello.json # the eternal handshake — read FIRST, dispatches to a world
├── repomd.json # manifest — hashes & metadata for all other files
├── primary.jsonl # one line per (group, name, version)
├── primary.jsonl.gz # gzipped variant
├── by-name/
│ └── <name>.json # candidate set — every (group, name) sharing one bare name
├── by-cap/
│ └── <capability-slug>.jsonl # provides-index — pkgrefs that advertise this capability
├── by-purl/
│ └── <purl-slug>.jsonl # describes-index — pkgrefs that describe this PURL
└── README.md # human-readable "what is this directory" pointer
42hello.json — the eternal handshake (PROP-044 §3): {vibe, worlds[], min_client?, notice?, successor?}, the one document whose keys never change meaning, through which a client of any age learns which worlds this index currently serves, where each lives, and where the handshake itself moved. Its shape is schemas/hello/e1/hello.jtd.json and its type is generated like every other wire type (§2.12); the index writes it, and §2.1 is where a consumer reads it.
43The handshake is deliberately absent from repomd.json::files, and the asymmetry is the design. repomd.json is the manifest of one world; the handshake stands above worlds and dispatches to them, so a world's manifest can no more vouch for it than a chapter can vouch for the table of contents. The consequence to hold: the handshake is the one served file the manifest's sha256 map does not cover, and what verifies it instead is that it parses — an HTTP 200 whose body is not a handshake is a loud refusal naming the broken index, never a quiet fall-through to repomd.json (§2.1).
44What the writer owns is a stated whitelist, not whatever the directory happens to hold — four root files (hello.json, repomd.json, primary.jsonl, primary.jsonl.gz) and three trees (by-name/, by-cap/, by-purl/), named once in the code so the two readers that ask "is this catalog still the projection of its journal?" — vibe-index rebuild <data-dir> --check (with cargo xtask rebuild --check <data-dir> as a thin compatibility wrapper) and the golden-corpus test — cannot compare different sets. The rest of the directory is not the writer's and is not compared: README.md and .gitignore are written once by init, and the whole of state/ (§2.13) is the server's own bookkeeping. A blacklist would have to enumerate the world and would rot the day the directory grew a file nobody listed.
45repomd.json — the manifest, modelled after RPM's repomd.xml:
46{
"schema_version": 1,
"registry": "vibespecs",
"registry_url": "https://github.com/vibespecs",
"naming": "fqdn",
"generated_at": "2026-05-06T12:00:00Z",
"generator": "vibe-index 0.1.0",
"package_count": 42,
"version_count": 117,
"files": {
"primary.jsonl": {
"kind": "file",
"size": 184522,
"sha256": "<hex>"
},
"primary.jsonl.gz": {
"kind": "file",
"size": 38421,
"sha256": "<hex>"
},
"by-name": {
"kind": "directory",
"entries": 42
},
"by-cap": {
"kind": "directory",
"entries": 18
},
"by-purl": {
"kind": "directory",
"entries": 5
}
}
}
- 47Every entry of
filescarries akindtag —"file"or"directory"— and a reader dispatches on it rather than on which shape happens to fit. The tag was half-present until 2026-08-14: directories carried it, files did not, and the union was matched by shape, so an entry that lost a field was silently re-read as the other kind instead of being refused. A wrong answer that looks like a right one is the one failure re-fetching cannot cure (PROP-044 §2), which is why the asymmetry was broken deliberately rather than tolerated; the break note isformats/breaks/001.md. Afileentry missing its tag is now a parse refusal. repomd.jsonis the format's point of trust: it records asha256for every file beneath it, which is what makes «verify the catalog» a single well-posed question. Today the party that asks it is the operator —vibe-index verifyre-hashes a data directory against the manifest — and that is the whole shipped claim (formulation narrowed to what exists, owner ruling 2026-08-20, BACKLOG.md B-084). The consumer-side half — fetch it once with an ETag round-trip, then verify each fetched sub-file against the recorded hash — is deliberately deferred to the post-1.0 campaign (campaigns/packages-2026-09/deferrals.md#release-1-0), together with the reader-predicate question it shares with B-080.- No shipped consumer does that yet, and saying so is the point of writing it down. The index client asks
repomd.jsonas an existence probe and then fetches the candidate-set file directly: it reads nosha256and sends noETag— measured as zero occurrences of either in the client, against 43 elsewhere in the same crate, so the zero is the client's silence and not the instrument's. The comparison the fact above describes exists only as the operator verbvibe-index verify, run against a data directory, which is a different party at a different time. - What that leaves exposed is metadata in transit, not content.
content_hashis verified against the actually-fetched bytes at fetch time no matter how the version was chosen (§2.3), so a tampered index can misdirect a consumer toward the wrong version — it cannot make it install bytes nobody checked. A substitutedby-namefile, by contrast, is read as it arrives. Both halves belong in one sentence: an integrity story stated only in its strong half is the kind of claim a reader plans around and a defect hides behind. - This pattern (manifest-with-checksums) is what RPM, Deb, and OCI all share, and is what gives us a path to GPG signing without re-architecting.
48primary.jsonl — newline-delimited JSON, one record per (group, name, version). Lines are sorted by (group, name, version) — the PROP-008 §2.2 identity ordering. Each line is one §2.6 entry. JSONL is chosen over JSON-array because:
- 49Append-friendly (a publish hook can append and re-sort + rewrite, or for incremental update merge a sorted insert).
- Streamable by consumers (line-at-a-time parse; no need to buffer the whole file).
grep-able (operators can inspect the index manually).- Diffable in git (per-line diffs survive re-sorts cleanly).
50primary.jsonl.gz — gzip-compressed equivalent for HTTP-bandwidth-conscious consumers. ~5× compression typical for JSON. Byte-identical content; gzip is deterministic at level 6 with the standard zlib dictionary so the file is reproducible across machines (we pin level and disable mtime in the gzip header to keep the SHA-256 stable). Both primary.jsonl and primary.jsonl.gz ship; consumer chooses based on Accept-Encoding.
51by-name/<name>.json — the candidate-set file for one bare package
name (PROP-008 §2.8). A single HTTP GET fetches every (group, name)
package that shares the short name <name>, each with all its versions —
~1–10 KB. This is the path short-name resolution (PROP-008 §2.6) walks:
one GET per registry yields the whole candidate set, so a collision
(PROP-008 §2.7) is detected at once. The directory level keyed on kind
before PROP-008; kind left package identity, so <name> alone is the key.
52{
"name": "wal",
"indexed_at": "2026-05-06T12:00:00Z",
"packages": [
{
"group": "org.vibevm",
"name": "wal",
"indexed_at": "2026-05-06T12:00:00Z",
"latest_stable": "0.1.0",
"versions": [
{ /* one §2.6 entry */ },
{ /* … */ }
]
}
]
}
53tombstone — {reason, superseded_by?}, carried by the candidate-set file only when the bare name is buried, and omitted from the wire otherwise (which is why the live example above does not show it). A name that ever existed answers with the current thing, a forwarding pointer, or a tombstone carrying a reason — never with silence (PROP-044 §2). The consequence that is easy to miss: the file is written even when the name has no packages left, because an absent file is the silence the law forbids. A buried name therefore looks like this:
54{
"name": "wal-old",
"indexed_at": "2026-05-06T12:00:00Z",
"tombstone": { "reason": "renamed to `wal`", "superseded_by": "org.vibevm/wal" },
"packages": []
}
55by-cap/<capability-slug>.jsonl — for vibe install --capability ui:landing-page style queries (PROP-003 capability-driven resolution). Each line: {"kind":"feat","name":"welcome-page","version":"0.3.0","capability":"ui:landing-page@0.3.0"}. <capability-slug> is the capability string with : and / and @ replaced by -- (filesystem-safe; reversible). Optional in v0; populated when present.
56by-purl/<purl-slug>.jsonl — for "what vibevm packages document this upstream library?" queries (PROP-003 §2.5.6 describes). Same shape as by-cap; <purl-slug> is the PURL with / replaced by --.
57Sort invariants. Every file with multiple entries is sorted deterministically:
- 58
primary.jsonl— sort key(group, name, version)with versions in ascending semver order. by-name/<name>.json—packagessorted bygroup; each package'sversionsarray sorted ascending.by-cap/<slug>.jsonlandby-purl/<slug>.jsonl— sort key(group, name, version).
59Determinism matters because the index repo lives in git: a non-deterministic order would produce churn diffs on every regenerate, defeating the value of git as the transport.
2.5 Two modes: CLI tool and HTTP server
60req r1
61Decision. A single binary, vibe-index, ships in two modes selected by subcommand:
- 62CLI mode (default, every subcommand except
serve) — operates directly on a data directory of index files. Reads on-disk state, mutates, writes back atomically. Suited for: scriptedvibe-index reindexinvocations, manual operator commands, CI pipelines, post-publish hooks. - Server mode (
vibe-index serve) — boots an HTTP server. Holds the index in RAM; persists every mutation back to disk. Single-writer (the server) — no other process should mutate the data dir while the server runs (file lock at<data-dir>/state/server.lock; broken lock → server refuses to start with a clear error). Suited for: hosted index endpoints, real-time publish-time updates fromvibe registry publish.
63Why one binary, not two. Same code paths (the in-memory Index struct, the persistence layer, the scanner) are shared. Two binaries would force consumers to install both. clap-style subcommand dispatch handles the mode selection.
64Distribution. The utility lives at crates/vibe-index/ as a member of the top-level vibevm workspace — built, tested and gated by the same cargo … --workspace invocations as every other crate. §6 records why the original standalone-workspace decision was reversed and what the reversal bought.
2.6 Index entry shape (the canonical record)
65req r1
66Decision. Every (group, name, version) entry carries the following fields. This is the shape lines of primary.jsonl follow, and the elements each by-name/<name>.json candidate's versions[] carry. The authority for that shape is schemas/index/e1/entry.jtd.json; this section is its reading aid. The JTD file is the source of truth, the Rust is generated from it into crates/vibe-wire/src/generated/ and re-exported through crates/vibe-index/src/types/entry/, and cargo xtask check-codegen refuses any drift between the two. The record is defined once, in the shared version_entry vocabulary, because the candidate-set file and the journal carry it transitively and the schema language has no cross-file reference — entry.jtd.json is the root that names it. So the index entry stands in exactly the same arrangement as the wire reports under the root schemas/, not a different one. Where this section and the schema disagree, the schema wins and this section is the defect. What this section carries that the schema cannot is the provenance below — where each field comes from — and that is a copy of nothing.
67{
"schema_version": 1,
"kind": "flow",
"group": "org.vibevm",
"name": "wal",
"version": "0.1.0",
"content_hash": "sha256:8136ecdbc25d4555cbab6e9574f153b252a05c62b55b5e0255def645458c9544",
"source_url": "git@gitverse.ru:vibespecs/flow-wal.git",
"source_ref": "v0.1.0",
"resolved_commit": "1c3a1355abcdef0123456789abcdef0123456789",
"registry": "vibespecs",
"workspace_origin": null,
"license": "EULA",
"authors": ["Oleg Chirukhin"],
"description": "Write-Ahead Log discipline for human-AI development sessions",
"homepage": null,
"keywords": ["wal", "memory", "discipline", "session-management"],
"describes": null,
"compatibility": {
"min_vibe_version": "0.1.0",
"requires_kinds": []
},
"provides": {
"capabilities": []
},
"requires": {
"packages": [],
"capabilities": []
},
"requires_any": [],
"obsoletes": { "packages": [] },
"conflicts": { "packages": [] },
"features": {
"default": [],
"exclusive": {}
},
"subskills": [],
"i18n": {
"available": ["en"],
"default": "en"
},
"boot_snippet": {
"source": "boot/10-flow-wal.md",
"category": "flow"
},
"files_count": 5,
"must_understand": [],
"yanked": false,
"frozen": false,
"indexed_at": "2026-05-06T12:00:00Z",
"indexed_by": "vibe-index 0.1.0"
}
68The three slots — must_understand, yanked, frozen — appear above in their significant form, but all three are omitted from the wire when empty (an empty list; false). A live record for an un-yanked snapshot package therefore carries none of them, and the same holds for tombstone on the candidate-set file (§2.4). Reading this example as "these keys are always present" is the one mistake it invites.
69Field provenance.
- 70
kind/group/name/version/license/authors/description/homepage/keywords/describes/compatibility/provides/requires/requires_any/obsoletes/conflicts/features/i18n/boot_snippet.source/boot_snippet.category— read directly fromvibe.tomlat the tagged ref. (M1.18's loading model, PROP-009, retired the author-chosenboot_snippet.filename; a snippet is now itssourcepath plus an orderingcategory.) group— the mandatory reverse-FQDN qualifier from[package].group(PROP-008 §2.1). Withnameit forms the package identity;kindis metadata and identifies nothing (PROP-008 §2.2 / §2.3).workspace_origin— the[origin]provenance marker (PROP-007 §2.8, PROP-008 §2.8), present only on a copyvibe workspace publishgenerated from a workspace member; absent (null) for a standalone publish.subskills— collected by walking<package-root>/subskills/<path>/vibe-subskill.tomlat the tagged ref; each entry:{path, delivery, describes, description, channels}. Same fields the lockfile records.content_hash— computed by the same algorithmvibe-registry::compute_content_hashuses (sha256 over deterministically-ordered file bytes). Index uses the same hash as the lockfile, so cross-checks are byte-equal.source_url— the canonical org URL (§2.4 of PROP-002) composed with the package repo name. Mirror URLs do not appear here (same invariant as the lockfile).source_ref—v<version>by default (the tag).resolved_commit— the commit SHA the tag pointed to at index time. Pinning to commit lets us notice tag-rewrites later.registry— local alias from[[registry]].name.files_count— informational, useful for sanity-checking integrity diffs.indexed_at/indexed_by— provenance for the index entry itself (when, by which tool version).must_understand— the reader capabilities a consumer must have to act on this record (PROP-044 §4.5) — a different vocabulary from the package's ownprovides.capabilities. Written by the projector; never read fromvibe.toml. A reader that does not understand every string in the list skips this record and says so — what «says so» is, exactly, is §2.19; unknown fields outside the list are ignored as before. This is the exact inversion of additive-only: the writer declares what is mandatory, per record, addressably and revocably, instead of the schema promising ignorability forever.yanked— the version is withdrawn. Journal-borne, not authored: frozen content cannot withdraw itself, so the fact arrives from the registry's facts journal and is projected here (PROP-044 §2a).frozen— projected from the package manifest's[package].frozen, never a registry's opinion. Absence =false= snapshot: content may flow under the same version string, and a hash mismatch is news.trueis the author's one-way freeze: bytes immutable, and a hash mismatch is an alarm. The flag lives inside the hashed content, so a version self-describes even offline and every registry serving those bytes necessarily agrees — a registry only observes the freeze in its journal and projects it (PROP-044 §2a; terminology §2b —snapshotandfrozenare the two states of one boolean axis, with no third).
71Forward compatibility. schema_version: 1 is recorded at file scope (in repomd.json) and at entry scope. Entries carrying fields a reader does not know coexist with it: unknown fields are tolerated, and known-but-absent ones default. This sentence was false for as long as it stood. Fifteen catalog aggregates carried deny_unknown_fields, so a reader meeting one unknown key refused the whole file — the opposite of what the fact promised — and the contradiction survived because nobody could remove the attribute safely. That was not caution: while every mutation read the catalog to rewrite it, tolerance would have SILENTLY DELETED the tolerated fields on the next write, which is worse than refusing. Phase 3 removed the condition rather than the symptom — a mutation is now an append to the journal plus a reprojection, so nothing read is ever written back, there is nothing to lose, and the strictness could finally go (PROP-044 §4.4). What tolerance still does NOT mean is acting on a record one does not understand; that refusal moved to the per-record capability set (##NEVER-SILENT-SCHEMA).
2.7 Identity and trust
72req r1
73Decision. The digest of content_hash is the join key between the index and the lockfile, and it joins only at a stated recipe (PROP-002 §2.1). A consumer that fetches flow:wal@0.1.0 via the index records the digest a no-index fetch would have produced — for every tree except the ones recipe 1 exists to disambiguate, where the two recipes deliberately disagree and that disagreement is the point. Index entries are advisory; the bytes are authoritative.
74The index and the lockfile currently stand at different recipes: the index emits sha256-tree/1:, the lockfile keeps the bare sha256: until its own format moves. So their strings are deliberately unequal, and nothing compares them as strings — the index client does not read the field at all. Written down because the asymmetry looks like a defect to a cold reader and is instead the thing that keeps a live lockfile from being rewritten by a change it did not ask for.
75The repomd.json::files[*].sha256 covers integrity of the index files themselves. The per-entry content_hash covers integrity of the package content, and carries its recipe so a check compares like with like. The two are independent: a tampered index file fails its file-hash check; a tampered package repo (force-pushed tag) fails its content-hash check at fetch time.
76Out of scope for v0: GPG-signed repomd.json.asc, Merkle-log audit trail (Go sumdb-style). §9 tracks both.
2.8 Reindexation: full and incremental
77req r1
78Decision. Two regeneration modes. Both are available via the CLI (vibe-index reindex); the HTTP trigger was withdrawn by owner ruling 2026-08-20 (##TRIGGER-HTTP):
79Full reindex. Walk every package repo in the org; for each repo, list tags; for each v<semver> tag, read vibe.toml and subskills/**/vibe-subskill.toml at that ref; compute content_hash; assemble §2.6 entry. Replace the in-memory index wholesale, then atomic-write the on-disk files.
80Sources for the walk:
- 81
--from-clones <org-dir>— local directory of bare/regular clones. Authoritative for the operator who already maintains a vendor mirror; offline-capable. Default path for owners who runvibe-index reindexon a cron against their own server's clone tree. --from-github <org>— REST API walk againstapi.github.com. Requires a token (read-onlyrepo-scope). Used by hosted index instances that don't keep a clone tree.--from-gitverse <org>— equivalent against GitVerse's API once it exposes org-scoped repo enumeration; today returns "not implemented" (mirrors the publish-stub pattern fromvibe-publish/src/gitverse.rs).
82Incremental reindex. Detect what changed since the last run and update only the affected entries.
- 83For
--from-clones: compare each repo'sgit rev-parse HEADandgit tag -loutput to a checkpoint stored at<data-dir>/state/checkpoint.json. Repos with new tags or a new HEAD commit onmain(in case a manifest changed without a tag) are re-walked; others skip. - For
--from-github: use theIf-Modified-Since/ ETag headers on/orgs/<org>/reposand/repos/<org>/<name>/tagsto skip unchanged repos.
84Incremental is the default cadence target (one run per minute on an active org); full is the bootstrap path and the "trust nothing" recovery option.
2.8.1 The organisation image, and what keeps caching it honest
85Enumerating the organisation on every operation is a cost, and the cost is the small half of the problem. For local clones it is a directory walk and cheap. For a git host it is a paged API walk on every single operation. But the reason to think about it is not speed — it is that the picture is stale the moment it is taken.
86The premise «between operations nobody can change the organisation» is already false, and not because of sibling workers (owner accepted this correction, 2026-08-06). A developer publishing a package creates a repository and pushes a tag straight to the git host, never passing through the index service. The image goes stale with one worker exactly as with ten. The real axis is whether every change goes through the index — and today none has to.
87--cache-org is on by default (owner
ruling, 2026-08-06), with an explicit negative form to turn it off. The name
describes the mechanism rather than the assumption, which is deliberate: the
assumption the default must NOT make is the one the fact above rejects.
88The cheap freshness check is what makes that default honest, and it is therefore not an enhancement. Git hosts answer «has anything changed» with a conditional request that costs almost nothing and needs no walk; the cached image carries the validator its enumeration came with, and every run offers it back. Without this step, «on by default» would silently mean the very assumption the owner rejected. With it, the image is cached and never treated as truth without asking.
89A host that gives no validator makes the index enumerate, never trust. The absence of an answer is not an answer. This is the direction the whole design has to fail in, because the opposite default — no validator, assume fresh — is indistinguishable from a working cache right up until a package cannot be found.
90An image is keyed to the organisation and the API base it came from, and a cache taken for one is never used for another. Cheap to state, expensive to omit: the failure would be an index confidently serving another organisation's picture.
91Hit and miss are reported, in both renderings. An operator must be able to tell «this came from the cache» from «this was enumerated», because a cache that silently serves stale data is indistinguishable from one that works — which is the disease this whole section is written against.
92rescan-org is its own verb and it is
unconditional (owner ruling, 2026-08-06). It enumerates regardless of the
cache and regardless of any validator, and refreshes the image. It exists
because a missed change is invisible from the inside: no freshness mechanism
promises completeness, and a full walk does. Webhooks (§2.16)
reduce how often it is needed; they never remove the need.
93Decision (owner, 2026-08-20), closing a fork that had been open since the journal phase: a full walk says nothing about packages that were in the previous catalog and are not in this one. No comparison, no warning, no tombstone.
- 94What is being declined is a real, cheap capability, not an impossible one. The walk already holds the previous set and the new set in memory at the same moment and simply never compares them — computing «forty before, thirty-nine now, this one is gone» costs almost nothing. The owner's answer is that the report is not wanted, which settles the question the machinery could not: a disappearance has too many innocent causes — a repository made private, renamed, moved between organisations, or an enumeration that was simply narrower — for the index to have an opinion about it.
- Why this does not contradict the no-silence law. The law that a name which ever existed must not answer with silence governs withdrawal — an act an operator performed and a record they chose to leave. A package absent from a walk performed nothing and chose nothing; the walk is a photograph, not a claim about intent. Burial is how a name is closed on purpose (§2.11), and it stays the only way one is.
95The cache and its freshness check govern the host-API path. For a local-clone walk the enumeration is a directory read, and wrapping a validator around it would buy nothing and add a way to be wrong.
96With no image on disk the behaviour is exactly what it was — enumerate, build, write the image — with no warning and no error, and turning the flag off leaves no image and no report field. A default that changes the first run's behaviour is a default that has to be explained; this one does not.
97Triggers.
- 98CLI:
vibe-index reindex <data-dir> --from-clones <org-dir>— direct invocation. - HTTP: withdrawn. The route this fact used to specify —
POST /v1/admin/reindexwith a pollable job id — was retired by owner ruling 2026-08-20 (BACKLOG.md B-085) without ever being built: reindexation is an operator verb, not a network operation. The reasoning is the one §2.16 already applied to webhooks: a network trigger for a heavy rebuild is a DoS lever, and handing it to the admin token deserves a deliberate decision, not an inherited TODO. The operator's trigger is the CLI verb — reached by cron, a host-native scheduler, or the host's own hook mechanism invoking the CLI (§11). A route may return post-1.0 at the first real operator's need (campaigns/packages-2026-09/deferrals.md#release-1-0). - git hook (server-side, on the index repo's host): owner installs a
post-receivehook on the org's hosted git that posts toPOST /v1/admin/reindexwhenever a package repo gets a push to av*tag. Documented in §11; not shipped as part of the binary. - cron:
crontabline invokesvibe-index reindex --incrementalevery N minutes. Documented; not enforced.
2.9 Single-writer server mode
99req r1
100Decision. The HTTP server is the only writer when running. It locks the data directory via a PID file (<data-dir>/state/server.lock) at startup; refuses to start if the lock is held by another live process; refuses CLI mutations against the same data directory by detecting the lock from CLI side (CLI-mode add / remove / reindex errors with "server is running on this data dir; use the HTTP API").
101In-memory state model:
102Arc<RwLock<Index>>
│
├─ readers (search, list, get) — RwLock::read()
└─ writers (add, remove, reindex) — RwLock::write()
103On every successful write, the server:
- 104Updates the in-memory
Index. - Re-serialises the affected files (
primary.jsonl, the touchedby-name/<name>.json, optionallyby-cap/by-purl). - Writes each file atomically:
tmpnext to the destination,fsync,rename. - Updates
repomd.jsonlast (the manifest is replaced as a whole; readers that hold the oldrepomd.jsonsee a consistent old view; readers that pick up the newrepomd.jsonsee consistent new files). - Optionally (if
--auto-commit-pushflag is on):git add -A && git commit -m "auto: index update" && git push origin <branch>against the data directory if it is a git working tree. v0 ships without this — operator runs commit/push manually or via separate cron. v1 adds--auto-commit-push.
105The writer never calls now(). Every timestamp a write stamps — the manifest's generated_at and each candidate-set file's indexed_at — arrives as an argument: the CLI passes the moment its command began, the server passes the moment of the mutation event, and the index and entry modules contain no clock call at all (a panel step refuses one). One state therefore produces one byte sequence, which is what makes "rebuild and compare" a real verification, an empty diff a real no-op, and a wire-diff a quantitative measure of a break rather than a wall of timestamp churn (PROP-044 §4.3). Determinism here is an instrument, not tidiness: without it every later recoverability check measures the clock instead of the content.
106The writer stamps its own schema version only into an artifact it creates from scratch. A catalog it read keeps the version that catalog carried: the value is state, not a constant of whichever binary happens to be running. Otherwise a catalog written by a later version and opened by an older binary for any mutation would silently shed its own marker and start claiming ours — a file that still looks consistent while asserting something untrue about itself, which is the failure re-fetching cannot cure (PROP-044 §2, law 1). What the writer does when the version it read is one it cannot serve is a separate question and not answered here; this fact only forbids the silent overwrite.
107The clause the journal adds: a projection has no version to keep. Once a mutation is an append to the journal followed by a reprojection, the writer no longer reads a catalog at all — it builds one from the facts. There is therefore nothing to preserve, and stamping this build's constant asserts something true about the artifact just written, rather than overwriting a claim some other writer made. The rule above is unchanged and still binds every path that does read a catalog before rewriting it; what changed is how many such paths exist. The protection it was reaching for did not disappear with them — it moved up a floor, to the journal's own epoch and to each record's must_understand set, where a build meeting facts from a newer world refuses them by name instead of quietly reading a subset and re-labelling the result (PROP-044 §4.5). Read the two together and the invariant is one thing said twice: a file never claims an authorship it does not have.
108Concurrency. axum + tokio. Reads do not block reads. Writes block reads (RwLock) for the duration of the in-memory mutation; disk I/O happens after lock release for any path it can (e.g. primary.jsonl rewrites are queued and serialised by a single dedicated writer task). For the request rates we target (max ~10 writes/min during a publish burst, ~1000 reads/min during a CI install storm), a coarse RwLock is sufficient.
109Process model. Single process. No replication. An operator who needs HA runs the server behind a load balancer with N replicas and a shared filesystem — but that's outside v0. v0 expects one process per data directory.
2.10 HTTP API surface
110req r1
111Decision. REST API, JSON over HTTP. CORS open on read endpoints (so a future web UI can hit it from a browser). Routes:
112GET /healthz # liveness
GET /readyz # readiness (index loaded, no in-flight reindex)
# Static index files (raw — same shape as the on-disk files; mirror-friendly).
# The handshake leads the block because it leads the client (§2.1).
GET /v1/index/hello.json
GET /v1/index/repomd.json
GET /v1/index/primary.jsonl
GET /v1/index/primary.jsonl.gz
GET /v1/index/by-name/{name}.json
GET /v1/index/by-cap/{slug}.jsonl
GET /v1/index/by-purl/{slug}.jsonl
# Structured query (richer than the raw files)
GET /v1/packages # ?kind=&q=&limit=&offset=
GET /v1/packages/{group}/{name} # all versions of one package
GET /v1/packages/{group}/{name}/{version} # one specific version (entry)
GET /v1/capabilities/{cap} # who provides this capability
GET /v1/purls/{purl} # who describes this upstream
# Mutations (auth required)
POST /v1/packages # body: full §2.6 entry — insert/upsert
DELETE /v1/packages/{group}/{name}/{version} # remove one version
DELETE /v1/packages/{group}/{name} # remove all versions of a package
# Admin (auth required)
POST /v1/admin/reindex # body: { mode, source, args } — SPECIFIED, NOT BUILT
GET /v1/admin/status # uptime, last reindex, pkg count, server version
# Observability
GET /metrics # Prometheus text format
113The admin surface the server builds is one route — GET /v1/admin/status — and since 2026-08-20 that is the specified surface, not a gap. The reindex trigger spent months as «specified and unbuilt»: the router registers 16 paths and none of them was it, the handler module holds status alone, and the code's own note («reindex POST lands in slice 6») described a slice that closed without it. The fork BACKLOG.md B-085 recorded — build it, or retire it in favour of the CLI verb — was resolved by owner ruling 2026-08-20: retired. ##TRIGGER-HTTP carries the reasoning, §11's hook recipe now invokes the CLI verb, and the route returns, if ever, at the first real operator's need.
114Authentication. Bearer tokens via Authorization: Bearer <token>. Tokens are read from <data-dir>/state/admin.tokens (one token per line; comment lines start with #). Read endpoints accept missing/invalid tokens silently. Write endpoints require a valid token; mismatch → 401 with a generic message ("authentication required"; do not echo the supplied token nor say which valid prefix it matched). Tokens never appear in logs (logging redacts the Authorization header).
115Per-host lockdown. By default the server binds to 127.0.0.1:8412 — local-only. Operators expose externally by setting --bind 0.0.0.0:8412 and putting it behind a reverse proxy with TLS. v0 does not ship TLS termination; this is the reverse proxy's job. (Same posture as cargo's sparse index protocol: the upstream is HTTP — TLS is for the CDN / proxy in front.)
116Errors. Application/json error shape, taken from RFC 7807 Problem Details (lightweight subset) — four members always, and one extension member when the error is a quarantine refusal (§2.19):
117{ "type": "vibe-index/error/integrity-mismatch", "title": "content_hash mismatch", "status": 409, "detail": "…" }
118instance is not emitted, and the subset is the four members above plus unavailable. RFC 7807 makes every member optional, so omitting instance is conformance rather than a gap — but naming a member the body does not carry teaches a client to look for it. What the body does carry beyond the four is the refusal row, as an extension member, which is the mechanism that RFC provides for precisely this and the reason the status can stay 404 while the answer stops being «not found».
2.11 CLI surface
119req r2
120Decision. vibe-index [--log-level LEVEL] <subcommand> <data-dir> [args]. The data directory is a required positional on every verb — the one argument no invocation can omit, so it is not dressed as an option. One global flag stands above the verbs: --log-level off|error|warn|info|debug|trace.
121# Lifecycle
vibe-index init <data-dir> --registry NAME --registry-url URL [--naming fqdn|kind-name|name|kind/name] [--force]
vibe-index dump <data-dir> [--format jsonl|json]
vibe-index verify <data-dir> [--json] # recompute file hashes, check repomd
vibe-index rebuild <data-dir> --check # journal-only reprojection + byte comparison
# Reindex
vibe-index reindex <data-dir> --from-clones <org-dir> [--full | --incremental] [--json]
vibe-index reindex <data-dir> --from-github <org> [--token-file FILE] [--api-base URL] [--clone-cache DIR]
[--cache-org | --no-cache-org] [--full | --incremental] [--json]
vibe-index reindex <data-dir> --from-gitverse <org> # emits stub-not-implemented today
vibe-index rescan-org <data-dir> --from-github <org> [--token-file FILE] [--api-base URL] [--clone-cache DIR] [--json]
# Read
vibe-index get <data-dir> <group> <name> [--version V] [--json]
vibe-index list <data-dir> [--kind K] [--limit N] [--offset M] [--json]
vibe-index search <data-dir> <query> [--kind K] [--limit N] [--json]
vibe-index capabilities <data-dir> <capability> [--json]
vibe-index purls <data-dir> <purl> [--json]
vibe-index outdated <data-dir> [--lockfile PATH] [--json] # given a vibe.lock, print upgrade candidates
# Write (CLI-mode; refused if server is holding the lock)
vibe-index add <data-dir> --manifest <package.toml-path> --repo-url URL [--ref REF --commit SHA]
vibe-index remove <data-dir> <group> <name> [--version V]
vibe-index yank <data-dir> <group> <name> --version V --reason TEXT
vibe-index bury <data-dir> <name> --reason TEXT [--superseded-by GROUP/NAME] # no group: the name is closed for all of them
# Server
vibe-index serve <data-dir> [--bind ADDR] [--auth-tokens-file FILE] [--read-only] [--auto-commit-push]
[--rate-limit-per-token N] [--rate-limit-per-ip N]
vibe-index stop <data-dir> # graceful shutdown via lock-file PID
122Decision (owner, 2026-08-19). Taking a package out of circulation is three distinct operations over two different entities, and the surface must offer all three rather than making one stand in for the others:
| operation | entity | what remains | state today |
|---|---|---|---|
full deletion — remove |
a version, or every version of a name | nothing. The package is indistinguishable from one that never existed | built — the verb emits its journal fact and the projector applies it |
withdraw one version — the yank verb |
one version | the record, carrying yanked on the wire: a build that already pinned this version keeps working, a fresh resolution passes over it |
built. The verb emits the fact, the projector sets the flag from it, and the wire omits the flag when false |
retire a name — the bury verb |
a bare name | a tombstone: the reason, and a successor to redirect to (§2.4) | built. The verb emits the fact; the projector is the only arm that PRODUCES a carrier from one, dropping the name's packages across every group and leaving the tombstone in their place |
- 124Why three verbs and not one with flags. They differ in what a reader is owed afterwards, which is the only thing a caller actually chooses between: deletion owes silence, yanking owes "still here, do not pick it", retiring owes "gone, and here is where to look instead". A single verb with a mode would let the wrong one be selected by a typo — and two of the three outcomes are not reversible by re-running the command.
- All three operations are built, and this line is the record of how the distance closed rather than a measurement of what remains. It has said three different things, each true when written, and the sequence is worth keeping because it is what a status marker is FOR: first «retirement is genuinely unbuilt» and the nearest existing fact, a rename, refusing outright as an unbuilt carrier; then «both remaining operations are one verb away», once the
buriedfact gave retirement its carrier; now neither is away from anything —yankemits its fact and the projector setsyanked = true,buryemits its fact and the projector drops the name's packages across every group and leaves the tombstone. The anchor is kept and its text corrected, never replaced, so a reader arriving from any of the three states lands on the same coordinate and learns which one they were reading. - A latent mine, measured while recording this, and it decides the shape of the work. The tombstone carrier is populated only by reading a catalog off disk; the projection built from the journal never sets it. Since the journal phase there is no read-then-write path — a mutation builds its state from the facts and writes that out — so a tombstone placed on disk by anything other than a journal fact would be erased by the next mutation, silently and with no failure anywhere. Nothing could reach this while nothing produced a tombstone at all, which is exactly what made it worth writing down before a producer existed: the retirement verb is not «write the field», it is «add the fact», and an implementation that takes the shorter route passes its own tests and loses the tombstone on the first unrelated publish. The producer built to this rule is the
buriedfact's projector arm, and the rule is now guarded rather than remembered — a test that buries a name, publishes something unrelated after it, and asserts the stone still stands, proved red by neutering the producer before it was believed green. - Decision (owner, 2026-08-19): renaming gets no carrier of its own — the tombstone already is one, and the journal gets ONE retirement fact carrying
reasonplus an optional successor. Therenamedarm of the event vocabulary is retired in the same act. - Why not a second thing. This document's own worked example of a tombstone is a rename —
{reason: "renamed to …", superseded_by: "org.vibevm/wal"}(§2.4) — andsuperseded_byis precisely the "go here instead" pointer a rename needs. More binding than the example: the standing naming law says a rename is a NEW IDENTITY, versions never transfer. A first-class rename relation would assert continuity between the old coordinate and the new one, which is the thing that law forbids; "the old name is closed, and here is where to look" is the only model consistent with it. And one question — where did this package go? — must have one place to look, or the two places eventually disagree. - Why the journal collapses them too, against this project's usual habit of keeping distinctions the projection folds. The deciding fact is measured, not aesthetic: the existing
renamedarm carriesfromandtoand no reason, while a tombstone requires one. So keeping it forces a choice between synthesising prose into a required field and adding a reason to it — after which the two facts differ only in whether the successor is optional, which is one thing spelled twice. The usual argument for keeping them apart (a rename is a stronger claim than a retirement-with-pointer) does not survive the naming law: under it, «renamed A→B» asserts nothing beyond «A is closed; B is where to look». The distinction the vocabulary would preserve is one this project has already decided not to make. - What it costs, and why now. This edits the truth layer's vocabulary, which is heavier than a catalog change and is done as a declared break with a note, never as a side effect. It was nearly free at the moment it was made and will not stay so for the next such change: nothing emitted
renamed— only tests constructed it, and the projector refused it by design — no rename had ever been recorded anywhere in the tree, and that was measured beside a control rather than assumed (the same search that found no"kind":"renamed"did find"kind":"yanked"in the golden corpus), and there is no external consumer of the journal. The reverse direction is not lost, only relocated: a reader holding the NEW name learns its old one from the journal, which keeps the retirement fact and its successor forever. That is the right home — the catalog answers where do I go, the journal answers what happened. - A consequence carried out with the change, named here before it was made so it would not be left lying — and discharged. §2.18 listed
Renamedamong the arms the projector refuses because their carriers are unbuilt;renamedleft the vocabulary and retirement gained a projected carrier in the same commit, so the sentence stopped being true in both halves at once and was corrected there. What this record could not name, and the landing had to find: the same commit falsifies three more statements of present state in this section — thestate todaycolumn above,##YANK-IS-A-VERB-AWAY's «retirement is genuinely unbuilt», and##A-TOMBSTONE-THAT-IS-NOT-A-JOURNAL-FACT-ERASES-ITSELF's «nothing produces a tombstone at all». A contract that predicts one stale sentence and carries four is the argument for measuring the perimeter by file rather than by naming what one remembers (harvest/renamed-perimeter.md). - <status stage="spec" state="void">Retired 2026-08-19, hours after it was written, when the owner named the verb. It recorded that
yankhad a precedent to borrow and retirement did not, and left the name to the owner. Heir:##THE-RETIREMENT-VERB-IS-BURY. This line stays so its name is never reused and inbound links do not break.</status> - Decision, taken while building the fact and recorded here because a future reader will re-open it. A
Publishedfact for a name that carries a tombstone clears the tombstone; the name lives again and the projection carries no stone beside its packages. Why: §2.4 says the candidate-set file carries a tombstone «only when the bare name is buried», and its worked example shows an empty package list beside it — so a file holding packages AND a stone is a shape this contract never describes, and a reader would have to consult something else to tell «gone» from «here». The fold has no veto: it answers with the state as of the last fact, and refusing a publish would make the projection a policy engine rather than a projection. Considered and rejected: keeping the stone as history (it would make the two states of a candidate file overlap, and history is the journal's job — the burial is recorded there forever either way); suppressing the stone at write time when packages exist (that is a rendering trick over untruthful state, and the state is whatrebuild --checkcompares). Revisit when: an operator needs «this name is closed» to outlive a re-publication — that is a different claim from a tombstone and would need its own carrier, not a change to this one. - The verbs are
yankandbury(owner, 2026-08-19).yankborrows the ecosystem precedent it already has.buryis not invented for the occasion: this contract already describes the state in that word — the tombstone is «carried by the candidate-set file only when the bare name is buried», and the worked example is introduced as «a buried name therefore looks like this» (§2.4). The command and the state it produces therefore speak one word instead of two, which is the property the neighbours' candidates (deprecate / retract / relocate) each failed in a different way.
125What every withdrawal verb refuses, and why the two refusals must not share a message. Each of the three verbs answers its question against the projection of the journal, never against the catalog on disk, and refuses before appending anything — so a refused command leaves the journal exactly as it found it. Two refusals apply to all three, and they come from different laws:
- 126The target is already in the state the verb produces — the version already yanked, the name already buried. Refused because it is already law here rather than new:
##A-MUTATION-THAT-CHANGES-NOTHING-COMMITS-NOTHING. A second identical record would be a trace left by a change that did not happen. - Otherwise, the target does not stand in the projection at all — no such version, or a bare name carrying neither packages nor a tombstone. Refused because the journal carries no false facts: a record of withdrawing what was never there asserts a state that never held. This is the rule
removehas enforced since it was built.
127Why that condition is a conjunction and not a convenience, stated because the tempting relaxation defeats another verb. An operator who removes every version of a name leaves it carrying nothing — and full deletion's whole guarantee is that the result is indistinguishable from a package that never existed (PROP-010 §2.6). Letting bury then plant a tombstone on that empty name would put the deleted package's name back on the wire, in a file written precisely so it can be read — quietly undoing the operation the operator chose. So «nothing to bury» is the correct answer for an emptied name, and a future reader who finds the refusal unhelpful is looking at the mechanism that makes deletion mean what it says.
128The two conditions overlap unless the verb is written to keep them apart, and bury is where that bites. A buried name has no packages in any group, so «nothing stands here» is also true of it — a verb that tested the plainer condition first would answer «no such name» about a name whose tombstone it is holding. The requirement is that the two partition; how is the implementation's business. Two ways are equivalent and both are correct: test «already in that state» first, or spell the plainer condition as the conjunction it really is («no packages anywhere AND no tombstone»). What is NOT correct is the naive reading, in which the more specific case never reaches its own message.
- 129The messages must differ, and that is not politeness. «Nothing to yank» told about an already-yanked version sends the operator looking for a problem that does not exist — they will re-publish, re-check the identity, or file a bug against the index. The two conditions are distinguishable at zero cost, so a verb that collapses them is discarding information it already holds.
burytakes a bare name and no group qualifier, unlike every other writing verb here. Not an oversight and not a convenience: the tombstone rides onby-name/<name>.json(§2.4), the candidate-set file that spans every group, so there is no per-group tombstone to address. Burying closes the name for all of its groups at once, and a--groupflag would promise a narrowing the format cannot express.--superseded-byis recorded as given and not resolved. The field is a redirect pointer «never an automatic rewrite» (§2.4): nothing follows it mechanically, so validating that it names a package this index knows would enforce an obligation the format never made — and would forbid the ordinary case of redirecting to a package published elsewhere, or not yet published at all. A wrong successor is a wrong sentence in a tombstone, which is the same class of error as a wrongreason.
130--json is a property of nine verbs, not of the binary. Every verb that ANSWERS a question carries it — get, list, search, capabilities, purls, outdated, verify, reindex, rescan-org; the six that perform an action and report only success (init, add, remove, dump, serve, stop) do not. dump is the instructive exception: it is machine output already, so a --json switch on it would be a second spelling of --format json. Saying «all subcommands» would send a script author looking for a flag six verbs do not have.
131--log-level is global (it may be written before or after the subcommand) and it folds into the one lever VIBE_LOG, which the subscriber reads exactly once at start-up: passing the flag SETS that variable, so the process environment always explains the output an operator is looking at. The flag speaks a closed set of six values while the variable keeps the full directive language — one thing with a coarse dial and a fine one, never two spellings of the same power.
132The tracing subscriber is installed on every invocation, at warn by default, and that is a decision rather than a default nobody chose. It is the binary's job and not the library's; there is no RUST_LOG fallback and no second lever. The reason is the refusal path: a version this build cannot act on is reported at WARN when a catalog loads (§2.19), and a publication failure is reported at WARN when the server pushes (§2.17). Both are things an operator must be able to see on any subcommand — so a subscriber installed only under some flag would make observability an accidental property of which verb happened to be running, which is how a message that exists is never read. The ordering that makes the flag honest is part of the same decision: the fold writes VIBE_LOG at the very top of main, after the parse and before the subscriber, so --help and parse errors still answer before any log and the variable is never left describing output it no longer governs.
133Help-text smoke lives under crates/vibe-index/tests/help_smoke.rs, mirroring every_subcommand_renders_help in vibe-cli.
2.12 Data structures
134req r1
135Decision. The catalog's wire types are generated from the schemas of §2.6 and re-exported, never written by hand: the definitions live in vibe_wire::generated beside the JTD they come from, crates/vibe-index/src/types/ re-exports them so every vibe_index::types::* path keeps its meaning, and cargo xtask check-codegen is the gate that refuses a drift between schema and type. VersionEntry comes from the shared version_entry vocabulary; NameEntry / PackageEntry / Tombstone from schemas/index/e1/by_name.jtd.json; BindingSite from by_purl.
136Two shapes stay hand-written, and each says why. Repomd / RepomdFileEntry — its size is a u64 where the schema language reaches only u32 (an open owner fork, BACKLOG.md B-091 — filed as B-056 and renumbered 2026-08-19, because that coordinate carried a closed row too and a reader following it landed there), and its files union is tagged by this document's own law (§2.4). And Index below, which is not a wire type at all: it is the server's in-RAM state, no single document ever carries it, and it holds two members the catalog deliberately never serialises — the reader's quarantine record and the per-name tombstones the writer projects back onto the candidate-set files:
137pub struct Index {
pub schema_version: u32,
pub registry: String,
pub registry_url: String,
pub naming: NamingConvention,
pub generator: String,
pub generated_at: DateTime<Utc>,
pub by_pkgref: BTreeMap<PkgKey, PackageEntry>,
/// The reader's record of the versions it refused to act on —
/// in memory only, never written into any catalog file.
pub quarantined: Vec<Quarantined>,
/// Per-name tombstones; `write_to` projects them back onto the
/// `by-name/<name>.json` it builds.
pub tombstones: BTreeMap<String, Tombstone>,
}
// generated — schemas/index/e1/by_name.jtd.json
pub struct PackageEntry {
pub group: Group,
pub name: String,
pub indexed_at: Timestamp,
pub versions: Vec<VersionEntry>, // ascending by version
pub latest_stable: Option<Version>,
}
// generated — the shared `version_entry` vocabulary (§2.6)
pub struct VersionEntry { /* … */ }
138The generated types carry a fixed trait floor — Debug, Clone, PartialEq, Eq beside the serde pair — and Default is deliberately not in it. The dividing question is whether the trait says anything about the FORMAT: the four are properties of the Rust representation and the wire knows nothing of them, so emitting them unconditionally is the same class of decision as canonical ordering. Default is different in kind — «does this type have a meaningful empty value» is a judgement about the type, not a fact about its form. An empty ProvidesEntry means «provides nothing»; an empty VersionEntry means nothing at all, since twenty-odd of its fields are required. So Default lives in hand-written impls beside the generated tree, on the sub-structures where it is meaningful, and VersionEntry has none.
139The consequence, which nothing in the tree guards: because the record derives no Default, every literal that builds one names all its fields, and there is no ..Default::default() tail anywhere to shorten them. That is not an inconvenience to be optimised away — it is what makes adding a field a decision at every construction site instead of a silent zero. A future session that "simplifies" by deriving Default on the record would change how records are built from fixtures without any test going red, which is why the boundary is written here rather than left to be re-derived.
140What moving to generated types cost, measured rather than estimated, so the price is not re-paid in reverse. Three traits left with the hand-written shapes and the tree was checked for each: Ord / Hash / PartialOrd — wanted by nobody, and a comment claiming one of them justified duplicating a vocabulary had been false since it was written; Copy on the kind vocabulary — really lost, and the call sites that had it became explicit clones; Default — restored by hand where it means something. The classification is what matters, not the counts, which is why the counts live in the dated measurement (campaigns/packages-2026-09/harvest/f42c-reexport-radius.md) and not in this sentence. Anyone proposing to restore a duplicate type for the sake of Copy is proposing to unpay this, and should read what it bought first.
141A vocabulary fragment that changes in a new epoch is a DIFFERENT fragment, and it is separated by its NAME, not by a directory. The shared home holds fragments once, by name, and every schema module that pulls one re-exports it; when a second epoch needs a changed shape, the changed shape gets its own name and sits beside the old one in the same home. Worlds stay separated because two names never denote one type — the same rule that makes the shared home possible at all. Putting epochs into the home's directory structure instead would fork the home and give one fragment two addresses, which is the collision the single-home phase exists to prevent. Revisit when: the first second-epoch schema pulls a first-epoch fragment — that is the day this rule is exercised for real, and until then it has never been tested.
142PkgKey = (Group, String) — the (group, name) identity of
PROP-008 §2.2, and the order by_pkgref walks in. kind is metadata and
identifies nothing, so it is not part of the key.
143Search keeps no stored index. The postings are built per query against the loaded Index (index/search.rs) rather than held as a field, so no mutation has anything to invalidate. Token = lowercased ASCII alphanumeric run; ~30-stopword filter (the same list vibe-check::activation_conflict uses, deliberately reused for consistency). Ranking is term-overlap — one point per query token a hit carries — with the (group, name) identity breaking ties. Good enough for ≤10k packages; tantivy is a v1 upgrade if it isn't. Search answers only over what this build can act on: it asks the quarantine::usable_* accessors, never pkg.versions raw (§2.6).
2.13 Persistence layer
144req r1
145Decision. <data-dir>/ layout:
146<data-dir>/
├── hello.json # the eternal handshake (§2.4)
├── repomd.json # the manifest (§2.4)
├── primary.jsonl
├── primary.jsonl.gz
├── by-name/
│ └── <name>.json # no <kind>/ level — kind left package identity (PROP-008)
├── by-cap/
│ └── <slug>.jsonl
├── by-purl/
│ └── <slug>.jsonl
├── README.md # auto-generated; explains "this is a vibevm index"
├── .gitignore # written by `init`; covers state/
└── state/ # NOT mirrored (gitignored when data-dir is a git working tree)
├── journal/<YYYY>-<MM>.ndjson # the registry facts journal — the AUTHORITATIVE layer
├── server.lock # PID file, present only when serve is running
├── admin.tokens # bearer tokens (gitignored)
├── checkpoint.json # incremental-reindex bookkeeping (last commit/tag per repo)
└── org-cache.json # the organisation image and its validator (§2.8.1)
147The one thing to notice in that tree: state/ is not mirrored, and the journal lives there. Everything above state/ is a projection and may be deleted and rebuilt; state/journal/ is the layer it is rebuilt FROM (§2.3), and the server refuses to start without it. So the directory's gitignore boundary and its truth boundary run in opposite directions — the served half is disposable, the unserved half is not — and an operator who backs up «the index» by copying what the mirror carries has backed up the derivative and left the original.
148/metrics counts from memory, not from a file. The counters are atomics in the server's own state and reset with the process, which is what an operational counter means; no state/stats.json exists, and a reader looking for one would find a durable-looking name for a volatile fact.
149The data-dir doubles as a git working tree of the org's index repo. state/ is .gitignored (the init subcommand writes a default .gitignore). Operators commit + push the rest manually, or via --auto-commit-push — built 2026-08-06, contract in §2.17.
150Atomic write protocol. For each file F to be replaced:
- 151Write
F.tmpnext toF. fsync(F.tmp).rename(F.tmp, F).fsync(parent_dir(F))on POSIX. (No-op on Windows where the directory has no fsync semantics; rename itself is atomic.)
152Step 4 went unperformed until 2026-08-20; it is now atomic_write's closing act. The anchor keeps the finding's name: for one campaign every sync_all in the crate was on a FILE and no code path opened a directory to flush it — the gap was filed as BACKLOG.md B-087 rather than quietly dropped from the protocol, because a durability step deleted for being unimplemented is how a guarantee becomes folklore. What step 4 buys is the durability of the rename across a power loss on POSIX: without it the new bytes are safe and the directory entry pointing at them may not be. The B-087 landing closed it the conservative way — the code caught up to the protocol as ratified: fsync_parent_dir in index/persistence.rs runs after the rename (POSIX real, errors surfaced, never swallowed; Windows a no-op with the reason recorded in code, exactly as this step's own parenthesis allows), and every production projection writer reaches it through the one atomic_write. The journal shard stays outside by measured verdict, not omission: it is an append-in-place NDJSON write with no tmp→rename, so this protocol does not describe it; the lockfile and auth sync_all sites are test fixtures on read-only production paths.
153repomd.json is replaced last among the files it vouches for, so a reader that fetches repomd.json first then chases hashes always sees consistent files. The handshake is written after it and is the one root file that does not weaken the rule, because the manifest never claimed it (§2.4) — the precedent being README.md and .gitignore, which init writes and the map does not carry either.
2.14 Integration with the rest of vibevm
154req r1
155Consumer side (vibe-cli, vibe-registry).
- 156
crates/vibe-registry/src/multi_registry_resolver/carries an optional index-aware fast path. Before falling back to per-repogit ls-remote, it opens a session by the discovery ladder of §2.1 — the handshake first, the manifest as the compatibility tail — and on success readsby-name/<name>.jsonfor the pkgref, selects the candidate whosegroupmatches, and picks the matching version locally: zero ls-remote calls. Anabsentprobe falls through to today's path; arefusedone surfaces its reason instead of pretending nothing was there. - Index-derived
content_hashdoes NOT replace fetch-time verification. The actualgit fetchstill happens; the post-fetchcompute_content_hashstill runs; mismatch still errors out per PROP-002 §2.1.
157Publisher side (vibe-publish).
- 158
crates/vibe-publish/src/post_hook.rscarries an optional post-publish hook: when bothVIBEVM_INDEX_URL_<REGISTRY>andVIBEVM_INDEX_TOKEN_<REGISTRY>are set for the registry being published to, the publisher POSTs the new entry to<index-url>/v1/packageswith a bearer token after a successfulpush_release. Failure of the index POST does NOT fail the publish — it logs a warning and the operator's nextvibe-index reindexcovers the gap. - Both settings are per-REGISTRY environment variables, and the distinction matters. The suffix is the registry's local alias from
[[registry]].name, not the host — one host can serve several registries and one registry can move hosts, so keying on the host would name the wrong thing in both directions. The manifest fields this document once promised (index_url,index_token) do not exist (§2.2), so the environment is not one source among several here: it is the only one. - Direct-push (
--repo-url) bypasses index updates entirely (no registry context).
159vibe outdated (M1.10 follow-up).
- 160Adds a fast path: when a registry has an index, query
by-name/<name>.jsonfor the latest version instead ofgit ls-remote. Same envelope shape; ~100× faster for large lockfiles.
161vibe search (M2.10 — this is what unblocks it).
- 162Walks every configured registry's index through the same client the resolver uses — probe, then query — rather than downloading
primary.jsonl.gzand scanning it locally. The whole-file scan was the shape this document first imagined and is not what shipped: asking the index a question keeps the bandwidth proportional to the answer instead of to the catalog, and it puts one discovery ladder (§2.1) under every consumer instead of two. Index is the enabling layer for M2.10;vibe searchis the headline consumer of this PROP.
163Each integration point is a separate slice. v0 of vibe-index ships without any of them — the index can be populated and consumed via raw HTTP / git clone before vibevm consumers know about it. Integration slices land in M2.10 / M1.10 follow-ups.
2.15 What index must NEVER do
164req r1
- 165Never replace
vibe.tomlas the source of truth. A package with a missing index entry still installs from git per the live path. A package with a divergent index entry triggersIntegrityError, never silent acceptance. - Never modify package repos. The index utility reads package repos (for the
--from-cloneswalk) but never writes to them. - Never echo tokens. Same discipline as PROP-000 §20. Auth tokens for the server, GitHub API tokens for
--from-github, publish tokens propagated through hooks — none ever appear in stdout / stderr / logs / JSON envelopes. - Never assume mirror infrastructure. The index is opt-in everywhere; no consumer or publisher fails because the index disappeared.
- Never make breaking schema changes silently. The refusal is what matters and it survives; where it LIVES moved with the journal. A build must never read the subset it understands and carry on as though it understood the whole — but the carrier of that refusal is no longer a version number on the catalog, because the catalog is a projection any build rewrites from facts. It is the per-record capability set: a record naming a capability the reader lacks is refused BY NAME, with a recipe, and the rest of the catalog still loads (PROP-044 §4.5; the answer's shape and the surfaces that owe it are §2.19). That is strictly louder than a version compare, which could only say "somewhere in here is something newer than you". Unknown FIELDS are a different question and are answered by
##FORWARD-COMPAT: they are tolerated, because tolerating them can no longer lose them.
2.16 Webhooks — feeding the index instead of polling it
166The problem, and it is not performance. To learn what changed, the index enumerates the organisation. That is a cost, but the cost is the small half. The large half is that the picture goes stale without anyone doing anything wrong: a developer publishing a package creates a repository and pushes a tag straight to the git host, never passing through the index service. So the index's image is behind from the moment it is taken, and a stale index is worse than a slow one — the package exists and cannot be found.
167This is what makes the organisation cache honest. §2.8's cache and its cheap freshness check reduce how often the index asks; they do not change who knows first. A webhook does: the image becomes authoritative because it is fed, rather than because we assumed nobody else writes. Both mechanisms ship — the freshness check is what keeps the cache truthful when no webhook is configured, and every deployment starts that way.
168The endpoint is per host flavour, deliberately.
POST /v1/hooks/{source} where {source} names the git host's flavour
(github today; other flavours are added when one is measured, not reserved in
advance). One route per flavour rather than one generic route, because the
payload shape and the signature scheme are host-specific and a single endpoint
would have to sniff which it received — which is guessing, dressed as
convenience.
169A webhook authenticates with its own shared secret, never with an admin token. The sender signs the request body under a per-hook secret and the index verifies that signature; an unverifiable request is refused and not processed. Two reasons, and the second is the important one. (i) The secret is held by a third party — the git host — and must be rotatable without touching the tokens that authorise real writes. (ii) Least authority: an admin token authorises arbitrary writes, while a webhook may only cause the index to re-read one repository. Handing a notification channel the authority to write anything is how a notification becomes an attack surface. Secret storage follows §7 — file, never a flag, never a log.
170The load-bearing rule: a
payload says that something changed, never what it now is. The index reads
the manifest from the git host itself and applies the per-package add / remove
of §2.8 for the named repository only. Nothing from the request body
is ever written into an index record. This follows from §2.3 — package
repos remain authoritative — and it is the rule that keeps an attacker-influenced
input from becoming index content. A webhook may never trigger a full
reindex, both because that is the expensive thing it exists to avoid and
because a cheap request that causes an expensive walk is a denial-of-service
lever.
171Deliveries arrive twice, out of order, or
not at all, and the design assumes all three. The handler is idempotent by
construction — re-reading a repository and upserting its versions is the same
operation performed twice — and it never infers from one event that it saw the
previous one. Consequence, stated so it is not quietly dropped later:
webhooks reduce staleness, they do not abolish the full walk. The explicit
rescan-org verb of §2.8 stays unconditional exactly because a
missed delivery is invisible from inside.
172What each failure answers. An unverifiable
signature is 401 and is not processed. A payload naming a repository outside
this server's configured organisation is 400 and is not processed — the scope
check is not optional, since the repository name is the one field of the payload
we act on. A verified, in-scope delivery whose re-read fails is 202: the
request was accepted and the work is ours to retry. Answering 5xx to a sender
that retries on a schedule we do not control would turn our own outage into a
retry storm — the failure is on our side, and the status code should say so
rather than invite the sender to hammer.
173The GitHub-Actions alternative, and why it is the
fallback rather than the default. The same effect is reachable without any
endpoint: an Action in the package repository POSTs to the write API of
§2.10 with an admin token. It is genuinely simpler — no new route, no
signature verification, nothing to specify. What it costs is exactly what
#WEBHOOK-SECRET-IS-NOT-THE-ADMIN-TOKEN protects: every participating
repository then holds a credential that can write anything, and the number of
places a broad secret lives grows with the organisation. The webhook keeps one
narrow secret in one place. So: webhook by default; the Action is the honest
answer for an organisation that already runs its publishing through Actions and
prefers one mechanism to two, and it has one capability the webhook does not —
it can also push the built index files, being a runner with a checkout.
174Named, not invented — two things this section does
NOT settle. (i) Whether the server should require the pushed ref to look
like a release tag before acting, or re-read on any push to the default branch
as well: the second catches a manifest edited without a tag, which
§2.8's incremental walk already treats as a real case, and the first
is cheaper. Decide it against a measured event volume, not here. (ii) The
GitVerse flavour is unmeasured: their public API could not enumerate an
organisation, and nothing here establishes what their webhooks can do. Writing a
gitverse route from that ignorance would be inventing a contract for a system
nobody in this repository has watched.
2.16.1 Setting one up — the operator's walkthrough
175Why this walkthrough sits inside the
specification and not in docs/ (owner ruling, 2026-08-06). It describes how
to configure a mechanism whose properties this document defines. Kept beside
the contract it changes when the contract changes, because the two are the same
file and the same commit; kept in docs/ it drifts. That is not a hypothetical:
this repository measured two independent instances of exactly that drift in a
single week — the index's own format documentation against its code, and an
owner guide promising a gate step that did not exist.
176Read the steps below as a
specification, not as instructions that work today. The endpoint is designed
here and not built; the block is therefore an example and carries no
@fact/code: marker, which is precisely the distinction that keeps a fenced
block from asserting something nobody can falsify. When the route ships, this
block becomes the fact's body and comes due with it.
177# 1. On the index host: put the shared secret where the server reads it.
# One secret per configured hook; file, not a flag (§7).
$ printf '%s' "$SECRET" > ./vibespecs-index/state/webhook.secret
# 2. Start the server with the hook route enabled.
$ vibe-index serve ./vibespecs-index --bind 0.0.0.0:8412 \
--auth-tokens-file ./vibespecs-index/state/admin.tokens \
--webhook-secret-file ./vibespecs-index/state/webhook.secret
# 3. On the git host, at the ORGANISATION level (not per repository —
# a per-repo hook is one more thing to remember on every new package):
# payload URL https://<index-host>/v1/hooks/github
# content type application/json
# secret the same $SECRET
# events pushes and tag/release creation only — not "everything"
#
# 4. Verify the wiring before trusting it: push a tag to any package repo,
# then ask the index what it now knows about that package.
$ vibe-index get ./vibespecs-index <group> <name>
178Step 4 is not politeness. A hook that is
configured and silently not arriving looks exactly like a hook that is arriving
and finding nothing to do, and the difference is invisible from the index side —
which is the same shape as #WEBHOOK-DELIVERY-IS-UNRELIABLE, seen at setup time
instead of at run time. Verify once against a known change; after that, trust
the mechanism and keep rescan-org for the deliveries you will never know you
missed.
2.17 Auto-publication — the server carries its own result to the host
179What it fixes. The server
already accepts an authenticated write, writes the files atomically, recomputes
the manifest and verifies integrity. The one thing it could not do was carry
the result to where it is served from. --auto-commit-push closes that: after
each successful mutation the server commits the data directory and pushes it.
Built 2026-08-06 on the owner's ruling; the flag had been declared and discarded
by one line since the server shipped.
180Where it publishes
is the operator's setting, and vibe-index does not mint a second place to say
so. The data directory is already a git working tree (§2.4), and a
working tree's remote and branch are configured with plain git. So the push
carries no refspec and names no remote — it goes where the tree is already
pointed. A private repository is a legitimate target by construction: it is
simply what the operator cloned. Rejected: a --push-remote / --push-url
pair, and a target block in the on-disk config — both would be a second home for
a value git already owns, which is the defect class this repository keeps
paying for.
181Startup refuses rather than warns,
and this is the flag's most important behaviour. state/ holds the bearer
tokens (§7). If the data directory's .gitignore does not cover
them — a directory created before init wrote one, or one edited since —
git add -A would stage those tokens and push them to a host that may be
public. So with the flag set, the server does not start unless it confirms
state/admin.tokens is ignored, and the refusal says so in those words. The
check runs once at startup rather than per mutation, because the operator must
learn the configuration is unsafe before the first token leaves the machine,
not after.
182The second refusal: the flag set over a data directory that is not a git working copy also stops the server, naming what to do. Publishing by committing a directory that is not tracked is not a degraded mode, it is a no-op that would look like success forever.
183A push failure never
turns a successful write into an error. By the time publication runs the
mutation is on disk and in memory; the HTTP write has happened. So a failure is
logged at warn with git's own message and counted as
vibe_index_publish_failures_total, and the request still answers as it would
have. It is not rolled back either: a network outage must not be able to corrupt
index state. Transient failures self-heal — git accumulates, and the next
successful push carries the queued commits.
184Nothing to commit is success, not an error — the opposite of the publish flow's rule for the same operation, and deliberately so. The index lock is released before the push, so a second mutation can land while the first is publishing and the first commit carries both. The second then finds nothing staged, and that is the normal course of events rather than a caller's mistake.
185One publication at a time, and the
response waits for it. Two concurrent mutations must not interleave two
commits in one working copy, so publication takes its own lock — not the index
lock, which is released earlier and correctly so. The handler awaits the result
on a blocking thread, which means a 200 says «persisted and published».
For an index that is what the operator asked for, and mutations are publish
events rather than a hot path.
186The commit message names what moved — the upsert or the removal, with the package coordinate. Each of the three mutating routes knows its own change, so the index's history reads as a log of publications rather than a wall of identical messages.
187<status stage="spec" state="void">Retired 2026-08-14 by the determinism phase. It recorded a real consequence of the writer stamping a fresh generation time on every write: a repeated identical upsert produced a diff and therefore a commit, so the empty-diff path fired only on the overlap case. Both halves of that consequence are gone — the writer takes its clock as an input (##THE-WRITER-TAKES-ITS-CLOCK-AS-AN-INPUT), and a mutation that changes nothing no longer writes at all (##A-MUTATION-THAT-CHANGES-NOTHING-COMMITS-NOTHING). Its closing sentence asked whether the index's own write should be deterministic and called that a question about the format rather than about this flag; the format answered yes. This tombstone stays so the old sentence's name is never reused and inbound links do not break.</status>
188A mutation that changes nothing writes nothing, and therefore commits nothing. An upsert whose entry equals the one already stored under that version number leaves the in-memory state untouched, never reaches the writer, and never reaches the publisher; the two removal routes have always behaved this way, and the upsert route now matches them. The response is still success — the resource is already in the requested state, which is what idempotency means over HTTP — and the distinction between created and changed is kept, because a differing entry under an existing version number is an update and must still land. Determinism alone would not have bought this: with the clock arriving per mutation event, a repeat would still have moved generated_at and produced a diff. The point is not to produce an empty diff but to not create the work, so that the catalog's history records events that actually happened.
189No identity is invented.
The commit uses whatever git identity the host is configured with; if there is
none, git commit fails and that failure takes the path above — logged and
counted, never fatal to the write. Inventing a fallback author would put a name
in an organisation's published history that nobody chose.
2.18 Channels — author-named version pointers
190req r2
191Decision (owner rulings, 2026-08-13; not
built — this section is the contract the build will follow). A channel
is an author-controlled named pointer (group, name, channel) → version —
npm's dist-tags and Docker's tags are the prior art. The pointer map is
flat: several channels may point at one version (a release that is both
latest and stable is the everyday case), and no promotion semantics
(beta → stable as a registry operation) are baked into the format —
promotion is the author's workflow, not registry law. A channel may point at
a snapshot or at a frozen version (PROP-044 §2b) — the axes are orthogonal.
- 192What already exists, so nobody mints it twice: the journal's event vocabulary carries
ChannelSet {group, name, channel, version}andChannelUnset {group, name, channel}in exactly the shape below, and the generated entry types carry achannelslist. What is NOT built is the projection and the surfaces. And the projector's treatment of that gap is the load-bearing part: meeting a channel act it refuses the whole projection by name — «the journal holds aChannelSetrecord, but its carrier (channels) is not built in this vibe-index; skipping the record would project a catalog the journal does not describe» — rather than skipping the record and continuing. The journal is truth (§2.3); a projector that quietly dropped an event it did not understand would publish a catalog asserting a state nobody recorded, which is the one failure re-fetching cannot cure.NoticeandForceReplacedstand in the same place for the same reason.Renamedstood there too until the retirement collapse: it left the vocabulary, and its successorBuriedis the one arm that went the other way — it gained a carrier and now PRODUCES a tombstone instead of refusing, which is what makes it the first producing arm this projector has ever had (§2.11). - Channel-name grammar:
[a-z][a-z0-9-]*— it must not start with a digit (versions do) or a version-requirement operator (^ ~ = < > *), which is what makes@betaunambiguous in a pkgref's version position. - Authority is the journal; the
catalog only projects. Channel state changes are registry facts:
Publishedcarries the manifest-declared channels (below), and the explicit actsChannelSet {group, name, channel, version}/ChannelUnsetretarget or clear a pointer. No hand-edited pointer file exists anywhere — a hand-writtenvibeversions.tomlinside the derived catalog would be a secretly-authoritative fact (PROP-044 law 2). The projection lands the map in theNameEntry(channels: {stable → 1.1.0, beta → 1.2.0-rc.1}) — the sameby-name/<name>.jsoncandidate file the resolver already fetches, so channels cost zero additional round-trips. - The manifest declares
membership as a publish-time fact, never as the pointer.
[package] channels = ["stable", "lts-2026"](a list — multiplicity is first-class; the singularchannel = "…"is rejected with a did-you-mean) records which channels this version was published into — immutable with the content, honest forever. Publication moves each named pointer to this version (npm'spublish --tagsemantics), so routine channel management is just publishing — no separate command. The pointer itself cannot live in the manifest: retargetingstableback to a frozen1.1.0(the rollback — the main use case) would require editing frozen bytes, which is forbidden; that act is the journal'sChannelSet, viavibe registry channel set <group>:<name> stable 1.1.0. - LATEST and STABLE are the two
built-in channels (Maven's
<latest>/<release>pair). A channel is authored from the first explicit act (a manifest declaration or achannel set) and stays authored untilchannel unset; while unauthored it is computed at projection time: STABLE = the greatest non-prerelease version, LATEST = the greatest version outright — both by the ordering below. A new publication without declarations does not move an authored pointer: if the author saidstable = 1.2.0, releasing 1.3.0 does not silently make it stable. - The ordering (owner ruling,
2026-08-13): SemVer precedence first, natural-sort tie-break on build
metadata second. SemVer 2.0.0 is not modified:
+buildis legal in published versions, the coordinate is the full version string including+…(uniqueness is hard on the string), and precedence ignores metadata exactly as the standard demands — every foreign semver library agrees with us. Where SemVer declares two versions equal, our resolver breaks the tie deterministically: versions with metadata outrank the bare version (a+stampis a rebuild atop it), and among metadata the greater under natural sort (digit runs compare numerically, text lexicographically) is the fresher — so+20260813…beats yesterday's stamp. Reproduction is never at stake — the lockfile pinscontent_hash— the tie-break only answers "which is latest", and only those who chose to publish+twins pay the axis any attention. - The resolver default (owner
ruling, 2026-08-13):
vibe install pkgwith no requirement and no channel takes STABLE when it exists, else LATEST — and the frozen/snapshot state does not influence selection at all. Selection and integrity are separate axes: the chosen version is pinned by hash in the lockfile, and after selection the freeze contract governs mismatches (frozen — alarm; snapshot — news). Requesting a channel is the explicit act:{ channel = "beta" }on a dependency invibe.toml(mutually exclusive with a version requirement) or@betain a pkgref's version position. - Resolution through a channel pins
{channel, resolved version, content_hash, locator}in the lockfile:vibe installreproduces the pin;vibe updatere-follows the pointer;--lockedturns any drift into a loud CI error. - An authored pointer at a dead target is a loud state. When a channel's target version is yanked or removed, computed channels simply recompute past it, but an authored pointer refuses at resolve time with a recipe («stable указывает на отозванную 1.2.0 — автор должен переставить или снять; потребитель может явно взять версию»). Silently hopping to "the next best" would be a choice the author never made.
- The degraded ladder (catalog
unreachable). (1) A lockfile answers without the catalog at all —
resolve-by-lock never needs it. (2) No lock but a local catalog cache →
resolve against the cache, loudly stamped «по снимку каталога от <даты>».
(3) Cold resolve with the provider alive → enumerate versions at the
provider (
ls-remote --tags-class), read manifests at tags, and let the local resolver reconstruct channels from the publish-timechannelsdeclarations — approximate exactly where the author manually retargeted, and the output says so («по перечислению провайдера, каталог недоступен»). (4) Neither reachable → refusal with a recipe. Degradation is always announced, never silent (PROP-044 law 1).
3. Architecture
3.1 Crate layout
202design r1
203crates/vibe-index/ # a member of the vibevm workspace
├── Cargo.toml # depends on vibe-core + vibe-wire; no [workspace] table
├── README.md # operator-facing — how to run, common recipes
├── src/
│ ├── main.rs # bin entrypoint — clap dispatch
│ ├── lib.rs # exports, top-level Error/Result
│ ├── error.rs
│ ├── cli/ # one file per verb (§2.11) + kinds.rs
│ ├── journal/ # THE AUTHORITATIVE LAYER (§2.3)
│ │ ├── record.rs # the event vocabulary
│ │ ├── store.rs # append-only shards under state/journal/
│ │ ├── project.rs # journal → catalog; refuses unbuilt carriers
│ │ └── mod.rs
│ ├── index/
│ │ ├── mod.rs # the writer's owned surface (§2.4)
│ │ ├── memory.rs # Index struct + ops
│ │ ├── quarantine.rs # the reader's judgement + the refusal (§2.19)
│ │ ├── persistence.rs # atomic write/read of files
│ │ ├── primary.rs # JSONL serialise/parse
│ │ ├── by_name.rs # candidate-set JSON
│ │ ├── inverted.rs # by-cap / by-purl
│ │ ├── repomd.rs # repomd.json
│ │ ├── checkpoint.rs # incremental-reindex state
│ │ └── search.rs # per-query postings, not a stored index
│ ├── scanner/
│ │ ├── mod.rs # source-of-truth walkers
│ │ ├── from_clones.rs # walk org-dir clones via shell git
│ │ ├── from_github.rs # GitHub REST API walk
│ │ ├── org_walk.rs # the organisation enumeration
│ │ ├── org_cache.rs # the org image + its validator (§2.8.1)
│ │ ├── manifest.rs # parses through vibe-core
│ │ └── git_cli.rs # the shelled-out git
│ ├── server/
│ │ ├── mod.rs # axum app builder — the 16 routes of §2.10
│ │ ├── routes/ # health · index_files · packages · capabilities
│ │ │ # · purls · admin · metrics
│ │ ├── auth.rs
│ │ ├── error.rs # RFC-7807 mapper + the refusal extension
│ │ ├── rate_limit.rs # per-token / per-IP buckets (§9 Q10)
│ │ ├── metrics.rs # hand-rolled text serialiser
│ │ └── state.rs # AppState
│ ├── types/ # re-export seam over the generated wire types
│ │ ├── mod.rs
│ │ ├── entry/ # aggregate · content · relations
│ │ ├── repomd.rs # the one hand-written shape (§2.12)
│ │ └── kinds.rs # PackageKind, NamingConvention dupes
│ ├── publish.rs # auto-commit-and-push (§2.17)
│ ├── lock.rs # the single-writer PID lock
│ ├── lockfile.rs # reading a vibe.lock for `outdated`
│ ├── hash_recipe.rs # the recipe a content_hash rides with
│ └── content_hash.rs # mirrors vibe-registry::compute_content_hash exactly
├── fixtures/
│ ├── golden-flow-wal-1.0.0/ # the parity fixture
│ └── golden-order-trap-0.1.0/ # the tree where recipes 0 and 1 disagree
├── tests/ # help_smoke · cli_{lifecycle,read,write} · server_e2e
│ # · server_writes · auto_publish · rate_limit_e2e
│ # · org_cache_e2e · scanner_e2e · from_github_e2e
│ # · golden_corpus · round_trip_published
│ # · content_hash_parity · six wire_parity_*
└── docs/
├── operator-handbook.md
├── consumer-protocol.md # HTTP API reference
└── format.md # repomd / primary / by-name / by-cap / by-purl
204There is no LICENSE inside the crate, and that is the correct state. Cargo.toml carries license-file.workspace = true, so the crate inherits the repository's licence rather than keeping a second copy that can disagree with it — the same single-home rule this document applies to normative values, applied to the one value a licence is. The repository's licence is UPL-1.0.
3.2 Dependencies
205design r1
206Minimal Rust crates to keep redistribution clean:
- 207
clap(derive) — CLI dispatch. tokio— async runtime for the server, with the four features named below.axum— HTTP framework. Mature, minimal, integrates withtowermiddleware.tower/tower-http— auth, CORS, tracing layers.serde/serde_json— JSON.toml— read package manifests.semver— version handling. Same depvibe-coreuses; pin same version.sha2— content_hash. Same asvibe-registry.flate2— gzip primary.jsonl.gz.walkdir— directory traversal (matchesvibe-registry).tracing/tracing-subscriber— logging.chrono— timestamps.thiserror— error enums.gix(or shell-out viastd::process::Command) — read git tags / show files at refs. Decision §3.3.reqwest—--from-githubHTTP client.tempfile— atomic write helpers.- <status stage="spec" state="void">Retired 2026-08-18 by measurement: the
prometheuscrate is not a dependency and never became one —/metricsrenders the exposition format from a hand-written serialiser. The heir is##THE-METRICS-DEPENDENCY-WAS-NOT-TAKENbelow, which records the choice and its reason. This tombstone stays so the anchor's name is never reused and inbound links do not break.</status> specmark— the in-code spec markers (scope!,#[spec]) the traceability map is built from.vibe-wire— the generated wire types this crate'stypesmodule re-exports (§2.12). A runtime dependency, not a test one: the library's types ARE the wire's types.
208No prometheus crate is pulled, and the omission is the design. /metrics renders the Prometheus text exposition format from a hand-written serialiser, because the surface is a handful of counters and the exposition format is stable text — so the dependency would buy formatting we can write once and cost a tree we then carry forever. This is what «minimal crates to keep redistribution clean» means when it is applied rather than stated.
209tokio is taken with four features — signal, sync, time, fs — not full. full is the shape a project reaches for before it knows what it uses; naming the four is the same discipline as the paragraph above, one level down.
210vibe-core dependency. vibe-index parses vibe.toml and vibe-subskill.toml through vibe-core's own Manifest / SubskillManifest types, so the index can never drift from the manifest schema. This reverses the proposal's original standalone-no-vibe-core stance — §6 records the reversal, §9 item 11 the de-rot finding that forced it. What stays duplicated is small and stable: the four-variant PackageKind / NamingConvention (src/types/kinds.rs, frozen by VIBEVM-SPEC.md §4, needing the Ord + clap::ValueEnum the vibe-core originals lack) and the compute_content_hash algorithm (src/content_hash.rs, gated by tests/content_hash_parity.rs). compute_content_hash folds into vibe-core once it is lowered out of vibe-registry.
211The parity gate is wider than one fixture and one algorithm. It runs BOTH implementations over BOTH fixtures in BOTH recipes: fixtures/golden-flow-wal-1.0.0/ is the ordinary package, and fixtures/golden-order-trap-0.1.0/ is the tree built to make recipes 0 and 1 disagree — a directory whose name is continued by a sibling file at a byte below /, the only shape on which component-wise and byte-wise ordering part company (PROP-002 §2.1). A single fixture would let the two implementations agree by accident on every tree that never exercises the difference, which is exactly how a hash regression once reached a consumer before any golden noticed.
212Deliberately NOT pulling:
- 213A database (SQLite / PostgreSQL). All state in RAM + flat files.
3.3 Git access in the scanner
214design r1
215Decision. Use shell-out to git via std::process::Command for the scanner's read paths (git tag, git show <ref>:<path>, git rev-parse <tag>). Same path vibe-registry::shell.rs already follows. Rationale matches PROP-001 §2.1: shell-out works on every platform git works on, no per-host bindings to maintain.
216Not gix for v0: smaller dep tree wins. v1 may switch if perf demands and gix's read API matures further.
3.4 Threading model
217design r1
- 218CLI mode: synchronous. tokio runtime is created only in
servesubcommand. - Server mode: tokio multi-thread runtime. Routes are async;
Arc<RwLock<Index>>istokio::sync::RwLock(async lock). - <status stage="spec" state="void">Retired 2026-08-18 by measurement. It described a dedicated
index_writertokio task fed by an mpsc channel, so that fsync stalls would not block the request handlers. Neither the task nor the channel was ever built — measured as zero occurrences of both names in the crate, against a live control — and the problem they were designed for dissolved when a mutation became an append to the journal plus a reprojection. The heir is##THE-MUTATION-IS-WRITTEN-BY-ITS-OWN-HANDLERbelow. This tombstone stays so the old sentence's name is never reused and inbound links do not break.</status>
219Each mutating handler does the whole write itself, under the index's async write-lock: replay the journal, project a probe, append the event, reproject, write the catalog, swap the in-memory index. There is no writer task and no channel to post to. The queue that the task was reaching for turned out to be unnecessary once mutations became journal appends: a handler holds the lock for one append plus one projection, and the operations it serialises are publish events rather than a hot path (§2.17 makes the same argument about awaiting the push). What IS serialised separately is publication — its own lock, and a blocking thread, because a git command must not run on the async executor.
3.5 Configuration precedence
220design r1
221For every flag with a default, precedence is: explicit CLI flag > env var (VIBE_INDEX_*) > on-disk config (<data-dir>/state/config.toml, optional) > built-in default. Same shape vibe show config already uses on the consumer side. (Built 2026-08-20, B-086: the machine is crates/vibe-index/src/config.rs — every resolution carries the value AND its source, surfaced by the vibe-index config <data-dir> verb; the file rung parses strictly, an unknown key is a loud refusal. Four members ride it (log-level — with VIBE_LOG as the recorded legacy synonym below VIBE_INDEX_LOG, git, api-base, dump-format); the remaining ~17 flags-with-defaults join by the worked pattern — the mechanical continuation and its two owner forks (boolean value dialect; the --limit naming collision) are listed in the R3-LADDER worker report and the release-TZ log. data-dir is not a member: it is the required positional (##CLI-SURFACE) and the file rung lives inside it.)
222None of that ladder exists. There is no config.toml anywhere in the crate and no VIBE_INDEX_* family: a flag with a default gets it from its own declaration, full stop. Two environment variables do exist and neither is part of a precedence chain — VIBE_INDEX_GIT overrides the git binary the scanner shells out to, and VIBE_LOG is the logging lever --log-level folds into (§2.11). The requirement stands and the fork is BACKLOG.md B-086: build the ladder, or say plainly that this binary is configured by flags and two named variables. What must not survive is the middle state, where a document describes a resolution order an operator can neither use nor observe.
4. Phase plan (slices)
223Each slice = one or more conventional commits. The utility becomes useful at slice 5 (read endpoints + reindex from clones); the rest are progressive enhancements.
4.1 Slice 1 — skeleton
224crates/vibe-index/ standalone crate with Cargo.toml + src/main.rs + src/lib.rs. clap dispatch with stub subcommands that all print "not yet implemented". vibe-index --version works. tests/help_smoke.rs passes.
225Commit: feat(services/vibe-index): skeleton crate + clap subcommand dispatch.
4.2 Slice 2 — types + persistence
226src/types/ (entry / repomd), src/index/ (memory, persistence, primary, by_name, repomd). JTD schemas in schemas/. Atomic write protocol. vibe-index init works (writes empty repomd.json + empty primary.jsonl). vibe-index dump works. vibe-index verify works (checks file hashes). Round-trip tests.
227Commits:
- 228
feat(services/vibe-index): index entry + repomd schemas + JTD feat(services/vibe-index): in-memory index + atomic persistencefeat(services/vibe-index): vibe-index init/dump/verify
4.3 Slice 3 — scanner + reindex from clones
229src/scanner/from_clones.rs walks <org-dir>/<repo>/.git directories; src/content_hash.rs mirrors vibe-registry::compute_content_hash; vibe-index reindex --from-clones works against fixtures/sample-org/. Parity test against vibe-registry. Incremental mode = full for now (deferred to slice 7).
230Commits:
- 231
feat(services/vibe-index): content_hash parity with vibe-registry feat(services/vibe-index): scanner — walk org-dir clonesfeat(services/vibe-index): vibe-index reindex --from-clones
4.4 Slice 4 — read CLI subcommands
232get, list, search, capabilities, purls, outdated. Inverted text index for search. JSON output for every subcommand. cli_e2e.rs covers each.
233Commits:
- 234
feat(services/vibe-index): inverted text index for search feat(services/vibe-index): vibe-index get/list/search/capabilities/purlsfeat(services/vibe-index): vibe-index outdated against a vibe.lock
4.5 Slice 5 — HTTP server (read-only)
235vibe-index serve --read-only. axum app exposes /healthz, /readyz, /v1/index/*, /v1/packages*, /v1/capabilities/*, /v1/purls/*, /metrics. PID lock file. CORS open. server_e2e.rs covers each route.
236Commits:
- 237
feat(services/vibe-index): axum server skeleton + healthz/readyz feat(services/vibe-index): GET /v1/index/* file routesfeat(services/vibe-index): GET /v1/packages query routesfeat(services/vibe-index): /metrics prometheus endpoint
238After slice 5: vibe-index is independently usable as a read-only server fed by reindex --from-clones. This is the "MVP" mark.
4.6 Slice 6 — write CLI + write HTTP + auth
239vibe-index add / vibe-index remove. HTTP POST /v1/packages, DELETE /v1/packages/.... Bearer-token auth via <data-dir>/state/admin.tokens. Write-side server-vs-CLI lock arbitration.
240Commits:
- 241
feat(services/vibe-index): vibe-index add/remove (CLI) feat(services/vibe-index): bearer-token auth + admin.tokens loaderfeat(services/vibe-index): POST/DELETE /v1/packages routes
4.7 Slice 7 — incremental reindex
242<data-dir>/state/checkpoint.json. vibe-index reindex --incremental --from-clones walks the diff between checkpoint and current state. Test: full vs incremental produce identical output.
243Commit: feat(services/vibe-index): incremental reindex via checkpoint.
4.8 Slice 8 — --from-github mode
244reqwest-based GitHub API walk. --token-file FILE. Rate-limit-aware backoff. Same shape as --from-clones from caller's POV.
245Commit: feat(services/vibe-index): reindex --from-github (REST API walk).
4.9 Slice 9 — vibe-publish post-publish hook
246crates/vibe-publish/src/lib.rs::Publisher::publish gains optional index POST after successful push. New env var VIBEVM_INDEX_TOKEN_<HOST>. New [[registry]].index_url / [[registry]].index_token fields in the project manifest.
247Commit: feat(vibe-publish): POST to registry index after publish (opt-in).
4.10 Slice 10 — consumer-side integration
248crates/vibe-registry/src/multi_registry_resolver.rs gains the index-aware fast path. Falls back transparently on 404 / connect-failure. Live e2e test against an index-equipped registry.
249Commit: feat(vibe-registry): consume registry index for resolve fast path (opt-in).
4.11 Slice 11 — docs + manual-test smoke
250crates/vibe-index/docs/ filled in (operator-handbook, consumer-protocol, format). manual-tests/M2.10-index-smoke.md walks the live e2e: bootstrap an index from a fresh org dir, serve it, install a package through it, search it.
251Commits:
- 252
docs(vibe-index): operator handbook + consumer protocol + format reference test: manual-test smoke for index bootstrap + consume
5. Test plan
253Per slice (specifics in §4); cumulative state at GA:
- 254Unit: every type round-trips through serde JSON / TOML; every CLI subcommand has at least one happy-path test; every server route has at least one happy-path + one auth-fail test.
- Integration: the byte-identity check moved out of the crate's own fixtures and into the campaign's golden corpus —
formats/corpora/index/e1/holds a journal and the catalog it projects to,tests/golden_corpus.rscompares them, andvibe-index rebuild <data-dir> --checktears the catalog down and rebuilds it from the journal. The maintainer-sidecargo xtask rebuild --check <data-dir>surface calls the same library implementation as a compatibility wrapper. Incremental reindex applied to the same starting state is still byte-identical to a full one. The move is the point: the golden now lives beside the format registry that governs it rather than beside one consumer of it. - Parity:
tests/content_hash_parity.rsruns both implementations over both fixtures in both recipes (§3.2), and asserts equality within each recipe. CI gates the merge if they diverge. - End-to-end:
tests/server_e2e.rsspawns the server in-process (axum'soneshotstyle), drives every documented route over HTTP, asserts response shapes. - Crash recovery:
tests/persistence_atomic.rssimulates mid-write crash by failing the rename step; asserts the previous version remains readable. - Hermetic, with no live tier at all: every test runs without network, and the GitHub API walk is proved against a mock REST server on a random port whose canned responses point at local bare repositories, so
git cloneresolves entirely against the filesystem (tests/from_github_e2e.rs). The opt-in live run against the real registry this section once promised does not exist and is not owed: a test that needs the internet and a real organisation is one nobody runs, so it proves nothing on the day it would have mattered, while the mock proves the walk's shape on every commit. What it deliberately does not prove is that the real host still answers the way the mock does — that question belongs to the manual tier, not to an ignored test.
6. Distribution — essential member of the VibeVM bundle
255design r2
256Decision (revised 2026-05-22). vibe-index lives at crates/vibe-index/ as a member of the top-level vibevm workspace. It is built, tested, clippy-gated, and fmt-checked by the same cargo … --workspace invocations as every other crate, and it depends on vibe-core directly.
257Why this reverses the original standalone-workspace decision. The proposal first placed vibe-index in its own Cargo workspace under services/, outside crates/, so an org owner could vendor just that subdirectory. The cost was a hand-duplicated vibe.toml parser with nothing tying it to vibe-core — and, sitting outside cargo test --workspace, nothing routinely exercising it. It rotted silently against the M1.17 / M1.18 manifest-schema churn (§9 item 11). Folding the crate back in kills both failure modes at once: the scanner now parses through vibe-core::Manifest (one source of truth — the schema cannot drift), and the routine workspace gate covers it (drift is caught the moment it appears).
258Redistribution. vibe-index is an Essential component of every native
VibeVM distribution, installed and switched atomically beside vibe; this is
the default on-premises operator path. Its matching source is available through
vibe self source. A developer may still build it from the workspace with
cargo install --path crates/vibe-index, but consumers no longer need to clone
the repository merely to operate an index. Keeping it in the workspace still
prevents manifest-schema drift through its direct vibe-core dependency.
259Gate. tools/self-check.sh no longer special-cases a second workspace — the workspace-wide steps (fmt, then cargo test --workspace, then cargo clippy --workspace --all-targets -- -D warnings) cover vibe-index like any member.
260What has changed since is the opposite of a special case: the panel grew steps that look specifically at this crate, and they are the interesting half. A clock gate greps crates/vibe-index/src/{index,types,journal} for any call that reads the wall clock, because determinism here is an instrument rather than a preference (§2.9); check-codegen refuses a drift between the schemas and the generated types (§2.12); specmap --check refuses a stale traceability map; and the wire-derive ratchet refuses a hand-written wire. None of them is a workspace step, and none of them would fire from cargo test alone — which is why the number of steps in the panel is not a proxy for what it checks.
7. Auth, secrets, scope
261req r1
262PROP-000 §20 covers the token-secrecy invariant; PROP-005 inherits it verbatim. Specifically:
- 263Server admin tokens (
<data-dir>/state/admin.tokens) — file mode 0600, never echoed in logs / responses / error messages, gitignored. - GitHub API tokens (
--from-github --token-file FILE) — same discipline. Read once into memory, scrubbed from the env, never persisted outside the source file. - Index POST tokens (
VIBEVM_INDEX_TOKEN_<HOST>for the publish-side hook) — per-host shape mirrorsVIBEVM_PUBLISH_TOKEN_<HOST>.
264Scope discipline. The server's mutation endpoints accept entries only for the registry the server was started with (<data-dir>/repomd.json::registry). A POST attempting to land an entry whose registry field disagrees with the server's configured registry → 400 with a clear message. Same shape vibe-publish::validate_scope enforces on the publish side.
8. Operations
265A typical setup for an org owner who wants to host an index:
266# One-time bootstrap (on a host with the org's clones available).
$ vibe-index init ./vibespecs-index --registry vibespecs --registry-url https://github.com/vibespecs --naming fqdn
$ vibe-index reindex ./vibespecs-index --from-clones /var/lib/vibespecs-mirror
# Push the static files to <org>/index repo (operators wire this once):
$ cd ./vibespecs-index
$ git init && git remote add origin https://github.com/vibespecs/index
$ git add . && git commit -m "initial index"
$ git push -u origin main
# Run the live server (optional — only if hosting the HTTP-API path):
$ vibe-index serve ./vibespecs-index --bind 0.0.0.0:8412 \
--auth-tokens-file ./vibespecs-index/state/admin.tokens
# Periodic incremental refresh (cron):
$ */5 * * * * vibe-index reindex /home/owner/vibespecs-index --incremental --from-clones /var/lib/vibespecs-mirror
267Most consumers see only the static raw-HTTP files; the server is for orgs that need real-time publish updates.
9. Open questions
- 268Index file location:
<org>/indexrepo vs<org>/<package-repo>/index/...per-package? PROP-005 picks<org>/index. The alternative — per-package files inside each package repo — was rejected because it leaves catalog discovery a chicken-and-egg problem (you need to enumerate the org first). If new evidence emerges that orgs object to a top-levelindexrepo (naming conflicts, permission boundaries), we revisit. primary.jsonl.gzcompression: gzip vs zstd? v0 picks gzip — universally supported by every HTTP client; deterministic withmtime=0. v1 may add aprimary.jsonl.zstalongside.- GPG signing of
repomd.json? Tracked here, not shipped in v0. Shape:repomd.json.ascnext torepomd.json; consumers verify against a per-registry public key recorded in[[registry]].pgp_key. v1. - Merkle log (Go sumdb-style transparent log)? Tracked here. v2+. Useful for adversarial environments; v0/v1 trust the host.
- Auto-commit-and-push from server — ANSWERED and built 2026-08-06; the contract is §2.17. The risk this question named — the server holding push credentials is a step up in trust — is unchanged and is why the flag stays opt-in with manual commit/push as the default. What the build added to the answer is a second risk the question had not seen: the credentials are not the only secret in reach, because the data directory also holds the server's own bearer tokens, and the publishing step is a
git add -Aaway from them. Hence the startup refusal in §2.17 rather than a warning. - Multi-registry server — should one server instance host multiple data dirs (one per registry)? v0 says no (one process per registry). Trivial scale-out via process supervision; we revisit if multi-tenancy demand emerges.
- WebSockets / Server-Sent Events for live publish notifications — out of scope. Polling
/v1/admin/status::last_reindexis sufficient at our scale. - OCI registry shape — could we host the index inside an OCI registry instead of git? Out of scope; revisit if the OCI tooling becomes universal among vibevm operators.
- Capability- vs PURL-driven search — v0 ships
by-capandby-purlas separate files. If usage shows one dominates, the loser may be folded into the inverted text index. Empirical question. - Rate-limiting on the server — shipped after the v0 plan:
server/rate_limit.rsis a per-token and per-IP token-bucket limiter, disabled by default and opt-in by flag. Production deployments may still front it with a reverse proxy's limiter; the two compose.
- 269Standalone-workspace duplication — RESOLVED 2026-05-22 by folding the crate in. The 2026-05-22 de-rot found
crates/vibe-index/had silently rotted: its duplicatedvibe.tomlparser still expected the pre-M1.17 shape ([writes],[dependencies],[boot_snippet].filename) and could not parse a current manifest, and itscontent_hashparity test had drifted off a fixture renamed by the M1.17 manifest unification. §3.2 had weighed the duplication cost forcompute_content_hashalone ("the algorithm doesn't change"); the manifest schema, by contrast, churned hard through M1.17 / M1.18, and the duplicate parser had no cross-check to catch the drift — the standalone workspace also sat outside the routinecargo test --workspacegate. Resolution: foldvibe-indexinto thecrates/workspace and parse throughvibe-core::Manifest(see §6). The duplicated parser is deleted; only the tiny, schema-frozenPackageKind/NamingConventionand thecompute_content_hashalgorithm remain duplicated, both justified in §3.2.
10. Acceptance criteria
270A given slice is considered accepted when:
- 271All tests in its slice pass.
cargo clippy --workspace --all-targets -- -D warningsis clean.cargo fmt --checkis clean.tools/self-check.shis green.- Help-text smoke covers any new subcommand.
- A manual walk through
crates/vibe-index/docs/operator-handbook.mdsucceeds againstfixtures/sample-org/.
272PROP-005 is considered closed once slices 1–8 land. Slices 9–11 are integration with the rest of vibevm and ship under their respective milestone PRs.
11. Wire-up scripts (informational, not shipped)
273For operators wiring the index into their hosting:
274git post-receive hook on the org's hosted git (Forgejo/Gitea/GitVerse-style) — push to a package repo triggers an incremental CLI reindex on the index host (the HTTP trigger was withdrawn — ##TRIGGER-HTTP; the hook therefore invokes the operator verb, over ssh when the index lives on another machine, directly when it is local):
275#!/bin/sh
# /var/git/<org>/<repo>.git/hooks/post-receive
# Runs on the git host. Same machine as the index: drop the ssh wrapper
# and call the verb directly.
while read oldrev newrev refname; do
case "$refname" in
refs/tags/v*)
ssh index-host 'vibe-index reindex /var/lib/vibespecs-index \
--incremental --from-clones /var/lib/vibespecs-mirror' \
>>/var/log/vibe-index-hook.log 2>&1 \
|| echo "vibe-index reindex failed (non-fatal)"
;;
esac
done
276cron line:
277*/5 * * * * vibe-index reindex /home/owner/vibespecs-index --incremental --from-clones /var/lib/vibespecs-mirror >>/var/log/vibe-index.log 2>&1
278Neither is shipped as a binary — operators integrate at their own host, and the hook shape varies enough across hosting platforms that one-size-fits-all isn't worth shipping.
279Of the two, only the cron line is in crates/vibe-index/docs/operator-handbook.md; the post-receive hook is documented here and nowhere else. Worth keeping stated: the hook's earlier form posted to POST /v1/admin/reindex — a route that never shipped and was withdrawn 2026-08-20 (##TRIGGER-HTTP) — so the artefact that did not reach the handbook was exactly the one that would not have worked from it. The recipe above now invokes the CLI verb; copying it into the handbook becomes legitimate the day an operator asks for it.
12. Version history
- 2802026-05-06 — draft 1. Initial proposal. Open for review.
- 2026-05-22 — reconciled with the implementation, then folded into the workspace. A state review found PROP-005 already implemented (slices 1–10 + M2.10
vibe search) but rotted; the de-rot realigned the scanner with the currentvibe.tomlschema and corrected this document (§2.6boot_snippet, thevibe.tomlfilename, §2.10 rate-limiter status). The fold then movedvibe-indexfrom its ownservices/workspace intocrates/vibe-index/and switched it to parse throughvibe-core::Manifest— §3.2, §6, and §9 item 11 are revised for the reversed standalone-workspace decision. - 2026-05-22 — group-native (PROP-008 Phase 7). The index entry gained the mandatory
groupfield and the optionalworkspace_origin(§2.6); theby-name/layer was re-keyed fromby-name/<kind>/<name>.jsonto the candidate-set fileby-name/<name>.json(§2.4) — one GET per registry now yields every group sharing a bare name;primary.jsonl/by-cap/by-purlsort on the(group, name, version)identity; the HTTP/v1/packages/{group}/{name}routes, thevibe-index get/removeCLI, and thenaming = "fqdn"default followed. Thevibe-registryindex client and thevibe-publishpost-publish hook were realigned to the new shape. PROP-008 §2.8's index extension is shipped.