Keep the Why: Board

Context search CLI: measure and ship the best way for agents to search a large context/

tsk-bzc9wafnnpba review KTW Coder 001 (ktw-coder-001)

Goal (Oliver, 82639/82730): a CLI tool that users install, and that is all. It makes a large context/ fast and token-efficient for an agent to search. The aim is to ship the best tool, decided by measurement. There is no fixed budget: work sensibly and get the most out of it.

Starting point:

• Maity's context/ has 28 topic files, about 730 KB and 1000+ entries, so roughly 26 KB per topic. Index-then-topic finds the file but not the entry.
• The FAQ names exactly this limit ("the index would be the thing to fix"). It also says the skill ships no search tool and does no similarity search. Shipping changes that, so it needs a context entry that supersedes the FAQ's position, plus an FAQ update.

Phase 1, data and harness (`experiments/context-search/`):

• Datasets: Maity's context/ at ×1, ×5 and ×10. The copies get new UUIDs and topic names, and their See / Superseded-by references are rewritten to match (Oliver: yes). The graph stays valid and no hits are duplicated.
• Questions: about 30 real questions from Maity's history ("why was X rejected", "what supersedes Y", "which constraint protects Z", thought chains), each with its expected entry ids.
• Metrics per variant and size:
• correct entry found (recall@k / exact);
• tokens read into the session;
• tool calls;
• wall time;
• cache build time and size.
• Baseline: today's way, index.md and then the topic.
• Agents: Claude Code first. Then at least one other harness (Codex CLI or opencode) for the winners. Use TMPDIR=/var/tmp.

Phase 2, candidates (each behind the same cache):

1. An entry-level index file (heading, id, type, status, a one-line gist, file and line range).
2. A query CLI with filters (type, status, topic, text, id) and graph queries over See / Superseded / thoughts, returning file plus line ranges. It reuses the dashboard's state model (state.json) instead of a second parser.
3. Ranked full text (BM25).
4. Local embeddings, only if they install with the tool and need no service or API key. Compare cost and size against the gain.

• Cache: keyed on git rev-parse HEAD:context plus a dirty check of context/. Fallback without git: mtime, or 10 minutes.

Phase 3, decision and ship: a results write-up with the numbers, and Oliver decides. Then the package placement (proposed in the design note), the skill wiring (how the agent learns to use it), docs and FAQ, and the superseding context entry.

Process: a design note on the card first, covering packaging, CLI surface and the harness. Each step is a diff before push and goes through the three reviews.

Notes

1. KTW Coder 001 (ktw-coder-001) · 2026-11-06

   Design: context search (1/2): packaging, CLI, cache

   Placement, my recommendation: inside `keep-the-why-lint`, as a second command, `ktw-context`.

   • The skill already installs and runs the linter (local-lint), so the tool needs no further install step. That is the "install it and that's it" the card asks for.
   • The entry parser already lives there (ktw_lint.entries.parse_topic_text), and the dashboard's state.py builds on it. A second parser is not needed. The records ktw-context uses are the state model's entries (id, type, status, evidence, file, line, end_line, see, superseded_by, body). They are built by the same function, which I move from ktw_dashboard/state.py into the lint package. The dashboard then imports it, and state.json stays byte-identical.
   • Index, filters, graph and BM25 are stdlib only. Embeddings only as an extra (keep-the-why-lint[embed]), and only if the measurement shows a gain. The core install stays pure Python.
   • Rejected:
   • the dashboard package, because the skill does not install it, and the dashboard is a viewer;
   • a new package, because it is one more install step and one more version counter, and the linter's release order already covers schema changes.

   CLI (draft). By default one line per hit (id · type/status · title · file:L1-L2 · gist), which keeps the tokens down. Every command also takes --json.

   | command | does |
   |---|---|
   | ktw-context index [--topic F] | the entry-level index: every entry in one line |
   | ktw-context find [TEXT] [--type T] [--status S] [--evidence E] [--topic F] [-n 10] [--rank bm25\|embed] | filters, plus ranked full text |
   | ktw-context show ID… | the entry text (or only file:range with --where) |
   | ktw-context graph ID [--up\|--down] [--depth N] | See / Superseded by in both directions, and thought chains |
   | ktw-context cache [--status\|--rebuild] | the cache |

   The project is found by .keep-the-why walking upward, as everywhere else.

   Cache.

   • Key: git rev-parse HEAD:context plus git status --porcelain -- context/; a dirty context/ adds the hash of the changed files.
   • Without git: mtime of context/**, otherwise a 10-minute TTL.
   • Location: ~/.cache/keep-the-why/<sha256(abs path)>/, outside the repo.
   • Contents: entries.json and the BM25 postings, plus embeddings if used.
   • Build time and size are measured (Phase 1).

2. KTW Coder 001 (ktw-coder-001) · 2026-11-06

   Design: context search (2/2): data, questions, harness

   Dataset (`experiments/context-search/make_dataset.py`):

   • Maity's context/ is pinned to a commit and read-only; the source is the public repo.
   • ×1, ×5, ×10. Copy k gets:
   • topic file <name>-k<k>.md and title … (k);
   • UUIDs via uuid5(ns, old_id + k), so they are reproducible;
   • See/Superseded by (UUID and file#anchor locators) rewritten into the same copy.
   • index.md is regenerated.
   • ktw-lint --strict must pass on every set: no dangling reference, the graph is valid.

   Open point, for Oliver or you: "no hits duplicated". A text-identical copy gives N equally good answers to every question. I propose two options:

   • (a) Each copy applies a reproducible vocabulary map to the ~150 most frequent domain terms (titles, glossary), so a copy reads naturally but does not answer the original question. The answer is only the copy-0 id.
   • (b) Copies stay text-identical, and any copy's equivalent counts as a hit. That way only size is measured, not discrimination.

   I recommend (a). It is the realistic case: many similar entries, one correct one.

   Questions (`questions.json`, ~30):

   • Source: Maity's history (PR titles, issue texts, review findings), not the entry text, so the wording does not leak from the answer.
   • Expected ids, two ways:
   • graph and filter questions ("what supersedes X", "which constraints …"): computed mechanically from the state graph;
   • semantic questions ("why was X rejected"): the entry the source PR added or changed, from git log -p context/, checked by hand.
   • Each question records its source and derivation.
   • Mix: why-rejected (≥8), supersedes (≥5), constraint-protects (≥5), thought chain (≥4), open/needs-review (≥3), fact by id/title (≥3).

   Harness (`harness.py`):

   • 1. Offline (no LLM, all variants × sizes, deterministic):
   • recall@1/@5/@10 and MRR of the CLI ranking;
   • result tokens (tiktoken-compatible count);
   • cache build time and size.
   • This is cheap and narrows the field.
   • 2. Agent:
   • The Claude Code driver from tools/evals/ktw_evals/drivers/claude.py (stream-json), in a fresh workdir per run: dataset plus a variant note in CLAUDE.md/AGENTS.md (baseline: today's skill wording, index then topic).
   • Prompt: "Which entry answers: …? Reply with its Id."
   • Metrics:

   - exact@1 (the answered id = expected; for multi-answer questions, recall);
   - input + cache-read + cache-creation + output tokens;
   - tool calls (tool_use events);
   - turns;
   - wall time.

   • Plan:

   - baseline + the 2 offline winners × (×1, ×10) × 30 questions × 1 run;
   - then the finalists × 3 runs at all three sizes;
   - then Codex CLI or opencode for the winner and the baseline.

   • TMPDIR=/var/tmp, my own PLAYWRIGHT_BROWSERS_PATH, one heavy run at a time.
   • Results go to experiments/context-search/results/<date>/ with summary.md (median and IQR per cell). Raw transcripts stay local and are not committed.

   Embeddings (candidate 4): small local models that install via pip with no service. First choice is model2vec (static embeddings, numpy only, about 30 MB); alternatively fastembed (ONNX, about 100 MB). Size and build time are compared against recall gain and token savings.

   Order of PRs:

   1. Generator + questions + offline harness + baseline (results).
   2. CLI prototype in experiments/, the candidates measured.
   3. Agent runs and write-up for Oliver's decision.
   4. Ship: lint package, skill wiring, docs/FAQ, and the superseding context entry.

3. KTW Coder 001 (ktw-coder-001) · 2026-11-06

   Design update after 82754 / 82767 (replaces the placement in 1/2)

   Package: `ktw` on PyPI, command `ktw`.

   • It depends on keep-the-why-lint. The entry builder moves from ktw_dashboard/state.py into lint, and the dashboard and ktw import it from there; state.json stays byte-identical.
   • Embeddings only as ktw[embed], if the measurement shows it is worth it.
   • Release order: lint first, then ktw.
   • The skill's install routes (pip, pipx, uv) get ktw instead of keep-the-why-lint, and the linter comes along as a dependency. I check every route in Phase 4. With pipx this also needs ktw-lint in the venv; pipx install ktw exposes only ktw's own entry points, which is fine since ktw lint replaces the separate command.

   CLI:

   | command | does |
   |---|---|
   | ktw lint [args] | the linter as a library call, with the same options and exit codes as ktw-lint |
   | ktw index [--topic F] | the entry-level index |
   | ktw find [TEXT] [--type/--status/--evidence/--topic] [-n] [--rank bm25\|embed] | filters and ranked full text |
   | ktw show ID… [--where] | entry text, or file:range |
   | ktw graph ID [--up\|--down] [--depth N] | See and Superseded in both directions, and thoughts |
   | ktw cache [--status\|--rebuild] | the cache |

   keep-the-why-lint and ktw-lint stay unchanged, and so does the Action.

   Dataset: (a), the vocabulary map, is the primary dataset. (b), text-identical, runs only offline as a control, with no agent runs on it.

   Agent metrics, in addition: the session's total context tokens at the answer.