m3tam3re/agent-prod
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first

Browse Source
This commit is contained in:
m3tam3re committed 2026-10-02 09:25:13 +02:00
commit ca65491831
119 files changed
+9184
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

external/okf/agents/bundle-weaver.md
Vendored
+187
View File
@@ -0,0 +1,187 @@
---
name: bundle-weaver
description: >-
Fold event analyses into an OKF knowledge bundle in deterministic sequence. This is the
only agent that writes `.okf/`, executing the reduce phase of backfill replay to evolve
concepts over time, enforce anti-degeneration rules, and guarantee no events are lost.
model: sonnet
effort: high
tools: Read, Write, Edit, Bash, Grep, Glob
color: green
---
You are the semantic weaver for OKF bundle reconstruction. Your task is to take the analyses
produced by `okf:event-analyzer` (one per event, in `analyses/` directory) and fold them into
the `.okf/` bundle in deterministic order. You are the only agent that writes to `.okf/`; your
decisions on concept naming, grouping, and log entries become the bundle's permanent record.
## Input you receive
- A `.okf/` bundle directory (may be empty or partially populated from a prior resume).
- An `analyses/` directory with one `.md` file per event, timestamped in frontmatter. File names
are event ids (content hashes) and carry no order.
- A cursor state file (`.okf/.backfill-state.json`) indicating which events have already been
folded (resume case).
- A list of "live" event ids (those without a `skip` field) to process in order.
- Each analysis carries a `truncated` flag in its frontmatter: `true` means the analyzer saw a
capped diff. Treat that rationale as partial evidence (lean even harder toward update over
create) and count it for your reply.
## Your responsibilities
### 1. Process events in order
Read analyses in chronological order: follow the live-event-id list you were given (it carries
the event-stream order), or sort by each analysis's frontmatter `timestamp`. Do NOT sort by
filename: event ids are content hashes and their lexical order is unrelated to time.
For each event:
1. Load the analysis file.
2. Extract candidate concept names and rationale.
3. Decide: update existing concepts, or create new ones (prefer update).
4. For each touched concept: append source reference (the event id, with `:` sanitized to `-`
in the filename). Every live event lands in at least one concept's `sources`: that, not
`log.md`, is what the coverage check relies on.
5. If the event created, deprecated or superseded a concept, add a `log.md` bullet under the
event's date (§4). A routine update gets no bullet: its "why" goes into the concept body.
6. Update cursor: save `event_id` and increment `done` count.
7. Repeat.
### 2. Concept lifecycle rules
- **Name by domain entity, not by change.** Existing concept names are canonical; candidate names
from analyses are suggestions. If an analysis proposes `cache-invalidation` and `cache-invalidation.md`
already exists, update it. If it proposes `invalidation-strategy` and `cache-invalidation` exists,
check if the analysis belongs in the existing concept (usually yes); only create `invalidation-strategy`
if its scope is truly distinct.
- **1–3 concepts per event is typical.** If an analysis touches many subsystems, group them into
higher-level concepts rather than creating one per subsystem.
- **Update > Create.** The bundle evolves, not explodes. Each concept's body grows with sources and
discoveries; the frontmatter's `sources` section records all events that touched it.
- **A reversal is not growth.** Events arrive in chronological order, so a later one can overturn
what an earlier one established: a phase dropped, a limit changed, an option moved in or out of
scope, a target downgraded. Growth appends; a reversal must replace, or the bundle ends up
asserting both states in the present tense. Before writing, `grep` the whole bundle (every
directory, not the one you are about to write in) for the candidate's distinctive tokens and
reread every hit. Then:
- The analysis contradicts the concept you are updating → rewrite the body to what holds today,
and move the previous position to a dated line under a `## History` section ("2026-08-24: was
X, replaced by Y").
- The new state already lives in a *different* concept → the outdated one gets
`status: deprecated` and a link to its successor, keeping its body and all its `sources` (so
the coverage check stays green). Note both sides in the log bullet.
- Contradiction you cannot resolve from the evidence → keep the newer state and mark the older
line `[UNCLEAR]` rather than leaving two present-tense claims standing.
Count each resolution for `superseded` in your reply.
### 3. Frontmatter contract for concepts
Each `.okf/*/*.md` file must have YAML frontmatter:
```yaml
type: <inferred from analysis; one of: skill, plugin, tool, decision, pattern, architecture>
title: <Human-readable title>
description: <One-liner for the concept's essence>
status: draft # or `deprecated` once superseded (§2); never `stable`
tags: [tag1, tag2]
generated:
by: okf-backfill/0.9.5
at: <ISO 8601 timestamp now>
sources:
- id: git-abc1234 # Sanitized event id (colons → dashes)
resource: git:abc1234 # Original id for reverse lookup
last_modified: <the event's timestamp, ISO 8601 with offset, as in events.jsonl>
- id: session-file-42
resource: session:file:42
last_modified: <the event's timestamp, ISO 8601 with offset, as in events.jsonl>
```
**`status` is always written, and never `stable`.** SPEC §5.4 reads an absent `status` as
`stable`, "ready for consumption", and defines `draft` as "not yet reviewed; possibly
incomplete" — the literal description of a machine-replayed history. Every concept you write is
`draft` until a human reviews it; the only other value you may write is `deprecated`, per §2.
**For session events, the `resource` and `id` differ:** `session:<file>:<lineno>` is the resource;
`session-<file>-<lineno>` is the sanitized id (all `:` → `-`).
### 4. Log entry format
`log.md` records lifecycle events only, not every update: a concept **created**, a concept
**deprecated**, one concept **superseding** another. The date heading is the event's date, not
today's. Format:
```
## 2026-09-01
- **Creation**: [Presales pipeline](/concepts/sales/presales-pipeline.md), the sales workflow
formalized to clarify handoff points (from session: "Discuss..."). Sources: `git-abc1234`
- **Supersession**: [Vendor sync](/concepts/sales/vendor-sync.md) supersedes
[Manual vendor import](/concepts/sales/manual-vendor-import.md), now `deprecated`. Sources: `git-def5678`
```
Rules:
- Link to the concept file with Markdown syntax.
- Explain the "why" from the analysis, not just what changed.
- Never a bare restatement of the commit subject (that's the analysis's job; here you explain intent).
- A routine update, growth or a reversal inside one concept gets no bullet: the body carries
today's state, `## History` carries what it replaced, `sources` carries the events.
### 5. Anti-degeneration enforcement
Before finalizing each concept, run these checks:
- **No change-derived names:** filenames must not match `merge-pull-request-*`, `feat:-*`, `fix:-*`, etc.
- **Kebab-case only:** `[a-z0-9-]` in filenames (no spaces, underscores, capitals, or special chars).
- **Concepts must be entities, not actions.** A concept represents "the thing", not "the change to the thing".
If an analysis proposes a violating name, keep it out of the bundle and note the conflict in the log
(as a comment, or a separate line marked `[CONFLICT]` if the rule is unclear).
### 6. Cursor and resume
After each event:
```json
{"last_id": "git-abc1234", "done": 42}
```
If the job crashes or resumes, start from `last_id` + 1. The cursor is your audit trail: it records
how far you got and enables recovery without re-analyzing or re-writing.
### 7. Directory structure
Create a hierarchy based on the domain you infer from concepts:
- `skills/` for skills (backfill, validate, etc.)
- `decisions/` for architectural choices
- `components/` for technical components
- `integrations/` or `services/` for external integrations
- `tools/` for utilities
If unsure, default to `concepts/`.
## Output signature
When done with all live events (or a batch, if resuming):
- `.okf/<dir>/<concept>.md` files created/updated with sources and frontmatter.
- `.okf/log.md` updated with dated lifecycle bullets (creations, deprecations, supersessions).
- `.okf/.backfill-state.json` cursor updated.
The bundle is production-ready after the orchestrator runs the finalize step (validator + coverage check).
## What you return
Your reply to the orchestrator is one line of counts and nothing else:
```
folded=<n> created=<n> updated=<n> bullets=<n> conflicts=<n> superseded=<n> truncated_inputs=<n> last_id=<event-id>
```
The bundle is on disk; never paste concept bodies, log sections, or analyses into the reply.
## Hard rules
- **Only you touch `.okf/`.** If you see modifications from earlier runs, they are canonical;
don't replay changes to concepts that already exist (compare `last_modified` in sources and
skip if you're re-analyzing the same event).
- **Atomic per event:** update multiple concepts, log once, save cursor once per event.
- **No external reasoning:** the bundle is built on the analyses you receive. Don't invent
concepts or connections not supported by the event record.
external/okf/agents/event-analyzer.md
Vendored
+144
View File
@@ -0,0 +1,144 @@
---
name: event-analyzer
description: >-
Analyze a single repository history event (git commit or session turn) to extract domain
concepts and semantic content. Use in parallel during the map phase of OKF backfill replay
to materialize decision rationale from raw commit diffs and session outcomes.
# Bulk-read worker: it moves evidence into structure, it does not judge the bundle.
# Retier by forking this file; the backfill skill resolves the agent by name.
# haiku since benchmark/map-tier/RESULTS.md: parity with sonnet on commits at under half the cost,
# on the condition that the two explicit instructions below (flag source, evidence-only claims) stay.
model: haiku
effort: medium
tools: Bash, Read, Write
color: blue
---
You are a semantic analyzer for repository history events. You examine one git commit or
Claude session turn (or a small batch of them) and extract the domain concepts it touches, the
rationale behind the change, and candidate names for OKF knowledge concepts. You are a bulk
reader: you move evidence into structure. Judgment about the bundle (naming, merging, what to
keep) belongs to the weaver, not to you.
## Input you receive
- One or more event ids, and `events`: the path of `events.jsonl`. Fetch each event yourself,
one call per id (the orchestrator never reads event content):
```bash
jq -c --arg id '<id>' 'select(.id==$id)' <events>
```
Each event is a JSON object with:
- `id`: unique event identifier (e.g., `git:abc1234` or `session:file.jsonl:42`)
- `source`: either `git` or `session`
- git events: `sha`, `subject`, `body`, `author`, `files` (list of `{path, add, del}`)
- session events: `user` (question/prompt), `outcome` (assistant wrap-up), `title`
- `ts`: ISO 8601 timestamp
- `repo`: the repository root path.
- `out`: the directory where you write analyses.
- `emitter`: the path of the diff emitter script (`okf_backfill_events.py`), plus any
`--skip-globs` the orchestrator wants applied.
## How to read the evidence
- **Git commits**: the subject is the claim, the diff is the evidence. Read the diff **only**
through the emitter:
```bash
uv run <emitter> <repo> --show <sha> [--skip-globs ...]
```
The emitter prints the complete stat (every file, with line counts), then the patches capped
per file and cut only at hunk or file boundaries, with a bracketed marker at every cut, and a
fixed last line: `[diff: shown=X total=Y files_shown=A files_total=B truncated=true|false]`.
- Never run a raw `git show <sha>`: the harness would cut it blindly, mid-hunk.
- If `truncated=true` and the stat points at one file whose patch you need for the rationale,
you may make **one** follow-up call: `uv run <emitter> <repo> --show <sha> --only <path>`.
That is the only permitted escalation; then write what you have.
- **Session turns**: the `user` text and the `outcome` text are the whole semantic unit. The
jq fetch is the only tool call they need; do not go looking for more.
Extract, per event:
1. **Claim**: what the event asserts or achieves.
2. **Entities**: which domain concepts, subsystems, files, or features it touches. Name only
paths and entities that appear in this event's own evidence.
3. **Rationale**: why, from the commit body, session context, or diff evidence.
4. **Candidate concepts**: kebab-case concept names (`presales-pipeline`, `vendor-sync`,
`cache-invalidation`; never `feat:-add-feature` or `merge-pull-request-...`). Suggestions
only; the weaver decides.
## Anti-degeneration rules
- **Concept names describe the domain entity, not the change type.** Forbidden:
- Names derived from a git subject (`feat:`, `fix:`, `chore:`, `merge-pull-request-...`)
- Non-kebab-case (allowed: `[a-z0-9-]` only; no spaces, underscores, capitals)
- Generic placeholders (`changes`, `update`, `task`)
- **Every candidate name justifies its existence.** Three subsystems touched, three candidates,
each with a phrase saying which subsystem it covers.
## Output format
One file per event: `<out>/<event-id>.md`, where `event-id` is the id with every `:` replaced
by `-` (`git-abc1234.md`, `session-file.jsonl-42.md`). Exactly this structure, nothing outside
it (no greeting, no preamble, no commentary, no restating the diff):
```markdown
---
event_id: <original id with colons>
source: git|session
timestamp: <ISO 8601>
truncated: true|false
---
# Claim
- <one sentence: the essential change or outcome>
# Entities
- <path or entity>: <its role in this event>
# Rationale
- <why, in one sentence>
- <a second bullet only if the evidence supports a distinct reason>
# Candidate concepts
- `concept-name`: <domain it represents>; <one phrase on why it exists>
```
- `truncated` copies the last line of the **first** `--show` call for the commit: `true` if it
said `truncated=true`; `false` for complete diffs and for session events. A `--only`
follow-up never changes it: its own summary line describes one file, not the commit.
- Bullets only, one line each, leading with a name or a path. Rationale is at most two bullets.
- The claim states what the evidence shows happened, nothing more. A session outcome that only
shows intent or a first step is reported as intent or a first step; never extend it to what
probably happened next. When the evidence does not settle the why, write `[UNCLEAR]` in the
bullet instead of a plausible guess; the weaver routes it.
- When the prompt carries several events, write one file per event, each from its own evidence
only. Never let one event's content leak into another's analysis.
## What you return
Your reply to the orchestrator is **one line per event** and nothing else:
```
<out>/<event-id>.md candidates=<n> truncated=<true|false>
```
The analysis is on disk; the reply carries counts. If an event could not be analyzed, the line
is `<event-id> FAILED: <one-phrase reason>`.
## Hard constraints
- **Never modify `.okf/`**: extraction only. The weaver is the only actor that writes the bundle.
- **No external knowledge**: describe what the evidence shows, never what you know from elsewhere.
- **No raw diffs**: the emitter and its `--only` follow-up are the only ways to read a commit.
## Notes on reading a diff
- `-` lines were removed, `+` lines replaced them. A hunk the emitter cut is marked; do not
guess what the missing part did.
- Filenames tell you the scope; hunks tell you the intent.
external/okf/skills/backfill/SKILL.md
Vendored
+338
View File
@@ -0,0 +1,338 @@
---
name: backfill
description: >-
Reconstruct an OKF bundle by event-sourcing a repository's history (git log
and Claude session transcripts). Use when creating an `.okf/` bundle for an
existing repository that predates this skill, or when resuming an interrupted
backfill session. Triggers on: "reconstruct the OKF bundle", "backfill the
knowledge bundle", "event-source the history".
user-invocable: true
argument-hint: "[repo-dir] [--no-sessions] [--sessions-dir DIR]"
allowed-tools: Bash Read Write Edit
---
# Reconstruct an OKF bundle from history
This skill replays a repository's decision-making history (git commits and
Claude session transcripts) to rebuild its OKF knowledge bundle as if the
[okf](/okf) skill's Stop hook had been active from the start.
The extraction layer is **deterministic** (same repo → byte-identical events);
the replay layer is an LLM loop, **replayable and auditable but not
byte-identical** (timestamps, summaries change per run). See the event schema
(§1) and skip rules (§3).
## 1. Event schema
Git commits and session turns become events (JSONL, one per line, sorted by
timestamp then source):
```json
{"id":"git:<sha>","source":"git","ts":"<ISO8601 Z>","sha":"...","author":"...","subject":"...","body":"...","files":[{"path":"...","add":N,"del":N}]}
{"id":"session:<file>:<lineno>","source":"session","ts":"<ISO8601 Z>","user":"...","outcome":"...","title":"...","branch":"...","skip":"..."}
```
- **Git events** come from `git log --first-parent --numstat`.
- **Session events** pair each user message with the **last** assistant text
block of that turn (the wrap-up), extracted from
`~/.claude/projects/<repo-slug>/*.jsonl` and worktree subdirs.
- **Timestamps** are normalized to UTC ISO 8601 strings ending in `Z`.
- **Skip field** (optional, added by extraction): marks low-signal events —
see §3, never overridden by replay.
## 2. Protocol: Preflight → Extract → Bootstrap → Replay (Map+Reduce) → Finalize
### Preflight: check if bundle can be rebuilt
```bash
if [ -d <repo>/.okf ] && ! [ -f <repo>/.okf/.backfill-state.json ]; then
echo "ERROR: .okf/ exists but is incomplete. Delete it to rebuild from scratch, or pass --resume to continue from the last checkpoint."
exit 1
fi
```
If `.backfill-state.json` exists, resume from the cursor; otherwise, fresh bootstrap.
### Extract: generate event stream
```bash
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_backfill_events.py" <repo-dir> \
--out events.jsonl \
[--no-sessions] [--sessions-dir ~/.claude/projects] \
[--max-text 2000] [--skip-globs "vendor/**"]
```
Write `events.jsonl` to the scratchpad (never committed). Report:
- Total events extracted, per-source counts, and per-rule skip counts.
### Bootstrap: initialize bundle (fresh only)
Create `.okf/` with:
```bash
mkdir -p <repo>/.okf
echo 'okf_version: "0.2"' > <repo>/.okf/index.md
echo '# Update Log' > <repo>/.okf/log.md
```
Cursor (`.okf/.backfill-state.json`):
```json
{"last_id": null, "done": 0}
```
### Replay: two-phase map/reduce protocol
The replay is now structured in two phases to enforce semantic depth and anti-degeneration rules:
#### Phase 1: Map (parallel analysis, per-event)
Launch `okf:event-analyzer` agents for the live events (those without `skip` field), in waves of
4 to 10 (see the cost note in §5), via Claude Code's Workflow `agentType` parameter. Each analyzer
receives its event ids, the `events.jsonl` path, the repo path, the output directory and the
emitter path, and:
- Fetches its own event JSON with `jq` (the orchestrator dispatches ids, never content)
- Reads git evidence only through the capped diff emitter (§6); session events need no other call
- Writes `analyses/<event-id>.md` (event id with `:` sanitized to `-`) to scratchpad, with a
`truncated` flag copied from the emitter's last line
- Replies with one line of counts per event; the analysis never travels back in the reply
- Never touches `.okf/`
- Is resumable: skip events already in `analyses/`
**Dispatch rule** (deterministic, from `events.jsonl`, no content read): a live event is
*small* when it is a session turn or a commit whose numstat totals at most 60 changed lines.
Small events are grouped chronologically, eight per analyzer call; large commits go one per
call. Measured on this repository (`benchmark/map-tier/RESULTS.md`): 57 calls instead of 147,
a third off the map phase at the same tier, no loss on any per-event metric and no
cross-event contamination.
```bash
jq -c 'select(.skip==null) | {id, small: (.source=="session" or (([.files[]?|.add+.del]|add)//0) <= 60)}' events.jsonl
```
**Host-agnostic fallback** (if no Workflow support): spawn generic subagents with the same
system prompt as `agents/event-analyzer.md`, one per event, collecting analyses to scratchpad.
#### Phase 2: Reduce (sequential folding, one agent)
Run a single `okf:bundle-weaver` agent that:
- Reads all analyses in chronological order
- Folds them into `.okf/`, updating or creating concepts
- Enforces anti-degeneration rules (§4)
- Adds a dated `log.md` bullet for each lifecycle event only (a concept created, deprecated or
superseded); the "why" of a routine update goes into the concept body
- Manages cursor (`.okf/.backfill-state.json`) for resume capability
- Is the only actor that writes `.okf/`
- Replies with one line of counts (folded, created, updated, bullets, conflicts, superseded,
truncated inputs); the bundle never travels back in the reply
**Resume behavior:** both phases support resumption. Phase 1 skips already-analyzed event ids;
Phase 2 restarts from `last_id` in the cursor.
### Domain priming (advisory)
Before starting the map phase, read the repository's README and directory structure to sketch
a candidate taxonomy of concepts (skills, integrations, decisions, etc.). This priming is
*advisory only* — it helps the analyzer emit better candidate names. The *guarantee* that
no events are lost comes from the deterministic coverage check in Finalize (§5), not from
priming; analyses without candidate names still flow to the weaver.
### Context hygiene (orchestrator)
The orchestrator handles ids and counts, never content. Events, analyses and diffs are read by
the agents; the orchestrator reads:
```bash
wc -l events.jsonl # total events
jq -r 'select(.skip==null) | .id' events.jsonl # live ids to dispatch
ls analyses | wc -l # analyses written
```
Never `cat events.jsonl` or open an analysis from the orchestrator: whatever enters its context
is billed at the frontier rate on every following turn, and the routing exists to prevent that.
### Finalize: validate and clean up
1. **Write `index.md` per directory by hand** (do NOT run `okf_init.py` — it
would scaffold placeholder files): one `# Section` per directory, one
bullet per concept, `* [Title](file.md) - description` taken from each
concept's frontmatter (SPEC §8). The root `index.md` keeps its
`okf_version: "0.2"` frontmatter and links every subdirectory.
2. **Add final log entry** (today's date, appended to existing dated section if present):
```
## 2026-09-01
- **Backfill**: reconstructed from N events by okf-backfill/0.9.5
```
3. **Delete cursor:**
```bash
rm -f .okf/.backfill-state.json
```
4. **Run self-checks for anti-degeneration rules:**
```bash
# No concept filenames derived from change type
! find .okf -name "*.md" | grep -E "merge-pull-request|(^|/)(feat|fix|chore|docs)[:-]"
# No non-kebab-case characters in filenames
! find .okf -name "*.md" | grep -E "[^a-z0-9/.-]"
# No identical consecutive log bullets
awk 'p==$0 && /^- / {exit 1} {p=$0}' .okf/log.md
# Every concept declares its lifecycle (absent status reads as `stable`, SPEC §5.4)
[ -z "$(grep -rL '^status:' --include='*.md' .okf | grep -v '/index.md$\|/log.md$')" ]
```
These are lexical guards; they do not catch a bundle that asserts a superseded state. That
check is semantic and belongs to the weaver, at fold time, when it has both the analysis and
the earlier concepts on disk (§4 rule 5).
5. **Run coverage check to guarantee every event is mapped:**
```bash
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_backfill_events.py" \
--check-coverage events.jsonl .okf
```
Exit code 0 means all live events (non-skipped) appear in the bundle's `sources` or log.
Exit code 1 with a list of unmapped ids means the weaver skipped some events; fix and re-run.
6. **Validate bundle schema:**
```bash
uv run "${CLAUDE_SKILL_DIR}/../validate/scripts/okf_validate.py" .okf --strict
```
Fix every error before finishing.
7. **Report** the resulting bundle:
- Concept count per directory
- Log entry samples (first and last)
- Coverage check result (all events mapped)
- Validation result (pass/fail, warnings)
- Superseded resolutions (the weaver's `superseded` count) and deprecated concepts:
`grep -rl '^status: deprecated' --include='*.md' .okf | wc -l`
- Agents spawned per phase (analyzers, weaver invocations) and, when the host reports it,
tokens per phase
- Truncated analyses: `grep -l '^truncated: true' analyses/*.md | wc -l`, next to the
deterministic estimate of git events whose full diff exceeds the default cap:
`jq -r 'select(.skip==null and .source=="git") | [.files[]|.add+.del] | add' events.jsonl | awk '$1>300' | wc -l`
## 3. Skip rules
Events matching a rule below are marked `skip: <rule-id>` and skipped during
replay (cursor advances, no log entry):
| Rule | Condition | Rationale |
|------|-----------|-----------|
| `paths-only-generated` | Git commit touches ONLY lockfiles or generated code (vendor/, node_modules/, dist/, *.min.*, *.lock, etc.) | Noise: build byproducts, no knowledge content |
| `merge-no-files` | Git merge commit with no file changes (first-parent only) | Housekeeping |
| `session-command-noise` | Session user message is a slash command or <20 chars and has no useful outcome | Ephemeral UI/chatter |
Extend the first rule with `--skip-globs "extra/**"` for repo-specific patterns.
## 4. Anti-degeneration rules
The analyzer and weaver enforce these rules to prevent the bundle from devolving into
a mechanical listing of commits or a taxonomy-by-accident:
1. **Concept names describe domain entities, not changes.** Forbidden:
- Filenames derived from git subject: `merge-pull-request-#2-....md`, `feat:-add-docs-skill.md`, `fix:-handle-edge-case.md`
- Bare action words: `update.md`, `fix.md`, `add-feature.md`
- Non-kebab-case: spaces, underscores, capitals, special characters (allowed: `[a-z0-9-]` only)
Example: a commit with subject "feat: add presales pipeline" touches a domain concept.
The *concept* is named `presales-pipeline.md` (the entity), not `feat:-add-presales-pipeline.md`
(the change). The change history lives in frontmatter `sources` and the body's `## History`;
`log.md` holds only lifecycle events.
2. **Prefer update over create.** Every event is analyzed for which concepts it touches; if
a concept already exists and the analysis fits, update its `sources` and body. Only create
a new concept if the analysis reveals a distinct domain entity not yet captured.
3. **Log bullets are lifecycle events, and explain intent.** Only a concept created,
deprecated or superseded gets a bullet; a routine update does not. A bullet answers "why?"
from the commit body or session outcome, never just re-reads the subject:
- Bad: `- Added presales-pipeline.md feature`
- Good: `- **Creation**: [Presales pipeline](...), formalized to clarify handoff points`
4. **No identical consecutive bullets.** A run of same-concept bullets means updates leaked
into the log:
```
- Feature X update
- Feature X update
- Feature X update
```
Instead: no bullet at all; the updates live in the concept's body and `sources`.
5. **A reversal is not growth.** A replay walks the history forward, so a later event routinely
overturns an earlier one (a phase dropped, a limit changed, a target downgraded). "Prefer
update over create" is a merge rule and says nothing about this: applied alone it leaves the
bundle asserting both states in the present tense, in one concept or in two. The weaver greps
the whole bundle before writing and, on contradiction, replaces rather than appends: the body
states what holds today, the previous position drops to a dated line under `## History`, and
an outdated concept that a newer one supersedes gets `status: deprecated` plus a link to its
successor. A deprecated concept keeps its `sources`, so coverage stays green. None of the
guards in Finalize can catch this — they are lexical, and a superseded claim is well-formed.
6. **Reconstructed concepts are `draft`, never `stable`.** SPEC §5.4 reads an absent `status` as
`stable` ("ready for consumption"); a bundle nobody has reread is `draft` ("not yet reviewed;
possibly incomplete"). The weaver writes `status:` on every concept.
## 5. Implementation notes
- **Determinism**: extraction is byte-identical; replay is not (time, LLM variance).
Both are auditable: events.jsonl is deterministic, map analyses are stored and resumable,
reduce loop is human-readable and cursor-backed.
- **Privacy**: events.jsonl goes to scratchpad, never committed. Session turn text is
truncated (head+tail) before extraction.
- **Cost note**: deep replay reads one capped diff per git commit (§6) to extract
rationale. For histories >500 events, consider splitting into sub-ranges and
replaying sequentially, or use `--skip-globs` to exclude low-signal paths (e.g.,
vendored dependencies, generated code). Map phase is parallel in waves of 4 to 10: the
gate benchmark lost 260 of 299 runs to a high-concurrency mass failure and finished at
concurrency 4 (`benchmark/gate/RESULTS.md`). Reduce is sequential but much cheaper
(concepts already analyzed).
- **Future sources** (not implemented): GitHub PR/issue text, release notes, CI/deploy logs
— all optional post-MVP. **Lore protocol** (arXiv 2603.15566): on repos that adopt git
trailers (structured decision metadata), the extracted event's `body` already contains
trailers in a machine-readable format. The analyzer and weaver should use trailers as
the primary source of "why" (overriding generic inference from diff content).
- **Trust metadata**: `generated.by` is the backfill agent (`okf-backfill/0.9.5`), not
claimed as human-reviewed (`human:...`); concepts are correctly `unverified` (SPEC §5.3) and
`status: draft` (§5.4) — the two axes are independent: trust is who says it, lifecycle is
whether it has been reread. Writing `status` is not optional here: absent, §5.4 reads it as
`stable`, so a bundle nobody has reread would declare itself ready for consumption.
Map-phase analyses are working artifacts (stored for auditability during reduce); the
weaver's output is the canonical bundle.
## 6. Capped diff emitter
Analyzers never read a raw `git show`: the harness cuts long tool output blindly (mid-hunk,
character-based, host-dependent) and a cheap worker may not notice the cut. The extractor reads
the whole diff instead and emits a deterministic sample:
```bash
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_backfill_events.py" <repo-dir> --show <sha> \
[--only <path>] [--max-diff-lines 300] [--per-file-lines 120] [--max-line-chars 400] \
[--skip-globs "vendor/**"]
```
- Diff against the first parent, so merge commits agree with the extracted numstat.
- The stat is always complete: entities survive any cap.
- Patches are capped per file (breadth over depth) and cut at hunk or file boundaries, with a
bracketed marker at every cut; lines longer than `--max-line-chars` are shortened with a
marker (a generated one-line file is "1 line" but can be hundreds of KB).
- Patches of generated files (lockfiles, `vendor/**`, `--skip-globs`) are omitted even inside
mixed commits; their stat line stays.
- The last line is fixed-form: `[diff: shown=X total=Y files_shown=A files_total=B truncated=true|false]`.
The analyzer copies `truncated` into its frontmatter; finalize reports the count.
- `--only <path>` is the one permitted follow-up when a truncated diff hides the rationale:
same cap logic, one file.
Defaults keep one call under the harness limits with margin; tune per repo with the flags.
The analyzer's tier is configuration, not infrastructure: `agents/event-analyzer.md` carries
`model` and `effort` in its frontmatter. Fork the file to retier the map phase; the skill
resolves the agent by name and nothing else changes. The default is `haiku` since the
map-tier benchmark (`benchmark/map-tier/RESULTS.md`): parity with `sonnet` on commits at
under half the map cost, on the condition that the two explicit instructions in the agent
file stay (which summary line the flag copies; claims report intent as intent).
external/okf/skills/backfill/scripts/okf_backfill_events.py
Vendored
+1007
View File
File diff suppressed because it is too large. Load diff
external/okf/skills/okf/SKILL.md
Vendored
+159
View File
@@ -0,0 +1,159 @@
---
name: okf
description: >-
Author, maintain, and consume Open Knowledge Format (OKF) knowledge bundles —
portable markdown + YAML frontmatter that both humans and agents read. Use when
capturing project knowledge (services, APIs, schemas, metrics, runbooks,
decisions) into an OKF bundle, when updating one after code or docs change, or
when a repository contains an `.okf/` (or other OKF) bundle that should inform
the task. Triggers on: "document this in OKF", "update the knowledge bundle",
"capture this as a concept", or any work in a repo that has an OKF bundle.
user-invocable: true
argument-hint: "[produce|maintain|consume] [path]"
allowed-tools: Read Write Edit Grep Glob Bash
---
# Open Knowledge Format (OKF) skill
OKF represents knowledge as a directory of markdown files with YAML frontmatter.
It is minimal by design: no schema registry, no runtime, no SDK. Your job is to
produce, maintain, and consume OKF bundles **conformant with the spec**, not your
memory of it.
**Always read the canonical spec before non-trivial work:**
[reference/SPEC.md](reference/SPEC.md). It is the verbatim OKF v0.2 specification
and the source of truth for every rule below.
## The one hard rule
A bundle is conformant (§11) iff: every non-reserved `.md` file has a parseable
YAML frontmatter block, and every such block has a **non-empty `type`** field.
Everything else is soft guidance. Consumers MUST tolerate missing optional
fields, unknown types, and broken links — never reject a bundle over them.
## Conventions to apply
- **One concept = one file.** The file path (minus `.md`) is the concept ID.
- **Frontmatter:** `type` is required. Add `title`, `description`, `tags` when
they aid consumption; add `resource` (a canonical URI) only for concepts bound
to a real asset — omit it for abstract concepts.
- **Body:** prefer structural markdown (headings, tables, lists, fenced code).
Conventional headings: `# Schema`, `# Examples`, `# Computation`.
- **Cross-links:** standard markdown links; prefer absolute bundle-relative
form (`/services/auth-api.md`). A link asserts a relationship; its *kind* lives
in the surrounding prose, not the link.
- **Reserved files:** `index.md` (directory listing, no frontmatter, except the
bundle-root index, which may carry `okf_version` and this plugin's
`upkeep: enforced` opt-in flag) and `log.md` (ISO-dated history, newest
first). Never use these names for concepts.
- **What goes in `log.md`:** lifecycle events only, written in the bundle-root
`log.md`: a bundle **created**, a concept **deprecated** or retired, one
concept **superseding** another, concepts **regenerated** after their source
changed, a **verification** pass. Routine edits to a concept do not get an
entry: its `generated` and git already record them. A log is append-only, so
never rewrite or drop an existing entry.
## The v0.2 families (all optional, all worth filling)
- **Trust (§5.2):** `generated: { by, at }` — who produced the current content
and when. `verified: [{ by, at }]` — who confirmed it since (a bare mapping is
one entry). Write `by` in the **actor convention** (§7): `<producer>/<version>`
for an agent, `human:<id>` for a person, `process:<id>` for an automated job.
Use `human:` whenever a person authored or signed off — consumers key trust
tiers off that prefix.
- **Lifecycle (§5.4–5.5):** `status: draft|stable|deprecated` (absent means
stable) and `stale_after`, an absolute ISO 8601 datetime (a bare date is
tolerated), not a TTL.
- **Provenance (§5.1):** `sources: [{ id, resource, title, author,
usage_count, last_modified }]` plus a `usage_window: { from, to }` sibling of
`sources` framing every `usage_count` (an entry may carry its own to override
it); a `usage_count` without a window warns. `resource` is required per entry
and may be a URL, a bundle path, or a scope descriptor. Attribute a specific
claim with a markdown footnote whose label is the source's `id`:
`…sharded daily.[^ga4-schema]` plus a `[^ga4-schema]: …` definition. The label
is the join key, it must match a `sources[].id`.
- **Attestation (§10):** a sanctioned computation is its own concept,
`type: Attested Computation`, carrying `runtime` (required), `parameters`,
`executor`, `attester`, and the computation itself under `# Computation` (or a
`computation:` path). Concepts that need the value link to it. Never inline a
number's SQL into the concept that narrates it.
**Reading a v0.1 bundle?** Two constructs were superseded (§13.1): `timestamp`
is now `generated.at`, and a body `# Citations` list is now `sources`. Read both,
write v0.2 — and when you touch a legacy concept in **maintain** mode, migrate
its frontmatter as part of the edit. The validator warns on both.
Templates to copy: [concept](templates/concept.md), [index](templates/index.md),
[log](templates/log.md).
## Default bundle location
Use `.okf/` at the repository root unless the project already uses another
location. Commit it alongside the code it describes — knowledge as code.
## Modes
### produce — create or extend a bundle
**Starting a brand-new bundle?** Use the init fast-path instead of hand-writing
the first files — it scaffolds a conformant `index.md`, `log.md`, and a
`getting-started.md` concept with full recommended frontmatter in one shot:
```bash
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_init.py" <target-dir> [--title "..."]
```
It refuses to touch a directory that already has `.md` files unless `--force`
is given. Then extend it:
1. Read [reference/SPEC.md](reference/SPEC.md).
2. Pick the source(s): **code** (derive concepts from source, READMEs,
docstrings, config), **docs/wiki** (distill pages into concepts, record the
originals in `sources`), **manual** (decisions, playbooks, metrics).
3. Choose a directory layout by domain (e.g. `services/`, `datasets/`,
`decisions/`). One concept per file.
4. Write each concept from [templates/concept.md](templates/concept.md): set a
descriptive `type`, fill recommended fields, record `generated` and the
`sources` you actually read, cross-link related concepts.
5. Add/refresh `index.md` per directory (and `okf_version: "0.2"` in the root
index). A new bundle gets a dated **Creation** entry in `log.md`. Extending
an existing one does not.
6. Validate (see below). Fix every error before finishing.
### maintain — keep a bundle in sync with reality
1. Identify which concepts the change affects (search by `resource`, path, or
topic). This bookkeeping is exactly what agents are good at — touch every
affected file in one pass.
2. Update the body and `generated.at` (with your own actor in `generated.by`);
fix or add cross-links; create new concepts for new assets; mark removed
assets `status: deprecated` and note the deprecation in `log.md` rather than
silently deleting context. Facing a whole v0.1 bundle rather than a stray
field? Do not hand-edit it — run the validator's `--migrate` once.
3. Update the relevant `index.md` files. Append a dated `log.md` entry only if
the change is a lifecycle event (see *What goes in `log.md`*).
4. Validate.
### consume — use a bundle as context
1. Read the bundle-root `index.md` first for progressive disclosure, then follow
links only into the concepts relevant to the task.
2. Weigh what you read: `status: draft`/`deprecated`, a `stale_after` already
past, or no `verified` entry all mean "check before relying on this". Treat
broken links as not-yet-written knowledge, not errors.
3. Need a number an `Attested Computation` covers? Run *its* computation with
values bound to the declared `parameters` — never write your own query.
4. If you learn something durable while working, switch to **maintain** and
write it back.
## Validation (do this before declaring done)
Never eyeball conformance — run the deterministic checker. Invoke the companion
**`validate`** skill (`/okf:validate <bundle-dir> --strict`), which ships the
checker. If that skill is not installed, run it directly:
```bash
uv run "${CLAUDE_SKILL_DIR}/../validate/scripts/okf_validate.py" <bundle-dir> --strict
```
Resolve every `ERROR` (hard §11 failures). Under `--strict` warnings also fail
the run (exit 1), so clear them too; drop `--strict` (or use `--max-warnings N`)
if you only want §11 conformance gated.
external/okf/skills/okf/reference/APACHE-2.0.txt
Vendored
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
external/okf/skills/okf/reference/SPEC.md
Vendored
+1016
View File
File diff suppressed because it is too large. Load diff
external/okf/skills/okf/scripts/okf_init.py
Vendored
+112
View File
@@ -0,0 +1,112 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.11"
# dependencies = ["pyyaml>=6"]
# ///
"""Scaffold a conformant starter Open Knowledge Format (OKF) v0.2 bundle.
Creates a root `index.md` (frontmatter carries only `okf_version`), a `log.md`
with a creation entry under today's ISO date heading, and one starter concept
(`getting-started.md`) with the full set of recommended frontmatter fields.
The scaffold is meant to be exemplary: `okf_validate.py --strict` should pass
it with zero warnings.
Run: uv run scripts/okf_init.py <target-dir> [--title "..."] [--force]
"""
from __future__ import annotations
import argparse
import sys
from datetime import datetime, timezone
from pathlib import Path
import yaml
OKF_VERSION = "0.2"
STARTER_CONCEPT = "getting-started.md"
# Actor convention (§7): an automated process, not an agent with a version.
ACTOR = "process:okf_init"
def humanize(name: str) -> str:
words = name.replace("-", " ").replace("_", " ").strip()
return words.title() if words else "Untitled"
def frontmatter(meta: dict) -> str:
return "---\n" + yaml.safe_dump(meta, sort_keys=False, allow_unicode=True) + "---\n"
def build_index(title: str) -> str:
meta = frontmatter({"okf_version": OKF_VERSION})
return (
f"{meta}\n"
f"# {title}\n\n"
f"* [Getting started]({STARTER_CONCEPT}) - starting point for this bundle.\n"
)
def build_log(today: str, title: str) -> str:
return (
"# Update Log\n\n"
f"## {today}\n"
f"* **Creation**: Scaffolded the {title} bundle with `okf_init.py` — "
f"see [getting started]({STARTER_CONCEPT}).\n"
)
def build_concept(title: str, now_iso: str) -> str:
meta = frontmatter({
"type": "Reference",
"title": f"Getting started — {title}",
"description": f"Starting point for the {title} OKF bundle.",
"tags": ["getting-started"],
"status": "stable",
"generated": {"by": ACTOR, "at": now_iso},
})
return (
f"{meta}\n"
"# Overview\n\n"
"This is the first concept in a freshly scaffolded OKF bundle. Replace "
"it with real knowledge — one concept per file, cross-linked with "
"standard markdown links — and keep `index.md` updated as you go. "
"`log.md` records lifecycle events only: a concept created, "
"deprecated, superseded, regenerated or verified.\n"
)
def main() -> int:
ap = argparse.ArgumentParser(description=f"Scaffold a starter OKF v{OKF_VERSION} bundle.")
ap.add_argument("target", type=Path, help="directory to create the bundle in")
ap.add_argument("--title", default=None, help="bundle title (default: humanized target dir name)")
ap.add_argument("--force", action="store_true", help="scaffold even if the target already has .md files")
args = ap.parse_args()
if args.target.exists() and not args.target.is_dir():
print(f"error: {args.target} exists and is not a directory", file=sys.stderr)
return 2
existing_md = list(args.target.rglob("*.md")) if args.target.is_dir() else []
if existing_md and not args.force:
print(f"refusing: {args.target} already contains .md files (pass --force to scaffold anyway)", file=sys.stderr)
for p in sorted(existing_md)[:5]:
print(f" {p.relative_to(args.target)}", file=sys.stderr)
return 1
title = args.title or humanize(args.target.resolve().name)
today = datetime.now(timezone.utc).date().isoformat()
now_iso = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
args.target.mkdir(parents=True, exist_ok=True)
(args.target / "index.md").write_text(build_index(title), encoding="utf-8")
(args.target / "log.md").write_text(build_log(today, title), encoding="utf-8")
(args.target / STARTER_CONCEPT).write_text(build_concept(title, now_iso), encoding="utf-8")
validator = Path(__file__).resolve().parents[2] / "validate" / "scripts" / "okf_validate.py"
print(f"created OKF bundle scaffold at {args.target}")
print(f"hint: validate it — uv run {validator} {args.target} --strict (or the validate skill)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
external/okf/skills/okf/templates/concept.md
Vendored
+38
View File
@@ -0,0 +1,38 @@
---
type: <Concept type, e.g. Service, BigQuery Table, Metric, Playbook, Decision>
title: <Human-readable display name>
description: <Single sentence summarizing the concept.>
resource: <Canonical URI of the underlying asset — omit for abstract concepts>
tags: [<tag>, <tag>]
status: stable # draft | stable | deprecated; absent means stable
generated: { by: <actor>, at: <ISO 8601, e.g. 2026-06-14T10:00:00Z> }
verified: { by: human:<id>, at: <ISO 8601> } # omit until someone confirms it
stale_after: <YYYY-MM-DDTHH:MM:SSZ> # omit when the content does not expire
sources: # what this was derived from; omit if nothing
- id: <short-key>
resource: <URL, bundle path, or scope descriptor>
title: <Human-readable label>
author: <actor> # optional credibility signal
last_modified: <YYYY-MM-DDTHH:MM:SSZ> # when the source itself last changed
---
<!--
Actors (§7): `<producer>/<version>` for an agent, `human:<id>` for a person,
`process:<id>` for an automated job. Use `human:` whenever a person authored
or signed off — consumers key trust tiers off that prefix.
-->
# Overview
<What this concept is and why it matters. Attribute a sourced claim with a
footnote whose label is a `sources[].id`.[^short-key]>
# Schema
<Use for assets with fields/columns; otherwise replace with relevant sections.>
| Field | Type | Description |
|-------|------|-------------|
| | | |
[^short-key]: <Human-readable label of the source>
external/okf/skills/okf/templates/index.md
Vendored
+8
View File
@@ -0,0 +1,8 @@
# <Directory / Group Heading>
* [<Title>](<relative-url>) - <short description from the concept's frontmatter>
* [<Title>](<relative-url>) - <short description>
# <Another Group>
* [<Subdirectory>](<subdir>/) - <short description of the subdirectory>
external/okf/skills/okf/templates/log.md
Vendored
+8
View File
@@ -0,0 +1,8 @@
# Update Log
## <YYYY-MM-DD>
* **Creation**: <what was created> — [<concept>](<bundle-relative-path>).
* **Deprecation**: <what was retired, and what replaces it> — [<concept>](<bundle-relative-path>).
* **Supersession**: [<new concept>](<bundle-relative-path>) supersedes [<old concept>](<bundle-relative-path>). Affirmed by <actor>.
* **Regeneration**: <which concepts, after what source change> — [<concept>](<bundle-relative-path>).
* **Verification**: <what was verified, by whom> — [<concept>](<bundle-relative-path>).
external/okf/skills/validate/SKILL.md
Vendored
+85
View File
@@ -0,0 +1,85 @@
---
name: validate
description: >-
Check that an Open Knowledge Format (OKF) bundle is conformant with the v0.2
spec (§11). Use when asked to validate, lint, or check an OKF bundle, or before
committing changes to one. Runs a deterministic Python checker — not an
eyeball pass. Also migrates a v0.1 bundle to v0.2 in place with `--migrate`.
user-invocable: true
argument-hint: "[bundle-dir] [--strict | --max-warnings N] [--migrate]"
allowed-tools: Bash
---
# Validate an OKF bundle
Run the deterministic conformance checker against the target bundle. Default to
the project's `.okf/` directory when no path is given.
```bash
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_validate.py" $ARGUMENTS
```
If `uv` is unavailable, fall back to:
```bash
python3 -m pip install --quiet pyyaml && \
python3 "${CLAUDE_SKILL_DIR}/scripts/okf_validate.py" $ARGUMENTS
```
`${CLAUDE_SKILL_DIR}` resolves whether this skill runs as part of the `okf`
plugin or is installed standalone (e.g. via `npx skills add`), so the checker is
always found alongside the skill.
Interpret the result:
- **ERROR** → a hard §11 conformance failure (no parseable frontmatter, a
missing/empty `type`, or a file the checker cannot read as UTF-8). The bundle
is non-conformant. Fix every one.
Only `index.md` and `log.md` are reserved: §11.1 makes *every* other `.md` in
the tree a concept, convention files included. An `AGENTS.md` or `README.md`
dropped next to a bundle is therefore an ERROR until it carries frontmatter with
a `type` (`Document` reads well), which is the fix; a skip-list would put the
checker out of conformance. Move the file above the bundle root if it is not
knowledge.
- **warn** → soft guidance (missing recommended field, non-ISO log date, broken
cross-link, a malformed v0.2 family, a footnote naming no source, an actor
that misses the §7 shapes, an Attested Computation missing `runtime`, a
computation path that resolves nowhere, frontmatter in a reserved file (§8,
§9), a root `index.md` carrying extra keys or declaring a different
`okf_version` (§12)). Never blocks; broken links in particular are explicitly
tolerated by the spec (§6.1). Fix when cheap.
One warning is worth more than the others: a §7 near-miss such as `Human:dana`
or `human/dana`. §5.3 keys trust tiers off the exact lowercase `human:` prefix,
so the concept silently reads as machine-confirmed when a person did review it.
Fix that one on sight.
v0.1 bundles validate too: a legacy `timestamp` or `# Citations` section is
reported as a warning naming its v0.2 replacement (`generated.at`, `sources`),
never as an error (§13.1). Under `--strict` those warnings do fail the run —
that is the migration nudge, and `--migrate` is the door.
## Migrating a v0.1 bundle
`--migrate` **rewrites the bundle in place** before validating — the one mode of
this skill that is not read-only, so say what it will touch before running it on
a bundle the user has not asked to migrate. It is textual (comments, key order
and quoting survive) and idempotent.
```bash
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_validate.py" .okf --migrate --strict
```
It hoists `timestamp` to `generated: { by: process:okf-migrate, at }`, lifts a
`# Citations` list into `sources`, and bumps `okf_version`. Two limits worth
repeating to the user: `generated.by` cannot be recovered for pre-v0.2 content
(hence the `process:` actor — the concept stays correctly `unverified` under
§5.3), and per-claim `[^id]` attribution was never encoded in v0.1, so only the
source list moves up.
## Exit codes
Non-zero if any error is present, or if warnings exceed the gate:
`--strict` allows none, `--max-warnings N` allows N, the default allows any.
Add `--json` for machine-readable output (useful in CI).
external/okf/skills/validate/scripts/okf_validate.py
Vendored
+573
View File
@@ -0,0 +1,573 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.11"
# dependencies = ["pyyaml>=6"]
# ///
"""Deterministic conformance checker for Open Knowledge Format (OKF) v0.2.
Implements the §11 conformance rules verbatim:
1. Every non-reserved `.md` file contains a parseable YAML frontmatter block.
2. Every frontmatter block contains a non-empty `type` field.
3. Reserved filenames (`index.md`, `log.md`) follow §8 / §9 when present.
Rules 1 and 2 are hard errors (a bundle that fails them is non-conformant).
Everything else the spec marks as soft guidance: reported as warnings, never
fatal unless `--strict` is given. In particular broken cross-links are NOT
errors — the spec requires consumers to tolerate them (§6.1).
v0.1 bundles still validate: the two superseded constructs (`timestamp`, a body
`# Citations` list) are read and reported as warnings pointing at their v0.2
replacements (`generated.at`, `sources`), never as errors (§13.1). `--migrate`
performs that rewrite in place, textually, so `--strict` has a door and not just
a wall.
Run: uv run scripts/okf_validate.py <bundle-dir>
[--strict | --max-warnings N] [--migrate] [--json]
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
import yaml
OKF_VERSION = "0.2"
RESERVED = {"index.md", "log.md"}
# `resource` is intentionally excluded: the spec (§4.1) states it is absent for
# concepts describing abstract ideas, so flagging it produces false noise.
RECOMMENDED = ("title", "description", "tags")
STATUS_VALUES = {"draft", "stable", "deprecated"}
ISO_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
# RFC 3339, as the spec writes `generated.at` / `verified[].at` throughout, and
# (since 0.9.6) `stale_after`, `sources[].last_modified` and `usage_window`. A
# date-only value is tolerated: it is common in the wild and loses only precision.
# PyYAML resolves an unquoted timestamp to a datetime whose str() separates with a
# space, so both spellings have to pass.
RFC3339 = re.compile(
r"^\d{4}-\d{2}-\d{2}"
r"(?:[Tt ]\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?:[Zz]|[+-]\d{2}:?\d{2})?)?$")
# §7 gives three actor shapes, and §5.1 reuses them for `sources[].author`. The
# spec's own example (`author: team:ga4-docs`) shows the `<prefix>:<id>` family is
# open, so this is deliberately not a whitelist — it catches the one failure that
# is silent and costly: a near-miss of `human:`, which §5.3 keys trust tiers off.
# `human/dana` or `Human:dana` reads as an agent and downgrades the tier with no
# other symptom.
ACTOR_SHAPE = re.compile(r"^(?:[^\s:/]+:\S+|\S+/\S+)$")
# Anything that opens with human/process in any case — `Human:dana` is the worst
# of these, since it satisfies the generic `<prefix>:<id>` shape and so looks
# entirely well-formed while §5.3 reads it as an agent. The lookahead keeps
# `humanoid_agent/v1` out: only a non-identifier boundary makes it a near-miss.
ACTOR_HUMANISH = re.compile(r"^(?:human|process)(?![A-Za-z0-9_])", re.I)
FENCE = re.compile(r"^(```|~~~)")
# markdown link target capture: [text](target) — ignores images ![...]
LINK = re.compile(r"(?<!\!)\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
CITATIONS = re.compile(r"^#{1,6}[ \t]+Citations[ \t]*$", re.M)
# a footnote *reference* in the body; the definition line `[^id]: …` matches too,
# which is harmless — both carry the same label, the join key into `sources`.
FOOTNOTE = re.compile(r"\[\^([^\]\s]+)\]")
# --migrate only. `timestamp` is a top-level key (§4.1), so anchor at column 0 —
# an indented one belongs to something else and must not be hoisted.
TIMESTAMP = re.compile(r"^timestamp:[ \t]*(.+?)[ \t]*$", re.M)
HEADING = re.compile(r"^#{1,6}[ \t]", re.M)
# same shape as LINK above, plus the link text — a citation written with a title
# attribute, `[Src](url "Title")`, must not be dropped: the section is deleted
# wholesale once anything parses, so a miss here is silent data loss.
MD_LINK = re.compile(r"(?<!\!)\[([^\]]+)\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
OKF_VERSION_LINE = re.compile(r"^(okf_version:[ \t]*)[\"']?0\.1[\"']?[ \t]*$", re.M)
# True, and it keeps the concept correctly `unverified` under §5.3: who wrote the
# content before `generated` existed is not recoverable, and inventing a `human:`
# actor would fake a review that never happened.
MIGRATE_ACTOR = "process:okf-migrate"
@dataclass
class Report:
errors: list[str] = field(default_factory=list)
warnings: list[str] = field(default_factory=list)
concepts: int = 0
indexes: int = 0
logs: int = 0
def err(self, rel: str, msg: str) -> None:
self.errors.append(f"{rel}: {msg}")
def warn(self, rel: str, msg: str) -> None:
self.warnings.append(f"{rel}: {msg}")
def split_frontmatter(text: str) -> tuple[str | None, str]:
"""Return (raw_yaml, body). raw_yaml is None when no frontmatter block."""
if not text.startswith("---"):
return None, text
# first line must be exactly '---' (allow trailing whitespace / BOM already stripped)
lines = text.splitlines(keepends=True)
if lines[0].strip() != "---":
return None, text
for i in range(1, len(lines)):
if lines[i].strip() == "---":
return "".join(lines[1:i]), "".join(lines[i + 1 :])
return None, text # unterminated block → treated as absent
def _read_text(path: Path, rel: str, report: Report) -> str | None:
"""Read a concept/index/log file, reporting (not raising) on a bad file.
A binary or non-UTF-8 `.md` file must fail as a per-file error, not crash
the whole validation run.
"""
try:
return path.read_text(encoding="utf-8").lstrip("")
except (UnicodeDecodeError, OSError) as exc:
report.err(rel, f"cannot read file: {exc}")
return None
def check_computation(meta: dict, path: Path, bundle: Path, rel: str, report: Report) -> None:
"""§10.2 — the Attested Computation contract, including its path-valued fields.
`computation`, `executor.resource` and `attester.resource` name files (§6.2).
They are exactly the pointer that rots: the concept keeps validating while the
script it names moves or disappears. A target outside the bundle is legitimate
and unverifiable from here, so only bundle-relative paths are resolved.
"""
if not str(meta.get("runtime", "")).strip():
report.warn(rel, "§10.2 an Attested Computation concept requires `runtime`")
fields = [("computation", meta.get("computation"))]
for key in ("executor", "attester"):
block = meta.get(key)
if isinstance(block, dict):
fields.append((f"{key}.resource", block.get("resource")))
for where, target in fields:
if target is None:
continue
t = str(target).strip()
if not t or re.match(r"^[a-z][a-z0-9+.-]*://", t) or t.startswith("mailto:"):
continue
candidate = (bundle / t.lstrip("/")) if t.startswith("/") else (path.parent / t)
try:
inside = candidate.resolve().is_relative_to(bundle.resolve())
except OSError:
inside = False
if inside and not candidate.exists():
report.warn(rel, f"§6.2 `{where}` points at `{t}`, which does not exist")
def check_concept(path: Path, rel: str, bundle: Path, report: Report) -> None:
report.concepts += 1
text = _read_text(path, rel, report)
if text is None:
return
raw, body = split_frontmatter(text)
if raw is None:
report.err(rel, "§11.1 no parseable YAML frontmatter block "
"(every non-reserved `.md` in a bundle is a concept: add "
"frontmatter with a `type` field, or move the file outside "
"the bundle)")
return
try:
meta = yaml.safe_load(raw)
except yaml.YAMLError as exc:
report.err(rel, f"§11.1 frontmatter is not valid YAML: {exc}".replace("\n", " "))
return
if not isinstance(meta, dict):
report.err(rel, "§11.1 frontmatter must be a YAML mapping")
return
type_val = meta.get("type")
if not (isinstance(type_val, str) and type_val.strip()):
report.err(rel, "§11.2 missing or empty required `type` field")
for key in RECOMMENDED:
if key not in meta:
report.warn(rel, f"recommended field `{key}` is absent (§4.1)")
check_trust(meta, rel, report)
check_lifecycle(meta, rel, report)
check_sources(meta, body, rel, report)
if isinstance(type_val, str) and type_val.strip() == "Attested Computation":
check_computation(meta, path, bundle, rel, report)
def check_actor(value, where: str, rel: str, report: Report) -> None:
"""§7 — the actor convention shared by `generated.by`, `verified[].by` and
`sources[].author`."""
actor = str(value).strip()
if not actor:
return
if ACTOR_HUMANISH.match(actor) and not actor.startswith(("human:", "process:")):
report.warn(rel, f"§7 `{where}` `{actor}` is a near-miss of "
f"`human:`/`process:` — §5.3 keys trust tiers off that exact "
f"lowercase prefix, so this silently reads as an agent")
elif not ACTOR_SHAPE.match(actor):
report.warn(rel, f"§7 `{where}` `{actor}` matches no actor shape "
f"(`human:<id>`, `process:<id>`, or `<producer>/<version>`)")
def check_instant(value, where: str, rel: str, report: Report) -> None:
"""§5.2 — `at` values are RFC 3339; a date-only value is tolerated."""
if value is not None and not RFC3339.match(str(value).strip()):
report.warn(rel, f"§5.2 `{where}` `{value}` is not an RFC 3339 timestamp")
def check_trust(meta: dict, rel: str, report: Report) -> None:
"""§5.2 — `generated` / `verified`, with the v0.1 `timestamp` fallback."""
gen = meta.get("generated")
if gen is None:
if "timestamp" in meta:
report.warn(rel, "legacy v0.1 `timestamp`: v0.2 records the last content "
"change as `generated: {by, at}` (§13.1)")
else:
report.warn(rel, "recommended field `generated` is absent (§5.2)")
elif not isinstance(gen, dict):
report.warn(rel, "§5.2 `generated` must be a mapping with `by` and `at`")
elif not str(gen.get("by", "")).strip():
report.warn(rel, "§5.2 `generated.by` is required within `generated`")
else:
check_actor(gen["by"], "generated.by", rel, report)
check_instant(gen.get("at"), "generated.at", rel, report)
ver = meta.get("verified")
if ver is None:
return
# A bare mapping is one verification event — consumers MUST treat it as a
# one-element list (§5.2), so normalize before checking the entries.
entries = [ver] if isinstance(ver, dict) else ver
if not isinstance(entries, list):
report.warn(rel, "§5.2 `verified` must be a `{by, at}` mapping or a list of them")
return
for i, entry in enumerate(entries):
if not (isinstance(entry, dict) and str(entry.get("by", "")).strip()):
report.warn(rel, "§5.2 every `verified` entry needs a `by` actor")
continue
check_actor(entry["by"], f"verified[{i}].by", rel, report)
check_instant(entry.get("at"), f"verified[{i}].at", rel, report)
def check_lifecycle(meta: dict, rel: str, report: Report) -> None:
"""§5.4 / §5.5 — `status` and `stale_after`. A concept is stale when
`now >= stale_after`."""
status = meta.get("status")
if status is not None and status not in STATUS_VALUES:
report.warn(rel, f"§5.4 unknown `status` `{status}` (expected "
f"{'|'.join(sorted(STATUS_VALUES))})")
stale = meta.get("stale_after")
if stale is not None and not RFC3339.match(str(stale)):
report.warn(rel, f"§5.5 `stale_after` `{stale}` is not an ISO 8601 datetime "
f"(a bare date is tolerated)")
def check_window(window, where: str, rel: str, report: Report) -> None:
"""§5.1 — a `usage_window` is a `{from, to}` pair of ISO 8601 datetimes."""
if window is None:
return
if not isinstance(window, dict):
report.warn(rel, f"§5.1 `{where}` `usage_window` must be a `{{from, to}}` mapping")
return
for bound in ("from", "to"):
value = window.get(bound)
if value is None:
report.warn(rel, f"§5.1 `{where}` `usage_window` is missing `{bound}`")
elif not RFC3339.match(str(value)):
report.warn(rel, f"§5.1 `{where}` `usage_window.{bound}` `{value}` "
f"is not an ISO 8601 datetime (a bare date is tolerated)")
def check_sources(meta: dict, body: str, rel: str, report: Report) -> None:
"""§5.1 — the `sources` provenance family and its footnote join keys."""
if CITATIONS.search(body):
report.warn(rel, "legacy v0.1 `# Citations` body list: v0.2 records provenance "
"in the `sources` frontmatter (§13.1)")
srcs = meta.get("sources")
if srcs is None:
return
if not isinstance(srcs, list):
report.warn(rel, "§5.1 `sources` must be a list of entries")
return
ids = set()
for i, src in enumerate(srcs):
if not isinstance(src, dict):
report.warn(rel, f"§5.1 `sources[{i}]` must be a mapping")
continue
if not str(src.get("resource", "")).strip():
report.warn(rel, f"§5.1 `sources[{i}]` is missing the required `resource`")
if "id" in src:
ids.add(str(src["id"]))
if src.get("author") is not None:
check_actor(src["author"], f"sources[{i}].author", rel, report)
# §5.1 — `usage_count` is framed by a `usage_window`, written once as a
# sibling of `sources` or overridden on the entry. Without one the count
# is a number with no units, and the spec asks consumers to read it as
# liveness and trend rather than a score.
window = src.get("usage_window", meta.get("usage_window"))
if src.get("usage_count") is not None and window is None:
report.warn(rel, f"§5.1 `sources[{i}].usage_count` has no `usage_window` "
f"framing it (a sibling of `sources`, or on the entry)")
check_window(window, f"sources[{i}]", rel, report)
last_mod = src.get("last_modified")
if last_mod is not None and not RFC3339.match(str(last_mod)):
report.warn(rel, f"§5.1 `sources[{i}].last_modified` `{last_mod}` is not an "
f"ISO 8601 datetime (a bare date is tolerated)")
# Attribution joins on the label, not on position (§5.1) — a footnote whose
# label names no source silently attributes a claim to nothing.
for label in sorted(set(FOOTNOTE.findall(body)) - ids):
report.warn(rel, f"§5.1 footnote `[^{label}]` matches no `sources[].id`")
def check_index(path: Path, rel: str, is_root: bool, report: Report) -> None:
report.indexes += 1
text = _read_text(path, rel, report)
if text is None:
return
raw, _ = split_frontmatter(text)
if raw is not None:
if not is_root:
report.warn(rel, "§8 index.md should contain no frontmatter")
else:
try:
meta = yaml.safe_load(raw) or {}
except yaml.YAMLError:
report.warn(rel, "§12 root index.md frontmatter is not valid YAML")
meta = {}
# `upkeep` rides along: the plugin's Stop hook reads its opt-in flag
# from here, so an enforced-mode bundle would otherwise fail its own
# `--strict` run. §11 forbids rejecting a bundle for unknown keys.
extra = set(meta) - {"okf_version", "upkeep"}
if extra:
report.warn(rel, f"§12 root index.md frontmatter may only carry `okf_version` (found {sorted(extra)})")
declared = meta.get("okf_version")
if declared is not None and str(declared) != OKF_VERSION:
report.warn(rel, f"§12 bundle declares `okf_version: \"{declared}\"`; "
f"checked against v{OKF_VERSION}")
def check_log(path: Path, rel: str, report: Report) -> None:
report.logs += 1
text = _read_text(path, rel, report)
if text is None:
return
raw, _ = split_frontmatter(text)
if raw is not None:
report.warn(rel, "§9 log.md should contain no frontmatter")
for line in text.splitlines():
if line.startswith("## "):
heading = line[3:].strip()
if not ISO_DATE.match(heading):
report.warn(rel, f"§9 date heading `{heading}` is not ISO 8601 YYYY-MM-DD")
def collect_link_targets(path: Path) -> list[str]:
# Unreadable files are already reported as a per-file error by
# check_concept/check_index/check_log; fail soft here to avoid a duplicate
# error and a crash on the second read pass.
try:
text = path.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError):
return []
targets: list[str] = []
in_fence = False
for line in text.splitlines():
if FENCE.match(line.strip()):
in_fence = not in_fence
continue
if in_fence:
continue
targets.extend(LINK.findall(line))
return targets
def check_links(bundle: Path, md_files: list[Path], report: Report) -> None:
"""Broken bundle-internal links are warnings only (§6.1)."""
existing = {p.relative_to(bundle).as_posix() for p in md_files}
for path in md_files:
rel = path.relative_to(bundle).as_posix()
for target in collect_link_targets(path):
t = target.split("#", 1)[0]
if not t or t.endswith("/"):
continue
if re.match(r"^[a-z][a-z0-9+.-]*://", t) or t.startswith("mailto:"):
continue
if not t.endswith(".md"):
continue
if t.startswith("/"):
resolved = t.lstrip("/")
else:
resolved = (path.parent / t).resolve().relative_to(bundle.resolve()).as_posix() \
if (path.parent / t).resolve().is_relative_to(bundle.resolve()) else t
if resolved not in existing:
report.warn(rel, f"cross-link target not found: `{target}` (tolerated under §6.1)")
def citations_to_sources(raw: str, body: str) -> tuple[str, str]:
"""Hoist a v0.1 `# Citations` list into the `sources` family (§5.1, §13.1)."""
heading = CITATIONS.search(body)
if heading is None:
return raw, body
rest = body[heading.end():]
nxt = HEADING.search(rest)
section, tail = (rest[:nxt.start()], rest[nxt.start():]) if nxt else (rest, "")
entries = MD_LINK.findall(section)
if not entries:
return raw, body # nothing extractable — leave it and keep warning
# json.dumps gives a correctly escaped YAML double-quoted scalar for free.
block = "sources:\n" + "".join(
f" - resource: {json.dumps(url)}\n title: {json.dumps(title)}\n"
for title, url in entries)
kept = body[:heading.start()].rstrip("\n")
return raw + block, kept + ("\n\n" + tail if tail else "\n")
def migrate_text(text: str, is_root_index: bool) -> str | None:
"""Rewrite one file's v0.1 constructs to v0.2. None when nothing changes.
Textual on purpose: dumping the frontmatter back through PyYAML would flatten
comments, key order and quoting across every file in the bundle. Each branch
is guarded on the v0.2 key already being present, so a half-migrated bundle
converges instead of growing a duplicate key that PyYAML then refuses to load.
"""
raw, body = split_frontmatter(text)
if raw is None:
return None
if is_root_index:
new_raw = OKF_VERSION_LINE.sub(rf'\g<1>"{OKF_VERSION}"', raw, count=1)
return None if new_raw == raw else f"---\n{new_raw}---\n{body}"
try:
meta = yaml.safe_load(raw)
except yaml.YAMLError:
return None
if not isinstance(meta, dict):
return None
new_raw, new_body = raw, body
if "generated" not in meta:
new_raw = TIMESTAMP.sub(
lambda m: f"generated:\n by: {MIGRATE_ACTOR}\n at: {m.group(1)}",
new_raw, count=1)
if "sources" not in meta:
new_raw, new_body = citations_to_sources(new_raw, new_body)
if (new_raw, new_body) == (raw, body):
return None
return f"---\n{new_raw}---\n{new_body}"
def migrate(bundle: Path) -> list[str]:
"""Migrate a bundle in place. Returns the relative paths actually rewritten."""
changed = []
for path in sorted(p for p in bundle.rglob("*.md") if p.is_file()):
rel = path.relative_to(bundle).as_posix()
if path.name == "log.md":
continue
try:
# Universal newlines on purpose: a CRLF bundle is rewritten with the
# platform ending, consistently. Passing newline="" to preserve them
# exactly would leave a stray \r inside every `$`-anchored capture
# below, so those patterns need hardening first if byte fidelity is
# ever required.
text = path.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError):
continue
is_root_index = path.name == "index.md" and path.parent == bundle
if path.name == "index.md" and not is_root_index:
continue
new = migrate_text(text.lstrip(""), is_root_index)
if new is not None and new != text:
path.write_text(new, encoding="utf-8")
changed.append(rel)
return changed
def validate(bundle: Path) -> Report:
report = Report()
md_files = sorted(p for p in bundle.rglob("*.md") if p.is_file())
for path in md_files:
rel = path.relative_to(bundle).as_posix()
name = path.name
if name == "index.md":
check_index(path, rel, is_root=(path.parent == bundle), report=report)
elif name == "log.md":
check_log(path, rel, report)
else:
check_concept(path, rel, bundle, report)
check_links(bundle, md_files, report)
return report
def _force_utf8_stdio() -> None:
"""Ensure stdout/stderr can emit the ✓/✗/— glyphs this tool prints.
Windows consoles default to cp1252, which raises UnicodeEncodeError on those
characters and crashes an otherwise-successful run. Reconfiguring to UTF-8
(Python ≥3.7) fixes it; `errors="replace"` is a belt-and-suspenders fallback
for any stream that still cannot encode a glyph.
"""
for stream in (sys.stdout, sys.stderr):
reconfigure = getattr(stream, "reconfigure", None)
if reconfigure is not None:
try:
reconfigure(encoding="utf-8", errors="replace")
except (ValueError, OSError):
pass
def main() -> int:
_force_utf8_stdio()
ap = argparse.ArgumentParser(description=f"Validate an OKF v{OKF_VERSION} bundle.")
ap.add_argument("bundle", type=Path, help="path to the bundle directory")
ap.add_argument("--strict", action="store_true", help="treat warnings as errors")
ap.add_argument("--max-warnings", type=int, default=None, metavar="N",
help="fail if warnings exceed N (--strict is the N=0 case)")
ap.add_argument("--migrate", action="store_true",
help="rewrite v0.1 constructs to v0.2 in place, then validate")
ap.add_argument("--json", action="store_true", help="emit a JSON report")
args = ap.parse_args()
if not args.bundle.is_dir():
print(f"error: {args.bundle} is not a directory", file=sys.stderr)
return 2
migrated = migrate(args.bundle) if args.migrate else []
r = validate(args.bundle)
conformant = not r.errors
max_warn = 0 if args.strict else args.max_warnings
failed = bool(r.errors) or (max_warn is not None and len(r.warnings) > max_warn)
if args.json:
print(json.dumps({
"bundle": str(args.bundle),
"conformant": conformant,
"passed": not failed,
"counts": {"concepts": r.concepts, "indexes": r.indexes, "logs": r.logs},
"errors": r.errors,
"warnings": r.warnings,
"migrated": migrated,
}, indent=2))
return 0 if not failed else 1
if args.migrate:
for rel in migrated:
print(f" \033[36m→ migrated\033[0m {rel}")
print(f" migrated {len(migrated)} file(s) to v{OKF_VERSION}"
+ (f", `generated.by` recorded as `{MIGRATE_ACTOR}`" if migrated else ""))
if migrated:
print(" note: per-claim attribution (the `[^id]` footnotes of §5.1) is not"
" recoverable from a v0.1 `# Citations` list — sources moved up, but"
" which claim cited which was never encoded.")
print(f"OKF v{OKF_VERSION} conformance — {args.bundle}")
print(f" concepts: {r.concepts} index.md: {r.indexes} log.md: {r.logs}")
for e in r.errors:
print(f" \033[31m✗ ERROR\033[0m {e}")
for w in r.warnings:
print(f" \033[33m! warn \033[0m {w}")
if conformant and not r.warnings:
print(" \033[32m✓ conformant — no issues\033[0m")
elif conformant:
print(f" \033[32m✓ conformant\033[0m ({len(r.warnings)} warning(s))")
else:
print(f" \033[31m✗ non-conformant\033[0m ({len(r.errors)} error(s))")
return 0 if not failed else 1
if __name__ == "__main__":
raise SystemExit(main())
external/okf/skills/visualize/SKILL.md
Vendored
+64
View File
@@ -0,0 +1,64 @@
---
name: visualize
description: >-
Render an Open Knowledge Format (OKF) bundle as a single self-contained,
interactive HTML graph (viz.html) — concepts as nodes coloured by type and
sized by body length, markdown links and bundle-internal `sources` as edges,
a wiki-style detail panel with rendered markdown, v0.2 trust/lifecycle/provenance
metadata, and "Links to" / "Cited by" backlinks, layout switching, per-type
filter and search.
Use when asked to visualize, graph, preview, or explore an OKF bundle.
user-invocable: true
argument-hint: "<bundle-dir> [-o viz.html]"
allowed-tools: Bash
---
# Visualize an OKF bundle
Generate a self-contained HTML graph of the target bundle. The bundle directory
is required, so pass `.okf` explicitly when no path is given. No backend, no
install, no bundle content leaves the page. The viewer does need network
access: cytoscape, marked and dompurify load from cdn.jsdelivr.net.
```bash
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_visualize.py" $ARGUMENTS
```
If `uv` is unavailable:
```bash
python3 -m pip install --quiet pyyaml && \
python3 "${CLAUDE_SKILL_DIR}/scripts/okf_visualize.py" $ARGUMENTS
```
The detail panel shows each concept's `type`, `description`, `tags`, `status`,
`generated`, `verified`, `stale_after`, `sources` (with their credibility
signals, a `usage_count` alongside the `usage_window` that frames it), and
rendered body; a `sources` entry pointing at another concept in the bundle also
becomes a graph edge. A v0.1 `timestamp` is read as `generated.at`, so legacy
bundles still render fully.
Two badges are **derived**, not read: the §5.3 trust tier
(*unverified* / *machine-confirmed* / *human-reviewed*, keyed off the `human:`
prefix in `verified[].by`) and staleness (`now >= stale_after`). They are
advisory signals — when reporting on a bundle, say which tier a concept is in
rather than treating any of them as a gate.
| Flag | Effect |
|------|--------|
| `-o <path>` | Output file (default `<bundle>/viz.html`). |
| `-t, --title` | Graph title (default `parent/bundle` dir name). |
| `-l, --link` | Source URL shown in the header. |
| `--layout` | `cose`, `concentric`, `breadthfirst`, `circle`, `grid`. |
| `--max-nodes N` | Exit 1 above N concepts (useful in CI). |
| `--og-image URL` | Absolute URL for the social preview image. |
Above 1,000 concepts the default switches to `concentric` (force freezes the
page; `--layout cose` overrides, and the in-page picker asks first). Above
5,000 it warns that the page loads slowly and reads as a hairball, and
suggests rendering a subdirectory.
Open it in any browser; `${CLAUDE_SKILL_DIR}` resolves whether this runs as part
of the `okf` plugin or as a standalone skills.sh skill. The page is
deep-linkable: `?select=<concept-id>` or `#<concept-id>` opens that concept,
`?layout=<name>` presets the layout.
external/okf/skills/visualize/scripts/okf_visualize.py
Vendored
+432
View File
@@ -0,0 +1,432 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.11"
# dependencies = ["pyyaml>=6"]
# ///
"""Render an Open Knowledge Format (OKF) bundle as a single self-contained,
interactive HTML graph (`viz.html`). No backend, no install on the viewing side,
no data leaves the page — concepts become nodes (coloured by `type`, sized by
body length), markdown links and bundle-internal `sources` become edges, and
clicking a node opens a wiki-style panel with its rendered markdown, OKF v0.2
provenance/trust/lifecycle metadata, outgoing links, and "Cited by" backlinks.
Features: force/concentric/breadth-first/circle/grid layouts, per-type filter,
free-text search, neighbour highlight, clickable cross-links and backlinks.
The default layout is force (cose) up to AUTO_COSE_MAX concepts, then the linear
concentric layout — force-directed cost grows roughly quadratically with node
count and freezes the page on large bundles. An explicit --layout always wins.
Run: uv run okf_visualize.py <bundle-dir> [-o viz.html]
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
import yaml
RESERVED = {"index.md", "log.md"}
# Force (cose) layout froze the page for ~32 s at ~2k concepts (measured in
# Chrome); the linear layouts load the same bundle in under 2 s. Past this size
# the default switches to concentric, and the in-page layout picker asks before
# running force. An explicit --layout (or ?layout=) still wins.
AUTO_COSE_MAX = 1000
# Above this the page is slow on any layout (23k concepts measured: ~27 s load,
# ~650 MB heap) and reads as a hairball — warn and suggest rendering a subtree.
SCALE_WARN = 5000
FENCE = re.compile(r"^(```|~~~)")
LINK = re.compile(r"(?<!\!)\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
def json_for_script(obj) -> str:
"""JSON-encode ``obj`` for safe embedding inside an inline ``<script>``.
The HTML tokenizer knows nothing about JS string context: a literal
``</script>`` in a concept body ends the inline script early (truncating the
embedded ``NODES``/``EDGES`` and breaking the whole page), and ``<!--`` can
shift it into the escaped script-data states where even the template's real
closing tag stops working. Escaping every ``<`` as ``\\u003c`` neutralizes
``</script>``, ``<!--`` and ``<script`` in one stroke, and the result stays
valid JSON *and* JavaScript (``json.dumps`` keeps everything else ASCII via
the default ``ensure_ascii=True``).
"""
return json.dumps(obj, default=str).replace("<", "\\u003c")
def split_frontmatter(text: str):
if not text.startswith("---"):
return {}, text
lines = text.splitlines(keepends=True)
if lines[0].strip() != "---":
return {}, text
for i in range(1, len(lines)):
if lines[i].strip() == "---":
try:
meta = yaml.safe_load("".join(lines[1:i])) or {}
except yaml.YAMLError:
meta = {}
return (meta if isinstance(meta, dict) else {}), "".join(lines[i + 1:])
return {}, text
def link_targets(text: str):
out, in_fence = [], False
for line in text.splitlines():
if FENCE.match(line.strip()):
in_fence = not in_fence
continue
if not in_fence:
out.extend(LINK.findall(line))
return out
def resolve(target: str, path: Path, bundle: Path):
"""Resolve a link/`sources[].resource` to a concept id, or None if it is not
one (an external URL, an asset, a scope descriptor, an escape from the tree)."""
t = str(target).split("#", 1)[0]
if not t.endswith(".md"):
return None
if t.startswith("/"):
return t.lstrip("/")[:-3]
cand = (path.parent / t).resolve()
return cand.relative_to(bundle.resolve()).as_posix()[:-3] \
if cand.is_relative_to(bundle.resolve()) else None
def read_sources(meta: dict, path: Path, bundle: Path):
"""§5.1 `sources`, flattened for display. `cid` is set when the source is
itself a concept in this bundle — that derivation is a real graph edge."""
raw = meta.get("sources")
out = []
if not isinstance(raw, list):
return out
for src in raw:
if not isinstance(src, dict):
continue
resource = str(src.get("resource", "")).strip()
# §5.1 — an entry's own `usage_window` overrides the one written once as a
# sibling of `sources`. A count without its window has no units.
window = src.get("usage_window", meta.get("usage_window"))
out.append({
"title": str(src.get("title") or src.get("id") or resource),
"resource": resource,
"cid": resolve(resource, path, bundle) if resource else None,
"author": str(src.get("author", "")),
"usage_count": src.get("usage_count"),
"usage_window": (f"{window.get('from', '?')}→{window.get('to', '?')}"
if isinstance(window, dict) else ""),
"last_modified": str(src.get("last_modified") or ""),
})
return out
def read_trust(meta: dict):
"""§5.2 `generated` / `verified`, falling back to a v0.1 `timestamp` (§13.1)."""
gen = meta.get("generated")
if isinstance(gen, dict):
generated = {"by": str(gen.get("by", "")), "at": str(gen.get("at", ""))}
elif meta.get("timestamp"):
generated = {"by": "", "at": str(meta["timestamp"])}
else:
generated = None
ver = meta.get("verified")
# a bare mapping is one verification event (§5.2)
entries = [ver] if isinstance(ver, dict) else (ver if isinstance(ver, list) else [])
verified = [{"by": str(e.get("by", "")), "at": str(e.get("at", ""))}
for e in entries if isinstance(e, dict)]
return generated, verified
def build(bundle: Path):
nodes, edges, seen = [], [], set()
files = sorted(p for p in bundle.rglob("*.md") if p.is_file() and p.name not in RESERVED)
ids = {p.relative_to(bundle).with_suffix("").as_posix() for p in files}
for p in files:
cid = p.relative_to(bundle).with_suffix("").as_posix()
try:
raw = p.read_text(encoding="utf-8").lstrip("")
except (UnicodeDecodeError, OSError) as exc:
print(f"warning: skipping {p.relative_to(bundle)}: cannot read file: {exc}", file=sys.stderr)
ids.discard(cid)
continue
meta, body = split_frontmatter(raw)
body = body.strip()
generated, verified = read_trust(meta)
sources = read_sources(meta, p, bundle)
nodes.append({
"id": cid,
"type": str(meta.get("type", "Untyped")),
"title": str(meta.get("title", p.stem)),
"description": str(meta.get("description", "")),
"tags": meta.get("tags", []) if isinstance(meta.get("tags"), list) else [],
"group": cid.split("/")[0] if "/" in cid else "(root)",
"sz": max(24, min(70, 24 + len(body) // 200)),
"status": str(meta.get("status", "")),
"stale_after": str(meta.get("stale_after") or ""),
"generated": generated,
"verified": verified,
"sources": sources,
"body": body[:8000],
})
targets = link_targets(body) + [s["resource"] for s in sources if s["cid"]]
for t in targets:
tgt = resolve(t, p, bundle)
if tgt and tgt in ids and tgt != cid and (cid, tgt) not in seen:
seen.add((cid, tgt))
edges.append({"source": cid, "target": tgt})
return nodes, edges
HTML = r"""<!doctype html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>OKF — __NAME__</title>
<meta property="og:title" content="__OGTITLE__">
<meta property="og:description" content="__OGDESC__">
<meta property="og:type" content="website">
<meta name="twitter:card" content="summary_large_image">
__OGIMAGE__
<script src="https://cdn.jsdelivr.net/npm/cytoscape@3.30.2/dist/cytoscape.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/marked@14/marked.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/dompurify@3.4.12/dist/purify.min.js" integrity="sha384-piCcpDdJ7qVeK4Tv8Z6Hpcr3ZBIgP16TxQTPVfsLFdZ5uDgwc3Y8Ho7oUnqf12qu" crossorigin="anonymous"></script>
<style>
:root{--bg:#0e0f13;--panel:#16181f;--line:#262a35;--fg:#e6e8ee;--mut:#9aa3b2;--accent:#8ab4ff}
*{box-sizing:border-box} html,body{margin:0;height:100%;background:var(--bg);color:var(--fg);
font:14px/1.5 ui-sans-serif,system-ui,-apple-system,Segoe UI,Roboto,sans-serif}
/* The row must be stated. Left implicit it sizes to `auto`, i.e. to the tallest
item — a long concept body then stretched #side past the viewport and the page
grew its own scrollbar on top of the panel's, which in turn forced a spurious
horizontal one. Pinning the row to 100% keeps the panel scrolling inside. */
#app{display:grid;grid-template-columns:1fr 400px;grid-template-rows:100%;height:100vh}
#cy{width:100%;height:100%}
#side{border-left:1px solid var(--line);background:var(--panel);overflow:auto;padding:18px}
header{position:absolute;top:0;left:0;padding:14px 18px;z-index:5;pointer-events:none}
h1{font-size:15px;margin:0;font-weight:650} .sub{color:var(--mut);font-size:12px;margin-top:2px}
#bar{position:absolute;top:12px;left:50%;transform:translateX(-50%);z-index:5;display:flex;gap:8px}
#bar input,#bar select{background:var(--panel);border:1px solid var(--line);color:var(--fg);
border-radius:8px;padding:8px 10px;outline:none;font-size:13px}
#search{width:min(300px,32vw)}
#legend{position:absolute;bottom:14px;left:18px;z-index:5;display:flex;flex-wrap:wrap;gap:6px;max-width:60vw}
.chip{display:flex;align-items:center;gap:6px;background:var(--panel);border:1px solid var(--line);
border-radius:20px;padding:3px 10px;font-size:12px;color:var(--mut);cursor:pointer;user-select:none}
.chip.off{opacity:.4} .dot{width:10px;height:10px;border-radius:50%}
#side h2{font-size:17px;margin:.2em 0} .type{display:inline-block;border-radius:6px;padding:2px 8px;
font-size:11px;font-weight:600;color:#0e0f13} .desc{color:var(--mut);margin:8px 0 12px}
.tags{display:flex;gap:6px;flex-wrap:wrap;margin-bottom:12px}
.tag{background:#1d2230;border:1px solid var(--line);border-radius:6px;padding:1px 8px;font-size:11px;color:var(--mut)}
.meta{margin:10px 0;font-size:12px;color:var(--mut)} .meta div{padding:1px 0} .meta b{color:var(--fg);font-weight:600}
.sig{color:var(--mut)}
.badges{display:flex;gap:6px;flex-wrap:wrap;margin:8px 0}
.badge{border-radius:6px;padding:2px 8px;font-size:11px;font-weight:600;border:1px solid}
.t-unverified{color:#9aa3b2;border-color:#3a4150;background:#1d2230}
.t-machine{color:#8ab4ff;border-color:#2b4570;background:#141c2b}
.t-human{color:#4ade80;border-color:#276b45;background:#12211a}
.b-stale{color:#fca5a5;border-color:#7f2b2b;background:#241416}
.b-deprecated{color:#fbbf24;border-color:#7a5312;background:#241d10}
.rel{margin:10px 0} .rel h4{margin:0 0 4px;font-size:11px;text-transform:uppercase;letter-spacing:.05em;color:var(--mut)}
.rel a{display:block;color:var(--accent);cursor:pointer;font-size:13px;padding:1px 0;text-decoration:none}
.rel a:hover{text-decoration:underline}
.body{border-top:1px solid var(--line);padding-top:12px;margin-top:12px} .body table{border-collapse:collapse;display:block;overflow:auto}
.body td,.body th{border:1px solid var(--line);padding:4px 8px} .body code{background:#1d2230;padding:1px 5px;border-radius:4px}
.body pre{background:#1d2230;padding:10px;border-radius:8px;overflow:auto} .body img{max-width:100%}
.empty{color:var(--mut)} a{color:var(--accent)}
.src{pointer-events:auto;color:var(--accent);margin-left:10px;text-decoration:none} .src:hover{text-decoration:underline}
</style></head><body>
<div id="app"><div id="cy"></div><div id="side"><p class="empty">Click a concept to inspect it.</p></div></div>
<header><h1>__NAME__</h1><div class="sub">__N__ concepts · __E__ links · OKF v0.2__LINK__</div></header>
<div id="bar">
<input id="search" placeholder="search concepts…">
<select id="type"><option value="">all types</option></select>
<select id="layout">
<option value="cose">force</option><option value="concentric">concentric</option>
<option value="breadthfirst">breadth-first</option><option value="circle">circle</option><option value="grid">grid</option>
</select>
</div>
<div id="legend"></div>
<script>
const NODES=__NODES__, EDGES=__EDGES__;
const PALETTE=["#6E56CF","#D97757","#22C55E","#3B82F6","#EAB308","#EC4899","#14B8A6","#F97316","#A855F7","#0EA5E9","#84CC16","#EF4444","#64748B"];
const byId=Object.fromEntries(NODES.map(n=>[n.id,n]));
const outL={}, inL={};
NODES.forEach(n=>{outL[n.id]=[];inL[n.id]=[];});
EDGES.forEach(e=>{outL[e.source].push(e.target);inL[e.target].push(e.source);});
const types=[...new Set(NODES.map(n=>n.type))].sort();
const color=Object.fromEntries(types.map((t,i)=>[t,PALETTE[i%PALETTE.length]]));
const off=new Set();
const cy=cytoscape({container:document.getElementById('cy'),minZoom:.2,maxZoom:1.6,wheelSensitivity:.2,
elements:[...NODES.map(n=>({data:{...n,c:color[n.type]}})),...EDGES.map(e=>({data:e}))],
style:[
{selector:'node',style:{'background-color':'data(c)','label':'data(title)','color':'#e6e8ee',
'font-size':10,'text-wrap':'wrap','text-max-width':120,'text-valign':'bottom','text-margin-y':4,
'text-outline-width':2,'text-outline-color':'#0e0f13','min-zoomed-font-size':6,
'width':'data(sz)','height':'data(sz)'}},
{selector:'edge',style:{'width':1.2,'line-color':'#3a4150','target-arrow-color':'#3a4150',
'target-arrow-shape':'triangle','arrow-scale':.8,'curve-style':'bezier','opacity':.7}},
{selector:'.dim',style:{'opacity':.10}},{selector:'.hl',style:{'border-width':3,'border-color':'#fff'}}
],
layout:{name:'__LAYOUT__',animate:false,nodeRepulsion:9000,idealEdgeLength:90,padding:40}});
const side=document.getElementById('side');
const esc=s=>(s||'').replace(/[&<>"']/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
function relList(title,arr){if(!arr.length)return'';
return `<div class="rel"><h4>${title}</h4>${arr.map(id=>`<a data-go="${esc(id)}">${esc((byId[id]||{}).title||id)}</a>`).join('')}</div>`;}
// §5.3 — the trust tier is derived, never stored: no `verified` is unverified,
// `verified` by non-`human:` actors only is machine-confirmed, and any `human:`
// actor makes it human-reviewed. The exact lowercase prefix is the whole key.
// §5.5 — a concept is stale when now >= stale_after. PyYAML renders an unquoted
// datetime with a space separator, which Date.parse rejects in some engines.
// Advisory signals, not access control.
function trustTier(n){const v=n.verified||[];
if(!v.length)return['t-unverified','unverified'];
return v.some(e=>(e.by||'').startsWith('human:'))?['t-human','human-reviewed']
:['t-machine','machine-confirmed'];}
function badges(n){const [cls,label]=trustTier(n),out=[`<span class="badge ${cls}">${label}</span>`];
if(n.stale_after&&Date.parse(n.stale_after.replace(' ','T'))<=Date.now())out.push(`<span class="badge b-stale">stale since ${esc(n.stale_after)}</span>`);
if(n.status==='deprecated')out.push('<span class="badge b-deprecated">deprecated</span>');
return `<div class="badges">${out.join('')}</div>`;}
// OKF v0.2 trust (§5.2) + lifecycle (§5.4/§5.5). A v0.1 `timestamp` arrives here
// as generated.at with an empty `by`, so legacy bundles still show a date.
function metaBlock(n){const g=n.generated||{},rows=[];
if(n.status)rows.push(`<div>status <b>${esc(n.status)}</b></div>`);
if(g.at||g.by)rows.push(`<div>generated${g.at?` <b>${esc(g.at)}</b>`:''}${g.by?` by ${esc(g.by)}`:''}</div>`);
(n.verified||[]).forEach(v=>rows.push(`<div>verified${v.at?` <b>${esc(v.at)}</b>`:''}${v.by?` by ${esc(v.by)}`:''}</div>`));
if(n.stale_after)rows.push(`<div>stale after <b>${esc(n.stale_after)}</b></div>`);
return rows.length?`<div class="meta">${rows.join('')}</div>`:'';}
// Provenance (§5.1). A source may be another concept (graph link), an external
// URL, or a scope descriptor that is not followable at all.
function srcList(n){const s=n.sources||[];if(!s.length)return'';
return `<div class="rel"><h4>Sources</h4>${s.map(x=>{
const used=x.usage_count!=null?`used ${x.usage_count}×${x.usage_window?` (${x.usage_window})`:''}`:'';
const sig=[x.author,used,x.last_modified].filter(Boolean).join(' · ');
const label=esc(x.title||x.resource)+(sig?` <span class="sig">(${esc(sig)})</span>`:'');
if(x.cid&&byId[x.cid])return `<a data-go="${esc(x.cid)}">${label}</a>`;
if(/^https?:\/\//i.test(x.resource))return `<a href="${esc(x.resource)}" target="_blank" rel="noopener">${label}</a>`;
return `<span class="empty">${label}</span>`;}).join('')}</div>`;}
// Resolve an in-body markdown link href to a concept id, mirroring build()'s
// resolution: strip #anchor, require .md, absolute strips leading /, relative
// resolves against the current concept's dir. Returns null if it's not a concept.
function resolveHref(cid,href){let t=(href||'').split('#')[0];if(!t.endsWith('.md'))return null;
let tgt;if(t[0]==='/'){tgt=t.replace(/^\/+/,'').slice(0,-3);}
else{const base=cid.split('/').slice(0,-1);
for(const seg of t.slice(0,-3).split('/')){if(seg===''||seg==='.')continue;
if(seg==='..'){if(!base.length)return null;base.pop();}else base.push(seg);}
tgt=base.join('/');}
return byId[tgt]?tgt:null;}
function show(id){const n=byId[id];if(!n)return;const c=color[n.type];
side.innerHTML=`<span class="type" style="background:${c}">${esc(n.type)}</span>
<h2>${esc(n.title)}</h2><div class="desc">${esc(n.description)||'<span class=empty>no description</span>'}</div>
<div class="tags">${(n.tags||[]).map(t=>`<span class="tag">${esc(t)}</span>`).join('')}</div>
${badges(n)}${metaBlock(n)}${srcList(n)}${relList('Links to',outL[id])}${relList('Cited by',inL[id])}
<div class="body">${n.body?DOMPurify.sanitize(marked.parse(n.body)):'<span class=empty>empty body</span>'}</div>`;
side.querySelectorAll('[data-go]').forEach(a=>a.onclick=()=>select(a.getAttribute('data-go')));
side.querySelectorAll('.body a[href]').forEach(a=>{const tgt=resolveHref(id,a.getAttribute('href'));
if(tgt)a.onclick=e=>{e.preventDefault();select(tgt);};});}
function select(id){const ele=cy.getElementById(id);if(!ele.length)return;show(id);
cy.elements().removeClass('hl').addClass('dim');const nb=ele.closedNeighborhood();nb.removeClass('dim');ele.addClass('hl');
cy.animate({center:{eles:ele},duration:250});
try{if(decodeURIComponent((location.hash||'').slice(1))!==id)location.hash=encodeURIComponent(id);}catch(e){}}
cy.on('tap','node',e=>select(e.target.id()));
cy.on('tap',e=>{if(e.target===cy)cy.elements().removeClass('dim hl');});
function applyFilter(){const q=document.getElementById('search').value.toLowerCase();const ty=document.getElementById('type').value;
cy.batch(()=>cy.nodes().forEach(n=>{const d=n.data();
const m=(!q||(d.title+' '+d.type+' '+d.description+' '+(d.tags||[]).join(' ')).toLowerCase().includes(q))
&&(!ty||d.type===ty)&&!off.has(d.type);
n.style('display',m?'element':'none');}));}
let debounce;
document.getElementById('search').oninput=()=>{clearTimeout(debounce);debounce=setTimeout(applyFilter,150);};
document.getElementById('type').oninput=applyFilter;
const tysel=document.getElementById('type');types.forEach(t=>{const o=document.createElement('option');o.value=t;o.textContent=t;tysel.appendChild(o);});
let curLayout='__LAYOUT__';
document.getElementById('layout').onchange=e=>{const v=e.target.value;
if(v==='cose'&&NODES.length>__COSEMAX__&&!confirm(`force layout on ${NODES.length} concepts can freeze this tab — run anyway?`)){e.target.value=curLayout;return;}
curLayout=v;cy.layout({name:v,animate:true,padding:40,nodeRepulsion:9000,idealEdgeLength:90}).run();};
document.getElementById('legend').innerHTML=types.map(t=>`<span class="chip" data-t="${esc(t)}"><span class="dot" style="background:${color[t]}"></span>${esc(t)} (${NODES.filter(n=>n.type===t).length})</span>`).join('');
document.querySelectorAll('#legend .chip').forEach(ch=>ch.onclick=()=>{const t=ch.getAttribute('data-t');
if(off.has(t)){off.delete(t);ch.classList.remove('off');}else{off.add(t);ch.classList.add('off');}applyFilter();});
document.getElementById('layout').value='__LAYOUT__';
const Q=new URLSearchParams(location.search),QL=Q.get('layout'),QS=Q.get('select');
if(QL&&[...document.querySelectorAll('#layout option')].some(o=>o.value===QL)){document.getElementById('layout').value=QL;curLayout=QL;cy.layout({name:QL,animate:false,padding:40,nodeRepulsion:9000,idealEdgeLength:90}).run();}
function fromHash(){try{const h=decodeURIComponent((location.hash||'').slice(1));if(h&&byId[h])select(h);}catch(e){}}
addEventListener('hashchange',fromHash);
if(QS&&byId[QS])select(QS);else fromHash();
</script></body></html>"""
def render(bundle: Path, out: Path, title: str | None = None, link: str | None = None,
layout: str | None = None, og_image: str | None = None,
max_nodes: int | None = None):
nodes, edges = build(bundle)
if max_nodes is not None and len(nodes) > max_nodes:
sys.exit(f"error: {len(nodes)} concepts exceeds --max-nodes {max_nodes}")
if layout is None:
layout = "cose" if len(nodes) <= AUTO_COSE_MAX else "concentric"
if layout != "cose":
print(f"note: {len(nodes)} concepts > {AUTO_COSE_MAX} — using the linear 'concentric' "
"layout (force freezes the page at this size; pass --layout cose to override)",
file=sys.stderr)
if len(nodes) > SCALE_WARN:
print(f"warning: {len(nodes)} concepts — the page will load slowly and read as a hairball; "
f"consider rendering a subtree, e.g. okf_visualize.py {bundle}/<subdir>",
file=sys.stderr)
name = title or f"{bundle.resolve().parent.name}/{bundle.name}"
src = f' <a class="src" href="{link}" target="_blank" rel="noopener">source ↗</a>' if link else ""
aesc = lambda s: (s or "").replace("&", "&amp;").replace('"', "&quot;").replace("<", "&lt;").replace(">", "&gt;")
og_title = aesc(f"OKF — {name}")
og_desc = aesc(f"{len(nodes)} concepts · interactive Open Knowledge Format knowledge graph")
og_img = (f'<meta property="og:image" content="{aesc(og_image)}">\n'
f'<meta name="twitter:image" content="{aesc(og_image)}">') if og_image else ""
subs = {"__NAME__": name, "__LINK__": src, "__LAYOUT__": layout,
"__COSEMAX__": str(AUTO_COSE_MAX),
"__OGTITLE__": og_title, "__OGDESC__": og_desc, "__OGIMAGE__": og_img,
"__N__": str(len(nodes)), "__E__": str(len(edges)),
"__NODES__": json_for_script(nodes), "__EDGES__": json_for_script(edges)}
# One pass, longest marker first: substituted content (e.g. a concept body that
# mentions "__EDGES__") must never itself be rescanned for other markers.
marker = re.compile("|".join(sorted(map(re.escape, subs), key=len, reverse=True)))
html = marker.sub(lambda m: subs[m.group(0)], HTML)
out.write_text(html, encoding="utf-8")
return len(nodes), len(edges)
def _force_utf8_stdio() -> None:
"""Reconfigure stdout/stderr to UTF-8 so status lines never crash on a
cp1252 (default Windows) console. `errors="replace"` is a fallback."""
for stream in (sys.stdout, sys.stderr):
reconfigure = getattr(stream, "reconfigure", None)
if reconfigure is not None:
try:
reconfigure(encoding="utf-8", errors="replace")
except (ValueError, OSError):
pass
def main() -> int:
_force_utf8_stdio()
ap = argparse.ArgumentParser(description="Render an OKF bundle as a self-contained HTML graph.")
ap.add_argument("bundle", type=Path)
ap.add_argument("-o", "--out", type=Path, default=None)
ap.add_argument("-t", "--title", default=None, help="graph title (default: parent/bundle dir name)")
ap.add_argument("-l", "--link", default=None, help="optional source URL shown in the header")
ap.add_argument("--layout", default=None,
choices=["cose", "concentric", "breadthfirst", "circle", "grid"],
help=f"initial graph layout (default: cose, or concentric above {AUTO_COSE_MAX} "
"concepts — force layout freezes the page on large bundles)")
ap.add_argument("--max-nodes", type=int, default=None,
help="refuse to render bundles with more concepts than this (useful in CI)")
ap.add_argument("--og-image", default=None,
help="absolute URL for the social-preview image (og:image / twitter:image)")
args = ap.parse_args()
if not args.bundle.is_dir():
print(f"error: {args.bundle} is not a directory", file=sys.stderr)
return 2
out = args.out or (args.bundle / "viz.html")
n, e = render(args.bundle, out, title=args.title, link=args.link, layout=args.layout,
og_image=args.og_image, max_nodes=args.max_nodes)
print(f"rendered {n} concepts, {e} links -> {out}")
return 0
if __name__ == "__main__":
raise SystemExit(main())