fix agent definitions
This commit is contained in:
m3tam3re committed
1 parent
ca65491831
commit
b8798db398
18 files changed
+128
-4475
No files matched your search
-338
@@ -1,338 +0,0 @@
|
||||
---
|
||||
name: backfill
|
||||
description: >-
|
||||
Reconstruct an OKF bundle by event-sourcing a repository's history (git log
|
||||
and Claude session transcripts). Use when creating an `.okf/` bundle for an
|
||||
existing repository that predates this skill, or when resuming an interrupted
|
||||
backfill session. Triggers on: "reconstruct the OKF bundle", "backfill the
|
||||
knowledge bundle", "event-source the history".
|
||||
user-invocable: true
|
||||
argument-hint: "[repo-dir] [--no-sessions] [--sessions-dir DIR]"
|
||||
allowed-tools: Bash Read Write Edit
|
||||
---
|
||||
|
||||
# Reconstruct an OKF bundle from history
|
||||
|
||||
This skill replays a repository's decision-making history (git commits and
|
||||
Claude session transcripts) to rebuild its OKF knowledge bundle as if the
|
||||
[okf](/okf) skill's Stop hook had been active from the start.
|
||||
|
||||
The extraction layer is **deterministic** (same repo → byte-identical events);
|
||||
the replay layer is an LLM loop, **replayable and auditable but not
|
||||
byte-identical** (timestamps, summaries change per run). See the event schema
|
||||
(§1) and skip rules (§3).
|
||||
|
||||
## 1. Event schema
|
||||
|
||||
Git commits and session turns become events (JSONL, one per line, sorted by
|
||||
timestamp then source):
|
||||
|
||||
```json
|
||||
{"id":"git:<sha>","source":"git","ts":"<ISO8601 Z>","sha":"...","author":"...","subject":"...","body":"...","files":[{"path":"...","add":N,"del":N}]}
|
||||
{"id":"session:<file>:<lineno>","source":"session","ts":"<ISO8601 Z>","user":"...","outcome":"...","title":"...","branch":"...","skip":"..."}
|
||||
```
|
||||
|
||||
- **Git events** come from `git log --first-parent --numstat`.
|
||||
- **Session events** pair each user message with the **last** assistant text
|
||||
block of that turn (the wrap-up), extracted from
|
||||
`~/.claude/projects/<repo-slug>/*.jsonl` and worktree subdirs.
|
||||
- **Timestamps** are normalized to UTC ISO 8601 strings ending in `Z`.
|
||||
- **Skip field** (optional, added by extraction): marks low-signal events —
|
||||
see §3, never overridden by replay.
|
||||
|
||||
## 2. Protocol: Preflight → Extract → Bootstrap → Replay (Map+Reduce) → Finalize
|
||||
|
||||
### Preflight: check if bundle can be rebuilt
|
||||
|
||||
```bash
|
||||
if [ -d <repo>/.okf ] && ! [ -f <repo>/.okf/.backfill-state.json ]; then
|
||||
echo "ERROR: .okf/ exists but is incomplete. Delete it to rebuild from scratch, or pass --resume to continue from the last checkpoint."
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
If `.backfill-state.json` exists, resume from the cursor; otherwise, fresh bootstrap.
|
||||
|
||||
### Extract: generate event stream
|
||||
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_backfill_events.py" <repo-dir> \
|
||||
--out events.jsonl \
|
||||
[--no-sessions] [--sessions-dir ~/.claude/projects] \
|
||||
[--max-text 2000] [--skip-globs "vendor/**"]
|
||||
```
|
||||
|
||||
Write `events.jsonl` to the scratchpad (never committed). Report:
|
||||
- Total events extracted, per-source counts, and per-rule skip counts.
|
||||
|
||||
### Bootstrap: initialize bundle (fresh only)
|
||||
|
||||
Create `.okf/` with:
|
||||
|
||||
```bash
|
||||
mkdir -p <repo>/.okf
|
||||
echo 'okf_version: "0.2"' > <repo>/.okf/index.md
|
||||
echo '# Update Log' > <repo>/.okf/log.md
|
||||
```
|
||||
|
||||
Cursor (`.okf/.backfill-state.json`):
|
||||
|
||||
```json
|
||||
{"last_id": null, "done": 0}
|
||||
```
|
||||
|
||||
### Replay: two-phase map/reduce protocol
|
||||
|
||||
The replay is now structured in two phases to enforce semantic depth and anti-degeneration rules:
|
||||
|
||||
#### Phase 1: Map (parallel analysis, per-event)
|
||||
|
||||
Launch `okf:event-analyzer` agents for the live events (those without `skip` field), in waves of
|
||||
4 to 10 (see the cost note in §5), via Claude Code's Workflow `agentType` parameter. Each analyzer
|
||||
receives its event ids, the `events.jsonl` path, the repo path, the output directory and the
|
||||
emitter path, and:
|
||||
- Fetches its own event JSON with `jq` (the orchestrator dispatches ids, never content)
|
||||
- Reads git evidence only through the capped diff emitter (§6); session events need no other call
|
||||
- Writes `analyses/<event-id>.md` (event id with `:` sanitized to `-`) to scratchpad, with a
|
||||
`truncated` flag copied from the emitter's last line
|
||||
- Replies with one line of counts per event; the analysis never travels back in the reply
|
||||
- Never touches `.okf/`
|
||||
- Is resumable: skip events already in `analyses/`
|
||||
|
||||
**Dispatch rule** (deterministic, from `events.jsonl`, no content read): a live event is
|
||||
*small* when it is a session turn or a commit whose numstat totals at most 60 changed lines.
|
||||
Small events are grouped chronologically, eight per analyzer call; large commits go one per
|
||||
call. Measured on this repository (`benchmark/map-tier/RESULTS.md`): 57 calls instead of 147,
|
||||
a third off the map phase at the same tier, no loss on any per-event metric and no
|
||||
cross-event contamination.
|
||||
|
||||
```bash
|
||||
jq -c 'select(.skip==null) | {id, small: (.source=="session" or (([.files[]?|.add+.del]|add)//0) <= 60)}' events.jsonl
|
||||
```
|
||||
|
||||
**Host-agnostic fallback** (if no Workflow support): spawn generic subagents with the same
|
||||
system prompt as `agents/event-analyzer.md`, one per event, collecting analyses to scratchpad.
|
||||
|
||||
#### Phase 2: Reduce (sequential folding, one agent)
|
||||
|
||||
Run a single `okf:bundle-weaver` agent that:
|
||||
- Reads all analyses in chronological order
|
||||
- Folds them into `.okf/`, updating or creating concepts
|
||||
- Enforces anti-degeneration rules (§4)
|
||||
- Adds a dated `log.md` bullet for each lifecycle event only (a concept created, deprecated or
|
||||
superseded); the "why" of a routine update goes into the concept body
|
||||
- Manages cursor (`.okf/.backfill-state.json`) for resume capability
|
||||
- Is the only actor that writes `.okf/`
|
||||
- Replies with one line of counts (folded, created, updated, bullets, conflicts, superseded,
|
||||
truncated inputs); the bundle never travels back in the reply
|
||||
|
||||
**Resume behavior:** both phases support resumption. Phase 1 skips already-analyzed event ids;
|
||||
Phase 2 restarts from `last_id` in the cursor.
|
||||
|
||||
### Domain priming (advisory)
|
||||
|
||||
Before starting the map phase, read the repository's README and directory structure to sketch
|
||||
a candidate taxonomy of concepts (skills, integrations, decisions, etc.). This priming is
|
||||
*advisory only* — it helps the analyzer emit better candidate names. The *guarantee* that
|
||||
no events are lost comes from the deterministic coverage check in Finalize (§5), not from
|
||||
priming; analyses without candidate names still flow to the weaver.
|
||||
|
||||
### Context hygiene (orchestrator)
|
||||
|
||||
The orchestrator handles ids and counts, never content. Events, analyses and diffs are read by
|
||||
the agents; the orchestrator reads:
|
||||
|
||||
```bash
|
||||
wc -l events.jsonl # total events
|
||||
jq -r 'select(.skip==null) | .id' events.jsonl # live ids to dispatch
|
||||
ls analyses | wc -l # analyses written
|
||||
```
|
||||
|
||||
Never `cat events.jsonl` or open an analysis from the orchestrator: whatever enters its context
|
||||
is billed at the frontier rate on every following turn, and the routing exists to prevent that.
|
||||
|
||||
### Finalize: validate and clean up
|
||||
|
||||
1. **Write `index.md` per directory by hand** (do NOT run `okf_init.py` — it
|
||||
would scaffold placeholder files): one `# Section` per directory, one
|
||||
bullet per concept, `* [Title](file.md) - description` taken from each
|
||||
concept's frontmatter (SPEC §8). The root `index.md` keeps its
|
||||
`okf_version: "0.2"` frontmatter and links every subdirectory.
|
||||
|
||||
2. **Add final log entry** (today's date, appended to existing dated section if present):
|
||||
```
|
||||
## 2026-09-01
|
||||
|
||||
- **Backfill**: reconstructed from N events by okf-backfill/0.9.5
|
||||
```
|
||||
|
||||
3. **Delete cursor:**
|
||||
```bash
|
||||
rm -f .okf/.backfill-state.json
|
||||
```
|
||||
|
||||
4. **Run self-checks for anti-degeneration rules:**
|
||||
```bash
|
||||
# No concept filenames derived from change type
|
||||
! find .okf -name "*.md" | grep -E "merge-pull-request|(^|/)(feat|fix|chore|docs)[:-]"
|
||||
|
||||
# No non-kebab-case characters in filenames
|
||||
! find .okf -name "*.md" | grep -E "[^a-z0-9/.-]"
|
||||
|
||||
# No identical consecutive log bullets
|
||||
awk 'p==$0 && /^- / {exit 1} {p=$0}' .okf/log.md
|
||||
|
||||
# Every concept declares its lifecycle (absent status reads as `stable`, SPEC §5.4)
|
||||
[ -z "$(grep -rL '^status:' --include='*.md' .okf | grep -v '/index.md$\|/log.md$')" ]
|
||||
```
|
||||
These are lexical guards; they do not catch a bundle that asserts a superseded state. That
|
||||
check is semantic and belongs to the weaver, at fold time, when it has both the analysis and
|
||||
the earlier concepts on disk (§4 rule 5).
|
||||
|
||||
5. **Run coverage check to guarantee every event is mapped:**
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_backfill_events.py" \
|
||||
--check-coverage events.jsonl .okf
|
||||
```
|
||||
Exit code 0 means all live events (non-skipped) appear in the bundle's `sources` or log.
|
||||
Exit code 1 with a list of unmapped ids means the weaver skipped some events; fix and re-run.
|
||||
|
||||
6. **Validate bundle schema:**
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/../validate/scripts/okf_validate.py" .okf --strict
|
||||
```
|
||||
Fix every error before finishing.
|
||||
|
||||
7. **Report** the resulting bundle:
|
||||
- Concept count per directory
|
||||
- Log entry samples (first and last)
|
||||
- Coverage check result (all events mapped)
|
||||
- Validation result (pass/fail, warnings)
|
||||
- Superseded resolutions (the weaver's `superseded` count) and deprecated concepts:
|
||||
`grep -rl '^status: deprecated' --include='*.md' .okf | wc -l`
|
||||
- Agents spawned per phase (analyzers, weaver invocations) and, when the host reports it,
|
||||
tokens per phase
|
||||
- Truncated analyses: `grep -l '^truncated: true' analyses/*.md | wc -l`, next to the
|
||||
deterministic estimate of git events whose full diff exceeds the default cap:
|
||||
`jq -r 'select(.skip==null and .source=="git") | [.files[]|.add+.del] | add' events.jsonl | awk '$1>300' | wc -l`
|
||||
|
||||
## 3. Skip rules
|
||||
|
||||
Events matching a rule below are marked `skip: <rule-id>` and skipped during
|
||||
replay (cursor advances, no log entry):
|
||||
|
||||
| Rule | Condition | Rationale |
|
||||
|------|-----------|-----------|
|
||||
| `paths-only-generated` | Git commit touches ONLY lockfiles or generated code (vendor/, node_modules/, dist/, *.min.*, *.lock, etc.) | Noise: build byproducts, no knowledge content |
|
||||
| `merge-no-files` | Git merge commit with no file changes (first-parent only) | Housekeeping |
|
||||
| `session-command-noise` | Session user message is a slash command or <20 chars and has no useful outcome | Ephemeral UI/chatter |
|
||||
|
||||
Extend the first rule with `--skip-globs "extra/**"` for repo-specific patterns.
|
||||
|
||||
## 4. Anti-degeneration rules
|
||||
|
||||
The analyzer and weaver enforce these rules to prevent the bundle from devolving into
|
||||
a mechanical listing of commits or a taxonomy-by-accident:
|
||||
|
||||
1. **Concept names describe domain entities, not changes.** Forbidden:
|
||||
- Filenames derived from git subject: `merge-pull-request-#2-....md`, `feat:-add-docs-skill.md`, `fix:-handle-edge-case.md`
|
||||
- Bare action words: `update.md`, `fix.md`, `add-feature.md`
|
||||
- Non-kebab-case: spaces, underscores, capitals, special characters (allowed: `[a-z0-9-]` only)
|
||||
|
||||
Example: a commit with subject "feat: add presales pipeline" touches a domain concept.
|
||||
The *concept* is named `presales-pipeline.md` (the entity), not `feat:-add-presales-pipeline.md`
|
||||
(the change). The change history lives in frontmatter `sources` and the body's `## History`;
|
||||
`log.md` holds only lifecycle events.
|
||||
|
||||
2. **Prefer update over create.** Every event is analyzed for which concepts it touches; if
|
||||
a concept already exists and the analysis fits, update its `sources` and body. Only create
|
||||
a new concept if the analysis reveals a distinct domain entity not yet captured.
|
||||
|
||||
3. **Log bullets are lifecycle events, and explain intent.** Only a concept created,
|
||||
deprecated or superseded gets a bullet; a routine update does not. A bullet answers "why?"
|
||||
from the commit body or session outcome, never just re-reads the subject:
|
||||
- Bad: `- Added presales-pipeline.md feature`
|
||||
- Good: `- **Creation**: [Presales pipeline](...), formalized to clarify handoff points`
|
||||
|
||||
4. **No identical consecutive bullets.** A run of same-concept bullets means updates leaked
|
||||
into the log:
|
||||
```
|
||||
- Feature X update
|
||||
- Feature X update
|
||||
- Feature X update
|
||||
```
|
||||
Instead: no bullet at all; the updates live in the concept's body and `sources`.
|
||||
|
||||
5. **A reversal is not growth.** A replay walks the history forward, so a later event routinely
|
||||
overturns an earlier one (a phase dropped, a limit changed, a target downgraded). "Prefer
|
||||
update over create" is a merge rule and says nothing about this: applied alone it leaves the
|
||||
bundle asserting both states in the present tense, in one concept or in two. The weaver greps
|
||||
the whole bundle before writing and, on contradiction, replaces rather than appends: the body
|
||||
states what holds today, the previous position drops to a dated line under `## History`, and
|
||||
an outdated concept that a newer one supersedes gets `status: deprecated` plus a link to its
|
||||
successor. A deprecated concept keeps its `sources`, so coverage stays green. None of the
|
||||
guards in Finalize can catch this — they are lexical, and a superseded claim is well-formed.
|
||||
|
||||
6. **Reconstructed concepts are `draft`, never `stable`.** SPEC §5.4 reads an absent `status` as
|
||||
`stable` ("ready for consumption"); a bundle nobody has reread is `draft` ("not yet reviewed;
|
||||
possibly incomplete"). The weaver writes `status:` on every concept.
|
||||
|
||||
## 5. Implementation notes
|
||||
|
||||
- **Determinism**: extraction is byte-identical; replay is not (time, LLM variance).
|
||||
Both are auditable: events.jsonl is deterministic, map analyses are stored and resumable,
|
||||
reduce loop is human-readable and cursor-backed.
|
||||
- **Privacy**: events.jsonl goes to scratchpad, never committed. Session turn text is
|
||||
truncated (head+tail) before extraction.
|
||||
- **Cost note**: deep replay reads one capped diff per git commit (§6) to extract
|
||||
rationale. For histories >500 events, consider splitting into sub-ranges and
|
||||
replaying sequentially, or use `--skip-globs` to exclude low-signal paths (e.g.,
|
||||
vendored dependencies, generated code). Map phase is parallel in waves of 4 to 10: the
|
||||
gate benchmark lost 260 of 299 runs to a high-concurrency mass failure and finished at
|
||||
concurrency 4 (`benchmark/gate/RESULTS.md`). Reduce is sequential but much cheaper
|
||||
(concepts already analyzed).
|
||||
- **Future sources** (not implemented): GitHub PR/issue text, release notes, CI/deploy logs
|
||||
— all optional post-MVP. **Lore protocol** (arXiv 2603.15566): on repos that adopt git
|
||||
trailers (structured decision metadata), the extracted event's `body` already contains
|
||||
trailers in a machine-readable format. The analyzer and weaver should use trailers as
|
||||
the primary source of "why" (overriding generic inference from diff content).
|
||||
- **Trust metadata**: `generated.by` is the backfill agent (`okf-backfill/0.9.5`), not
|
||||
claimed as human-reviewed (`human:...`); concepts are correctly `unverified` (SPEC §5.3) and
|
||||
`status: draft` (§5.4) — the two axes are independent: trust is who says it, lifecycle is
|
||||
whether it has been reread. Writing `status` is not optional here: absent, §5.4 reads it as
|
||||
`stable`, so a bundle nobody has reread would declare itself ready for consumption.
|
||||
Map-phase analyses are working artifacts (stored for auditability during reduce); the
|
||||
weaver's output is the canonical bundle.
|
||||
|
||||
## 6. Capped diff emitter
|
||||
|
||||
Analyzers never read a raw `git show`: the harness cuts long tool output blindly (mid-hunk,
|
||||
character-based, host-dependent) and a cheap worker may not notice the cut. The extractor reads
|
||||
the whole diff instead and emits a deterministic sample:
|
||||
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_backfill_events.py" <repo-dir> --show <sha> \
|
||||
[--only <path>] [--max-diff-lines 300] [--per-file-lines 120] [--max-line-chars 400] \
|
||||
[--skip-globs "vendor/**"]
|
||||
```
|
||||
|
||||
- Diff against the first parent, so merge commits agree with the extracted numstat.
|
||||
- The stat is always complete: entities survive any cap.
|
||||
- Patches are capped per file (breadth over depth) and cut at hunk or file boundaries, with a
|
||||
bracketed marker at every cut; lines longer than `--max-line-chars` are shortened with a
|
||||
marker (a generated one-line file is "1 line" but can be hundreds of KB).
|
||||
- Patches of generated files (lockfiles, `vendor/**`, `--skip-globs`) are omitted even inside
|
||||
mixed commits; their stat line stays.
|
||||
- The last line is fixed-form: `[diff: shown=X total=Y files_shown=A files_total=B truncated=true|false]`.
|
||||
The analyzer copies `truncated` into its frontmatter; finalize reports the count.
|
||||
- `--only <path>` is the one permitted follow-up when a truncated diff hides the rationale:
|
||||
same cap logic, one file.
|
||||
|
||||
Defaults keep one call under the harness limits with margin; tune per repo with the flags.
|
||||
|
||||
The analyzer's tier is configuration, not infrastructure: `agents/event-analyzer.md` carries
|
||||
`model` and `effort` in its frontmatter. Fork the file to retier the map phase; the skill
|
||||
resolves the agent by name and nothing else changes. The default is `haiku` since the
|
||||
map-tier benchmark (`benchmark/map-tier/RESULTS.md`): parity with `sonnet` on commits at
|
||||
under half the map cost, on the condition that the two explicit instructions in the agent
|
||||
file stay (which summary line the flag copies; claims report intent as intent).
|
||||
File diff suppressed because it is too large.
Load diff
Vendored
-159
@@ -1,159 +0,0 @@
|
||||
---
|
||||
name: okf
|
||||
description: >-
|
||||
Author, maintain, and consume Open Knowledge Format (OKF) knowledge bundles —
|
||||
portable markdown + YAML frontmatter that both humans and agents read. Use when
|
||||
capturing project knowledge (services, APIs, schemas, metrics, runbooks,
|
||||
decisions) into an OKF bundle, when updating one after code or docs change, or
|
||||
when a repository contains an `.okf/` (or other OKF) bundle that should inform
|
||||
the task. Triggers on: "document this in OKF", "update the knowledge bundle",
|
||||
"capture this as a concept", or any work in a repo that has an OKF bundle.
|
||||
user-invocable: true
|
||||
argument-hint: "[produce|maintain|consume] [path]"
|
||||
allowed-tools: Read Write Edit Grep Glob Bash
|
||||
---
|
||||
|
||||
# Open Knowledge Format (OKF) skill
|
||||
|
||||
OKF represents knowledge as a directory of markdown files with YAML frontmatter.
|
||||
It is minimal by design: no schema registry, no runtime, no SDK. Your job is to
|
||||
produce, maintain, and consume OKF bundles **conformant with the spec**, not your
|
||||
memory of it.
|
||||
|
||||
**Always read the canonical spec before non-trivial work:**
|
||||
[reference/SPEC.md](reference/SPEC.md). It is the verbatim OKF v0.2 specification
|
||||
and the source of truth for every rule below.
|
||||
|
||||
## The one hard rule
|
||||
|
||||
A bundle is conformant (§11) iff: every non-reserved `.md` file has a parseable
|
||||
YAML frontmatter block, and every such block has a **non-empty `type`** field.
|
||||
Everything else is soft guidance. Consumers MUST tolerate missing optional
|
||||
fields, unknown types, and broken links — never reject a bundle over them.
|
||||
|
||||
## Conventions to apply
|
||||
|
||||
- **One concept = one file.** The file path (minus `.md`) is the concept ID.
|
||||
- **Frontmatter:** `type` is required. Add `title`, `description`, `tags` when
|
||||
they aid consumption; add `resource` (a canonical URI) only for concepts bound
|
||||
to a real asset — omit it for abstract concepts.
|
||||
- **Body:** prefer structural markdown (headings, tables, lists, fenced code).
|
||||
Conventional headings: `# Schema`, `# Examples`, `# Computation`.
|
||||
- **Cross-links:** standard markdown links; prefer absolute bundle-relative
|
||||
form (`/services/auth-api.md`). A link asserts a relationship; its *kind* lives
|
||||
in the surrounding prose, not the link.
|
||||
- **Reserved files:** `index.md` (directory listing, no frontmatter, except the
|
||||
bundle-root index, which may carry `okf_version` and this plugin's
|
||||
`upkeep: enforced` opt-in flag) and `log.md` (ISO-dated history, newest
|
||||
first). Never use these names for concepts.
|
||||
- **What goes in `log.md`:** lifecycle events only, written in the bundle-root
|
||||
`log.md`: a bundle **created**, a concept **deprecated** or retired, one
|
||||
concept **superseding** another, concepts **regenerated** after their source
|
||||
changed, a **verification** pass. Routine edits to a concept do not get an
|
||||
entry: its `generated` and git already record them. A log is append-only, so
|
||||
never rewrite or drop an existing entry.
|
||||
|
||||
## The v0.2 families (all optional, all worth filling)
|
||||
|
||||
- **Trust (§5.2):** `generated: { by, at }` — who produced the current content
|
||||
and when. `verified: [{ by, at }]` — who confirmed it since (a bare mapping is
|
||||
one entry). Write `by` in the **actor convention** (§7): `<producer>/<version>`
|
||||
for an agent, `human:<id>` for a person, `process:<id>` for an automated job.
|
||||
Use `human:` whenever a person authored or signed off — consumers key trust
|
||||
tiers off that prefix.
|
||||
- **Lifecycle (§5.4–5.5):** `status: draft|stable|deprecated` (absent means
|
||||
stable) and `stale_after`, an absolute ISO 8601 datetime (a bare date is
|
||||
tolerated), not a TTL.
|
||||
- **Provenance (§5.1):** `sources: [{ id, resource, title, author,
|
||||
usage_count, last_modified }]` plus a `usage_window: { from, to }` sibling of
|
||||
`sources` framing every `usage_count` (an entry may carry its own to override
|
||||
it); a `usage_count` without a window warns. `resource` is required per entry
|
||||
and may be a URL, a bundle path, or a scope descriptor. Attribute a specific
|
||||
claim with a markdown footnote whose label is the source's `id`:
|
||||
`…sharded daily.[^ga4-schema]` plus a `[^ga4-schema]: …` definition. The label
|
||||
is the join key, it must match a `sources[].id`.
|
||||
- **Attestation (§10):** a sanctioned computation is its own concept,
|
||||
`type: Attested Computation`, carrying `runtime` (required), `parameters`,
|
||||
`executor`, `attester`, and the computation itself under `# Computation` (or a
|
||||
`computation:` path). Concepts that need the value link to it. Never inline a
|
||||
number's SQL into the concept that narrates it.
|
||||
|
||||
**Reading a v0.1 bundle?** Two constructs were superseded (§13.1): `timestamp`
|
||||
is now `generated.at`, and a body `# Citations` list is now `sources`. Read both,
|
||||
write v0.2 — and when you touch a legacy concept in **maintain** mode, migrate
|
||||
its frontmatter as part of the edit. The validator warns on both.
|
||||
|
||||
Templates to copy: [concept](templates/concept.md), [index](templates/index.md),
|
||||
[log](templates/log.md).
|
||||
|
||||
## Default bundle location
|
||||
|
||||
Use `.okf/` at the repository root unless the project already uses another
|
||||
location. Commit it alongside the code it describes — knowledge as code.
|
||||
|
||||
## Modes
|
||||
|
||||
### produce — create or extend a bundle
|
||||
|
||||
**Starting a brand-new bundle?** Use the init fast-path instead of hand-writing
|
||||
the first files — it scaffolds a conformant `index.md`, `log.md`, and a
|
||||
`getting-started.md` concept with full recommended frontmatter in one shot:
|
||||
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_init.py" <target-dir> [--title "..."]
|
||||
```
|
||||
|
||||
It refuses to touch a directory that already has `.md` files unless `--force`
|
||||
is given. Then extend it:
|
||||
|
||||
1. Read [reference/SPEC.md](reference/SPEC.md).
|
||||
2. Pick the source(s): **code** (derive concepts from source, READMEs,
|
||||
docstrings, config), **docs/wiki** (distill pages into concepts, record the
|
||||
originals in `sources`), **manual** (decisions, playbooks, metrics).
|
||||
3. Choose a directory layout by domain (e.g. `services/`, `datasets/`,
|
||||
`decisions/`). One concept per file.
|
||||
4. Write each concept from [templates/concept.md](templates/concept.md): set a
|
||||
descriptive `type`, fill recommended fields, record `generated` and the
|
||||
`sources` you actually read, cross-link related concepts.
|
||||
5. Add/refresh `index.md` per directory (and `okf_version: "0.2"` in the root
|
||||
index). A new bundle gets a dated **Creation** entry in `log.md`. Extending
|
||||
an existing one does not.
|
||||
6. Validate (see below). Fix every error before finishing.
|
||||
|
||||
### maintain — keep a bundle in sync with reality
|
||||
1. Identify which concepts the change affects (search by `resource`, path, or
|
||||
topic). This bookkeeping is exactly what agents are good at — touch every
|
||||
affected file in one pass.
|
||||
2. Update the body and `generated.at` (with your own actor in `generated.by`);
|
||||
fix or add cross-links; create new concepts for new assets; mark removed
|
||||
assets `status: deprecated` and note the deprecation in `log.md` rather than
|
||||
silently deleting context. Facing a whole v0.1 bundle rather than a stray
|
||||
field? Do not hand-edit it — run the validator's `--migrate` once.
|
||||
3. Update the relevant `index.md` files. Append a dated `log.md` entry only if
|
||||
the change is a lifecycle event (see *What goes in `log.md`*).
|
||||
4. Validate.
|
||||
|
||||
### consume — use a bundle as context
|
||||
1. Read the bundle-root `index.md` first for progressive disclosure, then follow
|
||||
links only into the concepts relevant to the task.
|
||||
2. Weigh what you read: `status: draft`/`deprecated`, a `stale_after` already
|
||||
past, or no `verified` entry all mean "check before relying on this". Treat
|
||||
broken links as not-yet-written knowledge, not errors.
|
||||
3. Need a number an `Attested Computation` covers? Run *its* computation with
|
||||
values bound to the declared `parameters` — never write your own query.
|
||||
4. If you learn something durable while working, switch to **maintain** and
|
||||
write it back.
|
||||
|
||||
## Validation (do this before declaring done)
|
||||
|
||||
Never eyeball conformance — run the deterministic checker. Invoke the companion
|
||||
**`validate`** skill (`/okf:validate <bundle-dir> --strict`), which ships the
|
||||
checker. If that skill is not installed, run it directly:
|
||||
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/../validate/scripts/okf_validate.py" <bundle-dir> --strict
|
||||
```
|
||||
|
||||
Resolve every `ERROR` (hard §11 failures). Under `--strict` warnings also fail
|
||||
the run (exit 1), so clear them too; drop `--strict` (or use `--max-warnings N`)
|
||||
if you only want §11 conformance gated.
|
||||
-202
@@ -1,202 +0,0 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-1016
File diff suppressed because it is too large.
Load diff
-112
@@ -1,112 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["pyyaml>=6"]
|
||||
# ///
|
||||
"""Scaffold a conformant starter Open Knowledge Format (OKF) v0.2 bundle.
|
||||
|
||||
Creates a root `index.md` (frontmatter carries only `okf_version`), a `log.md`
|
||||
with a creation entry under today's ISO date heading, and one starter concept
|
||||
(`getting-started.md`) with the full set of recommended frontmatter fields.
|
||||
The scaffold is meant to be exemplary: `okf_validate.py --strict` should pass
|
||||
it with zero warnings.
|
||||
|
||||
Run: uv run scripts/okf_init.py <target-dir> [--title "..."] [--force]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
OKF_VERSION = "0.2"
|
||||
STARTER_CONCEPT = "getting-started.md"
|
||||
# Actor convention (§7): an automated process, not an agent with a version.
|
||||
ACTOR = "process:okf_init"
|
||||
|
||||
|
||||
def humanize(name: str) -> str:
|
||||
words = name.replace("-", " ").replace("_", " ").strip()
|
||||
return words.title() if words else "Untitled"
|
||||
|
||||
|
||||
def frontmatter(meta: dict) -> str:
|
||||
return "---\n" + yaml.safe_dump(meta, sort_keys=False, allow_unicode=True) + "---\n"
|
||||
|
||||
|
||||
def build_index(title: str) -> str:
|
||||
meta = frontmatter({"okf_version": OKF_VERSION})
|
||||
return (
|
||||
f"{meta}\n"
|
||||
f"# {title}\n\n"
|
||||
f"* [Getting started]({STARTER_CONCEPT}) - starting point for this bundle.\n"
|
||||
)
|
||||
|
||||
|
||||
def build_log(today: str, title: str) -> str:
|
||||
return (
|
||||
"# Update Log\n\n"
|
||||
f"## {today}\n"
|
||||
f"* **Creation**: Scaffolded the {title} bundle with `okf_init.py` — "
|
||||
f"see [getting started]({STARTER_CONCEPT}).\n"
|
||||
)
|
||||
|
||||
|
||||
def build_concept(title: str, now_iso: str) -> str:
|
||||
meta = frontmatter({
|
||||
"type": "Reference",
|
||||
"title": f"Getting started — {title}",
|
||||
"description": f"Starting point for the {title} OKF bundle.",
|
||||
"tags": ["getting-started"],
|
||||
"status": "stable",
|
||||
"generated": {"by": ACTOR, "at": now_iso},
|
||||
})
|
||||
return (
|
||||
f"{meta}\n"
|
||||
"# Overview\n\n"
|
||||
"This is the first concept in a freshly scaffolded OKF bundle. Replace "
|
||||
"it with real knowledge — one concept per file, cross-linked with "
|
||||
"standard markdown links — and keep `index.md` updated as you go. "
|
||||
"`log.md` records lifecycle events only: a concept created, "
|
||||
"deprecated, superseded, regenerated or verified.\n"
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=f"Scaffold a starter OKF v{OKF_VERSION} bundle.")
|
||||
ap.add_argument("target", type=Path, help="directory to create the bundle in")
|
||||
ap.add_argument("--title", default=None, help="bundle title (default: humanized target dir name)")
|
||||
ap.add_argument("--force", action="store_true", help="scaffold even if the target already has .md files")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.target.exists() and not args.target.is_dir():
|
||||
print(f"error: {args.target} exists and is not a directory", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
existing_md = list(args.target.rglob("*.md")) if args.target.is_dir() else []
|
||||
if existing_md and not args.force:
|
||||
print(f"refusing: {args.target} already contains .md files (pass --force to scaffold anyway)", file=sys.stderr)
|
||||
for p in sorted(existing_md)[:5]:
|
||||
print(f" {p.relative_to(args.target)}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
title = args.title or humanize(args.target.resolve().name)
|
||||
today = datetime.now(timezone.utc).date().isoformat()
|
||||
now_iso = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
args.target.mkdir(parents=True, exist_ok=True)
|
||||
(args.target / "index.md").write_text(build_index(title), encoding="utf-8")
|
||||
(args.target / "log.md").write_text(build_log(today, title), encoding="utf-8")
|
||||
(args.target / STARTER_CONCEPT).write_text(build_concept(title, now_iso), encoding="utf-8")
|
||||
|
||||
validator = Path(__file__).resolve().parents[2] / "validate" / "scripts" / "okf_validate.py"
|
||||
print(f"created OKF bundle scaffold at {args.target}")
|
||||
print(f"hint: validate it — uv run {validator} {args.target} --strict (or the validate skill)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
-38
@@ -1,38 +0,0 @@
|
||||
---
|
||||
type: <Concept type, e.g. Service, BigQuery Table, Metric, Playbook, Decision>
|
||||
title: <Human-readable display name>
|
||||
description: <Single sentence summarizing the concept.>
|
||||
resource: <Canonical URI of the underlying asset — omit for abstract concepts>
|
||||
tags: [<tag>, <tag>]
|
||||
status: stable # draft | stable | deprecated; absent means stable
|
||||
generated: { by: <actor>, at: <ISO 8601, e.g. 2026-06-14T10:00:00Z> }
|
||||
verified: { by: human:<id>, at: <ISO 8601> } # omit until someone confirms it
|
||||
stale_after: <YYYY-MM-DDTHH:MM:SSZ> # omit when the content does not expire
|
||||
sources: # what this was derived from; omit if nothing
|
||||
- id: <short-key>
|
||||
resource: <URL, bundle path, or scope descriptor>
|
||||
title: <Human-readable label>
|
||||
author: <actor> # optional credibility signal
|
||||
last_modified: <YYYY-MM-DDTHH:MM:SSZ> # when the source itself last changed
|
||||
---
|
||||
|
||||
<!--
|
||||
Actors (§7): `<producer>/<version>` for an agent, `human:<id>` for a person,
|
||||
`process:<id>` for an automated job. Use `human:` whenever a person authored
|
||||
or signed off — consumers key trust tiers off that prefix.
|
||||
-->
|
||||
|
||||
# Overview
|
||||
|
||||
<What this concept is and why it matters. Attribute a sourced claim with a
|
||||
footnote whose label is a `sources[].id`.[^short-key]>
|
||||
|
||||
# Schema
|
||||
|
||||
<Use for assets with fields/columns; otherwise replace with relevant sections.>
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| | | |
|
||||
|
||||
[^short-key]: <Human-readable label of the source>
|
||||
-8
@@ -1,8 +0,0 @@
|
||||
# <Directory / Group Heading>
|
||||
|
||||
* [<Title>](<relative-url>) - <short description from the concept's frontmatter>
|
||||
* [<Title>](<relative-url>) - <short description>
|
||||
|
||||
# <Another Group>
|
||||
|
||||
* [<Subdirectory>](<subdir>/) - <short description of the subdirectory>
|
||||
-8
@@ -1,8 +0,0 @@
|
||||
# Update Log
|
||||
|
||||
## <YYYY-MM-DD>
|
||||
* **Creation**: <what was created> — [<concept>](<bundle-relative-path>).
|
||||
* **Deprecation**: <what was retired, and what replaces it> — [<concept>](<bundle-relative-path>).
|
||||
* **Supersession**: [<new concept>](<bundle-relative-path>) supersedes [<old concept>](<bundle-relative-path>). Affirmed by <actor>.
|
||||
* **Regeneration**: <which concepts, after what source change> — [<concept>](<bundle-relative-path>).
|
||||
* **Verification**: <what was verified, by whom> — [<concept>](<bundle-relative-path>).
|
||||
-85
@@ -1,85 +0,0 @@
|
||||
---
|
||||
name: validate
|
||||
description: >-
|
||||
Check that an Open Knowledge Format (OKF) bundle is conformant with the v0.2
|
||||
spec (§11). Use when asked to validate, lint, or check an OKF bundle, or before
|
||||
committing changes to one. Runs a deterministic Python checker — not an
|
||||
eyeball pass. Also migrates a v0.1 bundle to v0.2 in place with `--migrate`.
|
||||
user-invocable: true
|
||||
argument-hint: "[bundle-dir] [--strict | --max-warnings N] [--migrate]"
|
||||
allowed-tools: Bash
|
||||
---
|
||||
|
||||
# Validate an OKF bundle
|
||||
|
||||
Run the deterministic conformance checker against the target bundle. Default to
|
||||
the project's `.okf/` directory when no path is given.
|
||||
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_validate.py" $ARGUMENTS
|
||||
```
|
||||
|
||||
If `uv` is unavailable, fall back to:
|
||||
|
||||
```bash
|
||||
python3 -m pip install --quiet pyyaml && \
|
||||
python3 "${CLAUDE_SKILL_DIR}/scripts/okf_validate.py" $ARGUMENTS
|
||||
```
|
||||
|
||||
`${CLAUDE_SKILL_DIR}` resolves whether this skill runs as part of the `okf`
|
||||
plugin or is installed standalone (e.g. via `npx skills add`), so the checker is
|
||||
always found alongside the skill.
|
||||
|
||||
Interpret the result:
|
||||
|
||||
- **ERROR** → a hard §11 conformance failure (no parseable frontmatter, a
|
||||
missing/empty `type`, or a file the checker cannot read as UTF-8). The bundle
|
||||
is non-conformant. Fix every one.
|
||||
|
||||
Only `index.md` and `log.md` are reserved: §11.1 makes *every* other `.md` in
|
||||
the tree a concept, convention files included. An `AGENTS.md` or `README.md`
|
||||
dropped next to a bundle is therefore an ERROR until it carries frontmatter with
|
||||
a `type` (`Document` reads well), which is the fix; a skip-list would put the
|
||||
checker out of conformance. Move the file above the bundle root if it is not
|
||||
knowledge.
|
||||
- **warn** → soft guidance (missing recommended field, non-ISO log date, broken
|
||||
cross-link, a malformed v0.2 family, a footnote naming no source, an actor
|
||||
that misses the §7 shapes, an Attested Computation missing `runtime`, a
|
||||
computation path that resolves nowhere, frontmatter in a reserved file (§8,
|
||||
§9), a root `index.md` carrying extra keys or declaring a different
|
||||
`okf_version` (§12)). Never blocks; broken links in particular are explicitly
|
||||
tolerated by the spec (§6.1). Fix when cheap.
|
||||
|
||||
One warning is worth more than the others: a §7 near-miss such as `Human:dana`
|
||||
or `human/dana`. §5.3 keys trust tiers off the exact lowercase `human:` prefix,
|
||||
so the concept silently reads as machine-confirmed when a person did review it.
|
||||
Fix that one on sight.
|
||||
|
||||
v0.1 bundles validate too: a legacy `timestamp` or `# Citations` section is
|
||||
reported as a warning naming its v0.2 replacement (`generated.at`, `sources`),
|
||||
never as an error (§13.1). Under `--strict` those warnings do fail the run —
|
||||
that is the migration nudge, and `--migrate` is the door.
|
||||
|
||||
## Migrating a v0.1 bundle
|
||||
|
||||
`--migrate` **rewrites the bundle in place** before validating — the one mode of
|
||||
this skill that is not read-only, so say what it will touch before running it on
|
||||
a bundle the user has not asked to migrate. It is textual (comments, key order
|
||||
and quoting survive) and idempotent.
|
||||
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_validate.py" .okf --migrate --strict
|
||||
```
|
||||
|
||||
It hoists `timestamp` to `generated: { by: process:okf-migrate, at }`, lifts a
|
||||
`# Citations` list into `sources`, and bumps `okf_version`. Two limits worth
|
||||
repeating to the user: `generated.by` cannot be recovered for pre-v0.2 content
|
||||
(hence the `process:` actor — the concept stays correctly `unverified` under
|
||||
§5.3), and per-claim `[^id]` attribution was never encoded in v0.1, so only the
|
||||
source list moves up.
|
||||
|
||||
## Exit codes
|
||||
|
||||
Non-zero if any error is present, or if warnings exceed the gate:
|
||||
`--strict` allows none, `--max-warnings N` allows N, the default allows any.
|
||||
Add `--json` for machine-readable output (useful in CI).
|
||||
@@ -1,573 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["pyyaml>=6"]
|
||||
# ///
|
||||
"""Deterministic conformance checker for Open Knowledge Format (OKF) v0.2.
|
||||
|
||||
Implements the §11 conformance rules verbatim:
|
||||
1. Every non-reserved `.md` file contains a parseable YAML frontmatter block.
|
||||
2. Every frontmatter block contains a non-empty `type` field.
|
||||
3. Reserved filenames (`index.md`, `log.md`) follow §8 / §9 when present.
|
||||
|
||||
Rules 1 and 2 are hard errors (a bundle that fails them is non-conformant).
|
||||
Everything else the spec marks as soft guidance: reported as warnings, never
|
||||
fatal unless `--strict` is given. In particular broken cross-links are NOT
|
||||
errors — the spec requires consumers to tolerate them (§6.1).
|
||||
|
||||
v0.1 bundles still validate: the two superseded constructs (`timestamp`, a body
|
||||
`# Citations` list) are read and reported as warnings pointing at their v0.2
|
||||
replacements (`generated.at`, `sources`), never as errors (§13.1). `--migrate`
|
||||
performs that rewrite in place, textually, so `--strict` has a door and not just
|
||||
a wall.
|
||||
|
||||
Run: uv run scripts/okf_validate.py <bundle-dir>
|
||||
[--strict | --max-warnings N] [--migrate] [--json]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
OKF_VERSION = "0.2"
|
||||
RESERVED = {"index.md", "log.md"}
|
||||
# `resource` is intentionally excluded: the spec (§4.1) states it is absent for
|
||||
# concepts describing abstract ideas, so flagging it produces false noise.
|
||||
RECOMMENDED = ("title", "description", "tags")
|
||||
STATUS_VALUES = {"draft", "stable", "deprecated"}
|
||||
ISO_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
|
||||
# RFC 3339, as the spec writes `generated.at` / `verified[].at` throughout, and
|
||||
# (since 0.9.6) `stale_after`, `sources[].last_modified` and `usage_window`. A
|
||||
# date-only value is tolerated: it is common in the wild and loses only precision.
|
||||
# PyYAML resolves an unquoted timestamp to a datetime whose str() separates with a
|
||||
# space, so both spellings have to pass.
|
||||
RFC3339 = re.compile(
|
||||
r"^\d{4}-\d{2}-\d{2}"
|
||||
r"(?:[Tt ]\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?:[Zz]|[+-]\d{2}:?\d{2})?)?$")
|
||||
# §7 gives three actor shapes, and §5.1 reuses them for `sources[].author`. The
|
||||
# spec's own example (`author: team:ga4-docs`) shows the `<prefix>:<id>` family is
|
||||
# open, so this is deliberately not a whitelist — it catches the one failure that
|
||||
# is silent and costly: a near-miss of `human:`, which §5.3 keys trust tiers off.
|
||||
# `human/dana` or `Human:dana` reads as an agent and downgrades the tier with no
|
||||
# other symptom.
|
||||
ACTOR_SHAPE = re.compile(r"^(?:[^\s:/]+:\S+|\S+/\S+)$")
|
||||
# Anything that opens with human/process in any case — `Human:dana` is the worst
|
||||
# of these, since it satisfies the generic `<prefix>:<id>` shape and so looks
|
||||
# entirely well-formed while §5.3 reads it as an agent. The lookahead keeps
|
||||
# `humanoid_agent/v1` out: only a non-identifier boundary makes it a near-miss.
|
||||
ACTOR_HUMANISH = re.compile(r"^(?:human|process)(?![A-Za-z0-9_])", re.I)
|
||||
FENCE = re.compile(r"^(```|~~~)")
|
||||
# markdown link target capture: [text](target) — ignores images ![...]
|
||||
LINK = re.compile(r"(?<!\!)\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
|
||||
CITATIONS = re.compile(r"^#{1,6}[ \t]+Citations[ \t]*$", re.M)
|
||||
# a footnote *reference* in the body; the definition line `[^id]: …` matches too,
|
||||
# which is harmless — both carry the same label, the join key into `sources`.
|
||||
FOOTNOTE = re.compile(r"\[\^([^\]\s]+)\]")
|
||||
|
||||
# --migrate only. `timestamp` is a top-level key (§4.1), so anchor at column 0 —
|
||||
# an indented one belongs to something else and must not be hoisted.
|
||||
TIMESTAMP = re.compile(r"^timestamp:[ \t]*(.+?)[ \t]*$", re.M)
|
||||
HEADING = re.compile(r"^#{1,6}[ \t]", re.M)
|
||||
# same shape as LINK above, plus the link text — a citation written with a title
|
||||
# attribute, `[Src](url "Title")`, must not be dropped: the section is deleted
|
||||
# wholesale once anything parses, so a miss here is silent data loss.
|
||||
MD_LINK = re.compile(r"(?<!\!)\[([^\]]+)\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
|
||||
OKF_VERSION_LINE = re.compile(r"^(okf_version:[ \t]*)[\"']?0\.1[\"']?[ \t]*$", re.M)
|
||||
# True, and it keeps the concept correctly `unverified` under §5.3: who wrote the
|
||||
# content before `generated` existed is not recoverable, and inventing a `human:`
|
||||
# actor would fake a review that never happened.
|
||||
MIGRATE_ACTOR = "process:okf-migrate"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Report:
|
||||
errors: list[str] = field(default_factory=list)
|
||||
warnings: list[str] = field(default_factory=list)
|
||||
concepts: int = 0
|
||||
indexes: int = 0
|
||||
logs: int = 0
|
||||
|
||||
def err(self, rel: str, msg: str) -> None:
|
||||
self.errors.append(f"{rel}: {msg}")
|
||||
|
||||
def warn(self, rel: str, msg: str) -> None:
|
||||
self.warnings.append(f"{rel}: {msg}")
|
||||
|
||||
|
||||
def split_frontmatter(text: str) -> tuple[str | None, str]:
|
||||
"""Return (raw_yaml, body). raw_yaml is None when no frontmatter block."""
|
||||
if not text.startswith("---"):
|
||||
return None, text
|
||||
# first line must be exactly '---' (allow trailing whitespace / BOM already stripped)
|
||||
lines = text.splitlines(keepends=True)
|
||||
if lines[0].strip() != "---":
|
||||
return None, text
|
||||
for i in range(1, len(lines)):
|
||||
if lines[i].strip() == "---":
|
||||
return "".join(lines[1:i]), "".join(lines[i + 1 :])
|
||||
return None, text # unterminated block → treated as absent
|
||||
|
||||
|
||||
def _read_text(path: Path, rel: str, report: Report) -> str | None:
|
||||
"""Read a concept/index/log file, reporting (not raising) on a bad file.
|
||||
|
||||
A binary or non-UTF-8 `.md` file must fail as a per-file error, not crash
|
||||
the whole validation run.
|
||||
"""
|
||||
try:
|
||||
return path.read_text(encoding="utf-8").lstrip("")
|
||||
except (UnicodeDecodeError, OSError) as exc:
|
||||
report.err(rel, f"cannot read file: {exc}")
|
||||
return None
|
||||
|
||||
|
||||
def check_computation(meta: dict, path: Path, bundle: Path, rel: str, report: Report) -> None:
|
||||
"""§10.2 — the Attested Computation contract, including its path-valued fields.
|
||||
|
||||
`computation`, `executor.resource` and `attester.resource` name files (§6.2).
|
||||
They are exactly the pointer that rots: the concept keeps validating while the
|
||||
script it names moves or disappears. A target outside the bundle is legitimate
|
||||
and unverifiable from here, so only bundle-relative paths are resolved.
|
||||
"""
|
||||
if not str(meta.get("runtime", "")).strip():
|
||||
report.warn(rel, "§10.2 an Attested Computation concept requires `runtime`")
|
||||
fields = [("computation", meta.get("computation"))]
|
||||
for key in ("executor", "attester"):
|
||||
block = meta.get(key)
|
||||
if isinstance(block, dict):
|
||||
fields.append((f"{key}.resource", block.get("resource")))
|
||||
for where, target in fields:
|
||||
if target is None:
|
||||
continue
|
||||
t = str(target).strip()
|
||||
if not t or re.match(r"^[a-z][a-z0-9+.-]*://", t) or t.startswith("mailto:"):
|
||||
continue
|
||||
candidate = (bundle / t.lstrip("/")) if t.startswith("/") else (path.parent / t)
|
||||
try:
|
||||
inside = candidate.resolve().is_relative_to(bundle.resolve())
|
||||
except OSError:
|
||||
inside = False
|
||||
if inside and not candidate.exists():
|
||||
report.warn(rel, f"§6.2 `{where}` points at `{t}`, which does not exist")
|
||||
|
||||
|
||||
def check_concept(path: Path, rel: str, bundle: Path, report: Report) -> None:
|
||||
report.concepts += 1
|
||||
text = _read_text(path, rel, report)
|
||||
if text is None:
|
||||
return
|
||||
raw, body = split_frontmatter(text)
|
||||
if raw is None:
|
||||
report.err(rel, "§11.1 no parseable YAML frontmatter block "
|
||||
"(every non-reserved `.md` in a bundle is a concept: add "
|
||||
"frontmatter with a `type` field, or move the file outside "
|
||||
"the bundle)")
|
||||
return
|
||||
try:
|
||||
meta = yaml.safe_load(raw)
|
||||
except yaml.YAMLError as exc:
|
||||
report.err(rel, f"§11.1 frontmatter is not valid YAML: {exc}".replace("\n", " "))
|
||||
return
|
||||
if not isinstance(meta, dict):
|
||||
report.err(rel, "§11.1 frontmatter must be a YAML mapping")
|
||||
return
|
||||
type_val = meta.get("type")
|
||||
if not (isinstance(type_val, str) and type_val.strip()):
|
||||
report.err(rel, "§11.2 missing or empty required `type` field")
|
||||
for key in RECOMMENDED:
|
||||
if key not in meta:
|
||||
report.warn(rel, f"recommended field `{key}` is absent (§4.1)")
|
||||
check_trust(meta, rel, report)
|
||||
check_lifecycle(meta, rel, report)
|
||||
check_sources(meta, body, rel, report)
|
||||
if isinstance(type_val, str) and type_val.strip() == "Attested Computation":
|
||||
check_computation(meta, path, bundle, rel, report)
|
||||
|
||||
|
||||
def check_actor(value, where: str, rel: str, report: Report) -> None:
|
||||
"""§7 — the actor convention shared by `generated.by`, `verified[].by` and
|
||||
`sources[].author`."""
|
||||
actor = str(value).strip()
|
||||
if not actor:
|
||||
return
|
||||
if ACTOR_HUMANISH.match(actor) and not actor.startswith(("human:", "process:")):
|
||||
report.warn(rel, f"§7 `{where}` `{actor}` is a near-miss of "
|
||||
f"`human:`/`process:` — §5.3 keys trust tiers off that exact "
|
||||
f"lowercase prefix, so this silently reads as an agent")
|
||||
elif not ACTOR_SHAPE.match(actor):
|
||||
report.warn(rel, f"§7 `{where}` `{actor}` matches no actor shape "
|
||||
f"(`human:<id>`, `process:<id>`, or `<producer>/<version>`)")
|
||||
|
||||
|
||||
def check_instant(value, where: str, rel: str, report: Report) -> None:
|
||||
"""§5.2 — `at` values are RFC 3339; a date-only value is tolerated."""
|
||||
if value is not None and not RFC3339.match(str(value).strip()):
|
||||
report.warn(rel, f"§5.2 `{where}` `{value}` is not an RFC 3339 timestamp")
|
||||
|
||||
|
||||
def check_trust(meta: dict, rel: str, report: Report) -> None:
|
||||
"""§5.2 — `generated` / `verified`, with the v0.1 `timestamp` fallback."""
|
||||
gen = meta.get("generated")
|
||||
if gen is None:
|
||||
if "timestamp" in meta:
|
||||
report.warn(rel, "legacy v0.1 `timestamp`: v0.2 records the last content "
|
||||
"change as `generated: {by, at}` (§13.1)")
|
||||
else:
|
||||
report.warn(rel, "recommended field `generated` is absent (§5.2)")
|
||||
elif not isinstance(gen, dict):
|
||||
report.warn(rel, "§5.2 `generated` must be a mapping with `by` and `at`")
|
||||
elif not str(gen.get("by", "")).strip():
|
||||
report.warn(rel, "§5.2 `generated.by` is required within `generated`")
|
||||
else:
|
||||
check_actor(gen["by"], "generated.by", rel, report)
|
||||
check_instant(gen.get("at"), "generated.at", rel, report)
|
||||
|
||||
ver = meta.get("verified")
|
||||
if ver is None:
|
||||
return
|
||||
# A bare mapping is one verification event — consumers MUST treat it as a
|
||||
# one-element list (§5.2), so normalize before checking the entries.
|
||||
entries = [ver] if isinstance(ver, dict) else ver
|
||||
if not isinstance(entries, list):
|
||||
report.warn(rel, "§5.2 `verified` must be a `{by, at}` mapping or a list of them")
|
||||
return
|
||||
for i, entry in enumerate(entries):
|
||||
if not (isinstance(entry, dict) and str(entry.get("by", "")).strip()):
|
||||
report.warn(rel, "§5.2 every `verified` entry needs a `by` actor")
|
||||
continue
|
||||
check_actor(entry["by"], f"verified[{i}].by", rel, report)
|
||||
check_instant(entry.get("at"), f"verified[{i}].at", rel, report)
|
||||
|
||||
|
||||
def check_lifecycle(meta: dict, rel: str, report: Report) -> None:
|
||||
"""§5.4 / §5.5 — `status` and `stale_after`. A concept is stale when
|
||||
`now >= stale_after`."""
|
||||
status = meta.get("status")
|
||||
if status is not None and status not in STATUS_VALUES:
|
||||
report.warn(rel, f"§5.4 unknown `status` `{status}` (expected "
|
||||
f"{'|'.join(sorted(STATUS_VALUES))})")
|
||||
stale = meta.get("stale_after")
|
||||
if stale is not None and not RFC3339.match(str(stale)):
|
||||
report.warn(rel, f"§5.5 `stale_after` `{stale}` is not an ISO 8601 datetime "
|
||||
f"(a bare date is tolerated)")
|
||||
|
||||
|
||||
def check_window(window, where: str, rel: str, report: Report) -> None:
|
||||
"""§5.1 — a `usage_window` is a `{from, to}` pair of ISO 8601 datetimes."""
|
||||
if window is None:
|
||||
return
|
||||
if not isinstance(window, dict):
|
||||
report.warn(rel, f"§5.1 `{where}` `usage_window` must be a `{{from, to}}` mapping")
|
||||
return
|
||||
for bound in ("from", "to"):
|
||||
value = window.get(bound)
|
||||
if value is None:
|
||||
report.warn(rel, f"§5.1 `{where}` `usage_window` is missing `{bound}`")
|
||||
elif not RFC3339.match(str(value)):
|
||||
report.warn(rel, f"§5.1 `{where}` `usage_window.{bound}` `{value}` "
|
||||
f"is not an ISO 8601 datetime (a bare date is tolerated)")
|
||||
|
||||
|
||||
def check_sources(meta: dict, body: str, rel: str, report: Report) -> None:
|
||||
"""§5.1 — the `sources` provenance family and its footnote join keys."""
|
||||
if CITATIONS.search(body):
|
||||
report.warn(rel, "legacy v0.1 `# Citations` body list: v0.2 records provenance "
|
||||
"in the `sources` frontmatter (§13.1)")
|
||||
srcs = meta.get("sources")
|
||||
if srcs is None:
|
||||
return
|
||||
if not isinstance(srcs, list):
|
||||
report.warn(rel, "§5.1 `sources` must be a list of entries")
|
||||
return
|
||||
ids = set()
|
||||
for i, src in enumerate(srcs):
|
||||
if not isinstance(src, dict):
|
||||
report.warn(rel, f"§5.1 `sources[{i}]` must be a mapping")
|
||||
continue
|
||||
if not str(src.get("resource", "")).strip():
|
||||
report.warn(rel, f"§5.1 `sources[{i}]` is missing the required `resource`")
|
||||
if "id" in src:
|
||||
ids.add(str(src["id"]))
|
||||
if src.get("author") is not None:
|
||||
check_actor(src["author"], f"sources[{i}].author", rel, report)
|
||||
# §5.1 — `usage_count` is framed by a `usage_window`, written once as a
|
||||
# sibling of `sources` or overridden on the entry. Without one the count
|
||||
# is a number with no units, and the spec asks consumers to read it as
|
||||
# liveness and trend rather than a score.
|
||||
window = src.get("usage_window", meta.get("usage_window"))
|
||||
if src.get("usage_count") is not None and window is None:
|
||||
report.warn(rel, f"§5.1 `sources[{i}].usage_count` has no `usage_window` "
|
||||
f"framing it (a sibling of `sources`, or on the entry)")
|
||||
check_window(window, f"sources[{i}]", rel, report)
|
||||
last_mod = src.get("last_modified")
|
||||
if last_mod is not None and not RFC3339.match(str(last_mod)):
|
||||
report.warn(rel, f"§5.1 `sources[{i}].last_modified` `{last_mod}` is not an "
|
||||
f"ISO 8601 datetime (a bare date is tolerated)")
|
||||
# Attribution joins on the label, not on position (§5.1) — a footnote whose
|
||||
# label names no source silently attributes a claim to nothing.
|
||||
for label in sorted(set(FOOTNOTE.findall(body)) - ids):
|
||||
report.warn(rel, f"§5.1 footnote `[^{label}]` matches no `sources[].id`")
|
||||
|
||||
|
||||
def check_index(path: Path, rel: str, is_root: bool, report: Report) -> None:
|
||||
report.indexes += 1
|
||||
text = _read_text(path, rel, report)
|
||||
if text is None:
|
||||
return
|
||||
raw, _ = split_frontmatter(text)
|
||||
if raw is not None:
|
||||
if not is_root:
|
||||
report.warn(rel, "§8 index.md should contain no frontmatter")
|
||||
else:
|
||||
try:
|
||||
meta = yaml.safe_load(raw) or {}
|
||||
except yaml.YAMLError:
|
||||
report.warn(rel, "§12 root index.md frontmatter is not valid YAML")
|
||||
meta = {}
|
||||
# `upkeep` rides along: the plugin's Stop hook reads its opt-in flag
|
||||
# from here, so an enforced-mode bundle would otherwise fail its own
|
||||
# `--strict` run. §11 forbids rejecting a bundle for unknown keys.
|
||||
extra = set(meta) - {"okf_version", "upkeep"}
|
||||
if extra:
|
||||
report.warn(rel, f"§12 root index.md frontmatter may only carry `okf_version` (found {sorted(extra)})")
|
||||
declared = meta.get("okf_version")
|
||||
if declared is not None and str(declared) != OKF_VERSION:
|
||||
report.warn(rel, f"§12 bundle declares `okf_version: \"{declared}\"`; "
|
||||
f"checked against v{OKF_VERSION}")
|
||||
|
||||
|
||||
def check_log(path: Path, rel: str, report: Report) -> None:
|
||||
report.logs += 1
|
||||
text = _read_text(path, rel, report)
|
||||
if text is None:
|
||||
return
|
||||
raw, _ = split_frontmatter(text)
|
||||
if raw is not None:
|
||||
report.warn(rel, "§9 log.md should contain no frontmatter")
|
||||
for line in text.splitlines():
|
||||
if line.startswith("## "):
|
||||
heading = line[3:].strip()
|
||||
if not ISO_DATE.match(heading):
|
||||
report.warn(rel, f"§9 date heading `{heading}` is not ISO 8601 YYYY-MM-DD")
|
||||
|
||||
|
||||
def collect_link_targets(path: Path) -> list[str]:
|
||||
# Unreadable files are already reported as a per-file error by
|
||||
# check_concept/check_index/check_log; fail soft here to avoid a duplicate
|
||||
# error and a crash on the second read pass.
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8")
|
||||
except (UnicodeDecodeError, OSError):
|
||||
return []
|
||||
targets: list[str] = []
|
||||
in_fence = False
|
||||
for line in text.splitlines():
|
||||
if FENCE.match(line.strip()):
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
if in_fence:
|
||||
continue
|
||||
targets.extend(LINK.findall(line))
|
||||
return targets
|
||||
|
||||
|
||||
def check_links(bundle: Path, md_files: list[Path], report: Report) -> None:
|
||||
"""Broken bundle-internal links are warnings only (§6.1)."""
|
||||
existing = {p.relative_to(bundle).as_posix() for p in md_files}
|
||||
for path in md_files:
|
||||
rel = path.relative_to(bundle).as_posix()
|
||||
for target in collect_link_targets(path):
|
||||
t = target.split("#", 1)[0]
|
||||
if not t or t.endswith("/"):
|
||||
continue
|
||||
if re.match(r"^[a-z][a-z0-9+.-]*://", t) or t.startswith("mailto:"):
|
||||
continue
|
||||
if not t.endswith(".md"):
|
||||
continue
|
||||
if t.startswith("/"):
|
||||
resolved = t.lstrip("/")
|
||||
else:
|
||||
resolved = (path.parent / t).resolve().relative_to(bundle.resolve()).as_posix() \
|
||||
if (path.parent / t).resolve().is_relative_to(bundle.resolve()) else t
|
||||
if resolved not in existing:
|
||||
report.warn(rel, f"cross-link target not found: `{target}` (tolerated under §6.1)")
|
||||
|
||||
|
||||
def citations_to_sources(raw: str, body: str) -> tuple[str, str]:
|
||||
"""Hoist a v0.1 `# Citations` list into the `sources` family (§5.1, §13.1)."""
|
||||
heading = CITATIONS.search(body)
|
||||
if heading is None:
|
||||
return raw, body
|
||||
rest = body[heading.end():]
|
||||
nxt = HEADING.search(rest)
|
||||
section, tail = (rest[:nxt.start()], rest[nxt.start():]) if nxt else (rest, "")
|
||||
entries = MD_LINK.findall(section)
|
||||
if not entries:
|
||||
return raw, body # nothing extractable — leave it and keep warning
|
||||
# json.dumps gives a correctly escaped YAML double-quoted scalar for free.
|
||||
block = "sources:\n" + "".join(
|
||||
f" - resource: {json.dumps(url)}\n title: {json.dumps(title)}\n"
|
||||
for title, url in entries)
|
||||
kept = body[:heading.start()].rstrip("\n")
|
||||
return raw + block, kept + ("\n\n" + tail if tail else "\n")
|
||||
|
||||
|
||||
def migrate_text(text: str, is_root_index: bool) -> str | None:
|
||||
"""Rewrite one file's v0.1 constructs to v0.2. None when nothing changes.
|
||||
|
||||
Textual on purpose: dumping the frontmatter back through PyYAML would flatten
|
||||
comments, key order and quoting across every file in the bundle. Each branch
|
||||
is guarded on the v0.2 key already being present, so a half-migrated bundle
|
||||
converges instead of growing a duplicate key that PyYAML then refuses to load.
|
||||
"""
|
||||
raw, body = split_frontmatter(text)
|
||||
if raw is None:
|
||||
return None
|
||||
if is_root_index:
|
||||
new_raw = OKF_VERSION_LINE.sub(rf'\g<1>"{OKF_VERSION}"', raw, count=1)
|
||||
return None if new_raw == raw else f"---\n{new_raw}---\n{body}"
|
||||
try:
|
||||
meta = yaml.safe_load(raw)
|
||||
except yaml.YAMLError:
|
||||
return None
|
||||
if not isinstance(meta, dict):
|
||||
return None
|
||||
new_raw, new_body = raw, body
|
||||
if "generated" not in meta:
|
||||
new_raw = TIMESTAMP.sub(
|
||||
lambda m: f"generated:\n by: {MIGRATE_ACTOR}\n at: {m.group(1)}",
|
||||
new_raw, count=1)
|
||||
if "sources" not in meta:
|
||||
new_raw, new_body = citations_to_sources(new_raw, new_body)
|
||||
if (new_raw, new_body) == (raw, body):
|
||||
return None
|
||||
return f"---\n{new_raw}---\n{new_body}"
|
||||
|
||||
|
||||
def migrate(bundle: Path) -> list[str]:
|
||||
"""Migrate a bundle in place. Returns the relative paths actually rewritten."""
|
||||
changed = []
|
||||
for path in sorted(p for p in bundle.rglob("*.md") if p.is_file()):
|
||||
rel = path.relative_to(bundle).as_posix()
|
||||
if path.name == "log.md":
|
||||
continue
|
||||
try:
|
||||
# Universal newlines on purpose: a CRLF bundle is rewritten with the
|
||||
# platform ending, consistently. Passing newline="" to preserve them
|
||||
# exactly would leave a stray \r inside every `$`-anchored capture
|
||||
# below, so those patterns need hardening first if byte fidelity is
|
||||
# ever required.
|
||||
text = path.read_text(encoding="utf-8")
|
||||
except (UnicodeDecodeError, OSError):
|
||||
continue
|
||||
is_root_index = path.name == "index.md" and path.parent == bundle
|
||||
if path.name == "index.md" and not is_root_index:
|
||||
continue
|
||||
new = migrate_text(text.lstrip(""), is_root_index)
|
||||
if new is not None and new != text:
|
||||
path.write_text(new, encoding="utf-8")
|
||||
changed.append(rel)
|
||||
return changed
|
||||
|
||||
|
||||
def validate(bundle: Path) -> Report:
|
||||
report = Report()
|
||||
md_files = sorted(p for p in bundle.rglob("*.md") if p.is_file())
|
||||
for path in md_files:
|
||||
rel = path.relative_to(bundle).as_posix()
|
||||
name = path.name
|
||||
if name == "index.md":
|
||||
check_index(path, rel, is_root=(path.parent == bundle), report=report)
|
||||
elif name == "log.md":
|
||||
check_log(path, rel, report)
|
||||
else:
|
||||
check_concept(path, rel, bundle, report)
|
||||
check_links(bundle, md_files, report)
|
||||
return report
|
||||
|
||||
|
||||
def _force_utf8_stdio() -> None:
|
||||
"""Ensure stdout/stderr can emit the ✓/✗/— glyphs this tool prints.
|
||||
|
||||
Windows consoles default to cp1252, which raises UnicodeEncodeError on those
|
||||
characters and crashes an otherwise-successful run. Reconfiguring to UTF-8
|
||||
(Python ≥3.7) fixes it; `errors="replace"` is a belt-and-suspenders fallback
|
||||
for any stream that still cannot encode a glyph.
|
||||
"""
|
||||
for stream in (sys.stdout, sys.stderr):
|
||||
reconfigure = getattr(stream, "reconfigure", None)
|
||||
if reconfigure is not None:
|
||||
try:
|
||||
reconfigure(encoding="utf-8", errors="replace")
|
||||
except (ValueError, OSError):
|
||||
pass
|
||||
|
||||
|
||||
def main() -> int:
|
||||
_force_utf8_stdio()
|
||||
ap = argparse.ArgumentParser(description=f"Validate an OKF v{OKF_VERSION} bundle.")
|
||||
ap.add_argument("bundle", type=Path, help="path to the bundle directory")
|
||||
ap.add_argument("--strict", action="store_true", help="treat warnings as errors")
|
||||
ap.add_argument("--max-warnings", type=int, default=None, metavar="N",
|
||||
help="fail if warnings exceed N (--strict is the N=0 case)")
|
||||
ap.add_argument("--migrate", action="store_true",
|
||||
help="rewrite v0.1 constructs to v0.2 in place, then validate")
|
||||
ap.add_argument("--json", action="store_true", help="emit a JSON report")
|
||||
args = ap.parse_args()
|
||||
|
||||
if not args.bundle.is_dir():
|
||||
print(f"error: {args.bundle} is not a directory", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
migrated = migrate(args.bundle) if args.migrate else []
|
||||
|
||||
r = validate(args.bundle)
|
||||
conformant = not r.errors
|
||||
max_warn = 0 if args.strict else args.max_warnings
|
||||
failed = bool(r.errors) or (max_warn is not None and len(r.warnings) > max_warn)
|
||||
|
||||
if args.json:
|
||||
print(json.dumps({
|
||||
"bundle": str(args.bundle),
|
||||
"conformant": conformant,
|
||||
"passed": not failed,
|
||||
"counts": {"concepts": r.concepts, "indexes": r.indexes, "logs": r.logs},
|
||||
"errors": r.errors,
|
||||
"warnings": r.warnings,
|
||||
"migrated": migrated,
|
||||
}, indent=2))
|
||||
return 0 if not failed else 1
|
||||
|
||||
if args.migrate:
|
||||
for rel in migrated:
|
||||
print(f" \033[36m→ migrated\033[0m {rel}")
|
||||
print(f" migrated {len(migrated)} file(s) to v{OKF_VERSION}"
|
||||
+ (f", `generated.by` recorded as `{MIGRATE_ACTOR}`" if migrated else ""))
|
||||
if migrated:
|
||||
print(" note: per-claim attribution (the `[^id]` footnotes of §5.1) is not"
|
||||
" recoverable from a v0.1 `# Citations` list — sources moved up, but"
|
||||
" which claim cited which was never encoded.")
|
||||
|
||||
print(f"OKF v{OKF_VERSION} conformance — {args.bundle}")
|
||||
print(f" concepts: {r.concepts} index.md: {r.indexes} log.md: {r.logs}")
|
||||
for e in r.errors:
|
||||
print(f" \033[31m✗ ERROR\033[0m {e}")
|
||||
for w in r.warnings:
|
||||
print(f" \033[33m! warn \033[0m {w}")
|
||||
if conformant and not r.warnings:
|
||||
print(" \033[32m✓ conformant — no issues\033[0m")
|
||||
elif conformant:
|
||||
print(f" \033[32m✓ conformant\033[0m ({len(r.warnings)} warning(s))")
|
||||
else:
|
||||
print(f" \033[31m✗ non-conformant\033[0m ({len(r.errors)} error(s))")
|
||||
return 0 if not failed else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
-64
@@ -1,64 +0,0 @@
|
||||
---
|
||||
name: visualize
|
||||
description: >-
|
||||
Render an Open Knowledge Format (OKF) bundle as a single self-contained,
|
||||
interactive HTML graph (viz.html) — concepts as nodes coloured by type and
|
||||
sized by body length, markdown links and bundle-internal `sources` as edges,
|
||||
a wiki-style detail panel with rendered markdown, v0.2 trust/lifecycle/provenance
|
||||
metadata, and "Links to" / "Cited by" backlinks, layout switching, per-type
|
||||
filter and search.
|
||||
Use when asked to visualize, graph, preview, or explore an OKF bundle.
|
||||
user-invocable: true
|
||||
argument-hint: "<bundle-dir> [-o viz.html]"
|
||||
allowed-tools: Bash
|
||||
---
|
||||
|
||||
# Visualize an OKF bundle
|
||||
|
||||
Generate a self-contained HTML graph of the target bundle. The bundle directory
|
||||
is required, so pass `.okf` explicitly when no path is given. No backend, no
|
||||
install, no bundle content leaves the page. The viewer does need network
|
||||
access: cytoscape, marked and dompurify load from cdn.jsdelivr.net.
|
||||
|
||||
```bash
|
||||
uv run "${CLAUDE_SKILL_DIR}/scripts/okf_visualize.py" $ARGUMENTS
|
||||
```
|
||||
|
||||
If `uv` is unavailable:
|
||||
|
||||
```bash
|
||||
python3 -m pip install --quiet pyyaml && \
|
||||
python3 "${CLAUDE_SKILL_DIR}/scripts/okf_visualize.py" $ARGUMENTS
|
||||
```
|
||||
|
||||
The detail panel shows each concept's `type`, `description`, `tags`, `status`,
|
||||
`generated`, `verified`, `stale_after`, `sources` (with their credibility
|
||||
signals, a `usage_count` alongside the `usage_window` that frames it), and
|
||||
rendered body; a `sources` entry pointing at another concept in the bundle also
|
||||
becomes a graph edge. A v0.1 `timestamp` is read as `generated.at`, so legacy
|
||||
bundles still render fully.
|
||||
|
||||
Two badges are **derived**, not read: the §5.3 trust tier
|
||||
(*unverified* / *machine-confirmed* / *human-reviewed*, keyed off the `human:`
|
||||
prefix in `verified[].by`) and staleness (`now >= stale_after`). They are
|
||||
advisory signals — when reporting on a bundle, say which tier a concept is in
|
||||
rather than treating any of them as a gate.
|
||||
|
||||
| Flag | Effect |
|
||||
|------|--------|
|
||||
| `-o <path>` | Output file (default `<bundle>/viz.html`). |
|
||||
| `-t, --title` | Graph title (default `parent/bundle` dir name). |
|
||||
| `-l, --link` | Source URL shown in the header. |
|
||||
| `--layout` | `cose`, `concentric`, `breadthfirst`, `circle`, `grid`. |
|
||||
| `--max-nodes N` | Exit 1 above N concepts (useful in CI). |
|
||||
| `--og-image URL` | Absolute URL for the social preview image. |
|
||||
|
||||
Above 1,000 concepts the default switches to `concentric` (force freezes the
|
||||
page; `--layout cose` overrides, and the in-page picker asks first). Above
|
||||
5,000 it warns that the page loads slowly and reads as a hairball, and
|
||||
suggests rendering a subdirectory.
|
||||
|
||||
Open it in any browser; `${CLAUDE_SKILL_DIR}` resolves whether this runs as part
|
||||
of the `okf` plugin or as a standalone skills.sh skill. The page is
|
||||
deep-linkable: `?select=<concept-id>` or `#<concept-id>` opens that concept,
|
||||
`?layout=<name>` presets the layout.
|
||||
@@ -1,432 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["pyyaml>=6"]
|
||||
# ///
|
||||
"""Render an Open Knowledge Format (OKF) bundle as a single self-contained,
|
||||
interactive HTML graph (`viz.html`). No backend, no install on the viewing side,
|
||||
no data leaves the page — concepts become nodes (coloured by `type`, sized by
|
||||
body length), markdown links and bundle-internal `sources` become edges, and
|
||||
clicking a node opens a wiki-style panel with its rendered markdown, OKF v0.2
|
||||
provenance/trust/lifecycle metadata, outgoing links, and "Cited by" backlinks.
|
||||
|
||||
Features: force/concentric/breadth-first/circle/grid layouts, per-type filter,
|
||||
free-text search, neighbour highlight, clickable cross-links and backlinks.
|
||||
|
||||
The default layout is force (cose) up to AUTO_COSE_MAX concepts, then the linear
|
||||
concentric layout — force-directed cost grows roughly quadratically with node
|
||||
count and freezes the page on large bundles. An explicit --layout always wins.
|
||||
|
||||
Run: uv run okf_visualize.py <bundle-dir> [-o viz.html]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
RESERVED = {"index.md", "log.md"}
|
||||
# Force (cose) layout froze the page for ~32 s at ~2k concepts (measured in
|
||||
# Chrome); the linear layouts load the same bundle in under 2 s. Past this size
|
||||
# the default switches to concentric, and the in-page layout picker asks before
|
||||
# running force. An explicit --layout (or ?layout=) still wins.
|
||||
AUTO_COSE_MAX = 1000
|
||||
# Above this the page is slow on any layout (23k concepts measured: ~27 s load,
|
||||
# ~650 MB heap) and reads as a hairball — warn and suggest rendering a subtree.
|
||||
SCALE_WARN = 5000
|
||||
FENCE = re.compile(r"^(```|~~~)")
|
||||
LINK = re.compile(r"(?<!\!)\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
|
||||
|
||||
|
||||
def json_for_script(obj) -> str:
|
||||
"""JSON-encode ``obj`` for safe embedding inside an inline ``<script>``.
|
||||
|
||||
The HTML tokenizer knows nothing about JS string context: a literal
|
||||
``</script>`` in a concept body ends the inline script early (truncating the
|
||||
embedded ``NODES``/``EDGES`` and breaking the whole page), and ``<!--`` can
|
||||
shift it into the escaped script-data states where even the template's real
|
||||
closing tag stops working. Escaping every ``<`` as ``\\u003c`` neutralizes
|
||||
``</script>``, ``<!--`` and ``<script`` in one stroke, and the result stays
|
||||
valid JSON *and* JavaScript (``json.dumps`` keeps everything else ASCII via
|
||||
the default ``ensure_ascii=True``).
|
||||
"""
|
||||
return json.dumps(obj, default=str).replace("<", "\\u003c")
|
||||
|
||||
|
||||
def split_frontmatter(text: str):
|
||||
if not text.startswith("---"):
|
||||
return {}, text
|
||||
lines = text.splitlines(keepends=True)
|
||||
if lines[0].strip() != "---":
|
||||
return {}, text
|
||||
for i in range(1, len(lines)):
|
||||
if lines[i].strip() == "---":
|
||||
try:
|
||||
meta = yaml.safe_load("".join(lines[1:i])) or {}
|
||||
except yaml.YAMLError:
|
||||
meta = {}
|
||||
return (meta if isinstance(meta, dict) else {}), "".join(lines[i + 1:])
|
||||
return {}, text
|
||||
|
||||
|
||||
def link_targets(text: str):
|
||||
out, in_fence = [], False
|
||||
for line in text.splitlines():
|
||||
if FENCE.match(line.strip()):
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
if not in_fence:
|
||||
out.extend(LINK.findall(line))
|
||||
return out
|
||||
|
||||
|
||||
def resolve(target: str, path: Path, bundle: Path):
|
||||
"""Resolve a link/`sources[].resource` to a concept id, or None if it is not
|
||||
one (an external URL, an asset, a scope descriptor, an escape from the tree)."""
|
||||
t = str(target).split("#", 1)[0]
|
||||
if not t.endswith(".md"):
|
||||
return None
|
||||
if t.startswith("/"):
|
||||
return t.lstrip("/")[:-3]
|
||||
cand = (path.parent / t).resolve()
|
||||
return cand.relative_to(bundle.resolve()).as_posix()[:-3] \
|
||||
if cand.is_relative_to(bundle.resolve()) else None
|
||||
|
||||
|
||||
def read_sources(meta: dict, path: Path, bundle: Path):
|
||||
"""§5.1 `sources`, flattened for display. `cid` is set when the source is
|
||||
itself a concept in this bundle — that derivation is a real graph edge."""
|
||||
raw = meta.get("sources")
|
||||
out = []
|
||||
if not isinstance(raw, list):
|
||||
return out
|
||||
for src in raw:
|
||||
if not isinstance(src, dict):
|
||||
continue
|
||||
resource = str(src.get("resource", "")).strip()
|
||||
# §5.1 — an entry's own `usage_window` overrides the one written once as a
|
||||
# sibling of `sources`. A count without its window has no units.
|
||||
window = src.get("usage_window", meta.get("usage_window"))
|
||||
out.append({
|
||||
"title": str(src.get("title") or src.get("id") or resource),
|
||||
"resource": resource,
|
||||
"cid": resolve(resource, path, bundle) if resource else None,
|
||||
"author": str(src.get("author", "")),
|
||||
"usage_count": src.get("usage_count"),
|
||||
"usage_window": (f"{window.get('from', '?')}→{window.get('to', '?')}"
|
||||
if isinstance(window, dict) else ""),
|
||||
"last_modified": str(src.get("last_modified") or ""),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def read_trust(meta: dict):
|
||||
"""§5.2 `generated` / `verified`, falling back to a v0.1 `timestamp` (§13.1)."""
|
||||
gen = meta.get("generated")
|
||||
if isinstance(gen, dict):
|
||||
generated = {"by": str(gen.get("by", "")), "at": str(gen.get("at", ""))}
|
||||
elif meta.get("timestamp"):
|
||||
generated = {"by": "", "at": str(meta["timestamp"])}
|
||||
else:
|
||||
generated = None
|
||||
ver = meta.get("verified")
|
||||
# a bare mapping is one verification event (§5.2)
|
||||
entries = [ver] if isinstance(ver, dict) else (ver if isinstance(ver, list) else [])
|
||||
verified = [{"by": str(e.get("by", "")), "at": str(e.get("at", ""))}
|
||||
for e in entries if isinstance(e, dict)]
|
||||
return generated, verified
|
||||
|
||||
|
||||
def build(bundle: Path):
|
||||
nodes, edges, seen = [], [], set()
|
||||
files = sorted(p for p in bundle.rglob("*.md") if p.is_file() and p.name not in RESERVED)
|
||||
ids = {p.relative_to(bundle).with_suffix("").as_posix() for p in files}
|
||||
for p in files:
|
||||
cid = p.relative_to(bundle).with_suffix("").as_posix()
|
||||
try:
|
||||
raw = p.read_text(encoding="utf-8").lstrip("")
|
||||
except (UnicodeDecodeError, OSError) as exc:
|
||||
print(f"warning: skipping {p.relative_to(bundle)}: cannot read file: {exc}", file=sys.stderr)
|
||||
ids.discard(cid)
|
||||
continue
|
||||
meta, body = split_frontmatter(raw)
|
||||
body = body.strip()
|
||||
generated, verified = read_trust(meta)
|
||||
sources = read_sources(meta, p, bundle)
|
||||
nodes.append({
|
||||
"id": cid,
|
||||
"type": str(meta.get("type", "Untyped")),
|
||||
"title": str(meta.get("title", p.stem)),
|
||||
"description": str(meta.get("description", "")),
|
||||
"tags": meta.get("tags", []) if isinstance(meta.get("tags"), list) else [],
|
||||
"group": cid.split("/")[0] if "/" in cid else "(root)",
|
||||
"sz": max(24, min(70, 24 + len(body) // 200)),
|
||||
"status": str(meta.get("status", "")),
|
||||
"stale_after": str(meta.get("stale_after") or ""),
|
||||
"generated": generated,
|
||||
"verified": verified,
|
||||
"sources": sources,
|
||||
"body": body[:8000],
|
||||
})
|
||||
targets = link_targets(body) + [s["resource"] for s in sources if s["cid"]]
|
||||
for t in targets:
|
||||
tgt = resolve(t, p, bundle)
|
||||
if tgt and tgt in ids and tgt != cid and (cid, tgt) not in seen:
|
||||
seen.add((cid, tgt))
|
||||
edges.append({"source": cid, "target": tgt})
|
||||
return nodes, edges
|
||||
|
||||
|
||||
HTML = r"""<!doctype html><html lang="en"><head><meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>OKF — __NAME__</title>
|
||||
<meta property="og:title" content="__OGTITLE__">
|
||||
<meta property="og:description" content="__OGDESC__">
|
||||
<meta property="og:type" content="website">
|
||||
<meta name="twitter:card" content="summary_large_image">
|
||||
__OGIMAGE__
|
||||
<script src="https://cdn.jsdelivr.net/npm/cytoscape@3.30.2/dist/cytoscape.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/marked@14/marked.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/dompurify@3.4.12/dist/purify.min.js" integrity="sha384-piCcpDdJ7qVeK4Tv8Z6Hpcr3ZBIgP16TxQTPVfsLFdZ5uDgwc3Y8Ho7oUnqf12qu" crossorigin="anonymous"></script>
|
||||
<style>
|
||||
:root{--bg:#0e0f13;--panel:#16181f;--line:#262a35;--fg:#e6e8ee;--mut:#9aa3b2;--accent:#8ab4ff}
|
||||
*{box-sizing:border-box} html,body{margin:0;height:100%;background:var(--bg);color:var(--fg);
|
||||
font:14px/1.5 ui-sans-serif,system-ui,-apple-system,Segoe UI,Roboto,sans-serif}
|
||||
/* The row must be stated. Left implicit it sizes to `auto`, i.e. to the tallest
|
||||
item — a long concept body then stretched #side past the viewport and the page
|
||||
grew its own scrollbar on top of the panel's, which in turn forced a spurious
|
||||
horizontal one. Pinning the row to 100% keeps the panel scrolling inside. */
|
||||
#app{display:grid;grid-template-columns:1fr 400px;grid-template-rows:100%;height:100vh}
|
||||
#cy{width:100%;height:100%}
|
||||
#side{border-left:1px solid var(--line);background:var(--panel);overflow:auto;padding:18px}
|
||||
header{position:absolute;top:0;left:0;padding:14px 18px;z-index:5;pointer-events:none}
|
||||
h1{font-size:15px;margin:0;font-weight:650} .sub{color:var(--mut);font-size:12px;margin-top:2px}
|
||||
#bar{position:absolute;top:12px;left:50%;transform:translateX(-50%);z-index:5;display:flex;gap:8px}
|
||||
#bar input,#bar select{background:var(--panel);border:1px solid var(--line);color:var(--fg);
|
||||
border-radius:8px;padding:8px 10px;outline:none;font-size:13px}
|
||||
#search{width:min(300px,32vw)}
|
||||
#legend{position:absolute;bottom:14px;left:18px;z-index:5;display:flex;flex-wrap:wrap;gap:6px;max-width:60vw}
|
||||
.chip{display:flex;align-items:center;gap:6px;background:var(--panel);border:1px solid var(--line);
|
||||
border-radius:20px;padding:3px 10px;font-size:12px;color:var(--mut);cursor:pointer;user-select:none}
|
||||
.chip.off{opacity:.4} .dot{width:10px;height:10px;border-radius:50%}
|
||||
#side h2{font-size:17px;margin:.2em 0} .type{display:inline-block;border-radius:6px;padding:2px 8px;
|
||||
font-size:11px;font-weight:600;color:#0e0f13} .desc{color:var(--mut);margin:8px 0 12px}
|
||||
.tags{display:flex;gap:6px;flex-wrap:wrap;margin-bottom:12px}
|
||||
.tag{background:#1d2230;border:1px solid var(--line);border-radius:6px;padding:1px 8px;font-size:11px;color:var(--mut)}
|
||||
.meta{margin:10px 0;font-size:12px;color:var(--mut)} .meta div{padding:1px 0} .meta b{color:var(--fg);font-weight:600}
|
||||
.sig{color:var(--mut)}
|
||||
.badges{display:flex;gap:6px;flex-wrap:wrap;margin:8px 0}
|
||||
.badge{border-radius:6px;padding:2px 8px;font-size:11px;font-weight:600;border:1px solid}
|
||||
.t-unverified{color:#9aa3b2;border-color:#3a4150;background:#1d2230}
|
||||
.t-machine{color:#8ab4ff;border-color:#2b4570;background:#141c2b}
|
||||
.t-human{color:#4ade80;border-color:#276b45;background:#12211a}
|
||||
.b-stale{color:#fca5a5;border-color:#7f2b2b;background:#241416}
|
||||
.b-deprecated{color:#fbbf24;border-color:#7a5312;background:#241d10}
|
||||
.rel{margin:10px 0} .rel h4{margin:0 0 4px;font-size:11px;text-transform:uppercase;letter-spacing:.05em;color:var(--mut)}
|
||||
.rel a{display:block;color:var(--accent);cursor:pointer;font-size:13px;padding:1px 0;text-decoration:none}
|
||||
.rel a:hover{text-decoration:underline}
|
||||
.body{border-top:1px solid var(--line);padding-top:12px;margin-top:12px} .body table{border-collapse:collapse;display:block;overflow:auto}
|
||||
.body td,.body th{border:1px solid var(--line);padding:4px 8px} .body code{background:#1d2230;padding:1px 5px;border-radius:4px}
|
||||
.body pre{background:#1d2230;padding:10px;border-radius:8px;overflow:auto} .body img{max-width:100%}
|
||||
.empty{color:var(--mut)} a{color:var(--accent)}
|
||||
.src{pointer-events:auto;color:var(--accent);margin-left:10px;text-decoration:none} .src:hover{text-decoration:underline}
|
||||
</style></head><body>
|
||||
<div id="app"><div id="cy"></div><div id="side"><p class="empty">Click a concept to inspect it.</p></div></div>
|
||||
<header><h1>__NAME__</h1><div class="sub">__N__ concepts · __E__ links · OKF v0.2__LINK__</div></header>
|
||||
<div id="bar">
|
||||
<input id="search" placeholder="search concepts…">
|
||||
<select id="type"><option value="">all types</option></select>
|
||||
<select id="layout">
|
||||
<option value="cose">force</option><option value="concentric">concentric</option>
|
||||
<option value="breadthfirst">breadth-first</option><option value="circle">circle</option><option value="grid">grid</option>
|
||||
</select>
|
||||
</div>
|
||||
<div id="legend"></div>
|
||||
<script>
|
||||
const NODES=__NODES__, EDGES=__EDGES__;
|
||||
const PALETTE=["#6E56CF","#D97757","#22C55E","#3B82F6","#EAB308","#EC4899","#14B8A6","#F97316","#A855F7","#0EA5E9","#84CC16","#EF4444","#64748B"];
|
||||
const byId=Object.fromEntries(NODES.map(n=>[n.id,n]));
|
||||
const outL={}, inL={};
|
||||
NODES.forEach(n=>{outL[n.id]=[];inL[n.id]=[];});
|
||||
EDGES.forEach(e=>{outL[e.source].push(e.target);inL[e.target].push(e.source);});
|
||||
const types=[...new Set(NODES.map(n=>n.type))].sort();
|
||||
const color=Object.fromEntries(types.map((t,i)=>[t,PALETTE[i%PALETTE.length]]));
|
||||
const off=new Set();
|
||||
const cy=cytoscape({container:document.getElementById('cy'),minZoom:.2,maxZoom:1.6,wheelSensitivity:.2,
|
||||
elements:[...NODES.map(n=>({data:{...n,c:color[n.type]}})),...EDGES.map(e=>({data:e}))],
|
||||
style:[
|
||||
{selector:'node',style:{'background-color':'data(c)','label':'data(title)','color':'#e6e8ee',
|
||||
'font-size':10,'text-wrap':'wrap','text-max-width':120,'text-valign':'bottom','text-margin-y':4,
|
||||
'text-outline-width':2,'text-outline-color':'#0e0f13','min-zoomed-font-size':6,
|
||||
'width':'data(sz)','height':'data(sz)'}},
|
||||
{selector:'edge',style:{'width':1.2,'line-color':'#3a4150','target-arrow-color':'#3a4150',
|
||||
'target-arrow-shape':'triangle','arrow-scale':.8,'curve-style':'bezier','opacity':.7}},
|
||||
{selector:'.dim',style:{'opacity':.10}},{selector:'.hl',style:{'border-width':3,'border-color':'#fff'}}
|
||||
],
|
||||
layout:{name:'__LAYOUT__',animate:false,nodeRepulsion:9000,idealEdgeLength:90,padding:40}});
|
||||
const side=document.getElementById('side');
|
||||
const esc=s=>(s||'').replace(/[&<>"']/g,c=>({'&':'&','<':'<','>':'>','"':'"',"'":'''}[c]));
|
||||
function relList(title,arr){if(!arr.length)return'';
|
||||
return `<div class="rel"><h4>${title}</h4>${arr.map(id=>`<a data-go="${esc(id)}">${esc((byId[id]||{}).title||id)}</a>`).join('')}</div>`;}
|
||||
// §5.3 — the trust tier is derived, never stored: no `verified` is unverified,
|
||||
// `verified` by non-`human:` actors only is machine-confirmed, and any `human:`
|
||||
// actor makes it human-reviewed. The exact lowercase prefix is the whole key.
|
||||
// §5.5 — a concept is stale when now >= stale_after. PyYAML renders an unquoted
|
||||
// datetime with a space separator, which Date.parse rejects in some engines.
|
||||
// Advisory signals, not access control.
|
||||
function trustTier(n){const v=n.verified||[];
|
||||
if(!v.length)return['t-unverified','unverified'];
|
||||
return v.some(e=>(e.by||'').startsWith('human:'))?['t-human','human-reviewed']
|
||||
:['t-machine','machine-confirmed'];}
|
||||
function badges(n){const [cls,label]=trustTier(n),out=[`<span class="badge ${cls}">${label}</span>`];
|
||||
if(n.stale_after&&Date.parse(n.stale_after.replace(' ','T'))<=Date.now())out.push(`<span class="badge b-stale">stale since ${esc(n.stale_after)}</span>`);
|
||||
if(n.status==='deprecated')out.push('<span class="badge b-deprecated">deprecated</span>');
|
||||
return `<div class="badges">${out.join('')}</div>`;}
|
||||
// OKF v0.2 trust (§5.2) + lifecycle (§5.4/§5.5). A v0.1 `timestamp` arrives here
|
||||
// as generated.at with an empty `by`, so legacy bundles still show a date.
|
||||
function metaBlock(n){const g=n.generated||{},rows=[];
|
||||
if(n.status)rows.push(`<div>status <b>${esc(n.status)}</b></div>`);
|
||||
if(g.at||g.by)rows.push(`<div>generated${g.at?` <b>${esc(g.at)}</b>`:''}${g.by?` by ${esc(g.by)}`:''}</div>`);
|
||||
(n.verified||[]).forEach(v=>rows.push(`<div>verified${v.at?` <b>${esc(v.at)}</b>`:''}${v.by?` by ${esc(v.by)}`:''}</div>`));
|
||||
if(n.stale_after)rows.push(`<div>stale after <b>${esc(n.stale_after)}</b></div>`);
|
||||
return rows.length?`<div class="meta">${rows.join('')}</div>`:'';}
|
||||
// Provenance (§5.1). A source may be another concept (graph link), an external
|
||||
// URL, or a scope descriptor that is not followable at all.
|
||||
function srcList(n){const s=n.sources||[];if(!s.length)return'';
|
||||
return `<div class="rel"><h4>Sources</h4>${s.map(x=>{
|
||||
const used=x.usage_count!=null?`used ${x.usage_count}×${x.usage_window?` (${x.usage_window})`:''}`:'';
|
||||
const sig=[x.author,used,x.last_modified].filter(Boolean).join(' · ');
|
||||
const label=esc(x.title||x.resource)+(sig?` <span class="sig">(${esc(sig)})</span>`:'');
|
||||
if(x.cid&&byId[x.cid])return `<a data-go="${esc(x.cid)}">${label}</a>`;
|
||||
if(/^https?:\/\//i.test(x.resource))return `<a href="${esc(x.resource)}" target="_blank" rel="noopener">${label}</a>`;
|
||||
return `<span class="empty">${label}</span>`;}).join('')}</div>`;}
|
||||
// Resolve an in-body markdown link href to a concept id, mirroring build()'s
|
||||
// resolution: strip #anchor, require .md, absolute strips leading /, relative
|
||||
// resolves against the current concept's dir. Returns null if it's not a concept.
|
||||
function resolveHref(cid,href){let t=(href||'').split('#')[0];if(!t.endsWith('.md'))return null;
|
||||
let tgt;if(t[0]==='/'){tgt=t.replace(/^\/+/,'').slice(0,-3);}
|
||||
else{const base=cid.split('/').slice(0,-1);
|
||||
for(const seg of t.slice(0,-3).split('/')){if(seg===''||seg==='.')continue;
|
||||
if(seg==='..'){if(!base.length)return null;base.pop();}else base.push(seg);}
|
||||
tgt=base.join('/');}
|
||||
return byId[tgt]?tgt:null;}
|
||||
function show(id){const n=byId[id];if(!n)return;const c=color[n.type];
|
||||
side.innerHTML=`<span class="type" style="background:${c}">${esc(n.type)}</span>
|
||||
<h2>${esc(n.title)}</h2><div class="desc">${esc(n.description)||'<span class=empty>no description</span>'}</div>
|
||||
<div class="tags">${(n.tags||[]).map(t=>`<span class="tag">${esc(t)}</span>`).join('')}</div>
|
||||
${badges(n)}${metaBlock(n)}${srcList(n)}${relList('Links to',outL[id])}${relList('Cited by',inL[id])}
|
||||
<div class="body">${n.body?DOMPurify.sanitize(marked.parse(n.body)):'<span class=empty>empty body</span>'}</div>`;
|
||||
side.querySelectorAll('[data-go]').forEach(a=>a.onclick=()=>select(a.getAttribute('data-go')));
|
||||
side.querySelectorAll('.body a[href]').forEach(a=>{const tgt=resolveHref(id,a.getAttribute('href'));
|
||||
if(tgt)a.onclick=e=>{e.preventDefault();select(tgt);};});}
|
||||
function select(id){const ele=cy.getElementById(id);if(!ele.length)return;show(id);
|
||||
cy.elements().removeClass('hl').addClass('dim');const nb=ele.closedNeighborhood();nb.removeClass('dim');ele.addClass('hl');
|
||||
cy.animate({center:{eles:ele},duration:250});
|
||||
try{if(decodeURIComponent((location.hash||'').slice(1))!==id)location.hash=encodeURIComponent(id);}catch(e){}}
|
||||
cy.on('tap','node',e=>select(e.target.id()));
|
||||
cy.on('tap',e=>{if(e.target===cy)cy.elements().removeClass('dim hl');});
|
||||
function applyFilter(){const q=document.getElementById('search').value.toLowerCase();const ty=document.getElementById('type').value;
|
||||
cy.batch(()=>cy.nodes().forEach(n=>{const d=n.data();
|
||||
const m=(!q||(d.title+' '+d.type+' '+d.description+' '+(d.tags||[]).join(' ')).toLowerCase().includes(q))
|
||||
&&(!ty||d.type===ty)&&!off.has(d.type);
|
||||
n.style('display',m?'element':'none');}));}
|
||||
let debounce;
|
||||
document.getElementById('search').oninput=()=>{clearTimeout(debounce);debounce=setTimeout(applyFilter,150);};
|
||||
document.getElementById('type').oninput=applyFilter;
|
||||
const tysel=document.getElementById('type');types.forEach(t=>{const o=document.createElement('option');o.value=t;o.textContent=t;tysel.appendChild(o);});
|
||||
let curLayout='__LAYOUT__';
|
||||
document.getElementById('layout').onchange=e=>{const v=e.target.value;
|
||||
if(v==='cose'&&NODES.length>__COSEMAX__&&!confirm(`force layout on ${NODES.length} concepts can freeze this tab — run anyway?`)){e.target.value=curLayout;return;}
|
||||
curLayout=v;cy.layout({name:v,animate:true,padding:40,nodeRepulsion:9000,idealEdgeLength:90}).run();};
|
||||
document.getElementById('legend').innerHTML=types.map(t=>`<span class="chip" data-t="${esc(t)}"><span class="dot" style="background:${color[t]}"></span>${esc(t)} (${NODES.filter(n=>n.type===t).length})</span>`).join('');
|
||||
document.querySelectorAll('#legend .chip').forEach(ch=>ch.onclick=()=>{const t=ch.getAttribute('data-t');
|
||||
if(off.has(t)){off.delete(t);ch.classList.remove('off');}else{off.add(t);ch.classList.add('off');}applyFilter();});
|
||||
document.getElementById('layout').value='__LAYOUT__';
|
||||
const Q=new URLSearchParams(location.search),QL=Q.get('layout'),QS=Q.get('select');
|
||||
if(QL&&[...document.querySelectorAll('#layout option')].some(o=>o.value===QL)){document.getElementById('layout').value=QL;curLayout=QL;cy.layout({name:QL,animate:false,padding:40,nodeRepulsion:9000,idealEdgeLength:90}).run();}
|
||||
function fromHash(){try{const h=decodeURIComponent((location.hash||'').slice(1));if(h&&byId[h])select(h);}catch(e){}}
|
||||
addEventListener('hashchange',fromHash);
|
||||
if(QS&&byId[QS])select(QS);else fromHash();
|
||||
</script></body></html>"""
|
||||
|
||||
|
||||
def render(bundle: Path, out: Path, title: str | None = None, link: str | None = None,
|
||||
layout: str | None = None, og_image: str | None = None,
|
||||
max_nodes: int | None = None):
|
||||
nodes, edges = build(bundle)
|
||||
if max_nodes is not None and len(nodes) > max_nodes:
|
||||
sys.exit(f"error: {len(nodes)} concepts exceeds --max-nodes {max_nodes}")
|
||||
if layout is None:
|
||||
layout = "cose" if len(nodes) <= AUTO_COSE_MAX else "concentric"
|
||||
if layout != "cose":
|
||||
print(f"note: {len(nodes)} concepts > {AUTO_COSE_MAX} — using the linear 'concentric' "
|
||||
"layout (force freezes the page at this size; pass --layout cose to override)",
|
||||
file=sys.stderr)
|
||||
if len(nodes) > SCALE_WARN:
|
||||
print(f"warning: {len(nodes)} concepts — the page will load slowly and read as a hairball; "
|
||||
f"consider rendering a subtree, e.g. okf_visualize.py {bundle}/<subdir>",
|
||||
file=sys.stderr)
|
||||
name = title or f"{bundle.resolve().parent.name}/{bundle.name}"
|
||||
src = f' <a class="src" href="{link}" target="_blank" rel="noopener">source ↗</a>' if link else ""
|
||||
aesc = lambda s: (s or "").replace("&", "&").replace('"', """).replace("<", "<").replace(">", ">")
|
||||
og_title = aesc(f"OKF — {name}")
|
||||
og_desc = aesc(f"{len(nodes)} concepts · interactive Open Knowledge Format knowledge graph")
|
||||
og_img = (f'<meta property="og:image" content="{aesc(og_image)}">\n'
|
||||
f'<meta name="twitter:image" content="{aesc(og_image)}">') if og_image else ""
|
||||
subs = {"__NAME__": name, "__LINK__": src, "__LAYOUT__": layout,
|
||||
"__COSEMAX__": str(AUTO_COSE_MAX),
|
||||
"__OGTITLE__": og_title, "__OGDESC__": og_desc, "__OGIMAGE__": og_img,
|
||||
"__N__": str(len(nodes)), "__E__": str(len(edges)),
|
||||
"__NODES__": json_for_script(nodes), "__EDGES__": json_for_script(edges)}
|
||||
# One pass, longest marker first: substituted content (e.g. a concept body that
|
||||
# mentions "__EDGES__") must never itself be rescanned for other markers.
|
||||
marker = re.compile("|".join(sorted(map(re.escape, subs), key=len, reverse=True)))
|
||||
html = marker.sub(lambda m: subs[m.group(0)], HTML)
|
||||
out.write_text(html, encoding="utf-8")
|
||||
return len(nodes), len(edges)
|
||||
|
||||
|
||||
def _force_utf8_stdio() -> None:
|
||||
"""Reconfigure stdout/stderr to UTF-8 so status lines never crash on a
|
||||
cp1252 (default Windows) console. `errors="replace"` is a fallback."""
|
||||
for stream in (sys.stdout, sys.stderr):
|
||||
reconfigure = getattr(stream, "reconfigure", None)
|
||||
if reconfigure is not None:
|
||||
try:
|
||||
reconfigure(encoding="utf-8", errors="replace")
|
||||
except (ValueError, OSError):
|
||||
pass
|
||||
|
||||
|
||||
def main() -> int:
|
||||
_force_utf8_stdio()
|
||||
ap = argparse.ArgumentParser(description="Render an OKF bundle as a self-contained HTML graph.")
|
||||
ap.add_argument("bundle", type=Path)
|
||||
ap.add_argument("-o", "--out", type=Path, default=None)
|
||||
ap.add_argument("-t", "--title", default=None, help="graph title (default: parent/bundle dir name)")
|
||||
ap.add_argument("-l", "--link", default=None, help="optional source URL shown in the header")
|
||||
ap.add_argument("--layout", default=None,
|
||||
choices=["cose", "concentric", "breadthfirst", "circle", "grid"],
|
||||
help=f"initial graph layout (default: cose, or concentric above {AUTO_COSE_MAX} "
|
||||
"concepts — force layout freezes the page on large bundles)")
|
||||
ap.add_argument("--max-nodes", type=int, default=None,
|
||||
help="refuse to render bundles with more concepts than this (useful in CI)")
|
||||
ap.add_argument("--og-image", default=None,
|
||||
help="absolute URL for the social-preview image (og:image / twitter:image)")
|
||||
args = ap.parse_args()
|
||||
if not args.bundle.is_dir():
|
||||
print(f"error: {args.bundle} is not a directory", file=sys.stderr)
|
||||
return 2
|
||||
out = args.out or (args.bundle / "viz.html")
|
||||
n, e = render(args.bundle, out, title=args.title, link=args.link, layout=args.layout,
|
||||
og_image=args.og_image, max_nodes=args.max_nodes)
|
||||
print(f"rendered {n} concepts, {e} links -> {out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in new issue
Block a user