docs(memory): update skills for opencode-memory plugin, deprecate mem0
This commit is contained in:
m3tm3re
28
.sisyphus/notepads/memory-system/decisions.md
Normal file
28
.sisyphus/notepads/memory-system/decisions.md
Normal file
@@ -0,0 +1,28 @@
|
||||
|
||||
## Task 5: Update Mem0 Memory Skill (2026-02-12)
|
||||
|
||||
### Decisions Made
|
||||
|
||||
1. **Section Placement**: Added new sections without disrupting existing content structure
|
||||
- "Memory Categories" after "Identity Scopes" (line ~109)
|
||||
- "Dual-Layer Sync" after "Workflow Patterns" (line ~138)
|
||||
- Extended "Health Check" section with Pre-Operation Check
|
||||
- "Error Handling" at end, before API Reference
|
||||
|
||||
2. **Content Structure**:
|
||||
- Memory Categories: 5-category classification with table format
|
||||
- Dual-Layer Sync: Complete sync pattern with bash example
|
||||
- Health Check: Added pre-operation verification
|
||||
- Error Handling: Comprehensive graceful degradation patterns
|
||||
|
||||
3. **Validation Approach**:
|
||||
- Used `./scripts/test-skill.sh --validate` for skill structure validation
|
||||
- All sections verified with grep commands
|
||||
- Commit and push completed successfully
|
||||
|
||||
### Success Patterns
|
||||
|
||||
- Edit tool works well for adding sections to existing markdown files
|
||||
- Preserving existing content while adding new sections
|
||||
- Using grep for verification of section additions
|
||||
- `./scripts/test-skill.sh --validate` validates YAML frontmatter automatically
|
||||
0
.sisyphus/notepads/memory-system/issues.md
Normal file
0
.sisyphus/notepads/memory-system/issues.md
Normal file
47
.sisyphus/notepads/memory-system/learnings.md
Normal file
47
.sisyphus/notepads/memory-system/learnings.md
Normal file
@@ -0,0 +1,47 @@
|
||||
|
||||
## Core Memory Skill Creation (2026-02-12)
|
||||
|
||||
**Task**: Create `skills/memory/SKILL.md` - dual-layer memory orchestration skill
|
||||
|
||||
**Pattern Identified**:
|
||||
- Skill structure follows YAML frontmatter with required fields:
|
||||
- `name`: skill identifier
|
||||
- `description`: Use when (X), triggers (Y) pattern
|
||||
- `compatibility`: "opencode"
|
||||
- Markdown structure: Overview, Prerequisites, Workflows, Error Handling, Integration, Quick Reference, See Also
|
||||
|
||||
**Verification Pattern**:
|
||||
```bash
|
||||
test -f <path> && echo "File exists"
|
||||
grep "name: <skill>" <path>
|
||||
grep "key-term" <path>
|
||||
```
|
||||
|
||||
**Key Design Decision**:
|
||||
- Central orchestration skill that references underlying implementation skills (mem0-memory, obsidian)
|
||||
- 4 core workflows: Store, Recall, Auto-Capture, Auto-Recall
|
||||
- Error handling with graceful degradation
|
||||
|
||||
## Apollo Agent Prompt Update (2026-02-12)
|
||||
|
||||
**Task**: Add memory management responsibilities to Apollo agent system prompt
|
||||
|
||||
**Edit Pattern**: Multiple targeted edits to single file preserving existing content
|
||||
- Line number-based edits require precise matching of surrounding context
|
||||
- Edit order: Core Responsibilities → Quality Standards → Tool Usage → Edge Cases
|
||||
- Each edit inserts new bullet items without removing existing content
|
||||
|
||||
**Key Additions**:
|
||||
1. Core Responsibilities: "Manage dual-layer memory system (Mem0 + Obsidian CODEX)"
|
||||
2. Quality Standards: Memory storage, auto-capture, retrieval, categories
|
||||
3. Tool Usage: Mem0 REST API (localhost:8000), Obsidian MCP integration
|
||||
4. Edge Cases: Mem0 unavailable, Obsidian unavailable handling
|
||||
|
||||
**Verification Pattern**:
|
||||
```bash
|
||||
grep -c "memory" ~/p/AI/AGENTS/prompts/apollo.txt # Count occurrences
|
||||
grep "Mem0" ~/p/AI/AGENTS/prompts/apollo.txt # Check specific term
|
||||
grep -i "auto-capture" ~/p/AI/AGENTS/prompts/apollo.txt # Case-insensitive
|
||||
```
|
||||
|
||||
**Observation**: grep is case-sensitive by default - use -i for case-insensitive searches
|
||||
0
.sisyphus/notepads/memory-system/problems.md
Normal file
0
.sisyphus/notepads/memory-system/problems.md
Normal file
120
.sisyphus/notepads/opencode-memory/learnings.md
Normal file
120
.sisyphus/notepads/opencode-memory/learnings.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Opencode Memory Plugin — Learnings
|
||||
|
||||
## Session: ses_3a5a47a05ffeoNYfz2RARYsHX9
|
||||
Started: 2026-02-14
|
||||
|
||||
### Architecture Decisions
|
||||
- SQLite + FTS5 + vec0 replaces mem0+qdrant entirely
|
||||
- Markdown at ~/CODEX/80-memory/ is source of truth
|
||||
- SQLite DB at ~/.local/share/opencode-memory/index.db is derived index
|
||||
- OpenAI text-embedding-3-small for embeddings (1536 dimensions)
|
||||
- Hybrid search: 0.7 vector weight + 0.3 BM25 weight
|
||||
- Chunking: 400 tokens, 80 overlap (tiktoken cl100k_base)
|
||||
|
||||
### Key Patterns from Openclaw
|
||||
- MemoryIndexManager pattern (1590 lines) — file watching, chunking, indexing
|
||||
- Hybrid scoring with weighted combination
|
||||
- Embedding cache by content_hash + model
|
||||
- Two sources: "memory" (markdown files) + "sessions" (transcripts)
|
||||
- Two tools: memory_search (hybrid query) + memory_get (read lines)
|
||||
|
||||
### Technical Stack
|
||||
- Runtime: bun
|
||||
- Test framework: bun test (TDD)
|
||||
- SQLite: better-sqlite3 (synchronous API)
|
||||
- Embeddings: openai npm package
|
||||
- Chunking: tiktoken (cl100k_base encoding)
|
||||
- File watching: chokidar
|
||||
- Validation: zod (for tool schemas)
|
||||
|
||||
### Vec0 Extension Findings (Task 1)
|
||||
- **vec0 extension**: NOT AVAILABLE - requires vec0.so shared library not present
|
||||
- **Alternative solution**: sqlite-vec package (v0.1.7-alpha.2) successfully tested
|
||||
- **Loading mechanism**: `sqliteVec.load(db)` loads vector extension into database
|
||||
- **Test result**: Works with Node.js (better-sqlite3 native module compatible)
|
||||
- **Note**: better-sqlite3 does NOT work with Bun runtime (native module incompatibility)
|
||||
- **Testing command**: `node -e "const Database = require('better-sqlite3'); const sqliteVec = require('sqlite-vec'); const db = new Database(':memory:'); sqliteVec.load(db); console.log('OK')"`
|
||||
|
||||
### Bun Runtime Limitations
|
||||
- better-sqlite3 native module NOT compatible with Bun (ERR_DLOPEN_FAILED)
|
||||
- Use Node.js for any code requiring better-sqlite3
|
||||
- Alternative: bun:sqlite API (similar API, but not same library)
|
||||
|
||||
## Wave Progress
|
||||
- Wave 1: IN PROGRESS (Task 1)
|
||||
- Wave 2-6: PENDING
|
||||
|
||||
### Configuration Module Implementation (Task: Config Module)
|
||||
- **TDD approach**: RED-GREEN-REFACTOR cycle successfully applied
|
||||
- **Pattern**: Default config object + resolveConfig() function for merging
|
||||
- **Path expansion**: `expandPath()` helper function handles `~` → `$HOME` expansion
|
||||
- **Test coverage**: 10 tests covering defaults, overrides, path expansion, and config merging
|
||||
- **TypeScript best practices**: Proper type exports from types.ts, type imports in config.ts
|
||||
- **Defaults match openclaw**: chunking (400/80), search weights (0.7/0.3), minScore (0.35), maxResults (6)
|
||||
- **Bun test framework**: Fast execution (~20ms for 10 tests), clean output
|
||||
|
||||
### Database Schema Implementation (Task 2)
|
||||
- **TDD approach**: RED-GREEN-REFACTOR cycle successfully applied for db module
|
||||
- **Schema tables**: meta, files, chunks, embedding_cache, chunks_fts (FTS5), chunks_vec (vec0)
|
||||
- **WAL mode**: Enabled via `db.pragma('journal_mode = WAL')` for better concurrency
|
||||
- **Foreign keys**: Enabled via `db.pragma('foreign_keys = ON')`
|
||||
- **sqlite-vec integration**: Loaded via `sqliteVec.load(db)` for vector search capabilities
|
||||
- **FTS5 virtual table**: External content table referencing chunks for full-text search
|
||||
- **vec0 virtual table**: 1536-dimension float array for OpenAI text-embedding-3-small embeddings
|
||||
- **Test execution**: Use Node.js with tsx for TypeScript execution (not Bun runtime)
|
||||
- **Buffer handling**: Float32Array must be converted to Buffer via `Buffer.from(array.buffer)` for SQLite binding
|
||||
- **In-memory databases**: WAL mode returns 'memory' for :memory: DBs, 'wal' for file-based DBs
|
||||
- **Test coverage**: 9 tests covering table creation, data insertion, FTS5, vec0, WAL mode, and clean closure
|
||||
- **Error handling**: better-sqlite3 throws "The database connection is not open" for operations on closed DBs
|
||||
|
||||
### Node.js Test Execution
|
||||
- **Issue**: better-sqlite3 not compatible with Bun runtime (native module)
|
||||
- **Solution**: Use Node.js with tsx (TypeScript executor) for running tests
|
||||
- **Command**: `npx tsx --test src/__tests__/db.test.ts`
|
||||
- **Node.test API**: Uses `describe`, `it`, `before`, `after` from 'node:test' module
|
||||
- **Assertions**: Use `assert` from 'node:assert' module
|
||||
- **Cleanup**: Use `after()` hooks for database cleanup, not `afterEach()` (node:test difference)
|
||||
|
||||
### Embedding Provider Implementation (Task: Embeddings Module)
|
||||
- **TDD approach**: RED-GREEN-REFACTOR cycle successfully applied for embeddings module
|
||||
- **Mock database**: Created in-memory mock for testing since better-sqlite3 incompatible with Bun
|
||||
- **Float32 precision**: embeddings stored/retrieved via Float32Array has limited precision (use toBeCloseTo in tests)
|
||||
- **Cache implementation**: content_hash + model composite key in embedding_cache table
|
||||
- **Retry logic**: Exponential backoff (1s, 2s, 4s) for 429/500 errors, max 3 retries
|
||||
- **Test coverage**: 11 tests covering embed(), embedBatch(), cache hits/misses, API failures, retries, buffer conversion
|
||||
- **Helper functions**: embeddingToBuffer() and bufferToEmbedding() for Float32Array ↔ Buffer conversion
|
||||
- **Bun spyOn**: Use mockClear() to reset call count without replacing mock implementation
|
||||
- **Buffer size**: Float32 embedding stored as Buffer with size = dimensions * 4 bytes
|
||||
|
||||
### FTS5 BM25 Search Implementation (Task: FTS5 Search Module)
|
||||
- **TDD approach**: RED-GREEN-REFACTOR cycle successfully applied for search module
|
||||
- **buildFtsQuery()**: Extracts alphanumeric tokens via regex `/[A-Za-z0-9_]+/g`, quotes them, joins with AND
|
||||
- **FTS5 escaping**: Tokens are quoted to handle special characters (e.g., `"term"`)
|
||||
- **BM25 score normalization**: `bm25RankToScore(rank)` converts BM25 rank to 0-1 score using `1 / (1 + normalized)`
|
||||
- **FTS5 external content tables**: The schema uses `content='chunks', content_rowid='rowid'` but requires manual insertion into chunks_fts
|
||||
- **Test data setup**: Must manually insert into chunks_fts after inserting into chunks (external content doesn't auto-populate)
|
||||
- **BM25 ranking**: Results are ordered by `rank` column (lower rank = better match for FTS5)
|
||||
- **Error handling**: searchFTS catches SQL errors and returns empty array (graceful degradation)
|
||||
- **MaxResults parameter**: Respects LIMIT clause in SQL query
|
||||
- **SearchResult interface**: Includes id, filePath, startLine, endLine, text, contentHash, source, score (all required)
|
||||
- **Prefix matching**: FTS5 supports prefix queries automatically via token matching (e.g., "test" matches "testing")
|
||||
- **No matches**: Returns empty array when query has no valid tokens or no matches found
|
||||
- **Test coverage**: 7 tests covering basic search, exact keywords, partial words, no matches, ranking, maxResults, and metadata
|
||||
|
||||
### Hybrid Search Implementation (Task: Hybrid Search Combiner)
|
||||
- **TDD approach**: RED-GREEN-REFACTOR cycle successfully applied for hybrid search
|
||||
- **Weighted scoring**: Combined score = vectorWeight * vectorScore + textWeight * textScore (default: 0.7/0.3)
|
||||
- **Result merging**: Uses Map<string, HybridSearchResult> to merge results by chunk ID, preventing duplicates
|
||||
- **Dual-score tracking**: Each result tracks both vectorScore and textScore separately, allowing for degraded modes
|
||||
- **Graceful degradation**: Works with FTS5-only (vector search fails) or vector-only (FTS5 fails)
|
||||
- **minScore filtering**: Results below minScore threshold are filtered out after score calculation
|
||||
- **Score sorting**: Results sorted by combined score in descending order before applying maxResults limit
|
||||
- **Vector search fallback**: searchVector catches errors and returns empty array, allowing FTS5-only operation
|
||||
- **FTS5 query fallback**: searchFTS catches SQL errors and returns empty array, allowing vector-only operation
|
||||
- **Database cleanup**: beforeEach must delete from chunks_fts, chunks_vec, chunks, and files to avoid state bleed
|
||||
- **Virtual table corruption**: Deleting from FTS5/vec0 virtual tables can cause corruption - use try/catch to recreate
|
||||
- **SearchResult type conflict**: SearchResult is imported from types.ts, don't re-export in search.ts
|
||||
- **Test isolation**: Virtual tables (chunks_fts, chunks_vec) must be cleared and potentially recreated between tests
|
||||
- **Buffer conversion**: queryEmbedding converted to Buffer via Buffer.from(new Float32Array(array).buffer)
|
||||
- **Debug logging**: process.env.DEBUG_SEARCH flag enables detailed logging of FTS5 and vector search results
|
||||
- **Test coverage**: 9 tests covering combination, weighting, minScore filtering, deduplication, sorting, maxResults, degraded modes (FTS5-only, vector-only), and custom weights
|
||||
Reference in New Issue
Block a user