AGENTS/.sisyphus/notepads/opencode-memory/learnings.md

Opencode Memory Plugin — Learnings

Session: ses_3a5a47a05ffeoNYfz2RARYsHX9

Started: 2026-02-14

Architecture Decisions

• SQLite + FTS5 + vec0 replaces mem0+qdrant entirely
• Markdown at ~/CODEX/80-memory/ is source of truth
• SQLite DB at ~/.local/share/opencode-memory/index.db is derived index
• OpenAI text-embedding-3-small for embeddings (1536 dimensions)
• Hybrid search: 0.7 vector weight + 0.3 BM25 weight
• Chunking: 400 tokens, 80 overlap (tiktoken cl100k_base)

Key Patterns from Openclaw

• MemoryIndexManager pattern (1590 lines) — file watching, chunking, indexing
• Hybrid scoring with weighted combination
• Embedding cache by content_hash + model
• Two sources: "memory" (markdown files) + "sessions" (transcripts)
• Two tools: memory_search (hybrid query) + memory_get (read lines)

Technical Stack

• Runtime: bun
• Test framework: bun test (TDD)
• SQLite: better-sqlite3 (synchronous API)
• Embeddings: openai npm package
• Chunking: tiktoken (cl100k_base encoding)
• File watching: chokidar
• Validation: zod (for tool schemas)

Vec0 Extension Findings (Task 1)

• vec0 extension: NOT AVAILABLE - requires vec0.so shared library not present
• Alternative solution: sqlite-vec package (v0.1.7-alpha.2) successfully tested
• Loading mechanism: sqliteVec.load(db) loads vector extension into database
• Test result: Works with Node.js (better-sqlite3 native module compatible)
• Note: better-sqlite3 does NOT work with Bun runtime (native module incompatibility)
• Testing command: node -e "const Database = require('better-sqlite3'); const sqliteVec = require('sqlite-vec'); const db = new Database(':memory:'); sqliteVec.load(db); console.log('OK')"

Bun Runtime Limitations

• better-sqlite3 native module NOT compatible with Bun (ERR_DLOPEN_FAILED)
• Use Node.js for any code requiring better-sqlite3
• Alternative: bun:sqlite API (similar API, but not same library)

Wave Progress

• Wave 1: IN PROGRESS (Task 1)
• Wave 2-6: PENDING

Configuration Module Implementation (Task: Config Module)

• TDD approach: RED-GREEN-REFACTOR cycle successfully applied
• Pattern: Default config object + resolveConfig() function for merging
• Path expansion: expandPath() helper function handles ~ → $HOME expansion
• Test coverage: 10 tests covering defaults, overrides, path expansion, and config merging
• TypeScript best practices: Proper type exports from types.ts, type imports in config.ts
• Defaults match openclaw: chunking (400/80), search weights (0.7/0.3), minScore (0.35), maxResults (6)
• Bun test framework: Fast execution (~20ms for 10 tests), clean output

Database Schema Implementation (Task 2)

• TDD approach: RED-GREEN-REFACTOR cycle successfully applied for db module
• Schema tables: meta, files, chunks, embedding_cache, chunks_fts (FTS5), chunks_vec (vec0)
• WAL mode: Enabled via db.pragma('journal_mode = WAL') for better concurrency
• Foreign keys: Enabled via db.pragma('foreign_keys = ON')
• sqlite-vec integration: Loaded via sqliteVec.load(db) for vector search capabilities
• FTS5 virtual table: External content table referencing chunks for full-text search
• vec0 virtual table: 1536-dimension float array for OpenAI text-embedding-3-small embeddings
• Test execution: Use Node.js with tsx for TypeScript execution (not Bun runtime)
• Buffer handling: Float32Array must be converted to Buffer via Buffer.from(array.buffer) for SQLite binding
• In-memory databases: WAL mode returns 'memory' for :memory: DBs, 'wal' for file-based DBs
• Test coverage: 9 tests covering table creation, data insertion, FTS5, vec0, WAL mode, and clean closure
• Error handling: better-sqlite3 throws "The database connection is not open" for operations on closed DBs

Node.js Test Execution

• Issue: better-sqlite3 not compatible with Bun runtime (native module)
• Solution: Use Node.js with tsx (TypeScript executor) for running tests
• Command: npx tsx --test src/__tests__/db.test.ts
• Node.test API: Uses describe, it, before, after from 'node:test' module
• Assertions: Use assert from 'node:assert' module
• Cleanup: Use after() hooks for database cleanup, not afterEach() (node:test difference)

Embedding Provider Implementation (Task: Embeddings Module)

• TDD approach: RED-GREEN-REFACTOR cycle successfully applied for embeddings module
• Mock database: Created in-memory mock for testing since better-sqlite3 incompatible with Bun
• Float32 precision: embeddings stored/retrieved via Float32Array has limited precision (use toBeCloseTo in tests)
• Cache implementation: content_hash + model composite key in embedding_cache table
• Retry logic: Exponential backoff (1s, 2s, 4s) for 429/500 errors, max 3 retries
• Test coverage: 11 tests covering embed(), embedBatch(), cache hits/misses, API failures, retries, buffer conversion
• Helper functions: embeddingToBuffer() and bufferToEmbedding() for Float32Array ↔ Buffer conversion
• Bun spyOn: Use mockClear() to reset call count without replacing mock implementation
• Buffer size: Float32 embedding stored as Buffer with size = dimensions * 4 bytes

FTS5 BM25 Search Implementation (Task: FTS5 Search Module)

• TDD approach: RED-GREEN-REFACTOR cycle successfully applied for search module
• buildFtsQuery(): Extracts alphanumeric tokens via regex /[A-Za-z0-9_]+/g, quotes them, joins with AND
• FTS5 escaping: Tokens are quoted to handle special characters (e.g., "term")
• BM25 score normalization: bm25RankToScore(rank) converts BM25 rank to 0-1 score using 1 / (1 + normalized)
• FTS5 external content tables: The schema uses content='chunks', content_rowid='rowid' but requires manual insertion into chunks_fts
• Test data setup: Must manually insert into chunks_fts after inserting into chunks (external content doesn't auto-populate)
• BM25 ranking: Results are ordered by rank column (lower rank = better match for FTS5)
• Error handling: searchFTS catches SQL errors and returns empty array (graceful degradation)
• MaxResults parameter: Respects LIMIT clause in SQL query
• SearchResult interface: Includes id, filePath, startLine, endLine, text, contentHash, source, score (all required)
• Prefix matching: FTS5 supports prefix queries automatically via token matching (e.g., "test" matches "testing")
• No matches: Returns empty array when query has no valid tokens or no matches found
• Test coverage: 7 tests covering basic search, exact keywords, partial words, no matches, ranking, maxResults, and metadata

Hybrid Search Implementation (Task: Hybrid Search Combiner)

• TDD approach: RED-GREEN-REFACTOR cycle successfully applied for hybrid search
• Weighted scoring: Combined score = vectorWeight * vectorScore + textWeight * textScore (default: 0.7/0.3)
• Result merging: Uses Map<string, HybridSearchResult> to merge results by chunk ID, preventing duplicates
• Dual-score tracking: Each result tracks both vectorScore and textScore separately, allowing for degraded modes
• Graceful degradation: Works with FTS5-only (vector search fails) or vector-only (FTS5 fails)
• minScore filtering: Results below minScore threshold are filtered out after score calculation
• Score sorting: Results sorted by combined score in descending order before applying maxResults limit
• Vector search fallback: searchVector catches errors and returns empty array, allowing FTS5-only operation
• FTS5 query fallback: searchFTS catches SQL errors and returns empty array, allowing vector-only operation
• Database cleanup: beforeEach must delete from chunks_fts, chunks_vec, chunks, and files to avoid state bleed
• Virtual table corruption: Deleting from FTS5/vec0 virtual tables can cause corruption - use try/catch to recreate
• SearchResult type conflict: SearchResult is imported from types.ts, don't re-export in search.ts
• Test isolation: Virtual tables (chunks_fts, chunks_vec) must be cleared and potentially recreated between tests
• Buffer conversion: queryEmbedding converted to Buffer via Buffer.from(new Float32Array(array).buffer)
• Debug logging: process.env.DEBUG_SEARCH flag enables detailed logging of FTS5 and vector search results
• Test coverage: 9 tests covering combination, weighting, minScore filtering, deduplication, sorting, maxResults, degraded modes (FTS5-only, vector-only), and custom weights