173 Commits

Author SHA1 Message Date
Alex Newman
eaba21329c Refactor hooks codebase: reduce complexity and improve maintainability (#204)
* refactor: Clean up hook-response and new-hook files

- Removed 'PreCompact' hook type and associated logic from hook-response.ts for improved type safety.
- Deleted extensive architecture comments in new-hook.ts to streamline code readability.
- Simplified debug logging in new-hook.ts to reduce verbosity.
- Enhanced ensureWorkerRunning function in worker-utils.ts with a final health check before throwing errors.
- Added a new documentation file outlining the hooks cleanup process and future improvements.

* Refactor cleanup and user message hooks

- Updated cleanup-hook.js to improve error handling and remove unnecessary input checks.
- Simplified user-message-hook.js by removing time-sensitive announcements and streamlining output.
- Enhanced logging functionality in both hooks for better debugging and clarity.

* Refactor error handling in hooks to use centralized error handler

- Introduced `handleWorkerError` function in `src/shared/hook-error-handler.ts` to manage worker-related errors.
- Updated `context-hook.ts`, `new-hook.ts`, `save-hook.ts`, and `summary-hook.ts` to utilize the new error handler, simplifying error management and improving code readability.
- Removed repetitive error handling logic from individual hooks, ensuring consistent user-friendly messages for connection issues.

* Refactor user-message and summary hooks to utilize shared transcript parser; introduce hook exit codes

- Moved user message extraction logic to a new shared module `transcript-parser.ts` for better code reuse.
- Updated `summary-hook.ts` to use the new `extractLastMessage` function for retrieving user and assistant messages.
- Replaced direct exit code usage in `user-message-hook.ts` with constants from `hook-constants.ts` for improved readability and maintainability.
- Added `HOOK_EXIT_CODES` to `hook-constants.ts` to standardize exit codes across hooks.

* Refactor hook input interfaces to enforce required fields

- Updated `SessionStartInput`, `UserPromptSubmitInput`, `PostToolUseInput`, and `StopInput` interfaces to require `session_id`, `transcript_path`, and `cwd` fields, ensuring better type safety and clarity in hook inputs.
- Removed optional index signatures from these interfaces to prevent unintended properties and improve code maintainability.
- Adjusted related hook implementations to align with the new interface definitions.

* Refactor save-hook to remove tool skipping logic; enhance summary-hook to handle spinner stopping with error logging; update SessionRoutes to load skip tools from settings; add CLAUDE_MEM_SKIP_TOOLS to SettingsDefaultsManager for configurable tool exclusion.

* Document CLAUDE_MEM_SKIP_TOOLS setting in public docs

Added documentation for the new CLAUDE_MEM_SKIP_TOOLS configuration
setting in response to PR review feedback. Users can now discover and
customize which tools are excluded from observations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-09 22:45:22 -05:00
Alex Newman
005a80c540 Refactor SettingsDefaultsManager: Move to shared directory and update imports
- Moved SettingsDefaultsManager from worker/settings to shared directory.
- Updated all import paths across the codebase to reflect the new location.
- Removed early-settings.ts as its functionality is now handled by SettingsDefaultsManager.
- Adjusted logger and paths to utilize SettingsDefaultsManager for configuration values.
2025-12-09 15:29:17 -05:00
Alex Newman
c3761a2204 Refactor silent debugging to happy path error handling
- Replaced instances of silentDebug with happy_path_error__with_fallback across multiple files to improve error logging and handling.
- Updated the utility function to provide clearer semantics for error handling when expected values are missing.
- Introduced a script to find potential silent failures in the codebase that may need to be addressed with the new error handling approach.
2025-12-09 15:09:44 -05:00
Alex Newman
fc5c2d5e07 Refactor settings management to use ~/.claude-mem/settings.json
- Updated paths in troubleshooting documentation to reflect new settings file location.
- Modified diagnostics and reference files to read from ~/.claude-mem/settings.json.
- Introduced getWorkerPort utility for cleaner worker port retrieval.
- Enhanced ChromaSync and SDKAgent to load Python version and Claude path from settings.
- Updated SettingsRoutes to validate new settings: CLAUDE_MEM_LOG_LEVEL and CLAUDE_MEM_PYTHON_VERSION.
- Added early-settings module to load settings for logger and other early-stage modules.
- Adjusted logger to use early-loaded log level setting.
- Refactored paths to utilize early-loaded data directory setting.
2025-12-09 12:23:33 -05:00
Alex Newman
4321add69c refactor: rename search-server to mcp-server throughout codebase
- Updated all documentation references from search-server to mcp-server
- Removed legacy search-server.cjs file
- Updated debug log messages to use [mcp-server] prefix
- Updated build output references in docs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-09 01:06:43 -05:00
Alex Newman
9a27f380c3 docs: lint and publish platform integration guide
- Fix MDX parsing error in search-architecture.mdx (line 382: changed <10ms to "under 10ms")
- Fix broken internal links:
  - architecture-evolution.mdx: VIEWER → /architecture/worker-service
  - usage/private-tags.mdx: search-tools → /usage/search-tools
  - usage/private-tags.mdx: getting-started → /usage/getting-started
- Verify all links pass mintlify broken-links check
- Confirm platform-integration.mdx renders correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-08 22:40:34 -05:00
Alex Newman
577cac8831 Add Platform Integration Guide for claude-mem worker service
- Introduced comprehensive documentation for integrating claude-mem into VSCode extensions, IDE plugins, and CLI tools.
- Detailed worker service basics, including environment variables and build commands.
- Provided an overview of worker architecture and request flow.
- Documented API reference for session lifecycle, data retrieval, search operations, and settings configuration.
- Included integration patterns, error handling strategies, and development workflow guidelines.
- Added critical implementation notes and additional resources for developers.
2025-12-08 22:36:00 -05:00
Alex Newman
8da92c6569 docs: update hooks.mdx with improved architecture diagrams and data flow representation 2025-12-08 21:54:48 -05:00
Alex Newman
a18b43744c docs: comprehensive hook lifecycle guide for platform implementers
Rewrote architecture/hooks.mdx with complete technical reference including:
- 5-stage lifecycle overview with ASCII architecture diagram
- Detailed input/output specs for each hook
- Processing steps with TypeScript code examples
- Data flow diagram showing complete request lifecycle
- Session ID threading explanation
- Privacy tag stripping pipeline
- SDK agent processing details
- Implementation checklist for other platforms
- Common pitfalls and solutions table

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-08 21:49:55 -05:00
Alex Newman
be28c095e2 Release v6.5.1: Product Hunt Launch Day UI Updates
- Decorative Product Hunt announcement in terminal with rocket borders
- Product Hunt badge in viewer header with theme-aware switching
- Badge uses separate tracking URL for analytics

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-04 13:29:56 -05:00
Alex Newman
c6708b3684 Release v6.5.0: Documentation Overhaul
Comprehensive documentation update with current features:
- Updated "What's New" section to highlight v6.4.x features
- Added privacy tags and context configuration to key features
- Fixed default model (claude-haiku-4-5)
- Clarified lifecycle hook count (5 events, 6 scripts)
- Removed outdated MCP server references
- Updated version numbers across all docs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-03 21:47:59 -05:00
copilot-swe-agent[bot]
2b683f99bb fix: Update search-server references from .mjs to .cjs to match actual build output
- Update plugin/.mcp.json to reference search-server.cjs
- Update docs/public/configuration.mdx to reference search-server.cjs
- Update docs/public/development.mdx to reference search-server.cjs
- Remove stale plugin/scripts/search-server.mjs file

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>
2025-12-03 22:17:12 +00:00
Alex Newman
2b223b7cd9 feat: Add dual-tag system for meta-observation control (#153)
* feat: Add dual-tag system for meta-observation control

Implements <private> and <claude-mem-context> tag stripping at hook layer
to give users fine-grained control over what gets persisted in observations
and enable future real-time context injection without recursive storage.

**Features:**
- stripMemoryTags() function in save-hook.ts
- Strips both <private> and <claude-mem-context> tags before sending to worker
- Always active (no configuration needed)
- Comprehensive test suite (19 tests, all passing)
- User documentation for <private> tag
- Technical architecture documentation

**Architecture:**
- Edge processing pattern (filter at hook, not worker)
- Defensive type handling with silentDebug
- Supports multiline, nested, and multiple tags
- Enables strategic orchestration for internal tools

**User-Facing:**
- <private> tag for manual privacy control (documented)
- Prevents sensitive data from persisting in observations

**Infrastructure:**
- <claude-mem-context> tag ready for real-time context feature
- Prevents recursive storage when context injection ships

**Files:**
- src/hooks/save-hook.ts: Core implementation
- tests/strip-memory-tags.test.ts: Test suite (19/19 passing)
- docs/public/usage/private-tags.mdx: User guide
- docs/public/docs.json: Navigation update
- docs/context/dual-tag-system-architecture.md: Technical docs
- plugin/scripts/save-hook.js: Built hook

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Strip private tags from user prompts and skip memory ops for fully private prompts

Fixes critical privacy bug where <private> tags were not being stripped from
user prompts before storage in user_prompts table, making private content
searchable via mem-search.

Changes:

1. new-hook.ts: Skip memory operations for fully private prompts
   - If cleaned prompt is empty after stripping tags, skip saveUserPrompt
   - Skip worker init to avoid wasting resources on empty prompts
   - Logs: "(fully private - skipped)"

2. save-hook.ts: Skip observations for fully private prompts
   - Check if user prompt was entirely private before creating observations
   - Respects user intent: fully private prompt = no observations at all
   - Prevents "thoughts pop up" issue where private prompts create public observations

3. SessionStore.ts: Add getUserPrompt() method
   - Retrieves prompt text by session_id and prompt_number
   - Used by save-hook to check if prompt was private

4. Tests: Added 4 new tests for fully private prompt detection (16 total, all passing)

5. Docs: Updated private-tags.mdx to reflect correct behavior
   - User prompts ARE now filtered before storage
   - Private content never reaches database or search indices

Privacy Protection:
- Fully private prompts: No user_prompt saved, no worker init, no observations
- Partially private prompts: Tags stripped, content sanitized before storage
- Zero leaks: Private content never indexed or searchable

Addresses reviewer feedback on PR #153 about user prompt filtering.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Enhance memory tag handling and indexing in user prompts

- Added a new index `idx_user_prompts_lookup` on `user_prompts` for improved query performance based on `claude_session_id` and `prompt_number`.
- Refactored memory tag stripping functionality into dedicated utility functions: `stripMemoryTagsFromJson` and `stripMemoryTagsFromPrompt` for better separation of concerns and reusability.
- Updated hooks (`new-hook.ts` and `save-hook.ts`) to utilize the new tag stripping functions, ensuring private content is not stored or searchable.
- Removed redundant inline tag stripping functions from hooks to streamline code.
- Added tests for the new tag stripping utilities to ensure functionality and prevent regressions.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-30 22:57:26 -05:00
Alex Newman
01be3156fb fix: context hook updates and cleanup (#150)
* fix(context-hook): update savings message to reference mem-search skill

Changed "Use claude-mem search" to "Use the mem-search skill" for clarity.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: delete outdated docs, experiments, and test results

Removed:
- docs/context/ (moved to private/)
- experiment/ (obsolete)
- test-results/ (stale)
- tests/ (outdated)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(user-message-hook): update support link to Discord community

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-28 20:17:44 -05:00
Alex Newman
4e5913611a feat(search-server): enhance decision search with optional semantic query support
- Updated the 'decisions' tool to accept an optional 'query' parameter for semantic filtering.
- Implemented logic to handle semantic search using Chroma when a query is provided.
- Preserved ranking order of results based on Chroma's output.
- Added fallback to metadata-first search when no query is present.
2025-11-25 16:37:08 -05:00
Alex Newman
29e6e026b6 feat: Add beta channel for experimental features and update documentation 2025-11-25 15:44:52 -05:00
Alex Newman
9285826547 feat: implement Endless Mode for real-time context compression in Claude sessions 2025-11-16 23:19:43 -05:00
Alex Newman
c0778bef00 docs: Align search documentation with hybrid ChromaDB architecture (#116)
* feat: Add discovery_tokens for ROI tracking in observations and session summaries

- Introduced `discovery_tokens` column in `observations` and `session_summaries` tables to track token costs associated with discovering and creating each observation and summary.
- Updated relevant services and hooks to calculate and display ROI metrics based on discovery tokens.
- Enhanced context economics reporting to include savings from reusing previous observations.
- Implemented migration to ensure the new column is added to existing tables.
- Adjusted data models and sync processes to accommodate the new `discovery_tokens` field.

* refactor: streamline context hook by removing unused functions and updating terminology

- Removed the estimateTokens and getObservations helper functions as they were not utilized.
- Updated the legend and output messages to replace "discovery" with "work" for clarity.
- Changed the emoji representation for different observation types to better reflect their purpose.
- Enhanced output formatting for improved readability and understanding of token usage.

* Refactor user-message-hook and context-hook for improved clarity and functionality

- Updated user-message-hook.js to enhance error messaging and improve variable naming for clarity.
- Modified context-hook.ts to include a new column key section, improved context index instructions, and added emoji icons for observation types.
- Adjusted footer messages in context-hook.ts to emphasize token savings and access to past research.
- Changed user-message-hook.ts to update the feedback and support message for clarity.

* fix: Critical ROI tracking fixes from PR review

Addresses critical findings from PR #111 review:

1. **Fixed incorrect discovery token calculation** (src/services/worker/SDKAgent.ts)
   - Changed from passing cumulative total to per-response delta
   - Now correctly tracks token cost for each observation/summary
   - Captures token state before/after response processing
   - Prevents all observations getting inflated cumulative values

2. **Fixed schema version mismatch** (src/services/sqlite/SessionStore.ts)
   - Changed ensureDiscoveryTokensColumn() from version 11 to version 7
   - Now matches migration007 definition in migrations.ts
   - Ensures consistent version tracking across migration system

These fixes ensure ROI metrics accurately reflect token costs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Update search documentation to reflect hybrid ChromaDB architecture

The backend correctly implements ChromaDB-first semantic search with SQLite
temporal ordering and FTS5 fallback, but documentation incorrectly described
it as "FTS5 full-text search". This fix aligns all skill guides and tool
descriptions with the actual implementation.

Changes:
- Update SKILL.md to describe hybrid architecture with ChromaDB primary
- Update observations.md title and query parameter descriptions
- Update all three search tool descriptions in search-server.ts:
  * search_observations
  * search_sessions
  * search_user_prompts

All tools now correctly document:
- ChromaDB semantic search (primary ranking)
- 90-day recency filter
- SQLite temporal ordering
- FTS5 fallback (when ChromaDB unavailable)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Add discovery_tokens column to observations and session_summaries tables

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-16 13:36:17 -05:00
Alex Newman
f5b25e8fc4 docs: update user-facing documentation for v6.0.0
- Updated version badge in README.md to 6.0.0
- Updated 'What's New' sections in README.md and introduction.mdx
- Highlighted major session management and transcript processing improvements
- Removed restrictive permissions from .claude/settings.json

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 20:53:59 -05:00
Alex Newman
68290a9121 Performance improvements: Token reduction and enhanced summaries (#101)
* refactor: Reduce continuation prompt token usage by 95 lines

Removed redundant instructions from continuation prompt that were originally
added to mitigate a session continuity issue. That issue has since been
resolved, making these detailed instructions unnecessary on every continuation.

Changes:
- Reduced continuation prompt from ~106 lines to ~11 lines (~95 line reduction)
- Changed "User's Goal:" to "Next Prompt in Session:" (more accurate framing)
- Removed redundant WHAT TO RECORD, WHEN TO SKIP, and OUTPUT FORMAT sections
- Kept concise reminder: "Continue generating observations and progress summaries..."
- Initial prompt still contains all detailed instructions

Impact:
- Significant token savings on every continuation prompt
- Faster context injection with no loss of functionality
- Instructions remain comprehensive in initial prompt

Files modified:
- src/sdk/prompts.ts (buildContinuationPrompt function)
- plugin/scripts/worker-service.cjs (compiled output)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Enhance observation and summary prompts for clarity and token efficiency

* Enhance prompt clarity and instructions in prompts.ts

- Added a reminder to think about instructions before starting work.
- Simplified the continuation prompt instruction by removing "for this ongoing session."

* feat: Enhance settings.json with permissions and deny access to sensitive files

refactor: Remove PLAN-full-observation-display.md and PR_SUMMARY.md as they are no longer needed

chore: Delete SECURITY_SUMMARY.md since it is redundant after recent changes

fix: Update worker-service.cjs to streamline observation generation instructions

cleanup: Remove src-analysis.md and src-tree.md for a cleaner codebase

refactor: Modify prompts.ts to clarify instructions for memory processing

* refactor: Remove legacy worker service implementation

* feat: Enhance summary hook to extract last assistant message and improve logging

- Added function to extract the last assistant message from the transcript.
- Updated summary hook to include last assistant message in the summary request.
- Modified SDKSession interface to store last assistant message.
- Adjusted buildSummaryPrompt to utilize last assistant message for generating summaries.
- Updated worker service and session manager to handle last assistant message in summarize requests.
- Introduced silentDebug utility for improved logging and diagnostics throughout the summary process.

* docs: Add comprehensive implementation plan for ROI metrics feature

Added detailed implementation plan covering:
- Token usage capture from Agent SDK
- Database schema changes (migration #8)
- Discovery cost tracking per observation
- Context hook display with ROI metrics
- Testing and rollout strategy

Timeline: ~20 hours over 4 days
Goal: Empirical data for YC application amendment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add transcript processing scripts for analysis and formatting

- Implemented `dump-transcript-readable.ts` to generate a readable markdown dump of transcripts, excluding certain entry types.
- Created `extract-rich-context-examples.ts` to extract and showcase rich context examples from transcripts, highlighting user requests and assistant reasoning.
- Developed `format-transcript-context.ts` to format transcript context into a structured markdown format for improved observation generation.
- Added `test-transcript-parser.ts` for validating data extraction from transcript JSONL files, including statistics and error reporting.
- Introduced `transcript-to-markdown.ts` for a complete representation of transcript data in markdown format, showing all context data.
- Enhanced type definitions in `transcript.ts` to support new features and ensure type safety.
- Built `transcript-parser.ts` to handle parsing of transcript JSONL files, including error handling and data extraction methods.

* Refactor hooks and SDKAgent for improved observation handling

- Updated `new-hook.ts` to clean user prompts by stripping leading slashes for better semantic clarity.
- Enhanced `save-hook.ts` to include additional tools in the SKIP_TOOLS set, preventing unnecessary observations from certain command invocations.
- Modified `prompts.ts` to change the structure of observation prompts, emphasizing the observational role and providing a detailed XML output format for observations.
- Adjusted `SDKAgent.ts` to enforce stricter tool usage restrictions, ensuring the memory agent operates solely as an observer without any tool access.

* feat: Enhance session initialization to accept user prompts and prompt numbers

- Updated `handleSessionInit` in `worker-service.ts` to extract `userPrompt` and `promptNumber` from the request body and pass them to `initializeSession`.
- Modified `initializeSession` in `SessionManager.ts` to handle optional `currentUserPrompt` and `promptNumber` parameters.
- Added logic to update the existing session's `userPrompt` and `lastPromptNumber` if a `currentUserPrompt` is provided.
- Implemented debug logging for session initialization and updates to track user prompts and prompt numbers.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-13 18:22:44 -05:00
Alex Newman
ab5d78717f Merge remote-tracking branch 'refs/remotes/origin/main' 2025-11-12 15:59:02 -05:00
Alex Newman
cb4aea57a8 feat: Add comprehensive documentation for Language Model Tool API and related resources 2025-11-12 15:58:44 -05:00
Copilot
7bdf6dbfe1 Align user-facing documentation with v5.5.1 codebase state (#99)
* Initial plan

* Update documentation to reflect v5.5.1 state and mem-search skill

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>

* Update hooks documentation to clarify 6 hooks + pre-hook architecture

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>

* Complete documentation alignment with mem-search skill naming

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>

* Fix remaining old skill path references in troubleshooting docs

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>

* Documentation alignment complete - all tests pass

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>

* Fix hallucinated /skill command references - skills are auto-invoked

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>
2025-11-12 14:59:04 -05:00
Alex Newman
1c9da73d5f feat: Add VSCode Language Model Tool API documentation 2025-11-12 13:39:46 -05:00
basher83
97d565e3cd Replace search skill with mem-search (#91)
* feat: add mem-search skill with progressive disclosure architecture

Add comprehensive mem-search skill for accessing claude-mem's persistent
cross-session memory database. Implements progressive disclosure workflow
and token-efficient search patterns.

Features:
- 12 search operations (observations, sessions, prompts, by-type, by-concept, by-file, timelines, etc.)
- Progressive disclosure principles to minimize token usage
- Anti-patterns documentation to guide LLM behavior
- HTTP API integration for all search functionality
- Common workflows with composition examples

Structure:
- SKILL.md: Entry point with temporal trigger patterns
- principles/: Progressive disclosure + anti-patterns
- operations/: 12 search operation files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add CHANGELOG entry for mem-search skill

Document mem-search skill addition in Unreleased section with:
- 100% effectiveness compliance metrics
- Comparison to previous search skill implementation
- Progressive disclosure architecture details
- Reference to audit report documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add mem-search skill audit report

Add comprehensive audit report validating mem-search skill against
Anthropic's official skill-creator documentation.

Report includes:
- Effectiveness metrics comparison (search vs mem-search)
- Critical issues analysis for production readiness
- Compliance validation across 6 key dimensions
- Reference implementation guidance

Result: mem-search achieves 100% compliance vs search's 67%

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add comprehensive search architecture analysis document

- Document current state of dual search architectures (HTTP API and MCP)
- Analyze HTTP endpoints and MCP search server architectures
- Identify DRY violations across search implementations
- Evaluate the use of curl as the optimal approach for search
- Provide architectural recommendations for immediate and long-term improvements
- Outline action plan for cleanup, feature parity, DRY refactoring

* refactor: Remove deprecated search skill documentation and operations

* refactor: Reorganize documentation into public and context directories

Changes:
- Created docs/public/ for Mintlify documentation (.mdx files)
- Created docs/context/ for internal planning and implementation docs
- Moved all .mdx files and assets to docs/public/
- Moved all internal .md files to docs/context/
- Added CLAUDE.md to both directories explaining their purpose
- Updated docs.json paths to work with new structure

Benefits:
- Clear separation between user-facing and internal documentation
- Easier to maintain Mintlify docs in dedicated directory
- Internal context files organized separately

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Enhance session management and continuity in hooks

- Updated new-hook.ts to clarify session_id threading and idempotent session creation.
- Modified prompts.ts to require claudeSessionId for continuation prompts, ensuring session context is maintained.
- Improved SessionStore.ts documentation on createSDKSession to emphasize idempotent behavior and session connection.
- Refined SDKAgent.ts to detail continuation prompt logic and its reliance on session.claudeSessionId for unified session handling.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alex Newman <thedotmack@gmail.com>
2025-11-11 16:15:07 -05:00
Alex Newman
5d64df2ba5 docs: Update all documentation to reflect v5.4.0 skill-based search
Documentation Updates:
- README.md: Updated version badge, What's New, and search section
- docs/usage/search-tools.mdx: Rewrote for skill-based natural language approach
- docs/architecture/mcp-search.mdx → search-architecture.mdx: Complete rewrite
- docs/architecture/overview.mdx: Updated components and search pipeline
- docs/usage/getting-started.mdx: Added skill-based search section
- docs/configuration.mdx: Updated search configuration for v5.4.0
- docs/introduction.mdx: Updated key features
- docs/docs.json: Updated navigation to search-architecture

Key Changes:
- Emphasized ~2,250 token savings per session start
- Converted all examples to natural language queries
- Documented 10 HTTP API endpoints
- Explained progressive disclosure pattern
- Added migration notes (transparent, no user action required)
- Removed outdated MCP references

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 19:14:05 -05:00
Alex Newman
0475a57fb1 feat: Migrate to Skill-Based Search and enhance context handling
- Implemented a new skill-based search approach, replacing MCP search tools to reduce session start context from ~2,500 tokens to ~250 tokens.
- Added 10 new HTTP API endpoints to the worker service for various search functionalities.
- Created a new search skill with detailed instructions and examples for users.
- Removed MCP search server references and updated documentation to reflect the new skill-based approach.
- Enhanced context handling in the context hook to conditionally display the most recent summary based on observations.
- Introduced comprehensive documentation for Agent Skills, including creation, management, and best practices.
2025-11-09 17:34:00 -05:00
copilot-swe-agent[bot]
e029903a66 Add troubleshooting slash command skill
Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>
2025-11-08 21:47:03 +00:00
Alex Newman
809c9e6639 Implement code changes to enhance functionality and improve performance 2025-11-07 21:51:04 -05:00
claude[bot]
012774b83c docs: add CLAUDE.md explaining docs folder structure
Clarifies that docs/ is a Mintlify documentation site for official
user-facing documentation (.mdx files), while context/ should contain
planning documents, design docs, and internal references.

Lists files currently in docs/ that should be moved to context/:
- typescript-errors.md
- worker-service-*.md files
- processing-indicator-*.md files
- CHROMA.md and chroma-search-completion-plan.md

Co-authored-by: Alex Newman <thedotmack@users.noreply.github.com>
2025-11-07 23:23:28 +00:00
Alex Newman
980151a50e feat: Implement Worker Service v2 with improved architecture
- Complete rewrite of the Worker Service following object-oriented principles.
- Introduced a single long-lived database connection to reduce overhead.
- Implemented event-driven queues to eliminate polling.
- Added DRY utilities for pagination and settings management.
- Reduced code size significantly from 1173 lines to approximately 600-700 lines.
- Created various composed services including DatabaseManager, SessionManager, and SDKAgent.
- Enhanced SSE broadcasting capabilities for real-time client updates.
- Established a structured approach for session lifecycle management and event handling.
- Introduced type safety with shared TypeScript interfaces for better maintainability.
2025-11-06 23:56:25 -05:00
Alex Newman
9eddc51979 feat: Conduct comprehensive overhead analysis of worker service
- Added detailed documentation on performance issues within `worker-service.ts`.
- Identified high severity issues including unnecessary polling, artificial delays, and redundant database operations.
- Provided recommendations for immediate and long-term improvements to enhance performance and reduce complexity.
- Suggested architectural changes to replace polling with event-driven patterns and optimize database connection handling.
2025-11-06 23:10:33 -05:00
Alex Newman
f45c782c07 Refactor SDK exports and remove unused functions; update search-server formatting functions to eliminate unused parameters; adjust SQLite migration function parameter; add TypeScript error documentation with detailed fixes and priority order. 2025-11-06 22:27:12 -05:00
Alex Newman
3030f518b5 refactor: Complete rewrite of worker-utils.ts and cleanup of worker-service.ts
- Removed fragile PM2 string parsing and replaced with direct PM2 restart logic.
- Eliminated silent error handling in worker-utils.ts for better error visibility.
- Extracted duplicated session auto-creation logic into a new helper method getOrCreateSession() in worker-service.ts.
- Centralized configuration values and replaced magic numbers with named constants.
- Updated health check logic to ensure worker is restarted if unhealthy.
- Removed unnecessary getWorkerPort() wrapper function.
- Improved overall code quality and maintainability by applying DRY and YAGNI principles.
2025-11-06 22:00:07 -05:00
Alex Newman
b68ea38bcc Fix all broken internal links in documentation
Changes:
- Fixed 11 broken internal links across 3 documentation files
- Updated relative paths to use correct format without /docs/ prefix
- Removed broken CHANGELOG.md anchor links (mintlify doesn't support external file anchors)
- Changed /docs/progressive-disclosure → progressive-disclosure
- Changed /docs/hooks-architecture → hooks-architecture
- Changed /docs/context-engineering → context-engineering
- Changed /docs/architecture → architecture/overview
- Changed /docs/worker-service → architecture/worker-service
- Changed /docs/viewer-ui → VIEWER
- Verified with mintlify broken-links: 0 broken links remaining

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-06 14:03:44 -05:00
Alex Newman
12459eab3b docs: comprehensive v5.1.2 documentation update
This commit brings all documentation up to date with the current v5.1.2
codebase, addressing 12+ critical discrepancies and adding 2 major new
documentation files.

## Files Modified (18 documentation files):

### Root Documentation:
- README.md: Updated version badge (4.3.1 → 5.1.2), tool count (7 → 9),
  added viewer UI and theme toggle features, updated "What's New" section
- CHANGELOG.md: Added 8 missing releases (v4.3.2 through v5.1.2) with
  comprehensive release notes
- CLAUDE.md: Removed hardcoded personal paths, documented all 14 worker
  endpoints (was 8), added Chroma integration overview, updated v5.x releases

### Mintlify Documentation (docs/):
- introduction.mdx: Updated search tool count to 9, added viewer UI and
  theme toggle to features
- configuration.mdx: Added smart-install.js documentation, clarified data
  directory locations, added CLAUDE_CODE_PATH env var, explained observations
  vs sessions, updated hook configuration examples
- development.mdx: Added comprehensive viewer UI development section (103 lines),
  updated build output filenames (search-server.mjs)
- usage/search-tools.mdx: Added get_context_timeline and get_timeline_by_query
  documentation with examples, updated tool count to 9
- architecture/overview.mdx: Updated to 7 hook files, 9 search tools, added
  Chroma to tech stack, enhanced component details with viewer UI
- architecture/hooks.mdx: Added smart-install.js and user-message-hook.js
  documentation, updated hook count to 7
- architecture/worker-service.mdx: Documented all 14 endpoints organized by
  category (Viewer & Health, Data Retrieval, Settings, Session Management)
- architecture/mcp-search.mdx: Added timeline tools documentation, updated
  tool count to 9, fixed filename references (search-server.mjs)
- architecture-evolution.mdx: Added complete v5.x release history (v5.0.0
  through v5.1.2), updated title to "v3 to v5"
- hooks-architecture.mdx: Updated to "Seven Hook Scripts", added smart-install
  and user-message-hook documentation
- troubleshooting.mdx: Added v5.x specific issues section (viewer, theme toggle,
  SSE, Chroma, PM2 Windows fix)

### New Documentation Files:
- docs/VIEWER.md: Complete 400+ line guide to web viewer UI including architecture,
  features, usage, development, API integration, performance considerations
- docs/CHROMA.md: Complete 450+ line guide to vector database integration including
  hybrid search architecture, semantic search explanation, performance benchmarks,
  installation, configuration, troubleshooting

## Key Corrections Made:

1.  Updated version badges and references: 4.3.1 → 5.1.2
2.  Corrected search tool count: 7 → 9 (added get_context_timeline, get_timeline_by_query)
3.  Fixed MCP server filename: search-server.js → search-server.mjs
4.  Updated hook count: 5 → 7 (added smart-install.js, user-message-hook.js)
5.  Documented all 14 worker endpoints (was 8, incorrectly claimed 6 were missing)
6.  Removed hardcoded personal file paths
7.  Added Chroma vector database documentation
8.  Added viewer UI comprehensive documentation
9.  Updated CHANGELOG with all missing v4.3.2-v5.1.2 releases
10.  Clarified data directory locations (production vs development)
11.  Added smart-install.js caching system documentation
12.  Updated SessionStart hook configuration examples

## Documentation Statistics:

- Total files modified: 18
- New files created: 2
- Lines added: ~2,000+
- Version mismatches fixed: 2 critical
- Missing features documented: 5+ major
- Missing tools documented: 2 MCP tools
- Missing endpoints documented: 6 API endpoints

## Impact:

Documentation now accurately reflects the current v5.1.2 codebase with:
- Complete viewer UI documentation (v5.1.0)
- Theme toggle feature (v5.1.2)
- Hybrid search architecture with Chroma (v5.0.0)
- Smart install caching (v5.0.3)
- All 7 hook scripts documented
- All 9 MCP search tools documented
- All 14 worker service endpoints documented
- Comprehensive troubleshooting for v5.x issues

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-06 13:59:10 -05:00
Alex Newman
a46a028ddb Refactor worker management and cleanup hooks
- Removed ensureWorkerRunning calls from multiple hooks (cleanup, context, new, save, summary) to streamline code and avoid unnecessary checks.
- Introduced fixed port usage for worker communication across hooks.
- Enhanced error handling in newHook, saveHook, and summaryHook to provide clearer messages for worker connection issues.
- Updated worker service to start without health checks, relying on PM2 for management.
- Cached Claude executable path to optimize repeated calls.
- Improved logging for better traceability of worker actions and errors.
2025-11-04 14:21:19 -05:00
Alex Newman
5169cfa46d Merge branch 'main' into feature/hybrid-search
Resolved conflicts by:
- Keeping feature/hybrid-search build process documentation in CLAUDE.md
- Removing deleted plugin/scripts/search-server.js (intentionally deleted in feature branch)
- Removing usage logging from worker-service.ts (telemetry captured at SDK level)
- Rebuilt worker-service.cjs after resolving source file conflicts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-03 19:15:18 -05:00
Alex Newman
03ba89b703 fix: update user prompt formatting and correct 90-day cutoff logic
- Changed user prompt formatting to use full text instead of truncated version.
- Updated date filtering logic to use milliseconds instead of seconds for 90-day cutoff.
- Renamed doc_type values in ChromaSync to ensure consistency and prevent deduplication issues.
- Improved documentation for concept tags in input schema.
2025-11-03 19:05:12 -05:00
Alex Newman
633f89a5fb feat: Implement user prompt syncing to Chroma and enhance timeline querying
- Added `getObservationById` method to retrieve observations by ID in SessionStore.
- Introduced `getSessionSummariesByIds` and `getUserPromptsByIds` methods for fetching session summaries and user prompts by IDs.
- Developed `getTimelineAroundTimestamp` and `getTimelineAroundObservation` methods to provide a unified timeline of observations, sessions, and prompts around a specified anchor point.
- Enhanced ChromaSync to format and sync user prompts, including a new `syncUserPrompt` method.
- Updated WorkerService to sync the latest user prompt to Chroma after updating the worker port.
- Created tests for timeline querying and MCP handler logic to ensure functionality.
- Documented the implementation plan for user prompts and timeline context tool in the Chroma search completion plan.
2025-11-03 16:55:33 -05:00
copilot-swe-agent[bot]
192adf5cbc Fix: /clear DOES inject context via SessionStart hook
Corrected incorrect documentation that claimed /clear doesn't inject context.
The SessionStart hook fires with source: "clear" when /clear is used, which
re-injects context from recent sessions. Updated docs to accurately reflect
that /clear both clears conversation AND re-injects fresh context.

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>
2025-10-31 21:28:54 +00:00
copilot-swe-agent[bot]
140a5f3469 Clarify Stop hook behavior and /clear vs exit in docs
- Changed "Stop Claude" to "Claude finishes responding" to clarify that the Stop hook fires automatically
- Explained Stop hook triggers summary generation automatically, not as a user action
- Added detailed section explaining /clear behavior vs completely exiting Claude Code
- Clarified when new context is injected (new session start, not after /clear)
- Fixed session count: changed "last 3 sessions" to "last 10 sessions" (matches implementation)

Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>
2025-10-31 21:06:59 +00:00
Alex Newman
b0fae0cfd4 Add stderr test hook for UI experiment 2025-10-26 22:29:43 -04:00
Alex Newman
44b69b737b Remove memory toggle feature documentation as it is no longer relevant 2025-10-26 00:44:31 -04:00
Alex Newman
56213ef84a Fix: Update npm loglevel to silent in SessionStart hook to prevent context injection issues; consolidate hooks architecture and update documentation for v4.3.1 2025-10-26 00:44:04 -04:00
Alex Newman
9f3bf55c76 refactor: Remove hardcoded paths for project and Claude Code executable in various scripts, fixes issue #23 2025-10-25 13:28:20 -04:00
Alex Newman
e18f02e2af plan: Implement memory toggle feature for pausing/resuming recording 2025-10-25 01:51:42 -04:00
Alex Newman
81fdf28347 Fix critical bug: getSessionById missing claude_session_id
Critical bugfix for NOT NULL constraint violation.

Problem:
- Worker service calls getSessionById(sessionDbId) to fetch session data
- Worker then uses dbSession.claude_session_id to create ActiveSession
- But getSessionById was NOT selecting claude_session_id from database
- Result: claudeSessionId = undefined in worker
- Caused: "NOT NULL constraint failed: sdk_sessions.claude_session_id" errors
- Impact: Observations and summaries couldn't be stored

Root cause:
- SessionStore.getSessionById() SQL query missing claude_session_id column
- Line 710-713: "SELECT id, sdk_session_id, project, user_prompt"
- Should be: "SELECT id, claude_session_id, sdk_session_id, project, user_prompt"

Fix:
- Added claude_session_id to SELECT query in getSessionById
- Updated return type to include claude_session_id: string
- Now worker correctly receives claude_session_id from database
- Session ID from hook flows properly through entire system

Files changed:
- src/services/sqlite/SessionStore.ts (getSessionById method)

Testing:
- Build succeeded
- Ready for PM2 restart and live testing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-24 22:18:03 -04:00
Alex Newman
c48290a156 Refactor code structure for improved readability and maintainability 2025-10-24 19:29:16 -04:00
Alex Newman
226a52f8b8 Implement structural updates and optimizations across multiple modules 2025-10-24 19:26:46 -04:00