Files
claude-mem/scripts/transcript-to-markdown.ts
Alex Newman 68290a9121 Performance improvements: Token reduction and enhanced summaries (#101)
* refactor: Reduce continuation prompt token usage by 95 lines

Removed redundant instructions from continuation prompt that were originally
added to mitigate a session continuity issue. That issue has since been
resolved, making these detailed instructions unnecessary on every continuation.

Changes:
- Reduced continuation prompt from ~106 lines to ~11 lines (~95 line reduction)
- Changed "User's Goal:" to "Next Prompt in Session:" (more accurate framing)
- Removed redundant WHAT TO RECORD, WHEN TO SKIP, and OUTPUT FORMAT sections
- Kept concise reminder: "Continue generating observations and progress summaries..."
- Initial prompt still contains all detailed instructions

Impact:
- Significant token savings on every continuation prompt
- Faster context injection with no loss of functionality
- Instructions remain comprehensive in initial prompt

Files modified:
- src/sdk/prompts.ts (buildContinuationPrompt function)
- plugin/scripts/worker-service.cjs (compiled output)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Enhance observation and summary prompts for clarity and token efficiency

* Enhance prompt clarity and instructions in prompts.ts

- Added a reminder to think about instructions before starting work.
- Simplified the continuation prompt instruction by removing "for this ongoing session."

* feat: Enhance settings.json with permissions and deny access to sensitive files

refactor: Remove PLAN-full-observation-display.md and PR_SUMMARY.md as they are no longer needed

chore: Delete SECURITY_SUMMARY.md since it is redundant after recent changes

fix: Update worker-service.cjs to streamline observation generation instructions

cleanup: Remove src-analysis.md and src-tree.md for a cleaner codebase

refactor: Modify prompts.ts to clarify instructions for memory processing

* refactor: Remove legacy worker service implementation

* feat: Enhance summary hook to extract last assistant message and improve logging

- Added function to extract the last assistant message from the transcript.
- Updated summary hook to include last assistant message in the summary request.
- Modified SDKSession interface to store last assistant message.
- Adjusted buildSummaryPrompt to utilize last assistant message for generating summaries.
- Updated worker service and session manager to handle last assistant message in summarize requests.
- Introduced silentDebug utility for improved logging and diagnostics throughout the summary process.

* docs: Add comprehensive implementation plan for ROI metrics feature

Added detailed implementation plan covering:
- Token usage capture from Agent SDK
- Database schema changes (migration #8)
- Discovery cost tracking per observation
- Context hook display with ROI metrics
- Testing and rollout strategy

Timeline: ~20 hours over 4 days
Goal: Empirical data for YC application amendment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add transcript processing scripts for analysis and formatting

- Implemented `dump-transcript-readable.ts` to generate a readable markdown dump of transcripts, excluding certain entry types.
- Created `extract-rich-context-examples.ts` to extract and showcase rich context examples from transcripts, highlighting user requests and assistant reasoning.
- Developed `format-transcript-context.ts` to format transcript context into a structured markdown format for improved observation generation.
- Added `test-transcript-parser.ts` for validating data extraction from transcript JSONL files, including statistics and error reporting.
- Introduced `transcript-to-markdown.ts` for a complete representation of transcript data in markdown format, showing all context data.
- Enhanced type definitions in `transcript.ts` to support new features and ensure type safety.
- Built `transcript-parser.ts` to handle parsing of transcript JSONL files, including error handling and data extraction methods.

* Refactor hooks and SDKAgent for improved observation handling

- Updated `new-hook.ts` to clean user prompts by stripping leading slashes for better semantic clarity.
- Enhanced `save-hook.ts` to include additional tools in the SKIP_TOOLS set, preventing unnecessary observations from certain command invocations.
- Modified `prompts.ts` to change the structure of observation prompts, emphasizing the observational role and providing a detailed XML output format for observations.
- Adjusted `SDKAgent.ts` to enforce stricter tool usage restrictions, ensuring the memory agent operates solely as an observer without any tool access.

* feat: Enhance session initialization to accept user prompts and prompt numbers

- Updated `handleSessionInit` in `worker-service.ts` to extract `userPrompt` and `promptNumber` from the request body and pass them to `initializeSession`.
- Modified `initializeSession` in `SessionManager.ts` to handle optional `currentUserPrompt` and `promptNumber` parameters.
- Added logic to update the existing session's `userPrompt` and `lastPromptNumber` if a `currentUserPrompt` is provided.
- Implemented debug logging for session initialization and updates to track user prompts and prompt numbers.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-13 18:22:44 -05:00

210 lines
6.9 KiB
TypeScript

#!/usr/bin/env tsx
/**
* Transcript to Markdown - Complete 1:1 representation
* Shows ALL available context data from a Claude Code transcript
*/
import { TranscriptParser } from '../src/utils/transcript-parser.js';
import type { UserTranscriptEntry, AssistantTranscriptEntry, ToolResultContent } from '../types/transcript.js';
import { writeFileSync } from 'fs';
import { basename } from 'path';
const transcriptPath = process.argv[2];
const maxTurns = process.argv[3] ? parseInt(process.argv[3]) : 20;
if (!transcriptPath) {
console.error('Usage: tsx scripts/transcript-to-markdown.ts <path-to-transcript.jsonl> [max-turns]');
process.exit(1);
}
/**
* Truncate string to max length, adding ellipsis if needed
*/
function truncate(str: string, maxLen: number = 500): string {
if (str.length <= maxLen) return str;
return str.substring(0, maxLen) + '\n... [truncated]';
}
/**
* Format tool result content for display
*/
function formatToolResult(result: ToolResultContent): string {
if (typeof result.content === 'string') {
// Try to parse as JSON for better formatting
try {
const parsed = JSON.parse(result.content);
return JSON.stringify(parsed, null, 2);
} catch {
return truncate(result.content);
}
}
if (Array.isArray(result.content)) {
// Handle array of content items - extract text and parse if JSON
const formatted = result.content.map((item: any) => {
if (item.type === 'text' && item.text) {
try {
const parsed = JSON.parse(item.text);
return JSON.stringify(parsed, null, 2);
} catch {
return item.text;
}
}
return JSON.stringify(item, null, 2);
}).join('\n\n');
return formatted;
}
return '[unknown result type]';
}
const parser = new TranscriptParser(transcriptPath);
const entries = parser.getAllEntries();
const stats = parser.getParseStats();
let output = `# Transcript: ${basename(transcriptPath)}\n\n`;
output += `**Generated:** ${new Date().toLocaleString()}\n`;
output += `**Total Entries:** ${stats.parsedEntries}\n`;
output += `**Entry Types:** ${JSON.stringify(stats.entriesByType, null, 2)}\n`;
output += `**Showing:** First ${maxTurns} conversation turns\n\n`;
output += `---\n\n`;
let turnNumber = 0;
let inTurn = false;
for (const entry of entries) {
// Skip summary and file-history-snapshot entries
if (entry.type === 'summary' || entry.type === 'file-history-snapshot') continue;
// USER MESSAGE
if (entry.type === 'user') {
const userEntry = entry as UserTranscriptEntry;
turnNumber++;
if (turnNumber > maxTurns) break;
inTurn = true;
output += `## Turn ${turnNumber}\n\n`;
output += `### 👤 User\n`;
output += `**Timestamp:** ${userEntry.timestamp}\n`;
output += `**UUID:** ${userEntry.uuid}\n`;
output += `**Session ID:** ${userEntry.sessionId}\n`;
output += `**CWD:** ${userEntry.cwd}\n\n`;
// Extract user message text
if (typeof userEntry.message.content === 'string') {
output += userEntry.message.content + '\n\n';
} else if (Array.isArray(userEntry.message.content)) {
const textBlocks = userEntry.message.content.filter((c) => c.type === 'text');
if (textBlocks.length > 0) {
const text = textBlocks.map((b: any) => b.text).join('\n');
output += text + '\n\n';
}
// Show ACTUAL tool results with their data
const toolResults = userEntry.message.content.filter((c): c is ToolResultContent => c.type === 'tool_result');
if (toolResults.length > 0) {
output += `**Tool Results Submitted (${toolResults.length}):**\n\n`;
for (const result of toolResults) {
output += `- **Tool Use ID:** \`${result.tool_use_id}\`\n`;
if (result.is_error) {
output += ` **ERROR:**\n`;
}
output += ` \`\`\`json\n`;
output += ` ${formatToolResult(result)}\n`;
output += ` \`\`\`\n\n`;
}
}
}
}
// ASSISTANT MESSAGE
if (entry.type === 'assistant' && inTurn) {
const assistantEntry = entry as AssistantTranscriptEntry;
output += `### 🤖 Assistant\n`;
output += `**Timestamp:** ${assistantEntry.timestamp}\n`;
output += `**UUID:** ${assistantEntry.uuid}\n`;
output += `**Model:** ${assistantEntry.message.model}\n`;
output += `**Stop Reason:** ${assistantEntry.message.stop_reason || 'N/A'}\n\n`;
if (!Array.isArray(assistantEntry.message.content)) {
output += `*[No content]*\n\n`;
continue;
}
const content = assistantEntry.message.content;
// 1. Thinking blocks (show first, as they happen first in reasoning)
const thinkingBlocks = content.filter((c) => c.type === 'thinking');
if (thinkingBlocks.length > 0) {
output += `**💭 Thinking:**\n\n`;
for (const block of thinkingBlocks) {
const thinking = (block as any).thinking;
// Format thinking with proper line breaks and indentation
const formattedThinking = thinking
.split('\n')
.map((line: string) => line.trimEnd())
.join('\n');
output += '> ';
output += formattedThinking.replace(/\n/g, '\n> ');
output += '\n\n';
}
}
// 2. Text responses
const textBlocks = content.filter((c) => c.type === 'text');
if (textBlocks.length > 0) {
output += `**Response:**\n\n`;
for (const block of textBlocks) {
output += (block as any).text + '\n\n';
}
}
// 3. Tool uses - show complete input
const toolUseBlocks = content.filter((c) => c.type === 'tool_use');
if (toolUseBlocks.length > 0) {
output += `**🔧 Tools Used (${toolUseBlocks.length}):**\n\n`;
for (const tool of toolUseBlocks) {
const t = tool as any;
output += `- **${t.name}** (ID: \`${t.id}\`)\n`;
output += ` \`\`\`json\n`;
output += ` ${JSON.stringify(t.input, null, 2)}\n`;
output += ` \`\`\`\n\n`;
}
}
// 4. Token usage
if (assistantEntry.message.usage) {
const usage = assistantEntry.message.usage;
output += `**📊 Token Usage:**\n`;
output += `- Input: ${usage.input_tokens || 0}\n`;
output += `- Output: ${usage.output_tokens || 0}\n`;
if (usage.cache_creation_input_tokens) {
output += `- Cache creation: ${usage.cache_creation_input_tokens}\n`;
}
if (usage.cache_read_input_tokens) {
output += `- Cache read: ${usage.cache_read_input_tokens}\n`;
}
output += '\n';
}
output += `---\n\n`;
inTurn = false;
}
}
if (turnNumber < (stats.entriesByType['user'] || 0)) {
output += `\n*... ${(stats.entriesByType['user'] || 0) - turnNumber} more turns not shown*\n`;
}
// Write output
const outputPath = transcriptPath.replace('.jsonl', '-complete.md');
writeFileSync(outputPath, output, 'utf-8');
console.log(`\nComplete transcript written to: ${outputPath}`);
console.log(`Turns shown: ${Math.min(turnNumber, maxTurns)} of ${stats.entriesByType['user'] || 0}\n`);