Files
claude-mem/scripts/fix-corrupted-timestamps.ts
Alex Newman 266c746d50 feat: Fix observation timestamps, refactor session management, and enhance worker reliability (#437)
* Refactor worker version checks and increase timeout settings

- Updated the default hook timeout from 5000ms to 120000ms for improved stability.
- Modified the worker version check to log a warning instead of restarting the worker on version mismatch.
- Removed legacy PM2 cleanup and worker start logic, simplifying the ensureWorkerRunning function.
- Enhanced polling mechanism for worker readiness with increased retries and reduced interval.

* feat: implement worker queue polling to ensure processing completion before proceeding

* refactor: change worker command from start to restart in hooks configuration

* refactor: remove session management complexity

- Simplify createSDKSession to pure INSERT OR IGNORE
- Remove auto-create logic from storeObservation/storeSummary
- Delete 11 unused session management methods
- Derive prompt_number from user_prompts count
- Keep sdk_sessions table schema unchanged for compatibility

* refactor: simplify session management by removing unused methods and auto-creation logic

* Refactor session prompt number retrieval in SessionRoutes

- Updated the method of obtaining the prompt number from the session.
- Replaced `store.getPromptCounter(sessionDbId)` with `store.getPromptNumberFromUserPrompts(claudeSessionId)` for better clarity and accuracy.
- Adjusted the logic for incrementing the prompt number to derive it from the user prompts count instead of directly incrementing a counter.

* refactor: replace getPromptCounter with getPromptNumberFromUserPrompts in SessionManager

Phase 7 of session management simplification. Updates SessionManager to derive
prompt numbers from user_prompts table count instead of using the deprecated
prompt_counter column.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: simplify SessionCompletionHandler to use direct SQL query

Phase 8: Remove call to findActiveSDKSession() and replace with direct
database query in SessionCompletionHandler.completeByClaudeId().

This removes dependency on the deleted findActiveSDKSession() method
and simplifies the code by using a straightforward SELECT query.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: remove markSessionCompleted call from SDKAgent

- Delete call to markSessionCompleted() in SDKAgent.ts
- Session status is no longer tracked or updated
- Part of phase 9: simplifying session management

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: remove markSessionComplete method (Phase 10)

- Deleted markSessionComplete() method from DatabaseManager
- Removed markSessionComplete call from SessionCompletionHandler
- Session completion status no longer tracked in database
- Part of session management simplification effort

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: replace deleted updateSDKSessionId calls in import script (Phase 11)

- Replace updateSDKSessionId() calls with direct SQL UPDATE statements
- Method was deleted in Phase 3 as part of session management simplification
- Import script now uses direct database access consistently

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* test: add validation for SQL updates in sdk_sessions table

* refactor: enhance worker-cli to support manual and automated runs

* Remove cleanup hook and associated session completion logic

- Deleted the cleanup-hook implementation from the hooks directory.
- Removed the session completion endpoint that was used by the cleanup hook.
- Updated the SessionCompletionHandler to eliminate the completeByClaudeId method and its dependencies.
- Adjusted the SessionRoutes to reflect the removal of the session completion route.

* fix: update worker-cli command to use bun for consistency

* feat: Implement timestamp fix for observations and enhance processing logic

- Added `earliestPendingTimestamp` to `ActiveSession` to track the original timestamp of the earliest pending message.
- Updated `SDKAgent` to capture and utilize the earliest pending timestamp during response processing.
- Modified `SessionManager` to track the earliest timestamp when yielding messages.
- Created scripts for fixing corrupted timestamps, validating fixes, and investigating timestamp issues.
- Verified that all corrupted observations have been repaired and logic for future processing is sound.
- Ensured orphan processing can be safely re-enabled after validation.

* feat: Enhance SessionStore to support custom database paths and add timestamp fields for observations and summaries

* Refactor pending queue processing and add management endpoints

- Disabled automatic recovery of orphaned queues on startup; users must now use the new /api/pending-queue/process endpoint.
- Updated processOrphanedQueues method to processPendingQueues with improved session handling and return detailed results.
- Added new API endpoints for managing pending queues: GET /api/pending-queue and POST /api/pending-queue/process.
- Introduced a new script (check-pending-queue.ts) for checking and processing pending observation queues interactively or automatically.
- Enhanced logging and error handling for better monitoring of session processing.

* updated agent sdk

* feat: Add manual recovery guide and queue management endpoints to documentation

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-25 15:36:46 -05:00

244 lines
8.1 KiB
TypeScript
Executable File

#!/usr/bin/env bun
/**
* Fix Corrupted Observation Timestamps
*
* This script repairs observations that were created during the orphan queue processing
* on Dec 24, 2025 between 19:45-20:31. These observations got Dec 24 timestamps instead
* of their original timestamps from Dec 17-20.
*/
import Database from 'bun:sqlite';
import { resolve } from 'path';
const DB_PATH = resolve(process.env.HOME!, '.claude-mem/claude-mem.db');
// Bad window: Dec 24 19:45-20:31 (timestamps in milliseconds, not microseconds)
// Using actual observation epoch format (microseconds since epoch)
const BAD_WINDOW_START = 1766623500000; // Dec 24 19:45 PST
const BAD_WINDOW_END = 1766626260000; // Dec 24 20:31 PST
interface AffectedObservation {
id: number;
sdk_session_id: string;
created_at_epoch: number;
title: string;
}
interface ProcessedMessage {
id: number;
session_db_id: number;
tool_name: string;
created_at_epoch: number;
completed_at_epoch: number;
}
interface SessionMapping {
session_db_id: number;
sdk_session_id: string;
}
interface TimestampFix {
observation_id: number;
observation_title: string;
wrong_timestamp: number;
correct_timestamp: number;
session_db_id: number;
pending_message_id: number;
}
function formatTimestamp(epoch: number): string {
return new Date(epoch).toLocaleString('en-US', {
timeZone: 'America/Los_Angeles',
year: 'numeric',
month: 'short',
day: 'numeric',
hour: '2-digit',
minute: '2-digit',
second: '2-digit'
});
}
function main() {
const args = process.argv.slice(2);
const dryRun = args.includes('--dry-run');
const autoYes = args.includes('--yes') || args.includes('-y');
console.log('🔍 Analyzing corrupted observation timestamps...\n');
if (dryRun) {
console.log('🏃 DRY RUN MODE - No changes will be made\n');
}
const db = new Database(DB_PATH);
try {
// Step 1: Find affected observations
console.log('Step 1: Finding observations created during bad window...');
const affectedObs = db.query<AffectedObservation, []>(`
SELECT id, sdk_session_id, created_at_epoch, title
FROM observations
WHERE created_at_epoch >= ${BAD_WINDOW_START}
AND created_at_epoch <= ${BAD_WINDOW_END}
ORDER BY id
`).all();
console.log(`Found ${affectedObs.length} observations in bad window\n`);
if (affectedObs.length === 0) {
console.log('✅ No affected observations found!');
return;
}
// Step 2: Find processed pending_messages from bad window
console.log('Step 2: Finding pending messages processed during bad window...');
const processedMessages = db.query<ProcessedMessage, []>(`
SELECT id, session_db_id, tool_name, created_at_epoch, completed_at_epoch
FROM pending_messages
WHERE status = 'processed'
AND completed_at_epoch >= ${BAD_WINDOW_START}
AND completed_at_epoch <= ${BAD_WINDOW_END}
ORDER BY completed_at_epoch
`).all();
console.log(`Found ${processedMessages.length} processed messages\n`);
// Step 3: Match observations to their session start times (simpler approach)
console.log('Step 3: Matching observations to session start times...');
const fixes: TimestampFix[] = [];
interface ObsWithSession {
obs_id: number;
obs_title: string;
obs_created: number;
session_started: number;
sdk_session_id: string;
}
const obsWithSessions = db.query<ObsWithSession, []>(`
SELECT
o.id as obs_id,
o.title as obs_title,
o.created_at_epoch as obs_created,
s.started_at_epoch as session_started,
s.sdk_session_id
FROM observations o
JOIN sdk_sessions s ON o.sdk_session_id = s.sdk_session_id
WHERE o.created_at_epoch >= ${BAD_WINDOW_START}
AND o.created_at_epoch <= ${BAD_WINDOW_END}
AND s.started_at_epoch < ${BAD_WINDOW_START}
ORDER BY o.id
`).all();
for (const row of obsWithSessions) {
fixes.push({
observation_id: row.obs_id,
observation_title: row.obs_title || '(no title)',
wrong_timestamp: row.obs_created,
correct_timestamp: row.session_started,
session_db_id: 0, // Not needed for this approach
pending_message_id: 0 // Not needed for this approach
});
}
console.log(`Identified ${fixes.length} observations to fix\n`);
// Step 5: Display what will be fixed
console.log('═══════════════════════════════════════════════════════════════════════');
console.log('PROPOSED FIXES:');
console.log('═══════════════════════════════════════════════════════════════════════\n');
for (const fix of fixes) {
const daysDiff = Math.round((fix.wrong_timestamp - fix.correct_timestamp) / (1000 * 60 * 60 * 24));
console.log(`Observation #${fix.observation_id}: ${fix.observation_title}`);
console.log(` ❌ Wrong: ${formatTimestamp(fix.wrong_timestamp)}`);
console.log(` ✅ Correct: ${formatTimestamp(fix.correct_timestamp)}`);
console.log(` 📅 Off by ${daysDiff} days\n`);
}
// Step 6: Ask for confirmation
console.log('═══════════════════════════════════════════════════════════════════════');
console.log(`Ready to fix ${fixes.length} observations.`);
if (dryRun) {
console.log('\n🏃 DRY RUN COMPLETE - No changes made.');
console.log('Run without --dry-run flag to apply fixes.\n');
db.close();
return;
}
if (autoYes) {
console.log('Auto-confirming with --yes flag...\n');
applyFixes(db, fixes);
return;
}
console.log('Apply these fixes? (y/n): ');
const stdin = Bun.stdin.stream();
const reader = stdin.getReader();
reader.read().then(({ value }) => {
const response = new TextDecoder().decode(value).trim().toLowerCase();
if (response === 'y' || response === 'yes') {
applyFixes(db, fixes);
} else {
console.log('\n❌ Fixes cancelled. No changes made.');
db.close();
}
});
} catch (error) {
console.error('❌ Error:', error);
db.close();
process.exit(1);
}
}
function applyFixes(db: Database, fixes: TimestampFix[]) {
console.log('\n🔧 Applying fixes...\n');
const updateStmt = db.prepare(`
UPDATE observations
SET created_at_epoch = ?,
created_at = datetime(?/1000, 'unixepoch')
WHERE id = ?
`);
let successCount = 0;
let errorCount = 0;
for (const fix of fixes) {
try {
updateStmt.run(
fix.correct_timestamp,
fix.correct_timestamp,
fix.observation_id
);
successCount++;
console.log(`✅ Fixed observation #${fix.observation_id}`);
} catch (error) {
errorCount++;
console.error(`❌ Failed to fix observation #${fix.observation_id}:`, error);
}
}
console.log('\n═══════════════════════════════════════════════════════════════════════');
console.log('RESULTS:');
console.log('═══════════════════════════════════════════════════════════════════════');
console.log(`✅ Successfully fixed: ${successCount}`);
console.log(`❌ Failed: ${errorCount}`);
console.log(`📊 Total processed: ${fixes.length}\n`);
if (successCount > 0) {
console.log('🎉 Timestamp corruption has been repaired!');
console.log('💡 Next steps:');
console.log(' 1. Verify the fixes with: bun scripts/verify-timestamp-fix.ts');
console.log(' 2. Consider re-enabling orphan processing if timestamp fix is working\n');
}
db.close();
}
main();