Files
paperclip/packages/adapters/gemini-local/src/server/execute.ts
Dotta 236d11d36f [codex] Add run liveness continuations (#4083)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies.
> - Heartbeat runs are the control-plane record of each agent execution
window.
> - Long-running local agents can exhaust context or stop while still
holding useful next-step state.
> - Operators need that stop reason, next action, and continuation path
to be durable and visible.
> - This pull request adds run liveness metadata, continuation
summaries, and UI surfaces for issue run ledgers.
> - The benefit is that interrupted or long-running work can resume with
clearer context instead of losing the agent's last useful handoff.

## What Changed

- Added heartbeat-run liveness fields, continuation attempt tracking,
and an idempotent `0058` migration.
- Added server services and tests for run liveness, continuation
summaries, stop metadata, and activity backfill.
- Wired local and HTTP adapters to surface continuation/liveness context
through shared adapter utilities.
- Added shared constants, validators, and heartbeat types for liveness
continuation state.
- Added issue-detail UI surfaces for continuation handoffs and the run
ledger, with component tests.
- Updated agent runtime docs, heartbeat protocol docs, prompt guidance,
onboarding assets, and skills instructions to explain continuation
behavior.
- Addressed Greptile feedback by scoping document evidence by run,
excluding system continuation-summary documents from liveness evidence,
importing shared liveness types, surfacing hidden ledger run counts,
documenting bounded retry behavior, and moving run-ledger liveness
backfill off the request path.

## Verification

- `pnpm exec vitest run packages/adapter-utils/src/server-utils.test.ts
server/src/__tests__/run-continuations.test.ts
server/src/__tests__/run-liveness.test.ts
server/src/__tests__/activity-service.test.ts
server/src/__tests__/documents-service.test.ts
server/src/__tests__/issue-continuation-summary.test.ts
server/src/services/heartbeat-stop-metadata.test.ts
ui/src/components/IssueRunLedger.test.tsx
ui/src/components/IssueContinuationHandoff.test.tsx
ui/src/components/IssueDocumentsSection.test.tsx`
- `pnpm --filter @paperclipai/db build`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
ui/src/components/IssueRunLedger.test.tsx`
- `pnpm --filter @paperclipai/ui typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
server/src/__tests__/run-continuations.test.ts
ui/src/components/IssueRunLedger.test.tsx`
- `pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts -t "treats a
plan document update"`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts -t "activity
service|treats a plan document update"`
- Remote PR checks on head `e53b1a1d`: `verify`, `e2e`, `policy`, and
Snyk all passed.
- Confirmed `public-gh/master` is an ancestor of this branch after
fetching `public-gh master`.
- Confirmed `pnpm-lock.yaml` is not included in the branch diff.
- Confirmed migration `0058_wealthy_starbolt.sql` is ordered after
`0057` and uses `IF NOT EXISTS` guards for repeat application.
- Greptile inline review threads are resolved.

## Risks

- Medium risk: this touches heartbeat execution, liveness recovery,
activity rendering, issue routes, shared contracts, docs, and UI.
- Migration risk is mitigated by additive columns/indexes and idempotent
guards.
- Run-ledger liveness backfill is now asynchronous, so the first ledger
response can briefly show historical missing liveness until the
background backfill completes.
- UI screenshot coverage is not included in this packaging pass;
validation is currently through focused component tests.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5.4, local tool-use coding agent with terminal, git,
GitHub connector, GitHub CLI, and Paperclip API access.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

Screenshot note: no before/after screenshots were captured in this PR
packaging pass; the UI changes are covered by focused component tests
listed above.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-04-20 06:01:49 -05:00

478 lines
18 KiB
TypeScript

import fs from "node:fs/promises";
import type { Dirent } from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import type { AdapterExecutionContext, AdapterExecutionResult } from "@paperclipai/adapter-utils";
import {
asBoolean,
asNumber,
asString,
asStringArray,
buildPaperclipEnv,
buildInvocationEnvForLogs,
ensureAbsoluteDirectory,
ensureCommandResolvable,
ensurePaperclipSkillSymlink,
joinPromptSections,
ensurePathInEnv,
readPaperclipRuntimeSkillEntries,
resolveCommandForLogs,
resolvePaperclipDesiredSkillNames,
removeMaintainerOnlySkillSymlinks,
parseObject,
renderTemplate,
renderPaperclipWakePrompt,
stringifyPaperclipWakePayload,
DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE,
runChildProcess,
} from "@paperclipai/adapter-utils/server-utils";
import { DEFAULT_GEMINI_LOCAL_MODEL } from "../index.js";
import {
describeGeminiFailure,
detectGeminiAuthRequired,
isGeminiTurnLimitResult,
isGeminiUnknownSessionError,
parseGeminiJsonl,
} from "./parse.js";
import { firstNonEmptyLine } from "./utils.js";
const __moduleDir = path.dirname(fileURLToPath(import.meta.url));
function hasNonEmptyEnvValue(env: Record<string, string>, key: string): boolean {
const raw = env[key];
return typeof raw === "string" && raw.trim().length > 0;
}
function resolveGeminiBillingType(env: Record<string, string>): "api" | "subscription" {
return hasNonEmptyEnvValue(env, "GEMINI_API_KEY") || hasNonEmptyEnvValue(env, "GOOGLE_API_KEY")
? "api"
: "subscription";
}
function renderPaperclipEnvNote(env: Record<string, string>): string {
const paperclipKeys = Object.keys(env)
.filter((key) => key.startsWith("PAPERCLIP_"))
.sort();
if (paperclipKeys.length === 0) return "";
return [
"Paperclip runtime note:",
`The following PAPERCLIP_* environment variables are available in this run: ${paperclipKeys.join(", ")}`,
"Do not assume these variables are missing without checking your shell environment.",
"",
"",
].join("\n");
}
function renderApiAccessNote(env: Record<string, string>): string {
if (!hasNonEmptyEnvValue(env, "PAPERCLIP_API_URL") || !hasNonEmptyEnvValue(env, "PAPERCLIP_API_KEY")) return "";
return [
"Paperclip API access note:",
"Use run_shell_command with curl to make Paperclip API requests.",
"GET example:",
` run_shell_command({ command: "curl -s -H \\"Authorization: Bearer $PAPERCLIP_API_KEY\\" \\"$PAPERCLIP_API_URL/api/agents/me\\"" })`,
"POST/PATCH example:",
` run_shell_command({ command: "curl -s -X POST -H \\"Authorization: Bearer $PAPERCLIP_API_KEY\\" -H 'Content-Type: application/json' -H \\"X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID\\" -d '{...}' \\"$PAPERCLIP_API_URL/api/issues/{id}/checkout\\"" })`,
"",
"",
].join("\n");
}
function geminiSkillsHome(): string {
return path.join(os.homedir(), ".gemini", "skills");
}
/**
* Inject Paperclip skills directly into `~/.gemini/skills/` via symlinks.
* This avoids needing GEMINI_CLI_HOME overrides, so the CLI naturally finds
* both its auth credentials and the injected skills in the real home directory.
*/
async function ensureGeminiSkillsInjected(
onLog: AdapterExecutionContext["onLog"],
skillsEntries: Array<{ key: string; runtimeName: string; source: string }>,
desiredSkillNames?: string[],
): Promise<void> {
const desiredSet = new Set(desiredSkillNames ?? skillsEntries.map((entry) => entry.key));
const selectedEntries = skillsEntries.filter((entry) => desiredSet.has(entry.key));
if (selectedEntries.length === 0) return;
const skillsHome = geminiSkillsHome();
try {
await fs.mkdir(skillsHome, { recursive: true });
} catch (err) {
await onLog(
"stderr",
`[paperclip] Failed to prepare Gemini skills directory ${skillsHome}: ${err instanceof Error ? err.message : String(err)}\n`,
);
return;
}
const removedSkills = await removeMaintainerOnlySkillSymlinks(
skillsHome,
selectedEntries.map((entry) => entry.runtimeName),
);
for (const skillName of removedSkills) {
await onLog(
"stderr",
`[paperclip] Removed maintainer-only Gemini skill "${skillName}" from ${skillsHome}\n`,
);
}
for (const entry of selectedEntries) {
const target = path.join(skillsHome, entry.runtimeName);
try {
const result = await ensurePaperclipSkillSymlink(entry.source, target);
if (result === "skipped") continue;
await onLog(
"stderr",
`[paperclip] ${result === "repaired" ? "Repaired" : "Linked"} Gemini skill: ${entry.key}\n`,
);
} catch (err) {
await onLog(
"stderr",
`[paperclip] Failed to link Gemini skill "${entry.key}": ${err instanceof Error ? err.message : String(err)}\n`,
);
}
}
}
export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExecutionResult> {
const { runId, agent, runtime, config, context, onLog, onMeta, onSpawn, authToken } = ctx;
const promptTemplate = asString(
config.promptTemplate,
DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE,
);
const command = asString(config.command, "gemini");
const model = asString(config.model, DEFAULT_GEMINI_LOCAL_MODEL).trim();
const sandbox = asBoolean(config.sandbox, false);
const workspaceContext = parseObject(context.paperclipWorkspace);
const workspaceCwd = asString(workspaceContext.cwd, "");
const workspaceSource = asString(workspaceContext.source, "");
const workspaceId = asString(workspaceContext.workspaceId, "");
const workspaceRepoUrl = asString(workspaceContext.repoUrl, "");
const workspaceRepoRef = asString(workspaceContext.repoRef, "");
const agentHome = asString(workspaceContext.agentHome, "");
const workspaceHints = Array.isArray(context.paperclipWorkspaces)
? context.paperclipWorkspaces.filter(
(value): value is Record<string, unknown> => typeof value === "object" && value !== null,
)
: [];
const configuredCwd = asString(config.cwd, "");
const useConfiguredInsteadOfAgentHome = workspaceSource === "agent_home" && configuredCwd.length > 0;
const effectiveWorkspaceCwd = useConfiguredInsteadOfAgentHome ? "" : workspaceCwd;
const cwd = effectiveWorkspaceCwd || configuredCwd || process.cwd();
await ensureAbsoluteDirectory(cwd, { createIfMissing: true });
const geminiSkillEntries = await readPaperclipRuntimeSkillEntries(config, __moduleDir);
const desiredGeminiSkillNames = resolvePaperclipDesiredSkillNames(config, geminiSkillEntries);
await ensureGeminiSkillsInjected(onLog, geminiSkillEntries, desiredGeminiSkillNames);
const envConfig = parseObject(config.env);
const hasExplicitApiKey =
typeof envConfig.PAPERCLIP_API_KEY === "string" && envConfig.PAPERCLIP_API_KEY.trim().length > 0;
const env: Record<string, string> = { ...buildPaperclipEnv(agent) };
env.PAPERCLIP_RUN_ID = runId;
const wakeTaskId =
(typeof context.taskId === "string" && context.taskId.trim().length > 0 && context.taskId.trim()) ||
(typeof context.issueId === "string" && context.issueId.trim().length > 0 && context.issueId.trim()) ||
null;
const wakeReason =
typeof context.wakeReason === "string" && context.wakeReason.trim().length > 0
? context.wakeReason.trim()
: null;
const wakeCommentId =
(typeof context.wakeCommentId === "string" && context.wakeCommentId.trim().length > 0 && context.wakeCommentId.trim()) ||
(typeof context.commentId === "string" && context.commentId.trim().length > 0 && context.commentId.trim()) ||
null;
const approvalId =
typeof context.approvalId === "string" && context.approvalId.trim().length > 0
? context.approvalId.trim()
: null;
const approvalStatus =
typeof context.approvalStatus === "string" && context.approvalStatus.trim().length > 0
? context.approvalStatus.trim()
: null;
const linkedIssueIds = Array.isArray(context.issueIds)
? context.issueIds.filter((value): value is string => typeof value === "string" && value.trim().length > 0)
: [];
const wakePayloadJson = stringifyPaperclipWakePayload(context.paperclipWake);
if (wakeTaskId) env.PAPERCLIP_TASK_ID = wakeTaskId;
if (wakeReason) env.PAPERCLIP_WAKE_REASON = wakeReason;
if (wakeCommentId) env.PAPERCLIP_WAKE_COMMENT_ID = wakeCommentId;
if (approvalId) env.PAPERCLIP_APPROVAL_ID = approvalId;
if (approvalStatus) env.PAPERCLIP_APPROVAL_STATUS = approvalStatus;
if (linkedIssueIds.length > 0) env.PAPERCLIP_LINKED_ISSUE_IDS = linkedIssueIds.join(",");
if (wakePayloadJson) env.PAPERCLIP_WAKE_PAYLOAD_JSON = wakePayloadJson;
if (effectiveWorkspaceCwd) env.PAPERCLIP_WORKSPACE_CWD = effectiveWorkspaceCwd;
if (workspaceSource) env.PAPERCLIP_WORKSPACE_SOURCE = workspaceSource;
if (workspaceId) env.PAPERCLIP_WORKSPACE_ID = workspaceId;
if (workspaceRepoUrl) env.PAPERCLIP_WORKSPACE_REPO_URL = workspaceRepoUrl;
if (workspaceRepoRef) env.PAPERCLIP_WORKSPACE_REPO_REF = workspaceRepoRef;
if (agentHome) env.AGENT_HOME = agentHome;
if (workspaceHints.length > 0) env.PAPERCLIP_WORKSPACES_JSON = JSON.stringify(workspaceHints);
for (const [key, value] of Object.entries(envConfig)) {
if (typeof value === "string") env[key] = value;
}
if (!hasExplicitApiKey && authToken) {
env.PAPERCLIP_API_KEY = authToken;
}
const effectiveEnv = Object.fromEntries(
Object.entries({ ...process.env, ...env }).filter(
(entry): entry is [string, string] => typeof entry[1] === "string",
),
);
const billingType = resolveGeminiBillingType(effectiveEnv);
const runtimeEnv = ensurePathInEnv(effectiveEnv);
await ensureCommandResolvable(command, cwd, runtimeEnv);
const resolvedCommand = await resolveCommandForLogs(command, cwd, runtimeEnv);
const loggedEnv = buildInvocationEnvForLogs(env, {
runtimeEnv,
includeRuntimeKeys: ["HOME"],
resolvedCommand,
});
const timeoutSec = asNumber(config.timeoutSec, 0);
const graceSec = asNumber(config.graceSec, 20);
const extraArgs = (() => {
const fromExtraArgs = asStringArray(config.extraArgs);
if (fromExtraArgs.length > 0) return fromExtraArgs;
return asStringArray(config.args);
})();
const runtimeSessionParams = parseObject(runtime.sessionParams);
const runtimeSessionId = asString(runtimeSessionParams.sessionId, runtime.sessionId ?? "");
const runtimeSessionCwd = asString(runtimeSessionParams.cwd, "");
const canResumeSession =
runtimeSessionId.length > 0 &&
(runtimeSessionCwd.length === 0 || path.resolve(runtimeSessionCwd) === path.resolve(cwd));
const sessionId = canResumeSession ? runtimeSessionId : null;
if (runtimeSessionId && !canResumeSession) {
await onLog(
"stdout",
`[paperclip] Gemini session "${runtimeSessionId}" was saved for cwd "${runtimeSessionCwd}" and will not be resumed in "${cwd}".\n`,
);
}
const instructionsFilePath = asString(config.instructionsFilePath, "").trim();
const instructionsDir = instructionsFilePath ? `${path.dirname(instructionsFilePath)}/` : "";
let instructionsPrefix = "";
if (instructionsFilePath) {
try {
const instructionsContents = await fs.readFile(instructionsFilePath, "utf8");
instructionsPrefix =
`${instructionsContents}\n\n` +
`The above agent instructions were loaded from ${instructionsFilePath}. ` +
`Resolve any relative file references from ${instructionsDir}.\n\n`;
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
await onLog(
"stdout",
`[paperclip] Warning: could not read agent instructions file "${instructionsFilePath}": ${reason}\n`,
);
}
}
const commandNotes = (() => {
const notes: string[] = ["Prompt is passed to Gemini via --prompt for non-interactive execution."];
notes.push("Added --approval-mode yolo for unattended execution.");
if (!instructionsFilePath) return notes;
if (instructionsPrefix.length > 0) {
notes.push(
`Loaded agent instructions from ${instructionsFilePath}`,
`Prepended instructions + path directive to prompt (relative references from ${instructionsDir}).`,
);
return notes;
}
notes.push(
`Configured instructionsFilePath ${instructionsFilePath}, but file could not be read; continuing without injected instructions.`,
);
return notes;
})();
const bootstrapPromptTemplate = asString(config.bootstrapPromptTemplate, "");
const templateData = {
agentId: agent.id,
companyId: agent.companyId,
runId,
company: { id: agent.companyId },
agent,
run: { id: runId, source: "on_demand" },
context,
};
const renderedBootstrapPrompt =
!sessionId && bootstrapPromptTemplate.trim().length > 0
? renderTemplate(bootstrapPromptTemplate, templateData).trim()
: "";
const wakePrompt = renderPaperclipWakePrompt(context.paperclipWake, { resumedSession: Boolean(sessionId) });
const shouldUseResumeDeltaPrompt = Boolean(sessionId) && wakePrompt.length > 0;
const renderedPrompt = shouldUseResumeDeltaPrompt ? "" : renderTemplate(promptTemplate, templateData);
const sessionHandoffNote = asString(context.paperclipSessionHandoffMarkdown, "").trim();
const paperclipEnvNote = renderPaperclipEnvNote(env);
const apiAccessNote = renderApiAccessNote(env);
const prompt = joinPromptSections([
instructionsPrefix,
renderedBootstrapPrompt,
wakePrompt,
sessionHandoffNote,
paperclipEnvNote,
apiAccessNote,
renderedPrompt,
]);
const promptMetrics = {
promptChars: prompt.length,
instructionsChars: instructionsPrefix.length,
bootstrapPromptChars: renderedBootstrapPrompt.length,
wakePromptChars: wakePrompt.length,
sessionHandoffChars: sessionHandoffNote.length,
runtimeNoteChars: paperclipEnvNote.length + apiAccessNote.length,
heartbeatPromptChars: renderedPrompt.length,
};
const buildArgs = (resumeSessionId: string | null) => {
const args = ["--output-format", "stream-json"];
if (resumeSessionId) args.push("--resume", resumeSessionId);
if (model && model !== DEFAULT_GEMINI_LOCAL_MODEL) args.push("--model", model);
args.push("--approval-mode", "yolo");
if (sandbox) {
args.push("--sandbox");
} else {
args.push("--sandbox=none");
}
if (extraArgs.length > 0) args.push(...extraArgs);
args.push("--prompt", prompt);
return args;
};
const runAttempt = async (resumeSessionId: string | null) => {
const args = buildArgs(resumeSessionId);
if (onMeta) {
await onMeta({
adapterType: "gemini_local",
command: resolvedCommand,
cwd,
commandNotes,
commandArgs: args.map((value, index) => (
index === args.length - 1 ? `<prompt ${prompt.length} chars>` : value
)),
env: loggedEnv,
prompt,
promptMetrics,
context,
});
}
const proc = await runChildProcess(runId, command, args, {
cwd,
env,
timeoutSec,
graceSec,
onSpawn,
onLog,
});
return {
proc,
parsed: parseGeminiJsonl(proc.stdout),
};
};
const toResult = (
attempt: {
proc: {
exitCode: number | null;
signal: string | null;
timedOut: boolean;
stdout: string;
stderr: string;
};
parsed: ReturnType<typeof parseGeminiJsonl>;
},
clearSessionOnMissingSession = false,
isRetry = false,
): AdapterExecutionResult => {
const authMeta = detectGeminiAuthRequired({
parsed: attempt.parsed.resultEvent,
stdout: attempt.proc.stdout,
stderr: attempt.proc.stderr,
});
if (attempt.proc.timedOut) {
return {
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: true,
errorMessage: `Timed out after ${timeoutSec}s`,
errorCode: authMeta.requiresAuth ? "gemini_auth_required" : null,
clearSession: clearSessionOnMissingSession,
};
}
const clearSessionForTurnLimit = isGeminiTurnLimitResult(attempt.parsed.resultEvent, attempt.proc.exitCode);
// On retry, don't fall back to old session ID — the old session was stale
const canFallbackToRuntimeSession = !isRetry;
const resolvedSessionId = attempt.parsed.sessionId
?? (canFallbackToRuntimeSession ? (runtimeSessionId ?? runtime.sessionId ?? null) : null);
const resolvedSessionParams = resolvedSessionId
? ({
sessionId: resolvedSessionId,
cwd,
...(workspaceId ? { workspaceId } : {}),
...(workspaceRepoUrl ? { repoUrl: workspaceRepoUrl } : {}),
...(workspaceRepoRef ? { repoRef: workspaceRepoRef } : {}),
} as Record<string, unknown>)
: null;
const parsedError = typeof attempt.parsed.errorMessage === "string" ? attempt.parsed.errorMessage.trim() : "";
const stderrLine = firstNonEmptyLine(attempt.proc.stderr);
const structuredFailure = attempt.parsed.resultEvent
? describeGeminiFailure(attempt.parsed.resultEvent)
: null;
const fallbackErrorMessage =
parsedError ||
structuredFailure ||
stderrLine ||
`Gemini exited with code ${attempt.proc.exitCode ?? -1}`;
return {
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: false,
errorMessage: (attempt.proc.exitCode ?? 0) === 0 ? null : fallbackErrorMessage,
errorCode: (attempt.proc.exitCode ?? 0) !== 0 && authMeta.requiresAuth ? "gemini_auth_required" : null,
usage: attempt.parsed.usage,
sessionId: resolvedSessionId,
sessionParams: resolvedSessionParams,
sessionDisplayId: resolvedSessionId,
provider: "google",
biller: "google",
model,
billingType,
costUsd: attempt.parsed.costUsd,
resultJson: attempt.parsed.resultEvent ?? {
stdout: attempt.proc.stdout,
stderr: attempt.proc.stderr,
},
summary: attempt.parsed.summary,
question: attempt.parsed.question,
clearSession: clearSessionForTurnLimit || Boolean(clearSessionOnMissingSession && !resolvedSessionId),
};
};
const initial = await runAttempt(sessionId);
if (
sessionId &&
!initial.proc.timedOut &&
(initial.proc.exitCode ?? 0) !== 0 &&
isGeminiUnknownSessionError(initial.proc.stdout, initial.proc.stderr)
) {
await onLog(
"stdout",
`[paperclip] Gemini resume session "${sessionId}" is unavailable; retrying with a fresh session.\n`,
);
const retry = await runAttempt(null);
return toResult(retry, true, true);
}
return toResult(initial);
}