mirror of
https://github.com/paperclipai/paperclip
synced 2026-04-25 17:25:15 +02:00
[codex] Fix stale issue execution run locks (#4258)
## Thinking Path > - Paperclip is a control plane for AI-agent companies, so issue checkout and execution ownership are core safety contracts. > - The affected subsystem is the issue service and route layer that gates agent writes by `checkoutRunId` and `executionRunId`. > - PAP-1982 exposed a stale-lock failure mode where a terminal heartbeat run could leave `executionRunId` pinned after checkout ownership had moved or been cleared. > - That stale execution lock could reject legitimate PATCH/comment/release requests from the rightful assignee after a harness restart. > - This pull request centralizes terminal-run cleanup, applies it before ownership-gated writes, and adds a board-only recovery endpoint for operator intervention. > - The benefit is that crashed or terminal runs no longer strand issues behind stale execution locks, while live execution locks still block conflicting writes. ## What Changed - Added `issueService.clearExecutionRunIfTerminal()` to atomically lock the issue/run rows and clear terminal or missing execution-run locks. - Reused stale execution-lock cleanup from checkout, `assertCheckoutOwner()`, and `release()`. - Allowed the same assigned agent/current run to adopt an unowned `in_progress` checkout after stale execution-lock cleanup. - Updated release to clear `executionRunId`, `executionAgentNameKey`, and `executionLockedAt`. - Added board-only `POST /api/issues/:id/admin/force-release` with company access checks, optional `clearAssignee=true`, and `issue.admin_force_release` audit logging. - Added embedded Postgres service tests and route integration tests for stale-lock recovery, release behavior, and admin force-release authorization/audit behavior. - Documented the new force-release API in `doc/SPEC-implementation.md`. ## Verification - `pnpm vitest run server/src/__tests__/issues-service.test.ts server/src/__tests__/issue-stale-execution-lock-routes.test.ts` passed. - `pnpm vitest run server/src/__tests__/issue-stale-execution-lock-routes.test.ts server/src/__tests__/approval-routes-idempotency.test.ts server/src/__tests__/issue-comment-reopen-routes.test.ts server/src/__tests__/issue-telemetry-routes.test.ts` passed. - `pnpm -r typecheck` passed. - `pnpm build` passed. - `git diff --check` passed. - `pnpm lint` could not run because this repo has no `lint` command. - Full `pnpm test:run` completed with 4 failures in existing route suites: `approval-routes-idempotency.test.ts` (2), `issue-comment-reopen-routes.test.ts` (1), and `issue-telemetry-routes.test.ts` (1). Those same files pass when run isolated and when run together with the new stale-lock route test, so this appears to be a whole-suite ordering/mock-isolation issue outside this patch path. ## Risks - Medium: this changes ownership-gated write behavior. The new adoption path is limited to the current run, the current assignee, `in_progress` issues, and rows with no checkout owner after terminal-lock cleanup. - Low: the admin force-release endpoint is board-only and company-scoped, but misuse can intentionally clear a live lock. It writes an audit event with prior lock IDs. - No schema or migration changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5 coding agent (`gpt-5`), agentic coding with terminal/tool use and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
@@ -484,6 +484,7 @@ All endpoints are under `/api` and return JSON.
|
|||||||
- `DELETE /issues/:issueId/documents/:key`
|
- `DELETE /issues/:issueId/documents/:key`
|
||||||
- `POST /issues/:issueId/checkout`
|
- `POST /issues/:issueId/checkout`
|
||||||
- `POST /issues/:issueId/release`
|
- `POST /issues/:issueId/release`
|
||||||
|
- `POST /issues/:issueId/admin/force-release` (board-only lock recovery)
|
||||||
- `POST /issues/:issueId/comments`
|
- `POST /issues/:issueId/comments`
|
||||||
- `GET /issues/:issueId/comments`
|
- `GET /issues/:issueId/comments`
|
||||||
- `POST /companies/:companyId/issues/:issueId/attachments` (multipart upload)
|
- `POST /companies/:companyId/issues/:issueId/attachments` (multipart upload)
|
||||||
@@ -508,6 +509,8 @@ Server behavior:
|
|||||||
2. if updated row count is 0, return `409` with current owner/status
|
2. if updated row count is 0, return `409` with current owner/status
|
||||||
3. successful checkout sets `assignee_agent_id`, `status = in_progress`, and `started_at`
|
3. successful checkout sets `assignee_agent_id`, `status = in_progress`, and `started_at`
|
||||||
|
|
||||||
|
`POST /issues/:issueId/admin/force-release` is an operator recovery endpoint for stale harness locks. It requires board access to the issue company, clears checkout and execution run lock fields, and may clear the agent assignee when `clearAssignee=true` is passed. The route must write an `issue.admin_force_release` activity log entry containing the previous checkout and execution run IDs.
|
||||||
|
|
||||||
## 10.5 Projects
|
## 10.5 Projects
|
||||||
|
|
||||||
- `GET /companies/:companyId/projects`
|
- `GET /companies/:companyId/projects`
|
||||||
|
|||||||
286
server/src/__tests__/issue-stale-execution-lock-routes.test.ts
Normal file
286
server/src/__tests__/issue-stale-execution-lock-routes.test.ts
Normal file
@@ -0,0 +1,286 @@
|
|||||||
|
import { randomUUID } from "node:crypto";
|
||||||
|
import express from "express";
|
||||||
|
import request from "supertest";
|
||||||
|
import { eq } from "drizzle-orm";
|
||||||
|
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||||
|
import {
|
||||||
|
activityLog,
|
||||||
|
agents,
|
||||||
|
companies,
|
||||||
|
createDb,
|
||||||
|
heartbeatRuns,
|
||||||
|
issueComments,
|
||||||
|
issueRelations,
|
||||||
|
issues,
|
||||||
|
} from "@paperclipai/db";
|
||||||
|
import {
|
||||||
|
getEmbeddedPostgresTestSupport,
|
||||||
|
startEmbeddedPostgresTestDatabase,
|
||||||
|
} from "./helpers/embedded-postgres.js";
|
||||||
|
import { errorHandler } from "../middleware/index.js";
|
||||||
|
import { issueRoutes } from "../routes/issues.js";
|
||||||
|
|
||||||
|
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||||
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||||
|
|
||||||
|
if (!embeddedPostgresSupport.supported) {
|
||||||
|
console.warn(
|
||||||
|
`Skipping embedded Postgres stale execution lock route tests on this host: ${
|
||||||
|
embeddedPostgresSupport.reason ?? "unsupported environment"
|
||||||
|
}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
describeEmbeddedPostgres("stale issue execution lock routes", () => {
|
||||||
|
let db!: ReturnType<typeof createDb>;
|
||||||
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-stale-execution-lock-routes-");
|
||||||
|
db = createDb(tempDb.connectionString);
|
||||||
|
}, 20_000);
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await db.delete(issueComments);
|
||||||
|
await db.delete(issueRelations);
|
||||||
|
await db.delete(activityLog);
|
||||||
|
await db.delete(issues);
|
||||||
|
await db.delete(heartbeatRuns);
|
||||||
|
await db.delete(agents);
|
||||||
|
await db.delete(companies);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterAll(async () => {
|
||||||
|
await tempDb?.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
function createApp(actor: Express.Request["actor"]) {
|
||||||
|
const app = express();
|
||||||
|
app.use(express.json());
|
||||||
|
app.use((req, _res, next) => {
|
||||||
|
req.actor = actor;
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
app.use("/api", issueRoutes(db, {} as any));
|
||||||
|
app.use(errorHandler);
|
||||||
|
return app;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function seedCompanyAgentAndRuns() {
|
||||||
|
const companyId = randomUUID();
|
||||||
|
const agentId = randomUUID();
|
||||||
|
const failedRunId = randomUUID();
|
||||||
|
const currentRunId = randomUUID();
|
||||||
|
|
||||||
|
await db.insert(companies).values({
|
||||||
|
id: companyId,
|
||||||
|
name: "Paperclip",
|
||||||
|
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||||
|
requireBoardApprovalForNewAgents: false,
|
||||||
|
});
|
||||||
|
await db.insert(agents).values({
|
||||||
|
id: agentId,
|
||||||
|
companyId,
|
||||||
|
name: "CodexCoder",
|
||||||
|
role: "engineer",
|
||||||
|
status: "active",
|
||||||
|
adapterType: "codex_local",
|
||||||
|
adapterConfig: {},
|
||||||
|
runtimeConfig: {},
|
||||||
|
permissions: {},
|
||||||
|
});
|
||||||
|
await db.insert(heartbeatRuns).values([
|
||||||
|
{
|
||||||
|
id: failedRunId,
|
||||||
|
companyId,
|
||||||
|
agentId,
|
||||||
|
status: "failed",
|
||||||
|
invocationSource: "manual",
|
||||||
|
finishedAt: new Date(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: currentRunId,
|
||||||
|
companyId,
|
||||||
|
agentId,
|
||||||
|
status: "running",
|
||||||
|
invocationSource: "manual",
|
||||||
|
startedAt: new Date(),
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
|
||||||
|
return { companyId, agentId, failedRunId, currentRunId };
|
||||||
|
}
|
||||||
|
|
||||||
|
function agentActor(companyId: string, agentId: string, runId: string): Express.Request["actor"] {
|
||||||
|
return {
|
||||||
|
type: "agent",
|
||||||
|
agentId,
|
||||||
|
companyId,
|
||||||
|
runId,
|
||||||
|
source: "agent_jwt",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function boardActor(companyId: string): Express.Request["actor"] {
|
||||||
|
return {
|
||||||
|
type: "board",
|
||||||
|
userId: "board-user",
|
||||||
|
companyIds: [companyId],
|
||||||
|
memberships: [{ companyId, membershipRole: "admin", status: "active" }],
|
||||||
|
isInstanceAdmin: false,
|
||||||
|
source: "session",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
it("allows an assigned agent PATCH to recover a terminal stale executionRunId", async () => {
|
||||||
|
const { companyId, agentId, failedRunId, currentRunId } = await seedCompanyAgentAndRuns();
|
||||||
|
const issueId = randomUUID();
|
||||||
|
await db.insert(issues).values({
|
||||||
|
id: issueId,
|
||||||
|
companyId,
|
||||||
|
title: "Stale execution lock",
|
||||||
|
status: "in_progress",
|
||||||
|
priority: "high",
|
||||||
|
assigneeAgentId: agentId,
|
||||||
|
checkoutRunId: null,
|
||||||
|
executionRunId: failedRunId,
|
||||||
|
executionAgentNameKey: "codexcoder",
|
||||||
|
executionLockedAt: new Date(),
|
||||||
|
});
|
||||||
|
|
||||||
|
const res = await request(createApp(agentActor(companyId, agentId, currentRunId)))
|
||||||
|
.patch(`/api/issues/${issueId}`)
|
||||||
|
.send({ title: "Recovered execution lock" });
|
||||||
|
|
||||||
|
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||||
|
expect(res.body.title).toBe("Recovered execution lock");
|
||||||
|
|
||||||
|
const row = await db
|
||||||
|
.select({
|
||||||
|
title: issues.title,
|
||||||
|
checkoutRunId: issues.checkoutRunId,
|
||||||
|
executionRunId: issues.executionRunId,
|
||||||
|
})
|
||||||
|
.from(issues)
|
||||||
|
.where(eq(issues.id, issueId))
|
||||||
|
.then((rows) => rows[0]);
|
||||||
|
expect(row).toEqual({
|
||||||
|
title: "Recovered execution lock",
|
||||||
|
checkoutRunId: currentRunId,
|
||||||
|
executionRunId: currentRunId,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("allows the rightful assignee to release after the owning run failed", async () => {
|
||||||
|
const { companyId, agentId, failedRunId, currentRunId } = await seedCompanyAgentAndRuns();
|
||||||
|
const issueId = randomUUID();
|
||||||
|
await db.insert(issues).values({
|
||||||
|
id: issueId,
|
||||||
|
companyId,
|
||||||
|
title: "Failed run release",
|
||||||
|
status: "in_progress",
|
||||||
|
priority: "high",
|
||||||
|
assigneeAgentId: agentId,
|
||||||
|
checkoutRunId: failedRunId,
|
||||||
|
executionRunId: failedRunId,
|
||||||
|
executionAgentNameKey: "codexcoder",
|
||||||
|
executionLockedAt: new Date(),
|
||||||
|
});
|
||||||
|
|
||||||
|
const res = await request(createApp(agentActor(companyId, agentId, currentRunId)))
|
||||||
|
.post(`/api/issues/${issueId}/release`)
|
||||||
|
.send();
|
||||||
|
|
||||||
|
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||||
|
|
||||||
|
const row = await db
|
||||||
|
.select({
|
||||||
|
status: issues.status,
|
||||||
|
assigneeAgentId: issues.assigneeAgentId,
|
||||||
|
checkoutRunId: issues.checkoutRunId,
|
||||||
|
executionRunId: issues.executionRunId,
|
||||||
|
executionLockedAt: issues.executionLockedAt,
|
||||||
|
})
|
||||||
|
.from(issues)
|
||||||
|
.where(eq(issues.id, issueId))
|
||||||
|
.then((rows) => rows[0]);
|
||||||
|
expect(row).toEqual({
|
||||||
|
status: "todo",
|
||||||
|
assigneeAgentId: null,
|
||||||
|
checkoutRunId: null,
|
||||||
|
executionRunId: null,
|
||||||
|
executionLockedAt: null,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("restricts admin force-release to board users with company access and writes an audit event", async () => {
|
||||||
|
const { companyId, agentId, failedRunId, currentRunId } = await seedCompanyAgentAndRuns();
|
||||||
|
const issueId = randomUUID();
|
||||||
|
await db.insert(issues).values({
|
||||||
|
id: issueId,
|
||||||
|
companyId,
|
||||||
|
title: "Admin force release",
|
||||||
|
status: "in_progress",
|
||||||
|
priority: "high",
|
||||||
|
assigneeAgentId: agentId,
|
||||||
|
checkoutRunId: currentRunId,
|
||||||
|
executionRunId: failedRunId,
|
||||||
|
executionAgentNameKey: "codexcoder",
|
||||||
|
executionLockedAt: new Date(),
|
||||||
|
});
|
||||||
|
|
||||||
|
await request(createApp(agentActor(companyId, agentId, currentRunId)))
|
||||||
|
.post(`/api/issues/${issueId}/admin/force-release`)
|
||||||
|
.expect(403);
|
||||||
|
await request(createApp({
|
||||||
|
type: "board",
|
||||||
|
userId: "outside-user",
|
||||||
|
companyIds: [],
|
||||||
|
memberships: [],
|
||||||
|
isInstanceAdmin: false,
|
||||||
|
source: "session",
|
||||||
|
}))
|
||||||
|
.post(`/api/issues/${issueId}/admin/force-release`)
|
||||||
|
.expect(403);
|
||||||
|
|
||||||
|
const res = await request(createApp(boardActor(companyId)))
|
||||||
|
.post(`/api/issues/${issueId}/admin/force-release?clearAssignee=true`)
|
||||||
|
.send();
|
||||||
|
|
||||||
|
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||||
|
expect(res.body.issue).toMatchObject({
|
||||||
|
id: issueId,
|
||||||
|
assigneeAgentId: null,
|
||||||
|
checkoutRunId: null,
|
||||||
|
executionRunId: null,
|
||||||
|
executionLockedAt: null,
|
||||||
|
});
|
||||||
|
expect(res.body.previous).toEqual({
|
||||||
|
checkoutRunId: currentRunId,
|
||||||
|
executionRunId: failedRunId,
|
||||||
|
});
|
||||||
|
|
||||||
|
const audit = await db
|
||||||
|
.select({
|
||||||
|
action: activityLog.action,
|
||||||
|
actorType: activityLog.actorType,
|
||||||
|
actorId: activityLog.actorId,
|
||||||
|
details: activityLog.details,
|
||||||
|
})
|
||||||
|
.from(activityLog)
|
||||||
|
.where(eq(activityLog.action, "issue.admin_force_release"))
|
||||||
|
.then((rows) => rows[0]);
|
||||||
|
expect(audit).toMatchObject({
|
||||||
|
action: "issue.admin_force_release",
|
||||||
|
actorType: "user",
|
||||||
|
actorId: "board-user",
|
||||||
|
details: {
|
||||||
|
issueId,
|
||||||
|
actorUserId: "board-user",
|
||||||
|
prevCheckoutRunId: currentRunId,
|
||||||
|
prevExecutionRunId: failedRunId,
|
||||||
|
clearAssignee: true,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -9,6 +9,7 @@ import {
|
|||||||
createDb,
|
createDb,
|
||||||
executionWorkspaces,
|
executionWorkspaces,
|
||||||
goals,
|
goals,
|
||||||
|
heartbeatRuns,
|
||||||
instanceSettings,
|
instanceSettings,
|
||||||
issueComments,
|
issueComments,
|
||||||
issueInboxArchives,
|
issueInboxArchives,
|
||||||
@@ -1879,3 +1880,135 @@ describeEmbeddedPostgres("issueService.findMentionedProjectIds", () => {
|
|||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describeEmbeddedPostgres("issueService.clearExecutionRunIfTerminal", () => {
|
||||||
|
let db!: ReturnType<typeof createDb>;
|
||||||
|
let svc!: ReturnType<typeof issueService>;
|
||||||
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issues-execution-lock-");
|
||||||
|
db = createDb(tempDb.connectionString);
|
||||||
|
svc = issueService(db);
|
||||||
|
}, 20_000);
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await db.delete(issueComments);
|
||||||
|
await db.delete(issueRelations);
|
||||||
|
await db.delete(issueInboxArchives);
|
||||||
|
await db.delete(activityLog);
|
||||||
|
await db.delete(issues);
|
||||||
|
await db.delete(heartbeatRuns);
|
||||||
|
await db.delete(executionWorkspaces);
|
||||||
|
await db.delete(projectWorkspaces);
|
||||||
|
await db.delete(projects);
|
||||||
|
await db.delete(goals);
|
||||||
|
await db.delete(agents);
|
||||||
|
await db.delete(instanceSettings);
|
||||||
|
await db.delete(companies);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterAll(async () => {
|
||||||
|
await tempDb?.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
async function seedIssueWithRun(status: string | null) {
|
||||||
|
const companyId = randomUUID();
|
||||||
|
const agentId = randomUUID();
|
||||||
|
const issueId = randomUUID();
|
||||||
|
const runId = status ? randomUUID() : null;
|
||||||
|
|
||||||
|
await db.insert(companies).values({
|
||||||
|
id: companyId,
|
||||||
|
name: "Paperclip",
|
||||||
|
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||||
|
requireBoardApprovalForNewAgents: false,
|
||||||
|
});
|
||||||
|
await db.insert(agents).values({
|
||||||
|
id: agentId,
|
||||||
|
companyId,
|
||||||
|
name: "CodexCoder",
|
||||||
|
role: "engineer",
|
||||||
|
status: "active",
|
||||||
|
adapterType: "codex_local",
|
||||||
|
adapterConfig: {},
|
||||||
|
runtimeConfig: {},
|
||||||
|
permissions: {},
|
||||||
|
});
|
||||||
|
if (runId) {
|
||||||
|
await db.insert(heartbeatRuns).values({
|
||||||
|
id: runId,
|
||||||
|
companyId,
|
||||||
|
agentId,
|
||||||
|
status,
|
||||||
|
invocationSource: "manual",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
await db.insert(issues).values({
|
||||||
|
id: issueId,
|
||||||
|
companyId,
|
||||||
|
title: "Execution lock",
|
||||||
|
status: "in_progress",
|
||||||
|
priority: "medium",
|
||||||
|
assigneeAgentId: agentId,
|
||||||
|
executionRunId: runId,
|
||||||
|
executionAgentNameKey: runId ? "codexcoder" : null,
|
||||||
|
executionLockedAt: runId ? new Date() : null,
|
||||||
|
});
|
||||||
|
|
||||||
|
return { issueId, runId };
|
||||||
|
}
|
||||||
|
|
||||||
|
it("clears execution locks owned by terminal runs", async () => {
|
||||||
|
const { issueId } = await seedIssueWithRun("failed");
|
||||||
|
|
||||||
|
await expect(svc.clearExecutionRunIfTerminal(issueId)).resolves.toBe(true);
|
||||||
|
|
||||||
|
const row = await db
|
||||||
|
.select({
|
||||||
|
executionRunId: issues.executionRunId,
|
||||||
|
executionAgentNameKey: issues.executionAgentNameKey,
|
||||||
|
executionLockedAt: issues.executionLockedAt,
|
||||||
|
})
|
||||||
|
.from(issues)
|
||||||
|
.where(eq(issues.id, issueId))
|
||||||
|
.then((rows) => rows[0]);
|
||||||
|
expect(row).toEqual({
|
||||||
|
executionRunId: null,
|
||||||
|
executionAgentNameKey: null,
|
||||||
|
executionLockedAt: null,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not clear execution locks owned by live runs", async () => {
|
||||||
|
const { issueId, runId } = await seedIssueWithRun("running");
|
||||||
|
|
||||||
|
await expect(svc.clearExecutionRunIfTerminal(issueId)).resolves.toBe(false);
|
||||||
|
|
||||||
|
const row = await db
|
||||||
|
.select({
|
||||||
|
executionRunId: issues.executionRunId,
|
||||||
|
executionAgentNameKey: issues.executionAgentNameKey,
|
||||||
|
executionLockedAt: issues.executionLockedAt,
|
||||||
|
})
|
||||||
|
.from(issues)
|
||||||
|
.where(eq(issues.id, issueId))
|
||||||
|
.then((rows) => rows[0]);
|
||||||
|
expect(row?.executionRunId).toBe(runId);
|
||||||
|
expect(row?.executionAgentNameKey).toBe("codexcoder");
|
||||||
|
expect(row?.executionLockedAt).toBeInstanceOf(Date);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not update issues without an execution lock", async () => {
|
||||||
|
const { issueId } = await seedIssueWithRun(null);
|
||||||
|
|
||||||
|
await expect(svc.clearExecutionRunIfTerminal(issueId)).resolves.toBe(false);
|
||||||
|
|
||||||
|
const row = await db
|
||||||
|
.select({ executionRunId: issues.executionRunId, executionLockedAt: issues.executionLockedAt })
|
||||||
|
.from(issues)
|
||||||
|
.where(eq(issues.id, issueId))
|
||||||
|
.then((rows) => rows[0]);
|
||||||
|
expect(row).toEqual({ executionRunId: null, executionLockedAt: null });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -2574,6 +2574,52 @@ export function issueRoutes(
|
|||||||
res.json(released);
|
res.json(released);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
router.post("/issues/:id/admin/force-release", async (req, res) => {
|
||||||
|
if (req.actor.type !== "board") {
|
||||||
|
res.status(403).json({ error: "Board access required" });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (!req.actor.userId) {
|
||||||
|
throw forbidden("Board user context required");
|
||||||
|
}
|
||||||
|
|
||||||
|
const id = req.params.id as string;
|
||||||
|
const existing = await svc.getById(id);
|
||||||
|
if (!existing) {
|
||||||
|
res.status(404).json({ error: "Issue not found" });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
assertCompanyAccess(req, existing.companyId);
|
||||||
|
|
||||||
|
const clearAssignee = req.query.clearAssignee === "true";
|
||||||
|
const result = await svc.adminForceRelease(id, { clearAssignee });
|
||||||
|
if (!result) {
|
||||||
|
res.status(404).json({ error: "Issue not found" });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const actor = getActorInfo(req);
|
||||||
|
await logActivity(db, {
|
||||||
|
companyId: result.issue.companyId,
|
||||||
|
actorType: actor.actorType,
|
||||||
|
actorId: actor.actorId,
|
||||||
|
agentId: actor.agentId,
|
||||||
|
runId: actor.runId,
|
||||||
|
action: "issue.admin_force_release",
|
||||||
|
entityType: "issue",
|
||||||
|
entityId: result.issue.id,
|
||||||
|
details: {
|
||||||
|
issueId: result.issue.id,
|
||||||
|
actorUserId: req.actor.userId,
|
||||||
|
prevCheckoutRunId: result.previous.checkoutRunId,
|
||||||
|
prevExecutionRunId: result.previous.executionRunId,
|
||||||
|
clearAssignee,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
res.json(result);
|
||||||
|
});
|
||||||
|
|
||||||
router.get("/issues/:id/comments", async (req, res) => {
|
router.get("/issues/:id/comments", async (req, res) => {
|
||||||
const id = req.params.id as string;
|
const id = req.params.id as string;
|
||||||
const issue = await svc.getById(id);
|
const issue = await svc.getById(id);
|
||||||
|
|||||||
@@ -1242,7 +1242,87 @@ export function issueService(db: Db) {
|
|||||||
return adopted;
|
return adopted;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function adoptUnownedCheckoutRun(input: {
|
||||||
|
issueId: string;
|
||||||
|
actorAgentId: string;
|
||||||
|
actorRunId: string;
|
||||||
|
}) {
|
||||||
|
const now = new Date();
|
||||||
|
const adopted = await db
|
||||||
|
.update(issues)
|
||||||
|
.set({
|
||||||
|
checkoutRunId: input.actorRunId,
|
||||||
|
executionRunId: input.actorRunId,
|
||||||
|
executionLockedAt: now,
|
||||||
|
updatedAt: now,
|
||||||
|
})
|
||||||
|
.where(
|
||||||
|
and(
|
||||||
|
eq(issues.id, input.issueId),
|
||||||
|
eq(issues.status, "in_progress"),
|
||||||
|
eq(issues.assigneeAgentId, input.actorAgentId),
|
||||||
|
isNull(issues.checkoutRunId),
|
||||||
|
or(isNull(issues.executionRunId), eq(issues.executionRunId, input.actorRunId)),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.returning({
|
||||||
|
id: issues.id,
|
||||||
|
status: issues.status,
|
||||||
|
assigneeAgentId: issues.assigneeAgentId,
|
||||||
|
checkoutRunId: issues.checkoutRunId,
|
||||||
|
executionRunId: issues.executionRunId,
|
||||||
|
})
|
||||||
|
.then((rows) => rows[0] ?? null);
|
||||||
|
|
||||||
|
return adopted;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function clearExecutionRunIfTerminal(issueId: string): Promise<boolean> {
|
||||||
|
return db.transaction(async (tx) => {
|
||||||
|
await tx.execute(
|
||||||
|
sql`select ${issues.id} from ${issues} where ${issues.id} = ${issueId} for update`,
|
||||||
|
);
|
||||||
|
const issue = await tx
|
||||||
|
.select({ executionRunId: issues.executionRunId })
|
||||||
|
.from(issues)
|
||||||
|
.where(eq(issues.id, issueId))
|
||||||
|
.then((rows) => rows[0] ?? null);
|
||||||
|
if (!issue?.executionRunId) return false;
|
||||||
|
|
||||||
|
await tx.execute(
|
||||||
|
sql`select ${heartbeatRuns.id} from ${heartbeatRuns} where ${heartbeatRuns.id} = ${issue.executionRunId} for update`,
|
||||||
|
);
|
||||||
|
const run = await tx
|
||||||
|
.select({ status: heartbeatRuns.status })
|
||||||
|
.from(heartbeatRuns)
|
||||||
|
.where(eq(heartbeatRuns.id, issue.executionRunId))
|
||||||
|
.then((rows) => rows[0] ?? null);
|
||||||
|
if (run && !TERMINAL_HEARTBEAT_RUN_STATUSES.has(run.status)) return false;
|
||||||
|
|
||||||
|
const updated = await tx
|
||||||
|
.update(issues)
|
||||||
|
.set({
|
||||||
|
executionRunId: null,
|
||||||
|
executionAgentNameKey: null,
|
||||||
|
executionLockedAt: null,
|
||||||
|
updatedAt: new Date(),
|
||||||
|
})
|
||||||
|
.where(
|
||||||
|
and(
|
||||||
|
eq(issues.id, issueId),
|
||||||
|
eq(issues.executionRunId, issue.executionRunId),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.returning({ id: issues.id })
|
||||||
|
.then((rows) => rows[0] ?? null);
|
||||||
|
|
||||||
|
return Boolean(updated);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
|
clearExecutionRunIfTerminal,
|
||||||
|
|
||||||
list: async (companyId: string, filters?: IssueFilters) => {
|
list: async (companyId: string, filters?: IssueFilters) => {
|
||||||
const conditions = [eq(issues.companyId, companyId)];
|
const conditions = [eq(issues.companyId, companyId)];
|
||||||
const limit = typeof filters?.limit === "number" && Number.isFinite(filters.limit)
|
const limit = typeof filters?.limit === "number" && Number.isFinite(filters.limit)
|
||||||
@@ -2112,38 +2192,7 @@ export function issueService(db: Db) {
|
|||||||
|
|
||||||
const now = new Date();
|
const now = new Date();
|
||||||
|
|
||||||
// Fix C: staleness detection — if executionRunId references a run that is no
|
await clearExecutionRunIfTerminal(id);
|
||||||
// longer queued or running, clear it before applying the execution lock condition
|
|
||||||
// so a dead lock can't produce a spurious 409.
|
|
||||||
// Wrapped in a transaction with SELECT FOR UPDATE to make the read + clear atomic,
|
|
||||||
// matching the existing pattern in enqueueWakeup().
|
|
||||||
await db.transaction(async (tx) => {
|
|
||||||
await tx.execute(
|
|
||||||
sql`select id from issues where id = ${id} for update`,
|
|
||||||
);
|
|
||||||
const preCheckRow = await tx
|
|
||||||
.select({ executionRunId: issues.executionRunId })
|
|
||||||
.from(issues)
|
|
||||||
.where(eq(issues.id, id))
|
|
||||||
.then((rows) => rows[0] ?? null);
|
|
||||||
if (!preCheckRow?.executionRunId) return;
|
|
||||||
const lockRun = await tx
|
|
||||||
.select({ id: heartbeatRuns.id, status: heartbeatRuns.status })
|
|
||||||
.from(heartbeatRuns)
|
|
||||||
.where(eq(heartbeatRuns.id, preCheckRow.executionRunId))
|
|
||||||
.then((rows) => rows[0] ?? null);
|
|
||||||
if (!lockRun || (lockRun.status !== "queued" && lockRun.status !== "running")) {
|
|
||||||
await tx
|
|
||||||
.update(issues)
|
|
||||||
.set({ executionRunId: null, executionAgentNameKey: null, executionLockedAt: null, updatedAt: now })
|
|
||||||
.where(
|
|
||||||
and(
|
|
||||||
eq(issues.id, id),
|
|
||||||
eq(issues.executionRunId, preCheckRow.executionRunId),
|
|
||||||
),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
const dependencyReadiness = await listIssueDependencyReadinessMap(db, issueCompany.companyId, [id]);
|
const dependencyReadiness = await listIssueDependencyReadinessMap(db, issueCompany.companyId, [id]);
|
||||||
const unresolvedBlockerIssueIds = dependencyReadiness.get(id)?.unresolvedBlockerIssueIds ?? [];
|
const unresolvedBlockerIssueIds = dependencyReadiness.get(id)?.unresolvedBlockerIssueIds ?? [];
|
||||||
@@ -2272,12 +2321,14 @@ export function issueService(db: Db) {
|
|||||||
},
|
},
|
||||||
|
|
||||||
assertCheckoutOwner: async (id: string, actorAgentId: string, actorRunId: string | null) => {
|
assertCheckoutOwner: async (id: string, actorAgentId: string, actorRunId: string | null) => {
|
||||||
|
await clearExecutionRunIfTerminal(id);
|
||||||
const current = await db
|
const current = await db
|
||||||
.select({
|
.select({
|
||||||
id: issues.id,
|
id: issues.id,
|
||||||
status: issues.status,
|
status: issues.status,
|
||||||
assigneeAgentId: issues.assigneeAgentId,
|
assigneeAgentId: issues.assigneeAgentId,
|
||||||
checkoutRunId: issues.checkoutRunId,
|
checkoutRunId: issues.checkoutRunId,
|
||||||
|
executionRunId: issues.executionRunId,
|
||||||
})
|
})
|
||||||
.from(issues)
|
.from(issues)
|
||||||
.where(eq(issues.id, id))
|
.where(eq(issues.id, id))
|
||||||
@@ -2293,6 +2344,27 @@ export function issueService(db: Db) {
|
|||||||
return { ...current, adoptedFromRunId: null as string | null };
|
return { ...current, adoptedFromRunId: null as string | null };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
actorRunId &&
|
||||||
|
current.status === "in_progress" &&
|
||||||
|
current.assigneeAgentId === actorAgentId &&
|
||||||
|
current.checkoutRunId == null &&
|
||||||
|
(current.executionRunId == null || current.executionRunId === actorRunId)
|
||||||
|
) {
|
||||||
|
const adopted = await adoptUnownedCheckoutRun({
|
||||||
|
issueId: id,
|
||||||
|
actorAgentId,
|
||||||
|
actorRunId,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (adopted) {
|
||||||
|
return {
|
||||||
|
...adopted,
|
||||||
|
adoptedFromRunId: null as string | null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (
|
if (
|
||||||
actorRunId &&
|
actorRunId &&
|
||||||
current.status === "in_progress" &&
|
current.status === "in_progress" &&
|
||||||
@@ -2320,12 +2392,14 @@ export function issueService(db: Db) {
|
|||||||
status: current.status,
|
status: current.status,
|
||||||
assigneeAgentId: current.assigneeAgentId,
|
assigneeAgentId: current.assigneeAgentId,
|
||||||
checkoutRunId: current.checkoutRunId,
|
checkoutRunId: current.checkoutRunId,
|
||||||
|
executionRunId: current.executionRunId,
|
||||||
actorAgentId,
|
actorAgentId,
|
||||||
actorRunId,
|
actorRunId,
|
||||||
});
|
});
|
||||||
},
|
},
|
||||||
|
|
||||||
release: async (id: string, actorAgentId?: string, actorRunId?: string | null) => {
|
release: async (id: string, actorAgentId?: string, actorRunId?: string | null) => {
|
||||||
|
await clearExecutionRunIfTerminal(id);
|
||||||
const existing = await db
|
const existing = await db
|
||||||
.select()
|
.select()
|
||||||
.from(issues)
|
.from(issues)
|
||||||
@@ -2343,12 +2417,15 @@ export function issueService(db: Db) {
|
|||||||
existing.checkoutRunId &&
|
existing.checkoutRunId &&
|
||||||
!sameRunLock(existing.checkoutRunId, actorRunId ?? null)
|
!sameRunLock(existing.checkoutRunId, actorRunId ?? null)
|
||||||
) {
|
) {
|
||||||
throw conflict("Only checkout run can release issue", {
|
const stale = await isTerminalOrMissingHeartbeatRun(existing.checkoutRunId);
|
||||||
issueId: existing.id,
|
if (!stale) {
|
||||||
assigneeAgentId: existing.assigneeAgentId,
|
throw conflict("Only checkout run can release issue", {
|
||||||
checkoutRunId: existing.checkoutRunId,
|
issueId: existing.id,
|
||||||
actorRunId: actorRunId ?? null,
|
assigneeAgentId: existing.assigneeAgentId,
|
||||||
});
|
checkoutRunId: existing.checkoutRunId,
|
||||||
|
actorRunId: actorRunId ?? null,
|
||||||
|
});
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const updated = await db
|
const updated = await db
|
||||||
@@ -2357,6 +2434,9 @@ export function issueService(db: Db) {
|
|||||||
status: "todo",
|
status: "todo",
|
||||||
assigneeAgentId: null,
|
assigneeAgentId: null,
|
||||||
checkoutRunId: null,
|
checkoutRunId: null,
|
||||||
|
executionRunId: null,
|
||||||
|
executionAgentNameKey: null,
|
||||||
|
executionLockedAt: null,
|
||||||
updatedAt: new Date(),
|
updatedAt: new Date(),
|
||||||
})
|
})
|
||||||
.where(eq(issues.id, id))
|
.where(eq(issues.id, id))
|
||||||
@@ -2367,6 +2447,51 @@ export function issueService(db: Db) {
|
|||||||
return enriched;
|
return enriched;
|
||||||
},
|
},
|
||||||
|
|
||||||
|
adminForceRelease: async (id: string, options: { clearAssignee?: boolean } = {}) =>
|
||||||
|
db.transaction(async (tx) => {
|
||||||
|
await tx.execute(
|
||||||
|
sql`select ${issues.id} from ${issues} where ${issues.id} = ${id} for update`,
|
||||||
|
);
|
||||||
|
const existing = await tx
|
||||||
|
.select({
|
||||||
|
id: issues.id,
|
||||||
|
checkoutRunId: issues.checkoutRunId,
|
||||||
|
executionRunId: issues.executionRunId,
|
||||||
|
})
|
||||||
|
.from(issues)
|
||||||
|
.where(eq(issues.id, id))
|
||||||
|
.then((rows) => rows[0] ?? null);
|
||||||
|
if (!existing) return null;
|
||||||
|
|
||||||
|
const patch: Partial<typeof issues.$inferInsert> = {
|
||||||
|
checkoutRunId: null,
|
||||||
|
executionRunId: null,
|
||||||
|
executionAgentNameKey: null,
|
||||||
|
executionLockedAt: null,
|
||||||
|
updatedAt: new Date(),
|
||||||
|
};
|
||||||
|
if (options.clearAssignee) {
|
||||||
|
patch.assigneeAgentId = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const updated = await tx
|
||||||
|
.update(issues)
|
||||||
|
.set(patch)
|
||||||
|
.where(eq(issues.id, id))
|
||||||
|
.returning()
|
||||||
|
.then((rows) => rows[0] ?? null);
|
||||||
|
if (!updated) return null;
|
||||||
|
|
||||||
|
const [enriched] = await withIssueLabels(tx, [updated]);
|
||||||
|
return {
|
||||||
|
issue: enriched,
|
||||||
|
previous: {
|
||||||
|
checkoutRunId: existing.checkoutRunId,
|
||||||
|
executionRunId: existing.executionRunId,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}),
|
||||||
|
|
||||||
listLabels: (companyId: string) =>
|
listLabels: (companyId: string) =>
|
||||||
db.select().from(labels).where(eq(labels.companyId, companyId)).orderBy(asc(labels.name), asc(labels.id)),
|
db.select().from(labels).where(eq(labels.companyId, companyId)).orderBy(asc(labels.name), asc(labels.id)),
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user