Files

T

Wenxin Duan 9933672c2e docs: align implementation plan with behavior tests

2026-09-07 10:23:10 +08:00

49 KiB

Raw Blame History

Agent Studio Mini Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a runnable training-grade Pi Coding Agent harness with assistant-ui, server-authoritative events and approvals, session-scoped model credentials, and secure image input.

Architecture: A pnpm workspace separates portable protocols (packages/shared), the stateful agent harness (packages/harness), Fastify transport (apps/server), and the assistant-ui presentation layer (apps/web). Pi AgentSession and the bounded EventStore own runtime truth; the browser consumes HTTP/SSE through one external store and never calls a model or executes tools directly.

Tech Stack: TypeScript strict mode, Node.js 20+, pnpm workspace, React, Vite, assistant-ui, Fastify, @earendil-works/pi-coding-agent, TypeBox, Zod, SSE, sharp, Vitest, React Testing Library, ESLint.

Spec: docs/superpowers/specs/2026-09-07-agent-studio-mini-design.md

Global Constraints

Use Node.js 20+ and TypeScript strict mode.
Install current stable dependencies, then treat installed public exports and type definitions as authoritative.
packages/shared must not import Node-only modules; packages/harness must not import React or assistant-ui.
Pi AgentSession and Harness EventStore are runtime truth; assistant-ui is a projection layer.
API keys remain only in session-scoped server memory and never enter URLs, logs, SSE, browser storage, or snapshots.
Only read, grep, find, ls, mock_ads_metrics, and save_report_draft are registered; unknown tools are denied.
Default tests use a fake AgentSession and never call a real model or external network.
The server must not claim that Project Trust is an OS sandbox.
Every task follows red-green-refactor, runs its focused tests, and creates one reviewable commit.

Task 1: Workspace foundation and verified dependency baseline

Files:

Create: package.json
Create: pnpm-workspace.yaml
Create: tsconfig.base.json
Create: eslint.config.js
Create: .gitignore
Create: .env.example
Create: README.md
Create: packages/shared/package.json
Create: packages/shared/tsconfig.json
Create: packages/shared/src/index.ts
Create: packages/harness/package.json
Create: packages/harness/tsconfig.json
Create: apps/server/package.json
Create: apps/server/tsconfig.json
Create: apps/web/package.json
Create: apps/web/tsconfig.json
Create: apps/web/vite.config.ts
Create: apps/web/index.html

Interfaces:

Consumes: Node.js 20+ and pnpm 11 available in the development environment.
Produces: root commands dev, lint, typecheck, test, and build; workspace package names @agent-studio/shared and @agent-studio/harness.
Step 1: Create strict workspace configuration and package scripts

Root scripts must delegate with pnpm -r, keep dev parallel, and make test use Vitest in run mode. tsconfig.base.json must enable strict, noUncheckedIndexedAccess, exactOptionalPropertyTypes, useUnknownInCatchVariables, verbatimModuleSyntax, and noEmit for checking. .env.example must contain only the empty/default values from the design spec.

Step 2: Install current stable dependencies

Run:

pnpm add -w -D typescript eslint @eslint/js typescript-eslint vitest @vitest/coverage-v8
pnpm --filter @agent-studio/shared add zod
pnpm --filter @agent-studio/harness add @agent-studio/shared @earendil-works/pi-coding-agent @sinclair/typebox sharp
pnpm --filter @agent-studio/server add @agent-studio/shared @agent-studio/harness fastify @fastify/multipart zod
pnpm --filter @agent-studio/web add @agent-studio/shared react react-dom @assistant-ui/react lucide-react
pnpm --filter @agent-studio/web add -D vite @vitejs/plugin-react jsdom @testing-library/react @testing-library/user-event @testing-library/jest-dom

Step 3: Record and inspect the installed public API

Run:

pnpm list -r --depth 0
rg "createAgentSession|DefaultResourceLoader|ModelRuntime|InMemoryCredentialStore|defineTool|isToolCallEventType" node_modules/.pnpm/@earendil-works+pi-coding-agent*/node_modules/@earendil-works/pi-coding-agent -g '*.d.ts'
rg "useExternalStoreRuntime|AttachmentAdapter|ThreadPrimitive|ComposerPrimitive" node_modules/.pnpm/@assistant-ui+react*/node_modules/@assistant-ui/react -g '*.d.ts'

Write exact dependency versions and any material signature differences into a temporary commit note in README.md; Task 16 expands it into the final version/API section.

Step 4: Verify the baseline

Run: pnpm test && pnpm typecheck && pnpm lint

Expected: all packages typecheck/lint without ignored errors; packages that do not yet contain tests exit cleanly through their explicit --passWithNoTests baseline configuration.

Step 5: Commit

git add package.json pnpm-lock.yaml pnpm-workspace.yaml tsconfig.base.json eslint.config.js .gitignore .env.example packages apps README.md
git commit -m "chore: scaffold Agent Studio workspace"

Task 2: Shared schemas, typed events, and API errors

Files:

Create: packages/shared/src/events.ts
Create: packages/shared/src/model-config.ts
Create: packages/shared/src/messages.ts
Create: packages/shared/src/errors.ts
Modify: packages/shared/src/index.ts
Create: packages/shared/tests/events.test.ts
Create: packages/shared/tests/model-config.test.ts
Create: packages/shared/tests/messages.test.ts
Create: packages/shared/tests/errors.test.ts

Interfaces:

Consumes: Zod.
Produces: HarnessEventSchema, HarnessEvent, EventInput<T>, ModelConnectionInputSchema, MessageInputSchema, ImageAttachmentSchema, ApiErrorSchema, and createApiError().
Step 1: Write failing schema tests

it("rejects an event payload for the wrong discriminant", () => {
  expect(() => HarnessEventSchema.parse({
    eventId: "evt_1", sessionId: "ses_1", timestamp: new Date().toISOString(),
    sequence: 1, type: "assistant.delta", payload: { approvalId: "ap_1" },
  })).toThrow();
});

it("requires text or at least one attachment", () => {
  expect(MessageInputSchema.safeParse({ text: " ", attachmentIds: [] }).success).toBe(false);
});

it("strips no unknown model fields because the schema is strict", () => {
  expect(ModelConnectionInputSchema.safeParse({
    providerId: "openai", api: "openai-responses", modelId: "gpt-x",
    supportsImages: true, unexpected: true,
  }).success).toBe(false);
});

Step 2: Run focused tests to verify failure

Run: pnpm --filter @agent-studio/shared test

Expected: FAIL because the schema modules do not exist.

Step 3: Implement all protocol schemas

Define a strict base schema with positive integer sequence and ISO timestamp. Add every event discriminant required by the spec with typed payloads. assistant.delta contains messageId and delta; Tool events contain toolCallId, toolName, redacted arguments/results and status; approval events contain approvalId; attachment events contain metadata only. Export inferred TypeScript types. createApiError(code, message, requestId, details?) must return an object accepted by ApiErrorSchema.

Step 4: Run shared verification

Run: pnpm --filter @agent-studio/shared test && pnpm --filter @agent-studio/shared typecheck

Expected: PASS, including invalid discriminants, strict model input, message minimum content, attachment ID validation, and error shape.

Step 5: Commit

git add packages/shared
git commit -m "feat(shared): define harness protocols"

Task 3: Atomic bounded EventStore

Files:

Create: packages/harness/src/events/event-store.ts
Create: packages/harness/src/events/in-memory-event-store.ts
Create: packages/harness/tests/event-store.test.ts
Modify: packages/harness/src/index.ts

Interfaces:

Consumes: HarnessEvent and EventInput<T> from shared.
Produces: EventCursor, EventReplay, EventStore, and InMemoryEventStore(maxEventsPerSession = 1000).
Step 1: Write failing EventStore tests

it("allocates monotonic per-session sequence and bounds history", () => {
  const store = new InMemoryEventStore(2);
  const first = store.append("ses_a", sessionStartedInput);
  const second = store.append("ses_a", runStartedInput);
  store.append("ses_a", runCompletedInput);
  expect([first.sequence, second.sequence]).toEqual([1, 2]);
  expect(store.listAfter("ses_a").events.map((e) => e.sequence)).toEqual([2, 3]);
});

it("reports a gap for a cursor older than retained history", () => {
  expect(store.listAfter("ses_a", { sequence: 0 }).gap).toBe(true);
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/harness test -- event-store

Expected: FAIL because the store is missing.

Step 3: Implement the store

Keep one private state object per Session with nextSequence, bounded events, listeners, and eventId index. append must create eventId/timestamp, increment and append synchronously before notifying a snapshot of listeners. listAfter accepts eventId or sequence, gives eventId precedence, and returns { events, gap, oldestSequence, latestSequence }. subscribe returns an idempotent unsubscribe.

Step 4: Verify pass and types

Run: pnpm --filter @agent-studio/harness test -- event-store && pnpm --filter @agent-studio/harness typecheck

Expected: PASS for sequence isolation, bounded retention, cursor precedence, gap behavior and unsubscribe.

Step 5: Commit

git add packages/harness
git commit -m "feat(harness): add bounded event store"

Task 4: SafeWorkspace and deterministic custom tools

Files:

Create: packages/harness/src/workspace/safe-workspace.ts
Create: packages/harness/src/tools/mock-ads-metrics.ts
Create: packages/harness/src/tools/save-report-draft.ts
Create: packages/harness/src/tools/tool-policy.ts
Create: packages/harness/tests/safe-workspace.test.ts
Create: packages/harness/tests/tools.test.ts
Create: fixtures/ads-account.json
Modify: packages/harness/src/index.ts

Interfaces:

Consumes: current Pi defineTool and TypeBox signatures confirmed in Task 1.
Produces: SafeWorkspace, createMockAdsMetricsTool(fixturePath), createSaveReportDraftTool(workspace), and decideToolPolicy(toolName): "allow" | "ask" | "deny".
Step 1: Write failing security and tool tests

it.each(["../outside", "/tmp/outside"])("rejects escape path %s", async (path) => {
  await expect(workspace.resolveReadable(path)).rejects.toMatchObject({ code: "PATH_OUTSIDE_WORKSPACE" });
});

it("rejects a symlink parent", async () => {
  await fs.symlink(outsideDir, join(root, "link"));
  await expect(workspace.resolveReadable("link/secret.txt")).rejects.toMatchObject({ code: "SYMLINK_NOT_ALLOWED" });
});

it("computes zero-safe ad metrics", async () => {
  const result = await executeMock({ accountId: "demo-account", days: 7 });
  expect(result.structured.roas).toBeCloseTo(result.structured.revenue / result.structured.spend, 2);
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/harness test -- safe-workspace tools

Expected: FAIL because workspace guards and tools are missing.

Step 3: Implement SafeWorkspace

Canonicalize the Session root once. For reads, resolve the candidate, reject protected basenames, verify relative containment, lstat every path segment, and reject symlinks. For the fixed report, open a random sibling temp file with exclusive creation, write UTF-8, fsync/close, recheck the parent, then rename to report.md. Expose only fixed report writing rather than a general write method.

Step 4: Implement both TypeBox tools and policy

The fixture must contain demo-account and campaigns with one high-CPA and one low-CTR anomaly. Validate days as integer 1–30 and accountId as a strict string enum or runtime lookup. Return Pi-compatible text content plus structured data with guarded division and two-decimal metrics. save_report_draft accepts only { title?: string; body: string }. Policy returns ask only for save_report_draft, allow for five permitted tools, deny otherwise.

Step 5: Verify

Run: pnpm --filter @agent-studio/harness test -- safe-workspace tools

Expected: PASS for traversal, absolute escape, symlinks, fixed report path, invalid account/day, metric precision and policy.

Step 6: Commit

git add packages/harness fixtures
git commit -m "feat(harness): secure workspace and custom tools"

Task 5: Standard Skill discovery and inferred Skill events

Files:

Create: packages/harness/src/skills/skill-catalog.ts
Create: packages/harness/src/skills/skill-event-inference.ts
Create: packages/harness/tests/skills.test.ts
Create: .agents/skills/ads-analysis/SKILL.md
Create: .pi/skills/report-writer/SKILL.md
Modify: packages/harness/src/index.ts

Interfaces:

Consumes: installed DefaultResourceLoader, settings extension point, EventStore, and canonical paths.
Produces: SkillCatalog.reload(), SkillCatalog.list(), SkillIndex.matchCanonicalPath(), and SkillEventInference.observeToolEvent().
Step 1: Write failing discovery tests

it("discovers project and additional skill roots without reading bodies into catalog", async () => {
  const catalog = await createFixtureCatalog({ extraDirs: [extraRoot] });
  expect(catalog.list().map((s) => s.name)).toEqual(expect.arrayContaining(["ads-analysis", "report-writer", "extra-skill"]));
  expect(JSON.stringify(catalog.list())).not.toContain("full secret skill body");
});

it("emits loaded only after a successful read of the canonical SKILL.md", () => {
  inference.onRequested(readCall);
  inference.onFailed(readFailure);
  expect(events.map((event) => event.type)).toEqual(["skill.requested"]);
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/harness test -- skills

Expected: FAIL because catalog and inference are absent.

Step 3: Implement catalog loading

Parse AGENT_SKILLS_DIRS with path.delimiter; resolve real paths, verify readable directories, deduplicate and preserve diagnostics for invalid roots. Configure the Pi loader through its current public settings/resource API without touching global Pi files. Map discovered skills to { name, description, source, filePath, diagnostics }, using project-relative or redacted display paths.

Step 4: Implement canonical inference and example Skills

Build an index from canonical SKILL.md paths. On a read request exact match, append skill.requested; on the paired successful result append skill.loaded; failed reads do not append loaded. ads-analysis explicitly requires mock_ads_metrics first and disclaims real data. report-writer requires the specified Chinese sections and final save_report_draft.

Step 5: Verify

Run: pnpm --filter @agent-studio/harness test -- skills

Expected: PASS for both project locations, delimiter-based extra directories, duplicate canonical roots, invalid diagnostics and request/load inference.

Step 6: Commit

git add packages/harness .agents .pi
git commit -m "feat(harness): discover skills progressively"

Task 6: Abort-aware ApprovalBroker

Files:

Create: packages/harness/src/approvals/approval-broker.ts
Create: packages/harness/src/approvals/in-memory-approval-broker.ts
Create: packages/harness/src/security/redact.ts
Create: packages/harness/tests/approval-broker.test.ts
Create: packages/harness/tests/redact.test.ts
Modify: packages/harness/src/index.ts

Interfaces:

Consumes: EventStore and ApprovalRequest shared types.
Produces: ApprovalBroker.request(), approve(), deny(), cancelForSession(), cancelForRun(), and redactAndLimit(value, maxBytes).
Step 1: Write failing broker tests

it("resolves an approved request and is idempotent", async () => {
  const waiting = broker.request(input, controller.signal);
  const first = broker.approve(input.approvalId);
  const second = broker.approve(input.approvalId);
  await expect(waiting).resolves.toMatchObject({ status: "approved" });
  expect(second).toEqual(first);
});

it("rejects a conflicting second decision", () => {
  broker.approve("ap_1");
  expect(() => broker.deny("ap_1")).toThrowError(expect.objectContaining({ code: "APPROVAL_ALREADY_RESOLVED" }));
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/harness test -- approval redact

Expected: FAIL because broker and redaction are missing.

Step 3: Implement redaction and broker lifecycle

Recursively redact case-insensitive keys matching apiKey, authorization, token, secret, password and credential. Limit serialized arguments by UTF-8 bytes and replace overflow with a typed truncation summary. Broker state holds one timer and abort listener per pending request; every resolution clears both. Timeout resolves as expired/denied semantics, abort resolves cancelled, and cancellation APIs cover all matching pending approvals. Append required/resolved exactly once.

Step 4: Verify

Run: pnpm --filter @agent-studio/harness test -- approval redact

Expected: PASS for approve, deny, timeout with fake timers, abort, Session cancellation, idempotency, conflict and sensitive nested fields.

Step 5: Commit

git add packages/harness
git commit -m "feat(harness): add server-authoritative approvals"

Task 7: Session-scoped model connection and SSRF guard

Files:

Create: packages/harness/src/models/model-connection-store.ts
Create: packages/harness/src/models/in-memory-model-connection-store.ts
Create: packages/harness/src/models/base-url-policy.ts
Create: packages/harness/src/models/pi-model-runtime-factory.ts
Create: packages/harness/tests/model-connection.test.ts
Create: packages/harness/tests/base-url-policy.test.ts
Modify: packages/harness/src/index.ts

Interfaces:

Consumes: shared model schema; installed Pi ModelRuntime/credential APIs.
Produces: configure, getRedacted, clearCredentials, resolveConnection, createRuntime, and validateBaseUrl(url, options, resolver).
Step 1: Write failing model and SSRF tests

it("never returns the raw key", async () => {
  await store.configure("ses_1", validInput({ apiKey: "sk-test-secret" }));
  const json = JSON.stringify(store.getRedacted("ses_1"));
  expect(json).not.toContain("sk-test-secret");
  expect(store.getRedacted("ses_1")?.keyHint).toBe("••••cret");
});

it.each(["http://169.254.169.254/latest/meta-data", "https://127.0.0.1/v1", "file:///etc/passwd"])(
  "rejects restricted base URL %s", async (url) => expect(validate(url)).rejects.toMatchObject({ code: "BASE_URL_NOT_ALLOWED" }),
);

Step 2: Verify red state

Run: pnpm --filter @agent-studio/harness test -- model-connection base-url

Expected: FAIL because model storage and URL policy are absent.

Step 3: Implement URL policy

Use WHATWG URL. Reject userinfo, fragment, malformed ports, non-HTTP protocols and sensitive query names. Permit HTTPS by default; permit HTTP only for an explicit localhost development option. Resolve all A/AAAA results with an injected resolver and reject unspecified, loopback, private, carrier-grade NAT, link-local, multicast and metadata ranges unless the private-network flag explicitly permits non-metadata private ranges. Return a normalized URL and validated addresses for the connection layer; disallow cross-host redirects.

Step 4: Implement in-memory connection lifecycle and Pi runtime factory

Store raw keys in a private map separated from redacted metadata. Session input overrides environment config; absent config throws MODEL_NOT_CONFIGURED. Clearing credentials overwrites/removes the old entry and is idempotent. Build the installed Pi runtime with an in-memory credential store or public equivalent. If metadata requires a Session-local models.json, write no secrets and delete it with the Session. Use AbortSignal timeouts for runtime/catalog creation and minimum connection tests.

Step 5: Verify

Run: pnpm --filter @agent-studio/harness test -- model-connection base-url

Expected: PASS for priority, missing config, no fallback, key redaction/rotation/clear, provider/API mismatch, localhost development exception and all restricted address classes.

Step 6: Commit

git add packages/harness
git commit -m "feat(harness): secure session model configuration"

Task 8: Secure image AttachmentStore

Files:

Create: packages/harness/src/attachments/attachment-store.ts
Create: packages/harness/src/attachments/file-attachment-store.ts
Create: packages/harness/src/attachments/image-signatures.ts
Create: packages/harness/tests/attachments.test.ts
Modify: packages/harness/src/index.ts

Interfaces:

Consumes: SafeWorkspace, sharp, shared ImageAttachment schema, and Node streams.
Produces: add(sessionId, upload), get, openNormalized, remove, removeAll, and toPiImages(sessionId, attachmentIds).
Step 1: Write failing attachment tests

it("rejects declared PNG with JPEG bytes", async () => {
  await expect(store.add("ses_1", upload({ name: "fake.png", mime: "image/png", bytes: jpegBytes })))
    .rejects.toMatchObject({ code: "IMAGE_SIGNATURE_MISMATCH" });
});

it("does not expose cross-session attachment metadata", async () => {
  const item = await store.add("ses_a", validPngUpload);
  expect(store.get("ses_b", item.attachmentId)).toBeUndefined();
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/harness test -- attachments

Expected: FAIL because AttachmentStore is missing.

Step 3: Implement bounded streaming and normalization

Stream to a random exclusive temp file while enforcing byte limit. Detect JPEG/PNG/WebP magic bytes and compare extension and declared MIME. Use sharp with fail-on-error, pixel/dimension limits, auto-rotation and metadata stripping, then encode to a supported canonical format and atomically move to the Session attachment directory. Never use the user filename in a path. On failure remove temp files and append only safe rejection metadata.

Step 4: Implement ownership, deletion and Pi mapping

Index by Session and attachmentId. toPiImages validates ready state and count, opens normalized bytes only at prompt time, and returns the exact installed Pi ImageContent shape. Removal is idempotent within the owning Session; cross-Session lookup behaves as not found. removeAll clears files and index.

Step 5: Verify

Run: pnpm --filter @agent-studio/harness test -- attachments

Expected: PASS for JPEG/PNG/WebP, forged MIME, SVG, byte/pixel/dimension limits, count, cross-Session access, deletion and no base64 in events.

Step 6: Commit

git add packages/harness
git commit -m "feat(harness): validate image attachments"

Task 9: Pi adapter, Session lifecycle, and event mapping

Files:

Create: packages/harness/src/agent/agent-session-port.ts
Create: packages/harness/src/agent/pi-agent-session-adapter.ts
Create: packages/harness/src/agent/pi-event-adapter.ts
Create: packages/harness/src/sessions/harness-session.ts
Create: packages/harness/src/sessions/session-registry.ts
Create: packages/harness/src/sessions/in-memory-session-registry.ts
Create: packages/harness/tests/pi-event-adapter.test.ts
Create: packages/harness/tests/session-registry.test.ts
Modify: packages/harness/src/index.ts

Interfaces:

Consumes: Tasks 3–8 and installed Pi public types.
Produces: AgentSessionPort, PiAgentSessionAdapter, PiEventAdapter, HarnessSession.startRun(), cancelRun(reason), dispose(), and SessionRegistry.
Step 1: Write failing lifecycle tests with a fake Port

it("returns a run id before the fake prompt completes and rejects overlap", async () => {
  const first = await session.startRun({ text: "first", attachmentIds: [] });
  expect(first.runId).toMatch(/^run_/);
  await expect(session.startRun({ text: "second", attachmentIds: [] }))
    .rejects.toMatchObject({ code: "SESSION_BUSY" });
});

it("distinguishes user cancellation from timeout", async () => {
  await session.startRun(message);
  await session.cancelRun("user");
  expect(events.at(-1)?.type).toBe("run.cancelled");
  expect(events.at(-1)?.payload.reason).toBe("user");
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/harness test -- session-registry pi-event-adapter

Expected: FAIL because lifecycle components are missing.

Step 3: Implement Port and centralized Pi event mapping

Keep SDK-specific types inside the Pi adapter. Map message start/update/end, assistant text deltas, tool start/result/error and turn completion to shared events. Correlate messageId and toolCallId deterministically. Feed read tool facts through SkillEventInference. Ignore unknown events without throwing. Redact and size-limit tool payloads before append.

Step 4: Implement Session and Registry

Create a random Session directory and in-memory Pi SessionManager. Lazily create or recreate Pi AgentSession from the current explicit model configuration before a Run. startRun validates image support, obtains a lock, appends accepted events, launches prompt without awaiting it, installs a timeout and returns runId. The background completion path clears lock/timer and appends exactly one terminal run event. dispose follows the design ordering and validates the deletion root before recursive removal.

Step 5: Wire tool preflight

Register only the explicit allowlist. On Pi tool_call, apply policy: deny unknown; allow safe tools after protected-path checks; for save_report_draft, await ApprovalBroker with ctx.signal and block on denied/expired/cancelled. Ensure the save tool cannot execute before approval.

Step 6: Verify

Run: pnpm --filter @agent-studio/harness test -- session-registry pi-event-adapter

Expected: PASS for 202-compatible launch semantics, busy, user cancel, timeout, approval cancellation, unsubscribe/dispose, no-model failure, image capability and event mapping.

Step 7: Commit

git add packages/harness
git commit -m "feat(harness): manage Pi agent sessions"

Task 10: Fastify API, validation, and safe errors

Files:

Create: apps/server/src/app.ts
Create: apps/server/src/config.ts
Create: apps/server/src/errors/error-handler.ts
Create: apps/server/src/routes/health.ts
Create: apps/server/src/routes/skills.ts
Create: apps/server/src/routes/sessions.ts
Create: apps/server/src/routes/models.ts
Create: apps/server/src/routes/attachments.ts
Create: apps/server/src/routes/approvals.ts
Create: apps/server/src/index.ts
Create: apps/server/tests/api.test.ts
Create: apps/server/tests/model-api.test.ts
Create: apps/server/tests/attachment-api.test.ts

Interfaces:

Consumes: SessionRegistry, SkillCatalog, ApprovalBroker, AttachmentStore and shared schemas.
Produces: every HTTP route in the spec and buildApp(dependencies) for injection testing.
Step 1: Write failing injected HTTP tests

it("accepts a run without waiting for completion", async () => {
  const response = await app.inject({ method: "POST", url: `/api/sessions/${sessionId}/messages`, payload: { text: "hello", attachmentIds: [] } });
  expect(response.statusCode).toBe(202);
  expect(response.json()).toMatchObject({ runId: expect.stringMatching(/^run_/) });
});

it("never returns model credentials", async () => {
  await configureWithKey("sk-super-secret");
  const response = await app.inject({ method: "GET", url: `/api/sessions/${sessionId}/model-config` });
  expect(response.body).not.toContain("sk-super-secret");
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/server test -- api model-api attachment-api

Expected: FAIL because the app and routes are missing.

Step 3: Implement app composition and error handling

Build Fastify with request IDs and allowlisted logging/redaction. Parse environment settings through Zod. Convert domain errors to 400/404/409/413/422/500 with the unified body and requestId on every response. Production errors omit stack, filesystem paths and raw causes.

Step 4: Implement all non-SSE routes

Use strict schemas at route boundaries. Messages return 202 after startRun accepts. Model test uses a bounded AbortSignal and records no response body; if the installed provider cannot safely probe, return the explicit deferred-validation result. Multipart accepts exactly one supported image part and streams it to AttachmentStore. Approval routes verify active Session/run ownership through broker state.

Step 5: Verify

Run: pnpm --filter @agent-studio/server test -- api model-api attachment-api

Expected: PASS for health, skills, create/delete, 404, validation, 202, busy, cancel, model redaction/clear/test, SSRF errors, multipart limits and approve/deny.

Step 6: Commit

git add apps/server
git commit -m "feat(server): expose secure harness API"

Task 11: Replayable SSE and shutdown lifecycle

Files:

Create: apps/server/src/sse/sse-connection.ts
Create: apps/server/src/sse/sse-registry.ts
Create: apps/server/src/routes/events.ts
Create: apps/server/tests/sse.test.ts
Create: apps/server/tests/shutdown.test.ts
Modify: apps/server/src/app.ts
Modify: apps/server/src/index.ts

Interfaces:

Consumes: EventStore replay/subscribe and SessionRegistry disposal.
Produces: /api/sessions/:sessionId/events, SseRegistry.closeForSession(), and SseRegistry.closeAll().
Step 1: Write failing SSE tests

it("replays after Last-Event-ID and prefers it over afterSequence", async () => {
  const stream = await connect({ headers: { "last-event-id": second.eventId }, query: { afterSequence: "0" } });
  expect(await stream.nextEvent()).toMatchObject({ id: third.eventId, event: third.type });
});

it("clears subscriber and heartbeat when the client closes", async () => {
  const stream = await connect();
  stream.close();
  expect(eventStore.listenerCount(sessionId)).toBe(0);
  expect(clock.getTimerCount()).toBe(0);
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/server test -- sse shutdown

Expected: FAIL because SSE support is missing.

Step 3: Implement framing, replay and backpressure

Set text/event-stream, no-cache, keep-alive and X-Accel-Buffering: no. Frame eventId/type/JSON data safely. Parse Last-Event-ID first, then afterSequence. Replay retained events before subscribing without a race by buffering new events during replay. Emit a named stream.reset control event on gaps. Send : heartbeat\n\n every 15 seconds. Track queued bytes/events; close clients exceeding the configured limit.

Step 4: Implement cleanup and shutdown

Request close clears heartbeat, listener and queued writes. Session deletion closes only that Session’s streams without cancelling unrelated runs. Server close stops accepting traffic, closes all SSE clients and awaits SessionRegistry disposal before process exit.

Step 5: Verify

Run: pnpm --filter @agent-studio/server test -- sse shutdown

Expected: PASS for initial stream, heartbeat with fake timers, both cursor forms, precedence, replay, gap/reset, live event, close cleanup, slow client and shutdown.

Step 6: Commit

git add apps/server
git commit -m "feat(server): stream replayable harness events"

Task 12: Browser HarnessClientStore and assistant-ui runtime projection

Files:

Create: apps/web/src/api/harness-api.ts
Create: apps/web/src/store/harness-client-store.ts
Create: apps/web/src/store/reducer.ts
Create: apps/web/src/assistant/use-harness-runtime.tsx
Create: apps/web/src/assistant/message-mapper.ts
Create: apps/web/tests/reducer.test.ts
Create: apps/web/tests/runtime.test.tsx
Modify: apps/web/vite.config.ts

Interfaces:

Consumes: shared event types and current assistant-ui External Store API.
Produces: HarnessClientStore, reduceHarnessEvent, toThreadMessageLike, and useHarnessRuntime(store).
Step 1: Write failing reducer and runtime tests

it("deduplicates sequence and merges assistant deltas", () => {
  const once = reduceHarnessEvent(initialState, delta("hel", 2));
  const twice = reduceHarnessEvent(once, delta("hel", 2));
  const complete = reduceHarnessEvent(twice, delta("lo", 3));
  expect(complete.messages.byId.msg_1.text).toBe("hello");
});

it("maps run failure to incomplete assistant-ui status", () => {
  expect(toThreadMessageLike(failedMessage).status).toMatchObject({ type: "incomplete" });
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/web test -- reducer runtime

Expected: FAIL because the client store and runtime do not exist.

Step 3: Implement API client and deterministic reducer

API methods cover all HTTP endpoints with typed safe errors. The store creates a Session, opens EventSource-compatible SSE, exposes subscribe/getSnapshot, and maintains connection state, skills, messages, runs, tools, approvals and timeline. Reject duplicate/out-of-order sequence, request reset on gaps, and reconnect with the latest accepted sequence. Do not store API keys inside the store snapshot.

Step 4: Implement External Store projection

Use the exact installed useExternalStoreRuntime contract. Map user/assistant text, image metadata and tool-call/result parts to ThreadMessageLike. onNew uploads/uses attachments then calls messages API; cancel calls the server. Running, complete and incomplete/error statuses come only from reducer state. Do not configure assistant-ui transport or a second model request.

Step 5: Verify

Run: pnpm --filter @agent-studio/web test -- reducer runtime

Expected: PASS for delta merge, dedupe, gap/reconnect state, tool correlation, approval state and message status mapping.

Step 6: Commit

git add apps/web
git commit -m "feat(web): project harness events into assistant-ui"

Task 13: Three-panel application shell, Thread, Skills, and Timeline

Files:

Create: apps/web/src/main.tsx
Create: apps/web/src/App.tsx
Create: apps/web/src/components/layout/AppShell.tsx
Create: apps/web/src/components/topbar/TopBar.tsx
Create: apps/web/src/components/skills/SkillsPanel.tsx
Create: apps/web/src/components/chat/Thread.tsx
Create: apps/web/src/components/chat/Composer.tsx
Create: apps/web/src/components/timeline/EventTimeline.tsx
Create: apps/web/src/components/common/StateView.tsx
Create: apps/web/tests/app-shell.test.tsx
Create: apps/web/tests/states.test.tsx

Interfaces:

Consumes: HarnessClientStore and assistant-ui runtime from Task 12.
Produces: accessible three-panel UI and mobile Chat/Skills/Events navigation.
Step 1: Write failing UI structure tests

it("renders the three primary work areas", () => {
  render(<TestApp />);
  expect(screen.getByRole("region", { name: "Skills Catalog" })).toBeVisible();
  expect(screen.getByRole("main", { name: "Streaming Chat" })).toBeVisible();
  expect(screen.getByRole("region", { name: "Event Timeline" })).toBeVisible();
});

it("fills but does not send the demo prompt", async () => {
  await user.click(screen.getByRole("button", { name: "填入演示任务" }));
  expect(screen.getByRole("textbox")).toHaveValue(expect.stringContaining("demo-account"));
  expect(api.sendMessage).not.toHaveBeenCalled();
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/web test -- app-shell states

Expected: FAIL because UI components are missing.

Step 3: Build the accessible app shell and states

Wrap Thread primitives with the custom runtime. Skills list displays metadata/diagnostics and current requested/loaded indicators without bodies. Timeline sorts by sequence, filters categories and collapses older entries. Auto-scroll only when already near the bottom. TopBar exposes Session state, model/vision badges, Settings and reset. Model-missing disables send and points to Settings. Add Loading, Empty, Error, Offline/Reconnecting, Busy and Approval Pending state views.

Step 4: Implement responsive navigation semantics

Desktop shows all three regions. Tablet toggles side panels with accessible buttons. Mobile uses a tablist with panels for Chat, Skills and Events and never mounts duplicate interactive controls. Preserve composer draft and scroll positions while switching.

Step 5: Verify

Run: pnpm --filter @agent-studio/web test -- app-shell states

Expected: PASS for regions, diagnostics, demo fill-only behavior, model-disabled composer, filters, reconnect/busy states and mobile tabs.

Step 6: Commit

git add apps/web
git commit -m "feat(web): add Agent Studio workspace UI"

Task 14: Tool UI, approval interaction, settings, and attachments

Files:

Create: apps/web/src/components/tools/AdsMetricsTool.tsx
Create: apps/web/src/components/tools/SaveReportTool.tsx
Create: apps/web/src/components/tools/ToolFallback.tsx
Create: apps/web/src/components/approvals/ApprovalDialog.tsx
Create: apps/web/src/components/settings/ModelSettings.tsx
Create: apps/web/src/assistant/attachment-adapter.ts
Create: apps/web/src/components/attachments/ImageAttachment.tsx
Create: apps/web/src/components/attachments/ImagePreviewDialog.tsx
Create: apps/web/tests/tools-approval.test.tsx
Create: apps/web/tests/settings.test.tsx
Create: apps/web/tests/attachments.test.tsx
Modify: apps/web/src/App.tsx

Interfaces:

Consumes: assistant-ui Toolkit/render-only Tool UI and AttachmentAdapter signatures verified from installed types.
Produces: registered UIs for exact names mock_ads_metrics and save_report_draft, server-authoritative ApprovalDialog, model settings, and upload/remove adapter.
Step 1: Write failing interaction tests

it("disables both decisions after submitting approval", async () => {
  render(<ApprovalDialog request={pendingApproval} />);
  await user.click(screen.getByRole("button", { name: "批准并保存" }));
  expect(screen.getByRole("button", { name: "批准并保存" })).toBeDisabled();
  expect(screen.getByRole("button", { name: "拒绝" })).toBeDisabled();
});

it("does not persist an API key", async () => {
  await saveModelSettings({ apiKey: "sk-browser-secret" });
  expect(localStorage.setItem).not.toHaveBeenCalled();
  expect(sessionStorage.setItem).not.toHaveBeenCalled();
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/web test -- tools-approval settings attachments

Expected: FAIL because the interaction components are missing.

Step 3: Implement Tool UIs and approval

Register exact server tool names through the installed stable Toolkit API. Ads UI shows account, range and result metrics. Save UI shows redacted report summary, risk and all waiting/resolution states. Unknown tools use fallback without rendering raw oversized values. Approval buttons call APIs once and wait for SSE; countdown is advisory. Modal has dialog semantics, focus trap, Escape behavior and cannot be dismissed by backdrop while pending.

Step 4: Implement model settings

Provide editable OpenAI, Anthropic and compatible presets with no keys. Use a password field with temporary reveal. Save/test/clear call server APIs; after save clear the local input and display only server-redacted metadata. Translate URL policy errors into concise Chinese. Prevent save while a Run is active. Switching to text-only preserves history but blocks new image sends.

Step 5: Implement custom AttachmentAdapter and image UI

Adapter uploads immediately with multipart, maps progress/error/ready state, removes through the server and never converts large files to persistent base64. Use object URLs only for local previews and revoke on removal/unmount/success replacement. Composer supports picker, drop and paste, blocks send during upload, and exposes retry/remove. Preview dialog opens by keyboard and returns focus. Sent attachments render safe metadata plus authenticated Session-scoped preview behavior if implemented.

Step 6: Verify

Run: pnpm --filter @agent-studio/web test -- tools-approval settings attachments

Expected: PASS for all Tool states, approve/deny, focus, duplicate click prevention, key storage prohibition, redacted settings, upload/retry/remove/revoke, vision blocking and keyboard preview.

Step 7: Commit

git add apps/web
git commit -m "feat(web): add tools approvals models and images"

Task 15: Visual system, responsive behavior, and accessibility checks

Files:

Create: apps/web/src/styles/tokens.css
Create: apps/web/src/styles/base.css
Create: apps/web/src/styles/layout.css
Create: apps/web/src/styles/components.css
Create: apps/web/tests/accessibility.test.tsx
Modify: apps/web/src/main.tsx
Modify: UI components from Tasks 13–14 where labels/focus hooks require adjustment.

Interfaces:

Consumes: semantic UI from Tasks 13–14.
Produces: dark operational theme, three responsive breakpoints, reduced-motion support and keyboard-visible focus.
Step 1: Write failing accessibility assertions

it("gives icon-only actions accessible names", () => {
  render(<TestApp />);
  expect(screen.getByRole("button", { name: "停止当前运行" })).toBeVisible();
  expect(screen.getByRole("button", { name: "打开模型设置" })).toBeVisible();
});

it("announces completion but not every streaming token", () => {
  const { rerender } = render(<CompletionAnnouncer status="running" />);
  expect(screen.queryByRole("status")).toBeEmptyDOMElement();
  rerender(<CompletionAnnouncer status="complete" />);
  expect(screen.getByRole("status")).toHaveTextContent("回答已完成");
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/web test -- accessibility

Expected: FAIL until labels and live-region policy are present.

Step 3: Implement tokens and layouts

Define surfaces, borders, text, muted, cyan accent, pink danger, focus, spacing in 8px increments, radii and shadows. Keep body at #111111, use accent colors sparingly, and avoid wrapping every item in a card. Use CSS grid for desktop, collapsible two-column tablet, tabs/sheets mobile, min-width: 0, responsive media and overflow guards. Interactive targets are at least 44px.

Step 4: Implement focus, motion and announcement behavior

Use :focus-visible rings with sufficient contrast, text+icon status labels, tooltips for icon-only actions, and prefers-reduced-motion overrides. Announce only completed assistant responses through a polite live region. Confirm Dialog focus return and mobile tab keyboard navigation.

Step 5: Verify

Run: pnpm --filter @agent-studio/web test -- accessibility && pnpm --filter @agent-studio/web typecheck

Expected: PASS for accessible names, modal roles, tab semantics, live announcement policy and type safety.

Step 6: Commit

git add apps/web
git commit -m "feat(web): polish responsive accessible interface"

Task 16: Full fake-session integration path

Files:

Create: packages/harness/tests/fakes/fake-agent-session.ts
Create: apps/server/tests/full-flow.integration.test.ts
Create: apps/web/tests/full-flow.test.tsx
Modify: test setup files in Harness, Server and Web packages.

Interfaces:

Consumes: complete Harness/Server/Web APIs.
Produces: deterministic full-flow acceptance coverage without model credentials or network.
Step 1: Write the failing end-to-end integration test

it("runs image + metrics + approved report flow and cleans up", async () => {
  const session = await client.createSession();
  await client.configureFakeModel(session.id, { supportsImages: true });
  const image = await client.uploadFixture(session.id, "creative.png");
  const accepted = await client.sendMessage(session.id, { text: demoPrompt, attachmentIds: [image.attachmentId] });
  await fake.emitDemoRun(accepted.runId);
  const approval = await events.waitFor("approval.required");
  await client.approve(approval.payload.approvalId);
  await events.waitFor("run.completed");
  expect(await readSessionReport(session.id)).toContain("账户总结");
  await client.deleteSession(session.id);
  expect(await sessionPathExists(session.id)).toBe(false);
});

Step 2: Verify red state

Run: pnpm --filter @agent-studio/server test -- full-flow.integration

Expected: FAIL until the programmable fake and dependency wiring exist.

Step 3: Implement the programmable fake

The fake stores subscribers, keeps prompt deferred, records structured images, emits assistant deltas, read Skill calls/results, mock tool calls/results, a save tool request that waits on the real ApprovalBroker, and terminal events. Its abort and dispose expose assertions and release all pending operations.

Step 4: Complete server and browser flow tests

Assert exact causal ordering for session/run/user/skill/tool/approval/assistant/run completion, toolCallId and approvalId correlation, report contents, image metadata without base64, and deletion cleanup. Add Deny and fake-timer Timeout variants that assert no report exists while assistant content may still complete. Web flow asserts reducer/UI states through the same event sequence.

Step 5: Verify

Run: pnpm test

Expected: all package tests pass without environment API keys and without network access.

Step 6: Commit

git add packages/harness apps/server apps/web
git commit -m "test: cover complete Agent Studio flow"

Task 17: Teaching documentation and production boundary

Files:

Modify: README.md
Create: docs/architecture.md
Create: docs/event-mapping.md
Create: docs/security-boundaries.md
Create: docs/classroom-demo.md
Create: docs/production-roadmap.md

Interfaces:

Consumes: actual dependency versions, commands, routes and observed behavior from Tasks 1–16.
Produces: complete operator/trainer documentation with Mermaid diagrams and no unverified claims.
Step 1: Write README and five guides

Document purpose, actual versions/API differences, architecture and Agent Loop diagrams, boundaries, directory tree, setup, environment and Settings model configuration, credential lifetime, image chain, exact demo prompt, Approve/Deny behavior, progressive disclosure, inferred Skill events, in-memory/single-process limits, test commands and production roadmap. Security guide explicitly explains Node process inheritance and the need for Docker/VM/micro-VM/remote sandbox for untrusted work.

Step 2: Verify documentation against actual behavior

Run every documented setup, test and build command. Compare documented versions with pnpm list -r --depth 0, routes with the Fastify route registry, and environment names with apps/server/src/config.ts. Render Mermaid blocks in a compatible preview and inspect every internal file link.

Expected: all commands succeed and no documented version, route, environment name, link or security claim disagrees with the implementation.

Step 3: Commit

git add README.md docs
git commit -m "docs: add Agent Studio teaching guide"

Task 18: Final automated and manual verification

Files:

Modify only files implicated by observed verification failures.
Create: docs/verification-results.md

Interfaces:

Consumes: the finished workspace.
Produces: verified build plus an exact record of commands, test count, local URLs and remaining limitations.
Step 1: Run the complete automated gate

Run each command separately and preserve its real result:

pnpm lint
pnpm typecheck
pnpm test
pnpm build

Expected: all four exit 0. Record package/test counts from output rather than estimating.

Step 2: Start the application

Run: pnpm dev

Expected: Server and Vite Web start from the root command, health returns success, Web loads without console errors, and missing model state is explicit.

Step 3: Manually exercise desktop and mobile critical paths

At desktop width verify Skills diagnostics, Demo Prompt fill-only behavior, fake/demo stream, Tool cards, Approval approve and deny, Timeline filtering/replay and Settings redaction. At a phone viewport verify Chat/Skills/Events tabs, no horizontal overflow, attachment picker/preview, keyboard focus and reduced-motion behavior. Use the browser’s visible network/storage tools to confirm the API key is absent from URLs, responses, SSE, localStorage, sessionStorage and IndexedDB.

Step 4: Fix only observed defects and repeat affected gates

For every defect, first add a regression test that demonstrates it, run that test red, implement the smallest correction, run it green, then rerun all four root gates. Do not skip or weaken security assertions.

Step 5: Write verification record

docs/verification-results.md must include date, Node/pnpm versions, exact commands and exit results, total tests, browser widths checked, demo paths exercised, credential inspection result and concrete known limitations.

Step 6: Commit

git add .
git commit -m "chore: verify Agent Studio Mini"

49 KiB Raw Blame History Unescape Escape

Agent Studio Mini Implementation Plan

Global Constraints

Task 1: Workspace foundation and verified dependency baseline

Task 2: Shared schemas, typed events, and API errors

Task 3: Atomic bounded EventStore

Task 4: SafeWorkspace and deterministic custom tools

Task 5: Standard Skill discovery and inferred Skill events

Task 6: Abort-aware ApprovalBroker

Task 7: Session-scoped model connection and SSRF guard

Task 8: Secure image AttachmentStore

Task 9: Pi adapter, Session lifecycle, and event mapping

Task 10: Fastify API, validation, and safe errors

Task 11: Replayable SSE and shutdown lifecycle

Task 12: Browser HarnessClientStore and assistant-ui runtime projection

Task 13: Three-panel application shell, Thread, Skills, and Timeline

Task 14: Tool UI, approval interaction, settings, and attachments

Task 15: Visual system, responsive behavior, and accessibility checks

Task 16: Full fake-session integration path

Task 17: Teaching documentation and production boundary

Task 18: Final automated and manual verification

49 KiB

Raw Blame History