1120 lines
50 KiB
Markdown
1120 lines
50 KiB
Markdown
# Agent Studio Mini Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Build a runnable training-grade Pi Coding Agent harness with assistant-ui, server-authoritative events and approvals, session-scoped model credentials, and secure image input.
|
||
|
||
**Architecture:** A pnpm workspace separates portable protocols (`packages/shared`), the stateful agent harness (`packages/harness`), Fastify transport (`apps/server`), and the assistant-ui presentation layer (`apps/web`). Pi AgentSession and the bounded EventStore own runtime truth; the browser consumes HTTP/SSE through one external store and never calls a model or executes tools directly.
|
||
|
||
**Tech Stack:** TypeScript strict mode, Node.js 20+, pnpm workspace, React, Vite, assistant-ui, Fastify, `@earendil-works/pi-coding-agent`, TypeBox, Zod, SSE, sharp, Vitest, React Testing Library, ESLint.
|
||
|
||
**Spec:** `docs/superpowers/specs/2026-09-07-agent-studio-mini-design.md`
|
||
|
||
## Global Constraints
|
||
|
||
- Use Node.js 20+ and TypeScript strict mode.
|
||
- Install current stable dependencies, then treat installed public exports and type definitions as authoritative.
|
||
- `packages/shared` must not import Node-only modules; `packages/harness` must not import React or assistant-ui.
|
||
- Pi AgentSession and Harness EventStore are runtime truth; assistant-ui is a projection layer.
|
||
- API keys remain only in session-scoped server memory and never enter URLs, logs, SSE, browser storage, or snapshots.
|
||
- Only `read`, `grep`, `find`, `ls`, `mock_ads_metrics`, and `save_report_draft` are registered; unknown tools are denied.
|
||
- Default tests use a fake AgentSession and never call a real model or external network.
|
||
- The server must not claim that Project Trust is an OS sandbox.
|
||
- Every task follows red-green-refactor, runs its focused tests, and creates one reviewable commit.
|
||
|
||
---
|
||
|
||
### Task 1: Workspace foundation and verified dependency baseline
|
||
|
||
**Files:**
|
||
- Create: `package.json`
|
||
- Create: `pnpm-workspace.yaml`
|
||
- Create: `tsconfig.base.json`
|
||
- Create: `eslint.config.js`
|
||
- Create: `.gitignore`
|
||
- Create: `.env.example`
|
||
- Create: `README.md`
|
||
- Create: `packages/shared/package.json`
|
||
- Create: `packages/shared/tsconfig.json`
|
||
- Create: `packages/shared/src/index.ts`
|
||
- Create: `packages/shared/tests/smoke.test.ts`
|
||
- Create: `packages/harness/package.json`
|
||
- Create: `packages/harness/tsconfig.json`
|
||
- Create: `apps/server/package.json`
|
||
- Create: `apps/server/tsconfig.json`
|
||
- Create: `apps/web/package.json`
|
||
- Create: `apps/web/tsconfig.json`
|
||
- Create: `apps/web/vite.config.ts`
|
||
- Create: `apps/web/index.html`
|
||
|
||
**Interfaces:**
|
||
- Consumes: Node.js 20+ and pnpm 11 available in the development environment.
|
||
- Produces: root commands `dev`, `lint`, `typecheck`, `test`, and `build`; workspace package names `@agent-studio/shared` and `@agent-studio/harness`.
|
||
|
||
- [ ] **Step 1: Add the workspace smoke test**
|
||
|
||
```ts
|
||
// packages/shared/tests/smoke.test.ts
|
||
import { describe, expect, it } from "vitest";
|
||
|
||
describe("workspace", () => {
|
||
it("runs TypeScript tests", () => expect(true).toBe(true));
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Create strict workspace configuration and package scripts**
|
||
|
||
Root scripts must delegate with `pnpm -r`, keep `dev` parallel, and make `test` use Vitest in run mode. `tsconfig.base.json` must enable `strict`, `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`, `useUnknownInCatchVariables`, `verbatimModuleSyntax`, and `noEmit` for checking. `.env.example` must contain only the empty/default values from the design spec.
|
||
|
||
- [ ] **Step 3: Install current stable dependencies**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
pnpm add -w -D typescript eslint @eslint/js typescript-eslint vitest @vitest/coverage-v8
|
||
pnpm --filter @agent-studio/shared add zod
|
||
pnpm --filter @agent-studio/harness add @agent-studio/shared @earendil-works/pi-coding-agent @sinclair/typebox sharp
|
||
pnpm --filter @agent-studio/server add @agent-studio/shared @agent-studio/harness fastify @fastify/multipart zod
|
||
pnpm --filter @agent-studio/web add @agent-studio/shared react react-dom @assistant-ui/react lucide-react
|
||
pnpm --filter @agent-studio/web add -D vite @vitejs/plugin-react jsdom @testing-library/react @testing-library/user-event @testing-library/jest-dom
|
||
```
|
||
|
||
- [ ] **Step 4: Record and inspect the installed public API**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
pnpm list -r --depth 0
|
||
rg "createAgentSession|DefaultResourceLoader|ModelRuntime|InMemoryCredentialStore|defineTool|isToolCallEventType" node_modules/.pnpm/@earendil-works+pi-coding-agent*/node_modules/@earendil-works/pi-coding-agent -g '*.d.ts'
|
||
rg "useExternalStoreRuntime|AttachmentAdapter|ThreadPrimitive|ComposerPrimitive" node_modules/.pnpm/@assistant-ui+react*/node_modules/@assistant-ui/react -g '*.d.ts'
|
||
```
|
||
|
||
Write exact dependency versions and any material signature differences into a temporary commit note in `README.md`; Task 16 expands it into the final version/API section.
|
||
|
||
- [ ] **Step 5: Verify the baseline**
|
||
|
||
Run: `pnpm test && pnpm typecheck && pnpm lint`
|
||
|
||
Expected: the smoke test passes and all packages typecheck/lint without ignored errors.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add package.json pnpm-lock.yaml pnpm-workspace.yaml tsconfig.base.json eslint.config.js .gitignore .env.example packages apps README.md
|
||
git commit -m "chore: scaffold Agent Studio workspace"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: Shared schemas, typed events, and API errors
|
||
|
||
**Files:**
|
||
- Create: `packages/shared/src/events.ts`
|
||
- Create: `packages/shared/src/model-config.ts`
|
||
- Create: `packages/shared/src/messages.ts`
|
||
- Create: `packages/shared/src/errors.ts`
|
||
- Modify: `packages/shared/src/index.ts`
|
||
- Create: `packages/shared/tests/events.test.ts`
|
||
- Create: `packages/shared/tests/model-config.test.ts`
|
||
- Create: `packages/shared/tests/messages.test.ts`
|
||
- Create: `packages/shared/tests/errors.test.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: Zod.
|
||
- Produces: `HarnessEventSchema`, `HarnessEvent`, `EventInput<T>`, `ModelConnectionInputSchema`, `MessageInputSchema`, `ImageAttachmentSchema`, `ApiErrorSchema`, and `createApiError()`.
|
||
|
||
- [ ] **Step 1: Write failing schema tests**
|
||
|
||
```ts
|
||
it("rejects an event payload for the wrong discriminant", () => {
|
||
expect(() => HarnessEventSchema.parse({
|
||
eventId: "evt_1", sessionId: "ses_1", timestamp: new Date().toISOString(),
|
||
sequence: 1, type: "assistant.delta", payload: { approvalId: "ap_1" },
|
||
})).toThrow();
|
||
});
|
||
|
||
it("requires text or at least one attachment", () => {
|
||
expect(MessageInputSchema.safeParse({ text: " ", attachmentIds: [] }).success).toBe(false);
|
||
});
|
||
|
||
it("strips no unknown model fields because the schema is strict", () => {
|
||
expect(ModelConnectionInputSchema.safeParse({
|
||
providerId: "openai", api: "openai-responses", modelId: "gpt-x",
|
||
supportsImages: true, unexpected: true,
|
||
}).success).toBe(false);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run focused tests to verify failure**
|
||
|
||
Run: `pnpm --filter @agent-studio/shared test`
|
||
|
||
Expected: FAIL because the schema modules do not exist.
|
||
|
||
- [ ] **Step 3: Implement all protocol schemas**
|
||
|
||
Define a strict base schema with positive integer sequence and ISO timestamp. Add every event discriminant required by the spec with typed payloads. `assistant.delta` contains `messageId` and `delta`; Tool events contain `toolCallId`, `toolName`, redacted arguments/results and status; approval events contain `approvalId`; attachment events contain metadata only. Export inferred TypeScript types. `createApiError(code, message, requestId, details?)` must return an object accepted by `ApiErrorSchema`.
|
||
|
||
- [ ] **Step 4: Run shared verification**
|
||
|
||
Run: `pnpm --filter @agent-studio/shared test && pnpm --filter @agent-studio/shared typecheck`
|
||
|
||
Expected: PASS, including invalid discriminants, strict model input, message minimum content, attachment ID validation, and error shape.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add packages/shared
|
||
git commit -m "feat(shared): define harness protocols"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Atomic bounded EventStore
|
||
|
||
**Files:**
|
||
- Create: `packages/harness/src/events/event-store.ts`
|
||
- Create: `packages/harness/src/events/in-memory-event-store.ts`
|
||
- Create: `packages/harness/tests/event-store.test.ts`
|
||
- Modify: `packages/harness/src/index.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `HarnessEvent` and `EventInput<T>` from shared.
|
||
- Produces: `EventCursor`, `EventReplay`, `EventStore`, and `InMemoryEventStore(maxEventsPerSession = 1000)`.
|
||
|
||
- [ ] **Step 1: Write failing EventStore tests**
|
||
|
||
```ts
|
||
it("allocates monotonic per-session sequence and bounds history", () => {
|
||
const store = new InMemoryEventStore(2);
|
||
const first = store.append("ses_a", sessionStartedInput);
|
||
const second = store.append("ses_a", runStartedInput);
|
||
store.append("ses_a", runCompletedInput);
|
||
expect([first.sequence, second.sequence]).toEqual([1, 2]);
|
||
expect(store.listAfter("ses_a").events.map((e) => e.sequence)).toEqual([2, 3]);
|
||
});
|
||
|
||
it("reports a gap for a cursor older than retained history", () => {
|
||
expect(store.listAfter("ses_a", { sequence: 0 }).gap).toBe(true);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- event-store`
|
||
|
||
Expected: FAIL because the store is missing.
|
||
|
||
- [ ] **Step 3: Implement the store**
|
||
|
||
Keep one private state object per Session with `nextSequence`, bounded events, listeners, and eventId index. `append` must create eventId/timestamp, increment and append synchronously before notifying a snapshot of listeners. `listAfter` accepts eventId or sequence, gives eventId precedence, and returns `{ events, gap, oldestSequence, latestSequence }`. `subscribe` returns an idempotent unsubscribe.
|
||
|
||
- [ ] **Step 4: Verify pass and types**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- event-store && pnpm --filter @agent-studio/harness typecheck`
|
||
|
||
Expected: PASS for sequence isolation, bounded retention, cursor precedence, gap behavior and unsubscribe.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add packages/harness
|
||
git commit -m "feat(harness): add bounded event store"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: SafeWorkspace and deterministic custom tools
|
||
|
||
**Files:**
|
||
- Create: `packages/harness/src/workspace/safe-workspace.ts`
|
||
- Create: `packages/harness/src/tools/mock-ads-metrics.ts`
|
||
- Create: `packages/harness/src/tools/save-report-draft.ts`
|
||
- Create: `packages/harness/src/tools/tool-policy.ts`
|
||
- Create: `packages/harness/tests/safe-workspace.test.ts`
|
||
- Create: `packages/harness/tests/tools.test.ts`
|
||
- Create: `fixtures/ads-account.json`
|
||
- Modify: `packages/harness/src/index.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: current Pi `defineTool` and TypeBox signatures confirmed in Task 1.
|
||
- Produces: `SafeWorkspace`, `createMockAdsMetricsTool(fixturePath)`, `createSaveReportDraftTool(workspace)`, and `decideToolPolicy(toolName): "allow" | "ask" | "deny"`.
|
||
|
||
- [ ] **Step 1: Write failing security and tool tests**
|
||
|
||
```ts
|
||
it.each(["../outside", "/tmp/outside"])("rejects escape path %s", async (path) => {
|
||
await expect(workspace.resolveReadable(path)).rejects.toMatchObject({ code: "PATH_OUTSIDE_WORKSPACE" });
|
||
});
|
||
|
||
it("rejects a symlink parent", async () => {
|
||
await fs.symlink(outsideDir, join(root, "link"));
|
||
await expect(workspace.resolveReadable("link/secret.txt")).rejects.toMatchObject({ code: "SYMLINK_NOT_ALLOWED" });
|
||
});
|
||
|
||
it("computes zero-safe ad metrics", async () => {
|
||
const result = await executeMock({ accountId: "demo-account", days: 7 });
|
||
expect(result.structured.roas).toBeCloseTo(result.structured.revenue / result.structured.spend, 2);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- safe-workspace tools`
|
||
|
||
Expected: FAIL because workspace guards and tools are missing.
|
||
|
||
- [ ] **Step 3: Implement SafeWorkspace**
|
||
|
||
Canonicalize the Session root once. For reads, resolve the candidate, reject protected basenames, verify relative containment, lstat every path segment, and reject symlinks. For the fixed report, open a random sibling temp file with exclusive creation, write UTF-8, fsync/close, recheck the parent, then rename to `report.md`. Expose only fixed report writing rather than a general write method.
|
||
|
||
- [ ] **Step 4: Implement both TypeBox tools and policy**
|
||
|
||
The fixture must contain `demo-account` and campaigns with one high-CPA and one low-CTR anomaly. Validate `days` as integer 1–30 and accountId as a strict string enum or runtime lookup. Return Pi-compatible text content plus structured data with guarded division and two-decimal metrics. `save_report_draft` accepts only `{ title?: string; body: string }`. Policy returns ask only for `save_report_draft`, allow for five permitted tools, deny otherwise.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- safe-workspace tools`
|
||
|
||
Expected: PASS for traversal, absolute escape, symlinks, fixed report path, invalid account/day, metric precision and policy.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add packages/harness fixtures
|
||
git commit -m "feat(harness): secure workspace and custom tools"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Standard Skill discovery and inferred Skill events
|
||
|
||
**Files:**
|
||
- Create: `packages/harness/src/skills/skill-catalog.ts`
|
||
- Create: `packages/harness/src/skills/skill-event-inference.ts`
|
||
- Create: `packages/harness/tests/skills.test.ts`
|
||
- Create: `.agents/skills/ads-analysis/SKILL.md`
|
||
- Create: `.pi/skills/report-writer/SKILL.md`
|
||
- Modify: `packages/harness/src/index.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: installed `DefaultResourceLoader`, settings extension point, `EventStore`, and canonical paths.
|
||
- Produces: `SkillCatalog.reload()`, `SkillCatalog.list()`, `SkillIndex.matchCanonicalPath()`, and `SkillEventInference.observeToolEvent()`.
|
||
|
||
- [ ] **Step 1: Write failing discovery tests**
|
||
|
||
```ts
|
||
it("discovers project and additional skill roots without reading bodies into catalog", async () => {
|
||
const catalog = await createFixtureCatalog({ extraDirs: [extraRoot] });
|
||
expect(catalog.list().map((s) => s.name)).toEqual(expect.arrayContaining(["ads-analysis", "report-writer", "extra-skill"]));
|
||
expect(JSON.stringify(catalog.list())).not.toContain("full secret skill body");
|
||
});
|
||
|
||
it("emits loaded only after a successful read of the canonical SKILL.md", () => {
|
||
inference.onRequested(readCall);
|
||
inference.onFailed(readFailure);
|
||
expect(events.map((event) => event.type)).toEqual(["skill.requested"]);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- skills`
|
||
|
||
Expected: FAIL because catalog and inference are absent.
|
||
|
||
- [ ] **Step 3: Implement catalog loading**
|
||
|
||
Parse `AGENT_SKILLS_DIRS` with `path.delimiter`; resolve real paths, verify readable directories, deduplicate and preserve diagnostics for invalid roots. Configure the Pi loader through its current public settings/resource API without touching global Pi files. Map discovered skills to `{ name, description, source, filePath, diagnostics }`, using project-relative or redacted display paths.
|
||
|
||
- [ ] **Step 4: Implement canonical inference and example Skills**
|
||
|
||
Build an index from canonical `SKILL.md` paths. On a read request exact match, append `skill.requested`; on the paired successful result append `skill.loaded`; failed reads do not append loaded. `ads-analysis` explicitly requires `mock_ads_metrics` first and disclaims real data. `report-writer` requires the specified Chinese sections and final `save_report_draft`.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- skills`
|
||
|
||
Expected: PASS for both project locations, delimiter-based extra directories, duplicate canonical roots, invalid diagnostics and request/load inference.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add packages/harness .agents .pi
|
||
git commit -m "feat(harness): discover skills progressively"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Abort-aware ApprovalBroker
|
||
|
||
**Files:**
|
||
- Create: `packages/harness/src/approvals/approval-broker.ts`
|
||
- Create: `packages/harness/src/approvals/in-memory-approval-broker.ts`
|
||
- Create: `packages/harness/src/security/redact.ts`
|
||
- Create: `packages/harness/tests/approval-broker.test.ts`
|
||
- Create: `packages/harness/tests/redact.test.ts`
|
||
- Modify: `packages/harness/src/index.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: EventStore and `ApprovalRequest` shared types.
|
||
- Produces: `ApprovalBroker.request()`, `approve()`, `deny()`, `cancelForSession()`, `cancelForRun()`, and `redactAndLimit(value, maxBytes)`.
|
||
|
||
- [ ] **Step 1: Write failing broker tests**
|
||
|
||
```ts
|
||
it("resolves an approved request and is idempotent", async () => {
|
||
const waiting = broker.request(input, controller.signal);
|
||
const first = broker.approve(input.approvalId);
|
||
const second = broker.approve(input.approvalId);
|
||
await expect(waiting).resolves.toMatchObject({ status: "approved" });
|
||
expect(second).toEqual(first);
|
||
});
|
||
|
||
it("rejects a conflicting second decision", () => {
|
||
broker.approve("ap_1");
|
||
expect(() => broker.deny("ap_1")).toThrowError(expect.objectContaining({ code: "APPROVAL_ALREADY_RESOLVED" }));
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- approval redact`
|
||
|
||
Expected: FAIL because broker and redaction are missing.
|
||
|
||
- [ ] **Step 3: Implement redaction and broker lifecycle**
|
||
|
||
Recursively redact case-insensitive keys matching apiKey, authorization, token, secret, password and credential. Limit serialized arguments by UTF-8 bytes and replace overflow with a typed truncation summary. Broker state holds one timer and abort listener per pending request; every resolution clears both. Timeout resolves as expired/denied semantics, abort resolves cancelled, and cancellation APIs cover all matching pending approvals. Append required/resolved exactly once.
|
||
|
||
- [ ] **Step 4: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- approval redact`
|
||
|
||
Expected: PASS for approve, deny, timeout with fake timers, abort, Session cancellation, idempotency, conflict and sensitive nested fields.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add packages/harness
|
||
git commit -m "feat(harness): add server-authoritative approvals"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: Session-scoped model connection and SSRF guard
|
||
|
||
**Files:**
|
||
- Create: `packages/harness/src/models/model-connection-store.ts`
|
||
- Create: `packages/harness/src/models/in-memory-model-connection-store.ts`
|
||
- Create: `packages/harness/src/models/base-url-policy.ts`
|
||
- Create: `packages/harness/src/models/pi-model-runtime-factory.ts`
|
||
- Create: `packages/harness/tests/model-connection.test.ts`
|
||
- Create: `packages/harness/tests/base-url-policy.test.ts`
|
||
- Modify: `packages/harness/src/index.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: shared model schema; installed Pi ModelRuntime/credential APIs.
|
||
- Produces: `configure`, `getRedacted`, `clearCredentials`, `resolveConnection`, `createRuntime`, and `validateBaseUrl(url, options, resolver)`.
|
||
|
||
- [ ] **Step 1: Write failing model and SSRF tests**
|
||
|
||
```ts
|
||
it("never returns the raw key", async () => {
|
||
await store.configure("ses_1", validInput({ apiKey: "sk-test-secret" }));
|
||
const json = JSON.stringify(store.getRedacted("ses_1"));
|
||
expect(json).not.toContain("sk-test-secret");
|
||
expect(store.getRedacted("ses_1")?.keyHint).toBe("••••cret");
|
||
});
|
||
|
||
it.each(["http://169.254.169.254/latest/meta-data", "https://127.0.0.1/v1", "file:///etc/passwd"])(
|
||
"rejects restricted base URL %s", async (url) => expect(validate(url)).rejects.toMatchObject({ code: "BASE_URL_NOT_ALLOWED" }),
|
||
);
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- model-connection base-url`
|
||
|
||
Expected: FAIL because model storage and URL policy are absent.
|
||
|
||
- [ ] **Step 3: Implement URL policy**
|
||
|
||
Use WHATWG URL. Reject userinfo, fragment, malformed ports, non-HTTP protocols and sensitive query names. Permit HTTPS by default; permit HTTP only for an explicit localhost development option. Resolve all A/AAAA results with an injected resolver and reject unspecified, loopback, private, carrier-grade NAT, link-local, multicast and metadata ranges unless the private-network flag explicitly permits non-metadata private ranges. Return a normalized URL and validated addresses for the connection layer; disallow cross-host redirects.
|
||
|
||
- [ ] **Step 4: Implement in-memory connection lifecycle and Pi runtime factory**
|
||
|
||
Store raw keys in a private map separated from redacted metadata. Session input overrides environment config; absent config throws `MODEL_NOT_CONFIGURED`. Clearing credentials overwrites/removes the old entry and is idempotent. Build the installed Pi runtime with an in-memory credential store or public equivalent. If metadata requires a Session-local `models.json`, write no secrets and delete it with the Session. Use AbortSignal timeouts for runtime/catalog creation and minimum connection tests.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- model-connection base-url`
|
||
|
||
Expected: PASS for priority, missing config, no fallback, key redaction/rotation/clear, provider/API mismatch, localhost development exception and all restricted address classes.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add packages/harness
|
||
git commit -m "feat(harness): secure session model configuration"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: Secure image AttachmentStore
|
||
|
||
**Files:**
|
||
- Create: `packages/harness/src/attachments/attachment-store.ts`
|
||
- Create: `packages/harness/src/attachments/file-attachment-store.ts`
|
||
- Create: `packages/harness/src/attachments/image-signatures.ts`
|
||
- Create: `packages/harness/tests/attachments.test.ts`
|
||
- Modify: `packages/harness/src/index.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: SafeWorkspace, sharp, shared ImageAttachment schema, and Node streams.
|
||
- Produces: `add(sessionId, upload)`, `get`, `openNormalized`, `remove`, `removeAll`, and `toPiImages(sessionId, attachmentIds)`.
|
||
|
||
- [ ] **Step 1: Write failing attachment tests**
|
||
|
||
```ts
|
||
it("rejects declared PNG with JPEG bytes", async () => {
|
||
await expect(store.add("ses_1", upload({ name: "fake.png", mime: "image/png", bytes: jpegBytes })))
|
||
.rejects.toMatchObject({ code: "IMAGE_SIGNATURE_MISMATCH" });
|
||
});
|
||
|
||
it("does not expose cross-session attachment metadata", async () => {
|
||
const item = await store.add("ses_a", validPngUpload);
|
||
expect(store.get("ses_b", item.attachmentId)).toBeUndefined();
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- attachments`
|
||
|
||
Expected: FAIL because AttachmentStore is missing.
|
||
|
||
- [ ] **Step 3: Implement bounded streaming and normalization**
|
||
|
||
Stream to a random exclusive temp file while enforcing byte limit. Detect JPEG/PNG/WebP magic bytes and compare extension and declared MIME. Use sharp with fail-on-error, pixel/dimension limits, auto-rotation and metadata stripping, then encode to a supported canonical format and atomically move to the Session attachment directory. Never use the user filename in a path. On failure remove temp files and append only safe rejection metadata.
|
||
|
||
- [ ] **Step 4: Implement ownership, deletion and Pi mapping**
|
||
|
||
Index by Session and attachmentId. `toPiImages` validates ready state and count, opens normalized bytes only at prompt time, and returns the exact installed Pi `ImageContent` shape. Removal is idempotent within the owning Session; cross-Session lookup behaves as not found. `removeAll` clears files and index.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- attachments`
|
||
|
||
Expected: PASS for JPEG/PNG/WebP, forged MIME, SVG, byte/pixel/dimension limits, count, cross-Session access, deletion and no base64 in events.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add packages/harness
|
||
git commit -m "feat(harness): validate image attachments"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 9: Pi adapter, Session lifecycle, and event mapping
|
||
|
||
**Files:**
|
||
- Create: `packages/harness/src/agent/agent-session-port.ts`
|
||
- Create: `packages/harness/src/agent/pi-agent-session-adapter.ts`
|
||
- Create: `packages/harness/src/agent/pi-event-adapter.ts`
|
||
- Create: `packages/harness/src/sessions/harness-session.ts`
|
||
- Create: `packages/harness/src/sessions/session-registry.ts`
|
||
- Create: `packages/harness/src/sessions/in-memory-session-registry.ts`
|
||
- Create: `packages/harness/tests/pi-event-adapter.test.ts`
|
||
- Create: `packages/harness/tests/session-registry.test.ts`
|
||
- Modify: `packages/harness/src/index.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: Tasks 3–8 and installed Pi public types.
|
||
- Produces: `AgentSessionPort`, `PiAgentSessionAdapter`, `PiEventAdapter`, `HarnessSession.startRun()`, `cancelRun(reason)`, `dispose()`, and `SessionRegistry`.
|
||
|
||
- [ ] **Step 1: Write failing lifecycle tests with a fake Port**
|
||
|
||
```ts
|
||
it("returns a run id before the fake prompt completes and rejects overlap", async () => {
|
||
const first = await session.startRun({ text: "first", attachmentIds: [] });
|
||
expect(first.runId).toMatch(/^run_/);
|
||
await expect(session.startRun({ text: "second", attachmentIds: [] }))
|
||
.rejects.toMatchObject({ code: "SESSION_BUSY" });
|
||
});
|
||
|
||
it("distinguishes user cancellation from timeout", async () => {
|
||
await session.startRun(message);
|
||
await session.cancelRun("user");
|
||
expect(events.at(-1)?.type).toBe("run.cancelled");
|
||
expect(events.at(-1)?.payload.reason).toBe("user");
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- session-registry pi-event-adapter`
|
||
|
||
Expected: FAIL because lifecycle components are missing.
|
||
|
||
- [ ] **Step 3: Implement Port and centralized Pi event mapping**
|
||
|
||
Keep SDK-specific types inside the Pi adapter. Map message start/update/end, assistant text deltas, tool start/result/error and turn completion to shared events. Correlate messageId and toolCallId deterministically. Feed read tool facts through SkillEventInference. Ignore unknown events without throwing. Redact and size-limit tool payloads before append.
|
||
|
||
- [ ] **Step 4: Implement Session and Registry**
|
||
|
||
Create a random Session directory and in-memory Pi SessionManager. Lazily create or recreate Pi AgentSession from the current explicit model configuration before a Run. `startRun` validates image support, obtains a lock, appends accepted events, launches prompt without awaiting it, installs a timeout and returns runId. The background completion path clears lock/timer and appends exactly one terminal run event. `dispose` follows the design ordering and validates the deletion root before recursive removal.
|
||
|
||
- [ ] **Step 5: Wire tool preflight**
|
||
|
||
Register only the explicit allowlist. On Pi `tool_call`, apply policy: deny unknown; allow safe tools after protected-path checks; for `save_report_draft`, await ApprovalBroker with `ctx.signal` and block on denied/expired/cancelled. Ensure the save tool cannot execute before approval.
|
||
|
||
- [ ] **Step 6: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- session-registry pi-event-adapter`
|
||
|
||
Expected: PASS for 202-compatible launch semantics, busy, user cancel, timeout, approval cancellation, unsubscribe/dispose, no-model failure, image capability and event mapping.
|
||
|
||
- [ ] **Step 7: Commit**
|
||
|
||
```bash
|
||
git add packages/harness
|
||
git commit -m "feat(harness): manage Pi agent sessions"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 10: Fastify API, validation, and safe errors
|
||
|
||
**Files:**
|
||
- Create: `apps/server/src/app.ts`
|
||
- Create: `apps/server/src/config.ts`
|
||
- Create: `apps/server/src/errors/error-handler.ts`
|
||
- Create: `apps/server/src/routes/health.ts`
|
||
- Create: `apps/server/src/routes/skills.ts`
|
||
- Create: `apps/server/src/routes/sessions.ts`
|
||
- Create: `apps/server/src/routes/models.ts`
|
||
- Create: `apps/server/src/routes/attachments.ts`
|
||
- Create: `apps/server/src/routes/approvals.ts`
|
||
- Create: `apps/server/src/index.ts`
|
||
- Create: `apps/server/tests/api.test.ts`
|
||
- Create: `apps/server/tests/model-api.test.ts`
|
||
- Create: `apps/server/tests/attachment-api.test.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: SessionRegistry, SkillCatalog, ApprovalBroker, AttachmentStore and shared schemas.
|
||
- Produces: every HTTP route in the spec and `buildApp(dependencies)` for injection testing.
|
||
|
||
- [ ] **Step 1: Write failing injected HTTP tests**
|
||
|
||
```ts
|
||
it("accepts a run without waiting for completion", async () => {
|
||
const response = await app.inject({ method: "POST", url: `/api/sessions/${sessionId}/messages`, payload: { text: "hello", attachmentIds: [] } });
|
||
expect(response.statusCode).toBe(202);
|
||
expect(response.json()).toMatchObject({ runId: expect.stringMatching(/^run_/) });
|
||
});
|
||
|
||
it("never returns model credentials", async () => {
|
||
await configureWithKey("sk-super-secret");
|
||
const response = await app.inject({ method: "GET", url: `/api/sessions/${sessionId}/model-config` });
|
||
expect(response.body).not.toContain("sk-super-secret");
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/server test -- api model-api attachment-api`
|
||
|
||
Expected: FAIL because the app and routes are missing.
|
||
|
||
- [ ] **Step 3: Implement app composition and error handling**
|
||
|
||
Build Fastify with request IDs and allowlisted logging/redaction. Parse environment settings through Zod. Convert domain errors to 400/404/409/413/422/500 with the unified body and requestId on every response. Production errors omit stack, filesystem paths and raw causes.
|
||
|
||
- [ ] **Step 4: Implement all non-SSE routes**
|
||
|
||
Use strict schemas at route boundaries. Messages return 202 after `startRun` accepts. Model test uses a bounded AbortSignal and records no response body; if the installed provider cannot safely probe, return the explicit deferred-validation result. Multipart accepts exactly one supported image part and streams it to AttachmentStore. Approval routes verify active Session/run ownership through broker state.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/server test -- api model-api attachment-api`
|
||
|
||
Expected: PASS for health, skills, create/delete, 404, validation, 202, busy, cancel, model redaction/clear/test, SSRF errors, multipart limits and approve/deny.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add apps/server
|
||
git commit -m "feat(server): expose secure harness API"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 11: Replayable SSE and shutdown lifecycle
|
||
|
||
**Files:**
|
||
- Create: `apps/server/src/sse/sse-connection.ts`
|
||
- Create: `apps/server/src/sse/sse-registry.ts`
|
||
- Create: `apps/server/src/routes/events.ts`
|
||
- Create: `apps/server/tests/sse.test.ts`
|
||
- Create: `apps/server/tests/shutdown.test.ts`
|
||
- Modify: `apps/server/src/app.ts`
|
||
- Modify: `apps/server/src/index.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: EventStore replay/subscribe and SessionRegistry disposal.
|
||
- Produces: `/api/sessions/:sessionId/events`, `SseRegistry.closeForSession()`, and `SseRegistry.closeAll()`.
|
||
|
||
- [ ] **Step 1: Write failing SSE tests**
|
||
|
||
```ts
|
||
it("replays after Last-Event-ID and prefers it over afterSequence", async () => {
|
||
const stream = await connect({ headers: { "last-event-id": second.eventId }, query: { afterSequence: "0" } });
|
||
expect(await stream.nextEvent()).toMatchObject({ id: third.eventId, event: third.type });
|
||
});
|
||
|
||
it("clears subscriber and heartbeat when the client closes", async () => {
|
||
const stream = await connect();
|
||
stream.close();
|
||
expect(eventStore.listenerCount(sessionId)).toBe(0);
|
||
expect(clock.getTimerCount()).toBe(0);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/server test -- sse shutdown`
|
||
|
||
Expected: FAIL because SSE support is missing.
|
||
|
||
- [ ] **Step 3: Implement framing, replay and backpressure**
|
||
|
||
Set `text/event-stream`, `no-cache`, keep-alive and `X-Accel-Buffering: no`. Frame eventId/type/JSON data safely. Parse Last-Event-ID first, then afterSequence. Replay retained events before subscribing without a race by buffering new events during replay. Emit a named `stream.reset` control event on gaps. Send `: heartbeat\n\n` every 15 seconds. Track queued bytes/events; close clients exceeding the configured limit.
|
||
|
||
- [ ] **Step 4: Implement cleanup and shutdown**
|
||
|
||
Request close clears heartbeat, listener and queued writes. Session deletion closes only that Session’s streams without cancelling unrelated runs. Server close stops accepting traffic, closes all SSE clients and awaits SessionRegistry disposal before process exit.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/server test -- sse shutdown`
|
||
|
||
Expected: PASS for initial stream, heartbeat with fake timers, both cursor forms, precedence, replay, gap/reset, live event, close cleanup, slow client and shutdown.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add apps/server
|
||
git commit -m "feat(server): stream replayable harness events"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 12: Browser HarnessClientStore and assistant-ui runtime projection
|
||
|
||
**Files:**
|
||
- Create: `apps/web/src/api/harness-api.ts`
|
||
- Create: `apps/web/src/store/harness-client-store.ts`
|
||
- Create: `apps/web/src/store/reducer.ts`
|
||
- Create: `apps/web/src/assistant/use-harness-runtime.tsx`
|
||
- Create: `apps/web/src/assistant/message-mapper.ts`
|
||
- Create: `apps/web/tests/reducer.test.ts`
|
||
- Create: `apps/web/tests/runtime.test.tsx`
|
||
- Modify: `apps/web/vite.config.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: shared event types and current assistant-ui External Store API.
|
||
- Produces: `HarnessClientStore`, `reduceHarnessEvent`, `toThreadMessageLike`, and `useHarnessRuntime(store)`.
|
||
|
||
- [ ] **Step 1: Write failing reducer and runtime tests**
|
||
|
||
```ts
|
||
it("deduplicates sequence and merges assistant deltas", () => {
|
||
const once = reduceHarnessEvent(initialState, delta("hel", 2));
|
||
const twice = reduceHarnessEvent(once, delta("hel", 2));
|
||
const complete = reduceHarnessEvent(twice, delta("lo", 3));
|
||
expect(complete.messages.byId.msg_1.text).toBe("hello");
|
||
});
|
||
|
||
it("maps run failure to incomplete assistant-ui status", () => {
|
||
expect(toThreadMessageLike(failedMessage).status).toMatchObject({ type: "incomplete" });
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/web test -- reducer runtime`
|
||
|
||
Expected: FAIL because the client store and runtime do not exist.
|
||
|
||
- [ ] **Step 3: Implement API client and deterministic reducer**
|
||
|
||
API methods cover all HTTP endpoints with typed safe errors. The store creates a Session, opens EventSource-compatible SSE, exposes `subscribe/getSnapshot`, and maintains connection state, skills, messages, runs, tools, approvals and timeline. Reject duplicate/out-of-order sequence, request reset on gaps, and reconnect with the latest accepted sequence. Do not store API keys inside the store snapshot.
|
||
|
||
- [ ] **Step 4: Implement External Store projection**
|
||
|
||
Use the exact installed `useExternalStoreRuntime` contract. Map user/assistant text, image metadata and tool-call/result parts to `ThreadMessageLike`. `onNew` uploads/uses attachments then calls messages API; cancel calls the server. Running, complete and incomplete/error statuses come only from reducer state. Do not configure assistant-ui transport or a second model request.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/web test -- reducer runtime`
|
||
|
||
Expected: PASS for delta merge, dedupe, gap/reconnect state, tool correlation, approval state and message status mapping.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add apps/web
|
||
git commit -m "feat(web): project harness events into assistant-ui"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 13: Three-panel application shell, Thread, Skills, and Timeline
|
||
|
||
**Files:**
|
||
- Create: `apps/web/src/main.tsx`
|
||
- Create: `apps/web/src/App.tsx`
|
||
- Create: `apps/web/src/components/layout/AppShell.tsx`
|
||
- Create: `apps/web/src/components/topbar/TopBar.tsx`
|
||
- Create: `apps/web/src/components/skills/SkillsPanel.tsx`
|
||
- Create: `apps/web/src/components/chat/Thread.tsx`
|
||
- Create: `apps/web/src/components/chat/Composer.tsx`
|
||
- Create: `apps/web/src/components/timeline/EventTimeline.tsx`
|
||
- Create: `apps/web/src/components/common/StateView.tsx`
|
||
- Create: `apps/web/tests/app-shell.test.tsx`
|
||
- Create: `apps/web/tests/states.test.tsx`
|
||
|
||
**Interfaces:**
|
||
- Consumes: HarnessClientStore and assistant-ui runtime from Task 12.
|
||
- Produces: accessible three-panel UI and mobile Chat/Skills/Events navigation.
|
||
|
||
- [ ] **Step 1: Write failing UI structure tests**
|
||
|
||
```tsx
|
||
it("renders the three primary work areas", () => {
|
||
render(<TestApp />);
|
||
expect(screen.getByRole("region", { name: "Skills Catalog" })).toBeVisible();
|
||
expect(screen.getByRole("main", { name: "Streaming Chat" })).toBeVisible();
|
||
expect(screen.getByRole("region", { name: "Event Timeline" })).toBeVisible();
|
||
});
|
||
|
||
it("fills but does not send the demo prompt", async () => {
|
||
await user.click(screen.getByRole("button", { name: "填入演示任务" }));
|
||
expect(screen.getByRole("textbox")).toHaveValue(expect.stringContaining("demo-account"));
|
||
expect(api.sendMessage).not.toHaveBeenCalled();
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/web test -- app-shell states`
|
||
|
||
Expected: FAIL because UI components are missing.
|
||
|
||
- [ ] **Step 3: Build the accessible app shell and states**
|
||
|
||
Wrap Thread primitives with the custom runtime. Skills list displays metadata/diagnostics and current requested/loaded indicators without bodies. Timeline sorts by sequence, filters categories and collapses older entries. Auto-scroll only when already near the bottom. TopBar exposes Session state, model/vision badges, Settings and reset. Model-missing disables send and points to Settings. Add Loading, Empty, Error, Offline/Reconnecting, Busy and Approval Pending state views.
|
||
|
||
- [ ] **Step 4: Implement responsive navigation semantics**
|
||
|
||
Desktop shows all three regions. Tablet toggles side panels with accessible buttons. Mobile uses a tablist with panels for Chat, Skills and Events and never mounts duplicate interactive controls. Preserve composer draft and scroll positions while switching.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/web test -- app-shell states`
|
||
|
||
Expected: PASS for regions, diagnostics, demo fill-only behavior, model-disabled composer, filters, reconnect/busy states and mobile tabs.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add apps/web
|
||
git commit -m "feat(web): add Agent Studio workspace UI"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 14: Tool UI, approval interaction, settings, and attachments
|
||
|
||
**Files:**
|
||
- Create: `apps/web/src/components/tools/AdsMetricsTool.tsx`
|
||
- Create: `apps/web/src/components/tools/SaveReportTool.tsx`
|
||
- Create: `apps/web/src/components/tools/ToolFallback.tsx`
|
||
- Create: `apps/web/src/components/approvals/ApprovalDialog.tsx`
|
||
- Create: `apps/web/src/components/settings/ModelSettings.tsx`
|
||
- Create: `apps/web/src/assistant/attachment-adapter.ts`
|
||
- Create: `apps/web/src/components/attachments/ImageAttachment.tsx`
|
||
- Create: `apps/web/src/components/attachments/ImagePreviewDialog.tsx`
|
||
- Create: `apps/web/tests/tools-approval.test.tsx`
|
||
- Create: `apps/web/tests/settings.test.tsx`
|
||
- Create: `apps/web/tests/attachments.test.tsx`
|
||
- Modify: `apps/web/src/App.tsx`
|
||
|
||
**Interfaces:**
|
||
- Consumes: assistant-ui Toolkit/render-only Tool UI and AttachmentAdapter signatures verified from installed types.
|
||
- Produces: registered UIs for exact names `mock_ads_metrics` and `save_report_draft`, server-authoritative ApprovalDialog, model settings, and upload/remove adapter.
|
||
|
||
- [ ] **Step 1: Write failing interaction tests**
|
||
|
||
```tsx
|
||
it("disables both decisions after submitting approval", async () => {
|
||
render(<ApprovalDialog request={pendingApproval} />);
|
||
await user.click(screen.getByRole("button", { name: "批准并保存" }));
|
||
expect(screen.getByRole("button", { name: "批准并保存" })).toBeDisabled();
|
||
expect(screen.getByRole("button", { name: "拒绝" })).toBeDisabled();
|
||
});
|
||
|
||
it("does not persist an API key", async () => {
|
||
await saveModelSettings({ apiKey: "sk-browser-secret" });
|
||
expect(localStorage.setItem).not.toHaveBeenCalled();
|
||
expect(sessionStorage.setItem).not.toHaveBeenCalled();
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/web test -- tools-approval settings attachments`
|
||
|
||
Expected: FAIL because the interaction components are missing.
|
||
|
||
- [ ] **Step 3: Implement Tool UIs and approval**
|
||
|
||
Register exact server tool names through the installed stable Toolkit API. Ads UI shows account, range and result metrics. Save UI shows redacted report summary, risk and all waiting/resolution states. Unknown tools use fallback without rendering raw oversized values. Approval buttons call APIs once and wait for SSE; countdown is advisory. Modal has dialog semantics, focus trap, Escape behavior and cannot be dismissed by backdrop while pending.
|
||
|
||
- [ ] **Step 4: Implement model settings**
|
||
|
||
Provide editable OpenAI, Anthropic and compatible presets with no keys. Use a password field with temporary reveal. Save/test/clear call server APIs; after save clear the local input and display only server-redacted metadata. Translate URL policy errors into concise Chinese. Prevent save while a Run is active. Switching to text-only preserves history but blocks new image sends.
|
||
|
||
- [ ] **Step 5: Implement custom AttachmentAdapter and image UI**
|
||
|
||
Adapter uploads immediately with multipart, maps progress/error/ready state, removes through the server and never converts large files to persistent base64. Use object URLs only for local previews and revoke on removal/unmount/success replacement. Composer supports picker, drop and paste, blocks send during upload, and exposes retry/remove. Preview dialog opens by keyboard and returns focus. Sent attachments render safe metadata plus authenticated Session-scoped preview behavior if implemented.
|
||
|
||
- [ ] **Step 6: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/web test -- tools-approval settings attachments`
|
||
|
||
Expected: PASS for all Tool states, approve/deny, focus, duplicate click prevention, key storage prohibition, redacted settings, upload/retry/remove/revoke, vision blocking and keyboard preview.
|
||
|
||
- [ ] **Step 7: Commit**
|
||
|
||
```bash
|
||
git add apps/web
|
||
git commit -m "feat(web): add tools approvals models and images"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 15: Visual system, responsive behavior, and accessibility checks
|
||
|
||
**Files:**
|
||
- Create: `apps/web/src/styles/tokens.css`
|
||
- Create: `apps/web/src/styles/base.css`
|
||
- Create: `apps/web/src/styles/layout.css`
|
||
- Create: `apps/web/src/styles/components.css`
|
||
- Create: `apps/web/tests/accessibility.test.tsx`
|
||
- Modify: `apps/web/src/main.tsx`
|
||
- Modify: UI components from Tasks 13–14 where labels/focus hooks require adjustment.
|
||
|
||
**Interfaces:**
|
||
- Consumes: semantic UI from Tasks 13–14.
|
||
- Produces: dark operational theme, three responsive breakpoints, reduced-motion support and keyboard-visible focus.
|
||
|
||
- [ ] **Step 1: Write failing accessibility assertions**
|
||
|
||
```tsx
|
||
it("gives icon-only actions accessible names", () => {
|
||
render(<TestApp />);
|
||
expect(screen.getByRole("button", { name: "停止当前运行" })).toBeVisible();
|
||
expect(screen.getByRole("button", { name: "打开模型设置" })).toBeVisible();
|
||
});
|
||
|
||
it("announces completion but not every streaming token", () => {
|
||
const { rerender } = render(<CompletionAnnouncer status="running" />);
|
||
expect(screen.queryByRole("status")).toBeEmptyDOMElement();
|
||
rerender(<CompletionAnnouncer status="complete" />);
|
||
expect(screen.getByRole("status")).toHaveTextContent("回答已完成");
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/web test -- accessibility`
|
||
|
||
Expected: FAIL until labels and live-region policy are present.
|
||
|
||
- [ ] **Step 3: Implement tokens and layouts**
|
||
|
||
Define surfaces, borders, text, muted, cyan accent, pink danger, focus, spacing in 8px increments, radii and shadows. Keep body at `#111111`, use accent colors sparingly, and avoid wrapping every item in a card. Use CSS grid for desktop, collapsible two-column tablet, tabs/sheets mobile, `min-width: 0`, responsive media and overflow guards. Interactive targets are at least 44px.
|
||
|
||
- [ ] **Step 4: Implement focus, motion and announcement behavior**
|
||
|
||
Use `:focus-visible` rings with sufficient contrast, text+icon status labels, tooltips for icon-only actions, and `prefers-reduced-motion` overrides. Announce only completed assistant responses through a polite live region. Confirm Dialog focus return and mobile tab keyboard navigation.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm --filter @agent-studio/web test -- accessibility && pnpm --filter @agent-studio/web typecheck`
|
||
|
||
Expected: PASS for accessible names, modal roles, tab semantics, live announcement policy and type safety.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add apps/web
|
||
git commit -m "feat(web): polish responsive accessible interface"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 16: Full fake-session integration path
|
||
|
||
**Files:**
|
||
- Create: `packages/harness/tests/fakes/fake-agent-session.ts`
|
||
- Create: `apps/server/tests/full-flow.integration.test.ts`
|
||
- Create: `apps/web/tests/full-flow.test.tsx`
|
||
- Modify: test setup files in Harness, Server and Web packages.
|
||
|
||
**Interfaces:**
|
||
- Consumes: complete Harness/Server/Web APIs.
|
||
- Produces: deterministic full-flow acceptance coverage without model credentials or network.
|
||
|
||
- [ ] **Step 1: Write the failing end-to-end integration test**
|
||
|
||
```ts
|
||
it("runs image + metrics + approved report flow and cleans up", async () => {
|
||
const session = await client.createSession();
|
||
await client.configureFakeModel(session.id, { supportsImages: true });
|
||
const image = await client.uploadFixture(session.id, "creative.png");
|
||
const accepted = await client.sendMessage(session.id, { text: demoPrompt, attachmentIds: [image.attachmentId] });
|
||
await fake.emitDemoRun(accepted.runId);
|
||
const approval = await events.waitFor("approval.required");
|
||
await client.approve(approval.payload.approvalId);
|
||
await events.waitFor("run.completed");
|
||
expect(await readSessionReport(session.id)).toContain("账户总结");
|
||
await client.deleteSession(session.id);
|
||
expect(await sessionPathExists(session.id)).toBe(false);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/server test -- full-flow.integration`
|
||
|
||
Expected: FAIL until the programmable fake and dependency wiring exist.
|
||
|
||
- [ ] **Step 3: Implement the programmable fake**
|
||
|
||
The fake stores subscribers, keeps prompt deferred, records structured images, emits assistant deltas, read Skill calls/results, mock tool calls/results, a save tool request that waits on the real ApprovalBroker, and terminal events. Its `abort` and `dispose` expose assertions and release all pending operations.
|
||
|
||
- [ ] **Step 4: Complete server and browser flow tests**
|
||
|
||
Assert exact causal ordering for session/run/user/skill/tool/approval/assistant/run completion, toolCallId and approvalId correlation, report contents, image metadata without base64, and deletion cleanup. Add Deny and fake-timer Timeout variants that assert no report exists while assistant content may still complete. Web flow asserts reducer/UI states through the same event sequence.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
Run: `pnpm test`
|
||
|
||
Expected: all package tests pass without environment API keys and without network access.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add packages/harness apps/server apps/web
|
||
git commit -m "test: cover complete Agent Studio flow"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 17: Teaching documentation and production boundary
|
||
|
||
**Files:**
|
||
- Modify: `README.md`
|
||
- Create: `docs/architecture.md`
|
||
- Create: `docs/event-mapping.md`
|
||
- Create: `docs/security-boundaries.md`
|
||
- Create: `docs/classroom-demo.md`
|
||
- Create: `docs/production-roadmap.md`
|
||
- Create: `packages/harness/tests/documentation.test.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: actual dependency versions, commands, routes and observed behavior from Tasks 1–16.
|
||
- Produces: complete operator/trainer documentation with Mermaid diagrams and no unverified claims.
|
||
|
||
- [ ] **Step 1: Write documentation assertions**
|
||
|
||
Create a focused Vitest that reads the Markdown files and asserts README contains the exact phrases `Pi does not provide an OS sandbox`, `Project Trust is not a sandbox`, `skill.requested`, `skill.loaded`, all root commands, and every required documentation link.
|
||
|
||
- [ ] **Step 2: Verify red state**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- documentation`
|
||
|
||
Expected: FAIL because final documentation is incomplete.
|
||
|
||
- [ ] **Step 3: Write README and five guides**
|
||
|
||
Document purpose, actual versions/API differences, architecture and Agent Loop diagrams, boundaries, directory tree, setup, environment and Settings model configuration, credential lifetime, image chain, exact demo prompt, Approve/Deny behavior, progressive disclosure, inferred Skill events, in-memory/single-process limits, test commands and production roadmap. Security guide explicitly explains Node process inheritance and the need for Docker/VM/micro-VM/remote sandbox for untrusted work.
|
||
|
||
- [ ] **Step 4: Verify documentation**
|
||
|
||
Run: `pnpm --filter @agent-studio/harness test -- documentation`
|
||
|
||
Expected: PASS and no stale commands, versions or route names.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add README.md docs packages/harness/tests
|
||
git commit -m "docs: add Agent Studio teaching guide"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 18: Final automated and manual verification
|
||
|
||
**Files:**
|
||
- Modify only files implicated by observed verification failures.
|
||
- Create: `docs/verification-results.md`
|
||
|
||
**Interfaces:**
|
||
- Consumes: the finished workspace.
|
||
- Produces: verified build plus an exact record of commands, test count, local URLs and remaining limitations.
|
||
|
||
- [ ] **Step 1: Run the complete automated gate**
|
||
|
||
Run each command separately and preserve its real result:
|
||
|
||
```bash
|
||
pnpm lint
|
||
pnpm typecheck
|
||
pnpm test
|
||
pnpm build
|
||
```
|
||
|
||
Expected: all four exit 0. Record package/test counts from output rather than estimating.
|
||
|
||
- [ ] **Step 2: Start the application**
|
||
|
||
Run: `pnpm dev`
|
||
|
||
Expected: Server and Vite Web start from the root command, health returns success, Web loads without console errors, and missing model state is explicit.
|
||
|
||
- [ ] **Step 3: Manually exercise desktop and mobile critical paths**
|
||
|
||
At desktop width verify Skills diagnostics, Demo Prompt fill-only behavior, fake/demo stream, Tool cards, Approval approve and deny, Timeline filtering/replay and Settings redaction. At a phone viewport verify Chat/Skills/Events tabs, no horizontal overflow, attachment picker/preview, keyboard focus and reduced-motion behavior. Use the browser’s visible network/storage tools to confirm the API key is absent from URLs, responses, SSE, localStorage, sessionStorage and IndexedDB.
|
||
|
||
- [ ] **Step 4: Fix only observed defects and repeat affected gates**
|
||
|
||
For every defect, first add a regression test that demonstrates it, run that test red, implement the smallest correction, run it green, then rerun all four root gates. Do not skip or weaken security assertions.
|
||
|
||
- [ ] **Step 5: Write verification record**
|
||
|
||
`docs/verification-results.md` must include date, Node/pnpm versions, exact commands and exit results, total tests, browser widths checked, demo paths exercised, credential inspection result and concrete known limitations.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add .
|
||
git commit -m "chore: verify Agent Studio Mini"
|
||
```
|