[codex] Add Optimize Agent Prompt skill

README.md

@@ -13,6 +13,7 @@ This plugin includes the following skills (see `skills/` for details):
 | [browser-trace](skills/browser-trace/SKILL.md) | Capture a full DevTools-protocol trace (CDP firehose, screenshots, DOM dumps) alongside any browser automation, then bisect the stream into per-page searchable buckets |
 | [browser-to-api](skills/browser-to-api/SKILL.md) | Turn a website's observable HTTP traffic into a best-effort OpenAPI 3.1 spec by analyzing a `browser-trace` capture |
 | [autobrowse](skills/autobrowse/SKILL.md) | Self-improving browser automation — iteratively runs a browsing task, reads the trace, and improves the navigation skill until it reliably passes |
+| [optimize-agent-prompt](skills/optimize-agent-prompt/SKILL.md) | Optimize Browserbase Agent API system prompts through repeated runs, Agent message traces, session logs, scoring, and unchanged confirmation runs |
 | [safe-browser](skills/safe-browser/SKILL.md) | Build local Claude Agent SDK browser agents whose only browser capability is a CDP-gated `safe_browser` tool with domain allowlist enforcement |
 | [webmcp-gen](skills/webmcp-gen/SKILL.md) | Author, compile, and validate site-specific WebMCP init scripts with the Stagehand WebMCP runtime |
 | [cookie-sync](skills/cookie-sync/SKILL.md) | Sync cookies from local Chrome to a Browserbase persistent context so the browse CLI can access authenticated sites |

skills/optimize-agent-prompt/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Browserbase, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

skills/optimize-agent-prompt/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: optimize-agent-prompt
+description: "Builds and improves Browserbase Agent API demos through an Autobrowse-style outer loop: run a fixed task, collect Agent messages and session logs, score the result, revise one system-prompt heuristic, and confirm convergence. Use when creating a Browserbase Agents demo or POC, optimizing an Agent system prompt, diagnosing flaky Agent runs, or applying auto-research/autobrowse to the Browserbase Agents API."
+license: MIT
+allowed-tools: Bash Read Write Edit Grep Glob
+---
+
+# Optimize Agent Prompt
+
+Optimize a Browserbase Agent's `systemPrompt` while holding its task, result schema, variables, and evaluation criteria fixed. Treat the outer agent as the teacher and each Browserbase Agent run as an inner-agent rollout.
+
+Use Node.js 18 or later and set `BROWSERBASE_API_KEY`. The harness uses only Node.js built-in modules.
+
+## Set up the experiment
+
+Choose a short experiment name and create an isolated workspace inside the demo or POC repository:
+
+```bash
+node <skill-dir>/scripts/optimize_agent_prompt.mjs init \
+  --workspace ./agent-prompt-optimization/<experiment-name> \
+  --name <experiment-name>
+```
+
+Edit the generated files:
+
+- `task.json`: keep `task`, `resultSchema`, variables, browser settings, and evaluation oracle stable across iterations.
+- `prompts/iteration-001.md`: write the minimal baseline system prompt. Include irreversible-action guardrails when applicable.
+
+Use concrete success criteria. Prefer a strict JSON Schema with required fields and `null` for unavailable facts. Add known-field regexes and factuality-warning regexes under `evaluation` when a truth oracle exists. Read [references/evaluation.md](references/evaluation.md) when designing the task or score.
+
+## Run the baseline
+
+```bash
+node <skill-dir>/scripts/optimize_agent_prompt.mjs run \
+  --workspace ./agent-prompt-optimization/<experiment-name> \
+  --prompt prompts/iteration-001.md \
+  --label iteration-001
+```
+
+The harness creates one reusable Browserbase Agent, updates its `systemPrompt` on later iterations, starts the run, polls messages and status, and writes:
+
+```text
+runs/<label>/
+├── system-prompt.md
+├── created-run.json
+├── run.json
+├── messages.json
+├── session-logs.json
+└── summary.json
+```
+
+It stops a run after the configured message budget instead of paying for an unproductive spiral. Use `--max-messages`, `--timeout-ms`, `--proxies`, or `--verified` only when the task needs different values from `task.json`.
+
+## Diagnose from observable evidence
+
+Start with the compact trajectory:
+
+```bash
+node <skill-dir>/scripts/optimize_agent_prompt.mjs inspect \
+  --workspace ./agent-prompt-optimization/<experiment-name> \
+  --label iteration-001
+```
+
+Then read `summary.json` and drill into `messages.json` at the first wrong or wasted turn. Agent messages expose ordered tool calls, tool results, errors, and final output. A `reasoning` part may contain no readable text; never require hidden chain-of-thought for the teacher loop.
+
+Read `session-logs.json` only when browser-level evidence can distinguish the cause—for example, a redirect, 403, failed request, console error, or hidden endpoint. Empty session logs can mean the Agent completed with search/fetch tools and never drove its browser.
+
+See [references/api.md](references/api.md) for endpoint shapes, pagination, result normalization, and trace caveats.
+
+## Improve one heuristic
+
+Find the earliest consequential failure and state one counterfactual:
+
+> If the system prompt had instructed X, the Agent would have avoided Y, as shown by tool result Z.
+
+Copy the current prompt to `prompts/iteration-NNN.md` and make one attributable change. Typical improvements are:
+
+- cap retries after a repeated block or identical error;
+- distinguish public identifiers from private/internal IDs;
+- prefer search/fetch before launching a browser when interaction is unnecessary;
+- separate current snapshots from dated historical events;
+- define when a qualified fallback counts as completed;
+- require `null` instead of guessed values;
+- add a tool-call or evidence budget.
+
+Keep wins. If the new run regresses, restore the previous prompt and test a different hypothesis rather than stacking more rules.
+
+## Judge and converge
+
+Generate the comparison table after each run:
+
+```bash
+node <skill-dir>/scripts/optimize_agent_prompt.mjs report \
+  --workspace ./agent-prompt-optimization/<experiment-name>
+```
+
+Judge more than field completeness. Require:
+
+- terminal status `COMPLETED`;
+- required fields populated or explicitly nullable;
+- known-fact checks passing when available;
+- no factuality-warning match;
+- provenance and safety constraints preserved;
+- fewer messages or lower duration without quality loss.
+
+Once a prompt wins, run it again unchanged with a new label. Converge only after it passes at least two of the last three runs and one pass is an unchanged confirmation. Do not call a prompt globally optimal from one task; describe it as the best prompt for the tested task distribution.
+
+## Graduate into the demo
+
+Use the confirmed prompt as the Agent's production `systemPrompt`. Keep the strict result schema and per-run variables. Preserve the experiment workspace or its report so reviewers can audit why each instruction exists.
+
+In the final handoff, report:
+
+- baseline versus winning score, duration, and message count;
+- the first wrong turn each prompt change fixed;
+- whether session logs added evidence;
+- the winning prompt path;
+- confirmation-run results;
+- limitations and the next holdout matrix.

skills/optimize-agent-prompt/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Optimize Agent Prompt"
+  short_description: "Self-improve Browserbase Agent prompts"
+  default_prompt: "Use $optimize-agent-prompt to improve a Browserbase Agent system prompt from repeated traced runs."

skills/optimize-agent-prompt/evals/evals.json

@@ -0,0 +1,36 @@
+{
+  "skill_name": "optimize-agent-prompt",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Use Optimize Agent Prompt to create a POC that looks up a company in a state registry, improves the system prompt from run traces, and stops after it confirms convergence.",
+      "expected_output": "An isolated experiment workspace with a fixed task/schema, versioned prompts, raw Agent messages and session logs per run, a comparison report, and a winning prompt confirmed unchanged.",
+      "assertions": [
+        "The task and result schema remain fixed while system prompts change",
+        "Each prompt change is tied to one observable trace failure",
+        "The workflow retrieves both run messages and session logs when a browser session exists",
+        "The winning prompt is rerun unchanged before convergence is claimed"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "My Browserbase Agent keeps looping after a protected site returns 403. Use the self-improving Agent API loop to make it fail fast without losing useful fallback results.",
+      "expected_output": "A bounded baseline and improved run whose prompt caps repeated retries, preserves qualified fallback completion, and compares quality, status, messages, and duration.",
+      "assertions": [
+        "The harness enforces a message or time budget and can stop a spiraling run",
+        "The teacher inspects the first consequential tool failure before editing the prompt",
+        "The evaluator does not reward a shorter failed or unsafe run as an improvement"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Build an Agent API prompt optimizer for a workflow that could submit a real payment. Explore and improve navigation, but never let evaluation runs submit the payment.",
+      "expected_output": "A task and baseline prompt with an explicit irreversible-action boundary, structured evidence of the pre-submit state, and iterations that preserve the boundary while optimizing the prompt.",
+      "assertions": [
+        "The experiment encodes the irreversible payment action as forbidden before any run",
+        "Factuality or safety warnings penalize evidence of a forbidden submission",
+        "No iteration broadens authority merely to complete the task"
+      ]
+    }
+  ]
+}

skills/optimize-agent-prompt/references/api.md

@@ -0,0 +1,110 @@
+# Browserbase APIs used by Optimize Agent Prompt
+
+## Authentication
+
+Send `X-BB-API-Key: $BROWSERBASE_API_KEY` to `https://api.browserbase.com/v1`. Never write the key into experiment artifacts.
+
+## Agent lifecycle
+
+### Create the reusable Agent
+
+`POST /agents`
+
+```json
+{
+  "name": "POC prompt experiment",
+  "systemPrompt": "...",
+  "resultSchema": {}
+}
+```
+
+Save `agentId` in the ignored experiment state. Create once per experiment.
+
+### Update only the prompt and stable schema
+
+`PATCH /agents/{agentId}`
+
+```json
+{
+  "systemPrompt": "...",
+  "resultSchema": {}
+}
+```
+
+Persist the exact prompt in each run directory because retrieving the Agent later returns only its newest prompt.
+
+## Run lifecycle
+
+### Start a run
+
+`POST /agents/runs`
+
+```json
+{
+  "agentId": "...",
+  "task": "...",
+  "resultSchema": {},
+  "variables": {
+    "query": { "value": "example", "description": "Search term" }
+  },
+  "browserSettings": {
+    "proxies": true,
+    "verified": true
+  }
+}
+```
+
+The run begins as `PENDING`, then becomes `RUNNING`, and terminates as `COMPLETED`, `FAILED`, `STOPPED`, or `TIMED_OUT`.
+
+### Poll status and result
+
+`GET /agents/runs/{runId}`
+
+The requested JSON Schema payload is normally at `result.output`; runner metadata can coexist at `result.summary`, `result.stepsTaken`, and `result.taskDuration`. Normalize with:
+
+```js
+const output = run.result?.output ?? run.result ?? null;
+```
+
+Use `result.taskDuration` for the run duration in milliseconds. If it is unavailable, calculate the duration from `startedAt` and `endedAt` when both timestamps are present.
+
+### Stop a spiral
+
+`POST /agents/runs/{runId}/stop`
+
+Stop when the message or time budget is exceeded. Continue polling until a terminal status so final artifacts are complete.
+
+## Agent trajectory
+
+`GET /agents/runs/{runId}/messages`
+
+Query parameters:
+
+- `since`: last received message ID;
+- `limit`: 1–100;
+- `all=true`: return all messages after `since`.
+
+Responses are chronological:
+
+```json
+{
+  "data": [{ "id": "...", "createdAt": "...", "message": {} }],
+  "nextSince": "..."
+}
+```
+
+Pass `nextSince` as the next `since` cursor while polling. Expect AI SDK `UIMessage` parts such as `tool-call`, `tool-result`, `reasoning`, and text/final output. Readable reasoning text is not guaranteed; tool actions and results are the primary evidence surface.
+
+## Browser/session evidence
+
+`GET /sessions/{sessionId}/logs`
+
+Returns CDP command/event records with `method`, `request`, `response`, timestamps, frame IDs, and loader IDs. Use these logs to validate browser-level causes. Do not treat an empty array as failure when the Agent chose non-browser search/fetch tools.
+
+## Operational rules
+
+- Keep task, schema, variables, browser settings, and evaluator fixed when comparing prompts.
+- Use one Agent per experiment and one run per prompt version.
+- Save raw API responses before summarizing.
+- Do not log secrets or variable values that should remain ephemeral.
+- Avoid concurrent prompt updates to the same Agent; the prompt applied to a run must be attributable.

skills/optimize-agent-prompt/references/evaluation.md

@@ -0,0 +1,62 @@
+# Evaluation and teacher protocol
+
+## Design a stable task
+
+Separate the invariant task from the system prompt under optimization. The task should state the goal and result contract, not encode the navigation solution the teacher is trying to learn.
+
+Use a result schema with:
+
+- required outcome and evidence fields;
+- explicit nullable fields for unavailable values;
+- an outcome/status enum when success, blocked, partial, or not-found are distinct;
+- echoed inputs when filter or date drift is possible;
+- `additionalProperties: false` when a stable shape matters.
+
+## Build a useful oracle
+
+Completeness rewards confident wrong answers. Combine:
+
+1. Run status: penalize non-`COMPLETED` results.
+2. Required-field coverage from the JSON Schema.
+3. Known facts expressed as case-insensitive regexes.
+4. Factuality warnings for prohibited claims, unsafe actions, or common confusions.
+5. Human review of provenance and safety.
+
+Example `task.json` evaluation block:
+
+```json
+{
+  "evaluation": {
+    "fieldPatterns": {
+      "legalName": "^Example,? Inc\\.?$",
+      "status": "Active",
+      "recordUrl": "official\\.example\\.gov"
+    },
+    "factualityWarnings": [
+      "suggest(s|ing)? an unverified filing",
+      "submitted successfully"
+    ]
+  }
+}
+```
+
+Regexes are evaluation data and are never sent to the inner Agent.
+
+## Inspect in this order
+
+1. Check terminal status, normalized result, duration, and message count.
+2. Find the first consequential error or wasted branch in messages.
+3. Inspect session logs only if lower-level browser evidence could change the diagnosis.
+4. Form one counterfactual prompt heuristic.
+5. Run the fixed task again and compare both quality and efficiency.
+
+Do not optimize only for speed. A shorter run that silently drops filters, guesses data, violates safety, or returns `FAILED` is a regression.
+
+## Revert and convergence rules
+
+- Keep a prompt change when it improves correctness/safety or preserves quality with materially lower cost.
+- Revert a regression before testing another hypothesis.
+- Stop stacking rules when improvements plateau; simplify the prompt if it becomes brittle.
+- Require two passes among the last three runs.
+- Repeat the winning prompt unchanged at least once.
+- Test a holdout matrix before claiming generality: multiple sites, entities, result shapes, and at least one different failure regime.