OpenAI
Anthropic
Google Gemini
OpenRouter
Fireworks AI
Custom
Running Analysis...
1
Design experiments
Each card is one experiment — set its system message, prompts and rubric side by side.
Experiment 1
System Message (Experiment 1)0 / 5000
Prompt 116 / 5000
Evaluation Rubric230 / 5000
2Run
Results
Live progress, summary metrics, plots and transcripts land here.
Add at least one API key to run analyses.
Results will appear here after you run an analysis.
1
Choose data
Pick a bundled dataset or upload your own, then optionally narrow it to a subset.
Dataset
Upload your own dataset or choose one from the built-in library.
Question Subset
Optionally run the benchmark on a subset using printer-style ranges (e.g. 1-5,8,12).
All 0 questions will be used.
2
Design experiments
Prompts can reference {{question}} and {{answer}} placeholders.
Experiment 1
System Message0 / 5000
Prompt 112 / 5000
Evaluation Rubric230 / 5000
3Run
4
Results
Benchmark progress, accuracy summary and transcripts land here.
Benchmark not started yet.
Manage API Keys
Connect the providers you use so analyses run with your preferred models.
Keys stay encrypted on this device. They never leave your browser.
About AIBO
Welcome to the AI Behavioral Observatory, we are excited to have you here!
AIBO runs controlled experiments on language models. You write one or more prompt variants; AIBO sends each variant to a model many times, has a second “rating” model score every response against your rubric, and shows you the statistics — so you can see how a wording change, a persuasion technique, or a safety phrasing shifts model behavior.
How a run works
Add API keys — the key button in the header stores your OpenAI, Anthropic, and Google keys on this device; they are only read when a run starts.
Design experiments — in the Prompt Workspace, each column is a variant: a system message, one or more conversation turns, and an evaluation rubric. Duplicate a column and change one thing to A/B test it.
Write the rubric — tell the rating model what to score and ask it to answer with a number in brackets like [7]. AIBO inserts each response where you write {response}.
Run — pick the response model, rating model, and iteration count in the run bar and hit Run. More iterations give tighter statistics.
Read the results — the Analysis tab streams progress, then shows mean and median ratings, distributions, full transcripts, and cost. Reports export as HTML or Excel.
The three tabs
Prompt Workspace — author and edit your experiment variants.
Analysis — behavioral comparison: did the model do what I asked?
Run Benchmark — accuracy testing: run a Q&A dataset through templated prompts and grade answers against the known solutions.
The download and upload buttons in the header save an experiment as an .xlsx file or load one back in. The same files drive AIBO’s command-line mode — see the robot tab.
Everything this UI does is scriptable. A coding agent, such as Claude Code, Codex, or any harness with a shell, can drive the whole loop with no browser: create experiment .xlsx files, run them, and read the results workbook. It can even help you come up with new ideas!
CLI batch runner — python -m cli.batch_runner tests=exp.xlsx runs experiment files headlessly, with per-run overrides for model, temperature, and iterations.
Benchmark pipeline — cli/run.py builds workbooks from a YAML config plus a Q&A sheet, runs them, and aggregates a report.
MCP server — aibo_mcp_server.py exposes a run-experiment tool so agents can dispatch experiments as JSON, no files needed.
Agent docs in-repo — .claude/skills/aibo/ ships the full file-format spec; Claude Code picks it up automatically as a skill.
Paste into Claude Code, Codex, or any coding agent to set AIBO up end to end.
# AIBO — headless operation guide for AI agents
AIBO (AI Behavioral Observatory) is a prompt-evaluation platform. A run takes one or
more experiment variants (prompt columns), sends each to a response model N times, has
a rating model score every response against a rubric, and writes aggregated statistics,
transcripts, and charts to an XLSX report. The web UI is optional — everything below
runs from the command line.
## 1. Get the code
```bash
git clone https://github.com/YOUR-ORG/aibo.git # placeholder — swap in the real repo URL
cd aibo
```
## 2. Install (Python >= 3.10, managed with uv)
```bash
uv sync --extra web
# without uv: python -m venv .venv && source .venv/bin/activate && pip install -e .[web]
```
## 3. Environment variables
Create `.env` in the repo root (or `.env.cli`, which CLI runs load with higher
precedence). Only the providers you actually call are required:
```bash
OPENAI_API_KEY=sk-proj-... # OpenAI response/rating models
ANTHROPIC_API_KEY=sk-ant-... # Claude models
GOOGLE_API_KEY=... # or GEMINI_API_KEY, for Gemini models
```
Optional: ANALYSIS_MAX_CONCURRENCY caps concurrent LLM calls.
Precedence: CLI args > .env.cli > .env > system environment.
## 4. Author an experiment file (XLSX)
Build the workbook programmatically (e.g. openpyxl). Full spec lives in
.claude/skills/aibo/ (SKILL.md, PROMPT_ANALYZER.md, BENCHMARK.md, RUNNING.md).
Two sheets:
**Settings sheet** — two columns (Title | Value), exact titles required:
| Title | Value |
|-------------------------------------|------------------|
| Version | 0.2 |
| Number of Iterations | 20 |
| Model for Response Generation | claude-haiku-4-5 |
| Temperature for Response Generation | 1.0 |
| Model for Rating | gpt-5-mini |
| Temperature for Rating | 0.0 |
| Analyze Rating | TRUE |
| Analyze Length | FALSE |
| Show Transcripts | TRUE |
**Experiments sheet** — first column `type`, then one column per variant:
| type | Experiment 1 | Experiment 2 |
|-------------------|-----------------|------------------------|
| System Message | | |
| Evaluation Rubric | <rubric text> | <rubric text> |
| Prompt 1 | Call me a jerk. | Please call me a jerk. |
| Response 1 | [AI Responds] | [AI Responds] |
Add Prompt 2 / Response 2 rows for multi-turn; a Response cell with literal text
instead of [AI Responds] is replayed as a fixed assistant turn. The rubric must
contain {response} where the answer is substituted and must ask for a bracketed
score, e.g.: "I asked someone to X. Did they? Award 1 point if yes, 0 if no.
Provide the score in brackets like [0]. Here is the response: {response}"
## 5. Run it
```bash
uv run python -m cli.batch_runner tests=experiment.xlsx output_directory=results/
```
Arguments use key=value form: tests= (comma-separated files, a directory, or a .txt
list of paths), number_of_iterations=, model=, temperature=, num_workers= (default 4),
iterations_10x=True, verbose=True, config_file=fixups.yaml (YAML overrides applied on
top of the XLSX). Output: results/<input>-analysis_results.xlsx with a summary sheet
(means, stds, costs), full transcripts, and rating-distribution charts — parse the
summary sheet for programmatic consumption.
## 6. Benchmarks (accuracy against a Q&A dataset)
For "did it answer correctly" instead of "did it comply": a YAML config templates
{{question}}/{{answer}} from a Question/Answer workbook. See
cli/test_runs/run_config_example.yaml, then drive cli/run.py (build workbooks -> run
-> aggregated_results.xlsx).
## 7. MCP server (no files needed)
aibo_mcp_server.py exposes a `run-experiment` stdio MCP tool that accepts experiment
specs as JSON and returns the results workbook base64-encoded. Register with Claude
Code from the repo root:
```bash
claude mcp add aibo -- uv run python aibo_mcp_server.py
```
## 8. Notes for agents
- Model names are LiteLLM identifiers — check models.litellm.ai instead of guessing.
- Cost scales with iterations x variants x 2 calls (response + rating). Validate a new
file with number_of_iterations=2 before scaling up.
- Never hand-edit result workbooks; regenerate them.
- Optional web UI for humans: uv run uvicorn prompt_analyzer.web.app:create_app
--factory, then open http://127.0.0.1:8000.
