OpenAI Agents SDK v2 vs Agno vs Pydantic AI: A 2026 Field Guide (with a real benchmark)
A complete, apple-to-apple comparison of the three agent frameworks that matter in 2026, full feature matrix plus a real benchmark on live Daytona sandboxes with gpt-5. All three can code; they diverge on durability, cost, and architecture. Includes a pick-by-use-case guide for analytics, coding, multi-agent, RAG, voice, and enterprise.
June 2026. The benchmark numbers (tokens, turns, pass rates, wall clock) are from real runs: live Daytona cloud sandboxes, real gpt-5, the same locked test suite for every framework. Framework facts, versions, feature counts, and star counts are from current official docs and repos as of June 2026 (sources linked), not from the benchmark. Nothing here is mocked or estimated.
Three Python agent frameworks are worth arguing about in 2026: OpenAI's Agents SDK (the post-April-2026 "v2" with native sandboxing), Agno (with its AgentOS runtime), and Pydantic AI. I wanted a comparison solid enough to pick one for any kind of agent: an analytics bot, a long-horizon coding agent, a multi-agent system, a RAG assistant, a voice agent, an enterprise workflow with approvals. So this is two things stapled together: a full feature matrix, and a real benchmark that actually kills a container to see what survives.
At gpt-5, all three write correct code. They diverge on what "durable" means, how many tokens they burn, and where the agent loop runs. Pick by failure mode and use case, not by hype.
What changed in 2026
The harness moved down to the model. Both frontier labs shipped model-native agent harnesses (Anthropic's Claude Agent SDK and OpenAI's Agents SDK overhaul), and both converged on the same primitives: MCP, skills, subagents, filesystem tools, and a sandbox as the place code actually runs. The older "orchestration framework" layer kept maturing on a different axis: batteries, model-agnosticism, and a production runtime.
| Framework | One-line identity | Sandbox is… | Headline durability |
|---|---|---|---|
| OpenAI Agents SDK | model-native harness, OpenAI-first | the execution substrate | native workspace snapshot + Temporal |
| Agno | batteries-included, model-agnostic, AgentOS runtime | a tool | DB-backed session continuity |
| Pydantic AI | type-safe, model-agnostic, explicit loop control | a tool you wire | Temporal / DBOS / Prefect / Restate |
The single diagram that explains most of the differences: where the agent loop runs, and what the sandbox is to it.
The benchmark
To measure the harness and not the model, I held everything else constant:
- Same task, hard on purpose. Build a
ledgerpackage: double-entry accounting with integer money (no floats), balanced-entry enforcement that raises custom exceptions, multi-account balances, a trial balance that must sum to zero, JSON round-trip, and an argparse CLI. Multi-file and interdependent. - Same rubric, locked. 30 pytest tests written up front, uploaded into the sandbox before the agent ran, edits forbidden. Partial credit: how many of 30 pass.
- Same model.
gpt-5for all three. - Same sandbox. A real Daytona cloud container per run. For Agno and Pydantic AI (sandbox-as-tool) I created the container and handed its ID to the framework, so scoring runs in the exact place the agent worked.
- Objective scoring. I run
pytestin the sandbox myself and parse it. I never trust the agent's self-report.
| Metric | OpenAI v2 | Agno | Pydantic AI |
|---|---|---|---|
| Tests passed | 30/30 | 30/30 | 30/30 |
| Tokens (median of 3 trials) | 80,266 | 171,259 | 64,724 |
| Host secret leaked into sandbox | no | no | no |
I ran the synthetic task 3 times per framework, and the tokens are noisy: OpenAI v2 was stable (79K–81K), Agno landed 163K–197K, Pydantic AI swung 30K–92K. So read these as ballpark, not precision. The one stable signal is that all three pass 30/30, which is the point: with a frontier model, raw coding capability is not where these frameworks differ. A synthetic build-from-spec is a weak discriminator. The real test is the next section, where the gaps widen by an order of magnitude.
Round 2: a real GitHub bug (SWE-bench)
I ran a real one: SWE-bench flask-5014, an actual Flask issue. The agent gets the real repo at the buggy commit, a hidden failing test that encodes the fix ("an empty Blueprint name should raise ValueError"), and has to find where in thousands of lines of Flask to make a minimal change. Scoring is the real SWE-bench criterion: the target test must pass and an existing-test sample must stay green (no regression).
Getting the environment right was itself the lesson. This one instance needed three pinned versions, and that is why SWE-bench ships per-instance Docker images:
| Pin | Why |
|---|---|
| Python 3.11 | Flask 2.3 calls pkgutil.get_loader, removed in 3.12 |
pytest<8 | the repo's conftest uses an internal pytest 8 dropped |
Werkzeug<2.4 | 2.3-era Flask reads werkzeug.__version__, dropped in Werkzeug 3 |
The agent is the easy part; the environment is the tax.
| Metric | OpenAI v2 | Agno | Pydantic AI |
|---|---|---|---|
| Resolved the issue | yes | yes | yes |
| Target test (FAIL_TO_PASS) | 1/1 | 1/1 | 1/1 |
| Regression sample (PASS_TO_PASS) | 25/25 clean | 25/25 clean | 25/25 clean |
| Turns | 14 | 31 | 16 |
| Tokens | 212,471 | 1,002,231 | 177,503 |
| Wall clock | 106s | 329s | 149s |
All three resolved the bug (target test passes, no regression in the sample). I captured the actual diff only for Pydantic AI, which made the canonical one-line fix; OpenAI and Agno also passed but I didn't save their diffs. Capability is still a wash; the story is cost.
The cost story in one table:
| Framework | Tokens | What drove it |
|---|---|---|
| Pydantic AI | 177K | leanest, surgical edits, tight context |
| OpenAI v2 | 212K | in-sandbox harness has context compaction → bounded |
| Agno | 1.0M | no context compaction; the full transcript grows every turn |
Agno's answer was correct and regression-free, but a million tokens would hurt in production. Agno has no automatic context compaction by default, so the transcript grows unbounded (Agno docs; token-aware context management is still an open request, #4952). The obvious fix is its compress_tool_results flag, except enabling it hung the run in this version (2.6.16) on an infinite "tool count limit" loop, which is a known class of Agno bug (#7133), so the high cost is not a one-flag fix today. Pydantic AI, by contrast, is bring-your-own-tools: the tools you wire (full-fidelity reads, surgical edits) directly shape what it can do. Given good tools it was the leanest here. Tool design is the variable you own.
One instance, one trial per framework, read Round 2 as a worked example of how the three diverge under real load, not a leaderboard.
Round 3: multi-agent teams
Last workload: build a 3-module package with one worker per module, each framework using its multi-agent approach. Only Agno has a first-class team primitive here, a Team(mode=coordinate) with a leader routing three worker agents. OpenAI uses a coordinator with the workers exposed as tools (agents-as-tools), all sharing one sandbox session. Pydantic has no built-in team abstraction, so the coordinator delegates to three worker agents through ordinary tool calls. An 18-test suite scores the result.
| Framework | Tests | Turns | Tokens | Wall clock |
|---|---|---|---|---|
| OpenAI v2 (workers-as-tools) | 18/18 | 8 | 57,363 | 81s |
| Pydantic AI (agent delegation) | 18/18 | 9 | 62,827 | 212s |
| Agno (Teams, leader-routed) | 18/18 | — | leader-only* | 552s |
All three coordinated three workers to a clean 18/18, and the OpenAI and Pydantic wirings both ran first try with no fixes. Two things stood out. OpenAI was fastest by a wide margin (81s); Agno's leader-routed Team was the slowest at 552s. And Agno's token figure is marked leader-only because its Team run metrics captured the coordinator, not the three member agents, so I can't report a trustworthy aggregate. That metrics gap is itself a finding: if you run Agno Teams, instrument member token usage yourself.
Agno has the most opinionated, batteries-included team model, and it produces correct results, but it was the slowest here and the hardest to measure. OpenAI's agents-as-tools and Pydantic's delegation are leaner mechanisms that happened to be faster on this task.
Durability: the part everyone gets wrong
Every framework here claims "durable execution." The claim is close to meaningless until you ask which failure:
- Your orchestrator process dies (a deploy, an OOM, a pod reschedule). The container may be fine; the thing driving it is gone.
- The sandbox/container itself is lost (it crashed, hit a wall-clock cap, got reaped). The files the agent built die with it.
Most marketing is about failure 1. The one that bites long-running coding and migration agents is failure 2. I tested both.
What I actually ran, not what the docs promise:
| Framework | What I destroyed | How it recovered | Outcome |
|---|---|---|---|
| OpenAI v2 | the container (shutdown()) | persist_workspace() → fresh sandbox → hydrate_workspace() | 30/30 in a brand-new container ✓ |
| OpenAI v2 | the host process | RunState serialization | documented capability; I did not run this one |
| Pydantic AI + DBOS | the host process mid-workflow (SIGKILL after step 1) | DBOS checkpoints + restart | resumed from the last completed step, proven across two PIDs ✓ |
| Pydantic AI (no DBOS) | a new agent object (same process) | reload message_history | resumed with context — but this is not a real cross-process crash (DBOS above is) |
| Agno | the host process | reload session by session_id from SQLite | recovered prior messages + recalled a planted fact ✓ |
| Agno | the container | session DB holds reasoning, not files | workspace gone ✕ |
I tested the durable-execution paths directly, not just from docs. Pydantic AI + DBOS is the strongest process-crash story: I wrapped an agent run as a DBOS workflow with checkpointed steps, SIGKILLed the process after step 1, and on restart DBOS recovered and continued from the last completed step (step-level execution replay, not message re-feeding). Agno resumes context for a new turn (its session DB), not execution. So both survive a process crash, at different tiers. The honest asterisk: DBOS was proven on local SQLite single-executor, not Postgres/multi-worker. (Pydantic durable execution, Agno sessions)
Bottom line: only OpenAI v2 recovers the workspace after container loss natively. For process-crash recovery I proved two of three: Pydantic AI + DBOS (execution replay) and Agno (context replay). OpenAI v2 offers RunState serialization for the same, which I did not benchmark here.
Does Pydantic AI's loop control help?
Yes, for a specific class of problems. agent.iter() exposes every node of the run (UserPromptNode → ModelRequestNode → CallToolsNode → … → End), so you can observe, log, checkpoint, or interrupt at any step. That is the finest-grained control of the three and the natural foundation for human-in-the-loop, mid-task interruption, and checkpoint/resume of the reasoning.
But loop control is orthogonal to execution-environment durability. Pausing between nodes does nothing to recover a workspace whose container died. If the sandbox vanishes, loop control doesn't help; hydrate_workspace does. Loop control buys orchestration, observability, and host-crash resume, not sandbox durability.
A note on type-safety (it is not a differentiator)
Type-safety gets sold as Pydantic AI's edge. It isn't, anymore. OpenAI's Agents SDK is itself built on Pydantic, output_type= gives validated structured outputs and tool arguments are Pydantic-validated. Agno has native structured outputs too. All three give you typed, validated I/O. Pydantic AI's genuine edge is the surrounding application-architecture ergonomics and dependency injection, not type-safety per se.
Isolation
The host-only canary secret was not visible to code inside the sandbox in any of the three, Daytona is a separate container, so the host environment never reaches it. The architectural difference is where the loop runs: in OpenAI v2 the loop executes in the sandbox, so host credentials stay on the host by construction (OpenAI calls this separating harness from compute); in Agno and Pydantic AI the loop runs in your host process and only the executed code is isolated. Same canary result today, different blast radius if the host process itself is compromised.
The full feature matrix (2026)
Support level + a terse note, from current official docs.
| Dimension | OpenAI Agents SDK | Agno + AgentOS | Pydantic AI |
|---|---|---|---|
| Model support | OpenAI-first; others via LiteLLM | agnostic, 40+ providers | agnostic |
| Languages | Python + TypeScript | Python only | Python only |
| Native sandbox / code exec | loop in sandbox, 7 providers | tool (Daytona, E2B) | tool; "Code Mode" via Rust Monty |
| Durable execution | partial native + Temporal | DB-continuity only | Temporal, DBOS, Prefect, Restate |
| Type-safety / structured outputs | yes (native output_type) | yes (native) | yes (Pydantic-native) |
| Subagents / teams | handoffs, agents-as-tools | Teams, leader-routed | partial (delegation, A2A) |
| MCP | yes | yes (can be an MCP server) | full (client+server, OAuth) |
| Guardrails | custom tripwires | built-in (PII, injection) | output validators + ModelRetry |
| Long-term memory + RAG | partial (no vector DB) | 25+ vector DBs, Knowledge Protocol | partial (via tools/MCP) |
| Human-in-the-loop | tool-approval pause/resume | 4 HITL flows, DB-persisted | yes (pairs with durable exec) |
| Observability | built-in spans | OTel + AgentOS UI | OTel + first-party Logfire |
| Evals | hosted Evals sunset Nov 2026 | accuracy/perf/reliability | first-party pydantic-evals |
| Multimodal | vision + audio | text/image/audio/video in+out | input only |
| Voice / realtime | gpt-realtime-2, WebRTC/SIP | model-level audio | no |
| Deployment / control plane | SDK lib; AgentKit partly discontinued | AgentOS runtime + control plane + RBAC | AG-UI + managed Gateway |
| License / stars | MIT, ~27k | Apache-2.0, ~41k | MIT, ~18k |
Sources: OpenAI Agents SDK, Agno, Pydantic AI, OpenAI sandbox guide, AgentOS.
Beyond the sandbox: MCP, skills, memory, and the rest
The benchmark only stressed one thing: sandboxed coding. But most real agents lean on capabilities the benchmark never touched, and those are what actually decide a framework for a given build. The rows below are pulled from current official docs, not all of them were benchmarked, so treat this as a capability map rather than measured results.
| Capability | OpenAI Agents SDK | Agno | Pydantic AI |
|---|---|---|---|
| MCP support | yes | yes (can be an MCP server) | full (client+server, OAuth) |
| Skills / progressive disclosure | yes (model-native) | via tools | via tools |
| Short-term memory / sessions | yes | yes (session DB) | message history |
| Long-term memory + RAG / vector DBs | partial (no vector DB) | 25+ vector DBs, Knowledge Protocol | partial (via tools/MCP) |
| Guardrails | custom tripwires | built-in (PII, injection) | output validators + ModelRetry |
| Human-in-the-loop / approvals | tool-approval pause/resume | 4 HITL flows, DB-persisted | yes (pairs with durable exec) |
| Observability / tracing | built-in spans | OTel + AgentOS UI | OTel + first-party Logfire |
| Evals | hosted Evals sunset Nov 2026 | accuracy/perf/reliability | first-party pydantic-evals |
| Voice / realtime | gpt-realtime-2, WebRTC/SIP | model-level audio | no |
| Multimodal | vision + audio | text/image/audio/video in+out | input only |
| Streaming | yes | yes | yes |
The standouts cluster cleanly. Agno owns built-in RAG/memory, guardrails, and HITL: the batteries you'd otherwise wire yourself. Pydantic AI owns full MCP (client+server, OAuth) plus first-party pydantic-evals and Logfire. OpenAI owns native voice/realtime, model-native skills, and the only first-class TypeScript SDK.
Which one should you use?
| Use case | Pick | Why |
|---|---|---|
| Long-horizon coding / migration | OpenAI v2 | only native container-loss durability; context compaction keeps cost bounded |
| Voice / realtime | OpenAI v2 | native gpt-realtime-2, interruption, WebRTC/SIP |
| TypeScript / polyglot | OpenAI v2 | only one with a first-class TS SDK |
| RAG / knowledge-heavy | Agno | built-in memory + Knowledge Protocol over 25+ vector DBs |
| Multi-agent teams | OpenAI v2 or Agno | in our benchmark OpenAI's agents-as-tools was fastest + cleanest (18/18, 81s, first try); Agno has the most built-in Team abstraction but was slowest (552s) |
| Enterprise (approvals, audit, self-host) | OpenAI v2 or Agno | Agno has the built-in control plane + RBAC + HITL; OpenAI v2 fits if you're OpenAI-standardized and build the governance layer yourself |
| Analytics / NL-to-SQL / short-lived | OpenAI v2 or Pydantic AI | no durability exposure; both fit, pick on ergonomics/cost (Agno too if you want batteries) |
| Model-agnostic / cost-sensitive | Pydantic AI | leanest tokens here; model-agnostic |
| Conversational chatbot | OpenAI v2 | model-native loop, voice-ready; Agno runner-up |
| Agentic RAG (multi-step retrieval) | Agno | memory + vector DBs + teams |
| Workflow automation / ETL | Pydantic AI | explicit loop + Temporal/DBOS replay |
| Customer support / triage (HITL + audit) | Agno | built-in guardrails, HITL, RBAC |
| Deep research / web synthesis | OpenAI v2 or Pydantic AI | OpenAI has built-in web search/fetch; Pydantic for lean multi-step iteration |
| Document extraction / classification | any (pick on cost) | structured outputs are universal here, not a differentiator; Pydantic is cheapest, Agno has batteries |
| Tool/API orchestration, MCP-heavy | Pydantic AI | full MCP client+server, OAuth |
| Autonomous long-running (hours/days) | OpenAI v2 or Pydantic AI | Pydantic for durable workflow crash-replay (Temporal/DBOS); OpenAI v2 if the long run touches a sandbox/workspace (survives container loss) |
| Batch / high-throughput, cost-sensitive | Pydantic AI | leanest tokens |
| Computer-use / browser automation | OpenAI v2 | sandbox-as-substrate |
| Prototyping / fastest-to-working | Agno | batteries-included |
| Eval / testing agents | Pydantic AI | first-party pydantic-evals |
| Security / red-team agents | Agno | built-in PII/injection guardrails + HITL |
Short-lived agents are the easy case. A query that finishes in seconds can't lose a container, so none of the durability machinery earns its keep. Pick on cost and ergonomics.
Gotchas (all real, all hit during the runs)
| Gotcha | Detail |
|---|---|
| OpenAI v2 snapshot only covers the declared root | model cwd ≠ root by default; my first run persisted an empty tar. Pin the manifest root. |
| Agno tools reject invented kwargs | model called run_shell_command(working_directory=…) → TypeError mid-run; recovered but cost tokens |
| Daytona base image has no pytest | pip install it in-sandbox before agent + scorer (same prep for all three) |
Pydantic AI usage is now a property | deprecation warning if you call it as a method |
Limitations
gpt-5for all three keeps the model constant but slightly favors OpenAI v2's model-native harness.- One bounded coding task per round. A heavier, open-ended project might widen the spread.
One to watch: Vercel Eve
While I was finishing this, Vercel shipped Eve (repo), an open-source agent framework. It does not belong in the apples-to-apple table: it is TypeScript, defaults to Claude through AI Gateway, runs its own sandbox, and at the time of writing is an open preview (v0.11.3, "APIs may change before GA"). But it is the most interesting new shape, so here is where it fits.
I scaffolded one with npx eve@latest init. An agent is just a folder: one file for the model, one for the system prompt, and optional folders for tools, subagents, scheduled jobs, and channels like Slack or Discord. The pitch is "Next.js for agents" — you write what the agent does, and the framework handles the production plumbing.
What stands out is what Eve bundles natively, the things the other three make you assemble or split across systems.
| Capability | Eve | Closest equivalent elsewhere |
|---|---|---|
| Durable workflow + checkpointing | built-in | Pydantic AI + DBOS/Temporal; OpenAI v2 + Temporal |
| Isolated sandbox for agent code | built-in | OpenAI v2's substrate (Eve runs its own) |
| Subagents | built-in folder | Agno Teams; OpenAI handoffs |
| Human-in-the-loop approval | built-in | Agno HITL |
| OTel tracing + evals | built-in | Pydantic (Logfire); Agno |
| Channels (Slack/Discord/GitHub) | built-in | not built-in elsewhere (custom integration / adapters) |
So Eve is closest in spirit to Agno's batteries-included philosophy, but TypeScript-first, Vercel-native, with durability and channels as first-class primitives. For a TypeScript team that wants the plumbing handled, it is one of the most direct answers here.
So I gave it the real benchmark. Same ledger task, the same locked 30-test suite, gpt-5, scored by the same pytest run. Eve defaults to Claude through Vercel's AI Gateway, which wants a Vercel account I had not set up, so getting to a first run took more setup than the others, which started on just an API key. I pointed it at gpt-5 by hand, gave it write_file and run_shell tools, and let its own runner (eve eval) drive the build. It passed the whole suite on the first try.
| Metric | Eve (gpt-5) | for reference: the Python three |
|---|---|---|
| Tests passed | 30/30 | 30/30 each |
| Tokens | 268,876 | 65K–171K (median) |
| Wall clock | 68.9s | 79–128s |
| Tool calls | 44 | 7–25 turns |
Eve resolved the same task correctly, which is the headline: a brand-new TS framework holds its own on real code. It spent more tokens than the Python three (269K vs their 65K–171K), partly because it made more, smaller tool calls (44), and partly because, like Agno, it has no aggressive context trimming on by default. Read this as one run, not a 3-trial median, and on a different harness: TypeScript, and I ran its tools locally rather than in Daytona, so it is comparable on capability and rough cost but not a controlled head-to-head on durability or isolation.
Caveats, stated plainly: it is an open preview, TypeScript-only, oriented toward deploying on Vercel, and this run used gpt-5 wired in by hand rather than its default Claude path. It is the one I would watch.