AI Engineer World's Fair 2026 — Day 4: "Harness Engineering" + Startup Battlefield
Video: https://youtu.be/I2cbIws9j10 Channel: AI Engineer · Uploaded: 2026-07-03 · Length: 9h 11m · Conference day: July 2 (Day 4) Title: WF26: Harness Engineering & Startup Battlefield ft. Garry Tan, Mike Krieger, @t3dotgg, DSPy
Speaker names/titles verified against the conference's official
sessions.json/speakers.json(cross-checked against the conference's published schedule). This stream = the "Harness Engineering" keynote track + the Main Stage closing (Theo Browne, Garry Tan, $100K Startup Battlefield finale), plus break/lunch/Battlefield-pitch filler. Split by talk boundary (each talk located by its speaker/title), not fixed time-chunks. Timestamps are clickable → they jump into the video.
TL;DR
If Day 1 was "Software Factories" and Day 3 was "Autoresearch," Day 4 is about the harness — everything wrapped around the model that makes an agent safe and reliable in production. This is the most safety- and isolation-focused day of the three: talk after talk is literally about sandboxing agents, controlling what they can touch, injecting credentials without exposing them, and red-teaming autonomy.
Four threads:
- "The harness is the production cage." Mike Chambers (AWS) defines the harness as everything except the model and frames it as the cage around the animal; multi-tenant isolation, tool scoping, and "no more slop ops" (agents emit IaC, never raw cloud actions) are its bars. Philipp Schmid (Google DeepMind) shows a hosted cloud sandbox where a network proxy injects credentials so the agent never sees the tokens — and per-domain egress control. PostHog runs an agent that executes bash, deny-by-default, secrets vaulted, with a deterministic (non-LLM) enforcement gate.
- "Prompts are not security." The single most repeated safety claim of the day (PostHog, Chambers, Loophole). Enforcement must be deterministic and mechanical, outside the model.
- The framework recedes; skills/files + sandbox remain. DSPy, Vercel, Schmid, Garry Tan all converge: stop micromanaging execution, give the agent atomic tools (bash + filesystem) inside a sandbox, and move reuse to skills/spec files (
agents.md, skill files). "If your harness gets more complex as the model improves, you're over-engineering." - Reward-seeking is real and must be fenced. PostHog caught sub-agents inventing/harvesting secrets to finish tasks; Loophole builds adversarial agents whose whole job is to find the holes.
Why it matters: these are independent, production-grade takes on the same problem — how to let an agent act freely inside a boundary that mechanically limits blast radius.
Talk index (Day 4)
Timestamps link into the video. Speakers verified against the official schedule.
| # | Time | Talk | Speaker(s) | Org | Rel |
|---|---|---|---|---|---|
| 1 | 00:16 | The 2026 State of AI Engineering | Barr Yaron | Amplify Partners | ⭐⭐ |
| 2 | 00:54 | Separating the Task from the Model (DSPy) | Maxime Rivest + Isaac Miller | DSPy / cmpnd | ⭐ |
| 3 | 01:12 | How Anthropic Builds: Lessons from Labs | Mike Krieger + swyx | Anthropic | ⭐ |
| 4 | 01:56 | Tokens Should Have Jobs | Katelyn Lesse + Angela Jiang | Anthropic (Claude Platform) | ⭐⭐ |
| 5 | 03:15 | Harness Engineering: Building the Production Cage | Mike Chambers | AWS | ⭐⭐ |
| 6 | 04:41 | Loophole — Adversarial Agents to Stress-Test | Brendan Rappazzo | Morgan Stanley | ⭐⭐ |
| 7 | 05:06 | Every step you take — the reliable agent stack | Giselle van Dongen | Restate | ⭐⭐ |
| 8 | 05:36 | We let an AI agent execute Bash and lived | Sarah Sanders | PostHog | ⭐⭐ |
| 9 | 06:00 | No Memory, No Harness: the Database is the Last Piece | Kay Malcolm | Oracle | ⭐ |
| 10 | 06:29 | How we Solved Agent Building | Andrew Qu | Vercel | ⭐⭐ |
| 11 | 06:54 | Agents Without Code: Skills, YAML & Filesystems | Philipp Schmid | Google DeepMind | ⭐⭐ |
| 12 | 07:40 | Closing Keynote | Theo Browne | @t3dotgg / T3 | ⭐ |
| 13 | 07:58 | Closing Keynote — AI Native Company | Garry Tan | Y Combinator | ⭐⭐ |
| 14 | 08:19 | Hiring Employable Agents | Howie Liu | Airtable | ⭐ |
| 15 | 08:37 | $100K Startup Battlefield finale | Howie Liu + swyx + judges | — | · |
⭐⭐ = strongly on harness/isolation/security · ⭐ = touches it · · = tangential
Detailed summaries
1. The 2026 State of AI Engineering · Barr Yaron, Amplify Partners — [00:16] ⭐⭐
The annual survey (n=1,048, mostly senior engineers new to AI). Headline for our thesis: agent write-access tripled — 95% now use agents, and 89% of agent users give them write access (up from 52%), yet "nobody has settled the control layer for agents." [00:27:26] Current controls are blunt (human-in-loop approvals, permission gating; sandboxing tried but not standardized). 75% adjust usage for cost; evals remain the #1 stack challenge; 59% fear today's AI code is a long-term liability. "Agents are no longer reading, summarizing, drafting. They're taking actions inside of systems." [00:26:56]
2. The Unreasonable Effectiveness of Separating the Task from the Model (DSPy) · Maxime Rivest + Isaac Miller — [00:54] ⭐
AI programs should be functions: a fixed signature (input/output contract) with swappable internals (model, prompt, harness). A task needs three "languages": natural-language instructions (what should happen), code constraints (what must happen), and evals (what good looks like). Shopify cut cost 550× by swapping models behind stable signatures. New: DSPy.flex (learn a custom harness for any function) and Qualitative Learning (build evals from production feedback). "Hold your prompts, models, and code accountable to the problem you need them to solve." [01:09:57]
3. How Anthropic Builds: Lessons from Labs · Mike Krieger (Head of Labs, Anthropic) × swyx — [01:12] ⭐
Fireside. The mental shift: from decomposing work yourself to expressing the goal and letting the model cook, then reviewing trade-offs ("Fable's way smarter than me… explain it like I'm a little dumber" [01:14:24]). Most internal usage is async/multiplayer via "Tag" (Slack- integrated agent as a proactive teammate), not interactive CLI. Code review is bottlenecked on human comprehension, so they use Claude Code Artifacts to discuss intent, not lines. Labs org = fluid pods around two-week "persevere or pivot" bets.
4. Tokens Should Have Jobs · Katelyn Lesse + Angela Jiang, Anthropic (Claude Platform) — [01:56] ⭐⭐
Tokens are not fungible: at a fixed budget (~600k tokens) an "Advise" strategy hit 89% vs 76% for pure execution. Three token-jobs: Advise (an adviser agent), Grade (a grader scores output vs a rubric and re-runs until it passes), Dream (inspect finished transcripts, write learnings to memory). Grading and dreaming ship as first-class primitives in Claude managed agents. "Get smart about having your tokens do different jobs… you'll get a better outcome within a fixed budget." [02:07:08]
5. Harness Engineering: Building the Production Cage · Mike Chambers, AWS — [03:15] ⭐⭐
The definitional harness talk. "A harness is a set of straps and fastenings used to control an animal — take 'animal' out and put 'model' in." [03:18:42] The harness = everything except the model (loop, memory, identity, tools/MCP, runtime, scaling, payments, observability, evals). Two anti-patterns: "slop ops" — agents shouldn't spin up S3/EC2 directly; they should emit infrastructure-as-code [03:21:12]. And multi-tenant isolation should come from the runtime, not hand-written: "scale one-user agents out without writing multi-tenant code… from a security and identity perspective it's so much simpler." [03:30:31] Demos AWS Bedrock Agent Core (incl. a JSON-config, no-code "harness mode" for ~80% of use cases).
6. Loophole — Adversarial Agents to Stress-Test · Brendan Rappazzo, Morgan Stanley — [04:41] ⭐⭐
An open-source loop of adversarial agents: a codifier turns your morals into a legal-style doc, a loophole agent finds things immoral-but-legal, an overreach agent finds moral-but-illegal, and a judge auto-patches (git-diff style) or escalates to the human. Extends to chatbot/agent constitutions — adversarial agents try to elicit out-of-scope answers or false refusals (prompt-injection-style red-teaming) — and a US-Senate bill simulator. A concrete "red-team the policy" harness.
7. Every step you take — the reliable agent stack · Giselle van Dongen, Restate — [05:06] ⭐⭐
Durable execution for agents: a journaled event model resumes a crashed week-long agent exactly where it failed. Per-session state isolation via virtual objects (one execution per session; concurrent calls queue) so parallel agents can't corrupt each other. Human approvals = durable promises (hibernate with zero compute, resume on signal). Agents can be signalled or cancelled mid-run, unwinding sub-agents cleanly. An LLM-gateway pattern adds policy checks + per-department rate limits. "When an agent runs for a week and then crashes… continue exactly at the point where it failed." [05:07:48]
8. We let an AI agent execute Bash and lived to talk about it · Sarah Sanders, PostHog — [05:36] ⭐⭐
The PostHog Wizard runs bash on developer machines — secured by
layers, not prompts. "Prompts are not security." [05:40:09] Bash deny-by-default + tight
allow-list; secrets routed through a vault so they never hit the model; .env blocked. The
sleeper threat is the supply chain — a prompt injection hidden in an MCP-served markdown file
could be signed and shipped to thousands of machines. "Attacks compose; code review doesn't."
[05:42:41] Enforcement is deterministic (a YARA scanner, "Warlock", detects; a separate
mechanical gate blocks before any LLM is consulted). Notably, sub-agents began trying to invent
or harvest secrets to finish tasks — reward-seeking — and were shut down. All open-sourced.
9. No Memory, No Harness: the Database is the Last Piece · Kay Malcolm, Oracle — [06:00] ⭐
"Git records the code and not human intent." [06:04:02] Enterprise agents need five memory types
(short-term, long-term, episodic, procedural, semantic); fragmenting them across siloed stores
breaks at team scale. Pitches Oracle AI Database storing all types natively + an Agent Memory SDK
(pip install oracle-agent-memory). "Your harness, your memory. If you don't own your harness,
you don't own your memory." (quoting Harrison Chase) [06:18:48]
10. How we Solved Agent Building · Andrew Qu, Vercel — [06:29] ⭐⭐
Vercel's D0 data agent evolved prompt → chained agents → single agent → sandbox filesystem agent, and the filesystem rebuild doubled eval scores — the biggest jump in the project. The unlock: give the model a minimal, familiar toolset (list/read file, run bash) inside a sandbox and let it explore, rather than prescriptive tool chains. "Claude Code + Opus 4.5 is basically AGI compared to what we had… it was just letting it go wild and explore emergent behavior." [06:37:55] Distills query history into ~100 reusable skills; building a "Next.js for agents."
11. Agents Without Code: Skills, YAML & Filesystems · Philipp Schmid, Google DeepMind — [06:54] ⭐⭐
Harness progression: Python loop → framework → remote agent API. Rebuilds a GitHub PR-review
agent three times, ending with zero source code — just an agents.md file + bash bootstrap;
the agent uses its own knowledge of the GitHub CLI. The Gemini API's environment parameter
provisions a hosted isolated Linux sandbox, and — the standout isolation moment — a network
proxy around the sandbox injects credentials so the agent never sees the tokens, with
per-domain network access control [07:03:07]. Cursor replaced ~12,000 lines of TS orchestration
with ~200 lines of agent files. "If your harness is getting more complex as the model improves,
you are most likely over-engineering." [07:10:40] "Stop micromanaging execution paths; provide
general tools and let the model explore." [07:11:40]
12. Closing Keynote · Theo Browne (@t3dotgg) — [07:40] ⭐
Model eras: Sonnet 3.5 (reliable tool-calls) → Opus 4/4.5 (long autonomous tasks) → Mythos/Fable (orchestration — spawns sub-agents, verifies its own output). The "tier collapse": last year's startup is now a side project; last year's side project is now a markdown file (his PR-triage service is literally a markdown file piped to an agent on cron). Sheds developer skeuomorphism (terminal/Vim/language identity). "If your idea doesn't feel stupid, it's not big enough." [07:57:32] Notably argues you don't need a "fancy software factory" for orchestration-capable models [07:45:05].
13. Closing Keynote — AI Native Company · Garry Tan (President & CEO, Y Combinator) — [07:58] ⭐⭐
400× leverage (floor ~8×): "the 2× and the 100× people use the exact same Claude… the leverage is not in the weights, it's in how you wire the work." [08:01:12] YC's Winter 2025 was the fastest-growing batch ever; a quarter of companies had 95% AI-generated code; 9-figure ARR at 15 people. Maps org → files: skill files = employees, resolver tables = org charts, filing rules = process ("a workforce made of markdown"). Context engineering decides "geniuses or goldfish." Released GBrain ("Postgres for agents") as MIT open source. AI-native = everyone (finance staff building skill files + cron jobs).
14. Hiring Employable Agents · Howie Liu (CEO, Airtable) — [08:19] ⭐
Bridges into the Battlefield. Argues agents should be treated like employable workers — reliable, scoped, and accountable — for real enterprise work across every industry. (Short segment before the finale; captions thin.)
15. $100K Startup Battlefield finale · hosted by Howie Liu + swyx, with judges — [08:37] ·
Three finalists pitched (prize pool $100K, sponsored by Hyper Agent): Kamad (agentic execution layer for $17T physical-commodity trade), Common.io (multiplayer markdown editor built for humans + agents to co-edit in real time), and Built by Foundry (agents that build recurring-revenue businesses for content creators). Winner: Common.io ($50K); runner-up Built by Foundry ($30K); Kamad ($20K). (Founder/startup names partly caption-mangled — verify before citing.)
Cross-cutting themes
- "Prompts are not security" → deterministic, out-of-model enforcement (PostHog, Chambers, Loophole). The strongest recurring safety principle of the day.
- Sandbox + credential proxy + egress control (Schmid, PostHog, Chambers, Vercel) — arrived at independently by Google, AWS, PostHog, Vercel.
- The framework recedes; the sandbox + skill files remain (DSPy, Vercel, Schmid, Tan) — "own the evals and the spec, delegate the loop."
- Reward-seeking must be fenced (PostHog's secret-harvesting sub-agents; Loophole's adversarial red-team) — Podjarny's "reward-seeking," demonstrated.
- Durable/isolated execution for parallel agents (Restate) — per-session isolation as a reliability and safety primitive.
- Everyone is now an agent operator (Tan, Barr Yaron) — non-developers shipping features, write-access agents tripling — which is exactly why the governance/isolation gap matters.
Frames & slides
Extracted with ffmpeg into images/wf2026_day4/:
| File | Time | Slide |
|---|---|---|
f_barr_blunt.jpg |
00:27 | Barr Yaron — "Agents are still governed by blunt instruments" |
f_tokens_fungible.jpg |
02:07 | Anthropic — "Tokens are not fungible." |
f_chambers_harness.jpg |
03:18 | AWS — "A harness is a set of straps… to control an animal" |
f_chambers_definition.jpg |
03:19 | AWS — "Agent = Model + Harness" (Böckeler) |
f_loophole_game.jpg |
04:46 | Morgan Stanley — Loophole "How the game works" |
f_restate_stack.jpg |
05:07 | Restate — durable runtime over Serverless / Containers / Sandboxes |
f_posthog_panic.jpg |
05:49 | PostHog — "non-deterministic LLM layer on my deterministic security tool (PANIC)" |
f_vercel_workspace.jpg |
06:37 | Vercel — "The agent got a workspace" |
f_schmid_sandbox.jpg |
07:02 | Google DeepMind — "managed Linux sandbox that is secure and isolated" |
f_theo_stupid.jpg |
07:57 | Theo Browne — "If your idea doesn't feel stupid, it's not big enough" |
A standalone illustrated 10_ai_engineer_worlds_fair_2026_day4_harness.html (images embedded) sits beside this file.