Double-T DINER

Deploying Improvements in New Environments via Refined Tool Transfer
Henry Tsai*, Leo Huang*, Denae Cantrell*, Narek Harutyunyan, Hunter Tsai, Maggie Dong, David Jo, Claire Tomlin, S. Shankar Sastry
University of California, Berkeley
*Equal contribution
Paper Code coming soon BibTeX
Field trial · Environment A
01 / 05
Bucket, all five subtasksH2 G8 · Qwen · 8×
0 trials
real-world field trials on a Spot quadruped with a Spot Arm
+0%
mean gain in subtask completion on two-object pick-and-place
0 episodes
simulated across three hill-climb campaigns
R² 0
between simulated and real-world toolset performance
Overview

Improve the robot agent's tools in simulation, then test them in the field.

Agents can now write their own tools, but there is limited research on how an agent should empirically choose which tool edits improve task performance the most. We study this for loco-manipulation: a Spot quadruped must find a payload on a table and walk it to a cabinet to drop it off.

Read the abstract

Agentic robots powered by frontier LLMs are increasingly capable of solving complex physical tasks. Tool-calling is an important method for agentic robot control: agents can now write their own tools, but there is limited research on how an agent should empirically choose which tool edits improve task performance the most.

We study this for loco-manipulation with a Boston Dynamics Spot quadruped carrying a 6-DoF Spot Arm, tasked with finding a payload on a table and walking it to a cabinet to drop it off. A refiner agent hill-climbs the robot agent's toolset through local beam search in simulation, statistically gating every proposed edit against a control before it survives to the next generation.

We run three search campaigns and evaluate the resulting toolsets in 220 real-world field trials. Tools refined in simulation transfer: we measure a mean gain of 23% in subtask completion on a pick-and-place task with two objects, and an R² of 0.33 between simulated and real-world toolset performance.

Top row: a successful episode in Environment A. Bottom row: in Environment B, the robot grasps the bucket and avoids obstacles but cannot find the target chair, and places the bucket on its own back.

Bucket pick-and-place with Spot. Top: a successful episode in Environment A. Bottom: in Environment B, the robot grasps the bucket and avoids obstacles, but cannot find the target chair and places the bucket on its own back.

01 · Iterate

Propose tool edits

A refiner agent reads failure summaries from simulated episodes and proposes up to M edited copies of the robot agent's prompt and tool files per generation.

02 · Evaluate

Gate each edit

Every candidate runs on fixed seeds against a control. A one-sided paired sign-flip test decides whether the mean improvement is real; the best K survive.

03 · Deploy

Check in the field

Statistically significant toolsets are promoted to field trials on the real Spot, with no real-world tuning of the tools in between.

Method

A refiner agent hill-climbs the toolset with local beam search.

The robot agent is given a system prompt describing the Spot and Spot Arm embodiment, plus high-level tools that call the low-level Spot SDK. Only the prompt and the tools change between generations.

Stage 01 · Tool iteration

Search in simulation.

  1. Candidates. The refiner, an Opus 5.5 Ultracode session in Claude Code, proposes up to M edits to the toolset per generation; each candidate is one edited copy of the prompt and tool files.
  2. Search vs. held-out seeds. Every candidate runs on a fixed seed set that randomizes scenery and payload position: R search seeds the refiner can read, V held-out validation seeds it cannot.
  3. Prune and repeat. Low performers are pruned to the best K candidates, and the cycle repeats for G generations.
  4. Guardrails. The refiner may not hard-code constants or edit the robot harness or low-level API.
Stage 02 · Tool evaluation

Test for a real improvement.

  1. Dense score. Each of the N subtasks earns one point for completion and up to one more for speed against a target time, averaged into a dense score S per episode.
  2. Paired difference. For each seed i, di = Sicand − Sictrl; di > 0 means the candidate improved.
  3. Sign-flip test. A one-sided paired sign-flip permutation test asks whether the mean difference is positive, with the exact p-value in closed form.
  4. Field check. Significant toolsets are periodically promoted to real-world trials on the Spot.
Dense score, per episode

cj = 1 if subtasks 1…j all completed; tj is the time subtask j took; Tj is its target time.

Sign-flip permutation p-value

The fraction of the 2n sign-flips of the n paired differences whose sum is at least the observed sum.

What the search looks like.

Campaign H2, generations 0 to 2, from our run dashboard.

Dashboard view of campaign H2's first three generations: one card per candidate toolset with its success rate, two score bars, seed count and screen and confirm p-values; lines connect each candidate to its parent.

Each card is one candidate toolset, named for its edit, with its search-seed success and scores. Lines link children to parents; the dark line is the accepted lineage. A no-op control is an unchanged copy of the parent, run to measure noise. In generation 2 the edit held-find-top-by-body passed the screen (p = 0.043) and the confirmation (p = 0.011) and became the new head.

Simulation campaigns

Three campaigns, 160 toolsets written.

Each campaign searches from the baseline toolset (G0) in its own simulation setup, except for H3, which uses H1's final toolset. Low-fidelity primitive scenes drive early search; a Gaussian-splat reconstruction of our lab, built from iPhone 13 Pro LiDAR scans with the GaussGym pipeline, drives H2 and H3.

H1Primitive-geometry scene
+34%
field dense score vs. baseline · G12
12 gens5,112 episodes67 (5) toolsets

Bucket payload only, no inheritance. The first campaign.

H2Lab 3DGS scan
+5%
field dense score vs. baseline · G12
20 gens3,888 episodes50 (4) toolsets

Bucket payload; inherits H1's lessons.

H3Lab 3DGS scan
+32%
field dense score vs. baseline · G21
29 gens4,776 episodes43 (5) toolsets

Bucket and duffel bag; inherits H1's final tools as G1.

Toolsets: candidates written (field-tested in parentheses). Percentages compare each campaign's last field-tested toolset with the baseline.

Isaac Sim · 1×
H1
Primitive-geometry scene
A scripted grasp check of the bucket in the table scene, from ground truth. No agent drives this clip.
Gaussian-splat lab · 2×
H2
G12, seed 114: full mission
The robot agent picks the bucket, carries it across the reconstructed lab and sets it on the cabinet. A recorded re-run of a seed the search scored as a full success.
Gaussian-splat lab · 2×
H3
G19, seed 110: duffel bag, full mission
H3 draws the bucket or the duffel bag per seed. Here the agent picks the bag, carries it across the reconstructed lab and leaves it on the cabinet. A recorded re-run of a seed the search scored as a full success.
Field deployment

220 real-world trials on a Spot quadruped.

We ran 220 field trials in Environment A and 40 more in Environment B. Each toolset is evaluated in 20 trials, 10 carrying a bucket and 10 carrying a duffel bag, split evenly between Gemini 3.8 Live and Qwen3.8-27B as the agent model.

BaselineG0
2.15/5
mean subtasks done
H1G12
2.85/5
mean subtasks done
H2G12
2.20/5
mean subtasks done
H3G21
2.85/5
mean subtasks done

Every field-tested toolset, subtask by subtask.

Trials that passed each subtask, of 10 per object. Hover a cell for details.

010 / 10
swipe the table sideways →

Sim success (% of seeds with the full mission done) is on the 96 held-out seeds, or on the search seeds for H1 (†); G0's is averaged over the three campaigns. Field score is the mean number of subtasks done, of five, over all 20 trials. H3's G4 and G8 are one toolset, tested once as G8*. All field trials ran in Environment A.

Place succeeded once across the ten campaign toolsets, in H2's G8 with the bucket. The clearest gain is at the pick: duffel-bag picks rose from 2 of 10 for the baseline to 54 of 100 pooled across the ten campaign toolsets, likely from a rule H1 added to measure_grip that turns the gripper 90° when the vision model picks grip points at opposite ends of the handle, which H2 and H3 both copied.

Gains in the field are far smaller than gains in simulation. Every campaign's field-tested toolsets sit below the "field = sim" line, and with 20 trials per toolset, every 95% confidence interval still overlaps the baseline's.
H2 G8 · bucket
Bucket · 5 of 5 subtasks
Qwen3.8-27B
8×
H3 G21 · duffel bag
Duffel bag · 4 of 5 subtasks
Qwen3.8-27B
8×

Environment A. A large open cubicle area on carpet, reconstructed by our Gaussian splat; the trash-can obstacle and the payload pose are randomized every trial, and all 220 scored trials ran here. Left: H2's G8 walks around the trash can, re-grasps the bucket by its outer lip after two misses and carries it to the cabinet. Its gripper camera never sees the cabinet, but it places the bucket on it anyway, at 251 s: the only one of the 220 trials to pass all five subtasks. Right: H3's G21 grasps the duffel bag along its long axis on the first attempt, carries it to the cabinet and hovers it over the place spot, but keeps walking, pushes the cabinet away and times out at 300 s.

Results

Sim and field scores rise together, with the field gains much smaller.

Every toolset the refiner wrote.

Search-seed success per generation, H1 to H3.

Search-seed success per generation for campaigns H1, H2 and H3, with rejected candidates, no-op controls and the accepted lineage.

Rejected candidates are gray dots; no-op controls (crosses) are unchanged copies of their parent; colored lines trace the accepted lineage that went on to field testing. Search-seed success climbs within every campaign, from 0% at G0 to as high as 79% (H1), 54% (H2) and 100% (H3) by the final generation shown. (candidates.csv)

Field dense score vs. simulation dense score.

All 11 field-tested toolsets: the baseline and 10 campaign toolsets.

R² = 0.33
Field dense score vs. simulation dense score for every field-tested toolset across campaigns H1, H2 and H3.

Field dense scores are means with 95% bootstrap intervals; sim dense scores are split into held-out seeds (filled) and search seeds (hollow). Every campaign toolset sits below the dashed "field = sim" line. From the baseline through H1's three field-tested toolsets, every rise in simulation score came with a rise in field score. In H2 and H3, held-out simulation success rose from 0% to as high as 54% while field gains stayed modest.

Citation

BibTeX

@misc{tsai-double-t-diner-2026,
  title   = {Double-T DINER: Deploying Improvements in New
             Environments via Refined Tool Transfer},
  author  = {Tsai, Henry and Huang, Leo and Cantrell, Denae and
             Harutyunyan, Narek and Tsai, Hunter and Dong, Maggie and
             Jo, David and Tomlin, Claire and Sastry, S. Shankar},
  note    = {CoRL 2026 Workshop on Sim-to-Real-to-Field:
             The Science of Transfer},
  year    = {2026},
  url     = {https://www.alphaxiv.org/abs/2611.diner},
}

Acknowledgements

This work was performed in the FHL Vive Center for Enhanced Reality and was supported by the program "Design of Robustly Implementable Autonomous and Intelligent Machines (TIAMAT)," Defense Advanced Research Projects Agency award number HR00112490425.

We thank our collaborators at Stanford University, Lizzi Yin, Yousef Liang and Marco Pavone, for their support and collaboration. We thank Lizzi in particular for her help with the robot's run-recording tools.

We thank Rosario Scalise for the 3D scan of the duffel bag used in our simulation.

We also thank Jason Pusey, John Rogers, Sean Gart, Jayasuriya Suresh, and all of the staff at the Army Research Laboratory Robotics Research Collaboration Campus for their tireless efforts in building diverse challenges, including the bucket pick-and-place task and outdoor environments, for testing our Spot and evaluating our baseline agent.