DexNinja: Learning Robust Dexterous Cutting Policy with a Real-to-Sim-to-Real Data Engine

Abstract

Cutting a piece of food looks effortless in human hands. For a robot, every cut is a contact-rich interaction with a deformable object whose behavior changes with pose, material properties, and cutting stage. Real-world data collection is expensive and destructive, while simulation must bridge gaps in both physics and perception. We present DexNinja, a real-to-sim-to-real framework that turns a small set of real demonstrations into a dense, physically grounded training distribution. DexNinja reconstructs food instances from real captures, generates category-consistent variations in geometry and appearance, and randomizes physically meaningful parameters. A differentiable cutting simulator produces feasible trajectories and aligned visual, tactile, and proprioceptive observations. An ACT-style policy separates arm and hand control and adapts its action horizon to the manipulation stage. Across four food categories, combining real demonstrations with simulated data improves real-world cutting success from 31% to 50% in the main experiment. A separate scaling study reaches 63% with 300 simulated episodes. These results demonstrate the value of data densification for generalization to unseen instances within a food category.

Real2Sim2Real pipeline: Dexninja

DexNinja overview: real-world teleoperation and multimodal sensing, differentiable cutting simulation, and knowledge-guided generation of category-consistent 3D food assets. — The real-to-sim-to-real data engine, from the paper. Select any method figure to view it at full size.

Stage I: Teleoperation and Object Frame

Record synchronized vision, touch, and robot state. Reconstruct the food at metric scale and register the object and knife in a shared coordinate frame.

Stage II: Differentiable Cutting Simulator Details

Vary geometry, appearance, and material parameters within the food category. Refine cutting trajectories and retain rollouts that pass feasibility and cutting checks.

Training Details

Train on real and simulated episodes. Separate arm and hand prediction, then use shorter action chunks when tactile feedback matters most.

Continuation on a Shape Manifold for Data Augmentation

Successful demonstrations define the starting point for local shape variation and motion correction.

Local continuation along a shape manifold: start from a real observation, approximate its action neighborhood, fit a local corrective policy, and sample nearby food instances. — Local continuation adapts a successful motion to nearby instances on a category-specific shape family.

Knowledge-Guided Data Augmentation

Category and attribute descriptions guide the generation of new food assets. Candidates are checked for semantic and multi-view consistency, metric scale, valid geometry, and stable simulation.

The asset pipeline yields 1,200 accepted assets across strawberry, apple, banana, and potato, from 1,670 proposals. Simulated episodes also pass inverse-kinematics, collision, and cutting-success checks.

Strawberry appearance and shape variations alongside six real-world strawberry varieties. — Variation in shape, color, texture, and appearance within a food category.

Simulated AssetsDexNinja dataset on Hugging Face

Stage-conditioned action chunking

Separate ACT-style transformer policies predict arm and dexterous-hand actions, with variable action chunk sizes and a real-world execution sequence. — Separate arm and hand prediction with stage-conditioned, adaptive action chunking.

64Approach & grasp

128Knife positioning

16Tactile-guided cutting

Nominal chunk lengths in action steps; shorter chunks allow more frequent feedback during cutting.

Real-world strawberry and apple cutting paired with tactile vector fields showing how contact changes over time. — Touch reveals changing contact during deformation and cutting.

Tactile Simulation During Cutting

Tactile observations inform contact-sensitive hand control. In the matched-budget ablation, removing tactile observations reduces food integrity from 60% to 45% and cutting success from 50% to 43%.

Local cutting dynamics and the articulated robot simulator are connected through the knife: its trajectory and contact wrench form the interface between the two simulation layers.

Experimental Evaluation

Four food categories. Twenty-five held-out trials per category and training regime.

50%

Main experiment success

+19 pp

Over real-only training

63%

Separate 300-episode scaling study

Success: the cut fully traverses the food, producing separable halves with a safe termination.

Sim-only

34%34 / 100

Real-only

31%31 / 100

Sim + Real DexNinja

50%50 / 100

Sim + Real improves success by 19 percentage points over Real-only across all four foods.

Main protocol: 5 real demonstrations and 50 simulated sequences per instance. Each regime is evaluated over 100 real-world trials.

View the complete main-results table

Real-world results, 25 trials per food. Each cell reports Progress / Success / Integrity.
Food	Sim-only	Real-only	Sim + Real
Strawberry	10 / 6 / 10	12 / 8 / 10	19 / 12 / 14
Apple	5 / 4 / 5	6 / 4 / 4	14 / 8 / 7
Banana	16 / 12 / 14	17 / 12 / 14	24 / 18 / 23
Potato	15 / 12 / 11	9 / 7 / 9	19 / 12 / 16
Total / 100	46 / 34 / 40	44 / 31 / 37	76 / 50 / 60

Simulation scaling and generation cost

With the real dataset fixed, success improves as more simulated episodes are added.

36%10

50%50

56%100

63%300

Added simulated episodes · 100 real-world trials per setting

Component ablations and failure analysis

Matched-budget component ablations isolate the value of refinement, touch, and asset diversity.

Component ablation success rates across 100 trials
Variant	Success
Full DexNinja	50%
Without quality control	46%
Without physical randomization	44%
Without tactile observations	43%
Without generated assets	42%
Without trajectory refinement	39%

Complex Motion and Heterogeneous Objects

Repeated cutting introduces new geometry, new contacts, and accumulated error.

Real-world lemon experiments: a single cut succeeds in 8 of 10 trials, two cuts in 7 of 10, and multiple cuts in 3 of 10, with hand adjustment between cuts. — Lemon experiments: 8/10 for a single cut, 7/10 for two cuts, and 3/10 for multiple cuts.

BibTeX

@inproceedings{lou2026dexninja,
  title     = {{DexNinja}: Learning Robust Dexterous Cutting Policy
               with a {Real-to-Sim-to-Real} Data Engine},
  author    = {Lou, Haozhe and Yang, Runyi and Zhong, Wanqi and
               Liu, Caiyun and Liu, Yicheng and Liu, Wenhao and
               Ma, Wenlong and Xia, Jiawei and Wu, Kanzhi and
               Paudel, Danda Pani and Van Gool, Luc and
               Zhao, Hang and Li, Yiming},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}

TL;DR