Training robot policies for general multi-task pick and place use cases requires demonstrations on many objects, which handheld Universal Manipulation Interfaces (UMIs) can collect quickly and cheaply. However, existing UMIs depend on external base stations, use rigid fingers that give no signal about contact forces, or use parallel-jaw fingers that limit grip strength.
We present a battery-powered handheld interface, built on YUBI, whose soft, angled fingers wrap around objects, deform visibly under load as a visual proxy for force, and match the grippers of the robot. With it, we collect a 99.7-hour dataset over 53 objects, train π0.5 policies on 49 of them, and hold out the other 4 as unseen objects.
We find that a top camera raises success by 13 points even when only about half of the data includes it, and that mid-training on our multi-task data raises success after fine-tuning on unseen objects from 35% to 57%. We will release the finger design, the dataset, and a simplified version of the data processing code.
Our interface builds on the YUBI hardware and software stack: Meta Quest pose tracking, aperture encoders, and recording software. Each UMI gripper carries a wrist camera and a Meta Quest 3S controller tracked by the headset, which the operator wears around the neck with a top camera on it. A portable power station runs it for about 19 hours per charge, so it needs no base stations and can be used anywhere.
We replace YUBI's rigid fingertips with 1.5× longer soft TPU ones that open up to 90°, conform to more object shapes, and deform visibly under load, a visual proxy for grasp force. Our 450 g aluminum robot gripper shares the UMI gripper's dimensions, fingertips, and camera placement, so demonstrations map one-to-one.
The state contains the relative pose between the two grippers, both apertures, and the gravity direction in each gripper frame. Actions are chunks of 30 future poses, each relative to its arm's current pose. Nothing depends on the tracking frame's origin or yaw, so no calibration to the robot is needed.
Our dataset contains 99.7 hours of demonstrations (over 51,000 episodes) over 53 objects. We mid-train the π0.5 base checkpoint on 92 hours over 49 objects, collected by 5 operators, and hold out the other 4 objects as unseen. We run 50 trials per policy and object on a Galaxea R1 Pro or R1 Lite.
Only 48 of our 92 hours include a top camera. We mid-train a single policy that blacks out the top view with probability 0.5 and evaluate it with and without the top camera. On 5 in-distribution objects on the R1 Pro, the top camera raises the mean success rate from 47% to 60%. Both settings fail most often on the bottle, where the gripper does not close tightly enough and the bottle slips out.
| Top view in training | Success rate | ||
|---|---|---|---|
| Object | (% of episodes) | With top view | Without top view |
| Peas | 100% | 78% | 54% |
| Fruit puree | 100% | 62% | 52% |
| Dried mango | 80% | 72% | 62% |
| Almonds | 34% | 70% | 64% |
| Bottle | 0% | 18% | 2% |
| Mean | – | 60% | 47% |
Table 1: Success rate of the mid-trained policy on 5 in-distribution objects, on the R1 Pro.
We fine-tune on 4 unseen objects: a box and porridge, with shapes similar to the mid-training objects, and a wire and soup, with different shapes. Starting from the π0.5 base checkpoint (PT), we compare the mid-trained policy without task-specific data (PT + MT), PT fine-tuned on one object (PT + SFT), and the mid-trained policy fine-tuned on it (PT + MT + SFT).
On the R1 Pro, mid-training raises success after SFT from 35% to 57%, mostly on objects that tolerate imprecise grasps; both policies fail on the wire. On the R1 Lite, which cannot reach most of the UMI motions and is fine-tuned on its own teleoperation data, it raises success from 80% to 89%.
Figure 2: Success rate on each out-of-distribution (OOD) object, grouped by whether its shape is similar to or different from those in the mid-training data. Left: on the R1 Pro, we compare mid-training and SFT on UMI data. Right: on the R1 Lite, we compare the same policies fine-tuned on teleoperation data (hatched). Error bars show 95% Wilson intervals.
We solve inverse kinematics for every frame and call an episode unreachable when the 95th percentile of its tip error exceeds a threshold. With whole-body inverse kinematics, the R1 Pro reaches most of the episodes we collected for the unseen objects: 12% exceed 20 mm and 1% exceed 30 mm. Freezing torso joints shrinks the workspace quickly, so the torso is essential for executing our data.
Figure 3: Share of R1 Pro episodes whose 95th-percentile IK tip error exceeds 20 mm (left) or 30 mm (right), as torso joints are frozen from the chassis (joint 1) up to the arms (joint 4).
@inproceedings{kelemen2026softfingers,
title = {Robot-Matched Soft Fingers for {UMI} Data Collection: Lessons from 100 Hours of Pick and Place},
author = {Kelemen, Kobi and Niedzia{\l}kowski, Tomasz and Paszczak, Marcin and Montesinos, Victoriano},
booktitle = {The {UMI} Arena: From Handheld Devices to Real-World Policies, Workshop at the Conference on Robot Learning (CoRL)},
year = {2026}
}