See to Touch: Learning Tactile Dexterity through Visual Incentives
Summary
In this work, we present Tactile Adaptation from Visual Incentives (TAVI), a framework that enhances tactile-based dexterity by optimizing dexterous policies using vision-based rewards. First, we employ a novel contrastive-based objective to learn visual representations. Next, we construct a reward function using these visual representations through optimal-transport based matching on one human demonstration. Finally, we utilize online reinforcement learning on our robot to optimize tactile-based policies that maximize the visual reward. We show the results of our framework in five different tasks.
Policies
We evaluate our framework on several dexterous tasks that are difficult to solve with visual information alone.
Sponge Flipping
Eraser Turning
Mint Opening
Peg Insertion
Plier Picking
Bowl Unstacking
Generalization
We observe that our tactile-based policies are able to learn residuals for objects that are both visually and structurally different. A new residual policy is trained with each separate object. Each object is experimented following a similar experimental setting as with the original object and observed success rates are shown for each showcased object.
Bowl Unstacking
Robot rollout for bowl unstacking task on different bowls.
Peg Insertion
Robot rollout for peg insertion for different pegs and cups.
Training
Here, we showcase the training rollouts for each of our tasks. We observe that as time passes the rewards increase and the policies become more and more robust.
Long Horizon Robustness
We experimented on the robustness of TAVI policies by concatenating multiple learned policies. We observe that concatenated policies can zero-shot work with the perturbations from different tasks.
Bibtex
@misc{guzey2023touch,
title={See to Touch: Learning Tactile Dexterity through Visual Incentives},
author={Irmak Guzey and Yinlong Dai and Ben Evans and Soumith Chintala and Lerrel Pinto},
year={2023},
eprint={2309.12300},
archivePrefix={arXiv},
primaryClass={cs.RO}
}
All Training Rollouts
Here, we show all the robot experiments, trainings and evaluation rollouts for each of our tasks.