Mustafa Mukadam
I am a Staff Research Scientist at FAIR (Meta), where I lead the manipulation and tactile sensing group.
I am interested in robotics, machine learning, and AI. Prior to joining FAIR, I received a Ph.D. in Robotics from Georgia Tech where I was advised by Byron Boots.
Email / Google Scholar / GitHub / YouTube / Twitter
News
[2024/04] Invited talk in the Robotics Colloquium at UW
[2023/11] Guest lecture, MIT 16.485 - Visual Navigation for Autonomous Vehicles, host Luca Carlone
[2023/09] Serving as ICRA 2024 Associate Editor (Manipulation and Grasping)
[2023/05] Tactile Diffusion wins Best Paper Award at ICRA 2023 Workshop on Effective Representations, Abstractions, and Priors for Robot Learning (RAP4Robots)
[2023/05] Two invited talks at ICRA 2023 workshops: Distributed Graph Algorithms for Robotics and 2nd Workshop on Compliant Robot Manipulation
[2023/03] Invited talk in the robotics seminar at University of Toronto
[2022/11] Guest lecture, Learning to Plan, CMU 16-831: Statistical Techniques in Robotics, host David Held
[2022/07] Theseus: A library for differentiable nonlinear optimization (NeurIPS 2022) released; Meta AI Blog, Twitter
[2022/03] Invited talk in the Robotics Seminar at Cornell
Publications
Preprint
A Touch, Vision, and Language Dataset for Multimodal Alignment
Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, Ken Goldberg
arXiv:2402.13232, 2024
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
Sudharshan Suresh, Haozhi Qi, Tingfan Wu, Taosha Fan, Luis Pineda, Mike Lambeta, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess, Joseph Ortiz, Mustafa Mukadam
arXiv:2312.13469, 2023
Perceiving Extrinsic Contacts from Touch Improves Learning Insertion Policies
Carolina Higuera, Joseph Ortiz, Haozhi Qi, Luis Pineda, Byron Boots, Mustafa Mukadam
arXiv:2309.16652, 2023
arxiv | code | video | twitter
Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Models
Carolina Higuera, Byron Boots, Mustafa Mukadam
arXiv:2304.01182, 2023
Best Paper Award, ICRA 2023 Workshop on Effective Representations, Abstractions, and Priors for Robot Learning (RAP4Robots)
2023
TaskMet: Task-Driven Metric Learning for Model Learning
Dishank Bansal, Ricky T. Q. Chen, Mustafa Mukadam, Brandon Amos
Neural Information Processing Systems (NeurIPS), 2023
USA-Net: Unified Semantic and Affordance Representations for Robot Memory
Benjamin Bolte, Austin Wang, Jimmy Yang, Mustafa Mukadam, Mrinal Kalakrishnan, Chris Paxton
International Conference on Intelligent Robots and Systems (IROS), 2023
Decentralization and Acceleration Enables Large-Scale Bundle Adjustment
Taosha Fan, Joseph Ortiz, Ming Hsiao, Maurizio Monge, Jing Dong, Todd Murphey, Mustafa Mukadam
Robotics: Science and Systems (RSS), 2023
Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing
Carolina Higuera, Siyuan Dong, Byron Boots, Mustafa Mukadam
International Conference on Robotics and Automation (ICRA), 2023
Neural Grasp Distance Fields for Robot Manipulation
Thomas Weng, David Held, Franziska Meier, Mustafa Mukadam
International Conference on Robotics and Automation (ICRA), 2023
2022
Theseus: A Library for Differentiable Nonlinear Optimization
Luis Pineda, Taosha Fan, Maurizio Monge, Shobha Venkataraman, Paloma Sodhi, Ricky T. Q. Chen, Joseph Ortiz, Daniel DeTone, Austin Wang, Stuart Anderson, Jing Dong, Brandon Amos, Mustafa Mukadam
Neural Information Processing Systems (NeurIPS), 2022
arxiv | webpage | code | twitter
MidasTouch: Monte-Carlo inference over distributions across sliding touch
Sudharshan Suresh, Zilin Si, Stuart Anderson, Michael Kaess, Mustafa Mukadam
Conference on Robot Learning (CoRL), 2022
Oral, 6.5% Acceptance Rate
arxiv | webpage | code | video
In-Hand Gravitational Pivoting Using Tactile Sensing
Jason Toskov , Rhys Newbury, Mustafa Mukadam, Dana Kulic, Akansel Cosgun
Conference on Robot Learning (CoRL), 2022
iSDF: Real-Time Neural Signed Distance Fields for Robot Perception
Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, Mustafa Mukadam
Robotics: Science and Systems (RSS), 2022
arxiv | webpage | code | video
PatchGraph: In-hand tactile tracking with learned surface normals
Paloma Sodhi, Michael Kaess, Mustafa Mukadam, Stuart Anderson
International Conference on Robotics and Automation (ICRA), 2022
2021
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra
Neural Information Processing Systems (NeurIPS), 2021
Spotlight, 3% Acceptance Rate
No RL, No Simulation: Learning to Navigate without Navigating
Meera Hahn, Devendra Chaplot, Shubham Tulsiani, Mustafa Mukadam, James Rehg, Abhinav Gupta
Neural Information Processing Systems (NeurIPS), 2021
A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation
Bernardo Aceituno, Alberto Rodriguez, Shubham Tulsiani, Abhinav Gupta, Mustafa Mukadam
Conference on Robot Learning (CoRL), 2021
LEO: Learning Energy-based Models in Graph Optimization
Paloma Sodhi, Eric Dexheimer, Mustafa Mukadam, Stuart Anderson, Michael Kaess
Conference on Robot Learning (CoRL), 2021
arxiv | video | webpage | code
Taskography: Evaluating robot task planning over large 3D scene graphs
Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vineet, Mustafa Mukadam, Liam Paull, Florian Shkurti
Conference on Robot Learning (CoRL), 2021
Revitalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation
Taosha Fan, Kalyan Vasudev Alwala, Donglai Xiang, Weipeng Xu, Todd Murphey, Mustafa Mukadam
International Conference on Computer Vision (ICCV), 2021
arxiv | video | webpage | code
Where2Act: From Pixels to Actions for Articulated 3D Objects
Kaichun Mo, Leonidas Guibas, Mustafa Mukadam, Abhinav Gupta, Shubham Tulsiani
International Conference on Computer Vision (ICCV), 2021
Joint Sampling and Trajectory Optimization over Graphs for Online Motion Planning
Kalyan Vasudev Alwala, Mustafa Mukadam
International Conference on Intelligent Robots and Systems (IROS), 2021
arxiv | video | webpage | code
Learning Tactile Models for Factor Graph-based Estimation
Paloma Sodhi, Michael Kaess, Mustafa Mukadam, Stuart Anderson
International Conference on Robotics and Automation (ICRA), 2021
arxiv | video | webpage | code
Batteries, camera, action! Learning a semantic control space for expressive robot cinematography
Rogerio Bonatti, Arthur Bucker, Sebastian Scherer, Mustafa Mukadam, Jessica Hodgins
International Conference on Robotics and Automation (ICRA), 2021
2020
Neural Dynamic Policies for End-to-End Sensorimotor Learning
Shikhar Bahl, Mustafa Mukadam, Abhinav Gupta, Deepak Pathak
Neural Information Processing Systems (NeurIPS), 2020
Spotlight, 3% Acceptance Rate
RMPflow: A Geometric Framework for Generation of Multi-Task Motion Policies
Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, Nathan Ratliff
Transactions on Automation Science and Engineering (T-ASE), 2020
Invited Paper
Encoding Physical Constraints in Differentiable Newton-Euler Algorithm
Giovanni Sutanto, Austin Wang, Yixin Lin, Mustafa Mukadam, Gaurav Sukhatme, Akshara Rai, Franziska Meier
Learning for Dynamics and Control (L4DC), 2020
Differentiable Gaussian Process Motion Planning
Mohak Bhardwaj, Byron Boots, Mustafa Mukadam
International Conference on Robotics and Automation (ICRA), 2020
2019
Riemannian Motion Policy Fusion through Learnable Lyapunov Function Reshaping
Mustafa Mukadam, Ching-An Cheng, Dieter Fox, Byron Boots, Nathan Ratliff
Conference on Robot Learning (CoRL), 2019
Learning Reactive Motion Policies in Multiple Task Spaces from Human Demonstrations
Asif Rana*, Anqi Li*, Harish Ravichandar, Mustafa Mukadam, Sonia Chernova, Dieter Fox, Byron Boots, Nathan Ratliff
Conference on Robot Learning (CoRL), 2019
Multi-Objective Policy Generation for Multi-Robot Systems Using Riemannian Motion Policies
Anqi Li, Mustafa Mukadam, Magnus Egerstedt, Byron Boots
International Symposium on Robotics Research (ISRR), 2019
Structured Learning and Inference for Robot Motion Generation
Mustafa Mukadam
Ph.D. Dissertation, Georgia Institute of Technology, 2019
Online Motion Planning Over Multiple Homotopy Classes with Gaussian Process Inference
Keshav Kolur*, Sahit Chintalapudi*, Byron Boots, Mustafa Mukadam
International Conference on Intelligent Robots and Systems (IROS), 2019
Joint Inference of Kinematic and Force Trajectories with Visuo-Tactile Sensing
Alexander Lambert, Mustafa Mukadam, Balakumar Sundaralingam, Nathan Ratliff, Byron Boots, Dieter Fox.
International Conference on Robotics and Automation (ICRA), 2019
2018
RMPflow: A Computational Graph for Automatic Motion Policy Generation
Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, Nathan Ratliff
International Workshop on the Algorithmic Foundations of Robotics (WAFR), 2018
Learning Generalizable Robot Skills from Demonstrations in Cluttered Environments
Asif Rana, Mustafa Mukadam, Reza Ahmadzadeh, Sonia Chernova, Byron Boots
International Conference on Intelligent Robots and Systems (IROS), 2018
STEAP: Simultaneous Trajectory Estimation and Planning
Mustafa Mukadam*, Jing Dong*, Frank Dellaert, Byron Boots
Autonomous Robots (AuRo), 2018
Invited Paper
Continuous-Time Gaussian Process Motion Planning via Probabilistic Inference
Mustafa Mukadam*, Jing Dong*, Xinyan Yan, Frank Dellaert, Byron Boots
International Journal of Robotics Research (IJRR), 2018
Winner of IJRR Paper of the Year Award
Sparse Gaussian Processes on Matrix Lie Groups: A Unified Framework for Optimizing Continuous-Time Trajectories
Jing Dong, Mustafa Mukadam, Byron Boots, Frank Dellaert
International Conference on Robotics and Automation (ICRA), 2018
2017
Towards Robust Skill Generalization: Unifying Learning from Demonstration and Motion Planning
Asif Rana, Mustafa Mukadam, Reza Ahmadzadeh, Sonia Chernova, Byron Boots
Conference on Robot Learning (CoRL), 2017
Long Talk, 8% Acceptance Rate
Simultaneous Trajectory Estimation and Planning via Probabilistic Inference
Mustafa Mukadam, Jing Dong, Frank Dellaert, Byron Boots
Robotics: Science and Systems (RSS), 2017
Approximately Optimal Continuous-Time Motion Planning and Control via Probabilistic Inference
Mustafa Mukadam, Ching-An Cheng, Xinyan Yan, Byron Boots
International Conference on Robotics and Automation (ICRA), 2017
Motion Planning with Graph-Based Trajectories and Gaussian Process Inference
Eric Huang, Mustafa Mukadam, Zhen Liu, Byron Boots
International Conference on Robotics and Automation (ICRA), 2017
2016
Motion Planning as Probabilistic Inference using Gaussian Processes and Factor Graphs
Jing Dong, Mustafa Mukadam, Frank Dellaert, Byron Boots
Robotics: Science and Systems (RSS), 2016
Gaussian Process Motion Planning
Mustafa Mukadam, Xinyan Yan, Byron Boots
International Conference on Robotics and Automation (ICRA), 2016
Patent
Autonomous Vehicle Policy Generation
Mustafa Mukadam, Akansel Cosgun, Alireza Nakhaei, Kikuo Fujimura
US Patent 10,739,776 B2, 2020
Refereed Workshop
Interaction-Aware Planning via Nash Equilibria for Manipulation in a Shared Workspace
Shray Bansal, Mustafa Mukadam, Charles Isbell
ICRA Workshop on Human Movement Science for Physical Human-Robot Collaboration, 2019
Tactical Decision Making for Lane Changing with Deep Reinforcement Learning
Mustafa Mukadam, Akansel Cosgun, Alireza Nakhaei, Kikuo Fujimura
NIPS Workshop on Machine Learning for Intelligent Transportation Systems, 2017
Workshops
Pretraining for Robotics (ICRA 2023)
Organizers: Rogerio Bonatti (Microsoft), Sai Vemprala (Scaled Foundations), Mustafa Mukadam (FAIR), Luis Figueredo (TU Munich), Antonio Loquercio (Berkeley), Xingyu Liu (CMU), Valts Blukis (NVIDIA), Huang (Raven) Huang (Berkeley)
Good Citizens of Robotics Research (RSS 2020)
Organizers: Mustafa Mukadam (FAIR), Nima Fazeli (University of Michigan), Niko Sünderhauf (Queensland University of Technology)
Organizers: Arunkumar Byravan (DeepMind), Markus Wulfmeier (DeepMind), Franziska Meier (FAIR), Mustafa Mukadam (FAIR), Nicolas Heess (DeepMind), Angela Schoellig (UToronto), Dieter Fox (UW / NVIDIA)
Imitation Learning and its Challenges in Robotics (NeurIPS 2018)
Organizers: Mustafa Mukadam (Georgia Tech), Sanjiban Choudhury (University of Washington), Siddhartha Srinivasa (University of Washington)
Organizers: Mustafa Mukadam (Georgia Tech), Arunkumar Byravan (University of Washington), Byron Boots (Georgia Tech)