|
I am an Assistant Professor at the School of Interactive Computing at Georgia Tech and a (part-time) Research Scientist at NVIDIA AI. I work at the intersection of Robotics and Machine Learning. I received my Ph.D. in CS from Stanford University advised by Fei-Fei Li and Silvio Savarese (2015-2021) and B.S. from Columbia University (SEAS'15). I've spent time at DeepMind UK (2019), ZOOX (2017), Autodesk Research (2016), CMU RI (2014), and Columbia Robotics Lab (2013-2015). Ph.D. applicants: I'm hiring 2022 - 2023! Apply through the official application portal for the CS Ph.D. program (School of Interactive Computing) and and make sure that you mention my name in your application. Email / Google Scholar / CV (July 2022) / Github / Twitter |
|
|
My research goal is to enable physical autonomy in everyday human environments with minimum expert intervention. Towards this goal, my works draw equally from Robotics, Machine Learning, and Computer Vision, including topics such as imitation & reinforcement learning, representation learning, manipulation, and human-robot interaction. My current research focuses on visuomotor skill learning, structured world models for long-horizon planning, and data-driven approaches to human-robot collaboration. Thesis: Compositional Reasoning in Robot Learning (talk recording, slides) Talk at the Brown BigAI Seminar: Bridging Reasoning and Acting for Everyday Autonomy (recording). |
|
I direct the Robot Learning and Reasoning Lab (RL2) at Georgia Tech. The current members are:
|
|
|
|
|
|
|
|
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020) |
|
6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints (2020) |
|
|
|
Shuo Cheng, Danfei Xu Preprint 2022, CoRL LHP Workshop (Best Paper Runner Up) Keywords: Skill Learning, Task Planning TL;DR: We use Task and Motion Planning (TAMP) as guidance for learning generalizable and composable sensorimotor skills. |
|
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg ICRA 2023 Keywords: Task Planning, Manipulation, Language Model TL;DR: We harness the power of large language model to generate program-like task plans. |
|
Danfei Xu*, Yuxiao Chen*, Boris Ivanovic, Marco Pavone ICRA 2023 Keywords: Imitation Learning, Behavior Modeling TL;DR: Hierarchical imitation for generating diverse, realistic, and robust traffic behaviors. |
|
Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, Marco Pavone ICRA 2023 Keywords: Imitation Learning, Behavior Modeling TL;DR: Learn behavior prior through diffusion-based imitation. Sample desired behaviors by conditional diffusion and Signal-Temporal Logic (STL) rules. |
|
Yulong Cao, Danfei Xu, Xinshuo Weng, Zhuoqing Mao, Anima Anandkumar, Chaowei Xiao, Marco Pavone CoRL 2022 (Oral) Keywords: Adversarial Robustness, Safe AI TL;DR: Defending against attacks that target generative behavior models. |
|
Yulong Cao, Chaowei Xiao, Anima Anandkumar, Danfei Xu, Marco Pavone ECCV 2022 Keywords: Adversarial Robustness, Safe AI TL;DR: Physically-plausible attack on behavior forecasting models. |
|
Chen Wang, Danfei Xu, Li Fei-Fei Robotics and Automation Letters (RA-L) 2022 Keywords: Task Planning, World Models TL;DR: Building generalizable world models for planning with unseen objects and environments. |
|
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martin-Martin CoRL 2021 (Oral) [code+dataset][website][blogpost] Keywords: Imitation Learning TL;DR: A large-scale study on learning manipulation skills from human demonstrations. |
|
Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin, Yuke Zhu, Li Fei-Fei, Silvio Savarese Arxiv Preprint, 2021 Human-in-the-loop learning for complex manipulation tasks. |
|
Chen Wang, Claudia D'Arpino, Danfei Xu, Li Fei-Fei, Karen Liu, Silvio Savarese CoRL 2021 (to appear) Learning human-robot collaboration policies from human-human collaboration demonstrations. |
|
Chen Wang*, Rui Wang*, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, Danfei Xu IROS 2021 An learnable action space for recovering human's hand-eye coordination behaviors by learning from human demonstrations. |
|
Danfei Xu, Ajay Mandlekar, Roberto Martin-Martin, Yuke Zhu, Silvio Savarese, Li Fei-Fei (Long version) ICRA 2021 (Short version) Oral Presentation, NeurIPS Workshop on Object Representations for Learning and Reasoning, 2020 We extend the classical definition of affordance to enable generalizable long-horizon planning. |
|
Danfei Xu, Misha Denil (Long version) CoRL 2020 (Short version) Late-Breaking Paper, NeurIPS Deep Reinforcement Learning Workshop 2019 An algorithm framework that simultaneously addresses the reward delusion problem in supervised reward learning and the overfitting discriminator problem in adversarial imitation learning. |
|
Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei Juan Carlos Niebles ECCV, 2020 Learning to plan from instructional videos. |
|
Ajay Mandlekar*, Danfei Xu*, Roberto Martin-Martin, Silvio Savarese, Li Fei-Fei RSS, 2020 Learning visuomotor policies that can generalize across long-horizon tasks by modeling latent compositional structures. |
|
Chen Wang, Roberto Martin-Martin, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, Yuke Zhu ICRA, 2020 Real-time category-level 6D object tracking from RGB-D data. |
|
Danfei Xu, Roberto Martin-Martin, De-An Huang, Yuke Zhu, Silvio Savarese, Li Fei-Fei NeurIPS, 2019 A flexible neural network architecture for learning to plan from video demonstrations. |
|
De-An Huang, Danfei Xu, Yuke Zhu, Silvio Savarese, Li Fei-Fei, Juan Carlos Niebles IROS, 2019 One-shot imitation learning via hybrid neural-symbolic planning. |
|
William B. Shen, Danfei Xu, Yuke Zhu, Leonidas Guibas, Li Fei-Fei, Silvio Savarese ICCV, 2019 Learning generalizable navigation policy from mid-level visual representations. |
|
Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martin-Martin, Cewu Lu, Li Fei-Fei, Silvio Savarese CVPR, 2019 Dense RGB-depth sensor fusion for 6D object pose estimation. |
|
De-An Huang*, Suraj Nair*, Danfei Xu*, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, Juan Carlos Niebles CVPR, 2019 (Oral) Generate executable task graphs from video demonstrations. |
|
Danfei Xu*, Suraj Nair*, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, Silvio Savarese ICRA, 2018 [website] [video] [Two Minute Papers] [blog post] Neural Task Programming (NTP) is a meta-learning framework that learns to generate robot-executable neural programs from task demonstration video. |
|
Danfei Xu, Ashesh Jain, Dragomir Anguelov CVPR, 2018 End-to-end 3D Bounding Box Estimation via sensor fusion. |
|
Danfei Xu, Yuke Zhu, Christopher B. Choy, Li Fei-Fei CVPR, 2017 We propose an end-to-end model that jointly infers object category, location, and relationships. The model learns to iteratively improve its prediction by passing messages on a scene graph. |
|
Christopher B. Choy, Danfei Xu*, JunYoung Gwak*, Silvio Savarese ECCV, 2016 We propose an end-to-end 3D reconstruction model that unifies single- and multi-view reconstruction. |
 
|
Yinxiao Li , Yan Wang , Yonghao Yue , Danfei Xu, Michael Case , Shih-Fu Chang , Eitan Grinspun , Peter K. Allen IEEE TASE, 2016 Deformable object manipulation with an application of personal assitive robot. This is the journal paper of our "laundry robot" series: |
|
Danfei Xu, Hernan Badino, Daniel Huber IROS, 2015 Vision-based localization on a probabilistic road network. |
|
Danfei Xu, Gerald E. Loeb, Jeremy Fishel ICRA, 2013 Object classification using multi-modal tactile sensing. |
|
|
|
|