Danfei Xu

I am an Assistant Professor at the School of Interactive Computing at Georgia Tech and a (part-time) Research Scientist at NVIDIA AI. I work at the intersection of Robotics and Machine Learning.

I received my Ph.D. in CS from Stanford University advised by Fei-Fei Li and Silvio Savarese (2015-2021) and B.S. from Columbia University (SEAS'15). I've spent time at DeepMind UK (2019), ZOOX (2017), Autodesk Research (2016), CMU RI (2014), and Columbia Robotics Lab (2013-2015).

Email  /  Google Scholar  /  CV (Sep 2024)  /  Github  /  Twitter

I direct the Robot Learning and Reasoning Lab (RL2). We aim to build general-purpose and adaptable "brains" for robots in home, factory, healthcare, and search & rescue missions alike. Our work focuses on endowing robots with both flexible high-level planning abilities ("what to do next") and robust low-level sensorimotor control ("how to do it"). The research draws equally from Robotics and Machine Learning, with the following themes:

• Compositionality: Inspired by human's ability to "make infinite use of finite means", we aim to enable (good!) combinatorial explosions in robots' ability to solve new tasks. Examples include neural program synthesis for control [a][b], compositional diffusion models for planning [a][b], and learning neural-symbolic motor skills [a][b].
• Generative Modeling: Modeling complex, high-dimensional distributions is fundamental to many robotics problems. We aim to advance core generative modeling research and their applications to robotics problems. Examples include modeling human behaviors in human-robot collaboration [a][b], modeling dynamics in manipulation planning [a], and explorative "play" behaviors [a][b].
• Robot Learning from Human Data: Despite recent successes in Internet-scale AI, Robot Learning remains data-deprived. We are broadly interested in enabling robots to learn from human data, in particilar, data captured with wearable devices such as smart glasses. My CoRL'24 keynote talk covers our vision on this topic.
• Full-stack Robot Learning: Robotics is as much science as system building. We are committed to developing high-quality, open-source robot hardware and software systems to demonstrate and promote "full-stack" progress in learning-based robotics. We actively maintain Robomimic, a general-purpose Robot Learning framework and benchmark.

Robot Learning from Egocentric Human Data

Solving multi-step manipulation problems with zero-shot generalization.

Learning tactile control policies from human hand demonstrations.

A compositional generative model that can plan for long-horizon, complex coordinated manipulation tasks.

Learning to generate legible robot motion (motion with expressive intent) with Conditional Diffusion Models.

A principled Bayesian framework to model uncertainty in NeRF for active mapping.

Unsupervised 4D representation learning from videos.

High-precision manipulation with generative action modeling.

[website]

A new algorithm to chain together skill-level diffusion models to solve long-horizon manipulation tasks.

[website]

A new field-based representation (occupancy density) to model and optimize granular object manipulation.

An offline imitation learning algorithm to learn from mixed-quality human demonstrations.

[website]

We present HITL-TAMP, a system that merges human teleoperation with automated TAMP to efficiently train robots for complex tasks.

GPT4-guided trajectory diffusion model.

We leverage human video data to train a planner to guide robust low-level task execution.

We use Task and Motion Planning (TAMP) as guidance for learning generalizable and composable sensorimotor skills.

We harness the power of large language model to generate program-like task plans.

Hierarchical imitation for generating diverse, realistic, and robust traffic behaviors.

Learn behavior prior through diffusion-based imitation. Sample desired behaviors by conditional diffusion and Signal-Temporal Logic (STL) rules.

Defending against attacks that target generative behavior models.

Physically-plausible attack on behavior forecasting models.

Building generalizable world models for planning with unseen objects and environments.

[code+dataset][website][blogpost]

A large-scale study on learning manipulation skills from human demonstrations.

Human-in-the-loop learning for complex manipulation tasks.

Learning human-robot collaboration policies from human-human collaboration demonstrations.

An learnable action space for recovering human's hand-eye coordination behaviors by learning from human demonstrations.

We extend the classical definition of affordance to enable generalizable long-horizon planning.

[Video]

An algorithm framework that simultaneously addresses the reward delusion problem in supervised reward learning and the overfitting discriminator problem in adversarial imitation learning.

Learning to plan from instructional videos.

[website] [video] [blog post]

Learning visuomotor policies that can generalize across long-horizon tasks by modeling latent compositional structures.

[website] [video] [code]

Real-time category-level 6D object tracking from RGB-D data.

[code] [poster]

A flexible neural network architecture for learning to plan from video demonstrations.

[blog post]

One-shot imitation learning via hybrid neural-symbolic planning.

Learning generalizable navigation policy from mid-level visual representations.

[website] [video] [code]

Dense RGB-depth sensor fusion for 6D object pose estimation.

[blog post]

Generate executable task graphs from video demonstrations.

[website] [video] [Two Minute Papers] [blog post]

Neural Task Programming (NTP) is a meta-learning framework that learns to generate robot-executable neural programs from task demonstration video.

End-to-end 3D Bounding Box Estimation via sensor fusion.

[website] [code]

We propose an end-to-end model that jointly infers object category, location, and relationships. The model learns to iteratively improve its prediction by passing messages on a scene graph.

[website] [code]

We propose an end-to-end 3D reconstruction model that unifies single- and multi-view reconstruction.

[website]

Deformable object manipulation with an application of personal assitive robot.

This is the journal paper of our "laundry robot" series:
ICRA 2015
IROS 2015
ICRA 2016

Vision-based localization on a probabilistic road network.

Object classification using multi-modal tactile sensing.

Template source