About Me

I am an Associate Professor in the School of Interactive Computing at Georgia Tech and a Research Director in the Fundamental AI Research (FAIR) team at Meta.

From Jan 2013 - Aug 2016, I was an Assistant Professor in the Bradley Department of Electrical and Computer Engineering at Virginia Tech, where I led the VT Machine Learning & Perception group. From Sep 2010 - Dec 2012, I was a Research Assistant Professor at Toyota Technological Institute at Chicago (TTIC), a philanthropically endowed academic computer science institute located in the campus of University of Chicago.

I received my Ph.D. in 2010 from Carnegie Mellon University, advised by Tsuhan Chen. I had the pleasure of spending the last 1.5 years of my grad school life at Cornell University, following Tsuhan's move.

Research

I am fascinated by the natural phenomenon of intelligence, and I work on understanding and advancing the limits of artificial intelligence (AI). More specifically, my research lies at the intersection of machine learning and computer vision, with forays into robotics and natural language processing.

• see (or more generally perceive their environment through vision, audition, or other senses),
• talk (i.e. hold a natural language dialog grounded in their environment),
• act (e.g. navigate their environment and interact with it to accomplish goals), and
• reason (i.e., consider the long-term consequences of their actions).

Here are some representative projects from my lab:

• Summarizing beliefs of AI agents via diverse plausible predictions/hypotheses:
  Diverse Beam Search, Multiple Choice Learning, Tutorial on Diversity at CVPR '13 and CVPR '16,
• Vision-and-language:
  Image Captioning, Visual Question Answering, Visual Dialog, Audio-Video Dialog, Human-AI GuessWhich games,
• Embodied AI:
  Habitat: A Platform for Embodied AI, Decentralized Distributed PPO, Embodied Question Answering,
• Explainable, Unbiased, Trustworthy AI:
  Grad-CAM (Visual Explanations), Human-vs-machine attention,
• Platforms for reproducible AI research:
  EvalAI, a platform for evaluating AI algorithms.

Teaching

• Fall 2021: CS 4803/7643: Deep Learning.
• Fall 2020: CS 4803/7643: Deep Learning.
• Fall 2019: CS 4803/7643: Deep Learning.
• Fall 2018: CS 4803/7643: Deep Learning.
• Fall 2017: CS 7643: Deep Learning.
• Spring 2016: ECE 2574: Data Structures and Algorithms.
• Fall 2015: ECE 6504: Deep Learning for Perception. Video lectures available!
• Spring 2015: ECE 5984: Introduction to Machine Learning. All video lectures available!
• Fall 2014: ECE 2574: Data Structures and Algorithms.
• Spring 2014: ECE 6504: Advanced Topics in Machine Learning: Probabilistic Graphical Models.
• Fall 2013: ECE 4984/5984: Introduction to Machine Learning & Perception.
• Spring 2013: ECE 4984/5984: Introduction to Machine Learning & Perception.
• Spring 2012: Co-Instructor for TTIC 31070: Convex Optimization. Primary Instructor: Nati Srebro.

Group

Students

• Harsh Agrawal, PhD student (2018 — Present)
  Snap Inc. Research Fellowship
• Arjun Majumdar, PhD student (2019 — Present)
• Joanne Truong, PhD student (2019 — Present); Co-advised with Sonia Chernova
  NSF Graduate Research Fellowship
  Google Women Techmakers Scholarship
  Adobe Research Fellowship
  Apple Scholars in AI/ML PhD Fellowship
• Andrew Szot, PhD student (2020 — Present); Co-advised with Zsolt Kira
• Naoki Yokoyama, PhD student (2020 — Present); Co-advised with Sehoon Ha
  ARCS Scholar Award Fellowship
  

• Ram Ramrakhya, MS student (2021 — Present)
• Mukul Khanna, MS student (2022 — Present)

Visiting Students / Interns

Alumni

Former Research Scientists, Postdocs

• Stefan Lee, Postdoc 2016-17, Research Scientist 2017-19. Next position: Assistant Professor, Oregon State University
  Outstanding Research Scientist Award, College of Computing, Georgia Tech
  DARPA Riser
• Peter Anderson, Research Scientist 2018-19. Next position: Research Scientist in Language and Vision at Google.
• Zhile Ren, Postdoc 2018-19. Next position: Applied Research Scientist, Apple.

Former PhD Students

• Nirbhay Modhe, PhD 2022.
  PhD Thesis: Leveraging Value-awareness for Online and Offline Model-based Reinforcement Learning.
• Erik Wijmans, PhD 2022. Co-advised with Irfan Essa. Next position: Apple.
  ARCS Scholar Award Fellowship
  PhD Thesis: Emergence of Intelligent Navigation Behavior in Embodied Agents from Massive-Scale Simulation.
  Georgia Tech College of Computing PhD Dissertation Award, 2023
  Sigma Xi Best Ph.D. Thesis Award, 2023
• Ashwin Kalyan, PhD 2020. Next position: Allen Institute for Artificial Intelligence (AI2).
  JP Morgan PhD Fellowship
  PhD Thesis: Improved search techniques for structured prediction.
• Abhishek Das, PhD 2020. Next position: Facebook AI Research.
  Facebook Fellowship
  Adobe Research Fellowship
  Snap Inc. Research Fellowship
  Sigma Xi Best Ph.D. Thesis Award, 2021
  Georgia Tech College of Computing PhD Dissertation Award, 2021
  AAAI/ACM-SIGAI Doctoral Dissertation Award Runner-up, 2020
  PhD Thesis: Building agents that can see, talk, and act.
• Michael Cogswell, PhD 2020. Next position: SRI.
  Bradley Fellowship
  PhD Thesis: Disentangling neural network representations for improved generalization.
• Yash Goyal, PhD 2020. Next position: Samsung AI Lab, Montreal.
• Aishwarya Agrawal, PhD 2019. Next position: Google DeepMind / University of Montreal (MILA).
  NVIDIA Graduate Fellowship
  Sigma Xi Best Ph.D. Thesis Award, 2020
  Georgia Tech College of Computing PhD Dissertation Award, 2020
  AAAI/ACM-SIGAI Doctoral Dissertation Award Runner-up, 2019
  PhD Thesis: Visual Question Answering and Beyond.
• Qing Sun, PhD 2017. Next position: Facebook AI.
  PhD Thesis: Greedy Inference Algorithms for Structured and Neural Models.
• Gordon Christie, PhD 2016. Co-Advised with Kevin Kochersberger. Next position: Army Research Lab
  PhD Thesis: Collaborative Unmanned Air and Ground Vehicle Perception for Scene Understanding, Planning and GPS-denied Localization.

Former MS students, undergraduate advisees, and interns

• Harish Kamath, BS 2021 (next position: Scale.AI)
• Rishabh Jain, MS 2021 (next position: eBay)
The Marshall D. Williamson Fellowship Award 2021
• Joel Ye, MS 2021 (next position: PhD student at CMU)
Donald V. Jackson Fellowship award for academic excellence and leadership, 2021
• Sameer Dharur, MS 2021 (next position: Apple)
• Deshraj Yadav, MS 2018 (next position: Tesla Autopilot)
Snap Inc. Research Scholarship, 2017
• Akrit Mohapatra, MS 2017 (next position: eBay)
• Aroma Mahendru, MS 2017 (next position: Bloomberg)
• Latha Pemula, MS 2016 (next position: Amazon)
• Harsh Agrawal, MS 2016 (next position: Snapchat Research)
• Prakriti Banik, MS 2015 (next position: Bloomberg)
• Neelima Chavali, MS 2015 (next position: Many Trees Inc)
• Clint Solomon, MS 2015 (next position: PhotoKharma)


• Yash Kant, Intern (next position: PhD student at University of Toronto)
• Rishabh Jain, Intern (next position: MS student at GT)
• Sanyam Agrawal, Intern (next position: Google)
• Deshraj Yadav, Intern (next position: MS student at GT)
• Tejas Khot, Intern (next position: MS student at RI, CMU)
• Viraj Prabhu, Intern (next position: MS student at GT)
• Ahmed Osman, Intern (next position: MS student at UCL)
• Khushi Gupta, Intern (next position: MS student at RI, CMU)
• Ashwin Kalyan, Intern (next position: PhD student at GT)
• Abhishek Das, Intern (next position: PhD student at GT)
• Avi Singh, Intern (next position: PhD student at UC Berkeley)
• Faruk Ahmed, Intern (next position: PhD student at UMontreal)
• Senthil Purushwalkam, Intern (MS student at RI, CMU)
• Aroma Mahendru, Intern (next position: MS student at VT)
• Ratnesh Kumar, Intern (next position: MERL)
• Qi Lou, Intern (next position: PhD student at UC Irvine)
• Ankit Laddha, Intern (next position: MS student at RI, CMU)
• Adarsh Prasad, Intern (next position: PhD student at UT-Austin)
• Harsh Agrawal, Intern (next position: MS student at VT)
• Abhimanyu Dubey, Intern (next position: PhD student at MIT)

Former Student Collaborators

• Payman Yadollahpour (PhD student from TTIC)
• Kun Duan (PhD student from Indiana U)
• Abner Guzman-Rivera (PhD student from UIUC)
• Daozheng Chen (PhD student from UMD)
• Jason Lew (MEng student at Cornell)
• Tanmay Verma (BTech student from IIIT-D)
• Charvi Puri (BTech student from IIIT-D)
• Kevin Tang (BS student at Cornell, now grad student at Stanford)

Press Coverage

The research performed by my group (and our collaborators) has been covered here:

• Four Indian American Researchers Named Young Investigator Awardees for 2017 by Office of Naval Research
  
• Is Artificial Intelligence Permanently Inscrutable?
  
• Deep learning is creating computer systems we don't fully understand.
  
• Robot eyes and humans fix on different things to decode a scene.
  
• Teaching computers to describe images as people would. Project Lead: Meg Mitchell.
  
• A Giant Leap for Machine Kind; When Robots Can See.
  
• Artificial Intelligence Algorithm Taught to Recognise Humor.
  
• AI Algorithm Identifies Humorous Pictures.
  
• What’s in This Picture? AI Becomes as Smart as a Toddler.
  
• How to automatically detect the most important people in a photograph.
  
• Dhruv Batra receives NSF CAREER Award for machine perception research.
  

  ECE News	DAC News

• He aims to mimic the human brain’s capabilities.
  
• Researcher seeks to remove ambiguity in computer visual recognition systems.
  

  COE News	VT News	Phys Org News

• Stocking the visual toolbox.
  

  ECE News

Quoted in coverage not directly about my work:

• The Curious Case of Apple’s Supposed Self-Driving Car.
  
  • WIRED, Feb 2015.
  • Business Insider, Feb 2015.
  • Realty Today, Feb 2015.
  • Opp Trends, Feb 2015.
  • Rt.com, Feb 2015.

Funding

My group is supported by the following:

• Army Research Lab (ARL) and Army Research Office (ARO)
• Defense Advanced Research Projects Agency (DARPA)
• National Science Foundation (NSF)
• Office of Naval Research (ONR)
• Amazon
• Facebook
• Google
• Samsung
• Siemens

Talks

On the web

• Visual Question Answering
  Keynote talk, Multimodal Machine Learning Workshop at NIPS, Dec 2015.
  
  
• Visual Question Answering
  Data Science Summit & Dato Conference, 2015.
  
  
• Visual Question Answering
  Deep Learning Summit, 2015.
  
  
• Hedging Against Uncertainty via Multiple Diverse Predictions
  Microsoft Research Talk, 2014.
  
  
• Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets
  NIPS Workshop on Discrete and Combinatorial Problems in Machine Learning (DISCML), 2014.
  
  
• Hedging Against Uncertainty via Multiple Diverse Predictions
  CESCA Seminar, 2014.
  
  
• Beyond MAP: Making Multiple Predictions from Probabilistic Models: Diversity, DPPs and more
  Tutorial at the Conference on Computer Vision and Pattern Recognition (CVPR) 2013.
  
  
• Diverse M-Best Solutions in Markov Random Fields
  European Conference on Computer Vision (ECCV) 2012.
  
  
• Structured-Output Models for Computer Vision
  Microsoft Research Talk, 2012.
  
  
• Beyond Trees: MRF Inference via Outer-Planar Decomposition
  Microsoft Research Talk, 2009.
  
  

Slides from talks

• A tale of two negative results [slides (pptx)]
• Towards AI agents that can see, talk, and act [slides (pptx)]
• Visual Dialog [slides (pptx)]
• Towards Transparent VQA Models [slides (pptx)]
• Visual Question Answering [slides (pptx)]
• Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets.
  [ slides (pptx) ]
• M-Best and Diverse M-Best MAP Inference in Graphical Models [ slides ]
• CloudCV: Large-Scale Distributed Computer Vision as a Cloud Service [ slides ]
• Hedging Against Uncertainty via Multiple Diverse Predictions [ slides (pptx) ]
• Structured-Output Models for Computer Vision
• The M-Best Mode Problem [ slides (pptx) ]
• Focused Inference and the M-Best Mode Problem [ slides (pptx) ]
• Focused Inference with Local Primal-Dual Gaps [ slides (pptx) ]
• On Graph-Structured Discrete Labelling Problems in Computer Vision [ slides (pptx) ]
• Beyond Trees: MRF Inference via Outer-Planar Decomposition [ slides (pptx) ]