2023
Habitat-Matterport 3D Semantics Dataset .Highlight paper
PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav .
Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Second.
Emergence of Maps in the Memories of Blind Navigation Agents .Outstanding Paper Award
BC-IRL: Learning Generalizable Reward Functions from Demonstrations .Notable top-25% paper
ViNL: Visual Navigation and Locomotion Over Obstacles .Best Paper Award at Learning for Agile Robotics Workshop at CoRL, 2022.
Simple and Effective Synthesis of Indoor 3D Scenes .
2022
ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings.
Variable Experience Rollout: Training Robust Skill Policies for Rearrangement .
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning .
Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation .
Cross-Domain Transfer via Semantic Skill Imitation .
Housekeep: Tidying Virtual Households using Commonsense Reasoning .
Memory-Augmented Reinforcement Learning for Image-Goal Navigation .
Benchmarking Augmentation Methods for Learning Robust Navigation Agents: the Winning Entry of the 2021 iGibson Challenge .
Episodic Memory Question Answering .Oral Presentation
Around the World in 3,000 Hours of Egocentric Video .Best Paper Award Finalist
Learning Embodied Object-Search Strategies from Human Demonstrations at Scale .
Is Mapping Necessary for Realistic PointGoal Navigation?
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget .
2021
Habitat 2.0: Training Home Assistants to Rearrange their Habitat .Spotlight paper
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI .https://aihabitat.org/datasets/hm3d/ ]
SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation .
Model-Advantage Optimization for Model-Based Reinforcement Learning .
THDA: Treasure Hunt Data Augmentation for Semantic Navigation.
Contrast and Classify: Training Robust VQA Models .
Auxiliary Tasks and Exploration Enable ObjectNav .
Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation .
Waypoint Models for Instruction-guided Navigation in Continuous Environments .Oral Presentation
Large Batch Simulation for Deep Reinforcement Learning .
Success Weighted by Completion Time: A Dynamics-Aware Evaluation Criteria for Embodied Navigation .
Learning Navigation Skills for Legged Robots with Learned Robot Embeddings .
Bi-directional Domain Adaptation for Sim2Real Transfer of Embodied Navigation Agents .
SOrT-ing VQA Models: Contrastive Gradient Learning for Improved Consistency .
Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views .
2020
Rearrangement: A Challenge for Embodied AI .
Sim-to-Real Transfer for Vision-and-Language Navigation .
Auxiliary Tasks Speed Up Learning PointGoal Navigation .
Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agents .
Dialog without Dialog Data: Learning Visual Dialog Agents from VQA Data .
Where Are You? Localization from Embodied Dialog .
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects .
Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web .Spotlight
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments .
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation .
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline .
Spatially Aware Multimodal Transformers for TextVQA .
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames .
Analyzing Visual Representations in Embodied Navigation Tasks .
IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL .
Embodied Multimodal Multitask Learning .
2019
The Replica Dataset: A Digital Replica of Indoor Spaces .
Emergence of Compositional Language with Deep Generational Transmission .
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks .
Chasing Ghosts: Instruction Following as Bayesian State Tracking .
Habitat: A Platform for Embodied AI Research .https://aihabitat.org/ ] Best Paper Award Nominee
nocaps: novel object captioning at scale .https://nocaps.org ]
SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation .
Embodied Visual Recognition .
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded .
Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning .
Improving Generative Visual Dialog by Answering Diverse Questions .
Trainable Decoding of Sets of Sequences for Neural Sequence Models .
Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering .
TarMAC: Targeted Multi-Agent Communication .
Counterfactual Visual Explanations .
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication .
Embodied Question Answering in Photorealistic Environments with Point Cloud Perception .http://embodiedqa.org ] Oral Presentation
Multi-target Embodied Question Answering .http://embodiedqa.org ]
Towards VQA Models That Can Read .https://textvqa.org/ ]
Audio-Visual Scene-Aware Dialog .https://video-dialog.com// ]
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog .
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future .
2018
Pythia v0.1: The Winning Entry to the VQA Challenge 2018 .
Talk the Walk: Navigating New York City through Grounded Dialogue .
Neural Modular Control for Embodied Question Answering .
Learning to Ask Questions to Learn Visual Recognition .
Visual Coreference Resolution in Visual Dialog using Neural Module Networks .
Choose Your Neuron: Incorporating Domain Knowledge through Neuron Importance .
Graph R-CNN for Scene Graph Generation .
Learn From Your Neighbor: Learning Multi-Modal Distributions from Sparse Annotation .Oral Presentation (Long Talk)
Embodied Question Answering .http://embodiedqa.org ] Oral Presentation
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering .
Neural Baby Talk .code ] Spotlight
Visual Dialog .visualdialog.org | video ]
Neural Guided Deductive Search for Real-time Program Synthesis .
Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models .Code
| CloudCV demo ]
2017
Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
Resolving Vision and Language Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes.
Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model .
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning .visualdialog.org (data, code)
| CloudCV demo ] Oral Presentation
Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization .Code
| CloudCV demo
| video ]
Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog .Best Paper Award
Deal or No Deal? End-to-End Learning for Negotiation Dialogues .
The Promise of Premise: Harnessing Question Premises in Visual Question Answering .
ParlAI: A Dialog Research Software Platform .parl.ai ]
Evaluating Visual Dialog Agents via Cooperative Human-AI Games .visualdialog.org ]
Visual Dialog .visualdialog.org | video ] Spotlight
Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning .
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering .Webpage
| video ]
Counting Everyday Objects in Everyday Scenes .Spotlight
LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation .code ]
VQA: Visual Question Answering .visualqa.org (data, code, challenge)
| CloudCV demo
| video ]
2016
Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles .
Hierarchical Question-Image Co-Attention for Visual Question Answering .06/2016: State of art performance on the VQA dataset (62.06% Open-Ended / 66.33% Multiple-Choice ).here for latest numbers.
Sort Story: Sorting Jumbled Images and Captions into Stories .
Analyzing the Behavior of Visual Question Answering Models .
Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions .
Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions? Best Student Paper
Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes .
Towards Transparent AI Systems: Interpreting Visual Question Answering Models .Best Student Paper
Measuring Machine Intelligence Through Visual Question Answering .
Object-Proposal Evaluation Protocol is 'Gameable' .project page (data, code)
| Object Proposals Library ] Spotlight
We Are Humor Beings: Understanding and Predicting Visual Humor .Spotlight
Yin and Yang: Balancing and Answering Binary Visual Questions .
Simultaneously Discovering Image Clusters and Deep Representations .code
]
Visual Storytelling .project page (with data)
] GroupSight workshop AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2016.
A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories .project page (with data) ] Oral Presentation
Empirical Minimum Bayes Risk Prediction .project page |
code
]
Reducing Overfitting in Deep Networks by Decorrelating Representations .
Pose Tracking by Efficiently Exploiting Global Features .
2015
Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks .
SubmodBoxes: Near-Optimal Search for a Set of Diverse Object Proposals .
VQA: Visual Question Answering .visualqa.org (data, code, challenge)
| CloudCV demo
| video spotlight ]
Optimizing Expected Intersection-over-Union with Candidate-Constrained CRFs .
CloudCV: Large Scale Distributed Computer Vision as a Cloud Service .CloudCV ]
Active Learning for Structured Probabilistic Models with Histogram Approximation .project page (data, code, poster, slides)
| talk ] Oral Presentation
VIP: Finding Important People in Images .project page
| online demo on CloudCV ]
A Comparative Study of Modern Inference Techniques for Discrete Energy Minimization Problems .project page + data + code ]
2014
Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets .data ] Spotlight
Human Pose Estimation via Multi-layer Composite Models .
Empirical Minimum Bayes Risk Prediction: .project page |
code
]
Multimodal Learning in Loosely-organized Web Images .
Putting the User in the Loop for Image-Based Modeling .
Efficiently Enforcing Diversity in Multi-Output Structured Prediction .Oral Presentation
2013
Submodular Maximization and Diversity in Structured Output Spaces .
Group Norm for Learning Structured SVMs with Unstructured Latent Variables .
A Systematic Exploration of Diversity in Machine Translation .
Discriminative Re-ranking of Diverse Segmentations .leaderboard link ] 06/2013: State of art (48.1%) performance on PASCAL VOC Segmentation 2012.
See here for more recent results.
A Comparative Study of Modern Inference Techniques for Discrete Energy Minimization Problems .talk video ]Oral Presentation
DivMCuts: Faster Training of Structural SVMs with Diverse M-Best Cutting-Planes .talk video ]Oral Presentation
2012
Mode-Marginals: Expressing Uncertainty via Diverse M-Best Solutions .
Faster Training of Structural SVMs with Diverse M-Best Cutting-Planes .
Multiple Choice Learning: Learning to Produce Multiple Structured Outputs .
Diverse M-Best Solutions in Markov Random Fields . talk slides (pptx) |
talk video |
code
] Oral Presentation
An Efficient Message-Passing Algorithm for the M-Best MAP Problem . talk slides (pptx) ] Oral Presentation
MaxFlow Revisited: An Empirical Comparison of Maxflow Algorithms for Dense Vision Problems . project page ]
A Multi-layer Composite Model for Human Pose Estimation . project page ]
Learning the Right Model: Efficient Max-Margin Learning in Laplacian CRFs .
2011
M-Best Modes: Diverse M-Best Solutions in MRFs .
Group Norm for Learning Latent Structural SVMs .
Similarity Sensitive Nonlinear Embeddings .
Interactive Co-segmentation of Objects in Image Collections. Mini-Book [ Springer Page |
project page ]
Dynamic Tree-Block Coordinate Ascent . code ]
Making the Right Moves: Guiding Alpha-Expansion using Local Primal-Dual Gaps .
Inference for Order Reduction in Markov Random Fields .
Tighter Relaxations for MAP-MRF Inference: A Local Primal-Dual Gap based Separation Algorithm .
Scribble Based Interactive 3D Reconstruction via Scene Co-segmentation . project page |
dataset ] Oral Presentation
Interactively Co-segmenting Topically Related Images with Intelligent Scribble Guidance . Springer Page |
project page ] 03/2011: Among top downloaded articles from IJCV in the last 30 days.
2010
iModel: Interactive Co-segmentation for Object of Interest 3D Modeling . project page |
dataset ]
Beyond Trees: MRF Inference via Outer-Planar Decomposition .
iCoseg: Interactive Co-segmentation with Intelligent Scribble Guidance . project page ]
2009
Dynamic Planar-Cuts: Efficient Computation of Min-Marginals for Outer-Planar Models .
Seed Image Selection in Interactive Cosegmentation . project page ]
Cutout-Search: Putting a name to the Picture .
2008
Semi-Supervised Clustering via Learnt Codeword Distances .
Learning Class-Specific Affinities for Image Labelling . project page ]
Space-Time Shapelets for Action Recognition . project page ]
Patents
Finding Important People in Images.
Method for Generating Object Cutout for Topically Related Photographs. googe patent page |
faqs page ]
Demos
iModel: Object of Interest 3D Modeling via Interactive Co-segmentation on a Mobile Device .
Interactive Cosegmentation by Touch .