The computer vision community has made impressive progress on many problems in visual recognition. However, for any intelligent system to interact with its environment, it needs to understand much more than simply recognizing and labeling objects. In this workshop, our goal is to motivate and discuss what to explore next. Specifically, we will study the representations and algorithms necessary for a system to physically interact in everyday scenes. This involves studying problems such as learning predictive models of the future, deep reinforcement learning, self-supervised robotics, and understanding object physics and affordances. Our goal is to advance the field with several impacts. First of all, we will continue providing a yearly summary of new progress in the field through a combination of keynote talks, workshop papers, and a panel discussion. Additionally, we plan to have a session for invited student talks to give a chance for junior researchers to share their innovations. We will invite and encourage the participation from all related fields including computer vision, robotics, cognitive science, and HCI. This will provide an opportunity to share various perspectives for this exciting research agenda and encourage collaboration among multiple fields.