Embodied AI

We study the automatic generation of natural language navigation instructions in visually realistic indoor environments. Existing generators suffer from poor visual grounding, skip steps, and hallucinate objects. We address this using a large language model which incorporates visual landmark detection.. The model dramatically increases the quality of generated instructions, such that humans can follow them with a 71% success rate (SR); just shy of the 75% SR of real human instructions.