Ghost Autonomy Announces Investment from OpenAI Startup Fund to Bring Multi-Modal LLMs to Autonomous Driving

November 08, 2023 at 08:30 AM EST

Ghost Autonomy, a pioneer in scalable autonomy software for consumer cars, today announced a $5 million investment from the OpenAI Startup Fund to bring large-scale, multi-modal large language models (MLLMs) to autonomous driving. The funds will be used to accelerate ongoing research and development of LLM-based complex scene understanding required for urban autonomy. The new investment brings the company’s total funding to $220 million to date.

“Multi-modal models have the potential to expand the applicability of LLMs to many new use cases including autonomy and automotive. With the ability to understand and draw conclusions by combining video, images, and sounds, multi-modal models may create a new way to understand scenes and navigate complex or unusual environments,” said Brad Lightcap, OpenAI's COO and manager of the OpenAI Startup Fund.

MLLMs potentially represent a new architecture for self-driving software, capable of handling the long tail of rare and complex driving scenarios. Where existing single-task networks are limited to their narrow scope and training, LLMs allow autonomous driving systems to reason about driving scenes holistically, utilizing broad-based world knowledge to navigate complex and unusual situations, even those never seen before.

“Solving complex urban driving scenarios in a scalable way has long been the holy grail for this industry – LLMs provide a breakthrough that will finally enable everyday consumer vehicles to reason about and navigate through the toughest scenarios,” stated John Hayes, founder and CEO, Ghost Autonomy. “While LLMs have already proven valuable for offline tasks like data labeling and simulation, we are excited to apply these powerful models directly to the driving task to realize their full potential.”

Ghost’s platform allows leading automakers to bring artificial intelligence and advanced autonomous driving software into the next generation of vehicles, now expanding capabilities and use cases with MLLMs. Ghost is actively testing these capabilities via its development fleet today, and is partnering with automakers to jointly validate and integrate new large models into the autonomy stack.

For more information, including more videos and integration details, visit Ghost’s blog.

About Ghost Autonomy

Ghost makes autonomous driving software for the next generation of consumer cars. It is a software partner to automakers, accelerating new applications of artificial intelligence to help realize the software-defined vehicle. Ghost is pioneering the use of multi-modal large language models (MLLMs) in autonomy. This new software architecture applies the powerful human-like reasoning of large models to the driving task, enabling autonomous vehicles to understand and navigate the long tail of complex driving scenarios, even those never seen before. Ghost was founded in 2017 by John Hayes, who previously co-founded Pure Storage, taking the company public in 2015. Ghost is based in Mountain View, CA, with additional offices in Detroit, Dallas and Sydney.