Elastic Reports 8x Speed and 32x Efficiency Gains for Elasticsearch and Lucene Vector Database

April 26, 2024 at 07:45 AM EDT

The world’s most downloaded vector database is used by customers to store and search billions of vector embeddings

Elastic (NYSE: ESTC), the company behind Elasticsearch®, today announced new vector database performance gains with Elasticsearch and Apache Lucene, with up to 8x speed and 32x efficiency. These advancements provide developers with the most flexible and open tools needed to keep up with rapid generative AI innovation.

New optimization strategies and enhancements driving the increased gains in Elasticsearch and Apache Lucene include*:

Multi-threaded search capabilities run searches in independent segments, minimizing response times, ensuring users receive their search results as swiftly as possible
Accelerated multi-graph vector search and information exchange among segment searches, reducing query latencies up to 60%
Panama Vector API Integration enabling Java code to interface seamlessly with SIMD instructions, unlocking a new era of vector search performance optimization
Maximized memory efficiency with scalar quantization, slashing memory usage by approximately 75% without sacrificing search performance
Seamless compression improvements that keep search results accurate while making data smaller, laying the foundations for binary quantization that will deliver the full potential of vector search while maximizing resource utilization and scalability
Multi-vector integration in Lucene and Elasticsearch that enables searches across nested documents and joins, making document searches in Lucene more effective

“Our goal is to make Elasticsearch and Apache Lucene the best vector database and Elasticsearch the best retrieval platform for semantic search, RAG, and generative AI use cases,” said Shay Banon, founder and chief technology officer at Elastic. “Elastic’s contributions bring multifold performance gains in speed, scale, and efficiency, ensuring the Elasticsearch-Lucene integration continues to be a versatile vector database and search experience loved by developers and trusted by enterprises the world over.”

Customers are building the next generation of AI enabled search applications with Elastic’s vector database and vector search technology. For example, Roboflow is used by over 500,000 engineers to create datasets, train models, and deploy computer vision models to production. Roboflow uses Elastic vector database to store and search billions of vector embeddings.

Learn more about Lucene’s integration into Elasticsearch and how these investments deliver faster vector search performance and scale here.

*These features are available in Elasticsearch version 8.10 and above.

About Elastic

Elastic (NYSE: ESTC), the leading search analytics company, securely harnesses search powered AI to enable everyone to find the answers they need in real-time using all their data, at scale. Elastic’s solutions for security, observability and search are built on the Elasticsearch platform, the development platform used by thousands of companies, including more than 50% of the Fortune 500. Learn more at elastic.co.