Running AI search at scale requires sophisticated infrastructure for vector storage, indexing, and serving. Approximate nearest neighbor (ANN) search trades a small loss in recall for huge speed gains over exact kNN by using index structures like HNSW, IVF-PQ, or ScaNN, enabling sub-linear vector lookup over millions of embeddings. Hierarchical Navigable Small World (HNSW) is a graph-based ANN index that builds multi-layer proximity graphs, offering logarithmic-time search, high recall, and incremental inserts — making it the most popular vector index. Product quantization (PQ) splits each embedding into sub-vectors and replaces each with the nearest centroid id from a codebook, compressing vectors by roughly 32x with small recall loss for billion-scale retrieval. DiskANN is a Microsoft graph-based ANN index designed for SSD-resident vectors at billion scale with the Vamana graph, used in Azure AI Search and Bing's vector backend. Meta's FAISS (Facebook AI Similarity Search) is the open-source library for efficient similarity search and clustering of dense vectors, with GPU support and many index types — the toolkit beneath most vector databases. In a one-line comparison: HNSW is graph-based with high recall but more RAM, IVF-PQ is inverted-file plus product quantization with low memory and lower recall, and ScaNN is Google's anisotropic vector quantization that performs at top-tier on TPU/CPU benchmarks.
The vector database landscape offers many choices. Pinecone is a managed serverless vector database (built on proprietary infrastructure) popular for production RAG, offering metadata filtering, namespaces, and hybrid sparse+dense search. Weaviate is an open-source vector DB with built-in vectorization modules (OpenAI, Cohere, HuggingFace) and GraphQL/REST APIs, supporting hybrid search and generative RAG out of the box. Qdrant is a Rust-based open-source vector DB with high-performance HNSW, rich payload filtering, and quantization, widely used for on-prem and self-hosted RAG. Milvus (built by Zilliz) is an open-source, distributed vector DB with multiple index types targeting billion-scale deployments — the OSS counterpart to Pinecone at scale. pgvector is a PostgreSQL extension that adds vector types and ANN search to Postgres, popular for teams that want a single OLTP+vector store without a new system. LanceDB is an embedded, serverless vector DB built on the Lance columnar format, optimized for ML workloads with zero ops. For broader hybrid systems, Elasticsearch combines BM25 plus ELSER plus optional dense_vector in one engine, OpenSearch's neural_search plugin adds dense and sparse neural retrievers, and Vespa.ai is Yahoo's open-source engine for large-scale hybrid search plus ranking plus ML inference. The major difference between vector DBs and traditional search engines is that traditional engines optimize lexical recall and aggregations while vector DBs optimize semantic similarity at scale via ANN; hybrid systems increasingly offer both but trade-offs in freshness, filter expressiveness, and cost remain.
Distance metric selection matters: cosine is standard for normalized embeddings, dot product for unnormalized ones, and Euclidean for absolute-distance clustering. Metadata filters (e.g. tenant_id, date > 2024, language = "en") are pre-filters that restrict ANN search to a subset of vectors, critical for multi-tenant and time-bounded retrieval. The SIFT-1B benchmark (1 billion 128-dim SIFT descriptors) is the standard large-scale ANN benchmark; state-of-the-art systems achieve greater than 95% recall@10 at sub-millisecond latency per query on a single GPU. Latency budgets are tight in production: a typical RAG query targets about 1-3 seconds end-to-end (roughly 50-150ms retrieval, 100-300ms reranking, and 500-2000ms for the LLM to stream its first token), and anything over 3 seconds drops user engagement sharply. With HNSW on modern hardware, p95 hybrid search over 10M vectors runs in roughly 50-150ms, well within the RAG latency budget.
Optimizing cost and context length is just as important as optimizing latency. Embedding too many chunks per query wastes tokens and dilutes attention; it is better to retrieve, then compress or re-rank down to the 5-15 most relevant chunks before the LLM. LongLLMLingua (LLMLingua) uses a small model to remove non-essential tokens from retrieved documents (adjectives, restatements), achieving 4-20x compression with minimal quality loss. Anthropic's 2024 "contextual retrieval" technique prepends a 50-token LLM-generated context summary to each chunk before embedding, reducing retrieval failure rate by 35-49%. Scalar (int8) or binary (1-bit) quantization of embeddings cuts vector DB storage and compute by 4-32x, with typical recall drop of 1-3%, often offset by adding a reranker. Prompt caching, now offered by OpenAI, Anthropic, and Google, reuses a cached prefix for 5-10 minutes, cutting cost up to 90% and latency up to 80%; semantic caching stores previous (query, embedding, answer) tuples and returns a cached answer if a new query is within a similarity threshold of an old one. Token streaming returns LLM tokens as they are generated via server-sent events, so the user sees the answer build up in real time even when the full answer takes 5-10 seconds to complete. The UI also matters: AI search uses an "answer-first" design with the generated answer at the top followed by citations, sources, related questions, and traditional links — the inverse of classic "10 blue links" SERPs. Users now read the AI answer first and click through to verify 1-2 sources; the "zero-click" share of search queries is above 65% per SparkToro (2024).