At the heart of modern AI search is the embedding — a numerical vector representation of text, images, or other content that captures its semantic meaning. Embeddings allow AI systems to find conceptually similar content even when exact keywords do not match, which is precisely the limitation of older lexical search methods. Lexical search matches exact words and phrases; semantic search understands meaning and concepts, retrieving relevant results even when no keywords overlap.
Dense retrieval generalizes this idea by encoding both queries and documents into vector embeddings and matching them via cosine similarity, in contrast to sparse retrieval (such as BM25) which matches exact lexical tokens. BM25 itself remains a strong baseline and is still relevant in 2025 AI pipelines because it excels at exact-match recall on rare terms — a property that dense retrieval alone can struggle to preserve. Dense Passage Retrieval (DPR), introduced by Karpukhin et al. in 2020, was the first dense retriever to outperform BM25 on open-domain question answering, using two BERT encoders (one for the question, one for the passage) trained with contrastive loss on QA pairs.
Modern embedding systems rely on standardized benchmarks to measure progress. The Massive Text Embedding Benchmark (MTEB) covers 56 datasets across retrieval, classification, clustering, and semantic textual similarity, and is the de facto leaderboard for general-purpose embeddings. State-of-the-art open models in 2025 include BGE-M3, E5-Mistral-7B, and Nomic-Embed-Text-v1.5, all of which score above 65 on MTEB for English and offer strong multilingual performance. Typical embedding dimensions today are 384, 768, 1024, or 1536; OpenAI's text-embedding-3-small supports 512 or 1536, while text-embedding-3-large supports up to 3072. Matryoshka Representation Learning (MRL) trains embeddings so that the first \( d \) dimensions form a useful embedding on their own, allowing a single model to serve multiple storage and recall budgets (e.g. 64, 128, 256, 512) without retraining.
The advantages of AI search over keyword search are direct consequences of these technical foundations: AI search understands context, intent, and semantic meaning rather than just matching keywords, providing more relevant and personalized results. This semantic depth is also why Information Retrieval is foundational to broader AI development, particularly the five new retrieval tasks introduced in the context of Artificial General Intelligence — chief among them External Information Retrieval, which allows AI agents to access new information that was not seen during training, enabling continuous learning and up-to-date knowledge retrieval.