BM25

What is BM25?

BM25, also known as Okapi BM25, is a probabilistic information retrieval ranking function that scores documents according to how well their text matches a search query. It considers factors including how frequently query terms appear in a document, how rare those terms are across the document collection and the document’s length.

These factors help search systems rank matching documents by estimated relevance, making BM25 a widely used method for full-text search and enterprise document retrieval.

Key Features

Term Frequency: Considers how often a query term appears in a document, while limiting the benefit of repeated occurrences.

Inverse Document Frequency: Gives greater weight to terms that occur in fewer documents across the collection, helping distinguish more informative search terms.

Document Length Normalization: Adjusts scoring to account for document length so that longer documents do not automatically receive higher scores simply because they contain more words.

Relevance-Based Ranking: Produces scores that search systems can use to order matching documents, placing higher-scoring results ahead of lower-scoring ones.

Configurable Parameters: Allows implementations to adjust factors such as term-frequency saturation and document-length normalization to suit different search requirements.

Benefits

More Relevant Search Results: Helps prioritise documents containing meaningful matches to a user's query.

Efficient Lexical Retrieval: Provides a practical ranking method for searching large text collections without requiring a language model to interpret every query-document pair.

Improved Document Discovery: Helps users locate policies, contracts, reports and other records within repositories using terms found in their content.

Scalable Search Implementation: Is supported by widely used search technologies, including Elasticsearch and Apache Lucene.

Foundation for Hybrid Search: Can serve as the lexical retrieval component in systems that combine keyword-based ranking with vector search or other relevance methods.

Conclusion

BM25 is a foundational ranking method for text-based information retrieval. By balancing query-term frequency, term rarity and document length, it helps search systems order documents according to estimated relevance.

In document management environments, BM25 can support full-text search and serve as a baseline for more advanced retrieval pipelines. Its effectiveness still depends on factors such as text processing, query quality, indexed content and the relevance criteria of the application.