The Relevan search stack
Good search doesn’t come from one clever model. It comes from a stack of small decisions, and any one of them can quietly cost you relevance. Here’s the stack we run at Relevan, in the order a query meets it, with one detail per layer you can take back to your own system.
- 01Mapping and ingestthe index the query meets
- 02Read the querytokens, phrases, places
- 03Lexical retrievalBM25, weights, fuzzy, prefix
- 04Hybrid gatelexical only, or add vectors
- 05Rerank the top 30per-index LambdaMART
- 06Log the searchone UBI record
- 07FeedbackKinesis, Firehose, S3
- 08Scorecardwhat to fix next
1. The mapping and ingest
Everything downstream depends on the mapping. Ours says what each field is for: searchable, filterable, autocompletable, and so on. Your agent drafts it from a sample of your documents, and we turn it into OpenSearch analyzers and subfields. Documents land in Postgres first, and a worker bulk-indexes them into OpenSearch from a queue.
The detail: draft the mapping from 100 to 1,000 documents from different sources and dates, and include documents with optional fields. The field you didn’t sample is the one that breaks, and changing an analyzer later means a reindex.
2. Reading the query
First, classifiers tag the query: token count, language, quoted phrases, negation, and questions. On indexes that mark fields with an entity type, a GLiNER model also looks for places. “Software jobs in Virginia” becomes a text query plus a location to match against the index.
The detail: give every enrichment a hard time budget. Entity detection gets 150 ms, and inference on one vCPU takes 62 to 85 ms for a short query. If it fails or runs long, the search goes ahead with plain text.
3. Lexical retrieval
BM25 still carries most of the load, and we tune it per index. k1 and b live in Relevan’s own config, outside your mapping, together with field weights, fuzziness, and prefix matching.
The detail: OpenSearch’s AUTO fuzziness allows two edits on a six-letter word, so “womens” matches “mens” and “blouse” matches “blue”. We default to AUTO:3,8, which saves two edits for words of eight letters or more.
Field weights follow the shape of the corpus. When an index reaches 1,000 documents, we profile a sample. A catalog gets its name field boosted ×2, and prose gets its title cut to ×0.5. Offline, the name boost helped both catalogs we tested and hurt both prose sets, so the profile needs two catalog signals before it applies the boost.
Prefix matching uses an edge n-gram subfield of 2 to 20 characters. Keep preserve_original on, or a word longer than the largest gram never matches itself.
4. Hybrid, switched on per query
Vectors help natural-language queries and get in the way of part numbers, so hybrid retrieval has a per-query gate. A query stays lexical if it matches one of the index’s identifier patterns (a SKU format, say), holds a quoted phrase, or has fewer than three tokens. Everything else runs BM25 and a k-NN query with k = 100. Both scores are min-max scaled to 0 to 1, and the vector side gets 60% of the combined score by default.
The detail: a lexical query makes no embedding call, so the gate saves latency and cost too. We log the decision and its reason on every search, so we can measure each kind of query separately.
5. Reranking the top 30
On the first page, we fetch the top 30 candidates and reorder them with a per-index LambdaMART model, trained with LightGBM and stored as flat trees. One extra OpenSearch query scores named clauses on those 30 documents, and the model runs inside the API process. The step has a 300 ms budget, and on a local single node it takes 17 ms at p50.
The detail: you need more judged queries than you think. On Amazon ESCI, 302 queries with human labels gave +0.001 nDCG@10, and 1,000 queries gave +0.016. On WANDS, 480 queries took nDCG@10 from 0.758 to 0.770. Today we train each index’s model offline from feedback, and scheduled training is planned.
6. Logging every search
Every search writes one record in the UBI (User Behavior Insights) format. It holds the query, the top hits with position, id, and score, and a relevan extension with the retrieval mode and whether a fallback query ran.
The detail: use a standard schema and put your own fields in its extension point, so standard tools can still read the records. We cap each record at 50 hits to stay under the 1 MB Kinesis record limit.
7. Feedback to S3
Feedback comes in three types, click, conversion, and llm_judgment, with an optional grade from 0 to 2. Search and feedback events go to one Kinesis stream. Firehose buffers them for 60 seconds and writes gzipped NDJSON to S3, partitioned by organization, index, date, and event type. We keep the events for 13 months.
The detail: the caller doesn’t choose how much we trust a signal. We map each type to a tier: an agent’s judgment is low, a click is medium, and a conversion is high. Eval traffic carries its own flag, so a test suite never looks like real users.
8. The scorecard
The scorecard reads a window of those events (7 days by default) for one index. It reports zero-result queries, low-confidence queries where the top hit barely beats the rest, rewrite chains (similar queries retried within five minutes), and the fields people filter on. We run it on demand today.
The detail: run each zero-result query again with filters dropped and matching relaxed. If it still finds nothing, the document doesn’t exist. If it finds something, the query strategy is the problem. At Oberdex, zero-result queries went from 27 unique queries to 5, and none of those 5 has a matching document to find.
Where to start
None of these layers is exotic. The work is in the defaults, the time budgets, and the logs that tell you which layer to fix next. If you’re building your own stack, start with the logs.
Want to see what this stack does with your data? Try our free tier, or book a call.