Skip to content

Limits and evaluation

Use the component to rank supplied passages, and evaluate the complete application on representative authorized data. Public relevance diagnostics do not establish accuracy for your customers.

What ranking does not establish

The model does not retrieve documents, generate answers, detect missing answers, check facts, enforce access or tenant boundaries, or verify freshness and completeness. Apply those controls in your application before ranking and review the source afterward. Full legal documents and arbitrary long-input retrieval are outside the bounded passage API.

Human-judged public diagnostics

The frozen comparison retains original human relevance grades and full judged pools. It contains 1,517 original queries and 15,845 pairs. The bounded API admits 1,343 queries and 13,699 pairs. All 174 unsupported pools remain disclosed; no labeled passage was removed to improve a score.

Admitted source Queries Original-model Thor nDCG@5
MIRACL English dev 713 / 799 0.82882
MIRACL Spanish dev 606 / 648 0.82679
Quati Portuguese human supplement 24 / 24 0.73452
NormasTCU Portuguese full documents 0 / 46 Not admitted

nDCG@5 measures ordering within supplied judged pools, not corpus-wide retrieval recall, answer completeness or customer accuracy. The 24-query Portuguese slice is small. The sources are public and are not sealed upstream-training holdouts. Independent cross-language and native business-customer acceptance remain unmeasured.

Actual AWS checks

The qualified immutable runtime produced all 1,343 expected native outputs on each of A10G real-time and L40S batch. Seven real-time rejection probes passed, and Marketplace model-package certification completed. Both native runs passed the pre-frozen portability tolerance against Thor; small numerical/backend reorderings occurred.

Sequential real-time client round-trip p50 was 136 ms and p95 was 205 ms after startup. These are us-east-1 qualification measurements, not maximum capacity, throughput, an SLA or a customer-workload guarantee.

The package uses original BAAI weights without additional Software Sushi training. Earlier adapted complete-answer experiments failed their targets; those failures were not relabeled as relevance successes. Video examples are fictional development cases and do not replace customer acceptance.

Read the full evaluation card and sources and licenses for detailed coverage and distribution scope. Raw native evaluation texts are not shipped in the model or examples.

Last updated November 15, 2026 · Suggest an improvement