Limits and evaluation¶
These figures are from the published evaluation scope for version 1.0.0. Scores and review flags do not guarantee a correct label. This page is not a claim that the model is safe to route tickets on its own.
AWS timing¶
Measured from AWS in the endpoint's Region on one network-isolated
ml.c5.xlarge, using synthetic tickets of 75 to 145 characters:
| Measurement | Result |
|---|---|
| Warm serial p95, 200 requests after 10 warmups | 76.3 ms |
| Successful throughput, two concurrent requests | 28.9 tickets/s |
Cold starts, large documents, and maximum capacity were not measured. These numbers are not an SLA.
The unchanged 81-ticket synthetic regression passed the publisher's internal classification gates. It repeats 27 scenarios across Spanish, Portuguese, and English. It is not a representative customer sample.
Expanded synthetic diagnostic¶
The frozen 2,000-case set has 1,800 core tickets and 200 challenge cases. All labels were authored and separately reviewed by AI. None are human-reviewed. Earlier failures informed later changes that used separate training and development data. Repeating this set is a diagnostic. It is not a fresh blind benchmark and it is not an estimate of accuracy on your tickets.
Each core language has 600 cases. Five ambiguous Portuguese sentiment labels are excluded only from that task's denominator.
| Language | Intent macro-F1 | Urgency macro-F1 | Sentiment macro-F1 | Language accuracy |
|---|---|---|---|---|
| EN | 0.899 | 0.868 | 0.869 | 100.0% |
| ES | 0.893 | 0.833 | 0.873 | 99.8% |
| PT | 0.871 | 0.847 | 0.862 | 100.0% |
Review flags miss cases¶
On the challenge cases the evaluation expected to flag:
| Case | Flagged / expected |
|---|---|
| Long context | 45/45 |
| Multiple intents | 11/21 |
| Short or noisy | 4/10 |
| Unsupported or mixed language | 19/20 |
Review flags miss some ambiguous and short or noisy requests. An unflagged prediction can still be wrong. Keep a human review path, and test the model on tickets from your own queue before you automate any routing. Do not read a score as a calibrated probability.
The model does not redact personal data and does not make a customer-service decision for you.
Other bounds¶
Spanish, Portuguese, and English only. Unknown and mixed-language messages need review. Regional slang, multiple intents, and unusual domains need their own check. The first 128 tokenizer tokens are what the encoder sees. Custom labels are experimental, and the quality targets above apply to the default taxonomy only.
Last updated November 11, 2026 · Suggest an improvement