Skip to content

Limits and evaluation

These figures are from the published evaluation scope for version 1.0.0. Scores and review flags do not guarantee a correct label. This page is not a claim that the model is safe to route tickets on its own.

AWS timing

Measured from AWS in the endpoint's Region on one network-isolated ml.c5.xlarge, using synthetic tickets of 75 to 145 characters:

Measurement Result
Warm serial p95, 200 requests after 10 warmups 76.3 ms
Successful throughput, two concurrent requests 28.9 tickets/s

Cold starts, large documents, and maximum capacity were not measured. These numbers are not an SLA.

The unchanged 81-ticket synthetic regression passed the publisher's internal classification gates. It repeats 27 scenarios across Spanish, Portuguese, and English. It is not a representative customer sample.

Expanded synthetic diagnostic

The frozen 2,000-case set has 1,800 core tickets and 200 challenge cases. All labels were authored and separately reviewed by AI. None are human-reviewed. Earlier failures informed later changes that used separate training and development data. Repeating this set is a diagnostic. It is not a fresh blind benchmark and it is not an estimate of accuracy on your tickets.

Each core language has 600 cases. Five ambiguous Portuguese sentiment labels are excluded only from that task's denominator.

Language Intent macro-F1 Urgency macro-F1 Sentiment macro-F1 Language accuracy
EN 0.899 0.868 0.869 100.0%
ES 0.893 0.833 0.873 99.8%
PT 0.871 0.847 0.862 100.0%

Review flags miss cases

On the challenge cases the evaluation expected to flag:

Case Flagged / expected
Long context 45/45
Multiple intents 11/21
Short or noisy 4/10
Unsupported or mixed language 19/20

Review flags miss some ambiguous and short or noisy requests. An unflagged prediction can still be wrong. Keep a human review path, and test the model on tickets from your own queue before you automate any routing. Do not read a score as a calibrated probability.

The model does not redact personal data and does not make a customer-service decision for you.

Other bounds

Spanish, Portuguese, and English only. Unknown and mixed-language messages need review. Regional slang, multiple intents, and unusual domains need their own check. The first 128 tokenizer tokens are what the encoder sees. Custom labels are experimental, and the quality targets above apply to the default taxonomy only.

Last updated November 11, 2026 · Suggest an improvement