Calibration in the decision-model era
Turning confidence into type-safe action
A probability from Jev or Laya is not yet safe automation. The missing layer maps uncertainty into types, thresholds, rejection and human escalation.Sep 22, 2026
Conclusion: probability is an input to a contract, not a decision
- Confidence describes how concentrated one output distribution is; calibration asks whether cases assigned the same probability are correct at that rate over time.
- A confidence of 0.90 does not mean “90% correct.” Laya’s own documentation reports a Khmer case with 0.000 accuracy at 0.952 confidence.
- There is no single threshold for a model. Thresholds belong to the loss attached to an action. Reading and deleting, recommending and transferring money cannot share one gate.
- Sign does not make the model correct. It maps a probability into a type-safe set of allowed actions, rejection, re-evaluation and human review.
- Calibration is not a one-time setting. It is an operating process that must detect changing inputs and outcomes.
“Certain” and “well calibrated” are different
TypeSafe’s Jev returns a probability distribution and a confidence value for Choice and Score answers. The official definition is the shape of that distribution: concentrated on one option means high confidence, spread out means low confidence. Noul returns the probability of “yes” directly and has no separate confidence field. [TypeSafe Confidence]
Calibration is a population property. If a model’s 0.8 predictions are actually correct about 80% of the time, the probability is calibrated. A model can be highly confident on every individual answer and still be badly calibrated.
| Concept | What it describes | What it does not guarantee |
|---|---|---|
| Probability | A value assigned to an outcome | That the value repeats in deployment |
| Confidence | Concentration of one answer distribution | Accuracy or in-distribution input |
| Calibration | Agreement between forecast probability and observed frequency | Correctness of any single answer |
| Type guarantee | Output stays inside the allowed shape | Semantic correctness of the selected value |
The dangerous failure is a confident error
Laya’s official model card reports that its English checkpoint scored 0.000 accuracy at 0.952 confidence on Khmer input. It was confidently wrong on a script it could not read. The official Router therefore detects script and language before inference and sends the request to the multilingual checkpoint. [Laya model card]
This is the warning: confidence gating cannot protect you from out-of-distribution input. Language, length, schema, missing fields, date and domain need validation before the model is allowed to judge. Calibration can drift even with a fixed model when customers, products, writing styles or attacks change.
Thresholds belong to losses, not models
TypeSafe proposes three starting ranges: high confidence can act automatically, medium confidence can request confirmation or more context, and low confidence should not act. It also states explicitly that a threshold is not one number. A destructive operation should demand more than a read-only one.
| Action | Cost of error | Example design |
|---|---|---|
| Rank candidate articles | Low | Moderate gate; easy to reverse |
| Route customer mail | Medium | Low confidence returns to a shared queue |
| Send externally | High | High gate, content validation and reversibility |
| Delete, transfer money, change permissions | Extreme | Never model-only; require human approval |
What Sign guarantees
Sign’s role needs a precise boundary. It does not make a judgment true. It guarantees which answer type can be received, under what condition, and which branch targets are legal. A probability becomes an input to a contract; application code decides whether to adopt it.
choice route { billing, technical, sales, other }
threshold read_only # 0.72
threshold external_write # 0.94
p(route) < read_only -> request_more_context
p(route) >= read_only -> route_draft
p(route) < external_write -> human_review
p(route) >= external_write -> signed_action
The same probability now means different things for drafting and external sending. The probability does not call an action directly; only an action that passes a typed contract can run. That extra layer is the boundary between a demo and production.
Do not compress a complex decision into one question
TypeSafe recommends one snap judgment per question: something a knowledgeable person could decide in a second with the right evidence. Instead of “approve this transaction,” ask separately about identity, amount anomaly, prior-pattern match and policy compliance, then compose the results in code. The weights and hard rejection rules remain inspectable. [TypeSafe Primitives]
A seven-point production checklist
- Input eligibility: validate language, length, required fields, freshness and domain before inference.
- Atomic questions: use one judgment per question and compose in code.
- Validation data: keep labeled examples from the real deployment distribution, including severity.
- Calibration curves: inspect by language, customer segment and input type, not only globally.
- Risk-specific gates: separate reading, drafting, external writing and irreversible actions.
- Abstention paths: design “other,” insufficient evidence and out-of-distribution as normal outcomes.
- Monitoring: log model version, input features, probabilities, action and eventual ground truth; recalibrate on drift.
Accuracy alone is not enough
A thresholded system also needs Brier score, expected calibration error, selective accuracy after abstention, and coverage — the share of cases handled automatically. The operational question is the trade-off between accuracy and automation rate. Raising a threshold may improve precision while increasing human review. The optimum comes from the business loss function, not a leaderboard.
Calibration is an operating process
The contribution of decision models is not shorter text. It is exposing uncertainty as a value software can use. Converting that value into safe automation still requires eligibility checks, segment-level calibration, risk gates, abstention, escalation and continuous monitoring.
Sign’s objective is to stop treating probability as magic and express what may be automated — and what must not be — as an executable contract. The decision model supplies a judgment, Sign constrains the permitted action, and a person owns responsibility. Keeping those roles separate is the foundation of safe decision AI.
This is an architecture argument based on TypeSafe AI and Laya public documentation and our Sign Playground implementation. We have not independently validated either model’s calibration. Thresholds shown are explanatory; production values must come from each environment’s loss function and validation data.