Structured objects from System One decision models. Schema-guaranteed, calibrated, and field-level confident.
jsonformer (2023) proved you don't trust a language model with structural tokens: fill the JSON skeleton in code, generate only content tokens. deciform applies the same philosophy to the System One era: don't use a generative model for typed fields at all. Compile your schema into a batched program of choice / noul / score questions, run it on any decision-model backend, and escalate only the fields the model can't confidently decide to an LLM.
Your pipeline makes thousands of small structured decisions a day. Triage a ticket, flag a transaction, route a document, grade a lead. Each one is currently either an LLM call you overpay for and can't calibrate, or a decision-model call plus a few hundred lines of glue you wrote by hand, mapping fields to questions, batching, gating optional branches, thresholding confidence, reassembling the object, logging what happened. deciform is that layer, written once. If your whole output is a single yes/no, use the raw API; if your schema is mostly free text, you want a structured-output tool, not this.
from pydantic import BaseModel, Field
from typing import Literal
from deciform import Deciform, EscalationPolicy, SystemOneBackend
from deciform.backends import OpenAIGenerativeBackend
class TicketTriage(BaseModel):
category: Literal["billing", "bug", "account", "feature_request", "other"]
is_urgent: bool
sentiment: Literal["angry", "neutral", "positive"]
priority: int = Field(ge=1, le=5) # bounded int -> score
needs_human: Literal["none", "support", "engineering"] | None # Optional -> presence gate
reply_draft: str = "" # free text -> fallback LLM
sf = Deciform(
backend=SystemOneBackend(model=SystemOneBackend.ELEPHANT), # any /v1/systemone endpoint
fallback=OpenAIGenerativeBackend( # any OpenAI-compatible LLM
base_url="https://your-llm-endpoint/v1",
api_key="...", model="your-chat-model"),
escalation=EscalationPolicy(threshold=0.8, debias_k=3),
)
result = sf.extract(TicketTriage, state=ticket_json)What you get back is a validated object plus a per-field receipt:
result.value # TicketTriage(category='billing', is_urgent=True, ...)
result.fields[2] # FieldTrace for 'sentiment': backend='elephant', answer.top_probability=0.94
result.escalated # ['priority', 'reply_draft'] <- fell below 0.8 / not a decision primitive
result.usage # {'requests': 3, 'prompt_tokens': ..., 'total_tokens': ...}Runnable version with zero API keys in examples/ticket_triage.py; a real-endpoint wiring in examples/expense_review.py.
sf.extract(Model, state)
│ schema walk ┌───────────────────┐
compile(Model) ────────────────► │ question program │ rounds + gates
│ └─────────┬─────────┘
│ batched evaluate() one request / round
│ ┌─────────▼─────────┐
│ │ probabilities in │ per-field answers
│ confident fields └──────┬──────┬──────┘ uncertain fields
└────── keep decision │ └─► GenerativeBackend (escalation)
└─────────────────────► assemble → Pydantic validate → DecisionResult
| Schema element | Compiled to |
|---|---|
Literal[...] / Enum |
choice (option-order debiasing optional) |
bool |
noul |
bounded int (ge/le/Field) |
score (ordinal levels; level_labels via json_schema_extra) |
str / unbounded numbers |
generative fallback, schema-hinted |
list[Literal[...]] |
multinoul, one noul per member |
Optional[T] |
presence noul → child questions in a lazy later round |
nested BaseModel |
question group sharing the state prefix |
- Calibrated escalation:
EscalationPolicy(threshold=…, per_field={…});policy.fit(calibration_rows, target_error=0.05)picks per-field τ on your labeled data (Wilson-bound acceptance, conformal-flavored, honest about what the model can't promise). - Order debiasing:
debias_k=3rotates options and averages probabilities (the literature reports ~30% flip rates on some open models; measured 2–9% on elephant/mammoth). - Per-field traces: every field's questions, backend, top probability, latency, escalated-or-not, skipped-by-gate-or-not.
- Token accounting:
result.usageaggregates prompt/completion/total tokens across every call in the extract (when the API reports it). - Async:
await sf.aextract(Model, state)runs the sync pipeline in a worker thread; rounds are sequential by design, so the win is concurrency across calls in async services. - Backend-agnostic:
SystemOneBackendfor any/v1/systemoneprovider;OpenAIGenerativeBackendfor any OpenAI-compatible chat endpoint;LLMAsJevBackendturns any chat LLM withlogprobsinto a decision backend via first-token option-letter readout.
| task | elephant-1 (9B) | mammoth-1 (27B) |
|---|---|---|
| banking77 intent (77 options) | acc .929 · ece .013 · flip .024 | acc .937 · ece .010 · flip .020 |
| RTE entailment | .895 | .946 |
| fintech suite (5 fields) | violation .890 · flags .781 | violation .945 · flags .915 |
Plus: LLM-as-Jev readout verified on a 30B chat model; escalation Pareto showing a +1.5pp bump at ~25% fallback spend when the fallback is strong enough to deserve it. Full numbers and analysis in REPORT.md; raw records in results/.
pip install -e . # core: pydantic + httpx only
pip install -e .[eval] # benchmark harness: datasets + matplotlib
pip install -e .[local] # future local-encoder backends: torch + transformers# benchmarks: banking77 · sst5(_labeled) · rte · procedural fintech
python scripts/run_eval.py --tasks fintech rte --backends elephant mammoth \
--limit 500 --workers 8 --out results --fit-error 0.05
# escalation Pareto: decision model decides, LLM covers the low-confidence tail
python scripts/escalation_experiment.py --task fintech \
--llm-model nvidia/nemotron-3-ultra-550b-a55b --tag ultra
# summaries + coverage plots
python scripts/summarize.py --results resultsv0.x. Core pipeline, both backend kinds, eval suite, and threshold fitting are live-verified. Still coming: a published PyPI name and richer cost metering per field.
Named in homage to jsonformer by Rahul Sengottuvelu. Same thesis, next model class: the structure is yours; let the model do only what it must.