Skip to content

perf(models): reduce discriminated streaming event validation - #2045

Open
Linxiushen wants to merge 2 commits into
anthropics:mainfrom
Linxiushen:integrated/core-stream-1649
Open

Linxiushen wants to merge 2 commits into
anthropics:mainfrom
Linxiushen:integrated/core-stream-1649

Conversation

@Linxiushen

Copy link
Copy Markdown

This addresses the union-decoding part of #1649: each streaming event currently attempts validation against the ordinary union before falling back to the SDK discriminator. For eligible Pydantic v2 unions, this caches a tagged-union validator built from the existing core schema and selects the event branch by its literal tag.

The shortcut requires model branches with disjoint string tags, consistent simple aliases, and schemas without validation callbacks. A bounded JSON-value check excludes iterators and custom objects. Unsupported schemas, unknown tags, invalid payloads and larger values retain the existing whole-union validation and loose-construction fallback. Pydantic v1, strict response validation and the existing fallback discriminator cache are unchanged. Event variants are derived from the schema rather than a hardcoded event list.

Measurements and tradeoffs

An offline benchmark on one Windows host with Python 3.12.14 / Pydantic 2.12.5 compared 202db72 with this implementation. Two alternating baseline/candidate process pairs each ran seven measured repetitions with default deferred schemas. The stream cases use the actual stable/beta AsyncAnthropic.messages.stream().get_final_message() path through httpx2.MockTransport, with 3,000 text deltas and assertions on final text, usage and stop reason.

Median of process medians, in microseconds per event:

Workload Baseline Candidate Elapsed-time change
Warm text-delta decode 15.37 10.95 -28.8%
Complete stable mock stream 59.10 54.37 -8.0%
Complete beta mock stream 63.40 55.92 -11.8%
Large content-block-start decode 26.03 28.36 +9.0%
Large message-start decode 30.03 36.19 +20.5%

The two large cases contain 1,000 nested tool-input rows, exceed the eligibility budget, and pay the guard overhead before fallback. Cold first-decode samples were 13.22/12.90 ms before and 14.40/14.67 ms after; two samples per version are insufficient for a cold-latency conclusion. These results describe a text-delta-heavy synthetic workload, not API/model latency or a universal speedup. In particular, the 28.8% result applies only to isolated decoding.

Benchmark script and all four result files (local source paths omitted) are kept outside this PR's source diff.

Validation

Linux validation for exact candidate cb46b2a passed:

  • ./scripts/lint and uv build.
  • ./scripts/test tests/test_models.py tests/test_client.py tests/test_response.py tests/test_streaming.py tests/lib/streaming/test_messages.py tests/lib/streaming/test_beta_messages.py -n 2: Python 3.10/Pydantic v2 and Python 3.14/Pydantic v2 each had 453 passed / 2 skipped; Python 3.10/Pydantic v1 had 436 passed / 19 skipped. The script's MCP v2 suite also passed 39 tests on each Python version.
  • Added regression coverage for stable/beta event decoding, malformed and unknown values, aliases, duplicate tags, iterators, defaults, and callback side effects, including enum and nested callable discriminators. A separate 104-case first-call/warm-cache comparison matched baseline values, concrete types, extras and field sets.

This picks up the union-decoding direction from withdrawn #1695, following the invitation to continue it. It does not change stream draining/aggregation (#1663) or fallback discriminator caching (#2025), and does not close the broader #1649.

The documented generator-trailer comparison was attempted, but the referenced output commit could not be fetched (not our ref, also 404 via API). Existing lines are therefore not claimed to be handwritten; the change adds a guarded path in the mixed generated module while preserving its fallback.

Release note: Reduce Pydantic v2 decoding overhead for eligible discriminated streaming events, with workload-dependent eligibility-check costs.

@Linxiushen
Linxiushen requested a review from a team as a code owner October 29, 2026 12:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant