You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Capacity: INT8 context KV cache with in-kernel dequantization #34
Resident context KV caps how many contexts (users) a GPU holds. TabPFN already stores its kv cache at reduced precision; TabICL's is full precision.
Idea
INT8 context KV (per-head or per-block scales), dequantized inside the attention kernel. That's about 2× the contexts per GPU. A40/A100/L4 have INT8 tensor cores; FP8 is Hopper-only, so FP8 would be an optional path.
Tasks
Accuracy study: prediction agreement and probability drift vs full precision across the sweep's shapes (the TabPFN kv path drifts ~1e-2 with identical labels, per the integration test notes)
Quantize at fit and dequantize in the predict kernel (Triton), integrated with cache accounting
Measure contexts per GPU and predict latency, before vs after; GKE GPU node first
Activity
amrit110 commentedon Oct 18, 2026
Close after verifying on the prod A40 (once inference-platform#168 is deployed):
benchmarks/eval_kv_precision.pyon an A40 (sm_86): label agreement and probability drift, int8 vs fp16, both models