Skip to content

Capacity: INT8 context KV cache with in-kernel dequantization #34

Open

Description

@amrit110

Why

Resident context KV caps how many contexts (users) a GPU holds. TabPFN already stores its kv cache at reduced precision; TabICL's is full precision.

Idea

INT8 context KV (per-head or per-block scales), dequantized inside the attention kernel. That's about 2× the contexts per GPU. A40/A100/L4 have INT8 tensor cores; FP8 is Hopper-only, so FP8 would be an optional path.

Tasks

  • Accuracy study: prediction agreement and probability drift vs full precision across the sweep's shapes (the TabPFN kv path drifts ~1e-2 with identical labels, per the integration test notes)
  • Quantize at fit and dequantize in the predict kernel (Triton), integrated with cache accounting
  • Measure contexts per GPU and predict latency, before vs after; GKE GPU node first

Activity

  1. amrit110 commented on Oct 18, 2026

    @amrit110
    MemberAuthor

    Close after verifying on the prod A40 (once inference-platform#168 is deployed):

    • benchmarks/eval_kv_precision.py on an A40 (sm_86): label agreement and probability drift, int8 vs fp16, both models
    • Resident bytes per context and contexts per GPU, int8 vs fp16
    • Predict latency int8 vs fp16 (dequantization is per layer in PyTorch, not inside the attention kernel; a kernel follow-up only if this shows a cost)
  2. added
    P2Medium: planned
    area/memoryGPU and host memory, admission, calibration
    needs-prod-verificationClose only after it is verified on the prod A40 deployment
    on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium: plannedarea/memoryGPU and host memory, admission, calibrationenhancementNew feature or requestneeds-prod-verificationClose only after it is verified on the prod A40 deploymentroadmapTracked in ROADMAP.md

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions