Repository navigation
Open
Labels
P1High: next upHigh: next uparea/latencyFit and predict timeFit and predict timearea/memoryGPU and host memory, admission, calibrationGPU and host memory, admission, calibrationenhancementNew feature or requestNew feature or request
Description
On prod (4× A40, tabicl-v2), fit time jumps 3.2× for 1.5× more data: 500×20,000 takes 34.8 s, 500×30,000 takes 111.8 s. That's where tabicl's automatic offload starts copying stage outputs to pinned host memory. During the 500×65,000 fit, /readyz showed calls_offloading: 1 and a 162.5 GB host reservation for about 10 minutes.
TabICLBackend.may_offload (backends/tabicl.py:436) mirrors tabicl's trigger: stage output (about 2.5 KB per cell, calibrated on an A100) above 0.4 × the currently free GPU memory. So:
- other contexts cached on the chosen GPU make a large fit offload at smaller sizes, even when that GPU (or another) could be cleared;
- on the A40 (48 GB) the threshold sits near 10M cells only when the card is nearly empty.
Options to measure
- Before a fit that would offload, spill or evict cached contexts on the chosen GPU (the spill tier exists) so the stage outputs fit on the GPU, if that's cheaper than offloading. Compare the spill and restore cost with the offload cost.
- In placement, prefer the GPU with the most free memory for fits near the threshold, not just one that fits.
- Keep offloaded or staged outputs in bf16 to halve their size, if accuracy holds (
eval_kv_precision.py-style check). - Recalibrate
_OUTPUT_BYTES_PER_CELLon the A40 (Per-device TabICL calibration grids (L4, A40) for admission #37) and check whether the 0.4 factor is too cautious. - Fixed chunk sizes for column and row stages, so the peak stays bounded without full offload (Large tables: tiled/fused column+row stage kernels to cut the fit peak #32).
Accept
- 500×30,000 within 1.5× of the per-cell rate of 500×20,000 on an otherwise idle A40.
- No added latency for small fits and predicts (placement and eviction must not stall them).
- Host-memory accounting still correct (Host-memory reservation exceeds its budget and overestimates real use 8× #52).
Metadata
Metadata
Assignees
Labels
P1High: next upHigh: next uparea/latencyFit and predict timeFit and predict timearea/memoryGPU and host memory, admission, calibrationGPU and host memory, admission, calibrationenhancementNew feature or requestNew feature or request
Activity
[-]P1 latency: avoid the offload cliff on large fits (3× slower past ~10M cells)[/-][+]Avoid the offload cliff on large fits (3× slower past ~10M cells)[/+]