You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Multi-GPU replica: one engine per device with workload-aware placement #30
A replica can only use one GPU today. Prod gives tabctx a whole A40 node (4 × 48 GB), and the expected workload is a few concurrent users with a mix of small and large tables. Neither TabICL nor TabPFN can split one context across GPUs: TabICL is single-device, and TabPFN's device lists only parallelize ensemble members, and not in fit_with_cache.
Design (in progress on feat/multi-gpu-engine)
MultiDeviceEngine: one complete TabctxEngine per device, each with its own backends bound to cuda:i, cache, lock and spill subdirectory, behind the same public API.
Placement: prefer devices where the table fits without eviction, and among those the least active one. If none fits without eviction, use the device with the most headroom.
Predicts route to the owning device. After a restart, spilled contexts are found by scanning devices.
Unit and integration tests (fake backend, multiple devices)
Real multi-GPU run on a GKE GPU node: device scoping, per-device memory measurement, mixed-workload benchmark (benchmarks/bench_workload.py), L4 sweep re-run
Release, then inference-platform#168 switched to the whole A40 node
Activity
feat: multi-GPU replicas with workload-aware placement (0.12.0) (#38)