Skip to content

Multi-GPU replica: one engine per device with workload-aware placement #30

Closed

Description

@amrit110

Why

A replica can only use one GPU today. Prod gives tabctx a whole A40 node (4 × 48 GB), and the expected workload is a few concurrent users with a mix of small and large tables. Neither TabICL nor TabPFN can split one context across GPUs: TabICL is single-device, and TabPFN's device lists only parallelize ensemble members, and not in fit_with_cache.

Design (in progress on feat/multi-gpu-engine)

  • MultiDeviceEngine: one complete TabctxEngine per device, each with its own backends bound to cuda:i, cache, lock and spill subdirectory, behind the same public API.
  • Placement: prefer devices where the table fits without eviction, and among those the least active one. If none fits without eviction, use the device with the most headroom.
  • Predicts route to the owning device. After a restart, spilled contexts are found by scanning devices.
  • TABCTX_DEVICES=auto|cuda:0,.... Per-device /readyz breakdown.

Done when

  • Unit and integration tests (fake backend, multiple devices)
  • Real multi-GPU run on a GKE GPU node: device scoping, per-device memory measurement, mixed-workload benchmark (benchmarks/bench_workload.py), L4 sweep re-run
  • Release, then inference-platform#168 switched to the whole A40 node
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestroadmapTracked in ROADMAP.md

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions