Skip to content

GPU profiling harness: per-stage time and peak-memory breakdown (GKE first, then A40) #31

Open

Description

@amrit110

Why

Every CUDA-level item on the roadmap (#32, #33, #34, #35, #36) should be chosen and sized from profiles, not guesses. The single-L4 sweep showed admission limits scaling with cells (rows × features), not rows², which suggests per-cell activations in TabICL's column-embedding and row-interaction stages dominate the fit peak more than attention. That's a hypothesis to confirm.

Tasks

  • Profiling harness (benchmarks/profile_fit_predict.py): torch profiler (CUDA time, memory timeline per stage: col_embedder, row_interactor, icl_predictor) plus an Nsight Systems recipe, over the sweep's shapes (10–1,000 features, up to the admission limit) for TabICLv2 and TabPFN-3.5, in kv and repr modes.
  • Record what TabICL's own InferenceManager already does (stage batching from its activation model, CPU/disk offload, async copies) and how it interacts with tabctx's kv mode.
  • Run on a GKE GPU node first (L4 and A100), then on the prod A40 once tabctx is deployed (sm_86: no FA3, no FP8).
  • Write up: top kernels by time, peak-memory breakdown by stage and tensor, and a ranked target list for the kernel issues.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium: plannedarea/latencyFit and predict timeenhancementNew feature or requestroadmapTracked in ROADMAP.md

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions