Skip to content

Serving benchmark: cold fit, warm predict, many tenants #104

Open

Description

@amrit110

Publish a serving benchmark for tabular foundation models. The existing comparisons are accuracy leaderboards. Nobody publishes what it costs to serve these models.

Measure

  • Cold fit: time by rows × features.
  • Warm predict: p50 and p99 latency against a cached context, by batch size.
  • Many tenants: throughput and tail latency with N tenants and M tables on one GPU, including eviction and restores from disk.
  • Baselines: the model in your own process, and the hosted API where the terms allow it (benchmarks/bench_vs_hosted_api.py).

Deliver

  • One script that runs the whole suite and writes JSON, reusing bench_workload.py, sweep_admission.py and bench_concurrency.py.
  • Results on at least L4 and A40 in docs/benchmarks.md, with the exact commands to reproduce them.

Activity

  1. added
    enhancementNew feature or request
    roadmapTracked in ROADMAP.md
    P2Medium: planned
    help wantedExtra attention is needed
    on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium: plannedarea/latencyFit and predict timeenhancementNew feature or requesthelp wantedExtra attention is neededroadmapTracked in ROADMAP.md

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions