You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Serving benchmark: cold fit, warm predict, many tenants #104
Publish a serving benchmark for tabular foundation models. The existing comparisons are accuracy leaderboards. Nobody publishes what it costs to serve these models.
Measure
Cold fit: time by rows × features.
Warm predict: p50 and p99 latency against a cached context, by batch size.
Many tenants: throughput and tail latency with N tenants and M tables on one GPU, including eviction and restores from disk.
Baselines: the model in your own process, and the hosted API where the terms allow it (benchmarks/bench_vs_hosted_api.py).
Deliver
One script that runs the whole suite and writes JSON, reusing bench_workload.py, sweep_admission.py and bench_concurrency.py.
Results on at least L4 and A40 in docs/benchmarks.md, with the exact commands to reproduce them.
Activity