Repository navigation
Open
Labels
P3Low: later, or waiting on a needLow: later, or waiting on a needarea/servingServing layer: process model, routing, timeouts, healthServing layer: process model, routing, timeouts, healthroadmapTracked in ROADMAP.mdTracked in ROADMAP.md
Description
Why
Everything needed for a researcher-facing deployment is staged in VectorInstitute/inference-platform on branch test/tabicl-gke-onboard: the research overlay (helm-charts/ray-llm-app/values/tabctx-research-values.yaml: both models, one A100, spillover, 64Gi host RAM per the measured host-OOM boundary), the probe battery (tests/gke-tabicl-test/probe_*.py), and a parameterized onboard.sh. ROADMAP item 2 sequences a dedicated ephemeral A100 stress run after the release, followed by the production inference-platform deployment for Vector researchers.
What changed since that plan
- v0.10.0 was validated on the standing
tabctx-gatewayGKE cluster (one L4) withbenchmarks/probe_deployment.py(24/24). That probe handlesTABCTX_REQUIRE_TENANT=trueand a renamed session header, which the inference-platform probes do not -- they send neither a tenant header nor the renamed session header, so they 401 against that gateway. - Redeploying a same-named wheel requires restarting the head pod, not just the worker: Ray caches pip runtime envs by package spec per node, and the Serve controller ships the ingress-wrapped class from the head's cached env by value (CHANGELOG 0.10.0 "Validated").
Tasks
- Run the A100 stress battery (
probe_stress_v09.py,probe_soak.py,bench_concurrency.py) against a v0.10.0 wheel on an ephemeral A100 cluster viaonboard.sh(KEEP_ALIVE=false; it tears down on exit -- real A100 cost). Compare againstbenchmarks/baselines/v0.9.1-research-1replica.json. - Bump the wheel filename pinned in both overlays (
/wheels/tabctx-0.9.1-py3-none-any.whl) to 0.10.0. - Either port the inference-platform probes to
TabctxClient(tenant_id=..., session_header=...)or retire them in favour ofbenchmarks/probe_deployment.pyplus the stress/soak scripts. - Document the head-pod restart requirement in
onboard.sh/ the overlay comments, or make the wheel filename carry a build id so the pip spec changes per deploy. - Production deployment for researchers on inference-platform.
Pointers
ROADMAP.md item 2, benchmarks/README.md, benchmarks/baselines/.
Metadata
Metadata
Assignees
Labels
P3Low: later, or waiting on a needLow: later, or waiting on a needarea/servingServing layer: process model, routing, timeouts, healthServing layer: process model, routing, timeouts, healthroadmapTracked in ROADMAP.mdTracked in ROADMAP.md
Activity