After a few days, http 500 rpc error: code=unavailable desc = error reading from server: EOF

[nbonfire]

I'm running a localai server hosting a few models via llama.cpp (primarily qwen3.8-27b-q4, gemma4-31b-it, and mistral-nemo-instruct-2407) on a dgx spark (actually a lenovo pgx but same cpu/gpu and specs.) It works fine for a couple days in a row, but eventually degrades to a state where I cant get responses and just get this error in my agents. Not really sure how to troubleshoot or prevent it (beyond a nightly reboot or something, which I guess I could do but feels like kicking the can). I didn't see anything obvious in the error logs, but will capture them the next time it occurs if it would be helpful

[helloe365]

That error is the gRPC connection between LocalAI core and the model backend breaking — "Unavailable / error reading from server: EOF" means the llama.cpp backend process died or wedged mid-connection. Slow degradation over days on a box hosting several models is the classic pattern: a backend that's either leaking memory or silently hung, and everything routed to it fails until the process is recycled.

Before you schedule nightly reboots, there's a built-in answer for exactly this: the watchdog.

• --enable-watchdog-idle --watchdog-idle-timeout 15m — stops backends that have been idle for the timeout, which frees their memory instead of letting them accumulate
• --enable-watchdog-busy --watchdog-busy-timeout 5m — kills backends that have been "busy" for too long, i.e. the wedged ones

(both also settable via WATCHDOG_IDLE / WATCHDOG_BUSY etc., see the CLI reference). One caveat: if you have legitimately long generations, set the busy timeout above your worst-case generation time, or enable only the idle watchdog — otherwise a long batch job looks identical to a stuck one.

If you want to actually find the leak instead of managing it: when it next happens, check ps aux for leftover backend processes versus what LocalAI thinks is loaded (zombie backends after model reloads are a known leak shape), watch backend RSS over a couple of days — a slow creep is your smoking gun — and glance at dmesg for OOM activity, since on the GB10's unified memory the kernel reclaiming pages can take a backend down without anything obvious in the app logs. Running with debug logging (DEBUG=1) should capture the backend's last words when it dies, which makes for a much better issue report than "it stopped working after two days".

With the idle watchdog enabled you may find the problem simply stops occurring, since nothing sits around long enough to wedge. If it turns out to be a real leak in a specific model's backend, the RSS/dmesg evidence is what the maintainers will need.

If this gets it stable, marking the answer helps the next person whose server quietly rots after a few days.