feat: add per-model GPU and RAM placement

This commit is contained in:
Hermes Agent
2026-08-25 23:17:30 +10:00
parent 238de79607
commit 1029d23605
6 changed files with 60 additions and 17 deletions
+9 -1
View File
@@ -47,7 +47,15 @@ The Available downloads controls support:
Popularity and date ordering use upstream metadata only; the plugin does not invent popularity, dates, RAM requirements, or token metrics.
## Ollama capacity warnings
## Per-model placement
Each installed model in the Model pool has a persistent placement selector:
- **GPU + RAM (automatic offload)**: Ollama uses GPU layers where it can and keeps the remainder in system RAM. This is the default.
- **RAM only (CPU)**: the plugin sends Ollama `num_gpu: 0`, preventing GPU layer offload for that model.
Placement is sent both when loading models and when chatting, so a RAM-only model is not silently reloaded with GPU offload. RAM-only models will not increase GPU VRAM usage; GPU+RAM models can still be evicted by Ollama if the GPU/device-memory scheduler cannot fit the runner.
When Ollama accepts a load request but evicts one model while starting another, the dashboard displays an amber capacity warning rather than a red plugin error. On the verified host, Ollama logged that a 27.9 GiB runner would exceed available device memory with approximately 2.5 GiB GPU memory free. The T600 has 4 GiB VRAM, so two large multimodal models cannot be guaranteed resident simultaneously by the plugin. The actual resident set remains authoritative through `/api/ps`.