feat: add primary model validation harness

This commit is contained in:
Hermes Agent
2026-08-26 23:01:06 +10:00
parent 6348b23881
commit babd72f3a1
7 changed files with 345 additions and 79 deletions
+4 -2
View File
@@ -17,8 +17,10 @@ Native-like Hermes dashboard plugin for local Ollama model management and chat.
- Paste images directly into the composer and drag/drop images, PDFs, and text files
- Streamed Ollama responses with a real Stop action that cancels the active request
- Minimized-by-default expandable thinking/progress details with live stage, elapsed time, event, and character counters
- Validation harness mode: choose one primary model and two or more independent validator models; validators review the primary draft and the primary model compiles one final answer
The chat supports two modes. With one selected model, it sends a normal direct request. With one primary model and at least two validator models selected, the plugin runs a validation harness: the primary creates a draft, validators independently review the request and draft in parallel, and the primary compiles one final user-facing answer from the draft and validation reports. Validator reports are returned as supporting evidence, while only the compiled primary response is persisted and displayed as the answer.
The chat transcript and selected model are persisted in the Hermes server's SQLite database at `~/.hermes/ollama-manager/chat.sqlite3`, so conversations can be listed and resumed from another browser or after a dashboard restart. Use **New conversation** to start a separate thread and **Clear chat** to delete the selected shared conversation. Uploaded files remain temporary; attachment names/types/URLs are retained as metadata, not raw file contents.
## Performance metrics
@@ -93,7 +95,7 @@ When Ollama accepts a load request but evicts one model while starting another,
While Ollama is starting a runner, the highlighted runtime chart displays an animated **Loading into Ollama memory** state with the selected model names, current stage, and elapsed time. The runtime panel separately reports Ollama resident model-weight bytes and the estimated target weight bytes. This is separate from host `MemAvailable`: CPU-mapped model files may appear as Linux file cache rather than ordinary process RAM usage.
## Multi-model loading and resident state
The model pool now verifies every load request against Ollama `/api/ps` before reporting success. The UI shows an in-progress loading message, then reports which models are actually resident and which Ollama evicted. Resident models are highlighted in the pool with a green loaded state. After a browser refresh, resident models repopulate the pool selection and the **Models for this answer** selector, allowing multiple loaded models to be selected for parallel chat.
The model pool now verifies every load request against Ollama `/api/ps` before reporting success. The UI shows an in-progress loading message, then reports which models are actually resident and which Ollama evicted. Resident models are highlighted in the pool with a green loaded state. After a browser refresh, resident models repopulate the pool selection and the answer-harness controls. Select one primary loaded model and two or more validator models; validators review the primary draft and the primary compiles one final answer.
Ollama still controls the physical resident-model limit. If it cannot keep all requested models at once because of its scheduler, GPU policy, context allocation, or available memory, the plugin reports the non-resident names instead of claiming they were permanently loaded. Increasing that limit requires changing the Ollama service configuration; the plugin does not silently alter or restart the Ollama service.