Merge pull request #788 from nesquena/fix/ci-test-default-model-isolation

fix(tests): restore conftest default model in teardown — fixes CI ordering failure

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Hermes Web UI -- Changelog
 
+## [v0.50.123] — 2026-04-21
+
+### Fixed
+- **Default model change surfaced stale value after model-list TTL cache landed** — `set_hermes_default_model()` now explicitly invalidates `_available_models_cache` after `reload_config()`. The 60s TTL cache introduced in v0.50.121 (#780) only invalidates on config-file mtime change, but `reload_config()` resyncs `_cfg_mtime` before `get_available_models()` runs — so the mtime check never fires and the POST response (plus downstream reads within the TTL window) returned the previous model until the cache expired. Root cause of the `test_default_model_updates_hermes_config` CI flake as well. (#788)
+- **Test teardown restores conftest default deterministically** — `test_default_model_updates_hermes_config` now restores to the conftest-injected `TEST_DEFAULT_MODEL` (via `tests/_pytest_port.py`) instead of reading the pre-test value from `/api/models`, so teardown is stable regardless of ordering. Also updates `TESTING.md` automated-test count to 1578. (#788)
+
 ## [v0.50.122] — 2026-04-21
 
 ### Fixed

TESTING.md

@@ -8,7 +8,7 @@
 > Prerequisites: SSH tunnel is active on port 8787. Open http://localhost:8787 in browser.
 > Server health check: curl http://127.0.0.1:8787/health should return {"status":"ok"}.
 >
-> Automated coverage: 1353 tests collected via `pytest tests/ --collect-only -q`. Includes onboarding coverage for bootstrap/static wizard presence, real provider config persistence (`config.yaml` + `.env`), the `/api/onboarding/*` backend, the onboarding skip/existing-config guard, and CSS regression coverage for smooth thinking/tool card disclosure animation.
+> Automated coverage: 1578 tests collected via `pytest tests/ --collect-only -q`. Includes onboarding coverage for bootstrap/static wizard presence, real provider config persistence (`config.yaml` + `.env`), the `/api/onboarding/*` backend, the onboarding skip/existing-config guard, and CSS regression coverage for smooth thinking/tool card disclosure animation.
 > Run: `pytest tests/ -v --timeout=60`
 >
 > Local regression focus: verify that a previously closed workspace panel stays visually closed from first paint through boot completion on desktop refresh; there should be no brief open-then-close flash.

api/config.py

@@ -800,6 +800,10 @@ def set_hermes_default_model(model_id: str) -> dict:
         _save_yaml_config_file(config_path, config_data)
     # Reload outside the lock — reload_config() acquires _cfg_lock itself.
     reload_config()
+    # reload_config() resyncs _cfg_mtime to the new file mtime, so the mtime
+    # check inside get_available_models() won't trigger invalidation. Drop
+    # the TTL cache explicitly so the next call recomputes with the new model.
+    invalidate_models_cache()
     return get_available_models()
 
 

tests/_pytest_port.py

@@ -40,3 +40,7 @@ TEST_STATE_DIR = pathlib.Path(os.environ.get(
     'HERMES_WEBUI_TEST_STATE_DIR',
     str(_HERMES_HOME / _auto_state_dir_name(_REPO_ROOT))
 ))
+
+# Default model injected by conftest — tests that mutate the default model
+# must restore to this value so later tests see a consistent baseline.
+TEST_DEFAULT_MODEL = os.environ.get('HERMES_WEBUI_DEFAULT_MODEL', 'openai/gpt-5.4-mini')

tests/test_sprint12.py

@@ -3,7 +3,7 @@ Sprint 12 Tests: settings panel, session pinning, session import, SSE reconnect.
 """
 import json, pathlib, urllib.error, urllib.request, urllib.parse
 
-from tests._pytest_port import BASE
+from tests._pytest_port import BASE, TEST_DEFAULT_MODEL
 
 
 def get(path):
@@ -40,8 +40,6 @@ def test_settings_get_returns_defaults():
 
 def test_default_model_updates_hermes_config():
     """POST /api/default-model updates the effective Hermes default model."""
-    original, _ = get("/api/models")
-    original_model = original.get("default_model") or ""
     try:
         d, status = post("/api/default-model", {"model": "anthropic/claude-sonnet-4.6"})
         assert status == 200
@@ -50,9 +48,9 @@ def test_default_model_updates_hermes_config():
         # Both should resolve to the same model (may differ in prefix normalization)
         assert 'claude-sonnet-4.6' in d2['default_model']
     finally:
-        # Always restore — regardless of test ordering or failures
-        if original_model:
-            post("/api/default-model", {"model": original_model})
+        # Always restore to the conftest-injected default so later tests see
+        # a consistent baseline regardless of test ordering.
+        post("/api/default-model", {"model": TEST_DEFAULT_MODEL})
 
 
 def test_settings_does_not_persist_default_model():