Sókrates Wiki

LLM Fallback Provider Configuration

2 min read

LLM Fallback Provider Configuration

Summary

This page defines the architectural constraints and configuration schema for implementing LLM provider fallbacks within the Sokrates stack. It specifically addresses the transition between the legacy fallback_model structure used in version 0.5.0 and the fallback_providers chain introduced in version 0.6.0.

Details

The Sokrates project utilizes a resilience layer for its Large Language Model (LLM) integrations to ensure high availability, particularly when relying on external providers like OpenRouter. The configuration of these fallbacks is version-dependent and governed by the specific build of the Eidos or Hermes components.

Version Discrepancies

During the development and deployment of the Sokrates stack, a distinction was identified between the configuration capabilities of version 0.5.0 and version 0.6.0. While version 0.6.0 documentation references a fallback_providers chain (allowing for multiple sequential fallbacks), the v0.5.0 builds—which are currently prevalent in certain NixOS flake configurations—do not consume this key. In these earlier builds, the system relies on a singular fallback_model block.

Configuration Schema (v0.5.0)

For systems running on the v0.5.0 baseline, the fallback logic must be defined using the fallback_model key. This block specifies a secondary provider and model to be used if the primary inference request fails.

The standard implementation for the Sokrates project often utilizes OpenRouter as a fallback provider due to its broad model availability. A common configuration pattern identified in the system logs involves using Moonshot AI models as a cost-effective or high-availability alternative.

Example Configuration:

fallback_model:
  provider: openrouter
  model: moonshotai/kimi-k2.5

Transition to v0.6.0

In version 0.6.0 and later, the configuration schema evolves to support a fallback_providers list. Although the key fallback_providers may appear in default configuration files within the Nix store for v0.5.0 builds, it is effectively a “dead” key that is not processed by the application logic. Administrators must ensure that the configuration matches the binary version deployed via the Nix flake to avoid silent failures where fallbacks are defined but never triggered.

Implementation Notes

  • Provider Selection: OpenRouter is the preferred fallback gateway for the Sokrates project to minimize the need for multiple individual provider credentials (e.g., Anthropic, OpenAI) within the intelligence secrets directory.
  • Model Selection: The choice of moonshotai/kimi-k2.5 is noted as a specific target for free or low-cost fallback operations during development phases.
  • Validation: When updating the sokrates-box or sokrates-dev configurations, the fallback_model block should be verified against the actual Nix store source to confirm which schema version is active.

Graph View

Backlinks