Skip to content
Flue

Start typing to search the documentation.

Models

Last updated View as Markdown

Every agent is powered by exactly one LLM at a time, declared with useModel() — the single required hook in Flue. This guide covers choosing a model, tuning how it’s called, when a change takes effect, supplying provider credentials, and connecting providers Flue doesn’t know out of the box.

Declaring a model

Call useModel() in your agent function’s body with a model specifier string:

'use agent';
import { useModel } from '@flue/runtime';

export function TriageAgent() {
  useModel('anthropic/claude-sonnet-4-6');
  return 'Investigate the reported issue and recommend the next action.';
}

useModel() is a declaration, not a client: it returns nothing, and you never construct an SDK object or pass an API key through your agent code. You name the model; the Flue runtime owns the connection, authentication, streaming, and retries behind it.

Two rules:

  • The call is required. An agent render without a useModel() call cannot start.
  • Call it exactly once per render. An agent has one model. The argument may change from render to render (more on that below), but the call itself may not disappear or repeat.

The call can live in the agent body or in a custom hook the body calls. The one place it’s not available is a subagent render — a delegate’s model is set on its useSubagent() definition instead, and it inherits the parent’s model when unset. See Subagents.

Model specifier

A model specifier is a plain string in 'provider-id/model-id' format. Everything up to the first / names the provider; the rest is the provider’s own model ID, which may itself contain slashes:

  • anthropic/claude-sonnet-4-6 — provider anthropic, model claude-sonnet-4-6
  • openai/gpt-5.5 — provider openai, model gpt-5.5
  • openrouter/moonshotai/kimi-k2.6 — provider openrouter, model moonshotai/kimi-k2.6
  • cloudflare/@cf/moonshotai/kimi-k2.6 — provider cloudflare, model @cf/moonshotai/kimi-k2.6

Flue resolves specifiers against the model catalog from Pi, which ships entries for the major providers — anthropic, openai, google, amazon-bedrock, google-vertex, groq, mistral, xai, deepseek, cerebras, together, fireworks, openrouter, and more. A catalog entry carries the model’s wire protocol, endpoint, context-window size, output-token limit, cost rates, reasoning support, and accepted input modalities. That metadata decides when compaction triggers, whether a thinking level reaches the wire, and whether the model can accept images.

An unknown specifier fails fast: the run errors with the unresolved provider and model ID before any request is sent. To teach Flue a specifier the catalog doesn’t know — a local model, a proxy, a brand-new release — register it yourself; see Custom providers.

On the Cloudflare target there is one more built-in provider ID: cloudflare/... model specifiers run on Workers AI with no API key at all.

Model reasoning effort

useModel() accepts an options object as its second argument with two fields: thinkingLevel and compaction.

useModel('anthropic/claude-opus-4-6', {
  thinkingLevel: 'high',
  compaction: { keepRecentTokens: 16000 },
});

thinkingLevel sets the default reasoning effort for the agent’s model calls: 'off', 'minimal', 'low', 'medium', 'high', or 'xhigh'. When you don’t set it, the runtime uses 'medium'.

Higher levels increase reasoning depth at the cost of latency and tokens; 'off' disables extended thinking entirely. The value is a default: individual operations may override it — a subagent definition can pin its own thinkingLevel, and programmatic harness.prompt(...) calls accept one per operation.

Thinking only reaches the wire for models marked reasoning-capable. Catalog models carry that flag already, but if you register a custom provider and skip its reasoning metadata, a forwarded thinkingLevel is silently dropped. See Custom providers.

Compaction

Agent conversations can outlive any context window. As a conversation approaches the model’s limit, Flue automatically compacts it: older history is folded into a summary while recent messages stay verbatim, and the conversation continues. The compaction option tunes that behavior:

useModel('anthropic/claude-opus-4-6', {
  compaction: {
    // Trigger earlier or later: compaction runs when used tokens
    // exceed contextWindow - reserveTokens. Default: model-aware, ≤ 20000.
    reserveTokens: 30000,
    // How much recent history survives verbatim. Default: 8000.
    keepRecentTokens: 16000,
    // Summarize with a cheaper model than the session runs on.
    model: 'anthropic/claude-haiku-4-5',
  },
});

All three fields are optional; each overrides a model-aware default. Setting compaction.model offloads summarization to a cheaper model.

Passing compaction: false disables threshold compaction — the automatic trigger. Overflow recovery and explicit harness.compact() calls still compact when the conversation no longer fits. The full field reference lives at CompactionConfig.

Changing models mid-conversation

The agent function re-renders before every model call, and useModel() runs again each time — so the specifier can be computed, not constant. A common pattern is escalation, where durable state moves an agent from a cheap model to a strong one:

'use agent';
import { useModel, usePersistentState, useTool } from '@flue/runtime';

export function Reviewer() {
  const [escalated, setEscalated] = usePersistentState('escalated', false);
  useModel(escalated ? 'anthropic/claude-opus-4-6' : 'anthropic/claude-haiku-4-5');

  useTool({
    name: 'escalate_review',
    description: 'Escalate when the change is too complex for a quick pass.',
    async run() {
      setEscalated(true);
      return 'Escalated. A stronger model will take over.';
    },
  });

  return 'Review the proposed change and leave actionable feedback.';
}

The model, thinking level, and compaction settings are submission-scoped: the runtime reads them once, when the agent wakes to process an accepted input (a submission — see Durability). A different value computed by a re-render mid-run latches and takes effect on the next submission, not in the middle of the current one. In the example above, the response where escalate_review fires finishes on the cheap model; the conversation’s next message runs on the strong one.

There is no per-message model parameter on dispatch(...) or the HTTP surface: the model choice belongs to the agent function.

Provider credentials

Model providers authenticate with API keys, and the runtime resolves them from the environment — your agent code never handles them.

Local development

Put keys in a .env file at the project root, using the variable name each provider expects:

ANTHROPIC_API_KEY="sk-ant-..."
OPENAI_API_KEY="sk-..."
GEMINI_API_KEY="..."

anthropic reads ANTHROPIC_API_KEY, openai reads OPENAI_API_KEY, google reads GEMINI_API_KEY, groq reads GROQ_API_KEY — the pattern holds across providers (see Pi’s provider documentation for the full list). Both local entry points load the file for you, with shell-exported values always winning over file values:

  • flue run loads the project-root .env; pass --env <path> to select one alternate file.
  • vite dev loads Vite’s standard file set: .env, .env.local, .env.<mode>, .env.<mode>.local.

Don’t commit .env files.

Deployed environments

Deployed servers read only the real environment — no .env loading:

When the environment-variable convention doesn’t fit — a gateway with its own credential, a secret manager that hands you the value in code — pass apiKey to registerProvider(), which takes precedence over the environment lookup for that provider ID:

import { registerProvider } from '@flue/runtime';

registerProvider('anthropic', { apiKey: await secrets.get('anthropic-key') });

Custom providers

registerProvider() teaches the runtime a provider ID. Call it at module top level in app.ts, before any agent runs. Registrations are keyed by provider ID, and each call replaces that ID’s previous registration. One placement caveat: flue run loads only the agent module, never app.ts — when an agent must also work under flue run, put the registration in the agent module instead.

For a provider ID the catalog already knows, registration is an overlay: models keep their catalog metadata (cost, context window, wire protocol), and your options are layered on top. That makes routing a built-in provider through a gateway or proxy one call, without changing the specifiers your agents use:

import { registerProvider } from '@flue/runtime';

registerProvider('anthropic', {
  baseUrl: 'https://gateway.example.com/anthropic',
  apiKey: process.env.GATEWAY_KEY,
});

For a provider ID the catalog doesn’t know, you register it from scratch. api (the wire protocol) and baseUrl are required, and any OpenAI- or Anthropic-compatible endpoint works — here’s a local Ollama server:

import { registerProvider } from '@flue/runtime';

registerProvider('ollama', {
  api: 'openai-completions',
  baseUrl: 'http://localhost:11434/v1',
  contextWindow: 128000,
  reasoning: false,
});
useModel('ollama/llama3.1:8b');

The runtime trusts the declared metadata. A model ID no catalog knows defaults to reasoning: false (a thinkingLevel is silently dropped), input: ['text'] (attached images are replaced with an “(image omitted)” placeholder), and contextWindow: 0 (unknown, so threshold compaction can’t engage). Set reasoning, input, contextWindow, and maxTokens at the provider level, or per model ID via the models map.

For a wire protocol Pi doesn’t ship at all, registerApiProvider() registers a custom handler that registerProvider() can then reference by its api slug. The full option tables for both functions live in the Provider API reference.

Cloudflare Workers AI (Cloudflare only)

On the Cloudflare target, the cloudflare provider ID is registered automatically and runs models on Workers AI — no API key, no external account; authorization and billing follow the Worker, including the Workers AI pricing and daily free allocation:

export function Assistant() {
  useModel('cloudflare/@cf/moonshotai/kimi-k2.6');
  return 'Help the user with their question.';
}

Declare the AI binding the provider uses in your project’s Wrangler configuration:

{
  "$schema": "./node_modules/wrangler/config-schema.json",
  "ai": {
    "binding": "AI",
  },
}

Everything after cloudflare/ is passed as the model ID to env.AI.run(...). That can be a Workers AI model ID such as @cf/moonshotai/kimi-k2.6, or a binding-supported AI Gateway model ID such as openai/gpt-5.5 when the Worker should reach that model through Cloudflare’s gateway path — cloudflare/openai/gpt-5.5 bills through Cloudflare, while plain openai/gpt-5.5 uses Flue’s direct OpenAI provider and its API key.

By default, every cloudflare/... call routes through Cloudflare’s AI Gateway, giving you caching, logging, and budget controls in the dashboard out of the box. To target a named gateway, tune caching and logging, or opt out, register the cloudflare provider yourself in app.ts — your registration wins over the generated default:

import { registerProvider } from '@flue/runtime';
import { env } from 'cloudflare:workers';

registerProvider('cloudflare', {
  api: 'cloudflare-ai-binding',
  binding: env.AI,
  gateway: { id: 'my-gateway', cacheTtl: 300, metadata: { tenant: 'acme' } },
  // ...or `gateway: false` to bypass AI Gateway entirely.
});

Cloudflare’s model surface is also reachable from any target through two ordinary catalog providers: cloudflare-workers-ai/... (URL-backed Workers AI) and cloudflare-ai-gateway/... (URL-backed AI Gateway), both authenticating with CLOUDFLARE_API_KEY like any other hosted provider. See the Cloudflare target guide for the target’s full model story.

Next steps

  • Agent Hooks API — the full useModel() contract, ThinkingLevel, and CompactionConfig.
  • Provider API — every registerProvider() and registerApiProvider() option.
  • Subagents — give a delegate its own model and thinking level.
  • Durability — what a submission is and how interrupted model work recovers.
  • Observability — inspect model calls, token usage, and provider diagnostics.