Skip to content
Flue

Start typing to search the documentation.

Provider API

Last updated View as Markdown

The provider registration surface configures how the Flue runtime reaches model providers: which provider IDs exist, which wire protocol and endpoint each one uses, and what metadata its models carry. For a walkthrough, see Models — Custom providers; this page is the complete contract.

Exports:

import { registerProvider, registerApiProvider, ProviderRegistrationError } from '@flue/runtime';
import type { ProviderRegistration, HttpProviderRegistration } from '@flue/runtime';
import type {
  CloudflareAIBinding,
  CloudflareAIBindingRegistration,
  CloudflareGatewayOptions,
} from '@flue/runtime/cloudflare';

registerProvider()

function registerProvider(providerId: string, registration: ProviderRegistration): void;

Registers a model provider keyed by the provider ID used in model specifiers: after registerProvider('acme', ...), the specifier 'acme/some-model' resolves through this registration. Registered provider IDs take precedence over Flue’s built-in model catalog (from Pi) for that ID; all other provider IDs continue to resolve from the catalog unchanged. The registration is one of the two ProviderRegistration shapes.

// Catalog provider ID: options overlay the catalog metadata.
registerProvider('anthropic', { baseUrl: 'https://gateway.example.com/anthropic' });

// Unknown provider ID: registered from scratch; api and baseUrl required.
registerProvider('ollama', { api: 'openai-completions', baseUrl: 'http://localhost:11434/v1' });

Behavior:

  • Catalog provider IDs overlay; unknown provider IDs register from scratch. When providerId is a catalog provider (anthropic, openai, google, …), every option is optional: models keep their catalog metadata and the registration’s options are layered on top (see Model resolution). When the catalog has no models for providerId, the registration must supply api and baseUrl; omitting either throws ProviderRegistrationError synchronously. A 'cloudflare-ai-binding' registration always passes this validation, under any provider ID.
  • Each call replaces the provider ID’s previous registration. Calls do not accumulate or merge. The effective settings are always the catalog defaults (when the ID is a catalog provider) plus the latest call’s options — an option set by an earlier call and omitted from the latest one is gone.
  • The registry is module-scoped and in-memory. Call registerProvider() at module top level, before any agent runs. On the Node.js target one process hosts all agents, so a registration in app.ts covers everything. On the Cloudflare target each agent conversation runs in its own Durable Object isolate; app.ts is evaluated in every isolate, so top-level registrations apply everywhere. flue run loads only the agent module, never app.ts — put the registration in the agent module when it must also apply there.
  • Registration is declarative and deferred. The call performs no network I/O and no credential validation; a wrong baseUrl or apiKey surfaces as a provider error on the first model request. There is no public unregister function.

Model resolution

A model specifier is 'provider-id/model-id', split at the first /. Resolution happens per model call, against the live registry. When the provider ID is registered, the resolved model is built as follows.

For an HttpProviderRegistration, each field resolves independently with fixed precedence:

  • api and baseUrl — the registration’s value, else the model’s own catalog entry, else (for a model ID a catalog provider doesn’t list) the value from the provider’s first catalog entry. Both are guaranteed present: non-catalog registrations were required to supply them.
  • contextWindow, maxTokens, reasoning, input — the registration’s models[modelId] entry, else the registration-level option, else the model’s catalog entry, else the hydration default below.
  • headers — merged per key: the catalog entry’s headers first, the registration’s on top (the registration wins on conflict). Model IDs without a catalog entry contribute no catalog headers.
  • cost — the catalog entry’s rates, else all-zero. There is no override option.

A model ID that no catalog entry backs — any model of a from-scratch provider, or an unlisted model ID under a registered catalog provider — hydrates these defaults:

  • reasoning: false — a forwarded thinkingLevel is silently dropped; thinking never reaches the wire.
  • input: ['text'] — attached images are silently replaced with an "(image omitted)" placeholder.
  • contextWindow: 0 — treated as unknown; threshold compaction cannot engage.
  • maxTokens: 0 and all-zero cost rates.

Declare reasoning, input, contextWindow, and maxTokens in the registration (provider-level or per model ID via models) when the defaults are wrong for the models you run.

Resolution failures throw plain Errors (not FlueError categories), raised when the model call resolves the specifier:

  • A specifier with no / is invalid.
  • A specifier whose provider ID is registered but whose model ID is empty ('acme/') is invalid.
  • A specifier whose provider ID is neither registered nor a catalog provider, or a catalog specifier whose model ID the catalog doesn’t list (with no registration in play), is unknown.

ProviderRegistration

type ProviderRegistration = HttpProviderRegistration | CloudflareAIBindingRegistration;

The union of the two registration shapes registerProvider() accepts, discriminated by api:

  • HttpProviderRegistration — an HTTP endpoint speaking a Pi wire protocol. Any api value other than 'cloudflare-ai-binding', including omitted.
  • CloudflareAIBindingRegistration — a Cloudflare Workers AI binding (env.AI), dispatched in-process instead of over HTTP. api: 'cloudflare-ai-binding'.

ProviderRegistration and HttpProviderRegistration are exported from @flue/runtime; CloudflareAIBindingRegistration is exported from @flue/runtime/cloudflare only.

HttpProviderRegistration

interface HttpProviderRegistration {
  api?: Api; // Pi wire-protocol slug, e.g. 'anthropic-messages'
  baseUrl?: string;
  apiKey?: string;
  headers?: Record<string, string>;
  contextWindow?: number;
  maxTokens?: number;
  reasoning?: boolean;
  input?: ('text' | 'image')[];
  models?: Record<
    string,
    {
      contextWindow?: number;
      maxTokens?: number;
      reasoning?: boolean;
      input?: ('text' | 'image')[];
    }
  >;
  storeResponses?: boolean;
  telemetry?: {
    providerName?: string;
    serverAddress?: string;
    serverPort?: number;
  };
}

Registers an HTTP-backed provider ID. All fields are optional for catalog provider IDs; api and baseUrl are required otherwise.

  • api — the wire protocol used for requests. Pi ships handlers for 'anthropic-messages', 'openai-completions', 'openai-responses', 'azure-openai-responses', 'openai-codex-responses', 'google-generative-ai', 'google-vertex', 'bedrock-converse-stream', and 'mistral-conversations'; any other string is accepted and must have a handler registered via registerApiProvider(). Default: the catalog protocol.
  • baseUrl — the endpoint root, e.g. 'https://api.anthropic.com/v1'. The protocol handler appends its own route paths. Default: the catalog endpoint.
  • apiKey — the credential for this provider ID, looked up live on every model call, so it reflects the latest registration. Precedence: a non-empty registered apiKey wins over Pi’s environment-variable lookup for the provider ID (ANTHROPIC_API_KEY, OPENAI_API_KEY, …); unset, empty, or whitespace-only values fall through to that lookup. See Provider credentials.
  • headers — headers sent on every outgoing request, merged per key over the catalog entry’s headers; the registration’s values win on conflict. Default: none.
  • contextWindow — default context-window size in tokens for every model resolved through this registration. Default: the catalog value, then 0 (unknown — threshold compaction cannot engage).
  • maxTokens — default output-token limit for every model resolved through this registration. Default: the catalog value, then 0.
  • reasoning — declares models reasoning-capable. When false, a forwarded thinkingLevel is silently dropped. Default: the catalog value, then false.
  • input — the input modalities models accept. Without 'image', image blocks are silently replaced with an "(image omitted)" placeholder. Default: the catalog value, then ['text'].
  • models — per-model overrides for contextWindow, maxTokens, reasoning, and input, keyed by model ID. Each field beats its registration-level counterpart for that model ID only. Default: none.
  • storeResponses — when true, sends store: true on every request, opting into OpenAI-hosted item persistence and its retention policy. Applied only when the resolved api is 'openai-responses' or 'azure-openai-responses'; ignored for every other protocol. Default: false.
  • telemetry — observability identity overrides; see Provider telemetry. Default: derived.

CloudflareAIBindingRegistration

interface CloudflareAIBindingRegistration {
  api: 'cloudflare-ai-binding';
  binding: CloudflareAIBinding; // env.AI
  gateway?: CloudflareGatewayOptions | false;
  telemetry?: {
    providerName?: string;
    serverAddress?: string;
    serverPort?: number;
  };
}

Registers a provider ID backed by a Workers AI binding: requests dispatch through binding.run(modelId, payload, options) in-process, with no baseUrl, no apiKey, and no HTTP endpoint. The type is exported from @flue/runtime/cloudflare.

  • api — the literal discriminator 'cloudflare-ai-binding'.
  • binding — the captured env.AI reference, read once at registration time.
  • gateway — AI Gateway routing for every run call through this registration. Tri-state: omitted routes through Cloudflare’s default AI Gateway (the options object { id: 'default' }, which the binding provisions on demand for the account); a CloudflareGatewayOptions object replaces the default; false opts out — no gateway option is passed to run.
  • telemetry — observability identity overrides; see Provider telemetry. Default: derived.

Behavior:

  • Wire handler availability. The 'cloudflare-ai-binding' protocol handler is registered only by the generated Cloudflare Worker entry. On other targets the registration shape is accepted but no handler exists, so model calls fail.
  • Default registration on the Cloudflare target. The generated Worker entry registers the cloudflare provider ID with { api: 'cloudflare-ai-binding', binding: env.AI } — unless app.ts already registered cloudflare itself. app.ts imports are hoisted above the generated entry’s body, so a user registration always wins; this is how a project targets a named gateway, tunes caching, or opts out. See Models — Cloudflare Workers AI and the Cloudflare target guide.
  • Metadata hydration. Model IDs resolve against Pi’s cloudflare-workers-ai catalog regardless of the provider ID registered. Listed IDs keep their catalog metadata (context window, cost, reasoning, input); unlisted IDs get the same zero-metadata defaults as Model resolution describes. There are no metadata override options on this shape — no contextWindow, maxTokens, reasoning, input, or models map.
  • Wire format. Model IDs prefixed anthropic/ are sent as Anthropic Messages requests; all other IDs use the OpenAI-compatible chat-completions shape. Both go through binding.run with returnRawResponse: true and the resolved gateway option.
  • Failures. Non-OK binding responses throw CloudflareAIBindingError (type: 'cloudflare_ai_binding_error'), exported from @flue/runtime/cloudflare; a 413 additionally carries meta.reason: 'request_too_large' and triggers compaction recovery. See Errors — CloudflareAIBindingError.

CloudflareGatewayOptions

interface CloudflareGatewayOptions {
  id: string;
  skipCache?: boolean;
  cacheTtl?: number;
  cacheKey?: string;
  metadata?: Record<string, number | string | boolean | null | bigint>;
  collectLog?: boolean;
  eventId?: string;
  requestTimeoutMs?: number;
}

AI Gateway options forwarded verbatim as the gateway option on every binding.run(...) call routed through a CloudflareAIBindingRegistration. The shape mirrors Cloudflare’s Worker binding methods documentation, which defines each field’s provider-side semantics. Exported from @flue/runtime/cloudflare.

  • id — the AI Gateway ID (slug) to route through. Required whenever gateway options are specified.
  • skipCache — bypass the gateway cache for the request.
  • cacheTtl — cache TTL override, in seconds.
  • cacheKey — cache key override.
  • metadata — arbitrary metadata surfaced on the gateway log entry.
  • collectLog — force collecting (or not collecting) request logs.
  • eventId — custom event ID for log correlation.
  • requestTimeoutMs — per-request timeout enforced by the gateway, in milliseconds.

CloudflareAIBinding

interface CloudflareAIBinding {
  run(
    modelId: string,
    inputs: Record<string, unknown>,
    options?: Record<string, unknown>,
  ): Promise<Response | Record<string, unknown>>;
}

The minimal structural shape of a Workers AI binding, exported from @flue/runtime/cloudflare. It is deliberately structural — not Cloudflare’s Ai type — so @flue/runtime stays importable on Node.js. Pass the real env.AI binding as a CloudflareAIBindingRegistration’s binding.

Provider telemetry

Both registration shapes accept a telemetry object that overrides how the provider is identified in model-turn observability events (turn_request.request and turn.request). All fields are optional; the type of the object is not separately exported.

  • providerName — the semantic provider name (ModelRequestInfo.providerName). Default: a fixed normalization of the provider ID to observability conventions (table below); IDs outside the table pass through unchanged.
  • serverAddress — the reported server host. Default: parsed from the resolved model’s baseUrl.
  • serverPort — the reported server port. Default: parsed from the resolved model’s baseUrl.

Default providerName normalization:

Provider ID providerName
amazon-bedrock aws.bedrock
azure-openai-responses azure.ai.openai
google gcp.gemini
google-vertex gcp.vertex_ai
mistral mistral_ai
moonshotai, moonshotai-cn moonshot_ai
xai x_ai

telemetry affects reported identity only; it changes nothing about how requests are sent. The same events always report the registration key unmodified as ModelRequestInfo.providerId — telemetry cannot change it.

registerApiProvider()

function registerApiProvider(provider: ApiProvider): void;

// From '@earendil-works/pi-ai/compat':
interface ApiProvider<TApi extends Api = Api, TOptions extends StreamOptions = StreamOptions> {
  api: TApi;
  stream: (model: Model<TApi>, context: Context, options?: TOptions) => AssistantMessageEventStream;
  streamSimple: (
    model: Model<TApi>,
    context: Context,
    options?: SimpleStreamOptions,
  ) => AssistantMessageEventStream;
}

Registers a Pi wire-protocol handler for an api slug Pi doesn’t ship, making that slug usable as the api of an HttpProviderRegistration. The argument is a Pi ApiProvider; its parameter types (Model, Context, StreamOptions, AssistantMessageEventStream) come from @earendil-works/pi-ai/compat and are not re-exported by Flue.

import { registerApiProvider, registerProvider } from '@flue/runtime';

registerApiProvider({ api: 'my-novel-api', stream, streamSimple });
registerProvider('thing', { api: 'my-novel-api', baseUrl: 'https://thing.example/v1' });
  • The handler registry is module-scoped and last-write-wins per api slug: repeated calls with the same slug overwrite, so code can register on every process or isolate boot without dedupe bookkeeping.
  • Registering a built-in slug (for example 'openai-completions') replaces Pi’s shipped handler for the process.
  • registerApiProvider() supplies the transport implementation only. Associating a provider ID, endpoint, credentials, and model metadata with the slug is registerProvider()’s job.

ProviderRegistrationError

class ProviderRegistrationError extends FlueError {
  readonly type: 'invalid_provider_registration';
  readonly meta: { providerId: string };
}

Thrown synchronously by registerProvider() when the provider ID is not a catalog provider and the registration omits api or baseUrl. Exported from @flue/runtime. See Errors — FlueError for the FlueError contract.

What a registration does not change

  • The catalog itself. A registration shadows one provider ID at resolution time; it never adds, removes, or edits catalog entries. Unregistered provider IDs, and every model listed under them, resolve exactly as before.
  • Cost metadata. There is no cost option. Catalog-listed models keep their catalog rates under any overlay; models without a catalog entry report zero cost.
  • Model ID validity. Any model ID under a registered provider resolves without validation; a typo surfaces as a provider error on the first request, not at registration or resolution time.
  • Anything durable. Registrations live in process memory for the current module scope. They are not persisted, not shared across processes or isolates, and rebuilt from module top-level code on every boot.
  • Earlier registrations of other provider IDs. Each call affects exactly one provider ID.