Skip to content

Start typing to search the documentation.

Observability

Last updated View as Markdown

Flue emits everything its agents do — model turns, tool calls, structured logs, compactions, and settlements — as typed runtime events your application can observe in process. This surface is separate from the per-conversation message stream a chat UI reads, which belongs to Routing and the SDK. This guide covers the two surfaces and when to use each, subscribing with observe(), what the event stream contains, token usage and provider diagnostics on model turns, tool activity and logs, and exporting telemetry to Sentry, Braintrust, and OpenTelemetry.

Two event surfaces

Flue exposes agent activity on two distinct surfaces:

  • The conversation stream is the product surface: one conversation’s durable, render-ready messages, data parts, and settlements, consumed over HTTP with createFlueClient(...) observe() / history(). Routing covers it.
  • The runtime event stream is the operational surface: live activity across every agent in the process — model requests, tool executions, logs, token counts, failures — consumed in process with observe() from @flue/runtime. That stream is this guide’s subject.

The two APIs share a name but not a shape: the SDK client’s observe() maintains one conversation’s materialized message state, while the runtime’s observe() delivers raw activity events. Telemetry, metering, and error reporting belong on the runtime stream. The surfaces share correlation identifiers — a conversation message’s submissionId matches the runtime events its submission produced.

Subscribing with observe()

observe() from @flue/runtime registers a global subscriber for all agent activity in the current process. Register it once at startup, at module top level in app.ts (or a module app.ts imports):

import { observe } from '@flue/runtime';

observe((event) => {
  if (event.type === 'submission_settled' && event.outcome === 'failed') {
    console.error(`[${event.agentName}] submission ${event.submissionId} failed:`, event.error?.message);
  }
});

The subscriber receives every event from every agent — direct prompts, dispatched work, subagent tasks, and harness activity alike. observe() returns an unsubscribe function, but telemetry subscribers typically register once and never remove themselves; the return value exists for tests and dynamic wiring.

Three rules for subscribers:

  • Stay cheap. Subscribers run synchronously on the event emission path. Branch on event.type, return immediately for activity you don’t consume, and queue substantial async work instead of blocking emission.
  • Treat events as read-only. Each delivery is a detached, frozen observation; a subscriber can never alter what other subscribers or the runtime see.
  • Failures are contained. A throwing subscriber is logged and skipped — it never halts the agent or other subscribers. Returned promises are observed for rejection but not awaited.

The subscription is isolate-scoped and live-only: it sees activity emitted in the current process from the moment it registers, with no durable replay and no cross-process aggregation. On Node.js one process hosts all agents, so one registration sees everything. On Cloudflare, each agent conversation runs in its own Durable Object isolate — a subscriber registered from app.ts runs in each isolate and sees that isolate’s activity only. One placement caveat, shared with registerProvider(): flue run loads only the agent module, never app.ts — register in the agent module when a subscriber must also run under the CLI.

What the stream contains

Every event carries the event-format version (v: 3), a per-context eventIndex, a timestamp, and the correlation fields that apply to it, including agentName, conversationId, instanceId, submissionId, operationId, turnId, taskId, and dispatchId. The event families:

Events Activity
agent_start, agent_end, idle Agent loop lifecycle.
submission_settled A durable submission reached completed, failed, or aborted — the reliable terminal signal.
operation_start, operation Prompt, skill, task, shell, and compact operation boundaries, with duration and rolled-up usage.
turn_start, turn_request, turn, turn_messages Model turns (see below).
message_*, text_delta, thinking_* Live message and reasoning progress.
tool_start, tool Tool execution, correlated by toolCallId.
task_start, task Subagent task delegation, with result, error state, and duration.
compaction_start, compaction Context compaction, with message counts and usage.
log Structured logs written by your tools and hooks (see below).

Streaming deltas are live progress signals, not authoritative message state; the assistant message_end event carries the completed message. Nested errors do not necessarily fail the work that contains them — an agent can recover from a failed turn or tool call — so alert on submission_settled outcomes and read nested isError events as diagnostic context.

Two properties of the live stream go beyond what is durably recorded:

  • Live observations carry extra detail. observe() delivers each event as a FlueObservation — the event plus live-only fields such as normalized tool arguments, effective results, and classified errorInfo including the throw-site stack. These exporter-oriented fields are never persisted or replayed.
  • turn_request is in-process only. It contains the full model-visible request — provider identity, settings, system prompt, messages, and tools — and is delivered to observe() subscribers but never persisted or served over HTTP.

The Events Reference documents every event’s fields and which payloads are stable contract.

Token usage

Each completed model call emits a turn event whose request summarizes what was sent (provider, requested model, API, settings) and whose response carries the outcome — output, finish reason, and usage:

usage field Meaning
input, output Tokens sent to and generated by the model.
cacheRead, cacheWrite Prompt-cache tokens read and written.
totalTokens Total across all components.
cost Estimated cost from the model catalog’s rates, per component plus total.

Per-agent token metering is a single observer:

import { observe } from '@flue/runtime';
import { metrics } from './shared/metrics.ts';

observe((event) => {
  if (event.type !== 'turn' || !event.response.usage) return;
  const { usage } = event.response;
  metrics.increment('llm.tokens', usage.totalTokens, {
    agent: event.agentName,
    model: event.request.requestedModel,
    purpose: event.purpose, // 'agent' | 'compaction' | 'compaction_prefix'
  });
  metrics.increment('llm.cost', usage.cost.total, { agent: event.agentName });
});

Usage also rolls up at coarser boundaries: operation and compaction events carry aggregate usage for the work they bound. When summing usage across events, sum one level only — turn values are the leaves, and the roll-ups already include them. Duration values at different levels overlap the same way and should not be added together. Inside the agent, useResponseFinish() receives the whole response’s aggregate usage — the right place to stamp token counts onto response metadata for your client; see Event hooks.

Provider diagnostics

A turn event’s response is normalized — finishReason and error use Flue’s vocabulary regardless of provider. Alongside them, the response carries allowlisted raw provider metadata when the provider attaches it:

  • providerFinishReason — the provider’s exact finish value before normalization (for example, Workers AI’s tool_calls behind the normalized toolUse).
  • gatewayLogId — the response’s own Cloudflare AI Gateway log id (cf-aig-log-id), for correlating a specific turn with its entry in the gateway dashboard.

Both are telemetry only — they never affect execution or replay — and are present only when the provider records them. The Workers AI provider attaches both today. A diagnostic observer for failed turns reads them directly from the event:

import { observe } from '@flue/runtime';

observe((event) => {
  if (event.type !== 'turn' || !event.isError) return;
  console.error('model turn failed', {
    provider: event.request.providerName,
    model: event.request.requestedModel,
    finishReason: event.response.finishReason,
    providerFinishReason: event.response.providerFinishReason,
    gatewayLogId: event.response.gatewayLogId,
    error: event.response.error?.message,
  });
});

request.providerId is the registration key from the model specifier; request.providerName is the semantic provider identity, which differs when a gateway or custom registration fronts the model.

Tool activity and logs

Tool execution emits tool_start and tool events carrying the tool name, toolCallId, duration, error state, and result — for both model-driven calls and programmatic shell activity. The live observation adds the normalized arguments and effective result.

For progress inside a long-running tool, the tool’s run context provides a logger (see Tools); lifecycle hook contexts like useAgentStart carry the same log interface. Each call emits a log event with a level, a message, and your attributes — tool logs additionally stamped with tool and toolCallId, hook logs with the hook that wrote them. The model never sees log lines; they exist for your application:

async run({ data, log }) {
  log.info('sync started', { records: data.ids.length });
  const failed = await crm.sync(data.ids);
  if (failed.length > 0) log.error('sync incomplete', { failed: failed.length });
  return { synced: data.ids.length - failed.length };
}
import { observe } from '@flue/runtime';
import { logger } from './shared/logger.ts';

observe((event) => {
  if (event.type !== 'log') return;
  logger.log(event.level, event.message, {
    ...event.attributes,
    conversation: event.conversationId,
  });
});

Log lines are runtime events, not conversation content: they never appear in the messages a client renders, and they reach only in-process subscribers — forward them to your logging backend from an observer, or through one of the integrations below.

Choose an observability provider

For production telemetry, Flue ships integrations with three ecosystems rather than a bundled dashboard:

  • Sentry — error reporting: failed operations, failed settlements, and explicit error logs. Add with flue add tooling sentry.
  • Braintrust — LLM tracing: operations as traces with model, tool, task, and compaction spans plus usage. Add with flue add tooling braintrust.
  • OpenTelemetry — standards-based GenAI spans, metrics, and logs for any OTel-compatible backend. Add @flue/opentelemetry to your OTel SDK setup.

The Sentry and Braintrust blueprints generate a source-root module that app.ts imports — an observe() bridge exactly like the ones above, plus provider initialization. The OpenTelemetry adapter registers through instrument(...) instead, which pairs an observer with an execution interceptor so spans wrap live agent, model, tool, and task execution:

import { createOpenTelemetryInstrumentation } from '@flue/opentelemetry';
import { instrument } from '@flue/runtime';

instrument(createOpenTelemetryInstrumentation());

Choose Sentry when you want failures in an existing error tracker, Braintrust when you want content-bearing LLM traces for inspection and evaluation, and OpenTelemetry when your organization standardizes on an OTel backend. They compose — an error reporter and a tracer can subscribe side by side. On Cloudflare, each integration exports per isolate and final flushes are best-effort; each tooling page documents its target-specific behavior.

Protect sensitive content

Runtime events can contain prompts, system instructions, reasoning, tool arguments and results, and error details. Two protections are built in — turn_request never leaves the process, and image content blocks never carry raw bytes (their data is replaced with the IMAGE_DATA_OMITTED sentinel) — but everything else is your exporter’s responsibility. Restrict subscriptions to the event types you need, apply a sanitization policy before forwarding events to an external service, and review the retention and access controls of the receiving backend. The shipped integrations default conservatively: the Sentry bridge reports no prompts or tool payloads, the OpenTelemetry adapter disables content capture unless you enable it with a transform, and Braintrust is content-bearing by design with a masking hook. Make an explicit data-handling decision before widening any of them.

Next steps

  • Events Reference — the full event vocabulary, envelope fields, and the observe() contract.
  • Routing and the SDK — the conversation stream your UI consumes.
  • Agent Hooks — read usage and stamp response metadata from inside the agent.
  • Sentry, Braintrust, and OpenTelemetry — per-integration setup and content policies.
  • Evals — turn observed behavior into scored regression checks.