Distributed Tracing
Understand what ADK-TS traces automatically, how to read the trace hierarchy, and how to add custom spans
Every agent run, LLM request, and tool call in ADK-TS automatically creates an OpenTelemetry span. Spans are nested to reflect the real execution order, so a single trace shows the complete path from user input to final response — including which tools ran, in what order, and how long each LLM call took.
Automatic Trace Hierarchy
Each agent entry point calls telemetryService.traceAsyncGenerator() to wrap its execution in a span. LLM calls and tool calls do the same, creating a parent-child hierarchy:
Auto-instrumented HTTP spans only appear when enableAutoInstrumentation: true is set during initialization.
Agent Spans
Each agent execution creates a span named agent_run [agentName] #N, where N increments with each nested agent call in the same invocation. The span carries these attributes:
| Attribute | Value |
|---|---|
gen_ai.provider.name | iqai-adk |
gen_ai.operation.name | invoke_agent |
gen_ai.agent.name | Agent name |
gen_ai.agent.id | agentName-sessionId |
gen_ai.conversation.id | Session ID |
adk.session.id | Session ID |
adk.user.id | User ID (if set) |
adk.invocation.id | Invocation ID |
adk.environment | Environment name |
When captureMessageContent is enabled, the span also receives gen_ai.input.messages and gen_ai.output.messages events with the full conversation content.
LLM Spans
Each LLM request creates a span named llm_generate [modelName] #N for blocking calls or llm_stream [modelName] #N for streaming. These spans follow the OpenTelemetry GenAI semantic conventions:
| Attribute | Value |
|---|---|
gen_ai.provider.name | Provider name (e.g. openai, anthropic, google) |
gen_ai.operation.name | chat |
gen_ai.request.model | Model name |
gen_ai.request.max_tokens | Max output tokens |
gen_ai.request.temperature | Temperature |
gen_ai.response.id | Response ID from the provider |
gen_ai.response.finish_reasons | Array of finish reasons |
gen_ai.output.type | text or json |
gen_ai.usage.input_tokens | Prompt token count |
gen_ai.usage.output_tokens | Completion token count |
adk.llm.model | Model name |
adk.session.id | Session ID |
When captureMessageContent is enabled, the span also receives gen_ai.input.messages and gen_ai.output.messages attributes containing the full prompt and response, plus gen_ai.system_instructions if a system prompt was set.
Tool Spans
Each tool call creates a span named execute_tool [toolName] #N:
| Attribute | Value |
|---|---|
gen_ai.provider.name | iqai-adk |
gen_ai.operation.name | execute_tool |
gen_ai.tool.name | Tool name |
gen_ai.tool.description | Tool description |
gen_ai.tool.type | Tool class name |
gen_ai.tool.call.id | Tool call ID |
adk.tool.name | Tool name |
adk.session.id | Session ID |
When captureMessageContent is enabled, the span additionally carries gen_ai.tool.call.arguments and gen_ai.tool.call.result with the full serialized arguments and response.
Custom Spans
Wrap any async operation in a span using telemetryService.withSpan(). The callback receives the live Span object so you can add attributes mid-execution:
import { telemetryService } from "@iqai/adk";
const report = await telemetryService.withSpan(
"generate_report",
async span => {
span.setAttribute("report.type", "weekly");
span.setAttribute("report.userId", "user-123");
const data = await fetchReportData("user-123");
span.setAttribute("report.recordCount", data.length);
return compileReport(data);
},
{ "report.version": "2" },
);The initial attributes object (third argument) is set before the callback runs. The span is automatically closed and marked OK or ERROR depending on whether the callback throws.
Tracing Async Generators
Streaming operations that use async generators can be traced with traceAsyncGenerator(). The span stays open for the duration of the generator and closes when it finishes or throws:
import { telemetryService } from "@iqai/adk";
async function* generateItems() {
yield "item-1";
yield "item-2";
yield "item-3";
}
const traced = telemetryService.traceAsyncGenerator(
"generate_items",
generateItems(),
{ "generator.source": "database" },
);
for await (const item of traced) {
console.log(item);
}Adding Events and Attributes to the Active Span
Events are timestamped annotations attached to the currently active span. Use them to mark specific moments during a longer operation:
import { telemetryService } from "@iqai/adk";
telemetryService.addEvent("cache_miss", {
"cache.key": "user:123:preferences",
"cache.ttl": 300,
});To set attributes on the active span without creating an event:
telemetryService.setActiveSpanAttributes({
"processing.stage": "normalization",
"processing.inputSize": 4096,
});Recording Exceptions
Exceptions recorded on a span appear as structured events in the trace backend and set the span status to ERROR:
import { telemetryService } from "@iqai/adk";
try {
await callExternalApi();
} catch (error) {
telemetryService.recordException(error as Error, {
"error.context": "external_api_call",
"error.service": "payments",
});
throw error;
}Debug Mode: Reading Spans In-Process
When debug: true is set during initialization, spans are also written to an in-memory exporter. Call getTraces() or getTracesForSession() to inspect them without a running backend:
import { telemetryService, AgentBuilder } from "@iqai/adk";
await telemetryService.initialize({
appName: "my-agent-app",
debug: true,
});
const sessionId = "session-abc";
const { runner } = await AgentBuilder.create("my-agent")
.withModel("gemini-2.5-flash")
.build();
for await (const _ of runner.runAsync({
userId: "user-1",
sessionId,
newMessage: { role: "user", parts: [{ text: "Hello!" }] },
})) {
// consume events
}
const spans = telemetryService.getTracesForSession(sessionId);
for (const span of spans) {
console.log(span.name, span.attributes);
}getTraces() returns all spans across all sessions. Both return ReadableSpan[] from @opentelemetry/sdk-trace-base.
Semantic Conventions Reference
Use the exported constants when setting custom attributes to stay consistent with ADK-TS internals:
import { SEMCONV, ADK_ATTRS, OPERATIONS } from "@iqai/adk";
// Standard OpenTelemetry GenAI attributes
SEMCONV.GEN_AI_PROVIDER_NAME; // "gen_ai.provider.name"
SEMCONV.GEN_AI_OPERATION_NAME; // "gen_ai.operation.name"
SEMCONV.GEN_AI_AGENT_NAME; // "gen_ai.agent.name"
SEMCONV.GEN_AI_CONVERSATION_ID; // "gen_ai.conversation.id"
SEMCONV.GEN_AI_TOOL_NAME; // "gen_ai.tool.name"
SEMCONV.GEN_AI_REQUEST_MODEL; // "gen_ai.request.model"
SEMCONV.GEN_AI_USAGE_INPUT_TOKENS; // "gen_ai.usage.input_tokens"
SEMCONV.GEN_AI_USAGE_OUTPUT_TOKENS; // "gen_ai.usage.output_tokens"
// ADK-TS-specific attributes
ADK_ATTRS.SESSION_ID; // "adk.session.id"
ADK_ATTRS.USER_ID; // "adk.user.id"
ADK_ATTRS.INVOCATION_ID; // "adk.invocation.id"
ADK_ATTRS.TOOL_ARGS; // "adk.tool.args"
ADK_ATTRS.TOOL_RESPONSE; // "adk.tool.response"
ADK_ATTRS.LLM_REQUEST; // "adk.llm.request"
ADK_ATTRS.LLM_RESPONSE; // "adk.llm.response"
// Operation name constants
OPERATIONS.INVOKE_AGENT; // "invoke_agent"
OPERATIONS.EXECUTE_TOOL; // "execute_tool"
OPERATIONS.CHAT; // "chat"
OPERATIONS.TRANSFER_AGENT; // "transfer_agent"
OPERATIONS.SEARCH_MEMORY; // "search_memory"Auto-Instrumentation
When enableAutoInstrumentation: true is set, the OpenTelemetry Node SDK instruments outbound HTTP, Express, and NestJS automatically. These appear as child spans under the LLM or tool span that triggered them:
| What gets traced | Notes |
|---|---|
| Outbound HTTP/HTTPS | All node:http and node:https calls |
| Express routes | Incoming request handling |
| NestJS controllers | Route and middleware execution |
Opt-in
Auto-instrumentation is off by default (enableAutoInstrumentation: false).
Enable it when you want visibility into the HTTP calls your tools make to
external APIs.