Observability Is a Product Feature, Not an Operations Tool > **📖 Relationship to Part 3**: Part 3, Chapter 8 "Telemetry and Observability Deep Dive" and Chapter 17 "Telemetry and Analytics System Deep Dive" dissect the technical implementation of the telemetry system in detail— the architecture of five outbound data channels, eight OTel Counters, span hierarchy design, PII redaction mechanisms, and more. This chapter does not repeat the "what" and "how" questions, but focuses on the **engineering significance of observability as a product design philosophy**: why did the Claude Code engineering team choose to elevate observability from an "ops add-on" to a "core product feature"? What benefits did this choice bring, and what costs did it incur? In most software systems, observability (logs, metrics, traces) is "an ops concern"—developers write features, and the operations team adds monitoring. The Claude Code codebase demonstrates a different mindset: **observability is part of product design, directly impacting user experience**. > 💡 **Plain English**: Observability is like a **car dashboard**—not a repair manual for mechanics, but real-time information for the driver. You need to know your current fuel level (token usage), speed (response latency), engine temperature (cache hit rate), and navigation status (what the AI is doing). Without a dashboard, you're driving blind. > 🌍 **Industry Context**: The idea that "observability is a product feature" did not originate with Claude Code—**Google's Dapper** (2010) and the **OpenTelemetry** standard that followed elevated distributed tracing from an "ops add-on" to "infrastructure for system design." The *Google SRE* book (2016) systematically discussed the central role of observability in system design, and Charity Majors (founder of Honeycomb) further popularized the "Observability-First Design" concept in the broader engineering community around 2018. In product observability, **Stripe** is widely regarded as the gold standard—its Dashboard surfaces API call chains, error rates, and latency distributions directly to developer users, turning observability into a core product selling point. Among AI coding assistants, **Cursor** shows "Thinking..." and token consumption counts in its UI; **Aider** provides a `--verbose` mode that outputs API call details; and **LangSmith** (LangChain's observability platform) specifically addresses debugging and tracing for LLM applications, offering complete trace visualization, cost attribution, prompt version management, and evaluation feedback. > > Claude Code's practice needs to be positioned objectively within this industry context: **dual-layer telemetry separation** (OTel for infrastructure + `logEvent` for product analytics) has been the standard architecture for most B2B SaaS products since 2020—any SaaS of meaningful scale separates infrastructure monitoring (Datadog/New Relic/Grafana) from product analytics (Amplitude/Mixpanel/Segment). This is an industry baseline, not an innovation. **Feeding observability data back into runtime decisions** (e.g., dynamically suppressing speculative execution based on cache state) is also a variant of feature flags + A/B testing—LaunchDarkly and Split.io productized this approach back in 2018. What truly sets Claude Code apart is: (1) **closed-loop tracking of classifier decisions**—fine-grained online tracing of permission classifiers fed back into model iteration is rare among similar tools; (2) **SDK-subscribable hook events**—exposing observability as an API to third-party ecosystems; (3) **maturity of anomaly detection at the "whale session" level**—extreme use cases have spawned internal taxonomy and systematic tracking workflows. --- ## Evidence 1: Every Hook Execution Emits Three Events When a hook (a user-configured automation script that runs automatically at specific times; see Part 3, Chapter 4 for details) runs, the system produces three events, like three delivery notifications: ```typescript emitHookStarted(hookId, hookName, hookEvent) // "Package has departed" emitHookProgress(...) // "Package out for delivery" (updated every second) emitHookResponse(...) // "Package delivered" (includes result and error info) ``` These are not "debug logs." These events can be subscribed to by the SDK (via the `includeHookEvents` option), letting integrators know in real time what is happening. When a hook runs for more than a few seconds, users need to see "still running" feedback—otherwise they can't tell whether Claude is working or frozen. Observability = user experience. --- ## Evidence 2: Every Permission Decision Is Tracked The auto-mode classifier logs every run: ```typescript logEvent('tengu_yolo_classifier_result', { toolName, decision, // allow/deny durationMs, reason, }) ``` This is not just for monitoring system health—it is for improving the classifier itself. By tracking which actions are denied and why, engineers can discover patterns such as "the classifier is too conservative" or "the classifier allowed an operation it shouldn't have," and then adjust training data or classification rules. **Observability = data-driven product improvement**. > 📚 **Course Connection**: The closed loop in Evidence 2—"track classifier decisions → discover patterns → adjust rules"—maps to the **PDCA (Plan-Do-Check-Act)** continuous improvement cycle taught in **software engineering** courses, and is also a classic case of the **MLOps monitoring feedback loop** in **machine learning systems** courses—online model performance data is collected and used to improve the next model version. --- ## Evidence 3: Performance Metrics Determine Whether a Feature Lives or Dies Prompt Suggestion (which predicts the user's next message) has suppression logic like this: ```typescript // promptSuggestion.ts const MAX_PARENT_UNCACHED_TOKENS = 10_000 export function getParentCacheSuppressReason( lastAssistantMessage: ... ): string | null { const usage = lastAssistantMessage.message.usage const inputTokens = usage.input_tokens ?? 0 const cacheWriteTokens = usage.cache_creation_input_tokens ?? 0 const outputTokens = usage.output_tokens ?? 0 // If total input + cache write + output tokens exceeds 10,000, return 'cache_cold' return inputTokens + cacheWriteTokens + outputTokens > MAX_PARENT_UNCACHED_TOKENS ? 'cache_cold' : null } // Call site: const cacheReason = getParentCacheSuppressReason(lastAssistantMessage) if (cacheReason) { // cacheReason === 'cache_cold' logSuggestionSuppressed(cacheReason, ...) return null // Do not generate prediction } ``` The `MAX_PARENT_UNCACHED_TOKENS = 10_000` threshold was not plucked from thin air. It came from production data: by tracking prediction hit rates vs. parent request cache states, engineers discovered the pattern that "predictions are pointless during cold cache startup," and then encoded that pattern into a runtime decision. Notably, this threshold is a hard-coded constant—when models are upgraded (larger context windows, different caching strategies), it may need recalibration. The source code does not document how this number was derived (e.g., whether it was an inflection point found by linear regression or a judgment call from a scatter plot), so it is essentially an annotated magic number—a term programmers use to poke fun at "hard-coded values that appear out of nowhere with no explained origin." The underlying "data-driven" process is invisible to the reader. **Observability = runtime product decision**. But the actual degree of "data-driven" rigor depends on whether the threshold calibration process is documented—and documentation culture is itself part of observability culture. --- ## Evidence 4: WebhookHook Debug Logs ```typescript // hooks.ts emitHookResponse() // Always log full hook output to debug log for verbose mode debugging const outputToLog = data.stdout || data.stderr || data.output if (outputToLog) { logForDebugging( `Hook ${data.hookName} (${data.hookEvent}) ${data.outcome}:\n${outputToLog}`, ) } ``` Note the word "Always"—regardless of whether the hook succeeded or what type of event it is, the full stdout/stderr goes into the debug log (visible via `claude --debug`). This is user-facing observability: when your hook behaves unexpectedly, you have tools to troubleshoot. --- ## Evidence 5: Whale Sessions—A Complete Case of Observability-Driven Product Improvement This is the most compelling evidence in the chapter, because it shows the complete closed loop from "anomaly detection" to "root cause identification" to "code fix." ### How the Problem Was Discovered A BQ (BigQuery—Google's large-scale data analytics tool that enables ad-hoc queries over massive log volumes) analysis comment in the source code documents the full investigation chain. The following English comment is the development team's "case notes," recording how they discovered and pinpointed a serious memory issue: ```typescript // types.ts — InProcessTeammateTask /** * BQ analysis (round 9, 2026-03-20) showed ~20MB RSS per agent at 500+ turn * sessions and ~125MB per concurrent agent in swarm bursts. Whale session * 9a990de8 launched 292 agents in 2 minutes and reached 36.8GB. The dominant * cost is this array holding a second full copy of every message. */ export const TEAMMATE_MESSAGES_UI_CAP = 50 ``` "Round 9" indicates this was not a one-off query, but the team's ninth round of systematic BQ analysis—meaning there is a regular, numbered data review process. ### Pinpointing the Specific Session The engineering team did not stop at a statistical conclusion like "average memory is high." By grouping RSS memory distributions by session in BigQuery, they found an extreme outlier—session `9a990de8`. The behavioral signature of this session was unmistakable: **292 agents launched in 2 minutes**, with total RSS (Resident Set Size—the actual memory occupied by a program) reaching 36.8 GB (a typical laptop has only 8–16 GB of RAM, so this single session consumed the memory of two full computers). The team coined an internal term for such extreme sessions: **"whale sessions."** The very existence of this name shows that anomaly detection has matured into an internal taxonomy: it's not "some user reported a bug," but "we have a standard category called whale session, defined as…" ### Root Cause Analysis After pinpointing the session, they traced the main memory consumer: the `task.messages` array. This array was intended to feed data to the UI's "zoom into transcript" dialog, but it held a **full copy of every message**. In normal sessions this was not a problem, but in swarm mode, where hundreds of agents are spawned, each agent's `task.messages` independently held a complete copy of all messages, causing linear memory explosion. ### The Fix The fix also reflects the precision guided by observability data—rather than crudely limiting the number of agents, they applied two precise interventions targeting the root cause: 1. **Cap `task.messages` for UI purposes** (`TEAMMATE_MESSAGES_UI_CAP = 50`), retaining only the most recent 50 messages for UI display, while the full conversation remains in the local disk transcript file. 2. **Actively free memory during agent cleanup** (`runAgent.ts`)—clearing cloned fork contexts, file state caches, Perfetto registrations, transcript directory mappings, and orphaned todo entries: ```typescript // runAgent.ts — agent cleanup logic agentToolUseContext.readFileState.clear() // Free file state cache initialMessages.length = 0 // Free fork context messages unregisterPerfettoAgent(agentId) // Free Perfetto registration clearAgentTranscriptSubdir(agentId) // Free transcript mapping // Clean up orphaned todo entries—whale sessions spawn hundreds of agents, // and each orphaned key is a small leak that adds up ``` ### Why This Case Is the Most Persuasive This complete chain—**regular BQ analysis → anomalous session identification → memory attribution → precise fix**—is the most vivid proof of "observability-driven product improvement." It demonstrates three things: 1. **Observability must be continuous and organized**: "Round 9" means this was not a sporadic investigation, but a systematic data review cadence. 2. **Coarse-grained alerts are insufficient; session-level traceability is needed**: if the only metric were "average RSS," the 36.8 GB whale session would be hidden by the mean. 3. **The precision of the fix depends on the precision of the observability data**: because they could trace the memory consumer to "the full copy in the `task.messages` array," the fix could be a precise cap rather than a crude feature restriction. Without observability, this problem might have remained "some users report that Claude Code uses a lot of memory," then been dismissed as "AI just uses a lot of RAM." With observability, it became "session 9a990de8 launched 292 agents in 2 minutes; the `task.messages` array is the dominant memory consumer"—an engineering problem with a precise fix. --- ## Two-Layer Telemetry Separation (Industry Standard Practice) > **Positioning note**: Dual-layer telemetry separation has been the industry baseline for B2B SaaS products since 2020; it is not unique to Claude Code. Any SaaS of meaningful scale separates infrastructure monitoring (Datadog/New Relic/Grafana) from product analytics (Amplitude/Mixpanel/Segment). This section records Claude Code's concrete implementation of this standard practice, focusing not on *what* was done, but on *notable engineering details in how it was done*. For the complete technical architecture of the five data channels, see Part 3, Chapter 8. Claude Code has two kinds of telemetry, serving different purposes: **OpenTelemetry (OTel) — For Infrastructure** ```typescript // telemetry/sessionTracing.ts startHookSpan(), endHookSpan() startInteractionSpan(), endInteractionSpan() ``` Spans and traces for tracking request chains, performance bottlenecks, and error rates. Classic APM (Application Performance Monitoring) territory. **logEvent — For Product Analytics** ```typescript logEvent('tengu_conversation_forked', { message_count: serializedMessages.length, has_custom_title: !!title, }) ``` Product events for tracking feature usage, A/B tests, and feature flag effectiveness. Designed to avoid logging raw user content, containing only structured statistics—but note that structured fields may indirectly contain contextual clues (e.g., `toolName: 'bash'` + `reason: 'contains rm -rf'` can infer user intent), so the claim of "no PII" needs qualification. The two layers are separate, each serving different consumers—infrastructure teams look at OTel, product teams look at logEvent. **Notable engineering detail**: Because these two systems record independently, a correlation question arises—if a request shows as successful in an OTel trace but logEvent records an anomaly, how do you correlate them? Do the two systems share a correlation ID? This cross-layer correlation complexity is the hidden cost of dual-layer separation (discussed further in the "Costs and Reflections" section below). --- ## Design Principle: Treat Observability as Part of the Product This approach in software engineering is known as **"Observability-First Design."** Its conceptual roots trace back to the *Google SRE* book (2016), which systematically discussed the central role of observability in system design, and Charity Majors (founder of Honeycomb) further popularized this idea in the broader engineering community around 2018. The core tenet is that observability is not an add-on after features are complete, but should be planned during architecture design. Claude Code's practice aligns with this philosophy—but it is important to distinguish between standard implementation and distinctive practices: 1. **Emit points included at feature design time** (industry standard practice): the hook system, permission system, and speculation logic all have emits in their core logic, not added as afterthoughts. This is a basic requirement of Observability-First Design. 2. **Data-driven product decisions** (industry standard practice): prediction suppression thresholds, cache hit rate monitoring—these runtime behaviors come from data analysis. Essentially, this is the standard pattern of feature flags + data feedback. 3. **User-facing observability + SDK subscribability** (rare among AI coding assistants): `--debug` mode, hook execution event streams (subscribable by third-party SDKs via the `includeHookEvents` option)—users and integrators can see what the system is doing. This is a leap from "product feature" to "platform capability." 4. **Closed-loop classifier decision tracking** (rare among AI coding assistants): online behavior tracking of the permission classifier → pattern discovery → training data/rule adjustment is an MLOps feedback loop embedded in the core product flow. Cursor's permission model uses static rules; Aider has no automatic permission classification—this closed loop is Claude Code's true differentiator. 5. **Clear separation of purpose** (industry baseline): OTel for infrastructure, logEvent for product analytics. See the positioning note in the "Two-Layer Telemetry Separation" section above. > 📚 **Course Connection**: The dual-layer telemetry architecture (OTel + logEvent) maps to the **Separation of Concerns** principle taught in **distributed systems** courses. Understanding OTel's span/trace model requires knowledge of request chain tracing from **computer networking** courses; the structured event design of logEvent is an application of the event log (Event Sourcing) pattern from **database systems** courses. --- ## Implications for Your System When you are building an AI application, ask yourself: - When the AI behaves abnormally, do I have tools to troubleshoot? - How do I know whether an optimization (like prompt cache) is actually working? - Can I trace "which specific session triggered this bug"? - Can users know what the AI is doing right now (during long-running tasks)? The answers to these questions depend on whether you treated observability as part of the feature design—not something to "add after launch." --- ## Code Landmarks - `src/services/analytics/index.ts`: `logEvent()` / `logEventAsync()` functions—product analytics-level structured logs (see Part 3, Chapter 17) - `src/utils/telemetry/instrumentation.ts`: OTel initialization and Counters—infrastructure-level distributed tracing (see Part 3, Chapter 8) - `src/utils/telemetry/sessionTracing.ts`: Span hierarchy management—`startHookSpan` / `startInteractionSpan`, etc. - `src/services/PromptSuggestion/promptSuggestion.ts`: `getParentCacheSuppressReason()` + `MAX_PARENT_UNCACHED_TOKENS` threshold - `src/tasks/InProcessTeammateTask/types.ts`: Whale session BQ analysis comment + `TEAMMATE_MESSAGES_UI_CAP` - `src/tools/AgentTool/runAgent.ts`: Agent memory cleanup logic (whale session fix) - `src/utils/hooks/hookEvents.ts`: Hook event emission—the `emitHookStarted` / `emitHookProgress` / `emitHookResponse` three-event pattern (note: the function is named `emitHookResponse` rather than `emitHookCompleted`, semantically meaning "response data returned" rather than "execution finished," because this event carries the full stdout/stderr/exitCode/outcome) ## Costs and Reflections A discussion of engineering philosophy that only talks about benefits and not costs is not philosophy—it's advertising. The "embedded observability" model in Claude Code shows significant value (especially the whale session case), but its costs are equally worthy of systematic examination. ### 1. The Telemetry Code Maintenance Tax Analysis in Part 3, Chapter 8 shows that the Claude Code codebase contains **over 7,400 lines of telemetry-related code**, spread across `src/utils/telemetry/` (4 core files, 3,363 lines) and `src/services/analytics/` (9 files, 4,040 lines). This means telemetry code is itself a medium-sized subsystem requiring independent testing, maintenance, and evolution. Every emit point is a contract—the event name, field structure, and semantic meaning form an interface between producer (business logic) and consumer (analytics pipelines, dashboards, alerting rules). If the event format changes (e.g., adding a new field to `tengu_yolo_classifier_result` or changing the enum values of `reason`), all downstream consumers must be updated. Under the architecture of five outbound data channels, a single event format change may require synchronously updating BigQuery schemas, Datadog dashboards, first-party log parsers, and other downstream systems—a classic "distributed schema evolution" problem. ### 2. The Tension Between Privacy and Debuggability The `toolName` and `reason` fields logged by `logEvent` in Evidence 2 are designed not to contain raw user content, but structured fields may indirectly leak user intent. For example, the combination `toolName: 'bash'` + `reason: 'contains rm -rf'` can infer that the user is executing a dangerous operation. A more accurate statement is "designed to avoid logging raw user content (PII), but structured fields may contain contextual clues." This design choice creates a dilemma: when the classifier misclassifies, engineers only have `toolName` and `reason`, not the user's original prompt—how do they reproduce the issue? If logging granularity is increased for debuggability, it crosses the privacy red line; if logging content is strictly limited, some bugs can only be reproduced if users proactively submit reproduction steps. Claude Code leans toward the privacy-protection side—a reasonable but costly trade-off, especially in AI agent classifier scenarios where "decision correctness directly impacts user safety." ### 3. Consistency Challenges Across the Two Layers OTel and logEvent record events independently. This separation brings the benefit of concern isolation, but also introduces cross-layer correlation complexity: if a request shows as successful in an OTel trace (span status OK) but logEvent records a classifier anomaly for that request—the operations team and the product team see two different "truths" for the same request. The key question is: do the two systems share a correlation ID? From the source code, OTel uses the standard trace ID / span ID system, while logEvent uses a session ID + event timestamp system. Correlating the two requires inference via time windows and session granularity, rather than precise ID matching. This works adequately for everyday scenarios, but in high-concurrency swarm contexts (e.g., 292 agents launched in 2 minutes in a whale session), time-window-based correlation may become ambiguous. ### 4. The Long-Term Sustainability of the "Log Liberally" Strategy The design decision in Evidence 4 to "Always log full hook output" embodies the "better to log too much than too little" principle—regardless of hook success or event type, full stdout/stderr goes into the debug log. This creates tension with data bloat risk. In a single-user local execution context, debug log storage costs are essentially negligible. But logEvent product events sent to the server are different—every emit point, multiplied by millions of DAU across five outbound data channels, creates non-linear storage and transmission growth. The whitelist filtering in `datadog.ts` and the sampling mechanism in `sink.ts` show the team is already managing data volume, but this is itself a maintenance cost of the "log liberally" strategy—you need to continuously maintain sampling rules and whitelists to ensure you neither miss critical signals nor drown in noise. ### 5. Security Boundaries of `--debug` Mode The user-facing `--debug` mode is an important UI for observability, but it also exposes internal implementation details—function call chains, cache states, hook execution flows, and more. For benign users this is a powerful troubleshooting tool; for attackers it is also a source of reverse-engineering clues. The more critical question is: does `--debug` output have information classification? Is there a distinction between "user-level debug info" (e.g., "hook X timed out") and "internal implementation details" (e.g., internal OTel span structures, telemetry pipeline routing logic)? From the source code, `logForDebugging` is a unified output port with no obvious classification mechanism. This means `--debug` is an "all or nothing" switch, potentially exposing unnecessary internal information while helping users troubleshoot. ### Trade-off Summary | Dimension | Benefit | Cost | |-----------|---------|------| | Maintenance | Early problem detection (e.g., whale sessions) | Continuous maintenance of 7,400+ lines of telemetry code, schema evolution coordination | | Privacy | Structured stats without raw user content | Indirect contextual leakage risk + lack of reproduction data when classifiers misclassify | | Consistency | Each layer serves its own purpose, concern separation | Cross-layer correlation relies on time-window inference, potentially ambiguous under high concurrency | | Storage | Complete logs support deep troubleshooting | Five channels × millions of DAU = non-linear storage growth | | Security | Users have tools to troubleshoot | `--debug` has no classification, potentially exposing too much internal implementation | A mature engineering philosophy requires a clear-eyed awareness of its own costs. Claude Code's observability system is excellent in the dimension of "what was done"—the whale session case proves the ROI. But there is still room for improvement in the dimension of "cost management," particularly in cross-layer correlation, the privacy-debuggability balance, and debug information classification.