uptrace/uptrace
Open source APM: OpenTelemetry traces, metrics, and logs
10 hidden assumptions · 7-stage pipeline · 8 components
Like any codebase, this fullstack makes assumptions it never checks — most are routine. The ones worth your attention are below.
Ingests OpenTelemetry traces, metrics, and logs then stores and visualizes them
OpenTelemetry agents send traces/metrics/logs to HTTP endpoints, where they're authenticated by project tokens, converted from OTLP protobuf to internal formats, processed through task queues for aggregation and validation, then stored in ClickHouse (time-series data) and PostgreSQL (metadata). The Vue.js frontend queries this data through REST APIs to render dashboards, traces, and alerts.
Under the hood, the system uses 3 feedback loops, 4 data pools, 4 control points to manage its runtime behavior.
A 8-component fullstack. 398 files analyzed. Data flows through 7 distinct pipeline stages.
Hidden Assumptions
Most of what this code assumes is routine. These 3 are the ones most likely to cause trouble here. The rest are minor; they're under "Show everything".
If the PCG generator enters an invalid state or the mutex fails, trace ID generation could produce duplicate IDs across distributed services, breaking trace correlation
Invalid hex characters like 'G' or 'Z' cause hex.DecodeString to fail silently, falling back to ParseUint which interprets the string as decimal, producing wrong span IDs
Under high load, the producer can overwhelm Redis memory or task queue capacity, causing job loss or Redis OOM crashes
Show everything (7 more)
ClickHouse server is running on default port with 'test' database already created and accessible without authentication
If this fails: Application panics on db.Ping() if ClickHouse is down, database doesn't exist, or requires credentials - no graceful degradation
pkg/clickhouse/ch/example/basic/main.go:main
Running 1 million iterations with 1-second sleeps (≈11.5 days runtime) with 20 concurrent goroutines per iteration without any memory cleanup or connection pooling limits
If this fails: Memory leak from accumulated goroutines and Redis connections, eventually causing OOM or connection pool exhaustion
example/kvrocks/main.go:main
uptrace.ConfigureOpentelemetry() successfully initializes telemetry without checking return value or error state before creating database connections and HTTP handlers
If this fails: If OpenTelemetry setup fails silently, spans and metrics are lost without indication, breaking observability while application appears to work normally
example/gin-gorm/main.go:main
seed1() and seed2() functions provide sufficient entropy at process startup time and PCG generator state never needs reseeding during long-running processes
If this fails: In containerized environments with identical startup times or low-entropy systems, multiple processes could generate overlapping trace ID sequences
pkg/idgen/traceid.go:traceIDRand initialization
DSN format 'http://project2_secret_token@localhost:14318/2' follows exact Uptrace URL structure with project ID as path suffix and token in auth section
If this fails: If DSN parsing logic changes or project ID encoding differs, authentication fails silently with data being dropped rather than rejected with clear errors
example/sentry-go/main.go:main
Each of 20 goroutines can safely access Redis client concurrently without connection pool exhaustion, and sync.WaitGroup correctly handles concurrent Add/Done calls
If this fails: Under high concurrency, Redis connection pool could be exhausted leading to blocked goroutines or timeouts, causing trace spans to never complete
example/kvrocks/main.go:handleRequest
Input uint64 values represent valid 40-bit span IDs but function doesn't validate the upper 24 bits are zero before truncation to SpanID
If this fails: Large uint64 values get silently truncated to 40 bits, potentially creating span ID collisions that break parent-child span relationships in traces
pkg/idgen/spanid.go:SpanIDFromUint64
Open the standalone hidden-assumptions report for uptrace →
How Data Flows Through the System
OpenTelemetry agents send traces/metrics/logs to HTTP endpoints, where they're authenticated by project tokens, converted from OTLP protobuf to internal formats, processed through task queues for aggregation and validation, then stored in ClickHouse (time-series data) and PostgreSQL (metadata). The Vue.js frontend queries this data through REST APIs to render dashboards, traces, and alerts.
- Authenticate OTLP ingestion — HTTP handlers in pkg/bunapp/ extract project tokens from Authorization headers, validate them against PostgreSQL project records, and establish request context for data isolation [HTTP request with OTLP data → Authenticated context]
- Parse OTLP protobuf data — pkg/otlpconv/ deserializes incoming protobuf messages (traces, metrics, logs) into Go structs, handling different OTLP versions and extracting resource attributes [OTLP protobuf bytes → OTLP structs]
- Convert to internal format — Processors in pkg/tracing/ and pkg/metrics/ transform OTLP data into internal Span and Metric models, generating IDs via pkg/idgen/, extracting service names, and normalizing timestamps [OTLP structs → Span]
- Queue for background processing — pkg/taskq/ queues processing tasks using Redis/PostgreSQL backends, enabling horizontal scaling and failure recovery for high-volume telemetry ingestion [Span → Queued tasks]
- Aggregate and validate data — Background workers aggregate metrics by time windows, validate span relationships, compute derived metrics like error rates, and prepare batch inserts for ClickHouse [Queued tasks → Aggregated data]
- Store in ClickHouse — pkg/clickhouse/ CHClient batches data insertions into time-partitioned tables, with spans stored by trace ID and metrics pre-aggregated for fast dashboard queries [Aggregated data]
- Query and visualize — Vue.js frontend calls REST APIs that execute SQL queries against ClickHouse, building trace graphs, metric charts, and service maps with real-time updates [Query requests → Dashboard data]
Data Models
The data structures that flow between stages — the contracts that hold the system together.
pkg/tracing/struct with TraceID (16-byte array), SpanID (5-byte uint40), ParentSpanID, OperationName (string), StartTime/EndTime (time.Time), Tags (key-value map), Events (array of timestamped events), Status (code + message)
Created from OTLP protobuf messages, validated and converted to internal format, stored in ClickHouse partitioned by time, queried via SQL-like syntax for trace visualization
pkg/metrics/struct with Name (string), Type (gauge/counter/histogram), Value (float64 or distribution), Timestamp (time.Time), Labels (key-value map), Resource attributes
Received as OTLP metrics, aggregated by time windows and label combinations, stored as time-series data in ClickHouse with pre-computed aggregations for dashboard queries
pkg/idgen/traceid.go128-bit identifier represented as [16]byte array with hex string encoding/decoding, time-based generation using PCG random number generator
Generated using timestamp + randomness for uniqueness, encoded as 32-character hex strings in APIs, stored as binary in ClickHouse for efficient indexing
pkg/org/PostgreSQL model with ID (uint32), Name (string), Token (string for authentication), Settings (JSON configuration), CreatedAt/UpdatedAt timestamps
Created through UI or config, used to authenticate OTLP ingestion via tokens, determines data isolation and retention policies
pkg/bunapp/struct with ID, Name, ProjectID, GridRows (array of chart configurations), TableConfig, TimeRange settings, rendered from YAML templates
Defined in YAML templates, instantiated per project with metric queries, rendered in Vue.js frontend with real-time data from ClickHouse
System Behavior
How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.
Data Pools
Time-partitioned table storing distributed traces with spans indexed by trace ID, service name, and timestamp for fast retrieval
Pre-aggregated time-series metrics with materialized views for different time granularities (1m, 5m, 1h) to accelerate dashboard queries
Stores projects, users, alert rules, and dashboard configurations with transactional consistency for system management
Redis or PostgreSQL-backed queues buffer telemetry processing tasks, enabling async processing and horizontal scaling
Feedback Loops
- Metric aggregation pipeline (convergence, balancing) — Trigger: Incoming metrics trigger window-based aggregation. Action: Compute averages, sums, percentiles for each time window and label combination. Exit: Window closes and aggregated values are persisted.
- Alert evaluation loop (polling, reinforcing) — Trigger: Scheduled interval checks alert rules. Action: Query metrics/spans against rule conditions and update alert states. Exit: All rules evaluated, notifications sent if state changes.
- Task queue retry (retry, balancing) — Trigger: Task processing failure. Action: Exponential backoff delay then requeue failed task. Exit: Max retries reached or task succeeds.
Delays
- Batch insertion window (batch-window, ~configurable (default ~1s)) — Telemetry data appears in queries after batch window closes
- Metric aggregation window (batch-window, ~1m/5m/1h depending on resolution) — Metrics show in dashboards at window boundaries
- Task queue processing (queue-drain, ~depends on queue depth and worker count) — High ingestion rates can delay background processing
Control Points
- Project token authentication (feature-flag) — Controls: Which telemetry data is accepted and how it's isolated by project. Default: enabled
- ClickHouse retention policy (env-var) — Controls: How long telemetry data is stored before automatic deletion. Default: configurable per project
- Task queue backend (architecture-switch) — Controls: Whether to use Redis, PostgreSQL, or in-memory queues for background processing. Default: Redis default
- Metric aggregation intervals (runtime-toggle) — Controls: Time window sizes for pre-computing metric rollups (affects query performance vs storage). Default: 1m,5m,1h
Technology Stack
Primary time-series database for storing and querying observability data with columnar compression
Metadata storage for projects, users, alerts, and system configuration with ACID guarantees
Frontend framework for building interactive dashboards and trace visualization UI
Standard protocol for ingesting traces, metrics, and logs from instrumented applications
Default backend for distributed task queues enabling async telemetry processing
Dependency injection framework for managing application component lifecycle and configuration
HTTP router handling OTLP ingestion endpoints and API routes with OpenTelemetry instrumentation
Binary serialization format for efficient task queue payloads and internal data structures
Key Components
- App (orchestrator) — Main application coordinator that initializes all subsystems, configures databases, starts HTTP server, and manages application lifecycle using Uber FX dependency injection
pkg/bunapp/app.go - SpanProcessor (processor) — Converts incoming OTLP span data to internal format, validates span structure, extracts service information, and prepares spans for storage in ClickHouse
pkg/tracing/ - MetricProcessor (processor) — Aggregates incoming metrics by time windows and label combinations, computes histograms and percentiles, and batches data for efficient ClickHouse insertion
pkg/metrics/ - CHClient (adapter) — ClickHouse database client that provides ORM-like interface for inserting time-series data and executing complex analytical queries with automatic partitioning
pkg/clickhouse/ch/db.go - QueueFactory (factory) — Creates and manages distributed task queues with pluggable backends (Redis, PostgreSQL, in-memory) for background processing of telemetry data
pkg/taskq/factory.go - Router (gateway) — HTTP request router that handles OTLP ingestion endpoints, authentication, CORS, and serves the Vue.js frontend with OpenTelemetry instrumentation
pkg/bunapp/ - AlertManager (monitor) — Evaluates alerting rules against metrics and spans, sends notifications via email/Slack/webhooks, and manages alert state transitions
pkg/madalarm/ - IDGenerator (factory) — Generates globally unique trace and span IDs using time-based approach with PCG random number generator, ensuring uniqueness across distributed systems
pkg/idgen/
Package Structure
Core APM server that ingests OpenTelemetry data, stores it in ClickHouse/PostgreSQL, and serves a Vue.js web UI for observability analysis
ClickHouse database client with ORM-like features for storing and querying observability data
Distributed task queue framework with multiple backends (Redis, PostgreSQL, in-memory)
Collection of utility packages for ID generation, message packing, tag parsing, and other common operations
Example applications demonstrating Uptrace integration with various frameworks (Gin, Sentry, Redis, etc.)
Explore the interactive analysis
See the full architecture map, data flow, and code patterns visualization.
Analyze on CodeSeaRelated Fullstack Repositories
Frequently Asked Questions
What is uptrace used for?
Ingests OpenTelemetry traces, metrics, and logs then stores and visualizes them uptrace/uptrace is a 8-component fullstack written in Go. Data flows through 7 distinct pipeline stages. The codebase contains 398 files.
How is uptrace architected?
uptrace is organized into 5 architecture layers: Web Interface, API Server, Data Processing, Storage Layer, and 1 more. Data flows through 7 distinct pipeline stages. This layered structure keeps concerns separated and modules independent.
How does data flow through uptrace?
Data moves through 7 stages: Authenticate OTLP ingestion → Parse OTLP protobuf data → Convert to internal format → Queue for background processing → Aggregate and validate data → .... OpenTelemetry agents send traces/metrics/logs to HTTP endpoints, where they're authenticated by project tokens, converted from OTLP protobuf to internal formats, processed through task queues for aggregation and validation, then stored in ClickHouse (time-series data) and PostgreSQL (metadata). The Vue.js frontend queries this data through REST APIs to render dashboards, traces, and alerts. This pipeline design reflects a complex multi-stage processing system.
What technologies does uptrace use?
The core stack includes ClickHouse (Primary time-series database for storing and querying observability data with columnar compression), PostgreSQL (Metadata storage for projects, users, alerts, and system configuration with ACID guarantees), Vue.js (Frontend framework for building interactive dashboards and trace visualization UI), OpenTelemetry (Standard protocol for ingesting traces, metrics, and logs from instrumented applications), Redis (Default backend for distributed task queues enabling async telemetry processing), Uber FX (Dependency injection framework for managing application component lifecycle and configuration), and 2 more. A focused set of dependencies that keeps the build manageable.
What system dynamics does uptrace have?
uptrace exhibits 4 data pools (ClickHouse traces table, ClickHouse metrics table), 3 feedback loops, 4 control points, 3 delays. The feedback loops handle convergence and polling. These runtime behaviors shape how the system responds to load, failures, and configuration changes.
What design patterns does uptrace use?
5 design patterns detected: Multi-tenant data isolation, Time-partitioned storage, Configurable dashboard templates, Pluggable task queue backends, OpenTelemetry-native ingestion.
Analyzed on April 20, 2026 by CodeSea. Written by Karolina Sarna.