highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging, distributed tracing, and more.
13 hidden assumptions · 6-stage pipeline · 8 components
Like any codebase, this fullstack makes assumptions it never checks — most are routine. The ones worth your attention are below.
Captures user sessions, application errors, and performance data for full-stack monitoring
Client applications instrumented with Highlight SDKs capture user interactions, errors, and performance data, sending it via OpenTelemetry to the Go backend. The backend receives data through OTLP endpoints, queues it in Kafka for processing, then stores structured data in ClickHouse. Frontend applications query this data through GraphQL to display monitoring dashboards, session replays, and analytics.
Under the hood, the system uses 3 feedback loops, 4 data pools, 4 control points to manage its runtime behavior.
A 8-component fullstack. 2975 files analyzed. Data flows through 6 distinct pipeline stages.
Hidden Assumptions
Most of what this code assumes is routine. These 3 are the ones most likely to cause trouble here. The rest are minor; they're under "Show everything".
If client sends malformed JSON or non-JSON content type, the handler will crash with JSONDecodeError before Highlight can capture the error context
If Redis is down or unreachable, the application fails to start without graceful degradation, making entire monitoring stack unavailable
Under load or with slow logging backends, requests will timeout causing 504 errors while partial telemetry data may be lost mid-processing
Show everything (10 more)
OTLP endpoint at localhost:4318 is always reachable for telemetry export
If this fails: When OTLP collector is down, telemetry data is silently dropped without retry logic, making monitoring invisible during outages
e2e/nextjs/go-service/main.go:highlight.SetOTLPEndpoint
HTTP request to pub.highlight.io/v1/logs/json expects specific JSON structure with message, timestamp, level fields
If this fails: If external logging endpoint changes its API contract, the service will receive 400 errors but continue processing, silently losing log forwarding capability
e2e/nestjs/src/app.service.ts:firstValueFrom
span.end() is called after HTTP request completion but code structure doesn't guarantee this ordering if HTTP request throws
If this fails: If HTTP request fails, span remains open causing memory leaks in trace collection and incorrect duration measurements
e2e/nestjs/src/app.service.ts:span.end
traceparent header format '00-${traceId}-${spanId}-01' matches W3C trace context specification exactly
If this fails: If traceId or spanId contain invalid characters or wrong lengths, distributed tracing correlation breaks silently between frontend and backend
e2e/hono/src/index.ts:layout
External fetch to 'https://highlight.io/index.js' will always return valid response with predictable content-type headers
If this fails: If remote server returns unexpected content types or is unreachable, worker crashes without proper error handling for content type determination
e2e/cloudflare-worker/src/index.ts:fetch
H.parseHeaders expects request headers to contain secureSessionId and requestId in specific format from Highlight SDK
If this fails: If client sends malformed or missing Highlight headers, error reporting silently fails to associate with user sessions, losing crucial debugging context
e2e/express/src/index.mjs:H.parseHeaders
Heavy computation loop with Math.random() * 1000 for 1000 iterations can complete within single request lifecycle
If this fails: Under high concurrency, CPU-intensive random number generation blocks event loop causing request timeouts and degraded performance
e2e/express/src/index.mjs:Math.random
AWS environment variables (E2E_AWS_ACCESS_KEY, E2E_AWS_SECRET_KEY) contain valid credentials when present
If this fails: Invalid AWS credentials cause silent connection failures in try/catch block, but SQS operations later in code will fail cryptically
e2e/python/highlight_fastapi/main.py:boto3.client
1 second sleep in defer function doesn't exceed HTTP client timeout expectations
If this fails: Client connections may timeout waiting for response while server artificially delays with sleep, causing poor user experience
e2e/nextjs/go-service/main.go:time.Sleep
Project ID '1jdkoe52' is hardcoded and matches the backend's expected project configuration
If this fails: If project ID becomes invalid or changes, all telemetry data is rejected by backend but application continues running with silent monitoring failure
e2e/hono/src/index.ts:H.init
Open the standalone hidden-assumptions report for highlight →
How Data Flows Through the System
Client applications instrumented with Highlight SDKs capture user interactions, errors, and performance data, sending it via OpenTelemetry to the Go backend. The backend receives data through OTLP endpoints, queues it in Kafka for processing, then stores structured data in ClickHouse. Frontend applications query this data through GraphQL to display monitoring dashboards, session replays, and analytics.
- Instrument client applications — Highlight SDKs (highlight-react, highlight-node, etc.) are integrated into applications to automatically capture DOM events, console logs, network requests, errors, and performance metrics, wrapping them in OpenTelemetry format
- Transmit telemetry data — SDKs send captured data to the backend via OTLP (OpenTelemetry Protocol) over HTTP/gRPC, including session recordings, error reports, logs, and distributed traces with proper correlation IDs [TraceSpan]
- Receive and validate data — The OTLPReceiver component in the Go backend receives telemetry data, validates project IDs and authentication, then publishes validated messages to Kafka topics organized by data type
- Process telemetry streams — KafkaConsumer workers read from topic partitions, with SessionProcessor handling session event ordering, ErrorGrouper clustering similar errors, and trace processors building distributed trace trees [SessionEvent → Processed telemetry data]
- Store in time-series database — ClickHouseStore writes processed data to time-partitioned tables optimized for analytical queries, with separate tables for sessions, errors, logs, and traces including proper indexing [Processed telemetry data → Query results]
- Query and visualize data — Frontend applications send GraphQL queries to the public API, which translates them into optimized ClickHouse queries, returning data for dashboards, session replays, error tracking, and performance monitoring [Query results → GraphQL responses]
Data Models
The data structures that flow between stages — the contracts that hold the system together.
backend/store/session.gostruct with ID, ProjectID, SessionID, Type (click|navigation|error), Timestamp, and Payload containing event-specific data like coordinates, URLs, or error details
Created by client SDKs during user interactions, queued through Kafka, processed and stored in ClickHouse, then retrieved for session replay playback
backend/model/error.gostruct with ID, ProjectID, ErrorGroupID, SessionID, Message, StackTrace, Timestamp, Environment, and Source containing the original error information
Generated when applications throw exceptions, sent via SDK to backend, grouped by similarity into ErrorGroups, stored in database for querying and alerting
backend/store/logs.gostruct with Timestamp, Level (info|warn|error), Message, Attributes map[string]interface{}, TraceID, SpanID, and ServiceName for distributed tracing context
Emitted by application logging libraries, instrumented by Highlight SDKs, sent via OTLP to backend, stored in ClickHouse with full-text search indexing
backend/otel/trace.goOpenTelemetry span with TraceID, SpanID, ParentSpanID, OperationName, StartTime, Duration, Tags, and Logs following the OTEL specification
Created by OpenTelemetry instrumentation in application code, sent to backend via OTLP protocol, stored in ClickHouse for trace analysis and performance monitoring
backend/model/project.gostruct with ID, Name, WorkspaceID, Settings containing rate limits, retention policies, and feature flags that control data collection behavior
Created through frontend dashboard, stored in PostgreSQL, referenced by all telemetry data to enforce project-level settings and permissions
System Behavior
How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.
Data Pools
Temporary storage for incoming telemetry data partitioned by project and data type to enable parallel processing
Time-series database storing all telemetry data with columnar compression and time-based partitioning for fast analytical queries
Relational database storing projects, users, settings, and other metadata that controls system behavior and access permissions
In-memory cache for active session data and real-time features like live session viewing and immediate error notifications
Feedback Loops
- Error Rate Alerting (auto-scale, balancing) — Trigger: ErrorGrouper detects error rate spike above threshold. Action: Triggers alert webhooks and increases processing capacity for error ingestion. Exit: Error rate returns to normal baseline.
- Data Retention Cleanup (scheduled-job, balancing) — Trigger: Daily cron job checks data age against project retention policies. Action: Deletes old session recordings, logs, and traces to maintain storage limits. Exit: All expired data removed.
- SDK Retry Logic (retry, reinforcing) — Trigger: Network failure or backend unavailability when sending telemetry. Action: SDKs implement exponential backoff retry with local queuing to avoid data loss. Exit: Data successfully transmitted or retry limit reached.
Delays
- Session Replay Processing (batch-window, ~5-10 seconds) — Session events are batched before processing to build coherent timelines, causing slight delay in replay availability
- ClickHouse Ingestion (eventual-consistency, ~1-3 seconds) — Newly ingested data may not immediately appear in queries due to ClickHouse's async insert behavior and materialized view updates
- Error Grouping (async-processing, ~variable) — Similar errors are grouped asynchronously, so new errors may initially appear as separate groups before being merged
Control Points
- Project Rate Limits (rate-limit) — Controls: Maximum events per minute that can be ingested per project to prevent abuse and ensure fair resource usage. Default: configurable per project
- Session Recording Quality (architecture-switch) — Controls: Whether to capture full DOM mutations, network requests, and console logs vs minimal session data. Default: full capture by default
- Data Retention Period (hyperparameter) — Controls: How long telemetry data is stored before automatic deletion, affecting storage costs and query performance. Default: 30 days default
- Sampling Rate (sampling-strategy) — Controls: Percentage of sessions, traces, or errors that are actually captured and sent to reduce data volume. Default: 100% by default
Technology Stack
Backend service runtime for high-performance data ingestion and processing with strong concurrency support
Frontend framework for building the monitoring dashboard and session replay interfaces with real-time data visualization
Columnar database optimized for analytical queries on time-series telemetry data with compression and fast aggregations
Message queue system for reliable ingestion and processing of high-volume telemetry data streams with partitioning
Standardized telemetry collection across all SDKs and services for distributed tracing, metrics, and structured logging
API layer for frontend-backend communication with efficient querying and real-time subscriptions for monitoring data
Monorepo management for coordinated building, testing, and publishing of multiple JavaScript packages and applications
Type-safe JavaScript development across all frontend applications and Node.js SDKs with shared type definitions
Key Components
- KafkaConsumer (processor) — Consumes telemetry data messages from Kafka topics and routes them to appropriate storage handlers based on data type (sessions, errors, logs, traces)
backend/kafka-queue/consumer.go - ClickHouseStore (store) — Handles all database operations for time-series telemetry data, providing optimized queries for session events, logs, traces, and metrics with time-based partitioning
backend/clickhouse/client.go - SessionProcessor (processor) — Processes raw session events to build session timelines, handles session replay data compression, and manages session state transitions
backend/store/sessions.go - ErrorGrouper (processor) — Groups similar errors together based on stack trace fingerprints and error messages to reduce noise and enable effective error management
backend/errorgroups/grouper.go - GraphQLResolver (adapter) — Exposes telemetry data through GraphQL API for frontend consumption, handling authentication, rate limiting, and query optimization
backend/public-graph/resolver.go - OTLPReceiver (gateway) — Receives OpenTelemetry protocol data from SDKs and applications, validates the data format, and forwards it to Kafka for processing
backend/otel/otel.go - HighlightProvider (adapter) — React context provider that manages authentication state, project selection, and global application settings for the monitoring dashboard
frontend/src/components/Header/HighlightProvider.tsx - SessionReplayPlayer (processor) — Reconstructs and plays back user sessions by processing session events and DOM mutations to create a visual replay experience
frontend/src/pages/Player/PlayerHook.ts
Package Structure
Go-based data ingestion service that processes OpenTelemetry data through Kafka queues and stores it in ClickHouse for analysis
React-based monitoring dashboard that displays session replays, error tracking, logs, and performance analytics
Next.js marketing website and documentation site for the Highlight.io platform
React SDK for capturing user sessions, errors, and performance data in web applications
Node.js SDK for backend error monitoring, distributed tracing, and structured logging
Go SDK providing OpenTelemetry instrumentation for HTTP frameworks and error tracking
Python SDK with FastAPI, Django, and Flask integrations for backend monitoring and tracing
Shared React component library used across Highlight.io frontend applications
Integration test applications across multiple frameworks validating SDK functionality and data pipeline
Explore the interactive analysis
See the full architecture map, data flow, and code patterns visualization.
Analyze on CodeSeaRelated Fullstack Repositories
Frequently Asked Questions
What is highlight used for?
Captures user sessions, application errors, and performance data for full-stack monitoring highlight/highlight is a 8-component fullstack written in TypeScript. Data flows through 6 distinct pipeline stages. The codebase contains 2975 files.
How is highlight architected?
highlight is organized into 4 architecture layers: Client SDKs, Backend Services, Frontend Applications, E2E Testing. Data flows through 6 distinct pipeline stages. This layered structure keeps concerns separated and modules independent.
How does data flow through highlight?
Data moves through 6 stages: Instrument client applications → Transmit telemetry data → Receive and validate data → Process telemetry streams → Store in time-series database → .... Client applications instrumented with Highlight SDKs capture user interactions, errors, and performance data, sending it via OpenTelemetry to the Go backend. The backend receives data through OTLP endpoints, queues it in Kafka for processing, then stores structured data in ClickHouse. Frontend applications query this data through GraphQL to display monitoring dashboards, session replays, and analytics. This pipeline design reflects a complex multi-stage processing system.
What technologies does highlight use?
The core stack includes Go (Backend service runtime for high-performance data ingestion and processing with strong concurrency support), React (Frontend framework for building the monitoring dashboard and session replay interfaces with real-time data visualization), ClickHouse (Columnar database optimized for analytical queries on time-series telemetry data with compression and fast aggregations), Kafka (Message queue system for reliable ingestion and processing of high-volume telemetry data streams with partitioning), OpenTelemetry (Standardized telemetry collection across all SDKs and services for distributed tracing, metrics, and structured logging), GraphQL (API layer for frontend-backend communication with efficient querying and real-time subscriptions for monitoring data), and 2 more. A focused set of dependencies that keeps the build manageable.
What system dynamics does highlight have?
highlight exhibits 4 data pools (Kafka Message Queues, ClickHouse Analytics Database), 3 feedback loops, 4 control points, 3 delays. The feedback loops handle auto-scale and scheduled-job. These runtime behaviors shape how the system responds to load, failures, and configuration changes.
What design patterns does highlight use?
4 design patterns detected: OpenTelemetry Integration, Event Sourcing for Sessions, Multi-Language SDK Architecture, Time-Series Data Partitioning.
Analyzed on April 20, 2026 by CodeSea. Written by Karolina Sarna.