lightdash/lightdash
Self-serve BI to 10x your data team ⚡️
13 hidden assumptions · 6-stage pipeline · 10 components
Like any codebase, this fullstack makes assumptions it never checks — most are routine. The ones worth your attention are below.
Self-service BI platform that transforms dbt models into interactive dashboards and analytics
Data enters Lightdash through dbt model compilation (CLI deployment) or live queries (web interface). The system parses dbt YAML files to extract metric definitions, compiles custom formulas into warehouse-specific SQL, executes queries against configured data warehouses, and returns results as interactive charts or raw data. Authentication flows through passport strategies, while authorization uses CASL ability builders to control access to projects and resources.
Under the hood, the system uses 4 feedback loops, 4 data pools, 4 control points to manage its runtime behavior.
A 10-component fullstack. 3406 files analyzed. Data flows through 6 distinct pipeline stages.
Hidden Assumptions
Most of what this code assumes is routine. These 3 are the ones most likely to cause trouble here. The rest are minor; they're under "Show everything".
If a server returns a malformed Set-Cookie header or cookie values with embedded semicolons/equals, the cookie parsing silently fails or stores corrupted values, breaking session management
All API tests fail with connection errors if the target service is down or the URL is wrong, with no retry logic or health checks
If session data is corrupted or the user is deleted between login and subsequent requests, authentication silently fails or returns undefined user context
Show everything (10 more)
The maximum_bytes_billed setting in dbt profiles is sufficient to prevent runaway BigQuery costs, with no runtime validation that query costs stay within this limit
If this fails: Users can accidentally trigger expensive BigQuery queries that exceed their intended budget if the dbt profile setting is misconfigured or missing
packages/cli/src/dbt/targets/Bigquery/index.ts:maximum_bytes_billed
Project profiles array contains all current project memberships for the user in consistent order, with no duplicate projectUuid entries that could cause permission conflicts
If this fails: If projectProfiles has duplicates or stale data, the last processed role for each project wins, potentially granting wrong permissions or access to deleted projects
packages/common/src/authorization/index.ts:getUserAbilityBuilder
The CompileOptions.dialect field always maps to exactly one of the hardcoded warehouse types (postgres, bigquery, snowflake, duckdb), with no support for dialect versioning or custom warehouse types
If this fails: Adding new warehouse support or handling dialect variations requires code changes; unknown dialects throw runtime errors instead of falling back gracefully
packages/formula/src/codegen/index.ts:createGenerator
All required environment variables for each warehouse type (like FORMULA_TEST_BQ_PROJECT, FORMULA_TEST_SF_ACCOUNT) are set when those warehouses are included in test runs
If this fails: Tests fail with cryptic connection errors if warehouse-specific environment variables are missing, with no clear indication of which variables need to be set
packages/formula-tests/config.ts:getWarehouseConfig
The React Query cache invalidation for 'projectCompileLogs' happens atomically and the fresh data will be immediately available for re-rendering
If this fails: If the cache invalidation fails or the backend is slow to respond, users see stale compilation data even after clicking refresh, with success toast shown prematurely
packages/frontend/src/components/CompilationHistory/index.tsx:handleRefresh
The mapCronExpressionToFrequency function can always parse any cron expression string passed as the 'value' prop, even malformed or custom expressions
If this fails: Invalid cron expressions cause the frequency selector to show incorrect state or crash the component, breaking the scheduling UI
packages/frontend/src/components/CronInput/index.tsx:useEffect
The fetch() call will complete within reasonable time limits and network errors are properly handled by the underlying fetch implementation
If this fails: Long-running queries or network issues cause tests to hang indefinitely since there's no explicit timeout configuration
packages/api-tests/helpers/api-client.ts:request
All Lightdash API responses follow the exact shape { status: string, results: T } but only enforces this through TypeScript typing, not runtime validation
If this fails: If the backend returns differently structured responses, TypeScript compilation succeeds but runtime access to response.body.results fails with undefined errors
packages/api-tests/helpers/api-client.ts:Body<T>
The tier system (fast/tier1/tier2/all) represents increasing complexity and execution time, with DuckDB always being fastest and BigQuery/Snowflake being slowest
If this fails: If warehouse performance characteristics change or new warehouses are added with different performance profiles, the tier classifications become misleading
packages/formula-tests/config.ts:TIER_WAREHOUSES
The customRoleScopes record contains scopes for every roleUuid referenced in projectProfiles when customRolesEnabled is true, with no missing or orphaned role definitions
If this fails: Missing custom role definitions log console errors and skip permission grants, potentially leaving users with no project access despite having role assignments
packages/common/src/authorization/index.ts:customRoleScopes
Open the standalone hidden-assumptions report for lightdash →
How Data Flows Through the System
Data enters Lightdash through dbt model compilation (CLI deployment) or live queries (web interface). The system parses dbt YAML files to extract metric definitions, compiles custom formulas into warehouse-specific SQL, executes queries against configured data warehouses, and returns results as interactive charts or raw data. Authentication flows through passport strategies, while authorization uses CASL ability builders to control access to projects and resources.
- dbt Model Ingestion — CLI reads dbt project files (models/*.sql, schema.yml) and sends them to the backend's compilation service, which parses YAML metadata to extract dimension and metric definitions [dbt project files → Parsed model definitions]
- Formula Compilation — Formula parser converts metric expressions into ASTs, then warehouse-specific SQL generators (PostgresSqlGenerator, BigQuerySqlGenerator) produce executable SQL tailored to the target warehouse dialect [Formula strings → Warehouse SQL]
- User Authentication — Passport strategies validate login credentials (password, Google OAuth, SAML), create user sessions in PostgreSQL, and construct LightdashUser objects with role information [Login credentials → LightdashUser]
- Authorization Check — getUserAbilityBuilder combines the user's organization role with project-specific permissions to create CASL ability objects that control access to charts, dashboards, and data sources [LightdashUser → MemberAbility]
- Query Execution — Backend receives query requests via REST API, validates permissions, executes compiled SQL against warehouse adapters (BigQuery, Snowflake, Postgres), and returns structured results [QueryDefinition → QueryResult]
- Chart Rendering — Frontend consumes QueryResult data and renders it into interactive charts using visualization libraries, with state management handled by React Query for caching and updates [QueryResult → Chart components]
Data Models
The data structures that flow between stages — the contracts that hold the system together.
packages/api-tests/helpers/api-client.tsgeneric wrapper with ok: boolean, status: number, body: T — standardizes all API communication
Created by ApiClient.request(), consumed by all frontend API calls and CLI commands
packages/common/src/types/user.tsuser profile with role: string, organizationUuid: string, userUuid: string, plus authentication state
Populated during login, attached to requests via passport middleware, used for permission checks
packages/formula-tests/types.tstest definition with id: string, formula: string, columns: Record<string, string>, expectedRows: Record<string, any>[], warehouses: WarehouseType[]
Loaded from test files, executed against multiple warehouses, results compared for validation
packages/cli/src/dbt/targets/Bigquery/index.tsdbt target config with project: string, dataset: string, schema: string, plus optional auth and performance settings
Parsed from dbt profiles.yml, validated against JSON schema, converted to warehouse credentials
packages/iframe-test-app/src/types.tscross-frame message with type: string, payload: Record<string, unknown>, timestamp: number
Generated by embedded Lightdash instances, sent via postMessage, logged for debugging iframe integrations
packages/query-sdk/src/types.tsquery specification with columns: Column[], filters: Filter[], sorts: Sort[] — describes what data to fetch
Built by query SDK, sent to backend, compiled to warehouse SQL, executed to return QueryResult
System Behavior
How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.
Data Pools
Encrypted user session data managed by express-session — stores authentication state and user context across requests
Maintains active connections to configured data warehouses (BigQuery, Snowflake, etc.) with connection pooling for performance
Caches compiled SQL and dbt model metadata to avoid recompilation on repeated queries
Frontend query result cache managed by React Query — stores API responses and manages invalidation strategies
Feedback Loops
- API Client Cookie Management (cache-invalidation, reinforcing) — Trigger: ApiClient receives Set-Cookie headers from responses. Action: Updates internal cookie map and includes cookies in subsequent requests. Exit: Session expires or user logs out.
- Formula Test Validation (convergence, balancing) — Trigger: TestCase execution across multiple warehouses. Action: Compares actual query results against expected results, adjusting test classifications. Exit: All warehouses pass or test is marked as failing.
- dbt Compilation Refresh (polling, balancing) — Trigger: User clicks refresh in CompilationHistory component. Action: Invalidates React Query cache and refetches compilation logs from backend. Exit: Fresh data is loaded and displayed.
- Cron Expression Sync (self-correction, balancing) — Trigger: CronInternalInputs detects mismatch between frequency selection and cron value. Action: Updates frequency state to match current cron expression unless in CUSTOM mode. Exit: UI state aligns with cron value.
Delays
- Warehouse Query Execution (async-processing, ~Variable (seconds to minutes)) — Users wait for query results, UI shows loading states while warehouse processes SQL
- dbt Compilation (compilation, ~Variable (10s to several minutes)) — CLI deployment and UI refresh operations block until dbt parsing and model extraction completes
- Session Validation (cache-ttl, ~Per request) — Each request includes database lookup to validate session state and deserialize user
Control Points
- Warehouse Connection Config (env-var) — Controls: Which data warehouses are available and how to connect to them (credentials, endpoints, auth methods). Default: Environment-dependent
- Formula Test Tier (runtime-toggle) — Controls: Which warehouses are included in test runs (fast=DuckDB only, tier1=DuckDB+Postgres, tier2=all). Default: Configurable via TIER_WAREHOUSES
- Authentication Strategy (feature-flag) — Controls: Enables/disables OAuth providers (Google, SAML) vs local password authentication. Default: Multi-strategy enabled
- API Rate Limits (threshold) — Controls: Request throttling and payload size limits to prevent abuse. Default: LIGHTDASH_MAX_PAYLOAD=5mb
Technology Stack
Primary language across all packages providing type safety for complex BI data models and API contracts
Frontend framework for building the interactive dashboard and chart creation interface
Backend web server handling API routes, authentication, and dbt compilation orchestration
Authentication middleware supporting multiple strategies (local, Google OAuth, SAML) for user login
Authorization framework that builds permission abilities based on user roles and project membership
Primary database for storing user sessions, project metadata, and application state
Parser generator for compiling metric formula strings into abstract syntax trees
React component library providing UI components for forms, tables, and dashboard elements
End-to-end testing framework for validating complete user workflows across the web application
Package manager for monorepo workspace management and dependency resolution
Build orchestration tool managing compilation, testing, and deployment across the 14 packages
Containerization for deployment with services for database, MinIO object storage, and the main application
Key Components
- ApiClient (adapter) — HTTP client with cookie-based session management for testing API endpoints — handles authentication flow and request/response parsing
packages/api-tests/helpers/api-client.ts - AuthenticationController (orchestrator) — Manages user login flow using passport strategies (local password, Google OAuth, SAML) — coordinates session creation and user profile retrieval
packages/backend/src/controllers/authentication/index.ts - BigqueryTarget converter (transformer) — Parses dbt BigQuery profile configuration and converts it to Lightdash warehouse credentials — handles OAuth and service account authentication modes
packages/cli/src/dbt/targets/Bigquery/index.ts - getUserAbilityBuilder (factory) — Constructs CASL ability objects that define what actions a user can perform on specific resources — combines organization role with project-specific permissions
packages/common/src/authorization/index.ts - Formula Parser (processor) — Parses metric formula strings into abstract syntax trees using PEG.js grammar — validates syntax and prepares for SQL compilation
packages/formula/src/compiler/index.ts - SQL Generator Factory (factory) — Creates warehouse-specific SQL generators (PostgresSqlGenerator, BigQuerySqlGenerator, etc.) based on dialect configuration — enables cross-warehouse formula compilation
packages/formula/src/codegen/index.ts - Test Runner (orchestrator) — Coordinates formula testing across multiple data warehouses with tiered execution (fast/tier1/tier2) — manages warehouse connections and test result aggregation
packages/formula-tests/config.ts - CompilationHistory (monitor) — React component displaying dbt compilation history with refresh capability — shows deployment logs and compilation status for debugging
packages/frontend/src/components/CompilationHistory/index.tsx - CronInternalInputs (processor) — Converts between human-readable scheduling options (daily, weekly, etc.) and cron expressions — provides UI for configuring automated tasks
packages/frontend/src/components/CronInput/index.tsx - LightdashClient (gateway) — Primary SDK interface providing query building and data fetching capabilities — abstracts API communication for external developers
packages/query-sdk/src/index.ts
Package Structure
Integration test suite for the Lightdash API with authentication helpers and test clients.
Core Node.js API server handling authentication, dbt compilation, warehouse connections, and data queries.
Command-line tool for deploying dbt projects to Lightdash and managing warehouse connections.
Shared TypeScript types, validation schemas, authorization logic, and utilities used across all packages.
End-to-end test suite using Cypress for integration testing across the full application.
Expression parser and SQL compiler that translates metric formulas into warehouse-specific SQL.
Test framework for validating formula compilation across multiple data warehouses.
React-based web application providing the main Lightdash user interface for creating dashboards and exploring data.
Test application for validating iframe embedding functionality and cross-frame communication.
Client SDK providing programmatic access to Lightdash's query builder and data fetching capabilities.
Adapters and connection handlers for various data warehouses (BigQuery, Snowflake, Postgres, etc.).
Explore the interactive analysis
See the full architecture map, data flow, and code patterns visualization.
Analyze on CodeSeaRelated Fullstack Repositories
Frequently Asked Questions
What is lightdash used for?
Self-service BI platform that transforms dbt models into interactive dashboards and analytics lightdash/lightdash is a 10-component fullstack written in TypeScript. Data flows through 6 distinct pipeline stages. The codebase contains 3406 files.
How is lightdash architected?
lightdash is organized into 4 architecture layers: Client Applications, API Server, Query Engine, Shared Foundation. Data flows through 6 distinct pipeline stages. This layered structure keeps concerns separated and modules independent.
How does data flow through lightdash?
Data moves through 6 stages: dbt Model Ingestion → Formula Compilation → User Authentication → Authorization Check → Query Execution → .... Data enters Lightdash through dbt model compilation (CLI deployment) or live queries (web interface). The system parses dbt YAML files to extract metric definitions, compiles custom formulas into warehouse-specific SQL, executes queries against configured data warehouses, and returns results as interactive charts or raw data. Authentication flows through passport strategies, while authorization uses CASL ability builders to control access to projects and resources. This pipeline design reflects a complex multi-stage processing system.
What technologies does lightdash use?
The core stack includes TypeScript (Primary language across all packages providing type safety for complex BI data models and API contracts), React (Frontend framework for building the interactive dashboard and chart creation interface), Express (Backend web server handling API routes, authentication, and dbt compilation orchestration), Passport (Authentication middleware supporting multiple strategies (local, Google OAuth, SAML) for user login), CASL (Authorization framework that builds permission abilities based on user roles and project membership), PostgreSQL (Primary database for storing user sessions, project metadata, and application state), and 6 more. This broad technology surface reflects a mature project with many integration points.
What system dynamics does lightdash have?
lightdash exhibits 4 data pools (Session Store, Warehouse Connection Pool), 4 feedback loops, 4 control points, 3 delays. The feedback loops handle cache-invalidation and convergence. These runtime behaviors shape how the system responds to load, failures, and configuration changes.
What design patterns does lightdash use?
5 design patterns detected: Monorepo Package Architecture, Multi-Warehouse Abstraction, API Client with Session Management, Tiered Testing Strategy, Formula AST Compilation Pipeline.
Analyzed on April 20, 2026 by CodeSea. Written by Karolina Sarna.