great-expectations/great_expectations

Always know what to expect from your data.

How Data Flows Through the System

Users define expectations through DataContext, which manages datasources that load data into Batches. Validators execute expectations against batches using ExecutionEngines that compute metrics. Results flow to Renderers for documentation and Stores for persistence. Checkpoints orchestrate this pipeline for production workflows.

1. Initialize DataContext — AbstractDataContext loads configuration from stores, initializes datasources, and sets up validation infrastructure [DataContextConfig → DataContext]
2. Define Expectations — Users create ExpectationConfiguration objects defining data quality rules via expect_* methods on Validator instances
3. Load Data Batches — Datasource queries data sources and creates Batch objects containing data and metadata using BatchDefinition [BatchRequest → Batch]
4. Execute Validations — Validator.validate() converts expectations to MetricConfiguration, ExecutionEngine computes metrics, results compared to expectation thresholds [ExpectationConfiguration → ValidationResult]
5. Aggregate Results — Checkpoint collects ValidationResults from multiple batches and expectations into CheckpointResult [ValidationResult → CheckpointResult]
6. Render Documentation — Renderer transforms ValidationResults into HTML reports, data docs, and other human-readable formats [ValidationResult → RenderedContent]
7. Persist Results — Store implementations save ValidationResults, ExpectationSuites, and documentation to filesystem, databases, or cloud storage [Serializable objects]

Data Models

The data structures that flow between stages — the contracts that hold the system together.

Hidden Assumptions

Things this code relies on but never validates. These are the things that cause silent failures when the system changes.

System Behavior

How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Expectation Refinement (self-correction, balancing) — Trigger: Validation failures or unexpected results. Action: Users adjust expectation parameters based on validation outcomes. Exit: Expectations match data reality.
• Profiler Iteration (convergence, balancing) — Trigger: Data profiling to automatically generate expectations. Action: ProfilerRunner iteratively refines expectation parameters based on data samples. Exit: Statistical convergence or iteration limit reached.
• Checkpoint Retry (retry, balancing) — Trigger: Transient failures in data access or computation. Action: Checkpoint retries failed validations with exponential backoff. Exit: Success or maximum retry count reached.

Delays

• Lazy Metric Computation (async-processing, ~Variable based on data size) — Metrics computed only when needed by expectations, improving performance for large datasets
• Store Synchronization (eventual-consistency, ~Depends on storage backend) — Stores may have eventual consistency, affecting immediate availability of saved artifacts
• Data Source Connection (warmup, ~1-30 seconds) — Initial connection establishment to databases or cloud storage before data access

Control Points

• progress_bars (feature-flag) — Controls: Whether to display progress indicators during long-running operations. Default: True
• result_format (runtime-toggle) — Controls: Detail level of validation results (BOOLEAN_ONLY, BASIC, COMPLETE, SUMMARY). Default: SUMMARY
• catch_exceptions (runtime-toggle) — Controls: Whether to catch and return exceptions or let them propagate. Default: True
• evaluation_parameters (runtime-toggle) — Controls: Dynamic parameter substitution in expectation configurations. Default: null

Technology Stack

Key Components

• AbstractDataContext (orchestrator) — Central coordinator that manages datasources, expectations, checkpoints, and validation workflows great_expectations/data_context/data_context/abstract_data_context.py
• Validator (executor) — Executes expectations against data batches using configured execution engines great_expectations/validator/validator.py
• ExecutionEngine (adapter) — Abstracts computation across pandas, Spark, SQL databases for running metrics and validations great_expectations/execution_engine/execution_engine.py
• Expectation (validator) — Defines data quality validation logic and converts to executable metric configurations great_expectations/expectations/expectation.py
• Checkpoint (orchestrator) — Orchestrates batch validation workflows across multiple data assets with consistent configuration great_expectations/checkpoint/checkpoint.py
• Datasource (adapter) — Provides unified interface to different data sources and manages data asset discovery great_expectations/datasource/fluent/datasource.py
• Store (store) — Persists expectations, validation results, and other GX artifacts to various backends great_expectations/data_context/store/store.py
• Renderer (transformer) — Converts validation results into human-readable documentation and reports great_expectations/render/renderer/renderer.py

Related Library Repositories

Frequently Asked Questions

Analyzed on April 19, 2026 by CodeSea. Written by Karolina Sarna.