sodadata/soda-core

Data Contracts engine for the modern data stack. https://www.soda.io

Structural Verdict

How Data Flows Through the System

YAML contracts are parsed into check definitions, translated to SQL queries via the AST system, executed against target databases through adapter plugins, with results either returned locally or sent to Soda Cloud

1. Load YAML Contract — Parse data contract YAML files into structured check definitions
2. Generate SQL AST — Convert checks into database-agnostic SQL Abstract Syntax Tree
3. Render Database SQL — Transform AST into database-specific SQL using dialect adapters
4. Execute Queries — Run SQL queries against target database via connection adapters
5. Process Results — Aggregate query results and validate against contract expectations
6. Report Outcomes — Send results to Soda Cloud or return locally for pipeline integration

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Contract Verification Loop (polling, balancing) — Trigger: Scheduled or pipeline-triggered execution. Action: Execute all checks in contract and compare results to expectations. Exit: All checks pass or maximum retries reached.
• Plugin Discovery (recursive, reinforcing) — Trigger: System startup. Action: Scan for and load available data source adapter packages. Exit: All packages loaded or import failures.

Delays & Async Processing

• Database Query Execution (async-processing, ~varies by database size and complexity) — Contract verification waits for all SQL queries to complete
• Soda Cloud Upload (async-processing, ~network-dependent) — Results may be available locally before cloud synchronization

Control Points

• Data Source Type (env-var) — Controls: Which database adapter plugin is loaded and used
• Connection Properties (env-var) — Controls: Database connection parameters like host, credentials, timeouts
• Logging Level (runtime-toggle) — Controls: Verbosity of system logging and debugging output

Package Structure

This monorepo contains 14 packages:

Technology Stack

Key Components

• DataSourceImpl (class) — Abstract base class defining interface for all database adapters soda-core/src/soda_core/common/data_source_impl.py
• BigQueryDataSourceImpl (class) — BigQuery-specific implementation handling project namespaces and BigQuery SQL dialect soda-bigquery/src/soda_bigquery/common/data_sources/bigquery_data_source.py
• PostgresDataSourceImpl (class) — PostgreSQL adapter with metadata queries and regex pattern support soda-postgres/src/soda_postgres/common/data_sources/postgres_data_source.py
• SqlDialect (class) — Base class for database-specific SQL generation and type mapping soda-core/src/soda_core/common/sql_dialect.py
• ContractVerificationImpl (class) — Core contract verification engine executing checks against YAML contracts soda-core/src/soda_core/contracts/impl/contract_verification_impl.py
• DataSourceConnection (class) — Abstract connection manager handling database connectivity and query execution soda-core/src/soda_core/common/data_source_connection.py
• SQL_AST (module) — SQL Abstract Syntax Tree classes for building database-agnostic queries soda-core/src/soda_core/common/sql_ast.py
• load_plugins (function) — Discovers and registers all available data source adapter packages soda-core/src/soda_core/plugins.py
• MetadataTablesQuery (class) — Base class for querying database metadata like table and column information soda-core/src/soda_core/common/statements/metadata_tables_query.py
• DuckDBDataSourceImpl (class) — DuckDB adapter supporting both file-based and in-memory database connections soda-duckdb/src/soda_duckdb/common/data_sources/duckdb_data_source.py

Configuration

soda-bigquery/src/soda_bigquery/common/data_sources/bigquery_data_source.py (python-dataclass)

• project_id (str, unknown)
• dataset (str, unknown)
• location (Optional[str], unknown) — default: None

soda-core/src/soda_core/common/metadata_types.py (python-dataclass)

• database (str, unknown)
• schema (str, unknown)

soda-core/src/soda_core/common/metadata_types.py (python-dataclass)

• name (str, unknown)
• character_maximum_length (Optional[int], unknown) — default: None
• numeric_precision (Optional[int], unknown) — default: None
• numeric_scale (Optional[int], unknown) — default: None
• datetime_precision (Optional[int], unknown) — default: None

soda-core/src/soda_core/common/soda_cloud_dto.py (python-pydantic)

• check_attributes (list[CheckAttribute], unknown) — default: Field(..., alias="resourceAttributes")

Science Pipeline

1. Parse Contract YAML — YAML parsing with schema validation [YAML text → ContractVerification objects] soda-core/src/soda_core/contracts/contract_verification.py
2. Build SQL AST — Convert checks to database-agnostic SQL tree [ContractVerification → SQL AST nodes] soda-core/src/soda_core/common/sql_ast.py
3. Render Database SQL — Transform AST using dialect-specific rules [SQL AST → Database-specific SQL strings] soda-core/src/soda_core/common/sql_dialect.py
4. Execute Database Queries — Run SQL against target database [SQL strings → QueryResult objects with rows] soda-core/src/soda_core/common/data_source_connection.py
5. Validate Results — Compare actual vs expected values [QueryResult → Check pass/fail status] soda-core/src/soda_core/contracts/impl/contract_verification_impl.py

Assumptions & Constraints

• [warning] Assumes sampling parameters are positive numbers but no validation enforces reasonable limits (value-range)
• [info] Assumes table names follow database-specific naming conventions but validation varies by adapter (format)
• [info] Assumes column metadata fields like precision/scale are always integers when present (shape)

Related Data Pipeline Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.