kedro-org/kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

Structural Verdict

How Data Flows Through the System

Data flows through pipelines where nodes transform datasets loaded from the catalog, with configuration controlling both data sources and processing parameters

1. Load Configuration — ConfigLoader reads YAML/JSON configuration files with environment-specific overrides (config: catalog.*, parameters.*)
2. Initialize Catalog — DataCatalog instantiates dataset objects based on catalog configuration (config: catalog.*)
3. Resolve Pipeline — Pipeline dependencies are resolved and nodes are ordered for execution
4. Execute Nodes — Runner executes nodes with automatic dataset loading/saving and parameter injection (config: parameters.*)
5. Persist Results — Node outputs are saved to configured datasets in the catalog (config: catalog.*)

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Pipeline Validation (recursive, balancing) — Trigger: Pipeline creation or modification. Action: Validates node dependencies and detects circular references. Exit: All dependencies resolved or error raised.
• Configuration Reload (polling, balancing) — Trigger: Development mode or configuration change detection. Action: Reloads configuration files and reinitializes context. Exit: Configuration stabilizes.

Delays & Async Processing

• Dataset Lazy Loading (async-processing, ~varies by dataset size) — Datasets only loaded when accessed by nodes, improving memory efficiency
• Plugin Discovery (async-processing, ~startup time) — Plugins discovered at application startup, affecting initial load time
• Template Generation (async-processing, ~varies by template complexity) — Project scaffolding occurs during kedro new command execution

Control Points

• Environment Configuration (env-var) — Controls: Which configuration environment to load (base, local, etc.). Default: KEDRO_ENV
• Plugin Discovery Toggle (env-var) — Controls: Whether to automatically discover and load plugins
• Runner Selection (runtime-toggle) — Controls: Which pipeline runner to use (sequential, parallel, etc.). Default: SequentialRunner
• Logging Level (env-var) — Controls: Verbosity of framework logging output

Technology Stack

Key Components

• KedroContext (class) — Central project context managing configuration, catalogs, and pipelines kedro/framework/context/context.py
• Pipeline (class) — Directed acyclic graph representing data processing workflows kedro/pipeline/pipeline.py
• Node (class) — Individual computational unit that processes inputs to produce outputs kedro/pipeline/node.py
• DataCatalog (class) — Registry managing datasets and their load/save operations kedro/io/data_catalog.py
• AbstractDataset (class) — Base interface for all dataset implementations with load/save/exists methods kedro/io/core.py
• AbstractRunner (class) — Base class for pipeline execution strategies kedro/runner/runner.py
• ConfigLoader (class) — Loads and manages configuration from YAML/JSON files with environment support kedro/config/config.py
• PluginManager (class) — Discovers and manages Kedro plugins using pluggy hooks kedro/framework/plugins/manager.py
• find_run_command (function) — Discovers and returns the main run command for a Kedro project kedro/framework/cli/utils.py
• register_pipelines (function) — Project-level function to register all available pipelines kedro/framework/project/__init__.py

Configuration

asv.conf.json (json)

• version (number, unknown) — default: 1
• project (string, unknown) — default: Kedro
• project_url (string, unknown) — default: https://kedro.org/
• repo (string, unknown) — default: .
• install_command (array, unknown) — default: pip install -e . kedro-datasets[pandas-csvdataset]
• branches (array, unknown) — default: main
• environment_name (string, unknown) — default: kedro
• environment_type (string, unknown) — default: virtualenv
• +5 more parameters

kedro/pipeline/preview_contract.py (python-dataclass)

• content (str, unknown)

kedro/pipeline/preview_contract.py (python-dataclass)

• content (str, unknown)

kedro/pipeline/preview_contract.py (python-dataclass)

• renderer_key (str, unknown)
• content (JSONObject, unknown)

Science Pipeline

1. Configuration Loading — YAML/JSON parsing with OmegaConf then merge with environment overrides kedro/config/config.py
2. Dataset Resolution — Instantiate dataset objects from catalog configuration using factory pattern kedro/io/data_catalog.py
3. Pipeline Graph Construction — Build DAG from nodes and validate dependencies kedro/pipeline/pipeline.py
4. Node Execution — Load inputs from catalog, execute node function, save outputs back to catalog kedro/runner/runner.py

Assumptions & Constraints

• [warning] Assumes node functions accept parameters matching their signature but no runtime validation of parameter types (dependency)
• [info] Assumes datasets can be loaded/saved consistently but format validation left to implementations (format)
• [warning] Assumes all node dependencies can be resolved but circular dependency detection only occurs at pipeline creation (dependency)

Related Ml Inference Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.