zenml-io/zenml

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

Structural Verdict

How Data Flows Through the System

ML data flows through ingestion, preprocessing, training, evaluation, and deployment stages orchestrated by ZenML pipelines

1. Data Ingestion — Load raw datasets from various sources including LakeFS, S3, or local files
2. Preprocessing — Clean, transform, and prepare data using sklearn pipelines or custom transformers
3. Model Training — Train ML models using frameworks like HuggingFace Transformers with LoRA fine-tuning
4. Evaluation — Compute metrics and validate model performance using test datasets
5. Deployment — Deploy trained models as web services using FastAPI runners with dashboard interfaces

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Pipeline Retry Logic (retry, balancing) — Trigger: Step failure or infrastructure error. Action: Retry failed pipeline steps with exponential backoff. Exit: Max retries reached or step succeeds.
• Model Performance Monitoring (training-loop, balancing) — Trigger: Scheduled evaluation runs. Action: Compare model metrics against thresholds. Exit: Performance degradation detected triggering retraining.
• Agent Outer Loop (recursive, reinforcing) — Trigger: Agent receives task. Action: Plan, execute tools, evaluate results, and iterate. Exit: Task completed or max iterations reached.

Delays & Async Processing

• Model Training (async-processing, ~minutes to hours) — Pipeline blocks until training completes with artifacts stored
• Container Build (async-processing, ~1-10 minutes) — Pipeline execution waits for custom environment containerization
• LakeFS Commit (eventual-consistency, ~seconds) — Data changes become visible after commit operation completes

Control Points

• Pipeline Configuration (env-var) — Controls: Model hyperparameters, data paths, and infrastructure settings
• Integration Registry (runtime-toggle) — Controls: Which ML framework integrations are active
• Deployment Settings (feature-flag) — Controls: Endpoint configuration, middleware, and security settings for deployed models

Technology Stack

Key Components

• Client (class) — Main entry point for interacting with ZenML services, managing pipelines and artifacts src/zenml/client.py
• BaseDeploymentAppRunner (class) — Abstract base for creating web applications that serve ML models and pipelines src/zenml/deployers/server/app.py
• FastAPIDeploymentAppRunner (class) — FastAPI implementation of deployment runner with CORS, static files, and templating src/zenml/deployers/server/fastapi/app.py
• EndpointAdapter (class) — Converts framework-agnostic endpoint specifications to framework-specific endpoints src/zenml/deployers/server/adapters.py
• MiddlewareAdapter (class) — Adapts middleware specifications for different web frameworks src/zenml/deployers/server/adapters.py
• LakeFSRef (class) — Lightweight JSON-serializable pointer to datasets stored in LakeFS for data versioning examples/lakefs_data_versioning/utils/lakefs_ref.py
• lakefs_utils (module) — Helper functions for LakeFS interaction via SDK and S3-compatible gateway examples/lakefs_data_versioning/utils/lakefs_utils.py
• compute_metrics (function) — Computes accuracy metrics for NLP model evaluation using HuggingFace datasets examples/e2e_nlp/utils/misc.py
• sentiment_analysis (cli-command) — Launches Gradio interface for text classification sentiment analysis examples/e2e_nlp/gradio/app.py
• load_base_model (function) — Loads and configures base language models for fine-tuning with quantization options examples/llm_finetuning/utils/loaders.py

Sub-Modules

Configuration

pull_request_cloudbuild.yaml (yaml)

• steps (array, unknown) — default: [object Object],[object Object],[object Object],[object Object],[object Object]
• timeout (string, unknown) — default: 3600s
• availableSecrets.secretManager (array, unknown) — default: [object Object],[object Object]

release-cloudbuild-nightly.yaml (yaml)

• steps (array, unknown) — default: [object Object],[object Object],[object Object],[object Object],[object Object]
• timeout (string, unknown) — default: 3600s
• availableSecrets.secretManager (array, unknown) — default: [object Object],[object Object]

release-cloudbuild-preparation.yaml (yaml)

• steps (array, unknown) — default: [object Object],[object Object],[object Object]
• timeout (string, unknown) — default: 3600s
• availableSecrets.secretManager (array, unknown) — default: [object Object],[object Object]

release-cloudbuild.yaml (yaml)

• steps (array, unknown) — default: [object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
• timeout (string, unknown) — default: 3600s
• availableSecrets.secretManager (array, unknown) — default: [object Object],[object Object]

Science Pipeline

1. Load NLP dataset — datasets.load_dataset then tokenization examples/e2e_nlp/steps/
2. Model training — HuggingFace Trainer with LoRA fine-tuning [(batch_size, sequence_length) → (batch_size, num_classes)] examples/llm_finetuning/utils/loaders.py
3. Metric computation — argmax on logits then accuracy calculation [(batch_size, num_classes) → scalar] examples/e2e_nlp/utils/misc.py
4. LakeFS data read — S3 gateway read then pandas parquet parse examples/lakefs_data_versioning/utils/lakefs_utils.py

Assumptions & Constraints

• [warning] Assumes logits has shape (batch_size, num_classes) and labels has shape (batch_size,) but no assertion enforces this (shape)
• [critical] Assumes GPU availability when use_accelerate=True but no device validation occurs (device)
• [info] Assumes parquet format but no file type validation before pandas read_parquet call (format)

Related Ml Training Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.