orchest/orchest
Build data pipelines, the easy way 🛠️
5-stage pipeline · 10 components
Visual data pipeline orchestration platform with Kubernetes-native deployment
Users interact with the web UI to create pipelines, which are stored and executed by the API service, while the CLI manages cluster operations through Kubernetes CRDs monitored by the controller.
Under the hood, the system uses 3 feedback loops, 3 data pools, 4 control points to manage its runtime behavior.
A 10-component data pipeline. 980 files analyzed. Data flows through 5 distinct pipeline stages.
How Data Flows Through the System
Users interact with the web UI to create pipelines, which are stored and executed by the API service, while the CLI manages cluster operations through Kubernetes CRDs monitored by the controller.
- User Interface — React web app provides visual pipeline editor and project management
- API Processing — Orchest API service handles pipeline storage, job scheduling, and execution coordination
- Cluster Management — CLI writes to Kubernetes CRDs which trigger controller actions for resource provisioning
- Image Management — Node agents pull and manage Docker images for pipeline execution environments (config: image-puller-interval, image-puller-policy, image-puller-retries)
- Authentication — Auth server validates users and manages access control for all services
System Behavior
How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.
Data Pools
Cluster state and configuration managed through custom resources
Pipeline definitions, job histories, and execution state
Container images for pipeline execution environments
Feedback Loops
- Image Pull Retry (retry, balancing) — Trigger: Failed image pull operation. Action: Retry pull operation with backoff. Exit: Success or max retries reached.
- Kubernetes Controller Reconciliation (polling, balancing) — Trigger: CRD watch events. Action: Reconcile actual vs desired cluster state. Exit: State converged or error.
- Linear Issue Sync (polling, balancing) — Trigger: GitHub issue events. Action: Query Linear API and sync issue state. Exit: Successful sync or error.
Delays
- Image Pull Interval (scheduled-job, ~5 seconds (configurable)) — Controls frequency of image pull operations
- Async Pipeline Execution (async-processing, ~variable) — Pipeline steps execute asynchronously with result polling
- Build Environment Setup (async-processing, ~variable) — Environment builds complete before pipeline can execute
Control Points
- Image Puller Policy (env-var) — Controls: When to pull images (IfNotPresent, Always, etc.). Default: IfNotPresent
- Image Puller Threadiness (env-var) — Controls: Number of concurrent image pull operations. Default: 4
- Image Puller Retries (env-var) — Controls: Maximum retry attempts for failed pulls. Default: 3
- Orchest API Host (env-var) — Controls: API endpoint for node agent communication. Default: http://orchest-api:80
Technology Stack
Frontend framework for web interfaces
Python web framework for API services
Container orchestration and cluster management
Language for Kubernetes controller implementation
Type-safe JavaScript for frontend development
End-to-end testing framework
Container runtime for pipeline execution
Asynchronous programming for Python services
Python CLI framework
React component library for UI
Key Components
- OrchestCLI (cli-command) — Main CLI interface for managing Orchest clusters with commands for install, update, adduser, etc.
orchest-cli/orchestcli/cli.py - ImagePuller (service) — Manages Docker image pulling operations on Kubernetes nodes with async processing
services/node-agent/app/main.py - FetchError (class) — Custom error class for handling HTTP request failures with status codes and response bodies
lib/javascript/utils/src/fetcher.ts - ReactApp (component) — Main React application component for the pipeline visual editor and project management UI
services/orchest-webserver/client/src/index.tsx - AuthApp (component) — Authentication interface for user login and management operations
services/auth-server/client/src/main.tsx - LinearSyncAction (handler) — GitHub Action that synchronizes issues between GitHub and Linear project management tool
.github/custom-actions/github-linear-sync/src/index.js - CypressTestCommon (utility) — Shared constants and test identifiers for Cypress end-to-end tests
cypress/cypress/support/common.ts - DesignSystemComponents (module) — Exports shared React UI components like Box, Flex, Grid, Heading for consistent styling
lib/design-system/package/src/components/index.ts - IconComponents (component) — SVG icon components with forwardRef support for consistent iconography
lib/design-system/package/src/icons/index.tsx - StepExtension (type-def) — TypeScript definitions for allowed pipeline step file extensions (ipynb, py, R, sh, jl, js)
lib/javascript/utils/src/typed.ts
Package Structure
End-to-end testing suite for the Orchest web application using Cypress framework.
Shared React component library providing UI elements like Box, Flex, Grid, icons and typography components.
JavaScript utility library providing HTTP fetcher utilities, type helpers, and validation functions.
Command-line interface for managing Orchest clusters on Kubernetes, handling deployment, updates, and user management.
Core API service handling pipeline execution, job scheduling, project management, and system orchestration.
Main web application providing the visual pipeline editor, project management UI, and user interface.
Kubernetes controller managing Orchest cluster lifecycle, resource provisioning, and deployment orchestration.
Authentication service handling user management, login, and authorization for the Orchest platform.
Background service managing Docker image operations (pulling, pushing, cleanup) on Kubernetes nodes.
Explore the interactive analysis
See the full architecture map, data flow, and code patterns visualization.
Analyze on CodeSeaRelated Data Pipeline Repositories
Frequently Asked Questions
What is orchest used for?
Visual data pipeline orchestration platform with Kubernetes-native deployment orchest/orchest is a 10-component data pipeline written in TypeScript. Data flows through 5 distinct pipeline stages. The codebase contains 980 files.
How is orchest architected?
orchest is organized into 5 architecture layers: CLI Layer, Frontend Layer, API Layer, Infrastructure Layer, and 1 more. Data flows through 5 distinct pipeline stages. This layered structure keeps concerns separated and modules independent.
How does data flow through orchest?
Data moves through 5 stages: User Interface → API Processing → Cluster Management → Image Management → Authentication. Users interact with the web UI to create pipelines, which are stored and executed by the API service, while the CLI manages cluster operations through Kubernetes CRDs monitored by the controller. This pipeline design reflects a complex multi-stage processing system.
What technologies does orchest use?
The core stack includes React (Frontend framework for web interfaces), Flask (Python web framework for API services), Kubernetes (Container orchestration and cluster management), Go (Language for Kubernetes controller implementation), TypeScript (Type-safe JavaScript for frontend development), Cypress (End-to-end testing framework), and 4 more. This broad technology surface reflects a mature project with many integration points.
What system dynamics does orchest have?
orchest exhibits 3 data pools (Kubernetes CRDs, Pipeline Metadata), 3 feedback loops, 4 control points, 3 delays. The feedback loops handle retry and polling. These runtime behaviors shape how the system responds to load, failures, and configuration changes.
What design patterns does orchest use?
6 design patterns detected: Microservices Architecture, Kubernetes Custom Resources, Async Task Processing, Shared Component Library, Error Boundary Pattern, and 1 more.
Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.