kestra-io/kestra

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Structural Verdict

How Data Flows Through the System

Workflows are defined in YAML, stored in repositories, queued for execution, processed by workers, with results indexed and exposed via API/UI

1. Flow Definition — YAML workflow definitions are validated and stored in flow repositories
2. Execution Queue — Flow executions are queued and dispatched to available workers
3. Worker Processing — Workers execute flow tasks and report status back through controllers
4. Result Indexing — Execution results and logs are indexed for searchability and monitoring
5. API/UI Access — Results are exposed through REST API and consumed by web UI

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Worker Health Monitoring (polling, balancing) — Trigger: Worker controller checks worker status periodically. Action: Reschedule failed executions or spawn new workers. Exit: Healthy worker count maintained.
• Flow Validation Loop (retry, balancing) — Trigger: Invalid flow definition submitted. Action: Return validation errors to user for correction. Exit: Valid flow definition accepted.

Delays & Async Processing

• Scheduled Execution (scheduled-job, ~User-defined cron schedules) — Workflows wait for scheduled trigger time
• Queue Processing (queue-drain, ~Variable based on worker availability) — Executions wait in queue until worker capacity available

Control Points

• Worker Thread Count (env-var) — Controls: Number of concurrent worker threads for execution. Default: Math.max(3, Runtime.getRuntime().availableProcessors())
• Scheduler Enable/Disable (runtime-toggle) — Controls: Whether scheduled workflows are processed. Default: true
• Tenant ID Selection (env-var) — Controls: Multi-tenant isolation for enterprise deployments

Technology Stack

Key Components

• Kestra (cli-command) — Main CLI orchestrator that routes to subcommands for flows, servers, plugins, and system operations cli/src/main/java/io/kestra/cli/Kestra.java
• StandAloneRunner (service) — Runs the complete Kestra stack in standalone mode with workers, scheduler, indexer, and controller cli/src/main/java/io/kestra/cli/StandAloneRunner.java
• AbstractApiCommand (class) — Base class for CLI commands that interact with Kestra server API with authentication and HTTP client setup cli/src/main/java/io/kestra/cli/AbstractApiCommand.java
• FlowCommand (cli-command) — Flow management subcommands including test, export, delete, and synchronization operations cli/src/main/java/io/kestra/cli/commands/flows/FlowCommand.java
• FlowTestCommand (cli-command) — Executes flow tests in standalone mode with temporary execution environment and await completion cli/src/main/java/io/kestra/cli/commands/flows/FlowTestCommand.java
• useKvFilter (hook) — Vue composable exports for various filter configurations used across the UI ui/src/components/filter/configurations/index.ts
• flowExamples (config) — Collection of predefined workflow templates for onboarding including ETL, microservices, and automation examples ui/src/components/onboarding/flows/index.ts
• defineKestraHeyConfig (function) — Configuration builder for Kestra's API SDK plugin integration ui/heyapi-sdk-plugin/index.ts

Sub-Modules

Configuration

codecov.yml (yaml)

• component_management.individual_components (array, unknown) — default: [object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
• ignore (array, unknown) — default: ui/**
• flag_management.default_rules.carryforward (boolean, unknown) — default: true
• flag_management.default_rules.statuses (array, unknown) — default: [object Object],[object Object]

docker-compose-ci.yml (yaml)

• services.mysql.image (string, unknown) — default: mysql
• services.mysql.environment.MYSQL_DATABASE (string, unknown) — default: kestra_unit
• services.mysql.environment.MYSQL_USER (string, unknown) — default: kestra
• services.mysql.environment.MYSQL_PASSWORD (string, unknown) — default: k3str4
• services.mysql.environment.MYSQL_ROOT_PASSWORD (string, unknown) — default: p4ssw0rd
• services.mysql.entrypoint (string, unknown) — default: sh -c " echo \"CREATE DATABASE IF NOT EXISTS kestra_unit_webserver;GRANT ALL PRIVILEGES ON kestra_unit_webserver.* TO 'kestra'@'%' WITH GRANT OPTION;\" > /docker-entrypoint-initdb.d/init.sql; /usr/local/bin/docker-entrypoint.sh --log-bin-trust-function-creators=1 --innodb_ft_min_token_size=1 --ft_min_word_len=1 --sort-buffer-size=10485760 "
• services.mysql.ports (array, unknown) — default: 3306:3306
• services.mysql.restart (string, unknown) — default: on-failure
• +7 more parameters

docker-compose-dind.yml (yaml)

• volumes.postgres-data.driver (string, unknown) — default: local
• volumes.kestra-data.driver (string, unknown) — default: local
• volumes.dind-socket.driver (string, unknown) — default: local
• volumes.tmp-data.driver (string, unknown) — default: local
• services.postgres.image (string, unknown) — default: postgres:18
• services.postgres.volumes (array, unknown) — default: postgres-data:/var/lib/postgresql
• services.postgres.environment.POSTGRES_DB (string, unknown) — default: kestra
• services.postgres.environment.POSTGRES_USER (string, unknown) — default: kestra
• +21 more parameters

docker-compose.yml (yaml)

• volumes.postgres-data.driver (string, unknown) — default: local
• volumes.kestra-data.driver (string, unknown) — default: local
• services.postgres.image (string, unknown) — default: postgres:18
• services.postgres.volumes (array, unknown) — default: postgres-data:/var/lib/postgresql
• services.postgres.environment.POSTGRES_DB (string, unknown) — default: kestra
• services.postgres.environment.POSTGRES_USER (string, unknown) — default: kestra
• services.postgres.environment.POSTGRES_PASSWORD (string, unknown) — default: k3str4
• services.postgres.healthcheck.test (array, unknown) — default: CMD-SHELL,pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}
• +12 more parameters

Related Data Pipeline Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.