prefecthq/prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

How Data Flows Through the System

Flow definitions with @flow/@task decorators are submitted as deployments to the server database. The scheduler creates flow runs and places them in work queues. Workers poll queues, retrieve flow runs, and execute them using the flow engine which manages task execution, state transitions, and result persistence. Events are emitted throughout for observability and automation triggers.

1. Flow definition and deployment — User defines flows with @flow decorator and creates deployments via CLI using Flow.deploy() method, storing flow code references and configuration in server database [Flow → Deployment]
2. Schedule-based flow run creation — Scheduler service queries deployments with schedules, creates FlowRun instances using deployment parameters, and queues them in specified work queues [Deployment → FlowRun]
3. Worker polling and run acquisition — Workers poll work queues via PrefectClient.get_runs_in_work_queue(), acquire FlowRun assignments, and prepare execution environment based on infrastructure configuration [WorkQueue → FlowRun]
4. Flow execution and task orchestration — FlowRunner loads flow code, resolves parameters from blocks, executes @flow function which creates TaskRun instances for each @task call, managing parallel execution and dependencies [FlowRun → TaskRun]
5. Task execution with caching and retries — TaskRunner executes individual tasks, checks cache via cache_key lookup, handles retries on failure, and persists results using ResultStore [TaskRun → Task results]
6. State management and persistence — StateManager validates state transitions, updates FlowRun/TaskRun states in database via PrefectClient, and triggers event emission for state changes [State → Event]
7. Event processing and automation — EventEmitter publishes events to automation engine, which evaluates trigger conditions and executes actions like creating new flow runs or sending notifications [Event → Automation actions]

Data Models

The data structures that flow between stages — the contracts that hold the system together.

Hidden Assumptions

Things this code relies on but never validates. These are the things that cause silent failures when the system changes.

System Behavior

How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Task retry loop (retry, balancing) — Trigger: Task failure with retries configured. Action: TaskRunner waits exponential backoff delay then re-executes task with same parameters. Exit: Task succeeds or retry limit reached.
• Worker heartbeat polling (polling, reinforcing) — Trigger: Worker starts and joins work pool. Action: Worker sends heartbeat and polls for new flow runs every polling_interval seconds. Exit: Worker shutdown or pool deletion.
• Scheduler deployment polling (polling, reinforcing) — Trigger: Scheduler service startup. Action: Queries deployments for due schedules and creates flow runs every scheduling_interval. Exit: Service shutdown.
• Automation trigger evaluation (auto-scale, reinforcing) — Trigger: Event matching automation trigger condition. Action: Creates new flow run or performs configured action. Exit: Trigger condition no longer met or automation disabled.
• Flow run state propagation (self-correction, balancing) — Trigger: Flow run state change. Action: Updates child task run states and parent flow context. Exit: All state updates completed.

Delays

• Task retry backoff (rate-limit, ~exponential: retry_delay_seconds * (2 ** attempt)) — Prevents overwhelming failing resources while allowing recovery
• Worker polling interval (scheduled-job, ~PREFECT_WORKER_QUERY_SECONDS (default 10s)) — Controls frequency of work queue polling and resource usage
• Database connection pool (queue-drain, ~Connection acquisition timeout) — Blocks API requests when connection pool exhausted
• Event processing delay (async-processing, ~Event emission to automation trigger evaluation) — Automations react after event processing latency
• Flow run scheduling (scheduled-job, ~Schedule interval or cron expression) — Determines when scheduled flows execute

Control Points

• PREFECT_API_URL (env-var) — Controls: Which Prefect server the client connects to. Default: http://127.0.0.1:4200/api
• PREFECT_WORKER_QUERY_SECONDS (env-var) — Controls: How frequently workers poll for new flow runs. Default: 10
• task_runner (runtime-toggle) — Controls: Execution strategy for tasks (sequential, concurrent, distributed). Default: ConcurrentTaskRunner
• retries (hyperparameter) — Controls: How many times a failed task will retry execution. Default: 0
• cache_policy (feature-flag) — Controls: Whether task results are cached and cache key strategy. Default: None
• infrastructure (architecture-switch) — Controls: Where and how flow runs execute (local, Docker, Kubernetes). Default: Process
• log_level (env-var) — Controls: Verbosity of logging output. Default: INFO
• concurrency_limit (rate-limit) — Controls: Maximum concurrent executions for tagged runs

Technology Stack

Key Components

• FlowRunner (orchestrator) — Orchestrates flow execution including parameter resolution, task scheduling, and state management src/prefect/flow_engine.py
• TaskRunner (executor) — Executes individual tasks with retry logic, caching, and result persistence src/prefect/task_engine.py
• PrefectClient (gateway) — HTTP client for communicating with Prefect server API, handles authentication and retries src/prefect/client/orchestration.py
• Scheduler (scheduler) — Creates flow runs from deployments based on schedules and triggers src/prefect/server/services/scheduler.py
• Worker (executor) — Polls work queues and executes flow runs in configured infrastructure environments src/prefect/workers/base.py
• DatabaseInterface (adapter) — Abstracts database operations across SQLite and PostgreSQL with connection pooling src/prefect/server/database/interface.py
• StateManager (processor) — Manages state transitions and validation for flows and tasks src/prefect/states.py
• ResultStore (store) — Persists and retrieves flow/task results using pluggable storage backends src/prefect/results.py
• BlockRegistry (registry) — Manages registration and instantiation of configuration blocks with validation src/prefect/blocks/core.py
• EventEmitter (adapter) — Emits events during flow/task execution for automation triggers and observability src/prefect/events/client.py

Compare prefect

Related Repository Repositories

Frequently Asked Questions

Analyzed on April 19, 2026 by CodeSea. Written by Karolina Sarna.