robinhood/faust

Python Stream Processing

6,820 stars Python 10 components 14 connections

Python stream processing library built on Kafka with async agents and tables

Messages flow from Kafka topics through agents that transform and route data, with optional stateful processing via tables and output to other topics or external systems

Under the hood, the system uses 2 feedback loops, 3 data pools, 4 control points to manage its runtime behavior.

Structural Verdict

A 10-component ml inference with 14 connections. 391 files analyzed. Highly interconnected — components depend on each other heavily.

How Data Flows Through the System

Messages flow from Kafka topics through agents that transform and route data, with optional stateful processing via tables and output to other topics or external systems

Message Ingestion — Transport layer consumes messages from Kafka topics and deserializes them
Stream Processing — Agents process message streams with async iteration, transformations, and business logic
State Management — Tables maintain local state for joins, aggregations, and stateful computations
Message Production — Processed results are serialized and published to output topics or external systems

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Kafka Topics (queue)
Message queues where events accumulate for processing

Table Stores (state-store)
Local RocksDB or in-memory stores maintaining stateful data for joins and aggregations

Django ORM (database)
Relational database state managed by Django models

Feedback Loops

Consumer Group Rebalancing (auto-scale, balancing) — Trigger: Consumer joins/leaves or partition changes. Action: Reassign topic partitions across consumers. Exit: Stable partition assignment achieved.
LiveCheck Test Execution (polling, balancing) — Trigger: Test case execution. Action: Send test signals and wait for validation. Exit: Test passes or timeout.

Delays & Async Processing

Message Processing Latency (async-processing) — Stream processing pipeline introduces async processing delays
Table Changelog Sync (eventual-consistency) — Local table state may lag behind changelog topic
Cron Schedule Delays (scheduled-job, ~varies by cron expression) — Scheduled tasks wait for next cron interval

Control Points

Concurrency Level (threshold) — Controls: Number of concurrent agent instances processing messages
Topic Partitions (env-var) — Controls: Parallelism and scalability of message processing
Serializer Selection (runtime-toggle) — Controls: Message format and schema validation behavior
Store Backend (env-var) — Controls: Local state persistence mechanism (RocksDB vs in-memory)

Technology Stack

aiokafka (library)
Async Kafka client implementation

mode (framework)
Async service framework and utilities

RocksDB (database)
Local state store backend for tables

pytest (testing)
Testing framework with extensive test suite

Sphinx (build)
Documentation generation

mypy (build)
Static type checking

Django (framework)
Web framework integration example

Key Components

faust.App (class) — Main application class that orchestrates agents, topics, tables, and services faust/app/base.py
Agent (class) — Async stream processor that consumes from topics and processes messages faust/agents/agent.py
Topic (class) — Kafka topic abstraction for sending and receiving messages faust/topics.py
Table (class) — Stateful key-value store backed by changelog topics for stream joins and aggregations faust/tables/table.py
Transport (class) — Abstract base for Kafka transport implementing consumer/producer protocols faust/transport/base.py
Stream (class) — Async iterator over topic messages with transformation and filtering operations faust/streams.py
LiveCheck (class) — Testing framework for end-to-end stream processing validation faust/livecheck/app.py
Serializer (module) — Message serialization/deserialization with support for JSON, Avro, and custom formats faust/serializers/__init__.py
CLI (module) — Command-line interface with worker, send, tables, and other management commands faust/cli/__init__.py
Record (class) — Typed data model with automatic serialization and schema validation faust/models/record.py

Sub-Modules

LiveCheck (independence: medium)
End-to-end testing framework for stream processing applications with distributed test execution

Django Integration (independence: high)
Complete example showing Faust integration with Django ORM and settings

Configuration

environment.yml (yaml)

channels (array, unknown) — default: conda-forge
name (string, unknown) — default: py36
dependencies (array, unknown) — default: pip,python=3.6,setuptools,wheel,[object Object]

readthedocs.yml (yaml)

conda.file (string, unknown) — default: environment.yml

Explore the interactive analysis

See the full architecture map, data flow, and code patterns visualization.

Analyze on CodeSea

Related Ml Inference Repositories

Frequently Asked Questions

What is faust used for?

Python stream processing library built on Kafka with async agents and tables robinhood/faust is a 10-component ml inference written in Python. Highly interconnected — components depend on each other heavily. The codebase contains 391 files.

How is faust architected?

faust is organized into 5 architecture layers: Application Layer, Stream Processing, Storage Layer, Transport Layer, and 1 more. Highly interconnected — components depend on each other heavily. This layered structure enables tight integration between components.

How does data flow through faust?

Data moves through 4 stages: Message Ingestion → Stream Processing → State Management → Message Production. Messages flow from Kafka topics through agents that transform and route data, with optional stateful processing via tables and output to other topics or external systems This pipeline design keeps the data transformation process straightforward.

What technologies does faust use?

The core stack includes aiokafka (Async Kafka client implementation), mode (Async service framework and utilities), RocksDB (Local state store backend for tables), pytest (Testing framework with extensive test suite), Sphinx (Documentation generation), mypy (Static type checking), and 1 more. A focused set of dependencies that keeps the build manageable.

What system dynamics does faust have?

faust exhibits 3 data pools (Kafka Topics, Table Stores), 2 feedback loops, 4 control points, 3 delays. The feedback loops handle auto-scale and polling. These runtime behaviors shape how the system responds to load, failures, and configuration changes.

What design patterns does faust use?

4 design patterns detected: Agent Pattern, Type System, Plugin Architecture, Async Context Managers.

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.