apache/kafka

Apache Kafka - A distributed event streaming platform

Structural Verdict

How Data Flows Through the System

Messages flow from producers through brokers to consumers, with optional stream processing transformations

1. Message Production — Producers serialize and batch messages, then send to broker partition leaders (config: ProducerConfig.BATCH_SIZE_CONFIG, ProducerConfig.LINGER_MS_CONFIG)
2. Broker Storage — Brokers write messages to log segments and replicate to followers (config: log.segment.bytes, min.insync.replicas)
3. Consumer Fetch — Consumers poll brokers for new messages from assigned partitions (config: ConsumerConfig.FETCH_MIN_BYTES_CONFIG, ConsumerConfig.MAX_POLL_RECORDS_CONFIG)
4. Stream Processing — Optional stream processing applications transform messages in real-time (config: StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG)

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Leader Election (convergence, balancing) — Trigger: Broker failure or partition reassignment. Action: Controllers coordinate new leader selection. Exit: Stable leader elected and acknowledged.
• Consumer Rebalancing (convergence, balancing) — Trigger: Consumer group membership changes. Action: Coordinator reassigns partitions among active consumers. Exit: All consumers acknowledge new assignments.
• Producer Retry (retry, reinforcing) — Trigger: Retriable send failures or timeouts. Action: Resend records with backoff delay. Exit: Successful send or max retries exceeded.
• Replica Fetching (polling, reinforcing) — Trigger: Followers continuously poll leaders. Action: Fetch new log entries from partition leader. Exit: Never exits - continuous operation.

Delays & Async Processing

• Producer Batching (batch-window, ~linger.ms config (default 0ms)) — Trades latency for throughput by waiting to fill batches
• Consumer Poll (async-processing, ~Configurable timeout per poll() call) — Applications wait for message availability
• Log Segment Rolling (scheduled-job, ~segment.ms config (default 7 days)) — Periodic creation of new log segments affects cleanup timing
• Offset Commit (batch-window, ~auto.commit.interval.ms (default 5s)) — Automatic offset commits batched for efficiency

Control Points

• Replication Factor (threshold) — Controls: Number of replicas maintained for fault tolerance. Default: default.replication.factor
• Min In-Sync Replicas (threshold) — Controls: Minimum replicas that must acknowledge writes. Default: min.insync.replicas
• Message Max Bytes (threshold) — Controls: Maximum size allowed for individual messages. Default: message.max.bytes
• Auto Create Topics (feature-flag) — Controls: Whether topics are created automatically on first access. Default: auto.create.topics.enable

Technology Stack

Key Components

• KafkaProducer (class) — Main producer client for sending records to Kafka topics clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
• KafkaConsumer (class) — Main consumer client for reading records from Kafka topics clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
• KafkaRequestHandler (class) — Handles all client requests to the broker including produce, fetch, and metadata requests core/src/main/scala/kafka/server/KafkaRequestHandler.scala
• ReplicaManager (class) — Manages partition replicas, leader election, and data replication across brokers server-common/src/main/java/org/apache/kafka/server/ReplicaManager.java
• LogManager (class) — Manages the lifecycle and operations of topic partition logs storage/src/main/java/org/apache/kafka/storage/internals/log/LogManager.java
• GroupCoordinator (class) — Coordinates consumer group membership and partition assignments group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinator.java
• TransactionCoordinator (class) — Manages transactional producer state and coordinates distributed transactions transaction-coordinator/src/main/java/org/apache/kafka/coordinator/transaction/TransactionCoordinator.java
• KafkaStreams (class) — Main entry point for stream processing applications streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java
• MetadataCache (class) — Caches cluster metadata including broker information and topic partitions server-common/src/main/java/org/apache/kafka/server/MetadataCache.java
• RaftManager (class) — Implements Raft consensus protocol for metadata management in KRaft mode raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java

Sub-Modules

Configuration

config/connect-log4j2.yaml (yaml)

• Configuration.Properties.Property (array, unknown) — default: [object Object],[object Object]
• Configuration.Appenders.Console.name (string, unknown) — default: STDOUT
• Configuration.Appenders.Console.PatternLayout.pattern (string, unknown) — default: ${logPattern}
• Configuration.Appenders.RollingFile (array, unknown) — default: [object Object]
• Configuration.Loggers.Root.level (string, unknown) — default: INFO
• Configuration.Loggers.Root.AppenderRef (array, unknown) — default: [object Object],[object Object]

config/log4j2.yaml (yaml)

• Configuration.Properties.Property (array, unknown) — default: [object Object],[object Object]
• Configuration.Appenders.Console.name (string, unknown) — default: STDOUT
• Configuration.Appenders.Console.PatternLayout.pattern (string, unknown) — default: ${logPattern}
• Configuration.Appenders.RollingFile (array, unknown) — default: [object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
• Configuration.Loggers.Root.level (string, unknown) — default: INFO
• Configuration.Loggers.Root.AppenderRef (array, unknown) — default: [object Object],[object Object]
• Configuration.Loggers.Logger (array, unknown) — default: [object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

config/tools-log4j2.yaml (yaml)

• Configuration.Properties.Property (array, unknown) — default: [object Object]
• Configuration.Appenders.Console.name (string, unknown) — default: STDERR
• Configuration.Appenders.Console.target (string, unknown) — default: SYSTEM_ERR
• Configuration.Appenders.Console.PatternLayout.pattern (string, unknown) — default: ${logPattern}
• Configuration.Loggers.Root.level (string, unknown) — default: WARN
• Configuration.Loggers.Root.AppenderRef (array, unknown) — default: [object Object]

Related Fullstack Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.