keras-team/keras

Deep Learning for humans

Structural Verdict

How Data Flows Through the System

Data flows from raw inputs through layer transformations, with benchmarking measuring performance at each stage

1. Input Generation — Create synthetic input data matching layer specifications
2. Layer Initialization — Initialize layer with specified parameters and backend
3. Forward Pass — Run prediction or training forward pass through layer
4. Metrics Collection — Measure throughput, latency, and resource usage
5. Backend Comparison — Compare performance across JAX, TensorFlow, PyTorch backends

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Benchmark Calibration (convergence, balancing) — Trigger: Inconsistent timing measurements. Action: Adjust warmup iterations and measurement windows. Exit: Stable performance metrics achieved.
• API Generation (recursive, balancing) — Trigger: Source code changes. Action: Regenerate public API using namex. Exit: API matches source implementation.

Delays & Async Processing

• Benchmark Warmup (async-processing, ~configurable start_batch iterations) — Allows backend JIT compilation to stabilize before measurement
• Build Process (batch-window, ~full source tree copy) — Temporary build directory creation delays API generation

Control Points

• Backend Selection (env-var) — Controls: Which ML backend (JAX/TensorFlow/PyTorch) to use. Default: environment variable
• Benchmark Parameters (runtime-toggle) — Controls: num_samples, batch_size, jit_compile flags. Default: command-line flags
• Mixed Precision Policy (feature-flag) — Controls: Whether to use mixed_float16 or float32 precision. Default: mixed_float16

Technology Stack

Key Components

• api_gen (utility) — Generates the public keras API by copying and organizing source files using namex api_gen.py
• LayerBenchmark (class) — Base class for benchmarking individual layer performance across backends benchmarks/layer_benchmark/base_benchmark.py
• BenchmarkMetricsCallback (class) — Callback for measuring throughput and timing metrics during benchmarks benchmarks/layer_benchmark/base_benchmark.py
• auto-assignment (utility) — GitHub Actions script that automatically assigns issues to team members on rotation .github/workflows/scripts/auto-assignment.js
• labeler (utility) — GitHub Actions script that automatically adds labels based on issue content keywords .github/workflows/scripts/labeler.js
• pip_build (utility) — Custom build script for development installation with backend configuration pip_build.py
• copy_source_to_build_directory (function) — Copies keras source files to temporary build directory for API generation api_gen.py
• create_legacy_directory (function) — Creates legacy _tf_keras directory structure for backward compatibility api_gen.py
• benchmark_dense (function) — Benchmarks Dense layer performance with configurable parameters benchmarks/layer_benchmark/core_benchmark.py
• benchmark_conv2D (function) — Benchmarks Conv2D layer performance across different input sizes benchmarks/layer_benchmark/conv_benchmark.py
• BenchmarkMetricsCallback (class) — Model-level benchmark callback for measuring training throughput benchmarks/model_benchmark/benchmark_utils.py
• bert_benchmark (module) — End-to-end BERT model benchmark on GLUE/MRPC task benchmarks/model_benchmark/bert_benchmark.py

Configuration

codecov.yml (yaml)

• coverage.status.project.default.target (string, unknown) — default: auto
• coverage.status.patch.default (string, unknown) — default: target:auto
• comment.layout (string, unknown) — default: header, reach, diff, flags, files
• comment.behavior (string, unknown) — default: default
• comment.require_changes (string, unknown) — default: no
• comment.require_base (string, unknown) — default: no
• comment.require_head (string, unknown) — default: yes
• comment.show_carryforward_flags (string, unknown) — default: yes
• +3 more parameters

Science Pipeline

1. Data Generation — numpy.random or tf.random to create synthetic inputs matching input_shape benchmarks/layer_benchmark/base_benchmark.py
2. Layer Forward Pass — layer(inputs) call with optional JIT compilation [(batch_size, *input_shape) → depends on layer type] benchmarks/layer_benchmark/base_benchmark.py
3. Performance Measurement — time.time() before/after layer calls to measure throughput [timing scalars → throughput list] benchmarks/layer_benchmark/base_benchmark.py
4. Model Training — model.fit() with custom callback for epoch-level metrics [(num_samples, input_dims) → training history] benchmarks/model_benchmark/benchmark_utils.py

Assumptions & Constraints

• [warning] Assumes input_shape parameter matches layer's expected input dimensionality but no runtime validation (shape)
• [info] Assumes attention layer receives list input with shapes [[256, 64], [256, 64]] for query and key/value (shape)
• [warning] Mixed precision policy assumes float16 compatibility without checking backend support (dtype)

Related Ml Training Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.