optuna/optuna
A hyperparameter optimization framework
13 hidden assumptions · 6-stage pipeline · 10 components
Like any codebase, this ml training makes assumptions it never checks — most are routine. The ones worth your attention are below.
Suggests hyperparameters for machine learning models using Bayesian optimization and early stopping
Users define objective functions that take Trial objects and return scalar values to optimize. The Study orchestrates an optimization loop where samplers analyze past trial results to suggest new hyperparameter combinations, trials execute user code with those parameters while reporting intermediate values, pruners decide whether to terminate trials early, and storage backends persist all results. This creates a feedback cycle where each trial's outcome improves future parameter suggestions.
Under the hood, the system uses 4 feedback loops, 4 data pools, 6 control points to manage its runtime behavior.
A 10-component ml training. 380 files analyzed. Data flows through 6 distinct pipeline stages.
Hidden Assumptions
Most of what this code assumes is routine. These 3 are the ones most likely to cause trouble here. The rest are minor; they're under "Show everything".
Silent parameter bound violations where later suggest calls with different bounds are ignored, causing optimization to explore wrong parameter spaces without error messages
Incorrect pruning decisions where trials are terminated based on wrong median calculations, potentially killing promising trials or keeping poor ones
Lost trial results during network failures, or corrupted study state when multiple processes modify the same study simultaneously without coordination
Show everything (10 more)
Assumes objective function values follow a reasonable distribution where Tree-structured Parzen Estimators can model good vs bad parameter regions - breaks with pathological objectives like random noise, step functions, or objectives with extreme outliers spanning orders of magnitude
If this fails: TPE sampler suggests poor parameters indefinitely without convergence warning, wasting compute resources on optimization that cannot succeed
optuna/samplers/_tpe/sampler.py:TPESampler.sample
Assumes sufficient memory and CPU to fit Gaussian Process models that scale quadratically with trial history - no checks for memory usage as trial count grows into thousands
If this fails: Out-of-memory crashes or extreme slowdown when GP model fitting requires more resources than available, with no graceful degradation to simpler samplers
optuna/samplers/_gp/sampler.py:GPSampler model fitting
Assumes user's objective function always returns a single numeric value (or tuple for multi-objective) and that the return type matches the study's direction count - no validation that objective(trial) returns expected shape
If this fails: Type errors or wrong optimization behavior when objective returns None, strings, lists, or wrong number of values for multi-objective studies, potentially corrupting study results
optuna/study/_optimize.py:Study.optimize
Assumes study history fits in memory for visualization with reasonable rendering time - no pagination or sampling for studies with hundreds of thousands of trials
If this fails: Browser crashes or multi-minute rendering delays when visualizing large studies, making analysis impossible for long-running optimizations
optuna/visualization/plot_optimization_history.py and other plot functions
Assumes intermediate values represent monotonic progress metrics (like validation loss decreasing or accuracy increasing) where early values predict final performance - pruning logic breaks if metrics oscillate or improve late in training
If this fails: Premature termination of trials that would eventually converge, especially in deep learning where validation metrics often fluctuate before stabilizing
optuna/trial/_trial.py:Trial.report
Assumes database schema matches expected table structure and that migrations have been properly applied - no version checking between Optuna code and database schema
If this fails: Cryptic SQL errors or silent data corruption when connecting to databases created with different Optuna versions that have incompatible schemas
optuna/storages/_rdb/storage.py:RDBStorage initialization
Assumes that when should_prune() returns True, the user will immediately raise TrialPruned - if user ignores the pruning signal and continues execution, the trial's intermediate values and final result are still recorded normally
If this fails: Inconsistent pruning behavior where some trials marked for pruning complete anyway, skewing sampler's understanding of which parameter regions were actually explored
optuna/trial/_trial.py:Trial.should_prune
Assumes objective functions are thread-safe or process-safe when n_jobs > 1, and that shared resources like GPU memory or database connections won't cause conflicts between parallel trials
If this fails: Race conditions, deadlocks, or resource conflicts when multiple trials access shared resources simultaneously, leading to crashes or incorrect results
optuna/study/_optimize.py:parallel trial execution with n_jobs
Assumes that random sampling during warmup period provides representative coverage of the parameter space - if search space has rare but important regions, random sampling might miss them entirely
If this fails: TPE model trained on unrepresentative data leads to poor suggestions that avoid optimal parameter regions, reducing overall optimization effectiveness
optuna/samplers/_tpe/sampler.py:n_startup_trials warmup
Assumes parameter bounds make mathematical sense (low < high for numeric types, non-empty choices for categorical) and that log=True parameters have positive bounds - no validation of distribution consistency
If this fails: Infinite loops or NaN values when samplers try to generate parameters from invalid distributions, causing optimization to hang or crash
optuna/distributions/_base.py:BaseDistribution constraints
Open the standalone hidden-assumptions report for optuna →
How Data Flows Through the System
Users define objective functions that take Trial objects and return scalar values to optimize. The Study orchestrates an optimization loop where samplers analyze past trial results to suggest new hyperparameter combinations, trials execute user code with those parameters while reporting intermediate values, pruners decide whether to terminate trials early, and storage backends persist all results. This creates a feedback cycle where each trial's outcome improves future parameter suggestions.
- Create study with optimization configuration — create_study() instantiates a Study object with specified sampler (e.g. TPESampler), pruner (e.g. MedianPruner), storage backend (e.g. RDBStorage), and optimization direction (minimize/maximize)
- Generate trial with parameter suggestions — Study.ask() creates a new Trial object and invokes the sampler's sample() method, which analyzes past FrozenTrial results to suggest promising parameter values based on Bayesian optimization [FrozenTrial → Trial]
- Execute objective function with suggested parameters — User's objective function receives the Trial, calls trial.suggest_float/suggest_int/suggest_categorical to define the search space and get parameter values, then runs training/evaluation code with those hyperparameters [Trial → objective value]
- Report intermediate results for pruning decisions — During execution, objective function calls trial.report(value, step) to log intermediate metrics like validation loss, which pruners use to decide if the trial should be terminated early via trial.should_prune() [intermediate values → pruning decision]
- Complete trial and store results — Study.tell() marks the trial as complete with its final objective value, creates a FrozenTrial record, and persists all trial data (parameters, result, intermediate values) to the storage backend [objective value → FrozenTrial]
- Analyze results for visualization and next suggestions — Visualization functions like plot_optimization_history query completed FrozenTrial records from storage to generate plots, while samplers analyze the same data to improve future parameter suggestions [FrozenTrial → visualizations]
Data Models
The data structures that flow between stages — the contracts that hold the system together.
optuna/trial/_trial.pyObject with trial_id: int, study_id: int, number: int, state: TrialState, params: dict[str, Any], distributions: dict[str, BaseDistribution], user_attrs: dict[str, Any], intermediate_values: dict[int, float], datetime_start: datetime, datetime_complete: datetime
Created by Study, populated with suggested parameters from Sampler, executed by user's objective function, and stored in Storage backend
optuna/study/_study.pyObject with study_id: int, study_name: str, direction: StudyDirection, user_attrs: dict[str, Any], system_attrs: dict[str, Any], sampler: BaseSampler, pruner: BasePruner, storage: BaseStorage
Created via create_study(), manages the optimization loop by coordinating trials, samplers, and pruners until termination criteria are met
optuna/distributions/_base.pyAbstract base with subclasses like FloatDistribution(low: float, high: float, log: bool), IntDistribution(low: int, high: int, step: int), CategoricalDistribution(choices: Sequence[Any])
Defined when trial suggests parameters, used by samplers to understand parameter bounds and types, stored with trial results
optuna/trial/_frozen.pyImmutable version of Trial with same fields but read-only, used for analysis and visualization
Created from completed Trial objects, passed to samplers for suggesting new parameters and to visualization functions for analysis
optuna/study/_study_summary.pyObject with study_name: str, direction: StudyDirection, best_trial: FrozenTrial, user_attrs: dict[str, Any], system_attrs: dict[str, Any], n_trials: int, datetime_start: datetime
Generated by storage backends when querying study metadata, used for study selection and management operations
System Behavior
How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.
Data Pools
Persistent store of all completed trials including parameters, objective values, intermediate results, and metadata, used by samplers for Bayesian optimization
Registry of active and completed optimization studies with their configuration, best results, and progress statistics
Temporary storage of step-by-step metrics during trial execution, used by pruners for early stopping decisions
Cached search space definitions to avoid recomputing distribution parameters across multiple sampling requests
Feedback Loops
- Bayesian Optimization Loop (training-loop, reinforcing) — Trigger: Study.optimize() with remaining trial budget. Action: Sampler analyzes FrozenTrial history to model objective function and suggests parameters with high expected improvement for next trial. Exit: n_trials limit reached, timeout exceeded, or termination callback returns True.
- Trial Pruning Loop (convergence, balancing) — Trigger: trial.report() called with intermediate value. Action: Pruner compares current performance to historical trials at same step and determines if trial should be terminated early. Exit: Trial completes normally or raises TrialPruned exception.
- Multi-objective Pareto Optimization (convergence, reinforcing) — Trigger: Study with multiple optimization directions receives new trial results. Action: Updates Pareto front with non-dominated solutions and guides future sampling toward unexplored regions of the trade-off surface. Exit: Convergence criteria met or trial budget exhausted.
- Storage Transaction Retry (retry, balancing) — Trigger: Database connection failure or transaction conflict during trial persistence. Action: RDBStorage retries the operation with exponential backoff to handle concurrent access and temporary failures. Exit: Operation succeeds or maximum retry attempts exceeded.
Delays
- Sampler Warmup (warmup, ~random sampling until enough trials collected) — TPE sampler uses random sampling for first n_startup_trials before switching to Bayesian optimization
- Database Connection Pooling (eventual-consistency, ~milliseconds to seconds) — Concurrent studies may see slightly stale trial data until database transactions commit and replicate
- GP Model Fitting (compilation, ~seconds per trial) — GPSampler requires model fitting before each suggestion, creating computational overhead that scales with trial history
- Visualization Rendering (batch-window, ~100ms-1s depending on trial count) — Interactive plots must process all trial data and render visualizations, with latency increasing with study size
Control Points
- n_startup_trials (hyperparameter) — Controls: Number of random trials before TPE sampler switches to Bayesian optimization. Default: 10
- n_warmup_steps (hyperparameter) — Controls: Minimum trial steps before HyperbandPruner begins early stopping decisions. Default: 5
- storage_class (architecture-switch) — Controls: Choice between InMemoryStorage, RDBStorage, or JournalStorage for trial persistence strategy. Default: RDBStorage
- direction (hyperparameter) — Controls: Whether to minimize or maximize the objective function, affecting all optimization algorithms. Default: minimize
- n_jobs (runtime-toggle) — Controls: Number of parallel trial executions, enabling distributed hyperparameter search. Default: 1
- catch (feature-flag) — Controls: Which exceptions to catch and continue optimization vs. terminating the entire study. Default: []
Technology Stack
Database ORM for persisting studies and trials across SQL databases with schema migration support
Database migration tool for evolving Optuna's storage schema across versions
Numerical computing foundation for samplers and acquisition functions in Bayesian optimization
Interactive visualization backend for optimization history, parameter relationships, and Pareto fronts
Static plotting backend providing alternative visualizations for research and publication use
Evolution strategy optimization algorithm implemented as pluggable sampler for continuous parameter spaces
Feature importance calculation for hyperparameter analysis and visualization components
Numerical backends for Gaussian Process sampler fitting and acquisition function optimization
Configuration file parsing for CLI-based study execution and parameter specification
Progress bar display during long optimization runs with trial completion tracking
Key Components
- Study (orchestrator) — Main controller that coordinates the optimization loop by managing trials, invoking samplers for parameter suggestions, checking pruners for early stopping, and persisting results to storage
optuna/study/_study.py - TPESampler (optimizer) — Bayesian optimization algorithm that builds probabilistic models of the objective function using Tree-structured Parzen Estimators to suggest promising hyperparameters based on past trial results
optuna/samplers/_tpe/sampler.py - MedianPruner (validator) — Early stopping mechanism that terminates trials whose intermediate results fall below the median performance of previous trials at the same step, preventing waste of computational resources on poor configurations
optuna/pruners/_median.py - RDBStorage (store) — Database-backed storage engine that persists study metadata, trial results, and optimization history to SQL databases with ACID guarantees for concurrent access
optuna/storages/_rdb/storage.py - BaseStorage (adapter) — Abstract interface that standardizes how different storage backends (in-memory, RDB, journal-based) handle study and trial persistence operations
optuna/storages/_base.py - TrialPruned (validator) — Exception raised by user's objective function to signal early termination of a trial based on pruner recommendations or custom logic
optuna/exceptions.py - suggest_float/suggest_int/suggest_categorical (gateway) — API methods that capture hyperparameter definitions from user code, create appropriate distributions, and request values from the active sampler
optuna/trial/_trial.py - create_study (factory) — Factory function that instantiates Study objects with specified samplers, pruners, storage backends, and optimization directions
optuna/study/_create_study.py - plot_optimization_history (transformer) — Visualization component that transforms trial results into interactive plots showing objective value progression over trial iterations
optuna/visualization/_optimization_history.py - GPSampler (optimizer) — Gaussian Process-based Bayesian optimization that models the objective function as a probabilistic surface to suggest parameters with high expected improvement
optuna/samplers/_gp/sampler.py
Explore the interactive analysis
See the full architecture map, data flow, and code patterns visualization.
Analyze on CodeSeaRelated Ml Training Repositories
Frequently Asked Questions
What is optuna used for?
Suggests hyperparameters for machine learning models using Bayesian optimization and early stopping optuna/optuna is a 10-component ml training written in Python. Data flows through 6 distinct pipeline stages. The codebase contains 380 files.
How is optuna architected?
optuna is organized into 5 architecture layers: Study Management, Optimization Algorithms, Early Stopping, Storage & Persistence, and 1 more. Data flows through 6 distinct pipeline stages. This layered structure keeps concerns separated and modules independent.
How does data flow through optuna?
Data moves through 6 stages: Create study with optimization configuration → Generate trial with parameter suggestions → Execute objective function with suggested parameters → Report intermediate results for pruning decisions → Complete trial and store results → .... Users define objective functions that take Trial objects and return scalar values to optimize. The Study orchestrates an optimization loop where samplers analyze past trial results to suggest new hyperparameter combinations, trials execute user code with those parameters while reporting intermediate values, pruners decide whether to terminate trials early, and storage backends persist all results. This creates a feedback cycle where each trial's outcome improves future parameter suggestions. This pipeline design reflects a complex multi-stage processing system.
What technologies does optuna use?
The core stack includes SQLAlchemy (Database ORM for persisting studies and trials across SQL databases with schema migration support), Alembic (Database migration tool for evolving Optuna's storage schema across versions), NumPy (Numerical computing foundation for samplers and acquisition functions in Bayesian optimization), Plotly (Interactive visualization backend for optimization history, parameter relationships, and Pareto fronts), Matplotlib (Static plotting backend providing alternative visualizations for research and publication use), CMA-ES (Evolution strategy optimization algorithm implemented as pluggable sampler for continuous parameter spaces), and 4 more. This broad technology surface reflects a mature project with many integration points.
What system dynamics does optuna have?
optuna exhibits 4 data pools (Trial History, Study Registry), 4 feedback loops, 6 control points, 4 delays. The feedback loops handle training-loop and convergence. These runtime behaviors shape how the system responds to load, failures, and configuration changes.
What design patterns does optuna use?
6 design patterns detected: Strategy Pattern for Optimization Algorithms, Observer Pattern for Trial Lifecycle Events, Template Method for Pruning Logic, Factory Pattern for Study Creation, Command Pattern for Trial Operations, and 1 more.
Analyzed on April 20, 2026 by CodeSea. Written by Karolina Sarna.