clearml/clearml
ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
13 hidden assumptions · 6-stage pipeline · 10 components
Like any codebase, this ml training makes assumptions it never checks — most are routine. The ones worth your attention are below.
Instruments AI experiments with automatic tracking and orchestrates MLOps pipelines
ClearML instruments ML code by patching framework imports to automatically capture experiment data. When users import frameworks like PyTorch, the binding layer intercepts calls to track model architecture, hyperparameters, and training metrics. This data flows through the Logger to the backend API Session, which authenticates with ClearML server and stores experiment metadata. For pipeline workflows, the PipelineController schedules ClearmlJobs on compute queues, where each job inherits from a base task template but applies step-specific parameter overrides. Results flow back through the pipeline controller to update the overall workflow status.
Under the hood, the system uses 4 feedback loops, 4 data pools, 4 control points to manage its runtime behavior.
A 10-component ml training. 397 files analyzed. Data flows through 6 distinct pipeline stages.
Hidden Assumptions
Most of what this code assumes is routine. These 3 are the ones most likely to cause trouble here. The rest are minor; they're under "Show everything".
If worker registration uses different naming convention or includes special characters, the regex fails to parse worker details causing auto-scaling decisions based on incorrect worker inventory and potential resource waste or starvation
Circular dependencies or permanently stuck tasks cause pipeline to wait indefinitely without timeout, blocking all dependent steps and consuming queue resources while appearing to run normally
If callback returns non-deterministic hashes or identical hashes for different jobs, the caching mechanism fails silently - either rerunning identical jobs wastefully or reusing wrong cached results leading to incorrect pipeline outputs
Show everything (10 more)
The optimization algorithm (GridSearch, RandomSearch, BOHB, Optuna) libraries are installed and compatible with the parameter space definitions provided
If this fails: Missing optimization libraries cause runtime ImportError during optimization start, while incompatible parameter ranges cause silent fallback to basic search without notification, leading to suboptimal hyperparameter exploration
clearml/automation/optimization.py:HyperParameterOptimizer
All scheduled tasks execute within single timezone context and datetime parsing handles timezone-naive strings by assuming UTC rather than local system timezone
If this fails: When mixing timezone-aware and timezone-naive datetime inputs across different geographic regions, tasks may execute at wrong times causing data inconsistencies or missing time-sensitive processing windows
clearml/automation/scheduler.py:_datetime_from_isoformat_as_utc
Auto-scaling decisions operate on 60-second intervals and cloud provider APIs respond within this timeframe to provision/terminate instances
If this fails: Cloud API delays or instance boot times longer than 60 seconds cause scaling lag where new instances aren't counted in next scaling decision, potentially triggering excessive provisioning and unexpected costs
clearml/automation/auto_scaler.py:MINUTE
Parameter overrides in pipeline steps are JSON-serializable dictionaries that don't contain circular references or complex objects requiring custom serialization
If this fails: Non-serializable parameter values cause JSON encoding failures during job submission, while circular references create infinite serialization loops that hang pipeline creation without clear error messages
clearml/automation/controller.py:PipelineController.add_step
Job status updates arrive in chronological order and timestamp comparison accurately reflects the sequence of actual status changes on the server
If this fails: Out-of-order status updates due to network delays or server clock skew cause jobs to appear as reverting to previous states, confusing pipeline monitoring and potentially triggering incorrect retry logic
clearml/automation/job.py:BaseJob._last_batch_status_update_ts
Parameter ranges (UniformParameterRange, DiscreteParameterRange) have valid min/max bounds where min < max for continuous parameters and discrete values lists are non-empty
If this fails: Invalid parameter ranges with min >= max cause optimization algorithms to fail with unclear errors, while empty discrete value lists generate no parameter combinations leading to optimization that never starts
clearml/automation/optimization.py:Parameter
Target execution queues exist and are accessible by the scheduler, with sufficient worker capacity to handle scheduled job volume
If this fails: Missing queues cause scheduled jobs to be silently queued on non-existent queues where they wait indefinitely, while queue capacity mismatches cause unpredictable scheduling delays without visibility into resource constraints
clearml/automation/scheduler.py:BaseScheduleJob.queue
System clock synchronization between pipeline controller and ClearML server remains stable throughout pipeline execution for accurate status polling
If this fails: Clock drift causes status polling intervals to become unreliable, potentially missing rapid status changes or creating excessive polling load that impacts server performance
clearml/automation/controller.py:PipelineController
Cloud provider instance IDs use consistent naming conventions without colons and maintain stable mapping between instance metadata and ClearML worker registration
If this fails: Cloud provider changes to instance ID format break worker identification logic, causing auto-scaler to lose track of provisioned instances and potentially over-provision resources
clearml/automation/auto_scaler.py:WorkerId.__init__
Parameter override keys match exactly the parameter names expected by the base task, with type compatibility between override values and task parameter types
If this fails: Mismatched parameter names cause overrides to be ignored silently while type mismatches may cause task execution errors that are difficult to trace back to the parameter override source
clearml/automation/job.py:ClearmlJob.task_parameter_override
Open the standalone hidden-assumptions report for clearml →
How Data Flows Through the System
ClearML instruments ML code by patching framework imports to automatically capture experiment data. When users import frameworks like PyTorch, the binding layer intercepts calls to track model architecture, hyperparameters, and training metrics. This data flows through the Logger to the backend API Session, which authenticates with ClearML server and stores experiment metadata. For pipeline workflows, the PipelineController schedules ClearmlJobs on compute queues, where each job inherits from a base task template but applies step-specific parameter overrides. Results flow back through the pipeline controller to update the overall workflow status.
- Initialize experiment tracking — Task.init() creates a new experiment, registers with ClearML server via Session.send_api(), and patches ML framework imports through the binding layer to intercept training calls
- Capture training data — Framework bindings automatically detect model definitions, hyperparameters, and training loops, while Logger.report_scalar() and Logger.report_plot() capture user metrics and visualizations [ML framework calls → Experiment metadata]
- Define pipeline workflow — PipelineController.add_step() builds a dependency graph of tasks, each step specifying base_task_id, parameter_override dict, and target execution queue [Pipeline definition → PipelineController]
- Schedule pipeline execution — PipelineController.start() creates ClearmlJob instances for each step, applies parameter overrides, and enqueues jobs on specified compute queues respecting dependency order [PipelineController → ClearmlJob]
- Execute distributed jobs — ClearML agents poll queues, clone base tasks with parameter overrides, execute training code in isolated environments, and stream results back to server [ClearmlJob → Execution results]
- Collect pipeline results — PipelineController monitors job status via Session API calls, aggregates results from completed steps, and updates pipeline Task with final outputs and artifacts [Execution results → Pipeline completion]
Data Models
The data structures that flow between stages — the contracts that hold the system together.
clearml/task.pyCore experiment object with id: str, name: str, project: str, status: TaskStatus, parameters: dict, artifacts: dict, models: dict, and tracking state
Created when ML code starts, continuously updated during training with metrics and artifacts, finalized when experiment completes or fails
clearml/automation/controller.pyPipeline definition with name: str, project: str, version: str, steps: dict[str, PipelineStep], add_pipeline_tags: list, and execution state
Built by adding pipeline steps with dependencies, executed by scheduling jobs on queues, monitored until all steps complete
clearml/automation/job.pyJob execution wrapper with base_task_id: str, queue: str, task_overrides: dict, execution_queue: str, and status tracking
Created from pipeline step definition, queued for execution, picked up by worker agent, executed with parameter overrides, results collected back to pipeline
clearml/datasets/__init__.pyVersioned dataset with id: str, name: str, project: str, version: str, files: list[DatasetFile], parents: list[str], and metadata
Created as new version, populated with files and metadata, finalized to make immutable, consumed by training tasks
clearml/automation/parameters.pyOptimization space with parameters: dict[str, Parameter], where Parameter can be UniformParameterRange, DiscreteParameterRange, or UniformIntegerParameterRange with name, min/max/values, and sampling method
Defined for hyperparameter optimization, sampled by optimization algorithm to generate parameter combinations, applied to training jobs
System Behavior
How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.
Data Pools
Centralized storage for all experiment metadata, task definitions, pipeline states, and user data accessed through REST API
Job queue where ClearML agents poll for pending tasks, supporting priority ordering and resource-based routing
Cloud or local storage for models, datasets, logs, and other large artifacts with automatic upload/download
Runtime tracking of pipeline execution state, step dependencies, and parameter flow between pipeline stages
Feedback Loops
- Hyperparameter optimization loop (training-loop, reinforcing) — Trigger: HyperParameterOptimizer.start() begins search strategy. Action: Generate parameter combination, create ClearmlJob with overrides, execute training, evaluate objective function. Exit: Maximum iterations reached or convergence criteria met.
- Pipeline retry mechanism (retry, balancing) — Trigger: Job failure or timeout detected by PipelineController. Action: ClearmlJob.retry() creates new job instance with same parameters, increments retry count. Exit: Job succeeds or max_retries exceeded.
- Auto-scaling feedback (auto-scale, balancing) — Trigger: Queue length exceeds threshold or instances idle too long. Action: AutoScaler calculates required capacity, launches/terminates cloud instances, updates worker pool. Exit: Queue load matches target utilization.
- Session refresh cycle (polling, balancing) — Trigger: API token approaching expiration or authentication failure. Action: Session.refresh_token() requests new credentials, updates authentication headers. Exit: Valid token obtained or maximum refresh attempts exceeded.
Delays
- Queue processing latency (queue-drain, ~variable) — Jobs wait in queue until agent becomes available, affecting pipeline execution time
- Model artifact upload (async-processing, ~depends on model size and bandwidth) — Large models upload asynchronously to avoid blocking training code execution
- Pipeline step synchronization (eventual-consistency, ~polling interval) — Dependent steps wait for parent completion status to propagate through server
- Auto-scaler decision delay (scheduled-job, ~monitoring interval (default 5 minutes)) — Resource scaling decisions lag behind actual load changes
Control Points
- CLEARML_API_HOST (env-var) — Controls: Target ClearML server endpoint for all API communication. Default: configurable
- Pipeline retry policy (hyperparameter) — Controls: Maximum retry attempts and backoff strategy for failed pipeline steps. Default: configurable per pipeline
- Auto-scaler thresholds (threshold) — Controls: Queue length triggers for scaling up/down and instance idle timeout. Default: configurable
- Optimization search strategy (architecture-switch) — Controls: Algorithm selection between GridSearch, RandomSearch, BOHB, or Optuna. Default: runtime selection
Technology Stack
HTTP client for ClearML server API communication with retry logic and session management
Immutable dataclass definitions for configuration objects and API models
AWS integration for S3 storage and EC2 auto-scaling in cloud deployments
Bayesian optimization backend for hyperparameter search strategies
Cross-platform filesystem path handling for artifact storage and dataset management
Parallel pipeline execution and isolated job processing to prevent interference
Key Components
- Task (orchestrator) — Central experiment tracker that automatically captures ML framework calls, logs metrics and artifacts, manages model versioning, and coordinates with ClearML server
clearml/task.py - PipelineController (orchestrator) — Defines and executes multi-step ML pipelines with dependency management, parameter passing between steps, and distributed execution across compute queues
clearml/automation/controller.py - ClearmlJob (executor) — Wraps individual pipeline steps or optimization trials as executable jobs, handles parameter overrides, queue management, and result collection
clearml/automation/job.py - Logger (adapter) — Provides unified logging interface for metrics, plots, images, and debug samples that routes to appropriate storage backends and ClearML server
clearml/logger.py - Session (gateway) — Manages authenticated connections to ClearML server, handles API calls with retry logic, version compatibility, and token refresh
clearml/backend_api/session.py - HyperParameterOptimizer (scheduler) — Coordinates hyperparameter search strategies (grid, random, Bayesian), spawns optimization jobs with parameter combinations, and tracks results to guide search
clearml/automation/optimization.py - TaskScheduler (scheduler) — Schedules recurring tasks based on cron expressions or time intervals, manages job lifecycle, and handles timezone-aware scheduling
clearml/automation/scheduler.py - StorageManager (adapter) — Abstracts file storage operations across different backends (S3, GCS, Azure, local filesystem) with automatic upload/download and caching
clearml/storage/__init__.py - Dataset (registry) — Manages versioned datasets with immutable snapshots, tracks file changes between versions, and provides lineage tracking for data provenance
clearml/datasets/__init__.py - AutoScaler (allocator) — Monitors queue load and automatically provisions cloud instances to handle pending jobs, scaling up during high demand and down during idle periods
clearml/automation/auto_scaler.py
Explore the interactive analysis
See the full architecture map, data flow, and code patterns visualization.
Analyze on CodeSeaRelated Ml Training Repositories
Frequently Asked Questions
What is clearml used for?
Instruments AI experiments with automatic tracking and orchestrates MLOps pipelines clearml/clearml is a 10-component ml training written in Python. Data flows through 6 distinct pipeline stages. The codebase contains 397 files.
How is clearml architected?
clearml is organized into 5 architecture layers: Core SDK, Automation & Orchestration, Backend Interface, Data & Storage, and 1 more. Data flows through 6 distinct pipeline stages. This layered structure keeps concerns separated and modules independent.
How does data flow through clearml?
Data moves through 6 stages: Initialize experiment tracking → Capture training data → Define pipeline workflow → Schedule pipeline execution → Execute distributed jobs → .... ClearML instruments ML code by patching framework imports to automatically capture experiment data. When users import frameworks like PyTorch, the binding layer intercepts calls to track model architecture, hyperparameters, and training metrics. This data flows through the Logger to the backend API Session, which authenticates with ClearML server and stores experiment metadata. For pipeline workflows, the PipelineController schedules ClearmlJobs on compute queues, where each job inherits from a base task template but applies step-specific parameter overrides. Results flow back through the pipeline controller to update the overall workflow status. This pipeline design reflects a complex multi-stage processing system.
What technologies does clearml use?
The core stack includes requests (HTTP client for ClearML server API communication with retry logic and session management), attrs (Immutable dataclass definitions for configuration objects and API models), boto3 (AWS integration for S3 storage and EC2 auto-scaling in cloud deployments), optuna (Bayesian optimization backend for hyperparameter search strategies), pathlib2 (Cross-platform filesystem path handling for artifact storage and dataset management), multiprocessing (Parallel pipeline execution and isolated job processing to prevent interference). A focused set of dependencies that keeps the build manageable.
What system dynamics does clearml have?
clearml exhibits 4 data pools (ClearML Server Database, Compute Queue), 4 feedback loops, 4 control points, 4 delays. The feedback loops handle training-loop and retry. These runtime behaviors shape how the system responds to load, failures, and configuration changes.
What design patterns does clearml use?
4 design patterns detected: Monkey Patching for Framework Integration, Template-Based Job Creation, Lazy Proxy Objects, Multi-Strategy Plugin Architecture.
Analyzed on April 20, 2026 by CodeSea. Written by Karolina Sarna.