unidata/netcdf4-python

netcdf4-python: python/numpy interface to the netCDF C library

How Data Flows Through the System

Data enters through Dataset constructor which opens netCDF files via the C library and creates Python proxy objects for the file's hierarchical structure. Variables act as lazy array proxies - reads trigger netCDF C library calls that return raw data, which gets converted to numpy arrays with proper dtype mapping and missing value handling. Writes flow the opposite direction: numpy arrays get validated, optionally compressed/chunked, and passed to the C library for storage. Complex numbers get special handling through the nc_complex extension, either as compound types or dimensional encoding.

1. File Open and Introspection — Dataset.__init__ calls nc_open from the netCDF C library, then queries the file structure using nc_inq_* functions to populate Python objects for groups, dimensions, and variables without loading actual array data [file path → Dataset] (config: format, diskless, persist +1)
2. Variable Access Setup — When accessing dataset.variables['name'], a Variable proxy object is created that holds dimension info, chunking parameters, and compression settings from the netCDF file metadata [Dataset → Variable]
3. Array Data Reading — Variable.__getitem__ with slice notation triggers nc_get_vars calls to the C library, followed by automatic conversion to numpy arrays with proper dtype mapping and masked array creation for missing values [Variable → numpy.ndarray] (config: auto_mask, auto_scale)
4. Array Data Writing — Variable.__setitem__ validates numpy array shape/dtype compatibility, optionally applies compression (zlib, quantization), then calls nc_put_vars to write data through the C library [numpy.ndarray] (config: zlib, complevel, shuffle +1)
5. Complex Number Encoding — When auto_complex=True, the nc_complex extension detects complex arrays and either encodes them as compound types with r/i fields or as real arrays with an extra 'complex' dimension for storage compatibility [numpy.ndarray → CompoundType] (config: auto_complex)
6. File Close and Sync — Dataset.close() or context manager exit calls nc_close to flush pending writes and release the file handle, ensuring data integrity [Dataset]

Data Models

The data structures that flow between stages — the contracts that hold the system together.

Hidden Assumptions

Things this code relies on but never validates. These are the things that cause silent failures when the system changes.

System Behavior

How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• parallel write coordination (polling, balancing) — Trigger: MPI collective write mode. Action: Each process polls for completion of other processes' writes before proceeding. Exit: All processes complete their portion.
• compression optimization (, reinforcing) — Trigger: benchmark script execution. Action: Tests different compression levels and algorithms, measures performance. Exit: All parameter combinations tested.

Delays

• build-time feature detection (compilation, ~seconds) — Build process pauses to run nc-config and inspect headers to determine available netCDF features
• array chunking (batch-window, ~variable) — Large array operations get broken into chunks based on memory constraints and file chunk size
• diskless buffering (cache-ttl, ~until close) — Diskless files accumulate all changes in memory before writing to disk on close

Control Points

• parallel support detection (architecture-switch) — Controls: Whether mpi4py gets added as build dependency based on netCDF C library parallel support. Default: runtime detection
• compression level (hyperparameter) — Controls: Trade-off between file size and write/read performance. Default: 0-9
• auto_complex (feature-flag) — Controls: Whether to automatically detect and convert complex number representations. Default: False by default
• diskless mode (runtime-toggle) — Controls: Whether file operations happen in memory with optional persistence to disk. Default: False by default

Technology Stack

Key Components

• Dataset (gateway) — Main entry point that opens netCDF files and provides hierarchical access to groups, dimensions, variables, and attributes with automatic resource management src/netCDF4
• build backend (adapter) — Custom setuptools build backend that detects netCDF C library features at build time and conditionally adds mpi4py dependency for parallel support _build/backend.py
• config detector (resolver) — Introspects netCDF C library installation by parsing nc-config output and checking header files to determine available features like parallel I/O _build/utils.py
• compression benchmarks (validator) — Performance testing suite that measures read/write speeds across different compression algorithms (zlib, quantization) and compression levels examples/bench_compress*.py
• parallel I/O handler (orchestrator) — Demonstrates collective and independent parallel file access using MPI, coordinating multiple processes writing to the same netCDF file examples/mpi_example.py
• nc_complex (encoder) — Extends netCDF with complex number support by encoding them either as compound types with real/imaginary fields or as arrays with an extra dimension external/nc_complex/src/nc_complex.c

Related Library Repositories

Frequently Asked Questions

Analyzed on April 20, 2026 by CodeSea. Written by Karolina Sarna.