explosion/spacy

💫 Industrial-strength Natural Language Processing (NLP) in Python

Structural Verdict

How Data Flows Through the System

Text flows through tokenization, then sequential pipeline components (tagger, parser, NER, etc.), with training data flowing through alignment and loss computation

1. Text Input — Raw text strings are input to the Language pipeline (config: tokenizer)
2. Tokenization — Text is split into tokens using language-specific rules (config: tokenizer)
3. Pipeline Processing — Tokens pass through configured pipeline components (tagger, parser, NER) (config: pipeline, components)
4. Training Alignment — During training, predicted annotations are aligned with gold standard (config: training.batcher)
5. Loss Computation — Component losses are computed from aligned predictions and targets (config: training.dropout, training.optimizer)
6. Model Updates — Gradients are backpropagated to update model weights (config: training.optimizer, training.max_epochs)

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Training Loop (training-loop, balancing) — Trigger: train_cli command execution. Action: Process batch, compute loss, update weights. Exit: max_epochs or patience exhausted.
• Gradient Descent (convergence, balancing) — Trigger: Loss computation. Action: Backpropagate gradients and update model parameters. Exit: Convergence or max steps.

Delays & Async Processing

• Batch Processing (batch-window, ~configurable batch_size) — Examples accumulate before processing to optimize GPU utilization
• Evaluation Frequency (scheduled-job, ~eval_frequency steps) — Model evaluation is deferred to reduce training overhead

Control Points

• Pipeline Components (runtime-toggle) — Controls: Which NLP components are active in processing pipeline. Default: ['tok2vec', 'tagger', 'parser', 'ner']
• Training Dropout (threshold) — Controls: Regularization strength during training. Default: 0.1
• Batch Size (threshold) — Controls: Number of examples processed simultaneously. Default: 1000
• Max Epochs (threshold) — Controls: Maximum training iterations. Default: 0
• GPU Allocator (env-var) — Controls: GPU memory allocation strategy. Default: null

Technology Stack

Key Components

• Language (class) — Main NLP pipeline class that processes text through configurable components spacy/language.py
• setup_cli (function) — Sets up the complete CLI interface with all spaCy commands spacy/cli/_util.py
• ConfigSchemaTraining (class) — Pydantic schema defining training configuration parameters like epochs, dropout, patience spacy/schemas.py
• Example (class) — Training example class holding predicted and gold-standard annotations spacy/training/example.py
• train_cli (cli-command) — CLI command for training spaCy models from configuration files spacy/cli/train.py
• TrainablePipe (class) — Base class for trainable pipeline components like NER, parser, text classifier spacy/pipeline/
• convert (cli-command) — CLI command to convert training data between formats (CoNLL, JSON, etc.) spacy/cli/convert.py
• debug_data_cli (cli-command) — CLI command to analyze training data for issues and statistics spacy/cli/debug_data.py
• Vocab (class) — Vocabulary storage managing string-to-hash mappings and lexical attributes spacy/vocab.py
• DocBin (class) — Efficient serialization container for storing multiple Doc objects spacy/tokens/
• registry (module) — Component registry system for registering custom architectures, losses, and optimizers spacy/util.py
• Alignment (class) — Bidirectional token alignment between predicted and gold annotations spacy/training/alignment.py

Sub-Modules

Configuration

spacy/pipeline/_edit_tree_internals/schemas.py (python-pydantic)

• prefix_len (StrictInt, unknown) — default: Field(..., title="Prefix length")
• suffix_len (StrictInt, unknown) — default: Field(..., title="Suffix length")
• prefix_tree (StrictInt, unknown) — default: Field(..., title="Prefix tree")
• suffix_tree (StrictInt, unknown) — default: Field(..., title="Suffix tree")

spacy/pipeline/_edit_tree_internals/schemas.py (python-pydantic)

• orig (Union[int, StrictStr], unknown) — default: Field(..., title="Original substring")
• subst (Union[int, StrictStr], unknown) — default: Field(..., title="Replacement substring")

spacy/schemas.py (python-pydantic)

• IN (Optional[List[StrictStr]], unknown) — default: Field(None, alias="in")
• NOT_IN (Optional[List[StrictStr]], unknown) — default: Field(None, alias="not_in")
• IS_SUBSET (Optional[List[StrictStr]], unknown) — default: Field(None, alias="is_subset")
• IS_SUPERSET (Optional[List[StrictStr]], unknown) — default: Field(None, alias="is_superset")
• INTERSECTS (Optional[List[StrictStr]], unknown) — default: Field(None, alias="intersects")

spacy/schemas.py (python-pydantic)

• REGEX (Optional[StrictStr], unknown) — default: Field(None, alias="regex")
• IN (Optional[List[StrictInt]], unknown) — default: Field(None, alias="in")
• NOT_IN (Optional[List[StrictInt]], unknown) — default: Field(None, alias="not_in")
• IS_SUBSET (Optional[List[StrictInt]], unknown) — default: Field(None, alias="is_subset")
• IS_SUPERSET (Optional[List[StrictInt]], unknown) — default: Field(None, alias="is_superset")
• INTERSECTS (Optional[List[StrictInt]], unknown) — default: Field(None, alias="intersects")
• EQ (Optional[Union[StrictInt, StrictFloat]], unknown) — default: Field(None, alias="==")
• NEQ (Optional[Union[StrictInt, StrictFloat]], unknown) — default: Field(None, alias="!=")
• +4 more parameters

Science Pipeline

1. Load training data — Convert various formats (CoNLL, JSON) to spaCy Doc objects [variable text lengths → (n_docs, variable_tokens)] spacy/cli/convert.py
2. Tokenization — Split text into tokens using language-specific rules [raw text strings → (n_tokens, token_features)] spacy/lang/
3. Feature extraction — Extract token embeddings through tok2vec component [(n_tokens, vocab_size) → (n_tokens, embedding_dim)] spacy/pipeline/
4. Component processing — Apply NER, POS tagging, parsing through neural networks [(n_tokens, embedding_dim) → (n_tokens, n_classes)] spacy/pipeline/
5. Alignment — Align predicted tokens with gold standard for training [(pred_tokens, gold_tokens) → Alignment(x2y, y2x)] spacy/training/alignment.py
6. Loss computation — Compute component-specific losses from aligned predictions [(n_tokens, n_classes) → scalar loss] spacy/pipeline/

Assumptions & Constraints

• [warning] Assumes bidirectional alignment arrays x2y and y2x have compatible lengths but no validation enforces this (shape)
• [critical] Pipeline components assume Doc objects have consistent token indexing but no assertion verifies this (format)
• [critical] Training examples assume predicted and gold Doc objects have alignable token structures (dependency)

Related Ml Training Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.