Hidden Assumptions in stable-diffusion

Assumes cross-attention context tensors have exactly 768 dimensions (CLIP ViT-L/14 embedding size) and sequence length matches expected max_seq_len

If this fails: If using different text encoder (e.g., CLIP ViT-B/32 with 512 dims), attention weights are initialized for wrong dimensions causing matrix multiplication shape errors or silent attention computation on padded/truncated features

Assumes ddim_num_steps is much smaller than ddpm_num_timesteps (e.g., 50 vs 1000) for meaningful acceleration

If this fails: If user sets ddim_num_steps=999 when ddpm_num_timesteps=1000, timestep spacing becomes [0,1,2,...,998] instead of [0,20,40,...,980], defeating DDIM's purpose and causing near-identical quality but same computational cost as full DDPM

Assumes sufficient GPU memory for batch_size * image_resolution^2 * 4_channels * precision_bytes, typically requiring 24GB+ for batch_size=4 at 512x512

If this fails: With default config on smaller GPUs (8GB), training crashes with CUDA OOM after several steps when gradients accumulate, but error message doesn't indicate which config parameters to reduce

Assumes EMA weights are updated after every training step in consistent order without skipped steps

If this fails: If training is resumed from checkpoint but EMA decay rate changed, or if validation steps modify model parameters, EMA weights become desynchronized leading to worse inference quality without validation metrics detecting the degradation

Assumes CUDA is available and model.device points to a valid GPU device

If this fails: Hard-coded .to(torch.device('cuda')) call fails on CPU-only systems or when CUDA_VISIBLE_DEVICES hides GPUs, causing immediate crash during sampler initialization with unclear error about device mismatch

Assumes config dict contains '_target_' key pointing to importable class path and 'params' containing valid constructor arguments

If this fails: Typos in config YAML like '_target_': 'ldm.models.diffusion.ddpm.LatentDiffusion' (missing 's') cause AttributeError during model creation, but error doesn't indicate which config file or which component failed to instantiate

Assumes timestep embeddings are integers in range [0, num_timesteps-1] and spatial input dimensions match expected attention_resolutions

If this fails: If timesteps contain negative values or exceed training range, or if latent spatial size doesn't divide evenly by attention_resolutions, the model processes invalid timestep embeddings or misaligned spatial attention leading to generation artifacts

Assumes tensor dimension is reasonable (> 0) for computing 1/sqrt(dim) initialization standard deviation

If this fails: If attention head dimension is 0 due to misconfigured model architecture, init_ attempts 1/sqrt(0) causing division by zero, but error occurs during model initialization making it hard to trace back to attention configuration

Assumes boolean string arguments are always lowercase-normalized English words

If this fails: Command line arguments like --train True or --train YES fail to parse as boolean, falling back to string type causing type errors in downstream config processing, but argparse error message doesn't suggest correct format

Assumes dataset yields dicts with 'image' and 'caption' keys where image is RGB tensor and caption is string

If this fails: If custom datasets use different key names like 'img' or 'text', or if captions are pre-tokenized lists, the training loop silently skips samples or processes wrong data types without validation warnings