haotian-liu/llava

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

Structural Verdict

How Data Flows Through the System

Images and text flow through vision encoder, multimodal projector, and language model for instruction following

1. Load Data — Load image-text instruction pairs from configured datasets (config: data_path, image_folder, lazy_preprocess)
2. Process Images — Encode images through vision tower and project to language model dimension (config: vision_tower, mm_vision_select_layer, mm_projector_type)
3. Format Conversation — Apply conversation templates with special tokens for multimodal input (config: version, mm_use_im_start_end, mm_use_im_patch_token)
4. Train Model — Fine-tune language model with multimodal inputs and instruction targets (config: model_name_or_path, freeze_backbone, tune_mm_mlp_adapter +1)
5. Evaluate — Test on visual reasoning benchmarks with automated GPT-4 scoring

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Worker Heartbeat (polling, balancing) — Trigger: Timer every 15 seconds. Action: Workers send status to controller. Exit: Worker shutdown.
• GPT-4 Retry (retry, balancing) — Trigger: Rate limit or API error. Action: Sleep and retry API call. Exit: Successful response or max retries.
• Training Loop (training-loop, reinforcing) — Trigger: Start training. Action: Forward pass, compute loss, backprop. Exit: Max steps reached.

Delays & Async Processing

• Heart Beat Expiration (cache-ttl, ~30 seconds) — Worker marked as inactive
• API Rate Limiting (rate-limit, ~0.5-3 seconds) — Evaluation pipeline pauses
• Image Processing (async-processing) — Latency in multimodal inference

Control Points

• Vision Tower Selection (env-var) — Controls: Which vision encoder to use. Default: vision_tower
• Conversation Version (runtime-toggle) — Controls: Chat format and special tokens. Default: version
• Temperature (threshold) — Controls: GPT-4 evaluation randomness. Default: 0.2
• Freeze Backbone (feature-flag) — Controls: Whether to freeze language model weights. Default: freeze_backbone

Technology Stack

Key Components

• Conversation (class) — Manages conversation history and prompt formatting with different separator styles llava/conversation.py
• load_pretrained_model (function) — Factory function to load and initialize pretrained LLaVA models with vision components llava/model/builder.py
• LlavaLlamaForCausalLM (class) — Core multimodal architecture combining Llama language model with vision tower llava/model/llava_arch.py
• LLaVATrainer (class) — Custom trainer for multimodal instruction tuning with vision-language data llava/train/llava_trainer.py
• eval_model (function) — Evaluates LLaVA models on visual question answering benchmarks llava/eval/model_vqa.py
• get_eval (function) — Uses GPT-4 to evaluate model responses with automatic retry logic llava/eval/eval_gpt_review.py
• process_images (function) — Preprocesses images for model input with proper aspect ratio handling llava/mm_utils.py
• tokenizer_image_token (function) — Tokenizes text with special image tokens for multimodal processing llava/mm_utils.py
• ControllerManager (class) — Manages worker nodes and load balancing for distributed serving llava/serve/controller.py
• build_demo (function) — Creates Gradio web interface for interactive multimodal conversations llava/serve/gradio_web_server.py

Configuration

cog.yaml (yaml)

• build.gpu (boolean, unknown) — default: true
• build.python_version (string, unknown) — default: 3.11
• build.python_packages (array, unknown) — default: torch==2.0.1,accelerate==0.21.0,bitsandbytes==0.41.0,deepspeed==0.9.5,einops-exts==0.0.4,einops==0.6.1,gradio==3.35.2,gradio_client==0.2.9,httpx==0.24.0,markdown2==2.4.10,numpy==1.26.0,peft==0.4.0,scikit-learn==1.2.2,sentencepiece==0.1.99,shortuuid==1.0.11,timm==0.6.13,tokenizers==0.13.3,torch==2.0.1,torchvision==0.15.2,transformers==4.31.0,wandb==0.15.12,wavedrom==2.0.3.post3,Pygments==2.16.1
• build.run (array, unknown) — default: curl -o /usr/local/bin/pget -L "https://github.com/replicate/pget/releases/download/v0.0.3/pget" && chmod +x /usr/local/bin/pget
• predict (string, unknown) — default: predict.py:Predictor

llava/conversation.py (python-dataclass)

• system (str, unknown)
• roles (List[str], unknown)
• messages (List[List[str]], unknown)
• offset (int, unknown)
• sep_style (SeparatorStyle, unknown) — default: SeparatorStyle.SINGLE
• sep (str, unknown) — default: "###"
• sep2 (str, unknown) — default: None
• version (str, unknown) — default: "Unknown"

llava/serve/controller.py (python-dataclass)

• model_names (List[str], unknown)
• speed (int, unknown)
• queue_length (int, unknown)
• check_heart_beat (bool, unknown)
• last_heart_beat (str, unknown)

llava/train/train.py (python-dataclass)

• model_name_or_path (Optional[str], unknown) — default: field(default="facebook/opt-125m")
• version (Optional[str], unknown) — default: field(default="v0")
• freeze_backbone (bool, unknown) — default: field(default=False)
• tune_mm_mlp_adapter (bool, unknown) — default: field(default=False)
• vision_tower (Optional[str], unknown) — default: field(default=None)
• mm_vision_select_layer (Optional[int], unknown) — default: field(default=-1) # default to the last layer
• pretrain_mm_mlp_adapter (Optional[str], unknown) — default: field(default=None)
• mm_projector_type (Optional[str], unknown) — default: field(default='linear')
• +4 more parameters

Science Pipeline

1. Load Images — PIL.Image.open then convert RGB [Variable (H, W, 3) → (3, H, W)] llava/mm_utils.py
2. Vision Encoding — Vision tower forward pass with layer selection [(3, 336, 336) → (576, 1024)] llava/model/multimodal_encoder.py
3. Multimodal Projection — Linear/MLP projection to language model dimension [(576, 1024) → (576, 4096)] llava/model/multimodal_projector.py
4. Token Integration — Replace image tokens with vision features in text sequence [(seq_len, 4096) + (576, 4096) → (seq_len + 575, 4096)] llava/model/llava_arch.py
5. Language Generation — Autoregressive generation with multimodal context [(seq_len + 575, 4096) → (output_len, vocab_size)] llava/model/llava_arch.py

Assumptions & Constraints

• [warning] Assumes images are PIL Image objects but no type checking enforced (shape)
• [critical] Assumes input_ids tensor shape matches image token positions without validation (shape)
• [info] Assumes image_aspect_ratio 'square' produces consistent dimensions across datasets (format)
• [warning] Vision tower output dimension must match projector input but no runtime check (dependency)

Related Ml Training Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.