assafelovic/gpt-researcher

An autonomous agent that conducts deep research on any data using any LLM providers

Structural Verdict

How Data Flows Through the System

Research query flows through sub-query generation, parallel source retrieval, content scraping, context synthesis, and report generation with real-time progress streaming

1. Query Processing — Break down research question into focused sub-queries (config: services.gpt-researcher.environment.OPENAI_API_KEY, services.gpt-researcher.environment.OPENAI_BASE_URL)
2. Source Discovery — Search web using multiple retriever strategies in parallel (config: services.gpt-researcher.environment.TAVILY_API_KEY, services.gpt-researcher.environment.GOOGLE_API_KEY)
3. Content Extraction — Scrape and clean content from discovered URLs
4. Context Synthesis — Filter and consolidate relevant information from sources
5. Report Generation — Generate structured report with citations using LLM (config: services.gpt-researcher.environment.OPENAI_API_KEY, tone)
6. Output Formatting — Export report as PDF, DOCX, or JSON formats

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Query Refinement Loop (recursive, balancing) — Trigger: Insufficient or low-quality search results. Action: Generate additional sub-queries and search different sources. Exit: Sufficient relevant content gathered or max iterations reached.
• Source Validation Loop (retry, balancing) — Trigger: Failed URL scraping or invalid content. Action: Try alternative sources or retriever strategies. Exit: Successful content extraction or source exhaustion.

Delays & Async Processing

• API Rate Limiting (rate-limit, ~Variable per provider) — Delays between search API calls to avoid hitting rate limits
• LLM Token Processing (async-processing, ~1-30 seconds) — Time required for LLM to process context and generate responses
• Web Scraping Delays (async-processing, ~1-10 seconds per URL) — Network latency and content parsing time for each scraped page

Control Points

• LLM Provider Selection (env-var) — Controls: Which LLM provider to use (OpenAI, Anthropic, Google, etc.). Default: OPENAI_API_KEY
• Search Provider Priority (env-var) — Controls: Primary search service (Tavily, Google, DuckDuckGo). Default: TAVILY_API_KEY
• Report Tone (runtime-toggle) — Controls: Writing style and tone of generated reports. Default: Reflective
• Research Depth (feature-flag) — Controls: Type of research conducted (basic, deep, custom). Default: research_report

Technology Stack

Key Components

• GPTResearcher (class) — Main orchestrator class that conducts research and generates reports gpt_researcher/__init__.py
• conduct_research (function) — Executes the research workflow including query generation and source gathering gpt_researcher/actions/conduct_research.py
• get_sub_queries (function) — Generates focused sub-queries from the main research question gpt_researcher/actions/query_processing.py
• scrape_sites_by_query (function) — Scrapes web content for each sub-query using multiple retrievers gpt_researcher/actions/web_scraper.py
• WebSocketManager (class) — Manages real-time communication between frontend and research engine backend/server/websocket_manager.py
• run_agent (function) — Executes research agent with progress streaming via WebSocket backend/server/websocket_manager.py
• TavilySearch (class) — Primary search retriever using Tavily API for web search gpt_researcher/retrievers/tavily/tavily_search.py
• scrape_url (function) — Extracts and cleans content from web pages gpt_researcher/scraper/scraper.py
• create_chat_completion (function) — Unified LLM interface supporting multiple providers (OpenAI, Anthropic, Google) gpt_researcher/utils/llm.py
• write_report (function) — Synthesizes research data into structured reports with citations gpt_researcher/actions/write_report.py

Sub-Modules

Configuration

docker-compose.yml (yaml)

• services.gpt-researcher.pull_policy (string, unknown) — default: build
• services.gpt-researcher.image (string, unknown) — default: gptresearcher/gpt-researcher
• services.gpt-researcher.build (string, unknown) — default: ./
• services.gpt-researcher.environment.OPENAI_API_KEY (string, unknown) — default: ${OPENAI_API_KEY}
• services.gpt-researcher.environment.OPENAI_BASE_URL (string, unknown) — default: ${OPENAI_BASE_URL}
• services.gpt-researcher.environment.TAVILY_API_KEY (string, unknown) — default: ${TAVILY_API_KEY}
• services.gpt-researcher.environment.LANGCHAIN_API_KEY (string, unknown) — default: ${LANGCHAIN_API_KEY}
• services.gpt-researcher.environment.LOGGING_LEVEL (string, unknown) — default: INFO
• +36 more parameters

langgraph.json (json)

• python_version (string, unknown) — default: 3.11
• dependencies (array, unknown) — default: ./multi_agents
• graphs.agent (string, unknown) — default: ./multi_agents/agent.py:graph
• env (string, unknown) — default: .env

backend/server/app.py (python-pydantic)

• task (str, unknown)
• report_type (str, unknown)
• report_source (str, unknown)
• tone (str, unknown)
• repo_name (str, unknown)
• branch_name (str, unknown)
• generate_in_background (bool, unknown) — default: True

backend/server/app.py (python-pydantic)

• report (str, unknown)
• messages (List[Dict[str, Any]], unknown)

Related Ml Inference Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.