An AI-powered research assistant that performs iterative, deep research on any topic by combining web search, content analysis, and large language models.
🔬 Simple yet powerful research automation in Python
The goal is to provide the simplest implementation of a deep research agent that can refine its research direction over time and dive deep into any topic. Optimized for ease of use and understanding.
- 🔄 Iterative Research: Performs deep research by iteratively generating search queries, processing results, and diving deeper based on findings
- 🎯 Intelligent Query Generation: Uses LLMs to generate targeted search queries based on research goals and previous findings
- ⚙️ Depth & Breadth Control: Configurable parameters to control research scope (breadth: 1-20, depth: 1-10)
- 🔮 Smart Follow-up: Generates follow-up questions to better understand research needs
- 📊 Web Dashboard: Beautiful web interface with real-time progress tracking and organized result tabs
- 🔗 API Server: REST API for integration with other applications
- 🚀 Concurrent Processing: Handles multiple searches and result processing in parallel
- 🤖 Multiple AI Providers: Support for NVIDIA, OpenAI, Fireworks AI, OpenRouter, and local models
- ✨ Advanced Features:
- Semantic Re-ranking: Orders results by relevance using sentence transformers
- Smart Deduplication: Removes near-duplicate content automatically
- Freshness Filtering: Prioritizes recent information (configurable)
- Provenance Tracking: Transparent source attribution with supporting snippets and confidence scores
git clone https://github.com/Finance-LLMs/deep-research-python.git
cd deep-research-python
pip install -r requirements.txtCreate a .env.local file with your API keys:
# Required: Web search and scraping
FIRECRAWL_KEY="your_firecrawl_key"
# AI Provider (choose one)
OPEN_ROUTER_KEY="your_openrouter_key" # free models available
# NVIDIA_API_KEY="your_nvidia_api_key" # Alternative - Llama 3.1 70B, DeepSeek R1
# OPENAI_KEY="your_openai_key" # Alternative - GPT-4o-mini
# FIREWORKS_KEY="your_fireworks_key" # Alternative - DeepSeek R1
# Optional: Advanced features
USE_RERANKING=true # Enable semantic re-ranking (default: true)
DEDUP_THRESHOLD=0.9 # Deduplication threshold (default: 0.9)
MIN_YEAR=2020 # Minimum year for documents (default: 2020)python run_dashboard.py
# Open http://localhost:5000python -m src.runpython -m src.api
# Server starts on port 3051The system automatically selects the best available model in this order:
- Custom Model - if
CUSTOM_MODELandOPENAI_ENDPOINTare set - DeepSeek R1 (OpenRouter) - if
OPEN_ROUTER_KEYis set - NVIDIA Llama 3.1 70B - if
NVIDIA_API_KEYis set - DeepSeek R1 (Fireworks) - if
FIREWORKS_KEYis set - GPT-4o-mini (OpenAI) - Fallback option
OpenRouter
- Visit openrouter.ai
- Free tier includes access to many models
- Excellent DeepSeek R1 performance
NVIDIA API
- Visit build.nvidia.com
- Free access to Llama 3.1 70B and DeepSeek R1
- Great for research applications
Local/Custom Models
OPENAI_ENDPOINT="http://localhost:1234/v1"
CUSTOM_MODEL="your_model_name"
OPENAI_KEY="your_api_key_if_needed"Start the interactive web dashboard:
python run_dashboard.py
# Open http://localhost:5000Features:
- Interactive UI: User-friendly web interface
- Real-time Progress: Live updates with visual progress bars
- Organized Results: Separate tabs for Output, Learnings, Sources, and Feedback
- Download Reports: Export as Markdown files
- Provenance Display: See supporting snippets and confidence scores for each learning
python -m src.runYou'll be prompted to enter your research query and configure parameters:
- Breadth: Number of search queries per iteration (recommended: 2-10, default: 4)
- Depth: Number of research iterations (recommended: 1-5, default: 2)
- Mode: Generate report or specific answer
python -m src.api
# Server starts on port 3051Endpoints:
# Research with concise answer
curl -X POST http://localhost:3051/api/research \
-H "Content-Type: application/json" \
-d '{"query": "Tesla stock performance 2025", "breadth": 3, "depth": 2}'
# Generate detailed report
curl -X POST http://localhost:3051/api/generate-report \
-H "Content-Type: application/json" \
-d '{"query": "AI trends 2025", "breadth": 4, "depth": 2}'The system includes advanced retrieval processing for higher quality results:
- Semantic Re-ranking: Orders search results by relevance using AI embeddings
- Smart Deduplication: Automatically removes near-duplicate content (configurable threshold)
- Freshness Filtering: Prioritizes recent information while filtering outdated content
Configuration:
USE_RERANKING=true # Enable processing (default: true)
DEDUP_THRESHOLD=0.9 # Similarity threshold (default: 0.9)
MIN_YEAR=2020 # Minimum document year (default: 2020)Every research finding includes transparent source attribution:
- Source URLs: Direct links to original documents
- Supporting Snippets: Exact 1-2 sentence excerpts supporting each learning
- Confidence Scores: Similarity scores showing reliability (0-100%)
- Matched Terms: Key terms found in supporting evidence
Example Output:
### Learning #1
**Finding:** Python 3.12 introduces improved error messages
**Source:** "Python 3.12 now provides more detailed error messages..."
**From:** https://docs.python.org/3.12/whatsnew
**Confidence:** 95%Access Provenance Data:
from src.deep_research import deep_research
result = await deep_research("Your query", breadth=4, depth=2)
if result.learnings_with_provenance:
for provenance in result.learnings_with_provenance:
print(f"Learning: {provenance['learning']}")
print(f"Source: {provenance['source_url']}")
print(f"Evidence: {provenance['supporting_snippet']}")
print(f"Confidence: {provenance['confidence_score']:.1%}")| Variable | Description | Default | Options |
|---|---|---|---|
| Core APIs | |||
FIRECRAWL_KEY |
Firecrawl API key (required) | - | Your API key |
FIRECRAWL_BASE_URL |
Custom Firecrawl endpoint | https://api.firecrawl.dev |
URL |
OPEN_ROUTER_KEY |
OpenRouter API key | - | Your API key |
NVIDIA_API_KEY |
NVIDIA API key | - | Your API key |
OPENAI_KEY |
OpenAI API key | - | Your API key |
FIREWORKS_KEY |
Fireworks AI API key | - | Your API key |
| Local Models | |||
CUSTOM_MODEL |
Custom model name | - | Model name |
OPENAI_ENDPOINT |
Custom endpoint URL | - | http://localhost:1234/v1 |
| Search Quality | |||
USE_RERANKING |
Enable retrieval processing | true |
true/false |
DEDUP_THRESHOLD |
Deduplication threshold | 0.9 |
0.0-1.0 |
MIN_YEAR |
Minimum document year | 2020 |
2000-2025 |
| Performance | |||
FIRECRAWL_CONCURRENCY |
Concurrent requests | 2 |
1-10 |
CONTEXT_SIZE |
Max context size | 128000 |
Number |
- Create
.env.localfile with your API keys - Build and run:
docker build -t deep-research-python .
docker run -p 3051:3051 --env-file .env.local deep-research-pythonOr with Docker Compose:
docker compose up -dThe research process follows these steps:
- Query Analysis - Takes user query and generates follow-up questions for refinement
- Search Generation - Creates multiple targeted SERP queries based on research goals
- Content Retrieval - Searches web using Firecrawl API and scrapes relevant pages
- Quality Enhancement - Applies semantic re-ranking, deduplication, and freshness filtering
- Learning Extraction - Analyzes content to extract key insights and learnings
- Provenance Tracking - Links each learning to supporting source snippets with confidence scores
- Iterative Deepening - Generates new research directions and repeats if depth > 0
- Report Generation - Compiles findings into comprehensive markdown reports
flowchart TB
Q[User Query] --> DR[Deep Research]
DR --> SQ[Generate SERP Queries]
SQ --> SR[Search & Scrape]
SR --> QE[Quality Enhancement]
QE --> LE[Extract Learnings]
LE --> PT[Track Provenance]
PT --> D{Depth > 0?}
D -->|Yes| RD[Generate Directions]
RD --> DR
D -->|No| R[Generate Report]
Configure concurrency based on your plan:
# Free tier
FIRECRAWL_CONCURRENCY=1
# Paid tier or self-hosted
FIRECRAWL_CONCURRENCY=5- Start small: Use
breadth=2, depth=1for testing - Monitor rate limits: Watch for 429 errors and adjust concurrency
- Use faster models: NVIDIA models are generally quicker
- Self-host Firecrawl: For unlimited scraping
| Issue | Solution |
|---|---|
| "No model found" error | Ensure at least one AI provider API key is set |
| Rate limit errors | Reduce FIRECRAWL_CONCURRENCY or upgrade Firecrawl plan |
| Empty search results | Check Firecrawl API key and connectivity |
| Import/dependency errors | Run pip install -r requirements.txt |
| Slow processing | Enable GPU, use smaller parameters, or faster model |
Modify logging in src/deep_research.py:
def log(*args):
print(*args) # Enable all debug outputThis project is licensed under MIT LICENSE.