How to Scrape DuckDuckGo SERP Data: 4 Effective Approaches
Note: I am NOT affiliated with any provider, I am using Bright Data in this tutorial as it’s the one I am most familiar with. Feel free to choose any other reputable provider from my list!
Why Scrape DuckDuckGo?
DuckDuckGo has become an important search platform for many people who care about privacy. Because of this, more analysts and developers want to study its results to compare rankings, monitor trends, build datasets, or support AI workflows.
Scraping DuckDuckGo can help you:
- Analyze keyword performance
- Build datasets for machine learning
- Track changes in search trends
- Compare search engines
- Test SEO strategies
- Create automated research tools
DuckDuckGo SERP Versions
DuckDuckGo provides two different versions of its search results page. Understanding these versions helps you choose the right scraping approach.
1. Dynamic Version
- Standard version seen by most users
- Loads content using JavaScript
- Uses features like “More Results”
- Requires JavaScript rendering
- URL format:
https://duckduckgo.com/?q=
2. Static Version
- No JavaScript
- Loads simple HTML
- Uses standard pagination with “Next”
- URL format:
https://html.duckduckgo.com/html/?q=
This version is ideal for lightweight scrapers because you can fetch the page with a quick HTTP request.
Overview of the 4 Approaches
Press enter or click to view image in full size
Approach 1: Build Your Own DuckDuckGo Scraper
This method gives you full control. You write your own Python script to fetch and parse data from DuckDuckGo’s static SERP version.
This approach is great for:
- Learning how scraping works
- Small personal projects
- Custom extraction logic
Step 1: Install Required Libraries
pip install requests beautifulsoup4
requestssends HTTP requestsBeautifulSoupextracts data from HTML
Step 2: Request the Static SERP Page
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
url = "https://html.duckduckgo.com/html/"
params = {"q": "agentic rag"}
response = requests.get(url, headers=headers, params=params)
html = response.text
Step 3: Parse the HTML
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
results = soup.select("#links .result")
Step 4: Extract Useful Information
Each search result includes:
- Title
- URL
- Display link
- Snippet
scraped = []
for item in results:
title_tag = item.select_one(".result__a")
if not title_tag:
continue
title = title_tag.get_text(strip=True)
href = title_tag.get("href", "")
link = href if href.startswith("http") else f"https:{href}"
display_tag = item.select_one(".result__url")
display = display_tag.get_text(strip=True) if display_tag else ""
snippet_tag = item.select_one(".result__snippet")
snippet = snippet_tag.get_text(strip=True) if snippet_tag else ""
scraped.append({
"title": title,
"url": link,
"display_url": display,
"snippet": snippet
})
Step 5: Save to CSV
import csv
if scraped:
with open("results.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["title", "url", "display_url", "snippet"])
writer.writeheader()
writer.writerows(scraped)
print(f"Saved {len(scraped)} results to results.csv")
When to Use This Approach
✅ You want full customization
✅ You enjoy coding
✅ You do not need high-scale scraping
When to Avoid It
❌ You expect thousands of searches
❌ You want the dynamic version with JavaScript
❌ You do not want to handle blocks manually
Approach 2: Use the DDGS Library
DDGS is a Python library and CLI tool built to simplify DuckDuckGo scraping. It is the easiest free method when you don’t want to write your own parser.
Installation
pip install -U ddgs
Scrape Search Results Using the CLI
ddgs text -q "agentic rag" -b duckduckgo -o output.csv
This creates a CSV file with titles, URLs, and snippets.
Pros of DDGS
✅ No need to write code
✅ Works through CLI or Python
✅ Great for small-scale use
Cons
❌ Can still get blocked
❌ Limited flexibility
❌ Not suitable for large workloads
Using Proxies with DDGS (Optional)
If you want to reduce the chance of blocks:
ddgs text -q "agentic rag" -b duckduckgo
-o output.csv
-pr USER:[email protected]:33335
This routes all traffic through a rotating proxy.
Approach 3: Use Bright Data’s SERP API
For large-scale or highly reliable scraping, an API-based approach is the most effective. Bright Data’s SERP API handles:
- JavaScript rendering
- IP rotation
- Browser fingerprinting
- Blocks and CAPTCHAs
You simply ask for a URL, and the API gives you the fully rendered HTML or structured JSON.
Why This Approach Is Powerful
You can scrape:
- The dynamic DuckDuckGo SERP
- AI-generated “Search Assist” sections
- Infinite scroll or “More Results” content
- High volumes of queries
All without dealing with proxies, browsers, or blocks.
Example Request in Python
import requests
api_key = ""
zone = ""
target = "https://duckduckgo.com/?q=agentic rag"
response = requests.post(
"https://api.brightdata.com/request",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
},
json={
"zone": zone,
"url": target,
"format": "raw"
}
)
if response.status_code == 200:
html = response.text
print(html)
else:
print(f"Error: {response.status_code}")
Who Should Use This Approach
- Businesses
- SEO agencies
- AI developers
- High-scale data collectors
Pros
✅ Reliable
✅ High volume
✅ No blocks
✅ Works with every DuckDuckGo version
Cons
❌ Paid service
Approach 4: Scrape DuckDuckGo Using an MCP Serverfor AI Agents
If you’re working with AI assistants, autonomous agents, or advanced workflows, this approach is designed for you. Bright Data provides a MCP server that exposes a tool called search_engine.
This allows AI tools to fetch SERP data by simply “asking” the MCP server.
Use Cases for MCP
- AI coding assistants
- AI research agents
- Automated workflows
- Multi-agent systems
- Chat-based tools with internet access
How It Works
You install the MCP server and configure it with your API key (using Bright Data here, but choose whichever you want):
{
"mcpServers": {
"Bright Data Web MCP": {
"command": "npx",
"args": ["-y", "@brightdata/mcp"],
"env": {
"API_TOKEN": ""
}
}
}
}
Then your AI environment (such as Claude Code) can access DuckDuckGo SERP data through the MCP interface.
Benefits
✅ No scraping code needed
✅ Ideal for AI integrations
✅ Free tier available
✅ Same reliability as the SERP API
Which Approach Should You Choose?
Press enter or click to view image in full size
Tips to Avoid Blocks When Scraping
DuckDuckGo may block repeated requests from the same IP. To reduce this:
- Use realistic User-Agent strings
- Add delays between requests
- Use rotating proxies
- Use official scraping APIs
- Avoid very high request rates
Final Thoughts
Scraping DuckDuckGo can be simple or complex, depending on the method you select. If you only want to gather small amounts of data, a custom scraper or the DDGS library is usually enough. But if you need large-scale, reliable, and fully automated scraping, a solution like the Bright Data SERP API or the MCP server integration will save time and prevent issues. If you want other options, check my list of the best SERP APIs.
Each approach has strengths, so choose the one that fits your skill level and project needs. When used correctly, DuckDuckGo can be a valuable resource for search data, whether for research, applications, or analysis.
