Distributed Web Crawling Guide: System & Architecture

In this article, I’ll show you how to build a basic distributed web crawling system using Python, Celery, and Redis. I’ll walk you through the process of setting up the architecture, distributing tasks across multiple nodes, and ensuring that the crawler runs efficiently. So, you’ll have a scalable solution for scraping websites, even as the volume of data increases. Let’s dive in and build this together!

What is Distributed Web Crawling?

Web crawling involves automatically browsing the web, extracting useful data from different websites, and following links on those sites to gather more data. On the other hand, distributed web crawling takes this a step further by distributing the crawling process across multiple machines or nodes to scale the process and improve efficiency.

In a distributed system, the workload is divided between several computers, each working on a subset of tasks. By using this approach, we can crawl large numbers of pages in a shorter timeframe and handle massive amounts of data. This is essential for large-scale web scraping operations, such as those used in e-commerce analysis, news aggregation, and market research.

Key Technologies for Building a Distributed Web Crawler

Before we dive into building a distributed web crawling system, let’s discuss the essential technologies that we will be using:

1. Python: A popular programming language known for its simplicity and extensive library support. Python is ideal for web scraping, and it provides powerful libraries, such as requests and BeautifulSoup, for parsing HTML.
2. Celery: An open-source distributed task queue system that allows us to process tasks asynchronously. Celery enables us to distribute the crawling tasks to multiple workers or machines, making the web crawling process more scalable and efficient.
3. Redis: A high-performance in-memory data store used as a message broker for Celery. Redis helps manage the queue of URLs to crawl, store visited links, and keep track of the crawling progress.

Prerequisites for Building the Distributed Crawler

Before building the distributed web crawling system, there are a few prerequisites you need to have installed on your machine:

• Python 3: Make sure Python 3 is installed on your system.
• Redis: Install Redis, which will be used as the message broker.
• Celery and other dependencies: Install the necessary libraries for web scraping and asynchronous task handling.

You can install these dependencies using pip as follows:

pip install requests beautifulsoup4 celery[redis] playwright
npx playwright install

Step 1: Setting Up Celery and Redis

The first step in building the distributed web crawler is to configure Celery and Redis.

Celery will be used to distribute the tasks, while Redis will act as the message broker, keeping track of the URLs that need to be crawled. Here’s how to configure Celery:

Create a Celery Application

We will define a simple Celery application to handle tasks asynchronously. In the tasks.py file, we will initialize the Celery app and configure it to use Redis as the broker.

from celery import Celery
app = Celery('tasks', broker_url='redis://127.0.0.1:6379/1')
@app.task
def demo(str):
print(f'Str: {str}')

Start the Celery Worker

To process tasks, you need to start the Celery worker by running the following command in the terminal:

celery -A tasks worker - loglevel=info

This will start a worker that listens for tasks in the queue and processes them as they come in.

Step 2: Creating the Crawling Task

Once Celery is set up, the next step is to create a task for crawling URLs. This task will fetch the HTML content of a URL, extract the links from the page, and enqueue them for further processing.

Here’s how you can create a basic crawling task:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
@app.task
def crawl(url):
html = get_html(url)
soup = BeautifulSoup(html, 'html.parser')
links = extract_links(url, soup)
print(links)
def get_html(url):
try:
response = requests.get(url)
return response.content
except Exception as e:
print(e)
return ''
def extract_links(url, soup):
return list({
urljoin(url, a.get('href'))
for a in soup.find_all('a')
if a.get('href') and not(a.get('rel') and 'nofollow' in a.get('rel'))
})

This crawl task fetches the HTML of a webpage, parses it using BeautifulSoup, and extracts all the links. It then prints the list of links found on the page.

Step 3: Using Redis to Track URLs

To prevent the crawler from visiting the same page multiple times, we will use Redis to keep track of URLs. We will store URLs in a Redis list called crawling:to_visit, and the crawler will pop URLs from this list one by one for processing.

We will also maintain two sets in Redis: one for visited URLs and one for queued URLs. This will help prevent duplicate processing of URLs.

Here’s how you can modify the crawler to use Redis for tracking URLs:

from redis import Redis
connection = Redis(db=1)
starting_url = 'https://example.com/start-page'
# Push the starting URL to Redis
connection.rpush('crawling:to_visit', starting_url)
while True:
item = connection.blpop('crawling:to_visit', 60)
if item is None:
print('Timeout! No more items to process')
break
url = item[1].decode('utf-8')
print('Pop URL', url)
crawl.delay(url)

Step 4: Ensuring Scalability with Multiple Workers

Once the basic crawling system is in place, the next step is to distribute the workload across multiple workers. Celery allows you to run multiple workers on different machines or processes, making it easier to scale the crawling process.

To start multiple workers, you can run the following commands:

celery -A tasks worker - concurrency=20 -n worker1
celery -A tasks worker - concurrency=20 -n worker2

Each worker will process tasks independently, enabling the crawler to scale across multiple nodes. Celery handles distributing tasks to available workers, ensuring an even workload distribution.

Step 5: Customizing the Crawler for Different Websites

To make the crawler more flexible, you can customize the way it extracts data from different websites. Instead of hardcoding the data extraction logic in the crawl function, you can create a modular system that allows different parsers for different websites.

In the repo.py file, you can define a function to get a custom parser for each website:

from urllib.parse import urlparse
from parsers import defaults
parsers = {
'example.com': defaults,
'quotes.com': custom_parser,
}
def get_parser(url):
hostname = urlparse(url).hostname
if hostname in parsers:
return parsers[hostname]
return defaults

Then, in the crawl task, you can use the appropriate parser for each website:

@app.task
def crawl(url):
parser = get_parser(url)
html = parser.get_html(url)
soup = BeautifulSoup(html, 'html.parser')
links = extract_links(url, soup)
print(links)

Step 6: Avoiding Detection with Proxies and Headers

Websites often block crawlers by detecting unusual traffic patterns. To avoid getting blocked, you can rotate proxies and use custom headers for your requests. You can create a headers.py file to store different sets of headers and a proxies.py file for managing proxies.

Here’s an example of how to use random headers and proxies in the get_html function:

from headers import random_headers
from proxies import random_proxies
def get_html(url):
headers = random_headers()
proxies = random_proxies()
response = requests.get(url, headers=headers, proxies=proxies)
return response.content

Step 7: Handling Errors and Retries

Web scraping at scale often encounters errors, including timeouts, missing pages, or request failures. It’s essential to implement error handling and retry mechanisms to ensure the crawler keeps running smoothly.

Celery provides a retry feature that allows you to retry failed tasks automatically. Here’s an example of how to implement retries in your crawl task:

@app.task(bind=True, max_retries=3)
def crawl(self, url):
try:
html = get_html(url)
soup = BeautifulSoup(html, 'html.parser')
links = extract_links(url, soup)
print(links)
except Exception as e:
print(f'Error: {e}')
raise self.retry(exc=e)

Looking for an Easy & Scalable Solution?

Building your own distributed crawler lets you learn the technology in detail and adjust everything to your needs. However, if you ever need to collect data at an even larger scale or want to simplify some of the operational challenges, managed services like Bright Data or Firecrawl can be helpful. These platforms handle the underlying infrastructure and technical obstacles, allowing you to dedicate more time to working with your collected data.

Conclusion

Using Celery and Redis to build a distributed web crawler helps you scrape websites at scale, efficiently managing tasks across multiple workers. This approach allows you to handle large amounts of data and customize your crawler for different websites.

It also helps tackle challenges like proxies, headers, and retries. As web scraping becomes increasingly complex, a distributed system becomes essential for achieving optimal performance and scalability. By leveraging Celery and Redis, you can create a robust and reliable crawler that can handle millions of URLs quickly and accurately.