How to Use Puppeteer Cluster to Scale Up Web Scraping?

In this article, we’ll walk you through how to scale up your web scraping using Puppeteer Cluster, allowing you to handle more tasks at the same time and speed up the entire process. Let’s dive in!

What is Puppeteer Clustering?

Puppeteer clustering helps speed up web scraping by running multiple browser instances, or workers, at the same time. Instead of scraping each URL one after another with a single browser, clustering divides the workload across several workers, allowing them to process tasks concurrently. Each worker picks up a task, scrapes the data, and then moves on to the next one from the shared queue. This approach lets you scrape multiple pages at once, cutting down the time needed to scrape large datasets. Clustering is especially useful when you need to scrape large volumes of data quickly.

Why Use Puppeteer Clustering?

Here are the key benefits of using Puppeteer clustering:

Improved Speed and Efficiency: When scraping large websites or multiple URLs, running a single browser instance can take a long time. Puppeteer clustering allows you to speed up the process by running several browser instances concurrently. This reduces the total time required to scrape all the pages.
Resource Optimization: Running multiple browser instances concurrently ensures that you make the best use of available resources. Rather than running a single instance per task, which can be inefficient, clustering enables you to use resources optimally by sharing the workload among multiple workers.
Better Load Distribution: In cases where websites are slow or resource-heavy, clustering can help distribute the workload evenly across several workers. This ensures that no single worker gets overloaded, and tasks are completed as quickly as possible.
Avoiding Detection: When scraping multiple pages from the same website, clustering helps you distribute tasks across different browser instances, which can make it harder for websites to detect scraping behavior. Each worker can use its own set of cookies, user agents, and sessions, reducing the likelihood of being flagged as a bot.

Setting Up Puppeteer Cluster

To begin using Puppeteer clustering, you’ll need to install the necessary dependencies. These include both Puppeteer and the Puppeteer Cluster library.

Install Puppeteer and Puppeteer-Cluster: First, install the required libraries by running the following command:

npm install puppeteer puppeteer-cluster

Import the Libraries: After installation, import the necessary libraries into your script. You’ll need the puppeteer-cluster library to handle the clustering functionality:

const { Cluster } = require('puppeteer-cluster');

Launch the Cluster: Once the libraries are imported, you can launch the cluster. The Cluster.launch function allows you to configure several parameters, such as the concurrency model (how the workers operate), the maximum number of workers to run concurrently, and other cluster settings. For example, to create a cluster with three browser instances, you can use the following code:

const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_BROWSER, // Use separate browser instances
maxConcurrency: 3, // Limit the number of concurrent browser instances
});

How Puppeteer Cluster Works?

The puppeteer-cluster library provides an easy way to manage a pool of workers that scrape websites concurrently. Each worker is responsible for executing a task, which may include navigating to a URL, scraping data from a page, or interacting with the page. Here’s how the process works:

Task Queues: When you launch the cluster, you define a task that each worker will execute. A task could involve navigating to a specific URL and scraping some data. Tasks are added to the cluster’s task queue, and each worker picks up a task when it’s free.

Concurrency Models: There are three concurrency models in Puppeteer Cluster, each suited for different use cases:

CONCURRENCY_BROWSER: This model runs each worker in a separate browser instance. It’s ideal when you need complete isolation between tasks, such as when each task requires its own set of cookies, user agents, or proxies.
CONCURRENCY_CONTEXT: This model runs each worker in a separate browser context within the same browser instance. It allows you to share browser resources like user agents or cookies but still have some isolation between tasks.
CONCURRENCY_PAGE: This model runs each worker in the same browser instance and context, sharing resources like cookies, session data, and local storage. It’s the most memory-efficient but should be used when scraping different domains.

Running Tasks Concurrently: Once the cluster is set up, you can add tasks to the queue using cluster.queue(url). Each worker picks a task from the queue, processes it, and moves to the next one. This allows the tasks to run concurrently, significantly reducing the time required to scrape data.

Closing the Cluster: After all tasks are processed, you can call cluster.close() to close the cluster and release any resources used by the workers.

Example: Scraping Multiple URLs Concurrently

Here’s a simple example of how you can scrape multiple pages concurrently using Puppeteer Cluster. In this case, we will scrape the titles of different pages:

const { Cluster } = require('puppeteer-cluster');
(async () => {
// Launch the cluster with three browser instances
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_BROWSER,
maxConcurrency: 3,
});
// Define the task to scrape page titles
await cluster.task(async ({ page, data: url }) => {
await page.goto(url);
console.log(await page.title());
});
// Queue up multiple URLs to scrape
const URLs = [
'https://example.com/page1',
'https://example.com/page2',
'https://example.com/page3',
];
URLs.forEach((url) => cluster.queue(url));
// Wait for all tasks to finish
await cluster.idle();
// Close the cluster
await cluster.close();
})();

In the code above, three browser instances are launched concurrently. Each worker executes the task of navigating to a URL and logging the page title. Each URL is added to the queue, and the workers pick up the tasks and process them concurrently, significantly reducing the time needed to scrape all three pages.

Advanced Cluster Features

Puppeteer Cluster also offers several advanced features that make it more powerful and flexible. These options help you manage large-scale web scraping tasks more effectively:

1. Retry Limit and Delay: If a task fails (for example, due to a network error or page load failure), you can set a retry limit and delay to give the task another chance to succeed.

const cluster = await Cluster.launch({
retryLimit: 3, // Maximum retries per task
retryDelay: 2000, // Delay between retries (in milliseconds)
});

2. Timeouts: You can set a global timeout for tasks, ensuring that tasks that take too long are automatically stopped.

const cluster = await Cluster.launch({
timeout: 6000, // Timeout for each task (in milliseconds)
});

3. Skip Duplicate URLs: To avoid processing the same URL multiple times, you can enable the skipDuplicateUrls option.

const cluster = await Cluster.launch({
skipDuplicateUrls: true, // Prevent duplicate URLs from being processed
});

4. Worker Creation Delay: If you need to control the rate at which workers are created, you can introduce a delay between each worker’s creation.

const cluster = await Cluster.launch({
workerCreationDelay: 100, // Delay between worker creations (in milliseconds)
});

Scaling Web Scraping with Cloud Solutions

While Puppeteer Cluster can help you scale your scraper on a local machine, running too many browser instances concurrently may lead to memory consumption issues. To overcome these challenges, you can use a cloud-based proxy and browser management solution like Bright Data. Bright Data offers a powerful infrastructure that enables you to scale your web scraping operations effortlessly.

By connecting your Puppeteer scraper to a cloud-based solution like Bright Data, you can handle thousands of concurrent requests without worrying about system limitations. Bright Data offers rotating proxies, geo-targeting, and the ability to scale scraping operations easily across multiple cloud nodes, making it an ideal choice for large-scale scraping projects.

Here’s an example of integrating Bright Data’s Proxy Manager with your Puppeteer Cluster scraper:

const puppeteer = require('puppeteer-core');
const { Cluster } = require('puppeteer-cluster');
(async () => {
// Set up Bright Data's Proxy Manager
const browser = await puppeteer.connect({
browserWSEndpoint: 'wss://your-proxy-manager-endpoint',
});
// Launch the cluster with the Bright Data connection
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_BROWSER,
maxConcurrency: 3,
browser,
});
// Define scraping tasks
await cluster.queue('https://example.com', async ({ page, data }) => {
await page.goto(data);
console.log(await page.title());
});
// Close the cluster
await cluster.idle();
await cluster.close();
await browser.close();
})();

Conclusion

Puppeteer Cluster is a powerful tool for scaling web scraping tasks by running multiple browser instances concurrently. By using clustering, you can significantly improve the speed and efficiency of your scraper, handle large amounts of data, and reduce the risk of detection by websites. Whether you’re scraping data for research, analysis, or business intelligence, Puppeteer Cluster provides a simple and effective way to scale your web scraping operations.

For large-scale scraping tasks, consider using a cloud-based solution like Bright Data or Firecrawl to manage high volumes of requests and scale your scraper without running into system limitations.