How to Use ScrapydWeb for Managing Scrapy Projects

In this article, I’ll walk you through how to set up and use ScrapydWeb to deploy your spiders, monitor their performance, and schedule tasks — all from a single place. From installation to scaling your projects, I’ll cover everything you need to know to make your workflow smoother and more efficient.

What is ScrapydWeb?

ScrapydWeb is a web interface designed for managing Scrapyd clusters. Scrapyd itself is a powerful tool used for running and managing Scrapy spiders in a distributed environment. While Scrapyd runs in the background, handling the execution of spiders on different servers, ScrapydWeb offers a user-friendly interface to interact with the Scrapyd server. With ScrapydWeb, you can easily monitor, control, and manage your Scrapy spiders in real time, directly from your browser.

Why Use ScrapydWeb?

While the CLI interface of Scrapyd is useful, it can be cumbersome, especially when you’re managing multiple servers and spiders. ScrapydWeb provides the following benefits:

• Centralized management: Manage all your Scrapy projects from a single interface, no matter how many Scrapyd servers you have.
• Real-time monitoring: View real-time job statuses, logs, and statistics directly from the web interface.
• Scheduling: Schedule your spiders to run at specific times and intervals.
• Multi-server management: Manage multiple Scrapyd nodes (servers) within a single ScrapydWeb instance.

Key Features of ScrapydWeb

ScrapydWeb offers several features that make it easier to manage Scrapy projects:

• Task Monitoring: View real-time updates on your scraping jobs, including logs and performance metrics.
• Multi-Node Cluster Management: Manage multiple Scrapyd servers from a single interface, whether they are running locally or remotely.
• Scheduled Scraping Jobs: Schedule spiders to run at specific times and frequencies.
• Mobile Mode: ScrapydWeb allows you to manage servers from mobile devices, making it accessible on the go.
• Detailed Job Statistics: Access detailed logs and statistics for each job, project, and server.
• Alerts: Set up email, Slack, or Telegram notifications to get alerts on job status, errors, and performance.

Setting Up ScrapydWeb

Before you can start using ScrapydWeb, you need to set up both Scrapyd and ScrapydWeb on your system. Below is a step-by-step guide to get you started.

Step 1: Install Scrapyd and ScrapydWeb

ScrapydWeb requires Python 3 to run smoothly. Make sure Python is installed on your system before proceeding.

To install Scrapyd, Scrapyd-Client (CLI tool), and ScrapydWeb, you can use pip:

pip3 install scrapyd scrapyd-client scrapydweb

Once installed, you can proceed to the next step.

Step 2: Deploy Your Scrapy Project to Scrapyd

Now, you need to deploy your Scrapy project to the Scrapyd server. Start by running Scrapyd:

scrapyd

By default, Scrapyd runs on port 6800. You can access Scrapyd by visiting http://localhost:6800 in your browser.

Next, connect your Scrapy project to the Scrapyd server. Open the scrapy.cfg file in your project’s root directory and modify it as follows:

[settings]
default = product_scraper.settings
[deploy:local]
url = http://localhost:6800/
project = scraper

Replace product_scraper.settings with the settings for your Scrapy project and make sure the url is set to the Scrapyd server’s address (localhost:6800).

Deploy your project using the Scrapyd-Client:

scrapyd-deploy local -p scraper

After deployment, you should see your Scrapy project listed on the Scrapyd web interface (http://localhost:6800/).

Step 3: Start ScrapydWeb Server

To start ScrapydWeb, run the following command:

scrapydweb

This will start the ScrapydWeb server on http://localhost:5000/ by default. Open this URL in your browser, and you should see the ScrapydWeb interface.

Step 4: Configure ScrapydWeb

After starting the ScrapydWeb server, you need to configure it to connect to your Scrapyd server. Open the scrapydweb_settings_v11.py file in your project’s root directory and update the SCRAPYD_SERVERS list with your Scrapyd server’s address:

SCRAPYD_SERVERS = [
"127.0.0.1:6800",
]

Once done, restart the ScrapydWeb server. You can now manage your Scrapyd server from the ScrapydWeb interface.

Managing Multiple Scrapyd Servers

As your scraping needs grow, you might want to scale and use multiple Scrapyd servers. ScrapydWeb makes it easy to manage several servers in one place.

To set up another Scrapyd server on a different port, follow these steps:

Create a new scrapyd.conf file in the second project’s root directory, specifying a new port (e.g., 6802):

[scrapyd]
http_port = 6802

Update the scrapy.cfg file in the new project to point to this new port:

[settings]
default = product_scraper.settings
[deploy:local]
url = http://localhost:6802/
project = product_scraper

Deploy the new project using Scrapyd-Client:

scrapyd-deploy local -p product_scraper

Update the scrapydweb_settings_v11.py file in the first project to add the new server:

SCRAPYD_SERVERS = [
"127.0.0.1:6800",
"127.0.0.1:6802",
]

Now, when you open ScrapydWeb, you’ll see both servers listed. You can manage and deploy spiders to both servers from the same interface.

Running and Scheduling Scrapy Spiders

With ScrapydWeb, you can easily run and schedule spiders on different servers.

Running a Spider

To run a spider, follow these steps:

1. Open ScrapydWeb (http://localhost:5000/) and go to the Servers tab.
2. Select the Scrapyd server that contains the desired project.
3. Click on the Run Spider tab and select the spider you wish to run.
4. You can specify additional options such as the user-agent, cookies, and concurrency by toggling on the settings & arguments switch.
5. Click Check CMD, and then click Run Spider to execute the spider.

You’ll see the spider’s status change to “running” in ScrapydWeb.

Scheduling a Spider

To schedule a spider to run at a specific time or interval:

1. Select the Scrapyd server and project from the Servers tab.
2. Click on the Run Spider tab and select the spider.
3. Toggle on the timer task switch to set a schedule.
4. You can specify the start and end dates for the spider, as well as the interval at which it should run.
5. Click Check CMD, then click Add Task to schedule the spider.

Viewing Spider Logs and Stats

ScrapydWeb provides easy access to spider logs and statistics. You can monitor the performance and status of your scraping jobs directly through the interface.

To view logs and stats:

1. 1. Go to the Logs section on the left sidebar.
   2. Select the project and spider you want to check.
   3. You’ll see the spider’s execution logs, along with detailed statistics on the crawl.

A Powerful Alternative for Enterprise-Scale Scraping

While ScrapydWeb is excellent for managing self-hosted Scrapy clusters, sometimes you need a solution that’s truly scalable, fully managed, and can handle advanced challenges like IP rotation and global geo-targeting.

For example, I once needed to scrape thousands of product listings from a well-known international marketplace while avoiding IP bans and ensuring data quality. Using Bright Data’s Scraper API, I was able to seamlessly rotate IPs, solve CAPTCHAs, and pull clean, structured data — without worrying about infrastructure or local deployment. This saved me a tremendous amount of time and let me focus entirely on the data extraction logic.

If your projects are growing in complexity or volume, Bright Data offers an excellent, reliable alternative that’s ideal for large, high-frequency scraping tasks and compliance-sensitive use cases.

NOT AFFILIATED WITH ANY OF THE MENTIONED PROVIDERS!

Conclusion

ScrapydWeb is a handy tool for managing Scrapy projects and Scrapyd servers. It simplifies the process of deploying, monitoring, and scheduling spiders with an easy-to-use web interface. Whether you’re working with one server or many, ScrapydWeb helps you keep track of your projects and ensures your scraping tasks run smoothly.

With ScrapydWeb, scaling your web scraping projects becomes much easier. Its real-time monitoring, scheduling options, and detailed logs make it an essential tool for anyone using Scrapy, saving you time and effort in managing multiple servers and tasks.