In this guide, I’ll walk you through the basics of web scraping with TypeScript. We’ll cover everything from setting up your project to tackling more advanced scraping tasks, like handling multiple pages. By the end, you’ll be equipped to create your own TypeScript web scraping scripts easily and confidently. Let’s dive in!

Why Choose TypeScript for Web Scraping?

TypeScript offers several advantages over regular JavaScript, especially for larger projects. Here are some of the key reasons why TypeScript is a great choice for web scraping:

• Strong Typing: TypeScript’s strong typing helps avoid many bugs that often arise in JavaScript. This is particularly important for large scraping projects where much data is processed.
• Code Readability: TypeScript provides type annotations to make the code more readable and easier to maintain. This can save you time when debugging or revisiting the project.
• Compatibility with JavaScript Libraries: TypeScript is fully compatible with JavaScript, meaning you can still use popular JavaScript libraries like Axios and Cheerio for web scraping.

While Python is traditionally the most popular language for web scraping, TypeScript’s type safety and integration with JavaScript libraries make it an excellent choice for developers familiar with the language.

Prerequisites

Before you begin, you’ll need a few things set up on your machine:

Node.js: Ensure that Node.js is installed on your computer. You can download the latest version from the Node.js official website.

TypeScript: You’ll need to install TypeScript globally on your system. You can do so by running the following command in your terminal:

npm install -g typescript

Text Editor/IDE: Use any IDE that supports TypeScript, such as Visual Studio Code.

Once you have everything set up, you can start writing your scraper!

Setting Up Your Project

Create a Project Folder: First, create a new folder for your project. Open your terminal and run:

mkdir web-scraper-typescript
cd web-scraper-typescript

Initialize the Project: Run the following command to set up a new Node.js project:

npm init -y

This command will generate a package.json file.

Initialize TypeScript: Next, initialize TypeScript in your project by running:

npx tsc - init

This will create a tsconfig.json file that contains configuration options for TypeScript.

Install Dependencies: To perform web scraping, we will use two key packages: Axios for making HTTP requests and Cheerio for parsing HTML. Install them using the following commands:

npm install axios cheerio
npm install - save-dev @types/node @types/cheerio

The @types/ packages provide TypeScript definitions for Node.js and Cheerio, enabling code completion and type checking.

Writing Your First Scraper

Now that your environment is set up, it’s time to write your first web scraper. In this example, we will scrape product information from an online store. Here are the steps:

Step 1: Make an HTTP GET Request

The first step is to retrieve the HTML content of the page you want to scrape. We will use Axios to make an HTTP GET request.

import axios from "axios";
async function scrapeSite() {
const response = await axios.get("https://www.example.com");
const html = response.data;
console.log(html);
}
scrapeSite();

In this code:

• axios.get() is used to make the GET request.
• response.data contains the HTML content of the page.

Step 2: Parse the HTML Content

Once we have the HTML, we need to parse it to extract the data. This is where Cheerio comes in. Cheerio is a fast, lightweight HTML parser that mimics jQuery’s syntax.

import axios from "axios";
import { load } from "cheerio";
async function scrapeSite() {
const response = await axios.get("https://www.example.com");
const html = response.data;
const $ = load(html);
// Extract data using Cheerio
const title = $("h1").text();
console.log(title);
}
scrapeSite();

In this code:

• load(html) initializes Cheerio with the HTML content.
• $(“h1”).text() selects the first
  

  element and retrieves its text.

Step 3: Extract Data from Specific Elements

Now that we know how to parse HTML, let’s extract specific product details, such as the name, price, and URL of each product. Suppose each product is within a

import axios from "axios";
import { load } from "cheerio";
async function scrapeSite() {
const response = await axios.get("https://www.example.com/products");
const html = response.data;
const $ = load(html);
$("div.product").each((i, product) => {
const name = $(product).find("h2").text();
const price = $(product).find(".price").text();
const url = $(product).find("a").attr("href");
console.log(`Product Name: ${name}`);
console.log(`Price: ${price}`);
console.log(`URL: ${url}`);
});
}
scrapeSite();

Here:

• $(“div.product”).each() loops through all product elements on the page.
• find() is used to locate specific child elements, such as the product name, price, and URL.

Step 4: Storing Data in an Array

If you want to store the scraped data for further processing (such as exporting to a CSV file), you can push the data into an array. Let’s create a Product type and store the extracted data in an array:

import axios from "axios";
import { load } from "cheerio";
type Product = {
name: string;
price: string;
url: string;
};
async function scrapeSite() {
const response = await axios.get("https://www.example.com/products");
const html = response.data;
const $ = load(html);
const products: Product[] = [];
$("div.product").each((i, product) => {
const name = $(product).find("h2").text();
const price = $(product).find(".price").text();
const url = $(product).find("a").attr("href");
const productData: Product = {
name: name,
price: price,
url: url
};
products.push(productData);
});
console.log(products);
}
scrapeSite();

Step 5: Saving Data to CSV

You can use libraries like fast-csv to save the scraped data to a CSV file. First, install the fast-csv package:

npm install fast-csv

Then, modify your scraper to save the data to a CSV:

import axios from "axios";
import { load } from "cheerio";
import { writeToPath } from "@fast-csv/format";
type Product = {
name: string;
price: string;
url: string;
};
async function scrapeSite() {
const response = await axios.get("https://www.example.com/products");
const html = response.data;
const $ = load(html);
const products: Product[] = [];
$("div.product").each((i, product) => {
const name = $(product).find("h2").text();
const price = $(product).find(".price").text();
const url = $(product).find("a").attr("href");
const productData: Product = {
name: name,
price: price,
url: url
};
products.push(productData);
});
writeToPath("products.csv", products, { headers: true })
.on("error", (error) => console.error(error));
}
scrapeSite();

This script will save the scraped data to a file called products.csv.

Step 6: Scraping Multiple Pages (Pagination)

Many websites have multiple pages of products. To scrape data from multiple pages, you’ll need to navigate through the pagination links. This can be done by checking for a “Next” page link and scraping it.

Here’s how you can scrape multiple pages:

import axios from "axios";
import { load } from "cheerio";
async function scrapeSite() {
let currentPage = 1;
const products: Product[] = [];
while (currentPage <= 5) {
const response = await axios.get(`https://www.example.com/products?page=${currentPage}`);
const html = response.data;
const $ = load(html);
$("div.product").each((i, product) => {
const name = $(product).find("h2").text();
const price = $(product).find(".price").text();
const url = $(product).find("a").attr("href");
const productData: Product = {
name: name,
price: price,
url: url
};
products.push(productData);
});
currentPage  ;
}
writeToPath("products.csv", products, { headers: true })
.on("error", (error) => console.error(error));
}
scrapeSite();

In this example, we use a while loop to scrape five pages of products.

Advanced Techniques

Handling Dynamic Pages with Puppeteer

If the page content is dynamically loaded via JavaScript, Cheerio might not be enough. In such cases, you can use Puppeteer, a headless browser, to scrape the data. Puppeteer can render JavaScript and provide access to the final content.

Install Puppeteer:

npm install puppeteer

Then, you can write a script to scrape data from dynamically rendered pages.

Avoiding Detection and Blocking

Websites often implement anti-scraping measures. To avoid being detected and blocked, consider the following strategies:

• Rotate User Agents: Use different user-agent strings to make your requests look like they’re coming from different browsers.
• Proxy Rotation: Use proxies to hide your IP address. Check out my list of the best rotating proxies.
• Throttling Requests: Limit the rate at which you make requests to avoid triggering rate-limiting systems.

Conclusion

TypeScript is a powerful tool for web scraping, offering the benefits of strong typing, easy integration with JavaScript libraries, and scalability for large projects. By following this step-by-step tutorial, you’ve learned how to set up your environment, write basic scrapers, scrape multiple pages, and save data to a CSV file.

With the basics covered, you can now explore more advanced techniques, such as handling dynamic pages with Puppeteer and avoiding detection using proxies. TypeScript’s robust features will make your web scraping projects more reliable and easier to maintain. Happy scraping!