Web Scraping in C++ with libxml2 and libcurl
Why Use C for Web Scraping?
C is known for its high-performance capabilities. Its speed makes it a suitable choice for tasks requiring rapid data extraction and processing, especially when dealing with large datasets. While languages like Python offer simple libraries for web scraping, they may not match the raw performance C offers.
The key benefits of using C for web scraping are:
- Speed: C is faster than higher-level languages like Python, making it an ideal choice when speed is a critical factor.
- Control over Resources: C allows you to manage system resources (like memory) directly, which can help optimize performance.
- Scalability: C can handle large-scale web scraping operations efficiently, making it ideal for extracting data from multiple sources or large websites.
Tools Needed
To build a web scraper in C , you will need to install a few libraries that will help with HTTP requests and HTML parsing.
- libcurl: This is a powerful and widely used library for making HTTP requests. It supports multiple protocols, including HTTP, HTTPS, and FTP.
- libxml2: This library provides tools for parsing XML and HTML documents. It supports XPath and XSLT, making it an excellent choice for scraping data from websites.
- C Compiler: A standard C compiler like g (GCC) is required to compile the code.
Setting Up the Development Environment
Before we begin coding, you’ll need to set up your development environment by installing the required libraries.
Using vcpkg (for Windows, macOS, and Linux)
vcpkg is a cross-platform package manager that simplifies the installation of libraries like libcurl and libxml2.
Install vcpkg on your system. Follow the instructions on vcpkg’s GitHub page.
Once installed, open your terminal and run the following commands to install the libraries:
vcpkg install curl libxml2
For Visual Studio users, run the following command to integrate vcpkg with MSBuild:
vcpkg integrate install
Using apt (for Linux)
On Debian-based systems like Ubuntu, you can install libcurl and libxml2 directly using the package manager:
sudo apt-get install libcurl4-openssl-dev libxml2-dev
HTTP Basics: How Web Scraping Works
Web scraping begins with sending an HTTP request to a server. The server responds with an HTTP response containing the desired data, usually in the form of an HTML document. The HTTP protocol is crucial for web scraping as it defines how requests and responses are handled.
Get Data Journal’s stories in your inbox
When scraping a website, you’ll typically need to make a GET request, which fetches data from the server. Here’s an example of how a request to a website might look:
GET /dictionary/esoteric HTTP/1.1
Host: www.merriam-webster.com
User-Agent: curl/8.9.1
Accept: */*
Once the server receives this request, it returns an HTTP response with a status code (e.g., 200 OK) and the requested content, which is usually an HTML page.
Building the Web Scraper
Now that we understand the basics of HTTP, let’s dive into building the web scraper.
Step 1: Initialize Your Project
First, create a directory for your project:
mkdir scraper
cd scraper
touch scraper.cc
This will be the main file where we will write the code for our scraper.
Step 2: Writing the Main Function
In C , every program starts execution with the main() function. Here, we will write the main function that handles input, calls the request and scrape functions, and outputs the result.
#include
#include
#include
#include <curl/curl.h>
#include <libxml/HTMLparser.h>
#include <libxml/xpath.h>
int main(int argc, char **argv)
{
if (argc != 2)
{
std::cout << "Please provide a valid English word" << std::endl;
return EXIT_FAILURE;
}
std::string word = argv[1];
std::string page_content = request(word);
std::cout << scrape(page_content) << std::endl;
return EXIT_SUCCESS;
}
This function checks whether a word is passed as a command-line argument and calls the request() and scrape() functions.
Step 3: Request Function: Fetching HTML Content
To fetch the HTML content from the website, we will use libcurl. The request() function sends an HTTP request to the website and retrieves the HTML content.
std::string request(std::string word)
{
CURLcode res_code = CURLE_FAILED_INIT;
CURL *curl = curl_easy_init();
std::string result;
std::string url = "https://www.merriam-webster.com/dictionary/" word;
if (curl)
{
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_USERAGENT, "simple scraper");
// Function to handle the response
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
static_cast<size_t (*)(char *, size_t, size_t, std::string *)>(
[](char *contents, size_t size, size_t nmemb, std::string *data) -> size_t
{
size_t new_size = size * nmemb;
data->append(contents, new_size);
return new_size;
}));
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &result);
// Perform the request
res_code = curl_easy_perform(curl);
if (res_code != CURLE_OK)
{
return curl_easy_strerror(res_code);
}
curl_easy_cleanup(curl);
}
return result;
}
This function initializes the libcurl object, sends the request, and retrieves the HTML content of the word’s dictionary page.
Step 4: Scrape Function: Extracting Data Using XPath
Now that we have the HTML content, we need to extract the relevant data (the word definitions) using libxml2 and XPath.
std::string scrape(std::string markup)
{
std::string res = "";
// Parse the HTML content
htmlDocPtr doc = htmlReadMemory(markup.c_str(), markup.length(), NULL, NULL, HTML_PARSE_NOERROR);
xmlXPathContextPtr context = xmlXPathNewContext(doc);
// XPath to find the definitions
xmlXPathObjectPtr result = xmlXPathEvalExpression((xmlChar *)"//div[contains(@class, 'vg-sseq-entry-item')]/div[contains(@class, 'sb')]//span[contains(@class, 'dtText')]", context);
if (result->nodesetval == NULL)
{
return "No definitions found.";
}
for (int i = 0; i < result->nodesetval->nodeNr; i)
{
char *def = (char *)xmlNodeGetContent(result->nodesetval->nodeTab[i]);
res = def;
res = "n";
free(def);
}
xmlFreeDoc(doc);
xmlCleanupParser();
return res;
}
In this function, we parse the HTML content, evaluate the XPath expression to find the definitions, and store the results in a string.
Step 5: Helper Function to Convert to Lowercase
We also need a helper function to convert the word to lowercase before sending the request to the website.
std::string strtolower(std::string str)
{
std::transform(str.begin(), str.end(), str.begin(), ::tolower);
return str;
}
This function simply converts the input string to lowercase.
Compilation and Running the Scraper
Now that the code is ready, it’s time to compile and run the scraper. Use the following command to compile your code:
g scraper.cc -lcurl -lxml2 -std=c 11 -o scraper
After compiling, run the scraper with a word of your choice. For example:
./scraper esoteric
You should see the dictionary definitions for the word “esoteric” output to the console.
The Best AI Web Scraping Tools
Looking for more options to boost your data collection with automation or artificial intelligence? Explore the best AI web scraping tools available today. For a helpful breakdown, check out the best AI web scraping tools and see which platforms stand out, like Bright Data and ScrapingBee for example.
Conclusion
In this tutorial, we have explored how to build a custom web scraper in C using libcurl and libxml2. While Python may be more commonly used for web scraping, C offers significant advantages in performance and scalability, especially for large-scale scraping tasks. We have also demonstrated how to fetch HTML content from a website, parse it, and extract relevant data using XPath.
Although this example focused on a relatively simple use case, C web scraping can be expanded to handle more complex scenarios, such as dealing with JavaScript, using proxies, or avoiding anti-scraping measures. By leveraging the power of C , you can create efficient and scalable web scrapers for any data extraction task.
