How to Scrape Google’s “People Also Ask” Using Python

In this article, I’ll walk youthrough how to scrape PAA using Python, extract valuable information, and store the data for analysis. You’ll need basic Python knowledge, including the requests library and BeautifulSoup. I’ll also cover tips to avoid getting blocked by Google while scraping.

What is Google’s People Also Ask?

Google’s “People Also Ask” feature is helpful for search results pages. It displays questions related to a user’s initial search. Each question expands with a brief answer, offering quick insights into related topics. This is great for SEO because it helps identify popular questions that audiences frequently look up.

Best Paid Solutions

To collect this data automatically, you can use tools like Bright Data’s SERP API or Google Search Autocomplete API for gathering search suggestions. You can also link these tools with Google Sheets for easy organization and analysis.

Check out my article of the best 5 SERP APIs. I am not affiliated with any of the services listed in that article.

Step 1: Setting Up the Environment

To begin, you’ll need to set up a Python environment and install the necessary libraries. This tutorial requires requests for handling HTTP requests and BeautifulSoup for parsing HTML content. Open your command prompt or terminal and enter the following command:

pip install requests beautifulsoup4

Once installed, these libraries allow you to request Google’s search results page and parse the HTML for PAA questions.

Step 2: Connecting to Google’s Search Results Page

Create a file called main.py in your project folder. Then, import the required libraries and set up a primary Google search function.

Here’s how to build the function:

import requests
from bs4 import BeautifulSoup
def google_search(query):
query = query.replace(' ', '+')
url = f"https://www.google.com/search?q={query}"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}
response = requests.get(url, headers=headers)
if response.status_code == 200:
print("Connection successful!")
return BeautifulSoup(response.text, "html.parser")
else:
print(f"Error: Unable to fetch the search results. Status code: {response.status_code}")
return None

This code snippet sets up a function called google_search that takes a query as input. It formats the query for use in a URL and sets the appropriate headers to mimic a browser request. If the connection is successful, the HTML content is parsed with BeautifulSoup.

Step 3: Locating the PAA Questions

With the HTML page ready, the next step is identifying and extracting the PAA questions. By inspecting Google’s page source, you’ll notice that PAA questions are often contained in a specific HTML class. Let’s build a function to locate and extract the questions.

Add the following code below the previous snippet in main.py:

def extract_questions(soup):
questions = []
if soup:
for question in soup.select('span.CSkcDe'):
questions.append(question.get_text())
return questions

In this function, soup.select is used to locate all elements with the span.CSkcDe class, which is often the container for PAA questions. Adjustments might be necessary if Google’s HTML structure changes, so always verify the page source if you encounter issues.

Step 4: Saving the Results

After extracting the questions, it’s essential to save the data for further analysis. JSON is a suitable format for storing structured data because it’s easily readable and can be used in data pipelines or integrated with analytics tools.

Here’s how to save the results with a timestamp:

import json
import os
from datetime import datetime
def save_results(query, questions):
results = {
"date": datetime.now().strftime("%Y-%m-%d"),
"query": query,
"questions": questions
}
if os.path.exists("results.json"):
with open("results.json", "r", encoding="utf-8") as file:
data = json.load(file)
else:
data = []
data.append(results)
with open("results.json", "w", encoding="utf-8") as file:
json.dump(data, file, indent=4)
print("Results saved to results.json")

This function checks if a results.json file exists. If it does, the function appends new results; otherwise, it creates a new file. Each entry includes the current date, query, and questions list. By structuring the data, you can analyze historical trends over time.

Step 5: Running the Script

With the foundational functions complete, you can combine them to create the final product. Here’s the complete script:

def main(query):
soup = google_search(query)
questions = extract_questions(soup)
save_results(query, questions)
query = "how to start a blog"
main(query)

This script will perform a Google search, extract, and save PAA questions. Running main.py in your terminal will output the results, which you can view in the results.json file.

Additional Tips for Successful Scraping

1. Avoid Overloading Google’s Servers: Google’s anti-bot measures can block excessive requests, so introduce delays between requests. Consider using time.sleep() to add intervals.
2. Rotate User Agents and Proxies: Using a single user agent or IP for multiple requests can result in blocks. Consider using rotating proxies and varying user-agent strings to mimic genuine user behavior.
3. Set Up Regular Runs: Since PAA questions change over time, automating this script to run periodically can reveal evolving search trends. Scheduling with a tool like cron (for Linux/macOS) or Task Scheduler (for Windows) is highly effective.
4. Consider Third-Party Solutions: For high-volume or specialized needs, using APIs like Bright Data’s SERP API or Google Search Autocomplete API can streamline and simplify data gathering, especially for larger projects.

Conclusion

Scraping Google’s “People Also Ask” (PAA) section using Python provides valuable insights into what users are searching for, helping create more targeted content and improve SEO. With the Python code in this guide, you can automate data collection from the PAA section, store it in an organized format, and track trends over time.

Adding APIs and other tools to your setup can make the data even more useful, especially for large-scale projects. Follow Google’s usage policies while scraping, proxies, and rotating user agents to avoid blocks and maintain request limits. Over time, this approach can build a valuable dataset that informs content strategy, keeps pace with trending topics, and boosts SEO.