How to Scrape a Website to Markdown: 2026 Guide

In this guide, I’ll walk you through the process of scraping websites and converting the content into Markdown. I’ll cover both static and dynamic websites and explain the tools and techniques I use to get the job done. Whether you’re new to web scraping or just looking for a better way to organize data, this guide will give you the knowledge you need to get started!

What Does it Mean to Scrape a Website to Markdown?

When we talk about scraping a website to Markdown, we mean parsing the HTML content of a web page and converting it into Markdown format. Markdown is a simple text-based format that preserves the structure of the content while being much easier to read and edit compared to raw HTML. It can handle text, headings, links, images, and other media in a cleaner format.

For example, let’s say you’re scraping a news website. The content you get in HTML format might look something like this:

<div class="article">
  <h1>Breaking News: Web Scraping Made Easy</h1>
  <p>Today, we discuss how scraping can be a powerful tool.</p>
  <img src="image.jpg" alt="Web Scraping">
</div>

When converted into Markdown, it will look like this:

# Breaking News: Web Scraping Made Easy
Today, we discuss how scraping can be a powerful tool.
![Web Scraping](image.jpg)

As you can see, Markdown simplifies the structure, making it easy to read and edit.

Why Convert HTML to Markdown?

When scraping websites, HTML is the default format used to retrieve data from the web. However, there are several reasons why converting HTML to Markdown is a better approach for many use cases:

• Readability: Markdown is much easier to read than HTML. It’s designed to be simple and visually straightforward.
• Compactness: Markdown is more compact than HTML. It removes unnecessary tags, reducing token usage in AI and text-processing tools.
• LLM Compatibility: Large language models (LLMs) such as GPT understand Markdown better than raw HTML, making it the ideal format for AI processing.
• Cross-platform Support: Markdown can be rendered across platforms, making it a universal format for applications such as blog posts, documents, and codebases.

Step-by-Step Guide to Scrape a Website to Markdown

Scraping a website to Markdown involves converting HTML content into a clean, readable format for easy processing. Here are the essential steps to scrape both static and dynamic websites.

Step 1: Understand the Different Types of Websites

Before you start scraping, it’s important to know the difference between static and dynamic websites:

• Static Websites: These websites are straightforward and provide a fixed HTML file from the server. When you request a page, the HTML is sent to you as it is, without any need for further processing.
• Dynamic Websites: These websites rely on JavaScript to render parts of the page after the initial HTML is delivered. They load content via AJAX or similar methods, so the page is dynamically generated in the browser rather than by the server.

Step 2: Scraping Static Websites

Scraping static websites is relatively simple. Since the content is readily available in the HTML, you can use a basic HTTP request to retrieve the page’s source code. Here’s a simple step-by-step process using Python:

1. Make an HTTP Request

The first step is to make a GET request to the website’s URL. You can use libraries like requests in Python to do this:

import requests
# URL of the static page
url = 'https://example.com/page'
# Fetch the HTML content of the page
response = requests.get(url)
# Check if the request was successful
if response.status_code == 200:
    # Get the HTML as a string
    html_content = response.text
else:
    print(f"Failed to fetch page: {response.status_code}")

2. Convert HTML to Markdown

Once you have the HTML content, you can use a library like markdownify to convert it to Markdown format. Install it using pip:

pip install markdownify

Now, you can convert the HTML:

from markdownify import markdownify as md
# Convert HTML to Markdown
markdown_content = md(html_content)
# Print the result
print(markdown_content)

This will convert raw HTML into clean, readable Markdown.

3. Save the Markdown Content

You can save the resulting Markdown content to a .md file:

with open('page.md', 'w', encoding='utf-8') as file:
    file.write(markdown_content)

This is the basic method for scraping a static website and converting it to Markdown. It’s simple, effective, and works well for websites that don’t use JavaScript to load content.

Step 3: Scraping Dynamic Websites

Dynamic websites require a bit more effort because the content is not included in the initial HTML response. Instead, it is rendered by JavaScript after the page loads. To scrape such websites, you need tools that can render the JavaScript and retrieve the fully rendered HTML.

Using Playwright for Dynamic Websites

One of the best tools for scraping dynamic websites is Playwright, a Python library that interacts with the browser, executes JavaScript, and retrieves fully rendered HTML.

First, you need to install Playwright and the necessary dependencies:

pip install playwright
playwright install

Here’s how you can scrape a dynamic website:

from playwright.sync_api import sync_playwright
from markdownify import markdownify as md
# Scraping dynamic websites with Playwright
with sync_playwright() as p:
    # Launch a browser instance
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    # Target URL
    url = 'https://example.com/dynamic-page'
    
    # Go to the page
    page.goto(url)
    # Wait for specific content to load (wait for the element to be visible)
    page.locator('div.content').wait_for(state='visible')
    # Get the fully rendered HTML
    rendered_html = page.content()
    # Convert HTML to Markdown
    markdown_content = md(rendered_html)
    # Print the result
    print(markdown_content)
    # Close the browser
    browser.close()

This script opens a headless browser, loads the page, waits for the content to load, then scrapes the rendered HTML and converts it to Markdown.

Step 4: Handling Anti-Scraping Measures

Many websites implement measures to prevent scraping, such as CAPTCHAs, IP bans, or JavaScript challenges. These protections can make it difficult to scrape data using basic tools. To bypass these obstacles, you can use services like Bright Data, which offers a Web Unlocker API with advanced features such as CAPTCHA bypass and rotating proxies.

Bright Data’s Web Unlocker API allows you to access any website, regardless of anti-scraping measures, and convert the content directly into Markdown. Here’s an example of how to use it:

import requests
# Replace with your Bright Data API key and zone name
BRIGHT_DATA_API_KEY = ""
WEB_UNLOCKER_ZONE = ""
# URL to scrape
url_to_scrape = "https://example.com/protected-page"
# Prepare headers
headers = {
    "Authorization": f"Bearer {BRIGHT_DATA_API_KEY}",
    "Content-Type": "application/json"
}
# Payload for the Web Unlocker API
payload = {
    "url": url_to_scrape,
    "zone": WEB_UNLOCKER_ZONE,
    "format": "raw",
    "data_format": "markdown"
}
# Make the API request
response = requests.post(
    "https://api.brightdata.com/request",
    json=payload,
    headers=headers
)
# Check if the request was successful
if response.status_code == 200:
    # Get the response content in Markdown format
    markdown_content = response.text
    print(markdown_content)
else:
    print(f"Request failed with status code: {response.status_code}")
    print(f"Response: {response.text}")

You can also use the native proxy interface with the x-unblock-data-format: markdown header for more flexibility:

import requests
# Your Bright Data credentials
CUSTOMER_ID = ""
ZONE = ""
PASSWORD = ""
# Proxy configuration
proxy = f"@brd.superproxy.io:33335">http://brd-customer-{CUSTOMER_ID}-zone-{ZONE}:{PASSWORD}@brd.superproxy.io:33335"
proxies = {
    "http": proxy,
    "https": proxy
}
# Custom header to get Markdown output
headers = {
    "x-unblock-data-format": "markdown"
}
# URL to scrape
url = "https://example.com/protected-page"
# Make the request through the proxy
response = requests.get(
    url,
    proxies=proxies,
    headers=headers,
    verify=False  # Required for proxy SSL
)
if response.status_code == 200:
    markdown_content = response.text
    print(markdown_content)
else:
    print(f"Request failed: {response.status_code}")

Final Words

Scraping websites into Markdown is a powerful technique for extracting clean, structured content that AI tools, such as LLMs, can easily process. Whether you are working with static or dynamic websites, the process involves making an HTTP request, retrieving the HTML content, and converting it into Markdown. Tools like requests, markdownify, and Playwright make it relatively easy to scrape websites, while services like Bright Data's Web Unlocker can help bypass scraping protections and provide more robust solutions.

By following this guide and general best practices for web scraping, you can successfully scrape websites, convert their content into Markdown, and use it for a variety of applications, from content aggregation to data processing with advanced AI models.

Got questions? Let me know in the comments!