Scraping a website to migrate it to WordPress

I was tasked with migrating an old website from Yahoo Pages to WordPress. Normally, migrating a website is fairly straightforward if you can export its content or access the database.

Unfortunately, that wasn’t an option here.

The website contained a collection of articles about local and world history. I needed to get that content out of the existing website and into a format that could be used for the WordPress migration.

Copying and pasting every page manually wasn’t particularly appealing, so I decided to write a web scraper to do the repetitive work for me.

Using Scrapy

I hadn’t needed to build a particularly complicated scraper, and Python seemed like a good choice for the job.

I used Scrapy, a Python framework designed for crawling websites and extracting data from pages.

Rather than writing something to fetch each page manually, Scrapy handles the requests, crawling and processing of responses.

I started with a Scrapy project and created a spider to crawl the website.

 1from scrapy.spiders import SitemapSpider
 2import re
 3 
 4class HistoriesSpider(SitemapSpider):
 5    name = 'website'
 6    allowed_domains = ["example.com"]
 7    sitemap_urls = ['http://example.com/sitemap.xml']
 8    start_urls = [
 9        'http://example.com/',
10    ]
    

      
Copied!
      
      
    
    from scrapy.spiders import SitemapSpider
import re

class HistoriesSpider(SitemapSpider):
    name = 'website'
    allowed_domains = ["example.com"]
    sitemap_urls = ['http://example.com/sitemap.xml']
    start_urls = [
        'http://example.com/',
    ]
    

Scrapy has several different spider types, but SitemapSpider was particularly useful here.

The website already had an XML sitemap listing its pages, so there was no need to write logic to discover links and follow them around the website.

By pointing Scrapy at sitemap.xml, it could find the URLs and process them automatically.

Extracting the content

Once Scrapy had fetched a page, I needed to extract the information that would be useful for the migration.

For this first version, I kept it simple. I wanted the page title, URL identifier and content.

1def parse(self, response):
2    item = {}
3    item['title'] = response.css("p::text").get()
4    item['url'] = re.findall(
5        "(?<=\\/)[^\\/]*(?=\\.\\w+$)",
6        response.url
7    )
8    item['content'] = response.css("p::text").getall()
9    yield item
    

      
Copied!
      
      
    
    def parse(self, response):
    item = {}
    item['title'] = response.css("p::text").get()
    item['url'] = re.findall(
        "(?<=\\/)[^\\/]*(?=\\.\\w+$)",
        response.url
    )
    item['content'] = response.css("p::text").getall()
    yield item
    

Scrapy’s CSS selectors made extracting the content easy.

The selector p::text finds text directly inside paragraph elements. Calling .get() returns the first match, while .getall() returns a list of matches.

I used the first paragraph text as the title and collected the remaining paragraph text as content.

This isn’t a particularly sophisticated way of extracting article content. It doesn’t preserve HTML formatting, headings, links or images, and text inside nested elements can be missed.

But it provided a way of getting the text out of the old website without having to open every page individually.

Extracting the URL

I also wanted to keep track of where each article had come from.

For that, I used a regular expression to extract the filename from the URL, without its extension.

1re.findall(
2    "(?<=\\/)[^\\/]*(?=\\.\\w+$)",
3    response.url
4)
    

      
Copied!
      
      
    
    re.findall(
    "(?<=\\/)[^\\/]*(?=\\.\\w+$)",
    response.url
)
    

For example, a URL like:

https://example.com/cambridge.html

Would return:

['cambridge']

Having the original page identifier available would be useful when matching the extracted content to its new location in WordPress, particularly when dealing with redirects.

The expression was written for the URL structure of the original website, so it wouldn’t necessarily work for every website.

Exporting the results

One of the useful things about Scrapy is that it already supports exporting scraped data into formats such as JSON and CSV.

There’s no need to build a separate file writer for a simple project like this.

For example, running the spider with the following command exports the results to a JSON file:

1scrapy crawl website-O website.json
    

      
Copied!
      
      
    
    scrapy crawl website-O website.json
    

Each page produces an object containing the extracted title, URL identifier and paragraph content.

That gives me structured data to work with rather than a collection of HTML pages.

From there, the content can be cleaned up and transformed into something suitable for importing into WordPress.

Being considerate when crawling

The scraper also has two settings enabled which are worth mentioning.

1ROBOTSTXT_OBEY = True
2HTTPCACHE_ENABLED = True
    

      
Copied!
      
      
    
    ROBOTSTXT_OBEY = True
HTTPCACHE_ENABLED = True
    

The first tells Scrapy to respect the website’s robots.txt rules.

The second enables HTTP caching, which means responses can be reused between runs rather than fetching the same pages repeatedly.

That was useful while developing the scraper, as I could change the extraction logic and run it again without repeatedly requesting every page from the website.

Getting the content into WordPress

The scraper wasn’t intended to be a complete migration tool. Its job was to get the content out of Yahoo Pages and into a format that I could work with.

There was still work to do before the content was suitable for WordPress.

The extracted text would need cleaning up, formatting would need restoring where necessary, and the original URLs would need mapping to their new WordPress equivalents.

But automating the extraction removed a substantial amount of repetitive work from the process.

The whole scraper is only a few lines of actual logic. Scrapy does most of the heavy lifting, from reading the sitemap to requesting pages and exporting the results.

It’s a useful approach when migrating away from an old platform that doesn’t offer an easy way to export its content. As long as you can still access the pages, you have something to work with.