scrapy-feedexport-warc

A Scrapy feed exporter for storing crawled web pages in WARC (Web ARChive) format.

WARC is the standard archival format used by the Internet Archive and many web-crawling pipelines. This library makes it easy to write WARC files directly from a Scrapy spider using the standard FEEDS mechanism.

---

Installation

pip install scrapy-feedexport-warc

Requires Python 3.9+ and Scrapy 2.6+.

---

Quick start

1. Register the exporter in your Scrapy settings

# settings.py

from scrapy_feedexport_warc.exporters import WarcItemFilter

FEED_EXPORTERS = {
    "warc": "scrapy_feedexport_warc.exporters.WarcExporter",
}

FEEDS = {
    "output.warc": {
        "format": "warc",
        # Only write WarcItem instances; skip all other item types.
        "item_filter": WarcItemFilter,
    },
}

2. Yield WarcItem objects from your spider

# myspider.py

import scrapy
from scrapy_feedexport_warc.items import WarcItem


class MySpider(scrapy.Spider):
    name = "my_spider"
    start_urls = ["https://example.com/"]

    def parse(self, response):
        yield WarcItem(
            url=response.url,
            status=response.status,
            headers=[
                (k, v[0])
                for k, v in response.headers.items()
            ],
            body=response.body,
            protocol="HTTP/1.1",
        )

Run the spider:

scrapy crawl my_spider

This produces an output.warc file containing one WARC response record per crawled page.

---

WarcItem schema

WarcItem is the predefined item class that the exporter understands. All fields except protocol are required when an item is passed to the exporter; items with any required field missing are skipped with a warning.

Field	Type	Required	Description
url	str	Yes	The URL of the HTTP response (final URL after redirects). Used as the WARC Target-URI.
status	int	Yes	HTTP status code, e.g. 200.
headers	list	Yes	HTTP response headers as a list of (name, value) tuples.
body	bytes	Yes	Raw response body.
protocol	str	No	HTTP protocol version string. Defaults to "HTTP/1.1" when omitted.

Building headers from a Scrapy response

Scrapy stores response headers as bytes keys and list[bytes] values. The simplest way to convert them:

headers = [(k, v[0]) for k, v in response.headers.items()]

---

Configuration reference

FEED_EXPORTERS

Register the WARC format so Scrapy knows which class to use:

FEED_EXPORTERS = {
    "warc": "scrapy_feedexport_warc.exporters.WarcExporter",
}

FEEDS

Standard Scrapy FEEDS configuration. The item_filter key is strongly recommended so that only WarcItem instances reach the exporter:

from scrapy_feedexport_warc.exporters import WarcItemFilter

FEEDS = {
    "output.warc": {
        "format": "warc",
        "item_filter": WarcItemFilter,
    },
}

Gzip-compressed output (.warc.gz)

To write individually gzip-compressed WARC records (the standard .warc.gz format), use the provided GzipWarcExporter class. It is a thin WarcExporter subclass that defaults gzip=True, so you don't have to recreate it yourself. Because Scrapy does not forward arbitrary feed options to exporters, register it as a separate feed format:

# settings.py
from scrapy_feedexport_warc.exporters import WarcItemFilter

FEED_EXPORTERS = {
    "warc.gz": "scrapy_feedexport_warc.exporters.GzipWarcExporter",
}

FEEDS = {
    "output.warc.gz": {
        "format": "warc.gz",
        "item_filter": WarcItemFilter,
    },
}

Using item_filter with mixed item spiders

When a spider yields multiple item types (e.g. a ProductItem and a WarcItem), use WarcItemFilter to ensure that only WarcItem instances go to the WARC feed while other items continue to their own feeds:

from scrapy_feedexport_warc.exporters import WarcItemFilter

FEEDS = {
    # WARC archive – only WarcItems
    "pages.warc": {
        "format": "warc",
        "item_filter": WarcItemFilter,
    },
    # JSON Lines – only ProductItems (example)
    "products.jsonl": {
        "format": "jsonlines",
    },
}

---

Running the tests

pip install -e ".[dev]"
pytest

---

License

Apache-2.0. See LICENSE.