Extruct

Common errors

• ImportError: cannot import name '_ElementStringResult' from 'lxml.etree'

  cause An incompatibility between older `extruct` versions (<0.18.0) and `lxml` versions 5.1.0 or newer.
  fix Update `extruct` to version 0.18.0 or later. If updating `extruct` is not possible, downgrade `lxml` to a version prior to 5.1.0 (e.g., `pip install lxml==5.0.1`).

• Empty dictionary or unexpected missing metadata in extruct output.

  cause The target HTML either does not contain metadata in the formats `extruct` supports, or relative URLs were not resolved because `base_url` was omitted.
  fix First, inspect the source HTML for the presence of Microdata, JSON-LD, Open Graph, etc. Second, always provide the `base_url` parameter to `extruct.extract(html_string, base_url=actual_url)` to ensure proper resolution of relative URLs and images.

• ModuleNotFoundError: No module named 'requests' when running the `extruct` command-line tool.

  cause The `requests` library, which the command-line interface uses to fetch web pages, is an optional dependency and not installed by default.
  fix Install `extruct` with its command-line interface dependencies using `pip install 'extruct[cli]'`.

Warnings

• breaking Versions of `extruct` prior to 0.18.0 might encounter `ImportError: cannot import name '_ElementStringResult' from 'lxml.etree'` when used with `lxml` versions 5.1.0 or higher due to internal API changes in `lxml`.
  Fix: Upgrade `extruct` to version 0.18.0 or newer. If an upgrade is not feasible, temporarily pin `lxml` to a version less than 5.1.0 (e.g., `pip install lxml<5.1.0`).
• gotcha The output structure of `extruct` can be inconsistent for certain metadata types, sometimes returning a list of dictionaries and other times a single dictionary, which can lead to `TypeError` or `IndexError` if not handled carefully in post-processing.
  Fix: Use the `uniform=True` parameter in `extruct.extract()` to ensure a more consistent output structure (e.g., always a list). Always check the type of the returned data before attempting to access elements by index or key.
• gotcha Extracting all supported syntaxes from very large or complex HTML documents can be memory-intensive and slow. By default, `extruct.extract()` attempts all formats.
  Fix: To optimize performance and resource usage, specify only the required syntaxes using the `syntaxes` parameter (e.g., `syntaxes=['json-ld', 'opengraph']`).
• gotcha The command-line tool `extruct` (e.g., `extruct 'http://example.com'`) requires the `requests` library, which is an optional dependency and not installed by default with a basic `pip install extruct`.
  Fix: Install `extruct` with the `cli` extra to include `requests`: `pip install 'extruct[cli]'`.

Install

• pip install extruct Core library
• pip install 'extruct[cli]' With command-line tool dependencies

Imports

• extract

  from extruct import extract

  The primary function for all-in-one metadata extraction.
• get_base_url

  from w3lib.html import get_base_url

  Commonly imported for resolving relative URLs in extracted metadata.
• OpenGraphExtractor

  from extruct.opengraph import OpenGraphExtractor

  Example of importing a specific extractor if only certain formats are needed.

Quickstart

This quickstart fetches HTML content from a URL, determines the base URL for resolving relative paths, and then uses `extruct.extract` to retrieve structured metadata in common formats (JSON-LD, Microdata, Open Graph). The `uniform=True` parameter ensures a consistent output structure for easier processing.

import extruct
import requests
from w3lib.html import get_base_url
import pprint

pp = pprint.PrettyPrinter(indent=2)

# Replace with a real URL to test
url = 'http://quotes.toscrape.com/scroll'
r = requests.get(url)
base_url = get_base_url(r.text, r.url)

data = extruct.extract(r.text, base_url=base_url, uniform=True, syntaxes=['json-ld', 'microdata', 'opengraph'])

pp.pprint(data)