BoilerPy3

Common errors

• ImportError: No module named boilerpipe

  cause This error often occurs when trying to import `boilerpipe` instead of `boilerpy3`, or when an older `boilerpipe-py3` package (which might have been Java-dependent) was installed. `boilerpy3` is a distinct, native Python port.
  fix Ensure you have installed `boilerpy3` using `pip install boilerpy3` and that your import statements are `from boilerpy3 import extractors`.

• HTMLExtractionError: Could not extract content from the given HTML.

  cause This exception (or a similar error) can occur when the input HTML is malformed, unexpectedly structured, or when no discernible main content can be identified by the chosen extractor. Prior to v1.0.4, this would always raise an exception.
  fix Consider passing `raise_on_failure=False` to your `Extractor` constructor (e.g., `extractors.ArticleExtractor(raise_on_failure=False)`) to gracefully handle errors and retrieve any partial content. Also, try different extractors like `DefaultExtractor` or `KeepEverythingExtractor`.

• AttributeError: 'TextBlock' object has no attribute 'set_is_content'

  cause This specific method (`TextBlock.set_is_content()`) was temporarily missing or broken in some versions (e.g., prior to v1.0.5) and then restored. Other `AttributeError`s could arise from accessing internal camelCase variables after v1.0.5's snake_case conversion.
  fix Upgrade to the latest `boilerpy3` version (v1.0.5 or later fixed `set_is_content`). For other `AttributeError`s related to internal names, ensure you are using the documented public API and check if a camelCase attribute was converted to snake_case.

Warnings

• gotcha The `*_from_url()` methods (e.g., `get_content_from_url`) provided by `BoilerPy3` are intended for testing purposes only. For robust, production-grade URL content fetching, it is strongly recommended to use a dedicated HTTP library like `requests` to retrieve the HTML, then pass the raw HTML to `extractor.get_content()`.
  Fix: Use `requests` or a similar library to fetch HTML, then process with `extractor.get_content(html_string)`.
• gotcha From v1.0.4, the `Extractor` classes gained a `raise_on_failure` parameter, defaulting to `True`. If an HTML extraction error is encountered, an exception will be raised. Setting it to `False` will handle exceptions internally and return any partially extracted text, which changes default error handling behavior.
  Fix: Explicitly set `raise_on_failure=True` or `raise_on_failure=False` in the `Extractor` constructor based on desired error handling. If your code expects exceptions for all failures, ensure it's `True`.
• gotcha Version 1.0.5 introduced conversions from camelCase variable names to snake_case within the library. While internal, if you were accessing private or undocumented attributes from older versions, this might cause `AttributeError`.
  Fix: Review any code that might have accessed internal or undocumented camelCase attributes and update them to their snake_case equivalents if they were exposed. Stick to documented public APIs to avoid such issues.
• gotcha BoilerPy3 is a native Python port of Boilerpipe 1.2. It does not include features from Boilerpipe 1.3, as testing showed 1.3 performed worse in the Python port. Users expecting the latest Boilerpipe features may find the behavior different.
  Fix: Be aware that the underlying algorithm is equivalent to Boilerpipe 1.2. If a specific Boilerpipe 1.3 feature is needed, it might not be available in `boilerpy3`.

Install

• pip install boilerpy3 Install latest version

Imports

• ArticleExtractor
  wrong:

  from boilerpy3.extractors import ArticleExtractor

  correct:

  from boilerpy3 import extractors
  extractor = extractors.ArticleExtractor()

  While direct import works, the recommended pattern from the official examples is `from boilerpy3 import extractors`.
• DefaultExtractor

  from boilerpy3 import extractors
  extractor = extractors.DefaultExtractor()

  Used for general text extraction when `ArticleExtractor` is not suitable.

Quickstart

Demonstrates how to extract content from both a raw HTML string and a URL. For URL extraction, it is highly recommended to use the `requests` library for robust fetching, then pass the HTML content to the extractor.

from boilerpy3 import extractors
import requests

# Example 1: Extract from raw HTML string
html_content = "<html><body><h1>Title</h1><p>Main content here.</p><footer>Footer</footer></body></html>"
extractor = extractors.ArticleExtractor()
content_from_html = extractor.get_content(html_content)
print(f"Content from HTML: {content_from_html}")

# Example 2: Extract from a URL (recommended to use 'requests' for robustness)
# Replace with a real URL for testing
url = "https://example.com"

try:
    response = requests.get(url, timeout=5)
    response.raise_for_status() # Raise an exception for HTTP errors
    html_from_url = response.text
    content_from_url = extractor.get_content(html_from_url)
    print(f"\nContent from URL: {content_from_url}")
except requests.exceptions.RequestException as e:
    print(f"\nError fetching URL {url}: {e}")