justext

Warnings

• breaking justext v3.0.0 dropped support for Python 3.4 and older versions (including Python 2.x). Attempts to install or run on these versions will fail or lead to unexpected behavior.
  Fix: Upgrade to Python 3.5 or newer. For projects requiring older Python versions, consider pinning justext to < 3.0.0 (e.g., `justext<3.0.0`).
• gotcha The `justext.justext()` function expects raw HTML content as bytes (e.g., from `response.content`). Passing a decoded string (e.g., `response.text`) can lead to parsing errors or incorrect results due to encoding issues with `lxml`.
  Fix: Always provide HTML content as bytes. If you have a string, ensure it's encoded correctly before passing it: `html_string.encode('utf-8')`.
• gotcha Older versions of justext (specifically before v3.0.1) had compatibility issues with newer versions of `lxml`, leading to parsing errors. Similarly, versions before v3.0.0 would fail on Python 3.8+ due to the removal of `cgi.escape`.
  Fix: Ensure you are using `justext` version 3.0.1 or higher for better `lxml` compatibility, and 3.0.0 or higher for Python 3.8+ environments. Regularly update `justext` to its latest stable release.

Install

• pip install justext Install latest version

Imports

• justext

  import justext

  The primary function is justext.justext(). Accessing it via 'import justext' and then 'justext.justext()' is the most common pattern.

Quickstart

This quickstart demonstrates how to fetch an HTML document using `requests`, then pass its raw byte content to `justext.justext()` along with a predefined stoplist (e.g., 'English') to extract and print human-readable text, filtering out boilerplate.

import requests
import justext

# Example URL (replace with a real URL for actual testing)
url = "https://www.python.org"

try:
    response = requests.get(url, timeout=5)
    # justext expects bytes as input
    html_content = response.content

    # Get the English stoplist
    stoplist = justext.get_stoplist("English")

    # Process the HTML content
    paragraphs = justext.justext(html_content, stoplist)

    print(f"Extracted text from {url}:")
    for paragraph in paragraphs:
        if not paragraph.is_boilerplate:
            print(paragraph.text)

except requests.exceptions.RequestException as e:
    print(f"Error fetching URL: {e}")
except Exception as e:
    print(f"An error occurred: {e}")