MainContentExtractor

0.0.4 · active · verified Sat Apr 11

MainContentExtractor is a Python library designed to extract the core content from HTML documents. It aims to address limitations found in other extraction tools, such as the inability to output clean HTML directly. The library is useful for LLM-related tasks and for feeding data into frameworks like LangChain and LlamaIndex by providing output in HTML, Text, or Markdown formats. It is currently at version 0.0.4, with a relatively active development cadence.

Warnings

gotcha The library internally uses `trafilatura` and converts its XML output to HTML. This conversion is described as irreversible and may not perfectly match the original HTML structure.
Fix: Be aware that the output HTML might not be an exact replica of the original main content. Validate the extracted output against your specific use case, especially if pixel-perfect fidelity to the original HTML is required.
gotcha There can be confusion between the PyPI package name (`MainContentExtractor` - capitalized) and the Python module name for import (`main_content_extractor` - lowercase with underscores). A `ModuleNotFoundError` will occur if the import statement uses the incorrect casing or formatting.
Fix: Always use `pip install MainContentExtractor` for installation and `from main_content_extractor import MainContentExtractor` for importing. Pay close attention to casing and underscores.
breaking Given the library's early stage (version 0.0.4), API stability is not guaranteed. Minor version updates may introduce breaking changes or significant modifications to the API without extensive deprecation warnings.
Fix: Pin your project's dependency to a specific patch version (e.g., `MainContentExtractor==0.0.4`) and thoroughly test your application after any updates. Monitor the GitHub repository for release notes and changes.

Install

pip install MainContentExtractor PyPI

Imports

MainContentExtractor
```
from main_content_extractor import MainContentExtractor
```
The PyPI package name is `MainContentExtractor` (capitalized), but the Python module and class name use `main_content_extractor` (lowercase with underscore). Ensure correct casing for import statements.

Quickstart

This quickstart demonstrates how to extract the main content from an HTML string using MainContentExtractor. It shows output in HTML, Markdown, and plain text formats. If fetching HTML from a URL, ensure `requests` is installed (`pip install requests`).

import requests
from main_content_extractor import MainContentExtractor

# Example HTML content (or fetch from a URL)
html_content = """
<html>
<head><title>Example Page</title></head>
<body>
    <header>Navigation Bar</header>
    <main>
        <h1>Important Article Title</h1>
        <p>This is the main content paragraph.</p>
        <p>Another paragraph with <a href="#">a link</a> inside.</p>
    </main>
    <footer>Footer content</footer>
</body>
</html>
"""

# Or, fetch from a URL (requires 'requests')
# url = "https://www.example.com"
# response = requests.get(url)
# response.encoding = 'utf-8'
# html_content = response.text

# Extract main content as HTML
extracted_html = MainContentExtractor.extract(html_content)
print("--- Extracted HTML ---")
print(extracted_html)

# Extract main content as Markdown
extracted_markdown = MainContentExtractor.extract(html_content, output_format="markdown")
print("\n--- Extracted Markdown ---")
print(extracted_markdown)

# Extract main content as plain text
extracted_text = MainContentExtractor.extract(html_content, output_format="text")
print("\n--- Extracted Text ---")
print(extracted_text)

view raw JSON →