Newspaper4k

Warnings

• breaking Newspaper4k requires Python 3.10 or higher. Older Python versions (3.8 and 3.9) are no longer officially supported as of version 0.9.4, though they might still function. Ensure your Python environment meets this requirement.
  Fix: Upgrade your Python environment to 3.10 or newer.
• gotcha The Google News integration (`GoogleNewsSource`) can be unstable. Google frequently changes its HTML structure and URL encoding, which may cause this functionality to break without notice. This requires the `gnews` optional dependency.
  Fix: Be prepared for potential breakages and monitor for updates to the library. Consider alternative methods if reliability is critical.
• gotcha The `article.nlp()` method, which extracts keywords and summaries, currently works most reliably on Western languages. Its performance and accuracy might be limited for non-Western languages, even with language-specific optional dependencies installed.
  Fix: Verify NLP output for non-Western languages and consider external NLP libraries for more robust analysis if needed.
• gotcha Aggressively downloading many articles from a single source using multi-threading or rapid requests can lead to rate limiting, IP blocks, or CAPTCHA challenges from websites. Always respect `robots.txt` if enabled.
  Fix: Implement delays between requests, use proxy rotation, and rotate user-agent strings. Install `protego` optional dependency for robots.txt enforcement. For heavily protected sites (e.g., Cloudflare), `cloudscraper` (optional dependency) or external tools like Playwright might be necessary.
• gotcha When using the `Article` class directly (not `newspaper.article()`), you must explicitly call `article.download()` and `article.parse()` before attempting to access most article attributes (like `title`, `text`, `authors`, `publish_date`) or calling `article.nlp()`. Failure to do so will result in errors or empty data.
  Fix: Ensure `article.download()` and `article.parse()` are called sequentially before accessing article properties or running NLP.
• deprecated The `text_cleaned` and `clean_doc` attributes/methods have been deprecated and removed. Direct access to `article.clean_top_node` is also removed.
  Fix: Remove any usage of `text_cleaned` or `clean_doc`. The primary article text is available via `article.text`.

Install

• pip install newspaper4k Install base package
• pip install newspaper4k[all] Install with all optional dependencies
• pip install newspaper4k[gnews,cloudflare,zh] Install with specific optional dependencies (e.g., Google News, Cloudflare, Chinese language support)

Imports

• article
  wrong:

  from newspaper import Article; Article(url).download().parse()

  correct:

  import newspaper
  article = newspaper.article(url)

  The 'newspaper.article()' helper function streamlines downloading and parsing, combining the Article object instantiation, download(), and parse() calls into one step. The direct Article class import and method calls still work but are more verbose for a single article.
• build

  import newspaper
  source = newspaper.build(url)

  Used for building a 'Source' object to crawl an entire news website.

Quickstart

This quickstart demonstrates how to extract key information from a single news article using the `newspaper.article()` helper. It retrieves the title, authors, publish date, top image, and then performs NLP to get a summary and keywords. A commented-out example shows how to initialize and crawl an entire news source using `newspaper.build()` and iterate through its articles.

import newspaper

# Example for a single article
url = "https://edition.cnn.com/2023/11/08/china/china-blizzard-disruption-intl-hnk/index.html"
article = newspaper.article(url)

print(f"Title: {article.title}")
print(f"Authors: {article.authors}")
print(f"Publish Date: {article.publish_date}")
print(f"Top Image: {article.top_image}")

# Perform NLP for keywords and summary (requires NLTK and other NLP dependencies if installed)
article.nlp()
print(f"Summary: {article.summary}")
print(f"Keywords: {article.keywords}")

# Example for processing a news source (website)
# cnn_paper = newspaper.build('http://cnn.com')
# for article_obj in cnn_paper.articles:
#    print(article_obj.url)
#    article_obj.download()
#    article_obj.parse()
#    print(article_obj.title)