Beautiful Soup 4

Warnings

• breaking Install name (beautifulsoup4) and import name (bs4) differ. pip install beautifulsoup4 then from bs4 import BeautifulSoup. Attempting pip install bs4 installs an old, abandoned package that is NOT Beautiful Soup 4.
  Fix: pip install beautifulsoup4 then from bs4 import BeautifulSoup. Never pip install bs4.
• breaking The text= parameter to find() and find_all() was renamed to string= in 4.9.0. Using text= raises DeprecationWarning now and will be removed eventually.
  Fix: Replace find('tag', text='value') with find('tag', string='value').
• gotcha Omitting the parser argument raises UserWarning and picks a parser automatically based on what's installed — which may differ between dev and production environments. Code may parse HTML differently on different machines.
  Fix: Always specify parser explicitly: BeautifulSoup(html, 'html.parser'). Use lxml for speed, html.parser for zero extra deps, html5lib for broken HTML.
• gotcha The BeautifulSoup package (without 4) on PyPI is the old, dead Beautiful Soup 3. pip install beautifulsoup installs the wrong package. pip install bs4 installs a different abandoned package.
  Fix: Always use: pip install beautifulsoup4 (with the 4).
• gotcha soup.find('tag') returns None if not found, not an empty list. Calling .get_text() or ['href'] on None raises AttributeError.
  Fix: Always check for None: tag = soup.find('a'); if tag: href = tag.get('href'). Or use soup.find_all() which always returns a list.

Install

• pip install beautifulsoup4 Core (uses Python's built-in html.parser)
• pip install beautifulsoup4 lxml With lxml parser (faster, more lenient)
• pip install beautifulsoup4 html5lib With html5lib parser (most lenient, slowest)

Imports

• BeautifulSoup

  from bs4 import BeautifulSoup
  
  # Always specify parser explicitly
  soup = BeautifulSoup(html_content, 'html.parser')  # built-in, no extra install
  soup = BeautifulSoup(html_content, 'lxml')          # requires: pip install lxml
  soup = BeautifulSoup(html_content, 'html5lib')      # requires: pip install html5lib

  Install name is beautifulsoup4 (with 4), import name is bs4 (without 4). Always specify parser. Omitting it raises UserWarning and Beautiful Soup picks a parser automatically — which may differ between environments.

Quickstart

Basic parsing. Use html.parser (built-in), lxml (fast), or html5lib (lenient).

from bs4 import BeautifulSoup
import requests

response = requests.get('https://example.com')
soup = BeautifulSoup(response.text, 'html.parser')

# Find elements
title = soup.find('title').get_text()       # first match
links = soup.find_all('a', href=True)       # all <a> with href
divs = soup.select('div.content > p')       # CSS selectors via soupsieve

# Navigate
body = soup.body
first_p = soup.body.p
parent = first_p.parent

# Text extraction
text = soup.get_text(separator=' ', strip=True)

# Find with attributes
button = soup.find('button', {'class': 'submit', 'type': 'submit'})

# Find by string content (NOT text= — that's deprecated)
heading = soup.find('h1', string='Welcome')