MechanicalSoup

Warnings

• breaking MechanicalSoup does NOT execute JavaScript. Pages that render content via JS (React, Vue, Angular SPAs) will appear empty or broken. This is the most common misuse.
  Fix: For JavaScript-rendered pages use Playwright (pip install playwright + playwright install) or Selenium instead.
• gotcha Install name and import name differ in capitalization. pip install MechanicalSoup (capital M and S), but import mechanicalsoup (all lowercase). import MechanicalSoup raises ModuleNotFoundError.
  Fix: pip install MechanicalSoup then import mechanicalsoup
• gotcha raise_on_404=False by default — 404 responses are silently treated as successful. Code that navigates to a missing page gets no error and browser.page will contain the 404 HTML.
  Fix: Always pass raise_on_404=True: StatefulBrowser(raise_on_404=True). This raises LinkNotFoundError on 404 responses.
• gotcha Browser (stateless) and StatefulBrowser (stateful) have different APIs. Browser.get() returns a response. StatefulBrowser.open() returns a response AND updates internal state. Mixing them causes confusion.
  Fix: Use StatefulBrowser for multi-page workflows (login flows, form sequences). Use Browser only for simple one-shot requests.
• gotcha select_form() with no argument selects the first form on the page. On pages with multiple forms (login + newsletter), this may select the wrong form.
  Fix: Always pass a CSS selector: browser.select_form('#login-form'). Or use the nr= argument to select by index: browser.select_form('form', nr=1).

Install

• pip install MechanicalSoup Standard (capital M and S in package name)

Imports

• StatefulBrowser

  import mechanicalsoup
  
  # StatefulBrowser remembers state (cookies, current page)
  browser = mechanicalsoup.StatefulBrowser(
      soup_config={'features': 'lxml'},
      raise_on_404=True  # recommended: raise on 404 instead of silently failing
  )
  browser.open('https://example.com/login')
  browser.select_form('#login-form')
  browser['username'] = 'alice'
  browser['password'] = 'secret'
  browser.submit_selected()

  Install with pip install MechanicalSoup (capital). Import as import mechanicalsoup (lowercase). Use StatefulBrowser (not Browser) for multi-step workflows — it tracks the current page and form state.

Quickstart

StatefulBrowser for form-based workflows. Requires lxml or html.parser.

import mechanicalsoup

# StatefulBrowser is the high-level interface
browser = mechanicalsoup.StatefulBrowser(
    soup_config={'features': 'lxml'},
    raise_on_404=True
)

# Navigate to a page
browser.open('https://httpbin.org/forms/post')

# Select and fill a form
browser.select_form('form')  # or CSS selector like '#login-form'
browser['custname'] = 'Alice'
browser['custtel'] = '555-1234'
browser['comments'] = 'Hello!'

# Submit the form
response = browser.submit_selected()
print(response.status_code)

# Access the resulting page
print(browser.page.title.string)