{"id":230,"library":"MechanicalSoup","title":"MechanicalSoup","description":"Python library for automating website interaction — form submission, cookie handling, link following. Built on requests + BeautifulSoup. Current version is 1.4.0 (2023). Install name is MechanicalSoup (capital M and S), import is mechanicalsoup (all lowercase). Does NOT execute JavaScript — use Playwright or Selenium for JS-rendered pages. Maintenance status: low activity since 2023.","status":"active","version":"1.4.0","language":"python","source_language":"en","source_url":"https://mechanicalsoup.readthedocs.io/","tags":["web-scraping","forms","automation","requests","beautifulsoup","browser-automation"],"install":[{"cmd":"pip install MechanicalSoup","lang":"bash","label":"Standard (capital M and S in package name)"}],"dependencies":[{"reason":"Required. Installed automatically.","package":"requests>=2.22.0","optional":false},{"reason":"Required. Installed automatically.","package":"beautifulsoup4>=4.7","optional":false},{"reason":"Optional but recommended parser for BeautifulSoup.","package":"lxml","optional":true}],"imports":[{"note":"Install with pip install MechanicalSoup (capital). Import as import mechanicalsoup (lowercase). Use StatefulBrowser (not Browser) for multi-step workflows — it tracks the current page and form state.","wrong":"import MechanicalSoup  # ImportError — import name is lowercase\n\n# Browser (stateless) vs StatefulBrowser:\nbrowser = mechanicalsoup.Browser()  # stateless — does NOT track current page\nbrowser.open('https://example.com')  # AttributeError — Browser has no open()","symbol":"StatefulBrowser","correct":"import mechanicalsoup\n\n# StatefulBrowser remembers state (cookies, current page)\nbrowser = mechanicalsoup.StatefulBrowser(\n    soup_config={'features': 'lxml'},\n    raise_on_404=True  # recommended: raise on 404 instead of silently failing\n)\nbrowser.open('https://example.com/login')\nbrowser.select_form('#login-form')\nbrowser['username'] = 'alice'\nbrowser['password'] = 'secret'\nbrowser.submit_selected()"}],"quickstart":{"code":"import mechanicalsoup\n\n# StatefulBrowser is the high-level interface\nbrowser = mechanicalsoup.StatefulBrowser(\n    soup_config={'features': 'lxml'},\n    raise_on_404=True\n)\n\n# Navigate to a page\nbrowser.open('https://httpbin.org/forms/post')\n\n# Select and fill a form\nbrowser.select_form('form')  # or CSS selector like '#login-form'\nbrowser['custname'] = 'Alice'\nbrowser['custtel'] = '555-1234'\nbrowser['comments'] = 'Hello!'\n\n# Submit the form\nresponse = browser.submit_selected()\nprint(response.status_code)\n\n# Access the resulting page\nprint(browser.page.title.string)","lang":"python","description":"StatefulBrowser for form-based workflows. Requires lxml or html.parser."},"warnings":[{"fix":"For JavaScript-rendered pages use Playwright (pip install playwright + playwright install) or Selenium instead.","message":"MechanicalSoup does NOT execute JavaScript. Pages that render content via JS (React, Vue, Angular SPAs) will appear empty or broken. This is the most common misuse.","severity":"breaking","affected_versions":"all"},{"fix":"pip install MechanicalSoup then import mechanicalsoup","message":"Install name and import name differ in capitalization. pip install MechanicalSoup (capital M and S), but import mechanicalsoup (all lowercase). import MechanicalSoup raises ModuleNotFoundError.","severity":"gotcha","affected_versions":"all"},{"fix":"Always pass raise_on_404=True: StatefulBrowser(raise_on_404=True). This raises LinkNotFoundError on 404 responses.","message":"raise_on_404=False by default — 404 responses are silently treated as successful. Code that navigates to a missing page gets no error and browser.page will contain the 404 HTML.","severity":"gotcha","affected_versions":"all"},{"fix":"Use StatefulBrowser for multi-page workflows (login flows, form sequences). Use Browser only for simple one-shot requests.","message":"Browser (stateless) and StatefulBrowser (stateful) have different APIs. Browser.get() returns a response. StatefulBrowser.open() returns a response AND updates internal state. Mixing them causes confusion.","severity":"gotcha","affected_versions":"all"},{"fix":"Always pass a CSS selector: browser.select_form('#login-form'). Or use the nr= argument to select by index: browser.select_form('form', nr=1).","message":"select_form() with no argument selects the first form on the page. On pages with multiple forms (login + newsletter), this may select the wrong form.","severity":"gotcha","affected_versions":"all"}],"env_vars":null,"last_verified":"2026-05-12T12:06:29.311Z","next_check":"2026-09-27T00:00:00.000Z","problems":[{"fix":"Ensure the link exists on the page and is correctly specified. Use the `find_link` method to verify its presence before attempting to follow it.","cause":"This error occurs when the specified link cannot be found on the current page.","error":"mechanicalsoup.utils.LinkNotFoundError"},{"fix":"Specify the parser explicitly by setting the `soup_config` parameter, e.g., `soup_config={'features': 'lxml'}`.","cause":"This warning appears when BeautifulSoup defaults to an HTML parser because none was specified.","error":"UserWarning: No parser was explicitly specified, so I’m using the best available HTML parser for this system (“lxml”)."},{"fix":"Ensure to call `browser.close()` before the end of the script or use a context manager with the `with` statement to handle the browser session.","cause":"This error can occur when the browser session is not properly closed, leading to issues during garbage collection.","error":"ReferenceError: weakly-referenced object no longer exists"},{"fix":"Verify the link's presence and correctness on the page before attempting to follow it.","cause":"This error occurs when attempting to follow a link that does not exist on the page.","error":"mechanicalsoup.utils.LinkNotFoundError: No link found matching the criteria"},{"fix":"Ensure the form exists and is correctly specified before attempting to select it.","cause":"This error occurs when attempting to select a form that does not exist on the page.","error":"mechanicalsoup.utils.LinkNotFoundError: No form matching the criteria"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":0,"quickstart_tag":"stale","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.97,"mem_mb":13.5,"disk_size":"34.6M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.71,"mem_mb":13.5,"disk_size":"35M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.61,"mem_mb":15,"disk_size":"36.9M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.33,"mem_mb":15,"disk_size":"37M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.21,"mem_mb":14.6,"disk_size":"28.7M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.29,"mem_mb":14.6,"disk_size":"29M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.16,"mem_mb":15.4,"disk_size":"28.4M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.19,"mem_mb":15.4,"disk_size":"29M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.93,"mem_mb":13.4,"disk_size":"33.9M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.79,"mem_mb":13.4,"disk_size":"34M"}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"stale","tag_description":"widespread failures or data too old to trust","results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]}}