{"id":7402,"library":"mechanize","title":"mechanize","description":"mechanize provides a stateful, programmatic web browsing interface, allowing for opening URLs, following links, submitting forms, and handling cookies. It simulates a web browser's behavior without a GUI or JavaScript engine. The current version is 0.4.10, released in 2023, and it follows a slow release cadence, primarily for maintenance and bug fixes.","status":"maintenance","version":"0.4.10","language":"en","source_language":"en","source_url":"https://github.com/python-mechanize/mechanize","tags":["web scraping","browser automation","form submission","http client"],"install":[{"cmd":"pip install mechanize","lang":"bash","label":"Install latest version"}],"dependencies":[],"imports":[{"note":"While `from mechanize import Browser` works, the idiomatic and generally recommended way to import and use mechanize is `import mechanize` and then access its components via `mechanize.Browser()`.","wrong":"from mechanize import Browser","symbol":"Browser","correct":"import mechanize\nbr = mechanize.Browser()"}],"quickstart":{"code":"import mechanize\nimport http.cookiejar as cookielib\n\nbr = mechanize.Browser()\n\n# Cookie Jar setup (optional but recommended for stateful browsing)\ncj = cookielib.LWPCookieJar()\nbr.set_cookiejar(cj)\n\n# Browser options\nbr.set_handle_equiv(True)\nbr.set_handle_gzip(True)\nbr.set_handle_redirect(True)\nbr.set_handle_robots(False) # Often set to False for scraping\n\n# User-Agent\nbr.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]\n\n# Open a page\nurl = \"http://www.example.com/\"\n# Use a placeholder for environment variables if authentication is needed\n# url = os.environ.get('MECHANIZED_TARGET_URL', 'http://www.example.com/')\n\ntry:\n    response = br.open(url)\n    print(f\"Title: {br.title()}\")\n    print(f\"Status: {response.code}\")\n    # print(response.read().decode('utf-8'))\nexcept Exception as e:\n    print(f\"An error occurred: {e}\")\n","lang":"python","description":"This quickstart demonstrates how to initialize a mechanize Browser, configure it with a cookie jar and custom user-agent, and open a URL. It includes common settings like disabling robots.txt handling."},"warnings":[{"fix":"For JavaScript-heavy sites, consider a full-fledged browser automation library like Selenium or Playwright.","message":"mechanize does NOT execute JavaScript. It's a 'headless' browser in the sense it has no GUI, but it cannot render or interact with dynamic content generated by JavaScript. If a page relies on JavaScript for content loading, form submission, or navigation, mechanize will not see or interact with it.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure you are using `mechanize` version 0.3.0 or higher for Python 3. Refactor Python 2 code, paying attention to `str`/`bytes` conversions and any reliance on internal mechanize Python 2 specifics.","message":"Major breaking changes occurred during the transition from Python 2 to Python 3. Code written for mechanize on Python 2.x is likely incompatible with Python 3.x due to changes in internal modules (e.g., `_mechanize` C module removed) and string/bytes handling.","severity":"breaking","affected_versions":"<0.3.0 (Python 2) vs. >=0.3.0 (Python 3)"},{"fix":"To ignore `robots.txt`, set `br.set_handle_robots(False)` immediately after initializing the browser instance.","message":"By default, mechanize respects `robots.txt` rules. Many scraping tasks require bypassing this, which can lead to `HTTP Error 403: Forbidden` or simply not accessing desired content.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Always set a custom User-Agent string using `br.addheaders = [('User-agent', 'YOUR_USER_AGENT_STRING')]`.","message":"Many modern websites require specific User-Agent headers to display content correctly or to prevent blocking. Without setting a realistic User-Agent, you might receive errors or be served different content.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Check the target URL in a regular browser. Ensure your User-Agent is realistic. The server might be blocking your requests due to suspicious headers or rate limiting. Try increasing timeouts or retrying.","cause":"The server responded with an invalid or empty HTTP status line. This often indicates a malformed server response, or the server closing the connection unexpectedly.","error":"http.client.BadStatusLine: ''"},{"fix":"Wrap `br.open()` calls in a `try-except` block to catch `mechanize.URLError` or `mechanize.HTTPError`. Verify the URL and network connectivity. Inspect `br.response()` if available for details.","cause":"This error typically occurs when `br.open()` fails to retrieve a valid response object (e.g., due to a network error, DNS failure, or a very quick connection reset), and subsequent code tries to access headers or other attributes on a `None` object.","error":"AttributeError: 'NoneType' object has no attribute 'get_header' or 'AttributeError: 'NoneType' object has no attribute 'headers'"},{"fix":"Inspect the HTML content of the page (`response.read()`) to identify the correct form attributes (name, id) or its numerical index. You can iterate `for form in br.forms(): print(form)` to see all available forms.","cause":"You are trying to select a form that doesn't exist on the current page, or your selection criteria (name, id, index `nr`) do not match any available forms.","error":"mechanize._response.html.FormNotFoundError: no form matching name ... or nr ..."},{"fix":"Ensure `br.set_handle_robots(False)` is set if you intend to ignore `robots.txt`. Set a realistic User-Agent string. If persistent, consider rotating IP addresses or waiting before retrying.","cause":"The server denied access to the resource. This could be due to not respecting `robots.txt`, an invalid or missing User-Agent, IP blocking, or other security measures.","error":"mechanize._urllib2_fork.HTTPError: HTTP Error 403: Forbidden"}]}