urllib for Node.js

Common errors

• SyntaxError: Named export 'request' not found. The requested module 'urllib' does not provide an export named 'request'

  cause Attempting to use ES module `import` syntax in a Node.js CommonJS module, or running in an older Node.js version not configured for ESM.
  fix For Node.js CommonJS files, change `import { request } from 'urllib';` to `const { request } = require('urllib');`. If you intend to use ESM, ensure your `package.json` has `"type": "module"` or files end with `.mjs`.

• TypeError: Cannot destructure property 'data' of 'undefined' or 'null' as it is null.

  cause Missing `await` keyword when calling `urllib.request`, causing the result to be a Promise instead of the resolved value.
  fix Ensure you `await` the `request` call: `const { data, res } = await request(...);`. Remember that `await` can only be used inside `async` functions.

• Error: connect ETIMEDOUT <IP>:<PORT>

  cause The network connection timed out before a response could be established or fully received.
  fix Increase the `timeout` option in `request` (e.g., `timeout: 10000` milliseconds). Verify network connectivity and firewall rules. The target server might be down or heavily loaded.

• SyntaxError: Unexpected token '"', "{ "error": "In"..." is not valid JSON

  cause The `dataType: 'json'` option was used, but the server responded with non-JSON content or malformed JSON. The error message indicates an attempt to parse a string that starts like JSON but is invalid or unexpected.
  fix Inspect the raw response content (e.g., by logging `data.toString()`) to understand why it's not valid JSON. You might need to remove `dataType: 'json'` and manually parse the `data` buffer/string after checking `res.headers['content-type']` or handle non-JSON responses gracefully.

Warnings

• breaking urllib v4 introduces significant internal changes by basing its API on `undici`, which may lead to subtle behavioral differences or require Node.js 18.19.0+. While the `request` API remains largely consistent, deep integrations or reliance on internal `urllib` behaviors might require adjustments.
  Fix: Review the official changelog for specific breaking changes between v3 and v4. Ensure your Node.js environment meets the `>=18.19.0` requirement. Test thoroughly after upgrading.
• breaking The `compressed` option, which controls automatic decompression of response bodies, changed its default value to `false`. Previously, it might have defaulted to `true` or been implicitly handled, leading to uncompressed responses if not explicitly set.
  Fix: If your application relies on automatic decompression (e.g., `gzip`, `deflate`), explicitly set `compressed: true` in your request options. Otherwise, you might receive raw compressed data.
• gotcha The `timeout` option in `urllib` historically had variations in how it accepted string formats (e.g., '2s'). While v4.7.1 added compatibility for `urllib@2` string formats, relying on number-based timeouts (milliseconds) is the most robust and consistent approach across versions.
  Fix: Always provide the `timeout` option as a number representing milliseconds (e.g., `timeout: 5000` for 5 seconds) to ensure consistent behavior across all `urllib` versions and avoid potential parsing issues.
• deprecated The direct `urllib` class constructor or `new urllib()` pattern is generally deprecated in favor of named exports like `request` for simple calls or `create` for custom instances. Relying on default exports or class instantiation can lead to unexpected behavior or future breakage.
  Fix: Prefer `import { request } from 'urllib';` for making requests directly. If you need a custom `urllib` instance with specific configurations (e.g., a custom dispatcher), use `import { create } from 'urllib';`.
• gotcha Security vulnerabilities in `urllib` or its dependencies are tracked by Snyk. While the library actively addresses them, it's crucial to regularly update to the latest patch versions to mitigate known risks.
  Fix: Regularly run `npm audit` or `snyk test` in your project and update `urllib` to the latest recommended version, especially for patch releases, to ensure you have the most secure version.

Install

• npm install urllib npm
• yarn add urllib yarn
• pnpm add urllib pnpm

Imports

• request
  wrong:

  const { request } = require('urllib'); // Incorrect in a pure ESM module context

  correct:

  import { request } from 'urllib';

  For modern Node.js ESM modules, use named imports. While `require` might work via an interop layer in some ESM setups, directly using `import` is the idiomatic and most reliable approach.
• request (CommonJS)
  wrong:

  import { request } from 'urllib'; // Incorrect in a CommonJS module context

  correct:

  const { request } = require('urllib');

  For Node.js CommonJS modules, use the `require` syntax. Attempting to use `import` in a `.js` file without `"type": "module"` in `package.json` will result in a `SyntaxError`.
• create (for custom instance)
  wrong:

  import urllib from 'urllib'; // Default import is not the primary API
  const client = new urllib(); // Class instantiation is not the recommended pattern

  correct:

  import { create } from 'urllib';
  const client = create({ /* options */ });

  To create a custom `urllib` instance with specific global configurations or a custom `undici` dispatcher, use the named `create` factory function. Direct class instantiation or default imports are generally not recommended in current versions.

Quickstart

Demonstrates a basic GET request to a public API, parsing the response as JSON, and handling potential errors. It includes common options like `dataType`, `method`, `timeout`, and `headers`.

import { request } from 'urllib';

async function fetchData() {
  try {
    const { data, res } = await request('https://cnodejs.org/api/v1/topics?limit=5', {
      dataType: 'json', // Automatically parse response as JSON
      method: 'GET',
      timeout: 5000,   // 5 seconds timeout
      headers: {
        'User-Agent': 'urllib-checklist-example/1.0'
      }
    });
    console.log('Status: %s', res.status);
    console.log('Headers: %j', res.headers);
    console.log('First topic title: %s', data.data[0].title);
  } catch (error) {
    console.error('Request failed:', error.message);
  }
}

fetchData();