Got Scraping

Common errors

• ReferenceError: require is not defined

  cause Attempting to use CommonJS `require()` in an ES module context or with an ESM-only library like `got-scraping`.
  fix Change your import statement to `import { gotScraping } from 'got-scraping';` or configure your project as a CommonJS module and use dynamic `import()` for `got-scraping`.

• TypeError: gotScraping is not a function

  cause `gotScraping` is an already instantiated `got` instance, not a constructor. It's meant to be called directly for requests or to chain methods like `.get()` or `.post()`.
  fix Do not use `new` keyword or try to call `gotScraping()` directly. Instead, call its methods: `gotScraping.get(url)`, `gotScraping.post(options)`, or configure via `gotScraping(options)`.

• Error: Proxy CONNECT Error: 407 Proxy Authentication Required

  cause The configured proxy server requires authentication, but either no credentials were provided or they are incorrect.
  fix Ensure the `proxyUrl` includes valid username and password credentials: `http://username:password@proxy.example.com:port`. Verify that the credentials are correct for your proxy.

• Error: This module was compiled against Node.js version XX. You are currently running version YY. Please update Node.js to a compatible version.

  cause `got-scraping` requires Node.js >=16, and you are running an older, incompatible version.
  fix Upgrade your Node.js environment to version 16 or higher. Use a Node Version Manager (e.g., `nvm`) to easily switch versions: `nvm install 16 && nvm use 16`.

Warnings

• breaking The `got-scraping` package has been officially declared End-of-Life (EOL) by its maintainers. It will no longer receive updates, bug fixes, or support. Users are strongly advised to migrate to the recommended alternative, `impit`, for new projects and consider migrating existing projects.
  Fix: For new projects, use `impit` (npm install impit). For existing projects, evaluate the effort required to migrate to `impit` or understand that `got-scraping` will not be maintained further.
• breaking `got-scraping` is an ESM-only module since version 4. This means it can only be imported using `import` statements in ES modules or dynamically with `import()` in CommonJS contexts. Direct `require()` calls will result in errors.
  Fix: Ensure your project is configured as an ES module (e.g., add `"type": "module"` to `package.json`) and use `import { gotScraping } from 'got-scraping';`. If you must use CommonJS, employ dynamic imports: `const { gotScraping } = await import('got-scraping');` or `gotScraping ??= (await import('got-scraping')).gotScraping;`.
• breaking The package requires Node.js version 16 or newer. Running `got-scraping` on older Node.js versions may lead to instability, particularly with HTTP/2 connections, or outright failures.
  Fix: Upgrade your Node.js environment to version 16 or later using a version manager like `nvm` (e.g., `nvm install 16 && nvm use 16`).
• gotcha When using proxies, ensure the `proxyUrl` is correctly formatted with scheme, host, port, and credentials if required (e.g., `http://username:password@myproxy.com:1234`). Incorrectly formatted URLs or invalid credentials will lead to proxy connection failures.
  Fix: Double-check `proxyUrl` format and credentials. Ensure the proxy server is reachable and configured to accept connections from your environment. Use environment variables or a secure configuration for credentials, not hardcoded values in production.
• gotcha By default, `got-scraping` generates new browser-like headers for each request. To maintain a consistent set of headers across multiple requests, you must provide a unique `sessionToken` object (any non-primitive object will work, e.g., `{}`) to reuse the same generated headers for subsequent requests within that session.
  Fix: Pass a consistent `sessionToken` object: `gotScraping.get({ url: '...', sessionToken: mySessionObject });` to ensure headers are reused. A new object will always generate new headers.

Install

• npm install got-scraping npm
• yarn add got-scraping yarn
• pnpm add got-scraping pnpm

Imports

• gotScraping
  wrong:

  const { gotScraping } = require('got-scraping');

  correct:

  import { gotScraping } from 'got-scraping';

  The package is ESM-only since v4. For compatibility with CommonJS modules, use a dynamic import: `gotScraping ??= (await import('got-scraping')).gotScraping;`
• HeaderGeneratorOptions

  import type { HeaderGeneratorOptions } from 'got-scraping';

  Type definition for configuring `headerGeneratorOptions` when customizing browser-like headers.
• getAgents

  import { getAgents } from 'got-scraping';

  Exports a function to retrieve the internal HTTP/HTTPS agents used by `got-scraping`, which can be useful for advanced agent management or debugging purposes.

Quickstart

Demonstrates a basic GET request using `gotScraping` with proxy support and custom header generation options, showing how to fetch and log response data.

import { gotScraping } from 'got-scraping';

async function fetchExampleData() {
    try {
        const response = await gotScraping.get({
            url: 'https://httpbin.org/headers', // A simple endpoint to inspect request headers
            proxyUrl: process.env.PROXY_URL ?? 'http://username:password@proxy.example.com:8000',
            useHeaderGenerator: true,
            headerGeneratorOptions: {
                browsers: [
                    { name: 'chrome', minVersion: 90, maxVersion: 99 }
                ],
                devices: ['desktop'],
                locales: ['en-US', 'de-DE'],
                operatingSystems: ['windows', 'macos']
            },
            // Standard Got options are also available
            timeout: { request: 30000 }, // 30-second total request timeout
            retry: { limit: 2, methods: ['GET'] }, // Retry GET requests up to 2 times
            headers: {
                'X-Custom-Header': 'Scraping-Demo-Request'
            }
        });

        console.log('Status Code:', response.statusCode);
        console.log('Response Body (first 500 chars):', response.body.substring(0, 500));
        // Further processing of `response.body` (e.g., JSON.parse) would happen here.
    } catch (error) {
        console.error('Failed to fetch data:', error instanceof Error ? error.message : error);
    }
}

fetchExampleData();