{"id":13215,"library":"gatsby-worker","title":"Gatsby Worker Pool Utility","description":"gatsby-worker is a utility package within the Gatsby ecosystem, currently at version `2.16.0`, designed for creating worker pools to offload CPU-intensive tasks into separate Node.js processes. Inspired by `jest-worker`, it enables efficient parallel execution, improving build times and overall performance for Gatsby sites. The package follows Gatsby's release cadence, typically aligning with major and minor Gatsby releases, which often see minor versions published every two weeks for the core framework. It provides a type-safe API, allowing developers to define worker modules with explicit function signatures using TypeScript. Key features include queuing tasks on single or all workers, robust parent-worker messaging capabilities, and granular control over worker lifecycle and environment variables. It requires Node.js versions `>=18.0.0 <26` for operation.","status":"active","version":"2.16.0","language":"javascript","source_language":"en","source_url":"https://github.com/gatsbyjs/gatsby","tags":["javascript","gatsby","worker"],"install":[{"cmd":"npm install gatsby-worker","lang":"bash","label":"npm"},{"cmd":"yarn add gatsby-worker","lang":"bash","label":"yarn"},{"cmd":"pnpm add gatsby-worker","lang":"bash","label":"pnpm"}],"dependencies":[],"imports":[{"note":"Prefer ES Module import syntax. While `require.resolve` is used for worker path configuration, the library's primary API is designed for ES Modules.","wrong":"const { WorkerPool } = require('gatsby-worker')","symbol":"WorkerPool","correct":"import { WorkerPool } from 'gatsby-worker'"},{"note":"Used to conditionally execute code based on whether it's running in the main process or a worker thread.","wrong":"const isWorker = require('gatsby-worker').isWorker","symbol":"isWorker","correct":"import { isWorker } from 'gatsby-worker'"},{"note":"Enables type-safe messaging between parent and child worker processes. Define shared message types for best practice.","wrong":"const getMessenger = require('gatsby-worker').getMessenger","symbol":"getMessenger","correct":"import { getMessenger } from 'gatsby-worker'"}],"quickstart":{"code":"/* File: worker.ts */\n// This file defines the tasks that will be executed in a worker thread.\n// It should be a separate file resolved by the WorkerPool.\nexport async function heavyTask(param: string): Promise<string> {\n  // Simulate a CPU-intensive operation\n  let result = 0;\n  for (let i = 0; i < 10_000_000; i++) {\n    result += Math.sqrt(i);\n  }\n  return `Processed '${param}' with result: ${result.toFixed(2)}`;\n}\n\nexport async function setupStep(param: string): Promise<void> {\n  console.log(`Worker ${process.env.GATSBY_WORKER_ID || 'unknown'} setting up with: ${param}`);\n  // Simulate a heavy setup process\n  await new Promise(resolve => setTimeout(resolve, 100));\n}\n\n/* File: parent.ts */\nimport { WorkerPool } from 'gatsby-worker';\nimport * as path from 'path';\n\n// Assuming 'worker.ts' is compiled to 'worker.js' in the same directory for Node.js execution\nconst workerPath = path.resolve(__dirname, 'worker.js');\n\nasync function runWorkers() {\n  const workerPool = new WorkerPool<typeof import('./worker')>(\n    workerPath,\n    {\n      numWorkers: 2, // Use 2 worker threads\n      env: {\n        CUSTOM_ENV_VAR_TO_SET_IN_WORKER: 'foo',\n      },\n      silent: false, // Output from workers will be logged to parent's console\n    }\n  );\n\n  try {\n    console.log('--- Queuing setup on all workers ---');\n    // Queue a task on all workers for initial setup\n    await Promise.all(workerPool.all.setupStep('initial setup data'));\n    console.log('All workers setup complete.');\n\n    console.log('\\n--- Queuing heavy tasks ---');\n    // Queue a task on a single worker (gatsby-worker handles worker assignment)\n    const singleTaskPromise1 = workerPool.single.heavyTask('data-chunk-A');\n    const singleTaskPromise2 = workerPool.single.heavyTask('data-chunk-B');\n    const singleTaskPromise3 = workerPool.single.heavyTask('data-chunk-C');\n\n    const results = await Promise.all([\n      singleTaskPromise1,\n      singleTaskPromise2,\n      singleTaskPromise3\n    ]);\n    results.forEach(res => console.log(res));\n\n  } catch (error) {\n    console.error('An error occurred:', error);\n  } finally {\n    console.log('\\n--- Shutting down worker pool ---');\n    await workerPool.end(); // Shut down all workers\n    console.log('Worker pool shut down.');\n  }\n}\n\nrunWorkers();\n","lang":"typescript","description":"This quickstart demonstrates how to create a worker pool, define CPU-intensive tasks in a separate worker file, execute tasks on single or all workers, and gracefully shut down the pool, while also showcasing environment variable passing and logging."},"warnings":[{"fix":"Upgrade your Node.js runtime using a version manager like `nvm` (e.g., `nvm install 20 && nvm use 20`) or `volta` to a supported LTS version.","message":"Gatsby and its associated packages, including `gatsby-worker`, adhere to strict Node.js version compatibility policies. Major Gatsby releases often drop support for older Node.js versions, aligning with Node.js LTS schedules. Ensure your development and deployment environments use a compatible Node.js version (currently `>=18.0.0 <26`) to avoid installation or runtime errors.","severity":"breaking","affected_versions":">=2.0.0"},{"fix":"Always `await` the promises returned by `workerPool.single` or `workerPool.all` operations before calling `workerPool.end()` to ensure all tasks have a chance to complete. Implement `.catch()` for robust error handling on individual task promises.","message":"Calling `workerPool.end()` immediately after queuing tasks, or while tasks are still executing, will cause any pending or in-progress task promises to be rejected. This can lead to unhandled promise rejections or incomplete operations.","severity":"gotcha","affected_versions":">=2.0.0"},{"fix":"Reserve `gatsby-worker` for truly CPU-intensive computations that block the event loop, such as image processing, complex data transformations, or heavy computations. For I/O, rely on Node.js's native asynchronous capabilities.","message":"`gatsby-worker` is designed for CPU-bound tasks. Using worker threads for I/O-bound operations (e.g., heavy network requests, database queries that are already asynchronous) can introduce unnecessary overhead from serialization/deserialization and inter-process communication, potentially decreasing performance rather than improving it.","severity":"gotcha","affected_versions":">=2.0.0"},{"fix":"Always use `import { SymbolName } from 'gatsby-worker'` for importing components like `WorkerPool`. If in a CommonJS environment, ensure your build setup correctly transpiles or bundles ES Modules. For worker paths, `require.resolve()` remains appropriate.","message":"The primary API for `gatsby-worker` uses ES Module `import` syntax. Attempting to `require()` its named exports directly in a strict ES Module context will result in module resolution errors.","severity":"gotcha","affected_versions":">=2.0.0"}],"env_vars":null,"last_verified":"2026-04-19T00:00:00.000Z","next_check":"2026-07-18T00:00:00.000Z","problems":[{"fix":"Ensure your files are treated as ES Modules where `gatsby-worker` is imported and use `import { WorkerPool } from 'gatsby-worker';`. For `.js` files, this typically means having `\"type\": \"module\"` in your `package.json` or using `.mjs` file extensions.","cause":"Incorrect import syntax (often mixing CommonJS require with ES Module usage) or an environment mismatch where `gatsby-worker` is treated as CJS in an ESM context.","error":"TypeError: (0 , gatsby_worker_1.WorkerPool) is not a constructor"},{"fix":"Update your Node.js version to one supported by Gatsby (e.g., using `nvm install --lts && nvm use --lts` or `volta install node@20`). Consult Gatsby's documentation for the most up-to-date Node.js support matrix.","cause":"Your installed Node.js version does not meet the requirements specified in `gatsby-worker`'s `package.json` or Gatsby's overall compatibility policy.","error":"The engine \"node\" is incompatible with this module. Expected version \"X\" Got \"Y\""},{"fix":"Set `silent: false` in the `WorkerPool` options to allow worker stdout/stderr to be logged to the parent process's console for debugging. Add robust `try/catch` blocks around your worker function logic and ensure all promises are handled to prevent uncaught rejections.","cause":"An uncaught error or exception occurred within one of your worker threads, causing the process to terminate prematurely. This could be due to a bug in the worker's logic, resource exhaustion, or an unhandled promise rejection.","error":"Worker process exited with code 1"}],"ecosystem":"npm","meta_description":null,"install_score":null,"install_tag":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":null,"cli_name":""}