{"id":14761,"library":"node-worker-pool","title":"Node Worker Pool","description":"node-worker-pool is a JavaScript library for Node.js that provides a robust mechanism for managing a pool of child worker processes. It's designed for scenarios involving numerous highly parallelizable tasks, utilizing an exclusive message-passing paradigm rather than shared memory for inter-process communication. The library, currently at version 3.0.2, allows developers to define custom worker executables (though helper libraries are currently Node.js-specific) that communicate over `stdin`/`stdout` using a defined protocol. While its release cadence isn't explicitly documented, it focuses on stability and efficient task distribution. Key differentiators include its explicit message-passing design and the ability to use external executables as workers, offering flexibility beyond typical Node.js `worker_threads` when process isolation and custom language workers are desired.","status":"maintenance","version":"3.0.2","language":"javascript","source_language":"en","source_url":"https://github.com/jeffmo/node-worker-pool","tags":["javascript"],"install":[{"cmd":"npm install node-worker-pool","lang":"bash","label":"npm"},{"cmd":"yarn add node-worker-pool","lang":"bash","label":"yarn"},{"cmd":"pnpm add node-worker-pool","lang":"bash","label":"pnpm"}],"dependencies":[],"imports":[{"note":"The library primarily uses CommonJS `require` syntax. As of v3, ESM `import` is not officially supported and will likely result in a runtime error.","wrong":"import WorkerPool from 'node-worker-pool';","symbol":"WorkerPool","correct":"const WorkerPool = require('node-worker-pool');"},{"note":"Specific utilities for implementing a Node.js worker are exposed via a sub-path import, strictly using CommonJS `require`.","wrong":"import * as workerUtils from 'node-worker-pool/nodeWorkerUtils';","symbol":"workerUtils","correct":"const workerUtils = require('node-worker-pool/nodeWorkerUtils');"},{"note":"Functions like `startWorker` and `respondWithError` are named exports from `nodeWorkerUtils` and must be destructured using CommonJS `require`.","wrong":"import { startWorker, respondWithError } from 'node-worker-pool/nodeWorkerUtils';","symbol":"{ startWorker, respondWithError }","correct":"const { startWorker, respondWithError } = require('node-worker-pool/nodeWorkerUtils');"}],"quickstart":{"code":"const fs = require('fs');\nconst path = require('path');\nconst WorkerPool = require('node-worker-pool');\n\nconst workerScriptContent = `\nvar workerUtils = require('node-worker-pool/nodeWorkerUtils');\n\nvar initData;\nfunction onInitialize(data) {\n  initData = data;\n}\n\nfunction onMessage(data) {\n  return {\n    initData: initData,\n    receivedData: data,\n    pid: process.pid\n  };\n}\n\nif (require.main === module) {\n  try {\n    workerUtils.startWorker(onInitialize, onMessage);\n  } catch (e) {\n    workerUtils.respondWithError(e);\n  }\n}\n`;\n\n// Create a temporary worker file\nconst workerFilePath = path.join(__dirname, 'temp-worker.js');\nfs.writeFileSync(workerFilePath, workerScriptContent);\n\nasync function runPool() {\n  console.log('Starting worker pool...');\n  const workerPool = new WorkerPool(\n    2, // Use 2 workers for demonstration\n    process.execPath, // Path to the node binary\n    workerFilePath,   // Path to the temporary worker script\n    {\n      initData: { someUsefulConstant: 42 }\n    }\n  );\n\n  try {\n    console.log('Sending messages to workers...');\n    const responses = await Promise.all([\n      workerPool.sendMessage({ message: 'Hello from main 1!' }),\n      workerPool.sendMessage({ message: 'Hello from main 2!' }),\n      workerPool.sendMessage({ message: 'Hello from main 3!' })\n    ]);\n\n    responses.forEach((response, index) => {\n      console.log(`Response ${index + 1}:`, response);\n    });\n\n    console.log('Shutting down worker pool...');\n    await workerPool.shutDown();\n    console.log('All worker processes have now been killed.');\n  } catch (error) {\n    console.error('An error occurred:', error);\n  } finally {\n    // Clean up the temporary worker file\n    fs.unlinkSync(workerFilePath);\n    process.exit(0);\n  }\n}\n\nrunPool();","lang":"javascript","description":"This quickstart demonstrates how to set up a `WorkerPool` with a dynamically created worker script, send messages to the pool, and process responses, showing inter-process communication."},"warnings":[{"fix":"Adhere strictly to the `nodeWorkerUtils` helper library for Node.js workers. For non-Node.js workers, reverse-engineer the `stdin`/`stdout` messaging protocol by inspecting `node-worker-pool`'s source code.","message":"The communication protocol between the main process and workers is not officially documented, making it challenging to implement workers in languages other than Node.js or to debug complex communication issues.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Always ensure the return value of `onMessage` is a serializable object. Wrap any primitive return values in an object, e.g., `return { result: 'your_value' };`.","message":"Worker functions (`onInitialize`, `onMessage`) must only return plain JavaScript objects. Returning primitive values (strings, numbers, booleans) or `null`/`undefined` from `onMessage` will lead to a 'Worker protocol error: Response must be an object'.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Use absolute paths or paths relative to `process.cwd()` for both `process.execPath` (for the Node.js binary) and your worker script path (e.g., `path.join(__dirname, 'worker.js')`). Implement robust error handling around worker startup.","message":"The `workerExecutablePath` and `workerScriptPath` arguments to `WorkerPool` must be correct and accessible from the main process. Incorrect paths will result in worker startup failures that might be difficult to diagnose.","severity":"gotcha","affected_versions":">=1.0.0"}],"env_vars":null,"last_verified":"2026-04-19T00:00:00.000Z","next_check":"2026-07-18T00:00:00.000Z","problems":[{"fix":"Run `npm install node-worker-pool` or `yarn add node-worker-pool` in your project directory. Ensure your Node.js module resolution paths are correctly configured.","cause":"The `node-worker-pool` package is not installed or not accessible from the current working directory.","error":"Error: Cannot find module 'node-worker-pool'"},{"fix":"Add `const WorkerPool = require('node-worker-pool');` at the top of your file where `WorkerPool` is used.","cause":"The `WorkerPool` class was used without being correctly `require`d into the scope.","error":"ReferenceError: WorkerPool is not defined"},{"fix":"Modify your `onMessage` function in the worker script to always return an object, e.g., `return { data: yourResult };`.","cause":"The `onMessage` function in the worker script returned a value that was not a plain JavaScript object.","error":"Worker protocol error: Response must be an object"},{"fix":"Verify that `process.execPath` correctly points to your Node.js binary and that the `workerScriptPath` argument correctly points to your worker script file. Use absolute paths for robustness.","cause":"The path provided for the worker executable (e.g., `process.execPath` or a custom binary) or the worker script itself is incorrect or the file does not exist.","error":"Error: spawn [worker executable path] ENOENT"}],"ecosystem":"npm"}