Node Worker Pool

Common errors

• Error: Cannot find module 'node-worker-pool'

  cause The `node-worker-pool` package is not installed or not accessible from the current working directory.
  fix Run `npm install node-worker-pool` or `yarn add node-worker-pool` in your project directory. Ensure your Node.js module resolution paths are correctly configured.

• ReferenceError: WorkerPool is not defined

  cause The `WorkerPool` class was used without being correctly `require`d into the scope.
  fix Add `const WorkerPool = require('node-worker-pool');` at the top of your file where `WorkerPool` is used.

• Worker protocol error: Response must be an object

  cause The `onMessage` function in the worker script returned a value that was not a plain JavaScript object.
  fix Modify your `onMessage` function in the worker script to always return an object, e.g., `return { data: yourResult };`.

• Error: spawn [worker executable path] ENOENT

  cause The path provided for the worker executable (e.g., `process.execPath` or a custom binary) or the worker script itself is incorrect or the file does not exist.
  fix Verify that `process.execPath` correctly points to your Node.js binary and that the `workerScriptPath` argument correctly points to your worker script file. Use absolute paths for robustness.

Warnings

• gotcha The communication protocol between the main process and workers is not officially documented, making it challenging to implement workers in languages other than Node.js or to debug complex communication issues.
  Fix: Adhere strictly to the `nodeWorkerUtils` helper library for Node.js workers. For non-Node.js workers, reverse-engineer the `stdin`/`stdout` messaging protocol by inspecting `node-worker-pool`'s source code.
• gotcha Worker functions (`onInitialize`, `onMessage`) must only return plain JavaScript objects. Returning primitive values (strings, numbers, booleans) or `null`/`undefined` from `onMessage` will lead to a 'Worker protocol error: Response must be an object'.
  Fix: Always ensure the return value of `onMessage` is a serializable object. Wrap any primitive return values in an object, e.g., `return { result: 'your_value' };`.
• gotcha The `workerExecutablePath` and `workerScriptPath` arguments to `WorkerPool` must be correct and accessible from the main process. Incorrect paths will result in worker startup failures that might be difficult to diagnose.
  Fix: Use absolute paths or paths relative to `process.cwd()` for both `process.execPath` (for the Node.js binary) and your worker script path (e.g., `path.join(__dirname, 'worker.js')`). Implement robust error handling around worker startup.

Install

• npm install node-worker-pool npm
• yarn add node-worker-pool yarn
• pnpm add node-worker-pool pnpm

Imports

• WorkerPool
  wrong:

  import WorkerPool from 'node-worker-pool';

  correct:

  const WorkerPool = require('node-worker-pool');

  The library primarily uses CommonJS `require` syntax. As of v3, ESM `import` is not officially supported and will likely result in a runtime error.
• workerUtils
  wrong:

  import * as workerUtils from 'node-worker-pool/nodeWorkerUtils';

  correct:

  const workerUtils = require('node-worker-pool/nodeWorkerUtils');

  Specific utilities for implementing a Node.js worker are exposed via a sub-path import, strictly using CommonJS `require`.
• { startWorker, respondWithError }
  wrong:

  import { startWorker, respondWithError } from 'node-worker-pool/nodeWorkerUtils';

  correct:

  const { startWorker, respondWithError } = require('node-worker-pool/nodeWorkerUtils');

  Functions like `startWorker` and `respondWithError` are named exports from `nodeWorkerUtils` and must be destructured using CommonJS `require`.

Quickstart

This quickstart demonstrates how to set up a `WorkerPool` with a dynamically created worker script, send messages to the pool, and process responses, showing inter-process communication.

const fs = require('fs');
const path = require('path');
const WorkerPool = require('node-worker-pool');

const workerScriptContent = `
var workerUtils = require('node-worker-pool/nodeWorkerUtils');

var initData;
function onInitialize(data) {
  initData = data;
}

function onMessage(data) {
  return {
    initData: initData,
    receivedData: data,
    pid: process.pid
  };
}

if (require.main === module) {
  try {
    workerUtils.startWorker(onInitialize, onMessage);
  } catch (e) {
    workerUtils.respondWithError(e);
  }
}
`;

// Create a temporary worker file
const workerFilePath = path.join(__dirname, 'temp-worker.js');
fs.writeFileSync(workerFilePath, workerScriptContent);

async function runPool() {
  console.log('Starting worker pool...');
  const workerPool = new WorkerPool(
    2, // Use 2 workers for demonstration
    process.execPath, // Path to the node binary
    workerFilePath,   // Path to the temporary worker script
    {
      initData: { someUsefulConstant: 42 }
    }
  );

  try {
    console.log('Sending messages to workers...');
    const responses = await Promise.all([
      workerPool.sendMessage({ message: 'Hello from main 1!' }),
      workerPool.sendMessage({ message: 'Hello from main 2!' }),
      workerPool.sendMessage({ message: 'Hello from main 3!' })
    ]);

    responses.forEach((response, index) => {
      console.log(`Response ${index + 1}:`, response);
    });

    console.log('Shutting down worker pool...');
    await workerPool.shutDown();
    console.log('All worker processes have now been killed.');
  } catch (error) {
    console.error('An error occurred:', error);
  } finally {
    // Clean up the temporary worker file
    fs.unlinkSync(workerFilePath);
    process.exit(0);
  }
}

runPool();