Sharding Utility for CI Services

2.1.0 · maintenance · verified Sun Apr 19

ShardyMcShardFace is a JavaScript/TypeScript utility library designed to efficiently shard arrays of items across multiple parallel processes, primarily for Continuous Integration (CI) environments. It leverages standard CI environment variables like `CI_NODE_INDEX` and `CI_NODE_TOTAL` (via the `ci-parallel-vars` dependency) to determine the current shard and total shard count, distributing items as evenly as possible. The current stable version is 2.1.0, released in November 2019. Since then, the project appears to be in a maintenance-only state, with no new releases in over five years. Key features include stable, random-seeded item distribution, graceful handling of scenarios where the item count is less than the total shard count, and full TypeScript support, ensuring type safety. It offers both a programmatic API for direct integration into JavaScript/TypeScript applications and a command-line interface for simple shell scripting. Its primary differentiator is its direct, opinionated integration with common CI parallelization parameters, simplifying the setup for distributed test suites or build steps.

Common errors

```
TypeError: Expected options to be an object, but got string.
```
cause Attempting to pass a seed string directly as the last argument to the `shard` function in versions 2.0.0 or later.

fix Wrap the seed string in an options object: `shard(items, { seed: 'your-seed' });`
```
Items are not being sharded; all items are returned in every execution.
```
cause The `CI_NODE_INDEX` or `CI_NODE_TOTAL` environment variables are not correctly set or are missing, causing the library to assume a single execution node.

fix Verify that your CI environment or local setup correctly defines `CI_NODE_INDEX` and `CI_NODE_TOTAL` (e.g., `process.env.CI_NODE_INDEX = '0'; process.env.CI_NODE_TOTAL = '2';`). The library uses `ci-parallel-vars` to detect these.

Warnings

breaking The `shard` function's last parameter changed from a string to an options map.
Fix: If you were passing a string as the last argument (e.g., for a seed), you must now pass an object `{ seed: 'your-seed' }` or similar. Refer to the updated function signature: `shardyMcShardFace(items, options)`.
gotcha This package relies heavily on CI environment variables (`CI_NODE_INDEX`, `CI_NODE_TOTAL`, etc.) to determine sharding parameters. If these are not set, it will default to a single shard (index 0 of 1).
Fix: Ensure that your CI system correctly sets these environment variables. For local development or testing, you may need to manually set `process.env.CI_NODE_INDEX` and `process.env.CI_NODE_TOTAL` or use the `concurrency` option within the API if available in newer versions.
gotcha The project has not seen a new release since November 2019. While functional, active development and rapid bug fixes are unlikely.
Fix: Assess the stability and existing features carefully. Consider contributing fixes or new features if needed, or fork the repository. Be aware that future JavaScript/TypeScript ecosystem changes might not be immediately supported.

Install

npm install shardy-mc-shard-face npm
yarn add shardy-mc-shard-face yarn
pnpm add shardy-mc-shard-face pnpm

Imports

shard

wrong:

import shardyMcShardFace from 'shardy-mc-shard-face';

correct:

import { shard } from 'shardy-mc-shard-face';

The primary sharding function is a named export. There is no default export.

shard (CommonJS)
```
const { shard } = require('shardy-mc-shard-face');
```
For CommonJS environments, destructure the named 'shard' export. This package ships with TypeScript types, but CommonJS is still supported in Node.js environments.
shardy (CLI)
```
cat items.txt | shardy shard
```
The CLI is installed as 'shardy'. Input is newline-separated to stdin, output is newline-separated to stdout. Environment variables like CI_NODE_INDEX and CI_NODE_TOTAL are crucial for its operation.

Quickstart

This quickstart demonstrates how to programmatically shard an array of items using `shardy-mc-shard-face`. It simulates CI environment variables and shows both basic usage and how to apply custom options like a stable seed.

import { shard as shardyMcShardFace } from 'shardy-mc-shard-face';

// Simulate CI environment variables for local testing
process.env.CI_NODE_INDEX = '0'; // Current shard index (0-based)
process.env.CI_NODE_TOTAL = '2'; // Total number of shards

const allItems = [
  'test_file_A.spec.ts',
  'test_file_B.spec.ts',
  'test_file_C.spec.ts',
  'test_file_D.spec.ts',
  'test_file_E.spec.ts',
  'test_file_F.spec.ts'
];

// Basic sharding, options are optional
const myShard = shardyMcShardFace(allItems);

console.log(`Running on CI node ${process.env.CI_NODE_INDEX} of ${process.env.CI_NODE_TOTAL}`);
console.log('My assigned shard:', myShard);

// Example with custom options and seed for stable distribution
const customShard = shardyMcShardFace(allItems, {
  throwOnEmpty: false, // Don't throw if shard is empty
  seed: 'my-stable-seed' // Ensures same items always go to same shard
});

console.log('My assigned shard with custom seed:', customShard);

view raw JSON →