Micromark Utility for Chunked Array Operations

Common errors

• Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /path/to/node_modules/micromark-util-chunked/index.js

  cause Attempting to import `micromark-util-chunked` using CommonJS `require()` syntax.
  fix Change `const { push } = require('micromark-util-chunked');` to `import { push } from 'micromark-util-chunked';`

• TypeError: (0 , micromark_util_chunked__WEBPACK_IMPORTED_MODULE_0__.splice) is not a function

  cause Incorrect ESM import syntax, often trying to destructure a default export or using `import * as` when the module only provides named exports.
  fix Ensure you are using named imports: `import { splice } from 'micromark-util-chunked';`

• RangeError: Maximum call stack size exceeded

  cause Directly using `Array.prototype.splice` or `Array.prototype.push` with a very large number of arguments or an extremely large array, causing a V8 engine stack overflow.
  fix Refactor your array manipulation logic to use `micromark-util-chunked`'s `push` and `splice` functions, which are optimized for large arrays.

Warnings

• breaking This package is ESM-only. Attempting to use `require()` for importing will result in an `ERR_REQUIRE_ESM` error. This has been the case since its initial major release.
  Fix: Migrate your import statements to ES module syntax: `import { push, splice } from 'micromark-util-chunked';`
• gotcha Using native `Array.prototype.splice` with an extremely large number of items (e.g., 100,000+) can lead to a 'RangeError: Maximum call stack size exceeded' in V8, which this package is designed to prevent.
  Fix: Always use `micromark-util-chunked`'s `splice` function when dealing with potentially giant arrays to avoid V8's argument limits.
• gotcha The `push` function can return a *new* array reference if the `list` argument was initially empty, instead of modifying `list` in place. Always reassign the return value of `push`.
  Fix: Ensure you reassign the result of `push` back to your variable: `myArray = push(myArray, newItems);`

Install

• npm install micromark-util-chunked npm
• yarn add micromark-util-chunked yarn
• pnpm add micromark-util-chunked pnpm

Imports

• push
  wrong:

  const { push } = require('micromark-util-chunked')

  correct:

  import { push } from 'micromark-util-chunked'

  This package is ESM-only. CommonJS `require` will result in an `ERR_REQUIRE_ESM` error. There is no default export.
• splice
  wrong:

  import splice from 'micromark-util-chunked'

  correct:

  import { splice } from 'micromark-util-chunked'

  The `splice` function is a named export. Attempting to use a default import will fail.
• push, splice
  wrong:

  import * as chunked from 'micromark-util-chunked'; chunked.push(...);

  correct:

  import { push, splice } from 'micromark-util-chunked'

  While `import * as` works, directly named imports are generally preferred for clarity and tree-shaking benefits for this module's two primary exports.

Quickstart

This quickstart demonstrates how to use `push` and `splice` from `micromark-util-chunked` to efficiently manage large arrays of 'events' (e.g., markdown tokens), avoiding V8 limitations.

import { push, splice } from 'micromark-util-chunked';

// Simulate a context object often used in micromark tokenizers
const context = { document: 'Some markdown string with lots of tokens...' };

// Initialize an array of "events" (e.g., tokens or state changes).
// Start with a moderately large array to demonstrate the utility's purpose.
let events: Array<[string, any, typeof context]> = [];

// Populate with some initial data using `push`
for (let i = 0; i < 5000; i++) {
  events = push(events, [['text', { type: 'text', value: `item-${i}` }, context]]);
}

console.log(`Initial events length: ${events.length}`);
// Expected output: Initial events length: 5000

// Demonstrate `push` with a large chunk of new items, which avoids V8 stack limits
const newItemsToAdd: Array<[string, any, typeof context]> = [];
for (let i = 0; i < 10000; i++) {
  newItemsToAdd.push(['token', { type: 'new_token', value: `added-${i}` }, context]);
}
events = push(events, newItemsToAdd);
console.log(`Length after large push: ${events.length}`);
// Expected output: Length after large push: 15000

// Demonstrate `splice`: removing existing items and inserting new ones.
// Let's remove 500 items from index 1000 and insert 2000 new ones.
const itemsToInsert: Array<[string, any, typeof context]> = [];
for (let i = 0; i < 2000; i++) {
  itemsToInsert.push(['insert', { type: 'inserted', value: `inserted-${i}` }, context]);
}

const startIndex = 1000;
const itemsToRemove = 500;
splice(events, startIndex, itemsToRemove, itemsToInsert);
console.log(`Length after splice: ${events.length}`);
// Expected output: Length after splice: 16500 (15000 - 500 + 2000)

// Verify a few items around the spliced area
console.log('Item before splice start:', events[startIndex - 1]);
console.log('First inserted item:', events[startIndex]);
console.log('Last inserted item:', events[startIndex + itemsToInsert.length - 1]);
console.log('Item after inserted block:', events[startIndex + itemsToInsert.length]);

// Example of a small final push
events = push(events, [['final', { type: 'end' }, context]]);
console.log(`Length after final push: ${events.length}`);
// Expected output: Length after final push: 16501