micromark Subtokenization Utility

2.1.0 · active · verified Sun Apr 19

micromark-util-subtokenize is a low-level utility within the micromark ecosystem, designed to handle the internal process of tokenizing subtokens during markdown parsing. It provides the `subtokenize` function, which is crucial for extending micromark with custom parsing behaviors, particularly for nested content or specific markdown constructs. This package is currently at version 2.1.0 and is actively maintained as part of the unified collective's micromark project. Releases are typically coordinated with other micromark packages, following a frequent cadence in response to bug fixes, performance improvements, and new feature development within the core parser. Its primary differentiator is its specialized role in the micromark parsing pipeline, making it essential for advanced extension authors rather than direct end-user application.

Common errors

```
Error [ERR_REQUIRE_ESM]: require() of ES Module .../node_modules/micromark-util-subtokenize/index.js from ... not supported.
```
cause Attempting to import an ESM-only package using CommonJS `require()`.

fix Switch to `import` statements and ensure your project or file is treated as an ES module. For example, add `"type": "module"` to your `package.json` or use `.mjs` file extensions.
```
TypeError: subtokenize is not a function
```
cause This typically occurs if `micromark-util-subtokenize` was imported incorrectly, e.g., using `import subtokenize from '...'`, or if it's an environment issue.

fix Ensure you are using named imports: `import { subtokenize } from 'micromark-util-subtokenize'`. This package has no default export.

Warnings

breaking This package is ESM-only. Attempting to `require()` it in a CommonJS environment will result in an error.
Fix: Ensure your project is configured for ESM (e.g., `"type": "module"` in `package.json`) or use dynamic `import()` if you must consume it from CommonJS.
gotcha This package is an internal utility for `micromark` and is generally not intended for direct use by consumers unless developing custom `micromark` extensions. Incorrect usage outside its intended context can lead to unexpected parsing behavior or errors.
Fix: Consult the `micromark` extension development guide before using this utility directly. Most users should interact with `micromark` through its main API or higher-level utilities.
breaking The package is compatible with Node.js 16+ and `micromark@3`. Using it with older Node.js versions or incompatible `micromark` major versions may lead to runtime issues or unexpected behavior.
Fix: Upgrade your Node.js environment to version 16 or newer and ensure your `micromark` dependency is version 3 or higher for full compatibility.
gotcha Version 2.0.3 included a refactor to omit a spread operator that was causing crashes. Earlier versions might exhibit stability issues related to this, particularly in specific environments or with complex markdown structures.
Fix: Always use the latest patch release of `micromark-util-subtokenize` (currently 2.1.0) to ensure stability and benefit from bug fixes and performance improvements.

Install

npm install micromark-util-subtokenize npm
yarn add micromark-util-subtokenize yarn
pnpm add micromark-util-subtokenize pnpm

Imports

subtokenize
wrong:
```
const subtokenize = require('micromark-util-subtokenize')
```
correct:
```
import { subtokenize } from 'micromark-util-subtokenize'
```
This package is ESM-only and does not provide a CommonJS export. There is no default export, so curly braces are required.

Quickstart

This example demonstrates how `subtokenize` is typically used within a micromark resolver function to process nested token events.

import {subtokenize} from 'micromark-util-subtokenize'
import type {Event, Resolver} from 'micromark-util-types'

/**
 * A resolver function type from micromark-util-types.
 * For custom micromark extensions, resolvers process a list of events.
 * In this example, `subtokenize` is called on the events to parse subcontent.
 *
 * @type {Resolver}
 */
function resolveContent(events: Array<Event>): Array<Event> {
  // `subtokenize` processes the event stream to tokenize any nested content.
  // This makes sure that, for instance, definitions are parsed before
  // their references in paragraphs.
  subtokenize(events)
  return events
}

view raw JSON →