NLCST Normalize Utility

4.0.0 · active · verified Sun Apr 19

nlcst-normalize is a utility within the unified ecosystem, specifically designed for working with Natural Language Concrete Syntax Tree (NLCST) nodes. Its primary function is to serialize and clean words, making them easier to compare consistently. The current stable version is 4.0.0. The package generally follows a release cadence tied to Node.js LTS versions, dropping support for unmaintained Node.js versions with major releases. Key differentiators include its integration with the broader unified collective for natural language processing, its ability to normalize various word forms (e.g., smart vs. straight apostrophes) and handle specific punctuation like hyphens and apostrophes based on configurable options. This makes it particularly useful for tasks such as building keyword matchers or creating indexes where slight variations in word formatting need to be reconciled.

Common errors

```
TypeError: require is not a function
```
cause Attempting to use `require()` to import nlcst-normalize in a module context where only ESM is supported.

fix Change `const { normalize } = require('nlcst-normalize')` to `import { normalize } from 'nlcst-normalize'`.
```
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './lib/index.js' is not defined by "exports" in ...
```
cause Attempting to import from a non-public, internal path of the package, which is disallowed due to the `exports` field in package.json.

fix Always import symbols directly from the main package entry point: `import { normalize } from 'nlcst-normalize'`.
```
Property 'NormalizeOptions' does not exist on type 'typeof import("nlcst-normalize")'
```
cause Using the deprecated `NormalizeOptions` type after upgrading to version 4.0.0 or later.

fix Replace `NormalizeOptions` with `Options` in your TypeScript type imports and declarations.

Warnings

breaking Version 4.0.0 of nlcst-normalize requires Node.js version 16 or higher. Older Node.js versions are no longer supported.
Fix: Upgrade your Node.js environment to version 16 or newer. For projects that cannot upgrade, consider using nlcst-normalize@^3.
breaking nlcst-normalize switched to being an ESM-only package starting from version 3.0.0. CommonJS `require()` statements will fail.
Fix: Migrate your codebase to use ECMAScript Modules (ESM) syntax with `import` statements. Ensure your project is configured for ESM (e.g., `"type": "module"` in package.json or using `.mjs` file extensions).
breaking Version 4.0.0 removed the `NormalizeOptions` type. The correct type for configuration is now `Options`.
Fix: Update your TypeScript code to use `Options` instead of `NormalizeOptions` when defining or importing the configuration type.
gotcha When normalizing, smart apostrophes (`’`) are always converted to straight apostrophes (`'`) and then removed by default, along with hyphens (`-`), unless `allowApostrophes` or `allowDashes` options are explicitly set to `true`.
Fix: If you need to preserve apostrophes or hyphens, pass `{ allowApostrophes: true }` or `{ allowDashes: true }` respectively in the options object to the `normalize` function.

Install

npm install nlcst-normalize npm
yarn add nlcst-normalize yarn
pnpm add nlcst-normalize pnpm

Imports

normalize
wrong:
```
const normalize = require('nlcst-normalize').normalize
```
correct:
```
import { normalize } from 'nlcst-normalize'
```
nlcst-normalize is an ESM-only package since v3.0.0. CommonJS 'require' is not supported.
Options
wrong:
```
import { Options } from 'nlcst-normalize'
```
correct:
```
import type { Options } from 'nlcst-normalize'
```
For TypeScript, 'Options' is a type and should be imported using 'import type'.
normalize (Deno/Browser)
wrong:
```
import {normalize} from 'nlcst-normalize'
```
correct:
```
import {normalize} from 'https://esm.sh/nlcst-normalize@4'
```
When using in Deno or browsers via esm.sh, ensure you specify the version in the URL.

Quickstart

This quickstart demonstrates how to use the `normalize` function with various string inputs, including options to control punctuation stripping, and how to pass an NLCST `WordNode` object for normalization.

import { normalize } from 'nlcst-normalize';

// Normalize simple strings
console.log(normalize("Don't")); // => 'dont'
console.log(normalize('Don’t')); // => 'dont'

// Normalize with options to retain specific punctuation
console.log(normalize('Don’t', { allowApostrophes: true })); // => 'don\'t'
console.log(normalize('Block-level')); // => 'blocklevel'
console.log(normalize('Block-level', { allowDashes: true })); // => 'block-level'

// Normalize an NLCST WordNode object
const wordNode = {
  type: 'WordNode',
  children: [
    { type: 'TextNode', value: 'Example' },
    { type: 'PunctuationNode', value: '-' },
    { type: 'TextNode', value: 'word' }
  ]
};
console.log(normalize(wordNode)); // => 'exampleword'

view raw JSON →