js-tokens

10.0.0 · active · verified Sun Apr 19

js-tokens is a JavaScript tokenizer that leverages regular expressions to parse JavaScript code into a stream of tokens. It is known for its tiny footprint, lenient parsing approach, and robustness, designed to 'never fail' even on malformed input, making it suitable for tools that need to process potentially incomplete or invalid JavaScript. The current stable version is 10.0.0. While not strictly spec-compliant in all edge cases, its 'almost spec-compliant' nature makes it highly practical for tasks like syntax highlighting, basic static analysis, or code transformation where full AST parsing is overkill. It supports the latest ECMAScript features up to ES2025 and also offers an option for JSX support. The project maintains a steady release cadence for new ECMAScript features and bug fixes, with major versions often introducing changes to module formats or API surface to align with ecosystem shifts.

Common errors

```
SyntaxError: Cannot use import statement outside a module
```
cause Attempting to use ES module `import` syntax (`import jsTokens from 'js-tokens'`) in a CommonJS (`.js` file without `"type": "module"` in `package.json`) environment, especially on older Node.js versions.

fix For CommonJS projects, either enable `"type": "module"` in your `package.json` (requires Node.js 12+), or use dynamic `await import('js-tokens')`. For older Node.js versions or strict CommonJS, you might need Node.js 20.19.0+ to directly `require('js-tokens')`. Otherwise, refactor your project to use ES Modules.
```
TypeError: jsTokens is not a function
```
cause This usually occurs when attempting to use `require('js-tokens')` in a CommonJS context for `js-tokens` versions that export a default ES module, and Node.js doesn't correctly resolve the default export.

fix In CommonJS (for Node.js < 20.19.0 or without `"type": "module"`): use `const { default: jsTokens } = require("js-tokens");` or `const jsTokens = require("js-tokens").default;`. In ESM: ensure you use `import jsTokens from "js-tokens";`. For TypeScript, ensure `"esModuleInterop": true` in your `tsconfig.json`.

Warnings

breaking js-tokens v10.0.0 transitioned to being an ES Module (ESM) package, setting `"type": "module"` in its `package.json`. This fundamentally changes how it should be imported.
Fix: For CommonJS projects, use dynamic `import('js-tokens')` or ensure you are on Node.js 20.19.0 or later which supports `require(esm)`. For new projects, adopt `import` statements and ensure your project is configured for ESM.
breaking In v6.0.0, the main export changed from a direct regular expression to a generator function. Code relying on the raw regex pattern or its `matchToToken` helper will break.
Fix: Update your code to call `jsTokens(string, options?)` and iterate over the returned generator function, rather than using a regex directly.
breaking Version 4.0.0 introduced breaking changes including requiring Node.js 10+ (due to Unicode property escapes), aligning token names with the ECMAScript spec, and treating whitespace/line terminators as separate tokens.
Fix: Ensure your environment runs Node.js 10 or newer. Review code that relies on specific token structures, names, or how whitespace was previously handled, and adjust to the updated specification-aligned token output.
gotcha The tokenizer is described as 'almost spec-compliant' and takes 'a couple of shortcuts'. While it passes most `test262-parser-tests`, it has documented limitations (e.g., regarding `>>` and `>>>` operators in TypeScript type annotations, or HTML-like comments).
Fix: For scenarios requiring absolute strict ECMAScript compliance, be aware of these limitations and consider a full-fledged AST parser. For most general-purpose tokenization (like syntax highlighting), `js-tokens`'s leniency is often an advantage.
gotcha While designed to 'never fail', extremely large or malformed inputs (e.g., string literals with millions of repeated characters) can cause the underlying JavaScript regex engine to throw a `RangeError: Maximum call stack size exceeded` or similar error. This is a limitation of the regex engine, not `js-tokens`'s logic.
Fix: Avoid processing extraordinarily large single tokens. For inputs with potential extreme length, consider pre-validation or chunking strategies if feasible.

Install

npm install js-tokens npm
yarn add js-tokens yarn
pnpm add js-tokens pnpm

Imports

jsTokens
wrong:
```
const jsTokens = require('js-tokens');
```
correct:
```
import jsTokens from 'js-tokens';
```
Since v10.0.0, js-tokens is an ES Module. While Node.js 20.19.0+ supports `require(esm)`, it's generally recommended to use `import` for full compatibility and better tooling. For older CommonJS environments, dynamic `import()` or specific bundler configurations are necessary.
Token type
wrong:
```
import { Token } from 'js-tokens';
```
correct:
```
import type { Token } from 'js-tokens';
```
js-tokens ships its own TypeScript types since v5+, making `@types/js-tokens` unnecessary. Ensure `esModuleInterop: true` in your `tsconfig.json` for seamless type resolution.
jsTokens (JSX option)
```
import jsTokens from 'js-tokens';

jsTokens(jsString, { jsx: true });
```
The tokenizer supports an optional `jsx: true` flag to enable JSX tokenization, affecting how angle brackets and identifiers are parsed.

Quickstart

Demonstrates basic tokenization of a JavaScript string, extracting token values. Also shows enabling JSX support.

import jsTokens from 'js-tokens';

const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\t")';

const tokens = Array.from(jsTokens(jsString));

console.log(tokens.map(token => token.value).join('|'));
// Expected output: JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)

// Example with JSX support
const jsxString = 'const element = <div>Hello, {name}!</div>;';
const jsxTokens = Array.from(jsTokens(jsxString, { jsx: true }));
console.log(jsxTokens.map(token => token.type + ':' + token.value).join(', '));

view raw JSON →