{"id":11158,"library":"js-tokens","title":"js-tokens","description":"js-tokens is a JavaScript tokenizer that leverages regular expressions to parse JavaScript code into a stream of tokens. It is known for its tiny footprint, lenient parsing approach, and robustness, designed to 'never fail' even on malformed input, making it suitable for tools that need to process potentially incomplete or invalid JavaScript. The current stable version is 10.0.0. While not strictly spec-compliant in all edge cases, its 'almost spec-compliant' nature makes it highly practical for tasks like syntax highlighting, basic static analysis, or code transformation where full AST parsing is overkill. It supports the latest ECMAScript features up to ES2025 and also offers an option for JSX support. The project maintains a steady release cadence for new ECMAScript features and bug fixes, with major versions often introducing changes to module formats or API surface to align with ecosystem shifts.","status":"active","version":"10.0.0","language":"javascript","source_language":"en","source_url":"https://github.com/lydell/js-tokens","tags":["javascript","JavaScript","js","ECMAScript","es","token","tokens","tokenize","tokenizer","typescript"],"install":[{"cmd":"npm install js-tokens","lang":"bash","label":"npm"},{"cmd":"yarn add js-tokens","lang":"bash","label":"yarn"},{"cmd":"pnpm add js-tokens","lang":"bash","label":"pnpm"}],"dependencies":[],"imports":[{"note":"Since v10.0.0, js-tokens is an ES Module. While Node.js 20.19.0+ supports `require(esm)`, it's generally recommended to use `import` for full compatibility and better tooling. For older CommonJS environments, dynamic `import()` or specific bundler configurations are necessary.","wrong":"const jsTokens = require('js-tokens');","symbol":"jsTokens","correct":"import jsTokens from 'js-tokens';"},{"note":"js-tokens ships its own TypeScript types since v5+, making `@types/js-tokens` unnecessary. Ensure `esModuleInterop: true` in your `tsconfig.json` for seamless type resolution.","wrong":"import { Token } from 'js-tokens';","symbol":"Token type","correct":"import type { Token } from 'js-tokens';"},{"note":"The tokenizer supports an optional `jsx: true` flag to enable JSX tokenization, affecting how angle brackets and identifiers are parsed.","symbol":"jsTokens (JSX option)","correct":"import jsTokens from 'js-tokens';\n\njsTokens(jsString, { jsx: true });"}],"quickstart":{"code":"import jsTokens from 'js-tokens';\n\nconst jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, \"\\t\")';\n\nconst tokens = Array.from(jsTokens(jsString));\n\nconsole.log(tokens.map(token => token.value).join('|'));\n// Expected output: JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |\"\\t\"|)\n\n// Example with JSX support\nconst jsxString = 'const element = <div>Hello, {name}!</div>;';\nconst jsxTokens = Array.from(jsTokens(jsxString, { jsx: true }));\nconsole.log(jsxTokens.map(token => token.type + ':' + token.value).join(', '));","lang":"typescript","description":"Demonstrates basic tokenization of a JavaScript string, extracting token values. Also shows enabling JSX support."},"warnings":[{"fix":"For CommonJS projects, use dynamic `import('js-tokens')` or ensure you are on Node.js 20.19.0 or later which supports `require(esm)`. For new projects, adopt `import` statements and ensure your project is configured for ESM.","message":"js-tokens v10.0.0 transitioned to being an ES Module (ESM) package, setting `\"type\": \"module\"` in its `package.json`. This fundamentally changes how it should be imported.","severity":"breaking","affected_versions":">=10.0.0"},{"fix":"Update your code to call `jsTokens(string, options?)` and iterate over the returned generator function, rather than using a regex directly.","message":"In v6.0.0, the main export changed from a direct regular expression to a generator function. Code relying on the raw regex pattern or its `matchToToken` helper will break.","severity":"breaking","affected_versions":">=6.0.0 <10.0.0"},{"fix":"Ensure your environment runs Node.js 10 or newer. Review code that relies on specific token structures, names, or how whitespace was previously handled, and adjust to the updated specification-aligned token output.","message":"Version 4.0.0 introduced breaking changes including requiring Node.js 10+ (due to Unicode property escapes), aligning token names with the ECMAScript spec, and treating whitespace/line terminators as separate tokens.","severity":"breaking","affected_versions":">=4.0.0 <6.0.0"},{"fix":"For scenarios requiring absolute strict ECMAScript compliance, be aware of these limitations and consider a full-fledged AST parser. For most general-purpose tokenization (like syntax highlighting), `js-tokens`'s leniency is often an advantage.","message":"The tokenizer is described as 'almost spec-compliant' and takes 'a couple of shortcuts'. While it passes most `test262-parser-tests`, it has documented limitations (e.g., regarding `>>` and `>>>` operators in TypeScript type annotations, or HTML-like comments).","severity":"gotcha","affected_versions":"*"},{"fix":"Avoid processing extraordinarily large single tokens. For inputs with potential extreme length, consider pre-validation or chunking strategies if feasible.","message":"While designed to 'never fail', extremely large or malformed inputs (e.g., string literals with millions of repeated characters) can cause the underlying JavaScript regex engine to throw a `RangeError: Maximum call stack size exceeded` or similar error. This is a limitation of the regex engine, not `js-tokens`'s logic.","severity":"gotcha","affected_versions":"*"}],"env_vars":null,"last_verified":"2026-04-19T00:00:00.000Z","next_check":"2026-07-18T00:00:00.000Z","problems":[{"fix":"For CommonJS projects, either enable `\"type\": \"module\"` in your `package.json` (requires Node.js 12+), or use dynamic `await import('js-tokens')`. For older Node.js versions or strict CommonJS, you might need Node.js 20.19.0+ to directly `require('js-tokens')`. Otherwise, refactor your project to use ES Modules.","cause":"Attempting to use ES module `import` syntax (`import jsTokens from 'js-tokens'`) in a CommonJS (`.js` file without `\"type\": \"module\"` in `package.json`) environment, especially on older Node.js versions.","error":"SyntaxError: Cannot use import statement outside a module"},{"fix":"In CommonJS (for Node.js < 20.19.0 or without `\"type\": \"module\"`): use `const { default: jsTokens } = require(\"js-tokens\");` or `const jsTokens = require(\"js-tokens\").default;`. In ESM: ensure you use `import jsTokens from \"js-tokens\";`. For TypeScript, ensure `\"esModuleInterop\": true` in your `tsconfig.json`.","cause":"This usually occurs when attempting to use `require('js-tokens')` in a CommonJS context for `js-tokens` versions that export a default ES module, and Node.js doesn't correctly resolve the default export.","error":"TypeError: jsTokens is not a function"}],"ecosystem":"npm"}