Lindera WASM with Jieba Dictionary (Bundler)

Common errors

• TypeError: TokenizerBuilder is not a constructor

  cause The WebAssembly module was not initialized correctly, or the `__wbg_init()` promise was not awaited, leading to `TokenizerBuilder` not being fully available.
  fix Ensure you have `await __wbg_init();` called before attempting to instantiate `TokenizerBuilder`.

• Error: Failed to fetch dictionary

  cause The dictionary identifier passed to `setDictionary()` does not match the dictionary bundled with the package or is misspelled.
  fix Verify that `builder.setDictionary("embedded://jieba")` is used for the `lindera-wasm-jieba-bundler` package. Other dictionary names will not work.

• SyntaxError: require is not defined (for bundler or web packages)

  cause Attempting to use CommonJS `require()` syntax with packages intended for ESM (`import`) or bundler environments, especially after `v3.0.0` which consolidated ESM usage.
  fix Switch to ES module `import` statements (e.g., `import { TokenizerBuilder } from 'lindera-wasm-jieba-bundler';`) and ensure your build environment supports ESM.

Warnings

• breaking Version 3.0.0 introduced significant changes, including the removal of the direct Node.js WASM target and a renaming of npm packages. Users migrating from `v2.x` to `v3.x` should review the new package naming conventions (e.g., `-web`, `-nodejs`, `-bundler`) and adapt their imports accordingly.
  Fix: Review the `README.md` for the correct package suffix (`-web`, `-nodejs`, `-bundler`) matching your environment. For Node.js, `lindera-nodejs` is a separate native NAPI-RS binding, and WASM targets for Node.js were deprecated.
• gotcha Confusing the `-web`, `-nodejs`, and `-bundler` packages for different environments can lead to runtime errors or suboptimal performance. Each package is optimized for its target environment.
  Fix: Ensure you are using the correct package for your environment: `lindera-wasm-<dictionary>-web` for direct browser usage via `<script type="module">`, `lindera-wasm-<dictionary>-nodejs` for Node.js (pre-v3.0.0 WASM, or NAPI-RS for v3.0.0+ `lindera-nodejs`), and `lindera-wasm-<dictionary>-bundler` for projects using bundlers like Webpack or Rollup.
• gotcha The `__wbg_init()` function, which initializes the WebAssembly module, must be called and awaited before any other Lindera WASM functionality can be used. Forgetting to await it will lead to runtime errors.
  Fix: Always prepend `await __wbg_init();` at the beginning of your asynchronous function where you use Lindera, ensuring the WASM module is fully loaded and ready.
• gotcha Specifying the correct dictionary via `builder.setDictionary("embedded://<dictionary-name>")` is crucial. Using the wrong identifier (e.g., `"embedded://ipadic"` for a `jieba` package) will cause dictionary loading failures.
  Fix: For `lindera-wasm-jieba-bundler`, always use `builder.setDictionary("embedded://jieba")`. Refer to the package name and documentation for the correct dictionary identifier.

Install

• npm install lindera-wasm-jieba-bundler npm
• yarn add lindera-wasm-jieba-bundler yarn
• pnpm add lindera-wasm-jieba-bundler pnpm

Imports

• __wbg_init
  wrong:

  const __wbg_init = require('lindera-wasm-jieba-bundler');

  correct:

  import __wbg_init from 'lindera-wasm-jieba-bundler';

  Required to initialize the WebAssembly module; must be awaited. CommonJS `require` is not supported for bundler targets since v3.0.0.
• TokenizerBuilder
  wrong:

  import TokenizerBuilder from 'lindera-wasm-jieba-bundler'; // Not a default export
  import { TokenizerBuilder } from 'lindera-wasm-jieba-bundler/lindera_wasm';

  correct:

  import { TokenizerBuilder } from 'lindera-wasm-jieba-bundler';

  This is a named export for creating tokenizer instances. The import path is directly from the package name.
• Token

  import type { Token } from 'lindera-wasm-jieba-bundler';

  Use `import type` for type-only imports to ensure they are stripped during compilation, optimizing bundle size and avoiding runtime errors in some environments.

Quickstart

This example demonstrates how to initialize the Lindera WASM module, build a tokenizer with the embedded Jieba dictionary, and perform morphological analysis on Chinese text in a bundler environment.

import __wbg_init, { TokenizerBuilder, type Token } from 'lindera-wasm-jieba-bundler';

async function main() {
    // Initialize the WebAssembly module. This must be awaited.
    await __wbg_init();

    // Create a new TokenizerBuilder instance.
    const builder = new TokenizerBuilder();

    // Specify the Jieba dictionary to use.
    // 'embedded://jieba' refers to the dictionary bundled with this package.
    builder.setDictionary("embedded://jieba");

    // Set the tokenization mode. 'normal' is a common default.
    builder.setMode("normal");

    // Build the tokenizer.
    const tokenizer = builder.build();

    // Text to tokenize.
    const textToAnalyze = "上海东方明珠广播电视塔";

    // Perform tokenization.
    const tokens: Token[] = tokenizer.tokenize(textToAnalyze);

    console.log(`Tokens for: "${textToAnalyze}"`);
    tokens.forEach(token => {
        // Each token has a surface form and detailed information.
        console.log(`- ${token.surface}: ${token.details.join(" | ")}`);
    });

    // Example with another Chinese sentence
    const anotherText = "我爱北京天安门";
    const moreTokens: Token[] = tokenizer.tokenize(anotherText);
    console.log(`\nTokens for: "${anotherText}"`);
    moreTokens.forEach(token => {
        console.log(`- ${token.surface}: ${token.details.join(" | ")}`);
    });
}

main().catch(console.error);