Lindera WASM (IPADIC) for Bundlers

Common errors

• ReferenceError: require is not defined

  cause Using the CommonJS `require()` syntax with a bundler-targeted package in an ECMAScript Module (ESM) context, or using a `-bundler` package in a Node.js CJS project.
  fix For bundler environments, use `import init, { TokenizerBuilder } from 'lindera-wasm-ipadic-bundler';`. If targeting Node.js, use `lindera-wasm-ipadic-nodejs` with `const { TokenizerBuilder } = require('lindera-wasm-ipadic-nodejs');`.

• TypeError: init is not a function

  cause Incorrectly importing the default `init` function as a named export, or attempting to call it before it's properly assigned (e.g., due to an incorrect bundler configuration).
  fix Ensure `init` is imported as the default export: `import init from 'lindera-wasm-ipadic-bundler';`. Also, verify your bundler is correctly handling default WASM module exports.

• Error: The requested dictionary 'embedded://ipadic' is not found.

  cause The specified dictionary path (`embedded://ipadic`) is incorrect, or a Lindera WASM package without an embedded dictionary was installed (e.g., `lindera-wasm-bundler` instead of `lindera-wasm-ipadic-bundler`).
  fix Double-check the dictionary path. Ensure you have installed the correct package that bundles the desired dictionary, such as `lindera-wasm-ipadic-bundler` for IPADIC.

• TypeError: Failed to execute 'fetch' on 'WorkerGlobalScope': Failed to fetch dynamically imported module

  cause This usually occurs in browser/web worker environments when a bundler fails to correctly output or serve the WASM `.wasm` chunk file, or the server path for the WASM file is incorrect.
  fix Verify your bundler (e.g., Webpack, Rollup) is configured to handle `.wasm` files. Ensure the output directory of your build includes the `.wasm` asset and that your web server is correctly serving it with the appropriate MIME type (`application/wasm`).

Warnings

• breaking The `lindera-wasm-ipadic-bundler` package is currently at v2.3.4, while other Lindera WASM packages (e.g., `-web`, `-nodejs`) have progressed to v3.x. This means features, bug fixes, and breaking changes introduced in Lindera v3 are not yet present in this specific 'bundler' package. Users expecting v3 behavior will not find it here.
  Fix: Monitor the Lindera GitHub repository and npm for the release of `lindera-wasm-ipadic-bundler` v3.x if you require the latest features or architecture. For now, be aware of the version difference when consulting documentation for other Lindera targets.
• gotcha Lindera provides separate npm packages for different JavaScript environments (e.g., `-web`, `-nodejs`, `-bundler`). Using the incorrect package for your target environment (e.g., `lindera-wasm-ipadic-nodejs` in a browser bundler context) will lead to runtime errors due to environment-specific WASM compilation targets and API differences.
  Fix: Always install and import the package specifically designed for your environment. For projects using bundlers like Webpack, Rollup, or Vite, ensure you are using `lindera-wasm-ipadic-bundler`.
• gotcha The WebAssembly module must be asynchronously initialized by calling `await init()` (where `init` is the default export) before any other functions from the package can be used. Forgetting to call or await this initialization function will result in runtime errors.
  Fix: Ensure that `await init()` is the very first interaction with the Lindera module in your application's entry point or wherever the module is first used. Wrap your usage in an `async` function.
• gotcha WASM module loading and initialization are asynchronous operations. Attempting to use the `TokenizerBuilder` or other functionality synchronously before `init()` has resolved will lead to errors.
  Fix: Always use `async/await` or Promise `.then()` chains to ensure that all interactions with the Lindera WASM module occur only after it has been successfully initialized.

Install

• npm install lindera-wasm-ipadic-bundler npm
• yarn add lindera-wasm-ipadic-bundler yarn
• pnpm add lindera-wasm-ipadic-bundler pnpm

Imports

• init
  wrong:

  import { init, TokenizerBuilder } from 'lindera-wasm-ipadic-bundler';

  correct:

  import init, { TokenizerBuilder } from 'lindera-wasm-ipadic-bundler';

  The `init` function is the default export and must be called to initialize the WebAssembly module. Other utilities like `TokenizerBuilder` are named exports.
• TokenizerBuilder
  wrong:

  import TokenizerBuilder from 'lindera-wasm-ipadic-bundler';

  correct:

  import { TokenizerBuilder } from 'lindera-wasm-ipadic-bundler';

  `TokenizerBuilder` is a named export used to configure and create a tokenizer instance.
• Token

  import type { Token } from 'lindera-wasm-ipadic-bundler';

  Import the `Token` type for type-safe handling of tokenized results in TypeScript projects.

Quickstart

This quickstart initializes the Lindera WASM module, configures a tokenizer with the embedded IPADIC dictionary, and demonstrates how to tokenize a Japanese sentence and access token details.

import init, { TokenizerBuilder } from 'lindera-wasm-ipadic-bundler';

async function main() {
    // Initialize the WebAssembly module. This is crucial and must be awaited
    // before any other functions from the module can be used.
    await init();

    // Create a new TokenizerBuilder instance to configure the tokenizer.
    const builder = new TokenizerBuilder();

    // Specify the dictionary to use. 'embedded://ipadic' uses the dictionary
    // bundled with this package.
    builder.setDictionary("embedded://ipadic");

    // Set the tokenization mode, 'normal' is suitable for general text.
    builder.setMode("normal");

    // Build the tokenizer with the specified settings.
    const tokenizer = builder.build();

    // Define the Japanese sentence to be tokenized.
    const sentence = "すもももももももものうち";
    const tokens = tokenizer.tokenize(sentence);

    console.log(`Tokenizing sentence: "${sentence}"`);
    console.log("--- Tokens ---");

    // Iterate over the resulting tokens and print their surface form and details.
    tokens.forEach(token => {
        console.log(`${token.surface} [${token.details.join(", ")}]`);
    });

    // Demonstrate accessing specific details of a token.
    if (tokens.length > 0) {
        const firstToken = tokens[0];
        console.log(`\nFirst token surface: ${firstToken.surface}`);
        // The details array contains information like part-of-speech, conjugation, etc.
        console.log(`First token part of speech: ${firstToken.details[0]}`);
    }
}

// Execute the main asynchronous function.
main().catch(console.error);