Lindera WASM for Bundler Environments

Common errors

• Failed to load WASM module: CompileError: WebAssembly.instantiate(): expected magic word 00 61 73 6d, found 3c 21 44 4f

  cause This error often indicates that the browser or bundler attempted to parse an HTML document (e.g., an error page) as a WebAssembly module, usually because the WASM file could not be found at the expected path.
  fix Verify that your bundler is correctly configuring the output path and serving the WebAssembly `.wasm` file. For Vite, ensure the `optimizeDeps.exclude` configuration is correctly applied to prevent issues with pre-bundling that might interfere with WASM file resolution.

• ReferenceError: __wbg_init is not defined

  cause The default export for initializing the WebAssembly module, `__wbg_init`, was either not imported or was imported incorrectly (e.g., as a named import).
  fix Ensure `__wbg_init` is imported as the default export: `import __wbg_init from 'lindera-wasm-bundler';`

• TypeError: Cannot read properties of undefined (reading 'metadata')

  cause This error typically occurs when attempting to access properties like `metadata` from the object returned by `loadDictionaryFiles` before the dictionary files have been successfully loaded or if `loadDictionaryFiles` failed to retrieve them.
  fix Ensure `downloadDictionary` has completed successfully and that the dictionary name provided to `loadDictionaryFiles` matches the name used during download. Also, verify that the browser's Origin Private File System (OPFS) is accessible and not blocked by security policies.

Warnings

• breaking Version 3.0.0 introduced significant changes, including the removal of the dedicated Node.js WASM target and renaming of npm packages. Projects targeting Node.js should now use `lindera-nodejs` instead of WASM packages. Projects using WASM in bundler environments should use `lindera-wasm-bundler`.
  Fix: For Node.js environments, switch to `lindera-nodejs`. For bundler environments, ensure you are importing from `lindera-wasm-bundler` and update import paths/statements as necessary. Review the v3.0.0 changelog for specific refactoring details.
• gotcha When using `lindera-wasm-bundler` (or `lindera-wasm-web`) in a Vite project, you must explicitly exclude it from `optimizeDeps` in your `vite.config.js` to prevent issues with Vite's dependency pre-bundling.
  Fix: Add `lindera-wasm-bundler` to the `optimizeDeps.exclude` array in your `vite.config.js`: ```javascript // vite.config.js import { defineConfig } from 'vite' export default defineConfig({ optimizeDeps: { exclude: [ "lindera-wasm-bundler" // Use "lindera-wasm-web" if using that package ] }, }); ```
• gotcha For browser extension development, a `content_security_policy` (CSP) must be configured in `manifest.json` to allow WebAssembly execution using `wasm-unsafe-eval`. Additionally, Vite's dev server might require CORS configuration for `chrome-extension://` origins.
  Fix: In `manifest.json`, ensure your CSP includes `script-src 'self' 'wasm-unsafe-eval';` for `extension_pages`. For Vite development, configure `cors` in `vite.config.js`: ```json // manifest.json "content_security_policy": { "extension_pages": "script-src 'self' 'wasm-unsafe-eval';" } ``` ```javascript // vite.config.js import { defineConfig } from 'vite' export default defineConfig({ server: { cors: { origin: [ /chrome-extension:\/\//, ], }, }, }); ```
• gotcha All functions interacting with the WebAssembly module, including `__wbg_init` and methods on the `Tokenizer` instance, are asynchronous and return Promises. Forgetting to `await` them can lead to unhandled promise rejections or functions not completing before subsequent operations.
  Fix: Always `await` calls to `__wbg_init()` and any methods that return a Promise, typically those that interact with the underlying WASM or I/O, such as `downloadDictionary` and potentially `tokenizer.tokenize` in some complex scenarios. Wrap usage in an `async` function.

Install

• npm install lindera-wasm-bundler npm
• yarn add lindera-wasm-bundler yarn
• pnpm add lindera-wasm-bundler pnpm

Imports

• __wbg_init
  wrong:

  import { __wbg_init } from 'lindera-wasm-bundler';

  correct:

  import __wbg_init from 'lindera-wasm-bundler';

  This is the default export and initializes the WebAssembly module. It must be awaited before using other functions.
• TokenizerBuilder
  wrong:

  const { TokenizerBuilder } = require('lindera-wasm-bundler');

  correct:

  import { TokenizerBuilder } from 'lindera-wasm-bundler';

  CommonJS `require` is not supported for `lindera-wasm-bundler` since v3. Use named ESM imports.
• loadDictionaryFromBytes

  import { loadDictionaryFromBytes } from 'lindera-wasm-bundler';

  Used to load pre-downloaded dictionary files from byte arrays.
• downloadDictionary
  wrong:

  import { downloadDictionary } from 'lindera-wasm-bundler';

  correct:

  import { downloadDictionary, loadDictionaryFiles } from 'lindera-wasm-bundler/opfs';

  Functions related to OPFS-based dictionary management (downloading, loading files) are exposed from the 'lindera-wasm-bundler/opfs' subpath.

Quickstart

This quickstart demonstrates how to initialize the Lindera WASM module, download and load a dictionary using the OPFS API, and then tokenize a Japanese sentence, logging each token's surface form and morphological details.

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm-bundler';
import { downloadDictionary, loadDictionaryFiles } from 'lindera-wasm-bundler/opfs';

async function initializeAndTokenize() {
    // Initialize the WebAssembly module
    await __wbg_init();

    // Check if dictionary is already downloaded or download it for the first time.
    // In a real application, you might use local storage or IndexedDB to track this.
    console.log("Attempting to download dictionary (if not already present)...");
    try {
        await downloadDictionary(
            "https://github.com/lindera/lindera/releases/download/v3.0.0/lindera-ipadic-3.0.0.zip", 
            "ipadic"
        );
        console.log("Dictionary 'ipadic' ready in OPFS.");
    } catch (error) {
        console.warn("Error downloading dictionary, might be already present or network issue:", error);
    }

    // Load dictionary from OPFS
    console.log("Loading dictionary files from OPFS...");
    const files = await loadDictionaryFiles("ipadic");
    const dict = loadDictionaryFromBytes(
        files.metadata, files.dictDa, files.dictVals,
        files.dictWordsIdx, files.dictWords, files.matrixMtx,
        files.charDef, files.unk
    );
    console.log("Dictionary loaded into memory.");

    // Create a tokenizer instance
    const builder = new TokenizerBuilder();
    builder.setDictionaryInstance(dict);
    builder.setMode("normal"); // "normal", "decompose", or "search"
    const tokenizer = builder.build();

    // Tokenize a sentence
    const text = "すもももももももものうち";
    console.log(`Tokenizing: "${text}"`);
    const tokens = tokenizer.tokenize(text);

    tokens.forEach(token => {
        console.log(`Surface: ${token.surface}, Details: ${token.details.join(", ")}`);
    });
    console.log("Tokenization complete.");
}

initializeAndTokenize().catch(console.error);