Konoha: Japanese Tokenizer Wrapper

5.7.0 · active · verified Thu Apr 16

Konoha is a Python library (v5.7.0) that provides a unified, easy-to-use interface for various Japanese tokenizers, including MeCab, Sudachi, and Sentencepiece. It allows developers to seamlessly switch between different tokenizers and also offers rule-based tokenizers (whitespace, character) and a sentence splitter. The library is actively maintained with its latest release in March 2026.

Common errors

```
ModuleNotFoundError: No module named 'MeCab'
```
cause The MeCab tokenizer dependency (mecab-python3) was not installed along with konoha.

fix Install konoha with the mecab extra: `pip install 'konoha[mecab]'` or `pip install 'konoha[all]'`.
```
RuntimeError: 'sudachipy' is not installed. Please install it with 'pip install sudachipy'
```
cause The Sudachi tokenizer dependency (sudachipy and sudachidict_core) was not installed.

fix Install konoha with the sudachi extra: `pip install 'konoha[sudachi]'` or `pip install 'konoha[all]'`.
```
TypeError: __init__() missing 1 required positional argument: 'model_path'
```
cause Attempted to initialize `WordTokenizer('Sentencepiece')` without providing the `model_path` argument.

fix Provide the path to your Sentencepiece model file: `WordTokenizer('Sentencepiece', model_path="path/to/your/model.spm")`.

Warnings

gotcha Installing `konoha` without specifying extras (e.g., `pip install konoha`) will only install the sentence splitter, not any word tokenizers. To use tokenizers like MeCab or Sudachi, you must install `konoha` with the corresponding extra (e.g., `konoha[mecab]`) or `konoha[all]` for all supported tokenizers.
Fix: Use `pip install 'konoha[all]'` or `pip install 'konoha[<tokenizer_name>]'` to include desired tokenizer dependencies.
breaking The API endpoint paths for the Docker quickstart (e.g., `/api/v1/tokenize`) changed in v4.6.4. Older `curl` commands or Docker configurations might fail.
Fix: Refer to the latest documentation or GitHub README for updated API paths when using the Konoha Docker image. For example, check release notes at `https://github.com/himkt/konoha/releases/tag/v4.6.4`.
gotcha When using the `Sentencepiece` tokenizer, you must provide a valid `model_path` argument to `WordTokenizer`. Omitting it will result in an error or unexpected behavior.
Fix: Always pass the `model_path` argument when initializing `WordTokenizer('Sentencepiece', model_path="your/model.spm")`.

Install

pip install 'konoha[all]' Recommended: Install with all supported tokenizers
pip install 'konoha[mecab]' Install with a specific tokenizer (e.g., MeCab)
pip install konoha Minimal install (only sentence splitter)

Imports

WordTokenizer
```
from konoha import WordTokenizer
```
SentenceTokenizer
```
from konoha import SentenceTokenizer
```

Quickstart

Demonstrates basic word-level tokenization of a Japanese sentence using the `WordTokenizer` with MeCab. Ensure the necessary tokenizer is installed as an extra.

from konoha import WordTokenizer

sentence = '自然言語処理を勉強しています'

# Initialize with a supported tokenizer (e.g., MeCab)
# Ensure 'konoha[mecab]' or 'konoha[all]' is installed
tokenizer = WordTokenizer('MeCab')

tokens = tokenizer.tokenize(sentence)
print([token.surface for token in tokens])

view raw JSON →