Wordninja

Warnings

• gotcha The default `wordninja.split()` function removes punctuation from the input string and does not retain it in the output. For example, 'hello.world' becomes ['hello', 'world'].
  Fix: To preserve punctuation, you may need to pre-process the text, or consider using the `wordninja-enhanced` fork (e.g., `pip install wordninja-enhanced`) which includes punctuation preservation and other features not present in the original library.
• gotcha The probabilistic model can sometimes produce 'over-aggressive' or incorrect splits for specific words or abbreviations, such as 'patreon' splitting into ['pat', 're', 'on'] or 'esg' into ['e', 's', 'g'].
  Fix: For critical applications, consider reviewing results for specific edge cases or exploring custom language models. The library supports custom language models if provided as gzipped text files with one word per line in decreasing order of probability.
• gotcha The library is primarily designed for English. While custom language models are supported, creating them requires a specific gzipped text file format (one word per line, decreasing order of probability).
  Fix: If working with non-English languages or specific domains, ensure your custom word list adheres strictly to the required format. The `wordninja-enhanced` fork offers out-of-the-box support for several additional languages.

Install

• pip install wordninja Install stable version

Imports

• split

  import wordninja
  wordninja.split('text')

  The primary function `split` is accessed directly from the imported `wordninja` module, not imported directly.

Quickstart

Demonstrates how to import the library and use the `split` function to segment a concatenated string into a list of words.

import wordninja

split_words = wordninja.split('thisisateststring')
print(split_words)

split_phrase = wordninja.split('hellofromtheotherside')
print(split_phrase)