checklist.day

Tokenizer

JSON →
library 0.3.0 ·ruby
✓ verified Jun 18, 2026
utility

A multilingual tokenizer for NLP tasks supporting German, English, and French texts.

Resources

docswww.rubydoc.info/gems/tokenizer/ ↗
githubgithub.com/arbox/tokenizer ↗

API endpoints

full doc /v1/registry/tokenizer
install /v1/registry/tokenizer/install