Tiktoken Ruby
JSON →An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer used by OpenAI, for counting tokens in text before sending to OpenAI APIs.
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer used by OpenAI, for counting tokens in text before sending to OpenAI APIs.