Mistral Common

Warnings

• breaking Applications relying on strict parsing of streamed chunks may break due to a security-related change that added a new 'p' parameter to chunks. Update `mistral-common` to version 1.8.4 or higher to mitigate this.
  Fix: Upgrade to mistral-common >= 1.8.4.
• gotcha Directly importing internal modules (e.g., `MistralCommonBackend`) from `mistral-common` or other related libraries (like `transformers`) is generally discouraged. Internal file structures in fast-paced open-source libraries can change frequently, making your codebase brittle to minor updates. It is safer to rely on public APIs like `AutoTokenizer` when integrating with libraries such as Hugging Face Transformers.
  Fix: Use public API endpoints and classes as documented. Avoid deep imports from internal submodules unless explicitly instructed by official documentation.
• gotcha Tokenizer versions are closely tied to Mistral model versions. Using an older `mistral-common` version with newer models or vice-versa might lead to incorrect tokenization or unexpected behavior. Verify compatibility between your `mistral-common` version and the Mistral model you intend to use.
  Fix: Always check the official Mistral AI documentation or the `mistral-common` GitHub repository for recommended `mistral-common` versions compatible with specific Mistral models. Upgrade `mistral-common` when using new model releases.
• deprecated The `sentencepiece` tokenizer is now optional, as Mistral AI primarily releases `Tekken` tokenizers for recent models. While `sentencepiece` support is still available via an optional dependency, new projects might benefit from focusing on `Tekken`-based tokenization if working with the latest models.
  Fix: For new models, consider using `Tekken` tokenizers. If you require `sentencepiece`, ensure you install `mistral-common[sentencepiece]`.
• gotcha When using `mistral-common`'s tokenizer via a Hugging Face `PreTrainedTokenizerBase` compatible interface, be aware of key behavioral differences. Special tokens are not encoded directly, and pairs of sequences are not supported. This can lead to unexpected results if relying on standard Hugging Face `PreTrainedTokenizer` behavior.
  Fix: Refer to the `mistral-common` documentation for specific behavior of its tokenizer, especially concerning special tokens and sequence handling, when used with Hugging Face interfaces.

Install

• pip install mistral-common Basic Installation
• pip install "mistral-common[image,audio,hf-hub,sentencepiece,server]" Installation with all optional dependencies

Imports

• MistralTokenizer

  from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

• ChatCompletionRequest

  from mistral_common.protocol.instruct.request import ChatCompletionRequest

• UserMessage

  from mistral_common.protocol.instruct.messages import UserMessage

• Tool

  from mistral_common.protocol.instruct.tool_calls import Tool

• Function

  from mistral_common.protocol.instruct.tool_calls import Function

• MistralCommonBackend

  from mistral_common.tokens.tokenizers.mistral import MistralCommonBackend

  Directly importing internal modules like `MistralCommonBackend` from `transformers` or other libraries is discouraged as internal paths are unstable and can break. Rely on the `AutoTokenizer` or the official `mistral-common` import paths for stability.

Quickstart

This quickstart demonstrates how to tokenize a simple chat completion request using `mistral-common`. It involves defining user messages, creating a `ChatCompletionRequest`, and then encoding it into token IDs using a `MistralTokenizer`. For a real scenario, ensure the tokenizer model is correctly loaded, typically from a model name or a local file. The example includes a fallback for demonstration if a live tokenizer cannot be loaded.

from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
import os

# NOTE: For actual inference, you would typically load a model tokenizer 
# from a path or Hugging Face Hub. This example uses a placeholder.
# A real model_name would be 'mistral-large-latest' or 'open-mixtral-8x22b'
# For local development or specific tokenizer versions, a path can be used.
# For the purpose of a quickstart demonstration without an actual model download,
# we'll simulate the tokenizer loading or use a common one if available without heavy downloads.
# In a real scenario, you might do:
# tokenizer = MistralTokenizer.from_model("open-mixtral-8x22b")
# Or, if you have a local tokenizer:
# tokenizer = MistralTokenizer.from_file("path/to/tokenizer.model")

# For this quickstart, we will attempt to load a tokenizer that is generally available
# or illustrate the process. Assuming a tokenizer object can be instantiated for tokenization.
# In a production environment, ensure the tokenizer model is correctly loaded.

try:
    # Attempt to load a common tokenizer for demonstration. 
    # 'open-mixtral-8x22b' is often used in examples.
    tokenizer = MistralTokenizer.from_model("open-mixtral-8x22b")
except Exception as e:
    print(f"Could not load tokenizer directly (e.g., model not found or network issue): {e}")
    print("Please ensure you have the tokenizer model available or use a local path if preferred.")
    print("For this quickstart, we will use a dummy tokenizer for demonstration purposes only.")

    # Fallback to a dummy tokenizer if real one fails for demonstration
    class DummyTokenizer:
        def encode_chat_completion(self, request):
            print("Using dummy tokenizer. No actual tokenization performed.")
            return [1, 2, 3, 4, 5] # Dummy token IDs

    tokenizer = DummyTokenizer()

messages = [
    UserMessage(content="What is the capital of France?")
]

chat_completion_request = ChatCompletionRequest(messages=messages)

# Tokenize the chat completion request
token_ids = tokenizer.encode_chat_completion(chat_completion_request)

print(f"Original messages: {messages}")
print(f"Token IDs: {token_ids}")