{"library":"lmcache","title":"lmcache","description":"lmcache is a Python library that provides an LLM serving engine extension. It aims to reduce Time To First Token (TTFT) and increase throughput, particularly in scenarios involving long contexts. The current version is 0.4.3, and it appears to have an active development cadence.","language":"python","status":"active","last_verified":"Thu Apr 16","install":{"commands":["pip install lmcache"],"cli":null},"imports":["from lmcache.client import Client","from lmcache.schemas import ChatCompletionRequest","from lmcache.schemas import ChatCompletionMessage"],"auth":{"required":false,"env_vars":[]},"quickstart":{"code":"import os\nfrom lmcache.client import Client\nfrom lmcache.schemas import ChatCompletionRequest, ChatCompletionMessage\n\n# NOTE: An lmcache server must be running separately for this client to connect.\n# Default server host is 'localhost', port 13333.\n\ntry:\n    client = Client(host=os.environ.get('LMCACHE_HOST', 'localhost'), \n                    port=int(os.environ.get('LMCACHE_PORT', 13333)))\n\n    request = ChatCompletionRequest(\n        model=os.environ.get('LMCACHE_MODEL', 'gpt-3.5-turbo'), # Replace with a model supported by your lmcache server\n        messages=[\n            ChatCompletionMessage(role=\"user\", content=\"Hello, how are you?\"),\n            ChatCompletionMessage(role=\"assistant\", content=\"I am doing well, thank you!\"),\n            ChatCompletionMessage(role=\"user\", content=\"What is your purpose?\")\n        ]\n    )\n\n    response = client.chat_completion(request)\n    print(f\"Assistant: {response.choices[0].message.content}\")\n\nexcept Exception as e:\n    print(f\"An error occurred: {e}\")\n    print(\"Ensure the lmcache server is running and accessible at the specified host and port.\")\n","lang":"python","description":"This quickstart demonstrates how to use the lmcache client to interact with a running lmcache server. It sends a chat completion request similar to the OpenAI API. Please ensure that an lmcache server is running independently before executing this client code.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":null}