NVIDIA AI Endpoints for LangChain

1.2.1 · active · verified Sat Apr 11

This integration package connects NVIDIA AI Endpoints with the LangChain framework, providing seamless access to NVIDIA's state-of-the-art AI Foundation Models. It enables robust conversational AI and semantic embedding capabilities through classes like `ChatNVIDIA` and `NVIDIAEmbeddings`. Currently at version 1.2.1, the library maintains an active development pace with frequent updates.

Warnings

breaking With the release of LangChain 1.0, the broader LangChain ecosystem underwent significant changes, streamlining its core and moving some legacy functionality to `langchain-classic`. While `langchain-nvidia-ai-endpoints` adapted, ensure your overall `langchain` and `langchain-core` dependencies are compatible.
Fix: Review LangChain's v1 migration guide. For `langchain-nvidia-ai-endpoints`, ensure `langchain-core` is updated to a compatible version (e.g., `0.1.15` or newer as per examples).
gotcha Incorrect or missing `NVIDIA_API_KEY` will lead to authentication failures. The key must start with `nvapi-`.
Fix: Obtain your `NVIDIA_API_KEY` from the NVIDIA API Catalog and ensure it is correctly set as an environment variable or passed directly during class instantiation. Verify it starts with `nvapi-`.
gotcha Using older versions of `langchain-nvidia-ai-endpoints` with newer NVIDIA AI Foundation Models may result in uninformative errors or lack of support for latest model features. For example, specific models might require a minimum package version.
Fix: Always keep `langchain-nvidia-ai-endpoints` updated to the latest version to ensure compatibility with the newest NVIDIA models and features. Refer to GitHub releases for model-specific support.
gotcha When self-hosting NVIDIA NIM microservices, incorrect `base_url` or port configuration for models like `NVIDIARAGRetriever` can lead to `NVIDIARAGConnectionError`.
Fix: Ensure the `base_url` parameter points to the correct endpoint (e.g., `http://localhost:8081` for RAG server) and that the NIM container is running and accessible.

Install

pip install langchain-nvidia-ai-endpoints Install stable version

Imports

ChatNVIDIA

from langchain_nvidia_ai_endpoints import ChatNVIDIA

NVIDIAEmbeddings

from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings

NVIDIARerank

from langchain_nvidia_ai_endpoints import NVIDIARerank

ChatNVIDIADynamo
```
from langchain_nvidia_ai_endpoints import ChatNVIDIADynamo
```
Used for Dynamo KV cache optimization support, introduced in v1.1.0.
NVIDIARAGRetriever
```
from langchain_nvidia_ai_endpoints import NVIDIARAGRetriever
```
Connects to NVIDIA RAG Blueprint /v1/search endpoint, introduced in v1.2.1.

Quickstart

This quickstart demonstrates how to initialize `ChatNVIDIA` for text generation and `NVIDIAEmbeddings` for creating text embeddings. It requires `NVIDIA_API_KEY` to be set as an environment variable, obtainable from the NVIDIA API Catalog. Replace 'nvidia/nemotron-3-super-120b-a12b' and 'nvolveqa_40k' with desired model names available on NVIDIA AI Endpoints.

import os
from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings

nvapi_key = os.environ.get('NVIDIA_API_KEY', '')
if not nvapi_key.startswith('nvapi-'):
    print("Please set the NVIDIA_API_KEY environment variable. You can get one from the NVIDIA API Catalog.")
else:
    # Initialize ChatNVIDIA for conversational AI
    chat_model = ChatNVIDIA(model="nvidia/nemotron-3-super-120b-a12b", nvidia_api_key=nvapi_key)
    chat_response = chat_model.invoke("Explain the concept of large language models.")
    print("Chat Model Response:", chat_response.content)

    # Initialize NVIDIAEmbeddings for semantic embeddings
    embed_model = NVIDIAEmbeddings(model="nvolveqa_40k", nvidia_api_key=nvapi_key)
    embedding_output = embed_model.embed_query("What are vector embeddings?")
    print("Embedding Model Output Length:", len(embedding_output))

view raw JSON →