NVIDIA NeMo Agent Toolkit LangChain/LangGraph Integration

Common errors

• ModuleNotFoundError: No module named 'langchain.agents' (or similar for other langchain sub-modules)

  cause This typically occurs due to a mismatch between the installed `nvidia-nat` (and `nvidia-nat-langchain`) version and the `langchain` version. `nvidia-nat` might be trying to import from an older LangChain structure that no longer exists in your installed LangChain version.
  fix Check the `nvidia-nat` official documentation or GitHub for compatible LangChain versions. You may need to downgrade your `langchain` packages or upgrade `nvidia-nat` to a version that supports your desired `langchain` version. Using a `pyproject.toml` with strict dependency pinning is recommended.

• ERROR: [401] Unauthorized invalid response from UAM Please check or regenerate your API key.

  cause This error, or a similar 'Error: NGC_API_KEY is not set', indicates that the API key required for authentication with NVIDIA services (like NIM microservices or hosted LLMs) is either missing or invalid.
  fix Ensure that your `NVIDIA_API_KEY` (and potentially `NGC_CLI_API_KEY` for certain local deployments) environment variable is correctly set and contains a valid API key obtained from `build.nvidia.com`.

• RuntimeWarning: coroutine 'Server.serve' was never awaited (when running `nat serve`)

  cause This warning (and potential related issues) can arise in Python 3.11+ environments when using the `nat serve` command, often due to how event loops are managed.
  fix This is an internal `nvidia-nat` issue that was addressed in later versions. Ensure you are using the latest stable version of `nvidia-nat` (and thus `nvidia-nat-langchain`). If the problem persists, consult the `NVIDIA/NeMo-Agent-Toolkit` GitHub issues for potential workarounds or specific version recommendations for Python 3.11+.

Warnings

• breaking Potential version incompatibility with `langchain` and `langgraph`. Older `nvidia-nat` versions (and implicitly `nvidia-nat-langchain`) might rely on specific, older `langchain` versions, leading to `ModuleNotFoundError` or other import issues if your project uses newer LangChain versions.
  Fix: Consult the `nvidia-nat` GitHub repository or documentation for a compatibility matrix. For `nvidia-nat` versions before 1.5.0, you might need to pin `langchain` and its sub-packages to compatible versions. `nvidia-nat` version 1.5.0 and later aim to improve dependency management.
• deprecated The `aiqtoolkit-langchain` package is a transitional package that is deprecated. It will be removed in future releases.
  Fix: New projects should directly use `nvidia-nat-langchain`. If migrating an existing project, update your dependencies to `nvidia-nat-langchain` instead of `aiqtoolkit-langchain`.
• gotcha Many functionalities, especially those interacting with NVIDIA's hosted models or NIM microservices, require the `NVIDIA_API_KEY` environment variable to be set.
  Fix: Obtain an API key from `build.nvidia.com` and set it as an environment variable: `export NVIDIA_API_KEY="nvapi-xxxxxxxxxxxxxxxxxxxxxxxx"`. Also, ensure `NGC_CLI_API_KEY` is set for specific local NIM deployments.

Install

• pip install nvidia-nat-langchain Recommended installation
• pip install "nvidia-nat[langchain]" Install as an extra with nvidia-nat

Imports

• add_profiling_callbacks

  from nvidia_nat.langchain import add_profiling_callbacks

  Used to wrap LangChain chains or agents to enable automatic token counting, latency measurement, and tracing to observability platforms.
• LLMFrameworkEnum

  from nat.builder.framework_enum import LLMFrameworkEnum

  Used to specify the framework (e.g., LangChain) when configuring components with NAT's builder system.
• SyncBuilder

  from nat.builder.sync_builder import SyncBuilder

  Provides access to NAT's dependency injection system for configuring and retrieving LLMs and tools.

Quickstart

This quickstart demonstrates how to integrate `nvidia-nat-langchain` for basic profiling with a LangChain chain. It assumes `langchain-nvidia-ai-endpoints` is also installed for `ChatNVIDIA`. The core idea is to wrap your existing LangChain construct with `add_profiling_callbacks` to enable NAT's observability features. In a full NeMo Agent Toolkit workflow, this integration is often handled through YAML configurations and the `nat` CLI.

import os
from langchain_core.prompts import ChatPromptTemplate
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_core.output_parsers import StrOutputParser
from nvidia_nat.langchain import add_profiling_callbacks

# Ensure NVIDIA_API_KEY is set in your environment
# You can get one from https://build.nvidia.com/
# os.environ["NVIDIA_API_KEY"] = "nvapi-xxxxxxxxxxxxxxxxxxxxxxxx"

if not os.getenv("NVIDIA_API_KEY"):
    print("Please set the NVIDIA_API_KEY environment variable.")
    exit()

# 1. Define your LangChain components
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful AI assistant."),
    ("user", "{input}")
])
llm = ChatNVIDIA(model="meta/llama-3-8b-instruct")
output_parser = StrOutputParser()

# 2. Create your LangChain chain
chain = prompt | llm | output_parser

# 3. Add NeMo Agent Toolkit profiling callbacks to your chain
profiled_chain = add_profiling_callbacks(chain)

# 4. Invoke the profiled chain
response = profiled_chain.invoke({"input": "What is the capital of France?"})
print(response)

# In a real NAT setup, you would typically run this via the `nat` CLI
# with a YAML configuration, which automatically applies these integrations.