Autograd

Warnings

• breaking Autograd v1.7.0 (and subsequent versions) dropped support for Python 2.x. Python 3.9 or newer is now required.
  Fix: Migrate your project to Python 3.9+.
• breaking While Autograd v1.7.0 added compatibility for NumPy 2.0.0, the latest v1.8.0 explicitly pins dependencies to NumPy < 3 to ensure stability. Future NumPy 3.x versions might introduce breaking changes.
  Fix: Ensure your environment uses a NumPy version compatible with Autograd's constraints (currently <3.0). Check release notes for specific version requirements.
• gotcha Autograd does not properly handle in-place array operations (e.g., `A[0,0] = x`, `a += b`) or method-style dot products (`A.dot(B)`). Always use non-in-place assignments (`a = a + b`) and function-style operations (`np.dot(A, B)`).
  Fix: Rewrite code to use non-in-place operations and function-style calls for NumPy array manipulations.
• gotcha Avoid implicit casting of Python lists to NumPy arrays within functions that require differentiation (e.g., `np.sum([x, y])`). Explicitly convert lists to NumPy arrays: `np.sum(np.array([x, y]))`.
  Fix: Ensure all array creations and operations explicitly use `autograd.numpy` functions and array types.
• gotcha The `grad` function expects the function being differentiated to return a scalar value. If you need to differentiate a vectorized function that returns an array of scalar derivatives, use `autograd.elementwise_grad`.
  Fix: Ensure the target function for `grad` returns a scalar, or use `elementwise_grad` for vectorized outputs.
• gotcha Performance can be significantly impacted by extensive pure Python logic (e.g., complex loops, conditional statements) within the function being differentiated, as Python overheads are amplified during automatic differentiation.
  Fix: Vectorize operations as much as possible using `autograd.numpy` primitives to minimize Python-level computation. Re-evaluate algorithm design if performance remains an issue.

Install

• pip install autograd Base installation
• pip install "autograd[scipy]" With SciPy features

Imports

• numpy

  import autograd.numpy as np

  Use autograd's wrapped NumPy to enable automatic differentiation.
• grad

  from autograd import grad

  The primary function to obtain the gradient of a scalar-valued function.
• elementwise_grad

  from autograd import elementwise_grad as egrad

  For functions that vectorize over inputs and return an array of scalar derivatives.

Quickstart

This example defines a simple tanh function using `autograd.numpy`, then uses `autograd.grad` to obtain its derivative function. The gradient is then evaluated at a specific point.

import autograd.numpy as np
from autograd import grad

def tanh(x):
    return (1.0 - np.exp((-2 * x))) / (1.0 + np.exp(-(2 * x)))

grad_tanh = grad(tanh)
x_val = np.float64(1.0)

print(f"tanh({x_val}) = {tanh(x_val)}")
print(f"Gradient of tanh at {x_val} = {grad_tanh(x_val)}")