Watchtower (Python CloudWatch Logging)

Warnings

• breaking Starting with v3.0.0, non-JSON-serializable objects (e.g., `datetime` objects, custom classes) in log messages are now represented by their `repr()` string instead of being converted to `isoformat()` or `null`. This change might increase log data volume or alter parsing expectations.
  Fix: Review existing code that logs complex objects and adjust downstream log parsing or processing logic to account for `repr()` string representations. If the old behavior is desired, custom JSON serialization should be implemented before passing data to the logger.
• deprecated The `datetime.utcnow()` method, previously used internally by Watchtower, has been replaced due to deprecation. While this primarily affects internal implementation, users with custom logging formats or direct manipulation of `datetime` objects might encounter related issues or are encouraged to update their own code to use timezone-aware `datetime.now(timezone.utc)` for consistency.
  Fix: Ensure your Python environment uses a version of `watchtower` that includes the fix. For your own code, replace `datetime.utcnow()` with `datetime.now(timezone.utc)`.
• gotcha As of v3.0.1, Watchtower truncates log messages based on byte length (256 KB CloudWatch Logs limit) rather than Unicode character count. This can lead to unexpected truncation of messages containing multi-byte Unicode characters, potentially cutting off messages mid-character.
  Fix: Be aware of the byte-length limit for log messages. If logs frequently contain long Unicode strings, consider pre-truncating or encoding them to ensure meaningful parts are preserved within the byte limit before they reach Watchtower.
• gotcha CloudWatch log stream naming conventions changed in v3.4.0, specifically removing ':' from `program_name` to prevent issues. Also, `strftime` format strings are explicitly noted as being required for certain configurations. Incorrect stream naming can lead to logs not appearing where expected or creating many unintended log streams.
  Fix: If customizing log stream names, ensure they comply with CloudWatch naming rules and the expected `strftime` format. For high-volume applications or those using process pools, ensure the log stream name is unique per source using template variables like `{machine_name}/{program_name}/{logger_name}/{process_id}`.
• gotcha Watchtower uses `boto3`, which in turn relies on `botocore` and `urllib3`. These dependencies can produce a significant amount of `DEBUG` level log messages, which can overwhelm application logs if not properly filtered.
  Fix: To reduce noise, set the logging level for `boto3`, `botocore`, and `urllib3` to `WARNING` or higher. For example: `logging.getLogger('boto3').setLevel(logging.WARNING)`, `logging.getLogger('botocore').setLevel(logging.WARNING)`, `logging.getLogger('urllib3').setLevel(logging.WARNING)`.
• gotcha The process running Watchtower requires appropriate AWS Identity and Access Management (IAM) permissions to call the CloudWatch Logs API (e.g., `logs:CreateLogGroup`, `logs:CreateLogStream`, `logs:PutLogEvents`). Lack of these permissions is a common reason for logs not appearing in CloudWatch.
  Fix: Attach an AWS managed IAM policy (e.g., `CloudWatchLogsFullAccess` for testing, or a more restrictive custom policy with `logs:CreateLogGroup`, `logs:CreateLogStream`, `logs:PutLogEvents`) to the IAM role or user credentials used by your application. Refer to `boto3` credentials documentation for how credentials are loaded.

Install

• pip install watchtower Install Watchtower

Imports

• CloudWatchLogHandler

  from watchtower import CloudWatchLogHandler

• logging

  import logging

Quickstart

This quickstart demonstrates how to integrate Watchtower with the Python `logging` module to send logs to AWS CloudWatch. It sets up a basic logger and a `CloudWatchLogHandler`, then sends a few example log messages. Ensure your AWS credentials and default region are configured either via environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`) or through the AWS CLI (`aws configure`) for `boto3` to automatically pick them up.

import logging
import os
from watchtower import CloudWatchLogHandler

# Configure basic logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Set AWS credentials via environment variables for boto3 (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION)
# Or configure AWS CLI (aws configure) for boto3 to pick up credentials automatically

# Create a CloudWatchLogHandler instance
# Recommended: specify log_group_name and region_name explicitly
# boto3 will automatically pick up credentials from env vars or IAM roles.
handler = CloudWatchLogHandler(
    log_group_name=os.environ.get('AWS_LOG_GROUP', 'my-python-app'),
    region_name=os.environ.get('AWS_REGION', 'us-east-1')
)

# Add the handler to the logger
logger.addHandler(handler)

# Log some messages
logger.info('Hello from Watchtower!')
logger.warning('This is a warning message.')
logger.error(dict(error_code=500, message='Something went wrong'))

# For applications that might exit quickly, ensure logs are flushed
# In typical web applications or long-running services, this is handled on shutdown.
handler.flush()