Amazon SQS Extended Client Library for Python

1.0.1 verified Thu Apr 16 auth: no python

The Amazon SQS Extended Client Library for Python extends Amazon SQS to handle message payloads larger than the standard 256 KB limit, supporting sizes up to 2 GB. It achieves this by transparently storing the message payload in an Amazon S3 bucket and sending a reference to the S3 object in the SQS message. The library is currently at version 1.0.1 and maintains a release cadence tied to feature development and AWS SDK updates.

pip install amazon-sqs-extended-client

Common errors

error ERROR [S3Dao] Failed to get the S3 object which contains the payload. ↓

cause The SQS Extended Client could not retrieve the message payload from the referenced S3 bucket. This commonly indicates incorrect IAM permissions (e.g., missing s3:GetObject) or that the S3 object was deleted before the message was processed.

fix

Verify that the AWS credentials used by the consumer have s3:GetObject permission on the S3 bucket where large payloads are stored. Also, ensure that S3 objects are not being prematurely deleted.

error ModuleNotFoundError: No module named 'amazon_sqs_extended_client.SQSExtendedClient' ↓

cause Attempting to import the `SQSExtendedClient` class from an incorrect submodule path. The class is directly available at the package root.

fix

Correct the import statement to from amazon_sqs_extended_client import SQSExtendedClient.

error ClientError: An error occurred (InvalidParameterValue) when calling the SendMessage operation: The message body must not be empty. ↓

cause This error occurs when `send_message` is called with an empty or `None` `MessageBody`. While SQS itself might tolerate some forms of empty messages, the extended client might have internal validations or the underlying S3 put operation might fail for empty content.

fix

Ensure the MessageBody provided to send_message contains valid string data, even if it's a small placeholder.

Warnings

gotcha The Amazon SQS Extended Client Library for Python explicitly depends on an S3 bucket for storing large message payloads. Ensure the specified S3 bucket exists and that your AWS credentials (used by Boto3 clients) have the necessary permissions (s3:PutObject, s3:GetObject, s3:DeleteObject) for the bucket and sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage for the SQS queue. Lack of S3 permissions is a common cause of `Failed to get the S3 object which contains the payload` errors. ↓

fix Verify IAM policies for the AWS user/role sending and receiving messages. Grant `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject` on the S3 bucket and `sqs:*` on the SQS queue.

gotcha There are several similarly named Python libraries (`aws-sqs-ext-client`, `sqs-extended-client`, `pysqs-extended-client`) that aim to provide similar functionality. Ensure you are using `amazon-sqs-extended-client` (from `awslabs` on GitHub) for the official and actively maintained version to avoid compatibility issues or unexpected behavior. ↓

fix Always install using `pip install amazon-sqs-extended-client` and import with `from amazon_sqs_extended_client import SQSExtendedClient`.

gotcha When deleting messages, the extended client automatically handles the deletion of the corresponding S3 object if the message payload was stored in S3. However, if the S3 object is manually deleted before the SQS message is processed and deleted, the consumer might fail to retrieve the full message body, leading to `Failed to get the S3 object which contains the payload` errors. ↓

fix Rely on the `SQSExtendedClient.delete_message` or `delete_message_batch` calls to ensure both the SQS message and its S3 payload are removed synchronously. Avoid manual deletion of S3 objects referenced by active SQS messages.

Imports

SQSExtendedClient
wrong
```
from amazon_sqs_extended_client.SQSExtendedClient import SQSExtendedClient
```
correct
```
from amazon_sqs_extended_client import SQSExtendedClient
```
The primary class `SQSExtendedClient` is directly available under the main package namespace, not nested within a submodule of the same name.

Quickstart

This quickstart demonstrates how to initialize the `SQSExtendedClient` and use it to send and receive messages larger than the standard SQS limit. It requires an existing SQS queue and an S3 bucket configured in your AWS environment with appropriate permissions. The `always_through_s3` flag is set to `True` to ensure all messages, regardless of size, are handled via S3, which is often desired for consistency.

import boto3
from amazon_sqs_extended_client import SQSExtendedClient
import os

# Configure AWS credentials and region
# Ensure AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION are set as environment variables
aws_region = os.environ.get('AWS_REGION', 'us-east-1')
sqs_client = boto3.client('sqs', region_name=aws_region)
s3_client = boto3.client('s3', region_name=aws_region)

# SQS Queue URL and S3 Bucket Name must exist and be accessible
# Replace with your actual SQS Queue URL and S3 Bucket Name
SQS_QUEUE_URL = os.environ.get('SQS_QUEUE_URL', 'YOUR_SQS_QUEUE_URL')
S3_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME', 'YOUR_S3_BUCKET_NAME')

# Initialize the SQS Extended Client
sqs_extended_client = SQSExtendedClient(
    sqs_client=sqs_client,
    s3_client=s3_client,
    bucket_name=S3_BUCKET_NAME,
    always_through_s3=True # Always store messages in S3, regardless of size
)

# Example: Send a large message
large_message_body = 'A' * 300000  # 300KB message, exceeding SQS limit
print(f"Sending a large message ({len(large_message_body) / 1024} KB)...")
send_response = sqs_extended_client.send_message(
    QueueUrl=SQS_QUEUE_URL,
    MessageBody=large_message_body
)
print(f"Message sent! Message ID: {send_response.get('MessageId')}")

# Example: Receive the message
print("\nReceiving messages...")
receive_response = sqs_extended_client.receive_message(
    QueueUrl=SQS_QUEUE_URL,
    MaxNumberOfMessages=1
)

messages = receive_response.get('Messages', [])
if messages:
    message = messages[0]
    received_body = message.get('Body')
    print(f"Received message body (first 100 chars): {received_body[:100]}...")
    print(f"Received message length: {len(received_body)} bytes")
    
    # Delete the message after processing
    sqs_extended_client.delete_message(
        QueueUrl=SQS_QUEUE_URL,
        ReceiptHandle=message['ReceiptHandle']
    )
    print("Message deleted.")
else:
    print("No messages received.")