{"id":8823,"library":"amazon-sqs-extended-client","title":"Amazon SQS Extended Client Library for Python","description":"The Amazon SQS Extended Client Library for Python extends Amazon SQS to handle message payloads larger than the standard 256 KB limit, supporting sizes up to 2 GB. It achieves this by transparently storing the message payload in an Amazon S3 bucket and sending a reference to the S3 object in the SQS message. The library is currently at version 1.0.1 and maintains a release cadence tied to feature development and AWS SDK updates.","status":"active","version":"1.0.1","language":"en","source_language":"en","source_url":"https://github.com/awslabs/amazon-sqs-python-extended-client-lib","tags":["aws","sqs","s3","messaging","large-messages","cloud"],"install":[{"cmd":"pip install amazon-sqs-extended-client","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"The library is built upon the AWS SDK for Python (Boto3) for interacting with SQS and S3.","package":"boto3","optional":false}],"imports":[{"note":"The primary class `SQSExtendedClient` is directly available under the main package namespace, not nested within a submodule of the same name.","wrong":"from amazon_sqs_extended_client.SQSExtendedClient import SQSExtendedClient","symbol":"SQSExtendedClient","correct":"from amazon_sqs_extended_client import SQSExtendedClient"}],"quickstart":{"code":"import boto3\nfrom amazon_sqs_extended_client import SQSExtendedClient\nimport os\n\n# Configure AWS credentials and region\n# Ensure AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION are set as environment variables\naws_region = os.environ.get('AWS_REGION', 'us-east-1')\nsqs_client = boto3.client('sqs', region_name=aws_region)\ns3_client = boto3.client('s3', region_name=aws_region)\n\n# SQS Queue URL and S3 Bucket Name must exist and be accessible\n# Replace with your actual SQS Queue URL and S3 Bucket Name\nSQS_QUEUE_URL = os.environ.get('SQS_QUEUE_URL', 'YOUR_SQS_QUEUE_URL')\nS3_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME', 'YOUR_S3_BUCKET_NAME')\n\n# Initialize the SQS Extended Client\nsqs_extended_client = SQSExtendedClient(\n    sqs_client=sqs_client,\n    s3_client=s3_client,\n    bucket_name=S3_BUCKET_NAME,\n    always_through_s3=True # Always store messages in S3, regardless of size\n)\n\n# Example: Send a large message\nlarge_message_body = 'A' * 300000  # 300KB message, exceeding SQS limit\nprint(f\"Sending a large message ({len(large_message_body) / 1024} KB)...\")\nsend_response = sqs_extended_client.send_message(\n    QueueUrl=SQS_QUEUE_URL,\n    MessageBody=large_message_body\n)\nprint(f\"Message sent! Message ID: {send_response.get('MessageId')}\")\n\n# Example: Receive the message\nprint(\"\\nReceiving messages...\")\nreceive_response = sqs_extended_client.receive_message(\n    QueueUrl=SQS_QUEUE_URL,\n    MaxNumberOfMessages=1\n)\n\nmessages = receive_response.get('Messages', [])\nif messages:\n    message = messages[0]\n    received_body = message.get('Body')\n    print(f\"Received message body (first 100 chars): {received_body[:100]}...\")\n    print(f\"Received message length: {len(received_body)} bytes\")\n    \n    # Delete the message after processing\n    sqs_extended_client.delete_message(\n        QueueUrl=SQS_QUEUE_URL,\n        ReceiptHandle=message['ReceiptHandle']\n    )\n    print(\"Message deleted.\")\nelse:\n    print(\"No messages received.\")\n","lang":"python","description":"This quickstart demonstrates how to initialize the `SQSExtendedClient` and use it to send and receive messages larger than the standard SQS limit. It requires an existing SQS queue and an S3 bucket configured in your AWS environment with appropriate permissions. The `always_through_s3` flag is set to `True` to ensure all messages, regardless of size, are handled via S3, which is often desired for consistency."},"warnings":[{"fix":"Verify IAM policies for the AWS user/role sending and receiving messages. Grant `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject` on the S3 bucket and `sqs:*` on the SQS queue.","message":"The Amazon SQS Extended Client Library for Python explicitly depends on an S3 bucket for storing large message payloads. Ensure the specified S3 bucket exists and that your AWS credentials (used by Boto3 clients) have the necessary permissions (s3:PutObject, s3:GetObject, s3:DeleteObject) for the bucket and sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage for the SQS queue. Lack of S3 permissions is a common cause of `Failed to get the S3 object which contains the payload` errors.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Always install using `pip install amazon-sqs-extended-client` and import with `from amazon_sqs_extended_client import SQSExtendedClient`.","message":"There are several similarly named Python libraries (`aws-sqs-ext-client`, `sqs-extended-client`, `pysqs-extended-client`) that aim to provide similar functionality. Ensure you are using `amazon-sqs-extended-client` (from `awslabs` on GitHub) for the official and actively maintained version to avoid compatibility issues or unexpected behavior.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Rely on the `SQSExtendedClient.delete_message` or `delete_message_batch` calls to ensure both the SQS message and its S3 payload are removed synchronously. Avoid manual deletion of S3 objects referenced by active SQS messages.","message":"When deleting messages, the extended client automatically handles the deletion of the corresponding S3 object if the message payload was stored in S3. However, if the S3 object is manually deleted before the SQS message is processed and deleted, the consumer might fail to retrieve the full message body, leading to `Failed to get the S3 object which contains the payload` errors.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Verify that the AWS credentials used by the consumer have `s3:GetObject` permission on the S3 bucket where large payloads are stored. Also, ensure that S3 objects are not being prematurely deleted.","cause":"The SQS Extended Client could not retrieve the message payload from the referenced S3 bucket. This commonly indicates incorrect IAM permissions (e.g., missing s3:GetObject) or that the S3 object was deleted before the message was processed.","error":"ERROR [S3Dao] Failed to get the S3 object which contains the payload."},{"fix":"Correct the import statement to `from amazon_sqs_extended_client import SQSExtendedClient`.","cause":"Attempting to import the `SQSExtendedClient` class from an incorrect submodule path. The class is directly available at the package root.","error":"ModuleNotFoundError: No module named 'amazon_sqs_extended_client.SQSExtendedClient'"},{"fix":"Ensure the `MessageBody` provided to `send_message` contains valid string data, even if it's a small placeholder.","cause":"This error occurs when `send_message` is called with an empty or `None` `MessageBody`. While SQS itself might tolerate some forms of empty messages, the extended client might have internal validations or the underlying S3 put operation might fail for empty content.","error":"ClientError: An error occurred (InvalidParameterValue) when calling the SendMessage operation: The message body must not be empty."}]}