Cloud Custodian - Parallel Execution

Warnings

• breaking Python 3.9.2 is the minimum required version, and support is limited to Python 3.x (less than 4.0.0). Older Python versions are not supported.
  Fix: Ensure your environment uses Python >=3.9.2 and <4.0.0. Upgrade Python if necessary.
• gotcha Executing policies across multiple accounts requires properly configured cross-account IAM roles (e.g., `OrganizationAccountAccessRole` for AWS) in each target account that `c7n-org` will assume. Without correct permissions, policies will fail silently or with access denied errors.
  Fix: Verify that the IAM role specified in `accounts.yml` exists in each target account and grants sufficient permissions for `c7n-org` to assume it and execute policies.
• gotcha The `c7n-org report` command currently only supports generating reports from locally stored output directories. It cannot directly process output stored in cloud object storage (e.g., S3).
  Fix: Before generating a report, ensure that the policy execution output is synced or copied to a local directory accessible to the `c7n-org report` command.
• gotcha Logging from `c7n-org` can sometimes be too concise, hiding specific cloud provider error messages and making troubleshooting complex issues, especially permission-related ones, difficult.
  Fix: Increase logging verbosity where possible (e.g., `-v` flag if available), and consult underlying cloud provider logs (CloudTrail, CloudWatch Logs) for more detailed error information. Consider raising an issue for improved logging if specific gaps are identified.
• gotcha There have been reports of `c7n-org` encountering errors when targeting AWS accounts in non-default regions, potentially related to regional STS endpoints.
  Fix: Ensure the AWS CLI and Boto3 configurations properly handle regional STS endpoints. If issues persist, consider explicit region configurations in `accounts.yml` or check for updates to `c7n-org`.

Install

• pip install c7n-org Install c7n-org

Quickstart

c7n-org operates via the command line, requiring an `accounts.yml` file that defines the target cloud accounts and regions, and Custodian policy files (e.g., `policy.yml`). This example demonstrates setting up these files and shows the command to run `c7n-org` to execute policies across multiple accounts and regions. For AWS, the `accounts.yml` can be dynamically generated using `c7n-org aws-accounts -f accounts.yml` if AWS Organizations is configured.

import os

# Create a dummy accounts.yml for demonstration
accounts_yaml_content = '''
accounts:
  - account_id: '123456789012'
    name: dev-account
    regions:
      - us-east-1
      - us-west-2
    role: arn:aws:iam::123456789012:role/CloudCustodian
  - account_id: '987654321098'
    name: prod-account
    regions:
      - us-east-1
    role: arn:aws:iam::987654321098:role/CloudCustodian
'''

# Create a simple Custodian policy to find untagged S3 buckets
policy_yaml_content = '''
policies:
  - name: untagged-s3-buckets
    resource: aws.s3
    filters:
      - "tag:Project": absent
    actions:
      - type: notify
        subject: Untagged S3 Bucket Found
        to:
          - email@example.com # Replace with a valid email for actual use
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/123456789012/my-notification-queue
'''

# Write content to files
with open('accounts.yml', 'w') as f:
    f.write(accounts_yaml_content)

with open('policy.yml', 'w') as f:
    f.write(policy_yaml_content)

# Simulate running c7n-org via subprocess for demonstration
# In a real scenario, you would run this command in your shell
print("Simulating c7n-org execution...")
print("Command: c7n-org run -c accounts.yml -s output -p policy.yml")
print("\n--- Output Directory Structure ---")
print("output/")
print("├── dev-account/")
print("│   ├── us-east-1/")
print("│   │   └── untagged-s3-buckets/")
print("│   └── us-west-2/")
print("│       └── untagged-s3-buckets/")
print("└── prod-account/")
print("    └── us-east-1/")
print("        └── untagged-s3-buckets/")
print("----------------------------------")

# Cleanup dummy files
os.remove('accounts.yml')
os.remove('policy.yml')