AWS S3 log 使用及 CloudTrail 整合方式

（本篇當初為了公司而寫所以暫時為英文版，有空把它變成中文）

Introduction

1. Understand how to open AWS S3 bucket logs and how to query the stored logs

2. Understand how to control user read and write permissions for S3 buckets

3. Integrate CloudTrail to make the log query process more convenient

Create IAM user for different permission

Go to Identity and Access Management (IAM) for create 2 user, we are going to let

• sun: read/write access for S3

• moon: read only for S3
  

Go to Policies and give user sun full permission for S3

Click into policy and attach user for this policy

Also, give user moon “AmazonS3ReadOnlyAccess” policy.

Create S3 for log save

We can log in user sun for create the S3 bucket, login url can be found in IAM Dashboard.

So now, go to Amazon S3 for create.

Didn’t need to change anything during create.

After create the S3 for saving log (log-kang), we can create the other one for Production env (In the image, it display as prod-kang)

When your source bucket and destination bucket are the same bucket, additional logs are created for the logs that are written to the bucket, which creates an infinite loop of logs. We do not recommend doing this because it could result in a small increase in your storage billing. In addition, the extra logs about logs might make it harder to find the log that you are looking for.

Set server logging enable

Now, click in prod-kang and go to Properties, scroll down to find “Server access log”, enable it and define the save path


In this image, we have create “prod-log” folder in log-kang S3, so we can find our log in it.

Get the log

Go in log-kang S3 bucket, we can see the log inside (separate in many file)

We can open the log file by clicking “Open” button instead of URL

or we can use AWS CLI to download all the log and migrate it by

aws s3 cp s3://log-kang/prod-log . --recursive

and we can execute command in the file, so we can get the combined_log.txt

cat * >> combined_log.txt

Compare: CloudTrail vs S3 server access logs

CloudTrail Data Events	S3 server access logs
API calls at Object level	Access calls to S3 objects
All the S3 buckets for the AWS account or just for some folder in S3 bucket	individual bucket level
	give more comprehensive information about the logs like BucketOwner, HTTPStatus, ErrorCode
0.10 per 100,000 data events delivered	Free

CloudTrail Log

Example: PutBucketLogging

{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "Root",
        "principalId": "304226915015",
        "arn": "arn:aws:iam::304226915015:root",
        "accountId": "304226915015",
        "accessKeyId": "ASIAUNVKWKLDZHPIM7ZN",
        "sessionContext": {
            "attributes": {
                "creationDate": "2024-01-10T02:17:54Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2024-01-10T03:26:11Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "PutBucketLogging",
    "awsRegion": "ap-east-1",
    "sourceIPAddress": "202.39.237.203",
    "userAgent": "[S3Console/0.4, aws-internal/3 aws-sdk-java/1.12.488 Linux/5.10.201-168.748.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.372-b08 java/1.8.0_372 vendor/Oracle_Corporation cfg/retry-mode/standard]",
    "requestParameters": {
        "logging": "",
        "bucketName": "useless.s3",
        "BucketLoggingStatus": {
            "xmlns": "http://s3.amazonaws.com/doc/2006-03-01/",
            "LoggingEnabled": {
                "TargetPrefix": "log/",
                "TargetObjectKeyFormat": {
                    "SimplePrefix": ""
                },
                "TargetBucket": "useless.s3"
            }
        },
        "Host": "s3.ap-east-1.amazonaws.com"
    },
    "responseElements": null,
    "additionalEventData": {
        "SignatureVersion": "SigV4",
        "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "bytesTransferredIn": 258,
        "AuthenticationMethod": "AuthHeader",
        "x-amz-id-2": "Wu0MQhAUrPP2ZGo3WIQ7AM4SbnhVxMkpcoyTyWYMmLdVMLaAFBuaN8lMZZV79uhm9gJoJrOpuCY=",
        "bytesTransferredOut": 0
    },
    "requestID": "YW4JEQZ6020S2NV7",
    "eventID": "e3d5ad2b-8836-440f-bd1c-e6ec56779c52",
    "readOnly": false,
    "resources": [
        {
            "accountId": "304226915015",
            "type": "AWS::S3::Bucket",
            "ARN": "arn:aws:s3:::useless.s3"
        }
    ],
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "304226915015",
    "vpcEndpointId": "vpce-0e5823d23ba7d10ce",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "s3.ap-east-1.amazonaws.com"
    }
}

S3 server access logs

How to use CloudTrail

Go to CloudTrail, click on “Create Trail”


After created, we will go in “Choose log events”, there’ll have 3 Event type:

• Management events: Capture management operations performed on your AWS resources.

  • Management events capture management operations, such as API use, that are performed on resources in your AWS account. These are also known as control plane operations. Management events can also include non-API events that occur in your account, such as the ConsoleLogin event that occurs when a user signs in to your AWS account.

• Data events: Log the resource operations performed on or within a resource.

  • Data events show the resource operations performed on or within a resource, such as when a user uploads a file to an S3 bucket. These are also known as data plane operations. For more information about available data events, see Logging data events.

• Insights events: Identify unusual activity, errors, or user behavior in your account.

  • Helps you identify and respond to unusual activity associated with API calls and API error rates by continuously analyzing CloudTrail management events

We can select to record all the operation for all S3 bucket, or we can go into “custom” choice with specific S3 bucket.

Append: Full access for specific S3

Now we want to give user moon have full access to S3 bucket share-resource-kang, which create by user sun

Go to IAM policies and create

Just select JSON and copy paste below

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Describe*",
                "s3-object-lambda:Get*",
                "s3-object-lambda:List*"
            ],
            "Resource": [
                "arn:aws:s3:::share-resource-kang",
                "arn:aws:s3:::share-resource-kang/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::share-resource-kang/*"
        }
    ]
}

So that we can give user moon “s3:PutObject” and “s3:DeleteObject” permission.