通过Lambda捕获S3的增量文件

Database and Ruby, Python, History


  1. 创建两个 buckets ‘pzhong-test-bucket’ and ‘pzhong-incremental-bucket’
  2. 创建能够访问两个 buckets 的 policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws-cn:s3:::pzhong-test-bucket/*",
        "arn:aws-cn:s3:::pzhong-test-bucket"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws-cn:s3:::pzhong-incremental-bucket",
        "arn:aws-cn:s3:::pzhong-incremental-bucket/*"
      ]
    }
  ]
}
  1. 创建能够访问日志的 policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
  1. 创建一个新的角色,绑定上面两个 policy
  2. 创建 Lambda 函数
require 'aws-sdk-s3'

def lambda_handler(event:, context:)
  s3 = Aws::S3::Resource.new
  bucket = s3.bucket('pzhong-incremental-bucket')
  objects = event['Records'].map { |e| e['s3']['object'] }
  objects.each { |obj| bucket.put_object({ key: obj['key'], body: obj.to_json }) }
end
  1. 为 S3 配置 Lambda 函数,并指定运行 Lambda 的角色。
  2. 通过 cloudWatch 查看 Lambda 日志 https://console.amazonaws.cn/cloudwatch/home?region=cn-north-1#logsV2:log-groups