Tag Archives: lambda

Amazon ECS 可以跑 cron job 了...

Amazon ECS 上面固定時間跑某些東西,以前得自己用 AWS Lambda 帶 (或是自己架,不過這樣就要自己考慮 High Availability 架構了),現在則是直接支援:「Amazon ECS Now Supports Time and Event-Based Task Scheduling」。

Previously, you could start and stop Amazon ECS tasks manually, but running tasks on a schedule required writing and integrating an external scheduler with the Amazon ECS API.

Now you can schedule tasks through the Amazon ECS console on fixed time intervals (e.g.: number of minutes, hours, or days). Additionally, you can now set Amazon ECS as a CloudWatch Events target, allowing you to launch tasks by using CloudWatch Events.

用 Amazon Elasticsearch 看 VPC Flow Logs

在「How to Visualize and Refine Your Network’s Security by Adding Security Group IDs to Your VPC Flow Logs」這篇雖然是講特定功能,但還是把怎麼架設從頭到尾都講了一次...

比較特別的幾張圖:

然後再回來看怎麼串:

AWS Lambda 支援 Python 3.6 了

大家都等很久的需求總算推出來了 (雖然我覺得還好...),AWS Lambda 要支援 Python 3.6 了:「AWS Lambda Supports Python 3.6」。會逐步推上線:

You can now develop your AWS Lambda functions using Python 3.6. This feature will become available in all AWS Lambda regions within 24 hours.

Amazon Kinesis Firehole 可以先轉換再進 S3 了...

前陣子 Amazon Kinesis Firehole 推出來的新功能,可以轉完後再寫進 Amazon S3 (或是其他地方):「Amazon Kinesis Firehose can now prepare and transform streaming data before loading it to data stores」。

文件是「Amazon Kinesis Firehose Data Transformation」這份,開頭有說明是透過 Lambda 做到的:

When you enable Firehose data transformation, Firehose buffers incoming data up to 3 MB or the buffering size you specified for the delivery stream, whichever is smaller. Firehose then invokes the specified Lambda function with each buffered batch asynchronously. The transformed data is sent from Lambda to Firehose for buffering. Transformed data is delivered to the destination when the specified buffering size or buffering interval is reached, whichever happens first.

而文件下方可以看到有些現成寫好的 Lambda 可以用,而且是還蠻常見的 case,像是 apache log 的處理,或是 syslog 的處理:

Lambda Blueprints

Firehose provides the following Lambda blueprints that you can use to create a Lambda function for data transformation.

General Firehose Processing — Contains the data transformation and status model described in the previous section. Use this blueprint for any custom transformation logic.

  • Apache Log to JSON — Parses and converts Apache log lines to JSON objects, using predefined JSON field names.
  • Apache Log to CSV — Parses and converts Apache log lines to CSV format.
  • Syslog to JSON — Parses and converts Syslog lines to JSON objects, using predefined JSON field names.
  • Syslog to CSV — Parses and converts Syslog lines to CSV format.

這樣配合 Amazon Athena 就是一包 serverless 架構了...

在 CloudFront 的 edge 上跑 Lambda

所以 Amazon CloudFront 讓使用者在 edge 上跑程式了 (雖然目前是 limited preview):「Lambda@Edge – Preview」。

分成 Viewer Request、Origin Request、Origin Response 以及 Viewer Response 四個階段可以插入修改。另外有些限制:

Because your JavaScript code will be part of the request/response path, it must be lean, mean, and self-contained. It cannot make calls to other web services and it cannot access other AWS resources. It must run within 128 MB of memory, and complete within 50 ms.

要在 128MB 內搞定,而且不能呼叫其他資源。不過這樣已經可以做很多事了... 基本上就是一台 turing machine 了 :o

用 Lambda 做 DynamoDB 的 Auto Scaling

AWS Lambda 可以跑 cron job 後應該就不怎麼意外出現了:「Autoscale DynamoDB provisioned capacity using Lambda」。

不像 EC2Auto Scaling,或是 ELB 自己會成長或縮小,DynamoDB 跟其他 AWS 服務不同,雖然可以 scale,但需要自己手動設定 capacity 伸縮。

於是就有人寫了程式 (也就是這個專案),判斷目前的 r/w 用量來決定策略... 有點像是我在處理自家 bandwidth 的搞法,達到某個警戒值就自動增加導去 CDN 的量,或是降低回來 :o

AWS 的 scheduled events 可以設定為一分鐘一次了

Twitter 上看到 AWS 的 scheduled events 可以設定為一分鐘一次了:

在 AWS 的相關文件上也發現相關的限制都拿掉了:「Using AWS Lambda with Scheduled Events」。

實際測試也發現沒問題了,之前用 */1 會被擋下來,現在是 ok 的:

aws events put-rule --schedule-expression 'cron(*/1 * * * ? *)' --name test

AWS Lambda 支援 Node.js 4.3

AWS 宣佈 Lambda 支援 Node.js 4.3:「AWS Lambda Supports Node.js 4.3」:

You can now develop your AWS Lambda functions using Node.js 4.3.2 in addition to Node.js 0.10.4.

另外同步在「Node.js 4.3.2 Runtime Now Available on Lambda」這邊也有文章介紹。

這樣總算可以拿出新的套件以及語法了...

AWS Lambda 可以在 VPC 裡面執行了

AWS Lambda 可以存取 VPC 內的資源了:「New – Access Resources in a VPC from Your Lambda Functions」。

這包括了 RDSElastiCache 這類放資料或是狀態的資源。以往有些要自己處理 High Availability 但又不怎麼大的功能,可以改寫成 AWS Lambda 往上丟了...