AWS 推出 CloudWatch Metric Streams

AWS 推出了 CloudWatch Metric Streams,把 CloudWatch Metric 的資料往 Kinesis Data Firehose 裡面丟:「CloudWatch Metric Streams – Send AWS Metrics to Partners and to Your Apps in Real Time」。

其中一個賣點是即時性比用 API 去拉好很多:

In order to make it easier for AWS Partners and others to gain access to CloudWatch metrics faster and at scale, we are launching CloudWatch Metric Streams. Instead of polling (which can result in 5 to 10 minutes of latency), metrics are delivered to a Kinesis Data Firehose stream.

格式上可以是 JSON 或是 Open Telemetry

When you set up a stream you choose between the binary Open Telemetry 0.7 format, and the human-readable JSON format.

另外一個賣點是價位,每千次 $0.003:

Pricing – You pay $0.003 for every 1000 metric updates, and for any charges associated with the Kinesis Data Firehose. To learn more, check out the pricing page.

另外算一下 Kinesis Data Firehose 的價錢,是以資料量的大小計費,不過最小計價單位是 5KB (一筆應該是不會到),單價是 $0.029/GB (us-east-1) 或是 $0.037/GB (ap-southeast-1),算了一下跟 CloudWatch Metrics Streams 比起來只是零頭...

之前如果要自己拉出來的話是透過 API call 抓,每 1000 次是 USD$0.01,這個方法相較起來便宜不少,不過數量多的時候還是一筆費用 (而且有不少 metrics 是一分鐘更新一次)。

如果只是要備份起來或是跑分析的話,也許先前用 API 拉的作法可能還是比較好?一個小時拉一次對於備份與分析應該都很夠了,而 alarm 的機制還是掛在 CloudWatch 上。

這次產品的定位看起來是要把 ecosystem 做起來:

We designed this feature with the goal of making it easier & more efficient for AWS Partners including Datadog, Dynatrace, New Relic, Splunk, and Sumo Logic to get access to metrics so that the partners can build even better tools.

Amazon Kinesis Streams 的 Video 版本:Amazon Kinesis Video Streams

這次 AWS 推出的 Amazon Kinesis Video Streams 在技術上看起來跟 Amazon Media Services 有不少重疊 (參考先前提到的文章「AWS Media Services 推出一卡車與影音相關的服務...」),但產品面上區隔開的服務:「Amazon Kinesis Video Streams – Serverless Video Ingestion and Storage for Vision-Enabled Apps」。

開頭介紹就有提到適合用在各種 IoT 裝置,用在一直有影像資料產生的設備上:

Cell phones, security cameras, baby monitors, drones, webcams, dashboard cameras, and even satellites can all generate high-intensity, high-quality video streams. Homes, offices, factories, cities, streets, and highways are now host to massive numbers of cameras.

像這張圖的所介紹的流程,以及可以保留天數的設計:

底層用了不少與 Amazon Media Services 相同的技術,但是包裝成不同的產品...

AWS PrivateLink

AWS 計畫把先前設計的 VPC Endpoint 都併到 AWS PrivateLink 裡,統一管理:「New – AWS PrivateLink for AWS Services: Kinesis, Service Catalog, EC2 Systems Manager, Amazon EC2 APIs, and ELB APIs in your VPC」。

Today we are announcing AWS PrivateLink, the newest generation of VPC Endpoints which is designed for customers to access AWS services in a highly available and scalable manner, while keeping all the traffic within the AWS network. Kinesis, Service Catalog, Amazon EC2, EC2 Systems Manager (SSM), and Elastic Load Balancing (ELB) APIs are now available to use inside your VPC, with support for more services coming soon such as Key Management Service (KMS) and Amazon Cloudwatch.

這樣就不用弄 proxy server 然後在上面管一堆 policy 了... (先不講自己搞 HA 的麻煩事,光是有些程式還得 patch 才能支援 proxy 就會想翻桌了 XD)

Amazon Kinesis Firehole 可以先轉換再進 S3 了...

前陣子 Amazon Kinesis Firehole 推出來的新功能,可以轉完後再寫進 Amazon S3 (或是其他地方):「Amazon Kinesis Firehose can now prepare and transform streaming data before loading it to data stores」。

文件是「Amazon Kinesis Firehose Data Transformation」這份,開頭有說明是透過 Lambda 做到的:

When you enable Firehose data transformation, Firehose buffers incoming data up to 3 MB or the buffering size you specified for the delivery stream, whichever is smaller. Firehose then invokes the specified Lambda function with each buffered batch asynchronously. The transformed data is sent from Lambda to Firehose for buffering. Transformed data is delivered to the destination when the specified buffering size or buffering interval is reached, whichever happens first.

而文件下方可以看到有些現成寫好的 Lambda 可以用,而且是還蠻常見的 case,像是 apache log 的處理,或是 syslog 的處理:

Lambda Blueprints

Firehose provides the following Lambda blueprints that you can use to create a Lambda function for data transformation.

General Firehose Processing — Contains the data transformation and status model described in the previous section. Use this blueprint for any custom transformation logic.

  • Apache Log to JSON — Parses and converts Apache log lines to JSON objects, using predefined JSON field names.
  • Apache Log to CSV — Parses and converts Apache log lines to CSV format.
  • Syslog to JSON — Parses and converts Syslog lines to JSON objects, using predefined JSON field names.
  • Syslog to CSV — Parses and converts Syslog lines to CSV format.

這樣配合 Amazon Athena 就是一包 serverless 架構了...

把 AWS 的紀錄串起來倒進 Elasticsearch,用 Kibana 呈現

在「CloudWatch Logs Subscription Consumer + Elasticsearch + Kibana Dashboards」這篇文章裡,AWS 官方給了很棒的 screenshot,用一堆服務包起來後可以得到這樣的效果,提供給有興趣的人: