Amazon Kinesis Streams 的 Video 版本:Amazon Kinesis Video Streams

這次 AWS 推出的 Amazon Kinesis Video Streams 在技術上看起來跟 Amazon Media Services 有不少重疊 (參考先前提到的文章「AWS Media Services 推出一卡車與影音相關的服務...」),但產品面上區隔開的服務:「Amazon Kinesis Video Streams – Serverless Video Ingestion and Storage for Vision-Enabled Apps」。

開頭介紹就有提到適合用在各種 IoT 裝置,用在一直有影像資料產生的設備上:

Cell phones, security cameras, baby monitors, drones, webcams, dashboard cameras, and even satellites can all generate high-intensity, high-quality video streams. Homes, offices, factories, cities, streets, and highways are now host to massive numbers of cameras.


底層用了不少與 Amazon Media Services 相同的技術,但是包裝成不同的產品...

AWS PrivateLink

AWS 計畫把先前設計的 VPC Endpoint 都併到 AWS PrivateLink 裡,統一管理:「New – AWS PrivateLink for AWS Services: Kinesis, Service Catalog, EC2 Systems Manager, Amazon EC2 APIs, and ELB APIs in your VPC」。

Today we are announcing AWS PrivateLink, the newest generation of VPC Endpoints which is designed for customers to access AWS services in a highly available and scalable manner, while keeping all the traffic within the AWS network. Kinesis, Service Catalog, Amazon EC2, EC2 Systems Manager (SSM), and Elastic Load Balancing (ELB) APIs are now available to use inside your VPC, with support for more services coming soon such as Key Management Service (KMS) and Amazon Cloudwatch.

這樣就不用弄 proxy server 然後在上面管一堆 policy 了... (先不講自己搞 HA 的麻煩事,光是有些程式還得 patch 才能支援 proxy 就會想翻桌了 XD)

Amazon Kinesis Firehole 可以先轉換再進 S3 了...

前陣子 Amazon Kinesis Firehole 推出來的新功能,可以轉完後再寫進 Amazon S3 (或是其他地方):「Amazon Kinesis Firehose can now prepare and transform streaming data before loading it to data stores」。

文件是「Amazon Kinesis Firehose Data Transformation」這份,開頭有說明是透過 Lambda 做到的:

When you enable Firehose data transformation, Firehose buffers incoming data up to 3 MB or the buffering size you specified for the delivery stream, whichever is smaller. Firehose then invokes the specified Lambda function with each buffered batch asynchronously. The transformed data is sent from Lambda to Firehose for buffering. Transformed data is delivered to the destination when the specified buffering size or buffering interval is reached, whichever happens first.

而文件下方可以看到有些現成寫好的 Lambda 可以用,而且是還蠻常見的 case,像是 apache log 的處理,或是 syslog 的處理:

Lambda Blueprints

Firehose provides the following Lambda blueprints that you can use to create a Lambda function for data transformation.

General Firehose Processing — Contains the data transformation and status model described in the previous section. Use this blueprint for any custom transformation logic.

  • Apache Log to JSON — Parses and converts Apache log lines to JSON objects, using predefined JSON field names.
  • Apache Log to CSV — Parses and converts Apache log lines to CSV format.
  • Syslog to JSON — Parses and converts Syslog lines to JSON objects, using predefined JSON field names.
  • Syslog to CSV — Parses and converts Syslog lines to CSV format.

這樣配合 Amazon Athena 就是一包 serverless 架構了...

把 AWS 的紀錄串起來倒進 Elasticsearch,用 Kibana 呈現

在「CloudWatch Logs Subscription Consumer + Elasticsearch + Kibana Dashboards」這篇文章裡,AWS 官方給了很棒的 screenshot,用一堆服務包起來後可以得到這樣的效果,提供給有興趣的人: