Home » Posts tagged "cost"

Amazon EFS 的 IA Storage Class

Amazon EFS 一直找不太到好用的使用情境,因為 NFS 的關係所以大量 I/O 時的 latency issue 使得速度快不起來,而拿來堆 log 的成本又超級高...

最新推出的 storage class 則是透過提供低儲存成本的版本,解決了堆 log 這種使用情境:「New – Infrequent Access Storage Class for Amazon Elastic File System (EFS)」。

不過 EFS 不像 S3 可以直接選擇 storage class,是需要讓系統管理的:

開啟後 30 天沒有被碰過的檔案就會切過去:

Eligible Files – Files that are 128 KiB or larger and that have not been accessed or modified for at least 30 days can be transitioned to the new storage class. Modifications to a file’s metadata that do not change the file will not delay a transition.

而 latency 也會增加:

Files that have not been read or written for 30 days will be transitioned to the Infrequent Access storage class with no further action on your part. Files in the Standard Access class can be accessed with latency measured in single-digit milliseconds; files in the Infrequent Access class have latency in the double-digits.

us-east-1 為例子來說,Standard 是 USD$0.3/GB-month,而 IA 只要 USD$0.045/GB-month,但抓取時會有 USD$0.01/GB 的傳輸費用,可以看出價錢低不少。

不過文章裡沒提到什麼時候會把資料從 IA 跳回 Standard,可能得找機會問問看...

DynamoDB Autoscaling 的各種眉眉角角...

AdRollDynamoDB Autoscaling 的踩雷記錄,裡面有些資訊如果不是跳下去玩應該不會注意到 (魔鬼藏在細節裡的感覺):「Managing DynamoDB Autoscaling with Lambda and Cloudwatch」。

第一個提到的問題是 autoscaling 的觀察對象:

Ideally, the table should scale based on the number of requests that we are making , not the number of requests that are successful.

另外一個是 autoscaling 遇到完全不用的情況下不會 scale down,看起來是某種保護機制。但這使得平常只有拿來讀取的表格在跑完 batch job 後得自己處理 write scale down 問題:

Additionally, at the time of implementing this algorithm, the DynamoDB capacity could not be brought down automatically if the consumption was exactly zero, which can happen if you write to your table in batch instead of realtime, for example.

This meant that, when enabling autoscaling, tables that were read in realtime, but written to in batch, still needed manual intervention to bring the write capacity down after our jobs were done writing.

另外一個問題是 scale down 是有次數限制的:

Another interesting point that might bite users is that capacity decreases are an expensive operation for AWS, so they’re limited.

The number of decreases cited in the documentation can be achieved under very special conditions, since you need to have 4 decreases in the first hour of the day plus one for each of the remaining hours, for a total of 4 (first hour) + 23 (1 hourly) = 27.

後面就是自己研究什麼 algorithm 可以調整的更細,然後用 lambda 重寫... 最後省下 30% 的成本:

Here is where we detected our costs for our batch tables dropping to around 30% of the initial cost.

AdRoll 的規模應該是不小,所以為了省 30% 可以花不少力氣在上面...

Linode 推出 Dedicated CPU Instances

Linode 決定要推出 Dedicated CPU Instances,看起來相當於 AWSEC2 裡一般性的 EC2 instance (指非 t 系列的),跟之前 CPU 共用而導致 CPU 可能時快時慢不同,這個方案讓 CPU resource 穩定輸出:「Introducing: Linode Dedicated CPU Instances」。

Dedicated instances are optimized for workloads where consistent performance is required or where full-duty work (100% CPU all day, every day) needs doing.

拿 EC2 裡的 c5.large (2vCPU + 4GB) 比較差不多是一半的價錢,但實際不知道差多少... 回台灣的時候測試看看好了。看起來如果真的要在 Linode 上跑 Kubernetes 的話反而要用這種機器?

AWS 宣佈 Fargate 大幅降價

AWS Fargate 大幅降價:「Announcing AWS Fargate Price Reduction By Up To 50%」、「AWS Fargate Price Reduction – Up to 50%」。

先前 Fargate 的定價超級高,基本上是不會有正常應用會丟上去的情境... 如果可以接受 Fargate 的價錢,平常就用 ECS 或是 EKS 開三到五倍的機器在旁邊待機就好了,container scale 的速度還比 Fargate 快。

這次降價幅度很大,vCPU 降 20% 其實不算小,但相較於 Memory 降的 65% 就感覺好像沒降很多 (...):

Effective Jan 07, 2019, we are reducing the price for AWS Fargate by 20% for vCPU and 65% for memory across all regions where Fargate is currently available.

us-east-1m5.24xlarge 的價錢來看是 USD$4.608/hr。以他的 96 vCPU + 384 GiB RAM 對應到 Fargate 來算是 USD$5.59296/hr,大約多 21% 的價錢。


Twitch 用 VP9 直播...

Twitch 整理了一篇「How VP9 delivers value for Twitch’s esports live streaming」,說明他們用 VP9 的經驗談。

裡面有很大的篇幅是在講 VP9 與 H.264 的比較,不過這兩個用的技術就已經不是同一個年代了,沒有進步的話就不用出來玩了...

裡面有講到一些有趣的東西,像是提到是用 FPGA 即時壓縮:

In this article, we will show that the FPGA-based real-time VP9 encoding can deliver at least 25% bitrate savings compared to the highest-quality H.264 encoders deployed in Twitch’s production today.

然後提到 1080p60 至少省了 25% 的頻寬 (這邊應該是相較於 H.264):

VP9’s Compression Efficiency for Live 1080p60 Encoding: We Can Achieve At Least 25% Bitrate Savings


VP9 is implemented in these web browsers:

Chromium and Google Chrome (usable by default since version 29 from May and August 2013, respectively),
Opera (since version 15 from July 2013),
Mozilla Firefox (since version 28 from March 2014),
Microsoft Edge (as of summer 2016).

行動裝置的話 Android 4.4+ 有支援,但在 iOS 上沒有支援...

整體看起來普及率算是不低,可以引入當主力 codec 降低頻寬成本,當設備不支援 VP9 時 (應該只有 iOS 透過 Safari 觀看的情況) 就用 H.264 stream 提供服務。

EC2 開始陸續推出支援 100Gbps 網路的機器

AWS 開始陸陸續續在推出有 100Gbps 能力的 EC2 instance 了:「New – EC2 P3dn GPU Instances with 100 Gbps Networking & Local NVMe Storage for Faster Machine Learning + P3 Price Reduction」。

從「Amazon EC2 Instance Types」這邊可以看到先前只有 c5n.18xlarge 有支援 100Gbps 網路,現在推出的 p3dn.24xlarge 是第二個支援的...

另外是 P3 系列的降價消息,比較奇怪的是從 2018/12/06 開始生效,而不是從月初開始。另外區域與條件也有一些複雜,有常在用的人可以翻一下說明...

EC2 支援休眠

AWSEC2 instance 推出了休眠模式:「New – Hibernate Your EC2 Instances」。

休眠模式主要是透過保留記憶體內的內容 (會寫到 EBS 裡),讓機器開起來比較快,不過這邊有要求 root EBS volume 需要加密:

When an instance is instructed to hibernate, it writes the in-memory state to a file in the root EBS volume and then (in effect) shuts itself down. The AMI used to launch the instance must be encrypted, as must the root EBS volume of the instance. The encryption ensures proper protection for sensitive data when it is copied from memory to the EBS volume.

費用的部分是 EBS volume 與 Elastic IP:

While the instance is in hibernation, you pay only for the EBS volumes and Elastic IP Addresses attached to it; there are no other hourly charges (just like any other stopped instance).

一般對外服務的伺服器好像用不太到... 應該是內部系統或是開發機的需求?

Amazon S3 推出了一個自動分析後分類的 Storage Class

Amazon S3 推出了新的 Storage Class,後面直接用演算法分析 access pattern (所以要跑一陣子才會生效),然後決定要放到 Standard 或是 Standard IA 裡:「Announcing S3 Intelligent-Tiering — a New Amazon S3 Storage Class」。

混了 Standard 與 Standard IA:

S3 Intelligent-Tiering stores objects in two access tiers: one tier that is optimized for frequent access and another lower-cost tier that is optimized for infrequent access.

然後連續 30 天沒有被存取的就會被丟到 Standard IA,如果有被存取的話就會被搬回來,而搬移的部份不用收費:

For a small monthly monitoring and automation fee per object, S3 Intelligent-Tiering monitors access patterns and moves objects that have not been accessed for 30 consecutive days to the infrequent access tier. There are no retrieval fees in S3 Intelligent-Tiering. If an object in the infrequent access tier is accessed later, it is automatically moved back to the frequent access tier. No additional tiering fees apply when objects are moved between access tiers within the S3 Intelligent-Tiering storage class.

從費用上可以看到演算法本身是有費用的,換算一下 1M objects 是 USD$2.5/month,好像還可以...

Monitoring and Automation, All storage / Month$0.0025 per 1,000 objects

不過有蠻多要注意的 pattern。像是這邊有提到 128KB 以下的檔案不會搬到 IA 上,但不知道算不算 Monitoring 的費用?

S3 Intelligent-Tiering has a minimum eligible object size of 128KB for auto-tiering. Smaller objects may be stored but will always be charged at the Frequent Access tier rates.

另外這邊講 S3 Intelligent-Tiering 的三十天也不知道是不是 Standard + Standard IA,或是分開算:

S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA storage are charged for a minimum storage duration of 30 days.


Amazon EFS 也要推出 IA 版本了 (Infrequent Access)

Amazon EFS 也要推出 IA (Infrequent Access) 版本了,Infrequent Access 指的是不常存取的資料:「Coming Soon – Amazon EFS Infrequent Access Storage Class」。

這剛好配合上很多人拿 Amazon EFS 來堆 log 的行為... AWS 是說有機會到省到 85%,不過應該是非常大的量才有機會有這個價錢?

EFS IA reduces storage costs for files not accessed every day, with savings up to 85% compared to the EFS Standard storage class.

其實用過 Amazon EFS 的人都對效能抱怨頗嚴重 (透過 NFS 有太多操作沒辦法 cache,於是 network latency issue 就出現了),堆 log 或是當作跨機器的空間大概是目前的主流用法...

Amazon API Gateway 推出分級收費 (降價)

Amazon API Gateway 推出分級收費:「Amazon API Gateway Announces Tiered Pricing」。

原先的費用不變,大多數的地區是超過 333M reqs/month 的部分降價了... (不過雪梨跟南非是超過 1B reqs/month,而且北卡超過 333M 的部分也只降個零頭,實際比較深的折扣還是在 1B),333M reqs/month 這個量換算下來需要 11.1M reqs/day,平均值要 128 reqs/sec,看起來是設計給整個站都搬上去的折扣方案 (不然就是本身量就超大)。