AWS 上用空間買 IOPS 的故事...

在「A web performance issue」這邊講到 Mozilla 的系統產生效能問題,後續的 trouble shooting 以及解決問題的方案。

這個系統跑在 AWS 上,在一連串確認後發現是 RDS 所使用的 EBS 的 IOPS 滿了:

After reading a lot of documentation about Amazon’s RDS set-up I determined that slow downs in the database were related to IOPS spikes. Amazon gives you 3 IOPS per Gb and with a storage of 1 Terabyte we had 3,000 IOPS as our baseline. The graph below shows that at times we would get above that max baseline.

然後大家對於解法都差不多,因為 Provisioned IOPS 太貴,所以直接加大空間換 IOPS 出來 (因為 General SSD 裡 1 GB 給 3 IOPS):

To increase the IOPS baseline we could either increase the storage size or switch from General SSD to Provisioned IOPS storage. The cost of the different storage type was much higher so we decided to double our storage, thus, doubling our IOPS baseline. You can see in the graph below that we’re constantly above our previous baseline. This change helped Treeherder’s performance a lot.


In order to prevent getting into such a state in the future, I also created a CloudWatch alert. We would get alerted if the combined IOPS is greater than 5,700 IOPS for 6 datapoints within 10 minutes.

不過 General SSD 的 IOPS 是沒有 100% 保證的,只有這樣寫:

AWS designs gp2 volumes to deliver 90% of the provisioned performance 99% of the time.


Amazon EBS (gp2) 提昇效能...

AWS 宣佈提昇了 Amazon EBS (gp2) 的效能:「Amazon EBS Increases Performance of General Purpose SSD (gp2) Volumes」。

本來上限是 10k IOPS,現在提升到 16k IOPS 了。另外最大傳輸速度也從 160 MB/sec 提升到 250 MB/sec:

Today we are announcing a 60% improvement in performance of General Purpose SSD (gp2) Volumes from 10,000 IOPS to 16,000 IOPS and from 160 MB/s to 250 MB/s of throughput per volume.

應該還是維持 3 IOPS per GB 的設計,但這對於想要用 gp2 堆效能的人來說算是好消息,可以用更大的空間堆出更多 IOPS 了... (像是「Percona 的人接受 AWS 的建議,重新測試了 Percona XtraDB Cluster 在 gp2 上的效能...」這篇)

Percona 的人接受 AWS 的建議,重新測試了 Percona XtraDB Cluster 在 gp2 上的效能...

去年年底的時候 Percona 的人在 AWS 上測試 Percona XtraDB Cluster 的效能,尤其是針對底層應該選擇哪種 EBS 的部分給了一些建議。可以參考先前寫的「Percona 分析在 AWS 上跑 Percona XtraDB Cluster 的效能 (I/O bound)」這篇。

當時的建議是用 io1,雖然是比較貴,但對於效能比較好。

而後來 Percona 的人收到 AWS 工程師的建議,可以用另外一個方式,可以在 gp2 上拉出類似的效能,但成本會比 io1 低不少:「Percona XtraDB Cluster on Amazon GP2 Volumes」。

這個方式是利用 gp2 會依照空間大小,計算可用的 IOPS。在官方的文件裡是這樣描述 gp2 的效能 (IOPS):

General Purpose SSD (gp2) volumes offer cost-effective storage that is ideal for a broad range of workloads. These volumes deliver single-digit millisecond latencies and the ability to burst to 3,000 IOPS for extended periods of time. Between a minimum of 100 IOPS (at 33.33 GiB and below) and a maximum of 10,000 IOPS (at 3,334 GiB and above), baseline performance scales linearly at 3 IOPS per GiB of volume size. AWS designs gp2 volumes to deliver the provisioned performance 99% of the time. A gp2 volume can range in size from 1 GiB to 16 TiB.

在這個前提下,需要 10000 IOPS 的效能會需要 3.3TB 以上的空間,所以 Percona 就被 AWS 的工程師建議直接拉高空間重新測試:

After publishing our material, Amazon engineers pointed that we should try GP2 volumes with the size allocated to provide 10000 IOPS. If we allocated volumes with size 3.3 TiB or more, we should achieve 10000 IOPS.


接下來就比較儲存成本,大約是 io1 版本的一半價錢:

如上面文件中提到的,gp1 不完全保證效能,但統計出來經常能夠提供出 3 IOPS/GB 的效能。而 io1 則是保證效能,不太需要擔心效能不穩定的問題。就是這個差異,反應到成本上面就有蠻大的差距。善用這點設計系統,應該會對整體成本有蠻大的幫助... (但對 latency 就未必了,尤其是 P99 之類的數值)


AWS 提昇了 Amazon EBS 能提供的效能上限

AWS 宣佈 Amazon EBS 可以提供的效能往上提高了 (這邊講的是 Provisioned IOPS SSD,代號 io1):「Amazon EBS Improves Performance for io1 Volumes」。

單一 volume 的 IOPS 從 20K 變成 32K,thoughput 從 320MB/sec 變成 500MB/sec:

Today we are announcing an improvement in performance of Provisioned IOPS SSD (io1) Volumes from 20,000 IOPS to 32,000 IOPS and from 320 MB/s to 500 MB/s of throughput per volume.

應該是科技的進步帶動的 XD

Amazon RDS 支援更大的硬碟空間與更多的 IOPS

Amazon RDS 的升級:「Amazon RDS Now Supports Database Storage Size up to 16TB and Faster Scaling for MySQL, MariaDB, Oracle, and PostgreSQL Engines」。

空間上限從 6TB 變成 16TB,而且可以無痛升。另外 IOPS 上限從 30K 變成 40K:

Starting today, you can create Amazon RDS database instances for MySQL, MariaDB, Oracle, and PostgreSQL database engines with up to 16TB of storage. Existing database instances can also be scaled up to 16TB storage without any downtime.

The new storage limit is an increase from 6TB and is supported for Provisioned IOPS and General Purpose SSD storage types. You can also provision up to 40,000 IOPS for Provisioned IOPS storage volumes, an increase from 30,000 IOPS.

不過隔壁的 Amazon Aurora 還是大很多啊 (64TB),而且實際上不用管劃多大,他會自己長大:

Q: What are the minimum and maximum storage limits of an Amazon Aurora database?

The minimum storage is 10GB. Based on your database usage, your Amazon Aurora storage will automatically grow, up to 64 TB, in 10GB increments with no impact to database performance. There is no need to provision storage in advance.

AWS 的 General Purpose SSD (gp2) 可以看到 burst I/O 的 credit 數字了

AWS 宣佈把 gp2 的 I/O burst credit 數字給量化了:「New – Burst Balance Metric for EC2’s General Purpose SSD (gp2) Volumes」。

gp2 最小的也可以衝到 3000 IOPS,另外可以累積 5.4M credits:

Each volume can accumulate up to 5.4 million credits, and they can be spent at up to 3,000 per second per volume.

算了一下,1GB 的空間一個小時可以累積 10,800 IOPS,如果切 10GB 的系統碟,大約 50 個小時就會滿。如果 100GB 的話就是 5 個小時了,其實對於真的超級大量持續 I/O 的應用還是要考慮用 Provisioned IOPS SSD (io1)。

不過明顯的好處是可以建立 alarm,當機器的 burst I/O credit 快用完的時候可以叫一叫,這樣讓人可以評估下一步:

另外也可以藉由這個數字來評估是要加大空間以換取 IOPS,或是換到有保障的 Provisioned IOPS SSD (io1)。

Etsy 用 SSD 的故事

EtsyLaurie Denness 對於 Etsy 使用各種品牌 SSD 的情況給出了他的經歷:「SSDs: A gift and a curse」。


SSD firmware is buggy

可以看到當 SSD 配上 RAID controller 的時候,常常會需要找問題... (而且很難找)

Intel 的評價很不錯:

Okay, bad start, we’ve actually had no issues with Intel. This seems to be common across other companies we’ve spoken to.

OCZ 倒了,被 Toshiba 收購,而且 S.M.A.R.T. 資訊很差,很難預測什麼時候會掛掉 (有助於提前替換):

However, they had poor SMART info (none) so predicting failures was hard.

HP 是個大黑盒:

Unfortunately, HP have proprietary RAID controllers, and they don’t support SMART. Or rather, they refuse to talk to non-HP drives using off the shelf technology, they have their own methods.

Samsung 的評價不錯,C/P 值很高,而且有 S.M.A.R.T.:

Samsung saved the day and picked up from OCZ with a ludicrously cheap 960GB offering, the 840 EVO. A consumer drive, so very limited warranty, but for the price (~$400-500) you got great IOPS and they were reliable. They had better SMART info, and seemed to play nicely with our hardware.

不過 BB6Q 版的韌體搞爆了效能,雖然最後修好了:「Samsung Releases Firmware Update to Fix the SSD 840 EVO Read Performance Bug」。

LiteOn 則是掛在 GC 上 (RAID 裡同時掛掉兩顆以上):

The SSDs were having extended garbage collection periods, exacerbated by a smaller amount of SSDs with higher IO, in RAID6. This caused the controller to kick the drive out of the array… and unfortunately due to the write levelling across the drives, at least two of them were garbage collecting at the same time, destroying the array integrity.

不過後來 Dell 與 LiteOn 分別就 RAID controller 與 SSD 本身都跳下去修正,最後還是解決了:

Dell and LiteOn together identified and fixed weaknesses in their RAID controller, the backplane and the SSD firmware.

算是經驗分享,在 SSD 硬碟成熟的過程中間必經的道路 XD

AWS EBS 的改善...

AWS EBSAmazon Web Services 平台上的永久性儲存空間 (一般開起來的空間在 crash 後會消失),不過 EBS 的效能 (速度與 IOPS) 一直讓大家很頭痛,要硬撐 IOPS 的方式是透過四個 EBS volume,上面用 mdadm 跑 RAID0...

這幾個禮拜 AWS 丟出了兩個方案出來,提供不同的選擇...

首先是帶大容量 SSD 空間的 instance,參考「AWS 推出高速 I/O 的 EC2 instance」,這對於 cache 類的應用相當適合...

而昨天則是介紹了 Provisioned IOPS 與 EBS-Optimized instances:「Fast Forward - Provisioned IOPS for EBS Volumes」。

Provisioned IOPS 是保障 IOPS 的機制,每個 volume 最高可以到 1000 IOPS。除了每 GB 的價錢比較高以外,另外每個保障的 IOPS 要再收 USD$0.1/month。(本來 EBS 的 I/O 費用還是要計算)

標準的 EBS 約提供 100 IOPS,這個 Provisioned IOPS 架構讓使用者要在上面衝 IOPS 變得更容易...

EBS-Optimized instances 是把 EBS 使用的頻寬獨立出來,目前只支援 m1.large、m1.xlarge 以及 m2.4xlarge 這三種,其中 m1.large 跑到 500Mbps (理論值 62.5MB/sec),後面兩種可以上 1Gbps (125MB/sec),使得服務與 I/O 的頻寬不會互相干擾到...

AWS 推出高速 I/O 的 EC2 instance

早上就看到 AWS EC2 推出 hi1.4xlarge 的消息:「New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SSD-Backed Storage」(官方 blog)、「Expanding The Cloud – High Performance I/O Instances for Amazon EC2」(CTO Werner Vogels 的 blog)。


  • 60.5GB 記憶體
  • 10Gbps 網路
  • 1TB 的 SSD volume 兩個

前面兩個不會太意外,因為需要高速 I/O 的服務通常也都很需要用大量記憶體當作 cache 降低 I/O,也需要大量頻寬提供服務。用 SSD 也在預期的範圍內,不過提供的 SSD 空間居然這麼大...

當然,價位也不便宜,美東就要 USD$3.10/hour,冰島愛爾蘭則是 USD$3.41/hour (目前只有這兩區有提供)。如果以美東一個月 720hours 計算是 USD$2232,約台幣六萬六千多?