Amazon S3 變成 Strong Consistency 背後的改善方式

看到 Hacker News 上的討論「Diving Deep on S3 Consistency (」才想到該整理一下,原文的「Diving Deep on S3 Consistency」是 Amazon 的 CTO Werner Vogels 花了一些篇幅描述 Amazon S3 怎麼把 Eventually Consistent 變成 Strongly Consistent,當初 Amazon S3 公告時我也有寫一篇文章提到:「Amazon S3 現在變成 Strong Read-After-Write Consistency 啦...」。

Amazon S3 之所以會是 Eventually Consisient 是因為 Metadata Subsystem 的 cache 設計:

Per-object metadata is stored within a discrete S3 subsystem. This system is on the data path for GET, PUT, and DELETE requests, and is responsible for handling LIST and HEAD requests. At the core of this system is a persistence tier that stores metadata. Our persistence tier uses a caching technology that is designed to be highly resilient. S3 requests should still succeed even if infrastructure supporting the cache becomes impaired. This meant that, on rare occasions, writes might flow through one part of cache infrastructure while reads end up querying another. This was the primary source of S3’s eventual consistency.

如果要解決 Eventually Consistent,最直接的想法是拔掉 cache,但這樣對效能的影響太大,所以得在要保留 cache 的情況下設計,所以就想到用其他管道確保 cache 裡的資料狀態是正確的:

One early consideration for delivering strong consistency was to bypass our caching infrastructure and send requests directly to the persistence layer. But this wouldn’t meet our bar for no tradeoffs on performance. We needed to keep the cache. To keep values properly synchronized across cores, CPUs implement cache coherence protocols. And that’s what we needed here: a cache coherence protocol for our metadata caches that allowed strong consistency for all requests.

而接下來是設計一連串的邏輯確保每個 S3 object 的操作都有 serializability:

We had introduced new replication logic into our persistence tier that acts as a building block for our at-least-once event notification delivery system and our Replication Time Control feature. This new replication logic allows us to reason about the “order of operations” per-object in S3. This is the core piece of our cache coherency protocol.

後面又要確保這個 cache coherence 的 HA,最後要能夠驗證實做上的正確性,花的力氣比實做協定本身還多:

These verification techniques were a lot of work. They were more work, in fact, than the actual implementation itself. But we put this rigor into the design and implementation of S3’s strong consistency because that is what our customers need.

Amazon S3 算是 AWS 當初推出來的招牌,當時的 Amazon S3 底層的論文「Amazon's Dynamo」劇烈影響了後來整個產業 (雖然論文裡面是拿 Amazon 的購物車說明),這次的補充算是更新了原來論文的技術,告訴大家本來的 Eventually Consistent 是可以再拉到 Strongly Consistent。

Martin Fowler 在 2015 年寫的 MonolithFirst,以及 Microservice 的問題

Hacker News Daily 上看到「MonolithFirst」這篇,是 Martin Fowler 在 2015 年寫的文章,對應在 Hacker News 上的討論「Monolith First (2015) (」也頗有趣的,可以一起翻。

tl;dr:他的文章就是在講新專案用 monolith,不要去碰 microservice。

文章開頭提到了就他觀察到的情況:第一點是,幾乎所有成功的 microservice 案例都是從 monolith 起頭,再轉到 microservice 環境;第二點是,幾乎所有一開始用 microservice 的案例,在後來都遇到嚴重的問題:

As I hear stories about teams using a microservices architecture, I've noticed a common pattern.

  1. Almost all the successful microservice stories have started with a monolith that got too big and was broken up
  2. Almost all the cases where I've heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble.

This pattern has led many of my colleagues to argue that you shouldn't start a new project with microservices, even if you're sure your application will be big enough to make it worthwhile. .


第一點可能的原因是 Yagni (You Aren't Gonna Need It),在試驗市場時 PoC 或是 MVP 的商業邏輯反而不會那麼複雜,快速用 monolith 開發驗證比起擁抱 microservice 來的重要太多,而且可以快速修改。

第二點分成兩個現象:第一個現象是,即使是對產品的商業領域很有經驗的資深架構師,也很難一開始就切出正確的 BoundedContext;第二個現象是,在 microservice 裡,改架構的難度比起 monolith 高非常多 (i.e. 跨 boundary 的 refactoring)。這兩個現象加在一起就會造成一開始導入 microservice 的專案失敗。

文章接著想要提出一些建議,在一開始使用 monolith 可以注意的方向,這些注意的事項可能可以讓之後轉成 microservice 變得比較輕鬆。

第一種方式是在 monolith 架構下注意 API boundary 與 data 的儲存方式,當想要切換到 microservice 的時候就有機會比較簡單。

第二種是更常見的方式,一開始先是 monolith 架構,然後把 boundary 好切割的拆成 microservice,所以在過度期會是一組不那麼大的 sub-monolith 架構,然後週邊圍繞著很多 microservice。這個做久了就有機會轉成全部都是 microservice。


第四種方式是一開始的時候不用 monolith,而是幾個很大包的 service (所以一開始不太能叫 microservice),當商業模式成熟穩定後,切出更細緻的 boundary 的時候再拆成 microservice。

把這篇文章拿去搜尋 Hacker News 上的討論,可以看到除了 2021 年的這次討論外,在 2017 年的時候就有一批討論也蠻有趣的,可以看到有些不同的風向:「Monolith First (2015) (」。

當然也未必要信 Martin Fowler 的看法,軟體工程這塊還是有很多不同的流派...

ALB 支援 Sticky Session


ALB 支援使用 cookie 實現 sticky session 功能:「Application Load Balancer now supports Application Cookie Stickiness」。

使用者的 session 通常會使用 cookie 記錄,而如果有多台 server 提供服務時,session 裡的資訊就需要找一個 shared session storage 放,以確保使用者在連到不同的 server 時都還是可以讀到對應的 session,比較傳統的方案就是直接把 session 塞進資料庫,後來發展出 memcached 或是 Redis 可以用。

但有些買來的軟體並沒有考慮到這點 (常常都是內部系統),導致前面放 load balancer 時,必須想個辦法記錄使用者使用後端的哪台機器,這樣就可以在後端不支援 shared session storage 的情況下,還是可以讓應用正常運作。

透過 cookie 實做的 sticky session 算是蠻常見的作法,只是以為早就有了...

Backblaze 在 2020 年對機械硬碟的回顧

前幾天 Backblaze 放了 2020 年的回顧資料出來:「Backblaze Hard Drive Stats for 2020」。

整體的 AFR (Annualized Failure Rate) 在 0.93% 左右,而如果照品牌拆開,HGST 的數字依然是最漂亮的 (雖然他現在是 WD 的品牌),大約在 0.36% 左右 (111/(1083774+4663049+372000+820272+275779+3968475)),Toshiba 次之,大約低了平均值一些落在 0.89%,而 Seagate 光是看就就知道會超過 1%...

官方有提到,低於 250,000 drive days 以下的數據僅供參考,因為資料量太少,在統計上無法提供結論:

For drives which have less than 250,000 drive days, any conclusions about drive failure rates are not justified. There is not enough data over the year-long period to reach any conclusions. We present the models with less than 250,000 drive days for completeness only.

然後 WD 本家的硬碟回到戰線了,記得之前基本上算是被唾棄 XDDD

另外一張表則是講到這三年的情況,可以看出來 2020 年的 AFR 數字降了不少,裡面也解釋了為什麼 (看起來就是活下來的穩下來了...):

The answer: It was a group effort. To start, the older drives: 4TB, 6TB, 8TB, and 10TB drives as a group were significantly better in 2020, decreasing from a 1.35% AFR in 2019 to a 0.96% AFR in 2020. At the other end of the size spectrum, we added over 30,000 larger drives: 14TB, 16TB, and 18TB, which as a group recorded an AFR of 0.89% for 2020. Finally, the 12TB drives as a group had a 2020 AFR of 0.98%. In other words, whether a drive was old or new, or big or small, they performed well in our environment in 2020.

gp3 (Amazon EBS) 的 latency

昨天把手上所有的 Amazon EBSgp2 換到 gp3 了:「Amazon EBS 的 gp3 可以用在開機磁碟了」,今天早上來看一下狀態,整體看起來是還 OK,不過有些地方值得注意的,像是標題寫到的 latency。

我抓了跑 GitLab 的機器來看,可以很明顯看到讀寫的 latency 都變高了:

AWS 又有提到這些數字資料有經過轉換,看起來是 gp2gp3 的數字意義本來就不一樣,所以他必須想辦法轉換,所以也有可能是因為這個轉換導致的?

This graph has had transformations applied to it and will differ from what is natively found in CloudWatch. Due to this some functionality is reduced.


Amazon EBS 的 gp3 可以用在開機磁碟了

可以先參考「Amazon EBS 推出了 gp3」這篇,但剛出來的時候大家都有發現無論是透過 web console 還是透過 awscli,boot disk 都沒辦法改成 gp3,可是在官方的文件上又說可以用 gp3,所以就有人在 AWS 的 forum 上發問了:「EBS GP3 Boot Volume Issues」。

直到剛剛發現已經可以改成 gp3 了... 一個一個手動改當然也是 OK,但對於有一卡車 EBS 要換的人來說鐵定得弄指令來換,這邊搭配了 jq 一起改:

aws ec2 describe-volumes | jq '.Volumes[] | select(.VolumeType == "gp2") | .VolumeId' | xargs -n1 -P4 env aws ec2 modify-volume --volume-type gp3 --volume-id

這邊是把 gp2 都改成 gp3,沒有考慮到空間大小的問題 (因為超過 1TB 時 gp2 給的 IOPS 會比較多),另外 -P4 是平行四個 process 跑,改起來會快一些...

Amazon EC2 的新機種:R5b、D3 (D3en)、C6gn、M5zn、G4ad

Amazon EC2 除了昨天放出 Mac mini 消息打頭陣以外,其他機種的更新消息也陸陸續續公佈了:

比較有趣的 (對我而言),第一個是 ARM 架構的機器也推出 100Gbps 的 n 版本 c6gn,看起來很適合跑大流量的東西,馬上想到的就是自架的 memcached

另外是 m5zn,使用高頻率的 Intel Xeon,主打需要單核效率的程式,不過這是掛在 m 系列下,而不是 c 系列...

再來是使用 AMD GPU 的 g4ad,官方宣稱跟 NVIDIAg4dn 比起來,將會有 45% 的 C/P 值提昇,是個蘇媽跟老黃的對決:

However, when compared to G4dn the new G4ad instances enable up to 45% better price performance for graphics-intensive workloads, including the aforementioned game streaming, remote graphics workstations, and rendering scenarios. Compared to an equally-sized G4dn instance, G4ad instances offer up to 40% improvement in performance.

看起來 ARM 的消息沒有想像中的多...

Amazon EBS 的 io2 給了不少新消息...

Amazon EBS 的另外一個新推出的東西,是針對 io2 的改善:

前面兩則消息可以一起看,主要是推出了 EBS Block Express,有著效能上的提昇:

Built on our new EBS Block Express architecture that takes advantage of some advanced communication protocols implemented as part of the AWS Nitro System, the volumes will give you up to 256K IOPS & 4000 MBps of throughput and a maximum volume size of 64 TiB, all with sub-millisecond, low-variance I/O latency. Throughput scales proportionally at 0.256 MB/second per provisioned IOPS, up to a maximum of 4000 MBps per volume. You can provision 1000 IOPS per GiB of storage, twice as many as before. The increased volume size & higher throughput means that you will no longer need to stripe multiple EBS volumes together, reducing complexity and management overhead.

目前因為是 preview 階段,想要用的人需要申請測試。要注意目前支援的區域有限 (不像這次推出 gp3 的時候就是全區),而且需要搭配 r5b 的機器:

The preview is currently available in the US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), and Europe (Frankfurt) Regions. During the preview, we support the use of R5b instances, with support for other Nitro-powered instances in the works.

第三則消息則是在講 io2 的 IOPS 的折扣,針對購買 32K IOPS 以上的部份會有 30% 折扣:

Now, with the new tiered pricing structure, the first 32,000 IOPS provisioned on a volume are charged at the current base rate ($0.065 per provisioned IOPS-mo) and the second tier between 32,001 and 64,000 is charged at a 30% lower rate ($0.046 per provisioned IOPS-mo).

針對前面提到的 preview 版本 (EBS Block Express),因為可以超過 64K IOPS,這個部份的價錢會更低,再疊一次 30% 的折扣:

Furthermore, for customers who have even higher performance requirement than currently supported by a single io2 volume today, we are previewing io2 volumes that run on EBS Block Express, the next generation of our block storage architecture. io2 Block Express volumes can be provisioned to deliver peak IOPS of 256,000. For these volume, any IOPS provisioned over 64,000 IOPS will be charged at a further 30% lower rate than the second tier ($0.032 per provisioned IOP-mo for IOPS over 64,000). This lowers the effective rate to $0.038 per provisioned IOPS on a volume provisioned with 256,000 IOPS.

算是要衝效能的人用的,目前平常應該還是會用 gp2 或是 gp3 的 SSD...

Amazon EBS 推出了 gp3

今年的 AWS re:Invent 又開始了,不過因為疫情的關係,這次是線上為主... 這邊先來整理一下 Amazon EBS 相關的更新。

首先是推出了新的 gp3 類型,也是 SSD 類:「New – Amazon EBS gp3 Volume Lets You Provision Performance Apart From Capacity」。

每 GB 單位成本比 gp2 低 20%:

Today I would like to tell you about gp3, a new type of SSD EBS volume that lets you provision performance independent of storage capacity, and offers a 20% lower price than existing gp2 volume types.

然後直接給你 3000 IOPS 與 125MB/sec,有需要更高的話可以「加購」:

gp3 is designed to provide predictable 3,000 IOPS baseline performance and 125 MiB/s regardless of volume size. It is ideal for applications that require high performance at a low cost such as MySQL, Cassandra, virtual desktops and Hadoop analytics. Customers looking for higher performance can scale up to 16,000 IOPS and 1,000 MiB/s for an additional fee. The top performance of gp3 is 4 times faster than max throughput of gp2 volumes.

但照「Amazon EBS volume types」這邊的列表可以看到,要注意 gp2 可以 burst 的 throughput (250MB/sec) 比 gp3 的 baseline (125MB/sec) 高。

也因為這樣,可以把一些 random access 比較多的 /data 這類的 EBS 換過去,但如果是要大量 sequential access 的也許就不適合了。

IOPS 的部份,1TB 以下的 gp2 換過去應該是沒什麼太大問題,因為在 gp2 的時候是 1GB 給 3IOPS,所以 1TB 以下的 gp2 都低於 3000IOPS。

轉移的部份可以在 AWS 的 console 上直接 migrate 到 gp3

If you’re currently using gp2, you can easily migrate your EBS volumes to gp3 using Amazon EBS Elastic Volumes, an existing feature of Amazon EBS. Elastic Volumes allows you to modify the volume type, IOPS, and throughput of your existing EBS volumes without interrupting your Amazon EC2 instances.


但照「Amazon EBS volume types」這邊的列表,gp3 可以是開機硬碟,但是改不過去啊 XDDD


不知道哪邊搞錯了,過幾天看看吧 XDDD

AWS 推出了 Amazon S3 Storage Lens 可以看 S3 使用的概況

AWS 推出了 Amazon S3 Storage Lens,可以看 S3 使用的概況:「Introducing Amazon S3 Storage Lens – Organization-wide Visibility Into Object Storage」。

要使用者個功能需要授權 Amazon S3 Storage Lens 一些權限,照著說明去 IAM 開就可以了,開好後要等他一陣子,他需要去分析記錄才能產出 dashboard。

有免費版與付費版可以用,付費版的部份目前看到都是「$0.20 per million objects monitored per month」,但沒把所有的區域都翻完,所以不確定。

我自己看了一下免費版提供的預設 dashboard,就已經給出不少好用的資訊了,像是 30 天內的物件數與空間使用率變化,可以抓到一些成長數量的感覺。