Backblaze 採購硬碟的策略

在「How Backblaze Buys Hard Drives」這篇裡面提到了 Backblaze 採購硬碟的策略,可以看到完全都是偏成本走向,所以裡面的策略一般個人用不太到,一般企業也不應該照抄,但拿來看看還蠻有趣的...

像是因為硬碟太多,所以硬碟的使用電量是他們在評估成本時蠻重要的一環,這點在一般的情境下不太會考慮到:

Power draw is a very important metric for us and the high speed enterprise drives are expensive in terms of power cost. We now total around 1.5 megawatts in power consumption in our centers, and I can tell you that every watt matters for reducing costs.

另外也提到了 SMR 硬碟的特性,在單位成本雖然有比較高的容量,但導致架構面需要配合 (cache),而也會有工程端的成本提昇,所以不是很愛:

SMR would give us a 10-15% capacity-to-dollar boost, but it also requires host-level management of sequential data writing. Additionally, the new archive type of drives require a flash-based caching layer. Both of these requirements would mean significant increases in engineering resources to support and thereby even more investment. So all-in-all, SMR isn’t cost-effective in our system.

成本面上,他們觀察到的現象是每季會降 5%~10%:

Ideally, I can achieve a 5-10% cost reduction per terabyte per quarter, which is a number based on historical price trends and our performance for the past 10 years.

另外提到了用 SAS controller 可以接多個 SATA 硬碟的事情 (雖然還是成本考量),但這塊也蠻有趣的:

Longer term, one thing we’re looking toward is phasing out SATA controller/port multiplier combo. This might be more technical than some of our readers want to go, but: SAS controllers are a more commonly used method in dense storage servers. Using SATA drives with SAS controllers can provide as much as a 2x improvement in system throughput vs SATA, which is important to me, even though serial ATA (SATA) port multipliers are slightly less expensive. When we started our Storage Pod construction, using SATA controller/port multiplier combo was a great way to keep costs down. But since then, the cost for using SAS controllers and backplanes has come down significantly.

企業內的文件搜尋系統 Amazon Kendra

AWS 推出了具有語意分析的能力,可以直接丟自然語言進去搜尋的 Amazon Kendra:「Announcing Amazon Kendra: Reinventing Enterprise Search with Machine Learning」。

之前 Google 有推出過 Google Search Appliance 也是做企業內資料的整合 (2016 年收掉了),但應該沒有到可以用自然語言搜尋?

Amazon Kendra 的費用不算便宜,Enterprise Edition 提供 150GB 的容量與 50 萬筆文件,然後提供大約 40k query/day,這樣要 USD$7/hr,一個月大約是 USD$5,040,不過對於企業來說應該是很有用...

另外有提到這邊 query 收費的部份是估算,會依照 query 問題的難易度而不同:

Actual queries per day will vary based on query complexity, which greatly varies from customer to customer. Less complex queries (e.g. “leave policy”) consume less resources to run, and more complex queries (e.g. “What’s the daily parking allowance in Seattle?”) consume more resources to run. The total number of queries you can run with your allocated resources will depend on your mix of queries. The max queries per day provided above is an estimate, assuming 80% less complex queries and 20% more complex queries.

這樣頗有趣的,感覺可以處理簡單的分析了?

Rasmus 的平價 VPS 測試

Rasmus Lerdorf 整理的資料 (就是生出 PHP 的那個 @rasmus),關於平價 VPS 的測試:「Low-Cost VPS Testing」。

因為是 PHP 的大頭,所以把編 PHP 當作是 benchmark 的項目之一也不算太意外。另外還是有比較常見的 dd 與 iperf3 測試資料可以看。

算是一個挖掘的點,之後也可以租幾台測試看看...

Amazon Detective:用 Machine Learning 分析可能的安全問題

也是這次 AWS re:Invent 發表的服務,透過 Machine Learning 分析可能的安全問題:「Introducing Amazon Detective」。

透過現有的各種 log 建立模型分析:

Amazon Detective can analyze trillions of events from multiple data sources such as Virtual Private Cloud (VPC) Flow Logs, AWS CloudTrail, and Amazon GuardDuty, and automatically creates a unified, interactive view of your resources, users, and the interactions between them over time.

依照 log 的量算錢的,然後 preview 階段不收費,所以有興趣的人可以開起來跑看看?

AWS Lambda 的 Provisioned Concurrency

AWS Lambda 推出了 Provisioned Concurrency,降低冷啟動所需要的時間以確保效率:「New – Provisioned Concurrency for Lambda Functions」。

看 benchmark 的資料就很清楚,可以避免冷啟動所產生的延遲:

這邊也可以看出來就算是冷啟動,大約也是多個一秒多。如果這個延遲是需要被處理的,就可以考慮用 Provisioned Concurrency 了。

價位上是依照保留的量:

You only pay for the amount of concurrency that you configure and for the period of time that you configure it. Pricing in US East (N. Virginia) is $0.015 per GB-hour for Provisioned Concurrency and $0.035 per GB-hour for Duration. The number of requests is charged at the same rate as normal functions. You can find more information in the Lambda pricing page.

不過如果量再更大,而且考慮成本,應該會考慮改回傳統的架構,用多台 EC2 instance 跑...

AWS Fargate 推出 Spot

相較於 Amazon EC2 有 Spot Instance (可以利用 Spot Instance 的競價機制省下很多費用),這次 AWS re:InventFargate 也推出了對應的產品線:「AWS Fargate Spot Now Generally Available」。

跟 EC2 的相同,你在上面跑的應用程式必須可以接受隨時中斷服務 (i.e. 必須是 crash-safe),常見的情境是 worker 類的程式。

價錢上大約在三折 (寫這篇時 us-east-1 目前的價錢),考慮到啟動的速度比 EC2 快很多,這樣好像是個可以考慮的方案...

今年 AWS Workshop 的資料

在「Guest Post: AWS Workshop Links for AWS re:Invent 2019 (and more), by Jennine Townsend」這邊看到整理出來的資料,雖然沒有去今年的 AWS re:Invent 2019 參與實際操作,但還可以從教材裡面翻到一些有用的東西...

可以看到蠻多主題都單獨註冊了 .com 的網域 (robomakerworkshops.comawssecworkshops.comecsworkshop.comwww.appmeshworkshop.com、...),而且有些是放到 CloudFront 上,但也有些是丟到 GitHub Pages 上...

另外也有不少課程沒有申請網域,而是找個地方 (大多就是 GitHub) 丟著而已,看起來是 AWS 官方給了免費的資源,然後看課程的主辦方要不要用...

Amazon API Gateway 又在搞奇怪的東西了...

Amazon API Gateway 宣佈一個新的產品,提供 HTTP APIs 管理 RESTful APIs (???):「Amazon API Gateway Offers Faster, Cheaper, Simpler APIs Using HTTP APIs (Preview)」。

官方是這樣描述 HTTP APIs 的:

Use HTTP APIs to build high performance RESTful APIs that require API proxy functionality without API management features. HTTP APIs are optimized for serverless applications and HTTP backends, and offer up to 70% cost savings compared to REST APIs.

你已經有了 RESTful APIs,然後跑去接個沒有 API management features 的 API Gateway...?

然後翻了一下之前 API Gateway 的豐功偉業,本來打了一大堆,但還是留點口德好了... 看起來 API Gateway 團隊裡老大的後台很硬啊,搞成這樣都沒被幹掉?

話說回來,去年 ALB 宣佈支援 AWS Lambda,該不會是 API Gateway 實在太爛,所以 Serverless 的大方向逼 ALB 支援的啊?

Amazon RDS 推出了 Connection Pool 的產品

Amazon RDS 推出了 Connection Pool 的產品,叫做 Amazon RDS Proxy:「Introducing Amazon RDS Proxy (Preview)」。

目前支援 MySQL (包括了傳統的與 Aurora 版本的):

Amazon RDS Proxy supports Amazon RDS for MySQL and Amazon Aurora with MySQL compatibility, with support for additional RDS database engines coming soon.

定價策略看起來是依照後端資料庫的 vCPU 計算:

Pricing is simple and predictable: you pay per vCPU of the database instance for which the proxy is enabled.

翻了一下價錢頁是 USD$0.015/vCPU (用 us-east-1 的資料),而如果是 t2 系列的機器,最低是以 2 vCPUs 計算,不是照使用比例算:

RDS Proxy pricing correlates to the number of vCPUs of the database instance for which it is enabled, with a minimum charge for 2 vCPUs.

這樣一個 vCPU 一個月大約要 USD$21.6,算起來頗貴的... 如果 SLA 允許的話,用基本的方式 failover 也許就 ok 了...

如果 SLA 真的要追求到這麼高的話,可以在這些區域測試:

Amazon RDS Proxy is available in preview for RDS MySQL and Aurora MySQL in US East (N. Virginia), US East (Ohio), US West (Oregon), EU West (Ireland), and Asia Pacific (Tokyo) regions. Support for RDS PostgreSQL and Aurora PostgreSQL is coming soon.

CloudFront 的 BBR 效能提昇

這是在找一些 TCP congestion algorithm 相關的資訊時發現的,看起來 Amazon CloudFront 導入 BBR 一陣子了:「TCP BBR Congestion Control with Amazon CloudFront」。

不過 BBR 被研究的愈來愈多,大家開始發現這個演算法的霸道,跟其他的 TCP congestion algorithm 並不太能和平共存,但這就跟軍事武器一樣,隔壁升級了你就得跟著升級,抱怨沒有用,只會被消滅...