Wasabi 與 Storj DCS

WasabiStorj 是在看到「Will Cloudflare R2 Win Customers from Amazon S3?」這篇文章時翻到了三個 Cloud Storage Provider,文章本身我倒是沒什麼吸收...。

第一個是 BackblazeB2,這個產品平常的曝光度就還算夠。

另外是 Wasabi 的部份,其中一個賣點是免費的頻寬,但其實限制意外的多。首先是各地區的價錢:

我找了一下到底是什麼地區,目前只有看到「Wasabi Technologies Inc Status」這邊有編號 (US-East-1、US-East-2、US-Central-1、US-West-1、EU-Central-1、AP-Northeast-1),但也沒找到地區... US 的都在美國沒問題,AP-Northeast-1 應該是日本,但 EU-Central-1 是哪裡就找不到了。

另外是 pay-as-you-go 的方案最小是 1TB,如果是歐美區的話是 US$5.99:

For customers using the Wasabi pay-as-you-go pricing model, Wasabi has a minimum monthly charge associated with 1 TB of active storage. If you store less than 1 TB of active storage in your account, you will still be charged for 1 TB of storage based on the pricing associated with the storage region you are using.

然後也有 90 天的最短計價時間:

Wasabi has a minimum storage duration policy that means if stored objects are deleted before they have been stored with Wasabi for a certain number of days (90 days when using the Wasabi pay-go pricing model), a Timed Deleted Storage charge equal to the storage charge for the remaining days will apply.

另外 Wasabi's free egress policy 這邊也可以看出來他們的設計就是拿來當 storage 用,然後前面需要擋 CDN 之類的服務。

最後一個是 Storj 的 DCS,US$4/TB/month 的空間費用,與 US$7/TB 的流量費用感覺還算便宜?他的做法是把檔案拆成 80 份,然後取 29 份就可以算回來:

How many Nodes are files stored on?

80. We split each file into 80 different encrypted pieces, and each piece is stored on a different Node.

When you retrieve an object, only 29 of its 80 pieces are needed to reconstitute that object. With no central point of failure, your data is always quickly available, all over the world.

這部份是則是透過 Reed-Solomon error correction 實做:

Automate file repair and know that Reed-Solomon erasure coding enables the highest levels of durability for all files uploaded to Storj DCS.

好一陣子沒看到 Reed-Solomon 了,沒想到在這邊看到... 先不管技術的部份,看起來 Storj DCS 的價錢可以玩看看。

Cloudflare R2 Storage 的插曲...

Hacker News 首頁上看到「Cloudflare's Disruption (stratechery.com)」這篇,文章「Cloudflare’s Disruption」這篇其實還好,主要就是分析一下 Cloudflare R2 Storage 在下的棋,真的讓我想寫的是反而是 Hacker News 上的討論...

首先是提到了 S3 -> R2 -> Q1 -> P0 這個:

ksec 36 minutes ago | unvote [–]

^gt; The service will be called R2 — “one less than S3,” quipped Cloudflare CEO Matthew Prince in an interview with Protocol ahead of Cloudflare’s announcement

Oh I never thought of that. So the next one is Q1 and final one would be P0.

另外下面有也提到 IBMHAL

piaste 33 minutes ago | unvote [–]

And it is likely inspired by the old joke that 2001: A Space Odyssey's HAL was one less than "IBM".

下一個 Q1 是明年了,來看看 2022Q1 會不會有 P0 issue XDDD

Cloudflare 推出 Cloudflare R2 Storage,相容於 S3 API,但沒有傳輸費用

Cloudflare 宣佈了 Cloudflare R2 Storage,相容於 S3 API,但是沒有傳輸費用:「Announcing Cloudflare R2 Storage: Rapid and Reliable Object Storage, minus the egress fees」,Hacker News 上的「Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees (cloudflare.com)」可以看一下討論,裡面有負責 R2 的 PM (帳號是 greg-m) 回答一些東西。

R2 的第一個特點就是剛剛提到的傳輸費用:一般的雲端都是傳進去不用錢,但傳出來會很貴,而 R2 其中一個主打的點就是傳出來不用錢:

R2 builds on Cloudflare’s commitment to the Bandwidth Alliance, providing zero-cost egress for stored objects — no matter your request rate. Egress bandwidth is often the largest charge for developers utilizing object storage and is also the hardest charge to predict. Eliminating it is a huge win for open-access to data stored in the cloud.

另外 storage cost 也算低,S3 目前的費用是 US$0.023/GB/month (拿 us-east-1 相比),而 R2 目前的定價是 US$0.015/GB/month:

That doesn’t mean we are shifting bandwidth costs elsewhere. Cloudflare R2 will be priced at $0.015 per GB of data stored per month — significantly cheaper than major incumbent providers.

在 durability 的部份,與 S3 都是一年 11 個 9:

The core of what makes Object Storage great is reliability — we designed R2 for data durability and resilience at its core. R2 will provide 99.999999999% (eleven 9’s) of annual durability, which describes the likelihood of data loss.

目前還沒有公開,算是先對市場放話:

R2 is currently under development — you can sign up here to join the waitlist for access.

有幾個點還蠻有趣的,第一個是 Cloudflare 自己在推的 Bandwidth Alliance 裡有不少 VPS 跟 Cloudflare 之間的流量是不計頻寬費用的,所以等於是 VPS 到 R2 不計費,而 R2 到 VPS 也不計費,但要注意 VPS 自己也都有在推 object storage。

像是 Vultr 的 US$5 方案包括了 250GB 的空間與 1TB 的頻寬,扣掉頻寬的部份 (可以透過 Cloudflare 處理),相當於是 US$0.02/GB。

Linode 也類似,US$5 的方案包括了 250GB 的空間與 500GB 的頻寬,算出來也是 US$0.02/GB。

Backblaze 也有類似的產品 B2,US$0.005/GB/month 的儲存費用以及 $0.01/GB 的傳輸費用,但頻寬的部份也可以透過 Cloudflare 處理。

這個產品出來以後可以再看看如何,但看起來是蠻有趣的。對目前的雲端商應該還好 (因為資料進 R2 還是有費用),但對這些 VPS 來說應該是有蠻大的衝擊...

Cloudflare Images

Cloudflare Images 開放付費使用了:「Cloudflare Images Now Available to Everyone」。

檔案傳到 Cloudflare 上面,然後另外收處理費用:

You pay $5/month for every 100,000 stored images and $1 per 100,000 delivered images. There are no additional resizing, compute or egress costs.

檔案大小的限制是 10MB,所以 $5/month 的 storage 最多可以提供 1TB 的空間,$0.005/GB 算是很漂亮的數字,如果是小圖的話就會比較虧一些?看起來丟大圖會開心一點...

Cloudflare Images offers multiple ways to upload your images. We accept all the common file formats including JPEG, GIF and WEBP. Each image uploaded to Images can be up to 10 MB.

然後支援的檔案格式有常見的 GIFJPEGPNG 以及 WebP

When a client requests an image, Cloudflare Images will pick the optimal format between WebP, PNG, JPEG and GIF.

另外有計畫要支援 AVIF

We’re just getting started with Cloudflare Images. Here are some of the features we plan to support soon:

AVIF support for even smaller file sizes and faster load times.

沒提到 durability,不知道會有多少...

Cloudflare 開始在正式環境用 ARM server 了

在「Designing Edge Servers with Arm CPUs to Deliver 57% More Performance Per Watt」這邊 Cloudflare 提到了他們在正式環境用 ARM 架構了:

Our first Arm CPU was deployed in production earlier this month — July 2021.

記得測了很多年,其中遇到測試到一半看起來還不錯,但原廠商決定不繼續做的,直到後來又有廠商投入,到現在總算是有比較成熟的產品可以用。

隔壁棚 AWS 上的 ARM 伺服器用起來也是香到不行,還沒有用過的可以試看看,至少我這台 blog & wiki 也都是跑在上面。

另外文章裡有提到目前 x86 的效能,新一代的 AMD 大概只比前一代多了 39% 的每瓦效能,但如果是把 ARM 拿進來比的話會到 57%:

Our most recently deployed generation of edge servers, Gen X, used AMD Rome CPUs. Compared with that, the newest Arm based CPUs process an incredible 57% more Internet requests per watt. While AMD has a sequel, Milan (and which Cloudflare will also be deploying), it doesn’t achieve the same degree of energy efficiency that the Arm processor does — managing only 39% more requests per watt than Rome CPUs in our existing fleet.

開始推上 production 後應該會愈換愈快,而且代表 Cloudflare 也會開始針對 ARM 平台最佳化。

Cloudflare 在巴西的佈點

看到 Cloudflare 在講他們打算在巴西佈 25 個點:「Expanding Cloudflare to 25+ Cities in Brazil」,目前可以看出來是八個點:

比較了領土的大小,跟美國扣掉阿拉斯加差不多等級:

人口的話美國是 328M 左右 (阿拉斯加不到 1M,不太影響感覺),巴西 215M,依照 Cloudflare 在美國目前有 39 個 PoP 來說,的確是可以拉上去,不過看起來應該是因為有大的 ISP 合作的關係:

Today, we are excited to announce an expansion we’ve been working on behind the scenes for the last two years: a 25+ city partnership with one of the largest ISPs in Brazil.

不過沒提到是哪個 ISP,之後看看有沒有消息...

QUIC 成為標準,從 RFC 8999 到 RFC 9002

前幾天的新聞了,這兩天 FastlyCloudflare 也都發文章出來了,QUIC 成為標準:「QUIC is now RFC 9000」、「QUIC Version 1 is live on Cloudflare」。

主要是這兩家都發稿宣傳他們的平台都支援 QUIC 了,接下來可以等一些測試報告,看看在 web 這種已經有不少複雜的 workaround 機制下,TCP BBR 環境的 HTTP/2 跟 QUIC 環境會有多少差異... 記得 QUIC 也是 BBR-based 的演算法。

在 QUIC 下的 https 協定會走 443/udp,如果防火牆是預設阻擋所有連線,然後逐條開放的話,需要另外開這組設定。

另外就是等 nginx 支援了,在「NGINX QUIC Preview」這邊有些資料,然後「">nginx-quic: log」裡面可以看到東西,裡面不少 commit 只是跟 nginx 本家同步而已,不過還是可以看到一些跟 QUIC 有關的...

試用 Cloudflare 的 Argo Tunnel

Cloudflare 宣佈讓大家免費使用 Argo Tunnel 了,也順便改名為 Cloudflare Tunnel 了:「A Boring Announcement: Free Tunnels for Everyone」。

Starting today, we’re excited to announce that any organization can use the secure, outbound-only connection feature of the product at no cost. You can still add the paid Argo Smart Routing feature to accelerate traffic.

As part of that change (and to reduce confusion), we’re also renaming the product to Cloudflare Tunnel. To get started, sign up today.

Cloudflare Tunnel 的功能就像 ngrok,在用戶端的機器上跑一隻 agent 連到 Cloudflare 或是 ngrok 的伺服器,這樣外部連到 Cloudflare 或是 ngrok 的伺服器後就可以透過這組預先建好的連線連上本機的服務了,常見的應用當然就是 HTTP(S) server。

本來是付費功能,一般使用者應該也不會需要這個功能,這次把這個功能免費丟出來的用意不知道是什麼...

不過既然都免費了,還是花了點時間測了一下,可以發現 ngrok 的設定比較簡單,Cloudflare 的 cloudflared 設定起來複雜不少,不過文件還算清楚,照著設就好。

Anyway,有些事情有了 Cloudflare Tunnel 就更方便了,像是有些超小型的 VPS 是共用 IPv4 address 而且沒有 IPv6 address 的,可以透過 cloudflared 反向打進去提供服務,同樣的,在 NAT 後面的機器也可以透過這個方法很簡單的打通。

順便說一下,現在的 blog.gslin.org 就是跑在 cloudflared 上面了,官方提供的 ARM64 binary 跑在 EC2t4g 上面目前看起來沒有什麼問題,而且比起本來 nginx 都是抓到 Cloudflare 本身的 IP,現在加上這兩行後反而就可以抓到真的使用者 IP address 了:

    set_real_ip_from 127.0.0.1;
    real_ip_header X-Forwarded-For;

跑一陣子看看效果如何...

利用 Cloudflare Workers 繞過 Cloudflare 自家的阻擋機制

Hacker News 首頁上看到「How to bypass Cloudflare bot protection (jychp.medium.com)」這則,裡面的文章是「How to bypass CloudFlare bot protection ?」這篇,利用 Cloudflare Workers 繞過 Cloudflare 自家的 CAPTCHA 機制。

這個漏洞有先被送給 Cloudflare,但被認為不是問題,所以作者就決定公開:

Several months ago I submitted what appeared to be a security flaw to CloudFalre’s bugbounty program. According to them, this is not a problem, it’s up to you to make up your own mind.

技術上就是透過 Cloudflare Workers 當作 proxy server,只是看起來 Cloudflare 對自家 IP 有特別處理,在設定妥當後,用 Cloudflare Workers 的 IP address 去連 Cloudflare 的站台,幾乎不會觸發 Cloudflare 的阻擋機制。

不過 free tier 還是有限制,主要就是數量:

The first 100,000 requests each day are free and paid plans start at just $5/10 million requests, making Workers as much as ten-times less expensive than other serverless platforms.

作者也有提到這點:

So let’s enjoy the 100 000 request/day for your free Cloudflare account and go scrape the world !

但這是個有趣的方法,加上信用卡盜刷之類的方式,這整包看起來就很有威力...

Cloudflare 再次嘗試 ARM 伺服器

2018 年的時候寫過一篇 Cloudflare 在嘗試 ARM 伺服器的進展:「Cloudflare 用 ARM 當伺服器的進展...」,後來就沒有太多公開的消息,直到這幾天看到「ARMs Race: Ampere Altra takes on the AWS Graviton2」才看到原因:

By the time we completed porting our software stack to be compatible with ARM, Qualcomm decided to exit the server business.

所以是都測差不多,也都把 Cloudflare 自家的軟體搬上去了,但 Qualcomm 也決定收手,沒機器可以用...

這次再次踏入 ARM 領域讓人想到前陣子 AppleM1,讓大家看到 ARM 踏入桌機與筆電領域可以是什麼樣貌...

這次 Cloudflare 選擇了 Ampere Altra,這是基於 Neoverse N1 的平台,而這個平台的另外一個知名公司就是 AWSGraviton2,所以就拿來比較:

可以看到 Ampere Altra 的核心數多了 25% (64 vs. 80),運作頻率多了 20% (2.5Ghz vs. 3.0Ghz)。測試的結果也都有高有低,落在 10%~40% 都有。

不過其中比較特別的是 Brotli - 9 的測試特別差 (而且是 8 與 10 都正常的情況下):

依照 Cloudflare 的說法,他們其實不會用到 Brotli - 7 以及更高的等級,不過畢竟有測出來,還是花了時間找一下根本原因:

Although we do not use Brotli level 7 and above when performing dynamic compression, we decided to investigate further.

反追問題後發現跟 Page Faults 以及 Pipeline Backend Stalls 有關,不過是可以改寫避開,在避開後可以達到跟 Graviton2 類似的水準:

By analyzing our dataset further, we found the common underlying cause appeared to be the high number of page faults incurred at level 9. Ampere has demonstrated that by increasing the page size from 4K to 64K bytes, we can alleviate the bottleneck and bring the Ampere Altra at parity with the AWS Graviton2. We plan to experiment with large page sizes in the future as we continue to evaluate Altra.

但目前看起來應該都還算正向,看起來供貨如果穩定的話,應該有機會換過去?畢竟 ARM 平台可以省下來的電力太多了,現在因為 M1 對 ARM 的公關效果太驚人的關係,解釋起來會更輕鬆...