Category: CDN

用 Akamai 提供的 akahelp 分析 DNS Resolver 的資訊

整理資料的時候看到以前就看到的資訊，Akamai 有提供工具，可以看 DNS resolver 的資訊：「Introducing a New whoami Tool for DNS Resolver Information」。

這拿來分析 168.95.1.1 或是 8.8.8.8 這些服務還蠻好用的，這些對外雖然有一個 IP address 在服務，但後面是一整個 cluster，所以可以利用 Akamai 的這個工具來看分析。

像是 8.8.8.8 會給接近的 EDNS Client Subnet (ECS) 資訊 (ip 的部份看起來是隨便給一個)：

$ dig whoami.ds.akahelp.net txt @8.8.8.8

[...]

;; ANSWER SECTION:
whoami.ds.akahelp.net.  20      IN      TXT     "ns" "172.217.43.194"
whoami.ds.akahelp.net.  20      IN      TXT     "ecs" "111.250.35.0/24/24"
whoami.ds.akahelp.net.  20      IN      TXT     "ip" "111.250.35.149"

而 1.1.1.1 會給假的 ECS 資訊：

$ dig whoami.ds.akahelp.net txt @1.1.1.1

[...]

;; ANSWER SECTION:
whoami.ds.akahelp.net.  20      IN      TXT     "ns" "2400:cb00:80:1024::a29e:f134"
whoami.ds.akahelp.net.  20      IN      TXT     "ip" "2400:cb00:80:1024::a29e:f134"
whoami.ds.akahelp.net.  20      IN      TXT     "ecs" "111.250.0.0/24/24"

然後 168.95.1.1 則是連 ECS 都不給 XDDD

$ dig whoami.ds.akahelp.net txt @168.95.1.1

[...]

;; ANSWER SECTION:
whoami.ds.akahelp.net.  20      IN      TXT     "ns" "2001:b000:180:8002:0:2:9:114"

之前在找 DNS 類問題的時候還算可以用的工具...

把 Blog 丟到 CloudFront 上

先前在「AWS 流量相關的 Free Tier 增加不少...」這邊有提到一般性的流量從 1GB/month per region 升到 100GB/month，另外 CloudFront 則是大幅增加，從 50GB/month (只有註冊完的前 12 個月) 提升到 1TB/month (不限制 12 個月)，另外 CloudFront 到 EC2 中間的流量是不計費的。

剛剛花了點功夫把 blog 從 Cloudflare 搬到 CloudFront 上，另外先對預設的 /* 調整成 no cache，然後針對 /wp-content/* 另外加上 cache 處理，跑一陣子看看有沒有問題再說...

目前比較明顯的改善就是 latency，從 HiNet 連到免費版的 Cloudflare 會導去美國，用 CloudFront 的話就會是台灣了：

另外一方面，這樣國際頻寬的部份就會走進 AWS 的骨幹，比起透過 HiNet 自己連到美國的 PoP 上，理論上應該是會快一些...

來看 Intel + Varnish 的單機 500Gbps 的 PR 新聞稿

在「Varnish Software Achieves 500Gbps Throughput Per Server for UHD Video Content」這邊看到 PR 稿，由 Intel 與 Varnish 合作，宣稱達到單機 500Gbps 的 throughput 了：

According to Varnish Software, the following were the outcomes of the test:

  • 509.7 Gbps live-linear throughput, using a dual-processor configuration
  • 487.2 Gbps video-on-demand throughput, using a dual-processor configuration

白皮書在「Delivering up to 500 Gbps Throughput for Next-Gen CDNs」這頁可以用個資交換下載，不過用搜尋引擎找一下可以發現 Intel 那邊有放出 PDF (但不確定兩邊給的是不是同一份)：「Delivering up to 500 Gbps Throughput for Next-Gen CDNs」。

單 CPU 的伺服器是四個 100Gbps 界面接出來，雙 CPU 的伺服器是八個 (這邊 SUT 是 system under test 的縮寫)：

These client systems were connected to the CDN servers using 100 GbE links through a switch; 4x100 GbE connections for the single-processor SUT, and 8x100 GbE for the dualprocessor SUT. Testing was done using Wrk, a widely recognized open-source HTTP(S) benchmarking tool.

不過如果實際看圖會發現伺服器是兩個 100Gbps (單 CPU) 與四個 100Gbps (雙 CPU)，然後 wrk 也吃了兩個或是四個 100Gbps：

在白皮書最後面也有提到測試的配置，都是在 Ubuntu 20.04 上面跑，單 CPU 用的是兩張 Intel 的 100Gbps 網卡，雙 CPU 的用的是四張 Mellanox 的 100Gbps 網卡：

3rd generation Intel Xeon Scalable testing done by Intel in September 2021. Single processor SUT configuration was based on the Supermicro SMC 110P-WTR-TNR single socket server based on Intel® Xeon® Platinum 8380 processor (microcode: 0xd000280) with 40 cores operating at 2.3 GHz. The server featured 256 GB of RAM. Intel® Hyper-Threading Technology was enabled, as was Intel® Turbo Boost Technology 2.0. Platform controller hub was the Intel C620. NUMA balancing was enabled. BIOS version was 1.1. Network connectivity was provided by two 100 GbE Intel® Ethernet Network Adapters E810. 1.2 TB of boot storage was available via an Intel SSD. Application storage totaled 3.84TB per drive and was provided by 8 Intel P5510 SSDs. The operating system was Ubuntu Linux release 20.04 LTS with kernel 5.4.0-80 generic. Compiler GCC was version 9.3.0. The workload was wrk/master (April 17, 2019), and the version of Varnish was varnishplus-6.0.8r3. Openssl v1.1.1h was also used. All traffic from clients to SUT was encrypted via TLS.

3rd generation Intel Xeon Scalable testing done by Intel in September 2021. Dual processor SUT configuration was based on the Supermicro SMC 22OU-TNR dual socket server based on Intel® Xeon® Platinum 8380 processor (microcode: 0xd000280) with 40 cores operating at 2.3 GHz. The server featured 256 GB of RAM. Intel® Hyper-Threading Technology was enabled, as was Intel® Turbo Boost Technology 2.0. Platform controller hub was the Intel C620. NUMA balancing was enabled. BIOS version was 1.1. Network connectivity was provided by four 100 GbE Mellanox MCX516A-CDAT adapters. 1.2 TB of boot storage was available via an Intel SSD. Application storage totaled 3.84TB per drive and was provided by 12 Intel P5510 SSDs. The operating system was Ubuntu Linux release 20.04 LTS with kernel 5.4.0-80- generic. Compiler GCC was version 9.3.0. The workload was wrk/master (April 17, 2019), and the version of Varnish was varnish-plus6.0.8r3. Openssl v1.1.1h was also used. All traffic from clients to SUT was encrypted via TLS.

不過馬上就會滿頭問號，四張 100Gbps 是怎麼跑到 500Gbps 的頻寬...

這份 PR 馬上就讓人想到 Netflix 先前放出來的投影片 (先前有在「Netflix 在單機服務 400Gbps 的影音流量」這篇提到)，在 Netflix 的投影片裡面有提到他們在 Intel 平台上面受限於記憶體的頻寬，整台機器只能跑到 230Gbps。

另外一種猜測是，如果 Intel 與 Varnish 宣稱的 500Gbps 是算 switch 上的總流量 (有這樣算的嗎，你是 Juniper 嗎...)，那這邊的 500Gbps 換算回去差不多就是減半 (還很客氣的沒把 cache 沒中需要去 origin server 拉資料的流量扣掉)，跟 Netflix 在 FreeBSD 上跑出來的結果差不多啊...

坐等反駁 XDDD

AWS 流量相關的 Free Tier 增加不少...

Jeff Barr 出來公告增加 AWS 流量相關的 free tier：「AWS Free Tier Data Transfer Expansion – 100 GB From Regions and 1 TB From Amazon CloudFront Per Month」。

一般性的 data transfer 從 1GB/month/region 變成 100GB/mo，現在是 21 regions 所以不會有反例，另外大多數的人或是團隊也就固定用一兩個 region，這個 free tier 大概可以省個 $10 到 $20 左右？

Data Transfer from AWS Regions to the Internet is now free for up to 100 GB of data per month (up from 1 GB per region). This includes Amazon EC2, Amazon S3, Elastic Load Balancing, and so forth. The expansion does not apply to the AWS GovCloud or AWS China Regions.

另外是 CloudFront 的部份，本來只有前 12 個月有 free tier，現在是開放到所有帳號都有，另外從 50GB/month 升到 1TB/month，這個部份的 free tier 就不少了，大概是 $100 到 $200？

Data Transfer from Amazon CloudFront is now free for up to 1 TB of data per month (up from 50 GB), and is no longer limited to the first 12 months after signup. We are also raising the number of free HTTP and HTTPS requests from 2,000,000 to 10,000,000, and removing the 12 month limit on the 2,000,000 free CloudFront Function invocations per month. The expansion does not apply to data transfer from CloudFront PoPs in China.

今年十二月才生效，要注意一下不要現在就用爽爽：

This change is effective December 1, 2021 and takes effect with no effort on your part.

這樣好像可以考慮把 blog 與 wiki 都放上去玩玩看，目前這兩個服務都是用 Cloudflare 的 free tier，HiNet 用戶基本上都是連去 Cloudflare 的美西 PoP，偶而離峰時間會用亞洲的點，但都不會是台灣的 PoP...

不過記得之前 WordPress + CloudFront 有些狀況，再研究看看要怎麼弄好了...

Wasabi 與 Storj DCS

Wasabi 與 Storj 是在看到「Will Cloudflare R2 Win Customers from Amazon S3?」這篇文章時翻到了三個 Cloud Storage Provider，文章本身我倒是沒什麼吸收...。

第一個是 Backblaze 的 B2，這個產品平常的曝光度就還算夠。

另外是 Wasabi 的部份，其中一個賣點是免費的頻寬，但其實限制意外的多。首先是各地區的價錢：

我找了一下到底是什麼地區，目前只有看到「Wasabi Technologies Inc Status」這邊有編號 (US-East-1、US-East-2、US-Central-1、US-West-1、EU-Central-1、AP-Northeast-1)，但也沒找到地區... US 的都在美國沒問題，AP-Northeast-1 應該是日本，但 EU-Central-1 是哪裡就找不到了。

另外是 pay-as-you-go 的方案最小是 1TB，如果是歐美區的話是 US$5.99：

For customers using the Wasabi pay-as-you-go pricing model, Wasabi has a minimum monthly charge associated with 1 TB of active storage. If you store less than 1 TB of active storage in your account, you will still be charged for 1 TB of storage based on the pricing associated with the storage region you are using.

然後也有 90 天的最短計價時間：

Wasabi has a minimum storage duration policy that means if stored objects are deleted before they have been stored with Wasabi for a certain number of days (90 days when using the Wasabi pay-go pricing model), a Timed Deleted Storage charge equal to the storage charge for the remaining days will apply.

另外 Wasabi's free egress policy 這邊也可以看出來他們的設計就是拿來當 storage 用，然後前面需要擋 CDN 之類的服務。

最後一個是 Storj 的 DCS，US$4/TB/month 的空間費用，與 US$7/TB 的流量費用感覺還算便宜？他的做法是把檔案拆成 80 份，然後取 29 份就可以算回來：

How many Nodes are files stored on?

80. We split each file into 80 different encrypted pieces, and each piece is stored on a different Node.

When you retrieve an object, only 29 of its 80 pieces are needed to reconstitute that object. With no central point of failure, your data is always quickly available, all over the world.

這部份是則是透過 Reed-Solomon error correction 實做：

Automate file repair and know that Reed-Solomon erasure coding enables the highest levels of durability for all files uploaded to Storj DCS.

好一陣子沒看到 Reed-Solomon 了，沒想到在這邊看到... 先不管技術的部份，看起來 Storj DCS 的價錢可以玩看看。

Cloudflare R2 Storage 的插曲...

在 Hacker News 首頁上看到「Cloudflare's Disruption (stratechery.com)」這篇，文章「Cloudflare’s Disruption」這篇其實還好，主要就是分析一下 Cloudflare R2 Storage 在下的棋，真的讓我想寫的是反而是 Hacker News 上的討論...

首先是提到了 S3 -> R2 -> Q1 -> P0 這個：

ksec 36 minutes ago | unvote [–]

^gt; The service will be called R2 — “one less than S3,” quipped Cloudflare CEO Matthew Prince in an interview with Protocol ahead of Cloudflare’s announcement

Oh I never thought of that. So the next one is Q1 and final one would be P0.

另外下面有也提到 IBM 與 HAL：

piaste 33 minutes ago | unvote [–]

And it is likely inspired by the old joke that 2001: A Space Odyssey's HAL was one less than "IBM".

下一個 Q1 是明年了，來看看 2022Q1 會不會有 P0 issue XDDD

Netflix 在單機服務 400Gbps 的影音流量

在 Hacker News 首頁上看到 Netflix 在 EuroBSDCon 2021 上發表的投影片：「Serving Netflix Video at 400Gb/s on FreeBSD」，對應的討論則是在「Serving Netflix Video at 400Gb/s [pdf] (freebsd.org)」這邊可以翻到，投影片的作者有在上面回答一些問題。

投影片在講的應該就是 Netflix 的 Open Connect。

主要是因為 Open Connect 的伺服器是放到各家 ISP 機房，在單一 IP 且單一伺服器的限制下，要想辦法壓榨出最高的效能。

硬體是 AMD 的 EPYC，在先前的版本可以達到 240Gbps，經過分析與嘗試解決了一堆問題後，最後是在原來的 AMD 機器上跑到了 380Gbps (另外有測 ARM 以及 Intel 的數字)，然後之後機房有可能會有 800Gbps 的標準，他們又要繼續煩惱...

有看到 Mellanox ConnectX-6 Dx (CX6-DX) 這個東西，看起來很有趣啊，有 200Gbps 的能力，而且可以把 TLS 的事情推到卡上面處理... 然後這家公司被 Nvidia 買走了。

另外當然也會有人問為什麼不用 Linux，作者在討論串裡面也有回答一些，有興趣的可以自己去搜一下。

Cloudflare Images

Cloudflare Images 開放付費使用了：「Cloudflare Images Now Available to Everyone」。

檔案傳到 Cloudflare 上面，然後另外收處理費用：

You pay $5/month for every 100,000 stored images and $1 per 100,000 delivered images. There are no additional resizing, compute or egress costs.

檔案大小的限制是 10MB，所以 $5/month 的 storage 最多可以提供 1TB 的空間，$0.005/GB 算是很漂亮的數字，如果是小圖的話就會比較虧一些？看起來丟大圖會開心一點...

Cloudflare Images offers multiple ways to upload your images. We accept all the common file formats including JPEG, GIF and WEBP. Each image uploaded to Images can be up to 10 MB.

然後支援的檔案格式有常見的 GIF、JPEG、PNG 以及 WebP：

When a client requests an image, Cloudflare Images will pick the optimal format between WebP, PNG, JPEG and GIF.

另外有計畫要支援 AVIF：

We’re just getting started with Cloudflare Images. Here are some of the features we plan to support soon:

AVIF support for even smaller file sizes and faster load times.

沒提到 durability，不知道會有多少...

Cloudflare 開始在正式環境用 ARM server 了

在「Designing Edge Servers with Arm CPUs to Deliver 57% More Performance Per Watt」這邊 Cloudflare 提到了他們在正式環境用 ARM 架構了：

Our first Arm CPU was deployed in production earlier this month — July 2021.

記得測了很多年，其中遇到測試到一半看起來還不錯，但原廠商決定不繼續做的，直到後來又有廠商投入，到現在總算是有比較成熟的產品可以用。

隔壁棚 AWS 上的 ARM 伺服器用起來也是香到不行，還沒有用過的可以試看看，至少我這台 blog & wiki 也都是跑在上面。

另外文章裡有提到目前 x86 的效能，新一代的 AMD 大概只比前一代多了 39% 的每瓦效能，但如果是把 ARM 拿進來比的話會到 57%：

Our most recently deployed generation of edge servers, Gen X, used AMD Rome CPUs. Compared with that, the newest Arm based CPUs process an incredible 57% more Internet requests per watt. While AMD has a sequel, Milan (and which Cloudflare will also be deploying), it doesn’t achieve the same degree of energy efficiency that the Arm processor does — managing only 39% more requests per watt than Rome CPUs in our existing fleet.

開始推上 production 後應該會愈換愈快，而且代表 Cloudflare 也會開始針對 ARM 平台最佳化。

Cloudflare 在巴西的佈點

看到 Cloudflare 在講他們打算在巴西佈 25 個點：「Expanding Cloudflare to 25+ Cities in Brazil」，目前可以看出來是八個點：

比較了領土的大小，跟美國扣掉阿拉斯加差不多等級：

人口的話美國是 328M 左右 (阿拉斯加不到 1M，不太影響感覺)，巴西 215M，依照 Cloudflare 在美國目前有 39 個 PoP 來說，的確是可以拉上去，不過看起來應該是因為有大的 ISP 合作的關係：

Today, we are excited to announce an expansion we’ve been working on behind the scenes for the last two years: a 25+ city partnership with one of the largest ISPs in Brazil.

不過沒提到是哪個 ISP，之後看看有沒有消息...