Amazon EFS 漲價,再推出給更「冷」的資料儲存的空間:Amazon EFS Archive

Amazon EFS 這次推出的是再多推出一個 storage class:「Optimize your storage costs for rarely-accessed files with Amazon EFS Archive」。

先前應該是 2019 的時候推出了 IA:「Amazon EFS 的 IA Storage Class」,現在的 Archive 就是新的 storage class,儲存成本更便宜,但取用成本更高。

us-east-1 的價錢來看,可以到 Archive 的成本是 IA 的一半:

Standard (GB-Month)	$0.30
Infrequent Access (GB-Month)	$0.016
Archive (GB-Month)	$0.008
Backup - Warm / Cold (GB-Month)	$0.05 / $0.01

讀取成本則是 IA 的三倍:(這邊的 Tiering 指的是自動化的搬遷的服務)

All storage classes - Reads (per GB transferred)	$0.03
All storage classes - Writes (per GB transferred)	$0.06
Infrequent Access - Reads (incremental charge per GB transferred)	$0.01
Infrequent Access - Tiering (per GB transferred)*	$0.01
Archive - Reads (incremental charge per GB transferred)	$0.03
Archive - Tiering (per GB transferred)*	$0.03

基本上就是 Amazon S3 那套分級方法陸陸續續搬過來的感覺。

然後注意到這個「Regional (Multi-AZ) with Elastic Throughput」是新的計價方案,就算是 Standard storage class,I/O 是要算錢的。

在舊的方案「Regional (Multi-AZ) with legacy throughput modes」裡面,Standard 的 I/O 是不用額外付費,已經包在裡面,除非你直接購買 Provisioned (保證速度):

Standard (GB-Month)	$0.30
Infrequent Acces (GB-Month)	$0.025
Backup - Warm / Cold (GB-Month)	$0.05 / $0.01
Provisioned Throughput (MB/s-Month)	$6.00
Infrequent Access - Reads (per GB transferred)	$0.01
Infrequent Access - Tiering (per GB transferred)*	$0.01

翻了一下 Internet Archive 可以確認前幾天 2023/11/26 的 pricing 頁面還是舊的,也就是說這是這次推出來的改變:「Amazon EFS Pricing」。

看了一下目前 blog 上最近掛 Amazon EFS 類別的三篇都沒提到這件事情 (「New – Announcing Amazon EFS Elastic Throughput」、「Optimize your storage costs for rarely-accessed files with Amazon EFS Archive」以及「Replication failback and increased IOPS are new for Amazon EFS」),要用的人自己注意一下?

下載 YouTube 影片的技術限制與繞過方法

Hacker News 上看到這篇「How They Bypass YouTube Video Download Throttling」在講 YouTube 防止下載的各種方式。

透過 API 拿到的 URL 直接抓很慢,大約 40-70KB/sec:

However, attempting to download from this URL leads to really slow download:

The speed is always limited to around 40-70kB/s.

這邊需要一個 javascript 環境計算出 n,帶入後續的 request 以「證明」你是官方的網頁 client:

Since mid-2021, YouTube has included the query parameter n in the majority of file URLs. This parameter needs to be transformed using a JavaScript algorithm located in the file base.js, which is distributed with the web page. YouTube utilizes this parameter as a challenge to verify that the download originates from an “official” client. If the challenge is not resolved and n is not transformed correctly, YouTube will silently apply throttling to the video download.

The JavaScript algorithm is obfuscated and changes frequently, so it’s not practical to attempt reverse engineering to understand it. The solution is simply to download the JavaScript file, extract the algorithm code, and execute it by passing the n parameter to it. The following code accomplishes this.

但即使算出 n,也還是會限速,可以看到作者策出來大約是 4MB/sec,雖然比以前快很多了,但還是看得出來有限速。這主要是避免 client 端過度 buffer 浪費頻寬:

With this new URL containing the correctly transformed n parameter, the next step is to download the video. However, YouTube still enforces a throttling rule. This rule imposes a variable download speed limit based on the size and length of the video, aiming to provide a download time that’s approximately half the duration of the video. This aligns with the streaming nature of videos. It would be a massive waste of bandwidth for YouTube to always provide the media file as quickly as possible.

接下來的方式就是利用 Range 拆成很多個 HTTP request 打,這樣因為 buffering algorithm 在開始限速前會先全速塞資料給你,就可以用這點避開限速的問題了。

把多的 request 與處理時間都算進去後,整體大約可以到 50-70MB/sec,算是可以接受的下載速度了:

However, the average speeds typically ranged between 50-70 MB/s or 400-560 Mb/s, which is still pretty fast.

後面有一些合併處理的指令 (因為 YouTube 會把影與音分離成兩個檔案),就不是重點了...

Internet Archive 被打

Internet Archive 更新一篇文章,說明前幾天被打掛的事情:「Let us serve you, but don’t bring us down」。

有 64 台機器 (或是 64 個 IP) 從 AWS 打了幾萬 rps 進 Internet Archive:

Tens of thousands of requests per second for our public domain OCR files were launched from 64 virtual hosts on amazon’s AWS services. (Even by web standards,10’s of thousands of requests per second is a lot.)

然後擋掉這些 IP 後恢復正常,但過了幾個小時後又換 IP 被打了:

But, another 64 addresses started the same type of activity a couple of hours later.

找了一下之前有寫過「限制流量的方式 (rate limit)」這篇,裡面提到 Figma 怎麼處理,另外以前自己搞 apache module 後面接 memcached 達到跨機器統計的作法。

就 Internet Archive 的服務來說,是應該要搞個類似的東西來擋,不然可以預期會不斷發生?

跑 ArchiveTeam Warrior

Archive Team 是一個致力於保存數位資料的組織,而 ArchiveTeam Warrior 則是他們提供的軟體,可以讓你很方便直接跑 worker 加入他們的 cluster,幫忙抓資料並且保存到 Internet Archive 上。

他們提供三種方法跑 ArchiveTeam Warrior,第一種是 VM 的方式,文件裡面有介紹怎麼用 VirtualBox 或是 VMware Player 跑起來。

第二種與第三種都是 container 類的方式,DockerPodman 都能跑起來。

跑起來後可以連進 http://127.0.0.1:8001/,然後選擇想要加入的項目,或是接受指令選擇目前團隊主打的項目。

在「Projects」這頁可以看到目前主力是備份 Enjin 上的資料。

丟了兩台 VPS 的機器上去用 Docker 跑,CPU 使用率看起來很低,但網路流量看起來會因為所在的地點而差蠻多的,一台大約是 300KB/sec 到 400KB/sec,換算後大約是 1TB/mo,另外一台則只有 1/10 的量。

備份 Xuite Blog 的公開文章

中華的 Xuite 前陣子宣佈了服務中止的公告:「Xuite隨意窩平台服務終止公告」(這邊就先拉 Internet Archive 的連結了,看起來之後會消失...)。

Blog 的部份,除了作者本身可以拉資料下來放到其他平台以外,外人也可以把這些歷史遺跡保留下來,像是丟到 Internet Archive 的 Wayback Machine 上面。

所以用 Perl 寫了一隻 script,把 url 掃出來後,後續就可以用其他工具 submit 到 Wayback Machine 上面:「xuite-urldump」。

當年有不少 ACG 相關的 blog 在上面,先來備份起來...

Internet Archive 的 Flash 收藏

剛剛看到 Internet Archive 的這篇文章:「Flash Animations Live Forever at the Internet Archive」。

Internet Archive 把模擬器掛上去了,所以你可以直接在網站上用這些 Flash 程式:

Great news for everyone concerned about the Flash end of life planned for end of 2020: The Internet Archive is now emulating Flash animations, games and toys in our software collection.

看起來是透過 ruffle + WebAssembly 轉到瀏覽器上面跑:

Utilizing an in-development Flash emulator called Ruffle, we have added Flash support to the Internet Archive’s Emularity system, letting a subset of Flash items play in the browser as if you had a Flash plugin installed. While Ruffle’s compatibility with Flash is less than 100%, it will play a very large portion of historical Flash animation in the browser, at both a smooth and accurate rate.

You will not need to have a flash plugin installed, and the system works in all browsers that support Webassembly.

然後在「Software Library: Flash Showcase」這邊有一些 showcase 可以看,基本上就是測過沒問題的。另外在「Software Library: Flash」看起來就是整包了...

搜了一下以前有在玩的 Zookeeper,好像沒有在裡面...

Internet Archive 的頻寬...

看到「Thank you for helping us increase our bandwidth」這邊在說明 Internet Archive 的流量資訊:

看起來平常就是滿載的情況,然後加上去的流量馬上就被吃掉了... 這些資料看起來是放在 Cacti 這邊,不過只有 error rate 有開放讓大家翻,流量的部份看起來要登入才能看。

AWS 推出更便宜的儲存方案 Glacier Deep Archive

AWS 推出的這個方案價錢又更低了:「New Amazon S3 Storage Class – Glacier Deep Archive」。

在這之前在 us-east-1S3 最低的方案是 Glacier Storage,單價是 USD$0.004/GB (也就是 $4/TB)。

而這次推出的 Glacier Deep Archive Storage 在同一區則是直接到 USD$0.00099/GB ($0.99/TB),大約是 1/4 的價錢。

Glacier Deep Archive 在取得時 first byte 的保證時間是 12 小時,另外最低消費是 180 天:

Retrieval time within 12 hours

先前就有的 Glacier Storage 則是可以在取用時設定取得的 pattern (會影響 first byte 的時間),而最低消費是 90 天:

Configurable retrieval times, from minutes to hours

Pricing for each of these metrics is determined by the speed at which data is requested based on three options. "Expedited" queries <250 MB are typically returned in 1-5 minutes. "Standard" queries are typically returned in 3-5 hours. "Bulk" queries are typically returned in 5-12 hours.

多一個更便宜的選擇可以用。

保存網頁的工具 ArchiveBox

pirate/ArchiveBox 這個專案:

The open source self-hosted web archive. Takes browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

他的想法是用 command line 就可以保存:

echo 'https://example.com' | ./archive

然後提供一個網頁介面存取,類似於 Internet Archive 的技術架構?

不過 Internet Archive 因為在美國有拿到豁免權 (像是這篇所說的:「Internet Archive Gets DMCA Exemption To Help Archive Vintage Software」),還是有他的方便性...

HTTP Archive 分析 CDN 的使用情況...

Twitter 上看到 HTTP Archive 分析各家 CDN 的使用情況,給了一個蠻意外的結果,可以看到 Cloudflare 的佔比相當高:

進去看一下分析的方式,是拿 Google Chrome 提供的資料來再進一步分析:

As of December 15th 2018, the HTTP Archive is crawling the full list of desktop origins from the Chrome User Experience (CrUX) Report for the desktop crawls (mobile will be added as of January 1, 2019). The URL list used is the latest available at the time of the crawl (November 2018 in this case).

這應該是出自「Chrome User Experience Report」這邊:

The Chrome User Experience Report is powered by real user measurement of key user experience metrics across the public web, aggregated from users who have opted-in to syncing their browsing history, have not set up a Sync passphrase, and have usage statistic reporting enabled.

由於有條件限制,可以預期會有偏差,不過以目前 Google Chrome 的市占率來說,應該是有意義的資料。而這份資料跑出來的數據看到 Cloudflare 的影響力遠遠超過其他 CDN 讓人頗意外...