Backblaze 的資料量

Backblaze 在「Yes, Backblaze Just Ordered 100 Petabytes of Hard Drives」這篇提到他們這一次買了 100PB 的硬碟,不過這應該還好 XD


破 400PB 了... 而且看起來成長速度頗穩定的。

我也用 rclone 把自己的資料往上面備份 (先加密再傳上去),目前連 free tier 都還沒用完 (雖然信用卡卡號掛著)。

Facebook 把 InnoDB 換成 MyRocks 的計畫

在「Migrating a database from InnoDB to MyRocks」這邊講到了 FacebookMySQL 的 InnoDB 換成 MyRocks 的計畫。

Facebook 已經大量導入全 Flash 的環境,於是現在 InnoDB (Compressed) 的情況類似於這樣:

可以看到空間是最大的問題 (Space-bound),而非 CPU 或是 I/O,這在純 Flash 的機器上還蠻有可能的,因為純 Flash 能提供的 I/O capacity 超高,加上 Facebook 把 MySQL 當作後端儲存設備而已,CPU 的使用量可以預期不高...

然後他們估算 MyRocks 可以省下一半的空間,於是有可能變成:

不過對於一般單位來說,這些前提就未必成立了... 但還是可以看到 Mark Callaghan 花不少力氣在調校一般性的效能,希望讓使用者多一點...

Quotient filter

之前有提過「Cuckoo Filter:比 Bloom Filter 多了 Delete」,最近在「A general purpose counting filter: making every bit count」這邊看到 Quotient filter,也是類似 Bloom filter 的資料結構,但想要解決更多問題。

一般的 Bloom filter (BF) 會有這些問題:

  • The inability to delete items
  • Poor scaling out of RAM
  • The inability to resize dynamically
  • The inability to count the number of occurrences of each item, especially with skewed input distributions.

而文章裡提到的 Quotient filter (QF) 就是要解這些問題。另外還提到了 Rank-and-Select-based Quotient filter (RSQF) 以及 Counting Quotient filter (CQF)。雖然多了一些空間需求,但看起來解掉不少問題... (尤其是刪除的能力)

效能上也還不錯,尤其是讀取速度的部份... 不過不知道相對於 Cuckoo filter 差多少。

Amazon Redshift 壓縮率的改善

Amazon Redshift 對壓縮率的改善:「Data Compression Improvements in Amazon Redshift Bring Compression Ratios Up to 4x」。

首先是引入了 Zstandard

First, we added support for the Zstandard compression algorithm, which offers a good balance between a high compression ratio and speed in build 1.0.1172. When applied to raw data in the standard TPC-DS, 3 TB benchmark, Zstandard achieves 65% reduction in disk space. Zstandard is broadly applicable.


Second, we’ve improved the automation of compression on tables created by the CREATE TABLE AS, CREATE TABLE or ALTER TABLE ADD COLUMN commands. Starting with Build 1.0.1161, Amazon Redshift automatically chooses a default compression for the columns created by those commands. Automated compression happens when we estimate that we can reduce disk space without degrading query performance. Our customers have seen up to 40% reduction in disk space.


Third, we’ve been optimizing our internal on-disk data structures. Our preview customers averaged a 7% reduction in disk space usage with this improvement. This feature is delivered starting with Build 1.0.1271.


Finally, we have enhanced the ANALYZE COMPRESSION command to estimate disk space reduction.

不過其他幾個產品線的使用方式更成熟 (像是 Amazon Athena 這類產品),不知道會不會讓 Amazon Redshift 慢慢退出第一線...


這幾年對於開放式空間有不少反面意見出來,像是這幾天 BBC 登的「Why open offices are bad for us」。


Numerous companies have embraced the open office — about 70% of US offices are open concept — and by most accounts, very few have moved back into traditional spaces with offices and doors.

但人的效率會因為開放式空間大約掉 15%:

But research that we’re 15% less productive, we have immense trouble concentrating and we’re twice as likely to get sick in open working spaces, has contributed to a growing backlash against open offices.

採用開放式空間最常見的理由包括辦公室成本 (每個人平均分到的空間大小會比較低),另外一個是藉由開放式空間讓互相討論合作的成本降低,但因為開放式空間,反而是影響到別人的情況比討論合作的情況多,甚至是與工作無關的事情也會影響到期他人:

Beside the cheaper cost, one main argument for the open workspace is that it increases collaboration. However, it’s well documented that we rarely brainstorm brilliant ideas when we’re just shooting the breeze in a crowd. Instead, as many of us know, we’re more likely to hear about the Christmas gift a colleague is buying for a family member, or problems with your deskmate’s spouse.

其實科技的進步讓遠端溝通的成本降低了不少,像是 SlackZoom,現在未必要靠 open office 的架構讓大家溝通了。

debootstrap 可以把所有檔案都裝到 /usr 下了

從「You can now try merged /usr in Debian」這邊看到,早期因為硬碟空間比較小的關係,會切成 /bin/usr/lib 之類的目錄:

The original impetus for requiring these directories was due to space limitations in the first Unix implementations, developers favoring the change point out.

這個理由很明顯已經消失了,所以 Debian 就規劃要整併起來...

分析 GitHub 上的 Tab 與 Space

作者用 BigQuery 分析了 GitHub 上的 Tab 與 Space 的差異 (是個 flame war 開始的節奏 XDDD):「400,000 GitHub repositories, 1 billion files, 14 terabytes of code: Spaces or Tabs?」。

可以看到除了 C 與 Go 以外,大多數的程式語言都是 Space > Tab。另外在文章下面也有使用的 BigQuery 指令可以參考。

Netflix 對 sendfile() 在 TLS 情況下的加速

Netflix 對於寫了一篇關於隱私保護的技術細節:「Protecting Netflix Viewing Privacy at Scale」。

其中講到 2012 年的 Netflix Open Connect 中的 Open Connect Appliance (OCA,放伺服器到 ISP 機房的計畫) 只有單台伺服器 8Gbps,到現在 2016 可以達到 90Gbps:

As we mentioned in a recent company blog post, since the beginning of the Open Connect program we have significantly increased the efficiency of our OCAs - from delivering 8 Gbps of throughput from a single server in 2012 to over 90 Gbps from a single server in 2016.

早期的 Netflix 走 sendfile() 將影片丟出去,這在 kernel space 處理,所以很有效率:

當影片本身改走 HTTPS (TLS) 時,其中一個遇到的效能問題是導致 sendfile() 無法使用,而必須在 userland space 加密後改走回傳統的 write() 架構,這對於效能影響很大:

所以他們就讓 kernel 支援 AES 系列加密 (包括 AES-GCM 與 AES-CBC),效能的提昇大約是 30%:

Our changes in both the BoringSSL and ISA-L test situations significantly increased both CPU utilization and bandwidth over baseline - increasing performance by up to 30%, depending on the OCA hardware version.

文章開頭也有提到選 AES-GCM 與 AES-CBC 的一些來龍去脈,主要是 AES-GCM 的安全強度比較好,另外考慮到舊的 client 不支援 AES-GCM 時會使用 AES-CBC:

We evaluated available and applicable ciphers and decided to primarily use the Advanced Encryption Standard (AES) cipher in Galois/Counter Mode (GCM), available starting in TLS 1.2. We chose AES-CGM over the Cipher Block Chaining (CBC) method, which comes at a higher computational cost. The AES-GCM cipher algorithm encrypts and authenticates the message simultaneously - as opposed to AES-CBC, which requires an additional pass over the data to generate keyed-hash message authentication code (HMAC). CBC can still be used as a fallback for clients that cannot support the preferred method.

另外 OCA 機器本身也都夠新,支援 AES-NI 指令集,效能上不是太大的問題:

All revisions of Open Connect Appliances also have Intel CPUs that support AES-NI, the extension to the x86 instruction set designed to improve encryption and decryption performance. We needed to determine the best implementation of AES-GCM with the AES-NI instruction set, so we investigated alternatives to OpenSSL, including BoringSSL and the Intel Intelligent Storage Acceleration Library (ISA-L).

不過在「Netflix Open Connect Appliance Deployment Guide」(26 July 2016 版) 這份文件裡看起來還是用多條 10Gbps 透過 LACP 接上去:

You must be able to provision 2-4 x 10 Gbps ethernet ports in a LACP LAG per OCA. The exact quantity depends on the OCA type.

可能是下一版準備要上 40Gbps 或 100Gbps 的準備...?


在「The International Space Station (Finally) Gets an Espresso Machine」這篇看到的,原始的報導出自「The International Space Station (finally!) gets an espresso machine」。

幾個重點 XDDD

The ISSpresso requires 120V DC power which is obtained at the Utility Outlet Panel (UOP) on the ISS.

很特別的電力,是 120V DC 而非 120V AC...