Pinterest 在 InnoDB Compression 的努力

Pinterest 用 InnoDB 儲存各式資料,而且使用了 InnoDB Compression 的功能。他們花了不少力氣跟 Percona 合作改善 InnoDB Compression 的效能:「Evolving MySQL Compression - Part 1」。

文章有點長度,重點在於他們在 MySQL 裡面放了大量的 JSON:

A Pin is stored as a 1.2 KB JSON blob in sharded MySQL databases.

他們發現新版 zlib 的 predefined dictionary 可以讓壓縮率變得更高 (從本來的 ~50% 到 ~66%);而除了壓縮率變高外,由於事先定義了字典內容,對於效能的提昇也不少 (warm up):

Zlib version was released in early 2013 and added the ability to use a predefined “dictionary” to prefill the lookback window for LZ77. This seemed promising since we could “warm up” the lookback window with field names and other common strings. We ran a few tests using the Python Zlib library with a naive predefined dictionary consisting of an arbitrary Pin JSON blob. The compression savings increased from ~50% to ~66% at what appeared to be relatively little cost.

另外他們做了 read-only 的 benchmark (畢竟這是重點)。圖片資料有點糊,但可以看出 y 軸是 Queries/sec。而 x 軸上則用文字給了些說明,黃色是 TokuDB,紅色是本來的 InnoDB Compression,剩下的都是不同的字典集的成果:

Below is a graph from our presentation which showed a read-only version of our production workload at concurrency of 256, 128, 32, 16, 8, 4 and 1 clients. TokuDB is in yellow, InnoDB page compression is in red and the other lines are column compression with a variety of dictionaries.

整體效率都比之前高不少,尤其是當 concurrent query 的數量偏高的時候差距會很大。

而這個功能將會納入未來的 Percona 版本,對於在 MySQL 裡面會塞 JSON 或是 XML 的人應該會很有幫助:

We worked with Percona to create a specification for column compression with an optional predefined dictionary and then contracted with Percona to build the feature.

This entry was posted in Computer, Database, Murmuring, MySQL, Software and tagged , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *