KeyDB:使用 Multithreading 改善 Redis 的效能

Hacker News 上看到有支援 Multithreading 的 Redis fork:「KeyDB – A Multithreaded Fork of Redis (keydb.dev)」,官網在「KeyDB - The Faster Redis Alternative」這邊。

不過這篇是要記錄從 Hacker News 看到的雷點,這樣以後自己再找資料的時候會比較容找到。

36022425 這篇是跳下去用發現不太行,最後在 application 端實作需要的 feature,後端還是用原廠的 Redis:

To counter what the other active business said, we tried using KeyDB for about 6 months and every fear you concern you stated came true. Numerous outages, obscure bugs, etc. Its not that the devs aren’t good, its just a complex problem and they went wide with a variety of enhancements. We changed client architecture to work better with tradition Redis. Combined with with recent Redis updates, its rock solid and back to being an after-thought rather than a pain point. Its only worth the high price if it solves problems without creating worse ones. I wish those guys luck but I wont try it again anytime soon.

* its been around 2 years since our last use as a paying customer. YMMV.

另外是在專案裡搜尋「is:open is:issue label:"Priority 1"」的結果可以看到不太妙,在 36021108 這邊有提到的問題:

Filed July, eventually marked priority 1 in early December, not a single comment or signs of fix on it since. That doesn't look good at all.

然後 36020184 有提到 Snap 買進去後沒有什麼在管 open source project 的部分了:

I think I'll stay far away from this thing anyway. Numerous show-stopper bug reports open and there hasn't been a substantial commit on the main branch in at least a few weeks, and possibly months. I'll be surprised if Snap is actually paying anybody to work on this.

Meta (Facebook) 把 MySQL replication 丟上自製的 Raft 系統

看到「Building and deploying MySQL Raft at Meta」這篇,在講 Meta (Facebook) 把 MySQL 的 replication 架構換成自己用 Raft 的系統。

舊的系統是走 MySQL 的 semisync replication:

Previously, our replication solution used the MySQL semisynchronous (semisync) replication protocol.

其中 semisync replication 是在 MySQL 5.5 加入的功能,在至少一個遠端收到 replication log 後才傳回成功 (可以設定數量):「Semisynchronous Replication」。

Semisynchronous replication falls between asynchronous and fully synchronous replication. The source waits until at least one replica has received and logged the events (the required number of replicas is configurable), and then commits the transaction.

然後舊的系統是透過一包 Python 軟體在管理這些機器的各種 failover 操作:

The control plane operations (e.g., promotions, failover, and membership change) would be the responsibility of a set of Python daemons (henceforth called automation).

這個方法常遇到的問題是切換 primary server (以前叫做 master server) 時有可能會因為 binlog position 接不起來而失敗。

所以後來 MySQL 導入了 GTID,可以緩解這個問題,但還是有可能會發生不同的 secondary server (以前叫做 slave server) 會有不一樣的資料。

而在 Meta 改出來的架構裡面,把 replication data 直接寫到一個用 Raft 同步的系統,同步到其他的 secondary server 上面:

In MySQL Raft:

  • Primary writes to binlog via Raft, and Raft sends binlog to followers/replicas.
  • Replicas/followers receive in binlog and apply the transactions to the engine. An apply log is created during apply.
  • Binlog is the replicated log from the Raft point of view.

是個一般單位不太會遇到的架構,而且可以預期其他公司的人遇到類似問題應該也不會用這個方法解...

MariaDB 嘗試相容於 PostgreSQL 協定的產品

Twitter 上看到的消息,新聞在「MariaDB's Xpand offers PostgreSQL compatibility without the forking drama」這邊:

看起來是 SkySQL 的服務,這樣聽起來不像會 open source... 看起來賣點在於 globally distributed RDBMS 這個部分:

MariaDB is previewing a PostgreSQL-compatible front end in its SkySQL Database-as-a-Service which provides a globally distributed RDBMS on the back end.

再看看後續會不會有更多消息?

測了 PGroonga,PostgreSQL 上的 fulltext search engine

PostgreSQL 的 news 頁上看到「PGroonga 3.0.0 - Multilingual fast full text search」,想到一直沒有測過 PGroonga,就找台機器測了一下。

PGroonga 是以 Groonga 為引擎提供 PostgreSQL 全文搜尋能力的套件,是個能支援 CJK 語系的全文搜尋套件。

可以先看一下支援的 column type 與對應的語法:「Reference manual | PGroonga」,可以發現基本的 texttext[]varcharvarchar[] 都有支援,比較特別的是有 jsonb,看起來是對裡面的 text 欄位搜尋。

另外一個比較特別的是他會去配合 LIKE '%something%' 這樣的語法,對於無法修改的既有程式也會有幫助。

缺點方面,官方有提到產生出來的 index 會比其他的套件大,但畢竟我們在的環境要支援 CJK,場上的選手已經不多了。

另外一個缺點是目前 AWSRDSGCPCloud SQL 看起來都沒支援,要用的話得自己架 & 自己管,也許可以考慮用老方法,replication 接出來?

接下來就是安裝測試了,我在 x86-64 上的 Ubuntu 22.04 上面測試,就照著「Install on Ubuntu | PGroonga」這頁裡面的「How to install for system PostgreSQL」這段就可以了,裝系統的 PostgreSQL 14 以及 postgresql-14-pgroonga,之後要用 PostgreSQL 官方的新版的話可以參考「How to install for the official PostgreSQL」這段的安裝。

後續再到「Tutorial | PGroonga」頁,針對要搜尋的欄位下 index (這邊裱格式 memos,欄位是 content):

CREATE INDEX ON memos USING pgroonga (content);

官方的教學文件裡是用 SET enable_seqscan = off; 關閉 sequence scan,可以用 EXPLAIN 看到使用了 index:

test=# SELECT * FROM memos WHERE content &@ 'engine';
 id |                                content                                 
----+------------------------------------------------------------------------
  2 | Groonga is a fast full text search engine that supports all languages.
(1 row)

test=# EXPLAIN SELECT * FROM memos WHERE content &@ 'engine';
                                   QUERY PLAN                                    
---------------------------------------------------------------------------------
 Index Scan using memos_content_idx on memos  (cost=0.00..43.18 rows=1 width=36)
   Index Cond: (content &@ 'engine'::text)
(2 rows)

先拔掉 index:

test=# DROP INDEX pgroonga_content_index;
DROP INDEX

接著要塞資料,這邊拿 CQD 生的「中文假文產生器」來用,有 API 可以接比較方便。

test=# SELECT COUNT(*) FROM memos;
 count  
--------
 100000
(1 row)

Time: 15.495 ms

接著多跑幾次測試直接用 LIKE '%台北%' 去找,可以看到大概都在 150ms 以上:

test=# SELECT COUNT(*) FROM memos WHERE content LIKE '%台北%';
 count 
-------
   710
(1 row)

Time: 178.784 ms

接著來建立 index:

test=# CREATE INDEX ON memos USING pgroonga (content);
CREATE INDEX
Time: 17638.124 ms (00:17.638)

再跑幾次同樣的 query,可以看到巨大的改善:

test=# SELECT COUNT(*) FROM memos WHERE content LIKE '%台北%';
 count 
-------
   710
(1 row)

Time: 9.876 ms

SQL:2023 的新玩意

Hacker News 上看到「SQL: 2023 is finished: Here is what's new (eisentraut.org)」這篇題到了 SQL:2023 標準的新東西,對應的原文在「SQL:2023 is finished: Here is what's new」這邊。

「UNIQUE null treatment (F292)」讓你可以決定 NULL 到底要不要算 unique,剛好跟之前寫過的「PostgreSQL 15 將可以對透過 UNIQUE 限制 NULL 的唯一性了」要做的事情一樣。

「ORDER BY in grouped table (F868)」則是針對沒有出現在 SELECT 的欄位頁可以 ORDER BY,看了一下說明,主要是在 JOIN 的時候限制住了。很明顯的 workaround 是多加上這個欄位,但就代表會增加傳回的資料量。

「GREATEST and LEAST (T054)」這個因為 MIN()MAX() 已經被 aggregate function 用掉了,所以只好另外取名。

「String padding functions (T055)」與「Multi-character TRIM functions (T056)」是熟悉的語法,各家都有對應的 function 可以做,但這次就放進標準化。

「Optional string types maximum length (T081)」是 VARCHAR 可以不用指定大小了,實務上應該是還好?

「Enhanced cycle mark values (T133)」這編提到的 recursive 真的是每次用每次忘,然後 cycle 這個功能就沒看懂了...

「ANY_VALUE (T626)」看起來可以隨機取出資料,搭配 GROUP BY '' 就不用拿 ORDER BY RAND() 這種髒髒的東西出來了?

「Non-decimal integer literals (T661)」與「Underscores in numeric literals (T662)」都是讓數字更好讀以及操作。

後面講了很多 JSON 功能,看起來是 SQL:2016 有先納入一些,但 SQL:2023 補的更完整了。

然後有 Graph 相關的標準也被定義進 SQL:2023,原文介紹的也不是很多,看起來是要跨足過來?

MySQL 5.7 的支援只到今年十月 (Oct 2023)

剛剛翻資料才看到 OracleMySQL 5.7 的支援原來只剩下半年了,預定在 2023 年十月中止:「Oracle Technology Products - Oracle Lifetime Support Policy」。

隔壁棚 Percona 包的 Percona Server for MySQL 5.7 可以從「Percona Release Lifecycle Overview」這邊查,看起來也設定一樣的時間 (2023 年十月),但不確定會不會宣佈延長,至少提供 security fix 之類的。

一直沒注意,突然發現只剩下半年...

MongoDB 的替代方案 FerretDB 推出 1.0 (GA) 版本

Hacker News 上看到 FerretDB 推出 1.0 (GA) 版本:「FerretDB: open-source MongoDB alternative (ferretdb.io)」,原文在「Announcing FerretDB 1.0 GA - a truly Open Source MongoDB alternative」這邊。

當初有寫過「MangoDB 改名為 FerretDB (雪貂)」這篇,但沒注意到他們成立公司來開發?在「Careers at FerretDB」這邊可以看到 hiring 的訊息。

官網有整理出目標,像是他們提到不是以 drop-in replacement 為目標,而是實做核心功能與常用的功能,涵蓋大多數的使用者:

Is FerretDB 100% compatible with MongoDB?

It is not necessary, nor it is feasible to implement every single MongoDB feature out there. Our aim is to cover the core feature set of MongoDB, and then continue adding features which could enhance the experience or increase application compatibility. Non-OSS alernatives of MongoDB are similar in this sense, eg. none of these products are able to provide the full feature set of MongoDB. We are aiming to please 85% of MongoDB users, not all of them.

但這樣也讓想換的人會有一些顧慮... 而且這邊的 85% 不知道是怎麼喊出來的?

把 RabbitMQ 換成 PostgreSQL 的那篇文章...

Hacker News 上看到「SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue (prequel.co)」這篇文章,原文在「SQL Maxis: Why We Ditched RabbitMQ And Replaced It With A Postgres Queue」這邊,裡面在講他們把 RabbitMQ 換成 PostgreSQL 的前因後果。

文章裡面可以吐嘈的點其實蠻多的,而且在 Hacker News 上也有被點出來,像是有人就有提到他們遇到了 bug (或是 feature) 卻不解決 bug,而是決定直接改寫成用 PostgreSQL 來解決,其實很怪:

In summary -- their RabbitMQ consumer library and config is broken in that their consumers are fetching additional messages when they shouldn't. I've never seen this in years of dealing with RabbitMQ. This caused a cascading failure in that consumers were unable to grab messages, rightfully, when only one of the messages was manually ack'ed. Fixing this one fetch issue with their consumer would have fixed the entire problem. Switching to pg probably caused them to rewrite their message fetching code, which probably fixed the underlying issue.

另外一個吐嘈的點是量的部份,如果就這樣的量,用 PostgreSQL 降低使用的 tech stack 應該是個不錯的決定 (但另外一個問題就是,當初為什麼要導入 RabbitMQ...):

>To make all of this run smoothly, we enqueue and dequeue thousands of jobs every day.

If you your needs aren't that expensive, and you don't anticipate growing a ton, then it's probably a smart technical decision to minimize your operational stack. Assuming 10k/jobs a day, thats roughly 7 jobs per minute. Even the most unoptimized database should be able to handle this.

在同一個 thread 下面也有人提到這個量真的很小,甚至直接不講武德提到可以用 Jenkins 解 XD:

Years of being bullshitted have taught me to instantly distrust anyone who is telling me about how many things they do per day. Jobs or customers per day is something to tell you banker, or investors. For tech people it’s per second, per minute, maybe per hour, or self aggrandizement.

A million requests a day sounds really impressive, but it’s 12req/s which is not a lot. I had a project that needed 100 req/s ages ago. That was considered a reasonably complex problem but not world class, and only because C10k was an open problem. Now you could do that with a single 8xlarge. You don’t even need a cluster.

10k tasks a day is 7 per minute. You could do that with Jenkins.

然後意外看到 Simon Willison 提到了一個重點,就是 RabbitMQ 到現在還是不支援 ACID 等級的 job queuing (尤其是 Durability 的部份),也就是希望 MQ 系統回報成功收到的 task 一定會被處理:

The best thing about using PostgreSQL for a queue is that you can benefit from transactions: only queue a job if the related data is 100% guaranteed to have been written to the database, in such a way that it's not possible for the queue entry not to be written.

Brandur wrote a great piece about a related pattern here: https://brandur.org/job-drain

He recommends using a transactional "staging" queue in your database which is then written out to your actual queue by a separate process.

這也是當年為什麼用 MySQL 幹類似的事情,要 ACID 的特性來確保內容不會掉。

這也是目前我覺得唯一還需要用 RDBMS 當 queue backend 的地方,但原文公司的想法就很迷,遇到 library bug 後決定換架構,而不是想辦法解 bug,還很開心的寫一篇文章來宣傳...

Amazon DocumentDB 支援 MongoDB 5.0 協定

看到「Amazon DocumentDB (with MongoDB compatibility) adds support for MongoDB 5.0 wire protocol and client-side field level encryption」這篇,Amazon DocumentDB 推出了 MongoDB 5.0 的支援。

MongoDB 5.0 是 2021 年七月的版本,對使用者端比較重要的看起來是 PII 處理與 TSDB 的支援。

比較感興趣的是,TSDB 的部份是繼續用 PostgreSQL 魔改嗎...?

Etsy 使用 Vitess 的過程

Etsy 寫了三偏關於使用 Vitess 解決資料庫效能問題的文章:「Scaling Etsy Payments with Vitess: Part 1 – The Data Model」、「Scaling Etsy Payments with Vitess: Part 2 – The “Seamless” Migration」、「Scaling Etsy Payments with Vitess: Part 3 – Reducing Cutover Risk」。

Vitess 是 YouTube 團隊開發出來的東西,試著透過一層 proxy 解決後端 MySQL 資料庫在 sharding 後查詢邏輯的問題。

有一些地方的資訊整理出來:

首先是現代暴力解的能耐,從維基百科可以查到 Etsy 在 2015 年就上市了,但到了 2020 年年底撞到 vertically scaling 的天花板 (這邊是指 GCP 的上限),可以看到現在的暴力法可以撐超久... 如果再多考慮到實體機房的話應該可以找到更大台的機器。

第二個是 Etsy 在 2020 年年底開始從資料庫搬資料,一路到 2022 年五月,算起來差不多搬了一年半,總共轉移了 4 個 database 到 Vitess 的 cluster 上,共 23 張表格與 40B rows。

第三個是利用 Vindexes 這個技術降低 sharding 時所帶來的限制。這個之前沒研究過:

A Vindex provides a way to map a column value to a keyspace ID.

從「Older Version Docs」這邊翻舊版的文件,發現 5.0+ 都有,再往 GitHub 上面的資料翻,看起來從 2016 年的版本就有了,不過當時看起來還一直在擴充:「Vitess v2.0.0-rc.1」。

回來看現在的功能,有 primary vindex 的設計:

The Primary Vindex for a table is analogous to a database primary key. Every sharded table must have one defined. A Primary Vindex must be unique: given an input value, it must produce a single keyspace ID.

然後是 secondary vindex(es) 的設計,指到 keyspace id(s),然後這個資訊會被用在 routing 上:

Secondary Vindexes are additional vindexes against other columns of a table offering optimizations for WHERE clauses that do not use the Primary Vindex. Secondary Vindexes return a single or a limited set of keyspace IDs which will allow VTGate to only target shards where the relevant data is present. In the absence of a Secondary Vindex, VTGate would have to send the query to all shards (called a scatter query).

It is important to note that Secondary Vindexes are only used for making routing decisions. The underlying database shards will most likely need traditional indexes on those same columns, to allow efficient retrieval from the table on the underlying MySQL instances.

然後是 functional vindex 與 lookup vindex,前者用演算法定義 keyspace id,後者讓你查:

A Functional Vindex is a vindex where the column value to keyspace ID mapping is pre-established, typically through an algorithmic function. In contrast, a Lookup Vindex is a vindex that provides the ability to create an association between a value and a keyspace ID, and recall it later when needed. Lookup Vindexes are sometimes also informally referred to as cross-shard indexes.

然後 lookup vindex 還有對 consistent hashing 的支援:

Consistent lookup vindexes use an alternate approach that makes use of careful locking and transaction sequences to guarantee consistency without using 2PC. This gives the best of both worlds, with the benefit of a consistent cross-shard vindex without paying the price of 2PC. To read more about what makes a consistent lookup vindex different from a standard lookup vindex read our consistent lookup vindexes design documentation.

這樣整體看起來,Vitess 把所有常見的 sharding 方式都包進去了,如果以後真的遇到這個量的話,也不需要自己在 application 或是 library 做一堆事情了...