Kafka 拔掉 ZooKeeper 的計畫

目前 Kafka cluster 還是會需要透過 ZooKeeper 處理不少資料,但眾所皆知的,ZooKeeper 實在是不好維護,所以 Kafka 官方從好幾年前就一直在想辦法移除對 ZooKeeper 的相依性。

這篇算是其中一塊:「Kafka Needs No Keeper」。

真的自己架過 Kafka cluster 就會知道其中的 ZooKeeper 很不好維護,尤其是 Apache 官方版本的軟體與文件常常脫勾,設定起來就很痛苦。所以一般都會用 Confluent 出的包裝,裡面的 ZooKeeper 軟體與 Confluent 自己寫的文件至少都被測過,不太會遇到官方文件與軟體之間搭不上的問題。

另外一個常見的痛點是,因為 Kafka 推動拔掉 ZooKeeper 的計畫推很久了 (好幾年了),但進展不快,所以有時候會發現在 command line 下,有些指令會把 API endpoint 指到 ZooKeeper 伺服器上,但有些指令卻又指到 Kafka broker 上,這點一直在邏輯上困擾很久,直到看到官方的拔除計畫 (但又不快) 才理解為什麼這麼不一致...

給需要的人參考,當初在架設 Kafka cluster 時寫下來的筆記:「Confluent」。

Amazon MQ 在 ap-northeast-{1,2} 推出了...

Amazon MQap-northeast-{1,2} 推出了,先前自己架的,現在可以直接拿現成的服務了:「Amazon MQ is Now Available in the Asia Pacific (Seoul) and Asia Pacific (Tokyo) regions」。

不過 AWS 上的開發主要還是以 SQS 之類的服務為主,可以避免 scalability 的問題 (另外一種可能是一開始就打定要搬出來,所以選擇 open protocol 的方案)。在這樣的前提下,Amazon MQ 的定位就變成將現有軟體丟上去跑... (而不想自己管 XD)

SQS 可以打進 Lambda 了...

在昨天的 AWS 台北高峰會上,AWS 的人有提到這個功能應該要正式推出了,果然在回來不久後就看到消息了:「AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources」。

We can now use Amazon Simple Queue Service (SQS) to trigger AWS Lambda functions! This is a stellar update with some key functionality that I’ve personally been looking forward to for more than 4 years. I know our customers are excited to take it for a spin so feel free to skip to the walk through section below if you don’t want a trip down memory lane.

這算是 Serverless 架構下很自然會想要做的一環,當 SQS 裡面有東西的時候就呼叫 Lambda 起來做事,以往一般會透過 SNS 在中間接起來 (或是拿 S3 惡搞,因為 S3 也可以串 Lambda...),現在可以直接串了:

By adding support for SQS to Lambda we’re removing a lot of the undifferentiated heavy lifting of running a polling service or creating an SQS to SNS mapping.

這個功能本身不收費,但他需要的 SQS API call 與產生的 Lambda 當然是需要收費的:

There are no additional charges for this feature, but because the Lambda service is continuously long-polling the SQS queue the account will be charged for those API calls at the standard SQS pricing rates.

AWS 推出用 ActiveMQ 架設的服務,Amazon MQ

AWSActiveMQ 包起來賣服務:「Amazon MQ – Managed Message Broker Service for ActiveMQ」。

在 AWS 上已經有 Amazon SQS 這類服務的情況下,應該還是因為 ActiveMQ 的生態更豐富,所以決定支援 ActiveMQ... 光是支援的通訊協定就比自家多很多,有很多應用可以直接接上去:

With Amazon MQ, you get direct access to the ActiveMQ console and industry standard APIs and protocols for messaging, including JMS, NMS, AMQP, STOMP, MQTT, and WebSocket.

不過支援的地區還是有限:

Amazon MQ is available now and you can start using it today in the US East (Northern Virginia), US East (Ohio), US West (Oregon), EU (Ireland), EU (Frankfurt), and Asia Pacific (Sydney) Regions.

另外這個服務有提供 free tier,可以讓使用者測試:

The AWS Free Tier lets you use a single-AZ micro instance for up to 750 hours and to store up to 1 gigabyte each month, for one year. After that, billing is based on instance-hours and message storage, plus charges Internet data transfer if the broker is accessed from outside of AWS.

Amazon SQS 支援 FIFO 了

Amazon SQS 支援 FIFO 了:「FIFO (First-In-First-Out) Queues」。新的 FIFO Queue 有保證順序,但也因此效能上有限制:

In addition to having all the capabilities of the standard queue, FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated. FIFO queues also provide exactly-once processing but are limited to 300 transactions per second (TPS).

可以看到舊版的 FAQ 對於 FIFO 的回答是 Standard Queue 會盡力做到 FIFO,但不保證:(出自 2016/08/26 的版本)

Q: Does Amazon SQS provide first-in-first-out (FIFO) access to messages?

Amazon SQS provides a loose-FIFO capability that attempts to preserve the order of messages. However, we have designed Amazon SQS to be massively scalable using a distributed architecture. Thus, we can't guarantee that you will always receive messages in the exact order you sent them (FIFO).

If your system requires the order of messages to be preserved, place sequencing information in each message so that messages can be ordered when they are received.

而現在則是名正言順的說有提供 FIFO 了:

Q: Does Amazon SQS provide message ordering?

Yes. FIFO (first-in-first-out) queues preserve the exact order in which messages are sent and received. If you use a FIFO queue, you don't have to place sequencing information in your messages. For more information, see FIFO Queue Logic in the Amazon SQS Developer Guide.

Standard queues provide a loose-FIFO capability that attempts to preserve the order of messages. However, because standard queues are designed to be massively scalable using a highly distributed architecture, receiving messages in the exact order they are sent is not guaranteed.

Netflix 開發的 Delayed Queue

原來這個叫做 Delayed Queue,難怪之前用其他關鍵字都找不到什麼資料... (就不講其他關鍵字了 XD)

Netflix 發表了他們自己所開發的 Delayed Queue:「Distributed delay queues based on Dynomite」。

本來的架構是用 Cassandra + Zookeeper 來做:

Traditionally, we have been using a Cassandra based queue recipe along with Zookeeper for distributed locks, since Cassandra is the de facto storage engine at Netflix.

但可以馬上想到不少問題,就如同 Netflix 提到的:

Using Cassandra for queue like data structure is a known anti-pattern, also using a global lock on queue while polling, limits the amount of concurrency on the consumer side as the lock ensures only one consumer can poll from the queue at a time.

所以就改放到 Netflix 另外開發的 Dynamite 上:

Dynomite, inspired by Dynamo whitepaper, is a thin, distributed dynamo layer for different storage engines and protocols. Currently these include Redis and Memcached. Dynomite supports multi-datacenter replication and is designed for high availability.

後端是 RedisMemcached 的系統,可以對抗整個機房從 internet 上消失的狀態。

在設計上則是「保證會跑一次」,也就是有可能會有多次的情況,用 Dyno Queues 系統的人必需要考慮進去:

4. At-least-once delivery semantics

雖然整篇講的頗輕鬆,但實際看起來還是很厚重... 暫時還是不會用吧 :o

對各類 Message Queue 的效能測試

在「Benchmarking Message Queue Latency」這篇看到作者測了一輪 Message Queue 軟體:

RabbitMQ (3.6.0), Kafka (0.8.2.2 and 0.9.0.0), Redis (2.8.4) pub/sub, and NATS (0.7.3)

測試包括了從一個 9 到六個 9 的 latency (i.e. 90%、99%、99.9%、99.99%、99.999%、99.9999%),另外也測了 message 大小帶來的效能差異。

99.9% 表示 1/1000,而 99.99% 表示 1/10000,如果差距跟 90% 很大,表示系統反應時間會很不一致。另外有些 Message Queue 軟體有 disk persistence 的功能,也因為寫入資料,會看到更大的差距。

善用或是避開這些特性去規劃才能減少問題,像是關掉 disk persistence 之類的方法。