Home » Posts tagged "queue"

SQS 可以打進 Lambda 了...

在昨天的 AWS 台北高峰會上,AWS 的人有提到這個功能應該要正式推出了,果然在回來不久後就看到消息了:「AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources」。

We can now use Amazon Simple Queue Service (SQS) to trigger AWS Lambda functions! This is a stellar update with some key functionality that I’ve personally been looking forward to for more than 4 years. I know our customers are excited to take it for a spin so feel free to skip to the walk through section below if you don’t want a trip down memory lane.

這算是 Serverless 架構下很自然會想要做的一環,當 SQS 裡面有東西的時候就呼叫 Lambda 起來做事,以往一般會透過 SNS 在中間接起來 (或是拿 S3 惡搞,因為 S3 也可以串 Lambda...),現在可以直接串了:

By adding support for SQS to Lambda we’re removing a lot of the undifferentiated heavy lifting of running a polling service or creating an SQS to SNS mapping.

這個功能本身不收費,但他需要的 SQS API call 與產生的 Lambda 當然是需要收費的:

There are no additional charges for this feature, but because the Lambda service is continuously long-polling the SQS queue the account will be charged for those API calls at the standard SQS pricing rates.

AWS 推出用 ActiveMQ 架設的服務,Amazon MQ

AWSActiveMQ 包起來賣服務:「Amazon MQ – Managed Message Broker Service for ActiveMQ」。

在 AWS 上已經有 Amazon SQS 這類服務的情況下,應該還是因為 ActiveMQ 的生態更豐富,所以決定支援 ActiveMQ... 光是支援的通訊協定就比自家多很多,有很多應用可以直接接上去:

With Amazon MQ, you get direct access to the ActiveMQ console and industry standard APIs and protocols for messaging, including JMS, NMS, AMQP, STOMP, MQTT, and WebSocket.


Amazon MQ is available now and you can start using it today in the US East (Northern Virginia), US East (Ohio), US West (Oregon), EU (Ireland), EU (Frankfurt), and Asia Pacific (Sydney) Regions.

另外這個服務有提供 free tier,可以讓使用者測試:

The AWS Free Tier lets you use a single-AZ micro instance for up to 750 hours and to store up to 1 gigabyte each month, for one year. After that, billing is based on instance-hours and message storage, plus charges Internet data transfer if the broker is accessed from outside of AWS.

Amazon SQS 支援 FIFO 了

Amazon SQS 支援 FIFO 了:「FIFO (First-In-First-Out) Queues」。新的 FIFO Queue 有保證順序,但也因此效能上有限制:

In addition to having all the capabilities of the standard queue, FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated. FIFO queues also provide exactly-once processing but are limited to 300 transactions per second (TPS).

可以看到舊版的 FAQ 對於 FIFO 的回答是 Standard Queue 會盡力做到 FIFO,但不保證:(出自 2016/08/26 的版本)

Q: Does Amazon SQS provide first-in-first-out (FIFO) access to messages?

Amazon SQS provides a loose-FIFO capability that attempts to preserve the order of messages. However, we have designed Amazon SQS to be massively scalable using a distributed architecture. Thus, we can't guarantee that you will always receive messages in the exact order you sent them (FIFO).

If your system requires the order of messages to be preserved, place sequencing information in each message so that messages can be ordered when they are received.

而現在則是名正言順的說有提供 FIFO 了:

Q: Does Amazon SQS provide message ordering?

Yes. FIFO (first-in-first-out) queues preserve the exact order in which messages are sent and received. If you use a FIFO queue, you don't have to place sequencing information in your messages. For more information, see FIFO Queue Logic in the Amazon SQS Developer Guide.

Standard queues provide a loose-FIFO capability that attempts to preserve the order of messages. However, because standard queues are designed to be massively scalable using a highly distributed architecture, receiving messages in the exact order they are sent is not guaranteed.

Netflix 開發的 Delayed Queue

原來這個叫做 Delayed Queue,難怪之前用其他關鍵字都找不到什麼資料... (就不講其他關鍵字了 XD)

Netflix 發表了他們自己所開發的 Delayed Queue:「Distributed delay queues based on Dynomite」。

本來的架構是用 Cassandra + Zookeeper 來做:

Traditionally, we have been using a Cassandra based queue recipe along with Zookeeper for distributed locks, since Cassandra is the de facto storage engine at Netflix.

但可以馬上想到不少問題,就如同 Netflix 提到的:

Using Cassandra for queue like data structure is a known anti-pattern, also using a global lock on queue while polling, limits the amount of concurrency on the consumer side as the lock ensures only one consumer can poll from the queue at a time.

所以就改放到 Netflix 另外開發的 Dynamite 上:

Dynomite, inspired by Dynamo whitepaper, is a thin, distributed dynamo layer for different storage engines and protocols. Currently these include Redis and Memcached. Dynomite supports multi-datacenter replication and is designed for high availability.

後端是 RedisMemcached 的系統,可以對抗整個機房從 internet 上消失的狀態。

在設計上則是「保證會跑一次」,也就是有可能會有多次的情況,用 Dyno Queues 系統的人必需要考慮進去:

4. At-least-once delivery semantics

雖然整篇講的頗輕鬆,但實際看起來還是很厚重... 暫時還是不會用吧 :o

對各類 Message Queue 的效能測試

在「Benchmarking Message Queue Latency」這篇看到作者測了一輪 Message Queue 軟體:

RabbitMQ (3.6.0), Kafka ( and, Redis (2.8.4) pub/sub, and NATS (0.7.3)

測試包括了從一個 9 到六個 9 的 latency (i.e. 90%、99%、99.9%、99.99%、99.999%、99.9999%),另外也測了 message 大小帶來的效能差異。

99.9% 表示 1/1000,而 99.99% 表示 1/10000,如果差距跟 90% 很大,表示系統反應時間會很不一致。另外有些 Message Queue 軟體有 disk persistence 的功能,也因為寫入資料,會看到更大的差距。

善用或是避開這些特性去規劃才能減少問題,像是關掉 disk persistence 之類的方法。