Netflix 開發的 Delayed Queue

原來這個叫做 Delayed Queue,難怪之前用其他關鍵字都找不到什麼資料... (就不講其他關鍵字了 XD)

Netflix 發表了他們自己所開發的 Delayed Queue:「Distributed delay queues based on Dynomite」。

本來的架構是用 Cassandra + Zookeeper 來做:

Traditionally, we have been using a Cassandra based queue recipe along with Zookeeper for distributed locks, since Cassandra is the de facto storage engine at Netflix.

但可以馬上想到不少問題,就如同 Netflix 提到的:

Using Cassandra for queue like data structure is a known anti-pattern, also using a global lock on queue while polling, limits the amount of concurrency on the consumer side as the lock ensures only one consumer can poll from the queue at a time.

所以就改放到 Netflix 另外開發的 Dynamite 上:

Dynomite, inspired by Dynamo whitepaper, is a thin, distributed dynamo layer for different storage engines and protocols. Currently these include Redis and Memcached. Dynomite supports multi-datacenter replication and is designed for high availability.

後端是 RedisMemcached 的系統,可以對抗整個機房從 internet 上消失的狀態。

在設計上則是「保證會跑一次」,也就是有可能會有多次的情況,用 Dyno Queues 系統的人必需要考慮進去:

4. At-least-once delivery semantics

雖然整篇講的頗輕鬆,但實際看起來還是很厚重... 暫時還是不會用吧 :o

This entry was posted in Computer, Murmuring, Network, Programming, Software and tagged , , , , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *