Netflix 開發的 Delayed Queue

原來這個叫做 Delayed Queue,難怪之前用其他關鍵字都找不到什麼資料... (就不講其他關鍵字了 XD)

Netflix 發表了他們自己所開發的 Delayed Queue:「Distributed delay queues based on Dynomite」。

本來的架構是用 Cassandra + Zookeeper 來做:

Traditionally, we have been using a Cassandra based queue recipe along with Zookeeper for distributed locks, since Cassandra is the de facto storage engine at Netflix.

但可以馬上想到不少問題,就如同 Netflix 提到的:

Using Cassandra for queue like data structure is a known anti-pattern, also using a global lock on queue while polling, limits the amount of concurrency on the consumer side as the lock ensures only one consumer can poll from the queue at a time.

所以就改放到 Netflix 另外開發的 Dynamite 上:

Dynomite, inspired by Dynamo whitepaper, is a thin, distributed dynamo layer for different storage engines and protocols. Currently these include Redis and Memcached. Dynomite supports multi-datacenter replication and is designed for high availability.

後端是 RedisMemcached 的系統,可以對抗整個機房從 internet 上消失的狀態。

在設計上則是「保證會跑一次」,也就是有可能會有多次的情況,用 Dyno Queues 系統的人必需要考慮進去:

4. At-least-once delivery semantics

雖然整篇講的頗輕鬆,但實際看起來還是很厚重... 暫時還是不會用吧 :o

Amazon EC2 Spot Instance 的關機通知

AWS 改善了 EC2 Spot Instance 的關機通知,在決定 terminate 前兩分鐘 instance 可以得知消息,讓 instance 可以處理「後事」,像是將計算的資料寫回 EBS 或是 S3:「New – EC2 Spot Instance Termination Notices」。

Today we are improving the reclamation process with the addition of a two-minute warning, formally known as a Spot Instance Termination Notice.

不過目前不是 trigger,而是透過 polling 的方式提供:

The Termination Notice is accessible to code running on the instance via the instance’s metadata at This field will become available when the instance has been marked for termination (step 3, above), and will contain the time when a shutdown signal will be sent to the instance’s operating system.

建議的 polling 時間是五秒鐘一次:

We recommend that interested applications poll for the termination notice at five-second intervals.