Netflix 開發的 Delayed Queue

原來這個叫做 Delayed Queue,難怪之前用其他關鍵字都找不到什麼資料... (就不講其他關鍵字了 XD)

Netflix 發表了他們自己所開發的 Delayed Queue:「Distributed delay queues based on Dynomite」。

本來的架構是用 Cassandra + Zookeeper 來做:

Traditionally, we have been using a Cassandra based queue recipe along with Zookeeper for distributed locks, since Cassandra is the de facto storage engine at Netflix.

但可以馬上想到不少問題,就如同 Netflix 提到的:

Using Cassandra for queue like data structure is a known anti-pattern, also using a global lock on queue while polling, limits the amount of concurrency on the consumer side as the lock ensures only one consumer can poll from the queue at a time.

所以就改放到 Netflix 另外開發的 Dynamite 上:

Dynomite, inspired by Dynamo whitepaper, is a thin, distributed dynamo layer for different storage engines and protocols. Currently these include Redis and Memcached. Dynomite supports multi-datacenter replication and is designed for high availability.

後端是 RedisMemcached 的系統,可以對抗整個機房從 internet 上消失的狀態。

在設計上則是「保證會跑一次」,也就是有可能會有多次的情況,用 Dyno Queues 系統的人必需要考慮進去:

4. At-least-once delivery semantics

雖然整篇講的頗輕鬆,但實際看起來還是很厚重... 暫時還是不會用吧 :o

Amazon EC2 Spot Instance 的關機通知

AWS 改善了 EC2 Spot Instance 的關機通知,在決定 terminate 前兩分鐘 instance 可以得知消息,讓 instance 可以處理「後事」,像是將計算的資料寫回 EBS 或是 S3:「New – EC2 Spot Instance Termination Notices」。

Today we are improving the reclamation process with the addition of a two-minute warning, formally known as a Spot Instance Termination Notice.

不過目前不是 trigger,而是透過 polling 的方式提供:

The Termination Notice is accessible to code running on the instance via the instance’s metadata at http://169.254.169.254/latest/meta-data/spot/termination-time. This field will become available when the instance has been marked for termination (step 3, above), and will contain the time when a shutdown signal will be sent to the instance’s operating system.

建議的 polling 時間是五秒鐘一次:

We recommend that interested applications poll for the termination notice at five-second intervals.

至少有方法得知了...