這是 Jespen 第一次測試 PostgreSQL,就順利找出可重製的 bug 了:「PostgreSQL 12.3」。

第一個 bug 是 REPEATABLE READ 下的問題,不過因為 SQL-92 定義不夠嚴謹的關係,其實算不算是 bug 有討論的空間,這點作者 Kyle Kingsbury 在文章裡也有提出來:

Whether PostgreSQL’s repeatable-read behavior is correct therefore depends on one’s interpretation of the standard. It is surprising that a database based on snapshot isolation would reject the strict interpretation chosen by the seminal paper on SI, but on reflection, the behavior is defensible.

另外一個就比較沒問題了,是 SERIALIZABLE 下的 bug,在 SQL-92 下對 SERIALIZABLE 的定義是這樣:

The execution of concurrent SQL-transactions at isolation level SERIALIZABLE is guaranteed to be serializable. A serializable execution is defined to be an execution of the operations of concurrently executing SQL-transactions that produces the same effect as some serial execution of those same SQL-transactions. A serial execution is one in which each SQL-transaction executes to completion before the next SQL-transaction begins.

也就是說,在 SERIALIZABLE 下一堆 transaction 的執行結果,你至少可以找到一組排序,使得這些 transaction 的結果是等價的。

而 Jespen 順利找出了一組 transaction (兩個 transaction),在 SERIALIZABLE 下都成功 (但不應該成功):

對於這兩個 transaction,不論是上面這條先執行,還是下面這條先執行,都不存在等價的結果,所以不符合 SERIALIZABLE 的要求。

另外也找到一個包括三個 transaction 的情況:

把 transaction 依照執行的結果把 dependency 拉出來,就可以看出來裡面產生了 loop,代表不可能在 SERIALIZABLE 下三個都成功。

在 Jespen 找到這些 bug 後,PostgreSQL 方面也找到軟體內產生 bug 的部份,並且修正了:「Avoid update conflict out serialization anomalies.」,看起來是在 PostgreSQL 引入 Serializable Snapshot Isolation (SSI) 的時候就有這個 bug,所以 9.1 以後的版本都有這個問題...

這次順利打下來,測得很漂亮啊... 翻了一下 Jespen 上的記錄,發現好像還沒測過 MySQL,應該會是後續的目標?

MongoDB 的欺騙性廣告

Jepsen 最近丟出了一篇新的測試報告在測新版的 MongoDB 4.2.6,而且語氣看起來比以前兇很多,翻了一下前因後果,看起來起因是出自 Twitter 上的這則推,提到了 MongoDB 拿 Jepsen 宣傳的頁面:

然後 Jepsen 的官方帳號這邊也回應,覺得不可置信:

過兩個禮拜後 Jepsen 就丟出由老大 Kyle Kingsbury 發表的「Jepsen: MongoDB 4.2.6」,這篇測試 MongoDB 4.2.6 最新版的測試報告了。

在這篇報告裡面提到了很多不道德的行為,首先是在之前的測試發現有很多會掉資料的問題,但在 MongoDB 官方的宣傳文件「MongoDB and Jepsen」裡面則是完全沒提到,而且還宣稱有業界最強的資料一致性與正確性 (與 Jepsen 報告所提供的資料不符),所以 Jepsen 建議把這些問題列到這個頁面上,以避免使用者受到「誤解」:

Curiously, MongoDB omitted any mention of these findings in their MongoDB and Jepsen page. Instead, that page discusses only passing results, makes no mention of read or write concern, buries the actual report in a footnote, and goes on to claim:

MongoDB offers among the strongest data consistency, correctness, and safety guarantees of any database available today.

We encourage MongoDB to report Jepsen findings in context: while MongoDB did appear to offer per-document linearizability and causal consistency with the strongest settings, it also failed to offer those properties in most configurations. We think users might want to be aware that their database could lose data by default, but MongoDB’s summary of our work omits any mention of this behavior.

另外當然就是重測 MongoDB 4.2.6 版,沒時間看內容的人可以先瞄一下標題,裡面就已經點出不少東西了:

3 Results
3.1 Sometimes, Programs That Use Transactions… Are Worse
3.2 How ACID is Snapshot Isolation, Anyway
3.3 Indeterminate Errors
3.4 Duplicate Effects
3.5 Read Skew
3.6 Cyclic Information Flow
3.7 Read Your (Future) Writes

不過在最後面的 Discussion 比較清楚。

首先是批評 snapshot isolation 不是 ACID:

MongoDB 4.2.6 claims to offer “full ACID transactions” via snapshot isolation. However, the use of these transactions is complicated by weak defaults, confusing APIs, and undocumented error codes. Snapshot isolation is questionably compatible with the marketing phrase “full ACID”. Even at the highest levels of read and write concern, MongoDB’s transaction mechanism exhibited various anomalies which violate snapshot isolation.

Snapshot isolation is a reasonably strong consistency model, but claiming that snapshot isolation is “full ACID” is questionable.

而且即使把所有的資料安全性相關的設定都調到最高,也根本就做不到宣稱的 snapshot isolation:

Finally, even with the strongest levels of read and write concern for both single-document and transactional operations, we observed cases of G-single (read skew), G1c (cyclic information flow), duplicated writes, and a sort of retrocausal internal consistency anomaly: within a single transaction, reads could observe that transaction’s own writes from the future. MongoDB appears to allow transactions to both observe and not observe prior transactions, and to observe one another’s writes. A single write could be applied multiple times, suggesting an error in MongoDB’s automatic retry mechanism. All of these behaviors are incompatible with MongoDB’s claims of snapshot isolation.

過程中也發現就算設定了 snapshot 層級,MongoDB 在讀取時也不會遵守 snapshot isolation:

MongoDB’s default read and write concern for single-document operations remains local, which can observe uncommitted data, and w: 1, which can lose committed writes. Even when users select safer settings in their clients at the database or collection level, transactions ignore these settings and default again to local and w: 1. The snapshot read concern does not actually guarantee snapshot isolation, and must always be used in conjunction with write concern majority. This holds even for transactions which perform no writes.

然後所有的官方文件都沒有教 snapshot isolation 要怎麼設定,你必須在第三方的文件上才有機會找到:

Nor can users rely on examples to demonstrate snapshot isolated behavior. MongoDB’s transaction documentation and tutorial blog posts show only write-only transactions, using read concern local rather than snapshot. Other examples from MongoDB don’t specify a read concern or run entirely with defaults. Learn MongoDB The Hard Way uses read concern snapshot but write concern local, despite performing writes. Tutorials from DZone, Several Nines, Percona, The Code Barbarian, and all claim that transactions are either ACID or offer snapshot isolation, but none set either read or write concern. There are some examples of MongoDB transactions which are snapshot isolated—for instance, from BMC, +N Consulting, and Maciej Zgadzaj, but most uses of MongoDB transactions we found ran—either intentionally or inadvertently—with settings that would (in general) allow write loss and aborted reads.


Amazon 的 Elasticsearch 服務提供十四天免費 hourly snapshot

Amazon Elasticsearch Service 提供 14 天免費的 hourly snapshot:「Amazon Elasticsearch Service increases data protection with automated hourly snapshots at no extra charge」。

Amazon Elasticsearch Service has increased its snapshot frequency from daily to hourly, providing more granular recovery points. If you need to restore your cluster, you now have numerous, recent snapshots to choose from. These automated snapshots are retained for 14 days at no extra charge.

不過這是 5.3+ 版本才有,舊版只有 daily:

  • For domains running Elasticsearch 5.3 and later, Amazon ES takes hourly automated snapshots and retains up to 336 of them for 14 days.
  • For domains running Elasticsearch 5.1 and earlier, Amazon ES takes daily automated snapshots (during the hour you specify) and retains up to 14 of them for 30 days.

In both cases, the service stores the snapshots in a preconfigured Amazon S3 bucket at no additional charge. You can use these automated snapshots to restore domains.


EC2 簡化了 Hibernation 的需求

因為在記憶體內會有各種敏感資訊,所以 EC2 的 Hibernation 當初推出時要求在寫到 snapshot 時要有加密,而 EC2 的設計需要使用 encrypted AMI 啟動,才能產生 encrypted snapshot:

Hibernation requires an EC2 instance be an encrypted EBS-backed instance. This ensures protection of sensitive contents in memory (RAM) as they get copied to the EBS upon hibernation.

這點對一般人來說就比較麻煩了,因為 AMI 裡面可能沒有敏感資訊,所以當初都是設計成 unencrypted 狀態,變成要多一些步驟...

而現在 EC2 宣佈可以可以用一般的 AMI 啟動並且產生出加密的 snapshot:「Enable Hibernation on EC2 Instances when launching with an AMI without an Encrypted EBS Snapshot」。


AWS 給 EBS 用的 Data Lifecycle Manager 在東京可以用了?

先前在「Amazon EBS Snapshot 支援 Lifecycle Management」這邊提到 AWS 設計了 Data Lifecycle Manager,讓 EBS 磁碟可以自動產生 snapshot 並且管理保留份數,可以當作某種備份機制。


Availability – Data Lifecycle Manager is available in the US East (N. Virginia), US West (Oregon), and Europe (Ireland) Regions.

剛剛發現在東京也已經可以用了?但好像沒看到有公告提過... 設下去看看會不會動好了。

AWS 提供將 Lightsail 切換到 EC2 的功能

AWS 總算把 Lightsail 轉移到 EC2 的功能做出來了:「Amazon Lightsail Now Provides an Upgrade Path to EC2」。

這樣先從小站開始跑,搞大後想要改用 AWS 服務的切換成本就降低不少了。不過目前看起來是透過 export-and-import 做:

When you are ready to upgrade an instance to EC2, simply take a snapshot of your instance and follow the step-by-step process in Lightsail's console to export your snapshot to EC2. You can then use EC2 or the Upgrade to EC2 wizard in Lightsail's console to get your new EC2 instance up and running.

不過上次用 Lightsail 可以發現可用的 CPU 跟其他 VPS 比起來慢不少... 這點是一開始要不要選 Lightsail 的因素。

Amazon EBS Snapshot 支援 Lifecycle Management

以往用 Amazon EBS Snapshot 需要用 Lambda 當 cron job 建立 snapshot,以及管理要存的數量 (要刪掉舊的),現在 AWS 直接提供服務幫你處理:「New – Lifecycle Management for Amazon EBS Snapshots」。文章的截圖就說明了這個新功能:


You can create and use Data Lifecyle Manager policies at no charge; you pay the usual storage charges for the EBS snapshots that it creates.

Data Lifecycle Manager is available in the US East (N. Virginia), US West (Oregon), and EU (Ireland) Regions.

先繼續用 Lambda,等東京有的時候再換過去...

Amazon Aurora 支援快速複製

Amazon Aurora 宣佈支援快速複製:「Amazon Aurora Fast Database Cloning」。

對於 2TB 的資料大約五分鐘就完成了:

This means my 2TB snapshot restore job that used to take an hour is now ready in about 5 minutes – and most of that time is spent provisioning a new RDS instance.

主要是得力於後端 storage 的部份可以實做 copy-on-write 架構:

By taking advantage of Aurora’s underlying distributed storage engine you’re able to quickly and cheaply create a copy-on-write clone of your database.

可以快速複製就可以很快的驗證一些事情,像是可以直接測試 ALTER TABLE 需要的時間,或是事前演練...

V8 對 Hash Flooding 的防禦措施

Hash Flooding 問題是指 Hash 這個資料結構是可以被預測 collision 所造成的問題,在隨機的情況下會是 O(1) 的操作,在特定挑選故意讓他 collision 而變成 O(n),當有 n 個元素時,乘起來就會變成 O(n^2)。這算是一種阻斷攻擊 (DoS attack)。

在「About that hash flooding vulnerability in Node.js...」這邊提到了 V8 之前為了避免 Hash Flooding 的問題,關掉了 Startup snapshot 而造成的效能問題,以及後續的很多故事,最後找了長期的解法。

這個解法已經併入 Node.js 裡,預定下個包括的版本是 8.3.0:

The patch to re-enable startup snapshot has been merged into Node.js. It is part of the recent Node.js v8.3.0 release.

不過這表示現有的 LTS (4.8.4 以及 6.11.2) 還是... XD

DigitalOcean 開始加收 Snapshot 費用

收到 DigitalOcean 的信件通知,snapshot 會開始收費:

Starting October 1, 2016, we will begin charging for snapshot storage at $0.05 per gigabyte per month. This will first be reflected in the invoice posted to your account on November 1, 2016. Like other features, snapshot storage uses hourly pricing, and size is calculated from a compressed version of the snapshot—not the total disk space allocated to the Droplet.