分散式系統的 clock

前幾天在 Hacker News 上看到「Clocks and Causality – Ordering Events in Distributed Systems (2022) (exhypothesi.com)」這篇,講分散式系統上 clock 的設計,作者也有跑出來在 Hacker News 上面跟大家聊一下 (帳號是 thoughtlede),原文在「Clocks and Causality - Ordering Events in Distributed Systems」這邊。

文章裡面主要講空間是 O(1)Lamport timestamp 與空間是 O(n)Vector clock (這邊的 n 相對於節點數量),以及這兩個對應的擴充版本:

作者會整理這些資料的原因是因為在研究 CRDT 的時候看到演算法中常常會需要處理分散式系統裡面事件的順序,所以花了一些時間整理常見的方式:

Author here. Pleasantly surprised to see the article here.

Some context behind the article. I studied CRDTs for a few months, and noticed that different CRDT designs use logical clocks in different and clever ways. And I haven't seen anyone narrate all those ways of use in one article. My attempt with this article was to dredge up those flavors of logical clocks into one article and give them names for future reference.

(To respond to a couple of other comments, I ignored atomic (and gps-based) clocks in this discussion, as indicated in my footnote 3).

我記得還有一個 Interval Tree Clocks 可以參考 (在「Interval Tree Clocks」這邊講的比較清楚),是針對節點的動態增刪而改善的演算法,但不確定有什麼比較有名的系統有用。

大多數應該都是用 Vector clock,畢竟是在 2007 年的「Dynamo: Amazon’s Highly Available Key-value Store」被發揚光大,而且也算是還不錯的演算法?

在 CSS 裡面使用系統字型的選擇

一開始是看到「Show HN: Modern Font Stacks – New system font stack CSS for modern OSs (modernfontstacks.com)」這個,專案希望可以使用不用另外下載的字型 (系統內建字型),但這個網站本身不好用,留言區提到的一些其他網站好用多了...

首先是「CSS Font Stack: Web Safe and Web Font Family with HTML and CSS code.」這個,把 WindowsmacOS 上預設的字型都列出來,然後也列出對應的安裝比率。

而另外一個則是在 GitHub 上的專案「Modern Font Stacks」,把類似的字型放在一起。

看起來可以參考,比起只用 serif 或是 sans-serif 來說會多一些變化可以用。

GitHub 自己開發的搜尋引擎

前陣子 GitHub 發了一篇文章,說明自己開發搜尋引擎的心路歷程:「The technology behind GitHub’s new code search」。

看了一下其實就是自己幹了一套 search engine cluster,然後針對 code search 把一些功能放進去。

目前這套 search enginer 還是 beta 版本,全站兩億個 repository 只包括了 4500 萬 (大概 22% 左右),然後已經有 115TB 的程式碼了;另外也題到了先前導入 Elasticsearch 時的數字是 800 萬個 repository:

GitHub’s scale is truly a unique challenge. When we first deployed Elasticsearch, it took months to index all of the code on GitHub (about 8 million repositories at the time). Today, that number is north of 200 million, and that code isn’t static: it’s constantly changing and that’s quite challenging for search engines to handle. For the beta, you can currently search almost 45 million repositories, representing 115 TB of code and 15.5 billion documents.

目前是 32 台機器,沒有特別提到記憶體大小,也沒有提到 replication 之類的數字:

Code search runs on 64 core, 32 machine clusters.

然後各種 inverted index 與各種資料在壓縮後只有 25TB:

There are some big wins on the size of the index as well. Remember that we started with 115 TB of content that we want to search. Content deduplication and delta indexing brings that down to around 28 TB of unique content. And the index itself clocks in at just 25 TB, which includes not only all the indices (including the ngrams), but also a compressed copy of all unique content. This means our total index size including the content is roughly a quarter the size of the original data!

換算一下,就會發現現在已經是「暴力」可以解很多事情的年代了,而這已經是全世界最大的 code hosting。

以前隨便一個主題搞大一點就會撞到 Amdahl's law,現在輕鬆不少...

Twitter 的 2022 年架構 (?)

Elon Musk 自己貼了以後,有人把白板圖整理成數位圖後好讀一些:

不過心裡可以先有底,這是解釋給大老闆聽的架構圖,底層可能還會有不少東西是沒有在這上面的,尤其是其他重要的子系統,像是圖片與影音怎麼處理之類的。

先把圖拉出來看原圖大小 (可以自己點),比較清楚一點:

從最前端往後看,可以看到 TLS-API 被標成淘汰中,照圖上寫的剩下 Android 版本在用,後續應該是往 GraphQL 靠;另外內部有使用到 Thrift,但沒有標 gRPC,我猜是這邊的主線上沒有 gRPC 而已,其他系統可能還是會有。

另外是有些東西有標 next-gen system,應該就是正在做而還沒上線的東西。

Git 的「災難處理」

但印象中之前看過 (在 Internet Archive 上可以看到 4 Sep 2017 的版本),但搜尋 Hacker News 後發現沒有提過... 這幾天紅起來的「Dangit, Git!?!」,也有簡體中文版可以看。

裡面其實提到了很多要怎麼處理不小心塞錯資料進 Git 的情況,不過好像還是有些東西沒涵蓋到,像是遇到不小心塞到 credentials 進去後需要清除掉的 git rebase -i HASH,接著一連串的手動修 conflict 與 git rebase --continue,最後再接上 git push --force 這種禁招...

另外推一下「為你自己學 Git」這本書,裡面其實也有提到類似的情境:

第7章:修改歷史紀錄
7.1 狀況題 修改歷史訊息
7.2 狀況題 把多個 Commit 合併成一個Commit
7.3 狀況題 把一個 Commit 拆解成多個Commit
7.4 狀況題 想要在某些 Commit 之間再加新的Commit
7.5 狀況題 想要刪除某幾個 Commit 或是調整Commit 的順序
7.6 Reset、Revert 跟 Rebase 指令有什麼差別?

這本書也有網頁版,在 gitbook.tw 這邊。

觀察誰在存取剪貼簿的工具 (X11 下)

兩個月前在 Hacker News 上看到的討論,有人想要知道誰在 X11 下存取剪貼簿:「Who keeps an eye on clipboard access? (ovalerio.net)」,原文在「Who keeps an eye on clipboard access?」這邊,作者用 Python 寫的程式則是在「clipboard-watcher」這邊。

馬上有想到 iOS 在 2020 年推出的機制:「iOS 14 clipboard notifications are annoying, but developer adoption of a new API will improve the experience」。

不過在 X11 上跑起來會發現冒出來的資訊量有點大,像是在瀏覽器操作 WordPress 寫文章時剪剪貼貼的時候就會狂噴,如果可以提供程式的白名單的話就更好了,畢竟是我直接把 clipboard API 裡讀取的功能直接拔掉 (但網站還是可以寫進去就是了),對我來說不會在意 browser 寫進去的情況:

另外程式有時候會卡住 (尤其是遇到圖片的剪輯時),算是 bug 吧...

然後 Hacker News 的討論串裡面有人提到一個有趣的設計,他希望限制那些不在焦點上面的程式去碰 clipboard:

By far the worst offense I've seen in clipboard privacy on the Linux desktop is RedHat's virt-manager. It sends your clipboard AND selection content to all virtual machines, even when they are not focused, with no indication that it's happening, and with no GUI option to turn it off. This is at odds with the common practice of running untrusted code in virtual machines.

這個想法好像不賴,理論上 clipboard 應該是在有互動的時候才會碰到的東西...

Amazon EFS 效能提昇的一些討論

上一篇「Amazon EFS 的效能提昇」提到 Amazon EFS 的效能提昇,在 Hacker News 上看到 Amazon EFS 團隊的 PMT (Product-Manager-Technical) 出來回一些東西:「Amazon Elastic File System Update – Sub-Millisecond Read Latency (amazon.com)」,搜尋 geertj 應該就可以看到他回的東西了...

像是即使是 Jeff Barr 發表這篇文章,也還是經過 legal team 的同意才能發表:

(PMT on the EFS team).

Yes, the wordings are carefully formulated as they have to be signed off by the AWS legal team for obvious reasons. With that said, this update was driven by profiling real applications and addressing the most common operations, so the benefits are real. For example, a simple WordPress "hello world" is now about 2x as fast as before.

另外這次的效能提昇是透過 cache 層達成的:

I'm the PMT for this project in the EFS team. The "flip the switch" part was indeed one of the harder parts to get right. Happy to share some limited details. The performance improvement builds on a distributed consistent cache. You can enable such a cache in multiple steps. First you deploy the software across the entire stack that supports the caching protocol but it's disabled by configuration. Then you turn it for the multiple components that are involved in the right order. Another thing that was hard to get right was to ensure that there are no performance regressions due to the consistency protocol.

然後在每個 AZ 都有 cache:

The caches are local to each AZ so you get the low latency in each AZ, the other details are different. Unfortunately I can't share additional details at this moment, but we are looking to do a technical update on EFS at some point soon, maybe at a similar venue!

另外看起來主要就是 metadata cache 的幫助:

NFS workloads are typically metadata heavy and highly correlated in time, so you can achieve very high hit rates. I can't share any specific numbers unfortunately.

還是有很多細節數字不能透漏,但知道是透過 cache 達成的就已經可以大致上想像後面是怎麼弄出來的了...

Amazon EFS 的效能提昇

AWS 宣佈他們將 Amazon EFS 的 latency 大幅降低以提昇效能:「Amazon Elastic File System Update – Sub-Millisecond Read Latency」。

Linux 上一般是用 NFS 掛 EFS,個位數的 ms 的確對於效能影響超大,現在宣稱讀取的部份降到 0.6ms,應該會有蠻明顯的感覺:

Up until today, EFS latency for read operations (both data and metadata) was typically in the low single-digit milliseconds. Effective today, new and existing EFS file systems now provide average latency as low as 600 microseconds for the majority of read operations on data and metadata.

然後不另外收費:

This performance boost applies to One Zone and Standard General Purpose EFS file systems. New or old, you will still get the same availability, durability, scalability, and strong read-after-write consistency that you have come to expect from EFS, at no additional cost and with no configuration changes.

另外就是過去幾個禮拜他們把現有的 EFS 都轉移過去了:

We “flipped the switch” and enabled this performance boost for all existing EFS General Purpose mode file systems over the course of the last few weeks, so you may already have noticed the improvement. Of course, any new file systems that you create will also benefit.

不過 EFS 另外一個問題就是貴炸,用錢換方便...

其他非 Google 的 Email Hosting 服務

先前在「Cloudflare 的 Email Routing 功能可以讓一般人用 (包括免費帳號)」這邊有提到,因為 Google Workspace 要廢掉免費版本,所以 Cloudflare 推出了 E-mail forwarding 的方案讓大家用,所以一種簡單的解決方法就是註冊另外一個免費的 Gmail 帳號,然後用 Cloudflare 把信件 forward 到新的帳號裡。

但如果想要避開 Google 的服務的話,在 Hacker News 上的「Ask HN: Best hosted alternative to Google Workspace for email?」這邊就有討論不少可能的替代方案,主要是付費的為主。

討論裡常被提起的方案:

我自己是很愛 Fastmail,他的 webmail 界面速度很快 (即使 www.fastmail.com 需要連到美國的伺服氣上),而且鍵盤快速鍵與 Gmail 幾乎相同,所以學習成本很低,就把其中一個 domain 掛上去用,然後把一些服務的註冊 email 都往這邊掛。

而費用的部份,除了可以月繳外,年繳與三年繳都有另外再給 discount (30GB 版本是 US$5/mo、US$50/y 與 US$140/3y),第一次用一年後,後面就是刷三年了...

如果有在用 Office 的話,Microsoft 365 應該也是個選擇方案,不過我對 Outlook 的 folder-based 的設計邏輯用不慣...

然後是 iCloud+,看討論裡面提到穩定度似乎不太行,但也有人用的好好的,我自己是只有拿來同步 Apple 裝置用,沒有用到郵件服務的部份...

另外是自己有架設 Email service,話說這年頭自己搞 Email system 超麻煩的,對於沒搞過的人 SPFDKIM 加上 DMARC 絕對會搞死自己,乖乖丟雲端服務比較省事,我純粹是要保持多路暢通而已...