GitHub 自己開發的搜尋引擎

前陣子 GitHub 發了一篇文章,說明自己開發搜尋引擎的心路歷程:「The technology behind GitHub’s new code search」。

看了一下其實就是自己幹了一套 search engine cluster,然後針對 code search 把一些功能放進去。

目前這套 search enginer 還是 beta 版本,全站兩億個 repository 只包括了 4500 萬 (大概 22% 左右),然後已經有 115TB 的程式碼了;另外也題到了先前導入 Elasticsearch 時的數字是 800 萬個 repository:

GitHub’s scale is truly a unique challenge. When we first deployed Elasticsearch, it took months to index all of the code on GitHub (about 8 million repositories at the time). Today, that number is north of 200 million, and that code isn’t static: it’s constantly changing and that’s quite challenging for search engines to handle. For the beta, you can currently search almost 45 million repositories, representing 115 TB of code and 15.5 billion documents.

目前是 32 台機器,沒有特別提到記憶體大小,也沒有提到 replication 之類的數字:

Code search runs on 64 core, 32 machine clusters.

然後各種 inverted index 與各種資料在壓縮後只有 25TB:

There are some big wins on the size of the index as well. Remember that we started with 115 TB of content that we want to search. Content deduplication and delta indexing brings that down to around 28 TB of unique content. And the index itself clocks in at just 25 TB, which includes not only all the indices (including the ngrams), but also a compressed copy of all unique content. This means our total index size including the content is roughly a quarter the size of the original data!

換算一下,就會發現現在已經是「暴力」可以解很多事情的年代了,而這已經是全世界最大的 code hosting。

以前隨便一個主題搞大一點就會撞到 Amdahl's law,現在輕鬆不少...

Twitter 的 2022 年架構 (?)

Elon Musk 自己貼了以後,有人把白板圖整理成數位圖後好讀一些:


先把圖拉出來看原圖大小 (可以自己點),比較清楚一點:

從最前端往後看,可以看到 TLS-API 被標成淘汰中,照圖上寫的剩下 Android 版本在用,後續應該是往 GraphQL 靠;另外內部有使用到 Thrift,但沒有標 gRPC,我猜是這邊的主線上沒有 gRPC 而已,其他系統可能還是會有。

另外是有些東西有標 next-gen system,應該就是正在做而還沒上線的東西。

Git 的「災難處理」

但印象中之前看過 (在 Internet Archive 上可以看到 4 Sep 2017 的版本),但搜尋 Hacker News 後發現沒有提過... 這幾天紅起來的「Dangit, Git!?!」,也有簡體中文版可以看。

裡面其實提到了很多要怎麼處理不小心塞錯資料進 Git 的情況,不過好像還是有些東西沒涵蓋到,像是遇到不小心塞到 credentials 進去後需要清除掉的 git rebase -i HASH,接著一連串的手動修 conflict 與 git rebase --continue,最後再接上 git push --force 這種禁招...

另外推一下「為你自己學 Git」這本書,裡面其實也有提到類似的情境:

7.1 狀況題 修改歷史訊息
7.2 狀況題 把多個 Commit 合併成一個Commit
7.3 狀況題 把一個 Commit 拆解成多個Commit
7.4 狀況題 想要在某些 Commit 之間再加新的Commit
7.5 狀況題 想要刪除某幾個 Commit 或是調整Commit 的順序
7.6 Reset、Revert 跟 Rebase 指令有什麼差別?

這本書也有網頁版,在 這邊。

觀察誰在存取剪貼簿的工具 (X11 下)

兩個月前在 Hacker News 上看到的討論,有人想要知道誰在 X11 下存取剪貼簿:「Who keeps an eye on clipboard access? (」,原文在「Who keeps an eye on clipboard access?」這邊,作者用 Python 寫的程式則是在「clipboard-watcher」這邊。

馬上有想到 iOS 在 2020 年推出的機制:「iOS 14 clipboard notifications are annoying, but developer adoption of a new API will improve the experience」。

不過在 X11 上跑起來會發現冒出來的資訊量有點大,像是在瀏覽器操作 WordPress 寫文章時剪剪貼貼的時候就會狂噴,如果可以提供程式的白名單的話就更好了,畢竟是我直接把 clipboard API 裡讀取的功能直接拔掉 (但網站還是可以寫進去就是了),對我來說不會在意 browser 寫進去的情況:

另外程式有時候會卡住 (尤其是遇到圖片的剪輯時),算是 bug 吧...

然後 Hacker News 的討論串裡面有人提到一個有趣的設計,他希望限制那些不在焦點上面的程式去碰 clipboard:

By far the worst offense I've seen in clipboard privacy on the Linux desktop is RedHat's virt-manager. It sends your clipboard AND selection content to all virtual machines, even when they are not focused, with no indication that it's happening, and with no GUI option to turn it off. This is at odds with the common practice of running untrusted code in virtual machines.

這個想法好像不賴,理論上 clipboard 應該是在有互動的時候才會碰到的東西...

Amazon EFS 效能提昇的一些討論

上一篇「Amazon EFS 的效能提昇」提到 Amazon EFS 的效能提昇,在 Hacker News 上看到 Amazon EFS 團隊的 PMT (Product-Manager-Technical) 出來回一些東西:「Amazon Elastic File System Update – Sub-Millisecond Read Latency (」,搜尋 geertj 應該就可以看到他回的東西了...

像是即使是 Jeff Barr 發表這篇文章,也還是經過 legal team 的同意才能發表:

(PMT on the EFS team).

Yes, the wordings are carefully formulated as they have to be signed off by the AWS legal team for obvious reasons. With that said, this update was driven by profiling real applications and addressing the most common operations, so the benefits are real. For example, a simple WordPress "hello world" is now about 2x as fast as before.

另外這次的效能提昇是透過 cache 層達成的:

I'm the PMT for this project in the EFS team. The "flip the switch" part was indeed one of the harder parts to get right. Happy to share some limited details. The performance improvement builds on a distributed consistent cache. You can enable such a cache in multiple steps. First you deploy the software across the entire stack that supports the caching protocol but it's disabled by configuration. Then you turn it for the multiple components that are involved in the right order. Another thing that was hard to get right was to ensure that there are no performance regressions due to the consistency protocol.

然後在每個 AZ 都有 cache:

The caches are local to each AZ so you get the low latency in each AZ, the other details are different. Unfortunately I can't share additional details at this moment, but we are looking to do a technical update on EFS at some point soon, maybe at a similar venue!

另外看起來主要就是 metadata cache 的幫助:

NFS workloads are typically metadata heavy and highly correlated in time, so you can achieve very high hit rates. I can't share any specific numbers unfortunately.

還是有很多細節數字不能透漏,但知道是透過 cache 達成的就已經可以大致上想像後面是怎麼弄出來的了...

Amazon EFS 的效能提昇

AWS 宣佈他們將 Amazon EFS 的 latency 大幅降低以提昇效能:「Amazon Elastic File System Update – Sub-Millisecond Read Latency」。

Linux 上一般是用 NFS 掛 EFS,個位數的 ms 的確對於效能影響超大,現在宣稱讀取的部份降到 0.6ms,應該會有蠻明顯的感覺:

Up until today, EFS latency for read operations (both data and metadata) was typically in the low single-digit milliseconds. Effective today, new and existing EFS file systems now provide average latency as low as 600 microseconds for the majority of read operations on data and metadata.


This performance boost applies to One Zone and Standard General Purpose EFS file systems. New or old, you will still get the same availability, durability, scalability, and strong read-after-write consistency that you have come to expect from EFS, at no additional cost and with no configuration changes.

另外就是過去幾個禮拜他們把現有的 EFS 都轉移過去了:

We “flipped the switch” and enabled this performance boost for all existing EFS General Purpose mode file systems over the course of the last few weeks, so you may already have noticed the improvement. Of course, any new file systems that you create will also benefit.

不過 EFS 另外一個問題就是貴炸,用錢換方便...

其他非 Google 的 Email Hosting 服務

先前在「Cloudflare 的 Email Routing 功能可以讓一般人用 (包括免費帳號)」這邊有提到,因為 Google Workspace 要廢掉免費版本,所以 Cloudflare 推出了 E-mail forwarding 的方案讓大家用,所以一種簡單的解決方法就是註冊另外一個免費的 Gmail 帳號,然後用 Cloudflare 把信件 forward 到新的帳號裡。

但如果想要避開 Google 的服務的話,在 Hacker News 上的「Ask HN: Best hosted alternative to Google Workspace for email?」這邊就有討論不少可能的替代方案,主要是付費的為主。


我自己是很愛 Fastmail,他的 webmail 界面速度很快 (即使 需要連到美國的伺服氣上),而且鍵盤快速鍵與 Gmail 幾乎相同,所以學習成本很低,就把其中一個 domain 掛上去用,然後把一些服務的註冊 email 都往這邊掛。

而費用的部份,除了可以月繳外,年繳與三年繳都有另外再給 discount (30GB 版本是 US$5/mo、US$50/y 與 US$140/3y),第一次用一年後,後面就是刷三年了...

如果有在用 Office 的話,Microsoft 365 應該也是個選擇方案,不過我對 Outlook 的 folder-based 的設計邏輯用不慣...

然後是 iCloud+,看討論裡面提到穩定度似乎不太行,但也有人用的好好的,我自己是只有拿來同步 Apple 裝置用,沒有用到郵件服務的部份...

另外是自己有架設 Email service,話說這年頭自己搞 Email system 超麻煩的,對於沒搞過的人 SPFDKIM 加上 DMARC 絕對會搞死自己,乖乖丟雲端服務比較省事,我純粹是要保持多路暢通而已...

為資料庫提案新的 UUID 格式

前幾天在 Hacker News Daily 上看到的東西,今年四月的時候有人針對資料庫提案新的 UUID 格式:「New UUID Formats – IETF Draft (」。

在 draft 開頭有說明這個提案的目標:

This document presents new time-based UUID formats which are suited for use as a database key.

A common case for modern applications is to create a unique identifier for use as a primary key in a database table. This identifier usually implements an embedded timestamp that is sortable using the monotonic creation time in the most significant bits. In addition the identifier is highly collision resistant, difficult to guess, and provides minimal security attack surfaces. None of the existing UUID versions, including UUIDv1, fulfill each of these requirements in the most efficient possible way. This document is a proposal to update [RFC4122] with three new UUID versions that address these concerns, each with different trade-offs.

另外在 Hacker News 上有人整理出來,可以直接理解提案所提出的新格式是什麼:

A somewhat oversimplified summary of the new UUID formats:

UUID6: a timestamp with a weird epoch and 100 ns precision like in UUID1, but in a big-endian order that sorts naturally by time, plus some random bits instead of a predictable MAC address.

UUID7: like UUID6, but uses normal Unix timestamps and allows more timestamp precision.

UUID8: like UUID7, but relaxes requirements on where the timestamp is coming from. Want to use a custom epoch or NTP timestamps or something? UUID8 allows it for the sake of flexibility and future-proofing, but the downside is that there's no standard way to parse the time from one of these -- the time source could be anything monotonic.

這在不同的 storage engine 上面會有不同的討論,這邊先討論 MySQL 系列的 InnoDB,至於 PostgreSQL 的 engine 以及其他資料庫系統,就另外讓更熟悉的人討論了。

InnoDB 採用了 clustered index (可以參考「Database index」這邊的說明),也就是資料本體是以某種定義的大小順序存放。

在 InnoDB 裡面則是用 primary key 的順序來存放資料 (沒有指定 primary key 時會有 fallback 行為),其他的 unique key 與 index key 則是指到 primary key,所以你可以看到 primary key 的大小也會影響到其他的 index key。

所以 128 bits 的 UUID 在大型的 MySQL ecosystem 實在不怎麼受歡迎,在 2010 年的時候 FlickrTwitter 都有發表過 ticket system:「Ticket Servers: Distributed Unique Primary Keys on the Cheap」、「Announcing Snowflake」,兩個系統有不同的需求,但都是產生 64 bits 的 unique id。

其中 Flickr 的系統算是很簡單的,沒有要保證時間順序 (i.e. 先取的號碼一定比較小,以及後取的號碼一定比較大),就用兩台 MySQL 跑 active-active 架構,然後錯開產生的值:

auto-increment-increment = 2
auto-increment-offset = 1

auto-increment-increment = 2
auto-increment-offset = 2


EC2 總算支援透過 Serial Console 操作了...

以往 Amazon EC2 的機器爛到開不起來時只能「看」到 Console 的輸出,然後要把 root volume 掛到其他機器上修正,接著再掛回來 (然後沒修好就要再重複...),現在總算可以透過 EC2 Serial Console 來操作了:「Troubleshoot Boot and Networking Issues with New EC2 Serial Console」。

不過裡面有一些限制,首先機器必須是基於 AWS Nitro System,這個部份在「Amazon EC2 Instance Types」這邊可以翻到是不是 Nitro,比較新的 family type 應該都是 (像是 t3/t3a/t4g 都是 Nitro,但 t2 不是):

EC2 Serial Console access is available for EC2 instances based on the AWS Nitro System. It supports all major Linux distributions, FreeBSD, NetBSD, Microsoft Windows, and VMWare.

另外是支援的區域目前只有幾個主力區域,不過公司在用的主力區 (新加坡) 也進去了,之後遇到問題的時候可以測試看看:

  • US East (N. Virginia), US West (Oregon), US East (Ohio)
  • Europe (Ireland), Europe (Frankfurt)
  • Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Singapore)

Twitter 上看到有些人有提到「總算啊...」之類的感想...