Ingo Molnár 提出讓 Linux Kernel 編譯速度提昇的 mega patch

Hacker News 首頁上看到「Massive ~2.3k Patch Series Would Improve Linux Build Times 50~80% & Fix "Dependency Hell"」這個,對應到 mailing list 上的信件是「* [PATCH 0000/2297] [ANNOUNCE, RFC] "Fast Kernel Headers" Tree -v1: Eliminate the Linux kernel's "Dependency Hell"」這個,看到「0000/2297」這個 prefix XDDD

他主要是想要改善 Linux Kernel 的 compile 時間 (從 project 的名稱「Fast Kernel Headers」可以看到),只是沒想到會縮短這麼多。另外一方面也順便處理了 dependency hell 的問題 (改善維護性)。

測試出來的結果相當驚人,從 231.34 +- 0.60 secs (15.5 builds/hour) 到 129.97 +- 0.51 secs (27.7 builds/hour),以編譯次數來看的話是 78% 的改善。如果以 CPU time 來看的話,從 11,474,982.05 msec cpu-clock 降到 7,100,730.37 msec cpu-clock,也是以編譯次數來算的話,有 61.6% 的改善...

這是花了一年多的時間嘗試才達成的目標,嘗試不同的方法,前幾次雖然都有改善,但改善幅度太小,變動卻太大,他覺得不值得丟出來,直到第三次才達成這樣的目標...

第一次:

When I started this project, late 2020, I expected there to be maybe 50-100 patches. I did a few crude measurements that suggested that about 20% build speed improvement could be gained by reducing header dependencies, without having a substantial runtime effect on the kernel. Seemed substantial enough to justify 50-100 commits.

第二次:

But as the number of patches increased, I saw only limited performance increases. By mid-2021 I got to over 500 commits in this tree and had to throw away my second attempt (!), the first two approaches simply didn't scale, weren't maintainable and barely offered a 4% build speedup, not worth the churn of 500 patches and not worth even announcing.

第三次:

With the third attempt I introduced the per_task() machinery which brought the necessary flexibility to reduce dependencies drastically, and it was a type-clean approach that improved maintainability. But even at 1,000 commits I barely got to a 10% build speed improvement. Again this was not something I felt comfortable pushing upstream, or even announcing. :-/

然後基於第三次的成果覺得有望,意外的發現後續的速度改善比想像中的多非常多:

But the numbers were pretty clear: 20% performance gains were very much possible. So I kept developing this tree, and most of the speedups started arriving after over 1,500 commits, in the fall of 2021. I was very surprised when it went beyond 20% speedup and more, then arrived at the current 78% with my reference config. There's a clear super-linear improvement property of kernel build overhead, once the number of dependencies is reduced to the bare minimum.

這次的 patch 雖然超大包,但看起來對於 compile 時間改善非常多,應該會有不少討論... 消息還蠻新的 (台灣時間今天早上五點的信),晚點可以看一下其他大老出來回什麼...

在 ZFS 上跑 PostgreSQL 的調校

在「Everything I've seen on optimizing Postgres on ZFS」這邊看到如果要在 ZFS 上面跑 PostgreSQL 時的調校方式,看起來作者有一直在更新這篇,所以需要的時候可以跑去看...

主要的族群是要搞 self-hosted PostgreSQL 的人,相較於 ext4 或是 XFS,底層如果使用 ZFS 可以做許多事情,像是 compression 與 snapshot,這對於很多 DBA 相關的操作會方便不少,但也因為 ZFS 的關係,兩邊 (& PostgreSQL) 需要一起調整以確保效能...

不過短期應該還是用 RDS 就是了...

Linux Kernel 裡的 RNG 從 SHA-1 換成 BLAKE2s

Hacker News Daily 上看到的消息,Linux Kernel 裡的 RNG,裡面用到的 SHA-1 演算法換成 BLAKE2s 了:

SHA-1 已知的問題是個隱患,不過換成 BLAKE2s 應該是 maintainer 的偏好,Jason Donenfeld 在 WireGuard 裡面也是用 BLAKE2s...

用 Exodus 打包 Linux ELF 檔案到其他機器上

前幾天在 Hacker News Daily 上看到的工具:「Exodus」,官方的說明是這樣:

Painless relocation of Linux binaries–and all of their dependencies–without containers.

技術上是把 Linux ELF 檔案搬到其他機器上以外,也幫你把對應的 dynamic library 都一起包進去:

  • Finding and bundling all of a binary's dependencies.
  • Launching the binary in such a way that the proper dependencies are used without any potential interaction from system libraries on the destination machine.

而 Linux 的 Kernel 因為有儘量維持 ABI compatibility,應該是不會有太大的問題,除非剛好用到新的 API...

看起來是個除了用 static compile 以外的解法,好像可以來弄弄看 FFmpeg

XFCE 配上 Chromium 系列瀏覽器 (Chrome/Brave/...),視窗最大化時的問題

今天發現 Brave 在視窗最大化時會超出預期的邊界,而非放大到螢幕的邊緣,找了一下發現有人已經在 Brave 的 GitHub 上開了「Incorrect scale if browser is full screen #18964」這張票,後來看到有人說在 Chromium 的 bug system 上已經有人提出來了:「Issue 1257119: Goes under the taskbar when maximized」、「Issue 1260821: maximise gets screen dimension wron」與「Issue 1261797: [User Feedback - Stable] Reports that when Chrome is maximized after being minimized, it launches to beyond the window frame on Linux」。

這次遇到的 bug 看起來是只有用 XFCE 的使用者才會中獎,目前先摸索出一套 workaround 是透過 wmctrl 操作修改瀏覽器的位置與視窗大小。

方法是先用 wmctrl -l -G 列出所有視窗的資訊,包括 geometry 的資料,接著再用 wmctrl -i -r 0x12345678 -e 0,3760,15,1232,1935 這樣的指令去指定瀏覽器的位置與視窗大小。

不知道要撐多久就是了...

獨立遊戲創作者推出 Linux 版的好處

標題不知道怎麼下,大概就是這樣...

Hacker News 首頁上翻到的,以這個 upvote 數量來看,應該會收到今天的 Hacker News Daily 上:「Despite having just 5.8% sales, over 38% of bug reports come from Linux (reddit.com)」。

作者是一位獨立遊戲開發者,在兩年前推出了「ΔV: Rings of Saturn」這款遊戲,並且也發佈了 Linux 版。

作者先就數字來比較,他賣出了 12000 套,其中 700 套是 Linux 玩家;另外他收到了 1040 個 bug report,其中大約 400 個是從 Linux 玩家回報的:

As of today, I sold a little over 12,000 units of ΔV in total. 700 of these units were bought by Linux players. That’s 5.8%. I got 1040 bug reports in total, out of which roughly 400 are made by Linux players.

That’s one report per 11.5 users on average, and one report per 1.75 Linux players. That’s right, an average Linux player will get you 650% more bug reports.

看文章時可能會覺得「Linux 玩家真難伺候」,但實際上作者講到,這 400 個 bug 裡面只有 3 個 bug 是平台相關的 (只會發生在 Linux 上),其他的 bug 其實是所有平台都會發生:

A lot of extra work for just 5.8% of extra units, right?

Wrong. Bugs exist whenever you know about them, or not.

Do you know how many of these 400 bug reports were actually platform-specific? 3. Literally only 3 things were problems that came out just on Linux. The rest of them were affecting everyone[.]

原因是 Linux 社群在參與各種 open source project 養成的習慣,會盡可能把各種資訊講清楚,並且找出可以重製問題的方式:

The thing is, the Linux community is exceptionally well trained in reporting bugs. That is just the open-source way. This 5.8% of players found 38% of all the bugs that affected everyone. Just like having your own 700-person strong QA team. That was not 38% extra work for me, that was just free QA!

But that’s not all. The report quality is stellar.

與一般玩家的回報方式完全不同,Linux 玩家很習慣就會附上基本的環境資訊,系統紀錄,甚至有時候還會包括 core dump 與 reproducible steps:

I mean we have all seen bug reports like: “it crashes for me after a few hours”. Do you know what a developer can do with such a report? Feel sorry at best. You can’t really fix any bug unless you can replicate it, see it with your own eyes, peek inside and finally see that it’s fixed.

And with bug reports from Linux players is just something else. You get all the software/os versions, all the logs, you get core dumps and you get replication steps. Sometimes I got with the player over discord and we quickly iterated a few versions with progressive fixes to isolate the problem. You just don’t get that kind of engagement from anyone else.

不知道有沒有遇到回報 GDB 資訊的...

很特別的分享 XDDD

用 iptables 擋特定國家的封包

這兩天發現 ubuntu-20.04.3-live-server-amd64.iso 這個 BitTorrent 的 ISO image 有大量來自 CN 的連線在狂抓,導致整個上傳頻寬都被吃滿:

沒想到第一次用 iptables 的 xt_geoip 居然是這個用途... 主要是參考「GeoIP Blocking Ubuntu 20.04LTS」這邊的方法,不過因為我的 rtorrent 是跑在 Docker 裡面的,有另外要注意的地方。

首先是安裝軟體,這邊要裝 xtables-addons-commonlibtext-csv-xs-perl

sudo apt install -y libtext-csv-xs-perl xtables-addons-common

再來是建立目錄,並且下載一包 GeoIP 的資料 (從 DBIP 下載) 並且轉成 xt_geoip 可以用的格式:

sudo mkdir /usr/share/xt_geoip
cd /usr/share/xt_geoip
sudo /usr/lib/xtables-addons/xt_geoip_dl
sudo /usr/bin/perl /usr/lib/xtables-addons/xt_geoip_build

然後就是加到 iptables 的條件裡面了,我加到兩個地方,一個是 INPUT chain,另外一個是 DOCKER-USER chain (參考「Docker and iptables」這邊的說明),假設你是用 port 6991 的話就這樣加:

sudo iptables -I INPUT -p tcp -m geoip --source-country CN -m tcp --dport 6991 -j DROP
sudo iptables -I DOCKER-USER -p tcp -m geoip --source-country CN -m tcp --dport 6991 -j DROP

然後可以考慮每個禮拜更新一次資料庫。

另外在找資料的時候發現「Free updated GeoIP legacy databases」這邊有人放出 MaxMind 的版本,不過免費版的應該都差不多,這邊就用 xtables-addons-common 內預設的。

弄完以後就正常多了...

在 Linux (Ubuntu) 上跑透過 QEMU 跑 Windows/Mac/Linux 的工具

Hacker News Daily 上看到的工具:「Quickly create and run optimised Windows, macOS and Linux desktop virtual machines.」,對應的討論在「Quickemu: Quickly create and run optimised Win-10,11/macOS/Linux on Linux (github.com/wimpysworld)」這邊可以看到,可以減少自己要設定一堆 QEMU 參數。

雖然專案是支援多系統,但其實 Microsoft WindowsLinux 的部份在其他虛擬軟體都很簡單 (像是用 VirtaulBox),大家馬上會注意到的重點還是 macOS 的部份,如果有自己弄過就會知道這東西有夠難裝的,而且跨版本有不同的安裝方式...

目前 Quickemu 支援四個版本:

Supported macOS releases:

  • High Sierra
  • Mojave
  • Catalina (Recommended)
  • Big Sur

然後可以看到幾乎所有目前能支援的功能都有設定上去了,包括 VirtIO 與 USB 的部份。

然後一些經典的問題,像是 Big Sur 的音源問題還是沒解:

Full Duplex audio works on macOS High Sierra, Mojave and Catalina.

  • macOS Big Sur has no audio at all.

在 Hacker News 的討論串裡面有提到有很多地方沒有檢查,這會是風險:

While I appreciate the effort, and the code is very readable. I just want to give a friendly warning that these shell scripts just download random stuff from the internet and run this random stuff without checking any integrity/signature.

下面的討論另外看到個冷知識,關於蘋果故意走 HTTP 下載 recovery image 是因為 HTTPS 太複雜,在 UEFI firmware 裡面實做容易產生被攻擊的點,所以決定自己透過其他機制確認正確性:

Apple Internet recoveryOS images are served over plain http, on purpose. The macrecovery.py script used by Quickemu uses http¹, though the server supports https.

https://support.apple.com/guide/security/recoveryos-and-diagnostics-environments-sec2512a0c09/web

> When the internet recovery and diagnostic modes were added to Mac computers in 2011, it was decided that it would be better to use the simpler HTTP transport, and handle content authentication using the chunklist mechanism, rather than implement the more complicated HTTPS functionality in the UEFI firmware, and thus increase the firmwareʼs attack surface.

¹https://github.com/acidanthera/OpenCorePkg/blob/4a740c3f256e285c66ca3b65e42b60af6826d343/Utilities/macrecovery/macrecovery.py#L123

[edit] Added macrecovery.py info

另外為了避免直接在 shell script 裡面出現「神秘字串」,可以看到特別的寫法 XDDD

Took a little while to find the magic words in there: https://github.com/wimpysworld/quickemu/blob/af26f41440d63a069045660fad860c797011310a/quickemu#L351

可以想到一些用途,像是在機房裡面跑 CI 的 worker,但要注意這個搞法不符合蘋果的 EULA,現在不抓不代表以後也不會有事,請自己謹慎評估...

然後往 ARM-based 架構後應該門檻就更高了,現在還有 Intel-based 的環境可以用加減用...

Ubuntu 下的滑鼠滾輪速度

這陣子因為經常切回 WindowsD2R,發現 Windows 下的滾輪速度快多了,回到 Ubuntu 20.04 下發現無法調整滑鼠滾輪的速度,找了一些方案測試,發現居然地雷還是超多 XD

搜尋可以找到「Increase mouse wheel scroll speed」與「How to change my mouse wheel scroll rate?」這兩篇,被推最多的都是 imwheel,但這套軟體的最新版是 2004 年,實際上用就會發現配合現代的系統 bug 很多...

另外用的方案是「Mouse scroll wheel acceleration, implemented in user space」,作者用 Python 去控制加速,測了一下正常多了。範例給的 ./main.py -v --exp 1 其中的 --exp 1 實際用起來有點太快,我改成 0.75 比較習慣。

先照著作者提到的,把 dependency 都裝起來,接下來掛到 Session and Startup 裡面,在登入後跑起來就可以了:

Ubuntu 14.04 與 16.04 的 ESM 從八年延長到十年

本來的舊的 Ubuntu ESM 是額外的三年 (加上本來的 LTS 五年,共八年),14.04 會支援到 2022 年四月 (參考 Internet Archive 上的存檔資料「Ubuntu 14.04 LTS has transitioned to ESM support」),然後 16.04 會支援到 2024 年四月 (參考 Internet Archive 上的存檔資料「Ubuntu 16.04 LTS transitions to Extended Security Maintenance (ESM)」),而 18.04 與 20.04 以後的 Ubuntu ESM 則是額外五年。

現在則是宣佈 14.04 與 16.04 都切齊額外五年了,所以總共都是十年:「Ubuntu 14.04 and 16.04 lifecycle extended to ten years」。

另外在 Hacker News 上的討論可以看一下:「Ubuntu 14.04 and 16.04 lifecycle extended to ten years (ubuntu.com)」,有人覺得這個政策很糟,但我覺得還好,有些商業環境就是花錢解決懶得升級... (沒有 support 只有 security update 的方案,一台實體機器才 USD$225/year,如果是虛擬機的話就更便宜了)

算是對付 legacy application 還蠻重要的方案...