又再次看到了 Spectre Mitigation 的效能損失...

Hacker News 首頁上看到的文章,講 Spectre Mitigation 的效能損失:「Spectre Mitigations Murder *Userspace* Performance In The Presence Of Frequent Syscalls」,對應的討論串在「Spectre Mitigations Murder Userspace Performance (ocallahan.org)」。

看起來作者是在調校 rr 時遇到的問題,幾年前有提到過 rr:「Microsoft 的 TTD 與 Mozilla 的 RR」。

對此作者對 rr 上了一個 patch,減少了 mitigation code 會在 syscall 時清掉 cache 與 TLB,這個 patch 讓執行的速度大幅提昇:「Cache access() calls to avoid syscalls」。

另外作者提到了他的硬體是 IntelSkylake,他又再跑一次 pre-patch 與 post-patch 的速度,可以看到在 pre-patch 前,mitigation 會讓系統慢超多 (從 2m5.776s 到 3m19.648s),而 post-patch 後大幅降低 syscall 的使用,就不會影響那麼多 (從 0m33.422s 到 0m36.160s)。

就目前知道的 mitigation 方式來說,這個猜測應該是對的...

Windows 上現成的 KataGo + Lizzie 安裝程式

本來花了一些功夫整理了一下 KataGo + Lizzie 純 CPU 版本在 Windows 上的安裝方式 (在「KataGo/Windows」這邊),後來找了一下發現早就有人做的更簡單了:「BadukMegapack」。

底層的部份除了可以選擇裝 KataGo 外,也可以選擇目前已經停止維護的 Leela Zero,或是 SAIAQRay

而界面的部份除了可以用 Lizzie 外 (而且還是改過的加強版 XD),還可以選其他的界面... 除此之外還連 JVM (Java 8) 都一起拉進來幫你裝。

看起來算是包的好好的... 看起來最困難的應該是弄一張高階顯卡了?

居然在安全性漏洞的 PoC 上面看到拿 Bad Apple!! 當作範例

人在日本的資安專家 Hector Martin 找到了 Apple M1 的安全漏洞,可以不用透過 macOS Big Sur 提供的界面,直接透過 M1 的漏洞跨使用者權限傳輸資料,這可以用在突破 sandbox 的限制。而也如同目前的流行,他取了一個好記的名字:「M1RACLES: M1ssing Register Access Controls Leak EL0 State」,對應的 CVECVE-2021-30747

先講比較特別的點,PoC 的影片放在 YouTube 上,作者拿 Bad Apple!! 當作示範,這很明顯是個雙關的點:

這應該是當年的影繪版本,看了好懷念啊... 當年看到的時候有種「浪費才能」的感覺,但不得不說是個經典。

Hacker News 上有討論可以翻翻:「M1racles: An Apple M1 covert channel vulnerability (m1racles.com)」。

依照作者的說明,Apple A14 因為架構類似,也有類似的問題,不過作者沒有 iPhone,沒辦法實際測試:

Are other Apple CPUs affected?

Maybe, but I don't have an iPhone or a DTK to test it. Feel free to report back if you try it. The A14 has been confirmed as also affected, which is expected, as it is a close relative of the M1.

另外作者覺得這個安全漏洞在 macOS 上還好,主要是你系統都已經被打穿可以操控 s3_5_c15_c10_1 register 了,應該會有更好的方式可以用:

So you're telling me I shouldn't worry?

Yes.

What, really?

Really, nobody's going to actually find a nefarious use for this flaw in practical circumstances. Besides, there are already a million side channels you can use for cooperative cross-process communication (e.g. cache stuff), on every system. Covert channels can't leak data from uncooperative apps or systems.

Actually, that one's worth repeating: Covert channels are completely useless unless your system is already compromised.

比較明顯的問題應該是 iOS 這邊的 privacy issue,不過 iOS 上的 app store 有基本的保護機制:(不過想到作者可以故意寫成 RCE 漏洞...)

What about iOS?

iOS is affected, like all other OSes. There are unique privacy implications to this vulnerability on iOS, as it could be used to bypass some of its stricter privacy protections. For example, keyboard apps are not allowed to access the internet, for privacy reasons. A malicious keyboard app could use this vulnerability to send text that the user types to another malicious app, which could then send it to the internet.

However, since iOS apps distributed through the App Store are not allowed to build code at runtime (JIT), Apple can automatically scan them at submission time and reliably detect any attempts to exploit this vulnerability using static analysis (which they already use). We do not have further information on whether Apple is planning to deploy these checks (or whether they have already done so), but they are aware of the potential issue and it would be reasonable to expect they will. It is even possible that the existing automated analysis already rejects any attempts to use system registers directly.

從調校 HTTP Server 的文章中學各種奇技淫巧

在「Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 Instance」這篇文章裡面,作者在示範各種奇技淫巧調校 HTTP server。

Hacker News 上的討論也蠻有趣的:「Extreme HTTP Performance Tuning (talawah.io)」。

雖然是在講 HTTP server,但裡面有很多東西可以拿出來獨立用。

想特地拿出來聊的大項目是「Speculative Execution Mitigations」這段,作者有些說明,除非你真的知道你在做什麼,不然不應該關掉這些安全相關的修正:

You should probably leave the mitigations enabled for that system.

而作者是考慮到 AWS 有推出 AWS Nitro Enclaves 的前提下決定關掉,但我會建議在 *.metal 的機器上才這樣做,這樣可以避免這台機器上有其他 AWS 帳號的程式在跑。

測試中關了一卡車 mitigation,得到了 28% 的效能提昇:

Disabling these mitigations gives us a performance boost of around 28%.

這其實比預期中多了不少,這對於自己擁有實體機房跑 Intel 平台的使用者來說,很吸引人啊...

Amazon EC2 的 t3/t3a/t4g 的 CPU credit 保留七天的限制

Twitter 上看到朋友提到 t3 系列的機器有保留七天的 CPU credit:

在「CPU credits and baseline utilization for burstable performance instances」這邊有提到,t3/t3a/t4g 的設計都是讓你可以塞 24h 小時的量:

這邊講的七天是這段:

CPU credits on a running instance do not expire.

For T2, the CPU credit balance does not persist between instance stops and starts. If you stop a T2 instance, the instance loses all its accrued credits.

For T3 and T4g, the CPU credit balance persists for seven days after an instance stops and the credits are lost thereafter. If you start the instance within seven days, no credits are lost.

開著的機器的 CPU credit 不會過期,只會到最大上限 (在同一篇文件裡面的表格有提到),t2 的機器關掉後 (stop) CPU credit 就會直接消失,而 t3/t3a/t4g 則在關掉後會保留七天。

之前沒注意到文件上的這點。

另外之前在測試自己架設 Sentry 時還測過 t3a.medium -> r5a.large -> t3a.medium 這樣換過去又換回來的情況,本來的 CPU credit 是可以繼續用的,看起來 CPU credit 不會因為 family type 改變就不見 (不過不確定這個是不是 undefined behavior...)。

Google 釋出網頁版的 Spectre 攻擊 PoC,包括 Apple M1 在內

在大約三年前 (2018 年年初) 的時候,在讀完 Spectre 之後寫下了一些記錄:「讀書時間:Spectre 的攻擊方式」,結果在 Bruce Schneier 這邊看到消息,Google 前幾天把把 PoC 放出來了:「Exploiting Spectre Over the Internet」,在 Hacker News 上也有討論:「A Spectre proof-of-concept for a Spectre-proof web (googleblog.com)」。

首先是這個攻擊方法在目前的瀏覽器都還有用,而且包括 Apple M1 上都可以跑:

The demonstration website can leak data at a speed of 1kB/s when running on Chrome 88 on an Intel Skylake CPU. Note that the code will likely require minor modifications to apply to other CPUs or browser versions; however, in our tests the attack was successful on several other processors, including the Apple M1 ARM CPU, without any major changes.

即使目前的瀏覽器都已經把 performance.now() 改為 1ms 的精度,也還是可以達到 60 bytes/sec 的速度:

While experimenting, we also developed other PoCs with different properties. Some examples include:

  • A PoC which can leak 8kB/s of data at a cost of reduced stability using performance.now() as a timer with 5μs precision.
  • A PoC which leaks data at 60B/s using timers with a precision of 1ms or worse.

比較苦的消息是 Google 已經確認在軟體層沒辦法解乾淨,目前在瀏覽器上只能靠各種 isolation 降低風險,像是將不同站台跑在不同的 process 裡面:

In 2019, the team responsible for V8, Chrome’s JavaScript engine, published a blog post and whitepaper concluding that such attacks can’t be reliably mitigated at the software level. Instead, robust solutions to these issues require security boundaries in applications such as web browsers to be aligned with low-level primitives, for example process-based isolation.

Apple M1 也中這件事情讓人比較意外一點,看起來是當初開發的時候沒評估?目前傳言的 M1x 與 M2 不知道會怎樣...

GTA Online 釋出官方修正,大幅改善啟動效能

看到「GTA Online load time fix released, shaves off actual minutes of waiting for some」這邊的消息,先前在「GTA 的啟動讀取效能問題」這邊提到 GTA Online 啟動速度很慢的問題,官方正式推出修正版本了:「GTAV Title Update 1.53 Notes (PS4 / Xbox One / PC)」。

抓了一些在 Reddit 的討論「Loading Times Have FINALLY been patched - Discussion Thread」。

這則降的比率與當時 workaround 的修正差不多:

Insane. GTA menu -> GTA: Online.

Dropped from 7 minutes to 1:57

i7-2600k,GTX1070,16GB RAM and the game is on HDD.

這個就有點誇張了,這是 90% 吧?

Dropped from 5-8 minutes to 35 seconds

這個差不多 70%~80%:

Loading time 2m 20s for online directly from steam. Before it was like 8-10 minutes for me. Damn

Edit: 50s for story mode. 35s from story mode to online. So it seems it's still faster to load into online from story mode.

這個也差不多 70%:

From 4-5 minutes to 1 a minute and 22 seconds. Y e s p l e a s e

然後 PS4 的版本原來也受到一樣的影響?

Currently tested on PS4 , from main menu to online : 3min 45 sec From story mode to online: 1min 20sec (😩 i can't tell for sure )

整體看起來是正面的,畢竟大家等這個問題等超久了... 另外也可以看出來當初的 workaround patch 其實相當精準的把問題都解掉了,官方的修正並沒有快更多。

來繼續關注 libc 那邊的問題...

Cloudflare 再次嘗試 ARM 伺服器

2018 年的時候寫過一篇 Cloudflare 在嘗試 ARM 伺服器的進展:「Cloudflare 用 ARM 當伺服器的進展...」,後來就沒有太多公開的消息,直到這幾天看到「ARMs Race: Ampere Altra takes on the AWS Graviton2」才看到原因:

By the time we completed porting our software stack to be compatible with ARM, Qualcomm decided to exit the server business.

所以是都測差不多,也都把 Cloudflare 自家的軟體搬上去了,但 Qualcomm 也決定收手,沒機器可以用...

這次再次踏入 ARM 領域讓人想到前陣子 AppleM1,讓大家看到 ARM 踏入桌機與筆電領域可以是什麼樣貌...

這次 Cloudflare 選擇了 Ampere Altra,這是基於 Neoverse N1 的平台,而這個平台的另外一個知名公司就是 AWSGraviton2,所以就拿來比較:

可以看到 Ampere Altra 的核心數多了 25% (64 vs. 80),運作頻率多了 20% (2.5Ghz vs. 3.0Ghz)。測試的結果也都有高有低,落在 10%~40% 都有。

不過其中比較特別的是 Brotli - 9 的測試特別差 (而且是 8 與 10 都正常的情況下):

依照 Cloudflare 的說法,他們其實不會用到 Brotli - 7 以及更高的等級,不過畢竟有測出來,還是花了時間找一下根本原因:

Although we do not use Brotli level 7 and above when performing dynamic compression, we decided to investigate further.

反追問題後發現跟 Page Faults 以及 Pipeline Backend Stalls 有關,不過是可以改寫避開,在避開後可以達到跟 Graviton2 類似的水準:

By analyzing our dataset further, we found the common underlying cause appeared to be the high number of page faults incurred at level 9. Ampere has demonstrated that by increasing the page size from 4K to 64K bytes, we can alleviate the bottleneck and bring the Ampere Altra at parity with the AWS Graviton2. We plan to experiment with large page sizes in the future as we continue to evaluate Altra.

但目前看起來應該都還算正向,看起來供貨如果穩定的話,應該有機會換過去?畢竟 ARM 平台可以省下來的電力太多了,現在因為 M1 對 ARM 的公關效果太驚人的關係,解釋起來會更輕鬆...

把 blog 從 t4g.small 降到 t4g.micro

我在「把 blog 搬到 t4g.small 上」這邊有提到把這個 blog 搬到 Amazon EC2t4g.small 上 (2GB RAM + 20% CPU credit),跑了一陣子把 CPU usage 拉出來看:

當初估大約要 20% 的 CPU credit,結果發現 CPU credit 大概用 5% 就夠了。另外記憶體的部份大約要給 1GB,這個量可以看出來一些沒在用的 process 會被丟到 swap:

              total        used        free      shared  buff/cache   available
Mem:          952Mi       380Mi        79Mi       110Mi       492Mi       368Mi
Swap:         511Mi       152Mi       359Mi

把條件綜合起來計算,就往下降一階變成 t4g.micro 了 (1GB RAM + 10% CPU credit)。

另外新機種比較不用擔心淘汰速度,就看了一下 Reserved Instances 的價錢,一年 USD$44,三年 USD$84,看起來只要有用兩年就算是 OK,直接買三年解決掉...

Let's Encrypt 升級資料庫伺服器 (AMD YES?)

Let's Encrypt 升級了 MariaDB 資料庫的伺服器 (跑 InnoDB),特地寫了一篇文章出來講:「The Next Gen Database Servers Powering Let's Encrypt」。

CPU 的部份從本來的 2x Intel Xeon E5-2650 (Total 24 cores / 48 threads) 換成了 2x AMD EPYC 7542 (Total 64 cores / 128 threads),這點在本來就是 CPU 滿載的情境下改善很大:

而本來的瓶頸一解決,也使得 API 的 latency 直接降下去:

回頭看一下架構,可以看到他們提到沒有使用分散式的資料庫,而是單台 database 硬撐,驗證了即使到了 Let's Encrypt 這種規模,以暴制暴還是很有效的:

We run the CA against a single database in order to minimize complexity. Minimizing complexity is good for security, reliability, and reducing maintenance burden. We have a number of replicas of the database active at any given time, and we direct some read operations to replica database servers to reduce load on the primary.

除了 CPU 暴力外,2TB RAM 與 24 顆 NVMe SSD 的搞法也是很讚的,擺明就是用記憶體拼 cache 的量,以及用大量的 NVMe SSD 疊 IOPS。

然後硬體還在成長,看起來暴力解應該會變成以後的基本答案了...