Hacker News 前幾天炸很久的 root cause

前幾天 Hacker News 炸了很久,如果是從 Twitter 上的資料來看,是從 2022/07/08 14:08 UTC 這篇:

中間還原失敗 (2022/07/08 17:35 UTC):

到最後恢復 (2022/07/08 20:48 UTC):

Twitter 這邊的資料看起來差不多是六個小時多,以一個應該是只有 database 需要還原的站台來說的確是蠻久的,所以後續在「HN is up again」這邊就有在討論原因,裡面 HN 的老大 dang 也有提到 downtime 是七個小時多:

8 hours of downtime, but not data loss, since there was no data to lose during the downtime.

Last post before we went down (2022-07-08 12:46:04 UTC): https://news.ycombinator.com/item?id=32026565

First post once we were back up (2022-07-08 20:30:55 UTC): https://news.ycombinator.com/item?id=32026571 (hey, that's this thread! how'd you do that, tpmx?)

So, 7h 45m of downtime. What we don't know is how many posts (or votes, etc.) happened after our last backup, and were therefore lost. The latest vote we have was at 2022-07-08 12:46:05 UTC, which is about the same as the last post.

There can't be many lost posts or votes, though, because I checked HN Search (https://hn.algolia.com/) just before we brought HN back up, and their most recent comment and story were behind ours. That means our last backup on the ill-fated server was taken after the last API update (HN Search relies on our API), and the API gets updated every 30 seconds.

I'm not saying that's a rock-solid argument, but it suggests that 30 seconds is an upper bound on how much data we lost.

另外大家就在找 dang 的回應是什麼 (畢竟是第一手資料),用 Ctrl-F 找一下就看到有趣的猜測,從 32028511 這個節點可以看到這串有趣的討論,首先是 mikeiem

You are never going to guess how long the HN SSDs were in the servers... never ever... OK... I'll tell you: 4.5years. I am not even kidding.

然後是 kabdib 的回應:

Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours.

And that they were sold by HP or Dell, and manufactured by SanDisk.

Do I win a prize?

(None of us win prizes on this one).

接著就是 dang 說他覺得這個猜測很有可能:

Wow. It's possible that you have nailed this.

Edit: here's why I like this theory. I don't believe that the two disks had similar levels of wear, because the primary server would get more writes than the standby, and we switched between the two so rarely. The idea that they would have failed within hours of each other because of wear doesn't seem plausible.

But the two servers were set up at the same time, and it's possible that the two SSDs had been manufactured around the same time (same make and model). The idea that they hit the 40,000 hour mark within a few hours of each other seems entirely plausible.

Mike of M5 (mikiem in this thread) told us today that it "smelled like a timing issue" to him, and that is squarely in this territory.

後續他也從自家的 /newest 裡面撈了相關的資料出來,依照他撈出來的關鍵字,看起來是用 HPE 出的 SSD:

It's also an example of the dharma of /newest – the rising and falling away of stories that get no attention:

HPE releases urgent fix to stop enterprise SSDs conking out at 40K hours - https://news.ycombinator.com/item?id=22706968 - March 2020 (0 comments)

HPE SSD flaw will brick hardware after 40k hours - https://news.ycombinator.com/item?id=22697758 - March 2020 (0 comments)

Some HP Enterprise SSD will brick after 40000 hours without update - https://news.ycombinator.com/item?id=22697001 - March 2020 (1 comment)

HPE Warns of New Firmware Flaw That Bricks SSDs After 40k Hours of Use - https://news.ycombinator.com/item?id=22692611 - March 2020 (0 comments)

HPE Warns of New Bug That Kills SSD Drives After 40k Hours - https://news.ycombinator.com/item?id=22680420 - March 2020 (0 comments)

(there's also https://news.ycombinator.com/item?id=32035934, but that was submitted today)

這次 downtime 看起來很像是中了 SSD firmware bug,目前看起來先搬到 EC2 上面了:

$ host news.ycombinator.com
news.ycombinator.com has address 50.112.136.166
$ host 50.112.136.166      
166.136.112.50.in-addr.arpa domain name pointer ec2-50-112-136-166.us-west-2.compute.amazonaws.com.

看討論串應該是暫時性的?

Raspberry Pi 4 可以透過 USB 開機了

一樣是在 Hacker News Daily 上看到這篇:「Boot from USB · Issue #28 · raspberrypi/rpi-eeprom」,不過 GitHub 上的討論沒什麼重點,主要是 Hacker News 上的討論:「Raspberry Pi 4 can finally boot directly from USB (github.com)」。

這個功能在以前的 Raspberry Pi 上是可以用的,但大概是因為 RPi4 的 USB 模組整個換掉,所以在 RPi4 推出時是沒這個功能的,直到最近更新韌體才支援。

不過就算以前可以用 USB 2.0 開機,但因為 USB 2.0 的極限值只有 480Mbps,扣掉 USB mass storage 的 protocol overhead 後,跟 SD 卡相比沒有差太多速度,大家就沒有太在意這個項目。但現在 USB 3.0 的速度快不少,讓這邊有不少發揮的空間,之後來找機會來外接個硬碟測一些東西看看好了...

然後順便提一下,以前要弄這種東西都是同事之間一起團購,但後來發現 PChome 24h 上也有賣,而且價位也算是還算可以?(加上又是隔天到?)

伺服器的 BIOS 保護機制

NIST (National Institute of Standards and Technology) 是美國的政府機關 (國家標準技術研究所),訂定了非常多的標準。像是 DES 的制定 (NIST 前身,NBS - National Bureau of Standards) 與 AES 的選拔都是 NIST 的成果。

Slashdot 上看到 NIST 對於伺服器 BIOS 的保護機制提出建議:「NIST Publishes Draft Guidelines For Server BIOS Protection」,NIST 原始公告則是在「Security First: New NIST Guidelines on Securing BIOS for Servers」。都有原始 PDF 連結可以直接點。

BIOS Protection Guidelines for Servers ToC

標題是「BIOS Protection Guidelines for Servers」,目前還是在 draft 階段,頁數也還不太多...

請更新 HP 印表機的韌體...

HP 發了安全通告「HPSBPI02728 SSRT100692 rev.2 - Certain HP Printers and HP Digital Senders, Remote Firmware Update Enabled by Default」,在安全通告內列出的印表機都有安全問題,「允許遠端安裝未經授權的印表機韌體」,攻擊者可以遠端直接安裝有木馬的韌體:

A potential security vulnerability has been identified with certain HP printers and HP digital senders. The vulnerability could be exploited remotely to install unauthorized printer firmware.

另外可以參考 CVE-2011-4161 的說明:

The default configuration of the HP CM8060 Color MFP with Edgeline; Color LaserJet 3xxx, 4xxx, 5550, 9500, CMxxxx, CPxxxx, and Enterprise CPxxxx; Digital Sender 9200c and 9250c; LaserJet 4xxx, 5200, 90xx, Mxxxx, and Pxxxx; and LaserJet Enterprise 500 color M551, 600, M4555 MFP, and P3015 enables the Remote Firmware Update (RFU) setting, which allows remote attackers to execute arbitrary code by using a session on TCP port 9100 to upload a crafted firmware update.

能更新的就想辦法更新吧,無法更新的看看有沒有辦法處理 port 9100...