Tag: post

Cloudflare 前幾天 API 與 Dashboard 出事的 Post Mortem 記錄

前幾天 Cloudflare 的 API 與 dashboard 掛了一天多，少見的讓 Matthew Prince (CEO) 自己出來發 post mortem 記錄了：「Post Mortem on Cloudflare Control Plane and Analytics Outage」，在 Hacker News 上面也有蠻多討論的：「Post Mortem on Cloudflare Control Plane and Analytics Outage (cloudflare.com)」，這邊是整理我自己讀完後的感想。

從「Matthew Prince - The Cloudflare Blog」這邊可以看出來 Matthew Prince 上次是 2023/09/27 的公關文「Cloudflare’s 2023 Annual Founders’ Letter」，還有對應的多國翻譯，像是繁體中文的「Cloudflare 2023 年度創始人來信」，再往前的 2022/12/11 也是公關文「Welcome to Cloudflare’s Impact Week」(以及對應的繁體中文版本：「歡迎來到 Cloudflare 的 Impact Week」)。

這次的事情算是 Cloudflare 在 post-IPO 後很少見的長時間出事，就難得看到 Matthew Prince 自己出來坦了。為了重新建立信任，加上因為層級的關係，可以看到透漏出很多架構細節，算是這次可以窺視 Cloudflare 架構的一些資訊。

先大概提一下官方文章的著墨點：他們花了非常多的篇幅在機房服務商 Flexential 在處理 PDX-04 (這是 Cloudflare 訂的名稱) 機房電力問題的失職 (要注意這邊是 Cloudflare 的觀點，認為 Flexential 的失職，目前沒有從 Flexential 這邊的消息出來解釋)，淡化掉了 Cloudflare 自己的設計問題，這邊在 Hacker News 上有蠻多人都有指出來的。

這次事件一切的起因是 Flexential 的 PDX-04 機房整個電力系統斷線 offline 導致的，屬於標準的 data center failure 的情況，像是 2013 年二月時是方機房的火災 (可以參考 iThome 的整理)，或是 2021 年 OVH 的機房火災 (「去年 OVH 機房大火的部份情形最近被揭露」)，是個在設計架構時一定會規劃進去的項目。

所以 Matthew Prince 先是解釋 Cloudflare 的 HA 作法，是直接在 Hillsboro, Oregon 租三個機房建立起 low-latency network：

Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections.

但大家看到這計馬上就會去查，這個城市也才 66.9km2 的土地，大約是 1/4 個台北市 (約 291.8km2) 再小一些，拉了一下城市內的直線最遠距離，大約是 12km？

呃，這不是一個地震 (就在聖安地列斯斷層區域？) 或是一個核攻擊就把 Cloudflare 最核心的部分給擺平了嗎？

其中 analytics systems 就算了：整個 Hillsboro 掛了進入 gracefully degrading，我可以理解這個設計的考量，但 control plane 看起來不太妙？

雞蛋放在同一個籃子的問題裡先放著，雖然這個問題真的很...。

後續提到了有些重要的產品對沒有 HA 能力的服務上有相依性：

Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04.

這邊沒有講是哪些服務的相依性，但文章其他地方有提到有些基礎服務是沒有跨機房 HA 架構的，只有在 PDX-04 有跑，包括了 Kafka 與 ClickHouse：

In particular, two critical services that process logs and power our analytics — Kafka and ClickHouse — were only available in PDX-04 but had services that depended on them that were running in the high availability cluster.

這點讓人頗意外的，Kafka 因為自己架設 & 維護過，知道他的架構本身就很容易設計到跨機房的 case，而且這算是很基礎建設的東西，居然沒有跨機房 HA？

而 ClickHouse 只有研究過，沒有實際把 production 量丟上去跑，但從文件看到的東西，應該至少能做到 shared-everything 的架構，也居然沒有跨機房 HA？

這接基礎建設的問題，導致了雖然只是單一機房 PDX-04 掛掉，但在有重要基礎建設消失的情況下 (應該就是上面提到的 Kafka 與 ClickHouse)，加上 Flexential 沒有給出恢復的時間，決定直接跑災難重建的 SOP (也就是 Hillsboro 的三個機房都回不來的情境)。

而這也可以看到恢復時間比較久，從決定切到歐洲的 DR site 到整個切過去花了四個多小時：

Because more services were offline than we expected, and because Flexential could not give us a time for restoration of our services, we made the call at 13:40 UTC to fail over to Cloudflare's disaster recovery sites located in Europe.

By 17:57 UTC, the services that had been successfully moved to the disaster recovery site were stable and most customers were no longer directly impacted.

因為 Kafka 與 ClickHouse 在 Hillsboro 只有單一機房有服務，那就不確定歐洲 DR site 平常有沒有建起來，也許這邊的四個多小時有不少是在歐洲 DR site 把 Kafka 與 ClickHouse 建起來？(這個就只能猜測了)

回到 Flexential 這邊，在恢復供電的過程發現 Cloudflare 這邊迴路用的 breaker 掛了，直到十個小時後才供電，但也因為大家都忙了一整天，Matthew Prince 決定讓大家先回去休息，隔天早上再從歐洲的 DR site 切回 Hillsboro，也因此拉長了恢復的時間：

At 12:48 UTC, Flexential was able to get the generators restarted. [...] When Flexential attempted to power back up Cloudflare's circuits, the circuit breakers were discovered to be faulty.

Flexential replaced our failed circuit breakers, restored both utility feeds, and confirmed clean power at 22:48 UTC. Our team was all-hands-on-deck and had worked all day on the emergency, so I made the call that most of us should get some rest and start the move back to PDX-04 in the morning. That decision delayed our full recovery, but I believe made it less likely that we’d compound this situation with additional mistakes.

要注意報告慣例是用 UTC 時間 (這又是另外一個主題了，先前 HN 上也有其他文章討論過...)，而 Hillsboro 在美西，要減八個小時，所以 22:48 UTC 是下午兩點多左右，美東與歐洲的團隊時間則會更晚。

目前看起來 Cloudflare 的設計與流程有很大的改善空間？之後看看有沒有其他的八卦消息出來？

華盛頓郵報怎麼把 Mapbox 換成其他 open source 方案

在 Hacker News 上面看到「How The Post is replacing Mapbox with open source solutions」這篇，講華盛頓郵報怎麼把 Mapbox 換成 open source 方案，對應的討論在「Replacing Mapbox with open source solutions (kschaul.com)」這邊。

維基百科有提到大概兩年前，2020 年底的時候 Mapbox GL JS 從開源授權換成私有授權了 (也可以參考先前寫的「Mapbox GL JS 的授權改變，以及 MapLibre GL 的誕生」這篇)：

In December 2020, Mapbox released the second version of their JavaScript library for online display of maps, Mapbox GL JS. Previously open source code under a BSD license, the new version switched to proprietary licensing. This resulted in a fork of the open source code, MapLibre GL, and initiation of the MapLibre project.

裡面題到了四個工具。

第一個是 OpenMapTiles，下載部份的圖資使用，對於報導只需要某個區域很方便：

OpenMapTiles is an extensible and open tile schema based on the OpenStreetMap. This project is used to generate vector tiles for online zoomable maps. OpenMapTiles is about creating a beautiful basemaps with general layers containing topographic information.

第二個是 Maputnik，可以修改圖資呈現的方式：

A free and open visual editor for the Mapbox GL styles targeted at developers and map designers.

第三個是 PMTiles，可以將圖資檔案塞到一個大檔案裡面，然後透過 HTTP range requests 下載需要的部份就好，大幅下降 HTTP request 所需要的費用 (很多 CDN 會依照 HTTP request 數量收費)：

Protomaps is a serverless system for planet-scale maps.

An alternative to map APIs at 1% the cost, via single static files on your own cloud storage. Deploy datasets like OpenStreetMap for your site in minutes.

最後就是 fork 出來開源版本的 maplibre-gl-js：

MapLibre GL JS is an open-source library for publishing maps on your websites or webview based apps. Fast displaying of maps is possible thanks to GPU-accelerated vector tile rendering.

It originated as an open-source fork of mapbox-gl-js, before their switch to a non-OSS license in December 2020. The library's initial versions (1.x) were intended to be a drop-in replacement for the Mapbox’s OSS version (1.x) with additional functionality, but have evolved a lot since then.

這樣看起來好像可以用在像 KKTIX 這種下面顯示固定地圖的地方：

Post-Quantum 的 KEM，SIDH/SIKE 確認死亡

似乎是這幾天 cryptography 領域裡面頗熱鬧的消息，SIDH 以及 SIKE 確認有嚴重的問題：「SIKE Broken」，論文在「An efficient key recovery attack on SIDH (preliminary version)」這邊可以取得。

這次的成果是 Key recovery attack，算是最暴力的幹法，直接把 key 解出來。

另外 SIKE 剛好也是先前 Cloudflare 在解釋 Hertzbleed 時被拿來打的目標：「Cloudflare 上的 Hertzbleed 解釋」，這樣看起來連 patch 也都不用繼續研究了...

論文裡面的攻擊對象中，第一個是 Microsoft 在 $IKE challenges 內所定義的 $IKEp182 與 $IKEp217，在只用 single core 的情況下，分別在四分鐘與六分鐘就解出來：

Ran on a single core, the appended Magma code breaks the Microsoft SIKE challenges $IKEp182 and $IKEp217 in about 4 minutes and 6 minutes, respectively.

接著是四個參與 NIST 標準選拔的參數，分別是 SIKEp434、SIKEp503、SIKEp610 以及 SIKEp751，也都被極短的時間解出來：

A run on the SIKEp434 parameters, previously believed to meet NIST’s quantum security level 1, took about 62 minutes, again on a single core.

We also ran the code on random instances of SIKEp503 (level 2), SIKEp610 (level 3) and SIKEp751 (level 5), which took about 2h19m, 8h15m and 20h37m, respectively.

在 Ars Technica 的採訪「Post-quantum encryption contender is taken out by single-core PC and 1 hour」裡面，有問到 SIKE 的共同發明人 David Jao 的看法，他主要是認為密碼學界的人對於數學界的「武器」了解程度不夠而導致這次的情況：

It's true that the attack uses mathematics which was published in the 1990s and 2000s. In a sense, the attack doesn't require new mathematics; it could have been noticed at any time. One unexpected facet of the attack is that it uses genus 2 curves to attack elliptic curves (which are genus 1 curves). A connection between the two types of curves is quite unexpected. To give an example illustrating what I mean, for decades people have been trying to attack regular elliptic curve cryptography, including some who have tried using approaches based on genus 2 curves. None of these attempts has succeeded. So for this attempt to succeed in the realm of isogenies is an unexpected development.

In general there is a lot of deep mathematics which has been published in the mathematical literature but which is not well understood by cryptographers. I lump myself into the category of those many researchers who work in cryptography but do not understand as much mathematics as we really should. So sometimes all it takes is someone who recognizes the applicability of existing theoretical math to these new cryptosystems. That is what happened here.

這樣第四輪的選拔只剩下三個了...

NIST 選出了四個 Post-Quantum Cryptography 演算法

NIST (NSA) 選出了四個 Post-quantum cryptography 演算法 (可以抵抗量子電腦的演算法)：「NIST Announces First Four Quantum-Resistant Cryptographic Algorithms」。

四個演算法分別是：

  • CRYSTALS-Kyber：非對稱加密。
  • CRYSTALS-Dilithium：數位簽名。
  • FALCON：數位簽名。
  • SPHINCS+：數位簽名。

這次沒看到非對稱加解密的演算法...

然後翻了 Hacker News 上的討論，果然一堆人在討論 NIST 能不能信任的問題：「NIST Announces First Four Quantum-Resistant Cryptographic Algorithms (nist.gov)」。

然後據說 Kyber 這個名字出自 Star Wars，Dilithium 這個名字則是出自 Star Trek，這還真公平 XDDD

AWS KMS 與 AWS ACM 支援 post-quantum TLS ciphers

AWS 宣佈 AWS KMS 與 AWS ACM 支援 post-quantum TLS ciphers：「AWS KMS and ACM now support the latest hybrid post-quantum TLS ciphers」。

全區支援 Kyber、BIKE 與 SIKE 這三個演算法：

The three PQC key encapsulation mechanisms (KEMs) offered are Kyber, BIKE, and SIKE. Hybrid post-quantum TLS combines a classical key agreement, such as ECDHE, with one of these KEMs. The result is that your TLS connections inherit the security properties of both the classical and post-quantum key exchanges.

Hybrid post-quantum TLS for AWS KMS and ACM is available in all public AWS Regions.

不過這是 NIST Post-Quantum Cryptography Standardization 裡 Round 3 裡面其中幾個演算法而已：

AWS Key Management Service (KMS) and AWS Certificate Manager (ACM) now support hybrid post-quantum key establishment for transport layer security (SSL/TLS) connections using the latest post-quantum ciphers from Round 3 of the NIST Post-Quantum Cryptography (PQC) selection process.

順便補一下隔壁棚 Cloudflare 的研究：「Making protocols post-quantum」。

GET 與 POST 的差異

看到這篇在講 HTTP (& HTTPS) 裡面 GET 與 POST 的差異，剛好把一些標準的定義拿出來翻一翻，算是複習基本概念：「Get safe」。

第一個基本概念主要是 idempotence (& idempotent)，重複被呼叫不會造成狀態的再次改變：

Idempotence ([...]) is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

數學定義是這樣跑：

An element x of a magma (M, •) is said to be idempotent if:

x • x = x.

If all elements are idempotent with respect to •, then • is called idempotent. The formula ∀x, x • x = x is called the idempotency law for •.

這點在 HTTP 標準 (RFC 7231) 裡面的定義也類似：

A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request. Of the request methods defined by this specification, PUT, DELETE, and safe request methods are idempotent.

第二個基本概念是 Safe method (也是在同樣的 RFC 裡被提到)，主要的思想是 read-only，這也是文章作者的標題要講的事情：

Request methods are considered "safe" if their defined semantics are essentially read-only; i.e., the client does not request, and does not expect, any state change on the origin server as a result of applying a safe method to a target resource. Likewise, reasonable use of a safe method is not expected to cause any harm, loss of property, or unusual burden on the origin server.

然後標準的 HTTP method 是有定義的：

   +---------+------+------------+---------------+
   | Method  | Safe | Idempotent | Reference     |
   +---------+------+------------+---------------+
   | CONNECT | no   | no         | Section 4.3.6 |
   | DELETE  | no   | yes        | Section 4.3.5 |
   | GET     | yes  | yes        | Section 4.3.1 |
   | HEAD    | yes  | yes        | Section 4.3.2 |
   | OPTIONS | yes  | yes        | Section 4.3.7 |
   | POST    | no   | no         | Section 4.3.3 |
   | PUT     | no   | yes        | Section 4.3.4 |
   | TRACE   | yes  | yes        | Section 4.3.8 |
   +---------+------+------------+---------------+

不過文章裡面提到的第一個例子並沒有很好，POST 不保證 safe 沒錯，但不代表 safe operation 就不能用 POST。

這邊用 URI resource 的概念 (以及 SEO？) 或是用 Post/Redirect/Get 的概念來說明會比較好：

<form method="get" action="/search">
<input type="search" name="term">

不過文章後續提到的問題的確就是我自己都會犯錯的問題：

“Log out” links that should be forms with a “log out” button—you can always style it to look like a link if you want.

“Unsubscribe” links in emails that immediately trigger the action of unsubscribing instead of going to a form where the POST method does the unsubscribing. I realise that this turns unsubscribing into a two-step process, which is a bit annoying from a usability point of view, but a destructive action should never be baked into a GET request.

這兩個動作都會造成 server 端的狀態改變，不應該用 GET，而我自己常常忘記第一個... 這邊其實可以用 form 產生 POST 需求，並且用 css 效果包起來，達到看起來跟一般的連結一樣。

寫起來讓自己多一點印象，之後避免再犯一樣的錯誤...

Google 與 Cloudflare 測試 Post-Quantum 演算法的成果

這幾年量子電腦的進展不斷有突破，雖然到對於攻擊現有的密碼學看起來還有一段時間，但總是得先開始研究對量子電腦有抵抗性的演算法...

其中 Google Chrome 的團隊與 Cloudflare 的團隊手上都有夠大的產品，兩個團隊合作測試的結果在學界與業界都還蠻重視的：「Real-world measurements of structured-lattices and supersingular isogenies in TLS」、「The TLS Post-Quantum Experiment」。

Google Chrome 這邊是使用了 Canary 與 Dev 兩個 channel，有控制組與兩個新的演算法：

Google Chrome installs, on Dev and Canary channels, and on all platforms except iOS, were randomly assigned to one of three groups: control (30%), CECPQ2 (30%), or CECPQ2b (30%). (A random ten percent of installs did not take part in the experiment so the numbers only add up to 90.)

這兩個演算法有優點也有缺點。一個是 key 比較小，但運算起來比較慢 (SIKE，CECPQ2b)；另外一個是 key 比較大，但是運算比較快 (HRSS，CECPQ2)：

For our experiment, we chose two algorithms: isogeny-based SIKE and lattice-based HRSS. The former has short key sizes (~330 bytes) but has a high computational cost; the latter has larger key sizes (~1100 bytes), but is a few orders of magnitude faster.

We enabled both CECPQ2 (HRSS + X25519) and CECPQ2b (SIKE/p434 + X25519) key-agreement algorithms on all TLS-terminating edge servers.

感覺還是會繼續嘗試，因為這兩個演算法的缺點都還是有點致命...

Adobe Security Team 直接把 Private Key 貼到網誌上面...

Security Team 出這種包...：「In spectacular fail, Adobe security team posts private PGP key on blog」。

Adobe 這次的事情要怎麼說呢，hmmm...

Facebook 與 Google Chrome 以及 Firefox 的人合作降低 Reload 使用的資源

Facebook 花了不少時間對付 reload 這件事情：「This browser tweak saved 60% of requests to Facebook」。

Facebook 的人發現有大量對靜態資源的 request 都是 304 (not modified) 回應：

In 2014 we found that 60% of requests for static resources resulted in a 304. Since content addressed URLs never change, this means there was an opportunity to optimize away 60% of static resource requests.

而 Google Chrome 很明顯偏高：

於是他們找出原因後，發現 Google Chrome 只要 POST 後的頁面都會 revalidate：

A piece of code in Chrome hinted at the answer to our question. This line of code listed a few reasons, including reload, for why Chrome might ask to revalidate resources on a page. For example, we found that Chrome would revalidate all resources on pages that were loaded from making a POST request.

然後在討論後認為這個行為不必要，就修掉了，可以看到降了非常多：

We worked with Chrome product managers and engineers and determined that this behavior was unique to Chrome and unnecessary. After fixing this, Chrome went from having 63% of its requests being conditional to 24% of them being conditional.

但還是很明顯比起其他瀏覽器偏高不少，在追問題後發現當輸入同樣的 url 時 (像是 Ctrl-L 或是 Cmd-L 然後直接按 enter)，Google Chrome 會當作 reload：

The fact that the percentage of conditional requests from Chrome was still higher than other browsers seemed to indicate that we still had some opportunity here. We started looking into reloads and discovered that Chrome was treating same URL navigations as reloads while other browsers weren't.

不過這次推出修正後發現沒有大改變：(拿 production 測試 XDDD)

Chrome fixed the same URL behavior, but we didn't see a huge metric change. We began to discuss changing the behavior of the reload button with the Chrome team.

後來是針對 reload button 的行為修改，max-age 很長的就不 reload，比較短的就 reload。算是一種 workaround：

There was some debate about what to do, and we proposed a compromise where resources with a long max-age would never get revalidated, but that for resources with a shorter max-age the old behavior would apply. The Chrome team thought about this and decided to apply the change for all cached resources, not just the long-lived ones.

Google 也發了一篇說明這個新功能：「Reload, reloaded: faster and leaner page reloads」。

當 Facebook 的人找 Firefox 的人時，Firefox 決定另外定義哪些東西在 reload 時不需要 revalidate，而不像 Google Chrome 的 workaround：

Firefox chose to implement this directive in the form of a cache-control: immutable header.

Firefox 的人也寫了一篇「Using Immutable Caching To Speed Up The Web」解釋這個新功能。

所以之後規劃前後端的架構時又有東西要考慮進去...

利用隱藏的 form input 加上自動完成功能取得敏感資料

anttiviljami/browser-autofill-phishing 這邊示範了怎麼用隱藏的 form input 與自動完成功能取得敏感資料。在這邊可以看到示範 (把 POST 丟到 httpbin 上看 response)。

想法不算困難，但好像也不是很好防... 關掉 autofill 是比較簡單的解法 (我是裝好瀏覽器就會關掉，不過好像很多人都喜歡用這個功能)，所以這個問題就丟回給這些 browser vendor 想了 :o