Go 的 net/http 在 1.22 的 routing 新功能

Gonet/http 在 1.22 引入了更方便的 pattern matching:「Routing Enhancements for Go 1.22」。

用官方的範例,現在可以處理路徑裡的參數了:

http.Handle("GET /posts/{id}", handlePost2)

後續可以透過 PathValue() 取出來:

idString := req.PathValue("id")

而優先順序則是依照吻合度定義:

The precedence rule is simple: the most specific pattern wins. This rule matches our intuition that posts/latests should be preferred to posts/{id}, and /users/{u}/posts/latest should be preferred to /users/{u}/posts/{id}. It also makes sense for methods. For example, GET /posts/{id} takes precedence over /posts/{id} because the first only matches GET and HEAD requests, while the second matches requests with any method.

但是當有重疊卻無法判斷相對吻合度的 rule 被加進去時,會直接 panic()

What if two patterns overlap but neither is more specific? For example, /posts/{id} and /{resource}/latest both match /posts/latest. There is no obvious answer to which takes precedence, so we consider these patterns to conflict with each other. Registering both of them (in either order!) will panic.

這的確是種方法啦... 而且留有之後處理的空間,真的有好的方法就可以把 panic() 的邏輯改成新的共識。

德國法院認為 DNT header 具有法律的告知效力

HN 上面看到「German court declares Do Not Track to be legally binding (vzbv.de)」這個消息,原文是德文:「Gericht untersagt Datenschutzverstöße von LinkedIn」,Google Translation 翻譯的結果:「Court bans LinkedIn data protection violations」。

LinkedIn 告知使用者他們不會理會 DNT,德國法院則是認為 DNT header 是已經告知對方不願意被追蹤了:

„Wenn Verbraucher:innen die ,Do-Not-Track‘-Funktion ihres Browsers aktivieren, ist das eine klare Botschaft: Sie wollen nicht, dass ihr Surfverhalten für Werbe- und andere Zwecke ausgespäht wird“, sagt Rosemarie Rodden, Rechtsreferenin beim vzbv. „Webseitenbetreiber müssen dieses Signal respektieren.“

“When consumers activate the 'Do Not Track' function of their browser, it sends a clear message: They do not want their surfing behavior to be spied on for advertising and other purposes,” says Rosemarie Rodden, legal officer at vzbv. “Website operators must respect this signal.”

這好像是第一次看到 DNT 相關的法律判決?可以看看後續有沒有新的消息 (上訴之類的),來看看最終的判決會是怎麼樣。

Uptime Kuma:UptimeRobot 的 Open Source Clone

UptimeRobot 算是我自己還蠻喜歡用的監控服務,各種常見的類型都支援,像是 HTTP/HTTPS website 的監控,ICMP Ping 或是 TCP 的監控,另外也提供 SSL certificate 的監控。

前陣子找資料的時候找到 Uptime Kuma,這是香港的 Louis Lam 開發的 open source software,光看名字就懷疑是 UptimeRobot 的致敬版本,裝起來用過後就更確定...

差不多用了一個多月了,基本上功能還蠻完整的,我把他接到我自己的 Slack 以及 Pushover 上,就很方便的可以監控服務:

挑一個點:

ISP 偽造出合法的 SSL certificate,對放在德國的 xmpp.ru 進行 MITM 監聽

標題有點複雜,先講一下 http-01 認證,這是目前 Let's Encrypt 上最常被使用的認證方式,是透過 HTTP 協定完成認證,你只要能回答 http://www.example.com/.well-known/acme-challenge/XXX 的內容就能過。

一般來說,這個 HTTP 位置只有這台伺服器的 owner 才有辦法提供,也就能確保不是任何人都可以申請。

但因為這邊走的是 HTTP,對於 ISP 這種比較特別的身分來說,他可以從中架設 HTTP 的 MITM Proxy 做到這件事情。

而有了合法的 SSL certificate 之後,中間的 MITM Proxy 就不只能聽 HTTP 了,還可以聽 HTTPS 的內容 (甚至修改內容)。

這次發生的事情就是在德國的 LinodeHetzner 機房內的服務,俄羅斯最大的 XMPP 服務 xmpp.ru 被搞出這件事情 (XMPP 是一個開放協定,不熟的話可以想像成類似 Line 或是 Telegram 的服務,但 XMPP 是開放協定,可以用自己喜歡的軟體連上):「Encrypted traffic interception on Hetzner and Linode targeting the largest Russian XMPP (Jabber) messaging service」。

他們在 Hetzner 的伺服器上有發現 network offline 的訊號:

[Tue Jul 18 12:58:29 2023] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Down
[Tue Jul 18 12:58:48 2023] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

而在這個 network offline 的時間不久後 Let's Encrypt 發出了 xmpp.ru 與 jabber.ru 的 SSL certificate (crt.sh 上可以查到,在 99976947049997621208):

18 July 2023 issuing time is about the same when Hetzner server has lost network link for several seconds.

這些徵兆符合改接到 MITM Proxy 上的行為。

這次的事情很大條,因為這些伺服器是在德國,不是在俄羅斯... 事情才剛開始被報導出來,後續得繼續追蹤,而且應該也會促成新的機制被引入?

下載 YouTube 影片的技術限制與繞過方法

Hacker News 上看到這篇「How They Bypass YouTube Video Download Throttling」在講 YouTube 防止下載的各種方式。

透過 API 拿到的 URL 直接抓很慢,大約 40-70KB/sec:

However, attempting to download from this URL leads to really slow download:

The speed is always limited to around 40-70kB/s.

這邊需要一個 javascript 環境計算出 n,帶入後續的 request 以「證明」你是官方的網頁 client:

Since mid-2021, YouTube has included the query parameter n in the majority of file URLs. This parameter needs to be transformed using a JavaScript algorithm located in the file base.js, which is distributed with the web page. YouTube utilizes this parameter as a challenge to verify that the download originates from an “official” client. If the challenge is not resolved and n is not transformed correctly, YouTube will silently apply throttling to the video download.

The JavaScript algorithm is obfuscated and changes frequently, so it’s not practical to attempt reverse engineering to understand it. The solution is simply to download the JavaScript file, extract the algorithm code, and execute it by passing the n parameter to it. The following code accomplishes this.

但即使算出 n,也還是會限速,可以看到作者策出來大約是 4MB/sec,雖然比以前快很多了,但還是看得出來有限速。這主要是避免 client 端過度 buffer 浪費頻寬:

With this new URL containing the correctly transformed n parameter, the next step is to download the video. However, YouTube still enforces a throttling rule. This rule imposes a variable download speed limit based on the size and length of the video, aiming to provide a download time that’s approximately half the duration of the video. This aligns with the streaming nature of videos. It would be a massive waste of bandwidth for YouTube to always provide the media file as quickly as possible.

接下來的方式就是利用 Range 拆成很多個 HTTP request 打,這樣因為 buffering algorithm 在開始限速前會先全速塞資料給你,就可以用這點避開限速的問題了。

把多的 request 與處理時間都算進去後,整體大約可以到 50-70MB/sec,算是可以接受的下載速度了:

However, the average speeds typically ranged between 50-70 MB/s or 400-560 Mb/s, which is still pretty fast.

後面有一些合併處理的指令 (因為 YouTube 會把影與音分離成兩個檔案),就不是重點了...

Cloudflare 看 HTTP/3 的使用率

Cloudflare 利用自家平台分析過去一年 HTTP/3 的使用率:「Examining HTTP/3 usage one year on」。

首先是整體的使用率,看起來反而沒有什麼太大的變化?一上線就已經是巔峰的感覺?

然後各種 bot (像是搜尋引擎的 bot 或是 social media 的 bot) 看起來幾乎都沒有用 HTTP/3,少數的量應該都是實驗性質居多,唯一的例外是 LinkedIn 有試著在導入,可以看到慢慢的爬升:

看起來不算太順?

nginx 開始嘗試支援 HTTP/3

Hacker News 上看到「Nginx 1.25.0: experimental HTTP/3 support (nginx.org)」這則消息,從 nginx 1.25.0 開始可以用 HTTP/3

HTTP/2 最大的差異就是從以往的 TCP 改到 UDP 上了,這是基於 QUIC 的經驗弄出來的東西...

nginx 的支援算是等了一陣子了,不過沒有當初 HTTP/1.1SPDY 的進步這麼明顯,我自己就沒有跟的那麼緊了。

這樣以後 office firewall 預設應該會再開 443/udp?

網頁大小 14KB 與 15KB 的速度差異

Hacker News 上看到「Why your website should be under 14kB in size」這篇,對應的討論在「A 14kb page can load much faster than a 15kb page (endtimes.dev)」,在講網頁大小 14KB/15KB 的速度差異比 15KB/16KB 大很多的問題:

What is surprising is that a 14kB page can load much faster than a 15kB page — maybe 612ms faster — while the difference between a 15kB and a 16kB page is trivial.

原因是 TCP slow start 造成的:

This is because of the TCP slow start algorithm.

而網頁這邊 TCP slow start 目前大多數的實做都是 10 packets 後發動:

Most web servers TCP slow start algorithm starts by sending 10 TCP packets.

然後再組合 1500 bytes/packet 以及 overhead,就差不多是 14KB 了:

The maximum size of a TCP packet is 1500 bytes.

This this maximum is not set by the TCP specification, it comes from the ethernet standard

Each TCP packet uses 40 bytes in its header — 16 bytes for IP and an additional 24 bytes for TCP

That leaves 1460 bytes per TCP packet. 10 x 1460 = 14600 bytes or roughly 14kB!

然後 HTTP/3 也可以看到類似的設計 (出自「QUIC Loss Detection and Congestion Control」:

Sending multiple packets into the network without any delay between them creates a packet burst that might cause short-term congestion and losses. Implementations MUST either use pacing or limit such bursts to the initial congestion window, which is recommended to be the minimum of 10 * max_datagram_size and max(2* max_datagram_size, 14720)), where max_datagram_size is the current maximum size of a datagram for the connection, not including UDP or IP overhead.

算是一個小知識... 但對於現在肥滋滋的網頁效果來說就沒辦法了,而且考慮到大一點的網站會在一個 TCP 連線裡面可能會傳很多 request,其實早就超過 TCP slow start 的門檻了。

CloudFront 支援 HTTP/3

雖然 HTTP/3 還沒有進到 Standard Track,但看到 CloudFront 宣佈支援 HTTP/3 了:「New – HTTP/3 Support for Amazon CloudFront」。

只要在 CloudFront 的 console 上勾選起來就可以了:

看了看 RFC 9114: HTTP/3 文件裡的描述,client 可以試著建立 UDP 版本的 QUIC 連線,但要有機制在失敗時回去用 TCPHTTP/2 或是 HTTP/1.1

A client MAY attempt access to a resource with an "https" URI by resolving the host identifier to an IP address, establishing a QUIC connection to that address on the indicated port (including validation of the server certificate as described above), and sending an HTTP/3 request message targeting the URI to the server over that secured connection. Unless some other mechanism is used to select HTTP/3, the token "h3" is used in the Application-Layer Protocol Negotiation (ALPN; see [RFC7301]) extension during the TLS handshake.

Connectivity problems (e.g., blocking UDP) can result in a failure to establish a QUIC connection; clients SHOULD attempt to use TCP-based versions of HTTP in this case.

另外一條路是在 TCP 連線時透過 HTTP header 告訴瀏覽器升級:

An HTTP origin can advertise the availability of an equivalent HTTP/3 endpoint via the Alt-Svc HTTP response header field or the HTTP/2 ALTSVC frame ([ALTSVC]) using the "h3" ALPN token.

像是這樣:

Alt-Svc: h3=":50781"

然後 client 就可以跑上 HTTP/3:

On receipt of an Alt-Svc record indicating HTTP/3 support, a client MAY attempt to establish a QUIC connection to the indicated host and port; if this connection is successful, the client can send HTTP requests using the mapping described in this document.

另外在 FAQ 裡面有提到啟用 HTTP/3 是不另外計費的,就照著本來的 request 費用算:

Q. Is there a separate charge for enabling HTTP/3?

No, there is no separate charge for enabling HTTP/3 on Amazon CloudFront distributions. HTTP/3 requests will be charged at the request pricing rates as per your pricing plan.

先開起來玩看看...