SourceHut 被 DDoS 後的報告

SourceHut 在 DDoS 後發表了報告:「SourceHut network outage post-mortem」。

這次的攻擊在 L3 層,直接塞爆 upstream bandwidth:

At around 06:00 UTC on January 10th, a layer 3 distributed denial-of-service (DDoS) attack began to target SourceHut’s PHL infrastructure.

上游 Cogent 選擇 null route 掉:

In response to the attack, Cogent announced null routes for our downstream AS, causing our PHL network to become unreachable both for SourceHut staff and the general public.

中間有試著問 Cloudflare 以及其他的方案,但依照他們的說法,費用上無法承受:

We initially researched a number of solutions, and spoke to Cloudflare in particular due to their ability to provide a rapid response to ongoing incidents. However, given our complex requirements, Cloudflare quoted us a figure which was not attainable within our financial means as a small company. Other options we researched (though we did not seek additional quotes) had similar economical constraints.

後來的解法是在 OVH 放 proxy server (搭配 OVH 的 DDoS 保護服務),然後導到沒有公開的 subnet:

However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS; this met our needs for end-to-end encryption as well as service over arbitrary TCP protocols.

GitHub 在還沒被 Microsoft 併購前 (2018 年) 也有被打的記錄,2015 年的時候 Google 有放一些資料,當年有寫一篇記錄下來:「Google 對 GitHub 先前遭受 GFW 的 DDoS 攻擊的分析」,不過當年這波是 L7 的。

另外 2016 年的時候 GitHub 也有整理一篇關於 TCP SYN flood 的阻擋方式,這個看起來比較接近這次的攻擊:「GitHub 對抗 TCP SYN Flood 的方式:synsanity」。

Route 53 支援 DNS64,以及 NAT Gateway 支援 NAT64

AWS 宣佈了一套機制,讓 IPv6-only 的機器可以連到 IPv4-only 的服務:「Let Your IPv6-only Workloads Connect to IPv4 Services」。

首先是 DNS64,針對只有 IPv4-only 的 A record 自動加上 AAAA record (如果已經有 AAAA record 的則不變),這邊提到的 64:ff9b::/96 是來自 DNS64 標準內的規範:

The DNS resolver first checks if the record contains an IPv6 address (AAAA record). If it does, the IPv6 address is returned. The IPv6 host can connect to the service using just IPv6. When the record only contains an IPv4 address, the Route 53 resolver synthesizes an IPv6 address by prepending the well-known 64:ff9b::/96 prefix to the IPv4 address.

再來就是 NAT Gateway 可以把 64:ff9b::/96 透過 NAT64 轉到 IPv4 network 上:

You may configure subnet routing to send all packets starting with 64:ff9b::/96 to the NAT gateway. The NAT gateway recognizes the IPv6 address prefix, extracts the IPv4 address from it, and initiates an IPv4 connection to the destination. As usual, the source IPv4 address is the IPv4 address of the NAT gateway itself.

由於有些 protocol 會帶 IP address 資訊,所以不能保證 NAT64 一定會動,但大多數的情況應該是可以解決,至少提供了 IPv6-only server 連到 IPv4-only network 上的方法...

AWS 增加 CloudFront 的 AWS-managed prefix list 讓管理者使用

看到 AWS 公告提供 CloudFront 的 origin subnet 資訊 (AWS-managed prefix list) 讓管理者可以用:「Amazon VPC now supports an AWS-managed prefix list for Amazon CloudFront」。

以往會自己去「AWS IP address ranges」這邊提供的 JSON 檔案定時撈出來再丟到 managed prefix list 裡面,這次的功能等於是 AWS 自己管理這個 prefix list 讓管理者使用。

馬上想的到的用途就是 HTTP/HTTPS port 了,只開放給 CloudFront 的伺服器存取:

Starting today, you can use the AWS managed prefix list for Amazon CloudFront to limit the inbound HTTP/HTTPS traffic to your origins from only the IP addresses that belong to CloudFront’s origin-facing servers. CloudFront keeps the managed prefix list up-to-date with the IP addresses of CloudFront’s origin-facing servers, so you no longer have to maintain a prefix list yourself.

要注意的是這不應該當作唯一的 ACL 手段,因為其他人也可以建立 CloudFront distribution 來穿透打進你的 origin server。

另外有個比較特別的地方,這個 prefix list 的權重很重,使用他會算 55 條 rule 的量,在 security group 內很容易撞到 60 條的限制,在 route table 裡面則是直接撞到 50 條的限制;不過這兩個限制都可以跟 AWS 申請調昇:

The Amazon CloudFront managed prefix list weight is unique in how it affects Amazon VPC quotas:

  • It counts as 55 rules in a security group. The default quota is 60 rules, leaving room for only 5 additional rules in a security group. You can request a quota increase for this quota.
  • It counts as 55 routes in a route table. The default quota is 50 routes, so you must request a quota increase before you can add the prefix list to a route table.

如果 HTTP 一條,HTTPS 也一條,那就會算 110 rules 了,有暴力的感覺...

AWS 推出 Amazon Route 53 Resolver DNS Firewall

長久以來的洞總算有比較好的方法補上了,AWS 推出了 Amazon Route 53 Resolver DNS Firewall:「Introducing Amazon Route 53 Resolver DNS Firewall」。

Route 53 Resolver 是 AWS 官方提供的 DNS Resolver,沒有特殊的設定的話通常會在 x.x.x.2 (/24 或是更大的網段),先前一直沒有辦法解決 data leak 的問題,也就是透過 DNS 把敏感資料從 private network 裡丟出去。

以前的作法是透過 security group 擋掉對 Route 53 Resolver 的流量 (或是透過 VPC 的 Firewall 擋),然後自己架設兩台 DNS resolver 過濾,現在 Route 53 Resolver 支援 DNS Firewall,提供 allowlist 與 blocklist 這兩個功能使用,總算是把這件事情解的比較乾淨了:

Route 53 Resolver DNS Firewall lets you create “blocklists” for domains you don’t want your VPC resources to communicate with via DNS. You can also take a stricter, “walled-garden” approach by creating “allowlists” that permit outbound DNS queries only to domains you specify. You can also create alerts for when outbound DNS queries match certain firewall rules, allowing you to test your rules before deploying for production traffic.

另外這次的 DNS Firwall 提供了兩組由 AWS 維護的清單讓人使用,包括了 malware 與 botnet:

Route 53 Resolver DNS Firewall offers two managed domain lists—malware domains and botnet command and control domains—enabling you to get started quickly with managed protections against common threats.

這樣省事多了...

Amazon Route 53 支援 DNSSEC

也是個大家等蠻久的功能,AWS 總算在 Amazon Route 53 上推出 DNSSEC 了:「Announcing Amazon Route 53 support for DNSSEC」。

他需要掛 AWS KMS,這部份會有一些費用在裡面,不過應該是還好...

不過 web console 目前有個明顯的缺點:透過 Route 53 註冊的網域,又用 Route 53 自家服務的情況下,設定 DNSSEC 的整合沒有做的很好,不能直接快速設定。

現在得自己設定演算法,然後複製 public key 到另外一邊,當你有一堆網域要設定的時候就會覺得很煩了...

Amazon Route 53 對地區的微調功能

Amazon Route 53 推出新功能,針對地區微調資源的比重:「Amazon Route 53 Traffic Flow Announces Support For Geoproximity Routing With Traffic Biasing」。

範例大致上說明了這個功能的能力,假設你在兩個點都有服務可以提供,你可以利用這個功能微調某個比率到某個點:

For example, suppose you have EC2 instances in the AWS US East (Ohio) region and in the US West (Oregon) region. When a user in Los Angeles browses to your website, geoproximity routing will route the DNS query to the EC2 instances in the US West (Oregon) region because it's closer geographically. If you want a larger portion of users in the middle of the United States to be routed to one region, you can specify a positive bias for that region, a negative bias for the other region, or both.

有點 CDN 的想法在裡面...

EdgeCast 提供的 DNS 服務:EdgeCast Route

Zite 上看到 EdgeCast 也要進入 DNS 服務這個市場了:「CDN Provider EdgeCast Gets Into The DNS Market With Launch Of EdgeCast Route」。

服務的頁面已經公開,並且公開價錢:「Managed DNS Provider | Outsourced DNS Service | Route | EdgeCast」,服務分成三種:

Standard Routing

利用 EdgeCast 的 IP Anycast Network 服務。每百萬個 query 是 USD$0.4 (超過十億個 query 的部份降到半價 USD$0.2)。

Adaptive Availability

除了 IP Anycast Network 外,還可以設定 health check 與 ratio (以達到 load sharing 的功能)。每百萬個 query 是 USD$0.6 (超過十億個 query 的部份降到半價 USD$0.3)。

Advanced Policy Routing

可以依照這些條件分析:GeoIP、GeoCountry、GeoCity、ASN、IP Group、Network Groups、Anycast PoPs 或是 IP Type。每百萬個 query 是 USD$1.5 (超過十億個 query 的部份降到 USD$1)。

另外價錢還有 zone 的部份。前面 50 個 zone 是 USD$50/month (算是低消吧?),後面每 50 個 zone 是 USD$35/month。而 health check 的部份是每個 USD$0.5/month。

可以設這麼細,而且又取這個名字,算是跟 Amazon Route 53 打對台?不過那個 Geo 系列以及 ASN 的部份看起來不賴啊,以前是自己寫 DNS resolver 處理這部份,把國外流量指到 CDN 上,然後台灣流量放在台灣的機房 (因為 CDN 不一定有台灣機房的 PoP)。

找機會來看看效果如何...