GitHub 在 2/28 遭受的攻擊...

GitHub 在 2/28 遭受 DDoS 攻擊,蠻快就把事故報告丟出來了:「February 28th DDoS Incident Report」。

不過跟 GitHub 其他文章不太一樣,這篇算是 PR 稿吧,簡單來說就是花錢買 Akamai Prolexic 的過濾服務解決... Akamai 方的 PR 稿則是在「Memcached-fueled 1.3 Tbps attacks - The Akamai Blog」這邊可以看到。

17:21 UTC 發現問題,然後判斷超過 100Gbps,所以 17:26 決定讓 Akamai Prolexic 接管過濾:

At 17:21 UTC our network monitoring system detected an anomaly in the ratio of ingress to egress traffic and notified the on-call engineer and others in our chat system. This graph shows inbound versus outbound throughput over transit links:

Given the increase in inbound transit bandwidth to over 100Gbps in one of our facilities, the decision was made to move traffic to Akamai, who could help provide additional edge network capacity. At 17:26 UTC the command was initiated via our ChatOps tooling to withdraw BGP announcements over transit providers and announce AS36459 exclusively over our links to Akamai. Routes reconverged in the next few minutes and access control lists mitigated the attack at their border. Monitoring of transit bandwidth levels and load balancer response codes indicated a full recovery at 17:30 UTC. At 17:34 UTC routes to internet exchanges were withdrawn as a follow-up to shift an additional 40Gbps away from our edge.

就這樣而已,完全就是 PR 稿 XDDD

Akamai 阻擋 DDoS 能力的上限

這應該是最近在看 DDoS 事件中比較重要的新聞了,從這次的事件知道 Akamai 沒有能力擋下某種 620Gbps 以上的 DDoS 攻擊,而這是攻擊者已經有能力「示範」出來的量:「Akamai kicked journalist Brian Krebs' site off its servers after he was hit by a 'record' cyberattack」。

The assault has flooded Krebs' site with more than 620 gigabits per second of traffic — nearly double what Akamai has seen in the past.

然後現在 Krebs on Security 的整個站台都轉移到 GoogleProject Shield 計畫上了,接下來就是時間的考驗了:「The Democratization of Censorship」。

GitHub 對抗 TCP SYN Flood 的方式:synsanity

GitHub 提出了自己對抗 TCP SYN Floord 的方式:「SYN Flood Mitigation with synsanity」。

synsanity 是一個 netfilter (iptables) 用的 target,利用現有的理論阻擋 TCP SYN Flood 這種 DDoS:

synsanity is a netfilter (iptables) target for high performance lockless SYN cookies for SYN flood mitigation, as used in production at GitHub.

前人的作法 (SYNPROXY) 以 module 形式運作,需要過濾每一個封包,而這在 GitHub 這種規模上會導致效能不足並且 kernel panic:

This is quite an intrusive way of solving the problem since it touches every packet during the entire connection, but it does successfully mitigate SYN floods. Unfortunately we found that in practise under our load and with the amount of malformed packets we receive, it quickly broke down and caused a kernel panic.

GitHub 所開發的 synsanity 則是透過 netfilter (iptables) 的 target,只處理 initial packets,在撰寫的時候考慮多 CPU 的 lock 問題:

Slack 把服務丟上 CloudFront...

剛剛發現公司的 Slack 網域 已經上了 CloudFront,測了一些其他的 domain,應該是全部都上了:

gslin@home [~] [15:55/W4] mtr --report
Start: Tue May 17 15:56:06 2016
HOST:              Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|--      0.0%    10   16.3  11.7   9.3  18.8   3.1
  2.|--        0.0%    10   44.5  13.4   9.0  44.5  10.9
  3.|--        0.0%    10    8.8  12.7   8.8  26.8   5.9
  4.|--        0.0%    10   12.0  14.3  10.4  19.2   3.1
  5.|--         0.0%    10    9.8  10.4   9.0  11.4   0.5
  6.|-- 203-75-228-29.HINET-IP.hi  0.0%    10   10.3  10.6   9.2  13.7   1.2
  7.|--      0.0%    10   12.1  11.6  10.5  13.2   0.5
  8.|-- 202-133-255-122-static.un  0.0%    10   11.3  10.6   9.8  11.8   0.5
  9.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 10.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 11.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 12.|-- server-54-230-215-138.tpe  0.0%    10   11.9  11.2  10.1  12.2   0.3

這樣先前遇到從 HiNetLevel3 常常爛掉的問題應該是可以解決不少?

Linode 的被攻擊報告

Linode 這陣子一直被 DDoS 攻擊,前幾天放出報告:「The Twelve Days of Crisis – A Retrospective on Linode’s Holiday DDoS Attacks」。

其中這段提到了一些數字,Linode 有個小機房有 40Gbps 的能力,但以現在的 DDoS 規模會馬上爆掉:

Linode’s capacity management strategy for IP transit has been simple: when our peak daily utilization starts approaching 50% of our overall capacity, then it’s time to get more links.

This strategy is standard for carrier networks, but we now understand that it is inadequate for content networks like ours. To put some real numbers on this, our smaller datacenter networks have a total IP transit capacity of 40Gbps. This may seem like a lot of capacity to many of you, but in the context of an 80Gbps DDoS that can’t be blackholed, having only 20Gbps worth of headroom leaves us with crippling packet loss for the duration of the attack.

另外把 DNS 整個放上 CloudFlare 讓他們來擋:

Our nameservers are now protected by Cloudflare, and our websites are now protected by powerful commercial traffic scrubbing appliances.

後續的改善應該還要幾個月?完成後應該會再看到 blog post...

拿 Openvirtuals 的主機跑 Syncthing...

Low End Box 上逛到的主機商 Openvirtuals,在 LEB 上看到的優惠已經沒了,但點進去後看到 Buffalo 的主機年繳有 50% off,加上硬碟空間又大,就決定弄一台玩玩...

SSD-CACHED 的 Mini 是 256MB RAM + 512MB vSwap 以及 90GB 空間,要 USD$16/year,而 Standard 的都是兩倍,但只要 USD$20/year,就決定買 Standard 了...

後台的功能比想像中完整,這是系統資訊與狀態的畫面,功能其實不比 DigitalOcean 差 (不過畫面就普普通通了):

裝了 Ubuntu 14.04 64bits 跑,不過 Linux kernel 偏舊了點,是 2.6.32,查了一下維基百科上的資料,應該是 2009 年底的版本,是目前唯一一個 2.6 上有繼續維護的版本:

Linux two 2.6.32-042stab108.2 #1 SMP Tue May 12 18:07:50 MSK 2015 x86_64 x86_64 x86_64 GNU/Linux

網路的部份,實際測試時發現不是很穩定,HiNet 過去有時候會有不低的 packet loss,可能是中間有線路因為 DDoS 造成不穩定。

反正只是要跑 Synthing 也還好,就先這樣丟著... 上面順便跑個 rtorrent 幫忙 Ubuntu 分擔 ISO Image。

China's Great Cannon:中華大加農

在「China’s Great Cannon」這篇文章裡面把最近 GitHub 被攻擊的事件所使用的武器稱為 China’s Great Cannon。文章裡分析了攻擊的方式、造成的影響。

不過... 我只是對取名的人讚嘆而已,可以參考偽基百科的「先行者」條目 XDDD

CloudFlare 支援 WebSockets

CloudFlare 的官方 blog 上的公告:「CloudFlare Now Supports WebSockets」。

於是 CloudFlare 自豪的 DDoS 防護服務也涵蓋到 WebSockets 了:

The ability to protect and accelerate WebSockets has been one of our most requested features.

裡面其實還提到一些 CDN + WebSockets 的技術問題 (像是 port 的數量),有興趣的可以再仔細看 :o

BPF (Berkeley Packet Filter)

看到 CloudFlare 的「BPF - the forgotten bytecode」在文章裡提到 BPF (Berkeley Packet Filter),發現從大學畢業後就沒再看過... (然後也沒什麼印象了)

tcpdump 可以把 expression 轉成 BPF bytecode,再丟進 kernel 執行,拿 CloudFlare 文章裡的例子在自己電腦上跑:

gslin@GSLIN-DESKTOP [~] [07:27/W4] sudo tcpdump -p -ni eth1 -d "ip and udp"
(000) ldh      [12]
(001) jeq      #0x800           jt 2    jf 5
(002) ldb      [23]
(003) jeq      #0x11            jt 4    jf 5
(004) ret      #65535
(005) ret      #0

而對於複雜的過濾邏輯而需要拼效能時,可能會需要手動寫 bytecode (像是優先先判斷某些比較容易過濾的欄位,藉以降低判斷的量),可以透過 SOCK_RAWSO_ATTACH_FILTER 直接寫 bytecode 給 kernel 執行。

雖然文章內沒有明講,不過看起來 CloudFlare 有這樣做,尤其後面又有提到:

These kind of rules are very useful, they allow us to pinpoint the malicious traffic and drop it early. Just in the last couple of weeks we dropped 870,213,889,941 packets with few BPF rules. Recently during a flood we saw 41 billion packets dropped throughout a night due to a single well placed rule.


利用 Facebook Notes 對圖片 cache 的特性發動 DDoS 攻擊

在「Using Facebook Notes to DDoS any website」這篇文章裡提到了利用 Facebook Notes 允許使用者嵌入 <img> 標籤時的特性,利用 Facebook 的 server 進行 DDoS...

在 Notes 一般的 <img> 會被 Facebook 的伺服器 cache 起來,但如果是帶有 query string 的 <img> 就不會 cache (因為不同的 query string 表示不同的 url 是合理的),於是就可以利用這個特性打出超高的流量:

這個問題被 Facebook 認為不是問題,不會也不打算修正...

文章後提到的指令還蠻有趣的,要抓出某個 AS number 有哪些 IP address,可以用這樣的指令抓出來:

whois -h — '-i origin AS32934' | grep ^route

試著抓了 AS9916 與 AS18185,的確是蠻有趣的東西 XD