Amazon S3 改變 403 的收費方式

這算是一連串的故事,首先是四月底的時候「How an empty S3 bucket can make your AWS bill explode」這篇,提到了他一個晚上收到了 US$1,300 的帳單,因為有人 (沒有權限的人) 對他的 S3 bucket 狂打了 100M requests (一億筆),雖然都是 403 的 access denied,但還是得付 request 與頻寬的費用。

對於想要搞的人來說,us-east-1 的 Amazon S3 費用是 $0.005/1K requests (PUT, COPY, POST, LIST requests),換算大一點的單位是 $5/1M requests,拿個 ab 之類的工具超級簡單就可以打出破千 reqs/sec,如果是 k6 之類的工具,其實一台電腦就蠻容易打爆?

作者聯絡 AWS 客服後,客服回答你需要付這筆費用 (「這不是 bug,是 feature」):

Yes, S3 charges for unauthorized requests (4xx) as well[1]. That’s expected behavior.

然後這件事情就在社群傳開了,傳到 Jeff Barr 後直接公開提到他認為客戶不應該付 unauthorized request 的 cost (應該是先跟內部其他高層討論過了),等於是宣佈了會改掉:

不過這件事情之前應該就有人提過了,結果 Colin Percival 直接戳,他在 2006 年 Amazon S3 剛出來的時候就提過了:

Anyway,兩個禮拜過去後,剛剛看到宣佈收費方式修改:「Amazon S3 will no longer charge for several HTTP error codes」。

針對從不屬於自己帳號所產生的 403 不收費 (包括 request 與頻寬費用):

With this change, bucket owners will never incur request or bandwidth charges for requests that return an HTTP 403 (Access Denied) error response if initiated from outside their individual AWS account or AWS Organization.

然後多了一頁「Billing for Amazon S3 error responses」專門說明這件事情,這邊列的比較完整,除了 403 以外也包含了其他的 HTTP response code 是不收費的:

The current page shows a full list of HTTP 3XX and 4XX status codes that won't be billed.

補了一個 18 年的洞...

HP 印表機的 Port 與 Prometheus...

Twitter 上看到這個,HP 印表機的 Port 9100 跟 Prometheus 撞到,再加上 mistype,於是就出事了:

找了一下 HP 的文件,「HP Jetdirect Print Servers - HP Jetdirect Port Numbers for TCP/IP (UDP) Connections」:

9100 TCP port is used for printing. Port numbers 9101 and 9102 are for parallel ports 2 and 3 on the three-port HP Jetdirect external print servers.

翻了一下「Service Name and Transport Protocol Port Number Registry」這邊,看起來 HP 在很久前就登記了 9100/tcp 與 9100/udp...

不過這沒有誰對誰錯的問題,只是很好笑:Printer 在收到不認識的指令時會直接當做 text 印出來,加上 Prometheus 的 HTTP request 打進去...

這幾天 blog 被掃,用 nginx 的 limit_req_zone 擋...

Update:這個方法問題好像還是不少,目前先拿掉了...

這幾天 blog 被掃中單一頁面負載會比較重的頁面,結果 CPU loading 變超高,從後台可以看到常常滿載:

看了一下是都是從 Azure 上面打過來的,有好幾組都在打,IP address 每隔一段時間就會變,所以單純用 firewall 擋 IP address 的方法看起來沒用...

印象中 nginx 本身可以 rate limit,搜了一下文件可以翻到應該就是「Module ngx_http_limit_req_module」這個,就設起來暫時用這個方式擋著,大概是這樣:

limit_conn_status 429;
limit_req_status 429;
limit_req_zone $binary_remote_addr zone=myzone:10m rate=10r/m;

其中預設是傳回 5xx 系列的 service unavailable,但這邊用 429 應該更正確,從維基百科的「List of HTTP status codes」這邊可以看到不錯的說明:

429 Too Many Requests (RFC 6585)
The user has sent too many requests in a given amount of time. Intended for use with rate-limiting schemes.

然後 virtual host 的設定檔內把某個 path 放進這個 zone 保護起來,目前比較困擾的是需要 copy & paste try_filesFastCGI 相關的設定:

    location /path/subpath {
        limit_req zone=myzone;
        try_files $uri $uri/ /index.php?$args;

        include fastcgi.conf;
        fastcgi_intercept_errors on;
        fastcgi_pass php74;
    }

這樣一來就可以自動擋下這些狂抽猛送的 bot,至少在現階段應該還是有用的...

如果之後有遇到其他手法的話,再見招拆招看看要怎麼再加強 :o

拿 Cloudflare Workers 跑 Geolocation API

Hacker News Daily 上看到拿 Cloudflare Workers 跑 Geolocation API:「How to make simple Geolocation service」。

作者想要做一個很簡單的 Geolocation API,一開始的想法是在 AWS Lambda 上用 MaxMind 的資料,但 latency 偏高:

However, I quickly realized that the response time isn't what I've expected - on average the response took somewhere between from 200ms to 500ms. So I started looking for other options.

所以作者就想到是不是有有機會丟到 Cloudflare Workers 上,但發現 license 看起來是個問題,另外因為把 MaxMind 的資料庫丟進去,會超過 worker 的時間限制:

And for this case MaxMind offers GeoLite2 database, however you'll be in charge of hosting this database on your server and making regular updates of the database. You also need to make sure your project is compliant with MaxMind's License.

However, this solution had one really big caveat - MaxMind GeoLite2 database does not work on Cloudflare Workers due to some runtime limitations.

結果作者後來發現 Cloudflare Workers 上本身就會帶 Geoloation 資訊了,不需要另外拉 MaxMind 的資料查:

And after exploring their documentation, I realized that the Request object in function have an access to cf object, which contains some useful information about the visitor, including visitor's country!

另外我翻了一下價錢,主要是算次數的,看起來 Free Plan 就可以 100K/day (執行時間限制是 10ms),而如果是付費方案的話則是 USD$0.5/M (50ms),這樣對一些小專案來說,Free Plan 似乎是夠用了...

Amazon S3 要拿掉 Path-style 存取方式

Hacker News 上翻的時候翻到的公告:「Announcement: Amazon S3 will no longer support path-style API requests starting September 30th, 2020」。

現有的兩種方法,一種是把 bucket name 放在 path (V1),另外一種是把 bucket name 放在 hostname (V2):

Amazon S3 currently supports two request URI styles in all regions: path-style (also known as V1) that includes bucket name in the path of the URI (example: //s3.amazonaws.com/<bucketname>/key), and virtual-hosted style (also known as V2) which uses the bucket name as part of the domain name (example: //<bucketname>.s3.amazonaws.com/key).

這次要淘汰的是 V1 的方式,預定在 2020 年十月停止服務 (服務到九月底):

Customers should update their applications to use the virtual-hosted style request format when making S3 API requests before September 30th, 2020 to avoid any service disruptions. Customers using the AWS SDK can upgrade to the most recent version of the SDK to ensure their applications are using the virtual-hosted style request format.

Virtual-hosted style requests are supported for all S3 endpoints in all AWS regions. S3 will stop accepting requests made using the path-style request format in all regions starting September 30th, 2020. Any requests using the path-style request format made after this time will fail.

Amazon S3 提供更高的存取量...

AWS 宣佈提高了 Amazon S3 的效能:「Amazon S3 Announces Increased Request Rate Performance」。

每個 S3 prefix 都可以到 5500 RPS read 與 3500 RPS write:

Amazon S3 now provides increased performance to support up to 3,500 requests per second to add data and 5,500 requests per second to retrieve data, which can save significant processing time for no additional charge. Each S3 prefix can support these request rates, making it simple to increase performance exponentially.

舊的資料可以看「Request Rate and Performance Considerations」這邊,裡面沒有明講速度,但有提到如果超過 800 RPS read 與 300 RPS write 的門檻,建議開 case:

However, if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepare for the workload and avoid any temporary limits on your request rate.

不過如果有量的話,還是建議照著原來的 prefix 建議,打散處理會比較好,通常在前面的 CDN 通常可以跑簡單的 url rewrite 處理掉 (像是 CloudFront 自家或是 Cloudflare),像是把使用 unix timestamp (ms) 的 https://www.example.com/1531843366123.jpg 變成 https://www.example.com/6123/1531843366123.jpg,這樣可以讓 Amazon S3 的後端依照 prefix 打散 loading,避免當站愈來愈大的時候很難處理。

ALB 支援 Slow Start 了

這個功能在 ELB Classic 年代時有跟 AWS 提過,到 ALB 支援了 (總算...):「Application Load Balancer Announces Slow Start Support for its Load Balancing Algorithm」。

Application Load Balancers now support a slow start mode that allows you to add new targets without overwhelming them with a flood of requests. With the slow start mode, targets warm up before accepting their fair share of requests based on a ramp-up period that you specify.

然後時間可以設定,從 30 秒到 15 分鐘:

Slow start mode can be enabled by target group and can be configured for a duration of 30 seconds to 15 minutes. The load balancer linearly increases the number of requests sent to a new target in a target group up to its fair share during the slow start ramp-up window.

就之前的經驗來說,這在跑 PHP 的時候會很需要這個功能 (之前是在 F5 的設備上設定)。其他的語言因為性質不太一樣,可能不會這麼吃這個功能。

主要是因為 PHP 是在 request 進來時 compile 並且 cache。所以在機器剛起來時,儘量將 CPU 留給 opcache,把常用的頁面 compile 完並且放進 cache,而不是讓大量的連線灌進來,這樣對使用體驗不會太好... (要避免 CPU 吃滿 100% 很久,造成每個連線都很慢才跑完)

AWS 推出 Slow Start 後對 auto scaling 時的順暢度會好不少...

HTTPS Everywhere 改變更新 Ruleset 機制,變成定時更新...

HTTPS Everywhere 是我很喜歡的一個套件,裡面有 Ruleset,會將 Ruleset 表內認定有支援 HTTPS 網站的 HTTP request 都改成 HTTPS,這可以降低被攔截的風險。像是網站雖然有 HSTS 但第一次連線時走 HTTP 的情況,以及網站本身有支援 HTTPS 但沒有設定 HSTS 時,在網址列上誤打 HTTP 版本的情況。

先前版本的 Ruleset 是隨著軟體更新時,包在軟體內一起更新。這樣的缺點是更新速度比較慢,但好處是不需要伺服器端,而且隱私性也比較高。而現在 EFF 決定還是要推出線上更新的版本,以加速 Ruleset 更新的速度:「HTTPS Everywhere Introduces New Feature: Continual Ruleset Updates」。

We've modified the extension to periodically check in with EFF to see if a new list is available.

而頻寬的部份由 Fastly 贊助:

If you haven't already, please install and contribute to HTTPS Everywhere, and consider donating to EFF to support our work!

如果對這點有疑慮的,也還是可以關掉 auto updater 避免洩漏資訊給 EFF 或是 Fastly。

WebKit 對 HSTS Super Cookie 提出的改法

Twitter 上看到 WebKitHSTS 所產生的 Super Cookie 提出的改善方案:

拿原文的例子來說明,先指定一個隨機數給 user,像是 8396804 (二進位是 100000000010000000000100),所以就存取下面的網址:

https://bit02.example.com
https://bit13.example.com
https://bit23.example.com

在存取這些 HTTPS 網址時都會指定 HSTS,所以之後連到這三個網址的 HTTP request 就不會觸發到 HTTP 版本,會因為 HSTS 被轉到 HTTPS 版本。於是就可以用 32 個 HTTP request 測試 32bits 而判斷出身份。(當然你可以用更多)

WebKit 提出的改善方案大概有幾種,主要是就觀察到的現象來限制。

第一種解法「Mitigation 1: Limit HSTS State to the Hostname, or the Top Level Domain + 1」是因為會看到這樣的設計:

https://a.a.a.a.a.a.a.a.a.a.a.a.a.example.com
https://a.a.a.a.a.a.a.a.a.a.a.a.example.com
https://a.a.a.a.a.a.a.a.a.a.a.example.com
https://a.a.a.a.a.a.a.a.a.a.example.com
https://a.a.a.a.a.a.a.a.a.example.com
https://a.a.a.a.a.a.a.a.example.com
https://a.a.a.a.a.a.a.example.com
…etc...
https://bit00.example.com
https://bit01.example.com
https://bit02.example.com
...etc...
https://bit64.example.com

所以提出的方案是只有目前網站的 domain 以及 top domain + 1 (像是 example.com) 可以被設定 HSTS:

Telemetry showed that attackers would set HSTS across a wide range of sub-domains at once. Because using HSTS in this way does not benefit legitimate use cases, but does facilitate tracking, we revised our network stack to only permit HSTS state to be set for the loaded hostname (e.g., “https://a.a.a.a.a.a.a.a.a.a.a.a.a.example.com”), or the Top Level Domain + 1 (TLD+1) (e.g., “https://example.com”).

但其實廣告主只需要註冊 32 domains (或是 64) 就可以避開這個問題。

第二種是「Mitigation 2: Ignore HSTS State for Subresource Requests to Blocked Domains」,如果在 HTTPS 頁面上,某個 domain 的 cookie 已經因為某些原因被阻擋 (像是手動設定),那麼就忽略掉 HSTS 的設計:

We modified WebKit so that when an insecure third-party subresource load from a domain for which we block cookies (such as an invisible tracking pixel) had been upgraded to an authenticated connection because of dynamic HSTS, we ignore the HSTS upgrade request and just use the original URL. This causes HSTS super cookies to become a bit string consisting only of zeroes.

後面這點在現在因為 SEO 設計而使得各大網站都往 HTTPS 方向走,應該會有些幫助吧...