PyPy JIT 的改善

PyPy 這邊看到 JIT 的重大進展:「Better JIT Support for Auto-Generated Python Code」。

他們在 Tornado 上重製出來效能問題,後面也都是用這個例子在測試:

If you render a big HTML template (example) using the Tornado templating engine, the template rendering is really not any faster than CPython.

看起來上的 workaround 是在撞到 trace limit 時標記起來,之後再遇到時就可以跳進 special mode,接著處理下去避免浪費掉之前處理過的 trace:

After we have hit the trace limit and no inlining has happened so far, we mark the outermost function as a source of huge traces. The next time we trace such a function, we do so in a special mode. In that mode, hitting the trace limit behaves differently: Instead of stopping the tracer and throwing away the trace produced so far, we will use the unfinished trace to produce machine code.

效能可以看到改善很多:

看起來這個概念有打算在 3.8 的時候放進去:

The work described in this post tiny bit experimental still, but we will release it as part of the upcoming 3.8 beta release, to get some more experience with it. Please grab a 3.8 release candidate, try it out and let us know your observations, good and bad!

Django 的 template engine 不怎麼快,用 Jinja2 可能是一個方法,但既有的 project 如果有遇到 template engine 的效能問題,也許也可以翻看看 PyPy 解得如何...

EC2 的 Monitoring 提供更多關於限流的資訊

AWS 對於 EC2 的網路推出了五個新的監控指標:「Amazon EC2 announces new network performance metrics for EC2 instances」。

看起來都是跟限流有關的指標,看起來是在壓榨機器極限時會用到:

The new metrics inform customers in real time of network traffic impacted when instance allowances for inbound and outbound bandwidth, packets-per-second (PPS), connections tracked and PPS to link-local services are exceeded.

需要有最新的 ENA driver 才會提供 (看了一下現有的機器沒出現這些值 XD):

These metrics are available today in all global commercial AWS regions on instances running the latest version of the Elastic Network Adapter (ENA) driver with support for Linux, Windows ENA driver support will be available soon with version 2.2.2.0.

不另外收費:

They can be accessed from within the instance at no extra cost using simple command line tools.

這個功能在所有 AWS 商業區以及 GovCloud (US) 都已經上線:

Instance Level Network Performance Metrics is available in all AWS Commercial and GovCloud (US) Regions, with the exception of China (Beijing) and China (Ningxia).

先記錄起來就好,一般用法應該都還好...

這幾天 blog 被掃,用 nginx 的 limit_req_zone 擋...

Update:這個方法問題好像還是不少,目前先拿掉了...

這幾天 blog 被掃中單一頁面負載會比較重的頁面,結果 CPU loading 變超高,從後台可以看到常常滿載:

看了一下是都是從 Azure 上面打過來的,有好幾組都在打,IP address 每隔一段時間就會變,所以單純用 firewall 擋 IP address 的方法看起來沒用...

印象中 nginx 本身可以 rate limit,搜了一下文件可以翻到應該就是「Module ngx_http_limit_req_module」這個,就設起來暫時用這個方式擋著,大概是這樣:

limit_conn_status 429;
limit_req_status 429;
limit_req_zone $binary_remote_addr zone=myzone:10m rate=10r/m;

其中預設是傳回 5xx 系列的 service unavailable,但這邊用 429 應該更正確,從維基百科的「List of HTTP status codes」這邊可以看到不錯的說明:

429 Too Many Requests (RFC 6585)
The user has sent too many requests in a given amount of time. Intended for use with rate-limiting schemes.

然後 virtual host 的設定檔內把某個 path 放進這個 zone 保護起來,目前比較困擾的是需要 copy & paste try_filesFastCGI 相關的設定:

    location /path/subpath {
        limit_req zone=myzone;
        try_files $uri $uri/ /index.php?$args;

        include fastcgi.conf;
        fastcgi_intercept_errors on;
        fastcgi_pass php74;
    }

這樣一來就可以自動擋下這些狂抽猛送的 bot,至少在現階段應該還是有用的...

如果之後有遇到其他手法的話,再見招拆招看看要怎麼再加強 :o

AMD Ryzen Threadripper 3990X 在 Windows 上的效能

John Carmack 注意到在 AMD Ryzen Threadripper 3990X 上因為 Windows 的 group limit 限制而造成效能問題:

但這點可以透過打散到兩個 group 改善 (workaround) 而提昇速度:

然後順便看了一下目前 CPU Benchmark 網站上對於高階 CPU 的跑分數據「PassMark - CPU Mark High End CPUs)」,可以看到 AMD 最近真是香噴噴的,用 3950X (16C/32T,105W) 殺 Intel 目前最高分的 W-3275M (28C/56T,205W),然後那個價差:

Intel 的 14nm 牙膏繼續擠...

EC2 要從 Instance 數量限制改成 vCPU 數量限制

這算是 AWS 的保護機制,在 Amazon EC2 上能開的機器數量都是有限制的。

打算要用新的 vCPU 數量限制取代舊的 Instance 數量限制:「Using new vCPU-based On-Demand Instance limits with Amazon EC2」,然後現在可以先加入:「vCPU-based On-Demand Instance Limits are Now Available in Amazon EC2」。

這次改善的問題是,以往 m5.largem5.xlarge 是兩個不同的限制,所以用起來會比較卡,現在則改成用 vCPU 來管理。

這次的架構是改成,一般性的機器會有一個 vCPU 數量限制,其他不同特性的各自有自己的 vCPU 數量限制:

In addition to now measuring usage in number of vCPUs, there will only be five different On-Demand Instance limits—one limit that governs the usage of standard instance families such as A, C, D, H, I, M, R, T, and Z, and one limit per accelerated instance family for FPGA (F), graphic-intensive (G), general purpose GPU (P), and special memory optimized (X) instances.

9/24 可以先手動加入,會拿你現在的量會換算過去,然後 10/24 會全部都轉過去:

During a transition period from September 24, 2019, through October 24, 2019, you can opt in to receive vCPU-based instance limits. When you opt in, EC2 automatically computes your new limits, giving you access to launch at least the same number of instances (if not more) than you do currently. Beginning October 24, 2019, all accounts will switch to vCPU-based instance limits, and the current count-based instance limits will no longer be supported. Although the switchover will not impact your ability to launch EC2 instances, you should familiarize yourself with the new On-Demand Instance limits experience and opt into vCPU limits at a time of your choosing.

應該是會方便一些...

Stripe 遇到 AWS 上 DNS Resolver 的限制

當量夠大就會遇到各種限制...

這次 Stripe 在描述 trouble shooting 的過程:「The secret life of DNS packets: investigating complex networks」。

其中一個頗有趣的架構是他們在每台主機上都有跑 Unbound,然後導去中央的 DNS Resolver,再決定導去 Consul 或是 AWS 的 DNS Resolver:

Unbound runs locally on every host as well as on the DNS servers.

然後他們發現偶而會有大量的 SERVFAIL

接下來就是各種找問題的過程 (像是用 tcpdump 看情況,然後用 iptables 統計一些數字),最後發現是卡在 AWS 的 DNS Resolver 在 60 秒內只回應了 61,385 packets,換算差不多是 1,023 packets/sec,這數字看起來就很雷:

During one of the 60-second collection periods the DNS server sent 257,430 packets to the VPC resolver. The VPC resolver replied back with only 61,385 packets, which averages to 1,023 packets per second. We realized we may be hitting the AWS limit for how much traffic can be sent to a VPC resolver, which is 1,024 packets per second per interface. Our next step was to establish better visibility in our cluster to validate our hypothesis.

在官方文件「Using DNS with Your VPC」這邊看到對應的說明:

Each Amazon EC2 instance limits the number of packets that can be sent to the Amazon-provided DNS server to a maximum of 1024 packets per second per network interface. This limit cannot be increased. The number of DNS queries per second supported by the Amazon-provided DNS server varies by the type of query, the size of response, and the protocol in use. For more information and recommendations for a scalable DNS architecture, see the Hybrid Cloud DNS Solutions for Amazon VPC whitepaper.

iptables 看到的量則是:

找到問題後,後面就是要找方法解決了... 他們給了一個只能算是不會有什麼副作用的 workaround,不過也的確想不到太好的解法。

因為是查詢 10.0.0.0/8 網段反解產生大量的查詢,所以就在各 server 上的 Unbound 上指定這個網段直接問 AWS 的 DNS Resolver,不需要往中央的 DNS Resolver 問,這樣在這個場景就不會遇到 1024 packets/sec 問題了 XDDD

修改 Firefox 的 Pop Up 限制

這是在 Bazqux 使用時遇到的問題,我在 Bazqux 這邊看到想看的文章,會習慣用 c 鍵打開新的 tab,等下再一起看。在 Chrome 上面需要安裝 extension 才能開到背景,在 Firefox 上可以透過修改 about:config 裡的browser.tabs.loadDivertedInBackground,讓他預設開到背景 (雖然變成所有的行為都會到開到背景 tab,但我偏好這樣...)。

但在 Bazqux 用 c 鍵把連結打開到 tab 時,有時發現會跳出遇到開啟上限:

這其實很不方便... 所以故意連打測試發現是 20 個,找了一下資料發現是 dom.popup_maximum 這個值,改成 -1 就好了。這算是某種安全機制吧...

先繼續用看看 :o

Stripe 將 Redis 單機版轉到 Cluster 版本上降低了錯誤率

在「Scaling a High-traffic Rate Limiting Stack With Redis Cluster」這邊提到了 StripeRedis 單機版轉移到 10 個節點的 cluster 版本,然後錯誤率大幅下降:

Stripe’s rate limiters are built on top of Redis, and until recently, they ran on a single very hot instance of Redis. The server had followers in place for failover, but at any given time, one node was handling every operation.

We eventually solved it by migrating to a 10-node Redis Cluster.

另外也可以看出來,在轉移到 cluster 版本後有不少要注意的,像是因為 sharding 而需要調整平衡性。另外是 cluster 模式下寫入的 confirmation 跟一般預期的不太一樣,不過這對於 rate limit 的應用還好,可以接受某種程度的掉資料...

Amazon RDS 支援更大的硬碟空間與更多的 IOPS

Amazon RDS 的升級:「Amazon RDS Now Supports Database Storage Size up to 16TB and Faster Scaling for MySQL, MariaDB, Oracle, and PostgreSQL Engines」。

空間上限從 6TB 變成 16TB,而且可以無痛升。另外 IOPS 上限從 30K 變成 40K:

Starting today, you can create Amazon RDS database instances for MySQL, MariaDB, Oracle, and PostgreSQL database engines with up to 16TB of storage. Existing database instances can also be scaled up to 16TB storage without any downtime.

The new storage limit is an increase from 6TB and is supported for Provisioned IOPS and General Purpose SSD storage types. You can also provision up to 40,000 IOPS for Provisioned IOPS storage volumes, an increase from 30,000 IOPS.

不過隔壁的 Amazon Aurora 還是大很多啊 (64TB),而且實際上不用管劃多大,他會自己長大:

Q: What are the minimum and maximum storage limits of an Amazon Aurora database?

The minimum storage is 10GB. Based on your database usage, your Amazon Aurora storage will automatically grow, up to 64 TB, in 10GB increments with no impact to database performance. There is no need to provision storage in advance.

用 4.5+ 的 Linux Kernel 限制 I/O 速度

在「Using cgroups to limit I/O」這邊看到作者試著用 cgroups 限制 I/O 速度。

作者前面花了不少篇幅解釋 cgroups v1 無法正確限制 I/O 速度,後面就在講 cgroups v2 怎麼做:

So, in order to limit I/O when this I/O may hit the writeback kernel cache, we need to use both memory and io controllers in the cgroups v2!

這會需要 4.5+ 的 kernel,可能會需要手動更新,或是直接使用比較新的 distribution:

Since kernel 4.5, the cgroups v2 implementation was marked non-experimental.

然後照抄就可以了 (不過這邊的指定都需要 root,作者用 $ 表示 shell 有點怪):

# mount -t cgroup2 nodev /cgroup2
# mkdir /cgroup2/cg2
# echo "+io" > /cgroup2/cgroup.subtree_control
# echo "8:0 wbps=1048576" > io.max
# echo $$ > /cgroup2/cg2/cgroup.procs

然後就可以跑 dd 測試速度了,同時間也可以跑 iostat 看。