Home » Posts tagged "limit"

Amazon RDS 支援更大的硬碟空間與更多的 IOPS

Amazon RDS 的升級:「Amazon RDS Now Supports Database Storage Size up to 16TB and Faster Scaling for MySQL, MariaDB, Oracle, and PostgreSQL Engines」。

空間上限從 6TB 變成 16TB,而且可以無痛升。另外 IOPS 上限從 30K 變成 40K:

Starting today, you can create Amazon RDS database instances for MySQL, MariaDB, Oracle, and PostgreSQL database engines with up to 16TB of storage. Existing database instances can also be scaled up to 16TB storage without any downtime.

The new storage limit is an increase from 6TB and is supported for Provisioned IOPS and General Purpose SSD storage types. You can also provision up to 40,000 IOPS for Provisioned IOPS storage volumes, an increase from 30,000 IOPS.

不過隔壁的 Amazon Aurora 還是大很多啊 (64TB),而且實際上不用管劃多大,他會自己長大:

Q: What are the minimum and maximum storage limits of an Amazon Aurora database?

The minimum storage is 10GB. Based on your database usage, your Amazon Aurora storage will automatically grow, up to 64 TB, in 10GB increments with no impact to database performance. There is no need to provision storage in advance.

用 4.5+ 的 Linux Kernel 限制 I/O 速度

在「Using cgroups to limit I/O」這邊看到作者試著用 cgroups 限制 I/O 速度。

作者前面花了不少篇幅解釋 cgroups v1 無法正確限制 I/O 速度,後面就在講 cgroups v2 怎麼做:

So, in order to limit I/O when this I/O may hit the writeback kernel cache, we need to use both memory and io controllers in the cgroups v2!

這會需要 4.5+ 的 kernel,可能會需要手動更新,或是直接使用比較新的 distribution:

Since kernel 4.5, the cgroups v2 implementation was marked non-experimental.

然後照抄就可以了 (不過這邊的指定都需要 root,作者用 $ 表示 shell 有點怪):

# mount -t cgroup2 nodev /cgroup2
# mkdir /cgroup2/cg2
# echo "+io" > /cgroup2/cgroup.subtree_control
# echo "8:0 wbps=1048576" > io.max
# echo $$ > /cgroup2/cg2/cgroup.procs

然後就可以跑 dd 測試速度了,同時間也可以跑 iostat 看。

Twitter 打算放寬到 280 字...

Twitter 打算放寬 140 字限制:「Giving you more characters to express yourself」。

不過不包括日文、中文與韓文 XD

We want every person around the world to easily express themselves on Twitter, so we're doing something new: we're going to try out a longer limit, 280 characters, in languages impacted by cramming (which is all except Japanese, Chinese, and Korean).

然後也拿日文與英文當範例:

然後做了比較:

GitHub Apps (前身 GitHub Integrations) 的 Rate Limiting 變得更彈性

GitHub 宣佈了把 GitHub Integrations 改名為 GitHub Apps,另外 Rate Limiting 變得更彈性:「GitHub Apps (formerly Integrations) General Release」。

All GitHub Apps start with a rate limit of 5000 requests per hour. To facilitate growth we have added a dynamic rate limit for installations: organization installations with more than 20 users receive another 50 requests per hour for each user. Installations that have more than 20 repositories receive another 50 requests per hour for each repository.

所以對於超過 20 個使用者以及超過 20 個 repository 的 organization 都會增加 Rate Limiting。每個單位增加 50 requests/hour 不算太多,不過想一下好像也不少... (尤其對大一點的團體來說)

Stripe 對於控制 API 使用量的作法

在「Scaling your API with rate limiters」這篇 StripePaul Tarjan 提到了四種如何保護 API 的作法。

前兩種都是 rate limit。第一種是最標準的「你一分鐘可以用幾次」的方式,這是最容易理解的方式。第二種是「你同時間可以用幾個 API request」,這通常會用在大量消耗資源的 API 上,避免短時間內被打爆。

第三種是拉到整體來看,把 API 分成重要與不重要的,然後直接保留確保重要的 API 有一定的 capacity 可以用:

We always reserve a fraction of our infrastructure for critical requests. If our reservation number is 20%, then any non-critical request over their 80% allocation would be rejected with status code 503.

第四種方式是當過載時的自動化處理,平常就把各種工作排優先順序,當超量的時候自動先將低優先權的拿掉:

Only 100 requests were rejected this month from this rate limiter, but in the past it’s done a lot to help us recover more quickly when we have had load problems. This load shedder limits the impact of incidents that are already happening and provides damage control, while the first three are more preventative.

不過還是有點怪,Stripe 應該是全部都建在 AWS 上面 (AWS Case Study: Stripe),跟 auto scaling 的配合好像都沒提到?

歸類的方式還蠻有條理的,可以學這個方法來規劃...

Let's Encrypt 宣佈脫離 Beta

Let's Encrypt 宣佈脫離 beta,正式開放:「Leaving Beta, New Sponsors」。

翻資料的時候發現在今年 3/26 的時候,限制已經放寬了:「Rate Limits for Let’s Encrypt」。

首先一張證書只能包括 100 個 hostname,跟原來相同:

Names/Certificate is the limit on how many domain names you can include in a single certificate. This is currently limited to 100 names, or websites, per certificate issued.

再來是每個禮拜可以申請的數量從 5 個 hostname 變成 20 個,另外本來 renew 也算 quota,現在變成不會吃到 quota:

Certificates/Domain limits how many certificates can be issued that contain a single registered domain*.
This is limited to 20 certificates per domain per week. Exception: When you request a certificate with the same exact set of FQDNs as previously-issued certificate, this rate limit does not apply, but the one below does.

不知道會不會再放寬限制...

避免用 SQL 的 OFFSET 實做 Pager

發現以前沒有提到過...?

WordPress.com Developer Resources 寫的這篇「An efficient alternative to paging with SQL OFFSETs」提到了用 OFFSET 實做 Pager 的效率問題。

因為 RDBMS 是人寫的,所以人想不到的方法,程式也做不到,像是這樣的 SQL query 效率不會好:

SELECT * FROM my_table ORDER BY pk LIMIT 8000000, 100;

如果這是一般以 B+tree 儲存的 RDBMS (或是類似的資料結構),會先把 pk 拿出來,從最小的開始計算,掃過八百萬次後再去抓一百筆 pk 的值,最後再回到 my_table 內把所有資料拉出來。

這個方法當 offset 小的時候不會有感覺,但大了以後就會爆炸。就算 primary key 都在記憶體內,仍然是狂操 CPU L2/L3 以及記憶體的存取速度。

目前常見的分頁作法是在 url 上妥協,只提供「下一頁」或「下 n 頁」的功能。如此一來,url 變成:

https://www.example.com/blog?page=10&started_at=12345678
https://www.example.com/blog?page=11&started_at=12345690
https://www.example.com/blog?page=12&started_at=12345710

started_at 變成 unix timestamp,而資料庫對時間欄位 index。如此一來,資料庫在查詢的時候就可以先 B+tree 找到 started_at 的節點 (或是小於他最近的節點),然後連續抽出一百筆。

另外一種則是在產品面上「妥協」,也就是方法保持不變,但限制「一個 thread 最多只能回 1000 篇」,這在 2ch 或是論壇上很常看到。

Archives