Home » Posts tagged "space"

Amazon RDS 支援更大的硬碟空間與更多的 IOPS

Amazon RDS 的升級:「Amazon RDS Now Supports Database Storage Size up to 16TB and Faster Scaling for MySQL, MariaDB, Oracle, and PostgreSQL Engines」。

空間上限從 6TB 變成 16TB,而且可以無痛升。另外 IOPS 上限從 30K 變成 40K:

Starting today, you can create Amazon RDS database instances for MySQL, MariaDB, Oracle, and PostgreSQL database engines with up to 16TB of storage. Existing database instances can also be scaled up to 16TB storage without any downtime.

The new storage limit is an increase from 6TB and is supported for Provisioned IOPS and General Purpose SSD storage types. You can also provision up to 40,000 IOPS for Provisioned IOPS storage volumes, an increase from 30,000 IOPS.

不過隔壁的 Amazon Aurora 還是大很多啊 (64TB),而且實際上不用管劃多大,他會自己長大:

Q: What are the minimum and maximum storage limits of an Amazon Aurora database?

The minimum storage is 10GB. Based on your database usage, your Amazon Aurora storage will automatically grow, up to 64 TB, in 10GB increments with no impact to database performance. There is no need to provision storage in advance.

Linode 要推出「Linode Block Storage」了...

從「Linode Block Storage (Fremont beta)」這邊可以看到 Linode 推出 Block Storage 了,是 SSD-based,跟 Amazon EBS 的 gp2 也是 SSD-based 相同。

計價方式,價錢也相同,沒有 I/O fee:

They're affordable - $0.10 per GB (free during the beta) and no usage fees.

目前能從 1GB 開到 1TB:

How big of a Volume can I create?
Between 1 GB and 1024 GB for now. After the beta, the max volume size may be larger.

單台可以掛 8 個:

How many Volumes can I attach to a Linode at the same time?
Up to 8.

然後 2018 開始收費:

The beta is free through 2017. January 1, 2018 the meter starts running.

有了 Block Storage 後有些事情就比較好搭出來了,也不會受限於 local disk 的空間大小。

Facebook 把 InnoDB 換成 MyRocks 的計畫

在「Migrating a database from InnoDB to MyRocks」這邊講到了 FacebookMySQL 的 InnoDB 換成 MyRocks 的計畫。

Facebook 已經大量導入全 Flash 的環境,於是現在 InnoDB (Compressed) 的情況類似於這樣:

可以看到空間是最大的問題 (Space-bound),而非 CPU 或是 I/O,這在純 Flash 的機器上還蠻有可能的,因為純 Flash 能提供的 I/O capacity 超高,加上 Facebook 把 MySQL 當作後端儲存設備而已,CPU 的使用量可以預期不高...

然後他們估算 MyRocks 可以省下一半的空間,於是有可能變成:

不過對於一般單位來說,這些前提就未必成立了... 但還是可以看到 Mark Callaghan 花不少力氣在調校一般性的效能,希望讓使用者多一點...

Quotient filter

之前有提過「Cuckoo Filter:比 Bloom Filter 多了 Delete」,最近在「A general purpose counting filter: making every bit count」這邊看到 Quotient filter,也是類似 Bloom filter 的資料結構,但想要解決更多問題。

一般的 Bloom filter (BF) 會有這些問題:

  • The inability to delete items
  • Poor scaling out of RAM
  • The inability to resize dynamically
  • The inability to count the number of occurrences of each item, especially with skewed input distributions.

而文章裡提到的 Quotient filter (QF) 就是要解這些問題。另外還提到了 Rank-and-Select-based Quotient filter (RSQF) 以及 Counting Quotient filter (CQF)。雖然多了一些空間需求,但看起來解掉不少問題... (尤其是刪除的能力)

效能上也還不錯,尤其是讀取速度的部份... 不過不知道相對於 Cuckoo filter 差多少。

Amazon Redshift 壓縮率的改善

Amazon Redshift 對壓縮率的改善:「Data Compression Improvements in Amazon Redshift Bring Compression Ratios Up to 4x」。

首先是引入了 Zstandard

First, we added support for the Zstandard compression algorithm, which offers a good balance between a high compression ratio and speed in build 1.0.1172. When applied to raw data in the standard TPC-DS, 3 TB benchmark, Zstandard achieves 65% reduction in disk space. Zstandard is broadly applicable.


Second, we’ve improved the automation of compression on tables created by the CREATE TABLE AS, CREATE TABLE or ALTER TABLE ADD COLUMN commands. Starting with Build 1.0.1161, Amazon Redshift automatically chooses a default compression for the columns created by those commands. Automated compression happens when we estimate that we can reduce disk space without degrading query performance. Our customers have seen up to 40% reduction in disk space.


Third, we’ve been optimizing our internal on-disk data structures. Our preview customers averaged a 7% reduction in disk space usage with this improvement. This feature is delivered starting with Build 1.0.1271.


Finally, we have enhanced the ANALYZE COMPRESSION command to estimate disk space reduction.

不過其他幾個產品線的使用方式更成熟 (像是 Amazon Athena 這類產品),不知道會不會讓 Amazon Redshift 慢慢退出第一線...


這幾年對於開放式空間有不少反面意見出來,像是這幾天 BBC 登的「Why open offices are bad for us」。


Numerous companies have embraced the open office — about 70% of US offices are open concept — and by most accounts, very few have moved back into traditional spaces with offices and doors.

但人的效率會因為開放式空間大約掉 15%:

But research that we’re 15% less productive, we have immense trouble concentrating and we’re twice as likely to get sick in open working spaces, has contributed to a growing backlash against open offices.

採用開放式空間最常見的理由包括辦公室成本 (每個人平均分到的空間大小會比較低),另外一個是藉由開放式空間讓互相討論合作的成本降低,但因為開放式空間,反而是影響到別人的情況比討論合作的情況多,甚至是與工作無關的事情也會影響到期他人:

Beside the cheaper cost, one main argument for the open workspace is that it increases collaboration. However, it’s well documented that we rarely brainstorm brilliant ideas when we’re just shooting the breeze in a crowd. Instead, as many of us know, we’re more likely to hear about the Christmas gift a colleague is buying for a family member, or problems with your deskmate’s spouse.

其實科技的進步讓遠端溝通的成本降低了不少,像是 SlackZoom,現在未必要靠 open office 的架構讓大家溝通了。