Tag: pt-osc

MySQL InnoDB 的 OPTIMIZE TABLE 的 Lock

在 Backend Tw 的 https://www.facebook.com/groups/616369245163622/posts/2467225396744655/ 這邊看到：

先大概回答一下假設，DELETE 後的空間是可以被同一個表格重複使用的，所以應該是還好，不過離峰時間跑一下 OPTIMIZE TABLE 也沒什麼關係就是了。

裡面提到的「13.7.2.4 OPTIMIZE TABLE Statement」(MySQL 5.7 文件) 以及「13.7.2.4 OPTIMIZE TABLE Statement」(MySQL 5.6 文件) 都有講到目前比較新的版本都已經是 Online DDL 了：(這邊抓 5.6 的文件，有支援的版本資訊)

Prior to Mysql 5.6.17, OPTIMIZE TABLE does not use online DDL. Consequently, concurrent DML (INSERT, UPDATE, DELETE) is not permitted on a table while OPTIMIZE TABLE is running, and secondary indexes are not created as efficiently.

As of MySQL 5.6.17, OPTIMIZE TABLE uses online DDL for regular and partitioned InnoDB tables, which reduces downtime for concurrent DML operations. The table rebuild triggered by OPTIMIZE TABLE is completed in place. An exclusive table lock is only taken briefly during the prepare phase and the commit phase of the operation. During the prepare phase, metadata is updated and an intermediate table is created. During the commit phase, table metadata changes are committed.

文件上有提到會有一小段 lock 的時間，不過一般來說應該不會造成太大問題。

這邊要講的是早期的經典工具 pt-online-schema-change (pt-osc)，這是使用 TRIGGER-based 的方式在跑，他的範例就直接提供了一個不需要 Online DDL 支援的版本：

Change sakila.actor to InnoDB, effectively performing OPTIMIZE TABLE in a non-blocking fashion because it is already an InnoDB table:

pt-online-schema-change --alter "ENGINE=InnoDB" D=sakila,t=actor

這在早期的時候還蠻常被拿出來用的，如果還在維護一些舊系統的話還蠻推薦的...

Eventbrite 的 MySQL 升級計畫

在 2021 年看到 Eventbite 的 MySQL 升級計畫：「MySQL High Availability at Eventbrite」。

看起來是 2019 年年初的時候 MySQL 5.1 出問題，後續決定安排升級，在 2019 年年中把系統升級到 MySQL 5.7 (Percona Server 版本)：

Our first major hurdle was to get current with our version of MySQL. In July, 2019 we completed the MySQL 5.1 to MySQL 5.7 (v5.7.19-17-log Percona Server to be precise) upgrade across all MySQL instances.

然後看起來是直接在 EC2 上跑，不過這邊提到的空間問題就不太確定了，是真的把 EBS 的空間上限用完嗎？比較常使用的 gp2 與 gp3 上限都是 16TB，不確定是不是真的用到接近爆掉了：

Not only was support for MySQL 5.1 at End-of-Life (more than 5 years ago) but our MySQL 5.1 instances on EC2/AWS had limited storage and we were scheduled to run out of space at the end of July. Our backs were up against the wall and we had to deliver!

另外在升級到 5.7 的時候，順便把本來是 INT 的 primary key 都換成 BIGINT：

As part of the cut-over to MySQL 5.7, we also took the opportunity to bake in a number of improvements. We converted all primary key columns from INT to BIGINT to prevent hitting MAX value.

然後系統因為舊版的 Django 沒辦法配合 MySQL 5.7，得升級到 Django 1.6 (要注意 Django 1 系列的最新版是 1.11，看起來光是升級到 1.6 勉強會動就升不上去了？)：

In parallel with the MySQL 5.7 upgrade we also Upgraded Django to 1.6 due a behavioral change in MySQL 5.7 related to how transactions/commits were handled for SELECT statements. This behavior change was resulting in errors with older version of Python/Django running on MySQL 5.7

然後採用了 GitHub 家研發的 gh-ost 當作改變 schema 的工具：

In December 2019, the Eventbrite DBRE successfully implemented a table ALTER via gh-ost on one of our larger MySQL tables.

看起來主要的原因是有遇到 pt-online-schema-change 的限制 (在「GitHub 發展出來的 ALTER TABLE 方式」這邊有提到)：

Eventbrite had traditionally used pt-online-schema-change (pt-osc) to ALTER MySQL tables in production. pt-osc uses MySQL triggers to move data from the original to the “duplicate” table which is a very expensive operation and can cause replication lag. Matter of fact, it had directly resulted in several outages in H1 of 2019 due to replication lag or breakage.

另外一個引入的技術是 Orchestrator，看起來是先跟 HAProxy 搭配，不過他們打算要再換到 ProxySQL：

Next on the list was implementing improvements to MySQL high availability and automatic failover using Orchestrator. In February of 2020 we implemented a new HAProxy layer in front of all DB clusters and we released Orchestrator to production!

Orchestrator can successfully detect the primary failure and promote a new primary. The goal was to implement Orchestrator with HAProxy first and then eventually move to Orchestrator with ProxySQL.

然後最後題到了 Square 研發的 Shift，把 gh-ost 包裝起來變成有個 web UI 可以操作：

2021 還可以看到這類文章還蠻有趣的...