在本機用 pip 直接安裝 PostgreSQL server

看到 PostgreSQL 官方站台上的介紹,可以直接用 Pythonpip 指令安裝 PostgreSQL server:「Install a local, non-root PostgreSQL Server with Python "pip"」,專案在「postgresql-wheel」這邊。

GitHub 上面的說明跑了一下,還真的可以惡搞... 這樣如果真的要在 CI 裡面跑的話也簡單很多了?只要能 pip 裝軟體就能跟你拼 XDDD

也省掉需要設定一些權限跑 Docker-in-Docker...

Travis CI 停止提供服務給 Open Source 專案

Hacker News Daily 上看到這個,在講 Travis CI 終止對 open source 專案的服務:「Travis CI is no longer providing CI minutes for open source projects」。

有不少專案已經改用其他 CI 服務的 free tier 做,像是 GitHub ActionsCircleCI 都有提供。

看了一下 Hacker News 上的討論,不少人是可以理解不會有永遠免費的午餐,只是這次 Travis CI 的處理讓大家覺得比較討厭的是,一方面官方一直跟 open source community 說我們很願意支援 open source community (前幾個禮拜官方 blog 還有發文重申這件事情),另外一方面是一直在朝著關閉服務走:從限制 10K credits 開始,到現在不會再提供 credit 給 open source 專案。

有人整理了對應的時間軸:

? 2018: Travis CI announces they are starting the process of merging travis-ci.org, which provided free builds for OSS projects, into travis-ci.com, which until then was only for paying customers. They promise OSS builds will continue to be free.

? 2020: Travis CI announces they are shutting down travis-ci.org at the end of the year and all projects have to move to travis-ci.com. They promise OSS builds will continue to be free.

Early November 2020: travis-ci.com switches from providing unlimited builds for OSS to only providing 10k one-time credits by default. Projects that meet certain guidelines (e.g. no one paid to work on them) can apply for recurring credits.

Later in November 2020: CI for many OSS projects that had migrated to travis-ci.com starts to fail, as they've exhausted their 10K credits.

Dec 2020: If what is reported here is accurate, Travis CI stop providing any recurring OSS credits. CI breaks for the remaining OSS projects on travis-ci.com.

Jan 2021: travis-ci.org shuts down. CI will be broken for all projects using it. They'll have the option of migrating to travis-ci.com, but will soon break again as they exhaust their 10k credits.

這種服務非常吃 CPU resource,大型專案如果每個 push 或是每個 pull request 都跑一輪完整的測試,成本應該不低,大概可以理解為什麼 Travis CI 會這樣決定,不過態度就...

其他家有提供 free tier 的 CI 最近應該會湧入不少人,這幾個月可以看看其他家會不會也跟進,另外一方面應該也會有人開始自架?

CodeDeploy 的 codedeploy-agent 總算支援 Ubuntu 18.04 的環境了...

先前在『在 AWSUG Taiwan 上講的「用 AWS CodeDeploy 解決程式佈署」』這邊有分享過 CodeDeployUbuntu 18.04codedeploy-agent 會認為系統內建的 rubycodedeploy-agent 不支援的版本:「Support for ruby 2.5 (Ubuntu 18.04) · Issue #158 · aws/aws-codedeploy-agent」。

剛剛總算看到修正後的版本被推出去了:

這樣就不需要自己修正惡搞了...

Amazon DynamoDB 提供 Docker Image 讓開發者可以在本地端測試

AWS 推出了 Amazon DynamoDB 的相容 Docker Image,讓開發者可以在本地端測試 DynamoDB 的 API:「Use Amazon DynamoDB Local More Easily with the New Docker Image」,在 amazon/dynamodb-local 這邊可以拉到,裡面其實是包 Java:

DynamoDB local is now available to download as a self-contained Docker image or a .jar file that can run on Microsoft Windows, Linux, macOS, and other platforms that support Java.

這樣在 Continuous Integration (CI) 的過程裡面也可以拉起 service 測試...

GitHub 上的 Continuous Integration (CI)

GitHub 寫了一篇「GitHub welcomes all CI tools」,不過對我來說比較有趣的是各家 CI 在 GitHub 上的市占率:

可以看到 Travis CI 還是擁有壓倒性的佔有率 (大約一半),這跟他提供免費的 public repository 方案有關。但以另外一個角度來看,這也讓使用商用版本 (travis-ci.com) 的人有大量的範例可以參考,而不需要擔心資源太少。

Heroku CI 正式推出

月初才提到「Heroku 也要進入 CI 市場...」,前幾天就開放使用了:「Heroku CI Is Now Generally Available: Fast, Low Setup CI That’s Easy to Use」。

收費方式是依照 Heroku Pipelines 收費,一個 Heroku Pipelines 是 USD$10/month:

Each CI-enabled Heroku Pipeline is charged just $10/month for an unlimited number of test runs.

比起其他棚是無限量 Travis CI 的 $69/month (1 concurrent job,不限時間),或是 CircleCI 的 Free 方案 (1 concurrent job,限時每個月 1500 分鐘) 或是 USD$50/month (2 concurrent job,不限時間),感覺有些微妙...

感覺還是先繼續用 Travis CI 好了,目前 Open Source Project 的標準,路邊可以隨便檢到很多範例用... (像是用 Google".travis.yml" "language:php" site:github.com 就可以找到很多 PHP 的範例)

AWS CodeDeploy 支援 BlueGreenDeployment

AWS CodeDeploy 推出了 BlueGreenDeployment 的功能:「AWS CodeDeploy Introduces Blue/Green Deployments」。

BlueGreenDeployment 的目的不計成本想辦法把上線的 downtime 壓到最低,而且當出問題時 rollback 的時間壓到最低的方法:

One of the challenges with automating deployment is the cut-over itself, taking software from the final stage of testing to live production. You usually need to do this quickly in order to minimize downtime.

Blue-green deployment also gives you a rapid way to rollback - if anything goes wrong you switch the router back to your blue environment.

其實就是直接跑兩個環境 (所以成本比較高),一套跑舊的一套跑新的,然後在前面的 load balancer 切換:

The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible.

Facebook 備份 MySQL 資料並且確認正確性的方法

Facebook 再多花了一些篇幅數對於 MySQL 資料備份以及確認正確性的方法:「Continuous MySQL backup validation: Restoring backups」。

首先是 Continuous Restore Tier (CRT) 這塊,可以看到他們在這塊很仰賴 HDFS 當作備份的第一層基地,包括了 Full logical backups (用 mysqldump)、Differential (diff) backups 以及 Binary log (binlog) backups (stream 進 HDFS)。

另外上了 GTID,對於後續的處理會比較方便:

All of our database servers also use global transaction IDs (GTIDs), which gives us another layer of control when replaying transactions from binlog backups.

在 CRT 這塊可以看到其實是拿現成的工具堆起來,不同單位會因為規模而有不同的作法。真正的重點反而在 ORC Restore Coordinator (ORC) 這塊,可以看到 Facebook 開發了大量的程式將回復這件事情自動化處理:

在收到回復的需求後,可以看到 Peon 會從 HDFS 拉資料出來,並且用 binlog replay 回去:

Peons contain all relevant logic for retrieving backups from HDFS, loading them into their local MySQL instance, and rolling them forward to a certain point in time by replaying binlogs. Each restore job a peon works on goes through these five stages[.]

也是因為 Facebook 對 MySQL 的用量大到需要自動化這些事情,才有這些東西...