Google SRE 團隊整理出過去二十年的十一條心得

Google 的 SRE 團隊整理出過去二十年的心得,當看故事的心態在看的:「Lessons Learned from Twenty Years of Site Reliability Engineering」,在 Hacker News 上也有討論:「Lessons Learned from Twenty Years of Site Reliability Engineering (sre.google)」。

裡面的項目大多都會在公司成長時不斷的導入,都是夠大就會遇到的。

比較有趣的是第六條,這是唯一一條全部都用大寫字母列出來的:

COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!!

到 Google 這個規模的架構,這邊就會規劃找完全獨立於 Google 架構的方案來用;我猜應該是傳統的 colocation 機房 (像是 AT&T 之類的),上面跑 IRC server 之類的?

在 Hacker News 上面也有其他人提到 Netflix 也有類似的規劃,需要有一個備援的管道是完全獨立於 AWS 的;另外同一則 comment 裡也有提到 Reddit 的作法是在辦公室裡面放 IRC server 備援:

Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one we used was unavailable.

IRC 還是很好用的 XD

Ansible 的爭論

前幾天在 Hacker News Daily 上看到「Five Ansible Techniques I Wish I’d Known Earlier」這篇,裡面提到了一些 Ansible 的用法還蠻有用的,算是開始用 Ansible 後應該都會有幫助的用法... 不過 Hacker News 上的討論「Ansible Techniques I Wish I’d Known Earlier (zwischenzugs.com)」比較精彩...

目前在頂端的留言對 Ansible 幹到不行,尤其是那個 YAML 格式:

Ansible is abysmal. I don't know why anyone still chooses it. It's a mess of yaml and what feels like a million yaml files that is always extremely hard to follow. Honestly writing some python, or bash is a lot easier to follow, read, and understand. The only thing it has going for it is the inventory system. I wish ansible would die already.

然後講到 bash 與 python 之類的工具時有人提到 idempotent:

>bash and python
Neither of those of idempotent.

馬上就有人幹勦,大多數人在寫 Ansible playbook 時根本沒人在注意 idempotent,而且一堆 shell script 的東西被塞進 YAML 只能說痛苦 XDDD

Most of the ansible roles I come across written by my team are not idempotent either, its a huge lie that Ansible is idempotent. Its idempotent if you put the effort into make it be but if I see tons of shell or command module invocations without prerequisite checks to see if the work should be done. Most devs I see using Ansible treat it like a shell script written in YAML and to that purpose it sucks.

我自己目前會挑 Ansible 主要還是因為 server 不需要另外裝軟體,是個 production 為導向的設計,再更大的時候就要想一下要怎麼繼續搞下去了...