Google 的 SRE 團隊整理出過去二十年的心得,當看故事的心態在看的:「Lessons Learned from Twenty Years of Site Reliability Engineering」,在 Hacker News 上也有討論:「Lessons Learned from Twenty Years of Site Reliability Engineering (sre.google)」。
裡面的項目大多都會在公司成長時不斷的導入,都是夠大就會遇到的。
比較有趣的是第六條,這是唯一一條全部都用大寫字母列出來的:
COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!!
到 Google 這個規模的架構,這邊就會規劃找完全獨立於 Google 架構的方案來用;我猜應該是傳統的 colocation 機房 (像是 AT&T 之類的),上面跑 IRC server 之類的?
在 Hacker News 上面也有其他人提到 Netflix 也有類似的規劃,需要有一個備援的管道是完全獨立於 AWS 的;另外同一則 comment 裡也有提到 Reddit 的作法是在辦公室裡面放 IRC server 備援:
Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one we used was unavailable.
IRC 還是很好用的 XD