Home » Posts tagged "code" (Page 3)

Reddit 的 Deploy 機制 (的歷史)

Reddit 主要是用 Python 寫的,這邊介紹了他們歷年來的 Code Deploy 系統:「The Evolution of Code Deploys at Reddit」。

最早期的時候 (2007 到 2010) 是用 rsync 更新程式碼,然後跑個迴圈用 ssh 連進去重跑:

# build the static files and put them on the static server
`make -C /home/reddit/reddit static`
`rsync /home/reddit/reddit/static public:/var/www/`

# iterate through the app servers and update their copy
# of the code, restarting once done.
foreach $h (@hostlist) {
    `git push $h:/home/reddit/reddit master`
    `ssh $h make -C /home/reddit/reddit`
    `ssh $h /bin/restart-reddit.sh`
}

2011 時因為人變多了,用 IRC 把過程丟出來 (okay,我知道你想問的問題... Slack 是 2013 年推出的):

The process for actually doing the deploy looked the same, but now the system did the work for you and told everyone what you were doing.

另外值得一提的是,因為他們不是自己架 IRC server 而是用外面第三方的伺服器,所以他們決定 IRC 只有單向告知的功能:

There was a lot of talk of systems that managed deploys from chat around this time, but since we used third party IRC servers we weren’t able to fully trust the chat room with production control and so it remained a one-way flow of information.

2012 時則是把機器列表放到 DNS 上,某種 service discovery 系統:

First, it fetched its list of hosts from DNS rather than keeping it hard-coded. This allowed us to update the list of hosts without having to remember to update the deploy tool as well — a rudimentary service discovery system.

另外是固定的版本,而非拉 master 下來,這樣可以避免 race condition 的不一致性 (推到一半有人把 code 塞進 master):

Another small but important change was to always deploy a fixed version of the code. The previous version of the tool would update master on a given host, but what if master changed mid-deploy because someone accidentally pushed up code? By deploying a specific git revision instead of branch name, we ensured that the deploy got the same version everywhere in production.

2013 往雲上搬,於是遇到像是「開新機器時剛好在 deploy 會拉到舊的 code」這種 edge case。

What happens if a server is launched while a deploy is ongoing? We had to make sure each newly launched server checked in to get new code if present. What about servers going away mid-deploy? The tool had to be made smarter to detect when the server was gone legitimately rather than there being an issue with the deploy process itself that should be noisily alerted on.

2014 遇到機器數量太多,推一輪要一個小時而被迫要平行化處理:

Over time, the number of servers needed to serve peak traffic grew. This meant that deploys took longer and longer. At its worst, a normal deploy took close to an hour. This was not good.

2015 則是加上 deploy lock,避免同時間有兩個人在 deploy:

Engineers would ask for the deploy lock and either get it or get put in the queue. This helped keep order in deploys and let people relax a bit while waiting for the lock.

2017 的部份則是提到了伺服器的數量:

This new mechanism allows us to deploy to a lot more machines concurrently, and deploy timings are down to 7 minutes for around 800 servers despite the extra waiting for safety.

看起來到現在還是維持手動 deploy,而不是自動化... 這塊還蠻有趣的 :o

如何讓工程師一個禮拜工作 60~80 小時

從「How do you make programmers work 60-80 hours per week?」這邊看到的,出自 Quora 的「How do you make programmers work 60-80 hours per week?」。

看到標題的時候在想「這什麼詐騙集團類型問題 XDDD」,寫 code 的工程師一天可以專注三個小時就很了不起了好嗎,然後年紀愈大就愈難專注。每天可以工作十小時鐵定是一堆時間在看 YouTubeFacebookTwitter 的好嗎 XDDD

仔細看了文章以後,發現其實作者就是在講這個:

No programmers really work 60-80 hours a week, especially in a 5 day span. That is a 12-16 hour day, 5 days a week.

I promise you that any company that has programmers “working” that many hours is really only getting 2-4 hours of real work out of them each day. The rest of the time will be filled with pointless meetings, a fair amount of web browsing, and then a whole lot of looking busy.

真的有寫程式都知道的啦...

iOS 透過無線網路的 RCE...

在「About the security content of iOS 10.3.1」這邊的說明:

Available for: iPhone 5 and later, iPad 4th generation and later, iPod touch 6th generation and later
Impact: An attacker within range may be able to execute arbitrary code on the Wi-Fi chip
Description: A stack buffer overflow was addressed through improved input validation.
CVE-2017-6975: Gal Beniamini of Google Project Zero

這描述看起來就不太妙...

透過手機螢幕上的餘熱猜測 PIN 碼

利用手機螢幕上的餘熱分析可能的 PIN 碼:「Heat traces left by fingers can reveal your smartphone PIN」,在輸入完 PIN 碼的 30 秒內的準確度都還是很高 (80%):

The report further revealed that if the thermal image is collected within 15 seconds of a PIN being entered, the technique is accurate almost 90% of the time. At 30 seconds, this accuracy decreased slightly to 80%. At 45 seconds or more, the accuracy dropped to 35% and below.

CodeDeploy 的權限設定...

這陣子在弄 AWS CodeDeploy,調整了半天才把權限壓低到合理的範圍 (AWS 給的預設值還是有點大),記錄下來之後會比較好找...

在權限部份,AWS CodeDeploy 應該有三個權限要設定:

  • IAM user:給 CI 跑完後丟上 Amazon S3 並且呼叫 AWS CodeDeploy 佈署用的,像是 Travis CI 或是之類的服務。但如果是手動到 web console 觸發的話,這個部份就不需要了。
  • Service Role:給 AWS CodeDeploy 的服務本身用的。
  • EC2 Role:給跑在機器上 AWS CodeDeploy Agent 用的。

如同前面講的,IAM user 的部份有兩個要處理,一個是讓 CI 服務把檔案傳上 Amazon S3,另外一個是讓他有權限可以呼叫 AWS CodeDeploy 佈署新版本。

後者比較簡單,直接拉 AWSCodeDeployDeployerAccess 就可以了,前者比較麻煩一點,需要透過 Policy Generator 建出對應的權限:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1486156178000",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::codedeploy-testbucket/*"
            ]
        }
    ]
}

第二個的 Service Role 是最簡單的,直接拉 AWSCodeDeployRole 就可以了。

第三個的 EC2 Role 只需要給存取 Amazon S3 的權限。照 AWS 的建議用 AmazonEC2RoleforAWSCodeDeploy 是會動啦,但權限給太大了,整個帳號可以存取的 S3 bucket 他都可以拉到:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:ListObjects"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

改成自己用 Policy Generator 生,限制在 codedeploy-testbucket 上會比較好:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1486165995000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::codedeploy-testbucket/*"
            ]
        }
    ]
}

這樣權限的部份就差不多了...

AWS CodeDeploy 支援 BlueGreenDeployment

AWS CodeDeploy 推出了 BlueGreenDeployment 的功能:「AWS CodeDeploy Introduces Blue/Green Deployments」。

BlueGreenDeployment 的目的不計成本想辦法把上線的 downtime 壓到最低,而且當出問題時 rollback 的時間壓到最低的方法:

One of the challenges with automating deployment is the cut-over itself, taking software from the final stage of testing to live production. You usually need to do this quickly in order to minimize downtime.

Blue-green deployment also gives you a rapid way to rollback - if anything goes wrong you switch the router back to your blue environment.

其實就是直接跑兩個環境 (所以成本比較高),一套跑舊的一套跑新的,然後在前面的 load balancer 切換:

The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible.

Unix 程式碼演進的記錄

GitHub 上的「dspinellis/unix-history-repo」專案放進了 Unix 程式碼從 1970 年演進到 2016 年的記錄:

The history and evolution of the Unix operating system is made available as a revision management repository, covering the period from its inception in 1970 as a 2.5 thousand line kernel and 26 commands, to 2016 as a widely-used 27 million line system.

主要的目的是讓研究人員可以直接分析,減少重複的工作:

The project aims to put in the repository as much metadata as possible, allowing the automated analysis of Unix history.

後面的分支主要是以 FreeBSD 為主:(在列表的部份也可以看到)

It has been created by synthesizing with custom software 24 snapshots of systems developed at Bell Labs, the University of California at Berkeley, and the 386BSD team, two legacy repositories, and the modern repository of the open source FreeBSD system.

整個 repository 頗壯觀的 XD

把 CSC (卡片背面的三碼) 變成 OTP (動態密碼)

把信用卡背面的後三碼 (Card security code) 變成動態密碼,雖然一般只會有三碼,但對於網路消費應該會有不少幫助,不過這樣就不能完全不拿出卡片了...:「This high-tech card is being rolled out by French banks to eliminate fraud」。

產品叫做 MotionCode,會先從法國開始:

Today both Société Générale and Groupe BPCE, two of France’s largest banking groups, are preparing to roll out these cards across all their customers after completing a pilot scheme last year.

然後是波蘭、墨西哥以及英國在規劃:

There are other pilots underway in Poland and Mexico, and Davis is running Oberthur’s UK operation with the hope of getting a pilot or trial started with a UK bank soon.

Archives