用 pipsi 管 Python 的 command line 工具...

在「My Python Development Environment, 2018 Edition « Jacob Kaplan-Moss」這邊看到 Python 開發有哪些工具可以用 (介紹了三個),其中管理不同 Python 版本的 pyenv 用一陣子了,另外兩個則是之前沒用過...

pipsi 是將套件用 virtualenv 包起來,讓使用者在用的時候不會受到其他環境的干擾。我是拿來跟系統的 python3 (目前在 Ubuntu 16.04 上是指到 3.5.1) 使用,所以安裝 pipsi 時先切到 system 再透過 python3 安裝 (讓他偵測到系統的 python3):

$ pyenv shell system
$ which python3
$ curl https://raw.githubusercontent.com/mitsuhiko/pipsi/master/get-pipsi.py | python3

接著把 PATH 參數設好後 (設到 .bashrc 或是 .zshrc 之類的檔案),重新開一個 terminal 或是 shell (讓路徑生效),再把 awscli 裝進去:

$ pipsi install awscli

這樣這些工具就會吃系統的 python3 了...

Rust 版本 Mercurial

看到「Mercurial: 964212780daf」這則 commit log,看起來 MercurialPython 換成 Rust 的計畫正在進行中:

rust: implementation of `hg`

This commit provides a mostly-working implementation of the `hg` script in Rust along with scaffolding to support Rust in the repository.

之前從朋友那邊聽到,在 F 公司用 Mercurial 用到覺得很厭世,主要是因為 repository 太大,一跑下去就會發現記憶體用量與速度都很無奈 (即使內部已經有不少工具改善速度),所以就啟動專案要換一個程式語言,直接拼最後那段的效能 XD

如果是隔壁棚的 Git... 就沒這個問題,一開始 Git 就用 C 寫,所以如果厭世的話也不太容易生出什麼進展了 XDDD

微軟在考慮讓 Excel 支援 Python...

在「Excel team considering Python as scripting language: asking for feedback」這邊看到微軟正在考慮要不要讓 Excel 支援 Python,出自 UserVoice 上的:「How can we improve Excel for Windows (Desktop Application)?」。

比較感覺到有可能性應該是因為微軟做了一個問卷收集資訊:「Python and Excel」。

不過本來的功能就已經可以用到很出神入化了... XD (想到最近提到的「LINE 將內部的座位表由 Excel 改成 Web 界面...」)

GitHub 上有大量重複的程式碼...

扣除掉 fork 的程式碼後,研究人員在 GitHub 上還是發現有大量重複的程式碼:「DéjàVu: a map of code duplicates on GitHub」。

This paper analyzes a corpus of 4.5 million non-fork projects hosted on GitHub representing over 482 million files written in Java, C++, Python, and JavaScript. We found that this corpus has a mere 85 million unique files.

Java/C++/Python/JavaScript 寫的 4.5M 個專案有 482M 個檔案,但只有 85M 個檔案是不一樣的 XD

想一想其實也是... 現在愈來愈多工具產生程式碼了 XD (i.e. Scaffold)

在 MOPCON 2017 的 Unconference「MySQL to NoSQL & Search Engine」

把投影片傳到 Speaker Deck 上了:「MySQL to NoSQL & Search Engine」。

這是在介紹 noplay/python-mysql-replication 這個軟體,我在示範時用的 python script 有增加 blocking 參數讓他保持一直讀取 MySQL replication stream:

from pymysqlreplication import BinLogStreamReader

mysql_settings = {'host': '', 'port': 3306, 'user': 'root', 'passwd': ''}

stream = BinLogStreamReader(connection_settings = mysql_settings, server_id=100, blocking=True)

for binlogevent in stream:


利用這樣的工具可以做很多事情,像是當 post 表格更新時自動更新 search engine,並且清空 memcached 內的資料。這可以避免使用 library 時有可能會漏掉忘記做 (因為有些程式不用 library 處理),可靠度比較高。

另外一方面 replication protocol 本身就有考慮重連的問題,重新接上時是可以從上一次處理完的資料繼續處理 (只要不要隔太久),這讓寫應用的人不需要用太複雜的方式確保他不會漏掉。

PyPy 5.9 支援 Pandas 與 NumPy 了

PyPy 5.9 支援 machine learning 常用的 PandasNumPy 了:「PyPy v5.9 Released, Now Supports Pandas, NumPy」,包括 2.7 與 3.5 的相容版本:

The PyPy team is proud to release both PyPy3.5 v5.9 (a beta-quality interpreter for Python 3.5 syntax) and PyPy2.7 v5.9 (an interpreter supporting Python 2.7 syntax).

對於使用 Python 大量計算的人來說可以進場測試了 XD

Python 在高收入國家的成長

Stack Overflow 的內文其實有點奇怪的誤導... 主要是分析在 Stack Overflow 上 Python 成長的趨勢:「The Incredible Growth of Python」。




另外列出 YoY 成長:


Reddit 的 Deploy 機制 (的歷史)

Reddit 主要是用 Python 寫的,這邊介紹了他們歷年來的 Code Deploy 系統:「The Evolution of Code Deploys at Reddit」。

最早期的時候 (2007 到 2010) 是用 rsync 更新程式碼,然後跑個迴圈用 ssh 連進去重跑:

# build the static files and put them on the static server
`make -C /home/reddit/reddit static`
`rsync /home/reddit/reddit/static public:/var/www/`

# iterate through the app servers and update their copy
# of the code, restarting once done.
foreach $h (@hostlist) {
    `git push $h:/home/reddit/reddit master`
    `ssh $h make -C /home/reddit/reddit`
    `ssh $h /bin/restart-reddit.sh`

2011 時因為人變多了,用 IRC 把過程丟出來 (okay,我知道你想問的問題... Slack 是 2013 年推出的):

The process for actually doing the deploy looked the same, but now the system did the work for you and told everyone what you were doing.

另外值得一提的是,因為他們不是自己架 IRC server 而是用外面第三方的伺服器,所以他們決定 IRC 只有單向告知的功能:

There was a lot of talk of systems that managed deploys from chat around this time, but since we used third party IRC servers we weren’t able to fully trust the chat room with production control and so it remained a one-way flow of information.

2012 時則是把機器列表放到 DNS 上,某種 service discovery 系統:

First, it fetched its list of hosts from DNS rather than keeping it hard-coded. This allowed us to update the list of hosts without having to remember to update the deploy tool as well — a rudimentary service discovery system.

另外是固定的版本,而非拉 master 下來,這樣可以避免 race condition 的不一致性 (推到一半有人把 code 塞進 master):

Another small but important change was to always deploy a fixed version of the code. The previous version of the tool would update master on a given host, but what if master changed mid-deploy because someone accidentally pushed up code? By deploying a specific git revision instead of branch name, we ensured that the deploy got the same version everywhere in production.

2013 往雲上搬,於是遇到像是「開新機器時剛好在 deploy 會拉到舊的 code」這種 edge case。

What happens if a server is launched while a deploy is ongoing? We had to make sure each newly launched server checked in to get new code if present. What about servers going away mid-deploy? The tool had to be made smarter to detect when the server was gone legitimately rather than there being an issue with the deploy process itself that should be noisily alerted on.

2014 遇到機器數量太多,推一輪要一個小時而被迫要平行化處理:

Over time, the number of servers needed to serve peak traffic grew. This meant that deploys took longer and longer. At its worst, a normal deploy took close to an hour. This was not good.

2015 則是加上 deploy lock,避免同時間有兩個人在 deploy:

Engineers would ask for the deploy lock and either get it or get put in the queue. This helped keep order in deploys and let people relax a bit while waiting for the lock.

2017 的部份則是提到了伺服器的數量:

This new mechanism allows us to deploy to a lot more machines concurrently, and deploy timings are down to 7 minutes for around 800 servers despite the extra waiting for safety.

看起來到現在還是維持手動 deploy,而不是自動化... 這塊還蠻有趣的 :o

AWS Lambda 支援 Python 3.6 了

大家都等很久的需求總算推出來了 (雖然我覺得還好...),AWS Lambda 要支援 Python 3.6 了:「AWS Lambda Supports Python 3.6」。會逐步推上線:

You can now develop your AWS Lambda functions using Python 3.6. This feature will become available in all AWS Lambda regions within 24 hours.

django 的 LTS

看到「Django 1.11 released」這篇才發現 django 也有設計 LTS 機制... 在「Django’s release process」這邊有介紹:

Certain feature releases will be designated as long-term support (LTS) releases. These releases will get security and data loss fixes applied for a guaranteed period of time, typically three years.