Wikimedia 新計畫:WikiFunctions

Wikimedia 的新計畫 WikiFunctions:「Introducing Wikifunctions: first Wikimedia project to launch in a decade creates new forms of knowledge」。

官方的說明中提到 WikiFunctions 是提供給 Wikimedia 其他的計畫用的程式碼:

Wikifunctions is a Wikimedia project for everyone to collaboratively create and maintain a library of code functions to support the Wikimedia projects and beyond, in the world's natural and programming languages.

這樣有點抽象,看「Wikifunctions:Catalogue」這邊的範例會比較具體一點,可以看到都是實作很基本的功能,像是「Function: To Uppercase」。

Hacker News 上也有討論:「Wikifunctions (wikimediafoundation.org)」,在 id=38549081 這邊有人比較清楚的說明了要解決的問題:

A lot of Wikipedia sites have scripts embedded in the wikitext which automatically generate or transform information on a page, e.g. automatically performing unit conversions to generate text like "I would walk 500 miles (804.67 km)", performing date math to automatically generate and update a person's age based on their birthdate, or querying structured data from Wikidata [1] to display in an infobox. One example of these scripts is the {{convert}} [2] template on the English Wikipedia.

Initially, these scripts were written in MediaWiki template logic [3], and were maintained individually on each wiki. This quickly proved unmaintainable, and some of those scripts were rewritten in Lua using the Scribunto extension [4], but these were still per-wiki, and there were frequently issues where different wikis would copy scripts from each other and introduce their own incompatible features.

The WikiFunctions project is an attempt to centralize development of these scripts, much like how Wikimedia Commons [5] centralizes hosting of freely licensed images and other media.

MediaWiki 本身的 template engine 還是有極限,所以很多的需求會希望使用程式語言開發,一方面是比較好維護,另外一方面是效率也比 template engine 硬幹來的好。

先前大家在 MediaWiki 上比較常用的方案是透過 Extension:ScribuntoLua script,而這個 extension 後來變得太重要 (好用),直接包進 MediaWiki 了。

拿個例子來說,常見的引用功能 Template:Citation 裡面直接是透過模組實作:

<includeonly>{{#invoke:citation/CS1|citation
|CitationClass=citation
}}</includeonly><noinclude>
{{Documentation}}
</noinclude>

而這邊的 #invoke 就是去使用「Module:Citation/CS1」這邊的 Lua 程式。

目前的架構上面,每個語言的 Wikipedia 都需要使用 Lua 維護自己的 Module:*,而 WikiFunctions 計畫則是希望能夠整合起來,算是重新打造基礎建設的工程。

目前看了一些 WikiFunctions 的實作,主要是以 PythonJavaScript 為主。

這樣看起來,技術上是蠻有機會讓 Scribunto 退役,但實際上要不要這樣推還沒查到討論,在 Wikifunctions:FAQ 這邊則是沒提到 Scribunto,不確定是不是想雙軌跑...

MediaWiki 的 EmailDiff 套件

先前 MediaWiki 所提供的「變更通知」都只有在信件裡「通知」,而沒有在信件裡列出「改變的內容」,這使得讀信的人要再點進去看... (於是就懶的點了)

而前陣子看到有人寫了 extension 來輸出 diff,解決了這個問題:「MediaWiki extension EmailDiff: notification emails improved」。

信裡變成有這樣的內容:

Version differences:
@@ -846,5 +887,3 @@
 In cattle, temperament can affect production traits such as carcass and meat 
 quality or milk yield as well as affecting the animal's overall health and 
-reproduction. Cattle temperament is defined as "the consistent behavioral and physiological 
-difference observed between individuals in response to a stressor or environmental 
+reproduction. If you succeed in tipping a cow only partway, such that only one 
+of its feet is still on the ground, you have created lean beef. Such a feat is 
+well done. Naturally, being outside, the cow is unstable. When it falls over, 
+it becomes ground beef. Cattle temperament is defined as "the consistent behavioral 
+and physiological difference observed between individuals in response to a stressor or environmental 
 challenge and is used to describe the relatively stable difference in the behavioral 
 predisposition of an animal, which can be related to psychobiological mechanisms.

這樣可讀性高非常多,來找人裝看看好了...

維基基金會的 2014 年八月月報

維基基金會釋出八月月報 (好像晚了三個月?):「Wikimedia Foundation Report, August 2014」,在「Wikimedia Highlights, August 2014」有比較精簡的版本。

維基基金會在報告裡有提供一些 PV 相關的數據,包括 comScore 的數字與自己 server log 所統計出來的數據。另外也包含了財務狀況。

其中技術相關的是取自「Wikimedia Engineering/Report/2014/August」這頁。另外因為這是八月的資料,我順便偷看了九月與十月的「Wikimedia Engineering/Report/2014/September」與「Wikimedia Engineering/Report/2014/October」。

可以看到在測試 HHVM 的計畫,而且目前看起來還不錯:「[Wikitech-l] [Engineering] Migrating test.wikipedia.org to HHVM」,拿了 test.wikipedia.org 測試,其中 speed test 的部份有大幅改善:

1) Speed test: measure the time taken to request the page 1000 times over just 10 concurrent connections:

                        HHVM    Zend    diff
Mean time (ms):         233     441     -47%
99th percentile (ms):   370     869     -57%
Request/s:              43      22.6    +90%

而負載測試的成果更好:

2) Load test: measure how much thoughput we obtain when hogging the appserver with 50 concurrent requests for a grand total of 10000 requests. What I wanted to test in this case was the performance degradation and the systems resource consumption

                        HHVM    Zend    diff
Mean time (ms):         355     906     -61%
99th percentile (ms):   791     1453    -45%
Request/s:              141     55.1    +156%
Network (Mbytes/s)      17      7       +142%
RAM used (GBs):         5(1)    11(4)
CPU usage (%):          90(75)  100(90)

維基百科之所以沒有遇到太多問題,主要是因為所使用的軟體是 open source 而且夠大的關係,直接成為 HHVM 測試的一環:「Compatibility Update」。

不過目前看起來應該還是跑 PHP,沒有看到整個都轉換過去的計畫。

另外一方面,搜尋引擎的更換就沒有這麼順利,雖然換到 Elasticsearch 後改善不少,不過可以看到八月的報告這樣寫:

tarted deploying Cirrus as the primary search back-end to more of the remaining wikis and we found what looks like our biggest open performance bottleneck. Next month's goal is to fix it and deploy to more wikis (probably not all). We're also working on getting more hardware.

而九月時就講到沒有銀彈,要加硬體去拼:

In September we worked to mitigate the performance bottleneck that we found in August. We found there to be no silver bullet but used the information we learned to pick and order appropriate hardware to handle the remaining wikis. We also implemented out significantly improved wikitext Regular Expression search. In October we've begun rolling out the wikitext Regular Expression search and received some of the hardware we need to finish cutting over the remaining wikis. We believe we'll get it all installed in October and cut the remaining wikis over in November.

十月的時候弄到機器了:

In October we prepared for November in which we deployed Cirrus to all the remaining wikis by installing new servers installing new versions of Elasticsearch and our plugins. We also fixed up regex search which had caused a search outage.

這些報告的連結裡面其實有些不會在對外新聞稿上面的評語... XD

Wikipedia 把英文版資料庫的其中一個 slave 從 MySQL 5.1 換到 MariaDB 5.5...

維基百科的 mailing list 上丟出的消息,英文版 Wikipedia 資料庫的 slave server 目前已經在 MariaDB 5.5 上了:「mariadb 5.5 in production for english wikipedia」。

之前跑的版本是 MySQL 5.1 + Facebook patchset 版本,整體大約快了 8%:

Taking the times of 100% of all queries over regular sample windows, the average query time across all enwiki slave queries is about 8% faster with MariaDB vs. our production build of 5.1-fb. Some queries types are 10-15% faster, some are 3% slower, and nothing looks aberrant beyond those bounds. Overall throughput as measured by qps has generally been improved by 2-10%. I wouldn't draw any conclusions from this data yet, more is needed to filter out noise, but it's positive.

然後計畫在接下來一兩個月觀察,沒問題就全換:

MariaDB has some nice performance improvements that our workload doesn't really hit (better query optimization and index usage during joins, much better sub query support) but there are also some things, such as full utilization of the primary key embedded on the right of every secondary index that we can take advantage of (and improve our schema around) once prod is fully upgraded, hopefully over the next 1-2 months.

效能不是最主要考量,而是政治面的原因,官方說法是支持 open source 社群:(沒有講的就是「我們對 Oracle 不怎麼信任...」)

The main goal of migrating to MariaDB is not performance driven. More so, I think it's in WMF's and the open source communities interest to coalesce around the MariaDB Foundation as the best route to ensuring a truly open and well supported future for mysql derived database technology. Performance gains along the way are icing on the cake.

另外參考:「on wikipedia and mariadb」。