跑 ArchiveTeam Warrior

Archive Team 是一個致力於保存數位資料的組織,而 ArchiveTeam Warrior 則是他們提供的軟體,可以讓你很方便直接跑 worker 加入他們的 cluster,幫忙抓資料並且保存到 Internet Archive 上。

他們提供三種方法跑 ArchiveTeam Warrior,第一種是 VM 的方式,文件裡面有介紹怎麼用 VirtualBox 或是 VMware Player 跑起來。

第二種與第三種都是 container 類的方式,DockerPodman 都能跑起來。

跑起來後可以連進 http://127.0.0.1:8001/,然後選擇想要加入的項目,或是接受指令選擇目前團隊主打的項目。

在「Projects」這頁可以看到目前主力是備份 Enjin 上的資料。

丟了兩台 VPS 的機器上去用 Docker 跑,CPU 使用率看起來很低,但網路流量看起來會因為所在的地點而差蠻多的,一台大約是 300KB/sec 到 400KB/sec,換算後大約是 1TB/mo,另外一台則只有 1/10 的量。

EnterpriseDB 買下 2ndQuadrant

算是 PostgreSQL 社群裡面的大事情,看到大老在討論 EnterpriseDB (EDB) 買下 2ndQuadrant 的事情:「Community Impact of 2nd Quadrant Purchase」,這兩家公司都是 PostgreSQL 社群裡面重量級的台柱。

先翻了一下新聞稿,兩邊的官方新聞稿分別是「How EDB Became the Leader in the Postgres Market」與「How EDB Became the Leader in the Postgres Market」。

回到原來的文章,裡面提到了 core team 的不成文規定,這個部份可以從 Contributor Profiles 這邊看到目前 core team 有五位成員,Peter Eisentraut 來自 2ndQuadrant,而 Bruce Momjian (這是文章作者自己) 與 Dave Page 則是來自 EnterpriseDB:

First, there is an unwritten rule that the Postgres core team should not have over half of its members from a single company, and the acquisition causes edb's representation in the core team to be 60% — the core team is working on a solution for this.

裡面有提到目前正在找辦法解決中,但不知道目前會怎麼解決,讓出位置可能是一個方法,加到七個人應該也是個方法,反正方法不算少,就等著看...

另外他提出來的兩個問題我覺得都還好,就是併購本來就會發生的事情。

這次的併購算是 PostgreSQL 社群裡面蠻熱鬧的事情,雖然是商業公司之間的併購,但社群這邊應該也會有不少變化...

GitHub 擴大免費版功能,以及付費版降價

GitHub 宣佈了提昇免費版的功能,以及付費版的降價消息:「GitHub is now free for teams」。

昨天是這樣:(從這邊撈的,然後發現好像有人寫了個機器人,每天都叫 web.archive.org 去撈一份...)

現在變成:

本來付費的個人方案 (Pro) 的功能都直接下放到免費版本了,而一般公司用的 Team 版本從 $9/m/user 降到 $4/m/user。有個富爸爸之後就可以任性...

GrabFood 用定位資料修正餐廳的資訊

Grab 的「How we harnessed the wisdom of crowds to improve restaurant location accuracy」這篇是他們的 data team 整理出來,如何使用既有的資料快速的修正餐廳資訊。裡面提到的方法不需要用到 machine learning,光是一些簡單的統計算法就可以快速修正現有的架構。

這些資訊其實是透過司機用的 driver app 蒐集來的,在 driver app 上有大量的資訊傳回伺服器 (像是定時回報的 GPS 位置,以及取餐狀態),而這些司機因為地緣關係,腦袋裡的資訊比地圖會準不少:

One of the biggest advantages we have is the huge driver-partner fleet we have on the ground in cities across Southeast Asia. They know the roads and cities like the back of their hand, and they are resourceful. As a result, they are often able to find the restaurants and complete orders even if the location was registered incorrectly.

所以透過這些資訊他們就可以反過來改善地圖資料,像是透過司機按下「取餐」的按鈕的地點與待的時間,就可以估算餐聽可能的位置,然後拿這個資訊比對地圖上的資料,就很容易發現搬家但是地圖上沒更新的情況:

Fraction of the orders where the pick-up location was not “at” the restaurant: This fraction indicates the number of orders with a pick-up location not near the registered restaurant location (with near being defined both spatially and temporally as above). A higher value indicates a higher likelihood of the restaurant not being in the registered location subject to order volume

Median distance between registered and estimated locations: This factor is used to rank restaurants by a notion of “importance”. A restaurant which is just outside the fixed radius from above can be addressed after another restaurant which is a kilometer away.

另外也有不少其他的改善 (像是必須在離餐聽某個距離內才能點「取餐」,這個「距離」會因為餐聽可能在室內商場而需要的調整),整個成果就會反應在訂單的取消率大幅下降:

整體看起來是系統產生清單後讓人工後續處理 (像是打電話去店家問?),但這個方式所提供的清單準確度應該很高 (因為司機不會沒事跟自己時間過不去,跑到奇怪地方按下取餐),用這些資料跑簡單的演算法就能夠快速修正不少問題...

Adobe Security Team 直接把 Private Key 貼到網誌上面...

Security Team 出這種包...:「In spectacular fail, Adobe security team posts private PGP key on blog」。

Adobe 這次的事情要怎麼說呢,hmmm...

GitHub 的組織管理可以堆階層了...

GitHub 的組織管理可以堆階層了:「Nested teams add depth to your team structure」。

If you're a member of Engineering and someone creates a child team called Security, team members of Engineering aren't automatically direct team members of Security. Security and all other teams nested under the Engineering will inherit repository permissions and @mentions but nothing else.

包括了權限繼承的概念。

這功能等好久了,剛好最近會用到... 本來得硬幹做,現在看起來可以比較方便的管理了。

Stripe 的 Increment 雜誌

Stripe 推出了 Increment 雜誌,講團隊合作時的各種議題:「Introducing Increment」。

And so we've decided to start Increment, a software engineering magazine dedicated to providing practical and useful insight into what effective teams are doing so that the rest of us can learn from them more quickly.

雜誌網站上也有類似的描述:

A digital magazine about how teams build and operate software systems at scale.

Increment is dedicated to covering how teams build and operate software systems at scale, one issue at a time.

可以看一看 Stripe 對團隊合作的想法...

BGPmon 推出 BGP Stream 警告異常的 BGP 流量劫持

也是兩個禮拜前的新聞,在「OpenDNS BGP Stream Twitter Feed」這邊提到了 BGPmon 將會推出 BGP Stream 服務,將偵測到的 BGP 異常變化發到 Twitter 上。

其中 BGPmon 在幾個月前被 OpenDNS 併購 (2015 年 3 月),而 Cisco 則在上上個月底併購了 OpenDNS (2015 年 6 月)。而在過幾天的 DefCon 23 上將會透露更多細節。

前陣子 Hacking Team 洩漏的資料中就用到了 BGP hijack 來取回控制權:

That nugget that emerged from the 400 Gb of stolen Hacking Team data posted online where Italian law enforcement used Hacking Team’s Remote Control System monitoring software to regain control over a number IP addresses it was watching that were already infected with Hacking Team software by hijacking BGP routes in order to redirect traffic and regain control over a target’s machines.

除了示警外,另外一方面 BGP 上的簽名技術也愈來愈重要了,只是不知道最終會怎麼做...

Hacking Team 的 BGP Routing Hijack

Hacking Team 的事情告訴我們,只能是能做的,都有人會包成 Total Solution 賣。

洩漏出來的資料說明了 Hacking Team 在 2013 年幹的 BGP Routing Hijack:「How Hacking Team Helped Italian Special Operations Group with BGP Routing Hijack」。

The Wikileaks document described how the Italian ROS reached out to Hacking Team to work together on recovering the VPS server that ran on 46.166.163.175. In ROS terminology, the server was called “Anonymizer”. The emails also revealed that this server relays updates to another back end server called “Collector” from which ROS presumably recovers the targets’ data.

然後:

When we look at historical BGP data we can confirm that AS31034 (Aruba S.p.A) indeed started to announce the prefix 46.166.163.0/24 starting on Friday, 16 Aug at 2013 07:32 UTC. The Wikileaks emails outline how ROS complained to Hacking Team that the IP was reachable only via Fastweb but not yet through Telecom Italia, concluding not all RCS clients were able to connect back to the server immediately, since the prefix was not seen globally. BGP data further confirms this per the visualization below.

這些主要的 ISP 分別是:

AS12874 Fastweb
AS6939 Hurricane Electric, Inc.
AS49605 Reteivo.IT
AS4589 Easynet
AS5396 MC-link Spa

時間線:

這也證明了「鎖 IP」的方法其實還是很危險的。