Tag: zero

用更少訓練時間的 KataGo

最近開始在不同的地方會看到 KataGo 這個名字 (Twitter 與 YouTube 上都有看到)，翻了一下資料發現是在訓練成本上有重大突破，依照論文的宣稱快了五十倍...

在第一次跑的時候，只用了 35 張 V100 跑七天就有 Leela Zero 第 130 代的強度：

The first serious run of KataGo ran for 7 days in Februrary 2019 on up to 35xV100 GPUs. This is the run featured in the paper. It achieved close to LZ130 strength before it was halted, or up to just barely superhuman.

而第二次跑的時候用了 28 張 V100 跑 20 blocks 的訓練，跑了 19 天就已經超越 Facebook 當初提供的 ELFv2 版本，而對應到 Leela Zero 大約是第 200 代左右的強度 (要注意的是在 Leela Zero 這邊已經是用 40 blocks 的結構訓練了一陣子了)：

Following some further improvements and much-improved hyperparameters, KataGo performed a second serious run in May-June a max of 28xV100 GPUs, surpassing the February run after just three and a half days. The run was halted after 19 days, with the final 20-block networks reaching a final strength slightly stronger than LZ-ELFv2! (This is Facebook's very strong 20-block ELF network, running on Leela Zero's search architecture). Comparing to the yet larger Leela Zero 40-block networks, KataGo's network falls somewhere around LZ200 at visit parity, despite only itself being 20 blocks.

從論文裡面可以看到，跟 Leela Zero 一樣是逐步提昇 (應該也是用 Net2Net)，而不是一開始就拉到 20x256：

In KataGo’s main 19-day run, (b, c) began at (6, 96) and switched to (10, 128), (15, 192), and (20, 256), at roughly 0.75 days, 1.75 days, and 7.5 days, respectively. The final size approximately matches that of AlphaZero and ELF.

訓練速度上會有這麼大的改善，分成兩個類型，一種是一般性的 (在「Major General Improvements」這章)，另外一類是特定於圍棋領域的改進 (在「Major Domain-Specific Improvements」這章)。

在 Leela Zero 的 issue tracking 裡面也可以看到很多關於 KataGo 的消息，看起來作者也在裡面一起討論，應該會有一些結果出來...

日本圍棋界使用 AWS 分析棋局的情況

看到「圍棋ＡＩ與ＡＷＳ」這篇譯文，原文是「囲碁AIブームに乗って、若手棋士の間で「AWS」が大流行 その理由とは？」。

沒有太意外是使用 Leela Zero + Lizzle，畢竟這是 open source project，在軟體與資料的取得上相當方便，而且在好的硬體上已經可以超越人類頂尖棋手。

由於在 Lizzle 的介面上可以看到勝率，以及 Leela Zero 考慮的下一手 (通常會有多個選點)，而且當游標移到這些選點上以後，還會有可能的變化圖可以看，所以對於棋手在熟悉操作介面後，可以很快的擺個變化圖，然後讓 Leela Zero 分析後續的發展，而棋手就可以快速判斷出「喔喔原來是這樣啊」。

網路上也有類似的自戰解說，可以看到棋手對 Lizzle 的操作與分析 (大約從 50:50 開始才是 Lizzle 的操作)：

不過話說回來，幹壞事果然是進步最大的原動力... 讓一群對 AWS 沒什麼經驗的圍棋棋手用起 AWS，而且還透過 AMI 與 spot instance 省錢... XD

Spectre 與 Meltdown 兩套 CPU 的安全漏洞

The Register 發表了「Kernel-memory-leaking Intel processor design flaw forces Linux, Windows redesign」這篇文章，算是頗完整的說明了這次的安全漏洞 (以 IT 新聞媒體標準來看)，引用了蠻多資料並且試著說明問題。

而這也使得整個事情迅速發展與擴散超出本來的預期，使得 Google 的 Project Zero 提前公開發表了 Spectre 與 Meltdown 這兩套 CPU 安全漏洞。文章非常的長，描述的也比 The Register 那篇還完整：「Reading privileged memory with a side-channel」。

在 Google Project Zero 的文章裡面，把這些漏洞分成三類，剛好依據 CVE 編號分開描述：

  • Variant 1: bounds check bypass (CVE-2017-5753)
  • Variant 2: branch target injection (CVE-2017-5715)
  • Variant 3: rogue data cache load (CVE-2017-5754)

前兩個被稱作 Spectre，由 Google Project Zero、Cyberus Technology 以及 Graz University of Technology 三個團隊獨立發現並且回報原廠。後面這個稱作 Meltdown，由 Google Project Zero 與另外一個團隊獨立發現並且回報原廠。

這兩套 CPU 的安全漏洞都有「官網」，網址不一樣但內容一樣：spectreattack.com 與 meltdownattack.com。

影響範圍包括 Intel、AMD 以及 ARM，其中 AMD 因為架構不一樣，只有在特定的情況下會中獎 (在使用者自己打開 eBPF JIT 後才會中)：

(提到 Variant 1 的情況) If the kernel's BPF JIT is enabled (non-default configuration), it also works on the AMD PRO CPU.

這次的洞主要試著透過 side channel 資訊讀取記憶體內容 (會有一些條件限制)，而痛點在於修正 Meltdown 的方式會有極大的 CPU 效能損失，在 Linux 上對 Meltdown 的修正的資訊可以參考「KAISER: hiding the kernel from user space」這篇，裡面提到：

KAISER will affect performance for anything that does system calls or interrupts: everything. Just the new instructions (CR3 manipulation) add a few hundred cycles to a syscall or interrupt. Most workloads that we have run show single-digit regressions. 5% is a good round number for what is typical. The worst we have seen is a roughly 30% regression on a loopback networking test that did a ton of syscalls and context switches.

KAISER 後來改名為 KPTI，查資料的時候可以注意一下。

不過上面提到的是實體機器，在 VM 裡面可以預期會有更多 syscall 與 context switch，於是 Phoronix 測試後發現在 VM 裡效能的損失比實體機器大很多 (還是跟應用有關，主要看應用會產生多少 syscall 與 context switch)：「VM Performance Showing Mixed Impact With Linux 4.15 KPTI Patches」。

With these VM results so far it's still a far cry from the "30%" performance hit that's been hyped up by some of the Windows publications, etc. It's still highly dependent upon the particular workload and system how much performance may be potentially lost when enabling page table isolation within the kernel.

這對各家 cloud service 不是什麼好消息，如果效能損失這麼大，不太可能直接硬上 KPTI patch... 尤其是 VPS，對於平常就會 oversubscription 的前提下，KPTI 不像是可行的方案。

可以看到各 VPS 都已經發 PR 公告了 (先發個 PR 稿說我們有在注意，但都還沒有提出解法)：「CPU Vulnerabilities: Meltdown & Spectre (Linode)」、「A Message About Intel Security Findings (DigitalOcean)」、「Intel CPU Vulnerability Alert (Vultr)」。

現在可以預期會有更多人投入研究，要怎麼樣用比較少的 performance penalty 來抵抗這兩套漏洞，現在也只能先等了...

AlphaGo Zero 演算法的 Open Source 實做

Leela 的作者 Gian-Carlo Pascutto 依照 paper 的描述寫完了，放在 GitHub 上的 gcp/leela-zero：

不過他在 Twitter 上也提到了，open source 實做不是真正的困難，真正的困難在於訓練完的資料，那個部份需要大量的成本才有辦法作到：

另外他推估 AlphaGo Zero 的計算量是 1700 年 (以 1080 Ti 來計算)：「[Computer-go] Zero performance」。

另外 Leela 0.11.0 也推出了，還是先維持 policy + value 的方式，但引入了不少新演算法加強。另外一個蠻特別的地方是 Windows 版改用 clang 而變快不少：

Windows version is now compiled with Clang/LLVM 5.0 instead of MSVC2017. This makes the Monte Carlo evaluations about 15% faster.

雖然 DeepMind 說要收手，但還是留下不少方向讓大家走...

U2F Security Key 產品測試？

Adam Langley 的「Testing Security Keys」這篇測試了不少有支援 U2F Security Key 的產品，這邊作者是以 Linux 環境測試。

tl;dr：在 Linux 環境下，除了 Yubico 的產品沒問題外，其他的都有問題... (只是差在問題多與少而已)

Yubico 的沒找到問題：

Easy one first: I can find no flaws in Yubico's U2F Security Key.

VASCO SecureClick 的則是 vendor ID 與 product ID 會跑掉：

If you're using Linux and you configure udev to grant access to the vendor ID & product ID of the token as it appears normally, nothing will work because the vendor ID and product ID are different when it's active. The Chrome extension will get very confused about this.

Feitian ePass 的 ASN.1 DER 編碼是錯的：

ASN.1 DER is designed to be a “distinguished” encoding, i.e. there should be a unique serialisation for a given value and all other representations are invalid. As such, numbers are supposed to be encoded minimally, with no leading zeros (unless necessary to make a number positive). Feitian doesn't get that right with this security key: numbers that start with 9 leading zero bits have an invalid zero byte at the beginning. Presumably, numbers starting with 17 zero bits have two invalid zero bytes at the beginning and so on, but I wasn't able to press the button enough times to get such an example. Thus something like one in 256 signatures produced by this security key are invalid.

Thetis 根本沒照 spec 走，然後也有相同的 ASN.1 DER 編碼問題：

With this device, I can't test things like key handle mutability and whether the appID is being checked because of some odd behaviour. The response to the first Check is invalid, according to the spec: it returns status 0x9000 (“NO_ERROR”), when it should be 0x6985 or 0x6a80. After that, it starts rejecting all key handles (even valid ones) with 0x6a80 until it's unplugged and reinserted.

This device has the same non-minimal signature encoding issue as the Feitian ePass. Also, if you click too fast, this security key gets upset and rejects a few requests with status 0x6ffe.

U2F Zero 直接 crash 沒辦法測 XDDD：

A 1KiB ping message crashes this device (i.e. it stops responding to USB messages and needs to be unplugged and reinserted). Testing a corrupted key handle also crashes it and thus I wasn't able to run many tests.

KEY-ID (網站連 HTTPS 都沒上...) / HyperFIDO 也有編碼問題但更嚴重：

The Key-ID (and HyperFIDO devices, which have the same firmware, I think) have the same non-minimal encoding issue as the Feitian ePass, but also have a second ASN.1 flaw. In ASN.1 DER, if the most-significant bit of a number is set, that number is negative. If it's not supposed to be negative, then a zero pad byte is needed. I think what happened here is that, when testing the most-significant bit, the security key checks whether the first byte is > 0x80, but it should be checking whether it's >= 0x80. The upshot is the sometimes it produces signatures that contain negative numbers and are thus invalid.

所以還是乖乖用 GitHub 帳號買 Yubico 吧...

Raspberry Pi Zero 新增了對外的無線網路與藍牙...

Raspberry Pi Zero 本來是沒有網路功能的，對於 IoT 應用有點綁手綁腳，現在則出了加上無線網路與藍牙的版本，Zero W：「New product! Raspberry Pi Zero W joins the family」。

本來的 Zero 是 USD$5，有無線網路與藍牙的 Zero W 要 $10，然後也有出專用的殼：

這樣應用好像多了一些...

Cloudbleed：Cloudflare 這次的安全問題

Cloudflare 把完整的時間軸與影響範圍都列出來了：「Incident report on memory leak caused by Cloudflare parser bug」。

出自於 2/18 時 Google 的 Tavis Ormandy 直接在 Twitter 上找 Cloudflare 的人：

Google 的 Project Zero 上的資料：「cloudflare: Cloudflare Reverse Proxies are Dumping Uninitialized Memory」。

起因在於 bug 造成有時候會送出不應該送的東西，可能包含了敏感資料：

It turned out that in some unusual circumstances, which I’ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data.

不過這邊不包括 SSL 的 key，主要是因為隔離開了：

For the avoidance of doubt, Cloudflare customer SSL private keys were not leaked. Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug.

不過由於這些敏感資料甚至還被 Google 收進 search engine，算是相當的嚴重，所以不只是 Cloudflare 得修好這個問題，還得跟眾多的 search engine 合作將這些資料移除：

Because of the seriousness of such a bug, a cross-functional team from software engineering, infosec and operations formed in San Francisco and London to fully understand the underlying cause, to understand the effect of the memory leakage, and to work with Google and other search engines to remove any cached HTTP responses.

bug 影響的時間從 2016/09/22 開始：

2016-09-22 Automatic HTTP Rewrites enabled
2017-01-30 Server-Side Excludes migrated to new parser
2017-02-13 Email Obfuscation partially migrated to new parser
2017-02-18 Google reports problem to Cloudflare and leak is stopped

而以 2/13 到 2/18 的流量反推估算，大約是 0.00003% 的 request 會可能產生這樣的問題：

The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).

不過不得不說 Tavis Ormandy 真的很硬，在沒有 source code 以及 Cloudflare 幫助的情況下直接打出可重製的步驟：

I worked with cloudflare over the weekend to help clean up where I could. I've verified that the original reproduction steps I sent cloudflare no longer work.

事發後完整的時間軸：

2017-02-18 0011 Tweet from Tavis Ormandy asking for Cloudflare contact information
2017-02-18 0032 Cloudflare receives details of bug from Google
2017-02-18 0040 Cross functional team assembles in San Francisco
2017-02-18 0119 Email Obfuscation disabled worldwide
2017-02-18 0122 London team joins
2017-02-18 0424 Automatic HTTPS Rewrites disabled worldwide
2017-02-18 0722 Patch implementing kill switch for cf-html parser deployed worldwide
2017-02-20 2159 SAFE_CHAR fix deployed globally
2017-02-21 1803 Automatic HTTPS Rewrites, Server-Side Excludes and Email Obfuscation re-enabled worldwide

另外在「List of Sites possibly affected by Cloudflare's #Cloudbleed HTTPS Traffic Leak」這邊有人整理出受影響的大站台有哪些 (小站台就沒列上去了)。

電信商對 Zero Rating 與網路中立性的問題

在「AT&T users will be able to stream DirecTV Now without using their data」這邊才看到 FCC 在這個月月初針對電信商對特定服務的 zero rating 發出警告：「The FCC tells AT&T it may be violating net neutrality with its DirecTV plans」：

AT&T is far from the only US carrier to zero rate data. T-Mobile has been ostentatiously offering free data for music and movies for a year now, and Verizon also zero rates video from its Go90 app. But in zero rating DirecTV, the FCC thinks AT&T may have gone too far.

AT&T 說任何人只要付錢都可以參加這個 plan：

AT&T’s argument is that any company that participates in its Sponsored Data program has to pay AT&T for it, and that includes DirecTV.

但問題還是在 AT&T 擁有 DirecTV，所以是左手付到右手：

Except, again, AT&T owns DirecTV, so even if one division is paying another, the overall company still ends up not paying any money.

而且這筆金額其實不小：

The situation for other companies is very different — and the FCC believes that the price they’d have to pay is “significant[.]”

不過總統快換人了，很有可能會往更糟的方向前進...

Apple 打算把 iCloud 加密用的 Key 放到用戶端

在經過最近 FBI 與 Apple 的戰鬥中 (FBI–Apple encryption dispute)，Apple 正規劃把 iCloud 加密所使用的 key 放到用戶端裝置上，而非放在伺服器端：「Apple to Hand iCloud Encryption Key Management to Account Holders」：

In effect, Apple is following the lead of secure cloud services such as SpiderOak which has been offering what it calls “Zero Knowledge” cloud storage. By that, SpiderOak retains no information about whatever is stored in its cloud service, nor the means of gaining access to it.

也就是加解密都放在 client 端處理，server 端只是 storage。

這類型最大的問題是 server 端沒辦法運用資料，但 iCloud 的確可以放掉這些功能 (搜尋之類的)，純粹當 storage 使用，藉以讓使用者自己裝置保護。

而蘋果在使用者的裝置上把類似於 HSM 的系統做的頗強大... 不知道 Android 有沒有機會也跟進。(雖然我自己是用 Apple 家的東西...)

Raspberry Pi Zero

Raspberry Pi 出了新的系列，只要 USD$5 的 Raspberry Pi Zero：「Raspberry Pi Zero: the $5 computer - Raspberry Pi」。

機器更小了，不過要注意的是沒有網路介面。看起來比較像是接機器人之類的？先用 USB 裝一裝，裝好後就都不需要了...

  • A Broadcom BCM2835 application processor
    • 1GHz ARM11 core (40% faster than Raspberry Pi 1)
  • 512MB of LPDDR2 SDRAM
  • A micro-SD card slot
  • A mini-HDMI socket for 1080p60 video output
  • Micro-USB sockets for data and power
  • An unpopulated 40-pin GPIO header
    • Identical pinout to Model A+/B+/2B
  • An unpopulated composite video header
  • Our smallest ever form factor, at 65mm x 30mm x 5mm