Recently we had a report from a user who had seen a stunning 90% performance regression after upgrading his server to a Linux kernel with KPTI (kernel page-table isolation – a remedy for the Meltdown vulnerability).
他們發現 90% 是因為 VMware 舊版本無法使用 CPU feature 加速，在新版應該可以改善不少。但即使如此，文章內還是在實體機器上看到了 40% 的效能損失：
A big deal of those 90% was caused by running in an old version of VMware which doesn’t pass the PCID and INVPCID capabilities of the CPU to the guest. But I could reproduce a regression around 40% even on bare metal.
To understand the KPTI overhead, there are at least five factors at play. In summary:
Syscall rate: there are overheads relative to the syscall rate, although high rates are needed for this to be noticable. At 50k syscalls/sec per CPU the overhead may be 2%, and climbs as the syscall rate increases. At my employer (Netflix), high rates are unusual in cloud, with some exceptions (databases).
Context switches: these add overheads similar to the syscall rate, and I think the context switch rate can simply be added to the syscall rate for the following estimations.
Page fault rate: adds a little more overhead as well, for high rates.
Working set size (hot data): more than 10 Mbytes will cost additional overhead due to TLB flushing. This can turn a 1% overhead (syscall cycles alone) into a 7% overhead. This overhead can be reduced by A) pcid, available in Linux 4.14, and B) Huge pages.
Cache access pattern: the overheads are exacerbated by certain access patterns that switch from caching well to caching a little less well. Worst case, this can add an additional 10% overhead, taking (say) the 7% overhead to 17%.
重點在於給了量測的方式，以第一個 Syscall rate 來說好了，他用 sudo perf stat -e raw_syscalls:sys_enter -a -I 1000 測試而得到程式的 syscall 數量，然後得到下面的表格，其中 X 軸是每秒千次呼叫數，Y 軸是效能損失：
We wanted to provide a bit more context for the most recent login issues and service instability. All of our cloud services are affected by updates required to mitigate the Meltdown vulnerability. We heavily rely on cloud services to run our back-end and we may experience further service issues due to ongoing updates.
原因是 Linux 與 OS X 上有 direct-physical map 的機制，會把整塊 physical memory 對應到 virtual memory 的固定位置上，這些位置不會再發給 user space 使用，所以是通的：
On Linux and OS X, this is done via a direct-physical map, i.e., the entire physical memory is directly mapped to a pre-defined virtual address (cf. Figure 2).
而在 Windows 上則是比較複雜，但大部分的 physical memory 都有對應到 kernel address space，而每個 process 裡面也都還是有完整的 kernel address space (只是受到權限控制)，所以 Meltdown 的攻擊仍然有效：
Instead of a direct-physical map, Windows maintains a multiple so-called paged pools, non-paged pools, and the system cache. These pools are virtual memory regions in the kernel address space mapping physical pages to virtual addresses which are either required to remain in the memory (non-paged pool) or can be removed from the memory because a copy is already stored on the disk (paged pool). The system cache further contains mappings of all file-backed pages. Combined, these memory pools will typically map a large fraction of the physical memory into the kernel address space of every process.
