Amazon EFS 的 file lock 限制

看到「Amazon EFS now supports a larger number of concurrent file locks」這篇提到:

This Amazon EFS update increases the number of simultaneous file locks an NFS mount can acquire to 65,536 (from 8,192 previously), enabling Amazon EFS to be used for a broader set of applications that heavily leverage file locking (including message broker and distributed analytics applications).

所以 NFS 的部份先前有 8192 個的上限在,現在則是拉 65536 個了。

不過提到 message broker,應該是以前的應用利用 NFS file lock 來做一些事情?

Amazon EFS 效能提昇的一些討論

上一篇「Amazon EFS 的效能提昇」提到 Amazon EFS 的效能提昇,在 Hacker News 上看到 Amazon EFS 團隊的 PMT (Product-Manager-Technical) 出來回一些東西:「Amazon Elastic File System Update – Sub-Millisecond Read Latency (amazon.com)」,搜尋 geertj 應該就可以看到他回的東西了...

像是即使是 Jeff Barr 發表這篇文章,也還是經過 legal team 的同意才能發表:

(PMT on the EFS team).

Yes, the wordings are carefully formulated as they have to be signed off by the AWS legal team for obvious reasons. With that said, this update was driven by profiling real applications and addressing the most common operations, so the benefits are real. For example, a simple WordPress "hello world" is now about 2x as fast as before.

另外這次的效能提昇是透過 cache 層達成的:

I'm the PMT for this project in the EFS team. The "flip the switch" part was indeed one of the harder parts to get right. Happy to share some limited details. The performance improvement builds on a distributed consistent cache. You can enable such a cache in multiple steps. First you deploy the software across the entire stack that supports the caching protocol but it's disabled by configuration. Then you turn it for the multiple components that are involved in the right order. Another thing that was hard to get right was to ensure that there are no performance regressions due to the consistency protocol.

然後在每個 AZ 都有 cache:

The caches are local to each AZ so you get the low latency in each AZ, the other details are different. Unfortunately I can't share additional details at this moment, but we are looking to do a technical update on EFS at some point soon, maybe at a similar venue!

另外看起來主要就是 metadata cache 的幫助:

NFS workloads are typically metadata heavy and highly correlated in time, so you can achieve very high hit rates. I can't share any specific numbers unfortunately.

還是有很多細節數字不能透漏,但知道是透過 cache 達成的就已經可以大致上想像後面是怎麼弄出來的了...

Amazon EFS 的效能提昇

AWS 宣佈他們將 Amazon EFS 的 latency 大幅降低以提昇效能:「Amazon Elastic File System Update – Sub-Millisecond Read Latency」。

Linux 上一般是用 NFS 掛 EFS,個位數的 ms 的確對於效能影響超大,現在宣稱讀取的部份降到 0.6ms,應該會有蠻明顯的感覺:

Up until today, EFS latency for read operations (both data and metadata) was typically in the low single-digit milliseconds. Effective today, new and existing EFS file systems now provide average latency as low as 600 microseconds for the majority of read operations on data and metadata.

然後不另外收費:

This performance boost applies to One Zone and Standard General Purpose EFS file systems. New or old, you will still get the same availability, durability, scalability, and strong read-after-write consistency that you have come to expect from EFS, at no additional cost and with no configuration changes.

另外就是過去幾個禮拜他們把現有的 EFS 都轉移過去了:

We “flipped the switch” and enabled this performance boost for all existing EFS General Purpose mode file systems over the course of the last few weeks, so you may already have noticed the improvement. Of course, any new file systems that you create will also benefit.

不過 EFS 另外一個問題就是貴炸,用錢換方便...

OpenSSH 的 scp 改用 SFTP 協定

在「By default, scp(1) now uses SFTP protocol」這邊看到的,OpenSSH 的 scp 改用 SFTP 協定了,原因也有附在文章裡:

SFTP offers more predictable filename handling and does not require expansion of glob(3) patterns via the shell on the remote side.

要注意這是 BC-break change,有些之前會動的 case 在改用 SFTP 後會爛掉,但這算是前進了一大步,scp 因為 spec 的關係很難維護安全性。

在「Deprecating scp」這邊也有提到相關的問題,另外也給出了一些範例。

rsync 的預設值是傳整個檔案,不是 delta

剛好最近工作上需要透過 4G 網路傳大檔案,但希望大檔案傳到一半斷掉後可以續傳,而不要浪費頻寬整個重傳,所以查了資料並且測了一些東西...

其中一個比較特別的是發現 rsync 的預設是傳整個檔案 (當檔案有變化時),而不是傳 delta (有變更的部份),不過還好,可以透過指令強制使用 delta。

在「Does rsync --inplace write to the entire file, or just to the parts that need to be updated? (for btrfs+rsync backups)」這邊有提到幾個需要設定的指令。

首先是標題就有提到的 --inplace,在 manpage 裡面有提到是直接更新檔案,而非建立一個新檔案再 rename,這樣做的缺點是其他的應用程式可能會讀到改到一半的檔案:

update destination files in-place

另外一個提到的是 --no-whole-file,這個要看 --whole-file 的說明來理解,後者就是不開 delta:

copy files whole (w/o delta-xfer algorithm)

第三個是 -c,強制使用 checksum 比對:

skip based on checksum, not mod-time & size

不過我的應用裡面不太想管這個,就沒設定 -c 了,基本上是靠 ssh 的保護,不會有收到錯誤封包的問題。

整體來說,這個方法對兩邊的機器都比較吃資源,而且會遇到應用程式在還在傳輸時讀到檔案的問題,但如果可以克服,而且目標是省頻寬的話,算是個還不錯的方法...

Neovim 在選擇檔案名稱時的操作按鍵

Neovim 時操作檔案名稱時會是下拉選單,在 insert mode 時的畫面是這樣 (進到 insert mode 後 Ctrl-X + F):

這時候可以用上下鍵選擇檔案名稱。

在 command mode 下也有類似的功能,像是 :sp 後按 tab 選擇檔案名稱:

問題在於只能用 Ctrl-N 與 Ctrl-P 移動,而不能用上下鍵操作,兩者的 UI 類似但是操作的方式不一樣,於是就翻了翻 manual,找出對應的模式,得到是 command mode,然後用 <expr> + pumvisible() 判斷是否是在 popup menu,接著把上下鍵對應到 Ctrl-N 與 Ctrl-P:

cnoremap <expr> <Down> pumvisible() ? "\<C-n>" : "\<Down>"
cnoremap <expr> <Up> pumvisible() ? "\<C-p>" : "\<Up>"

這樣就搞定了...

Apple 推出讓 iCloud 可以複製到 Google Photos 上的服務

Apple 推出了從 iCloud 的內容複製到 Google Photos 上的服務:「Apple Launches Service for Transferring iCloud Photos and Videos to Google Photos」。

我登入進去 iCloud 沒看到,後來才看到:

Apple's transfer service is available to customers in Australia, Canada, the European Union, Iceland, Liechtenstein, New Zealand, Norway, Switzerland, the United Kingdom, and the United States at this time.

服務的區域主要是歐洲北美紐西蘭,台灣暫時還沒辦法用,只能先看一下 screenshot 了:

既然是 Google Photos,能搬過去的就只有 Google Photos 有支援的類型,包括了大部分的圖片與影片格式:

Smart Albums, Live Photos, photo stream content, some metadata, and some RAW photos are not able to be transferred, but formats including .jpg, .png, .webp, .gif, some RAW files, .mpg, .mod, .mmv, .tod, .wmv, .asf, .avi, .divx, .mov, .m4v, .3gp, .3g2, .mp4, .m2t, .m2ts, .mts, and .mkv are compatible.

不知道推出這個功能後面的意義是什麼...

改變 Xfce Terminal 的 Alt-Number 快速鍵功能

前陣子桌機重裝 Ubuntu,順便把桌面環境換成 Xubuntu 用看看,也把本來再用的 GNOME Terminal 換成 XfceTerminal

我的習慣會把 GNOME Terminal 的 Alt-Number (像是 Alt-1) 快速鍵改掉,因為有不少程式會吃這組快速鍵,像是 tmux 切換視窗內玻璃 (pane) 排列的 preset 以及 IRC client 在不同頻道的切換。

但 Xfce Terminal 沒有 GUI 讓你改這組快速鍵 (其他的快速鍵有,但也雷雷的,後面會提到...),翻了翻看起來只有「Disable alt-n tab shortcut in xfce-terminal?」這邊有提到,算是堪用:

~/.config/xfce4/terminal/accels.scm looked promising but my changes were undone after a restart, so I made it read-only but it turns out commenting out the relevant lines makes no difference anyway.

雖然作者有提到它改了 ~/.config/xfce4/terminal/accels.scm 沒用,我自己發現這邊的確是很 buggy,但暫時還是可以找到 workaround。

解法是直接改沒錯,但不能直接註解掉,而需要改空,也就是本來的:

(gtk_accel_path "<Actions>/terminal-window/goto-tab-1" "<Alt>-1")

不能改成:

; (gtk_accel_path "<Actions>/terminal-window/goto-tab-1" "<Alt>-1")

而是要改空:

(gtk_accel_path "<Actions>/terminal-window/goto-tab-1" "")

另外要注意,透過 GUI 修改快速鍵後,~/.config/xfce4/terminal/accels.scm 裡面的內容也會被重製,也就是 Xfce Terminal 寫入這個檔案時是直接把預設值寫進去,而非把有效值寫進去:

這點算是比較地雷的地方...

Windows 上包裝 Syncthing 的 SyncTrayzor

在「SyncTrayzor is an open source Syncthing client for Windows」這邊看到有人將 Syncthing 包裝好,讓使用者在 Windows 上直接設定,而不需要另外開瀏覽器設定:

Syncthing is a popular peer-to-peer file sharing/synchronization software. It uses a web GUI which can be a little confusing for beginners. SyncTrayzor is an open source client that makes the P2P tool more user-friendly.

Syncthing 比較特別的觀念就是每一台都要設定允許其他台分享 (通常是這樣)。假設你有四台 Syncthing 要設定,每一台都要設定允許其他三台的分享。

不過也可以有其他的設計,像是你可以在 VPS hosting 上租一台空間很大的機器,然後其他機器都只對 VPS 這台機器同步,這樣就比較像有中央 server 的架構。

對於有多電腦的人還蠻好用的東西...

AirBnB 送出 Form S-1

Hacker News Daily 上看到 AirBnB 送出 Form S-1 了:「d81668ds1.htm」,在 TechCrunch 上面也有一系列的文章可以看...

印象中差不多試 2018 年就有一些傳言了,結果在 2019 年的時候沒看到,到了 2020 年被 COVID-19 重挫... 但長期來看是個成功的商業模式,所以 2020 年不論是有要上還是沒有要上都不意外,現在丟出 Form S-1 給了個答案。

接下來就是等開張那天的股價了...