llama.cpp 有全 GPU 版本了

Hacker News 首頁上看到「Llama.cpp: Full CUDA GPU Acceleration (github.com/ggerganov)」,對應得原頁面在「CUDA full GPU acceleration, KV cache in VRAM #1827」這邊。

裡面是在講 llama.cpp 之前的 GPU 加速還是有不少事情是在 CPU 上面做,這次是把目前 ggml 支援的操作都實作 GPU 版本了:

This PR adds GPU acceleration for all remaining ggml tensors that didn't yet have it. Especially for long generations this makes a large difference because the KV cache is still CPU only on master and gets larger as the context fills up.

蠻多人有不同測試的結果,要注意這次不是把 CPU 搬到 GPU 上面做,而是把本來因為比較 light 而還沒搬上 GPU 的部分搬上去,所以不會是數量級的加速,但看起來改善也已經很不賴了:

Early attempt this morning we're getting ~2.5-2.8x perf increase on 4090s and about 1.8-2x on 3090Ti.

然後 Falcon... 目前看起來還沒有必較好的進展 XD

Georgi Gerganov 成立公司 GGML

Hacker News 首頁上看到 Georgi Gerganov 成立公司的計畫:「GGML – AI at the Edge (ggml.ai)」,官網在「GGML - AI at the edge」。

如同 Georgi Gerganov 提到的,llama.cpp 這些專案本來是他的 side project,結果意外的紅起來:

另外他提到了 Nat FriedmanDaniel Gross 也幫了一把:

在官網則是有提到是 pre-seed funding:

ggml.ai is a company founded by Georgi Gerganov to support the development of ggml. Nat Friedman and Daniel Gross provided the pre-seed funding.

現在回頭來看,當初 llama.cpp 會紅起來主要是因為 CPU 可以跑 LLaMA 7B,而且用 CPU 跑起來其實也不算慢。

後來吸引了很多人一起幫忙,於是有了不少 optimization (像是「llama.cpp 的載入速度加速」這邊用 mmap 減少需要載入的時間,並且讓多個 process 之間可以重複使用 cache),接下來又有 GPU 的支援...


llama.cpp 開始支援 GPU 了

前陣子因為重灌桌機,所以在重建許多環境... 其中一個就是 llama.cpp,連到專案頁面上時意外發現這兩個新的 feature:

OpenBLAS support
cuBLAS and CLBlast support

這代表可以用 GPU 加速了,所以就照著說明試著編一個版本測試。

編好後就跑了 7B 的 model,看起來快不少,然後改跑 13B 的 model,也可以把完整 40 個 layer 都丟進 3060 (12GB 版本) 的 GPU 上:

./main -m models/13B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -n 512 -ngl 40

從 log 可以看到 40 layers 到都 GPU 上面,吃了 7.5GB 左右:

llama.cpp: loading model from models/13B/ggml-model-q4_0.bin
llama_model_load_internal: format     = ggjt v2 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =  90.75 KB
llama_model_load_internal: mem required  = 9807.48 MB (+ 1608.00 MB per state)
llama_model_load_internal: [cublas] offloading 40 layers to GPU
llama_model_load_internal: [cublas] total VRAM used: 7562 MB
llama_init_from_file: kv self size  =  400.00 MB

30B 的 model 我也試著丟上去跑,但只能丟 28 layers 上去 (全部是 60 layers),再多 GPU 的記憶體就撐不住了。

但能用 GPU 算是一個很大的進展,現在這版只快了一半的時間,不知道後面還有沒有 tune 的空間...

llama.cpp 的載入速度加速

Hacker News 上看到「Llama.cpp 30B runs with only 6GB of RAM now (github.com/ggerganov)」這個消息,原 pull request 在「Make loading weights 10-100x faster #613」這邊。

這個 PR 的作者 Justine Tunney 在 PR 上有提到他改變 model 檔案格式,以便改用 mmap(),大幅降低了需要預先讀取的時間 (因為變成 lazy-loading style),而且這也讓系統可以利用 cache page,避免了 double buffering 的問題:

This was accomplished by changing the file format so we can mmap() weights directly into memory without having to read() or copy them thereby ensuring the kernel can make its file cache pages directly accessible to our inference processes; and secondly, that the file cache pages are much less likely to get evicted (which would force loads to hit disk) because they're no longer competing with memory pages that were needlessly created by gigabytes of standard i/o.

這讓我想到在資料庫領域中,PostgreSQL 也會用 mmap() 操作,有點類似的概念。

另外 Justine Tunney 在這邊的 comment 有提到一個意外觀察到的現象,他發現實際在計算的時候用到的 model 內容意外的少:他用一個簡單的 prompt 測試,發現 20GB 的 30B model 檔案在他的 Intel 機器上實際只用到了 1.6GB 左右:

If I run 30B on my Intel machine:


As we can see, 400k page faults happen, which means only 1.6 gigabytes ((411522 * 4096) / (1024 * 1024)) of the 20 gigabyte weights file actually needed to be used.

這點他還在懷疑是不是他的修改有 bug,但目前他覺得不太像,也看不太出來:

Now, since my change is so new, it's possible my theory is wrong and this is just a bug. I don't actually understand the inner workings of LLaMA 30B well enough to know why it's sparse. Maybe we made some kind of rare mistake where llama.cpp is somehow evaluating 30B as though it were the 7B model. Anything's possible, however I don't think it's likely. I was pretty careful in writing this change, to compare the deterministic output of the LLaMA model, before and after the Git commit occurred. I haven't however actually found the time to reconcile the output of LLaMA C++ with something like PyTorch. It'd be great if someone could help with that, and possibly help us know why, from more a data science (rather than systems engineering perspective) why 30B is sparse.

如果不是 bug 的話,這其實冒出了一個很有趣的訊號,表示這些 model 是有可能再瘦身的?

Stanford Alpaca 與 Alpaca.cpp

先前提到的「玩最近 Facebook Research (Meta) 放出來的 LLaMA」,在 2/24 公佈後,過沒多久 Stanford CRFM 團隊就丟出 fine-tuned 過的版本:「Alpaca: A Strong, Replicable Instruction-Following Model」,對應的討論在「Alpaca: A strong open-source instruction-following model (stanford.edu)」。

這個 tuning 是基於 LLaMA-7B:

We are releasing our findings about an instruction-following language model, dubbed Alpaca, which is fine-tuned from Meta’s LLaMA 7B model. We train the Alpaca model on 52K instruction-following demonstrations generated in the style of self-instruct using text-davinci-003. On the self-instruct evaluation set, Alpaca shows many behaviors similar to OpenAI’s text-davinci-003, but is also surprisingly small and easy/cheap to reproduce.

後來就再被包成一個專案 (改自 llama.cpp),讓大家可以很方便的跑:「Alpaca.cpp」。

剛好可以比較,用 llama.cpp 跑的話是:

$ ./main -m ./models/7B/ggml-model-q4_0.bin -t 12 -n 128 -p 'Explain FAANG. '

Explain FAANG. 🙂
The acronym “FAAN” is used for the five tech giants (Facebook, Apple, Amazon and Netflix) that are
 taking over our lives:
15 November 2018 / by Amaraksha S [end of text]

而用 alpaca.cpp 跑的話回答品質好不少:

> Explain FAANG.
FAANG (or FANNG) refers to five technology companies, Facebook Inc., Apple Inc., Amazon.com Inc., Netflix Inc and Alphabet's Google division that have become some of the most influential in recent years due to their rapid growth rates as well as their market capitalization values: 
Facebook (FB), $596 billion; Apple (AAPL) ,$1,084.72 billion; Amazon (AMZN), $394.5 billion ; Netflix( NFLX ), $154.8 billion and Google's parent company Alphabet Inc., ($GOOG,$GOOGL).
These companies have become some of the most influential in recent years due to their rapid growth rates as well as their market capitalization values:  Facebook (FB),$307.15 billion; Apple(AAPL) , $924.86 billion ; Amazon (AMZN ), $783.

可以看到 instruction-following 後的差異,很像是 GPT-3.5 API 與 ChatGPT API 的輸出結果差異。

而這只是 LLaMA-7B 的成果,好像可以期待有人對 LLaMA-65B 做些事情?