Nvidia CUDA 的 EULA 禁止跑在模擬層上面

Hacker News 上看到「Nvidia bans using translation layers for CUDA software to run on other chips (tomshardware.com)」這個討論串,原報導「Nvidia bans using translation layers for CUDA software — previously the prohibition was only listed in the online EULA, now included in installed files」。

文章最上面更新的這段把重點講完了,在 11.6 之後的版本就有加入這段文字了:

[Edit 3/4/24 11:30am PT: Clarified article to reflect that this clause is available on the online listing of Nvidia's EULA, but has not been in the EULA text file included in the downloaded software. The warning text was added to 11.6 and newer versions of the installed CUDA documentation.]

主要是這條:

You may not reverse engineer, decompile or disassemble any portion of the output generated using SDK elements for the purpose of translating such output artifacts to target a non-NVIDIA platform.

從「CUDA Toolkit Archive」這邊可以看到 11.6.0 是 2022 年一月的事情:

CUDA Toolkit 11.6.0 (January 2022), Versioned Online Documentation

看起來跟 ZLUDA 有關 (參考「IntelAMD GPU 直接跑 CUDA 程式的 ZLUDA」)。

時間軸看起來有對起來,2021 年的時候被 Intel 找去聊,2022 年的時候改跟 AMD 簽約,然後 AMD 後來也放掉:

In 2021 I was contacted by Intel about the development of ZLUDA. I was an Intel employee at the time. While we were building a case for ZLUDA internally, I was asked for a far-reaching discretion: not to advertise the fact that Intel was evaluating ZLUDA and definitely not to make any commits to the public ZLUDA repo. After some deliberation, Intel decided that there is no business case for running CUDA applications on Intel GPUs.

Shortly thereafter I got in contact with AMD and in early 2022 I have left Intel and signed a ZLUDA development contract with AMD. Once again I was asked for a far-reaching discretion: not to advertise the fact that AMD is evaluating ZLUDA and definitely not to make any commits to the public ZLUDA repo. After two years of development and some deliberation, AMD decided that there is no business case for running CUDA applications on AMD GPUs.

IntelAMD GPU 直接跑 CUDA 程式的 ZLUDA

先前提過「在 Intel 內顯上面直接跑 CUDA 程式的 ZLUDA」,結果後來事情大翻轉,AMD 跑去贊助專案,變成支援 AMD GPU 了:「AMD Quietly Funded A Drop-In CUDA Implementation Built On ROCm: It's Now Open-Source」,專案在 GitHubvosen/ZLUDA 這邊,而這包支援 AMD GPU 的 commit log 則是在 1b9ba2b2333746c5e2b05a2bf24fa6ec3828dcdf 這包巨大的 commit:

Nobody expects the Red Team

Too many changes to list, but broadly:
* Remove Intel GPU support from the compiler
* Add AMD GPU support to the compiler
* Remove Intel GPU host code
* Add AMD GPU host code
* More device instructions. From 40 to 68
* More host functions. From 48 to 184
* Add proof of concept implementation of OptiX framework
* Add minimal support of cuDNN, cuBLAS, cuSPARSE, cuFFT, NCCL, NVML
* Improve ZLUDA launcher for Windows

其中的轉折以及後續的故事其實還蠻不知道怎麼說的... 作者一開始在 Intel 上班,弄一弄 Intel 覺得這沒前景,然後 AMD 接觸後贊助這個專案,到後面也覺得沒前景,於是依照後來跟 AMD 的合約,如果 AMD 覺得沒前景,可以 open source 出來:

Why is this project suddenly back after 3 years? What happened to Intel GPU support?

In 2021 I was contacted by Intel about the development od ZLUDA. I was an Intel employee at the time. While we were building a case for ZLUDA internally, I was asked for a far-reaching discretion: not to advertise the fact that Intel was evaluating ZLUDA and definitely not to make any commits to the public ZLUDA repo. After some deliberation, Intel decided that there is no business case for running CUDA applications on Intel GPUs.

Shortly thereafter I got in contact with AMD and in early 2022 I have left Intel and signed a ZLUDA development contract with AMD. Once again I was asked for a far-reaching discretion: not to advertise the fact that AMD is evaluating ZLUDA and definitely not to make any commits to the public ZLUDA repo. After two years of development and some deliberation, AMD decided that there is no business case for running CUDA applications on AMD GPUs.

One of the terms of my contract with AMD was that if AMD did not find it fit for further development, I could release it. Which brings us to today.

這個其實還蠻好理解的,CUDA 畢竟是 Nvidia 家的 ecosystem,除非你反超越後自己定義一堆自家專屬的功能 (像是當年 MicrosoftIE 上的玩法),不然只是幫人抬轎。

Phoronix 在 open source 前幾天先拿到軟體進行測試,而他這幾天測試的結果給了「頗不賴」的評價:

Andrzej Janik reached out and provided access to the new ZLUDA implementation for AMD ROCm to allow me to test it out and benchmark it in advance of today's planned public announcement. I've been testing it out for a few days and it's been a positive experience: CUDA-enabled software indeed running atop ROCm and without any changes. Even proprietary renderers and the like working with this "CUDA on Radeon" implementation.

另外為了避免測試時有些測試軟體會回傳到伺服器造成資訊外洩,ZLUDA 在這邊故意設定為 Graphics Device,而在這次 open source 公開後會改回正式的名稱:

In my screenshots and for the past two years of development the exposed device name for Radeon GPUs via CUDA has just been "Graphics Device" rather than the actual AMD Radeon graphics adapter with ROCm. The reason for this has been due to CUDA benchmarks auto-reporting results and other software that may have automated telemetry, to avoid leaking the fact of Radeon GPU use under CUDA, it's been set to the generic "Graphics Device" string. I'm told as part of today's open-sourcing of this ZLUDA on Radeon code that the change will be in place to expose the actual Radeon graphics card string rather than the generic "Graphics Device" concealer.

作者的測試看起來在不同的測試項目下差異頗大,但如果依照作者的計算方式,整體效能跟 OpenCL 版本差不多:

Phoronix 那邊則是做了與 Nvidia 比較的測試... 這邊拿的是同樣都有支援 Nvidia 與 AMD 家的卡的 Blender 測試,然後跑出來的結果讓人傻眼,透過 ZLUDA 轉譯出來的速度比原生支援的速度還快,這 optimization 看起來又有得討論了:(這是 BMW27 的測試,在 Classroom 的測試也發現一樣的情況)

但即使如此,CUDA over AMD GPU 應該還是不會起來,官方會儘量讓各 framework 原生支援,而大多數的開發者都是在 framework 上面開發,很少會自己從頭幹...

AMD 平台上的 LLM 計算

前幾天在 Hacker News 上看到的文章:「Making AMD GPUs competitive for LLM inference (mlc.ai)」,原文在「Making AMD GPUs competitive for LLM inference」這邊。

Nvidia 在 GPU 上的各種運算這塊進來的很早,除了本家開發了很多工具以外,社群的支援度也很好。而 AMD 這邊就差了不少,但這也反應在顯卡的售價上面。

作者整理了同樣是 24GB VRAM 的顯卡出來,分別是 AMD 的 7900XTX,以及 Nvidia 的 3090 Ti 與新的 4090

可以看出來縮然同樣 fp16 對應到的功耗差蠻多的,但單價低很多,對於業餘玩家偶而用來說,其實是個可以考慮的方案。

而他們的成果可以看出來效果其實不差,跑 Llama 2 的 model 可以看到 CP 值相當高:

看起來支援的主力在 ROCm 上,就效能與功耗的筆直來說其實是超越的?(或者保守一點的說,是在同一個水平上的)

現在算是 AMD 顯卡在追趕的過程,社群的力量看起來會是主力...

在 Intel 內顯上面直接跑 CUDA 程式的 ZLUDA

Hacker News 首頁上看到的有趣東西:「Zluda: Run CUDA code on Intel GPUs, unmodified (github.com/vosen)」,專案在「CUDA on Intel GPUs」這邊,這是個最後更新在 2021 年的專案。

這個專案的想法可以猜得出來,想要吃 CUDA 的 ecosystem,把現有用 CUDA 的應用程式直接跑在 Intel 的 GPU 上面,這樣對於一些只有 CUDA 卻沒有 OpenCL 的實作就有機會拿來用。

一開始本來以為是給 Intel 新的獨立顯卡 Arc,結果發現是 2021 年就停更的專案,是以內顯來測試的:

ZLUDA performance has been measured with GeekBench 5.2.3 on Intel UHD 630.

從 benchmark 的結果看起來,大多數的功能應該都有 porting 上去,所以至少測試是能跑的,而不是 crash:

不過 Hacker News 的討論上可以看到似乎還是有問題,而且大多數的 AI 應用還是會回頭支援 OpenCL,似乎沒有那麼好用...

雲端上面的 GPU 資源費用,以及地端的 GPU 決策圖

Hacker News 上面看到「Cloud GPU Resources and Pricing (fullstackdeeplearning.com)」這篇,原網頁是「Cloud GPUs - The Full Stack」,裡面有些有用的資源可以拉出來獨立看。

雲端的選擇上,因為 H100 看起來還沒普及,所以用上一代的 A100 (80GB) 來看,可以看到大的雲端跟其他家的差異還是蠻大的:

不過這邊好像沒把 vast.ai 放進來。

地端的資訊主要是直接購買顯示卡時的選擇,可以看到如果除了各系列的旗艦卡外 (4090 & 3090 & 2080),3060 是一張會在考慮到「便宜」而上榜的卡,應該是因為他是一張入門價位的顯卡,卻有 12GB VRAM 的關係:

在接下來七月要推出的 4060 會出 16GB VRAM 版本,應該會取代現在 3060 12GB VRAM 的地位...

llama.cpp 開始支援 GPU 了

前陣子因為重灌桌機,所以在重建許多環境... 其中一個就是 llama.cpp,連到專案頁面上時意外發現這兩個新的 feature:

OpenBLAS support
cuBLAS and CLBlast support

這代表可以用 GPU 加速了,所以就照著說明試著編一個版本測試。

編好後就跑了 7B 的 model,看起來快不少,然後改跑 13B 的 model,也可以把完整 40 個 layer 都丟進 3060 (12GB 版本) 的 GPU 上:

./main -m models/13B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -n 512 -ngl 40

從 log 可以看到 40 layers 到都 GPU 上面,吃了 7.5GB 左右:

llama.cpp: loading model from models/13B/ggml-model-q4_0.bin
llama_model_load_internal: format     = ggjt v2 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =  90.75 KB
llama_model_load_internal: mem required  = 9807.48 MB (+ 1608.00 MB per state)
llama_model_load_internal: [cublas] offloading 40 layers to GPU
llama_model_load_internal: [cublas] total VRAM used: 7562 MB
llama_init_from_file: kv self size  =  400.00 MB

30B 的 model 我也試著丟上去跑,但只能丟 28 layers 上去 (全部是 60 layers),再多 GPU 的記憶體就撐不住了。

但能用 GPU 算是一個很大的進展,現在這版只快了一半的時間,不知道後面還有沒有 tune 的空間...

Hashcat 在 3090 與 4090 上的速度差異

想到的時候查了一下,發現進步頗大的...

可以參考「Hashcat v6.1.1 benchmark on the Nvidia RTX 3090」與「Hashcat v6.2.6 benchmark on the Nvidia RTX 4090」這兩份資料。

已經被打到爛的 $1$ 或是 $apr1$ 就不看了,主要還是看比較有抵抗性的 KDF 的速度。

撞老牌的 bcrypt,在 4090 的速度快要是 3090 的兩倍了:

Hashmode: 3200 - bcrypt $2*$, Blowfish (Unix) (Iterations: 32)
Speed.#1.........:    96662 H/s (36.29ms) @ Accel:8 Loops:16 Thr:11 Vec:1
* Hash-Mode 3200 (bcrypt $2*$, Blowfish (Unix)) [Iterations: 32]
Speed.#1.........:   184.0 kH/s (50.22ms) @ Accel:4 Loops:32 Thr:24 Vec:1

但是在看 scrypt 的時候不知道為什麼 3090 跑的 iterations 是 1,這兩個不對等:

Hashmode: 8900 - scrypt (Iterations: 1)
Speed.#1.........:  2308.2 kH/s (8.64ms) @ Accel:16 Loops:1 Thr:16 Vec:1
* Hash-Mode 8900 (scrypt) [Iterations: 16384]
Speed.#1.........:     7126 H/s (29.16ms) @ Accel:128 Loops:1024 Thr:32 Vec:1

改看 9300 這組的話,iterations 還是 1:

Hashmode: 9300 - Cisco-IOS $9$ (scrypt) (Iterations: 1)
Speed.#1.........:    85331 H/s (122.39ms) @ Accel:16 Loops:1 Thr:8 Vec:1
* Hash-Mode 9300 (Cisco-IOS $9$ (scrypt)) [Iterations: 16384]
Speed.#1.........:    83890 H/s (1.76ms) @ Accel:128 Loops:1024 Thr:32 Vec:1

再來是看 PBKDF2 類的,這邊先抓 9200 這組,可以看到 4090 的速度是 3090 的兩倍多:

Hashmode: 9200 - Cisco-IOS $8$ (PBKDF2-SHA256) (Iterations: 19999)
Speed.#1.........:   187.2 kH/s (91.61ms) @ Accel:16 Loops:256 Thr:1024 Vec:1
* Hash-Mode 9200 (Cisco-IOS $8$ (PBKDF2-SHA256)) [Iterations: 19999]
Speed.#1.........:   430.1 kH/s (60.71ms) @ Accel:16 Loops:512 Thr:512 Vec:1

另外 10900 這組也是兩倍多:

Hashmode: 10900 - PBKDF2-HMAC-SHA256 (Iterations: 999)
Speed.#1.........:  3785.4 kH/s (87.58ms) @ Accel:4 Loops:999 Thr:1024 Vec:1
* Hash-Mode 10900 (PBKDF2-HMAC-SHA256) [Iterations: 999]
Speed.#1.........:  8865.7 kH/s (50.50ms) @ Accel:64 Loops:124 Thr:512 Vec:1

考慮到單位價錢的效能,與單位功率的效能,4090 都是大幅領先的情況 (台積電幫了不少?)。

美國政府禁止 NVIDIA 將高階顯卡輸出到中國與俄羅斯

Hacker News 首頁上看到「US Government Bans Export of Nvidia A100 and H100 GPUs to China and Russia (sec.gov)」這篇,是 NVIDIA 發出了 Form 8-K,說明美國政府禁止 A100 與 H100 或是更高階 (更快) 的卡以及產品輸出到中國 (包括香港) 與俄羅斯:「nvda-20220826.htm」。

先是指出 A100、H100 以及 A100X (Ampere) 被管制:

On August 26, 2022, the U.S. government, or USG, informed NVIDIA Corporation, or the Company, that the USG has imposed a new license requirement, effective immediately, for any future export to China (including Hong Kong) and Russia of the Company’s A100 and forthcoming H100 integrated circuits. DGX or any other systems which incorporate A100 or H100 integrated circuits and the A100X are also covered by the new license requirement.

另外是禁止新產品的部份,效能與 A100 相等或是更好的卡也被禁止輸出,除非有取得授權:

The license requirement also includes any future NVIDIA integrated circuit achieving both peak performance and chip-to-chip I/O performance equal to or greater than thresholds that are roughly equivalent to the A100, as well as any system that includes those circuits.

然後有提到軍事相關考量:

A license is required to export technology to support or develop covered products. The USG indicated that the new license requirement will address the risk that the covered products may be used in, or diverted to, a ‘military end use’ or ‘military end user’ in China and Russia. The Company does not sell products to customers in Russia.

有看到一些報導指出 AMD 也有收到類似的禁令 (畢竟也是個顯卡大廠),但在「SEC Filings」這邊沒看到...

Netflix 的 Open Connect 機器往 800Gbps 推進

2021 年的時候曾經提過 Netflix 試著用單機推出 400Gbps 的流量 (用在 Netflix 的 Open Connect):「Netflix 在單機服務 400Gbps 的影音流量」,快一年後的目前,Netflix 的人已經成功推到接近 800Gbps 了:「Serving Netflix Video Traffic at 800Gb/s and Beyond」。另外在 Hacker News 上的討論「Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] (nabstreamingsummit.com)」也可以看看。

翻了一下投影片,最後衝到 720Gbps,主要是因為 NIC output drop,而非其他部份。

裡面還是把之前的故事也都講了一遍 (不然簡報的時間會太短?),如果有看過前面的內容可以快速看一下就好,這次新的東西從 page 89 開始:

  • Asynchronous Sendfile (2014)
  • Kernel TLS (2016)
  • Network-centric NUMA (2019)
  • Inline Hardware (NIC) kTLS (2022)
  • 800G initial results

最後面幾張投影片裡面有提到往 800Gbps 衝的硬體平台:

  • AMD (EPYC 7713 CPUs)
  • Dell (PowerEdge R7525)
  • Mellanox/NVIDIA (ConnectX-6 Dx NICS)
  • Intel (P5316 NVME)

下個目標不知道是什麼,看起來目前已經壓榨到 memory bandwidth 也有點極限的感覺了...

跑在本機的 GitHub Copilot 替代品

Hacker News 上看到「FauxPilot – an attempt to build a locally hosted version of GitHub Copilot (github.com/moyix)」這個本機上跑 GitHub Copilot 協定的專案。專案的 GitHub 在「FauxPilot - an open-source GitHub Copilot server」這邊。

裡面用的是 Salesforce 放出來的 CodeGen,不過 Salesforce 提供了 350M、2B、6B 與 16B 的 model,但在 FauxPilot 這邊目前只看到 350M、6B 與 16B 的 model 可以用,少了 2B 這組,然後需要的 VRAM 就有點尷尬了:

[1] codegen-350M-mono (2GB total VRAM required; Python-only)
[2] codegen-350M-multi (2GB total VRAM required; multi-language)
[3] codegen-6B-mono (13GB total VRAM required; Python-only)
[4] codegen-6B-multi (13GB total VRAM required; multi-language)
[5] codegen-16B-mono (32GB total VRAM required; Python-only)
[6] codegen-16B-multi (32GB total VRAM required; multi-language)

13GB 剛好超過 3080 Ti 的 12GB,所以不是 3090 或 3090 Ti 的使用者就只能跑 350M 這個版本?看 Hacker News 上的討論似乎是有打算要弄 2B 的版本啦...

然後我自己雖然是 11GB 的 1080 Ti,想跑個 350M 的版本測試看看,但看起來相關的 Nvidia driver 沒裝好造成他識別不到,加上我是用 neovim,看了一下目前 ~/.config/github-copilot/hosts.json 的內容,程式碼應該是寫死到 GitHub API 上使用:

{"github.com":{"user":"gslin","oauth_token":"x"}}

先暫時放著好了,晚點等 2B 版本出現後再回來看看有沒有比較完整的指示...