fp32、fp16 與 bf16

nanoGPT 這個專案下面看到 bf16 這個浮點格式：(這段在新版的 `README.md` 被改寫了)

Currently fp16 is much faster than bf16. Potentially revert back to using fp16 and re-introduce the gradient scaler?

0.1 + 0.2 = 0.30000000000000004

Your language isn't broken, it's doing floating point math. Computers can only natively store integers, so they need some way of representing decimal numbers. This representation comes with some degree of inaccuracy. That's why, more often than not, .1 + .2 != .3.

It's actually pretty simple. When you have a base 10 system (like ours), it can only express fractions that use a prime factor of the base. The prime factors of 10 are 2 and 5. So 1/2, 1/4, 1/5, 1/8, and 1/10 can all be expressed cleanly because the denominators all use prime factors of 10. In contrast, 1/3, 1/6, and 1/7 are all repeating decimals because their denominators use a prime factor of 3 or 7. In binary (or base 2), the only prime factor is 2. So you can only express fractions cleanly which only contain 2 as a prime factor. In binary, 1/2, 1/4, 1/8 would all be expressed cleanly as decimals. While, 1/5 or 1/10 would be repeating decimals. So 0.1 and 0.2 (1/10 and 1/5) while clean decimals in a base 10 system, are repeating decimals in the base 2 system the computer is operating in. When you do math on these repeating decimals, you end up with leftovers which carry over when you convert the computer's base 2 (binary) number into a more human readable base 10 number.

DigitalOcean 提供 Floating IP 功能

DigitalOcean 推出的新功能，可以註冊 IP 並且動態掛到某個 droplet 上：「Floating IPs: Start Architecting Your Applications for High Availability」。

DEC64 浮點數

value = coefficient * 10exponent

0 與 NaN 是特別處理：

There are 255 possible representations of zero. They are all considered to be equal.

There is a special value called nan that has a coefficient of 0 and an exponent of -128. The result of division by zero is nan. nan is also the result of operations that produce results that are too large to be represented. nan is equal to itself.

```   Domain Name: DEC64.COM
Registrar: 1 & 1 INTERNET AG
Whois Server: whois.schlund.info
Referral URL: http://1and1.com
Name Server: NS57.1AND1.COM
Name Server: NS58.1AND1.COM
Status: ok
Updated Date: 03-jun-2013
Creation Date: 02-jun-2009
Expiration Date: 02-jun-2014```

Fast Inverse Square Root 演算法...

Update：請參考 comment，看起來中文版有誤譯...