Tag: pypy

PyPy JIT 的改善

在 PyPy 這邊看到 JIT 的重大進展：「Better JIT Support for Auto-Generated Python Code」。

他們在 Tornado 上重製出來效能問題，後面也都是用這個例子在測試：

If you render a big HTML template (example) using the Tornado templating engine, the template rendering is really not any faster than CPython.

看起來上的 workaround 是在撞到 trace limit 時標記起來，之後再遇到時就可以跳進 special mode，接著處理下去避免浪費掉之前處理過的 trace：

After we have hit the trace limit and no inlining has happened so far, we mark the outermost function as a source of huge traces. The next time we trace such a function, we do so in a special mode. In that mode, hitting the trace limit behaves differently: Instead of stopping the tracer and throwing away the trace produced so far, we will use the unfinished trace to produce machine code.

效能可以看到改善很多：

看起來這個概念有打算在 3.8 的時候放進去：

The work described in this post tiny bit experimental still, but we will release it as part of the upcoming 3.8 beta release, to get some more experience with it. Please grab a 3.8 release candidate, try it out and let us know your observations, good and bad!

Django 的 template engine 不怎麼快，用 Jinja2 可能是一個方法，但既有的 project 如果有遇到 template engine 的效能問題，也許也可以翻看看 PyPy 解得如何...

快速產生 SQLite 資料的方式：一分鐘內產生十億筆資料

在「Towards Inserting One Billion Rows in SQLite Under A Minute」這邊看到作者想要在一分鐘內在 MBP 2019 上面寫 1B 筆資料進 SQLite，裡面有些方法還蠻值得玩一下的，這台 MBP 2019 機器的規格是：

The machine I am using is MacBook Pro, 2019 (2.4 GHz Quad Core i5, 8GB, 256GB SSD, Big Sur 11.1)

第一版是 Python 寫的，塞 10M 筆花了 15 分鐘：

In this script, I tried to insert 10M rows, one by one, in a for loop. This version took close to 15 minutes, sparked my curiosity and made me explore further to reduce the time.

加了五個 PRAGMA 的版本變成 100M 筆十分鐘：

The naive for loop version took about 10 minutes to insert 100M rows.

用批次處理則可以降到八分半：

The batched version took about 8.5 minutes to insert 100M rows.

再來是拿經典神器 PyPy 出來用，降到兩分半：

All I had to do was run my existing code, without any change, using PyPy. It worked and the speed bump was phenomenal. The batched version took only 2.5 minutes to insert 100M rows. I got close to 3.5x speed :)

接下來就是跳槽到 Rust 了，中間也有不少 tuning 相關的討論，但直接先跳到最後面好了... 最後 100M 只用了 33 秒：

I created a threaded version, where I had one writer thread that received data from a channel and four other threads which pushed data to the channel. This is the current best version which took about 32.37 seconds.

能用 PyPy 的地方還是可以考慮一下的...

PyPy 5.9 支援 Pandas 與 NumPy 了

PyPy 5.9 支援 machine learning 常用的 Pandas 與 NumPy 了：「PyPy v5.9 Released, Now Supports Pandas, NumPy」，包括 2.7 與 3.5 的相容版本：

The PyPy team is proud to release both PyPy3.5 v5.9 (a beta-quality interpreter for Python 3.5 syntax) and PyPy2.7 v5.9 (an interpreter supporting Python 2.7 syntax).

對於使用 Python 大量計算的人來說可以進場測試了 XD