Pragtical v3.9.0 Release
The time has finally come! A new contestant enters the arena and promises some performance gains to help make the editor snappier and more responsive, with another new friend to make sure it works.
Native Tokenizer
I'm happy to announce we finally got a native tokenizer in the form of a Lua module and benchmarks have proved to provide performance gains that can reach a 20X to 30X improvement over the previous Lua tokenizer.
Here is a video showcasing the evident performance gains with a 4000 lines file.
This native tokenizer is enabled by default (config.native_tokenizer) and can be toggled from the settings UI.
You can benchmark it with the new provided repo script:
pragtical run scripts/lua/benchmarks/tokenizer.lua \
<path-to-source-file> <iterations_count|defaults-to-3>
To keep compatibility I ported most functionality from utf8extra and regex Lua modules into non Lua dependent C for usage on the native tokenizer. Some time ago I started an initial tokenizer port from Lua to C++ and with the help of Copilot got it ported to C and working! GitHub Copilot basically implemented it since the original code was mostly interfaces (with the exception of the utf8extra and regex ports). Hopefully if it breaks it will not come biting at us 😂, but is incredible how far these LLM models have come...
Testing Framework
Another worthwhile addition on this release is the new testing framework, accompanied of tests to check our Lua C modules and some pure Lua component tests like core.common courtesy of Walkero!
Here is how the tests look at the terminal emulator:
This test framework supports running the tests inside coroutines, so you can
coroutine.yield() from your tests in the same way done with
core.add_thread/core.add_background_thread (check the runner.lua test for an
example). And the most important aspect, the whole pragtical runtime is
available to the written tests!
A new core.test module was added to easily describe and write tests. Also,
a new cli command test (eg: pragtical test <dir_or_test_file>) will allow
to automatically run a directory containing lua test files or directly a single
test file.
It should help us better catch errors on the API's, hopefully more test scripts keep coming up to help us battle test the project :)
Other Enhancements and Fixes
Some improvements were made for macOS (at least that is what I think since macOS is not my daily driver for better testing):
- The shared memory module used for IPC was rewritten to better support of macOS.
- PPM was not been packaged for official macOS binary releases, which is fixed now.
Fixes and enhancements thanks to badiku issue reports:
- Drawing operations where processing before an update was called, this caused various issues with plugins that relied on update been call before draw.
- The core:open-file command now trims double quotes from the given path
to allow opening it, for example
"/path/to/file"gets trim into/path/to/file, this allows properly opening files which path was copied from a file explorer and pasted into Pragtical.
Finally, a new zig meson cross file was added to allow targeting older glibc v2.17. This should permit more easily targeting older Linux distributions. Building with it seems to work but haven't plugged it yet into our GitHub CI builds, hopefully we can do it for future releases.
Downloads on GitHub.
Changes Log
New Features
- Native tokenizer implementation (#467)
Fixes
-
drawwhitespace plugin: properly handle nil selections (#463)
-
Ensure first update ran on views before draw (#469)
Enhancements
-
Add testing framework and initial set of tests (#465)
-
Compatible rewrite of shmem (#466)
-
Trim double quotes from path on core:open-file (#470)