6.8 KiB
Hot Reloading ("Live" Development)
Summary
When doing development/debug builds with the nightly toolchain, conduwuit is modular using dynamic libraries and various parts of the application are hot reloadable such as the APIs, database, service, admin room, RocksDB, etc. These are all split up into individual workspace crates. This design also only reloads the libraries that actually changed. hot_lib_reload was considered during early research of this, however its too limited in its functionality for conduwuit's use-case where we have many things going on (e.g. async code, tower/tower-http, axum, hyper, hickory_resolver, ruma, RocksDB, admin room, etc etc etc), so a custom solution using libloading was required.
Release builds are unaffected and still function as fully normal static binaries. Some bugs such as shutdown issues have been fixed as apart of this implementation, but there is no functional difference with release builds as apart of this. You cannot hot reload release binaries.
Currently, this development setup only works on x86_64 and aarch64 Linux glibc. musl explicitly does not support hot reloadable libraries, and does not implement dlclose. macOS does not fully support our usage of RTLD_GLOBAL possibly due to some thread-local issues. This Rust issue may be of relevance, specifically this comment. It may be possible to get it working on only very modern macOS versions such as at least Sonoma, as currently loading dylibs is supported, but not unloading them in our setup, and the cited comment mentions an Apple WWDC confirming there have been TLS changes to somewhat make this possible.
Additionally, at the time of writing io_uring and jemalloc are unstable at runtime when using hot reloading. Release builds are not affected.
Requirements
As mentioned above this requires the nightly toolchain. This is due to reliance on various Cargo.toml features that are only available on nightly, most specifically RUSTFLAGS in Cargo.toml. Some of the implementation could also be simpler based on other various nightly features. We hope lots of nightly features start making it out of nightly sooner as there have been dozens of very helpful features that have been stuck in nightly ("unstable") for at least 5+ years that would make this a lot more cleaner, simpler, etc and have worked perfectly fine in testing.
This currently only works on x86_64/aarch64 Linux with a glibc C library. musl C library, macOS, and likely other host architectures are not supported (if other architectures work, feel free to let us know and/or make a PR updating this).
This should work on GNU ld and lld (rust-lld) and gcc/clang, however if you happen to have linker issues it's recommended to try using mold or gold linkers, and please let us know in the conduwuit Matrix room the linker error and what linker solved this issue so we can figure out a solution. Ideally there should be minimal friction to using this, and in the future a build script (build.rs) may be suitable to making this easier to use if the capabilities allow us.
Usage
As of 19 May 2024, the instructions for using this are:
-
Have patience. Don't feel hesitated to join the conduwuit Matrix room to receive help in using this. As indicated by the various rustflags used and some of the interesting issues linked at the bottom, this is definitely not something the Rust ecosystem or toolchain is used to doing.
-
Install the nightly toolchain using rustup. You may need to use
rustup override set nightlyin your local conduwuit directory, or usecargo +nightlyfor all actions. -
Uncomment
cargo-featuresat the top level / root Cargo.toml -
Scroll down to the
# Developer profilesection and uncomment ALL the rustflags for each dev profile and their respective packages. -
In each workspace crate's Cargo.toml (everything under
src/*ANDdeps/rust-rocksdb/Cargo.toml), uncomment thedylibcrate type under[lib]. -
Due to this rpath issue, you must export the
LD_LIBRARY_PATHenvironment variable to your nightly Rust toolchain library directory. If using rustup (hopefully), use this:export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/ -
Start the server. You can use
cargo +nightly runfor this along with the standard. -
Make some changes where you need to.
-
In a separate terminal window in the same directory (or using a terminal multiplexer like tmux), run the build Cargo command
cargo +nightly build. Cargo should only rebuild what was changed / what's necessary, so it should not be rebuilding all the crates. -
In your conduwuit server terminal, hit/send
CTRL+Csignal. This will tell conduwuit to find which libraries need to be reloaded, and reloads them as necessary. -
If there were no errors, it will tell you it successfully reloaded
#modules, and your changes should now be visible. Repeat 7 - 9 as needed.
To shutdown conduwuit in this setup, hit/send CTRL+\. Normal builds still shutdown with CTRL+C as usual.
Steps 1 - 5 are the initial first-time steps for using this. To remove the hot reload setup, revert/comment all the Cargo.toml changes.
As mentioned in the requirements section, if you happen to have some linker issues, try using the -fuse-ld= rustflag and specify mold or gold in all the rustflags definitions in the top level Cargo.toml, and please let us know in the conduwuit Matrix room the problem. mold can be installed typically through your distro, and gold is provided by the binutils package.
It's possible a helper script can be made to do all of this, or most preferably a specially made build script (build.rs). cargo watch support will be implemented soon which will eliminate the need to manually run cargo build all together.
Design
The initial implementation PR is available here.
TODO: document a lot more stuff about all the design here
