luxolus
/
pyo3
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

auto-initialize: better error messages and embedding docs

This commit is contained in:
David Hewitt 2021-04-11 09:27:50 +01:00
parent a9942bd987
commit 355df5a37f
5 changed files with 89 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
build.rs
View File

@ -17,10 +17,15 @@ const CFG_KEY: &str = "py_sys_config";
type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;
// A simple macro for returning an error. Resembles failure::bail and anyhow::bail.
// A simple macro for returning an error. Resembles anyhow::bail.
macro_rules! bail {
($msg: expr) => { return Err($msg.into()); };
($fmt: literal $(, $args: expr)+) => { return Err(format!($fmt $(,$args)+).into()); };
($fmt: literal $($args: tt)+) => { return Err(format!($fmt $($args)+).into()); };
}
// A simple macro for checking a condition. Resembles anyhow::ensure.
macro_rules! ensure {
($condition:expr, $($args: tt)+) => { if !($condition) { bail!($($args)+) } };
}
// Show warning. If needed, please extend this macro to support arguments.
@ -757,8 +762,30 @@ fn configure(interpreter_config: &InterpreterConfig) -> Result<()> {
_ => {}
}
if interpreter_config.shared {
println!("cargo:rustc-cfg=Py_SHARED");
if env::var_os("CARGO_FEATURE_AUTO_INITIALIZE").is_some() {
ensure!(
interpreter_config.shared,
"The `auto-initialize` feature is enabled, but your python installation only supports \
embedding the Python interpreter statically. If you are attempting to run tests, or a \
binary which is okay to link dynamically, install a Python distribution which ships \
with the Python shared library.\n\
\n\
Embedding the Python interpreter statically does not yet have first-class support in \
PyO3. If you are sure you intend to do this, disable the `auto-initialize` feature.\n\
\n\
For more information, see \
https://pyo3.rs/v{pyo3_version}/\
building_and_distribution.html#embedding-python-in-rust",
pyo3_version = env::var("CARGO_PKG_VERSION").unwrap()
);
// TODO: PYO3_CI env is a hack to workaround CI with PyPy, where the `dev-dependencies`
// currently cause `auto-initialize` to be enabled in CI.
// Once cargo's `resolver = "2"` is stable (~ MSRV Rust 1.52), remove this.
ensure!(
!interpreter_config.is_pypy() || env::var_os("PYO3_CI").is_some(),
"The `auto-initialize` feature is not supported with PyPy."
);
}
let is_abi3 = is_abi3();
@ -854,10 +881,6 @@ fn abi3_without_interpreter() -> Result<()> {
// complains that the crate using pyo3 does not contains a `#[link(...)]`
// attribute with pythonXY.
println!("cargo:rustc-link-lib=pythonXY:python3");
// Match `get_config_from_interpreter()` and `windows_hardcoded_cross_compile()`:
// assume "Py_ENABLE_SHARED" to be set on Windows.
println!("cargo:rustc-cfg=Py_SHARED");
}
Ok(())
@ -920,12 +943,6 @@ fn main_impl() -> Result<()> {
}
}
// TODO: this is a hack to workaround compile_error! warnings about auto-initialize on PyPy
// Once cargo's `resolver = "2"` is stable (~ MSRV Rust 1.52), remove this.
if env::var_os("PYO3_CI").is_some() {
println!("cargo:rustc-cfg=__pyo3_ci");
}
Ok(())
}

45
guide/src/building_and_distribution.md
View File

@ -119,6 +119,50 @@ cargo build --target x86_64-pc-windows-gnu
Any of the `abi3-py3*` features can be enabled instead of setting `PYO3_CROSS_PYTHON_VERSION` in the above examples.
## Embedding Python in Rust
If you want to embed the Python interpreter inside a Rust program, there are two modes in which this can be done: dynamically and statically. We'll cover each of these modes in the following sections. Each of them affect how you must distribute your program. Instead of learning how to do this yourself, you might want to consider using a project like [PyOxidizer] to ship your application and all of its dependencies in a single file.
PyO3 automatically switches between the two linking modes depending on whether the Python distribution you have configured PyO3 to use ([see above](#python-version)) contains a shared library or a static library. The static library is most often seen in Python distributions compiled from source without the `--enable-shared` configuration option. For example, this is the default for `pyenv` on macOS.
### Dynamically embedding the Python interpreter
Embedding the Python interpreter dynamically is much easier than doing so statically. This is done by linking your program against a Python shared library (such as `libpython.3.9.so` on UNIX, or `python39.dll` on Windows). The implementation of the Python interpreter resides inside the shared library. This means that when the OS runs your Rust program it also needs to be able to find the Python shared library.
This mode of embedding works well for Rust tests which need access to the Python interpreter. It is also great for Rust software which is installed inside a Python virtualenv, because the virtualenv sets up appropriate environment variables to locate the correct Python shared library.
For distributing your program to non-technical users, you will have to consider including the Python shared library in your distribution as well as setting up wrapper scripts to set the right environment variables (such as `LD_LIBRARY_PATH` on UNIX, or `PATH` on Windows).
### Statically embedding the Python interpreter
Embedding the Python interpreter statically means including the contents of a Python static library directly inside your Rust binary. This means that to distribute your program you only need to ship your binary file: it contains the Python interpreter inside the binary!
On Windows static linking is almost never done, so Python distributions don't usually include a static library. The information below applies only to UNIX.
The Python static library is usually called `libpython.a`.
Static linking has a lot of complications, listed below. For these reasons PyO3 does not yet have first-class support for this embedding mode. See [issue 416 on PyO3's Github](https://github.com/PyO3/pyo3/issues/416) for more information and to discuss any issues you encounter.
The [`auto-initialize`](features.md#auto-initialize) feature is deliberately disabled when embedding the interpreter statically because this is often unintentionally done by new users to PyO3 running test programs. Trying out PyO3 is much easier using dynamic embedding.
The known complications are:
- To import compiled extension modules (such as other Rust extension modules, or those written in C), your binary must have the correct linker flags set during compilation to export the original contents of `libpython.a` so that extensions can use them (e.g. `-Wl,--export-dynamic`).
- The C compiler and flags which were used to create `libpython.a` must be compatible with your Rust compiler and flags, else you will experience compilation failures.
Significantly different compiler versions may see errors like this:
```ignore
lto1: fatal error: bytecode stream in file 'rust-numpy/target/release/deps/libpyo3-6a7fb2ed970dbf26.rlib' generated with LTO version 6.0 instead of the expected 6.2
```
Mismatching flags may lead to errors like this:
```ignore
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libpython3.9.a(zlibmodule.o): relocation R_X86_64_32 against `.data' can not be used when making a PIE object; recompile with -fPIE
```
If you encounter these or other complications when linking the interpreter statically, discuss them on [issue 416 on PyO3's Github](https://github.com/PyO3/pyo3/issues/416). It is hoped that eventually that discussion will contain enough information and solutions that PyO3 can offer first-class support for static embedding.
## Bazel
For an example of how to build python extensions using Bazel, see https://github.com/TheButlah/rules_pyo3
@ -126,3 +170,4 @@ For an example of how to build python extensions using Bazel, see https://github
[maturin]: https://github.com/PyO3/maturin
[setuptools-rust]: https://github.com/PyO3/setuptools-rust
[PyOxidizer]: https://github.com/indygreg/PyOxidizer

2
guide/src/features.md
View File

@ -30,7 +30,7 @@ See the [building and distribution](building_and_distribution.md#minimum-python-
## Features for embedding Python in Rust
### `auto-initalize`
### `auto-initialize`
This feature changes [`Python::with_gil`]({{#PYO3_DOCS_URL}}/pyo3/struct.Python.html#method.with_gil) and [`Python::acquire_gil`]({{#PYO3_DOCS_URL}}/pyo3/struct.Python.html#method.acquire_gil) to automatically initialize a Python interpreter (by calling [`prepare_freethreaded_python`]({{#PYO3_DOCS_URL}}/pyo3/fn.prepare_freethreaded_python.html)) if needed.

45
src/gil.rs
View File

@ -46,9 +46,6 @@ pub(crate) fn gil_is_acquired() -> bool {
/// this function has no effect.
///
/// # Availability
/// This function is only available when linking against Python distributions that contain a
/// shared library.
///
/// This function is not available on PyPy.
///
/// # Panics
@ -70,7 +67,7 @@ pub(crate) fn gil_is_acquired() -> bool {
/// });
/// }
/// ```
#[cfg(all(Py_SHARED, not(PyPy)))]
#[cfg(not(PyPy))]
#[allow(clippy::clippy::collapsible_if)] // for if cfg!
pub fn prepare_freethreaded_python() {
// Protect against race conditions when Python is not yet initialized and multiple threads
@ -110,9 +107,6 @@ pub fn prepare_freethreaded_python() {
/// initialize correctly on the second run.)
///
/// # Availability
/// This function is only available when linking against Python distributions that contain a shared
/// library.
///
/// This function is not available on PyPy.
///
/// # Panics
@ -137,7 +131,7 @@ pub fn prepare_freethreaded_python() {
/// }
/// }
/// ```
#[cfg(all(Py_SHARED, not(PyPy)))]
#[cfg(not(PyPy))]
#[allow(clippy::clippy::collapsible_if)] // for if cfg!
pub unsafe fn with_embedded_python_interpreter<F, R>(f: F) -> R
where
@ -219,32 +213,9 @@ impl GILGuard {
// auto-initialize so this avoids breaking existing builds.
// - Otherwise, just check the GIL is initialized.
cfg_if::cfg_if! {
if #[cfg(all(feature = "auto-initialize", Py_SHARED, not(PyPy)))] {
if #[cfg(all(feature = "auto-initialize", not(PyPy)))] {
prepare_freethreaded_python();
} else if #[cfg(all(feature = "auto-initialize", not(Py_SHARED), not(__pyo3_ci)))] {
compile_error!(concat!(
"The `auto-initialize` feature is not supported when linking Python ",
"statically instead of with a shared library.\n\n",
"Please disable the `auto-initialize` feature, for example by entering the following ",
"in your cargo.toml:\n\n",
" pyo3 = { version = \"",
env!("CARGO_PKG_VERSION"),
"\", default-features = false }\n\n",
"Alternatively, compile PyO3 using a Python distribution which contains a shared ",
"libary."
));
} else if #[cfg(all(feature = "auto-initialize", PyPy, not(__pyo3_ci)))] {
compile_error!(concat!(
"The `auto-initialize` feature is not supported by PyPy.\n\n",
"Please disable the `auto-initialize` feature, for example by entering the following ",
"in your cargo.toml:\n\n",
" pyo3 = { version = \"",
env!("CARGO_PKG_VERSION"),
"\", default-features = false }\n\n",
));
} else {
// extension module feature enabled and PyPy or static linking
// OR auto-initialize feature not enabled
START.call_once_force(|_| unsafe {
// Use call_once_force because if there is a panic because the interpreter is
// not initialized, it's fine for the user to initialize the interpreter and
@ -252,12 +223,18 @@ impl GILGuard {
assert_ne!(
ffi::Py_IsInitialized(),
0,
"The Python interpreter is not initalized and the `auto-initialize` feature is not enabled."
"The Python interpreter is not initalized and the `auto-initialize` \
feature is not enabled.\n\n\
Consider calling `pyo3::prepare_freethreaded_python()` before attempting \
to use Python APIs."
);
assert_ne!(
ffi::PyEval_ThreadsInitialized(),
0,
"Python threading is not initalized and the `auto-initialize` feature is not enabled."
"Python threading is not initalized and the `auto-initialize` feature is \
not enabled.\n\n\
Consider calling `pyo3::prepare_freethreaded_python()` before attempting \
to use Python APIs."
);
});
}

2
src/lib.rs
View File

@ -131,7 +131,7 @@ pub use crate::conversion::{
ToBorrowedObject, ToPyObject,
};
pub use crate::err::{PyDowncastError, PyErr, PyErrArguments, PyResult};
#[cfg(all(Py_SHARED, not(PyPy)))]
#[cfg(not(PyPy))]
pub use crate::gil::{prepare_freethreaded_python, with_embedded_python_interpreter};
pub use crate::gil::{GILGuard, GILPool};
pub use crate::instance::{Py, PyNativeType, PyObject};