From 825ec086814b5b4b8f3f8e2e2531cc9db1e68606 Mon Sep 17 00:00:00 2001 From: David Hewitt <1939362+davidhewitt@users.noreply.github.com> Date: Fri, 21 May 2021 08:34:51 +0100 Subject: [PATCH] pyo3-build-config: docs wip --- guide/src/SUMMARY.md | 1 + guide/src/building_and_distribution.md | 7 ++- .../multiple_python_versions.md | 60 +++++++++++++++++++ guide/src/faq.md | 2 +- pyo3-build-config/src/lib.rs | 2 + pyo3-macros-backend/Cargo.toml | 2 +- 6 files changed, 69 insertions(+), 5 deletions(-) create mode 100644 guide/src/building_and_distribution/multiple_python_versions.md diff --git a/guide/src/SUMMARY.md b/guide/src/SUMMARY.md index 28c4a6fb..4d2df2f9 100644 --- a/guide/src/SUMMARY.md +++ b/guide/src/SUMMARY.md @@ -19,6 +19,7 @@ - [Features Reference](features.md) - [Advanced Topics](advanced.md) - [Building and Distribution](building_and_distribution.md) + - [Supporting multiple Python versions](building_and_distribution/multiple_python_versions.md) - [PyPy support](building_and_distribution/pypy.md) - [Useful Crates](ecosystem.md) - [Logging](ecosystem/logging.md) diff --git a/guide/src/building_and_distribution.md b/guide/src/building_and_distribution.md index b1293241..57a45f26 100644 --- a/guide/src/building_and_distribution.md +++ b/guide/src/building_and_distribution.md @@ -43,8 +43,7 @@ There are two ways to distribute your module as a Python package: [setuptools-ru By default, Python extension modules can only be used with the same Python version they were compiled against -- if you build an extension module with Python 3.5, you can't import it using Python 3.8. [PEP 384](https://www.python.org/dev/peps/pep-0384/) introduced the idea of the limited Python API, which would have a stable ABI enabling extension modules built with it to be used against multiple Python versions. This is also known as `abi3`. -Note that [maturin] >= 0.9.0 or [setuptools-rust] >= 0.11.4 support `abi3` wheels. -See the [corresponding](https://github.com/PyO3/maturin/pull/353) [PRs](https://github.com/PyO3/setuptools-rust/pull/82) for more. +The advantage of building extension module using the limited Python API is that you only need to build and distribute a single copy (for each OS / architecture), and your users can install it on all Python versions from your [minimum version](#minimum-python-version-for-abi3) and up. The downside of this is that PyO3 can't use optimizations which rely on being compiled against a known exact Python version. It's up to you to decide whether this matters for your extension module. It's also possible to design your extension module such that you can distribute `abi3` wheels but allow users compiling from source to benefit from additional optimizations - see the [support for multiple python versions](./building_and_distribution/multiple_python_versions.html) section of this guide, in particular the `#[cfg(Py_LIMITED_API)]` flag. There are three steps involved in making use of `abi3` when building Python packages as wheels: @@ -55,7 +54,8 @@ There are three steps involved in making use of `abi3` when building Python pack pyo3 = { {{#PYO3_CRATE_VERSION}}, features = ["abi3"] } ``` -2. Ensure that the built shared objects are correctly marked as `abi3`. This is accomplished by telling your build system that you're using the limited API. +2. Ensure that the built shared objects are correctly marked as `abi3`. This is accomplished by telling your build system that you're using the limited API. [maturin] >= 0.9.0 and [setuptools-rust] >= 0.11.4 support `abi3` wheels. +See the [corresponding](https://github.com/PyO3/maturin/pull/353) [PRs](https://github.com/PyO3/setuptools-rust/pull/82) for more. 3. Ensure that the `.whl` is correctly marked as `abi3`. For projects using `setuptools`, this is accomplished by passing `--py-limited-api=cp3x` (where `x` is the minimum Python version supported by the wheel, e.g. `--py-limited-api=cp35` for Python 3.5) to `setup.py bdist_wheel`. @@ -76,6 +76,7 @@ not work when compiling for `abi3`. These are: - `#[text_signature]` does not work on classes until Python 3.10 or greater. - The `dict` and `weakref` options on classes are not supported until Python 3.9 or greater. - The buffer API is not supported. +- Optimizations which rely on knowledge of the exact Python version compiled against. ## Cross Compiling diff --git a/guide/src/building_and_distribution/multiple_python_versions.md b/guide/src/building_and_distribution/multiple_python_versions.md new file mode 100644 index 00000000..153d6e32 --- /dev/null +++ b/guide/src/building_and_distribution/multiple_python_versions.md @@ -0,0 +1,60 @@ +# Supporting multiple Python versions + +PyO3 supports all actively-supported Python 3 and PyPy versions. As much as possible, this is done internally to PyO3 so that your crate's code does not need to adapt to the differences between each version. However, as Python features grow and change between versions, PyO3 cannot a completely identical API for every Python version. This may require you to add conditional compilation to your crate or runtime checks for the Python version. + +This section of the guide first introduces the `pyo3-build-config` crate, which you can use as a `build-dependency` to add additional `#[cfg]` flags which allow you to support multiple Python versions at compile-time. + +Second, we'll show how to check the Python version at runtime. This can be useful when building for multiple versions with the `abi3` feature, where the Python API compiled against is not always the same as the one in use. + +## Conditional compilation for different Python versions + +The `pyo3-build-config` exposes multiple [`#[cfg]` flags](https://doc.rust-lang.org/rust-by-example/attribute/cfg.html) which can be used to conditionally compile code for a given Python version. PyO3 itself depends on this crate, so by using it you can be sure that you are configured correctly for the Python version PyO3 is building against. + +This allows us to write code like the following + +```rust,ignore +#[cfg(Py_3_7)] +fn function_only_supported_on_python_3_7_and_up() { } + +#[cfg(not(Py_3_8))] +fn function_only_supported_before_python_3_8() { } + +#[cfg(not(Py_LIMITED_API))] +fn function_incompatible_with_abi3_feature() { } +``` + +The following sections first show how to add these `#[cfg]` flags to your build process, and then cover some common patterns flags in a little more detail. + +To see a full reference of all the `#[cfg]` flags provided, see the [`pyo3-build-cfg` docs](https://docs.rs/pyo3-build-config). + +### Using `pyo3-build-config` + +You can use the `#[cfg]` flags in just two steps: + +1. Add `pyo3-build-config` it to your crate's build dependencies in `Cargo.toml`: + + ```toml + [build-dependencies] + pyo3-build-config = "{{#PYO3_CRATE_VERSION}}" + ``` + +2. Add a [`build.rs`](https://doc.rust-lang.org/cargo/reference/build-scripts.html) file to your crate with the following contents: + + ```rust,ignore + fn main() { + // If you have an existing build.rs file, just add this line to it. + pyo3_build_config::use_pyo3_cfgs(); + } + ``` + +After these steps you are ready to annotate your code! + +### Common usages of `pyo3-build-cfg` flags + +The following are some common patterns implemented using these flags: + +// TODO + +## Checking the Python version at runtime + +// TODO diff --git a/guide/src/faq.md b/guide/src/faq.md index aeb2bc00..38b5260a 100644 --- a/guide/src/faq.md +++ b/guide/src/faq.md @@ -21,7 +21,7 @@ Currently, [#341](https://github.com/PyO3/pyo3/issues/341) causes `cargo test` t ```toml [dependencies.pyo3] -version = "{{#PYO3_VERSION}}" +version = "{{#PYO3_CRATE_VERSION}}" [features] extension-module = ["pyo3/extension-module"] diff --git a/pyo3-build-config/src/lib.rs b/pyo3-build-config/src/lib.rs index 300ed443..13525e29 100644 --- a/pyo3-build-config/src/lib.rs +++ b/pyo3-build-config/src/lib.rs @@ -2,6 +2,8 @@ //! //! The only public API currently exposed is [`use_pyo3_cfgs`], which is intended to be used in //! build scripts to add a standard set of `#[cfg]` flags for handling multiple Python versions. +//! +//! TODO: tabulate all the flags here #[allow(dead_code)] // TODO cover this using tests mod impl_; diff --git a/pyo3-macros-backend/Cargo.toml b/pyo3-macros-backend/Cargo.toml index d8e0c443..91979046 100644 --- a/pyo3-macros-backend/Cargo.toml +++ b/pyo3-macros-backend/Cargo.toml @@ -23,4 +23,4 @@ default-features = false features = ["derive", "parsing", "printing", "clone-impls", "full", "extra-traits"] [build-dependencies] -pyo3-build-config = { path = "../pyo3-build-config", version = "0.14.0-alpha.0" } +pyo3-build-config = { path = "../pyo3-build-config", version = "=0.14.0-alpha.0" }