diff --git a/guide/src/SUMMARY.md b/guide/src/SUMMARY.md index 0227dbd4..e07b41a7 100644 --- a/guide/src/SUMMARY.md +++ b/guide/src/SUMMARY.md @@ -4,35 +4,36 @@ --- -- [Getting Started](getting_started.md) -- [Python Modules](module.md) -- [Python Functions](function.md) -- [Python Classes](class.md) +- [Getting started](getting_started.md) +- [Python modules](module.md) +- [Python functions](function.md) + - [Function signatures](function/signature.md) +- [Python classes](class.md) - [Class customizations](class/protocols.md) - [Basic object customization](class/object.md) - [Emulating numeric types](class/numeric.md) - [Emulating callable objects](class/call.md) -- [Type Conversions](conversions.md) +- [Type conversions](conversions.md) - [Mapping of Rust types to Python types](conversions/tables.md)] - [Conversion traits](conversions/traits.md)] -- [Python Exceptions](exception.md) +- [Python exceptions](exception.md) - [Calling Python from Rust](python_from_rust.md) - [GIL, mutability and object types](types.md) - [Parallelism](parallelism.md) - [Debugging](debugging.md) -- [Features Reference](features.md) -- [Memory Management](memory.md) -- [Advanced Topics](advanced.md) -- [Building and Distribution](building_and_distribution.md) +- [Features reference](features.md) +- [Memory management](memory.md) +- [Advanced topics](advanced.md) +- [Building and distribution](building_and_distribution.md) - [Supporting multiple Python versions](building_and_distribution/multiple_python_versions.md) -- [Useful Crates](ecosystem.md) +- [Useful crates](ecosystem.md) - [Logging](ecosystem/logging.md) - - [Async / Await](ecosystem/async-await.md) -- [FAQ & Troubleshooting](faq.md) + - [Using `async` and `await`](ecosystem/async-await.md) +- [FAQ and troubleshooting](faq.md) --- -[Appendix A: Migration Guide](migration.md) +[Appendix A: Migration guide](migration.md) [Appendix B: PyO3 and rust-cpython](rust_cpython.md) [Appendix C: Trait bounds](trait_bounds.md) [Appendix D: Python typing hints](python_typing_hints.md) diff --git a/guide/src/advanced.md b/guide/src/advanced.md index 327790d4..8264c14d 100644 --- a/guide/src/advanced.md +++ b/guide/src/advanced.md @@ -6,7 +6,7 @@ PyO3 exposes much of Python's C API through the `ffi` module. The C API is naturally unsafe and requires you to manage reference counts, errors and specific invariants yourself. Please refer to the [C API Reference Manual](https://docs.python.org/3/c-api/) and [The Rustonomicon](https://doc.rust-lang.org/nightly/nomicon/ffi.html) before using any function from that API. -## Memory Management +## Memory management PyO3's `&PyAny` "owned references" and `Py` smart pointers are used to access memory stored in Python's heap. This memory sometimes lives for longer diff --git a/guide/src/building_and_distribution.md b/guide/src/building_and_distribution.md index e59525b2..2c39f2db 100644 --- a/guide/src/building_and_distribution.md +++ b/guide/src/building_and_distribution.md @@ -1,4 +1,4 @@ -# Building and Distribution +# Building and distribution This chapter of the guide goes into detail on how to build and distribute projects using PyO3. The way to achieve this is very different depending on whether the project is a Python module implemented in Rust, or a Rust binary embedding Python. For both types of project there are also common problems such as the Python version to build for and the [linker](https://en.wikipedia.org/wiki/Linker_(computing)) arguments to use. diff --git a/guide/src/class.md b/guide/src/class.md index f3857072..58400267 100644 --- a/guide/src/class.md +++ b/guide/src/class.md @@ -1,4 +1,4 @@ -# Python Classes +# Python classes PyO3 exposes a group of attributes powered by Rust's proc macro system for defining Python classes as Rust structs. @@ -614,29 +614,10 @@ impl MyClass { ## Method arguments -By default, PyO3 uses function signatures to determine which arguments are required. Then it scans -the incoming `args` and `kwargs` parameters. If it can not find all required -parameters, it raises a `TypeError` exception. It is possible to override the default behavior -with the `#[args(...)]` attribute. This attribute accepts a comma separated list of parameters in -the form of `attr_name="default value"`. Each parameter has to match the method parameter by name. +Similar to `#[pyfunction]`, the `#[args]` attribute can be used to specify the way that `#[pymethods]` accept arguments. Consult the documentation for [`function signatures`](./function/signature.md) to see the parameters this attribute accepts. -Each parameter can be one of the following types: +The following example defines a class `MyClass` with a method `method`. This method has an `#[args]` attribute which sets default values for `num` and `name`, and indicates that `py_args` should collect all extra positional arguments and `py_kwargs` all extra keyword arguments: - * `"/"`: positional-only arguments separator, each parameter defined before `"/"` is a - positional-only parameter. - Corresponds to python's `def meth(arg1, arg2, ..., /, argN..)`. - * `"*"`: var arguments separator, each parameter defined after `"*"` is a keyword-only parameter. - Corresponds to python's `def meth(*, arg1.., arg2=..)`. - * `args="*"`: "args" is var args, corresponds to Python's `def meth(*args)`. Type of the `args` - parameter has to be `&PyTuple`. - * `kwargs="**"`: "kwargs" receives keyword arguments, corresponds to Python's `def meth(**kwargs)`. - The type of the `kwargs` parameter has to be `Option<&PyDict>`. - * `arg="Value"`: arguments with default value. Corresponds to Python's `def meth(arg=Value)`. - If the `arg` argument is defined after var arguments, it is treated as a keyword-only argument. - Note that `Value` has to be valid rust code, PyO3 just inserts it into the generated - code unmodified. - -Example: ```rust # use pyo3::prelude::*; use pyo3::types::{PyDict, PyTuple}; @@ -665,35 +646,26 @@ impl MyClass { name: &str, py_args: &PyTuple, py_kwargs: Option<&PyDict>, - ) -> PyResult { + ) -> String { + let num_before = self.num; self.num = num; - Ok(format!( - "py_args={:?}, py_kwargs={:?}, name={}, num={}", - py_args, py_kwargs, name, self.num - )) - } - - fn make_change(&mut self, num: i32) -> PyResult { - self.num = num; - Ok(format!("num={}", self.num)) + format!( + "py_args={:?}, py_kwargs={:?}, name={}, num={} num_before={}", + py_args, py_kwargs, name, self.num, num_before, + ) } } ``` -N.B. the position of the `"/"` and `"*"` arguments (if included) control the system of handling positional and keyword arguments. In Python: -```python -import mymodule -mc = mymodule.MyClass() -print(mc.method(44, False, "World", 666, x=44, y=55)) -print(mc.method(num=-1, name="World")) -print(mc.make_change(44, False)) -``` -Produces output: -```text -py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44 -py_args=(), py_kwargs=None, name=World, num=-1 -num=44 -num=-1 +In Python this might be used like: + +```python +>>> import mymodule +>>> mc = mymodule.MyClass() +>>> print(mc.method(44, False, "World", 666, x=44, y=55)) +py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44, num_before=-1 +>>> print(mc.method(num=-1, name="World")) +py_args=(), py_kwargs=None, name=World, num=-1, num_before=44 ``` ## Making class method signatures available to Python diff --git a/guide/src/conversions.md b/guide/src/conversions.md index f535e083..991c2061 100644 --- a/guide/src/conversions.md +++ b/guide/src/conversions.md @@ -1,3 +1,3 @@ -# Type Conversions +# Type conversions In this portion of the guide we'll talk about the mapping of Python types to Rust types offered by PyO3, as well as the traits available to perform conversions between them. diff --git a/guide/src/ecosystem.md b/guide/src/ecosystem.md index 0937a048..478edaec 100644 --- a/guide/src/ecosystem.md +++ b/guide/src/ecosystem.md @@ -1,4 +1,4 @@ -# The PyO3 Ecosystem +# The PyO3 ecosystem This portion of the guide is dedicated to crates which are external to the main PyO3 project and provide additional functionality you might find useful. diff --git a/guide/src/ecosystem/async-await.md b/guide/src/ecosystem/async-await.md index 262b27e6..1ab50065 100644 --- a/guide/src/ecosystem/async-await.md +++ b/guide/src/ecosystem/async-await.md @@ -1,4 +1,4 @@ -# Async / Await +# Using `async` and `await` If you are working with a Python library that makes use of async functions or wish to provide Python bindings for an async Rust library, [`pyo3-asyncio`](https://github.com/awestlake87/pyo3-asyncio) diff --git a/guide/src/exception.md b/guide/src/exception.md index 68b4846e..518122de 100644 --- a/guide/src/exception.md +++ b/guide/src/exception.md @@ -1,4 +1,4 @@ -# Python Exceptions +# Python exceptions ## Defining a new exception diff --git a/guide/src/faq.md b/guide/src/faq.md index 6b0dd57f..1389f131 100644 --- a/guide/src/faq.md +++ b/guide/src/faq.md @@ -1,4 +1,4 @@ -# Frequently Asked Questions / Troubleshooting +# Frequently Asked Questions and troubleshooting ## I'm experiencing deadlocks using PyO3 with lazy_static or once_cell! diff --git a/guide/src/features.md b/guide/src/features.md index 5974ba0f..8cb5b0bc 100644 --- a/guide/src/features.md +++ b/guide/src/features.md @@ -1,4 +1,4 @@ -# Features Reference +# Features reference PyO3 provides a number of Cargo features to customise functionality. This chapter of the guide provides detail on each of them. diff --git a/guide/src/function.md b/guide/src/function.md index 41550c18..008c9200 100644 --- a/guide/src/function.md +++ b/guide/src/function.md @@ -1,4 +1,4 @@ -# Python Functions +# Python functions The `#[pyfunction]` attribute is used to define a Python function from a Rust function. Once defined, the function needs to be added to a [module](./module.md) using the `wrap_pyfunction!` macro. @@ -19,17 +19,20 @@ fn my_extension(py: Python<'_>, m: &PyModule) -> PyResult<()> { } ``` -This chapter of the guide explains full usage of the `#[pyfunction]` attribute. The following topics are covered: +This chapter of the guide explains full usage of the `#[pyfunction]` attribute. In this first section, the following topics are covered: - [Function options](#function-options) - [`#[pyo3(name = "...")]`](#name) - [`#[pyo3(text_signature = "...")]`](#text_signature) - [`#[pyo3(pass_module)]`](#pass_module) -- [Argument parsing](#argument-parsing) - - [`#[pyo3(from_py_with = "...")]`](#from_py_with) +- [Per-argument options](#per-argument-options) - [Advanced function patterns](#advanced-function-patterns) - [`#[pyfn]` shorthand](#pyfn-shorthand) +There are also additional sections on the following topics: + +- [Function Signatures](./function/signature.md) + ## Function options The `#[pyo3]` attribute can be used to modify properties of the generated Python function. It can take any combination of the following options: @@ -118,27 +121,7 @@ The `#[pyo3]` attribute can be used to modify properties of the generated Python } ``` -## Argument parsing - -The `#[pyfunction]` attribute supports specifying details of argument parsing. The details are given in the section ["Method arguments" of the Classes chapter](class.md#method-arguments). Here is an example for a function that accepts arbitrary keyword arguments (`**kwargs` in Python syntax) and returns the number that was passed: - -```rust -use pyo3::prelude::*; -use pyo3::types::PyDict; - -#[pyfunction(kwds="**")] -fn num_kwds(kwds: Option<&PyDict>) -> usize { - kwds.map_or(0, |dict| dict.len()) -} - -#[pymodule] -fn module_with_functions(py: Python<'_>, m: &PyModule) -> PyResult<()> { - m.add_function(wrap_pyfunction!(num_kwds, m)?).unwrap(); - Ok(()) -} -``` - -### Per-argument options +## Per-argument options The `#[pyo3]` attribute can be used on individual arguments to modify properties of them in the generated function. It can take any combination of the following options: diff --git a/guide/src/function/signature.md b/guide/src/function/signature.md new file mode 100644 index 00000000..259fda19 --- /dev/null +++ b/guide/src/function/signature.md @@ -0,0 +1,97 @@ +# Function signatures + +The `#[pyfunction]` attribute also accepts parameters to control how the generated Python function accepts arguments. Just like in Python, arguments can be positional-only, keyword-only, or accept either. `*args` lists and `**kwargs` dicts can also be accepted. These parameters also work for `#[pymethods]` which will be introduced in the [Python Classes](../class.md) section of the guide. + +Like Python, by default PyO3 accepts all arguments as either positional or keyword arguments. The extra arguments to `#[pyfunction]` modify this behaviour. For example, below is a function that accepts arbitrary keyword arguments (`**kwargs` in Python syntax) and returns the number that was passed: + +```rust +use pyo3::prelude::*; +use pyo3::types::PyDict; + +#[pyfunction(kwds="**")] +fn num_kwds(kwds: Option<&PyDict>) -> usize { + kwds.map_or(0, |dict| dict.len()) +} + +#[pymodule] +fn module_with_functions(py: Python<'_>, m: &PyModule) -> PyResult<()> { + m.add_function(wrap_pyfunction!(num_kwds, m)?).unwrap(); + Ok(()) +} +``` + +The following parameters can be passed to the `#[pyfunction]` attribute: + + * `"/"`: positional-only arguments separator, each parameter defined before `"/"` is a + positional-only parameter. + Corresponds to python's `def meth(arg1, arg2, ..., /, argN..)`. + * `"*"`: var arguments separator, each parameter defined after `"*"` is a keyword-only parameter. + Corresponds to python's `def meth(*, arg1.., arg2=..)`. + * `args="*"`: "args" is var args, corresponds to Python's `def meth(*args)`. Type of the `args` + parameter has to be `&PyTuple`. + * `kwargs="**"`: "kwargs" receives keyword arguments, corresponds to Python's `def meth(**kwargs)`. + The type of the `kwargs` parameter has to be `Option<&PyDict>`. + * `arg="Value"`: arguments with default value. Corresponds to Python's `def meth(arg=Value)`. + If the `arg` argument is defined after var arguments, it is treated as a keyword-only argument. + Note that `Value` has to be valid rust code, PyO3 just inserts it into the generated + code unmodified. + +Example: +```rust +# use pyo3::prelude::*; +use pyo3::types::{PyDict, PyTuple}; +# +# #[pyclass] +# struct MyClass { +# num: i32, +# } +#[pymethods] +impl MyClass { + #[new] + #[args(num = "-1")] + fn new(num: i32) -> Self { + MyClass { num } + } + + #[args( + num = "10", + py_args = "*", + name = "\"Hello\"", + py_kwargs = "**" + )] + fn method( + &mut self, + num: i32, + name: &str, + py_args: &PyTuple, + py_kwargs: Option<&PyDict>, + ) -> PyResult { + self.num = num; + Ok(format!( + "py_args={:?}, py_kwargs={:?}, name={}, num={}", + py_args, py_kwargs, name, self.num + )) + } + + fn make_change(&mut self, num: i32) -> PyResult { + self.num = num; + Ok(format!("num={}", self.num)) + } +} +``` +N.B. the position of the `"/"` and `"*"` arguments (if included) control the system of handling positional and keyword arguments. In Python: +```python +import mymodule + +mc = mymodule.MyClass() +print(mc.method(44, False, "World", 666, x=44, y=55)) +print(mc.method(num=-1, name="World")) +print(mc.make_change(44, False)) +``` +Produces output: +```text +py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44 +py_args=(), py_kwargs=None, name=World, num=-1 +num=44 +num=-1 +``` diff --git a/guide/src/getting_started.md b/guide/src/getting_started.md index 16bb98ef..66ee841a 100644 --- a/guide/src/getting_started.md +++ b/guide/src/getting_started.md @@ -1,33 +1,32 @@ -# Instalation +# Installation -To get started using PyO3 you will need three things: a rust toolchain, a python environment, and a way to build. We'll cover each of these below. +To get started using PyO3 you will need three things: a rust toolchain, a python environment, and a way to build. We'll cover each of these below. ## Rust First, make sure you have rust installed on your system. If you haven't already done so you can do so by following the instructions [here](https://www.rust-lang.org/tools/install). PyO3 runs on both the `stable` and `nightly` versions so you can choose whichever one fits you best. The minimum required rust version is Rust 1.48. -if you can run `rustc --version` and the version is high enough you're good to go! +if you can run `rustc --version` and the version is high enough you're good to go! ## Python -To use PyO3 you need at least Python 3.7. While you can simply use the default Python version on your system, it is recommended to use a virtual environment. +To use PyO3 you need at least Python 3.7. While you can simply use the default Python version on your system, it is recommended to use a virtual environment. +## Virtualenvs -## Virtualenvs - -While you can use any virtualenv manager you like, we recommend the use of `pyenv` especially if you want to develop or test for multiple different python versions, so that is what the examples in this book will use. The installation instructions for `pyenv` can be found [here](https://github.com/pyenv/pyenv#getting-pyenv). +While you can use any virtualenv manager you like, we recommend the use of `pyenv` especially if you want to develop or test for multiple different python versions, so that is what the examples in this book will use. The installation instructions for `pyenv` can be found [here](https://github.com/pyenv/pyenv#getting-pyenv). Note that when using `pyenv` you should also set the following environment variable ```bash PYTHON_CONFIGURE_OPTS="--enable-shared" ``` + ### Building +There are a number of build and python package management systems such as [`setuptools-rust`](https://github.com/PyO3/setuptools-rust) or [manually](https://pyo3.rs/latest/building_and_distribution.html#manual-builds) we recommend the use of `maturin` which you can install [here](https://maturin.rs/installation.html). It is developed to work with PyO3 and is the most "batteries included" experience. `maturin` is just a python package so you can add it in any way that you install python packages. -There are a number of build and python package management systems such as [`setuptools-rust`](https://github.com/PyO3/setuptools-rust) or [manually](https://pyo3.rs/latest/building_and_distribution.html#manual-builds) we recommend the use of `maturin` which you can install [here](https://maturin.rs/installation.html). It is developed to work with PyO3 and is the most "batteries included" experience. `maturin` is just a python package so you can add it in any way that you install python packages. - -System Python: +System Python: ```bash pip install maturin --user ``` @@ -48,12 +47,11 @@ poetry: poetry add -D maturin ``` -after installation, you can run `maturin --version` to check that you have correctly installed it. - +after installation, you can run `maturin --version` to check that you have correctly installed it. # Starting a new project -Firstly you should create the folder and virtual environment that are going to contain your new project. Here we will use the recommended `pyenv`: +Firstly you should create the folder and virtual environment that are going to contain your new project. Here we will use the recommended `pyenv`: ```bash mkdir pyo3-example @@ -61,19 +59,19 @@ cd pyo3-example pyenv virtualenv pyo3 pyenv local pyo3 ``` -after this, you should install your build manager. In this example, we will use `maturin`. After you've activated your virtualenv add `maturin` to it: +after this, you should install your build manager. In this example, we will use `maturin`. After you've activated your virtualenv add `maturin` to it: ```bash pip install maturin ``` -After this, you can initialise the new project +After this, you can initialise the new project ```bash maturin init ``` -If `maturin` is already installed you can create a new project using that directly as well: +If `maturin` is already installed you can create a new project using that directly as well: ```bash maturin new -b pyo3 pyo3-example @@ -84,16 +82,16 @@ pyenv local pyo3 # Adding to an existing project -Sadly currently `maturin` cannot be run in existing projects, so if you want to use python in an existing project you basically have two options: +Sadly currently `maturin` cannot be run in existing projects, so if you want to use python in an existing project you basically have two options: 1. create a new project as above and move your existing code into that project -2. Manually edit your project configuration as necessary. +2. Manually edit your project configuration as necessary. -If you are opting for the second option, here are the things you need to pay attention to: +If you are opting for the second option, here are the things you need to pay attention to: ## Cargo.toml -Make sure that the rust you want to be able to access from Python is compiled into a library. You can have a binary output as well, but the code you want to access from python has to be in the library. Also, make sure that the crate type is `cdylib` and add PyO3 as a dependency as so: +Make sure that the rust you want to be able to access from Python is compiled into a library. You can have a binary output as well, but the code you want to access from python has to be in the library. Also, make sure that the crate type is `cdylib` and add PyO3 as a dependency as so: ```toml @@ -111,7 +109,8 @@ pyo3 = { version = "0.16.5", features = ["extension-module"] } ``` ## pyproject.toml -You should also create a `pyproject.toml` with the following contents: + +You should also create a `pyproject.toml` with the following contents: ```toml [build-system] @@ -130,7 +129,7 @@ classifiers = [ ## Running code -After this you can setup rust code to be available in python as such: +After this you can setup rust code to be available in python as such: ```rust use pyo3::prelude::*; @@ -151,7 +150,7 @@ fn string_sum(_py: Python<'_>, m: &PyModule) -> PyResult<()> { } ``` -After this you can run `maturin develop` to prepare the python package after which you can use it like so: +After this you can run `maturin develop` to prepare the python package after which you can use it like so: ```bash $ maturin develop @@ -162,4 +161,4 @@ $ python '25' ``` -For more instructions on how to use python code from rust see the [Python from Rust](python_from_rust.md) page. \ No newline at end of file +For more instructions on how to use python code from rust see the [Python from Rust](python_from_rust.md) page. diff --git a/guide/src/memory.md b/guide/src/memory.md index 5b43dd78..aca624ab 100644 --- a/guide/src/memory.md +++ b/guide/src/memory.md @@ -1,4 +1,4 @@ -# Memory Management +# Memory management Rust and Python have very different notions of memory management. Rust has a strict memory model with concepts of ownership, borrowing, and lifetimes, @@ -13,7 +13,7 @@ PyO3 bridges the Rust and Python memory models with two different strategies for accessing memory allocated on Python's heap from inside Rust. These are GIL-bound, or "owned" references, and GIL-independent `Py` smart pointers. -## GIL-bound Memory +## GIL-bound memory PyO3's GIL-bound, "owned references" (`&PyAny` etc.) make PyO3 more ergonomic to use by ensuring that their lifetime can never be longer than the duration the @@ -119,7 +119,7 @@ dropped you do not retain access to any owned references created after the [documentation for `Python::new_pool()`]({{#PYO3_DOCS_URL}}/pyo3/prelude/struct.Python.html#method.new_pool) for more information on safety. -## GIL-independent Memory +## GIL-independent memory Sometimes we need a reference to memory on Python's heap that can outlive the GIL. Python's `Py` is analogous to `Rc`, but for variables whose diff --git a/guide/src/module.md b/guide/src/module.md index 4146fb18..9e52ab93 100644 --- a/guide/src/module.md +++ b/guide/src/module.md @@ -1,4 +1,4 @@ -# Python Modules +# Python modules You can create a module using `#[pymodule]`: @@ -64,7 +64,7 @@ print(my_extension.__doc__) ## Python submodules -You can create a module hierarchy within a single extension module by using +You can create a module hierarchy within a single extension module by using [`PyModule.add_submodule()`]({{#PYO3_DOCS_URL}}/pyo3/prelude/struct.PyModule.html#method.add_submodule). For example, you could define the modules `parent_module` and `parent_module.child_module`. @@ -99,9 +99,9 @@ fn func() -> String { # }) ``` -Note that this does not define a package, so this won’t allow Python code to directly import -submodules by using `from parent_module import child_module`. For more information, see -[#759](https://github.com/PyO3/pyo3/issues/759) and +Note that this does not define a package, so this won’t allow Python code to directly import +submodules by using `from parent_module import child_module`. For more information, see +[#759](https://github.com/PyO3/pyo3/issues/759) and [#1517](https://github.com/PyO3/pyo3/issues/1517#issuecomment-808664021). It is not necessary to add `#[pymodule]` on nested modules, which is only required on the top-level module.