From 81afdaf6cda2b73eb9c01fb3052e319ad49f594c Mon Sep 17 00:00:00 2001 From: David Hewitt <1939362+davidhewitt@users.noreply.github.com> Date: Tue, 1 Sep 2020 22:40:41 +0100 Subject: [PATCH] Start splitting long guide pages into subchapters --- guide/src/SUMMARY.md | 3 + guide/src/class.md | 305 +---------------------- guide/src/class/protocols.md | 303 +++++++++++++++++++++++ guide/src/conversions.md | 421 -------------------------------- guide/src/conversions/tables.md | 96 ++++++++ guide/src/conversions/traits.md | 324 ++++++++++++++++++++++++ guide/src/index.md | 2 +- guide/src/migration.md | 2 +- guide/src/trait_bounds.md | 2 +- 9 files changed, 730 insertions(+), 728 deletions(-) create mode 100644 guide/src/class/protocols.md create mode 100644 guide/src/conversions/tables.md create mode 100644 guide/src/conversions/traits.md diff --git a/guide/src/SUMMARY.md b/guide/src/SUMMARY.md index 2b7570fa..d400bf96 100644 --- a/guide/src/SUMMARY.md +++ b/guide/src/SUMMARY.md @@ -7,7 +7,10 @@ - [Python Modules](module.md) - [Python Functions](function.md) - [Python Classes](class.md) + - [Class customizations](class/protocols.md) - [Type Conversions](conversions.md) + - [Mapping of Rust types to Python types](conversions/tables.md)] + - [Conversion traits](conversions/traits.md)] - [Python Exceptions](exception.md) - [Calling Python from Rust](python_from_rust.md) - [GIL, mutability and object types](types.md) diff --git a/guide/src/class.md b/guide/src/class.md index 4ee46687..3e8d8c82 100644 --- a/guide/src/class.md +++ b/guide/src/class.md @@ -15,7 +15,7 @@ For ease of discovery, below is a list of all custom attributes with links to th - [`#[call]`](#callable-objects) - [`#[classattr]`](#class-attributes) - [`#[args]`](#method-arguments) -- [`#[pyproto]`](#class-customizations) +- [`#[pyproto]`](class/protocols.html) ## Defining a new class @@ -695,307 +695,6 @@ num=44, debug=false num=-1, debug=false ``` -## Class customizations - -Python's object model defines several protocols for different object behavior, like sequence, -mapping or number protocols. PyO3 defines separate traits for each of them. To provide specific -Python object behavior, you need to implement the specific trait for your struct. Important note, -each protocol implementation block has to be annotated with the `#[pyproto]` attribute. - -All `#[pyproto]` methods which can be defined below can return `T` instead of `PyResult` if the -method implementation is infallible. In addition, if the return type is `()`, it can be omitted altogether. - -### Basic object customization - -The [`PyObjectProtocol`] trait provides several basic customizations. - -#### Attribute access - -To customize object attribute access, define the following methods: - - * `fn __getattr__(&self, name: FromPyObject) -> PyResult>` - * `fn __setattr__(&mut self, name: FromPyObject, value: FromPyObject) -> PyResult<()>` - * `fn __delattr__(&mut self, name: FromPyObject) -> PyResult<()>` - -Each method corresponds to Python's `self.attr`, `self.attr = value` and `del self.attr` code. - -#### String Conversions - - * `fn __repr__(&self) -> PyResult>` - * `fn __str__(&self) -> PyResult>` - - Possible return types for `__str__` and `__repr__` are `PyResult` or `PyResult`. - - * `fn __bytes__(&self) -> PyResult` - - Provides the conversion to `bytes`. - - * `fn __format__(&self, format_spec: &str) -> PyResult>` - - Special method that is used by the `format()` builtin and the `str.format()` method. - Possible return types are `PyResult` or `PyResult`. - -#### Comparison operators - - * `fn __richcmp__(&self, other: impl FromPyObject, op: CompareOp) -> PyResult` - - Overloads Python comparison operations (`==`, `!=`, `<`, `<=`, `>`, and `>=`). - The `op` argument indicates the comparison operation being performed. - The return type will normally be `PyResult`, but any Python object can be returned. - If `other` is not of the type specified in the signature, the generated code will - automatically `return NotImplemented`. - - * `fn __hash__(&self) -> PyResult` - - Objects that compare equal must have the same hash value. - The return type must be `PyResult` where `T` is one of Rust's primitive integer types. - -#### Other methods - - * `fn __bool__(&self) -> PyResult` - - Determines the "truthyness" of the object. - -### Emulating numeric types - -The [`PyNumberProtocol`] trait allows [emulate numeric types](https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types). - - * `fn __add__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __sub__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __mul__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __matmul__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __truediv__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __floordiv__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __mod__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __divmod__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __pow__(lhs: impl FromPyObject, rhs: impl FromPyObject, modulo: Option) -> PyResult` - * `fn __lshift__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rshift__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __and__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __or__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __xor__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - -These methods are called to implement the binary arithmetic operations -(`+`, `-`, `*`, `@`, `/`, `//`, `%`, `divmod()`, `pow()` and `**`, `<<`, `>>`, `&`, `^`, and `|`). - -If `rhs` is not of the type specified in the signature, the generated code -will automatically `return NotImplemented`. This is not the case for `lhs` -which must match signature or else raise a TypeError. - - -The reflected operations are also available: - - * `fn __radd__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rsub__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rmul__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rmatmul__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rtruediv__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rfloordiv__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rmod__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rdivmod__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rpow__(lhs: impl FromPyObject, rhs: impl FromPyObject, modulo: Option) -> PyResult` - * `fn __rlshift__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rrshift__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rand__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __ror__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - * `fn __rxor__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` - -The code generated for these methods expect that all arguments match the -signature, or raise a TypeError. - -*Note*: Currently implementing the method for a binary arithmetic operations -(e.g, `__add__`) shadows the reflected operation (e.g, `__radd__`). This is -being addressed in [#844](https://github.com/PyO3/pyo3/issues/844). to make -these methods - - -This trait also has support the augmented arithmetic assignments (`+=`, `-=`, -`*=`, `@=`, `/=`, `//=`, `%=`, `**=`, `<<=`, `>>=`, `&=`, `^=`, `|=`): - - * `fn __iadd__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __isub__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __imul__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __imatmul__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __itruediv__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __ifloordiv__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __imod__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __ipow__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __ilshift__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __irshift__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __iand__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __ior__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - * `fn __ixor__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` - -The following methods implement the unary arithmetic operations (`-`, `+`, `abs()` and `~`): - - * `fn __neg__(&'p self) -> PyResult` - * `fn __pos__(&'p self) -> PyResult` - * `fn __abs__(&'p self) -> PyResult` - * `fn __invert__(&'p self) -> PyResult` - -Support for coercions: - - * `fn __complex__(&'p self) -> PyResult` - * `fn __int__(&'p self) -> PyResult` - * `fn __float__(&'p self) -> PyResult` - -Other: - - * `fn __index__(&'p self) -> PyResult` - * `fn __round__(&'p self, ndigits: Option) -> PyResult` - -### Garbage Collector Integration - -If your type owns references to other Python objects, you will need to -integrate with Python's garbage collector so that the GC is aware of -those references. -To do this, implement the [`PyGCProtocol`] trait for your struct. -It includes two methods `__traverse__` and `__clear__`. -These correspond to the slots `tp_traverse` and `tp_clear` in the Python C API. -`__traverse__` must call `visit.call()` for each reference to another Python object. -`__clear__` must clear out any mutable references to other Python objects -(thus breaking reference cycles). Immutable references do not have to be cleared, -as every cycle must contain at least one mutable reference. -Example: -```rust -extern crate pyo3; - -use pyo3::prelude::*; -use pyo3::PyTraverseError; -use pyo3::gc::{PyGCProtocol, PyVisit}; - -#[pyclass] -struct ClassWithGCSupport { - obj: Option, -} - -#[pyproto] -impl PyGCProtocol for ClassWithGCSupport { - fn __traverse__(&self, visit: PyVisit) -> Result<(), PyTraverseError> { - if let Some(ref obj) = self.obj { - visit.call(obj)? - } - Ok(()) - } - - fn __clear__(&mut self) { - // Clear reference, this decrements ref counter. - self.obj = None; - } -} -``` - -Special protocol trait implementations have to be annotated with the `#[pyproto]` attribute. - -It is also possible to enable GC for custom classes using the `gc` parameter of the `pyclass` attribute. -i.e. `#[pyclass(gc)]`. In that case instances of custom class participate in Python garbage -collection, and it is possible to track them with `gc` module methods. When using the `gc` parameter, -it is *required* to implement the `PyGCProtocol` trait, failure to do so will result in an error -at compile time: - -```compile_fail -#[pyclass(gc)] -struct GCTracked {} // Fails because it does not implement PyGCProtocol -``` - -### Iterator Types - -Iterators can be defined using the -[`PyIterProtocol`](https://docs.rs/pyo3/latest/pyo3/class/iter/trait.PyIterProtocol.html) trait. -It includes two methods `__iter__` and `__next__`: - * `fn __iter__(slf: PyRefMut) -> PyResult>` - * `fn __next__(slf: PyRefMut) -> PyResult>>` - -Returning `None` from `__next__` indicates that that there are no further items. -These two methods can be take either `PyRef` or `PyRefMut` as their -first argument, so that mutable borrow can be avoided if needed. - -Example: - -```rust -use pyo3::prelude::*; -use pyo3::PyIterProtocol; - -#[pyclass] -struct MyIterator { - iter: Box + Send>, -} - -#[pyproto] -impl PyIterProtocol for MyIterator { - fn __iter__(slf: PyRef) -> PyRef { - slf - } - fn __next__(mut slf: PyRefMut) -> Option { - slf.iter.next() - } -} -``` - -In many cases you'll have a distinction between the type being iterated over (i.e. the *iterable*) and the iterator it -provides. In this case, you should implement `PyIterProtocol` for both the iterable and the iterator, but the iterable -only needs to support `__iter__()` while the iterator must support both `__iter__()` and `__next__()`. The default -implementations in `PyIterProtocol` will ensure that the objects behave correctly in Python. For example: - -```rust -# use pyo3::prelude::*; -# use pyo3::PyIterProtocol; - -#[pyclass] -struct Iter { - inner: std::vec::IntoIter, -} - -#[pyproto] -impl PyIterProtocol for Iter { - fn __iter__(slf: PyRef) -> PyRef { - slf - } - - fn __next__(mut slf: PyRefMut) -> Option { - slf.inner.next() - } -} - -#[pyclass] -struct Container { - iter: Vec, -} - -#[pyproto] -impl PyIterProtocol for Container { - fn __iter__(slf: PyRef) -> PyResult> { - let iter = Iter { - inner: slf.iter.clone().into_iter(), - }; - Py::new(slf.py(), iter) - } -} - -# let gil = Python::acquire_gil(); -# let py = gil.python(); -# let inst = pyo3::PyCell::new( -# py, -# Container { -# iter: vec![1, 2, 3, 4], -# }, -# ) -# .unwrap(); -# pyo3::py_run!(py, inst, "assert list(inst) == [1, 2, 3, 4]"); -# pyo3::py_run!(py, inst, "assert list(iter(iter(inst))) == [1, 2, 3, 4]"); -``` - -For more details on Python's iteration protocols, check out [the "Iterator Types" section of the library -documentation](https://docs.python.org/3/library/stdtypes.html#iterator-types). - -#### Returning a value from iteration - -This guide has so far shown how to use `Option` to implement yielding values during iteration. -In Python a generator can also return a value. To express this in Rust, PyO3 provides the -[`IterNextOutput`](https://docs.rs/pyo3/latest/pyo3/class/iter/enum.IterNextOutput.html) enum to -both `Yield` values and `Return` a final value - see its docs for further details and an example. - - ## How methods are implemented Users should be able to define a `#[pyclass]` with or without `#[pymethods]`, while PyO3 needs a @@ -1089,8 +788,6 @@ impl pyo3::pyclass::PyClassSend for MyClass { [`GILGuard`]: https://docs.rs/pyo3/latest/pyo3/struct.GILGuard.html -[`PyGCProtocol`]: https://docs.rs/pyo3/latest/pyo3/class/gc/trait.PyGCProtocol.html -[`PyObjectProtocol`]: https://docs.rs/pyo3/latest/pyo3/class/basic/trait.PyObjectProtocol.html [`PyTypeInfo`]: https://docs.rs/pyo3/latest/pyo3/type_object/trait.PyTypeInfo.html [`PyTypeObject`]: https://docs.rs/pyo3/latest/pyo3/type_object/trait.PyTypeObject.html diff --git a/guide/src/class/protocols.md b/guide/src/class/protocols.md new file mode 100644 index 00000000..efefa48d --- /dev/null +++ b/guide/src/class/protocols.md @@ -0,0 +1,303 @@ +## Class customizations + +Python's object model defines several protocols for different object behavior, like sequence, +mapping or number protocols. PyO3 defines separate traits for each of them. To provide specific +Python object behavior, you need to implement the specific trait for your struct. Important note, +each protocol implementation block has to be annotated with the `#[pyproto]` attribute. + +All `#[pyproto]` methods which can be defined below can return `T` instead of `PyResult` if the +method implementation is infallible. In addition, if the return type is `()`, it can be omitted altogether. + +### Basic object customization + +The [`PyObjectProtocol`] trait provides several basic customizations. + +#### Attribute access + +To customize object attribute access, define the following methods: + + * `fn __getattr__(&self, name: FromPyObject) -> PyResult>` + * `fn __setattr__(&mut self, name: FromPyObject, value: FromPyObject) -> PyResult<()>` + * `fn __delattr__(&mut self, name: FromPyObject) -> PyResult<()>` + +Each method corresponds to Python's `self.attr`, `self.attr = value` and `del self.attr` code. + +#### String Conversions + + * `fn __repr__(&self) -> PyResult>` + * `fn __str__(&self) -> PyResult>` + + Possible return types for `__str__` and `__repr__` are `PyResult` or `PyResult`. + + * `fn __bytes__(&self) -> PyResult` + + Provides the conversion to `bytes`. + + * `fn __format__(&self, format_spec: &str) -> PyResult>` + + Special method that is used by the `format()` builtin and the `str.format()` method. + Possible return types are `PyResult` or `PyResult`. + +#### Comparison operators + + * `fn __richcmp__(&self, other: impl FromPyObject, op: CompareOp) -> PyResult` + + Overloads Python comparison operations (`==`, `!=`, `<`, `<=`, `>`, and `>=`). + The `op` argument indicates the comparison operation being performed. + The return type will normally be `PyResult`, but any Python object can be returned. + If `other` is not of the type specified in the signature, the generated code will + automatically `return NotImplemented`. + + * `fn __hash__(&self) -> PyResult` + + Objects that compare equal must have the same hash value. + The return type must be `PyResult` where `T` is one of Rust's primitive integer types. + +#### Other methods + + * `fn __bool__(&self) -> PyResult` + + Determines the "truthyness" of the object. + +### Emulating numeric types + +The [`PyNumberProtocol`] trait allows [emulate numeric types](https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types). + + * `fn __add__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __sub__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __mul__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __matmul__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __truediv__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __floordiv__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __mod__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __divmod__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __pow__(lhs: impl FromPyObject, rhs: impl FromPyObject, modulo: Option) -> PyResult` + * `fn __lshift__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rshift__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __and__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __or__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __xor__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + +These methods are called to implement the binary arithmetic operations +(`+`, `-`, `*`, `@`, `/`, `//`, `%`, `divmod()`, `pow()` and `**`, `<<`, `>>`, `&`, `^`, and `|`). + +If `rhs` is not of the type specified in the signature, the generated code +will automatically `return NotImplemented`. This is not the case for `lhs` +which must match signature or else raise a TypeError. + + +The reflected operations are also available: + + * `fn __radd__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rsub__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rmul__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rmatmul__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rtruediv__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rfloordiv__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rmod__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rdivmod__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rpow__(lhs: impl FromPyObject, rhs: impl FromPyObject, modulo: Option) -> PyResult` + * `fn __rlshift__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rrshift__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rand__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __ror__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + * `fn __rxor__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult` + +The code generated for these methods expect that all arguments match the +signature, or raise a TypeError. + +*Note*: Currently implementing the method for a binary arithmetic operations +(e.g, `__add__`) shadows the reflected operation (e.g, `__radd__`). This is +being addressed in [#844](https://github.com/PyO3/pyo3/issues/844). to make +these methods + + +This trait also has support the augmented arithmetic assignments (`+=`, `-=`, +`*=`, `@=`, `/=`, `//=`, `%=`, `**=`, `<<=`, `>>=`, `&=`, `^=`, `|=`): + + * `fn __iadd__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __isub__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __imul__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __imatmul__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __itruediv__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __ifloordiv__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __imod__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __ipow__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __ilshift__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __irshift__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __iand__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __ior__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + * `fn __ixor__(&'p mut self, other: impl FromPyObject) -> PyResult<()>` + +The following methods implement the unary arithmetic operations (`-`, `+`, `abs()` and `~`): + + * `fn __neg__(&'p self) -> PyResult` + * `fn __pos__(&'p self) -> PyResult` + * `fn __abs__(&'p self) -> PyResult` + * `fn __invert__(&'p self) -> PyResult` + +Support for coercions: + + * `fn __complex__(&'p self) -> PyResult` + * `fn __int__(&'p self) -> PyResult` + * `fn __float__(&'p self) -> PyResult` + +Other: + + * `fn __index__(&'p self) -> PyResult` + * `fn __round__(&'p self, ndigits: Option) -> PyResult` + +### Garbage Collector Integration + +If your type owns references to other Python objects, you will need to +integrate with Python's garbage collector so that the GC is aware of +those references. +To do this, implement the [`PyGCProtocol`] trait for your struct. +It includes two methods `__traverse__` and `__clear__`. +These correspond to the slots `tp_traverse` and `tp_clear` in the Python C API. +`__traverse__` must call `visit.call()` for each reference to another Python object. +`__clear__` must clear out any mutable references to other Python objects +(thus breaking reference cycles). Immutable references do not have to be cleared, +as every cycle must contain at least one mutable reference. +Example: +```rust +extern crate pyo3; + +use pyo3::prelude::*; +use pyo3::PyTraverseError; +use pyo3::gc::{PyGCProtocol, PyVisit}; + +#[pyclass] +struct ClassWithGCSupport { + obj: Option, +} + +#[pyproto] +impl PyGCProtocol for ClassWithGCSupport { + fn __traverse__(&self, visit: PyVisit) -> Result<(), PyTraverseError> { + if let Some(ref obj) = self.obj { + visit.call(obj)? + } + Ok(()) + } + + fn __clear__(&mut self) { + // Clear reference, this decrements ref counter. + self.obj = None; + } +} +``` + +Special protocol trait implementations have to be annotated with the `#[pyproto]` attribute. + +It is also possible to enable GC for custom classes using the `gc` parameter of the `pyclass` attribute. +i.e. `#[pyclass(gc)]`. In that case instances of custom class participate in Python garbage +collection, and it is possible to track them with `gc` module methods. When using the `gc` parameter, +it is *required* to implement the `PyGCProtocol` trait, failure to do so will result in an error +at compile time: + +```compile_fail +#[pyclass(gc)] +struct GCTracked {} // Fails because it does not implement PyGCProtocol +``` + +### Iterator Types + +Iterators can be defined using the +[`PyIterProtocol`](https://docs.rs/pyo3/latest/pyo3/class/iter/trait.PyIterProtocol.html) trait. +It includes two methods `__iter__` and `__next__`: + * `fn __iter__(slf: PyRefMut) -> PyResult>` + * `fn __next__(slf: PyRefMut) -> PyResult>>` + +Returning `None` from `__next__` indicates that that there are no further items. +These two methods can be take either `PyRef` or `PyRefMut` as their +first argument, so that mutable borrow can be avoided if needed. + +Example: + +```rust +use pyo3::prelude::*; +use pyo3::PyIterProtocol; + +#[pyclass] +struct MyIterator { + iter: Box + Send>, +} + +#[pyproto] +impl PyIterProtocol for MyIterator { + fn __iter__(slf: PyRef) -> PyRef { + slf + } + fn __next__(mut slf: PyRefMut) -> Option { + slf.iter.next() + } +} +``` + +In many cases you'll have a distinction between the type being iterated over (i.e. the *iterable*) and the iterator it +provides. In this case, you should implement `PyIterProtocol` for both the iterable and the iterator, but the iterable +only needs to support `__iter__()` while the iterator must support both `__iter__()` and `__next__()`. The default +implementations in `PyIterProtocol` will ensure that the objects behave correctly in Python. For example: + +```rust +# use pyo3::prelude::*; +# use pyo3::PyIterProtocol; + +#[pyclass] +struct Iter { + inner: std::vec::IntoIter, +} + +#[pyproto] +impl PyIterProtocol for Iter { + fn __iter__(slf: PyRef) -> PyRef { + slf + } + + fn __next__(mut slf: PyRefMut) -> Option { + slf.inner.next() + } +} + +#[pyclass] +struct Container { + iter: Vec, +} + +#[pyproto] +impl PyIterProtocol for Container { + fn __iter__(slf: PyRef) -> PyResult> { + let iter = Iter { + inner: slf.iter.clone().into_iter(), + }; + Py::new(slf.py(), iter) + } +} + +# let gil = Python::acquire_gil(); +# let py = gil.python(); +# let inst = pyo3::PyCell::new( +# py, +# Container { +# iter: vec![1, 2, 3, 4], +# }, +# ) +# .unwrap(); +# pyo3::py_run!(py, inst, "assert list(inst) == [1, 2, 3, 4]"); +# pyo3::py_run!(py, inst, "assert list(iter(iter(inst))) == [1, 2, 3, 4]"); +``` + +For more details on Python's iteration protocols, check out [the "Iterator Types" section of the library +documentation](https://docs.python.org/3/library/stdtypes.html#iterator-types). + +#### Returning a value from iteration + +This guide has so far shown how to use `Option` to implement yielding values during iteration. +In Python a generator can also return a value. To express this in Rust, PyO3 provides the +[`IterNextOutput`](https://docs.rs/pyo3/latest/pyo3/class/iter/enum.IterNextOutput.html) enum to +both `Yield` values and `Return` a final value - see its docs for further details and an example. + +[`PyGCProtocol`]: https://docs.rs/pyo3/latest/pyo3/class/gc/trait.PyGCProtocol.html +[`PyGCProtocol`]: https://docs.rs/pyo3/latest/pyo3/class/number/trait.PyNumberProtocol.html +[`PyObjectProtocol`]: https://docs.rs/pyo3/latest/pyo3/class/basic/trait.PyObjectProtocol.html diff --git a/guide/src/conversions.md b/guide/src/conversions.md index 29d90f93..f535e083 100644 --- a/guide/src/conversions.md +++ b/guide/src/conversions.md @@ -1,424 +1,3 @@ # Type Conversions In this portion of the guide we'll talk about the mapping of Python types to Rust types offered by PyO3, as well as the traits available to perform conversions between them. - -## Mapping of Rust types to Python types - -When writing functions callable from Python (such as a `#[pyfunction]` or in a `#[pymethods]` block), the trait `FromPyObject` is required for function arguments, and `IntoPy` is required for function return values. - -Consult the tables in the following section to find the Rust types provided by PyO3 which implement these traits. - -### Argument Types - -When accepting a function argument, it is possible to either use Rust library types or PyO3's Python-native types. (See the next section for discussion on when to use each.) - -The table below contains the Python type and the corresponding function argument types that will accept them: - -| Python | Rust | Rust (Python-native) | -| ------------- |:-------------------------------:|:--------------------:| -| `object` | - | `&PyAny` | -| `str` | `String`, `Cow`, `&str` | `&PyUnicode` | -| `bytes` | `Vec`, `&[u8]` | `&PyBytes` | -| `bool` | `bool` | `&PyBool` | -| `int` | Any integer type (`i32`, `u32`, `usize`, etc) | `&PyLong` | -| `float` | `f32`, `f64` | `&PyFloat` | -| `complex` | `num_complex::Complex`[^1] | `&PyComplex` | -| `list[T]` | `Vec` | `&PyList` | -| `dict[K, V]` | `HashMap`, `BTreeMap`, `hashbrown::HashMap`[^2] | `&PyDict` | -| `tuple[T, U]` | `(T, U)`, `Vec` | `&PyTuple` | -| `set[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^2] | `&PySet` | -| `frozenset[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^2] | `&PyFrozenSet` | -| `bytearray` | `Vec` | `&PyByteArray` | -| `slice` | - | `&PySlice` | -| `type` | - | `&PyType` | -| `module` | - | `&PyModule` | -| `datetime.datetime` | - | `&PyDateTime` | -| `datetime.date` | - | `&PyDate` | -| `datetime.time` | - | `&PyTime` | -| `datetime.tzinfo` | - | `&PyTzInfo` | -| `datetime.timedelta` | - | `&PyDelta` | -| `typing.Optional[T]` | `Option` | - | -| `typing.Sequence[T]` | `Vec` | `&PySequence` | -| `typing.Iterator[Any]` | - | `&PyIterator` | -| `typing.Union[...]` | See [`#[derive(FromPyObject)]`](#deriving-a-hrefhttpsdocsrspyo3latestpyo3conversiontraitfrompyobjecthtmlfrompyobjecta-for-enums) | - | - -There are also a few special types related to the GIL and Rust-defined `#[pyclass]`es which may come in useful: - -| What | Description | -| ------------- | ------------------------------------- | -| `Python` | A GIL token, used to pass to PyO3 constructors to prove ownership of the GIL | -| `Py` | A Python object isolated from the GIL lifetime. This can be sent to other threads. | -| `PyObject` | An alias for `Py` | -| `&PyCell` | A `#[pyclass]` value owned by Python. | -| `PyRef` | A `#[pyclass]` borrowed immutably. | -| `PyRefMut` | A `#[pyclass]` borrowed mutably. | - -For more detail on accepting `#[pyclass]` values as function arguments, see [the section of this guide on Python Classes](class.md). - -#### Using Rust library types vs Python-native types - -Using Rust library types as function arguments will incur a conversion cost compared to using the Python-native types. Using the Python-native types is almost zero-cost (they just require a type check similar to the Python builtin function `isinstance()`). - -However, once that conversion cost has been paid, the Rust standard library types offer a number of benefits: -- You can write functionality in native-speed Rust code (free of Python's runtime costs). -- You get better interoperability with the rest of the Rust ecosystem. -- You can use `Python::allow_threads` to release the Python GIL and let other Python threads make progress while your Rust code is executing. -- You also benefit from stricter type checking. For example you can specify `Vec`, which will only accept a Python `list` containing integers. The Python-native equivalent, `&PyList`, would accept a Python `list` containing Python objects of any type. - -For most PyO3 usage the conversion cost is worth paying to get these benefits. As always, if you're not sure it's worth it in your case, benchmark it! - -### Returning Rust values to Python - -When returning values from functions callable from Python, Python-native types (`&PyAny`, `&PyDict` etc.) can be used with zero cost. - -Because these types are references, in some situations the Rust compiler may ask for lifetime annotations. If this is the case, you should use `Py`, `Py` etc. instead - which are also zero-cost. For all of these Python-native types `T`, `Py` can be created from `T` with an `.into()` conversion. - -If your function is fallible, it should return `PyResult` or `Result` where `E` implements `From for PyErr`. This will raise a `Python` exception if the `Err` variant is returned. - -Finally, the following Rust types are also able to convert to Python as return values: - -| Rust type | Resulting Python Type | -| ------------- |:-------------------------------:| -| `String` | `str` | -| `&str` | `str` | -| `bool` | `bool` | -| Any integer type (`i32`, `u32`, `usize`, etc) | `int` | -| `f32`, `f64` | `float` | -| `Option` | `Optional[T]` | -| `(T, U)` | `Tuple[T, U]` | -| `Vec` | `List[T]` | -| `HashMap` | `Dict[K, V]` | -| `BTreeMap` | `Dict[K, V]` | -| `HashSet` | `Set[T]` | -| `BTreeSet` | `Set[T]` | -| `&PyCell` | `T` | -| `PyRef` | `T` | -| `PyRefMut` | `T` | - -## Traits - -PyO3 provides some handy traits to convert between Python types and Rust types. - -### `.extract()` and the `FromPyObject` trait - -The easiest way to convert a Python object to a Rust value is using -`.extract()`. It returns a `PyResult` with a type error if the conversion -fails, so usually you will use something like - -```ignore -let v: Vec = obj.extract()?; -``` - -This method is available for many Python object types, and can produce a wide -variety of Rust types, which you can check out in the implementor list of -[`FromPyObject`]. - -[`FromPyObject`] is also implemented for your own Rust types wrapped as Python -objects (see [the chapter about classes](class.md)). There, in order to both be -able to operate on mutable references *and* satisfy Rust's rules of non-aliasing -mutable references, you have to extract the PyO3 reference wrappers [`PyRef`] -and [`PyRefMut`]. They work like the reference wrappers of -`std::cell::RefCell` and ensure (at runtime) that Rust borrows are allowed. - -#### Deriving [`FromPyObject`] - -[`FromPyObject`] can be automatically derived for many kinds of structs and enums -if the member types themselves implement `FromPyObject`. This even includes members -with a generic type `T: FromPyObject`. Derivation for empty enums, enum variants and -structs is not supported. - -#### Deriving [`FromPyObject`] for structs - -``` -use pyo3::prelude::*; - -#[derive(FromPyObject)] -struct RustyStruct { - my_string: String, -} -``` - -The derivation generates code that will per default access the attribute `my_string` on -the Python object, i.e. `obj.getattr("my_string")`, and call `extract()` on the attribute. -It is also possible to access the value on the Python object through `obj.get_item("my_string")` -by setting the attribute `pyo3(item)` on the field: -``` -use pyo3::prelude::*; - -#[derive(FromPyObject)] -struct RustyStruct { - #[pyo3(item)] - my_string: String, -} -``` - -The argument passed to `getattr` and `get_item` can also be configured: - -``` -use pyo3::prelude::*; - -#[derive(FromPyObject)] -struct RustyStruct { - #[pyo3(item("key"))] - string_in_mapping: String, - #[pyo3(attribute("name"))] - string_attr: String, -} -``` - -This tries to extract `string_attr` from the attribute `name` and `string_in_mapping` -from a mapping with the key `"key"`. The arguments for `attribute` are restricted to -non-empty string literals while `item` can take any valid literal that implements -`ToBorrowedObject`. - -#### Deriving [`FromPyObject`] for tuple structs - -Tuple structs are also supported but do not allow customizing the extraction. The input is -always assumed to be a Python tuple with the same length as the Rust type, the `n`th field -is extracted from the `n`th item in the Python tuple. - -``` -use pyo3::prelude::*; - -#[derive(FromPyObject)] -struct RustyTuple(String, String); -``` - -Tuple structs with a single field are treated as wrapper types which are described in the -following section. To override this behaviour and ensure that the input is in fact a tuple, -specify the struct as -``` -use pyo3::prelude::*; - -#[derive(FromPyObject)] -struct RustyTuple((String,)); -``` - -#### Deriving [`FromPyObject`] for wrapper types - -The `pyo3(transparent)` attribute can be used on structs with exactly one field. This results -in extracting directly from the input object, i.e. `obj.extract()`, rather than trying to access -an item or attribute. This behaviour is enabled per default for newtype structs and tuple-variants -with a single field. - -``` -use pyo3::prelude::*; - -#[derive(FromPyObject)] -struct RustyTransparentTupleStruct(String); - -#[derive(FromPyObject)] -#[pyo3(transparent)] -struct RustyTransparentStruct { - inner: String, -} -``` - -#### Deriving [`FromPyObject`] for enums - -The `FromPyObject` derivation for enums generates code that tries to extract the variants in the -order of the fields. As soon as a variant can be extracted succesfully, that variant is returned. -This makes it possible to extract Python types like `Union[str, int]`. - -The same customizations and restrictions described for struct derivations apply to enum variants, -i.e. a tuple variant assumes that the input is a Python tuple, and a struct variant defaults to -extracting fields as attributes but can be configured in the same manner. The `transparent` -attribute can be applied to single-field-variants. - -``` -use pyo3::prelude::*; - -#[derive(FromPyObject)] -enum RustyEnum<'a> { - Int(usize), // input is a positive int - String(String), // input is a string - IntTuple(usize, usize), // input is a 2-tuple with positive ints - StringIntTuple(String, usize), // input is a 2-tuple with String and int - Coordinates3d { // needs to be in front of 2d - x: usize, - y: usize, - z: usize, - }, - Coordinates2d { // only gets checked if the input did not have `z` - #[pyo3(attribute("x"))] - a: usize, - #[pyo3(attribute("y"))] - b: usize, - }, - #[pyo3(transparent)] - CatchAll(&'a PyAny), // This extraction never fails -} -``` - -If none of the enum variants match, a `PyValueError` containing the names of the -tested variants is returned. The names reported in the error message can be customized -through the `pyo3(annotation = "name")` attribute, e.g. to use conventional Python type -names: - -``` -use pyo3::prelude::*; - -#[derive(FromPyObject)] -enum RustyEnum { - #[pyo3(transparent, annotation = "str")] - String(String), - #[pyo3(transparent, annotation = "int")] - Int(isize), -} -``` - -If the input is neither a string nor an integer, the error message will be: -`"Can't convert to Union[str, int]"`, where `` is replaced by the type name and -`repr()` of the input object. - -#### `#[derive(FromPyObject)]` Container Attributes -- `pyo3(transparent)` - - extract the field directly from the object as `obj.extract()` instead of `get_item()` or - `getattr()` - - Newtype structs and tuple-variants are treated as transparent per default. - - only supported for single-field structs and enum variants -- `pyo3(annotation = "name")` - - changes the name of the failed variant in the generated error message in case of failure. - - e.g. `pyo3("int")` reports the variant's type as `int`. - - only supported for enum variants - -#### `#[derive(FromPyObject)]` Field Attributes -- `pyo3(attribute)`, `pyo3(attribute("name"))` - - retrieve the field from an attribute, possibly with a custom name specified as an argument - - argument must be a string-literal. -- `pyo3(item)`, `pyo3(item("key"))` - - retrieve the field from a mapping, possibly with the custom key specified as an argument. - - can be any literal that implements `ToBorrowedObject` - -### `IntoPy` - -This trait defines the to-python conversion for a Rust type. It is usually implemented as -`IntoPy`, which is the trait needed for returning a value from `#[pyfunction]` and -`#[pymethods]`. - -All types in PyO3 implement this trait, as does a `#[pyclass]` which doesn't use `extends`. - -Occasionally you may choose to implement this for custom types which are mapped to Python types -_without_ having a unique python type. - -``` -use pyo3::prelude::*; - -struct MyPyObjectWrapper(PyObject); - -impl IntoPy for MyPyObjectWrapper { - fn into_py(self, py: Python) -> PyObject { - self.0 - } -} -``` - -### The `ToPyObject` trait - -[`ToPyObject`] is a conversion trait that allows various objects to be -converted into [`PyObject`]. `IntoPy` serves the -same purpose, except that it consumes `self`. - - -### `*args` and `**kwargs` for Python object calls - -There are several ways how to pass positional and keyword arguments to a Python object call. -[`PyAny`] provides two methods: - -* `call` - call any callable Python object. -* `call_method` - call a specific method on the object, shorthand for `get_attr` then `call`. - -Both methods need `args` and `kwargs` arguments, but there are variants for less -complex calls, such as `call1` for only `args` and `call0` for no arguments at all. - -```rust -use pyo3::prelude::*; -use pyo3::types::{PyDict, PyTuple}; - -struct SomeObject; -impl SomeObject { - fn new(py: Python) -> PyObject { - PyDict::new(py).to_object(py) - } -} - -fn main() { - let arg1 = "arg1"; - let arg2 = "arg2"; - let arg3 = "arg3"; - - let gil = Python::acquire_gil(); - let py = gil.python(); - - let obj = SomeObject::new(py); - - // call object without empty arguments - obj.call0(py); - - // call object with PyTuple - let args = PyTuple::new(py, &[arg1, arg2, arg3]); - obj.call1(py, args); - - // pass arguments as rust tuple - let args = (arg1, arg2, arg3); - obj.call1(py, args); -} -``` - -`kwargs` can be `None` or `Some(&PyDict)`. You can use the -[`IntoPyDict`] trait to convert other dict-like containers, -e.g. `HashMap` or `BTreeMap`, as well as tuples with up to 10 elements and -`Vec`s where each element is a two-element tuple. - -```rust -use pyo3::prelude::*; -use pyo3::types::{IntoPyDict, PyDict}; -use std::collections::HashMap; - -struct SomeObject; - -impl SomeObject { - fn new(py: Python) -> PyObject { - PyDict::new(py).to_object(py) - } -} - -fn main() { - let key1 = "key1"; - let val1 = 1; - let key2 = "key2"; - let val2 = 2; - - let gil = Python::acquire_gil(); - let py = gil.python(); - - let obj = SomeObject::new(py); - - // call object with PyDict - let kwargs = [(key1, val1)].into_py_dict(py); - obj.call(py, (), Some(kwargs)); - - // pass arguments as Vec - let kwargs = vec![(key1, val1), (key2, val2)]; - obj.call(py, (), Some(kwargs.into_py_dict(py))); - - // pass arguments as HashMap - let mut kwargs = HashMap::<&str, i32>::new(); - kwargs.insert(key1, 1); - obj.call(py, (), Some(kwargs.into_py_dict(py))); -} -``` - -[`IntoPy`]: https://docs.rs/pyo3/latest/pyo3/conversion/trait.IntoPy.html -[`FromPyObject`]: https://docs.rs/pyo3/latest/pyo3/conversion/trait.FromPyObject.html -[`ToPyObject`]: https://docs.rs/pyo3/latest/pyo3/conversion/trait.ToPyObject.html -[`PyObject`]: https://docs.rs/pyo3/latest/pyo3/type.PyObject.html -[`PyTuple`]: https://docs.rs/pyo3/latest/pyo3/types/struct.PyTuple.html -[`PyAny`]: https://docs.rs/pyo3/latest/pyo3/struct.PyAny.html -[`IntoPyDict`]: https://docs.rs/pyo3/latest/pyo3/types/trait.IntoPyDict.html - -[`PyRef`]: https://docs.rs/pyo3/latest/pyo3/pycell/struct.PyRef.html -[`PyRefMut`]: https://docs.rs/pyo3/latest/pyo3/pycell/struct.PyRefMut.html - -[^1]: Requires the `num-complex` optional feature. -[^2]: Requires the `hashbrown` optional feature. diff --git a/guide/src/conversions/tables.md b/guide/src/conversions/tables.md new file mode 100644 index 00000000..be1750af --- /dev/null +++ b/guide/src/conversions/tables.md @@ -0,0 +1,96 @@ +## Mapping of Rust types to Python types + +When writing functions callable from Python (such as a `#[pyfunction]` or in a `#[pymethods]` block), the trait `FromPyObject` is required for function arguments, and `IntoPy` is required for function return values. + +Consult the tables in the following section to find the Rust types provided by PyO3 which implement these traits. + +### Argument Types + +When accepting a function argument, it is possible to either use Rust library types or PyO3's Python-native types. (See the next section for discussion on when to use each.) + +The table below contains the Python type and the corresponding function argument types that will accept them: + +| Python | Rust | Rust (Python-native) | +| ------------- |:-------------------------------:|:--------------------:| +| `object` | - | `&PyAny` | +| `str` | `String`, `Cow`, `&str` | `&PyUnicode` | +| `bytes` | `Vec`, `&[u8]` | `&PyBytes` | +| `bool` | `bool` | `&PyBool` | +| `int` | Any integer type (`i32`, `u32`, `usize`, etc) | `&PyLong` | +| `float` | `f32`, `f64` | `&PyFloat` | +| `complex` | `num_complex::Complex`[^1] | `&PyComplex` | +| `list[T]` | `Vec` | `&PyList` | +| `dict[K, V]` | `HashMap`, `BTreeMap`, `hashbrown::HashMap`[^2] | `&PyDict` | +| `tuple[T, U]` | `(T, U)`, `Vec` | `&PyTuple` | +| `set[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^2] | `&PySet` | +| `frozenset[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^2] | `&PyFrozenSet` | +| `bytearray` | `Vec` | `&PyByteArray` | +| `slice` | - | `&PySlice` | +| `type` | - | `&PyType` | +| `module` | - | `&PyModule` | +| `datetime.datetime` | - | `&PyDateTime` | +| `datetime.date` | - | `&PyDate` | +| `datetime.time` | - | `&PyTime` | +| `datetime.tzinfo` | - | `&PyTzInfo` | +| `datetime.timedelta` | - | `&PyDelta` | +| `typing.Optional[T]` | `Option` | - | +| `typing.Sequence[T]` | `Vec` | `&PySequence` | +| `typing.Iterator[Any]` | - | `&PyIterator` | +| `typing.Union[...]` | See [`#[derive(FromPyObject)]`](traits.html#deriving-a-hrefhttpsdocsrspyo3latestpyo3conversiontraitfrompyobjecthtmlfrompyobjecta-for-enums) | - | + +There are also a few special types related to the GIL and Rust-defined `#[pyclass]`es which may come in useful: + +| What | Description | +| ------------- | ------------------------------------- | +| `Python` | A GIL token, used to pass to PyO3 constructors to prove ownership of the GIL | +| `Py` | A Python object isolated from the GIL lifetime. This can be sent to other threads. | +| `PyObject` | An alias for `Py` | +| `&PyCell` | A `#[pyclass]` value owned by Python. | +| `PyRef` | A `#[pyclass]` borrowed immutably. | +| `PyRefMut` | A `#[pyclass]` borrowed mutably. | + +For more detail on accepting `#[pyclass]` values as function arguments, see [the section of this guide on Python Classes](class.md). + +#### Using Rust library types vs Python-native types + +Using Rust library types as function arguments will incur a conversion cost compared to using the Python-native types. Using the Python-native types is almost zero-cost (they just require a type check similar to the Python builtin function `isinstance()`). + +However, once that conversion cost has been paid, the Rust standard library types offer a number of benefits: +- You can write functionality in native-speed Rust code (free of Python's runtime costs). +- You get better interoperability with the rest of the Rust ecosystem. +- You can use `Python::allow_threads` to release the Python GIL and let other Python threads make progress while your Rust code is executing. +- You also benefit from stricter type checking. For example you can specify `Vec`, which will only accept a Python `list` containing integers. The Python-native equivalent, `&PyList`, would accept a Python `list` containing Python objects of any type. + +For most PyO3 usage the conversion cost is worth paying to get these benefits. As always, if you're not sure it's worth it in your case, benchmark it! + +### Returning Rust values to Python + +When returning values from functions callable from Python, Python-native types (`&PyAny`, `&PyDict` etc.) can be used with zero cost. + +Because these types are references, in some situations the Rust compiler may ask for lifetime annotations. If this is the case, you should use `Py`, `Py` etc. instead - which are also zero-cost. For all of these Python-native types `T`, `Py` can be created from `T` with an `.into()` conversion. + +If your function is fallible, it should return `PyResult` or `Result` where `E` implements `From for PyErr`. This will raise a `Python` exception if the `Err` variant is returned. + +Finally, the following Rust types are also able to convert to Python as return values: + +| Rust type | Resulting Python Type | +| ------------- |:-------------------------------:| +| `String` | `str` | +| `&str` | `str` | +| `bool` | `bool` | +| Any integer type (`i32`, `u32`, `usize`, etc) | `int` | +| `f32`, `f64` | `float` | +| `Option` | `Optional[T]` | +| `(T, U)` | `Tuple[T, U]` | +| `Vec` | `List[T]` | +| `HashMap` | `Dict[K, V]` | +| `BTreeMap` | `Dict[K, V]` | +| `HashSet` | `Set[T]` | +| `BTreeSet` | `Set[T]` | +| `&PyCell` | `T` | +| `PyRef` | `T` | +| `PyRefMut` | `T` | + +[^1]: Requires the `num-complex` optional feature. + +[^2]: Requires the `hashbrown` optional feature. diff --git a/guide/src/conversions/traits.md b/guide/src/conversions/traits.md new file mode 100644 index 00000000..7d997538 --- /dev/null +++ b/guide/src/conversions/traits.md @@ -0,0 +1,324 @@ +## Conversion traits + +PyO3 provides some handy traits to convert between Python types and Rust types. + +### `.extract()` and the `FromPyObject` trait + +The easiest way to convert a Python object to a Rust value is using +`.extract()`. It returns a `PyResult` with a type error if the conversion +fails, so usually you will use something like + +```ignore +let v: Vec = obj.extract()?; +``` + +This method is available for many Python object types, and can produce a wide +variety of Rust types, which you can check out in the implementor list of +[`FromPyObject`]. + +[`FromPyObject`] is also implemented for your own Rust types wrapped as Python +objects (see [the chapter about classes](class.md)). There, in order to both be +able to operate on mutable references *and* satisfy Rust's rules of non-aliasing +mutable references, you have to extract the PyO3 reference wrappers [`PyRef`] +and [`PyRefMut`]. They work like the reference wrappers of +`std::cell::RefCell` and ensure (at runtime) that Rust borrows are allowed. + +#### Deriving [`FromPyObject`] + +[`FromPyObject`] can be automatically derived for many kinds of structs and enums +if the member types themselves implement `FromPyObject`. This even includes members +with a generic type `T: FromPyObject`. Derivation for empty enums, enum variants and +structs is not supported. + +#### Deriving [`FromPyObject`] for structs + +``` +use pyo3::prelude::*; + +#[derive(FromPyObject)] +struct RustyStruct { + my_string: String, +} +``` + +The derivation generates code that will per default access the attribute `my_string` on +the Python object, i.e. `obj.getattr("my_string")`, and call `extract()` on the attribute. +It is also possible to access the value on the Python object through `obj.get_item("my_string")` +by setting the attribute `pyo3(item)` on the field: +``` +use pyo3::prelude::*; + +#[derive(FromPyObject)] +struct RustyStruct { + #[pyo3(item)] + my_string: String, +} +``` + +The argument passed to `getattr` and `get_item` can also be configured: + +``` +use pyo3::prelude::*; + +#[derive(FromPyObject)] +struct RustyStruct { + #[pyo3(item("key"))] + string_in_mapping: String, + #[pyo3(attribute("name"))] + string_attr: String, +} +``` + +This tries to extract `string_attr` from the attribute `name` and `string_in_mapping` +from a mapping with the key `"key"`. The arguments for `attribute` are restricted to +non-empty string literals while `item` can take any valid literal that implements +`ToBorrowedObject`. + +#### Deriving [`FromPyObject`] for tuple structs + +Tuple structs are also supported but do not allow customizing the extraction. The input is +always assumed to be a Python tuple with the same length as the Rust type, the `n`th field +is extracted from the `n`th item in the Python tuple. + +``` +use pyo3::prelude::*; + +#[derive(FromPyObject)] +struct RustyTuple(String, String); +``` + +Tuple structs with a single field are treated as wrapper types which are described in the +following section. To override this behaviour and ensure that the input is in fact a tuple, +specify the struct as +``` +use pyo3::prelude::*; + +#[derive(FromPyObject)] +struct RustyTuple((String,)); +``` + +#### Deriving [`FromPyObject`] for wrapper types + +The `pyo3(transparent)` attribute can be used on structs with exactly one field. This results +in extracting directly from the input object, i.e. `obj.extract()`, rather than trying to access +an item or attribute. This behaviour is enabled per default for newtype structs and tuple-variants +with a single field. + +``` +use pyo3::prelude::*; + +#[derive(FromPyObject)] +struct RustyTransparentTupleStruct(String); + +#[derive(FromPyObject)] +#[pyo3(transparent)] +struct RustyTransparentStruct { + inner: String, +} +``` + +#### Deriving [`FromPyObject`] for enums + +The `FromPyObject` derivation for enums generates code that tries to extract the variants in the +order of the fields. As soon as a variant can be extracted succesfully, that variant is returned. +This makes it possible to extract Python types like `Union[str, int]`. + +The same customizations and restrictions described for struct derivations apply to enum variants, +i.e. a tuple variant assumes that the input is a Python tuple, and a struct variant defaults to +extracting fields as attributes but can be configured in the same manner. The `transparent` +attribute can be applied to single-field-variants. + +``` +use pyo3::prelude::*; + +#[derive(FromPyObject)] +enum RustyEnum<'a> { + Int(usize), // input is a positive int + String(String), // input is a string + IntTuple(usize, usize), // input is a 2-tuple with positive ints + StringIntTuple(String, usize), // input is a 2-tuple with String and int + Coordinates3d { // needs to be in front of 2d + x: usize, + y: usize, + z: usize, + }, + Coordinates2d { // only gets checked if the input did not have `z` + #[pyo3(attribute("x"))] + a: usize, + #[pyo3(attribute("y"))] + b: usize, + }, + #[pyo3(transparent)] + CatchAll(&'a PyAny), // This extraction never fails +} +``` + +If none of the enum variants match, a `PyValueError` containing the names of the +tested variants is returned. The names reported in the error message can be customized +through the `pyo3(annotation = "name")` attribute, e.g. to use conventional Python type +names: + +``` +use pyo3::prelude::*; + +#[derive(FromPyObject)] +enum RustyEnum { + #[pyo3(transparent, annotation = "str")] + String(String), + #[pyo3(transparent, annotation = "int")] + Int(isize), +} +``` + +If the input is neither a string nor an integer, the error message will be: +`"Can't convert to Union[str, int]"`, where `` is replaced by the type name and +`repr()` of the input object. + +#### `#[derive(FromPyObject)]` Container Attributes +- `pyo3(transparent)` + - extract the field directly from the object as `obj.extract()` instead of `get_item()` or + `getattr()` + - Newtype structs and tuple-variants are treated as transparent per default. + - only supported for single-field structs and enum variants +- `pyo3(annotation = "name")` + - changes the name of the failed variant in the generated error message in case of failure. + - e.g. `pyo3("int")` reports the variant's type as `int`. + - only supported for enum variants + +#### `#[derive(FromPyObject)]` Field Attributes +- `pyo3(attribute)`, `pyo3(attribute("name"))` + - retrieve the field from an attribute, possibly with a custom name specified as an argument + - argument must be a string-literal. +- `pyo3(item)`, `pyo3(item("key"))` + - retrieve the field from a mapping, possibly with the custom key specified as an argument. + - can be any literal that implements `ToBorrowedObject` + +### `IntoPy` + +This trait defines the to-python conversion for a Rust type. It is usually implemented as +`IntoPy`, which is the trait needed for returning a value from `#[pyfunction]` and +`#[pymethods]`. + +All types in PyO3 implement this trait, as does a `#[pyclass]` which doesn't use `extends`. + +Occasionally you may choose to implement this for custom types which are mapped to Python types +_without_ having a unique python type. + +``` +use pyo3::prelude::*; + +struct MyPyObjectWrapper(PyObject); + +impl IntoPy for MyPyObjectWrapper { + fn into_py(self, py: Python) -> PyObject { + self.0 + } +} +``` + +### The `ToPyObject` trait + +[`ToPyObject`] is a conversion trait that allows various objects to be +converted into [`PyObject`]. `IntoPy` serves the +same purpose, except that it consumes `self`. + + +### `*args` and `**kwargs` for Python object calls + +There are several ways how to pass positional and keyword arguments to a Python object call. +[`PyAny`] provides two methods: + +* `call` - call any callable Python object. +* `call_method` - call a specific method on the object, shorthand for `get_attr` then `call`. + +Both methods need `args` and `kwargs` arguments, but there are variants for less +complex calls, such as `call1` for only `args` and `call0` for no arguments at all. + +```rust +use pyo3::prelude::*; +use pyo3::types::{PyDict, PyTuple}; + +struct SomeObject; +impl SomeObject { + fn new(py: Python) -> PyObject { + PyDict::new(py).to_object(py) + } +} + +fn main() { + let arg1 = "arg1"; + let arg2 = "arg2"; + let arg3 = "arg3"; + + let gil = Python::acquire_gil(); + let py = gil.python(); + + let obj = SomeObject::new(py); + + // call object without empty arguments + obj.call0(py); + + // call object with PyTuple + let args = PyTuple::new(py, &[arg1, arg2, arg3]); + obj.call1(py, args); + + // pass arguments as rust tuple + let args = (arg1, arg2, arg3); + obj.call1(py, args); +} +``` + +`kwargs` can be `None` or `Some(&PyDict)`. You can use the +[`IntoPyDict`] trait to convert other dict-like containers, +e.g. `HashMap` or `BTreeMap`, as well as tuples with up to 10 elements and +`Vec`s where each element is a two-element tuple. + +```rust +use pyo3::prelude::*; +use pyo3::types::{IntoPyDict, PyDict}; +use std::collections::HashMap; + +struct SomeObject; + +impl SomeObject { + fn new(py: Python) -> PyObject { + PyDict::new(py).to_object(py) + } +} + +fn main() { + let key1 = "key1"; + let val1 = 1; + let key2 = "key2"; + let val2 = 2; + + let gil = Python::acquire_gil(); + let py = gil.python(); + + let obj = SomeObject::new(py); + + // call object with PyDict + let kwargs = [(key1, val1)].into_py_dict(py); + obj.call(py, (), Some(kwargs)); + + // pass arguments as Vec + let kwargs = vec![(key1, val1), (key2, val2)]; + obj.call(py, (), Some(kwargs.into_py_dict(py))); + + // pass arguments as HashMap + let mut kwargs = HashMap::<&str, i32>::new(); + kwargs.insert(key1, 1); + obj.call(py, (), Some(kwargs.into_py_dict(py))); +} +``` + +[`IntoPy`]: https://docs.rs/pyo3/latest/pyo3/conversion/trait.IntoPy.html +[`FromPyObject`]: https://docs.rs/pyo3/latest/pyo3/conversion/trait.FromPyObject.html +[`ToPyObject`]: https://docs.rs/pyo3/latest/pyo3/conversion/trait.ToPyObject.html +[`PyObject`]: https://docs.rs/pyo3/latest/pyo3/type.PyObject.html +[`PyTuple`]: https://docs.rs/pyo3/latest/pyo3/types/struct.PyTuple.html +[`PyAny`]: https://docs.rs/pyo3/latest/pyo3/struct.PyAny.html +[`IntoPyDict`]: https://docs.rs/pyo3/latest/pyo3/types/trait.IntoPyDict.html + +[`PyRef`]: https://docs.rs/pyo3/latest/pyo3/pycell/struct.PyRef.html +[`PyRefMut`]: https://docs.rs/pyo3/latest/pyo3/pycell/struct.PyRefMut.html diff --git a/guide/src/index.md b/guide/src/index.md index b7e12841..80534cae 100644 --- a/guide/src/index.md +++ b/guide/src/index.md @@ -1,4 +1,4 @@ -# The PyO3 Guide +# The PyO3 user guide Welcome to the PyO3 user guide! This book is a companion to [PyO3's API docs](https://docs.rs/pyo3). It contains examples and documentation to explain all of PyO3's use cases in detail. diff --git a/guide/src/migration.md b/guide/src/migration.md index bd0dddba..4aefab99 100644 --- a/guide/src/migration.md +++ b/guide/src/migration.md @@ -248,7 +248,7 @@ impl MyClass { ``` Basically you can return `Self` or `Result` directly. -For more, see [the constructor section](https://pyo3.rs/master/class.html#constructor) of this guide. +For more, see [the constructor section](class.html#constructor) of this guide. ### PyCell PyO3 0.9 introduces [`PyCell`], which is a [`RefCell`]-like object wrapper diff --git a/guide/src/trait_bounds.md b/guide/src/trait_bounds.md index e3aac085..c11d585c 100644 --- a/guide/src/trait_bounds.md +++ b/guide/src/trait_bounds.md @@ -1,6 +1,6 @@ # Using in Python a Rust function with trait bounds -PyO3 allows for easy conversion from Rust to Python for certain functions and classes (see the [conversion table](https://pyo3.rs/master/conversions.html)). +PyO3 allows for easy conversion from Rust to Python for certain functions and classes (see the [conversion table](conversions/tables.html). However, it is not always straightforward to convert Rust code that requires a given trait implementation as an argument. This tutorial explains how to convert a Rust function that takes a trait as argument for use in Python with classes implementing the same methods as the trait.