luxolus
/
pyo3
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #2173 from davidhewitt/deprecate-pyproto 

pyproto: deprecate protocol traits
This commit is contained in:
David Hewitt 2022-02-24 23:54:40 +00:00 committed by GitHub
parent 15046a2812 424644181b
commit 7c865fcc25
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
25 changed files with 375 additions and 236 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
Architecture.md
View File

@ -144,20 +144,12 @@ For example, you can see `type({})` shows `dict` and `type(type({}))` shows `typ
## 4. Protocol methods
Python has some built-in special methods called dunder, such as `__iter__`.
They are called [abstract objects layer](https://docs.python.org/3/c-api/abstract.html) in
Python has some built-in special methods called dunder methods, such as `__iter__`.
They are called "slots" in the [abstract objects layer](https://docs.python.org/3/c-api/abstract.html) in
Python/C API.
We provide a way to implement those protocols by using `#[pyproto]` and specific traits, such
as `PyIterProtocol`.
[`src/class`] defines these traits.
Each protocol method has a corresponding FFI function.
For example, `PyIterProtocol::__iter__` has
`pub unsafe extern "C" fn iter<T>(slf: *mut PyObject) -> *mut PyObject`.
When `#[pyproto]` finds that `T` implements `PyIterProtocol::__iter__`, it automatically
sets `iter<T>` on the type object of `T`.
Also, [`src/class/methods.rs`] has utilities for `#[pyfunction]` and [`src/class/impl_.rs`] has
some internal tricks for making `#[pyproto]` flexible.
We provide a way to implement those protocols similarly, by recognizing special
names in `#[pymethods]`, with a few new ones for slots that can not be
implemented in Python, such as GC support.
## 5. Procedural macros to simplify usage for users.

3
CHANGELOG.md
View File

@ -73,7 +73,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `PyDate_FromTimestamp`
- Deprecate the `gc` option for `pyclass` (e.g. `#[pyclass(gc)]`). Just implement a `__traverse__` `#[pymethod]`. [#2159](https://github.com/PyO3/pyo3/pull/2159)
- The `ml_meth` field of `PyMethodDef` is now represented by the `PyMethodDefPointer` union [2166](https://github.com/PyO3/pyo3/pull/2166)
- The `PyTypeError` thrown when argument parsing failed now contains the nested causes [2177](https://github.com/PyO3/pyo3/pull/2178)
- Deprecate the `#[pyproto]` traits. [#2173](https://github.com/PyO3/pyo3/pull/2173)
### Removed
@ -92,6 +92,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Add missing FFI definitions `_PyLong_NumBits` and `_PyLong_AsByteArray` on PyPy. [#2146](https://github.com/PyO3/pyo3/pull/2146)
- Fix memory leak in implementation of `AsPyPointer` for `Option<T>`. [#2160](https://github.com/PyO3/pyo3/pull/2160)
- Fix the signature of `_PyLong_NumBits` [#2161](https://github.com/PyO3/pyo3/pull/2161)
- Fix `TypeError` thrown when argument parsing failed missing the originating causes. [2177](https://github.com/PyO3/pyo3/pull/2178)
## [0.15.1] - 2021-11-19

5
benches/bench_pyclass.rs
View File

@ -1,5 +1,5 @@
use criterion::{criterion_group, criterion_main, Criterion};
use pyo3::{class::PyObjectProtocol, prelude::*, type_object::LazyStaticType};
use pyo3::{prelude::*, type_object::LazyStaticType};
/// This is a feature-rich class instance used to benchmark various parts of the pyclass lifecycle.
#[pyclass]
@ -19,10 +19,7 @@ impl MyClass {
self.elements.push(new_element);
self.elements.len()
}
}
#[pyproto]
impl PyObjectProtocol for MyClass {
/// A basic __str__ implementation.
fn __str__(&self) -> &'static str {
"MyClass"

8
guide/src/class.md
View File

@ -2,7 +2,7 @@
PyO3 exposes a group of attributes powered by Rust's proc macro system for defining Python classes as Rust structs.
The main attribute is `#[pyclass]`, which is placed upon a Rust `struct` or a fieldless `enum` (a.k.a. C-like enum) to generate a Python type for it. They will usually also have *one* `#[pymethods]`-annotated `impl` block for the struct, which is used to define Python methods and constants for the generated Python type. (If the [`multiple-pymethods`] feature is enabled each `#[pyclass]` is allowed to have multiple `#[pymethods]` blocks.) Finally, there may be multiple `#[pyproto]` trait implementations for the struct, which are used to define certain python magic methods such as `__str__`.
The main attribute is `#[pyclass]`, which is placed upon a Rust `struct` or a fieldless `enum` (a.k.a. C-like enum) to generate a Python type for it. They will usually also have *one* `#[pymethods]`-annotated `impl` block for the struct, which is used to define Python methods and constants for the generated Python type. (If the [`multiple-pymethods`] feature is enabled each `#[pyclass]` is allowed to have multiple `#[pymethods]` blocks.) `#[pymethods]` may also have implementations for Python magic methods such as `__str__`.
This chapter will discuss the functionality and configuration these attributes offer. Below is a list of links to the relevant section of this chapter for each:
@ -16,7 +16,7 @@ This chapter will discuss the functionality and configuration these attributes o
- [`#[classmethod]`](#class-methods)
- [`#[classattr]`](#class-attributes)
- [`#[args]`](#method-arguments)
- [`#[pyproto]`](class/protocols.html)
- [Magic methods and slots](class/protocols.html)
## Defining a new class
@ -926,9 +926,9 @@ enum BadSubclass{
## Implementation details
The `#[pyclass]` macros rely on a lot of conditional code generation: each `#[pyclass]` can optionally have a `#[pymethods]` block as well as several different possible `#[pyproto]` trait implementations.
The `#[pyclass]` macros rely on a lot of conditional code generation: each `#[pyclass]` can optionally have a `#[pymethods]` block.
To support this flexibility the `#[pyclass]` macro expands to a blob of boilerplate code which sets up the structure for ["dtolnay specialization"](https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.md). This implementation pattern enables the Rust compiler to use `#[pymethods]` and `#[pyproto]` implementations when they are present, and fall back to default (empty) definitions when they are not.
To support this flexibility the `#[pyclass]` macro expands to a blob of boilerplate code which sets up the structure for ["dtolnay specialization"](https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.md). This implementation pattern enables the Rust compiler to use `#[pymethods]` implementations when they are present, and fall back to default (empty) definitions when they are not.
This simple technique works for the case when there is zero or one implementations. To support multiple `#[pymethods]` for a `#[pyclass]` (in the [`multiple-pymethods`] feature), a registry mechanism provided by the [`inventory`](https://github.com/dtolnay/inventory) crate is used instead. This collects `impl`s at library load time, but isn't supported on all platforms. See [inventory: how it works](https://github.com/dtolnay/inventory#how-it-works) for more details.

461
guide/src/class/protocols.md
View File

@ -1,19 +1,15 @@
# Class customizations
# Magic methods and slots
Python's object model defines several protocols for different object behavior, such as the sequence, mapping, and number protocols. You may be familiar with implementing these protocols in Python classes by "magic" methods, such as `__str__` or `__repr__`. Because of the double-underscores surrounding their name, these are also known as "dunder" methods.
In the Python C-API which PyO3 is implemented upon, many of these magic methods have to be placed into special "slots" on the class type object, as covered in the previous section. There are two ways in which this can be done:
- [Experimental for PyO3 0.15, may change slightly in PyO3 0.16] In `#[pymethods]`, if the name of the method is a recognised magic method, PyO3 will place it in the type object automatically.
- [Stable, but expected to be deprecated in PyO3 0.16] In special traits combined with the `#[pyproto]` attribute.
- [New in PyO3 0.15, recommended in PyO3 0.16] In `#[pymethods]`, if the name of the method is a recognised magic method, PyO3 will place it in the type object automatically.
- [Deprecated in PyO3 0.16] In special traits combined with the `#[pyproto]` attribute.
(There are also many magic methods which don't have a special slot, such as `__dir__`. These methods can be implemented as normal in `#[pymethods]`.)
This chapter gives a brief overview of the available methods. An in depth example is given in the following sub-chapters.
### Magic methods in `#[pymethods]`
In PyO3 0.15, if a function name in `#[pymethods]` is a recognised magic method, it will be automatically placed into the correct slot in the Python type object. The function name is taken from the usual rules for naming `#[pymethods]`: the `#[pyo3(name = "...")]` attribute is used if present, otherwise the Rust function name is used.
If a function name in `#[pymethods]` is a recognised magic method, it will be automatically placed into the correct slot in the Python type object. The function name is taken from the usual rules for naming `#[pymethods]`: the `#[pyo3(name = "...")]` attribute is used if present, otherwise the Rust function name is used.
The magic methods handled by PyO3 are very similar to the standard Python ones on [this page](https://docs.python.org/3/reference/datamodel.html#special-method-names) - in particular they are the the subset which have slots as [defined here](https://docs.python.org/3/c-api/typeobj.html). Some of the slots do not have a magic method in Python, which leads to a few additional magic methods defined only in PyO3:
- Magic methods for garbage collection
@ -41,14 +37,16 @@ given signatures should be interpreted as follows:
string object. This is indicated by `object (Python type)`.
#### Basic object customization
### Basic object customization
- `__str__(<self>) -> object (str)`
- `__repr__(<self>) -> object (str)`
- `__hash__(<self>) -> isize`
Objects that compare equal must have the same hash value.
<details>
<summary>Disabling Python's default hash</summary>
By default, all `#[pyclass]` types have a default hash implementation from Python. Types which should not be hashable can override this by setting `__hash__` to `None`. This is the same mechanism as for a pure-Python class. This is done like so:
```rust
@ -66,33 +64,142 @@ given signatures should be interpreted as follows:
</details>
- `__richcmp__(<self>, object, pyo3::basic::CompareOp) -> object`
Overloads Python comparison operations (`==`, `!=`, `<`, `<=`, `>`, and `>=`).
The `CompareOp` argument indicates the comparison operation being performed.
<details>
<summary>Return type</summary>
The return type will normally be `PyResult<bool>`, but any Python object can be returned.
If the `object` is not of the type specified in the signature, the generated code will
automatically `return NotImplemented`.
</details>
- `__getattr__(<self>, object) -> object`
- `__setattr__(<self>, object, object) -> ()`
- `__setattr__(<self>, value: object) -> ()`
- `__delattr__(<self>, object) -> ()`
Overrides attribute access.
- `__bool__(<self>) -> bool`
Determines the "truthyness" of an object.
- `__call__(<self>, ...) -> object` - here, any argument list can be defined
as for normal `pymethods`
#### Iterable objects
### Iterable objects
Iterators can be defined using these methods:
- `__iter__(<self>) -> object`
- `__next__(<self>) -> Option<object> or IterNextOutput` ([see details](#returning-a-value-from-iteration))
#### Awaitable objects
Returning `None` from `__next__` indicates that that there are no further items.
Example:
```rust
use pyo3::prelude::*;
#[pyclass]
struct MyIterator {
iter: Box<dyn Iterator<Item = PyObject> + Send>,
}
#[pymethods]
impl MyIterator {
fn __iter__(slf: PyRef<Self>) -> PyRef<Self> {
slf
}
fn __next__(mut slf: PyRefMut<Self>) -> Option<PyObject> {
slf.iter.next()
}
}
```
In many cases you'll have a distinction between the type being iterated over
(i.e. the *iterable*) and the iterator it provides. In this case, the iterable
only needs to implement `__iter__()` while the iterator must implement both
`__iter__()` and `__next__()`. For example:
```rust
# use pyo3::prelude::*;
#[pyclass]
struct Iter {
inner: std::vec::IntoIter<usize>,
}
#[pymethods]
impl Iter {
fn __iter__(slf: PyRef<Self>) -> PyRef<Self> {
slf
}
fn __next__(mut slf: PyRefMut<Self>) -> Option<usize> {
slf.inner.next()
}
}
#[pyclass]
struct Container {
iter: Vec<usize>,
}
#[pymethods]
impl Container {
fn __iter__(slf: PyRef<Self>) -> PyResult<Py<Iter>> {
let iter = Iter {
inner: slf.iter.clone().into_iter(),
};
Py::new(slf.py(), iter)
}
}
# Python::with_gil(|py| {
# let container = Container { iter: vec![1, 2, 3, 4] };
# let inst = pyo3::PyCell::new(py, container).unwrap();
# pyo3::py_run!(py, inst, "assert list(inst) == [1, 2, 3, 4]");
# pyo3::py_run!(py, inst, "assert list(iter(iter(inst))) == [1, 2, 3, 4]");
# });
```
For more details on Python's iteration protocols, check out [the "Iterator Types" section of the library
documentation](https://docs.python.org/library/stdtypes.html#iterator-types).
#### Returning a value from iteration
This guide has so far shown how to use `Option<T>` to implement yielding values
during iteration. In Python a generator can also return a value. To express
this in Rust, PyO3 provides the [`IterNextOutput`] enum to both `Yield` values
and `Return` a final value - see its docs for further details and an example.
### Awaitable objects
- `__await__(<self>) -> object`
- `__aiter__(<self>) -> object`
- `__anext__(<self>) -> Option<object> or IterANextOutput`
#### Mapping & Sequence types
### Mapping & Sequence types
- `__len__(<self>) -> usize`
Implements the built-in function `len()` for the sequence.
- `__contains__(<self>, object) -> bool`
Implements membership test operators.
Should return true if `item` is in `self`, false otherwise.
For objects that don’t define `__contains__()`, the membership test simply
traverses the sequence until it finds a match.
<details>
<summary>Disabling Python's default contains</summary>
By default, all `#[pyclass]` types with an `__iter__` method support a default implementation of the `in` operator. Types which do not want this can override this by setting `__contains__` to `None`. This is the same mechanism as for a pure-Python class. This is done like so:
By default, all `#[pyclass]` types with an `__iter__` method support a
default implementation of the `in` operator. Types which do not want this
can override this by setting `__contains__` to `None`. This is the same
mechanism as for a pure-Python class. This is done like so:
```rust
# use pyo3::prelude::*;
@ -107,38 +214,62 @@ given signatures should be interpreted as follows:
}
```
</details>
- `__getitem__(<self>, object) -> object`
Implements retrieval of the `self[a]` element.
*Note:* Negative integer indexes are not handled specially.
- `__setitem__(<self>, object, object) -> ()`
Implements assignment to the `self[a]` element.
Should only be implemented if elements can be replaced.
- `__delitem__(<self>, object) -> ()`
#### Descriptors
Implements deletion of the `self[a]` element.
Should only be implemented if elements can be deleted.
* `fn __concat__(&self, other: impl FromPyObject) -> PyResult<impl ToPyObject>`
Concatenates two sequences.
Used by the `+` operator, after trying the numeric addition via
the `__add__` and `__radd__` methods.
* `fn __repeat__(&self, count: isize) -> PyResult<impl ToPyObject>`
Repeats the sequence `count` times.
Used by the `*` operator, after trying the numeric multiplication via
the `__mul__` and `__rmul__` methods.
* `fn __inplace_concat__(&self, other: impl FromPyObject) -> PyResult<impl ToPyObject>`
Concatenates two sequences.
Used by the `+=` operator, after trying the numeric addition via
the `__iadd__` method.
* `fn __inplace_repeat__(&self, count: isize) -> PyResult<impl ToPyObject>`
Concatenates two sequences.
Used by the `*=` operator, after trying the numeric multiplication via
the `__imul__` method.
### Descriptors
- `__get__(<self>, object, object) -> object`
- `__set__(<self>, object, object) -> ()`
- `__delete__(<self>, object) -> ()`
#### Numeric types
### Numeric types
Binary arithmetic operations (`+`, `-`, `*`, `@`, `/`, `//`, `%`, `divmod()`,
`pow()` and `**`, `<<`, `>>`, `&`, `^`, and `|`) and their reflected versions:
(If the `object` is not of the type specified in the signature, the generated code
will automatically `return NotImplemented`.)
- `__pos__(<self>) -> object`
- `__neg__(<self>) -> object`
- `__abs__(<self>) -> object`
- `__invert__(<self>) -> object`
- `__index__(<self>) -> object (int)`
- `__int__(<self>) -> object (int)`
- `__float__(<self>) -> object (float)`
- `__iadd__(<self>, object) -> ()`
- `__isub__(<self>, object) -> ()`
- `__imul__(<self>, object) -> ()`
- `__imatmul__(<self>, object) -> ()`
- `__itruediv__(<self>, object) -> ()`
- `__ifloordiv__(<self>, object) -> ()`
- `__imod__(<self>, object) -> ()`
- `__ipow__(<self>, object, object) -> ()`
- `__ilshift__(<self>, object) -> ()`
- `__irshift__(<self>, object) -> ()`
- `__iand__(<self>, object) -> ()`
- `__ixor__(<self>, object) -> ()`
- `__ior__(<self>, object) -> ()`
- `__add__(<self>, object) -> object`
- `__radd__(<self>, object) -> object`
- `__sub__(<self>, object) -> object`
@ -168,21 +299,91 @@ given signatures should be interpreted as follows:
- `__pow__(<self>, object, object) -> object`
- `__rpow__(<self>, object, object) -> object`
#### Buffer objects
In-place assignment operations (`+=`, `-=`, `*=`, `@=`, `/=`, `//=`, `%=`,
`**=`, `<<=`, `>>=`, `&=`, `^=`, `|=`):
- `__iadd__(<self>, object) -> ()`
- `__isub__(<self>, object) -> ()`
- `__imul__(<self>, object) -> ()`
- `__imatmul__(<self>, object) -> ()`
- `__itruediv__(<self>, object) -> ()`
- `__ifloordiv__(<self>, object) -> ()`
- `__imod__(<self>, object) -> ()`
- `__ipow__(<self>, object, object) -> ()`
- `__ilshift__(<self>, object) -> ()`
- `__irshift__(<self>, object) -> ()`
- `__iand__(<self>, object) -> ()`
- `__ixor__(<self>, object) -> ()`
- `__ior__(<self>, object) -> ()`
Unary operations (`-`, `+`, `abs()` and `~`):
- `__pos__(<self>) -> object`
- `__neg__(<self>) -> object`
- `__abs__(<self>) -> object`
- `__invert__(<self>) -> object`
Coercions:
- `__index__(<self>) -> object (int)`
- `__int__(<self>) -> object (int)`
- `__float__(<self>) -> object (float)`
### Buffer objects
- `__getbuffer__(<self>, *mut ffi::Py_buffer, flags) -> ()`
- `__releasebuffer__(<self>, *mut ffi::Py_buffer)` (no return value, not even `PyResult`)
#### Garbage Collector Integration
### Garbage Collector Integration
- `__traverse__(<self>, visit: pyo3::class::gc::PyVisit) -> Result<(), pyo3::class::gc::PyTraverseError>`
If your type owns references to other Python objects, you will need to integrate
with Python's garbage collector so that the GC is aware of those references. To
do this, implement the two methods `__traverse__` and `__clear__`. These
correspond to the slots `tp_traverse` and `tp_clear` in the Python C API.
`__traverse__` must call `visit.call()` for each reference to another Python
object. `__clear__` must clear out any mutable references to other Python
objects (thus breaking reference cycles). Immutable references do not have to be
cleared, as every cycle must contain at least one mutable reference.
- `__traverse__(<self>, pyo3::class::gc::PyVisit) -> Result<(), pyo3::class::gc::PyTraverseError>`
- `__clear__(<self>) -> ()`
Example:
```rust
use pyo3::prelude::*;
use pyo3::PyTraverseError;
use pyo3::gc::PyVisit;
#[pyclass]
struct ClassWithGCSupport {
obj: Option<PyObject>,
}
#[pymethods]
impl ClassWithGCSupport {
fn __traverse__(&self, visit: PyVisit) -> Result<(), PyTraverseError> {
if let Some(obj) = &self.obj {
visit.call(obj)?
}
Ok(())
}
fn __clear__(&mut self) {
// Clear reference, this decrements ref counter.
self.obj = None;
}
}
```
[`IterNextOutput`]: {{#PYO3_DOCS_URL}}/pyo3/class/iter/enum.IterNextOutput.html
### `#[pyproto]` traits
PyO3 can use the `#[pyproto]` attribute in combination with special traits to implement the magic methods which need slots. The special traits are listed below. See also the [documentation for the `pyo3::class` module]({{#PYO3_DOCS_URL}}/pyo3/class/index.html).
Before PyO3 0.15 this was the only supported solution for implementing magic methods. Due to complexity in the implementation and usage, these traits are expected to be deprecated in PyO3 0.16 in favour of the `#[pymethods]` solution.
Due to complexity in the implementation and usage, these traits are deprecated in PyO3 0.16 in favour of the `#[pymethods]` solution.
All `#[pyproto]` methods can return `T` instead of `PyResult<T>` if the method implementation is infallible. In addition, if the return type is `()`, it can be omitted altogether.
@ -190,44 +391,15 @@ All `#[pyproto]` methods can return `T` instead of `PyResult<T>` if the method i
The [`PyObjectProtocol`] trait provides several basic customizations.
##### Attribute access
To customize object attribute access, define the following methods:
* `fn __str__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>`
* `fn __repr__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>`
* `fn __hash__(&self) -> PyResult<impl PrimInt>`
* `fn __richcmp__(&self, other: impl FromPyObject, op: CompareOp) -> PyResult<impl ToPyObject>`
* `fn __getattr__(&self, name: impl FromPyObject) -> PyResult<impl IntoPy<PyObject>>`
* `fn __setattr__(&mut self, name: impl FromPyObject, value: impl FromPyObject) -> PyResult<()>`
* `fn __delattr__(&mut self, name: impl FromPyObject) -> PyResult<()>`
Each method corresponds to Python's `self.attr`, `self.attr = value` and `del self.attr` code.
##### String Conversions
* `fn __repr__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>`
* `fn __str__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>`
Possible return types for `__str__` and `__repr__` are `PyResult<String>` or `PyResult<PyString>`.
##### Comparison operators
* `fn __richcmp__(&self, other: impl FromPyObject, op: CompareOp) -> PyResult<impl ToPyObject>`
Overloads Python comparison operations (`==`, `!=`, `<`, `<=`, `>`, and `>=`).
The `op` argument indicates the comparison operation being performed.
The return type will normally be `PyResult<bool>`, but any Python object can be returned.
If `other` is not of the type specified in the signature, the generated code will
automatically `return NotImplemented`.
* `fn __hash__(&self) -> PyResult<impl PrimInt>`
Objects that compare equal must have the same hash value.
The return type must be `PyResult<T>` where `T` is one of Rust's primitive integer types.
##### Other methods
* `fn __bool__(&self) -> PyResult<bool>`
Determines the "truthyness" of the object.
#### Emulating numeric types
The [`PyNumberProtocol`] trait can be implemented to emulate [numeric types](https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types).
@ -247,13 +419,7 @@ The [`PyNumberProtocol`] trait can be implemented to emulate [numeric types](htt
* `fn __or__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult<impl ToPyObject>`
* `fn __xor__(lhs: impl FromPyObject, rhs: impl FromPyObject) -> PyResult<impl ToPyObject>`
These methods are called to implement the binary arithmetic operations
(`+`, `-`, `*`, `@`, `/`, `//`, `%`, `divmod()`, `pow()` and `**`, `<<`, `>>`, `&`, `^`, and `|`).
If `rhs` is not of the type specified in the signature, the generated code
will automatically `return NotImplemented`. This is not the case for `lhs`
which must match signature or else raise a TypeError.
These methods are called to implement the binary arithmetic operations.
The reflected operations are also available:
@ -275,9 +441,6 @@ The reflected operations are also available:
The code generated for these methods expect that all arguments match the
signature, or raise a TypeError.
This trait also has support the augmented arithmetic assignments (`+=`, `-=`,
`*=`, `@=`, `/=`, `//=`, `%=`, `**=`, `<<=`, `>>=`, `&=`, `^=`, `|=`):
* `fn __iadd__(&'p mut self, other: impl FromPyObject) -> PyResult<()>`
* `fn __isub__(&'p mut self, other: impl FromPyObject) -> PyResult<()>`
* `fn __imul__(&'p mut self, other: impl FromPyObject) -> PyResult<()>`
@ -294,7 +457,7 @@ This trait also has support the augmented arithmetic assignments (`+=`, `-=`,
* `fn __ixor__(&'p mut self, other: impl FromPyObject) -> PyResult<()>`
The following methods implement the unary arithmetic operations (`-`, `+`, `abs()` and `~`):
The following methods implement the unary arithmetic operations:
* `fn __neg__(&'p self) -> PyResult<impl ToPyObject>`
* `fn __pos__(&'p self) -> PyResult<impl ToPyObject>`
@ -305,9 +468,6 @@ Support for coercions:
* `fn __int__(&'p self) -> PyResult<impl ToPyObject>`
* `fn __float__(&'p self) -> PyResult<impl ToPyObject>`
Other:
* `fn __index__(&'p self) -> PyResult<impl ToPyObject>`
#### Emulating sequential containers (such as lists or tuples)
@ -406,139 +566,26 @@ For a mapping, the keys may be Python objects of arbitrary type.
The same exceptions should be raised for improper key values as
for the `__getitem__()` method.
#### Garbage Collector Integration
If your type owns references to other Python objects, you will need to
integrate with Python's garbage collector so that the GC is aware of
those references.
To do this, implement the [`PyGCProtocol`] trait for your struct.
It includes two methods `__traverse__` and `__clear__`.
These correspond to the slots `tp_traverse` and `tp_clear` in the Python C API.
`__traverse__` must call `visit.call()` for each reference to another Python object.
`__clear__` must clear out any mutable references to other Python objects
(thus breaking reference cycles). Immutable references do not have to be cleared,
as every cycle must contain at least one mutable reference.
Example:
```rust
use pyo3::prelude::*;
use pyo3::PyTraverseError;
use pyo3::gc::{PyGCProtocol, PyVisit};
#[pyclass]
struct ClassWithGCSupport {
obj: Option<PyObject>,
}
#[pyproto]
impl PyGCProtocol for ClassWithGCSupport {
fn __traverse__(&self, visit: PyVisit) -> Result<(), PyTraverseError> {
if let Some(obj) = &self.obj {
visit.call(obj)?
}
Ok(())
}
fn __clear__(&mut self) {
// Clear reference, this decrements ref counter.
self.obj = None;
}
}
```
#### Iterator Types
Iterators can be defined using the
[`PyIterProtocol`]({{#PYO3_DOCS_URL}}/pyo3/class/iter/trait.PyIterProtocol.html) trait.
Iterators can be defined using the [`PyIterProtocol`] trait.
It includes two methods `__iter__` and `__next__`:
* `fn __iter__(slf: PyRefMut<Self>) -> PyResult<impl IntoPy<PyObject>>`
* `fn __next__(slf: PyRefMut<Self>) -> PyResult<Option<impl IntoPy<PyObject>>>`
Returning `None` from `__next__` indicates that that there are no further items.
These two methods can be take either `PyRef<Self>` or `PyRefMut<Self>` as their
first argument, so that mutable borrow can be avoided if needed.
Example:
For details, look at the `#[pymethods]` regarding iterator methods.
```rust
use pyo3::prelude::*;
use pyo3::PyIterProtocol;
#### Garbage Collector Integration
#[pyclass]
struct MyIterator {
iter: Box<dyn Iterator<Item = PyObject> + Send>,
}
#[pyproto]
impl PyIterProtocol for MyIterator {
fn __iter__(slf: PyRef<Self>) -> PyRef<Self> {
slf
}
fn __next__(mut slf: PyRefMut<Self>) -> Option<PyObject> {
slf.iter.next()
}
}
```
In many cases you'll have a distinction between the type being iterated over (i.e. the *iterable*) and the iterator it
provides. In this case, you should implement `PyIterProtocol` for both the iterable and the iterator, but the iterable
only needs to support `__iter__()` while the iterator must support both `__iter__()` and `__next__()`. The default
implementations in `PyIterProtocol` will ensure that the objects behave correctly in Python. For example:
```rust
# use pyo3::prelude::*;
# use pyo3::PyIterProtocol;
#[pyclass]
struct Iter {
inner: std::vec::IntoIter<usize>,
}
#[pyproto]
impl PyIterProtocol for Iter {
fn __iter__(slf: PyRef<Self>) -> PyRef<Self> {
slf
}
fn __next__(mut slf: PyRefMut<Self>) -> Option<usize> {
slf.inner.next()
}
}
#[pyclass]
struct Container {
iter: Vec<usize>,
}
#[pyproto]
impl PyIterProtocol for Container {
fn __iter__(slf: PyRef<Self>) -> PyResult<Py<Iter>> {
let iter = Iter {
inner: slf.iter.clone().into_iter(),
};
Py::new(slf.py(), iter)
}
}
# Python::with_gil(|py| {
# let container = Container { iter: vec![1, 2, 3, 4] };
# let inst = pyo3::PyCell::new(py, container).unwrap();
# pyo3::py_run!(py, inst, "assert list(inst) == [1, 2, 3, 4]");
# pyo3::py_run!(py, inst, "assert list(iter(iter(inst))) == [1, 2, 3, 4]");
# });
```
For more details on Python's iteration protocols, check out [the "Iterator Types" section of the library
documentation](https://docs.python.org/3/library/stdtypes.html#iterator-types).
##### Returning a value from iteration
This guide has so far shown how to use `Option<T>` to implement yielding values during iteration.
In Python a generator can also return a value. To express this in Rust, PyO3 provides the
[`IterNextOutput`]({{#PYO3_DOCS_URL}}/pyo3/class/iter/enum.IterNextOutput.html) enum to
both `Yield` values and `Return` a final value - see its docs for further details and an example.
Implement the [`PyGCProtocol`] trait for your struct.
For details, look at the `#[pymethods]` regarding GC methods.
[`PyGCProtocol`]: {{#PYO3_DOCS_URL}}/pyo3/class/gc/trait.PyGCProtocol.html
[`PyMappingProtocol`]: {{#PYO3_DOCS_URL}}/pyo3/class/mapping/trait.PyMappingProtocol.html
[`PyNumberProtocol`]: {{#PYO3_DOCS_URL}}/pyo3/class/number/trait.PyNumberProtocol.html
[`PyObjectProtocol`]: {{#PYO3_DOCS_URL}}/pyo3/class/basic/trait.PyObjectProtocol.html
[`PySequenceProtocol`]: {{#PYO3_DOCS_URL}}/pyo3/class/sequence/trait.PySequenceProtocol.html
[`PyIterProtocol`]: {{#PYO3_DOCS_URL}}/pyo3/class/iter/trait.PyIterProtocol.html

57
guide/src/migration.md
View File

@ -7,7 +7,60 @@ For a detailed list of all changes, see the [CHANGELOG](changelog.md).
### Drop support for older technologies
PyO3 0.16 has increased minimum Rust version to 1.48 and minimum Python version to 3.7. This enables ore use of newer language features (enabling some of the other additions in 0.16) and simplifies maintenance of the project.
PyO3 0.16 has increased minimum Rust version to 1.48 and minimum Python version to 3.7. This enables use of newer language features (enabling some of the other additions in 0.16) and simplifies maintenance of the project.
### `#[pyproto]` has been deprecated
In PyO3 0.15, the `#[pymethods]` attribute macro gained support for implementing "magic methods" such as `__str__` (aka "dunder" methods). This implementation was not quite finalized at the time, with a few edge cases to be decided upon. The existing `#[pyproto]` attribute macro was left untouched, because it covered these edge cases.
In PyO3 0.16, the `#[pymethods]` implementation has been completed and is now the preferred way to implement magic methods. To allow the PyO3 project to move forward, `#[pyproto]` has been deprecated (with expected removal in PyO3 0.18).
Migration from `#[pyproto]` to `#[pymethods]` is straightforward; copying the existing methods directly from the `#[pyproto]` trait implementation is all that is needed in most cases.
Before:
```rust,ignore
use pyo3::prelude::*;
use pyo3::class::{PyBasicProtocol, PyIterProtocol};
use pyo3::types::PyString;
#[pyclass]
struct MyClass { }
#[pyproto]
impl PyBasicProtocol for MyClass {
fn __str__(&self) -> &'static [u8] {
b"hello, world"
}
}
#[pyproto]
impl PyIterProtocol for MyClass {
fn __iter__(slf: PyRef<self>) -> PyResult<&PyAny> {
PyString::new(slf.py(), "hello, world").iter()
}
}
```
After
```rust,ignore
use pyo3::prelude::*;
use pyo3::types::PyString;
#[pyclass]
struct MyClass { }
impl MyClass {
fn __str__(&self) -> &'static [u8] {
b"hello, world"
}
fn __iter__(slf: PyRef<self>) -> PyResult<&PyAny> {
PyString::new(slf.py(), "hello, world").iter()
}
}
```
### Container magic methods now match Python behavior
@ -593,7 +646,7 @@ impl PySequenceProtocol for ByteSequence {
```
After:
```rust
```rust,ignore
# use pyo3::prelude::*;
# use pyo3::class::PySequenceProtocol;
#[pyclass]

5
guide/src/python_from_rust.md
View File

@ -165,7 +165,7 @@ quickly testing your Python extensions.
```rust
use pyo3::prelude::*;
use pyo3::{PyCell, PyObjectProtocol, py_run};
use pyo3::{PyCell, py_run};
# fn main() {
#[pyclass]
@ -179,10 +179,7 @@ impl UserData {
fn as_tuple(&self) -> (u32, String) {
(self.id, self.name.clone())
}
}
#[pyproto]
impl PyObjectProtocol for UserData {
fn __repr__(&self) -> PyResult<String> {
Ok(format!("User {}(id: {})", self.name, self.id))
}

10
pyo3-macros-backend/src/pymethod.rs
View File

@ -53,6 +53,12 @@ impl PyMethodKind {
"__contains__" => PyMethodKind::Proto(PyMethodProtoKind::Slot(&__CONTAINS__)),
"__concat__" => PyMethodKind::Proto(PyMethodProtoKind::Slot(&__CONCAT__)),
"__repeat__" => PyMethodKind::Proto(PyMethodProtoKind::Slot(&__REPEAT__)),
"__inplace_concat__" => {
PyMethodKind::Proto(PyMethodProtoKind::Slot(&__INPLACE_CONCAT__))
}
"__inplace_repeat__" => {
PyMethodKind::Proto(PyMethodProtoKind::Slot(&__INPLACE_REPEAT__))
}
"__getitem__" => PyMethodKind::Proto(PyMethodProtoKind::Slot(&__GETITEM__)),
"__pos__" => PyMethodKind::Proto(PyMethodProtoKind::Slot(&__POS__)),
"__neg__" => PyMethodKind::Proto(PyMethodProtoKind::Slot(&__NEG__)),
@ -606,6 +612,10 @@ const __CONTAINS__: SlotDef = SlotDef::new("Py_sq_contains", "objobjproc")
.ret_ty(Ty::Int);
const __CONCAT__: SlotDef = SlotDef::new("Py_sq_concat", "binaryfunc").arguments(&[Ty::Object]);
const __REPEAT__: SlotDef = SlotDef::new("Py_sq_repeat", "ssizeargfunc").arguments(&[Ty::PySsizeT]);
const __INPLACE_CONCAT__: SlotDef =
SlotDef::new("Py_sq_concat", "binaryfunc").arguments(&[Ty::Object]);
const __INPLACE_REPEAT__: SlotDef =
SlotDef::new("Py_sq_repeat", "ssizeargfunc").arguments(&[Ty::PySsizeT]);
const __GETITEM__: SlotDef = SlotDef::new("Py_mp_subscript", "binaryfunc").arguments(&[Ty::Object]);
const __POS__: SlotDef = SlotDef::new("Py_nb_positive", "unaryfunc");

2
src/class/basic.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Basic Python Object customization
@ -14,6 +15,7 @@ use std::os::raw::c_int;
/// Basic Python class customization
#[allow(unused_variables)]
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PyObjectProtocol<'p>: PyClass {
fn __getattr__(&'p self, name: Self::Name) -> Self::Result
where

2
src/class/buffer.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Represent Python Buffer protocol implementation
@ -13,6 +14,7 @@ use std::os::raw::c_int;
/// For more information check [buffer protocol](https://docs.python.org/3/c-api/buffer.html)
/// c-api.
#[allow(unused_variables)]
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PyBufferProtocol<'p>: PyClass {
// No default implementations so that implementors of this trait provide both methods.

2
src/class/descr.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Python Description Interface
@ -11,6 +12,7 @@ use std::os::raw::c_int;
/// Descriptor interface
#[allow(unused_variables)]
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PyDescrProtocol<'p>: PyClass {
fn __get__(
slf: Self::Receiver,

2
src/class/gc.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Python GC support
@ -9,6 +10,7 @@ use std::os::raw::{c_int, c_void};
pub struct PyTraverseError(c_int);
/// GC support
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PyGCProtocol<'p>: PyClass {
fn __traverse__(&'p self, visit: PyVisit) -> Result<(), PyTraverseError>;
fn __clear__(&'p mut self);

3
src/class/iter.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Python Iterator Interface.
//! Trait and support implementation for implementing iterators
@ -16,6 +17,7 @@ use crate::{PyClass, PyObject};
/// the integers 1 to 5, before raising `StopIteration("Ended")`.
///
/// ```rust
/// # #![allow(deprecated)]
/// use pyo3::class::iter::IterNextOutput;
/// use pyo3::prelude::*;
/// use pyo3::PyIterProtocol;
@ -43,6 +45,7 @@ use crate::{PyClass, PyObject};
/// # }); // test of StopIteration is done in pytests/src/pyclasses.rs
/// ```
#[allow(unused_variables)]
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PyIterProtocol<'p>: PyClass {
fn __iter__(slf: Self::Receiver) -> Self::Result
where

2
src/class/mapping.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Python Mapping Interface
@ -8,6 +9,7 @@ use crate::{FromPyObject, PyClass, PyObject};
/// Mapping interface
#[allow(unused_variables)]
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PyMappingProtocol<'p>: PyClass {
fn __len__(&'p self) -> Self::Result
where

0
src/class/methods.rs
View File

1
src/class/mod.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Python object protocols

2
src/class/number.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Python Number Interface
@ -8,6 +9,7 @@ use crate::{ffi, FromPyObject, PyClass, PyObject};
/// Number interface
#[allow(unused_variables)]
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PyNumberProtocol<'p>: PyClass {
fn __add__(lhs: Self::Left, rhs: Self::Right) -> Self::Result
where

2
src/class/pyasync.rs
View File

@ -1,3 +1,4 @@
#![allow(deprecated)]
// Copyright (c) 2017-present PyO3 Project and Contributors
//! Python Async/Await Interface.
@ -15,6 +16,7 @@ use crate::{PyClass, PyObject};
///
/// Each method in this trait corresponds to Python async/await implementation.
#[allow(unused_variables)]
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PyAsyncProtocol<'p>: PyClass {
fn __await__(slf: Self::Receiver) -> Self::Result
where

1
src/class/sequence.rs
View File

@ -11,6 +11,7 @@ use std::os::raw::c_int;
/// Sequence interface
#[allow(unused_variables)]
#[deprecated(since = "0.16.0", note = "prefer `#[pymethods]` to `#[pyproto]`")]
pub trait PySequenceProtocol<'p>: PyClass + Sized {
fn __len__(&'p self) -> Self::Result
where

1
tests/test_buffer_protocol_pyproto.rs
View File

@ -1,6 +1,7 @@
#![cfg(feature = "macros")]
#![cfg(feature = "pyproto")]
#![cfg(any(not(Py_LIMITED_API), Py_3_11))]
#![allow(deprecated)]
use pyo3::buffer::PyBuffer;
use pyo3::class::PyBufferProtocol;

1
tests/test_gc_pyproto.rs
View File

@ -1,5 +1,6 @@
#![cfg(feature = "macros")]
#![cfg(feature = "pyproto")]
#![allow(deprecated)]
use pyo3::class::PyGCProtocol;
use pyo3::class::PyTraverseError;

1
tests/test_mapping_pyproto.rs
View File

@ -1,5 +1,6 @@
#![cfg(feature = "macros")]
#![cfg(feature = "pyproto")]
#![allow(deprecated)]
use std::collections::HashMap;

1
tests/test_pyproto.rs
View File

@ -1,5 +1,6 @@
#![cfg(feature = "macros")]
#![cfg(feature = "pyproto")]
#![allow(deprecated)]
use pyo3::class::{
PyAsyncProtocol, PyDescrProtocol, PyIterProtocol, PyMappingProtocol, PyObjectProtocol,

20
tests/test_sequence.rs
View File

@ -66,12 +66,17 @@ impl ByteSequence {
}
}
fn __concat__(&self, other: PyRef<Self>) -> Self {
fn __concat__(&self, other: &Self) -> Self {
let mut elements = self.elements.clone();
elements.extend_from_slice(&other.elements);
Self { elements }
}
fn __inplace_concat__(mut slf: PyRefMut<Self>, other: &Self) -> Py<Self> {
slf.elements.extend_from_slice(&other.elements);
slf.into()
}
fn __repeat__(&self, count: isize) -> PyResult<Self> {
if count >= 0 {
let mut elements = Vec::with_capacity(self.elements.len() * count as usize);
@ -83,6 +88,19 @@ impl ByteSequence {
Err(PyValueError::new_err("invalid repeat count"))
}
}
fn __inplace_repeat__(mut slf: PyRefMut<Self>, count: isize) -> PyResult<Py<Self>> {
if count >= 0 {
let mut elements = Vec::with_capacity(slf.elements.len() * count as usize);
for _ in 0..count {
elements.extend(&slf.elements);
}
slf.elements = elements;
Ok(slf.into())
} else {
Err(PyValueError::new_err("invalid repeat count"))
}
}
}
/// Return a dict with `s = ByteSequence([1, 2, 3])`.

1
tests/test_sequence_pyproto.rs
View File

@ -1,5 +1,6 @@
#![cfg(feature = "macros")]
#![cfg(feature = "pyproto")]
#![allow(deprecated)]
use pyo3::class::PySequenceProtocol;
use pyo3::exceptions::{PyIndexError, PyValueError};