luxolus/pyo3
2
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Apply suggestions from birkenfeld

Browse source 
Co-authored-by: Georg Brandl <georg@python.org>
This commit is contained in:
Yuji Kanagawa 2021-03-06 23:59:46 +09:00 committed by GitHub
parent a591b55219
commit 7c1cc29612
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 17 additions and 22 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37
Architecture.md
View file

@ -1,18 +1,18 @@
<!-- This file contains a rough overview of PyO3 codebase. --> <!-- This file contains a rough overview of the PyO3 codebase. -->
<!-- Please do not make descriptions too specific, so that we can easily --> <!-- Please do not make descriptions too specific, so that we can easily -->
<!-- keep this file in sync with the codebase. --> <!-- keep this file in sync with the codebase. -->
# PyO3: Architecture.md # PyO3: Architecture.md
This document roughly describes the high-level architecture of PyO3. This document roughly describes the high-level architecture of PyO3.
If you are to become familiar with the codebase, you are in the right place! If you want to become familiar with the codebase, you are in the right place!
## Overview ## Overview
PyO3 provides a bridge between Rust and Python, based on the [Python C/API]. PyO3 provides a bridge between Rust and Python, based on the [Python C/API].
Thus, PyO3 has low-level bindings of these API as its core. Thus, PyO3 has low-level bindings of these API as its core.
On top of that, we have higher-level bindings to operate Python objects safely. On top of that, we have higher-level bindings to operate Python objects safely.
Also, to define Python class and functions in Rust code, we have `trait PyClass<T>` and a set of Also, to define Python classes and functions in Rust code, we have `trait PyClass<T>` and a set of
protocol traits (e.g., `PyIterProtocol`) for supporting object protocols (i.e., `__dunder__` methods). protocol traits (e.g., `PyIterProtocol`) for supporting object protocols (i.e., `__dunder__` methods).
Since implementing `PyClass` requires lots of boilerplates, we have a proc-macro `#[pyclass]`. Since implementing `PyClass` requires lots of boilerplates, we have a proc-macro `#[pyclass]`.
@ -25,18 +25,16 @@ To summarize, we have mainly four parts in the PyO3 codebase.
- [`src/pycell.rs`], [`src/pyclass.rs`] - [`src/pycell.rs`], [`src/pyclass.rs`]
4. Protocol methods like `__getitem__`. 4. Protocol methods like `__getitem__`.
- [`src/class`] - [`src/class`]
5. Defining a Python class requires lots of glue codes, so we provide proc-macros to simplify the procedure. 5. Defining a Python class requires lots of glue code, so we provide proc-macros to simplify the procedure.
- [`src/derive_utils.rs`] - [`src/derive_utils.rs`]
- [`pyo3-macros`], [`pyo3-macros-backend`] - [`pyo3-macros`], [`pyo3-macros-backend`]
## Low-level bindings of CPython API ## Low-level bindings of CPython API
[`src/ffi`] contains wrappers of [Python C/API](https://docs.python.org/3/c-api/). [`src/ffi`] contains wrappers of [Python C/API](https://docs.python.org/3/c-api/).
We aim to provide straight-forward Rust wrappers and resemble the file structure of the We aim to provide straight-forward Rust wrappers resembling the file structure of
[`cpython/Include`](https://github.com/python/cpython/tree/v3.9.2/Include). [`cpython/Include`](https://github.com/python/cpython/tree/v3.9.2/Include).
However, we still lack some API and continue to refactor the module to resemble the CPython's
file structure completely.
However, we still lack some API and continue to refactor the module to completely resemble However, we still lack some API and continue to refactor the module to completely resemble
the CPython's file structure. the CPython's file structure.
The tracking issue is [#1289](https://github.com/PyO3/pyo3/issues/1289), and contribution is welcome. The tracking issue is [#1289](https://github.com/PyO3/pyo3/issues/1289), and contribution is welcome.
@ -44,13 +42,12 @@ The tracking issue is [#1289](https://github.com/PyO3/pyo3/issues/1289), and con
## Bindings to Python Objects ## Bindings to Python Objects
[`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html) [`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html)
of Python, such as `dict` and `list`. of Python, such as `dict` and `list`.
Due to a historical reason, Python's `object` is called `PyAny` and placed in [`src/types/any.rs`]. Due to historical reasons, Python's `object` is called `PyAny` and placed in [`src/types/any.rs`].
Currently, `PyAny` is a straight-forward wrapper of `ffi::PyObject`, like: Currently, `PyAny` is a straight-forward wrapper of `ffi::PyObject`, like:
```rust ```rust
#[repr(transparent)] #[repr(transparent)]
pub struct PyAny(UnsafeCell<ffi::PyObject>); pub struct PyAny(UnsafeCell<ffi::PyObject>);
``` ```
.
All built-in types are defined as a C struct. All built-in types are defined as a C struct.
For example, `dict` is defined as: For example, `dict` is defined as:
@ -66,7 +63,6 @@ typedef struct {
PyObject **ma_values; PyObject **ma_values;
} PyDictObject; } PyDictObject;
``` ```
.
However, we cannot access such a specific data structure with `#[cfg(Py_LIMITED_API)]` set. However, we cannot access such a specific data structure with `#[cfg(Py_LIMITED_API)]` set.
Thus, all builtin objects are implemented as opaque types by wrapping `PyAny`, like: Thus, all builtin objects are implemented as opaque types by wrapping `PyAny`, like:
@ -80,7 +76,7 @@ Python heap, as `&PyAny`.
This design choice can be changed This design choice can be changed
(see the discussion in [#1056](https://github.com/PyO3/pyo3/issues/1056)). (see the discussion in [#1056](https://github.com/PyO3/pyo3/issues/1056)).
Since we need lots of boilerplates for implementing common traits for these types Since we need lots of boilerplate for implementing common traits for these types
(e.g., `AsPyPointer`, `AsRef<PyAny>`, and `Debug`), we have some macros in (e.g., `AsPyPointer`, `AsRef<PyAny>`, and `Debug`), we have some macros in
[`src/types/mod.rs`]. [`src/types/mod.rs`].
@ -99,9 +95,9 @@ pub struct PyObject {
... ...
} }
``` ```
Thanks to this gurantee, casting `*mut A` to `*mut PyObject` is valid if `A` is a Python object. Thanks to this guarantee, casting `*mut A` to `*mut PyObject` is valid if `A` is a Python object.
To ensure this gurantee, we have a wrapper struct `PyCell<T>` in [`src/pycell.rs`] which is roughly: To ensure this guarantee, we have a wrapper struct `PyCell<T>` in [`src/pycell.rs`] which is roughly:
```rust ```rust
#[repr(C)] #[repr(C)]
pub struct PyCell<T: PyClass> { pub struct PyCell<T: PyClass> {
@ -109,9 +105,8 @@ pub struct PyCell<T: PyClass> {
inner: T, inner: T,
} }
``` ```
. Thus, when copying a Rust struct to a Python object, we first allocate `PyCell` on the Python heap and then
Thus, when copying a Rust struct to a Python object, we first allocate `PyCell` on a Python heap and then copy `T`.
copies `T`.
Also, `PyCell` provides [RefCell](https://doc.rust-lang.org/std/cell/struct.RefCell.html)-like methods Also, `PyCell` provides [RefCell](https://doc.rust-lang.org/std/cell/struct.RefCell.html)-like methods
to ensure Rust's borrow rules. to ensure Rust's borrow rules.
See [the document](https://docs.rs/pyo3/latest/pyo3/pycell/struct.PyCell.html) for more. See [the document](https://docs.rs/pyo3/latest/pyo3/pycell/struct.PyCell.html) for more.
@ -120,9 +115,9 @@ See [the document](https://docs.rs/pyo3/latest/pyo3/pycell/struct.PyCell.html) f
This trait is somewhat complex and derives many traits, but the most important one is `PyTypeObject` This trait is somewhat complex and derives many traits, but the most important one is `PyTypeObject`
in [`src/type_object.rs`]. in [`src/type_object.rs`].
`PyTypeObject` is also implemented for built-in types. `PyTypeObject` is also implemented for built-in types.
Type objects are singleton, and all Python types have their unique type objects. Type objects are singletons, and all Python types have their unique type objects.
For example, you can see `type({})` shows `dict` and `type(type({}))` shows `type` in Python REPL. For example, you can see `type({})` shows `dict` and `type(type({}))` shows `type` in Python REPL.
`T: PyTypeObject` implies that `T` has a corresponding type object `T: PyTypeObject` implies that `T` has a corresponding type object.
## Protocol methods ## Protocol methods
Python has some built-in special methods called dunder, such as `__iter__`. Python has some built-in special methods called dunder, such as `__iter__`.
@ -132,10 +127,10 @@ We provide a way to implement those protocols by using `#[pyproto]` and specific
as `PyIterProtocol`. as `PyIterProtocol`.
[`src/class`] defines these traits. [`src/class`] defines these traits.
Each protocol method has a corresponding FFI function. Each protocol method has a corresponding FFI function.
For example, when `PyIterProtocol::__iter__` has For example, `PyIterProtocol::__iter__` has
`pub unsafe extern "C" fn iter<T>(slf: *mut PyObject) -> *mut PyObject`. `pub unsafe extern "C" fn iter<T>(slf: *mut PyObject) -> *mut PyObject`.
When `#[pyproto]` finds that `T` implements `PyIterProtocol::__iter__`, it automatically When `#[pyproto]` finds that `T` implements `PyIterProtocol::__iter__`, it automatically
set `iter<T>` to the type object of `T`. sets `iter<T>` on the type object of `T`.
Also, [`src/class/methods.rs`] has utilities for `#[pyfunction]` and [`src/class/impl_.rs`] has Also, [`src/class/methods.rs`] has utilities for `#[pyfunction]` and [`src/class/impl_.rs`] has
some internal tricks for making `#[pyproto]` flexible. some internal tricks for making `#[pyproto]` flexible.
@ -143,7 +138,7 @@ some internal tricks for making `#[pyproto]` flexible.
## Proc-macros ## Proc-macros
[`pyo3-macros`] provides six proc-macro APIs: `pymodule`, `pyproto`, `pyfunction`, `pyclass`, [`pyo3-macros`] provides six proc-macro APIs: `pymodule`, `pyproto`, `pyfunction`, `pyclass`,
`pymethods`, and `#[derive(FromPyObject)]`. `pymethods`, and `#[derive(FromPyObject)]`.
[`pyo3-macros-backend`] has actual implementations of these APIs. [`pyo3-macros-backend`] has the actual implementations of these APIs.
[`src/derive_utils.rs`] contains some utilities used in codes generated by these proc-macros, [`src/derive_utils.rs`] contains some utilities used in codes generated by these proc-macros,
such as parsing function arguments. such as parsing function arguments.

2
README.md
View file

@ -14,7 +14,7 @@
* Contributing Notes: [github](https://github.com/PyO3/pyo3/blob/master/Contributing.md) * Contributing Notes: [github](https://github.com/PyO3/pyo3/blob/master/Contributing.md)
* Berif Architecture Guide: [github](https://github.com/PyO3/pyo3/blob/master/Architecture.md) * Brief Architecture Guide: [github](https://github.com/PyO3/pyo3/blob/master/Architecture.md)
A comparison with rust-cpython can be found [in the guide](https://pyo3.rs/master/rust_cpython.html). A comparison with rust-cpython can be found [in the guide](https://pyo3.rs/master/rust_cpython.html).