pyo3/README.md

229 lines
12 KiB
Markdown
Raw Normal View History

2018-05-17 21:59:11 +00:00
# PyO3
2015-04-19 03:29:19 +00:00
2021-03-28 20:56:10 +00:00
[![actions status](https://github.com/PyO3/pyo3/workflows/CI/badge.svg)](https://github.com/PyO3/pyo3/actions)
[![benchmark](https://github.com/PyO3/pyo3/actions/workflows/bench.yml/badge.svg)](https://pyo3.rs/dev/bench/)
2021-03-16 22:14:57 +00:00
[![codecov](https://codecov.io/gh/PyO3/pyo3/branch/main/graph/badge.svg)](https://codecov.io/gh/PyO3/pyo3)
[![crates.io](https://img.shields.io/crates/v/pyo3)](https://crates.io/crates/pyo3)
2021-11-18 11:38:47 +00:00
[![minimum rustc 1.48](https://img.shields.io/badge/rustc-1.48+-blue.svg)](https://rust-lang.github.io/rfcs/2495-min-rust-version.html)
2021-03-28 20:56:10 +00:00
[![dev chat](https://img.shields.io/gitter/room/nwjs/nw.js.svg)](https://gitter.im/PyO3/Lobby)
[![contributing notes](https://img.shields.io/badge/contribute-on%20github-Green)](https://github.com/PyO3/pyo3/blob/main/Contributing.md)
2018-05-17 21:59:11 +00:00
2021-09-18 18:48:45 +00:00
[Rust](https://www.rust-lang.org/) bindings for [Python](https://www.python.org/), including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.
2015-04-19 03:29:19 +00:00
2021-06-26 22:37:30 +00:00
- User Guide: [stable](https://pyo3.rs) | [main](https://pyo3.rs/main)
2015-04-19 03:29:19 +00:00
2021-06-26 22:37:30 +00:00
- API Documentation: [stable](https://docs.rs/pyo3/) | [main](https://pyo3.rs/main/doc)
2018-10-04 06:46:15 +00:00
2018-05-17 21:59:11 +00:00
## Usage
2017-05-21 12:37:56 +00:00
PyO3 supports the following software versions:
2021-11-19 10:45:27 +00:00
- Python 3.7 and up (CPython and PyPy)
2021-11-18 11:38:47 +00:00
- Rust 1.48 and up
Support rust extensions for PyPy via cpyext (#393) * wip * removed stuff * removed another change * implemented minimum amouth of ifdefs to make pypy3 hello world to compile * implemented minimum amount of ifdefs to make pypy3 hello world to compile * hacking on build.rs * compiler is happy! * few todos remain * extracted build logic to seperate module * added pypy test * finally fixed pypy structs * removed some todos * test should now be machine independent * fixed all pypy3 symbols * added pypy feature * removed `is_pypy` * added pypy2 declerations also * fix for cpython2 * improved libpypy detection * added all pypy2 macros * fixed errneous type * more fixes * fix python2 string macros * modsupport symbol * fix * fixed and added many symbols * fixes * remove dup * remove mac-specific config * fix all name mangling macros * unite imports * missing symbol * fix pybool * implemented another missing symbol * it works * fix merge conflict * uncomment non default features * cargo.toml * Cargo fmt * small merge fixes * use newer build version * whoops * fix build script * more build hacks * some random hiccups * small fixes * it builds! * it builds and runs * revert everything in FFI2 * revert changes to ffi2 * check python3 for pypy * tiny fix * revert ffi2 for real * revert weird formatting changes * bring back missing feature * tiny error * fix py3.7 issue * add pypy3.5 6.0 to travis * remove dbg! * another tiny fix * removed some useless annotations, and fixed inlines annotations * removed `pretty_assertions` * removed pypy feature from cargo.toml * fix for Py_CompileStringFlags * tox runs word_count! * __dict__ changes are not supported for PyPy * fix 3.7 and copy comment * fix test script :flushed: * transfer ownership of strings to cpython when possible * remove cstr! macro * added missing nuls * as_bytes() -> b’’ string * symbol removed by mistake * properly shim pypy date time API, some tests are passing! * extension_module tests now not crashing! (some still skipped) * maybe travis has new pypy version? * small error on windows (build script) * fix conditional compilation * try to make tests run on travis.. * invert condition * added pytest-faulthandler to facilitate debugging * correctly name dir * use full paths * say —yes to conda * fix * syntax error * change PATH * fixed a terrible bug with PyTypeObjects in PyPy * fix PyTypeObject defs * re-enabled tests! * all tests are passing! * make the fix ad-hoc for now * removed build module * revert changes that cause an additional GC bug * prevented buggy test from failing pypy * removed unused comment * don’t run coverage on pypy * removed some erroneous symbols from function calls which are actually macros * restore py37 pyunicode missing def * use only `link_name` in PyPy specific declarations * only setup PyPy when testing against PyPy * annotation that was eaten during merge * remove change to comment by mistake + unnecessary changes to cargo.toml * xfail dates test only on pypy * changed comment to be a little more helpful * cleaned up some warnings * Update src/ffi3/ceval.rs Co-Authored-By: omerbenamram <omerbenamram@gmail.com> * @konstin PR notes * rustfmt * some documentation * if configured via env var only, default to cpython * remove extra unsafe * refer users to guide for pypy * Update guide/src/pypy.md Co-Authored-By: omerbenamram <omerbenamram@gmail.com> * Update guide/src/pypy.md Co-Authored-By: omerbenamram <omerbenamram@gmail.com> * @konstin applied patch * check that pypy at least build * search explicitly for libpypy * added note about some known unsupported features * use ld_version * export PYTHON_SYS_EXECUTABLE to `cargo build` test * inverted if * always link pypy dynamically * remove unused imports * Apply @kngwyu’s suggestion * fix tox configuration * try conda virtualenv * try to simply not install python at all inside pypy environment * setup pypy before using “python" * use system_site_packages * revert change to .travis * moved cpyext datetime documentation to module level, and revised it. * Update src/ffi/datetime.rs Co-Authored-By: omerbenamram <omerbenamram@gmail.com> * rustfmt * Update src/ffi/datetime.rs Co-Authored-By: omerbenamram <omerbenamram@gmail.com> * kept only notes that are relevant to users. * invert if * use bash and not sh
2019-04-23 11:18:42 +00:00
You can use PyO3 to write a native Python module in Rust, or to embed Python in a Rust binary. The following sections explain each of these in turn.
2015-04-19 03:29:19 +00:00
### Using Rust from Python
2018-11-26 04:31:49 +00:00
2021-11-29 18:51:21 +00:00
PyO3 can be used to generate a native Python module. The easiest way to try this out for the first time is to use [`maturin`](https://github.com/PyO3/maturin). `maturin` is a tool for building and publishing Rust-based Python packages with minimal configuration. The following steps install `maturin`, use it to generate and build a new Python package, and then launch Python to import and execute a function from the package.
2018-11-26 04:31:49 +00:00
2021-11-29 18:51:21 +00:00
First, follow the commands below to create a new directory containing a new Python `virtualenv`, and install `maturin` into the virtualenv using Python's package manager, `pip`:
```bash
# (replace string_sum with the desired package name)
$ mkdir string_sum
$ cd string_sum
$ python -m venv .env
$ source .env/bin/activate
$ pip install maturin
```
Still inside this `string_sum` directory, now run `maturin init`. This will generate the new package source. When given the choice of bindings to use, select pyo3 bindings:
```bash
$ maturin init
✔ 🤷 What kind of bindings to use? · pyo3
✨ Done! New project created string_sum
```
The most important files generated by this command are `Cargo.toml` and `lib.rs`, which will look roughly like the following:
2019-02-01 13:01:18 +00:00
**`Cargo.toml`**
2017-05-21 12:37:56 +00:00
```toml
[package]
2021-11-29 18:51:21 +00:00
name = "string_sum"
version = "0.1.0"
2019-02-01 13:01:18 +00:00
edition = "2018"
[lib]
2022-02-15 07:27:28 +00:00
# The name of the native library. This is the name which will be used in Python to import the
# library (i.e. `import string_sum`). If you change this, you must also change the name of the
# `#[pymodule]` in `src/lib.rs`.
2018-08-30 21:19:23 +00:00
name = "string_sum"
# "cdylib" is necessary to produce a shared library for Python to import from.
#
# Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
# to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
# crate-type = ["cdylib", "rlib"]
2016-12-17 20:41:14 +00:00
crate-type = ["cdylib"]
2021-11-29 18:51:21 +00:00
[dependencies]
2022-03-15 17:42:31 +00:00
pyo3 = { version = "0.16.2", features = ["extension-module"] }
```
**`src/lib.rs`**
2017-05-21 12:37:56 +00:00
```rust
2018-07-30 20:52:22 +00:00
use pyo3::prelude::*;
/// Formats the sum of two numbers as string.
2020-06-07 12:06:15 +00:00
#[pyfunction]
2018-07-30 20:52:22 +00:00
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
Ok((a + b).to_string())
}
2018-05-01 13:31:11 +00:00
/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
2020-06-07 12:06:15 +00:00
#[pymodule]
2022-03-23 07:07:28 +00:00
fn string_sum(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
Ok(())
2017-06-12 06:57:21 +00:00
}
```
2017-03-12 11:54:21 +00:00
2021-11-29 18:51:21 +00:00
Finally, run `maturin develop`. This will build the package and install it into the Python virtualenv previously created and activated. The package is then ready to be used from `python`:
2017-05-13 05:51:14 +00:00
```bash
$ maturin develop
# lots of progress output as maturin runs the compilation...
$ python
>>> import string_sum
>>> string_sum.sum_as_string(5, 20)
'25'
```
2021-11-29 18:51:21 +00:00
To make changes to the package, just edit the Rust source code and then re-run `maturin develop` to recompile.
To run this all as a single copy-and-paste, use the bash script below (replace `string_sum` in the first command with the desired package name):
```bash
mkdir string_sum && cd "$_"
python -m venv .env
source .env/bin/activate
pip install maturin
maturin init --bindings pyo3
maturin develop
```
As well as with `maturin`, it is possible to build using [`setuptools-rust`](https://github.com/PyO3/setuptools-rust) or [manually](https://pyo3.rs/latest/building_and_distribution.html#manual-builds). Both offer more flexibility than `maturin` but require more configuration to get started.
### Using Python from Rust
To embed Python into a Rust binary, you need to ensure that your Python installation contains a shared library. The following steps demonstrate how to ensure this (for Ubuntu), and then give some example code which runs an embedded Python interpreter.
2021-08-02 21:29:37 +00:00
To install the Python shared library on Ubuntu:
```bash
sudo apt install python3-dev
```
2018-07-30 20:52:22 +00:00
2021-08-02 21:29:37 +00:00
Start a new project with `cargo new` and add `pyo3` to the `Cargo.toml` like this:
2018-07-30 20:52:22 +00:00
```toml
[dependencies.pyo3]
2022-03-15 17:42:31 +00:00
version = "0.16.2"
features = ["auto-initialize"]
2018-07-30 20:52:22 +00:00
```
Example program displaying the value of `sys.version` and the current user name:
2018-07-30 20:52:22 +00:00
```rust
use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
2018-07-30 20:52:22 +00:00
2021-08-02 21:29:37 +00:00
fn main() -> PyResult<()> {
2020-07-13 21:37:40 +00:00
Python::with_gil(|py| {
2021-08-02 21:29:37 +00:00
let sys = py.import("sys")?;
let version: String = sys.getattr("version")?.extract()?;
2019-09-15 10:55:01 +00:00
2021-08-02 21:29:37 +00:00
let locals = [("os", py.import("os")?)].into_py_dict(py);
let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
let user: String = py.eval(code, None, Some(&locals))?.extract()?;
println!("Hello {}, I'm Python {}", user, version);
Ok(())
})
2018-07-30 20:52:22 +00:00
}
```
The guide has [a section](https://pyo3.rs/latest/python_from_rust.html) with lots of examples
about this topic.
2019-09-15 10:55:01 +00:00
## Tools and libraries
2021-06-26 22:37:30 +00:00
- [maturin](https://github.com/PyO3/maturin) _Build and publish crates with pyo3, rust-cpython or cffi bindings as well as rust binaries as python packages_
2021-06-26 22:37:30 +00:00
- [setuptools-rust](https://github.com/PyO3/setuptools-rust) _Setuptools plugin for Rust support_.
2021-09-18 18:48:45 +00:00
- [pyo3-built](https://github.com/PyO3/pyo3-built) _Simple macro to expose metadata obtained with the [`built`](https://crates.io/crates/built) crate as a [`PyDict`](https://docs.rs/pyo3/*/pyo3/types/struct.PyDict.html)_
2021-06-26 22:37:30 +00:00
- [rust-numpy](https://github.com/PyO3/rust-numpy) _Rust binding of NumPy C-API_
- [dict-derive](https://github.com/gperinazzo/dict-derive) _Derive FromPyObject to automatically transform Python dicts into Rust structs_
- [pyo3-log](https://github.com/vorner/pyo3-log) _Bridge from Rust to Python logging_
- [pythonize](https://github.com/davidhewitt/pythonize) _Serde serializer for converting Rust objects to JSON-compatible Python objects_
- [pyo3-asyncio](https://github.com/awestlake87/pyo3-asyncio) _Utilities for working with Python's Asyncio library and async functions_
2022-03-26 17:14:17 +00:00
- [rustimport](https://github.com/mityax/rustimport) _Directly import Rust files or crates from Python, without manual compilation step. Provides pyo3 integration by default and generates pyo3 binding code automatically._
## Examples
2018-07-30 20:52:22 +00:00
2021-06-26 22:37:30 +00:00
- [hyperjson](https://github.com/mre/hyperjson) _A hyper-fast Python module for reading/writing JSON data using Rust's serde-json_
- [html-py-ever](https://github.com/PyO3/setuptools-rust/tree/main/examples/html-py-ever) _Using [html5ever](https://github.com/servo/html5ever) through [kuchiki](https://github.com/kuchiki-rs/kuchiki) to speed up html parsing and css-selecting._
- [point-process](https://github.com/ManifoldFR/point-process-rust/tree/master/pylib) _High level API for pointprocesses as a Python library_
- [autopy](https://github.com/autopilot-rs/autopy) _A simple, cross-platform GUI automation library for Python and Rust._
2021-09-18 18:48:45 +00:00
- Contains an example of building wheels on TravisCI and appveyor using [cibuildwheel](https://github.com/pypa/cibuildwheel)
2021-06-26 22:37:30 +00:00
- [orjson](https://github.com/ijl/orjson) _Fast Python JSON library_
2021-09-18 18:48:45 +00:00
- [inline-python](https://github.com/fusion-engineering/inline-python) _Inline Python code directly in your Rust code_
2021-06-26 22:37:30 +00:00
- [Rogue-Gym](https://github.com/kngwyu/rogue-gym) _Customizable rogue-like game for AI experiments_
- Contains an example of building wheels on Azure Pipelines
- [fastuuid](https://github.com/thedrow/fastuuid/) _Python bindings to Rust's UUID library_
- [wasmer-python](https://github.com/wasmerio/wasmer-python) _Python library to run WebAssembly binaries_
- [mocpy](https://github.com/cds-astro/mocpy) _Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere_
- [tokenizers](https://github.com/huggingface/tokenizers/tree/master/bindings/python) _Python bindings to the Hugging Face tokenizers (NLP) written in Rust_
2021-09-18 18:48:45 +00:00
- [pyre](https://github.com/Project-Dream-Weaver/pyre-http) _Fast Python HTTP server written in Rust_
2021-06-26 22:37:30 +00:00
- [jsonschema-rs](https://github.com/Stranger6667/jsonschema-rs/tree/master/bindings/python) _Fast JSON Schema validation library_
- [css-inline](https://github.com/Stranger6667/css-inline/tree/master/bindings/python) _CSS inlining for Python implemented in Rust_
- [cryptography](https://github.com/pyca/cryptography/tree/main/src/rust) _Python cryptography library with some functionality in Rust_
- [polaroid](https://github.com/daggy1234/polaroid) _Hyper Fast and safe image manipulation library for Python written in Rust_
- [ormsgpack](https://github.com/aviramha/ormsgpack) _Fast Python msgpack library_
- [bed-reader](https://github.com/fastlmm/bed-reader) _Read and write the PLINK BED format, simply and efficiently_
- Shows Rayon/ndarray::parallel (including capturing errors, controlling thread num), Python types to Rust generics, Github Actions
2022-01-28 10:33:18 +00:00
- [pyheck](https://github.com/kevinheavey/pyheck) _Fast case conversion library, built by wrapping [heck](https://github.com/withoutboats/heck)_
- Quite easy to follow as there's not much code.
2018-07-30 20:52:22 +00:00
2021-09-02 20:28:14 +00:00
## Articles and other media
- [Nine Rules for Writing Python Extensions in Rust](https://towardsdatascience.com/nine-rules-for-writing-python-extensions-in-rust-d35ea3a4ec29?sk=f8d808d5f414154fdb811e4137011437) - Dec 31, 2021
- [Calling Rust from Python using PyO3](https://saidvandeklundert.net/learn/2021-11-18-calling-rust-from-python-using-pyo3/) - Nov 18, 2021
2021-09-02 20:28:14 +00:00
- [davidhewitt's 2021 talk at Rust Manchester meetup](https://www.youtube.com/watch?v=-XyWG_klSAw&t=320s) - Aug 19, 2021
- [Incrementally porting a small Python project to Rust](https://blog.waleedkhan.name/port-python-to-rust/) - Apr 29, 2021
2021-09-02 20:28:14 +00:00
- [Vortexa - Integrating Rust into Python](https://www.vortexa.com/insight/integrating-rust-into-python) - Apr 12, 2021
- [Writing and publishing a Python module in Rust](https://blog.yossarian.net/2020/08/02/Writing-and-publishing-a-python-module-in-rust) - Aug 2, 2020
2021-09-02 20:28:14 +00:00
## Contributing
Everyone is welcomed to contribute to PyO3! There are many ways to support the project, such as:
2021-06-26 22:37:30 +00:00
2021-09-18 18:48:45 +00:00
- help PyO3 users with issues on GitHub and Gitter
- improve documentation
- write features and bugfixes
- publish blogs and examples of how to use PyO3
2021-09-18 18:48:45 +00:00
Our [contributing notes](https://github.com/PyO3/pyo3/blob/main/Contributing.md) and [architecture guide](https://github.com/PyO3/pyo3/blob/main/Architecture.md) have more resources if you wish to volunteer time for PyO3 and are searching where to start.
2021-09-18 18:48:45 +00:00
If you don't have time to contribute yourself but still wish to support the project's future success, some of our maintainers have GitHub sponsorship pages:
2021-06-26 22:37:30 +00:00
- [davidhewitt](https://github.com/sponsors/davidhewitt)
2018-05-17 21:59:11 +00:00
## License
2017-05-13 05:51:14 +00:00
2021-09-18 18:48:45 +00:00
PyO3 is licensed under the [Apache-2.0 license](https://opensource.org/licenses/APACHE-2.0).
Python is licensed under the [Python License](https://docs.python.org/3/license.html).