Rust bindings for the Python interpreter
Find a file
David Hewitt b441e0b4ab
Merge pull request #3295 from davidhewitt/pypy-pystate
add PyState_*Module definitions for PyPy
2023-07-04 08:29:31 +00:00
.cargo Drop the xtask helper as it is superseded by Nox. 2023-06-05 17:41:41 +02:00
.github ci: run valgrind and careful with 'CI-build-full' label 2023-07-03 21:20:49 +01:00
.netlify add http headers to help search engines index the docs 2023-06-21 23:10:09 +01:00
assets Fix CI 2022-05-03 11:36:24 +02:00
benches support ordering magic methods for #[pyclass] 2023-06-04 13:58:18 +01:00
emscripten Drop PYPRERELEASE variable from Emscripten Makefile as we use a proper release now. 2022-11-07 08:31:47 +01:00
examples release: 0.19.1 2023-07-03 16:18:24 +01:00
guide add some missing type conversions to the guide 2023-07-03 21:36:17 +01:00
newsfragments add PyState_*Module definitions for PyPy 2023-07-04 08:57:50 +01:00
pyo3-build-config release: 0.19.1 2023-07-03 16:18:24 +01:00
pyo3-ffi add PyState_*Module definitions for PyPy 2023-07-04 08:57:50 +01:00
pyo3-ffi-check move pyo3-ffi-check into its own workspace 2023-06-27 22:54:10 +01:00
pyo3-macros release: 0.19.1 2023-07-03 16:18:24 +01:00
pyo3-macros-backend release: 0.19.1 2023-07-03 16:18:24 +01:00
pytests Merge pull request #3240 from davidhewitt/maturin-versions 2023-06-14 08:24:31 +00:00
src Allow #[new] to return existing instances 2023-07-02 19:57:53 -04:00
tests Allow #[new] to return existing instances 2023-07-02 19:57:53 -04:00
.gitignore guide: deployment to netlify 2022-08-19 12:46:56 +01:00
.python-version unpin 3.11 ci jobs 2023-02-27 22:25:07 +00:00
.towncrier.template.md use towncrier to generate CHANGELOG 2022-09-08 07:58:53 +01:00
Architecture.md pyproto: remove deprecated feature 2022-09-06 08:38:44 +01:00
build.rs Enable v2 resolver on the workspace to enable dropping PYO3_CI workaround. 2023-06-05 18:54:35 +02:00
Cargo.toml release: 0.19.1 2023-07-03 16:18:24 +01:00
CHANGELOG.md release: 0.19.1 2023-07-03 16:18:24 +01:00
CITATION.cff add CITATION.cff 2022-10-04 19:33:00 +01:00
Code-of-Conduct.md
codecov.yml Fix IntoPyCallbackOutput paper cuts (#2326) 2022-05-09 18:15:43 +01:00
Contributing.md Drop the xtask helper as it is superseded by Nox. 2023-06-05 17:41:41 +02:00
LICENSE
netlify.toml guide: deployment to netlify 2022-08-19 12:46:56 +01:00
noxfile.py Merge pull request #3278 from davidhewitt/ffi-check-ci 2023-06-28 08:10:54 +00:00
pyproject.toml release: 0.19.1 2023-07-03 16:18:24 +01:00
README.md release: 0.19.1 2023-07-03 16:18:24 +01:00
Releasing.md release: 0.18.0 2023-01-17 19:04:30 +00:00
rust-toolchain.toml

PyO3

actions status benchmark codecov crates.io minimum rustc 1.56 dev chat contributing notes

Rust bindings for Python, including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.

Usage

PyO3 supports the following software versions:

  • Python 3.7 and up (CPython and PyPy)
  • Rust 1.56 and up

You can use PyO3 to write a native Python module in Rust, or to embed Python in a Rust binary. The following sections explain each of these in turn.

Using Rust from Python

PyO3 can be used to generate a native Python module. The easiest way to try this out for the first time is to use maturin. maturin is a tool for building and publishing Rust-based Python packages with minimal configuration. The following steps install maturin, use it to generate and build a new Python package, and then launch Python to import and execute a function from the package.

First, follow the commands below to create a new directory containing a new Python virtualenv, and install maturin into the virtualenv using Python's package manager, pip:

# (replace string_sum with the desired package name)
$ mkdir string_sum
$ cd string_sum
$ python -m venv .env
$ source .env/bin/activate
$ pip install maturin

Still inside this string_sum directory, now run maturin init. This will generate the new package source. When given the choice of bindings to use, select pyo3 bindings:

$ maturin init
✔ 🤷 What kind of bindings to use? · pyo3
  ✨ Done! New project created string_sum

The most important files generated by this command are Cargo.toml and lib.rs, which will look roughly like the following:

Cargo.toml

[package]
name = "string_sum"
version = "0.1.0"
edition = "2021"

[lib]
# The name of the native library. This is the name which will be used in Python to import the
# library (i.e. `import string_sum`). If you change this, you must also change the name of the
# `#[pymodule]` in `src/lib.rs`.
name = "string_sum"
# "cdylib" is necessary to produce a shared library for Python to import from.
#
# Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
# to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
# crate-type = ["cdylib", "rlib"]
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.19.1", features = ["extension-module"] }

src/lib.rs

use pyo3::prelude::*;

/// Formats the sum of two numbers as string.
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Ok((a + b).to_string())
}

/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn string_sum(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
    Ok(())
}

Finally, run maturin develop. This will build the package and install it into the Python virtualenv previously created and activated. The package is then ready to be used from python:

$ maturin develop
# lots of progress output as maturin runs the compilation...
$ python
>>> import string_sum
>>> string_sum.sum_as_string(5, 20)
'25'

To make changes to the package, just edit the Rust source code and then re-run maturin develop to recompile.

To run this all as a single copy-and-paste, use the bash script below (replace string_sum in the first command with the desired package name):

mkdir string_sum && cd "$_"
python -m venv .env
source .env/bin/activate
pip install maturin
maturin init --bindings pyo3
maturin develop

If you want to be able to run cargo test or use this project in a Cargo workspace and are running into linker issues, there are some workarounds in the FAQ.

As well as with maturin, it is possible to build using setuptools-rust or manually. Both offer more flexibility than maturin but require more configuration to get started.

Using Python from Rust

To embed Python into a Rust binary, you need to ensure that your Python installation contains a shared library. The following steps demonstrate how to ensure this (for Ubuntu), and then give some example code which runs an embedded Python interpreter.

To install the Python shared library on Ubuntu:

sudo apt install python3-dev

To install the Python shared library on RPM based distributions (e.g. Fedora, Red Hat, SuSE), install the python3-devel package.

Start a new project with cargo new and add pyo3 to the Cargo.toml like this:

[dependencies.pyo3]
version = "0.19.1"
features = ["auto-initialize"]

Example program displaying the value of sys.version and the current user name:

use pyo3::prelude::*;
use pyo3::types::IntoPyDict;

fn main() -> PyResult<()> {
    Python::with_gil(|py| {
        let sys = py.import("sys")?;
        let version: String = sys.getattr("version")?.extract()?;

        let locals = [("os", py.import("os")?)].into_py_dict(py);
        let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
        let user: String = py.eval(code, None, Some(&locals))?.extract()?;

        println!("Hello {}, I'm Python {}", user, version);
        Ok(())
    })
}

The guide has a section with lots of examples about this topic.

Tools and libraries

  • maturin Build and publish crates with pyo3, rust-cpython or cffi bindings as well as rust binaries as python packages
  • setuptools-rust Setuptools plugin for Rust support.
  • pyo3-built Simple macro to expose metadata obtained with the built crate as a PyDict
  • rust-numpy Rust binding of NumPy C-API
  • dict-derive Derive FromPyObject to automatically transform Python dicts into Rust structs
  • pyo3-log Bridge from Rust to Python logging
  • pythonize Serde serializer for converting Rust objects to JSON-compatible Python objects
  • pyo3-asyncio Utilities for working with Python's Asyncio library and async functions
  • rustimport Directly import Rust files or crates from Python, without manual compilation step. Provides pyo3 integration by default and generates pyo3 binding code automatically.

Examples

  • autopy A simple, cross-platform GUI automation library for Python and Rust.
    • Contains an example of building wheels on TravisCI and appveyor using cibuildwheel
  • ballista-python A Python library that binds to Apache Arrow distributed query engine Ballista.
  • bed-reader Read and write the PLINK BED format, simply and efficiently.
    • Shows Rayon/ndarray::parallel (including capturing errors, controlling thread num), Python types to Rust generics, Github Actions
  • cryptography Python cryptography library with some functionality in Rust.
  • css-inline CSS inlining for Python implemented in Rust.
  • datafusion-python A Python library that binds to Apache Arrow in-memory query engine DataFusion.
  • deltalake-python Native Delta Lake Python binding based on delta-rs with Pandas integration.
  • fastbloom A fast bloom filter | counting bloom filter implemented by Rust for Rust and Python!
  • fastuuid Python bindings to Rust's UUID library.
  • feos Lightning fast thermodynamic modeling in Rust with fully developed Python interface.
  • forust A lightweight gradient boosted decision tree library written in Rust.
  • html-py-ever Using html5ever through kuchiki to speed up html parsing and css-selecting.
  • hyperjson A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.
  • inline-python Inline Python code directly in your Rust code.
  • jsonschema-rs Fast JSON Schema validation library.
  • mocpy Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere.
  • opendal A data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.
  • orjson Fast Python JSON library.
  • ormsgpack Fast Python msgpack library.
  • point-process High level API for pointprocesses as a Python library.
  • polaroid Hyper Fast and safe image manipulation library for Python written in Rust.
  • polars Fast multi-threaded DataFrame library in Rust | Python | Node.js.
  • pydantic-core Core validation logic for pydantic written in Rust.
  • pyheck Fast case conversion library, built by wrapping heck.
    • Quite easy to follow as there's not much code.
  • pyre Fast Python HTTP server written in Rust.
  • ril-py A performant and high-level image processing library for Python written in Rust.
  • river Online machine learning in python, the computationally heavy statistics algorithms are implemented in Rust.
  • rust-python-coverage Example PyO3 project with automated test coverage for Rust and Python.
  • tiktoken A fast BPE tokeniser for use with OpenAI's models.
  • tokenizers Python bindings to the Hugging Face tokenizers (NLP) written in Rust.
  • wasmer-python Python library to run WebAssembly binaries.

Articles and other media

Contributing

Everyone is welcomed to contribute to PyO3! There are many ways to support the project, such as:

  • help PyO3 users with issues on GitHub and Gitter
  • improve documentation
  • write features and bugfixes
  • publish blogs and examples of how to use PyO3

Our contributing notes and architecture guide have more resources if you wish to volunteer time for PyO3 and are searching where to start.

If you don't have time to contribute yourself but still wish to support the project's future success, some of our maintainers have GitHub sponsorship pages:

License

PyO3 is licensed under the Apache-2.0 license. (The PyO3 project is in the process of collecting permission from past contributors to additionally license under the MIT license, see #3108. Once complete, PyO3 will additionally be licensed under the MIT license, the same as the Rust language itself is both Apache and MIT licensed.)

Python is licensed under the Python License.

Deploys by Netlify