I decided to not implement the full protocol for truth value testing [1] as it
seems confusing in the context of function arguments if basically any instance
of custom class or non-empty collection turns into `true`.
[1] https://docs.python.org/3/library/stdtypes.html#truth
Closes#3615
This simply adds a new method which uses the existence of a `PyVisit`
object as proof that the GIL is held instead of a `Python` object. This
allows `GILProtected` to be used in instances where contained Python
objects need to participate in garbage collection. Usage in this
situation should be valid since no Python calls are made and this does
not provide any additional mechanism for accessing a `Python` object.
I chose to make the dependency mandatory instead of optional as portable-atomic
itself just forwards to the native atomics when they are available so making
that choice part of our build system is not really necessary. Personally, I was
unable to perceive any noticeable compile-time hit from adding it.
In pyca/cryptography this function is the #1 source of lines of generated LLVM IR, because it is duplicated 42x (and growing!). By rewriting it so most of the logic is monomorphic, we reduce the generated LLVM IR for this function by 4x.
Python code doesn't like to run without `__builtins__`, so adding them
if missing seems to be a good idea. Also that's what CPython >3.10 does.
See https://github.com/python/cpython/pull/24564Resolves#3370
When the Python iterator backing `PyIterator` has a `__length_hint__` special
method, we can use this as a lower bound for Rust's `Iterator::size_hint` to
e.g. support pre-allocation of collections.
This is implemented using `PyObject_LengthHint` which is not available in the
stable ABI and hence so is `Iterator::size_hint`. This should be fine since this
is an optimization in any case and the stable ABI is expected to have slightly
worse performance overall.
Most native types have static type objects which just need to be turned into a
pointer using addr_of_mut!, but sometimes like for exceptions we do call a
function which lazily initializes a type object.
This makes the pyobject_native_type_info! macro expect callable and passes it
the GIL token so that we do not need to call Python::assume_gil_acquired. The
case of static type objects is handled by the
pyobject_native_static_type_object! helper macro.
3209: Remove the conditional compilation flags which are made redundant by the MSRV bump r=davidhewitt a=adamreichold
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3203: support ordering magic methods for `#[pyclass]` r=adamreichold a=davidhewitt
Closes#2089
This adds `__lt__`, `__le__`, `__eq__`, `__ne__`, `__gt__`, and `__ge__` as per the Python implementations of what we call `__richcmp__`.
There's a UI test confirming that the user cannot implement split forms and `__richcmp__` simultaneously.
There's also a benchmark comparing implementing these split methods against using `__richcmp__`. I couldn't see a meaningful performance difference, so I'm tempted to deprecate `__richcmp__`, given that's not a magic method which exists in Python. Potentially we can provide options such as the opt-in `#[pyclass(eq, ord)]` to avoid boilerplate for people who don't want to implement six different methods.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3198: Add abi3 + num_bigint conversion r=davidhewitt a=youknowone
<!--
Please consider adding the following to your pull request:
- an entry for this PR in newsfragments - see [https://pyo3.rs/main/contributing.html#documenting-changes]
- docs to all new functions and / or detail in the guide
- tests for all new or changed functions
PyO3's CI pipeline will check your pull request. To run its tests
locally, you can run ```cargo xtask ci```. See its documentation
[here](https://github.com/PyO3/pyo3/tree/main/xtask#readme).
-->
Fix#3196
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
Python classes that were not `complex` but implemented the `__complex__`
magic would have that method called via `PyComplex_AsCComplex` when
running against the full API, but the limited-API version
`PyComplex_RealAsDouble` does not attempt this conversion. If the input
object is not already complex, we can call the magic before proceeding.
`PyAny::lookup_special` is an approximate equivalent to the CPython
internal `_PyObject_LookupSpecial`, which is used to resolve lookups of
"magic" methods. These are only looked up from the type, and skip the
instance dictionary during the lookup. Despite this, they are still
required to resolve the descriptor protocol.
Many magic methods have slots on the `PyTypeObject` or respective
subobjects, but these are not necessarily available when targeting the
limited API or PyPy. In these cases, the requisite logic can be worked
around using safe but likely slower APIs.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Fix up lookup-special
3168: Do not apply deferred ref count updates and prevent the GIL from being acquired inside of __traverse__ implementations. r=davidhewitt a=adamreichold
Closes#2301Closes#3165
3176: Prevent dropping unsendable classes on other threads. r=davidhewitt a=adamreichold
Continuing the discussed from https://github.com/PyO3/pyo3/pull/3169#issuecomment-1556571504 and https://github.com/PyO3/pyo3/pull/3169#issuecomment-1556661723:
We already have checks in place to avoid borrowing these classes on other threads but it was still possible to send them to another thread and drop them there (while holding the GIL).
This change avoids running the `Drop` implementation in such a case even though Python will still free the underlying memory. This might leak resources owned by the object, but it avoids undefined behaviour due to access the unsendable type from another thread.
This does assume that the object was not unsafely integrated into an intrusive data structures which still point to the now freed memory. In that case, the only recourse would be to abort the process as freeing the memory is unavoidable when the tp_dealloc slot is called. (And moving it elsewhere into a new allocation would still break any existing pointers.)
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
We already have checks in place to avoid borrowing these classes on other
threads but it was still possible to send them to another thread and drop them
there (while holding the GIL).
This change avoids running the `Drop` implementation in such a case even though
Python will still free the underlying memory. This might leak resources owned by
the object, but it avoids undefined behaviour due to access the unsendable type
from another thread.
This does assume that the object was not unsafely integrated into an intrusive
data structures which still point to the now freed memory. In that case, the
only recourse would be to abort the process as freeing the memory is unavoidable
when the tp_dealloc slot is called. (And moving it elsewhere into a new
allocation would still break any existing pointers.)
3177: RFC: Use a const initializer for `GIL_COUNT` if possible r=adamreichold a=lifthrasiir
Normally [`LocalKey::try_with`](https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.try_with) needs to check for the initialization flag to lazily initialize the TLS. This behaves badly with a compiler optimization and it seems that multiple calls to `gil_is_acquired()` cannot be correctly eliminated. Rust 1.59 added a `const { ... }` initializer (a special form only allowed here for now) in `thread_local!` which removes the initialization flag, allowing those kind of optimizations.
I should note that the performance impact is probably minimal, because the check branch is extremely well predicted and the optimization is only possible when every PyO3 code that leads to `gil_is_acquired()` is inlined, a pretty uncommon situation. I couldn't demonstrate any consistent improvement or regression from my machine, which performance seems to be flaky as well. But at least we would have one less branch there. I'll leave this as an RFC so that someone else can prove or disprove that this is indeed beneficial.
Co-authored-by: Kang Seonghoon <public+git@mearie.org>
2593: docs: mention PyBuffer r=adamreichold a=davidhewitt
Uses PEP 688 `types.Buffer` to describe `PyBuffer<T>` in the conversion tables. Will leave as draft until PEP 688 is finalised.
Closes#954
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3029: use dynamic trampoline for all getters and setters r=adamreichold a=davidhewitt
This is an extension to the "trampoline" changes made in #2705 to re-use a single trampoline for all `#[getter]`s (and similar for all `#[setters]`). It works by setting the currently-unused `closure` member of the `ffi::PyGetSetDef` structure to point at a new `struct GetSetDefClosure` which contains function pointers to the `getter` / `setter` implementations.
A universal trampoline for all `getter`, for example, then works by reading the actual getter implementation out of the `GetSetDefClosure`.
Advantages of doing this:
- Very minimal simplification to the macro code / generated code size. It made a 4.4% reduction to `test_getter_setter` generated size, which is an exaggerated result as most code will probably have lots of bulk that isn't just the macro code.
Disadvantages:
- Additional level of complexity in the `getter` and `setter` trampolines and accompanying code.
- To keep the `GetSetDefClosure` objects alive, I've added them to the static `LazyTypeObject` inner.
- Very slight performance overhead at runtime (shouldn't be more than an additional pointer read). It's so slight I couldn't measure it.
Overall I'm happy to either merge or close this based on what reviewers think!
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3004: Unwrap dynamic error types when inner is simple `PyErr` r=davidhewitt,adamreichold,mejrs a=BlueGlassBlock
This is the first part of suggested improvements in #2998.
This change will make bubbled `PyErr` wrapped in `eyre::Report` / `anyhow::Error` bubble up unchanged, instead of being wrapped in a `PyRuntimeError`.
Co-authored-by: BlueGlassBlock <blueglassblock@outlook.com>
2980: support `text_signature` on `#[new]` r=adamreichold a=davidhewitt
Closes#2866
This is a breaking change for 0.19.0, because it starts autogenerating `text_signature` for `#[new]`. This could affect runtime behaviour if the user is relying on the class docs at runtime for some reason.
Guide & tests all updated accordingly.
`#[pyclass(text_signature = "...")]` is deprecated by this PR, however if it's set, it will be used in preference to `#[new]`.
(The signature / `text_signature` from `#[new]` will simply be ignored in this case. I figure that when users fix their deprecation warnings by removing `#[pyclass(text_signature = "...")]` then the `#[new]` signatures will start flowing properly, and this is good enough.)
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
Implement conversion between rust_decimal::Decimal and decimal.Decimal
from Python's stdlib. The C API does not appear to be exposed on the
Python side so we need to call into it via Python.
3015: Implement wrapper for `PyASCIIObject.state` bitfield accesses r=davidhewitt a=decathorpe
This is a first draft of my attempt to fix#1824 "properly" by writing a C wrapper for the `PyASCIIObject.state` bitfield accesses, as proposed here: https://github.com/PyO3/pyo3/issues/1824#issuecomment-1406909504
---
The original argument for making these functions `unsafe` is still valid, though - bitfield memory layout is not guaranteed to be stable across different C compilers, as it is "implementation defined" in the C standard. However, short of having CPython upstream provide non-inlined public functions to access this bitfield, this is the next best thing, as far as I can tell.
I've removed the `#[cfg(target_endian = "little")]` attributes from all things that are un-blocked by fixing this issue on big-endian systems, except for three tests, which look like expected failures considering that they do not take bit/byte order into account (for example, when writing to the bitfield).
- `ffi::tests::ascii_object_bitfield`
- `types::string::tests::test_string_data_ucs2_invalid`
- `types::string::tests::test_string_data_ucs4_invalid`
All other tests now pass on both little-endian and big-endian systems.
---
I am aware that some parts of this PR are probably not in a state that's acceptable for merging as-is, which is why I'm filing this as a draft. Feedback about how to better integrate this change with pyo3-ffi would be great. :)
In particular, I'm unsure whether the `#include` statements in the C files are actually correct across different systems. I have only tested this on Fedora Linux so far.
I'm also open to changing the names of the C functions that are implemented in the wrapper. For now I chose the most obvious names that shouldn't cause collisions with other symbols.
Co-authored-by: Fabio Valentini <decathorpe@gmail.com>
2975: RFC: Add GILProtected synchronization primitive and use it for LazyTypeObjectInner. r=davidhewitt a=adamreichold
I would also like to use that type in rust-numpy and it seems we can avoid ~~both a manual unsafe impl and~~ a full blown mutex if we apply it to `LazyTypeObjectInner`.
One downside might be that it ties us closer to the GIL when we want to enable nogil experimentation, but on the other hand, it may also help by reifying the GIL usage. (This is currently limited to comments in unsafe code in rust-numpy for example.)
3022: Fix function name shadowing r=davidhewitt a=mejrs
Fixes https://github.com/PyO3/pyo3/issues/3017
3023: Emit a better error for bad argument names r=davidhewitt a=mejrs
This will emit a better error for code like
```rust
#[pyfunction]
fn output([a,b,c]: [u8;3]) {}
```
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
Co-authored-by: mejrs <59372212+mejrs@users.noreply.github.com>
3044: Add `PyTuple::to_list` r=davidhewitt a=davidhewitt
Companion to #3043. I've included benchmarks which suggests that this method is indeed about 25% faster on my machine.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3014: feat: add #[pyo3(get, set)] for Cell r=davidhewitt a=AntoineRR
This fixes#2659.
The types for which `#[pyo3(get, set)]` should now work are `Cell`, `Arc` and `Box`.
There is one issue regarding `Box`, the implementation of `FromPyObject` conflicts with another one. I could not find what the issue was, especially since the other implementations for `Arc` and `Cell` work as expected. The related code and test has been commented out for now. Maybe someone could help me fix this issue if I don't figure it out myself? There is also the possibility to remove the implementation for `Box` of course.
Co-authored-by: Antoine Romero-Romero <ant.romero2@orange.fr>
2899: RFC: Provide a special purpose FromPyObject impl for byte slices r=davidhewitt a=adamreichold
This enables efficiently and safely getting a byte slice from either bytes or byte arrays.
The main issue I see here is discoverability, i.e. should this be mention in the docs of `PyBytes` and `PyByteArray` or in the guide?
It is also not completely clear whether this really _fixes_ the issue.
Closes#2888
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
2979: allow `create_exception!` to place the exception in a `dotted.module` r=adamreichold a=davidhewitt
Closes#2946
Credit fully to `@BlueGlassBlock`
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2952: Fix allow_threads segfault r=davidhewitt a=OliverBalfour
Please see the corresponding issue **#2951** for details. This PR adds the failing test from the issue and then a fix for it. The fix simply calls `ReferencePool::update_counts` at the end of `allow_threads` to ensure objects aren't accidentally deleted too soon.
Co-authored-by: Oliver Balfour <oliver.leo.balfour@gmail.com>
2947: change PyModule::add_class to return an error if class creation fails r=adamreichold a=davidhewitt
Related to #2942
At the moment there are panics deep in the `#[pyclass]` machinery when initialising the type fails. This PR adjusts a number of these functions to return `PyResult` instead, so that we can handle the error more appropriately further down the pipeline.
For example, take the following snippet:
```rust
#[pyclass(extends = PyBool)]
struct ExtendsBool;
#[pymodule]
fn pyo3_scratch(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_class::<ExtendsBool>()?;
Ok(())
}
```
Currently, importing this module will fail with a panic:
```
TypeError: type 'bool' is not an acceptable base type
thread '<unnamed>' panicked at 'An error occurred while initializing class ExtendsBool', /Users/david/Dev/pyo3/src/pyclass.rs:412:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/david/.virtualenvs/pyo3/lib/python3.10/site-packages/pyo3_scratch/__init__.py", line 1, in <module>
from .pyo3_scratch import *
pyo3_runtime.PanicException: An error occurred while initializing class ExtendsBool
```
After this PR, this import still fails, but with a slightly cleaner, more Pythonic error:
```
TypeError: type 'bool' is not an acceptable base type
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/david/.virtualenvs/pyo3/lib/python3.10/site-packages/pyo3_scratch/__init__.py", line 1, in <module>
from .pyo3_scratch import *
RuntimeError: An error occurred while initializing class ExtendsBool
```
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
2944: optimize sequence conversion for list and tuple r=adamreichold a=davidhewitt
closes#2943
Avoid using `PyObject_IsInstance` for checking if lists or tuples are sequences, as we know they are always sequences.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2914: correct ffi definition of PyIter_Check r=davidhewitt a=davidhewitt
Closes#2913
It looks like what is happening is that PyO3 was relying on an outdated macro form of `PyIter_Check` which is now a CPython implementation detail, which would explain why it was behaving inconsistently on different platforms (likely due to differences in linkers / implementations).
The test I've pushed succeeds, but fails to compile due to a hygiene bug! I'm done for tonight so I'll take a look at that soon and then rebase this after.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2912: Add `PyDict.update()` and `PyDict.update_if_missing()` r=davidhewitt a=samuelcolvin
Fix#2910
Note, I'd also be happy to remove the `override_` argument from merge and perhaps rename it to `update_missing` or similar to give a cleaner API. LMK what you think.
Please consider adding the following to your pull request:
- [x] an entry for this PR in newsfragments
- [x] docs to all new functions and / or detail in the guide
- [x] tests for all new or changed functions
Co-authored-by: Samuel Colvin <s@muelcolvin.com>
2923: hygiene: fix `#[pymethods(crate = "...")]` r=davidhewitt a=davidhewitt
Got to the bottom of the hygiene issue in test of #2914
Turns out that `#[pymethods] #[pyo3(crate = "...")]` works, but `#[pymethods(crate = "...")]` was ignoring the argument.
Added a tweak to fix this and a snippet in the hygiene test (which fails on `main`).
2924: remove unneeded into_iter calls r=davidhewitt a=davidhewitt
Clippy complaining about these to me this morning locally.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2904: refactor docstring generation code r=mejrs a=davidhewitt
As a first step towards #2866 this is a tweak to the macro code which generates Python docstrings. This PR refactors the behaviour so that instead of always creating a `concat!` expression to generate a nul-terminated string, this will only happen if a Rust 1.54+ macro doc is present (e.g. `#[doc = include_str!(...)]`).
The default case now just collects all the `#[doc]` attributes into a single string.
This should make it easier to factor out the `text_signature` formatting, and avoids wasting compile time invoking the `concat!` macro when not necessary.
2921: Check to see if object is `None` before traversing r=davidhewitt a=neachdainn
Closes#2915
When using the C API directly, the intended way to call `visitproc` is via the `Py_VISIT` macro, which checks to see that the provided pointer is not null before passing it along to `visitproc`. Because PyO3 isn't using the macro, it needs to manually check that the pointer isn't null. Without this check, calling `visit.call(&obj)` where `let obj = None;` will segfault.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Nate Kent <nate@nkent.net>
Closes#2915
When using the C API directly, the intended way to call `visitproc` is
via the `Py_VISIT` macro, which checks to see that the provided pointer
is not null before passing it along to `visitproc`. Because PyO3 isn't
using the macro, it needs to manually check that the pointer isn't null.
Without this check, calling `visit.call(&obj)` where `let obj = None;`
will segfault.
2893: rename `wrap_pyfunction` impl to `wrap_pyfunction_impl` r=messense a=davidhewitt
I mistyped the other day and wrote `wrap_pyfunction(f, py)` without the `!` to invoke the macro, and `rust-analyzer` imported the hidden symbol `pyo3::impl_::pyfunction::wrap_pyfunction`. Took me a moment to realise the compilation failure was because I'd forgotten the `!`.
This PR just renames that hidden symbol to `pyo3::impl_::pyfunction::wrap_pyfunction_impl` so it doesn't clash with the user-facing macro.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2882: inspect: gate behind `experimental-inspect` feature r=davidhewitt a=davidhewitt
This is the last thing I want to do before preparing 0.18 release.
The `pyo3::inspect` functionality looks useful as a first step towards #2454. However, we don't actually make use of this anywhere within PyO3 yet (we could probably use it for better error messages). I think we also have open questions about the traits which I'd like to resolve before committing to these additional APIs. (For example, this PR adds `IntoPy::type_output`, which seems potentially misplaced to me, the `type_output` function probably wants to be on a non-generic trait e.g. `ToPyObject` or maybe #2316.)
As such, I propose putting these APIs behind an `experimental-inspect` feature gate for now, and invite users who find them useful to contribute a finished-off design.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2796: Forward cfgs on pyclass fields to the method defs r=davidhewitt a=mejrs
With this and the cfg_attr PR, I don't need cfg_eval at all anymore :)
- [x] needs some more tests
Co-authored-by: mejrs <>
2843: remove functionality deprecated in 0.16 r=davidhewitt a=davidhewitt
Simple cleanup to remove all functionality marked deprecated in the 0.16 releases.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2842: Stop leaking in `new_closure` r=adamreichold a=davidhewitt
This is a rebase of #2690 which simplifies the `MaybeLeaked` abstraction from that PR with just `Cow<'static, CStr>`.
This enabled me to annotate with `FIXME` all the places where we still leak; I wonder if we could potentially use `GILOnceCell` in future and statics to avoid those. As those callsities are in `#[pyclass]` and `#[pyfunction]` these are effectively in statics anyway, but it would be nice to tidy up.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
