3168: Do not apply deferred ref count updates and prevent the GIL from being acquired inside of __traverse__ implementations. r=davidhewitt a=adamreichold
Closes#2301Closes#3165
3176: Prevent dropping unsendable classes on other threads. r=davidhewitt a=adamreichold
Continuing the discussed from https://github.com/PyO3/pyo3/pull/3169#issuecomment-1556571504 and https://github.com/PyO3/pyo3/pull/3169#issuecomment-1556661723:
We already have checks in place to avoid borrowing these classes on other threads but it was still possible to send them to another thread and drop them there (while holding the GIL).
This change avoids running the `Drop` implementation in such a case even though Python will still free the underlying memory. This might leak resources owned by the object, but it avoids undefined behaviour due to access the unsendable type from another thread.
This does assume that the object was not unsafely integrated into an intrusive data structures which still point to the now freed memory. In that case, the only recourse would be to abort the process as freeing the memory is unavoidable when the tp_dealloc slot is called. (And moving it elsewhere into a new allocation would still break any existing pointers.)
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
We already have checks in place to avoid borrowing these classes on other
threads but it was still possible to send them to another thread and drop them
there (while holding the GIL).
This change avoids running the `Drop` implementation in such a case even though
Python will still free the underlying memory. This might leak resources owned by
the object, but it avoids undefined behaviour due to access the unsendable type
from another thread.
This does assume that the object was not unsafely integrated into an intrusive
data structures which still point to the now freed memory. In that case, the
only recourse would be to abort the process as freeing the memory is unavoidable
when the tp_dealloc slot is called. (And moving it elsewhere into a new
allocation would still break any existing pointers.)
3029: use dynamic trampoline for all getters and setters r=adamreichold a=davidhewitt
This is an extension to the "trampoline" changes made in #2705 to re-use a single trampoline for all `#[getter]`s (and similar for all `#[setters]`). It works by setting the currently-unused `closure` member of the `ffi::PyGetSetDef` structure to point at a new `struct GetSetDefClosure` which contains function pointers to the `getter` / `setter` implementations.
A universal trampoline for all `getter`, for example, then works by reading the actual getter implementation out of the `GetSetDefClosure`.
Advantages of doing this:
- Very minimal simplification to the macro code / generated code size. It made a 4.4% reduction to `test_getter_setter` generated size, which is an exaggerated result as most code will probably have lots of bulk that isn't just the macro code.
Disadvantages:
- Additional level of complexity in the `getter` and `setter` trampolines and accompanying code.
- To keep the `GetSetDefClosure` objects alive, I've added them to the static `LazyTypeObject` inner.
- Very slight performance overhead at runtime (shouldn't be more than an additional pointer read). It's so slight I couldn't measure it.
Overall I'm happy to either merge or close this based on what reviewers think!
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2981: Remove 0.17 deprecations r=adamreichold,davidhewitt a=davidhewitt
Since #2980 starts a breaking change for 0.19, let's also clean up all 0.17's deprecations.
I've removed `Python::acquire_gil` in its own commit, as that was a reasonably chunky removal.
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3142: Do not store return values in locals so that holders benefit from lifetime extension for temporaries. r=davidhewitt a=adamreichold
Closes#3138
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3004: Unwrap dynamic error types when inner is simple `PyErr` r=davidhewitt,adamreichold,mejrs a=BlueGlassBlock
This is the first part of suggested improvements in #2998.
This change will make bubbled `PyErr` wrapped in `eyre::Report` / `anyhow::Error` bubble up unchanged, instead of being wrapped in a `PyRuntimeError`.
Co-authored-by: BlueGlassBlock <blueglassblock@outlook.com>
2980: support `text_signature` on `#[new]` r=adamreichold a=davidhewitt
Closes#2866
This is a breaking change for 0.19.0, because it starts autogenerating `text_signature` for `#[new]`. This could affect runtime behaviour if the user is relying on the class docs at runtime for some reason.
Guide & tests all updated accordingly.
`#[pyclass(text_signature = "...")]` is deprecated by this PR, however if it's set, it will be used in preference to `#[new]`.
(The signature / `text_signature` from `#[new]` will simply be ignored in this case. I figure that when users fix their deprecation warnings by removing `#[pyclass(text_signature = "...")]` then the `#[new]` signatures will start flowing properly, and this is good enough.)
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
Implement conversion between rust_decimal::Decimal and decimal.Decimal
from Python's stdlib. The C API does not appear to be exposed on the
Python side so we need to call into it via Python.
3066: Improve default value for `None` in `text_signature` r=davidhewitt a=messense
xref #2863
3098: readme: add new pyo3 article r=adamreichold a=davidhewitt
With thanks to `@ohadravid` for the great piece!
Co-authored-by: messense <messense@icloud.com>
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3015: Implement wrapper for `PyASCIIObject.state` bitfield accesses r=davidhewitt a=decathorpe
This is a first draft of my attempt to fix#1824 "properly" by writing a C wrapper for the `PyASCIIObject.state` bitfield accesses, as proposed here: https://github.com/PyO3/pyo3/issues/1824#issuecomment-1406909504
---
The original argument for making these functions `unsafe` is still valid, though - bitfield memory layout is not guaranteed to be stable across different C compilers, as it is "implementation defined" in the C standard. However, short of having CPython upstream provide non-inlined public functions to access this bitfield, this is the next best thing, as far as I can tell.
I've removed the `#[cfg(target_endian = "little")]` attributes from all things that are un-blocked by fixing this issue on big-endian systems, except for three tests, which look like expected failures considering that they do not take bit/byte order into account (for example, when writing to the bitfield).
- `ffi::tests::ascii_object_bitfield`
- `types::string::tests::test_string_data_ucs2_invalid`
- `types::string::tests::test_string_data_ucs4_invalid`
All other tests now pass on both little-endian and big-endian systems.
---
I am aware that some parts of this PR are probably not in a state that's acceptable for merging as-is, which is why I'm filing this as a draft. Feedback about how to better integrate this change with pyo3-ffi would be great. :)
In particular, I'm unsure whether the `#include` statements in the C files are actually correct across different systems. I have only tested this on Fedora Linux so far.
I'm also open to changing the names of the C functions that are implemented in the wrapper. For now I chose the most obvious names that shouldn't cause collisions with other symbols.
Co-authored-by: Fabio Valentini <decathorpe@gmail.com>
2975: RFC: Add GILProtected synchronization primitive and use it for LazyTypeObjectInner. r=davidhewitt a=adamreichold
I would also like to use that type in rust-numpy and it seems we can avoid ~~both a manual unsafe impl and~~ a full blown mutex if we apply it to `LazyTypeObjectInner`.
One downside might be that it ties us closer to the GIL when we want to enable nogil experimentation, but on the other hand, it may also help by reifying the GIL usage. (This is currently limited to comments in unsafe code in rust-numpy for example.)
3022: Fix function name shadowing r=davidhewitt a=mejrs
Fixes https://github.com/PyO3/pyo3/issues/3017
3023: Emit a better error for bad argument names r=davidhewitt a=mejrs
This will emit a better error for code like
```rust
#[pyfunction]
fn output([a,b,c]: [u8;3]) {}
```
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
Co-authored-by: mejrs <59372212+mejrs@users.noreply.github.com>
3055: Fix compile error when trying to use static slot methods, fixes: #3039 r=davidhewitt a=willstott101
Whilst having static slot methods is of dubious value in the first place, there's little reason not to support it by bringing the different function call macros more in-line with each other.
I had no idea which test file to use so I just made a new one... `test_methods.rs` has no slots in and the test files that do use slots seem to be specific to their protocol. Let me know if you'd like the test somewhere else.
Thanks!
Co-authored-by: Will Stott <willstott101@gmail.com>
3044: Add `PyTuple::to_list` r=davidhewitt a=davidhewitt
Companion to #3043. I've included benchmarks which suggests that this method is indeed about 25% faster on my machine.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3031: mangle exported functions for PyPy r=davidhewitt a=mattip
I run HEAD of PyO3 with HEAD of some PyPy branches, and noticed that the py3.10 branch (which implements python3.10) [fails to properly build](https://github.com/pypy/binary-testing/actions/runs/4334873617/jobs/7569059852#step:6:179). The failure was a missing export:
```
/home/runner/work/binary-testing/binary-testing/pyo3/src/exceptions.rs:715: \
undefined reference to `PyExc_EncodingWarning'
collect2: error: ld returned 1 exit status
```
So I grepped around in the code for `Py_3_10` to see what new functions were added and found a few that needed PyPy-specific exports.
Co-authored-by: Matti Picus <matti.picus@gmail.com>
bors[bot]