3168: Do not apply deferred ref count updates and prevent the GIL from being acquired inside of __traverse__ implementations. r=davidhewitt a=adamreichold
Closes#2301Closes#3165
3176: Prevent dropping unsendable classes on other threads. r=davidhewitt a=adamreichold
Continuing the discussed from https://github.com/PyO3/pyo3/pull/3169#issuecomment-1556571504 and https://github.com/PyO3/pyo3/pull/3169#issuecomment-1556661723:
We already have checks in place to avoid borrowing these classes on other threads but it was still possible to send them to another thread and drop them there (while holding the GIL).
This change avoids running the `Drop` implementation in such a case even though Python will still free the underlying memory. This might leak resources owned by the object, but it avoids undefined behaviour due to access the unsendable type from another thread.
This does assume that the object was not unsafely integrated into an intrusive data structures which still point to the now freed memory. In that case, the only recourse would be to abort the process as freeing the memory is unavoidable when the tp_dealloc slot is called. (And moving it elsewhere into a new allocation would still break any existing pointers.)
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
We already have checks in place to avoid borrowing these classes on other
threads but it was still possible to send them to another thread and drop them
there (while holding the GIL).
This change avoids running the `Drop` implementation in such a case even though
Python will still free the underlying memory. This might leak resources owned by
the object, but it avoids undefined behaviour due to access the unsendable type
from another thread.
This does assume that the object was not unsafely integrated into an intrusive
data structures which still point to the now freed memory. In that case, the
only recourse would be to abort the process as freeing the memory is unavoidable
when the tp_dealloc slot is called. (And moving it elsewhere into a new
allocation would still break any existing pointers.)
3177: RFC: Use a const initializer for `GIL_COUNT` if possible r=adamreichold a=lifthrasiir
Normally [`LocalKey::try_with`](https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.try_with) needs to check for the initialization flag to lazily initialize the TLS. This behaves badly with a compiler optimization and it seems that multiple calls to `gil_is_acquired()` cannot be correctly eliminated. Rust 1.59 added a `const { ... }` initializer (a special form only allowed here for now) in `thread_local!` which removes the initialization flag, allowing those kind of optimizations.
I should note that the performance impact is probably minimal, because the check branch is extremely well predicted and the optimization is only possible when every PyO3 code that leads to `gil_is_acquired()` is inlined, a pretty uncommon situation. I couldn't demonstrate any consistent improvement or regression from my machine, which performance seems to be flaky as well. But at least we would have one less branch there. I'll leave this as an RFC so that someone else can prove or disprove that this is indeed beneficial.
Co-authored-by: Kang Seonghoon <public+git@mearie.org>
3166: RFC: Drop EnsureGIL to remove one layer of indirection from the implementation of Python::with_gil r=davidhewitt a=adamreichold
I am not sure if other people would consider this a simplification as well, but it helped me to remove one layer of indirection used by the implementation of `Python::with_gil`, especially flattening the `Option<_>` layers in `EnsureGIL` and `GILGuard::pool`. This makes it obvious that we always create `GILPool` when we actually acquire the GIL. (Of course, there still might be extra `GILPool` instances created manually.)
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3178: Fix a bad span of `_slf` for custom receivers in `#[pymethods]`. r=adamreichold a=lifthrasiir
This turned out to be a remnant of #1506. It notably resulted in a very confusing error:
```
error[E0308]: mismatched types
--> tests\test_methods.rs:1456:9
|
1456 | #[pymethods]
| ^^^^^^^^^^^^
| |
| expected `Py<Issue1506>`, found `*mut PyObject`
| arguments to this function are incorrect
...
1461 | / issue_1506!(
1462 | | #[pymethods]
1463 | | impl Issue1506 {
1464 | | fn issue_1506(
... |
1536 | | }
1537 | | );
| |_- in this macro invocation
|
= note: expected struct `pyo3::Py<Issue1506>`
found raw pointer `*mut pyo3::ffi::PyObject`
```
The actual cause is that `SelfType::receiver` is entirely ignored in this case and `_slf` refers to the original argument, but it sounds like that `TryFrom::try_from` somehow resulted in `*mut PyObject`...
Co-authored-by: Kang Seonghoon <public+git@mearie.org>
3161: add PyAny::is_exact_instance and PyAny::is_exact_instance_of r=adamreichold a=davidhewitt
Split off from #2881. I'll rebase after that merges.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2881: Rework `PyAny::is_instance_of` for performance r=adamreichold a=davidhewitt
I was talking to `@samuelcolvin` about the fastest way to identify object types (relevant e.g. for `pythonize` and also `pydantic-core`) and noticed that `PyAny::is_instance_of` is quite unoptimised because it expands to an ffi call to `PyObject_IsInstance`.
This PR proposes `PyAny::is_instance_of::<T>(obj)` is changed to be equivalent to `T::is_type_of(obj)`, plus add a sprinkling of inlining. We often have implementations such as `PyDict_Check` which can pretty much be optimised away to just checking a bit on the type object.
The accompanying benchmark to run through a bunch of object types is approx 40% faster after making this change.
For completeness, I've also added `PyAny::is_exact_instance` and `PyAny::is_exact_instance_of`, to pair with `T::is_exact_type_of`. (This could be split into a separate PR if preferred.)
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3158: Add Py::get for GIL-independent access to frozen classes. r=davidhewitt a=adamreichold
`@davidhewitt` Is this what you had in mind for #3154?
The name is an obvious candidate for bikeshedding.
Trying to write an example, I noticed that making `PyCell::get_frozen` public is most likely not useful as there is no way to safely get a `&PyCell` without acquiring the GIL first?
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3159: Add OpenDAL to examples r=adamreichold a=suyanhanx
Hi there,
We have been using PyO3 to build our project's python binding and have been extremely satisfied with it.
And we want to say thank you for providing such a wonderful tool.
Co-authored-by: suyanhanx <suyanhanx@gmail.com>
3150: Fix Clippy CI job cache to really include the platform. r=messense a=adamreichold
Doesn't really matter since we never build the same target on different platforms, but fixing it will prevent future confusion when reading it. Sorry for the extra lap. 😫
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3149: Enable rust-cache on clippy CI jobs as we do for build CI jobs. r=davidhewitt a=adamreichold
Not sure if there was a reason we did not enable this for the Clippy jobs?
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
2593: docs: mention PyBuffer r=adamreichold a=davidhewitt
Uses PEP 688 `types.Buffer` to describe `PyBuffer<T>` in the conversion tables. Will leave as draft until PEP 688 is finalised.
Closes#954
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3146: Added a few lines to document the main difference between maturin and… r=adamreichold a=why-not-try-calmer
… setuptools-rust as far as building manylinux-compliant wheels is concerned.
Co-authored-by: Adrien <mrnycticorax@gmail.com>
3029: use dynamic trampoline for all getters and setters r=adamreichold a=davidhewitt
This is an extension to the "trampoline" changes made in #2705 to re-use a single trampoline for all `#[getter]`s (and similar for all `#[setters]`). It works by setting the currently-unused `closure` member of the `ffi::PyGetSetDef` structure to point at a new `struct GetSetDefClosure` which contains function pointers to the `getter` / `setter` implementations.
A universal trampoline for all `getter`, for example, then works by reading the actual getter implementation out of the `GetSetDefClosure`.
Advantages of doing this:
- Very minimal simplification to the macro code / generated code size. It made a 4.4% reduction to `test_getter_setter` generated size, which is an exaggerated result as most code will probably have lots of bulk that isn't just the macro code.
Disadvantages:
- Additional level of complexity in the `getter` and `setter` trampolines and accompanying code.
- To keep the `GetSetDefClosure` objects alive, I've added them to the static `LazyTypeObject` inner.
- Very slight performance overhead at runtime (shouldn't be more than an additional pointer read). It's so slight I couldn't measure it.
Overall I'm happy to either merge or close this based on what reviewers think!
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2981: Remove 0.17 deprecations r=adamreichold,davidhewitt a=davidhewitt
Since #2980 starts a breaking change for 0.19, let's also clean up all 0.17's deprecations.
I've removed `Python::acquire_gil` in its own commit, as that was a reasonably chunky removal.
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
