`PyAny::lookup_special` is an approximate equivalent to the CPython
internal `_PyObject_LookupSpecial`, which is used to resolve lookups of
"magic" methods. These are only looked up from the type, and skip the
instance dictionary during the lookup. Despite this, they are still
required to resolve the descriptor protocol.
Many magic methods have slots on the `PyTypeObject` or respective
subobjects, but these are not necessarily available when targeting the
limited API or PyPy. In these cases, the requisite logic can be worked
around using safe but likely slower APIs.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Fix up lookup-special
3187: release: 0.19.0 r=davidhewitt a=davidhewitt
As per #3154 I think we're ready to push 0.19.
I'll put this live on Tuesday evening unless I hear a reason to postpone.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3188: Verify that macros do work without any imports r=adamreichold a=lifthrasiir
Currently virtually all (positive) tests import `pyo3::prelude::*`, making it hard to detect a certain class of bugs. This PR adds an explicit test that never imports from `pyo3` to fix this.
Also this fixes a minor bug from #3157 which didn't work without `use pyo3::types::PyType;`. I think this should be a part of 0.19.0 (#3187), so no additional changelog would be required (as this feature is new in 0.19.0).
Co-authored-by: Kang Seonghoon <public+git@mearie.org>
3183: Add benchmark showing that extract::<i64> is costly due to PyNumber_Index trying hard. r=davidhewitt a=adamreichold
The benchmarks from #3182 to keep around for later and to play around with.
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3171: RFC: Simplify version handling of UI tests. r=davidhewitt a=adamreichold
Due to checking in error outputs these tests only work reliably on stable and not on intermediate version between our MSRV (currently 1.48) and the current stable version.
Hence this simplifies things to run only MSRV-compatible tests for the MSRV builds, anything else for stable builds except for those tests which require the nightly feature, i.e. the `Ungil` being distinct from the `Send` trait.
Finally, `not_send3` is disabled when using the nightly feature since while `Rc` is not send, it also not GIL-bound and hence can be passed into `allow_threads` as it does not actually spawn a new thread.
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
Due to checking in error outputs these tests only work reliably on stable and
not on intermediate version between our MSRV (currently 1.48) and the current
stable version.
Hence this simplifies things to run only MSRV-compatible tests for the MSRV
builds, anything else for stable builds except for those tests which require the
nightly feature, i.e. the `Ungil` being distinct from the `Send` trait.
Finally, `not_send3` is disabled when using the nightly feature since while `Rc`
is not send, it also not GIL-bound and hence can be passed into `allow_threads`
as it does not actually spawn a new thread.
3184: Additional tests for #3168 r=adamreichold a=adamreichold
These were a part of tests `@lifthrasiir` was preparing for https://github.com/PyO3/pyo3/issues/3165, and I believe it's worthy to add them (any single of them fails in the current main branch).
Co-authored-by: Kang Seonghoon <public+git@mearie.org>
3168: Do not apply deferred ref count updates and prevent the GIL from being acquired inside of __traverse__ implementations. r=davidhewitt a=adamreichold
Closes#2301Closes#3165
3176: Prevent dropping unsendable classes on other threads. r=davidhewitt a=adamreichold
Continuing the discussed from https://github.com/PyO3/pyo3/pull/3169#issuecomment-1556571504 and https://github.com/PyO3/pyo3/pull/3169#issuecomment-1556661723:
We already have checks in place to avoid borrowing these classes on other threads but it was still possible to send them to another thread and drop them there (while holding the GIL).
This change avoids running the `Drop` implementation in such a case even though Python will still free the underlying memory. This might leak resources owned by the object, but it avoids undefined behaviour due to access the unsendable type from another thread.
This does assume that the object was not unsafely integrated into an intrusive data structures which still point to the now freed memory. In that case, the only recourse would be to abort the process as freeing the memory is unavoidable when the tp_dealloc slot is called. (And moving it elsewhere into a new allocation would still break any existing pointers.)
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
We already have checks in place to avoid borrowing these classes on other
threads but it was still possible to send them to another thread and drop them
there (while holding the GIL).
This change avoids running the `Drop` implementation in such a case even though
Python will still free the underlying memory. This might leak resources owned by
the object, but it avoids undefined behaviour due to access the unsendable type
from another thread.
This does assume that the object was not unsafely integrated into an intrusive
data structures which still point to the now freed memory. In that case, the
only recourse would be to abort the process as freeing the memory is unavoidable
when the tp_dealloc slot is called. (And moving it elsewhere into a new
allocation would still break any existing pointers.)
3177: RFC: Use a const initializer for `GIL_COUNT` if possible r=adamreichold a=lifthrasiir
Normally [`LocalKey::try_with`](https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.try_with) needs to check for the initialization flag to lazily initialize the TLS. This behaves badly with a compiler optimization and it seems that multiple calls to `gil_is_acquired()` cannot be correctly eliminated. Rust 1.59 added a `const { ... }` initializer (a special form only allowed here for now) in `thread_local!` which removes the initialization flag, allowing those kind of optimizations.
I should note that the performance impact is probably minimal, because the check branch is extremely well predicted and the optimization is only possible when every PyO3 code that leads to `gil_is_acquired()` is inlined, a pretty uncommon situation. I couldn't demonstrate any consistent improvement or regression from my machine, which performance seems to be flaky as well. But at least we would have one less branch there. I'll leave this as an RFC so that someone else can prove or disprove that this is indeed beneficial.
Co-authored-by: Kang Seonghoon <public+git@mearie.org>
3166: RFC: Drop EnsureGIL to remove one layer of indirection from the implementation of Python::with_gil r=davidhewitt a=adamreichold
I am not sure if other people would consider this a simplification as well, but it helped me to remove one layer of indirection used by the implementation of `Python::with_gil`, especially flattening the `Option<_>` layers in `EnsureGIL` and `GILGuard::pool`. This makes it obvious that we always create `GILPool` when we actually acquire the GIL. (Of course, there still might be extra `GILPool` instances created manually.)
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3178: Fix a bad span of `_slf` for custom receivers in `#[pymethods]`. r=adamreichold a=lifthrasiir
This turned out to be a remnant of #1506. It notably resulted in a very confusing error:
```
error[E0308]: mismatched types
--> tests\test_methods.rs:1456:9
|
1456 | #[pymethods]
| ^^^^^^^^^^^^
| |
| expected `Py<Issue1506>`, found `*mut PyObject`
| arguments to this function are incorrect
...
1461 | / issue_1506!(
1462 | | #[pymethods]
1463 | | impl Issue1506 {
1464 | | fn issue_1506(
... |
1536 | | }
1537 | | );
| |_- in this macro invocation
|
= note: expected struct `pyo3::Py<Issue1506>`
found raw pointer `*mut pyo3::ffi::PyObject`
```
The actual cause is that `SelfType::receiver` is entirely ignored in this case and `_slf` refers to the original argument, but it sounds like that `TryFrom::try_from` somehow resulted in `*mut PyObject`...
Co-authored-by: Kang Seonghoon <public+git@mearie.org>
3161: add PyAny::is_exact_instance and PyAny::is_exact_instance_of r=adamreichold a=davidhewitt
Split off from #2881. I'll rebase after that merges.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2881: Rework `PyAny::is_instance_of` for performance r=adamreichold a=davidhewitt
I was talking to `@samuelcolvin` about the fastest way to identify object types (relevant e.g. for `pythonize` and also `pydantic-core`) and noticed that `PyAny::is_instance_of` is quite unoptimised because it expands to an ffi call to `PyObject_IsInstance`.
This PR proposes `PyAny::is_instance_of::<T>(obj)` is changed to be equivalent to `T::is_type_of(obj)`, plus add a sprinkling of inlining. We often have implementations such as `PyDict_Check` which can pretty much be optimised away to just checking a bit on the type object.
The accompanying benchmark to run through a bunch of object types is approx 40% faster after making this change.
For completeness, I've also added `PyAny::is_exact_instance` and `PyAny::is_exact_instance_of`, to pair with `T::is_exact_type_of`. (This could be split into a separate PR if preferred.)
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3158: Add Py::get for GIL-independent access to frozen classes. r=davidhewitt a=adamreichold
`@davidhewitt` Is this what you had in mind for #3154?
The name is an obvious candidate for bikeshedding.
Trying to write an example, I noticed that making `PyCell::get_frozen` public is most likely not useful as there is no way to safely get a `&PyCell` without acquiring the GIL first?
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
3159: Add OpenDAL to examples r=adamreichold a=suyanhanx
Hi there,
We have been using PyO3 to build our project's python binding and have been extremely satisfied with it.
And we want to say thank you for providing such a wonderful tool.
Co-authored-by: suyanhanx <suyanhanx@gmail.com>
