3015: Implement wrapper for `PyASCIIObject.state` bitfield accesses r=davidhewitt a=decathorpe
This is a first draft of my attempt to fix#1824 "properly" by writing a C wrapper for the `PyASCIIObject.state` bitfield accesses, as proposed here: https://github.com/PyO3/pyo3/issues/1824#issuecomment-1406909504
---
The original argument for making these functions `unsafe` is still valid, though - bitfield memory layout is not guaranteed to be stable across different C compilers, as it is "implementation defined" in the C standard. However, short of having CPython upstream provide non-inlined public functions to access this bitfield, this is the next best thing, as far as I can tell.
I've removed the `#[cfg(target_endian = "little")]` attributes from all things that are un-blocked by fixing this issue on big-endian systems, except for three tests, which look like expected failures considering that they do not take bit/byte order into account (for example, when writing to the bitfield).
- `ffi::tests::ascii_object_bitfield`
- `types::string::tests::test_string_data_ucs2_invalid`
- `types::string::tests::test_string_data_ucs4_invalid`
All other tests now pass on both little-endian and big-endian systems.
---
I am aware that some parts of this PR are probably not in a state that's acceptable for merging as-is, which is why I'm filing this as a draft. Feedback about how to better integrate this change with pyo3-ffi would be great. :)
In particular, I'm unsure whether the `#include` statements in the C files are actually correct across different systems. I have only tested this on Fedora Linux so far.
I'm also open to changing the names of the C functions that are implemented in the wrapper. For now I chose the most obvious names that shouldn't cause collisions with other symbols.
Co-authored-by: Fabio Valentini <decathorpe@gmail.com>
2975: RFC: Add GILProtected synchronization primitive and use it for LazyTypeObjectInner. r=davidhewitt a=adamreichold
I would also like to use that type in rust-numpy and it seems we can avoid ~~both a manual unsafe impl and~~ a full blown mutex if we apply it to `LazyTypeObjectInner`.
One downside might be that it ties us closer to the GIL when we want to enable nogil experimentation, but on the other hand, it may also help by reifying the GIL usage. (This is currently limited to comments in unsafe code in rust-numpy for example.)
3022: Fix function name shadowing r=davidhewitt a=mejrs
Fixes https://github.com/PyO3/pyo3/issues/3017
3023: Emit a better error for bad argument names r=davidhewitt a=mejrs
This will emit a better error for code like
```rust
#[pyfunction]
fn output([a,b,c]: [u8;3]) {}
```
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
Co-authored-by: mejrs <59372212+mejrs@users.noreply.github.com>
3055: Fix compile error when trying to use static slot methods, fixes: #3039 r=davidhewitt a=willstott101
Whilst having static slot methods is of dubious value in the first place, there's little reason not to support it by bringing the different function call macros more in-line with each other.
I had no idea which test file to use so I just made a new one... `test_methods.rs` has no slots in and the test files that do use slots seem to be specific to their protocol. Let me know if you'd like the test somewhere else.
Thanks!
Co-authored-by: Will Stott <willstott101@gmail.com>
3044: Add `PyTuple::to_list` r=davidhewitt a=davidhewitt
Companion to #3043. I've included benchmarks which suggests that this method is indeed about 25% faster on my machine.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
3031: mangle exported functions for PyPy r=davidhewitt a=mattip
I run HEAD of PyO3 with HEAD of some PyPy branches, and noticed that the py3.10 branch (which implements python3.10) [fails to properly build](https://github.com/pypy/binary-testing/actions/runs/4334873617/jobs/7569059852#step:6:179). The failure was a missing export:
```
/home/runner/work/binary-testing/binary-testing/pyo3/src/exceptions.rs:715: \
undefined reference to `PyExc_EncodingWarning'
collect2: error: ld returned 1 exit status
```
So I grepped around in the code for `Py_3_10` to see what new functions were added and found a few that needed PyPy-specific exports.
Co-authored-by: Matti Picus <matti.picus@gmail.com>
2899: RFC: Provide a special purpose FromPyObject impl for byte slices r=davidhewitt a=adamreichold
This enables efficiently and safely getting a byte slice from either bytes or byte arrays.
The main issue I see here is discoverability, i.e. should this be mention in the docs of `PyBytes` and `PyByteArray` or in the guide?
It is also not completely clear whether this really _fixes_ the issue.
Closes#2888
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
2952: Fix allow_threads segfault r=davidhewitt a=OliverBalfour
Please see the corresponding issue **#2951** for details. This PR adds the failing test from the issue and then a fix for it. The fix simply calls `ReferencePool::update_counts` at the end of `allow_threads` to ensure objects aren't accidentally deleted too soon.
Co-authored-by: Oliver Balfour <oliver.leo.balfour@gmail.com>
2947: change PyModule::add_class to return an error if class creation fails r=adamreichold a=davidhewitt
Related to #2942
At the moment there are panics deep in the `#[pyclass]` machinery when initialising the type fails. This PR adjusts a number of these functions to return `PyResult` instead, so that we can handle the error more appropriately further down the pipeline.
For example, take the following snippet:
```rust
#[pyclass(extends = PyBool)]
struct ExtendsBool;
#[pymodule]
fn pyo3_scratch(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_class::<ExtendsBool>()?;
Ok(())
}
```
Currently, importing this module will fail with a panic:
```
TypeError: type 'bool' is not an acceptable base type
thread '<unnamed>' panicked at 'An error occurred while initializing class ExtendsBool', /Users/david/Dev/pyo3/src/pyclass.rs:412:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/david/.virtualenvs/pyo3/lib/python3.10/site-packages/pyo3_scratch/__init__.py", line 1, in <module>
from .pyo3_scratch import *
pyo3_runtime.PanicException: An error occurred while initializing class ExtendsBool
```
After this PR, this import still fails, but with a slightly cleaner, more Pythonic error:
```
TypeError: type 'bool' is not an acceptable base type
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/david/.virtualenvs/pyo3/lib/python3.10/site-packages/pyo3_scratch/__init__.py", line 1, in <module>
from .pyo3_scratch import *
RuntimeError: An error occurred while initializing class ExtendsBool
```
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Adam Reichold <adam.reichold@t-online.de>
2944: optimize sequence conversion for list and tuple r=adamreichold a=davidhewitt
closes#2943
Avoid using `PyObject_IsInstance` for checking if lists or tuples are sequences, as we know they are always sequences.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2914: correct ffi definition of PyIter_Check r=davidhewitt a=davidhewitt
Closes#2913
It looks like what is happening is that PyO3 was relying on an outdated macro form of `PyIter_Check` which is now a CPython implementation detail, which would explain why it was behaving inconsistently on different platforms (likely due to differences in linkers / implementations).
The test I've pushed succeeds, but fails to compile due to a hygiene bug! I'm done for tonight so I'll take a look at that soon and then rebase this after.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2912: Add `PyDict.update()` and `PyDict.update_if_missing()` r=davidhewitt a=samuelcolvin
Fix#2910
Note, I'd also be happy to remove the `override_` argument from merge and perhaps rename it to `update_missing` or similar to give a cleaner API. LMK what you think.
Please consider adding the following to your pull request:
- [x] an entry for this PR in newsfragments
- [x] docs to all new functions and / or detail in the guide
- [x] tests for all new or changed functions
Co-authored-by: Samuel Colvin <s@muelcolvin.com>
2923: hygiene: fix `#[pymethods(crate = "...")]` r=davidhewitt a=davidhewitt
Got to the bottom of the hygiene issue in test of #2914
Turns out that `#[pymethods] #[pyo3(crate = "...")]` works, but `#[pymethods(crate = "...")]` was ignoring the argument.
Added a tweak to fix this and a snippet in the hygiene test (which fails on `main`).
2924: remove unneeded into_iter calls r=davidhewitt a=davidhewitt
Clippy complaining about these to me this morning locally.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
2904: refactor docstring generation code r=mejrs a=davidhewitt
As a first step towards #2866 this is a tweak to the macro code which generates Python docstrings. This PR refactors the behaviour so that instead of always creating a `concat!` expression to generate a nul-terminated string, this will only happen if a Rust 1.54+ macro doc is present (e.g. `#[doc = include_str!(...)]`).
The default case now just collects all the `#[doc]` attributes into a single string.
This should make it easier to factor out the `text_signature` formatting, and avoids wasting compile time invoking the `concat!` macro when not necessary.
2921: Check to see if object is `None` before traversing r=davidhewitt a=neachdainn
Closes#2915
When using the C API directly, the intended way to call `visitproc` is via the `Py_VISIT` macro, which checks to see that the provided pointer is not null before passing it along to `visitproc`. Because PyO3 isn't using the macro, it needs to manually check that the pointer isn't null. Without this check, calling `visit.call(&obj)` where `let obj = None;` will segfault.
Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Nate Kent <nate@nkent.net>
