Merge pull request #1968 from KRunchPL/guide-python-typing-hints

#1959 - Add Python Typing Hints to the Guide

guide/src/SUMMARY.md

@@ -31,4 +31,5 @@
 [Appendix A: Migration Guide](migration.md)
 [Appendix B: PyO3 and rust-cpython](rust_cpython.md)
 [Appendix C: Trait bounds](trait_bounds.md)
+[Appendix D: Python typing hints](python_typing_hints.md)
 [CHANGELOG](changelog.md)

guide/src/python_typing_hints.md

@@ -0,0 +1,163 @@
+# Typing and IDE hints for you Python package
+
+PyO3 provides an easy to use interface to code native Python libraries in Rust. The accompanying Maturin allows you to build and publish them as a package. Yet, for the better user experience, Python libraries should provide typing hints and documentation for all public entities, so that IDEs can show them during development and type analyzing tools such as `mypy` can use them to properly verify the code.
+
+Currently the best solution for the problem is to maintain manually the `*.pyi` files and ship them along with the package.
+
+## The `pyi` files introduction
+
+`pyi` (an abbreviation for `Python Interface`) is called a `Stub File` in most of the documentations related to them. Very good definition of what it is can be found in [old MyPy documentation](https://github.com/python/mypy/wiki/Creating-Stubs-For-Python-Modules):
+
+> A stubs file only contains a description of the public interface of the module without any implementations.
+
+Probably most Python developers encountered them already when trying to use the IDE "Go to Definition" function on any builtin type. For example the definitions of few standard exceptions look like this:
+
+```python
+class BaseException(object):
+    args: Tuple[Any, ...]
+    __cause__: BaseException | None
+    __context__: BaseException | None
+    __suppress_context__: bool
+    __traceback__: TracebackType | None
+    def __init__(self, *args: object) -> None: ...
+    def __str__(self) -> str: ...
+    def __repr__(self) -> str: ...
+    def with_traceback(self: _TBE, tb: TracebackType | None) -> _TBE: ...
+
+class SystemExit(BaseException):
+    code: int
+
+class Exception(BaseException): ...
+
+class StopIteration(Exception):
+    value: Any
+```
+
+As we can see those are not full definitions containing implementation, but just a description of interface. It is usually all that is needed by the user of the library.
+
+### What does the PEPs say?
+
+As of the time of writing this documentation the `pyi` files are referenced in three PEPs.
+
+[PEP8 - Style Guide for Python Code - #Function Annotations](https://www.python.org/dev/peps/pep-0008/#function-annotations) (last point) recommends all third party library creators to provide stub files as the source of knowledge about the package for type checker tools.
+
+> (...) it is expected that users of third party library packages may want to run type checkers over those packages. For this purpose [PEP 484](https://www.python.org/dev/peps/pep-0484) recommends the use of stub files: .pyi files that are read by the type checker in preference of the corresponding .py files. (...)
+
+[PEP484 - Type Hints - #Stub Files](https://www.python.org/dev/peps/pep-0484/#stub-files) defines stub files as follows.
+
+> Stub files are files containing type hints that are only for use by the type checker, not at runtime.
+
+It contains a specification for them (highly recommended reading, since it contains at least one thing that is not used in normal Python code) and also some general information about where to store the stub files.
+
+[PEP561 - Distributing and Packaging Type Information](https://www.python.org/dev/peps/pep-0561/) describes in detail how to build packages that will enable type checking. In particular it contains information about how the stub files must be distributed in order for type checkers to use them.
+
+## How to do it?
+
+[PEP561](https://www.python.org/dev/peps/pep-0561/) recognizes three ways of distributing type information:
+
+* `inline` - the typing is placed directly in source (`py`) files;
+* `separate package with stub files` - the typing is placed in `pyi` files distributed in their own, separate package;
+* `in-package stub files` - the typing is placed in `pyi` files distributed in the same package as source files.
+
+The first way is tricky with PyO3 since we do not have `py` files. When it will be investigated and necessary changes are implemented, this document will be updated.
+
+The second way is easy to do, and the whole work can be fully separated from the main library code. The example repo for the package with stub files can be found in [PEP561 references section](https://www.python.org/dev/peps/pep-0561/#references): [Stub package repository](https://github.com/ethanhs/stub-package)
+
+The third way is described below.
+
+### Including `pyi` files in your PyO3/Maturin build package
+
+When source files are in the same package as stub files, they should be placed next to each other. We need a way to do that with Maturin. Also, in order to mark our package as typing-enabled we need to add an empty file named `py.typed` to the package.
+
+#### If you do not have other Python files
+
+If you do not need to add any other Python files apart from `pyi` to the package, the Maturin provides a way to do most of the work for you. As documented in [Maturin Guide](https://github.com/PyO3/maturin/blob/084cfaced651b28616aeea1f818bdc933a536bfe/guide/src/project_layout.md#adding-python-type-information) the only thing you need to do is create a stub file for your module named `<module_name>.pyi` in your project root and Maturin will do the rest.
+
+```text
+my-rust-project/
+├── Cargo.toml
+├── my_project.pyi  # <<< add type stubs for Rust functions in the my_project module here
+├── pyproject.toml
+└── src
+    └── lib.rs
+```
+
+For example of `pyi` file see [`my_project.pyi` content](#my_projectpyi-content) section.
+
+#### If you need other Python files
+
+If you need to add other Python files apart from `pyi` to the package, you can do it also, but that requires some more work. Maturin provides easy way to add files to package ([documentation](https://github.com/PyO3/maturin/blob/0dee40510083c03607834c821eea76964140a126/Readme.md#mixed-rustpython-projects)). You just need to create a folder with the name of your module next to the `Cargo.toml` file (for customization see documentation linked above).
+
+The folder structure would be:
+
+```text
+my-project
+├── Cargo.toml
+├── my_project
+│   ├── __init__.py
+│   ├── my_project.pyi
+│   ├── other_python_file.py
+│   └── py.typed
+├── pyproject.toml
+├── Readme.md
+└── src
+    └── lib.rs
+```
+
+Let's go a little bit more into details on the files inside the package folder.
+
+##### `__init__.py` content
+
+As we now specify our own package content, we have to provide the `__init__.py` file, so the folder is treated as a package and we can import things from it. We can always use the same content that the Maturin creates for us if we do not specify a python source folder. For PyO3 bindings it would be:
+
+```python
+from .my_project import *
+```
+
+That way everything that is exposed by our native module can be imported directly from the package.
+
+##### `py.typed` requirement
+
+As stated in [PEP561](https://www.python.org/dev/peps/pep-0561/):
+> Package maintainers who wish to support type checking of their code MUST add a marker file named py.typed to their package supporting typing. This marker applies recursively: if a top-level package includes it, all its sub-packages MUST support type checking as well.
+
+If we do not include that file, some IDEs might still use our `pyi` files to show hints, but the type checkers might not. MyPy will raise an error in this situation:
+
+```text
+error: Skipping analyzing "my_project": found module but no type hints or library stubs
+```
+
+The file is just a marker file, so it should be empty.
+
+##### `my_project.pyi` content
+
+Our module stub file. This document does not aim at describing how to write them, since you can find a lot of documentation on it, starting from already quoted [PEP484](https://www.python.org/dev/peps/pep-0484/#stub-files).
+
+The example can look like this:
+
+```python
+class Car:
+    """
+    A class representing a car.
+
+    :param body_type: the name of body type, e.g. hatchback, sedan
+    :param horsepower: power of the engine in horsepower
+    """
+    def __init__(self, body_type: str, horsepower: int) -> None: ...
+
+    @classmethod
+    def from_unique_name(cls, name: str) -> 'Car':
+        """
+        Creates a Car based on unique name
+
+        :param name: model name of a car to be created
+        :return: a Car instance with default data
+        """
+
+    def best_color(self) -> str:
+        """
+        Gets the best color for the car.
+
+        :return: the name of the color our great algorithm thinks is the best for this car
+        """
+```

src/lib.rs

@@ -405,6 +405,10 @@ pub mod doc_test {
     doctest!("guide/src/trait_bounds.md", guide_trait_bounds_md);
     doctest!("guide/src/types.md", guide_types_md);
     doctest!("guide/src/faq.md", faq);
+    doctest!(
+        "guide/src/python_typing_hints.md",
+        guide_python_typing_hints
+    );
 
     // deliberate choice not to test guide/ecosystem because those pages depend on external crates
     // such as pyo3_asyncio.