blog

commit

970a58570e15a9077043bfcafe026ba01f0f8aba

parent

70c77db9f384e5a2cb32282eab69412258aec666

Author

Tobias Bengfort <tobias.bengfort@posteo.de>

Date

2023-11-04 22:07

add post on python packaging

Diffstat

3 files changed, 137 insertions, 2 deletions

diff --git a/_content/posts/2023-01-29-python-async-loops/index.md b/_content/posts/2023-01-29-python-async-loops/index.md

@@ -1,7 +1,7 @@
    1     1 ---
    2     2 title: Eight different ways to implement an asyncronous loop in python
    3     3 date: 2023-01-29
    4    -1 tags: [code, linux]
   -1     4 tags: [code, python, linux]
    5     5 ---
    6     6 
    7     7 [asyncio](https://peps.python.org/pep-3156/) was first added to the python

diff --git a/_content/posts/2023-10-13-logging/index.md b/_content/posts/2023-10-13-logging/index.md

@@ -1,7 +1,7 @@
    1     1 ---
    2     2 title: Another look at python's logging library
    3     3 date: 2023-10-13
    4    -1 tags: [code]
   -1     4 tags: [code, python]
    5     5 ---
    6     6 
    7     7 Python's [logging](https://docs.python.org/3/library/logging.html) library is a

diff --git a/_content/posts/2023-11-04-python-packaging/index.md b/_content/posts/2023-11-04-python-packaging/index.md

@@ -0,0 +1,135 @@
   -1     1 ---
   -1     2 title: Python packaging with pyproject.toml and setuptools
   -1     3 date: 2023-11-04
   -1     4 tags: [code, python]
   -1     5 ---
   -1     6 
   -1     7 Python packaging has been in a bad state for ages. I recently read a [post by
   -1     8 Gregory
   -1     9 Szorc](https://gregoryszorc.com/blog/2023/10/30/my-user-experience-porting-off-setup.py/)
   -1    10 that resonated with me a lot. Still, I do not personally have any issues
   -1    11 in practice. So in this post I am going to explain how I do package management
   -1    12 without loosing my mind.
   -1    13 
   -1    14 ## Be aware of the different kinds of tools in package management
   -1    15 
   -1    16 Package management in python is highly modular, and each part of the process
   -1    17 can have multiple implementations. In this article I will use `setuptools` as a
   -1    18 build backend (the part that actually builds the package), `build` as a build
   -1    19 frontend (the part that creates a build environment) and `pip` to install
   -1    20 packages. But I could also use `poetry` to cover all of those roles with a
   -1    21 single tool.
   -1    22 
   -1    23 There are standards that allow all of these tools to work together.
   -1    24 [PEP 427](https://peps.python.org/pep-0427/) defines the "wheel" package
   -1    25 format. [PEP 517](https://peps.python.org/pep-0517/) defines the interface
   -1    26 between build backends and frontends.
   -1    27 
   -1    28 Note that the separation is not always razor sharp. For example, a build
   -1    29 frontend may also have to install packages into the build environment. And an
   -1    30 installer might have to act as a build frontend if the package is not available
   -1    31 as a wheel.
   -1    32 
   -1    33 ## Don't create a package if you want an environment
   -1    34 
   -1    35 Many modern package managers like npm, cargo, or Poetry automatically create
   -1    36 lockfiles and recommend to commit them to version control. I really don't
   -1    37 understand why they are doing this. A package is supposed to be installed along
   -1    38 with other packages, so it needs to be compatible with as many versions as
   -1    39 possible. I do understand that you sometimes want a reproducible environment.
   -1    40 But those are two separate things.
   -1    41 
   -1    42 If you want to create a reproducible environment, you can use a simple
   -1    43 [requirements.txt
   -1    44 file](https://pip.pypa.io/en/stable/reference/requirements-file-format/) and
   -1    45 install it with `python -m pip install -r requirements.txt`. The file could
   -1    46 look like this:
   -1    47 
   -1    48 ```
   -1    49 # allow a range of versions
   -1    50 foo >= 1.1, < 2.0
   -1    51 
   -1    52 # select optional features
   -1    53 bar[feature]
   -1    54 
   -1    55 # pin a specific version and specifiy a hash for supply chain integrity
   -1    56 baz == 1.2.3 --hash=sha256:f22fa1e554c9ddfd16e6e41ac79759e17be9e492b3587efa038054674760e72d
   -1    57 ```
   -1    58 
   -1    59 There are some tools that can help you generate these files, e.g. `pip freeze`
   -1    60 or [pip-tools](https://github.com/jazzband/pip-tools). Still, I find that you
   -1    61 should not have too much automation in this area. The goal is that you have
   -1    62 control over the environment, not the other way around.
   -1    63 
   -1    64 ## Use venv instead of virtualenv
   -1    65 
   -1    66 You usually want to install the requirements for each project into a separate
   -1    67 environment. That approach was pioneered by the package
   -1    68 [`virtualenv`](https://virtualenv.pypa.io/en/latest/). However, the
   -1    69 functionality was so useful that it was integrated into the standard library in
   -1    70 python 3.3 (2012). I still see references to `virtualenv` more than 10 years
   -1    71 later, but you really don't need it. Just run `python -m venv` instead.
   -1    72 
   -1    73 ## Use pyproject.toml to specify package meta data
   -1    74 
   -1    75 I stuck with `setup.py` and `setup.cfg` pretty long. The most important reason
   -1    76 was that editable installs were not supported when using `pyproject.toml`. But
   -1    77 now, setuptools (>= 64) and pip (>= 21.3) both have all the features I need.
   -1    78 However, [Ubuntu 22.04 is still on setuptools
   -1    79 59](https://packages.ubuntu.com/jammy/python3-setuptools). I have started
   -1    80 porting some projects to the new system. But I will probably wait with some
   -1    81 more critical projects until the new features are widely available.
   -1    82 
   -1    83 The [setuptools
   -1    84 documentation](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html)
   -1    85 on pyproject.toml is solid and even though to my knowledge there is no tool for
   -1    86 automatic migration, porting an existing `setup.py` or `setup.cfg` to the new
   -1    87 syntax should be simple enough.
   -1    88 
   -1    89 You can then build your package either by using `python -m build` (which I see
   -1    90 recommended in most places) or `python -m pip wheel .` (which doesn't require
   -1    91 an additional tool). To upload your package to PyPI you can use
   -1    92 [twine](https://twine.readthedocs.io/en/stable/).
   -1    93 
   -1    94 ## Include data files
   -1    95 
   -1    96 setuptools will automatically include python files in the package. If you need
   -1    97 to include other files, e.g. templates or translations, you traditionally had
   -1    98 to use a separate `MANIFEST.in` file. That still works, but it can also be
   -1    99 included in `pyproject.toml` directly:
   -1   100 
   -1   101 ```
   -1   102 [tool.setuptools.package-data]
   -1   103 mypackage = [
   -1   104     "**/*.html",
   -1   105     "**/*.csv",
   -1   106 ]
   -1   107 ```
   -1   108 
   -1   109 ## Configure other tools
   -1   110 
   -1   111 Most tools can be configured using `pyproject.toml`, e.g.
   -1   112 [pytest](https://docs.pytest.org/en/7.1.x/reference/customize.html#pyproject-toml),
   -1   113 [coverage](https://coverage.readthedocs.io/en/latest/config.html), or
   -1   114 [isort](https://pycqa.github.io/isort/docs/configuration/config_files.html#pyprojecttoml-preferred-format).
   -1   115 A prominent exception is
   -1   116 [flake8](https://github.com/PyCQA/flake8/issues/234#issuecomment-812800722),
   -1   117 but you can replace most of it by
   -1   118 [ruff](https://docs.astral.sh/ruff/configuration/#using-pyprojecttoml).
   -1   119 
   -1   120 ## Conclusion
   -1   121 
   -1   122 Python's packaging infrastructure is certainly not it's best feature, but it is
   -1   123 still usable. The transition to pyproject.toml took far too long and was far
   -1   124 too messy, but I am confident that we will finally be done with it in just a
   -1   125 few years.
   -1   126 
   -1   127 In this article I stuck with setuptools, because that is the build backend I
   -1   128 know best. However, setuptools has accumulated a lot of legacy code over the
   -1   129 years. [flit](https://flit.pypa.io/en/latest/rationale.html) is another backend
   -1   130 that has "not being setuptools" as its main feature. I am not sure if this is
   -1   131 enough of a reason for a switch. But I might try it anyways.
   -1   132 
   -1   133 I wouldn't say that python packaging is good now. But at least it has
   -1   134 stabilized to a degree that I feel like we could actually, finally reap the
   -1   135 benefits of blowing up everything.