I am very excited to announce the 0.8 release of
PyOxidizer, a modern
Python application packaging tool. You can find the full changelog
in the docs.
First time user? See the
Getting Started
documentation.
Foremost, I apologize that this release took so long to publish (0.7 was
released on 2020-04-09). I fervently believe that frequent releases are
a healthy software development practice. And 6 months between PyOxidizer
releases was way too long. Part of the delay was due to world events
(it has proven difficult to focus on… anything given a global pandemic,
social unrest, and wildfires further undermining any resemblance of
lifestyle normalcy in California). Another contributing factor was I was
waiting on a few 3rd party Rust crates to have new versions published to
crates.io (you can’t release a crate to crates.io unless all your
dependencies are also published there).
Release delay and general life hardships aside, the 0.8 release is here
and it is full of notable improvements!
Python 3.8 and 3.9 Support
PyOxidizer 0.8 now targets Python 3.8 by default and support for Python
3.9 is available by tweaking configuration files. Previously, we only
supported Python 3.7 and this release drops support for Python 3.7. I feel
a bit bad for dropping compatibility. But Python 3.8 introduced a
new C API for initializing
Python interpreters (thank you Victor Stinner!) and this makes PyOxidizer’s
run-time code for interfacing with Python interpreters vastly simpler.
I decided that given the beta nature of PyOxidizer, it wasn’t worth
maintaining complexity to continue to support Python 3.7. I’m optimistic
that I’ll be able to support Python 3.8 as a baseline for a while.
PyOxidizer started as a science experiment of sorts to see if I could
achieve the elusive goal of producing a single file executable providing
a Python application. I was successful in proving this hypothesis. But the
cost to achieving this outcome was rather high in terms of end-user
experience: in order to produce single file executables, you had to break
a lot of assumptions about how Python typically works and this in turn broke
a lot of Python code and packages in the wild.
In other words, PyOxidizer’s opinionated defaults of producing a single file
executable were externalizing hardship on end-users and preventing them from
using PyOxidizer.
PyOxidizer 0.8 contains a handful of changes to defaults that should hopefully
lessen the friction.
On Windows, the default Python distribution now has a more traditional
build configuration (using .pyd extension modules and a pythonXY.dll
file). This means that PyOxidizer can consume pre-built extension modules
without having to recompile them from source. If you publish a Windows
binary wheel on PyPI, in many cases it will just work with PyOxidizer
0.8! (There are some notable exceptions to this, such as numpy, which is
doing wonky things with the location of shared libraries in wheels – but
I aim to fix this soon.)
Also on Windows, we no longer attempt to embed Python extension modules
(.pyd files) and their shared library dependencies in the produced
binary and load them from memory by default. This is because PyOxidizer’s
from-memory library loader didn’t work in all cases. For example, some
OpenSSL functionality used by the _ssl module in the standard library
didn’t work, preventing Python from establishing TLS connections. The old
mode enabling you to produce a single file executable on Windows is still
available. But you have to opt in to it (at the likely cost of more
packaging and compatibility pain).
Starlark Configuration Overhaul
PyOxidizer 0.8 contains a ton of changes to its Starlark configuration
files. There are so many changes that you may find it easier to port to
PyOxidizer 0.8 by creating a new configuration file rather than attempting
to port an existing one.
I apologize for this churn and recognize it will be disruptive. However,
this churn needed to happen for various reasons.
Much of the old Starlark configuration semantics was rooted in the days
when configuration files were static TOML files. Now that configuration
files provide the power of a (Python-inspired) programming language, we
are free to expose much more flexibility. But that flexibility requires
refactoring things so the experience feels more native.
Many changes to Starlark were rooted in necessity. For example,
the methods for invoking setup.py or pip install used to live on a
Python distribution type
and have been moved to a
type representing executables.
This is because the binary we are targeting influences how
packaging actions behave. For example, if the binary only supports
loading resources from memory (as opposed to standalone files), we need
to know that when invoking the packaging tool so we can produce files
(notably Python extension modules) compatible with the destination.
A major change to Starlark in 0.8 is around resource location handling.
Before, you could define a static string denoting the resources policy
for where things should be placed. And there were 10+ methods for
adding different resource types (source, bytecode, extensions, package
data) to different load locations (memory, filesystem). This mechanism
is vastly simplified and more powerful in PyOxidizer 0.8!
In PyOxidizer 0.8, there is a single
add_python_resource()
method for adding a resource to a binary and the Starlark objects you add
can denote where they should be added by
defining attributes on those objects.
Furthermore, you can
define a Starlark function
that is called when resource objects are created to apply custom packaging
rules using custom Starlark code defined in your PyOxidizer config file.
So rather than having everyone try to abide by a few pre-canned policies for
packaging resources, you can define a proper function in your config file
that can be as complex as you want/need it to be! I feel this is vastly simpler
and more powerful than implementing a custom DSL in static configuration files
(like TOML, JSON, YAML, etc).
While the ability to implement your own arbitrarily complex packaging
policies is useful, there is a new
PythonPackagingPolicy
Starlark type with enough flexibility to suit most needs.
Shipping oxidized_importer
During the development of PyOxidizer 0.8, I broke out the custom
Rust-based Python meta-path importer used by PyOxidizer’s run-time code
into a standalone Python package. This sub-project is called
oxidized_importer and I previously
blogged about it.
PyOxidizer 0.8 ships oxidized_importer and all of its useful APIs
available to Python. Read more in the
official docs.
The new Python APIs should make debugging issues with PyOxidizer-packaged
applications vastly simpler: I found them invaluable when tracking down
user-reported bugs!
Tons of New Tests and Refactored Code
PyOxidizer was my first non-toy Rust project. And the quality of the Rust
code I produced in early versions of PyOxidizer clearly showed it. And when I
was in the rapid-prototyping phase of PyOxidizer, I eschewed writing tests
in favor of short-term progress.
PyOxidizer 0.8 pays down a ton of technical debt in the code base. Lots of
Rust code has been refactored and is using somewhat reasonable practices.
I’m not yet a Rust guru. But I’m at the point where I cringe when I look at
some of the early code I wrote, which is a good sign. I do have to say that
Rust has been a dream to work with during this transition. Despite being a
low-level language, my early misuse of Rust did not result in crashes like
you would see in languages like C/C++. And Rust’s seemingly omniscient compiler
and IDE tools facilitating refactoring have ensured that code changes aren’t
accompanied by subtle random bugs that would occur in dynamic programming
languages. I really need to write a dedicated post espousing the virtues of
Rust…
There are a ton of new tests in PyOxidizer 0.8 and I now feel somewhat
confident that the main branch of PyOxidizer should be considered
production-ready at any time assuming the tests pass. This will hopefully
lead to more rapid releases in the future.
There are now tests for the pyembed Rust crate, which provides the
run-time code for PyOxidizer-built binaries. We even have
Python-based unit tests
for validating the Python-exposed APIs behave as expected. These tests have
been invaluable for ensuring that the run-time code works as expected. So now
when someone files a bug I can easily write a test to capture it and keep
the code working as intended through various refactors.
The packaging-time Rust code has also gained its fair share of tests.
We now have fairly comprehensive test coverage around how resources
are added/packaged. Python extension modules have proved to be highly
nuanced in how they are handled. Tremendously helping testing of extension
modules is that we’re able to run tests for platform non-native extensions!
While not yet exposed/supported by Starlark configuration files, I’ve taught
PyOxidizer’s core Rust code to be cross-compiling aware so that we can
e.g. test Windows or macOS behavior from Linux. Before, I’d have to test
Windows wheel handling on Windows. But after writing a wheel parser in Rust
and teaching PyOxidizer to use a different Python distribution for the
host architecture from the target architecture, I’m now able to write
tests for platform-specific functionality that run on any platform that
PyOxidizer can run on. This may eventually lead to proper cross-compiling
support (at least in some configuration). Time will tell. But the foundation
is definitely there!
New Rust Crates
As part of the aforementioned refactoring of PyOxidizer’s Rust code, I’ve
been extracting some useful/generic functionality built as part of
developing PyOxidizer to their own Rust crates.
As part of this release, I’m publishing the initial 0.1 release of the
python-packaging crate
(docs). This crate
provides pure Rust code for various Python packaging related functionality.
This includes:

  • Rust types representing Python resource types (source modules, bytecode
    modules, extension modules, package resources, etc).
  • Scanning the filesystem for Python resource files .
  • Configuring an embedded Python interpreter.
  • Parsing PKG-INFO and related files.
  • Parsing wheel files.
  • Collecting Python resources and serializing them to a data structure.

The crate is somewhat PyOxidizer centric. But if others are interested
in improving its utility, I’ll happily accept pull requests!
PyOxidizer’s crates footprint now includes:
Major Documentation Updates
I strongly believe that software should be documented thoroughly and I strive
for PyOxidizer’s documentation to be useful and comprehensive.
There have been a lot of changes to PyOxidizer’s documentation since the
0.7 release.
All configuration file documentation
has been consolidated.
Likewise, I’ve attempted to consolidate a lot of the paved road documentation
for how to use PyOxidizer in the
Packaging User Guide
section of the docs.
I’ll be honest, since I have so much of PyOxidizer’s workings internalized,
it can be difficult for me to empathize with PyOxidizer’s users. So if you
have difficult with the readability of the documentation, please
file an issue and report
what is confusing so the documentation can be improved!
Mercurial Shipping With PyOxidizer 0.8
PyOxidizer is arguably an epic yak shave of mine to help the
Mercurial version control tool transition
to Python 3 and Rust.
I’m pleased to report that Mercurial is
now shipping
PyOxidizer-built distributions on Windows as of the 5.2.2 release a few days
ago! If a complex Python application like Mercurial can be
configured
to work with PyOxidizer, chances are your Python application will work as
well.
Whats Next
I view PyOxidizer 0.8 as a pivotal release where PyOxidizer is turning the
corner from a prototyping science experiment to something more generally
usable. The investments in test coverage and refactoring of the Rust
internals are paving the way towards future features and bug fixes.
In upcoming releases, I’d like to close remaining known compatibility
gaps with popular Python packages (such as numpy and other packages in
the scientific/data space). I have a general idea of what work needs to
be done and I’ve been laying the ground work via various refactorings to
execute here.
I want a general theme of future releases to be eliminating reasons why
people can’t use PyOxidizer. PyOxidizer’s historical origin was as a
science experiment to see if single file Python applications were possible.
It is clear that achieving this is fundamentally incompatible with
compatibility with tons of Python packages in the wild. I’d like to find a
way where PyOxidizer can achieve 99% package compatibility by default
so new users don’t get discouraged when using PyOxidizer. And for the
subset of users who want single file executables, they can spend the
magnitude of additional effort to achieve that.
At some point, I also want to make a pivot towards focusing on producing
distributable artifacts (Debian/RPM packages, MSI installers, macOS DMG
files, etc). I’m slightly bummed that I haven’t made much progress here.
But I have a vision in my mind of where I want to go (I’ll be making
a standalone Rust crate + Starlark dialect to facilitate producing
distributable artifacts for any application) and I’m anticipating
starting this work in the next few months. In the mean time, PyOxidizer
0.8 should be able to give people a directory tree that they can coerce
into distributable artifacts using existing packaging tooling. That’s not as
turnkey as I would like it to be. But the technical problems around
building a distributable Python application binary still needs some work
and I view that as the most pressing need for the Python ecosystem. So
I’ll continue to focus there so there is a solid foundation to build upon.
In conclusion, I hope you enjoy the new release! Please report any issues
or feedback in the
GitHub issue tracker.