Talk

Deinventing the Wheel

Saturday, May 25

11:40 - 12:25
RoomLasagna
LanguageEnglish
Audience levelIntermediate
Elevator pitch

The ability to include pre-built binaries into Python packages has been crucial for Python but now 75% of PyPi’s 66TB total data is occupied by binary blobs.

What would happen if we didn’t have to bundle binaries into Python packages anymore?

Abstract

When a Python package depends on non-python code (usually C/C++ libraries), it’s customary to include in the package also pre-built binary blobs of those external dependencies. This is done for good reasons as it’s the only way to ensure that clients won’t have problems when installing the package. Or, at least, it used to be the only way.

Bundling binary blobs into packages also comes with some downsides, the biggest of which is that, of the current 66TB of package data stored in PyPI, 50TB (75%) is just binary blobs (see https://py-code.org/stats), which is only going to get worse as the rise of new architectures will mean that even more pre-built blobs will have to be bundled in each package.

In this talk I will explain why building C/C++ projects can be problematic and will show how those problems are solved by the Zig compiler toolchain. After that I will explain the exciting things that could be achieved once building C/C++ code stops being a problem, from helping PyPI free some space, to creating maximally efficient builds for the target machine.

By the end of the talk attendees should have gained some insight into how Python tooling interacts with C/C++ dependencies, have gained some sympathy for PyPI (running a package index is expensive!), and be inspired by the positive effects that advances in the systems programming space can bring to Python.

TagsTooling, Packaging, Performance, PyPi
Participant

Loris Cro

Hi, I’m VP of Community at the Zig Software Foundation, the 501(c)(3) non-profit organization behind the Zig Programming Language. I run a small YouTube channel (https://youtube.com/c/ZigSHOWTIME) and often livecode on Twitch (https://twitch.tv/kristoff_it). I also run a conference titled Software You Can love (https://softwareyoucan.love). My current main hobby project is a static site generator called Zine (named after “fanzine”) in which I’m experimenting with, the idea of having a different type of templating language than the usual style used by Jinja, Mustache, Handlebars, etc. If you too have opinions about static site generators, come chat me up!