• About
• Latest Posts

Jonathon Hall

Latest posts by Jonathon Hall (see all)

• PureOS Crimson Development Report: November 2024 - December 16, 2024
• Purism’s PureBoot is Not Affected by UEFI Key Leaks (Again) - July 29, 2024
• From the Hackdesk: Librem 16 - May 30, 2024

Welcome to Part Five

Welcome back!  Since our first report, this is now our fifth installment.  We appreciate our community’s feedback about these updates, and we are happy to continue providing them!

Your support in PureOS subscriptions, volunteer efforts, and community feedback helps us advance PureOS for all Librem devices as well as the larger FOSS ecosystem.

In our last installment, we mentioned a number of package updates we published, now that Crimson again synchronizes from Debian.  Most of these updated packages built right away, but one failed.  Fixing it took us through a deep dive in apt, debootstrap, and debspawn.

From source packages to binaries

Most of our readers will know that software is often written as source code, then compiled to the binaries that the computer can actually execute.  When you install a package in a Debian derivative like PureOS, you are installing a binary package that someone else compiled from source code.

Like Debian, PureOS prefers that the binaries are built on common infrastructure.  Developers only upload source code.  This ensures that binaries are built using the correct dependencies, that those dependencies are properly present in PureOS, and that a problem (or compromise) on a developer system cannot result in a broken (or compromised) package in the archive.

Preparing the build environment

Having the right build environment is critical.  If you build against the wrong dependencies, the build might not complete, or it might not work when users install it.

For Laniakea, debspawn is in charge of creating the build environment and updating it as we build each package.  It uses debootstrap and apt.

We want the same build output no matter whether the build environment was from yesterday or a month ago.  Updating the environment before the build makes it consistent.

In other words, you and I can both get the same build result, even if our build environments initially were different.  If your environment was created in July, and mine was created yesterday, they’ll be the same after we both update.

The same, until they’re different

Most of the updated packages in October were fine, they built as expected.

For util-linux, one of the builds failed on the new build worker we set up in September.  The older worker was fine.  There was a difference in the build environment, and apt would not install build dependencies for util-linux on the new worker due to a conflict:

The following packages have unmet dependencies:
 libsystemd-dev : Depends: libsystemd0 (= 247.3-7+deb11u4)
 libudev-dev : Depends: libudev1 (= 247.3-7+deb11u4) but 247.3-7+deb11u6 is to be installed
 systemd : Depends: libsystemd0 (= 247.3-7+deb11u4)
E: Unable to correct problems, you have held broken packages

This was puzzling.  Why would the two environments differ?  Updating is supposed to make them the same, and that was fine.  It shouldn’t even matter whether we build against the ‘u4 or ‘u6 version of this dependency.  Why does apt refuse to install this package entirely?

The key was apt pinning.  You can tell apt to prefer packages from a particular source.  Packages from another source can be made available, but with lower preference than the first source.

debspawn does that.  Since we’re building a package that will go to byzantium-updates-proposed, it prefers to use other packages from that source.  Other sources are made available (like the full byzantium archive), but only used if we can’t choose a package from the preferred source.

This answered the latter question – why apt refused to install this build dependency.  The environment already had a package from our non-preferred suite installed.  To install this build dependency, apt would have to select the same version from the non-preferred suite.  We told it not to do that, so it could not find a way to proceed.

We have to go deeper

This still left the first question – why the environments were different.  debspawn had updated the environments, and apt said that went fine.  That’s our key assumption – if your environment was created in July, and mine was created yesterday, they’ll be the same after we update.  That did not happen here.

The difference again relates to apt pinning – specifically, pinning does not apply when we’re creating the environment.  The bootstrap process (debootstrap) does not support pinning, it only applies later when updating (apt).

This means that in specific circumstances, bootstrapping an environment might select a package that we will not select later during an update.  This breaks the key assumption – updating an old environment might not be exactly the same as creating one today.

Even so, this rarely breaks a build.  We don’t install very many packages during bootstrap, so only a small set of packages could trigger this behavior.  Even then, most builds will be fine with slightly different versions of dependencies.  It only breaks when a precise constellation of direct and indirect dependencies exposes this conflict.

Re-prioritize

Once we found the conflict, we determined that we could tweak the pinning manually as a workaround until this is addressed in debspawn.  This particular build doesn’t need to prefer byzantium-updates-proposed over the other sources.  We adjusted the pinning, and the builds succeeded.

We also opened an issue in debspawn, for a proper fix.

Moving forward

With that fixed, and util-linux published, we can move forward to more packaging updates!

If you’d like to see by-the-minute changes of code uploads to PureOS, subscribe to the PureOS-changes mailing list!  If you have suggestions or would like to get involved, the PureOS-project list is a great place to start a discussion.  We recently updated the list manager to mailman 3, so give it a try!  You can also participate from the web, thanks to HyperKitty.

We are also looking forward to soliciting feedback from our community and offering some straightforward tasks for newcomers who would like to help out.  At the moment, we need to solidify the foundation, but we’re eager to take those steps when we are ready.

Have a great end of 2024, and we will see you again in our next update!