Purism

Beautiful, Secure, Privacy-Respecting Laptops, Tablets, PCs, and Phones

Latest posts by Purism (see all)

PureOS Crimson Development Report: September 2024 - October 16, 2024
Quantum Safe Communication with Purism - September 23, 2024
Purism Releases STEP File for Librem 5 - September 23, 2024

It’s good to see you again

Welcome back! As always, we greatly appreciate the support of PureOS subscribers. Advancing PureOS benefits all Librem devices as well as the larger FLOSS ecosystem. Your support is crucial to that mission.

In our last update, PureOS development was blocked in August as the Laniakea maintainer was away. In September, we tasked another team member to start learning Laniakea and address these issues. We’ve now addressed the two issues that we discussed previously, and we expect to have a build worker processing the job queue soon. This effort was funded by Purism, providing us more resources to get Crimson out to devices.

Learning Laniakea

Laniakea is an archive management tool for Debian derivatives. It integrates with existing Debian tooling to bring together all the packages that form the PureOS archive. That includes:

Synchronizing packages from Debian, when we use the package with no changes
Building and publishing packages modified for PureOS
Hosting the archive files and the packages themselves
Providing tools to manage the archive

Expired release

Some of you may have noticed that on September 20, the PureOS release files expired. If an update failed for you around this time, that was why.

The Laniakea server was out of disk space. We will address the monitoring issues too, but first we needed to get the server working again. With the maintainer away, we had to identify this cause and address it. Then, we had Laniakea generate the release files. This worked properly.

Solving dependency issues

Synchrotron, a part of Laniakea, synchronizes packages from upstream Debian to PureOS. PureOS excludes some packages (those that we have modified, and a few that we do not import for other reasons). While many packages have modifications, Debian contains an incredible number of packages. PureOS imports most of its 58,000 packages directly.

Laniakea tells us when packages have dependency issues. That’s when a package cannot be built or installed, because it needs another package that doesn’t exist yet.

We knew that there were many issues in Crimson. Binary package issues spanned 96 pages, while source package issues produced a whopping 319 pages:

Some of these were due to packages that should be synchronized from Debian, but were not. There were no synchronization issues for those packages – they didn’t conflict with our changes. Something else was wrong.

Manual cleanup

After learning more about Laniakea’s design and how to troubleshoot this, we ran a manual sync. It gave this error:

Destination source file `pool/main/e/expat/expat_2.2.10-2+deb11u6.dsc` already exists. Can not continue

The source package already existed in the pool, but there was no record of it in the database, and Laniakea did not import the binary packages. This isn’t a normal situation, and Laniakea rightly stopped here to let us rectify it.

Evidently, the disk space issue interrupted a previous synchronization. Laniakea downloaded some source packages, but the binary packages failed because the disk was too full. These source packages were much smaller than their binaries, so this happened to several packages before the disk was completely full.

We just needed to remove the errant source package files, since they were not referenced yet.

After cleaning these up, Synchrotron was again able to sync all of the PureOS suites from Debian.

Addressing a packaging conflict

This solved some synchronization issues, but one key conflict still remained – we were missing the linux-libc-dev package on arm64. This is due to a conflict with the Librem 5 kernel package that had occurred earlier in development of Crimson.

In Debian, the linux source package builds many binary packages. Besides the kernel itself, a number of userspace tools, optional modules, and development packages are all built from the Linux kernel source. Since you can install any combination of those separately, Debian splits them up into separate binary packages. PureOS imports 45 of these binary packages from Debian. linux-libc-dev is one of these, and a large number of packages need it to compile on a Linux platform.

On PCs, we use the Debian kernel unmodified. The Librem 5 uses a different kernel based on the latest kernel LTS release.

Although a source package can produce many binary packages, those binary packages must be unique. Two source packages can’t produce binary packages with the same name. One would overwrite the other.

So, the Librem 5 Linux kernel package does not produce a linux-libc-dev binary. It produces the kernel itself and headers, nothing else. linux-libc-dev and other binaries come from the Debian arm64 kernel, then use them with the newer Librem 5 kernel. Linux presents a stable userspace ABI and API, so we can use these packages together.

Forgetting a version

At some point, the Librem 5 kernel package must have built and published a linux-libc-dev package by mistake (likely when it was being forked). This errant package had since been removed, but Laniakea would not synchronize it again from Debian:

Unable to import binary package `linux-libc-dev/6.1.106-3`: We have already seen higher version "6.4.16pureos1" in this repository/suite before.

Laniakea had remembered a newer version of linux-libc-dev, which the Librem 5 kernel had provided by mistake. Package versions should not go backwards. If they did, anyone that had installed the higher version would not see the older version as an “update”. Since Crimson is not stable for arm64 yet, we can revert the remembered package version. This had to be done manually in the database.

Following a short brush-up on SQL and tests on a replica, this package synchronized again.

99% solved

These two fixes solved an enormous number of dependency issues! Both source and binary issues now fit on a single page, and we can see the remaining dependency issues:

Going just by the numbers, that’s a 99.3% reduction in source package issues, and a 99.7% reduction in binary package issues. While it’s of course not 99% of the work, it’s always nice to see a number like that come down.

Some of these are not critical for general release, either. While those of you that work with bacterial genomes on-the-go may miss shovill and unicycler on the Librem 5, the many other updates in Crimson will make the release worthwhile even if a few packages aren’t ready yet.

Next month: the job queue

The remaining issue in Laniakea is that the job queue has a large backlog of build jobs for amd64. Due to a hosting issue, we deployed a new amd64 worker (fennel). The important jobs for Crimson and Byzantium are in line behind a lot of jobs for Landing, the unstable development distribution. As Landing won’t bootstrap right now, those jobs won’t complete, but Laniakea won’t skip over them to process the other jobs.

One option is to fix the bootstrapping issue for Landing. Otherwise, we could tweak Laniakea in some way to let the Byzantium and Crimson jobs through.

Fixing Landing is a forward fix that we wouldn’t revert later. However, with our eyes on a Crimson release, that might not be the best path in the immediate term. We’ll let you know what we find next month!

Milestone visibility

Many supporters have asked for more visibility into the PureOS work, and we would like to provide it.

We initially tracked this work in internal milestones, just because we already had tasking there. We’ve now moved the PureOS milestones to the public PureOS group, and we’ve moved the issues to appropriate repositories within that group. This way, everyone can see the work that is planned for each milestone, and the time that is logged to each issue.

Here’s the first milestone we’re targeting – an image alpha testable by clean flash:

You can see that we’ve made progress on several issues, and we’ve now completed some of the Laniakea work that was blocking the others. We also have a task to identify any remaining problems that aren’t yet known, which might result in additional work for that milestone (the “Affects milestone scope” label indicates that).

We again thank all our supporters and PureOS subscribers. Your subscriptions make this work possible, and we share these benefits with the broader FLOSS community.

PureOS Crimson Development Report: September 2024