How to be Upstream-First

• About
• Latest Posts

Purism

Beautiful, Secure, Privacy-Respecting Laptops, Tablets, PCs, and Phones

Latest posts by Purism (see all)

• PureOS Crimson Development Report: October 2024 - November 14, 2024
• PureOS Crimson Development Report: September 2024 - October 16, 2024
• Quantum Safe Communication with Purism - September 23, 2024

One of the things I’m most proud of about my work on the Librem 5 team is the productive collaborations we have with upstream free software projects, and the contributions to the broader ecosystem we’ve been able to make as a result.

Over the past four years the Librem 5 software team has worked on dozens of upstream GNOME modules, resulting in many cool new features and improvements for people using GNOME across all devices, not just on mobile. These include helping to modernize apps such as Software, Contacts, Calculator, or Geary, adding and improving Settings panels, and of course the massive leap in consistency and developer experience that came with Libhandy/Libadwaita.

The result is that we now not only have a free software mobile platform and vibrant third party app ecosystem based on the GNOME stack, but also that GNOME on the desktop has gotten much better at the same time.

Given the small size of the team what we’ve achieved is pretty incredible, especially compared to previous efforts to build a free software mobile platform.

I think a big part of the success of this initiative has been our upstream-first approach to development.

What is upstream-first?

In the free software world “upstream” refers to a project you are building something on top of. For the Librem 5 software stack upstreams include GNOME, Debian, and the Linux kernel. If you are building something on top of an upstream project, your project is “downstream” from them.

One way to build on top of free software is to develop downstream: Take the upstream code, change it until it does what you want, and ship the result. This works, and can be a quick way to get something out the door, but it comes with a very important drawback: Maintenance.

If you develop a feature downstream this means you have two choices for maintaining it longer-term. One is to periodically take the newest version of the upstream project and re-apply all your changes. This is often very difficult and time-consuming, depending on how much changed upstream. Alternatively you can also fork the upstream project entirely, but then you’re responsible for maintaining that entire code base forever. Either way you have to invest significant resources into keeping your downstream version working, in perpetuity.

This kind of downstream approach is how everything works in the Android world, from drivers to UI customizations. This is a major reason why the ecosystem is so fragmented and why devices get so few updates: It’s incredibly expensive to keep rebasing your changes to the software stack for every new version, and it gets harder as time goes on.

Upstream-first, on the other hand, means investing more time and effort up-front by making the necessary changes to the upstream project itself. This can be difficult depending on the case because it may require agreement between different stakeholders, code and design reviews, and aligning with the upstream release cycle. However, in the long run it’s a much cleaner solution because it means not having to spend time on rebasing your changes forever or maintaining a huge amount of code in the form of a fork.

In this post I want to go over some best practices for how to do upstream-first development. I’ll draw primarily on our experiences at Purism with GNOME as an upstream, as that’s what I’m personally most involved in. Most of this is relatively evergreen though, and should apply to many projects.

Understanding the terrain

Before embarking on an upstream-first development effort, it’s important to be aware of the landscape you’re navigating: Where in the stack is the best place to implement what you need? Which upstream projects would be affected by that? What are those communities like in terms of developer base, goals, momentum, history, and culture?

At the most basic level you should make sure that what you’re trying to do is actually going to be welcomed by the community upstream. If you don’t know whether it fits into the technical and UX vision upstream has, you might end up in an awkward situation medium term. For example, had we tried to build a new mobile app platform on top of a desktop stack that was ideologically opposed to supporting mobile it’s unlikely that upstream-first would have been possible. Conversely, if you pick a stack where community’s goals are well-aligned with yours it’s possible that things will end up being easier than expected, because people join the effort and help it go faster.

That said, your downstream goals might be more reconcilable with upstream ones than it can seem at a glance. For example, adaptiveness can also be framed in terms of making it easier to tile apps, so it’s nice to have even if you only care about desktop use. Aligning goals like that can be non-trivial, and is much easier if you’re intimately familiar with the inner workings of the upstream project. This is why the next point is crucial.

Hiring the right people

Probably the single most important thing for a company to do upstream-first successfully is hiring the right people. Obviously it’s important for people to have the right technical skills (e.g. GTK frontend development or UX design), but that’s only a part of the picture.

Because much of the work is done by loosely organized volunteers without clear hierarchies, free software development is very dependent on shared culture and values, as well as individual personalities and relationships. An important aspect of this is that the size of a proposed change should be proportionate to how much trust someone enjoys in the community. If you’re trying to do something big it’s much more likely to succeed if someone who has the community’s trust is leading the charge.

In the case of adaptive apps I think it’d have been a very different story if instead of people like Adrien Plazas and myself the Librem 5 team would have consisted of developers and designers with no connection to the GNOME project. The history of the Librem 5 team has also shown that people who don’t have that connection are less likely to stick around.

If you’re interested in this topic I’ve written about this in more detail in my series on how power works in the GNOME project.

To summarize: The ideal hire (or contractor) is a trusted long-term upstream contributor who is well-integrated into the community (attending hackfests, writing blog posts, giving talks etc.) and can reconcile upstream and downstream goals productively.

High level goals, long term focus

Having the right people is a great start, but if they’re made to work on the wrong things the effort can still be unsuccessful. There are companies that have employed plenty of smart upstream GNOME people over the years, but whose long-term impact has been limited because of how they focused their efforts.

Aligning upstream and downstream goals can be quite challenging, and works best when people who understand upstream goals and culture are the ones leading the charge. It also helps if it’s not imposed from above in monolithic waterfall fashion, but rather a more organic process of discussion and experimentation, iterating towards solutions together with the wider community.

There’s of course a balance to be struck between meeting downstream goals in a timely fashion and aligning with upstream. In my experience what works best in practice is hiring upstream people, giving them high-level goals, and letting them figure out the best way to get there.

For example, solution-oriented goals like “app X should have a sidebar” or “add a GSettings key for Y” are often too specific to be imposed from above and you end up with lots of drama and little progress. More high level goals like “make categories more accessible in app X”, or “offer a way for apps to access setting Y” are much more productive, because they leave room for upstream processes to play out organically.

Individual contributors pushing for specific solutions (whether they have a corporate goal in mind or not) isn’t a problem of course, and happens all the time. However, it’s important to be open-minded and go along with community consensus if it ends up going in a different direction than the one you were pushing for initially.

Collaboration over control

Ultimately the goal of upstream-first development is to not hoard power for yourself, but to share it with many other people in the community whose goals are aligned with yours. The entire reason for doing upstream-first is that you don’t have the resources to maintain the whole thing by yourself, or have realized it’s not an efficient use of your resources.

That doesn’t mean the goal is to just dump code upstream and hope someone else will maintain it, of course. The ideal case is co-maintainership with other stakeholders, including people from other companies and community volunteers. That way the project is less reliant on a single company or funding source, and there is a larger pool of people who can review merge request, make releases, and help with other maintenance tasks.

One example where this works quite well is GNOME Settings, which has a several co-maintainers, working for Endless, Red Hat, Canonical, and Purism, among others.

Short term vs. long term

When conceiving of a feature downstream it’s only natural to be impatient about the timeline for when it will actually ship. There are business goals to be met, hardware to be sold, and customers to be billed. It can take months for upstream to reach consensus on how a feature should work, for the code to be reviewed and merged, and the next release to be out the door. Who has time to wait that long?

While it’s true that upstream-first development can take longer than just doing everything downstream, it doesn’t necessarily have to. Depending on how well-integrated the developers are into the upstream community it can take about the same time to get something designed, reviewed, and merged upstream as just building it independently downstream. What really helps speed things along is employing people who co-maintain the relevant module, since it removes a lot of uncertainty around how long reviews can take.

What’s trickier is release cycles: For example, if you decide you want to add a feature to GNOME in February (when the spring GNOME release is already in its feature freeze period), you won’t get it in a release until October (when the fall release starts landing in distributions).

Of course, depending on how complex the feature is it might require changes in a bunch of modules across the stack, which all have different release cycles. In these cases there is often no way around downstream patching of some kind.

The important thing to keep in mind though is that eventually, new versions of all these modules will be released, and have everything you need. That’s ultimately what you’re working towards. Doing upstream-first development well means always focusing on this long-term view, and filling in the gaps downstream in the mean time where necessary.

Temporary downstream patching

When there is an absolute necessity to ship a feature on a specific timeline (for example because it’s needed by hardware you’re about to start shipping), there’s often no way way around some downstream patching.

For example, a project as large as the Librem 5 software stack will necessarily take years, and requires working upstream on dozens of projects, from the kernel all the way to individual GNOME apps. Because of this we initially had quite a few hacky downstream patches on apps that shipped with the phone, just to have something that works. However, we’ve always kept the big picture in mind, and have (in collaboration with the wider community) been working towards making individual apps fully adaptive, often redesigning them and making them nicer on desktop too in the process.

So downstream patches aren’t necessarily a problem, as long as you’re being thoughtful about it and working towards getting rid of them in the long run.

A few things to keep in mind:

• Don’t invest too much effort into temporary downstream patches. What goes upstream is what you’ll be using, supporting, and building upon in the long run, so making that great should be the primary focus.
• Never ship downstream patches that might break third party apps (e.g. by hacking the stylesheet to force dark mode on apps that don’t support it yet). Things that affect the app developer platform can only be done upstream.
• Wherever possible, avoid downstream patches by pushing things forward as fast as possible upstream and/or just holding off a bit longer downstream. Does a certain feature really need to ship in the next version, or could it just wait until you get it from upstream? Every hour you invest in patching could be spent making things better for everyone upstream instead.

Conclusion

I think our experience with the Librem 5 software stack and the general direction of the GNOME ecosystem over the past few years shows clearly how well the upstream-first development approach can work.

We’re of course far from perfect and there are many areas where we could do better (the GTK dialog patches are still a problem, for example), but overall I think it’s impressive how far we’ve come. This will become even more obvious once all the cool new GTK4 and Libadwaita stuff makes it to the phone!

Something I find very rewarding about my work on the Librem 5 team is that I almost never feel like what I’m doing is wasted effort, or a dead end. Anything you improve upstream on a project used in as many different places as GNOME will benefit people for a long time to come, in ways that we can probably not imagine yet.

Happy hacking!

Thanks to Alexander Mikhaylenko for helping with some factual details and screenshots.