mattst88's blog - Does Anyone Care About Fixing Bugs?

GNOME 40 available in Gentoo

GNOME 40 was released at the end of March, and yesterday I added the last bits of it to Gentoo. You may not think that's fast, and you'd be right, but it's a lot faster than any GNOME release has been added to Gentoo that I can recall. I wasn't looking to become Gentoo's GNOME maintainer when I joined the team 18 months ago. I only wanted to use a GNOME release that was a little less stale. So how did I get here?

I asked about the GNOME 3.26 status when 3.28 and 3.30 were already out. Repeat that story until I got tired of waiting and added myself to the Gentoo/GNOME team and started updating glib... then I started updating mutter and gnome-shell... then I started updating everything...

— Matt Turner (@mattst88) May 1, 2021

GNOME has two major releases per year (in March and September), so to be more than two major releases behind is significant. At least two of my coworkers on the Mesa team at Intel switched to Gentoo for one reason or another, but ultimately switched back to their old distribution because Gentoo's GNOME packages were so out of date. That was pretty disappointing to hear, but I sympathized with them.

I maintain the X11/Wayland stack in Gentoo, and I think I do a good job of keeping on top of it. I make upstream releases of X packages and contribute to Mesa professionally so I'm often able to make the upstream and downstream changes at the same time.

But for GNOME I was just a user who happened to be a Gentoo Developer, so I started by just poking and asking if there was anything I could do to help. Unfortunately the answer was "no" nearly every time.

So I just watched and occasionally asked how things were going. And occasionally GNOME updates happened, but the gap between Gentoo and upstream never really closed. GNOME 3.26 was added to Gentoo, and before significant progress was made on adding 3.28 or 3.30 a new major version 3.32 was released upstream. It looked like we were just treading water.

What's worse, there were multiple unofficial overlays often providing newer versions of GNOME than what the ::gentoo repository contained. For reasons that were never clear to me, it seemed that none of the external overlay contributors (one of whom was a full Gentoo Developer!) were willing or able to collaborate with the Gentoo GNOME team.

I started small by adding new versions of GNOME packages and making pull request on GitHub for more experienced GNOME team members to review. Unfortunately by this time, the GNOME team had only one active member.

I joined the GNOME team in October 2019 and worked around the edges, doing small version bumps of non-critical packages.

Since most of the GNOME packages were behind, I began adding the next major GNOME's glib to the tree to get extra testing. I figured if that additional testing caught issues before they could block the rest of GNOME from being updated that I could save us some time.

That worked out pretty well, and I felt a little more confident so I added the next major GNOME's mutter and gnome-shell. Kind of scary.

But that worked out well too. Users tested, filed bugs, and I fixed them. And since the most critical GNOME packages entered the ::gentoo repo long before the ancillary applications we didn't have any big surprises when it was time to ask for stabilization.

Initially I had no idea which packages were related or if there were particular problems to look out for. This knowledge existed only in the head of one Gentoo Developer, so as I squeezed it out of him (as I made mistakes and he let me know!) I began documenting it on the Wiki.

As I updated packages, I encountered various build system bugs. Gentoo naturally uncovers problems binary distributions don't notice. Whenever possible, I made a merge request upstream so that the next time we added a new version we wouldn't have to carry a patch. So far I've had 13 merge requests accepted!

Starting on March 20 I added the first bits of GNOME 40 to the tree (glib and some other packages are often released before the official release date). I added glib first, and then I figured I couldn't break anything too badly if I just bumped the GNOME games. I added gnome-shell (behind package.mask), and then sort of forgot that's where I normally stopped. Less than 8 weeks later, all of GNOME is entirely up to date in Gentoo!

The bookends of adding GNOME 40 are commits 71e9245b05e6 and b93e3e581161. In that time I made 610 commits. The vast majority are GNOME-related (511 of them by my count). Categorized, they are:

Those commits closed 120 bugs (and referenced 21 more), which made a nice dent in the Gentoo GNOME team's bug backlog. At the time of this writing, there are 514 bugs assigned to the GNOME team or with the GNOME team in the Cc list. By default, Bugzilla only shows 500 bugs on a single page, so the GNOME bug list doesn't even fit. That was a bit of a psychological hurdle for me to get started. It'll be a nice moment when we get to the other side of 500.

I hope that with the gap to upstream now closed that some other Gentoo Developers and users will be more willing to help contribute. GNOME fell behind in Gentoo because it was too much work for a single person to maintain sustainably. I've remedied the most glaring symptom of the situation but not the underlying problem. Reach out to me if you'd like to help!

Because it's fun to look at, here's the output of our gnome-bumpchecker.py tool, showing that we're indeed up-to-date on everything.

– Tags: gentoo gnome linux

Ideas for the Future of SkyOS

In December 2004, I paid thirty dollars for a SkyOS Beta membership. I was intrigued by the project. I couldn't explain it. It was just different. I ported simple software and came to be a part of the SkyOS community. I wrote SkyOS programming guides. I spent hours talking with other SkyOS users on IRC. I reported bugs and followed development closely. There was no other project that I devoted more time to than SkyOS. But for the last year or so, SkyOS has progressed at an ever slowing pace. With long periods of downtime between SkyOS releases, many users, myself included, gradually moved on to other projects. Finally, Robert made the announcement that SkyOS development is officially halted. I wasn't surprised, but I was saddened. Given what I know of the project and of Robert, and having seen in four years of development the same concerns raised, problems faced and subsequently fixed over and over again, I wrote a letter to Robert outlining what needs to be done to save his hobby of the last thirteen years.

Hi Robert,

This is Matt (mattst88) from SkyOS.

I read your response on OSNews. I want to comment on your possible resorts:

Do all of them, by

  1. Releasing SkyOS as open source/free software under the MIT or BSD license. By licensing under a very permissive license, it will allow projects from whom SkyOS has used code (e.g. Haiku) to use SkyOS's code where possible.

  2. Implementing a Linux-style development model. Source code would be available in a public git repository, of which you would be the only one with commit access. You, and you alone, would:

    1. control what goes into the repository
    2. define targets and make releases
    3. point people in the right direction

    This way, others do the heavy lifting and you help where needed.

    Once this has been done, I see a number of things that can be done to improve SkyOS.

  3. Move SkyOS-specific code in ported software upstream. As an open source project, it should be much easier to have official support from other open source projects.

  4. Reduce development time and duplication of effort by replacing SkyOS components with open source counterparts. As I have not viewed the SkyOS source, I cannot comment further on which components could be replaced.

  5. Port the Direct Rendering Manager to the SkyOS kernel. With DRM in place port Gallium3D [1], which should _greatly_ simplify graphics drivers. Using Gallium3D, SkyOS would be able to leverage support from a much wider base of developers.

  6. Finally, define a niche for SkyOS to fill. Laptops are more likely to have nonstandard hardware, thereby requiring all kinds of extra support, from strange integrated graphics chips, to countless wireless chipsets, to docking stations. (I do not mention power management, as it is becoming increasingly necessary even for desktop systems.) I therefore propose supporting only standard desktop systems.

I hope you find time for SkyOS in the future and implement my suggestions. With this plan, SkyOS should live on with community support, and you should be able to devote time to your daughter (I guess she has just turned one?) and family.

Thanks Robert, for everything. I'll turn 21 this month, graduate with a degree in Physics next year, and go to graduate school in either Computer Science or Computer Engineering the year after. I can definitely say that SkyOS has contributed heavily to developing my passion -- computers. I've learned so much from this project. I know there's still much to learn from you and from SkyOS, so I hope you continue the project.

Thanks again,

Matt Turner

[1] http://www.tungstengraphics.com/technologies/gallium3d.html
     http://www.tungstengraphics.com/wiki/index.php/Gallium3D
     https://en.wikipedia.org/wiki/Gallium3D

I sent this email almost two weeks ago. I have received no response.

As an addendum to my email: SkyOS should further limit its hardware support by supporting only x86-64 systems in 64-bit long mode.

Another long time SkyOS user told me via IRC that he believes Robert has entirely lost interest in SkyOS. Evidently, he didn't even notice that SkyOS.org was down for two to three days after OSNews ran its article.

I sincerely hope he is wrong, but I can only doubt his claim so much. If SkyOS is not dead – great, let's see how we can give it new life and contribute to projects on which we've drawn support an inspiration. If it is dead, it was a fun hobby for Robert, me, and others involved. I certainly learned a lot and cut my teeth on SkyOS.

– Tags: skyos

Does Anyone Care About Fixing Bugs?

As time goes on, alternative architectures like Alpha and PA-RISC slowly lose their userbase. Experienced developers move on to things that interest them more. Emphasis isn't put on fixing bugs for these aging platforms, and the level of support slowly erodes. Eventually a small hardcore userbase is all that is left. The Gentoo Bugzilla showed this effect on the Alpha platform. All nontrivial bugs were left to rot. What's worse, many bugs were so old that the software containing them wasn't even in Portage anymore, yet no one closed the bug report or asked if it was fixed. One, a two-and-a-half-year-old bug about a failing cipher algorithm in libmcrypt caught my eye. I decided I'd give fixing it a shot.

The project's KNOWN-BUGS file stated

- cast-256 and rc6 do not work properly on Alpha (64 bit) machines

Fittingly, the bug was filed by a developer who has since retired. An automated test suite included with libmcrypt reported a failing cipher, CAST-256. Maybe it's a bug with gcc. Months pass. If it is, it's a bug across both 3.x and 4.x series. Years pass. Maybe we'll just mask the failure.

No one seemed to want to fire up vi and check the code.

I decided I'd compile the same version side by side on my AMD64 desktop and my UP1500 Alpha. Both compile cleanly, and I can reproduce the failing case quickly. The first thing I check is the test suite itself by adding print statements and comparing the output between the AMD64 and Alpha systems. All the start-up code looks fine. The problem has to be in the library itself, which is what I expect.

Finally, I find that the results begin to vary during a function call to _mcrypt_set_key. Continuing, I slowly isolate the failing code to the k_rnd macro, then the f1 macro, and finally to the rotl32 macro.

The rotl32 macro rotates bits left in a 32-bit memory cell. The macro and its siblings look like

#define rotl32(x,n)   (((x) << ((word32)(n))) | ((x) >> (32 - (word32)(n))))
#define rotr32(x,n)   (((x) >> ((word32)(n))) | ((x) << (32 - (word32)(n))))
#define rotl16(x,n)   (((x) << ((word16)(n))) | ((x) >> (16 - (word16)(n))))
#define rotr16(x,n)   (((x) >> ((word16)(n))) | ((x) << (16 - (word16)(n))))

I confirmed that this function did yield different results on AMD64 and Alpha by writing a small test program. Guessing, I figured that this implementation wasn't compatible with Alphas and that I could easily find another working implementation. In the Linux Kernel's include/linux/bitops.h file, they had virtually the same implementation. No luck there.

After a few hours of scouring the internet for quick-fix solutions, I turned to the Alpha Architecture Handbook and look up Alpha's shift instructions, sll and srl.

SxL   Ra.rq,Rb.rq,Rc.wq
Rc <- LEFT_SHIFT (Rav, Rbv<5:0>)      !SLL
Rc <- RIGHT_SHIFT(Rav, Rbv<5:0>)      !SRL

Beyond the terse syntax, this means that only six bits of the shift argument matter. The designers did this because with the Alpha's 64-bit wide registers, it doesn't make sense to implement instructions (and circuitry) to shift more than 63 times. Just the same, the rotl32 macro is only supposed to operate on 32-bit numbers, so it doesn't make sense to rotate more than 31 times.

The result of rotating 32 times should be the same as the number input, since it would rotate the bits the entire width of the field. On Alpha though there are more than 32-bits in each register, so shifting 32 times doesn't leave the bits in place. It moves them into the upper part of the register.

By masking the shift argument and ignoring all but the first five bits, I fixed the problem.

#define rotl32(x,n)   (((x) << ((word32)(n & 31))) | ((x) >> (32 - (word32)(n & 31))))
#define rotr32(x,n)   (((x) >> ((word32)(n & 31))) | ((x) << (32 - (word32)(n & 31))))
#define rotl16(x,n)   (((x) << ((word16)(n & 15))) | ((x) >> (16 - (word16)(n & 15))))
#define rotr16(x,n)   (((x) >> ((word16)(n & 15))) | ((x) << (16 - (word16)(n & 15))))

This bug didn't affect AMD64, since it has 32-bit shift instructions as well as 64-bit. Undoubtedly though, had this been a problem on AMD64 instead of an obscure and aging architecture such as Alpha, it would have been fixed in a heartbeat.

It's amazing that such a simple fix was needed to squash a bug that (1) was reported by a Gentoo/Alpha developer, and (2) had been in the tracker for two-and-a-half years.

Now, I need to check on that Kernel code. Who knows how long it's contained this bug!

– Tags: alpha gentoo linux