Get it while it’s hot: Debian Installer 7.0 Alpha1 release.

See how yummy it is:

Time for a DXN#11 follow-up.

1. The recent xserver-xorg-input-synaptics rc4 upload solved a lot of issues, but the 1.6.0 release (just uploaded) should fix some more. Enjoy!

2. I’ve also uploaded a new xorg-server, merging from upstream server-1.12-branch to get many XI2.2 bug fixes, along with an infinite loop bug fix (also seen with synaptics).

3. Many drivers can no longer work on ia64 due to the recent changes, so we requested they be removed, which happened promptly!

4. All XSF-maintained packages build happily against X server 1.12, meaning users can get back to running apt-get dist-upgrade blindly without having to fear the consequences. Pro tip: when you see something like xserver-xorg go away during a dist-upgrade, think twice before confirming!

5. xf86-input-mtrack was recently fixed; xf86-video-glamo and xf86-video-msm fail to build (#671028, #671806), so they stay uninstallable for now. Thankfully nothing appears to depend on them, so they can be temporarily removed from testing if needs be.

6. In the meanwhile, xserver-xorg-video-intel 2.19 was released. It will probably land into experimental first, until the new server and its drivers make it into testing.

7. Andreas Beckmann asked me to mention the status of the binary drivers, so here is my take about them: fglrx still doesn’t support X server 1.12 (LOL!). The other, big fat blobby driver is installable, and supposed to work.

Long time no see…

Summary

Anyway, here are some quick news from the X packaging front, focussing on the past few weeks.

1. While I was busy doing other things, Julien took care of uploading lots of libraries (making most of them multiarch-aware) along with our usual meta-packages (like x11-apps, x11-utils, etc.), preparing for the upcoming 7.7 katamari.

2. Mesa 8.0.2 was finally uploaded to experimental, where it seems to be building fine, and working fine, at least for me.

3. Accordingly, I’m planning an upload to unstable very soon, and I won’t be disabling wayland support this time, since wayland/weston reached their first milestone with a public release (0.85). There’s nothing very interesting to see here, but since wayland support doesn’t really hurt, I thought I’d just keep it. That’s why wayland hit unstable yesterday; weston should follow after mesa. (In case somebody is in a hurry, they have been in experimental for quite some time already.)

4. Scrolling issues with the synaptics driver seem to be finally fixed thanks to rc4: #665004 was confirmed as fixed by many reporters (thanks everyone for the quick feedback after my ping). Accordingly, I’ve pinged the release team to get an age-days 3 to make it migrate faster, and a kind RM added that hint to his file, so testing is getting the fixed package thanks to this midnight’s britney run.

5. libcairo2/server/EXA/ati/nouveau fun: will be added through an edition of this post. Executive summary: downgrade libcairo2 to testing’s version if you’re seeing text corruptions.

Upcoming X server transition

X server 1.12 has been prepared in experimental for a while, from release candidates to the first stable release from server-1.12-branch. Since a bunch of drivers were uploaded to cope with that version when it shows up, we should be able to upload xorg-server 1.12 to unstable VERY SOON.

What does that mean?

Basically, that’s a transition. In details:

1. We upload xorg-server, and wait for it to be built everywhere, then we trigger binNMUs for all architectures at once when it’s ready on all of them.

2. Unlike shared libraries, there’s no old and new library packages (old+new SONAME); instead, the server itself provides different packages (because of the different input and video ABI). That means that drivers won’t be installable until they are rebuilt against the new server. Usually, we’re talking about a few dinstall runs, meaning less than a day.

Frequently Yelled Phrases:

1. Upgrades are broken! No, it just means you can’t upgrade the server alone, you need the drivers too (see above).

2. Installations are broken! Well, that’s unstable, and you probably know installability-related issues are trivially solved by pulling packages from testing when needed. This is such a case.

Please don’t report bugs on this topic, thanks already, and enjoy that new server.

Fixing hardware

Thanks to the nice Debian Swiss Knives initiative!

Fixing software

Besides trying to maintain the whole X stack as usual, and preparing the funky wayland, mesa, and weston trio for experimental, I got myself into other areas.

Debugging gzip

Jakup Wilk noticed a while ago that some multiarchified packages couldn’t be co-installed due to gzipped files being different between architectures. That was reported as #647522 against the gzip package.

While some developers consider the lack of determinism (as in: it’s not in the gzip specification, and the current implementation generates different packages anyway!) as a dead-end for the multiarch experiment, it didn’t look like undebuggable or unfixable. And indeed, as confirmed by a quick analysis: a lack of clean-up when several files are processed at once can lead to different results, depending on the order in which files are specified on the command line; the patch for that is trivial, and was later made smaller and the need for it was explained.

Working in the Release Team

I reviewed/approved a few packages for squeeze, but also took care of handling or finishing a few transitions (as seen in my hints file), among which the funny iceweasel9/libmozjs ones; release tools are nice, but one gets to learn lots of stuff at once (which isn’t a negative aspect). I hacked a tiny “collision detector” so that packages involved in several transitions can be easily spotted. Given the huge number of ongoing transitions we had lately, that didn’t look overkill.

Squashing bugs

It had been a very long time since my last working on totally random RC bugs. The Bug Squashing Party organized at IRILL was a very nice opportunity to get back to totally crazy bugs in unknown packages, but also to meet with other Debian contributors; sponsoring maintainers with good patches on their first try was nice, but walking other contributors through their first patches sent to the BTS or through their first NMUs was nice too.

Waving good bye to yada

Back in the Mérida QA meeting I attended in 2007, there were already jokes about how bad yada was, and how it should die in a fire etc. When I started looking at the RC bug list, I quickly switched to scrolling it backwards, and I wondered how much work was left. The details can be seen in #660548, and the result is: yada is gone! sanity++;

packages.debian.org

Both long descriptions and translations for packages in some suites disappeared after the switch to Description-md5. Thanks to a quick and reduced packages.debian.org setup (mostly: en+de for squeeze+sid+experimental), I managed to find my way through Perl and DB files to propose patches for #657557. I’m still waiting for a confirmation, but in case it works fine, we could even get a fix for DDTP/DDTSS.

dpkg was uploaded to experimental a few times lately, let’s sum it up quickly:

1. It started with my NMU on 2012-01-31, since I hadn’t received any reasons not to upload (except “NACK”, which doesn’t count as such in my book).
2. It got reverted a few minutes after that.
3. Since this was getting nowhere, the Debian Technical Committee finally got involved, and voted in a few days only (wow, thanks!) for an “it’s time to experiment” decision.
4. A new dpkg upload to experimental finally happened. One shall note the prominent “This is a WIP release, command line interfaces will change.” notice.

I’ve updated my previous dpkg with multiarch page accordingly, only keeping the interesting bits.

Playing around with grep-aptavail -F Multi-Arch $i -sPackage (with $i being same or foreign), I found some bugs:

• #658792: libstdc++6-4.6-dbg: libstdc++.so.6.0.16-gdb.py is not an ELF file
• #658793: libisl-dbg: libisl.so.8.0.0-gdb.py is not an ELF file

Also, some wishlist bugs (keeping the prominent notice in mind):

• #658794: apt-file: Please implement multiarch support
• #658795: reportbug: Please report foreign architectures

Finally, one dpkg improvement and one possible dpkg bug:

• #658812: dpkg: Please add a hint in arch_remove about installed foreign packages
• #658814: dpkg: Inconsistent installation state with buggy multiarchified packages?

(This page was edited a few times, details are available.)

Grab it while it’s hot!

Thanks to the hard work of dpkg developers and many (generations of) developers, multiarch is becoming a reality.

If you want to give it a try, install dpkg from experimental, add a foreign architecture, and start trying to install packages. Example on amd64:

# dpkg --add-architecture i386

# dpkg --print-foreign-architectures
i386

# apt-get update
[ lots of amd64 and i386 Packages files get downloaded ]

# apt-get install mksh:i386
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  ed:i386
The following NEW packages will be installed:
  mksh:i386
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 414 kB of archives.
After this operation, 707 kB of additional disk space will be used.
Get:1 http://ftp.fr.debian.org/debian/ sid/main mksh i386 40.4-2 [414 kB]
Fetched 414 kB in 0s (664 kB/s)
Selecting previously unselected package mksh.
(Reading database ... 171933 files and directories currently installed.)
Unpacking mksh (from .../archives/mksh_40.4-2_i386.deb) ...
Processing triggers for menu ...
Processing triggers for man-db ...
Setting up mksh (40.4-2) ...
update-alternatives: using /bin/mksh to provide /bin/ksh (ksh) in auto mode.
Processing triggers for menu ...

# mksh
$ ldd $(which mksh)
linux-gate.so.1 =>  (0xf779c000)
libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xf75d6000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf75b9000)
/lib/ld-linux.so.2 (0xf779d000)

Of course, installing an i386 mksh package isn’t exactly what multiarch is about. Dozens of packages have been patched already to add Multi-Arch fields, but until their (recursive) dependencies have been multiarch-ified, foreign packages can be uninstallable, as can be seen below, with the usual why is it uninstallable? hunt (shortened output):

# apt-get install psutils:i386
The following packages have unmet dependencies:
 psutils:i386 : Depends: libpaper1:i386 but it is not going to be installed

# apt-get install libpaper1:i386
The following packages have unmet dependencies:
 libpaper1:i386 : Depends: ucf:i386 (>= 0.28) but it is not installable
                  Recommends: libpaper-utils:i386 but it is not going to be installed

# apt-get install ucf:i386
E: Package 'ucf:i386' has no installation candidate

Another example, successful handling of the installation of a foreign package, while it’s already installed with the host architecture:

# apt-get install xz-utils:i386
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  liblzma5:i386
Suggested packages:
  xz-lzma:i386
The following packages will be REMOVED:
  xz-utils
The following NEW packages will be installed:
  liblzma5:i386 xz-utils:i386
0 upgraded, 2 newly installed, 1 to remove and 9 not upgraded.
Need to get 440 kB of archives.
After this operation, 410 kB of additional disk space will be used.
Do you want to continue [Y/n]?
Get:1 http://ftp.fr.debian.org/debian/ sid/main liblzma5 i386 5.1.1alpha+20110809-3 [205 kB]
Get:2 http://ftp.fr.debian.org/debian/ sid/main xz-utils i386 5.1.1alpha+20110809-3 [235 kB]
Fetched 440 kB in 0s (478 kB/s)
dpkg: xz-utils: dependency problems, but removing anyway as you requested:
 dpkg depends on xz-utils.
 xz-lzma depends on xz-utils.
 dpkg-dev depends on xz-utils.
(Reading database ... 171952 files and directories currently installed.)
Removing xz-utils ...
Processing triggers for man-db ...
Selecting previously unselected package liblzma5:i386.
(Reading database ... 171908 files and directories currently installed.)
Unpacking liblzma5:i386 (from .../liblzma5_5.1.1alpha+20110809-3_i386.deb) ...
Selecting previously unselected package xz-utils.
Unpacking xz-utils (from .../xz-utils_5.1.1alpha+20110809-3_i386.deb) ...
Processing triggers for man-db ...
Setting up liblzma5:i386 (5.1.1alpha+20110809-3) ...
Setting up xz-utils (5.1.1alpha+20110809-3) ...

# dpkg -l xz-utils xz-utils:i386 'liblzma*:*'
[ … ]
un  xz-utils            <none>
ii  xz-utils            5.1.1alpha+20110809-3
ii  liblzma5:amd64      5.1.1alpha+20110809-3
ii  liblzma5:i386       5.1.1alpha+20110809-3

# ldd $(which xz)
    linux-gate.so.1 =>  (0xf776b000)
    liblzma.so.5 => /usr/lib/i386-linux-gnu/liblzma.so.5 (0xf7724000)
    librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xf771b000)
    libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xf7701000)
    libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xf75a4000)
    /lib/ld-linux.so.2 (0xf776c000)

# zgrep -a '_("Activities")' gnome-shell-3.2.2.1.tar.xz
        this._label = new St.Label({ text: _("Activities") });

zgrep is a shell script, but it calls xz, which is from the i386 package, and everything runs just fine. Running it through strace -f -e '', I discovered those messages, which I had never seen before:

[ Process PID=8900 runs in 32 bit mode. ]
[ Process PID=8899 runs in 64 bit mode. ]
[ Process PID=8900 runs in 32 bit mode. ]
[ Process PID=8899 runs in 64 bit mode. ]
…

What’s next?

We’re late! It’s time to check what happens with that dpkg package, report bugs, and convert more libraries! Please think of the kittens, and coordinate with the release team to make sure you don’t delay a transition when uploading a shiny, multiarch-ified package. In a nutshell, if you received a patch from Steve Langasek or Riku Voipio, it’s a good indication your package will be helpful quickly when it’s multiarchified.

Since zlib is directly depended on by 2000+ packages, #569697 was pinged right after the dpkg upload; but many other packages will need patches and heavy testing.

Hurry up, the freeze is coming, we need to shake it up as soon as possible!

It’s this time of the year again: a new X release approaches. The upstream schedule has been delayed a tiny bit so that the multitouch changes could land in master during this merge window. Which is what happened, a few days before Christmas.

The other components involved in XI2.2 (aka. “multitouch”) were packaged in experimental while the server side was getting prepared:

• inputproto
• libXi

Given we reached the 1.12 RC1 stage, it’s a good idea to start building drivers against the new server to make it possible for users to perform some tests. And as usual, a new server means bumped input and video ABI, meaning a rebuild for the drivers to pick new dependencies. Some of them also needed a few fixes to make them build against the new server. Nothing that can’t be fixed with a few merges from upstream master branches.

The usual set of “major” drivers was uploaded to experimental:

• xserver-xorg-input-evdev
• xserver-xorg-input-keyboard
• xserver-xorg-input-mouse
• xserver-xorg-input-synaptics
• xserver-xorg-input-void
• xserver-xorg-video-ati
• xserver-xorg-video-dummy
• xserver-xorg-video-nouveau
• xserver-xorg-video-fbdev
• xserver-xorg-video-intel
• xserver-xorg-video-vesa

Significant change on the intel side: a snapshot of the current master branch was created, and packaged with SNA support enabled. SandyBridge's New Acceleration has apparently reached a point where it can be published to the masses, so here we go. Some details about it can be found in the initial commit.

Also, for those wondering about the xorg-server FTBFS on some architectures, fixes are pending.

Happy new year.

(It’s a new day, it’s a new bug…)

A regression appeared in the synaptics input driver, between the 1.4.1 and the 1.5.0 release, as reported in Debian #649002 by many users. Basically, no more touchpad on PowerPC (hi, iBook G4 users!). :-(

What follows is how I tracked it down. The upstream bug report is fdo#43728. Anyone could have done so, there’s no black magic involved. A little hint maybe: git bisect is really easy on X drivers, since one can build and test without the Debian packaging bits.

1st step: Getting started

You need sources and build dependencies, nothing fancy. Since packages might be named a bit differently, there’s a reference to the relevant upstream repositories in debian/watch for X packages.

# You need git here:
debcheckout xserver-xorg-input-synaptics synaptics.git
cd synaptics.git
# Let’s get the upstream tags and branches:
cat debian/watch
git remote add upstream git://anongit.freedesktop.org/xorg/driver/xf86-input-synaptics
git fetch --all
# Let’s install build dependencies:
sudo apt-get build-dep xserver-xorg-input-synaptics

2nd step: Reproducing

It was said 1.5.0 was the first known buggy version, so let’s make sure:

# Get the appropriate tag:
git checkout xf86-input-synaptics-1.5.0
# Prepare the build system:
autoreconf -vfi
# X finds its modules under /usr, not /usr/local:
./configure --prefix=/usr
# Build and install:
make && sudo make install

Now time to restart the display manager, and confirm that the pointer is not moving → bug reproduced.

Now let’s check the 1.4.1 release isn’t affected:

# Clean everything:
git clean -xdf
git checkout xf86-input-synaptics-1.4.1
autoreconf -vfi
./configure --prefix=/usr
make && sudo make install

And after a display manager restart, the pointer is moving again, good! For reference, the log has very different lines about appletouch, so it can be used to programmatically learn about the status (good or bad).

3rd step: Narrowing it down

So, we have a known good version and a known bad version. Time for a binary search to find the guilty commit which introduced that regression!

git bisect start
git bisect good xf86-input-synaptics-1.4.1
git bisect bad  xf86-input-synaptics-1.5.0

You don’t even need to know that 1.4.x and 1.5.x live in different branches, git figures that out for you:

Bisecting: a merge base must be tested
[de0dfb76444ad4160468d00515876c91a9fa20bf] synaptics 1.4.0

It’s just a matter of running autoreconf … && ./configure … && make && sudo make install && sudo /etc/init.d/xdm restart every step of the way, so it’s pretty easy.

No wonder, 1.4.0 was working OK, so it can be recorded as such through git bisect good, and the binary search goes between that commit and 1.5.0. After a few iterations of git bisect good and git bisect bad, we get to the commit reported on the upstream bug.

As an extra bonus, mentioning the upstream bug number/URL in the Debian bug means it can be marked as forwarded there and we’ll receive automatic notifications when its status changes, thanks to bts-link.

Now, if you want to provide a patch for this bug, you may want to try and revert this commit.

4th step: Providing a patch

Let’s go back to the affected release and try to revert the guilty commit:

git checkout xf86-input-synaptics-1.5.0
git revert 13543b156d78bc4d01a19844a5ee8f283269621b

Unfortunately, some later commits modified the affected areas so we’re running into conflicts:

error: could not revert 13543b1... eventcomm: replace synaptics-custom TEST_BIT with server's BitIsOn.
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

Let’s try to understand what the offending commit did: it replaced a set of macros with a single one, taken from the server. Both the old TEST_BIT() and the new BitIsOn() take 2 arguments, but the order is reversed. Reverting it should only be a matter of introducing the TEST_BIT() macro again (along with the two other ones it needs), and using it.

The problem is replacing all BitIsOn(foo,bar) with TEST_BIT(bar,foo): foo and bar might not be trivial, they could be complex C expressions, and a replacement in your favourite editor might not get all occurrences right.

Thankfully, there’s a nice tool to handle such things, called coccinelle. I won’t go into details, just show how it can help here. Basically you just need to craft a tiny patch, called a semantic patch, which describes what I wrote in the previous paragraph. You only need to declare two expressions, say a and b, and declare that all BitIsOn(a,b) must be replaced with TEST_BIT(b,a). Here’s the syntax:

@@
expression a,b;
@@
-BitIsOn(a,b)
+TEST_BIT(b,a)

Save it under testbit.cocci and apply it through spatch (from the coccinelle package), asking it to perform in-place replacement (we’re in a git checkout copy, remember):

# Apply:
spatch -sp_file testbit.cocci -in_place .
# Profit:
git diff

Then you can run git commit -s, write a nice commit message, send it upstream, and enjoy the rest of the night.

Coming soon to a mirror near you: fixing commit. As mentioned on the upstream bug report, I got the Debian reference wrong in the commit message, I really meant #649002.

If you’re having issues with video playback in unstable on Intel hardware, this post is for you.

Some buffer freeing code was added in libdrm 2.4.28 to try and fix exhaustion of per-process limits. Some safeguards (assert()) were put in place to see whether some other components needed fixing (like unbalanced map()/unmap()). It’s hard to miss those, since the server crashes when something’s wrong. Good news: for affected users, downgrading to libdrm 2.4.27 (currently available in testing) should restore a functional setup.

So far, patches have floated around for libva[1], xserver-xorg-video-intel[1,2] and libdrm[1,2,3] itself. Since libdrm has already been exposed to a wide audience for a while, a new upstream release is planned in a few days, disabling those assert()s. Other packages will then have some extra time to get new releases with the appropriate patches.

this is not a robbery, only a regular X server branch switch, from 1.10 to 1.11. That means X drivers (for input and video) need to be rebuilt against the new server, which is happening through binNMUs. That also means the X stack from sid might be uninstallable for a few hours, but impatient people can still use the stack from wheezy in the meanwhile.

Please refrain from reporting uninstallability bugs. That’s expected, can’t be avoided, and only for a few hours (assuming no driver starts to Fail To Build From Source).

Many thanks to the following people, who joined the “let’s do a pre-upload crash test” effort:

• Marc Dequènes
• Jakub Wilk
• David Bremner