debian
KiBi’s blog
http://mraw.org/blog/tags/debian/
KiBi’s blog
ikiwiki
2019-08-17T06:11:30Z
Sending HTML messages with Net::XMPP (Perl)
http://mraw.org/blog/2019/08/17/Sending_HTML_messages_with_Net_XMPP/
2019-08-17T06:11:30Z
2019-08-17T06:15:00Z
<h3>Executive summary</h3>
<p>It’s perfectly possible! <a href="http://mraw.org/blog/tags/debian/#html-demo">Jump to the HTML demo!</a></p>
<h3>Longer version</h3>
<p>This started with a very simple need: wanting to improve the
notifications I’m receiving from various sources. Those include:</p>
<ul>
<li>changes or failures reported during <a href="https://puppet.com">Puppet</a>
runs on my own infrastructure, and on at a customer’s;</li>
<li>build failures for the <a href="https://d-i.debian.org/daily-images/daily-build-overview.html">Debian Installer</a>;</li>
<li>changes in banking amounts;</li>
<li>and lately: build status for jobs in a customer’s Jenkins instance.</li>
</ul>
<p>I’ve been using plaintext notifications for a number of years but I
decided to try and pimp them a little by adding some colors.</p>
<p>While the XMPP-sending details are usually hidden in a local module,
here’s a small self-contained example: connecting to a server, sending
credentials, and then sending a message to someone else. Of course,
one might want to tweak the <code>Configuration</code> section before trying to run
this script…</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc">#!/usr/bin/perl</span>
<span class="hl kwa">use</span> strict<span class="hl opt">;</span>
<span class="hl kwa">use</span> warnings<span class="hl opt">;</span>
<span class="hl kwa">use</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">;</span>
<span class="hl slc"># Configuration:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$hostname</span> <span class="hl opt">=</span> <span class="hl str">'example.org'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$username</span> <span class="hl opt">=</span> <span class="hl str">'bot'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$password</span> <span class="hl opt">=</span> <span class="hl str">'call-me-alan'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$resource</span> <span class="hl opt">=</span> <span class="hl str">'demo'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$recipient</span> <span class="hl opt">=</span> <span class="hl str">'human</span><span class="hl ipl">@example</span><span class="hl str">.org'</span><span class="hl opt">;</span>
<span class="hl slc"># Open connection:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$con</span> <span class="hl opt">=</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">::</span>Client<span class="hl opt">-></span><span class="hl kwd">new</span><span class="hl opt">();</span>
<span class="hl kwc">my</span> <span class="hl kwb">$status</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">Connect</span><span class="hl opt">(</span>
hostname <span class="hl opt">=></span> <span class="hl kwb">$hostname</span><span class="hl opt">,</span>
connectiontype <span class="hl opt">=></span> <span class="hl str">'tcpip'</span><span class="hl opt">,</span>
tls <span class="hl opt">=></span> <span class="hl num">1</span><span class="hl opt">,</span>
ssl_ca_path <span class="hl opt">=></span> <span class="hl str">'/etc/ssl/certs'</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
<span class="hl kwc">die</span> <span class="hl str">'XMPP connection failed'</span>
<span class="hl kwa">if</span> <span class="hl opt">!</span> <span class="hl kwc">defined</span><span class="hl opt">(</span><span class="hl kwb">$status</span><span class="hl opt">);</span>
<span class="hl slc"># Log in:</span>
<span class="hl kwc">my</span> <span class="hl kwb">@result</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">AuthSend</span><span class="hl opt">(</span>
hostname <span class="hl opt">=></span> <span class="hl kwb">$hostname</span><span class="hl opt">,</span>
username <span class="hl opt">=></span> <span class="hl kwb">$username</span><span class="hl opt">,</span>
password <span class="hl opt">=></span> <span class="hl kwb">$password</span><span class="hl opt">,</span>
resource <span class="hl opt">=></span> <span class="hl kwb">$resource</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
<span class="hl kwc">die</span> <span class="hl str">'XMPP authentication failed'</span>
<span class="hl kwa">if</span> <span class="hl kwb">$result</span><span class="hl opt">[</span><span class="hl num">0</span><span class="hl opt">]</span> <span class="hl kwc">ne</span> <span class="hl str">'ok'</span><span class="hl opt">;</span>
<span class="hl slc"># Send plaintext message:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$msg</span> <span class="hl opt">=</span> <span class="hl str">'Hello, World!'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$res</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">MessageSend</span><span class="hl opt">(</span>
to <span class="hl opt">=></span> <span class="hl kwb">$recipient</span><span class="hl opt">,</span>
body <span class="hl opt">=></span> <span class="hl kwb">$msg</span><span class="hl opt">,</span>
type <span class="hl opt">=></span> <span class="hl str">'chat'</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
<span class="hl kwc">die</span><span class="hl opt">(</span><span class="hl str">'ERROR: XMPP message failed'</span><span class="hl opt">)</span>
<span class="hl kwa">if</span> <span class="hl kwb">$res</span> <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">;</span>
</pre></div>
<p>For reference, here’s what the XML message looks like in Gajim’s XML
console (on the receiving end):</p>
<div class="highlight-xml"><pre class="hl"><span class="hl kwa"><message</span> type='chat' to='human@example.org' from='bot@example.org/demo'<span class="hl kwa">></span>
<span class="hl kwa"><body></span>Hello, World!<span class="hl kwa"></body></span>
<span class="hl kwa"></message></span>
</pre></div>
<p>Issues start when one tries to send some HTML message, e.g. with the
last part changed to:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc"># Send plaintext message:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$msg</span> <span class="hl opt">=</span> <span class="hl str">'This is a <b>failing</b> test'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$res</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">MessageSend</span><span class="hl opt">(</span>
to <span class="hl opt">=></span> <span class="hl kwb">$recipient</span><span class="hl opt">,</span>
body <span class="hl opt">=></span> <span class="hl kwb">$msg</span><span class="hl opt">,</span>
type <span class="hl opt">=></span> <span class="hl str">'chat'</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
</pre></div>
<p>as that leads to the following message:</p>
<div class="highlight-xml"><pre class="hl"><span class="hl kwa"><message</span> type='chat' to='human@example.org' from='bot@example.org/demo'<span class="hl kwa">></span>
<span class="hl kwa"><body></span>This is a <span class="hl kwd">&lt;</span>b<span class="hl kwd">&gt;</span>failing<span class="hl kwd">&lt;</span>/b<span class="hl kwd">&gt;</span> test<span class="hl kwa"></body></span>
<span class="hl kwa"></message></span>
</pre></div>
<p>So tags are getting encoded and one gets to see the uninterpreted
“HTML code”.</p>
<p>Trying various things to embed that inside <code><body></code> and <code><html></code> tags,
with or without namespaces, led nowhere.</p>
<p>Looking at a message sent from Gajim to Gajim (so that I could craft
an HTML message myself and inspect it), I’ve noticed it goes this way (edited to concentrate on important parts):</p>
<div class="highlight-xml"><pre class="hl"><span class="hl kwa"><message</span> <span class="hl kwb">xmlns</span>=<span class="hl str">"jabber:client"</span> <span class="hl kwb">to</span>=<span class="hl str">"human@example.org/Gajim"</span> <span class="hl kwb">type</span>=<span class="hl str">"chat"</span><span class="hl kwa">></span>
<span class="hl kwa"><body></span>Hello, World!<span class="hl kwa"></body></span>
<span class="hl kwa"><html</span> <span class="hl kwb">xmlns</span>=<span class="hl str">"http://jabber.org/protocol/xhtml-im"</span><span class="hl kwa">></span>
<span class="hl kwa"><body</span> <span class="hl kwb">xmlns</span>=<span class="hl str">"http://www.w3.org/1999/xhtml"</span><span class="hl kwa">></span>
<span class="hl kwa"><p></span>Hello, <span class="hl kwa"><strong></span>World<span class="hl kwa"></strong></span>!<span class="hl kwa"></p></span>
<span class="hl kwa"></body></span>
<span class="hl kwa"></html></span>
<span class="hl kwa"></message></span>
</pre></div>
<p>Two takeaways here:</p>
<ul>
<li><p>The message is send both in plaintext and in HTML. It seems Gajim
archives the plaintext version, as opening the history/logs only
shows the textual version.</p></li>
<li><p>The fact that the HTML message is under a different path
(<code>/message/html</code> as opposed to <code>/message/body</code>) means that one
cannot use the <code>MessageSend</code> method to send HTML messages…</p></li>
</ul>
<p>This was verified by checking the documentation and code of the
<code>Net::XMPP::Message</code> module. It comes with various <em>getters</em> and
<em>setters</em> for attributes. Those are then automatically collected when
the message is serialized into XML (through the <code>GetXML()</code>
method). Trying to add handling for a new HTML attribute would mean being
extra careful as that would need to be treated with <code>$type = 'raw'</code>…</p>
<p>Oh, wait a minute! While using <code>git grep</code> in the sources, looking for
that <code>raw</code> type thing, I’ve discovered what sounded promising: an
<code>InsertRawXML()</code> method, that doesn’t appear anywhere in either the
code or the documentation of the <code>Net::XMPP::Message</code> module.</p>
<p>It’s available, though! Because <code>Net::XMPP::Message</code> is derived from
<code>Net::XMPP::Stanza</code>:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl kwa">use</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">::</span>Stanza<span class="hl opt">;</span>
<span class="hl kwa">use</span> base <span class="hl kwc">qw</span><span class="hl opt">(</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">::</span>Stanza <span class="hl opt">);</span>
</pre></div>
<p>which then in turn comes with this function:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc">##############################################################################</span>
<span class="hl slc">#</span>
<span class="hl slc"># InsertRawXML - puts the specified string onto the list for raw XML to be</span>
<span class="hl slc"># included in the packet.</span>
<span class="hl slc">#</span>
<span class="hl slc">##############################################################################</span>
</pre></div>
<p>Let’s put that aside for a moment and get back to the <code>MessageSend()</code>
method. It wants parameters that can be passed to the
<code>Net::XMPP::Message</code> <code>SetMessage()</code> method, and here is its entire
code:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc">###############################################################################</span>
<span class="hl slc">#</span>
<span class="hl slc"># MessageSend - Takes the same hash that Net::XMPP::Message->SetMessage</span>
<span class="hl slc"># takes and sends the message to the server.</span>
<span class="hl slc">#</span>
<span class="hl slc">###############################################################################</span>
<span class="hl kwa">sub</span> MessageSend
<span class="hl opt">{</span>
<span class="hl kwc">my</span> <span class="hl kwb">$self</span> <span class="hl opt">=</span> <span class="hl kwc">shift</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$mess</span> <span class="hl opt">=</span> <span class="hl kwb">$self</span><span class="hl opt">-></span><span class="hl kwd">_message</span><span class="hl opt">();</span>
<span class="hl kwb">$mess</span><span class="hl opt">-></span><span class="hl kwd">SetMessage</span><span class="hl opt">(</span><span class="hl kwb">@_</span><span class="hl opt">);</span>
<span class="hl kwb">$self</span><span class="hl opt">-></span><span class="hl kwd">Send</span><span class="hl opt">(</span><span class="hl kwb">$mess</span><span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<p>The first assignment is basically equivalent to <code>my $mess = Net::XMPP::Message->new();</code>, so
what this function does is: creating a <code>Net::XMPP::Message</code> for us, passing all
parameters there, and handing the resulting object over to the <code>Send()</code>
method. All in all, that’s merely a proxy.</p>
<p><a name="html-demo"></a></p>
<h4>HTML demo</h4>
<p>The question becomes: what if we were to create that object ourselves,
then tweaking it a little, and then passing it directly to <code>Send()</code>,
instead of using the slightly limited <code>MessageSend()</code>? Let’s see what
the rewritten sending part would look like:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc"># Send HTML message:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$text</span> <span class="hl opt">=</span> <span class="hl str">'This is a working test'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$html</span> <span class="hl opt">=</span> <span class="hl str">'This is a <b>working</b> test'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$message</span> <span class="hl opt">=</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">::</span>Message<span class="hl opt">-></span><span class="hl kwd">new</span><span class="hl opt">();</span>
<span class="hl kwb">$message</span><span class="hl opt">-></span><span class="hl kwd">SetMessage</span><span class="hl opt">(</span>
to <span class="hl opt">=></span> <span class="hl kwb">$recipient</span><span class="hl opt">,</span>
body <span class="hl opt">=></span> <span class="hl kwb">$text</span><span class="hl opt">,</span>
type <span class="hl opt">=></span> <span class="hl str">'chat'</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
<span class="hl kwb">$message</span><span class="hl opt">-></span><span class="hl kwd">InsertRawXML</span><span class="hl opt">(</span><span class="hl str">"<html><body></span><span class="hl ipl">$html</span><span class="hl str"></body></html>"</span><span class="hl opt">);</span>
<span class="hl kwc">my</span> <span class="hl kwb">$res</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">Send</span><span class="hl opt">(</span><span class="hl kwb">$message</span><span class="hl opt">);</span>
</pre></div>
<p>And tada!</p>
<div class="highlight-xml"><pre class="hl"><span class="hl kwa"><message</span> type='chat' to='human@example.org' from='bot@example.org/demo'<span class="hl kwa">></span>
<span class="hl kwa"><body></span>This is a working test<span class="hl kwa"></body></span>
<span class="hl kwa"><html></span>
<span class="hl kwa"><body></span>This is a <span class="hl kwa"><b></span>working<span class="hl kwa"></b></span> test<span class="hl kwa"></body></span>
<span class="hl kwa"></html></span>
<span class="hl kwa"></message></span>
</pre></div>
<p>I’m absolutely no expert when it comes to XMPP standards, and one
might need/want to set some more metadata like <code>xmlns</code> but I’m happy
enough with this solution that I thought I’d share it as is. <code>;)</code></p>
Quick recap of 2017
http://mraw.org/blog/2018/01/15/Quick_recap_of_2017/
2018-01-15T11:02:01Z
2018-01-15T11:00:00Z
<p>I haven’t been posting anything on my personal blog in a long while,
let’s fix that!</p>
<p>Partial reason for this is that I’ve been busy documenting progress on
the Debian Installer on
<a href="https://debamax.com/blog/">my company’s blog</a>. So far, the following
posts were published there:</p>
<ul>
<li><a href="https://debamax.com/blog/2016/11/22/debian-installer-stretch-alpha-8/">Debian Installer: Stretch Alpha 8 released</a>
with details on the release process, and on the <code>debootstrap</code>
attempt regarding merged-<code>/usr</code>
(granted, that one was from late 2016).</li>
<li><a href="https://debamax.com/blog/2017/02/13/debian-installer-stretch-rc-2/">Debian Installer: Stretch RC 2 released</a>:
wrapping up both RC 1 and RC 2 for Stretch, mentioning major changes
instead of all the tiny details one would usually find in the release
announcements published on the <code>debian-devel-announce@</code> mailing list.</li>
<li><a href="https://debamax.com/blog/2017/08/04/debian-installer-stretch-final/">Debian Installer: Stretch released</a>:
aggregating RC 3 to RC 5 this time, since the last few weeks before
the Stretch release date were quite busy!</li>
</ul>
<p>After the Stretch release, it was time to attend
<a href="http://debconf17.debconf.org/">DebConf’17 in Montreal, Canada</a>. I’ve
<a href="https://debconf17.debconf.org/talks/167/">presented the latest news</a>
on the Debian Installer front there as well. This included a quick
demo of
<a href="https://anonscm.debian.org/cgit/d-i/kadit">my little framework</a> which
lets me run automatic installation tests. Many attendees mentioned
<a href="http://open.qa/">openQA</a> as the current state of the art technology
for OS installation testing, and Philip Hands started
<a href="https://lists.debian.org/debian-boot/2017/11/msg00292.html">looking into it</a>.
Right now, my little thing is still useful as it is, helping me
reproduce regressions quickly, and testing bug fixes… so I haven’t
been trying to port that to another tool yet.</p>
<p>I also gave another presentation in two different contexts: once at a
<a href="http://rrll.alliance-libre.org/">local FLOSS meeting in Nantes, France</a>
and once during the
<a href="https://wiki.debian.org/DebianEvents/fr/2017/Toulouse">mini-DebConf in Toulouse, France</a>. Nothing
related to Debian Installer this time, as the topic was how I helped a
company upgrade thousands of machines from Debian 6 to Debian 8 (and
to Debian 9 since then). It was nice to have
<a href="https://evolix.com/">Evolix</a> people around, since we shared our
respective experience around automation tools like Ansible and Puppet.</p>
<p>After the mini-DebConf in Toulouse, another event: the
<a href="https://wiki.debian.org/DebianEvents/gb/2017/MiniDebConfCambridge">mini-DebConf in Cambridge, UK</a>. I
tried to give a lightning talk about <em>“how snapshot.debian.org helped
saved the release(s)”</em> but clearly speed was lacking, and/or I had too
many things to present, so that didn’t work out as well as I
hoped. Fortunately, no time constraints when I presented that during a
Debian meet-up in Nantes, France. <code>:)</code></p>
<p>Since
<a href="https://tails.boum.org/news/reproducible_Tails/">Reproducible Tails builds</a>
were announced, it seemed like a nice opportunity to document how my
company
<a href="https://debamax.com/blog/2017/11/20/tails-early-work-on-reproducibility/">got involved into early work on reproducibility</a>
for the <a href="https://tails.boum.org/">Tails project</a>.</p>
<p>On an administrative level, I’m already done with all the paperwork
related to the second financial year. <code>\o/</code></p>
<p>Next things I’ll likely write about: the first two D-I Buster Alpha
releases (many blockers kept popping up, it was really hard to
release), and a few more recent release critical bug reports.</p>
Freelance Debian consultant: running DEBAMAX
http://mraw.org/blog/2016/08/08/Freelance_Debian_consultant_running_DEBAMAX/
2016-08-22T06:00:01Z
2016-08-22T06:00:00Z
<h3>Executive summary</h3>
<p>Since October 2015, I've been running a
<acronym title="Free/Libre and Open-Source Software">FLOSS</acronym> consulting company,
specialized on Debian, called <a href="https://debamax.com/">DEBAMAX</a>.</p>
<p><a href="https://debamax.com/"><img src="http://mraw.org/blog/2016/08/08/debamax-logo.png" width="300" height="60" alt="DEBAMAX logo" class="img" /></a></p>
<h3>Longer version</h3>
<p>Everything started two years ago. Back then I blogged about one of the
biggest changes in my life: trying to find the right balance between
volunteer work as a Debian Developer, and entrepreneurship as a
<a href="http://mraw.org/blog/2014/09/15/Freelance_Debian_consultant">Freelance Debian consultant</a>.
Big change because it meant giving up the comfort of the salaried
world, and figuring out whether working this way would be sufficient to earn
a living…</p>
<p>I experimented for a while under a simplified status. It comes with a
number of limitations but that’s a huge win compared to France’s heavy
company-related administrativia. Here’s what it looked like,
everything being done online:</p>
<ul>
<li><p>1 registration form to begin with: wait a few days, get an
identifier from <a href="https://en.wikipedia.org/wiki/INSEE">INSEE</a>,
mention it in your invoices, there you go!</p></li>
<li><p>4 tax forms a year: taxes can be declared monthly or quarterly, I
went for the latter.</p></li>
</ul>
<p>A number of things became quite clear after a few months:</p>
<ul>
<li><p>I love this new job! Sharing my Debian knowledge with customers,
and using it to help them build/improve/stabilise their products
and their internal services feels great!</p></li>
<li><p>Even if I wasn't aware of that initially, it seems like I've got a
decent network already: Debian Developers, former coworkers, and
friends thought about me for their Debian-related tasks. It was
nice to hear about their needs, say yes, sign paperwork, and start working
right away!</p></li>
<li><p>While I'm trying really hard not to get too optimistic (achieving a
given turnover on the first year doesn't mean you're guaranteed to
do so again the following year), it seemed to go well enough for me to
consider switching from this simplified status to a full-blown
company.</p></li>
</ul>
<p>Thankfully I was eligible to being accompanied by the local Chamber of
Commerce and Industry (<a href="http://www.rennes.cci.fr/">CCI Rennes</a>), which
provides teaching sessions for new entrepreneurs, coaching, and
meeting opportunities (accountants, lawyers, insurance companies, …).
Summer in France is traditionally rather quiet (<em>read:</em> almost everybody is on
vacation), so <a href="https://debamax.com/">DEBAMAX</a> officially started
operating in October 2015. Besides different administrative and
accounting duties, running this company doesn't change the way I've
been working since July 2014, so everything is fine!</p>
<p>As before, I won't be writing much about it through my personal blog,
except for an occasional update every other year; if you want to
follow what's happening with DEBAMAX:</p>
<ul>
<li>Website: <a href="https://debamax.com/">debamax.com</a> — in addition to the
usual <a href="https://debamax.com/company">company</a>,
<a href="https://debamax.com/services">services</a>, and
<a href="https://debamax.com/references">references</a> sections, it features
a <a href="https://debamax.com/blog/">blog</a> (with
<a href="https://debamax.com/blog/rss.xml">RSS</a>) where some missions are
going to be detailed (when it makes sense to share and when
customers are fine with it). Spoiler alert:
<a href="https://tails.boum.org/">Tails</a> is likely to be the first success
story there. <code>;)</code></li>
<li>Twitter: <a href="https://twitter.com/debamax">@debamax</a> — which is going
to be retweeted for a while from my personal account,
<a href="https://twitter.com/CyrilBrulebois">@CyrilBrulebois</a>.</li>
</ul>
D-I Stretch Alpha 1
http://mraw.org/blog/2015/07/22/D-I_Stretch_Alpha_1/
2016-08-03T17:44:34Z
2015-07-22T11:50:00Z
<p>Time for a quick recap of the beginning of the <em>Stretch</em> release cycle
as far as the Debian Installer is concerned:</p>
<ul>
<li>It took nearly 3 months after the <em>Jessie</em> release, but <code>linux</code>
finally managed to get into shape and fit for migration to
<code>testing</code>, which unblocked the way for an <code>debian-installer</code>
upload.</li>
<li>Trying to avoid last-minute fun, I’ve
<a href="https://lists.debian.org/debian-release/2015/07/msg00168.html">updated the britney freeze hints file</a>
to put into place a <code>block-udeb</code> on all packages.</li>
<li>Unfortunately, a recent change in <code>systemd</code> (implementation of
<a href="https://lists.debian.org/debian-devel/2015/06/msg00018.html">Proposal v2: enable stateless persistant network interface names</a>)
found its way into <code>testing</code> a bit before that, so I’ve had my
share of last-minute fun anyway! Indeed, that resulted in installer
system and installed system having different views on interface
naming. Thankfully I was approached by Michael Biebl right before
my final tests (and <code>debian-installer</code> upload) so there was little
head scratching involved. Commits were already in the <code>master</code>
branch so a little plan was proposed in
<a href="https://lists.debian.org/debian-release/2015/07/msg00234.html">Fixing udev-udeb vs. net.ifnames for Stretch Alpha 1</a>. This was implemented in two shots, given the
<a href="https://lists.debian.org/debian-release/2015/07/msg00254.html">extra round trip</a>
due to having dropped a binary package in the meanwhile and due to
<code>dak</code>’s complaining about it.</li>
<li>After the usual round of build (see
<a href="https://buildd.debian.org/status/package.php?p=debian-installer&suite=sid">logs</a>),
and <code>dak copy-installer</code> to get installer files from <code>unstable</code> to
<code>testing</code>, and <code>urgent</code> to get the source into <code>testing</code> as well
(see
<a href="https://lists.debian.org/debian-release/2015/07/msg00271.html">request</a>),
I’ve asked Steve McIntyre to start building images through <code>debian-cd</code>. As
expected, some troubles were run into, but they were swiftly fixed!</li>
<li>While Didier Raboud and Steve were performing some tests with the
built images, I’ve prepared the
<a href="https://lists.debian.org/debian-devel-announce/2015/07/msg00005.html">announcement for dda@</a>,
and updated the usual pages in the <code>debian-installer</code> corner of the website:
<a href="https://www.debian.org/devel/debian-installer/News/2015/20150721">news entry</a>,
<a href="https://www.debian.org/devel/debian-installer/errata">errata</a>, and
<a href="https://www.debian.org/devel/debian-installer/">homepage</a>.</li>
<li>Once the website was rebuilt to include these changes, I’ve sent
the announce, and
<a href="https://lists.debian.org/debian-release/2015/07/msg00350.html">lifted all <code>block-udeb</code></a>.</li>
</ul>
<p><em>(On a related note, I’ve started tweeting rather regularly about my
actions, wins & fails, using the
<a href="https://twitter.com/hashtag/debianinstaller">#DebianInstaller</a>
hashtag. I might try and aggregate
<a href="https://twitter.com/CyrilBrulebois">my tweets as @CyrilBrulebois</a>
into more regular blog posts, time permitting.)</em></p>
<p><strong>Executive summary:</strong>
<a href="https://lists.debian.org/debian-devel-announce/2015/07/msg00005.html">D-I Stretch Alpha 1</a>
is released, time to stretch a bit!</p>
<p><a href="http://mraw.org/blog/2015/07/22/stretching-cat.svg"><img src="http://mraw.org/blog/2015/07/22/stretching-cat.svg" alt="Stretching cat" class="img" /></a></p>
<p><em>(Credit: rferran on <a href="https://openclipart.org/detail/193429/gat-4">openclipart</a>)</em></p>
Freelance Debian consultant
http://mraw.org/blog/2014/09/15/Freelance_Debian_consultant/
2018-01-15T11:02:01Z
2014-09-15T09:20:00Z
<p>I’m not used to talking about my day job but here’s an exception.</p>
<p>Over the past few years I worked in two startups (3 years each). It
was nice to spend time in different areas: one job was mostly about
research and development in a Linux cluster environment; the other one
was about maintaining a highly-customized, Linux-based operating
system, managing a small support team, and performing technological
surveillance in IT security.</p>
<p>In the meanwhile I’ve reached a milestone: 10 years with Debian. I had
been wondering for a few months whether I could try my luck going
freelance, becoming a Debian consultant. I finally decided to go ahead
and started in August!</p>
<p>The idea is to lend a hand for various Debian-related things like
systems administration, development/debugging, packaging/repository
maintenance, or Debian Installer support, be it one-shot or on a
regular basis. I didn’t think about trainings/workshops at first but
sharing knowledge is something I’ve always liked, even if I didn’t
become a teacher.</p>
<p>For those interested, details can be found on my website:
<a href="https://mraw.org/">https://mraw.org/</a>.</p>
<p><strong>Update:</strong> It moved to <a href="https://debamax.com/">https://debamax.com/</a> since then.</p>
<p>Of course this doesn’t mean I’m going to put an end to my volunteer
activities within Debian, especially as a Debian Installer release
manager. Quite the contrary in fact! See the August and September
<a href="https://lists.debian.org/debian-boot/">debian-boot@ archives</a>, which
have been busy months. <code>:)</code></p>
Mark a mail as read across maildirs
http://mraw.org/blog/2014/08/11/Mark_a_mail_as_read_across_maildirs/
2014-08-11T18:18:33Z
2014-08-11T18:20:00Z
<h2>Problem</h2>
<p>Discussions are sometimes started by mailing a few different mailing
lists so that all relevant parties have a chance to be aware of a new
topic. It’s all nice when people can agree on a single venue to send
their replies to, but that doesn’t happen every time.</p>
<p>Case in point, I’m getting <strong>5</strong> copies of a bunch of mails, through
the following <code>debian-*</code> lists: accessibility, boot, cd, devel,
project.</p>
<p>Needless to say: Reading, or marking a given mail as read once per
maildir rapidly becomes a burden.</p>
<h2>Solution</h2>
<p>I know some people use a duplicate killer at <code>procmail</code> time (hello
gregor) but I’d rather keep all mails in their relevant maildirs.</p>
<p>So here’s
<a href="http://mraw.org/blog/2014/08/11/mark-as-read-everywhere.pl">mark-read-everywhere.pl</a>
which seems to do the job just fine for my particular setup: all
maildirs below <code>~/mails/*</code> with the usual <code>cur</code>, <code>new</code>, <code>tmp</code>
subdirectories.</p>
<p>Basically, given a mail piped from <code>mutt</code>, compute a hash on various
headers, look at all new mails (<code>new</code> subdirectories), and mark the
matching ones as read (move to the nearby <code>cur</code> subdirectories, and
change suffix from <code>,</code> to <code>,S</code>).</p>
<p>Mutt key binding (where <em>X</em> is short for <em>cross</em> post):</p>
<pre><code>macro index X "<pipe-message>~/bin/mark-as-read-everywhere.pl<enter>"
</code></pre>
<p>This isn’t pretty or bulletproof but it already started saving time!</p>
<p>Now to wonder: <a href="http://xkcd.com/1205/">was it worth the time</a> to
<a href="http://xkcd.com/1319/">automate</a> that?</p>
How to serve Perl source files
http://mraw.org/blog/2014/08/11/How_to_serve_Perl_source_files/
2014-08-11T18:25:13Z
2014-08-11T17:45:00Z
<p>I noticed a while ago a Perl script file included on my blog wasn’t
served properly, since the charset wasn’t announced and web browsers
didn’t display it properly. The received file was still valid UTF-8
(hello, little <code>©</code> character), at least!</p>
<h2>First, wrong intuition</h2>
<p>Reading Apache’s <code>/etc/apache2/conf.d/charset</code> it looks like the
following directive might help:</p>
<pre><code>AddDefaultCharset UTF-8
</code></pre>
<p>but comments there suggest reading the documentation! And indeed that alone
isn’t sufficient since this would only affect <code>text/plain</code> and
<code>text/html</code>. The above directive would have to be combined with
something like this in <code>/etc/apache2/mods-enabled/mime.conf</code>:</p>
<pre><code>AddType text/plain .pl
</code></pre>
<h2>Real solution</h2>
<p>To avoid any side effects on other file types, the easiest way forward
seems to <strong>avoid</strong> setting <code>AddDefaultCharset</code> and to associate the
<code>UTF-8</code> charset with <code>.pl</code> files instead, keeping the <code>text/x-perl</code>
MIME type, with this single directive (again in
<code>/etc/apache2/mods-enabled/mime.conf</code>):</p>
<pre><code>AddCharset UTF-8 .pl
</code></pre>
<p>Looking at response headers (<code>wget -d</code>) we’re moving from:</p>
<pre><code>Content-Type: text/x-perl
</code></pre>
<p>to:</p>
<pre><code>Content-Type: text/x-perl; charset=utf-8
</code></pre>
<h2>Conclusion</h2>
<p>Nothing really interesting, or new. Just a small reminder that tweaking
options too hastily is sometimes a bad idea. In other news, another Perl
script is coming up soon. <code>:)</code></p>
Why is my package blocked?
http://mraw.org/blog/2014/08/10/Why_is_my_package_blocked/
2014-08-10T19:48:22Z
2014-08-10T17:45:00Z
<p>A bit of history: A while ago udeb-producing packages were getting
frozen on a regular fashion, when a d-i release was about to be
cut. While I wasn’t looking at the time, I can easily understand the
reasons behind that: d-i is built upon many components, it takes some
time to make sure it’s basically in shape for a release, and it’s very
annoying when a regression sneaks in right before the installation
images get built.</p>
<p>I took over d-i release maintenance in May 2012 and only a few uploads
happened before the <code>wheezy</code> freeze. I was only discovering the job at
the time, and I basically released whatever was in <code>testing</code> then. The
freeze began right after that (end of June), so I started double
checking things affecting d-i (in addition to or instead of the review
performed by other release team members), and unblocking packages when
changes seemed safe, or once they were tested.</p>
<p>A few uploads happened after the <code>wheezy</code> release and there’s already
a <code>Jessie Alpha 1</code> release. I was about to release <code>Jessie Beta 1</code>
after some fair bits of testing, a <code>debian-installer</code> upload, and the
only remaining bits were: building installation images (hello Steve),
and of course communication (mail announce and website update).</p>
<p>Unfortunately a new upstream release reached <code>testing</code> in the
meanwhile, breaking the installer in several ways. I’ll give details
below, of course not because I want to point finger at the maintainer,
but to illustrate the ramifications that a single package’s migrating to
<code>testing</code> can induce.</p>
<ul>
<li><p>parted 3.2-1 was uploaded on 2014-07-30 and migrated on 2014-08-05.</p></li>
<li><p>parted 3.2-2 fixed a regression reported in Ubuntu only
(<a href="https://bugs.launchpad.net/ubuntu/+source/parted/+bug/1352252">LP#1352252</a>)
which I also hit with images built locally after that migration.</p></li>
<li><p>I then built some images locally using fixed parted packages but
then discovered that auto-lvm was still broken, which I reported in
<a href="https://bugs.debian.org/757417">#757417</a>.</p></li>
<li><p>After some investigation Colin confirmed some behavioral changes in
this new parted release, which imply the need for an update of
several other <code>partman-*</code> components:
<a href="https://bugs.debian.org/757661">#757661</a>,
<a href="https://bugs.debian.org/757662">#757662</a>,
<a href="https://bugs.debian.org/757663">#757663</a>,
<a href="https://bugs.debian.org/757664">#757664</a>,
<a href="https://bugs.debian.org/757665">#757665</a>,
<a href="https://bugs.debian.org/757666">#757666</a>.</p></li>
<li><p>Thankfully fixes have been added for all of those, but more testing is needed
before possibly urgenting those packages so that they get into
<code>testing</code> as soon as possible.</p></li>
</ul>
<p>Since I’d like to avoid such experience in the future, I’ll probably
reintroduce the old method and freeze all udeb-producing packages
during next d-i releases.</p>
<p>So you know why it might happen. Your next question might be: “What to
do when your package is getting caught in that net?”. In that case,
please get in touch with both <code>debian-release@</code> and <code>debian-boot@</code>
asking for an unblock. I’ll then review your particular package, and
either let it migrate to <code>testing</code>, or delay it until after the
release.</p>
<p><strong>Update:</strong> <a href="https://lists.debian.org/debian-devel-announce/2014/08/msg00003.html">official announcement</a>.</p>
D-I Jessie Alpha 1
http://mraw.org/blog/2014/03/19/D-I_Jessie_Alpha_1/
2014-03-19T02:05:04Z
2014-03-19T02:05:00Z
<p><a href="http://mraw.org/blog/2014/03/19/wazza.jpg"><img src="http://mraw.org/blog/2014/03/19/wazza.jpg" width="487" height="280" class="img" /></a></p>
<p><strong>What’s up?</strong> That was the sound of a new Debian Installer release,
an early
<a href="https://lists.debian.org/debian-devel-announce/2014/03/msg00009.html">Alpha 1 for the Jessie release cycle</a>.</p>
<p>As usual, see the
<a href="http://www.debian.org/devel/debian-installer/">Debian Installer section</a>
of the Debian website, in particular the
<a href="http://www.debian.org/devel/debian-installer/errata">errata page</a>
which lists important, known issues. Then give it a try, and
<a href="http://www.debian.org/releases/stable/amd64/ch05s04.html.en#problem-report">report bugs</a>!</p>
<p>Let’s see if we can keep the
<a href="http://qa.debian.org/data/bts/graphs/by-maint/debian-boot@lists.debian.org.png">debian-boot@ bug count</a>
below 1400 for a while, even if we get many new installation
reports. Last weeks have been busy:</p>
<p><a href="http://mraw.org/blog/2014/03/19/debian-boot-lately.png"><img src="http://mraw.org/blog/2014/03/19/debian-boot-lately.png" width="145" height="329" class="img" /></a></p>
<p>Speaking of which, if you feel like helping with the triaging effort,
there’s now a bug type called “installer bugs” on
<a href="http://udd.debian.org/bugs/">UDD’s Bugs Search page</a>. <code>;)</code></p>
Fixing bugs
http://mraw.org/blog/2013/09/17/Fixing_bugs/
2013-12-17T02:11:29Z
2013-09-17T01:10:00Z
<p>It’s a somewhat strange feeling to spend time fixing things that broke
instead of implementing new things, but it’s not like I’m a creative
guy anyway…</p>
<ul>
<li><p><code>buildd.debian.org</code>
<a href="http://lists.debian.org/debian-infrastructure-announce/2013/09/msg00001.html">went down</a>,
but
<a href="http://lists.debian.org/debian-infrastructure-announce/2013/09/msg00002.html">it’s now up again</a>
thanks to the relevant admins. The
<a href="http://debblog.philkern.de/2013/09/buildddebianorg.html">CGI part mentioned by Philipp</a>
was fun to debug: the CGI worked when called from a shell, but not
from Apache (with an apparently strange DNS resolution error). I
suggested attaching it with <code>strace</code>, and the result was even
stranger: a connection to <code>nscd</code>’s socket was attempted but the DNS
resolution through the network wasn’t. A closer look revealed the
<code>nsswitch.conf</code> file was read, and its configuration was OK, so
what? Looking at <code>libnss_dns.so.2</code>, one could see the usual
syscalls for library loading: <code>open</code>, <code>read</code>, <code>fstat</code>, <code>mmap</code>… oh
wait, <code>mmap</code> failed with <code>ENOMEM (Cannot allocate memory)</code>. Tollef
<a href="http://anonscm.debian.org/gitweb/?p=mirror/dsa-puppet.git;a=commitdiff;h=721ad7172f27a55de960d3f77c00ea649692be22">bumped the <code>rlimits</code> on the Apache side</a>,
and that was it.</p></li>
<li><p>Linux kernel ABI bump notifications: they were
<a href="https://lists.debian.org/E1VKL1F-0001pH-No@ravel.debian.org">acting weird</a>
since kernel people resumed uploading <code>-trunk-</code> kernels to
<code>experimental</code>. That was identified a while ago, but since I like
<a href="https://lists.debian.org/20130914212358.GB27003@mraw.org">tested bugfixes</a>,
I actually did wait for it to happen; fixed in
<a href="http://anonscm.debian.org/viewvc/d-i?view=revision&revision=68876">r68876</a>.</p></li>
<li><p>Of course, d-i’s git repository
<a href="http://anonscm.debian.org/gitweb/?p=d-i/debian-installer.git;a=commitdiff;h=5edeb70ea1406d57909e44d0f0becbf5137f1738">was updated</a>
to bump the ABI from <code>3.10-2-*</code> to <code>3.10-3-*</code> accordingly.</p></li>
<li><p><a href="http://bugs.debian.org/722939">#722939</a> was reported against
<code>udev-udeb</code> since depending on a deb library isn’t a good thing
when you’re an udeb binary.</p></li>
<li><p>Trying to look into d-i build failures earlier this month,
<a href="https://lists.debian.org/20130916231527.GB21028@mraw.org">I thought I managed to reproduce the issue</a>,
but that was just my misunderstanding the way apt works. Long story
short, <code>*InRelease</code> and <code>*Packages</code> files are checked before
they’re put under <code>lists/</code>; modifying the contents isn’t expected:
the modification time is used to set the <code>If-Modified-Since</code> header
when trying to fetch a new version from the server, and it’s OK to
get a <code>304 Not Modified</code> in that case. As a side effect, that means
the files can be hand-edited, for example because you need to work
around the dependency bug mentioned above, and one will still get
<code>Hit</code> lines for all those files. If something goes wrong, one can
still <code>rm</code> the files and start over, but that looks like a large
hammer. Let’s see if the issue can get reproduced (and debugged!)
later on.</p></li>
</ul>
<p>For those last two points, the long term plan is:</p>
<ul>
<li><p>To check udeb installability automatically to catch such dependency
issues as early as possible.</p></li>
<li><p>To keep many more logs, so that one can still read the whole output
a few weeks after the fact, and so that one can compare some logs
to try and spot difference in a semi-automated fashion.</p></li>
</ul>
<p>Hopefully getting this done this week (famous last words, eh?).</p>