Now that we have KMS, it happens things go bad enough that when X goes down, one can no longer access any console to see what happened. Mainly, one can distinguish between two cases.
- The “good” one: the machine is still accessible over the
network. Unfortunately, that requires having a spare machine to ssh
into the first machine, but that means one can run
dmesg, look into X logs, possibly run X inside
gdband see where it breaks (because
backtraceisn‘t exactly an excellent tool to get the idea of what’s going on — and that’s the tool used in X to print a backtrace in the log through a signal handler).
- The “bad” one: the machine is no longer accessible over the network. Likely, the kernel is in such a bad shape that one is never going to know what happened. Usually, writing logs to disc didn’t happen, so logs won’t magically show up at next reboot.
I’m going to discuss the second case. There’s a nice tool in the
kernel which makes it possible to send kernel logs over the
network. It’s called
netconsole. As far as limitations are
concerned, one shall note that it’s UDP only, and over Ethernet (in
other words: no wireless). The good news is that it can usually make
the last crucial lines available.
Let’s call the crashy machine a patient and the logging machine a doctor.
One should keep in mind that if something goes wrong while configuring
netconsole module, it can be unloaded at anytime through:
sudo modprobe -r netconsole
Easy case: on a local network
Here’s an example: Patient’s IP is
192.168.0.1, doctor's IP is
What one needs on the receiving side: only
netcat. A tiny warning,
there are several implementations. Here’s the appropriate syntax:
# netcat-traditional: nc -l -u -p 6666 | tee ~/netconsole.log # netcat-openbsd: nc -l -u 6666 | tee ~/netconsole.log
nc command is handled through the alternatives system, one can
update-alternative --config nc to pick the appropriate one if
netcat-* packages are installed. To list the current one, an
easy way is
readlink -f $(which nc).
tee means that incoming messages are going to be printed, both
to the standard output and to the specified file.
Running as non-privileged user is sufficient.
Now, to have the patient send stuff to the doctor, a simple
sudo modprobe netconsole netconsole=@/,firstname.lastname@example.org/
What happens here? One requests the
netconsole module to be loaded,
and one specifies the parameters. Details can be read in the Linux
kernel documentation (
concentrating on the points of interest here:
- 192.168.0.2: The doctor’s IP.
- 6666: The UDP port to which packets are sent. 6666 is the default and can be omitted.
That’s all! On the doctor side, one should see something along those lines:
[2920549.188090] console [netcon0] enabled [2920549.188102] netconsole: network logging started
Slightly harder case: over internet
Because one might not have a second machine handy, it’s also possible to go through a router and send stuff across the internet.
Here’s an example: Patient’s IP is
192.168.0.1, connecting to
internet through a router, which IP is
192.168.0.254. The doctor,
available over the internet, is
Exactly the same as in the previous case. The main issue one can run
into is firewalls at this point. That’s why one may want to switch
from the default 6666 port to another one which wouldn’t be
filtered. One can think of 53 (DNS), but that means running
root since it’s a “privileged” port (below 1024).
Routing is also possible, but that requires an extra parameter: the
MAC address of the router. To obtain it, one can use the
Supposing it returned the
01:02:03:04:05:06 address, loading the
sudo modprobe netconsole netconsole=@/,email@example.com/01:02:03:04:05:06
Now, if you’re running into firewall-related issues, you can change the source for the UDP packets. The default is 6665, but assuming you want to send from an unfiltered 1234 port, that becomes:
sudo modprobe netconsole netconsole=1234@/,firstname.lastname@example.org/01:02:03:04:05:06