Recently in Syshacking Category

I think the breakage is more or less over, so it’s time to (a) enjoy the results, and (b) write up the war stories. I’m now running a mostly-IPv6 network.

Actually, the network was mostly-IPv6 to begin with. The main missing piece was DNS: I wasn’t advertising AAAA records for my hosts, even though they nearly all actually had IPv6 addresses. Getting things to actually use IPv6, then, mostly involved hacking on my unncesssarily complicated zone file generator (written in Common Lisp, natch).

At this point, everything went wrong. It turns out that if I breathe wrong, then Linux loses the routing table entries for its own local interface addresses. It turns out that I’d breathed wrong quite a lot before pulling the big DNS switch. I still don’t know what causes this; my current theory is that it’s BIRD, but I don’t have any convincing evidence (and I know that the obvious alternative — Quagga — is hopeless in my environment. I just have a cron job which checks that the interface routes haven’t vanished.

Then I found out that SSH connections from my laptop crybaby to a colocated (virtual) server stratocaster were wedging in a not-quite MTU-blackhole way when the server had a lot of stuff to say. Specifically, the first few large segments would get through, and then they’d dry up. (MTU problems on IPv6 are especially nasty, because IPv6 routers don’t fragment packets — only the sender is allowed to do that.)

The MTU bottleneck is the VPN hop between the colocated server (the endpoint is on precision) and the house (on radius). Capturing packets showed that I was right: stratocaster was sending a TCP segment that was too large, and being told this by precision. So stratocaster sent the data again, in two pieces — and then sent the next segment full size. After a while, precision gave up sending packet-too-big ICMP messages (some rate-limiting thing, presumably), and stratocaster continued throwing large TCP segments into the void, having forgotten the path MTU, and the connection jammed up.

I’ve never trusted the standard wireless security stuff since the old WEP disaster, so even when I’m at home, traffic between crybaby and the home servers goes over the VPN. To make all of this work efficiently, mobile devices like crybaby use a simple heuristic to decide which static VPN endpoint is likely to be best to associate with (radius in the house, otherwise precision), and that endpoint advertises a host route for the mobile device through my dynamic routing machinery.

Unfortunately, Linux has a bug, and won’t attach path MTU information to host routes! Simple solution: since IPv6 address space is cheap, allocate a little network to each mobile device. Rather than 2001:ba8:1d9:6000::{1,2,3}, we’ll use 2001:ba8:1d9:6000::{1,2,3}:0/112.

Yes, I realise that allocating …:0 to the mobile device was brave. Too brave: it didn’t work. In particular, now connections wedged in the other direction. I checked: this time nobody was actually sending packet-too-big messages. Eventually, I noticed that radius’s kernel was logging plaintive — if gnomic — messages of the form

2014-04-19T14:05:29+01:00 radius kernel: [363728.436583] icmpv6_send: acast source

Tracking down this message didn’t take too long. Apparently, it means that the sender (to whom we’d be sending an ICMPv6 error) has been identified as an anycast address — and for some reason we shouldn’t actually send it at all. It’s true that …:0/112 is defined to be the any-router anycast address, and radius could see that it was indeed a /112 network, so I can’t argue with that; but I think refusing to send ICMP back to an anycast address is rather poor.

Anyway, another renumbering ensued, and now the mobile devices (and my actual IPv6 anycast services) are on …:1/112. And IPv6 works.

None of these are really very clever, but they’re slightly creative abuses of the standard tools, and maybe someone will find them useful.

1. Browsing lots of small files. Suppose you’re looking at something like a Qmail configuration directory or settings in /proc or /sys. where there are lots of one- or two-line files. You could try to get a handle on them by using cat(1) but I find that the right answer is

   grep ^ files …

   For trees of stuff, like in /proc/sys/net/*/conf, say, GNU grep’s -r flag is useful.

2. Clobbering files as root. I try to run as a non-root user most of the time, and elevate to root as needed, using sudo(8). (The vulnerability history of sudo makes for exciting reading, but I like that it maintains a log. I’m much less impressed with the security theatre of typing passwords, and its tendency to send loud SECURITY mail is very annoying.)

   So, there’s a file you want to hack: it’s owned by root, and the easiest way to hack it is with a shell rune. But if you redirect something’s stdout to the file, that’ll be done in your shell, with your privileges, so it won’t work. I use tee(1) here.

   … | sudo tee output >/dev/null

   (This at least leaves a log that I hacked the file, and even if it doesn’t say exactly how, it’s no less informative than sudoedit.)

I’m sure there were going to be more of these. Maybe some of them will come to me later.

If you’re certifiably crazy, you serve multiple views of your DNS zones from the same BIND instance. Suppose one of those zones is dynamic — maybe it’s updated by the DHCP server or something — and wants to be shared between both views.

Apparently enough people are mad enough to want to do this that there’s an entry about it on the BIND FAQ page (entitled How do I share a dynamic zone between multiple views?). The problem is tricky because, for timely updates, each view has to be able to contact the other reliably: the slave must be able to request zone transfers from the master, and the master must be able to send notifications to the slave. The official solution involves inventing magical TSIG keys that the views can use to identify each other.

Unfortunately, this doesn’t actually work properly when you have more than two views. In the scenario the FAQ answer appears to be considering, the nameserver in question has only one IP address. There’s therefore no way for the master view to send update notifications to both slaves.

I run OSPF at home. I realise that this is an obvious sign of madness, but let’s leave that aside for now. I tried running BIRD, but I couldn’t get that to work properly: something to do with not reporting changes in VPN arrangements, I think. So I run Quagga.

This seems to work well. Just one snag: after running perfectly happily, one of my routers (vampire, for the curious) started being shunned by the others after running fine for a while. Worse, once in this strange state, vampire’s ospfd process started consuming lots of memory. This is a shame, since vampire only has 512MB of physical memory to start with.

This story starts with SSH agents. Briefly, they’re programs which load your SSH keys and use them on behalf of SSH in order to authenticate to remote hosts. Very handy. Now you don’t need to type your password all the time.

When there’s no SSH agent running, your keys are usually encrypted, so you have to type a passphrase to tell the agent how to decrypt them. The default behaviour of the standard SSH is to bind a shinyfresh Unix-domain socket somewhere in /tmp and listen on it; it tells the SSH process how to find it using an environment variable. If you log in twice, you get two agents. Both of them need to be told your passphrase. This is annoying. Why do I need both? The answer appears to be that I want both because the agents’ lifetimes are tied to their respective sessions. All of this seems rather daft to me. Why not just have one agent, which listens on a well-known socket and lives on until it dies of natural causes? Now I only need to type my passphrase once each time the machine boots.

(Critics might argue that I’m extending my window of vulnerability to include times when I’m not logged in. True; but (a) I usually spend most of the time logged in, and (b) there’s only a sensible attack if the bad guy can run processes with my uid — and if he can do that, I’m already toast.)

Enter the Gnome keyring manager. It’s a handy looking gadget which remembers secrets. It protects the secrets with a passphrase. By default, it uses your login password — which is somewhat sane, since if a bad guy knows your password, you’re toast anyway. And it gets your login password through PAM — when you log into GDM. It drops its keyring on the floor when you lock the screen, and picks it up again when you type your password again — again, through PAM. Quite clever, really.

So I twiddled my login profile to try to pick up gnome-keyring-daemon and set the environment variable from it. I managed to botch this edit in such a way that I ended up starting new gnome-keyring-daemon processes if I couldn’t contact the one that was already meant to be running. Oh, well.

Now I notice that — every now and then — lots of new gnome-keyring-daemon processes appear out of thin air. SSH logins provoke this, as you’d expect, but I don’t SSH into my laptop very often so this isn’t too bad.

Last weekend was mostly spent fighting with Gnome/DBus/HAL/MumbleKit.

The story starts with me upgrading my laptop to Karmic. (Well, no, actually it starts a bit before that, but we’ll pretend.) After deciding that the new version of GDM blows goats (see Google passim: it seems that it got HFPed) I downgraded to the gdm-2.20 package, which actually has documentation and can be configured. It was about this time that I discovered a bug.

The other day, a colleague asked me how to multiplex lines from a number of asynchronous sources from a Unix shell script. Here’s what I came up with.

#! /bin/sh

set -e
for i; do
  while read line; do echo "$i: $line"; done <"$i"&
done | cat
wait