I think the breakage is more or less over, so it’s time to (a) enjoy the results, and (b) write up the war stories. I’m now running a mostly-IPv6 network.
Actually, the network was mostly-IPv6 to begin with. The main missing piece was DNS: I wasn’t advertising AAAA records for my hosts, even though they nearly all actually had IPv6 addresses. Getting things to actually use IPv6, then, mostly involved hacking on my unncesssarily complicated zone file generator (written in Common Lisp, natch).
At this point, everything went wrong. It turns out that if I breathe wrong, then Linux loses the routing table entries for its own local interface addresses. It turns out that I’d breathed wrong quite a lot before pulling the big DNS switch. I still don’t know what causes this; my current theory is that it’s BIRD, but I don’t have any convincing evidence (and I know that the obvious alternative — Quagga — is hopeless in my environment. I just have a cron job which checks that the interface routes haven’t vanished.
Then I found out that SSH connections from my laptop crybaby to a colocated (virtual) server stratocaster were wedging in a not-quite MTU-blackhole way when the server had a lot of stuff to say. Specifically, the first few large segments would get through, and then they’d dry up. (MTU problems on IPv6 are especially nasty, because IPv6 routers don’t fragment packets — only the sender is allowed to do that.)
The MTU bottleneck is the VPN hop between the colocated server (the endpoint is on precision) and the house (on radius). Capturing packets showed that I was right: stratocaster was sending a TCP segment that was too large, and being told this by precision. So stratocaster sent the data again, in two pieces — and then sent the next segment full size. After a while, precision gave up sending packet-too-big ICMP messages (some rate-limiting thing, presumably), and stratocaster continued throwing large TCP segments into the void, having forgotten the path MTU, and the connection jammed up.
I’ve never trusted the standard wireless security stuff since the old WEP disaster, so even when I’m at home, traffic between crybaby and the home servers goes over the VPN. To make all of this work efficiently, mobile devices like crybaby use a simple heuristic to decide which static VPN endpoint is likely to be best to associate with (radius in the house, otherwise precision), and that endpoint advertises a host route for the mobile device through my dynamic routing machinery.
Unfortunately, Linux has a bug, and won’t attach path MTU information to host routes! Simple solution: since IPv6 address space is cheap, allocate a little network to each mobile device. Rather than 2001:ba8:1d9:6000::{1,2,3}, we’ll use 2001:ba8:1d9:6000::{1,2,3}:0/112.
Yes, I realise that allocating …:0 to the mobile device was brave. Too brave: it didn’t work. In particular, now connections wedged in the other direction. I checked: this time nobody was actually sending packet-too-big messages. Eventually, I noticed that radius’s kernel was logging plaintive — if gnomic — messages of the form
2014-04-19T14:05:29+01:00 radius kernel: [363728.436583] icmpv6_send: acast source
Tracking down this message didn’t take too long. Apparently, it means that the sender (to whom we’d be sending an ICMPv6 error) has been identified as an anycast address — and for some reason we shouldn’t actually send it at all. It’s true that …:0/112 is defined to be the any-router anycast address, and radius could see that it was indeed a /112 network, so I can’t argue with that; but I think refusing to send ICMP back to an anycast address is rather poor.
Anyway, another renumbering ensued, and now the mobile devices (and my actual IPv6 anycast services) are on …:1/112. And IPv6 works.
