I run OSPF at home. I realise that this is an obvious sign of madness, but let’s leave that aside for now. I tried running BIRD, but I couldn’t get that to work properly: something to do with not reporting changes in VPN arrangements, I think. So I run Quagga.
This seems to work well. Just one snag: after running perfectly happily, one of my routers (vampire, for the curious) started being shunned by the others after running fine for a while. Worse, once in this strange state, vampire’s ospfd process started consuming lots of memory. This is a shame, since vampire only has 512MB of physical memory to start with.
My routing arrangements are sufficiently robust that vampire’s rejection by the other routers doesn’t actually cause a significant problem in practice. There’s some unpleasant tromboning between the main LAN and the VPN, with redirects and stuff, but packets basically get through, and the VPN is only carried over slower links like wireless ethernet so I don’t really notice, except that vampire is pretty heavily overloaded at the best of times, and 256MB of leaky ospfd really doesn’t help.
Last night, I decided to investigate this problem and solve it once and for all.
I picked another router, ibanez, and captured packets on both. The captures revealed that ibanez was sending Hello packets every 10s, as it should, and vampire was sending nothing at all. Well, that explains why the other routers had decided that they weren’t going to talk to vampire: if they never see Hello packets from it, they don’t know it’s there. So, why does it clam up?
I spent a while flailing about staring at the way my firewall handles multicast. It’s not really very satisfactory, but I couldn’t see anything especially wrong. Then I got vampire’s ospfd to produce debugging output, and compared that to ibanez. That told me that vampire wasn’t even trying to send Hello packets. Well, it noticed that it was time for another, and built them, and then, well, got on with the rest of its life.
For a different view of the situation, I also captured strace traces of the daemons on the two machines. Sure enough, vampire never called sendmsg(2) at all. But why? I noticed another difference between the two: vampire repeatedly called select(2) asking whether fd 5 was writable; ibanez just got on and sent the packets. File descriptor 5 turned out to be ospfd’s raw socket which it uses for sending and receiving OSPF messages. I also dug about in the Quagga source code, and found that the relevant code constructed Hello packets and put them on a queue for output. A theory develops: for some reason, fd 5 stops showing up as writable on vampire after a while; then ospfd has a fit of rampant optimism and queues up all of its (actually rather boring) Hello packets in the hope of being able to splurge them onto the network later, all at once. OK, so this is a stupid approach, but really we shouldn’t be in this mess to begin with.
Why does a raw socket stop being writable? More diving through source code, this time for the Linux kernel. Each socket has a transmit buffer, from which sk_buffs for outgoing packets are allocated. Linux keeps track of the total size of these pending sk_buffs, and the socket reports that it’s writable only if their total size is below some threshold. Oooh, I wonder whether I can see what the transmit buffer sizes are for the raw sockets in the system. Turns out that catting /proc/net/raw tells me all I wanted. So now I run ospfd again, but this time I also watch /proc/net/raw in another window, and again I capture the packets. Sure enough, every 10s, vampire’s ospfd sends a Hello packet and its socket’s transmit buffer grows by 320 bytes. Once the buffer hits 57600 bytes, the socket stops being writable and ospfd clams up (and starts growing, but I kill it at that point).
Searching the Quagga mailing lists tells me that people have trouble setting up their routers to receive OSPF packets (multicasting is great fun if you’re actually interested in security) but not so much with sending them. One interesting thread cast aspersions on the e100 ethernet driver, which it so happens that vampire runs. So, we have a suspect.
I failed to get ping to leak memory in a stupid way. I had a wade through the kernel sources and found that tracing the life of a packet from raw_sendmsg through to transmission was going to be difficult. At this point, I decided that the thing to do was to write a kludgy program to generate packets in the right way, and see which things I could tweak to determine what things were actually causing the leak. While I was doing this, I stared at my strace spew more carefully. Of course, ospfd wasn’t just transmitting on the one network: it was sending to three different nets. And one of them is the virtual IP-over-DNS network which isn’t doing anything of any great use. On a hunch, I brought down iodined and ran ospfd again. No leak.
I’ve not worked out exactly what’s going on with iodined and its TUN/TAP interface, but the IP-over-DNS net is a stub with only one router (vampire, natch) so sending Hello packets there is pointless. So I said
passive-interface 172.29.198.128/28
into Quagga’s configuration file. Problem solved.
[NP: Epica, Design Your Universe]
Leave a comment