So I’ve been poking at Jason Wessel’s kgdb patches recently (specifically, the ones in kgdb-next – you do believe in kernel debuggers, right? Good). They came in very handy when trying to track down an obscure netfilter brokenness last week that was causing Fedora kernels to fall over reproducibly when running KVM. That particular issue was caused by libvirt’s namespace code that attempts to create additional network namespaces on startup, just to see if it’s possible (for optional containers support). After a very long weekend, I pointed out a number of bugs that got fixed. But it got me thinking about kgdb and being able to easily debug stuff that rolls over and plays dead.
Traditionally, I have used a (somewhat loud, and sometimes therefore unfortunately annoying) PC attached to my debugging target via a serial crossover cable. Actually, it’s the inverse of the usual setup in which said other PC is intended to be the target of experimental test kernels, with my desktop generally not being anticipated to fall over with kernel bugs (as it has been doing increasingly of late). In any case, it’s not optimal to leave that PC running and I prefer it being used for evil test experiments. An opportunity to buy random crap on eBay presented itself in the form of an awesome Cyclades (now some other random company) terminal server. I bought a TS-3000 for $115, which is less than a tenth of what they used to go for retail. 48 ports of serial terminal server goodness for the home.
Photo: My Cyclades TS-3000 sitting atop an APC Masterswitch Plus
I was never very good at waiting for santa. I was tracking this damned thing several times a day for the two days it was in transit. And when it arrived – shock! – it might not have the latest firmware! Quick! Time to fix that. I hadn’t even used it in anger before I managed to brick the thing with an update not intended for this model. Cursing myself, I figured I would just rescue it via TFTP. But that requires a special console cable (not quite the same as some others) in order to interrupt the standard boot. Obviously I had none of these cables, and all of the ones here were useless. And I wasn’t prepared to wait ten minutes to order another one. So I went to Microcenter, and bought two RJ45-DB9 generic converters you can click together to wire yourself.
I followed a diagram online to make the RJ45-DB9 cable for the Cyclades – twice. But all of the posted diagrams were incorrect (this is nothing like a Cisco cable, even if you’re a moron and think that it is when you incorrectly make a website with the wrong pinouts, especially if you’re Cyclades and write a manual with the wrong information contained within it…thanks a bunch!). Not to be discouraged, the soldering iron came out, and I rummaged around in a box of parts to find some serial connectors. Fortunately, I had a female DB9 and plenty of old crappyish network cables. I soldered, desoldered, and resoldered this thing about 4 times before finding the correct Cyclades console cable pinout (ADB0036 female DB9) (repeated below, for the benefit of others who read this). Finally, I reflashed the unit with the same firmware it had had when it arrived (zImage_ts_140-3.bin) – the “new” firmware was only for specific other units of which mine was not one thereof, there is a newer “GPL” kit I will poke at sometime – and booted it up.
Photo: A homebrew Cyclades ADB0036 Cable
| RJ45 pin | DB9 pin |
|---|---|
| 1 | 8 (CTS) |
| 2 | 1 (DCD) and 6 (DSR) |
| 3 | 2 (RD) |
| 4 | 5 (SGND) |
| 5 | 7 (RTS) |
| 6 | 3 (TD) |
| 7 | 4 (DTR) |
| 8 | 4 (DTR) |
Figure: The correct pinout for a Cyclades ADB0036 console cable (RJ45 to Female DB9 connector)
Cyclades made good (fanless) hardware, but they were hardly the most adept at making configuration straightforward. Sure, you can configure the network easily (this one is called “morse” after the inventor – in the US – of the coding used for telegraphs, which are an ancient precursor to the RS232 standard used on modern serial ports), but when it comes to the port setup…what you want to know is that you’re looking for the “Socket SSH” option, set to increment (e.g. from “1″ – no need to use the “7001″ example, you’re not directly sshing into the port anyway, as with telnet), and based upon a simple “CAS profile” with local authentication (make sure you add a new “system” user for those SSH logins), unless you want to use RADIUS (I have home KRB5, but haven’t deployed RADIUS at the moment). Always make sure you “Run Configuration” before flashing – it seems the former writes to the actual config files that the latter will use, so you cannot necessarily flash and then “Run Configuration” that way around, depending upon the particular operation you are performing.
Once you have the terminal server running, you can talk to it:
$ ssh user_name:port_number@terminal_server.address
More importantly perhaps, you can use the gdb remote target:
(gdb) target remote | ssh -t -t user_name:port_number@terminal_server.address
Remember to tell ssh not to ruin the day (fail to allocate a pty for your friendly conversation) by specifying the “-t -t”, then you can talk to Jason’s kgdb stub.
Next steps? I need to make some more of these damned ADB0036 cables (or find some more on eBay – anyone want some useless Cisco cables I bought thinking they were the same?) and hook them up to all of my systems at home. They will then constantly log via the awesomeness of GNU screen to a remote VM, and I can jump in if something rolls over and catch it so I won’t miss panic/debug opportunities.
Kernel debuggers FTW.
Jon.