Sysadmins’ Inconceivable Tales: When All You Have is a Serial Console

by

As a colleague of mine once astutely observed: “All problems are DBA problems.” Recently I had to overcome a problem that had nothing to do with databases. Thankfully, I had an experience our colleague Niall shared with me well over two decades ago to fall back on.

Problem

There was a bricked OpenWRT device. So bricked that the only way of contacting it was via a TTL serial console. It had a buggy firmware on it that completely rendered all network interfaces inoperative. So – all we have is a serial console, and a booted relatively Linux system. The program to flash the firmware is there, but how do we get the firmware image onto the device in the first place?

File Through a Pipe
Feeding a file through a small pipe.

Getting to the Solution

This where a variation on the 25 year old trick comes in – we paste it in! But we cannot paste in a binary, that just won’t transfer in over a serial console because various control and escape values will be in the stream. And this is where the difficulties really begin. OpenWRT base image doesn’t seem to contain a base64 decoder program. The thing that came to mind, primed with the story I was told 25 years before, was – do I have awk?

OpenWRT has awk

We do! Great! So we now need an awk program small enough to paste in to decode something like base64. A quick search lead me to a base64 decoder in a about 100 lines of awk. Great, that’s small enough that we can just paste it into vi and save it as /tmp/base64decode.awk

Now we need to encode the firmware image for sending:

Encode the file for sending

For those of you who are hoping for some pastables:

cat openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk | base64 > openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk.base64
cat openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk.base64 | xclip -sel clip

Now we have the base64 encoded firmware image on the clipboard, we need to get it pasted into the console and save it. On the OpenWRT side, we can use the following pastable:

cat - > openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk.base64

This will take a while over serial console, and you will see 12MB of base64 encoded data scroll past your serial console terminal. When it stops scrolling, press Ctrl-D.

Then we have to use the awk program to decode the base64 encoded program. This is not a powerful device, and awk isn’t a particularly fast language, so this will also take a while.

Compare the checksums, and if they match – flash the firmware:

Decode base64 with awk, compare checksums and flash the firmware.

And here are the pastables:

awk -f base64decode.awk openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk.base64 > openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk

md5sum openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk*
5a8ba43131c78bf08301648cd7ea5cc9  openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk
69fa521cccc01a775c8412d7b5d5990e  openwrt-22.03.5-bcm53xx-generic-netgear_r8000-squashfs.chk.base64

And that is how the day and the device were saved by an experience a colleague shared 25 years prior.