The Personal Blog of Stephen Sekula

Home Server Upgrade – Phase One

I’ve been putting it off for over a year: our home server is running an old Ubuntu Linux installation, and needs to be upgraded. Research and teaching and life intervened, and this upgrade fell farther and farther down the list. In the meantime, Ubuntu produced two stable releases… making the chore of upgrading twice as hard. Let’s take a look at some lessons I learned by not upgrading, and my state of mind as I proceeded through the first phase of the upgrade today.

Lessons from NOT upgrading

One of the key lessons from holding back on upgrading was the following: when they stop updating kernels for your old release, you’re playing a dangerous game of kernel roulette. What bugs might be left in that last kernel they pushed to the old release? Well, I learned recently: the “Out-of-Memory Killer” kernel bug. A well-documented bug was left in the last range of kernel versions released to Ubuntu 14.04, wherein a Linux Kernel aggressively kills large-memory programs because it perceives a lack of system memory, even when plenty is available.

It took me weeks to track this down. It began with having to write a script to recover strange failures of our Gluster disk, the one shared across computers in our house. From log files, I could see that processes like Spamassassin were occasionally being killed by the Kernel. All very strange. I expanded the size of my “swap” file (a disk-based memory overflow area). Nothing helped. Finally, I went in search of kernel bugs. By this point, I could not keep our Gluster disk up for more than a few minutes and any significant file input/output operation led to its murder by the kernel.

So the lesson here is: grab upstream kernels, even if you’re not going to upgrade your OS. But, seriously…. upgrade your OS.

Other problems that cropped up in the last 6 months:

  • The PHP version on my server was so old I couldn’t even do basic things like upgrade the Dokuwiki software I run there.
  • OwnCloud server versions were pretty unhappy with running on something as old as Ubuntu 14.04, and we rely on OwnCloud for distributed and cloud-based file sharing and hosting.

The upgrade

You can follow my activities today (still not complete!) from my tweets:

https://twitter.com/drsekula/status/1080467267454824448

https://twitter.com/drsekula/status/1080479606908600320
https://twitter.com/drsekula/status/1080480054617034752
https://twitter.com/drsekula/status/1080512649119649792
https://twitter.com/drsekula/status/1080514666915344390
https://twitter.com/drsekula/status/1080516617434525697
https://twitter.com/drsekula/status/1080522998636077056
https://twitter.com/drsekula/status/1080562605327085568

What now?

Where am I now? Email is all working fine again. Most basic web services, like blogs and Dokuwiki, seem to be behaving themselves. My social media instance of GNU Social seems to be OK, and tbe bridging to the Pump.io instance is also fine. I am presently backing up my old OwnCloud data folder before deploying a new OwnCloud server, and trying to get Diaspora* working again.

And I still have to make the leap from 16.04 to 18.04…!