My 1st & 2nd SystemD Bite On The Ass

SysAdmins with a long experience develop a sense… a sense of Impending Doom… about some changes. After 20+ years of sewing your own toes back on and looking for the foot that got shot off and watching your friends and neighbors go up in flames (sometime self immolated, others with outside help), well, you get cautious about some things. Rather like someone from battle experience who reacts badly to “SURPRISE!” from a darkened room suddenly full of moving people and lights…

That was my initial response to SystemD.

No, not a single “reason”. No, no bad experiences At All. Yes, lots of folks who had plenty of experience and credentials said the Cool-Aid was Just FINE… But it smelled a bit fishy to me…

And, it would seem, to plenty of other ‘old hands’ at Unix and Linux. Part of the Unix philosophy, carried over until recently in Linux, was a program ought to be small, do one thing really well, and modularly connect to other programs.

So you could glue different bits of code together to avoid making giant monolithic bug ridden things… For example, the “list the files” command is “ls”; while the “count words program” is “wc” and it can also count lines if you give it the flag “-l”. Then you can glue those two small simple programs together with a “pipe” symbol “|” to create a new command:

ls | wc -l

That, then, counts how many files there are in that particular directory. Neat, eh?

So when someone starts making a Giant Binary Blob that sits in the middle of dozens of key parts of the operating system and has hooks into all sorts of things (like desktop windows systems and browsers and more) and is claimed to be an “init system” (that really is only supposed to be “initializing” the system – i.e. “start it up and get the hell out of the way”); well, that’s like tossing Molotov Cocktails while shouting “Surprise!” in a dark alley as cars backfire in the distance. It’s just soooo wrong! And makes ones “Spidy Sense” tingle.

But, being the “benefit of the doubt” sort, I went ahead and lived with it, if grudgingly. Not too much bad seemed to happen. But I drifted over to Debian away from Fedora / Gnome anyway… then it infested Debian… and I went to Wheezy on the Raspberry Pi M2… and now it’s in Jesse Debian and Wheezy is hard to find anymore. OK, I’ll bite…

So yesterday I got a fresh 32 GB Samsung SD Ultra 10 card. Nice, fast, big. Did a Berryboot install on it, and loaded up the current batch of OSs they offer (Arch – has SystemD, Fedora – has SystemD, Debian Jesse – has SystemD, …) and started my “usual config process”. OK, I find I’m “biting the bullet”…

First off, the Debian was quasi-OK for one config, then I ran my BuildIt Script (from prior posting) in a second config. It hung just after the intall of tightvnc IIRC. A Java process just spinning like crazy sucking up a whole CPU. It won’t die. Stopping the script doesn’t work. I send it a “SIG HUP” ( kill -HUP pidFOO)- nothing. OK, don’t like to do it, but… time for the UNSTOPPABLE UNCATCHABLE SIGNAL OF DEATH TO ALL THINGS PROCESS!!!, the “kill -9 pidFOO” as root. NOTHING survives that!…

It would not die… No, it didn’t become a Zombie Process (Process ID survives, but zero CPU, no, this PID just kept on chewing that core…) Well, I’ll reboot… Then the reboot hangs in a spin for a couple of minutes waiting for that process to die before it finally shuts down… Sigh.

Now, can I prove that was SystemD related? No, not really. But everything in my history says it is. The Kill -9 is supposed to go straight to the process and kill it. I suspect something else in the way now… Remember too that part of the advantage of a “build script” is that you KNOW you are doing things exactly the same way as worked last time. “It isn’t me” become much easier to accept. And if not me, then who?…

FWIW, on a rerun of the script, I was instructed to run a particular apt-get command to clean out the uncompleted step. The whole cycle with spinning core repeated… AND a third time after another reboot on another attempt as… oh, screw it, RESTORE TO BASE STATE and start over… (Nice thing about BerryBoot, it lets you snapshot your system installs, so I just had to drop back to “my logins and fstab, but not full buildit script yet”…

Along the way ran into Bite #2

Frankly, I’ve forgotten if this bit happened in Debian or its derivative Ubuntu. Doesn’t really matter.

I’d had a disk mounted on /Foo/ext as my file mount standard du jour is /Diskname/filesystemtype/ for mount points. After one of those reboots above, while /Foo was still there, /Foo/ext was GONE. Now I get very nervous when a file or directory just goes “POOF!” and is deleted silently. Still, it can happen if a file system is corrupt. BUT, I was running an ext4 Journaling File System… and nothing had complained of corruption. This directory was on the / file system on the SD card and they can have corruption issues… but it passed fsck at boot.

While unlikely to be related, I fsck file system check the disk. No Problems.

OK, maybe it was just “one of those things”, I think… but in the back of my brain a small patch is saying “WHAT things? The File System was fine. It’s a relatively new but not worn high quality disk with zero errors to date. The SD card is brand new, not worn, and of very high quality giving zero errors. NOTHING ought to have a directory just go away and not be found by an FSCK unless it was deleted… but *I* didn’t delete it… A 3 or 4 levels down wee voice mutters “maybe you ought to look at SystemD and see if it does anything”… dismissed as that Paranoid Guy In The Corner… and I “move on”.

The Rest of The Day

The rest of the day was spent trying various things. Re-doing the work done in the morning. Getting Jesse into a workable state, minus the ‘issue’ part of the build. Along the way suffering a few more minor unexpected things. Rebooting more than expected.

Finally, I decided to just “move on” and see if the other OSs were any better. Cycled again through Ubuntu MATE, Fedora, Puppy4, etc.

Puppy on the Raspberry Pi is still very rough. I think it has an early Chromium browser on it. Whatever it is, the graphics are essentially non-functional and it’s a pain. Way behind the rest of the Puppy litter.

Ubuntu was still Big, Fat, and slow. Still Slime Green theme. Still no sound function at all with youtube. I think it was the one with the lost directory problem, so I moved on again.

Fedora worked relatively well. Firefox uses multiple cores so isn’t hobbled as much as the browsers on early Debian limited to one core. Yet is was the same old security straight jacket. Oh, and Firefox had no sound for youtube either. Maybe there’s some Secret Sauce, but not on my todo list for the day…

With a couple of ordinary pages (sourceforge) in the browser open and the system monitor running, one core is more or less busy with Xorg (windows driving) one with Firefox (occasionally bursting to 1.5 cores) and one with “other stuff”. So about 75% of the system “busy” with modestly passive “activity” for reading an already loaded web page. Can you say “Sucky efficiency”? I knew you could… But it worked. Didn’t hang. Didn’t seem to lose anything. Oh, and SystemD was constantly sucking up some few tenths of a CPU for “whatever” in the process monitor screen…

At the end of the day, I was just not a happy camper. 4 Cores at almost a GHz to do nearly nothing. Questionable reliability and stability. Evident “bit rot” in at least one directory just “gone missing”. Failure To Perform on sound and music. An “init” process sucking up 1/10 to 1/4 of a core for???? Oh, and processes that just ignore the Super User. Systems Admin? We don’t need no Steeenking Systems Admin!! Sorry, but when the SysAdmin as Root says “You Die!”, you die… at least in my world.

But I didn’t KNOW any of this was SystemD related, for certain.

Going Looking

So today I went looking for alternatives. Found a nice FreeBSD (with modestly painful install process ‘long hand’ and not BerryBoot ready) and an interesting Arch release that is fully populated with useful things like browsers and Xwindows… And even an old Wheezy Debian image to download (that surely matches the one in my archive disk).

And began to work through how to get an Old School “Init is just init” type install running. And found this page after this search:

http://systemd-free.org/

An init system must be an init system

Systemd problems

Since the adoption of systemd by Arch Linux I’ve encountered many problems with my systems, ranging from lost temporary files which systemd deemed proper to delete without asking (changing default behaviour on a whim), to total boot lockups because systemd-210+ couldn’t mount /usr/local (whereas systemd-208 could; go figure).

As each “upgrade” of systemd aggressively assimilated more and more key system components into itself, it became apparent that the only way to avoid this single most important point of failure was to stay as far away from it as possible.

The solution: Remove systemd, install OpenRC

“Coincidentally”, there were others before me who had had similar concerns and had prepared the way. Their efforts and experience is summarised in these pages. Sincere, warm thanks go to artoo and Aaditya who have done most of the work in Archland and, of course, the Gentoo developers who have made this possible in the first place.

I administer a handsome lot of linux boxes and I’ve performed the migration procedure (successfully and without exception) in all of them, even remote ones.

That page is full of good links, so “hit the link” if interested.

I found the ‘mysteriously disappearing file’ oddly familiar, and the boot problems…

FWIW there’s also a very nice list of SystemD Free systems and much much more here:

http://without-systemd.org/wiki/index.php/Main_Page

with lots of links to:

Free/Open Source Operating systems without systemd in the default installation

GNU/Linux distributions

0Linux (SysVinit)
4MLinux (BusyBox)
Absolute Linux
AgiliaLinux (formerly MOPSLinux)
Alpine Linux (BusyBox/OpenRC)
Amazon Linux AMI
antiX antiX ships JWM+iceWM+fluxbox desktops. Separate, co-developed MX Linux distro ships XFCE desktop.
AUSTRUMI (Slackware based bootable live CD, to be run from RAM)
AV Linux
Calculate Linux (Gentoo based using OpenRC)
ConnochaetOS
Crux
DeLi(cate) Linux (legacy hardware)
Devil-Linux live, firewall distro
Devuan
Dragora GNU/Linux Libre
Elive Linux
Exe GNU/Linux
Finnix (Debian based lightweight live CD using SysVinit)
Funtoo Linux (Using OpenRC)
Gentoo Linux
While an option is provided to install systemd for those that want it, the default init system in Gentoo Linux as of May 2015 is OpenRC. If Portage is pulling in systemd, please read this Gentoo wiki article before removing Gentoo from this list. Other suggested reading, [1]
gNewSense GNU/Linux
GNUinos Libre distro based on Devuan
GoboLinux
Uses sysvinit + BootScripts. Nice approach to sysvinit!
Grml Live Linux
GuixSD
Kwort Linux
Leenux
Linux from Scratch
Linux Console (Lightweight distribution for children and kids)
Manjaro OpenRC Forum Wiki
NuTyX based on Linux From Scratch
Obarun (Arch/Runit)
OpenMediaVault
Openwall GNU/*/Linux (Owl)
Overclockix (architecture: i386, x86_64) based on Debian stable. Project forum @overclockers.com
Parted Magic
PCLinuxOS
Pentoo (Gentoo based security-focused live CD)
Pisi Linux (sysvinit + python init scripts)
Plamo Linux japanese distro, based on slackware
Porteus (Slackware based lightweight modular live CD/USB)
Porteus Kiosk (Gentoo based lightweight kiosk using BusyBox)
Puppy Linux
Pür Linux
Refracta
RLSD
Sabotage Linux
Salix
Simplicity Linux (Puppy Linux based with LXDE using SysVinit)
Slackel
Slackware
Slax
SliTaz (Lightweight live CD/USB using BusyBox/SysVinit)
Source Mage GNU/Linux
Spark Linux
sta.li
SystemRescueCd (Gentoo/OpenRC based system rescue disk)
Tiny Core Linux
TLD Linux
TRIOS Linux (Live, based on Debian jessie, XFCE, rEFInd, OpenRC init, very stable, iso images)
Unity Linux (MUSL, OpenRC, BusyBox, RPM5, and YUM)
Univention Corporate Server
Vector Linux
Void Linux
Volumio (architecture: armhf)
wifislax
Zenwalk
Embedded
CeroWrt
DD-WRT
Gargoyle
LibreCMC
OpenWRT
Unix-like and derivatives

9atom
9front
9legacy
Akaros
Darwin
Debian GNU/Hurd
Debian GNU/kFreeBSD
DesktopBSD
DragonFlyBSD
Dyson
FreeBSD
Fuguita (architecture: i386) Japanese, based on OpenBSD
Genode
GhostBSD
GNU/Hurd
Harvey OS
Inferno
MidnightBSD
MINIX
MirBSD
NetBSD
NitrOS-9
OmniOS (illumos)
OpenBSD
OpenIndiana (modern OpenSolaris, illumos)
PC-BSD
Plan 9 from Bell Labs (mirror)
Plan B
PureDarwin
Redox
SmartOS (illumos)
ubuntuBSD
Embedded
FreeNAS
NAS4Free
pfSense
Others

AROS Research Operating System
FreeDOS
FreeVMS
Haiku
HelenOS
Oberon
ReactOS

Each one a live link in the original (and no, I’m not going to do the work of copying all those links here, just hit their link)

Which of those have a hope of being on the Pi? I have no clue for now. Fewer still will be pre-made for the BerryBoot that wants a squashfs file system image. Though that isn’t very hard at all, really. See:

http://www.berryterminal.com/doku.php/berryboot/adding_custom_distributions

Adding your own custom operating systems to the menu

You can add your own extra operating systems to the Berryboot menu. However this requires that you convert your file system image to SquashFS format first.

Most Raspberry Pi operating system images are disk images containing two partitions. A FAT partition with the boot loader and kernel files, and a second ext4 partition with everything else. We are interested in the second partition.

With a regular Linux desktop computer that has kpartx and mksquashfs installed, you can convert the second partition to SquashFS like this:

$ sudo kpartx -av image_you_want_to_convert.img 
add map loop0p1 (252:5): 0 117187 linear /dev/loop0 1
add map loop0p2 (252:6): 0 3493888 linear /dev/loop0 118784
$ sudo mount /dev/mapper/loop0p2 /mnt
$ sudo sed -i 's/^\/dev\/mmcblk/#\0/g' /mnt/etc/fstab
$ sudo mksquashfs /mnt converted_image_for_berryboot.img -comp lzo -e lib/modules
$ sudo umount /mnt
$ sudo kpartx -d image_you_want_to_convert.img 

Notes:

If kpartx reports it created a mapping different than loop0p2 (e.g. loop4p2) mount that instead. This can happen if loop0 is already in use by something else on the system.
We are excluding /lib/modules from the image, because the kernel modules shipped with Berryboot are used instead, and shared with all distributions.
Some older versions of mksquashfs do not support the ”-comp lzo” option. You can leave it out to let it use gzip compression instead. Advantage of LZO is that it is faster to uncompress, which is a big plus on slow ARM devices, and therefore preferred. This does come at a cost of reduced compression ratio (LZO images are larger than gzip ones).
Put your SquashFS formatted image on a USB stick, go to the “Operating system installer”, hold down your mouse button over “Add OS” and select “Install from USB stick” If your image prefers to have a certain memory split use the extension .img128 .img192, .img224 or .img240 instead of .img.

Tweaks

If the image you are converting is based on Debian/Raspbian delete the etc/console-setup/cached_UTF-8_del.kmap.gz file before converting the image to force regeneration of the cached keyboard mapping on first boot. This is to make sure it uses the keyboard layout set in Berryboot instead of default British.

That’s the whole thing. Basically, just take the .iso and make a squashfs version of it. I’m “good with that” as it is part of what I wanted to do anyway for making a Live System that uses the SD card less and RAMdisk more (ala Puppy and Knoppix and Tails).

I downloaded a FreeBSD for the PiM2 in .iso format, so today will be “make it a squashfs and try it” day. I’d not mind at all being back on Real Unix ™!

In Conclusion

So it looks like, against my will to some extent, I’m going to be forced into the roll of “sort of a system developer” and “roll my own” distro.

My basic boot loader will be BerryBoot. It’s nice for debugging what with image captures and recovery. It’s also very compact with a basic squashfs format. It is easy to make an .iso into a BerryBoot image.

I’m starting the “search for a better tomorrow” with the Arch port (probably needing to retrofit the OpenRC bits into the RaspberryPi Arch release; and also with FreeBSD that might just be “squash and go”… and that I’m likely to try first.

If neither of those looks good enough, then I’ll wander down the other options and see what comes of it all.

This has implications.

For one, when I’ve spent 3 days screwing around with computers (like now) I don’t post as much about Global Warming BS. Then again, since Paris was signed against all obvious reality, I’m not sure reality, data integrity, and Real Science ™ matter much to the outcome.

It also means I’m likely to be quiet for a day or two at a time. It’s called a “coding frenzy” in the biz. You get to chewing on a problem and it takes over your brain. Need to hold lots of bits in there at once and distractions break the gel… So you hunker down with the coffee pot and just burn through it. There can sometimes be quick distraction posts from a “pop up for air” from time to time, but not deep “data dives” on other topics until the frenzy ends…

We’ll see.

IFF I’m very lucky, I’ll find others who have “gone there” already and I can “move on” to other bits. I can also “fall back” to my prior “daily driver” chip in the PiM2 and be functional in it. But, as I’m moving off of the old WBPCs for actual production work, I’d rather have something I liked better. (I’d just go toss $300 at a new laptop at Walmart but for them being infested with EUFI and Windoz and likely Linux Hostile too…)

For now, I’m very happy with the ChromeBox as a “Daily Driver” posting and music video box. (Editing, file saving, database management, code development, etc. etc. not so much…) So I’ve got a good box for “internet stuff”, and a good enough selection of “Linux of Old” for basic files, edits, etc… I’d just like to narrow it down into one Really Good For Everything *Nix machine.

Expect to see postings like “Here’s where to get a BSD and how to put it into BerryBoot” postings as various steps happen.

I’m also going to find myself sucked off into learning more than a minimal use of “git” and maybe even learning “how to put releases up on SourceForge” and such… IFF I actually get some release built with some virtue in it.

Not what I intended to be doing with my life right now, but “Oh Well”…

Subscribe to feed

Advertisement