My 1st & 2nd SystemD Bite On The Ass

SysAdmins with a long experience develop a sense… a sense of Impending Doom… about some changes. After 20+ years of sewing your own toes back on and looking for the foot that got shot off and watching your friends and neighbors go up in flames (sometime self immolated, others with outside help), well, you get cautious about some things. Rather like someone from battle experience who reacts badly to “SURPRISE!” from a darkened room suddenly full of moving people and lights…

That was my initial response to SystemD.

No, not a single “reason”. No, no bad experiences At All. Yes, lots of folks who had plenty of experience and credentials said the Cool-Aid was Just FINE… But it smelled a bit fishy to me…

And, it would seem, to plenty of other ‘old hands’ at Unix and Linux. Part of the Unix philosophy, carried over until recently in Linux, was a program ought to be small, do one thing really well, and modularly connect to other programs.

So you could glue different bits of code together to avoid making giant monolithic bug ridden things… For example, the “list the files” command is “ls”; while the “count words program” is “wc” and it can also count lines if you give it the flag “-l”. Then you can glue those two small simple programs together with a “pipe” symbol “|” to create a new command:

ls | wc -l

That, then, counts how many files there are in that particular directory. Neat, eh?

So when someone starts making a Giant Binary Blob that sits in the middle of dozens of key parts of the operating system and has hooks into all sorts of things (like desktop windows systems and browsers and more) and is claimed to be an “init system” (that really is only supposed to be “initializing” the system – i.e. “start it up and get the hell out of the way”); well, that’s like tossing Molotov Cocktails while shouting “Surprise!” in a dark alley as cars backfire in the distance. It’s just soooo wrong! And makes ones “Spidy Sense” tingle.

But, being the “benefit of the doubt” sort, I went ahead and lived with it, if grudgingly. Not too much bad seemed to happen. But I drifted over to Debian away from Fedora / Gnome anyway… then it infested Debian… and I went to Wheezy on the Raspberry Pi M2… and now it’s in Jesse Debian and Wheezy is hard to find anymore. OK, I’ll bite…

So yesterday I got a fresh 32 GB Samsung SD Ultra 10 card. Nice, fast, big. Did a Berryboot install on it, and loaded up the current batch of OSs they offer (Arch – has SystemD, Fedora – has SystemD, Debian Jesse – has SystemD, …) and started my “usual config process”. OK, I find I’m “biting the bullet”…

First off, the Debian was quasi-OK for one config, then I ran my BuildIt Script (from prior posting) in a second config. It hung just after the intall of tightvnc IIRC. A Java process just spinning like crazy sucking up a whole CPU. It won’t die. Stopping the script doesn’t work. I send it a “SIG HUP” ( kill -HUP pidFOO)- nothing. OK, don’t like to do it, but… time for the UNSTOPPABLE UNCATCHABLE SIGNAL OF DEATH TO ALL THINGS PROCESS!!!, the “kill -9 pidFOO” as root. NOTHING survives that!…

It would not die… No, it didn’t become a Zombie Process (Process ID survives, but zero CPU, no, this PID just kept on chewing that core…) Well, I’ll reboot… Then the reboot hangs in a spin for a couple of minutes waiting for that process to die before it finally shuts down… Sigh.

Now, can I prove that was SystemD related? No, not really. But everything in my history says it is. The Kill -9 is supposed to go straight to the process and kill it. I suspect something else in the way now… Remember too that part of the advantage of a “build script” is that you KNOW you are doing things exactly the same way as worked last time. “It isn’t me” become much easier to accept. And if not me, then who?…

FWIW, on a rerun of the script, I was instructed to run a particular apt-get command to clean out the uncompleted step. The whole cycle with spinning core repeated… AND a third time after another reboot on another attempt as… oh, screw it, RESTORE TO BASE STATE and start over… (Nice thing about BerryBoot, it lets you snapshot your system installs, so I just had to drop back to “my logins and fstab, but not full buildit script yet”…

Along the way ran into Bite #2

Frankly, I’ve forgotten if this bit happened in Debian or its derivative Ubuntu. Doesn’t really matter.

I’d had a disk mounted on /Foo/ext as my file mount standard du jour is /Diskname/filesystemtype/ for mount points. After one of those reboots above, while /Foo was still there, /Foo/ext was GONE. Now I get very nervous when a file or directory just goes “POOF!” and is deleted silently. Still, it can happen if a file system is corrupt. BUT, I was running an ext4 Journaling File System… and nothing had complained of corruption. This directory was on the / file system on the SD card and they can have corruption issues… but it passed fsck at boot.

While unlikely to be related, I fsck file system check the disk. No Problems.

OK, maybe it was just “one of those things”, I think… but in the back of my brain a small patch is saying “WHAT things? The File System was fine. It’s a relatively new but not worn high quality disk with zero errors to date. The SD card is brand new, not worn, and of very high quality giving zero errors. NOTHING ought to have a directory just go away and not be found by an FSCK unless it was deleted… but *I* didn’t delete it… A 3 or 4 levels down wee voice mutters “maybe you ought to look at SystemD and see if it does anything”… dismissed as that Paranoid Guy In The Corner… and I “move on”.

The Rest of The Day

The rest of the day was spent trying various things. Re-doing the work done in the morning. Getting Jesse into a workable state, minus the ‘issue’ part of the build. Along the way suffering a few more minor unexpected things. Rebooting more than expected.

Finally, I decided to just “move on” and see if the other OSs were any better. Cycled again through Ubuntu MATE, Fedora, Puppy4, etc.

Puppy on the Raspberry Pi is still very rough. I think it has an early Chromium browser on it. Whatever it is, the graphics are essentially non-functional and it’s a pain. Way behind the rest of the Puppy litter.

Ubuntu was still Big, Fat, and slow. Still Slime Green theme. Still no sound function at all with youtube. I think it was the one with the lost directory problem, so I moved on again.

Fedora worked relatively well. Firefox uses multiple cores so isn’t hobbled as much as the browsers on early Debian limited to one core. Yet is was the same old security straight jacket. Oh, and Firefox had no sound for youtube either. Maybe there’s some Secret Sauce, but not on my todo list for the day…

With a couple of ordinary pages (sourceforge) in the browser open and the system monitor running, one core is more or less busy with Xorg (windows driving) one with Firefox (occasionally bursting to 1.5 cores) and one with “other stuff”. So about 75% of the system “busy” with modestly passive “activity” for reading an already loaded web page. Can you say “Sucky efficiency”? I knew you could… But it worked. Didn’t hang. Didn’t seem to lose anything. Oh, and SystemD was constantly sucking up some few tenths of a CPU for “whatever” in the process monitor screen…

At the end of the day, I was just not a happy camper. 4 Cores at almost a GHz to do nearly nothing. Questionable reliability and stability. Evident “bit rot” in at least one directory just “gone missing”. Failure To Perform on sound and music. An “init” process sucking up 1/10 to 1/4 of a core for???? Oh, and processes that just ignore the Super User. Systems Admin? We don’t need no Steeenking Systems Admin!! Sorry, but when the SysAdmin as Root says “You Die!”, you die… at least in my world.

But I didn’t KNOW any of this was SystemD related, for certain.

Going Looking

So today I went looking for alternatives. Found a nice FreeBSD (with modestly painful install process ‘long hand’ and not BerryBoot ready) and an interesting Arch release that is fully populated with useful things like browsers and Xwindows… And even an old Wheezy Debian image to download (that surely matches the one in my archive disk).

And began to work through how to get an Old School “Init is just init” type install running. And found this page after this search:

An init system must be an init system

Systemd problems

Since the adoption of systemd by Arch Linux I’ve encountered many problems with my systems, ranging from lost temporary files which systemd deemed proper to delete without asking (changing default behaviour on a whim), to total boot lockups because systemd-210+ couldn’t mount /usr/local (whereas systemd-208 could; go figure).

As each “upgrade” of systemd aggressively assimilated more and more key system components into itself, it became apparent that the only way to avoid this single most important point of failure was to stay as far away from it as possible.

The solution: Remove systemd, install OpenRC

“Coincidentally”, there were others before me who had had similar concerns and had prepared the way. Their efforts and experience is summarised in these pages. Sincere, warm thanks go to artoo and Aaditya who have done most of the work in Archland and, of course, the Gentoo developers who have made this possible in the first place.

I administer a handsome lot of linux boxes and I’ve performed the migration procedure (successfully and without exception) in all of them, even remote ones.

That page is full of good links, so “hit the link” if interested.

I found the ‘mysteriously disappearing file’ oddly familiar, and the boot problems…

FWIW there’s also a very nice list of SystemD Free systems and much much more here:

with lots of links to:

Free/Open Source Operating systems without systemd in the default installation

GNU/Linux distributions

0Linux (SysVinit)
4MLinux (BusyBox)
Absolute Linux
AgiliaLinux (formerly MOPSLinux)
Alpine Linux (BusyBox/OpenRC)
Amazon Linux AMI
antiX antiX ships JWM+iceWM+fluxbox desktops. Separate, co-developed MX Linux distro ships XFCE desktop.
AUSTRUMI (Slackware based bootable live CD, to be run from RAM)
AV Linux
Calculate Linux (Gentoo based using OpenRC)
DeLi(cate) Linux (legacy hardware)
Devil-Linux live, firewall distro
Dragora GNU/Linux Libre
Elive Linux
Exe GNU/Linux
Finnix (Debian based lightweight live CD using SysVinit)
Funtoo Linux (Using OpenRC)
Gentoo Linux
While an option is provided to install systemd for those that want it, the default init system in Gentoo Linux as of May 2015 is OpenRC. If Portage is pulling in systemd, please read this Gentoo wiki article before removing Gentoo from this list. Other suggested reading, [1]
gNewSense GNU/Linux
GNUinos Libre distro based on Devuan
Uses sysvinit + BootScripts. Nice approach to sysvinit!
Grml Live Linux
Kwort Linux
Linux from Scratch
Linux Console (Lightweight distribution for children and kids)
Manjaro OpenRC Forum Wiki
NuTyX based on Linux From Scratch
Obarun (Arch/Runit)
Openwall GNU/*/Linux (Owl)
Overclockix (architecture: i386, x86_64) based on Debian stable. Project forum
Parted Magic
Pentoo (Gentoo based security-focused live CD)
Pisi Linux (sysvinit + python init scripts)
Plamo Linux japanese distro, based on slackware
Porteus (Slackware based lightweight modular live CD/USB)
Porteus Kiosk (Gentoo based lightweight kiosk using BusyBox)
Puppy Linux
Pür Linux
Sabotage Linux
Simplicity Linux (Puppy Linux based with LXDE using SysVinit)
SliTaz (Lightweight live CD/USB using BusyBox/SysVinit)
Source Mage GNU/Linux
Spark Linux
SystemRescueCd (Gentoo/OpenRC based system rescue disk)
Tiny Core Linux
TLD Linux
TRIOS Linux (Live, based on Debian jessie, XFCE, rEFInd, OpenRC init, very stable, iso images)
Unity Linux (MUSL, OpenRC, BusyBox, RPM5, and YUM)
Univention Corporate Server
Vector Linux
Void Linux
Volumio (architecture: armhf)
Unix-like and derivatives

Debian GNU/Hurd
Debian GNU/kFreeBSD
Fuguita (architecture: i386) Japanese, based on OpenBSD
Harvey OS
OmniOS (illumos)
OpenIndiana (modern OpenSolaris, illumos)
Plan 9 from Bell Labs (mirror)
Plan B
SmartOS (illumos)

AROS Research Operating System

Each one a live link in the original (and no, I’m not going to do the work of copying all those links here, just hit their link)

Which of those have a hope of being on the Pi? I have no clue for now. Fewer still will be pre-made for the BerryBoot that wants a squashfs file system image. Though that isn’t very hard at all, really. See:

Adding your own custom operating systems to the menu

You can add your own extra operating systems to the Berryboot menu. However this requires that you convert your file system image to SquashFS format first.

Most Raspberry Pi operating system images are disk images containing two partitions. A FAT partition with the boot loader and kernel files, and a second ext4 partition with everything else. We are interested in the second partition.

With a regular Linux desktop computer that has kpartx and mksquashfs installed, you can convert the second partition to SquashFS like this:

$ sudo kpartx -av image_you_want_to_convert.img 
add map loop0p1 (252:5): 0 117187 linear /dev/loop0 1
add map loop0p2 (252:6): 0 3493888 linear /dev/loop0 118784
$ sudo mount /dev/mapper/loop0p2 /mnt
$ sudo sed -i 's/^\/dev\/mmcblk/#\0/g' /mnt/etc/fstab
$ sudo mksquashfs /mnt converted_image_for_berryboot.img -comp lzo -e lib/modules
$ sudo umount /mnt
$ sudo kpartx -d image_you_want_to_convert.img 


If kpartx reports it created a mapping different than loop0p2 (e.g. loop4p2) mount that instead. This can happen if loop0 is already in use by something else on the system.
We are excluding /lib/modules from the image, because the kernel modules shipped with Berryboot are used instead, and shared with all distributions.
Some older versions of mksquashfs do not support the ”-comp lzo” option. You can leave it out to let it use gzip compression instead. Advantage of LZO is that it is faster to uncompress, which is a big plus on slow ARM devices, and therefore preferred. This does come at a cost of reduced compression ratio (LZO images are larger than gzip ones).
Put your SquashFS formatted image on a USB stick, go to the “Operating system installer”, hold down your mouse button over “Add OS” and select “Install from USB stick” If your image prefers to have a certain memory split use the extension .img128 .img192, .img224 or .img240 instead of .img.


If the image you are converting is based on Debian/Raspbian delete the etc/console-setup/cached_UTF-8_del.kmap.gz file before converting the image to force regeneration of the cached keyboard mapping on first boot. This is to make sure it uses the keyboard layout set in Berryboot instead of default British.

That’s the whole thing. Basically, just take the .iso and make a squashfs version of it. I’m “good with that” as it is part of what I wanted to do anyway for making a Live System that uses the SD card less and RAMdisk more (ala Puppy and Knoppix and Tails).

I downloaded a FreeBSD for the PiM2 in .iso format, so today will be “make it a squashfs and try it” day. I’d not mind at all being back on Real Unix ™!

In Conclusion

So it looks like, against my will to some extent, I’m going to be forced into the roll of “sort of a system developer” and “roll my own” distro.

My basic boot loader will be BerryBoot. It’s nice for debugging what with image captures and recovery. It’s also very compact with a basic squashfs format. It is easy to make an .iso into a BerryBoot image.

I’m starting the “search for a better tomorrow” with the Arch port (probably needing to retrofit the OpenRC bits into the RaspberryPi Arch release; and also with FreeBSD that might just be “squash and go”… and that I’m likely to try first.

If neither of those looks good enough, then I’ll wander down the other options and see what comes of it all.

This has implications.

For one, when I’ve spent 3 days screwing around with computers (like now) I don’t post as much about Global Warming BS. Then again, since Paris was signed against all obvious reality, I’m not sure reality, data integrity, and Real Science ™ matter much to the outcome.

It also means I’m likely to be quiet for a day or two at a time. It’s called a “coding frenzy” in the biz. You get to chewing on a problem and it takes over your brain. Need to hold lots of bits in there at once and distractions break the gel… So you hunker down with the coffee pot and just burn through it. There can sometimes be quick distraction posts from a “pop up for air” from time to time, but not deep “data dives” on other topics until the frenzy ends…

We’ll see.

IFF I’m very lucky, I’ll find others who have “gone there” already and I can “move on” to other bits. I can also “fall back” to my prior “daily driver” chip in the PiM2 and be functional in it. But, as I’m moving off of the old WBPCs for actual production work, I’d rather have something I liked better. (I’d just go toss $300 at a new laptop at Walmart but for them being infested with EUFI and Windoz and likely Linux Hostile too…)

For now, I’m very happy with the ChromeBox as a “Daily Driver” posting and music video box. (Editing, file saving, database management, code development, etc. etc. not so much…) So I’ve got a good box for “internet stuff”, and a good enough selection of “Linux of Old” for basic files, edits, etc… I’d just like to narrow it down into one Really Good For Everything *Nix machine.

Expect to see postings like “Here’s where to get a BSD and how to put it into BerryBoot” postings as various steps happen.

I’m also going to find myself sucked off into learning more than a minimal use of “git” and maybe even learning “how to put releases up on SourceForge” and such… IFF I actually get some release built with some virtue in it.

Not what I intended to be doing with my life right now, but “Oh Well”…

Subscribe to feed


About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in Tech Bits and tagged , , . Bookmark the permalink.

18 Responses to My 1st & 2nd SystemD Bite On The Ass

  1. Larry Ledwick says:

    I hate it when kill -9 does not pull the plug on a process!
    Like you say it should die before you can run ps -ef | grep
    Hope this is not a symptom of things to look forward to on Centos at work!

  2. E.M.Smith says:

    I probably ought to add a 3rd BonTA…

    I wanted to stop the “automatic log in and launch of window system” in Debian… Normally, I’d just do a change in the init.d script. Or noobs would do a web search and find directions like:

    (which pops up when you search on “raspbian how to stop automatic login”… as any good noob would do, knowing nothing of the swap to SystemD in progress…)

    down vote
    Following was taken from RPi Debian Auto Login page:

    Auto Login:

    In Terminal:

    sudo nano /etc/inittab
    Scroll down to:

    1:2345:respawn:/sbin/getty 115200 tty1
    and change to

    #1:2345:respawn:/sbin/getty 115200 tty1
    Under that line add:

    1:2345:respawn:/bin/login -f pi tty1 /dev/tty1 2>&1
    Ctrl+X to exit, Y to save followed by enter twice

    Auto StartX (Run LXDE):

    In Terminal:

    sudo nano /etc/rc.local
    Scroll to the bottom and add the following above exit 0:

    su -l pi -c startx

    Basically start, or don’t start, a ‘getty” (get tty) process on init of a login on tty1…

    Something I’ve done that way “almost forever”…

    But now, thanks to SystemD “Everything You Know Is Wrong” and such things will become ever more bad over time. The “old way” will live on, and the new will be randomly mixed in, on millions of web advice pages…

    So I searched further… Adding “Systemd” to the search gave a page that suggested changing where a symbolic link buried in a systemd directory pointed… I tried it, it didn’t work… Then found that there’s “an app for that” under a dropdown menu to set system settings. Great! It’s all integrated now… So I clicked “no” for autologin. Rebooting gave me an unrecoverable boot with a blank screen and no prompt…

    Several reboots and restore of system later… I found that if I clicked BOTH “no” and “not a GUI give a a CLI” choices, then and only then I could not hang at boot… Really really crappy when choosing exactly what you want from the menu (don’t auto login) gives you an unrecoverably hung system… LOTS of noobs are going to crash on that one…

    Perhaps this one is “more correct”, if you can figure out what it says after you spend an hour on it…

    But “whatever”… That one is 100% SystemD driven.

    That was the moment when I decided to just “Chuck it” and avoid SystemD. When it has no simple “change this parameter” and either forces you to use a “click box GUI” that bricks your system or deal with an arcania of nonsense… well, I’ll take init that works and is simple.

    So maybe I have 3 SystemD bites on the ass …

    For now, I’m staying on Debian Wheezy and working to port FreeBSD to BerryBoot. Arch and maybe Gentoo for later as ‘fall back’ (unless someone can point me to an already made non-SystemD release of them with window system and browser already built in…)

    In short, this kind of crap is what you expect with MicroSnot, or a Devo raw early release major feature change release of a Linux (and nearly never on BSD…); not something that’s supposedly a ‘stable’ release for the public…

    (I can say that kind of thing as I was Software Q.A. manager at one company and Build Master for a Unix system at another… I’ve done this stuff professionally… even if a long time ago… In any case, that level of Aw Shit would not make it past my Q.A. department and would never be allowed to ship. How do I know? For one thing, any panel with ‘click buttons’ would have ALL combinations tried before saying “OK, no bad thing happens”… )

  3. curly says:

    Hey EM.
    W/you on systemd.
    Worked on *nix systems since V7, and X-window since the MIT X10 releases, and linux custom builds from source for embedded and appliances (CoreOS reinventing that again).
    Something about SW from that causes me to think “uh-oh” before I have to use it. Don’t know if they’re good ideas badly executed, or just bad ideas.

  4. E.M.Smith says:

    So far, I’m impressed with Arch Linux. Install was not as horrid as I’d expected (just the regular “chose from what Berrryboot gives you” but on the second page / tab; but comes up as “prompt only”. It’s about 148 MB? Something like that. Dinky. Then you get to start adding things.

    I’ve added X-Windows support and Firefox so far. I’m posting this from Firefox on Arch.

    At first I ‘had issues’ adding X, then discovered the “magic sauce” of upgrading the whole thing first, since what’s on the BB is older and Arch runs as a big synchronous horde of folks… So do a:

    pacman -Syu

    then wait a long time as packages get changed…

    After that, you can add X via:

    pacman -S xf86-video-fbdev lxde xorg-xinit dbus

    I added a .xinitrc file in the home directory of root (as I’ve not created accounts yet) with:

    exec startlxde

    in it, then did a


    and was in LXDE “right quick”.

    I did notice that X depended on SystemD (there are supposedly ways to “kill and replace” systemD in Arch, but I’m not up to that step yet…) but God only knows why…

    At any rate, then I added Firefox with

    pacman -S firefox.

    The system was using nearly nothing of memory and comes up astoundingly fast. With LXDE running under X-windows, it was a bit under 200 MB memory. Now, with Firefox running, and a terminal window open, and the system monitor running, it’s at:

    312 MB memory and about 12% of CPU depending on how fast I type ;-)

    the “top” command has a truly horrid Red Font that is hard to read too. Not figured out yet how to ‘fix’ it… BUT, it also has a nice 4-bar graphic for what the 4 CPUs are doing. Generally it looks like the machine is Damn Fast this way, and careful with memory use.

    So far, I’m likeing it, even if the process isn’t as ‘hand holding’ as the other OSs.

    So general impression is that it’s very much fast enough to be Very Comfortable. Adding packages is easy and quick (even if a different method than I’m used to…) and it is delightfully careful with resources.

    My major barrier to using ARCH had just been the PITA of it not having a windows system by default and no browser so “go fish” in the big Systems Administration Pond and see if you can find something… Now that I’m past that hump (that turned out to be a speck of dirt…) it is very nice. Except that damn red font… I like to always run “top” in a window while doing anything for constant system awareness…

    Oh, and the ‘terminal’ window has tiny mice type, so you need to jump it up to something readable under the “edit ; preferences” tab…

    More later…

  5. E.M.Smith says:

    OK, I”m sold. ARCH is now going to be my “daily driver” on the PiM2 until something better comes along.

    Just hit my “way too heavy” top URL with a zillion music videos pounding it, and then clicked on the Grumpy one… then played “Pub With No Beer”…

    First off, it actually made sound! YAAYYYY! I can now use my PiM2 for things YouTube!!!

    Second, using FireFox, it had all 4 cores more or less evenly used, bouncing between about 50% and 80% each. I.e. it used the processors well and balanced AND had enough “umpph” for the job with some left over! There was one small video glitch moment (that might have been “not me”…) but otherwise smooth motion video and good stereo.

    And that from a class 4 slow card…

    So now I’m going to ‘reimplement’ on a faster Class 10 Ultra card and one with more space… While this starts out small, it rapidly blows up to huge when you start slamming things into it. I added: Chromium, Firefox, Midori, Seamonkey Suite

    About then, the 4 GB chip is getting full… only 917 MB of free space left (though also in the Berryboot are the memtester and PXE server, but that’s all of a few hundred MB…)

    In short: between the speed gain, the memory gain, and the music / sound working along with “4 core aware” browser, the experience is Damn Fine and I’m happy using it; instead of “slightly reluctant longing for more…” as with Ubuntu or Fedora or increasingly Debian. They have all gotten too fat and slow to be really comfortable with this small of hardware.

    Arch is right at home on it… I’ve got about 620 MB of memory unused at the moment, for example, and swap? Who needs to even think about swap…

    OK, time for me to get back to work…

  6. E.M.Smith says:

    Just reran the video on the tablet, same video glitch… it isn’t Arch, or me…

  7. E.M.Smith says:

    OK, on the “reimplementation” there was another “issue”… Firefox (and others” would not boot under LXDE…

    OK, why….

    On the first try, I’d done a “pacman -S Xorg” that put a bunch of stuff in, but didn’t make anything go…

    I’d figured “nada”… but… turns out something WAS needed. So I did it after the above… and it crashed about 1/2 way through saying some file already existed… whatver it was, I moved it to and redid the pacman -S Xorg…

    Now everything works… Again… (clearly there’s an ordering and dependency issue…)

    Someday I’ll need to redo the reinstall and document exactly what it was, but for now, good enough… (Why we script things and why we have reimplimentations…)

    OH, and both times at boot, FireFox complains about some error of something vaguely SystemD / Dbus like (procman or something to do with access or permissions or… Did I mention I really like Gin in Tangelo juice?…. but I digress… just ignore the God Damn error at first Firefox boot, it doesn’t mean a thing that matters and “move along”…

    Meanwhile back at Shakiria…

  8. E.M.Smith says:

    OK, I still have a reason to want a R.PiM3…

    When you make the YouTube window full sized on a 27 inch diagonal screen it has halting video… It is enough to drive video fluidly in a smaller window, but not full screen high def… So I can get that R.Pi Model 3 ;-)

    Still, pretty darned good for $35…

  9. E.M.Smith says:

    ANnd from the “everything you know is wrong. now…’ department…

    While trying to figure out why I could not get an exported file system to mount ( I’d done ‘exportfs -a’ and it gave no errors, so presumed it was exporting…) Seems that SystemD doesn’t actually bother exporting NFS files unless you tell it, too, to export…

    [root@ARCH_pi_64 etc]# systemctl enable nfs-server
    Created symlink from /etc/systemd/system/ to /usr/lib/systemd/system/nfs-server.service.
    [root@ARCH_pi_64 etc]# systemctl start nfs-server

    Now I just need to find out the New Improved Obscured! way that’s used for enabling nfs to mount things onto this box. Not enough to have nfs installed, and a server exporting, and an entry in /etc/fstab, and being root, and issuing a mount command… No, something “else” is missing… and I suspect SystemD has stuck it’s tongue down the NFS mount mouth and I need to find out how to extract it…


    And they wonder why folks hate them…

    BTW, the “ought to be obvious” choice doesn’t work:

    [root@ARCH_pi_64 etc]# systemctl enable nfs-client
    Failed to execute operation: No such file or directory
  10. E.M.Smith says:

    Oh Great… IFF this is really the way “things are done now”… it’s a royal PITA…

    Client systemd NFS Mounts

    With the advent of systemd, the use of a service like netfs is no longer required or present – the mounts are still configured in /etc/fstab but how they work is completely different mechanically. The new systemd methodology of mounting filesystems at boot time is using the generator infrastructure with RHEL/CentOS 7, Debian 8, Arch and others.

    Upon boot, systemd-fstab-generator examines the configured mounts and writes each one as a systemd Unit in the runtime directory, /run/systemd/generator, that are then “started” as if they were traditional systemd unit files. These Unit files have the normal dependency chains listed for that particular mount point; using the example mount point in this wiki from fstab:

    /etc/fstab /data nfs vers=3,proto=tcp,hard,intr,rsize=32768,wsize=32768,noatime 0 0

    …the generator creates this file on boot, data.mount:


    # Automatically generated by systemd-fstab-generator



    These units are then instantiated by the Unit that is Wanted by the Unit. This methodology ensures that the networking layer is up and running, the rpcbind/nfs-lock service running and local filesystem ready to perform the mount in a fully automated method. No longer does a tech need to remember to enable a service like netfs to get NFS shares mounted at boot.

    The use of Generators goes beyond just NFS mounts – please see the official documentation for a full overview.

    So now I get to go down a Giant Rat Hole of something called “Generators” before I can say:

    mount server:/my/files /my/homedir


    Maybe tomorrow… For now I think I’m going to put an hour into that whole ‘make BSD boot on the R.PiM2 under BerryBoot…

  11. E.M.Smith says:

    Ok, that whole /run/systemd/generator/{files.mount} junk is in place… I thought maybe rpcbind was the hang up? But no…

    [root@ARCH_pi_64 generator]# systemctl enable rpcbind
    Created symlink from /etc/systemd/system/ to /usr/lib/systemd/system/rpcbind.socket.
    [root@ARCH_pi_64 generator]# systemctl start rpcbind
    [root@ARCH_pi_64 generator]# mount /SG15/ext
    mount.nfs: an incorrect mount option was specified

    And those “incorrect mount options”? “defaults”

    [root@ARCH_pi_64 generator]# grep SG15 /etc/fstab	    /Seagate/ext	nfs	defaults  0 0

    So just maddening… and then there was this ‘weird bit’ where at one moment a directory appears empty, but then magically it all reapears when you renter the directory…

    [root@ARCH_pi_64 generator]# pwd
    [root@ARCH_pi_64 generator]# ls
    [root@ARCH_pi_64 generator]# ls .
    [root@ARCH_pi_64 generator]# ls -al
    total 0
    [root@ARCH_pi_64 generator]# pwd
    [root@ARCH_pi_64 generator]# cd /run/systemd/generator
    [root@ARCH_pi_64 generator]# ls
    Pig-ext.mount	    'dev-disk-by\x2dlabel-PIG_swap.swap'
    Pig-ms.mount	    'dev-disk-by\x2dlabel-Seagate_500swap.swap'

    I’ve had mysterious disappearance of directories under SystemD before, but this one where it goes away, then returns, is just strange. I suspect it was changed when I did some command while in the directory and SystemD “did something trick” to rewrite it leaving me in an abandoned dir / inode… but really… The system is just NOT supposed to delete / remake directories if someone is in them… Just crappy design.

    While I’m really liking Arch Linux, I’m finding the systemD parts of it ever more irksome…

  12. beng135 says:

    Unfortunate about your Puppy experience — maybe it’s a driver issue. My Precise-Puppy 5.71 is fast & works fine (it’s expendable so I’ve abused it for experimenting and it holds up well) — no video or sound issues. Youtube or any other type of video I’ve come across works fine using the included seamonkey browser. Manually updating to the latest version of seamonkey is simple using its update feature. Tower is a circa-2007 HP w/a ATI Radeon video chip & Realtek HD audio chip.

  13. E.M.Smith says:


    I generally like Puppy, and on the Pi B+ it was actually better. Thus my statement that I think it is an early in the cycle release. They have chromium working, where other releases just don’t have it working at all. I could likely find a way to install FireFox / Seamonkey / whatever on Puppy and be happy, but just didn’t want to sink the time into it.

    On x86, Puppy is great.

    On my “someday” list is to unroll how Puppy works and “roll my own”. Just not right now, and not on the R.PiM2 (for now).

    Right now I needed “Something that Just Works right now and lets me be productive without too much pain and annoyance”. Arch seems to be that vehicle. (Despite my grousing about SystemD and NFS not mounting things… but that ‘failure to mount’ may not be Arch. I tried to get the same system to mount onto Fedora and Debian with no-joy, leading me to think something is “protecting” me… Maybe even my own iptables and firewall rules… The NFS Server was my DNS Server that is in the DMZ network, so has extra security issues… I WAS able to mount from Arch onto the DNS server…so it is a non-symmetrical filtering issue, I think…)

    At any rate, to clarify my position on Puppy:

    I am NOT disparaging Puppy in general, nor even Puppy on the original Raspberry Pi.
    I am saying that the port to Pi Model 2 is “young” and needs more polish / porting work.

    Not much of a surprise, really, when you consider how recently the board came out. About a year?

    So I generally install Puppy on every chip I have. I like to have it around for a “fast and light boot” and “alternate testing and / or emergency recovery” platform. I have Puppy CDs that I use in x86 boxes as a ‘quick system’ too. Yes, it has some quirks, and I’ve not felt in love with it enough to make it my Daily Driver, but I can see that moment from here… (It passes through that “roll my own” process… so will be a while). But on my “product glide path” is an eventual “light weight distro running mostly or entirely from memory and leaving no footprint, but able to save my state to USB” – sound familiar ;-)

    But at present, given my desire to do things like a distcc build farm, a FORTRAN compile and climate data / simulation environment over multicore Beowulf / MPI cluster or similar; well, that just doesn’t shout “Puppy!” at me… (But then again “I need to boot up and check a web page” doesn’t shout “configure and boot up your build farm cluster”: at me either…)

    And yes, I run way too many different OSs and chips. It’s in the DNA of systems admin types to want to play around with different distros and boot a dozen different things in a day ;-) At any one time I’m running about 3 computers at once, on at least 3 different distributions, and occasionally booting 4 different OSs on the same computer in one sitting… One of them often Puppy. Not your typical usage profile, I know ;-)

    Tablet (Android), DNS Server Pi (old Debian), Desktop Pi (any of Fedora, Ubuntu, Arch, Puppy, Debian, Kali, depending on what I’m doing / need), ChromBox (ChromOS), x86 Du Jour (Windows in several eras, Debian, Devuan, Red Hat of several eras, CentOs, Caldera Linux, and a dozen or so CD based Linux releases, oh, and a Knoppix on a chip for when I need a quick Linux on a laptop that normally is Windows along with a “Solaris in a QEMU chip” when I want a very slow Solaris for something…)

    So folks may have noticed I’m often carping about one or the other of the releases at any one time, yet never abandon them…

    My present “motivation” comes from a simple source: Hardware Failure.

    Basically, the old White Box PCs need to go. So I need to ‘catch up’ about 20 years of crap on them into the New World. (as noted, one of them is running NFS original..) They still boot, but a bit sporadically now. The EVO has a video system that nobody seems to support anymore, and running a decade old Browser doesn’t work on any web pages anymore. That, and the CD/DVD works some days, others not so much. Then there’s that 64 Bit Antek… Filled the role of emergency box for cheap, but has a temperature dependent disk failure starting. Works fine, until an hour or two of warmth, then the disk suddenly goes read only and eventually the system hangs. It’s time for that collection of “old x86 PCs” to be let go.

    To do that, I needed a place to rapidly suck data, that works with modern browsers, and is fast enough to be comfortable. That was the Pi Model 2. I used Debian for the data suck over, but Arch is faster as a Daily Driver for things like posting, and music works…. It also can serve NFS OK so I’m just going to skip the question of mounting that other system and “move the disk” to a different Arch box as my “file server”. I can move the USB disks to any box, so the use of NFS is mostly for my amusement, now that I’ve gotten all the data off the hard disks in the old WB PCs.

    Once I’m a bit more consolidated onto the R.Pi for All Things Linux, then I’m down to three major environments: Pi / Linux (of many colors), Tablet (Android), and the occasional Chromebox (ChromeOS). Yet even there, Arch is fast enough I’ve not booted the ChromeBox to do postings / browsing / videos… so it is potentially next on the chopping block (i.e. conversion to Linux )

    I know TMI (Too Much Information)… but I wanted the context shown. I’m not using just The One True OS! but instead wander around as needed, and one of them is Puppy. I’m not abandoning it, just limiting the use to things I find it best at doing. I often use it for a quick “boot and look at an SD card” or on the old Pi and PCs “check a quick web page”. I had it on a USB stick for the PCs for such things. Eventually I’ll install another browser on the PiM2 Puppy Alpha4, but maybe after it’s matured a few months… I’d have it already done that but for Puppy having their own way of doing things, so needing learn time, and being pressed for time as the hardware clock is ticking…

    The key take-away here being that each hardware / OS couple has their own quirks. Puppy is an example, with Puppy on a PC being very mature and usable, while Puppy on the Pi Model 2 is still young, growing, and with bits missing. ALL OSs are like that. FreeBSD is solid. Debian is fairly solid. DebianBSD is not so complete or mature… not enough folks porting Debian user space to BSD Kernel. But it will get there eventually. Devuan is pretty complete, but an early port, just now Beta, so in 6 months or so it will be even better. Devuan on the Pi? Not even a dream yet…

    So now it’s time for me to get back to decommissioning 3 WB PCs, an Evo, an Antek, about 4 old laptops from the last 2 decades, etc. etc. and finishing the consolidation onto hardware made in this millennium ;-) After that, I’ll do more with things like Puppy Custom Build and such…

  14. beng135 says:

    Tried to install FreeBSD and the installation menu was so convoluted I gave up — didn’t see an option to install to a single partition (already have 4 primaries). But I’m not at a high-level. Puppy didn’t have dnsmasq originally (I liked it in Linux Mint), but it was easy to install and work with minor tweaking of the conf file.

    Love the hardware clock ticking comment — yes, I’ll use hardware/software until it breaks or is simply unusable (like dial-up modems/ISPs and Windows95 & NT). Use it up, wear it out — an engineer’s mantra. The coal power plant I worked at in the oldest part of the plant (the units there had been long retired/scrapped) had some 1930s pneumatic/mechanical controls, electric motors (most never rebuilt), transformers, pumps, valves, etc still working fine (rest of the plant closed down now by the EPA).

  15. E.M.Smith says:

    Yes, it seems to be part of the Engineer’s Core Values to keep using a thing as long as it goes…and never waste… Part of my early resentment toward the Global Warming Crap since then wanted to trash a LOT of working stuff… Offended my “use it up first” core value…

    FreeBSD is not for most folks. BSD has long been “User Hostile, but Sysadmin Friendly”. So only for those who have a near religious zeal for all things BSD…. Maybe my destiny is to make a user friendly BSD install… oxymoronic as that may be…

    Right now I’m happy with Arch as a ‘middle ground’ between BSD and Debian. Next month? Well, that will be something different…. And my faithful Puppy is always there when I screw up something and need a quick-boot recovery system ;-)

  16. anon says:

    Void Linux ships ready-to-go images for Raspberry Pi.

  17. anon says:

    So does Alpine Linux. Void uses runit. Alpine uses OpenRC for now, but has general plans for s6 in 1-2 years. The s6 dev is active on Alpine IRC.

  18. E.M.Smith says:


    As a sysadmin. I understand the desire for privacy and security, but I would encourage you to use a real name (even if pseudonym) and real email address (even if disposo address as I use).

    That said:

    Yes, and I have run that Alpine and will be test driving the Void one in a couple of days ( when my bills are paid and the beer runs out :-)

    My intent is to make a “spin” (to use the Red Hat term) with my idea of the most useful applications installed and with a critical eye watching the process. Then script it so that the less technical can get the same result with less effort.

    Essentially, I care a lot about security and privacy AND want to make a spin meeting those ends available to the aversge Joe and Jane without pain and suffering. Preferably in a way they can choose to just download a binary package or do a source compile at their option and easily.

Comments are closed.