Some folks might remember some years ago I got an Odroid C2 and loved it. Then it mysteriously “died”. Then a year or two later is was working again. Then the Rock64 “died” (it is still in a “Dead Boards” box). Then several months back the Odroid N2 “died”, and now I’ve figured out that while it LOOKED dead (not booting, nothing on screen) it was really hung in boot on /etc/fstab changes.
Well, the N2 was too expensive for me to just ignore, so after a certain amount of “set it aside for me to cool” time I whacked at it for a few days and figured it out. Now I’ve installed Armbian on it; that seems more resistant to that particular SystemD / fstab interaction bug. (I’m not willing quite yet to say it has no chance of it…)
So now I’m wondering if the mysterious “death and resurrection” of the C2 was a similar thing. This has encouraged me to dig the Rock64 out of the “Dead Boards” box and try a re-spin of that OS to test it, too. The particularly pernicious nature of that bug would mean that, for example, if you removed a disk (and it was noauto 00 in fstab so you think it need not be connected…) even if you restored a back up of the OS, it would still “act dead”. I did several restores of backups for each of those boards…
So sometime this month I’m going to put a day into recovery of the Rock64 with that in mind.
Why mention this?
This particular bug is strongest, near as I’ve been able to tell, in Ubuntu. BUT it could be in any SystemD operating system. Ubuntu (and other SystemD infested things like the “upstream” Debian) are THE most common OS shipped with the boards or provided on their support sites.
LOTS of folks are likely to have experienced this.
So IF you have a board “suddenly die”, it has a good chance of not really being dead.
I’m a natural “parts” packrat, so didn’t toss mine out. Other folks? If it is “dead” they toss. So don’t toss, OK?
When the C2 came back to life, I had no idea why. IIRC I’d just thought “Well, OK, lets just try a pristine OS copy”… which would argue for the fstab thing being causal as it would be gone then, but present in the backup copies (needing only a disk removed from the system but the mount description left in /etc/fstab and why would you plug back in a disk you had removed?…) I still have those old backup images, so I can test this thesis (but not any time soon… too much other to do).
I wonder how many hundreds of working Single Board Computers have hit the scrap heap from that one bug.
So, I would advise that as soon as you have the board booted up, change the configuration so that it does NOT give you the blank screen during boot, but gives you the running commentary and a login prompt for maintenance if needed. IF you already have a “Dead Board” try editing the uSD chip on another system to comment out any disk partition mounts.
I spent a good chunk of yesterday trying a Gentoo Linux installation on the Odroid XU4. Gentoo starts with a core “tarball” and you unpack it into a “chroot” partition where you can “chroot” into it and it looks like it is a new isolated computer. (chroot is a way cool thing for security. It lets you turn one set of hardware into what looks like two isolated computers. It was a key part of how we kept projects isolated on the Cray. Folks on the project side could not see the systems admin side, for example.) Then you do a build of the rest of the system from source tarballs.
I really really like the idea of building the system from source. However…
Their package manager, portage, is a PITA. It has too many knobs and configuration flags. I know, I know, it’s just enough for “the experienced operator” to get it to run on a a zillion different computers and just the way they want… But…
After about 6 hours of compiling things and completing something like 138 packages, it barfed on the next step. Some config line in one of the half dozen (or more?) configuration places isn’t right. Or something.
Near as I can tell there’s a missing directory / file:
>>> Unmerging (6 of 11) acct-group/render-0... >>> Unmerging (7 of 11) mail-mta/ssmtp-2.64-r3... >>> Unmerging (8 of 11) sys-devel/binutils-2.30-r2... * Switching to armv7a-unknown-linux-gnueabihf-2.32 ... [ ok ] !!! Section 'gentoo' in repos.conf has location attribute set to nonexistent directory: '/var/db/repos/gentoo' !!! Invalid Repository Location (not a dir): '/var/db/repos/gentoo' * Please remember to run: * # . /etc/profile >>> Unmerging (9 of 11) perl-core/File-Path-2.130.0... >>> Unmerging (10 of 11) net-mail/mailbase-1.1... >>> Unmerging (11 of 11) sys-devel/automake-1.15.1-r2... Packages installed: 206 Packages in world: 0 Packages in system: 43 Required packages: 206 Number removed: 11 * Regenerating GNU info directory index... * Processed 96 info files. odroidxu4 /usr #
These are the packages that would be merged, in order: Calculating dependencies... done! emerge: there are no ebuilds to satisfy "app-portage/gentoolkit". emerge: searching for similar names... emerge: Maybe you meant any of these: app-portage/gemato, app-portage/portage-utils, app-portage/elt-patches?
So where is repos.conf file? Unknown. They don’t give the full path name. One hopes it is the same as the config file that was edited per the “DIY write-up”. “The experienced operator will just know” is the usual sniditude applied to such circumstances. To what ought it be changed? Unknown. Ought it be changed? Unknown.
I’m hoping that I can just make the directory. That it doesn’t need to have anything in it, it can just be made and that’s it. Will that work? Unknown.
FWIW, I’m following the not-quite-complete cook-book here:
It has a different set of choices for a few items than those in the default as it is found in the tarball. I suspect this is where things go off the rails. I used the settings in the page to replace those in the tarball. This is done in a “repos.conf” file that one hopes is the same “repos.conf” file the error message is talking about. But maybe it isn’t… There’s a bunch of available config files and some of them seem to have the same or similar names, just different paths (per various pages I’ve read). Or maybe it’s the same file, just they “configured it” to a different location, or the location changed over time. (This is WHY I complain about capricious changes. It leads to confusion “downstream” or “later” unseen by the high dick-with-factor upstream developers [cough, Pottering, for example on the Debian side}).
There is a HUGE advantage to consistency and standards for things like the name space, location of files, methods of operation. The BSD folks “get that”, but are accused of being old fuddy duddies for it.
CHOST="armv7a-unknown-linux-gnueabihf" PORTDIR="/var/db/repos/gentoo" DISTDIR="/var/cache/distfiles" PKGDIR="/var/cache/binpkgs"
As unpacked, those values are not in /var.
So can I just mkdir /var/db/repos /var/db/repos/gentoo and be “good to go”? Unknown. But I’ll give it a try when I’m interested in wasting a day or two, again…
Oh, and that day of compiling things isn’t all lost ( I think) in that I can just make a tarball of it and stuff it off on a disk as “XU4 base install plus make world” and pick it up again some other day.
I got Gentoo to work on the Raspberry Pi some long time ago, so it isn’t like I’m a complete Noob with it. But it really does illustrate the “User Hostile” side of Linux. I appreciate that their whole thing is “build it all from sources” so you know it is clean. It avoids all sorts of potential hidden Bad Things from Bad Guys. But really, would it hurt them all that much to have a pre-built image for the dozen or two major platforms where a Noob could just “download and go”? It would remove a huge barrier to adoption. Then, over time, those folks could expand into the build from source mainline… WHILE having a working model to study and learn from.
A similar thing afflicts Slackware. They pride themselves on NOT having a package manager. As though forcing folks to keep track of a zillion dependencies that are most readily found once and codified in code is “a good thing”, when it isn’t. Oh Well.
So, for now, for whatever reason, it looks like the XU4 is not a widely supported bit of hardware for “variety” OS choices. There’s some DIY Kits (like Gentoo & BSD) but for “unpack and go” it is mostly Debian, Ubuntu and Armbian – all SystemD afflicted, or the Devuan port that is’t working right. So also, for now, I’m just going to leave it Armbian and “move on”.
There’s a LOT more “variety OS” support for the RockPro64. So I’m going to move my quest for “Variety OS free of systemD” over to it. Their site at least claims a working Slackware and BSD image. We’ll see. Sometimes they are just pointers to DIY recipes.
Also, it looks like there’s a lot more OS chioces on the Odroid C2. Now it isn’t the Hot Hardware these days, but… it’s faster than the R.Pi M3. So on the Odroid front I’m moving my Variety OS quest onto it (where it is more likely to succeed).
Hopefully this will let me get something free of SystemD running on either the Odroid C2 or the RockPro64. Yes, I’ve got a wonderfully working just fine Devuan on the x86 PC and on the R.Pi M2, M3, etc. So that’s nice and they are all taken care of. I’ve also got a “workable enough” Armbian with systemD running on the Ordoid N2 and XU4 (and without that fsck / systemD interaction bug) so they are “nervously workable” and I’m just going to leave them that way for a while. (Using the XU4 to post this BTW). So that’s a pretty good collection of “things that are working well, seem stable and reliable” and don’t really need more futz with factor.
BUT: As Devuan on the XU4 was clearly at a minimum not really tested well at all and buggy: I want a choice other than Devuan for a SystemD free OS.
So I’ll be whacking on The Usual Suspects, but on the “other” platforms (so Gentoo, Slackware, BSD on RockPro64, Odroid C2) and we’ll see where it ends up.
“I love what Linux / Unix let’s me do, but I hate how it makes me do it. -E.M.Smith”