The SystemD Armbian Pine64 Swap Trap

Well that was fun… NOT!

In a comment on an open discussion thread, I’d mentioned I was using the Pine64 as a “Daily Driver” for a while to see if it was suitable (in the PineBook) as my tablet / mobile stuff replacer. The answer is more or less “almost”, maybe.

It plays videos OK in a Theater Mode on Chromium (FireFox in small size only). That was with 2 tabs open. Memory use was about 1/2 of the GB (a bit more in FFox). I can live with that.

Then, since Alex Jones is busy being attacked and deplatformed, I decided to just open a tab to his website and run a video there. I’d been doing this on the Roku (partly just as a ‘thumb in the eye’ of the SILENCE!!! crowd) but he got nuked there, to. So I’m setting up to have it just run on a browser… but on the Chromebox it would run for about an hour then get a framing error of some sort. OK, I’ve got lots of other hardware… (I won’t mention that I have zero such errors on any OTHER site…)

So I launched his front page an in about 2 minutes the whole system turned to slow-mo mollasses in January in Fargo S.D.

WT? It looked like it was locked up hard, but every so often the “htop” panel would change a little. OK, we’re in some kind of thrash lock. I clicked on htop (slowly nursing the mouse over, one inch of mouse and wait a minute to see where the cursor moves, one inch, move…) and waited. It showed ALL of the GB of memory was used and ALL of the 480-ish MB of SWAP was used. OK, we’ve run out of swap. The OS is trying desperately to load everything and is busy just playing with swap and trying not to misplace its mind. Got it.

Clicking on the “close box” for Chromium, or for the individual tabs didn’t do anything. Likely because too much of the browser was swapped out.

Now, one thing I typically do is have a “root window” open. I’m usually playing with some kind of system stuff anyway, and it’s an old habit from being the System Police to be ready to kill any rogue process or user or “whatever” threatens the system. Lucky for me it was just visible at the bottom of the screen. After about 3 minutes of inching the cursor down to it, click. Then typed “halt” and carriage return. Then waited. After a few more minutes, the command showed up and the system halted.

FWIW, on the Odroid XU4 I have the same InfoWars Browser window open and I’ve filled all of the 2 GB of memory along with 64 MB of swap space. Make sure you have more than 2GB total of memory + swap. I have 3.5 GB now on the Pine64; 4 GB on the XU4. Page Weight is reaching a whole ‘nother level… So “note to self”, for that PineBook get extra memory… if possible. Hitting the wrong web page can put you in thrash lock.

Quasi Fixing It

Looking into things, after a reboot… it looks like Armbian Ubuntu puts all of swap into a compressed temp disk image in RAM. Yeah, to swap out of RAM you use a RAMdisk… This avoids write latency to the uSD card and wear on the uSD card and it effectively turns memory usage into CPU usage (for the compression / decompression), so “I get it”. But really? NO provision for an overflow area out of RAM entirely?

So I’ve added a 2 GB swap area on the uSD card. Using it first, the YouTube music video gets a bit more “choppy”, but Alex didn’t crash my system anymore. I’m now going to change the priority so that the uSD swap is used as a ‘backing store’ (i.e. lower priority) behind the RAMdisk. Given the thermal limiting I saw on videos in larger screen sizes, this might work well (with a heat sink) or it might be better to reduce that CPU use by having the uSD swap come first.

In doing the swap setup, I discovered (of course…) that “SystemD Does It Different”. Sigh. Now it has taken over swap and swap administration too. The cancer spreads… I’m not yet at the point of converting this install to Devuan ( I want a bit more performance information first) mostly because the PineBook is likely to have special drivers that may or may not be happy with Devuan. I might end up stuck with a systemD system for a while if I buy one. (another “Dig Here”…)

So here’s the Tech Talk part:

As root, inspecting where swap presently is. Oh, 4 small ramdisks of about 125 MB each.

root@pine64:/# swapon -s
Filename				Type		Size	Used	Priority
/dev/zram1                             	partition	125292	0	5
/dev/zram2                             	partition	125292	0	5
/dev/zram3                             	partition	125292	0	5
/dev/zram4                             	partition	125292	0	5

Then adding a swap file (as I don’t have a Real Disk partition…) On the “someday” list is to shrink the filesystem on the uSD card and add a Linux Swap partition. Yes, I know a swapfile is just as good… I just like to see my swap space called out when I do things like gparted or look at fstab. Knowing the device it is on is helpful sometimes (especially when you move devices around a lot like I do). Normally I put swap on a USB real disk too. In this case I’m not doing that as the intent is to model what the PineBook experience will likely be.

I typically name this file SWAPFILE. So the first step is to see if it will have a name conflict:

root@pine64:/etc# ls S*
ls: cannot access ‘S*’: No such file or directory

OK, good to go. Now we make a 2 GB file, make it read / write only for Root, make a swap format space in it, and turn it on:

root@pine64:/etc# fallocate -l 2G SWAPFILE
root@pine64:/etc# ls -l SWAPFILE
-rw-r--r-- 1 root root 2147483648 Feb 18 18:05 SWAPFILE
root@pine64:/etc# chmod 600 SWAPFILE
root@pine64:/etc# mkswap SWAPFILE
Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
no label, UUID=a6a94d05-e5b7-4010-89bb-95438d60636a
root@pine64:/etc# swapon SWAPFILE

Did it work?

root@pine64:/etc# swapon -s
Filename				Type		Size	Used	Priority
/dev/zram1                             	partition	125292	0	5
/dev/zram2                             	partition	125292	0	5
/dev/zram3                             	partition	125292	0	5
/dev/zram4                             	partition	125292	0	5
/etc/SWAPFILE                          	file    	2097148	0	-1

Yes, but with a “don’t use me unless desperate” priority setting. OK for a backing store, not so good for testing it… So I decided to make it higher priority for testing, and put the mount of it as swap in the /etc/fstab place where it has always been.

Being just a tiny bit (justifiably…) paranoid that SystemD had screwed this up too, I did a quicky web search that turned up “Yes! Screwed the Pooch Here TOO!”…

“Why? Don’t ask why. Down that path lies insanity and ruin. -E.M.Smith” It is just what Pottering does. There is a high dick-with factor in that one.

Manual page here has the gory details.

A unit configuration file whose name ends in “.swap” encodes information about a swap
device or file for memory paging controlled and supervised by systemd.

This man page lists the configuration options specific to this unit type. See
systemd.unit(5) for the common options of all unit configuration files. The common
configuration items are configured in the generic [Unit] and [Install] sections. The swap
specific configuration options are configured in the [Swap] section.

Additional options are listed in systemd.exec(5), which define the execution environment
the swapon(8) binary is executed in, in systemd.kill(5), which define the way the these
processes are terminated, and in systemd.resource-control(5), which configure resource
control settings for these processes of the unit.

Swap units must be named after the devices or files they control. Example: the swap device
/dev/sda5 must be configured in a unit file dev-sda5.swap. For details about the escaping
logic used to convert a file system path to a unit name, see systemd.unit(5).

AUTOMATIC DEPENDENCIES

All swap units automatically get the BindsTo= and After= dependencies on the device units
or the mount units of the files they are activated from.

Swap units with DefaultDependencies= enabled implicitly acquire a Conflicts= and an After=
dependency on umount.target so that they are deactivated at shutdown, unless
DefaultDependencies=no is specified.

Additional implicit dependencies may be added as result of execution and resource control
parameters as documented in systemd.exec(5) and systemd.resource-control(5).

BLAH Blah blah blah BLAH blah Blah….

The only bit that matters is that they left a hook to let you do it by the “old way” of one easy line in the /etc/fstab file

FSTAB

Swap units may either be configured via unit files, or via /etc/fstab (see fstab(5) for
details). Swaps listed in /etc/fstab will be converted into native units dynamically at
boot and when the configuration of the system manager is reloaded. See systemd-fstab-
generator(8) for details about the conversion.

If a swap device or file is configured in both /etc/fstab and a unit file, the
configuration in the latter takes precedence.

So I can ignore that crap unless someone else has not ignored it. OK… I guess maybe he’s starting to hear the folks who have complained that changing how everything is administered is a Royal Pain ita and Breaks Things.

I can live with that (barely)… So I edited the file and added the one line in bold at the bottom:

root@pine64:/etc#  cat /etc/fstab
UUID=f56fc0fc-5c24-4529-86fa-d3819850ed05 / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1
tmpfs /tmp tmpfs defaults,nosuid 0 0
/etc/SWAPFILE none swap 	sw,pri=6 0 0

then rebooted. Now the uSD is one unit of priority ahead of the RAMdisk. I did this for testing and will change that pri= to a 1 after this, so ramdisk will come first and then the uSD space. Then performance ought to be like it was before while the system won’t hang when a Huge Fat page comes along.

Here’s the present swapon output:

root@pine64:/# swapon -s
Filename				Type		Size	Used	Priority
/etc/SWAPFILE                          	file    	2097148	165.3M	6
/dev/zram1                             	partition	125292	0	5
/dev/zram2                             	partition	125292	0	5
/dev/zram3                             	partition	125292	0	5
/dev/zram4                             	partition	125292	0	5

More useful information on swap and tuning here:

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04

Includes things about adjusting swappiness and cache pressure settings.

In Conclusion

Once again my loathing of SystemD has been reinforced and extended. Swap worked FINE. It wasn’t broken, don’t “fix” it.

For any 1 GB memory system, you will need to check where any configured SWAP actually goes. If it is all ramdisk, and too small, add some more space so your system doesn’t go into thrash lock on SWAP.

This demonstrates that “limited hardware” is limited, and limiting. You can only trade one part for another if you have some part in excess. (RAM, CPU, Disk I/O swap, etc.).

It’s good to keep a “reboot me” window open where you can slide a mouse over it and just type “halt”.

If at all possible, get more than 1 GB of memory in systems where you expect to visit random web pages. Some can kill a 1 GB system (or at least wound it horribly).

With that, I now return you to your regular programming…

Subscribe to feed

Advertisements

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in Tech Bits and tagged , , , , . Bookmark the permalink.

9 Responses to The SystemD Armbian Pine64 Swap Trap

  1. E.M.Smith says:

    Running the Infowars site “live” feed, I’m now at 544 MB of swap used (so definitely out on the uSD swap space). I’ve done the “shrink file system and add swap partition” so it’s now got 4.48 GB of swap, with the zRAMdisk first, then the swap partition, then I’ve left the SWAPFILE in place for now just to see what’s really needed, but last in the priority list.

    Interesting to note that the ‘jerky bits’ seem a bit less now that it’s rolled to swap partition from zRAMdisk. Likely as the CPU demand goes away. Temperature running at 83 C to 85 C and clock down to about 1 GHz ( 912 MHz to 1.01 GHz wobbling between them). So a heat sink is a necessity.

    Some other day I’ll go back to “swap partition first” and see if it makes any difference.

  2. E.M.Smith says:

    Here’s a look at swap use now. Oddly, it’s spreading around more than I’d expected:

    root@pine64:/#  swapon -s
    Filename				Type		Size	Used	Priority
    /etc/SWAPFILE                          	file    	2097148	166220	1
    /dev/zram1                             	partition	125292	125204	5
    /dev/mmcblk0p2                         	partition	2097148	166264	4
    /dev/zram2                             	partition	125292	125192	5
    /dev/zram3                             	partition	125292	125208	5
    /dev/zram4                             	partition	125292	125176	5
    

    So that’s kind of interesting. Might be amusing to make priority more “diverse” and move the SWAPFILE out to -1 and the uSD partition up to 1024. Leave the zram in the middle… That ought to force the partition to be used first, then use the zram and only if desperate move to the SWAPFILE.

    I may do that later… For now it’s interesting to watch that live video on Chromium means constantly growing memory use so constantly growing swap use. Now at 840 MB as I type, so up about 100 MB over the time I’ve been writing this. Some tuning of cache pressure and such may be helpful… or not ;-)

    I’m curious, now, when it decides to release that memory? IF you watch a Live Broadcast for all day, do you end up with 20 GB of swap? Or at some time does cache pressure and other settings start to trim the use?

    I’m also going to try this on FFox later. See if they both have the same behaviour. It seems a rather “less than good” behaviour. Hmm… just saw swap drop from 858 to 857 MB so it is doing some space management. Maybe there’s some parameter saying to let it run up to almost the same as real mem then trim? Well, also for later to explore that…

  3. E.M.Smith says:

    Interesting… both cashe_pressure and swappiness are set to 100. That says dump cache information as fast as you can (so inodes and such get dumped rather than swapped) from memory, and we’ll try to put as much as possible into swap. That seems a bit silly, even with ramdisk as swap.

    I’m changing those (in /etc/systcl.conf) to a swappiness of 60 and cache_pressure of 50. More normal values. (Because I’m using uSD not just all a few small RAMdisks).

    To see the valued:

    cat /proc/sys/vm/swappiness
    cat /proc/sys/vm/vfs_cache_pressure
    

    Overall, I find the system barely OK as a “Daily Driver” IF you intend to watch videos. It is fine if you don’t load it up with a video load. When running videos, it needs a big added heat sink and some tuning along with some big swap (that will likely burn your uSD lifetime fairly fast).

    A much better system would be about 2 GB of memory+ and at least 1.5 GHz CPU clock (for this level of Arm arch – the A56 – as each arch has a different performance / clock the arch matters too..) This one is better at 1.2 GHz but not perfect, and degrades as heat causes clock to drop to 1 GHz. My guess is that about 1.5 GHz is the tipping point. On the Odroid XU4, at about 2 GHz, (on an A15 arch) it is just fine; so that’s the upper bound of needed.

    So I think I’ve got my answer. The PineBook ought to be barely OK for light use, but don’t expect to hit heavy web pages with lots of video with hurky jerky performance, heat issues, and uSD / SSD burn.

    I think the Chromebook is likely the winner here. Yes, it’s close to double the price (or more), but you can run ChromOS and Linux so videos in ChromOS ought to be OK and you still have a full Linux so it is usable for more than a browser station with Google looking over your shoulder… More on that after I do a test run on my ChromeBox a while from now.

    For now, I think I’m going to run this SBC the rest of today, but “move on” to the Rock64 and / or RockPro64 with similar “no disk” limits to see what (should it show up in a PineBook “someday”) would be a “buy one” marker for me.

    FWIW, swap usage seems to have stabilized at a bit under 1 GB. 878 to 950 MB or so wandering back and forth.

  4. Graeme No.3 says:

    Obviously computers are planning to take over the World, it’s just that they won’t be able to tell us when they do.

  5. E.M.Smith says:

    The performance of the Pine64 was starting to wear on me (sound pauses and some occasional Max Headroom moments); so I swapped over to the Rock64. It is a 2 GB memory system.

    So now InfoWars is running with stable sound, and only modest video ‘jerking’. I’ll do a test of a YouTube music video later.

    This is also a straight Armbian with SystemD build and no disk (though it has a free USB 3.0 port so I could in fact put a nice fast disk with fast /tmp and swap on it.. Swap is also on RAMdisk, but these are twice as big. /tmp is also on a tempfs file system.

    Sound worked when booted ( I think it worked when I first booted it long ago… I don’t think I did anything special then to make it “go”…).

    Running with a heatsink on it, temps have stabilized about 80C to 82 C and it is holding a steady 1.3 GHz from bootup to full warm. CPU use is running 95% to 99%, so this puppy is pegged. We’ll see if that backs off on a music video a bit later.

    Swap has started to be used at 28 MB, but memory is only saying 1.55 G out of 1.95 Gb so not sure what that’s about. (Maybe that is due to the tempfs space used…)

    Overall, it’s quite acceptable to me as a background news show. The video is a bit jerky for direct watching IMHO, so need to cut down resolution (not as easy as on youtube or at least I’ve not found a way yet). It is also A53 cores so directly comparable to the Pi M3 and the Pine64.

    So this is an improvement, but not perfect. It continues to argue for 1.5 GHz + on A53 or better architecture cores, with 2 GB of memory + some swap; as being more or less the “lower bound” of acceptable for a laptop used for videos.

    Oddly, the YouTube music video is only completely smooth at 480p in the small size on Chromium. Something is different between this OS / Browser build and the one on the Pine64, or the graphics engine is different enough to cause a bit of reduced smooth.

    This is even after a big update-upgrade. (So only release level issue would be the general age of the kernel used and the specific build). It is quite possible that one of them as the specter / meltdown patches that slow the kernel and the other one does not?

    FWIW, running music video didn’t result in any swap usage.

    So the Rock64 starts getting closer to “fine” for a music browser experience, but still has some issues with rendering speed. I suspect they still have not got full use of the Video Cores going on… Lesser chips sets on Android do way better, but Android strongly cares about the video experience.

    In general, though, it is looking like 1.5 GHz minimum processor speed in an A53 or better core ( 4 of them…) with good heat management and 2 GB of memory + swap is about the minimum config for running videos.

    The other interesting bit is that on this box. the newly installed FireFox runs a 480p music video smoothly in theater mode size. CPU use at 68% to 96%.

    So looks like there are some specific software / hardware interactions that can favor one browser over the other. No idea why. Maybe one is favored by a given GPU over the other? Specific release levels? Way lots of memory gets used? Who knows… But it looks like FireFox on this hotter box and Chromium on the Pine64. Probably needs a better benchmark that just my running the same video ;-)

    So my conclusion is that with some tweeking the Rock64 is “good enough” for music videos in less than full frame less than HD.

    Since I seem to be on a roll about testing these boards for video, I think I’ll just move on and see about the RockPro64 now. Last time the software was a bit “young” and rough in spots. With A72 cores it ought to have not issues, though, so it would be a hot one in the laptop. (Wonder if it could retrofit ;-)

    It does look like I’m generally gravitating toward the $50+ boards and away from the $40- ones…

  6. E.M.Smith says:

    Well we have more answers on the RockPro64.

    Sound doesn’t work on Ubuntu, Debian, or even on the DietPi latest releases.

    Debian Stretch even breaks lightdm after an update upgrade so can’t even get a windowing system login going.

    It all seems to come down to ALSA and PulseAudio being a fat complicated system and folks not making it work right (likely because it isn’t “the unix way” of some small thing doing one job well).

    Maybe in a few more months… So not usable as a web browsing media station, but otherwise OK.

    I’m presently using the XU4 in that mode (the DVI monitor doesn’t get HDMI audio through the converter and the direct cable plugin is way too soft a volume level), So it might be reasonable to swap it over as media player and use the RockPro64 as the workstation.

    Or just forget about it and put it in the box again for a few more release cycles.

  7. CoRev says:

    E.M. forHDMI sound on the rockpro64 load the 4.4..138.1100 kernel. Anything later sound is broken, but you still will have your SystemD nemesis installed. https://forum.pine64.org/showthread.php?tid=7006

    I’ve been locked on this version for the past several weeks until it is permanently fixed. I’m using the Rockpro64 as my daily driver and have found nearly everything is stable except for a network problem with mail in Firefox and my ISP. For mail I use one of my XU4s.

  8. E.M.Smith says:

    @CoRev:

    Thanks for that pointer to the kernel issue. As I was deliberately using the newest build / kernel, that’s likely the problem. The SystemD thing isn’t an absolute for me. It’s a PITA if you do much admin (especially across different systems so “everything you know is wrong” on SystemD…) but for “low use” systems where I don’t change much I can mostly ignore it. Unless it crashes and hangs…

    So, for example, I’ve got an NFS file server on an Orange Pi One that’s running generic Armbian with SystemD. I got it running, then have had zero need to change it so… why take on the workload of a “Devuan Uplift”? Just preference? Where I never log into it? So “someday” I’ll do it, if an issue arises or I need to work on it a lot so the admin changes are a big PITA. Or, whenever Devuan makes a release specifically for it. (They have some other chips supported, but not that one yet).

    So I’d be fine with a “media server” running that way too. I’d likely try the Devuan Uplift anyway, as that’s a hot enough board to want to do more stuff on it too… but if it failed, I’d go ahead and use the generic version for media. (Easy enough to change chips for purposes.)

    Guess I’m off to find a release copy with the 4.4.138-1100 kernel. (Or just find the kernel and do a “mix and match” into userland… I’ve done that before.)

  9. E.M.Smith says:
    Armbian_5.75_Rockpro64_Debian_stretch_default_4.4.174.7z
    Armbian_5.75_Rockpro64_Ubuntu_bionic_default_4.4.174_desktop.7z
    

    Yeah, that looks like the problem…

Comments are closed.