Mounting Read Only / Compressed System Spaces

Archivemount

There’s an interesting tool that lets you mount compressed tarball archives as a live file system. Why? Well, you don’t need to unpack the whole thing to see what is in it, nor to update some one small file.

root@odroidxu4:/# apt-get install archivemount
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  archivemount
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 22.8 kB of archives.
After this operation, 88.1 kB of additional disk space will be used.
Get:1 http://auto.mirror.devuan.org/merged/ jessie/main archivemount armhf 0.8.3-1 [22.8 kB]
Fetched 22.8 kB in 1s (12.1 kB/s) 
Selecting previously unselected package archivemount.
(Reading database ... 95617 files and directories currently installed.)
Preparing to unpack .../archivemount_0.8.3-1_armhf.deb ...
Unpacking archivemount (0.8.3-1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up archivemount (0.8.3-1) ...
root@odroidxu4:/# which archivemount
/usr/bin/archivemount

Then you can just make a mount point directory and mount your tarball:

root@odroidxu4:/SG2/ext# ls
lost+found  TARball
root@odroidxu4:/SG2/ext# ls -l TARball/Orange_xfs.tar 
-rw-r--r-- 1 root root 647618560 Aug 13 09:53 TARball/Orange_xfs.tar

root@odroidxu4:/SG2/ext# mkdir mountpoint

root@odroidxu4:/SG2/ext# archivemount TARball/Orange_xfs.tar mountpoint

root@odroidxu4:/SG2/ext# df
Filesystem      1K-blocks     Used  Available Use% Mounted on
udev                10240        0      10240   0% /dev
tmpfs              204484      592     203892   1% /run
/dev/mmcblk1p1   30335916  3219452   26778676  11% /
tmpfs                5120        4       5116   1% /run/lock
tmpfs             1247820    42196    1205624   4% /run/shm
[...]
archivemount   1048576000        0 1048576000   0% /SG2/ext/mountpoint

root@odroidxu4:/SG2/ext# ls mountpoint/
Orange_xfs
root@odroidxu4:/SG2/ext# ls mountpoint//*

chiefio      Desktop	Downloads  ems.movedWD	Pictures  Templates
chiefiolink  Documents	ems	   Music	Public	  Videos

Nice, eh? Now the fun bit is that the file system is r/w.

root@odroidxu4:/SG2/ext# cd mountpoint/*
root@odroidxu4:/SG2/ext/mountpoint/Orange_xfs# pwd
/SG2/ext/mountpoint/Orange_xfs

root@odroidxu4:/SG2/ext/mountpoint/Orange_xfs# ls
chiefio      Desktop	Downloads  ems.movedWD	Pictures  Templates
chiefiolink  Documents	ems	   Music	Public	  Videos

root@odroidxu4:/SG2/ext/mountpoint/Orange_xfs# touch JUNK
root@odroidxu4:/SG2/ext/mountpoint/Orange_xfs# ls -l JUNK 
-rw-r--r-- 0 root root 0 Aug 13 13:29 JUNK

root@odroidxu4:/SG2/ext/mountpoint/Orange_xfs# ls
chiefio      Desktop	Downloads  ems.movedWD	Music	  Public     Videos
chiefiolink  Documents	ems	   JUNK		Pictures  Templates

So, at least in theory, you could mount a compressed tarball as your /tmp directory or /bin or “whatever”… as long as it isn’t needed before the facilities of archivemount are available… (i.e. boot time is a problem for some things). OTOH, mounting, say, a compressed /bin on top of the disk version of /bin after booted would work fine, and would protect that first version from being written (so any hacker messes up your bits, they just messed up the tarball bag of bits and it’s easy to recreate from the original image.)

I’m more likely to do this as play than as a serious systems protection behaviour, but it is an interesting option. Someone could still become root, see the odd mounts, figure out that they need to unmount them to get to the original bits and then diddle those (provided the present mounted file system image is not “in use” which prevents the unmount… and for many file systems that means you can’t unmount them until shutdown…)

To do that would take a few more options to the mount command. Here’s my script with them added:

root@odroidxu4:/# bcat mtar
echo archivemount -o nonempty -o allow_other -o intr -o default_permissions /SG2/ext/TARball/${1-tmp.tar} ${2-/tmp}
echo

archivemount -o nonempty -o allow_other -o intr -o default_permissions /SG2/ext/TARball/${1-tmp.tar} ${2-/tmp}

The “nonempty” lets you mount a file system on top of another mounted file system (so tmp.tar over /tmp that’s on a disk) while the “allow_other” lets other folks use it and the “default_permissions” prevents everything being assigned your usr and group ids. The “intr” lets you interrupt operations if things go wrong…

I may try that over /tmp some time later… It ought to work. I did test it to a different mount point just to see if I had the command right aand it worked fine. Only leaving interactions with a live running operating system as potential issues in a hot swap of /tmp to a tarball. (Isn’t it fun tormenting machines and changing their brains around while they are using them? ;-)

But my real interest in is using read-only file systems as a further security hurdle for attackers.

Squashfs

So while you could mount a tarball as Read Only if desired (for file systems that do not need writing), there is another choice. For that I’d be more inclined to use a squashfs compressed file system that is, by design and default, read only. I’ve done that in the past with /usr on the Raspberry Pi. “Stuff that changes” is supposed to all be in /var, so /usr only needs to be writable when doing system updates. (As long as folks are following the “rules”…) and I had no problem mounting a squashfs /usr and running.

If you have not installed it as part of your regular build, you will need to install squashfs. On Debian / Devuan like systems, this is done via apt-get install squashfs-tools.

root@odroidxu4:/# apt-get install squashfs-tools
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  squashfs-tools
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 79.3 kB of archives.
After this operation, 367 kB of additional disk space will be used.
Get:1 http://auto.mirror.devuan.org/merged/ jessie/main squashfs-tools armhf 1:4.2+20130409-2 [79.3 kB]
Fetched 79.3 kB in 2s (30.9 kB/s)       
Selecting previously unselected package squashfs-tools.
(Reading database ... 95610 files and directories currently installed.)
Preparing to unpack .../squashfs-tools_1%3a4.2+20130409-2_armhf.deb ...
Unpacking squashfs-tools (1:4.2+20130409-2) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up squashfs-tools (1:4.2+20130409-2) ...
root@odroidxu4:/# 

then you can use mksquashfs to make your squash file system files. I tend to do this with a small script so that I don’t have to type as much. As we saw before, I have two versions. One that compresses in place, the other that compresses to a given alternative directory. Named “sqit” and “squish”:

root@odroidxu4:/# bcat sqit
echo mksquashfs ${1-/tmp} ${1-/tmp}.sqsh -b 65536
echo
mksquashfs ${1-/tmp} ${1-/tmp}.sqsh -b 65536

root@odroidxu4:/# bcat squish
echo mksquashfs ${1-/tmp} ${2-/tmp}.sqsh $3 $4
echo
mksquashfs ${1-/tmp} ${2-/tmp}.sqsh $3 $4

As a reminder, the first one sets a block size (-b) 1/2 the default size. The second one uses the default but lets you chose a new directory as argument #2 and then add too more options as desired ( $3 and $4 ) to your invocation. Yeah, the lengths I go through to avoid typing a dozen characters… The use of /tmp in the commands is just so that if you do them with no argument, no bad thing happens:

root@odroidxu4:/# sqit
mksquashfs /tmp /tmp.sqsh -b 65536

Parallel mksquashfs: Using 8 processors
Creating 4.0 filesystem on /tmp.sqsh, block size 65536.
[===================================================================|] 4/4 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 65536
	compressed data, compressed metadata, compressed fragments, compressed xattrs
	duplicates are removed
Filesystem size 3.60 Kbytes (0.00 Mbytes)
	15.41% of uncompressed filesystem size (23.35 Kbytes)
Inode table size 266 bytes (0.26 Kbytes)
	43.68% of uncompressed inode table size (609 bytes)
Directory table size 360 bytes (0.35 Kbytes)
	73.92% of uncompressed directory table size (487 bytes)
Number of duplicate files found 0
Number of inodes 21
Number of files 4
Number of fragments 1
Number of symbolic links  1
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 6
Number of directories 10
Number of ids (unique uids + gids) 2
Number of uids 2
	root (0)
	chiefio (1616)
Number of gids 2
	root (0)
	chiefio (1616)

root@odroidxu4:/# ls -l /tmp.sqsh
-rw-r--r-- 1 root root 4096 Aug 13 13:14 /tmp.sqsh

So all that happened was that I got a squashfs of the /tmp file system in /tmp.sqsh.

Once you have a squashed file system, it’s easy enough to mount it. You use a “loopback” device for the mount.

I do this via a script:

root@odroidxu4:/# bcat mtsq

mount -o loop -t squashfs ${1-/SG2/sq/tmp.sqsh} ${2-/mnt}

but you could just type in:

mount -o loop -t squashfs /path/to/foo.sqsh /mount/point

Then it is mounted:

root@odroidxu4:/# df
Filesystem      1K-blocks     Used  Available Use% Mounted on
udev                10240        0      10240   0% /dev
tmpfs              204484      712     203772   1% /run
/dev/mmcblk1p1   30335916  3219480   26778648  11% /
tmpfs                5120        4       5116   1% /run/lock
tmpfs             1247820    83060    1164760   7% /run/shm[...]
/dev/loop0             64       64          0 100% /mnt

root@odroidxu4:/# ls /mnt
fileKEuurt  mozilla_chiefio0	pulse-PKdhtXMmr18n  serverauth.481Ng67uNU  ssh-XRKpwpiug2h0
lost+found  pulse-2L9K88eMlGn7	pulse-sA0qDsBuoAX6  ssh-pSUFgYCfEFzD	   xsession-chiefio

And there it is.

On A USB Stick

Not a long term good idea for active file systems like /tmp with lots of writes, as “bit wear” will eventually cause it to die, but for read only file systems, not so bad. I’m using a particularly slow cheap one ($5) just to see how bad it might be…

So what would it look like to move, say, /lib onto a squashfs file system on a USB stick, live?

First, you make the squashed image:

root@odroidxu4:/# squish /lib /O/ext/lib
mksquashfs /lib /O/ext/lib.sqsh

Parallel mksquashfs: Using 8 processors
Creating 4.0 filesystem on /O/ext/lib.sqsh, block size 131072.
[=======================================================================================/] 2561/2561 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments, compressed xattrs
	duplicates are removed
Filesystem size 33576.29 Kbytes (32.79 Mbytes)
	35.34% of uncompressed filesystem size (95015.21 Kbytes)
Inode table size 26853 bytes (26.22 Kbytes)
	30.82% of uncompressed inode table size (87120 bytes)
Directory table size 27888 bytes (27.23 Kbytes)
	47.03% of uncompressed directory table size (59300 bytes)
Number of duplicate files found 64
Number of inodes 2606
Number of files 2129
Number of fragments 262
Number of symbolic links  126
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 351
Number of ids (unique uids + gids) 1
Number of uids 1
	root (0)
Number of gids 1
	root (0)

It is interesting to watch it light up all 8 cores on the XU4 to 100% ;-)

Then you mount it:

root@odroidxu4:/# bcat mtsq
mount -o loop -t squashfs ${1-/SG2/sq/tmp.sqsh} ${2-/mnt}

root@odroidxu4:/# mtsq /O/ext/lib.sqsh /lib

Which issued: mount -o loop -t squashfs /O/ext/lib.sqsh /lib

With the result:

root@odroidxu4:/SG2/home/chiefio/bin# df
Filesystem      1K-blocks     Used  Available Use% Mounted on
udev                10240        0      10240   0% /dev
tmpfs              204484      592     203892   1% /run
/dev/mmcblk1p1   30335916  3219464   26778664  11% /
tmpfs                5120        4       5116   1% /run/lock
tmpfs             1247820    42228    1205592   4% /run/shm
/dev/sda5         3997376     8496    3779168   1% /tmp
/dev/sda3        20511356  9770196    9692584  51% /var
/dev/loop1          33664    33664          0 100% /lib
[...]

And at this very moment I’m running with /lib from a squashfs file system on a cheap USB stick.

IF anything went wrong, in the reboot it would not be automounted, so I’d come up on the regular one. Note that the df shows no free space on the device. That’s because squashfs is read-only. I can do the same thing with /usr /bin /sbin etc and make a lot of my system very very hard to change for any hacker. NOT impossible. They could copy the bits to writable space, unsquash them, rewrite, resquash, and then put them back on the source device. However, they would need to “become root” first and that’s made significantly harder if all the things they would like to overwrite in that process are locking them out. It becomes a “chicken and egg” problem where it is easier to put in place the binaries to become root once you become root… if only you could put them in place to become root.

Also, this device has a “flashy light” so any attempt at a lot of writes would have a flashing orange light near my right hand…. I ought to notice ;-)

Finally, note that you could simply put a real, writable (and perhaps even encrypted if you wished) file system on such a device. Keep your main system pristine, and then all the “tracking” information in places like /var or your home directory cache files could go to that device. On any AwShit (or shutdown or just not putting it in at boot time) the system itself would look like a normal pristine box. All the “history” would be on the encrypted dongle “somewhere”… I’d only do that with a GOOD USB device that could take the write cycles. Likely a u-SD card in an adapter.

SO that’s it for this posting. I’m going to continue playing with putting some other file systems on this USB stick and if anything interesting comes up, I’ll put it in comments (or maybe as an update).

UPDATE:

Just for grins, I put /usr as a tarball and mounted it:

root@odroidxu4:/usr# df
Filesystem      1K-blocks     Used  Available Use% Mounted on
udev                10240        0      10240   0% /dev
tmpfs              204484      592     203892   1% /run
/dev/mmcblk1p1   30335916  3219464   26778664  11% /
tmpfs                5120        4       5116   1% /run/lock
tmpfs             1247820       68    1247752   1% /run/shm
/dev/sda5         3997376     8496    3779168   1% /tmp
/dev/sda3        20511356  9770460    9692320  51% /var
/dev/loop0          33664    33664          0 100% /lib
archivemount   1048576000        0 1048576000   0% /usr
/dev/sda8          999320     2232     944660   1% /root
/dev/sda9          999320     6684     940208   1% /bin
/dev/sda10         999320    10496     936396   2% /sbin
/dev/sda11         999320     6868     940024   1% /etc
/dev/sda12      103081248 61166676   36671692  63% /SG2/home
/dev/sda14     1765835816  2499324 1673630508   1% /SG2/ext
cgroup                 12        0         12   0% /sys/fs/cgroup
/dev/sdb9          999320   756252     190640  80% /O/ext

With /lib as a squashfa on the USB stick it wasn’t bad. Putting /usr as a tarball, even from Real USB Disk on USB 3.0 interface, was way way slow. As a tarball is not a file system, every lookup for every file must traverse the file looking for the bits.

hiefio@odroidxu4:~$ ls -l /SG2/ext/TARball/
total 2455564
-rw-r--r-- 1 root root  647618560 Aug 13 09:53 Orange_xfs.tar
-rw-r--r-- 1 root root     604160 Aug 13 16:01 root.tar
-rw-r--r-- 1 root root      33280 Aug 13 14:03 tmp.tar
-rw-r--r-- 1 root root      40960 Aug 13 13:42 tmp.tar.orig
-rw-r--r-- 1 root root   26705920 Aug 13 16:26 usr.tar

Searching a 26 MB file every time to want a few bits is slow.

But, surprisingly, things DID work! Very surprising, really. I’d have expected more issues.

Still, strongly not recommended unless you like long waits…

Subscribe to feed

Advertisements

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in Tech Bits and tagged , . Bookmark the permalink.

13 Responses to Mounting Read Only / Compressed System Spaces

  1. wyzelli says:

    Interesting. Could you create a tarball of /bin and then mount that on top of bin – say each time you boot? That could be an interesting way of enforcing a known system state every reboot.

  2. E.M.Smith says:

    @wyzelli:

    Yes, you can, but I’d not recommend it simply because I did it (with /usr) and it was painfully slow. The tarball, not being a file system, it must do a linear search of the tarball each time you access it.

    I’ve also done it with squashfs (that is a real file system) and it does not have the penalty hit.

    FWIW at present I actually boot from the u_SD card /bin /sbin /usr /lib etc. and then it mounts other versions on disk partitions over them, so in actual production it’s the disk copies that are “at risk”. Simply reboot with the disk unplugged (and as super user change the fstab to not mount those partitions) and you are back at running off the copy on the SD card.

    You can also put the disk mount entry as “read only” for file systems like /lib and /bin to further prevent changes to the system.

    In Live Linux Systems they copy all the relevant directories from the CD or DVD or USB stick into a memory based file system (tmpfs or ramdisk) and run from that. This is another way to get a consistent starting point each time.

    FWIW, tonight I made my own first tmpfs file system. It is just wayyyyy tooo easy.

    mount -t tmpfs -o size=100M,mode=0755 tmpfs /target/mountpoint/directory
    

    That’s IT. just say that magic phrase as root, you get a 100M tmpfs file system that is nominally in memory, but bits are only used if you put something in it. Then, if filled with a lot of stuff, low use blocks can roll out to swap space. As I typically have 2 to 4 GB of swap (and sometimes up to 8); that’s a LOT of stuff you can put on very fast access space.

    At boot time, you could automate the loading of copies of things like /bin into such a space, so keep a copy of /bin somewhere like /O/bin, and while still single user, have it copied to a tmpfs partition /bin and that’s it. Now you start with the same /bin each time, and it has the advantage of high speed…

    Filesystem      1K-blocks     Used  Available Use% Mounted on
    /dev/sda6         3997376   110520    3677144   3% /lib
    /dev/sda7        12254384  1987488    9637752  18% /usr
    /dev/sda8          999320     2232     944660   1% /root
    /dev/sda9          999320     6684     940208   1% /bin
    /dev/sda10         999320    10496     936396   2% /sbin
    /dev/sda11         999320     6868     940024   1% /etc
    

    I get about 135 MB for all of /lib /root /bin /sbin and /etc combined. With something like 4 GB of swap, I could even toss in the 2 GB of /usr.

    IF you want to make a tmpfs file system automagically at boot time, you can just put an entry in /etc/fstab like:

    tmpfs   /tmp    tmpfs   size=1g     0 0
    

    UPDATE: The actual line I put in my /etc/fstab was:

    tmpfs	/tmp		tmpfs	nodev,nosuid,size=2g,mode=1777 	0 0
    

    as I needed a big /tmp, had lots of swap, and wanted to set some permissions and protections.
    END UPDATE.

    And your /tmp is automatically brought up as a tmpfs file system.

    So were I going to have that persistent /bin file always starting in a known state, I’d have a few GB of swap (just so I’d know that memory would never get full) and unpack that tarball into a tmpfs /bin at boot time.

    That way you get BOTH a known clean start state AND a damn fast system…

  3. E.M.Smith says:

    Well…

    What had lead me to bother with this stuff was working my way up to using an overlay file system on top of a squashfs file system to make a very fast system that had any changes “evaporate” on shutdown.

    This is essentially what a Live CD or Live System does. You have the compressed base system on some media (CD / USB) and it boots, making tmpfs file systems for things like logs or /tmp; then it mounts an overlay writable / file system over the squashed bits on the original media and you are up and running; with any changes written to the “overlay” in memory. It all “Goes POOF!” on a reboot and you are back to the pristine original system image.

    Essentially I’m looking to make a “roll your own live boot”. But why?

    So that when doing things “inside my own shop” I can boot regularly. Install new code, have edited files stay changed. Etc. HOWEVER, I can choose to boot as volatile and then browse the internet. Any changes, hacks, cookies, trackers, “infestations” or whatever just “go away” at the end of the session. Essentially making a system that can boot like a Live CD but not requiring a CD drive… and where I can customize the base system easily.

    Traditionally this is all done with aufs. Another Union File System:
    https://www.thegeekstuff.com/2013/05/linux-aufs/

    Linux AuFS Examples: Another Union File System Tutorial (UnionFS Implementation)
    by Lakshmanan Ganapathy on May 8, 2013

    AuFS stands for Another Union File System.

    AuFS started as an implementation of UnionFS Union File System.

    An union filesystem takes an existing filesystem and transparently overlays it on a newer filesystem. It allows files and directories of separate filesystem to co-exist under a single roof. AuFS can merge several directories and provide a single merged view of it.

    AuFS is used in many of the opensource projects like, Slax, Knoppix, and many other live CD and live USB distributions.

    On Debian based systems, for example on Ubuntu, do the following to install aufs.

    # apt-get install aufs-tools
    

    And it isn’t there by default on my Armbian system on the XU4:

    root@odroidxu4:/# man aufs
    No manual entry for aufs
    

    So I installed it:

    root@odroidxu4:/# apt-get install aufs-tools
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim
    Use 'apt-get autoremove' to remove them.
    The following NEW packages will be installed:
      aufs-tools
    0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
    Need to get 91.6 kB of archives.
    After this operation, 268 kB of additional disk space will be used.
    Get:1 http://auto.mirror.devuan.org/merged/ jessie/main aufs-tools armhf 1:3.2+20130722-1.1 [91.6 kB]
    Fetched 91.6 kB in 2s (37.5 kB/s)
    Selecting previously unselected package aufs-tools.
    (Reading database ... 95626 files and directories currently installed.)
    Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1_armhf.deb ...
    Unpacking aufs-tools (1:3.2+20130722-1.1) ...
    Processing triggers for man-db (2.7.0.2-5) ...
    Setting up aufs-tools (1:3.2+20130722-1.1) ...
    Processing triggers for libc-bin (2.19-18+deb8u10) ...
    root@odroidxu4:/# 
    

    BUT….

    Along the way I found that the Folks Who Love To DIddle have decided to change all that. They are deprecating aufs in favor of “overlayfs” – something new (to me, it’s been ‘around’ for several years).

    So once again there will be a few bumps in the road as things all get realigned.

    https://askubuntu.com/questions/109413/how-do-i-use-overlayfs

    How do I use OverlayFS?

    Asked 7 years, 5 months ago

    This answer and email message indicate that something called “OverlayFS” is available in Ubuntu 11.10 and will forcefully replace aufs in Ubuntu 12.04.

    How do I use it? Where is its documentation?

    Now I sort of knew about overlayFS and had heard about it, but mostly I’ve only used overlay file systems in Knoppix and other more “legacy” CD based things on an old PC; so it wasn’t top of my stack of awareness. Sort of “know it exists, don’t care, leave me alone”… Until now. NOW I care. That “FORCEFULLY REPLACE” is a bit, erm, less than satisfying… something more from the “everything you know is wrong” capricious change folks?

    So, whatever. It’s a 7 year old question there, so by now ought to be just ancient history. Except looking into it, it depends on kernel versions that are not all that old. So it is highly likely that I’ll have “OverlayFS” on the newer boards / systems (like the RockPro64 / Odroid N2 / Odroid XU4 ) but perhaps not on the cheap Chinese boards with poor support for new software like the Orange Pi One. IIRC it was back level a ways on the kernel.

    Once again, split between two worlds, and who knows where the R. Pi falls out.

    Part of my interest in this is working my way up to using Docker to do some of the stuff I’ve got going on the climate data so that I can just toss a “Docker Image” up and then anyone can run it (without all that system dependency unwinding step…) but Docker can have “requirements”.

    This is from 3 years ago:

    https://stackoverflow.com/questions/37110291/how-to-enable-aufs-on-debian

    When I try to install docker via:

    curl -sSL https://get.docker.com/ | sh
    

    I get the message:

    Warning: current kernel is not supported by the linux-image-extra-virtual package. We have no AUFS support. Consider installing the packages linux-image-virtual kernel and linux-image-extra-virtual for AUFS support.

    However, neither package seems to exist on Debian Jessie:

    # apt-get install linux-image-virtual linux-image-extra-virtual
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    E: Unable to locate package linux-image-virtual
    E: Unable to locate package linux-image-extra-virtual
    

    What am I missing here?
    […]

    aufs is not supported by modern kernels, so you should skip to overlayfs from aufs. Just restart your docker daemon with the option:

    --storage-driver=overlay2
    

    (or add this option to /etc/default/docker)

    In some systems you should add the processing of the file /etc/default/docker to start procedure by creating /etc/systemd/system/docker.service with content:

    etc. etc… So now we’re already up to “overlay2″… and Docker can be cranky (at least a couple of years back) if you don’t get it set right and the “fix” is in systemD service files… Um, no.

    Further down we get a statement that as of January 2017 overlay2 is not considered ready for production. One hopes it is ready now, but…

    Then:

    Actually installing a stock jessie gives you a kernel that comes with aufs support. My guess is you upgraded to a higher kernel version through jessie-backports which is not not the standard jessie setup.

    This was tested with current Debian jessie 8.7.1 amd64 and kernel 3.16.0.4.

    As I’m running Jessie on some of my systems, that’s an issue. Not a big one, just something to know. Now most of my stuff is on 4.x kernels so “not a problem”, but some older stuff… So something to be aware of. But worse is that this problem is persistent over years and was still hanging around a year ago:

    So to answer your question:

    You could re-install Debian Jessie or down-grade to the default kernel and you have a jessie with aufs support.

    For installing aufs on Debian 9 aka Debian stretch one just have to issue an

    apt-get install aufs-dkms. 
    

    This is sadly not available for jessie-backports at this time.

    At our company we are driving our docker hosts with an jessie aufs setup and everything is running flawlessly.

    Update 08-2018

    Even now I can not recommend overlay2 for production. We currently have an issue with containers where mysql is not able to create the /var/run/mysqld.sock when running in a container with the overlay2 storage.

    So the install method they give doesn’t seem right, now, on this system of mine. But I was able to install it as aufs-tools. I’m OK with that. I’ve also got to play around a bit with OverlayFS and see if it’s working well now. Then decide between them. Install, test… etc. etc.

    It sure looks like both Red Hat (originators of systemD) and Ubuntu have a high “Dick With Factor” in their systems development work; and doesn’t spend enough time thinking about what the downstream impacts on the rest of the community might be.

    Every x2 increase in commands that do the same job, or options inside some command, or dependencies that are different between commands makes a 2^2 increase in complexity and “bother” for the downstream. That was a major reason for the Unix Way of “do one SMALL thing and do it well”…

    Oh Well. Too many folks from the MS Way world moving into Linux development without having ever been seasoned in the Unix World first…

    Back from that “how to use it” link, at least OverlayFS looks easy to use:

    Example 1, overlaying the root filesystem

    I’ve been working on an Ubuntu hybrid boot disk where the base Ubuntu system exists as filesystem.squashfs and I have files called ubuntu.overlay kubuntu.overlay xubuntu.overlay and lubuntu.overlay. The .overlay files are base installs of said systems with the contents of filesystem.squashfs pruned (to save space). Then I modified the init scripts to overlay the correct distro’s .overlay file (from a boot parameter) using overlayfs and the above options and it works like a charm!

    These are the lines that I used in my init scripts (once all variables are translated):

    mkdir -p /overlay
    mount -t squashfs /cdrom/casper/ubuntu.overlay /overlay
    mount -t overlayfs -o lowerdir=/filesystem.squashfs,upperdir=/overlay overlayfs /
    

    Note that filesystem.squashfs above is a directory created by casper, not a file.

    These three statements create an /overlay directory, mount a squashfs filesystem on the /overlay directory and then use OverlayFS to essentially merge the contents of /overlay over /.
    Example 2, transparent merging of two directories

    In the process of re-building my live USB for each release, I use OverlayFS to save a bunch of time. I start off with a directory called ubuntu-base which contains the contents of the ubuntu-core image which is the most basic install. I will then create directories called ubuntu, kubuntu, lubuntu, and xubuntu.

    Then, I use OverlayFS to make the files from the ubuntu-base show up in the individual directories. I would use something like this:

    mount -t overlayfs -o lowerdir=ubuntu-base,upperdir=kubuntu overlayfs kubuntu
    

    This makes the files from ubuntu-base show up in the kubuntu folder. Then, I can chroot to the kubuntu folder and do something like apt-get install kubuntu-desktop. Any changes that are made while in this OverlayFS mount will remain in the upper directory, in this case the kubuntu folder. Then, once I unmount the OverlayFS mount the files that really exist in ubuntu-base but are “mirrored” into the kubuntu folder vanish unless they have been changed. This keeps me from having to have multiple copies of the files in ubuntu-base while still being able to use them as if they physically exist in each location.

    So there’s that.

    But I’m still left with the question of just how well Docker is working NOW on the systems I’m running, with an OverlayFS instead of AUFS.

    I am able to install AUFS still, so it is there if I need it. I guess the “option” I have is to just push on and try Docker on OverlayFS and then if it still ‘has issues” drop back to aufs.

    Until then it looks like either one ought to let me make a “volatile system overlay” as a safe browsing protection layer. (Yes, I presently do this by the expedient of just booting up a u-SD chip in a R.Pi that is NOT my Daily Driver. But sometimes it would be easier to not have to start up a different computer. Especially on portable things like an On The Road laptop. Probably still at least one Christmas away, but yes, I’m working toward an On The Road build that is highly flexible in use… so I can just keep on “doing what I do” even if not at home.)

    Sorry for the long comment, but this leaves a documentation trail for me should I drop this project for a few months and need to pick it up again, while also letting everyone else see that “Here there be dragons” even if little tiny ones…

    I’ll likely do a test case of both aufs and overlayfs and then post examples for each. But maybe not today…

  4. wyzelli says:

    Appreciate the detail. I run a couple of production Debian systems and fiddle with a few Pi devices – the most recent version of stretch doesn’t seem to work on the Pi 4 properly so we back install to jessie, but as you point out – there are always little niggles. The production stuff I use is all on vmware and I usually install a prebuilt package via iso from Turnkey Linux because they have done most of the hard stuff to make it ‘just work’.

  5. E.M.Smith says:

    Just for completion, here’s how to do it with aufs:

    https://www.thegeekstuff.com/2013/05/linux-aufs/

    Linux AuFS Examples: Another Union File System Tutorial (UnionFS Implementation)
    by Lakshmanan Ganapathy on May 8, 2013

    AuFS stands for Another Union File System.

    AuFS started as an implementation of UnionFS Union File System.

    So, yeah, there’s another version that’s older… but not going to go there…

    I just need one of the current two to work. (aufs or overlayfs).

    An union filesystem takes an existing filesystem and transparently overlays it on a newer filesystem. It allows files and directories of separate filesystem to co-exist under a single roof. AuFS can merge several directories and provide a single merged view of it.

    AuFS is used in many of the opensource projects like, Slax, Knoppix, and many other live CD and live USB distributions.

    Which is where I’ve run into it and used it the most. Largely live CDs on a PC, especially Knoppix and some Slax.

    On Debian based systems, for example on Ubuntu, do the following to install aufs.

    # apt-get install aufs-tools
    

    Which I’ve already done, but nice to see he has the correct current command for Debian / Devuan installation.

    Example 1 – Understanding How AuFS Works

    This example shows how to mount two directories of a same filesystem.

    # mkdir /tmp/dir1
    
    # mkdir /tmp/aufs-root
    
    # mount -t aufs -o br=/tmp/dir1:/home/lakshmanan none /tmp/aufs-root/
    

    The first two commands created 2 new directories. The mount.aufs is the command to mount the filesystem as Union mount.

    The mount command, specifies it is going to union mount “/tmp/dir1″ and /home/lakshmanan” under “/tmp/aufs-root”. The directory “/tmp/aufs-root” will have the content of both “/tmp/dir1” and “/home/lakshmanan”.

    The following options are used in the above mount command example:

    -o – specifies options to be passed to the filesystem
    br – specifies a branch, where each branch is separated by colon (:). A branch is nothing but a directory on a system.
    none – specifies we don’t have any device associated with it, since we are going to mount two directories

    As you see from the below ls -l command output, we can see that aufs is merging the contents of the 2 separate directories and brings a unified view.

    # ls -l /tmp/dir1/
    -rw-r--r-- 1 root       root       23 Mar 25 14:21 file_in_tmp_dir1
    
    # ls -l /home/lakshmanan
    -rw-r--r-- 1 root       root            26 Mar 25 14:20 file_in_home_dir
    
    # ls -l /tmp/aufs-root/
    -rw-r--r-- 1 root       root            26 Mar 25 14:20 file_in_home_dir
    -rw-r--r-- 1 root       root            23 Mar 25 14:21 file_in_tmp_dir1
    

    By default, if no permissions are specified, first branch will be mounted as writable, and the remaining branches will be mounted as readonly.

    So when you create any file inside ‘/tmp/aufs-root/’, physically it will be created under “/tmp/dir1”, since it is the only writable branch.

    Example 2 – Unified View of Home Directories

    At times, system administrators end up having multiple disk partitions which has home directories of multiple users. We will see an example of how we can make it as unified view and simplify the admin process.

    In this example:

    /home -> is the mount point of /dev/sda2 having “lakshmanan” and “sysadmin” users
    /home1 -> is the mount point of /dev/sdb2 having “test” user.

    # mount -t aufs -o br=/home=rw:/home1=rw -o udba=reval  none /common/
    
    # ls -l /common/
    drwxr-xr-x 39 lakshmanan lakshmanan  4096 Mar 25 15:52 lakshmanan
    drwxr-xr-x 26 sysadmin   sysadmin    4096 Mar 25 15:51 sysadmin
    drwxr-xr-x  2 root       root        4096 Mar 25 16:36 test
    

    The above mount command has an extra option called “udba”, which refers to “User’s Direct Branch Access”. This option says what to do, if the user directly access a branch and create/update files without going through AuFS.

    The following are possible values for udba:

    udba=none – With this option, the AuFS will be faster, but may show incorrect data, if the user created any files/directories without going through the AuFS.
    udba=reval – With this option, the AuFS will re-lookup the branches and update it. So any changes done on any directories within the branch, will be reflected in “/common”.
    udba=notify – With this option, the AuFS will register for inotify for all the directories in the branches. This affects the performance of AuFS.

    # touch /home/lakshmanan/TGS
    
    # ls -l /common/lakshmanan/
    .
    .
    -rw-r--r-- 1 root       root             0 Mar 25 17:17 TGS
    

    The touch command created a file named “TGS” in the home directory of “lakshmanan” without going through AuFS. Since we mounted using “udba=retval”, when we execute ls, AuFS did a re-lookup and displays the new file that was created.

    Also note that, in the previous mount command, we specified the permission for each branches as readwrite. So when a file is created under /common/test/, it will be physically created in “/home1/test/”, which is the actual physical location. Same is applicable for other directories also.

    # touch /common/test/Testing
    
    # ls -l /home1/test/
    -rw-r--r-- 1 root root  0 Mar 25 18:26 Testing
    

    Example 3 – Mount with readonly Permission for Branches

    You can also mount by having permissions set for each branches.

    # mount -t aufs -o br=/tmp/dir1=rw:/home/lakshmanan=ro -o udba=reval none /tmp/aufs-root/
    

    The above command will mount the /tmp/aufs-root, by having “/tmp/dir1” as writable, and “/home/lakshmanan” as readonly. Any change you do in /tmp/aufs-root, will be saved only under “/tmp/dir1”, since that is the only writable directory.

    The following sequence of commands can be used to verify the above.

    # cat > /home/lakshmanan/1.txt
    This is a new file
    
    # cat /tmp/aufs-root/1.txt
    This is a new file
    
    # cat >> /tmp/aufs-root/1.txt
    This is updated on AuFS space
    
    # cat /home/lakshmanan/1.txt
    This is a new file
    
    # cat /tmp/dir1/1.txt
    This is a new file
    This is updated on AuFS space
    

    In the above sequence of commands, we did the following:

    We are creating a file called 1.txt under “/home/lakshmanan”.
    This file is reflected in the union mount directory, because of the udba=retval option that we explained above
    Update the file that is present in the union mount directory
    Even tough the file is present inside /home/lakshmanan, since it is mounted as readonly, AuFS takes a copy of the file and places it on “/tmp/dir1”.
    On top of that copy, it appends the content, and saves the file
    In /home/lakshmanan/1.txt, the change is not reflected
    A new file named “1.txt”, is created under “/tmp/dir1” which has the updated content

    Example 4 – Apply Round Robin Policy for Creating files under AuFS

    I’m not going to bother with quoting the Round Robin part as I’m not planing on creating files across several file systems as a distributed workload… Hit the link if you want it.

    I’m only quoting so much of it as it stands as protection against a dead link in the future.

    Since I want to mount a read-only base version and overlay with a life (if temporary) file system to absorb any changes, I think that’s a straight forward case.

    I’m going to give it a go with aufs first, and then with overlayfs later and see if I can get both versions to work.

    One note in comments on that aufs page stated to be aware that making hard links in an aufs mounted file system can create a broken file system when you remove the overlay. That does make sense if you have a hard link in one file system to a file that really isn’t in it…

    I don’t believe that will be an issue for my use case, as the underlaying will be static and the overlay will be disposed. I may test it anyway…

  6. E.M.Smith says:

    Well, this is interesting. On the Odroid XU4 running an Armbian “uplifted” to Devuan, it doesn’t seem to have overlayfs (at least in a first trivial test):

    root@odroidxu4:/# man mount | grep overlay
    root@odroidxu4:/# man fstab | grep overlay
    
    root@odroidxu4:/# apt-get install overlayfs
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    E: Unable to locate package overlayfs
    

    and then just in case it is there and not documented:

    root@odroidxu4:/# mount -t overlayfs -o lowerdir=/JUNK,upperdir=/tmp/JUNK overlayfs /tmp/JUNK
    mount: unknown filesystem type 'overlayfs'
    

    And the system seems to not know about it:

    root@odroidxu4:/# cat /proc/filesystems | grep overlay
    

    This is in theory a 4.14 kernel:

    root@odroidxu4:/# ls /boot
    armbian_first_run.txt	   dtb-4.9.61-odroidxu4
    boot.bmp		   dtb.old
    boot-desktop.png	   initrd.img-4.14.133-odroidxu4
    boot.ini		   System.map-4.14.133-odroidxu4
    config-4.14.133-odroidxu4  uInitrd
    dtb			   uInitrd-4.14.133-odroidxu4
    dtb-4.14.133-odroidxu4	   vmlinuz-4.14.133-odroidxu4
    dtb-4.14.94-odroidxu4	   zImage
    

    so I would expect it to be there… I hope it isn’t one of those things where it is only there if you poke the right module to load… or some other quirky thing…

    One of my other pet peeves. When the answer is “Oh, just modprobe to load the module…”

    Do developers Really think Joe Sixpack or Mama Jane Sixpack are going to use a system that doesn’t work unless you know about kernel modules and modprobe? I sure hope that isn’t it. Heck, just the fact that I need to wonder if that is it shows a fundamental issue…

    So, OK, for now, on this system, I’ll be proceeding with AUFS while OverlayFS gets backburnered. I’ll ether find it is used on some other system, find that there’s some obscure way to activate it, or find out people were staying away in droves and Armbian just said “Nah, not in this release…”, or something…

  7. E.M.Smith says:

    Looks like maybe an Armbian thing:

    https://docs.armbian.com/User-Guide_Advanced-Features/

    How to freeze your filesystem?

    In certain situations it is desirable to have a virtual read-only root filesystem. This prevents any changes from occurring on the root filesystem that may alter system behavior and it allows a simple reboot to restore a system to its clean state.

    You need an ODROID XU4 or Allwinner A10, A20 or H3 board with legacy kernel where we added support for overlayfs. Works only on Ubuntu Xenial.
    Login as root and execute:

    apt-get install overlayroot
    echo 'overlayroot="tmpfs"' >> /etc/overlayroot.conf
    reboot
    

    After your system boots up it will always remain as is. If you want to make any permanent changes, you need to run:

    overlayroot-chroot
    

    Changes inside this will be preserved.

    Oh Well.

    At least this is a good example of how the 2 x options results in 2^2 ( or sometimes 2^x where x is larger…) of time suck.

    Here I’ve been fooling around for a couple of days on sorting out OverlayFS vs AUFS only to find out that on THIS system I can’t do OverlayFS… somebody somewhere made some decision…

    Now were it a KISS world, there would be “one small thing done well”, I’d have just started using AUFS and be done already.

    Oh Well, it’s only a couple of days of my life down the chute… grrr…. but at least I’m ready for when I run into some board / system where someone else decided “Only OverlayFS / deprecate aufs”…

  8. E.M.Smith says:

    And it looks like it is working…

    So I made a /tmp/JUNK and a /JUNK directories with /tmp/JUNK being a tmpfs type. Also made a mount point directory of /AUFSmnt.

    Then cobbled them all together with a mount command:

    root@odroidxu4:/# mount -t aufs -o br=/tmp/JUNK=rw:/JUNK=ro -o udba=reval none /AUFSmnt/ 
    

    In /JUNK I had a junk file

    root@odroidxu4:/# ls /JUNK
    junk
    

    As the mount command makes /JUNK read only, it ought not change and that one file ought to stay there through the other operations. Writes of a file ought to only end up in the memory based /tmp/JUNK and be visible in the union mount directory /AUFSmnt:

    then copied /etc/fstab over

    root@odroidxu4:/# df
    Filesystem      1K-blocks     Used  Available Use% Mounted on
    [...]
    tmpfs              102400       12     102388   1% /tmp/JUNK
    none               102400       12     102388   1% /AUFSmnt
    
    root@odroidxu4:/# cp /etc/fstab /AUFSmnt/
    root@odroidxu4:/# df
    Filesystem      1K-blocks     Used  Available Use% Mounted on
    [...]
    tmpfs              102400       20     102380   1% /tmp/JUNK
    none               102400       20     102380   1% /AUFSmnt
    
    
    We see that both of them show the increase of size of 12 to 20 and /tmp/junk shows as a tmpfs type file system so memory resident.  What's it look like when we do an 'ls' in each directory?
    
    root@odroidxu4:/# ls -l /tmp/JUNK /JUNK /AUFSmnt/
    /AUFSmnt/:
    total 4
    -rw-r--r-- 1 root root 3119 Aug 17 06:32 fstab
    -rw-r--r-- 1 root root    0 Aug 13 14:00 junk
    
    /JUNK:
    total 0
    -rw-r--r-- 1 root root 0 Aug 13 14:00 junk
    
    /tmp/JUNK:
    total 4
    -rw-r--r-- 1 root root 3119 Aug 17 06:32 fstab
    

    Just like it ought. /JUNK still has only the “junk” file in it. /tmp/JUNK in memory got the one copy of fstab. BOTH show up in the unified mount point of /AUFSmnt.

    So the next step is just to make my home directory a read-only file system (either as squashfs or just a regular ro disk partition mount0 and do a tmpfs overlay rw mount. At that point, sites can send me cookies, put trackers in my .cache, scribble crap in my home directory, whatever. It all goes POOF! when I power down.

    At the same time, sites that I want to have a cookie (since it is used as a ‘just log me in” marker on some sites I like0 will have one there. (It just won’t update unless I log in without the overlay).

    So that’s what I’m going to try next.

    I’ll also try stuffing /bin and maybe some others into ro format and see how much I can get done over one cup of morning coffee ;-)

    But this, at least, works.

  9. E.M.Smith says:

    Well, that was quick and fun.

    There’s still some cruft laying about as I did all this “live”, but first I made some squashfs file systems and some directories:

    root@odroidxu4:/# mkdir /tmpfs
    root@odroidxu4:/# mkdir /tmpfs/bin /tmpfs/sbin /tmpfs/lib
    

    makes the directories to mount the tmpfs file systems. Then I create / mount them with this little script:

    root@odroidxu4:/SG2/ext/sq# bcat mktfs
    mount -t tmpfs -o size=100M,mode=0755 tmpfs /tmpfs/bin
    mount -t tmpfs -o size=100M,mode=0755 tmpfs /tmpfs/sbin
    mount -t tmpfs -o size=100M,mode=0755 tmpfs /tmpfs/lib
    

    The squashfs file systems were made using my “squish” script by by hand, one each. So I’d cd into the directory then make it:

    root@odroidxu4:/sbin# cd ../lib
    root@odroidxu4:/lib# squish . /SG2/ext/sq/lib
    mksquashfs . /SG2/ext/sq/lib.sqsh
    

    I did that for all three directories: /bin /sbin /lib

    And made mount points for them:

    root@odroidxu4:/lib# cd /SG2/ext/sq
    root@odroidxu4:/SG2/ext/sq# ls
    bin.sqsh lib.sqsh sbin.sqsh
    root@odroidxu4:/SG2/ext/sq# mkdir sbin.mnt lib.mnt bin.mnt

    So just to summarize, I made a squashfs file system for each of the three and it’s all in /SG2/ext/sq directory with .sqsh qualifiers on the file names. I also made a mount point for each of them in the same directory with a .mnt qualifier on the directory name.

    Then you need to mount them all. Remember I have a script that makes this easier to type:

    root@odroidxu4:/SG2/ext/sq# bcat mtsq
    
    mount -o loop -t squashfs ${1-/SG2/ext/sq/tmp.sqsh} ${2-/mnt}
    

    So I just need the three lines in the script mtsqfs:

    mtsq /SG2/ext/sq/bin.sqsh /SG2/ext/sq/bin.mnt
    mtsq /SG2/ext/sq/sbin.sqsh /SG2/ext/sq/sbin.mnt
    mtsq /SG2/ext/sq/lib.sqsh /SG2/ext/sq/lib.mnt
    

    That mounts the 3 squashfs file systems on their three mount points as read only file systems (since squashfs is only read only…)

    Now we just need to connect the dots and make squashfs the base, put tmpfs over it, and mount them over the existing working directories:

    root@odroidxu4:/SG2/ext/sq# bcat aumnt
    mount -t aufs -o br=/tmpfs/bin=rw:/SG2/ext/sq/bin.mnt=ro -o udba=reval none /bin 
    mount -t aufs -o br=/tmpfs/sbin=rw:/SG2/ext/sq/sbin.mnt=ro -o udba=reval none /sbin 
    mount -t aufs -o br=/tmpfs/lib=rw:/SG2/ext/sq/lib.mnt=ro -o udba=reval none /lib 
    

    After that, we have:

    root@odroidxu4:/SG2/ext/sq# df
    Filesystem      1K-blocks     Used  Available Use% Mounted on
    udev                10240        0      10240   0% /dev
    tmpfs              204484      536     203948   1% /run
    /dev/mmcblk1p1   30335916  3219476   26778652  11% /
    tmpfs                5120        4       5116   1% /run/lock
    tmpfs             1247820    31704    1216116   3% /run/shm
    /dev/sda5         3997376     8444    3779220   1% /tmp
    /dev/sda3        20511356  9765932    9696848  51% /var
    none               102400       16     102384   1% /lib
    /dev/sda7        12254384  1987716    9637524  18% /usr
    /dev/sda8          999320     2232     944660   1% /root
    none               102400       12     102388   1% /bin
    none               102400       12     102388   1% /sbin
    /dev/sda11         999320     6872     940020   1% /etc
    /dev/sda12      103081248 61172628   36665740  63% /SG2/home
    /dev/sda14     1765835816  2566712 1673563120   1% /SG2/ext
    cgroup                 12        0         12   0% /sys/fs/cgroup
    tmpfs              102400       20     102380   1% /tmp/JUNK
    none               102400       20     102380   1% /AUFSmnt
    /dev/loop0           2688     2688          0 100% /SG2/ext/sq/bin.mnt
    /dev/loop1           4992     4992          0 100% /SG2/ext/sq/sbin.mnt
    /dev/loop2          33664    33664          0 100% /SG2/ext/sq/lib.mnt
    tmpfs              102400       12     102388   1% /tmpfs/bin
    tmpfs              102400       12     102388   1% /tmpfs/sbin
    tmpfs              102400       16     102384   1% /tmpfs/lib
    

    So now you can see the three AUFS mount points in production as type “none” /bin /sbin and /lib along with the squashfs mount points in /SG2/ext/sq/{FOO}.mnt and the tmpfs mount points as /tmpfs/{BAR}.

    (You can also see the /tmp/JUNK and related that I’ve not cleaned up yet but ignore them ;-)

    So right now, that’s what I’m running on, done live. In the future, the swuashfs files and mount points exist already so i’d just need to issue the commands:

    mtsqfs
    mktfs
    aumnt
    

    And all that stuff is read only under an in memory writable layer.

    I do need to add my home directory and integrate some of this into the boot process, but just “as is” it is a nice added bit of protective stuff.

    In any case, concept proven.

    Guess it’s time for that second cup of coffee ;-0

  10. E.M.Smith says:

    Interesting… some things just seem to feel a bit “snappier”. Open a bit faster, crisper. Might be a wishful thinking thing…

    CPU use is less than that taken by things like FireFox. Might be an issue on dinky hardware but the XU4 doesn’t even notice.

    I did a reboot to clear everything and test a clean restart. Also had compressed my home directory… and realized maybe I Did NOT really need to compress all 61 GB of it… as it has a lot of things like compressed linux tarball releases…

    Made another layer of “make it easy” with a trivial script named “protect”, so all I had to do was, after booting, log in as root, type “protect”, then login as me:

    chiefio@odroidxu4:~$ bcat protect
    mtsqfs
    mktfs
    aumnt
    

    I’ve added my home directory to those three scripts, so the result is:

    root@odroidxu4:/# df
    Filesystem      1K-blocks     Used  Available Use% Mounted on
    udev                10240        0      10240   0% /dev
    tmpfs              204484      536     203948   1% /run
    /dev/mmcblk1p1   30335916  3219476   26778652  11% /
    tmpfs                5120        4       5116   1% /run/lock
    tmpfs             1247820       68    1247752   1% /run/shm
    /dev/sda5         3997376     8408    3779256   1% /tmp
    /dev/sda3        20511356  9766168    9696612  51% /var
    none               102400      156     102244   1% /lib
    /dev/sda7        12254384  1987720    9637520  18% /usr
    /dev/sda8          999320     2232     944660   1% /root
    none               102400       12     102388   1% /bin
    none               102400       12     102388   1% /sbin
    /dev/sda11         999320     6872     940020   1% /etc
    /dev/sda12      103081248 61166472   36671896  63% /SG2/home
    /dev/sda14     1765835816 46145252 1629984580   3% /SG2/ext
    cgroup                 12        0         12   0% /sys/fs/cgroup
    /dev/loop0           2688     2688          0 100% /SG2/ext/sq/bin.mnt
    /dev/loop1           4992     4992          0 100% /SG2/ext/sq/sbin.mnt
    /dev/loop2          33664    33664          0 100% /SG2/ext/sq/lib.mnt
    /dev/loop3       43536000 43536000          0 100% /SG2/ext/sq/chiefio.mnt
    tmpfs              102400       12     102388   1% /tmpfs/bin
    tmpfs              102400       12     102388   1% /tmpfs/sbin
    tmpfs              102400      156     102244   1% /tmpfs/lib
    tmpfs              102400        0     102400   0% /tmpfs/chiefio
    none               102400      156     102244   1% /SG2/home/chiefio
    

    The one surprise,that ought not to have been, is that after mounting /SG2/home/chiefio as an aufs mount, the directory was owned by root and not writable by me… so I had to chown chiefio:chiefio it… I could log in, but launching x windows hung on inability to write it’s launch file…

    SO, OK, as of now this:

    a) Works very well.
    b) Is nearly trivial to launch
    c) is easy to repicate and extend to other file systems.

    As I already mount the various system file systems from real disk partitions over the u-SD card copies, to then mount a squashfs version over that is just a bit over the top / rude. So some neatening to do.

    On my ToDo list:

    1) Change the home-dir to be a cut down version without all my historical baggage in it. A fresh “just enough for browsing” version.

    2) Change the home-dir squashfs mount to mount /SG2/home and with the subdir of chiefio having proper ownership by me permissions.

    3) Sort out just which of these various sub-directories I really want to mount as squashfs only and which I want to have a writable overlay (like, in theory, /bin /sbin and /lib could all really just be squashfs direct mounts and not have any writable overlay; unless someone did something stupid and writes in them). But while /usr usually can be left non-writable, things like my /usr/local/bin would be an annoyance. Then, /var must be writable so a writable overlay for it is a Very Good Idea.

    4) After #3 is sorted, create it all and replace the present mount of USB disk partitions with those squashfs origin versions in the raw boot. (First it boots from the u-SD card and it is that /etc/fstab which controls the next step of mounting things, and so mounts the USB disk partitions over the top of the u-SD ones. I can change that to, by default, go with the squashfs versions if I like).

    5) Do the same thing on the Raspberry Pi to see if there’s any differences / gotcha / overlayfs vs aufs / whatever weirdness.

    So as it stands now, I can fairly easily make a “just let me browse without much risk” configuration via just saying “protect”, and then reboot to clear it all when done. Since many of these file systems are widely used, you can’t always unmount them once mounted. You get a “file system in use” message for things like /lib and /usr.

    When I do the R.Pi version, I’ll put that in a new posting as a start-to-end HowTo that avoids the need to wander through all this cruft to figure out the particulars…

    I’m rather pleased with this. There’s one little bit of “finesse” I’m thinking might be nice to do. As the squashfs partitions are read-only, they cause no u-SD card wear in use. It would be nice to have them on a Read Only mounted partition from the SD card, and then mount them as per the rest of the above. No USB disk needed and the source partition write protected. At some point I’ll check sizes and such and see about using gparted to make such a partition on the SD card.

    After that, most of the use of the USB disk can just go away… Especially if I put /tmp in one of these tmpfs filesystems too. Essentially, it is only SWAP where I’d like it on real disk (for preventing bit wear on the SD cards) and that becomes “any hard disk will do” as I have swap partitions on all of them

    Well, that’s a polish point for after I’ve done the itemized things…

    FWIW, with the browser open, and 28 tabs, memory use is as 499 MB out of 2 GB and no swap is in use. Even with all those tmpfs file systems defined. Why? Because nothing much is being written to them… they stay empty until some write happens then they accept the write and use it as overlay to the read-only squashfs under it.

    Overall, I’m pretty happy with how this is working.

  11. E.M.Smith says:

    Looks like aufs on the Raspberry Pi Devuan is going to be a bit cranky. I’ll have to explore OverlayFS on it and see if they have moved to that. The package is there, but complains about kernel headers:

    root@PiM3Devuan2:/# apt-get install aufs-tools
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following additional packages will be installed:
      aufs-dkms dkms linux-kbuild-4.9
    Suggested packages:
      aufs-dev python3-apport
    Recommended packages:
      linux-headers-686-pae | linux-headers-amd64 | linux-headers-generic
      | linux-headers
    The following NEW packages will be installed:
      aufs-dkms aufs-tools dkms linux-kbuild-4.9
    0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
    Need to get 820 kB/1,164 kB of archives.
    After this operation, 2,992 kB of additional disk space will be used.
    Do you want to continue? [Y/n] y
    Get:1 http://pkgmaster.devuan.org/merged ascii-security/main arm64 linux-kbuild-4.9 arm64 4.9.168-1+deb9u5 [820 kB]
    Fetched 820 kB in 1s (500 kB/s)           
    Selecting previously unselected package dkms.
    (Reading database ... 118528 files and directories currently installed.)
    Preparing to unpack .../archives/dkms_2.3-2_all.deb ...
    Unpacking dkms (2.3-2) ...
    Selecting previously unselected package linux-kbuild-4.9.
    Preparing to unpack .../linux-kbuild-4.9_4.9.168-1+deb9u5_arm64.deb ...
    Unpacking linux-kbuild-4.9 (4.9.168-1+deb9u5) ...
    Selecting previously unselected package aufs-dkms.
    Preparing to unpack .../aufs-dkms_4.9+20161219-1_arm64.deb ...
    Unpacking aufs-dkms (4.9+20161219-1) ...
    Selecting previously unselected package aufs-tools.
    Preparing to unpack .../aufs-tools_1%3a4.1+20161219-1_arm64.deb ...
    Unpacking aufs-tools (1:4.1+20161219-1) ...
    Setting up aufs-tools (1:4.1+20161219-1) ...
    Setting up dkms (2.3-2) ...
    Setting up linux-kbuild-4.9 (4.9.168-1+deb9u5) ...
    Processing triggers for libc-bin (2.24-11+deb9u4) ...
    Processing triggers for man-db (2.7.6.1-2) ...
    Setting up aufs-dkms (4.9+20161219-1) ...
    Loading new aufs-4.9+20161219 DKMS files...
    It is likely that 4.16.14-v8+ belongs to a chroot's host
    Building for 4.16.14-v8+
    Module build for kernel 4.16.14-v8+ was skipped since the
    kernel headers for this kernel does not seem to be installed.
    
    [...]
    root@PiM3Devuan2:/# mount -t aufs -o br=/tmp/JUNK:/JUNK none /tmp/Junk2
    mount: unknown filesystem type 'aufs'
    

    I doubt it will change after a reboot… but we’ll see… later.

    Did nicely add a tmpfs /tmp file system;

    tmpfs            2097152        20   2097132   1% /tmp
    
  12. E.M.Smith says:

    Well, I’m nearly done on the XU4. I’ve got it running almost entirely from squashfs file systems and some with tmpfs overlays if I think they need to be writable.

    It does seem “snappier”, which makes sense since there’s less total bits being moved, it has 8 cores so decompression not going to be noticed, and there’s no write overhead. Then a lot of stuff will be activity to memory (like launching a browser where it wants so scribble stuff back to your cache and home dir.

    Here’s what the df looks like:

    Filesystem      1K-blocks      Used  Available Use% Mounted on
    udev                10240         0      10240   0% /dev
    tmpfs              204484       552     203932   1% /run
    /dev/mmcblk1p1    8224952   3221008    4900172  40% /
    tmpfs                5120         4       5116   1% /run/lock
    tmpfs             1247820         0    1247820   0% /run/shm
    tmpfs             2097152        16    2097136   1% /tmp
    /dev/mmcblk1p2    8191416   4033352    3738636  52% /SD/squash
    none               102400       144     102256   1% /SG2/home
    cgroup                 12         0         12   0% /sys/fs/cgroup
    tmpfs              204484         0     204484   0% /run/user/0
    /dev/loop0           2688      2688          0 100% /bin
    /dev/loop1           4992      4992          0 100% /sbin
    /dev/loop2          33664     33664          0 100% /lib
    /dev/loop3            256       256          0 100% /SD/squash/root.mnt
    tmpfs               20480        24      20456   1% /tmpfs/root
    none                20480        24      20456   1% /root
    /dev/loop4         713088    713088          0 100% /usr
    /dev/loop5        2605696   2605696          0 100% /SD/squash/var.mnt
    tmpfs              204800       204     204596   1% /tmpfs/var
    none               204800       204     204596   1% /var
    /dev/loop6         650240    650240          0 100% /SD/squash/home.mnt
    tmpfs              102400       144     102256   1% /tmpfs/home
    /dev/sda14     1765835816 106248320 1569881512   7% /SG2/ext
    

    In some ways pretty ugly and could likely be made cleaner by better ordering of processes and a better selection of what to have writable.

    There’s a bunch of default from the Armbian system tmpfs bits up top (/run /run/lock /run/shm) then the /tmp that I’ve grown a lot. On the u-SD card in the mmcblk1p1 partition is a normal / partition that gets overlayed with these other copies, protecting them. Then there’s the mmcblk1p2 mounted as /SD/squash.

    I used gparted to shrink the / partition and used 8 GB of free space as a place to put the squashfs images and mount points. It is presently mounted r/w but once I’m sure I’m done changing things, it will be mounted r/o as a redundant protection. The squashfs images are r/o file systems, but the files themselves could be overwritten by a serious attack. So I’ll be doing some polishing of permissions and mount options for that space:

    root@odroidxu4:/# ls /SD/squash
    bin.mnt      etc.mnt   home.sqsh  lost+found  sbin.mnt	 usr.sqsh
    bin.sqsh     etc.sqsh  lib.mnt	  root.mnt    sbin.sqsh  var.mnt
    chiefio.mnt  home.mnt  lib.sqsh   root.sqsh   usr.mnt	 var.sqsh
    

    At this point, the chiefio.sqsh is deprecated. I’ve moved the big binaries to another space and pruned it back to much smaller, then did a mksquashfs on the whole /SG2/home directory. Now my /SG2/home/chiefio directory permissions are correct as mounted.

    There are several of these where I just mount the squashfs directly without a writable overlay.
    /bin
    /sbin
    /lib
    /usr

    Then there are the ones where I make a /tmpfs/FOO writable overlay:

    /SG2/home
    /var
    /root

    At that point, there isn’t much left of / that’s writable and can be messed with.

    It’s come through a reboot fine. I boot up using the fstab on the u_SD card, then login as root, issue the “protect” command, and login again as me.

    I am set up to overlay /etc, but presently have not just until I’m sure everything is working like I want it. Then there will be an overlay of /etc as well. I think it can just be a straight mount of squashfs… but we’ll see.

    /SG2/ext is just a big writable workspace for me to use in case I want to save things. If really worried I can just not mount it. I have another 14 GB of space on the u_SD card as mountable partitions and expect that’s what I’ll use longer term.

    So at this point it’s darned close to something I can comfortably use to browse various places and not care what kind of crap they want to shove at me. Trackers, beacons, cookies, etc.

    All that’s really left to do is make a new, dummy, user account (and a re-spin of the /SG2/home squashfs with it…) that has nothing about me included, and no legacy “crap” from my usage to date.

    I’ll be running this way for the next few weeks while I tweak and polish bits. So far, ti’s working great.

  13. E.M.Smith says:

    Interesting… After just an hour or two of “browsing”, the home directory tmpfs partition has:

    tmpfs              102400     35992      66408  36% /tmpfs/home
    

    36 MB of “stuff” in it. (those are 1 k blocks)

    That’s a LOT of cruft I don’t need stuffing up my disk drive… and it all goes POOF as soon as I reboot.

    I think next on my To Do list is that “make a pristine new account home directory” step and dump all the OLD cached crap, trackers, beacons, cookies, etc. etc….and then squashfs it ;-)

    I’d been mildly “bothered” at the fact I didn’t have a Live CD like experience for random browsing and general “whatever”. Yes, moving to a new system every so often does have some of the same effect… but you are still tagged by someone for a while (granularity depending on your rate of moving on – and if you copy your home dir forward you stay tracked…)

    This is now feeling much more like that live CD experience. I’m liking it.

    Then, that I just boot up and do NOT do the “protect” thing and get my entire account on real disk, means things like sites that use a security cookie to let you back in again still work (and without that whole “authenticate me again” process).

    So, OK, it has some complexity to it. Not as much as keeping track of 3 or 4 different computers / chip sets… And iit has all the “live CD” issues with loosing everything when you are done (so save things to a writable real store…). Updates are also going to be more complicated. (So need to boot regularly, do the apt-get stuff, re-squash, etc.)

    But still. It’s nice to just do a df and realize nearly nothing is writable on this system right now.

Anything to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.