From the “Well that was fun, NOT!” department…
I’ve had a ‘site scrape’ on GHCN, NOAA/NCDC, GIStemp of various degrees for ‘a while’. The last increment downloaded a huge chunk of ‘superghcnd’ data and essentially ‘filled my disk’. So I added some dribs and drabs of other disks as a stop gap. A 2 TB physically small WD and a 1.5 TB Seagate that is in that “awkward stage” between full TB increments. That, then, filled to about 6 TB out of a 6.82 TB formatted capacity.
Thanks to generous donations (with a special h/t to P.G. Sharrow for a very generous donation) I’ve added another 4 TB (gross size, 3.64 TB formatted) disk.
So I decided to remove the 1.5 TB disk ( 1.36 TB formatted and likely only about 900 MB of it actually in use at the moment as it was added last so all the free space ought to be on it).
No Problem, I think. It is all built on LVM and that’s the whole point of LVM. Letting you add and remove physical volumes without disturbing the logical structure.
But why do this at all?
Well, the Orange Pi One I’ve been using as a scraping engine has one USB port, so you plug in a hub to get more. I have 2 hubs. A 4 port and a 7 port. The 7 port is used on my primary machine as it typically runs with mouse, keyboard, and a few disks. So with 3 disks already on the Orange Pi, adding a disk fills all the ports. This means no more disks AND that it can’t be run ‘headful’ with keyboard, mouse, and monitor. It’s better to keep a port or two open so at least you can boot with keyboard and monitor for command line recovery purposes.
Well, I plugged the new disk into the 4 port hub and added that disk. I did a shutdown / reboot prior to doing the data move, just to assure all things were fresh at boot time. It would not let me open a new SSH session after the reboot / power up. Well, something is not right… Skipping a lot, it seems I had left a ‘mount this disk’ in the /etc/fstab file from something months ago during the initial set-up and at boot time, no longer having that disk, it fails to finish the boot. Fixed that, and rebooted. It came up fine.
By then I’d also moved the mouse, keyboard, monitor, large USB hub etc. over to the Orange Pi One so as to make maintenance quicker and avoid those “ssh denied” problems. Added the disk to the LVM and all is good, right? I used gparted to format the disk as mostly LVM with a 1 GB linux swap instead of using pvcreate as in the docs, and then did a ‘vgextend’. Shows 10 TB of disk and ready to migrate off the old disk.
Well, at that point one uses pvmove in a nearly trivial command. My old disk partition was named /dev/sdd1 so the command is “pvmove /dev/sdd1”.
I got the error message that the kernel module dm-mirror was not installed. What the?…
Seems that on Armbian, when you install LVM, it does not install all the dependencies. In particular the kernel module used by pvmove. That’s the kind of stuff you find in “young ports”. Over time, as folks run into those bugs, report them, and future builds are fixed, they “go away”. That doesn’t help me, now, though.
OK, I can either get into the land of rebuilding kernels and modules and / or system updating and / or all the attendant QA et all, or I can “move on”. I chose to “move on”. Simply export the whole LVM, then import it to the Raspberry Pi on Devuan. Debian is a MUCH more robust and older project so that kind of stuff is pretty much stamped out. Devuan is a special purpose build on top of Debian, so benefits from all that.
The move was flawless, and the file system came up nicely. Now the Orange Pi is free for other tasks, or I can later move the LVM back if desired. Time for that pvmove to empty the odd disk (and make room for larger disks later…).
I launched pvmove and it complained that there were no extents to allocate. What the? I just added 4 TB of extents! But pvdisplay said they were all spoken for. After some degree of head scratching, I found that I needed to do a lvreduce command to release some of the extents. Since I’d not expanded the actual file system built on top of the logical volume, I didn’t need to shrink that first. It was already at 6.82 TB. So I did an lvreduce to about 7.5 TB (leaving lots of room beyond the end of the file system, while also having plenty of room to move the 1.5 TB disk). Then the pvmove worked. Or rather “is working”.
The docs warn that “pvmove is slow”. That is a gross understatement. It Oh So Slowly makes a copy, the whole time not just moving data blocks but file system structures as well, then eventually releases the extents on the old disk when the new copy is ready to go. I launched it last night at about 10 PM? something like that. Today, after lunch, I’m at 68% moved… I figure it will be done at about the 24 hour mark.
Then I can do a vgreduce and eventually physically remove the disk from the system. All told, it will be about 48 hours from start to finish to add the 4 TB disk and remove the 1.5 TB disk. “someday” I get to do this all over again for the 2 TB disk when it gets replaced by something bigger… or maybe I’ll just go buy another 7 port USB hub and forget about it…
So that’s the kind of adventure you end up in, when using data center production techniques like LVM on a $16 brand new card with an operating version (Armbian) aimed mostly at small internet appliances.
For now, I’m planning to leave the Orange Pi One powered off. Eventually I’ll put a newer copy of Armbian on it (i.e. do an update cycle) and make sure it has dm-mirror installed. Until then, running the LVM stack on the Raspberry Pi is fine. When I get it slammed with a model run, I’ll want to move that scrape process somewhere else, but for now it’s fine. This also will let me explore some kind of case for the Orange Pi (that is currently loose on the desktop as it doesn’t fit Pi cases) and make a more lead dressed and clean system out of the present “pile of parts” on the desktop. With luck, a re-scrape of cdiac will not take more than the 2 x 4 TB big disks and I can recover that 2 TB Western Digital for other things. IFF it ends up spilling out to the whole stack, well, I’ll add some big disk and ‘move on’.
What I’d like to do is get the CDIAC files on the 2 TB disk and leave NOAA on the LVM group as it is big enough to need it. That way they can be moved between systems separately and without all the fuss involved in moving an LVM group. But we’ll see as the process sorts out and the final scrape is done. I think I’m not going to re-scrape NOAA, just because I have a clean scrape of it and the whole point is to avoid loss if the site gets ‘reduced’, so I do NOT want to stay in sync with it. But that is for the future… in a day or two when moving to the new disk finally, oh so slowly, completes…