Pi Cluster Parallel Script First Fire

Today was spent “slaving over a hot computer”. Part of that was getting GNU parallel to work with a couple of commands.

I first mentioned GNU parallel command here:

https://chiefio.wordpress.com/2017/11/10/parallel-scripts-clusters-easy-use/

That page includes some on how to install it and a video on basic operation. Today I decided to make a couple of scripts that would automatically spread work around to the cluster instead of me needing to type arcane command lines. I started by reviewing the tutorial and getting a basic example onto my desktop. That was fairly easy.

I chose to have a naming convention that any command I added to my $HOME/bin would have p_ as the first letters if it was a parallel command that just used my local cores, and ps_ if it was a parallel across systems command spreading things over the whole cluster of systems. In this way, I have to know a command is going to use parallel when I’m typing the name.

The example in the video used gzip and gunzip. Since I’ll often need to gzip or gunzip whole directories full of stuff, I thought I’d start with a p_gzip and a p_gunzip. They worked relatively well and somewhat easily.

Here’s the p_gzip text:

parallel -j0 --eta --progress gzip {}

It can be used by doing an ls to list the files you want zipped and send their names into standard-in like:

ls junk* | p_gzip 

The command says to do things in parallel, the -j0 says use all cores available (in this case 4 on the Pi M3) so doing 4 gzips at a time, give me status on progress and eta to complete if it takes long, and run the gzip command on each name that comes in standard in.

When I first ran it, I thought I’d failed. I was staring at my monitor screen and saw no increase of activity. I expected to see the 4 cores load up. (That’s when I added some of the status information). Doing an “ls” command showed all the files were gziped. What? Yeah, it finished fast. I’m working in a directory where I’d copied all my log files. Guess it wasn’t enough load.

chiefio@DevuanPiM2:~/pp$ ls | wc -l
64
chiefio@DevuanPiM2:~/pp$ du -ks .
572

64 files, compressed size 572 kB.

I then ran it with “time” to see just how long it was taking:

chiefio@DevuanPiM2:~/pp$ time ls * | p_gzip

Computers / CPU cores / Max jobs to run
1:local / 4 / 64

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s 0left 0.00avg  local:0/64/100%/0.0s 

real	0m1.671s
user	0m1.510s
sys	0m0.590s

Oh. 1 2/3 seconds. No wonder I didn’t see a blip on the load… Notice the top status report. 1 computer, the local one (i.e. my desktop) with 4 cores running 64 jobs. As all of them total only took 1.6 seconds, each individual job was too little to show up on the htop monitor. I need bigger files to compress ;-)

Clearly from this example it is Real Easy to use parallel to load up all the cores on your local computer for the various tasks that are annoying to systems admin types, but need doing.

Here’s the equivalent parallel gunzip:

ls *.gz | parallel -j0 --eta --progress gunzip 

For this one I chose to put the “ls” inside the script itself. Just gunzip anything that has a .gz suffix in a given directory. How long did it take to unzip all those 64 files?

chiefio@DevuanPiM2:~/pp$ time p_gunzip

Computers / CPU cores / Max jobs to run
1:local / 4 / 64

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s 0left 0.00avg  local:0/64/100%/0.0s 

real	0m1.735s
user	0m1.210s
sys	0m0.770s

About the same. Notice the “ETA:” line. 0 seconds to complete, 0 left to do. By the time it got set up for the first ETA report, it was done. 64 jobs and 100% done on the local machine.

Clearly I needed something that would load up a core so I could see it actually doing something. I have such a “do nothing” load script, named load.

while true
do
ANS="foo"
done

All it does is repeatedly stuff “foo” into the variable ANS forever in a loop. Locks up a CPU Core at 100% very nicely.

chiefio@DevuanPiM2:~/pp$ bcat p_load
ls ${1-d*} | parallel -j0 --eta load 

This, run in my parallel program testing directory, is doing an ls of all the files starting with a ‘d’ (that’s about 20 of them with names like daemon.log.1) as a cheap hack to let me vary the number of invocations of the “load” script. Works a champ, and here’s the display it gives when run:

chiefio@DevuanPiM2:~/pp$ p_load

Computers / CPU cores / Max jobs to run
1:local / 4 / 20

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete

Note that it never completes and so can’t really estimate when it will complete. A dirty trick to play on –eta ;-)

1 computer, the local one, with 4 cores in use, and 20 instances of “load” pegging it at 100% loaded.

Great! I thought, now lets just bring in the rest of the cluster.

It Is Much More Complicated To Cluster Parallel

While not all that hard for an old hand at Linux / Unix, it is, IMHO, more complicated than it needs to be for simple things (while letting you do all sorts of complicated things no doubt…)

When I ran my first test case I got something like 64 prompts for my login / password. Ooops.

First off, the underlying transport to the distributed cluster nodes is SSH so you must have it installed AND have a shared key set to let you log in without password prompting. Oh Joy. I got to go through the generate a key pair and copy it to each machine in the cluster exercise. Gee, that would be a great thing for a parallel automated task… oh, right…

Here’s a helpful page on how to do that:

https://www.ostechnix.com/configure-passwordless-ssh-login-linux/

It has you install openssh. Debian / Devuan already has ssh for me so I just skipped that step and dropped down to:

Generate SSH keypair on local system

ssh-keygen creates a keypair, private and public keys. The private key should be kept secret. You shouldn’t disclose it to anyone else. And, the public key should be shared with the remote systems that you want to access via ssh.

Important note: Do not generate keypair as root, as only root would be able to use those keys. Create key pairs as normal user.

Run the following command in local system (Ubuntu in my case) to generate the keypair.

ssh-keygen

So on my “headend” machine, logged in as “me” (chiefio) I did that ssh-keygen command. Easy.

THEN, I got to do this next step 5 times for the 5 other computers presently in the cluster… Easy but tedious.

Now, we have created the key pair in the client system. Now, copy the SSH public key to the remote system that you want to access via SSH.

In my case, I want to access my remote system which is running with CentOS. So, I ran the following command to copy my public key to the remote system:

ssh-copy-id ostechnix@192.168.43.150

As my login was “chiefio” and my machines are in a 10.x.x.x network, I got to do something more like:

ssh-copy-id chiefio@10.168.168.141
ssh-copy-id chiefio@10.168.168.142
ssh-copy-id chiefio@10.168.168.143
ssh-copy-id chiefio@10.168.168.144
ssh-copy-id chiefio@10.168.168.145

All from the headend machine.

I could likely have put that in a script-lette but it was a one-off so I just did it long hand. I could likely have also used the hostnames, but just followed the model shamelessly out of sloth.

That, then, let me do a password free login from headend to all the headless nodes over SSH. Oh Joy. This time for sure!

Well, I launched the parallel system version of my load command and it showed work blips show up on the various machines as they prepared to be infinitely loaded forever, and then it was done. WT?… There were also a fair number of error messages. First try was to just put a copy of “load” in my home directory/bin on each machine. Figuring if it logs in as me, it will be in the search path. Nope. Still telling me “cannot find command load” and just ending.

Well, cutting to the end, it’s got ‘fiddly bits’ that need a good fiddle. Probably to protect you from stray programs of the same name but different content, or perhaps to make it ‘easier’ so you never have to install things on the headless nodes, the parallel command has its own way of distributing the program / script to be run. I also got some nags about SSH wanting to have the max connections made larger (that I’ve not done yet) and some about Python not being happy at “locale” being unset (so it uses the default that I want anyway) and some more housekeeping clean up to do. But it DOES run and it DOES work.

The “feel the other systems and set up to distribute work” seems to take a few seconds, so anything needing less than 30 seconds to run on your local box probably will not benefit from a cluster command. Use this only for things that take long enough locally to bother you enough to write and debug a command / script.

Here’s the final result:

ls ${1-d*} | parallel -j0 --eta -S:,h1,h2,h3,h4,h5 --basefile /home/chiefio/bin/load '/home/chiefio/bin/load' 

Again, I’m feeding it a set of bogus input just to stimulate a number of jobs being farmed out. This goes into the pipe into parallel with -j0 for jobs = cores on any given computer. The –eta to give us a useless eta on something that will never end, then the interesting bit. The -S lists the systems to run this command upon. The “:” is special and means “localhost” (in this case ‘headend’ machine) then a comma separated list of other computers. For me, headless1 has an alias of h1, and headless2 an alias of h2, etc. So I’ve listed my 5 headless nodes.

Now, the “trick”, is you must tell it where your “script” or command is located. That –basefile that points at my bin directory and the command named “load”. That –basefile option can also share data files around too. Next, in single quotes, you put the actual command and arguments to execute. As load takes no arguments, it’s the same as the –basefile listing, but in single quotes. I’ve not yet gotten into the whole argument passing thing. That will be for tomorrow and making a parallel systems version of gzip and gunzip.

For now, this proves I’ve got the cluster running parallel scripts and paves the way for me to do things like a set of parallel FORTRAN compiles. While there is a dedicated commend for parallel C compiles (“distcc”) I’ve not found an equal for FORTRAN. I have distcc on each node of the cluster, so parallel C complies are ready to go (but having different libraries might be an issue – identical systems, releases, and libraries are best). Hopefully FORTRAN will be a bit forgiving on that front. We’ll see (eventually).

To the extent that parallel FORTRAN compiles require the same library and OS images, I can just use the 4 boards ( 16 cores) that are all Devuan matched R.Pi systems Leaving the Odroid C1 and Orange Pi One out of it. At least until a properly matched Devuan is available for them. Just changing the -S system list is all it takes.

Oh, here’s the result of running my ps_load command and loading up all 6 systems. A couple only had 3 cores in use, but that’s because I had more cores than files starting with the letter d ;-)

chiefio@DevuanPiM2:~/pp$ ps_load
parallel: Warning: ssh to h2 only allows for 15 simultaneous logins.
You may raise this by changing /etc/ssh/sshd_config:MaxStartup on h2.
Using only 14 connections to avoid race conditions.
parallel: Warning: ssh to h4 only allows for 17 simultaneous logins.
You may raise this by changing /etc/ssh/sshd_config:MaxStartup on h4.
Using only 16 connections to avoid race conditions.
parallel: Warning: ssh to h3 only allows for 14 simultaneous logins.
You may raise this by changing /etc/ssh/sshd_config:MaxStartup on h3.
Using only 13 connections to avoid race conditions.
parallel: Warning: ssh to h1 only allows for 17 simultaneous logins.
You may raise this by changing /etc/ssh/sshd_config:MaxStartup on h1.
Using only 16 connections to avoid race conditions.
parallel: Warning: ssh to h5 only allows for 16 simultaneous logins.
You may raise this by changing /etc/ssh/sshd_config:MaxStartup on h5.
Using only 15 connections to avoid race conditions.

So I need to go to each of those headless systems and raise the SSH logins limit, or set parallel to ask for fewer. It still runs, just nags you.

Warning: Permanently added 'h4,10.168.168.144' (ECDSA) to the list of known hosts.
Warning: Permanently added 'h1,10.168.168.141' (ED25519) to the list of known hosts.
Warning: Permanently added 'h5,10.168.168.145' (ECDSA) to the list of known hosts.
Warning: Permanently added 'h2,10.168.168.142' (ED25519) to the list of known hosts.
Warning: Permanently added 'h3,10.168.168.143' (ECDSA) to the list of known hosts.
Warning: Permanently added 'h1,10.168.168.141' (ED25519) to the list of known hosts.

Since I’m frequently changing host boards, I set the file where this “permanent” list is kept to be a link to /dev/null, so it nags me about this each time, but I don’t have to remove old identities to let new ones be added. IF the cluster stabilizes, I’ll put it back to normal and these will go away.

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Warning: Permanently added 'h2,10.16.16.42' (ED25519) to the list of known hosts.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

So Perl on a couple of the boards needs to have ‘locale’ fixed. Maybe someday… I did it on a couple of others. Really a stupid Perl behaviour.

Warning: Permanently added 'h3,10.168.168.143' (ECDSA) to the list of known hosts.
Warning: Permanently added 'h4,10.168.168.144' (ECDSA) to the list of known hosts.
Warning: Permanently added 'h5,10.168.168.145' (ECDSA) to the list of known hosts.

Computers / CPU cores / Max jobs to run
1:local / 4 / 20
2:h1 / 4 / 16
3:h2 / 4 / 14
4:h3 / 4 / 13
5:h4 / 4 / 16
6:h5 / 5 / 15

Computer:jobs running/jobs completed/%of started jobs

And there you can see the 6 computers, each with 4 cores, and what max number of jobs each can get. I don’t know why h5 claims to have 5 cores. It’s an Orange Pi One, with Armbian, so maybe it’s quirky… Ran fine in any case, just might get assigned more work units than it ought to get.

In Conclusion

The way distributing scripts / commands is handled is a bit obtuse. Passing parameters and file names is a PITA (partly as they have 3 or 4 ways you can do it to try pleasing everyone and that just makes it 3 or 4 times as much to learn and 9 to 16 times as many disconnects between the way you read and what the example shows… Oh Well.)

The good bit is I now have model parallel scripted commands for both machine local and cluster distributed execution, and they work. Now I can just, one thing I do at a time, model off them to make a local parallel or cluster parallel script to distribute the work, as appropriate. I’ll need to learn some more options as I do that, but those will be “as needed / where needed”.

Over time, as I run into things that are slow, I can spend time while they run making a parallel scripted version of them. For now, gzip, gunzip, xz, unxz, perhaps some data move / preen tasks, and making FORTRAN modules in a distributed way. Certainly testing them on different data sets. I also need to integrate a Real Disk NFS data store for cluster shared data. That can cut down on some of the headend communicating data copies to the headless nodes, but at the cost of an NFS server bottleneck. I also need to find the powersupply for my 100 Mb network switch, since runnig the cluster through the old crusty 10 Mb hub it is on now will certainly make “network” the hang-up point.

But now that I’ve got things for it to actually DO, I’ll be motivated to fix those things as they give me a bit of a pinch ;-)

The other thing I’ll end up doing is building more cluster control / management tools. Doing a log-in to 6 systems every time you want to shut it down will get old quick ;-) Not bad when playing with 2 to 4 boards at a time, but a PITA as a normal operations process.

With that, I’m signing off for the evening. I’ve got a cluster that works. I can distribute just about any Linux command or script to 24 cores as it stands and another 12 if I bother to configure them too. Now it’s just putting it to work. One script and one task at a time…

Subscribe to feed