distcc – Success! That was fun.

I know, not too many folks will care, this being a ‘computer geek’ thing, but for me it was a moment of success that will save me many hours “going forward”.

As y’all know, I’m presently using Arch Linux as my “daily driver” on the Pi Model 2 since it is fairly fast, and despite it being afflicted with SystemD. But I really want something not so afflicted, and faster would be nicer.

That honor falls to Gentoo Linux. It is small, fast, and a bit techy oriented. The Gentoo Penguin is the fastest penguin on the planet, and Gentoo Linux styles itself efficient, fast, and effective. It also has decided to have BOTH a SystemD option and an OpenRC option. “Why” is interesting too. Gentoo is used on many small embedded devices where a complicated ‘init’ (initialization) system would just be a royal PITA, so many users want OpenRC. It also is used by other folks who don’t want a lot of work doing things like porting the Gnome Desktop and it’s huge swath of tools, that are now firmly tied to SystemD.

So they let you choose.

Now one of the peculiar things about Gentoo is they are really over the top about security. I like that. As do all the folks making embedded systems for things like routers and such. It also cares a great deal about small size and efficient operation (since there may not be much processor at all and nearly no memory in that $50 router box you buy, that wholesales for $25 and is $10 to the manufacturer…). To that end, they also can compile with a smaller more efficient set of C libraries called uClib. Ideal for the R.Pi as they are small, fast, and efficient. (Though likely not ported yet and a large body of work to do… someday…)

BUT, all this comes at the price of “You must compile it yourself”. Now doing that is sometimes as simple as typing “make” in the right directory (if everything is configured right and works right). But on a box as small as the Pi, it can take a few days…

Thus the desire for a faster, multi core, multi machine, distributed compiler farm.

(Yes, many folks just do a cross compile from a large beefy PC / Intel chip box instead, but then the compiler can’t do quite as many optimized choices for exactly your chipset and hardware… While that may not matter much for most things, for something that is just marginally comfortable to use, it can make a lot of difference).

So I was looking for a way to do faster compiles, in a native Raspberry Pi environment. It might only cut the 24 hours down to 4, but that is enough. Or I could pop for a couple of more cards and make it twice again as fast, if desired. Nice, eh? ;-)

But…

But before I can even ‘go there’, I need to learn how to install and use distcc, configure it, and make it go; showing that it will actually work on the hardware I have with the operating systems I have.

One good thing, is that it already expects to work on ‘variety hardware and OS’ types, so being on, say, Arch doing a compile of Gentoo is easy for distcc.

Still, I needed to do the work. Any time you are learning new tools and installing them, it’s work.

Today I did that.

Sweet Success

I just launched a “make” and had all 8 cores of my two Raspberry Pi Model 2 boards cooking. It was a dramatic speed up in rate of compilation, clearly visible. Normal complies will often only use one core, even on a multi-core machine. Sometimes you can add some directives to use more of them, but it can be a PITA to do all that. The distcc code makes it easy.

My test case was a simple “make” plodded along, one core, one cc at a time.

My distributed “make -j8” where the -j8 says “start 8 complies at once” went much faster. I’d guess about 5 or 6 times as fast. Due to overlap of I/O and such, the optimal number of jobs is usually about 2 x the number of cores, so I ought to do it with -j16 set, but I just wanted it to work first. Optimize later…

How? What does it take?

This is where it gets a bit hairy.

In an ideal world, you would just search for “Install and configure distcc” and do what the web page says. The world is now far from ideal. Thanks to the explosion of flavors of Unix, dsitros of Linux, and versions over time, seasoned with the “Everything you KNEW is now WRONG!” mandate of SystemD for how to configure things, there is now an N x M x Y x Z set of possible “ways to do it” (or worse…).

Then some parts may not actually work, so you need a particular work around for THAT system. Plus programmers like adding all sorts of configuration options, so you get to read through 100 of them to find the one that you need to set to “fix it”… etc. etc.

The bottom line is that the online pages are nice pointers, but there’s a considerable “dick with factor” before it actually works. Mine now works. So here’s some of the specific bits I learned along the way, that may well be different next year on a new release or a different context.

First off, the page with pointers and clue about doing it as a cross compiler from a PC. More complicated than I wanted as I’d rather NOT fire up the PCs, but some folks may care:

http://jeremy-nicola.info/portfolio-item/cross-compilation-distributed-compilation-for-the-raspberry-pi/

The page is Raspberry Pi oriented, but not Arch Linux (Debian instead) and I think non-SystemD for the service starting commands. (Either that, or it IS SystemD and my box isn’t… my target server box that is… or mayube it is just a bug in starting the daemon… but more on that below). It is also cross compiler and I’m not. But the general idea is presented well.

Introduction

Compiling large programs such as a Linux Kernel, or big libraries like OpenCV, OpenNI directly on your Raspberry Pi will take a lot of time, and sometimes will even fail ( I was not able to reach more than 6% in the compilation process of the PCL, when compiling directly on the Pi ).

In this howto I will show you how to:

1) cross compile programs, i.e.: how you can compile a program on your PC so that it will run on your Raspberry Pi

2) distribute the compilation so that when you compile a program from your Raspberry Pi it actually gets cross-compiled on your remote PC(s), in a totally transparent manner

I diverged right out the gate in that I’ve got the Model 2, so the OS is a bit different, as is the speed, and I’ve got 4 cores, and I’m running Arch, and I’m not cross compiling from the PC. But about that Arch…

My “workstation” is Arch, but for the “server” I just grabbed an existing chip and stuck it in. Debian Jessie. I think it is non-SystemD. Bottom line, though, is that to install distcc on the Arch box I had to do a:

pacman -S distcc

while for the Debian I had to do:

apt-get install distcc

I also found that the Arch Linux didn’t have ‘git’ installed, and doing a

pacman -S git

tries, but fails on a ‘404 error’ with the package not being available. Good thing I didn’t need to install the toolchain as they say to do… much already being there from my install script. Though I did find that “make” was missing, so a pacman -S make was needed.

For me, the best bits were the pointers about environment variables, why you set some things, and starting the service. Which near as I can tell didn’t start on my server… So I did it long hand.

But first I started at the point where he complies an “Hello World” program. By convention, Linux / Unix folks usually write as their first program in any language a simple one that just says “Hello World” and ends. His text was:

#include 
int main(void)
{
    printf("Hello, cross-compilation world !\n");
    return 0;
}

I initially complied it just with cc the C Compiler. It puts out an executable named a.out and you can type a.out like a command and it runs and says hello. It also gives you a date stamped a.out file that you can look at, then if the time stamp doesn’t change, you know your latest compile attempt didn’t happen. I had that a few times before I got things to work.

They use a ‘trick’ of symbolic links and your Search Path to trick the use of ‘cc’ to point at ‘distcc’, but before I did that, I just tried “distcc hello.c”. The a.out time stamp changed, but it still ran, so I knew distcc would actually compile something. Then I did the links.

They have you put them in a fairly long path name. I chose to make it simple. At the root level, I made a directory named /distcc and put them there.

pi@raspberrypi ~ $ which gcc # This tells you where gcc lies
/usr/bin/gcc
pi@raspberrypi ~ $ which distcc # This tells you where distcc has been installed
/usr/local/bin/distcc
pi@raspberrypi ~ $ sudo ln -s /usr/local/bin/distcc /usr/local/bin/gcc
pi@raspberrypi ~ $ sudo ln -s /usr/local/bin/distcc /usr/local/bin/cc
pi@raspberrypi ~ $ sudo ln -s /usr/local/bin/distcc /usr/local/bin/g++
pi@raspberrypi ~ $ sudo ln -s /usr/local/bin/distcc /usr/local/bin/c++
pi@raspberrypi ~ $ sudo ln -s /usr/local/bin/distcc /usr/local/bin/cpp
export PATH=/usr/local/bin:$PATH
pi@raspberrypi ~ $ which gcc # Now, when gcc is called, distcc is called instead
/usr/local/bin/gcc

Instead, I just “became root” via (on the Arch: su while on the Debian: sudo bash) and did:

mkdir /distcc
cd /distcc
ln -s /usr/bin/distcc gcc
ln -s /usr/bin/distcc cc
ln -s /usr/bin/distcc g++
ln -s /usr/bin/distcc c++

I forgot to do the cpp C PreProcessor, so might need to go back to do that one, or just let the preprocessing not be distributed ;-)

Note that the path is differernt at “/usr/bin” instead of “/usr/local/bin”. That’s the kind of thing you run into with different releases and flavors… On Debian Jessie it was also /usr/bin so I’m not sure when it changed from what he did. Doing a “which distcc” gives you your path.

I did that mkdir /distcc and the ln -s set on both systems.

Now he has a set of his search path (where Linux finds a command, searching many directories) in an ‘export’ command. This was supposed to be put in your ~homedirecory/.bashrc file so it is always there when you launch a shell. I put it there, but it didn’t work. I suspect that an lxterm window may not be running a real bash shell? Either that, or not running the .bashrc file.

But this was another of those “Oh, need to work around that” moments. So before running a program, by hand, I’d type:

export PATH=/distcc;$PATH

That takes our old path variable value ( $PATH ) and glues /distcc on the front of it, then sticks it back in your PATH variable. That way when you say “cc hello.c”, it looks for ‘cc’ first in /distcc where it finds that symbolic link, and uses distcc instead.

I also needed to similarly explicitly set the machine list to use for compiles.

At the bottom of my .bashrc file I added:

#added by EMS 6May2016

export PATH=/distcc:$PATH
export DISTCC_HOSTS="10.16.16.253/24"
export DISTCC_TO_TIMEOUT=3000
#
# To prevent local try of compile, uncomment this line:
#
#export DISTCC_SKIP_LOCAL_RETRY=1

Yet when a new terminal is opened:

sh-4.3$ echo $DISTCC_HOSTS

sh-4.3$ 

It is empty. So for now, I type it long hand in the new window and it is there for that session. A “dig here” for later is how to fix that.

At first I had an error message that it could not distribute jobs. Then I found the variable not set by the script. Then I put in 10.116.16.253 by hand (export DISTCC_HOSTS=”10.16.16.253″ ) that then let me discover the server was NOT running the distccd deamon. More on that below, but I got it started…

What I then found was that it was only running one job. So initially I only had 10.16.16.253 and it would only run one process and only on the ‘server’ box… a nice trick, but not what I wanted.

Trying to fix THAT, I set the DISTCC_HOSTS variable to:

sh-4.3$ echo $DISTCC_HOSTS
localhost/4 10.16.16.253/8
sh-4.3$ 

That says to use the localhost, your box, for up to 4 processes, and the box at 10.16.16.253 for up to 8 processes.

“This Time For SURE!!!”…

And I got one process running only on the workstation (Localhost, the Arch box).

I had “missed a trick” in that “make” takes an option. You need to tell it how many jobs.

Doing “make -j8” and I had a nice set of 8 compiles running on 8 cores on two boxes.

But Wait, There’s More!

I’ve skipped over a couple of ‘gotchas”. The man page says to start the service with

service distcc start

That didn’t seem to work on Debian.

]      If  you  installed  distcc  using a packaged version you may be able to
       start the server using the standard mechanism for your  operating  sys���
       tem, such as

              # service distcc start

       To  start  distccd  as  a  standalone  service, run a command like this
       either as root or an ordinary user:

              # distccd --daemon

I wonder at it having ‘distcc’ for the service, but distccd as the daemon, but who knows…

Instead I used the distccd –daemon form. And promptly got an error message. It seems that the ‘-a’ or ‘–allow’ option is now mandatory. Back to the man page…

  -a, --allow IPADDR[/MASK]
              Instructs  distccd  to  accept  connections  from the IP address
              IPADDR.  A CIDR mask length can be supplied optionally  after  a
              trailing  slash,  e.g.  192.168.0.0/24,  in which case addresses
              that match in the most significant MASK bits  will  be  allowed.
              If  no  --allow options are specified, distccd will exit immedi���
              ately!  Unauthorized connections are rejected by closing the TCP
              connection  immediately.   A warning is logged on the server but
              nothing is sent to the client.

so doing: distccd -a 10.16.16.0/24 –daemon

as root, finally got the service running. (I suspect it would not start as a service due to no ‘allow’ list, wherever one puts it for a service…) That was the server side.

So loose ends there are to clean up or capture the start up process and environment variable stuff so I don’t need to do this long hand each time. On the client side, also fix the environment variable handling and maybe set it up to be daemon / server too (so things can distribute any to any).

There’s one last hitch…

When it came time to compile “something big”, his example has you grab from the git repository. But on Arch, git wasn’t installed. Attempting to install git failed on git not being in the Arch repository (so is it just not there, or do I need to expand my list of sites to search?…) I decided to just make a git tarball my target for compiles instead.

https://www.kernel.org/pub/software/scm/git/

has the tarballs. I did the download with the browser (SeaMonkey), remembered that the .xz ending was Yet Another Compression Method (that doesn’t uncomprss with the usual old tools…) and did an unxz on it, then made a ./git directory for it, moved it in, untarred it with ‘tar xvf git-2.8.2….” and was then ready to type ‘make’.

At that point the above story of getting the service to start, finding out about ENV variables unset, learning to use -j8 etc. actually unfolded.

In Conclusion?

So is that all of it? I hope so.

This is just a postmortem stream of consciousness posting of what I remember of the process.

Often ‘systems work’ is like this. Wandering lost in the desert for days, at the end, trying to scrawl down a map of where you ought to have gone. Noting the traps, you hope in the right places. Leaving bread crumbs for your future self on the next trip past.

Over time, more of this will be scripted, and the odd bits ironed out or work arounds put in scripts.

For me the brightest bit is just that I got ALL of git to compile in one go. It is fairly big and obscure code, but no doubt well written. Watching the two screens with system monitoring running in them finally “light up” with 8 cores in use was a treat. ( I skipped all the part of the article where he’s installing a bunch of stuff I didn’t care about, or were not on Arch, like automake, subversion and whatever gave him his tracking window.)

For several minutes I got to watch 2 Raspberry Pi Model 2 boxes cranking in sync compiling a decent chunk of code. Nice. My first successful highly parallel task on the Pi. As build master at other places, I had set up a Beowulf Cluster with pppmake and other bits and done a distributed build. Nobody else really cared, but I did. This distcc method is a lot better and easier. Then again, it is now about 20 years later…

On my “todo list” is to clean this up so that I can start taking a whack at Gentoo. Explore adding OpenMPI and see if it works better in some context:

https://apparatusd.wordpress.com/2012/04/18/hpc-high-performance-compute-cluster-with-mpi-and-arch/

Folks will remember that my tests of OpenMPI FORTRAN were not positive. Slower with it than without. This distcc experience shows it isn’t the Pi that’s the problem, and not the process of distributed computing. It is either the OpenMPI implementation or FORTRAN that’s sucking, and I think it is unlikely to be the FORTRAN as those folks are all about speed and parallel when possible.

For now, I’ve got a great distributed compile process. Distributed execution TBD… We’ll see if C does it better with OpenMPI.

Perhaps, if I feel really ambitious one day, I’ll set up the old PCs as Linux distcc cross compile servers and try that, too. But not, I think, today (or really, tonight). Now it is time to clean up, write up, and tidy up the space for tomorrow.

Oh, and re-read those how to install Gentoo on Pi pages ;-)

Subscribe to feed

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in Tech Bits. Bookmark the permalink.

8 Responses to distcc – Success! That was fun.

  1. p.g.sharrow says:

    @EMSmith;”For several minutes I got to watch 2 Raspberry Pi Model 2 boxes cranking in sync compiling a decent chunk of code. Nice”
    NICE! Wonderful! :-) After pawing through mountains of less then useful stuff, something that works right is a pleasant outcome. I’m glad that you are the squirrel that is doing the digging as I can barely follow your description of the path you have followed in this journey. Getting 2 times 4 cores to work together effectively is good news to me…pg

  2. E.M.Smith says:

    Yeah… it was pushing midnight as I did this write-up, brain was starting to sputter, mostly needed to grab the bits before they evaporated so today would not be “Now what did I do?”…but continuity suffers then, so it is disjoint, compared to my usual orderly process flow descriptions. More like what really happens on one of these devo trips. Eventually I’ll have it in a final build script and it will be less murky ;-)

    For now, I can use the cluster to bootstrap Gentoo on an empty chip, and work on polish later. It also does a decent “here thar be dragons” warning about where the models didn’t quite match reality…

    After my first Gentoo build, I’ll know if I need to add the PCs as cross compiler servers too. Given that one has dying DVD drive, another a dodgy hard disk, they would make nice PXE Booted CPU servers as diskless nodes (but I’m avoiding that time cost unless needed…)

    Amusing sidebar: My old Beowulf Cluster that I’ve been pining after for years, was an 8 box White Box PC cluster with about 400 MHz CPUs and an average about 80 MB memory each. Cost new about $4000 I’d guess. (I got them as castoffs) Runnung Red Hat 7.2 release (pre Fedora). The 2 x PiM2 cluster has a similar 8 cores, running 900 MHz, with about 250 Mb /core, cost $125 or so, all in, and has a couple of TB of disk available. (Old cluster had 8 x 10 Gb, or 80 GB total). Old cluster was 8 x 250 W power, or about 2 kW. This one is minescule at under 10 W (how far under is unclear… the server node is on a 5 V 1 A power supply, so that sets an upper bound). My how times change :-)

    I’m now ready to let go of the last two nodes of the old cluster out in the garage :-0

    As soon as I can find a good efficient distributed FORTRAN solution for the Pi I’ll be all set for running climate codes… but frankly, at $100 for the Nvidia board, it is way attractive as a compute server node…

    That’s the problem of having a working *nix cluster. .. you start shopping for more nodes…

    Well, time for that morning beverage and day planning.

  3. E.M.Smith says:

    It would appear that after a complete logout, and then logging back in (don’t think the shutdown / restart was important) this morning my .bashrc file is being used…

    I would speculate that it is set at login, then any new lxterm window opened just inherits the set state, so then changing the .bashrc file doesn’t have immediate effect. OK, I can live with that.

    At least this morning I didn’t have to set the $PATH or $DFISTCC_HOSTS environment variables by hand:

    sh-4.3$ echo $PATH
    /distcc:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
    sh-4.3$ which cc
    /distcc/cc
    sh-4.3$ echo $DISTCC_HOSTS
    10.16.16.253/24
    sh-4.3$ 
    

    sh-43$ is the user prompt for the Arch box… I’ve not fully customized that name ;-)

    Unfortunately, I didn’t ‘clean up’ the .bashrc entries last night, so I now need to go change the DISTCC_HOSTS entry, log out, log back in again…

    The Debian Jessie server node is using System V init (that ‘service FOO start’) and it shows it doesn’t start:

    root@RPiM2_server:/home/pi# service distcc start
    root@RPiM2_server:/home/pi# service distcc status
    �� distcc.service - LSB: simple distributed compiler server
       Loaded: loaded (/etc/init.d/distcc)
       Active: active (exited) since Fri 2016-05-06 06:46:40 UTC; 9h ago
      Process: 633 ExecStart=/etc/init.d/distcc start (code=exited, status=0/SUCCESS)
    
    May 06 06:46:40 RPiM2_server distcc[633]: STARTDISTCC is set to false in /et....
    May 06 06:46:40 RPiM2_server distcc[633]: /usr/bin/distccd not starting ... ....
    Hint: Some lines were ellipsized, use -l to show in full.
    root@RPiM2_server:/home/pi# 
    

    So I think my thesis that it’s the missing now obligatory allow list has validity. I need to find out how to put that into the init file / process (or startup config file, but the man page says there isn’t one…)

    So for now, ti stays with the manual Daemon launch process.

    distccd –daemon -a 10.16.16.0/24

    So doing that on the server was all the “startup” I needed this morning. I’ve now done a “make clean” in the git directory and another “make -j16” and things are cranking again, so it clearly is working. The Server Node is loaded up to 0.0 idle percent at -j16, so that’s pretty good! (Sporadically going to single digit percent idle as jobs finish / swap).

    I’m generally a “happy camper” with it this morning. Just need to tune up the .bashrc file and I’m “good to go”. Though, at present, with the compiles all sent off to the server, I can still use the head node for posting ;-) (The queue management, linking, etc. are taking about 1/2 to 1/3 the CPU on the head node at the moment while running a browser). So to some extent I can “tune” it for “all compiles remote” or “load up my workstation node too”. I think I like that…

  4. Bill S says:

    Please tell me you are retired! I have yet to finish failing to install arch linux on my Pi 3 and you have moved on to Gentoo.

  5. Larry Ledwick says:

    Nice to see the project is moving along, I can’t really add anything to your efforts so as in many of these threads just lurking, but do appreciate the inside look to the project, so please continue.

  6. E.M.Smith says:

    @Bill S.:

    I am “involuntarily between contracts”, or “on sabbatical”, or “underunemployed”, or, if you like, semi-retired…

    I don’t have anything against getting a job, it is just that looking has become less interesting, and harder…

    Don’t feel bad about my speed of bringing up a Linux release… remember that I have been doing Unix since about 1980, and Linux since it began, professionally. On one contract, my job was to spin up and install Red Hat 7 or so (pre Fedora and pre Enterprise) on about 30 Dell deskside engineering machines in a couple of days…. do that for a decade or two, you get good at it :-)

    Biggest slowdown for me is just the jerks constantly mutating the sysadmin tools, commands, and world… so now I have to sort out “What Style?” for every box and command… is it { BSD, Version 7, System V, POSIX, Sun OS, Solaris, Irix, Linux – a coat of many colors…, and now SystemD} for almost every sysadmin command. I’m sure I left out a few (HP/UX and UniCOS come to mind) plus the dozen ways Linux has diverged… But usually 10 min on the box and that profile is loaded back in the brain…

    I imagine car mechanics have the same gripe…

    FWIW, for an uncomplicated Linux install, I figure I’m blowing it at the one hour mark if I’m not done with the basic install (at least now with fast I/O and no need to compile your system). FWIW, I’m adding LFS Linux From Scratch to my build list… It is just a very well done project. Reminds me of The Old Days and booting BSD from a source tape onto cold iron… and compiling for a day or two :-)

    While I enjoy this, so do it for fun, it also keeps me current and skills high should I get / take a contract. The Customer does NOT want to hear “Oh, SystemD? I heard about that, wonder what it changed?…” They expect you to have had hands on recently, not 2 years ago, or read about it online… Besides, it is kind of fun in an interview to ‘name drop’ that you have in your home office a mix of BSD, Red Hat, Arch and Debian and run a Beowulf Cluster at home for big compile jobs ;-)

  7. E.M.Smith says:

    After a long dive into “How To Install Gentoo” that took a sidetrack into LFS Linux From Scratch, I’ve learned some more about what is recommended, and where their might be shortcuts that save a day or three if you are willing to accept a bit of “I know what I’m doing so it isn’t risk!”…risk.

    One bit learned is just that the whole issue of how one handles libraries is rather central to risk-free success (and all the models are based on the notion you are cross compiling from at least a different library set and possibly also a different hardware type – reasonable approach most of the time…)

    This means that to be really “risk free” with parallel compiles, you need not only the same compiler release level (ignoring minor number) but also the same libraries and the same library changes during the build.

    During a cross-build (be it hardware type OR library release level…) there’s an interesting “two step” dance with libraries. Compiler A using Libraries A makes new-arch new-tool-chain. The New-Arch New-Tools are made with A libraries. These then compile the libraries B. THEN, you get to use New-B-A-Lib tools, to compile a New-B-B-Lib Tool-chain… that is then used for the rest of the process.

    The upshot of all this is that for the specific purpose of doing a distcc distributed build of Gentoo using Arch I need to take the regular “do it on one Pi” scripts / processes and convert them to use two boxes with two sets of libraries and keep it all in sync. That will be a ‘good trick’… since even on matched build platforms some bits of the build fail in a parallel make (due to ill defined dependencies…)

    This, too, implies that having two matched Arch workstations for the first round is the ideal way to go (as by definition their A libraries and tool-chain match). So yesterday I bought 2 x 32 GB Ultra Speed SD cards on which to put two matched clones of Arch as “Build Monster” core. Both will get the distcc treatment in the same way too.

    Now, the fun bit: There are some directions for using distcc in a ‘from source’ Linux build. I need to sort them out, integrate that with the “make in on a Pi” build instructions (since BOTH are mutually orthogonal… the distcc directions inevitably are for Intel architechture and not Pi, that has quirks…)

    From the looks of it, my best approach is likely to be “put the shared libs on a shared NFS disk”, so as one core or another builds a chunk of Lib-B is is available to both (just like a build on one box). In that way, only mutual timing dependencies are ‘an issue’. (I.e. core 3 builds Lib-B1 and core4 builds Lib-B2, but core2 runs off to make Lib-B3 and expects that Lib-B1 is already built and available, and it isn’t yet…) But I’m willing to sort those out when the time comes. Some of the parallel build pages list Here_There_Be_Dragons bits on parallel makes, so one can know when to single thread the build… most likely ;-)

    FWIW, in the dive into LFS, it was a marvelous discovery. A VERY well written book / how-to that is fairly hand-holdy on the whole build a system stuff. It’s a VERY long process (couple of days IF everything goes well, and that is just to the basic login prompt. No X windows or other fancy stuff, that is in BLFS Beyond Linux From Scratch; so likely a week all-up to be working and done.

    In comparison, Gentoo is a hacker paradise. Things like “Oh, and watch the libraries, in the usual way” and “It was a bitch but got it working, watch out in the cross compile step.” and other not so helpful things. OTOH, there’s available a pre-built Gentoo download image that you can put in the /os directory on a NOOBs card and likely boot it OK. On my “todo” list is test that… IF that works, that’s the shortcut I’ll take. Strait to a bootable Gentoo (and skip the usual build from source) as my 2 x Build Boxes… then backfill that whole build from sources process…

    IF that doesn’t work (or maybe just one day when bored ;-) I’m going to do the whole LFS process. It is just so well written and has some nostalgia for me as it is very much like the old “bare metal installs” of 30 years ago when you loaded a boot compiler to compile the compiler and libraries and then the rest of the tape… It would also make a Great Refresher on the whole process of slowly swapping compilers and libraries as things build… It also makes a fairly clean Linux, with all sources in your own hands, so nobody can swap binaries under you by surprise (or by capturing your ‘apt-get’ or ‘pacman’ or… fetch of binaries over the internet). Also, like Gentoo, there’s some folks who have made a ‘hardened’ version of the kernel and tools. Nice.

    So we’ll see how progress goes on this. It’s a bit slow, partly as I’m doing a lot of other things, and partly as it isn’t exactly easy to do. Lots of moving parts, some different from ‘before times’, and any minor screw-up you might end up throwing it all away when you reach the end and starting over, so slow-and-no-screw-ups is important. (Thus the 2 days spent reading about doing it on various systems instead of doing it…)

    So, summary:

    Test a ‘quicky’ gentoo-blob into NOOBS/os and boot process:
    Good: Proceed to build 2 x Gentoo build systems and system-build-script.
    Bad: Proceed to build 2 x Arch build systems and system-build-script.
    In Parallel: Work out the likely layout of a distcc parallel build system process with shared disk. Then config the build system (when done) with it. Test.

    Do a slow steady build on the build systems, documenting what worked and what has failures and how recovered. As appropriate, post bits. Build a “build script” along the way. When the system is built from sources, test it. On acceptance:

    Run the build script to make a clone of it. Test and compare.

    At the end of that I ought to have a nice end to end build process to crank out a generic Linux OS from scratch sources at the push of a script button (and waiting and waiting ;-). Then I can start working on the customizing ideas. Hardened kernel. Maybe a swap to uClibc libraries for smaller size and hardened function (the router guys harden things a LOT more than the desktop folks…), etc. etc. Also that whole “load into RAM to run and lock /usr from being written and add Tor and… make a secure station aspect of it all. Onion-Torrifying it.”

    Oh, and I got a new 2 TB disk just so I can free up one of my others as ‘build farm disk’. I suspect that some (much?) of the sloth of the regular build process is writing to the SD cards. Even cards with fast read have slow write. So I intend to put the build sources, and output, onto a real disk NFS shared to the second system. (Fairly typical in clusters. One node has the data, the others are compute nodes with local scratch. Often a dedicated NFS Raid Farm, but I don’t have one of those… yet ;-) So I’ve spent much of today “shoveling bits around” and making things tidy and available…

    With that, I’m back to “building the tools to build the tool-chain to build the tools to build the system”… bootstrap development is so recursive ;-)

  8. Pingback: Why Multi-Boot and Time To New Distro Boot | Musings from the Chiefio

Anything to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s