Just as a ‘working draft’ and with a lot of potential variations “going forward”, here’s what I’m planning to build for my general compute environment.
I’ll be using Raspberry Pi SBC (Single Board Computer) products as the exemplar, but in fact any of Orange Pi, Banana Pi, Cubieboard, Beagleboard, etc. hardware will work fine too. I’m just looking at the $/compute (and to a small extent the interfaces and hardware on the board) and choosing the one that is the best bang for the buck. As that changes rapidly over time, on any given day the exact answer will be different.
As I’m fairly lousy with artwork, I’m going to use “text pictures” for the list of hardware servers. It’s “good enough” and doesn’t take a lot of bits.
^ ^ |-Boundary Router === WiFi router| === Server Stack | |_ semi-public host - browser station | DMZ Hosts (eventually VPN, private cloud, private Email server, bittorrent)
The Boundary Router is, at present, an old one from the parts box that was used to replace the AT&T prefered model after a power surge killed it. The line with a ‘hat’ at the start is the connection to the ISP. “Someday” it might become a better one. For now, all it needs to do is connect from The Internet to The Private Side network and provide NAT Network Address Translation. (That lets me use non-routing numbers like 192.168.1.1 internally. There are three non-routing blocks including 172.16-31.x.x and 10.x.x.x and you can use any of them).
In production shops we’d have two routers to two different ISPs via two different communications links out two ends of the building (POP or Point Of Presence) and a load balancer between them. For my ‘backup’, I have a WiFi hotspot that I can bridge in if ever needed (like, oh, post quake or hurricane…)
“Someday” it would also be expected to provide a DMZ or Demilarized Zone service where a VPN target for personal VPNs from anywhere can land, a “private cloud” where you can have all the cloud features you like, but without the spying by Google and PRISM program members, and a private Email server. These typically come from a filtered subnet with limited connections “internally” to keep your private side private.
At present, I’m not using any VPN, but intend to get a service in a foreign land just so that my traffic can, if desired, originate / land outside US jurisdiction. Why? Just to add yet more encrypted VPN traffic to the world and suck up more resources just to find out I’m reading the Wiki… a “nose tweak”… Yeah, I’m like that…
I presently use relatively crappy, but free, AOL mail. HOWEVER, they managed to dump my decade of accumulated mail archives and lose a years worth of recent mail and I’m just pissed at them. So eventually I’ll go back to running my own Email server. I’d done it prior to AOL and can easily do it again. (I’ve already changed to them just being a ‘pass through’ to a private mail handler).
To the extent I want those services prior to building out the whole magilla, I’l put them on a semi-public host right off of the WiFi internal router and use it to filter. Likely put them on one Raspberry Pi all together.
The WiFi router gives access via WiFi to me and any guests. The Netgear I just put in supports having a Guest network (like a DMZ) that can be prevented from seeing the private side, and that’s where the Guest WiFi and any DMZ-in-process hosts would go. All my stuff will be on the private side. The fully private side.
When doing “general browsing”, it is done from a dedicated tablet, or from a machine that can be booted “fresh” each time from a CD-ROM. Something like Tails for really paranoid things, or just MacPuppy for general browsing of low risk. Doing this, nothing from a “risky” web site can crawl into the fully private side. Also, things like bill paying are done from a ‘fresh boot from CD-ROM’ each time, so there is very little chance of being hacked or having a trojan or “whatever” stealing your information. So one tablet (or the Chromebox), plus one Old White Box PC with a CD-Drive dedicated here.
Finally, there’s the fully Private Side with the Server Stack. That’s the “back room”. General web browsing not done from there. (At MOST, download of software from known sources or similar, though better is for those to be downloaded on the semi-public platform and moved over via secure means like DVD, CD, or USB.) Eventually I might put an isolation router between that “server stack” and the “WiFi router” just to make it that much more filtered, but frankly I’m not that interesting a target and would be doing it only for fun, really.
That Server Stack includes a private storage farm (RAID like disks, with eventual encryption at the disk layer) along with backup systems (tape, DVD, whatever is du jour… I’ve got an old 8mm drive somewhere and might get it going again). It also includes a DHCP / PXE boot server for any machines in the “back room” along with NFS / SAMBA disk services; that way any “node” can come up bare metal, with OS and Files provided from the “farm”. Only one place to maintain, and one place to defend. Much of the OS mounted “read only” to the target machines.
For now, that will also include all hosts in the DMZ and on the WiFi network as there is no filtering between “back room” and WiFi at present, and DMZ hosts are not ‘outside’ yet. (Or even really built yet). When those network segments become actual subnets, a dedicated Raspberry Pi gets the job of DNS, DHCP, and PXE Boot server for each of those subnets. OS images served via an SD card with a locked status and mounted RO on the clients. (Good luck finding a way to hack that remotely…) and only the “live” NFS mounted files can be written. Likely that those NFS (Network File System) files will be coming from a USB on that Raspberry Pi too (just so ‘leakage’ further inside is even less likely).
DNS, DHCP, PXE Boot, NFS / Samba is the core set on infrastructure servers, and can run from one R.Pi.
Email, VPN, Private Cloud, Bittorrent are the DMZ set and can also be run from one R.Pi
(or from a zoo of old junk like I have now), though I’d likely make the WiFi router dedicated.
Boundary Router and any internal routers locked down and dedicated. Harder to crack into them that way. Likely minimal R.Pi configs or gentoo based, though commercial gear with OpenWRT or DD-WRT ought to be fine too.
Yes, all that is way overkill for a home system. It is very similar to what I’ve built for clients for a few decades. For them, though, the routers are usually Cisco gear or similar and the servers were often higher end PCs in racks or things like Sun boxes and Network Appliance file servers. Also, there was typically a matching “inside” Email server and the inside and outside could only talk to each other over a single protocol of the email exchange program and port. Even if a box outside was compromised, all it could do was squirt email at the inside box. For “pro” scale we would also add intrusion detection systems / servers and do “penetration testing” with an attack server / platform. That can come later for the home toy stack.
Also a ‘pro” site would have an FTP server and a Web Server in the DMZ (and potentially inside as well if needed).
That’s the basic layout. I’ll use what I have for now, and add Raspberry Pi bits of kit as time, money, and interest (or broken dead old junk) warrants.
Order of Build
I already have the tablet / Chromebox as browser stations using the WiFi.
I already have the WiFi router (as a new purchase, someday to get OpenWRT).
I already have the boundary router – a low end box, but adequate. Someday to be upscaled. Maybe.
I’ve been working on the private side “personal compute stations”. This has been less productive than desired, as some old gear has had parts die. At this point I’ve got a couple of working Windows Boxes for ‘legacy’ crap. I’ll not be buying any new MicroSoft stuff. Eventually some of the legacy stuff may be repurposed to dedicated Linux use (but for now I’ve just made them dual boot as that’s ‘good enough’).
I’ve got a nicely working CentOS 64 bit station with GIStemp compiled and installed, along with an archive of old copies of the temperature data. (It is also way too full of old crap as I try to sort out the couple of Terabytes of accumulated trash of a few decades…)
Not working as desired is the personal private side Linux Station. It had Debian on it, but X-windows caused hangs. I moved to Devuan (without systemd) and it stabilized the video some, but still has hung once in a few boots / hours of use; and crashed twice for unknown reasons. It “needs work” to make it what I’d accept as a stable reliable box. Then the CD reader died, so the hardware has ongoing issues. I’ve decided to ‘depricate it’ to being just a legacy XP box and Flavor Of The Day linux boot. With those Flavors coming from a PXE Boot server To Be Built.
On order is a Raspberry Pi Model 2 B kit. That’s a quad core 900 MHz 1 GB memory $42 card, in a kit with power supply, SD card, case, etc. etc. $70 from Amazon.
That’s going to become my Daily Driver Linux Box for all things Linux / Unix / Tech and non-GIStemp non-models. The 64 Bit Asus / Antek will be essentially dedicated to being a temperature and models research station (for now…)
Initially I’m going to add a TB or two of USB Disks to the Daily Driver Linux and it will do double duty as NFS Server for a while.
An old R.Pi B model (one core 700 MHz) is presently doing lite NFS service (an SD card) and will perhaps get the NFS TB Disk. It is also being the Bittorrent Server and DNS server and with a full load on it the DNS can be slow. Eventually the Bittorrent part goes to the DMZ and the DNS gets split between one for the DMZ, one for the private site. Also the PXE Boot server needs a dedicated system on the same subnet as the clients… so…
After that desktop Daily Driver is running, a second R.Pi gets bought to become the PXE Boot / DHCP / DNS server for semi-private side (and eventually one for the fully private side… one feels much more secure doing fully private things when the network to the internet is shut down… but you need some services in place for that. The “air gap” is your friend.) Why my own DNS servers? Since some TLAs have begun to play with DNS servers and feeding you bogus IPs, It’s nice to be able to hard code those that you depend upon for security, like your remote VPN company and Email server feed. Yes, you get to do ongoing maintenance and may suffer an outage if they suddenly change numbers, but you will know…
That second R.Pi M2 along with the old B model will likely be in a ‘dogbone case’ like this one:
I’m also looking at the Cubieboard with built in SATA as the disk sever farm. A lot faster than USB. On the “someday” list. I need to learn more about the Cubieboard product line to make a smart choice here. The Banana Pi is also a possible. Some of the boards have built in WiFi, and some have SATA, and some have more and different cores than the others. It takes a while to sort out. In a test on media center use, the Cubieboard was something like twice as fast as the Model B R.Pi, and 1.4 x the Banana Pi IIRC. No data on the R.Pi. M2 with 4 cores (or any newer boards with 4 cores from other vendors).
Unfortunately, the ARM chip cores have one of the most confusing naming conventions possible where larger numbers often mean slower performance and where all sorts of arcane difference exist in the chipsets. Sigh. Add that the ARM chips can be built into several SOC System On Chip products by different vendors (that brings their own variations in performance / features / reliability) and those go on various boards with different goezintas and goesoutas for connectivity and it can take a weekend and a couple of bottles of wine to sort them all out by speed / performance / gotchas… Only THEN do you get to start matching feature sets vs price to desired server performance requirements. Sigh.
So I just “bit the bullet” on the Quad Core Pi to get started. I know, at $40-$50 for a board, it isn’t exactly a huge cost and failure to “optimize” is worth about 1/2 hour of time wasted, but I’d rather avoid the “order them all and trial / error” process.
At present, I’m going to have 2 Raspberry Pi boards from two very different eras. When the File Server farm is added, likely a SATA capable addition. As the PXE Bootservers build out, 2 more of something fairly cheap. Darned near anything can feed a fixed copy of data to an ethernet once or twice an hour, so any old thing ought to work fine. Heck, even old B boards at maybe $20 each? With an old SD card of which I have several already to hold the system images. Again, not exactly a big cost needing a lot of optimizing.
Eventually, as time and interest allows, I’ll slowly replace the “commercial bits” like the new Netgear and my boundary router with dedicated SBCs in a Dogbone stack. (Or put OpenWRT on the Netgear). Why? Better control. I don’t need something where a vendor in the PRISM System can send new flash downloads into my routers without my permission. I’d rather have a locked SD card or ROM that can’t be rewritten. I also want to be able to assure I know just which encryption code is running and that someone didn’t just decide to swap out my preferred version for something more “Key Escrow Friendly”…
At this point I’m into it about $200 for the new WiFi router, Antek/ASUS 64 bit box with XP, and the new Raspberry Pi kit. Not bad. You could add $180 for the Chromebox and $400 ish for the Samsung tablet, but those were bought in prior years and for prior reasons and really need to be seen as ‘legacy’ stuff at this point (as, too, the two LCD screens I’m using of very old vintage and the keyboards and mice from the “junk pile” and legacy kit.)
New buys likely to run about $160 for the Dogbone with boards and added bits. Then it will just be “how much SATA disk to I want” and how many working parts do I want to replace to play with or prove something…
At the end of it all, about $400 for a rather fully over the top system with mostly Open Source, loads of security, more performance than I can use, and a load of disk too big to keep well organized ;-)
Oh, and I expect to make the ChromeBox the TV driving media server for Netflix, Youtube, et. al. once I buy a new wide screen TV with HDMI. I can’t really see needing it for a ‘Daily Driver’ once the R.Pi. Linux box is running and I can PXE boot the System Du Jour onto the Evo (or old Vectra where the DVD / CD drive died too, or the old…). It does fine with Netflix, and the other things are better for browsing and word processing / office work without “sharing” your goods with Google. So one really needs a line from the Boundary Router or WiFi to the Chromebox as Media Server in the final design. But that’s a few months away.
Hopefully this shows just how much you can build with a couple of hundred dollars of Pi Boards and some time. Frankly, the time spent configuring things will be more of a ‘cost’ than the bucks. OTOH, it will look pretty cool in the Dogbone stack ;-)
After that? Well, after that I’ll likely (slowly) move on to finally building that Beowulf Cluster I’ve been wanting. I’d made a 7 node one out of old White Box PCs some 20 years ago. It’s slowly “gone away” as boxes died. As of now I think there’s just one or 2 nodes left in running condition in the garage (and they are way slow nodes now). For another $40 case and $200 of R.PiM2 boards, I could have a very nice cluster setup. I’ve already got a 100 Mb switch in the junk pile. IFF I ever ran it faster than that, adding a Gb Switch would be cheap. It would be a nice setup for running some of the climate models… 5 boards at 4 cores each is 20 total cores. Then there are 5 GPUs that can be made to do non-graphics duty. (Display is done on a workstation – so that Daily Driver Linux…) That’s the “dream box” to build in the very back end, after everything else is built and proven. Essentially a replacement for the Antek / ASUS as the Temperature Compute Engine.
Lots of work? Well, yes. But I’ve done all of it before on a variety of different equipment with a variety of different vendors and software. “It’s what I do” in many ways. Finally doing it for myself is kind of fun, in a ‘cobler gets new shoes’ kind of way ;-)
Besides, thanks to Moore’s Law it no longer costs $1/2 Million to build, but just “one night on the town” money ;-)