Raspberry Pi M3 i2p Server Full Install

What this posting is about

This posting is a “cook book” of bringing up a Raspberry Pi Model 3 on the i2p privacy overlay network. This is somewhat like the Tor .onion Dark Net, but a little different. Tor uses 3 hops in a tunnel and uses the same hops for send and receive. Only your traffic is in your tunnel. The i2p network lets you choose how many hops to use for your service (so you can trade more security for less performance, or more performance for less security) but with the default of 3. It also blends different users traffic into the same tunnels (so is called ‘garlic routing’ instead of ‘onion routing’) and it uses different tunnels for send and receive. Also tunnels are swapped every 10 minutes making tracing very difficult even with some access to the underlying network equipment.

I found if very easy, nearly trivial, to install i2p on 2 very different Linux machines. All of them were based on some flavor of Debian, so the process was essentially the same. “apt install i2p”. That’s it. This ought to work on your choice of: Raspbian, Debian, Devuan, Armbian, or Ubuntu. I know that it works on Armbian and Debian on other hardware.

Anyone interested in my Tail Of Woe read on. Those just wanting it to work, skip down to the WORKED! heading.

Armbian – No Joy

When you check the Armbian download pages, you find no Raspberry Pi. Reading the discussion page for that, they are dead set against doing a Raspberry Pi port. Why? Mostly they don’t like the closed source boot loader. They also talk dirt about the hardware some (poor I/O bandwidth, light weight video cores, etc.) and make some claims of failure prone hardware. I’ve not seen that. I did have one of the original Pi single core boards die on me, but it was being hauled back and forth from California to Florida and used & handled without any static protection at all. Its mate, also bought about 8? years ago, has been running continuously for a few years now as DNS, Time, and Squid proxy server. But whatever, they have no OS for the Pi, so ‘moving on’:

Devuan – No Joy on Ascii

I’m going to do this install trial first on Devuan, the SystemD free OS version. Furthermore, I’m going to do it ‘backlevel’ by one release.

Why? Well, 2 reasons really. First off, the ASCII release is still in the mirrors as a regular release. For the new 3.0 Beowulf, Devuan have gone to more of a “user community” process for builds for ARM chips, and that’s a bit more chaotic. I want something more standardized for this posting. Then secondly, if it works in Ascii, then it’s been around a good while and that’s nice to know too. Sometimes things don’t make it into the ARM chip world as fast as they do in the Intel side of things.

You may, if you wish, follow all the regular install instructions for any of those Debian based OS alternatives on the R. Pi and it all ought to work the same after the OS is installed.

Download and install a Devuan OS

This section includes a long detailed careful example of loading an OS onto a uSD card. I’ll not be repeating that in other sections.

As mentioned, I’m downloading a copy of the 2.0 Ascii build that’s one level back-leveled. I’m getting mine from here:

http://mirror.alpix.eu/devuan/devuan_ascii/embedded/

u-boot/                                            06-Jun-2018 11:28                   -
README.txt                                         06-Jun-2018 11:28                2795
SHA256SUMS                                         06-Jun-2018 11:28                2412
SHA256SUMS.asc                                     06-Jun-2018 11:28                1528
ddroid-2018-06-05.zip                              06-Jun-2018 11:28             5778067
devuan_ascii_2.0.0_arm64_raspi3.img.xz             06-Jun-2018 11:28           160068308
devuan_ascii_2.0.0_arm64_raspi3.tar.gz             06-Jun-2018 11:28           231711445
devuan_ascii_2.0.0_armel_raspi1.img.xz             06-Jun-2018 11:28           152467380
devuan_ascii_2.0.0_armel_raspi1.tar.gz             06-Jun-2018 11:28           220286010
devuan_ascii_2.0.0_armhf_beagleboneblack.img.xz    06-Jun-2018 11:28           153267156
devuan_ascii_2.0.0_armhf_beagleboneblack.tar.gz    06-Jun-2018 11:28           215303387
devuan_ascii_2.0.0_armhf_chromeacer.img.xz         06-Jun-2018 11:28           197930992
devuan_ascii_2.0.0_armhf_chromeacer.tar.gz         06-Jun-2018 11:28           294143167
devuan_ascii_2.0.0_armhf_droid4.img.xz             06-Jun-2018 11:28           138730596
devuan_ascii_2.0.0_armhf_droid4.tar.gz             06-Jun-2018 11:28           194227526
devuan_ascii_2.0.0_armhf_n9.img.xz                 06-Jun-2018 11:28           140260756
devuan_ascii_2.0.0_armhf_n9.tar.gz                 06-Jun-2018 11:28           195496761
devuan_ascii_2.0.0_armhf_n900.img.xz               06-Jun-2018 11:28           137080000
devuan_ascii_2.0.0_armhf_n900.tar.gz               06-Jun-2018 11:28           197467631
devuan_ascii_2.0.0_armhf_n950.img.xz               06-Jun-2018 11:28           140166116
devuan_ascii_2.0.0_armhf_n950.tar.gz               06-Jun-2018 11:28           195477021
devuan_ascii_2.0.0_armhf_odroidxu4.img.xz          06-Jun-2018 11:28           151147248
devuan_ascii_2.0.0_armhf_odroidxu4.tar.gz          06-Jun-2018 11:28           215414703
devuan_ascii_2.0.0_armhf_raspi2.img.xz             06-Jun-2018 11:28           159664056
devuan_ascii_2.0.0_armhf_raspi2.tar.gz             06-Jun-2018 11:28           225220011
devuan_ascii_2.0.0_armhf_sunxi.img.xz              06-Jun-2018 11:28           136928708
devuan_ascii_2.0.0_armhf_sunxi.tar.gz    

Though any mirror from their mirrors page ought to still have the embedded/ascii builds in it:

https://www.devuan.org/get-devuan

In your browser, just click on the version you need (raspi3 in this case) and it ought to start to download.

Those looking for the newer Beowulf 3.0 build can try here:

https://arm-files.devuan.org/

Though as I type this, there is no Beowulf build (though the forum pages show people who are running it as an experimental install straight from the armel / armhf / aarch64 repositories for generic builds.)

../
README.txt                                         19-Nov-2020 21:44    5671
SHA256SUMS                                         25-Dec-2020 13:50     688
devuan_beowulf_3.0.0_arm64_npir2s_0.2.img.xz       20-Dec-2020 23:10    468M
devuan_beowulf_3.0.0_arm64_orangepi_one_plus_0...> 25-Dec-2020 13:51    462M
devuan_beowulf_3.0.0_arm64_rockpi4_0.2.img.xz      05-Apr-2020 23:49    525M
devuan_beowulf_3.0.0_arm64_rockpro64_0.2.tar.xz    08-May-2020 18:34    249M
devuan_beowulf_3.0.0_armel_rpi1_0.3.img.xz         05-Apr-2020 23:49    473M
devuan_beowulf_3.0.0_armhf_olinuxino_lime2_0.5...> 04-Jun-2020 21:39    456M

So that’s why I’m using Ascii for this… Just a ‘download install and go’.

The image is an xz compressed file, so you need to uncompress it. This is done with the unxz program. If you don’t have it installed, as root issue “apt-get install unxz”.

Then you can just do:

ems@OdroidN2:/T1/ext/Downloads$ unxz devuan_ascii_2.0.0_arm64_raspi3.img.xz 
ems@OdroidN2:/T1/ext/Downloads$ ls -l devuan_ascii_2.0.0_arm64_raspi3.img 
-rw-r--r-- 1 ems ems 1982857216 Feb 12 20:34 devuan_ascii_2.0.0_arm64_raspi3.img

Here you can see that the 160 MB we downloaded have expanded into 1.9 GB. XZ does a great job of compression ;-)

Now we put it on a uSD card. IF you are on a Windows box, I can’t help you, but a lot of folks talk about something called “etcher”. Being a Linux / Unix guy, I have a dozen of them available, so just do it with “dd”. (Which stands for “convert and copy” because “cc” was already taken by the “C Compiler”… *nix is full of inside jokes like that…)

At this point there’s an obligatory Noobs Nag: IF you do a “dd” wrong, you can completely destroy your system, any attached disk, and everything you care about. NOTHING in *Nix will protect you from yourself. If you want to play in this world, “grow a set” and buy a lot of mental bandaids. You will need them.

For those who are not afraid or need no nagging:

The hard / important bit here is to KNOW just what the proper device name is for your target uSD card. These are typically put into some kind of adapter so that they can talk to a USB port, then stuck into that USB port. Your job is to triple check that you know what your machine disks are named AND what the uSD is named. Often when you first insert the adapter, you will get a “file manager” window pop up and ask if you want to have it mounted. Sometimes you don’t get that. Feel free to mount it to the system and have a good look at it. You want to know what letter is assigned to it in the /dev/sd{X} device address.

FWIW, my favorite “paranoia check” that I do when I have important archive disks and such up and running (when the pain of a mistake is brutal) is to launch “gparted”. This is a nice graphical look at all the “disks” attached, what their names are, and what the partitions look like. It lets me see sizes and lets me look over the target device before nuking it. Somtimes I realize maybe there’s something on that uSD card I’d like to keep.

IF your base system is running from a real hard disk, it is usually /dev/sda (with numbers for each partition so /dev/sda1 might be boot and /dev/sda2 swap and /dev/sda3 / etc.) If you are using an SBC (Single Board Computer) with emmc or uSD for booting, it will show up as /dev/mmcblk{X}/p{Y} on a ‘df’ (disk free) and as similar names in gparted. Here’s a partial df of my current system:

ems@OdroidN2:/T1/ext/Downloads$ df

Filesystem     1K-blocks     Used Available Use% Mounted on
udev             1637948        0   1637948   0% /dev
tmpfs             380132      644    379488   1% /run
/dev/mmcblk1p1  16453544 12249348   3982300  76% /
tmpfs            1900648    99616   1801032   6% /dev/shm
tmpfs               5120        4      5116   1% /run/lock
tmpfs            1900648        0   1900648   0% /sys/fs/cgroup
/dev/mmcblk1p3  10057972    22664   9521056   1% /SD/ext
/dev/mmcblk0p2   7251432  6411364    535128  93% /mmc/ext
/dev/mmcblk0p1    130798    42954     87844  33% /mmc/BOOT
tmpfs            2097152   106732   1990420   6% /tmp
armbian-ramlog     51200    30500     20700  60% /var/log
/dev/sda4      830428208 85390716 702847516  11% /T1/ext
tmpfs             380128   226628    153500  60% /run/user/1000

The device /dev/mmcblk0 is the emmc chip that contains Ubuntu as shipped from the factory. The /dev/mmcblk1 device is the uSD card that boots into Armbian. Yes, I set this one up to dual boot. The “pX” part is the individual partitions on the “disk”. The other stuff tends to be some neat trick stuff that Armbian does, like put /tmp on a tmpfs temporary partition in memory as it does with run locks and other active stuff, saving your uSD from the hit. The /T1/ext partition is on a Real Hard Disk on a USB plug, and shows up as /dev/sda4 as it is the 4th partition on that disk}.

So first do a “df” and note all the things you do NOT want as targets. /dev/mmc{anything} and /dev/sda on this system. Then plug in your adapter with the target uSD. (Note that if you do NOT already have a disk on the system, your adapter may show up as /dev/sda… )

At this point, you could also inspect things with gparted. IF you don’t already have it installed, just as root do “apt-get install gparted”.

On my present system, plugging in the adapter did NOT cause it to auto-mount the chip. IF it does that, it will show up in a new ‘df’ command. You will eventually need to do a “umount /dev/sd{whatever}” in that case to make it available to molest.

For me, I just ‘guessed’ that it would go in the next slot after ‘a’ which is ‘b’ and did a basic partition check with “file”. Again, if you don’t have the ‘file’ command installed, as root, do “apt-get install file”.

ems@OdroidN2:/T1/ext/Downloads$ file -s /dev/sdb1
/dev/sdb1: Linux rev 1.0 ext4 filesystem data, UUID=a738ae0c-b3ea-4fd9-947a-2cbfbf16116d (extents) (64bit) (large files) (huge files)
ems@OdroidN2:/T1/ext/Downloads$ file -s /dev/sdb2
/dev/sdb2: cannot open `/dev/sdb2' (No such file or directory)
ems@OdroidN2:/T1/ext/Downloads$ file -s /dev/sdb3
/dev/sdb3: cannot open `/dev/sdb3' (No such file or directory)

So this shows me there is only one partition, it is an ext4 type (so a linux is on it most likely), and I may want to have a bit of a think about nuking it if I don’t know for sure why I put that on it in the first place.

Looking at it with gparted:

gparted view of uSD

gparted view of uSD

Here we can see that there’s something in the first partition for about 1 GB. That’s almost certainly a bare dd onto the chip of some system.img that has not been expanded into the whole chip. We can also see that there’s another 2.6 GB not allocated to anything (that can likely have that image in the first partition extended into it to give it some room to grow. Some systems do that automagically at first boot. For others, gparted has a ‘grow file system’ option.)

As I don’t remember what that image was, I’m briefly going to mount it, look it over, and then umount it so I can do things with it, or make a backup prior to nuking it.

So I “become root” via sudo bash, then check that I’ve got a mount point I can use. I typically make a directory named “/SD” for mounting SD cards, and in it have ‘ms’ for any Microsoft format partitions like Fat32 or NTFS and an ext directory for mounting linux partitions:

ems@OdroidN2:/T1/ext/Downloads$ cd /
ems@OdroidN2:/$ sudo bash
[sudo] password for ems: 
root@OdroidN2:/# ls /
dev   lib         mmc  proc  sbin     srv  usr
SD       T1      bin   etc   lost+found  mnt  root 
selinux  sys  var MACHINE_README  SG2      boot  home  
media       opt  run       tmp
root@OdroidN2:/# ls /SD
ext  ms

Note that “MACHINE_README” is a file I put one on each chip to tell me what it is and what I used it for. This can make sorting out the chips easier ;-) So my working machine has one of those. Now lets mount the Mystery Chip and see what it has on it.

root@OdroidN2:/# mount /dev/sdb1 /SD/ext
root@OdroidN2:/# ls /SD/ext
bin   dev  home  lost+found  mnt  proc  run   selinux  sys  usr
boot  etc  lib   media       opt  root  sbin  srv      tmp  var

OK, no “MACHINE_README” so not likely anything I used for more than a test case.

root@OdroidN2:/# cat /SD/ext/etc/hostname 
odroidxu4
root@OdroidN2:/# cat /SD/ext/etc/debian_version 
10.5

So I did boot it up and gave it a name. It is for the Odroid XU4. Also it is a Debian of 10.5 version. Rather generic and relatively new. IIRC this was a test of a Beowulf install that worked, but I realized the chip was just way small for it. Checking the end of the password file for any user logins added:

root@OdroidN2:/# tail /SD/ext/etc/passwd
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
systemd-timesync:x:101:102:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin
systemd-network:x:102:103:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin
systemd-resolve:x:103:104:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin
messagebus:x:104:105::/nonexistent:/usr/sbin/nologin
_chrony:x:105:111:Chrony daemon,,,:/var/lib/chrony:/usr/sbin/nologin
avahi-autoipd:x:106:113:Avahi autoip daemon,,,:/var/lib/avahi-autoipd:/usr/sbin/nologin
sshd:x:107:65534::/run/sshd:/usr/sbin/nologin

No user logins. So never did do any add-user or anything really. My first step after test install is to do the adduser thing.

So pretty much I can know this is an experimental chip that’s not in use for anything at the moment. Good to go on the nuking it.

WHY spend so much time on how to “look over a chip”? Because once nuked, there’s no going back…

OK, now to stuff the image onto the chip:

root@OdroidN2:/T1/ext/XU4# df /SD/ext
Filesystem     1K-blocks   Used Available Use% Mounted on
/dev/sdb1        1148472 974348     97736  91% /SD/ext
root@OdroidN2:/T1/ext/XU4# umount /SD/ext

Now on THIS system we know the uSD card is at /dev/sdb and we want to overwrite the whole thing, not just a single partition, so we use that name without any number at the end. Note too that we had to ‘umount’ unmount the “disk” prior to writing the raw image to it.

root@OdroidN2:/T1/ext/RPiM3# dd bs=20M of=/dev/sdb if=devuan_ascii_2.0.0_arm64_raspi3.img 
94+1 records in
94+1 records out
1982857216 bytes (2.0 GB, 1.8 GiB) copied, 235.628 s, 8.4 MB/s

That gives the basic Operating System installation. BUT, it doesn’t use the whole chip, nor does it ‘auto expand’ to use the whole chip as does Armbian. So first we launch gparted again, and tell it to grow the file system into the rest of the space available. I’ve done that, and captured images. They will be included a bit later for those unfamiliar with the process.

Essentially, the only ‘odd bit’ is you can’t just say “leave zero space after” as it assumes you want to do a MOVE instead of a GROW. So you have to find the “XXX MB max size” near the top of the panel and change the file system size number in the panel, then if you click on any of the other entries (like ‘free’ or whatever) you will see it change all the numbers appropriately. At that point you do the “accept” stuff and eventually click the green check mark at the top and it grows the file system.

Now I need to swap systems and boot it up. It will boot up into a simple root login where the default password for Devuan is “toor”. As the terminal is not as convenient as a full on windowing environment, the very first thing I do is:

apt-get update
apt-get upgrade
apt-get install task-lxde-desktop

That will install about 2.2 GB of “stuff” in 1108 files. Then you can reboot and launch a proper window environment and we’ll continue from there.

And… “only” 2 hours and 26 minutes later… a “apt install i2p” and an “apt-get install i2prouter” both failed to find a target. OK…

Looks like at least Devuan Ascii 2.0 didn’t bother to port it. Welcome to the world of REAL Systems Admin. This is when your boss asks you “You said it would be up in about an hour. It’s been 3 already. Why isn’t it up?” and you get to tell them that “I’m starting over from scratch with a different OS version as that one won’t do it”. And dread the 5 to 12 hours it would take to get it into their head that it isn’t always deterministic and there was no real good way to know in advance if it was there or not until you tried it… or pour through a listing of in-vs-out contents… But “it is what it is”, so, moving on.

Starting Over with Debian Buster – No Joy.

Pretty much everything above stays the same, but this time I’m going to try straight Debian.

https://raspi.debian.net/verified/20210210_raspi_3_buster.img.xz

Sidebar on Why: Why am I showing all this cruft and mistakes? Because this is what being a Systems Admin really is like, and when you are building systems and downloading operating systems and doing all this stuff, that’s exactly what you are. A Systems Administrator (and maybe a little bit of a Systems Programmer if you need to hack some stuff…). This is The Life. Deal with it…

So I did all that same stuff as above with this build here’s a sample:

ems@OdroidN2:/T1/ext/RPiM3$ file -s /dev/sdb1
/dev/sdb1: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "mkfs.fat", sectors/cluster 4, reserved sectors 4, root entries 512, Media descriptor 0xf8, sectors/FAT 256, sectors/track 32, heads 64, sectors 262144 (volumes > 32 MB), serial number 0xeea4304a, unlabeled, FAT (16 bit)
ems@OdroidN2:/T1/ext/RPiM3$ file -s /dev/sdb2
/dev/sdb2: Linux rev 1.0 ext4 filesystem data, UUID=589c7999-62be-47ac-8847-32b62fd9fdc9 (extents) (64bit) (large files) (huge files)
ems@OdroidN2:/T1/ext/RPiM3$ file -s /dev/sdb3
/dev/sdb3: cannot open `/dev/sdb3' (No such file or directory)
ems@OdroidN2:/T1/ext/RPiM3$ 

Then did the same dd onto the chip, booted it, and then did the same update stuff.

root@OdroidN2:/T1/ext/RPiM3# dd bs=20M of=/dev/sdb if=20210210_raspi_3_buster.img 
75+0 records in
75+0 records out
1572864000 bytes (1.6 GB, 1.5 GiB) copied, 224.766 s, 7.0 MB/s

And One More Time we do the boot the chip, update, upgrade, install a windows environment and try the i2p install and find out in a few hours if it worked… But this time there wasn’t enough space on the uSD card.

So I got to do it all over AGAIN, but only doing “apt-get install lxde” and leaving out a bunch of the desktop stuff / applications. “Only” 782 files and 36 minutes…

It booted up nicely into the graphical interface. I installed ‘htop’ to monitor things with “apt-get install htop”, and launched it… and the system hung. Just went unresponsive. Long story short, it did this a couple of more times with a browser and a terminal window. So once again “moving on”…

There may well be a better build of Debian Buster out there somewhere, but I’ve not found it. Didn’t really look either. I don’t remember visiting raspi.debian before, then again things do change sometimes. Rather than looking for a better Debian source, I decided to just download Raspian and give it a go.

Raspberry Pi OS aka Raspian – it WORKED!

But not without some suffering. Again mostly related to the use of a 4 GB uSD card. This uSD card is also very slow so things like “apt-get upgrade” take time reading and writing it. I’d figured if Debian fit, Raspian ought to fit. It doesn’t.

Download OS from here:

https://www.raspberrypi.org/software/operating-systems/

Well, really, it BARELY does. But as soon as you do the update / upgrade it fills the space and you get a failed upgrade. I got to toss log files, Chromium browser, a couple of programming tools I never use, etc. then got the upgrade to complete. EVENTUALLY I got to where I could do the install of i2p.

It was simple and worked first time. Either put “sudo” in front of the first command, or just do “sudo bash” and then become root. Then it is just:

apt install i2p

It installs just fine. Next, you exit root and as a simple user, launch it:

i2prouter start

This starts the router and launches your default browser to finish the configuration. As I’d deleted Chromium, it launched Dillo that was unable to handle the ‘click a language hit next’. So I got to exit Dillo, do a “i2prouter stop”, and a “apt-get install firefox-esr” then launch the i2prouter again and was this time put into Firefox. Along the way, learned that FFox is about 1/2 the size of the 340 MB or so Chromium…

In the end, I had a nice running i2p router, on wifi to my “TV_net”, and a console open on FireFox. With about 38 MB (and shrinking) of free space. Have I mentioned you ought to use at least an 8 GB uSD card? :-{

So in summary, to run a Raspberry Pi i2p router, the easiest way to do it is simple. Get an 8 GB or larger uSD card, install Raspberry Pi OS aka Raspbian on it. As root do “apt install i2p” then start it running when the install finishes. Done that way, ought to be about an hour all told. Done my way was 2 days, on and off…

After some suitable time away from keyboard, I’ll look for a better Debian image and maybe give Ubuntu a try. I’ll also dig though my box of bits and find a 16 GB uSD card… or buy one.

Subscribe to feed

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in Tech Bits. Bookmark the permalink.

31 Responses to Raspberry Pi M3 i2p Server Full Install

  1. E.M.Smith says:

    This looks like a better place to have picked up a Debian image. At least, it says “tested” in the names ;-)

    https://raspi.debian.net/tested-images/

    But it has the exact same name as the one that failed, so may be the same…

  2. E.M.Smith says:

    Detail on how to do it in various other OSs (probably mostly / all Intel based…) here:
    https://chiefio.wordpress.com/2021/02/06/installing-i2p-privacy-overlay-network/#comment-139662

  3. Mikeb says:

    Only downside I see running on Pi is that it needs java so is not going to be speedy I’d be interested to know what the idle core usage is

    Hate jave. Love coffee and tea :)

    Appreciate the time you put in and the resulting reports

  4. E.M.Smith says:

    On the Rock64 ( 1.4 MHz vs 1.2 Mhz both systems A53 Cores so basically a slightly faster pi) with both Firefox and Chromium launched and serving web pages and with 27 active peers and 24 tunnels in operation, HTOP is showing roughly load average about 1.08 and Cores bouncing between about 5% and 22% used. Lots of headroom

    More worry some is memory at 1.42 GB (then again, two browsers open at once on both clear and dark nets…) but it was about 800 MB with ony FFox and the router.

    I’d make sure you have some swap configured.

    I’ll do an actual R.Pi M3 test sometime tomorrow…

  5. E.M.Smith says:

    The Light Dawns:

    https://geti2p.net/en/download/debian

    Debian I2P Packages
    Debian Buster, Ubuntu Bionic, and later
    I2P is available in the official repositories for Ubuntu Bionic and later, and Debian Buster and Sid. However, Debian Buster and Ubuntu Bionic (LTS) distributions will have older I2P versions. If you are not running Debian Sid or the latest Ubuntu release, use our Debian repo or Launchpad PPA to ensure you’re running the latest I2P version.

    So that’s why it wasn’t in Devuan Ascii It also tells me I don’t need to test Ubuntu, it is there, just in Bionic or later. So finding a Stable Buster for the R.Pi would have the simple install work, or, they have a ‘direct from their repository’ method:

    Instructions for Debian
    Currently supported architectures: amd64, i386, armhf, arm64, powerpc, ppc64el, s390x
    Note: The steps below should be performed with root access (i.e., switching user to root with su or by prefixing each command with sudo).

    Ensure that apt-transport-https and curl are installed.

        sudo apt-get update
        sudo apt-get install apt-transport-https curl
        

    Check which version of Debian you are using on this page at the Debian wiki and verify with /etc/debian_version on your system. Then, add lines like the following to /etc/apt/sources.list.d/i2p.list.

    For Buster (stable):

        deb https://deb.i2p2.de/ buster main
        deb-src https://deb.i2p2.de/ buster main
        

    For Stretch (oldstable):

        deb https://deb.i2p2.de/ stretch main
        deb-src https://deb.i2p2.de/ stretch main
        

    For Jessie (oldoldstable):

        deb https://deb.i2p2.de/ jessie main
        deb-src https://deb.i2p2.de/ jessie main
        

    For Wheezy (obsolete):

        deb https://deb.i2p2.de/ wheezy main
        deb-src https://deb.i2p2.de/ wheezy main
        

    Note: If you are running Debian Sid (testing), then you can install I2P directly from Debian’s main repository:

      sudo apt-get install i2p
          

    Download the key used to sign the repository:

      curl -o i2p-debian-repo.key.asc https://geti2p.net/_static/i2p-debian-repo.key.asc
        

    Check the fingerprint and owner of the key without importing anything:

      gpg -n --import --import-options import-show i2p-debian-repo.key.asc
            

    Add the key to APT’s keyring:

      sudo apt-key add i2p-debian-repo.key.asc
            

    Notify your package manager of the new repository by entering

      sudo apt-get update
            

    This command will retrieve the latest list of software from every repository enabled on your system, including the I2P repository added in step 1.

    You are now ready to install I2P! Installing the i2p-keyring package will ensure that you receive updates to the repository’s GPG key.

      sudo apt-get install i2p i2p-keyring
          

    After the installation process completes you can move on to the next part of starting I2P and configuring it for your system.

    It would seem that Java or the java code in which i2p is written, either has a memory leak, or just keeps hanging on to a lot of blocks of memory from long finished activity.

    I saw this the last time I left this instance running for a day+ but thought maybe it was all the chaotic usage I had done bringing it up. This time I brought it up, got it working again, and went to bed with about 1 GB of memory used and nearly no swap. Something like 20 MB from when I’d launched both browsers.

    Here’s the “top” command from logging in with only FFox and I2p running:

    ems@rock64:~$ top
    
    top - 18:16:26 up 15:37,  1 user,  load average: 0.88, 0.52, 0.37
    Tasks: 213 total,   1 running, 212 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  9.9 us,  2.4 sy,  0.0 ni, 87.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    MiB Mem :   1991.8 total,    181.1 free,   1525.3 used,    285.5 buff/cache
    MiB Swap:   1995.9 total,   1518.7 free,    477.1 used.    267.1 avail Mem 
    

    477 MB on swap and 1525 MB used for ‘other than file buffer/cache’. That’s a bit huge for one browser with a set of 6 pages open. Two of them on the i2p version of static web pages from my copy / archive and the others just management pages for the i2p router.

    Closing the browser dropped it by about 990 MB used and swap is now over 500 MB, but some of that will be from launching this second browser. I’ll put a new number here after closing this browser.

    So my recommendation would be to not leave it running with the browser open to monitor it. Just launch the browser when you want to do something with it that involves management or browsing.

  6. E.M.Smith says:

    Both browsers shut down, only 3 terminal windows open, one with htop, the other top, and one where I’d launched the router last night and done nothing else.

    root@rock64:/# top
    
    top - 18:42:06 up 16:03,  1 user,  load average: 0.53, 1.08, 1.08
    Tasks: 204 total,   2 running, 201 sleeping,   0 stopped,   1 zombie
    %Cpu(s):  3.3 us,  1.7 sy,  0.0 ni, 95.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    MiB Mem :   1991.8 total,    719.5 free,    899.5 used,    372.8 buff/cache
    MiB Swap:   1995.9 total,   1491.9 free,    504.0 used.    964.2 avail Mem 
    

    Actual used memory drops to about 900 MB which seems high to me, but with 372 buffer cache that’s just locations of files on disk and very disposable (i.e. it is held by the OS to make it easy to find the file again IF you use it again, but the memory can be freed whenever needed). I have to wonder if the 504 MB on swap is also buffer / cache of some kind and the OS is just being lazy about freeing it…

    In any case, it does look like there’s a fair amount of memory churn on long runs with lots of pipes open and moving bits & bytes. So make sure you have some swap space configured.

    Later, likely tomorrow, I’ll try the R. Pi M3 with minimal swap and see if it chokes of if it is just ‘using all available space’ for buffer because, well, why not? It’s there, nothing else needs it, you don’t waste cycles freeing it until something else needs it…

  7. Pinroot says:

    I don’t have anything up and running yet. I want to check out Freedom box which includes i2p. However I also want to check out i2pplus (i2pplus.com). It’s an upgrade to the interface, and supposedly has some network enhancements, which might help with my connectivity problems. It’s compatible with i2p and easy to install:


    If you’re running a manually installed version of I2P from the Java installer (not a repo or .deb installed version), updating is as simple as dropping the i2pupdate.zip file into your I2P application folder and restarting the router.
    Note: Do not extract the zip file, simply copy it to the I2P application folder and restart I2P… I2P will extract the zip automatically and update.
    Alternatively, if you haven’t already installed I2P, you can download a full Windows or Linux installer from i2pplus.com

    Based on the screenshots, it looks like a cleaner user interface, along with an email client that includes an address book (the email client with the standard i2p doesn’t seem to include an address book). Plus it includes a lot of site suggestions. And I’m curious about the network tweaks, so I’ll definitely be looking into this as i like the whole idea of i2p and would like to help by running a router. I just wish it was written in something faster than Java…

  8. E.M.Smith says:

    @Pinroot:

    Nice find!

    I’ll bring it up on “something”.

    I’ve basically got my “contributed router” up and running (modulo my occasional fubar on reboot…). While it is mostly a learning test bed for me ATM, it’s also acting as a router and seems to be getting well connected. (More than 1k known others and about 50 tunnels of one kind or another going). It seems to take a couple of days to become “well known” enough that a lot happens. I intend to just leave it up and running in perpetuity in my “TV Network” (worst that can happen is it gets trashed or hacked and someone can find out I watch TV… “Who knew?” ;-)

    This is a “DMZ” off the Telco router that by definition can’t see my inside networks. Even has a different non-routing IP range that my networks don’t know about ;-)

    My cost is essentially zero (sunk cost in the SBC a couple? of years ago, negligible power, and “by the month” bandwidth so marginal cost zero). Might as well have it doing something.

    From what I’ve seen so far, CPU speed isn’t the issue. Memory gets used up faster. I MAY find the C version of “just a router” and set it up as my “contribution”, reserving the Java instance for “boot and browse and email and such, then down again” use.

    But that’s an investigation for another day. Here’s one:
    https://github.com/tayler1/i2pd

    About
    Simplified C++ implementation of I2P client. Binary downloads: https://download.i2p.io/purplei2p/i2pd/releases/

    But haven’t dug around looking to see if it is The One True C++, or Yet Another Fork…

    Or boot Java service version with the memory limit setting active.

    My workload right now is just to reach mastery on operations / configuration issues. I want to get that put to bed before I move on to one of: Chat, “Tweets”, “Social Media”.

    @ALL:

    Any suggestions on preferred clients accepted. For what ever reason, the “microblogging” (“tweeters”) folks seem to love strange languages and loads of dependencies. So, IIRC, Twister requires you install Erlang and something else first. Pleroma? Is in Ruby / Rails I think, so you get to install that build environment, etc. Bit of a PITA for an “average joe” and I’m trying to assure that “any old Joe or Jane” can bring up what’s needed. Starting off with “First install an Erlang build environment” is not my idea of user friendly…

    Chat looks built in to i2p, and I’d love to test it, but have nobody with whom I “chat”, so there’s that. Not real intersted on dropping in to a random chat room just to waste folks time… but if anyone has clue on “chats” that might be a reasonable test bed, let me know.

    As I don’t really “do” ‘tweets’ or “social media” I’m both a bit poorly motivated AND completely lacking in any knowing what matters, is good, bad, etc. So again, any suggestions welcome. (I’m leaning toward strict P2P no server needed simply because things like Mastodon start off with ‘first register your domain name’ and that’s a whole bucket of non-anon right there…)

  9. Pinroot says:

    I took a look at that link to the C++ version, but it’s from 2015. However there was a link I followed to a site which had a link to a more recent version (2020). There are a ton of different versions (~70), including ARM architecture so I guess they might be binaries. At the bottom of the listing is the source.
    https://github.com/PurpleI2P/i2pd/releases/tag/2.35.0

    It would be nice to run something like that, especially on a Pi. None of the Java overhead. It might even run on a Pi Zero since it doesn’t need the JVM, which the Zero’s hardware didn’t like.

    Chat is ok for really quick things, like “Hey, I’m testing something, can you go here and do x and let me know what happens? Thanks!”. On the other hand, I don’t like getting caught up in a never-ending texting/chat session (looking at my sister… a phone call would be faster). All of those other things sound interesting, I do like the peer to peer over the rest. I think right now you’ve about found the right thing for you, just mirroring your site on an i2p site. If this site ever gets nuked for wrong-think, you’ve at least got a backup that is safe. What would really be nice would to be able to cross-post between the two sites :) I did see a diaspora ‘pod’ on the i2p network, so it might be possible to set up a Pleroma or Mastodon server up the same way on i2p. And there’s always the whole Tor world to consider, which I know nothing about, but I suspect that there’s a Fediverse on the darknet.

  10. E.M.Smith says:

    how to for connecting to chat servers in i2p along with pointer to setting up a private chat server:

    https://i2pd.readthedocs.io/en/latest/tutorials/irc/

    @Pinroot:

    Yeah, I’ve got all the services I wanted set up. But I also want to get a kind of ‘cook book’ for anyone else wanting some other services too. But as I’m not a user of twitter or faceplant I don’t have a good basis for choosing other than ease of installation and operation.

    It looks like a private chat room is the next reasonable thing for me to set up. An example of how a small group can have ‘in-group’ communications that are private.

    Sometime after that a ‘microblogging’ twitter like “one to the encrypted world” type service. (Still leaning to Pleroma but maybe others…)

    FWIW, my Java based i2p router looks to slowly accumulate java threads that are inactive but holding memory. Not effectively garbage collecting from the looks of it. Takes a day or two of slow memory usage creep to see it. Probably why the “service” installation has a memory usage limit setting. I’m going to let it run longer this time and see if it memory limits by itself at some point (i.e. looks at total memory and stops when swap is getting full…) of if it just grows over days until the system is swap bound.

    I’ve also started an update-the-archieve-chiefio-copy process. Discovered AFTER I started it running that there’s a special flag to NOT download everything all over again.

    httrack --updatehack  --update
    

    Next time I’ll run it with that (checks file size and such and doesn’t download it again if they all match). As it stands now every update takes a couple of days, so hopefully that will go faster if it doesn’t re-do every file ;-)

  11. E.M.Smith says:

    I’ve been down the rat-hole of IRC configuration and some history. It looks like a festering blob of infighting and childish food fights.

    There’s a half dozen protocol splits and version divergence and different ‘networks’ of chat servers. All off it looks a hideous mess. Some entry points here:
    https://en.wikipedia.org/wiki/IRCd

    This came about as I was looking for an easy “IRC Server” to bring up and was discovering several names for a lot of different ircd-foo things. Yes, they each get their own little daemon for their own peculiar service spins. Sigh.

    OK, the only really good news here is that for a stand alone server, I don’t give a damn. Anything ought to do. But realize you are “picking sides in a food fight” between historical clans when you pick one…

    SO I’m going to have a bit more of a think before I set about picking anything.

    https://serverfault.com/questions/269988/best-lightweight-linux-irc-server-for-a-team

    looks like it is aimed at the “small team not in a food fight” usage, and the recommendation I think I like the most there is:
    https://github.com/jrosdahl/miniircd
    though I don’t really have the background to choose properly. The i2p docs point at one named “Eris” but searching on Eris return a dozen uses that are not it. Be warned. I think this is it:
    https://prologic.github.io/eris/
    Maybe. Possibly…

    So I’m sort of kinda leaning toward a binary choice between miniircd (mini-internet relay chat daemon) and Eris. But have no basis to decide. So flip a coin time unless someone has ideas to share.

  12. Pinroot says:

    Well, Eris is written in GO, so it looks like you’ll need to install that to get Eris to run (I could be wrong tho…). Looking at GO, the linux version wants a 64 bit Intel chip, nothing there about ARM support. miniircd looks to be a little simpler and requires Python, which definitely runs on a Pi. I’m not familiar with all the irc lingo (I’ve used irc some in the past, but not enough to be familiar with a lot of aspects), so I can’t speak to the limitations section of miniircd, but for a small private chat, I don’t think those limitations will really matter a whole lot. To me, it looks like miniirc would be the simplest route.

  13. E.M.Smith says:

    Well, it’s in there:

    root@OdroidN2:/home/ems# apt-get install golang
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following additional packages will be installed:
      golang-1.11 golang-1.11-doc golang-1.11-go golang-1.11-src golang-doc
      golang-go golang-src
    Suggested packages:
      bzr | brz mercurial subversion
    Recommended packages:
      pkg-config
    The following NEW packages will be installed:
      golang golang-1.11 golang-1.11-doc golang-1.11-go golang-1.11-src golang-doc
      golang-go golang-src
    0 upgraded, 8 newly installed, 0 to remove and 1 not upgraded.
    Need to get 52.3 MB of archives.
    After this operation, 276 MB of additional disk space will be used.
    Do you want to continue? [Y/n] 
    

    But that looks like 1.11 version. From June 2019 (so a good 1.5 years ago):
    https://www.e-tinkers.com/2019/06/better-way-to-install-golang-go-on-raspberry-pi/

    The current stable version available at Golang official website is v1.12.6, and there is a distribution packaged for ArmV6 CPU available that is suitable for Raspberry Pi 3 (and some earlier models).

    Not keen on a v6 code base on a v8 chip…

    https://golang.org/dl/

    Has them at:

    go1.15.8.linux-amd64.tar.gz (115 MB)
    

    So 0.04.8 revisions back. Could be a problem… but you can install newer

    go1.15.8.linux-arm64.tar.gz	Archive	Linux	ARMv8	93MB
    

    HOWEVER, since I’m looking for “easy for semi-tech folks to do” starting off with install an obscure language long hand as root from the site that makes it is not exactly “simple”…

  14. Pinroot says:

    Oops, I didn’t think about the repos. Still I think your last sentence sums it up. It needs to be simple enough for the average person to at least feel like they could do it without having to figure out obscure languages. I still don’t understand why so many of these projects are written in such weird and obscure languages. Proof of concept? Who knows…

  15. E.M.Smith says:

    Well, I just finished doing a ‘consolidation’ on my i2p router. I’d done the basic install test on an 8 GB card, but needed ANOTHER 8 GB uSD to hold the web site archive. I’ve now combined those two onto a single 32 GB uSD with lots of room to spare. (So now I can try that backup / archive copy of P.G.s site too ;-)

    I’ll ether screw it all up (and so get to assemble the bits from backups again) or it will serve up both sites and be an exemplar of how “Some Guy” can take it on himself to produce backup DarkNet copies of selected Web Site properties…

    So “starting soon”….

    I’ve also expanded swap space to 2 x memory at 6 GB. I doubt that the Linux Swap daemon / facility can handle that as it seems to get confused over about 1.8 GB, but we’ll find out. It looks to me like most of the memory suckage is defunct Java instances, so it may be that Linux will just happily roll them to swap and not notice a thing… But it is likely to take a few days to find out. The memory leakage isn’t that big and it takes a long time. 13.4% of 2 GB memory per Java thread.

    I’ve decided to just keep the web pages n a separate partition and mount it at BOTH locations. In /var and in ~ems. That ought to mean whichever the router chooses will work. Crude, but eliminates a potential issue.

    So with that, my DarkNet site ought to be back up at:
    http://73jnlf22dkg2sfcem2zbqjge4xh7nx47q4kefbnuxyh5vifogjrq.b32.i2p/chiefio.wordpress.com/index.html
    (At least that reaches it from my console / browser)

  16. Pinroot says:

    If I put that full link in, I get a 404 error. If I end the link at .i2p, I get an odd looking page with a link to PG’s site, which automatically (5-10 seconds) redirects to PG’s site.

  17. E.M.Smith says:

    Well that was interesting…

    It seems that despite what the user manual page implies, if you download another site, it overlays with the new site.

    So, as of now, “my” Dark Net server is serving up P.G.’s site mirror….

    So either there is some option I need to set, or some trick I need to learn, or it is just that it only really handles one site at a time…

    Well, OK…

    So sometime tomorrow I’ll revert it to MY site backup, and then I’ll try to figure out if there’s a different way to get multiples. Or whatever…

    @Pinroot:

    I suspect it only adjusts links at the end and you may have stepped in mid-PG-copy.

    So thanks for trying it, anyway ;-)

    For now, I’m calling it a night and I’ll worry about the details of httrack options tomoroow… (I have a pristine “my site” backup on disk from before launching the P.G. backup attempt, so it isn’t exactly any risk involved. This is all, after all, just a big experiment…)

  18. Pinroot says:

    @EM – I’m curious, are you running WordPress on your Pi? And if so, how difficult was it to set it all up? When I finally get i2p running, I’d like to run some type of simple site just to document the things I’m doing and possibly get feedback from people, and would like something more than some type of “static” site, so I’m just what was involved in getting a wordpress site to run on your i2p system.

  19. E.M.Smith says:

    @Pinroot:

    I’ve installed WordPress software as a test case on a R. Pi M3 that was NOT on i2p, then did a WordPress extract from this live site, but was then unable to load that copy into the WordPress software I’d installed as it was a GB or 2 over the “2 GB” limit in the loader. Since then I’ve found some references that claim to show how to make the limit larger and get it loaded, but I’ve not done that step.

    Installing WordPress was very easy.

    UN-related to that, I brought up i2p (Currently on a Rock64). It comes equipped with a built in web server (go to ‘hidden services manager panel and click “Start” after Webserver…)

    I then used “httrack” linux command to duplicate this site to the i2p server. BUT it is not running WordPress software (though some web pages still do some things if the html is sufficient).

    If you know html, then the easy thing to do is just start the webserver in i2r and type in the html you want.

  20. Pinroot says:

    Thanks EM. My html is rusty but there are plenty of references available, so that isn’t an issue. I just wanted something more than static pages, so that people could comment/reply and such, and I don’t think my html is sufficient for that :) I should be able to find something that will work on a Pi, I just need to look around.

  21. E.M.Smith says:

    Well, for that I’d install WordPress.
    https://wordpress.org/support/article/how-to-install-wordpress/
    I’ve seen a wordpress site on i2p so it can be done.

  22. Pinroot says:

    I found this: Install Ghost CMS on a Raspberry Pi
    It’s supposed to be lighter than wordpress, but I don’t need or want a lot of bells and whistles, just something simple. There are several other options available. And this (I think you posted this recently, or something similar by the same guy) should also help: How to set up untraceable websites (eepsites) on I2P. It even touches on how to use a different web server than the one built into i2p. The weather here is supposed to be lousy (ice and rain all night tonight, then into tomorrow) and I’ll probably lay out of work if the roads are bad. If the power doesn’t go out, it will give me something to do :)

  23. Pinroot says:

    Well, not sure how I did that… :(

    [Reply: You had /s tag that started the ‘strike out’ and it looks like WordPress auto closed it at the end (as it does with dangling open tags.). I didn’t know ‘s’ was the same as ‘strike’ but looks like it is. I fixed it. -E.M.S.]

  24. E.M.Smith says:

    Interesting. When I launched the mirror of PG’s site it said something like “there’s already an index.html and {something} file, do you want to do an update instead?” and I answered no, figuring it would just leave the other stuff (chiefo.wordpress.com) alone. But it didn’t. It completely replaced everything in that directory…

    root@rock64:/home/ems/.i2p/eepsite/docroot# ls
    backblue.gif  favicon.ico  hts-log.txt  pgtruspace.wordpress.com
    cookies.txt   help         index.html   robots.txt
    fade.gif      hts-cache    lost+found
    

    So, OK, maybe I DID want to do an “update”…

    I’ve got my web server shut down ATM as I’m going to copy off this backup copy of PG’s site, and then restore mine from the save uSD chip copy from just before this experiment.

    Then I think I need to do a web search on “How to host multiple httrack images” or some such… The “greeting panel” says “these are the sites available” implying it is common to have several. Perhaps it IS just as simple as “mirror once, update after”… but I need to validate that before I end up in the land of endless restorations…

    Also, on “Memory Usage”:

    There’s a “graphs” item on the left side menu, click it, you get usage graphs. Mine say it’s using all of 60 MB of memory. So something else is causing this particular system to rack up memory use over time. FWIW, I averaged 209 “peers” overnight with a peek of 350. With use, my “known peers” has slowly climbed to 1980 and it gets started with tunnels almost instantly on launch now. Running for a few days really matters to performance improvements.

    Running for almost 24 hours has 1.6 GB memory used out of 1.95 GB but that’s with 2 browsers open (FireFox for i2p and console, Chromium for Clearnet and this comment). Yet there’s 225 MB rolled out to swap.

    I suspect this is something odd that Linux is doing and maybe I need to inspect the “swappiness” setting and / or how many file buffers it hangs onto after the process using them has exited. For Example: IF I do a complete httrack update of my site again, every one of those files will at some point have a pointer / address on disk held in memory. Default is to hold on to almost all of them until the memory is needed for something else (even if the file was only opened and written once then closed).

    Basically it might not be i2p that’s using the memory but things I’m going AROUND i2p in the set-up…

    So I also need to figure out what memory profile program might be out there…

  25. E.M.Smith says:

    I’ve done several experiments with httrack and so far been singularly unsuccessful at downloading 2 web sites to one target directory. If anyone else cares to pitch in on this, that might be a good thing, as whatever I’m thinking isn’t quite right ;-)

  26. Rienk says:

    @E.M.Smith says:
    18 February 2021 at 5:30 am

    Httrack should have made a base directory. Something like /home/pi/websites. Inside should be 3 files, index.html, backblue.gif and fade.gif plus subdirectories for every website you downloaded.
    Inside each subdirectory those same 3 files plus a hts-log.txt which may have sensitive information (login and password) and any number of subdirectories again. One of which is de website you wanted and that one will have the proper index.html for the site.

    Httrack should have asked for a category and a project name. The category name is just for the websites directory index file. The project name is the one given to the subdirectories. If you don’t have this structure then something went wrong when starting a new project. If on the other hand you do have this structure it may be your webserver needs configuring differently.

    Rienk

  27. E.M.Smith says:

    @Rienk:

    “Httrack should have asked for a category and a project name. ”

    Implies an interactive mode. I’ve been using the straight command line non-interactive imperative command form. Which means I’m on the hook to provide “all the right options”…

    So good clue there. I’ll try it with the -W ‘let the wizard guide you’ option instead ;-)

    There’s also a completely different command, “webhttrack” version, that looks like it is also interactive. https://debian.httrack.com/

    HTTrack Debian Packages

    HTTrack is available on GNU/Linux and Un*x platforms, as a standalone commandline program, or as a GUI (using a regular browser).
    […]

    apt-get update 
    apt-get install webhttrack
    to get the full GUI version, or
    apt-get update
    apt-get install httrack
    

    So I think that’s what I’ll try.

    FWIW, I get the index.html and the 2 gifs along with the two directories (chiefio.wordpress.com and the 2nd test target) but then the download stops. Even with the –mirror flag and -s0 ignore robots flag, and starting from scratch.

    Trying to do one at a time, with –mirror the second one arrives but the first one is deleted. With update set, it doesn’t download the 2nd one.

    It may be a bug in the command line code on ARM chips. I don’t imagine every option has been fully tested on aarch64 given that most code was still armhf just a year or two ago (and Raspbian is still running armhf on aarch64 chips…)

    So I’ll give it a try with the webhttrack and see if anything changes. Heck, I might even boot up my Intel Box Devuan that hasn’t been run in a year or 2 and give it a try. Probably time to do an update / upgrade on it anyway ;-)

    That would sort out CLI vs GUI and ARM vs x86 as potential issues anyway… Removing a false issue is as valuable as finding a real one some times, so either answer is useful.

  28. E.M.Smith says:

    Well, that was interesting…

    It looks like I’m right now actively doing a download of 2 different websites. They are going into a different place than the command line version (~ems/websites/{projectname}) where the first part is the default set in a config file that I don’t have (so must be defaults) and the “projectname” is something you pick at time of launch of the GUI copier.

    Installing webhttracker puts TWO things in your “Applications/internet” menu on Linux. One at the top of the list just launches your ‘default browser’ on those saved web sites so you can look at them “Browse Mirrored Websites” (which threw me for a few tries as I noticed “the new thing” and didn’t look all the way down the list…)

    At the BOTTOM of the list, it also added “WebHTTrack Website Copier” which also launches your default browser BUT does it with their web app open to drive a site copy process. Which worked. Eventually…

    Issues & Lessons Learned:

    First off, it wanted FireFox which is set up to run against the i2p proxy server that isn’t a good idea for downloading clearnet sites. I opened Chromium and told IT to be the “default browser”, and again FFox was chosen. Checking, they BOTH now claim to be the “Default Browser”… so some kind of food fight between the browsers over who’s on top seems to be happening. OK, I just shut off proxy services (Advanced / Network) in preferences and let it use FFox.

    And it proceeded to duplicate almost nothing. Reliably.

    After a fiddle of several other parameters that may not matter, I realized that the way the GUI asks for site names, they end up http://foo while these are HTTPS:/foo so had to manually insert the “S” in the approprate (3rd or 4th?) GUI screen where it displays the URLs in an editable window.

    Other things changed? I set “robots.txt” to the middle value between “ignore” and “accept”. Something like “Ignore other than Mumble”… This is hiding under the “Spyders” options menu… It likely matters to tell it to ignore robots.txt as that was needed for the ‘longhand’ version.

    I also removed “footer” text and cut “httrack” out of the “browser” type text. Sort of like described here:
    https://forum.httrack.com/readmsg/33697/1129/index.html

    Subject: Re: How to completely ignore ‘robots.txt’?
    Author: Daniel
    Date: 01/11/2015 22:18

    I found a way to bypass the robots.txt from one popular website (the wordpress
    codex site).
    Here’s what I did: In the spiders category, switch it to “accept robots except
    for filters” (something like that, I´m using the German version)
    Then change the Browser ID to Java (something else might work, too but I
    haven´t tested it). Turn off sending a HTML footer and make sure under
    connections you use a value lower than 8 because many websites even ban ips
    which open up more than 8 connections. So 1-4 is a good value here. Hope this
    helps. I´m really glad I could finally bypass the wordpress ban of httrack.

    Also in that “project” folder a whole lot of other URLs are getting dribs and drabs of stuff, so this command is clearly doing a more complete job of collecting odd little ancillary things that normally come from ‘some other site’ where my simple mirror didn’t get those bits. Like:

    ems@OdroidN2:~/websites/TestingCopy$ ls
    0.gravatar.com                            en.wikipedia.org                  s.pubmine.com
    1.gravatar.com                            fade.gif                          s0.wp.com
    2.gravatar.com                            faq.files.wordpress.com           s1.wp.com
    a.teads.tv                                fastlane.rubiconproject.com       s2.wp.com
    aax.amazon-adsystem.com                   gads.pubmatic.com                 sciencespeak.com
    ads.pubmatic.com                          gmpg.org                          secure.gravatar.com
    adserver-us.adtech.advertising.com        googleads.g.doubleclick.net       smartech.gatech.edu
    allnewspipeline.com                       gum.criteo.com                    static.criteo.net
    apex.go.sonobi.com                        hb-api.omnitagjs.com              stats.wp.com
    [...]
    

    I’d wondered when I’d launched a browser against the i2p copy of my site and saw it polling “s0.wp.com” about why it was doing that… non-local file copy needed… so looks like this fixed that.

    As they (several times) go out of their way to state that the GUI version is just a front end to the same back end HTTRack command, I need to capture whatever command they are sending to find out just what all flags are being set. (Either that, or study the manual page a LOT longer ;-)

    For now I’m reasonably happy that there’s a solution. If nothing else I can just stick with the Browser GUI method (though that means the occasional update is a manual launch thing…). Eventually it would be nice to do this all automated, but it is now a lower priority thing to work out.

    So: H/T. Rienk! for pointing me in a direction toward success.

    Now it’s all just “polish and improve, then document the process.”

    (Oh, and BethTheSurf: I used your site as a testing target, so now I’ve got an offline copy of it ;-) Hope you don’t mind…)

  29. Rienk says:

    @ E.M.Smith
    18 February 2021 at 4:59 pm

    Indeed, I do use webhttrack on the pi (raspbian). It runs in the browser and it does work. I just ran across one oddity though, you do need an explicit web address. “https://chiefio.wordpress.com” will work but “chiefio.wordpress.com” does not. I have not come across that on the windows version. And it could be something as silly as chromium hiding that part of the address.

    Apart from setting robots.txt to no I’ve also set the user agent string to something more modern than “mozilla 4.5 something something widows 98”.

    Rienk

  30. E.M.Smith says:

    I had to kill this last test run as it was busy downloading 1 TB+ of Wikipedia into my 1 TB partition with all the other stuff. I need to tell it not to go so deep down links ;-)

    yeah, browsers hiding the http vs https vs whatever is an annoyance…

    But at least I’m past the hump of “can’t download more than one”. Now I just need to figure out the CLI way ;-)

  31. E.M.Smith says:

    Well, looks like the “scuttlebutt” suite of social media software are out:
    https://scuttlebutt.nz/docs/code-of-conduct/

    Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

    Just up stream of that is more “touchy feely” verbiage that indicates “on board” with loony lefty biases.

    In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.

    This if from the “sounds good in principle / ends in the Hateful Left banning anything right of Stalin” problem.

    As soon as you allow banning and hold anything that makes anyone unhappy to be a justification for the banning, The Left finds everything not hard Alt-Left to be making them unhappy…

    Unfortunately, one must be willing to accept “hateful” speech if you would have “free” speech as otherwise someone will find everything “hateful”…

    The Scuttlebutt “products” include: Planetary, Patchbay, Patchfox, Patchwork, and Manyverse (Actually end node apps).

    So not really a free and open P2P product. Also has some IP tracking information available in the connecting / interacting, so a potential for Doxing exists with a bit of work.

    With that, moving on…

Anything to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.