Raspberry Pi GIStemp – It Compiled

I’ve not done any testing yet, just a first cut “unpack and compile”.

The good news is that it did, in fact, compile. And STEP0 ran.

I had the same two “Makefile” issues from the port to Centos (Red Hat for the Enterprise). No surprise as it was an unpack of the same tar file archive with the same two issues in it. A bogus “:q” from a botched exit from the editor in one source file, and the need to remove a trace directive from the compile statement on another. After that, it compiled fine with one warning.

That warning might be a significant bug in the GIStemp code. I’ll need to investigate it further. I don’t remember if any of the other compilations found that type mis-match. The importance of it is still TBD and might be nothing.

Here’s the captures on what I did:

Install FORTRAN

Whereas in the Centos install, FORTRAN is part of “build-essentials”, it was not included in the R.Pi version. That required a specific call out of gfortan as a package. Here’s the call to “apt-get install gfortran” and the result.

root@RaPiM2:/WD/home/BUPSfromGTemp# apt-get install gfortran

Reading package lists… Done
Building dependency tree
Reading state information… Done
The following extra packages will be installed:
gfortran-4.6
Suggested packages:
gfortran-doc gfortran-4.6-doc libgfortran3-dbg
The following NEW packages will be installed:
gfortran gfortran-4.6
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 4,834 kB of archives.
After this operation, 12.0 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://mirrordirector.raspbian.org/raspbian/ wheezy/main gfortran armhf 4:4.6.3-8 [1,134 B]
Get:2 http://mirrordirector.raspbian.org/raspbian/ wheezy/main gfortran-4.6 armhf 4.6.3-14+rpi1 [4,833 kB]
Fetched 4,834 kB in 34s (140 kB/s)
Selecting previously unselected package gfortran-4.6.
(Reading database … 89899 files and directories currently installed.)
Unpacking gfortran-4.6 (from …/gfortran-4.6_4.6.3-14+rpi1_armhf.deb) …
Selecting previously unselected package gfortran.
Unpacking gfortran (from …/gfortran_4%3a4.6.3-8_armhf.deb) …
Processing triggers for man-db …
Setting up gfortran-4.6 (4.6.3-14+rpi1) …
Setting up gfortran (4:4.6.3-8) …
update-alternatives: using /usr/bin/gfortran to provide /usr/bin/f95 (f95) in auto mode

root@RaPiM2:/WD/home/BUPSfromGTemp# which gfortran
/usr/bin/gfortran

So I’ll need to add that “apt-get install gfortran” to my Raspberry Pi software build script.

Here’s the change to the top of the Makefile to tell it to use gfortran instead of the FORTRAN compilers of old:

FC=/usr/bin/gfortran
FC95=/usr/bin/gfortran
FC77=/usr/bin/gfortran

#FC=/usr/local/bin/g95
#FC95=/usr/local/bin/g95
#FC77=/usr/bin/f77
SRCDIR=.
BINDIR=../bin

I inserted “script variables” to use gfortran for all three of “generic FORTRAN”, “FORTRAN 95”, and “FORTRAN 77” compilers. Commenting out the older assignments that used g95 for the first two, and a real f77 for the third. (gfortran being smart enough to do F77 things with F77 code). But at least, due to having made that Makefile, I didn’t have to edit every single script that their version used where each program was complied anew each time it was run, so EVERY run script had a call out to the compiler to use… Really crappy “technique”…

Now with gfortran installed and located as in /usr/bin/gfortran, I went to the GIStemp src directory (remember that I moved sources to one directory and made a “Makefile” as opposed to leaving them scattered all over the place in run-scripts as built). At that point, the two edits were done and I typed “make”.

Here’s the errors that showed the two things I had to fix. Yes, the compiler nicely tells you what you screwed up:

/usr/bin/gfortran -o ../bin/toANNanom.exe ./toANNanom.f
./toANNanom.f:40.1:

:q
1
Error: Non-numeric character in statement label at (1)
./toANNanom.f:40.1:

:q
1

In the program “toANNanom.f” when last I’d edited it back in about 2010, I’d tried to leave the “vi” (visual editor) and not realized I was in “insert” mode. That “:q” is the vi command to “exit”. So I’d exited from edit mode, and then saved and exited the program, but left that artifact inserted. OK. Took it out. (And yes, one then needs to go regression test the code against earlier copies to make sure nothing important got messed up).

The next run found the trace directive in the Makefile and didn’t like it.

pi@RaPiM2 /WD/home/BUPSfromGTemp/gistemp/src $ make
/usr/bin/gfortran -o ../bin/sorts.exe ./sorts.f
/usr/bin/gfortran -o ../bin/antarc_comb.exe ./antarc_comb.f
/usr/bin/gfortran -o ../bin/cmb2.ushcn.v2.exe ./cmb2.ushcn.v2.f
/usr/bin/gfortran -o ../bin/cmb.hohenp.v2.exe ./cmb.hohenp.v2.f
/usr/bin/gfortran -o ../bin/dump_old.exe ./dump_old.f
/usr/bin/gfortran -o ../bin/hohp_to_v2.exe ./hohp_to_v2.f
/usr/bin/gfortran -o ../bin/USHCN2v2.exe ./USHCN2v2.f
#echo “USHCN2v2.f is compiler senstive at run time. 1 digit rounding variation
/usr/bin/gfortran -o ../bin/dif.ushcn.ghcn.exe ./dif.ushcn.ghcn.f
#echo “Still a custom makefile inside STEP1”
/usr/bin/gfortran -o ../bin/flags.exe ./flags.f
/usr/bin/gfortran -o ../bin/padjust.exe ./padjust.f
/usr/bin/gfortran -o ../bin/toANNanom.exe ./toANNanom.f
/usr/bin/gfortran -o ../bin/PApars.exe ./PApars.f ./tr2.f ./t2fit.f
/usr/bin/gfortran -o ../bin/split_binary.exe ./split_binary.f
/usr/bin/gfortran -o ../bin/text_to_binary.exe ./text_to_binary.f
/usr/bin/gfortran -o ../bin/trim_binary.exe ./trim_binary.f
/usr/bin/gfortran -o ../bin/invnt.exe ./invnt.f
/usr/bin/gfortran -o ../bin/trimSBBX.exe ./trimSBBX.f
/usr/bin/gfortran -o ../bin/toSBBXgrid.exe ./toSBBXgrid.f
/usr/bin/gfortran -o ../bin/zonav.exe ./zonav.f
/usr/bin/gfortran -o ../bin/annzon.exe ./annzon.f
/usr/bin/gfortran -o ../bin/convert1.HadR2_mod4.exe ./convert1.HadR2_mod4.f
/usr/bin/gfortran -o ../bin/convert.HadR2_mod4.upto15full_yrs.exe ./convert.HadR2_mod4.upto15full_yrs.f
/usr/bin/gfortran -ftrace=full -o ../bin/SBBXotoBX.exe ./SBBXotoBX.f
f951: error: unrecognized command line option ‘-ftrace=full’
Makefile:91: recipe for target ‘SBBXotoBX’ failed
make: *** [SBBXotoBX] Error 1
pi@RaPiM2 /WD/home/BUPSfromGTemp/gistemp/src $ vi Makefile

I have a couple of notes to myself about STEP1 having it’s own makefile that I need to go set up and test ( it uses Python for some special things and needs some more complicated set up) and that USHCN2V2.f has a rounding error sensitivity to what compiler is used (so needs specific testing / validation).

Then, down near the bottom, you can see that it complains about the -ftrace=full directive. OK, some compiler flags are different for doing traces. For now I just pulled that trace out. I don’t remember why I wanted to trace it 5 years ago anyway ;-) Then the compile ran to completion:

pi@RaPiM2 /WD/home/BUPSfromGTemp/gistemp/src $ make
/usr/bin/gfortran -o ../bin/sorts.exe ./sorts.f
/usr/bin/gfortran -o ../bin/antarc_comb.exe ./antarc_comb.f
/usr/bin/gfortran -o ../bin/cmb2.ushcn.v2.exe ./cmb2.ushcn.v2.f
/usr/bin/gfortran -o ../bin/cmb.hohenp.v2.exe ./cmb.hohenp.v2.f
/usr/bin/gfortran -o ../bin/dump_old.exe ./dump_old.f
/usr/bin/gfortran -o ../bin/hohp_to_v2.exe ./hohp_to_v2.f
/usr/bin/gfortran -o ../bin/USHCN2v2.exe ./USHCN2v2.f
#echo “USHCN2v2.f is compiler senstive at run time. 1 digit rounding variation
/usr/bin/gfortran -o ../bin/dif.ushcn.ghcn.exe ./dif.ushcn.ghcn.f
#echo “Still a custom makefile inside STEP1”
/usr/bin/gfortran -o ../bin/flags.exe ./flags.f
/usr/bin/gfortran -o ../bin/padjust.exe ./padjust.f
/usr/bin/gfortran -o ../bin/toANNanom.exe ./toANNanom.f
/usr/bin/gfortran -o ../bin/PApars.exe ./PApars.f ./tr2.f ./t2fit.f
/usr/bin/gfortran -o ../bin/split_binary.exe ./split_binary.f
/usr/bin/gfortran -o ../bin/text_to_binary.exe ./text_to_binary.f
/usr/bin/gfortran -o ../bin/trim_binary.exe ./trim_binary.f
/usr/bin/gfortran -o ../bin/invnt.exe ./invnt.f
/usr/bin/gfortran -o ../bin/trimSBBX.exe ./trimSBBX.f
/usr/bin/gfortran -o ../bin/toSBBXgrid.exe ./toSBBXgrid.f
/usr/bin/gfortran -o ../bin/zonav.exe ./zonav.f
/usr/bin/gfortran -o ../bin/annzon.exe ./annzon.f
/usr/bin/gfortran -o ../bin/convert1.HadR2_mod4.exe ./convert1.HadR2_mod4.f
/usr/bin/gfortran -o ../bin/convert.HadR2_mod4.upto15full_yrs.exe ./convert.HadR2_mod4.upto15full_yrs.f
/usr/bin/gfortran -o ../bin/SBBXotoBX.exe ./SBBXotoBX.f
./SBBXotoBX.f:181.39:

CALL SREAD(10,MNOW,AVG(I1TIN,NC),XTRL,MNEW)
1
Warning: Type mismatch in argument ‘itrl’ at (1); passed REAL(4) to INTEGER(4)

Here’s the relevant part of the Makefile:

all: inputsort step0 step1 step2 step3 step4_5

step0 : antarc cmb2 cmbho dumpold hotov2 u2v2 dif.ushcn.ghcn

step1:
#echo “Still a custom makefile inside STEP1”

step2: flags padjust toANNanom PApars split_binary text_to_binary trim_binary invnt

step3: trimSBBX toSBBXgrid zonav annzon

step4_5: convert1.HadR2_mod4 convert.HadR2_mod4.upto15full_yrs SBBXotoBX

“Make”ing all does the input sort, then steps in order ending with step4_5. Further down, we see that step4_5 includes three parts, the last of which was SBBXotoBX that complained about a type mis-match with a warning. So it got through the compiles OK.

Since sometimes it is OK to pack a real into an int, and sometimes it is an error, I’ll need to get very familiar with that code to see which it is. As that’s part of the very last grid box portion, I’m not likely to bother.

This is the 5 year old version of GIStemp. I mostly wanted to compile it just to see if it would, generally, go. I’m most likely to get the newer version (that uses different input data now too) and port it first before sinking too much more time into this old version.

At some much later time, I might want to compare the two. That would be a good time to come back to this potential error in their old code.

Trial Run of STEP0

I also did a trial run of STEP0. Turns out that it doesn’t like [[ in shell scripts even if #!/bin/sh is called out. I changed the shell scripts to have “#!/bin/bash” and everything then worked as expected. Oh, and I also had to change “tail +100” to “tail -n 100” in the script as the options are different (and I ought to check that those are really the same between the two versions… it’s been a while since I used each version of “tail” – that gives the last set of lines in a file.)

Most of the time, it “pegs” one core on the FORTRAN. Here’s a grab of “top” showing USHCN2v2.exe at 99.8%

Top - 06:56:37 up  9:26,  4 users,  load average: 1.30, 0.92, 0.81
Tasks: 148 total,   2 running, 146 sleeping,   0 stopped,   0 zombie
%Cpu(s): 25.7 us,  0.6 sy,  0.0 ni, 73.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    949408 total,   928900 used,    20508 free,    16272 buffers
KiB Swap:  3174392 total,       12 used,  3174380 free,   481096 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND            
16388 pi        20   0  2964  532  464 R  99.8  0.1   0:17.22 USHCN2v2.exe       
14405 pi        20   0  588m 289m  41m S   4.6 31.3  55:55.12 iceape-bin         
12141 root      20   0 66284  39m 9132 S   0.7  4.3   8:09.38 Xorg               
15272 pi        20   0  4676 2460 2076 R   0.7  0.3   0:16.34 top                
12243 pi        20   0  106m  16m  13m S   0.3  1.8   0:48.41 lxterminal         
14518 mysql     20   0  308m  34m 7844 S   0.3  3.8   0:08.20 mysqld             

Most of the memory used is for stale buffers. The performance tool showed about 375 MB actually used by everything at this point.

And here is the result of the run on STEP0:

pi@RaPiM2 /WD/home/BUPSfromGTemp/gistemp/STEP0 $ time ./do_comb_step0.sh v2.mean
Clear work_files directory? (Y/N) y
Bringing Antarctic tables closer to input_files/v2.mean format
collecting surface station data
… and autom. weather stn data
… and australian data
replacing ‘-‘ by -999.9, blanks are left alone at this stage
adding extra Antarctica station data to input_files/v2.mean
created v2.meanx from v2_antarct.dat and input_files/v2.mean
removing pre-1880 data:

GHCN data:

removing data before year 1880.00000
created v2.meany from v2.meanx

replacing USHCN station data in v2.mean by USHCN_noFIL data (Tobs+maxmin adj+SHAPadj+noFIL)
reformat USHCN to v2.mean format
extracting FILIN data

-rw-r–r– 1 pi pi 20924805 Jul 29 06:56 hcn_doe_mean_data_fil
-rw-r–r– 1 pi pi 81574970 Feb 17 2010 input_files/hcn_doe_mean_data

getting inventory data for v2-IDs

-rw-r–r– 1 pi pi 32967 Sep 16 1998 input_files/ushcn.tbl

After the sort of ushcn.tbl into ID_US_G

Doing ../bin/USHCN2v2.exe the ***OLD*** USHCN.v2 Version

USHCN data end in 2007
finding offset caused by adjustments
extracting US data from GHCN set
removing data before year 1980.00000
getting USHCN data:
-rw-r–r– 1 pi pi 11111793 Jul 29 06:56 USHCN.v2.mean_noFIL
-rw-r–r– 1 pi pi 10189487 Jul 29 06:57 xxx
doing dump_old.exe
removing data before year 1880.00000
-rw-r–r– 1 pi pi 10124653 Jul 29 06:57 yyy
Sorting into USHCN.v2.mean_noFIL
-rw-r–r– 1 pi pi 10124653 Jul 29 06:57 USHCN.v2.mean_noFIL
done with ushcn

created ushcn-ghcn_offset_noFIL

Doing cmb2.ushcn.v2.exe

created v2.meanz

replacing Hohenspeissenberg data in v2.mean by more complete data (priv.comm.)
disregard pre-1880 data:

At Cleanup

created v2.mean_comb

-rw-r–r– 1 pi pi 44774884 Jul 29 06:59 to_next_step/v2.mean_comb

move this file from to_next_step/. to ../STEP1/to_next_step/.
Copy the file to_next_step/v2.mean_comb to ../STEP1/to_next_step/v2.mean_comb? (Y/N) y

and execute in the STEP1 directory the command:
do_comb_step1.sh v2.mean_comb

real 4m48.070s
user 4m5.370s
sys 0m10.660s
pi@RaPiM2 /WD/home/BUPSfromGTemp/gistemp/STEP0 $

At the very bottom is the result of the “time” command that tells you elapsed time and how much CPU time was used. So this step took 4 minutes 48 seconds to run, used 4 minutes 5 seconds of user CPU time and 10 seconds of system CPU time. Essentially pegging one core for 5 minutes. Not bad. IIRC, each step is about the same run time. So this would be in keeping with the 20 to 30 minutes end-to-end I think I remember from before, on the old x86 CPU box with faster disks.

This was on a USB disk with EXT3 partitions and I added a swap partition which you can see in the “top” listing as well. The R.Pi uses a 1 GB swap file, but swapping to SD card is just soo wrong, so I added a real partition… that never gets used… oh well…).

Just in case anyone else wants to know how to do that, here’s the /etc/fstab lines:

/dev/sda3       swap            swap    sw,pri=1024       0       0
/dev/sda4       /WD/home        ext3    defaults          0       1

That “pri=1024” is the priority for that swap partition. It ranges from zero (last) to 30,000 ish (first). I just set it to 1k so it comes ahead of the swap file set by default. That ought to direct any swapping first to my real USB disk, only when that 2 GB get used up, spill over to their 1 GB file. But if I’m ever using 1 GB of memory and 2+GB of swap, I’ll have other issues to worry about ;-)

Also, since GIStemp repeatedly writes 10 to 40 MB files, I decided to not burn that on my expensive fast 64 Gig SD card and instead put it on a real USB disk that doesn’t care about write cycles.

In Conclusion

That’s basically it. Initial unpack and install done. Compiler check and install done. Make done, and make file updated / bugs ironed out. STEP0 first run completed. Next step to do the STEP1 “make” that is “special” and try doing a couple of runs end to end. Then download a fresh copy of the source, and see how much has changed in 1/2 decade.

Don’t know when I’ll get all that done. At some point I’m supposed to go looking for a real job to pay the rent. And the garden calls my name when the weather is nice. Then there are the 1001 other interesting things happening to / in the world. But I’m not going to just let GIStemp lay there. I’ll get to it. Maybe mañana ;-)

I do find it rather fun that GIStemp compiles and runs at a reasonably fast clip on a $60 computer. Kind of makes you wonder what they need all the super duper new expensive computers for…

Subscribe to feed

Advertisement