GHCN v4 Adding Statistics & Anomaly Tables

This is just an intermediate layout for these tables. They are Version 4 of GHCN specific. Eventually I would like to make replacements that will hold the various versions (v1, v2, v3…) but that will require a bit more “integration” of the difference between the versions.

So for now, to shake out the bugs and make sure I’ve got the right ideas, I’ve made these versions of the tables; and scripts to load them.

Statistics Table

First off, the statistics. I’ve added a standard deviation field by Station for each month.

Here’s the table layout:

CREATE TABLE mstats4 (
    stnID CHAR(11) NOT NULL,
    month CHAR(4) NOT NULL,
    mean  FLOAT NOT NULL,
    big   DECIMAL(7,2) NOT NULL,
    small DECIMAL(7,2) NOT NULL,
    num   INTEGER NOT NULL,
    trang FLOAT NOT NULL,
    stdev FLOAT NOT NULL,
    pctd  FLOAT,
    pctm  FLOAT,
    PRIMARY KEY (stnID,month)
    ) ;

I’ve still not done the percent missing and percent data statistics so those two fields at the bottom are still empty.

Here’s the script that loads the table:

chiefio@PiM3Devuan2:~/SQL/bin$ cat lmstats4.sql 
INSERT INTO  mstats4 (stnID,month,mean,big,small,num,trang,stdev)
SELECT stnID,month,
AVG(deg_C),MAX(deg_C),MIN(deg_C),COUNT(deg_C), 
MAX(deg_C)-MIN(deg_C), STDDEV(deg_C)
FROM temps4 
WHERE deg_C>-90 
GROUP BY stnID,month;

When run, this took a fairly long time to complete. It has one core pegged all the time, so clearly MariaDB doesn’t do as much multi-core as it might. It is not disk limited on the Pi M3. This strongly implies a system with a single faster core would be a lot faster. This implies boards like the Odroid XU4 or the RockPro64 with very fast cores would make this a lot faster.

Here’s the run result:

MariaDB [temps]> source tables/mstats4
Query OK, 0 rows affected (0.57 sec)
[...]
MariaDB [temps]> source tables/anom4
Query OK, 0 rows affected (0.31 sec)
[...]
MariaDB [temps]> source bin/lmstats4.sql
Query OK, 328327 rows affected (29 min 32.63 sec)
Records: 328327  Duplicates: 0  Warnings: 0

MariaDB [temps]> 

So just about 1/2 an hour to create the statistics and load them… As this only needs to be done once, it can be done on the Pi M3, but at the 1/2 hour point I’m starting to look longingly at my fast SBCs ;-)

Anomaly

Next, the anomaly table:

This one was (and still is) a challenge. It, too, pegs one core at 100% use. After running for about 24 hours, it crapped out with an error message. I think this was taking so long as it has a JOIN on a non-index field, so about 300,000 records to match against 240 with most likely repeated linear searches and lots of “is it equal?” tests… While it was running I figured that was likely the issue but just left it to run to find out what would come of it.

First the table schema:

CREATE TABLE anom4 (
    stnID CHAR(11) NOT NULL,
    abrev CHAR(2) NOT NULL,
    region CHAR(1) NOT NULL,
    cnum CHAR(3) NOT NULL,
    year CHAR(4) NOT NULL,
    month CHAR(4) NOT NULL,
    deg_C DECIMAL(5,2) NOT NULL,
    PRIMARY KEY (stnID,year,month)
    ) ;

Pretty basic as there’s a lot of fields in v3 that are dropped in v4. I add the region (continent) number and the country number (cnum) from v3 with the intent to enable comparisons of like things with v3. That has an issue with Russia and Kazakhstan… which I’ll show down below.

Then the code:

INSERT INTO  anom4 (stnID,abrev,region,cnum,year,month,deg_C)
SELECT T.stnID,C.abrev,C.region,C.cnum,T.year,T.month,T.deg_C-ST.mean
FROM temps4 AS T
INNER JOIN country AS C
ON T.abrev=C.abrev
INNER JOIN mstats4 AS ST 
ON
        ST.stnID=T.stnID
	AND ST.month=T.month 
WHERE T.deg_C > -90 
;

The problem is that join to the “country” table so I can pick up things like the country number used in version 3 along with the continent number.

MariaDB [temps]> source bin/lanom4.sql
ERROR 1062 (23000) at line 1 in file: 'bin/lanom4.sql': Duplicate entry 'KZ000028676-1891- APR' for key 'PRIMARY'

So I checked where was this duplicate data? (Not remembering yet that I had created it…)

MariaDB [temps]> SELECT stnID, year, month, type FROM temps4
    -> WHERE year=1891 AND month=' APR' and stnID='KZ000028676';
+-------------+------+-------+------+
| stnID       | year | month | type |
+-------------+------+-------+------+
| KZ000028676 | 1891 |  APR  | TAVG |
+-------------+------+-------+------+
1 row in set (0.84 sec)

MariaDB [temps]> SELECT cnum,abrev,region,version,cname FROM country
    -> WHERE abrev='KZ';
+------+-------+--------+---------+------------+
| cnum | abrev | region | version | cname      |
+------+-------+--------+---------+------------+
| 211  | KZ    | 2      | Gv4     | Kazakhstan |
| 625  | KZ    | 6      | Gv4     | Kazakhstan |
+------+-------+--------+---------+------------+
2 rows in set (0.00 sec)

Oh, Yeah… Russia and Kazakhstan were split between Europe and Asia in v3, but only one country in V4, so there are two country numbers assigned to the same abbreviation… Which also means my attempt to make an index on it will fail.

Grumble…

So I need to choose to either just assign all of those two to one continent or the other (and one cnum) or change the abbreviation to something else for 1/2 of the country (and have it not match V4).

Decisions decisions… unscrambling eggs can be so much bother…

I’d intended to just pick up the v3 bits and move on, not needing to get them again, but that match clearly “has issues” due to those two countries.

I think I’ll have a cup-a tea while mulling this over. Most likely I’ll just assign all of both of them to Asia and “move on” with v4 as is for now. Sorting out the individual stations into European / Asian later with different names and abbreviations (like ARussia and EKazakhstan … or something…) Alternatively I can move the attachment of a name and number elsewhere in the process. But somewhere some how I need to be able to match two things that don’t exactly match in order to do direct comparisons.

It isn’t needed for the direct anomaly inspection of just the v4 data, so simply skipping that step would be ‘OK’ for now. Decisions, decidsions…

Subscribe to feed

Advertisements

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in NCDC - GHCN Issues, Tech Bits and tagged , , , , , . Bookmark the permalink.

22 Responses to GHCN v4 Adding Statistics & Anomaly Tables

  1. E.M.Smith says:

    I’ve decided that, for now, I’m just going to assign all of Russia and Kazakhstan to Asia and put a “!” in the second character of their abbreviations for the European part. That will both make them a unique field (so can be a key) and visually distinct so they can’t be confused with some other country in that list of 240 two letter abbreviations.

    With that, I ought to be able to proceed with v4 stuff and “sometime” later come back to sorting the European stations into R! and K! Europe…

    So here’s the changed “country” schema:

    CREATE TABLE country (
        cnum CHAR(3) NOT NULL,
        abrev   CHAR(2),
        region CHAR(1) NOT NULL,
        cname  CHAR(60),
        version CHAR(5),
        ascension CHAR(10),
        PRIMARY KEY(abrev)
        ) ;
    

    And then the re-load with the two changed entries (in the editor I just replaced the 2nd CHAR of the 6xx RU and KZ to be a “!” so they are R! and K! in the input file).:

    MariaDB [temps]> source bin/LC4.sql
    Query OK, 240 rows affected, 1 warning (0.99 sec)    
    Records: 240  Deleted: 0  Skipped: 0  Warnings: 1
    
    MariaDB [temps]> describe country;
    +-----------+----------+------+-----+---------+-------+
    | Field     | Type     | Null | Key | Default | Extra |
    +-----------+----------+------+-----+---------+-------+
    | cnum      | char(3)  | NO   |     | NULL    |       |
    | abrev     | char(2)  | NO   | PRI | NULL    |       |
    | region    | char(1)  | NO   |     | NULL    |       |
    | cname     | char(60) | YES  |     | NULL    |       |
    | version   | char(5)  | YES  |     | NULL    |       |
    | ascension | char(10) | YES  |     | NULL    |       |
    +-----------+----------+------+-----+---------+-------+
    6 rows in set (0.22 sec)
    
    MariaDB [temps]> 
    

    So even with abrev as the primary key (now) it loaded fine.

    However… attempting to add an index to the temps4 file craps out with this message:

    MariaDB [temps]> CREATE INDEX abrevx ON temps4(abrev);
    ERROR 1034 (HY000): Incorrect key file for table 'temps4'; try to repair it
    MariaDB [temps]>       
    

    Searching on it lead to the assertion it was likely an out of disk error. On /tmp. But I have /tmp on a 1 GB dedicated disk partition. A whole GB just to create an index? Of temp space?

    I ran

    while true
    do
    df / /tmp
    echo
    sleep 10
    done
    

    In one terminal window while the CREATE INDEX was running… Along with finding I’m close on the system space on / (probably all the added programs I’d got on this system now) I also found that right on queue when it ran to full on /tmp the program crapped and then removed the evidence from /tmp

    Filesystem     1K-blocks    Used Available Use% Mounted on
    /dev/root        4062912 3718252    134948  97% /
    /dev/sda9        1038336  591404    446932  57% /tmp
    
    Filesystem     1K-blocks    Used Available Use% Mounted on
    /dev/root        4062912 3718252    134948  97% /
    /dev/sda9        1038336  689708    348628  67% /tmp
    
    Filesystem     1K-blocks    Used Available Use% Mounted on
    /dev/root        4062912 3718252    134948  97% /
    /dev/sda9        1038336  820780    217556  80% /tmp
    
    Filesystem     1K-blocks    Used Available Use% Mounted on
    /dev/root        4062912 3718252    134948  97% /
    /dev/sda9        1038336  902700    135636  87% /tmp
    
    Filesystem     1K-blocks    Used Available Use% Mounted on
    /dev/root        4062912 3718252    134948  97% /
    /dev/sda9        1038336 1013376     24960  98% /tmp
    
    Filesystem     1K-blocks    Used Available Use% Mounted on
    /dev/root        4062912 3718252    134948  97% /
    /dev/sda9        1038336   34452   1003884   4% /tmp
    
    ^C
    root@PiM3Devuan2:/SG500# 
    

    So now I need to make /tmp a few GB…

    So just an FYI for anyone playing with v4 and adding indexes, you need a BIG /tmp file system.

  2. E.M.Smith says:

    And 15 minutes of processing later, I have an index.

    Unfortunately, Chromium locked up the system as I was ready to paste in the result and that text did not survived the forced reboot. I’ve had Chromium do that a few times now. Lock up the system. Either when I’d click in the comment box while it was still loading the page, or just if left up overnight. Usually with “waiting for cache” displayed in the lower left corner. Hopefully that will be fixed sooner than later…

    I’d also captured the usage on /tmp, but now you’ll just have to take my word for it. It wasn’t much over 1 GB, just enough to exceed my prior /tmp partition. It got to about 1.03 GB.

    So now, with my 8 GB /tmp partition dedicated… I can get back to trying that anomaly load step…

    AFTER shutting down Chromium…

  3. E.M.Smith says:

    And with index fields, it all went mch quicker on the anomalies:

    MariaDB [temps]> source bin/lanom4.sql
    Query OK, 15448107 rows affected, 65535 warnings (19 min 20.46 sec)
    Records: 15448107  Duplicates: 0  Warnings: 15444656
    

    But with some warnings…

    | Note  | 1265 | Data truncated for column 'deg_C' at row 1 |
    | Note  | 1265 | Data truncated for column 'deg_C' at row 1 |
    | Note  | 1265 | Data truncated for column 'deg_C' at row 1 |
    | Note  | 1265 | Data truncated for column 'deg_C' at row 1 |
    | Note  | 1265 | Data truncated for column 'deg_C' at row 1 |
    | Note  | 1265 | Data truncated for column 'deg_C' at row 1 |
    +-------+------+--------------------------------------------+
    64 rows in set (0.01 sec)
    
    MariaDB [temps]> 
    

    Oh Dear…

    Oh, but wait, that’s a calculated field and I’m stuffing it into a DECIMAL 5.2 so of course it gets truncated. OK, at some point I need to make it a FLOAT instead…

    As I skip the Missing Data Flag records, but subtracting that number of records from the count in temps4 we ought to get a rough idea of the number of -99.99 records:

    MariaDB [temps]> SELECT COUNT(deg_C) FROM temps4;
    +--------------+
    | COUNT(deg_C) |
    +--------------+
    |     16809258 |
    +--------------+
    1 row in set (2 min 22.97 sec)
    

    So 16,809,258 – 15,448,107 = 1.361 Million months with a missing data flag. About 8%.

    A quick spot check looks “about right”:

    MariaDB [temps]> SELECT stnID,abrev,region,cnum,year,month,deg_C
        -> FROM anom4 WHERE abrev='KZ' and year=2000;
    +-------------+-------+--------+------+------+-------+-------+
    | stnID       | abrev | region | cnum | year | month | deg_C |
    +-------------+-------+--------+------+------+-------+-------+
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  APR  |  4.90 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  AUG  |  0.40 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  DEC  |  1.30 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  FEB  |  5.35 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  JAN  |  1.93 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  MAR  |  3.48 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  MAY  | -0.95 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  NOV  | -5.13 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 |  OCT  | -1.64 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 | JULY  |  0.51 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 | JUNE  |  1.96 |
    | KZ000028676 | KZ    | 2      | 211  | 2000 | SEPT  | -0.86 |
    

    Interesting that this station had both a hot anomaly Feb at +5 and a cold Nov at -5.

    [...]
    | KZ000028867 | KZ    | 2      | 211  | 2000 | JUNE  |  1.11 |
    | KZ000028867 | KZ    | 2      | 211  | 2000 | SEPT  | -1.44 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 |  APR  |  4.31 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 |  JAN  |  2.38 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 |  MAR  |  3.40 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 |  MAY  | -1.09 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 |  NOV  | -4.37 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 |  OCT  | -2.62 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 | JULY  | -0.40 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 | JUNE  |  1.20 |
    | KZ000028879 | KZ    | 2      | 211  | 2000 | SEPT  | -0.74 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  APR  |  5.51 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  AUG  |  0.61 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  DEC  |  2.27 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  FEB  |  6.31 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  JAN  |  3.66 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  MAR  |  3.23 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  MAY  | -2.31 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  NOV  | -1.15 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 |  OCT  | -1.13 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 | JULY  |  0.18 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 | JUNE  |  1.56 |
    | KZ000028952 | KZ    | 2      | 211  | 2000 | SEPT  | -0.65 |
    | KZ000028966 | KZ    | 2      | 211  | 2000 |  APR  |  3.97 |
    | KZ000028966 | KZ    | 2      | 211  | 2000 |  AUG  |  0.66 |
    | KZ000028966 | KZ    | 2      | 211  | 2000 |  DEC  |  1.23 |
    [...]
    | KZ000034798 | KZ    | 2      | 211  | 2000 |  OCT  | -2.46 |
    | KZ000035085 | KZ    | 2      | 211  | 2000 |  MAR  |  0.28 |
    | KZ000035085 | KZ    | 2      | 211  | 2000 |  NOV  | -5.30 |
    | KZ000035085 | KZ    | 2      | 211  | 2000 |  OCT  | -3.24 |
    | KZ000035085 | KZ    | 2      | 211  | 2000 | JULY  | -0.28 |
    | KZ000035085 | KZ    | 2      | 211  | 2000 | SEPT  | -0.63 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  APR  |  5.59 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  AUG  |  0.75 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  DEC  |  4.73 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  FEB  |  7.93 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  JAN  |  6.04 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  MAR  |  3.72 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  MAY  | -3.85 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  NOV  | -0.27 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 |  OCT  |  0.19 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 | JULY  | -0.08 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 | JUNE  |  0.03 |
    | KZ000035108 | KZ    | 2      | 211  | 2000 | SEPT  | -0.75 |
    | KZ000035173 | KZ    | 2      | 211  | 2000 |  APR  |  3.26 |
    | KZ000035173 | KZ    | 2      | 211  | 2000 |  AUG  |  1.08 |
    | KZ000035173 | KZ    | 2      | 211  | 2000 |  DEC  |  1.68 |
    | KZ000035173 | KZ    | 2      | 211  | 2000 |  FEB  |  3.56 |
    | KZ000035173 | KZ    | 2      | 211  | 2000 |  JAN  |  2.31 |
    | KZ000035173 | KZ    | 2      | 211  | 2000 |  MAR  | -0.31 |
    [...]
    | KZ000038439 | KZ    | 2      | 211  | 2000 | JUNE  |  0.69 |
    | KZ000038439 | KZ    | 2      | 211  | 2000 | SEPT  |  1.13 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  APR  |  4.75 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  AUG  |  1.69 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  DEC  |  3.40 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  FEB  |  7.66 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  JAN  |  6.08 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  MAR  |  4.54 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  MAY  | -3.44 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  NOV  | -1.36 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 |  OCT  | -1.73 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 | JULY  |  1.30 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 | JUNE  | -0.03 |
    | KZXLT317325 | KZ    | 2      | 211  | 2000 | SEPT  | -0.94 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 |  AUG  |  1.20 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 |  DEC  |  3.70 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 |  FEB  |  5.27 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 |  JAN  |  2.88 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 |  MAR  |  2.71 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 |  MAY  | -1.98 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 |  NOV  | -3.31 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 |  OCT  | -2.94 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 | JULY  |  1.46 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 | JUNE  |  1.23 |
    | KZXLT409590 | KZ    | 2      | 211  | 2000 | SEPT  |  0.13 |
    | KZXLT517296 | KZ    | 2      | 211  | 2000 |  DEC  |  2.65 |
    | KZXLT517296 | KZ    | 2      | 211  | 2000 |  MAR  |  2.84 |
    | KZXLT517296 | KZ    | 2      | 211  | 2000 |  MAY  |  1.16 |
    | KZXLT517296 | KZ    | 2      | 211  | 2000 |  NOV  | -4.65 |
    | KZXLT517296 | KZ    | 2      | 211  | 2000 |  OCT  | -2.89 |
    | KZXLT517296 | KZ    | 2      | 211  | 2000 | JUNE  |  0.62 |
    | KZXLT517296 | KZ    | 2      | 211  | 2000 | SEPT  |  1.08 |
    +-------------+-------+--------+------+------+-------+-------+
    780 rows in set (1 min 23.11 sec)
    
    MariaDB [temps]> 
    

    So looks like all the right stuff in all the right fields and an anomaly value that is in a sane range for the volatile Steppes of Kazakhstan.

    With that, I think I can proceed to doing some analysis and reports stuff.

  4. andrewsjp says:

    I believe that you are not doing the standard deviation correctly because of autocorrelation. Here is an article from WattsUpWithThat by Willis Eschenbach that discusses the subject. Worth a look.

  5. E.M.Smith says:

    @Andrewsjp:

    Not seeing the link….

    FWIW, all I’m doing it finding the StdDev of a Given Month for a Given Instrument. I would hope the instrument has a very high “auto-correlation” with itself… So, for example, I will have a StdDev for the SFO Airport Thermometer for ALL of the February Readings. Then I can ask things like how many StdDev is any given Monthly Average of Februaries at SFO away from the MEAN for February at SFO. If, for example, it is 10 Std Dev above the mean, it is likely a jet exhaust was directed at the station a lot that month.

    Similarly, if the range of February data starts out 2 Std Dev and decreases to 1/2 Std Dev over the years, then “something interesting” is going on.

    I suspect the Willis Article is about the Standard Deviation of some large group of thermometers. I’m not doing anything with the Std Dev of a large group of disbursed instruments. I’m not trying to find an “average global temperature” nor a std deviation of the creation of it. I’m looking at individual instruments and individual MONTH statistics (where, for example, August in Sacramento ought to be pretty much the same every August. About 100 F, ranging mostly 90 F to 110 F. (I lived there a lot of decades. It’s always like that in August…) IF that is changing and the statistics for that thermometer in that month are changing, something is happening there.

    I doubt anyone else in the “Climate Science” world is looking at the data in that way. They all seem fixated on “Grid Box” anomaly and averaging a bunch of thermometers together and using artificial “data food product” in stead of real readings from real instruments. In that context lots of auto-correlation issues pop up.

  6. E.M.Smith says:

    I’ve been putting some tables of lat and long and such from v4 in a thread about v3 here:
    https://chiefio.wordpress.com/2019/03/13/anomaly-by-continent-graphs-ghcn-v3-3/#comment-110149

    That’s just where the discussion broke out and I had the v4 data to hand…

    I’m going to put some of that here, instead, as I’ve drifted a bit away from the topic there, and it is more about statistics here on v4 data.

    This is 263 rows (for 263 years) of the average Latitude and average Longitude in Russia. I find it interesting how the station distribution moves… It starts with just one station in the European side of things and then spreads out into Asia, having a bit of a pull back at the end. Can you make an average of Russia out of that migrating herd? I wonders…

    MariaDB [temps]> source bin/LatAve.sql
    +------+---------+----------+---------+
    | year | Avg.LAT | Avg.LONG | Country |
    +------+---------+----------+---------+
    | 1743 |   59.97 |    30.30 | RS      |
    | 1744 |   59.97 |    30.30 | RS      |
    | 1745 |   59.97 |    30.30 | RS      |
    | 1751 |   59.97 |    30.30 | RS      |
    | 1752 |   59.97 |    30.30 | RS      |
    | 1753 |   59.97 |    30.30 | RS      |
    | 1754 |   59.97 |    30.30 | RS      |
    | 1755 |   59.97 |    30.30 | RS      |
    | 1756 |   59.97 |    30.30 | RS      |
    | 1757 |   59.97 |    30.30 | RS      |
    | 1758 |   59.97 |    30.30 | RS      |
    | 1759 |   59.97 |    30.30 | RS      |
    | 1760 |   59.97 |    30.30 | RS      |
    | 1761 |   59.97 |    30.30 | RS      |
    | 1762 |   59.97 |    30.30 | RS      |
    | 1763 |   59.97 |    30.30 | RS      |
    | 1764 |   59.97 |    30.30 | RS      |
    | 1765 |   59.97 |    30.30 | RS      |
    | 1766 |   59.97 |    30.30 | RS      |
    | 1767 |   59.97 |    30.30 | RS      |
    | 1768 |   59.97 |    30.30 | RS      |
    | 1769 |   59.97 |    30.30 | RS      |
    | 1770 |   59.97 |    30.30 | RS      |
    | 1771 |   59.97 |    30.30 | RS      |
    | 1772 |   59.97 |    30.30 | RS      |
    | 1773 |   59.97 |    30.30 | RS      |
    | 1774 |   59.97 |    30.30 | RS      |
    | 1775 |   59.97 |    30.30 | RS      |
    | 1776 |   59.97 |    30.30 | RS      |
    | 1777 |   59.97 |    30.30 | RS      |
    | 1778 |   59.97 |    30.30 | RS      |
    | 1779 |   57.86 |    33.93 | RS      |
    | 1780 |   57.86 |    33.93 | RS      |
    | 1781 |   57.86 |    33.93 | RS      |
    | 1782 |   57.86 |    33.93 | RS      |
    | 1783 |   57.86 |    33.93 | RS      |
    | 1784 |   59.97 |    30.30 | RS      |
    | 1785 |   57.86 |    33.93 | RS      |
    | 1786 |   57.86 |    33.93 | RS      |
    | 1788 |   57.86 |    33.93 | RS      |
    | 1789 |   57.86 |    33.93 | RS      |
    | 1790 |   59.97 |    30.30 | RS      |
    | 1791 |   57.86 |    33.93 | RS      |
    | 1792 |   57.86 |    33.93 | RS      |
    | 1794 |   59.97 |    30.30 | RS      |
    | 1798 |   59.97 |    30.30 | RS      |
    | 1799 |   59.97 |    30.30 | RS      |
    | 1800 |   59.97 |    30.30 | RS      |
    | 1805 |   59.97 |    30.30 | RS      |
    | 1806 |   59.97 |    30.30 | RS      |
    | 1807 |   59.97 |    30.30 | RS      |
    | 1808 |   59.97 |    30.30 | RS      |
    | 1809 |   59.97 |    30.30 | RS      |
    | 1810 |   59.97 |    30.30 | RS      |
    | 1811 |   59.97 |    30.30 | RS      |
    | 1812 |   57.85 |    39.75 | RS      |
    | 1813 |   60.07 |    40.08 | RS      |
    | 1814 |   60.07 |    40.08 | RS      |
    | 1815 |   60.07 |    40.08 | RS      |
    | 1816 |   60.50 |    38.62 | RS      |
    | 1817 |   60.74 |    41.07 | RS      |
    | 1818 |   60.74 |    41.07 | RS      |
    | 1819 |   60.74 |    41.07 | RS      |
    | 1820 |   58.81 |    49.61 | RS      |
    | 1821 |   59.33 |    49.68 | RS      |
    | 1822 |   59.33 |    49.68 | RS      |
    | 1823 |   59.33 |    49.68 | RS      |
    | 1824 |   59.33 |    49.68 | RS      |
    | 1825 |   59.33 |    49.68 | RS      |
    | 1826 |   59.33 |    49.68 | RS      |
    | 1827 |   58.81 |    49.61 | RS      |
    | 1828 |   58.81 |    49.61 | RS      |
    | 1829 |   59.21 |    59.62 | RS      |
    | 1830 |   59.21 |    59.62 | RS      |
    | 1831 |   58.95 |    59.73 | RS      |
    | 1832 |   57.60 |    62.02 | RS      |
    | 1833 |   57.69 |    58.09 | RS      |
    | 1834 |   57.69 |    58.09 | RS      |
    | 1835 |   57.75 |    56.03 | RS      |
    | 1836 |   57.39 |    57.09 | RS      |
    | 1837 |   56.72 |    58.32 | RS      |
    | 1838 |   56.52 |    58.87 | RS      |
    | 1839 |   56.59 |    62.03 | RS      |
    | 1840 |   56.09 |    62.13 | RS      |
    | 1841 |   56.09 |    62.13 | RS      |
    | 1842 |   56.11 |    61.24 | RS      |
    | 1843 |   56.37 |    61.58 | RS      |
    | 1844 |   56.54 |    58.96 | RS      |
    | 1845 |   56.20 |    55.34 | RS      |
    | 1846 |   56.42 |    54.30 | RS      |
    | 1847 |   56.21 |    56.57 | RS      |
    | 1848 |   56.15 |    55.13 | RS      |
    | 1849 |   56.34 |    55.51 | RS      |
    | 1850 |   56.34 |    55.51 | RS      |
    | 1851 |   56.34 |    55.51 | RS      |
    | 1852 |   56.22 |    55.30 | RS      |
    | 1853 |   56.54 |    56.35 | RS      |
    | 1854 |   55.59 |    60.04 | RS      |
    | 1855 |   55.38 |    58.42 | RS      |
    | 1856 |   55.44 |    57.08 | RS      |
    | 1857 |   55.41 |    60.82 | RS      |
    | 1858 |   55.46 |    60.61 | RS      |
    | 1859 |   55.46 |    60.61 | RS      |
    | 1860 |   55.93 |    60.33 | RS      |
    | 1861 |   56.32 |    60.30 | RS      |
    | 1862 |   56.71 |    64.00 | RS      |
    | 1863 |   56.06 |    62.69 | RS      |
    | 1864 |   56.16 |    62.95 | RS      |
    | 1865 |   56.06 |    62.69 | RS      |
    | 1866 |   55.96 |    62.84 | RS      |
    | 1867 |   55.96 |    62.84 | RS      |
    | 1868 |   55.46 |    56.81 | RS      |
    | 1869 |   55.46 |    56.81 | RS      |
    | 1870 |   55.84 |    61.10 | RS      |
    | 1871 |   55.99 |    62.82 | RS      |
    | 1872 |   55.16 |    65.44 | RS      |
    | 1873 |   55.09 |    68.09 | RS      |
    | 1874 |   54.44 |    63.38 | RS      |
    | 1875 |   54.44 |    63.38 | RS      |
    | 1876 |   54.73 |    66.57 | RS      |
    | 1877 |   54.73 |    66.57 | RS      |
    | 1878 |   54.61 |    65.00 | RS      |
    | 1879 |   54.61 |    65.00 | RS      |
    | 1880 |   54.61 |    65.00 | RS      |
    | 1881 |   54.64 |    61.85 | RS      |
    | 1882 |   55.01 |    65.89 | RS      |
    | 1883 |   55.36 |    63.75 | RS      |
    | 1884 |   55.71 |    63.60 | RS      |
    | 1885 |   55.98 |    65.16 | RS      |
    | 1886 |   55.53 |    66.81 | RS      |
    | 1887 |   55.57 |    66.57 | RS      |
    | 1888 |   55.78 |    64.91 | RS      |
    | 1889 |   55.92 |    64.16 | RS      |
    | 1890 |   55.59 |    67.93 | RS      |
    | 1891 |   55.38 |    71.38 | RS      |
    | 1892 |   55.60 |    70.80 | RS      |
    | 1893 |   55.96 |    70.94 | RS      |
    | 1894 |   56.26 |    72.29 | RS      |
    | 1895 |   56.42 |    70.96 | RS      |
    | 1896 |   56.10 |    72.11 | RS      |
    | 1897 |   56.25 |    71.50 | RS      |
    | 1898 |   56.38 |    73.34 | RS      |
    | 1899 |   56.46 |    73.15 | RS      |
    | 1900 |   56.26 |    73.28 | RS      |
    | 1901 |   56.10 |    72.82 | RS      |
    | 1902 |   55.98 |    73.31 | RS      |
    | 1903 |   55.86 |    72.97 | RS      |
    | 1904 |   56.08 |    71.76 | RS      |
    | 1905 |   56.06 |    72.08 | RS      |
    | 1906 |   56.43 |    71.99 | RS      |
    | 1907 |   56.15 |    71.01 | RS      |
    | 1908 |   56.02 |    74.20 | RS      |
    | 1909 |   55.97 |    76.95 | RS      |
    | 1910 |   55.60 |    77.09 | RS      |
    | 1911 |   55.50 |    78.75 | RS      |
    | 1912 |   55.74 |    78.76 | RS      |
    | 1913 |   55.65 |    78.29 | RS      |
    | 1914 |   55.85 |    78.59 | RS      |
    | 1915 |   56.01 |    78.01 | RS      |
    | 1916 |   56.43 |    78.87 | RS      |
    | 1917 |   56.36 |    77.96 | RS      |
    | 1918 |   56.46 |    76.85 | RS      |
    | 1919 |   56.54 |    79.20 | RS      |
    | 1920 |   56.91 |    76.77 | RS      |
    | 1921 |   56.77 |    77.21 | RS      |
    | 1922 |   56.70 |    74.40 | RS      |
    | 1923 |   56.52 |    73.81 | RS      |
    | 1924 |   56.33 |    75.71 | RS      |
    | 1925 |   56.12 |    78.93 | RS      |
    | 1926 |   56.21 |    77.03 | RS      |
    | 1927 |   56.08 |    76.85 | RS      |
    | 1928 |   56.23 |    75.56 | RS      |
    | 1929 |   56.31 |    76.34 | RS      |
    | 1930 |   56.48 |    76.97 | RS      |
    | 1931 |   56.54 |    76.42 | RS      |
    | 1932 |   57.15 |    76.12 | RS      |
    | 1933 |   57.22 |    78.09 | RS      |
    | 1934 |   57.89 |    76.45 | RS      |
    | 1935 |   57.88 |    78.42 | RS      |
    | 1936 |   57.50 |    81.45 | RS      |
    | 1937 |   57.59 |    81.39 | RS      |
    | 1938 |   57.46 |    83.05 | RS      |
    | 1939 |   57.53 |    83.27 | RS      |
    | 1940 |   57.64 |    83.90 | RS      |
    | 1941 |   57.69 |    83.88 | RS      |
    | 1942 |   57.79 |    86.95 | RS      |
    | 1943 |   57.93 |    87.59 | RS      |
    | 1944 |   57.71 |    86.43 | RS      |
    | 1945 |   57.79 |    86.09 | RS      |
    | 1946 |   57.96 |    86.36 | RS      |
    | 1947 |   58.00 |    86.78 | RS      |
    | 1948 |   57.67 |    88.71 | RS      |
    | 1949 |   57.28 |    88.21 | RS      |
    | 1950 |   57.24 |    86.50 | RS      |
    | 1951 |   57.26 |    87.12 | RS      |
    | 1952 |   57.26 |    88.28 | RS      |
    | 1953 |   57.69 |    88.36 | RS      |
    | 1954 |   57.68 |    87.23 | RS      |
    | 1955 |   57.92 |    86.62 | RS      |
    | 1956 |   57.87 |    89.92 | RS      |
    | 1957 |   57.97 |    87.09 | RS      |
    | 1958 |   58.23 |    86.16 | RS      |
    | 1959 |   57.79 |    86.49 | RS      |
    | 1960 |   57.66 |    85.33 | RS      |
    | 1961 |   57.57 |    84.98 | RS      |
    | 1962 |   57.88 |    85.16 | RS      |
    | 1963 |   57.83 |    85.12 | RS      |
    | 1964 |   58.05 |    81.10 | RS      |
    | 1965 |   58.02 |    81.31 | RS      |
    | 1966 |   58.03 |    82.08 | RS      |
    | 1967 |   58.02 |    82.06 | RS      |
    | 1968 |   58.02 |    82.06 | RS      |
    | 1969 |   57.09 |    83.94 | RS      |
    | 1970 |   57.10 |    83.81 | RS      |
    | 1971 |   57.18 |    84.93 | RS      |
    | 1972 |   58.23 |    85.25 | RS      |
    | 1973 |   57.03 |    85.94 | RS      |
    | 1974 |   57.03 |    85.57 | RS      |
    | 1975 |   57.11 |    85.28 | RS      |
    | 1976 |   56.99 |    85.99 | RS      |
    | 1977 |   56.90 |    85.72 | RS      |
    | 1978 |   56.96 |    86.33 | RS      |
    | 1979 |   57.02 |    86.40 | RS      |
    | 1980 |   57.16 |    85.91 | RS      |
    | 1981 |   57.22 |    86.02 | RS      |
    | 1982 |   57.26 |    87.01 | RS      |
    | 1983 |   57.33 |    85.43 | RS      |
    | 1984 |   57.21 |    85.81 | RS      |
    | 1985 |   57.22 |    85.70 | RS      |
    | 1986 |   57.19 |    86.04 | RS      |
    | 1987 |   57.18 |    85.36 | RS      |
    | 1988 |   57.17 |    85.99 | RS      |
    | 1989 |   57.14 |    85.72 | RS      |
    | 1990 |   57.06 |    85.20 | RS      |
    | 1991 |   57.15 |    86.02 | RS      |
    | 1992 |   57.14 |    85.36 | RS      |
    | 1993 |   57.09 |    85.69 | RS      |
    | 1994 |   57.10 |    85.44 | RS      |
    | 1995 |   57.12 |    84.71 | RS      |
    | 1996 |   56.94 |    85.00 | RS      |
    | 1997 |   56.89 |    85.21 | RS      |
    | 1998 |   56.86 |    84.48 | RS      |
    | 1999 |   56.95 |    84.16 | RS      |
    | 2000 |   56.90 |    83.84 | RS      |
    | 2001 |   56.85 |    82.62 | RS      |
    | 2002 |   56.93 |    84.01 | RS      |
    | 2003 |   56.92 |    84.27 | RS      |
    | 2004 |   56.90 |    83.94 | RS      |
    | 2005 |   56.96 |    84.12 | RS      |
    | 2006 |   56.94 |    83.90 | RS      |
    | 2007 |   56.95 |    84.07 | RS      |
    | 2008 |   56.90 |    84.42 | RS      |
    | 2009 |   56.91 |    84.26 | RS      |
    | 2010 |   56.90 |    84.22 | RS      |
    | 2011 |   56.93 |    84.14 | RS      |
    | 2012 |   56.89 |    83.11 | RS      |
    | 2013 |   56.86 |    83.14 | RS      |
    | 2014 |   56.79 |    80.44 | RS      |
    | 2015 |   56.81 |    80.23 | RS      |
    | 2016 |   56.91 |    80.14 | RS      |
    | 2017 |   56.83 |    80.16 | RS      |
    | 2018 |   56.79 |    80.25 | RS      |
    | 2019 |   56.98 |    80.62 | RS      |
    +------+---------+----------+---------+
    263 rows in set (6.18 sec)
    
  7. E.M.Smith says:

    Oh, and the code that made that table:

    SELECT T.year,
           ROUND(AVG(I.latitude),2) AS "Avg.LAT", 
           ROUND(AVG(I.longitude),2) AS "Avg.LONG", 
           T.abrev AS 'Country'
    FROM invent4 AS I 
    INNER JOIN temps4 as T on I.stnID=T.stnID 
    WHERE T.abrev='RS' 
    GROUP BY T.year
    ;
    
  8. E.M.Smith says:

    Argentina is interesting. I’ve added elevation too.

    ROUND is used for formatting, to chop off the dozen bits of non-information from the math of doing the floating point math. The /12 is to turn 12 monthly readings into one thermometer count. Since an elevation of 9999 is used for missing data, I’ve added a test for “too high”:

    SELECT T.year,
           ROUND(COUNT(I.stnID)/12,0) AS 'Instr.#',
           ROUND(AVG(I.latitude),2) AS "Avg.LAT", 
           ROUND(AVG(I.longitude),2) AS "Avg.LONG", 
           ROUND(AVG(I.stn_elev),0) AS "Avg.Elev", 
           T.abrev AS 'Country'
    FROM invent4 AS I 
    INNER JOIN temps4 as T on I.stnID=T.stnID 
    WHERE T.abrev='AR' AND I.stn_elev source bin/LatAve.sql
    +------+---------+---------+----------+----------+---------+
    | year | Instr.# | Avg.LAT | Avg.LONG | Avg.Elev | Country |
    +------+---------+---------+----------+----------+---------+
    | 1856 |       1 |  -34.58 |   -58.48 |       25 | AR      |
    | 1857 |       1 |  -34.58 |   -58.48 |       25 | AR      |
    | 1858 |       1 |  -34.58 |   -58.48 |       25 | AR      |
    | 1859 |       1 |  -34.58 |   -58.48 |       25 | AR      |
    | 1860 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1861 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1862 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1863 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1864 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1865 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1866 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1867 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1868 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1869 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1870 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1871 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1872 |       2 |  -36.66 |   -60.32 |       50 | AR      |
    | 1873 |       3 |  -34.91 |   -61.61 |      175 | AR      |
    | 1874 |       3 |  -34.91 |   -61.61 |      175 | AR      |
    | 1875 |       3 |  -34.91 |   -61.61 |      175 | AR      |
    | 1876 |       3 |  -34.91 |   -61.61 |      175 | AR      |
    | 1877 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1878 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1879 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1880 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1881 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1882 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1883 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1884 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1885 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1886 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1887 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1888 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1889 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1890 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1891 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1892 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1893 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1894 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1895 |       3 |  -31.69 |   -60.66 |      162 | AR      |
    | 1896 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1897 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1898 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1899 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1900 |       4 |  -33.45 |   -61.03 |      140 | AR      |
    | 1901 |       8 |  -36.86 |   -63.54 |      235 | AR      |
    | 1902 |      10 |  -40.20 |   -64.09 |      194 | AR      |
    | 1903 |      10 |  -39.51 |   -64.16 |      220 | AR      |
    | 1904 |      13 |  -38.82 |   -64.73 |      246 | AR      |
    | 1905 |      13 |  -38.82 |   -64.73 |      246 | AR      |
    | 1906 |      14 |  -38.39 |   -65.03 |      288 | AR      |
    | 1907 |      14 |  -38.39 |   -65.03 |      288 | AR      |
    | 1908 |      13 |  -37.32 |   -64.77 |      310 | AR      |
    | 1909 |      13 |  -37.32 |   -64.77 |      310 | AR      |
    | 1910 |      14 |  -38.39 |   -65.03 |      288 | AR      |
    | 1911 |      14 |  -36.23 |   -64.83 |      536 | AR      |
    | 1912 |      15 |  -37.31 |   -65.06 |      501 | AR      |
    | 1913 |      15 |  -37.31 |   -65.06 |      501 | AR      |
    | 1914 |      15 |  -37.31 |   -65.06 |      501 | AR      |
    | 1915 |      14 |  -36.23 |   -64.83 |      536 | AR      |
    | 1916 |      15 |  -37.31 |   -65.06 |      501 | AR      |
    | 1917 |      15 |  -37.31 |   -65.06 |      501 | AR      |
    | 1918 |      15 |  -37.31 |   -65.06 |      501 | AR      |
    | 1919 |      15 |  -37.31 |   -65.06 |      501 | AR      |
    | 1920 |      14 |  -36.07 |   -65.13 |      532 | AR      |
    | 1921 |      14 |  -36.07 |   -65.13 |      532 | AR      |
    | 1922 |      14 |  -36.07 |   -65.13 |      532 | AR      |
    | 1923 |      14 |  -36.07 |   -65.13 |      532 | AR      |
    | 1924 |      14 |  -36.07 |   -65.13 |      532 | AR      |
    | 1925 |      14 |  -36.07 |   -65.13 |      532 | AR      |
    | 1926 |      14 |  -36.07 |   -65.13 |      532 | AR      |
    | 1927 |      14 |  -36.07 |   -65.13 |      532 | AR      |
    | 1928 |      25 |  -35.35 |   -64.98 |      451 | AR      |
    | 1929 |      25 |  -35.35 |   -64.98 |      451 | AR      |
    | 1930 |      25 |  -35.35 |   -64.98 |      451 | AR      |
    | 1931 |      52 |  -34.87 |   -64.55 |      392 | AR      |
    | 1932 |      52 |  -34.87 |   -64.55 |      392 | AR      |
    | 1933 |      53 |  -34.87 |   -64.48 |      386 | AR      |
    | 1934 |      52 |  -34.72 |   -64.47 |      393 | AR      |
    | 1935 |      52 |  -34.72 |   -64.47 |      393 | AR      |
    | 1936 |      52 |  -34.72 |   -64.47 |      393 | AR      |
    | 1937 |      51 |  -34.64 |   -64.52 |      400 | AR      |
    | 1938 |      45 |  -34.91 |   -64.43 |      388 | AR      |
    | 1939 |      46 |  -35.00 |   -64.38 |      381 | AR      |
    | 1940 |      46 |  -35.00 |   -64.38 |      381 | AR      |
    | 1941 |      47 |  -35.12 |   -64.27 |      368 | AR      |
    | 1942 |      53 |  -34.83 |   -64.38 |      381 | AR      |
    | 1943 |      53 |  -34.83 |   -64.38 |      381 | AR      |
    | 1944 |      53 |  -34.83 |   -64.38 |      381 | AR      |
    | 1945 |      47 |  -35.12 |   -64.27 |      368 | AR      |
    | 1946 |      47 |  -35.12 |   -64.27 |      368 | AR      |
    | 1947 |      47 |  -35.12 |   -64.27 |      368 | AR      |
    | 1948 |      46 |  -34.74 |   -64.18 |      376 | AR      |
    | 1949 |      48 |  -34.65 |   -64.26 |      385 | AR      |
    | 1950 |      48 |  -34.47 |   -64.10 |      371 | AR      |
    | 1951 |      58 |  -34.84 |   -64.05 |      344 | AR      |
    | 1952 |      57 |  -35.01 |   -64.11 |      347 | AR      |
    | 1953 |      57 |  -34.53 |   -63.97 |      350 | AR      |
    | 1954 |      56 |  -34.63 |   -64.06 |      355 | AR      |
    | 1955 |      56 |  -34.63 |   -64.06 |      355 | AR      |
    | 1956 |      57 |  -34.53 |   -63.97 |      350 | AR      |
    | 1957 |      61 |  -34.73 |   -63.92 |      346 | AR      |
    | 1958 |      60 |  -34.45 |   -63.83 |      352 | AR      |
    | 1959 |      59 |  -34.61 |   -63.95 |      343 | AR      |
    | 1960 |      59 |  -34.61 |   -63.95 |      343 | AR      |
    | 1961 |      73 |  -34.41 |   -63.78 |      347 | AR      |
    | 1962 |      74 |  -34.36 |   -63.61 |      337 | AR      |
    | 1963 |      75 |  -33.98 |   -63.51 |      339 | AR      |
    | 1964 |      75 |  -33.98 |   -63.51 |      339 | AR      |
    | 1965 |      76 |  -34.29 |   -63.57 |      331 | AR      |
    | 1966 |      75 |  -34.35 |   -63.57 |      331 | AR      |
    | 1967 |      75 |  -33.75 |   -63.44 |      343 | AR      |
    | 1968 |      78 |  -34.06 |   -63.55 |      336 | AR      |
    | 1969 |      77 |  -33.88 |   -63.70 |      349 | AR      |
    | 1970 |      77 |  -33.93 |   -63.67 |      347 | AR      |
    | 1971 |      83 |  -34.50 |   -63.82 |      336 | AR      |
    | 1972 |      83 |  -34.50 |   -63.82 |      336 | AR      |
    | 1973 |      88 |  -34.66 |   -63.72 |      327 | AR      |
    | 1974 |      88 |  -34.66 |   -63.72 |      327 | AR      |
    | 1975 |      89 |  -34.75 |   -63.73 |      324 | AR      |
    | 1976 |      88 |  -34.57 |   -63.68 |      327 | AR      |
    | 1977 |      89 |  -34.71 |   -63.76 |      328 | AR      |
    | 1978 |      88 |  -34.39 |   -63.75 |      339 | AR      |
    | 1979 |      86 |  -34.20 |   -63.62 |      339 | AR      |
    | 1980 |      89 |  -34.56 |   -63.69 |      329 | AR      |
    | 1981 |      92 |  -34.69 |   -63.80 |      321 | AR      |
    | 1982 |      89 |  -34.55 |   -63.76 |      324 | AR      |
    | 1983 |      90 |  -34.69 |   -63.77 |      323 | AR      |
    | 1984 |      89 |  -34.62 |   -63.76 |      326 | AR      |
    | 1985 |      89 |  -34.82 |   -63.77 |      322 | AR      |
    | 1986 |      88 |  -34.82 |   -63.84 |      325 | AR      |
    | 1987 |      88 |  -34.84 |   -63.75 |      318 | AR      |
    | 1988 |      91 |  -34.98 |   -63.86 |      322 | AR      |
    | 1989 |      89 |  -34.56 |   -63.76 |      329 | AR      |
    | 1990 |      90 |  -34.76 |   -63.81 |      325 | AR      |
    | 1991 |      79 |  -34.78 |   -63.85 |      327 | AR      |
    | 1992 |      80 |  -34.81 |   -63.84 |      334 | AR      |
    | 1993 |      80 |  -34.72 |   -63.80 |      333 | AR      |
    | 1994 |      80 |  -34.63 |   -63.80 |      334 | AR      |
    | 1995 |      85 |  -34.81 |   -63.91 |      335 | AR      |
    | 1996 |      86 |  -34.72 |   -63.87 |      332 | AR      |
    | 1997 |      86 |  -34.83 |   -63.86 |      332 | AR      |
    | 1998 |      84 |  -34.76 |   -63.90 |      338 | AR      |
    | 1999 |      84 |  -34.76 |   -63.90 |      338 | AR      |
    | 2000 |      83 |  -34.89 |   -63.92 |      339 | AR      |
    | 2001 |      82 |  -35.04 |   -63.89 |      301 | AR      |
    | 2002 |      85 |  -34.84 |   -63.87 |      333 | AR      |
    | 2003 |      85 |  -34.84 |   -63.87 |      333 | AR      |
    | 2004 |      85 |  -34.84 |   -63.87 |      333 | AR      |
    | 2005 |      85 |  -34.84 |   -63.87 |      333 | AR      |
    | 2006 |      85 |  -34.84 |   -63.87 |      333 | AR      |
    | 2007 |      79 |  -34.57 |   -63.70 |      328 | AR      |
    | 2008 |      73 |  -34.27 |   -63.69 |      329 | AR      |
    | 2009 |      73 |  -34.33 |   -63.65 |      325 | AR      |
    | 2010 |      77 |  -34.28 |   -63.69 |      333 | AR      |
    | 2011 |      75 |  -34.28 |   -63.71 |      323 | AR      |
    | 2012 |      75 |  -34.28 |   -63.71 |      323 | AR      |
    | 2013 |      75 |  -34.51 |   -63.79 |      332 | AR      |
    | 2014 |      71 |  -34.59 |   -63.80 |      336 | AR      |
    | 2015 |      73 |  -34.74 |   -63.89 |      338 | AR      |
    | 2016 |      71 |  -34.59 |   -63.80 |      336 | AR      |
    | 2017 |      71 |  -34.50 |   -63.60 |      332 | AR      |
    | 2018 |      71 |  -34.50 |   -63.60 |      332 | AR      |
    | 2019 |      70 |  -34.49 |   -63.69 |      336 | AR      |
    +------+---------+---------+----------+----------+---------+
    164 rows in set (1.56 sec)
    

    Think losing 200 m of elevation average might matter?

  9. E.M.Smith says:

    Oh Man, Bolivia loses a lot of altitude between the “baseline” era (1950-1990) and now:

    +------+---------+---------+----------+----------+---------+
    | year | Instr.# | Avg.LAT | Avg.LONG | Avg.Elev | Country |
    +------+---------+---------+----------+----------+---------+
    | 1898 |       1 |  -16.50 |   -68.17 |     3701 | BL      |
    | 1899 |       1 |  -16.50 |   -68.17 |     3701 | BL      |
    | 1900 |       1 |  -16.50 |   -68.17 |     3701 | BL      |
    | 1901 |       1 |  -16.50 |   -68.17 |     3701 | BL      |
    | 1902 |       1 |  -16.50 |   -68.17 |     3701 | BL      |
    | 1903 |       1 |  -16.50 |   -68.17 |     3701 | BL      |
    | 1913 |       1 |  -19.03 |   -65.28 |     2884 | BL      |
    | 1915 |       1 |  -19.03 |   -65.28 |     2884 | BL      |
    | 1916 |       1 |  -19.03 |   -65.28 |     2884 | BL      |
    | 1917 |       1 |  -19.03 |   -65.28 |     2884 | BL      |
    | 1918 |       4 |  -17.14 |   -67.44 |     3576 | BL      |
    | 1919 |       4 |  -17.14 |   -67.44 |     3576 | BL      |
    | 1920 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1921 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1922 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1923 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1924 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1925 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1926 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1927 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1928 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1929 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1930 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1931 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1932 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1933 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1934 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1935 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1936 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1937 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1938 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1939 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1940 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1941 |       3 |  -16.50 |   -68.15 |     3807 | BL      |
    | 1942 |      13 |  -18.18 |   -65.81 |     2577 | BL      |
    | 1943 |      18 |  -18.26 |   -65.75 |     2504 | BL      |
    | 1944 |      19 |  -18.28 |   -65.82 |     2567 | BL      |
    | 1945 |      18 |  -18.33 |   -65.88 |     2499 | BL      |
    | 1946 |      19 |  -18.28 |   -65.82 |     2567 | BL      |
    | 1947 |      19 |  -18.28 |   -65.82 |     2567 | BL      |
    | 1948 |      18 |  -18.21 |   -65.82 |     2487 | BL      |
    | 1949 |      21 |  -18.17 |   -65.70 |     2523 | BL      |
    | 1950 |      21 |  -18.17 |   -65.70 |     2523 | BL      |
    | 1951 |      28 |  -17.47 |   -65.18 |     2050 | BL      |
    | 1952 |      27 |  -17.39 |   -65.16 |     1977 | BL      |
    | 1953 |      26 |  -17.23 |   -65.14 |     1940 | BL      |
    | 1954 |      13 |  -16.47 |   -64.53 |     1432 | BL      |
    | 1955 |      27 |  -17.48 |   -65.11 |     1990 | BL      |
    | 1956 |      25 |  -17.32 |   -65.15 |     2025 | BL      |
    | 1957 |      23 |  -17.14 |   -65.25 |     2102 | BL      |
    | 1958 |      28 |  -17.37 |   -65.21 |     2086 | BL      |
    | 1959 |      26 |  -17.17 |   -65.18 |     1975 | BL      |
    | 1960 |      23 |  -17.00 |   -64.94 |     1880 | BL      |
    | 1961 |      21 |  -17.09 |   -64.89 |     1861 | BL      |
    | 1962 |      22 |  -17.21 |   -64.93 |     1959 | BL      |
    | 1963 |      21 |  -17.20 |   -65.01 |     2032 | BL      |
    | 1964 |      21 |  -16.97 |   -65.13 |     1909 | BL      |
    | 1965 |      14 |  -16.58 |   -64.71 |     1595 | BL      |
    | 1966 |      15 |  -17.09 |   -65.00 |     1777 | BL      |
    | 1967 |      11 |  -17.10 |   -65.86 |     2096 | BL      |
    | 1968 |      12 |  -16.88 |   -65.42 |     1835 | BL      |
    | 1969 |      13 |  -16.94 |   -65.73 |     2006 | BL      |
    | 1970 |      13 |  -16.94 |   -65.73 |     2006 | BL      |
    | 1971 |      19 |  -16.29 |   -65.39 |     1333 | BL      |
    | 1972 |      20 |  -16.13 |   -65.35 |     1274 | BL      |
    | 1973 |      23 |  -16.41 |   -65.33 |     1413 | BL      |
    | 1974 |      24 |  -16.35 |   -65.39 |     1362 | BL      |
    | 1975 |      23 |  -16.28 |   -65.31 |     1260 | BL      |
    | 1976 |      24 |  -16.35 |   -65.39 |     1362 | BL      |
    | 1977 |      24 |  -16.35 |   -65.39 |     1362 | BL      |
    | 1978 |      26 |  -16.42 |   -65.41 |     1418 | BL      |
    | 1979 |      27 |  -16.53 |   -65.42 |     1514 | BL      |
    | 1980 |      27 |  -16.37 |   -65.28 |     1248 | BL      |
    | 1981 |      24 |  -16.27 |   -65.09 |     1104 | BL      |
    | 1982 |      23 |  -16.33 |   -65.01 |     1144 | BL      |
    | 1983 |      24 |  -16.46 |   -65.04 |     1263 | BL      |
    | 1984 |      24 |  -16.64 |   -65.00 |     1273 | BL      |
    | 1985 |      24 |  -16.64 |   -65.00 |     1273 | BL      |
    | 1986 |      24 |  -16.64 |   -65.00 |     1273 | BL      |
    | 1987 |      27 |  -16.50 |   -64.72 |     1146 | BL      |
    | 1988 |      22 |  -16.26 |   -64.77 |     1143 | BL      |
    | 1989 |      18 |  -16.23 |   -64.82 |     1260 | BL      |
    | 1990 |      18 |  -16.78 |   -63.99 |      935 | BL      |
    | 1991 |      19 |  -16.92 |   -64.08 |     1093 | BL      |
    | 1992 |      22 |  -16.74 |   -63.91 |      974 | BL      |
    | 1993 |      20 |  -16.97 |   -63.91 |     1053 | BL      |
    | 1994 |      31 |  -16.68 |   -64.44 |     1231 | BL      |
    | 1995 |      31 |  -16.61 |   -64.46 |     1229 | BL      |
    | 1996 |      35 |  -16.94 |   -64.31 |     1258 | BL      |
    | 1997 |      37 |  -16.78 |   -64.43 |     1232 | BL      |
    | 1998 |      37 |  -16.78 |   -64.43 |     1232 | BL      |
    | 1999 |      37 |  -16.78 |   -64.43 |     1232 | BL      |
    | 2000 |      37 |  -16.78 |   -64.43 |     1232 | BL      |
    | 2001 |      38 |  -16.73 |   -64.49 |     1204 | BL      |
    | 2002 |      38 |  -16.73 |   -64.49 |     1204 | BL      |
    | 2003 |      38 |  -16.73 |   -64.49 |     1204 | BL      |
    | 2004 |      38 |  -16.73 |   -64.49 |     1204 | BL      |
    | 2005 |      38 |  -16.73 |   -64.49 |     1204 | BL      |
    | 2006 |      38 |  -16.73 |   -64.49 |     1204 | BL      |
    | 2007 |      38 |  -16.73 |   -64.49 |     1204 | BL      |
    | 2008 |      36 |  -16.76 |   -64.24 |     1119 | BL      |
    | 2009 |      36 |  -16.76 |   -64.24 |     1119 | BL      |
    | 2010 |      36 |  -16.76 |   -64.24 |     1119 | BL      |
    | 2011 |      36 |  -16.76 |   -64.24 |     1119 | BL      |
    | 2012 |      36 |  -16.76 |   -64.24 |     1119 | BL      |
    | 2013 |      36 |  -16.76 |   -64.24 |     1119 | BL      |
    | 2014 |      32 |  -16.94 |   -64.12 |     1119 | BL      |
    | 2015 |      32 |  -16.94 |   -64.12 |     1119 | BL      |
    | 2016 |      27 |  -16.64 |   -63.83 |      908 | BL      |
    | 2017 |      26 |  -16.55 |   -63.77 |      831 | BL      |
    | 2018 |      26 |  -16.55 |   -63.77 |      831 | BL      |
    | 2019 |      26 |  -16.55 |   -63.77 |      831 | BL      |
    +------+---------+---------+----------+----------+---------+
    112 rows in set (1.10 sec)
    
  10. Larry Ledwick says:

    Yes on lapse rate alone 200 m should shift the temperature about 2.6 deg F.

    Interesting observation on Russia that probably applies to Africa, as population (or technological sophistication changes) you get a “migration of thermometers” into new microclimates which adds yet another complication in addition to latitude, altitude, industrialization, ( changes in micro climate such as huge reservoirs ) all added on top of what ever true temperature trend there is.

    As an thought experiment example if you had been plotting average temperatures of all thermometers in the US at the time the pioneers swept across the plains and settled small towns as they went, you would have had a huge shift in the pool of thermometers along with elevation going from 500 ft elevation in St. Louis to 3000 some ft in Nebraska, peaking at about 7000 ft as they crossed and settled Wyoming, then dropping back down to about 4000 ft in Salt Lake and the great basin etc.

    I am sure the same impacts occurred as settlement shifted across Siberia with the development on long distance rail movement, and likewise as advanced urban communities built up in Africa.

  11. E.M.Smith says:

    I’m seeing the same general thing in most countries with mountains. Some only 100 m or so, but that can matter. Ecuador, Peru even Norway. Average elevation higher in the baseline, lower in the present.

    New Zealand was just low elevation a lot of the time. Thought those guys had mountains!?
    ;-)

    I think I need to turn them into graphs before I do much more of this – large blocks of numbers are not something most folks like to look at …

  12. Chiefio, re the drift of ‘country parent’ as between versions. I work in data warehousing, especially of accounting data, and consolidating tens of companies tends to throw up the exact same issue: a given code is used for different meanings in different companies, or is unique to one company only, despite the best efforts of CFO/CIO types to maintain congruence. The standard star-schema warehouse solution is to ‘uniqify’ the code by suffixing the company or something similar to the raw code, then adding attributes – grouping columns – which specify ‘this here Account ABC is really CDE’. So if a Country Attributes table, common to all versions, is made up, the key would be Code+Version, and grouping tags could then be attached to bring back the commonality that’s needed to make sensible comparisons. The join would be (T-SQL, Microsoft syntax) and assuming main table has column ‘CountryCode’ and is aliased as T:
    LEFT OUTER JOIN Country_Attributes C on T.code = C.Code + ‘4’
    The advantage of Attributes tables are that multiple alternative groupings can be held in one place. The alternates are then used for different audiences/purposes. If this is old news, so be it. If it helps, it would be worth a try in order to rationalise/harmonise codes across versions.

    As a NZ’er (and Christchurch at that) the main reason for the low altitudes shown is simple – there are no large cities at much more than sea level. And in some cases (e.g. Napier airport) the land actually used to Be sea, until (in Napier’s case) the 1931 earthquake….

    Keep up the wonderful work. Your work rate is inspiring.

  13. Bill in Oz says:

    E M
    1: Point of clarification: Is the elevation for Argentina and Bolivia weather stations, in meters or in feet ? I assume it is meters. But better to know for sure.
    2 : There is a similar pattern for Argentina & Bolivia. The altitude of stations for both countries has go less over the past 50 years. I suggest that reflects agricultural development in each country. Agriculture in general thrives in warmer locations and is limited by low temperatures at high altitudes.
    In the case of Bolivia that is a reflection of the development of the low tropical area of Bolivia bordering Brazil, since the 1960’s.

    In the case of Argentina the early high altitudes probably reflect that the wine industry developed in such provinces as Mendoza, San Juan and La Rioja from the 1890’s-1920’s. These are all Andean provinces in the West of Argentina.. Then later on less intensive agricultural development happened in the more far flung lowland areas of Northern ( eg. Missiones & Entre Rios ) and Southern Argentina ( eg. Patagonia ).

    ( BTW My knowledge & interest in Argentina comes as a side effect of learning & dancing Argentine tango. And living in BA’s for a while in 2014. )

    These were the processes that lead to the average altitude of weather stations becoming less.

    And surely this process also had an impact on the ‘summed’ Average temperature.

    I wonder what will emerge from this work on Australia ?

    Interesting work !

  14. Bill in Oz says:

    E M , Re Russia, Well Russia was a global spanning empire like the others. It just expanded over land whereas all the others expanded over seas. And when the empire collapsed lots of countries emerged with weird boundaries..Modern Russia’s weather station data reflects that process.

    Russia is curious because the average latitude gets less over time by 3 degrees… I imagine this had an impact. But there is no significant change post 1990 when the USSR morphed into 17 different countries. I guess someone has re-allocated all the earlier temp readings to the new countries which emerged ost 1990.

  15. E.M.Smith says:

    https://chiefio.wordpress.com/2019/03/18/ghcn-v4-first-peek/

           STELEV: is the station elevation in meters. -999.0 = missing.
    

    I think an important part is also the move of more stations to Jet Airports. It is hard to find 10,000 feet of flat level unobstructed ground in the mountains…

    Some of it, though, is just selection choice. California was dropped to “4 on the beach” all next t the ocean. There’s still plenty of stations, with long lives, up in the mountains. It was a choice to not use them.

    I’m not so much worried about the lack of high altitude stations as I am about having CHANGE of altitude over time in the comparisons. Basically I think the “Reference Station Method” is not perfect at removing variation in volatility and that high altitude cold stations are more volatile AND have more variation to the downside than to the upside. Then that effect shows up as “higher lows” that get interpreted as a “warming” average when it is really a loss of cold excursions.

  16. Bill in Oz says:

    Re : “CHANGE of altitude over time in the comparisons” Yes, I agree absolutely with you..

    Each version is is a 4-5,000+ piece jig saw picture. And with that many small bits of info, it’s dead easy to think each is the same when they are NOT.

  17. E.M.Smith says:

    WOW! On the RockPro64, the load of monthly statistics for version 4 is way faster. It still is pegging on CPU core, but now it’s an A72 core at about 2 x the clock rate. It finishes in 1/3 the time. Instead of about 1/2 hour (30 minutes) it’s:

    MariaDB [temps]> source bin/lmstats4.sql
    Query OK, 328327 rows affected (10 min 42.20 sec)
    Records: 328327  Duplicates: 0  Warnings: 0
    
    MariaDB [temps]> 
    

    That’s a whole lot more comfortable for kicking around frequently.

    It pegs one of the BIG cores at between 92% and 99% with the occasional 100%. This board is a BIG/little chip with 4 ‘little’ cores at 1.5 GHz and 2 BIG cores at 1.8 GHz. The BIG A72 cores also get more done / cycle than the A53 cores so it isn’t just clock rate.

    That said, in most cases it will only be necessary to make the statistics one time and then use them for days to weeks. So a Raspberry Pi M3 is ‘enough computer’. The Pi pegs at a flat 100% and this is a little under that most of the time, so while it is still CPU limited, it is likely at about the limit where memory transfers and cache speed and other bits start to matter.

    For development cycles, though, this one is a lot more pleasing ;-)

    One nice thing is that once computed, the statistics can just be moved from one machine to another if desired. So really, just computing it once ought to be enough ;-)

    Guess the next thing is to compute the anomaly tables…

  18. E.M.Smith says:

    So 19 min per:
    https://chiefio.wordpress.com/2019/03/24/ghcn-v4-adding-statistics-anomaly-tables/#comment-110143

    only drops to 16 minutes:

    MariaDB [temps]> source bin/lanom4.sql
    Query OK, 15448107 rows affected, 65535 warnings (16 min 32.44 sec)
    Records: 15448107  Duplicates: 0  Warnings: 15444656
    

    For the loading of the anomaly table. Don’t get me wrong, I’ll take 3 minutes, but….

    Mostly only loading up one core about 30% (and a little one at that). Occasionally doing about 50% of a BIG core (but some of that might have been the browser asking for something…). Sporadic Red D disk waits in htop.

    Now all this is running FROM uSD card and TO uSD card (as it takes from one DB Table and loads into another after minimal math (one subtraction) so I/O speed will dominate. Still, with the card there are no “head seeks” so that’s not an issue.

    Overall, my conclusion so far is faster IO maters more than an A72 core for everything other than the one math heavy statistics creating table load. Most likely a quad core at 1.5 to 1.8 GHz with a USB 3.0 will be about as much as you can need (and even then most of the time it will be idle or using all of one core). I have noticed that the Pi with only 1 GB of memory has a few hundred MB in swap so a 2 GB memory is helpful.

    So I think that’s the “spec” for a system that can do the climate data manipulation in MariaDB at about as fast as you need and more money won’t get much faster until you are into the Very Big Bucks. (Think SSD and lots of memory, cache, and ramdisk).

    About a $40 to $50 SBC. 2 GB Memory. USB 3.0 preferred for a real disk. FAST uSC card ( I think mine is a 90 MB read rate), Quad Core (or even dual core if anyone still makes that small) at about 1.5 GHz or better. A53 cores or better. (The XU4 from Odroid with the A15 cores is also nice).

    OK, I’ve got a few reports to run for speed comparison, but as they are mostly disk intensive don’t expect much speedup. The graphs might speed up, depending on how much math goes into making a scatter plat.

    But at this point, I’m thinking the Raspberry Pi is “just enough” and after not too much more it’s about as fast as you will get – so like an Odroid C2 or Rock64 kind of thing gets you most of the way there. The Odroid XU4 and RockPro64 become “a little nicer and faster” but not enough to make putting up with the funky OS limitations “worth it”. (Being BIG/little with 8 or 6 cores and not a lot sold means there’s a lot of things that don’t work, or are annoying. Like sound or systemD or various browsers not being available or… OTOH, they all have MariaDB and being able to flip rapidly though web pages in a browser while a 1/2 hour data load is done in 10 minutes was nice ;-)

    But the problem is it will be a week or more before I need (or plausibly can think of a reason to try something…) do it again. The $30 NOT spent on an upgrade from a Pi M3 to a RockPro64 would buy a lot of 20 minute beer breaks… Just sayin’…

  19. E.M.Smith says:

    Well, this kind of settles it then….

    I went to run a report I’d done before (the one with altitude blocks) only using v4 data.

    It uses about 53% to 65% of ONE of the A53 cores for almost all of the run. At the point where it makes the plot, if very briefly, like 5 seconds max 10, runs one of the BIG A72 cores at 100%, then puts up the plot.

    So for graphing and generic reports, the RockPro64 is just loafing and you are using the Odroid C2 (or Rock64) 1/2 of it (but even then only 2/3 of one core…)

    Then “the kicker’: The graph doesn’t look at all right and the axis lables are screwed up. As in blobs that look like 6 lines of over typed gibberish as in smear like things.

    All of which leads me to think something is not quite right in the Matplotlib or Numpty or some package or other. That it’s not fully debugged and ported yet on this particular platform. Complied yes, right not so much.

    I’d not be surprised at all to find out it was trying to use the graphics processor for the graph and was ported from some slightly different model… and the compile succeeded and the QA was limited (or nil…)

    I’ll try a few more graphs, perhaps something simpler than two colors of different altitude blocks… but at this point IMHO it isn’t something I can trust to be correct.

    Were it not for the fact that I copied over exactly the stuff I’ve already run, I’d think I’d gotten the code wrong. Were it just the plotted points, I’d think maybe I did something wrong in the data load / stats creation / whatever steps.

    But this is the legend that’s specified with a trivial (and unchanged) directive:

        plt.title("v4 Global Thermometer Above 2000 m  Base (blue) vs 2018")
        plt.xlabel("Longitude")
        plt.ylabel("Latitude")
        plt.xlim(-180,180)
        plt.ylim(-90,90)
    

    I get the “Longitude” word OK but the “Latitude” and the range numbers are a garble.

    That’s not me. That’s a bug.

    So other than a couple of more to confirm that graphing is screwed up, I’m done with the RockPro64 testing for MariaDB / Python3 plots.

    Don’t need that much hardware, do need more mature and stable software.

    BTW, I also got a message that the “mysql” connector wasn’t there. I had to do this bit to get it to connect to the database at all:

    ============= RESTART: /home/chiefio/Py3/BaseVsNow/BaseVsNow2.py =============
    Traceback (most recent call last):
      File "/home/chiefio/Py3/BaseVsNow/BaseVsNow2.py", line 7, in 
        import mariadb.connector as mariadb
    ModuleNotFoundError: No module named 'mariadb'
    

    https://askubuntu.com/questions/1014947/mysql-connector-python-importerror-no-module-named-mysql

    The solution is to install corresponding Python 3 module:
    
    sudo apt-get install python3-mysql.connector
    It fixes import mysql.connector error:
    

    With that, I think it’s time to call it a night…

  20. Pingback: Ubuntu As Advertiser Monster? | Musings from the Chiefio

  21. E.M.Smith says:

    So over night we get 3 MB of update LISTS. Not the actual software, just what is to be updated.

    root@rockpro64:~# apt-get update
    Hit:2 http://ports.ubuntu.com bionic InRelease
    Get:3 http://ports.ubuntu.com bionic-security InRelease [88.7 kB]
    Get:4 http://ports.ubuntu.com bionic-updates InRelease [88.7 kB]               
    Hit:1 https://apt.armbian.com bionic InRelease                                 
    Get:5 http://ports.ubuntu.com bionic-backports InRelease [74.6 kB]       
    Get:6 http://ports.ubuntu.com bionic-security/main arm64 Packages [205 kB]
    Get:7 http://ports.ubuntu.com bionic-security/main armhf Packages [185 kB]
    Get:8 http://ports.ubuntu.com bionic-security/universe arm64 Packages [130 kB] 
    Get:9 http://ports.ubuntu.com bionic-security/universe armhf Packages [127 kB] 
    Get:10 http://ports.ubuntu.com bionic-updates/main armhf Packages [439 kB]     
    Get:11 http://ports.ubuntu.com bionic-updates/main arm64 Packages [454 kB]     
    Get:12 http://ports.ubuntu.com bionic-updates/universe arm64 Packages [634 kB] 
    Get:13 http://ports.ubuntu.com bionic-updates/universe armhf Packages [599 kB] 
    Fetched 3,026 kB in 11s (287 kB/s)                                             
    Reading package lists... Done
    

    Yeah, that’s a “young port” undergoing “active development”…

  22. Pingback: GHCN v3.3 vs v4 – Top Level Entry Point | Musings from the Chiefio

Anything to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.