GHCN v3.3 vs v4 – Selected Country Anomaly Differences

This will be a bit of a pot-luck of some semi-random country graphs that I just chose by whim. There’s a LOT of countries so doing all of them would be a bit much for one posting. I tried to spread them around the world and do a mix of large and small.

There are two graphs for each country. One is an A/B graph of the v3.3 anomaly data vs the v4 anomaly data. The other is a graph of one minus the other for a “delta” or difference number.

It was all a ‘rush job’ with several surprise turns, so hopefully I didn’t miss some subtle point. Please help by scrutinizing some graph or other and making sure they look right.

Yes, there’s a LOT of graphs. Just remember all you have to do is look at them and that’s a lot less than it took to create them and upload each one and put them in a posting… I’ve scattered a few snide comments along the way, but I’m not “fresh” enough after this effort to do detailed analysis. If you see something interesting, say something! ;-)

One country in particular, Lesotho, was chosen for the very small number of data points as that ought to let me “QA Check” that the two graphs match where they both have data and that the Delta graph does not include data for years where only one of the two versions has data. With that, here’s Lesotho:

Lesotho – Introduction and Validity Check

GHCN v3.3 vs v4 Lesotho Anomaly

GHCN v3.3 vs v4 Lesotho Difference

This may be a lousy nation for thermometer data, but it’s a good test case. Notice that the v3 data are only in a few years on the first graph. Eyeballing the second graph, it looks like it correctly only has spots for those years where both data sets have data. The size of the value in the second graph looks like a match to the difference on the first graph, read off the graph.

Around 1975 there is a hotter v4 value and the difference graph has that as a positive while the colder years are negative values, so “trend” is in the right direction.

With that out of the way, on to the rest of the graphs. These will be in roughly alphabetical order. At the end there will be a Tech Talk section with the table schema and Python code along with the SQL used.

Other Countries Graphs

Argentina

GHCN v3.3 vs v4 Argentina Difference

Not much going on in Argentina. Some of the “baseline” period from 1950-1990 gets nudged up, some down, and the recent data becomes more changable.

GHCN v3.3 vs v4 Argentina Anomaly

Australia

GHCN v3.3 vs v4 Australia Difference

Interesting that the general trend is a roll off of heat. But a couple of years get a hot bump at the end.

GHCN v3.3 vs v4 Australia Anomaly

Brazil

GHCN v3.3 vs v4 Brazil Difference

Not a lot of trend, but certainly some odd changes. Deep past gets cooled a lot, the baseline gets a divot, then a couple of hot fliers recently. Wonder what justified those changes?

GHCN v3.3 vs v4 Brazil Anomaly

Burundi

GHCN v3.3 vs v4 Burundi Difference

Not much happening in Burundi. A bit of cooling the baseline period. They drop the oldest v3 data (see the plot below)

GHCN v3.3 vs v4 Burundi Anomaly

Canada

GHCN v3.3 vs v4 Canada Difference

What’s the deal with recent Canada data? That’s just crazy change. 2 C of change in the anomaly range.

GHCN v3.3 vs v4 Canada Anomaly

China

GHCN v3.3 vs v4 China Difference

A little dip into the start of the baseline window, and recent larger changes, but over all China is fairly flat. The general cooling of the past is a bother. Just wrong, even if only a 1/10 C or two.

GHCN v3.3 vs v4 China Anomaly

France

GHCN v3.3 vs v4 France Difference

Looks like the French are not fooling around with their data much. Substantially flat. I wonder if the dip around 2000 was removing some fudge ’cause someone got caught? Wonder what was in their news then…

GHCN v3.3 vs v4 France Anomaly

French Polynesia

GHCN v3.3 vs v4 French Polynesia Difference

Again with the cooling of the baseline window… I think we’re getting a trend here… but not in the climate.

GHCN v3.3 vs v4 French Polynesia Anomaly

Germany

GHCN v3.3 vs v4 Germany Difference

Nice “tuck” taken in 1950. Then recently a couple of years with a hot pop.

GHCN v3.3 vs v4 Germany Anomaly

India

GHCN v3.3 vs v4 India Difference

Looks like India isn’t with the “adjusting” program ;-)

GHCN v3.3 vs v4 India Anomaly

Indonesia

GHCN v3.3 vs v4 Indonesia Difference

Either the historic Indonesia data are crap and need a lot of fixes, or they can’t decide what their temperature was in the past.. Nice warming jump added at the recent end.

GHCN v3.3 vs v4 Indonesia Anomaly

Italy

GHCN v3.3 vs v4 Italy Difference

WOW. Deep past changes by at least 2 C (and may have gone off my graph range), then they cool most of the past by about another 1/2 C, then run up the recent data by 1/2 C. Looks like Italy is with the program of Data Diddle.

GHCN v3.3 vs v4 Italy Anomaly

Japan

GHCN v3.3 vs v4 Japan Difference

Well. Japan can only manage 1/4 C of cooling of the early baseline period. The Japanese must feel guilty about changing historical data…

GHCN v3.3 vs v4 Japan Anomaly

Johnston Atoll

GHCN v3.3 vs v4 Johnston Atoll Difference

Not much going on with Johnston Atoll. Then again they already have 2 C range in the anomaly (see next graph) so maybe nothing more was needed…

GHCN v3.3 vs v4 Johnston Atoll Anomaly

New Zealand

GHCN v3.3 vs v4 New Zealand Difference

Abut 1/2 C cooling of the deep past, but not much else.

GHCN v3.3 vs v4 New Zealand Anomaly

Saudi Arabia

GHCN v3.3 vs v4 Saudi Arabia Difference

1.5 C range of variation? I wonder if they play with their thermometers much?…

GHCN v3.3 vs v4 Saudi Arabia Anomaly

Seychelles

GHCN v3.3 vs v4 Seychelles Difference

WARMING in the baseline? Cooling recent data? They have a bit over 2 C of anomaly delta in the data (see below graph) so maybe it was standing out too much? Looking too much like jet exhaust and not enough like CO2?

GHCN v3.3 vs v4 Seychelles Anomaly

South Africa

GHCN v3.3 vs v4 South Africa Difference

GHCN v3.3 vs v4 South Africa Anomaly

South Korea

GHCN v3.3 vs v4 South Korea Difference

Ah, the Classics. Just a smooth gentle 1/2 C colder past… Korean tech is neat and careful like that.

GHCN v3.3 vs v4 South Korea Anomaly

Spain

GHCN v3.3 vs v4 Spain Difference

OMG what a mess. 1.2 C colder deep past, baseline only 1/4 C colder. Recent data about 1/4 C of lift.

GHCN v3.3 vs v4 Spain Anomaly

Tonga

GHCN v3.3 vs v4 Tonga Difference

GHCN v3.3 vs v4 Tonga Anomaly

Turkey

GHCN v3.3 vs v4 Turkey Difference

Turkey had complained that GHCN was only using the few thermometers that showed warming and ignoring the ones that showed cooling. Wonder if that “sensitized” folks to not fool with Turkey? Looks like some W.W.I data got adjusted.

GHCN v3.3 vs v4 Turkey Anomaly

United States Of America

GHCN v3.3 vs v4 USA Anomaly Difference

Looks like somebody fooled around a lot with the 1800s. Then recently a couple of years get a big bump up.

GHCN v3.3 vs v4 USA Anomaly

What Does It Mean?

I’m not sure. I’d guess that it shows some countries have “activist” BOM’s who are adjusting their data, with emphasis on the English Speaking countries and maybe those expecting to pick up a bit of cash. It’s possible this was done in the creation of the GHCN, but given how different the countries look that would require a deliberate decision to obfuscate the changes. I suppose it is possible some “new idea” was applied but only to some countries (based on some attribute like the thermometer type or their standards).

To me, it looks like for at least some of the larger and more important countries there is a bias toward cooling the past and warming the “present”, even in data that were considered “just fine” in v3.3 all of a couple of years ago. That, IMHO, is not Science. That is “action for effect”.

Any of this say anything to you?

Tech Talk

I added a new table that holds the average anomaly for each year in each country. This makes the V4 minus V3.3 math easier to do. The straight graph of the anomalies average was done first and by a different path, and I’d already made those graphs prior to making the new table for the Delta or Difference graphs, so I’ve left that code as-is for now (so it can also serve as a QA check on the rest of the process).

The Table Schema:

chiefio@PiM3Devuan2:~/SQL/tables$ cat yrcastats 
CREATE TABLE yrcastats (
    year   CHAR(4) NOT NULL,
    abrev  CHAR(2) NOT NULL,
    mean3  FLOAT,
    mean4  FLOAT,
    big3   DECIMAL(7,2) NOT NULL,
    big4   DECIMAL(7,2) NOT NULL,
    small3 DECIMAL(7,2) NOT NULL,
    small4 DECIMAL(7,2) NOT NULL,
    num3   INTEGER NOT NULL,
    num4   INTEGER NOT NULL,
    trang3 FLOAT NOT NULL,
    trang4 FLOAT NOT NULL,
    stdev3 FLOAT NOT NULL,
    stdev4 FLOAT NOT NULL,
    PRIMARY KEY (year,abrev)
    ) ;

The SQL code to create the average anomalies and load it:

chiefio@PiM3Devuan2:~/SQL/bin$ cat Lyrcastats 
INSERT INTO  yrcastats (year,abrev,mean4,big4,small4,num4,trang4,stdev4)
SELECT year,abrev,
AVG(deg_C),MAX(deg_C),MIN(deg_C),COUNT(deg_C), MAX(deg_C)-MIN(deg_C), STDDEV(deg_C)
FROM anom4 
GROUP BY year,abrev
;
show warnings;

UPDATE  yrcastats AS Y
SET 
mean3 = (SELECT AVG(A.deg_C) 
FROM anom3 AS A
WHERE Y.year=A.year AND Y.abrev=A.abrev),

big3 = (SELECT MAX(A.deg_C) 
FROM anom3 AS A
WHERE Y.year=A.year AND Y.abrev=A.abrev),

small3 = (SELECT MIN(A.deg_C) 
FROM anom3 AS A
WHERE Y.year=A.year AND Y.abrev=A.abrev),

num3 = (SELECT COUNT(A.deg_C) 
FROM anom3 AS A
WHERE Y.year=A.year AND Y.abrev=A.abrev),

trang3 = (SELECT MAX(A.deg_C)-MIN(A.deg_C) 
FROM anom3 AS A
WHERE Y.year=A.year AND Y.abrev=A.abrev),

stdev3 = (SELECT STDDEV(A.deg_C) 
FROM anom3 AS A
WHERE Y.year=A.year AND Y.abrev=A.abrev)
;
show warnings;

This is rather slow on the UPDATE section. There may well be a faster way… The initial insert of the v4 data that’s much larger in volume than the v3.3 data takes only 4 minutes on the Raspberry Pi M3. Then the Update to add the v3.3 values takes over 4 hours…

MariaDB [temps]> source bin/Lyrcastats
Query OK, 28423 rows affected, 5 warnings (4 min 7.05 sec)
Records: 28423  Duplicates: 0  Warnings: 5

Query OK, 23987 rows affected, 17744 warnings (4 hours 20 min 58.52 sec)
Rows matched: 28423  Changed: 23987  Warnings: 17744

The Python3 code that makes the graphs for the two graph types:

Here’s the one that calculates and plots the two anomaly values on one graph, directly from the anomaly by month tables.

chiefio@PiM3Devuan2:~/Py3/v4$ cat cnASdelta.py 
# -*- coding: utf-8 -*-
import datetime
import pandas as pd
import numpy as np
import matplotlib.pylab as plt
import math
import mysql.connector as MySQLdb

plt.title("GHCN v3 black vs v4 red Anomaly by Years")
plt.ylabel(" Country AS Australia Anomaly C")
plt.xlabel("Year")
plt.ylim(-3,2)
#plt.ylim(1850,2020)

try:
    db=MySQLdb.connect(user="chiefio",password="LetMeIn!",database='temps')
    cursor=db.cursor()

    
    sql="SELECT year,AVG(deg_c) FROM anom4 WHERE abrev='AS' GROUP BY year;"

    print("stuffed SQL statement")
    cursor.execute(sql)
    print("Executed SQL")
    stn=cursor.fetchall()
#    print(stn)
    data = np.array(list(stn))
    print("Got data")
    xs = data.transpose()[0]   # or xs = data.T[0] or  xs = data[:,0]
    ys = data.transpose()[1]
    print("after the transpose")

    plt.scatter(xs,ys,s=1,color='red',alpha=1)
#    plt.show()


    
    sql="SELECT year,AVG(deg_c) FROM anom3 AS A INNER JOIN country AS C ON A.country=C.cnum  WHERE C.abrev='AS' GROUP BY year;"

    print("stuffed v3.3 SQL statement")
    cursor.execute(sql)
    print("Executed SQL")
    stn=cursor.fetchall()

    data = np.array(list(stn))
    print("Got data")
    xs = data.transpose()[0]   # or xs = data.T[0] or  xs = data[:,0]
    ys = data.transpose()[1]
    print("after the transpose")

    plt.scatter(xs,ys,s=1,color='black',alpha=1)
    plt.show()



except:
    print ("This is the exception branch")

finally:
    print ("All Done")
    if db:
        db.close()

This is the one that uses the new table of yearly statistics and does the difference or change of the anomaly values between GHCN version 3.3 and version 4:

chiefio@PiM3Devuan2:~/Py3/v4$ cat a3v4deltaAS.py 
# -*- coding: utf-8 -*-
import datetime
import pandas as pd
import numpy as np
import matplotlib.pylab as plt
import math
import mysql.connector as MySQLdb

plt.title("GHCN v3 vs v4 Anomaly Difference by Years")
plt.ylabel(" Country AS Australia Anomaly C")
plt.xlabel("Year")
plt.ylim(-2,2)
#plt.ylim(1850,2020)

try:
    db=MySQLdb.connect(user="chiefio",password="LetMeIn!",database='temps')
    cursor=db.cursor()

    
    sql="SELECT year,(mean4-mean3) FROM yrcastats WHERE abrev='AS' AND num3>0 AND num4>0 GROUP BY year;"

    print("stuffed SQL statement")
    cursor.execute(sql)
    print("Executed SQL")
    stn=cursor.fetchall()
#    print(stn)
    data = np.array(list(stn))
    print("Got data")
    xs = data.transpose()[0]   # or xs = data.T[0] or  xs = data[:,0]
    ys = data.transpose()[1]
    print("after the transpose")

    plt.scatter(xs,ys,s=4,color='purple',alpha=1)
    plt.show()

except:
    print ("This is the exception branch")

finally:
    print ("All Done")
    if db:
        db.close()

Subscribe to feed

Advertisement