GIStemp STEP0 data changes profile

Benchmarking Changes In The Data in STEP0 of GIStemp

OK, I’ve settled on a benchmark procedure and characterized the data (that is, we know what the data look like and how they are structured in a general sense.) So how does the pattern of the data change as they move through the STEP0 processing? Does STEP0 introduce significant changes to the data?

I’ll be using the “Decade Averages of Thermometers” code published earlier. The result will be a table of the average temperature in each decade, by month. Realize that this is NOT the actual temperature of the planet in any given time period. It is just a way to see what happens in bulk to the data set as it flows through that part of GIStemp. A bulk measure of the degree of change in the data. The intent is just to see how much and in what way a processing step changes the bulk data.

Part way through STEP0 the data series gets truncated at 1880. We will not be looking at the data from prior to 1880, since it doesn’t change prior to deletion. There were not a lot of temperature records from the 1700s in Antarctica. The number of thermometers in the years prior to 1880 is rather small so the data do not represent the planet surface well.

The Base Case

This is the base case of the benchmark. The benchmark code is run on the input dataset: v2.mean from NOAA.

DecadeAV: 1889  
 0.4  1.9  5.2 10.6 15.3 18.8 21.0 20.3 17.2 12.0  6.5  2.7 11.0
DecadeAV: 1899 
-0.2  0.9  4.7 10.6 15.3 19.4 21.4 20.8 17.5 12.0  6.0  1.7 10.8
DecadeAV: 1909  
 0.9  1.3  6.0 10.6 15.2 18.9 21.0 20.6 17.3 12.4  6.8  2.0 11.1
DecadeAV: 1919  
 0.9  1.9  6.1 11.0 14.9 18.6 20.8 20.1 16.9 12.3  6.9  2.2 11.0
DecadeAV: 1929   
 1.0  2.5  6.3 10.7 15.0 18.6 20.7 20.1 17.2 12.5  6.9  2.4 11.2
DecadeAV: 1939  
 1.1  2.2  5.9 10.9 15.4 18.9 21.2 20.6 17.4 12.4  6.8  2.5 11.3
DecadeAV: 1949  
 0.6  1.9  5.8 10.9 15.0 18.4 20.5 20.0 16.9 12.4  6.6  2.2 10.9
DecadeAV: 1959  
 3.1  4.4  7.5 12.2 16.1 19.2 20.9 20.4 17.7 13.5  8.2  4.7 12.3
DecadeAV: 1969  
 3.7  5.1  8.3 12.6 16.2 19.0 20.6 20.2 17.6 13.9  9.2  5.2 12.6
DecadeAV: 1979  
 3.5  5.1  8.4 12.4 15.9 18.8 20.3 19.8 17.3 13.3  8.7  5.0 12.4
DecadeAV: 1989  
 2.7  4.0  7.5 11.9 15.7 18.5 20.4 20.0 17.0 12.8  7.8  3.8 11.8
DecadeAV: 1999  
 4.8  6.4  9.3 13.1 17.0 20.1 21.9 21.5 18.7 14.4  9.2  5.8 13.5
DecadeAV: 2009  
 5.1  6.2  9.7 13.8 17.2 20.3 22.0 21.6 18.9 14.8 10.1  6.2 13.8

STEP0 largely just glues together the base data (GHCN) with some data from Antarctica, and updated set for a single city (Hohenspiessenberg), truncates the data at 1880, and substitutes some USHCN records into GHCN with a different modification history (the history of “corrections” for things like TOBS – Time of Observation, or equipment changes). Along the way it spits out intermediate files (with names like meanx, meany, and meanz). These files are all in GHCN format and all can be processed in the same general way.

Adding Antarctica

The first major process is adding in the Antarctic records. We would expect this to lower the average of thermometers by a small amount. While antarctic temperatures are low, there are fairly few records. The result is:

DecadeAV: 1889  
 0.4  1.9  5.2 10.6 15.3 18.8 21.0 20.3 17.2 12.0  6.5  2.7 11.0
DecadeAV: 1899
-0.2  0.9  4.7 10.6 15.3 19.4 21.4 20.8 17.5 12.0  6.0  1.7 10.8
DecadeAV: 1909  
 0.9  1.3  6.0 10.6 15.2 18.8 21.0 20.5 17.3 12.4  6.9  2.1 11.1
DecadeAV: 1919  
 1.0  1.9  6.1 11.0 14.9 18.5 20.7 20.0 16.9 12.3  6.9  2.2 11.0
DecadeAV: 1929  
 1.1  2.5  6.3 10.7 15.0 18.5 20.7 20.1 17.1 12.4  7.0  2.5 11.2
DecadeAV: 1939  
 1.1  2.2  5.9 10.9 15.3 18.9 21.2 20.6 17.3 12.4  6.8  2.5 11.3
DecadeAV: 1949  
 0.7  2.0  5.8 10.9 15.0 18.4 20.5 20.0 16.9 12.4  6.6  2.2 11.0
DecadeAV: 1959  
 3.2  4.5  7.5 12.2 16.0 19.1 20.8 20.4 17.6 13.4  8.2  4.7 12.3
DecadeAV: 1969  
 3.8  5.1  8.3 12.5 16.1 18.9 20.5 20.0 17.5 13.8  9.2  5.2 12.6
DecadeAV: 1979  
 3.5  5.1  8.4 12.3 15.8 18.6 20.1 19.7 17.1 13.2  8.7  5.0 12.3
DecadeAV: 1989  
 2.7  4.0  7.3 11.8 15.5 18.3 20.1 19.7 16.8 12.6  7.7  3.8 11.7
DecadeAV: 1999  
 4.7  6.0  8.7 12.2 16.0 19.0 20.8 20.5 17.8 13.7  8.8  5.6 12.8
DecadeAV: 2009  
 4.9  5.9  9.2 13.1 16.5 19.4 21.1 20.7 18.2 14.2  9.7  6.0 13.2

And what we find is substantially what we expected. Not much changes at all, with only a few cells moving down, and then only in the 1/10 C position and often by only 1/10C. The changes seem much stronger in the final decade than in the early years, which is most likely due to more thermometers in later years in Antarctica.

For example, look at the 3rd column: March. The series are identical down to the 1989 decade ending values of 7.5 vs 7.3 respectively. Not much of an impact. We still have the pronounced N.H. pattern to the months and the warming signal is still concentrated into the N.H. Winter months.

OK, on to the next one.

Deleting Prior to 1880

DecadeAV: 1889  
 0.4  1.9  5.2 10.6 15.3 18.8 21.0 20.3 17.2 12.0  6.5  2.7 11.0 1224
DecadeAV: 1899 
-0.2  0.9  4.7 10.6 15.3 19.4 21.4 20.8 17.5 12.0  6.0  1.7 10.8 1972
DecadeAV: 1909  
 0.9  1.3  6.0 10.6 15.2 18.8 21.0 20.5 17.3 12.4  6.9  2.1 11.1 2549
DecadeAV: 1919  
 1.0  1.9  6.1 11.0 14.9 18.5 20.7 20.0 16.9 12.3  6.9  2.2 11.0 2912
DecadeAV: 1929  
 1.1  2.5  6.3 10.7 15.0 18.5 20.7 20.1 17.1 12.4  7.0  2.5 11.2 3347
DecadeAV: 1939  
 1.1  2.2  5.9 10.9 15.3 18.9 21.2 20.6 17.3 12.4  6.8  2.5 11.3 3992
DecadeAV: 1949  
 0.7  2.0  5.8 10.9 15.0 18.4 20.5 20.0 16.9 12.4  6.6  2.2 11.0 5298
DecadeAV: 1959  
 3.2  4.5  7.5 12.2 16.0 19.1 20.8 20.4 17.6 13.4  8.2  4.7 12.3 7610
DecadeAV: 1969  
 3.8  5.1  8.3 12.5 16.1 18.9 20.5 20.0 17.5 13.8  9.2  5.2 12.6 9440
DecadeAV: 1979  
 3.5  5.1  8.4 12.3 15.8 18.6 20.1 19.7 17.1 13.2  8.7  5.0 12.3 9031
DecadeAV: 1989  
 2.7  4.0  7.3 11.8 15.5 18.3 20.1 19.7 16.8 12.6  7.7  3.8 11.7 8006
DecadeAV: 1999  
 4.7  6.0  8.7 12.2 16.0 19.0 20.8 20.5 17.8 13.7  8.8  5.6 12.8 2842
DecadeAV: 2009  
 4.9  5.9  9.2 13.1 16.5 19.4 21.1 20.7 18.2 14.2  9.7  6.0 13.2 1570                                     

Nothing changes. Hardly a surprise. But I’ve added a “count of active thermometers” field so we can at least see the number of active thermometers in each record.

Merging USHCN with GHCN and adjusting for history

The code claims to be replacing USHCN station data in the input file by USHCN_noFIL data (Tobs+maxmin adj+SHAPadj+noFIL) where available and converting any USHCN sourced records to v2.mean format. This is also where the F to C conversion happens. I’ve documented a small “warming” of the data from this F to C conversion earlier (1/1000 C). The code now uses my “improved” method that does not warm the data in that step; so we are measuring all the other impacts here.

DecadeAV: 1889  
 0.6  1.9  5.1 10.3 14.9 18.5 20.6 20.0 16.9 11.8  6.6  2.7 10.8 1150
DecadeAV: 1899 
-0.3  0.7  4.6 10.4 15.1 19.3 21.2 20.6 17.4 11.9  5.9  1.5 10.7 1916
DecadeAV: 1909  
 0.7  1.1  5.8 10.4 15.0 18.7 20.9 20.4 17.2 12.2  6.7  1.9 10.9 2528
DecadeAV: 1919  
 0.8  1.7  5.9 10.8 14.7 18.4 20.5 19.9 16.8 12.1  6.7  2.0 10.9 2898
DecadeAV: 1929  
 0.9  2.3  6.0 10.4 14.8 18.4 20.5 19.9 17.0 12.3  6.8  2.2 11.0 3333
DecadeAV: 1939 
 0.9  2.0  5.7 10.7 15.2 18.8 21.1 20.4 17.2 12.3  6.6  2.3 11.1 3981
DecadeAV: 1949  
 0.5  1.8  5.7 10.8 14.9 18.3 20.4 19.9 16.8 12.3  6.5  2.1 10.8 5291
DecadeAV: 1959  
 3.1  4.4  7.4 12.1 16.0 19.1 20.8 20.3 17.6 13.4  8.1  4.7 12.2 7609
DecadeAV: 1969  
 3.7  5.1  8.2 12.5 16.1 18.8 20.4 20.0 17.4 13.8  9.1  5.1 12.5 9440
DecadeAV: 1979  
 3.5  5.0  8.4 12.3 15.8 18.6 20.1 19.7 17.1 13.2  8.6  5.0 12.3 9031
DecadeAV: 1989  
 2.7  3.9  7.3 11.7 15.5 18.3 20.1 19.7 16.8 12.6  7.7  3.8 11.7 8006
DecadeAV: 1999  
 4.6  6.0  8.7 12.2 16.0 19.0 20.8 20.5 17.8 13.7  8.8  5.5 12.8 2842
DecadeAV: 2009  
 4.5  5.5  9.1 12.9 16.5 19.5 21.3 20.9 18.2 14.1  9.6  5.8 13.1 1570

Here we do get some changes to the data, but they appear to be mostly minor (1/10 C range) and by cursory examination roughly evenly split between warming and cooling. Number of stations stays the same and, despite my distrust for the methodology and dislike of the idea of blending two disjoint data streams, the effect seems to be largely nil.

I just don’t see any changes in the benchmark result that would point at any significant pathological issue to explore (or even much of an insignificant issue.) There just doesn’t seem to be much “there there”.

As a final step, a private communications “more complete” set of data are substituted in the record of one single city: Hohenspeissenberg. We can pretty much expect the result to be “not much verging on nothing”.

DecadeAV: 1889  
 0.6  1.9  5.1 10.3 14.9 18.5 20.6 20.0 16.9 11.8  6.6  2.7 10.8 1150
DecadeAV: 1899 
-0.3  0.7  4.6 10.4 15.1 19.3 21.2 20.6 17.4 11.9  5.9  1.5 10.7 1916
DecadeAV: 1909  
 0.7  1.1  5.8 10.4 15.0 18.7 20.9 20.4 17.2 12.2  6.7  1.9 10.9 2528
DecadeAV: 1919  
 0.8  1.7  5.9 10.8 14.7 18.4 20.5 19.9 16.8 12.1  6.7  2.0 10.9 2898
DecadeAV: 1929  
 0.9  2.3  6.0 10.4 14.8 18.4 20.5 19.9 17.0 12.3  6.8  2.2 11.0 3333
DecadeAV: 1939  
 0.9  2.0  5.7 10.7 15.2 18.8 21.1 20.4 17.2 12.3  6.6  2.3 11.1 3981
DecadeAV: 1949  
 0.5  1.8  5.7 10.8 14.9 18.3 20.4 19.9 16.8 12.3  6.5  2.1 10.8 5291
DecadeAV: 1959 
 3.1  4.4  7.4 12.1 16.0 19.1 20.8 20.3 17.6 13.4  8.1  4.7 12.2 7609
DecadeAV: 1969  
 3.7  5.1  8.2 12.5 16.1 18.8 20.4 20.0 17.4 13.8  9.1  5.1 12.5 9440
DecadeAV: 1979  
 3.5  5.0  8.4 12.3 15.8 18.6 20.1 19.7 17.1 13.2  8.6  5.0 12.3 9031
DecadeAV: 1989  
 2.7  3.9  7.3 11.7 15.5 18.3 20.1 19.7 16.8 12.6  7.7  3.8 11.7 8007
DecadeAV: 1999 
 4.6  6.0  8.7 12.2 16.0 19.0 20.8 20.5 17.8 13.7  8.8  5.5 12.8 2842
DecadeAV: 2009  
 4.5  5.5  9.1 12.9 16.5 19.5 21.3 20.9 18.2 14.1  9.6  5.8 13.1 1570

And that’s what we get. While 1979 has one more active thermometer in it (8007 vs 8006) the effect in the averages is nil.

So that’s it for STEP0. I’m left to conclude that, despite the process being a bit byzantine, the result is innocuous.

Advertisements

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in AGW GIStemp Specific, Favorites and tagged , , , , , . Bookmark the permalink.