GHCN v4 First Peek

I’m starting the process of examining the GHCN Global Historical Climate Network Version 4 data set. This is just a “first peek” at it to see how much looks different and get an idea what kind of work is ahead. First blush it looks like a lot has changed and it is designed to make comparisons between the “versions” difficult. I hope to fix that…

Where To Get It

I downloaded this set “long hand” from:

https://www1.ncdc.noaa.gov/pub/data/ghcn/v4/

Then unpacked the “un-adjusted” version (that DOES have adjustments in it – they note that the data may come pre-adjusted from the various country BOMs) along with the descriptions and text files. There is also a directory that looks to be individual station graphs for every station. I’m pretty sure they are all “adjusted” graphs so will consistently show warming (as their “adjustments” plot exactly to the CO2 predictions… strange that, all the warming is in the adjustments and they just happen to exactly match the CO2 theory predictions…)

Eventually, after I’ve sized things up, I want to create a “GHCN v4 Analysis Station From Scratch” posting and whenever (if ever?) that happens I’ll put in an rsync command / script that does the download.

The Downloaded Stuff

Here’s what you get:

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4$ ls -l
total 124484
-rw-r–r– 1 chiefio chiefio 3955 Mar 18 17:53 ghcnm-countries.txt
-rw-r–r– 1 chiefio chiefio 6221 Mar 18 17:53 ghcnm-flags.txt
-rw-r–r– 1 chiefio chiefio 40872107 Mar 18 17:55 ghcnm.tavg.latest.qcf.tar.gz
-rw-r–r– 1 chiefio chiefio 42708888 Mar 18 17:50 ghcnm.tavg.latest.qcu.tar.gz
-rw-r–r– 1 chiefio chiefio 43854929 Mar 18 17:55 ghcnm.tavg.latest.qfe.tar.gz
-rw-r–r– 1 chiefio chiefio 11472 Mar 18 17:51 readme.txt
-rw-r–r– 1 chiefio chiefio 4416 Mar 18 17:51 status.txt
drwxr-xr-x 3 chiefio chiefio 68 Mar 18 o17:57 unpacking

I created the “unpacking” directory where I could lookover the .qcu file (that one is the “unadjusted”).

Here’s that record:

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4$ mkdir unpacking
chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4$ cp ghcnm.tavg.latest.qcu.tar.gz unpacking/chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4$ cd unpacking/
chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking$ ls
ghcnm.tavg.latest.qcu.tar.gz
chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking$ gunzip ghcnm.tavg.latest.qcu.tar.gz
chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking$ ls
ghcnm.tavg.latest.qcu.tar

You can see that I only work on a copy of the gziped file, and I uncompress it with the gunzip command in Linux. That gives a “tar” or tape archive format file. That gets “untarred”. First I did it with a “tv” that says “table of contents verbose” to see what I was going to get, then used “xvf” that is “eXtract, Verbosly, from the File” where the file is that .tar file named in the command:

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking$ tar tvf ghcnm.tavg.latest.qcu.tar
-rw-rw-r– jared.rennie/jared.rennie 162513448 2019-03-18 08:36 ./ghcnm.v4.0.0.20190317/ghcnm.tavg.v4.0.0.20190317.qcu.dat
-rw-rw-r– jared.rennie/jared.rennie 1888285 2019-03-18 08:36 ./ghcnm.v4.0.0.20190317/ghcnm.tavg.v4.0.0.20190317.qcu.inv
chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking$ tar xvf ghcnm.tavg.latest.qcu.tar
./ghcnm.v4.0.0.20190317/ghcnm.tavg.v4.0.0.20190317.qcu.dat
./ghcnm.v4.0.0.20190317/ghcnm.tavg.v4.0.0.20190317.qcu.inv

That unpacks into a directory as two files, so I’ll go into that directory (“cd” or Change Directory)

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking$ cd ghcnm.v4.0.0.20190317/
chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking/ghcnm.v4.0.0.20190317$ ls
ghcnm.tavg.v4.0.0.20190317.qcu.dat ghcnm.tavg.v4.0.0.20190317.qcu.inv
chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking/ghcnm.v4.0.0.20190317$ ls -l
total 160556
-rw-r–r– 1 chiefio chiefio 162513448 Mar 18 08:36 ghcnm.tavg.v4.0.0.20190317.qcu.dat
-rw-r–r– 1 chiefio chiefio 1888285 Mar 18 08:36 ghcnm.tavg.v4.0.0.20190317.qcu.inv

You may notice that “owner” and “group” changed to me, since I unpacked it as me, not as the superuser “root”.

The .inv file is the “inventory” and tells you about the stations. That .dat file is the temperatures.

At this point, it is an unpacked version of GHCN v4 “unadjusted”. About 162 MB of data. So what’s it look like?

The Files

Here’s a bit of the inventory file:

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking/ghcnm.v4.0.0.20190317$ head *.inv
ACW00011604  57.7667   11.8667   18.0 SAVE                          
AE000041196  25.3330   55.5170   34.0 SHARJAH_INTER_AIRP            
AEM00041184  25.6170   55.9330   31.0 RAS_AL_KHAIMAH_INTE           
AEM00041194  25.2550   55.3640   10.4 DUBAI_INTL                    
AEM00041216  24.4300   54.4700    3.0 ABU_DHABI_BATEEN_AIR          
AEM00041217  24.4330   54.6510   26.8 ABU_DHABI_INTL                
AEM00041218  24.2620   55.6090  264.9 AL_AIN_INTL                   
AF000040930  35.3170   69.0170 3366.0 NORTH_SALANG                  
AFM00040911  36.7000   67.2000  378.0 MAZAR_I_SHARIF                
AFM00040938  34.2100   62.2280  977.2 HERAT   

I notice first that it starts with 3 letters instead of the Region / Country digits. Then 8 of what looks like the old WMO# but with the 000 in front leads me to think they swapped the “modification” or “near” number to the front and put the 5 digit WMO at the end. (Hopefully they didn’t make the whole thing backwards or ‘all new’ like they did in the v1 to v2 transition).

I’d guess the next two are LAT and LONG then an altitude and station name. Gone are all those “annoying” bits of data like nearness to water, airport flag, urban /rural flag, vegetation type and other things that would show they have largely gone to thermometers at airports in urban areas near or over pavement…

So right out the gate there will be “some assembly required” to match these up with the prior data. I’m pretty sure I can work that out. One table with country flag number and country flag text to match them up (about 160 lines – probably manual to create it) and perhaps another with WMO# to Station Name by version to match all of them up. So 7280 lines in the v2 / v3 case, more for this one as they have a bunch of USHCN and Germany added (rest of world not so much…)

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking/ghcnm.v4.0.0.20190317$ wc -l *.inv
27361 ghcnm.tavg.v4.0.0.20190317.qcu.inv

So 27,361 instrument records now? OK… It will be interesting looking at a global map of them. From other sources I’ve heard they were mostly added in the USA and Germany. I may need to move off of the Raspberry Pi with that data size. It is already 10 minutes for some long reports. Make that 4 x as long we’re talking 40 minutes? So I’ll plan on a validation of times on the Pi M3 but expect I’ll need one of the USB 3.0 boards with faster processors to make this really fast.

I figure that’s about 2 weeks of full time work to get ‘er done, so probably a few months elapsed time (unless someone can scare up a “Grant” to pay me a salary… otherwise I have to put a roof on the house this spring… yeah, me, on the roof, with a hammer…)

So, moving on to the “data” file:

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking/ghcnm.v4.0.0.20190317$ head *.dat
ACW000116041961TAVG -142  k  183  k  419  k  720  k 1075  k 1546  k 1517  k 1428  k 1360  k 1121  k  457  k  -92  k
ACW000116041962TAVG   60  k   32  k -207  k  582  k  855  k 1328  k 1457  k 1340  k 1110  k  941  k  270  k -179  k
ACW000116041963TAVG -766  k -606  k -152  k  488  k 1171  k 1574  k 1567  k 1543  k 1279  k  887  k  513  k -161  k
ACW000116041964TAVG    9  k -138  k    2  k  685  k 1166  k 1389  k 1453  k 1504  k 1168  k  735  k  493  k   59  k
ACW000116041965TAVG   -9  k -158  k  -15  k  537  k  934  k 1447  k 1434  k 1424  k 1324  k  921  k  -22  k -231  k
ACW000116041966TAVG -490  k -614  k  108  k  246  k 1082  k 1642  k 1620  k 1471  k 1195  k  803  k  329  k    2  k
ACW000116041967TAVG -270  k   36  k  397  k  481  k 1052  k 1373  k 1655  k 1598  k 1318  k  997  k  559  k  -96  k
ACW000116041968TAVG -306  k -183  k  220  k  714  k  935  k 1635  k 1572  k 1718  k 1331  k  781  k  180  k  -56  k
ACW000116041969TAVG -134  k -494  k -185  k  497  k  962  k 1634  k 1687  k 1773  k 1379  k  932  k  321  k -275  k
ACW000116041970TAVG -483  k -704  k  -75  k  261  k 1093  k 1724  k 1470  k 1609  k 1163  k  836  k  300  k   73  k

So 12 sets of “k” (or some flag for who knows what) and what looks like a temperature as 1/100 C. Cheeky little bastards to think they have clue to 1/100 C. That takes exceptional calibration and a lab; not some bit of gear stuck out by the runway at Dallas Ft. Worth and ignored for a year+. But, OK, I can use my basic data prep-loading FORTRAN program to get that loaded. Now, for the front of the record…

We’ve got 3 Letters, then what looks like 8 digits of WMO# + “near” flag (again with the minor number ‘near’ part in front instead of in the back as they did in the past), then what looks like a 4 digit year and a flag to tell you type (in this case Temperature Average).

I think I can work with this lot too. Mostly it will be making a map of the 3 letter flag to the 3 digit Region / Country of the past. A manual task, but not too many countries nor continents to deal with.

The Descriptor Files

It comes with files that describes all this stuff. I generally like to just “eyeball” things first, then check it against the documentation. I can find more errors in the documentation that way (if any) and at the same time check my quality of instinct.

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4$ wc -l *.txt
  238 ghcnm-countries.txt
  125 ghcnm-flags.txt
  274 readme.txt
  103 status.txt
  740 total
chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4$ 

None of these are so big as to be horrible. 238 Countries? Didn’t know there were that many… Maybe some are sub-sets like overseas departments of France and such? The “status” file is a list of things that changed as they made the set. Announcement of ‘beta’ availability and such. I’ll leave it for last. Flags ought to be interesting. The Readme can be helpful too. So I’ll start with them.

Flags File

Here’s the flags:

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4$ cat *flags*
GHCN-Monthly, Version 4, Flag File 

This file describes all of the data source flags used in GHCN-Monthly
version 4. They are formatted as follows:

Column1: Alpha-Numeric Flag
Column2: Dataset Origin (Note that some sources exist in more than one dataset)
             GHCND: Global Historical Climatology Network: Daily Dataset
                    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/
             GHCNM: Global Historical Climatology Network: Monthly Dataset
                    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/
             ISTI:  International Surface Temperature Initiative 
                    ftp://ftp.ncdc.noaa.gov/pub/data/globaldatabank/

Column3: Description of Flag

********************************************************************************
0    GHCND       U.S. Cooperative Summary of the Day (NCDC DSI-3200)
6    GHCND       CDMP Cooperative Summary of the Day (NCDC DSI-3206)
7    GHCND       U.S. Cooperative Summary of the Day -- Transmitted via WxCoder3 
                 (NCDC DSI-3207)
A    GHCND       U.S. Automated Surface Observing System (ASOS) real-time data 
                 (since January 1, 2006)
a    GHCND       Australian data from the Australian Bureau of Meteorology
B    GHCND       U.S. ASOS data for October 2000-December 2005 (NCDC DSI-3211)
b    GHCND       Belarus update
C    GHCND       Environment Canada
E    GHCND       European Climate Assessment and Dataset (Klein Tank et al., 2002)	   
G    GHCND       Official Global Climate Observing System (GCOS) or other 
                 government-supplied data
H    GHCND       High Plains Regional Climate Center real-time data
I    GHCND       International collection (non U.S. data received through
                 personal contacts)
K    GHCND       U.S. Cooperative Summary of the Day data digitized frompaper 
                 observer forms (from 2011 to present)
M    GHCND       Monthly METAR Extract (additional ASOS data)
N    GHCND       Community Collaborative Rain, Hail,and Snow (CoCoRaHS)
Q    GHCND       Data from several African countries that had been "quarantined", 
                 that is, withheld from public releaseuntil permission was granted 
                 from the respective meteorological services
R    GHCND       NCDC Reference Network Database (Climate Reference Network and 
                 Historical Climatology Network-Modernized)
r    GHCND       All-Russian Research Institute of Hydrometeorological Information
                 World Data Center
S    GHCND       Global Summary of the Day (NCDC DSI-9618)
s    GHCND       China Meteorological Administration/National Meteorological 
                 Information Center/Climatic Data Center (http://cdc.cma.gov.cn)
T    GHCND       SNOwpack TELemtry (SNOTEL) data obtained from the Western 
                 Regional Climate Center
U    GHCND       Remote Automatic Weather Station (RAWS) data obtainedfrom the 
                 Western Regional Climate Center	   
u    GHCND       Ukraine update	   
W    GHCND       WBAN/ASOS Summary of the Day from NCDC's Integrated Surface 
                 Data (ISD).  
X    GHCND       U.S. First-Order Summary of the Day (NCDC DSI-3210)
z    GHCND       Uzbekistan update
Z    GHCND/GHCNM Datzilla
m    ISTI        mexico                        
V    ISTI        vietnam                       
c    ISTI        channel-islands               
e    ISTI        ecuador                       
p    ISTI        pitcairnisland                
j    ISTI        giessen                       
i    ISTI        brazil-inmet                  
1    ISTI        brazil                        
2    ISTI        argentina                     
4    ISTI        india                         
5    ISTI        gsn-sweden                    
8    ISTI        canada-raw                    
9    ISTI        east-africa                   
!    ISTI        uganda                        
#    ISTI        antarctica-aws                
$    ISTI        antarctica-palmer             
%    ISTI        antarctica-southpole          
&    ISTI        ispd-swiss                    
(    ISTI        ispd-ipy                      
)    ISTI        ispd-sydney                   
*    ISTI        antarctica-scar-reader        
+    ISTI        spain                         
-    ISTI        uruguay-inia                  
.    ISTI        uruguay                       
d    ISTI        swiss-digihom                 
/    ISTI        ispd-tunisia-morocco          
:    ISTI        sacad_non-blended             
;    ISTI        japan                         
    ISTI        russsource-australia_de       
^    ISTI        russsource-australia_wwr      
_    ISTI        russsource-ghcn               
'    ISTI        russsource-climat             
`    ISTI        russsource-conus_climat       
~    ISTI        russsource-ak_hi_climat       
?    ISTI        germany                       
w    ISTI        wmssc                         
@    ISTI        central-asia                  
[    ISTI        arctic                        
h    ISTI        histalp                       
]    ISTI        hadisd 
F    ISTI/GHCND  usforts 
D    ISTI/GHCNM  mcdw-unpublished
"    ISTI/GHCNM  ghcnsource
J    ISTI/GHCNM  colonialera
|    ISTI/GHCNM  climat-uk 
}    ISTI/GHCNM  mcdw
k    ISTI/GHCNM  knmi
P    ISTI/GHCNM  climat-ncdc
{    ISTI/GHCNM  wwr
,    ISTI/GHCNM  climat-bufr
3    ISTI/GHCNM  climat-prelim

So, was that that “K”? Well, I’ll sort that out later. What is clear is that there’s a lot of sources and they used funny letters to code them. Got it. Other than that, not so useful.

README File

How about the README? I’ve bolded some bits of interest. Looks like I’m generally right on my guesses.

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4$ cat readme.txt 
GLOBAL HISTORICAL CLIMATOLOGY NETWORK - MONTHLY (GHCN-M) Version 4
(Last Updated: 10/26/2018)

********************************************************************************

The underlying data, its software and all content are provided on an 'as is' 
and 'as available' basis. We do not give any warranties, whether express 
or implied, as to the suitability or usability of the product, 
its software or any of its content.

Feedback is welcome and should be sent via email to:
ncdc.ghcnm@noaa.gov

Should you have any questions or concerns regarding the product,
please let us know immediately so we can rectify these accordingly.
Your help in this regard is greatly appreciated.

********************************************************************************

1. INTRODUCTION

    1.1 OVERVIEW

        GHCN-M version 4 currently contains monthly mean temperature for over
        25,000 stations across the globe. In large part GHCN-M version 4 uses
        the same quality control and bias correction algorithms as version 3. 
        The greatest difference from previous version is a greatly expanded 
        set of stations based on the large data holdings in GHCN-Daily as well 
        as data collected as part of the International Surface Temperatue 
        Initiative databank (ISTI; Rennie et al. 2013).  

    1.2 INTERNET ACCESS

        The GHCNM v4 product can be found here:

        FTP: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v4

    1.3 DOWNLOADING AND INSTALLING

        WINDOWS (example):

	GHCNM files are compressed into gzip format.  For more information on gzip, 
        please see the following web site:

        www.gzip.org

        and for potential software that can compress/decompress gzip files, please see:

        www.gzip.org/#faq4


        LINUX (example):

        wget ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v4/ghcnm.latest.qcu.tar.gz
        tar -zxvf ghcnm.latest.qcu.tar.gz

        (alternatively, if "tar" does not support decompression, a user can try:

        wget ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v4/ghcnm.latest.qcu.tar.gz
        gzip -d ghcnm.latest.qcu.tar.gz
        tar -xvf ghcnm.latest.qcu.tar.gz)


        Note: the data are placed in their own separate directory, that is named
              according to the following specification:

              ghcnm.v4.x.y.YYYYMMDD where 

              x = integer to be incremented with major data additions 
              y = integer to be incremented with minor data additions
              YYYY = year specific dataset was processed and produced
              MM   = month specific dataset was processed and produced
              DD   = day specific dataset was processed and produced
 
              Two files (per element) should be present in the directory a 
              1) metadata and 2) data file. Note: there will be no 
              increments to "x" and "y" above during the phase.

2. DATA

    2.1 METADATA

       The metadata has been carried over from GHCN-Monthly v3.  This would 
       include basic geographical station information such as latitude, 
       longitude, elevation, station name, etc. The extended metadata
       information, such as surrounding vegetation, etc. was not carried 
       over, however still exists in version 3

    2.1.1 METADATA FORMAT (.inv file)

       Variable          Columns      Type
       --------          -------      ----

       ID                 1-11        Integer
       LATITUDE          13-20        Real
       LONGITUDE         22-30        Real
       STNELEV           32-37        Real
       NAME              39-68        Character

       Variable Definitions:

       ID: Station identification code. First two characters are FIPS country code

       LATITUDE: latitude of station in decimal degrees

       LONGITUDE: longitude of station in decimal degrees

       STELEV: is the station elevation in meters. -999.0 = missing.

       NAME: station name

    2.2  DATA (.dat file)

         The data within GHCNM v4 for the time being consist of monthly
         average temperature.

    2.2.1 DATA FORMAT

          Variable          Columns      Type
          --------          -------      ----

          ID                 1-11        Integer
          YEAR              12-15        Integer
          ELEMENT           16-19        Character
          VALUE1            20-24        Integer
          DMFLAG1           25-25        Character
          QCFLAG1           26-26        Character
          DSFLAG1           27-27        Character
            .                 .             .
            .                 .             .
            .                 .             .
          VALUE12          108-112       Integer
          DMFLAG12         113-113       Character
          QCFLAG12         114-114       Character
          DSFLAG12         115-115       Character

          Variable Definitions:

          ID: Station identification code. First two characters are FIPS country code

          YEAR: 4 digit year of the station record.
 
          ELEMENT: element type, monthly mean temperature="TAVG"

          VALUE: monthly value (MISSING=-9999).  Temperature values are in
                 hundredths of a degree Celsius, but are expressed as whole
                 integers (e.g. divide by 100.0 to get whole degrees Celsius).

          DMFLAG: data measurement flag, nine possible values for QCU/QCF and one for QFE:

                  Quality Controlled Unadj/Adj (QCU/QCF) Flags:

                  blank = no measurement information applicable
                  a-i = number of days missing in calculation of monthly mean
                        temperature (currently only applies to the 1218 USHCN
                        V2 stations included within GHCNM)

                  Quality Controlled Adj and estimated (QFE) Flags:
                  E = Data has been estimated from neighbors within the PHA

          QCFLAG: quality control flag, eleven possibilities within
                  quality controlled unadjusted (qcu) dataset, and three 
                  possibilities within the quality controlled adjusted (qcf) 
                  dataset.

                  Quality Controlled Unadjusted (QCU) QC Flags:
         
                  BLANK = no failure of quality control check or could not be
                          evaluated.

                  Note: the following QC checks are listed in order of execution. When
                        a value is flagged by a QC check it is not subjected to 
                        testing in subsequent checks.

                  E = Identify different stations that have the "same" 12 
                      monthly values for a given year (e.g. when all 12 months
                      are duplicated and there are at least 3 or more non-
                      missing data months, and furthermore values are considered 
                      the "same" when the absolute difference between the two values 
                      is less than or equal to 0.015 deg C)

                  D = monthly value is part of an annual series of values that
                      are exactly the same (e.g. duplicated) within another
                      calendar year in the station's record.

                  R = Flag values greater than or less than known world 
                      TAVG extremes.

                  K = Identifies and flags runs of the same value (non-missing)
                      in five or more consecutive months

                  W = monthly value is duplicated from the previous month,
                      based upon regional and spatial criteria (Note: test is only 
                      applied from the year 2000 to the present, and in general
                      only for near real time produced data sources).                   

                  I = checks for internal consistency between TMAX and TMIN. 
                      Flag is set when TMIN > TMAX for a given month. 

                  L = monthly value is isolated in time within the station
                      record and flagged, when:

                      1) a non-missing value has at least 18 missing values
                         before AND after in time., or

                      2) a non-missing value belongs to a "cluster" of 2
                         adjacent (in time) non-missing values, and the cluster of
                         values has at least 18 missing values before AND after the
                         cluster, or

                      3) a non-missing value belongs to a "cluster" of 3
                         adjacent (in time) non-missing values, and the cluster of
                         values has at least 18 missing values before AND after the
                         cluster.

                  O = monthly value that is >= 5 bi-weight standard deviations
                      from the bi-weight mean.  Bi-weight statistics are
                      calculated from a series of all non-missing values in 
                      the station's record for that particular month.

                  S = Flags value when the station z-score satisfies any of the
                      following algorithm conditions.

                      Definitions:

                      neighbor = any station within 500 km of target station.
                      zscore = (bi-weight standard deviation / bi-weight mean)
                      S(Z) = station's zscore
                      N(Z) = the set of the "5" closest non-missing neighbor zscores.
                             (Note: this set may contain less than 5 neighbors,
                             but must have at least one neighbor zscore for
                             algorithm execution)

                      Algorithm:

                      S(Z) >= 4.0 and < 5.0 and "all" N(Z) = 3.0 and < 4.0 and "all" N(Z) = 2.75 and < 3.0 and "all" N(Z) = 2.5 and < 2.75 and "all" N(Z) < 1.6
                      S(Z)  -5.0 and "all" N(Z) > -1.9
                      S(Z)  -4.0 and "all" N(Z) > -1.8
                      S(Z)  -3.0 and "all" N(Z) > -1.7
                      S(Z)  -2.75 and "all" N(Z) > -1.6
                                   
                  T = Identifies and flags when the temperature z-score compared 
                      to the inverse distance weighted z-score of all neighbors 
                      within 500 km (at least 2 or more neighbors are required)
                      is greater than or equal to 3.0. 
                 

                  M = Manually flagged as erroneous.

                  Quality Controlled Adjusted (QCF) QC Flags:

                  A = alternative method of adjustment used.
 
                  M = values with a non-blank quality control flag in the "qcu"
                      dataset are set to missing the adjusted dataset and given
                      an "M" quality control flag.

                  X = pairwise algorithm removed the value because of too many
                      inhomogeneities.


          DSFLAG: data source flag for monthly value

                  For more information on data source flags, please refer to
                  ghcnm-flags.txt 

3. CONTACT

    3.1 QUESTIONS AND FEEDBACK

        NCDC.GHCNM@noaa.gov

So looks like 3 flags in a row around that “k” and I’ll need to get a finer grain reading on just what row that is to know what flag it is. Got it.

I also note they say it is an 11 “digit” value starting in column 1, but also that the first two characters are the FIPS Country Code. Looks like a documentation error to me. They have 3 CHAR and then 8 INT so “some interpretation required”… I’ll work that out when I do a detailed read of the description and match the 3 CHAR to “whatever”…

Taking a peek at the bottom of the data file is not encouraging:

chiefio@PiM3Devuan2:/WDTB/xfs/GHCNv4/unpacking/ghcnm.v4.0.0.20190317$ tail *.dat
ZIXLT6221161961TAVG 1965  J 1945  J 2004  J 1850  J 1690  J 1310  J 1240  J 1320  J 1785  J 2065  J 1950  J 2000  J
ZIXLT6221161962TAVG 2035  J 2035  J 1900  J 1835  J 1315  J 1395  J-9999   -9999   -9999   -9999   -9999   -9999   
ZIXLT6221161963TAVG-9999   -9999   -9999   -9999   -9999    1265  J-9999   -9999   -9999   -9999   -9999   -9999   
ZIXLT6221161964TAVG-9999   -9999   -9999   -9999   -9999    1120  J-9999   -9999   -9999   -9999   -9999   -9999   
ZIXLT6221161965TAVG-9999   -9999   -9999   -9999   -9999    1150  J 1200  " 1420  " 1660  " 1840  " 2040  " 2220  "
ZIXLT6221161966TAVG 2180  " 2040  " 1840  " 1690  " 1430  " 1280  " 1210  " 1460  " 1770  " 1980  " 2090  " 2110  "
ZIXLT6221161967TAVG 2110  " 1990  " 1890  " 1920  " 1510  " 1350  " 1100  " 1380  " 1660  " 2080  " 1990  " 1910  "
ZIXLT6221161968TAVG 2180  " 2000  " 1930  " 1820  " 1560  " 1080  " 1370  " 1630  " 1760  " 2180  " 1840  " 2070  "
ZIXLT6221161969TAVG 2090  " 2150  " 1950  " 1830  " 1410  " 1310  " 1160  " 1460  " 1780  " 2100  " 2040  " 1910  "
ZIXLT6221161970TAVG 2070  " 1990  " 1930  " 1720  " 1560  " 1330  " 1330  " 1540  " 2040  " 2030  " 2130  " 2150  "

It has 5 leading CHAR and then 6 INT of what looks like WMO#. My guess would be the leading 5 CHAR are a complicated code and not just a “minor station number near a WMO” as in the past. OK, harder to work with and match up, but not impossible. IF as described, the first two are Country, and then the last 5 are WMO# (a fella can hope can’t he?) that just leaves 4 of “mystery meat” to work out inside the “ID” of 11 “Integer” that is really including characters.

But at least now you see why I don’t just blindly trust the documentation right out the gate…

So again “some assembly required”…

I’ll need to ponder a bit how to unscramble the “Region / Country” business. The basic problem being that countries change over time (is Crimea Ukraine or Russia right now?…) and really it ought not to be part of the data structure. Oh Well. Maybe just a table of WMO and the FIPS country and the 3 digit v2 / v3 country and then the v1 country? IIRC v1 WMO was a different length, so might need it in the table too to show that mapping.

IF the WMO Number can be used as the Primary Key, that would be ideal, then make all the rest of the metadata and country code(s) dependent values. That ought to straighten out the inventory data.

Having the bulk of all stations devoid of things like Airstation Flag and distance to water will make it impossible to do that kind of report, but I can still do altitude and lat / long. Thin gruel that…

From here on down I’ve included, without comment, the “Status” and “Countries” files, just as simple documentation. I think I have a handle on what I need to do to suck this into a MariaDB database and compare it with GHCN v3. So I’ll start working on that now. The rest of this can “catch up” ;-)

Status File

GHCN-Monthly, Version 4, Status File 

(users can use this file to determine overall current status, including 
information related to previous changes and errata).

DISCLAIMER: The status file does not record every possible change that may
have occurred from one version to another.  This would be impractical for a
variety of reasons. Therefore if a user is interested in discovering every 
possible difference between two different file versions, they will need to 
construct their own program to determine all possible changes between the file
versions. 

********************************************************************************

10/26/2018

GHCN-M version 4.0.0 is now operational. Issues may still arise during the 
first few days of operations. For any questions, please email
ncdc.ghcnm@noaa.gov

********************************************************************************

09/24/2018

With the early online release of the GHCN-M v4 paper, this product is being 
transitioned from a public beta (v4.b.2) to operations (v4.0.0). During this
transition, issues may arise with data output. For questions, please email
ncdc.ghcnm@noaa.gov 

********************************************************************************

02/16/2018

Beginning with the 20180215 run, a third version of v4 beta will be produced.
In addition to QCU and QCF, a QFE inventory and data file will be generated.
The QFE includes estimated data for stations for the period of 1961-1990. Data
are estimated based upon neighboring station info in the Pairwise Homogeneity
Algorithm (PHA). Data is only provided between 1961 and 1990, and will be used
to assess climate trends.

********************************************************************************

11/22/2017

An update to the public BETA of GHCN-M version 4 has been made available, known
as v4.b.2. Changes include:

    - Incorporating the update from the ISTI Databank, now at version 1.1.1.
    - Incorporating changes and updates to the quality control procedure. For
      more information about the flags, please refer to the readme file.
    - Other minor bug fixes, as pointed out from the user community.

Please remember this is still a beta version of GHCN-M version 4, which is 
undergoing final testing before its official release. The underlying
data, its software and all content are provided on an 'as is' and 'as available'
basis. We do not give any warranties, whether express or implied, as to the
suitability or usability of the product, its software or any of its content.

Feedback is welcome and should be sent via email to:
ncdc.ghcnm@noaa.gov

Should you have any questions or concerns regarding the product,
please let us know immediately so we can rectify these accordingly.
Your help in this regard is greatly appreciated.

********************************************************************************

07/20/2017

Beginning this week, v4 beta will begin ingesting new data and adjusting it 
through the Pairwise Homogeneity Algorithm. This update essentially reverts the
change that was made on March 20th, 2017.

********************************************************************************

03/20/2017

Starting with the 20170310 run, v4 Beta will stop adjusting new data ingested. 
Instead, quality controlled unadjusted (QCU) data will be appended
to both QCU and QCF files of v4 beta. During the first month of the new year,
the Pairwise Homogenity Adjustment (PHA) will be run again to adjust the datasets
entire period of record. 

********************************************************************************

10/15/2015

A public BETA of GHCN-M version 4 has been made available,
effective 10/15/2015. This is the first beta distribution, and its version
is known as v4.b.1

Please note that this is a beta version of GHCN-M version 4, which is
still undergoing final testing before its official release. The underlying
data, its software and all content are provided on an 'as is' and 'as available' 
basis. We do not give any warranties, whether express or implied, as to the 
suitability or usability of the product, its software or any of its content.

Feedback is welcome and should be sent via email to:
ncdc.ghcnm@noaa.gov

Should you have any questions or concerns regarding the product,
please let us know immediately so we can rectify these accordingly.
Your help in this regard is greatly appreciated.

Countries File

AC Antigua and Barbuda 
AE United Arab Emirates 
AF Afghanistan
AG Algeria 
AJ Azerbaijan 
AL Albania
AM Armenia 
AO Angola 
AQ American Samoa [United States] 
AR Argentina 
AS Australia 
AU Austria 
AY Antarctica 
BA Bahrain 
BB Barbados 
BC Botswana 
BD Bermuda [United Kingdom] 
BE Belgium 
BF Bahamas, The 
BG Bangladesh
BH Belize
BK Bosnia and Herzegovina 
BL Bolivia 
BM Burma
BN Benin 
BO Belarus 
BP Solomon Islands 
BR Brazil
BU Bulgaria
BX Brunei
BY Burundi
CA Canada 
CB Cambodia
CD Chad 
CE Sri Lanka 
CF Congo (Brazzaville) 
CG Congo (Kinshasa)
CH China  
CI Chile 
CJ Cayman Islands [United Kingdom] 
CK Cocos (Keeling) Islands [Australia] 
CM Cameroon 
CN Comoros
CO Colombia 
CQ Northern Mariana Islands [United States] 
CS Costa Rica 
CT Central African Republic 
CU Cuba 
CV Cape Verde
CW Cook Islands [New Zealand]
CY Cyprus 
DA Denmark 
DJ Dijibouti
DO Dominica
DR Dominican Republic 
EC Ecuador 
EG Egypt 
EI Ireland 
EK Equatorial Guinea
EN Estonia 
ER Eritrea 
ES El Salvador 
ET Ethiopia 
EU Europa Island [France]
EZ Czech Republic 
FG French Guiana [France] 
FI Finland 
FJ Fiji
FK Falkland Islands (Islas Malvinas) [United Kingdom] 
FM Federated States of Micronesia 
FP French Polynesia 
FR France 
FS French Southern and Antarctic Lands [France]
GA Gambia, The 
GB Gabon 
GG Georgia
GH Ghana
GI Gibraltar [United Kingdom]
GJ Grenada
GK Guernsey
GL Greenland [Denmark] 
GM Germany 
GP Guadeloupe [France] 
GQ Guam [United States] 
GR Greece 
GT Guatemala 
GV Guinea 
GY Guyana 
HA Haiti
HK Hong Kong
HO Honduras 
HR Croatia
HU Hungary 
IC Iceland 
ID Indonesia 
IM Isle of Man
IN India 
IO British Indian Ocean Territory [United Kingdom] 
IR Iran 
IS Israel 
IT Italy 
IV Cote D'Ivoire 
IZ Iraq 
JA Japan 
JE Jersey
JM Jamaica 
JN Jan Mayen [Norway] 
JO Jordan
JQ Johnston Atoll [United States] 
JU Juan De Nova Island [France]
KE Kenya 
KG Kyrgyzstan 
KN Korea, North 
KR Kiribati 
KS Korea, South 
KT Christmas Island [Australia] 
KU Kuwait 
KZ Kazakhstan 
LA Laos 
LE Lebanon
LG Latvia 
LH Lithuania
LI Liberia 
LO Slovakia 
LQ Palmyra Atoll [United States] 
LS Liechtenstein
LT Lesotho 
LU Luxembourg 
LY Libya 
MA Madagascar 
MB Martinique [France]
MC Macau S.A.R
MD Moldova 
MF Mayotte [France]
MG Mongolia 
MH Montserrat
MI Malawi
MJ Montenegro 
MK Macedonia 
ML Mali 
MO Morocco 
MP Mauritius 
MQ Midway Islands [United States} 
MR Mauritania 
MT Malta 
MU Oman 
MV Maldives 
MX Mexico 
MY Malaysia 
MZ Mozambique 
NC New Caledonia [France] 
NE Niue [New Zealand]
NF Norfolk Island [Australia]
NG Niger 
NH Vanuatu 
NI Nigeria
NL Netherlands 
NO Norway 
NP Nepal 
NR Nauru
NS Suriname
NT Netherlands Antilles [Netherlands]
NU Nicaragua 
NZ New Zealand 
PA Paraguay 
PC Pitcairn Islands [United Kingdom] 
PE Peru 
PK Pakistan 
PL Poland 
PM Panama 
PO Portugal 
PP Papua New Guinea 
PS Palau
PU Guinea-Bissau
QA Qatar
RE Reunion [France]
RI Serbia
RM Marshall Islands 
RO Romania 
RP Philippines 
RQ Puerto Rico [United States] 
RS Russia 
RW Rwanda
SA Saudi Arabia 
SB Saint Pierre and Miquelon [France]
SC Saint Kitts and Nevis
SE Seychelles 
SF South Africa 
SG Senegal 
SH Saint Helena [United Kingdom] 
SI Slovenia 
SL Sierra Leone 
SN Singapore
SO Somalia
SP Spain 
ST Saint Lucia 
SU Sudan 
SV Svalbard [Norway] 
SW Sweden 
SX South Georgia and the South Sandwich Islands [United Kingdom]
SY Syria 
SZ Switzerland 
TD Trinidad and Tobago 
TE Tromelin Island [France]
TH Thailand 
TI Tajikistan 
TL Tokelau [New Zealand] 
TN Tonga 
TO Togo 
TP Sao Tome and Principe
TS Tunisia 
TT Timor-Leste
TU Turkey 
TV Tuvalu 
TW Taiwan
TX Turkmenistan 
TZ Tanzania 
UG Uganda
UK United Kingdom 
UP Ukraine 
US United States 
UV Burkina Faso 
UY Uruguay 
UZ Uzbekistan 
VC Saint Vincent and the Grenadines
VE Venezuela 
VM Vietnam 
VQ Virgin Islands [United States] 
WA Namibia 
WF Wallis and Futuna [France]
WI Western Sahara 
WQ Wake Island [United States]
WS Samoa
WZ Swaziland 
XX Unknown
YM Yemen
ZA Zambia 
ZI Zimbabwe 

Subscribe to feed