## MAX – MIN vs MEAN _ Well it is a sort of answer

I was all set to do a bunch of MIN and MAX reports and compare a graph of each vs the MEAN (with lots of ponderings to ponder if “odd things” happened…) when “odd things” started to happen.

I got zero data reports (lots of them). Even some reports with MIN in some year, but no MAX ((or vs verse visa). So just how do you compute a mean from a Min and Max if you don’t have the Min or Max? My guess is that the data exist, but just not in the Min / Max files for GHCN (i.e. the country BOM will sell it…) or the MEAN was computed long long ago and the MIN / MAX are “long gone”.

OK, some numbers…

### From the Dec 2009 Vintage GHCN

While it may seem odd to have Vintages of what ought to be static “historical data”, in GHCN you must state the vintage to know if it was “a very good year” or not…

Number of Countries

[chiefio@Hummer wrk]\$ wc -l v2.country.codes
233 v2.country.codes

How many do I report on with my program? (That looks for MEAN records in the v2.mean file):

[chiefio@Hummer wrk]\$ grep ” Mean……..” MMM_report | wc -l
233

Oh good. How many of them have a ZERO for MEAN data record count in the v2.mean file?

[chiefio@Hummer wrk]\$ grep ” Mean……. 0″ MMM_report | wc -l
11

OK, eleven countries are just a empty. No data for the mean at all. That leaves 222 countries as “carrying the freight”. So how many of them have a ZERO for the MIN or MAX record counts? That is, how many programs have NO DATA in the GHCN v2.min or v2.max files?

Number with MIN of zero

[chiefio@Hummer wrk]\$ grep ” Min…….. 0″ MMM_report | wc -l
139

Number of MAX of zero

[chiefio@Hummer wrk]\$ !! | wc -l
grep ” Max…….. 0″ MMM_report | wc -l
139

Well, at least they are consistent…

So we have 139 countries missing MIN and MAX (and one presumes 11 of those have no MIN, MAX and MEAN, leaving 128 as “With Mean, no min/max” and presumably 222 – 128 = 94 as having a MIN, MAX and MEAN (though it isn’t that clean).

Going to be a bit hard to compare those MAX – MIN / 2 records with the MEANS for a whole lot of countries… But then I compared the number of MIN / MAX records to the number of MEAN records when they all DO exist. Even for the best reported countries they diverge. A LOT.

So I’m now looking at the Min / Max reporting possibilities and thinking them “bleak”. If the MEAN data was way too sparse globally to give much of a valid temperature trend for the planet, the MIN and MAX are hopeless. I note, for example, the complete absence of South America… They are likely also worthless as a ‘cross check’ on the MEAN trends. Yeah, I’ll plug on through looking for something of interest, but I don’t see where it can come from…

### Report of Thermometer Years for Mean, Min, and Max

With that said, here is the report of “Thermometer Years of Mean, Min, and Max records by country”. For each country, I find all records starting with that country code, and count them. They could be all “missing data flags” or all perfect. I don’t select on the temperature values at all. And a value of 20 could be one thermometer reporting for 20 years (and not necessarily contiguous…) or 20 thermometers in one year (or 10 thermometers with 2 Duplicate Numbers each; in one year). It’s just a raw count of “thermometer records”. And it’s not pretty:

[chiefio@Hummer wrk]\$ cat MMM_report

```For Country Code: 101 ALGERIA
Min:      643
Max:      643
Mean:    3378

For Country Code: 102 ANGOLA
Min:        0
Max:        0
Mean:     933

For Country Code: 103 BENIN
Min:        0
Max:        0
Mean:     539

For Country Code: 104 BOTSWANA
Min:      100
Max:       99
Mean:     302

For Country Code: 105 BURKINA FASO
Min:        0
Max:        0
Mean:     757

For Country Code: 106 BURUNDI
Min:        0
Max:        0
Mean:      20

For Country Code: 107 CAMEROON
Min:        0
Max:        0
Mean:    1102

For Country Code: 108 CAPE VERDE
Min:        0
Max:        0
Mean:     210

For Country Code: 109 CENTRAL AFRICAN REPUBLIC
Min:        0
Max:        0
Mean:    1154

For Country Code: 110 CHAD
Min:        0
Max:        0
Mean:     717

For Country Code: 111 COMOROS
Min:        0
Max:        0
Mean:     309

For Country Code: 112 CONGO
Min:        0
Max:        0
Mean:    1297

For Country Code: 113 COTE D IVOIRE
Min:        0
Max:        0
Mean:    1251

For Country Code: 114 DJIBOUTI
Min:        0
Max:        0
Mean:      74

For Country Code: 115 EGYPT
Min:      463
Max:      469
Mean:    1823

For Country Code: 116 ERITREA
Min:        0
Max:        0
Mean:     130

For Country Code: 117 ETHIOPIA
Min:        0
Max:        0
Mean:     731

For Country Code: 118 GABON
Min:        0
Max:        0
Mean:    1256

For Country Code: 119 GHANA
Min:      103
Max:      104
Mean:     535

For Country Code: 120 GUINEA
Min:        0
Max:        0
Mean:     436

For Country Code: 121 GUINEA-BISSAU
Min:        0
Max:        0
Mean:      66

For Country Code: 122 KENYA
Min:      115
Max:      117
Mean:     634

For Country Code: 123 LIBERIA
Min:        0
Max:        0
Mean:     150

For Country Code: 124 LIBYA
Min:      295
Max:      295
Mean:    1424

For Country Code: 125 MADAGASCAR
Min:        0
Max:        0
Mean:     972

For Country Code: 126 MALAWI
Min:        0
Max:        0
Mean:     303

For Country Code: 127 MALI
Min:        0
Max:        0
Mean:    1876

For Country Code: 128 MAURITANIA
Min:        0
Max:        0
Mean:    1040

For Country Code: 129 MAURITIUS
Min:        0
Max:        0
Mean:     383

For Country Code: 130 MOROCCO
Min:       66
Max:       66
Mean:     679

For Country Code: 131 MOZAMBIQUE
Min:        0
Max:        0
Mean:    1069

For Country Code: 132 NAMIBIA
Min:       54
Max:       54
Mean:     234

For Country Code: 133 NIGER
Min:        0
Max:        0
Mean:    1162

For Country Code: 134 NIGERIA
Min:      302
Max:      313
Mean:    1169

For Country Code: 136 SAO TOME AND PRINCIPE
Min:        0
Max:        0
Mean:     107

For Country Code: 137 SENEGAL
Min:        0
Max:        0
Mean:    1273

For Country Code: 138 SEYCHELLES
Min:        0
Max:        0
Mean:     251

For Country Code: 139 SIERRA LEONE
Min:       91
Max:       91
Mean:     461

For Country Code: 140 SOMALIA
Min:      185
Max:      187
Mean:     350

For Country Code: 141 SOUTH AFRICA
Min:     2431
Max:     2432
Mean:    4678

For Country Code: 148 SUDAN
Min:      795
Max:      794
Mean:    3248

For Country Code: 149 TANZANIA
Min:      198
Max:      197
Mean:     870

For Country Code: 150 THE GAMBIA
Min:        0
Max:        0
Mean:     193

For Country Code: 151 TOGO
Min:        0
Max:        0
Mean:     643

For Country Code: 152 TUNISIA
Min:        0
Max:        0
Mean:     634

For Country Code: 153 UGANDA
Min:      152
Max:      153
Mean:     307

For Country Code: 154 ZAIRE
Min:        0
Max:        0
Mean:     731

For Country Code: 155 ZAMBIA
Min:      268
Max:      277
Mean:     843

For Country Code: 156 ZIMBABWE
Min:      183
Max:      178
Mean:     840

For Country Code: 157 AMSTERDAM ISLAND FRANCE
Min:        0
Max:        0
Mean:     133

For Country Code: 158 ASCENSION ISLAND U.K.
Min:        0
Max:        0
Mean:      81

For Country Code: 159 CANARY ISLANDS SPAIN
Min:        0
Max:        0
Mean:     541

For Country Code: 160 CEUTA SPAIN
Min:        0
Max:        0
Mean:      31

For Country Code: 161 CHAGOS ARCHIPELAGO U.K.
Min:       30
Max:       30
Mean:      71

For Country Code: 162 LESOTHO
Min:        0
Max:        0
Mean:      10

For Country Code: 163 MAYOTTE FRANCE
Min:        0
Max:        0
Mean:      83

For Country Code: 164 MELILLA SPAIN
Min:        0
Max:        0
Mean:       0

For Country Code: 165 REUNION ISLAND FRANCE
Min:        0
Max:        0
Mean:     127

For Country Code: 166 RWANDA
Min:        0
Max:        0
Mean:       0

For Country Code: 167 SWAZILAND
Min:        0
Max:        0
Mean:       0

For Country Code: 168 TROMELIN ISLAND FRANCE
Min:        0
Max:        0
Mean:     103

For Country Code: 169 WESTERN SAHARA MOROCCO
Min:        0
Max:        0
Mean:      85

For Country Code: 201 AFGHANISTAN
Min:        0
Max:        0
Mean:     252

For Country Code: 202 BAHRAIN
Min:        0
Max:        0
Mean:     111

For Country Code: 203 BANGLADESH
Min:      698
Max:      702
Mean:    1875

For Country Code: 204 CAMBODIA
Min:        0
Max:        0
Mean:     114

For Country Code: 205 CHINA
Min:    15608
Max:    15599
Mean:   31325

For Country Code: 206 DEMOCRATIC PEOPLES REPUBLIC OF KOREA
Min:        0
Max:        0
Mean:     726

For Country Code: 207 INDIA
Min:        0
Max:        0
Mean:    7625

For Country Code: 208 IRAN
Min:        0
Max:        0
Mean:    1229

For Country Code: 209 IRAQ
Min:        0
Max:        0
Mean:     964

For Country Code: 210 JAPAN
Min:    11916
Max:    11917
Mean:   16698

For Country Code: 211 KAZAKHSTAN
Min:     1556
Max:     1462
Mean:    6742

For Country Code: 212 KUWAIT
Min:       31
Max:       31
Mean:     193

For Country Code: 213 KYRGYZSTAN
Min:      131
Max:      120
Mean:     505

For Country Code: 214 LAOS
Min:        0
Max:        0
Mean:     382

For Country Code: 215 MONGOLIA
Min:        0
Max:        0
Mean:    2531

For Country Code: 216 MYANMAR
Min:        0
Max:        0
Mean:     790

For Country Code: 217 NEPAL
Min:        0
Max:        0
Mean:      68

For Country Code: 218 OMAN
Min:        0
Max:        0
Mean:     217

For Country Code: 219 PAKISTAN
Min:     2237
Max:     2259
Mean:    5055

For Country Code: 220 QATAR
Min:       32
Max:       32
Mean:      87

For Country Code: 221 REPUBLIC OF KOREA
Min:     2165
Max:     2166
Mean:    4773

For Country Code: 222 RUSSIAN FEDERATION ASIAN SECTOR
Min:     7937
Max:     7157
Mean:   33198

For Country Code: 223 SAUDI ARABIA
Min:       29
Max:       29
Mean:     761

For Country Code: 224 SRI LANKA
Min:        0
Max:        0
Mean:    1592

For Country Code: 227 TAJIKISTAN
Min:      239
Max:      237
Mean:     660

For Country Code: 228 THAILAND
Min:      974
Max:      973
Mean:    2046

For Country Code: 229 TURKMENISTAN
Min:      617
Max:      574
Mean:    2175

For Country Code: 230 UNITED ARAB EMIRATES
Min:        0
Max:        0
Mean:      31

For Country Code: 231 UZBEKISTAN
Min:      466
Max:      447
Mean:    1658

For Country Code: 232 VIETNAM
Min:        0
Max:        0
Mean:     470

For Country Code: 233 YEMEN
Min:        0
Max:        0
Mean:     116

For Country Code: 234 MACAU PORTUGAL
Min:        0
Max:        0
Mean:      94

For Country Code: 235 MALDIVES
Min:        0
Max:        0
Mean:       0

For Country Code: 236 TAIWAN
Min:        0
Max:        0
Mean:      79

For Country Code: 301 ARGENTINA
Min:        0
Max:        0
Mean:    7953

For Country Code: 302 BOLIVIA
Min:        0
Max:        0
Mean:    1431

For Country Code: 303 BRAZIL
Min:        0
Max:        0
Mean:    4434

For Country Code: 304 CHILE
Min:        0
Max:        0
Mean:    2543

For Country Code: 305 COLOMBIA
Min:        0
Max:        0
Mean:    1072

For Country Code: 306 ECUADOR
Min:        0
Max:        0
Mean:     733

For Country Code: 307 GUYANA
Min:        0
Max:        0
Mean:     199

For Country Code: 308 PARAGUAY
Min:        0
Max:        0
Mean:    1466

For Country Code: 309 PERU
Min:        0
Max:        0
Mean:    2863

For Country Code: 312 SURINAME
Min:        0
Max:        0
Mean:     306

For Country Code: 313 URUGUAY
Min:        0
Max:        0
Mean:     959

For Country Code: 314 VENEZUELA
Min:        0
Max:        0
Mean:    1830

For Country Code: 315 FRENCH GUIANA FRANCE
Min:        0
Max:        0
Mean:     245

For Country Code: 316 FALKLAND ISLANDS U.K.
Min:        0
Max:        0
Mean:     167

For Country Code: 317 SOUTH GEORGIA U.K.
Min:        0
Max:        0
Mean:     190

For Country Code: 401 BARBADOS
Min:        0
Max:        0
Mean:     161

For Country Code: 402 BELIZE
Min:        0
Max:        0
Mean:     243

For Country Code: 403 CANADA
Min:    32640
Max:    32628
Mean:   49595

For Country Code: 405 COSTA RICA
Min:        0
Max:        0
Mean:     309

For Country Code: 406 CUBA
Min:       60
Max:       60
Mean:     518

For Country Code: 407 DOMINICAN REPUBLIC
Min:        0
Max:        0
Mean:     712

For Country Code: 408 EL SALVADOR
Min:        0
Max:        0
Mean:     131

For Country Code: 409 GRENADA
Min:        0
Max:        0
Mean:      63

For Country Code: 410 GUATEMALA
Min:        0
Max:        0
Mean:     174

For Country Code: 411 HAITI
Min:        0
Max:        0
Mean:      88

For Country Code: 412 HONDURAS
Min:       23
Max:       23
Mean:     964

For Country Code: 413 JAMAICA
Min:        0
Max:        0
Mean:     271

For Country Code: 414 MEXICO
Min:        0
Max:        0
Mean:    7533

For Country Code: 415 NICARAGUA
Min:        0
Max:        0
Mean:     170

For Country Code: 416 PANAMA
Min:       26
Max:       26
Mean:     197

For Country Code: 417 SAINT KITTS AND NEVIS
Min:        0
Max:        0
Mean:      10

For Country Code: 423 THE BAHAMAS
Min:        0
Max:        0
Mean:     229

For Country Code: 424 TRINIDAD AND TOBAGO
Min:        0
Max:        0
Mean:     221

For Country Code: 425 UNITED STATES OF AMERICA
Min:   155224
Max:   155251
Mean:  194577

For Country Code: 426 ANTIGUA AND BARBUDA
Min:        0
Max:        0
Mean:       0

For Country Code: 427 BERMUDA U.K.
Min:       45
Max:       45
Mean:     236

For Country Code: 428 BRITISH VIRGIN ISLANDS U.K.
Min:        0
Max:        0
Mean:       0

For Country Code: 429 CAYMAN ISLANDS U.K.
Min:       24
Max:       24
Mean:      24

For Country Code: 430 DOMINICA
Min:        0
Max:        0
Mean:       0

For Country Code: 431 GREENLAND DENMARK
Min:       44
Max:       45
Mean:    1294

For Country Code: 432 GUADELOUPE FRANCE
Min:        0
Max:        0
Mean:     121

For Country Code: 433 MARTINIQUE FRANCE
Min:        0
Max:        0
Mean:     119

For Country Code: 434 NETHERLANDS ANTILLES NETHERLANDS
Min:        0
Max:        0
Mean:     181

For Country Code: 435 PUERTO RICO U.S.A.
Min:      127
Max:      127
Mean:     803

For Country Code: 436 SAINT LUCIA
Min:        0
Max:        0
Mean:       0

For Country Code: 437 SAINT VINCENT AND THE GRENADINES
Min:        0
Max:        0
Mean:       0

For Country Code: 438 SAINT PIERRE and MIQUELON ISLAND
Min:        0
Max:        0
Mean:      58

For Country Code: 439 TURKS AND CAICOS ISLANDS
Min:        0
Max:        0
Mean:       0

For Country Code: 440 VIRGIN ISLANDS U.S.A.
Min:       27
Max:       27
Mean:     277

For Country Code: 501 AUSTRALIA
Min:    29024
Max:    29089
Mean:   35989

For Country Code: 502 FIJI
Min:        0
Max:        0
Mean:     457

For Country Code: 503 INDONESIA
Min:      829
Max:      831
Mean:    3275

For Country Code: 504 KIRIBATI
Min:       19
Max:       19
Mean:     220

For Country Code: 505 MALAYSIA
Min:      793
Max:      758
Mean:    1346

For Country Code: 506 NAURU
Min:       27
Max:       27
Mean:      93

For Country Code: 507 NEW ZEALAND
Min:        0
Max:        0
Mean:    1747

For Country Code: 508 PAPUA NEW GUINEA
Min:      188
Max:      184
Mean:     710

For Country Code: 509 PHILIPPINES
Min:     1364
Max:     1378
Mean:    2614

For Country Code: 511 SINGAPORE
Min:        0
Max:        0
Mean:     119

For Country Code: 512 SOLOMON ISLANDS
Min:      114
Max:      114
Mean:     211

For Country Code: 517 TONGA
Min:        0
Max:        0
Mean:      61

For Country Code: 518 TUVALU
Min:        0
Max:        0
Mean:     159

For Country Code: 520 VANUATU
Min:       57
Max:       57
Mean:     329

For Country Code: 521 AMERICAN SAMOA U.S.A.
Min:       41
Max:       41
Mean:     146

For Country Code: 522 BRUNEI
Min:        0
Max:        0
Mean:       0

For Country Code: 523 CHRISTMAS ISLAND AUSTRALIA
Min:       23
Max:       23
Mean:      51

For Country Code: 524 COCOS ISLANDS AUSTRALIA
Min:       55
Max:       55
Mean:     167

For Country Code: 525 COOK ISLANDS NEW ZEALAND
Min:        0
Max:        0
Mean:     413

For Country Code: 526 CORAL SEA ISLANDS AUSTRALIA
Min:       67
Max:       67
Mean:     174

For Country Code: 527 FEDERATED STATES OF MICRONESIA
Min:      177
Max:      177
Mean:     556

For Country Code: 528 FRENCH POLYNESIA FRANCE
Min:        0
Max:        0
Mean:     988

For Country Code: 529 GUAM U.S.A.
Min:       48
Max:       48
Mean:     105

For Country Code: 530 JOHNSTON ATOLL U.S.A.
Min:       27
Max:       27
Mean:      76

For Country Code: 531 MARSHALL ISLANDS
Min:      133
Max:      133
Mean:     302

For Country Code: 532 NEW CALEDONIA FRANCE
Min:        0
Max:        0
Mean:     280

For Country Code: 533 NIUE NEW ZEALAND
Min:        0
Max:        0
Mean:      57

For Country Code: 534 NORFOLK ISLAND AUSTRALIA
Min:       69
Max:       69
Mean:     212

For Country Code: 535 NORTHERN MARIANA ISLANDS U.S.A.
Min:       75
Max:       75
Mean:     165

For Country Code: 536 BELAU
Min:       58
Max:       58
Mean:     195

For Country Code: 537 PITCAIRN ISLAND U.K.
Min:        0
Max:        0
Mean:      39

For Country Code: 538 TOKELAU
Min:        0
Max:        0
Mean:      50

For Country Code: 539 WAKE ISLAND U.S.A.
Min:       58
Max:       58
Mean:     141

For Country Code: 540 WALLIS AND FUTUNA FRANCE
Min:        0
Max:        0
Mean:     116

For Country Code: 541 SAMOA
Min:        0
Max:        0
Mean:     160

For Country Code: 601 ALBANIA
Min:        0
Max:        0
Mean:     103

For Country Code: 602 ARMENIA
Min:      185
Max:      146
Mean:     448

For Country Code: 603 AUSTRIA
Min:      169
Max:      169
Mean:    1866

For Country Code: 604 AZERBAIJAN
Min:       83
Max:       80
Mean:     429

For Country Code: 605 BELARUS
Min:      229
Max:      194
Mean:     835

For Country Code: 606 BELGIUM
Min:        0
Max:        0
Mean:     254

For Country Code: 607 BOSNIA AND HERZEGOVINA
Min:      163
Max:      162
Mean:     288

For Country Code: 608 BULGARIA
Min:        0
Max:        0
Mean:     589

For Country Code: 609 CROATIA
Min:      197
Max:      195
Mean:     759

For Country Code: 610 CYPRUS
Min:        0
Max:        0
Mean:     203

For Country Code: 611 CZECH REPUBLIC
Min:       85
Max:       85
Mean:     988

For Country Code: 612 DENMARK
Min:        0
Max:        0
Mean:     984

For Country Code: 613 ESTONIA
Min:      122
Max:      122
Mean:     591

For Country Code: 614 FINLAND
Min:        0
Max:        0
Mean:    1394

For Country Code: 615 FRANCE
Min:       73
Max:       73
Mean:    3667

For Country Code: 616 GEORGIA
Min:      159
Max:      144
Mean:     564

For Country Code: 617 GERMANY
Min:      113
Max:      113
Mean:    5443

For Country Code: 618 GREECE
Min:       33
Max:       33
Mean:    1349

For Country Code: 619 HUNGARY
Min:        0
Max:        0
Mean:     840

For Country Code: 620 ICELAND
Min:       56
Max:       56
Mean:    1078

For Country Code: 621 IRELAND
Min:      436
Max:      436
Mean:    1979

For Country Code: 622 ISRAEL
Min:        0
Max:        0
Mean:     627

For Country Code: 623 ITALY
Min:     2264
Max:     2265
Mean:    4961

For Country Code: 624 JORDAN
Min:        0
Max:        0
Mean:     260

For Country Code: 625 KAZAKHSTAN
Min:        0
Max:        0
Mean:      75

For Country Code: 626 LATVIA
Min:      141
Max:      120
Mean:     565

For Country Code: 627 LEBANON
Min:        0
Max:        0
Mean:     432

For Country Code: 628 LITHUANIA
Min:      195
Max:      171
Mean:     817

For Country Code: 629 LUXEMBOURG
Min:        0
Max:        0
Mean:     144

For Country Code: 630 MALTA
Min:        0
Max:        0
Mean:     218

For Country Code: 631 MOLDOVA
Min:       74
Max:       61
Mean:     373

For Country Code: 632 MONTENEGRO
Min:       86
Max:       86
Mean:     172

For Country Code: 633 NETHERLANDS
Min:        0
Max:        0
Mean:     406

For Country Code: 634 NORWAY
Min:        0
Max:        0
Mean:    3410

For Country Code: 635 POLAND
Min:     2055
Max:     2055
Mean:    4194

For Country Code: 636 PORTUGAL
Min:       21
Max:       21
Mean:    1386

For Country Code: 637 ROMANIA
Min:        0
Max:        0
Mean:    1526

For Country Code: 638 RUSSIAN FEDERATION EUROPEAN SECTOR
Min:     3176
Max:     2992
Mean:   13497

For Country Code: 639 SERBIA
Min:      354
Max:      352
Mean:     452

For Country Code: 641 SLOVAKIA
Min:       39
Max:       39
Mean:     392

For Country Code: 643 SPAIN
Min:       75
Max:       75
Mean:    2603

For Country Code: 645 SWEDEN
Min:        0
Max:        0
Mean:    1714

For Country Code: 646 SWITZERLAND
Min:        0
Max:        0
Mean:    1410

For Country Code: 647 SYRIA
Min:        0
Max:        0
Mean:     657

For Country Code: 648 MACEDONIA
Min:      100
Max:      106
Mean:     228

For Country Code: 649 TURKEY
Min:     7155
Max:     7155
Mean:    9301

For Country Code: 650 UKRAINE
Min:     1031
Max:     1012
Mean:    4864

For Country Code: 651 UNITED KINGDOM
Min:      172
Max:      172
Mean:    4471

For Country Code: 652 FAROE ISLANDS DENMARK
Min:        0
Max:        0
Mean:     205

For Country Code: 653 GIBRALTAR U.K.
Min:        0
Max:        0
Mean:     161

For Country Code: 654 MADEIRA ISLANDS PORTUGAL
Min:        0
Max:        0
Mean:     249

For Country Code: 700 ANTARCTICA
Min:      271
Max:      293
Mean:    2741

For Country Code: 800 SHIP STATIONS
Min:        0
Max:        0
Mean:     437
```

Well, at least it’s an answer of a sort. We know that swapping over to using MIN or MAX to compute the “Global Warming” trend is not going to cut it; and we know that even using them for “Spot Checks” is going to be largely useless.

Postscript: Some AGW True Believers have ‘had issues’ with my talking about my “average of temperature” reports as a way of “measuring the data”. That is, they don’t tell me the actual “Average Temperature” (by which I presume they really meant ‘Mean Temperature’) but rather they tell me about the structure of the temperature records.

Well, here is a very simple and very explicit example of “measuring the data”, though in this case it’s looking at “do they exist at all” vs “how big are they”… but this “characterize the data” is a needed first step. It tells you “what is possible” and it tells you “what is a fools errand”. And computing a Global Average Temperature MIN or a Global Average Temperature MAX are even worse fools errands than computing a Global Average Temperature MEAN.

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in AGW Science and Background, NCDC - GHCN Issues and tagged , . Bookmark the permalink.

### 15 Responses to MAX – MIN vs MEAN _ Well it is a sort of answer

1. Steven Schuman says:

Chiefio,
As I look at yours and other data, I tell people I cannot disprove AGW, but I don’t think anyone can have any reasonable certainty that it does exist. I know this is a rather broad question, but after all the time you’ve spent looking at the data where do you stand?

REPLY: [ Well, sit, actually, fresh coffee to hand, watching CNBC as I need to get back “in sync” with markets and make some money to feed my “hobby”… OK, to know where I am now, it’s best to know where I came from (then “now” becomes more useful).

I started on this journey about 4 years ago (? a guess) with a “Gee, Global Warming looks like a big deal and must be an interesting thing to look at. Both from an investing potential impacts and from just a natural sciences point of view.” The more I looked (and the more books I read about it) the more I noticed a pattern: Most of the published “stuff” was of the form “ASSUMING Global Warming, what bad thing will happen?”

Well, I know that pattern very well. It is the standard formula for Science Fiction. Assume space aliens want to invade earth, what bad thing will happen? Assume an astroid will strike Earth (send Bruce Willis!) what bad thing will happen (other than TWO movies with the same theme coming out together…) Assume then Panic. And as much as I love SciFi, I hate assuming… So I bought more books and hit more web sites.

Then I made the mistake of asking “An Inconvenient Question” or two at some Warmer sites. RealClimate, for example. I just wanted to know the truth, so I had “things that don’t fit” (and nothing bothers A Tidy Mind, especially a borderline Aspe one, quite so much as things that will not ‘fit’… heck, I even wash the spoons in one batch and the forks in another batch so they don’t have to be mixed together… well, and I figured out that they must eventually be sorted anyway and “sort first’ is slightly more efficient… but I digress 8-) So I asked questions like: “It was cool in the ’60s and ’70s and we had the Ice Age scare – it snowed in my home town, where it almost never snows: that means the “baseline” is set in a broken place, doesn’t it?” or “How come the past keeps getting re-written colder?” ( I really don’t like folks moving my past around…) and just got attacked.

The more I looked, the more I found “broken stuff” masquerading as Science. And a big chunk of it was hanging on this “GIStemp” thing. So I bitched about the code and the data being “unavailable” ( a bit of a phishing expedition, truth be known… I was feeling a bit lazy and didn’t want to spend the time needed to find out on my own, so I just tossed a “bitch” out on WUWT and a “kindly Warmer” responded with some vehemence that I was an idiot and it was all published at (links to places). That made my life easier as they gave me what I wanted.) Then, after about 5 months of complaining that “Somebody” needed to make this thing go and test it (I’d looked inside and recoiled in horror…) I decided that if nobody else was going to do it, well, “I was somebody”.

By this time I was pretty firmly in the Skeptic camp. Having moved from “lukewarmer” with a bit of doubt that maybe they were just confounding a LIA rebound and ’70s cold excursion in the baseline with a human caused event. At WUWT I learned a great deal about climatology and weather and other links to interesting places (like solar observatories and NCDC and other countries Bureaus of Meteorology). And I noticed another pattern. When you go turning over rocks, if things are “normal” you get about as many “bad” surprises as “good” surprises. In an unbiased search you would expect to find a bag of quarters missing from the vault, but two bags of dimes in the vault without a record and a bag of nickels that is twice the size it’s supposed to be. Basically, random error tends to cancel. But almost every time a rock got turned over, there was another roach running for cover on the Warmer side… The classic indicia of a hidden “awshit” somewhere. So over time (and with a library now of about 3 feet length on the bookcase) I’ve become a modestly “Confirmed Skeptic”.

OK, that brings us up to about 1 year (?guess) ago. Having gotten GIStemp to run and read the code “end to end” a couple of times. It’s pretty rough. It has some “issues” that impact the actual function, but they tend to be fairly small. Most of the “issues” are “quality control” issues. Basically, I think it “works as advertized, mostly” but still has some “issues” that need investigation that could have a major impact. (In particular, investigating “anomaly processing” has left me with 2 ways in which it fails that are not addressed in GIStemp as near as I can tell – though I exploit one of them in dT/dt to make it better able to “spot” discontinuities in the data… that “Bullseye” that happens at transistions… where all the monthly anomalies pass through zero…) I’ve run a couple of benchmarks that show a 1/2 C or so “lift” to the data as it is processed prior to the “grid / box” anomaly step, and I’ve benchmarked STEP3 and found that about that much “warming” leaks through (that is, the “Box of Thermometers A compared to Box of Thermometers B” method of doing grid/box anomalies fails to do a clean anomaly and ‘leaks’ about 1/2 C of bias through). So I was all wound up to pronounce GIStemp as “Bogus in the 1/2 C degree” when 2 things happened. One was that I noticed “strange things” in the input data itself, the other was that Climategate broke. I spend ‘a while’ examining the data – all those “by altitude” and “by latitude” and “by airport flag” etc. reports.

That pointed me firmly at the data as “cooked”. And that brings us up to “now”.

Having gone through this dT/dt driven anomaly graphing exercise it’s pretty clear to me what’s going on. Frankly, the Fiji graph as compared to Micronesia and Samoa (with Tonga as seasoning…) pushed me “over the line”. It just screams fudge. If I were doing an undercover investigation (as I’ve done at some sites) I’d be sitting in the office of the V.P. (or whoever knew my actual job as investigator) discussing how to proceed: “Walk them out, bring in law enforcement, or bait and trap for a bigger catch?” (Most companies would just ‘walk them out’. Japanese companies especially want no bad news ever. Had a batch of guys stealing equipment from the dock and doctoring the books; just walked out. Nutty, IMHO, but the customer wanted discretion.)

So we take a look at a couple of places in the middle of the Pacific with ONE thermometer and they don’t screw around with it:

Dead Flat. Now you don’t get dead flat in a world of warming.

Then we look over at Singapore:

And get a great little “Hockey Stick” right at the thermometer rotation ( that blip up in the yellow line followed by a blip back down as the swap completes). That is just not consistent with steadily rising CO2 levels as causal NOR with UHI from industrial growth. It IS consistent with fudging the books.

Then we take a look at French Polynesia. Generally the French have been good indicators of what’s really happening, but here we have a ‘drop;’ in the baseline:

Notice how the baseline “sags” as a mirror image of the thermometer count line rising? In theory, thermometer count ought to be unrelated (if the process were clean the actual count of thermometers ought to have little relationship, sometimes up, often neutral, sometimes down – what we find is a bit different):

And you see that pattern in other places as well. Counts change at the start of the baseline, and the baseline drops, then they change at the end and the baseline rises, then in 1990 ‘something changes’ that makes a dramatic hockey stick blade in some place and a pivot / ramp in almost all of them. It is coincident with thermometer dropping on a massive basis, but could be just a change of processing that happened at the same time (i.e. “duplicate number” or “mod flag” changes on the survivors, but we don’t know if the choice to survive is the same as the change of flag… nor if the dropped records could be subjected to the same treatment to get the same ‘lift”…)

Then finally, there is Fiji. So much change in that ‘thermometer line’ right when the temperature line makes a great “dip” baseline and a “lift” coming out of it, then the “pivot” right at the “bullseye” about 1992 start of the Rocket Ride hot pink segment. All worked into a generally ‘steady rising’ trend of +2 C over all? Now look at Cooks and Cocos again… SOMEBODY is wrong in this picture… and its the guys screwing around with the thermometers…

https://chiefio.files.wordpress.com/2010/03/fiji_full_hair.png?w=500&h=277

The only thing I can’t speak to is “motivation”. Folks have a remarkable capacity for self deception. (Google “Clever Hans” – everyone thought the horse understood language… when he just had a great ‘make boss happy’ detector..) So was this a ‘deliberate act of data vandalism’ or was it ‘expectation driven accidental consequence’? That’s not my job. That’s for the jury to decide.

It’s quite possible, for example, that someone decided to put in place an “outlier QA process” that has asymmetrical impact, tossing out more low lows than high highs. ( In fact, I’m almost certain that’s the case given the suppression of low going excursions). They may have done it knowing the effect as a deliberate act, or they may have simply “picked some parameters out of thin air that ‘seemed good’ and were not”. So, for example, downside excursions are further (cold air does not convectively mix by rising, it just lays there) than hot excursions (hot air on the surface rises, mixing). If you ‘clip equally’ you have unequal impact when compared to a past data base that is “unclipped” ( which is, btw, what I think is happening at the 1990 “bullseye” pivot…)

So if this were a contract (as I’ve done before for security departments – that whole FBI background check fingerprints on file thing…) I’d be sitting with the Chief Mumble and explaining that we had enough goods to walk them out on the spot. I would be recommending involvement of law enforcement at the ‘getting to know the story’ level but not yet at the ‘raid / bust em’ level. And I’d be asking if they wanted to ‘bait the trap’ to get motivational information. I’d also ask that a ‘Disaster Recovery Set’ of all data be dumped so as to prevent any ability to ‘erase the evidence’. (But in fact, I prefer to do that as soon as I show up on a site. One of my first recommendations is a “We need a disaster recovery set and don’t have one so we’re just going to dump everything to tape while we design the process” fire drills. Most folks don’t “spook” over that and it gives you an archive you can place out of reach in a very private vault… but you do need someone watching the creation to detect when someone “decides to skip that irrelevant box over there” ;-)

So, OK, another 100x response to a 1 x question ;-) The “short form” would be:

“Book ’em Danno!”

Then I’d start over from ONLY the original records of the real raw data and I’d specifically forbid mixing different “processes” over time and I’d stabilize what thermometers were used as much as absolutely possible. And I’d have the work done by someone completely unrelated to the prior crop of “climate scientists” (preferably 2 isolated teams). Then I’d think we might have some clue what’s really going on… And I’d expect to find a LIA, a cold ’70s, and not much different between now and before the LIA and the 1930’s in North America.
-E.M.Smith ]

2. j ferguson says:

When you are doing a divination using sheep entrails as they did in Rome, you start with a live sheep – assumption is that it’s more likely to have a complete set to work with.

It really boggles to see that they didn’t even go looking for a live sheep. Maybe they did look, and there weren’t any.

I had sort of been hoping to see how the min temp trends compared to the max and whether min was rising and max wasn’t.

certainly there the inertia of mass/temperature has to vary by location – think Miami, where, in the summer it gets up to 93-95F everyday but very seldom higher and then cools into the low 80s (IIRC) at night.

one might imagine that different feedback mechanisms impinge on minimum than maximum. There seems to be a lot to think about.

One other thing. The notion that “they don’t do it this way” seems bogus. As you evolve the ways you look at the data, it’s true that you may see some logic in the way “they’ve” looked at it. At the same time, maybe not.

REPLY: [ Well, there are still some countries with MIN / MAX data and it might be possible to find ranges over time that match the MEAN during “interesting times”. But the quantity of work to make things “match” is going to be an issue. The big PITA to me is that we can’t do a “cross-foot validation”. One would expect a plot of MIN MAX and MEAN to have them all moving together and with MEAN more or less in the middle. An excursion of MAX down during the baseline and MIN up after the 1990 pivot would be a big Smoking Gun (and was what I saw hints of in a couple of countries). But all the “most interesting countries” were coming up “Zero Data”… which is why I did this report. Now I’m just trying to figure out if I’ve got enough ‘sheep’ left to even make a decent Haggis out of it ;-) Maybe, well see… -E. M. Smith ]

3. KevinM says:

Linkage between the use of “mean = max – min” and modern suppression of downward spikes in the raw data?

REPLY: [ I don’t know. I suspect it’s an “improved QA process” that it tossing “excursions” and having a differential impact, but that is just a working hypothesis at this point. It’s what I was hoping to figure out by looking at changes to MIN and MAX over time… So now I need a new approach compared to what was planned. -E.M.Smith ]

4. E.M.Smith says:

Oh, one point that does matter: The docs per min and max mention that the “Duplicate Number flag” represents that duplicate number used on the MEAN as “there is only one way to calculate a MIN” (or MAX) average. That being a simple average.

So for some of those countries with a large MEAN count, but a MIN or MAX of about 1/2 that size, it could simply be that the MEAN has many “Duplicate Numbers” so more duplicated data that do not make sense in a MIN or MAX file as they would be identical for the different “Duplicate Number” series.

So there are potentials for valid MIN/MAX vs MEAN reports from those places that have a MIN / MAX count smaller than the MEAN count. But it will just be a PITA to match the data up and verify what’s being compared to whom…

5. Chuckles says:

Occams Chainsaw – the MEAN only, no MIN/MAX records are places that are not using electronic MMTS type instruments?

Maybe just casting the Mk1. eyeball at the mercury at the designated time each day.

6. POUNCER says:

Dec 2009 Vintage GHCN?

Is there any chance a fan has laid down an earlier vintage?

Assume, just for fun, a copy of GHCN data from 2007. How troublesome would it be to run a diff or file compare?

7. E.M.Smith says:

@Pouncer: I have a pointer at an earlier Vintage. It’s not that hard ( download, run about 10 minutes worth of scripted programs, do graph). Maybe an hour all told? But I’m trying to complete one bright idea before I start another… It’s a “Polish Point” for me… (as HR would say when telling you how you tended to screw up… ;-)

@Chuckles: I think New Zealand would answer your question. It’s got no MIN/MAX but I’m pretty sure it’s gone electronic at the airports that make up almost all it’s data. Someone in Kiwi Land could probably confirm pretty quickly.

8. Chuckles says:

Good idea Chiefio, I noticed glancing down the list that a lot of the places that zero mins and maxes were, shall we say, places where hollywood types went to publicise and exhibit their compassion, and this triggered the thought.

9. Larry Geiger says:

Ed taught me to look at the data. All of it. Every table. Every field. Find out what the field names mean. Figure out the relationships.

So many people just start typeing SELECT and assuming what field names mean and spitting out data from queries. And they’re wrong and they’re meaningless and they’re convoluted.

Then I started reading your stuff. Look at the data. Look at the basic analysis first before you put more layers of abstraction and complication on top. It just makes sense.

It seems to me that someone came up with a “model” that sounded good to a couple of folks and they grabbed some undergraduate programmer to try and encapsulate their logic in a program and then started feeding in data. And they’ve done this every day, all over the world for the past 20 years.

PCs are great for word processing but maybe we should take them away from certain “scientists” and create a stat/code/data department that they MUST go through to get their data manipulations and models created and run and to store their data.

I guess I’m a little frustrated. I’m sitting here right now looking at a pile of data. The task order number is named differently in every table that it appears in. TO_NO, TONum, TaskOrderNo, TaskOrderNum, etc. It’s a different data type in the tables. It’s an integer in some, 1, 2, 5, 6, etc. It’s a varchar(3) in some “1”, “2”, “5”, “6” in others (we’re up to 700 and some task orders now so what happens when we get to “999”?). It’s a varchar(5) in some tables. The same number with leading 0s, “00001”, “00002”, “00005”, “00006” in other tables. They join these tables with all sorts of substr’s and string functions. Aaaaaaaaaaaargh!!

Anyway, I really appreciated your answer to Steven and your other posts. Thanks.

REPLY: [ Been there, done that, have the scars… “I feel your pain” doesn’t quite cover it…

Same thing runs throughout the code of GIStemp. “Station ID” can be, variously, 12 digits (including “Duplicate Number” or “Modification History Flag”), 11 digits (without it), 8 or 9 digits (the same two, but leaving off the leading 3 digits of “country code” that usually are not needed to make a unique number…), various lengths shorter than 8 or 9 when these were printed out such that leading “0” was suppressed (i.e. they printed an “Integer” without thinking that it’s not an integer it’s an identifier string of digits…), 5 digit “major station number” or 5+3 (that is, the 8 but as distinct parts) to get the minor modifier with the major number (and with leading zero variations possible)… And that is just ONE of the variables that are treated in this messy and cavalier way. The explosion of complexity from rampant carelessness is just stupidity on steroids, IMHO…

I’d like to see a test where, before they were allowed to write programs, folks were placed in a room and told someone would be with them in an hour or two and that there were magazines on the table near the cake and coffee. They would be in the room with some clean dishes in semi-random piles and some dirty dishes in the sink. Those who organized the dishes (without being told they were expected to do anything…) into Very tidy orderly stacks, and preferably washed and put away the dirty ones too, would be advanced immediately… Folks who organized the whole set of cupboards by form, structure, or use would be sent down the supervisor track… ( i.e. Plates in size order, or “glass with glass, ceramic with ceramic”, or cooking pots here, service pieces there, stemware over here…) Those who had cake and coffee and added dirty dishes to the “clean side’ of the sink would be walked out immediately…

Basically a “Chaos vs Order” filter. Does this person fight entropy, or create it? Those who create entropy do not belong in code. Ever. You are better off walking them out and hiring someone else (or doing nothing). It will save you vast amounts of time, money, and error in the long run. It can suck down 2 good programmers trying to control the damage done by one chaotic person.

Per my postings and answers: You are most welcome. Glad just to know someone gains something from it…

-E.M.Smith ]

10. Cement a friend says:

Chiefio, Your reply to the first post is interesting. Thanks for your thoughts and calculations. I sometimes wish I had your computer skills. I managed some basic with my Apple II (1975) but got lost with Fortran and Pascal. I use Metastock to look at some price trends but don’t have the discipline to make much for my super fund.
Your post shows considerable max/min data for Australia. You can download raw data from here http://www.bom.gov.au/climate/data/weather-data.shtml. You should be able to find all the GISS identified stations plus others nearby. Also, you should find stations that may have been dropped.
For interest I have looked at a small rural town Gayndah in Queensland. The number for the Post Office is 039039. The data for temperature dates from 1894 (rainfall from 1870). There appears to be a jump in minimum temperatures about 1975. The station closed in 2009 but there was six years overlap with the Gayndah Airport (2003 and still operating) number 039066. In my assessment this shows + 0.2C UHI for the Gayndah PO. I have not calculated a regression because my version of Excel (home2003)does not do it-I will need to fiddle with the formula. However, I can say that the highest average temperature (also the highest Max. temp) occurred in 1897. The 1890’s was a period of high rainfall while the period following 1900 to 1907 was a drought period with a record low rainfall in 1902. If one takes away the 0.2 UHI effect and then the jump in 1975 then there has been no temperature increase. I can send my spreadsheet with data if you email me.
Warwick Hughes (mentioned in climategate emails) has lots on Australian temperatures eg http://www.warwickhughes.com/blog/?cat=3

REPLY: [ Glad you found something of merit in it ;-) Per computer skills: It mostly takes a willingness to deal with details, and a long time “just doing it”. My first programs were fairly ugly and crude in retrospect… I find computer languages a lot easier than natural languages (more orderly and predictable…) for example. FWIW, I’ve been using OpenOffice and it’s not bad. A bit too “modal” but once you get past the idea that you need to “right click while hovering over the particular item” to get the menu you want, it starts to come together 8-} That it’s a free download is nice too…

Per Aussie data: Yeah, I’m going to be working on that for about a year, I’d guess… I’ve got to finish the current crop of dT/dt based graphs before I start a new thing, though. I’d just figured out that I’ve done roughly 100 graphs in the last couple of days… S. America – 15, Pacific Basin – 33, Asia = 33, N. America about a dozen out of 28. And some of them 2 or 3 times to find the interesting bits to graph. To say I’m a bit “worn” by it is an understatement 8-| so I’m going to knock out the last dozen or so N. America graphs and take a break from “cranking” on ’em. Europe and Africa will have to wait a day or so… Lest I “burn out” on the graphing thing… (Besides, a “look ahead” at the tabular reports shows a lot of the same – flats with hockey sticks with the occasional “little dipper” in key places. So it’s not like some great mystery will be waiting to be uncovered. The present batch pretty much does that.

So don’t be surprised if it takes me a bit to get around to doing an “in depth” on Australia… though I really want to… loved my time “down under” and have some family there.

Though I’ll probably work in a couple of small “for fun” postings along the way ;-)
-E.M.Smith ]

11. Larry Geiger says:

“The explosion of complexity from rampant carelessness is just stupidity on steroids.”

I’m saving this in a special place. If I find the time, I would like to turn this into one of those “inspirational” posters. I just need to find the ideal graphic. Hmmmmmm. Maybe a hockey stick:-) Thanks

REPLY: [ Be my guest. Though a small “-E.M.Smith” attribution would be a nice touch ;-) -E.M.Smith ]

12. A C Osborn says:

Re: POUNCER

Dec 2009 Vintage GHCN?

Is there any chance a fan has laid down an earlier vintage?

Assume, just for fun, a copy of GHCN data from 2007. How troublesome would it be to run a diff or file compare?

That was the main reason that I pointed Chiefio at the “old” data, to see if the “Raw” data had actually been changed.

It is interesting to note how different it is.

REPLY: [ And I’m going to get to it Real Soon Now ;-) but I’m trying to be disciplined about things and actually complete the canonical set of graphs before I take on “new” projects. Though, frankly, I’m getting REALLY tired of doing the same repetitive graph making steps and I’m probably going to do something else for a day or two right after North America and just before Europe as a ‘restorative’… -E.M.Smith ]

13. suricat says:

E M Smith.

I admire your dogged approach to posting. I can only imagine the anxiety this may generate for the desire to alter your direction and post other subject matter. This field moves so quickly at times.:)

The awareness of particular (unpublished) papers has led me to a more engineering orientated conceptualisation (hardly surprising, as I’m an engineer) generated from the introduction of econometrics to climate science. This econometric view was shown to me by Doug L. Hoffman here:

http://theresilientearth.com/?q=content/econometrics-vs-climate-science

In case his mods don’t permit my post response I repeat it:

I think you’re right here Doug. As an engineer, I can never understand why climate science ‘throws away’ most of the temperature signal and also works with an average global temperature that doesn’t have anything like enough resolution within the station network nodes to give a realistic average temperature to an area. Insufficient baud rate and scattered pixel appearance, respectively, just about says it all.

I was always of the understanding that AGW theory encompasses events where diurnal solar forcing would be reflected as an unaltered warming rate from insolation, but with OLR restricted, the night time cooling rate (given cloud change parameters) would reduce. We’d need 5-10 minute daily temperature readings with a realistic clear sky comparison standard to see this phenomenon and I’ve not seen any data of the like, or any movement towards data collation in this manner. Instead, all I’ve seen is a continual march of average temperatures, carbon dioxide content, ocean heat content, etc.

Back to predictive econometrics. What cointegration does show, in engineering terms, is the relationship of the data to a system attractor. However, from there, the system attractor really needs to be identified. Once the attractor is identified there will be other data relationships that are affected by that attractor. The climate system has many attractors and the greater the understanding of climate subsystem attractors that are known, the greater our understanding of climate per se.

I know what you mean by language terminology. I tried to learn ‘climate speak’ because people didn’t know what I was talking about. Though, I don’t see a paradigm shift taking place without a swift kick up the !”£\$%^&*. :)

Best regards, suricat.”

Yes, I know! This doesn’t help you to decide what to do with your spare time!

Best regards, suricat.

14. Jantar says:

The min and max data for New Zealand sites are available at http://cliflo.niwa.co.nz/