I was all set to do a bunch of MIN and MAX reports and compare a graph of each vs the MEAN (with lots of ponderings to ponder if “odd things” happened…) when “odd things” started to happen.
I got zero data reports (lots of them). Even some reports with MIN in some year, but no MAX ((or vs verse visa). So just how do you compute a mean from a Min and Max if you don’t have the Min or Max? My guess is that the data exist, but just not in the Min / Max files for GHCN (i.e. the country BOM will sell it…) or the MEAN was computed long long ago and the MIN / MAX are “long gone”.
OK, some numbers…
From the Dec 2009 Vintage GHCN
While it may seem odd to have Vintages of what ought to be static “historical data”, in GHCN you must state the vintage to know if it was “a very good year” or not…
Number of Countries
[chiefio@Hummer wrk]$ wc -l v2.country.codes
How many do I report on with my program? (That looks for MEAN records in the v2.mean file):
[chiefio@Hummer wrk]$ grep ” Mean……..” MMM_report | wc -l
Oh good. How many of them have a ZERO for MEAN data record count in the v2.mean file?
[chiefio@Hummer wrk]$ grep ” Mean……. 0″ MMM_report | wc -l
OK, eleven countries are just a empty. No data for the mean at all. That leaves 222 countries as “carrying the freight”. So how many of them have a ZERO for the MIN or MAX record counts? That is, how many programs have NO DATA in the GHCN v2.min or v2.max files?
Number with MIN of zero
[chiefio@Hummer wrk]$ grep ” Min…….. 0″ MMM_report | wc -l
Number of MAX of zero
[chiefio@Hummer wrk]$ !! | wc -l
grep ” Max…….. 0″ MMM_report | wc -l
Well, at least they are consistent…
So we have 139 countries missing MIN and MAX (and one presumes 11 of those have no MIN, MAX and MEAN, leaving 128 as “With Mean, no min/max” and presumably 222 – 128 = 94 as having a MIN, MAX and MEAN (though it isn’t that clean).
Going to be a bit hard to compare those MAX – MIN / 2 records with the MEANS for a whole lot of countries… But then I compared the number of MIN / MAX records to the number of MEAN records when they all DO exist. Even for the best reported countries they diverge. A LOT.
So I’m now looking at the Min / Max reporting possibilities and thinking them “bleak”. If the MEAN data was way too sparse globally to give much of a valid temperature trend for the planet, the MIN and MAX are hopeless. I note, for example, the complete absence of South America… They are likely also worthless as a ‘cross check’ on the MEAN trends. Yeah, I’ll plug on through looking for something of interest, but I don’t see where it can come from…
Report of Thermometer Years for Mean, Min, and Max
With that said, here is the report of “Thermometer Years of Mean, Min, and Max records by country”. For each country, I find all records starting with that country code, and count them. They could be all “missing data flags” or all perfect. I don’t select on the temperature values at all. And a value of 20 could be one thermometer reporting for 20 years (and not necessarily contiguous…) or 20 thermometers in one year (or 10 thermometers with 2 Duplicate Numbers each; in one year). It’s just a raw count of “thermometer records”. And it’s not pretty:
[chiefio@Hummer wrk]$ cat MMM_report
For Country Code: 101 ALGERIA Min: 643 Max: 643 Mean: 3378 For Country Code: 102 ANGOLA Min: 0 Max: 0 Mean: 933 For Country Code: 103 BENIN Min: 0 Max: 0 Mean: 539 For Country Code: 104 BOTSWANA Min: 100 Max: 99 Mean: 302 For Country Code: 105 BURKINA FASO Min: 0 Max: 0 Mean: 757 For Country Code: 106 BURUNDI Min: 0 Max: 0 Mean: 20 For Country Code: 107 CAMEROON Min: 0 Max: 0 Mean: 1102 For Country Code: 108 CAPE VERDE Min: 0 Max: 0 Mean: 210 For Country Code: 109 CENTRAL AFRICAN REPUBLIC Min: 0 Max: 0 Mean: 1154 For Country Code: 110 CHAD Min: 0 Max: 0 Mean: 717 For Country Code: 111 COMOROS Min: 0 Max: 0 Mean: 309 For Country Code: 112 CONGO Min: 0 Max: 0 Mean: 1297 For Country Code: 113 COTE D IVOIRE Min: 0 Max: 0 Mean: 1251 For Country Code: 114 DJIBOUTI Min: 0 Max: 0 Mean: 74 For Country Code: 115 EGYPT Min: 463 Max: 469 Mean: 1823 For Country Code: 116 ERITREA Min: 0 Max: 0 Mean: 130 For Country Code: 117 ETHIOPIA Min: 0 Max: 0 Mean: 731 For Country Code: 118 GABON Min: 0 Max: 0 Mean: 1256 For Country Code: 119 GHANA Min: 103 Max: 104 Mean: 535 For Country Code: 120 GUINEA Min: 0 Max: 0 Mean: 436 For Country Code: 121 GUINEA-BISSAU Min: 0 Max: 0 Mean: 66 For Country Code: 122 KENYA Min: 115 Max: 117 Mean: 634 For Country Code: 123 LIBERIA Min: 0 Max: 0 Mean: 150 For Country Code: 124 LIBYA Min: 295 Max: 295 Mean: 1424 For Country Code: 125 MADAGASCAR Min: 0 Max: 0 Mean: 972 For Country Code: 126 MALAWI Min: 0 Max: 0 Mean: 303 For Country Code: 127 MALI Min: 0 Max: 0 Mean: 1876 For Country Code: 128 MAURITANIA Min: 0 Max: 0 Mean: 1040 For Country Code: 129 MAURITIUS Min: 0 Max: 0 Mean: 383 For Country Code: 130 MOROCCO Min: 66 Max: 66 Mean: 679 For Country Code: 131 MOZAMBIQUE Min: 0 Max: 0 Mean: 1069 For Country Code: 132 NAMIBIA Min: 54 Max: 54 Mean: 234 For Country Code: 133 NIGER Min: 0 Max: 0 Mean: 1162 For Country Code: 134 NIGERIA Min: 302 Max: 313 Mean: 1169 For Country Code: 136 SAO TOME AND PRINCIPE Min: 0 Max: 0 Mean: 107 For Country Code: 137 SENEGAL Min: 0 Max: 0 Mean: 1273 For Country Code: 138 SEYCHELLES Min: 0 Max: 0 Mean: 251 For Country Code: 139 SIERRA LEONE Min: 91 Max: 91 Mean: 461 For Country Code: 140 SOMALIA Min: 185 Max: 187 Mean: 350 For Country Code: 141 SOUTH AFRICA Min: 2431 Max: 2432 Mean: 4678 For Country Code: 148 SUDAN Min: 795 Max: 794 Mean: 3248 For Country Code: 149 TANZANIA Min: 198 Max: 197 Mean: 870 For Country Code: 150 THE GAMBIA Min: 0 Max: 0 Mean: 193 For Country Code: 151 TOGO Min: 0 Max: 0 Mean: 643 For Country Code: 152 TUNISIA Min: 0 Max: 0 Mean: 634 For Country Code: 153 UGANDA Min: 152 Max: 153 Mean: 307 For Country Code: 154 ZAIRE Min: 0 Max: 0 Mean: 731 For Country Code: 155 ZAMBIA Min: 268 Max: 277 Mean: 843 For Country Code: 156 ZIMBABWE Min: 183 Max: 178 Mean: 840 For Country Code: 157 AMSTERDAM ISLAND FRANCE Min: 0 Max: 0 Mean: 133 For Country Code: 158 ASCENSION ISLAND U.K. Min: 0 Max: 0 Mean: 81 For Country Code: 159 CANARY ISLANDS SPAIN Min: 0 Max: 0 Mean: 541 For Country Code: 160 CEUTA SPAIN Min: 0 Max: 0 Mean: 31 For Country Code: 161 CHAGOS ARCHIPELAGO U.K. Min: 30 Max: 30 Mean: 71 For Country Code: 162 LESOTHO Min: 0 Max: 0 Mean: 10 For Country Code: 163 MAYOTTE FRANCE Min: 0 Max: 0 Mean: 83 For Country Code: 164 MELILLA SPAIN Min: 0 Max: 0 Mean: 0 For Country Code: 165 REUNION ISLAND FRANCE Min: 0 Max: 0 Mean: 127 For Country Code: 166 RWANDA Min: 0 Max: 0 Mean: 0 For Country Code: 167 SWAZILAND Min: 0 Max: 0 Mean: 0 For Country Code: 168 TROMELIN ISLAND FRANCE Min: 0 Max: 0 Mean: 103 For Country Code: 169 WESTERN SAHARA MOROCCO Min: 0 Max: 0 Mean: 85 For Country Code: 201 AFGHANISTAN Min: 0 Max: 0 Mean: 252 For Country Code: 202 BAHRAIN Min: 0 Max: 0 Mean: 111 For Country Code: 203 BANGLADESH Min: 698 Max: 702 Mean: 1875 For Country Code: 204 CAMBODIA Min: 0 Max: 0 Mean: 114 For Country Code: 205 CHINA Min: 15608 Max: 15599 Mean: 31325 For Country Code: 206 DEMOCRATIC PEOPLES REPUBLIC OF KOREA Min: 0 Max: 0 Mean: 726 For Country Code: 207 INDIA Min: 0 Max: 0 Mean: 7625 For Country Code: 208 IRAN Min: 0 Max: 0 Mean: 1229 For Country Code: 209 IRAQ Min: 0 Max: 0 Mean: 964 For Country Code: 210 JAPAN Min: 11916 Max: 11917 Mean: 16698 For Country Code: 211 KAZAKHSTAN Min: 1556 Max: 1462 Mean: 6742 For Country Code: 212 KUWAIT Min: 31 Max: 31 Mean: 193 For Country Code: 213 KYRGYZSTAN Min: 131 Max: 120 Mean: 505 For Country Code: 214 LAOS Min: 0 Max: 0 Mean: 382 For Country Code: 215 MONGOLIA Min: 0 Max: 0 Mean: 2531 For Country Code: 216 MYANMAR Min: 0 Max: 0 Mean: 790 For Country Code: 217 NEPAL Min: 0 Max: 0 Mean: 68 For Country Code: 218 OMAN Min: 0 Max: 0 Mean: 217 For Country Code: 219 PAKISTAN Min: 2237 Max: 2259 Mean: 5055 For Country Code: 220 QATAR Min: 32 Max: 32 Mean: 87 For Country Code: 221 REPUBLIC OF KOREA Min: 2165 Max: 2166 Mean: 4773 For Country Code: 222 RUSSIAN FEDERATION ASIAN SECTOR Min: 7937 Max: 7157 Mean: 33198 For Country Code: 223 SAUDI ARABIA Min: 29 Max: 29 Mean: 761 For Country Code: 224 SRI LANKA Min: 0 Max: 0 Mean: 1592 For Country Code: 227 TAJIKISTAN Min: 239 Max: 237 Mean: 660 For Country Code: 228 THAILAND Min: 974 Max: 973 Mean: 2046 For Country Code: 229 TURKMENISTAN Min: 617 Max: 574 Mean: 2175 For Country Code: 230 UNITED ARAB EMIRATES Min: 0 Max: 0 Mean: 31 For Country Code: 231 UZBEKISTAN Min: 466 Max: 447 Mean: 1658 For Country Code: 232 VIETNAM Min: 0 Max: 0 Mean: 470 For Country Code: 233 YEMEN Min: 0 Max: 0 Mean: 116 For Country Code: 234 MACAU PORTUGAL Min: 0 Max: 0 Mean: 94 For Country Code: 235 MALDIVES Min: 0 Max: 0 Mean: 0 For Country Code: 236 TAIWAN Min: 0 Max: 0 Mean: 79 For Country Code: 301 ARGENTINA Min: 0 Max: 0 Mean: 7953 For Country Code: 302 BOLIVIA Min: 0 Max: 0 Mean: 1431 For Country Code: 303 BRAZIL Min: 0 Max: 0 Mean: 4434 For Country Code: 304 CHILE Min: 0 Max: 0 Mean: 2543 For Country Code: 305 COLOMBIA Min: 0 Max: 0 Mean: 1072 For Country Code: 306 ECUADOR Min: 0 Max: 0 Mean: 733 For Country Code: 307 GUYANA Min: 0 Max: 0 Mean: 199 For Country Code: 308 PARAGUAY Min: 0 Max: 0 Mean: 1466 For Country Code: 309 PERU Min: 0 Max: 0 Mean: 2863 For Country Code: 312 SURINAME Min: 0 Max: 0 Mean: 306 For Country Code: 313 URUGUAY Min: 0 Max: 0 Mean: 959 For Country Code: 314 VENEZUELA Min: 0 Max: 0 Mean: 1830 For Country Code: 315 FRENCH GUIANA FRANCE Min: 0 Max: 0 Mean: 245 For Country Code: 316 FALKLAND ISLANDS U.K. Min: 0 Max: 0 Mean: 167 For Country Code: 317 SOUTH GEORGIA U.K. Min: 0 Max: 0 Mean: 190 For Country Code: 401 BARBADOS Min: 0 Max: 0 Mean: 161 For Country Code: 402 BELIZE Min: 0 Max: 0 Mean: 243 For Country Code: 403 CANADA Min: 32640 Max: 32628 Mean: 49595 For Country Code: 405 COSTA RICA Min: 0 Max: 0 Mean: 309 For Country Code: 406 CUBA Min: 60 Max: 60 Mean: 518 For Country Code: 407 DOMINICAN REPUBLIC Min: 0 Max: 0 Mean: 712 For Country Code: 408 EL SALVADOR Min: 0 Max: 0 Mean: 131 For Country Code: 409 GRENADA Min: 0 Max: 0 Mean: 63 For Country Code: 410 GUATEMALA Min: 0 Max: 0 Mean: 174 For Country Code: 411 HAITI Min: 0 Max: 0 Mean: 88 For Country Code: 412 HONDURAS Min: 23 Max: 23 Mean: 964 For Country Code: 413 JAMAICA Min: 0 Max: 0 Mean: 271 For Country Code: 414 MEXICO Min: 0 Max: 0 Mean: 7533 For Country Code: 415 NICARAGUA Min: 0 Max: 0 Mean: 170 For Country Code: 416 PANAMA Min: 26 Max: 26 Mean: 197 For Country Code: 417 SAINT KITTS AND NEVIS Min: 0 Max: 0 Mean: 10 For Country Code: 423 THE BAHAMAS Min: 0 Max: 0 Mean: 229 For Country Code: 424 TRINIDAD AND TOBAGO Min: 0 Max: 0 Mean: 221 For Country Code: 425 UNITED STATES OF AMERICA Min: 155224 Max: 155251 Mean: 194577 For Country Code: 426 ANTIGUA AND BARBUDA Min: 0 Max: 0 Mean: 0 For Country Code: 427 BERMUDA U.K. Min: 45 Max: 45 Mean: 236 For Country Code: 428 BRITISH VIRGIN ISLANDS U.K. Min: 0 Max: 0 Mean: 0 For Country Code: 429 CAYMAN ISLANDS U.K. Min: 24 Max: 24 Mean: 24 For Country Code: 430 DOMINICA Min: 0 Max: 0 Mean: 0 For Country Code: 431 GREENLAND DENMARK Min: 44 Max: 45 Mean: 1294 For Country Code: 432 GUADELOUPE FRANCE Min: 0 Max: 0 Mean: 121 For Country Code: 433 MARTINIQUE FRANCE Min: 0 Max: 0 Mean: 119 For Country Code: 434 NETHERLANDS ANTILLES NETHERLANDS Min: 0 Max: 0 Mean: 181 For Country Code: 435 PUERTO RICO U.S.A. Min: 127 Max: 127 Mean: 803 For Country Code: 436 SAINT LUCIA Min: 0 Max: 0 Mean: 0 For Country Code: 437 SAINT VINCENT AND THE GRENADINES Min: 0 Max: 0 Mean: 0 For Country Code: 438 SAINT PIERRE and MIQUELON ISLAND Min: 0 Max: 0 Mean: 58 For Country Code: 439 TURKS AND CAICOS ISLANDS Min: 0 Max: 0 Mean: 0 For Country Code: 440 VIRGIN ISLANDS U.S.A. Min: 27 Max: 27 Mean: 277 For Country Code: 501 AUSTRALIA Min: 29024 Max: 29089 Mean: 35989 For Country Code: 502 FIJI Min: 0 Max: 0 Mean: 457 For Country Code: 503 INDONESIA Min: 829 Max: 831 Mean: 3275 For Country Code: 504 KIRIBATI Min: 19 Max: 19 Mean: 220 For Country Code: 505 MALAYSIA Min: 793 Max: 758 Mean: 1346 For Country Code: 506 NAURU Min: 27 Max: 27 Mean: 93 For Country Code: 507 NEW ZEALAND Min: 0 Max: 0 Mean: 1747 For Country Code: 508 PAPUA NEW GUINEA Min: 188 Max: 184 Mean: 710 For Country Code: 509 PHILIPPINES Min: 1364 Max: 1378 Mean: 2614 For Country Code: 511 SINGAPORE Min: 0 Max: 0 Mean: 119 For Country Code: 512 SOLOMON ISLANDS Min: 114 Max: 114 Mean: 211 For Country Code: 517 TONGA Min: 0 Max: 0 Mean: 61 For Country Code: 518 TUVALU Min: 0 Max: 0 Mean: 159 For Country Code: 520 VANUATU Min: 57 Max: 57 Mean: 329 For Country Code: 521 AMERICAN SAMOA U.S.A. Min: 41 Max: 41 Mean: 146 For Country Code: 522 BRUNEI Min: 0 Max: 0 Mean: 0 For Country Code: 523 CHRISTMAS ISLAND AUSTRALIA Min: 23 Max: 23 Mean: 51 For Country Code: 524 COCOS ISLANDS AUSTRALIA Min: 55 Max: 55 Mean: 167 For Country Code: 525 COOK ISLANDS NEW ZEALAND Min: 0 Max: 0 Mean: 413 For Country Code: 526 CORAL SEA ISLANDS AUSTRALIA Min: 67 Max: 67 Mean: 174 For Country Code: 527 FEDERATED STATES OF MICRONESIA Min: 177 Max: 177 Mean: 556 For Country Code: 528 FRENCH POLYNESIA FRANCE Min: 0 Max: 0 Mean: 988 For Country Code: 529 GUAM U.S.A. Min: 48 Max: 48 Mean: 105 For Country Code: 530 JOHNSTON ATOLL U.S.A. Min: 27 Max: 27 Mean: 76 For Country Code: 531 MARSHALL ISLANDS Min: 133 Max: 133 Mean: 302 For Country Code: 532 NEW CALEDONIA FRANCE Min: 0 Max: 0 Mean: 280 For Country Code: 533 NIUE NEW ZEALAND Min: 0 Max: 0 Mean: 57 For Country Code: 534 NORFOLK ISLAND AUSTRALIA Min: 69 Max: 69 Mean: 212 For Country Code: 535 NORTHERN MARIANA ISLANDS U.S.A. Min: 75 Max: 75 Mean: 165 For Country Code: 536 BELAU Min: 58 Max: 58 Mean: 195 For Country Code: 537 PITCAIRN ISLAND U.K. Min: 0 Max: 0 Mean: 39 For Country Code: 538 TOKELAU Min: 0 Max: 0 Mean: 50 For Country Code: 539 WAKE ISLAND U.S.A. Min: 58 Max: 58 Mean: 141 For Country Code: 540 WALLIS AND FUTUNA FRANCE Min: 0 Max: 0 Mean: 116 For Country Code: 541 SAMOA Min: 0 Max: 0 Mean: 160 For Country Code: 601 ALBANIA Min: 0 Max: 0 Mean: 103 For Country Code: 602 ARMENIA Min: 185 Max: 146 Mean: 448 For Country Code: 603 AUSTRIA Min: 169 Max: 169 Mean: 1866 For Country Code: 604 AZERBAIJAN Min: 83 Max: 80 Mean: 429 For Country Code: 605 BELARUS Min: 229 Max: 194 Mean: 835 For Country Code: 606 BELGIUM Min: 0 Max: 0 Mean: 254 For Country Code: 607 BOSNIA AND HERZEGOVINA Min: 163 Max: 162 Mean: 288 For Country Code: 608 BULGARIA Min: 0 Max: 0 Mean: 589 For Country Code: 609 CROATIA Min: 197 Max: 195 Mean: 759 For Country Code: 610 CYPRUS Min: 0 Max: 0 Mean: 203 For Country Code: 611 CZECH REPUBLIC Min: 85 Max: 85 Mean: 988 For Country Code: 612 DENMARK Min: 0 Max: 0 Mean: 984 For Country Code: 613 ESTONIA Min: 122 Max: 122 Mean: 591 For Country Code: 614 FINLAND Min: 0 Max: 0 Mean: 1394 For Country Code: 615 FRANCE Min: 73 Max: 73 Mean: 3667 For Country Code: 616 GEORGIA Min: 159 Max: 144 Mean: 564 For Country Code: 617 GERMANY Min: 113 Max: 113 Mean: 5443 For Country Code: 618 GREECE Min: 33 Max: 33 Mean: 1349 For Country Code: 619 HUNGARY Min: 0 Max: 0 Mean: 840 For Country Code: 620 ICELAND Min: 56 Max: 56 Mean: 1078 For Country Code: 621 IRELAND Min: 436 Max: 436 Mean: 1979 For Country Code: 622 ISRAEL Min: 0 Max: 0 Mean: 627 For Country Code: 623 ITALY Min: 2264 Max: 2265 Mean: 4961 For Country Code: 624 JORDAN Min: 0 Max: 0 Mean: 260 For Country Code: 625 KAZAKHSTAN Min: 0 Max: 0 Mean: 75 For Country Code: 626 LATVIA Min: 141 Max: 120 Mean: 565 For Country Code: 627 LEBANON Min: 0 Max: 0 Mean: 432 For Country Code: 628 LITHUANIA Min: 195 Max: 171 Mean: 817 For Country Code: 629 LUXEMBOURG Min: 0 Max: 0 Mean: 144 For Country Code: 630 MALTA Min: 0 Max: 0 Mean: 218 For Country Code: 631 MOLDOVA Min: 74 Max: 61 Mean: 373 For Country Code: 632 MONTENEGRO Min: 86 Max: 86 Mean: 172 For Country Code: 633 NETHERLANDS Min: 0 Max: 0 Mean: 406 For Country Code: 634 NORWAY Min: 0 Max: 0 Mean: 3410 For Country Code: 635 POLAND Min: 2055 Max: 2055 Mean: 4194 For Country Code: 636 PORTUGAL Min: 21 Max: 21 Mean: 1386 For Country Code: 637 ROMANIA Min: 0 Max: 0 Mean: 1526 For Country Code: 638 RUSSIAN FEDERATION EUROPEAN SECTOR Min: 3176 Max: 2992 Mean: 13497 For Country Code: 639 SERBIA Min: 354 Max: 352 Mean: 452 For Country Code: 641 SLOVAKIA Min: 39 Max: 39 Mean: 392 For Country Code: 643 SPAIN Min: 75 Max: 75 Mean: 2603 For Country Code: 645 SWEDEN Min: 0 Max: 0 Mean: 1714 For Country Code: 646 SWITZERLAND Min: 0 Max: 0 Mean: 1410 For Country Code: 647 SYRIA Min: 0 Max: 0 Mean: 657 For Country Code: 648 MACEDONIA Min: 100 Max: 106 Mean: 228 For Country Code: 649 TURKEY Min: 7155 Max: 7155 Mean: 9301 For Country Code: 650 UKRAINE Min: 1031 Max: 1012 Mean: 4864 For Country Code: 651 UNITED KINGDOM Min: 172 Max: 172 Mean: 4471 For Country Code: 652 FAROE ISLANDS DENMARK Min: 0 Max: 0 Mean: 205 For Country Code: 653 GIBRALTAR U.K. Min: 0 Max: 0 Mean: 161 For Country Code: 654 MADEIRA ISLANDS PORTUGAL Min: 0 Max: 0 Mean: 249 For Country Code: 700 ANTARCTICA Min: 271 Max: 293 Mean: 2741 For Country Code: 800 SHIP STATIONS Min: 0 Max: 0 Mean: 437
Well, at least it’s an answer of a sort. We know that swapping over to using MIN or MAX to compute the “Global Warming” trend is not going to cut it; and we know that even using them for “Spot Checks” is going to be largely useless.
Postscript: Some AGW True Believers have ‘had issues’ with my talking about my “average of temperature” reports as a way of “measuring the data”. That is, they don’t tell me the actual “Average Temperature” (by which I presume they really meant ‘Mean Temperature’) but rather they tell me about the structure of the temperature records.
Well, here is a very simple and very explicit example of “measuring the data”, though in this case it’s looking at “do they exist at all” vs “how big are they”… but this “characterize the data” is a needed first step. It tells you “what is possible” and it tells you “what is a fools errand”. And computing a Global Average Temperature MIN or a Global Average Temperature MAX are even worse fools errands than computing a Global Average Temperature MEAN.
As I look at yours and other data, I tell people I cannot disprove AGW, but I don’t think anyone can have any reasonable certainty that it does exist. I know this is a rather broad question, but after all the time you’ve spent looking at the data where do you stand?
REPLY: [ Well, sit, actually, fresh coffee to hand, watching CNBC as I need to get back “in sync” with markets and make some money to feed my “hobby”… OK, to know where I am now, it’s best to know where I came from (then “now” becomes more useful).
I started on this journey about 4 years ago (? a guess) with a “Gee, Global Warming looks like a big deal and must be an interesting thing to look at. Both from an investing potential impacts and from just a natural sciences point of view.” The more I looked (and the more books I read about it) the more I noticed a pattern: Most of the published “stuff” was of the form “ASSUMING Global Warming, what bad thing will happen?”
Well, I know that pattern very well. It is the standard formula for Science Fiction. Assume space aliens want to invade earth, what bad thing will happen? Assume an astroid will strike Earth (send Bruce Willis!) what bad thing will happen (other than TWO movies with the same theme coming out together…) Assume then Panic. And as much as I love SciFi, I hate assuming… So I bought more books and hit more web sites.
Then I made the mistake of asking “An Inconvenient Question” or two at some Warmer sites. RealClimate, for example. I just wanted to know the truth, so I had “things that don’t fit” (and nothing bothers A Tidy Mind, especially a borderline Aspe one, quite so much as things that will not ‘fit’… heck, I even wash the spoons in one batch and the forks in another batch so they don’t have to be mixed together… well, and I figured out that they must eventually be sorted anyway and “sort first’ is slightly more efficient… but I digress 8-) So I asked questions like: “It was cool in the ’60s and ’70s and we had the Ice Age scare – it snowed in my home town, where it almost never snows: that means the “baseline” is set in a broken place, doesn’t it?” or “How come the past keeps getting re-written colder?” ( I really don’t like folks moving my past around…) and just got attacked.
The more I looked, the more I found “broken stuff” masquerading as Science. And a big chunk of it was hanging on this “GIStemp” thing. So I bitched about the code and the data being “unavailable” ( a bit of a phishing expedition, truth be known… I was feeling a bit lazy and didn’t want to spend the time needed to find out on my own, so I just tossed a “bitch” out on WUWT and a “kindly Warmer” responded with some vehemence that I was an idiot and it was all published at (links to places). That made my life easier as they gave me what I wanted.) Then, after about 5 months of complaining that “Somebody” needed to make this thing go and test it (I’d looked inside and recoiled in horror…) I decided that if nobody else was going to do it, well, “I was somebody”.
By this time I was pretty firmly in the Skeptic camp. Having moved from “lukewarmer” with a bit of doubt that maybe they were just confounding a LIA rebound and ’70s cold excursion in the baseline with a human caused event. At WUWT I learned a great deal about climatology and weather and other links to interesting places (like solar observatories and NCDC and other countries Bureaus of Meteorology). And I noticed another pattern. When you go turning over rocks, if things are “normal” you get about as many “bad” surprises as “good” surprises. In an unbiased search you would expect to find a bag of quarters missing from the vault, but two bags of dimes in the vault without a record and a bag of nickels that is twice the size it’s supposed to be. Basically, random error tends to cancel. But almost every time a rock got turned over, there was another roach running for cover on the Warmer side… The classic indicia of a hidden “awshit” somewhere. So over time (and with a library now of about 3 feet length on the bookcase) I’ve become a modestly “Confirmed Skeptic”.
OK, that brings us up to about 1 year (?guess) ago. Having gotten GIStemp to run and read the code “end to end” a couple of times. It’s pretty rough. It has some “issues” that impact the actual function, but they tend to be fairly small. Most of the “issues” are “quality control” issues. Basically, I think it “works as advertized, mostly” but still has some “issues” that need investigation that could have a major impact. (In particular, investigating “anomaly processing” has left me with 2 ways in which it fails that are not addressed in GIStemp as near as I can tell – though I exploit one of them in dT/dt to make it better able to “spot” discontinuities in the data… that “Bullseye” that happens at transistions… where all the monthly anomalies pass through zero…) I’ve run a couple of benchmarks that show a 1/2 C or so “lift” to the data as it is processed prior to the “grid / box” anomaly step, and I’ve benchmarked STEP3 and found that about that much “warming” leaks through (that is, the “Box of Thermometers A compared to Box of Thermometers B” method of doing grid/box anomalies fails to do a clean anomaly and ‘leaks’ about 1/2 C of bias through). So I was all wound up to pronounce GIStemp as “Bogus in the 1/2 C degree” when 2 things happened. One was that I noticed “strange things” in the input data itself, the other was that Climategate broke. I spend ‘a while’ examining the data – all those “by altitude” and “by latitude” and “by airport flag” etc. reports.
That pointed me firmly at the data as “cooked”. And that brings us up to “now”.
Having gone through this dT/dt driven anomaly graphing exercise it’s pretty clear to me what’s going on. Frankly, the Fiji graph as compared to Micronesia and Samoa (with Tonga as seasoning…) pushed me “over the line”. It just screams fudge. If I were doing an undercover investigation (as I’ve done at some sites) I’d be sitting in the office of the V.P. (or whoever knew my actual job as investigator) discussing how to proceed: “Walk them out, bring in law enforcement, or bait and trap for a bigger catch?” (Most companies would just ‘walk them out’. Japanese companies especially want no bad news ever. Had a batch of guys stealing equipment from the dock and doctoring the books; just walked out. Nutty, IMHO, but the customer wanted discretion.)
So we take a look at a couple of places in the middle of the Pacific with ONE thermometer and they don’t screw around with it:
Dead Flat. Now you don’t get dead flat in a world of warming.
Then we look over at Singapore:
And get a great little “Hockey Stick” right at the thermometer rotation ( that blip up in the yellow line followed by a blip back down as the swap completes). That is just not consistent with steadily rising CO2 levels as causal NOR with UHI from industrial growth. It IS consistent with fudging the books.
Then we take a look at French Polynesia. Generally the French have been good indicators of what’s really happening, but here we have a ‘drop;’ in the baseline:
Notice how the baseline “sags” as a mirror image of the thermometer count line rising? In theory, thermometer count ought to be unrelated (if the process were clean the actual count of thermometers ought to have little relationship, sometimes up, often neutral, sometimes down – what we find is a bit different):
And you see that pattern in other places as well. Counts change at the start of the baseline, and the baseline drops, then they change at the end and the baseline rises, then in 1990 ‘something changes’ that makes a dramatic hockey stick blade in some place and a pivot / ramp in almost all of them. It is coincident with thermometer dropping on a massive basis, but could be just a change of processing that happened at the same time (i.e. “duplicate number” or “mod flag” changes on the survivors, but we don’t know if the choice to survive is the same as the change of flag… nor if the dropped records could be subjected to the same treatment to get the same ‘lift”…)
Then finally, there is Fiji. So much change in that ‘thermometer line’ right when the temperature line makes a great “dip” baseline and a “lift” coming out of it, then the “pivot” right at the “bullseye” about 1992 start of the Rocket Ride hot pink segment. All worked into a generally ‘steady rising’ trend of +2 C over all? Now look at Cooks and Cocos again… SOMEBODY is wrong in this picture… and its the guys screwing around with the thermometers…
The only thing I can’t speak to is “motivation”. Folks have a remarkable capacity for self deception. (Google “Clever Hans” – everyone thought the horse understood language… when he just had a great ‘make boss happy’ detector..) So was this a ‘deliberate act of data vandalism’ or was it ‘expectation driven accidental consequence’? That’s not my job. That’s for the jury to decide.
It’s quite possible, for example, that someone decided to put in place an “outlier QA process” that has asymmetrical impact, tossing out more low lows than high highs. ( In fact, I’m almost certain that’s the case given the suppression of low going excursions). They may have done it knowing the effect as a deliberate act, or they may have simply “picked some parameters out of thin air that ‘seemed good’ and were not”. So, for example, downside excursions are further (cold air does not convectively mix by rising, it just lays there) than hot excursions (hot air on the surface rises, mixing). If you ‘clip equally’ you have unequal impact when compared to a past data base that is “unclipped” ( which is, btw, what I think is happening at the 1990 “bullseye” pivot…)
So if this were a contract (as I’ve done before for security departments – that whole FBI background check fingerprints on file thing…) I’d be sitting with the Chief Mumble and explaining that we had enough goods to walk them out on the spot. I would be recommending involvement of law enforcement at the ‘getting to know the story’ level but not yet at the ‘raid / bust em’ level. And I’d be asking if they wanted to ‘bait the trap’ to get motivational information. I’d also ask that a ‘Disaster Recovery Set’ of all data be dumped so as to prevent any ability to ‘erase the evidence’. (But in fact, I prefer to do that as soon as I show up on a site. One of my first recommendations is a “We need a disaster recovery set and don’t have one so we’re just going to dump everything to tape while we design the process” fire drills. Most folks don’t “spook” over that and it gives you an archive you can place out of reach in a very private vault… but you do need someone watching the creation to detect when someone “decides to skip that irrelevant box over there” ;-)
So, OK, another 100x response to a 1 x question ;-) The “short form” would be:
“Book ’em Danno!”
Then I’d start over from ONLY the original records of the real raw data and I’d specifically forbid mixing different “processes” over time and I’d stabilize what thermometers were used as much as absolutely possible. And I’d have the work done by someone completely unrelated to the prior crop of “climate scientists” (preferably 2 isolated teams). Then I’d think we might have some clue what’s really going on… And I’d expect to find a LIA, a cold ’70s, and not much different between now and before the LIA and the 1930’s in North America.
When you are doing a divination using sheep entrails as they did in Rome, you start with a live sheep – assumption is that it’s more likely to have a complete set to work with.
It really boggles to see that they didn’t even go looking for a live sheep. Maybe they did look, and there weren’t any.
I had sort of been hoping to see how the min temp trends compared to the max and whether min was rising and max wasn’t.
certainly there the inertia of mass/temperature has to vary by location – think Miami, where, in the summer it gets up to 93-95F everyday but very seldom higher and then cools into the low 80s (IIRC) at night.
one might imagine that different feedback mechanisms impinge on minimum than maximum. There seems to be a lot to think about.
One other thing. The notion that “they don’t do it this way” seems bogus. As you evolve the ways you look at the data, it’s true that you may see some logic in the way “they’ve” looked at it. At the same time, maybe not.
REPLY: [ Well, there are still some countries with MIN / MAX data and it might be possible to find ranges over time that match the MEAN during “interesting times”. But the quantity of work to make things “match” is going to be an issue. The big PITA to me is that we can’t do a “cross-foot validation”. One would expect a plot of MIN MAX and MEAN to have them all moving together and with MEAN more or less in the middle. An excursion of MAX down during the baseline and MIN up after the 1990 pivot would be a big Smoking Gun (and was what I saw hints of in a couple of countries). But all the “most interesting countries” were coming up “Zero Data”… which is why I did this report. Now I’m just trying to figure out if I’ve got enough ‘sheep’ left to even make a decent Haggis out of it ;-) Maybe, well see… -E. M. Smith ]
Linkage between the use of “mean = max – min” and modern suppression of downward spikes in the raw data?
REPLY: [ I don’t know. I suspect it’s an “improved QA process” that it tossing “excursions” and having a differential impact, but that is just a working hypothesis at this point. It’s what I was hoping to figure out by looking at changes to MIN and MAX over time… So now I need a new approach compared to what was planned. -E.M.Smith ]
Oh, one point that does matter: The docs per min and max mention that the “Duplicate Number flag” represents that duplicate number used on the MEAN as “there is only one way to calculate a MIN” (or MAX) average. That being a simple average.
So for some of those countries with a large MEAN count, but a MIN or MAX of about 1/2 that size, it could simply be that the MEAN has many “Duplicate Numbers” so more duplicated data that do not make sense in a MIN or MAX file as they would be identical for the different “Duplicate Number” series.
So there are potentials for valid MIN/MAX vs MEAN reports from those places that have a MIN / MAX count smaller than the MEAN count. But it will just be a PITA to match the data up and verify what’s being compared to whom…
Occams Chainsaw – the MEAN only, no MIN/MAX records are places that are not using electronic MMTS type instruments?
Maybe just casting the Mk1. eyeball at the mercury at the designated time each day.
Pingback: Arctic sea ice continues to rise « TWAWKI
Dec 2009 Vintage GHCN?
Is there any chance a fan has laid down an earlier vintage?
Assume, just for fun, a copy of GHCN data from 2007. How troublesome would it be to run a diff or file compare?
@Pouncer: I have a pointer at an earlier Vintage. It’s not that hard ( download, run about 10 minutes worth of scripted programs, do graph). Maybe an hour all told? But I’m trying to complete one bright idea before I start another… It’s a “Polish Point” for me… (as HR would say when telling you how you tended to screw up… ;-)
@Chuckles: I think New Zealand would answer your question. It’s got no MIN/MAX but I’m pretty sure it’s gone electronic at the airports that make up almost all it’s data. Someone in Kiwi Land could probably confirm pretty quickly.
Good idea Chiefio, I noticed glancing down the list that a lot of the places that zero mins and maxes were, shall we say, places where hollywood types went to publicise and exhibit their compassion, and this triggered the thought.
Ed taught me to look at the data. All of it. Every table. Every field. Find out what the field names mean. Figure out the relationships.
So many people just start typeing SELECT and assuming what field names mean and spitting out data from queries. And they’re wrong and they’re meaningless and they’re convoluted.
Then I started reading your stuff. Look at the data. Look at the basic analysis first before you put more layers of abstraction and complication on top. It just makes sense.
It seems to me that someone came up with a “model” that sounded good to a couple of folks and they grabbed some undergraduate programmer to try and encapsulate their logic in a program and then started feeding in data. And they’ve done this every day, all over the world for the past 20 years.
PCs are great for word processing but maybe we should take them away from certain “scientists” and create a stat/code/data department that they MUST go through to get their data manipulations and models created and run and to store their data.
I guess I’m a little frustrated. I’m sitting here right now looking at a pile of data. The task order number is named differently in every table that it appears in. TO_NO, TONum, TaskOrderNo, TaskOrderNum, etc. It’s a different data type in the tables. It’s an integer in some, 1, 2, 5, 6, etc. It’s a varchar(3) in some “1”, “2”, “5”, “6” in others (we’re up to 700 and some task orders now so what happens when we get to “999”?). It’s a varchar(5) in some tables. The same number with leading 0s, “00001”, “00002”, “00005”, “00006” in other tables. They join these tables with all sorts of substr’s and string functions. Aaaaaaaaaaaargh!!
Anyway, I really appreciated your answer to Steven and your other posts. Thanks.
REPLY: [ Been there, done that, have the scars… “I feel your pain” doesn’t quite cover it…
Same thing runs throughout the code of GIStemp. “Station ID” can be, variously, 12 digits (including “Duplicate Number” or “Modification History Flag”), 11 digits (without it), 8 or 9 digits (the same two, but leaving off the leading 3 digits of “country code” that usually are not needed to make a unique number…), various lengths shorter than 8 or 9 when these were printed out such that leading “0” was suppressed (i.e. they printed an “Integer” without thinking that it’s not an integer it’s an identifier string of digits…), 5 digit “major station number” or 5+3 (that is, the 8 but as distinct parts) to get the minor modifier with the major number (and with leading zero variations possible)… And that is just ONE of the variables that are treated in this messy and cavalier way. The explosion of complexity from rampant carelessness is just stupidity on steroids, IMHO…
I’d like to see a test where, before they were allowed to write programs, folks were placed in a room and told someone would be with them in an hour or two and that there were magazines on the table near the cake and coffee. They would be in the room with some clean dishes in semi-random piles and some dirty dishes in the sink. Those who organized the dishes (without being told they were expected to do anything…) into Very tidy orderly stacks, and preferably washed and put away the dirty ones too, would be advanced immediately… Folks who organized the whole set of cupboards by form, structure, or use would be sent down the supervisor track… ( i.e. Plates in size order, or “glass with glass, ceramic with ceramic”, or cooking pots here, service pieces there, stemware over here…) Those who had cake and coffee and added dirty dishes to the “clean side’ of the sink would be walked out immediately…
Basically a “Chaos vs Order” filter. Does this person fight entropy, or create it? Those who create entropy do not belong in code. Ever. You are better off walking them out and hiring someone else (or doing nothing). It will save you vast amounts of time, money, and error in the long run. It can suck down 2 good programmers trying to control the damage done by one chaotic person.
Per my postings and answers: You are most welcome. Glad just to know someone gains something from it…
Chiefio, Your reply to the first post is interesting. Thanks for your thoughts and calculations. I sometimes wish I had your computer skills. I managed some basic with my Apple II (1975) but got lost with Fortran and Pascal. I use Metastock to look at some price trends but don’t have the discipline to make much for my super fund.
Your post shows considerable max/min data for Australia. You can download raw data from here http://www.bom.gov.au/climate/data/weather-data.shtml. You should be able to find all the GISS identified stations plus others nearby. Also, you should find stations that may have been dropped.
For interest I have looked at a small rural town Gayndah in Queensland. The number for the Post Office is 039039. The data for temperature dates from 1894 (rainfall from 1870). There appears to be a jump in minimum temperatures about 1975. The station closed in 2009 but there was six years overlap with the Gayndah Airport (2003 and still operating) number 039066. In my assessment this shows + 0.2C UHI for the Gayndah PO. I have not calculated a regression because my version of Excel (home2003)does not do it-I will need to fiddle with the formula. However, I can say that the highest average temperature (also the highest Max. temp) occurred in 1897. The 1890’s was a period of high rainfall while the period following 1900 to 1907 was a drought period with a record low rainfall in 1902. If one takes away the 0.2 UHI effect and then the jump in 1975 then there has been no temperature increase. I can send my spreadsheet with data if you email me.
Warwick Hughes (mentioned in climategate emails) has lots on Australian temperatures eg http://www.warwickhughes.com/blog/?cat=3
REPLY: [ Glad you found something of merit in it ;-) Per computer skills: It mostly takes a willingness to deal with details, and a long time “just doing it”. My first programs were fairly ugly and crude in retrospect… I find computer languages a lot easier than natural languages (more orderly and predictable…) for example. FWIW, I’ve been using OpenOffice and it’s not bad. A bit too “modal” but once you get past the idea that you need to “right click while hovering over the particular item” to get the menu you want, it starts to come together 8-} That it’s a free download is nice too…
Per Aussie data: Yeah, I’m going to be working on that for about a year, I’d guess… I’ve got to finish the current crop of dT/dt based graphs before I start a new thing, though. I’d just figured out that I’ve done roughly 100 graphs in the last couple of days… S. America – 15, Pacific Basin – 33, Asia = 33, N. America about a dozen out of 28. And some of them 2 or 3 times to find the interesting bits to graph. To say I’m a bit “worn” by it is an understatement 8-| so I’m going to knock out the last dozen or so N. America graphs and take a break from “cranking” on ’em. Europe and Africa will have to wait a day or so… Lest I “burn out” on the graphing thing… (Besides, a “look ahead” at the tabular reports shows a lot of the same – flats with hockey sticks with the occasional “little dipper” in key places. So it’s not like some great mystery will be waiting to be uncovered. The present batch pretty much does that.
So don’t be surprised if it takes me a bit to get around to doing an “in depth” on Australia… though I really want to… loved my time “down under” and have some family there.
Though I’ll probably work in a couple of small “for fun” postings along the way ;-)
“The explosion of complexity from rampant carelessness is just stupidity on steroids.”
I’m saving this in a special place. If I find the time, I would like to turn this into one of those “inspirational” posters. I just need to find the ideal graphic. Hmmmmmm. Maybe a hockey stick:-) Thanks
REPLY: [ Be my guest. Though a small “-E.M.Smith” attribution would be a nice touch ;-) -E.M.Smith ]
Dec 2009 Vintage GHCN?
Is there any chance a fan has laid down an earlier vintage?
Assume, just for fun, a copy of GHCN data from 2007. How troublesome would it be to run a diff or file compare?
That was the main reason that I pointed Chiefio at the “old” data, to see if the “Raw” data had actually been changed.
It is interesting to note how different it is.
REPLY: [ And I’m going to get to it Real Soon Now ;-) but I’m trying to be disciplined about things and actually complete the canonical set of graphs before I take on “new” projects. Though, frankly, I’m getting REALLY tired of doing the same repetitive graph making steps and I’m probably going to do something else for a day or two right after North America and just before Europe as a ‘restorative’… -E.M.Smith ]
E M Smith.
I admire your dogged approach to posting. I can only imagine the anxiety this may generate for the desire to alter your direction and post other subject matter. This field moves so quickly at times.:)
The awareness of particular (unpublished) papers has led me to a more engineering orientated conceptualisation (hardly surprising, as I’m an engineer) generated from the introduction of econometrics to climate science. This econometric view was shown to me by Doug L. Hoffman here:
In case his mods don’t permit my post response I repeat it:
I think you’re right here Doug. As an engineer, I can never understand why climate science ‘throws away’ most of the temperature signal and also works with an average global temperature that doesn’t have anything like enough resolution within the station network nodes to give a realistic average temperature to an area. Insufficient baud rate and scattered pixel appearance, respectively, just about says it all.
I was always of the understanding that AGW theory encompasses events where diurnal solar forcing would be reflected as an unaltered warming rate from insolation, but with OLR restricted, the night time cooling rate (given cloud change parameters) would reduce. We’d need 5-10 minute daily temperature readings with a realistic clear sky comparison standard to see this phenomenon and I’ve not seen any data of the like, or any movement towards data collation in this manner. Instead, all I’ve seen is a continual march of average temperatures, carbon dioxide content, ocean heat content, etc.
Back to predictive econometrics. What cointegration does show, in engineering terms, is the relationship of the data to a system attractor. However, from there, the system attractor really needs to be identified. Once the attractor is identified there will be other data relationships that are affected by that attractor. The climate system has many attractors and the greater the understanding of climate subsystem attractors that are known, the greater our understanding of climate per se.
I know what you mean by language terminology. I tried to learn ‘climate speak’ because people didn’t know what I was talking about. Though, I don’t see a paradigm shift taking place without a swift kick up the !”£$%^&*. :)
Best regards, suricat.”
Yes, I know! This doesn’t help you to decide what to do with your spare time!
Best regards, suricat.
The min and max data for New Zealand sites are available at http://cliflo.niwa.co.nz/