GIStemp USHCN.v2 and the Filter Q Continuum

Things Move Forward

GISS has put up an announcement that they have now modified the GIStemp code to use the USHCN.v2 format file. This will “put back in” the 93% of USA thermometers that were dropped when NOAA changed from the USHCN original format to the USHCN.v2 file format, and GIStemp declined to follow. OK, after some degree of badgering, they have fixed that error.

Thanks for doing it. It is welcome progress.

Now if you will just get all the Rest Of World thermometers put back into the GHNC data set, we will be well on our way to stabilizing the “instrument” that is being used in this calorimetry experiment of Climate Change analysis.

Awaiting QA

The rest of this analysis is in the QA pool until I’m feeling better. I’ve got a bug right now and it’s making me a bit fuzzy / dizzy…

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in AGW GIStemp Specific and tagged , . Bookmark the permalink.

8 Responses to GIStemp USHCN.v2 and the Filter Q Continuum

  1. StrongContinental says:

    Dear E.M. Many many congratulations on forcing a change out of GISS! No doubt they will claim it was nothing to do with your demolishment of their code. I love the way on the previous updates, they say such things as “this change affects mainly part of South Africa”, and “This did not significantly change any results” etc. Your work has triggered a similarly pithy update remark, but with not even an estimate of what the “update” affects – only the entire record! Hilarious.

    This whole project has been and still is such an enjoyable tour-de-force. I have been richly enjoying each and every post since you began dismantling the utter shambles of the global “average” temperature calculations, and now refresh this site even more enthusiastically than any other, eagerly awaiting the next shoe-ing of GISSTemps dwindling credibility. Looking back, I’m really quite chuffed that I was already checking out your musings when it all started, so I’ve been able to enjoy the whole sorry tale from start to finish (or rather from start to wherever you have got to here, as I suspect the finish is not even close). It’s been a roller-coaster ride, via the sad story of contaminated cat food, the merits of natural wasp predation of vegetable gardens, and a plethora of other fascinating diversions, not least a close look at the delightful Miss Halle Berry (entirely justifiable on almost any occasion, imho).

    Anyway, I just wanted to chip in with a big thank you for what is not only a highly enjoyable read (very reminiscent of the Cuckoo’s egg as another commenter suggested, or my favourite in that genre: Between Silk and Cyanide – another classic of dogged determination and refusal to accept “anomalies” as not worth digging into), but also a fantastic contribution to the whole “global warming” debate that deserves much more widespread attention and acclaim than it currently appears to be getting – other than it seems from GISS (attention that is, not acclaim of course)…

    I noticed a few comments back that you had a visit from Big Anthony Watts, and I have been refreshing WUWT with nearly as much enthusiasm as Chiefio Musings ever since, assuming a guest post was imminent and some major coverage of your work would follow… but nothing yet. Is this on the cards?

    I have a couple of questions for you, if you can somehow find the time to answer them:

    1) When you refer to the dying of thermometers around the world, can you explain to me exactly who has taken them outside and shot them? Is it the parameters in the GISStemp code, or are these thermometers already missing before it hits the code – in which case who is responsible for maintaining the GHCN records? I guess what I am trying to say is who are the Thermometer Langoliers? I’m sure you explained this somewhere, but I’ve lost track…

    2) It is clear that you have shown that GISSTemp is conclusively broken, even after the restoration of the thousands of thermometers they “lost”, and the only question remaining on this front, is whether this was deliberate or not (and how much more embarrassment you can possibly heap on them – seemingly there is an inexhaustible supply… Each post you make pushes the weight of circumstantial evidence more and more towards the deliberate. I think your readers can make their own mind up on this issue – this question is not that question). So GISSTemp is not “robust”, and HADCrut have grudgingly admitted their data “was eaten by the dog”, and as such can be considered to have since been deposited by said dog, probably on some pavement in Norwich, and is awaiting collection by pooper-scooper, or some unfortunate person’s shoe. This leaves the satellite data to base trillion dollar decisions on, which obviously has a much shorter time-frame than the land-based records. Do you know if any of the satellite calibration is carried out using any of the land-based temp records? Or is the satellite data thankfully immune from this sort of shenanigans and anthropogenic biassing?

    3) Lastly, given your other interests, and that part of successful market investment is spotting when the crowd are heading in the wrong direction, would you care to speculate which stocks/commodities/ETFs etc are worth a punt on the basis that there are still a sizeable proportion of people under the misapprehension that the world is imminently overheating, when in fact it may well be cooling? Not financial advice in any way of course, just idle speculation…

    Keep up the awesome work, and the very enjoyable commentary. I am off now to tip your beer jar… which seems to be shockingly under-utilized – will I really be the first? You really deserve more than one beer for all this investigation. Perhaps it is the requirement for a website to promote that puts people off. I’ve linked to the ridiculous polar bear story from the Telegraph recently, which coined a whole new word that perfectly describes a large amount of the recent publications on our allegedly imminent eco-apocalypse. See if you can spot it – there is a hint in the roll-over if I have used ScratchBack correctly.

    P.S. While I think of beer, I strongly suspect that the regular use of my iPhone to check for new posts or comments is being registered as a new visitor each time, and as such I think that I may have single-handedly propelled tiny Singapore’s flag above the might of the Frenchies and Dutchies plus the Scandawegians to boot. I am now hot on the heels of what is always a well-organized and highly efficient German team… And if I’m not on my own in the Big Red Dot when it comes to Singapore-based Chiefio readers, then please add a comment below fellow Red Dotters, and we can meet up and raise a glass to E.M. – hopefully containing some nice fluorocarbon-refrigerated and CO2-spewing lager, and preferably in an air-conned bar: the best, and only, anthropogenic climate change as far as I can see here in Singapore…

  2. StrongContinental says:

    Hmm, rollover referred to above doesn’t seem to work – so if you need a hint, check out 5th para, 5th last “word”.

    And I am the first. Come on fellow readers, buy E.M. a beer…

  3. pyromancer76 says:

    Congrats, E.M. Smith, for scaring the pants off GISSTemp such that in their embarrassment — bare @$$, yes, for that is what they have been showing to the world — they restore 93% of the dropped USA thermometers. Shame on them. Did they give you credit?

    But you have other surprises waiting as we see in this latest post. Hmmm, the importance of 1/10 of a degree F and 1/10 degree C anomaly rounding “error”. Bet they thought no one would ever notice.

    I imagine your tip jar soon will be overflowing. I subscribe, not tip. So as soon as the next print subscription comes due for renewal, I will be sending that sum to you. Wish others would do so, too, for the cutting edge research and analysis/discussion based on real data. This change would go a long way toward shifting the financial balance of power away from pseudo-science for personal-political purposes. (My contribution will be based on the gratitude as well.)

  4. E.M.Smith says:

    Well, don’t know where to begin! Such wonderful compliments!

    Don’t ever expect to see GISS acknowledge my existence. Hansen would blow a gasket… so nobody will do that while he is the “big boss”. (Especially not with the comment I put in the code ;-)

    One of the things that leaves me a bit rankled is that whole issue of “funding”. Why is it that it falls onto a volunteer nobody to do the basic QA, validation, and performance measurement of this “product”? Why is there no published (peer reviewed, too!) performance data for end to end “Q” and response to thermometer deletion / change? Why is it just assumed that it will be a perfect filter? (When it can not be…)

    WHY has not a congressman had the OMB do a forensic audit of this code (that holds the fate of the world in the balance)? They can audit millions of hotdog stand owners and they can audit car repair places nationwide. They can force folks into bankruptcy on an investigative whim (and do), but not a single sniff up this pipe? Heck, you can’t even register your car without having it’s butt checked… The asymmetry of this arrogance of power and influence is maddening.

    Yes, the “next shoe” that is queued up is just that the USHCN and USHCN.v2 “data” for the same records are significantly different. Some are even different in the whole degrees F position. How can we be worked up over 1/10 C “anomalies” when the input “data” have 1 C “anomalies” of there own? And all subject to human choice?

    Along with that comes the need to even more precisely define and measure “Q”. And right behind that is to “normalize and stabilize the input over time” and see if “AGW” evaporates. (Basically, now that I’ve got the ability to run data end to end via swapping in ‘missing data flags’, what happens if you have the unstable thermometers ‘go missing’ and only keep the stable set?)

    But all that can sit just a while. I really do need to put a day or two into “that money thing”… (The Son called from college and wants a bit more in the till, so I need to go back to the money well and dip a bit more out…)

    The whole ‘beer tip’ thing was an early whim. I’ve not taken the time to go back and make it pretty or work right. So far it isn’t worth the effort to keep it, but I’ve been too busy to remove it. We’ll see. Not high on the priority list…

    Yes, each visit your flag will go up (if it can figure out what to do to increment it) if your IP changes (I think) and a mobile device ought to be a new IP fairly often, maybe every time the phone is turned on.

    I did find it amusing that GISS, just in passing, noted the USHCN.v2 change as though it was nothing; just a little file format change caused by NOAA, common, trivial really… (Just about 2 years late…)

    Per A.W.: We had some discussions, and I was doing an ‘end to end’ revalidate of the benchmark and impact as a more formal Q.A. and defensible write up when NASA GISS decided to “change the game”. So at this point I doubt that it is worth a WUWT posting to say “What was a big deal, is now history, but the history of change, is now changed, so was what was a big deal still a big deal or did the deal change, change the change?”… Anthony is “on the road” and I’m “in the moment”, so who knows where it will lead.

    I’m mostly now looking forward to what the GIStemp output change will be. If it is negligible, then I get to do a new ‘end to end’ code review to see where they put the magic sauce…

    On a pleasant note: Both cats are doing well now. Himself has added weight and has a better fuller coat (he had never ‘looked sick’ but clearly likes the food change better). Both eat much more (they clearly like the food better!). Madam has had a complete recovery from the drippy eyes, sneezing, etc. And most thankfully, the incontinence and upchucking are both in the past. Her weight has roughly doubled and she is a very happy kitty now. The “avoid urea formaldehyde byproduct contamination risks” thesis continues to be a valid predictor of acceptable foods (we’ve got 1/2 dozen choices now, all picked via that predictor and all accurate).

    The “Great Dying of Thermometers” is a metaphorical device. The thermometers are still there, still reporting. But only to the national bureaus. Just like the USHCN vs GHCN sets. The “data set manager” for GHCN is the person responsible for “The Great Dying of Thermometers” from the GHCN record. I put his contact information up on one of the postings. I’ll dig it out again later (after morning tea ;-) He is a NASA employee who manages the NOAA data set, so it is rather significantly disingenuous for NASA to say “oh, NOAA did it” when their boy is in the drivers seat…

    Finally, the “investing” thing: Predicting where the crowd will go can be useful, but more useful is watching where the big money is moving (it moves ahead of the crowd). So you see the earliest usable indications in the market prices. That is why I’m mostly a “rapid trend follower”. The major problem with market predictions based on what I’d call “expectational analysis” is that you can be completely right about the expectation and very wrong about the market price movement. Then you are either “in early” (also known as “being wrong”…) or “out late” (also known as “getting killed”…). I’ve been “completely right – just early” so many times and lost so much money doing it that I finally learned to just put those ideas on the wall and not put a penny into them until the PRICE ACTION shows the world has ‘caught up’. Basically, you don’t want to spend a year or two waiting for the market to figure out you are right.

    “It’s not about you, and it’s not about being right. It’s all about them. – E.M.Smith”

    So I’m up to my eyeballs in oil trusts and some LNG tankers paying 10%+ dividends (and rising…). And I’d buy a bunch of coal and coal hauling railroads, but Warren Buffet is already doing that and I hold a chunk of Birkshire Hathaway (so I would be doing too much in that area to “double dip” and lever up on top of what Buffet is already doing).

    Basically, the best “theme” along the lines of “contrary to AGW” is simply to bet on China. I’m largely following the dictate to “Own what China wants to buy”, and right now that is energy and resources. Oh, and some gold. China is, by it’s actions, clearly in the “AGW is bunk (but maybe I can get the fools to pay me for nothing)” camp. So just bet on them staying greedy and selfish. In Singapore that ought to be an ‘easy play’ ;-)

    I’ve asked several times “HOW are the satellites calibrated? Against what?” and not seen the answer. I’ve seen enough circumspect references to ‘adjustments’ to the data after the bird is on orbit to strongly suspect they calibrate against a ground temperature series; but it needs investigation. Maybe next year ;-)

  5. Al S. says:

    About satellite temperature sensor calibration: there’s supposed to be an onboard device to check the sensors against , and Jeff ID (The Air Vent blog) has done a post on adjustments to readings, based on time of day, etc.
    I believe Dr. Roy Spencer has written online articles also.

  6. JP Miller says:

    I surely hope you and Anthony collaborate (with some willing respected, tenured faculty) on an article that is submitted to and published by one of the more “respected” peer reviewed climatology journals. I was a bit disappointed to see Anthony self-publishing some of his findings. I may be wrong, but I believe that your kinds of findings (and Anthony’s) will not have the impact they need to have unless the broader scientific community sees your work going public in this manner. The press, the public, and ultimately the politicians will not pay attention until a lot of climate scientists start saying “our data has problems.” I know that the Hansen’s, Jones’s, Mann’s, Briffa’s will never recant on the basis of you demonstrating the data to be faulty, but there are hundreds of other scientists who have more scruples and will be willing to deal with the facts as they are — even if those facts undermine 10+ years of work. But, they will only do this if your facts get the imprimatur of the leading peer reviewed journals. Guerilla science has its place, but it will never outgun the “estabishment” if it is the only strategy trying to bring sanity to climatology. Wish you well however you proced. Keep it up!

  7. E.M.Smith says:

    @Al S:

    I’m not so much worried about calibrating the sensor against a standard to correct for sensor drift; as I am wondering “OK, I’ve got {foo} nano-watts of microwaves reflected from {something} below me that matches exactly {bar} reflection from the sensor calibration standard: What is that in “Degrees on the ground?”

    There are 2 aspects to this calibration thing. 1) How do I assure my sensor is not drifting and apply an appropriate adjustment if it loses 1% of output per unit of reflected microwaves? and 2) How do I map any given computed ‘microwave percent reflected AND detected’ into a ‘Degrees On The Ground’. If #2 is based on ground references, then drift in the ground references can be reflected into a satellite error.

    My onboard calibration continues to say “1 nano watt emmitted, 0.8 nano watt detected so multiply by 1.25 for sensor loss”.

    My ground temp map says “We measured against SFO and Death Valley and set the first marker at 50 F and the second marker at 100 F based on the ground reference data. But SFO was really 48 F in the air, due to UHI, while Death Valley was really 88 F at the measured altitude (it being hellishly hot right at the ground, while at zero Mean Sea Level – a few hundred feet above ground, much cooler).

    Now we have a 40 F slope being interpreted as a 50 F slope.

    We get a 5/4 reported increase from a 4/4 actual increase.

    We look down at, oh, Dallas, and now we find that it’s 100F even though it really is closer to 90F, and we just did a bad job of matching reflected microwaves from altitude to measured temperatures on the ground.

    I know, it’s all hypothetical, but that’s why I’d like to know how does one turn a microwave return energy signal into a calibrated temperature on the ground? I’m quite happy to presume that a decent E.E. can make a target and emitter to assure that any changes in the microwave SENSOR are detected and measured correctly. I’m much less convinced that x nano-watts from y altitude have been correctly mapped to real ground temperatures and even less convinced that they have the “Delta Temp / Delta Energy” worked out right… (That is: starting point, and slope)

    And do they change this mapping over time if Dallas land is reporting a consistent 2 F higher than when the original mapping was done a decade ago?

    Frankly, the whole idea that you can use the same technique, microwave reflection, to measure any and all of: Clouds, tornadoes, wind speeds, ice vs water vs cloud, rain, and on and on… without many opportunities for error, and that they have all been handled with 100% accuracy and 0.001 C precision … that idea just needs more proof than I’ve been able to dig up.

    Do they have it “good enough for nightly weather news”? Sure. Good enough for Climate Czars? Um, I’d like a bit more information on that, please…

  8. Level_Head says:

    On the change at GISS, congratulations!
    You have diligently tracked relations
    Of the thermos that were
    Versus those they prefer
    And the Southerly March of the Stations

    I hope you’re as fit as the felines
    And your wallet has gained some new re-lines
    From your work with the bourse
    So, best wishes, of course
    I look forward to SmithTemp-to-be lines

    ===|==============/ Level Head

    REPLY: [ Thanks! Both I and the cats are doing well, though I’m still recovering some. “I’ts a process”…

    And yes, I’ve done OK will “calling the market”. Not a “killing” but keeping kids in school, food on the table, etc. -ems ]

Comments are closed.