KUSI – Coleman TV show discussion

Just an open discussion thread

I’ve had a request or two for a place to discuss the KUSI “Global Warming, The Other Side” shows. OK, here it is.

Please keep the discussion centered on the shows or issues they discuss.

Also, please note that I’ve not yet seen the second show. ( I know, I ought to, but I’ve just been a bit too busy. It takes more time to be IN a show than to watch it ;-) But I’ll get to it in a day or so. I do remember all of my interview, but don’t know how much of it made it to the air. Yet. (Then, a couple of my days this week were taken up by medical stuff. No, nothing bad, just a bunch of ‘delayed 10 years’ required inspections and maintenance visits :-).

My Part

My major point has simply been that much of the available data is not used. It is dropped on the floor. You can call it “deleted” or “dropped” or “ignored” or whatever. It is still NOT in the GHCN data set. The pattern of these “droppings” is that high latitude and high altitude stations are dropped, while low altitude and low latitude stations are kept (with an ever increasing percentage at warm heat islands of airports). There is a clear ‘survivor bias’ toward warmer stations (and with warming trends, like airports), with warmer winters, more heat islands, and lots of tarmac.

(I’ve preferred to call them “deletions from the record” but some folks have complained about that implying intent and active process. I’m not fond of the implications of “droppings” [ it reminds me of something chickens do… that farm background ] but I’m willing to use that as it is more passive in intent. In either case, a “high cold” thermometer that is in the baseline period is left out in the present. Thousands of them, in fact. That is horrific ‘survivor bias’.)

The major “defense” of the “droppings” has been from two directions.

First, that there is no person actively pruning thermometers. While the “spin” put on my position has tended to say there is active intentional removal of thermometers for malicious effect; I have gone out of my way to point out that I can not know any person’s intent, only the result. I’ve also said that it is possible, though increasingly remote as a possibility, that the thermometer drops are a passive action from accidental ignorance. I do hold to Hanlon’s Razor. (Roughly paraphrased as “Never attribute to malice that which is adequately explained by stupidity”.) But “adequately explained” is becoming ever harder to accept… But in either case, I’m more interested in the FACT of the thermometer deletions (or drops) from the record and what that says about data bias; than about whether there has been a sin of omission or of commission. It’s a sin in either case. Was it murder or involuntary thermometer slaughter? In either case “It’s dead Jim”, and it’s wrong.

My favorite poster child for this is Bolivia (though there are several options) where no data are in GHCN for the last 20 years, yet CLIMAT reports are available at least since 2007 ( I’ve seen them online from that period). So something other than CLIMAT reports being “missing” must be the cause for the droppings. Whatever it is, other folks have no problem finding the temperature in Bolivia (or the omitted parts of Russia, Canada, etc.)

Second defense, that the “anomaly” process will prevent thermometer drops from having an impact. ( This is usually followed by a theoretical example of comparing a thermometer only to itself and showing that with perfect anomaly processing and an idealized unbroken record, there is no problem.) But the reality is that we don’t compare thermometers only to themselves and the records are horridly broken and with massive “fill in” with fantasy “data”. So we have “fantasy basket A” to “fantasy basket B” that change over time.

Well, thermometer change / drops / deletions DO have an impact. I’ve run a benchmark through the GIStemp code and using exactly the stations GISS dropped (from the USHCN data from 5/2007 to 11/2009 -when they put them back in, after some postings pointing out how to do it and that it was an ‘issue’… – perhaps just a coincidence…) and the anomaly map shows warming from those station being dropped. We can argue about the price of this streetwalker, but what it does is not in dispute.

The reason it fails to stop all survivor bias impact is two fold. One fixable, one less so.

First, it does not do the anomaly comparison “self to self” [ I call that “selfing” after the pollination process ] but rather “Basket A in time A” to “Basket B in time B”. And once the two things being used to create an “anomaly” are different from each other, you have opened the door for a variety of very subtile biases to change the result. This, too, is not in dispute. (Well, it is by some who are a bit slow to catch on, but it is not in dispute by the folks who wrote the code. One of the NASA FOIA emails admit to this problem and bias.)

The second reason is rather subtile, and it is one I’m “in discussions” about publishing; so I’ll not make it public until some decision is reached on that front. ( I may tire of the whole backbiting “peer reviewed publishing” process and just go for “public reviewed self published”. That is my leaning, but folks keep telling me it’s important to be “in the literature”… We’ll see.) Lets just say that it depends on some assumptions everyone makes that are wrong, and looking at what is ignored. It will impart survivor bias into the First Differences Method, and the Reference Station Method. I believe it will impart bias into the Climatology Anomaly Method as well, but the definition of that method might allow for an approach that would dampen the bias (i.e. “perfect” selfing and lifetime), so I have a bit more homework to do before asserting it as fact for all variations. Basically, for any system where the thermometers change over time, it allows for bias to show in the product. And that’s as far as I can go on that point right now. It is this property that, IMHO, lets the benchmark change for GIStemp.

The bottom line is that survivor bias from thermometer change matters, and there has been a heck fo a lot of biased thermometer change.

Other Folks

I’ve not seen other folks stuff yet (for part 2). I can guess that Anthony Watts covered the biases in station locations and poor maintenance. What surfacestations.org has turned up ought to call the whole temperature history quality into question. Thermometers over asphalt and under AC waste heat vents. On roofs covered with black tar. Near waste water treatment plants (always warm). And increasingly at airports (and even if low traffic locations, the tarmac and concrete runways stay year round). Heck, the lack of tree cover, and the snow removal, will cause several degrees of local heating at some times of the year.

I would further guess that the issues of “adjustment” and “homogenizing” were covered. Not only do we have sparse crappy data with too poor a temporal or geographic coverage, but then we fudge it out of all recognition and make up large chunks where we have gaps. Hardly inspiring trust and confidence.

What else? For now, other folks will have to provide those talking points. Once I’ve seen the show I can add something here.

Updates

Well, I’ve now seen parts 5 and 8 via the links here:

http://wattsupwiththat.com/2010/02/20/must-see-john-colemans-global-warming-special-2-now-online-at-youtube/

So far I like what I’m seeing ;-)

It is interesting to note that this time it looks like Coleman had so much to work with that individual interviews got cut down some to fit the time available. (You spend a half hour on camera in the studio and a couple of minutes make the show.) So the Bolivia reference was not in, but the Russia and Canada were. Probably the better choice. Lots of regular folks don’t really seem to care about Bolivia.

I also very much liked the segment with Dr. John Christy at UAH on the disconnect developing between the satellites and GIStemp. Part of “the game” in GIStemp looks to me like a rewrite of the (unverifiable by other means) old past of land data; then to anchor the land data in the present so it stays in sync with the verifiable period of time (since 1980 or so). But time passes. And that “anchored” segment is starting to be a millstone around the neck. Now there is either a drift downward of the early part of the verifiable segment (as it get re-written) OR you get a continually rising curve that does not match the satellite product.

It looks to me very much like a decision that has reached it’s ‘end of life’. It was workable for the 1980-1990 period of time, became a bit dodgy in the 1990’s but was saved by the 1998 warm spike, and has now had the wheels come off in the 2000’s as the satellite data drop away from the 1998 peak and GIStemp tries to flatten it (GIStemp tends to ‘iron out’ the past) and get “lift” recently (so the increasing divergence to the satellites will show). It will be fun to watch this verifiable period of time gnaw away at the GIStemp product over time. Especially now that the PDO flip has us lined up for 30 years of cooling. For the period from 1978 until now we have largely been using satellites to measure the warming phase of the PDO cycle. Now we have a cooling phase.

Yeah, it will take 20 years to get a reasonably complete satellite sample, but at least then we’ll have something decent to work with (not just a 30 year up slope of 1/2 the cycle). And yeah, there will be a couple of years of ‘lack of cooling’ as this anemic solar cycle sputters out a few sun spots. But we’re looking at about a 7 year cycle this time, so 3 years of ‘support’ at most. Then we’re back headed ‘off the cliff’ ( if prior Grand Minimum periods spot cycles are a guide).

I also enjoyed the fact that the “warming causes cooling” mantra is getting trounced. The ‘story’ that the snow this year is an ‘extreme event’ is not going to fly with anyone who is over 50 years old and remembers the 1970’s or 1960’s. (or even worse, someone who remembers history and either the Little Ice Age or even just the picture of Washington, crossing a frozen Delaware river, and the bitter snows his army faced.) Frankly, watching them take a bit of ridicule for a ridiculous position is a joy to behold. It will be far harder to re-write all that past history to put in warmth (so this cold period can be classed as ‘extreme’) than it was to re-write the recent past of the temperature data to put in extra cold to show us warming in comparison.

Basically, the “Warming Alarmists” have made a schizoid choice. To both warm the past, so the present is “an extreme cold event” and cool the past, so that our trend is warming today. “This dog won’t hunt.” The past is well attested outside of the temperature records, even by loads of folks who were alive then. The snows are present in photographs and paintings, and even works of literature. It’s now just too much history to re-write. That cold past “worked” with the warming slope mantra, and it is lethal to the “extreme cold event” mantra.

This cold weather period with snow all over the planet is NOT an extreme event. It is simply a return to the 1950’s weather patterns (and no where near the 1740-1840 period cold events – though add in one big volcano and we could be back at an 1816-1817 Year Without A Summer “Eighteen Hundred And Froze To Death” scenario). And trying to paint “Cold is Hot” and “Ordinary is Extreme” and even “Weather is cold but Climate is hot” is just going to cause an “Emperors Clothes” moment. Gotta love it.

More notes as I get time to watch the other segments.

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in AGW and Weather News Events and tagged , . Bookmark the permalink.

22 Responses to KUSI – Coleman TV show discussion

  1. Hoi Polloi says:

    Spencer mentions in his latest message (http://www.drroyspencer.com/2010/02/new-work-on-the-recent-warming-of-northern-hemispheric-land-areas/)

    “I’ll have to admit I was a little astounded at the agreement between Jones’ and my analyses, especially since I chose a rather ad-hoc method of data screening that was not optimized in any way. Note that the linear temperature trends are essentially identical; the correlation between the monthly anomalies is 0.91.

    and

    “But at face value, this plot seems to indicate that the rapid decrease in the number of stations included in the GHCN database in recent years has not caused a spurious warming trend in the Jones dataset — at least not since 1986.

    What do you think about that?

    REPLY: [ Nothing. I’ve not looked at it. See my reply to Keri Lantto. Though the description of it is sounding rather like Yet Another Hypothetical Cow story. -E.M.Smith ]

  2. Brian Bishop says:

    E.M. (aka Chiefio),

    I am trying to reconcile virtually counterposed substantive analysis of the effect of station dropping on the global temperature record. I cannot find you e-mail address to send a query I also dropped a note to Joe D’Aleo who paraphrased your posting on the Day the Temperature Record died in a white paper on climategate. ( http://icecap.us/images/uploads/NOAAroleinclimategate.pdf ). He may have been incorrect to say that you found they had systematically eliminated 75% of weather stations. Rather it seems that you offered statistical refutation of Basil Copeland’s comment/hypothesis that there was no systematic bias. That is a slightly different proposition.

    The countervaling analysis ( http://www.yaleclimatemediaforum.org/2010/01/kusi-noaa-nasa/ ) was prepared by Zeke Hausfather at Yale but linked by a purported skeptic (not necessarily a climate skeptic but one claiming skeptical credentials and posting on Skepticblog by the name of Steven Novella (who is a neurologist at Yale).

    In any event, I can’t figure out if any of the links on the page are actually an analog to Hausfather’s graph comparing temperatures of dropped and not dropped stations in his 1491 station sample of the NCAR. So if you could point me to an analogous link, I’d appreciate it.

    The gravamen of Hausfather’s complaint is that his analysis (using NCAR and not GHCN stations) into the dropping of stations suggests it has actually created a cool bias in the record, not a warm bias. And that those alleging a warm bias are anecdotally citing dropped stations in high latitude or altitudes and rural settings but have not globally examined the temperature record to demonstrate a statistically significant bias in the temperature resulting from station drops.

    Secondarily, he contends that the falloff in stations is a data collection lag, not an elimination arguing specifically:

    “It is quite likely that, a decade or two from now, the number of stations available for the 1990s and 2000s will exceed the 6,000-station peak reached in the 1970s.”

    Honestly, on a quick runthrough of your legion efforts, I can’t really find a link to a similar analysis or graph created in your work to that in Hausfeather. So I am unclear whether the difference in your outcomes might be related to the differing datasets you are examining.

    Of course the NCAR is a subset of the GHCN. And Hausfeather uses a subset of the NCAR, which may further divide your outlooks. Of the 3563 NCAR stations he only looks at 1491 or which 402 dropped. His criterion are stations that had a continuous record from 1960 to 1970, so this may be what results in a drop in station count in his method.

    I’m not sure why continuity between 1960 to 1970 would be the relevant calculus. This was the time of the greatest run-up in station numbers in the GHCN. I assume NCAR’s graph might look similar but I have not seen it.

    If you could do me (and the public) the service of having a look at Hausfeather’s contention and highlighting why his substantive results seem at odds with yours, I would greatly appreciate it.

    Best Regards,

    Brian Bishop

    REPLY: [ At some point I might get to it, but right now I’ve got a large backlog of things I’ve said I would do that I’ve not yet done. And, as I’ve said before, finding out where other folks have gotten things wrong does not interest me as much as finding out what is the truth. To the extent they overlap, yeah, I’ll toss rocks at wrong results, but my major interest is “what is actually happening”.

    Per an email address: Read the WORDS in the ‘about’ tab up top. It is buried in there such that spam bots can’t fish it out.

    In order to make things more approachable for the average person ( Lord knows this stuff is dense enough to make a geek’s head swim as it is) I have tried to work in picturesque descriptive phrases that are “easy to get”. Things like “The Great Dying of Thermometers” and “Taken out back and shot”. I can only hope that folks do not believe thermometers can actually die or that they were literally shot. So how folks choose to interpret such speech is not up to me. All I can really do is point at the available data and truths and say “This Happened”. Some folks highlight one potential causality, others leap to malice of intent. I’m neutral. I just don’t know “why”. So while mostly that leads to a “thermometer dropped due to accident of collection process” as a possible explanation, it also leaves you with “issues”.

    Bolivia. CLIMAT reports for at least 4 years (2007-2010) and not in GHCN. It is not for lack of electronic reporting. It is not for lack of access to the data (link is online in another posting). Panama also cuts off in 1980, for no good reason. (A promised posting that I’ve not yet done…)

    Canada. Dropped to 45 stations in GHCN last I counted. Canada says they are reporting more in real time today. It’s not for lack of reporting nor for these being “historical compilations”.

    Russia. Has stated that the pattern of their stations that are dropped vs kept puts a bias into the data.

    GUAM. All the METARS are there, and the NOAA weather site can’t make a CLIMAT report out of it for themselves? From their data at their site? It isn’t an uncooperative foreign sources issue.

    The list goes on and on.

    Now I can’t say for certain that it isn’t just an astounding lack of competence. It could easily be a profound laziness coupled with a belief that “any old stations will do, so Don’t Worry, Be Happy” and a degree of stupidity that would make a grown man cry. But frankly, that is no better than a malicious intent. I don’t see any way that arguing for one of these is an improvement over the other. So I don’t. While it would be fun to know which is was, it does not change the facts. The facts are that the temperature history is broken from too sparse a coverage and it need not be in the period from 1990 to now. And saying it might be fixed ‘in a decade or two’ when folks want to commit economic suicide NOW based on the data NOW is small comfort.

    I have done no ‘dropped vs kept’ analysis per se (since I don’t know where to get the temperatures for the dropped thermometers and I’m chasing other issues). I have done a “what do the good stations look like” simple average of the temperatures. And it shows no warming. Look in the GIStemp tab up top for the “Quartiles of Age” postings. I want to re-run those using dT/dt methods, but for now, that’s what is there. The shortest lived records carry the warming signal, the long lived thermometers do not. “Global Warming” is very non-global and only shows up in selected thermometers with the worst track record. More of them are in the data now. It is a data bias problem. QED.

    For now, I can only give you a GUESS. I would surmise that by looking at subsets you get Yet Another Biased Sample. Not good. Further, when you pick a period of time that is shorter than one whole PDO cycle, you have another type of selection bias. And finally, I could explain why “High Cold Places” have this issue, but that gets back into the area I’m in discussions about on publication, so I can’t go there. Yet.
    -E.M.Smith ]

  3. Kari Lantto says:

    Did you see Roy Spence’s (20 February) calculation of surface temperature without dropping stations. He finds the dropping hardly affects the trend at all.
    He seems to be commenting your work, without saying that though. But to me it seems he should have waited until he had a timeseries long enough. He is just comparing the trends 1986-. So he hardly has anything before the big death, right?

    REPLY: [ I’ve barely got enough time to keep up with my own work ;-) never mind following what everyone else does (and often gets wrong) so no, I’ve not looked at what Spencer has done. FWIW, taking your assertion of a 1986 start date as valid, I see an immediate issue that he is only sampling a warming portion of a PDO cycle. A “time bias” error. The effect of selection bias is dependent on the longer term cycles. Having a station “in” during the cold PDO and “out” during the following period. He’s missed that, at a minimum. -E.M.Smith ]

  4. Hoi Polloi says:

    “Yet Another Hypothetical Cow story.” That’s a bit easy, Chief. Spencer is a well respected, sceptic, climate scientist. I’m sure this will be used by the alarmists, so it might be wise to have a further look at it.

  5. boballab says:

    Brian in another thread there is a link to a post on my site that shows that the claim by Zeke or anyone else that it’s a reporting problem is BS, complete and total BS.

    Now This Is Interesting: A Different, Larger Dataset at NCDC?

    In that post I found a document on the NCDC site about a different data set compiled by the US Air Force. That document is from 2003, well after the great die off of thermomters from GHCN and in it NCDC states there is 10,000 active reporting stations in the dataset the Air Force is compiling using their own AWN system and the WMO’s Global Telecommunictaions System (GTS)

    National Climatic Data Center

    DATA DOCUMENTATION FOR

    DATA SET 9950 (DSI-9950)

    DATSAV2 SURFACE

    January 6, 2003

    1. Abstract: DATSAV2 is the official climatological database for surface observations. The database is composed of worldwide surface weather observations from about 10,000 currently active stations, collected and stored from sources such as the US Air Force’s Automated Weather Network (AWN) and the WMO’s Global Telecommunications System (GTS). Most collected observations are decoded at the Air Force Weather Agency (AFWA) formerly known as the Air Force Global Weather Central (AFGWC) at Offutt AFB, Nebraska, and then sent electronically to the USAF Combat Climatology Center (AFCCC), collocated with NCDC in the Federal Climate Complex in Asheville, NC. AFCCC builds the final database through decode, validation, and quality control software. All data are stored in a single ASCII format. The database is used in climatological applications by numerous DoD and civilian customers.

    AFCCC sorts the observations into station-date-time order, validates each station number against the Air Weather Service Master Station Catalog (AWSMSC), runs several quality control programs, and then merges and sorts the data further into monthly and yearly station-ordered files. AFCCC then provides the data to the collocated National Climatic Data Center (NCDC).

    Click to access td9950.pdf

    Now there is roughly 1500 active stations in GHCN, NCDC is given the DATSAV2 dataset by the Air Force with over 10,000 active stations. There is not 8500 US Air Force bases in the world nor is there even 8500 total US military bases in the world. So most of the data is comming from the GTS which is what the GHCN is suppose to be made up from as per the NCDC mandate as one of the three world climatological archives. So now you tell me how the US Air Force is able to get that many CLIMAT reports, QC them and give them to the NCDC but NCDC doesn’t include them into GHCN. Now try and blame that on not being “real time”reporting.

    Also according to NASA the DATSAV2 dataset is up to 13,000 active stations:

    Contains worldwide surface observations (synoptic, airways, METAR, synoptic ship) for about 13,000 stations. All weather elements transmitted are retained; in some cases, computed/derived values are incorporated into the record. Also available are “station files-individual station data sets for selected stations–that have … received more quality control. Elements reported are: Wind direction, Snowfall and snow depth data, Wind speed, Runway data, Barometric pressures, Hail data, Pressure tendency & change, Sunshine data, Dry bulb temperature, ground temperature and conditions, Dew point temperature, Maximum and minimum temperatures, Total sky cover, Ship data, Visibility, Sea surface temperature, Past and present weather, Wave data, Cloud layer data, Swell data, Ceiling, Ship ice reports, Precipitation data.

    The DATSAV2 Surface Data is also available from NOAA/NESDIS/NCDC (National Climatic Data Center) in Asheville, NC.

    http://gcmd.nasa.gov/records/GCMD_USAFETAC_SFFMG.html
    The thing about the DATSAV2 dataset is for us to get a copy you better be prepared to pony up a lot of money.

    So the question is:
    How can the US Air Force find 13,000 active stations world wide, update their dataset every 3 months provide this dataset to the NCDC and yet there is only 1500 active stations in GHCN?

  6. I enjoy viewing your blog, your writing style is a and I enjoy reading your blog posts

  7. marchesarosa says:

    Dear Chiefio

    This highly informative and persuasive graphic presented by lucy skywalker and based on data compiled by John Daly “Circling the Arctic” badly needs updating where possible to 2009 so it can go into neutralpedia.

    http://www.greenworldtrust.org.uk/Science/Scientific/Arctic.htm

    There are several people on this board who have the forensic data handling skills to identify the source of these data and to update them IF the databases are still available in their original form.

    May I exhort someone to volunteer for this this job and provide the source material on-line. It would be a great asset to neutralpedia and a valuable resource for sceptics.

    Someone please step forward for the job!

  8. Chiefio — This is just a note of admiration and encouragement from a non-scientist computer-illiterate human person, junior grade.

    There is a huge chance that the work you’re doing will prevent my unborn great-grandchildren from toiling, birth to death, in a worker’s paradise, that was created to avoid a fake disaster that was never going to happen in the first place.

    Your work is critically important. I never had a talent or true aptitude for science and math. I just want to thank you sincerely for what your work means, and how it could prevent future human tragedies, all planned in the name of “compassion” and “respect for the planet.”

    REPLY: [ Thanks for the compliment. All I really do is ask simple questions and only accept an answer if it makes sense. If it can’t be explained to me, it goes into the “smoke and mirrors” pile. That “Joe Sixpack” behaviour is all you really need to be a decent “scientist”. Observe. Question. Don’t be gullible. Unfortunately, many folks called ‘scientists’ today are too enamored of computer games and don’t spend enough time on the “question” and “don’t be gullible” parts… -E.M.Smith ]

  9. 3x2 says:

    EM & boballab,

    Never mind the cost of the data set!

    Have I missed something here?

    * Ten times the number of stations found in v2_mean actually arrive at NCDC in reasonably real time and have been doing so forever?

    * These are currently active stations and presumably v2_mean is a selected and truncated (time wise) sub-set?

    * Most had stopped being reported in v2_mean by the early ’90s? (many for no apparent reason.)

    (There is then a good chance that a station such as Edinburgh (and others) with continuous record back to the mid 1700’s didn’t really just die in the early 90’s for no apparent reason)

    * GISS (and presumably the CRU) are also well aware of the much larger and up to date set but still use v2_mean as a base instead? (all independently of course because they are, after all independent products)

    I think the next stage should be a station list for DATSAV2!

  10. 3x2 says:

    Colour me really confused here.

    We have the UKMO today proposing a grand scheme for an international effort on climate data.

    … Meanwhile back in the mid 90’s (95 or 98 as far as I can see) we have this document (FOREIGN WEATHER DATA SERVICING AT NCDC) ..

    The NCDC’s digital data base currently contains 180
    terabytes of data (not including back-up copies). New data
    streams will add an average of 70 terabytes per year for the remainder of the 1990s.

    The USAFETAC/OL-A archives more than 40 million global surface observations per year in its DATSAV2 data base.

    (…) It also archives nearly 2.5 million original manuscript forms each year. These records are indexed, placed on microfiche, and added to the more than 200 million pages already archived.

    The USAFETAC/OL-A archives more than 40 million global surface observations per year in its DATSAV2 data base.

    It also archives nearly 2.5 million original manuscript forms each year.

    Due to its superior quality and quantity compared to other worldwide surface data sources, the NCDC uses DATSAV2 as its primary source for foreign digital surface weather data.

    This data base contains hourly and/or synoptic data for about 20,000 worldwide stations, with nearly 10,000 of these stations currently active.

    More than 40 million observations are added to the data base each year.

    Are you weeping yet? (and that was back in the mid 90’s)

  11. boballab says:

    3×2

    You got it right. NASA knows of this dataset, they even have a page in their dataset sub directory on one of the NASA sites devoted to it.

    Now I’m not claiming that GHCN is taken FROM DatSav2 but that NCDC has available to it the same information the Air Force does.

    For data from foreign National Weather Services they send out monthly CLIMAT reports over the WMO’s GTS which the USAF and NCDC both get. From that and the readings from the USAF’s own AWN system the USAF makes DatSav2.

    From those same CLIMAT reports that the USAF uses the NCDC is only using a part of them for GHCN. What makes it worse is that the USAF then turns around and every 3 months gives them the updated DatSav2.

    So it is not once but twice NCDC spurns the data that came over the GTS and doesn’t put it into GHCN.

    Also you can see this in the GHCN’s own adjusted data file. There is Canadian stations in the raw file that GISS uses but NCDC drops out of their analysis and adjusted V2 mean_adj file.

  12. 3x2 says:

    This looks like the list of data sets. 463.2 (U.S.A.F. DATSAV3 Surface Observations, 1901-continuing)
    seems to be the big one.

    The earliest data is for 1901. The earliest years have the least volume of data, a few hundred megabytes; the volume in the 1950s-early 1960s increased by a factor of 10-20; 1965-1972 shows a decrease because of a loss of data; beginning in 1973 with about 5 gigabytes of data, volume has gradually increased to about 20 gigabytes per year. There are now about 10,000 active stations.

    Seems like anyone can register for access but there is a slight snag – it is 335GB worth. Hmm .. 2000-present file much smaller at 55G

    I think that this is the station list for the various DS463.X data sets. Out of interest I had a look for UK stations, found them .. all 500+. (!)

  13. 3x2 says:

    boballab

    Also you can see this in the GHCN’s own adjusted data file. There is Canadian stations in the raw file that GISS uses but NCDC drops out of their analysis and adjusted V2 mean_adj file.

    The station list I’m looking at has 1100 Canadian stations listed

  14. boballab says:

    The Station List for GHCN has every station that has data in their raw file, including stations that no longer exist. I have been working on a post for my site on this but here is a Preview. Go get a copy of GHCN V2 mean and V2 mean_adj.

    Ok now once you do that look up this WMO station ID: 71082 Alert Nwt Canada

    In the V2 mean file you see multiple thermometers that have been used for that station and the data goes up to 1991

    In the V2 mean_Adj file you only see one of the thermometers listed and the data goes up to 1991.

    Now head over to the GISS site here:
    http://data.giss.nasa.gov/cgi-bin/gistemp/findstation.py?datatype=gistemp&data_set=0&name=40371082

    Notice there this is the “raw” input to GISS and that there is 6 thermometers listed and the data goes to 1991.

    Now look at the GISS adjusted data here:
    http://data.giss.nasa.gov/cgi-bin/gistemp/findstation.py?datatype=gistemp&data_set=2&name=40371082

    Again back to one station and data up to 1990.

    Now go to the Canadian government site here:
    http://www.climate.weatheroffice.gc.ca/climateData/hourlydata_e.html?timeframe=1&Prov=XX&StationID=1731&Year=2010&Month=2&Day=24

    Monthly data between 1950 and 2006

    So there you see that there is data from the that station past 1991 not used in both GISS and GHCN.

    I have been working my way through every station listed on this map shown here:
    http://www.climate.weatheroffice.gc.ca/climateData/menu_e.html?timeframe=1&Prov=NU&StationID=99999&Year=2010&Month=2&Day=24

    Eureka station #71917
    Canada has data from 1947 to present
    GHCN raw: multiple thermometers listed for that station, data covers from 1947 to 2009
    GHCN adj: multiple thermometers listed for that station, data from 1947 to 1991 where it was cut off
    GISS adj: combined the thermometers, data from 1947 to present.

    So just because a station is in the Station list doesn’t mean the station is used today nor that GHCN uses all the data from that station, it only means that there is data that has been used from that station at one time by GHCN. So ig a station only existed from 1950 to 1990 and GHCN used that data it would still show up in their station list.

  15. boballab says:

    As to DatSav3 that was the Global hourly dataset and is no longer in use it was combined into a different dataset and that being an hourly dataset is also why that one file is so big (that is alot of data), but you did notice that the USAF does seem to do a better job of tracking down stations.

    Here is a list of all NCDC datasets:

    http://www.ncdc.noaa.gov/oa/documentlibrary/?choice=complete&searchstring=&submitted=1&submit_form=Search

    DatSav2 is dataset 9950
    Datsav3 is dataset 9956

    Also Datsav2 is only free to certain users ie: people with a .edu or .mil in their IP

    GENERAL STATEMENT
    Due to various Federal Laws and Regulations, NNDC is required to charge for some of its online data to recover the cost of data dissemination. This includes hardware and personnel costs incurred by each Data Center. Charges are required for most domains (e.g., .com, .org, .net). All online data are now free for all .gov, .edu, .k12, .mil, .us, and a few other specific domains. Please see NNDC’s Free Data Distribution Statement (PDF Format) for further information on our FREE data policy. For information on how free access is granted via our web systems, please visit the Free Access section of the NCDC help page at http://www.ncdc.noaa.gov/oa/about/ncdchelp.html#FREE

    Questions/Comments can be directed to: nndc.webmaster@noaa.gov

    NOTICE TO ACADEMIC USERS
    We provide free data access for .edu domains. We have also expanded the free access to include .k12.xx.us users. Please continue to provide information on your data usage via the NNDC Feedback Page. Thank you for your interest and usage of our system.

    NOTICE TO MILITARY USERS
    Due to funding provided through the DoD, all online data are now free to “.mil” customers. Our military customers may choose to contact Air Force Combat Climatology Center or Fleet Numerical Meteorology and Oceanography Detachment Asheville (FNMOD) for further information regarding support to military customers.

    and if you go the page to access the data you will notice that the ftp site where you can normally download the entire dataset, like with GHCN, is restricted.

  16. boballab says:

    EM I think you might want to take a look what I discovered when you play around the GISS Map Maker program when it set to display trends:

    Interesting GISTemp Trend Maps

  17. E.M.Smith says:

    @boballab:

    Nicely done. Very nicely done.

    I ought to warn you though, once you start turning up those kinds of “broken behaviours” in the GISS product it can be addicting ;-)

    I especially liked the way that you used “All GISS tools” to show that the argument that “The Anomaly Will Fix It” is broken. Thermometer change does show through in the trend maps.

    Wonder how long it will take them to put up a screen saying “You don’t need trend maps like that” ;-)

  18. Ruhroh says:

    Rightio, be sure to snag copies of all of your interesting discoveries.

    I (manually) ran a big series of ‘comparison-to-self’ anomalies one night, and by the next afternoon they had turned off the ability to make that run.

    I’ll email you a PDF of the results.

    Very nice work by the way.
    I want to PYLON the kind words of el jefe…

    RR

  19. boballab says:

    EM you should take a look at Dr. Spencers latest post here:
    http://www.drroyspencer.com/2010/02/spurious-warming-in-the-jones-u-s-temperatures-since-1973/

    He has taken his analysis now back to 1973 and he states clearly that all of the data he uses from the ISH dataset (which is what DATSAV3 turned into and over time has a growing number of active stations) have no station dropouts between 73 and 2009. He is also taking the average of 4 readings per day not just min/max. When he did this and gridded it he found that the Jones CRUTEM3 dataset has a 20% divergence in from his.

    REPLY:[ Nice. Very nice. His first cut ended too soon (as I noted when folks were all lathered up that he had not yet found any issue). As he extends further back in time he will find greater divergence. What he is finding is the same “instrument change” impact I’ve found. We got there by different means (I’m going out of my way to use methods other than the usual “peer reviewed” ones so as to avoid any systematic error in them). But he is clearly on the right track with using a more complete data set. He also “has clue” about the need to avoid those data that are highly “adjusted” as the “unscramble the eggs” problem becomes very large. I suspect that the hidden bias causing the divergence he found is the effect I’ve identified (that I still can’t talk about due to publication potential… dang it.)

    And the direction he is pointing toward is the right one. A “do over” with full and complete data with “stability of the instrumentation” as a major goal. -E. M. Smith ]

  20. boballab says:

    EM I just posted up a comparison of any and all changes to the trends in the overlapping grid boxes when you change from 1200km infill to 250 km infill. What you find is that almost every land grid is changed, meaning that just about no grid trends are based on obsereved data only, even over averaged observed data.

    REPLY: [ Maybe I could talk you into including a link in these notes? ;-) -E.M.Smith ]

  21. boballab says:

    Well the link here is a direct link to this one:

    GISTemp Trend Map Data Analysis

Comments are closed.