NASA has responded to a 2 year old FOIA request
These were released as a result of the FOIA request put in by Chris Horner as described on WUWT:
It comes in four parts from this URL as pdf docs:
http://www.nasa.gov/centers/goddard/business/foia/GISS.html
Part one is:
Part two:
Part Three:
Part Four:
http://www.nasa.gov/centers/goddard/pdf/417760main_part4.pdf
A scanned in OCR’d a searchable set is at:
http://www.neutralpedia.com/wiki/NASA_FOIA_Emails
An Example
Subject: Re: Your Reply to: GISS Temperature Correction Problem?
From: Gavin Schmidt gschmidt@giss.nasa.gov
Date: 19 Feb 2008 14:38:47 -0500
To: rruedy@giss.nasa.gov
I had a look at the data, and this whole business seems to be related to the infilling of seasonal and annual means. There is no evidence for any step change in any of the individual months.
The only anomalous point (which matches nearby deltas) is for Set 2005. Given the large amount of missing data in lampasas this gets propagated to the annual (D-N) mean – I think – with a little more weight then in the nearby stations. The other factor might be that lampasas is overall cooling, if we use climatology to infill in recent years, that might give a warm bias. But I’m not sure on how the filling-in happens.
Gavin
So as I read this, the folks at NASA responsible for GIStemp are saying that large data dropouts (i.e. Zombie Stations for a while or Dropouts for longer periods) “gets propagated” to the means (and thus the map products) and that if “we use climatology” (i.e. the way GIS uses the relationships between areas ‘climatology’ as it calculates ‘offsets’ – that’s the jargon for their process) that might “give a warm bias”.
Gee.
Maybe I don’t need to convince them that missing data can lead to climatology based infill giving a warming bias. Maybe I only need to get them to ADMIT it publicly… Oh, wait, this FOIA email looks like it does that… Though I’m sure we will get quibbling about it being only one swallow and not a whole spring…
And this one admires the way that you can make up yearly data with just a collection of months data… and that it might have issues. But he’s “pretty sure” it is “just a fluke”… unless, of course, you have a constantly shrinking number of thermometers with ever more gaps in the data to be “made up” from ever less real data…
But including one month of dropped data would “only” drop the temperature by 0.4 C for the annual mean of the cell in question…
Subject: Re: [Fwd: Re: Your Reply to: GISS Temperature Correction Problem?]
From: Gavin Schmidt
Date: 20 Feb 2008 15:01:26 -0500
To: rruedy
That works.
That implies that the missing months are assumed to have the same mean anomaly as the other two months, and that the missing seasons are assumed to have the same mean anomaly as the seasons present. Hence, one strong anomaly in a couple of months (ie. Sept and Nov 2005) can have a large impact on the annual mean.
I’m pretty sure that the Lampasas spike is just a fluke of the annual average construction. There are only eight months – of which only 7 are used to calculate the annual mean. The missing month (May) has the smallest anomaly, and so including it would bring down the annual mean by about 0.4 deg C.
There may be some improvements that could be made here. i.e. annual means could use as many months as there are available (rather than just whether the seasons are available), and it should be made clearer that this is a Dec-Nov mean, not the calendar year mean, Somewhere it should also be stated that the seas/ann values in the printout and figures are not used in the gridded data.
Thanks
Gavin
Musings from the Chiefio 
“we use climatology” I wonder if that is like scientology. also someone called “Jay” is the wizard the massages the code and data to create the needed output.
I wonder how it is that all the databases have all the same problems with FOI requests for data,”we lost it” the code is secret or the code released is not the one used and the data needs to be adjusted, “climatogized” to get the “real” output?
We know there can not be collusion. ;?q
E.M. – Apologies if you were aware of this, but this email series seems to be directly related to this WUWT post regarding the Lampasas, Texas USHCN on 2-14-2008.
Anthony’s post was five days before these emails and commented on a large temperature spike in 2005 that seemed to correlate with a station move. The site is terrible, right next to a building and a highway surrounded by asphalt. Anthony also commented on how the GISS algorithm failed to pick up and correct the discontinuity.
For all of their bluster, looks like these guys really were following the surface stations project closely and were concerned about what was being uncovered (although their concern was probably more on how to spin the story than fix the problem).
REPLY: [ No apologies needed. I was (and am) completely unaware of the “history” of this stuff prior to about 2? years ago max. I was pretty much a ‘newby’ about then to WUWT and much of my memory of my ‘early times’ at WUWT is a bit murky as I was still figuring out who was who and what was happening. I didn’t know who a “Gavin” or “Jones” were and was only barely aware of Hansen. And didn’t know what GIStemp was either (ah, such bliss ;-)
The point remains that some years ago, they were seeing that “the anomaly will save us” does not. So it interests me a little bit “why” they were figuring it out, it interests me a whole lot more that they admit that the “climatology” method lets temperatures change with station and data dropouts. -E.M. Smith ]
Moving from “lurker” to a leader of the pack in 2 years is awesome. congrats! and thanks! to you.
First thing that struck me was that they’ve labeled the page as “Temperature data”. No clue as to the true nature of the release.
It’s also NOT accessible from the FOI page, but rather under the “Reading room.”
So they’re hiding it “in plain sight.”
I worked asa contractor at Goddard for 42 years, retired in 2006 – partially because of the bureaucracy. The bureaucracy has doubled since I retired.
Pingback: The rats are starting to jump ship! « TWAWKI
Can’t help myself. I must second P.G.S.’s statement:
On February 19, 2010 at 9:31 am P.G. Sharrow:
“Moving from “lurker” to a leader of the pack in 2 years is awesome. congrats! and thanks! to you.”
We — all citizens of the developed world — are most fortunate that you have been willing to turn your considerable mind and expertise to examine the “bowels” of the problem. As part of my metaphor, you have been able to send your camera into the miles and miles of intestines (computer code) hidden in the depths of this conspiracy. You’ve enabled us to see the ways in which we were slowly being poisoned (good, historical data exchanged for lies) or starved to death (dropping of thermometers at their evil whims).
I think it will take a sustained fight (e.g., Refounding/Re-establishing America) if we are to regain our health. The most fundamental medicine: truth, transparency, accountability. You certainly have established an excellent model for this on “Chiefio”.
Mundane communication. I was just ready to send my renewal to “Eating Well”, a food magazine I have enjoyed for over 10 years. I tore it up after I read their column on “Green Eating”. Summary: a MacDonald’s Big Mac is too expensive (to continue to exist) — raising cattle and getting them market creates too many greenhouse gasses; corn is the most highly subsidized crop and cattle are fed corn for fattening; fast-food companies pay poverty-line wages requiring government support for workers; a Big Mack is not healthy.
Anyone who thinks this jumble mess of confused thoughts — bull-product of Uranus? — will soon be out of business. I did my part.
Best wishes for the publishing aspect of your career — we need more “truths” in the public sphere from those can think clearly.
E.M.Smith: I’ll also second PGS’s comment. You have made a great contribution in just 2 years. I watched you on John Coleman’s video-you were great. It was very informative to hear your description of the temperature collection/synthesis/calculation process. K.I.S.S-Keep It Simple Stupid was always a good rule to work by. Simplify the process as much as possible before you try to write the progam. I am sure as a programmer you can appreciate that. In the case of the temperature records, the most important thing is to start with good data. The “Harry Read Me” file indicated that there might have been some issues with the data. The first thing I would do would be to get accurate GPS locations of the global stations and find them on Google Earth. Secondly, get someone at the station to take some pictures and measurements to heat sources such as was done with the surfacestations.org project. How difficult can that be? Then take the raw data and for each grid block, select stations that properly represent the grid. Instead of using a program to make bad data good, just select good stations. If there is missing data in the good stations, they could be calculated relative to the other good stations. All of the raw data from all of the stations should be readily avaiable. The grid block data should be correlated with satellite data to create a robust process. Satellite data could also be used to fill in missing grid blocks. I would like to see something simple enough so that I can manually go in, pick a grid block and check all of the calculations.
fluke of the annual average construction.
There may be some improvements that could be made here. i.e. annual means could use as many months as there are available (rather than just whether the seasons are available), and it should be made clearer that this is a Dec-Nov mean, not the calendar year mean
Does this imply that the annual means for a location are constructed by combining the seasonal means, if available? What a back-arse way of doing it. And if so, and you are missing two months of data in seperate seasons, each contributing .4C in some cases, then that location could show a spurious warming of .8C based on two months of missing data. And why isn’t the data infilled in the case of Lampasas?
This is bizarre. Your analogy of GISS as a blended food product seems to be degrading to a mechanically pulverized sump pump discharge. I couldn’t take it as you do, looking at this inept collection of lazy science. It would drive me to……….?
Though I’m sure we will get quibbling about it being only one swallow and not a whole spring…
Looks pretty damning, but then so does “hide the decline” and that hasn’t rattled the consensus AGW dogma to the degree you might expect.
OTT
I just was looking at Yellowstone (kind of a weekend catch-up) and found this link to an animation:
http://www.quake.utah.edu/recenteqs/Anim/anim_imw.html
Wouldn’t it be neat if the USGS site webmaster could figure out how to run this routine( I can’t find it if the site does). You could follow the progression of quakes around the rim. Oh yeh, another guvmunt agency.
Just saw you on Coleman’s “Meltdown” at http://www.kusi.com, E.M., congratulations, i think you delivered your point very well again!
Tim as to your question about how GISS gets their official annual mean, the answer is yes. They first take the monthly means and make them into 4 seasonal means. As an example we will use the year 2009. They take Dec 2008, Jan 2009 and Feb 2009 and get the DJF mean. From there work the months out through the year. From those 4 seasonal averages of monthly averages of dailey averages they get another average the Meteological Annual Mean.
Whats bad about this is if you have missing months and how you get a seasonal average. When CLimategate first broke a noticed something strange in the data GISS uses for figuring out the State College PA annual mean. You see there you two different methods to calculate the seasonal mean if there is a missing month, al depending on which month it is. Examples are if you look at the this data:
1973 -1.5 -3.2 6.4 9.1 2.9 21.1 22.5 22.4 17.5 12.5 999.9
-0.2
1974 -0.3 -2.7 2.9 9.7 14.2 18.4 21.4 21.2 15.5 9.1 999.9 0.3
1975 -0.9 -1.3 1.1 5.9 16.7 20.0 22.5 21.8 14.5 999.9 8.6
-0.6
So there we have 3 years of data from State Collage. In 1973 and 74 the month of November is missing data in 1975 the month of October is missing. So now lets look at what the Sept, Oct, Nov (SON) seasonal average is:
1973 – 11.9
1974 – 9.2
1975 – 11.6
Now lets cut the yearly data down to just the months that make up that season:
1973 – 17.5 12.5 999.9
1974 – 15.5 9.1 999.9
1975 – 14.5 999.9 8.6
Just by looking at the values you should be able to spot one problem right away, but lets state it here. First look at 1975 the seasonal average is 11.6 and if you take a calculator and average the values for Sep and Nov of 75 you get: 11.55 or rounded to 11.6 [(14.6 + 8.6)/2]. So as you can see when the middle month is missing they average the two end months together. However look at the seasonal average for 73 and 74 and compare it to the 2 months that we do have data for. In 1973 they got an “average” that is below the value of both the Sept average and the Oct average. When you average those two number together you can not get 11.9 [(17.5 + 12.5)/2]. Same thing with 1974 even though the seasonal average is at least above the Oct average (.1 above, but above).
I have checked this on other seasonal averages that had missing data and not just on this one record but on several. When the missing month is the middle month they average the two end months, when it’s one of the end months they use some other method to “make up” that seasonal average. From those four averages they make the annual.
For confirmnation on how GISS makes their annual average hit this link:
http://data.giss.nasa.gov/gistemp/station_data/seas_ann_means.html
But here is an abstract:
“Climate is what we expect, Weather is what we get”. -Mark Twain. :)
Chiefio: Off topic: I watched the latest Coleman videos and would like to discuss your comments about GISTEMP, SST, University of East Anglia, Optimal Interpolation, etc. We can do it here, or if you’d prefer, you can leave me your email address in a comment at my blog. I won’t publish the comment (or you email address).
Regards
REPLY: [ You will find an email address encoded such that spam scammers can’t pick it up in the “about” tab. It is Pub the numeral 4, then “all” all as one word, followed by the at sign and aol dot com. -E.M. Smith ]
on February 20, 2010 at 12:43 pm boballab
Thanks
hi chiefio
O/T
i take you are aware Lucia at
http://rankexploits.com/musings/2010/guest-post-invitiation-to-chiefio/
would like a chat with tea & cookies:)
sorry if this is old news, keep up the good work, you are getting there.
REPLY: [ Yes, and dealing with such requests and stuff has now consumed all my time for today. I’m really more interested in getting my work queue done than in discussing with other folks where they didn’t get it… so the request to discuss how detractors have got it wrong and indulge in a food fight (even with tea and cookies) is not high on my ToDo List. -E. M. Smith ]