UPDATE: Since comments have become a repetative “Does So, Does Not – this has been covered already above, read it.” Then folks don’t read it. I’ve turned comments off. Perhaps for a while, perhaps for a long while, perhaps forever. Frankly, I have much more valuable and interesting things to do than tell people they are repeating already repeated too many times points. I don’t see where it removes anything from the issue, both sides are already well represented below.
On WhatsUpWithThat, one of the commenters asked, roughly:
Regarding the GISS output over time, is there anyone here who has any confidence in the second decimal place in either reported temperatures or reported temperature anomalies?
There can be no confidence in anything smaller than whole degrees F.
For some reason, it seems that many folks stop here and jump to the “law of large numbers” shiny thing. I’m going to add a note here so that those who feel frustrated by not getting to do that will, with luck, understand why. This is well covered in comments, but folks don’t seem to be bothering to find it there.
One common error is to presume that GIStemp takes thousands of temperature data points and averages them together. Then folks launch into a grand description of how this average can very well have greater precision than whole degrees F. One Small Problem: That is not what is done and that is not my complaint. “The central limit theorem” is just not applicable. GIStemp takes in monthly averages of daily averages of the daily MIN and MAX. When you are averaging temperatures, you are averaging exactly 2 of them. The daily MIN and MAX for one place for one day. Everything after THAT is just averaging averages (and sometimes averaging averages of averages of averages…). So the central limit theorm says you can get much more precision about that AVERAGE but not about the temperatures that made it up.
The next common error is to jump to “but it’s anomalies so it doesn’t matter”. It isn’t. The anomaly map is produced at the last stages of GIStemp. These averages of averages of averages are smeared around in all sorts of ways before the anomaly map step is done. They are used to “in fill” missing data. They are used to adjust during “homgenizing” data. And they are used in the Urban Heat Island Effect adjustments. They clearly can spread their influence up to 1200 km in some steps, and 1000 km in others. It is even possible for a 1000 km “in-fill” to be used for a further 1000 km “UHI” and then to be used for a further 1200 km step. While I think this is unlikely, it is possible.
So long before any “large numbers” or “anomalies”, the averages of daily MIN/MAX averages have been spread all over.
The Raw Data
The historic data were measured to 1/10 th degree F then they were rounded to whole degrees F for reporting. Each day had three samples (min / max / TimeOfObservation I believe). If a datum was missing it was fine to “guess” and fill in what you thought it ought to have been on the form.
“Temperature is measured electronically. High or low temperatures for the day may be estimated when necessary. Temperatures are measured in degrees and tenths fahrenheit, and reported as whole degrees, rounding down from .4 and up from .5 “
Averaging is not Over Sampling
Now you can “oversample” a single thing and get a synthetic accuracy that exceeds your actual accuracy; that requires measuring the same thing repeatedly. We don’t measure each day/location repeatedly. We measure it once. Then NOAA averages those data points (min / max) to get a daily average. Exactly two items are averaged.
Then they take those daily averages for the month and average them to get a monthly average mean of the daily means. At most 31 items are averaged.
This is what is used by GIStemp. At this point we are already 2 “averaging steps” away from “temperatures”. While we can gather interesting meta statistics about those averages of averages, such as a highly precise “mean of the mean of means”, that does not tell us things about the mean of the temperatures. The order of averaging things does matter.
In my opinion, NOAA does a better job than GIStemp, but they do manage to start the ball rolling on the false precision front by creating this data with 1/100 F precision when the raw data only have 1 F accuracy. (And remembering that they average at most 2 temperature items then average at most 31 of those averages. There is no ‘large number of items’ from which to have a “central limit theorem” improvement in precision.)
I have also found a great simple example and explanation of how the math ought to be done at: http://mathforum.org/library/drmath/view/58335.html
GIStemp – Where data manipulation begins
UPDATE: As of the end of 2009, GIStemp is using USHCN.v2 for US data and that is now in 1/10 F (one presumes NOAA / NCDC realized some of the folly in the 1/100 F place.) I do not yet know if they use the 1/10 F or 1/100 F as they calculate the GHCN data. They DID use a new “adjustment” on the USHCN.v2 vs the USHCN original method that causes the 1/10 F place to jump all over in comparison, so we now have about 1/2 F of “jitter” from “adjustments” that completely swamps the “precision” of the averaging process.
GIStemp then takes those 1/100 precision whole F accuracy temperatures (or 1/10 F in the new version with 1/2 F variations from the prior versions) and starts mathematically manipulating them (adding, dividing) to make many averages of averages. Typically in fairly small groups. These are what becomes the basic input to further GISS processes. NOAA provides a table of monthly averages of daily means in F with false precision into the 1/100 F. GISS then converts these to C. The relevant bit of code is in USHCN2v2.f
10 read(2,'(a)',end=100) line read(line(1:6),*) idus0 [...] read(line(indd:indd+5),'(f6.2)') temp if(temp.gt.-99.00) itemp(m)=nint( 50.*(temp-32.)/9 ) ! F->.1C end do write(nout,'(a11,i1,i4,12i5)') idg,iz,iyear,itemp
There is a bit of sloppiness here in that “9” is an integer and “32.” is a floating point number as is “50.” then they do a cast with nint into an integer type. I’m not sure what this mixing of data types will do to the precision in the low end bits (probably nothing) but a FORTRAN expert ought to pass judgement.
(In a later posting I found a compiler dependent bug in the way GIStemp does this step that “warms” 1/10 of the data by 1/10 C. Yes, 1/10 C is dependent on what computer compiler you run. It’s an easy fix, documented in the “F to C Conversion Issues” posting.)
A cleaner approach would have been to leave everything in F and avoid false precision, but they probably decided C was more trendy… They do have to at some point face the fact that the USHCN set is in F and the rest of the world is in C; so doing it this way might make sense IFF you watch the false precision properly (which they don’t do).
Some folks want to think this can be treated like an over sample of the month, but it can’t. There is no monthly average of daily min/max average temperatures to repeatedly sample. There are only individual days with their own precision and accuracy being sampled. The monthly average of MIN/MAX averages is only a computed artifact, not a real thing to over sample. (IF you were to take all the daily temperatures and average them in one go, then you could begin to apply the central limit theorem; but then you would have to confront the Nyquist limit…)
This is where the “fun” begins.
The C number already has some false precision in it; but you can almost forgive it since they have the choice of giving a bit of false precision (but preserving all the information that was in the original full degree F number) or giving a full degree C precision (and having no false precision, but losing the difference in range that a degree F vs C has). IMHO it would be easier and better to put everything in F and have avoided the issue, then convert to C “at the end”. But folks for some reason hold the politics of being International and using SI units as more important that having an easier time tracking the actual accuracy and precision. (It is easier to go to the unit of measure with the “finer grain”; but given the 1/2 F “jitter” from choices about “adjustment method” it is at most a nit, and not a very relevant one.)
And these folks at Colorado.edu in discussing Geographical Information Systems (that they abbreviate GIS) do suggest that the custom in some cases is to carry forward 1 digit of false precision so that the user can decide what to do with it.
So I can see that as a reasonable choice. That, in the end, GIStemp comes up with 1/100 C anomaly maps leads me to believe they do not do the right thing with it. And for those folks just chomping at the bit to say “but with thousands of temperatures we can get that precision in the anomaly” let me just point out that many anomaly boxes have all of ONE location reporting. Many more are in the 2 to a dozen range. You just don’t get to apply the “central limit theorem” to things counted in ones and twos…
Too often GIS projects fall prey to the problem of False Precision , that is reporting results to a level of accuracy and precision unsupported by the intrinsic quality of the underlying data. Just because a system can store numeric solutions down to four, six, or eight decimal places, does not mean that all of these are significant. Common practice allows users to round down one decimal place below the level of measurement. Below one decimal place the remaining digits are meaningless.
So they recognize that the false precision digits are meaningless, but have common practice bringing forward one of them.
Now GISS takes the input numbers and starts doing strange and wondrous things with them. see:
if you want to start touring the actual computer code and process.
Mr. McGuire Would Not Approve!
My high school Chemistry & Physics Teacher was one Mr. McGuire. He was a retired Lt. Colonel from the U.S. Air Force (WWII with combat) and a retired chemist from U.S. Steel.
He ran a very precise, very accurate, and very disciplined class. One did not cut corners nor stretch the truth one single decimal point in his class. Any problem in any homework, lab, test, you name it (even if otherwise perfectly done) that had a digit of precision beyond its accuracy would lose points. Sometimes all of them. By his rules, GIStemp would get an F (and that’s not for Fahrenheit).
The GIStemp process includes a great deal of math and averaging.
Now what I learned in school (“Never let your precision exceed your accuracy!” – Mr. McGuire) was that any time you did a calculation, the result of that calculation could only have the accuracy (and thus ought to only be presented with the precision of) your least accurate term. Average 10 12.111111111 and 8.0004 and you can only say “10”, not 10.0 and certainly not 10.1111 or 10.04 as that is false precision.
(In fact, it’s slightly worse than this, due to accumulation of errors in long strings of calculations and the repeated conversion that GISS does from decimal in intermediate files to binary at execution and back to decimal in the next file… but that’s a rather abstruse topic and most people glaze so I’ll skip it here. But just keep in mind that repeated type changing corrupts the purity of the low order bits.)
So what gets trumpeted and ballyhooed?
Things (Not temperatures! Calculated anomalies based on averages of interpolations of averages of averages of averates of two temperatures – no, that is not an exaggeration! In fact I’m leaving out a few averaging and interpolating and extrapolating steps! ) measured as X.yz C! Not only is the “z” a complete fabrication, but any residual value in “y” from the greater precision of F over C in the raw data has long ago been lost in the extended calculations and type conversions. (And, as seen in the changes from USHCN to USHCN Version 2; swamped by various adjustments and manipulations.)
IF you are lucky the X has some accuracy to it.
(Though GISS even manages to corrupt this via “the reference station method” that lets them rewrite the whole thing based on other temperatures or anomalies up to 1200 km away… In the “Slice of Pisa” thread we see that the past of Pisa is made 1.4C COLDER in a broken UHI adjustment that goes the wrong way.)
Under the Italy thread on WUWT there was a blink chart of Pisa that showed about 1.4C “adjustment” to one of the data points. So if we are creating 1.4 of “fantasy” how much truth is left in the 0.01 C place?
GISS data are thoroughly cooked and, IMHO, only useful for fairy tails and monster stories…
To the inevitable assertion that it’s only the US data so the global number is still fine, see:
The fact is that the only long term records we have are dominated severely by the U.S.A. and Europe. GISS makes up much of the rest by various sorts of in-filling of average boxes and in-filling over time.