Over in the WUWT ‘critique’ of my look at differences between GHCN v1 and v3, one of the more “vociferous” complaints was that I was using a ‘discredited’ or somehow deprecated method in using First Differences.
So this posting is really just a page in my notebook where I’m collecting pointers and information about First Differences. Not very exciting, but useful.
OK, don’t know if I can find the whole First Differences article text anywhere, but here is a link to the abstract:
Their “citations” page lists:
Chen, Fahu, Jinsong Wang, Liya Jin, Qiang Zhang, Jing Li, and Jianhui Chen (2009), Rapid warming in mid-latitude central Asia for the past 100 years, Front Earth Sci China, 3(1), 42.[CrossRef]
Christy, John R., William B. Norris, Kelly Redmond, and Kevin P. Gallo (2006), Methodology and Results of Calculating Central California Surface Temperature Trends: Evidence of Human-Induced Climate Change?, J Clim, 19(4), 548.[CrossRef]
DeGaetano, Arthur T., and Robert J. Allen (2002), Trends in Twentieth-Century Temperature Extremes across the United States, J Clim, 15(22), 3188.[CrossRef]
Free, Melissa, James K. Angell, Imke Durre, John Lanzante, Thomas C. Peterson, and Dian J. Seidel (2004), Using First Differences to Reduce Inhomogeneity in Radiosonde Temperature Datasets, J Clim, 17(21), 4171.[CrossRef]
Free, Melissa, Dian J. Seidel, James K. Angell, John Lanzante, Imke Durre, and Thomas C. Peterson (2005), Radiosonde Atmospheric Temperature Products for Assessing Climate (RATPAC): A new data set of large-area anomaly time series, J Geophys Res, 110, D22101.[CrossRef]
Hu, Qi (2005), How have soil temperatures been affected by the surface temperature and precipitation in the Eurasian continent?, Geophys Res Lett, 32, L14711.[CrossRef]
Jin, Menglin, and Robert E Dickinson (2010), Land surface skin temperature climatology: benefitting from the strengths of satellite observations, Environ Res Lett, 5(4), 044004.[CrossRef]
Jones, P. D., M. New, D. E. Parker, S. Martin, and I. G. Rigor (1999), Surface air temperature and its changes over the past 150 years, Rev Geophys, 37(2), 173.[CrossRef]
Jones, P. D., and A. Moberg (2003), Hemispheric and Large-Scale Surface Air Temperature Variations: An Extensive Revision and an Update to 2001, J Clim, 16(2), 206.[CrossRef]
Jones, P. D. (2004), Climate over past millennia, Rev Geophys, 42, RG2002.[CrossRef]
Jones, P. D., D. H. Lister, and Q. Li (2008), Urbanization effects in large-scale temperature records, with an emphasis on China, J Geophys Res, 113, D16122.[CrossRef]
Jones, P. D., D. H. Lister, T. J. Osborn, C. Harpham, M. Salmon, and C. P. Morice (2012), Hemispheric and large-scale land-surface air temperature variations: An extensive revision and an update to 2010, Journal of Geophysical Research—Atmospheres, 117, D05127.[CrossRef]
Karl, Thomas R., Richard W. Knight, and Bruce Baker (2000), The record breaking global temperatures of 1997 and 1998: Evidence for an increase in the rate of global warming?, Geophys Res Lett, 27(5), 719.[CrossRef]
Lawrimore, Jay H., Matthew J. Menne, Byron E. Gleason, Claude N. Williams, David B. Wuertz, Russell S. Vose, and Jared Rennie (2011), An overview of the Global Historical Climatology Network monthly mean temperature data set, version 3, Journal of Geophysical Research—Atmospheres, 116, D19121.[CrossRef]
Li, Qingxiang, Wei Li, Peng Si, Gao Xiaorong, Wenjie Dong, Phil Jones, Jiayou Huang, and Lijuan Cao (2010), Assessment of surface air warming in northeast China, with emphasis on the impacts of urbanization, Theor Appl Climatol, 99(3-4), 469.[CrossRef]
Li, QingXiang, WenJie Dong, Wei Li, XiaoRong Gao, P. Jones, J. Kennedy, and D. Parker (2010), Assessment of the uncertainties in temperature change in China during the last century, Chinese Sci Bull, 55(19), 1974.[CrossRef]
Menne, Matthew J., and Claude N. Williams (2005), Detection of Undocumented Changepoints Using Multiple Test Statistics and Composite Reference Series, J Clim, 18(20), 4271.[CrossRef]
Menne, Matthew J., and Claude N. Williams (2009), Homogenization of Temperature Series via Pairwise Comparisons, J Clim, 22(7), 1700.[CrossRef]
Montandon, Laure M., Souleymane Fall, Roger A. Pielke, and Dev Niyogi (2011), Distribution of Landscape Types in the Global Historical Climatology Network, Earth Interact, 15(6), 1.[CrossRef]
Peterson, Thomas C., Kevin P. Gallo, Jay Lawrimore, Timothy W. Owen, Alex Huang, and David A. McKittrick (1999), Global rural temperature trends, Geophys Res Lett, 26(3), 329.[CrossRef]
Selvam, A. M. (2011), Signatures of universal characteristics of fractal fluctuations in global mean monthly temperature anomalies, Jrl Syst Sci & Complex, 24(1), 14.[CrossRef]
Shen, S. S. P., H. Yin, and T. M. Smith (2007), An Estimate of the Sampling Error Variance of the Gridded GHCN Monthly Surface Air Temperature Data, J Clim, 20(10), 2321.[CrossRef]
Shreve, Cheney (2010), Working towards a community-wide understanding of satellite skin temperature observations, Environ Res Lett, 5(4), 041002.[CrossRef]
Smith, Thomas M. (2005), New surface temperature analyses for climate monitoring, Geophys Res Lett, 32, L14712.[CrossRef]
Thorne, Peter W., John R. Lanzante, Thomas C. Peterson, Dian J. Seidel, and Keith P. Shine (2011), Tropospheric temperature trends: history of an ongoing controversy, WIREs Clim Chang, 2(1), 66.[CrossRef]
Trewin, Blair (2010), Exposure, instrumentation, and observing practice effects on land temperature measurements, WIREs Clim Chang, 1(4), 490.[CrossRef]
Vose, Russell S. (2005), An intercomparison of trends in surface air temperature analyses at the global, hemispheric, and grid-box scale, Geophys Res Lett, 32, L18718.[CrossRef]
Vuille, Mathias, and Raymond S. Bradley (2000), Mean annual temperature trends and their vertical structure in the tropical Andes, Geophys Res Lett, 27(23), 3885.[CrossRef]
Wu, Zhuoting, Hongjun Zhang, Crystal M. Krause, and Neil S. Cobb (2010), Climate change and human activities: a case study in Xinjiang, China, Clim Change, 99(3-4), 457.[CrossRef]
Zaiki, Masumi (2002), A statistical estimate of daily mean temperature derived from a limited number of daily observations, Geophys Res Lett, 29, 1892.[CrossRef]
Which raises the rather amusing question of “Have all those papers been withdrawn for using First Differences? Hmmmmm?”. Perhaps those authors will wish to ‘have a conversation’ with Steven Moser about his claims and decide how best to withdraw THEIR works. Right after that, I’ll consider it… /sarcoff>;
Near as I can tell, FD is still used, the papers not withdrawn, and it is just that the limitations of the method and any quirks it might have ought to be kept in mind when you use it (rather like all methods of doing things…)
One of the complaints was that I used a method that “bridged the gap” on dropouts. I just hold onto the last value of a valid data time, then whenever the next valid data item arrives, use that to compute the “difference”. This gives a slope to that segment that is essentially an interpolation of that space. ( One could interpolate each value in between, compute all those small linear segments discreetly, and then compute the overall slope and you end up at the same point.) But there was some significant vitriol applied toward the idea that this was just horrible and very unacceptable… and insinuation I must have some moral defect with regards to claiming First Differences was Peer Reviewed, then doing this ‘other thing’ that isn’t… (While I’ve regularly said First Differences is a peer reviewed method – which it is- and that I use a variation on it which I have not claimed is peer reviewed – but which I think is trivially demonstrated to be more accurate, not less, and certainly not very much different on the bulk of the GHCN data.)
Of interest here might be the opinion at Climate Audit:
Changes that are likely to cause a level shift in the series, such as a TOBS or equipment change or a station move, should simply be treated as the closing of the old station and the creation of a new one, thereby eliminating the need for the arcane TOBS adjustment program or a one-size-fits-all MMTS adjustment.
Missing observations may simply be interpolated for the purposes of computing first differences (thereby splitting the 2 or more year observed difference into 2 or more equal interpolated differences). When these differences are averaged into the composite differences and then cumulated, the valuable information the station has for the long-run change in temperature will be preserved. (When computing a standard error for the index itself, however, it should be remembered in counting stations that the station in question is missing for the period in question).
The Common Anomalies Method (CAM), used by most of the major indices, including MOSHtemp ;-) , requires restricting the data base to stations that are observed during a common base period, or at least during most of it. This requires throwing out a large portion of the data, and/or relying heavily on estimated TOBS and MMTS adjustments to artificially extend stations with these changes. In Table 1, for example, there is no period longer than 1 year for which more than 1 station has complete data, so that no trend could be computed by this method at all. If we were to settle for a base period in which each station has at least 2/3 of its data, a 3-year period such as Yr2-Yr4 could be used, as illustrated below:
So it looks like “bridging the gap” has passed muster at Climate Audit. (Though via interpolation, not just “hold and do anomaly over the gap”.) As I do FD only on a single month for a single record, one would expect that THAT month in THAT place is the same basic entity this year as last year, or 2 years ago, or even 5 years ago. Any change in a very long gap, like 10 years, is more likely the result of a valid action (such as ENSO or PDO changes) than a “random artifact”. (The only exception being, as noted, equipment changes or process changes, where a ‘reset’ on FD ought to take care of it.) The flip side of that point is that if you are looking for those splice artifacts not taking a reset will highlight them (which is what I’ve said was my goal all along). So using my dT/dt method on unadjusted vs adjusted (that has supposedly done TOBS et.al. to do ‘proper’ adjustments) ought to let us see how much that ‘splice’ was ‘repaired’. One of my eventual goals is to compare the two in just this way. (In earlier use of dT/dt I’ve looked at variously unadjusted and adjusted sets. Yet Another Work In Progress -WIP)
Part of the reason I restrict the comparison in doing dT/dt anomaly creation (or dP/dt that differs only in the sign; difference present to past vs past to present) to a SINGLE instrument CountryCode/WMO/modifier string is to prevent interaction between different stations or changed WMO numbers. An instrument is compared ONLY to itself and ONLY inside a single month series. So any change large enough to cause a WMO or substation number change will be in a different FD series / entry.
As noted above, the CAMS method depends on all the adjustments to remove TOBS and station change issues. So if you are looking at comparing unadjusted GHCN over two versions of released data and want to know how much “bias” might be in the data (so how much must be removed by things such as TOBS and could influence CAM based systems) taking a reset on gaps within a station will hide exactly what you are trying to find. The bias in the change of data. Not looking for what you wish to examine makes it rather hard to see…
So using a method that “spans the gap” and using First Differences on a large body of data will give closer match to the actual trend in the longer term for those stations. Just what I wanted to show (and just what I have asserted ought to happen). Though, yes, a stringent comparison of interpolation vs ‘lump sum’ needs to be done. ( I put all the change in a single ‘lump sum’ in the year where the next valid data is observed – keeping the change synchronized with the actual arrival of data; interpolation ASSUMES it can be linearly spread over the gap. As I don’t know that, I’d rather just preserve what the data actually said: ‘This change showed up here.’) It is, in some ways, part of that difference in mind set between “looking at the shape of the data” (as I’ve called it) and the quest for The One True Global Average Temperature. I want to know what THE DATA looks like and what it has to say, not what I can turn it into that tells me what I think it says about something else. I want to know if the data has large lumps in it, not hide them (even if the method of the ‘hide’ is approved.)
Is First Differences ideal? IMHO nothing is ideal. So in looking for “the best method” one has to ask “best for what?”. Again, at Climate Audit, they find some “issues” with FD and like a different method for the ‘best’ least biased calculation of the actual warming trend in a temperature series. Realize that this doesn’t say much about comparing two data set versions where, frankly, using any method ought to be about the same validity. Any systematic error in a method will tend to give the same offset in both results so the difference between them will tend to neutralize. (Apparently the Warmers love anomalies for offset suppression when used for temperatures but suddenly don’t like them when doing bulk set compares…) It is the comparison of a method applied to ONE data set version compared to reality (that Holy Grail…) that has some comparative “better” or “worse” in the match. For the simple reason that you can not make an anomaly between those two to remove the bias in the method; as that anomaly is unknowable (as the “reality” is what you are trying to find in the first place.)
Update 8/29 Just for the record, as noted below at http://climateaudit.org/2010/08/19/the-first-difference-method/#comment-240064, Jeff Id has convinced me that while FDM solves one problem, it just creates other problems, and hence is not the way to go.
But “way to go” for what? Well, for that comparison of A data set version to the unknown “reality”…
And is the suggested ‘better’ method (“Plan B”) free of all issues?
As stated, it gives equal weights to all stations. But estimating it by GLS with an appropriate covariance matrix would be straightforward.
One small drawback is that adding new data can change earlier estimates in the combined series because the latest values will add new information on station differences. However, these differences will generally be relatively small.
True — but in live time this just means that you have to settle on a set of stations (with at least 10 years or so of readings), compute offsets, and then go with that formula for several years. 5 or 10 years later, you come out with Version 2 of your index, with new stations added and slightly modified offsets for the old stations.
And so on.
Now that doesn’t sound so good if your goal is the absolute minimal changes in the data used and wish to use EVERY data item in the set that it is possible to use. “Measuring the data” is NOT the same as “making up a number that I think matches reality the best” especially when it comes to things like “must have 10 years of data” or it doesn’t get used as part of the comparison base set.
That, BTW, is one of my complaints about the various CAM based methods. Overweight is given to some stations that are held to have some special merit due to a particular length of coverage in a particular span of time. That presents opportunities for those data to have “special effect” and for changes in a given instrument during those years to have greater impact on the comparison.
My goal in making a comparison between v1 and v3 is not to show how a particular 10 year period varies nor to give any decadal scale span overweight. It is to compare the two data sets, minimally biased, shifted, or weighted, directly to each other. Giving overweight to given decades just violates that goal.
That’s what I’ve got for this now; for today. I’ll add some to it from time to time as things of interest show up.
Right now I’ve got to find P.G. who is waiting for me at a local hotel lobby. We have a very important project* that just Can Not Wait. So further R&D on this topic is on hold for the rest of the day… and likely 1/2 of tomorrow…