Classic FD vs dT or dP
In the GHCN v1 vs v3 comparison I used a minor variation on First Differences that I think is slightly superior to the Classic Peer Reviewed version.
I do not take a gratuitous ‘reset’ on missing data items.
This caused some folks to get their panties in a bunch, so I’ve redone the process using Classic First Differences for the “All Data” case and made a graph of the difference between the two (Classic First Differences and my dT or dP variation on it). That graph is at the top of this page. I use the period of time that covers the common climate codes, such as GIStemp (1880) and Hadley (1850) and running to 1990 when v1 ends ( I align v3 with v1 on that date).
Frankly, I expected more difference. There is nearly no difference for the latest years. About 1960 we open up a 0.05 C gap and then it closes again about 1919. Another separation happens of about 0.03 to 0.05 C then closes again. At the far right side at the “start of time” for programs like HadCRUT and GIStemp, the gap widens a little more to about 0.12 C. Which is to be expected. As the data series is a running total of difference, any differences also ought to accumulate. Also, as the purpose of the dT variation was to preserve trend through dropouts, the more dropouts happen in the data, the more the two series will diverge. The farther back in time you go, the more there are dropouts in the data.
While I would assert that my method is more accurate as it does preserve all valid data and does not take a ‘gratuitous reset’ just because one monthly value is missing; the effect is clearly limited.
The Difference In Method
In Classic First Differences, a missing data item will cause a reset of the running total of ‘difference’.
In the dT and dP process, I ‘bridge the gap’ via hanging on to the last valid temperature and just waiting for the next new valid data item.
The thesis being that January in New York is still January in New York even if it has a 2 year gap between Januaries rather than one, so there really isn’t a lot of difference between comparing January 2012 to 2011, or to 2010 (or even to 2009). If you find a difference, it is still a valid difference.
So, for the hypothetical series of temperatures:
10.4, missing, 10.1, 9.8
Classical First Differences would find 0 for the first anomaly (as the first year matches itself) then reset on ‘missing’ recording a 0, then record a zero on 10.1 (as it is again a ‘first value’) and finally compare 10.1 to 9.8 and record a -0.3 change. By inspection, we can see that the temperature dropped from 10.4 to 9.8 for a difference of -0.6 or 50% more.
0,0,0,-0.3
The dT method would also record a 0 for the first value, and a zero for the missing value, but on that 10.1, it would compare it to 10.4 and find -0.3 of difference. Then compare 10.1 to 9.8 and find another -0.3 of difference. A total of -0.6.
0,0,-0.3,-0.3
While I would continue to assert that, for series like this where I only compare one month of one thermometer to itself, this is a more accurate method, the total impact on the result is rather low when used on the entire body of the GHCN data.
I still hold out hope that it can be demonstrated superior, especially in more sparse sets of data; but that will have to wait for another day. It’s now after 3 AM and time to wrap up for the day.
Musings from the Chiefio 
EM – for what it’s worth I also think your method gives a better representation of the data change with time, and as the dataset becomes sparser will produce even clearer results. I don’t know whether the classical First Difference method was intended for use with such data as temperatures, where it is by definition a continuum that we have samples of. Either holding the last datum point as you do or generating a straight-line interpolation would be better than restarting with each lost datapoint. Interpolation would give the series 0,-0.15,-0.15,-0.3 in your example above – maybe a closer approximation to what happened though of course we really can’t tell since the data is missing.
It seems that the general agreement of the two methods is pure chance – around as many datapoints were lost on a downward tend as on an upward tend. If there was a bias in the lost data, such that it only occurred on one side or the other, you’d see a much bigger variance between the results. Something to think about in test-data sets passed to the programs. You could give their program a dataset it would choke on and give wildly (and obviously) invalid answers by carefully missing out the optimum points.
In the limiting case where every other data point is missing, your method would give valid results whilst the classical method would give zero.
“Ringside, and you are there. Now the ball is in the ‘Bunched Panties’ court. Will they address the latest serve by EM in the classic style of considered and reflective return or bunch their panties higher into a New Age wedgie and scream bloody murder about something that has nothing to do with anything of real value applicable to the matter at hand? But first a word from our Sponsor: “How much beer will it take to become bold enough to pilot a neon sign charger? Let’s ask the Michelob Man.. back in a flash sports fans”
Let temperatures and their differences rest in peace, there is no problem with actual temperatures, they are and they will be just temperatures: You are mesmerized by them, and that could be part of a “trick”.
I agree, adolfogiurfa.
The AGW scam has nothing, absolutely nothing, to do with Earth’s global climate. It is all about a 1945 decision to build a tyrannical, one-world government to:
a.) Hide energy (E) stored as mass (m) in cores of atoms, stars, galaxies
b.) Protect world leaders and society from possible destruction by “nuclear fires”
Frightened world leaders betrayed Galileo’s brave commitment to give the public observational reality instead of the model of a geo-centric universe that was favored by ego-centric leaders of nations and religions the 1600s.
http://tinyurl.com/7qx7zxs
See: “Neutron repulsion,” The Apeiron Journal 19, 123-150 (2012) http://tinyurl.com/7t5ojrn
I’m sorry for the early way-way off-topic comment, but can someone tell me why this is not totally overtly stupid:
http://www.dailymail.co.uk/news/article-2164164/New-York-Washington-threatened-global-warming-study-shows-seas-rising-faster-East-Coast.html#ixzz1ynR8VLiF
I don’t understand how sea level could rise faster on one side of the ocean than on the other. Its a fluid. Its connected. The change they posit is very slow (cm per year). What kind of physics would allow water on one side of an ocean to go up (or down) faster than the other for more than one cycle of tides?!
Nobody I talk with regularly uses their brain often enough to see why this bothers me, so I’m here spamming Chiefio’s comments in search of a thought.
Kevin – run an experiment on it. Get a piece of flexible plastic tubing with water in, and hold one end in each hand. Wait for the levels to become equal, then move one side down. The response is not instant – again you have to wait. Now put a cork in one end or hold your thumb over it and move the ends level again. The water will not reach level until you release the cork and let the air pressures equalise on both ends of the water.
The water-level is affected by both dynamic changes and by the air pressure above it. If you look at a river, you might also see that it cannot be truly level, since otherwise it wouldn’t flow. The momentum of a current-flow can also give you a higher water-level in one place than you’d expect from static considerations. On a river-bend, the water on the outside normally runs faster – the river is obviously not level across its width, either.
@KevinM We are not suppose to use our brains in the postmodern world that was introduced by Fred Hoyle [1,2] himself in 1946.
I do not know but strongly suspect that Hoyle warned George Orwell about the future [3].
I wish I were joking.
Oliver K. Manuel
Former NASA Principal
Investigator for Apollo
http://www.omatumr.com
01. Fred Hoyle, “The synthesis of the elements from hydrogen,” Monthly Notices Royal Astronomical Society 106, 343-83 (1946): http://tinyurl.com/8aal4oy
02. Fred Hoyle, “The chemical composition of the stars,” Monthly Notices Royal Astronomical Society 106, 255-59 (1946): http://tinyurl.com/6uhm4xv
03. George Orwell, “1984″ (Harcourt Brace Jovanovich, Inc, 1949, First Signet Classic Printing, 1950, 328 pages): http://www.online-literature.com/orwell/1984/
@KevinM:
That story is horridly written. What it really said, down toward the bottom, was that as the gulf stream changes velocity their’s a bit of back up in the Carolinas… and at other times, not so much.
To then wrap that in “Global Warming” causality is just silly. And bogus.
Simon D,
I don’t see how your examples apply.
In the case of the equalization not being instant in a tube, these sea level projections are over decades. With the Atlantic and pacific connected by such wide open gaps, and the moon yanking water up and dropping it twice a day, I don’t think that effect lasts more than a month.
Likewise the river flow example. The river has water flowing from high to low, propelled by gravity, along a constrained path. When the river is forced to turn, I can see momentum carrying it up the bank. Add more water, get more momentum, get a greater carry uphill. But sea level rise should equalize under gravity because there is no constrained path. Or is the onrush of water dragged East-to-west by Earth spinning under the moon like a periodic river… In that case I see a higher increase at high tide, but a lesser increase at low tide, and so a mure energetic churning action.
I don’t have my mind wrapped around air presure yet, except that it is exerting a force. In this case the proposition would be that the air on the east coast consistently exerts less downward force on water than on the West coast (or vice-versa). So you add water everywhere and it gathers preferentially in the lower pressure area. I need to google and think.
Thanks, that’s the kind of help I was looking for. Needed a direction to start running.
Oh, thanks EM. Saw Simons response first. If you’re read is right then that’s just a tragic abuse of journalism, and I’ll stick with my initial suspicion. But Simon gave me an idea or two worth thought experimenting. Like if you add ten feet (just an out of the hat big number) of water to the oceans, does high tide increase more on (all of the) east coasts than wests, and does it net out to the same average increase with greater volatility? I don’t know, I need to think it.
And sorry again for the thread hijack.
Kevin – The main idea I was trying to get across is that the actual surface-height of the water is never going to be level since there is always movement from tides and ocean currents at least. A persistent low atmospheric pressure in one area will raise the sea level there, and will drop the sea-level in places where the pressure is higher.
If there is a real problem when the sea-level is raised 4.8″, then virtually any hurricane (low atmospheric pressure for long enough to gather a “storm surge” of higher water level) is going to be catastrophic anyway.
I Googled the question and found a lot of answers. Regardless of the reason its only a few centimeters. The answer I liked best had to do with how much salt is in each ocean.
http://wiki.answers.com/Q/Are_the_sea_levels_different_in_the_Atlantic_and_the_Pacific_oceans
“Wasn’t that a Wonderful Commercial? Anyway, here we are back at Ringside, and you are there. Here. Whatever! Now the ball has been in the ‘Bunched Panties’ court for a while now. Will they address the latest serve by EM? They must still be scratching their heads in the classic style of consideration and reflection or bunching their panties ever higher-and-higher into a Tremendious New Age Wedgie, if so, I’ll bet their scream of Bloody Murder (about something that has nothing to do with anything of real value applicable to the matter at hand) will be a Lu-Lu. While we wait, we’ll rejoin the fans here at Ringside and get some more of their interesting conversations about various subjects, but first, the answer to that trick question we asked a few minutes ago, ‘How much beer will it take to become bold enough to pilot a neon sign charger?’ Let’s ask EM, the Michelob Man, Chiefio how much beer will it take to become bold enough to pilot a neon sign charger?”
;-)
Pingback: WIP on First Differences | Musings from the Chiefio
Pascvaks, Mr Mosher rarely hangs around to investigate, question, debate ideas anymore. Usually it is a “this is the way it is” statement, perhaps a condecending remark or two, and then he is gone as though he solved all puzzles. Really it is quite disappointing as I thought him quite capable.
@Pascvaks:
After due consideration, that is not the limiting factor. After 1.5 L of Gordon Biersch and about the same at Tied House, I offered interest in attending the inaugural event.
This was declined by the creator of the device.
So it looks like we have an existence proof that the quantity need to elicit Pilot Volunteers is less than that needed to elicit opportunity to be fried, er flied…
Test Pilots are a dime a dozen, The chief information officer is not replaceable! Besides the early test efforts will be ground tied. No fun at all, sorry. pg