dMT/dt an improved version

A Review

I’d made a couple of trial runs at a “monthly anomaly trend” graph based roughly on a running total of the monthly dT/dt values. There were two issues.

First, the graphs were very “busy” and it was hard to hear what they had to say in the clutter.

Second, they were ‘quirky’ in that the same attribute that gave the dT/dt graphs the nice “bullseye” marker where things changed oddly, also put some odd processing artifacts in the graphs. This was basically a side effect of how “Duplicate Numbers” were handled.

The “Duplicate Number” (as NCDC calls them) or the “Modification History Flag” as the code named it, tells you that some change of processing history applies to a thermometer data run. Most of the thermometer records have multiple “Dup Flags”. For the dT/dt series I wanted to emphasize this ‘splice’ so I deliberately let the “difference” show up as a line of zeros in a data set when the “Dup Number” changed. (As in First Differences where a new record starts with a line of zeros as the first time you see a value it has no difference from itself). This involved deliberately suppressing the “overlap” period that most records have. (Since the records are almost always identical, doing an average of X with Y when both are the same yields the same answer…) and the “bullseye” was a great diagnostic artifact.

However.

On the monthly running total graphs the discontinuity was enhanced too much. For Mauritius, for example, we got a jump up in December that other months did not flaunt so much, due entirely to that ‘reset on dup number change’.

The Change

In this version I’ve gone ahead and “feathered the splice”. There is typically a couple of years of overlap between each Dup Number. Rather than suppress that and enhance the detection of a transition, in this version I gently ‘feather the edge’ by blending the disjoint Duplicate Number records into one record (and thus also avoid the ‘line of zeros’ on a reset from a record change).

This is done by the expedient of simply averaging the temperatures together for a given thermometer during the overlap period. I may someday change this to averaging anomalies, but it doesn’t really matter. Inspection of the records shows that they are almost universally identical, and where there is a difference it is usually in the 1/10 C space or is a missing datum in one series but present in the other.

So for an overlapping record, I’ll average the temperatures for EACH single thermometer and splice the different Duplicate Number series together into a single data series. This can have a small impact on the “warming” found. In most cases I looked at, it flattens the overall graph by a fractional amount. This is likely an artifact of the “lost data” in the gap of zeros being from, typically, the same year for large groups of thermometers. That is, it is a ‘splice artifact’ from the concentration of the change in a single year. This blending via averaging method of splicing will give a smoother (and more accurate) rendition of actual temperature changes in the thermometer record since we will only take a line of zeros on the very first record for a given thermometer.

Basically, the forensic value of flagging the splice drops, but the “fidelity to actual history of thermometer reading” goes up. Especially in the detail of individual months where a single dropped datum might put a 2 C ‘step’ into a line of total anomaly to date at that ‘enhanced splice effect join’ point.

Also, in these graphs, I’ve gone to emphasizing the trend line for each month. You can still see individual months ‘cumulative anomaly’ volatility and relative position, but I think that seeing the overall trend line is usually the most informative.

OK, enough on process. If folks really care, I’ll provide code and tutoring on how to reproduce any particular variation on these graphs and methods. For now, on to some more graphs and analysis…

Australia

As before, you can click on a graph to get a very large version to look at in more detail.

Australia Dup Number blended cumulative monthly anomalies

Australia Dup Number blended cumulative monthly anomalies

Not really much different overall. We still see some months falling while others are rising. Neat trick, eh? With the emphasis on trend lines you can see, for example, that the orange November line is strongly dropping, yet when The Great Dying of Thermometers begins, it makes a hard “U Turn” and heads up. It is a month with a strong down trend into the baseline, but it also is clearly behaving differently from some of the other months.

IMHO, this is a strong “Smoking Gun” that the ‘warming’ in the record is not a CO2 effect, but that it is rather a “thermometer selection and splicing” artifact. We will see very similar effects in each graph.

In this case we have about 1/2 the months dropping and about 1/2 rising. Not quite a “global warming” effect… It’s a race condition between these two groups of changes that results in a net warming or cooling. The “dT” line is the net. That’s the “hot pink” trend line. You will notice that for Australia it has a very small (trivial, really) positive slope. The bulk of the “warming” for GISS or CRU would come from that “dip” in the baseline during that 1951 to 1991 span; and with most of that coming from just a couple of individual monthly trends.

What’s more, there are even times when adjacent countries or thermometers have the same months going in different directions (or different months going in the same direction). The sheer chaos of which way months are trending argues strongly against the CO2 ‘broadly warming’ thesis and argues strongly for an “instrumentation and process” artifacts explanation.

Some Specific Australian Stations

We saw this Marble Bar set earlier as part of a ‘splice set’ example in the “Mysterious Marble Bar” posting. In this graph we can see the “boom town” warming, but with monthly detail on trends.

Marble Bar / Nullagine P.O. blended cumulative monthly anomalies

Marble Bar / Nullagine P.O. blended cumulative monthly anomalies

Interesting that even in a “boom town” with growing UHI there were some months rising while others fall. August dropping, May flat, October on a rocket ride. Looks more like local effects than CO2 to me.

Here we see Darwin, dropping:

Darwin blended duplicate number cumulative anomalies

Darwin blended duplicate number cumulative anomalies

So it’s getting hot in Marble Bar, but Darwin is cooling off. What’s “Global” about that?

And Hobart down in Tasmania:

Hobart blended duplicate number cumulative monthly anomalies

Hobart blended duplicate number cumulative monthly anomalies

January and December falling, June July August and September rising. So it’s cooling off in summer but winter is warming? Sounds like a UHI issue to me. That the ‘shoulder months’ are basically flat also argues for fuel driven effect.

South Africa

What a pretty “bow tie” effect!

For comparison, we have some more southern hemisphere graphs:

South Africa blended duplicate number cumulative monthly anomalies

South Africa blended duplicate number cumulative monthly anomalies

We note that unlike Australia, November is rising nicely. And April is flat where it was rising nicely in Australia. One is left to wonder how such trends can be reconciled with a generally diffuse physical effect as CO2 is supposed to be.

Nearby is Mauritius:

Mauritius blended duplicates cumulative monthly anomalies

Mauritius blended duplicates cumulative monthly anomalies

And for comparison at the top of Africa:

Egypt blended duplicates cumulative monthly anomalies

Egypt blended duplicates cumulative monthly anomalies

Over time thermometers move away from the coastal water and into the hotter interior. I suspect that contributes to things like that very hot December.

South America

Another “bow tie” with a different set of risers and fallers:

South America blended duplicate numbers cumulative monthly anomalies

South America blended duplicate numbers cumulative monthly anomalies

Brazil anyone?

Brazil blended duplicates cumulative monthly anomalies

Brazil blended duplicates cumulative monthly anomalies

June, July, and August are just flying, the rest of the year not so much… Wonder if there is more jet traffic at the airports in those months?

Just south in Argentina:

Argentina blended duplicates cumulative monthly anomalies

Argentina blended duplicates cumulative monthly anomalies

What a mess. More dynamic range than Brazil, but still has months going in all directions.

I like Chile:

Chile blended duplicates cumulative monthly anomalies

Chile blended duplicates cumulative monthly anomalies

Though I’m wondering what the deal is with September and especially June. Sudden outburst of heaters in June or something?

Elsewhere in the Pacific

Dropping like a rock in most months.

French Polynesia blended duplicate numbers cumulative monthly anomalies

French Polynesia blended duplicate numbers cumulative monthly anomalies

Though a lot of the drop looks like it comes out of that high start, prior to the added thermometers splice. Then again, the 1930’s – 40’s were reported warm in many places.

How about New Zealand?

New Zealand blended duplicates cumulative monthly anomalies

New Zealand blended duplicates cumulative monthly anomalies

What a mix! August on fire, September calming, but back to warming again in December. March relatively flat (slight down trend) with April and July dropping. But what is happening in May and June! Who lit them on fire? I’m sure someone can figure out what’s going on here, but it won’t be easy. And I’d just love to hear how CO2 is supposed to cause that selective warming over that many years.

Antarctica

Still with a strong wedgee going on. Amazing really, how divergent the months are.

Antarctica blended duplicates cumulative monthly anomalies

Antarctica blended duplicates cumulative monthly anomalie

Conclusions

Blending or “feathering together” the duplicate numbers smoothes some of the individual months but does not strongly impact the overall character of the graphs nor the conclusions I would draw from them. It does make the “trend line” a good method for seeing the divergent nature of individual months; by station, by country, and even between regions.

The warming seen in some places looks, to me, like it varies more by individual location issues than by any global effect.

Subscribe to feed

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in dT/dt and tagged , , , . Bookmark the permalink.

11 Responses to dMT/dt an improved version

  1. Robert says:

    How about graphing the gradient of the linear-fit slope against the month for all the graphs you have just done. That way can see trends in the month by month change for all the data sets around the world on the one graph, without clutter.

    REPLY: [ Interesting idea, except the slope by month is often highly divergent for very similar places. Chile and Argentina for example. Or Bermuda, Jamaica and The Bahamas. I’m suspect we get more information out of side by side comparisons (from the two different graphs) than out of one single super graph. Then again, perhaps I’m not understanding your vision… It would be interesting to try in any case to “see what the data say”. -E.M.Smith ]

  2. Chuckles says:

    We ‘feather the splice’, because ‘hiding the decline’ is just so last year.

  3. E.M.Smith says:

    @Chuckles:

    Got a smile out of that one ;-)

    Yes, the ‘splice’ behaviour is a prime example of how forensics is different from “academics”. In an academic world view the only thing you would want to do is “blend” the splice so it doesn’t show. In the forensics world, you want to enhance any (even trivial) discontinuity to see if there is something to investigate.

    Like shining light at high obliquity on a surface; or using odd spectra; or even doing various surface etchings. You are more interested in seeing if an acid etch shows a weld line in a metal than in putting a nice blued finish over it to disguise the weld.

    So I’m typically making my first cut at things with a forensics mind set. Emphasize what’s going on. Then depending on what I find, I may come back and polish the surface. This, BTW, seems to be the behaviour that causes He Who Shall Not Be Named to go all bat shit and claim “the ravings of a lunatic”. One can only surmise that he has no law enforcement background. At least not from the investigators side of the table…

    So having, I think, gotten a fair amount of milage out of emphasizing the splice (by suppression of the overlap of the duplicate numbers at the splice) and seeing where there were a couple of massive splice artifacts; it was now time to smooth over those splices and step back. Look at the larger picture a bit more polished; but with the monthly trends emphasized.

    To move to a different spectra and put the light at a different obliquity and see what turns up.

    And we find widely divergent trends when looked at by month. Months with smooth splices and rough edges feathered together. (The enhanced splice artifacts tended to be lumped in particular months so much that they masked some of the individual monthly trends, or at least messed them up a bit.) That individually the months diverge so much is rather fascinating.

    I’m left to wonder if what we’re seeing is an emphasized version of the various errors in the data. UHI in high heating fuel use months. Airport Heat Islands in places with travel spikes or when the winds are low and tarmac heat can build up. (It would be interesting to plot monthly average wind speed vs these data… Airport heat islands are known to be strongest when winds are lowest.) Splicing in different locations with divergent monthly trends at different portions of long duration weather events (like PDO flips). Etc.

    I’ve often said that GISS et. al. are just “dancing in the error bands of the data” and I think these divergent monthly trends shine a very bright light on that.

  4. Mike Fox says:

    Dear Mr. Smith,

    It seems to me that the fascinating analysis (analyses) to which you’ve devoted yourself over the past few months deserve a wider audience.

    May I respectfully suggest that you compile and digest your analyses and findings and submit them as an Op-Ed to the Wall Street Journal? Editorially, that paper has been remarkably responsible (i.e., skeptical) with regard to the “AGW question” and Climategate. Today, e.g., it published an Op-Ed by Richard Lindzen.

    Dissemination of your findings by the most widely read and respected paper in the country (and one with an increasing circulation) couldn’t hurt.

    Best regards,

    Mike Fox

    REPLY: [ Not a bad idea. Though it would take a “Cliffs Notes” version. I’m slated to give a short presentation at a conference soon and I’m thinking I’ll make a “short summary” version for that. I do think I’m close to the point where I’ve found out enough of what’s going on to make a decent article out of it. This has been an interesting experiment in “letting folks look over the shoulder” as you investigate something… but eventually it has to become a formal summary paper or you are just being a talk show host… There have been a couple of ‘spin offs’ that have taken wing (something I’d hoped for) but not quite as much collaborative barn raising as I’d hoped to see. I also was a bit unprepared for the impact of notoriety. Oh well. Times move on. We’ll see where things end up. -E.M.Smith ]

  5. Ecotretas says:

    I’ve done some analysis of one of the most northern weather stations: Eureka, in Nunavut Canada: http://ecotretas.blogspot.com/2010/04/eureka.html
    I believe further investigation should be conducted, as temperature variations are clearly not of scientific quality.

    Ecotretas

  6. Athelstan says:

    Some good, thorough and conscientious work done here, thank you.

  7. E.M.Smith says:

    @Athelstan

    Thanks! I’m trying as best I can to do a “bottoms up” evaluation of what’s the best way to do things AND “how the other guys did it”. The major problem is that I’ve got to make enough money to eat and pay the rent while I’m doing this, so it goes more slowly than I’d like.

    So far it looks like a combination of ‘splice artifacts’ and ‘instrument change’ along with a bit of ‘broken QA’… all I have to do is tie it all together and demonstrate conclusively how it all works ;-)

  8. E.M.Smith says:

    @ecotretas

    Wow! Very well done. Has Anthony at WUWT seen this? This, IMHO, is the kind of detailed no doubt about it work that will shape the “debate”.

    Sidebar: FWIW between Spanish and French I can mostly read Portuguese just fine. Yet there are some local speakers of Portuguese (who run a nearby store) and for the life of me I can’t follow the spoken form. Pick out words here and there, sure. Follow the speech? No way. There is also a local Portuguese radio station that’s got great music, and I can follow a bit more of the “happy talk” during breaks. Don’t know what it is but there is something in the phonology that throws me… I’ve got a Portuguese cd and book and probably ought to get back to it.. I love the sound of the language… more than Spanish. Oddly as well, I can read Catalan and find it an ‘easier sound’ than Castilian on the ears. I was looking at the language map of the Iberian peninsula the other day and was surprised at how much mixture there was; especially how much non-Castilian was spoken in Spain. At any rate, you can count me among the Portuguese speaker wanna-bees… ( The Language Map thing came about after watching “Tres Dias” and realizing that I could understand the Spanish on the TV in the movie, but what people were speaking was different though intelligible… Still not sure what it was, but it was a non-Castilian dialect of Spanish and ‘sounded better’ to me. Was it more Catalan or more from one of the Portuguese influenced areas? Still don’t know… But a fascinating movie. I think the English title was “After the Fall”. A very spooky and compelling movie, in any language.)

  9. David Jones says:

    Do you have any source code published for this revised analysis? I think other people or groups would be interested in running it.

    It would be a simple matter to create a googlecode project and publish code there.

    Thanks.

    REPLY: [ I’d started with publishing all code, but most folks didn’t want to see it, so I’ve gone to publishing ‘bits of code’ for the more interesting bits. It’s pretty darned simple code and I doubt that it justifies a whole project. If anyone wants the code they can let me know and I’ll provide or post it. The only difference between the “old” and the “new” is how duplicate data are averaged together (and thus how the “duplicate data flag” change / reset impacts the monthlies). I suspect most folks would prefer a pseudo-code description of what the program does rather than the somewhat ancient FORTRAN it’s written in now. (I got started using FORTRAN again as GIStemp uses it and I just found it easier to keep on language in my head (and on the machine) at a time…) -E.M.Smith ]

  10. David Jones says:

    Well, could I have the code please? You can e-mail it to me at drj@ravenbrook.com

    I think I’m getting pretty good at reading Fortran now.

    PS: Your habit of making inline replies means I didn’t see your reply for ages. A new comment would’ve alerted me by sending me an e-mail, as I’m subscribed to the comment feed for this thread.

  11. E.M.Smith says:

    The original version of the anomaly creation code was posted in the comments in this link:

    Germany – Not Warming

    The difference between that one and this one is documented above (and ought to be pretty trivial for you to implement).

    I’m assuming you don’t need the ‘wrapper script’ that just sorts the files in the right order, but if you do, holler and I’ll post it.

    Also, as described several times (and I think, in that link) once you have the anomaly file, making the report (that is fed into a hand graphing stage) is pretty trivial. I can post the (rather ugly) FORTRAN to do it, but I’d really encourage folks to ‘roll their own’ report writer. (It is really just making a running total of the year over year anomalies by month and ought to be fairly trivial in Python or R.

Comments are closed.