GISS Benchmarking The Baseline

GISS 2009 Anomaly Map default baseline

GISS 2009 Anomaly Map Default 1951-1980 Baseline

This is the 2009 anomaly map measured against the default 1951-1980 benchmark. How much of those colors represent bias from the baseline? How much represent the benchmark period being cold? My suggestion:

Inverted Baseline vs Total Data

Total Data compared to baseline or Baseline deviation from all data

So, what we find is that several of the persistent patterns in the anomaly maps seem to be artifacts of the baseline years chosen:

In particular, those persistent Hot Hot Hot Arctic temperatures. Here we can see that there is a consistent warming contributed to the area north of Alaska and the north of Siberia. Gee, Siberia being warm, where have I heard that one before?

In Antarctica, we have both the “Warm Peninsula” and that persistent odd cold patch that keeps showing up.

The warm central Pacific patch looks like it has a precursor as does the warm spot in the middle south of Africa along with Madagascar and the ocean south of it.

In my opinion, many of the persistent “odd” features of the anomaly maps I’ve looked at over the last couple of years can be explained by the peculiarities of the baseline chosen by GISS for their default.

Benchmarking The Baseline

I had set out to measure any bias that might be in the GIStemp “baseline” interval of 1951 to 1980 a couple of days ago and ran into an odd bug (where the world’s oceans turned blood red and GISS stated they were 9999 degrees) that took a day or so to explore. Ok, now it’s time to get back to my original track.

Basically, the question is: To what extent is there evidence that the “baseline” interval has bias in it?

This is not as easy to answer as you might think. Against what do you measure the baseline measurement? Where is your ‘gold standard’ or your ‘platinum meter’ or your ‘caesium clock’ for global temperatures against which to measure your ‘standard’ ?

Ok, we’ll have to ‘make one up’. While it isn’t an ideal choice, the best I can think of for this test is just to use the total temperature history in GIStemp. While we know the start date of 1880 is a bit of a “cherry pick” for warming (the world was warmer in 1720 than in 1880 and was just coming out of the Little Ice Age when GIStemp starts the series) it also includes roughly a full half cycle of temperatures. To use from 1720 to 2009 would put 2 ‘high’ ends in with only one cold middle. So I’ve chosen to use the entire life span of the data.

Even this ‘has issues’ in that the data are geographically sparse in the early years. GIStemp uses what it has to ‘fill in’ some boxes. Part of what we will be seeing is in the ‘unadjusted’ data and part of it is the effect of GIStemp adjustments and fill in. But this is a valid thing to do, since it is what GIStemp does when it makes the maps we look at.

Finally, if we compare the baseline to the whole data, we get one view (what those years looked like) but if we invert that image, by measuring ‘all data’ against the baseline we see how that baseline would contribute to the bias of any given year as compared to the total of all data. That is what was done in the above graph.

OK, So What Does The 1951-1980 Interval vs All Data Look Like?

This is the more normal view. What those 1951-1980 years look like in comparison to All Data. And they do look cold in general, but with a couple of slightly warmish spots.

GISS Default baseline 1951-1980 vs All Data

GISS Default baseline 1951-1980 vs All Data. A bit cold.

It all looks a bit cold to me, except for that one really hot spot in Antarctica. Oddly, right next to is a Very Cold Antarctic Peninsula. That cold baseline goes a long way toward explaining the “hot peninsula” stories of the last few decades.

For folks wanting to see if the 1961-1990 baseline used by other temperature series, like HadCrut, has a similar bias, here is that baseline compared to ‘all data’. First for the ‘unadjusted’ data (as used by those other codes) then for the GISS processed version.

1961 to 1990 "unadjusted" vs All Data

This graph is the ‘as re-imagined by GISS’ data:

1961 to 1990 'baseline' using GISS data compared to all data

1961 to 1990 'baseline' using GISS data compared to all data

In it, we see a bit more of the Antarctic (due to GISS adding SCAR data) and again the arctic blues are a bit more enhanced. That same Arctic spot that was deep red for the default GISS interval is now blue. I wonder if that is a flaky station or a place with wide swings of “30 year weather”?

What 2009 vs All Data Looks Like If You Use

the “Unadjusted” Input Data?

For what it’s worth, I was going to make a “2009 vs All Data” map using the ‘unadjusted data’, that GIStemp uses as input, to see how much of the “anomaly” was baseline, how much was GISS processing, and how much was in the “unadjusted” data. But when you try to do that you get this message;

Surface Temperature Analysis: Maps

Error

I cannot construct a plot from your input. GHCN unadjusted plots only available through 1999

So one is left to wonder what they use for GHCN data input if they do not have any “unadjusted” data after 1999. Are the current GISS maps based on “unadjusted” through 1999 and adjusted afterwards? One hopes not. Yet it simply is not known. The documentation says it uses “unadjusted”, but this interface to the actual product being run can’t find it for 2000 – 2009. OK, we have a big “Dig Here!”, but I’m doing other things right now, so this one will need to wait; or fall to someone else to figure out.

The “unadjusted” data file downloaded from NCDC has data through 2009, so what GISS has done to lose those years data in their Anomaly Map product is, er, an open issue…

So the best I can do is this 1951-1980 vs All Data GHCN ‘unadjusted’ graph. (But that leaves me wondering where that 2009 ‘unadjusted’ data came from… )

GHCN unadjusted 1951-1980 vs All Data Anomaly Map

GHCN unadjusted 1951-1980 vs All Data Anomaly Map

It looks generally more “muted” than the GISS processed version. This implies to me that a fair degree of what we see in the anomaly maps for today comes directly from how GIStemp processes the “unadjusted” data that it uses to make the baseline period.

The Default 1951-1980 Baseline By Decades

So is there any particular part of that baseline that looks like it “stands out” as not a typical period of time? I’m going to look at it ‘by decades’. Single years probably don’t do that much to bias a 30 year span, and “by decades” ought to show up where there is any issue. Other folks can dig into the individual decades if they wish and see if any particular years are spectacular or not using the GISS web site.

1951-1960

1951-1960 vs All Data Anomaly Map

1951-1960 vs All Data Anomaly Map

Looks a bit cold to me, but not too bad. Looks like we get our cold Siberia and our cold Antarctic Peninsula from here. Also that cold spot near South American coastal Ecuador looks like it influences the present as well.

1961-1970

1961-1970 vs All Data Anomaly Map

1961-1970 vs All Data Anomaly Map

Well, quite a bit of cold. That North of Siberia Arctic is “way cold” and likely to provide bias for decades to come.

1971-1980

1971-1980 vs All Data Anomaly Map

1961-1970 vs All Data Anomaly Map

A continued cold Arctic. OK, looks like a pretty good “cherry pick” (accidental or otherwise) for a very cold time in the Arctic. We also see that ‘hot patch’ in Antarctica. In the other two decades, that patch is NULL, so we now know that this single decade (and perhaps even a subset of it) contributes all that “hot spot” to the baseline and that explains that particular “cool” feature in our present maps vs this baseline.

Scanning back over these three maps, we also see a persistent cold spot in the central / southern area of Africa.

In general, it looks to me like 30 years is just too short a time period to use as a baseline, especially given the sparse data, limited coverage, and data dropouts of those early years. Artifacts of that sparse data and short time period “bleed through” into the present anomaly maps and bias the perception of those maps.

Isn’t That Just Showing The Arctic Has Warmed?

For the inevitable complaint that I’m just showing that the arctic has warmed since 1950, we can look at this graph. This is the period from 1931 to 1940 measured against ‘all data’. It, as you can see, has a very warm Arctic. So at the end of the day I’m left to conclude that a significant part of our present “Hot Arctic” is a result of comparing it to an abnormally cold period in the baseline interval.

!931-40 vs All Data Anomaly map

!931-40 vs All Data Anomaly map

For Boballab: 1931 start, 60 years duration

Your wish is my command:

2009 Calendar Annual vs 1931-1990 baseline

2009 Calendar Annual vs 1931-1990 baseline

If you open two copies of this page, you can compare this version with the one at the very top of the page. Both are calendar year 2009, but the top one is the default baseline while this is the “boballab” suggested baseline of 1931 for 60 years. (FWIW, I liked the 70 duration graph better, but this one has some interesting features too).

OK, what I noticed is that “baselines matter” in that there is an overall ‘cooling’ of the map. Not dramatic, but hey, if we’re supposed to panic over 10/100 C then I think 5/100 C is significant! (Top map is 0.68 this one is 0.63 in the upper right corner).

Most dramatic is that odd patch to the west of the Antarctic Peninsula. It has entirely changed color. Specifics of baseline matter, especially in places with sparse data. Regional effects DO show up based on the baseline.

Lesser drama, but still very interesting: Africa inland on the side near Madagascar gets a white blob. The cool area in North America stretches slightly further south. Siberia cools off. French Polynesia gets a bit of cooling as does southern Alaska.

It’s like one of those “what’s different in these two pictures” games. Subtile changes, but very real. So: “baseline matters” and longer is better.

Advertisements

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in AGW GIStemp Specific. Bookmark the permalink.

47 Responses to GISS Benchmarking The Baseline

  1. rob r says:

    Fantastic stuff EM.

    Looks like they need to consider lengthening the baseline period and starting at least as far back as 1930.

    Also they should push the baseline out at least to 1990 so that there is better coverage for Antarctica.

    They seem to be operating under the unproven assumption that regional average temperatures from 1950-1980 contain no inter-regional anomalies. It seems you have demonstrated that there actually are inter-regional anomalies even in 30 years of continuous data.

    By extension I would speculate that the Hadcrut temp series based on 1960-1990 baselines are similarly afflicted but that the set of inter-regional anomalies be significantly different.

    The whole idea that the baseline period should be 30 years is a completely artificial construct. I have seen complaints about this issue around the climate blogs for some time. The issue just seems to get a casual brush-off from the AGW scientists as if it is of no consequence.

    It would be useful to see this demonstration of yours as a new topic/thread on WUWT and TAV (for instance).

  2. boballab says:

    The thing with baselines that I have gathered from a little diggin and backed up by the Climategate emails is that they are not supposed to stay static, they were supposed to shift everytime we ended a decade, but didn’t because certain groups didn’t want to go back and redo their baseline work.

    When the process started in the 1980’s, 1980 was the last year with 0 in it so they went back 30 years from there. That got us the 1951 to 1980 Baseline. The WMO then put out that for use, in the IPCC works the baseline needed to be 1961 to 1990 when the date switched to 1991. When that date came around the CRU switched as stated in the Climategae emails, but GISS stayed with 51-80.

    Now this procedure was suppose to continue and that is when in the climategate emails you see Phil Jones talking about not wanting to switch to what was the new baseline of 1971-2000 (which some organizations did such as the BOM in Australia if I remember right) for the TAR in 2001.

    Jones view prevailed and the IPCC ditched the WMO program and why not it wasn’t enforced on anyone with the previous switch. This then brings us to the here and now and everyone groundstation and satellite is suppose to switch to the 1981-2010 baseline next year. For some reason I suspect that RSS and UAH will switch to that baseline but CRU and GISS wanting to stay with the old one since those baselines give you such nice rosy red anomaly maps.

  3. boballab says:

    Ack forgot to put this in the other post.

    IMHO opinion the baseline should be at a minimum 60 years and cover a full cycle of the PDO since the temperature record seems to follow its shift from warm to cold and vice versa.

  4. E.M.Smith says:

    @boballab:

    Take a look at the end of the posting now. I’ve added your suggested baseline and guess what: Baseline matters and longer is better ;-)

  5. The selection of a base period wil not impact the rate of warming seen. What it will do is given different patterns of warming, different spatial patterns. The longer the period the more likely you will see the difference between ALL and the base period shrink to a uniform map. at 30 years it’s obvious that there still are some “weather patterns” in the data and some noise.

  6. Pingback: Al Gores getting his hands on your super! « TWAWKI

  7. boballab says:

    @ Moshpit

    Thats why I like the roughly 60 year PDO cycle as the baseline, knock that “weather pattern” out. Also you should take a look at the maps here:

    http://jisao.washington.edu/pdo/

    Whats interesting in the global maps is that when the PDO is in a “warm” phase it shows a cooling arctic and when in a “Cool” phase a warming arctic during the wintertime. So with the PDO in a “cool” phase at least a part of the “warm” arctic this winter might be attributed to natural causes.

  8. Boris says:

    As mosh points out (hey, GFY mosh), the trends don’t change at all. I guess this is some sort of argument about presentation or something? I don’t get it.

    (Also GFY=good for you, obviously:) )

  9. Jim Masterson says:

    After reading John Daly’s comment about the move of Death Valley’s thermometer to Badwater, I downloaded the 2001 temperature table for Death Valley from Gistemp. It was interesting to notice that the third coolest year was in 1998–the supposedly hottest year. If the greenhouse gas model is correct, then deserts should show the warming first–not so in Death Valley. I downloaded the table again in 2003. The graph looked the same except for the two additional years.

    I’ve downloaded the latest table of Gistemp temperatures for Death Valley and the plot looks nothing like the previous versions. It appears that Hansen’s been hard at work homogenizing these temperatures. He’s definitely added a positive increasing trend and has raised the lows–including the 1998 year. I can’t proved that my previous downloads are the official ones, because–like in Orwell’s 1984–the evidence is conveniently gone.

    Jim

  10. Bruce says:

    There is response above entitled, “on February 2, 2010 at 11:12 pm Al Gores getting his hands on your super! « TWAWKI”

    I’m not a computer geek, but isn’t this some type of a ping to get access to your site. I have seen the same comment today on another site, and have had similar comments on my site. I have also heard that you can hack into wordpress by sending something, I think the word was a “null message.” Do you know anything about this?

    Back to the subject, I am making an attempt to determine if the topic above has been peer-reviewed, as if that means anything. Using AR4 chapter 3 as a base, I am looking up the reference papers. Basically, from what I can determine, basic elements of the temperature collection and calculation process have been peer-reviewed, but the entire system has not been audited. You are doing a good job of uncovering many questionable issues with the system. Keep up the good work.

    REPLY: The comment you saw is a “pingback”. It means someone has put a link to this article on their web page. Many are just announcing that someone found the article of interest and provide a way to see other places of kindred spirit (and sometimes folks of critical spirit) talking about the same topic. There is a SPAM technique that harvests a line of text from the topic page and uses it to construct a phony page that just tries to suck you in to be pestered with adverts. Those mostly end up in the SPAM queue which I empty daily. Others end up in the Moderation queue where I inspect the link to see if it looks ‘reasonable’ before I let that site escape moderation. TWAWKI looked ‘reasonable’ though I only spent a few minutes looking at it.

    So if you follow any pingback and find a spammer, just let me know. But they mostly ought to be ‘real folks and real pages’ just pointing here.

    Per ‘null attacks’ there are many ways to hack into things. One of the common ones is to send a null loaded bucket of ‘stuff’ in response to an input request and see if something breaks such as to let you into the site. That is not a pingback. It’s done at a ‘login’ screen typically (though a null message might be interesting to try). But basically, yes, we are all dependent on someone finding and fixing such ‘exploits’ ever day on every system from every vendor on the planet. If you are worried about ‘being under attack’, don’t ever look into the world of computer security. There is a constant flood of attacks every minute of every day. Often thousands per hour on a single site. There is even a kind of attack based on a high level of failed attacks. The DDOS – Distributed Denial Of Service attack. You just get a few thousand machines to all ‘ask for a page’ at the same time, repeatedly, until the server crashes or just grinds into molasses. (And yes, there are ways to protect against that kind of attack too… all it takes is time and hardware and software and people and… ) So don’t look too closely if you are prone to sleeping poorly at night ;-)

    Finally, thanks for the compliment. Peer review is fine and all, but it does not address code quality. It addresses if the “bright idea” sounds like a good “bright idea” to other folks in the field. It does nothing to say if the “bright idea” can be instantiated in code, if that code has bugs, were there ‘shortcuts’ taken with the “bright idea” as it was being implemented or did the “bright idea” have issues of vagueness that a programer had to ‘pick one’ and maybe picked one that didn’t work so well, or was the ‘bright idea’ overdone to the point of breaking it, etc.

    So, in particular, “The Reference Station Method” has been peer reviewed. That means that they found that anomalies at one station have a correlation with anomalies at another station up to 1000 km way. That does not address if that correlation was valid in all geographies (only in the study geography) or all times (so a study all during one 30 year PDO phase might break down in the next). Nor does it say that a baseline for that correlation of 30 years (and potentially all in one PDO phase…) can be used to “correct” temperatures from 1/2 Century later and in a different PDO phase… Finally, the peer reviewed “Reference station method” is only saying ONE level of reference is valid. It does not say you can do this 3 times in a row on the same ‘data’ getting further and further from the truth as GIStemp does. Basically, it does not say you can apply “The Reference Station Method” recursively 3 times in a row and still be valid. And finally, there may well have been some specific sample size used in the papers (so, hypothetically, valid results might be found with 10 reference stations averaged together) yet the code might well make a decision such as: “If I can’t get 10, just pass the data through unchanged” as several parts of GIStemp does (for example, UHI is only applied if “enough” reference stations are found, otherwise data just flows past unchanged) bypassing the whole step you thought was being done. Oh, and the code could just have a bug in it from bad coding.

    You you NEVER NEVER EVER accept that a published paper will tell you that the code is right. It can’t. It can only tell you if the basic “bright idea” sounded good to a bunch of other folks in the area of interest.

    Or, put more briefly: Peer review of the idea is not a code audit and not a benchmark validation and not a QA suite.

    -E.M.Smith ]

  11. e.m.smith says:

    steven mosher
    The selection of a base period wil not impact the rate of warming seen.

    Um, I think that is an assumption stated as a conclusion… IFF my baseline is, say, 1720 when temps were the same as today, I would see no warming today, so no rate of warming, but I would see a cool 1880.

    Similarly, that “Boballab” baseline was 5/100 C cooler “warming” and other baselines are even less. The mid point of the “Boballab” baseline is 1961 where the GISS midpoint is 1966 so you have 5/100 C less “warming” over a 5 year longer time period from center of baseline. i.e. less ‘rate of warming’. Not dramatic, but then again we’re supposed to be stressing out about 10/100 C so half that seems ‘important’ given their standards…

    Your assumption rests on the replacement baseline being nearer in time than the old baseline (and that is largely a reasonable assumption, since the data get very sparse before 1920) but that is not a mathematical truth, only an accident of the data.

    In particular, using a 1931-40 baseline gives a ‘warming’ in 2009 of 0.59 C so we have a case where the ‘warming’ rate is 0.59/70 years as opposed to 0.68/30 years in the graph up top (measured from the end of the baseline – you can do midpoint if you like). This implies significantly different rates of “warming”.

    http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=0112&year1=2009&year2=2009&base1=1931&base2=1940&radius=1200&pol=reg

    What it will do is given different patterns of warming, different spatial patterns. The longer the period the more likely you will see the difference between ALL and the base period shrink to a uniform map. at 30 years it’s obvious that there still are some “weather patterns” in the data and some noise.

    It also points out that at 30 years and with sparse data some of those “weather patterns” and “noise” can have dramatic effects. Like that blob of Antarctica that can be fiery hot or icy cold based only on a small shift of baseline. While that small spot may not ‘matter much’ it also implies other areas with sparse data might well be ‘squirming’ but just enough less to be believable.

    Also, I think that “uniform map” idea has a couple of issues… Yes, you are correct that as the baseline approaches “ALL” the delta between them approaches zero. (“uniform map”) But I think that ought to be the goal, not a point of dismissal. If 2009 were cold, I would still get a blue map compared to all data. If hot, a yellow/red one. What becomes more “uniform” is only the degree of variation each year has in comparison to the baseline. i.e. hot years look less fiery red than when compared to the present cold baseline, and cold years less blue than if compared to a hot baseline period. But this does not approach zero, it approaches truth. That is, the “cherry pick” bias of the baseline becomes more uniform and approaches zero over time periods longer than weather cycles. 60 years at least, but there also look to be some ‘few hundred’ year patterns as well… like 1720 to today.) A heating location will still be red and a cooling place will still be blue on the maps, even against “all data” and even as “all data” becomes quite large. In particular, if we had data from 1720 to today in a baseline, we would see two ends about equally hot. The 1816 ‘year without a summer’ would still show as dramatically blue, the ‘uniform’ baseline just gives us a less biased view of both the ends and the middle.

    Again: I would not care if the maps were being used to inform our ignorance and guide researchers as to where to look for better information. But the maps are being used to scare the children and congressmen and to drive policy decisions. All based on fractional degrees that may, in fact, bit fictional degrees as well. That, I care about.

    So yes, baseline matters. And it changes the ‘rate of warming’.

  12. MicHussey says:

    Am I the only one that finds it suspicious that there is near-zero anonomly in continental Europe, US, Australia and Sub-saharan Africa when comparing 51-80 vs the “long-baseline”?

    To my jaundiced eye this indicates that the GISS hot and cold spots are simply artefacts of bad / low measurement density.

  13. AnthonyH says:

    As you suspected one of us would, I looked at the 1971-1980 time period to see where the hot area near the Antarctic peninsula comes from. All those years are null except for 1980. http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=0112&year1=1980&year2=1980&base1=1880&base2=2009&radius=1200&pol=reg

    That means that of 30 years in their baseline, 29 of them show no data for that region, and 1 shows data. That data is relatively hot (2-4 deg C) compared to the full data set.

    Considering how many ‘nulls’ there are in the baseline, shouldn’t there be an error estimate for the baseline. That red area, with just 1 year of data, should show up as having a very low confidence for the baseline quality.

    I wonder if we could figure out a method to calculate the number of ‘non-null’ months in the baseline for each cell, which would help identify areas that were less usable for comparing to other temps.

  14. E.M.Smith says:

    @MicHussey:

    I’d been thinking about the “sparse data” as causing some of the most obvious “issues” and artifacts but had not made the leap to the flips side. That “good measurements” showed no anomaly.

    Very Good Catch!

    I like to think of myself as being aware of “The Negative Space Question” (whatever you see, look at the opposite of it, or look at what is missing). But in this case I was too distracted with other things to even think of it.

    Thank you for being so observant!

    It really is almost a match for the inverse of thermometer density and quality. The only real exception I notice is Japan, where the “base” data has a warm Japan (though the way GISS processes it seems to take even that warmth away.)

    If you look at the second graph from the top (the “bias map”) the white unbiased areas are largely the areas with significant population and some money. (Remember that in the years of the baseline China was still, er, having issues…). Even Mediterranean Africa is fine. It’s places where there were few people (and likely very few thermometers) that are yellow and orange.

    Golly.

    And those places are now the ones being most touted as “warming the fastest”. Siberia, Arctic Peninsula, Alaska and the Arctic, and there is this persistent warm blob over south central Africa / Madagascar too..

  15. E.M.Smith says:

    AnthonyH

    As you suspected one of us would, I looked at the 1971-1980 time period to see where the hot area near the Antarctic peninsula comes from.

    Oh I love it when a plan comes together ;-)

    That means that of 30 years in their baseline, 29 of them show no data for that region, and 1 shows data. That data is relatively hot (2-4 deg C) compared to the full data set.

    Gee a 30 year baseline made up of one datum… wonder how often something like that happens. I feel a new project coming on. For each station in the baseline, count the actual number of data items.

    Considering how many ‘nulls’ there are in the baseline, shouldn’t there be an error estimate for the baseline. That red area, with just 1 year of data, should show up as having a very low confidence for the baseline quality.

    You would think so. But that requires a willingness to think…

    I wonder if we could figure out a method to calculate the number of ‘non-null’ months in the baseline for each cell, which would help identify areas that were less usable for comparing to other temps.

    Well, modulo that some stations get dropped at various stages of the GIStemp code, so there would be a little slop depending on what “step” you were benchmarking, it ought not to be that hard.

    Filter GHCN / USHCN.v2 (the v2.mean file or the v2.mean_comb file) for years 1951-1980. For each StationID, count non-null cells by year. I can probably knock together the code to do that in one coding session. But it will be a few days before I could get to it. Then plot the stations by geography for a ‘coverage’ estimate, (The bit I’m not good at… yet ;-)

    Maybe if we plead really nicely vjones and the DB Guys would do it in 10 minutes…. (there are times I’m starting to really regret sticking with the FORTRAN for benchmark validity purposes… )

    But I think its pretty clear that GIStemp is sensitive to sparse data. And that the data are sparse…

  16. vjones says:

    @EM Smith,

    your wish is my command ;-)

    (although we’re finding so much to chase up that there is a queue)

  17. AnthonyH says:

    I never could get the GISTemp sources to work on my Mac (suggestions?). If I do, I’ll have to post how I did it.

    So, I wrote up a Scilab script (like Matlab) to scrape the data from the GISS maps site you linked and add up the number of nulls in each 2 degree by 2 degree cell.

    When I get home this evening, I’ll have it count how many months of data are in that spot during the baseline.

    REPLY: [ See the “GIStemp tab” up top. It points to what I had to do to get it to run. Both the (small) changes to the programs and the fact that you need TWO compilers. One f77 vintage and one f90 (or g95) vintage. -E.M.Smith ]

  18. Harry says:

    FWIW I like the 1931 baseline as well.

    We have meticulous records for the glaciers on Mt Rainier going back to 1931. Tacoma City Water had become concerned that the ‘unprecendented melting’ would effect water supplies.

  19. Roger Sowell says:

    Some CRU data plots for US cities now up on my site; some going back to the 1820’s. These show more shenanigans by warmists with cherry-picking, in particular the start date of 1975 for “warming.” No warming in most cities, (see Abilene, Texas for example), but there were several cold winters around 1980 followed by normal winters afterward. This gave an appearance of a warming trend.

    http://sowellslawblog.blogspot.com/2010/01/cold-winters-created-global-warming.html

  20. Curt says:

    The standard argument that the choice of baseline period is unimportant (and one that I at least used to believe) is that changing baseline period would simply offset any trend graph up or down without changing the characteristics of the trend curve. I have seen many comments on the alarmist blogs implying that anyone who does not believe this is an idiot who does not understand math at the junior high level.

    I think this would be true if we had totally uniform and complete coverage over the entire globe over the entire period of interest. But with all of the infilling, interpolation, and adjustment going on, none of which is uniform over time, I am starting to believe that this is not just wrong, but significantly so.

    It is fundamentally the same argument that some have advanced here that the anomaly method makes the official metrics that we see insensitive to even massive station dropout. It seems like this would be so at first glance, but as you start to consider how all the measurements get interrelated by the adjustment and infill algorithms that occur before anomalies are taken, the waters get far muddier.

  21. E.M.Smith says:

    AnthonyH: …from the GISS maps site you linked and add up the number of nulls in each 2 degree by 2 degree cell.

    When I get home this evening, I’ll have it count how many months of data are in that spot during the baseline.

    Curt: But with all of the infilling, interpolation, and adjustment going on, none of which is uniform over time, I am starting to believe that this is not just wrong, but significantly so.

    It is fundamentally the same argument that some have advanced here that the anomaly method makes the official metrics that we see insensitive to even massive station dropout. It seems like this would be so at first glance, but as you start to consider how all the measurements get interrelated by the adjustment and infill algorithms that occur before anomalies are taken, the waters get far muddier.

    Yup.

    Both of you ought to take a look at what’s being discussed on the Madagascar Thread.
    https://chiefio.wordpress.com/2010/01/31/mysterious-madagascar-muse/

    In digging through the code it turns out that the “annual anomaly” can be based on as few as 3 seasons. And each season only needs to have 2 months in it. And we don’t know how many days NCDC requires to be in a month to make the monthly mean.

    So you can drop out the entire winter, a month on each side of it, and one more random month: And still get an accepted annual anomaly…

    (And that is after all the homogenizing, ‘in filling’, etc.)

    And yes, I keep asking “Where’s the beef in the code?” and folks keep telling me “If we assume a hypothetical cow…”

    Folks love their perfect theoretical, but that is not what the code is doing.

    So keep that in mind when you are looking at cells that contain anomaly ‘data’… you don’t know where it’s been…

  22. AnthonyH says:

    OK. I counted all the monthly data, and it looks like a line of stations on the Antarctic coast started reporting in March 1980. That line of stations is completely responsible for the presence of that cold area in the southwest corner of the map. Apparently, a baseline only requires 3 seasons of data, as well, even out of 30 years.

    If you look at the 2009 anomaly against the 1951 to 1979 baseline, that area is null.

    http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=0112&year1=2009&year2=2009&base1=1951&base2=1979&radius=1200&pol=reg

    If you look in the 250k smoothed data, you can see the horizontal line of those stations, even as late as Dec 2009.

    I’ll look into getting GISTemp compiled and working again, sometime.

    E.M., I’ve seen you talking about creating graphs and charts. Since you’re comfortable around code, you might consider looking at Scilab (http://www.scilab.org), especially if you’re thinking about producing a lot of similar graphs with data that’s produced from some sort of code (sounds like your latitude/altitude data to me). Once you get the base code written, it’s easy to modify it, and Scilab can create PNG or GIF output directly, always exactly the same size, if that’s what you want.

  23. Ruhroh says:

    Well, since I posted a lot of stuff about the baseline over on the Madagascar thread, that arguably belonged here,

    I’ve decided to post something here that may not belong here.

    I’m suddenly wondering about that ‘Estimated Global Mean’ in the upper right corner of the gistemp mapping page.

    Does anyone know whether it has been cos weighted?
    (vs straight average of all gridboxes)?
    RR

  24. AnthonyH says:

    Ruhroh,

    I skimmed through the GISTemp source yesterday, thinking the same thing. The comments clearly state that they weight for area. They also state that they don’t include area of null cells when weighting. There is a weighting factor in the code, but I haven’t checked to see how it’s calculated or used.

    There’s also something in there about calculating averages for regions, and a note that since the NH an SH averages may be based on different non-null areas (at each time step!), the global average will not simply be a straight average of the NH and SH values.

    Seems to me that all those null values could cause the area being sampled to slop around a lot, when the null values enter and leave the calculations. With the area changing so much, I can’t see how it would be possible to calculate a global temperature average and know whether the differences between time steps were due to changes in coverage or changes in temperature.

  25. marchesarosa says:

    Great co-operative effort, lads! You’re steadily amassing the ammunition to bring the climatology establishment down. Well done!

    Who’d’ve thunk it? Baselines, the Achilles heel!

  26. docmartyn says:

    What would be nice is to run a double difference spectra; getting rid of the baseline problem.
    Run a movie of (Y+1)-Y; that is 1961-1960, next image 1962-1961, next image 1963-1962; e.t.c.
    What this will do is get rid of the baseline and show you the change, year on year.
    It will show you the march of the thermometers and ‘funny bits’.

    I have no idea who one makes such a movie BTW, but I know spectroscopy.

  27. e.m.smith says:

    GIStemp weights each station as it is used in a cell based on distance from the center of the cell IIRC. Then the individual cells get averaged together by yet another method…

    From:

    https://chiefio.wordpress.com/2009/03/07/gistemp-step345_zonav/

    which lists the source code we have this comment per zonav.f:

    C**** JBM zonal means are computed first, combining successively
    C**** the appropriate regional data (AR with weight WTR). To remove
    C**** the regional bias, the data of a new region are shifted
    C**** so that the mean over the common period remains unchanged
    C**** after its addition. If that common period is less than
    C**** 20(NCRIT) years, the region is disregarded. To avoid that
    C**** case as much as possible, regions are combined in order of
    C**** the length of their time record. A final shift causes the
    C**** 1951-1980 mean to become zero (for each month).
    C****
    C**** All other means (incl. hemispheric and global means) are
    C**** computed from these zonal means using the same technique.
    C**** NOTE: the weight of a zone may be smaller than its area
    C**** since data-less parts are disregarded; this also causes the
    C**** global mean to be different from the mean of the hemispheric
    C**** means.

    Those wishing to know more can hit the link and read the code.

    FWIW, I’m made a bit nervous by the part about “shifting the mean to be zero” and averaging in another record with shifting of the mean. I haven’t taken the time to work through what the code does and prove / disprove validity, but it looks to me like the first record ‘in the pot’ gets to set a bias to which later records are conformed. It could be valid, but it just smells a bit funny to me…

    A bit further down the same listing is annzon.f where we find the annual mean is calculated 2 different ways! :

    C*********************************************************************
    C *** program reads ZONAL monthly means and recomputes REGIONAL means
    C *** as well as annual means.
    C *** Input file: 10 zonal.means (ZON1977.T1200)

    C****
    C**** Find the annual means
    C****
    DO 100 JZ=1,JZM
    DO 100 IY=1,IYRS
    ANN(IY,JZ)=XBAD
    ANNW(IY,JZ)=0.
    ANNIY=0.
    ANNWIY=0.
    MON=0
    DO 50 M=1,KM
    IF(DATA(M,IY,JZ).EQ.XBAD) GO TO 50
    MON=MON+1
    ANNIY=ANNIY+DATA(M,IY,JZ)
    ANNWIY=ANNWIY+WT(M,IY,JZ)
    50 CONTINUE
    IF(MON.GE.MONMIN) ANN(IY,JZ)=ANNIY/MON
    ANNW(IY,JZ)=ANNWIY/12.

    100 CONTINUE
    C****
    C**** Alternate global mean (from North.lats,Equ.reg,South.lats)
    C****
    IF(IAVGGH.EQ.0) GO TO 180
    IGLB=IAVGGH
    DO 120 IY=1,IYRS
    GLOB=0.
    ANN(IY,JZM)=XBAD
    DO 110 J=1,4
    IF(ANN(IY,JZG(J,IGLB)).EQ.XBAD) GO TO 120
    GLOB=GLOB+ANN(IY,JZG(J,IGLB))*WTSP(J)
    110 CONTINUE
    ANN(IY,JZM)=.1*GLOB
    120 CONTINUE
    DO 130 IY=1,IYRS
    DO 130 M=1,12
    DATA(M,IY,JZM)=XBAD
    GLOB=0.
    DO 125 J=1,4
    IF(DATA(M,IY,JZG(J,IGLB)).EQ.XBAD) GO TO 130
    GLOB=GLOB+DATA(M,IY,JZG(J,IGLB))*WTSP(J)
    125 CONTINUE
    DATA(M,IY,JZM)=.1*GLOB
    130 CONTINUE

    Haven’t dug through it enough to figure out why it wants two ways or if there is a ‘hidden cherry pick’ where they did it both ways, but only one got picked to print / use.

  28. Ruhroh says:

    Wholly Smoke Chief!

    Hmm, maybe I’m confused again.
    This paragraph is written after blasting out the stuff below.
    To what do they refer with the terms ‘Region’ and ‘Regional’ ??? I had assumed it was the process of combining the gridcells, but maybe this is wrong?

    Did you really mean to say that
    “GIStemp weights each station as it is used in a cell based on distance from the center of the cell IIRC. Then the individual cells get averaged together by yet another method…” ?

    Why would distance from the center of the cell imply that the data is less worthy? This seems to be aimed solely at smoothing out the map for eventual colorization that looks nice.

    One other code comment that really hit me;

    “A final shift causes the
    C**** 1951-1980 mean to become zero (for each month).”

    Wow, that seems to be the step where a poorly represented month (within the baseline period for the cell (region? Gridbox?) being thus ‘scrappled’) is awarded full credence for the period.

    If I understand, this ‘shift’ is a 12 element array of offsets, that are the arithmetic inverses of the baseline monthly means, which is then ‘subtracted’ from the period whose anomaly is being calculated.

    Overall, GISTemp seems reasonable for the situation where there is plenty of data. I think that no one has systematically looked for the magnitude of distortions due to physically or temporally sparse data.

    [I take it there is no ‘weighting’ step for ‘data quality’ in any of the various ‘averaging’ steps that are taken by GISTemp.
    What would the ‘data quality function’ look like if ‘we’ were calling the shots?]

    All of the effort to highlight specific glitches seems to be unfortunately rather moot, as far as persuading non-skeptical folks that there is a general problem.
    Uninvolved parties lack the insight to move from ‘isolated’ problems to the general case. The ‘dismissal of anecdotal evidence’ is arguably a healthy tendency, but it is working against ‘us’ here.

    One other vague thought is that all of this math could be represented much more cleanly (while retaining the arcane warts and weirdness) for some math whiz to calculate the expected deviation (distortion?), due to the improperly excessive influence of sparse data cells. What would be the name of this branch of math? In EE realm this would be like ‘excess idle channel noise’ or ‘tape hiss’ or ? Herewith I am only hoping for someone to say the name of the kind of analytical armamentarium we must access to systematically characterize the intrinsic distortions of gistemp, without regard to the particular input data set.
    IF the temperature record was music, and GIStemp was the DSP filter, how would audiophiles characterize the behavior?

    Ok some more rhetorical questions now;

    The entire ‘mohoginization’ step seems mathematically dubious to me. The unstated intent seems to be that ‘urban’ stations are not trustworthy, and they must be made more trustworthy by ‘mohogining’ them according to their ‘rural’ brethren.
    Why not just discard them and only use the ‘trustworthy’ rural stations?
    What is the rationale for including urban records if they must first be made to conform to other records?

    Why are the ‘rural’ records not also subject to being ‘mohoginized’ with each other?

    So much text, so little content…
    RR

  29. e.m.smith says:

    Ruhroh
    Wholly Smoke Chief!

    Hmm, maybe I’m confused again.

    I know, it’s a little like the first time you got really drunk and it was interesting, but you wondered if folks really did this for fun (just before puking your guts out…)

    But this code really does this stuff… even if it does make you feel light headed and like you would like to toss your cookies somewhere…

    So, from STEP 2, we have the code:

    https://chiefio.wordpress.com/2009/03/07/gistemp-step345_tosbbxgrid/

    And what toSBBXgrid.f does is take station data and put it on the ‘grid’ of 80 ‘regions’ that have 100 ‘boxes’ in each grid cell.

    In that program we find the comments:

    C**** The spatial averaging is done as follows:
    C**** Stations within RCRIT km of the grid point P contribute
    C**** to the mean at P with weight 1.- d/1200, (d = distance
    C**** between station and grid point in km). To remove the station
    C**** bias, station data are shifted before combining them with the
    C**** current mean. The shift is such that the means over the time
    C**** period they have in common remains unchanged (individually
    C**** for each month). If that common period is less than 20(NCRIT)
    C**** years, the station is disregarded. To decrease that chance,
    C**** stations are combined successively in order of the length of
    C**** their time record. A final shift then reverses the mean shift
    C**** OR (to get anomalies) causes the 1951-1980 mean to become
    C**** zero for each month.
    C****
    C**** Regional means are computed similarly except that the weight
    C**** of a grid box with valid data is set to its area.
    C**** Separate programs were written to combine regional data in
    C**** the same way, but using the weights saved on unit 11.

    What does that mean? Why is it done that way? Lord only knows. All I can do is point you at the FORTRAN code and say “THIS is what is done”. Why is for a higher pay grade than I’m at…

    I suspect that the station with “pole position” has too much influence on the final result, but I’ve not done the work yet to prove that…

    In any case, you can see that an individual station has influence based on how close it is to the center of the cell. “Why? Don’t ask why. Down that path lies insanity and ruin. -E. M. Smith”

    Did you really mean to say that
    “GIStemp weights each station as it is used in a cell based on distance from the center of the cell IIRC. Then the individual cells get averaged together by yet another method…” ?

    Yes. See above code comments.

    Why would distance from the center of the cell imply that the data is less worthy? This seems to be aimed solely at smoothing out the map for eventual colorization that looks nice.

    “WHY? Don’t ask why. Down that path lies insanity and ruin. -E. M. Smith” So, you were saying??…

    One other code comment that really hit me;

    “A final shift causes the
    C**** 1951-1980 mean to become zero (for each month).”

    Wow, that seems to be the step where a poorly represented month (within the baseline period for the cell (region? Gridbox?) being thus ’scrappled’) is awarded full credence for the period.

    Yes. A couple of times now I’ve raised the issue that it looks to me like the “first cell in” both here and in PApars.f; gets to SET the bias that all other stations are forced iton complying with. But I’ve not followed up on it and nobody else has taken up the point…

    Overall, GISTemp seems reasonable for the situation where there is plenty of data. I think that no one has systematically looked for the magnitude of distortions due to physically or temporally sparse data.

    BINGO! It all looks good on the surface, but you start peeling back the wall paper and you find rotten timbers… And “sparse data” is a 4 foot snow fall on the roof…

    [I take it there is no ‘weighting’ step for ‘data quality’ in any of the various ‘averaging’ steps that are taken by GISTemp.
    What would the ‘data quality function’ look like if ‘we’ were calling the shots?]

    Are you kidding? I’ve seen ABSOLUTELY NO EVIDENCE OF QUALITY CONTROL AT ALL. No test suite. No acceptance criterion. No sanity checking code in the code. Nothing. And the data set is worse. NOTHING checks it but maybe some folks eyeballing it at NCDC… maybe.

    some math whiz to calculate the expected deviation (distortion?), due to the improperly excessive influence of sparse data cells. What would be the name of this branch of math? In EE realm this would be like ‘excess idle channel noise’ or ‘tape hiss’ or ?

    I think it would take a pretty good mathematician of the statistical analysis type, Someone versed in stats and both inductive and deductive reasoning ought to do it.

    .
    IF the temperature record was music, and GIStemp was the DSP filter, how would audiophiles characterize the behavior?

    IMHO it would be: “High Impedance Head Phones”. It makes a lot of sound from not very much signal, but the sound is very distorted from reality (and from the input) and with poor fidelity; but it sounds like and impressive name…

    The entire ‘mohoginization’ step seems mathematically dubious to me.

    BINGO! again. It is largely smoke and mirrors, IMHO. It is taking what you have and making from it what you can, knowing all the time that you are trying to turn an old leather glove into dinner for 12 and you just can’t infill and stretch that far…

    What is the rationale for including urban records if they must first be made to conform to other records?

    As near as I can tell, it all comes from the notion that there is very little data, so you need to use it all; and then some. So stretch, in fill, fabricate, interpolate, whatever. Just at the end of the day have a number in all the buckets you need; wherever it came from…

    Why are the ‘rural’ records not also subject to being ‘mohoginized’ with each other?

    What makes you think they are not?…

  30. DirkH says:

    ChiefIO, have you seen this?
    http://crapstats.wordpress.com/2010/01/21/global-warming-%e2%80%93-who-knows-we-all-care/
    Very simple elimination of the thermometer death effect IMHO, with stunning results.

  31. DirkH says:

    “It all looks a bit cold to me, except for that one really hot spot in Antarctica. Oddly, right next to is a Very Cold Antarctic Peninsula. That cold baseline goes a long way toward explaining the “hot peninsula” stories of the last few decades.”

    ChifIO, could it be that the warm/cold pattern in the south pacific in the 1950-1980 baseline is an artefact produced by El Ninos during that time? See Bob Tisdales comment here:
    http://wattsupwiththat.com/2010/02/05/spencer-record-january-warmth-is-mostly-sea/#more-16097
    Bob Tisdale (17:44:08) :

    “The big red blob (scientific term) in the South Pacific is a common response during El Nino events.
    http://bobtisdale.blogspot.com/2010/01/south-pacific-hot-spot.html

    It also occurs at other times.

  32. Ruhroh says:

    Sparseness Metrics and the Disproportionate Role of Lonely Stations

    OK, Cheif, here’s another big bowl of brain salad;
    it’s your job to pick out the non-soggy croutons
    whilst avoiding the brown bits of lettuce.

    Perhaps those Wizards of graphical presentation can make some of those cool Geo-referenced maps of Data Sparsity….
    where the each record is blue if long and red if short.
    Perhaps a second map would plot the inverse of some metric such as cumulative duration of records within that cell or region, after having paid attention to the 1/0 issue.

    We always see plots of Stations. I don’t remember ever having seen a plot of ‘non-stations’ except on those now-hidden comparison-to-identical-period-as-baseline graphs.

    Regarding the disproportionate influence of the lonely stations, consider the following;

    1. Create a flat-line test version of GHCN data by plugging 10.00000 (hehe) degrees into each element of each record that has a non-invalid data point.

    2. Run GIStemp to observe 0 GAT anomaly for any choices of baseline period and comparison period.

    3. Pick a (cell, region, station, year of a station, whatever…)
    and set each non-invalid element to another temperature, say 1000.00 degrees (chosen to avoid underflow/overflow),
    (also chosen to avoid being rejected by mohoginizer,)

    4 Rerun GIStemp to observe the net effect of this perturbation.

    5 Repeat steps 3 and 4 for (initially judiciously chosen stations, eventually randomly chosen or) all stations.
    Probably this is a good job for the folks running the ported versions after doing spot validations of single stations.

    6. A grand map of the ‘sensitivity’ of the reported GAT anomaly to the particular region

    OK, already I see a problem; the anomilization process might well wipe out simple offsets. Maybe the smart fellers in the audience can fix this by proposing a slightly modified version that has the proper mean value and other traits to survive the process, and probe the system as intended, in a way to allow characterization by the ratio of the perturbation/response.

    The virtue of this approach is that no changes are needed to GIStemp, and the spatial/temporal disturbution of GHCN records in the test case is identical to the ‘real world as we think we know it’…

    I remember the CCC crew asking for guidance about ‘test cases’, so this might be a valuable way to get them started, at first by single hand-checked examples.
    Obviously a lot of work being proposed here, but I think it would move us toward some very compelling graphix produced by UnModified GIStemp, that could have the same ‘apparent credibility’ as the nasa version, by virtue of visual identicality. (Not to mention the intrinsic credibility of complex computer-generated printout [vs. the same stuff done with paper/pencil/sliderule]).

    Remember now, don’t bother with the soggy croutons!

    RR

  33. E.M.Smith says:

    DirkH
    ChiefIO, have you seen this?
    http://crapstats.wordpress.com/2010/01/21/global-warming-%e2%80%93-who-knows-we-all-care/
    Very simple elimination of the thermometer death effect IMHO, with stunning results.

    Folks, hit that link! A VERY well done clear and simple presentation. (It does have one ‘naughty word’ in the text, so not ‘family friendly’… send the kids to the next room and open a beer first ;-)

  34. e.m.smith

    “The selection of a base period wil not impact the rate of warming seen.

    Um, I think that is an assumption stated as a conclusion… IFF my baseline is, say, 1720 when temps were the same as today, I would see no warming today, so no rate of warming, but I would see a cool 1880.”

    Wrong. You’ve misinterpreted my rather simple statement.

    Create a Time series. Select any period you like as the period over which to average. The base period. Subtract this constant from the raw series. Now pick any segment of that anomaly series. Compute the trend.
    Repeat picking some other arbitrary segment as the base period.

    The anomaly period selected does not change trends.

  35. EM the baseline does not change the rate of warming.

    Take a series t1,t2,t3,tn

    Select any start point you like and end point. Calculate the trend.
    the rate of warming ( or cooling if negative)

    NOW,

    Take that same series and subtract a constant from every term.
    ( hint the constant is the average of the base period)

    Now compute the same trend as above.

    Its simple algebra.

  36. E.M.Smith says:

    @Steven Mosher:

    Ah, I think we’re saying two different things.

    You are saying that from -5 to zero is the same as from zero to plus 5. Both are a +5 trend. I’m saying that if you have measured your delta from 0 to +5 you will get all excited about calling it “warming”, but if you look further back and set your baseline on 1930-1940 you have, instead, a (hypothetical) -5 in the 1970 area, and would call that a ‘cold’ 1970 and not a ‘hot’ 2009. You are speaking to the slope, I’m speaking to the interpretation. (Which then gets more interesting if you extend back to 1720 which was also the same as now, so no net warming at all from starting with a very old baseline. That is, changing the length of the data set. Which by your POV would be not kosher as it is changing the total slope.)

  37. Ruhroh says:

    @mosh;

    R U implying a constant trend dT/dt over all possible baseline periods? If yes, then your assertion might be correct. ( when calculated from (center of baseline period) to (center of compared period).

    Ex. 1; temperature decreases 1deg/year for 50 years starting 1900-1949, then increases 1 deg/year for 50 years, 1950-1999. If baseline is 1950-1959 and comparison period is 1998-1999, dT/dt would be 44 degrees/44 years. OK so far.

    Ex.2; Same Temp record as (1). Compare 1998-1999 to baseline of 1940-1949. dT/dt is now 44 degrees/54 years.

    Ex 3; Same Temp record as (1). Compare 1999 to baseline of 1900-1909. dT/dt is 4/94.

    Other than some possible minor math errors, am I still ‘wrong’ in understanding your assertions?
    RR

  38. boballab says:

    I think I laid out what both Mosh and EM are driving at in this post I made here:
    http://boballab.wordpress.com/2010/02/08/anomaly-baslines-do-they-matter/

  39. Ruhroh

    The easiest way for me to explain this is with a synthetic
    data series.

    Let your series of numbers t1,t2,t3,t4, t5, etc

    be this:

    1,2,3,4,5,4,3,2,1

    Or pick any series of numbers you like.

    Now pick a baseline period. Lets pick the first two years.
    Average them and you get 1.5.

    Subtract:

    -.5,.5,1.5,2.5,3.5,2.5,1.5,-.5

    Now pick a period over which to calculate a trend.

    Any period will do. lets pick the last 5. Its decreased 4 in 5 years. lets pick the entire series. oh, its decreased 0 in the full period.

    Go pick another base period. repeat.
    look at the original series. repeat.

    It doesnt matter what you do because you are only subtracting a constant. Do the algebra if you like and prove that its mathemtically CERTAIN. the baseline you pick has NO effect on any trend you wish to report for that series. NONE.

  40. EM

    “You are saying that from -5 to zero is the same as from zero to plus 5. Both are a +5 trend. ”

    Yes. Pick whatever baseline you want for a GIVEN series.
    Calculate whateverr trend you like within that series.
    The selction of baseline periods is immaterial to the reported slope. And trend is what we are concerned about.

    “I’m saying that if you have measured your delta from 0 to +5 you will get all excited about calling it “warming”, but if you look further back and set your baseline on 1930-1940 you have, instead, a (hypothetical) -5 in the 1970 area, and would call that a ‘cold’ 1970 and not a ‘hot’ 2009. “You are speaking to the slope, I’m speaking to the interpretation. ”

    Since slope is metric that matters, since slope is the metric that the IPCC reports out, nothing else matters. To be sure
    some people speak loosely or incompletely when using the term warming or the term cooling. Warming relative to what
    period? But since the baseline periods are always reported
    only the confused get confused.

    “(Which then gets more interesting if you extend back to 1720 which was also the same as now, so no net warming at all from starting with a very old baseline. That is, changing the length of the data set. Which by your POV would be not kosher as it is changing the total slope.)”

    There is hardly enough data going back to 1720 to conclude anything about a global temperature index that would not be conditioned by some heavy caveats WRT certainty. However, I remain open to looking at actual temperature readings from that year. Not proxies, not documentary evidence, but actual thermometer readings.

    The only relevant data for the 1720-1750 period That I can find are 4 european stations. In 1720 the average of these 4 stations was roughly -.2C ( relative to a 1961-1990 baseline)
    those same 4 stations average out to around .9C today.
    So since 1720 those 4 stations have increased roughly 1C

  41. Ruhroh says:

    OK, I think I understand what you are saying.

    Your approach to measuring trends is to look at dT/dt after subtracting a constant from all values of T in a sequence.
    Said again differently, you want to look at the differences between 2 periods after subtracting the ‘average’ from the baseline period from all data in the record.

    Yes?

    Your approach would certainly have merit in a rational world.
    (in those situations where disparate records require some kind of offset management prior to subsequent calculations.)

    However, the first graphic in this thread is a comparison of Dec-Nov 2009 vs. the Gistemp baseline period of 1951-1980. Note the calculated overall mean ‘anomaly’ is 0.68 degree.

    Whereas, the last graphic ‘ For Boballab’ ( sounds like a song title…) is a comparison of Dec-Nov 2009 vs. the Gistemp baseline period of 1931-1990. Note the calculated overall mean ‘anomaly’ is 0.63 degree.

    This seems to be a prima facie counterexample, that your mathematical formulation is not in use with this live version of Gistemp.

    Note this key sentence from the Gistemp definitions;

    “Anomalies: Mean temperature (°C) averaged over a specified mean period and time interval relative to a given base period.”

    So, by definition, they are not looking at the difference between 2 comparison periods after subtracting a baseline;
    They are simply comparing a specified period with a baseline.

    Was there a hint in those climategate emails on which you did such heroic duty; was there some discussion that changing the baseline period should be avoided, to avoid ‘confusion’ of changes in the tale…?

    RR

  42. Ruhroh says:

    Mr. Mosh;

    Ah, I’ve never noticed the Cheif to be offering analyses of IPCC machinations.

    This site seems to be the clearest exposition of the guts of GIStemp, and the discussion herein is implicitly related to that arena.

    I’m not aware of the optimal forum for IPCC analyses of temperature record.

    I think the point of this forum is to examine the numerous questionable steps that ‘underpin’ the purported ‘temperature FACTS’ on which the IPCC aerial castles rely.

    As a simple example, the ongoing, non-explained changes to NCDC ‘raw’ temperature history call the whole stinkin’ mess into question, as far as I am concerned.

    And the hash-function ‘homoginization’ of the record by Gistemp seems to be a world-class distortion function, that can scarcely be characterized in 25 words or less.

    So, your arguments have some merit, but their relevance is unclear to me anyway.

    RR

  43. boballab says:

    Ruh Roh I have a post on my site about what Mosh is saying and I posted it here but its still sitting in the “waiting for Mod” folder. If you mosey on over to my post you will see what Mosh is talking about and also how the Climategate emails apply.

  44. ruhroh,

    .68 is the difference between the current year and the 50-81 baseline.

    Change the baseline to a different period (31-900 and you change the math.

    WHY? BECAUSE you are subtracting a different constant.

    The fight about GISS is on a different battlefield.
    Now you may like this battlefield, but its a wheel that doesn’t turn in the debate.

  45. E.M.Smith says:

    @Steve Mosher:

    It turns the perceptual debate.

    The baseline serves 2 purposes. The one you describe (slope up from 1951-80 average point, which does not change). AND the absolute color applied to the graph. (As shown in the examples above). Both matter.

    Further, the “interesting bit” to me was discovering that the chosen baseline is very much non-robust. The two regions in Antarctica show that. The data are sparse enough to put artifacts in the maps from the baseline chosen.

    So to make it more robust, we need a longer baseline with more data in it. And to avoid a “color code cherry pick” it needs to have both hot and cold phases of the PDO in it. If you put your baseline at the bottom of a cold period, as GIStemp does, you always measure slope up hill from it. (zero to plus 5 ) and can never have a nice deep blue graph ( -5 to zero) … Perception matters.

    Basically, both the ‘rate of change’ and the ‘position relative to a static past point’ matter; but for very different reasons.

    I’ve just run into this on a little effort I was working through today – while the mod queue was untended…

    I’ve now got a program that, for each thermometer in the data set, does a dT/dt form of processing. It ignores missing values and just sets the start of time for each thermometer in each month as the first valid value. From that point onward, each valid value is turned into a delta Temp / delta time.

    Running it has been enlightening (and the results will be in postings ‘soon’). But one thing I did run into was that for some thermometers with short histories and especially those with holely data, the ‘accidental baseline’ of that first value can have the whole series end up being a positive or a negative ‘anomaly’ or ‘offset’ from that start point. (If, for example, the ‘first year’ was 5 high, then all subsequent readings become negative numbers until you accumulate 5 degrees of ‘warming’, which may be never).

    It works reasonably well with more ‘rich’ data series and it produces ‘reasonable’ results on larger masses of data, but on single or low numbers of thermometers with very divergent start dates there can ‘be issues’. Yes, they are only perceptual issues. But perception matters.

    (This is a variation on a “Constant Sample” method. As soon as you get two values in a row, you can start turning everything into offset / delta values and then they can be validly averaged. The only real wrinkle is that I let the 2 first values be more than one year apart so that there is a bit more sensitivity to original state, but you don’t through out several years worth of data if it has some holes in it.)

    So while it is a valid approach, it can present perceptual ‘issues’.

    I’ve also considered using the average of all values for a thermometer in a month as the baseline for that thermometer. This has the virtue of less sensitivity to the random act of the first reading, but is more complex to program. (And while I think it makes a better more representative baseline, it is not as well attested as the “constant sample” method for data with missing bits – at least in the few hours time I spent on it.)

    Finally, I’m planning to ‘invert the direction’ and try a run with looking backward in time. (the latest datum vs just prior will set the start of the baseline, with the past re-written. I’m less fond of this because each new month changes all past values… but the greater ‘trust’ in modern readings would argue for a more reasonable perception and fewer cases where the whole series is offset by 4 for no reason…

    (That is, the odds of being -4 degrees for most of your readings ought to be lower, since in 18xx someone might have gotten a more bogus reading and not really noticed and you might accidentally hit it as the ‘start of baseline’ moment. And I have… while the ‘today’ value ought to have less of that issue… with the result that the ‘near now’ history would look less ‘scary’ than when it’s full of larger ‘anomalies’ that are really just reflecting that one early ‘wrong’ value. The same thing hitting GISS with their ‘one temp in Antarctica’ blue blob… )

    At any rate, the choice of baseline has an impact on a lot of the processing, results and perception. And the GIStemp baseline has both data sparseness issues and duration issues that impact the results (blue blob from 1 data point, always hot absolute values from cold bottom being ‘zero’)

    FWIW, the dT/dt reports generally show no warming to speak of in places with modest economic growth / change. A lot of the “warming” is concentrated in times and places of economic growth. So the Pacific Basin and South America generally show little “warming” as does Africa. But Chinese Urban and Airport areas are warming nicely…

    I did run into an odd case in Canada that will end up another posting. A “hot spot” in Saskatchewan. Chased it back to a couple of thermometers that start in a very cold individual year. So a couple of ‘odd balls’ can put a 5 C ‘warm spot’ on the map while all their neighbors are just dead flat. (One case had a thermometer come back in 2005 after being gone since 1945 or so. January and Feb had 18 C hot anomalies while the other months are normal (near zero / single digits) with that one spot making the annual running total anomaly jump a couple of degrees. (and then bleeding into the region…)

    So was there ‘something wrong’ with the data? Or was 1919 really -22 and 2006 really -4 for a +18 C anomaly for that individual thermometer for that month? Hard to know …

    But the end result is that this ‘baseline’ effect warms a chunk of rural Canada and does it with a concentration in the ‘really rural stations”. (I can now make reports filtered for: with / without the A airport flag and the USR Urban Suburban Rural flag.) It is very ‘odd’ to find the ‘really rural R no A’ report showing much faster ‘warming’ than anything else around it.

    And while that fact showed up partly due to the kind of ‘baseline’ done in this report, the same data ‘issues’ will show up for that region against other baselines chosen and with other processing. So it, too, will be influenced by what other baseline is chosen… Longer baselines and those with more data items in them will mute the effect (as would simply tossing out the 2005-2010 segment when it “comes back” after being gone for a few decades… so ‘keepers and tossers’ will impact the baseline that impacts the presentation too…)

    And I could simply have a missing thermometer that is ‘out of service’ for 5 years or so reset the start of the ‘constant sample’. (Basically, stay closer to the true definition of ‘constant sample’ with 2 adjacent samples and only allow a limited gap; instead of letting my ‘constant sample’ turn into an ‘any gap will do’ sample… I had not counted on the same thermometer returning after so many years away… so I need to measure how many such cases there are and make a decision about it…)

    So the bottom line is that choice of ‘baseline’ is very much an important thing, and has significant impact on results and presentation. It isn’t just a ‘pick one and ignore it’ …

  46. Ruhroh says:

    @mosh

    OK, so now you say that changing the baseperiod will change the anomaly. On this point, we are all in violent agreement. Indeed, I thought that was the whole point of the post by Cheifio

    The rest of your point seems to revolve around the axle of Trends Vs. Anomaly_to_a_baseperiod . and the ‘proper’ way to calculate a trend, and this seems to be ~ the 3rd or 4th wrap…

    It does not seem to be your style to acknowledge the non-erroneous fraction of a post, but I hope you will tell me if I’m getting closer to guessing your point;

    The TRUE TREND between two time_periods is to divide the Delta Temp by delta Time.

    The only way I see to do that with GISTemp is to calculate the anomalies (vs. ‘appropriate’ baseperiod) for each Time Period, and then point_by_point-subtract the two anomaly maps to get the Delta Temp, and then divide that by the temporal difference of the midpoints of the two time_periods.

    In an ideal world of spatially complete (1:1:1 correspondence on an element-by-element basis) data sets, this should give the same dT/dt between two time_periods, for ‘any’ baseperiod if implemented properly. However, with incomplete data, this difference of differences seems to be a ticket to trouble.

    I am unable to find that option within GISTemp. It insists on calling one of the timeperiods a baseperiod.

    What is your perception of the general public usage of the ‘anomaly vs. base period’ maps generated by GISTemp and similar programs? The lurid reds seem to be frequently represented in my limited exposure.

    Perhaps your point is that the Trends in ‘GAT’ are a separate calculation than the GISTemp reported numbers (in the upper right above the graphic.

    So, what exactly was your point? I’m unable to discern it if it has not been covered above.
    RR

  47. E.M.Smith says:

    @Ruhroh:

    Here is my attempt at dT/dt (which seems to indicate we’re both looking at the world the same way…):

    https://chiefio.wordpress.com/2010/02/09/dtdt-agw-ddt/

    which is about as close as I could get to taking the whole basket of “baseline’ issues and setting them aside.

    There may be better ways, but frankly, this shows no warming for the entire Pacific Region… I think I can live with that ;-)

    The “hot spots” that I’ve found have, by and large, been related to station issues. A cluster of them look to be measuring temperatures in Canada just after they became an oil country and started having fuel for warming the place up ;-) “Hey, If ya’ caunt moove to Flooreda, ya’ can warm th’ place up ‘ere a bit! Eh?” 8-)

    But seriously, it does look like ‘bad stations” and “station change” are big issues. CO2 (smooth action over time distributed around the planet) just don’t even show up…

    And taking the ‘pick a baseline’ issues out of the method seem to be very helpful.

Comments are closed.