GIStemp – who needs Antarctic data or temps near ice.

I finally got GIStemp source code loaded into the Qemu SPARC emulator, unpacked it, and did a preliminary assessment.

First off, it is way out of date. It looks like it is a 2011 release from just prior to a major revision. So no source code for 1/3 of a decade or so AND a major revision. Not exactly caring about that whole ‘currency’ thing…

Next up, in looking at the things that did change, I noticed some things. First off, it points to a link for the Antarctic data that no longer works. Did it work in 2011? Who knows. But now it has a nice glitzy interactive web site that lets you download bits and pieces and look at bits and pieces… but not grab the data sets that are used as input to GIStemp. This means that either they have a very different way of getting the Antarctic data, or they just don’t bother to use it any more.

In this download, the same source is used as in the older version described here:

http://chiefio.wordpress.com/2009/02/24/gistemp-start/

Except that site now has a user friendly interface, not somewhere that you can download the data as a file.

http://www.antarctica.ac.uk/met/READER/surface/stationpt.html

http://www.antarctica.ac.uk/met/READER/temperature.html

http://www.antarctica.ac.uk/met/READER/aws/awspt.html

So look around at the interface on those pages. You can ‘hunt and peck’ for a station detail, on a point in time basis. Download the data set? Not so much… Then again, notice on the ‘temperatures’ page linked above; the last update date:

“Last modified : Friday 6 February 2009″

So… just what Antarctic data ARE used by GIStemp? Data, we don’t need no stieenking data! (A reference to a movie with the line “Badges? Badges! We don’t need no steenking badges!” give by some Mexican banditos pretending to be Federal agents.) Now this isn’t as bad as it looks ( or maybe it is worse…) since most of those stations don’t have any modern data anyway. Click on a few. Couple of decades for many. Often from disjoint moments in time. So again: Just what Antarctic data are used in GIStemp? Say, from this millennium?

A screen capture of the listing of the input_files directory where the Antarctic data is bundled in with GIStemp sources is somewhat revealing.

GIStemp input_files datestamps

GIStemp input_files datestamps

Notice that the Antarctic data included in the download are from ‘about’ the same vintage as the source code. 2011 is when I think they last updated. Some of the antarc data files are 2011, some 2010, some 2009… though the 2009 is marked ‘old’. So when did the Antarctic site break the download? Has there been any update since? Does GIStemp even use any Antarctic data anymore? Who knows.

But wait, there’s more…

Since GIStemp smears data via ‘homogenizing’ to places where it has no data, from places where it does have data, and since those places can be up to 1200 km away (though it does this smear in three different steps, so really might be ‘smearing a smear’ from up to 3600 km away) it can just fill in the temp data using ‘nearby’ stations. In the final steps GIStemp adds in the Hadley Sea Surface Temperature. So what is wrong with using cold temps from near the ice to fill in over the ice? Well… How about if you decide to just not use temperatures from places where the water is near the ice?

Remember that you can click on the images to make them larger and easier to read. This is an interesting bit of code. It is in STEP4 and the whole listing (minus the added bit described in a panel below) is in this posting:

http://chiefio.wordpress.com/2009/03/07/gistemp-step4_convhadr2/

In looking at the date stamps in STEP4_5, I noticed that it had changed. Why? I wondered. It isn’t a USHCN related step, and doesn’t depend on USHCN.V1 vs V2 changes (like the rest of the changed files).

GIStemp 3vs4_5 datestamps

GIStemp 3vs4_5 datestamps

Notice that this program is the only one with a 2011 date stamp. Everything else is quite old. So what changed? First off, to get oriented, let’s look at this part from the top of the code. (I downloaded this copy of the sources today, so this is as ‘fresh’ as it gets). In particular, we are looking for the meaning of three variables that are used in a very small fragment of code that was inserted into this program in 2011. They are I, J, and M. Look at the comments about lines 9, 10. They state that I is longitude and J is latitude. (You can also see where they are dimensioned and I gets 1-360 while J gets 1-180 – so 360 degrees of longitude and from -90 to +90 mapped onto 1 to 180). M gets the temperature for that location for any given month.

convert1.HadR2_mod4.f  top

convert1.HadR2_mod4.f top

Why focus on those variables? Because this code was added. It says to mark any cell that is anywhere in longitude ( from 1 to 360 degrees around the globe ) that are in the range near the pole, with ‘bad’, meaning to toss out the data. Just don’t use it. Notice the comment.

GISTemp throwing out data from water near the ice at the pole

GISTemp throwing out data from water near the ice at the pole

Yes, you read that right. The comment says:

“Skip some regions where SST is impacted by nearby ice floats”

That 166 to 180 loop says that anything from 76 N to 90 N is to have the data tossed out.

UPDATE:

From http://data.giss.nasa.gov/pub/gistemp/README.txt

I found a description of which way the arrays run. I’d assumed N to S, but they run S to N, so the above line has been changed from 76 S to 76 N. So it is the N. Pole (where there is more water data anyway) that has the SST data tossed.

The output GRID is rectangular (i:West to East, j:South to North),
where the pole boxes might be of a different latitudinal size than
the other boxes (but North and South pole boxes having the same size).
Examples:
if offI=0, Western edges of i=1 boxes lie on the international date line;
if offI=-.5, centers of i=1 boxes lie on the international date line;
if dlat=180./JM, all boxes have the same latitudinal extent;
if dlat=180./(JM-1), pole boxes are half boxes.

Though notice that there are some selections possible.

So any Arctic water temperatures can only come from places closer to the equator, and the Antarctic itself has very few currently reporting stations; and that data may or may not actually be making it into GIStemp.
END UPDATE.

Perhaps that is why, when we have an all time record ice area in the Antarctic, with record cold temperatures being recorded (by others…), GIStemp thinks it looks like this:

GIStemp polar view, May to October, 2013

GIStemp polar view, May to October, 2013

This is the May to October period, so when it is cold at the South Pole. BTW, I have a series of sea surface graphs from that period, all showing cold water anomalies. Nowhere to be seen in GIStemp. But that is for another day to sort out. IMHO, this image is showing GISTemp having a Jump The Shark moment.

In Conclusion

I’ve got some more on GIStemp, but that will wait for another posting. The big lumps are just that the source code is out of date, the data looks like it is missing a chunk from ‘down south’, and in 2011 the code was modified to avoid water temps from near ice. One can only wonder what possible rationalization could exist for that change. IMHO, it is totally unwarranted by any means. Might as well just start dropping any thermometers in cold places…

Subscribe to feed

About these ads

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in AGW and GIStemp Issues, GISStemp Technical and Source Code and tagged , , , , . Bookmark the permalink.

33 Responses to GIStemp – who needs Antarctic data or temps near ice.

  1. Robertv says:

    The moment government took over science , science you can trust in stopped to exist. Government only needs science to gain more power to control population with fear. Only those get funding who tell the population that you need more (big) government.

  2. A C Osborn says:

    What a find, they just don’t care do they.
    This needs more exposure.

  3. Robertv says:

    2013 anomaly vs 1951 – 1980 ?

    Were we not in a cooling trend those days ?

  4. philjourdan says:

    Several others have been poking around and have found similar shenanigans. The “steenk” is just getting worse.

  5. hifast says:

    Nicely done, thank you.

  6. ilma630 says:

    Having worked for a data/analytics company, the data collection, handling and archiving looks distinctly amateurish at best. It would never pass any sort of audit in the real world, and based on that quality assessment, it would gain ZERO customers.

  7. Ralph B says:

    What year was it that the Antarctic warming paper was blasted to smithereens? Looks like that is the time frame they stopped updating data. If it supports “The Cause” update, if it doesn’t…obfuscate

  8. Peter says:

    Since the data sets which do include Arctic temperatures say that the Arctic is getting warmer, perhaps leaving out the sea containing Arctic ice floes is why GISTEMP shows a lower temperature than the others.

  9. tom0mason says:

    E.M. thanks for the heads-up.
    Wow, GISS, what are they playing at, certainly they never thought they would be exposed.
    GISS = Guessed Input Sample Synthesizer ?

  10. Steve C says:

    As Charlie Brown said, “Good Grief!”. Now you put it in front of us, it’s as obvious as Basic 101, whether you speak the language or not. It’s certainly one heck of an Easter egg for those of an AGW disposition.

    Makes you wonder just how many more of these little logic bombs are out there, tucked away in climate data sources and models alike.

  11. Dmh says:

    Now I know why the “scientific consensus” is saying that the icecaps at the poles are melting (and fast!), with all that heat of GIStemp charts that’s exactly what *would be* happening… if it was true.
    But, we all know that neither the heat nor the melting are happening.
    As you said Chiefio, the oceans around Antarctica have been cooling since (at least) 2011, meaning *negative anomalies*, not the red hot ones shown in GIStemp. To have the real trend, as shown by other sources, you’d have to replace the “white” and “yellow” regions by “blue” and “dark blue”.
    I also have a series of charts from UNISYS showing the cooling trend in the SST of the S. Pole.
    We already knew that GIStemp was wrong, now we know why.
    Thanks Chiefio, great find!

  12. A scoop: The french Antarctic scientific station Dumont-d’Urville just recorded the june 2014 month as the coldest month ever since the station was established in 1956.

  13. NikFromNYC says:

    The silliest situation is how NASA’s main climate office ignores NASA satellite data except to estimate urban heating via nightime illumination. Might they estimate urban heating with satellite *temperature* data instead?! It would really worry them that actual satellite temperature plots are maintained by outspoken climate alarm skeptics, were the GISS guys not clear sociopaths. Now the alarmist team has started citing amateur hour Cowtan & Way’s similar use of satellite data to smear ground data out onto the Arctic to turn the recent lull into a new peak in a way that the satellite data itself simply falsifies. What’s most frustrating is the straight faced seriousness with which they present this crap to the media and to policy makers.

    “I have no idea how one deals with this– to be candid, McIntyre or Watts in handcuffs is probably the only thing that will slow things down.” – Robert Way in the exposed secret forum of John Cook’s site.

    -=NikFromNYC=-, Ph.D. in carbon chemistry (Columbia/Harvard)

  14. NikFromNYC says:

    Jacques, I see the black box at Berkekey BEST fixed your claimed record low by using a station move and alleged neighboring stations to suddenly boost the last decade by about half a degree:

    http://berkeleyearth.lbl.gov/stations/151563

  15. Pingback: NCEP Data Show June 2014 Among The Coldest This Century! Four Of Five Coldest In The Last 5 Years

  16. Bob Tisdale says:

    Thanks, Chiefio I’ll have to link this post the next time someone want to argue with me about GISS deleting SST data in the polar oceans.

    Cheers.

  17. Where exactly did you download your Gistemp source from? Download the tar.gz file from http://data.giss.nasa.gov/gistemp/sources_v3/

    What exactly is the link for the Antarctic data that no longer works? I have had no difficulty downloading current data. Your input listing shows no sign of the Byrd data either – see below and see the documentation link on the same page as the tar.gz file, and see http://data.giss.nasa.gov/gistemp/updates_v3/

    See http://data.giss.nasa.gov/gistemp/station_data/v3.mean_GISS_homogenized.txt.gz to check that the Antarctic stations are being updated to match the download sources. The GISS source download does not get updated monthly to match the latest data uses, although a mirror, as once existed, would be useful, and an archive too, as also once existed. The source is updated when code changes, not data.

    I’ve verified STEP0 through STEP2 for v3 against my own implementation by comparing v3.mean_GISS_homogenized.txt with my own implementation. Minor discrepancies due to platform dependent rounding to 0.1C only.

    Here is STEP0\input_files as I downloaded it on February 6th:

    Directory of C:\Climate\MyGistemp\Intermediates\2014_01\STEP0\input_files

    2014-02-06 03:10 .
    2014-02-06 03:10 ..
    2013-06-04 21:38 1,080 add_in_byrd.ksh
    2008-05-13 21:15 3,337 antarc1.list
    2013-06-04 21:26 236,714 antarc1.txt
    2005-02-15 19:03 2,485 antarc2.list
    2010-09-10 13:36 228,791 antarc2.txt
    2010-09-12 22:18 228,783 antarc2.txt_orig
    2009-08-10 21:00 4,757 antarc3.list
    2013-06-04 21:26 174,010 antarc3.txt
    2013-05-06 18:59 5,162 byrd.txt
    2013-06-04 20:32 1,003 byrd_reformat.f
    2009-11-14 16:23 512 do_corr_id2
    2008-06-09 17:30 694 do_sort
    2011-04-19 14:46 24,360 hohp_a
    2011-04-19 14:46 26,796 hohp_u
    2009-03-09 21:45 257 id_corrections
    2013-06-04 21:32 832 preliminary_manual_steps.txt
    2009-11-02 17:48 266 reduce_strange.sh
    2008-02-29 00:47 813 sorts.f
    2012-02-11 16:59 2,675 Ts.strange.v3.list.IN_full
    2007-08-20 17:23 18,995 t_hohenpeissenberg_200306.txt_as_received_July17_2003
    20 File(s) 962,322 bytes

  18. E.M.Smith says:

    @Peter O’Neil:

    That’s an interesting link. the V3 bit. It looks like they have the old link up here:

    http://data.giss.nasa.gov/gistemp/sources/

    I went to where they had always put the sources. There.

    Interesting to note that some search engines only return that like I used, but Google returns your choice of two links at the top level:

    Search Results

    Data.GISS: GISS Surface Temperature Analysis: Sources
    data.giss.nasa.gov/gistemp/sources/
    Goddard Institute for Space Studies
    Sep 10, 2010 – Sources. Source code and documentation for GISTEMP software is available here. The programs are intended for installation and use on a …

    GISTEMP: Sources Documentation – Data and Images – Nasa
    data.giss.nasa.gov/gistemp/sources/gis…
    Goddard Institute for Space Studies
    Dec 14, 2011 – USHCN and SCAR contain single source reports but in different formats/units and with different or no identification numbers. For USHCN, the …

    So a search on “Gistemp sources” on the most widely used search engine takes you to two links that both go to the ‘old’ sources…

    First one: http://data.giss.nasa.gov/gistemp/sources/
    Second one: http://data.giss.nasa.gov/gistemp/sources/gistemp.html

    OK… starting from: http://data.giss.nasa.gov/gistemp/ doesn’t say where to get sources… but clicking on lots of stuff eventually finds that “current analysis” links to the v3 stuff.
    http://data.giss.nasa.gov/gistemp/sources_v3/gistemp.html as a descriptive page… but not the actual sources.

    though if you prune the link back to: http://data.giss.nasa.gov/gistemp/sources_v3/ it shows up…

    Is there some simple way to find out that sources_v3 exists?

    So both are up, eh? I presume you got to the v3 stuff from some indirect navigation from their top page?

    OK, so the new stuff is up, but so is the old stuff…

    I’ll go get the new stuff and diff it as well, then.

    If, in the new code, they have directions to some new Antarctic data source, that would be nice to know. It isn’t available as a file download at the prior links.

  19. E.M.Smith says:

    The antarctic links lead to the data, but they are in a display format, not a download bundle. Yes, I can transcribe them, but that’s not what the program says… It looks like the historic data are in the ‘temperature.html’ and end some while back, while more recent data are in the ‘aws’ link. So there is current data; but how to get it into the program is a bit unclear at present. (Maybe this weekend I can work on it again…)

    The real ‘surprise’ to me is that the v3 source code looks like it is only linked from the GIStemp page in a link that says ‘current analysis’ rather than ‘v3 source code’. Not exactly misleading, but not very clear either. Especially when a reasonable google search turns up the old ‘sources’ link and it says nothing about being ‘old’. So, OK, it’s not horrible; but it is still not very cleanly done.

    The newer v3 sources do still have the ‘ignore water near ice’ code in them. So I’ll need to ‘update’ the posting to reflect where the new sources can be found and that it is more a question of ‘easy to find’ than ‘not available’ for the v3 codes. More to do for this weekend…

  20. David A says:

    “The newer v3 sources do still have the ‘ignore water near ice’ code in them”

    How is this defined? you wrote, “Yes, you read that right. The comment says:

    “Skip some regions where SST is impacted by nearby ice floats”

    That 166 to 180 loop says that anything from 76 N to 90 N is to have the data tossed out.”
    —————————–
    Does this mean for all seasons, including the SH summer? Could this be in a large part why GISS diverges from RSS so much of late?

  21. David A says:

    oops “76 N to 90 N” I guess I meant the N.H. summer.

  22. David A says:

    Wait, the post is about the S.H. Does this apply to both?

  23. omanuel says:

    Thanks to the Climategate emails that surfaced in late November 2009 and official responses from government scientists, we now know why government propaganda replaced government science after the end of the Second World War:

    https://dl.dropboxusercontent.com/u/10640850/Preprint_Solar_Energy.pdf

    Comments or corrections to the preprint would be appreciated.

  24. “The antarctic links lead to the data, but they are in a display format, not a download bundle. Yes, I can transcribe them, but that’s not what the program says…”

    It would be nice to have a tidy download bundle, but that is not within GISS control.

    It’s quite a while since I worked through the GISS program code and scripts doing my own implementation in C# rather than FORTRAN, Python and shell scripts. But it was all there. You do need to look at the scripts as well as the FORTRAN/Python as I recall.

  25. manicbeancounter says:

    It is interesting that you point to a major revision of the Gistemp series in 2011. I have recently looked at the consequences to the global warming message between data downloaded in early 2011 and last month. In particular
    – In 2011 Gistemp was reporting a warming rate for 1998-2010 unchanged from the rate of 1976-1998. In 2014 Gistemp reports the rate halved.
    – In 2011 Gistemp was reporting a warming rate for 1976-2010 was 2.2 times the rate of the early twentieth century warming. In 2014 Gistemp reports just 1.2 times. Early C20th warming has increased from 0.33 to 0.53 degrees.
    What is most significant is that Gistemp has come into line with HADCRUT. For instance in 2011 Gistemp was saying the peak of the early C20th warming was 1940. HADCRUT3 reported 1944 as the warmest year. Now both Gistemp and HADCRUT4 agree on 1944 as the peak year.
    I posted the details at http://manicbeancounter.com/2014/06/28/nasa-corrects-errors-in-the-gistemp-data/

  26. David A says:

    Peter says…”It would be nice to have a tidy download bundle, but that is not within GISS control.”
    Why?

    The title of the post is…who needs Antarctic data or temps near ice.” If the code says… “Skip some regions where SST is impacted by nearby ice floats”
    That 166 to 180 loop says that anything from 76 N to 90 N is to have the data tossed out.”
    Then, even if more recent data is found, the title is accurate, is it not?

  27. kirkmyers says:

    At the end of the day, NASA GISS has its marching orders. And those orders, clearly, are to fudge the data enough to create a warming signal. Those employees who aren’t on-board and object to dutifully producing what the bosses demand will soon find themselves on the street. So, for many otherwise decent GiSS “scientists,” it’s a choice between making the car and mortgage payments and putting some bread on the table, or commiting seppuku in the name of honesty. So the unexpressed rallying cry at GISS is, “Take the paycheck and keep the mouth shut.”

  28. As to why that is not within GISS control:

    This data does not come from GISS, so they can hardly order that it be made available in a different format. At most they might make available the intermediate files from their monthly Gistemp run, as indeed they once did, possibly unintentionally. This would include the Antarctic data, but only for a single date each month. Anyone requiring the data at any other date in the month would still need to go to the original source.

    For now, anyone wanting the data for the date of the Gistemp run can simply download http://data.giss.nasa.gov/gistemp/station_data/v3.mean_GISS_homogenized.txt.gz and delete all records not starting with the digit “7”. The “country codes for the Antarctic stations are 700 and 701, and as all are classed as rural the homogenized temperatures are the unadjusted raw temperatures. (But note that Byrd data comes from a different source, so you might still need to look up the SCAR data for Byrd if you need that)

  29. E.M.Smith says:

    @David A:

    It is a blanket operation, no limitation based on any other variable (date, time, whatever…); so yes, it is just excluding the water temps ‘up there’.

    The post is about GISTemp, both poles. The original question was antarctic temps and temps ‘near ice’, with a bit of confusion over ‘which ice’. It has been clarified to “antarctic temperature data and north polar ice”.

    @Peter O’Neil:

    Once it is homogenized, it is no longer data… it is at best a ‘data food product’…

    @OManuel:

    I’ve not been invited that I know of. Then again, I’ve not checked email in a few months. (Sorry, but being slave to email is not high on my life goals in the last few decades of my life. It’s an ‘as available’ thing…)

    I’m reading through your pdf. On page 3 you have “, with pairs of neutron-proton (n, p+) pairs” and I think the word pairs is in there once too many ;-)

    In the graph you have Xe1 and Xe2, in the text you talk of ‘strange’ Xenon. It might be nice to state that “strange Xenon is of mass (foo)” in the text. I think it is the Xe2 that is higher in both 124 and 136 and not the Xe1 that is 130, but that’s based on one caption and you made me ‘work at it’… which some folks will not do. Might be easier to include a definition along the lines of “Xe1 is mostly isotope 130 and found on earth, while Xe2 is a bit ‘strange’, found in space, and is mostly isotopes 124 and 136″ or whatever is correct….

    You use the term “s-products”, but I’ve not found a definition of it yet. Maybe I just missed it?

    Similarly p-process and r-process. Perhaps all that is clear to the intended audience, but to me it’s a bit un-clear.

    I think it would be improved with the leaving out of the yellow highlights. They give it a ‘shouting’ quality…

    I’m not sure that the ‘political’ statements mixed in with the science ones ‘works’. Then again, I don’t know the intended publishing venue or what rules they might have on such things. I think the science in it is true, and the political spin likely has some merit; but it could just as easily be that the Powers That Be in the world of nuclear physics simply screwed up. There are fashions science, and often the fashion is to make a ‘radical’ change in things that do not need changing just to get more grant money or more prestige. It doesn’t really need more motive than that for folks making up things that overturn clear understanding with pseudo-bs…

Well? Say something!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s