So sometimes something gets my attention and I’m “down the rabbit hole” for a few days (weeks… months… ye…)
In this case, it was an old problem, back again.
I never got the last steps of GIStemp to run, as all the machines I had at hand were ‘little-endian’ and GIStemp was ‘big-endian’ in the last steps. FORTRAN is a bit unforgiving about endian-ness in data structures. Data written in an unstructured way really gets the fundamental structure of the endian character of the processor. Endian is an oblique reference to Gulliver’s Travels and the wiki mentions it:
“In the discipline of computer architecture, the terms big-endian and little-endian are used to describe two possible ways of laying out bytes in memory. The terms derive from one of the satirical conflicts in the book, in which two religious sects of Lilliputians are divided between those who crack open their soft-boiled eggs from the little end, and those who use the big end.”
Basically, if you write 1234 into a computer, is it stored as 1234 or as 4321? (There is also a byte order endian issue, so it could also be 2143 or 3412 in some odd cases…)
Most of the time for almost everyone this endian issue is completely hidden.
For programmer geeks like me who keep the world sorted out for people who don’t care, or know, that endian issues are all over the place.
So GIStemp keeps endian issues out of the way until the very end, in Steps_4_5 code.
So I had most of everything that mattered by Step_3, and didn’t get the last bit running, as the machines I had at home were either little-endian PCs (Intel chips) or were Macintosh boxes with Motorola or PowerPC chips (big endian) but running Mac O/S and not something I was willing to blow away to install a Linux port.
So I left it an unfinished bit of the GIStemp port that I never did get the Big Endian bits to run.
Different Time Perception
I sense time differently than other folks.
I know this, but can’t change it. I remember being 3 or so years old and running down a dirt track with two tire tracks and grass in the middle and falling down and realizing that falling hurts skinned knees in just the same way that I remember being a ’20 something’ and dealing with a romantic rejection in just the same way that I remember missing a meeting at work a week ago; and in just the same way that I see what the world will be like a year from now, but can’t stop it from happening. To other people those are very different experiences. To me, it is all the same perception. All of it is “now”.
So I’ll set something aside to get back to it ‘later’, and then a 1/2 decade later pick it up again at just that spot. Only ‘lately’ have I realized that other folks don’t do that. That a decade ago is faded and lost to them. That it doesn’t just ‘pick up again’. So I’ll say things like “I need to do a posting on that”, and it is the same to me if it is tomorrow or a decade later. Other folks, not so much. They see it as not delivering as time ends shortly after the statement… Oh Well…
So, some ‘long time ago’ I looked at GIStemp and noted that it wasn’t using USHCN.V2:
That clearly shows it was 2009. It is now 2014. I make that 1/2 decade. To me it was just yesterday. Oh Well…
So why dwell on this? Because sometimes exact dates matter. Note that 2009 well.
So around 2009 to 2010 there was a change of USCHN and some GIStemp code changed. This is after I did the GIStemp port. This posting is about what has changed in GIStemp, so that date matters.
So let’s look at some GIStemp date stamps, and along the way look at new ways to run very old code.
Does anybody know what time it is?
I finally, and apparently about a 1/2 decade later, found a solution to my Big Endian machine need. That being a bit of emulator software that lets you make emulated Big Endian machines on Little Endian Intel chip machines. That being Qemu or the “Quick EMUlator” software. http://wiki.qemu.org/Main_Page is the home page.
Qemu is an open source software bit that lets you emulate other hardware. Now, on my laptop, I have a Sun SPARC bigendian emulator running. Basically, I have a Sun SPARC based Sun 5 or Sun 10 running. Now, to make this particularly ironic, I have a SparcStation 5 and 10 in my garage. I bought them for about $5 each when some company in Silicon Valley was going out of business. Yet it is quicker and easier to make the emulator run on my laptop. (Not to mention that by now the lithium batteries are likely dead and the machines have lost their identity as their battery backup ROM evaporates…)
So where to get Qemu? Well… that depends. I got the “for windows” version that is scarce. The “for Linux”
version is available for just about any Linux. But… my laptop and my machine at work are Windows Intel machines. So is there a way to get a SPARCstation running on a WinTel box? Yes.
Has a couple of Qemu for Windows releases. I installed the 1.5.3 one. It worked fine. (Details in a future posting, though how many decades is an open perceptual difference ;-)
I now have a SPARC running Debian Linux on my laptop. (Technically, on an SD Chip in a slot in the side of my laptop… but…)
The GIStemp Download
So first I tried using “wget” to get the GIStemp sources (as I was running a non-windowing version of Qemu). No joy. So I went whole hog and started a full on X-Windows based Debian On SPARC emulation. It is slow. Very slow. But livable. Barely. And I got the “current” GIStemp source code downloaded via HTTP / Web Browser to my Virtual Machine.
Some Day I’ll write up the details of how to make this go. For now, the important bit is that I did get the download into a Virtual Machine on my laptop. I did get the compressed archive unpacked. It is ready to configure, compile, and make go. But what surprised me was the time stamp. To me, they are not very new. Most of GIStemp is unchanged. Yes, I need to do a ‘diff’ of the sources and figure out exactly what changed. But at a cursory level, it looks like ‘not much’.
Does this mean that they “double dip” and do the GISS adjustments on top of the GHCN / NOAA / NCDC adjustments? I don’t know yet. That depends on the exact differences in the code. That will come in the future. For now, the date stamps do not show much difference.
So what does it look like? Here are some screen shots.
Images Of GIStemp / Qemu Now
This is the top level picture of Qemu running in a window on my WinTel PC.
So here is a screen shot of a Big Endian SPARC (emulated) processor running Linux. Note the tar ball of GIStemp sources and the unpacked directory of them.
Note the Date Stamps on the GIStemp sources as unpacked. Not much change in the last 1/2 decade or so…
Step0 is largely unchanged. Step1, the Python step, is also not much changed.
To me, at a first glance, it looks like the “pick up the data” processes in Step0 have changed a little, but the actual “apply processing” in Step1 has not changed much.
Has GIStemp “double dipped” by having NCDC apply adjustments and GIStemp do “the same old same old” on top of them? I don’t know. But the date stamp pattern does not look like much change. Yes, I’ll go through the code “line by line” and see what did change. It just isn’t really looking like they ripped out a lot of stuff on a first glance…
How about Steps 2 and 3?
Again, it doesn’t look like a lot of change.
For completion, here is the last bit. Steps4_5:
The only notable change looking like it involves HadCrut R2 release changes.
OK, I’m definitely “down the rat hole” as I’m going to do a character by character compare of old and new. But it will take some time. My old copies are on a Mac from about 20 years ago (though it is with me). So it will take a while to get it booted and figure out how to get the old stuff from it to the new Virtual Machine SparcStation 10 emulator. And time is what I don’t have a lot of right now.
My sense of it is that GIStemp will not have really changed much from ‘last look’ and that they double dip the NCDC homogination / adjustments. But close examination will answer it, or end it.
Qemu is your friend. Running full on X-Windows is slow, but livable if needed. Running it as a text only ‘small’ shell Linux is quite fast. The SPARC didn’t have hardware video processing, so running in just a single core and having that do graphics is surprisingly effective; though the slow graphics isn’t everything.
(Someday I hope to make a multi-processor Linux Damn Fast machine, but that is a ways off…)
What has been demonstrated? That a BigEndian solution is in hand, if slow, and that GIStemp has not changed much, per the date stamps. More work needed on that to show exactly what changed, and what it does.
I will be spending the next few week making a GIStemp On A Chip, with GIStemp and data loaded onto a BigEndian system image under Qemu; all on a modest SD Card. If this works out well, then anyone can run GIStemp via a small emulator download.
So that’s what I’m doing now. Yet Another GISTemp Port.
Details as they are available.
I’ll be making a post or two about how to set up a Qemu SPARC 10, how to make it portable (on a chip / thumbdrive) and how to have a portable GIStemp on a chip that runs on most any Windows machine you have laying around.