QA or Tossing Data – You Decide

Well, Is It An Improvement Or Not?

I found this paper:

http://journals.ametsoc.org/doi/full/10.1175/JTECH1963.1

that gives an overview of some of the approaches used in the “QA Process” at NCDC. I’ve read it through once (but not followed all the links to the other papers it references).

I find reason to think that the methods detailed in this paper are the “Smoking Gun” for the extraordinary compression of the “range” of the monthly anomalies that we see in 1990. There are some alternative “Duplicate Number” series that begin in the late 1980’s with dates that match those of the papers listed in this paper (1987, 1989 etc.).

The process is claimed to have been tested by how many “seeded errors” it found, but I don’t see much mention of how many valid values were rejected nor a discussion of any method to assure the data are not biased by this procedure. Maybe I missed it or maybe it is in the links I’ve not yet followed. ( It was a quick first read).

But frankly, what worries me the most about this is the, in my opinion, rather extreme and cavalier changes made to data that are then presented as ‘nearly raw with a bit of QA’. Things like replacing values with computed values based on nearby stations. Not exactly my idea of a simple QA process..

A couple of quotes:

Recently, the use of multiple stations in quality assurance procedures has proven to provide valuable information for quality control (QC) compared with the single-station checking. Spatial tests compare a station’s data against the data from neighboring stations (Wade 1987; Gandin 1988; Eischeid et al. 1995; Hubbard 2001a). They involve the use of neighboring stations to make an estimate of the measurement at the station of interest. This estimate can be formed by weighting according to distance separating the locations (Guttman et al. 1988; Wade 1987), or through other statistical approaches [e.g., multiple regression (Eischeid et al. 1995) and weighting of linear regressions (Hubbard et al. 2005)].

Nothing like requiring that a station never have an ‘extreme event’ to assure that it never has an extreme event…. So what happens as there are ever fewer ‘reference stations’ for comparison and as more of them are Airports? What happens to the “nearby” rural cold station when it must be in compliance with the dozen Airports or get killed?

I’ve bolded a bit here:

The SRT approach has been found in a previous study (Hubbard and You 2005) to be more accurate than the inverse distance weighting (IDW) approach for the maximum air temperature (Tmax) and the minimum air temperature (Tmin). It was found that the RMSE was smaller for SRT estimates than for IDW estimates in all areas including the coastal and mountainous regions. Both the spatial regression and inverse distance methods were found to perform relatively poorer when the weather stations are sparsely distributed as compared to areas with higher station densities. The success of the spatial regression approach is in part due to its ability to implicitly resolve the systematic differences caused by temperature lapse rate with elevation; these differences are not accounted for in the IDW method.

So the more stations you drop the worse your performance becomes? AND they know this? AND they are doing it anyway?

2) Step change (SC). This is a check to see whether or not the change in consecutive values of the variable x falls within the climatologically expected lower and upper limits on rate of change (ROC) for the month in question. In this case the step is defined as the difference di between values on day i and i − 1; that is, di = xi − xi−1. The step change test is performed by using this definition of d, calculating the associated mean and the variance, and using Eq. (2) with d substituted for x. Again fsc takes a value of 3.0.

“Climatologically expected” by whom? This looks like it’s just imposing statistical bounds on what the real data can report based on what they deem acceptable realities… I was in Dallas once when a 50 F drop happened from a cold front moving in. Step Change does happen in weather. In my experience, more to the cold side than the hot side. Will this toss out more cold step changes than hot?

4) Maximum and minimum air temperature mixed up (Mixup). This is a check to see whether the maximum and minimum air temperatures were interchanged by the observer. The record will not pass the test when the maximum air temperature of the current day is lower than the minimum air temperature of this day, previous day, or next day.
Similarly, the record will also not pass the test when the minimum air temperature of current day is higher than the maximum air temperature of this day, previous day, or next day.

So we can never transition from hot to cold in one day. Nor even two. Need to tell that to everyone hit by a Canada Express… Not stated is what happens to that QA process when that day is “tossed out”. If it was invalid, what is the NEXT day compared with? I’d like to see some field work that shows this is valid for all places on the planet. Again we see no recognition of the fact that places can be made cold faster than they can be made warm (Think snowfall vs clouds clearing. Delivering a few tons of ice per hectare will cool you down “right quick”. A sunny winter day warms you not so much… )

5) Spatial regression test (Hubbard et al. 2005). This test is a quality control approach that checks whether the variable falls inside the confidence interval formed from surrounding station data. Linear regression estimates are calculated for all pairs {x, yi}, where x is the station whose data is being quality checked, yi are the data for the i surrounding stations (i = 1, n), a and b are the regression coefficients, and the data record spans N days. For an observed yi, n estimates of x are calculated. A weighted estimate x′ of these n estimates of x is obtained by utilizing the standard error (root-mean-square error) sei of the n regression estimates:

And it goes on…

Well, nothing like a little regression testing to make sure everything fits inside your preconceived notions of what’s allowable. But who does the pre-conception? Based on? Is nature so predictable that it never delivers surprises? Are surprises to be dropped on the floor because nature is not allowed them? Is this The Procrustean Bed of Weather? If so, pray thee Gods, make me Theseus…

The NCDC Approach

As described in the paper: (I’ve bolded bits)

b. NCDC approach
The NCDC quality assessment is based on accepting all observed data that are plausible. There are five steps in the evaluation of temperature data. Because of the volume of data that are processed as well as requirements to provide quality assessed digital data to customers in near–real time, a goal of the approach is to automate as much evaluation as possible.

1) Pre-edit. This step checks the input data records for format and coding errors. Improper station identifiers, invalid characters, duplications, values that are not in a valid range, unexpected data, and other similar problems are identified and corrected if possible. If it is not possible to correct these errors, then a datum is labeled as missing for follow-on processing.

2) Climate division consistency. Departures of a station’s data from the monthly average of the data are calculated for all stations within a climatic division [see Guttman and Quayle (1996) for a description and history of the 344 climatic divisions in the contiguous United States]. The average departure for each day is then calculated. A datum is flagged for further review if the departure for a given station and day differs from the divisional average for the day by more than ±10°F. For a given day, temperature means and variances are estimated from all the divisional data that have not been flagged for further review. Any flagged data that exceed ±3 standard deviations from the mean for the day are then flagged for replacement. Replacement values are calculated from data for up to six nearby stations by the following procedure.

OK so we use only the accepted data to determine what is accepted. This opens the door for a recursive accumulation of error. And it’s automated as much as possible. How much?

We use a fixed size (10F) and ignore that cold anomalies are more common than hot (Hot air rises so we convectively limit. Cold air does not rise, it accumulates. Further, as we drop stations with extreme values – those mountains and high latitudes – our “average” narrows so what we will accept narrows. Station dropout matters here.) Then we just make up replacements…

For all nonflagged data, for a station, compute Z scores to standardize the data with zero mean and unit standard deviation.

For all combinations of the nearby stations, compute the average daily Z score.

For each combination of surrounding stations, multiply the average daily Z score by the standard deviation of the nonflagged data for the station for which a daily value is to be estimated (replacement station).

For each combination, subtract the estimated departures from observed, nonflagged departures for the replacement station.

For each combination, compute the error variances of the data obtained in 4.

For the combination with the smallest error variance obtained in 5, for the day being estimated at the replacement station, add the estimated departure to the replacement station mean calculated from the nonflagged data.

What a complicated stew just to make up a number… and it’s validity rests on it being statistically close to it’s peer group? It must match the consensus or be shot?

Replacement values that differ from the original observation by more than ±15°F may be manually adjusted if a validator believes the flagged values are in error by more than ±8°F.

Or they can just make something up the old fashioned way… Wonder where they keep the hat they pick from? At least, I hope they are pulling numbers out of their hat, considering the alternative…

Validators also compare the divisional data to the top 10 and bottom 10 observed extremes on a statewide basis. This comparison is intended to identify gross keying errors and anomalous extreme values and is performed both on the observed data and on the replacement values.

And as an ever greater percentage of stations are Airports and Urban we will have more “top” extremes to pass data and fewer “bottom” extremes… Station Drops Matter to this process. Drop all your high cold mountains and your high latitude stations and you will have a drastically reduced set of “bottom extremes” to validate against. So California with SFO, LA, San Diego and Santa Maria will have how many “observed extremes on a STATEWIDE basis” against which to compare the snow presently falling in the High Sierra Nevada?

3) Consistency. This check ensures that maximum, minimum, and observation time temperatures are internally consistent. Physically impossible relationships, such as the minimum temperature for a day being greater than the maximum temperature for the same day, are flagged as suspect. Often, these errors result from incorrect dates that are assigned to an observation (sometimes called “date shifting”); if possible, the flagged data are corrected.

4) TempVal. This spatial check uses grid fields derived from Automated Surface Observation System (ASOS)/Automated Weather Observing System (AWOS) hourly and daily temperature values as a “ground truth” to quality ensure the Cooperative Network daily temperature data (Angel et al. 2003). Note that the previously described steps are only applied to the cooperative data; this step compares the cooperative data to an independent data network.

Wasn’t there a problem with the ASOS having trouble with internal heat from the electronics and not reporting bottom values well? And were they not found to be held close to buildings by their wiring, too close to be acceptable to the published standards? And are not the bulk of them at Airports with loads of Tarmac and motor vehicles? So all thermometers must conform to the one that ‘had issues’?

Grid fields of departures from monthly averages are derived from the ASOS/AWOS data using a squared distance weighting function to estimate values at the corners of half-degree latitude–longitude boxes. Three grids were produced for each day of data corresponding to midnight, a.m., and p.m. observation times. The ASOS/AWOS data were quality assessed through an independent processing system that is more extensive than the cooperative data processing system. The nature of the checks are similar to those described above (sections 3a–c), but observations at both hourly and daily time scales, as well as the observation of more meteorological elements, lead to many more checks. Even though the data have been extensively assessed and processed, each grid used in TempVal is examined for suspect data. Every gridpoint value is compared to the average value of surrounding grid points. Suspect grid points (bull’s eyes), along with a list of surrounding ASOS/AWOS stations and their temperature values, are brought to the attention of a validator, and corrections and/or deletions are made as necessary. The grid values half-degree north, south, east, and west of the Cooperative Network site are also calculated. These values are used to determine the gradient (or slope) of the grid at this location.

The data for a cooperative site are compared to the grid estimates at the site. When the difference between a cooperative value and the estimated value is greater than ±(7°F + slope), the cooperative datum is flagged as suspect, and the grid estimate becomes the replacement value for the suspect observation. Note that the constant 7°F is usually much greater than the slope, so that the threshold is approximately a fixed value; the acceptance range for an observed datum is of the order of 16°–20°F. The grid estimate becomes the replacement value for the suspect observation.

We have an average used as the replacement. Averages always have lower range than simple measurements. Is there any metric for how much of the GHCN data set are so constructed?

5) Last look. Validators once again compare the cooperative data to the top 10 and bottom 10 extremes for the state to ensure that replacement values are not anomalous extreme values. Consistency and range checks are also performed again to ensure that the assessment process did not introduce errors.

And we do it all again. No wonder at the end of this everything is just squashed into submission as an ever flattening range. AND the concentration of stations at Airports and use of ASOS as gatekeeper assures that “Everywhere is in conformance with the Airports” No Matter What.

These “daily values” then get used to compute the “monthly mean” that is all we see in the GHCN and USHCN data sets. So how much of the “daily data” is made up or tossed out on the way to the “monthly mean”? We just don’t know.

OK, so how do I get data that has not be so adjusted on it’s way to being sold as UN-adjusted?