Deconstructing the HadCrut Data

In this post I took a look at the PDO, AMO, and ENSO data and went through the exercise of fitting a sine wave to see how the fit looked.  On all three, the fit seemed to be a good and reasonable way of estimating the general level of the curves at a given time.  Obviously, there are fluctuations about those curves, but all in all it seemed to be fairly adequate.

So, the other day I was thinking about the implications of hypothesizing the contribution of these
periodic elements to the temperature data, and I figured that it may be an interesting exercise to deconstruct the HadCrut data in the same way.  Understanding that there may be multiple oscillations going on, I set out to fit multiple waves to the data to see what I could make of it.

I used HadCrut because their records data back the furthest.  And also because the older data has not undergone the continual adjustments that the GISS data has.  Since HadCrut protects their process in determining the anomaly, it is not transparent to the user of the data what kind of spreading and adjustments they make to it.  However, I am inclined to place a little more trust in it than I do GISS because the data is more in line over the last 30 years with the satellite data than GISS is.  (Admittedly, based on observation.  I haven’t done a rigorous analysis on that).

Let me start by introducing the most recent chart that shows the overall linear trend on HadCrut over time. I didn’t do a January update, but the additional data point won’t have much of an effect.

The overall trend since January 1850 has a slope of 0.0003649, which corresponds to warming of 0.4378 degrees Celsius per Century. Light blue lines are raw anomalies, and the black line is a 12-month smoothed number.

Here are the required parameters and procedures to get the best fit:

1. A parameter that assigns the optimal wave length. This is done by determining a degree increment per month that gets added to the previous degree amount. That value is then converted to radians, and the sine value is calculated.
2. A wave scale factor is used to determine amplitude of the wave.
3. The starting wave position in January 1850
4. Linear trend that the wave is centered on
5. Vertical wave shift – since the anomalies over time aren’t necessarily centered about zero, and even if they were there may still be a shift needed depending on wave cycle weights, the wave needs to be shifted up or down to achieve optimal fit.
6. Parameters 1-3 are wave specific when optimizing multiple waves, while parameters 4 and 5 are applied after the cooperative effect of all the waves tested are determined.
7. To determine the best fit, I set up my calculations using the parameters above to compare against the historical HadCrut anomalies, determine the difference, and square the result. I then minimized the sum of the squares.

The first run I did was against only one wave. Even this fit provides substantial improvement over a simple linear fit, and starts to demonstrate the fact that there are clear cycles in the data.

The best-fit single sine wave along a linear trend.

While we can see visually that it is not entirely perfect, we can definitely see the general wave it’s tracing. The parameters on this wave are as follows:

• Wave length parameter = 0.45658. This corresponds to a 788.5 month complete cycle, or 65.7 years. This fit lines up right along with the cycle lengths determined in the PDO/AMO/ENSO study.
• Wave scale parameter = 0.13222. Basically, this means that the overall wave doesn’t deviate from the trend line by more than this value.
• Start Wave position = -31.71316. (0 would be right on the trend line, and +/-90 would be max deviation from the trend line in either direction.)
• Linear trend = 0.00037. This is spot on with the overall linear trend observed without the sine curve applied.
• Vertical Wave Shift = -0.53644 means that the start of the sin wave had to be shifted downward by this amount.

The least squares result was 56.18. This compares to the least squares fit on the linear trend line only of 72.58. This is nearly a 25% improvement.

The conclusions of this:

• there is still a definite linear trend, but most of the fluctuation about that trend can be explained by adding a single sin wave.
• the most recent decade or two is not satisfactorily explained by the sine wave, and the latest anomalies are above the wave. This could be consistent with the idea that the something has changed (e.g. increased Carbon Dioxide has accelerated what had been a linear trend).  Alternatively, it may simply be that a single sine wave is insufficient and there are other periodic influences that need to be examined.

An interesting exercise is to extrapolate the linear trend with the single sine curve forward. Taking this to 2050 shows us the following:

The best-fit single sine wave along a linear trend, extrapolated to 2050.

Observations:

• The graph indicates that we are at or near the peak of the single sine curve fit, and that the next 23 years will cool. 
• There is still the linear trend line observed at just over 0.4 degrees Celsius per Century, so the anticipated trough won’t be as severe, but we will still be cooler than today for the next 30 years or so.
• The trough of the curve occurs October 2032, where single-digit anomalies would be the norm.

While the single sine wave fit provides interesting information, from observing the AMO/PDO information, it seemed clear that there would likely be at least one additional periodic wave in the data. So I added an additional wave and the following chart ensued:

The best-fit double sine wave along a linear trend.

The interesting thing about this fit is that the cycles of both waves are shorter than the singularly combined wave (59.1 years and 58.5 years), and the amplitude of both waves is around 1.5. The combined wave is only 0.13222 because the phases of these two waves at the start of the period are almost perfectly offset (182 degrees apart). The linear trend is still apparent, though just a shade less (0.42 degrees per century). The vertical wave shift is nearly the same as the single wave. Overall least squares fit isn’t remarkably better, at 54.118.

All in all, the waves do seem to do a pretty good job of fitting the curve. In the early 1900s, the wave seemed to ride above the curve a bit and in the last decade or two the wave rides below a bit. So, it’s not perfect. It is possible that a linear curve is not the best approximation to have the sine curve fluctuate around. I may do further tests with other alternatives, such as geometric or exponential approaches.

The extrapolated chart, though, shows a little more severity in projected cooling (the title is wrong – it should read “Double Sin”  I would have corrected it except that I need to re-create it and don’t have time at the moment.  My apologies for the oversight:

The best-fit double sine wave along a linear trend, extrapolated to 2050.

Observations:

• The shifting of the wave phases over time lead to more fluctuation in the peaks and troughs, which explains why the right hand side of the chart fluctuates more than the left.  Since the amplitudes of these waves are 1.5, one can imagine a time centuries from now where there would be astonishing swings in the temperature trends over 50-year periods of time.
• The projected trough in temperature is expected to be June 2030 according to this chart.   Average anomalies at that time would be slightly negative.   The last half of next century would warm considerably.
• All this fluctuation occurs over a linear trend that is less than a haf-degree Celsius per Century.

I didn’t stop there. I tested three waves. However, the results that provide the best fit don’t make a lot of sense. The fit of existing data is certainly impressive, so let’s look at that first:

The best-fit triple sine wave along a linear trend.

Adding a third sine wave certainly appears to put our temperature observations spot on with the generated curve. One may feel inclined to get all puffy and declare that the case has been solved.

But looks can be deceiving. The best fit with three waves – a reduction to 45.47 – uses parameters that are not sensible. And this becomes a case study in trying to be “too accurate” with model forecasting.

When the third wave is introduced, I was able to generate a number of “best fit” scenarios all around the same value of 45. One such scenario shows that we will go into dramatic cooling by 2051, and the anomaly will be -2 at that time. The next scenario had a best fit of 46.43, with dramatically different parameters. That model shows no downturn whatever in temperature after the current period, and suggests unabated warming through 2050, where the anomaly will be around 1.

This is common when adding multiple parameters to models. In the quest to get more accurate, you actually introduce so much additional uncertainty that the range of reasonable projections becomes meaningless.

My conclusion is that the best representation using a sine wave analysis is a simpler 2-wave representation.

The conclusions are basically in line with the PDO/AMO analysis, as well. These drive the longer-term cycles about a trend. Whether it is a linear trend or something else is worth looking into. Also, is the trend related to the sun or carbon dioxide, or other things? The cycles still do not explain the overall trend, but they do help explain why the recent linear trends cannot simply be extrapolated into the future.

A Closer Look At Oceanic Oscillation Cycles

In the past, I’ve presented some charts on the different Oceanic Oscillations for PDO, AMO, and ENSO. I’ve started to take a look at these again with an eye towards running a correlation analysis. The initial work I’ve done today is something I considered somewhat interesting, so I thought I’d share it.

The first thing I’ll present is the chart for Arctic Ocean Oscillation Indices since 1950, smoothed at one year, 5 years, and 10 years. These are presented below:

The overall Arctic Oscillation index data since 1950 - 1 year smoothing.

The overall Arctic Oscillation index data since 1950 - 5 year smoothing.

The overall Arctic Oscillation index data since 1950 - 10 year smoothing.

Unlike the AMO, PDO, and ENSO charts, there is no apparent cyclicality showing up in the Arctic Oscillation chart. There does appear to be a trend upward overall, and there are certainly ups and downs within that. The 1-year chart looks much like an ENSO chart would be. Unlike ENSO, though, I’m not picking up a longer cycle.

Well, I wanted to show that chart to start, since the Arctic seems to be the focus of a lot of attention. I guess it bears musing whether or not the Oscillation has a root cause from the Ocean itself, or the sun, or melting ice, or freezing ice, or other factors that override any cyclical nature that would otherwise be apparent.

That’s all I really did on that piece. But I’d like to move on to some work I did with the AMO, PDO, and ENSO (as well as a look at those Arctic Oscillations).

A Look at the Atlantic Multidecadal Oscillation (AMO) Index

Previously, I took a look at the Pacific Decadal Oscillation (PDO) Index and showed that after a fairly long period of persistence in an average state above zero, it has dipped back down into the negative. It has now been in the negative level enough that the 10-year smoothed chart is actually negative. This has coincided with generally flat to cooling global temperatures, and colder weather patterns in the United States over the last few years.

I also took a look at the ENSO index over timeand showed that, while the ENSO cycle is much shorter-term, it also has been in a persistently warm state since the late 70s. I was one of the first I am aware of to point out the very recent readings in the Index pointing to another La Nina (there is some debate as to whether or not it is a “true” La Nina, since the cooler PDO could be driving this measure down. That seems silly to me. If you’re going to start arguing about influences to the index, then you can’t stop at the PDO.) In any case, the ENSO index is also running into recent negatives, and the persistent warm state appears to have finally come to an end.

It is reasonable, then, to wonder why global temperatures haven’t plummetted. While it is true that, current non-warming trends go back to May 1997, according to UAH anomaly dataand that all temperature measures show a declining trend line since 2001 to one extent or another, we have not been in a freefall on a global basis.

Certain regions appear to be more impacted than otehrs to recent cooler temperatures. The United States and Canada have had a much cooler time of it than Russia, for example. In fact, I previously took a look at the temperature anomaly maps and questioned why Russia seems to show consistently higher anomaliesthan the rest of the globe. I’ve checked the NOAA maps pretty consistently over the last couple years, and I have seen a lot of blue in the United States and Canada, and Russia burns bright red. This appears to be the most significant driver in temperature anomalies not hitting bottom.

It would appear that the PDO and ENSO do not drive Russian anomalies. At least, anyway, not as significantly as they seem to drive our region.

And so I decided to take a closer look at the other Ocean Index data. I have downloaded a boatload of data, and will be getting to each one in time. But I started with the other index we seem to hear a lot about – the AMO index. I started here because from what I understand, this is a pretty key index to keep track of. It also has the longest data history. In full disclosure here, I have not yet taken the time to look at the history of how these temperatures are recorded. I don’t know if they are direct measurements or modeled or some combination of the two. So, I simply present the data as given.

Starting out with the Raw Data:

The overall AMO index data since 1856.

Pacific Decadal Oscillation (PDO) Index – Back into the Negative

The PDO index is a measure of another one of those weird oceanic oscillation patterns.  The ENSO index is a shorter-term varying event, whereas the PDO index is considered a longer-term event.  Interestingly, however, when you collapse the ENSO index into longer average time periods, there does appear to be persistency in the ENSO index, as well. However, the short-term cyclical nature of the ENSO index is much more evident than the PDO index, even if the averages show similar persistency.

The PDO index is predominant in the Northern Pacific and secondarily evident in the tropics, whereas the ENSO index is the other way around. ENSO receives so much attention because of the large short-term cyclical variations, but the PDO has very significant swings as well, just usually over a longer period of time.

The index, however, is measured on a month-to-month basis and the data can be located here.

November has not yet been updated, but the October index value was -1.76. This was the single coolest reading in any month since November 1999, and was the 14th consecutive negative reading in a row. While there have been other stretches of 14 consecutive negative readings – the last being 1999-2000, it is interesting to note that the current streak of seven consecutive readings of less than -1 has not been seen since the period ending February 1976. That particular stretch ended at 7 months. If this month’s reading is below -1, it will mark the first time since the period ending June 1972. And that streak ended at 8. Beyond that point takes us to a streak in the early 1960s. So I’ll keep an eye on that.

As I did with the ENSO index, I collapsed the raw data into different average periods. Since the PDO does seem to vary over a longer period of time, I went up to a 10 year average. The persistency in the index is very clear once you take longer-term averages.

Below are a few charts of the PDO. The first is the raw data. Taking the edges off with 12-month smoothing follows that. I then present 5-year and 10-year averages.

The PDO monthly index has been negative for a bit already, as has the five-year average. But the 10-year average recently went negative, as well. Of course, all the same arguments apply as in the ENSO discussion – the persistence matters. You keep a heater running and the temperature gets warmer than the temperature an hour ago, depending on insulatory effects. Persistence in both ENSO and PDO correspond with the warming we see in the global temperature readings. The 10-year average PDO is slightly negative, and we’ve cooled slightly in the last few years. Coincidence?

The overall PDO index data since 1900.

The overall PDO index data since 1900 with 12-month smoothing.