This page is where I will post follow-ups and revisions to my recent article *The Great Climate Shift of 1878*, posted on WattsUpWithThat?

Reader Greg Goodman had a number of constructive suggestions, one of which was to remove the oscillatory modes from the data and run a fit to equation 2 on the residual. This removes any reliance on the SSA for extraction of the trend and thus precludes the possibility that the observed response is SSA related. If the fit parameters come out close to that which was achieved by fitting equation 1 to the differenced trend, it will be confirmation of the result.

Here is the hadcrut4 NH data with the 2 main oscillatory modes removed:

In order to use equation 2 on the data, we need to add a constant of integration, C and solve for the boundary condition of continuity at the step. The modified equation 2 is:

Eqn 2a – Modified 2nd order system response allowing for non-zero initial condition and with boundary condition of continuity at t=τ

Fitting Eqn 2a to the data of figure 1 yields the following parameters:

Compared to the previous fit obtained by fitting eq1 to the differentiated SSA mode.

We see the new fit yields a slightly more damped system with a slightly higher natural frequency. Given the data of figure 1 is much noisier, especially near t=tau, and the difference in detection method, I believe these results are confirmative. Of significance is the match in τ, the time at which the shift occurs.

Figure 2 shows the two responses

The fit is shown below (left) along with the residual (right)

Adding the oscillatory SSA modes to eqn (2a) is shown below against the unaltered data

The correlation coefficient R= .8476 is insignificantly better than the R=.8471 achieved with the original model using the integrated eqn 1. The correlation is dominated by the unpredictable events in the data. Smoothing the data with a HP smoothing factor of only 5 yields R= .971 for both models.The plot below shows a three harmonic sine fit to the two SSA main modes. Note how the combined SSA modes (red) seem to change during the step response transition region (t=27-50). The three sine waves comprising the fit are shown on the right (fundamental period = 209 years). All three components are near their peaks. Time to get out your winter woollies for the next few decades.

**Update: Monday,Oct. 7**

I’ve done the above analysis on the SST data.

The eqn 2a parameters are

The fit to the differenced trend is shown below.

**Update: Monday,Oct. 7 PM**

If we take equation 2a above to represent a system whose output is the temperature anomaly, then that output is the convolution of the input forcing with eqn 2a. The plot below is the cyclical convolution of the yearly sunspot number with eqn 2a, scaled by a gain constant (.0005) and offset by .9 degrees.

The correlation of the convolution with the data is excellent until about 1970 until the kernel end-of-record effects become apparent.

This result is strong empirical evidence both for the skill in which equation 2a encapsulates the climatic response and for the sun spot number being a primary forcing function.

**[Note: In order to improve readability I have removed some earlier updates which were in error or incomplete ]**

**Update Tuesday, Oct 8, 8PM**

Here’s the result of matching the kernel length to the SSN data compared to Gregs data (red) and the hadcrut4 SST data (gold).

In the plot below, the system response delay taken out.

**Update Tuesday, Oct 8, 3:45AM**

The plots above were convolved against a kernel derived from the SST NH data. In the plot below, the system response was derived as above (parameter fit of eqn 2a) but using Greg Goodman’s adjusted SST data. The correlation with the data is excellent until about 1963. The apparent cooling (shaded area below) might be explainable by volcanic activity [although Greg’s not buying it – see his comments below].

The graph below is Figure 1 in the last IPCC report. The black line shows temperatures. The four biggest tropical eruptions over the past century had slight cooling effects.

**Update Tuesday, Oct 8, 8PM**

The plot below adds in a single mode (SSA[L=81,k=3,4]) to the convolved SSN record

I had to reduce the kernel damping factor from the derived .135 to .5 to get the last 25 years to match. Here’s why the damping factor came in so low: As the plot above shows, the trend plus just a single SSA mode describes Greg’s adjusted SSA data (R=.963). When we remove the mode from the data we get: Fitting eqn 2a to what is essentially a sinusoid results in a low damping factor.One difference is that I have been using unsmoothed monthly data decimated to yearly by sampling each June. The rationale for this strategy can be seen by looking at the SST stack plot.

Each column is a month. June has the least variance which I suppose is due to calmer seas. Since we want to avoid filtering to preserve as much transient behavior as possible and since the winter variance is uninteresting to the model, we sample in June. Greg’s stack plot is more homogeneous which indicates that perhaps the data was smoothed across months.

In either case (low damping + volcanic cooling / higher damping and no effect from volcanoes), it points out how much the Hadcrut adjustments have biased the IPCC conclusions regarding AGW. If Greg’s “unadjustment” is correct, one glance at the SSA analysis disproves any notion of AGW. And one glance at SSN convolution shows what really controls the climate. The sun, whoda thunk?

**Update Wednesday, Oct 9, 6AM**

Here are the remaining modes not included in the model.

Beyond mode 5, the reconstructions are noise-like.

The plot below shows the residual error of the model (SSN convolved with eqn 2a + mode 2).

**Update Wednesday, Oct 9, 8AM**

Here’s a backcast of the model. The assumption is that mode 1 above (the 58 year cycle) remained constant which may not be true. I don’t know how to check this since it is a SST record but it seems to match the BEST land temp record pretty well.

**Update: Friday, Oct. 11, 3PM**

Appended Greg’s SSN prediction to data set. Convolved with unmodified kernel.

Greg Goodman

said:Glad you found my suggestions useful.

From your fig2 it would seem that the step is very similar in both cases but the over-shoot is only seen in SSA derived case. The non SSA result looks very close to what is called critical damping. That leaves the question of whether the over-shoot/under-damped behaviour is an artefact of SSA or some added detail that it is extracting.

My gut feeling is artefact but I’m open to more objective proof of the contrary 😉

I really like the 2nd order approach. Gross simplifications of making everything first order linear have well past their usefulness.

It would be really valuable if you could look at my other suggestion of doing this processing on ICOADS. Comparing hadSST3 run with uncorrected SST would tell us something about what these speculative ‘corrections’ do to the data. My guess is stronger negative 19th c. higher final level for 20th c. and faster transition with similar tau. What it does to damping would be interesting.

One small point, it would be nice to have your fitted coeffs as copyable text as well as (instead of) the images of text.

Greg Goodman

said:If you like I have have icoads monthly data from KNMI that I’ve cleaned up to remove gaps etc. and cropped off earlier discontinuous sections.

If your method does not require continuous monthly series, better to get it straight from source.

email me if you want a ready to roll time series.

Greg Goodman

said:Your earlier hadSST plot showed a clearer signal than the land+sea hadCRUT. Global averages tend to muddy the water already if we are looking basic mechanism. Mixing land and sea further reduces S/N ratios.

If there is century scale damping in the system it is almost certainly to be found in the thermal mass of the oceans. Residual atmospheric CO2 is primarily determined by how much the oceans absorb. Also dT/dt of land SAT is very close to twice that of SST. Adding the two is probably corrupting the relationship you are modelling.

All reasons I tend to focus on SST rather than mixed land-sea amalgams.

J Martin

said:Time to get out your winter woollies for the next few decades.Can you project it forwards to say, 2050, or even 2140 ?

How far down will the same formula take us and when ?

Two other projections you have made;

This graph shows a .5 C drop to 2040

from

https://montpeliermonologs.wordpress.com/2013/09/

this graph projects a 1.0 degree drop to 2040 and rising thereafter

from

http://wattsupwiththat.com/2013/09/11/digital-signal-processing-analysis-of-global-temperature-data-suggests-global-cooling-ahead/

And is your SSA technique likely to give a more accurate projection than a Fourier analysis of the last 400 years projected forward another 130 years ?

Jeff Patterson

said:I learned a lesson about making forecasts with the one you mentioned. They become the focus (even when you say they are just for fun and only as good as the underlying assumptions, which is to say meaningless) and cause readers to lose sight of the bigger picture. The analyses here (and the HD analysis for that matter) all show that the CAGW hypothesis is not supported by the data which makes forecasting an exercise in futility. The climate will do what it has always done, evolve in unpredictable ways within the bounds set by nature, unperturbed by man’s influence.

Speaking of bigger pictures, let’s suppose I’m wrong about the triggering event and have actually detected the onset of AGW in 1880. Since then, by the IPCC own model, CO2 has doubled

~~7 times~~once. If we completely ignore the ocean oscillations and ignore the fact that each doubling effects warming less than the last, and we attribute the entire rise from 1910 to 2000 (about 1 degC) then the climate sensitivity to each CO2 doubling is about~~1/7 = .14~~1 degC per doubling. Between now and 2100 the IPCC model projects 3.2 more CO2 doublings which equates to an additional~~.5~~3.2 degC attributable to CO2. [But if we remove the PDO+AMO+? oscillations, the true sensitivity is more like .4 degsC per doubling]. The IPCC can’t have it both ways. If the effect of CO2 became detectable at turn of the last century, the climate sensitivity to it must be exceedingly small. If on the other hand, the climate sensitivity is as large as they say, then as I pointed out in the Haystack article, it would be easily detectable.Greg Goodman

said:“And is your SSA technique likely to give a more accurate projection than a Fourier analysis”

Neither FA nor SSA produce projections outside the domain of the data.

Both are techniques for analysing existing data (only).

J Martin

said:I can accept that within the somewhat random-ish nature of our climate system any model that is used to project forwards will break down at some point. But to me, the most important point of a model is to provide projections, unreliable though they must inevitably be.

That a number of analyses point to cooling is a tantalising prospect that will destroy all remaining credibility of the IPCC should cooling arrive, as afaik they completely ignored that possibility in their latest religious proclamation AR5.

Greg Goodman

said:You don’t seem to get the point. Doing a FA or SSA does NOT provide a “model” with predictive ability. It described the data you have.

Jeff’s “170 year” FA is erroneous since the FT is only defined for a stationary signal , not a ramp. If you do FA of a ramp it will by definition produce a repeat of that same ramp over later and earlier periods. This in NO WAY suggests, predicts or “models” that is what the system will do.

What Jeff has done here is investigate whether the last 150 years matches a second order response to a step change in forcing.

It turns out this model fits quite well. The current part of that response is a steady rise in temperature. That means unless there is another change in forcing it will continue to rise. The harmonic component will neutralise the rise for the next 15 years or so then it will being to add to the rise.

However, even assuming that this 2nd order model is an accurate description of the system response , that still tells us NOTHING about whether there will be another change in the near or distant future nor in what directions such a change will be. It may give us an idea how the system would respond to such a change.

Hadley “corrections” to the data are as big as the most of the 20th c warming and include a 67 and 184 year component.

http://climategrog.wordpress.com/?attachment_id=48

This is close to 170 year of Jeffs first analysis and the 60 year harmonic component added to the 2nd order response here.

This is why I suggested that he does the same processing on the unadjusted data.

Jeff Patterson

said:I’ve read some peer reviewed papers that claim SSA shows forecasting skill to about 3*L/4 where L is the window length. I’ll try and dig them up.

Greg Goodman

said:SSA or any other data analysis can not have predictive ability. What may have predictive ability is the assumption that harmonic analysis of previous years reveals a pattern that has bearing on future climate.

This is a function of the climate system, not the data analysis, which is agnostic. Though some analysis methods will obviously be better than others. It is important to realise where any predictive ability comes from.

I think there are harmonic components in climate which means this assumption will work over short range. Though defining “short” and putting uncertainty figures on that is not trivial.

A CO2 signal , if correctly detected, will have predictive ability since we know what is likely to happen with future CO2 emissions. So I think the most useful exercise is to work on the available data to determine whether the CO2 signal is a significant part of the overall variability.

If “most of” turns out to be 80% we have good predictability. If it’s about 50% it’s still useful though leaves a lot to things we don’t understand.

If it’s 25% – 33% (which is my personal “expert opinion consensus” range, it is enough to counteract the downward trend of the Holocene which is probably a good thing but leaves the future largely dominated by things we do not understand.

Jeff Patterson

said:SSA and FA are fundamentally different. FA is a memoryless analysis while SSA is most akin to adaptive (Kalman) filtering. It has memory (L taps) which is “charged” at the end of the analysis. This represents information in the pipeline so to speak that can be teased out by extending with random data from a process matched to the data. As the real information becomes a smaller and smaller percentage of the filters content, the uncertainty grows.

Greg Goodman

said:http://climategrog.files.wordpress.com/2013/10/icoads_monthly1850.doc

This is straight ascii text file, rename as .txt if necessary.

Jeff Patterson

said:Thanks. Is the corrected ICOADS data or the raw data? I’d really like to get my hands on the corrected data Judith used in the article you pointed me to. Do you know if that is available? BTW, my decimation technique which preserves transient behavior requires monthly data.

Greg Goodman

said:IFASK, there is no ‘correction’ in ICOADS. It is ship and buoy data, the only processing done is 2×2 degree gridding .

Judith Curry’s article ( BEST team in fact ) was based on the BEST land SAT record, so totally separate from SST.

Both are available from KNMI climate explorer, if you want to go direct.

What I posted above is produced from that source with “at least 2% ” coverage selected. That means some areas can (will) be under represented in earlier years leading to potential geographical bias. I have an awk script that trims off early sparse periods with significant breaks and does some in filling where just a couple of month breaks occur. That’s virtually nothing on the global data and a little more if you break down to ocean basin level.

In-filling ( which is minimal ) looks to same month of previous or following year. I’ll post that if you would like to see it in detail.

So what I’ve linked to is continuous monthly data.

Greg Goodman

said:“Speaking of bigger pictures, let’s suppose I’m wrong about the triggering event and have actually detected the onset of AGW in 1880. Since then, by the IPCC own model, CO2 has doubled 7 times.”

What?!

IPCC assumptions are 280 ppmv pre-industrial, we have not doubled once on that so far. If you work out CS on that basis you will scare even Al Gore!

I don’t think there’s much chance that what you have is all AGW onset since that would assume the late 19th c. downward trend was typical of the longer term record, which it is not.

I’ll be interested to see how much of the pattern you found is a result of Hadley ‘corrections’. I don’t know what way that will go.

If the WWII glitch is too disruptive to your method, you could try subtracting a fixed 0.34K as I did in my Curry article:

judithcurry.com/2012/03/15/on-the-adjustments-to-the-hadsst3-data-set-2

You can see this is effectively what Hadley now do but they are sliding in -0.5K from 1920-1975 as well.

http://climategrog.wordpress.com/?attachment_id=48

Jeff Patterson

said:My bad. V. Pratt in his model says the IPCC CO2 model gives a doubling every 28.6 years. I used that figure blindly without checking it against the data. I’m not sure where the disconnect is. Maybe the 28.6 year doubling figure starts at some later date. Thanks for the correction.

The SSA chokes on the raw ICOADS data. It ground for an hour before I aborted it. It’s odd, I’ve never seen it not be able to converge. It may be that the covariance matrix is near singular due to the WWII glitch but that’s just a guess.

Greg Goodman

said:OMG that disingenuous Pratt. I wasted about a month listing all that was wrong with his crock of shit. It took me a week before he could admit he’d got his filter wrong.

What he has there is _excess_ CO2 doubling every … x years.

Not the same thing as total CO2. You need to apply something like his formula to get the forcing. There may be some mileage in that , rather than straight linear but the rest was a joke. He got pretty much taken apart on climate etc.

His poster is full of false claims and deception. I don’t wish to go into all that again.

Greg Goodman

said:Try subtracting 0.34 for the war years. If you’re working in d/dt that will be causing a +/ve then -ve spike that may be messing up convergence.

If you plot and zoom in it’s obvious which months it is.

In ICOADS v2.4 it was a step change from one month to the next. They merged in some UK Admiralty data into v2.5 that blurred the edges a bit.

I don’t know whether that goes someway to correcting sampling imbalance or simply muddies the waters. I suspect the later.

There were notable changes in maritime traffic and routing during WWII. I don’t see my -0.34 hack as any worse than their speculative changes.

It’s certainly better than doing nothing and should not have significant impact even if it’s not quite right.

For reference in hadSST2 they simply did -0.5 from one day in 1946 onwards. Bang, fixed it 😉 That lasted about 20 years until in hadSST3 they slide the same 0.5 in over about 20 years to smooth it to the eye.

IMHO that was no more justified but less obviously wrong to the eye.

McIntyre had blasted them for it and they felt they had to do something but seems they did not want to change the TS everyone was working to, so they came up with a “right for the wrong reason” explanation and found another justification to do the same thing.

I assume you can subtract over few years easy enough. If not I will have something I can did out.

Greg Goodman

said:“The correlation of the convolution with the data is excellent until about 1970 until the kernel end-of-record effects become apparent.”

I would point out that the divergence starts around 1950 which is the Met office adjustment I am questioning. Another reason to look at ICOADS.

What do you mean about end-effect? What is the length of the kernel you refer to?

Jeff Patterson

said:0 is 1850 on the graph. To my eye the divergence starts around 1975 (sample 125).

The kernel is just the discret-time version of equation 2a evaluated with t=t + τ (i.e. with the step moved to t=0). The kernel length is 70 years (compromise to capture the transient response with as short a “filter” as possible) so the edge effects start at K/2 or 35 years from the end.

Greg Goodman

said:Thanks for the detail.

There should be no “end effects” in a convoltuion since is starts and ends a half window from each end. Here you have earlier data , so just the late end data.

I don’t know what you are padding the SSN with, are you doing this in a spread sheet or something? It look rather like you run off the end of data. This reminiscent of the protracted arguments I had with Pratt about his running his filter convolution into the buffers.

Either you have to pad with something or you truncate the kernel, which means it is not longer the 2nd order resp. that you carefully derived.

Since the latter bit has no analytic value or legitimacy it should not be there.

There’s little point in showing the result of an incorrect calculation only to have to dismiss it as “end effects”. There is only one end effect in convolution : the end of the data. The rest is perfectly accurate.

If you truncate correctly the end divergence (140 onwards) just isn’t there to be explained.

The divergence I commented on is a divergence first visible in rate of change around 100-105 (1950-55) . This may be coincidence but that is about where Hadley start to phase in a 2% per year change over that replaces the earlier 0.5 down step in 1946.

I’m wondering whether without this your convolution would not be a much better fit to the post-war data.

Hence my interest in ICOADS as an input TS.

Jeff Patterson

said:I’m using

circularconvolution. Instead of padding, at each step the last entry becomes the first. the first becomes the second etc. The sunspot data starts at 1750 so by 1850 the filter is fully charged but at about 1975, the filter is seeing the wrapped samples and so diverges.I’m also anxious to try your corrected data but work beckons. I might not be able to post the results for a day or two.

Greg Goodman

said:” The sunspot data starts at 1750 so by 1850 the filter is fully charged but at about 1975, the filter is seeing the wrapped samples and so diverges. ”

OK, if you “padding” late 20th with 1750 data , hmm. not surprised.

I see not reason to that. Just cut the line off as soon as it’s less than half window length from the end.

Yeah, it’s a bitch when work gets in the way of the really important stuff 😉

Greg Goodman

said:Interestingly I did a similar thing recently but just using a linear relaxation first order equation. It will be interesting to so what your 2nd order does.

Keep us posted.

Jeff Patterson

said:I’ve added a standard zero-padded convolution (puts filter effects at the beginning) compared to your corrected data. It produces both the early 20th century dip and matches current warming quite well. Very interesting.

Greg Goodman

said:I still don’t quite understand your convolutions. I would expect no problem at the beginning since SSN predates the SST record. I would expect it to have to stop 35 years before the end of SST when the 70 kernel bumps up against the end of SSN data.

I don’t really understand why you are ‘allowed’ to shift the result but I may be missing something. It does seem to fit well.

Am I correct in thinking this is still the kernel you fitted to hadSST3 rather than a full rework tuned to the icoads time series?

That could explain lesser variability in late 19th c.

Greg Goodman

said:PS. I prefer to call this data adjusted rather than corrected. If I use the latter it is in quotation marks, indicating an ironic use of the term. 😉

I think it is a credible adjustment , I would not claim it is correct.

Greg Goodman

said:“I’d really like to get my hands on the corrected data Judith used in the article you pointed me to. ”

We may be talking a cross purposes here. What article are you referring to?

Judith did an article about the BEST paper of which she was a co-author.

I wrote an article on Judith’s site about hadSST “corrections”.

Greg Goodman

said:Here is ICOADS with -0.34K war-time ‘correction’ and 12mo triple RM filter to remove annual and faster variation.

http://climategrog.wordpress.com/?attachment_id=544

There is still a small peak around 1940 but this is likely to be genuine climate. There was a strong El Nina around that time and IIRC an unusually close perihelion approach.

Hopefully your processing will be able to converge on that . If it still doesn’t I suppose the conclusion is that without the Hadley fiddling it not longer fits that model.

My guess is it will now converge but with somewhat different params , should be interesting.

Greg Goodman

said:plot of above

http://climategrog.wordpress.com/?attachment_id=546

Your kernel will be slightly different but already it looks closer to your convoluted SSN.

Greg Goodman

said:I would expect this to require refitting the 60 y etc. components via SSA since this will likely be smaller magnitude.

It would then be good to compare to your figure 2 which showed the fitted response to both hadCRUT4 and hadSST3.

ICOADS will probably show a different initial negative dT/dt as well as different damped response.

Hopefully when all retuned bits gets added back together it will fit a bit better.

Whether you use kernel or formula, I don’t see how you can run to end since there is a significant response delay in the system you need have data ahead of where you end the convolution line. The truncated kernel made more sense to my way of thinking (possibly just familiarity) though there must be a way to figure out where you need to stop an equation because it does not have enough read-ahead data.

If you have time to post the impulse response (kernel) that may help visualise what’s going on.

Greg Goodman

said:Land temp changes about twice as fast as sea. So hadCRUT will be more variable. hadSST has the huge ‘correction’ so that has IMO too much 60 y amplitude as well.

Your initial ‘modes’ will need redoing for this data to get a meaningful fit.

Greg Goodman

said:Last one (at time of writing) seems to be about what I would expect of cumulative effect of SSN, but haven’t we lost the modal components?

Greg Goodman

said:Yeh, I knew it would tie up better, getting interesting.

Your initial caption read: “Best-fit eqn (2a) + SSA[L=82,k=3-6] vs. Hadcrut4 NH data”

It seems the last few plots are just eqn (2a) , what happened to the SSA ‘modes’ ?

I don’t buy the volcanic explanation. That’s severly hyped up by IPCC to justify exaggerated AGW. The pause is largely because the got this wrong and they’re just left with exaggerated AGW, no volcanism and it don’t work.

The other part of the pause is what you seem close to showing here and they reluctantly start to talk about in AR5: solar.

I think if you add the modal element back in you will have a better model that 34 IPCC GCMs.

If I’m miss understanding your plots please point out my error.

Greg Goodman

said:” Greg’s stack plot is more homogeneous which indicates that perhaps the data was smoothed across months.”

I provided two versions of -0.34K time series, with notes explaining what they were. It seems the one you used is the one with a 12 month R3M low-pass filter.

That is what I would have recommended because visual comparison would not be possible if the annual variation is retrained.

The idea of taking June only seem fraught with potential problems. I would have howled about that if you had noted it earlier.

Jeff Patterson

said:“The idea of taking June only seem fraught with potential problems.”

The only one I can think of is the potential for aliasing which can be neglected here because global data like this is already averaged both temporally and spatially. This means the amplitude of any high frequency noise is small and non-coherent and thus can not effect the trend we are looking for. Noise aliases to noise whose variance ca not exceed that of the original source.

Greg Goodman

said:So, if I follow what you’ve done so far, the single k=1,2 mode is similar to what you had form hadCRUT NH etc, ie close to 95 years.

The k=3,4 mode is (by eye from your graph) about 58.4 year.

Plus a linear ‘mode’ which I don’t see any figures for. I’m not clear when this linear cmpt is present or not in the various plots. It would be helpful if you were more explicit on that point.

In line with what you did last time, I would again suggest subtracting the mode you retain from data and fitting eqn2 to the residual so as to retain as much information as possible. As you found fitting one purely mathematical fn to another is less interesting.

So, if I’m following correctly what you’ve done, you are now fitting eqn2 to linear plus k=1,2 mode to derive a system response function which you then express as a convolution kernel.

You are applying that response to SSN and adding back in mode 3,4 such that SSN is providing the linear plus 95y component, not the SSA.

This is what you labelled as “Convolved SSN + SSA[L=81,k=3,4]”

At this stage (once kernel is refitted to data rather than model) it would be interesting to look at the residual of that plot.

Do we see any evidence of a volcanic signal in late 19th or 20th centuries?

What is the correlation coeff. of this model ?

We now have to ask whether Hadley processing is bias correction or “correction” bias?

Jeff Patterson

said:k =1,2 is a near perfect fit to a sine wave of period 189.5 years plus a constant offset of .04 degs.

k = 3,4 is a near perfect fit to a sine wave of period 58.4 years

As always, k=1,2 represents the trend, i.e. there is no “linear ‘mode'” in your data. The second plot in the update labeled Tueday, Oct 8, 8PM shows the SSA reconstruction using just those two modes plotted against your data. I’ve added a plot showing the remaining modes above. As you can see, their amplitudes are small and become noise-like above mode 3.

I followed the same procedure throughout. The third plot show your data minus the k=3,4 (58 yr cycle). That data is what I used to derive the eqn 2a parameters.

Correct, except there is no linear and the “trend” period is more like 190 years. The model (SSN convolved with eqn 2a parameterized to your data + the k=3,4 mode gives R=.941.

Greg Goodman

said:Now since we have some idea what the next solar cycle is likely to look like, we could also extend SSN to get another 15 years to feed in.

A rough guess could be made at the timing and amplitude of cycle 25 that would be a reasonable ball-park estimation. If we assume the regular SSA component is persistent too we can produce another 15 years on the model.

This extends the model up to beginning of the plateau and we can see whether reduced SSN in recent decades still fits data. (BTW have you shortened the kernel to 50 here?)

How does it back-cast?

SSN from 1750 will provide a convolution signal starting before 1800, adding in SSA 3,4 will give a back-cast. Does it look in agreement with any proxy histories of post LIA period?

Greg Goodman

said:Just out of interest, what is k=5,6 period? 21.years ?

Greg Goodman

said:I think the red prediction line is Hathaway’s. Don’t know if he publishes a formula.

Just needs scaling down by about 20% , then dupe for next cycle.

Looking at it , this cycle’s likely to run 2020-21 add on another cycle and you should run to near current end of temp data.

The anticipated SSN will not be too large a part of the result for most of it but will permit running convolution further. There is possibility that it runs lower than SST. A point Svalgaard uses to dismiss solar link.

Also uncorrected ICOADS does tend to run a little hotter than most other current assessments post-2000.

Greg Goodman

said:data for predicted SSN

http://solarscience.msfc.nasa.gov/images/ssn_predict.txt

Greg Goodman

said:I’ve done simplistic dupe of Hathaway’s cycle24 to assume a similar pattern for cycle 25

http://climategrog.wordpress.com/?attachment_id=547

You could graft that onto the end of you SSN data and run the convolution. Should give dates well into ‘the pause’.

Greg Goodman

said:BTW that’s updated Oct 2013 so should match current data without any scaling.down.

Greg Goodman

said:Thanks for the extra modes.

mode 2 indeed has something a round 21.6y ; mode 3 has 9.2

The former seems obviously Schwabe cycle.

The latter seems to be Scafetta’s and BEST’s circa 9.1 and is of lunar origin.

both modulated by something around 220y to judge by eye.

I have no understanding of properties of SSA so I don’t know how meaningful these long periods are, data related or an artefact of dataset length.

However, it does tie up with Gomez Dome d18O study also reported (well actually avoided reporting it but showed it):

I don’t think ICOADS is without biases and errors but this is proving interesting.

If you have actual numbers for those modes figures would be useful for reference.

Greg Goodman

said:“I don’t know how to check this since it is a SST record but it seems to match the BEST land temp record pretty well.”

Well the trend is heading high and it has one bump for two in BEST, but at least with the major troughs in the right place, it’s not completely getting lost.

There was some heavy volcanism in the early period to that may redtress things somewhat.

Troughs in the residual seem correctly placed for activity either side of the start of 20th c. It would be interesting to see whether Willis’ 0.3 K/W/m2 works better than classic values around 3.0

The distribution plot of the residual looks interesting too. It look like two almost perfect gaussian distributions one centred on +0.5 the other on -0.5

This suggests that the data is flipping rapidly from one state to another, yet once on either side variation is normally distributed, ie random.

I have noted several such flips in SST in different records. True climate or sampling error?

Jeff Patterson

said:Given the uncertainty in both the early SSN numbers and the BEST reconstruction I think it’s a pretty astounding match. To my eye it is within the BEST uncertainty bands all the way back.

I believe the residual distribution is basically Gaussian + outlier “events” (like 1878, 1917 etc.) .

Greg Goodman

said:I think overall it’s quite impressive but I like to be fairly hard-nosed and realistic about these kinds of model.

Adding a volcanic signal would almost certainly improve it but Willis/BEST derived 0.3 will still leave it hot in the hind case. IPCC flavoured 3.0 probably mean you need to put IPCC flavoured AGW in too.

The hind-cast reassures me that it’s not just coincidental curve fitting and the early period will need some kind of volcanic input. Activity at that time makes Mt Pinatubo look like wet fart.

At the same time if you add in enough extra forcings , most of which will be speculative back of envelop guesses, you’re back to something like IPCC which can be ‘tuned’ to fit whatever matches your personal preconceptions.

What I like about this is it’s just 58y cosine plus SSN and it gets amazingly close in the calibration period. Neither has the SST data been adjusted in ten different ways into being something fundamentally different to observations.

On the negative side there are other data sets , like BEST and ACE (cyclones) that support the need for some post-war adjustment in icoads. There’s also US surface record that suggest 30s were on a par with today.

Several other SST records simply adopted Hadley adjustments , so are not corroborative.

If you have time, could you post the actual numbers of modes 2 and 3 ?

Jeff Patterson

said:I find the R=.94 correlation to the observed data more compelling. If you think about how convolution works, each SSN input sample produces the impulse response described by eqn 2a at the output, which doesn’t completely die out for 70 years or so. The next input sample produces it’s own response, scaled and delayed and so on The output at each sample is the sum of all these individual responses. The odds that each sum would coincidentally produce the observed temperature at each time point is exceedingly small. Correlation in itself does not imply causation, but output = input @ impulse response does.

Another way to look at it is from information theory which states that a transfer function (i.e. eqn 2a) cannot create information (because there is no contingency in it’s deterministic response), only process it. This means that 94% of the information required to produce the temperature record is contained in the SSN input. The other 6% is in the residual which to all intent and purposes is just noise.

Greg Goodman

said:I agree the correlation is impressive. I saw straight away this was going to work well with the ICOADS, that’s why I suggested it.

But it is useful to play devil’s advocate and try to anticipate any possible criticism. That is why I’m interested in pushing it to 2007 by adding predictions for cycle24/25. Does it reproduce the plateau or is the solar decline too strong?

Similarly the back-cast is one of the first things to look at and it drifts off badly. That could be used to suggest that it is to a degree a fortuitous fit.

It is interesting that your SSA finds 9 and 22 , and no 11. Many reject solar because there’s no correlation to Hale cycle and clearly a strong 9y signal will disrupt 22.

My volcano stack plots suggest there is a temporal coincidence of volcanic events with pre-existing downward trends in SST. This has led to much false attribution and hence exaggerated estimations of volcanic impact on climate.

http://climategrog.wordpress.com/?attachment_id=278

If you left in modes 2 and 3 the residual should be smaller and maybe any true volcanic signal will be more visible.

It may sharpen up the distribution plot for the residual too.

Greg

said:Thanks, curious neg. offset in model. Presumably due to not updating the kernel. Also correlation slightly worse and residual noisier

Main parameters won’t change much but maybe previous damping no longer optimal.

In principal , I would expect the more parameters you use, the closer it should fit.

It would be good to look at BEST, since that would be less contentious than my icoads-adj34.

Jeff Patterson

said:The offset is because we’re convolving against the SSN anomaly where I’ve arbitrarily chosen the zero-value to be the mean of the un-appended SSN record and adjusted the kernel offset parameter (which just moves the whole Ta curve vertically) for best match. When we appended the predicted SSN values the mean shifted slightly and I didn’t re-adjust the kernel offset. Some work will have to be done to determine the true mid-point. I need to remove the kernel offset parameter and do the match adjustment by altering the SSN mid-point. Since the offset is somewhat dependent on the SSN starting value, the analysis will need to be done over multiple time intervals to find the best average value.

The correlation coefficient (Pearson’s r) actually improved with the appended record, from .947 to .960.The R^2 value decreased from .928 to .894. But when r^2 doesn’t = R^2, it implies the residual isn’t gaussian which is certainly the case here. There are unpredictable events which for the purpose of system identification are uninteresting but whose presence adversely effect the correlation. It would be nice to see what the correlation is without them but I don’t know how to do that with rigor. I have noticed empirically that R^2 is impacted by these more than r but I’m not sure why.

I’m convinced this will not work with land records. We know they lag the SST and the correlation between them, while significant isn’t that great. This implies there is another transfer function between the SST and the land temperature which is not included in our model.

When judging the backcast, don’t forget that the underlying assumption is that the mode can be back extended that far. This is unproven.

Greg

said:“This implies there is another transfer function between the SST and the land temperature which is not included in our model.”

Land seems to warm of its own accord, though all is linked. I found best was quite a close copy of SST but with twice the dT/dt.

http://climategrog.wordpress.com/?attachment_id=219

“When judging the backcast, don’t forget that the underlying assumption is that the mode can be back extended that far. This is unproven.”

That is whole point, you look and if it’s disproven you start scrabbling for excuses 😉

I think it will be necessary to play the volcano card on early 19th , there is some fairly strong impact in BEST with a good correlation in the form (unlike SST). At least with BEST, if you calibrate on 1850 onwards there is a hind-cast check available.

Do you have any idea why including the extra modes did not reduce the residual, that does not seem correct to me. In principal, more params should produce a better fit even it’s by over-fitting the data. Also 21.5 and 9 years are credible cycles known to be in the data.

Odd.

Greg

said:Interesting that with the fake cycle 24/25 data the SSN convolution is still following nicely.

One of Lief Svalgaard’s arguments is that it cannot be a significant influence because temps have not dropped with the weaker cycle 23 and the late and low c24.

This model has enough delay that it only just peaks around 2011.

Jeff Patterson

said:I think that was to be expected. The data that come out the end of the filter is not too dependent on the first sample. The info was in the pipeline, it just needed something reasonable to shove it out. We added 20 points, which didn’t even make it half way through. I’d have been disappointed if the correlation didn’t hold up pretty well to the end.

climategrog

said:Land SAT is more volatile than SST.

Perhaps going directly to BEST would be useful since it goes further back.

It has a little more down turn than my icoads_adj34 without the redictulous step of HadSST

Maybe fit to BEST 1850 on , then compare to back-cast.

Jeff Patterson

said:“When judging the backcast, don’t forget that the underlying assumption is that the mode can be back extended that far. This is unproven.”

That is whole point, you look and if it’s disproven you start scrabbling for excuses

Maybe you misread “mode” for “model” above, otherwise you comment doesn’t make sense. The model is comprised of two components- a convolutional component and and the unexplained oscillatory modes. For proving the climate since 1850 is well modeled by these components we need to show a strong correlation, since we know the characteristics of the primary mode ( the 58 year sinusoid) remained constant throughout this period. For backcasting, it is a different story. The convolutional portion should hold up, assuming the parameters of eqn 2a are invariant. But the mode is unknown. 1sky1 claims it peters out before 1900. I don’t know if that’s true or if there is even high enough resolution data available to tell. I’d like to get Judith’s latest paper and see if she sheds light.

In any case, there are other things to like about this model. It shows the right .03-.06 deg ripple due to the direct effect of TSI that others have found, with the right phase relative to the SSN peaks. What people are missing (on the “it’s the sun” thread on WUWT for example) is that in dynamic systems, the group delay is a function of frequency. This is why the TSI cycle shows very little phase lag but the primary trend lags by twenty years or so. They’re also missing the fact that the climate is a giant integrator (actually the equation I’m using is a double integration). This means that it’s the small asymmetries over time in the SSN that are determinative, not the peak to peak value of any cycle which only has a small ( but almost immediate effect).

Jeff Patterson

said:“Do you have any idea why including the extra modes did not reduce the residual, that does not seem correct to me.”

It did improve, from .947 to .960. There’s a limit to how well we’re going to do. The amplitude of these minor modes are small. At some point the unpredictable noise in the climate itself and the inaccuracies in the data will dominate the residual.

Jeff Patterson

said:Note that mode 3 ( the 11 year SSN ripple) is in the convolutional component already so adding it in would double count it. The 21 year cycle you mentioned, is it also TSI related?

Greg

said:OK, yes I did misread mode/model.

Indeed, we don’t know whether these modes are still present further back, that what a back-cast will tell is. These SSA modes are perhaps interesting in that sense since a lot of these climate oscillations to seem to phase in and out like that sort of two frequency interference pattern.

Svalgaard recently linked to a rather old meta-study that had looked at attempts to detect 11 and 22 year signals, they all seem to fade in and out. sometimes even anti-correlating.

I don’t have any a priori about the mechanism of the solar influence. TSI, EUV-stratosphere, magnitic-AO…. If SSN can give consistent results we go digging. It is well known that SSN counting rectifies the circa 22y Schwabe cycle in to the 11yHale cycle. Detecting one or both or neither is part of the mission that will help identify mechanism.

My simplistic attempts at cosine fitting SST indicated both to be present.

As you say climate science needs to get up to speed on the very basics of systems analysis before trying to analyse systems, especially one so complex.

I see very little that gets beyond multivariate linear regression. As you say expecting to see direct linear correlation is often not correct. Not finding such is not sufficient to reject a possible driver.

Just to clarify terminology is what you have referred to as “trend” effectively mode 0 ie k=1,2, a two frequency mode, like the others? I was a bit confused at one stage when you said ‘only one mode’ which was k=3,4 but you seemed to include k=1-4 .

Greg

said:I think I see your point about double counting. That explains why adding modes 3 and 4 as well as conv SSN did not reduce the magnitude of the residual. My confusion.

Does that imply that mode 0 (k=1,2) is the only mode independent of SSN and the only mode that should be subtracted for tuning the kernel?

The corollary being that HadSST3 leads us to subtract mode 1 (k=3,4) as well implying it is not SSN, whereas icoads_adj suggests it is solar but is of smaller magnitude.

Again, if you could provide some numbers for these modes it would improve understanding.

robinedwards36

said:I wish I’d seen this thread when it was active. The maths is way beyond me, I’m afraid, but I do dabble with these climate time series and have formed opinions, especially with regard to possible step changes. If I have uderstood correctly, your work seems to me to rule these out, though i still like my old-fashioned stat quality control approach to detecting rapid change.