I’ve found the LOESS algorithm to be pretty robust. In the back of my head, the number of 4 points for the moving regression sounds familiar as a limit I’ve encountered, though most of my data sets are at least dozens of points, and I don’t think I’ve routinely gone below a double-digit N.

Stay tuned. I’ve made some enhancements to the utility that make range selection much easier. I’ll be posting the new version in a few days.

Each time I was using the contiguous range option. It can accept fewer input points for separate I/O ranges without the overflow error, but with small N the output still isn’t right.

Since you aren’t having this problem, I’ll try again tomorrow with Excel 2003 on a different PC.

1. The utility is constrained by how the RefEdit control works. You can select one cell, then use the mouse and scroll bars to find the opposite cell, nhold Shift, and click on the last cell.

1A. The latest version improves on this. If you only select the top cell of a column, the program uses the whole column, down to the first blank cell.

2. I didn’t add the command to the Chart menu bar, because I figured it was really a data kind of utility. You’ve convinced me otherwise. This will go into the newest edition, and I’ll post it sometime soon.

3. The utility uses the same number of points, but it is a one-sided fit, with points further away given a lower weighting. It’s not perfect, but it beats simple moving averages. Statisticians use this methodology, with the knowledge that there may be end effects.

Some thoughts on improvement:

1. One thing that would be better is if when selecting a range you could use Shift + Cntl + Down to select really long ranges rather than having to drag down with the mouse. Is this possible? I see that Name Manager doesn’t let you do this either – so maybe it’s an artifact of constructing such addins?

2. In excel 2007, the PTS menu disappears from the addins ‘menu commands’ if you select the chart. For a moment, I thought it was suddenly uninstalled from my machine.

3. The function can also give a misleading match at each end of the series, because it has less data points with which to perform a regression on points in a moving range around the X value than it does everywhere else (and it might be substituting zero values for any missing data here, but I’m not sure). For an example of how a sine wave is affected, see my uploaded speadsheet at http://cid-f380a394764ef31f.skydrive.live.com/self.aspx/.Public/LOESS%20on%20SineWave.xls

To counter this, you could amend the function so that either:
- if it goes to perform a regression with say a moving range of 6 points then it doesn’t return data for the first 3 or the last 3 values; or
- in the above case, it substitutes any missing data from data it does have. For instance, in the above case, for the first point it would use the first 3 numbers twice instead of a moving range that has missing data points.

Alternately, the user could simply amend their range so that they use a subset of the LOESS function output that doesn’t include end values. But they might not be aware of this.

Regardless, this is staying on my ribbon! THanks Jon

Since alpha is a fraction, the number of points in a LOESS analysis of larger data sets is larger than that for a smaller data set, given a constant alpha. If you get a reasonable smoothing for a week of data for a given alpha, note the number of points used for smoothing. Then carry out LOESS smoothing on the larger data set, using the same number of points (and a correspondingly smaller alpha).

I’m guessing this is an artifact of who the LOESS technique works, rather than due to your fine utility.

In your original LOESS post, you say “For each X value where a Y value is to be calculated, the LOESS technique performs a regression on points in a moving range around the X value, where the values in the moving range are weighted according to their distance from this X value.” I’m guessing that the larger your dataset, the larger the moving range around the X value.

I was using it on some half-hourly electricity metering data. When I used the LOESS utility to smooth a week’s worth of data (336 points), I get a great looking curve compared to the rather ‘jagged’ appearance of the raw data. But when I use it to smooth the whole year (17520 points), I get a flat line. I was expecting to get a similar effect for the whole year’s worth of data as I did with a week’s worth…that is I thought I’d still see some swings etc.

Is this how you’d expect LOESS to work with such a large dataset?

Regards

Jeff